500 Internal Server Error Command `encryption-key.hash.get' Failed During Reading of Encryption Key.

This page helps yous understand and resolve error letters written to stderr or your logs.

Topic	Bulletin
Client connection	connectedness refused
Client connection	node is running secure mode, SSL connection required
Transaction retries	restart transaction
Node startup	node belongs to cluster <cluster ID> simply is attempting to connect to a gossip network for cluster <some other cluster ID>
Node configuration	clock synchronization error: this node is more than 500ms away from at to the lowest degree half of the known nodes
Node configuration	open up file descriptor limit of <number> is under the minimum required <number>
Replication	replicas declining with "0 of 1 store with an attribute matching []; likely non enough nodes in cluster"
Split failed	split failed while applying backpressure; are rows updated in a tight loop?
Deadline exceeded	context deadline exceeded
Incremental backups	protected ts verification error...
Ambiguous results	result is ambiguous
Import primal collision	checking for key collisions: ingested key collides with an existing 1
SQL memory budget exceeded	memory budget exceeded

connexion refused

This message indicates a customer is trying to connect to a node that is either not running or is non listening on the specified interfaces (i.eastward., hostname or port).

To resolve this outcome, practice one of the following:

• If the node hasn't yet been started, start the node.
• If you lot specified a --mind-addr and/or a --advertise-addr flag when starting the node, you must include the specified IP accost/hostname and port with all other cockroach commands or alter the COCKROACH_HOST environs variable.

If y'all're not sure what the IP accost/hostname and port values might have been, y'all tin await in the node's logs.

If necessary, you can also close down and so restart the node:

                          $              cockroach outset              [flags]                      

node is running secure fashion, SSL connexion required

This message indicates that the cluster is using TLS encryption to protect network advice, and the customer is trying to open up a connexion without using the required TLS certificates.

To resolve this outcome, apply the cockroach cert create-customer command to generate a client certificate and cardinal for the user trying to connect. For a secure deployment walkthrough, including generating security certificates and connecting clients, encounter Manual Deployment.

restart transaction

Messages with the fault code 40001 and the string restart transaction indicate that a transaction failed because information technology conflicted with another concurrent or recent transaction accessing the same information. The transaction needs to be retried by the client. For more data about how to implement client-side retries, run into client-side retry handling.

For more information about the different types of transaction retry errors such every bit "retry write too quondam", "read inside uncertainty interval", etc., meet the Transaction Retry Fault Reference.

node belongs to cluster <cluster ID> but is attempting to connect to a gossip network for cluster <another cluster ID>

This message normally indicates that a node tried to connect to a cluster, but the node is already a member of a different cluster. This is determined by metadata in the node'due south data directory. To resolve this issue, do one of the following:

• Cull a different directory to store the CockroachDB data:

                                    $                  cockroach starting time                  [flags]                  --shop                  =[new directory]                  --join                  =[cluster host]:26257                              

• Remove the existing directory and beginning a node joining the cluster again:

                                    $                  cockroach first                  [flags]                  --join                  =[cluster host]:26257                              

clock synchronization error: this node is more than than 500ms away from at to the lowest degree half of the known nodes

This fault indicates that a node has spontaneously shut down considering it detected that its clock is out of sync with at least half of the other nodes in the cluster by fourscore% of the maximum offset immune (500ms past default). CockroachDB requires moderate levels of clock synchronization to preserve information consistency, and so the node shutting down in this fashion avoids the risk of consistency anomalies.

To prevent this from happening, you should run clock synchronization software on each node. For guidance on synchronizing clocks, see the tutorial for your deployment environs:

Environment	Recommended Approach
Manual	Apply NTP with Google's external NTP service.
AWS	Utilise the Amazon Fourth dimension Sync Service.
Azure	Disable Hyper-V time synchronization and use NTP with Google's external NTP service.
Digital Body of water	Use NTP with Google'southward external NTP service.
GCE	Use NTP with Google's internal NTP service.

open file descriptor limit of <number> is under the minimum required <number>

CockroachDB can use a large number of open file descriptors, frequently more than than is available past default. This message indicates that the automobile on which a CockroachDB node is running is under CockroachDB's recommended limits.

For more details on CockroachDB's file descriptor limits and instructions on increasing the limit on various platforms, see File Descriptors Limit.

replicas declining with "0 of 1 shop with an attribute matching []; probable not enough nodes in cluster

When running a single-node cluster

When running a single-node CockroachDB cluster, an error most replicas failing volition eventually show upwards in the node's log files, for example:

            E160407 09:53:50.337328 storage/queue.go:511              [replicate] seven replicas failing with              "0 of one store with an attribute matching []; probable not enough nodes in cluster"                      

This happens because CockroachDB expects three nodes by default. If y'all do not intend to add additional nodes, yous tin stop this error by using Alter RANGE ... CONFIGURE ZONE to update your default zone configuration to expect simply one node:

                          # Insecure cluster:              $              cockroach sql              --execute              =              "ALTER RANGE default CONFIGURE ZONE USING num_replicas=1;"              --insecure                      

                          # Secure cluster:              $              cockroach sql              --execute              =              "ALTER RANGE default CONFIGURE ZONE USING num_replicas=1;"              --certs-dir              =[path to certs directory]                      

The zone'south replica count is reduced to 1. For more information, see ALTER RANGE ... CONFIGURE ZONE and Configure Replication Zones.

When running a multi-node cluster

When running a multi-node CockroachDB cluster, if you see an fault like the 1 higher up about replicas failing, some nodes might not be able to talk to each other. For recommended actions, see Cluster Setup Troubleshooting.

split failed while applying backpressure; are rows updated in a tight loop?

In CockroachDB, a tabular array row is stored on deejay as a primal-value pair. Whenever the row is updated, CockroachDB also stores a distinct version of the key-value pair to enable concurrent request processing while guaranteeing consistency (run across multi-version concurrency command (MVCC)). All versions of a key-value pair belong to a larger "range" of the total key infinite, and the historical versions remain until the garbage drove period defined by the gc.ttlseconds variable in the applicable zone configuration has passed (25 hours by default). Once a range reaches a size threshold (512 MiB by default), CockroachDB splits the range into two ranges. Withal, this message indicates that a range cannot be divide equally intended.

One possible crusade is that the range consists only of MVCC version data due to a row being repeatedly updated, and the range cannot be split because doing and so would spread MVCC versions for a single row across multiple ranges.

To resolve this outcome, make certain y'all are non repeatedly updating a single row. If frequent updates of a row are necessary, consider i of the following:

• Reduce the gc.ttlseconds variable in the applicative zone configuration to reduce the garbage collection period and forbid such a large build-up of historical values.
• If a row contains large columns that are not existence updated with other columns, put the large columns in dissever column families.

context deadline exceeded

This message occurs when a component of CockroachDB gives upwards considering information technology was relying on another component that has non behaved as expected, for example, another node dropped a network connection. To investigate further, await in the node's logs for the master failure that is the root cause.

protected ts verification error...

Messages that brainstorm with protected ts verification error… bespeak that your incremental fill-in failed because the data you are trying to fill-in was garbage nerveless. This happens when incremental backups are taken less frequently than the garbage collection periods for whatever of the objects in the base of operations fill-in. For instance, if your incremental backups recur daily, but the garbage collection flow of i table in your fill-in is less than one day, all of your incremental backups volition neglect.

The error message will specify which office of your backup is causing the failure. For instance, range span: /Table/771 indicates that table 771 is part of the problem. Yous can so inspect this tabular array by running SELECT * FROM crdb_internal.tables WHERE id=771. You can too run SHOW ZONE CONFIGURATIONS and look for any gc.ttlseconds values that are ready lower than your incremental fill-in frequency.

To resolve this outcome, accept a new full backup subsequently doing either of the post-obit:

• Increase the garbage collection period by configuring the gc.ttlseconds replication zone variable, or
• Increase the frequency of incremental backups.

result is ambiguous

In a distributed organization, some errors tin can have ambiguous results. For instance, if you receive a connection airtight error while processing a COMMIT statement, you cannot tell whether the transaction successfully committed or not. These errors are possible in whatever database, merely CockroachDB is somewhat more likely to produce them than other databases because cryptic results can be caused by failures betwixt the nodes of a cluster. These errors are reported with the PostgreSQL error lawmaking 40003 (statement_completion_unknown) and the message effect is ambiguous.

Ambiguous errors can be caused by nodes crashing, network failures, or timeouts. If yous experience a lot of these errors when things are otherwise stable, look for functioning bug. Note that ambiguity is only possible for the last statement of a transaction (COMMIT or RELEASE SAVEPOINT) or for statements outside a transaction. If a connection drops during a transaction that has not notwithstanding tried to commit, the transaction volition definitely be aborted.

In full general, you should handle ambiguous errors the same manner as connectedness closed errors. If your transaction is idempotent, it is safe to retry it on ambiguous errors. UPSERT operations are typically idempotent, and other transactions tin can be written to exist idempotent by verifying the expected state before performing any writes. Increment operations such as UPDATE my_table Fix x=ten+1 WHERE id=$one are typical examples of operations that cannot easily be made idempotent. If your transaction is not idempotent, and then y'all should decide whether to retry or not based on whether it would be better for your awarding to apply the transaction twice or return an fault to the user.

checking for cardinal collisions: ingested central collides with an existing 1

When importing into an existing table with IMPORT INTO, this error occurs because the rows in the import file conflict with an existing primary primal or another UNIQUE constraint on the tabular array. The import will fail as a result. IMPORT INTO is an insert-only statement, then you lot cannot use it to update existing rows. To update rows in an existing tabular array, use the UPDATE statement.

retention budget exceeded

This bulletin usually indicates that --max-sql-memory, the memory allocated to the SQL layer, was exceeded by the operation referenced in the error. A memory budget exceeded mistake also suggests that a node is close to an OOM crash, which might be prevented by declining the query.

If you find queries that are consuming too much retention, cancel the queries to free up retentiveness usage. For information on optimizing query operation, meet SQL Performance Best Practices.

Increasing --max-sql-memory can convalesce memory budget exceeded errors. However, allocating more --max-sql-memory tin can likewise increase the probability of OOM crashes relative to the amount of memory currently provisioned on each node. For guidance on configuring this flag, see Cache and SQL memory size.

For disk-spilling operations such every bit hash joins that are memory-intensive, some other solution is to increase the sql.distsql.temp_storage.workmem cluster setting to allocate more retentivity to the functioning earlier information technology spills to disk and probable consumes more than memory. This improves the functioning of the query, though at a possible reduction in the concurrency of the workload.

For example, if a query contains a hash join that requires 128 MiB of memory before spilling to disk, values of sql.distsql.temp_storage.workmem=64MiB and --max-sql-retention=1GiB permit the query to run with a concurrency of 16 without errors. The 17th concurrent instance will exceed --max-sql-memory and produce a memory upkeep exceeded error. Increasing sql.distsql.temp_storage.workmem to 128MiB reduces the workload concurrency to 8, but allows the queries to finish without spilling to deejay. For more information, see Disk-spilling operations.

Something else?

Endeavour searching the rest of our docs for answers or using our other support resources, including:

• CockroachDB Community Forum
• CockroachDB Community Slack
• StackOverflow
• CockroachDB Support Portal
• Transaction retry error reference

Was this page helpful?

Yes No

hopkinswintelpom.blogspot.com

Source: https://www.cockroachlabs.com/docs/stable/common-errors.html

Share this post