Skip to main content
Loading

Storage Format Upgrade in 6.0 Release

In server 6.0, the Aerospike Database internal storage format was changed to include a four-byte record end mark. This change addresses a potential, but unlikely, data-loss condition where a partial write block persisted, causing a record to be corrupted during an unclean shutdown.

When upgrading from version 5.x (or 4.9), each persisted namespace storage device, with the exception of PMem backed namespaces, must be erased and its Aerospike header wiped (zeroized). The header is stored in the first 8MiB of the device.

note

PMem-backed namespaces do not need to be zeroized because they do not require the use of an end-mark.

info

Versions 6.0 and pre-6.0 are cluster-compatible, so a rolling restart upgrade with mixed versions is supported. Keep a copy of your original aerospike.conf configuration file in case of an unlikely event requiring a downgrade.

caution

The record overhead increased by 4 bytes in server 6.0, so records which previously met the write-block-size or max-record-size value might exceed that limit. The former will cause replication and migration issues, while the latter will only cause master writes to fail that would have previously succeeded. Check your object size histogram to identify any records at the write-block-sizelimit.

See Important pre-upgrade considerations for more information.

Configuration changes​

The following configuration items have changed in 6.0:

  1. scan-max-done has moved to query-max-done.
  2. scan-threads-limit has moved to query-threads-limit.
  3. background-scan-max-rps has moved to background-query-max-rps.
  4. single-scan-threads has moved to single-query-threads.

The following configuration items have been removed in 6.0:

  1. query-threads
  2. query-worker-threads
  3. query-microbenchmark
  4. query-batch-size
  5. query-in-transaction-thread
  6. query-long-q-max-size
  7. query-priority
  8. query-priority-sleep-us
  9. query-rec-count-bound
  10. query-req-in-query-thread
  11. query-short-q-max-size
  12. query-threshold
  13. query-untracked-time-ms
  14. batch-without-digests

Important pre-upgrade considerations​

Aerospike recommends that you check for records that approach the configured write-block-size before you upgrade. Due to the overhead increase of 4 bytes per record in server 6.0, some records could exceed the write-block-size after the upgrade causing write transactions to fail and migrations to get stuck.

tip

If you are upgrading from a version prior to server 5.7, consider upgrading to server 5.7 before you upgrade to server 6.0 so that you can take advantage of max-record-size. Introduced in server 5.7, max-record-size provides a way to configure an extra threshold to prevent adding records above the configured size. It can be used to account for the extra overhead and prevent adding records that would then breach the write-block-size.

Find potential problem records before you upgrade​

Here are different ways to check for record sizes that may cause problems:

  • Leverage the object-size and object-size-linear histograms.
  • Run a query with a filter expression to identify records that have a size equal to write-block-size. An example Python script (using the Python client) is provided at aerospike-examples/6.0-record-size-checker. This script handles uncompressed records, and has a configurable compression ratio variance threshold, which it uses to identify compressed records that will exceed the write-block size.
# after editing global dry_run = True
$ python3 main.py
2023-08-02 17:44:53,097 [INFO]: Scanning node: BB9C00F800A0142
2023-08-02 17:44:53,098 [INFO]: Node BB9C00F800A0142 does not have compression enabled.
2023-08-02 17:44:53,098 [INFO]: Checking for records of compressed size larger than 1048560 bytes
2023-08-02 17:46:59,366 [INFO]: Namespace: bar, Set: testset, Primary Key: None, Digest: fe0f17700e1b7fcc82401b535f7933667634f8bf
2023-08-02 17:46:59,366 [INFO]: Namespace: bar, Set: testset, Primary Key: None, Digest: fe0f14b652ecbaf4afc46de605d7a4a0b6452f3a
...
2023-08-02 17:46:59,366 [INFO]: Node: BB9C00F800A0142 Returned Record Count: 328633
2023-08-02 17:46:59,366 [INFO]: Node: BB93E00800A0142 Returned Record Count: 330107
2023-08-02 17:46:59,366 [INFO]: Node: BB93F00800A0142 Returned Record Count: 329541

If no records are close in size to the configured write-block-size, the upgrade should not be impacted.

Mitigations, prior to upgrading​

If you find records that are close to the size limit, consider the following options and their risks:

  • You can double the write-block-size. However, this doubles the memory used by the configured post-write-queue. If you choose to double the write-block-size, you can reduce the post-write-queue by a factor of two to keep the same memory footprint for the post-write-queue. If the use case allows it, you can also break up the records that would not fit with the overhead, and delete the original records.

  • As of 6.0, Aerospike logs the digests of records that prevent migrations for some specific failure types. Failures to migrate due to an excessive record size are logged under the drv_ssd context at the DETAIL level. Refer to the Server Log Reference for details on dynamically changing the log level.

    Make sure to revert to the default INFO level after a few seconds to avoid polluting the logs and risking running out of log disk space.

How to fix a stuck migration​

When upgrading from server 5.7

  1. Use max-record-size configuration to set the value to write-block-size - 16 bytes.

  2. Identify the record(s) causing the issue by enabling detail log level:

asadm -e “asinfo -v ‘log-set:id=0;drv_ssd=detail’”

The server will log a message similar to this:

`DETAIL (drv_ssd): (drv_ssd.c:1550) {namespace} write: size 1048577 - rejecting <digest id>`

  1. Delete the record, or shorten it, using the printed digest.

  2. Issue the recluster command, to force migrations to reprocess the updated or deleted record(s).

When upgrading from a version prior to server 5.7

  1. Identify the record(s) causing the issue by enabling detail log level:
asadm -e “asinfo -v ‘log-set:id=0;drv_ssd=detail’”

The server will log a message similar to this:

`DETAIL (drv_ssd): (drv_ssd.c:1550) {namespace} write: size 1048577 - rejecting <digest id>`

  1. Delete the record, or shorten it, using the printed digest.

  2. Issue the recluster command, to force migrations to reprocess the updated or deleted record(s).

For more information, contact Aerospike Support.

Upgrade steps​

info

Be familiar with the general guidelines for upgrading a cluster for the common steps involved in an Aerospike cluster upgrade.

caution

Persisted data must be deleted prior to starting a node with server 6.0. Aerospike recommends that you backup your data, or have a redundant cluster, prior to proceeding with the upgrade.

note

Namespaces with replication-factor 1 require that the node be quiesced and that migrations complete before stopping and upgrading the node. Alternatively, these namespaces may have their data restored from a backup or through XDR or other clients.

For each node in the cluster:

  1. (Optional) Quiesce the node and wait for migrations to complete. This optional step protects against the unlikely event of an irrecoverable node crash while a node has been taken out to be upgraded.
  2. Stop the Aerospike daemon.
  3. Delete the stored data for storage-engine device-configured namespaces. Delete the file if using a file. For raw devices, Initializing Solid State Drives (SSDs).
  4. If you trimmed the raw device (using blkdiscard or similar), you also need to wipe (zeroize with dd or blkdiscard -z) the Aerospike header on the device. The header takes the first 8MiB of the raw device.
  5. Start the Aerospike daemon.
  6. Wait for migrations to complete after the node joins the cluster to allow all the data to be repopulated from other nodes (assuming replication factor 2 or more). Refer to the knowledge base article, Monitoring Migrations on a Live Cluster.
  7. Proceed with the next node.

Post upgrade​

The truncate privilege is now a standalone granular permission, and no longer part of the write privilege. Users representing applications that perform truncates should be granted the truncate privilege to one of their roles.

Last updated on