Upgrade to Aerospike Server 7.0

This page describes how to upgrade to Aerospike Database 7.0 from 6.x versions. If you are upgrading from versions prior to 6.0, see Upgrade from pre-6.0 versions.

Overview

Aerospike Database 7.0 consolidates all three storage engines into using the same flat storage format, so the amount of storage space consumed is the same regardless if the namespace uses solid state drives (SSD), memory, or Intel Optane Persistent Memory (PMem). This means that an in-memory namespace with storage-backed persistence now has the exact same write-blocks mirrored from memory to the persistence layer. To accomplish this, some configuration parameters have been added, renamed, or removed. This is also true of some metrics you use to monitor your system. Some of these changes are breaking changes: Your newly upgraded Aerospike node will refuse to start up unless you reconfigure to account for these changes. All breaking changes are listed in the 7.0 server release notes.

Important pre-upgrade considerations

Starting server 7.0, the write-block size for in-memory namespaces is hard-coded to 8MiB. If you are using an in-memory namespace without storage-backed persistence, we recommend that you check such a namespace for records that exceed 8MiB. Otherwise, some records could exceed the write-block size after the upgrade, causing write transactions to fail and migrations to get stuck. Other storage engines won't have records larger than 8MiB.

tip

If you have a storage-engine memory namespaces, you can take advantage of max-record-size ahead of upgrading to explicitly reject client writes that would then breach 8MiB after upgrading to server 7.0 or later.

Pre-upgrade tasks

1. (Required) - Dynamically set cluster-name on all nodes in the pre-7.0 cluster. The cluster name is mandatory in server 7.0, meaning that the name “null” is no longer assigned by default.

   • The maximum length of a cluster name in server 7.0 is 63 characters (previously 64).
   • If the cluster is used as a destination for Cross-Datacenter Replication (XDR), set the cluster name as the same name used in the XDR configuration. This helps with the monitoring stack.

2. (Optional) - Upgrade to Monitoring Stack 3.0 (Grafana and Prometheus Exporter) or later.

3. Download the server 7.0 software. See the download FAQs.

4. Install Aerospike Tools 10.1 or later. You need asadm and asconfig from Aerospike Tools 10.1 or later to use the following steps.

5. Install the monitoring stack and import your pre-7.0 monitoring stack configurations. The new dashboard displays information from servers 6.x and 7.0 on the same sceen.

Upgrade

1. Make a copy of your current configuration file.
2. Run the following command to generate a configuration file which includes all dynamic configuration of the running node, and convert the config file to YAML. You can also specify the output location of the generated config and YAML files.

tip

You can get a list of invalid configuration details by running the following command.

asadm -h 172.17.0.4 -e 'generate config with 172.17.0.4' | asconfig validate -a '7.0.0'

3. Verify that the configuration file contains the correct cluster name that you specified.

4. Make the required changes.

5. Install server 7.0 from the package you downloaded.

6. Restart the Aerospike node. Once the node joins the cluster you can configure the next cluster node. If the nodes use exactly the same configuration, you can reuse the previous configuration information.

   note

   You must upgrade all nodes in the cluster before deploying applications that explicitly persist Map indexes (a server 7.0 feature).

Converting in-memory namespaces

If your Aerospike cluster includes in-memory namespaces, you need to reconfigure them when you move to server 7.0 or later. See Configuring Namespace Storage for more details.

Starting in server 7.0, in-memory data storage is pre-allocated and static, rather than progressively growing and bound by the now obsolete memory-size configuration parameter. The amount of memory consumed by indexes in your current, pre-7.0 namespace is reflected in the metrics memory_used_index_bytes, memory_used_sindex_bytes, and memory_used_set_index_bytes. After upgrading a cluster node, the namespace indexes consume the same amount of memory as before, out of the system memory not pre-allocated for namespace data storage. When calculating the data storage size (according to the sub-sections below), make sure to leave room for your indexes, plus expected growth.

Converting in-memory with storage-backed persistence

To convert a pre-7.0 data-in-memory namespace with storage backing, first determine the size of your data storage.

The rule-of-thumb approach is to divide the device_used_bytes metric by 0.6. That gives you the rough storage size estimate with 60% used and the remainder left for defragmentation. Add extra room for your namespace data to grow to arrive at your data storage estimate.

The memory consumed by your indexes (see the "Converting in-memory namespaces" section above), plus data storage, should fit within your host machine’s RAM and within the previously declared memory-size.

1. Remove the obsolete memory-size configuration line.
2. Change the storage-engine from device to memory.
3. Remove the obsolete data-in-memory true configuration line.
4. If configured, rename the write-block-size configuration parameter to max-record-size. The in-memory write-block is hard-coded to 8MiB.
5. Adjust the persistent storage associated with the namespace. Starting in server 7.0, Aerospike pre-allocates in-memory data storage to match the size of the persistent storage (filesize times the number of files or the total storage of the device partitions of the namespace). In earlier versions of Aerospike, we recommended that persistent storage be a 4x multiple of the memory-size, but these two layers of data storage are now mirrored 1:1. If you have multiple files or devices, you can drop several of them from the configuration to adjust the amount of memory that will be allocated for in-memory storage on startup. Alternatively, you can erase your persistent storage. In either case, the Aerospike node will fill any missing records by migrating them over the network from other cluster nodes.
6. If your namespace has very low or intermittent write rates, consider using flush-max-ms to reduce the frequency of flushing partially filled write-blocks to the persistence layer.
7. Wait for migration to finish before continuing with the rolling restart of another node for the upgrade.

Converting in-memory with no persistence

To convert a pre-7.0 data-in-memory namespace without storage backing, first determine the size of your data storage.

The rule-of-thumb approach is to divide the memory_used_data_bytes by 0.6. That gives you the rough storage size estimate with 60% used and the remainder left for defragmentation. Add extra room for your namespace data to grow to arrive at your data storage estimate.

The memory consumed by your indexes (see above), plus data storage, should fit within your host machine’s RAM and within the previously declared memory-size.

1. Remove the obsolete memory-size configuration parameter.
2. Add a context to storage-engine memory.
3. Declare the amount of data storage to pre-allocate for the namespace in a data-size configuration line within the storage engine context.
4. Consider setting the max-record-size. Starting with server 7.0, the in-memory write-block is hard-coded to 8MiB.

Other configuration changes

Server 7.0 simplifies the configuration of eviction and stop-writes, and you may need to reconfigure accordingly. See Configuring Namespace Data Retention for more details. The following changes apply to any namespace configuration, regardless of storage engine:

1. Remove the obsolete memory-size, stop-writes-pct, high-water-memory-pct, commit-min-size from any namespace configuration. serialize-tomb-raider only applies to namespaces with storage-engine device.
2. The namespace config parameter evict-sys-memory-pct can be used as an eviction threshold for any type of storage engine, similar to how stop-writes-sys-memory-pct is a stop-writes threshold for any type of storage engine.
3. Rename high-water-disk-pct to evict-used-pct. This eviction threshold works in any type of storage engine.
4. Rename min-avail-pct to stop-writes-avail-pct. This stop-writes threshold works in any type of storage engine.
5. Rename max-used-pct to stop-writes-used-pct. This stop-writes threshold works in any type of storage engine.
6. Rename mounts-size-limit in PMem or flash index-type/sindex-type contexts to mounts-budget.
7. Rename mounts-high-water-pct in PMem or flash index-type/sindex-type contexts to evict-mounts-pct.

Upgrade from pre-6.0 versions

tip

Always review the special upgrade instructions of the interim server releases between the current version of your cluster and the target version.

• Before you upgrade from server 4.9 or 5.x, follow the instructions in Storage Format Upgrade in 6.0 Release.
• Back up /opt/aerospike/smd/security.smd in your system metadata (SMD) Directory Structure and aerospike.conf, the Aerospike configuration file.
• Find and fix any secondary index (sindex) names that are longer than 64 characters. See Upgrade to Server 6.1.
• If you're an Enterprise Edition customer, ensure that your feature-key file has the explicit feature keys needed in server 6.3 and later.

Changes and updates

For information about new features, breaking changes, and late breaking news, see the 7.0 server release notes.