Fast Restart

The Fast Restart feature in Aerospike Database Enterprise Edition (EE) enables nodes to start up and re-join their cluster much faster when they restart. A node with over one billion records will restart in about 10 seconds, as opposed to 40+ minutes. This makes cluster upgrades much faster.

How does it work?

Fast Restart (aka warmstart) uses the Linux operating system's shared memory. The server node’s primary index, along with other critical metadata, is allocated in the system shared memory, which persists even when the Aerospike server process (asd) is stopped. On restart, the new asd process reattaches to the persisted shared memory, quickly rebuilds its internal state, and re-joins the cluster. The EE asd does not need to read data from the storage drives to rebuild its primary index.

As of server 6.1, secondary indexes are also allocated in the system shared memory, making the restart process faster for namespaces with secondary indexes.

What does Aerospike Shared Memory Tool (ASMT) do?

The ASMT tool enables primary and secondary indexes to be backed up from shared memory to file system files and restored from file system files to shared memory. Warm restart can be accomplished across system restarts, thus enabling warm restart when, for example, upgrading the operating system.

What do I have to do to make this happen?

Nothing. For namespaces configured with flash storage or Data In Index, the node default behavior is to always try to fast restart, so just start up normally:

$ sudo systemctl start aerospike

You can verify in logs to see if the node performs a fast restart (also referred to as WARM restart) for a namespace.

INFO (namespace): (namespace_ee.c:361) {test} beginning warm restart

When does fast restart NOT happen?

There are some situations in which a server node will try to fast restart but determines that it cannot do so, and will switch to a “cold restart”, where all the record data is read from storage drives to rebuild the index:

• The namespace is configured as "data-in-memory". Since record data is not stored in system shared memory, all record data must be read to restore the data to memory. As of Enterprise Edition version 3.15.1.3, though, the index is not rebuilt, avoiding cases of deleted records being restored from storage in such cases (records not durably deleted while the node was not in the cluster would be brought back).

  With data-in-memory true, on server versions prior to 3.15.1.3

  INFO (namespace): (namespace_ee.c:162) {test} can't warm restart if data-in-memory
  INFO (namespace): (namespace_ee.c:243) {test} beginning COLD start

  As of server version 3.15.1.3 and later

  INFO (namespace): (namespace_ee.c:360) {test} beginning cool restart

• Changing the value of partition-tree-sprigs, for example, original value of 512 was changed to 1024:

  WARNING (namespace): (namespace_ee.c:495) {nsSSD} persistent memory partition-tree-sprigs
  512 doesn't match config 1024

• The storage drives were wiped clean while the server node was down. Very occasionally this is done, e.g. when repairing a drive with physical failures. Obviously if the record data was erased, the persisted index isn’t valid. Fortunately "cold restart" will be fast in this case — since there’s no data to read!

• The previous shutdown is not trusted. If a node stops unexpectedly, and we can’t be sure the persisted index is reliable and consistent with the stored data, it’s best to rebuild the index.

• The machine was rebooted. System shared memory survives the server process stopping, but not the machine rebooting.

• Prior to 3.13.0, if the order of namespaces in the config was changed.

If a node does switch to "cold restart", the log file will mention it, and should indicate the reason for the switch.

note

When secondary indexes are defined, the start up time of a node can significantly slow down in versions prior to server 6.1, depending on the number of secondary indexes defined and their size. If a namespace meets the conditions to fast restart but has secondary indexes defined, after the fast restart (primary index only), the namespace has to rebuild every secondary index defined, which would require looking up record values from the persistent storage.
As of server 6.1, this issue is much less significant, assuming the secondary indexes are successfully restored from shared memory during start up and therefore do not need to be rebuilt.

Refer to Cold Restart.

Can I force a “cold restart”?

Yes, although not necessary, you can manually force a “cold start” by using the command:

sudo /etc/init.d/aerospike coldstart

Can I monitor system shared memory used by Aerospike?

You can see the system’s shared memory blocks, using the command:

$ sudo ipcs -m

All blocks listed that have keys starting with "0xae" or "0xa2" are Aerospike shared memory blocks. With 6.1 and above, the shared memory blocks used for secondary indexes have keys that start with “0xa2”.

0xa2...

'ae' (as in Aerospike) are Aerospike shared memory blocks.
An instance of ASD runs with both "0xae" and "0xa2" segments. The "0xae" segments are associated with the primary indexes, while the "0xa2" segments are associated with the secondary indexes, but they are used for a single instance. If there are no secondary indexes, there are no "0xa2" segments.

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0xa2001100 163846     citrusleaf 666        1073741824 0
0xa2001000 163847     citrusleaf 666        16720      0
0xae001001 163848     citrusleaf 666        5259264    0
0xae001000 163849     citrusleaf 666        2097152    0
0xae001100 131135     citrusleaf 666        1073741824 0
------ Semaphore Arrays --------
key        semid      owner      perms      nsems

Fast Restart for Data-In-Memory Namespace

If your data-in-memory namespace is a single-bin namespace with only numeric data (integer or double), it can also take advantage of the fast-restart feature.

To do so, add the "data-in-index" configuration to your namespace stanza:

namespace <namespace-name> {
...
    single-bin true
    data-in-index true
...
}

Restart Aerospike and fast restart happens.

During the fast restart, the Aerospike process logs a message and exits if any non-numeric bin values are found while reducing the index. Otherwise, once the fast restart has completed successfully, any attempts to write non-numeric values to the namespace will fail with error code 12 (INCOMPATIBLE_TYPE).