Configuring Strong Consistency

This section discusses configuring a namespace with Strong Consistency (SC). Summary:

Get the feature key entry for Strong Consistency.
Make sure you have NTP configured.
Add config file entries to denote strong consistency namespaces.
Configure the initial roster.
Configure Node IDs with SC.
Configure Rack Awareness.

Feature key for strong consistency

Strong Consistency requires an entry in your feature-key-file. The feature key file contains a digitally signed key to enable the Strong Consistency feature. Contact Aerospike Sales for further information on Strong Consistency licenses.

Install and Configure NTP ( or other clock synchronization )

Strong Consistency mode is sensitive to clock skew and will enter 'stop-writes' clock skew exceeds 27 seconds.

Computer hardware tends to skew from true time, sometimes as much as by seconds a day. Aerospike's gossip clustering protocol will monitor the amount of skew in a cluster, and alert if the a large amount of skew has been detected. SC namespaces will stop taking writes if skew is detected at a larger interval - lower than the point where data loss occurs.

In order to avoid any issues, Aerospike recommends using a clock synchronization system. The system does not need to be exactly accurate. Second, or 10 second, accuracy is acceptable. It is also acceptable if clocks pause or go through periods of correction (accelerated clocks), as long as the total skew in the cluster is less than 27 seconds.

NTP will assist in keeping clocks synchronized. The most common method of synchronizing clocks is the common NTP protocol, which easily exceeds the granularity that Aerospike demands.

For more information, see the guide to installing NTP: How to configure NTP for Aerospike. Use the time synchronization software common to your operating system.

Configure Node IDs

By default, Aerospike derives the node-id from the server's mac address (or configured node-id-interface).

You will see in the subsequent commands for roster maintenance, use of hardware generated node ids, while convenient for Availability mode's automatic algorithms, can become cumbersome when managing Strong Consistency.

Consider using configured node-ids for your Strong Consistency cluster.

To allow friendlier node-ids, which simplify roster management, and including IDs that can be stable in the face of network interface hardware changes, the user may now optionally configure a specific 1 to 16 character hexadecimal node-id for each node. For example, "01" and "B2" and "beef" are valid node ids, but "fred" is not.

This feature is also desirable with cloud providers such as Amazon AWS. Specifying a node ID which matches a particular EBS volume allows a node to be taken out, a new instance created, then Aerospike brought up with the new instance with the previous node ID. This minimizes data migration.

If you mistakenly reuse a node-id, the cluster will detect the duplicate IDs, and refuse to allow these nodes to join the cluster. The log file(s) within the cluster will state that the cluster can't be joined due to duplicate cluster IDs.

In order to operationalize specified node IDs, it is best to start with specified node-ids during initial cluster formation and installation.

Follow the subsequent technique for starting with a set of initial node-ids.

If you wish to convert a cluster from automatic node-ids to specified node-ids, you may change the configuration file for a server and restart. For an SC namespace, you will have to change the roster. To prevent data unavailability, if you are changing multiple servers, update a single server, restart it, modify and commit the new roster, then repeat for the next server. You do not have to wait until data migration finishes, but you must validate and apply the new roster.

To configure the node-id, edit your configuration file and add node-id <HEX NODE ID> to the service context.

caution

The configuration file options node-id and node-id-interface are mutually exclusive.

service {
    user root
    group root
    service-threads 20 # Should be 5 times number of vCPUs for 4.7+ and at least one
                       # SSD namespace, otherwise number of vCPUs
    nsup-period 16
    proto-fd-max 15000
    
    node-id a1
}

When the server starts, validate that the node-id is as expected, then change the roster of any SC namespace intending to include this server.

Strong Consistency and parameters for expiration

Strong consistency should be used carefully with expiration. For background on expiration and eviction, see Definition of expiration and eviction.

For each namespace where you desire strong consistency, add strong-consistency true and default-ttl 0 to the namespace stanza.

namespace test {
    replication-factor 2
    memory-size 1G
    default-ttl 0
    strong-consistency true
    storage-engine device {
       file /var/lib/aerospike/test.dat
       filesize 4G
       data-in-memory true
    }
}

Non-durable deletes and configuration settings

A key consideration for SC is the length of time records exist, which is a function of several configuration parameters.

If a record is non-durably deleted in one master partition but has not yet been consistently marked as deleted in another partition, if the master has to heal by restoring a prior replica, the "deleted" record is restored in the live partition.

You need to be sure that when expunge, expiration, or eviction of non-durable-delete events occur, there are no transactions updating the record. With strong-consistency-allow-expunge true, for highly active or essential records, you have the following options:

Set the default-ttl parameter far enough into the future to be reasonably confident that updates do not occur during expiration. For example, default-ttl 365D sets the TTL to 365 days, or one year.
Disable eviction with the parameters:
- high-water-memory-pct 0.
- high-water-disk-pct 0.
- mounts-high-water-pct 0.

Commit-to-device

Determine whether you need commit-to-device. If you are running in a situation where no data loss is acceptable even in the case of simultaneous server hardware failures, you can choose that a namespace commits to disk. This will cause performance degradation, and only provides benefit where hardware failures are within milliseconds of each other. Add commit-to-device true within the storage engine scope of the namespaces in question.

commit-to-device requires serializing writes to the output buffer, and will thus flush more data than is strictly necessary. Although Aerospike automatically determines the smallest flush increment for a given drive, this can be raised with the optional commit-min-size parameter.

Start the servers.

systemctl start aerospike

The cluster forms with all nodes but without a per-roster namespace. In order to use your Strong Consistency namespace, you'll need to operationally add the roster. Until you do so, client requests will fail.

Admin> show stat -flip like cluster_size
~~~~~~~~~~~~~~~~~~~~~~~~Service Statistics~~~~~~~~~~~~~~~~~~~~~
                          NODE            cluster_size
node2.aerospike.com:3000                             5
node4.aerospike.com:3000                             5
node5.aerospike.com:3000                             5
node6.aerospike.com:3000                             5
node7.aerospike.com:3000                             5
Number of rows: 5

~~~~~~~~~~~~~~~~~~~~~~~~test Namespace Statistics~~~~~~~~~~~~~~~~~~~~~
                          NODE            ns_cluster_size
node2.aerospike.com:3000                             0
node4.aerospike.com:3000                             0
node5.aerospike.com:3000                             0
node6.aerospike.com:3000                             0
node7.aerospike.com:3000                             0
Number of rows: 5

This result is expected, since the test namespace's roster has not yet been modified. To make this namespace useful we will need to create a roster, please see below.

Create the Roster

The roster is the list of nodes which are expected to be in the cluster, for this particular namespace. This list is stored persistently in a distributed table within each Aerospike server, similar to how index configuration is stored. In order to change and manipulate this roster, please use the following Aerospike real-time tools. The tools referenced here are part of the aerospike-tools package, which should be installed.

Rosters are specified using node IDs. With the advent of Aerospike Server 4, you can specify the node ID of each server, as well as using the automatically generated node ID that was supported in Aerospike Server 3 and previous. For further information about specifying node IDs, please see that section.

The general process of managing a roster is to first bring together the cluster of nodes, then list the node IDs within the cluster that have the namespace defined, and finally add the node's IDs to the roster. To commit the changes, execute a recluster command.

If using Tools package 6.2.x or later, use the following commands:

Admin> enable;
Admin+> manage roster stage observed ns test
Pending roster now contains observed nodes.
Run "manage recluster" for your changes to take affect.
Admin+> manage recluster
Admin+> pager on
Admin+> show roster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2021-10-22 20:14:01 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                    Node|         Node ID|Namespace|                                 Current Roster|                                 Pending Roster|                                 Observed Nodes
node2.aerospike.com:3000|BB9070016AE4202 |test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202
node4.aerospike.com:3000|BB9060016AE4202 |test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202
node6.aerospike.com:3000|*BB9050016AE4202|test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202
Number of rows: 3

Otherwise, use the equivalent asinfo commands described below in detail:

Get a list of nodes which have been observed to have this namespace definition,

asinfo -v 'roster:namespace=[ns-name]'

The observed_nodes are the nodes which are in the cluster and have the namespace defined. Copy the observed_nodes list.

Admin+> asinfo -v "roster:namespace=test"
node6.aerospike.com:3000 (192.168.10.6) returned:
roster=null:pending_roster=null:observed_nodes=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202

node4.aerospike.com:3000 (192.168.10.4) returned:
roster=null:pending_roster=null:observed_nodes=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202

node2.aerospike.com:3000 (192.168.10.2) returned:
roster=null:pending_roster=null:observed_nodes=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202

Now set the roster to the observed_nodes

roster-set:namespace=[ns-name];nodes=[observed nodes list]

Note it is safe to issue this command multiple times, once for each observed node, or create a list. The roster is not active until we recluster.

Admin+> asinfo -v "roster-set:namespace=test;nodes=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202" with BB9020016AE4202
node2.aerospike.com:3000 (192.168.10.2) returned:
ok

We now have a roster but it isn't active yet.

Validate your roster by using the roster: command.

Admin+> asinfo -v "roster:"
node2.aerospike.com:3000 (192.168.10.2) returned:
ns=test:roster=null:pending_roster=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202:observed_nodes=null

node6.aerospike.com:3000 (192.168.10.6) returned:
ns=test:roster=null:pending_roster=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202:observed_nodes=null

node4.aerospike.com:3000 (192.168.10.4) returned:
ns=test:roster=null:pending_roster=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202:observed_nodes=null

Notice that roster is null but pending_roster is set with the provided roster.

Apply the pending_roster by issuing the recluster: command.

Admin+> asinfo -v "recluster:"
node2.aerospike.com:3000 (192.168.10.2) returned:
ignored-by-non-principal
                
node6.aerospike.com:3000 (192.168.10.6) returned:
ignored-by-non-principal

node4.aerospike.com:3000 (192.168.10.4) returned:
ok

Verify that the new roster was applied with the roster: command.

Admin+> asinfo -v "roster:"
node2.aerospike.com:3000 (192.168.10.2) returned:
ns=test:roster=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202:pending_roster=BB9070016AE4202,
                BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202
                
node6.aerospike.com:3000 (192.168.10.6) returned:
ns=test:roster=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202:pending_roster=BB9070016AE4202,
                BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202

node4.aerospike.com:3000 (192.168.10.4) returned:
ns=test:roster=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202:pending_roster=BB9070016AE4202,
                BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202

Notice that both roster and pending_roster are set to the provided roster.

Validate that the namespace cluster size agrees with the service cluster size.

The namespace ns_cluster_size should now agree with the service cluster_size (assuming all nodes in the service have this namespace).

Validate this with
show stat -flip like cluster_size

Admin> show stat -flip like cluster_size
~~~~~~~~~~~~~~~~~~~~~~~~Service Statistics~~~~~~~~~~~~~~~~~~~~~
                          NODE            cluster_size
node2.aerospike.com:3000                             5
node4.aerospike.com:3000                             5
node5.aerospike.com:3000                             5
node6.aerospike.com:3000                             5
node7.aerospike.com:3000                             5
Number of rows: 5

~~~~~~~~~~~~~~~~~~~~~~~~test Namespace Statistics~~~~~~~~~~~~~~~~~~~~~
                          NODE            ns_cluster_size
node2.aerospike.com:3000                             5
node4.aerospike.com:3000                             5
node5.aerospike.com:3000                             5
node6.aerospike.com:3000                             5
node7.aerospike.com:3000                             5
Number of rows: 5

Configuring Rack Aware

Strong Consistency namespaces cause some changes to rack aware configurations.

For a strong-consistency namespace to be rack-aware, the roster list becomes a series of node-id@rack-id pairs. Each entry in the roster list needs to define which rack a node is on (defaulting to rack-id 0 if none is defined). The roster can be manually constructed by appending @rack-id to each node-id in a comma separated list.

Note: the rack-id is specified in each namespace. It is possible to set different rack-ids for different namespaces, and it is possible to have some namespaces not be rack aware.

However, management of this list may be simplified by just configuring rack-id on each node.

These configured rack-ids will automatically appear in the observed_nodes list, which can then be used directly as a roster list in the set-roster command. The following will describe configuring Strong Consistency with rack aware by using the rack-id configuration to generate the observed_nodes list.

To configure rack aware for a namespace configured with strong-consistency, modify the rack-id configuration for each namespace in the /etc/aerospike/aerospike.conf file.

namespace test {
   replication-factor 2
   memory-size 1G
   default-ttl 0
   strong-consistency true
   rack-id 101
   
   storage-engine device {
     file /var/lib/aerospike/test.dat
     filesize 4G
     data-in-memory false
     commit-to-device true
   }
}

note

In strong consistency mode, the rack-id configuration is actually ignored and is only used to facilitate setting the roster by copying and pasting the observed_nodes list. Only the rack-id configured when setting the roster will be applied, and it could be different than what has been configured in the config file (or directly dynamically through the rack-id configuration parameter).

note

Tools package 6.2.x or later is required to use asadm's manage roster commands. Otherwise, use the equivalent asinfo - roster-set command described above.

This can also be done dynamically (refer to the details at the bottom of this page).

Start the Aerospike server

systemctl start aerospike

Copy the observed nodes into the pending roster with asadm's manage roster commands

Admin+> manage roster stage observed ns test

View the roster with asadm's show roster and notice that the Pending Roster has been updated.

Apply the roster with asadm's manage recluster command.

Admin+> manage recluster
Successfully started recluster

Verify the configured rack-ids are active by issuing the asadm's show racks command and verifying that the displayed rack-id_ matches what is configured.

Admin+> show racks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Racks (2021-10-21 20:33:28 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|Rack|                                                                          Nodes
         |  ID|                                                 
test     |101 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202                
Number of rows: 1

The cluster is now configured as rack aware.

Dynamically configuring rack ids

note

Tools package 6.0.x or later is required to use asadm's manage config commands. Otherwise, use the equivalent asinfo - set-config command described above.

Dynamically set rack-id with asadm's manage config command:

Admin+> manage config namespace test param rack-id to 101 with 192.168.10.2 192.168.10.4 192.168.10.5
~Set Namespace Param rack-id to 101~
                    Node|Response
node2.aerospike.com:3000|ok      
node4.aerospike.com:3000|ok      
node5.aerospike.com:3000|ok      
Number of rows: 3

Admin+> manage config namespace test param rack-id to 102 with 192.168.10.6 192.168.10.7
~Set Namespace Param rack-id to 102~
                    Node|Response
node6.aerospike.com:3000|ok      
node7.aerospike.com:3000|ok      
Number of rows: 2

Issue a recluster with asadm's manage recluster command:

Note: Tools package 6.0.x or later is required to use asadm's manage recluster command. Otherwise, use the equivalent asinfo - recluster command.

Admin+> manage recluster
Successfully started recluster

Managing Strong Consistency

In order to add nodes, remove nodes, and manage availability, please see Consistency Management documentation.

Feature key for strong consistency​

Install and Configure NTP ( or other clock synchronization )​

Configure Node IDs​

Strong Consistency and parameters for expiration​

Non-durable deletes and configuration settings​

Commit-to-device​

Create the Roster​

Configuring Rack Aware​

Dynamically configuring rack ids​

Managing Strong Consistency​

Feature key for strong consistency

Install and Configure NTP ( or other clock synchronization )

Configure Node IDs

Strong Consistency and parameters for expiration

Non-durable deletes and configuration settings

Commit-to-device

Create the Roster

Configuring Rack Aware

Dynamically configuring rack ids

Managing Strong Consistency