Configuring Namespace Storage

Aerospike stores data according to the storage engine you select for a namespace. Your choices are persistent storage on disk, non-persistent storage in memory, or a combination of both. Your choice of storage engine affects the durability, cost, and performance of your cluster.

Comparing storage engines

Storage engine	Device (data not in memory)	All Flash	Device (data in memory)	Memory only (no persistence)	Persistent Memory^***
Index In Memory	✓		✓	✓	✓
Fast Restarts^*	✓	✓	✓^**		✓
Survive a Power Outage	✓	✓	✓		✓

_{^*Available on Aerospike Enterprise Edition.}
_{^**Primary index persisted upon restart for data-in-memory with persistence in Enterprise Edition version 3.15.1.3 and later.}
_{^**Fast Restarts are also supported for a data-in-index configuration.}
_{^***Available in Aerospike Enterprise Edition 4.8 and later, with a feature enabled in the feature-key-file.}

Storage engine configuration

Storage engines are configured in the Aerospike Server configuration file, /etc/aerospike/aerospike.conf. Each section describes the minimal configuration to enable a particular storage engine, and the storage sizing parameters used by that engine.

Setup for an SSD storage engine

The minimal configuration for a solid-state drive (SSD) namespace requires:

Setting the storage-engine parameter to device.
Adding a device parameter for each SSD to be used by the namespace.
Optionally changing the memory-size from the default of 4 GiB to a size appropriate for the expected primary index size. Refer to the Sizing Guide to learn about sizing memory.

Each storage device must be properly initialized as an Aerospike device, including zeroizing the 8MiB header. Refer to Initializing Solid State Drives (SSDs). The maximum size of a device is 2 TiB. Larger devices must be partitioned into multiple equally-sized partitions that are less than 2 TiB each.

For performance, we recommend reducing the write-block-size from the default of 1 MiB to 128 KiB on SSD-backed namespaces. This may vary based on the specific workload and record average size. Run benchmarks with asbench to find the right setting.

namespace NAMESPACE-NAME {
    memory-size SIZE            # Maximum memory allocation for primary
                                # and secondary indexes.
    storage-engine device {     # Configure the storage-engine to use persistence
        device /dev/DEVICE      # raw device. Maximum size is 2 TiB
        # device /dev/DEVICE  # (optional) another raw device.
        write-block-size 128K   # adjust block size to make it efficient for SSDs.
    }
}

note

A device partition must be associated with only a single namespace. How to add, replace and remove disks describes how to add and remove device partitions from namespaces.

Setup for a persistent memory storage engine

The minimal configuration for the persistent memory storage namespace requires setting two parameters in aerospike.conf for each pmem storage file to be used by this namespace:

storage-engine
file Use the file parameter with the namespace context, and not the logging context.

The filesize must be large enough to support the size of the data, up to the maximum allowed value of 2 TiB.

You may need to change the memory-size from the default of 4 GiB to a size appropriate for the expected primary index size. Refer to the Sizing Guide to learn about sizing memory.

In server version 5.1 and later, persistent memory namespaces are treated equivalently to data-in-memory namespaces for the purpose of computing the default number of service-threads. The value of service-threads will default to the number of CPUs, unless there is at least one SSD namespace.

On systems with hyperthreading, only physical cores are counted. In multi-socketed systems, if non-uniform memory access (NUMA) pinning is enabled, each Aerospike instance counts only the CPU cores on the socket it is servicing.

namespace NAMESPACE-NAME {
    memory-size SIZE                # Maximum memory allocation for secondary indexes (if any).
    storage-engine pmem {           # Configure the storage-engine to use
                                    # persistence. Maximum size is 2 TiB.
        file /mnt/pmem/FILENAME   # Location of pmem data file on server, where /mnt/pmem is the
                                    # mount point of an EXT4 or XFS file system that resides in pmem
                                    # and has been mounted with the DAX option.
        # file /mnt/pmem/ANOTHER  # (optional) Location of pmem data file on server.
        filesize SIZE               # Max size of each file in GiB.
    }
}

Setup for an HDD storage engine with data-in-memory

The minimal configuration for a data-in-memory namespace includes:

Setting storage-engine to device.
Setting data-in-memory to true.
Entering a list of file parameters to configure where data will be persisted. Use the file parameter with the namespace context, and not the logging context.

The filesize must be large enough to support the size of the data on disk, with a maximum allowed value of 2 TiB. For common use cases, this should roughly be four times the memory-size.

You may need to change the memory-size from the default of 4 GiB to a size appropriate to handle the expected primary index size, and the expected size of the data in memory. Refer to the Sizing Guide to learn about sizing memory.

namespace NAMESPACE-NAME {
    memory-size SIZE                # Maximum memory allocation for secondary indexes (if any).
    storage-engine device {         # Configure the storage-engine to use
                                    # persistence. Maximum size is 2 TiB
        file /opt/aerospike/FILENAME  # Location of data file on server.
        # file /opt/aerospike/ANOTHER # (optional) Location of data file on server.
        filesize SIZE               # Max size of each file in GiB.
        data-in-memory true             # Indicates that all data should also be
                                        # in memory.
    }
}

Setup for a HDD storage engine with data-in-index engine

danger

As of server 6.4, this configuration is not supported; the single-bin and data-in-index configuration settings are removed.

A data-in-index configuration is a highly-specialized namespace for a niche use case. Use the data-in-index engine if your data is single-bin and fits in 8 bytes, and you need the performance of an in-memory namespace but do not want to lose the fast restart capability provided in Aerospike Enterprise Edition.

The minimal configuration for a data-in-index namespace includes:

Setting single-bin to true.
Setting data-in-index to true.
Setting data-in-memory to true.
The storage-engine must be device
The file or device parameters must be configured to map to the persisted storage device to be used by this namespace. Use the file parameter with the namespace context, and not the logging context.

You may need to change the memory-size from its default of 4 GiB to a size that can accommodate the primary index, and filesize from its 16 GiB default to the size of the data on disk, with a maximum allowed value of 2 TiB. Refer to the Sizing Guide to learn about sizing memory.

namespace NAMESPACE-NAME {
    memory-size SIZE                # Maximum memory allocation for data and
                                    # primary and secondary indexes.
    single-bin true                 # Required true by data-in-index.
    data-in-index true              # Enables in index integer store.
    storage-engine device {         # Configure the storage-engine to use
                                    # persistence.
    file /opt/aerospike/FILENAME  # Location of data file on server.
    # file /opt/aerospike/ANOTHER # (optimal) Location of data file on server.
    # device /dev/DEVICE          # Optional alternative to using files.

    filesize SIZE                  # Max size of each file in GiB. Maximum size is 2TiB
    data-in-memory true            # Required true by data-in-index.
    }
}

Updating the `filesize` parameter

If you need to change the filesize parameter of a namespace with storage-engine set to device, use the following procedure.

Increase `filesize`

Perform the following steps on your cluster one node at a time.

Change the filesize parameter in the configuration file. Ensure that the relevant partition has sufficient free disk space. To add a new file to the configuration, place it as the last entry in the Aerospike storage configuration.
note
You do not have to delete and recreate the file when increasing the configured size of the file.
Restart Aerospike:
```
/etc/init.d/aerospike restart
```
Wait for port 3000 to open and for the node to rejoin the cluster.
The following shell command is useful for discovering whether a node has started successfully:
```
cat /var/log/aerospike/aerospike.log | grep -i 'cake'
```
Proceed with the other nodes in the cluster one by one, repeating the above steps. To avoid data inconsistency, wait for migrations to complete between each restart.

Reduce `filesize`

Reducing the size of an existing data file may result in the loss of data. Proceed with caution.

To avoid data inconsistency, delete the data file and update the configuration file on one node at a time, allowing the data to migrate to other nodes before proceeding to the next node.

Stop Aerospike:

/etc/init.d/aerospike stop

Delete the file and update the configuration file with the new filesize.
Start Aerospike:

/etc/init.d/aerospike start

Setup for data-in-memory without persistence

The minimal configuration for a namespace without persistence is to set storage-engine to memory. If your namespace requires more than the default 4 GiB memory-size allocation for the primary index and data in memory, then you must also adjust memory-size. Refer to the Sizing Guide to learn about sizing memory.

namespace NAMESPACE-NAME {
    memory-size SIZE      # Maximum memory allocation for data and primary and
                          # secondary indexes.
    storage-engine memory # Configure the storage-engine to not use persistence.
}

Setup for shadow device

The shadow device storage model is designed for cloud environments with extremely high-performance SSDs that are ephemeral (not persistent), and where the persisted devices are not providing the necessary performance.

Shadow devices act as persisted stores, and must be greater than or equal to the size of the primary device. The primary device receives all read/write operations as usual, and all writes are duplicated to a shadow device. This results in a persisted data volume with lower input/output operations per second (IOPS) requirements, while gaining the IOPS benefit of the non-persisted volume without using large amounts of RAM. The shadow device only needs to satisfy the write IOPS requirements of your workload, not reads.

note

The shadow device storage model is an extension of the SSD storage engine.

When you are using network-attached shadow devices, for example on Amazon Elastic Block Store, or re-assigning shadow devices to a different instance, we recommend initially configuring the node-id across all the nodes in the cluster. This preserves the node-id on any potential new instances that will be re-attached to the shadow device of an instance, and avoids re-distribution of the partitions in the cluster.

To use shadow devices, add the persisted volume after the declaration of the non-persisted volume on the same line in aerospike.conf.

namespace NAMESPACE-NAME {
    ...
    storage-engine device {
        device /dev/sdb /dev/sdf  # sdb is the fast ephemeral volume,
                                  # and sdf is the slower persisted volume
        ...
    }
}

In the example, /dev/sdb is the fast non-persisted device. /dev/sdf is the persisted device. The device order is important, with the fast non-persisted device named first, and the shadow device named second. The two devices must be listed on the same line.

You may configure multiple shadow devices, with each device pair on its own line. Each shadow device must be paired with only one primary device:

    storage-engine device{
        device /dev/sdb /dev/sdf
        device /dev/sdc /dev/sdg
        device /dev/sdd /dev/sdh
        ...
    }

danger

When configuring a namespace to use persistence of any form, take care that a given file or device partition is associated with a single namespace only. Two namespaces cannot share the same file or partition. Configuration of the same file or partition for multiple namespaces could cause issues with the node starting, and possibly damage existing data in that file or partition.

Instance recovery

If the ephemeral device is damaged - missing header information for example - and there is a valid shadow device, the server will load data from the EBS shadow device into the ephemeral disk and into memory (primary index, secondary index, data-in-memory). Refer to the instance failure section of the AWS deployment guide.
If the ephemeral device fails, when the instance restarts it populates the data from the shadow device and the server will read transactions as usual once the node has rejoined the cluster.

Where to next?

Configure Data Retention Policy, which determines how long records are kept after they are initially written.
Configure Data Durability Policy which determines how many replica copies of a record to keep in the cluster.
Or return to Configure Page.

Comparing storage engines​

Storage engine configuration​

Setup for an SSD storage engine​

Setup for a persistent memory storage engine​

Setup for an HDD storage engine with data-in-memory​

Setup for a HDD storage engine with data-in-index engine​

Updating the filesize parameter​

Increase filesize​

Reduce filesize​

Setup for data-in-memory without persistence​

Setup for shadow device​

Instance recovery​

Where to next?​