Skip to main content

Configuring Storage Compression

Storage Compression

Aerospike's storage compression feature provides lossless compression of records written to persistent storage. Aerospike supports three different compression algorithms, which provide superior compression and decompression speed:

Compression is only applied to records that are written to persistent storage, e.g., SSDs or persistent memory (pmem). Record data in memory (RAM) is not compressed.

The trade-off of compression is increased transaction latency and higher CPU load, because in addition to accessing the SSD or pmem file, compression and decompression have to take place. Typically, write latency is affected more than read latency, because compression is more CPU-intensive than decompression.

An algorithm's performance - speed as well as compression ratio - is highly specific to the data to be compressed. When picking an algorithm, it is good practice to evaluate each algorithm's performance with records that are representative of real-world data.

An empirical approach towards evaluating the optimal compression algorithm is to configure different compression algorithms on multiple servers and then reviewing the compression percentages between the servers.

Note that compression does not allow for individual records larger than the configured write block size; it's the uncompressed record size that counts for the record size check. This ensures that a record will always fit into a write block, even if it is later stored without compression. However, as long as each individual record's uncompressed size is no larger than the write block size, multiple compressed records can be stored in one physical write block, up to its size.

Although different (or no) compression settings may be made for the same namespace on different nodes, this should only be considered as a temporary approach during a transition towards a cluster where that namespace has the same compression settings on each node.

Later Aerospike clients support compression on the client side in addition to server side compression. When this is configured the server will decompress bin data (metadata is never compressed) as it comes off the disk before recompressing to send to the client.

In the same way, incoming data from the client is decompressed by the server before being rewritten. For inbound data this allows operations to be performed on the data before it is written. For both inbound and outbound data it allows for the fact that the client may be running a different compression level or type than the server.

Configuration

Storage compression is configured per namespace per node, using two configuration directives, compression and compression-level.

  • compression selects the compression algorithm. Valid parameters are none, lz4, snappy, and zstd.

    If this setting is not specified, it defaults to none, i.e., no compression.

  • compression-level controls the trade-off between compression speed and compression ratio.

    The valid parameter values 1 through 9 are used as a scale. On one end of the scale, 1 selects a faster and less efficient compression. On the other end, 9 selects a slower and more efficient compression.

    For compression algorithm zstd, in Aerospike Server versions prior to 4.6.x, if this setting has never been specified when using compression zstd, a default flag of 0 is displayed and the compression-level of 9 will be used.

    For compression algorithm zstd, in Aerospike Server versions 4.6.x or newer, if this setting has never been specified when using compression zstd, a default flag of 9 is displayed and the compression-level of 9 will be used.

    This setting is only honored by Zstandard. It does not have any effect for LZ4 and Snappy.

The configuration directives belong to a namespace's storage-engine section, for example:

namespace test {
...
storage-engine device {
device /dev/sda1
...
compression zstd
compression-level 1
}
...
}

or

namespace test {
...
storage-engine pmem {
file /dev/pmem1/test.dat
...
compression zstd
compression-level 1
}
...
}

Both settings can also be configured dynamically with asadm, for example:

Note: Tools package 6.0.x or later is required to use asadm's manage config commands. Otherwise, use the equivalent asinfo - set-config command.

asadm -e 'enable; manage config namespace test param compression to zstd'
asadm -e 'enable; manage config namespace test param compression-level to 1'

Or, with asinfo:

asinfo -v 'set-config:context=namespace;id=test;compression=zstd'
asinfo -v 'set-config:context=namespace;id=test;compression-level=1'

The currently configured compression settings are consulted whenever a newly inserted or an updated record is compressed on its way to storage. The compression settings then control how this record is compressed.

This means that changing the compression settings doesn't affect existing records on storage. It only affects records that are newly inserted or updated after the compression settings are changed. If you switch compression from lz4 to snappy, then existing records will remain compressed with LZ4, until they get updated. Then they will be compressed according to the current settings, i.e., with Snappy.

This also means that storage may contain a mix of uncompressed records as well as records compressed with any of the three compression algorithms. This is fine, because decompression works independently from the compression setting; it always applies the correct algorithm when decompressing a record, i.e., the algorithm that was used to compress it.

Note that a record may not be compressible, so that applying compression would increase its size. In this case, the record is stored uncompressed instead of storing the larger compressed version.

From Aerospike 4.5.3 onwards when records are migrated between nodes the data remains in the same format as it is on disk. This is extremely efficient and means that if a record is compressed, it remains compressed during migration. It also means that, unlike in previous releases, migration cannot be used as a means of changing compression level.

To change the compression level of existing records it is necesssary for the application to trigger a re-write of each record in the cluster. These record re-writes can also be triggered by touch operations. This would entail the following steps:

  • Dynamically change the compression settings on all cluster nodes.

  • Read each record in the cluster and write it back.

Another approach would be to touch existing records. This can be done using a background scan running a touch operation or a scan and background UDF. This would serve to re-write the records using the new compression setting. This can be done without changing the TTL of the record dependant on the method chosen. Use Filter Expressions to ensure only records prior to the compression level change are touched. The How to Create a Scan and Touch UDF knowledge base article goes into greater detail on how multiple records can be touched simultaneously.

Statistics

Aerospike reports the current compression ratio as part of the namespace statistics. For example:

$ asadm -e 'show statistics for namespace test like device_compression_ratio'
~~~~~~~~test Namespace Statistics (2021-10-22 22:18:16 UTC)~~~~~~~
Node |10.0.0.1:3000|10.0.0.1:3000|10.0.0.1:3000
device_compression_ratio|1.000 |1.000 |1.000

The given number is the average compressed size : uncompressed size ratio. Thus, the above value, 1.000, means no compression at all. In contrast, 0.100 would indicate a ratio of 1 : 10, i.e., a size reduction by 90%.

Note that device_compression_ratio will be included in the namespace statistics unless compression is set to none.

The compression ratio is a moving average. It is calculated based on the most recently written (= most recently compressed) records. Read records do not factor into the ratio.

If the written data - and how compressible it is - changes over time, then the compression ratio will change with it. In case of a sudden change in data, the indicated compression ratio may lag behind a bit. As a rule of thumb, assume that the compression ratio covers the most recently written 100,000 to 1,000,000 records.

In particular, this means that the compression ratio might not accurately reflect the compression ratio across all records on storage. The actual storage savings across all records might be higher or lower than the current compression ratio, which just covers the most recently written records.

When evaluating different compression settings for real-world data, proceed as follows:

  • Set compression and compression-level.

  • Write 1,000,000 representative records to make the ratio converge to reflect the actual compression ratio of these records.

  • Query the compression ratio via the above info command.

Repeat these steps for all compression settings to be evaluated.