Managing Sets in a Namespace
An Aerospike set is similar to a table in a relational database, except you do not need to define a schema for a set. Sets are created dynamically and implicitly on the first record insertion in set. For more see Data Model.
Protecting a set from evictions
A set can be protected from evictions with a dynamic configuration. If evictions are in effect for a namespace, sets with evictions disabled will be skipped, regardless of the void-time of the records.
- Starting with Aerospike 5.6.0, use the configuration parameter
disable-eviction
- In Aerospike 3.6.1 to Aerospike 5.5.0, use the configuration parameter
set-disable-eviction
The asadm
command to dynamically disable set evictions is:
asadm -e "enable; manage config namespace test set demo param disable-eviction to true"
Or using the older info command:
asadm -e "enable; asinfo -v 'set-config:context=namespace;id=test;set=demo;disable-eviction=true' "
Configuration parameters are applied per-node. asadm
sends the same info command disabling evictions to all the nodes in the cluster.
Capping the size of a set
Starting with Aerospike 6.3, you can use the stop-writes-size
configuration parameter to specify a size limit for a set.
If this threshold is breached, client writes to the set are refused. Deletions, replica writes, and migration writes are still allowed when the set is in stop-writes mode.
Once the limit is reached, the server does not allow any additional writes, even those that would decrease the size of a record. There are two ways to get under the stop-writes-size limit: increase or remove the limit, or delete records. However, record deletions via UDF, delete-all-bins ops, or background ops queries will also be rejected – only regular record deletes (including batch deletes) and nsup deletes will be allowed.
The asadm
command to dynamically cap the size of a set is:
asadm -e "enable; manage config namespace test set demo param stop-writes-size to 250M"
Or using the older info command:
asadm -e "enable; asinfo -v 'set-config:context=namespace;id=test;set=demo;stop-writes-size=250M' "
Configuration parameters are applied per-node. asadm
sends the same info command limiting the set size to all the nodes in the cluster.
The cluster-wide size quota for the set is number-of-cluster-nodes x stop-writes-size
.
The threshold is checked and stop-writes triggered independently on each node.
Capping the number of records in a set
A set can be capped to a maximum number of records. If this threshold is breached, client writes to the set are refused. Deletions, replica writes, and migration writes are still allowed when the set is in stop-writes mode.
- Starting with Aerospike 5.6.0, use the configuration parameter
stop-writes-count
- In Aerospike 3.6.1 to Aerospike 5.5.0, use the configuration parameter
set-stop-writes-count
The asadm
command to dynamically cap the number of records in a set is:
asadm -e "enable; manage config namespace test set demo param stop-writes-count to 1000000"
Or using the older info command:
asadm -e "enable; asinfo -v 'set-config:context=namespace;id=test;set=demo;stop-writes-count=1000000' "
Configuration parameters are applied per-node. asadm
sends the same info command limiting the number of records in the set to all the nodes in the cluster.
The cluster-wide record-count quota for the set is number-of-cluster-nodes x stop-writes-count
.
The threshold is checked and stop-writes triggered independently on each node.
Adding and Removing a Set Index
Introduced in Aerospike Database 5.6, a set index
can be dynamically configured in the set context through declaring
enable-index
to
be true
. This may significantly improve performance when scanning the records in
the set.
asadm -e "enable; manage config namespace test set demo param enable-index to true"
Or using the older info command:
asadm -e "enable; asinfo -v 'set-config:context=namespace;id=test;set=demo;enable-index=true' "
Removing the set index is simply a matter of changing the value to
false
, which is also the default for each set.
asadm -e "enable; manage config namespace test set demo param enable-index to false"
Or using the older info command:
asadm -e "enable; asinfo -v 'set-config:context=namespace;id=test;set=demo;enable-index=false' "
Monitoring Set Indexes
The memory used for all set indexes in a namespace is available via the
memory_used_set_index_bytes
namespace statistic and also via the
"memory-usage"
server log ticker line.
Clearing a Set
There is a limit of 1023 on the total number of sets in a namespace. Deleting a set metadata (to reclaim space from this 1023 limit) requires durably deleting all the data in the set and cold restarting the node. See the following How to clear up set names when they exceed the limit for details.
Truncating a Set in a Namespace
Server versions 4.3.1.11, 4.4.0.11, 4.5.0.6 and 4.5.1.5 added 2 different commands for truncation:
truncate
for truncating sets.truncate-namespace
for truncating namespaces.
It is recommended to use theasadm
manage truncate
command due to the clear separation between truncating a set and truncating a namespace (added in Tools package 6.0).
To initiate a set truncation, the asadm
, asinfo
, or aql
tools can be used to issue a truncate command to one Aerospike cluster node at a time.
To truncate a set, use either of the following commands.
Using the asadm
command-line interface:
asadm
Seed: [('127.0.0.1', 3000, None)]
Aerospike Interactive Shell, version 2.11.0
Found 1 nodes
Online: 127.0.0.1:3000
Admin> info set
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Set Information (2023-03-26 22:59:40 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace| Set| Node| Memory| Disk|~~~~~Quota~~~~~~| Objects| Stop| Disable| Set
| | | Used| Used| Total|Used%| | Writes|Eviction|Index
| | | | | | | | Count| |
test |demo|127.0.0.1:3000 |1.236 MB|0.000 B |250.000 MB|0.0 %|99.711 K|1000000|True |No
test |demo| |1.236 MB|0.000 B |250.000 MB|0.0 %|99.711 K| | |
Number of rows: 1
Admin> enable
Admin+> manage truncate ns test set demo
You're about to truncate up to 0 records from set demo for namespace test
Confirm that you want to proceed by typing 6cdc8c, or cancel by typing anything else.
6cdc8c
Successfully started truncation for set demo of namespace test
Admin+> info set
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Set Information (2023-03-26 23:04:08 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace| Set| Node| Memory| Disk|~~~~~Quota~~~~~~|Objects| Stop| Disable| Set
| | | Used| Used| Total|Used%| | Writes|Eviction|Index
| | | | | | | | Count| |
test |demo|127.0.0.1:3000 |0.000 B |0.000 B |250.000 MB|0.0 %|0.000 |1000000|True |No
test |demo| |0.000 B |0.000 B |250.000 MB|0.0 %|0.000 | | |
Number of rows: 1
Or in non-interactive mode:
asadm --enable -e "manage truncate ns test set demo --no-warn"
Or with asinfo
send truncate a single node. The truncate operation is then
communicated to the other nodes as shared metadata (SMD):
asinfo -h 127.0.0.1 -v "truncate:namespace=test;set=demo"
Or with aql
:
aql -c "TRUNCATE <namespace_name>.<set_name>"
Mechanism
The mechanism involved in truncating a namespace or set within a namespace is distinct from NSUP.
Before server 6.3, the truncate
info command started a single background thread on each of the nodes in the cluster.
From server 6.3, the truncate
info command starts a number of threads for each truncation job equal to the value of the truncate-threads
configuraiton parameter. The truncate threads then traverse the primary index of the namespace and remove the necessary records.
From server 6.3, set truncation is optimized to use set indexes to rapidly find all the records belonging to the set being truncated. This optimization significantly increases the speed of set truncation, with the caveat that it is skipped if the set has tombstones.
The truncate command takes an optional lut
(last-update-time) threshold parameter, which limits the truncation to records with an LUT earlier than the specified timestamp . If no timestamp is specified, the now
timestamp of the receiving node is used.
Set truncation can be issued repeatedly. It updates the effective last-update-time of affected set, and is reflected in the lut
field of the set information.
For truncation on sets with secondary indexes, each record data is actively read, and any secondary index entries associated with the record are also removed.
Read the truncate
info command for advise on optimally truncating in the presence of secondary indexes.
In the Enterprise Edition, truncation is durable and preserves record deletions through a cold-restart. In the Community Edition, similar to record deletes, records in previously truncated sets are not durable and deletes can return through a cold-start.
The truncate-undo
and
truncate-namespace-undo
commands can be used to remove the truncate related entry(ies) in SMD to allow
for potential recovery of inadvertently truncated records upon a subsequent Cold Restart. Truncated records that would have been already overwritten on persistent storage would not be recoverable.