Skip to main content

Aerospike Validation Tool (asvalidation)

The asvalidation tool checks the validity of collection data type (CDT) bins in a namespace. It produces a report of problem bins it discovers. It also has an option to fix some of the discovered problems.

Recommendations and Server Versions for When to Run asvalidation

The Aerospike CDT Validation Tool addresses the following types of CDT issues, which require certain Aerospike Server versions to detect or correct.

  • Order: The CDT bin has elements out of order. Performing operations on Ordered Lists under Aerospike 3.16.0.1 to 3.16.0.5 may have corrupted records. Fixing this problem requires Aerospike Server version 4.5.3 or newer. If also using XDR, before upgrading to Aerospike 4.6.0 or newer, you must run asvalidation.
  • Padding: The bin has garbage bytes after the valid list or map. This issue is resolved in Aerospike Server version 4.6.0 or newer. For Aerospike Server version 4.5.3 or earlier this issue can be fixed by running the asvalidation tool.

Download asvalidation

Prior to Aerospike Tools version 6.0 asvalidation is available in the Aerospike Tools package. Otherwise, download asvalidation from https://github.com/aerospike/aerospike-tools-validation/releases .

Descriptions of possible corruption reasons

ReasonDescriptionDisposition
Has non-storageThe bin contains an infinite or wildcard element which is not allowed as storage.This type of error is unfixable without your manual intervention.
Has duplicate keysA map bin has duplicate key entries.This type of error is unfixable without your manual intervention.
CorruptedA problem not attributable to any of the other categories of errors.This type of error is unfixable without your manual intervention.
OrderThe bin has elements out of order.Can be fixed by reordering the list with the --cdt-fix-ordered-list-unique option. See server version requirements in Recommendations and Server Versions for When to Run asvalidation.
PaddingThe bin has garbage bytes after the valid list or map.Can be fixed by truncating the extra bytes. See server version requirements in Recommendations and Server Versions for When to Run asvalidation.

asvalidation Modes

asvalidation can be run in the following modes. Records without CDTs or detected errors are ignored. Records with detected errors are backed up unless otherwise specified. By default, no fixes are applied.

  • "Validation" mode discovers problems and produces a report.
  • "Fix" mode, triggered by the --cdt-fix-ordered-list-unique option, attempts to correct discovered problems where possible.

You should probably run asvalidation first in validation mode to see the kinds of errors it discovers before running it in fix mode to fix them.

Basic options for asvalidation

This is a minimal set of asvalidation options.

OptionDescription
--cdt-fix-ordered-list-uniqueFix lists whose elements were not stored in order and remove duplicate elements. Without this option, the tool only validates, not fixes.
-nNamespace to validate or fix.
-oOutput file name for validation report.
-dOutput directory for backed-up records.
--helpGet a comprehensive list of options for tool.

Running asvalidation in validation mode

Example of asvalidation in validation mode

In the following example, myNamespace is checked and its output stored in the file asvalidationOutput.txt.

Notice CDT Mode: validate.

> asvalidation -n myNamespace -o  asvalidationOutput.txt
...
2020-01-06 22:12:28 GMT [INF] [24662] Found 10 invalid record(s) from 1 node(s), 2620 byte(s) in total (~262 B/rec)
2020-01-06 22:12:28 GMT [INF] [24662] CDT Mode: validate
2020-01-06 22:12:28 GMT [INF] [24662] 100 Lists
2020-01-06 22:12:28 GMT [INF] [24662] 0 Unfixable
2020-01-06 22:12:28 GMT [INF] [24662] 0 Has non-storage
2020-01-06 22:12:28 GMT [INF] [24662] 0 Corrupted
2020-01-06 22:12:28 GMT [INF] [24662] 10 Need Fix
2020-01-06 22:12:28 GMT [INF] [24662] 0 Fixed
2020-01-06 22:12:28 GMT [INF] [24662] 0 Fix failed
2020-01-06 22:12:28 GMT [INF] [24662] 10 Order
2020-01-06 22:12:28 GMT [INF] [24662] 0 Padding
2020-01-06 22:12:28 GMT [INF] [24662] 0 Maps
2020-01-06 22:12:28 GMT [INF] [24662] 0 Unfixable
2020-01-06 22:12:28 GMT [INF] [24662] 0 Has duplicate keys
2020-01-06 22:12:28 GMT [INF] [24662] 0 Has non-storage
2020-01-06 22:12:28 GMT [INF] [24662] 0 Corrupted
2020-01-06 22:12:28 GMT [INF] [24662] 0 Need Fix
2020-01-06 22:12:28 GMT [INF] [24662] 0 Fixed
2020-01-06 22:12:28 GMT [INF] [24662] 0 Fix failed
2020-01-06 22:12:28 GMT [INF] [24662] 0 Order
2020-01-06 22:12:28 GMT [INF] [24662] 0 Padding

Interpreting the validation mode report

In the example above:

  • asvalidation was run in validation mode, because the --cdt-fix-ordered-list-unique option was not specified.
  • It found 100 lists: 100 Lists.
  • It found no maps: 0 Maps.
  • 10 of the lists were corrupted: 10 Need Fix under Lists.
  • The reason for corruption is shown as out of order: 10 Order.
  • Because it was run in validation mode, no fixes were applied: 0 Fixed.

Numbers under a heading do not necessarily add up to the count of the line. For example, there could be 1 Need Fix record with both an Order and Padding error.

Running asvalidation in fix mode

Fixing is triggered by the --cdt-fix-ordered-list-unique option.

Fixes are applied to the server. Failed fixes can be due to (but not limited to) an unsupported server version. See Recommendations and Server Versions for When to Run asvalidation.

Example of asvalidation fix mode

In the following example, myNamespace is checked and its output stored in the file asvalidationOutput.txt.

Notice CDT Mode: fix.

asvalidation -n test -o temp.txt -r --cdt-fix-ordered-list-unique
validation of 127.0.0.1 (namespace: test, set: [all], bins: [all], after: [none], before: [none]) to temp.txt
2020-01-23 22:08:25 GMT [INF] [10999] [src/main/aerospike/as_cluster.c:132][as_cluster_add_nodes_copy] Add node BB909000027000A 127.0.0.1:3000
2020-01-23 22:08:25 GMT [INF] [10999] Processing 1 node(s)
2020-01-23 22:08:25 GMT [INF] [10999] Node ID Objects Replication
2020-01-23 22:08:25 GMT [INF] [10999] BB909000027000A 130 1
2020-01-23 22:08:25 GMT [INF] [10999] Namespace contains 130 record(s)
2020-01-23 22:08:25 GMT [INF] [10999] Created new output file temp.txt
2020-01-23 22:08:25 GMT [INF] [11018] Starting validation for node BB909000027000A
2020-01-23 22:08:25 GMT [INF] [11018] Completed validation for node BB909000027000A, records: 30, size: 9200 (~306 B/rec)
2020-01-23 22:08:26 GMT [INF] [11017] 23% complete (~8 KiB/s, ~30 rec/s, ~306 B/rec)
2020-01-23 22:08:26 GMT [INF] [11017] ~3s remaining
2020-01-23 22:08:26 GMT [INF] [11017] Found 30 invalid record(s) from 1 node(s), 9200 byte(s) in total (~306 B/rec)
2020-01-23 22:08:26 GMT [INF] [11017] CDT Mode: fix
2020-01-23 22:08:26 GMT [INF] [11017] 110 Lists
2020-01-23 22:08:26 GMT [INF] [11017] 0 Unfixable
2020-01-23 22:08:26 GMT [INF] [11017] 0 Has non-storage
2020-01-23 22:08:26 GMT [INF] [11017] 0 Corrupted
2020-01-23 22:08:26 GMT [INF] [11017] 10 Need Fix
2020-01-23 22:08:26 GMT [INF] [11017] 10 Fixed
2020-01-23 22:08:26 GMT [INF] [11017] 0 Fix failed
2020-01-23 22:08:26 GMT [INF] [11017] 10 Order
2020-01-23 22:08:26 GMT [INF] [11017] 0 Padding
2020-01-23 22:08:26 GMT [INF] [11017] 20 Maps
2020-01-23 22:08:26 GMT [INF] [11017] 10 Unfixable
2020-01-23 22:08:26 GMT [INF] [11017] 10 Has duplicate keys
2020-01-23 22:08:26 GMT [INF] [11017] 0 Has non-storage
2020-01-23 22:08:26 GMT [INF] [11017] 0 Corrupted
2020-01-23 22:08:26 GMT [INF] [11017] 10 Need Fix
2020-01-23 22:08:26 GMT [INF] [11017] 0 Fixed
2020-01-23 22:08:26 GMT [INF] [11017] 0 Fix failed
2020-01-23 22:08:26 GMT [INF] [11017] 10 Order
2020-01-23 22:08:26 GMT [INF] [11017] 0 Padding