Skip to main content

Filters

Filters allow users to specify which records will be shipped by XDR. Users need to use the Aerospike Expressions to build these filters. It provides a very rich interface in which users can express very complex logic. Users can select records based on the meta data of the record as well as content of the bins. These filters will be evaluated when the record is about to be shipped to the destination DC.

A few use cases in which these filters can be used are :

  • GDPR - If a record needs to be shipped based on the user’s profile.
  • Conserve network bandwidth - These filters allow a fine-grained control on which records will get shipped.
  • Change notification - XDR is also used to notify external systems. These filters can be used to send the notifications only if the record meets specific critera. For e.g., only if the account balance falls below a threshold.

Remember that the expressions are only meant to decide if a record will be shipped or not. It does not determine which bins get shipped based on the expression. The bin-policy still determines which bins get shipped.

Writing the filter

As mentioned above, Aerospike Expressions are used to express these filters. Get yourself familiar with the Expressions API. Minimally, look at the options available under

The following is an example of a filter written in C language. It selects all the records with a bin named 'age' and the bin-type is an integer and the integer value is >= 21.

#include <stdio.h>
#include <citrusleaf/alloc.h>
#include <aerospike/as_exp.h>
int main()
{
as_exp_build_b64(exp_b64,
as_exp_cmp_ge(as_exp_bin_int("age"), as_exp_int(21)));
printf("%s\n", exp_b64);
cf_free(exp_b64);
}

The following is the same example in Java

import java.util.Base64;
import com.aerospike.client.exp.Exp;
import com.aerospike.client.exp.Expression;
public class ExpressionTest {
public static void main(String args[]) {
Expression filter = Exp.build(Exp.ge(Exp.intBin("age"), Exp.val(21)));
System.out.println(filter.getBase64());
}
}

Dealing with deletes

If a namespace is configured to ship via XDR, an operation which deletes a record will leave a tombstone in place of the original record (until XDR successfully ships the record). The filters apply to these tombstones as well in case of XDR shipping. Tombstones will have metadata but no storage data (bin-level data). An important thing to note is that the storage expressions (bin-level) will evaluate to false for a tombstone. Metadata checks will apply as usual. If the overall expression evaluates to false, the record will not get shipped.

In the above example where we are checking age >= 21, it will not ship record delete operations as the “age” bin is no longer present and the expression will evaluate to false. To ship record deletes as well as records with age >= 21, the above code should be modified as below.

#include <stdio.h>
#include <citrusleaf/alloc.h>
#include <aerospike/as_exp.h>

int main()
{
as_exp_build_b64(exp_b64, as_exp_or(
as_exp_is_tombstone(),
as_exp_cmp_ge(as_exp_bin_int("age"), as_exp_int(21))));
printf("%s\n", exp_b64);

cf_free(exp_b64);
}

Setting the filter

XDR filters use the SMD infrastructure of Aerospike. The filter needs to be set via an info command on a single node in the cluster and SMD layer will distributed across the rest of the cluster. Any new node which joins the cluster will automatically get the existing filters via SMD.

Filters in XDR should be set for each namespace in a DC via the xdr-set-filter info command. This command expects the filter in base64 representation. The above examples illustrate how to convert the expression into the corresponding base64 representation.

Use the command below to set the filter:

asinfo -v "xdr-set-filter:dc=DC1;namespace=test;exp=kxGRSJMEk1ECo2FnZRU="

Use 'exp=null' as below to remove an existing filter:

asinfo -v "xdr-set-filter:dc=DC1;namespace=test;exp=null"

Setting the filter via client code

The latest C and Java client libraries provide an API to set the filter directly without needing to generate the base-64 output and setting it via the xdr-set-filter info command.

  • In C-client the API is aerospike_set_xdr_filter
  • In Java-client the API is setXDRFilter in AerospikeClient class.

Sample code to set the filter via C-client code will be as below:

#include <stdio.h>
#include <citrusleaf/alloc.h>
#include <aerospike/aerospike.h>
#include <aerospike/as_exp.h>
#include <aerospike/as_error.h>
#include <aerospike/as_config.h>

int main()
{
as_config config;
as_config_init(&config);
if (!as_config_add_hosts(&config, "127.0.0.1", 3000)) {
printf("Invalid host");
exit(-1);
}

aerospike as;
aerospike_init(&as, &config);

as_error err;
if (aerospike_connect(&as, &err) != AEROSPIKE_OK) {
printf("Failed to connect : %d - %s", err.code, err.message);
exit(-1);
}

as_exp_build_b64(exp_b64, as_exp_or(
as_exp_is_tombstone(),
as_exp_cmp_ge(as_exp_bin_int("age"), as_exp_int(21))));

if (aerospike_set_xdr_filter(&as, &err, NULL, "DC1", "test", exp_b64) != AEROSPIKE_OK) {
printf("Failed to set filter : %d - %s", err.code, err.message);
}

cf_free(exp_b64);

return 0;
}

The following is the same example in Java

import com.aerospike.client.AerospikeClient;
import com.aerospike.client.AerospikeException;
import com.aerospike.client.command.ParticleType;
import com.aerospike.client.exp.Exp;
import com.aerospike.client.exp.Expression;

public class ExpressionTest {

public static void main(String args[]) throws AerospikeException {

AerospikeClient client = new AerospikeClient("127.0.0.1", 3000);

Expression filter = Exp.build(Exp.or(Exp.isTombstone(), Exp.ge(Exp.intBin("age"), Exp.val(21))));

try {
client.setXDRFilter(null, "DC1", "test", filter);
} catch (AerospikeException e) {
System.out.println("Failed to set filter " + e.getMessage());
}

client.close();
}
}

Getting the filter

Use the command below to see which filter is set:

asinfo -v "xdr-get-filter:dc=DC1;namespace=test"

output: namespace=test:exp=or(is_tombstone(), ge(bin_int("age"), 21))

Note: In version 5.3, the above command used to display the filter in base-64 format.

In version 5.4 and above, use the command below to get the filter in base-64 format:

asinfo -v "xdr-get-filter:dc=DC1;namespace=test;b64=true"

output: namespace=test:exp=kxGRSJMEk1ECo2FnZRU=

Examples of filters

  • Ships records from sets 'shipped_set1' and 'shipped_set2'. Note that this filter will ship tombstones too from these sets as metadata checks will apply as usual for tombstones.

    as_exp_build_b64(exp_b64, as_exp_or(
    as_exp_cmp_eq(as_exp_set_name(), as_exp_str("shipped_set1")),
    as_exp_cmp_eq(as_exp_set_name(), as_exp_str("shipped_set2"))));
    printf("%s\n", exp_b64);
    cf_free(exp_b64);
  • Ships records only if 'bin1' is an integer bin.

    as_exp_build_b64(exp_b64,
    as_exp_cmp_ne(as_exp_bin_type("bin1"), as_exp_int(AS_BYTES_INTEGER)));
    printf("%s\n", exp_b64);
    cf_free(exp_b64);
  • Ships records only if 'country' is 'NL'

    as_exp_build_b64(exp4_b64
    as_exp_cmp_eq(as_exp_bin_str("country"), as_exp_str("NL")));
    printf("%s\n", exp_b64);
    cf_free(exp_b64);