Configuring streaming from Aerospike to Elasticsearch involves setting up your Aerospike database to send change notifications and modifying the configuration file
Set Up Aerospike Database
In your Aerospike database, you must configure Cross-Datacenter Replication (XDR) and enable change notification. You must also configure your Elasticsearch outbound cluster to be a "connector" XDR datacenter and the namespace must point to this as xdr remote datacenter.
Modify the Config File
You configure streaming from Aerospike to elasticsearch by modifying the
/etc/aerospike-elasticsearch-outbound/aerospike-elasticsearch-outbound.yml YAML configuration file.
There are two levels at which various configuration can be defined:
|Root||This type of property can be configured only once at the root level of the configuration file.|
|All||This type of property can be configured at the root level, namespace level and/or at any of the sets' level of that namespace. Generally, this type of configuration is applied to the Aerospike record. The set level configuration takes the higher precedence. If absent, namespace level configuration is used for the record of that particular set or if set is absent. If configuration is also absent at the namespace level, the root level configuration is used for the given record.|
The configuration file has the following sections:
|Configures the connector's listening ports, TLS, and network interface.||Root|
|Configures Elasticsearch client properties for the target elasticsearch cluster.||Root|
|Specifies how to generate Elasticsearch's document id.||All|
|Specifies how to batch Aerospike records to send them to Elasticsearch in a single HTTP request.||All|
|Specifies the bin transformations to apply on the Aerospike record.||All|
|Specifies the message format to use for the outbound messages sent to the Elasticsearch cluster.||All|
|Configures how incoming record updates/deletes from Aerospike are routed to the Elasticsearch cluster.||All|
|Configures bin transforms, format, routing at namespace and set level.||Root|
|Configures the destination and level for the connector's logs.||Root|
|Configures the properties of Elasticsearch's Bulk API.||All|
|Specifies a set of error codes to ignore from Elasticsearch.||All|
If the route is not specified for an Aerospike record, then the record is skipped and logged at the warn level.
An example Aerospike Connect for Elasticsearch configuration:
# Optional TLS section
mutual-auth: true # For tcp server.
# Mutual auth is triggered only if this property is non-empty in http proxy server.
# Aerospike record routing to an Elasticsearch index.
# The digest is used as Elasticsearch's doc-id unless overridden at the child level.
# We support 3 types of authentication methods which is explained in the other file. Using api-key method here.
# Tls config to be used if Elasticsearch is configured to use TLS.
# There are several other options which are defined in the separate file.
# Overriding one of the batching parameter. Defaults are mentioned in the separate file.
# We are explicitly disabling batching here as we are using class, not a batch-formatter-class. This means that it only
# gets one record as an input. We get multiple records when batching is configured. This disables batching at
# customTransformer namespace unless some specific set overrides it for itself.
# We will consider the record to be successfully processed if Elasticsearch returns error code 400.
# Using bin-value type to generate a doc id. The default is digest at top.
# Example of bulk-request-config for the update operation. There are more supported config options which are
# defined on a separate page.
# Example for bulk-request-config for create operation. There are more supported config options which are
# defined on a separate page. Create operation means that this will succeed only if the doc doesn't exist in
# Elasticsearch. That's why we have used static doc-id for this set. Static doc-id won't be used practically
# by anyone. This is just for illustrative purpose.
Mapping between Aerospike and Elasticsearch models
|Table||Set||Index (version 6.0.0+)|
|Primary key||Primary key||document_id|
Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.