Configure Aerospike Connect for Elasticsearch

Configuring streaming from Aerospike to Elasticsearch involves setting up your Aerospike database to send change notifications and modifying the configuration file /etc/aerospike-elasticsearch-outbound/aerospike-elasticsearch-outbound.yml.

Set Up Aerospike Database

In your Aerospike database, you must configure Cross-Datacenter Replication (XDR) and enable change notification. You must also configure your Elasticsearch outbound cluster to be a "connector" XDR datacenter and the namespace must point to this as xdr remote datacenter.

See the change notification configuration parameters and example config.

Modify the Config File

You configure streaming from Aerospike to elasticsearch by modifying the /etc/aerospike-elasticsearch-outbound/aerospike-elasticsearch-outbound.yml YAML configuration file.

Configuration levels

There are two levels at which various configuration can be defined:

Level	Description
Root	This type of property can be configured only once at the root level of the configuration file.
All	This type of property can be configured at the root level, namespace level and/or at any of the sets' level of that namespace. Generally, this type of configuration is applied to the Aerospike record. The set level configuration takes the higher precedence. If absent, namespace level configuration is used for the record of that particular set or if set is absent. If configuration is also absent at the namespace level, the root level configuration is used for the given record.

Configuration sections

The configuration file has the following sections:

Section	Description	Configuration level
service	Configures the connector's listening ports, TLS, and network interface.	Root
es-client	Configures Elasticsearch client properties for the target elasticsearch cluster.	Root
doc-id	Specifies how to generate Elasticsearch's document id.	All
batching	Specifies how to batch Aerospike records to send them to Elasticsearch in a single HTTP request.	All
bin-transforms	Specifies the bin transformations to apply on the Aerospike record.	All
format	Specifies the message format to use for the outbound messages sent to the Elasticsearch cluster.	All
routing	Configures how incoming record updates/deletes from Aerospike are routed to the Elasticsearch cluster.	All
namespaces	Configures bin transforms, format, routing at namespace and set level.	Root
logging	Configures the destination and level for the connector's logs.	Root
bulk-request-config	Configures the properties of Elasticsearch's Bulk API.	All
ignore-error-codes	Specifies a set of error codes to ignore from Elasticsearch.	All

caution

If the route is not specified for an Aerospike record, then the record is skipped and logged at the warn level.

Configuration Example

An example Aerospike Connect for Elasticsearch configuration:

service:
  address: 0.0.0.0
  # Optional TLS section
  tls:
    port: 8080
    mutual-auth: true # For tcp server.
    # Mutual auth is triggered only if this property is non-empty in http proxy server.
    allowed-peer-names:
      - asd.aerospike.com
    trust-store:
      store-type: PEM
      certificate-files: tls/ca.aerospike.com.crt
    key-store:
      store-type: PEM
      store-file: tls/connector.aerospike.com.key.encrypted.pem
      store-password-file: tls/storepass
      certificate-chain-files: tls/connector.aerospike.com.crt

  manage:
    address: 0.0.0.0
    port: 8902

logging:
  file: /var/log/aerospike-elasticsearch-outbound/aerospike-elasticsearch-outbound.log

# Aerospike record routing to an Elasticsearch index.
routing:
  mode: static
  destination: aerospike

doc-id:
  # The digest is used as Elasticsearch's doc-id unless overridden at the child level.
  source: digest

es-client:
  cluster-config:
    type: on-prem
    nodes:
      - host:
          hostname: 192.168.123.234
          port: 9200
          scheme: https
  auth-config:
    # We support 3 types of authentication methods which is explained in the other file. Using api-key method here.
    type: api-key
    api-key-id-file: /tmp/api-key-id-file
    api-key-secret-file: /tmp/api-key-secret-file
  # Tls config to be used if Elasticsearch is configured to use TLS.
  tls-config:
    trust-store:
      store-file: tls/ca.aerospike.com.truststore.jks
      store-password-file: tls/storepass
  elasticsearch-request-config:
    # There are several other options which are defined in the separate file.
    connect-timeout: 10000

batching:
  # Overriding one of the batching parameter. Defaults are mentioned in the separate file.
  batch-interval: 1000

namespaces:
  customTransformer:
    batching:
      # We are explicitly disabling batching here as we are using class, not a batch-formatter-class. This means that it only
      # gets one record as an input. We get multiple records when batching is configured. This disables batching at
      # customTransformer namespace unless some specific set overrides it for itself.
      enabled: false
    format:
      mode: custom
      class: com.aerospike.connect.elasticsearch.outbound.TestElasticsearchOutboundCustomFormatter
    custom-transformer:
      class: com.aerospike.connect.outbound.TestOutboundCustomTransformer
    sets:
      ignoreErrorCodeSet:
        batching:
          enabled: false
        format:
          mode: custom
          class: com.aerospike.connect.elasticsearch.outbound.TestElasticsearchOutboundIgnoreErrorCodeCustomFormatter
        # We will consider the record to be successfully processed if Elasticsearch returns error code 400.
        ignore-error-codes:
          - 400

  root:
    sets:
      binTransformsSet:
        bin-transforms:
          map:
            original: transformed
          transforms:
            - uppercase
      docIdBinValueSet:
        doc-id:
          # Using bin-value type to generate a doc id. The default is digest at top.
          source: bin-value
          bin-name: es_doc_id
          failure-strategy: fail
      updateOperationSet:
        # Example of bulk-request-config for the update operation. There are more supported config options which are
        # defined on a separate page.
        bulk-request-config:
          ignore-aerospike-delete: true
          aerospike-write-operation-mapping:
            operation-type: update
            doc-as-upsert: true
      createOperationSet:
        doc-id:
          source: static
          value: UhP3zQBGIGbdqFIV5pqzdZB6rKA=
        # Example for bulk-request-config for create operation. There are more supported config options which are
        # defined on a separate page. Create operation means that this will succeed only if the doc doesn't exist in
        # Elasticsearch. That's why we have used static doc-id for this set. Static doc-id won't be used practically
        # by anyone. This is just for illustrative purpose.
        bulk-request-config:
          aerospike-write-operation-mapping:
            operation-type: create

Mapping between Aerospike and Elasticsearch models

RDBMS	Aerospike	Elasticsearch
Database	Namespace	Index
Table	Set	Index (version 6.0.0+)
Row	Record	Document
Column	Bin	Fields
	Partitions	Shards
Schema		Mappings
Primary key	Primary key	document_id

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.