Configuring the Kafka Sink (Inbound) Connector

The Aerospike Kafka sink (inbound) connector reads data from Apache Kafka and writes data to Aerospike.

Configuring streaming from Kafka to Aerospike involves setting the Kafka sink Connector to transform Kafka records into Aerospike records. This is driven by a YAML configuration file located at <connector-directory>/etc/aerospike-kafka-inbound.yml on each Kafka connect node.

Version 2.2.1 and later

The configuration has the following options:

Option	Required	Default	Description
max-queued-records	no	32768	The maximum number of records queued up with the connector. The size of the queue can go over this before topics are paused. All topics resume once the size of the queue drops under half of the maximum size.
processing-threads	no	Available processors	Number of threads to use for processing Kafka records and converting them to Aerospike records.
aerospike	yes		Configures the connection properties that the connector must use when connecting to your Aerospike database.
topics	yes		Configures the Kafka topics the connector listens to and the transformations to Aerospike records.

Here is an example:

max-queued-records: 10000

aerospike:
  seeds:
    - 192.168.50.1:
        port: 3000
        tls-name: red
    - 192.168.50.2
  cluster-name: east

topics:
  users:
    invalid-record: ignore
    mapping:
      namespace:
        mode: static
        value: users
      set:
        mode: dynamic
        source: value-field
        field-name: city
      key-field:
        source: key
      ttl:
        mode: dynamic
        source: value-field
        field-name: ttl
      bins:
        type: multi-bins
        map:
          name:
            source: value-field
            field-name: firstName

Version 2.2.0 and earlier

The configuration has the following options:

Option	Required	Default	Description
feature-key-file	yes for version before 2.2.0 no for version 2.2.0 and later		The location of the feature key file. Not required for versions 2.2.0 and later.
topics	yes		Configures the Kafka topics the connector listens to and the transformations to Aerospike records.
aerospike	yes		Configures the connection properties that the connector must use when connecting to your Aerospike database.

caution

feature-key-file is deprecated in version 2.2.0 and later. Make sure that the feature file with the feature key mesg-kafka-connector turned on is loaded onto the Aerospike server. The feature key is read directly from the Aerospike server.

Here is an example:

aerospike:
  seeds:
    - 192.168.50.1:
        port: 3000
        tls-name: red
    - 192.168.50.2
  cluster-name: east

topics:
  users:
    invalid-record: ignore
    mapping:
      namespace:
        mode: static
        value: users
      set:
        mode: dynamic
        source: value-field
        field-name: city
      key-field:
        source: key
      ttl:
        mode: dynamic
        source: value-field
        field-name: ttl
      bins:
        type: multi-bins
        map:
          name:
            source: value-field
            field-name: firstName