Skip to main content

Streaming to Aerospike from Apache Kafka

Streaming from Apache Kafka to Aerospike Database Enterprise Edition relies on Aerospike's Kafka inbound connector, which is implemented as a "sink" connector in the open-source Kafka Connect framework. This inbound connector allows you to stream messages from Kafka, translates each message to an Aerospike record, and performs the corresponding insert, update, or delete for that record in Aerospike.

The inbound connector supports streaming data from one or more Kafka topics and persists the data in an Aerospike database. Each Kafka record is converted to an Aerospike record and persisted to an Aerospike cluster. You can even write custom code to manipulate Kafka records by performing operations on them that the connector does not support natively.

An instance of a connector can persist data only to a single Aerospike cluster. Aerospike acknowledges records that have been written successfully.

info

There might be instances where records are re-ordered on the network or across Kafka workers. In such cases, an older version of a record might overwrite a newer version of the record in Aerospike.