Install the Aerospike Kafka Source Connector
Installer
- Version 5.0.0 and later
- Version 4.1.0 and earlier
Starting with version 5.0.0
, there is a single installer that handles both XDR connector 5.0 wire protocol and HTTP.
Go to the Aerospike Enterprise downloads
site to download the connector package for your platform and install it.
Remember to set the protocol
based on the change notification source. See
protocol for details.
The communication protocol between Aerospike Database and the Kafka source connector has changed for Aerospike 5.0 and later. There are separate Aerospike Kafka source (outbound) installers and container images for both these versions.
- aerospike-kafka-outbound-[version].[package] for Aerospike 5.0 and later
- aerospike-kafka-outbound-http-[version].[package] for Aerospike prior to 5.0
- aerospike/aerospike-kafka-outbound:[version]
- aerospike/aerospike-kafka-outbound-http:[version]
Prerequisites
We have tested and verified the source connector with:
- Apache Kafka version 2.0.x or later
- Confluent Platform 5.0
The connector has not been tested with Apache Kafka 3.0.x or Confluent Platform 7.0.
Aerospike Server change notification configuration
You must configure your Aerospike database to send change notifications in order to use a source connector to stream from Aerospike to Kafka. For the steps, see Configuring Change Notification for Use with Connectors.
Deploying on Docker
Aerospike provides Docker images for the Aerospike Kafka source connector on Docker Hub. Choose the version compatible with your Aerospike database.
- Version 5.0.0 and later
- Version before 4.0.0
sudo docker run -p 8080:8080 -v
/home/aerospike/Downloads/aerospike-kafka-outbound.yml:/etc/aerospike-kafka-outbound/aerospike-kafka-outbound.yml aerospike/aerospike-kafka-outbound:5.0.0
# With Aerospike server versions 5.0.0.0 and later
sudo docker run -p 8080:8080 -v /home/aerospike/Downloads/aerospike-kafka-outbound.yml:/etc/aerospike-kafka-outbound/aerospike-kafka-outbound.yml aerospike/aerospike-kafka-outbound:4.1.0
# With Aerospike server versions before 5.0.0.0
sudo docker run -p 8080:8080 -v /home/aerospike/Downloads/aerospike-kafka-outbound.yml:/etc/aerospike-kafka-outbound/aerospike-kafka-outbound.yml aerospike/aerospike-kafka-outbound-http:4.1.0
Installing on Linux
Prerequisites
Supported operating systems
The connector is supported on the following operating systems:
- RHEL (8 and 9) / CentOS 7
- Ubuntu 20.04+
- Ubuntu 22.04+
- Debian 10 (only on x86_64 hardware)
- Debian 11
Java Runtime Environment
This connector is a Java web-application that requires Java 8 or later. If you plan to use TLS v1.3, then you must use Java 11 or later. Both Oracle and OpenJDK Java Runtime Environments are supported.
Support for Java 8 will be dropped in future releases.
Java 17 is supported starting with Aerospike Kafka source connector (outbound) version 4.1.0.
Compatibility with versions of Aerospike Database Enterprise Edition
If you are using Aerospike Database Enterprise Edition version 5.6 or later, ensure that you use version 4.0.0 or later of the Aerospike Kafka source (outbound) connector.
If you are using Aerospike Database Enterprise Edition version 5.5 or earlier, you may use any version of the Aerospike Kafka source (outbound) connector.
Installation
The installation steps below apply to both the Debian (*.deb) and RHEL (*.rpm) packages.
Install Java 8 or later version
The Aerospike Kafka source connector package does not include a Java runtime environment. Most supported platforms provide official JDK 8 packages. For platforms that do not, Oracle's JDK 8 builds can be obtained from jdk.java.net/8/.
RHEL, CentOS, etc.
sudo yum install java-1.8.0-openjdk
Debian, Ubuntu, etc.
sudo apt-get install openjdk-8-jre
Install connector package
Go to the Aerospike Enterprise downloads site to download the Aerospike Kafka source (outbound) connector package for your platform and install it.
RHEL, CentOS, etc.
- Version 5.0.0 and later
- Version 4.1.0 and earlier
sudo rpm -i aerospike-kafka-outbound-5.0.0-1.noarch.rpm
# With Aerospike server versions 5.0.0.0 and later
sudo rpm -i aerospike-kafka-outbound-4.1.0-1.noarch.rpm
# With Aerospike server versions before 5.0.0.0
sudo rpm -i aerospike-kafka-outbound-http-4.1.0-1.noarch.rpm
Debian, Ubuntu, etc.
- Version 5.0.0 and later
- Version 4.1.0 and earlier
sudo dpkg -i aerospike-kafka-outbound-5.0.0.all.deb
# With Aerospike server versions 5.0.0.0 and later
$ sudo dpkg -i aerospike-kafka-outbound-4.1.0.all.deb
# With Aerospike server versions before 5.0.0.0
$ sudo dpkg -i aerospike-kafka-outbound-http-4.1.0.all.deb
Verify/update source connector configuration
The source connector's configuration can be found in the /etc/aerospike-kafka-outbound/
directory. For details on how to configure the Kafka provider, enable
TLS, etc., please refer to the configuration guide.
Start the source connector
The source connector package includes a Systemd service definition. The installation procedure creates an aerospike-kafka-outbound service.
Enable the source connector to start on system startup or reboot.
- Version 5.0.0 and later
- Version 4.1.0 and earlier
$ sudo systemctl enable aerospike-kafka-outbound
# With Aerospike server versions 5.0.0.0 and later
$ sudo systemctl enable aerospike-kafka-outbound
# With Aerospike server versions before 5.0.0.0
$ sudo systemctl enable aerospike-kafka-outbound-http
To start the source connector service run.
- Version 5.0.0 and later
- Version 4.1.0 and earlier
$ sudo systemctl start aerospike-kafka-outbound
# With Aerospike server versions 5.0.0.0 and later
$ sudo systemctl start aerospike-kafka-outbound
# With Aerospike server versions before 5.0.0.0
$ sudo systemctl start aerospike-kafka-outbound-http