Skip to main content
Loading

Configuring Streaming from Aerospike to JMS

Configuring streaming from Aerospike to JMS involves setting up your Aerospike database to send change notifications and modifying the configuration file for the JMS outbound connector: /etc/aerospike-jms-outbound/aerospike-jms-outbound.yml. "Outbound" means that the connector receives change notifications from Aerospike, translates them into JSON messages, and places the messages on topics or queues in a JMS message broker. This file is located on the system where you installed the connector package.

Setting Up Aerospike Database

In your Aerospike database, you must configure Cross-Datacenter Replication (XDR) and enable change notification. You must also configure your JMS cluster to be a "connector" XDR datacenter and the namespace must point to this as xdr remote datacenter.

See the change notification configuration parameters and example config.

Modifying aerospike-jms-outbound.yml

You configure streaming from Aerospike to JMS by modifying the /etc/aerospike-jms-outbound/aerospike-jms-outbound.yml YAML configuration file.

The configuration file has the following sections:

  • service Configures the connector's listening ports, TLS, and network interface.
  • jms Configures connection properties for the target JMS message broker.
  • bin-transforms Specifies the bin transformations to apply on the Aerospike record.
  • format Specifies the message format to use for the outbound messages sent to the message broker.
  • batching Specifies how to collect a group of Change Notification Records into a single batch of outbound messages sent to the message broker.
  • delivery-mode Chooses between persistent and non-persistent delivery of messages to the message broker.
  • routing Configures how incoming record updates/deletes from Aerospike are routed to the message broker.
  • namespaces Configures bin transforms, format, routing at namespace and set level.
  • logging Configures the destination and level for the connectors logs.
caution

If the route or format is not specified for an Aerospike record, then the record is skipped and logged at the warn level.

Cascading Configuration Values

You can configure the bin-transforms, format, and routing sections to include default values, values for each particular namespace, and values for each particular set within a namespace. For each Aerospike record, the appropriate values are applied.

Here is an annotated example:

routing:
mode: static
destination: default <1>
‎‎
namespaces:
users:
routing:
mode: static
destination: users <2>
format:
mode: flat-json
metadata-key: metadata
sets:
premium:
routing:
mode: static
destination: premium <3>
  1. Specifies that records in all namespaces are to be shipped to the default topic.
  2. Specifies that records in the users namespace are to be shipped to the users topic.
  3. Specifies that records in the premium set in the users namespace are to be shipped to the premium topic.

Example

A sample version of /etc/aerospike-jms-outbound/aerospike-jms-outbound.yml for connecting to IBM MQ is shown below:

service:
port: 8080

logging:
file: /var/log/aerospike-jms-outbound/aerospike-jms-outbound.log

jms:
factory: com.ibm.mq.jms.MQConnectionFactory
spec: 2.0
config:
hostName: 10.0.2.5
port: 1414
queueManager: QM1
transportType: 1
channel: DEV.APP.SVRCONN
credentials:
username: admin
password-file: /path/to/password/file.txt

delivery-mode: persistent

format: json

bin-transforms:
map:
yellow: red
transforms:
- uppercase

routing:
mode: bin
type: queue
bin: category
default: test-queue
transforms:
- trim
- regex:
pattern: '[^A-Za-z0-9]'
replacement: '-'
- lowercase

namespaces:
users:
routing:
mode: static
type: queue
destination: users
format:
mode: flat-json
metadata-key: metadata
sets:
premium:
routing:
mode: static
type: queue
destination: premium
bin-transforms:
map:
gold: platinum