Skip to main content

Configuring ESP Outbound Connector

Configure the ESP outbound connector with the file aerospike-esp-outbound.yml. The location of this file depends on how you plan to deploy, or have already deployed, the ESP outbound connector:

  • If you deploy in Docker, you can locate this file anywhere in your system, as long as you include the path to it in the docker run command.
  • If you deploy in Kubernetes, this file is in a directory that also includes the deployment file.
  • If you deploy in a non-containerized environment, this file is located in the root-level /etc/aerospike-esp/ directory.

Setting Up Aerospike Database

In your Aerospike database, you must configure Cross-Datacenter Replication (XDR) and enable change notification. You must also configure your ESP cluster to be a "connector" XDR datacenter. The namespace must point to this as XDR remote datacenter.

See the change notification configuration parameters and example config.

Modifying aerospike-esp-outbound.yml

You configure streaming from Aerospike for ESP by modifying the /etc/aerospike-esp-outbound/aerospike-esp-outbound.yml YAML configuration file.

The configuration file has the following sections:

  • service Configures the connector's listening ports, TLS, and network interface.
  • destinations section configures HTTP-based destinations where change notifications are sent.
  • bin-transforms Specifies the bin transformations to apply on the Aerospike record.
  • format Specifies the message format to use for the outbound messages sent to the message broker.
  • routing Configures how incoming record updates and deletes from Aerospike are routed to the message broker.
  • namespaces Configures bin transforms, format, routing at namespace and set level.
  • logging Configures the destination and level for the connectors logs.
  • port-based-config Allows alternate configuration based on which port received the change notification.
  • executor-thread-pool-size Size of the thread pool used to dispatch HTTP requests. The default value is twice the number of available processors.
  • record-ordering Configure ordering of incoming records sent to the outbound destination.
caution

If the route is not specified for an Aerospike record, then the record is skipped and logged at the warn level.

Cascading Configuration Values

You can configure the bin-transforms, format, and routing sections to include default values, values for each particular namespace, and values for each particular set within a namespace. For each Aerospike record, the appropriate values are applied.

Here is an annotated example:

routing:
mode: static
destination: default <1>
‎‎
namespaces:
users:
routing:
mode: static
destination: users <2>
format:
mode: flat-json
metadata-key: metadata
sets:
premium:
routing:
mode: static
destination: premium <3>
  1. Specifies that records in all namespaces are to be shipped to the default topic.
  2. Specifies that records in the users namespace are to be shipped to the users topic.
  3. Specifies that records in the premium set in the users namespace are to be shipped to the premium topic.

Example

A sample version of /etc/aerospike-esp-outbound/aerospike-esp-outbound.yml is as follows:

service:
port:
- 8901
- 9901
manage:
port: 8902

logging:
enable-console-logging: true

# Destinations
destinations:
xdr-proxy-1:
urls:
- http://proxy1.example.com
max-requests-queued-per-endpoint: 10240
connection-ttl: 15000
max-connections-per-endpoint: 100
call-timeout: 10000
connect-timeout: 2000
health-check:
call-timeout: 10000
headers:
authority-header-omit-default-port: true
send-digest-header: true
additional-headers:
custom-header-1: custom-value-1
custom-header-2: custom-header-2

xdr-proxy-2:
urls:
- http://proxy2.example.com/

port-based-config:
8901:
routing:
mode: static-multi-destination
destinations: xdr-proxy1
9901:
routing:
mode: static-multi-destination
destinations: xdr-proxy2