Orchestrating Aerospike Clusters with Docker Compose
This guide is accurate with the Docker 19.03.2 release. As Docker changes its APIs and recommendations this guide will evolve. Every attempt will be made to keep it current.
Background
Docker Compose is part of the Docker tool chain that enable multi-container applications to be orchestrated. This is an obvious mechanism to orchestrate the deployment and scaling of an Aerospike cluster.
Define the Aerospike Service
...
services:
aerospikedb:
image: aerospike/aerospike-server:latest
networks:
- aerospikenetwork
deploy:
replicas: 2
endpoint_mode: dnsrr
labels:
com.aerospike.cluster: "myproject"
command: [ "--config-file","/run/secrets/aerospike.conf"]
secrets:
- source: conffile
target: aerospike.conf
mode: 0440
...
Key points:
- Aerospike server version is defined (e.g. latest, 4.6.0.4)
- A declaration to use the network
aerospikenetwork
. This ensures that the discovery process can connect to the aerospike nodes to perform clustering. - A deploy parameter that specifies initial cluster size.
- A deploy parameter that specifies the discovery endpoint as
dnsrr
which is DNS round-robin. This is a Compose 3.2 feature - A label defines the cluster name (e.g. "com.aerospike.cluster=myproject")
- Append a command to use a custom conf file
- A secret is mounted that contains our aerospike.conf configuration file, as read-only
Define the Meshworker Discovery Service
...
meshworker:
image: aerospike/aerospike-tools:latest
networks:
- aerospikenetwork
depends_on:
- aerospike
entrypoint:
- /run/secrets/discovery
- "--servicename"
- aerospikedb
- "-i"
- "5"
- "-v"
secrets:
- source: discoveryfile
target: discovery
mode: 0750
...
Key points:
- Uses the aerospike/aerospike-tools image to trigger cluster reconfigurations
- A declaration to use the network
aerospikenetwork
. This ensures that the discovery process can connect to the aerospike nodes to perform clustering. - The discovery script can be found at discovery.py
- Depends_on: this is ignored in swarm mode, but obeyed with single node with compose. Ensures this service starts after the
aerospikedb
service. - Overwrite the entrypoint for
aerospike/aerospike-tools
container to run our discovery script- Passes in various parameters like: the service name to resolve
- The interval to poll the service name
- And a verbose flag for debugging/logging
- Mounts the discoveryfile secret as an executable script
Network definition
Starting with Compose v2, overlay networks can be defined as well.
...
networks:
aerospikenetwork:
driver: overlay
attachable: true
...
Key points:
- Uses the overlay driver for this network
- Turns on the attachable flag for easier debugging. Allows non-compose managed containers to also utilize this network.
Secrets definition
Secrets are used instead of volume mounts for two reasons:
- Secrets are propagated internally by docker, whereas volume mounts need to exist on the docker host (or swarm node) in order to be utilized.
- Long format declaration allows renaming and setting permissions/
...
secrets:
conffile:
file: ./aerospike.conf
discoveryfile:
file: ./discovery.py
Use Stack/Compose to deploy the Aerospike Containers
docker stack
can then be used in conjunction with your compose file and optionally your Docker Swarm cluster.
$ docker stack deploy -c aerospike.yml aerospike
You can see the Docker container running Aerospike on the Docker Swarm cluster using the docker ps
command.
$ docker ps
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
360e1da2d43f aerospike/aerospike-tools:latest "/run/secrets/disc..." 3 hours ago Up 3 hours aerospike_meshworker.1.zvs6cfhz0z8cebej5tdg4qdou
4d8fc0bd20f8 aerospike/aerospike-server:latest "/entrypoint.sh --..." 3 hours ago Up 3 hours 3000-3002/tcp aerospike_aerospikedb.3.v7edr5cvnnxvtep81beam9lqh
...
You can see your service status using the docker service ls command
docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
rfee38b7u9mt aerospike_meshworker replicated 1/1 aerospike/aerospike-tools:latest
ws864n7ivhy2 aerospike_aerospikedb replicated 2/2 aerospike/aerospike-server:latest
The name of the container is made up of
- Stack Name (by the `docker stack deploy` command) e.g. "aerospike"
- Service Name from the yaml file e.g. "aerospike"
- Numeric Id (indicates how many of the service has been started) e.g. "1"
- Task Id (internal id for every docker action)
For running Aerospike Enterprise Edition
The discovery.py
, aerospike_ee.yml
and aerospike_ee.conf
for EE edition usage can be found here.
Clone the repo aerospike/aerospike-docker-swarm
$ git clone https://github.com/aerospike/aerospike-docker-swarm
$ cd aerospike-docker-swarmCopy the
features.conf
file to the current path (same path asaerospike_ee.yml
,aerospike_ee.conf
anddiscovery.py
files)Note that
aerospike_ee.conf
file containsfeature-key-file
configuration underservice
stanza and it points to path/run/secrets/features.conf
. The path/run/secrets/
stores all the secrets defined inaerospike_ee.yml
.services:
aerospikedb:
image: aerospike/aerospike-server-enterprise:latest
networks:
- aerospikenetwork
deploy:
replicas: 3
endpoint_mode: dnsrr
labels:
com.aerospike.cluster: "myproject"
command: [ "--config-file","/run/secrets/aerospike_ee.conf"]
secrets:
- source: conffile
target: aerospike_ee.conf
mode: 0440
- source: featurekeyfile
target: features.conf
mode: 0440
....
....
secrets:
conffile:
file: ./aerospike_ee.conf
discoveryfile:
file: ./discovery.py
featurekeyfile:
file: ./features.confDeploy the stack
$ docker stack deploy -c aerospike_ee.yml aerospike
Use Service to scale the Aerospike Cluster
Docker services can be scaled dynamically using docker service scale
.
Ensure two Aerospike services (Containers) are running
$ docker service scale aerospike_aerospikedb=2
At this point, the discovery container would see the updated hostnames and automatically cluster the containers
View the cluster config
View the Aerospike cluster topology using asadm
.
$ docker run --net aerospike_aerospikenetwork aerospike/aerospike-tools asadm -h aerospikedb -e info
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node Node Ip Build Cluster Cluster Cluster Principal Client Uptime
. Id . . Size Key Integrity . Conns .
aerospike_aerospikedb.1.o0eujlsuyc8bylgzxt3svr61b.aerospike_aerospikenetwork:3000 *BB90301000A4202 10.0.1.3:3000 C-3.14.0 2 41339CCF5C67 True BB90301000A4202 5 02:50:39
aerospike_aerospikedb.3.v7edr5cvnnxvtep81beam9lqh.aerospike_aerospikenetwork:3000 BB90201000A4202 10.0.1.2:3000 C-3.14.0 2 41339CCF5C67 True BB90301000A4202 5 02:50:39
Number of rows: 2
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace Node Avail% Evictions Master Replica Repl Stop Pending Disk Disk HWM Mem Mem HWM Stop
. . . . (Objects,Tombstones) (Objects,Tombstones) Factor Writes Migrates Used Used% Disk% Used Used% Mem% Writes%
. . . . . . . . (tx%,rx%) . . . . . . .
test aerospike_aerospikedb.1.o0eujlsuyc8bylgzxt3svr61b.aerospike_aerospikenetwork:3000 99 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) 2 false (0,0) 0.000 B 0 50 0.000 B 0 60 90
test aerospike_aerospikedb.3.v7edr5cvnnxvtep81beam9lqh.aerospike_aerospikenetwork:3000 99 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) 2 false (0,0) 0.000 B 0 50 0.000 B 0 60 90
test 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) (0,0) 0.000 B 0.000 B
Number of rows: 3
Establishing an Aerospike Mesh Cluster dynamically
If the discovery container was not used, you would need to manually mesh the aerospike containers.
While both Containers are now on the same network, the Docker Overlay network does not support multi-cast. Therefore, each node has to be added into the Aerospike Cluster with the tip
command.
$ docker run --net aerospike_aerospikenetwork aerospike/aerospike-tools asinfo -v "tip:host=$(docker inspect -f '{{.NetworkSettings.Networks.prod.IPAddress }}' aerospike.2.nl30psiorabhcu2rj1kpghq6c );port=3002" -h aerospike.1.i6httgeicb9ix0vdevw8ye7xt
Additional Information
- See the Aerospike Docker Swarm GitHub Repo.
For namespaces using index-type
flash
, Aerospike container needs to be run in privileged
mode.
Aerospike writes to /proc/sys/vm/
during the startup to set certain parameters (dirty_bytes
, dirty_background_bytes
, dirty_expire_centisecs
, dirty_writeback_centisecs
) which control how data is written from page cache (dirty pages) to disk.
Note that swarm mode doesn't support running containers in privileged
mode yet.