TLS Configuration
Aerospike Database Enterprise Edition (EE) and Aerospike Database Enterprise Edition for United States Federal (FE) support Transport Layer Security (TLS) for encrypting the following network traffic:
- Client-to-Cluster - traffic between clients and all Database Nodes
- Cluster-to-Cluster - cross-datacenter replication (XDR) traffic between clusters
- Node-to-Node - fabric and heartbeat traffic between nodes in the same cluster
Planning TLS encryption and type
If you are planning to use TLS encryption, contact Aerospike Support for a smooth migration.
Refer to TLS Guide for an overview of the TLS functionality. If you require standard authentication or mutual authentication (PKI authentication with mTLS), select what to encrypt and the certificate type.
When using TLS you can choose to encrypt:
- Traffic between your clients and cluster nodes
- Fabric and/or heartbeat traffic between cluster nodes
- Cross-datacenter (XDR) traffic
Select the certificate match type:
- Cluster-name Match
- Wildcard Hostname Match
- Match with Common Server Certificate, or
- Hostname Match with Individual Server Certificate
Once the TLS deployment artichecture is determined, generate the appropriate certificates and follow the upgrade scenario below.
This page covers Aerospike EE versions 3.15 and above. Configuration syntax for TLS is different for versions 3.11 to 3.14. Refer to the TLS Guide pre 3.15 and the Network Configuration/TLS/Pre 3.15 page for details for the specific versions.
Dynamically rotating ECDSA private keys and certificates as well as password-protected private keys is not supported in Aerospike server versions prior to 4.7.0.5, 4.6.0.8, 4.5.3.10, 4.5.2.10, 4.5.1.15, 4.5.0.19. More details in the FAQ on TLS.
Configuring TLS
Refer to Configuration Reference for details regarding required and optional parameters.
Legend
- (*) required parameters for TLS
- (**) required parameters for standard authentication or mutual authentication
- (+) optional parameters for mutual authentication
network {
tls tlsname1 { # (*) This is the TLS name to be referred to in subsequent network
# sub-stanzas that woudld require TLS. Will be populated in the
# tls-name wire field for incoming connections. Could be set to
# <cluster-name> or <hostname> or user-defined.
# Set to `tlsname1` in this example.
cert-file path-to-file # (**) Required for standard authentication or mutual authentication
key-file path-to-file # (**) Required for standard authentication or mutual authentication
ca-file path-to-file # (+) For mutual authentication or for XDR.
# Only one of ca-file / ca-path required.
ca-path directory-path # (+) For mutual authentication or for XDR.
# Only one of ca-file / ca-path required.
protocols TLSv1.2 # Protocol version to include
# note - In version 4.6 the default protocols configuration parameter
# was changed from “-all,+TLSv1.2” to “TLSv1.2”.
cipher-suite ALL:!COMPLEMENTOFDEFAULT:!eNULL
# Ciphers to includes.
cert-blacklist path-to-file # Blacklist including serial number of certs to revoke
}
tls tlsname2 {
ca-file /tls/ca-cert2.pem
cert-file /tls/rem-chain2.pem
key-file /tls/rem-key2.pem
...
}
tls tlsname3.source.domain {
...
}
tls <cluster-name> {
...
}
...
service {
port 3000
address 192.168.80.100 # Bind address for non TLS traffic.
# Default to `tls-address` if not specified.
tls-port 4333 # (*) Enables tls
tls-address 192.168.90.150 # Bind address for TLS.
# Will default to `address` if not specified.
tls-authenticate-client false|any|user-defined
# If not specified, default is any, which is mutual authentication.
# Multiple tls-authenticate-client entries can be defined in order
# to accept multiple subject names. At typical example would be to
# use different subject names between local clients and XDR clients
# (when specifying the subject name of course).
tls-name <cluster-name>|<hostname>|user-defined
# (*) Which TLS config to use (from tls sub-stanza).
# This is also the name that would be populated in the tls-name
# wire field. (For example, tlsname1 to refer to the first
# tls sub-stanza described above).
}
heartbeat {
tls-port 3012
tls-name tlsname2
tls-mesh-seed-address-port 192.168.90.151 3012
...
}
fabric {
tls-port 3011
tls-name tlsname2
...
}
...
}
...
The TLS name (specified as part of the tls sub-stanza and referred to within the service, fabric and heartbeat sub-stanzas as well as XDR stanzas)
has the ability to be substituted. For example:
tls
/tls-name
<cluster-name>
means the server will use what's defined incluster-name
as the tls-name attribute.tls
/tls-name
<hostname>
means the server will use what's returned by the underlying OS's "hostname" system call as the tls-name attribute.
ca-path
: it may be required to run a "c_rehash" to generate symlinks based on hash values.
Configuring XDR with TLS
For the details on configuring XDR with TLS, see Securing with TLS.
XDR configuration stanza for pre-5.0 servers
The following xdr
stanza applies only to versions of Aerospike Server prior to 5.0.
xdr {
dc DC1 {
node-address-port 10.10.1.1 3010 tlsname2
# Remote nodes IP address along with TLS name a cluster node expects
# the remote DC to present on XDR connections.
...
tls-name tlsname3.source.domain # The remote cluster should have a tls-authenticate-client directive
# specifying this TLS name, or `false`, or `any`.
...
}
dc DC2 {
... # Can have its own set of tls configs
}
...
}
Blacklisting certificates
Certificates that have been compromised can be blacklisted, preventing malicious applications from connecting to the server.
Configuring the certificates blacklist
The serial number assumes hex representation, with no non-hex characters such as 'x' or ":" etc.
To add a certificate's information to the blacklist, you must have the certificate available to extract the properly formatted serial number:
cat chain.pem | openssl x509 -noout -serial -issuer | \
awk 'match($0,/[^=]*=(.*)/, a) { printf "%s", a[1] } END {printf "\n"}'
Refreshing the certificates blacklist
The certificates blacklist is automatically refresh when using Aerospike EE version 3.15 and above. For versions 3.11 to 3.14, use the following command to refresh the list:
asinfo -v 'set-config:context=xdr;dc=REMOTE;tls-cert-blacklist=.../path/blacklist.txt'
Upgrading to TLS
High level overview for upgrades.
Upgrade client to cluster connections to TLS
For an existing deployment and to upgrade to TLS, follow these steps:
- Upgrade Server
- Rolling Upgrade each node in the cluster to a software version that is TLS aware (3.11 and above), while adding in new TLS enabled configurations. Do not shutdown clear ports.
- Repeat for all nodes in the cluster.
- During this time, client will continue to communicate with cluster via clear transport.
- Upgrade Client
- Once every node is TLS enabled, start upgrading client to a software version that is TLS aware, while adding in new TLS enabled client configurations.
- During this time, old clients will continue to communicate with cluster via clear transport, while new clients will communicate with cluster via TLS.
- Repeat for all application nodes.
- Shut down clear (non tls) ports (optional)
- If no clients should be allowed to reach the server, an additional Rolling Restart is required to remove and shutdown the clear port.
Upgrade from standard-authentication to mutual-authentication
For an existing deployment already running standard-authentication to be upgraded to mutual-authentication, follow these steps:
- Upgrade Client
- Add the client certificate/keys.
- Rolling restart all clients so they are aware of the new certificates when asked.
- Update Server
- Change server configuration to include
ca-file
orca-path
- Update
tls-authenticate-client
toany
or a user defined string representing a tls name to be validated (if validation is required). - Rolling restart each of the node.
- As each node joins the cluster back, the newly established TLS connections will require the clients to present their certificates.
Upgrade intra node connections to TLS
Rolling upgrades from non-TLS to TLS for intra node connections is supported. The upgrade happens in two rounds:
- Enable TLS for heartbeat and fabric across all nodes in a rolling fashion.
- Keep non-TLS ports enabled while doing so.
- This first step results in a full cluster supporting both, non-TLS as well as TLS.
When upgrading from a version prior to 3.13.0.10 or 3.14.1.9 (which do not support intra node TLS), first upgrade all nodes to
version 3.15 and then enable TLS for heartbeat and fabric in a rolling fashion. Nodes running an older version
will try connecting to the TLS port first and will fail causing the cluster to not properly form. This will cause warnings to be
printed in the log file for each fabric connection attempt. Warning message will look as such:
WARNING (tls): (tls_ee.c:277) SSL_accept failed: error:1408F10B:SSL routines:
SSL3_GET_RECORD:wrong version number
WARNING (fabric): (fabric.c:1400) fabric TLS server handshake with 10.10.1.1:50001 failed
Versions 3.13.0.10 and 3.14.1.9 do support such upgrade due to the following improvement:
[AER-5826] - (CLUSTERING) Support rolling upgrades to later versions in which fabric TLS is enabled.
Under normal circumstances, such cluster configured with both non-TLS and TLS fabric/heartbeat will default to TLS. However, downgrade attacks are possible, where an attacker prevents the cluster from establishing TLS connections forcing the cluster to fall back to non-TLS.
- Disable non-TLS for heartbeat and fabric across all nodes in a rolling fashion.
- At this point, the cluster only supports TLS connection for its node to node communication. This prevents downgrade attacks.
Upgrade XDR connections to TLS
For an existing deployment with XDR which will be upgraded to TLS, follow these steps
- Upgrade Destination Clusters
- For each one of the destination cluster, Follow the procedure in "Upgrade client to cluster connections to TLS" to enable TLS ports.
- Upgrade Source Cluster
- Then for each Source Cluster node, upgrade XDR's DC configuration to use the newly configured TLS port.
- Shutdown clear ports (optional)
- An additional Rolling Restart is required to shutdown destination cluster nodes's clear port (port 3000).
Examples
Using cluster-name standard authentication
Server Configuration
service {
...
cluster-name as-cluster-west
}
network {
tls <cluster-name> {
cert-file /home/citrusleaf/x509_certificates/chainless_cluster_chain.pem
key-file /home/citrusleaf/x509_certificates/Chainless_cluster/key.pem
}
service {
tls-port 4333
tls-name <cluster-name>
tls-authenticate-client false
...
}
heartbeat {
...
protocol v3
}
}
C Client Benchmark Call
target/benchmarks -h "192.168.113.203:as-cluster-west:4333" --tlsEnable --tlsCaFile ~/x509_certificates/Platinum/cacert.pem
Java Client Benchmark Call
java -Djavax.net.ssl.trustStore=<TrustStorePath> -Djavax.net.ssl.trustStorePassword=<TrustStorePassword> -jar target/aerospike-benchmarks-*-jar-with-dependencies.jar -h "192.168.113.203:as-cluster-west:4333" -tlsEnable
Using hostname match with a common server certificate, mutual authentication (without name validation), and client-certificate blacklisting
Server Configuration
network {
tls <hostname> {
ca-file /home/citrusleaf/x509_certificates/Platinum/cacert.pem # Root CA's cert
cert-file /home/citrusleaf/x509_certificates/multi_chain.pem
key-file /home/citrusleaf/x509_certificates/MultiServer/key.pem
cert-blacklist /root/certs_subject/blacklist.txt
}
service {
tls-port 4333
tls-name <hostname>
tls-authenticate-client any
...
}
}
C Client Benchmark Call
target/benchmarks -h "192.168.113.203:t3.t-cluster.aerospike.com:4333" --tlsEnable --tlsCaFile ~/x509_certificates/Platinum/cacert.pem --tlsChainFile ~/x509_certificates/client_chain.pem --tlsKeyFile ~/x509_certificates/Client/key.pem
Java Client Benchmark Call
java -Djavax.net.ssl.trustStore=<TrustStorePath> -Djavax.net.ssl.trustStorePassword=<TrustStorePassword> Djavax.net.ssl.keyStore=<KeyStorePath> -Djavax.net.ssl.keyStorePassword=<KeyStorePassword> -jar target/aerospike-benchmarks-*-jar-with-dependencies.jar -h "192.168.113.203:t-cluster.aerospike.com:4333" -tlsEnable
Using hostname match with individual server certificate, and server-certificate blacklisting
Server Configuration
network {
tls <hostname> {
cert-file /home/citrusleaf/x509_certificates/server1_chain.pem
# note - different for each server node.
key-file /home/citrusleaf/x509_certificates/Server1/key.pem
# note - different for each server node.
}
service {
tls-port 4333
tls-name <hostname>
tls-authenticate-client false
}
}
C Client Benchmark Call
target/benchmarks -h "192.168.113.203:t3.t-cluster.aerospike.com:4333" --tlsEnable --tlsCaFile ~/x509_certificates/Platinum/cacert.pem --tlsCertBlackList ~/bad_serials.txt
Java Client Benchmark Call
java -Djavax.net.ssl.trustStore=<TrustStorePath> -Djavax.net.ssl.trustStorePassword=<TrustStorePassword> -jar target/aerospike-benchmarks-*-jar-with-dependencies.jar -h "192.168.113.203:t-cluster.aerospike.com:4333" -tlsEnable -tlsRevoke 0xfaa7f3e06e288466c5420e0c7ccd4c45
Generating Self-Signed TLS Certificates
Developers may want to generate self-signed TLS certificates for their own development environment. The following gives an example for doing this.
Prepare the certificate directories
Prepare certificate directories for input, local, output, and follow this directory structure:
mkdir -p rootca 2>/dev/null
rm -rf rootca/* 2>/dev/null
mkdir -p rootca/input
mkdir -p rootca/local
mkdir -p rootca/output
cd rootca
Configure OpenSSL
cat <<'EOF' > openssl.conf
HOME = .
oid_section = new_oids
[ new_oids ]
tsa_policy1 = 1.2.3.4.1
tsa_policy2 = 1.2.3.4.5.6
tsa_policy3 = 1.2.3.4.5.7
[ ca ]
default_ca = CA_default # The default ca section
[ CA_default ]
dir = rootca # Where everything is kept
certs = $dir/certs # Where the issued certs are kept
crl_dir = $dir/crl # Where the issued crl are kept
database = $dir/index.txt # database index file
# several certs with same subject
new_certs_dir = $dir/newcerts # default place for new certs
certificate = $dir/cacert.pem # The CA certificate
serial = $dir/serial # The current serial number
crlnumber = $dir/crlnumber # the current crl number
# must be commented out to leave a V1 CRL
crl = $dir/crl.pem # The current CRL
private_key = $dir/private/cakey.pem# The private key
x509_extensions = usr_cert # The extensions to add to the cert
name_opt = ca_default # Subject Name options
cert_opt = ca_default # Certificate field options
default_days = 365 # how long to certify for
default_crl_days= 30 # how long before next CRL
default_md = default # use public key default MD
preserve = no # keep passed DN ordering
policy = policy_match
[ policy_match ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
[ policy_anything ]
countryName = (2 letter code, for example, US)
stateOrProvinceName = State or Province Name (full name)
localityName = Locality Name (such as, city)
organizationName = Organization Name (such as, company)
organizationalUnitName = Organizational Unit Name (such as, section or department)
commonName = Common Name (such as, server FQDN or YOUR name)
emailAddress = joedo@joedocompany.com
[ req ]
default_bits = 2048
default_keyfile = privkey.pem
distinguished_name = req_distinguished_name
attributes = req_attributes
x509_extensions = v3_ca # The extensions to add to the self signed cert
string_mask = utf8only
[ req_distinguished_name ]
countryName = Country Name (2 letter code, for example, US)
countryName_default = US
countryName_min = 2
countryName_max = 2
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = Some-State
localityName = Locality Name (such as, city)
0.organizationName = Organization Name (such as, company)
0.organizationName_default = Your company name
organizationalUnitName = Organizational Unit Name (such as, section)
commonName = Common Name (such as, server FQDN or YOUR name)
commonName_max = 64
emailAddress = joedo@joedocompany.com
emailAddress_max = 64
[ req_attributes ]
challengePassword = A challenge password
challengePassword_min = 4
challengePassword_max = 20
unstructuredName = An optional company name
[ usr_cert ]
basicConstraints=CA:FALSE
nsComment = "OpenSSL Generated Certificate"
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = critical,CA:true
[ crl_ext ]
authorityKeyIdentifier=keyid:always
[ proxy_cert_ext ]
basicConstraints=CA:FALSE
nsComment = "OpenSSL Generated Certificate"
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
proxyCertInfo=critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo
[ tsa ]
default_tsa = tsa_config1 # the default TSA section
[ tsa_config1 ]
dir = ./demoCA # TSA root directory
serial = $dir/tsaserial # The current serial number (mandatory)
crypto_device = builtin # OpenSSL engine to use for signing
signer_cert = $dir/tsacert.pem # The TSA signing certificate
# (optional)
certs = $dir/cacert.pem # Certificate chain to include in reply
# (optional)
signer_key = $dir/private/tsakey.pem # The TSA private key (optional)
signer_digest = sha256 # Signing digest to use. (Optional)
default_policy = tsa_policy1 # Policy if request did not specify it
# (optional)
other_policies = tsa_policy2, tsa_policy3 # acceptable policies (optional)
digests = sha1, sha256, sha384, sha512 # Acceptable message digests (mandatory)
accuracy = secs:1, millisecs:500, microsecs:100 # (optional)
clock_precision_digits = 0 # number of digits after dot. (optional)
ordering = yes # Is ordering defined for timestamps?
# (optional, default: no)
tsa_name = yes # Must the TSA name be included in the reply?
# (optional, default: no)
ess_cert_id_chain = no # Must the ESS cert id chain be included?
# (optional, default: no)
ess_cert_id_alg = sha1 # algorithm to compute certificate
# identifier (optional, default: sha1)
EOF
Generate the Certificate Authority (CA) root directory
openssl genrsa -out local/rootCA.key 2048
openssl req -x509 -new -nodes -key local/rootCA.key -sha256 -days 3650 -out local/rootCA.pem -subj "/C=UK/ST=London/L=London/O=fed/OU=Support/CN=rootca.fed"
Generate TLS certificate requests
The following example generates a TLS certificate for the server, and a certificate for the admin user. This pair of certs can used for Mutual TLS (mTLS) with the clients, as well as PKI authentication for the users.
The tls-name
of the TLS context used in the service stanza must also be set as
Common Name (CN
) of the server's certificate. For example CN=server
.
For PKI auth the CN
of the user's certificate must match their username.
For example CN=admin
.
openssl req -new -nodes -config openssl.conf -extensions v3_req -out input/server.req -keyout output/server.key -subj "/C=UK/ST=London/L=London/O=fed/OU=Server/CN=server"
openssl req -new -nodes -config openssl.conf -extensions v3_req -out input/admin.req -keyout output/admin.key -subj "/C=UK/ST=London/L=London/O=fed/OU=Admin/CN=admin"
Example: Generate a user certificate
For each user, create a user certificate
export USERNAME=app
openssl req -new -nodes -config openssl.conf -extensions v3_req -out input/${USERNAME}.req -keyout output/${USERNAME}.key -subj "/C=UK/ST=London/L=London/O=fed/OU=Admin/CN=${USERNAME}"
openssl x509 -req -extfile openssl.conf -in input/${USERNAME}.req -CA local/rootCA.pem -CAkey local/rootCA.key -extensions v3_req -days 3649 -outform PEM -out output/${USERNAME}.pem -set_serial 310
openssl verify -verbose -CAfile local/rootCA.pem output/${USERNAME}.pem
Sign the certificates
openssl x509 -req -extfile openssl.conf -in input/server.req -CA local/rootCA.pem -CAkey local/rootCA.key -extensions v3_req -days 3649 -outform PEM -out output/server.pem -set_serial 110
openssl x509 -req -extfile openssl.conf -in input/admin.req -CA local/rootCA.pem -CAkey local/rootCA.key -extensions v3_req -days 3649 -outform PEM -out output/admin.pem -set_serial 310
Verify the certificates
openssl verify -verbose -CAfile local/rootCA.pem output/server.pem
openssl verify -verbose -CAfile local/rootCA.pem output/admin.pem
TLS-related configuration parameters and references
List of the TLS related configuration parameters:
To be used in the network's
tls
sub-stanza:
- To be used in the network's service, fabric, heartbeat sub-stanzas and xdr stanza:
tls-name
tls-port
(service, fabric and heartbeat sub-stanzas only)tls-authenticate-client
(service sub-stanza only)tls-address
(service, fabric and heartbeat sub-stanzas only)tls-mesh-seed-address-port
(heartbeat sub-stanza only)