TLS Configuration

Aerospike Database Enterprise Edition (EE) and Aerospike Database Enterprise Edition for United States Federal (FE) support Transport Layer Security (TLS) for encrypting the following network traffic:

• Client-to-Cluster - traffic between clients and all Database Nodes
• Cluster-to-Cluster - cross-datacenter replication (XDR) traffic between clusters
• Node-to-Node - fabric and heartbeat traffic between nodes in the same cluster

Planning TLS encryption and type

If you are planning to use TLS encryption, contact Aerospike Support for a smooth migration.

Refer to TLS Guide for an overview of the TLS functionality. If you require standard authentication or mutual authentication (PKI authentication with mTLS), select what to encrypt and the certificate type.

When using TLS you can choose to encrypt:

• Traffic between your clients and cluster nodes
• Fabric and/or heartbeat traffic between cluster nodes
• Cross-datacenter (XDR) traffic

Select the certificate match type:

• Cluster-name Match
• Wildcard Hostname Match
• Match with Common Server Certificate, or
• Hostname Match with Individual Server Certificate

Once the TLS deployment artichecture is determined, generate the appropriate certificates and follow the upgrade scenario below.

caution

This page covers Aerospike EE versions 3.15 and above. Configuration syntax for TLS is different for versions 3.11 to 3.14. Refer to the TLS Guide pre 3.15 and the Network Configuration/TLS/Pre 3.15 page for details for the specific versions.

caution

Dynamically rotating ECDSA private keys and certificates as well as password-protected private keys is not supported in Aerospike server versions prior to 4.7.0.5, 4.6.0.8, 4.5.3.10, 4.5.2.10, 4.5.1.15, 4.5.0.19. More details in the FAQ on TLS.

Configuring TLS

Refer to Configuration Reference for details regarding required and optional parameters.

Legend

• (*) required parameters for TLS
• (**) required parameters for standard authentication or mutual authentication
• (+) optional parameters for mutual authentication

network {
  tls tlsname1 {                    # (*) This is the TLS name to be referred to in subsequent network
                                    # sub-stanzas that woudld require TLS. Will be populated in the
                                    # tls-name wire field for incoming connections. Could be set to
                                    # <cluster-name> or <hostname> or user-defined.
                                    # Set to `tlsname1` in this example.  
    cert-file path-to-file          # (**) Required for standard authentication or mutual authentication
    key-file path-to-file           # (**) Required for standard authentication or mutual authentication
    ca-file path-to-file            # (+) For mutual authentication or for XDR.
                                    # Only one of ca-file / ca-path required.
    ca-path directory-path          # (+) For mutual authentication or for XDR.
                                    # Only one of ca-file / ca-path required.
    protocols TLSv1.2               # Protocol version to include
                                    # note - In version 4.6 the default protocols configuration parameter
                                    # was changed from “-all,+TLSv1.2” to “TLSv1.2”.
    cipher-suite ALL:!COMPLEMENTOFDEFAULT:!eNULL
                                    # Ciphers to includes.
    cert-blacklist path-to-file     # Blacklist including serial number of certs to revoke
  }

  tls tlsname2 {
    ca-file /tls/ca-cert2.pem
    cert-file /tls/rem-chain2.pem
    key-file /tls/rem-key2.pem
    ...
  }

  tls tlsname3.source.domain {
    ...
  }

  tls <cluster-name> {
    ...
  }

  ...

  service {
    port 3000
    address 192.168.80.100          # Bind address for non TLS traffic.
                                    # Default to `tls-address` if not specified.

    tls-port 4333                   # (*) Enables tls
    tls-address 192.168.90.150      # Bind address for TLS.
                                    # Will default to `address` if not specified.
    tls-authenticate-client false|any|user-defined
                                    # If not specified, default is any, which is mutual authentication.
                                    # Multiple tls-authenticate-client entries can be defined in order
                                    # to accept multiple subject names. At typical example would be to
                                    # use different subject names between local clients and XDR clients
                                    # (when specifying the subject name of course).
    tls-name <cluster-name>|<hostname>|user-defined  
                                    # (*) Which TLS config to use (from tls sub-stanza).
                                    # This is also the name that would be populated in the tls-name
                                    # wire field. (For example, tlsname1 to refer to the first
                                    # tls sub-stanza described above).
  }

  heartbeat {
    tls-port 3012
    tls-name tlsname2
    tls-mesh-seed-address-port 192.168.90.151 3012
    ...
  }

  fabric {
    tls-port 3011
    tls-name tlsname2
    ...
  }

  ...
}

...

info

The TLS name (specified as part of the tls sub-stanza and referred to within the service, fabric and heartbeat sub-stanzas as well as XDR stanzas) has the ability to be substituted. For example:

• tls / tls-name <cluster-name> means the server will use what's defined in cluster-name as the tls-name attribute.
  
• tls / tls-name <hostname> means the server will use what's returned by the underlying OS's "hostname" system call as the tls-name attribute.
  
  

info

ca-path: it may be required to run a "c_rehash" to generate symlinks based on hash values.

Configuring XDR with TLS

For the details on configuring XDR with TLS, see Securing with TLS.

XDR configuration stanza for pre-5.0 servers

The following xdr stanza applies only to versions of Aerospike Server prior to 5.0.

xdr {
  dc DC1 {
    node-address-port 10.10.1.1 3010 tlsname2
                                    # Remote nodes IP address along with TLS name a cluster node expects
                                    # the remote DC to present on XDR connections.
    ...
    tls-name tlsname3.source.domain # The remote cluster should have a tls-authenticate-client directive
                                    # specifying this TLS name, or `false`, or `any`.
    ...
  }

  dc DC2 {
    ...                           # Can have its own set of tls configs
  }

  ...
}

Blacklisting certificates

Certificates that have been compromised can be blacklisted, preventing malicious applications from connecting to the server.

Configuring the certificates blacklist

The serial number assumes hex representation, with no non-hex characters such as 'x' or ":" etc.

To add a certificate's information to the blacklist, you must have the certificate available to extract the properly formatted serial number:

cat chain.pem | openssl x509 -noout -serial -issuer | \
awk 'match($0,/[^=]*=(.*)/, a) { printf "%s", a[1] } END {printf "\n"}'

Refreshing the certificates blacklist

The certificates blacklist is automatically refresh when using Aerospike EE version 3.15 and above. For versions 3.11 to 3.14, use the following command to refresh the list:

asinfo -v 'set-config:context=xdr;dc=REMOTE;tls-cert-blacklist=.../path/blacklist.txt'

Upgrading to TLS

High level overview for upgrades.

Upgrade client to cluster connections to TLS

For an existing deployment and to upgrade to TLS, follow these steps:

1. Upgrade Server

• Rolling Upgrade each node in the cluster to a software version that is TLS aware (3.11 and above), while adding in new TLS enabled configurations. Do not shutdown clear ports.
• Repeat for all nodes in the cluster.
• During this time, client will continue to communicate with cluster via clear transport.

2. Upgrade Client

• Once every node is TLS enabled, start upgrading client to a software version that is TLS aware, while adding in new TLS enabled client configurations.
• During this time, old clients will continue to communicate with cluster via clear transport, while new clients will communicate with cluster via TLS.
• Repeat for all application nodes.

3. Shut down clear (non tls) ports (optional)

• If no clients should be allowed to reach the server, an additional Rolling Restart is required to remove and shutdown the clear port.

Upgrade from standard-authentication to mutual-authentication

For an existing deployment already running standard-authentication to be upgraded to mutual-authentication, follow these steps:

1. Upgrade Client

• Add the client certificate/keys.
• Rolling restart all clients so they are aware of the new certificates when asked.

2. Update Server

• Change server configuration to include ca-file or ca-path
• Update tls-authenticate-client to any or a user defined string representing a tls name to be validated (if validation is required).
• Rolling restart each of the node.
• As each node joins the cluster back, the newly established TLS connections will require the clients to present their certificates.

Upgrade intra node connections to TLS

Rolling upgrades from non-TLS to TLS for intra node connections is supported. The upgrade happens in two rounds:

1. Enable TLS for heartbeat and fabric across all nodes in a rolling fashion.

• Keep non-TLS ports enabled while doing so.
• This first step results in a full cluster supporting both, non-TLS as well as TLS.

info

When upgrading from a version prior to 3.13.0.10 or 3.14.1.9 (which do not support intra node TLS), first upgrade all nodes to version 3.15 and then enable TLS for heartbeat and fabric in a rolling fashion. Nodes running an older version
will try connecting to the TLS port first and will fail causing the cluster to not properly form. This will cause warnings to be printed in the log file for each fabric connection attempt. Warning message will look as such:
WARNING (tls): (tls_ee.c:277) SSL_accept failed: error:1408F10B:SSL routines:
SSL3_GET_RECORD:wrong version number
WARNING (fabric): (fabric.c:1400) fabric TLS server handshake with 10.10.1.1:50001 failed Versions 3.13.0.10 and 3.14.1.9 do support such upgrade due to the following improvement:
[AER-5826] - (CLUSTERING) Support rolling upgrades to later versions in which fabric TLS is enabled.

note

Under normal circumstances, such cluster configured with both non-TLS and TLS fabric/heartbeat will default to TLS. However, downgrade attacks are possible, where an attacker prevents the cluster from establishing TLS connections forcing the cluster to fall back to non-TLS.

2. Disable non-TLS for heartbeat and fabric across all nodes in a rolling fashion.

• At this point, the cluster only supports TLS connection for its node to node communication. This prevents downgrade attacks.

Upgrade XDR connections to TLS

For an existing deployment with XDR which will be upgraded to TLS, follow these steps

1. Upgrade Destination Clusters

• For each one of the destination cluster, Follow the procedure in "Upgrade client to cluster connections to TLS" to enable TLS ports.

2. Upgrade Source Cluster

• Then for each Source Cluster node, upgrade XDR's DC configuration to use the newly configured TLS port.

3. Shutdown clear ports (optional)

• An additional Rolling Restart is required to shutdown destination cluster nodes's clear port (port 3000).

Examples

Using cluster-name standard authentication

Server Configuration

service {
   ...
   cluster-name as-cluster-west
}

network {

    tls <cluster-name> {
        cert-file   /home/citrusleaf/x509_certificates/chainless_cluster_chain.pem
        key-file    /home/citrusleaf/x509_certificates/Chainless_cluster/key.pem    
    }

    service {
        tls-port 4333
        tls-name <cluster-name>
        tls-authenticate-client false
        ...
    }

    heartbeat {
        ...
        protocol v3
    }
}

C Client Benchmark Call

target/benchmarks -h "192.168.113.203:as-cluster-west:4333" --tlsEnable --tlsCaFile ~/x509_certificates/Platinum/cacert.pem

Java Client Benchmark Call

java -Djavax.net.ssl.trustStore=<TrustStorePath> -Djavax.net.ssl.trustStorePassword=<TrustStorePassword> -jar target/aerospike-benchmarks-*-jar-with-dependencies.jar -h "192.168.113.203:as-cluster-west:4333" -tlsEnable

Using hostname match with a common server certificate, mutual authentication (without name validation), and client-certificate blacklisting

Server Configuration

network {


    tls <hostname> {

        ca-file      /home/citrusleaf/x509_certificates/Platinum/cacert.pem  # Root CA's cert

        cert-file    /home/citrusleaf/x509_certificates/multi_chain.pem
        key-file     /home/citrusleaf/x509_certificates/MultiServer/key.pem

        cert-blacklist /root/certs_subject/blacklist.txt
    }

    service {
        tls-port 4333
        tls-name <hostname>
        tls-authenticate-client any
        ...
    }

}

C Client Benchmark Call

target/benchmarks -h "192.168.113.203:t3.t-cluster.aerospike.com:4333" --tlsEnable --tlsCaFile ~/x509_certificates/Platinum/cacert.pem --tlsChainFile ~/x509_certificates/client_chain.pem --tlsKeyFile ~/x509_certificates/Client/key.pem

Java Client Benchmark Call

java -Djavax.net.ssl.trustStore=<TrustStorePath> -Djavax.net.ssl.trustStorePassword=<TrustStorePassword> Djavax.net.ssl.keyStore=<KeyStorePath> -Djavax.net.ssl.keyStorePassword=<KeyStorePassword> -jar target/aerospike-benchmarks-*-jar-with-dependencies.jar -h "192.168.113.203:t-cluster.aerospike.com:4333" -tlsEnable

Using hostname match with individual server certificate, and server-certificate blacklisting

Server Configuration

network {

    tls <hostname> {

        cert-file    /home/citrusleaf/x509_certificates/server1_chain.pem  
                     # note - different for each server node.
        key-file     /home/citrusleaf/x509_certificates/Server1/key.pem    
                     # note - different for each server node.
    }

    service {
        tls-port      4333
        tls-name      <hostname>
        tls-authenticate-client false

    }
}

C Client Benchmark Call

target/benchmarks -h "192.168.113.203:t3.t-cluster.aerospike.com:4333" --tlsEnable --tlsCaFile ~/x509_certificates/Platinum/cacert.pem --tlsCertBlackList ~/bad_serials.txt

Java Client Benchmark Call

java -Djavax.net.ssl.trustStore=<TrustStorePath> -Djavax.net.ssl.trustStorePassword=<TrustStorePassword> -jar target/aerospike-benchmarks-*-jar-with-dependencies.jar -h "192.168.113.203:t-cluster.aerospike.com:4333" -tlsEnable -tlsRevoke 0xfaa7f3e06e288466c5420e0c7ccd4c45

Generating Self-Signed TLS Certificates

Developers may want to generate self-signed TLS certificates for their own development environment. The following gives an example for doing this.

Prepare the certificate directories

Prepare certificate directories for input, local, output, and follow this directory structure:

mkdir -p rootca 2>/dev/null
rm -rf rootca/* 2>/dev/null
mkdir -p rootca/input
mkdir -p rootca/local
mkdir -p rootca/output
cd rootca

Configure OpenSSL

cat <<'EOF' > openssl.conf
HOME                    = .
oid_section             = new_oids

[ new_oids ]
tsa_policy1 = 1.2.3.4.1
tsa_policy2 = 1.2.3.4.5.6
tsa_policy3 = 1.2.3.4.5.7

[ ca ]
default_ca      = CA_default            # The default ca section

[ CA_default ]
dir             = rootca                # Where everything is kept
certs           = $dir/certs            # Where the issued certs are kept
crl_dir         = $dir/crl              # Where the issued crl are kept
database        = $dir/index.txt        # database index file
                                        # several certs with same subject
new_certs_dir   = $dir/newcerts         # default place for new certs

certificate     = $dir/cacert.pem       # The CA certificate
serial          = $dir/serial           # The current serial number
crlnumber       = $dir/crlnumber        # the current crl number
                                        # must be commented out to leave a V1 CRL
crl             = $dir/crl.pem          # The current CRL
private_key     = $dir/private/cakey.pem# The private key
x509_extensions = usr_cert              # The extensions to add to the cert
name_opt        = ca_default            # Subject Name options
cert_opt        = ca_default            # Certificate field options
default_days    = 365                   # how long to certify for
default_crl_days= 30                    # how long before next CRL
default_md      = default               # use public key default MD
preserve        = no                    # keep passed DN ordering
policy          = policy_match

[ policy_match ]
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ policy_anything ]
countryName             = (2 letter code, for example, US)
stateOrProvinceName     = State or Province Name (full name)
localityName            = Locality Name (such as, city)
organizationName        = Organization Name (such as, company)
organizationalUnitName  = Organizational Unit Name (such as, section or department)
commonName              = Common Name (such as, server FQDN or YOUR name)
emailAddress            = joedo@joedocompany.com

[ req ]
default_bits            = 2048
default_keyfile         = privkey.pem
distinguished_name      = req_distinguished_name
attributes              = req_attributes
x509_extensions = v3_ca # The extensions to add to the self signed cert
string_mask = utf8only

[ req_distinguished_name ]
countryName                     = Country Name (2 letter code, for example, US)
countryName_default             = US
countryName_min                 = 2
countryName_max                 = 2
stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = Some-State
localityName                    = Locality Name (such as, city)
0.organizationName              = Organization Name (such as, company)
0.organizationName_default      = Your company name
organizationalUnitName          = Organizational Unit Name (such as, section)
commonName                      = Common Name (such as, server FQDN or YOUR name)
commonName_max                  = 64
emailAddress                    = joedo@joedocompany.com
emailAddress_max                = 64

[ req_attributes ]
challengePassword               = A challenge password
challengePassword_min           = 4
challengePassword_max           = 20
unstructuredName                = An optional company name

[ usr_cert ]
basicConstraints=CA:FALSE
nsComment                       = "OpenSSL Generated Certificate"
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer


[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment

[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = critical,CA:true

[ crl_ext ]
authorityKeyIdentifier=keyid:always

[ proxy_cert_ext ]
basicConstraints=CA:FALSE
nsComment                       = "OpenSSL Generated Certificate"
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
proxyCertInfo=critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

[ tsa ]
default_tsa = tsa_config1       # the default TSA section

[ tsa_config1 ]
dir             = ./demoCA              # TSA root directory
serial          = $dir/tsaserial        # The current serial number (mandatory)
crypto_device   = builtin               # OpenSSL engine to use for signing
signer_cert     = $dir/tsacert.pem      # The TSA signing certificate
                                        # (optional)
certs           = $dir/cacert.pem       # Certificate chain to include in reply
                                        # (optional)
signer_key      = $dir/private/tsakey.pem # The TSA private key (optional)
signer_digest  = sha256                 # Signing digest to use. (Optional)
default_policy  = tsa_policy1           # Policy if request did not specify it
                                        # (optional)
other_policies  = tsa_policy2, tsa_policy3      # acceptable policies (optional)
digests     = sha1, sha256, sha384, sha512  # Acceptable message digests (mandatory)
accuracy        = secs:1, millisecs:500, microsecs:100  # (optional)
clock_precision_digits  = 0     # number of digits after dot. (optional)
ordering                = yes   # Is ordering defined for timestamps?
                                # (optional, default: no)
tsa_name                = yes   # Must the TSA name be included in the reply?
                                # (optional, default: no)
ess_cert_id_chain       = no    # Must the ESS cert id chain be included?
                                # (optional, default: no)
ess_cert_id_alg         = sha1  # algorithm to compute certificate
                                # identifier (optional, default: sha1)
EOF

Generate the Certificate Authority (CA) root directory

openssl genrsa -out local/rootCA.key 2048
openssl req -x509 -new -nodes -key local/rootCA.key -sha256 -days 3650 -out local/rootCA.pem -subj "/C=UK/ST=London/L=London/O=fed/OU=Support/CN=rootca.fed"

Generate TLS certificate requests

The following example generates a TLS certificate for the server, and a certificate for the admin user. This pair of certs can used for Mutual TLS (mTLS) with the clients, as well as PKI authentication for the users.

The tls-name of the TLS context used in the service stanza must also be set as Common Name (CN) of the server's certificate. For example CN=server.

For PKI auth the CN of the user's certificate must match their username. For example CN=admin.

openssl req -new -nodes -config openssl.conf -extensions v3_req -out input/server.req -keyout output/server.key -subj "/C=UK/ST=London/L=London/O=fed/OU=Server/CN=server"
openssl req -new -nodes -config openssl.conf -extensions v3_req -out input/admin.req -keyout output/admin.key -subj "/C=UK/ST=London/L=London/O=fed/OU=Admin/CN=admin"

Example: Generate a user certificate

For each user, create a user certificate

export USERNAME=app
openssl req -new -nodes -config openssl.conf -extensions v3_req -out input/${USERNAME}.req -keyout output/${USERNAME}.key -subj "/C=UK/ST=London/L=London/O=fed/OU=Admin/CN=${USERNAME}"
openssl x509 -req -extfile openssl.conf -in input/${USERNAME}.req -CA local/rootCA.pem   -CAkey local/rootCA.key  -extensions v3_req -days 3649 -outform PEM -out output/${USERNAME}.pem -set_serial 310
openssl verify -verbose -CAfile local/rootCA.pem output/${USERNAME}.pem

Sign the certificates

openssl x509 -req -extfile openssl.conf -in input/server.req -CA local/rootCA.pem   -CAkey local/rootCA.key  -extensions v3_req -days 3649 -outform PEM -out output/server.pem -set_serial 110
openssl x509 -req -extfile openssl.conf -in input/admin.req -CA local/rootCA.pem   -CAkey local/rootCA.key  -extensions v3_req -days 3649 -outform PEM -out output/admin.pem -set_serial 310

Verify the certificates

openssl verify -verbose -CAfile local/rootCA.pem output/server.pem
openssl verify -verbose -CAfile local/rootCA.pem output/admin.pem

TLS-related configuration parameters and references

List of the TLS related configuration parameters:

• To be used in the network's tls sub-stanza:

• ca-file

• ca-path

• cert-blacklist

• cert-file

• cipher-suite

• key-file

• key-file-password

• protocols

• To be used in the network's service, fabric, heartbeat sub-stanzas and xdr stanza:
• tls-name
• tls-port (service, fabric and heartbeat sub-stanzas only)
• tls-authenticate-client (service sub-stanza only)
• tls-address (service, fabric and heartbeat sub-stanzas only)
• tls-mesh-seed-address-port (heartbeat sub-stanza only)