Skip to main content

Qualifying Cloud Instances

The Aerospike Cloud Qualifying Process evaluates cloud providers' instance and device performance by simulating a steady read and write workload, and measuring response time. These are tests that should be performed to get an idea of the level of performance you can expect from cloud instances.

If your instance is not one that has been tested by Aerospike, you can test it yourself by downloading the open sourced Aerospike Cloud Qualification package to test your instances, or by utilizing YCSB directly.

Qualified Instance Types

Cloud instance types will give you a certain amount of resources (CPU, Memory, Disk and Networking), but it can be hard to translate that into expected performance. The Cloud Qualification Process will take these instances and qualify it capable of a certain operational level, as an overall metric. It does not focus on individual resource components or optimizations.

info

This page will hopefully provide a starting point into what you can expect from each instance and how the instance class/family is more suitable toward different object sizes and use cases.

The Cloud Qualification Process is done using Aerospike's fork of the Yahoo! Cloud Serving Benchmark.

The testing methodology is as follows:

WorkloadAccess PatternRecord Count*Record Size*Cluster SizeYCSB Duration
50/50 R/UUniform60% of memory50% of disk33 * Record Count

* Based on a 2 node cluster

The testing client is a single instance of of a top-end system focused on compute power and network.

The entire test is repeated at least 3 times for verification

caution

This workload is chosen as a worst-case scenario for Aerospike, as a uniform access pattern will result in all blocks entering defragmentation at nearly the same time.

Aerospike is configured to utilize local SSD storage (when available), but is otherwise left as default, as are the instances that Aerospike is deployed on. You may see improved performance from additional tuning. You may also choose to alter these numbers and percentages in your own run of YCSB.

An instance is considered passing at an operational level (eg: 30,000 ops/s) if the average read latency does not exceed 1 millisecond from the client. 95 and 99 percentiles, as well as object size and count, are provided as reference.

Larger operational levels are better.

Your results may vary from these published numbers due to differences in server, topology, and even variation between systems of the same type.

The table below shows the results from the Aerospike Cloud Qualification Process on some popular instance types.

Qualified Instances

AWS

Instance TypeClient Transactions/s+95th Percentile (ms)99th Percentile (ms)Object Size (bytes)Object CountDisk SizeCost++
i3.xlarge68,0003.35.41424240,000,000950GB$0.312/hr
c3.4xlarge+++66,0004.111554240,000,0002x160GB$0.84/hr
i2.xlarge36,0003.55.81424240,000,000800GB$0.424/hr
+ As measured from the client, 50/50 read/update averaging under 1ms.
++ Price in us-west-2, at time of writing.
+++ All available ephemeral drives were utilized concurrently.
note

Some ephemeral drives require initialization before reaching maximum performance. You can read more on Amazon's Documentation.

Initialization is not done for these tests. The loading phase, as well as having multiple verification runs, acts as an effective initialization process.

Your numbers may not neccessarily match these exactly, due to variance within AWS's environment.

Azure

Instance TypeClient Transactions/s+95th Percentile (ms)99th Percentile (ms)Object Size (bytes)Object CountDisk SizeCost++
Standard_Gs131,0003.59.113+++240,000,00056GB$0.61/hr
Standard_L4s41,0001.25.81500240,000,000678GB$0.344/hr
+ As measured from the client, 50/50 read/update averaging under 1ms.
++ Price in West US, at time of writing.
+++ Low object size is due to fitting 240m objects into a mere 56GB disk. Larger object sizes can be used with the following optimzations:
  • shorten set name from 9 characters
  • Enable single-bin for the namespace. This will make 28 additional bytes available.
  • 3 additional bytes are available for Integer or Float types.
This results in up to an additional 39 bytes for data per record without impacting storage space (using a single character set name).


Your numbers may not neccessarily match these exactly, due to variance within Azure's environment.

GCP

Instance TypeClient Transactions/s+95th Percentile (ms)99th Percentile (ms)Object Size (bytes)Object CountDisk SizeCost++
n1_standard_882,0003.25.9768240,000,000375GB$0.492/hr+++
+ As measured from the client, 50/50 read/update averaging under 1ms.
++ Price in us-central1-a, at time of writing.
+++ Cost includes $0.112/hr for a single local ssd.
Your numbers may not neccessarily match these exactly, due to variance within Google's environment.

IBM Cloud

Instance TypeClient Transactions/s+95th Percentile (ms)99th Percentile (ms)Object Size (bytes)Object CountDisk SizeCost++
Xeon E3 1270v3
1Gig Networking
VM to Baremetal
75,0002.74.3500240,000,0001.2TB$415/mo ($0.576/hr)
Xeon E3 1270v3
1Gig Networking
Baremetal to Baremetal
130,0003.45.5500240,000,0001.2TB$415/mo ($0.576/hr)
Xeon E3 1270v3
1Gig Networking
Baremetal to Baremetal
30,0003.04.63000240,000,0001.2TB$415/mo ($0.576/hr)
2x Xeon E5 2620v4
10Gig Networking
Baremetal
222,0001.73.2500240,000,0001.2TB$1109/mo ($1.54/hr)
2x Xeon E5 2620v4
10Gig Networking
Baremetal +++
100,0002.63.43000240,000,0001.2TB$1109/mo ($1.54/hr)
+ As measured from the client, 50/50 read/update averaging under 1ms.
++ Price in SJC01/SJC04, at time of writing. System, RAM, Networking and Disk are all separate billable items, shown as a combined sum. Servers were on monthly provisioning, with hourly costs extrapolated using 30-day months.
+++ SSD Device overloaded and testing was terminated early. Testing proceeded at a faster pace than the device was capable of while still able to maintain latency profile. This caused defragmentation to fall behind and exhausted free blocks, therefore Aerospike Server entered into stop_writes. Read more about stop-writes and how it relates to

defragmentation.



Your numbers may not neccessarily match these exactly, due to variance within IBM's environment.

See also