Skip to main content
Loading

Managing Batch Operations

Batch-Index protocol

A batch is a series of requests that are sent together to the database server. A batch groups multiple operations into one unit and passes it in a single network trip to each database node. Refer to Batch Operations.

Tuning batches

Background

An incoming batch request from a client is assigned to a specific batch response thread if the status is not "full". A thread is declared "full" when batch-max-buffers-per-queue (255 default) are in use in a queue.

Each batch response thread has a queue of 128KiB buffers. Existing batch requests assigned to a response thread can allocate buffers beyond batch-max-buffers-per-queue, as needed. The “full” designation prevents a queue or response thread from assigning new incoming batch requests. If all response threads are at "full" status, new incoming batch requests are not accepted and an error is returned to the client.

When response threads need a new buffer, they take a buffer from a single pool of unused buffers. This pool is empty at startup and serves all the response threads. Response threads allocate the buffers, and then return them to the pool of unused buffers when done.

A destination client is associated with each 128KB buffer allowing the same response thread to serve multiple batch requests. Excess buffers are destroyed if the number of buffers in the unused pool exceeds batch-max-unused-buffers (default 256). Records that require more than 128KB are allocated a "huge" buffer which is destroyed after use and is not saved in the unused buffer pool.

The server provides the following configuration variables for batch performance tuning. Refer to Configuration Reference.

NameDefaultMaxDynamicDescription
batch-max-requests5000trueDetermines the maximum number of keys in a sub-batch sent to the cluster node. It prevents unexpectedly large batch requests from causing server instability due to excessive memory consumption. If the sub-batch size is exceeded the server returns error code 151 (AS_ERR_BATCH_MAX_REQUESTS)
batch-max-buffers-per-queue255trueMaximum number of 128KiB response buffers allowed in each batch queue before it is designated "full". Additional buffers beyond batch-max-buffers-per-queue can be allocated for accepted batch requests, if needed. If all batch queues are full, new batch requests are rejected with error code 152 (AS_ERR_BATCH_QUEUES_FULL).
batch-max-unused-buffers256trueMaximum number of 128KiB response buffers allowed in the unused buffer pool for reuse by any response thread. If the limit is reached, new buffers created by response threads at runtime are destroyed upon completion of the batch request. This limits the size of the unused buffer pool that serves all response threads.
batch-index-threads#cpu256trueNumber of batch index response worker threads. Each thread has its own queue. These threads only handle returning batch response buffers to the client using sockets. The maximum memory consumption can be computed as: batch-index-threads x batch-max-buffer-per-queue x 128KB. Tuning batch threads to 0 will disable batch functionality, rejecting batch commands with error code 150 (AS_ERR_BATCH_DISABLED)

The tools package 6.0.x or later is required to use asadm manage config commands. Otherwise, use the equivalent asinfo - set-config command.

Example: Increasing the number of idle batch response buffers:

asadm -e "enable; manage config service param batch-max-unused-buffers to 512"

Statistics

The server provides the following batch statistics variables. Refer to Metrics Reference.

NameDescription
batch_index_initiateNumber of batch requests received.
batch_index_queueNumber of batch requests and response buffers remaining on each batch queue. Format: <q1 requests> :<q1 buffers> ,<q2 requests> :<q2 buffers>,...
batch_index_completeNumber of completed batch requests.
batch_index_timeoutNumber of timed-out batch requests.
batch_index_errorNumber of batch requests rejected because of errors.
batch_index_unused_buffersNumber of available 128KB response buffers in the buffer pool.
batch_index_huge_buffersNumber temporary response buffers created that exceeded 128KB. Huge buffers are created when one of the records is retrieved that is greater than 128KB. Huge records do not benefit from batching and can result in excessive memory thrashing on the server.
batch_index_created_buffersNumber of 128KB response buffers created. Response buffers are created when there are no buffers left in the pool. If this number consistently increases and there is available memory, then batch-max-unused-buffers should be increased.
batch_index_destroyed_buffersNumber of 128KB response buffers destroyed. Response buffers are destroyed when there is no slot left to put the buffer back into the pool. The maximum response buffer pool size is batch-max-unused-buffers.
batch-indexBatch performance histogram.

Batch sub transactions

The server also provides the following batch sub transactions for read, writes, deletes, udfs, and (Lua) language. Refer to Metrics Reference.

NameDescription
batch_sub_delete_successNumber of batch delete sub transactions that were completed.
batch_sub_delete_errorNumber of batch delete sub transactions that failed with an error.
batch_sub_delete_timeoutNumber of batch delete sub transactions that timed out.
batch_sub_delete_not_foundNumber of batch delete sub transactions that were not found.
batch_sub_delete_filtered_outNumber of batch delete sub transactions that were filtered out. Transactions filtered out at the bin level by a filter expression.
batch_sub_lang_read_successNumber of batch sub language read transactions that were completed.
batch_sub_lang_write_successNumber of batch sub language write transactions that were completed.
batch_sub_lang_delete_successNumber of batch sub language delete transactions that were completed.
batch_sub_lang_errorNumber of batch language sub transactions that failed with an error.
batch_sub_udf_completeNumber of batch udf sub transactions that were completed.
batch_sub_udf_errorNumber of batch udf sub transactions that failed with an error.
batch_sub_udf_timeoutNumber of batch udf sub transactions that timed out.
batch_sub_udf_filtered_outNumber of batch udf sub transactions that were filtered out. Transactions filtered out at the bin level by a filter expression.
batch_sub_write_successNumber of batch write sub transactions that were completed.
batch_sub_write_errorNumber of batch write sub transactions that failed with an error.
batch_sub_write_timeoutNumber of batch write sub transactions that timed out.
batch_sub_write_filtered_outNumber of batch write sub transactions that were filtered out. Transactions filtered out at the bin level by a predicate expression.
batch_sub_proxy_completeNumber of proxied batch sub transactions that completed.
batch_sub_proxy_errorNumber of proxied batch sub transactions that failed with an error.
batch_sub_proxy_timeoutNumber of proxied batch sub transactions that timed out.
batch_sub_read_errorNumber of batch read sub transaction that failed with an error.
batch_sub_read_not_foundNumber of batch read sub transactions that resulted in not found.
batch_sub_read_successNumber of successful batch read sub transactions.
batch_sub_read_timeoutNumber of batch read sub transactions that timed out.
batch_sub_tsvc_errorNumber of batch read sub transactions that failed with an error in the transaction service, before attempting to handle the transaction. For example protocol errors or security permission mismatch.
batch_sub_tsvc_timeoutNumber of batch read sub transactions that timed out in the transaction service, before attempting to handle the transaction. For example, protocol errors or security permission mismatch.
retransmit_all_batch_sub_dup_resNumber of retransmits that occurred during batch sub transactions that were being duplicate resolved. Note this includes retransmits originating on the client as well as proxying nodes.
retransmit_batch_sub_dup_resNumber of retransmits that occurred during batch sub transactions that were being duplicate resolved. Replaced with retransmit_all_batch_sub_dup_res as of version 4.5.1.5.
early_tsvc_batch_sub_errorNumber of errors early in the transaction for batch sub transactions. For example, bad/unknown namespace name or security authentication errors.
from_proxy_batch_sub_delete_successNumber of records successfully deleted by batch sub delete transaction proxied from another node.
from_proxy_batch_sub_delete_errorNumber of records that were not deleted and failed with an error by batch sub delete transaction.
from_proxy_batch_sub_delete_timeoutNumber of records that were not deleted due to time out by batch sub delete transaction.
from_proxy_batch_sub_delete_not_foundNumber of records that were not deleted because that were not found by batch sub transaction.
from_proxy_batch_sub_delete_filtered_outNumber of records that were not deleted because they were filtered out by batch sub transaction. Transactions filtered out at the bin level by a filter expression.
from_proxy_sub_lang_read_successNumber of records that were completed by batch sub language read transactions.
from_proxy_sub_lang_write_successNumber of records that were completed by batch sub language write transactions.
from_proxy_sub_lang_delete_successNumber of records that succeeded by batch sub language delete transactions proxied from another node.
from_proxy_sub_lang_errorNumber of records that failed with an error by batch sub language transaction.
from_proxy_sub_udf_completeNumber of records that were completed by batch sub udf transaction.
from_proxy_sub_udf_errorNumber of records that failed with an error by batch sub udf transaction.
from_proxy_sub_udf_timeoutNumber of records of batch udf sub transactions proxied from another node that timed out, before attempting to handle this transaction.
from_proxy_sub_udf_filtered_outNumber of records of batch udf sub transactions proxied from another node that did not happen because the record was filtered out via a filter expression.
from_proxy_sub_write_successNumber of records successfully written by batch sub transactions proxied from another node.
from_proxy_sub_write_errorNumber of batch write sub transactions proxied from another node that failed with an error.
from_proxy_sub_write_timeoutNumber of batch write sub transactions proxied from another node that timed out.
from_proxy_sub_write_filtered_outNumber of batch write sub transactions proxied from another node that did not happen because the record was filtered out via a predicate expression.

Batch log file histograms

Periodically, the Aerospike server writes histograms to the log file. Refer to Latency Monitoring for latency histograms of batch and its sub transactions. In addition, review the Batch Transaction Analysis.

NameDescription
batch_sub_write_masterTime taken for writing all the copies of a record to the master . This only applies for strong consistency enabled namespaces.\n
batch_sub_udf_masterTime taken from partition reserved or after duplicate resolution to an actual master record applied.\n
batch_sub_repl_writeTime taken from the master record written to replica(s) written.\n