Queries, Long and Short
Overview
This section describes Aerospike's support for short and long queries on the primary index (PI queries, formerly known as scans), and secondary indexes (SI queries, formerly known as queries).
Unified query interface
Database 6.0 unified PI queries and SI queries in a single query subsystem. As a result, PI and SI queries share the following features:
- Queries are partitioned. By default, queries target all the partitions, allowing for large data sets to be processed in parallel. You can also target specific partitions, or a single partition with a query.
- Queries can be paginated. A query may return a large number of results (billions of records), which could overwhelm the client. The query can be paginated, with the size of the result, using the max records query policy. The client partition filter object keeps track of the query progress across multiple partitions, and can be used to continue the query each time.
- Queries are rebalancing-tolerant. Compatible clients in Database 6.0 and later ensure that queries can handle automatic partition migration when cluster sizes change.
Differences between Short and Long Queries
The following table shows the differences between short and long queries.
| Short Query | Long Query |
|---|---|
| Use the short query for a small number of records in the result, and when the query execution is highly concurrent | Use the long query for a large number of records in the result |
Short queries are not tracked. Useful when short-lived queries run frequently in order to avoid polluting the active_jobs queue | Long queries are tracked in active_jobs queue when they are active and in finished_jobs queue when they complete |
| Measured by query latency and record count histograms | Do not provide latency and record count histograms |
| Short queries cannot be aborted | Long queries can be aborted |
| Short query execution cannot be throttled | Long query execution can be throttled by setting an RPS (records per second) cap |
| Short queries have a default 1s transaction timeout (on the socket) | Long queries do not time out on the transaction length |
Developers can specify whether the query is long or short (long is the default) from the Query policy class of compatible clients.
A "long query" is a long-running query that retrieves a large number of records from the database. The amount of time it takes to execute a long query depends on factors such as complex filtering conditions, the size of the dataset, indexing strategies, and the IOPS (input/output operations per second) capacity of the cluster. The client reserves each partition it needs to query on the database cluster node that owns it. In an unstable cluster that has recently changed size and is rebalancing by migrating partitions, the client will rearrange the order it reserves partitions to ensure that it queries full partitions. Setting a long query with an adequate sleep between retries should ensure that the query is not negatively affected by migrations. However, if a long query in fact returns a small number of records, its expectedDuration query policy can be used to decide whether read consistency should be relaxed against AP mode namespaces (this is unnecessary when querying SC namespaces). This allows the client to reserve partially full migrating partitions if needed, otherwise the client fails with an error.
Short queries against AP namespaces always relax read consistency if they must reserve migrating partitions.
The Scan classes are deprecated starting with the Database 6.0-compatible clients, such as Java client 6.0.0.