Specifying the JMS Queues and Topics to Consume from

The queues and topics sections of the /etc/aerospike-jms-inbound/aerospike-jms-inbound.yml file configure the properties of the JMS queues and topics from which you want the JMS inbound connector to consume messages.

Example

Here is an example of a topics section that configures one topic. The name of the topic is employees. A queues section includes the same properties.

Version 2.0.0 and later

topics:
  employees:
    session-ratio: 1
    jms-transaction-size: 16
    jms-message-receive-timeout: 100
    aerospike-operation:
      type: write
      max-retries: 2
      total-timeout: 0
      record-exists-action: update
      send-key: false
    mapping:
      bins:
        all-value-fields: false
        type: multi-bins
        map:
          binOne:
            source: value-field
            field-name: field-one
      key-field:
        source: value-field
        field-name: employeeID
      namespace:
        mode: static
        value: east
      set:
        mode: dynamic
        source: value-field
        field-name: subLocation
      ttl:
        mode: dynamic
        source: value-field
        field-name: ttl
    parsing: json

Version 1.1.0 and earlier

topics:
  employees:
    session-ratio: 1
    jms-transaction-size: 16
    jms-message-receive-timeout: 100
    aerospike-operation:
      type: write
      max-retries: 2
      total-timeout: 0
      record-exists-action: update
      send-key: false
    mapping:
      all-fields: false
      bins:
        field-one: binOne
      key-field: employeeID
      namespace:
        mode: static
        value: east
      set:
        mode: dynamic
        field-name: subLocation
      ttl:
        mode: dynamic
        field-name: ttl
    parsing: json

This table shows how, using this example configuration, a JMS inbound connector maps JSON messages from the JMS message broker to Aerospike records:

JSON	Aerospike Namespace	Aerospike Set	Aerospike Key	Aerospike Bin	TTL
{"field-one":"one","subLocation":"BNG", "employeeID": E123, ttl: "100H"}	east	BNG	E123	Bin("binOne", one)	100H
{"field-one":["one"],"subLocation":"MYS", "employeeID": E456, ttl: "10M"}	east	MYS	E456	Bin("binOne", List("one"))	10M
{"field-one":{"one": "one"},"subLocation":"DLH", "employeeID": E789, ttl: "1D"}	east	DLH	E789	Bin("binOne", Map("one" to "one"))	1D
{"field-one":"one","subLocation":"MUM", "employeeID": E987, ttl: "100"}	east	MUM	E987	Bin("binOne", one)	100

Procedure

Listing queues and topics to configure

The configuration file must include at least one of these sections:

Property	Required	Description
queues	no	A list of the queues to consume messages from.
topics	no	A list of the topics to consume messages from.

The first level in a topics or queues section is the name of a topic or queue. For example, suppose you want to configure three topics: employees, offices, and buildings. Your topics section would, in outline, look like this:

topics:
  employees:
    ...
  offices:
    ...
  buildings:
    ...

Configuring queues and topics

For each topic or a queue that you name in their corresponding section, you can set the following properties:

Version 2.0.0 and later

Option	Required	Default	Description
session-ratio	no	1	The ratio of total JMS sessions to use for this source. See "Distributing JMS sessions across queues and topics" for details.
jms-transaction-size	no	16	The number of JMS messages to consume in a single JMS transaction.
jms-message-receive-timeout	no	100	The number of milliseconds to wait for a JMS message before timing out. A value of zero signifies that there is no timeout.
aerospike-operation	no	write	Specifies how to apply writes and deletes to your Aerospike database. See "Configuring Aerospike operations to perform on records" for details.
mapping	in some cases		The mapping of the JMS message to an Aerospike record. It is required if you do not implement message-transformer for the topic/queue or implement a message-transformer with InboundMessage<K, M>. See "Configuring how to map JMS messages to Aerospike records" for details.
parsing	yes		The JMS message parsing config. See "Specifying the format of inbound messages" for details.
message-transformer	no		The configuration parameters for a message transformer. See "Aerospike JMS Inbound Message Transformer" for details.

Version 1.1.0 and earlier

Option	Required	Default	Description
session-ratio	no	1	The ratio of total JMS sessions to use for this destination. See "Distributing JMS sessions across queues and topics" for details.
jms-transaction-size	no	16	The number of JMS messages to consume in a single JMS transaction.
jms-message-receive-timeout	no	100	The number of milliseconds to wait for a JMS message before timing out. A value of zero signifies that there is no timeout.
aerospike-operation	no	write	The Aerospike operation. See "Configuring Aerospike operations to perform on records" for details.
mapping	yes		The JMS message to Aerospike record mapping. See "Configuring how to map JMS messages to Aerospike records" for details.
parsing	yes		The JMS message parsing config. See "Specifying the format of inbound messages" for details.

Distributing JMS sessions across queues and topics

You can set this property for each topic or queue that you specify in the queues and topics sections. However, to set it for an individual topic or queue, you must first know the values for all of the queues and topics that you plan to configure in /etc/aerospike-jms-inbound/aerospike-jms-inbound.yml.

To calculate the values of this property for all the queues and topics that you want to use:

1. Taking the total number of JMS sessions as a whole (1), represent the whole as the fraction n/n, where n represents the number of equal divisions of the whole that you plan to distribute among your queues and topics.

   The value of n must be greater than or equal to the number of your queues and topics, and less than or equal to the number of JMS sessions. The number of JMS sessions is the product of two values that you set in your JMS message broker: max-connections and max-sessions-per-connection. For example, if you set max-connections to 4 and max-sessions-per-connection to 2, then the number of JMS sessions is 8. Therefore, if the number of your topics or queues is 4, n can be greater than or equal to 4, and less than or equal to 8.

   For example, suppose that you plan to distribute sessions across two topics and one queue. You smallest n you could use would be 3 because you have three sources. The fractional representation of the whole would be 3/3. (You don't have to use a value for n that equals the number of your queues and topics, however.)

2. Split this fraction across your queues and topics.

   For example, you have two topics (T1 and T2) and one queue (Q1) and the fraction is 3/3. Each source must be allocated some portion of the sessions. So, you give each source 1/3.

3. Take the numerator of each fraction.

4. Specify the numerators as the values of session-ratio for your queues and topics.

To continue with the example from step 2, this table shows the value of session-ratio for each source:

Source	Type	session-ratio
T1	Topic	1
T2	Topic	1
Q1	Queue	1

You might choose to distribute the sessions unequally across your queues and topics. For example, if you have topics T1 and T2, and queue Q1, you could use a value of 5 for n, making 5/5, and then set these values for session-ratio:

Source	Type	session-ratio
T1	Topic	2
T2	Topic	2
Q1	Queue	1

In this case, topics T1 and T2 would each be allocated two-fifths of the sessions, while topic T3 would be allocated one-fifth.

If the number of JMS sessions is not evenly divisible by n, then the remaining sessions are distributed randomly to your queues and topics at startup.

info

If you set session-ratio for one of the queues and topics that you plan to use, then you must set it for all of those queues and topics.

Configure Aerospike operations

Version 2.0.0 and later

Use the aerospike-operation option to specify how to apply writes and deletes to your Aerospike database.

Option	Required	Default	Description
type	no	write	Specifies the type of operation to perform in Aerospike. Valid values are: write, delete.
max-retries	no	2	Specifies the maximum number of times to retry a transaction.
total-timeout	no	0	Specifies the number of milliseconds to wait before a transaction time. A value of 0 indicates that there is no time limit.
record-exists-action	no	update	Specifies the action to take if a record already exists and type is set to write. Valid values are update, update-only, replace, replace-only, create-only.
send-key	no	false	Specifies to send a user-defined key, in addition to a hash digest. Applies only when the value for the property type is write.
durable-delete	no	false	Specifies to leave a tombstone for the record. Applies only when the value for the property type is delete.
ignore-error-codes	no	empty set	Consider a record as successfully processed if it throws AerospikeException with the given ResultCode.

Examples

# An aerospike write operation with all relevant options.
aerospike-operation:
  type: write
  max-retries: 3
  total-timeout: 0
  record-exists-action: update-only
  send-key: true
  ignore-error-codes:
    - 22
    - 13

# An aerospike delete operation with all relevant options.
aerospike-operation:
  type: delete
  max-retries: 3
  total-timeout: 0
  durable-delete: true
  ignore-error-codes:
    - 22
    - 13

Version 1.1.0 and earlier

Use the aerospike-operation option to specify how to apply writes and deletes to your Aerospike database.

Option	Required	Default	Description
type	no	write	Specifies the type of operation to perform in Aerospike. Valid values are: write, delete.
max-retries	no	2	Specifies the maximum number of times to retry a transaction.
total-timeout	no	0	Specifies the number of milliseconds to wait before a transaction times. A value of 0 indicates that there is no time limit.
record-exists-action	no	update	Specifies the action to take if a record already exists and type is set to write. Valid values are update, update-only, replace, replace-only, create-only.
send-key	no	false	Specifies to send a user-defined key, in addition to a hash digest. Applies only when the value for the property type is write.
durable-delete	no	false	Specifies to leave a tombstone for the record. Applies only when the value for the property type is delete.

Map JMS messages to Aerospike records

Use the mapping option to specify how to map JMS messages in a topic or queue to Aerospike records.

Version 2.0.0 and later

Option	Required	Description
namespace	yes	Specifies whether the name of the namespace is set in the configuration file or in messages. See Specify namespace, set, and ttl for details.
set	no	Specifies whether the name of the set is set in the configuration file or in messages. See Specify namespace, set, and ttl for details.
ttl	no	Specifies whether the time to live is set in the configuration file or in messages. See Specify namespace, set, and ttl for details.
bins	yes	Mapping of incoming message to Aerospike bins. See Mapping fields in messages to bins in Aerospike records for details.
key-field	no*	The Field selection configuration value for the Aerospike record key. See Key-field and Digest config for details.
digest	no*	The Field selection configuration value for the Aerospike record digest. The field has to be a valid Aerospike RIPEMD-160 digest. The digest field should be plain bytes or a Base64 encoded string. See Key-field and Digest config for details.

caution

Either key-field or digest has to be specified in a mapping.

Key-field and Digest config

Use these two options to generate key-field or digest.

Option	Required	Description
source	yes	The source of the value.
field-name	no	The name of the field to extract the value from. Only applies when source is value-field.

Possible values for the source option

Value	Description
value	Use the whole of the JMS message value as the value.
value-field	Extract a value from the JMS message value.

Version 1.1.0 and earlier

Option	Required	Default	Description
all-fields	yes	false	Indicates whether all fields from the JMS message should be converted to Aerospike record bins.
bins	no		Mapping of JMS message field names to Aerospike bin names.
key-field	yes		The JMS message field to use as the Aerospike record key.
namespace	yes		Specifies whether the name of the namespace is set in the configuration file or in messages. See "Specifying where to find values for namespace, set, and ttl" for details.
set	no		Specifies whether the name of the set is set in the configuration file or in messages. See "Specifying where to find values for namespace, set, and ttl" for details.
ttl	no		Specifies whether the time to live is set in the configuration file or in messages. See "Specifying where to find values for namespace, set, and ttl" for details.

Specify namespace, set, and ttl

Configuration to obtain values for namespace, set, and ttl requires definitions for the following properties:

Option	Required	Description
mode	yes	How the value of a given field should be obtained. The possible options are static and dynamic.

Static: Specify the values in the configuration file

To set a static or unchanging name for the namespace, name for the set, or value for the time to live for an Aerospike record, you can use this option:

Option	Required	Description
value	yes	The static name or value.

This example sets the name of a namespace to east for all records that an inbound connector creates from the messages that it receives:

mapping:
  ...
  namespace:
    mode: static
    value: east
  ...

This example sets the time to live for all records that an inbound connector creates from the messages that it receives:

mapping:
  ...
  ttl:
    mode: static
    value: -1
  ...

Choose from among the following values when setting the time to live for Aerospike records:

Value	Description
-2	Specifies not to reset the time to live when a record is updated.
-1	Specifies that the time to live never expires.
0	Specifies to set the time to live to the default that is configured for the namespace in the Aerospike database.
Integer greater than 0	Specifies by default the time to live in seconds. However, you can use one of the following suffixes to specify a different unit of time: M for minutes, H for hours, D for days.

Dynamic: Specify message fields from which to obtain values

Select a value dynamically from the JMS message.

Version 2.0.0 and later

Option	Required	Description
source	yes	The source of the value.
field-name	no	The name of the field to extract the value from. Only applies when source is value-field.

Possible values for the source option

Value	Description
value	Use the whole of the JMS message value as the value.
value-field	Extract a value from the JMS message value.

Example

mapping:
  ...
  namespace:
    mode: dynamic
    source: value-field
    field-name: category
  set:
    mode: dynamic
    source: value-field
    field-name: brand
  ttl:
    mode: dynamic
    source: value-field
    field-name: validForSec
  ...

Mapping fields in messages to bins in Aerospike records

For topics and queues that you list in their corresponding section, you must map fields from messages to bins in Aerospike records.

Option	Required	Description
type	yes	The number of bins that exist in Aerospike records that correspond to messages in the topic/queue. Valid values are single-bin, multi-bins.

Single bin

Use these two options to specify the source of the value to write to single-bin Aerospike records.

Option	Required	Description
source	yes	The source of the value.
field-name	no	The name of the field to extract the value from. Only applies when source is value-field.

Possible values for the source option

Value	Description
value	Use the whole of the JMS message value as the value.
value-field	Extract a value from the JMS message value.

Examples

# A single bin Aerospike record with the bin value extracted from the JMS
# record field "name".
bins:
  type: single-bin
  source: value-field
  field-name: name

# A single bin Aerospike record with the bin value equal to the whole of the
# JMS message value.
bins:
  type: single-bin
  source: value

Multiple bins

Use these options to specify the source of the values to write to Aerospike records that are comprised of multiple bins.

Option	Required	Default	Description
all-value-fields	no	false	Indicates if all fields from the JMS message value should be converted to Aerospike record bins.
map	no		A map of the Aerospike record bin name to the source of the bin value. Use this option only when all-value-fields is false.
source	yes		The source of the value. See the table below for the possible values. Used with the map option
field-name	no		The name of the field to extract the value from. Only applies when source is value-field. Used with the map option.

Possible values for the source option

Value	Description
value	Use the whole of the JMS message value as the value.
value-field	Extract a value from the JMS message value.

Examples

# Write all the JMS message value fields as Aerospike record bins.
bins:
  type: multi-bins
  all-value-fields: true

# Extract the values of Aerospike bins "name" and "city" from the JMS fields `firstName` and `residence`.
bins:
  type: multi-bins
  map:
    name:
      source: value-field
      field-name: firstName
    city:
      source: value-field
      field-name: residence

# Use the whole of the JMS message value as the Aerospike bin "metadata".
bins:
  type: multi-bins
  map:
    metadata:
      source: value

Version 1.1.0 and earlier

Option	Required	Expected Value	Description
mode	yes	dynamic	Specifies that the name or value comes from a field.
field-name	yes		The name of field in the message to extract the value from.

Specifying the format of inbound messages

The message parsing config. Valid values -

Value	Description
json	Specifies that the message is a JMS TextMessage in JSON format.
msgpack	Specifies that the message is a JMS BytesMessage in MsgPack format.
map	Specifies that the message is a JMS MapMessage.
java-serialized	Specifies that the message is a JMS ObjectMessage in Java-serialized object format. The serialized object should be in the class path. The JAR containing the class file and its dependencies should be put in `/opt/aerospike-jms-inbound/usr-lib.