Log Analyzer Commands

The asadm command has a built-in interactive shell for analysis of Aerospike server logs.

Starting the log analyzer

To start the log analyzer shell, run the following command:

asadm -l -f <asLogpath-1> <asLogpath-2> <asLogpath-3> ...

where <asLogpath*> are the space-separated paths of single log files or directories of log files.

Per-session lists of log files

In each session, the log analyzer keeps lists of log files for analysis:

• All list: the complete list of all log files added to the session.
• Selected list: those log files currently active for analysis.

In the log analyzer shell itself, you can add, select, or remove logs.

Summary of log analyzer commands

The log analyzer has the following commands. Some of them are similar to common UNIX shell commands.

Log analyzer command	Description
add	Add more log files to the All and Selected lists.
count	Display number of lines that match a specified pattern and time range.
diff	Display counts of a specified search pattern and the deltas of that pattern in consectuive log lines.
grep	Display lines that match a specified pattern and time range.
help	Display usage information about a specified command.
histogram	Calculate and display histogram measurements for a specific time range.
list	Display names of logs in the All and Selected lists.
pager	pager on: Sets paging size of output before pausing. Paging size is specified with the -p option. pager off: Turn off pausing at paging size.
remove	Remove logs from the All and Selected lists.
select	Select log files from the All list to put in the Selected lists.

Command help

The help command displays usage information about the specified command.

Log-analyzer> help grep
Displays all lines from server logs matched with input strings.
  Options:
    -s <string>  - Space seprated Strings to search in log files, MANDATORY - NO DEFAULT
                   Format -s 'string1' 'string2'... 'stringn'
.
.
.

About these examples

For details about command options in these examples, use help <command_name>.

Omission of lengthy output lines in some examples is indicated by three periods:

Line 1
.
.
.
Line n

add

The add command makes new logs from different nodes of the cluster available for analysis. It adds logs to both the All to the Selected lists.

Note: You can add only a single log file per node.

Arguments to add are the space-separated paths of a log or a directory of logs.

Note for Aerospike version 3.7.1 and earlier: the displayed name of a log file is the MD5 hash of the specified path.

In the following example, a directory of logs and a single log are added.

Log-analyzer> add /home/vagrant/shared/data/testlogs/ /var/log/aerospike/aerospike2.log
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/1.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/2.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/3.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/4.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/5.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/6.log.
6 server logs added for server analysis.
INFO: Added Log File /var/log/aerospike/aerospike2.log.
1 server log added for server analysis.

count

The count command displays the number of lines that match a specified pattern and time range.

You can specify any of the following:

• Multiple match patterns.
• Log selection.
• Ignore case.
• Invert match.
• Filter lines that are identical so only a single line is displayed, similar to the Unix uniq command.
• Time range.
• Use the-p option to set the paging number of lines to display, with default 10 lines.

Log-analyzer> count -s "reads" -n "5,6" -f "Jan 31 2017 07:00:00" -d 100 -t 20
~~~~~~~~~~~~~~~~~~~~~~~~~cluster (Page-1)~~~~~~~~~~~~~~~~~~~~~~~~
NODE                :   c1d635c2a270008   |   c81635c2a270008   |   
Jan 31 2017 07:00:00:   2                 |   2                 |   
Jan 31 2017 07:00:20:   2                 |   2                 |   
Jan 31 2017 07:00:40:   2                 |   2                 |   
Jan 31 2017 07:01:00:   2                 |   2                 |   
Jan 31 2017 07:01:20:   4                 |   4                 |   
total               :   12                |   12                |   

diff

The diff command displays the deltas of a specified pattern in consecutive values for search.

It supports following five patterns of search string S and values Vi :

• S V0
• S (V0)
• S (V0, V1, …, Vn)
• S (V0
• V0(V1) S

You can specify any of the following:

• Log selection.
• Ignore case.
• Output filter.
• Filter lines that are identical so only a single line is displayed, similar to the Unix uniq command.
• Time range.
• The-p option to set the paging number of lines to display, with default 10 lines.

Log-analyzer> diff -s "fds - proto" -n "5,6" -f "Jan 31 2017 07:00:00" -d 100
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~fds - proto Diff (Page-1)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                :   c1d635c2a270008   .               |   c81635c2a270008   .               |   
.                   :   Total             Diff            |   Total             Diff            |   
Jan 31 2017 07:00:00:   [0, 510, 510]     [0, 510, 510]   |   [0, 508, 508]     [0, 508, 508]   |   
Jan 31 2017 07:00:10:   [0, 510, 510]     [0, 0, 0]       |   [0, 508, 508]     [0, 0, 0]       |   
Jan 31 2017 07:00:20:   [0, 510, 510]     [0, 0, 0]       |   [0, 508, 508]     [0, 0, 0]       |   
Jan 31 2017 07:00:30:   [0, 510, 510]     [0, 0, 0]       |   [0, 508, 508]     [0, 0, 0]       
.
.
.

grep

The grep command displays lines that match a specified pattern. You can merge lines from a different file based on timestamp and display lines in ascending order of the timestamp.

You can specify any of the following:

• Multiple match patterns.
• Log selection.
• Ignore case.
• Invert match.
• Filter lines that are identical so only a single line is displayed, similar to the Unix uniq command.
• Time range.
• The-p option to set the paging number of lines to display, with default 10 lines.

Log-analyzer> grep -s "reads" -n "5,6" -f "Jan 31 2017 07:00:00" -d 100
  c81635c2a270008::Jan 31 2017 07:00:05 GMT: INFO (info): (hist.c:137) histogram dump: reads (0 total) msec
  c1d635c2a270008::Jan 31 2017 07:00:09 GMT: INFO (info): (hist.c:137) histogram dump: reads (0 total) msec
  c81635c2a270008::Jan 31 2017 07:00:15 GMT: INFO (info): (hist.c:137) histogram dump: reads (0 total) msec
.
.
.

Histogram

The histogram command analyzes server logs and displays histogram measurements. For background information and types of histograms, see Histograms.

The histogram command parses histogram lines for a specified time range and calculates the percentages of average and maximum operations in each time range.

Note: By default, histogram displays only three buckets of latency percentages. For an example of displaying more buckets with finer granularity, see Increasing buckets and granularity of measurement.

For Aerospike versions 3.9 and later, you can use the -N <namespace> option for a latency histogram for a specific namespace.

Use the -p option to set the paging number of lines to display (default 10 lines).

Log-analyzer> histogram -h reads -f head -d 150 -p 20 -f 'Jan 27 2016 22:26:00'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~reads Latency (Page-1)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                :   node0    .        .         .         |   node1    .        .         .         |   node2    .        .         .         |   
.                   :   % >1ms   % >8ms   % >64ms   ops/sec   |   % >1ms   % >8ms   % >64ms   ops/sec   |   % >1ms   % >8ms   % >64ms   ops/sec   |   
Jan 27 2016 22:26:10:   9.09     6.06     0.00      3.3       |   []       []       []        []        |   []       []       []        []        |   
Jan 27 2016 22:26:20:   0.00     0.00     0.00      0.8       |   3.70     0.00     0.00      2.7       |   11.11    0.00     0.00      0.9       |   
Jan 27 2016 22:26:30:   0.00     0.00     0.00      0.0       |   0.00     0.00     0.00      0.1       |   0.00     0.00     0.00      0.0       |   
.
.
.
avg                 :   0.61     0.40     0.00      1.0       |   0.42     0.00     0.00      1.0       |   0.89     0.00     0.00      1.0       |   
max                 :   9.09     6.06     0.00      19.8      |   3.70     0.00     0.00      19.0      |   11.11    0.00     0.00      14.5      |   

Syntax variations for histograms for a specific namespace

This same output can be displayed with either of the following command lines:

• Namespace prefix on histogram type. The braces are required: histogram -h {<namespace>}-write
• Namespace with -N: histogram -h write -N <namespace>

Increasing buckets and granularity of measurement

In a "stretch cluster" configuration, where nodes are geographically distant from each other, you might need percentage of latencies larger than histogram's default display.

Example

You can use a combination of pager on, -b, and -e to more sharply isolate where the latency occurs.

1. So you can avoid wrapped lines and scroll the display horizontally and vertically, turn on paging: pager on

2. Use a combination of -b and -e to specify the granulaity of percentages and number of buckets. For example, a combination of-b 4 -e 2 displays four buckets of latency in milliseconds in increasing 1, 4, 16, and 64 ratios. The output shows these buckets:

   % >1ms % >4ms % >16ms % >64ms.

3. Scroll vertically with the arrow keys to see all the buckets and horizontally to see all the values.

4. Turn off pager mode: pager off

list

The list command displays the names of server logs currently added and selected for processing.

Log-analyzer> list
Added Logs:
1  : bb9635c2a270008     /var/log/aerospike/aerospike.log

Selected Logs:
     bb9635c2a270008     /var/log/aerospike/aerospike.log

pager

The pager command sets paging size of output before pausing or turns the paging off.

• pager on: Sets paging size of output before pausing. Paging size is specified with the -p option. When pager mode is on, you can also scroll the output display vertically or horizontally.

  The command pager on is especially useful when looking at more than the log analyzer's default number of histogram buckets. For an example, see histogram.

• pager off: Turn off pausing at paging size.

remove

The remove command removes logs from the All and Selected lists.

To remove all server logs, specify the all argument.

Log-analyzer>
Log-analyzer> remove 1 3 7
INFO: Removed Log File /home/vagrant/shared/data/testlogs/files/6.log.
INFO: Removed Log File /home/vagrant/shared/data/testlogs/files/3.log.
INFO: Removed Log File /var/log/aerospike/aerospike.log.
Log-analyzer>

select

The select command provides a way to select server log from all list to selected list. It removes existing logs from the All list and adds newly selected logs.

Log-analyzer> list
Added Logs:
1  : MD5_0b32a78b084     /home/vagrant/shared/data/testlogs/files/6.log
2  : MD5_13e2b3a769b     /home/vagrant/shared/data/testlogs/files/5.log
3  : c1d635c2a270008     /var/log/aerospike/aerospike1.log
.
.
.
Log-analyzer> select 1 2 3
Log-analyzer>     
Log-analyzer> list
Added Logs:
1  : MD5_0b32a78b084     /home/vagrant/shared/data/testlogs/files/6.log
2  : MD5_13e2b3a769b     /home/vagrant/shared/data/testlogs/files/5.log
3  : c1d635c2a270008     /var/log/aerospike/aerospike1.log

Selected Logs:
     MD5_0b32a78b084     /home/vagrant/shared/data/testlogs/files/6.log
     MD5_13e2b3a769b     /home/vagrant/shared/data/testlogs/files/5.log
     c81635c2a270008     /var/log/aerospike/aerospike2.log
.
.
.