Metrics
Prometheus
Configuration
To configure a Prometheus metrics endpoint, you need to configure the HTTP endpoint in the Secret Agent's configuration file as below.
Sample Secret Agent's configuration file to configure the HTTP endpoint.
service:
http:
endpoint: 0.0.0.0:8080
Secret Agent uses the configured HTTP service to expose Prometheus endpoint via /manage/rest/v1/prometheus.
Full URL will be something like http://localhost:8080/manage/rest/v1/prometheus
Exposed Metrics
Secret Agent exposes metrics for
- Listeners like TCP, UDS.
- Secret Managers like AWS, GCP.
All metrics have the label module which value represents the module emitting the metric.
Metrics for Listeners
| Metrics Name | Description | Modules | Type |
|---|---|---|---|
aerospike_sa_connections_active | Number of active connections | TCP, UDS | Gauge |
aerospike_sa_connections_open | Number of opened connections | TCP, UDS | Counter |
aerospike_sa_connections_closed | Number of closed connections | TCP, UDS | Counter |
aerospike_sa_connections_err | Number of connection errors | TCP, UDS | Counter |
aerospike_sa_read_err | Number of errors while reading from connections | TCP, UDS | Counter |
aerospike_sa_write_err | Number of errors while writing to connections | TCP, UDS | Counter |
aerospike_sa_tls_err | Number of errors during the tls handshake for connections | TCP, UDS | Counter |
aerospike_sa_parse_err | Number of errors while parsing the requests | TCP, UDS | Counter |
Metrics for Secret Managers
| Metrics Name | Description | Modules | Type |
|---|---|---|---|
aerospike_sa_success | Number of successful fetch requests | AWS, GCP | Counter |
aerospike_sa_fetch_latency_p50 | p50 latency for fetch requests | AWS, GCP | Gauge |
aerospike_sa_fetch_latency_p95 | p95 latency for fetch requests | AWS, GCP | Gauge |
aerospike_sa_resource_err | Number of errors due to bad resource name in the request | AWS, GCP | Counter |
aerospike_sa_fetch_err | Number of errors while fetching secrets from external secret managers | AWS, GCP | Counter |
aerospike_sa_key_not_found_err | Number of errors where the requested key is not found in the fetched key-value pairs | AWS | Counter |
aerospike_sa_session_err | Number of errors while creating a client session to external secret managers | AWS, GCP | Counter |
aerospike_sa_role_err | Number of errors while assuming role (AWS)/impersonating (GCP) | AWS, GCP | Counter |
Custom labels
You can configure custom labels for Prometheus metrics as well. Labels can be provided
as key-value pair in the metrics context.
Sample Secret Agent's configuration file to configure custom labels.
...
...
http:
endpoint: 0.0.0.0:8080
metrics:
prometheus:
labels:
label1: val1
label2: val2
...
...
Log Ticker
Along with Prometheus metrics, Secret Agent also prints metrics in the log file every 10 seconds. Log ticker format is:
<module name>: <metric1> <val1>, <metric2> <val2>, ...
Sample ticker log.
[INFO] 2023/09/09 11:22:34 metrics.go:65: tcp: connections_active 1, connections_opened 2, connections_closed 1, connections_err 0, read_err 0, write_err 0, tls_err 0, parse_err 0
[INFO] 2023/09/09 11:22:34 metrics.go:65: uds: connections_active 0, connections_opened 0, connections_closed 0, connections_err 0, read_err 0, write_err 0, tls_err 0, parse_err 0
[INFO] 2023/09/09 11:22:34 metrics.go:65: aws: success 2, fetch_latency_p50 38, fetch_latency_p95 108, resource_err 0, fetch_err 0, key_not_found_err 0, session_err 0, role_err 0
[INFO] 2023/09/09 11:22:44 metrics.go:65: tcp: connections_active 1, connections_opened 2, connections_closed 1, connections_err 0, read_err 0, write_err 0, tls_err 0, parse_err 0
[INFO] 2023/09/09 11:22:44 metrics.go:65: uds: connections_active 0, connections_opened 0, connections_closed 0, connections_err 0, read_err 0, write_err 0, tls_err 0, parse_err 0
[INFO] 2023/09/09 11:22:44 metrics.go:65: aws: success 2, fetch_latency_p50 38, fetch_latency_p95 108, resource_err 0, fetch_err 0, key_not_found_err 0, session_err 0, role_err 0