Skip to main content

Resonate component metrics

The Resonate server exposes a metrics endpoint :9090/metrics that is compatible with Prometheus.

The following metrics are available:

Shell
# HELP aio_connection number of aio subsystem connections
# TYPE aio_connection gauge
aio_connection{type="sender:poll"} 0
# HELP aio_in_flight_submissions number of in flight aio submissions
# TYPE aio_in_flight_submissions gauge
aio_in_flight_submissions{type="store"} 0
# HELP aio_total_submissions total number of aio submissions
# TYPE aio_total_submissions counter
aio_total_submissions{status="success",type="store"} 0
# HELP aio_worker_count number of aio subsystem workers
# TYPE aio_worker_count gauge
aio_worker_count{type="router"} 0
aio_worker_count{type="sender"} 0
aio_worker_count{type="sender:http"} 0
aio_worker_count{type="sender:poll"} 0
aio_worker_count{type="store:sqlite"} 0
# HELP aio_worker_in_flight_submissions number of in flight aio submissions
# TYPE aio_worker_in_flight_submissions gauge
aio_worker_in_flight_submissions{type="router",worker="0"} 0
aio_worker_in_flight_submissions{type="sender",worker="0"} 0
aio_worker_in_flight_submissions{type="sender:http",worker="0"} 0
aio_worker_in_flight_submissions{type="sender:poll",worker="0"} 0
aio_worker_in_flight_submissions{type="store:sqlite",worker="0"} 0
# HELP coroutines_in_flight number of in flight coroutines
# TYPE coroutines_in_flight gauge
coroutines_in_flight{type="EnqueueTasks"} 0
coroutines_in_flight{type="SchedulePromises"} 0
coroutines_in_flight{type="TimeoutLocks"} 0
coroutines_in_flight{type="TimeoutPromises"} 0
coroutines_in_flight{type="TimeoutTasks"} 0
# ... For all coroutine types
# HELP coroutines_total total number of coroutines
# TYPE coroutines_total counter
coroutines_total{type="EnqueueTasks"} 0
coroutines_total{type="SchedulePromises"} 0
coroutines_total{type="TimeoutLocks"} 0
coroutines_total{type="TimeoutPromises"} 0
coroutines_total{type="TimeoutTasks"} 0
# ... For all coroutine types

The aio prefix refers to stuff that “goes out” of the Server such as requests to the store, sending tasks to nodes, etc. Coroutines refers to the units of business logic in the Server.

Using Prometheus and Grafana​

The way Grafana works is by pulling data from any timeseries db, in our case this db will be Prometheus. The way Prometheus works is by pulling data from the Resonate Server every X seconds. X is part of the config.

Get Prometheus and Grafana both from their websites and follow the instructions to get both projects onto your machine, whether that is via Docker or direct binary download

Run Grafana.

Shell
./grafana server

Define a prometheus.yml config file so Prometheus understands how to connect to the Resonate Server.

YAML
# prometheus.yml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "Resonate"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]
        # The label name is added as a label `label_name=<label_value>` to any timeseries scraped from this config.
        labels:
          app: "resonate"

The Resonate specific part is the scrape_configs section.

Run Prometheus with the config file you just created, specifying port 9091 in the --web.listen-address to avoid port conflicts with the Resonate Server. You can customize the port that Resonate exports metrics to with the Resonate Server config.

Shell
./prometheus --config.file=prometheus.yml --web.listen-address=:9091

In the Grafana UI, add Prometheus as a data source.