Skip to main content

Resonate component metrics

The Resonate server exposes a metrics endpoint :9090/metrics that is compatible with Prometheus.

The aio prefix refers to stuff that “goes out” of the server such as requests to the store, sending tasks to nodes, etc. Coroutines refers to the units of business logic in the Server.

Metrics exposed​

aio_connection​

Number of aio subsystem connections.

  • gauge
  • aio_connection{type="sender:poll"} 0

aio_in_flight_submissions​

Number of in flight aio submissions.

  • gauge
  • aio_in_flight_submissions{type="store"} 0

aio_total_submissions​

Total number of aio submissions.

  • counter
  • aio_total_submissions{status="success",type="store"} 0

aio_worker_count​

Number of aio subsystem workers.

  • gauge
  • aio_worker_count{type="router"} 0
  • aio_worker_count{type="sender"} 0
  • aio_worker_count{type="sender:http"} 0
  • aio_worker_count{type="sender:poll"} 0
  • aio_worker_count{type="store:sqlite"} 0

aio_worker_in_flight_submissions​

Number of in flight aio submissions.

  • gauge
  • aio_worker_in_flight_submissions{type="router",worker="0"} 0
  • aio_worker_in_flight_submissions{type="sender",worker="0"} 0
  • aio_worker_in_flight_submissions{type="sender:http",worker="0"} 0
  • aio_worker_in_flight_submissions{type="sender:poll",worker="0"} 0
  • aio_worker_in_flight_submissions{type="store:sqlite",worker="0"} 0

coroutines_in_flight number​

Number of in flight coroutines.

  • gauge
  • coroutines_in_flight{type="EnqueueTasks"} 0
  • coroutines_in_flight{type="SchedulePromises"} 0
  • coroutines_in_flight{type="TimeoutLocks"} 0
  • coroutines_in_flight{type="TimeoutPromises"} 0
  • coroutines_in_flight{type="TimeoutTasks"} 0

coroutines_total​

Total number of coroutines.

  • counter
  • coroutines_total{type="EnqueueTasks"} 0
  • coroutines_total{type="SchedulePromises"} 0
  • coroutines_total{type="TimeoutLocks"} 0
  • coroutines_total{type="TimeoutPromises"} 0
  • coroutines_total{type="TimeoutTasks"} 0

Using Prometheus and Grafana​

The way Grafana works is by pulling data from any timeseries db, in our case this db will be Prometheus. The way Prometheus works is by pulling data from the Resonate Server every X seconds. X is part of the config.

Get Prometheus and Grafana both from their websites and follow the instructions to get both projects onto your machine, whether that is via Docker or direct binary download

Run Grafana.

Shell
./grafana server

Define a prometheus.yml config file so Prometheus understands how to connect to the Resonate Server.

YAML
# prometheus.yml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "Resonate"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]
        # The label name is added as a label `label_name=<label_value>` to any timeseries scraped from this config.
        labels:
          app: "resonate"

The Resonate specific part is the scrape_configs section.

Run Prometheus with the config file you just created, specifying port 9091 in the --web.listen-address to avoid port conflicts with the Resonate Server. You can customize the port that Resonate exports metrics to with the Resonate Server config.

Shell
./prometheus --config.file=prometheus.yml --web.listen-address=:9091

In the Grafana UI, add Prometheus as a data source.