Prometheus business metric alerts with PromQL in arcodash

Connect Prometheus

The Prometheus form has three fields: host, port (9090 by default) and a Use HTTPS toggle. That is all. There is no username, password, bearer token or custom header, so arcodash can only talk to a Prometheus that accepts unauthenticated requests from the internet. See the connectors overview for the connection test, which runs the query 1 against the HTTP API, and for schema refresh, which lists metric names.

That is a real limit. A Prometheus behind basic auth, a managed service that requires a token, or a private server on an internal network will not work with this connector today. If yours is open, restrict it as far as you can with whatever your setup allows. arcodash does not publish fixed outbound IP addresses, so IP allowlists are not an option either.

There is no database user to create. Prometheus has no per-user permissions of its own, which is part of why the connection must be treated carefully.

Where Prometheus alerting already lives

Prometheus alerting rules, Alertmanager and Grafana alerting are the native answer for infrastructure: a service is down, memory is full, p99 latency is high. They give you grouping, silences, inhibition and on-call routing designed for that job. arcodash does not replace them and you should not rebuild them here.

The gap is business metrics. Teams often export orders_created_total or signups_total into Prometheus because it is convenient. Those numbers deserve the same alert routes as your revenue tables, a dashboard next to your SQL data and an incident history in one place. That is what arcodash is for here.

How a query becomes a number

A query is a PromQL expression, sent to /api/v1/query as an instant query. Prometheus evaluates it at that moment and arcodash turns the result into rows with three columns: metric (the label set as JSON text), timestamp (Unix seconds) and value (a float).

A monitor reads the value column of the first row, so the expression should produce exactly one series. Aggregate with sum, max or min and drop the labels. NaN and infinity become null, which makes the monitor unknown. Request timeout is 30 seconds. Put the query on a schedule, because the monitor only evaluates when the query runs.

One more behavior to know. If the expression returns a bare scalar, such as time() - 5 or 2 * 3, arcodash gets a two-element array instead of a list of series, and the rows come out with null values. Read from the connector source, so test your own expression. Wrapping it with vector(...) turns the scalar into a one-element vector, which works. The freshness query below relies on this.

A freshness check

If your job exports the time of its last successful run as a Unix timestamp gauge (a common pattern, often named like pipeline_last_success_timestamp_seconds):

time() - max(orders_pipeline_last_success_timestamp_seconds)

time() is the evaluation time in seconds, max() collapses to one series, and scalar minus vector yields a one-sample vector. The value is the age in seconds. Monitor it with > and 3600 for “no success in the last hour”.

If the exporter disappears entirely, the series goes stale after about five minutes and the expression returns nothing, so the monitor goes unknown. To get a number in that case, use absent():

absent(orders_pipeline_last_success_timestamp_seconds) or vector(0)

absent() returns 1 when the metric has no series and an empty vector otherwise. The or vector(0) fills the empty case with 0, so the result is 1 when the metric is missing and 0 when it is present. Monitor == 1. Run it as a second monitor next to the age check.

A volume check

Events in the last hour, from a counter:

sum(increase(orders_created_total[1h]))

increase takes a counter over a range and returns how much it grew, extrapolated to the window; sum gives one series. Monitor value with < and a floor tuned to a normal hour. Because the number is extrapolated it is a decimal and slightly fuzzy, so do not set a floor that a normal quiet hour sits right on. Counters that reset on a restart are handled by increase. A counter that was never incremented in the window is fine and gives 0, but a counter that does not exist yet gives no series at all.

A metric alert

Share of payments failing over 15 minutes:

sum(rate(payments_failed_total[15m])) / sum(rate(payments_total[15m]))

The result is a ratio between 0 and 1. Monitor value with > and 0.05. When there are no payments the denominator is 0 and the division gives NaN, which arrives as null and leaves the monitor unknown, rather than falsely green.

Gotchas

• Instant, not range. The evaluation time is when the query runs, so a late scheduled run reads the data as of then.
• Rate windows need scrapes. rate(x[5m]) wants at least two samples in the window. A 1 minute window on a 30 second scrape is noisy.
• Labels in the first row. Without aggregation the first row is whichever series Prometheus returns first, not the worst one. Aggregate or use topk(1, ...) deliberately.
• High cardinality. An unaggregated query across thousands of series returns thousands of rows. Aggregate to keep tiles readable.
• Recording rules. If a ratio is heavy, precompute it as a recording rule on your Prometheus and query that metric name here.

Dashboard tiles

• Orders in the last 24 hours. sum(increase(orders_created_total[24h])) as a single number tile.
• Failure rate trend. A subquery in an instant query returns a matrix, one row per sample, and can feed a line chart: sum(rate(payments_failed_total[5m])) / sum(rate(payments_total[5m])) wrapped as (...)[6h:5m]. The subquery syntax is in the Prometheus querying basics.
• Orders by region. sum by (region) (increase(orders_created_total[24h])) as a bar chart, reading the label from the metric column.

If you only want infrastructure dashboards and alerts, stay with Grafana. Add arcodash when the number is a business one and a non-engineer needs to see it.

InfluxDB metrics are covered on the InfluxDB connector page.

docs

Prometheus setup in the docs

Connection fields, an example configuration and caveats.