InfluxDB alerts and checks with InfluxQL in arcodash

Connect InfluxDB

This page is about InfluxDB 1.x. The form asks for host, port (8086 by default), database, username, password and a Use HTTPS toggle. InfluxDB 2 uses tokens, organizations and Flux, and arcodash does not support it. If you run version 2, only a 1.x compatibility setup that exposes the v1 query API and a database mapping would let this connector reach it.

See the connectors overview for the connection test and schema refresh. The test runs SHOW MEASUREMENTS, and arcodash authenticates with HTTP basic auth when a username is set. Create a user that can only read one database:

CREATE USER "arcodash_ro" WITH PASSWORD 'a-long-random-password'
GRANT READ ON "app_metrics" TO "arcodash_ro"

User management only has an effect when authentication is enabled on the server. Without it, anyone who can reach port 8086 can query. Turn it on, and use HTTPS, since basic auth sends the password with each request. arcodash does not publish fixed outbound IP addresses, so the server has to be reachable from the internet.

Where Grafana already lives

Grafana, with its alerting, is the native tool for InfluxDB dashboards and for infrastructure alerts such as CPU, disk and request latency. Kapacitor, Influx’s own alerting engine, covers streaming rules. arcodash is not a replacement. It fits when the measurement holds a business figure, such as orders per minute or checkout errors, and you want that figure on a dashboard with your SQL data and alerting through the same destinations.

How a query becomes a number

A query is any InfluxQL statement. arcodash sends it as an HTTP GET to /query for the configured database and flattens the response into rows. Every row has a series column (the measurement name plus tags, like orders[region=eu]), and then one column per returned field: time, count, mean and so on. Multiple series and multiple statements are all unioned into one result.

A monitor reads one column from the first row and needs a number. Two InfluxQL facts shape every check below. Rows come back oldest first unless you add ORDER BY time DESC, so a query that returns several rows must be ordered for the row you want. And per the InfluxQL data exploration docs, fill() is ignored if no data falls in the query’s time range, so a window with no data at all returns no rows. No rows means unknown, not triggered.

A freshness check

InfluxQL cannot subtract the latest timestamp from now() in the select list, and the timestamp comes back as text. So measure freshness as a count in a recent bucket, with a wider time range so there is always something for fill(0) to work with:

SELECT count("value") FROM "orders"
WHERE time > now() - 24h
GROUP BY time(15m) fill(0)
ORDER BY time DESC LIMIT 1 OFFSET 1

Reading it: the range covers 24 hours, so as long as the measurement wrote anything in that time, the 15 minute buckets are all returned, empty ones filled with 0. ORDER BY time DESC puts the newest bucket first, and OFFSET 1 skips it, because the newest bucket is the current one and is only partly filled. The row left is the last complete 15 minute bucket. Monitor the count column with == and 0.

The cost is delay. A feed that stops is noticed after a full empty bucket has passed, which is between 15 and 30 minutes. Use a bigger bucket for a slower feed. If the measurement has had no points at all for 24 hours, there are no rows and the monitor is unknown, so a very old outage needs a longer range.

A volume check

Points written in the last hour:

SELECT count("value") FROM "orders"
WHERE time > now() - 1h

With no GROUP BY time, this returns one row with a count column and a time of 1970-01-01T00:00:00Z, which you can ignore. Monitor count with < and a floor. The same caveat applies: if the hour was completely empty, no row is returned, so pair this with the gap check above, or use the grouped form with fill(0) and a wider range when “zero” has to alert.

Count a single field. Points can have several fields, and count("value") counts only points that have that field.

A metric alert

Average failed-checkout percentage over 30 minutes:

SELECT mean("failed_pct") FROM "checkout"
WHERE time > now() - 30m

The column is mean. Monitor it with > and 5. Avoid GROUP BY a tag here: it returns one series per tag value and a monitor would read only the first. If you need per-region alerts, create one query per region with a WHERE "region" = 'eu' filter. A silent writer leaves the window empty and the monitor unknown, which is why the freshness check sits beside it.

Gotchas

• Time zones. Timestamps are UTC. GROUP BY time(1d) buckets on UTC day boundaries, and the bucket boundaries are InfluxDB’s preset round-number boundaries.
• Default time range. A SELECT without a lower time bound scans the whole measurement. Always put a time > clause in.
• Retention. Data older than the retention policy is gone, so a long window can quietly return less than you expect.
• Cardinality. Queries grouped by a high-cardinality tag return a row per series. Keep tiles to a handful of groups.
• Fields versus tags. Tags are strings and cannot be aggregated with mean(). Only fields can.
• Timeout. Requests time out after 30 seconds.

Dashboard tiles

• Orders per 15 minutes, last day. SELECT count("value") FROM "orders" WHERE time > now() - 1d GROUP BY time(15m) fill(0) as a line chart on the time and count columns.
• Latest reading by region. SELECT last("value") FROM "orders_per_min" WHERE time > now() - 1h GROUP BY "region" as a table, one row per region from the series column.
• Checkout error average. The metric alert query above as a single number tile.

If the data is infrastructure telemetry, build it in Grafana. Bring it to arcodash when the number belongs next to your business data.

For Prometheus metrics, see the Prometheus connector.

docs

InfluxDB setup in the docs

Connection fields, an example configuration and caveats.