Elasticsearch alerts and checks with arcodash

Connect Elasticsearch

Add an Elasticsearch connector with host, port (9200 by default), username, password, a Use HTTPS toggle and an optional default index. See the connectors overview for the connection test and schema refresh. Authentication is HTTP basic only, so an API key or a cloud token is not an option. arcodash does not publish fixed outbound IP addresses, which means the cluster must accept connections from the internet, over HTTPS.

Create a role that reads only the indices you chart, and a user that holds it:

PUT _security/role/arcodash_ro
{
  "indices": [
    { "names": [ "orders-*", "metrics-business" ], "privileges": [ "read", "view_index_metadata" ] }
  ]
}

view_index_metadata is what lets the schema browser read mappings, which it does with a request across all indices. If the connection test fails on a role this narrow, the test calls the root endpoint, so check whether your cluster wants the monitor cluster privilege added.

How a query becomes rows

The query text is Query DSL JSON. A first line without braces is treated as the index:

orders-*
{ "size": 1, "sort": [ { "@timestamp": "desc" } ], "_source": [ "ingest_lag_seconds" ] }

The connector sends it to /<index>/_search. Index names allow letters, digits and _ - . + , *, so wildcards and comma lists work. With no index line it uses the default index from the form, and with neither it searches everything.

Then comes the part that decides what you can monitor. arcodash reads hits.hits and nothing else. Each hit becomes a row: _id plus every key in _source. Aggregations, hits.total, fields and script_fields are all ignored. A monitor then reads one column of the first row and needs a number.

So the pattern is the opposite of SQL. You do not ask Elasticsearch to compute the number at query time. You make sure a numeric field exists on a document, and the query finds the right document.

A freshness check

If your shipper or an ingest pipeline writes a numeric ingest_lag_seconds on every event (arrival time minus event time), the newest event’s lag is a number you can monitor:

orders-*
{ "size": 1, "sort": [ { "@timestamp": "desc" } ], "_source": [ "ingest_lag_seconds" ] }

Monitor ingest_lag_seconds with > and your tolerance. Be clear about what this does not catch. It measures how late the newest document was when it arrived. It cannot see a feed that stopped, because no new document means the newest one still looks fine, and a stopped feed gives no new lag number. Query DSL has no way to compute “now minus the newest timestamp” in a field arcodash will read. For a stopped feed, check the data where it lands in a store that can compute it, or use Kibana’s alerting rules, which can evaluate the hit count in a window.

A volume check

Because counts are dropped, store them. A continuous transform can group events by hour and write one summary document per hour, with an event count field. Name it event_count, in an index such as orders-hourly. Then read the last complete hour:

orders-hourly
{
  "size": 1,
  "sort": [ { "hour": "desc" } ],
  "query": { "range": { "hour": { "lt": "now/h" } } },
  "_source": [ "event_count" ]
}

now/h rounds to the start of the current hour, so lt excludes the hour still being filled. The monitor reads event_count with < and a floor. The transform’s frequency and delay settings decide how soon an hour’s number is final, so give the monitor a schedule slower than that. Same caveat as above: an hour with no events produces no summary document at all, so the query returns the last busy hour and the number looks healthy. Treat this check as a drop detector, not an outage detector.

A metric alert

This is where the connector fits best: a document that already holds the figure. Suppose a job writes a business metric every five minutes to metrics-business:

metrics-business
{
  "size": 1,
  "sort": [ { "@timestamp": "desc" } ],
  "query": { "bool": { "filter": [
    { "term": { "metric": "checkout_error_pct" } },
    { "range": { "@timestamp": { "gte": "now-30m" } } }
  ] } },
  "_source": [ "value" ]
}

Monitor value with > and 2. The now-30m filter stops an old reading from standing in for a current one: if the writer stops, no document matches, there is no row, and the monitor goes unknown instead of staying green on stale data.

Gotchas

• Field names with dots. _source keys come back as written. A nested object arrives as one JSON value in one column, not flattened, so alert on top-level numeric fields.
• Index aliases and data streams. A wildcard like logs-* covers rolled-over backing indices. Sort on @timestamp across all of them, not on a per-index counter.
• Timestamps are UTC. now in date math is always UTC, and date fields come back as strings in whatever format the mapping stores.
• Mapping types. A numeric field mapped as keyword sorts alphabetically and returns a string. A monitor needs a real number, so check the mapping.
• Default size. No size means 10 hits. Set "size": 1 on checks so the first row is the one you meant.
• Timeout. Requests time out after 30 seconds.

Dashboard tiles

arcodash charts rows, so aim tiles at documents that are already summaries:

• Latest value stat. The metric query above without the now-30m filter, as a counter tile.
• Hourly volume trend. orders-hourly with "size": 48 and "sort": [ { "hour": "asc" } ] and _source of hour and event_count, as a line chart.
• Recent slow requests table. { "size": 20, "sort": [ { "duration_ms": "desc" } ], "query": { "range": { "@timestamp": { "gte": "now-1h" } } }, "_source": [ "@timestamp", "path", "duration_ms" ] }.

Honestly, if you mostly want to explore logs, Kibana is the right tool and does it better. arcodash earns its place when a number in Elasticsearch needs to sit next to your database dashboards and page the same people.

MongoDB is the other document store arcodash can query.

docs

Elasticsearch setup in the docs

Connection fields, an example configuration and caveats.