MongoDB data freshness check and alerts with arcodash

Connect MongoDB

Add a MongoDB connector with host, port (27017 by default), username, password, database, auth source (admin by default) and an optional replica set name. The generic flow, including the connection test (a ping command) and schema refresh, is in the connectors overview. Schema introspection samples one document from each collection, so the field list in the editor is whatever that one document happens to have.

Create a user that can only read the database arcodash will query:

use admin
db.createUser({
  user: "arcodash_ro",
  pwd: passwordPrompt(),
  roles: [{ role: "read", db: "app" }]
})

Creating it in admin matches the default auth source. If you create the user inside app instead, set Auth Source to app.

Two connection details to check before you start. First, the connector builds a plain mongodb://host:port/db address from the form fields. There is no TLS switch and no mongodb+srv support, so a deployment that only accepts TLS connections, or one you reach through an SRV record such as a typical Atlas cluster, will not connect through the current form. Second, arcodash does not publish fixed outbound IP addresses, so the server has to be reachable from the internet. Both are worth confirming with a connection test before you plan around this connector.

What a query looks like

A query is one JSON document:

{ "collection": "orders", "aggregate": [ { "$match": { "status": "paid" } }, { "$count": "paid" } ] }

Without an aggregate key it is a plain find: {"collection": "orders", "filter": {"status": "paid"}, "limit": 50}. A find has no sort and no projection, so use aggregate for everything a monitor or chart needs. Pipelines are capped at 10,000 returned documents (arcodash adds a $limit stage and errors if more come back), and the server applies the workspace statement timeout as maxTimeMS.

Each returned document becomes a row and its top-level keys become columns. Dates arrive as epoch milliseconds, ObjectIds as hex strings. A monitor reads one column of the first row, and it must be a number.

A freshness check

The age of the newest document, in seconds:

{
  "collection": "events",
  "aggregate": [
    { "$sort": { "created_at": -1 } },
    { "$limit": 1 },
    { "$project": {
        "_id": 0,
        "seconds_since_last": { "$divide": [ { "$subtract": [ "$$NOW", "$created_at" ] }, 1000 ] }
    } }
  ]
}

How it becomes the monitor’s number: the pipeline returns one document with one key, seconds_since_last. $subtract on two dates returns milliseconds, $divide by 1000 gives seconds, and $$NOW is the same instant for the whole pipeline. Point a monitor at the seconds_since_last column with > and 3600 to alert when nothing has landed for an hour, and put the query on a schedule so it runs without anyone looking.

I used $sort plus $limit rather than a $group with $max on purpose. With an index on created_at the sort reads one index entry. A $group reads the whole collection every time the check runs. If created_at is the time a document was created rather than loaded, a quiet night will look stale. Prefer a loaded-at field if your pipeline writes one.

A volume check

Count documents in the last hour, and still return a number when there are none:

{
  "collection": "orders",
  "aggregate": [
    { "$match": { "$expr": { "$gte": [ "$created_at", { "$dateSubtract": { "startDate": "$$NOW", "unit": "day", "amount": 7 } } ] } } },
    { "$group": {
        "_id": null,
        "orders_last_hour": { "$sum": { "$cond": [
          { "$gte": [ "$created_at", { "$dateSubtract": { "startDate": "$$NOW", "unit": "hour", "amount": 1 } } ] }, 1, 0 ] } }
    } },
    { "$project": { "_id": 0, "orders_last_hour": 1 } }
  ]
}

The seven-day $match bounds the scan, and the $group then counts only the last hour. If the load died this morning, the pipeline still returns a document with orders_last_hour: 0, which a < monitor can catch. A plain $match then $count would return nothing at all when the count is zero, and that is the exact case you care about. $dateSubtract needs MongoDB 5.0 or later. If the collection has no documents in the whole seven days, the result is empty and the monitor is unknown.

Pick the floor from what a normal hour looks like on a weekday, or use a baseline monitor on the column, which learns the usual level from its own history instead of a hand-picked number.

A metric alert

Share of failed payments in the last 30 minutes:

{
  "collection": "payments",
  "aggregate": [
    { "$match": { "$expr": { "$gte": [ "$created_at", { "$dateSubtract": { "startDate": "$$NOW", "unit": "minute", "amount": 30 } } ] } } },
    { "$group": { "_id": null, "total": { "$sum": 1 },
        "failed": { "$sum": { "$cond": [ { "$eq": [ "$status", "failed" ] }, 1, 0 ] } } } },
    { "$project": { "_id": 0, "failed_pct": { "$multiply": [ 100, { "$divide": [ "$failed", "$total" ] } ] } } }
  ]
}

Monitor failed_pct with > and 5. No payments in the window means no document, so the monitor goes unknown rather than dividing by zero. That is the right outcome for a quiet window, and the freshness check above covers the silent-feed case.

Gotchas for monitoring

• Dates are UTC. $$NOW and stored BSON dates are both UTC. Day buckets from $dateTrunc default to UTC too, so “today” means UTC today unless you pass a timezone.
• Date literals. The query text is plain JSON, so you cannot write a BSON date literal. Build every date relative to $$NOW as above.
• Unindexed scans. An $expr filter on a large collection with no index on the date field scans it every run. Index the timestamp you filter on, and check cadence against load before choosing every 5 minutes.
• First row only. A pipeline that returns several documents is read from the first. End with $limit: 1 or a $group with _id: null when the check is a single number.

Dashboard tiles

Orders per day for two weeks:

{ "collection": "orders", "aggregate": [
  { "$match": { "$expr": { "$gte": [ "$created_at", { "$dateSubtract": { "startDate": "$$NOW", "unit": "day", "amount": 14 } } ] } } },
  { "$group": { "_id": { "$dateTrunc": { "date": "$created_at", "unit": "day" } }, "orders": { "$sum": 1 } } },
  { "$sort": { "_id": 1 } }
] }

Revenue by status:

{ "collection": "orders", "aggregate": [
  { "$group": { "_id": "$status", "revenue": { "$sum": "$total" }, "orders": { "$sum": 1 } } },
  { "$sort": { "revenue": -1 } }
] }

Top ten customers by spend this month:

{ "collection": "orders", "aggregate": [
  { "$group": { "_id": "$customer_id", "spend": { "$sum": "$total" } } },
  { "$sort": { "spend": -1 } },
  { "$limit": 10 }
] }

Rename _id with a $project stage when you want a friendlier column header on a table tile. If a tile is slow, the usual fix is a $match on an indexed field as the first stage.

Connecting MongoDB to arcodash starts with the 14 day free trial, no card needed.

Elasticsearch is the other document-style source.

docs

MongoDB setup in the docs

Connection fields, an example configuration and caveats.