Data observability for small teams, without a platform

Who this is for

You have a Postgres or MySQL database behind a product, maybe a warehouse fed by a couple of sync jobs, and a handful of dashboards people trust. Nobody’s title says data engineer. When a number looks wrong, the founder or the one backend developer finds out from a customer or a Monday meeting.

Data observability tools sell to teams with a platform group. You don’t need a lineage graph, a catalog or 40 integrations. You need to hear about three kinds of failure before the people who read the dashboards do. The glossary entry defines the term; this page is the starter plan.

What to monitor first

Don’t cover every table. Pick the two or three that, if they were wrong for a day, someone would make a bad decision or a customer would notice. Usually that’s orders or payments, signups or accounts, and the main event table.

For each, ask four questions in this order:

1. Did new data arrive? (freshness)
2. Did the right amount arrive? (volume)
3. Is it shaped correctly? (nulls, duplicates, impossible values)
4. Does it still agree with the table it joins to? (orphans)

Stop there. Everything else, schema drift, distribution shifts, lineage, is worth adding once these five are quiet and trusted.

Five checks to start with

Each query returns exactly one row with one number. That matters: an arcodash monitor reads the first row of the query result, takes the named column, and compares it with a threshold. Tables here are an orders table with loaded_at, total, status and customer_id, and a customers table.

1. Freshness. Hours since the pipeline last wrote. Alert when it passes the limit you choose, for example > 6. The coalesce makes an empty table count as stale instead of silently returning nothing.

SELECT coalesce(
  round((extract(epoch FROM now() - max(loaded_at)) / 3600)::numeric, 1),
  9999
) AS hours_since_last_load
FROM orders;

2. Daily volume. Yesterday’s rows as a percent of the median day over the last 14 complete days. Alert when it is < 50. Using complete days avoids a false alarm at 9am when today has barely started.

WITH daily AS (
  SELECT created_at::date AS d, count(*) AS n
  FROM orders
  WHERE created_at >= current_date - 14 AND created_at < current_date
  GROUP BY 1
), base AS (
  SELECT percentile_cont(0.5) WITHIN GROUP (ORDER BY n) AS med FROM daily
)
SELECT round((100.0 * (
  SELECT count(*) FROM orders
  WHERE created_at >= current_date - 1 AND created_at < current_date
) / nullif(med, 0))::numeric, 0) AS pct_of_normal
FROM base;

3. Null rate. Share of recent rows missing a value that should always be there. Alert on > 1, not > 0, because a few legitimate nulls are normal.

SELECT round((100.0 * count(*) FILTER (WHERE total IS NULL)
  / nullif(count(*), 0))::numeric, 1) AS null_total_pct
FROM orders
WHERE created_at >= now() - interval '7 days';

4. Duplicate keys. A replayed sync or a retried job inserts the same record twice and revenue doubles quietly. Alert on > 0.

SELECT count(*) AS duplicate_orders
FROM (SELECT id FROM orders GROUP BY id HAVING count(*) > 1) t;

5. Orphans and invalid values. Rows that point at a customer that doesn’t exist, or carry a status your code never writes. Alert on > 0.

SELECT count(*) AS orphan_orders
FROM orders o
LEFT JOIN customers c ON c.id = o.customer_id
WHERE c.id IS NULL;

Repeat the pattern for the next table. The row count post goes deeper on check 2, and data freshness covers check 1.

How arcodash runs them

In arcodash a check is a saved query on a schedule. A monitor attaches a column, an operator and a value to that query, and notifies whoever is subscribed when the state changes between ok and triggered. See alerts overview for the exact semantics and data observability monitors for these recipes in the product’s own words.

Destinations include email, Slack, Teams, webhook, PagerDuty and others; see notification destinations. Put each query on a schedule or nothing runs until someone opens it.

Where AI helps, and where it does not

Three shipped pieces, all optional:

• Baseline monitors. Pick Baseline instead of Threshold and arcodash learns normal from the column’s own history. How it works is described on metric monitoring; it is statistics, not a model, so it needs no AI key. It has no seasonality model, so a strong weekly pattern can cause false positives. If that’s your data, use check 2 as written, which compares against the median day.
• Suggest checks. On a connector’s detail page, Suggest checks with AI proposes checks like the five above from your schema and the connector’s AI profile. Accepting one creates the query, the monitor and your subscription. Row-count suggestions default to a baseline monitor. You bring your own Gemini, OpenAI or Anthropic key. Data handling lists exactly what is sent.
• Why did this fire? Alerts carry a one-line statistics summary. With an AI provider configured, arcodash also runs up to three read-only queries on the monitor’s data source and adds a short narrative with a confidence level. If that takes more than 20 seconds, the alert goes out without it.

None of this replaces choosing which tables matter. It saves typing and the first hour of a debugging session.

Cost of ownership

Be honest about what this costs, in the order you’ll feel it:

• Setup: an afternoon for 5 to 10 checks across three tables.
• Tuning: the first two weeks. Every check will fire once for a reason you decide is fine, and you’ll move a threshold. Start looser than feels right.
• Upkeep: a few minutes when a table changes shape. A new column or renamed status breaks a query, and the monitor goes to unknown rather than ok.
• Money: arcodash is $59 a month on Team or $300 on Org after a 14 day free trial (no card), unlimited seats, hosted only. You can also run these queries from cron with a script and a Slack webhook for nothing, which is a fine answer if you like maintaining the script.

Honest limits

• A monitor reads one value from the first row. It checks what you wrote and nothing else; it won’t notice a column quietly changing meaning.
• Checks run on your schedule, not on arrival of data. A 15 minute schedule means up to 15 minutes of delay.
• arcodash isn’t a lineage tool, a catalog or a data contract system. It tells you the number is wrong, and the AI explanation points you at where to look.

If five checks and a Slack channel is what you were hoping to find, you can start the trial and have the first one running today.