Databricks data freshness check and alerts

Connect Databricks

Add a Databricks connector with five fields: workspace host, HTTP path, token, and optionally a default catalog and schema. The field table is on the Databricks connector page in the docs, and the test flow is in the connectors overview.

arcodash runs your SQL through the statement execution API against one SQL warehouse, so you need a warehouse (the HTTP path comes from its connection details tab) and a personal access token. Create the token for a service principal or a dedicated user rather than for yourself, and give that principal read access in Unity Catalog:

GRANT USE CATALOG ON CATALOG main TO `arcodash-reader`;
GRANT USE SCHEMA ON SCHEMA main.analytics TO `arcodash-reader`;
GRANT SELECT ON SCHEMA main.analytics TO `arcodash-reader`;

USE CATALOG and USE SCHEMA only let the principal see into the containers; SELECT on the schema covers current and future tables in it through inheritance. A service principal is named by its application ID in backticks. The principal also needs permission to use the warehouse itself, which you set on the warehouse.

Set the default catalog and schema in the connector if you want unqualified table names to work. Setting a catalog also lets the schema browser read the catalog’s information_schema.columns for full column types. Without a catalog, or on the legacy hive_metastore, arcodash falls back to a coarser table listing.

Network access: the workspace host must be reachable from the internet, and arcodash does not publish fixed outbound IP addresses, so an IP access list on the workspace will block it. A result keeps at most 10,000 rows (see Limits), which a one-row check never approaches.

A freshness check

A check is a scheduled query returning one numeric row. The monitor reads one column of it and compares it with a threshold, as described in data quality monitors.

SELECT timestampdiff(
         SECOND,
         coalesce(max(loaded_at), TIMESTAMP'1970-01-01 00:00:00'),
         current_timestamp()
       ) AS seconds_since_last_row
FROM main.analytics.events

Monitor seconds_since_last_row with > and 3600. timestampdiff counts whole elapsed units in UTC and returns a negative number if the start is after the end. The coalesce handles an empty table, where max() is NULL and the monitor would otherwise show unknown instead of triggering.

Use a load timestamp, not an event timestamp, so a quiet weekend does not look like an outage. The idea is covered in data freshness.

Freshness without scanning: Delta history

For a Delta table you can ask the transaction log instead of the data:

SELECT timestampdiff(SECOND, max(timestamp), current_timestamp()) AS seconds_since_last_write
FROM (DESCRIBE HISTORY main.analytics.events)
WHERE operation <> 'OPTIMIZE'

The docs show DESCRIBE HISTORY wrapped in a subquery and filtered by operation, and they list version, timestamp, operation, operationParameters and operationMetrics among the output. This measures when the table was last written, not when the newest business event happened, so a job that writes zero rows still counts as fresh. Maintenance entries other than OPTIMIZE (vacuum runs, for example) can also appear, so look at your own history and exclude whatever is not a data write. History is kept for 30 days by default.

Do not use information_schema.tables.last_altered for this. It is documented as when the relation’s definition was last altered, which is a schema signal.

A volume check

SELECT count(*) AS rows_last_hour
FROM main.analytics.events
WHERE loaded_at >= current_timestamp() - INTERVAL '1' HOUR

Monitor rows_last_hour with < and a floor. If the table’s volume rises and falls with the day, use a baseline monitor so you do not hand-pick a number. A walkthrough of the logic is in row count anomaly detection in SQL.

A metric alert

Failed payments as a share of today’s attempts:

SELECT coalesce(
         round(100.0 * count_if(status = 'failed') / nullif(count(*), 0), 2),
         0
       ) AS failed_pct
FROM main.billing.payments
WHERE created_at >= current_date()

Monitor failed_pct with > and 5. Send it to the channel that owns billing; the setup is in notification destinations.

Gotchas that matter for monitoring

• The 50 second wait. arcodash submits each statement with a 50 second server-side wait and the option to cancel on timeout. A statement still running or queued at that point is cancelled and the check reports a failure. A stopped warehouse that is slow to start can hit this on the first check after a quiet period.
• Warehouse cost. Every scheduled check is a statement on the warehouse. A short interval plus a long auto-stop keeps it running around the clock. Match the interval to the threshold, and think about a small dedicated warehouse for checks.
• Time zones. current_timestamp() is the time at the start of query evaluation. timestampdiff works in UTC. If your timestamps are stored as local time without a zone, convert them before comparing.
• Views and tables. Checks on a view run the view’s full query on every run. Point freshness checks at the base table.
• Silent path errors. A malformed http_path is not rejected up front, so a bad value shows up as an odd error from the API.
• Result size. Results are capped at 10,000 rows, and a result that arrives as external links instead of inline data is not supported. Keep tile queries aggregated.

To generate freshness, row count, null and duplicate checks for a Databricks table, use the Databricks data check generator.

What a dashboard looks like

Daily volume with a trailing week:

SELECT date_trunc('DAY', created_at) AS day, count(*) AS orders
FROM main.analytics.orders
WHERE created_at >= date_sub(current_date(), 7)
GROUP BY 1
ORDER BY 1

Table freshness for a dashboard status tile, one row per table you care about, using UNION ALL. It is for the dashboard only: a monitor reads just the first row, and UNION ALL does not guarantee row order, so give each table its own monitor query.

SELECT 'events' AS tbl, timestampdiff(MINUTE, max(loaded_at), current_timestamp()) AS minutes_old
FROM main.analytics.events
UNION ALL
SELECT 'orders', timestampdiff(MINUTE, max(loaded_at), current_timestamp())
FROM main.analytics.orders

Table counts per schema and type, from Unity Catalog metadata:

SELECT table_schema, table_type, count(*) AS tables
FROM main.information_schema.tables
WHERE table_schema <> 'information_schema'
GROUP BY table_schema, table_type
ORDER BY tables DESC

The information_schema view only lists relations the principal has privileges on, so the count matches what the token can see, not the whole catalog. Schedule the queries that feed monitors, as described in scheduled refresh. Suggest checks with AI can draft them from your schema, using your own AI key. arcodash is hosted only, with 14 days free and no card.

The Snowflake page has the warehouse-cost advice that applies here as well.

docs

Databricks setup in the docs

Connection fields, an example configuration and caveats.