Data incident: what it is and how it's handled

The lifecycle

Every data incident passes through the same three stages, whether or not you name them.

• Detect. Something notices. Ideally a scheduled check, such as freshness or a row count, fires. Otherwise a stakeholder notices a strange number, which is the slow and embarrassing path.
• Triage. Decide whether it is real, how bad it is, and who owns it. Most triage time goes to one question: did the data change, or did the check? A threshold set too tight produces false alarms, and those should be fixed rather than worked.
• Resolve. Fix the cause, backfill if needed, and confirm the check clears. Then ask the follow-up question: would an earlier or sharper check have caught this sooner?

The useful numbers are time to detect and time to resolve. You can only improve what you log.

How arcodash tracks them

A monitor in arcodash is a threshold check on one column of a saved query’s result. An incident is the stretch where that monitor is triggered. Going from ok or unknown into triggered starts a new incident and notifies the people subscribed. Notifications fire on state changes, so a monitor that stays triggered does not repeat itself.

From there, the documented actions map onto triage:

• Acknowledge says “I’ve got this” and stops further escalation steps. It does not mute the monitor. If the monitor clears and triggers again, that is a new incident and the acknowledgment is cleared.
• Snooze silences notifications for a set number of minutes, then lifts itself. Mute silences them until you unmute.
• Run now forces an immediate check, which is how you confirm a fix without waiting for the next schedule.
• Delivery history lists the last 50 notification attempts per monitor, so you can see whether an alert actually went out.

If an AI provider is configured, a triggered alert can also carry an AI root-cause note. See alerts overview for what that sends and its limits.

Measure your own incidents

Keep a simple log, even a spreadsheet or a table, and query it. This computes average time to acknowledge and resolve per monitor over 30 days, and counts incidents still open.

SELECT monitor,
       count(*)                                         AS incidents,
       count(*) FILTER (WHERE resolved_at IS NULL)      AS still_open,
       avg(acked_at - opened_at)                        AS avg_time_to_ack,
       avg(resolved_at - opened_at)                     AS avg_time_to_resolve
FROM incidents
WHERE opened_at >= now() - interval '30 days'
GROUP BY monitor
ORDER BY incidents DESC;

avg ignores NULL, so open incidents do not drag the resolve time down to zero, but they also vanish from it. That is why still_open sits next to it. A monitor with a great average and three open incidents is not doing well.

The monitor that opens the most incidents is rarely the one to fix first. Look for the one that fires often and gets acknowledged without action. That is a threshold problem, covered under alert thresholds.