Responding to an incident

An incident means one of four detectors fired. Which one tells you almost everything about what to do next.

Triage by trigger

unrecovered_execution, drop everything

This is the only one that is an emergency. An action ran, its postcondition failed, and the rollback also failed. Nobody knows what state the system is in.

1. Read the evidence events to find what ran and what the postcondition checked.
2. Verify the actual state by hand. Do not trust the agent's report, the reason this is an incident is that reporting broke.
3. Restore or roll forward deliberately, by a human.
4. Only then resolve the incident, with what you actually found in the note.

break_glass_override, a governance question, not a technical one

The override is already audited as critical and carries a reason. The work here is deciding whether the reason was good.

If it was, the rule needs a narrower scope or a time window. If it was not, that is a conversation with a person, not a policy change.

repeated_blocks, usually the rule, sometimes the agent

Five blocks in thirty minutes. Look at whether it is the same rule each time.

• Same rule, legitimate work → the rule is too broad. Narrow it.
• Same rule, work the agent should not be doing → the agent's capabilities are too wide. That is the fix, not the policy.
• Different rules, escalating → treat as probing. Suspend the agent and investigate what is driving it.

memnox agents suspend <id>

critical_risk_score, read the factors

Scoring is deterministic, so the explanation is the calculation. A tainted session plus a novel action plus first-production is a very different story from a burst on a routine action.

Then: what did it reach?

memnox graph explain src/utils/money.ts

The blast radius answers what a change to a file can actually affect, through imports. This is the difference between "it edited a utility" and "it edited a utility that payments imports".

Then: what else did that session do?

memnox replay <sessionId>

Every decision in one agent session, in order. An incident is rarely a single event, it is one event that got noticed inside a session worth reading whole.

Check whether the session was tainted. A tainted session that reached a privileged action is a different investigation: the question becomes what did it read, and who wrote that?

Resolve, with a note somebody will read

Write what was actually true. The note is read during the next incident of the same shape, by somebody who was not here.

Bad: fixed. Good: Deploy job used the shared agent token, which carries deploy.*. Split into its own agent with staging-only capabilities. Rule was correct.

Finally: make it not happen again

The fix is almost always upstream of the incident:

Then record the decision, so the reasoning survives the people who were on the call:

memnox memory add "Deploy agents carry staging-only capabilities"

Write it the way somebody would search for it later. It is matched as text, and from then on it escalates any action that contradicts it. See Decisions and memory.