Agent-First Incident Response
Replace hand-set alert thresholds with anomaly detection and an agent on first response.
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 7
- Confidence
- 72%
Most engineering organisations still detect problems with thresholds a human guessed and typed in by hand. This framework replaces the guess with anomaly detection and puts an agent on first response. Any metric that moves oddly, latency, throughput or error rate, fires an anomaly signal rather than a pre-set alarm. An agent picks it up, investigates, and decides whether it warrants an incident. If it files one, humans are looped in and the agent has already begun assembling remediation, so engineers receive a proposed solution rather than a blank page. The deliberate boundary is write access: agents investigate and propose, humans still hold the tools that change production. The same pattern runs offline as a batch, pointing thousands of concurrent agents at your codebase on a schedule for security research and optimisation work that no human team could realistically staff.
Origin
Extracted from Naval, where Guillermo Rauch describes how Vercel automated much of its site reliability engineering and security research with autonomous agents.
Core principles
- 01Hand-typed alert thresholds are guesses that age badly.
- 02The expensive part of an incident is the blank page, not the fix.
- 03Agents should investigate and propose; humans keep write access to production.
- 04The same pattern runs offline as a scheduled sweep, not only live.
- 05Compute is cheaper than the security and optimisation headcount you cannot hire.
How to run it
- 1
Instrument broadly instead of choosing alarms
Stop deciding in advance which metrics deserve a threshold. Capture everything meaningful, then let statistical anomaly detection decide what is unusual.
Pro tip Include deploy events and config changes in the same stream so the agent can correlate them.
Watch out Hand-tuned thresholds encode last year's failure modes, not this year's.
- 2
Fire on anomaly, not on threshold
Any metric that slows down, speeds up or shifts throughput unexpectedly emits an anomaly signal into a queue an agent consumes.
Watch out Expect a noisy first month. Tune the detector, not the agent.
- 3
Give the agent investigative context
The agent needs read access to logs, traces, recent deploys and prior incidents so its investigation is grounded rather than speculative.
Pro tip Feed it past incident write-ups; they are the highest-signal training context you already own.
- 4
Let the agent triage and escalate
The agent decides whether the anomaly warrants an incident. If it files one, humans are looped in at that point rather than at the raw alert.
Pro tip Track the agent's false-positive and missed-incident rates as an explicit quality metric.
Watch out Escalation policy is a judgement call. Review the agent's threshold monthly.
- 5
Serve remediation on a platter
By the time a human joins, the agent should have assembled the likely cause and a proposed fix, so the engineer is verifying a diagnosis rather than starting one.
- 6
Hold the line on production write access
Deliberately withhold the tools that let the agent change production. Do everything up to the change, and keep the mutation behind a human.
Pro tip Revisit this boundary as your evaluators mature, not as your confidence grows.
Watch out This is the boundary that keeps an agent-driven investigation from becoming an agent-driven outage.
- 7
Run the same pattern as a scheduled sweep
Point large numbers of concurrent agents at your codebase periodically for security research, red teaming and optimisation, treating it as recurring infrastructure rather than a one-off audit.
Pro tip Budget it in tokens against the headcount it replaces, then run it on a cadence.
Watch out Adversaries are running the same sweeps. A single pass is not a posture.
In the wild
Rather than an engineer configuring an alarm for an error-rate increase at one API endpoint, every metric is watched for anomalies. When one fires, an agent investigates, decides whether to open an incident, and if it does, people are looped in while the agent has already started assembling remediation. Everything short of changing production is automated.
→ Engineers receive a diagnosis and a proposed fix instead of a raw alert and a blank page.
Vercel ran its open-sourced security research tool against its entire monorepo using ten thousand concurrent agents in the cloud. The sweep surfaced what the team judged to be several quarters of security research progress, compressing months of red teaming across entire teams of people into a couple of days.
→ Several quarters of security research progress in roughly two days for about $14,000 of tokens.
Common mistakes
Keeping hand-set thresholds alongside
Running both systems means the old alarms keep paging humans first, so the agent loop never gets the reps it needs to be trusted.
Granting production write access early
An agent that can change production before your evaluators are mature turns a false-positive investigation into a self-inflicted outage.
Treating the security sweep as an audit
A single agent sweep produces a report that ages immediately. The value comes from running it periodically as the codebase and the adversaries both move.
Is it for you?
Best for
Platform and infrastructure teams with real observability data and more surface area than their on-call rotation can cover.
Not ideal for
Small products with little traffic, thin telemetry or no on-call function, where the instrumentation cost exceeds the benefit.
From the transcript
“most engineering organizations are responding to this by setting up alarms or uh like monitoring thresholds by hand which is pretty insane but that's actually…”
“any metric that uh slows down speeds up uh throughput changes whatever fires off an anomaly alert an agent investigates that an agent can decide…”
“using 10,000 concurrent agents in the cloud and it found basically several quarters worth of security research uh progress was made in um basically a…”
From the episode
Full Episode: The AI Industrial Revolution