← All posts

Engineering · 2026.10.10 · 6 min

Letting Claude Triage Alerts Without Letting It Change Anything: Read-Only Triage Design and Pitfalls

When an alert fires, a Claude session with read-only tools hunts the root cause and posts to Discord; here's how I drew the permission boundary and two pitfalls: rewritten command paths and Sentry noise.

Reading
Theme
Size
Width

When an alert fires, I first have a Claude session with only read-only tools go find clues, reach a conclusion, and post it to Discord. Whether to change anything is my call. "Can't change anything" is enforced in --allowedTools, not requested politely in the prompt. After launch I hit two pitfalls. The model rewrote command paths on its own and kept getting blocked by the allowlist until it gave up. And Sentry treated self-recovering WebSocket disconnects as errors, so a triage ran for every exception type. I treat triage conclusions as leads only and check them again myself before acting.

At first, this lending platform's exceptions only lived in logs. When a white screen or a production anomaly showed up, I had to SSH into the machine and grep. Later I added three observability tracks. Prometheus, Grafana, and Alertmanager answer whether the system is broken. Sentry answers which user or route hit an unhandled exception. Micrometer Tracing plus OTLP answers where a request's time goes. With an empty DSN, Sentry doesn't initialize on either the frontend or the backend, and when it's on, sendDefaultPii is off and query strings are stripped, because cookies, API keys, and TOTP have no business in a third-party error service. Triage unifies the signals from Alertmanager and Sentry and hands them to Claude to investigate.

Diagnose only, change nothing

--allowedTools grants only Read, Grep, and Glob, plus a read-only ops query script and GitHub queries. No Edit, no Write, no arbitrary Bash. Mechanically, this session can't modify anything.

Why stop at diagnosis? The root cause of an alert might be that deploys came so close together they starved the scheduler, not that the verifier itself was broken. An agent that can fix things on its own would most likely go and change that verifier. Triage keeps a human at the "decide whether to change it" step, which is exactly where machines are least reliable.

Every triage costs real API money, so the same signal is investigated at most once every 6 hours and at most 8 times a day, with mkdir as a lock so only one runs at a time. Late arrivals don't queue, because queueing only produces stale conclusions. Reports follow a fixed format that must include a verdict line. If the verdict line can't be read, the report is treated as needing a fix and an issue is opened. One issue too many is the better mistake.

Flow chart: signals from Alertmanager and Sentry pass cost controls (the same signal at most once every 6 hours, at most 8 a day, an mkdir lock so one runs at a time) before reaching a read-only Claude session whose --allowedTools grants only Read, Grep, Glob, a read-only query script, and GitHub queries, with no Edit, Write, or arbitrary Bash. Reports go to Discord, a missing verdict line opens an issue, and whether to change anything stays a human call.

An issue with a single sentence

One day triage opened an issue whose body was just one sentence: "I need your authorization to run this command."

I dug up the transcript of that session on that machine. The prompt gave the absolute path of the query script, but the model rewrote it as a relative ./scripts/... path, then again with bash in front. The allowlist only recognized the absolute-path form, and after getting blocked 5 times in a row, the model gave up. The sentence it gave up with had no verdict line, so by the rules an issue was opened.

This wasn't a one-off. Of 89 triage sessions, 17 had been blocked at some point. Among the blocks, 10 were relative paths or a bash prefix, and another 28 were output piped to python3.

The fix was to add three equivalent forms to the allowlist: bash plus the absolute path, a relative path starting with ./, and bash plus the relative path. Claude always runs from the repo root, so all of these point to the same read-only script, and no capability was widened. Piping to python3 stays blocked, since that amounts to allowing arbitrary code execution.

Table: the allowlist originally accepted only the absolute path; bash plus the absolute path, a ./ relative path, and bash plus the relative path were added, covering 10 blocks; piping to python3 accounted for 28 blocks and stays blocked because it means arbitrary code execution. Of 89 triage sessions, 17 had been blocked.

I also added two sentences to the prompt: type commands with exactly the path given and don't pipe them, and if evidence can't be obtained, say so plainly under "Issues" and still give a verdict in the fixed format. I originally wanted to write "if you get blocked, retype the command, don't ask for authorization," but the Claude Code auto mode classifier flagged that sentence as instruction poisoning and blocked it. Looking back, that's fair. Telling an agent not to ask a human when it gets blocked really does look like teaching it to bypass permissions.

Cutting noise at the source

What triggered that issue was a network blip on the connection to Bitfinex. REST timed out first, the WebSocket disconnected, reconnecting timed out once more, and everything was back within 30 seconds.

The WebSocket client logged connection failures and onError with log.error and the exception attached, and Sentry ingests logs at error level and above, so every network exception type opened its own Sentry issue and ran its own triage. Over two days the WebSocket disconnected 14 times, and it recovered on its own every time.

The fix was to downgrade those two spots to log.warn and log only ex.toString(), without the stack trace full of JDK internal frames. The line for synchronous connection construction failures stays at ERROR, since that one is more likely a code or config problem. If the connection really stays down for a long time, the "disconnected for over 1 minute" alert is the backstop. I chose not to set ignore rules on individual Sentry issues, because new exception types (SSL handshake, DNS) keep spawning new issues, and each one would burn a triage.

Before and after: WebSocket disconnects from network blips were logged with log.error and the exception attached, and since Sentry ingests error level and above, every exception type opened its own issue and triage, with 14 self-recovering disconnects in two days. Now connection failures and onError log at warn with only ex.toString() and stay out of Sentry, synchronous construction failures stay at ERROR, and a disconnected-for-over-1-minute alert is the backstop.

The new allowlist can only be verified the next time the model rewrites a path, and the blocked python3 pipes haven't been dealt with, so triage can still come back incomplete.

Conclusions are only leads

Later the reconciliation alert fired twice, and the triage report was wrong both times. The first time it correctly guessed which new lane was involved but got wrong where the math broke. The second time it was wrong even about what was broken. Following its suggestions would have meant changing code that wasn't broken, both times. The investigation is written up in a separate post on reconciliation.

When a triage conclusion looks wrong, my first step is to open that session's transcript. Every tool call and every reason for a block is in there, which is much faster than guessing what's wrong with the prompt.

What I learned

  • "Can't change anything" belongs in permissions, not in the prompt. The model will rewrite commands, but it can't rewrite its way into tools it doesn't have.
  • An allowlist should cover the equivalent forms of the same capability, but every time you add one, ask whether it widens the capability.
  • When writing prompts for an agent, describe the right way to do things. Don't teach it how to get around a block.
  • Cut noise at the source. Transient errors are warnings, and persistent states belong to alerts.
#Sentry#Alerting#Observability#Claude Code

Letters

Loading comments…

Next

Engineering

An AI Quoting On Chain Every 300ms: Measuring Gas Costs and Margin Accounting

Running an AI trading demo on Monad mainnet showed requoting every block costs ten times more gas than the spread earns; I also fixed a margin bug reverting 83% to 95% of orders.