← All posts

Tools · 2026.10.10 · 5 min

GPT as a Read-Only Reviewer for Claude Code: Adversarial Review With Pi

I wrapped Pi, connected to GPT models over an API, as a read-only reviewer Claude Code can call, and found out why permission rules can't stop file writes and how adversarial review converges.

Reading
Theme
Size
Width

I use Pi, a terminal coding agent, connected to GPT models through an API, and wrapped it as a read-only reviewer that Claude Code can call. Two conclusions came out of it. Permission rules are not a sandbox: if you really want read-only, don't give it any tool that can write in the first place. And for adversarial review to converge, you have to tell the reviewer what's in scope and how the previous round's findings were handled. In its first real use, it caught a regression that Claude had written and then missed in its own review.

When Claude reviews code it just wrote, it mostly catches things it would have thought of anyway, so I wanted a model from another company as a second opinion. There was one requirement: the reviewer can't modify files. Whether and how to change anything is up to me and Claude.

To use it, I say "have pi review this" or type /pi-review in Claude Code, and Claude hands Pi the uncommitted changes, a branch, a PR, or a plan document. When the findings come back, Claude opens the code and checks each one, marks it as agree, disagree, or can't confirm, and then reports to me.

Flow chart: I say "have pi review this" or type /pi-review in Claude Code; Claude hands Pi the uncommitted changes, a branch, a PR, or a plan document; Pi, running GPT models through an API, reviews read-only and returns findings; Claude opens the code, marks each finding agree, disagree, or can't confirm, and reports to me. Whether and how to change anything is up to me and Claude.

Only two packages installed

  • earendil-works/pi-review: the official /review, with findings graded P0 to P3
  • @gotgenes/pi-permission-system: allow / ask / deny rules for tools, bash commands, and paths

Ones I looked at but didn't install: pi-web-access uses Exa search by default, which sends code snippets to an outside service, and pi-simplify edits files directly. I also considered wrapping Pi as an MCP server for Claude Code to call, but the only pi-mcp-server I found is no longer maintained, and the common approach in the community is to just shell out.

Pi automatically reads the project's CLAUDE.md and AGENTS.md, so reviews apply the same project rules without me spelling them out again.

Permission rules are not a sandbox

The rules: edit and write are always denied. Bash defaults to ask, with only read-type commands plus git diff/log/show/blame and gh pr diff/view allowed. .env, .ssh, .aws, and Pi's own auth.json are blocked even from reading. In a test repo I confirmed the write tool was hidden and touch was blocked.

But that's a rule layer, not a sandbox. In interactive mode, an allowed read command combined with a redirect, like cat a > b, can still write a file. The first time I used it for real, I started pi by hand without --tools. Pi had bash, actually ran rg and git diff, and the permission rules were the only thing in the way.

Since then, the skill always starts Pi with -na --tools read,grep,find,ls, so there's no bash at all. The diff is written to a file first and Pi reads it with read, and the permission rules drop back to being a second layer of insurance. When I asked it directly which tools it had, it listed only read, grep, find, and ls.

Comparison: before, pi was started by hand without --tools, had bash, and only the permission rules stood in the way, though an allowed read command plus a redirect could still write a file; after, the skill always starts Pi with -na --tools read,grep,find,ls, so it has only read, grep, find, and ls with no bash, and the permission rules become a second layer of insurance.

There are two ways to run it. Inside herdr (a terminal multiplexer for coding agents), it opens a Pi panel on the right, so I can see what it's reading, and the session sticks around for multiple rounds of back and forth. The other is headless pi -p, with the diff fed in on stdin.

Getting adversarial review to converge

The prompt to Pi is adversarial. It has to verify whether the design the author claims actually holds up, and attack spots like edge-case inputs, rolling deploys, and transaction boundaries. If nothing is wrong, the first line has to say "No findings that need fixing", so Claude can tell from the first line whether another round is needed. Only when I explicitly say to fix things does it go into the fix loop, and code fixes need new tests plus mutation checks.

The first real run took three rounds. Round one caught a regression Claude had introduced: on write, it filled in fields with old values it had read earlier, which would overwrite values someone else had just changed. Claude hadn't seen this in its self-review.

Converging wasn't that smooth. At first every round turned up problems in existing mechanisms that had nothing to do with this change. It only settled after I added three requirements to the prompt: mark issues that already exist on the base branch as "pre-existing", report only issues within the scope of this change, and include a list of how each finding from the previous round was handled. The result: round one had 2 High and 1 Medium, round two had 1 needing confirmation and 2 Medium, and round three came back with "No findings that need fixing".

Figure: three requirements added to the prompt, marking issues already on the base branch as pre-existing, reporting only issues within the scope of the change, and including how each previous finding was handled. In the first real run, round one had 2 High and 1 Medium and caught a regression Claude had introduced, round two had 1 needing confirmation and 2 Medium, and round three came back with "No findings that need fixing".

Code comments written while fixing findings only explain why the code is written the way it is now. They don't say "raised by pi" and don't describe the old version. The review history goes only into my own notes, never into docs that get committed.

What I learned

Permission rules block most cases, but they aren't a sandbox and only work as a backup. If you want read-only, take the write tools out of the launch arguments.

Without the scope and the previous round's dispositions, adversarial review keeps picking at pre-existing issues unrelated to the change and never converges.

A model from another company really does see things self-review misses. The regression in round one is the example.

#Code Review#Claude Code#GPT#Pi

Letters

Loading comments…

Next

Engineering

Speeding Up Rust and Next.js CI on a 2 vCPU GitHub Runner

After moving the backend to Rust, the PR backend job went from 11.5 minutes to 1.5 and frontend checks from 81 seconds to 30, mostly through test databases, bcrypt, and caches that broke silently.