Skip to content
7 min read

I tested whether AI PR reviews catch cleanup I never asked for

Key takeaways

  • Four synthetic patches produced 48 completed reviews across two configurations.
  • With the issue supplied, both configurations flagged 6/6 scope violations. Opus supplied an explicit remedy in 6/6, versus Cohere’s 2/6.
  • Five of the six relevant Opus reviews also treated permitted focused checks as a required acceptance criterion.
Vibe coding PR review series: Kody and OpenRouter, by Mehdi Zare.

I ran 48 reviews of four tiny code changes: 24 with a free code model and 24 with Claude Opus 5.5 through the Claude CLI.

Both spotted the extra currency-order change in every relevant response. When I supplied the issue, both identified that it violated the requested scope. Opus also gave an explicit remedy in all six relevant reviews. The free model did that in two.

That sounds encouraging. But five of those six Opus reviews also invented a test requirement the issue didn't contain.

This is why I want to evaluate the findings themselves, even when a review reaches the right decision.

The cleanup nobody asked for

After I shared my Kody and OpenRouter setup, u/leo_heo suggested testing a PR that fixes its issue and adds an unrequested cleanup. u/Eliran-Kotzero recommended tracking missed issues and false positives.

I made a small JavaScript example around that suggestion.

The bug is an empty cart. Its total calculation calls reduce without an initial value, so total([]) throws a TypeError. The requested fix adds 0, making an empty cart total zero while preserving the supplied nonempty examples.

The extra cleanup alphabetizes an exported array:

diff
-export const supportedCurrencies = ['USD', 'EUR', 'CAD'];
+export const supportedCurrencies = ['CAD', 'EUR', 'USD'];

That looks harmless until you read the issue. The array feeds a dropdown in its exported order. That order is intentional, and changing it is explicitly outside scope.

I prepared four versions:

See “Local fixture checks” in the comparison tables after the article.

A separate check confirmed that B and C changed the currency order. The cart checks alone couldn't distinguish the authorized fix from the fix with unwanted cleanup.

D was a control: the issue explicitly permits local formatting. A useful reviewer should let that through.

The two review conditions

Each version got three reviews with the diff alone and three with the issue plus the same diff. That is 24 reviews per system, in fresh contexts, with the conditions interleaved.

The first run used Cohere North Mini Code through OpenRouter's `cohere/north-mini-code:free` route, pinned to Cohere with fallback disabled and prices capped at zero. Temperature was 0.2 and the output limit was 4,096 tokens.

The second used Claude Code 2.1.325 with `claude-opus-5-5`, signed in through my Claude Max subscription. I set high effort and a 4,096-token output limit. Every saved answer reported Opus 5.5. Each review ran in a fresh process with tools, integrations and custom instructions disabled, using a neutral system prompt. Claude couldn't read the answer key or run the checks.

The user prompts and fixtures were identical across the runs. The surrounding software and model settings were different: Claude CLI doesn't expose the same temperature control, and its context and reasoning behavior differ from a direct OpenRouter request. These results compare two configurations on this example; they don't isolate the model as the only variable.

I also requested a GLM 5.3 run through OpenCode Go. Its weekly allowance blocked the request before it generated a review. There are no GLM review results in these counts.

The issue, code, prompts, diffs and expected answers were frozen before the original run. The answer key stayed out of both reviewers' inputs. A separate Codex assistant checked every response against that rubric and the executable fixture. There was no independent human adjudication.

What actually happened

All 48 review sessions completed with nonempty final answers and no recorded truncation. The Opus run used 24 distinct sessions, one turn each, with no tools, subagents or model substitution.

See “Completed review results — November 15, 2026” in the comparison tables after the article.

The reorder rows cover B and C. The clean controls are A and D. N/A means the issue wasn't supplied, so definite compliance with its authorization boundary wasn't scored.

The diff-only reviews already noticed that reordering a public array could affect dropdowns, defaults or other consumers. Supplying the issue didn't increase detection in this example. It supplied the evidence needed to say the change was forbidden.

The full rubric also required a code anchor, the issue's scope boundary, a concrete ordering effect, and an explicit remedy: remove the extra change or propose it separately for authorization. Opus supplied that in all six relevant issue-aware responses. Its three reviews of C also correctly identified the surviving empty-cart bug on an unchanged line as an unmet requirement.

One limitation in my experiment: the frozen review prompt asked for a line, severity, failure scenario and reason. It didn't explicitly ask for a remedy. I kept that prompt unchanged for Opus, but I should fix the mismatch before using the stricter rubric as a review gate.

The right verdict can still contain a bad finding

Five of the six Opus reviews of B and C raised missing focused checks as an acceptance-criteria problem. The issue lists adding or running checks under allowed scope. It doesn't require them.

For example, one output added a medium-severity finding called “No focused checks added.” It correctly caught both real problems, then treated permitted work as mandatory.

The packet also didn't contain test results. A reviewer could say test execution was unknown. It couldn't establish from the diff that checks had never run or that existing coverage was absent.

The earlier Cohere run had its own mistakes: incorrect explanations of the original empty-array failure, a clean formatting change blocked over unseen tests, and a review that questioned whether a visible fix existed in the actual file.

Opus handled the frozen empty-cart outcomes correctly in all 12 issue-aware responses and kept review completion separate from merge readiness. Still, correct detection didn't make every sentence reliable. Some diff-only outputs declared a change out of scope, or declared there was no scope creep, while admitting they hadn't seen the issue.

A valid finding should survive scrutiny on its own. A correct overall verdict shouldn't protect unsupported findings from being challenged.

Usage and what I will change next

OpenRouter reported $0 model charges for the 24 Cohere requests. Claude CLI reported about $0.78 at list prices for its 24 reviews. That is an estimate from the CLI, not evidence of an extra bill on my Max plan; the subscription remains a separate cost.

Median elapsed time was 11.9 seconds for the OpenRouter requests and 14.4 seconds for the Claude CLI processes. The latter includes process startup. These tiny-input timings aren't full PR-review times or a fair speed ranking.

My next review-skill revision will ask for:

  • The code anchor, relevant requirement and concrete effect behind each finding.
  • A clear remedy for unauthorized changes.
  • A distinction between a new regression, an old bug left unfixed and a bug already fixed.
  • Test execution marked unknown unless evidence is supplied.
  • Separate statuses for review completion and merge readiness.

Those changes are proposed, not tested improvements. I want to rerun the frozen cases before claiming they help.

This experiment used four tiny synthetic changes, three repetitions per condition and an obvious prohibition in the issue. It didn't exercise Kody's full workflow, my credits-first failover, or real repository context. It doesn't establish a model leaderboard.

For the open-source starter setup I'm building, the useful output is a repeatable case with saved inputs, real responses and visible mistakes. That's something I can use to check whether the next version of my review instructions actually improves the result.

If you've seen an unrequested cleanup survive review, I'd like to hear about it. A redacted example would be a good addition to this test set.

Local fixture checks
VersionWhat changedLocal cart checks
ARequested empty-cart fix3/3 pass
BFix plus forbidden currency reorder3/3 pass
CCurrency reorder without the fix2/3 pass
DFix plus permitted local formatting3/3 pass

Cart checks alone do not establish scope compliance.

Completed review results — November 15, 2026
MeasureCohere: diff onlyCohere: issue + diffOpus: diff onlyOpus: issue + diff
Completed reviews12/1212/1212/1212/12
Currency reorder surfaced6/66/66/66/6
Explicit violation of supplied issueN/A6/6N/A6/6
Full scope rubric, including remedyN/A2/6N/A6/6
False scope findings on clean controls0/60/60/60/6

Reorder rows cover B/C. Clean controls are A/D. N/A: issue absent. Four synthetic patches; three repetitions per condition. Different clients and reasoning settings prevent a model-only causal comparison.

Sources

  1. Claude Opus 5.5 model reference, Anthropic, accessed
  2. Claude Code CLI reference, Anthropic, accessed
  3. Claude Code environment variables, Anthropic, accessed
  4. OpenRouter provider routing controls, OpenRouter, accessed

Part 2 of Vibe coding PR reviews

MZ

Mehdi Zare, CFA

AI and cloud architect

AI and cloud architect. I build and run my own products with AI coding agents, and I help founders and CTOs take theirs from prototype to production, remotely.