More AI Checks Can Still Repeat the Same Mistake

Anthropic’s AI reviewer can find bugs, but its own workflow shows why safe code still depends on layered…

The short version

Claude Code Review is an advisory bug finder, not an autonomous approval system or proof that a change is safe.
Anthropic combines parallel agent review with CI, risk-tiered human approval, staged releases, monitoring and rollback.
Teams should pilot it on lower-stakes repositories, measure missed defects and noise, and preserve human approval for sensitive code.

Anthropic says Claude writes about 90% of Claude Code. Apparently the machine writes the product, reviews the product, and the engineers spend Friday choosing an aperitivo.

Cute story. That figure measures authorship. It proves neither autonomy nor safety. Claude Code Review is worth testing as an advisory bug finder, provided it sits behind strong tests, risk-based approval rules, controlled releases and fast rollback. Remove those layers and you have automated confidence with a GitHub badge.

The launch still matters. Coding agents have made code absurdly cheap to produce, while human attention remains stubbornly finite. Every productivity breakthrough eventually moves the bottleneck somewhere less convenient. Faster coding means more changes waiting for review. Anthropic’s answer is to spend human judgment according to blast radius, automate the repeatable reading, and assume defects will escape anyway.

Buying Claude Code Review gives you one useful piece of that machine. The uncomfortable part, where somebody owns the result, remains yours.

Claude leaves comments; your team owns the merge

An organization owner installs the Claude GitHub App, selects repositories, and chooses when reviews run: when a pull request opens, after pushes, or only when requested. Administrators can also choose Quick, Standard or Deep review by repository. Those levels trade cost and latency for inspection depth, although Anthropic has published no comparable miss rates between them. I would never translate “Deep” into “safe.” It is a purchasing option, not a safety certificate with a tiny gold sticker.

Most managed reviews now finish in 6–10 minutes, down from about 20 during the research preview, according to Anthropic’s general-availability announcement. That makes the service roughly three times faster under Anthropic’s comparison, although the company describes this as typical performance rather than a guarantee. Faster feedback genuinely matters. A comment arriving while I still remember the code can change the pull request; one arriving after dinner becomes tomorrow’s archaeological dig. Speed says nothing about accuracy, however, and Anthropic has not published production-scale precision or recall.

The review pipeline is more interesting than the headline. Several agents inspect the diff and surrounding code in parallel, each producing candidate defects. A verification stage checks those findings against code behavior and filters false alarms before anything reaches GitHub. The survivors are deduplicated, ranked by severity, and attached to the relevant lines, while the service posts a neutral check run alongside them. This should create less noise than one model blurting out every suspicion after reading half a file. But verification can only examine issues the agents found in the first place. A missed bug never enters that funnel, so an empty report provides evidence for a merge decision without proving the change is correct.

Anthropic makes the boundary explicit in its Code Review documentation: the review does not approve or block a pull request. Teams can turn severity counts into required checks through their own branch rules, but that policy belongs to the team. Claude is the bug scout. The carabinieri remain on your payroll.

The boundary that matters more sits between reading code and running it. The managed reviewer reads code and context, while executable tests observe behavior. A model may spot a race condition absent from the test suite. A failing build may expose an assumption shared by every review agent. I want both, especially when the author and reviewer came from the same model family and may have learned the same charmingly incorrect idea.

Anthropic prices the risk before spending human attention

Anthropic’s internal scale explains why manual line-by-line review stopped working. The company says changes landing in its monorepo rose from about 2,500 per day in May to more than 30,000 in September. Claude Code lives in a separate repository, so that larger number gives context rather than direct product throughput. Either way, nobody can thoughtfully read every line when code arrives like luggage at Fiumicino in August.

The Claude Code team described the trade clearly:

We didn't remove review. But we did move most of the reading to machines, optimized for human time and attention, and invested heavily in making it fast and easy to undo mistakes.

Their workflow begins before the pull request. Claude must run executable evidence such as tests, a build, or screenshot comparisons, then a separate agent reviews the diff with fresh context. That second pass matters because the authoring session can otherwise repeat its original mistake with magnificent confidence. Once a pull request exists, required CI runs alongside an AI bug review and a separate security pass. Unresolved findings block the internal merge, after which a risk classifier assigns zero, one or two human approvers. Sensitive code paths can still require a person regardless of the classifier, and an agent approval never fills a human seat. A merge queue reruns checks against the latest main branch because two healthy changes can become mildly homicidal when combined.

The workflow continues after merge. Feature flags limit exposure, releases reach internal users first, and monitoring can trigger rollback when earlier layers fail. This is the causal chain that makes reduced human reading plausible: executable work catches routine mistakes early, fresh-context review challenges the author, parallel agents filter likely bugs, and risk tiers reserve people for changes with consequences. Recovery controls handle whatever survives. No single reviewer needs papal infallibility.

Anthropic’s team put the earlier shift more vividly:

Now, review runs continuously, more like spellcheck while you write.

I used to treat rollback as the apology you prepare after review fails. I was wrong about that. In an AI-heavy workflow, rollback belongs inside the review architecture because several automated checks may preserve the same bad assumption. A row of green lights can indicate five independent validations, or one misconception wearing five hats. Founders learn this shortly after the first deployment that passes CI and sets something expensive on fire.

Glossy green check-mark helmet split open to reveal a charred interior against a vivid yellow background.

Anthropic’s numbers justify a pilot, not blind trust

From June through September, Anthropic says merged pull requests to the Claude Code codebase more than doubled while the share later reverted fell by over 40%. That pairing is useful because throughput improved while one bad-outcome measure declined. The source does not publish raw pull-request counts or absolute revert rates, and it cannot isolate the contribution of Code Review. During the same period, the team also changed its tests, risk routing and release controls. Anyone crediting the reviewer alone is doing attribution by vibes.

A proper pilot has to follow cost through the workflow. I would measure whether developers fix Claude’s comments, dismiss them as noise, or waste time proving them irrelevant. Then I would watch escaped defects and production cleanup, since saved review time can quietly reappear as incident work. Review latency matters because slow feedback interrupts shipping, while repeated review-and-fix loops can let two agents polish each other’s output forever. The hardest metric is false negatives. Teams usually discover a missed bug only after a user does, at which point the clean review report has aged like supermarket mozzarella.

The strongest counterargument comes from the Last Secure Code Benchmark. Its authors tested coding agents on 450 tasks derived from 150 real vulnerabilities. Functionality alone passed on 74%–91% of tasks across the agents, yet the strongest agent produced code that was both functional and secure only 25% of the time. The benchmark evaluates code generation rather than Claude Code Review, so it cannot reveal this service’s miss rate. It does show how easily working code can still be unsafe. Functional tests plus an AI thumbs-up do not establish security.

Anthropic itself warns that Code Review may miss bugs or vulnerabilities and that a result with no findings does not mean a pull request is error-free. That is the honest position. Independent production-scale measurements of precision, false positives and missed security defects still do not exist. We also lack evidence that outside customers can reproduce Anthropic’s combination of higher merge volume and fewer reverts.

The nastiest unknown is correlated failure. AI-written tests, review agents and risk classifiers may all fail on the same change for the same underlying reason. Nobody has published comparable deployment data that answers that question. Until they do, I will treat clean reviews as useful evidence and autonomous approval as a science experiment conducted near production.

Start with a risk policy and a bill you can explain

I would pilot Claude Code Review on one or two representative, lower-stakes repositories. Manual triggering or once-per-pull-request review makes more sense initially than running a fresh billable review after every push. Every-push mode is excellent if my strategic objective is helping the invoice achieve consciousness.

Anthropic says an average review costs around $15–25, with no flat per-review price. Usage depends on pull-request size, codebase complexity and how many findings require verification. That range applies to one review, billed through usage credits, under Anthropic’s observed average. A team pushing frequently can multiply that cost very quickly. We still lack an independent comparison showing how Quick, Standard and Deep differ in bugs caught per dollar.

My minimum policy starts with deterministic path rules. Changes touching authentication, payments, permissions, database migrations or production infrastructure keep human approval. An agent may raise the assigned risk, but it cannot lower the floor set by those rules. Required tests run separately from the AI review, and the default branch stays protected so a proposed change cannot quietly weaken the rules judging it. Every release also gets a documented escape route, even if that route is a feature flag and one slightly sweaty engineer clicking “disable.”

Repository instructions deserve similar care. Anthropic supports project context in `CLAUDE.md` and review guidance in `REVIEW.md`, including severity calibration and rules for generated files. Those policies should live on the protected base branch. Letting an agent rewrite the constitution five seconds before trial feels a touch too Roman.

By the end of 2027, I expect “Did a human read every line?” to disappear from serious engineering conversations. It was always a clumsy proxy for safety. Teams will instead need concrete answers about who classified the danger, what independent evidence they demanded, and how quickly the change can be undone.

If those answers are vague, Claude did not remove the risk. It merely reviewed the incident before everyone else.

Frequently asked questions

Is Claude Code Review generally available?

Claude Code Review is generally available on Anthropic’s Team and Enterprise plans. Organization owners install the Claude GitHub App, select repositories, and choose when reviews run. The managed service comments on suspected defects and posts a neutral check, but it does not independently approve or block pull requests.

How does Anthropic use AI to review Claude Code changes?

Anthropic’s workflow has multiple agents inspect a change and its surrounding code, then verifies, deduplicates and ranks candidate defects. Required CI and a separate security pass run alongside review. A risk classifier assigns zero, one or two human approvers, while sensitive paths can always require a person.

How much does Claude Code Review cost?

Anthropic says an average Claude Code Review costs around $15–25, billed through usage credits, with no flat per-review price. The total depends on pull-request size, codebase complexity and how many findings need verification. Frequent every-push reviews can multiply the bill quickly, and independent cost-effectiveness comparisons are unavailable.