Reproduce Before Report

A reviewer is allowed to suspect anything. The reader is allowed to see only what an independent verifier could actually reproduce.

Why this, for you: you've felt the noise — an AI reviewer confidently flags a null-deref that the code can never hit, and you spend ten minutes proving it wrong. That false positive reached you because nothing stood between "the model said so" and your inbox. This lesson installs that gate.

The fix is structural, not a better prompt. Put an independent verifier between the reviewer's findings and the human, and drop anything it cannot reproduce against actual code behavior. A reproduction is harder to fake than a judgment — and that asymmetry is the whole pattern.

1 The gate sits at the reporter boundary

A reviewer raises candidate findings freely — high recall, speculative, cheap. A fresh-context verifier then tries to build a concrete reproduction for each one before it ships. The gate sits exactly where findings would otherwise reach the user.

A reproduce-before-report gate drops any reviewer finding the verifier cannot reproduce against actual code behavior. The reviewer flags freely; the verifier turns soft suspicion into hard evidence — or discards it.

"Reproduce" is concrete, not a second opinion. Given the diff plus the claim ("function X crashes on empty input at line 42"), the verifier must produce one of: an input that triggers the failure, a code path that shows the contradiction, or a file:line citation to the behavior in the source. The canonical bar is a behavior claim needs a citation in the code — not an inference from a variable's name.

2 Independence is the load-bearing property

A verifier that shares context with the reviewer inherits its reasoning errors and rubber-stamps its own false positives. Two dimensions make the verifier independent:

Context freshness — the verifier starts from the diff and the claim, never the reviewer's chain of thought.
Model diversity — LLM judges show documented same-family bias, scoring related model families higher. Same-family verifiers reproduce same-family confabulations.

A reproduction is harder to fake than a judgment. A reviewer can hallucinate a defect with no external grounding; a verifier that must produce a reproduction cannot — it either runs or it doesn't.

The gain is empirical. Multi-agent verification with conditionally independent agents lifted accuracy from 32.8% to 72.4% in CodeX-Verify, with measured reviewer-pair error correlations of just 0.05–0.25 — confirming the agents fail on non-overlapping populations.

3 Silent drop is the default — not a confidence flag

Non-reproducing findings are dropped without a trace, not appended as low-confidence noise. Surfacing them with a "maybe" flag re-creates exactly the noise problem the gate exists to solve.

Fan-out without a gate is the anti-pattern

A pipeline that spawns many reviewers but reports every candidate isn't running this gate — it's raw fan-out, which amplifies false positives instead of filtering them. The verify step is what converts diversity into precision.

Part	Job	Implementation
Reviewer	High-recall candidate generation	Per-class agents (correctness, security, regressions) in parallel
Verifier	Reproduction against actual code	Fresh-context agent given diff + claim; reproduced or dropped
Drop policy	Discard non-reproducing findings	Silent — not surfaced to the user
Artifact	Expose the evidence	"How it verified the problem" reasoning, on demand

# REVIEW.md — make the reproduction bar a per-class dial Behavior claims need a file:line citation in the source, not an inference from naming. Can't cite it? Drop it. Data-flow claims (taint, leakage): cite BOTH source and sink line. Race suspicions: cite the two interleavings, or route to the "unverified-suspicion" channel — never an inline finding.

4 Know when the gate backfires

The gate is for substantial pre-merge changes, not every diff. Watch four failure modes:

Trivial diffs — on a typo fix, the verification cost dominates the false-positive cost. Use a single-pass diff review instead.
No real independence — same model plus same session is a gate in name only; it reproduces the reviewer's confabulations.
Reproduction-incapable classes — races, distributed failures, and intermittent flakes can't be reproduced on a static diff, so silent-drop becomes silent-drop of your highest-stakes findings. Route them to a separate channel.
High-iteration PRs — push-triggered re-verification compounds linearly; suppress it after the first pass.

↪ Your win: evidence, not opinion, reaches the human

Gate at the reporter boundary — reviewers flag freely; the user sees only what survives reproduction.
Make "reproduce" concrete — a triggering input, a contradicting path, or a file:line citation, never a naming inference.
Keep the verifier independent — fresh context, ideally a different model family; it's the load-bearing property.
Silently drop non-reproductions — a low-confidence flag rebuilds the noise the gate removed.
Skip the gate on trivial diffs and route reproduction-incapable classes elsewhere — don't silent-drop the high-stakes ones.

Retrieval practice — recall, don't peek

Question 1A reproduce-before-report gate drops a reviewer finding when…

Question 2The load-bearing property that makes the verifier work is its…

Question 3A finding the verifier cannot reproduce should be…

Question 4Fanning out many reviewers but reporting every candidate is…

Question 5 · spaced recall from an earlier lessonIn the committee pattern, running several specialized reviewers and synthesizing one verdict beats…

Ask me anything. Want a REVIEW.md verification bar tuned to your worst false-positive class, or help deciding which defect classes to route to an unverified-suspicion channel? Next: Human-AI Synergy — splitting review by comparative advantage so agents triage breadth and humans judge intent.