Reference · Canonical Language

Agent-Assisted Code Review — Glossary

The working vocabulary for this course. Once a term lives here, every lesson uses this word for it. Grows as we go.

Volume, Value, and Signal

Merge rate: The share of opened PRs that actually land — the course's outcome metric, not PR count, because agents can multiply volume while merge rate falls. CRA-only review merges at 45.2% against 68.4% for human-only review, a 23-point gap.; Avoid: treating PR count or merge success as proof of value — post-merge code quality can still be poor.; Source: cra-merge-rate-gap · agent-pr-volume-vs-value · Lesson: volume-vs-value
Acceptance rate: The fraction of an author's PRs that reviewers accept. Across the AIDev dataset agents trail the human baseline (~77%) by 13 to 42 points, and speed does not close the gap. Documentation is the one task type where agents beat humans.; Source: agent-pr-volume-vs-value · Lesson: volume-vs-value
Code-review agent · CRA: An autonomous agent that reviews PRs and posts comments. Deployed alone as a human substitute it underperforms — CRA-only review is a credibility gap, not a like-for-like replacement. Adding one human reviewer recovers most of the merge-rate deficit.; Source: cra-merge-rate-gap · Lesson: volume-vs-value
Signal ratio: The fraction of a CRA's comments that are actionable, from 0.0 (all noise) to 1.0 (all signal). The single best calibration metric: below ~60% actionable, PRs stall and get abandoned. Twelve of thirteen studied CRAs fell under that line.; Source: cra-merge-rate-gap · Lesson: signal-over-volume
Adoption rate: How often a reviewer's suggestions are actually applied — 16.6% for AI versus 56.5% for humans across 278,790 reviews. The gap is a design input, not a defect: over half of unadopted AI suggestions were wrong or superseded by the developer's own fix.; Source: human-ai-review-synergy · Lesson: human-ai-synergy
Alert fatigue: The learned response where a reviewer discounts all of a tool's output — including its rare critical findings — after repeated low-value comments. The primary failure mode noisy AI review creates, and the reason silence is a feature, not a gap.; Source: signal-over-volume-in-ai-review · Lesson: signal-over-volume
Signal over volume: The design principle that AI review should stay silent when it has nothing high-confidence to say. 29% of Copilot reviews intentionally produce no comments; as GitHub puts it, "more comments don't necessarily mean a better review."; Avoid: maximizing comment frequency — it trains reviewers to ignore the tool, burying the one finding that mattered.; Source: signal-over-volume-in-ai-review · Lesson: signal-over-volume
Confidence floor: A minimum certainty bar — the canonical example is ≥90% — each finding must clear before it surfaces. Findings below the floor are suppressed, not queued. Set too strict, it silences rare-but-real defect classes that look low-confidence only because they are rare in training data.; Source: signal-over-volume-in-ai-review · Lesson: signal-over-volume
Silent drop: Discarding a finding entirely rather than appending it with a low-confidence flag. The default for both signal-over-volume suppression and the reproduce-before-report gate — a flagged "maybe" re-creates the noise the discipline exists to remove.; Source: signal-over-volume-in-ai-review · reproduce-before-report-verification-gate · Lesson: signal-over-volume
Cyclomatic complexity: A measure of a change's structural difficulty — the count of independent paths through the code. Agent PRs alter it far less often than human PRs (9.1% vs 23.3%), so raw volume overstates how hard the work was; adopted AI suggestions also push it up where humans simplify.; Source: agent-pr-volume-vs-value · Lesson: volume-vs-value
Verbosity gap: AI reviewers emit ~29.6 tokens per line of code reviewed against 4.1 for humans — a 7x difference — and over 95% of AI comments target only defects and improvements. Unconstrained, the gap is alert fatigue by another name.; Source: human-ai-review-synergy · Lesson: human-ai-synergy

The Security Gap

Security review gap: The systematic failure of human-PR review heuristics to catch AI-specific security defects: 52.4% of agent-authored security PRs merge, only 1.8% of rejections cite distrust of AI code, and commit-message quality — a strong human-PR signal — has zero predictive value for agent PRs.; Source: security-review-gap-in-ai-prs · Lesson: signal-over-volume
CWE-1333 · ReDoS: Inefficient regular-expression complexity — a regex whose backtracking blows up to polynomial or exponential time on crafted input, a denial-of-service vector. The single most common weakness in agent security PRs (36.2%), because LLMs reproduce backtracking-prone patterns from their training corpus.; Source: security-review-gap-in-ai-prs · Lesson: signal-over-volume
Severity routing: Classifying each finding as critical, high, medium, or low and surfacing only the top tiers as PR comments. Keeps the one SQL-injection finding from drowning under twenty style nits; the merge gate fires only on action-required findings.; Source: signal-over-volume-in-ai-review · tiered-code-review · Lesson: signal-over-volume

How an Agentic Reviewer Works

Agentic review: Review that replaces static diff analysis with a tool-calling loop — the reviewer reads files outside the diff, traces dependencies, and reads linked issues before commenting. It judges how a change fits the wider codebase, not just line-level syntax. The cost is latency: GitHub traded an 8.1% feedback gain for slower reviews.; Source: agentic-code-review-architecture · Lesson: anatomy-of-agentic-review
Tool-calling context gathering: The mechanism that makes agentic review possible — the reviewer calls file-read, search, and dependency-trace tools to assemble project context on demand, rather than receiving only the diff. Give the reviewer tools, not just data.; Source: agentic-code-review-architecture · Lesson: anatomy-of-agentic-review
Hybrid LLM-plus-deterministic analysis: Pairing LLM judgment (architectural fit, pattern recognition) with deterministic tools (CodeQL, ESLint, SAST, static refactor detectors) so each does what it is best at. The deterministic leg supplies reproducibility the LLM cannot; structure-aware labeling makes the same call by reserving static analysis for the categories that matter most.; Source: agentic-code-review-architecture · structure-aware-diff-labeling · Lesson: anatomy-of-agentic-review
Strategic review planning: Having the agent map its review strategy across a large PR before reading individual files, so early discoveries are not "forgotten" by the time it finalizes. The fix for context loss on long diffs — the agent catches problems as it reads rather than at the end.; Source: agentic-code-review-architecture · Lesson: anatomy-of-agentic-review
Diff-based review: Reviewing what changed, not the full artifact — mistakes live in the delta, and review effort scales with what you read. Defect detection peaks at 200–400 lines and drops sharply beyond. Design pipelines so review lands at PR and checkpoint boundaries.; Source: diff-based-review · Lesson: anatomy-of-agentic-review
Cross-file blind spot: The defect class diff-only review cannot see — a one-line change that breaks a downstream consumer, an API misuse, a violated global invariant. The reason a strict confidence floor or pure diff scope misses real bugs, and why agentic context-gathering exists.; Source: diff-based-review · agentic-code-review-architecture · Lesson: anatomy-of-agentic-review
Hunk: A contiguous block of changed lines within a diff — the unit a structure-aware labeler classifies and the granularity at which findings attach.; Source: structure-aware-diff-labeling · Lesson: structure-aware-diffs
Change taxonomy: A fixed label set of change types — the canonical version has twelve: rename, logic change, error handling, documentation, style change, and so on — applied per hunk so a reviewer can route mechanical categories away from substantive ones.; Source: structure-aware-diff-labeling · Lesson: structure-aware-diffs
Two-stage labeling pipeline: Label each hunk with few-shot LLM classification (the Labeler), then run one whole-patch pass (the Refiner) to resolve cross-hunk relationships. It trades determinism and cost for polyglot coverage; the paper itself recommends a static-analysis hybrid for the categories where accuracy matters.; Source: structure-aware-diff-labeling · Lesson: structure-aware-diffs
Refiner: The second stage that sees the whole patch at once and links isolated labels into structural patterns — which hunk declares a renamed symbol and which hunks consume it. Knowing four hunks are renames is less useful than knowing they all consume one declaration.; Source: structure-aware-diff-labeling · Lesson: structure-aware-diffs

Routing Review Effort

Tiered review · risk-based routing: Treating review as a risk-routing problem — AI reviews everything, non-critical code (tests, docs, config) merges on AI-only review, and critical paths (auth, payments, crypto) escalate to mandatory human review. Only as good as the classification that routes each PR.; Avoid: assuming a path pattern catches every security change — a migration that adds a PII column matches no /src/auth/ rule and routes to the lowest tier.; Source: tiered-code-review · Lesson: tiered-and-tunable-review
CODEOWNERS routing: Using GitHub's CODEOWNERS file plus branch protection to approximate path-based tiers — critical directories require named human reviewers, while unmatched paths get AI review and green CI only. The native-tooling way to build tiered review today.; Source: tiered-code-review · Lesson: tiered-and-tunable-review
AI-first, human-last: The collaboration model the synergy data supports — the agent runs the mechanical first pass (defects, improvements) and the human makes the final merge decision (design judgment, knowledge transfer, architectural fit). The human owns the merge button.; Source: human-ai-review-synergy · tiered-code-review · Lesson: human-ai-synergy
Tunable effort dial: A per-PR review-depth control backed by a published bug-discovery curve, letting a reviewer or routing policy trade thoroughness against cost where it matters. Cursor's High finds 35% more bugs at constant resolution rate; the dial only pays off when routing concentrates High on high-stakes runs.; Avoid: pinning High by default — it burns attention on routine PRs and recreates the fatigue the dial existed to avoid.; Source: tunable-review-effort · Lesson: tiered-and-tunable-review
Bug-discovery curve: The published metric that gives an effort level meaning — bugs found per run at each level, measured on a benchmark and held against a constant resolution rate. Without the curve, "high effort" is a hedge word.; Source: tunable-review-effort · Lesson: tiered-and-tunable-review
Risk-score threshold calibration: Exposing the auto-approval cutoff on a learned diff-risk score as an explicit yield-vs-safety knob — moving it up automates more diffs at strictly higher marginal risk. Requires per-bucket revert/incident telemetry, a deterministic-validation backstop, and stable features; without all three the knob is opaque. Meta's RADAR is the industrial case.; Source: risk-score-threshold-calibration · Lesson: tiered-and-tunable-review

Multi-Agent Review and Verification

Committee review pattern: Routing one diff through several reviewer agents in parallel, each scoped to a single dimension (correctness, security, test coverage), then aggregating structured verdicts and routing feedback back to the implementer. Separate implementer and reviewer prompts remove the self-review confirmation bias.; Avoid: combining concerns in one reviewer — scope each to a single lens, or verdicts contradict and get harder to act on.; Source: committee-review-pattern · Lesson: the-committee-pattern
Cross-model review: Assigning each committee reviewer to a different model provider so failure modes are independent. A panel that all runs on one model shares its blind spots and confirms its own errors.; Source: committee-review-pattern · Lesson: the-committee-pattern
Risk-proportional scaling: Scaling reviewer count to the change's risk — zero reviewers for typos, one for ordinary changes, three cross-model for auth, crypto, payments, or schema migrations. File-level risk classification drives automatic escalation regardless of task scope.; Source: committee-review-pattern · Lesson: the-committee-pattern
Cloud parallel review: Fan out reviewers in a remote sandbox, verify each candidate finding against actual code behavior, deduplicate and severity-rank, then post one review. Distinguished from committee voting and tiered escalation by the verification step, cloud execution, and a single posted artifact. Claude Code's /ultrareview is the canonical implementation.; Source: cloud-parallel-review-pattern · Lesson: the-committee-pattern
Verification gate · reproduce-before-report: A step that drops any reviewer finding an independent verifier cannot reproduce against actual code behavior. It sits at the reporter boundary: reviewers flag freely, and only reproduced findings ship. Load-bearing in cloud parallel review — without it, fan-out amplifies false positives instead of reducing them.; Source: reproduce-before-report-verification-gate · Lesson: reproduce-before-report
Verifier independence: The load-bearing property of the gate — the verifier starts from fresh context (the diff plus the claim, not the reviewer's chain of thought) and ideally a different model family. A same-family, same-session verifier reproduces the reviewer's own confabulations.; Source: reproduce-before-report-verification-gate · Lesson: reproduce-before-report
Reproduction artifact: The concrete evidence the verifier must build — a triggering input, a contradicting code path, or a file:line citation. A reproduction is harder to fake than a judgment: it either runs or it does not. Race conditions and intermittent flakes are the defect classes it cannot capture on a static diff.; Source: reproduce-before-report-verification-gate · Lesson: reproduce-before-report
Correlated failure: When an AI reviewer reasons from the same training distribution as the AI author and misses the same defect classes — agreement that echoes error rather than cancelling it. The reason self-review has a structural ceiling; cross-family panels or executable specs are the mitigation.; Source: human-ai-review-synergy · agent-self-review-loop · Lesson: reproduce-before-report

Working With the Human Reviewer

Triage: Evaluating each AI suggestion on its own rather than treating the review output as a to-do list. With a 16.6% adoption rate and most rejections caused by wrong suggestions, rubber-stamping AI output works against you.; Avoid: waving through a large AI diff — the law-of-triviality dynamic sails big changes past review while one-liners attract debate.; Source: human-ai-review-synergy · Lesson: human-ai-synergy
PR slicing: Having the agent that built a branch propose its own decomposition into smaller, independently reviewable PRs — using session intent and the dependency graph, not diff clustering, as the cut signal. Targets the 200–400-line reviewer envelope; fails on cross-cutting refactors and atomic-revert paths.; Source: agent-driven-pr-slicing · Lesson: pr-slicing-and-descriptions
Stacked PRs: Dependent slices that each target the one before, landing in order. The partial order is half the work — without it, reviewers merge dependent slices out of sequence and intermediate states break. Practitioner ceiling is three to four per stack before rebase cascades dominate.; Source: agent-driven-pr-slicing · Lesson: pr-slicing-and-descriptions
Review churn: Back-and-forth review rounds that signal reviewers reconstructing intent, not productive alignment. The outcome to minimize: Copilot generated the most reviewer discussion yet had the lowest merge rate, and each extra comment on an agent PR correlates with lower merge probability.; Source: pr-description-style-lever · Lesson: pr-slicing-and-descriptions
PR description template: A required description structure — Summary, Changes, Testing, Breaking Changes — configured in CLAUDE.md or AGENTS.md. A secondary but measurable merge-rate lever; neither Claude Code nor Copilot enforces structure by default, and code quality stays the primary determinant.; Source: pr-description-style-lever · Lesson: pr-slicing-and-descriptions
Deletion pre-flag: Predicting, from structural features (method-name length, LOC, Halstead volume, call count), which agent-generated functions reviewers will delete — and returning the report to the agent before a human spends time on it. Fewer surviving functions beats more functions with a higher deletion rate.; Source: predicting-reviewable-code · Lesson: pr-slicing-and-descriptions
Inspection order: A time-boxed priority sequence for reviewing an agent PR — CI changes first (the cheapest hard stop), then duplicated utilities, the critical path, security boundaries, and evidence last. Sequenced by cost; running it out of order wastes the cheapest stops.; Source: reviewers-playbook-agent-authored-prs · Lesson: feedback-to-rule-loop
Context, not correctness: The reframing of what reviewing an agent PR is for — the code already parses and passes its own tests, so the human verifies that the agent understood the codebase, the contract, and the intent. The strongest test is demanding a test that fails on the pre-change behavior.; Source: reviewers-playbook-agent-authored-prs · Lesson: feedback-to-rule-loop

Closing the Loop

Feedback-to-rule loop: Promoting a recurring review comment into a mechanical check once it fires across three or more PRs, then retiring it when its hit count reaches zero. Converts an invariant living in one reviewer's head into a deterministic check that fires every time, not probabilistically.; Source: review-feedback-to-rule-loop · Lesson: feedback-to-rule-loop
Promotion threshold · three strikes: The load-bearing trigger — one or two occurrences of a comment is a hypothesis, three or more is a pattern worth encoding. Premature promotion freezes a guess as a rule and breeds suppression comments.; Source: review-feedback-to-rule-loop · Lesson: feedback-to-rule-loop
Encoding layer: The mechanism a promoted comment belongs in, matched to its category — linter for style, AST/import check for architectural boundaries, pre-completion checklist for safety invariants, evaluator rubric or test for spec violations. Forcing a semantic check into a regex linter is a category error that becomes the new noise source.; Source: review-feedback-to-rule-loop · Lesson: feedback-to-rule-loop
Remediation text: The "what to do instead" paired with every rule. A rule that only says what is wrong relocates the bottleneck from review to comprehension; the working shape is what-is-wrong, what-to-do, where-the-rationale-lives.; Source: review-feedback-to-rule-loop · Lesson: feedback-to-rule-loop
Priority saturation: The failure mode where every rule carries equal weight, so nothing signals priority and adherence degrades. The reason promoted rules need retirement discipline — context rot means models recall earlier rules less accurately as the instruction set grows.; Source: review-feedback-to-rule-loop · Lesson: feedback-to-rule-loop
Learned review rules: A reviewer that persists rules extracted from accept/reject signals — accepted fixes become positive rules, dismissed comments become suppression rules — sharpening on a codebase without manual reconfiguration. Cursor's Bugbot drove resolution from 52% to ~78% this way. Distinct from the feedback-to-rule loop: this tunes the reviewer; that promotes the invariant out of the reviewer entirely.; Avoid: leaving suppression rules un-pruned — they encode team blind spots and go stale after a refactor.; Source: learned-review-rules · Lesson: feedback-to-rule-loop