Tiered & Tunable Review

Stop reviewing every PR as if it could take down production. Route depth by risk, dial effort to the runs that pay for it, and calibrate the cutoff instead of guessing it.

Why this, for you: uniform human depth does not scale when AI is opening PRs that are ~18% larger with change-failure rates up ~30%. Review hard on everything and your best reviewers drown; review soft on everything and an auth bypass slips through. The fix is not "more review" — it is routing review.

Three dials sit on top of each other, each at a different grain. Tiers route by static path criticality, per-PR effort dials thoroughness on one run, and threshold calibration decides whether human review happens at all. They compose — they do not substitute.

1 Tier by criticality, not by line count

Tiered review treats review as a risk-routing problem: classify code by criticality, then match review effort to risk. AI runs the first pass on everything; human depth is reserved for the costly or irreversible.

Tier	Code types	Merge gate
Automated	Tests, docs, config, CSS, reversible migrations	AI review only, CI green
AI + Human	Business logic, API contracts, data models	AI first pass + CODEOWNERS approval
Human-only	Auth, payments, crypto, PII handling	Mandatory human / security sign-off

GitHub offers no native path-based routing, so you approximate it: enable automatic AI review on all PRs, define CODEOWNERS for the critical paths, and require CODEOWNERS approval in branch protection. A PR touching only Tier-1 paths matches no owner and merges on AI + CI alone.

Classification is the whole game

Tiered review fails exactly when classification is wrong: a migration that adds a PII column, a config change that broadens CORS, or a utility called from an auth path — none match /src/auth/, so they route to Tier 1 and merge unreviewed. Cross-cutting PRs that touch both a Tier-1 and a Tier-2 path can under-escalate if the logic stops at the first match.

2 Make effort a per-PR dial, backed by a curve

Tiers route who reviews; the effort dial sets how hard the agent looks on one PR. A tunable review agent ships named effort levels, each pinned to a point on a published bug-discovery curve. Cursor's Bugbot externalized exactly this:

Level	Bug-discovery rate	Trade-off
Default	0.7 bugs/run	Optimised for efficiency and speed
High	0.95 bugs/run	More reasoning, more expensive, finds more bugs
Custom	Operator-defined	Natural-language routing policy, evaluated per PR

High finds 35% more bugs at a constant 80% resolution rate. Holding resolution rate steady across levels is the calibration commitment — High costs more but does not flood you with low-quality flags. Effort labels without a published curve are just hedge words.

The dial composes with structured routing across four axes — file-path criticality (auth, payments, crypto), change size (LOC over threshold), author trust (first-time contributors, external forks), and historical defect rate (dirs with elevated post-merge bugs). Pin Default at the team level and let a Custom policy escalate — opt-in by policy, not by per-PR clicking.

# Custom routing policy — concentrate High where it pays Use High effort when: - the PR touches src/auth/, src/payments/, or src/crypto/ - the diff exceeds 500 lines - the author has fewer than 10 merged PRs in this repo - most changed files saw a post-incident commit in the last 90 days Use Default effort otherwise.

Resolution rate is not precision

The 35%-more-bugs figure counts both "developer fixed it" and "developer made it go away" — it does not rule out more false positives that authors muted. And pinning High on every PR burns the premium and accumulates the alert fatigue the dial existed to avoid, collapsing it back to a fixed-pipeline calibration.

3 Calibrate the threshold; don't guess it

The third dial is organization-wide. A learned diff-risk model scores each diff by its likelihood of revert or production incident, and a single percentile threshold separates auto-approved diffs from those routed to a human. Where you set that point is an operator choice, not a property of the model.

Meta's RADAR is the documented industrial case — a six-stage pipeline (classification → eligibility gates → static heuristics → ML risk score → LLM review → deterministic validation) where the risk score is the knob. Diffs at or below the chosen percentile proceed; the rest go to a human.

Threshold	Auto-approve rate	Safety vs. baseline
25th percentile	not reported	revert ~1/3, incident ~1/50
50th percentile	60.31%	revert ~1/3, incident ~1/50
Non-RADAR baseline	—	reference

Moving the threshold up automates more diffs at strictly higher marginal risk; moving it down trades yield for safety. The dial only works with three things at once: a monotonic risk score, per-bucket revert/incident telemetry, and a deterministic validation backstop. Remove any one and the knob is opaque.

Where calibration is the wrong dial

Small or low-telemetry teams cannot measure revert rate per percentile — the threshold becomes an unverifiable guess, so use static path-tiering instead. Regulated domains may mandate human sign-off regardless of empirical safety. And a calibration-aware adversary can structure a malicious diff to land in the auto-approved tier — a single knob optimizes for the average diff, not the worst case.

↪ Your win: three dials, one routing posture

Tier by criticality — AI on everything; humans on auth, payments, crypto, and irreversible state.
Classification is the failure mode — watch PII migrations, CORS configs, and cross-cutting PRs that dodge the path patterns.
Make effort a per-PR dial backed by a published curve — labels without numbers are hedge words.
Default-low, escalate by policy — concentrate High on path, size, author, and defect-rate axes; High-on-everything is alert fatigue.
Calibrate, don't guess — a percentile threshold needs a monotonic score, per-bucket telemetry, and a validation backstop.

Retrieval practice — recall, don't peek

Question 1In tiered review, the reserved-for-humans tier covers…

Question 2What makes a named effort level more than a hedge word?

Question 3Pinning High effort on every PR collapses the dial because it…

Question 4A risk-score auto-approval threshold stays measurable only with…

Question 5 · spaced recall from an earlier lessonAn agentic review is best understood as…

Ask me anything. Want help drawing the path patterns for your monorepo's tiers, or deciding whether you have the telemetry to calibrate a threshold at all? Next up: Structure-Aware Diffs — labelling each hunk by what it touches so the reviewer spends attention where structure says it matters.