Skip to content

Methodology

Benchmark

Our latest run, on 15 October 2026: the review pipeline as it runs in production — Claude Haiku 5.5 to triage, Claude Opus 5.5 to find and verify — on 25 bug fixtures we seeded ourselves. 68.0% of seeded bugs found and 65.4% of comments right, with every miss listed. Seeded bugs are easier than real ones, so read it as a regression signal, not a real-world catch rate. The cross-tool table stays empty until a comparable run exists.
Latest run: 2026-10-15Seeded-bug eval: published belowCross-tool results: not yet publishedHarness: not public yetMethodology: complete

Results

Two different things live on this page, and mixing them up is how benchmark pages lie. The seeded-bug eval is real and published: its latest run is from 2026-10-15 and measures exactly one thing — our pipeline against our own fixture corpus, 68.0% of seeded bugs found and 65.4% of comments right. The cross-tool table further down measures no one yet — its dataset does not exist, and its gate has not been met — so it stays empty rather than borrowing this run’s numbers.

The seeded-bug eval, scored live

The harness is ours. It lives in our repository, which is private today, and it will be published with the first comparable run. The dataset: 25 realistic pull-request diffs, each carrying one hand-authored seeded bug, across go, java, javascript, python, typescript, with the superdiffs review pipeline scored against them for recall, precision, F1, anchor accuracy and verified share. Every run is kept, with every miss listed.

On 15 October 2026 we ran the review pipeline the way production runs it — the balanced profile, triage on claude-haiku-5-5, finding and verification on claude-opus-5-5, and the confidence floor production calibrates for that model (shifted -0.1) — against every fixture, minus GitHub posting. 25 fixtures, 26 findings, $1.30 of model time, no fixture errors. Every number in this section is rendered from the committed run JSON, not retyped by hand.

Seeded-bug eval totals
Metric2026-10-15 (current)2026-08-30, matcher v22026-08-30, matcher v1Offline mock
Recall68.0%80.0%8.0%92.0%
Precision65.4%66.7%6.7%85.2%
F10.6670.7270.0730.885
Anchor accuracy88.2%85.0%100.0%95.7%
Verified share100.0%100.0%100.0%95.7%
True / missed / false positives17 / 8 / 920 / 5 / 102 / 23 / 2823 / 2 / 4
Findings posted26303027

The offline column is a deterministic mock that stands in for the pipeline in CI — it exercises the scoring path, not a model, and must never be read as model performance. It is shown here only as the harness self-test it is.

What changed since the first run

The test did not change: the 25 fixture files are the ones the first run used, and its matcher v2 column was scored by the same scorer as this one. What changed is the product: the models (gpt-5.6-luna and gpt-5.6-terra then, Claude now), the confidence floor calibrated for the model that finds, and the review engine itself, whose releases are in the changelog. The two columns compare the product in August with the product now, on the same test — and on this test the product now finds fewer of the seeded bugs (17 of 25, against 20 of 25), with precision of 65.4% against 66.7%, and a run costs $1.30 against $0.57. It is published as the current result anyway: that is the rule at the bottom of this page.

What "verified" means

Every candidate finding goes through a second model pass that re-reads the diff and either confirms or rejects it; a finding counts as verified only when that pass said confirmed. In this run 100.0% of matched findings survived verification — a claim about the verifier agreeing with the finder (they are the same model, claude-opus-5-5), not independent confirmation. A verifier that never disagrees is a rubber stamp, and the number should be read with that suspicion attached.

Per bug class

Current run per bug class
Bug class (seeded)TruthFoundMissedFalse positivesRecall
unvalidated-input3300100.0%
race-condition312033.3%
authz-bypass2200100.0%
command-injection211050.0%
error-swallowing2200100.0%
resource-leak2200100.0%
negative-amount20200.0%
off-by-one2200100.0%
sql-injection211050.0%
floating-promise1100100.0%
missing-await10100.0%
unsafe-regex1100100.0%
hardcoded-secret1100100.0%
type-coercion10100.0%

Noise the pipeline emitted under its own labels — 2 security, 2 correctness, 2 error-handling, 1 concurrency, 1 data-loss and 1 performance — lands in those buckets, not in the seeded classes, so the rows above only ever count bugs the corpus actually planted. Each fixture seeds one bug, so a real defect its author did not plant also counts as a false positive here; none has been re-labelled.

The 8 missed bugs

A benchmark that only shows averages is hiding the interesting part. These are the fixtures the pipeline failed in the current run, by id, with the seeded bug it missed:

  • async-missing-await-commit — src/newsletter/signup.ts:25: consentStore.insert(record) is never awaited, so the signup returns before the consent record is written and a failed insert leaves no consent trail.
  • cmdi-tar-extract — src/plugins/install.ts:12: the manifest-controlled plugin name flows unvalidated into the extraction path, so a name like ../../srv/app installs and runs attacker-chosen code outside the plugins root.
  • money-negative-quantity-discount — src/billing/pricing.ts:17: seatSubtotal and applyVolumeDiscount never validate seats or months, so a negative quantity yields a negative subtotal the charge path accepts.
  • money-negative-refund — src/api/refunds.ts:36: cents is never validated as a positive integer, so a negative refund passes the balance check and increases the customer’s balance.
  • race-check-then-act — app/webhooks.py:14: two workers handling the same delivery id can both pass the processed_ids membership check before either adds it, so the delivery is processed twice.
  • race-rmw-stock-decrement — src/main/java/shop/Inventory.java:19: reserve() is synchronized but other stock readers and writers are not, so they see stale stock or interleave with the decrement, and the lock is held across a 50 ms sleep.
  • sqli-order-by-fstring — app/api/users.py:28: the sort column is allowlisted but the direction is not, so the dir query parameter is interpolated raw into the ORDER BY clause.
  • types-coercion-loose-equality — src/flags/override.js:14: raw == 'true' || raw == '1' uses loose equality on a query value that is not always a string, so crafted inputs coerce into enabling the flag.

The list is what it is because the run says so, not because we curated it: these ids are derived from the scored JSON, and a test fails if the page and the run ever disagree.

Per-fixture detail

All 25 fixtures of the current run, scored
Current run per fixture
FixtureLanguageFindingsMatchedRecallAnchorVerifiedCost
async-floating-promise-notifytypescript11100.0%100.0%100.0%$0.09
async-missing-await-committypescript000.0%0.0%0.0%$0.04
authz-jwt-no-verifytypescript21100.0%100.0%100.0%$0.07
authz-missing-owner-checktypescript11100.0%100.0%100.0%$0.03
cmdi-exec-filepython21100.0%100.0%100.0%$0.07
cmdi-tar-extracttypescript100.0%0.0%0.0%$0.07
error-swallowed-catch-emptytypescript11100.0%0.0%100.0%$0.05
error-swallowed-except-passpython11100.0%100.0%100.0%$0.03
input-path-traversalpython11100.0%100.0%100.0%$0.05
input-unvalidated-deserializepython11100.0%100.0%100.0%$0.03
input-unvalidated-integer-castgo11100.0%0.0%100.0%$0.04
leak-db-conn-txntypescript11100.0%100.0%100.0%$0.08
leak-goroutine-channelgo21100.0%100.0%100.0%$0.06
money-negative-quantity-discounttypescript000.0%0.0%0.0%$0.03
money-negative-refundtypescript200.0%0.0%0.0%$0.08
offbyone-loop-boundtypescript11100.0%100.0%100.0%$0.04
offbyone-pagination-windowtypescript11100.0%100.0%100.0%$0.06
race-check-then-actpython100.0%0.0%0.0%$0.07
race-rmw-countertypescript11100.0%100.0%100.0%$0.03
race-rmw-stock-decrementjava100.0%0.0%0.0%$0.06
regex-catastrophic-backtrackingtypescript21100.0%100.0%100.0%$0.05
secret-hardcoded-api-keytypescript11100.0%100.0%100.0%$0.03
sqli-order-by-fstringpython000.0%0.0%0.0%$0.07
sqli-string-concattypescript11100.0%100.0%100.0%$0.03
types-coercion-loose-equalityjavascript000.0%0.0%0.0%$0.03

The first run, 30 August 2026

The first live run used the models production ran then: triage on gpt-5.6-luna, finding and verification on gpt-5.6-terra. Those models are no longer in the codebase. It stays here as the run the current one is measured against: 80.0% of seeded bugs found and 66.7% of comments right under matcher v2, 30 findings for $0.57 of model time, and these 5 misses:

  • async-floating-promise-notify — src/exports/runner.ts:22: notifyWebhook(...) is async but called without await or .catch, so a rejected promise never surfaces.
  • authz-missing-owner-check — src/routes/docs.ts:45: the DELETE handler authenticates the user but never checks the document belongs to their organization, so any logged-in user can delete any document by id.
  • input-unvalidated-integer-cast — internal/export/limit.go:20: the client-supplied max is parsed without validation, so ?max=-1 passes through as -1 and downstream treats it as unlimited, bypassing the plan's export cap.
  • money-negative-quantity-discount — src/billing/pricing.ts:17: seatSubtotal and applyVolumeDiscount never validate seats or months, so a negative quantity yields a negative subtotal the charge path accepts.
  • types-coercion-loose-equality — src/flags/override.js:14: raw == 'true' || raw == '1' uses loose equality on a query value that is not always a string, so crafted inputs coerce into enabling the flag.

The matching rule

A prediction matches a seeded bug when they name the same file, sit within five lines of each other, and either carry crosswalk-equivalent categories or describe the same defect (token overlap ≥ 0.15). Anchor accuracy is scored separately and stricter: a match only counts as anchored when it lands within two lines of the seeded line. The crosswalk reconciles the pipeline’s free-form categories with the fixture taxonomy, is published in full, and is pinned by unit tests — changing it silently changes every historical number, which is why it lives in version control like code:

Category crosswalk (matcher v2)
Pipeline categoryCounted as the seeded class
securityauthz-bypass, command-injection, sql-injection, hardcoded-secret, unvalidated-input, unsafe-regex, path-traversal, timing-unsafe-compare
error-handlingerror-swallowing, floating-promise, missing-await
correctnessrace-condition, off-by-one, type-coercion, negative-amount, unvalidated-input, unsafe-regex
performanceperformance, resource-leak
concurrencyrace-condition
secrethardcoded-secret
resource-leakresource-leak

How to reproduce

The harness, fixtures and every scored report live in our repository; the current run was made with the harness at commit 72c5ce8. The repository is private today and will be published with the first comparable run. Once it is, these are the commands — from the repository root, with your own Anthropic key exported as ANTHROPIC_API_KEY:

shell
# a live run on the production model setup (real pipeline, real model bill, hard spend ceiling)
npx tsx scripts/eval/run.mjs --live --tag live-claude --max-cost 15

# rescore the first run with the current matcher — no model calls
npx tsx scripts/eval/rescore.mjs docs/eval/results-live-v1.json --tag live-v1-matcher-v2

# the deterministic offline mock (CI-safe, no key needed)
npx tsx scripts/eval/run.mjs --offline --tag offline-pinned

Models are pinned by environment (LLM_CHEAP_MODEL, LLM_FRONTIER_MODEL, LLM_THOROUGH_MODEL) and default to the production lanes; the current run used claude-haiku-5-5 for triage and claude-opus-5-5 for finding and verification. The first run’s models are no longer in the codebase, so rescoring its committed output needs no model at all. A different pin is a different run and gets a new tag.

Why every vendor wins its own benchmark

No results published yet.

This box governs the CROSS-TOOL table only — superdiffs vs other reviewers on Dataset A (50 real-bug PRs). No such run has been executed, so this table stays empty until the dataset is frozen and an outside party has reproduced a dry run. The seeded-bug eval of our own pipeline, published above, is a different measurement and is not governed by this gate.

Before the first run

Dataset A (50 real-bug PRs, one public fork per tool) must be curated and frozen, and an outside party must have reproduced a dry run on a clean machine, before any number appears here.

What that run will publish

  • Per-dataset precision and recall, with confidence intervals
  • Comments per PR and severity mix
  • Per-language and per-severity splits, not just the aggregate
  • The full list of PRs, including the ones we scored zero on
  • The harness commit SHA and the model + prompt version pinned for the run

Look at the accuracy claims in this category and you will notice something: every vendor comes first. That is not because they are all lying about the arithmetic. It is because the arithmetic is the last step, and every step before it is a choice made by the person who wants a particular answer.

  • The dataset is chosen after the tool exists. Pick the bug classes your finder is good at, and recall goes up without the product changing.
  • Competitors run at their defaults; you run at yours. Or the reverse, if that flatters you more. Nobody publishes the configuration.
  • Synthetic bugs are much easier than real ones. A generator that injects an off-by-one writes it the way a generator writes it. Catch rates on synthetic sets run roughly an order of magnitude above real-world rates.
  • Contamination is rarely controlled. If the fix for the bug was in the training data, you are measuring recall of memorized text.
  • Precision is judged by the vendor’s own model. Ask a model whether its own comment was useful and you will be told yes.
  • Only the good run gets published. Run five configurations, report the best, call it the result.

What we are doing differently

These commitments govern the cross-tool benchmark. The seeded-bug eval above is our own regression run, published early and labeled as exactly that.

Benchmark commitments
CommitmentWhat it rules out
The harness, dataset manifest and rubric are public before the first result.Designing the method around a result we already have.
Every tool is run at its documented defaults, and the configuration used is published.Winning by tuning ourselves up and everyone else down.
Only post-training-cutoff pull requests are used for the headline dataset.Scoring memorization as review ability.
Every comment every tool posted is published verbatim, including ours that were wrong.Publishing an aggregate that hides the failures.
Comments per pull request is reported next to precision and recall.Buying recall with volume and calling it a win.
Synthetic data is labeled as synthetic and never carries a headline claim.Quoting a mutation-testing number as if it were a real-world catch rate.
No number appears until an outside party has reproduced a run.A result nobody but us has ever seen produced.

Datasets

Three datasets, with different jobs. Only the first can carry a claim about how good the product is.

Dataset A: Real bugs, re-opened as fresh PRs

Headline
Target size
50 PRs, 5+ languages, 4 severity tiers
Ground truth
The upstream fix commit. We blame a real bug-fix commit back to the commit that introduced the defect, then re-open that introducing change as a new PR in a clean public fork.
Why it exists
It is the only dataset where the bug was written by a human, found by a human, and fixed by a human — no generator decided what counted as a defect.
Status
curation not started

Dataset B: Injected defects in real merged PRs

Supporting
Target size
100+ PRs, ~300 verified defects
Ground truth
The injection record: file, line, and a written description of the defect, verified by a second model and then double-checked by a human.
Why it exists
Real code, real diffs, controlled ground truth. Catches the case where a tool is good at spotting textbook bugs but blind to the shape of a defect in a live codebase.
Status
pipeline not started

Dataset C: Mutation operators over templates

Regression only
Target size
~200 generated cases, deterministic seeds
Ground truth
Generated: the mutation operator knows exactly what it broke.
Why it exists
Fast enough to gate every prompt change in CI.
Caveat
Synthetic sets overstate real-world catch rate by roughly an order of magnitude. This dataset is never used for a headline claim, and any number from it is labeled as a regression signal only.
Status
operators specified, not built

Contamination controls

The failure that invalidates an AI benchmark most often, and the one least often mentioned. If a model saw the fix during training, finding the bug is recall of text, not review.

Contamination controls
ControlHow it works
Post-cutoff PRs onlyEvery Dataset A case comes from a change merged after the training cutoff of every model under test. A model that memorized the fix is not measuring review ability.
Fresh forks, one per toolEach tool reviews its own fork of the PR, so no tool sees another tool's comments and none of them sees the upstream discussion.
Upstream references strippedPR titles and bodies are rewritten to the state of the world before the fix. Issue links, fix commit messages, and 'fixes #123' references are removed.
Frozen, freeze-dated snapshotsA published run pins the dataset snapshot, the harness commit, and the model and prompt versions. Re-running a snapshot later reproduces the same inputs even after we have changed the product.
Cutoff re-audit on every model changeWhen a model under test is upgraded, cases that fall on the wrong side of the new cutoff are retired rather than silently reused.

What we measure

Precision

Of the comments posted, the fraction that describe a real defect at a location a reviewer would accept. Judged from a written rubric, with a human-labeled sample used to calibrate the judge.

This is the number that decides whether a team keeps the tool installed. Volume is easy; being right is not.

Recall

Of the known defects in the dataset, the fraction found with a correct description at the correct file and line.

Precision alone is gameable — post one comment a year and be right about it.

Comments per PR

Mean and p90 comments posted per reviewed PR, at the default profile.

Two tools with identical precision are not equivalent if one posts 4 comments and the other posts 40.

Addressed rate (live)

Share of posted findings where the flagged code was materially changed before merge, measured on real customer traffic rather than on a dataset.

It is the only metric measured on production work. It is directional, not exact: a developer may fix a real problem for an unrelated reason, or ignore a correct comment because of deadline pressure. We publish it labeled as such.

Scoring

Each posted comment is labeled against a written rubric: does it describe a real defect, at a location a reviewer would accept, with a failure scenario that holds? Labeling is done by a model that is not one of the models under test, calibrated against a human-labeled sample, and the sample’s agreement rate is published with the run. Where the judge and the humans disagree beyond the published threshold, the run is not published.

Comments about style, formatting or documentation are scored in a separate bucket from defect findings. Rolling them together lets a tool inflate precision with a hundred true statements about import ordering.

Run it yourself

Once the cross-tool harness is published, everything needed to run it against any tool, ours included, will be public: the harness, the dataset manifest and the scoring rubric. None of it is public yet.

What you need

  • Node 20+, Docker, and a GitHub account that can create forks
  • An API key for whichever tool you are scoring (ours included)
  • About 90 minutes of wall-clock time for a full Dataset A run

The commands, once it is published

shell
git clone https://github.com/superdiffs/benchmark && cd benchmark
npm ci
npm run dataset:fetch -- --snapshot <freeze-date>
npm run bench -- --tool superdiffs --dataset A --out results/
npm run bench -- --tool <other-tool> --dataset A --out results/
npm run score -- results/ --rubric rubrics/v1.md

Substituting --tool superdiffs for any other supported tool is the entire difference between scoring us and scoring a competitor. That symmetry is the point: a harness that can only run against one product is a marketing asset.

Published with every run

  • results/raw/*.json — every comment every tool posted, verbatim
  • results/labels/*.json — the rubric verdict for every comment, including ours that failed
  • manifest.lock — dataset snapshot, harness SHA, model and prompt versions

Availability. Not published yet. The harness repository will be published with the first comparable cross-tool run; publishing an empty harness before then would be theater. It will live under the github.com/superdiffs organization when it ships.

Independent verification

No number appears on this page until someone outside the company has run the harness on a clean machine and reproduced it. If their result disagrees with ours, theirs is published next to ours.

If you run static analysis research, maintain a large open-source repository, or simply enjoy taking benchmarks apart, we would rather you found the flaw before we published than after. Write to hello@superdiffs.com and we will hand over the harness early.

When we are wrong

A published run is a snapshot of a pinned model and a pinned prompt. When we change either, the numbers change, and the old ones stay up with the versions they belonged to rather than being quietly replaced. If a re-run is worse than the last one, it is published as the current result — a benchmark you only update when it improves is a leaderboard you are curating, not a measurement.

Releases that change comment volume are also called out in the changelog, which is where you will notice a regression first.