Methodology
Benchmark
Results
No results published yet.
The methodology below is complete and frozen. No run has been executed against it yet, so this page publishes no numbers. When the first run lands, its raw outputs, per-PR fork links, and the harness commit that produced them are published together — or nothing is.
First run
Scheduled for beta. Dataset A (50 real-bug PRs, one public fork per tool) must be curated and frozen, and an outside party must have reproduced a dry run on a clean machine, before any number appears here.
What that run will publish
- Per-dataset precision and recall, with confidence intervals
- Comments per PR and severity mix
- Per-language and per-severity splits, not just the aggregate
- The full list of PRs, including the ones we scored zero on
- The harness commit SHA and the model + prompt version pinned for the run
Why every vendor wins its own benchmark
Look at the accuracy claims in this category and you will notice something: every vendor comes first. That is not because they are all lying about the arithmetic. It is because the arithmetic is the last step, and every step before it is a choice made by the person who wants a particular answer.
- The dataset is chosen after the tool exists. Pick the bug classes your finder is good at, and recall goes up without the product changing.
- Competitors run at their defaults; you run at yours. Or the reverse, if that flatters you more. Nobody publishes the configuration.
- Synthetic bugs are much easier than real ones. A generator that injects an off-by-one writes it the way a generator writes it. Catch rates on synthetic sets run roughly an order of magnitude above real-world rates.
- Contamination is rarely controlled. If the fix for the bug was in the training data, you are measuring recall of memorised text.
- Precision is judged by the vendor’s own model. Ask a model whether its own comment was useful and you will be told yes.
- Only the good run gets published. Run five configurations, report the best, call it the result.
What we are doing differently
| Commitment | What it rules out |
|---|---|
| The harness, dataset manifest and rubric are public before the first result. | Designing the method around a result we already have. |
| Every tool is run at its documented defaults, and the configuration used is published. | Winning by tuning ourselves up and everyone else down. |
| Only post-training-cutoff pull requests are used for the headline dataset. | Scoring memorisation as review ability. |
| Every comment every tool posted is published verbatim, including ours that were wrong. | Publishing an aggregate that hides the failures. |
| Comments per pull request is reported next to precision and recall. | Buying recall with volume and calling it a win. |
| Synthetic data is labelled as synthetic and never carries a headline claim. | Quoting a mutation-testing number as if it were a real-world catch rate. |
| No number appears until an outside party has reproduced a run. | A result nobody but us has ever seen produced. |
Datasets
Three datasets, with different jobs. Only the first can carry a claim about how good the product is.
Dataset A: Real bugs, re-opened as fresh PRs
Headline- Target size
- 50 PRs, 5+ languages, 4 severity tiers
- Ground truth
- The upstream fix commit. We blame a real bug-fix commit back to the commit that introduced the defect, then re-open that introducing change as a new PR in a clean public fork.
- Why it exists
- It is the only dataset where the bug was written by a human, found by a human, and fixed by a human — no generator decided what counted as a defect.
- Status
- curation not started
Dataset B: Injected defects in real merged PRs
Supporting- Target size
- 100+ PRs, ~300 verified defects
- Ground truth
- The injection record: file, line, and a written description of the defect, verified by a second model and then double-checked by a human.
- Why it exists
- Real code, real diffs, controlled ground truth. Catches the case where a tool is good at spotting textbook bugs but blind to the shape of a defect in a live codebase.
- Status
- pipeline not started
Dataset C: Mutation operators over templates
Regression only- Target size
- ~200 generated cases, deterministic seeds
- Ground truth
- Generated: the mutation operator knows exactly what it broke.
- Why it exists
- Fast enough to gate every prompt change in CI.
- Caveat
- Synthetic sets overstate real-world catch rate by roughly an order of magnitude. This dataset is never used for a headline claim, and any number from it is labelled as a regression signal only.
- Status
- operators specified, not built
Contamination controls
The failure that invalidates an AI benchmark most often, and the one least often mentioned. If a model saw the fix during training, finding the bug is recall of text, not review.
| Control | How it works |
|---|---|
| Post-cutoff PRs only | Every Dataset A case comes from a change merged after the training cutoff of every model under test. A model that memorized the fix is not measuring review ability. |
| Fresh forks, one per tool | Each tool reviews its own fork of the PR, so no tool sees another tool's comments and none of them sees the upstream discussion. |
| Upstream references stripped | PR titles and bodies are rewritten to the state of the world before the fix. Issue links, fix commit messages, and 'fixes #123' references are removed. |
| Frozen, freeze-dated snapshots | A published run pins the dataset snapshot, the harness commit, and the model and prompt versions. Re-running a snapshot later reproduces the same inputs even after we have changed the product. |
| Cutoff re-audit on every model change | When a model under test is upgraded, cases that fall on the wrong side of the new cutoff are retired rather than silently reused. |
What we measure
Precision
Of the comments posted, the fraction that describe a real defect at a location a reviewer would accept. Judged from a written rubric, with a human-labelled sample used to calibrate the judge.
This is the number that decides whether a team keeps the tool installed. Volume is easy; being right is not.
Recall
Of the known defects in the dataset, the fraction found with a correct description at the correct file and line.
Precision alone is gameable — post one comment a year and be right about it.
Comments per PR
Mean and p90 comments posted per reviewed PR, at the default profile.
Two tools with identical precision are not equivalent if one posts 4 comments and the other posts 40.
Addressed rate (live)
Share of posted findings where the flagged code was materially changed before merge, measured on real customer traffic rather than on a dataset.
It is the only metric measured on production work. It is directional, not exact: a developer may fix a real problem for an unrelated reason, or ignore a correct comment because of deadline pressure. We publish it labelled as such.
Scoring
Each posted comment is labelled against a written rubric: does it describe a real defect, at a location a reviewer would accept, with a failure scenario that holds? Labelling is done by a model that is not one of the models under test, calibrated against a human-labelled sample, and the sample’s agreement rate is published with the run. Where the judge and the humans disagree beyond the published threshold, the run is not published.
Comments about style, formatting or documentation are scored in a separate bucket from defect findings. Rolling them together lets a tool inflate precision with a hundred true statements about import ordering.
Run it yourself
Everything needed to run this against any tool, including ours, is public: the harness, the dataset manifest, and the scoring rubric.
What you need
- Node 20+, Docker, and a GitHub account that can create forks
- An API key for whichever tool you are scoring (ours included)
- About 90 minutes of wall-clock time for a full Dataset A run
The commands
git clone https://github.com/superdiffs/benchmark && cd benchmark
npm ci
npm run dataset:fetch -- --snapshot <freeze-date>
npm run bench -- --tool superdiffs --dataset A --out results/
npm run bench -- --tool <other-tool> --dataset A --out results/
npm run score -- results/ --rubric rubrics/v1.mdSubstituting --tool superdiffs for any other supported tool is the entire difference between scoring us and scoring a competitor. That symmetry is the point: a harness that can only run against one product is a marketing asset.
Published with every run
results/raw/*.json — every comment every tool posted, verbatimresults/labels/*.json — the rubric verdict for every comment, including ours that failedmanifest.lock — dataset snapshot, harness SHA, model and prompt versions
Availability. The repository is published with the first run. Publishing an empty harness before the dataset is frozen would be theatre. The repository will live at https://github.com/superdiffs/benchmark.
Independent verification
No number appears on this page until someone outside the company has run the harness on a clean machine and reproduced it. If their result disagrees with ours, theirs is published next to ours.
If you run static analysis research, maintain a large open-source repository, or simply enjoy taking benchmarks apart, we would rather you found the flaw before we published than after. Write to support@superdiffs.com and we will hand over the harness early.
When we are wrong
A published run is a snapshot of a pinned model and a pinned prompt. When we change either, the numbers change, and the old ones stay up with the versions they belonged to rather than being quietly replaced. If a re-run is worse than the last one, it is published as the current result — a benchmark you only update when it improves is a leaderboard you are curating, not a measurement.
Releases that change comment volume are also called out in the changelog, which is where you will notice a regression first.