Benchmark · Martian Code Review Bench data
How many comments do AI code reviewers post, and how many are right?
What the data shows
- Volume varies widely. The quietest reviewer, Graphite, raised 0.32 issues per pull request and nothing at all on 40 of the 50. The loudest, CodeRabbit, raised 6.36. For scale, the pull requests carry 3.16 known issues each, on average.
- Saying more finds more. Issues raised per pull request and recall rank together at +0.82 (Spearman, 20 tools).
- Saying more is right less often. Against precision the same ranking runs at −0.65, and the issues that match nothing rise almost in step with volume: +0.90.
- Nearly twice the volume found about the same. The three reviewers with the highest F1 — Qodo Deep, Cubic Dev and Augment — raised 3.0–3.6 issues per pull request and found 2.0–2.1 known issues. The two loudest, CodeRabbit and GitHub Copilot, raised 6.0–6.4 and found 1.9–2.1. The difference landed in the unmatched column: 3.4–3.9 per pull request against 1.0–1.4.
- Every judge agrees on the direction. Under Claude Opus 4.5, Claude Sonnet 4.5 and GPT-5.2, volume ranks with precision between −0.73 and −0.65 and with recall between +0.77 and +0.82.
Issues raised per pull request, by tool
Martian’s site default: the Claude Opus 4.5 judge and the Core profile, which counts bugs, security, concurrency, data, API, performance, test gaps and documentation defects — 158 known issues across the 50 pull requests. Sorted from the quietest reviewer to the loudest. The bar is known issues found (dark) and unmatched issues (orange) per pull request on one scale, so its dark share is the tool’s precision.
| Reviewer | Raised / PR | Found / PR | Unmatched / PR | Precision | Recall | Comments / PR |
|---|---|---|---|---|---|---|
| Graphitegraphite | 0.32 | 0.24 | 0.00 | 100.0% | 7.6% | 0.26 |
| Kilo+Grokkg | 0.96 | 0.50 | 0.42 | 54.3% | 15.8% | 1.80 |
| Devindevin | 2.08 | 1.30 | 0.58 | 69.1% | 41.1% | 2.74 |
| Bazbaz | 2.10 | 1.18 | 0.74 | 61.5% | 37.3% | 3.68 |
| Koduskodus-v2 | 2.10 | 1.22 | 0.84 | 59.2% | 38.6% | 5.80 |
| Macroscopemacroscope | 2.58 | 1.54 | 0.98 | 61.1% | 48.7% | 3.34 |
| Cursor Bugbotbugbot | 2.60 | 1.48 | 1.12 | 56.9% | 46.8% | 4.48 |
| Claude Code Pluginclaude | 2.94 | 1.20 | 1.58 | 43.2% | 38.0% | 2.38 |
| Qodo Deepqodo-extended-v2 | 3.04 | 2.04 | 1.00 | 67.1% | 64.6% | 1.00 |
| Greptilegreptile-v4-1 | 3.36 | 1.62 | 1.56 | 50.9% | 51.3% | 3.68 |
| GitLab Duo Code Reviewgitlab | 3.40 | 1.66 | 1.44 | 53.5% | 52.5% | 3.34 |
| CodeAntcodeant-v2 | 3.44 | 1.30 | 1.94 | 40.1% | 41.1% | 6.24 |
| Claude Code Reviewerclaude-code | 3.46 | 1.52 | 1.76 | 46.3% | 48.1% | 2.84 |
| Cubic Devcubic-v2 | 3.50 | 2.08 | 1.30 | 61.5% | 65.8% | 6.46 |
| Qodoqodo-v2 | 3.50 | 1.96 | 1.58 | 55.4% | 62.0% | 3.08 |
| Augmentaugment | 3.56 | 2.06 | 1.40 | 59.5% | 65.2% | 6.40 |
| Geminigemini-v2 | 4.36 | 1.74 | 2.10 | 45.3% | 55.1% | 5.02 |
| Sourcery AIsourcery | 4.54 | 1.74 | 2.50 | 41.0% | 55.1% | 5.82 |
| GitHub Copilotcopilot-v2 | 6.00 | 2.10 | 3.36 | 38.5% | 66.5% | 6.52 |
| CodeRabbitcoderabbit | 6.36 | 1.88 | 3.88 | 32.6% | 59.5% | 8.86 |
Precision and recall are Martian’s published figures, and they match its offline leaderboard (dated 8 September 2026) as we read it on 19 October 2026. Names are the leaderboard’s; the result id under each is Martian’s, so a row can be traced to its files. Martian collected the reviews between 23 January 2026 and 25 June 2026; the CSV has each tool’s dates. Gitar is on the leaderboard and not here: see Methodology.
Does saying more find more?
On this benchmark, up to a point, and at a cost in precision. A rank correlation of +1 would mean the reviewers line up in exactly the same order on both measures; −1, in exactly the opposite order. Martian judged every review three times, with three different models, so the same question can be asked three times:
| Judge | Tools | Raised vs precision | Raised vs recall | Raised vs unmatched | Comments vs precision |
|---|---|---|---|---|---|
| Claude Opus 4.5 (Martian’s default) | 20 | −0.65 | +0.82 | +0.90 | −0.47 |
| Claude Sonnet 4.5 | 20 | −0.70 | +0.77 | +0.93 | −0.39 |
| GPT-5.2 | 20 | −0.73 | +0.80 | +0.96 | −0.54 |
The more useful comparison is between the reviewers that score best and the ones that say the most. Under the default judge:
| Reviewer | Raised / PR | Found / PR | Unmatched / PR | Precision | Recall |
|---|---|---|---|---|---|
| Qodo Deepqodo-extended-v2 | 3.04 | 2.04 | 1.00 | 67.1% | 64.6% |
| Cubic Devcubic-v2 | 3.50 | 2.08 | 1.30 | 61.5% | 65.8% |
| Augmentaugment | 3.56 | 2.06 | 1.40 | 59.5% | 65.2% |
| CodeRabbitcoderabbit | 6.36 | 1.88 | 3.88 | 32.6% | 59.5% |
| GitHub Copilotcopilot-v2 | 6.00 | 2.10 | 3.36 | 38.5% | 66.5% |
Raising nearly twice as much found about the same number of known issues. What it added was issues that matched nothing on Martian’s list, each one something a developer has to read and decide about.
Comments are not issues
Counting GitHub comments is the obvious way to measure noise in a trial, and it is a weak one. Some reviewers put every finding in one summary comment; others post a status line, a summary and a dozen inline threads. Qodo Deep posted 1.00 comments per pull request and raised 3.04 issues in them; Kodus posted 5.80 and raised 2.10. Across the 20 tools, comments per pull request rank with precision at only −0.47, against −0.65 for issues raised.
If you trial a reviewer, count the distinct issues inside its comments, and note which ones your team acted on. That is the measure the rest of this page uses, and the one our guide to choosing a reviewer walks through.
What this does not show
- That unmatched means wrong. An unmatched issue matched none of the issues Martian’s curators listed. Some are wrong; some are real problems the list does not contain. Martian notes the list is sparse on style and nitpicks, so a reviewer that flags many correct small issues still sees some counted as false positives.
- Today’s products. Martian collected these reviews between 23 January 2026 and 25 June 2026. Vendors ship changes every few weeks, and a reviewer’s current version may say more or less.
- Causes. These are correlations across 20 products that differ in models, prompts, context and defaults, not an experiment that turned one dial.
- Your code. 50 pull requests from five large open-source projects. They are public, so a model may have seen them in training; Martian runs an online tracker on fresh pull requests partly for that reason, and it measures something different: how many of a bot’s suggestions developers went on to fix.
- One judge’s opinion. Matches are decided by a language model. The three judges move individual numbers by several points, which is why the correlations are shown under all three.
Methodology
The dataset: Per-tool figures for the 20 AI code reviewers on the Martian Code Review Bench offline leaderboard, computed from Martian’s public data at commit e616e84: the issues each reviewer raised per pull request, the known issues it found, the issues that matched no known issue, the GitHub comments it posted, and Martian’s precision and recall under the Core profile, for each of the benchmark’s three LLM judges. 50 pull requests from Sentry, Grafana, Cal.com, Discourse and Keycloak; 173 human-curated known issues, 158 of them in the Core profile.
Where the numbers come from
The Martian Code Review Bench is an open benchmark for AI code reviewers, published under the MIT License. Martian forked 50 pull requests from Sentry, Grafana, Cal.com, Discourse and Keycloak, let each reviewer review them, downloaded every comment each one posted, and had three language models judge the results against 173 human-curated issues. We used its repository at commit e616e84 (30 August 2026), and nothing else: no model calls of our own, no reviews of our own, no editing of Martian’s verdicts.
What each column means
- Raised: the distinct issues Martian’s extractor found in everything a reviewer posted on the pull request — inline comments, review bodies and summary comments. The extractor is the judge model, so the count differs slightly by judge.
- Found: known issues in the Core profile that the judge matched to something the reviewer raised.
- Unmatched: raised issues that matched no known issue and did not repeat one that did. Martian counts these as false positives.
- Precision and recall: found ÷ (found + unmatched), and found ÷ the Core known issues, as Martian computes them. Found plus unmatched is a little less than raised: repeats of a matched issue, and matches to style or speculative issues outside the profile, count as neither.
- Comments: every comment the reviewer’s account left on the pull request, as Martian downloaded them. It does not depend on the judge.
- Correlations: Spearman’s rank correlation across the reviewers in the table, ties given their average rank.
Checks
The script recomputes every reviewer’s counts from Martian’s per-review verdicts and refuses to write anything unless each count, and each percentage to its published decimal, equals Martian’s own dashboard file in the same commit. All 60 rows pass.
Left out: Gitar (gitar). It is on the leaderboard, but in this commit its evaluations were scored against 137 golden comments where its 50 pull requests have 173, and 27 of its matched or missed entries are not in the current golden files, so the leaderboard’s figures for it cannot be reproduced from Martian’s files. We left it out rather than estimate it.
Download and reproduce
- ai-code-review-noise.csv — one row per reviewer and judge (60 rows), with the per-PR averages, precision, recall and F1, the raw counts behind them, and the days Martian collected each reviewer’s comments.
- ai-code-review-noise.mjs — the script that produced it. Node 18 or later, no dependencies, no network.
git clone https://github.com/withmartian/code-review-benchmark martian-crb
git -C martian-crb checkout e616e849755441da38f18bf3adba2c9583b03803
curl -O https://www.superdiffs.com/data/ai-code-review-noise.mjs
node ai-code-review-noise.mjs --repo martian-crb
# writes ai-code-review-noise.csv and ai-code-review-noise.json; the CSV matches ours byte for byteSource data: Martian Code Review Bench, MIT License, Copyright (c) 2025 Martian (withmartian.com). Found a mistake? Write to hello@superdiffs.com. For how we measure our own reviewer, see the benchmark page; for how the same tools bill, see AI code review pricing, compared.