Skip to content
Documentation menu

Choosing an AI code reviewer for GitHub

An AI code reviewer reads each pull request and comments on it the way a colleague would. On GitHub there are three ways to run one, they differ in where your code goes and what they can post, and the products differ most in how much they say. This guide covers how each works, what to check before you pick one, and how to trial it fairly.

Three ways to run one

An AI reviewer on GitHub usually runs one of three ways: as a feature of GitHub itself, as a GitHub App that a vendor runs, or as a workflow you run in GitHub Actions. Which one it is decides who holds the model key, where the code is processed, and what happens on pull requests from forks.

Three ways to run an AI code reviewer on GitHub
Built into GitHubA GitHub AppA GitHub Actions workflow
ExampleCopilot code reviewA hosted reviewer you install from GitHubAn open-source review action, or a coding agent run in CI
Where it runsOn GitHub, with agentic steps on GitHub Actions runnersOn the vendor’s servers, triggered by GitHub webhooksOn your runners, GitHub-hosted or self-hosted
Who holds the model keyGitHubThe vendorYou, as a repository or organization secret
What it costsA paid Copilot plan, plus AI credits per review and Actions minutesThe vendor’s price: per seat, per author, per review or per creditYour model bill plus Actions minutes

Copilot code review’s costs and availability are from GitHub’s documentation, read on 19 October 2026: each review consumes AI credits (GitHub estimates $0.05 to $1 at Lite effort, $0.25 to $5 at Balanced) and, for its agentic steps, GitHub Actions minutes; it is not part of Copilot Free.

What a reviewer can post

Whichever way it runs, a reviewer has the same few ways to talk to you:

  • Review comments anchored to lines of the diff, and usually one summary comment for the whole pull request.
  • Suggested changes: a comment that carries replacement code. Anyone with write access can apply one, or a batch of them, as a single commit; the reviewer that suggested it never pushes.
  • A check run: a result in the pull request’s checks, such as success, failure or neutral. Only GitHub Apps can create check runs through the API.
  • An approval or a change request, from reviewers that submit them. GitHub says Copilot’s reviews do not count toward required approvals by default.

A check run only blocks a merge if you make it a required status check in branch protection; GitHub then requires it to finish as successful, skipped or neutral, and you can insist it comes from a particular app. Leave the reviewer’s check optional until you trust it.

What to check before you choose

How much it says

The biggest difference between reviewers is volume. On the same 50 pull requests in the public Martian Code Review Bench, 20 reviewers raised between 0.3 and 6.4 issues per pull request. The three with the highest F1 — Qodo Deep, Cubic Dev and Augment — raised 3.0–3.6 and found 2.0–2.1 known issues per pull request; the two loudest raised 6.0–6.4 and found about as many. Every reviewer’s numbers, and the data, are on how many comments AI code reviewers post.

What happens on the second push

Resolve a comment, push a fix, and watch whether it comes back. A reviewer that re-posts what you already dismissed teaches a team to stop reading it. Check too whether it answers replies, and whether a reply can change what it does next time.

The permissions it asks for

GitHub shows an app’s permissions on the install screen, lets you grant it all repositories or only selected ones, and advises apps to request the minimum. A reviewer needs to read code and to write pull request comments and checks. Write access to contents lets an app push commits: reasonable for a tool that fixes code for you, unnecessary for one that only reviews.

Where your code goes

Ask four questions: is the code stored after the review, and for how long; is it used to train models, and can you opt out; which model providers see it, and under what retention terms; and which certifications back the answers. Vendors answer differently — Greptile and CodeRabbit give two different answers to the training question, for example. If none of the answers is acceptable, the options are self-hosting or a workflow on your own runners.

How it bills

Per seat, per pull request author, per active developer, per review, per credit or per reviewed file: the unit decides whether a busy month costs more. The reviewers and their units are on AI code review pricing, compared.

Everything else

  • Other Git hosts. If some of your code lives on GitLab, Bitbucket or Azure DevOps, check each vendor’s list; coverage varies widely.
  • Configuration in the repository. A file such as .coderabbit.yaml, Greptile’s .greptile/ folder or .superdiffs.yaml keeps settings reviewable, and lets you exclude generated code and state rules a reviewer cannot infer from the diff.
  • What it skips. Lockfiles, generated code and very large pull requests are commonly skipped or capped; find out how the tool tells you.

How to set one up

For a reviewer installed as a GitHub App:

  1. Pick one or two repositories with steady pull request traffic that your team knows well. A first impression on familiar code is worth more than coverage.
  2. Install the app from GitHub and choose Only select repositories. Read the permissions before you approve them.
  3. Open a pull request, or push to one, and read what arrives: the summary, the inline comments and the check run.
  4. Commit the tool’s configuration file with path filters for generated code, fixtures and vendored directories, and write down the rules your team already enforces in review.
  5. Exclude bot authors such as Dependabot and Renovate if the tool reviews them by default, and check whether they count toward your bill.
  6. Keep the check optional for the trial. Decide about requiring it afterwards.

For superdiffs, the quickstart covers each step, including the permissions it asks for and a starter configuration file.

How to run a fair trial

Two weeks on the same repositories is enough to see a reviewer’s habits. If you are choosing between two, run both on the same pull requests over the same weeks. Then count:

  1. Issues raised per pull request — the distinct problems inside its comments, not the number of comments. A single summary comment can hold five.
  2. Acted on: issues that led to a change or a reply that settled them. Acted on ÷ raised is the precision your team actually experienced.
  3. Wrong: issues that were simply incorrect. These cost the most, because someone had to work out that they were wrong.
  4. Missed: bugs in code the reviewer looked at that turned up later, in review, testing or production. Reviewing a pull request you already know has a bug in it is the quickest way to get one data point.

Compare the result with the public numbers on the noise benchmark, and with what each vendor says it does on our comparison pages.

Common questions

Can an AI code reviewer approve pull requests on GitHub?

Some can. GitHub says Copilot code review’s reviews do not count toward required approvals by default, and that approving reviews are in public preview. Reviewers installed as GitHub Apps decide for themselves; superdiffs only ever comments and reports a check run.

Do AI code reviewers work on pull requests from forks?

A reviewer that runs as a GitHub Actions workflow usually cannot use its model key there: GitHub does not pass secrets, other than a read-only GITHUB_TOKEN, to workflows triggered from a fork. The pull_request_target event runs with the base repository’s context instead, and GitHub warns against using it to build or run code from the pull request. A reviewer installed as a GitHub App runs on its vendor’s side, so that rule does not apply to it.

What permissions does an AI code reviewer need on GitHub?

Enough to read the code and to write comments and checks on pull requests. GitHub advises apps to request the minimum they need. superdiffs asks for read access to contents and metadata and write access to pull requests, issues and checks; write access to contents would let an app push commits.

How many comments should an AI code reviewer post?

There is no right number, but there is public data. On the Martian Code Review Bench, the three reviewers with the highest F1 raised 3.0–3.6 issues per pull request; the two loudest raised 6.0–6.4, found about as many known issues, and had 3.4–3.9 per pull request that matched none, against 1.0–1.4.

Sources

Every statement about GitHub on this page was checked against its documentation on 19 October 2026:

We make superdiffs, one of the reviewers this guide is about, so weigh our advice accordingly. The guide names other products only where their own pages back the statement, and the benchmark figures are Martian’s.