Get started
How reviews work
The two-stage pipeline behind every review: free local checks first, then a focused AI pass.
Every review runs in two stages. The first is deterministic and free; the second is a focused AI review that only sees what it needs. The aim is a few findings that are right, not many that might be.
Stage 1: local checks#
These run on our servers without any AI and never count toward your plan’s limits:
- Linters and type checks: ESLint, the TypeScript compiler and Prettier for JavaScript and TypeScript, plus Python checks.
- Semgrep rules and risky-pattern detection.
- Secret scanning for keys, tokens and passwords added in the change.
- Complexity and nesting analysis. Functions that are deeply nested or have too many paths are flagged, and the riskiest functions are picked out for the AI to look at closely.
Stage 2: the AI review#
Each changed file gets one AI call. For a pull request, the model is given:
- The changed lines with a little surrounding code.
- Callers: short excerpts of code elsewhere in the repository that uses the changed functions, so it can tell when a change breaks something that isn’t in the diff.
- Related tests: test files for the changed code, so it can judge whether behavior is covered.
- The pull request’s title and description, your team’s rules and your repository’s
auditbench.md.
Repository context is read for repositories up to 50 MB. Larger ones are reviewed with the diff and the pull request alone.
What the model is asked for#
The model has to back every finding with evidence from the code: the line that fails, the input that triggers it, and what happens next (“When it breaks”). It rates two things separately:
- Severity: how bad it would be if it’s real.
- Confidence: how sure it is that it’s real. Above 0.85 means proven by the code in view; 0.6–0.85 strongly suggested; below 0.6 needs checking.
Calibration#
Before anything is posted, findings go through fixed rules, so the same model output always publishes the same way:
| Rule | Why |
|---|---|
| Findings below the minimum confidence (0.4 by default) are dropped | A guess shouldn’t be presented as a defect |
| Critical and high findings below 0.6 confidence become medium | An unconfirmed concern shouldn’t block a merge on its own |
| Maintainability and Architecture findings stop at medium | Code health is worth raising, but rarely a reason to stop a merge |
| Two findings of the same kind on overlapping lines are merged | One problem, one comment |
| Kinds of finding your team keeps dismissing are dropped | See learned suppressions in Rules and strictness |
The thresholds are adjustable per team. See Rules and strictness.
No repeated comments#
When you push again, the new review is compared with comments it already left on the pull request, matched by file and the kind of problem, not by line number, so moving code around doesn’t create duplicates:
- Still open from an earlier review: not posted again.
- Dismissed by your team (👎 or agreed “not an issue”): not posted again.
- Marked fixed but detected again: posted, marked ♻️ reappeared.
- Its lines changed but the problem is still there: posted, marked 🔁 still present (GitHub and Bitbucket).
The summary says how many findings were held back and why.
Caching#
Results are cached by the code, your rules, the context and the engine version. Re-reviewing unchanged code is served from the cache and doesn’t use your plan’s limits. Changing your rules or strictness starts fresh reviews.