Configuration

Rules and strictness

Team rules, learned suppressions and the strictness thresholds that decide what gets posted.

You can shape reviews in three ways: a notes file in the repository, rules in the web app, and strictness thresholds. Changing any of them starts fresh reviews instead of reusing cached results.

The auditbench.md file#

Put your conventions in plain language in a file at the root of the repository. Audit Bench Ai reads the first of these it finds:

auditbench.md
.auditbench.md
.github/auditbench.md

auditbench.md

# Review notes for Audit Bench Ai

- Money is always integer cents (see src/money.ts). Flag any float arithmetic on amounts.
- All database access goes through src/db/ repositories; flag raw SQL elsewhere.
- Feature flags are read with flags.isOn(); hard-coded env checks are a bug.
- src/legacy/ is frozen: only flag security issues there.
- We target Node 20; don't suggest polyfills.

The file is added to the reviewer’s instructions for that repository. Keep it short and specific: the first 4,000 characters are used. Write what a new senior engineer would need to know to review your code, not general advice.

Note

There’s no YAML or JSON configuration file. Notes go in auditbench.md; rules and thresholds are set in the app.

Rules#

The Rules page holds rules that apply across your repositories, or to one repository. There are three kinds:

Kind	What it does	Example
Instruction	Tells the reviewer something to check or keep in mind	“Every API handler must check the caller owns the resource.”
Ignore files	Skips files matching a glob pattern	`src/generated/**`
Don’t flag	Stops a kind of finding from being raised	“Don’t flag missing JSDoc on internal functions.”

Each rule is up to 500 characters and can be limited to one repository (owner/repo).
Rules can be personal, or shared with your whole team.
Switch a rule off without deleting it.
Learned rules appear here too: kinds of finding your team kept dismissing. See how it learns.

Only team owners and admins can create or change shared rules. Anyone can manage their own personal rules.

Strictness#

Under Review strictness on the Rules page, you decide how sure the reviewer must be before it posts, and when code is complex enough to mention.

Setting	Default	Range	Effect
Minimum confidence	0.4	0.2 – 0.8	Findings less certain than this aren’t posted
Confidence for critical and high	0.6	0.4 – 0.95	Critical and high findings less certain than this are posted as medium
Deep nesting	4 levels	2 – 8	Nesting depth that earns a complexity finding
Very high complexity	25 paths	12 – 60	Paths through a function that earn a finding even when it’s flat
Optional suggestions	On	On / off	List 💡 suggestions in the pull request summary

Choosing values#

Too noisy? Raise the minimum confidence to 0.5–0.6, and turn optional suggestions off.
Missing things? Lower the minimum confidence to 0.3. Expect more findings that need a second look.
Blocking too often on uncertain issues? Raise the critical and high confidence to 0.75 or more.
Legacy code full of deep nesting? Raise the nesting and complexity limits rather than ignoring the files.

Team thresholds are set by owners and admins. If you’re not in a team, you set your own. Watch the review quality metrics after a change to see whether it helped.

Configuration

Rules and strictness

Team rules, learned suppressions and the strictness thresholds that decide what gets posted.

You can shape reviews in three ways: a notes file in the repository, rules in the web app, and strictness thresholds. Changing any of them starts fresh reviews instead of reusing cached results.

The auditbench.md file#

Put your conventions in plain language in a file at the root of the repository. Audit Bench Ai reads the first of these it finds:

auditbench.md
.auditbench.md
.github/auditbench.md

auditbench.md

# Review notes for Audit Bench Ai

- Money is always integer cents (see src/money.ts). Flag any float arithmetic on amounts.
- All database access goes through src/db/ repositories; flag raw SQL elsewhere.
- Feature flags are read with flags.isOn(); hard-coded env checks are a bug.
- src/legacy/ is frozen: only flag security issues there.
- We target Node 20; don't suggest polyfills.

Note

There’s no YAML or JSON configuration file. Notes go in auditbench.md; rules and thresholds are set in the app.

Rules#

The Rules page holds rules that apply across your repositories, or to one repository. There are three kinds:

Kind	What it does	Example
Instruction	Tells the reviewer something to check or keep in mind	“Every API handler must check the caller owns the resource.”
Ignore files	Skips files matching a glob pattern	`src/generated/**`
Don’t flag	Stops a kind of finding from being raised	“Don’t flag missing JSDoc on internal functions.”

Each rule is up to 500 characters and can be limited to one repository (owner/repo).
Rules can be personal, or shared with your whole team.
Switch a rule off without deleting it.
Learned rules appear here too: kinds of finding your team kept dismissing. See how it learns.

Only team owners and admins can create or change shared rules. Anyone can manage their own personal rules.

Strictness#

Under Review strictness on the Rules page, you decide how sure the reviewer must be before it posts, and when code is complex enough to mention.

Setting	Default	Range	Effect
Minimum confidence	0.4	0.2 – 0.8	Findings less certain than this aren’t posted
Confidence for critical and high	0.6	0.4 – 0.95	Critical and high findings less certain than this are posted as medium
Deep nesting	4 levels	2 – 8	Nesting depth that earns a complexity finding
Very high complexity	25 paths	12 – 60	Paths through a function that earn a finding even when it’s flat
Optional suggestions	On	On / off	List 💡 suggestions in the pull request summary

Choosing values#

Too noisy? Raise the minimum confidence to 0.5–0.6, and turn optional suggestions off.
Missing things? Lower the minimum confidence to 0.3. Expect more findings that need a second look.
Blocking too often on uncertain issues? Raise the critical and high confidence to 0.75 or more.
Legacy code full of deep nesting? Raise the nesting and complexity limits rather than ignoring the files.

Team thresholds are set by owners and admins. If you’re not in a team, you set your own. Watch the review quality metrics after a change to see whether it helped.