Criteria

This page says what a criterion is, what it compares, the rules it can apply, and how rb reads when it was declared.

A criterion says what would count, before the numbers come in: “INT8 loses at most 0.5 points of top-1 against FP32.” It reads one comparison and applies one rule, and rb records who declared it and when. rb computes its finding from the runs every time the record is read.

rb criterion add "INT8 loses at most 0.5 points of top-1 against FP32" \
  -e int8-ptq -q q-int8 --metric change.top1 --at-least -0.5 --min-n 3 --why "docs/plan.md:3"

What it compares

--metric names a number the runs report:

--metric	What it reads
top1	a number named top1, or the only candidate’s when the runs name it by variant
int8.top1	the int8 variant’s number, a level
candidate.top1	the candidate’s number
change.top1	the candidate’s number minus the baseline’s, from the same run
top1 with --subject fp32 --reference int8	FP32’s number against INT8’s

Record each variant’s own numbers, such as {"fp32": {"top1": 91.71}, "int8": {"top1": 91.6}}, never a difference you worked out: rb works out change.top1 itself. --where, --subject-where and --reference-where select runs by their per-run settings.

Its rule

Each criterion has exactly one:

Rule	Holds when
--at-most X, --at-least X	the estimate, the mean over the runs’ units, is at most X, or at least X
--equals X --tolerance T	the estimate is within T of X
--each (with a threshold)	every independent unit meets the threshold, not only the estimate
--count K --of N, --best --of N, --relative, --means	see Four more kinds of rule
--bound (with a threshold)	the interval’s bound meets it, not only the estimate
--better	the interval is clear of 0 on the better side
--noninferior M	the interval’s worse bound is inside M
--equivalent M	the interval is inside ±M

The last three read the interval, so they need one: rb gives an interval from 3 units, such as three seeds. --better and --noninferior also need to know which way is better, which the metric says: rb metric add top1 --maximize. Without it they are refused.

A threshold is read one way. Two of --each, --bound, --count and --best together are refused: “A threshold is read one way: --each and --bound each say a different one.”

Four more kinds of rule

rb 0.8.0 adds four kinds of rule. Each reads a threshold its own way, and each is said in the same words by rb criterion add, rb show, rb status, rb context, the SDK and MCP:

Rule	What it reads	As rb says it
--count K --of N	whether at least K of the N units meet the threshold, each unit judged as --each judges it	“4 of 5 units’ int8.top1 ≥ 91.58”
--best --of N	the unit best in the threshold’s direction, its own value, read against the threshold	“the best of 5 units’ int8.top1 ≥ 91.65”
--relative	the relative change of the two arms’ means, (subject - reference) / reference, in percent; its threshold is a percent	“change.top1 as a relative change ≥ -0.1%”
--means, beside --source and --quote	the cited number as a difference of the arms’ means over all their units	“change.val_loss as a difference of means = 0.0317 ± 0.00005”

• A count names both numbers: “recorded values meet criterion: 4 of 5 seeds at least 91.58; the rule asks for 4”. Where the runs hold another number of units than the rule is of, or fewer than K, it reads undetermined: “criterion undetermined: the rule is of 5 seeds; the runs hold 1”.
• A best of N takes --at-most or --at-least, and has no interval: “a best of N is one unit’s own value, with no interval”.
• A relative change compares two arms. From 3 units its interval is a delta-method t interval: “relative change of the means, paired, delta-method t interval”.
• A cited difference of means reads “difference of means, as cited”, with a Welch interval once each arm holds 3 units. rb sets the cited number against that difference of means, never against its paired reading, so a unit that only one arm ran does not make the rule read “not comparable”.

A met count or best of N reads recorded values meet criterion, since the recorded units themselves meet it. A relative change and a difference of means take a threshold on their estimate, or with --bound on their interval’s bound.

A criterion that holds one of the four is kept in record format 2 (rb: 2), in its own file. Every other file stays in format 1: see The record’s format. Read a rule over seeds declares a count, a best of N and a relative change over five seeds.

--min-n 3 says how many independent units the reading must rest on: with fewer, the rule reads undetermined. --noise 0.01 says how much a result moves from run to run: an estimate that close to the threshold gets a note that another unit may land on the other side.

Where its number comes from

A criterion’s threshold is a file’s or yours, never the agent’s own:

• A plan that states it: --why "docs/plan.md:3". rb refuses the criterion when the line is not there, is a comment, or does not state the threshold.
• A paper or another document: --source docs/baselines/resnet-paper.md --quote "| ResNet | 20 | 0.27M | 8.75 |" --locator "Table 6". rb checks that the file holds the quote and that it states the number. A cited criterion reads reproduced or not reproduced, and reproduced on the estimate where the reading has no interval or one that crosses its line.
• Your words: --words "dropout passes only if the whole 95% interval stays within 0.02 nats", which rb keeps verbatim beside the criterion. Words an agent passes read as words (as given), since rb cannot know you said them.

Its role

--role primary, the default, is what the work claims. secondary and exploratory are context. A gate is a priority policy for where to spend effort, not a claim: a missed gate asks for a call to park or continue the line, and never refutes anything.

When it was declared

A criterion is declared when it is written. rb compares that moment with when its evidence existed, and says it on every finding:

• declared before the evidence: written before the runs it reads were produced;
• declared after the evidence existed: some or all of those runs already existed, and rb says how many;
• declared after the data was seen: the data it evaluates on was looked at first;
• order unknown: nobody knows when some of its evidence was produced. Numbers typed in on the command line or through MCP with no --produced-at have no time of their own, so a rule over them reads this.

Evidence that already existed still counts. It is never hidden, and never read as if the criterion came first. Declare the criterion before the runs shows both timings side by side.

Freezing

rb freeze <experiment> -m "before the runs" locks the experiment’s variants, settings and criteria, and records a hash of them. After that, a change fails with E_FROZEN unless the same command gives its reason with --amend --why "...". The amendment is kept, with who made it and the hash before and after.

Which question it answers

-q <question> links a criterion to the question its finding bears on, and a question’s reading counts only the criteria linked to it. --when-met "..." keeps, verbatim, what meeting the rule would mean for that question. When the rule misses, rb never says the opposite.