Read a rule over seeds

This cookbook declares three rules that read the seeds themselves, not only their mean: a count, a best of N and a relative change.

A mean can hide a bad seed, and a plan can ask about each seed rather than the mean. rb 0.8.0 reads a rule over the units themselves: how many meet a threshold, which one is best, or by what percent the arms’ means differ.

Paste this into your agent

Record the five seeds, and the plan's three rules over them.

Use Rabbit Brain to check INT8 against FP32 over the five seeds in results/int8-ptq/. docs/plan.md says what counts, one rule a line.

- Run `rb context --model <your model id>` first.
- Record the question and the experiment, with each setting cited to the config that states it.
- Record each of the plan's three rules as rb reads it, cited to its line: a count (`--count 4 --of 5`), a best of N (`--best --of 5`) and a relative change (`--relative`).
- Record each seed as a run.
- Compute nothing yourself: rb computes the findings.

Finish with `rb status` and `rb show` of each criterion exactly as printed.

The repository evaluates an FP32 ResNet-20 and its INT8 copy on CIFAR-10 for five seeds, each into results/int8-ptq/seed<N>/. docs/plan.md holds the three rules, on lines 4 to 6:

“INT8 reaches at least 91.58% top-1 on at least 4 of the 5 seeds.”
“INT8’s best seed of the 5 reaches at least 91.65% top-1.”
“INT8 loses at most 0.1% of FP32’s top-1, as a relative change of the means.”

What you’ll see

rb status in your own terminal, once the agent is done:

$ rb status
INT8 ResNet-20 on CIFAR-10 · int8-resnet-20-on-cifar-10
Kept with rb since 2026-11-06. All 5 runs were recorded afterwards, not as they happened.

Does INT8 hold up against FP32 across the five seeds? (open)
  On every int8-seeds criterion below: question link declared after the evidence existed: all 5 units were produced before it was declared; units: 5 seeds.
  INT8's best of 5 seeds reaches 91.65% top-1. int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb); point only: a best of N is one unit's own value, with no interval; recorded values meet criterion: best of 5 seeds, seed 3: 91.69 ≥ 91.65; rule and question link declared after the evidence existed: all 5 units were produced before it was declared.
  INT8 reaches 91.58% top-1 on at least 4 of 5 seeds. int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb); measured: int8 on its own, with its interval, so neither better nor worse; recorded values meet criterion: 4 of 5 seeds at least 91.58; the rule asks for 4; rule and question link declared after the evidence existed: all 5 units were produced before it was declared.
  INT8 loses at most 0.1% of FP32's top-1. change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb); loss: the interval clear of 0 on the worse side (higher is better); estimate meets criterion: estimate -0.0916% ≥ -0.1%; rule and question link declared after the evidence existed: all 5 units were produced before it was declared. the interval [-0.1329, -0.05029] crosses -0.1: met on the estimate, not across the whole interval.

Rules: 3 met, all 3 on the estimate alone

The design of 1 experiment is not confirmed: rb confirm lists it.

Facts that ask for nothing (1)
  - int8-seeds: 3 criteria were declared after their evidence existed

rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-06 20:25 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.

Rules
  four-of-five INT8 reaches 91.58% top-1 on at least 4 of 5 seeds
          recorded values meet criterion · measured · 4 of 5 units' int8.top1 ≥ 91.58
          reads:     int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb)
          result:    measured: int8 on its own, with its interval, so neither better nor worse
          rule:      recorded values meet criterion: 4 of 5 seeds at least 91.58; the rule asks for 4
          timing:    declared after the evidence existed: all 5 units were produced before it was declared
          units:     5 seeds
  best-of-five INT8's best of 5 seeds reaches 91.65% top-1
          recorded values meet criterion · point only · the best of 5 units' int8.top1 ≥ 91.65
          reads:     int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb)
          result:    point only: a best of N is one unit's own value, with no interval
          rule:      recorded values meet criterion: best of 5 seeds, seed 3: 91.69 ≥ 91.65
          timing:    declared after the evidence existed: all 5 units were produced before it was declared
          units:     5 seeds
  relative-drop INT8 loses at most 0.1% of FP32's top-1
          estimate meets criterion · loss · change.top1 as a relative change ≥ -0.1%
          reads:     change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb)
          result:    loss: the interval clear of 0 on the worse side (higher is better)
          rule:      estimate meets criterion: estimate -0.0916% ≥ -0.1%
          timing:    declared after the evidence existed: all 5 units were produced before it was declared
          units:     5 seeds
          · the interval [-0.1329, -0.05029] crosses -0.1: met on the estimate, not across the whole interval

Experiments
  int8-seeds INT8 post-training quantization, five seeds · open · fp32 (baseline) vs int8 (candidate) · varies precision, calib_images · not frozen · settings 7 verified · 0 provisional · 0 unknown · 1 per-run · runs 5
          to run again: code unknown · environment unknown · files ok

Runs: 5 recorded on 1 experiment · every receipt is what rb wrote

Next: rb confirm

four-of-five, best-of-five and relative-drop are the criteria’s ids: their names in every command.
The count names both numbers: how many seeds meet the threshold, and how many the rule asks for.
The best of N names its seed and that seed’s own value. It has no interval: “a best of N is one unit’s own value, with no interval”.
The relative change is said in percent. Its interval crosses -0.1, so it reads estimate meets criterion, met on the estimate, not across the whole interval.
A met count or best of N reads recorded values meet criterion, since the recorded seeds themselves meet it. The tally counts all three among the rules met on the estimate alone.

What rb did, step by step

Declare the three rules

Each is said in rb’s own words: “4 of 5 units’ int8.top1 ≥ 91.58”, “the best of 5 units’ int8.top1 ≥ 91.65” and “change.top1 as a relative change ≥ -0.1%”.

$ rb criterion add "INT8 reaches 91.58% top-1 on at least 4 of 5 seeds" -e int8-seeds -q q-seeds --metric int8.top1 --at-least 91.58 --count 4 --of 5 --why docs/plan.md:4 --id four-of-five
four-of-five: INT8 reaches 91.58% top-1 on at least 4 of 5 seeds
  4 of 5 units' int8.top1 ≥ 91.58 · primary · on int8-seeds
  declared; no evidence recorded yet

Next: rb run add int8-seeds --from <the results file a run wrote> --set seed=<value>

$ rb criterion add "INT8's best of 5 seeds reaches 91.65% top-1" -e int8-seeds -q q-seeds --metric int8.top1 --at-least 91.65 --best --of 5 --why docs/plan.md:5 --id best-of-five
best-of-five: INT8's best of 5 seeds reaches 91.65% top-1
  the best of 5 units' int8.top1 ≥ 91.65 · primary · on int8-seeds
  declared; no evidence recorded yet

Next: rb run add int8-seeds --from <the results file a run wrote> --set seed=<value>

$ rb criterion add "INT8 loses at most 0.1% of FP32's top-1" -e int8-seeds -q q-seeds --metric change.top1 --at-least -0.1 --relative --why docs/plan.md:6 --id relative-drop
relative-drop: INT8 loses at most 0.1% of FP32's top-1
  change.top1 as a relative change ≥ -0.1% · primary · on int8-seeds
  declared; no evidence recorded yet

Next: rb run add int8-seeds --from <the results file a run wrote> --set seed=<value>

Record each seed

After each seed, rb run add reads all three. The count and the best of N read undetermined until the runs hold the 5 seeds the rule is of.

$ rb run add int8-seeds --from results/int8-ptq/seed1/metrics.json --set seed=1 --config results/int8-ptq/seed1/config.yaml
run7hiwzs on int8-seeds: fp32.top1=91.71, int8.top1=91.6 · seed=1
  Ran with the spec: 5 of its settings found in results/int8-ptq/seed1/config.yaml, all matching.
  Produced: 2026-10-20 10:00:00 (file)
  four-of-five criterion undetermined · point only: int8.top1: 91.6 % · mean, 1 seed (rb)
  best-of-five criterion undetermined · point only: int8.top1: 91.6 % · best unit seed 1, 1 seed (rb)
  relative-drop criterion not met · point only: change.top1: -0.1199 % · relative change of the means, paired, 1 seed (rb)
  rb put the code in git at refs/rb/snapshots/d5832f1e0ac5298950edf64261220c331f21c020, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/d5832f1e0ac5298950edf64261220c331f21c020 removes it.

Since this session last looked:
  - four-of-five: criterion undetermined: the rule is of 5 seeds; the runs hold 1   (rb decide four-of-five accept|reject|investigate --why "...")
  - best-of-five: criterion undetermined: the rule is of 5 seeds; the runs hold 1   (rb decide best-of-five accept|reject|investigate --why "...")
  - relative-drop misses its rule and no call on it is recorded: INT8 loses at most 0.1% of FP32's top-1   (rb decide relative-drop accept|reject|investigate --why "...")
  - 4 results folders hold results written before the record began, which no recorded experiment reads yet: results/int8-ptq/seed2/, results/int8-ptq/seed3/, results/int8-ptq/seed4/ and 1 more   (rb reconcile)

Next: rb show four-of-five  ·  rb show best-of-five  ·  rb show relative-drop

$ rb run add int8-seeds --from results/int8-ptq/seed2/metrics.json --set seed=2 --config results/int8-ptq/seed2/config.yaml
runh4wdyl on int8-seeds: fp32.top1=91.66, int8.top1=91.58 · seed=2
  Ran with the spec: 5 of its settings found in results/int8-ptq/seed2/config.yaml, all matching.
  Produced: 2026-10-20 10:00:00 (file)
  four-of-five criterion undetermined · point only: int8.top1: 91.59 % · mean, 2 seeds (rb)
  best-of-five criterion undetermined · point only: int8.top1: 91.6 % · best unit seed 1, 2 seeds (rb)
  relative-drop criterion not met · point only: change.top1: -0.1036 % · relative change of the means, paired, 2 seeds (rb)

Since this session last looked:
  - 3 results folders hold results written before the record began, which no recorded experiment reads yet: results/int8-ptq/seed3/, results/int8-ptq/seed4/, results/int8-ptq/seed5/   (rb reconcile)

Next: rb show four-of-five  ·  rb show best-of-five  ·  rb show relative-drop

$ rb run add int8-seeds --from results/int8-ptq/seed3/metrics.json --set seed=3 --config results/int8-ptq/seed3/config.yaml
runcmjjy2 on int8-seeds: fp32.top1=91.74, int8.top1=91.69 · seed=3
  Ran with the spec: 5 of its settings found in results/int8-ptq/seed3/config.yaml, all matching.
  Produced: 2026-10-20 10:00:00 (file)
  four-of-five criterion undetermined · measured: int8.top1: 91.62 [91.48, 91.77] % · mean, t interval, 3 seeds, 95% (rb)
  best-of-five criterion undetermined · point only: int8.top1: 91.69 % · best unit seed 3, 3 seeds (rb)
  relative-drop estimate meets criterion · loss: change.top1: -0.08724 [-0.1685, -0.005936] % · relative change of the means, paired, delta-method t interval, 3 seeds, 95% (rb)

Since this session last looked:
  - 2 results folders hold results written before the record began, which no recorded experiment reads yet: results/int8-ptq/seed4/, results/int8-ptq/seed5/   (rb reconcile)

Next: rb show four-of-five  ·  rb show best-of-five  ·  rb show relative-drop

$ rb run add int8-seeds --from results/int8-ptq/seed4/metrics.json --set seed=4 --config results/int8-ptq/seed4/config.yaml
runn4ysbk on int8-seeds: fp32.top1=91.69, int8.top1=91.57 · seed=4
  Ran with the spec: 5 of its settings found in results/int8-ptq/seed4/config.yaml, all matching.
  Produced: 2026-10-20 10:00:00 (file)
  four-of-five criterion undetermined · measured: int8.top1: 91.61 [91.52, 91.7] % · mean, t interval, 4 seeds, 95% (rb)
  best-of-five criterion undetermined · point only: int8.top1: 91.69 % · best unit seed 3, 4 seeds (rb)
  relative-drop estimate meets criterion · loss: change.top1: -0.09815 [-0.153, -0.04325] % · relative change of the means, paired, delta-method t interval, 4 seeds, 95% (rb)

Since this session last looked:
  - results/int8-ptq/seed5/ holds results written before the record began, which no recorded experiment reads yet   (rb reconcile)

Next: rb show four-of-five  ·  rb show best-of-five  ·  rb show relative-drop

$ rb run add int8-seeds --from results/int8-ptq/seed5/metrics.json --set seed=5 --config results/int8-ptq/seed5/config.yaml
runaiu2l6 on int8-seeds: fp32.top1=91.72, int8.top1=91.66 · seed=5
  Ran with the spec: 5 of its settings found in results/int8-ptq/seed5/config.yaml, all matching.
  Produced: 2026-10-20 10:00:00 (file)
  four-of-five recorded values meet criterion · measured: int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb)
  best-of-five recorded values meet criterion · point only: int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb)
  relative-drop estimate meets criterion · loss: change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb)

Next: rb show four-of-five  ·  rb show best-of-five  ·  rb show relative-drop

Find each rule in format 2

A criterion that holds one of these rules is kept in record format 2. Every other file stays rb: 1:
Terminal window
```
$ head -3 .rb/experiments/int8-seeds/criteria/four-of-five.yaml
rb: 2
kind: criterion
id: four-of-five
```

Read each criterion

$ rb show four-of-five
Criterion four-of-five on int8-seeds (primary, own): INT8 reaches 91.58% top-1 on at least 4 of 5 seeds
  four-of-five INT8 reaches 91.58% top-1 on at least 4 of 5 seeds
          recorded values meet criterion · measured · 4 of 5 units' int8.top1 ≥ 91.58
          reads:     int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb)
          result:    measured: int8 on its own, with its interval, so neither better nor worse
          rule:      recorded values meet criterion: 4 of 5 seeds at least 91.58; the rule asks for 4
          timing:    declared after the evidence existed: all 5 units were produced before it was declared
          units:     5 seeds
          integrity: ok
  run7hiwzs: int8.top1 91.6 (file) · unit seed=1 · produced before the declaration · seed=1
  runh4wdyl: int8.top1 91.58 (file) · unit seed=2 · produced before the declaration · seed=2
  runcmjjy2: int8.top1 91.69 (file) · unit seed=3 · produced before the declaration · seed=3
  runn4ysbk: int8.top1 91.57 (file) · unit seed=4 · produced before the declaration · seed=4
  runaiu2l6: int8.top1 91.66 (file) · unit seed=5 · produced before the declaration · seed=5

$ rb show best-of-five
Criterion best-of-five on int8-seeds (primary, own): INT8's best of 5 seeds reaches 91.65% top-1
  best-of-five INT8's best of 5 seeds reaches 91.65% top-1
          recorded values meet criterion · point only · the best of 5 units' int8.top1 ≥ 91.65
          reads:     int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb)
          result:    point only: a best of N is one unit's own value, with no interval
          rule:      recorded values meet criterion: best of 5 seeds, seed 3: 91.69 ≥ 91.65
          timing:    declared after the evidence existed: all 5 units were produced before it was declared
          units:     5 seeds
          integrity: ok
  run7hiwzs: int8.top1 91.6 (file) · unit seed=1 · produced before the declaration · seed=1
  runh4wdyl: int8.top1 91.58 (file) · unit seed=2 · produced before the declaration · seed=2
  runcmjjy2: int8.top1 91.69 (file) · unit seed=3 · produced before the declaration · seed=3
  runn4ysbk: int8.top1 91.57 (file) · unit seed=4 · produced before the declaration · seed=4
  runaiu2l6: int8.top1 91.66 (file) · unit seed=5 · produced before the declaration · seed=5

$ rb show relative-drop
Criterion relative-drop on int8-seeds (primary, own): INT8 loses at most 0.1% of FP32's top-1
  relative-drop INT8 loses at most 0.1% of FP32's top-1
          estimate meets criterion · loss · change.top1 as a relative change ≥ -0.1%
          reads:     change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb)
          result:    loss: the interval clear of 0 on the worse side (higher is better)
          rule:      estimate meets criterion: estimate -0.0916% ≥ -0.1%
          timing:    declared after the evidence existed: all 5 units were produced before it was declared
          units:     5 seeds
          integrity: ok
          · the interval [-0.1329, -0.05029] crosses -0.1: met on the estimate, not across the whole interval
  run7hiwzs: int8.top1 91.6 (file) · unit seed=1 · produced before the declaration · seed=1
  run7hiwzs: fp32.top1 91.71 (file) · unit seed=1 · produced before the declaration · seed=1
  runh4wdyl: int8.top1 91.58 (file) · unit seed=2 · produced before the declaration · seed=2
  runh4wdyl: fp32.top1 91.66 (file) · unit seed=2 · produced before the declaration · seed=2
  runcmjjy2: int8.top1 91.69 (file) · unit seed=3 · produced before the declaration · seed=3
  runcmjjy2: fp32.top1 91.74 (file) · unit seed=3 · produced before the declaration · seed=3
  runn4ysbk: int8.top1 91.57 (file) · unit seed=4 · produced before the declaration · seed=4
  runn4ysbk: fp32.top1 91.69 (file) · unit seed=4 · produced before the declaration · seed=4
  runaiu2l6: int8.top1 91.66 (file) · unit seed=5 · produced before the declaration · seed=5
  runaiu2l6: fp32.top1 91.72 (file) · unit seed=5 · produced before the declaration · seed=5

A team on rb 0.7.0 or 0.7.1 cannot write to this record: they write nothing to a record that holds a format-2 file, and say to upgrade. Upgrade everyone who shares a record together.

Why it matters

“The mean meets 91.58” and “4 of 5 seeds meet 91.58” are different claims. rb reads the one the plan makes.

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0