Compare against a published baseline

This cookbook checks your runs against a number a paper reports, on the paper’s own settings.

“We match the paper” is only as good as the comparison under it: the same number, read the same way, with the same settings. Save the paper’s text in the repository, cite it, and let rb read your runs against it.

Paste this into your agent

Check our run against the paper's number, with the paper's settings cited.

Check our ResNet-20 against the paper's 8.75% error with Rabbit Brain. The paper's numbers are saved in docs/baselines/resnet-paper.md. Count it as reproduced if we land within 0.3 points of the paper's 8.75% error.

- Run `rb context --model <your model id>` first.
- Record the experiment, and each training setting from configs/train.yaml with the value the paper used, cited to its line in docs/baselines/resnet-paper.md (`--cited`, `--cited-source`).
- Record the paper's number as a criterion cited to its table row (`--source`, `--quote`, `--locator "Table 6"`), with my tolerance and my words.
- Run seeds 1 to 3 through `rb journal run`, and record each run.

Finish with `rb show` of the criterion exactly as printed, and every setting that differs from the paper's.

The repository trains a ResNet-20 on CIFAR-10. It keeps the paper’s text it relies on in docs/baselines/resnet-paper.md: the training settings from section 4.2 and the ResNet-20 row of Table 6.

# Deep Residual Learning for Image Recognition (2016): excerpts

Section 4.2, CIFAR-10:

We use a weight decay of 0.0001 and momentum of 0.9.
These models are trained with a mini-batch size of 128.
We start with a learning rate of 0.1.

Table 6, classification error on the CIFAR-10 test set:

| method | layers | params | error (%) |
|---|---|---|---|
| ResNet | 20 | 0.27M | 8.75 |
| ResNet | 32 | 0.46M | 7.51 |

What you’ll see

$ rb show matches-paper-error
Criterion matches-paper-error on repro (primary, cited): Our ResNet-20 lands within 0.3 points of the paper's 8.75% error · words (as given): "Count it as reproduced if we land within 0.3 points of the paper's 8.75% error."
  cited: docs/baselines/resnet-paper.md "| ResNet | 20 | 0.27M | 8.75 |" (Table 6) · states 8.75, read '| ResNet | 20 | 0.27M | 8.75 |'
  matches-paper-error Our ResNet-20 lands within 0.3 points of the paper's 8.75% error
          reproduced · measured · resnet20.error = 8.75 ± 0.3 (n ≥ 3)
          reads:     resnet20.error: 8.747 [8.597, 8.896] % · mean, t interval, 3 seeds, 95% (rb)
          result:    measured: resnet20 on its own, with its interval, so neither better nor worse
          rule:      reproduced: estimate 8.747 within 8.75 ± 0.3
          timing:    declared before the evidence (by production times)
          units:     3 seeds
          integrity: ok
  run5s7da5: resnet20.error 8.81 (file) · unit seed=1 · seed=1
  runlycbpg: resnet20.error 8.69 (file) · unit seed=2 · seed=2
  run6aw4sv: resnet20.error 8.74 (file) · unit seed=3 · seed=3

matches-paper-error is the criterion’s id: its name in every command. Reproduced: the three seeds average 8.747% error, within 0.3 points of the paper’s 8.75%. The criterion cites the table row it read, and your words stay beside it, as given.

When a setting differs from the paper’s

We ran the same cookbook again with one change: our config trains with a batch size of 256, where the paper used 128. rb does not compare numbers made under different settings:

$ rb show matches-paper-error
Criterion matches-paper-error on repro (primary, cited): Our ResNet-20 lands within 0.3 points of the paper's 8.75% error
  cited: docs/baselines/resnet-paper.md "| ResNet | 20 | 0.27M | 8.75 |" (Table 6) · states 8.75, read '| ResNet | 20 | 0.27M | 8.75 |'
  matches-paper-error Our ResNet-20 lands within 0.3 points of the paper's 8.75% error
          reproduced · measured · resnet20.error = 8.75 ± 0.3 (n ≥ 3)
          reads:     resnet20.error: 8.747 [8.597, 8.896] % · mean, t interval, 3 seeds, 95% (rb)
          result:    measured: resnet20 on its own, with its interval, so neither better nor worse
          rule:      reproduced: estimate 8.747 within 8.75 ± 0.3
          timing:    declared before the evidence (by production times)
          units:     3 seeds
          integrity: ok
  runlvsb7g: resnet20.error 8.81 (file) · unit seed=1 · seed=1
  runjifz4w: resnet20.error 8.69 (file) · unit seed=2 · seed=2
  runv2wads: resnet20.error 8.74 (file) · unit seed=3 · seed=3

The same three error rates now read criterion undetermined: not comparable. The difference is an integrity problem until the settings match.

What rb did, step by step

Record the experiment’s purpose

$ rb experiment add "ResNet-20 against the ResNet paper (2016)" --id repro -q q-repro --purpose "Reproduce the ResNet-20 CIFAR-10 test error of the ResNet paper (2016), Table 6" --baseline resnet20
Experiment repro: ResNet-20 against the ResNet paper (2016) · resnet20 (baseline) · under q-repro
  purpose: Reproduce the ResNet-20 CIFAR-10 test error of the ResNet paper (2016), Table 6
  Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis.
  Made 1 variant: resnet20 (baseline); a setting of one is rb spec set repro resnet20.<setting> <value>.

Next: rb spec set repro --from <config.yaml> --keys "<the keys that define it>"  ·  rb criterion add "<what it should show>" -e repro -q q-repro --metric <name> --at-most <x>

Cite each setting, with the value the paper used

--cited keeps the paper’s value beside yours, and rb says it back under the setting. A difference makes every criterion that cites the paper not comparable.

$ rb spec set repro optim.lr --source "configs/train.yaml#optim.lr" --cited 0.1 --cited-source docs/baselines/resnet-paper.md:7
repro/optim.lr = 0.1 · verified in configs/train.yaml#optim.lr
  cited: 0.1 (docs/baselines/resnet-paper.md:7)

$ rb spec set repro optim.momentum --source "configs/train.yaml#optim.momentum" --cited 0.9 --cited-source docs/baselines/resnet-paper.md:5
repro/optim.momentum = 0.9 · verified in configs/train.yaml#optim.momentum
  cited: 0.9 (docs/baselines/resnet-paper.md:5)

$ rb spec set repro optim.weight_decay --source "configs/train.yaml#optim.weight_decay" --cited 0.0001 --cited-source docs/baselines/resnet-paper.md:5
repro/optim.weight_decay = 0.0001 · verified in configs/train.yaml#optim.weight_decay
  cited: 0.0001 (docs/baselines/resnet-paper.md:5)

$ rb spec set repro data.batch_size --source "configs/train.yaml#data.batch_size" --cited 128 --cited-source docs/baselines/resnet-paper.md:6
repro/data.batch_size = 128 · verified in configs/train.yaml#data.batch_size
  cited: 128 (docs/baselines/resnet-paper.md:6)

$ rb spec set repro seed --per-run
repro/seed · per-run: each run gives its own, with --set seed=<value> on rb run add

$ rb show repro/data.batch_size
repro/data.batch_size = 128 · verified
  source: configs/train.yaml#data.batch_size · read 'data.batch_size: 128' @9d87bd1
  cited source used 128 (docs/baselines/resnet-paper.md:6)
  history:
    2026-11-06T20:25:18 spec set: setting data.batch_size: new, 128
    2026-11-06T20:25:18 spec check: verified

Cite the paper’s number as a criterion

rb checks that the quoted row is in the file and states 8.75.

$ rb criterion add "Our ResNet-20 lands within 0.3 points of the paper's 8.75% error" -e repro -q q-repro --metric resnet20.error --equals 8.75 --tolerance 0.3 --source docs/baselines/resnet-paper.md --quote "| ResNet | 20 | 0.27M | 8.75 |" --locator "Table 6" --min-n 3 --words "Count it as reproduced if we land within 0.3 points of the paper's 8.75% error." --id matches-paper-error
matches-paper-error: Our ResNet-20 lands within 0.3 points of the paper's 8.75% error
  resnet20.error = 8.75 ± 0.3 (n ≥ 3) · primary · on repro
  cited: docs/baselines/resnet-paper.md "| ResNet | 20 | 0.27M | 8.75 |" (Table 6) · states 8.75 (read '| ResNet | 20 | 0.27M | 8.75 |')
  declared; no evidence recorded yet

Next: rb run add repro --from <the results file a run wrote> --set seed=<value>

Run and record each seed

$ rb journal run -- python train.py --seed 1
{"metrics": {"resnet20.error": 8.81}}
exit 0. The command, its exit code and what it wrote are in the journal.
It printed its metrics: resnet20.error=8.81. rb reconcile proposes the run of repro that records them.

Since this session last looked:
  - out/seed1/ holds results no recorded experiment reads yet   (rb reconcile)
  - out/seed1/metrics.json: results landed and are not in the record   (rb reconcile)

Next: rb reconcile

$ rb run add repro --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1"
run5s7da5 on repro: resnet20.error=8.81 · seed=1
  Ran with the spec: 5 of its settings found in out/seed1/config.yaml, all matching.
  Produced: 2026-11-06 20:25:19 (file)
  matches-paper-error criterion undetermined · point only: resnet20.error: 8.81 % · mean, 1 seed (rb)
  rb put the code in git at refs/rb/snapshots/e8626933b5fe8d9b4040088d8db7c87428d15d30, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/e8626933b5fe8d9b4040088d8db7c87428d15d30 removes it.

Next: rb show matches-paper-error

$ rb run add repro --from out/seed2/metrics.json --set seed=2 --config out/seed2/config.yaml --command "python train.py --seed 2"
runlycbpg on repro: resnet20.error=8.69 · seed=2
  Ran with the spec: 5 of its settings found in out/seed2/config.yaml, all matching.
  Produced: 2026-11-06 20:25:19 (file)
  matches-paper-error criterion undetermined · point only: resnet20.error: 8.75 % · mean, 2 seeds (rb)

Next: rb show matches-paper-error

$ rb run add repro --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3"
run6aw4sv on repro: resnet20.error=8.74 · seed=3
  Ran with the spec: 5 of its settings found in out/seed3/config.yaml, all matching.
  Produced: 2026-11-06 20:25:20 (file)
  matches-paper-error reproduced · measured: resnet20.error: 8.747 [8.597, 8.896] % · mean, t interval, 3 seeds, 95% (rb)

Next: rb show matches-paper-error

Why it matters

A match to a paper counts only on the paper’s settings. rb checks them before it says reproduced.

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0