Record one table of many runs

This cookbook records a repository whose runs are the rows of one results table in its top folder.

Your training script adds a line to results.tsv after each run, and a RESULTS.md says what the runs showed. You want each row recorded as its own run, without typing a command per row.

Paste this into your agent

Record every row of results.tsv as a run, and check what RESULTS.md says.

My runs are the rows of results.tsv. Record them with Rabbit Brain, and check what RESULTS.md says against them.

- Run `rb context --model <your model id>` first, then `rb reconcile`.
- Record the question and the experiment, with each setting cited to the config that states it.
- Record every row of results.tsv as its own run, one `rb run add ... --every-row` per arm, with each row's seed and finish time.
- Turn the claim in RESULTS.md into the criterion it states, cited to its line.
- Compute nothing yourself: rb computes the findings.

Finish with `rb reconcile` and `rb status` exactly as printed.

The repository trains a ViT-Tiny on CIFAR-100 with no label smoothing and with smoothing 0.1, four seeds each. train.py adds one line to results.tsv per run:

$ head -n 4 results.tsv
smoothing  seed  val_acc  finished_at
0.0  1  71.42  2026-10-27T09:00:00Z
0.1  1  72.06  2026-10-27T10:00:00Z
0.0  2  71.18  2026-10-27T11:00:00Z

$ cat RESULTS.md
# Results

Label smoothing 0.1 raises validation accuracy by about 0.66 points over four seeds.

What you’ll see

rb status in your own terminal, once the agent is done:

$ rb status
Label smoothing on a small ViT · label-smoothing-on-a-small-vit
Kept with rb since 2026-11-06. All 8 runs were recorded afterwards, not as they happened.

Does label smoothing 0.1 raise validation accuracy on ViT-Tiny? (open)
  Label smoothing 0.1 raises validation accuracy by about 0.66 points. change.val_acc: +0.66 [+0.6256, +0.6944] % · paired difference, t interval, 4 seeds, 95% (rb); gain: the interval clear of 0 on the better side (higher is better); reproduced on the estimate: estimate +0.66 within 0.66 ± 0.005; rule and question link declared after the evidence existed: all 4 units were produced before it was declared. units: 4 seeds from 8 runs; paired by seed: each seed's ls01 run against the same seed's none run; unpaired, the same runs read +0.66 [+0.3946, +0.9254] % · unpaired difference, Welch interval, 8 runs, 95% (rb); whether runs that share a seed are linked is not rb's to say. The repeat form is for a counter only (a seed, an epoch, a repeat's number), never a condition: where seed only numbers repeats, rb spec set ls population.repeat seed --no-verify reads each run as its own unit; no run config was compared with the spec (record runs with --config); the interval [+0.6256, +0.6944] crosses 0.655 and 0.665: reproduced on the estimate, not across the whole interval.

Rules: 1 reproduced, on the estimate alone

The design of 1 experiment is not confirmed: rb confirm lists it.

Facts that ask for nothing (1)
  - ls: 1 criterion was declared after its evidence existed

rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-06 20:25 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.

Rules
  ls-gain-claim Label smoothing 0.1 raises validation accuracy by about 0.66 points
          reproduced on the estimate · gain · change.val_acc = 0.66 ± 0.005
          reads:     change.val_acc: +0.66 [+0.6256, +0.6944] % · paired difference, t interval, 4 seeds, 95% (rb)
          result:    gain: the interval clear of 0 on the better side (higher is better)
          rule:      reproduced on the estimate: estimate +0.66 within 0.66 ± 0.005
          timing:    declared after the evidence existed: all 4 units were produced before it was declared
          units:     4 seeds from 8 runs
          · paired by seed: each seed's ls01 run against the same seed's none run; unpaired, the same runs read +0.66 [+0.3946, +0.9254] % · unpaired difference, Welch interval, 8 runs, 95% (rb); whether runs that share a seed are linked is not rb's to say. The repeat form is for a counter only (a seed, an epoch, a repeat's number), never a condition: where seed only numbers repeats, rb spec set ls population.repeat seed --no-verify reads each run as its own unit
          · no run config was compared with the spec (record runs with --config)
          · the interval [+0.6256, +0.6944] crosses 0.655 and 0.665: reproduced on the estimate, not across the whole interval

Experiments
  ls      Label smoothing 0.1 vs none · open · none (baseline) vs ls01 (candidate) · varies smoothing · not frozen · settings 5 verified · 0 provisional · 0 unknown · 1 per-run · runs 8
          to run again: code unknown · environment unknown · files ok

Runs: 8 recorded on 1 experiment · every receipt is what rb wrote

Next: rb confirm

All 8 runs come from rows of results.tsv, and the line under the title says they were recorded afterwards, not as they happened.
ls-gain-claim is the criterion’s id: its name in every command. It is RESULTS.md’s claim, and it reads reproduced on the estimate. The estimate, +0.66, is within 0.005 of 0.66, but the interval is wider than that. The tally says so too: “1 reproduced, on the estimate alone”.
The arms were recorded apart, a run per row, so the reading says what it pairs them by: “paired by seed”, each seed’s ls01 run against the same seed’s none run. The unpaired reading of the same runs is beside it. Whether runs that share a seed are linked is not rb’s to say. Where a seed only numbers repeats, rb gives the command that reads each run as its own unit, rb spec set ls population.repeat seed --no-verify, for a counter only.
Its timing reads declared after the evidence existed: every row was written before the criterion.
“no run config was compared with the spec”: the runs were recorded from rows, with no --config.

What rb did, step by step

Read what the repository already says

rb reconcile names each results file in the top folder. results.tsv is a table: rb gives its rows, the columns that hold numbers, and the rb run add ... --where line that reads a row. RESULTS.md is a report in prose: rb reads no run from it, and a number it states is checked by a cited rule.

$ rb reconcile
The repository at d17af26: 2 results files in the top folder.

Results files in no results folder (2):
  RESULTS.md: prose: rb reads no run from it; a number there is a cited rule
    a number it states is checked by a cited rule: rb criterion add "<what it claims>" -e <experiment> --metric <metric> --equals <the number> --tolerance <half its last digit> --source RESULTS.md --quote "<its words>"
  results.tsv: a TSV table of 8 rows, numbers in smoothing, seed, val_acc
    a row is a run: rb run add <experiment> --from results.tsv --where <column>=<value>,...

Record the question, the experiment and its settings

$ rb question add "Does label smoothing 0.1 raise validation accuracy on ViT-Tiny?" --id q-ls
q-ls: Does label smoothing 0.1 raise validation accuracy on ViT-Tiny?

Next: rb experiment add "<what would answer it>" -q q-ls --hypothesis "<what you expect>"

$ rb experiment add "Label smoothing 0.1 vs none" --id ls -q q-ls --baseline none --candidate ls01 --varies smoothing
Experiment ls: Label smoothing 0.1 vs none · none (baseline) vs ls01 (candidate) · varies smoothing · under q-ls
  Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis.
  Made 2 variants: none (baseline), ls01 (candidate); a setting of one is rb spec set ls none.<setting> <value>.
  ls declares it varies smoothing, but no variant has a value for it yet; give each variant its own: rb spec set ls <variant>.smoothing <value> --source <file#key>

Next: rb spec set ls --from <config.yaml> --keys "<the keys that define it>"  ·  rb criterion add "<what it should show>" -e ls -q q-ls --metric <name> --at-most <x>

$ rb spec set ls --from configs/train.yaml --keys "optim.*,data.*"
ls: 3 settings from configs/train.yaml @d17af26 · 3 verified

$ rb spec set ls none.smoothing 0.0 --source "configs/none.yaml#smoothing"
ls/none.smoothing = 0.0 · verified in configs/none.yaml#smoothing

$ rb spec set ls ls01.smoothing 0.1 --source "configs/ls01.yaml#smoothing"
ls/ls01.smoothing = 0.1 · verified in configs/ls01.yaml#smoothing

$ rb spec set ls seed --per-run
ls/seed · per-run: each run gives its own, with --set seed=<value> on rb run add

$ rb metric add val_acc --unit % --maximize
Metric val_acc (%): higher is better

Record every row, one command per arm

--where smoothing=0.0 picks the four rows of one arm, and --every-row records each as its own run. --set seed=@seed and --produced-at @finished_at read each run’s seed and time from its own row. Every row is read before any is written, and a row already recorded is passed.

$ rb run add ls --from results.tsv --where smoothing=0.0 --variant none --every-row --set seed=@seed --produced-at @finished_at
runhbpnqq on ls: none.val_acc=71.42 · seed=1
runga43yw on ls: none.val_acc=71.18 · seed=2
runm5jqh4 on ls: none.val_acc=71.55 · seed=3
run2fovlj on ls: none.val_acc=71.3 · seed=4
4 of the 4 rows --where picks in results.tsv recorded as runs.
  rb put the code in git at refs/rb/snapshots/2f2ced2447df1446e5704536fc328b9bc50e2bb1, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/2f2ced2447df1446e5704536fc328b9bc50e2bb1 removes it.

$ rb run add ls --from results.tsv --where smoothing=0.1 --variant ls01 --every-row --set seed=@seed --produced-at @finished_at
runpv56tf on ls: ls01.val_acc=72.06 · seed=1
runggqhb6 on ls: ls01.val_acc=71.87 · seed=2
runo774w2 on ls: ls01.val_acc=72.21 · seed=3
run5hr7aj on ls: ls01.val_acc=71.95 · seed=4
4 of the 4 rows --where picks in results.tsv recorded as runs.

A .jsonl or .ndjson file is read the same way, one JSON object a row. Without --every-row, a --where that picks several rows is refused, and offers it.

Since rb 0.8.0, --every-row records a SQLite table’s rows too: rb run add <exp> --from results.db --table evals --where precision=int8 --set seed=@seed --every-row.

Check the report’s claim

$ rb compare ls
ls  Label smoothing 0.1 vs none · varies smoothing · 8 runs under the current spec · means over seeds
  variant  role       smoothing  val_acc (%)
  none     baseline   0.0        71.36 (n=4)
  ls01     candidate  0.1        72.02 +0.66 better (n=4)

$ rb criterion add "Label smoothing 0.1 raises validation accuracy by about 0.66 points" -e ls -q q-ls --metric change.val_acc --equals 0.66 --tolerance 0.005 --source RESULTS.md --quote "Label smoothing 0.1 raises validation accuracy by about 0.66 points over four seeds." --id ls-gain-claim
ls-gain-claim: Label smoothing 0.1 raises validation accuracy by about 0.66 points
  change.val_acc = 0.66 ± 0.005 · primary · on ls
  cited: RESULTS.md "Label smoothing 0.1 raises validation accuracy by about 0..." · states 0.66 (read 'Label smoothing 0.1 raises validation accuracy by about 0.66 points over four seeds.')
  declared after the evidence existed: all 4 units were produced before it was declared
  it already reads change.val_acc: +0.66 [+0.6256, +0.6944] % · paired difference, t interval, 4 seeds, 95% (rb): reproduced on the estimate: estimate +0.66 within 0.66 ± 0.005

Next: rb run add ls --from <the results file a run wrote> --set seed=<value>

$ rb show ls-gain-claim
Criterion ls-gain-claim on ls (primary, cited): Label smoothing 0.1 raises validation accuracy by about 0.66 points
  cited: RESULTS.md "Label smoothing 0.1 raises validation accuracy by about 0..." · states 0.66, read 'Label smoothing 0.1 raises validation accuracy by about 0.66 points over four seeds.'
  ls-gain-claim Label smoothing 0.1 raises validation accuracy by about 0.66 points
          reproduced on the estimate · gain · change.val_acc = 0.66 ± 0.005
          reads:     change.val_acc: +0.66 [+0.6256, +0.6944] % · paired difference, t interval, 4 seeds, 95% (rb)
          result:    gain: the interval clear of 0 on the better side (higher is better)
          rule:      reproduced on the estimate: estimate +0.66 within 0.66 ± 0.005
          timing:    declared after the evidence existed: all 4 units were produced before it was declared
          units:     4 seeds from 8 runs
          integrity: no run config was compared, as said below
          · paired by seed: each seed's ls01 run against the same seed's none run; unpaired, the same runs read +0.66 [+0.3946, +0.9254] % · unpaired difference, Welch interval, 8 runs, 95% (rb); whether runs that share a seed are linked is not rb's to say. The repeat form is for a counter only (a seed, an epoch, a repeat's number), never a condition: where seed only numbers repeats, rb spec set ls population.repeat seed --no-verify reads each run as its own unit
          · no run config was compared with the spec (record runs with --config)
          · the interval [+0.6256, +0.6944] crosses 0.655 and 0.665: reproduced on the estimate, not across the whole interval
  runhbpnqq: none.val_acc 71.42 (row) · unit seed=1 · produced before the declaration · seed=1
  runga43yw: none.val_acc 71.18 (row) · unit seed=2 · produced before the declaration · seed=2
  runm5jqh4: none.val_acc 71.55 (row) · unit seed=3 · produced before the declaration · seed=3
  run2fovlj: none.val_acc 71.3 (row) · unit seed=4 · produced before the declaration · seed=4
  runpv56tf: ls01.val_acc 72.06 (row) · unit seed=1 · produced before the declaration · seed=1
  runggqhb6: ls01.val_acc 71.87 (row) · unit seed=2 · produced before the declaration · seed=2
  runo774w2: ls01.val_acc 72.21 (row) · unit seed=3 · produced before the declaration · seed=3
  run5hr7aj: ls01.val_acc 71.95 (row) · unit seed=4 · produced before the declaration · seed=4

Each run’s number reads “(row)”: it came from a row of the table.

Read the repository again

Once the table’s rows are recorded, rb reconcile says how many:

$ rb reconcile
The repository at d17af26: 2 results files in the top folder.

Results files in no results folder (2):
  RESULTS.md: prose: rb reads no run from it; a number there is a cited rule
    a number it states is checked by a cited rule: rb criterion add "<what it claims>" -e <experiment> --metric <metric> --equals <the number> --tolerance <half its last digit> --source RESULTS.md --quote "<its words>"
  results.tsv: a TSV table of 8 rows, numbers in smoothing, seed, val_acc; 8 of 8 rows recorded as runs
    a row is a run: rb run add <experiment> --from results.tsv --where <column>=<value>,...

Why it matters

A table of runs becomes a run per row, each with its own seed and time, and a claim in prose is checked against those runs, not copied from the report.

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0