Think step by step: a small model on grade-school math

rb looks in out/, outputs/, results/, eval-out/ and metrics/, and at the files in the project's top folder. It has not looked yet. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.

Recent changes

Latest changes: 2026-10-23, 23:07 to 23:11 UTC · 4 changes · no calls made

Recent calls and changes

2026-10-23, 23:07 to 23:11 UTC · 4 changes · no calls made

23:11
Added a note on the project
23:11
Retracted a note on e1 · why: c-ref now reads the published 49.6 against the plain prompt
23:11
Retracted a note on e1 · why: r010 (worked examples) and r012 (plain) both ran on seed 0 at 250 problems under last_number, so a pair of prompts has been run alike; only step by step has no 250-problem run under last_number
23:07
Added rule c-ref to e1

Every change, newest first →

Notes on the project (2)

note on the project (2026-10-23): Next stated in notes.md · line 9: a write-up of the prompt comparison for the reading group. No such write-up is in the repository yet. · not confirmed
note on the project (2026-10-23): Left out of the record for now: the smoke run r001 (10 problems) and the 50-problem pilot r002 and r003, all graded by answer_tag at commit 929a9c4.

Reproduce this project

Everything on this page was computed from the record this bundle carries. Rerun that computation, and the runs behind it, on your machine:

Rerun this research

Research

Each question has its own answer, evidence and open questions.

Open question

Does telling a small AI model to think step by step make it solve more grade-school math problems?

No answer yet: the plain baseline doesn't match its cited number, and 1 of 4 rules is not met.

Rules: 2 reproduced · 1 not met · 1 not reproduced

Open question

Do worked examples help more than "think step by step"?

Evidence includes point estimates only

Rules: 1 not reproduced

Open question

How many problems do we need before the difference is real?

No rule tests this question yet

Research›Does telling a small AI model to think… q1

Does telling a small AI model to think step by step make it solve more grade-school math problems?

Open question

No answer yet: the plain baseline doesn't match its cited number, and 1 of 4 rules is not met.

Rules: 2 reproduced · 1 not met · 1 not reproduced

Its rules (4)

All 4 rules were written after their results came in

No note on these rules yet.

Step by step scores at least 5 points above the plain prompt rule not met
step by step is 3.60 points higher than the plain baseline (+0.62 to +6.58), 3 seeds. Open calls c1
details of c1
step by step is 3.60 points higher than the plain baseline (+0.62 to +6.58) · gain
3 seeds from 6 runs · all 6 of its runs recorded afterwards · estimate +3.6 < 5
rule
c1 · primary · on e1
as declared
rule not met: estimate +3.6 < 5accuracy: step_by_step vs plain [where checker=answer_tag, problems=250] ≥ 5rule ≥ 5 points · margin -1.40 points
reading
accuracy: step_by_step vs plain [where checker=answer_tag, problems=250]: +3.6 [+0.619, +6.581] percent · paired difference, t interval, 3 seeds, 95% (rb)
units
3 seeds from 6 runs · declared: unit = seed
timing
declared after the evidence existed: all 3 units were produced before it was declared
question link
q1 · 2026-10-23 22:58 UTC
review
Open callsYour call
The plain prompt reaches the published 49.6 on GSM8K not reproduced
plain baseline is 13.07% (11.00% to 15.14%), 3 seeds; the cited number is 49.6%. Open calls c-ref
details of c-ref
plain baseline is 13.07% (11.00% to 15.14%); the cited number is 49.6% · measured
3 runs · all 3 of its runs recorded afterwards · estimate 13.07 outside 49.6 ± 0.05
rule
c-ref · secondary · on e1 · cited
as declared
not reproduced: estimate 13.07 outside 49.6 ± 0.05accuracy: plain [where checker=answer_tag, problems=250] = 49.6 ± 0.05rule = 49.6% ± 0.05 points · margin -36.48 points
reading
accuracy: plain [where checker=answer_tag, problems=250]: 13.07 [11, 15.14] percent · mean, t interval, 3 seeds, 95% (rb)
units
3 seeds · declared: unit = seed
timing
declared after the evidence existed: all 3 units were produced before it was declared
question link
q1 · 2026-10-23 23:07 UTC
review
Open callsYour call
The claim that step by step is 3.6 points above plain on the seed averages is checked here reproduced
step by step is 3.60 points higher than the plain baseline (+0.62 to +6.58), 3 seeds. c2
details of c2
step by step is 3.60 points higher than the plain baseline (+0.62 to +6.58) · gain
3 seeds from 6 runs · all 6 of its runs recorded afterwards
rule
c2 · secondary · on e1 · cited
as declared
reproduced: estimate +3.6 within 3.6 ± 0.05accuracy: step_by_step vs plain [where checker=answer_tag, problems=250] = 3.6 ± 0.05rule = 3.6 ± 0.05 points · margin 0.05 points
reading
accuracy: step_by_step vs plain [where checker=answer_tag, problems=250]: +3.6 [+0.619, +6.581] percent · paired difference, t interval, 3 seeds, 95% (rb)
units
3 seeds from 6 runs · declared: unit = seed
timing
declared after the evidence existed: all 3 units were produced before it was declared
question link
q1 · 2026-10-23 22:58 UTC
The claim that the step-by-step gain over plain holds on every seed is checked here reproduced
step by step is 3.60 points higher than the plain baseline (+0.62 to +6.58), 3 seeds. c3
details of c3
step by step is 3.60 points higher than the plain baseline (+0.62 to +6.58) · gain
3 seeds from 6 runs · all 6 of its runs recorded afterwards
rule
c3 · secondary · on e1 · cited
as declared
reproduced: all 3 units ≥ 0each unit's accuracy: step_by_step vs plain [where checker=answer_tag, problems=250] ≥ 0rule each unit ≥ 0 points
reading
accuracy: step_by_step vs plain [where checker=answer_tag, problems=250]: +3.6 [+0.619, +6.581] percent · paired difference, t interval, 3 seeds, 95% (rb)
units
3 seeds from 6 runs · declared: unit = seed
timing
declared after the evidence existed: all 3 units were produced before it was declared
question link
q1 · 2026-10-23 22:58 UTC

Open callsYour call (2)

Rule not met · estimate +3.6 < 5 · on the estimate, not across the whole intervalStep by step scores at least 5 points above the plain prompt c1 Not reproduced · estimate 13.07 outside 49.6 ± 0.05The plain prompt reaches the published 49.6 on GSM8K c-ref

Calls are made with rb in a terminal, never on this page.

Your calls are made with rb in a terminal; this page gives you the command.

Calls are made with rb in a terminal, not on this page; it shows where they stand.

Experiments behind this question

Prompt style on 250 GSM8K test problems: plain, step by step, worked examples

Experiment · open · 9 runs

Rerun recipe: 1 of 5 parts ready

Sub-question

Open question

Do worked examples help more than "think step by step"?

Evidence includes point estimates only

Rules: 1 not reproduced

Open question

How many problems do we need before the difference is real?

No rule tests this question yet

Which comparisons carry these qualifications?

rule written after the results

the rule or its question link was declared after the evidence existed or after the data was seen: never read as a test planned in advance.

c-ref The plain prompt reaches the published 49.6 on GSM8K
c1 Step by step scores at least 5 points above the plain prompt
c2 The claim that step by step is 3.6 points above plain on the seed averages is checked here
c3 The claim that the step-by-step gain over plain holds on every seed is checked here

By reading

not reproduced · 1

c-ref The plain prompt reaches the published 49.6 on GSM8K

rule not met · 1

c1 Step by step scores at least 5 points above the plain prompt

reproduced · 2

c2 The claim that step by step is 3.6 points above plain on the seed averages is checked here
c3 The claim that the step-by-step gain over plain holds on every seed is checked here

Research›Does telling a small AI model to think… q1›Do worked examples help more than… q2

Do worked examples help more than "think step by step"?

Open question

Evidence includes point estimates only

Rules: 1 not reproduced

Its rules (1)

The one rule was written after its results came in

No note on these rules yet.

The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from not reproduced
worked examples is 20.80 points higher than step by step, 1 seed. Open calls c4
details of c4
worked examples is 20.80 points higher than step by step · no interval
1 seed from 2 runs · 2 runs not counted · both its runs recorded afterwards · estimate +20.8 outside 22.9 ± 0.05
rule
c4 · secondary · on e1 · cited
as declared
not reproduced: estimate +20.8 outside 22.9 ± 0.05accuracy: worked_examples vs step_by_step [subject where checker=last_number; reference where checker=answer_tag, problems=250] = 22.9 ± 0.05rule = 22.9 ± 0.05 points · margin -2.05 points
reading
accuracy: worked_examples vs step_by_step [subject where checker=last_number; reference where checker=answer_tag, problems=250]: +20.8 percent · paired difference, 1 seed (rb)
units
1 seed from 2 runs · declared: unit = seed
timing
declared after the evidence existed: the only unit was produced before it was declared, and so were 2 unpaired runs (run step_by_step · seed 1 · checker answer_tag · problems 250, run step_by_step · seed 2 · checker answer_tag · problems 250)
question link
q2 · 2026-10-23 22:58 UTC
review
Open callsYour call

Open callsYour call (1)

Too little evidence · 1 seed · not reproducedThe claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from c4

Calls are made with rb in a terminal, never on this page.

Your calls are made with rb in a terminal; this page gives you the command.

Calls are made with rb in a terminal, not on this page; it shows where they stand.

Experiments behind this question

Prompt style on 250 GSM8K test problems: plain, step by step, worked examples

Experiment · open · 9 runs

Rerun recipe: 1 of 5 parts ready

Which comparisons carry these qualifications?

no interval

one estimate with no interval, so how much it could vary is unknown. rb gives one from 3 units, and none across fixed design cells.

c4 The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from

rule written after the results

the rule or its question link was declared after the evidence existed or after the data was seen: never read as a test planned in advance.

c4 The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from

By reading

not reproduced · 1

c4 The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from

Research›Does telling a small AI model to think… q1›How many problems do we need before the… q3

How many problems do we need before the difference is real?

Open question

No rule tests this question yet.

Experiments behind this question

No experiment is filed under this question.

Research›Rules

Rules (5)

Rules: 2 reproduced · 1 not met · 2 not reproduced

All 5 rules were written after their results came in

Prompt style on 250 GSM8K test problems: plain, step by step, worked examples

e15 rules

Step by step scores at least 5 points above the plain promptrule not metstep by step is 3.60 points higher than the plain baseline (+0.62 to +6.58), 3 seeds. c1

step by step is 3.60 points higher than the plain baseline (+0.62 to +6.58) · gain

rule: c1 · primary · on e1
as declared: rule not met: estimate +3.6 < 5accuracy: step_by_step vs plain [where checker=answer_tag, problems=250] ≥ 5rule ≥ 5 points · margin -1.40 points
reading: accuracy: step_by_step vs plain [where checker=answer_tag, problems=250]: +3.6 [+0.619, +6.581] percent · paired difference, t interval, 3 seeds, 95% (rb)
units: 3 seeds from 6 runs · declared: unit = seed
timing: declared after the evidence existed: all 3 units were produced before it was declared
review: Open callsYour call

3 seeds from 6 runs · all 6 of its runs recorded afterwards · estimate +3.6 < 5 Open calls

The claim that step by step is 3.6 points above plain on the seed averages is checked herereproducedstep by step is 3.60 points higher than the plain baseline (+0.62 to +6.58), 3 seeds. c2

step by step is 3.60 points higher than the plain baseline (+0.62 to +6.58) · gain

rule: c2 · secondary · on e1 · cited
as declared: reproduced: estimate +3.6 within 3.6 ± 0.05accuracy: step_by_step vs plain [where checker=answer_tag, problems=250] = 3.6 ± 0.05rule = 3.6 ± 0.05 points · margin 0.05 points
reading: accuracy: step_by_step vs plain [where checker=answer_tag, problems=250]: +3.6 [+0.619, +6.581] percent · paired difference, t interval, 3 seeds, 95% (rb)
units: 3 seeds from 6 runs · declared: unit = seed
timing: declared after the evidence existed: all 3 units were produced before it was declared

3 seeds from 6 runs · all 6 of its runs recorded afterwards

The claim that the step-by-step gain over plain holds on every seed is checked herereproducedstep by step is 3.60 points higher than the plain baseline (+0.62 to +6.58), 3 seeds. c3

step by step is 3.60 points higher than the plain baseline (+0.62 to +6.58) · gain

rule: c3 · secondary · on e1 · cited
as declared: reproduced: all 3 units ≥ 0each unit's accuracy: step_by_step vs plain [where checker=answer_tag, problems=250] ≥ 0rule each unit ≥ 0 points
reading: accuracy: step_by_step vs plain [where checker=answer_tag, problems=250]: +3.6 [+0.619, +6.581] percent · paired difference, t interval, 3 seeds, 95% (rb)
units: 3 seeds from 6 runs · declared: unit = seed
timing: declared after the evidence existed: all 3 units were produced before it was declared

3 seeds from 6 runs · all 6 of its runs recorded afterwards

The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed fromnot reproducedworked examples is 20.80 points higher than step by step, 1 seed. c4

worked examples is 20.80 points higher than step by step · no interval

rule: c4 · secondary · on e1 · cited
as declared: not reproduced: estimate +20.8 outside 22.9 ± 0.05accuracy: worked_examples vs step_by_step [subject where checker=last_number; reference where checker=answer_tag, problems=250] = 22.9 ± 0.05rule = 22.9 ± 0.05 points · margin -2.05 points
reading: accuracy: worked_examples vs step_by_step [subject where checker=last_number; reference where checker=answer_tag, problems=250]: +20.8 percent · paired difference, 1 seed (rb)
units: 1 seed from 2 runs · declared: unit = seed
timing: declared after the evidence existed: the only unit was produced before it was declared, and so were 2 unpaired runs (run step_by_step · seed 1 · checker answer_tag · problems 250, run step_by_step · seed 2 · checker answer_tag · problems 250)
review: Open callsYour call

1 seed from 2 runs · 2 runs not counted · both its runs recorded afterwards · estimate +20.8 outside 22.9 ± 0.05 Open calls

The plain prompt reaches the published 49.6 on GSM8Knot reproducedplain baseline is 13.07% (11.00% to 15.14%), 3 seeds; the cited number is 49.6%. c-ref

plain baseline is 13.07% (11.00% to 15.14%); the cited number is 49.6% · measured

rule: c-ref · secondary · on e1 · cited
as declared: not reproduced: estimate 13.07 outside 49.6 ± 0.05accuracy: plain [where checker=answer_tag, problems=250] = 49.6 ± 0.05rule = 49.6% ± 0.05 points · margin -36.48 points
reading: accuracy: plain [where checker=answer_tag, problems=250]: 13.07 [11, 15.14] percent · mean, t interval, 3 seeds, 95% (rb)
units: 3 seeds · declared: unit = seed
timing: declared after the evidence existed: all 3 units were produced before it was declared
review: Open callsYour call

3 runs · all 3 of its runs recorded afterwards · estimate 13.07 outside 49.6 ± 0.05 Open calls

Experiments

Open an experiment for its comparison, settings and record.

Experiments›Prompt style on 250 GSM8K test… e1

Prompt style on 250 GSM8K test problems: plain, step by step, worked examples

Experiment · open · 9 runs

step by step and worked examples against the plain baseline · varies prompt

design not confirmed · list it to confirm: rb confirm e1

The design is not confirmed yet.

hypothesis: Telling Qwen2.5-0.5B-Instruct to think step by step makes it solve more grade-school math problems than the plain prompt; worked examples are compared against step by step
note: README.md · line 3 and README.md · line 7 · not confirmed

Comparison

Accuracy (%)

	Answer tag · 250 problems	Last number · 250 problems	Last number · 500 problems
Plain baseline	13.07 · 3 seeds	43.20 · 1 run	Not run
Step by step	16.67 · 3 seeds3.60 points higher than plain (+0.62 to +6.58)	Not run	42.20 · 1 run
Worked examples	Not run	39.60 · 1 run3.60 points lower than plain · 1 seed, so no interval	Not run

Answer tag · 250 problems

Last number · 250 problems

Last number · 500 problems

Plain baseline

13.07 · 3 seeds

43.20 · 1 run

Not run

Step by step

16.67 · 3 seeds3.60 points higher than plain (+0.62 to +6.58)

Not run

42.20 · 1 run

Worked examples

Not run

39.60 · 1 run3.60 points lower than plain · 1 seed, so no interval

Not run

Each column is measured under its own settings. Compare within a column, not across.

From 9 runs.

Runs (9)

Recorded 2026-10-23

Seed	Checker	Problems	Accuracy (%)			Produced	run
Seed	Checker	Problems	Plain	Step by step	Worked examples	Produced	run
0	answer_tag	250	14.00			2026-10-06	recorded afterwards
1	answer_tag	250	12.80			2026-10-06	recorded afterwards
2	answer_tag	250	12.40			2026-10-06	recorded afterwards
0	answer_tag	250		18.80		2026-10-06	recorded afterwards
1	answer_tag	250		15.20		2026-10-06	recorded afterwards
2	answer_tag	250		16.00		2026-10-06	recorded afterwards
0	last_number	250			39.60	2026-10-09	recorded afterwards
0	last_number	250	43.20			2026-10-14	recorded afterwards
0	last_number	500		42.20		2026-10-12	recorded afterwards

Its rules (5)

All 5 rules were written after their results came in

Step by step scores at least 5 points above the plain promptrule not met

The claim that step by step is 3.6 points above plain on the seed averages is checked herereproduced

The claim that the step-by-step gain over plain holds on every seed is checked herereproduced

The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed fromnot reproduced

The plain prompt reaches the published 49.6 on GSM8Knot reproduced

c-ref

Reproduce

Rerun recipe: 1 of 5 parts ready

to run again: ✓ 1 ok · ◐ 1 partial · ✗ 2 missing · ○ 1 unknown

Rerun this research with Rabbit Brain (rb). Its record: https://rabbitbrain.ai/u/sample/think-step-by-step-a-small-model-on-grade-school-math/

1. Install rb: pip install rabbit-brain
2. Clone the record into a new folder: rb clone https://rabbitbrain.ai/u/sample/think-step-by-step-a-small-model-on-grade-school-math/bundle.zip think-step-by-step-a-small-model-on-grade-school-math. This bundle names the code: rb clone gets it too, at the commit the record names, and puts the record in it. If it says it could not get the code, tell me what it says and stop: once I give you an address of the code, rb clone https://rabbitbrain.ai/u/sample/think-step-by-step-a-small-model-on-grade-school-math/bundle.zip <another new folder> --code <that address> gets it.
3. In that folder, run rb context and read it before anything else. Its "Rerun the parent" part names the code and whether it is here, each resource, such as a model or a dataset, with how to get it, and the first command for each experiment.
4. Get those resources. Run experiment e1, “Prompt style on 250 GSM8K test problems: plain, step by step, worked examples”: its commands in the order rb context names them, each as rb journal run -- <command>; where rb context names a command that puts a run's code here, run that one first. After each, run rb reconcile and record the run it proposes with the rb run add line it prints.
5. Then run rb status, and tell me which of its rules (c-ref, c1, c2, c3, c4) reproduced and which did not, with their numbers. Do not change a rule. If something cannot run here, say what and why.

The whole recipe as one script

# Reproduce e1: Prompt style on 250 GSM8K test problems: plain, step by step, worked examples
# Recipe, from what the record holds. rb did not run these commands.
# code: missing · commit 32c91d6, 7265bd1, 738002e and 1 more · not in this repository: 32c91d6, 7265bd1, 738002e and 1 more (get the code repository to rerun)
# environment: unknown · the runs were made before they were recorded, in environments rb did not see
# checkpoint: ok · qwen25-05b-instruct: present
# dataset: partial · gsm8k-test: reconstructible
# files: missing · none of the 18 files the runs named is here unchanged; not here: results/r004_plain_s0/replies.jsonl, results/r004_plain_s0/metrics.json, results/r005_plain_s1/replies.jsonl and 15 more

# checkpoint · qwen25-05b-instruct: no recipe recorded · configs/model.yaml; Apache 2.0

# dataset · gsm8k-test: no recipe recorded · Cobbe et al. 2021, MIT license

# per run (9): seed · checker · problems
#   plain: 0 answer_tag 250 · 1 answer_tag 250 · 2 answer_tag 250 · 0 last_number 250
#   step_by_step: 0 answer_tag 250 · 1 answer_tag 250 · 2 answer_tag 250 · 0 last_number 500
#   worked_examples: 0 last_number 250
# settings: name=Qwen/Qwen2.5-0.5B-Instruct revision=7ae557604adf67be50417f59c2c2f167def9a775 dtype=bfloat16 temperature=0.7 top_p=0.8 top_k=20 repetition_penalty=1.1 max_new_tokens=512 split=test subset_seed=2026
#   plain: prompt=plain
#   step_by_step: prompt=step_by_step
#   worked_examples: prompt=worked_examples
# command (as given) · recorded for all 9 runs · 1 given later, not observed
#   plain · seed 0 · checker answer_tag · problems 250: command (as given): python solve.py --prompt plain --seed 0 --out results/r004_plain_s0 (results/r004_plain_s0/config.yaml:1 records this launch)
#   plain · seed 1 · checker answer_tag · problems 250: command (as given): python solve.py --prompt plain --seed 1 --out results/r005_plain_s1 (results/r005_plain_s1/config.yaml:1 records this launch)
#   plain · seed 2 · checker answer_tag · problems 250: command (as given): python solve.py --prompt plain --seed 2 --out results/r006_plain_s2 (results/r006_plain_s2/config.yaml:1 records this launch)
#   step_by_step · seed 0 · checker answer_tag · problems 250: command (as given): python solve.py --prompt step_by_step --seed 0 --out results/r007_step_s0 (results/r007_step_s0/config.yaml:1 records this launch)
#   step_by_step · seed 1 · checker answer_tag · problems 250: command (as given): python solve.py --prompt step_by_step --seed 1 --out results/r008_step_s1 (results/r008_step_s1/config.yaml:1 records this launch)
#   step_by_step · seed 2 · checker answer_tag · problems 250: command (as given): python solve.py --prompt step_by_step --seed 2 --out results/r009_step_s2 (results/r009_step_s2/config.yaml:1 records this launch)
#   worked_examples · seed 0 · checker last_number · problems 250: command (as given): python solve.py --prompt worked_examples --seed 0 --out results/r010_worked_s0 (results/r010_worked_s0/config.yaml:1 records this launch)
#   plain · seed 0 · checker last_number · problems 250: command (as given): python solve.py --prompt plain --seed 0 --out results/r012_plain_s0 (results/r012_plain_s0/config.yaml:1 records this launch)
#   step_by_step · seed 0 · checker last_number · problems 500: command given later, not observed: python solve.py --prompt step_by_step --seed 0 --out results/r011_step_n500_s0 --problems 500

What it takes to run it again

codemissing · commit 32c91d6, 7265bd1, 738002e and 1 more · not in this repository: 32c91d6, 7265bd1, 738002e and 1 more (get the code repository to rerun)
environmentunknown · the runs were made before they were recorded, in environments rb did not see
checkpointok · qwen25-05b-instruct: present
datasetpartial · gsm8k-test: reconstructible
filesmissing · none of the 18 files the runs named is here unchanged; not here: results/r004_plain_s0/replies.jsonl, results/r004_plain_s0/metrics.json, results/r005_plain_s1/replies.jsonl and 15 more

Recipe, from what the record holds

code · missing: commit 32c91d6, 7265bd1, 738002e and 1 more · not in this repository: 32c91d6, 7265bd1, 738002e and 1 more (get the code repository to rerun)
checkpoint · qwen25-05b-instruct: no recipe recorded · configs/model.yaml; Apache 2.0
dataset · gsm8k-test: no recipe recorded · Cobbe et al. 2021, MIT license
command (as given) · recorded for all 9 runs · 1 given later, not observed
per run · seed · checker · problems
plain0 answer_tag 250 · 1 answer_tag 250 · 2 answer_tag 250 · 0 last_number 250step_by_step0 answer_tag 250 · 1 answer_tag 250 · 2 answer_tag 250 · 0 last_number 500worked_examples0 last_number 250
settings
sharedname=Qwen/Qwen2.5-0.5B-Instruct revision=7ae557604adf67be50417f59c2c2f167def9a775 dtype=bfloat16 temperature=0.7 top_p=0.8 top_k=20 repetition_penalty=1.1 max_new_tokens=512 split=test subset_seed=2026plainprompt=plainstep_by_stepprompt=step_by_stepworked_examplesprompt=worked_examples

Commands as recorded (history, not the recipe)

plain · seed 0 · checker answer_tag · problems 250 logged

python solve.py --prompt plain --seed 0 --out results/r004_plain_s0

plain · seed 1 · checker answer_tag · problems 250 logged

python solve.py --prompt plain --seed 1 --out results/r005_plain_s1

plain · seed 2 · checker answer_tag · problems 250 logged

python solve.py --prompt plain --seed 2 --out results/r006_plain_s2

step_by_step · seed 0 · checker answer_tag · problems 250 logged

python solve.py --prompt step_by_step --seed 0 --out results/r007_step_s0

step_by_step · seed 1 · checker answer_tag · problems 250 logged

python solve.py --prompt step_by_step --seed 1 --out results/r008_step_s1

step_by_step · seed 2 · checker answer_tag · problems 250 logged

python solve.py --prompt step_by_step --seed 2 --out results/r009_step_s2

worked_examples · seed 0 · checker last_number · problems 250 logged

python solve.py --prompt worked_examples --seed 0 --out results/r010_worked_s0

plain · seed 0 · checker last_number · problems 250 logged

python solve.py --prompt plain --seed 0 --out results/r012_plain_s0

step_by_step · seed 0 · checker last_number · problems 500 given later

python solve.py --prompt step_by_step --seed 0 --out results/r011_step_n500_s0 --problems 500

Next step

"run one more accuracy measurement on Prompt style on 250 GSM8K test problems, step_by_step vs plain, at checker=answer_tag, problems=250, and record it in rb"

"run one more accuracy measurement on Prompt style on 250 GSM8K test problems, step_by_step vs plain, at checker=answer_tag, problems=250, and record it in rb"

Source details · settings: 3 per-run · 4 provisional · 10 verified

shared	name	Qwen/Qwen2.5-0.5B-Instruct	verified	configs/model.yaml#name
shared	revision	7ae557604adf67be50417f59c2c2f167def9a775	verified	configs/model.yaml#revision
shared	dtype	bfloat16	verified	configs/model.yaml#dtype
shared	temperature	0.7	verified	configs/sampling.yaml#temperature
shared	top_p	0.8	verified	configs/sampling.yaml#top_p
shared	top_k	20	verified	configs/sampling.yaml#top_k
shared	repetition_penalty	1.1	verified	configs/sampling.yaml#repetition_penalty
shared	max_new_tokens	512	verified	configs/sampling.yaml#max_new_tokens
shared	split	test	verified	configs/eval.yaml#split
shared	subset_seed	2026	verified	configs/eval.yaml#subset_seed
plain	prompt	plain	provisional	results/r004_plain_s0/config.yaml#prompt
step_by_step	prompt	step_by_step	provisional	results/r007_step_s0/config.yaml#prompt
worked_examples	prompt	worked_examples	provisional	results/r010_worked_s0/config.yaml#prompt

notes (5)

note on e1: Same prompt and seed, different grader: plain seed 0 gives 14.0 under answer_tag (results/r004_plain_s0) and 43.2 under last_number (results/r012_plain_s0); step by step seed 0 gives 18.8 at 250 problems under answer_tag (results/r007_step_s0) and 42.2 at 500 problems under last_number (results/r011_step_n500_s0). r012 was launched as a stability rerun (logs/agent.log, 2026-10-14) but differs from r004 in the grader, so it does not test reseeding stability. r012's config.yaml records dirty: true at commit 9ba0f86. · not confirmed
note on e1: Planned, not run: configs/sweep.yaml (uncommitted) holds worked_examples on seeds 1 and 2, 250 problems; at the current configs/eval.yaml these would be graded with last_number, against step-by-step and plain seeds graded with answer_tag. · not confirmed
note on e1: plain.prompt, step_by_step.prompt and worked_examples.prompt are sourced from each run's own config.yaml, written by the run; the sweep config before the runs lists the prompts only as a grid (configs/sweep.yaml at 32c91d6), which rb cannot read as one variant's value. · not confirmed
note on e1: The grader changed between runs: r004–r009 were graded with checker answer_tag, r010–r012 with checker last_number (configs/eval.yaml, commit 608a287, 2026-10-08; each run's config.yaml states its checker). logs/agent.log · lines 581-584 (Not quoted on a shared page) (misses.py on r004–r006) finds 501 of 652 wrong plain replies have no 'Answer:' line and 211 of those end with the right number. Accuracies from the two graders are not on one scale. · not confirmed
note on e1: q3 (how many problems) has no rule: r011 was run at 500 problems to ask whether 250 is enough (logs/agent.log, 2026-10-12), but it also changed the grader, and one seed at 500 gives no reseed spread there (logs/agent.log · line 669 (Not quoted on a shared page)). · not confirmed

Record

Runs

Each run keeps when its output was produced and when it was recorded, the repository's state when it was recorded, and what it read and made.

9 runs

Prompt style on 250 GSM8K test problems: plain, step by step, worked examples

e19 runs

Recorded 2026-10-23

Seed	Checker	Problems	Accuracy (%)			Produced	run
Seed	Checker	Problems	Plain	Step by step	Worked examples	Produced	run
0	answer_tag	250	14.00			2026-10-06	recorded afterwards
1	answer_tag	250	12.80			2026-10-06	recorded afterwards
2	answer_tag	250	12.40			2026-10-06	recorded afterwards
0	answer_tag	250		18.80		2026-10-06	recorded afterwards
1	answer_tag	250		15.20		2026-10-06	recorded afterwards
2	answer_tag	250		16.00		2026-10-06	recorded afterwards
0	last_number	250			39.60	2026-10-09	recorded afterwards
0	last_number	250	43.20			2026-10-14	recorded afterwards
0	last_number	500		42.20		2026-10-12	recorded afterwards

Resources

What the runs read and made, how each is identified, how to get it again, and what depends on it.

qwen25-05b-instructQwen2.5-0.5B-Instruct 7ae557604adf67be50417f59c2c2f167def9a775presentcheckpoint · read by 9 runs · 5 rules depend on it

identity: version 7ae557604adf67be50417f59c2c2f167def9a775
where: hf:Qwen/Qwen2.5-0.5B-Instruct · recorded · local:/srv/rb/demo/cache/qwen2.5-0.5b-instruct · here
read by: plain · seed 0 · checker answer_tag · problems 250 plain · seed 1 · checker answer_tag · problems 250 plain · seed 2 · checker answer_tag · problems 250 step_by_step · seed 0 · checker answer_tag · problems 250 step_by_step · seed 1 · checker answer_tag · problems 250 step_by_step · seed 2 · checker answer_tag · problems 250 worked_examples · seed 0 · checker last_number · problems 250 plain · seed 0 · checker last_number · problems 250 step_by_step · seed 0 · checker last_number · problems 500

gsm8k-testGSM8K test setreconstructibledataset · read by 9 runs · 5 rules depend on ithow to get it again · Cobbe et al. 2021, MIT license

identity: sha256 3730d312f6e3
where: local:data/gsm8k/test.jsonl · gone · local:data/gsm8k/test.jsonl · gone
read by: plain · seed 0 · checker answer_tag · problems 250 plain · seed 1 · checker answer_tag · problems 250 plain · seed 2 · checker answer_tag · problems 250 step_by_step · seed 0 · checker answer_tag · problems 250 step_by_step · seed 1 · checker answer_tag · problems 250 step_by_step · seed 2 · checker answer_tag · problems 250 worked_examples · seed 0 · checker last_number · problems 250 plain · seed 0 · checker last_number · problems 250 step_by_step · seed 0 · checker last_number · problems 500

History

Each call keeps its reason and its date. A call does not change a finding.

75 changes recorded

2026-10-23, 23:07 to 23:11 UTC · 4 changes · no calls made
Raw log
1. 23:11
  Added a note on the project
2. 23:11
  Retracted a note on e1 · why: c-ref now reads the published 49.6 against the plain prompt
3. 23:11
  Retracted a note on e1 · why: r010 (worked examples) and r012 (plain) both ran on seed 0 at 250 problems under last_number, so a pair of prompts has been run alike; only step by step has no 250-problem run under last_number
4. 23:07
  Added rule c-ref to e1
2026-10-23, 22:57 to 22:59 UTC · 71 changes · no calls made
1. 22:59
  Added 3 notes on e1
2. 22:59
  Retracted a note on e1 · note: cited README.md:9; the line is README.md:12 · not confirmed
3. 22:59
  Retracted a note on e1 · note: cited logs/agent.log:653-655; the line is 669 · not confirmed
4. 22:59
  Retracted a note on e1 · note: cited logs/agent.log:582-585; the lines are 581-584 · not confirmed
5. 22:59
  Added a note on the project
6. 22:59
  Added 7 notes on e1
7. 22:58
  Recorded 9 runs: 9 on e1
8. 22:58
  Added 3 rules to e1: c2, c3, c4
9. 22:58
  Added rule c1 to e1 · note: notes.md:7 · not confirmed
10. 22:58
  Added 2 resources: qwen25-05b-instruct, gsm8k-test
11. 22:58
  Set 17 settings on e1: plain.prompt, step_by_step.prompt, worked_examples.prompt, name, revision and 12 more; rb found 13 of them in their sources
12. 22:58
  Added an experiment: Prompt style on 250 GSM8K test problems: plain, step by step, worked examples (e1)
13. 22:58
  Added metric accuracy
14. 22:58
  Added 2 questions under q1: q2, q3
15. 22:58
  Added question q1
16. 22:57
  Started the record: Think step by step: a small model on grade-school math

Raw log

2026-10-23 23:11 UTC note project think-step-by-step-a-small-model-on-grade-school-math
2026-10-23 23:11 UTC retract a note on e1: c-ref now reads the published 49.6 against the plain prompt
2026-10-23 23:11 UTC retract a note on e1: r010 (worked examples) and r012 (plain) both ran on seed 0 at 250 problems under last_number, so a pair of prompts has been run alike; only step by step has no 250-problem run under last_number
2026-10-23 23:07 UTC add criterion c-ref: accuracy: plain [where checker=answer_tag, problems=250] = 49.6 ± 0.05 · e1
2026-10-23 22:59 UTC note experiment e1
2026-10-23 22:59 UTC note experiment e1
2026-10-23 22:59 UTC note experiment e1
2026-10-23 22:59 UTC retract a note on e1: note: cited README.md:9; the line is README.md:12 · not confirmed
2026-10-23 22:59 UTC retract a note on e1: note: cited logs/agent.log:653-655; the line is 669 · not confirmed
2026-10-23 22:59 UTC retract a note on e1: note: cited logs/agent.log:582-585; the lines are 581-584 · not confirmed
2026-10-23 22:59 UTC note project think-step-by-step-a-small-model-on-grade-school-math
2026-10-23 22:59 UTC note experiment e1
2026-10-23 22:59 UTC note experiment e1
2026-10-23 22:59 UTC note experiment e1
2026-10-23 22:59 UTC note experiment e1
2026-10-23 22:59 UTC note experiment e1
2026-10-23 22:59 UTC note experiment e1
2026-10-23 22:59 UTC note experiment e1
2026-10-23 22:58 UTC record run step_by_step · seed 0 · checker last_number · problems 500 · e1
2026-10-23 22:58 UTC record run plain · seed 0 · checker last_number · problems 250 · e1
2026-10-23 22:58 UTC record run worked_examples · seed 0 · checker last_number · problems 250 · e1
2026-10-23 22:58 UTC record run step_by_step · seed 2 · checker answer_tag · problems 250 · e1
2026-10-23 22:58 UTC record run step_by_step · seed 1 · checker answer_tag · problems 250 · e1
2026-10-23 22:58 UTC record run step_by_step · seed 0 · checker answer_tag · problems 250 · e1
2026-10-23 22:58 UTC record run plain · seed 2 · checker answer_tag · problems 250 · e1
2026-10-23 22:58 UTC record run plain · seed 1 · checker answer_tag · problems 250 · e1
2026-10-23 22:58 UTC record run plain · seed 0 · checker answer_tag · problems 250 · e1
2026-10-23 22:58 UTC add criterion c4: accuracy: worked_examples vs step_by_step [subject where checker=last_number; reference where checker=answer_tag, problems=250] = 22.9 ± 0.05 · e1
2026-10-23 22:58 UTC add criterion c3: each unit's accuracy: step_by_step vs plain [where checker=answer_tag, problems=250] ≥ 0 · e1
2026-10-23 22:58 UTC add criterion c2: accuracy: step_by_step vs plain [where checker=answer_tag, problems=250] = 3.6 ± 0.05 · e1
2026-10-23 22:58 UTC add criterion c1: note: notes.md:7 · not confirmed · e1
2026-10-23 22:58 UTC add resource gsm8k-test
2026-10-23 22:58 UTC add resource qwen25-05b-instruct
2026-10-23 22:58 UTC spec check experiment e1: worked_examples.prompt: checked as stated
2026-10-23 22:58 UTC spec set experiment e1: setting worked_examples.prompt: source configs/prompts.yaml#worked_examples @7265bd1 -> results/r010_worked_s0/config.yaml#prompt
2026-10-23 22:58 UTC spec check experiment e1: step_by_step.prompt: checked as stated
2026-10-23 22:58 UTC spec set experiment e1: setting step_by_step.prompt: source configs/prompts.yaml#step_by_step @32c91d6 -> results/r007_step_s0/config.yaml#prompt
2026-10-23 22:58 UTC spec check experiment e1: plain.prompt: checked as stated
2026-10-23 22:58 UTC spec set experiment e1: setting plain.prompt: source configs/prompts.yaml#plain @32c91d6 -> results/r004_plain_s0/config.yaml#prompt
2026-10-23 22:58 UTC spec set experiment e1: setting population.unit: new, seed (optional)
2026-10-23 22:58 UTC spec set experiment e1: setting problems: new, unknown (per-run)
2026-10-23 22:58 UTC spec set experiment e1: setting checker: new, unknown (per-run)
2026-10-23 22:58 UTC spec set experiment e1: setting seed: new, unknown (per-run)
2026-10-23 22:58 UTC spec check experiment e1: subset_seed: checked as stated
2026-10-23 22:58 UTC spec set experiment e1: setting subset_seed: new, 2026
2026-10-23 22:58 UTC spec check experiment e1: split: checked as stated
2026-10-23 22:58 UTC spec set experiment e1: setting split: new, test
2026-10-23 22:58 UTC spec check experiment e1: max_new_tokens: checked as stated
2026-10-23 22:58 UTC spec set experiment e1: setting max_new_tokens: new, 512
2026-10-23 22:58 UTC spec check experiment e1: repetition_penalty: checked as stated
2026-10-23 22:58 UTC spec set experiment e1: setting repetition_penalty: new, 1.1
2026-10-23 22:58 UTC spec check experiment e1: top_k: checked as stated
2026-10-23 22:58 UTC spec set experiment e1: setting top_k: new, 20
2026-10-23 22:58 UTC spec check experiment e1: top_p: checked as stated
2026-10-23 22:58 UTC spec set experiment e1: setting top_p: new, 0.8
2026-10-23 22:58 UTC spec check experiment e1: temperature: checked as stated
2026-10-23 22:58 UTC spec set experiment e1: setting temperature: new, 0.7
2026-10-23 22:58 UTC spec check experiment e1: dtype: checked as stated
2026-10-23 22:58 UTC spec set experiment e1: setting dtype: new, bfloat16
2026-10-23 22:58 UTC spec check experiment e1: revision: checked as stated
2026-10-23 22:58 UTC spec set experiment e1: setting revision: new, 7ae557604adf67be50417f59c2c2f167def9a775
2026-10-23 22:58 UTC spec check experiment e1: name: checked as stated
2026-10-23 22:58 UTC spec set experiment e1: setting name: new, Qwen/Qwen2.5-0.5B-Instruct
2026-10-23 22:58 UTC spec check experiment e1: worked_examples.prompt: provisional
2026-10-23 22:58 UTC spec set experiment e1: setting worked_examples.prompt: new, worked_examples
2026-10-23 22:58 UTC spec check experiment e1: step_by_step.prompt: provisional
2026-10-23 22:58 UTC spec set experiment e1: setting step_by_step.prompt: new, step_by_step
2026-10-23 22:58 UTC spec check experiment e1: plain.prompt: provisional
2026-10-23 22:58 UTC spec set experiment e1: setting plain.prompt: new, plain
2026-10-23 22:58 UTC add experiment e1
2026-10-23 22:58 UTC add metric accuracy
2026-10-23 22:58 UTC add question q3 · q1
2026-10-23 22:58 UTC add question q2 · q1
2026-10-23 22:58 UTC add question q1
2026-10-23 22:57 UTC init project think-step-by-step-a-small-model-on-grade-school-math

Open callsYour call (3)

Calls are made with rb in a terminal, never on this page.

Your calls are made with rb in a terminal; this page gives you the command.

Calls are made with rb in a terminal, not on this page; it shows where they stand.

Not reproduced · estimate 13.07 outside 49.6 ± 0.05The plain prompt reaches the published 49.6 on GSM8K c-ref

Review this result

Accept
```
rb decide c-ref accept --why '‹why›'
```
The record will say: accepted ‹day›: ‹why›
Reject
```
rb decide c-ref reject --why '‹why›'
```
The record will say: rejected ‹day›: ‹why›
Investigate
```
rb decide c-ref investigate --why '‹why›'
```
The record will say: investigating since ‹day›: ‹why›

Rule not met · estimate +3.6 < 5 · on the estimate, not across the whole intervalStep by step scores at least 5 points above the plain prompt c1

Review this result

Accept
```
rb decide c1 accept --why '‹why›'
```
The record will say: accepted ‹day›: ‹why›
Reject
```
rb decide c1 reject --why '‹why›'
```
The record will say: rejected ‹day›: ‹why›
Investigate
```
rb decide c1 investigate --why '‹why›'
```
The record will say: investigating since ‹day›: ‹why›

Too little evidence · 1 seed · not reproducedThe claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from c4

Review this result

Accept
```
rb decide c4 accept --why '‹why›'
```
The record will say: accepted ‹day›: ‹why›
Reject
```
rb decide c4 reject --why '‹why›'
```
The record will say: rejected ‹day›: ‹why›
Investigate
```
rb decide c4 investigate --why '‹why›'
```
The record will say: investigating since ‹day›: ‹why›

Waiting for a confirmation

1 design waits for your confirmation, listed apart

Also open in the record (3) →

Research › q1 › e1 › c1

Step by step scores at least 5 points above the plain prompt

ruleprimary in e1c1

Not met: step by step is 3.60 points higher than the plain baseline (+0.62 to +6.58), short of the 5 points the rule asks for.

What rb measured

+3.60 points (+0.62 to +6.58)

3 seeds from 6 runs

shaded: what the rule accepts · dot: estimate · bar: interval

The rule as written

passes when: The estimate is at least 5 points (step by step against the plain baseline).
margin: -1.40 points
timing: Written after its results came in.
units: 3 seeds from 6 runs · declared: unit = seed
checks: ok
review: No call made on it. · Open callsYour call

Runs it rests on 3 seeds from 6 runs

Every run here: answer-tag, problems 250.

arm	run	value
plain	seed 0	14.00%
plain	seed 1	12.80%
plain	seed 2	12.40%
step by step	seed 0	18.80%
step by step	seed 1	15.20%
step by step	seed 2	16.00%

What depends on this

question link · q1
Does telling a small AI model to think step by step make it solve more grade-school math problems?

Source details

rule: accuracy: step_by_step vs plain [where checker=answer_tag, problems=250] ≥ 5
rb reads: estimate +3.6 < 5
timing: declared after the evidence existed: all 3 units were produced before it was declared
comparison: accuracy: step_by_step vs plain [where checker=answer_tag, problems=250]: +3.6 [+0.619, +6.581] percent · paired difference, t interval, 3 seeds, 95% (rb)
evidence: all 6 of its runs recorded afterwards
declared: 2026-10-23 22:58 UTC
experiment: e1 · primary
metric: accuracy: step_by_step vs plain [where checker=answer_tag, problems=250]: step_by_step against plain
method: paired difference, t interval · 95% interval
question link: q1 · 2026-10-23 22:58 UTC

Record a call

Still open

Nothing is recorded on it yet.

Its actions and their lines are on its call card.

Research › q1 › e1 › c2

The claim that step by step is 3.6 points above plain on the seed averages is checked here

rulesecondary in e1citedc2

Reproduced: step by step is 3.60 points higher than the plain baseline (+0.62 to +6.58), within 0.05 points of the 3.6 points.

What rb measured

+3.60 points (+0.62 to +6.58)

3 seeds from 6 runs

shaded: what the rule accepts · dot: estimate · bar: interval

The rule as written

passes when: The estimate is within 0.05 points of 3.6 points (step by step against the plain baseline).
margin: 0.05 points
timing: Written after its results came in.
units: 3 seeds from 6 runs · declared: unit = seed
checks: ok
review: No call made on it.

Runs it rests on 3 seeds from 6 runs

Every run here: answer-tag, problems 250.

arm	run	value
plain	seed 0	14.00%
plain	seed 1	12.80%
plain	seed 2	12.40%
step by step	seed 0	18.80%
step by step	seed 1	15.20%
step by step	seed 2	16.00%

What depends on this

question link · q1
Does telling a small AI model to think step by step make it solve more grade-school math problems?

Source details

rule: accuracy: step_by_step vs plain [where checker=answer_tag, problems=250] = 3.6 ± 0.05
rb reads: estimate +3.6 within 3.6 ± 0.05
timing: declared after the evidence existed: all 3 units were produced before it was declared
comparison: accuracy: step_by_step vs plain [where checker=answer_tag, problems=250]: +3.6 [+0.619, +6.581] percent · paired difference, t interval, 3 seeds, 95% (rb)
evidence: all 6 of its runs recorded afterwards
declared: 2026-10-23 22:58 UTC
experiment: e1 · secondary · cited
metric: accuracy: step_by_step vs plain [where checker=answer_tag, problems=250]: step_by_step against plain
reference: 3.6 ± 0.05
method: paired difference, t interval · 95% interval
source: logs/agent.log:719 · sha256 d49a12fd8e19
question link: q1 · 2026-10-23 22:58 UTC
cited: logs/agent.log:719Step by step against plain: +3.6 points on the averages; the rule in notes.md asks for 5.sha256 d49a12fd8e19776d68b400288bdc6ca6ff77e6cfcca4be272f6603088768b94b

Research › q1 › e1 › c3

The claim that the step-by-step gain over plain holds on every seed is checked here

rulesecondary in e1citedc3

Reproduced: step by step is 3.60 points higher than the plain baseline (+0.62 to +6.58); every unit's value meets it.

What rb measured

+3.60 points (+0.62 to +6.58)

3 seeds from 6 runs

shaded: what the rule accepts · dot: estimate · bar: interval

The rule as written

passes when: Every unit's value is at least 0 points (step by step against the plain baseline).
timing: Written after its results came in.
units: 3 seeds from 6 runs · declared: unit = seed
checks: ok
review: No call made on it.

Runs it rests on 3 seeds from 6 runs

Every run here: answer-tag, problems 250.

arm	run	value
plain	seed 0	14.00%
plain	seed 1	12.80%
plain	seed 2	12.40%
step by step	seed 0	18.80%
step by step	seed 1	15.20%
step by step	seed 2	16.00%

What depends on this

question link · q1
Does telling a small AI model to think step by step make it solve more grade-school math problems?

Source details

rule: each unit's accuracy: step_by_step vs plain [where checker=answer_tag, problems=250] ≥ 0
rb reads: all 3 units ≥ 0
timing: declared after the evidence existed: all 3 units were produced before it was declared
comparison: accuracy: step_by_step vs plain [where checker=answer_tag, problems=250]: +3.6 [+0.619, +6.581] percent · paired difference, t interval, 3 seeds, 95% (rb)
evidence: all 6 of its runs recorded afterwards
declared: 2026-10-23 22:58 UTC
experiment: e1 · secondary · cited
metric: accuracy: step_by_step vs plain [where checker=answer_tag, problems=250]: step_by_step against plain
reference: 0
method: paired difference, t interval · 95% interval
source: logs/agent.log:557 · sha256 d49a12fd8e19
question link: q1 · 2026-10-23 22:58 UTC
cited: logs/agent.log:557The gap holds on every seed.sha256 d49a12fd8e19776d68b400288bdc6ca6ff77e6cfcca4be272f6603088768b94b

Research › q2 › e1 › c4

The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from

rulesecondary in e1citedc4

Not reproduced: worked examples is 20.80 points higher than step by step (no interval yet), not within 0.05 points of the 22.9 points.

What rb measured

+20.80 points

1 seed from 2 runs · 2 runs not counted

shaded: what the rule accepts · dot: estimate (no interval)

The rule as written

passes when: The estimate is within 0.05 points of 22.9 points (worked examples against step by step).
margin: -2.05 points
timing: Written after its results came in.
units: 1 seed from 2 runs · declared: unit = seed
checks: ok
review: No call made on it. · Open callsYour call

Runs it rests on 1 seed from 2 runs · 2 runs not counted

Every run here: problems 250.

2 runs are not counted: its unit has no worked_examples run, so the paired reading leaves it out.

arm	run	value
step by step	seed 0, answer-tag	18.80%
worked examples	seed 0, last-number	39.60%
step by step	seed 1, answer-tagnot counted	15.20%
step by step	seed 2, answer-tagnot counted	16.00%

What depends on this

question link · q2
Do worked examples help more than "think step by step"?

Source details

rule: accuracy: worked_examples vs step_by_step [subject where checker=last_number; reference where checker=answer_tag, problems=250] = 22.9 ± 0.05
rb reads: estimate +20.8 outside 22.9 ± 0.05
timing: declared after the evidence existed: the only unit was produced before it was declared, and so were 2 unpaired runs (run step_by_step · seed 1 · checker answer_tag · problems 250, run step_by_step · seed 2 · checker answer_tag · problems 250)
comparison: accuracy: worked_examples vs step_by_step [subject where checker=last_number; reference where checker=answer_tag, problems=250]: +20.8 percent · paired difference, 1 seed (rb)
evidence: both its runs recorded afterwards
declared: 2026-10-23 22:58 UTC
experiment: e1 · secondary · cited
metric: accuracy: worked_examples vs step_by_step [subject where checker=last_number; reference where checker=answer_tag, problems=250]: worked_examples against step_by_step
reference: 22.9 ± 0.05
method: paired difference · no interval
source: logs/agent.log:720 · sha256 d49a12fd8e19
question link: q2 · 2026-10-23 22:58 UTC
cited: logs/agent.log:720Worked examples against step by step: +22.9 points.sha256 d49a12fd8e19776d68b400288bdc6ca6ff77e6cfcca4be272f6603088768b94b

Record a call

Still open

Too little evidence

Its actions and their lines are on its call card.

Research › q1 › e1 › c-ref

The plain prompt reaches the published 49.6 on GSM8K

rulesecondary in e1citedc-ref

Not reproduced: plain baseline: 13.07% (11.00% to 15.14%), not within 0.05% of the cited 49.6%.

What rb measured

13.07% (11.00% to 15.14%)

3 runs

shaded: what the rule accepts · dot: estimate · bar: interval

The rule as written

passes when: The estimate is within 0.05% of 49.6%.
margin: -36.48 points
timing: Written after its results came in.
units: 3 seeds · declared: unit = seed
checks: ok
review: No call made on it. · Open callsYour call

Runs it rests on 3 runs

Every run here: answer-tag, problems 250.

Every run here is plain baseline.

arm	run	value
	seed 0	14.00%
	seed 1	12.80%
	seed 2	12.40%

What depends on this

question link · q1
Does telling a small AI model to think step by step make it solve more grade-school math problems?

Source details

rule: accuracy: plain [where checker=answer_tag, problems=250] = 49.6 ± 0.05
rb reads: estimate 13.07 outside 49.6 ± 0.05
timing: declared after the evidence existed: all 3 units were produced before it was declared
comparison: accuracy: plain [where checker=answer_tag, problems=250]: 13.07 [11, 15.14] percent · mean, t interval, 3 seeds, 95% (rb)
evidence: all 3 of its runs recorded afterwards
declared: 2026-10-23 23:07 UTC
experiment: e1 · secondary · cited
metric: accuracy: plain [where checker=answer_tag, problems=250]
reference: 49.6 ± 0.05
method: mean, t interval · 95% interval
source: README.md:12 · read at commit 6adf8a5 · sha256 7029e0be441f
question link: q1 · 2026-10-23 23:07 UTC
cited: README.md:12 · README.md:12Qwen2.5-0.5B-Instruct scores 49.6 on GSM8Kread at commit 6adf8a50067e093830a647d62fce0d68bc332f70 · sha256 7029e0be441faa31bb434075899b69a46f0c607ff2e05ff232b06a971ad19e63

Record a call

Still open

Nothing is recorded on it yet.

Its actions and their lines are on its call card.

plain, seed 0, answer-tag, problems 250

runin e1 · produced 2026-10-06 · recorded 2026-10-23

note: results/r004_plain_s0 · not confirmed

Counts toward 4 rules

Step by step scores at least 5 points above the plain prompt c1 · rule not met
The claim that step by step is 3.6 points above plain on the seed averages is checked here c2 · reproduced
The claim that the step-by-step gain over plain holds on every seed is checked here c3 · reproduced
The plain prompt reaches the published 49.6 on GSM8K c-ref · not reproduced

Results

Accuracy, plain: 14.00

Source details

command (as given)python solve.py --prompt plain --seed 0 --out results/r004_plain_s0how rb knowsresults/r004_plain_s0/config.yaml:1 records this launch

produced: 2026-10-06 01:42 UTC (as reported) · recorded 2026-10-23 22:58 UTC
inputs: read: qwen25-05b-instruct · present when recorded
inputs: read: gsm8k-test · present when recorded · sha256 3730d312f6e3
repository: when recorded, not when run: commit 6adf8a5006 · tree dirty
config: 14 matched
basis: file · spec not locked
environment: python 3.12.15 · rb 0.6.0 · pydantic 2.14.1 (where it was recorded: the run was made before it was recorded)
results: results/r004_plain_s0/replies.jsonl · sha256 2618abedd74c · 257180 bytes
results: results/r004_plain_s0/metrics.json · sha256 6c9de9a3c018 · 151 bytes
note: results/r004_plain_s0 · not confirmed
type: recorded afterwards
results as rb gives them: plain: accuracy=14.0

Reproduce this run · 1 of 5 parts ready

to run again: ✓ 1 ok · ◐ 1 partial · ✗ 2 missing · ○ 1 unknown

the whole recipe for e1

missing · commit 32c91d6 · not in this repository: 32c91d6 (get the code repository to rerun)

unknown · the run was made before it was recorded, in an environment rb did not see

ok · qwen25-05b-instruct: present

partial · gsm8k-test: reconstructible

missing · neither of the 2 files the runs named is here unchanged; not here: results/r004_plain_s0/replies.jsonl, results/r004_plain_s0/metrics.json

# Reproduce a run of e1 · plain · seed 0 · checker answer_tag · problems 250
# Recipe, from what the record holds. rb did not run these commands.
# code: missing · commit 32c91d6 · not in this repository: 32c91d6 (get the code repository to rerun)
# environment: unknown · the run was made before it was recorded, in an environment rb did not see
# checkpoint: ok · qwen25-05b-instruct: present
# dataset: partial · gsm8k-test: reconstructible
# files: missing · neither of the 2 files the runs named is here unchanged; not here: results/r004_plain_s0/replies.jsonl, results/r004_plain_s0/metrics.json

# checkpoint · qwen25-05b-instruct: no recipe recorded · configs/model.yaml; Apache 2.0

# dataset · gsm8k-test: no recipe recorded · Cobbe et al. 2021, MIT license

# per run: seed 0 · checker answer_tag · problems 250
# settings: name=Qwen/Qwen2.5-0.5B-Instruct revision=7ae557604adf67be50417f59c2c2f167def9a775 dtype=bfloat16 temperature=0.7 top_p=0.8 top_k=20 repetition_penalty=1.1 max_new_tokens=512 split=test subset_seed=2026
#   plain: prompt=plain
#   step_by_step: prompt=step_by_step
#   worked_examples: prompt=worked_examples
# command (as given) · results/r004_plain_s0/config.yaml:1 records this launch
python solve.py --prompt plain --seed 0 --out results/r004_plain_s0

plain, seed 1, answer-tag, problems 250

runin e1 · produced 2026-10-06 · recorded 2026-10-23

note: results/r005_plain_s1 · not confirmed

Counts toward 4 rules

Step by step scores at least 5 points above the plain prompt c1 · rule not met
The claim that step by step is 3.6 points above plain on the seed averages is checked here c2 · reproduced
The claim that the step-by-step gain over plain holds on every seed is checked here c3 · reproduced
The plain prompt reaches the published 49.6 on GSM8K c-ref · not reproduced

Results

Accuracy, plain: 12.80

Source details

command (as given)python solve.py --prompt plain --seed 1 --out results/r005_plain_s1how rb knowsresults/r005_plain_s1/config.yaml:1 records this launch

produced: 2026-10-06 01:52 UTC (as reported) · recorded 2026-10-23 22:58 UTC
inputs: read: qwen25-05b-instruct · present when recorded
inputs: read: gsm8k-test · present when recorded · sha256 3730d312f6e3
repository: when recorded, not when run: commit 6adf8a5006 · tree dirty
config: 14 matched
basis: file · spec not locked
environment: python 3.12.15 · rb 0.6.0 · pydantic 2.14.1 (where it was recorded: the run was made before it was recorded)
results: results/r005_plain_s1/replies.jsonl · sha256 50a7d0eff8d9 · 258222 bytes
results: results/r005_plain_s1/metrics.json · sha256 b56c3780c0aa · 151 bytes
note: results/r005_plain_s1 · not confirmed
type: recorded afterwards
results as rb gives them: plain: accuracy=12.8

Reproduce this run · 1 of 5 parts ready

to run again: ✓ 1 ok · ◐ 1 partial · ✗ 2 missing · ○ 1 unknown

the whole recipe for e1

missing · commit 32c91d6 · not in this repository: 32c91d6 (get the code repository to rerun)

unknown · the run was made before it was recorded, in an environment rb did not see

ok · qwen25-05b-instruct: present

partial · gsm8k-test: reconstructible

missing · neither of the 2 files the runs named is here unchanged; not here: results/r005_plain_s1/replies.jsonl, results/r005_plain_s1/metrics.json

# Reproduce a run of e1 · plain · seed 1 · checker answer_tag · problems 250
# Recipe, from what the record holds. rb did not run these commands.
# code: missing · commit 32c91d6 · not in this repository: 32c91d6 (get the code repository to rerun)
# environment: unknown · the run was made before it was recorded, in an environment rb did not see
# checkpoint: ok · qwen25-05b-instruct: present
# dataset: partial · gsm8k-test: reconstructible
# files: missing · neither of the 2 files the runs named is here unchanged; not here: results/r005_plain_s1/replies.jsonl, results/r005_plain_s1/metrics.json

# checkpoint · qwen25-05b-instruct: no recipe recorded · configs/model.yaml; Apache 2.0

# dataset · gsm8k-test: no recipe recorded · Cobbe et al. 2021, MIT license

# per run: seed 1 · checker answer_tag · problems 250
# settings: name=Qwen/Qwen2.5-0.5B-Instruct revision=7ae557604adf67be50417f59c2c2f167def9a775 dtype=bfloat16 temperature=0.7 top_p=0.8 top_k=20 repetition_penalty=1.1 max_new_tokens=512 split=test subset_seed=2026
#   plain: prompt=plain
#   step_by_step: prompt=step_by_step
#   worked_examples: prompt=worked_examples
# command (as given) · results/r005_plain_s1/config.yaml:1 records this launch
python solve.py --prompt plain --seed 1 --out results/r005_plain_s1

plain, seed 2, answer-tag, problems 250

runin e1 · produced 2026-10-06 · recorded 2026-10-23

note: results/r006_plain_s2 · not confirmed

Counts toward 4 rules

Step by step scores at least 5 points above the plain prompt c1 · rule not met
The claim that step by step is 3.6 points above plain on the seed averages is checked here c2 · reproduced
The claim that the step-by-step gain over plain holds on every seed is checked here c3 · reproduced
The plain prompt reaches the published 49.6 on GSM8K c-ref · not reproduced

Results

Accuracy, plain: 12.40

Source details

command (as given)python solve.py --prompt plain --seed 2 --out results/r006_plain_s2how rb knowsresults/r006_plain_s2/config.yaml:1 records this launch

produced: 2026-10-06 02:07 UTC (as reported) · recorded 2026-10-23 22:58 UTC
inputs: read: qwen25-05b-instruct · present when recorded
inputs: read: gsm8k-test · present when recorded · sha256 3730d312f6e3
repository: when recorded, not when run: commit 6adf8a5006 · tree dirty
config: 14 matched
basis: file · spec not locked
environment: python 3.12.15 · rb 0.6.0 · pydantic 2.14.1 (where it was recorded: the run was made before it was recorded)
results: results/r006_plain_s2/replies.jsonl · sha256 15d577f5e0d7 · 258186 bytes
results: results/r006_plain_s2/metrics.json · sha256 91ef13fdf236 · 151 bytes
note: results/r006_plain_s2 · not confirmed
type: recorded afterwards
results as rb gives them: plain: accuracy=12.4

Reproduce this run · 1 of 5 parts ready

to run again: ✓ 1 ok · ◐ 1 partial · ✗ 2 missing · ○ 1 unknown

the whole recipe for e1

missing · commit 32c91d6 · not in this repository: 32c91d6 (get the code repository to rerun)

unknown · the run was made before it was recorded, in an environment rb did not see

ok · qwen25-05b-instruct: present

partial · gsm8k-test: reconstructible

missing · neither of the 2 files the runs named is here unchanged; not here: results/r006_plain_s2/replies.jsonl, results/r006_plain_s2/metrics.json

# Reproduce a run of e1 · plain · seed 2 · checker answer_tag · problems 250
# Recipe, from what the record holds. rb did not run these commands.
# code: missing · commit 32c91d6 · not in this repository: 32c91d6 (get the code repository to rerun)
# environment: unknown · the run was made before it was recorded, in an environment rb did not see
# checkpoint: ok · qwen25-05b-instruct: present
# dataset: partial · gsm8k-test: reconstructible
# files: missing · neither of the 2 files the runs named is here unchanged; not here: results/r006_plain_s2/replies.jsonl, results/r006_plain_s2/metrics.json

# checkpoint · qwen25-05b-instruct: no recipe recorded · configs/model.yaml; Apache 2.0

# dataset · gsm8k-test: no recipe recorded · Cobbe et al. 2021, MIT license

# per run: seed 2 · checker answer_tag · problems 250
# settings: name=Qwen/Qwen2.5-0.5B-Instruct revision=7ae557604adf67be50417f59c2c2f167def9a775 dtype=bfloat16 temperature=0.7 top_p=0.8 top_k=20 repetition_penalty=1.1 max_new_tokens=512 split=test subset_seed=2026
#   plain: prompt=plain
#   step_by_step: prompt=step_by_step
#   worked_examples: prompt=worked_examples
# command (as given) · results/r006_plain_s2/config.yaml:1 records this launch
python solve.py --prompt plain --seed 2 --out results/r006_plain_s2

step by step, seed 0, answer-tag, problems 250

runin e1 · produced 2026-10-06 · recorded 2026-10-23

note: results/r007_step_s0 · not confirmed

Counts toward 4 rules

Step by step scores at least 5 points above the plain prompt c1 · rule not met
The claim that step by step is 3.6 points above plain on the seed averages is checked here c2 · reproduced
The claim that the step-by-step gain over plain holds on every seed is checked here c3 · reproduced
The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from c4 · not reproduced

Results

Accuracy, step by step: 18.80

Source details

command (as given)python solve.py --prompt step_by_step --seed 0 --out results/r007_step_s0how rb knowsresults/r007_step_s0/config.yaml:1 records this launch

produced: 2026-10-06 02:16 UTC (as reported) · recorded 2026-10-23 22:58 UTC
inputs: read: qwen25-05b-instruct · present when recorded
inputs: read: gsm8k-test · present when recorded · sha256 3730d312f6e3
repository: when recorded, not when run: commit 6adf8a5006 · tree dirty
config: 14 matched
basis: file · spec not locked
environment: python 3.12.15 · rb 0.6.0 · pydantic 2.14.1 (where it was recorded: the run was made before it was recorded)
results: results/r007_step_s0/replies.jsonl · sha256 a96256fbc0d8 · 271352 bytes
results: results/r007_step_s0/metrics.json · sha256 25ab5a992436 · 158 bytes
note: results/r007_step_s0 · not confirmed
type: recorded afterwards
results as rb gives them: step_by_step: accuracy=18.8

Reproduce this run · 1 of 5 parts ready

to run again: ✓ 1 ok · ◐ 1 partial · ✗ 2 missing · ○ 1 unknown

the whole recipe for e1

missing · commit 32c91d6 · not in this repository: 32c91d6 (get the code repository to rerun)

unknown · the run was made before it was recorded, in an environment rb did not see

ok · qwen25-05b-instruct: present

partial · gsm8k-test: reconstructible

missing · neither of the 2 files the runs named is here unchanged; not here: results/r007_step_s0/replies.jsonl, results/r007_step_s0/metrics.json

# Reproduce a run of e1 · step_by_step · seed 0 · checker answer_tag · problems 250
# Recipe, from what the record holds. rb did not run these commands.
# code: missing · commit 32c91d6 · not in this repository: 32c91d6 (get the code repository to rerun)
# environment: unknown · the run was made before it was recorded, in an environment rb did not see
# checkpoint: ok · qwen25-05b-instruct: present
# dataset: partial · gsm8k-test: reconstructible
# files: missing · neither of the 2 files the runs named is here unchanged; not here: results/r007_step_s0/replies.jsonl, results/r007_step_s0/metrics.json

# checkpoint · qwen25-05b-instruct: no recipe recorded · configs/model.yaml; Apache 2.0

# dataset · gsm8k-test: no recipe recorded · Cobbe et al. 2021, MIT license

# per run: seed 0 · checker answer_tag · problems 250
# settings: name=Qwen/Qwen2.5-0.5B-Instruct revision=7ae557604adf67be50417f59c2c2f167def9a775 dtype=bfloat16 temperature=0.7 top_p=0.8 top_k=20 repetition_penalty=1.1 max_new_tokens=512 split=test subset_seed=2026
#   plain: prompt=plain
#   step_by_step: prompt=step_by_step
#   worked_examples: prompt=worked_examples
# command (as given) · results/r007_step_s0/config.yaml:1 records this launch
python solve.py --prompt step_by_step --seed 0 --out results/r007_step_s0

step by step, seed 1, answer-tag, problems 250

runin e1 · produced 2026-10-06 · recorded 2026-10-23

note: results/r008_step_s1 · not confirmed

Counts toward 3 rules · not counted in 1

Counted in 3 rules.

Step by step scores at least 5 points above the plain prompt c1 · rule not met
The claim that step by step is 3.6 points above plain on the seed averages is checked here c2 · reproduced
The claim that the step-by-step gain over plain holds on every seed is checked here c3 · reproduced

Not counted in 1 rule: its unit has no worked_examples run, so the paired reading leaves it out.

The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from c4 · not reproduced

Results

Accuracy, step by step: 15.20

Source details

command (as given)python solve.py --prompt step_by_step --seed 1 --out results/r008_step_s1how rb knowsresults/r008_step_s1/config.yaml:1 records this launch

produced: 2026-10-06 02:24 UTC (as reported) · recorded 2026-10-23 22:58 UTC
inputs: read: qwen25-05b-instruct · present when recorded
inputs: read: gsm8k-test · present when recorded · sha256 3730d312f6e3
repository: when recorded, not when run: commit 6adf8a5006 · tree dirty
config: 14 matched
basis: file · spec not locked
environment: python 3.12.15 · rb 0.6.0 · pydantic 2.14.1 (where it was recorded: the run was made before it was recorded)
results: results/r008_step_s1/replies.jsonl · sha256 871d35df03a6 · 273020 bytes
results: results/r008_step_s1/metrics.json · sha256 2fe4fe0e92de · 158 bytes
note: results/r008_step_s1 · not confirmed
type: recorded afterwards
results as rb gives them: step_by_step: accuracy=15.2

Reproduce this run · 1 of 5 parts ready

to run again: ✓ 1 ok · ◐ 1 partial · ✗ 2 missing · ○ 1 unknown

the whole recipe for e1

missing · commit 32c91d6 · not in this repository: 32c91d6 (get the code repository to rerun)

unknown · the run was made before it was recorded, in an environment rb did not see

ok · qwen25-05b-instruct: present

partial · gsm8k-test: reconstructible

missing · neither of the 2 files the runs named is here unchanged; not here: results/r008_step_s1/replies.jsonl, results/r008_step_s1/metrics.json

# Reproduce a run of e1 · step_by_step · seed 1 · checker answer_tag · problems 250
# Recipe, from what the record holds. rb did not run these commands.
# code: missing · commit 32c91d6 · not in this repository: 32c91d6 (get the code repository to rerun)
# environment: unknown · the run was made before it was recorded, in an environment rb did not see
# checkpoint: ok · qwen25-05b-instruct: present
# dataset: partial · gsm8k-test: reconstructible
# files: missing · neither of the 2 files the runs named is here unchanged; not here: results/r008_step_s1/replies.jsonl, results/r008_step_s1/metrics.json

# checkpoint · qwen25-05b-instruct: no recipe recorded · configs/model.yaml; Apache 2.0

# dataset · gsm8k-test: no recipe recorded · Cobbe et al. 2021, MIT license

# per run: seed 1 · checker answer_tag · problems 250
# settings: name=Qwen/Qwen2.5-0.5B-Instruct revision=7ae557604adf67be50417f59c2c2f167def9a775 dtype=bfloat16 temperature=0.7 top_p=0.8 top_k=20 repetition_penalty=1.1 max_new_tokens=512 split=test subset_seed=2026
#   plain: prompt=plain
#   step_by_step: prompt=step_by_step
#   worked_examples: prompt=worked_examples
# command (as given) · results/r008_step_s1/config.yaml:1 records this launch
python solve.py --prompt step_by_step --seed 1 --out results/r008_step_s1

step by step, seed 2, answer-tag, problems 250

runin e1 · produced 2026-10-06 · recorded 2026-10-23

note: results/r009_step_s2 · not confirmed

Counts toward 3 rules · not counted in 1

Counted in 3 rules.

Step by step scores at least 5 points above the plain prompt c1 · rule not met
The claim that step by step is 3.6 points above plain on the seed averages is checked here c2 · reproduced
The claim that the step-by-step gain over plain holds on every seed is checked here c3 · reproduced

Not counted in 1 rule: its unit has no worked_examples run, so the paired reading leaves it out.

The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from c4 · not reproduced

Results

Accuracy, step by step: 16.00

Source details

command (as given)python solve.py --prompt step_by_step --seed 2 --out results/r009_step_s2how rb knowsresults/r009_step_s2/config.yaml:1 records this launch

produced: 2026-10-06 02:32 UTC (as reported) · recorded 2026-10-23 22:58 UTC
inputs: read: qwen25-05b-instruct · present when recorded
inputs: read: gsm8k-test · present when recorded · sha256 3730d312f6e3
repository: when recorded, not when run: commit 6adf8a5006 · tree dirty
config: 14 matched
basis: file · spec not locked
environment: python 3.12.15 · rb 0.6.0 · pydantic 2.14.1 (where it was recorded: the run was made before it was recorded)
results: results/r009_step_s2/replies.jsonl · sha256 1c50f9bc39d3 · 273707 bytes
results: results/r009_step_s2/metrics.json · sha256 71e83bc2e289 · 158 bytes
note: results/r009_step_s2 · not confirmed
type: recorded afterwards
results as rb gives them: step_by_step: accuracy=16.0

Reproduce this run · 1 of 5 parts ready

to run again: ✓ 1 ok · ◐ 1 partial · ✗ 2 missing · ○ 1 unknown

the whole recipe for e1

missing · commit 32c91d6 · not in this repository: 32c91d6 (get the code repository to rerun)

unknown · the run was made before it was recorded, in an environment rb did not see

ok · qwen25-05b-instruct: present

partial · gsm8k-test: reconstructible

missing · neither of the 2 files the runs named is here unchanged; not here: results/r009_step_s2/replies.jsonl, results/r009_step_s2/metrics.json

# Reproduce a run of e1 · step_by_step · seed 2 · checker answer_tag · problems 250
# Recipe, from what the record holds. rb did not run these commands.
# code: missing · commit 32c91d6 · not in this repository: 32c91d6 (get the code repository to rerun)
# environment: unknown · the run was made before it was recorded, in an environment rb did not see
# checkpoint: ok · qwen25-05b-instruct: present
# dataset: partial · gsm8k-test: reconstructible
# files: missing · neither of the 2 files the runs named is here unchanged; not here: results/r009_step_s2/replies.jsonl, results/r009_step_s2/metrics.json

# checkpoint · qwen25-05b-instruct: no recipe recorded · configs/model.yaml; Apache 2.0

# dataset · gsm8k-test: no recipe recorded · Cobbe et al. 2021, MIT license

# per run: seed 2 · checker answer_tag · problems 250
# settings: name=Qwen/Qwen2.5-0.5B-Instruct revision=7ae557604adf67be50417f59c2c2f167def9a775 dtype=bfloat16 temperature=0.7 top_p=0.8 top_k=20 repetition_penalty=1.1 max_new_tokens=512 split=test subset_seed=2026
#   plain: prompt=plain
#   step_by_step: prompt=step_by_step
#   worked_examples: prompt=worked_examples
# command (as given) · results/r009_step_s2/config.yaml:1 records this launch
python solve.py --prompt step_by_step --seed 2 --out results/r009_step_s2

worked examples, seed 0, last-number, problems 250

runin e1 · produced 2026-10-09 · recorded 2026-10-23

note: results/r010_worked_s0 · not confirmed

Counts toward 1 rule

The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from c4 · not reproduced

Results

Accuracy, worked examples: 39.60

Source details

command (as given)python solve.py --prompt worked_examples --seed 0 --out results/r010_worked_s0how rb knowsresults/r010_worked_s0/config.yaml:1 records this launch

produced: 2026-10-09 02:31 UTC (as reported) · recorded 2026-10-23 22:58 UTC
inputs: read: qwen25-05b-instruct · present when recorded
inputs: read: gsm8k-test · present when recorded · sha256 3730d312f6e3
repository: when recorded, not when run: commit 6adf8a5006 · tree dirty
config: 14 matched
basis: file · spec not locked
environment: python 3.12.15 · rb 0.6.0 · pydantic 2.14.1 (where it was recorded: the run was made before it was recorded)
results: results/r010_worked_s0/replies.jsonl · sha256 3997fcd8b135 · 145253 bytes
results: results/r010_worked_s0/metrics.json · sha256 3ed436b74478 · 161 bytes
note: results/r010_worked_s0 · not confirmed
type: recorded afterwards
results as rb gives them: worked_examples: accuracy=39.6

Reproduce this run · 1 of 5 parts ready

to run again: ✓ 1 ok · ◐ 1 partial · ✗ 2 missing · ○ 1 unknown

the whole recipe for e1

missing · commit 7265bd1 · not in this repository: 7265bd1 (get the code repository to rerun)

unknown · the run was made before it was recorded, in an environment rb did not see

ok · qwen25-05b-instruct: present

partial · gsm8k-test: reconstructible

missing · neither of the 2 files the runs named is here unchanged; not here: results/r010_worked_s0/replies.jsonl, results/r010_worked_s0/metrics.json

# Reproduce a run of e1 · worked_examples · seed 0 · checker last_number · problems 250
# Recipe, from what the record holds. rb did not run these commands.
# code: missing · commit 7265bd1 · not in this repository: 7265bd1 (get the code repository to rerun)
# environment: unknown · the run was made before it was recorded, in an environment rb did not see
# checkpoint: ok · qwen25-05b-instruct: present
# dataset: partial · gsm8k-test: reconstructible
# files: missing · neither of the 2 files the runs named is here unchanged; not here: results/r010_worked_s0/replies.jsonl, results/r010_worked_s0/metrics.json

# checkpoint · qwen25-05b-instruct: no recipe recorded · configs/model.yaml; Apache 2.0

# dataset · gsm8k-test: no recipe recorded · Cobbe et al. 2021, MIT license

# per run: seed 0 · checker last_number · problems 250
# settings: name=Qwen/Qwen2.5-0.5B-Instruct revision=7ae557604adf67be50417f59c2c2f167def9a775 dtype=bfloat16 temperature=0.7 top_p=0.8 top_k=20 repetition_penalty=1.1 max_new_tokens=512 split=test subset_seed=2026
#   plain: prompt=plain
#   step_by_step: prompt=step_by_step
#   worked_examples: prompt=worked_examples
# command (as given) · results/r010_worked_s0/config.yaml:1 records this launch
python solve.py --prompt worked_examples --seed 0 --out results/r010_worked_s0

plain, seed 0, last-number, problems 250

runin e1 · produced 2026-10-14 · recorded 2026-10-23

note: results/r012_plain_s0 · not confirmed

Counts toward 0 rules

No recorded rule reads it.

Results

Accuracy, plain: 43.20

Source details

command (as given)python solve.py --prompt plain --seed 0 --out results/r012_plain_s0how rb knowsresults/r012_plain_s0/config.yaml:1 records this launch

produced: 2026-10-14 01:38 UTC (as reported) · recorded 2026-10-23 22:58 UTC
inputs: read: qwen25-05b-instruct · present when recorded
inputs: read: gsm8k-test · present when recorded · sha256 3730d312f6e3
repository: when recorded, not when run: commit 6adf8a5006 · tree dirty
config: 14 matched
basis: file · spec not locked
environment: python 3.12.15 · rb 0.6.0 · pydantic 2.14.1 (where it was recorded: the run was made before it was recorded)
results: results/r012_plain_s0/replies.jsonl · sha256 c721c50cd173 · 256861 bytes
results: results/r012_plain_s0/metrics.json · sha256 3a10486fedde · 152 bytes
note: results/r012_plain_s0 · not confirmed
type: recorded afterwards
results as rb gives them: plain: accuracy=43.2

Reproduce this run · 1 of 5 parts ready

to run again: ✓ 1 ok · ◐ 1 partial · ✗ 2 missing · ○ 1 unknown

the whole recipe for e1

missing · commit 9ba0f86 · not in this repository: 9ba0f86 (get the code repository to rerun)

unknown · the run was made before it was recorded, in an environment rb did not see

ok · qwen25-05b-instruct: present

partial · gsm8k-test: reconstructible

missing · neither of the 2 files the runs named is here unchanged; not here: results/r012_plain_s0/replies.jsonl, results/r012_plain_s0/metrics.json

# Reproduce a run of e1 · plain · seed 0 · checker last_number · problems 250
# Recipe, from what the record holds. rb did not run these commands.
# code: missing · commit 9ba0f86 · not in this repository: 9ba0f86 (get the code repository to rerun)
# environment: unknown · the run was made before it was recorded, in an environment rb did not see
# checkpoint: ok · qwen25-05b-instruct: present
# dataset: partial · gsm8k-test: reconstructible
# files: missing · neither of the 2 files the runs named is here unchanged; not here: results/r012_plain_s0/replies.jsonl, results/r012_plain_s0/metrics.json

# checkpoint · qwen25-05b-instruct: no recipe recorded · configs/model.yaml; Apache 2.0

# dataset · gsm8k-test: no recipe recorded · Cobbe et al. 2021, MIT license

# per run: seed 0 · checker last_number · problems 250
# settings: name=Qwen/Qwen2.5-0.5B-Instruct revision=7ae557604adf67be50417f59c2c2f167def9a775 dtype=bfloat16 temperature=0.7 top_p=0.8 top_k=20 repetition_penalty=1.1 max_new_tokens=512 split=test subset_seed=2026
#   plain: prompt=plain
#   step_by_step: prompt=step_by_step
#   worked_examples: prompt=worked_examples
# command (as given) · results/r012_plain_s0/config.yaml:1 records this launch
python solve.py --prompt plain --seed 0 --out results/r012_plain_s0

step by step, seed 0, last-number, problems 500

runin e1 · produced 2026-10-12 · recorded 2026-10-23

note: results/r011_step_n500_s0 · not confirmed

Counts toward 0 rules

No recorded rule reads it.

Results

Accuracy, step by step: 42.20

Source details

command given later, not observedpython solve.py --prompt step_by_step --seed 0 --out results/r011_step_n500_s0 --problems 500how rb knowsit was made before it was recorded, and neither rb's journal nor a log line written then shows this command made it; logs/agent.log was written 2026-10-16, after the run

produced: 2026-10-12 01:57 UTC (as reported) · recorded 2026-10-23 22:58 UTC
inputs: read: qwen25-05b-instruct · present when recorded
inputs: read: gsm8k-test · present when recorded · sha256 3730d312f6e3
repository: when recorded, not when run: commit 6adf8a5006 · tree dirty
config: 14 matched
basis: file · spec not locked
environment: python 3.12.15 · rb 0.6.0 · pydantic 2.14.1 (where it was recorded: the run was made before it was recorded)
results: results/r011_step_n500_s0/replies.jsonl · sha256 f4ca17e2295a · 551606 bytes
results: results/r011_step_n500_s0/metrics.json · sha256 248def1e7a68 · 160 bytes
note: results/r011_step_n500_s0 · not confirmed
type: recorded afterwards
results as rb gives them: step_by_step: accuracy=42.2

Reproduce this run · 1 of 5 parts ready

to run again: ✓ 1 ok · ◐ 1 partial · ✗ 2 missing · ○ 1 unknown

the whole recipe for e1

missing · commit 738002e · not in this repository: 738002e (get the code repository to rerun)

unknown · the run was made before it was recorded, in an environment rb did not see

ok · qwen25-05b-instruct: present

partial · gsm8k-test: reconstructible

missing · neither of the 2 files the runs named is here unchanged; not here: results/r011_step_n500_s0/replies.jsonl, results/r011_step_n500_s0/metrics.json

# Reproduce a run of e1 · step_by_step · seed 0 · checker last_number · problems 500
# Recipe, from what the record holds. rb did not run these commands.
# code: missing · commit 738002e · not in this repository: 738002e (get the code repository to rerun)
# environment: unknown · the run was made before it was recorded, in an environment rb did not see
# checkpoint: ok · qwen25-05b-instruct: present
# dataset: partial · gsm8k-test: reconstructible
# files: missing · neither of the 2 files the runs named is here unchanged; not here: results/r011_step_n500_s0/replies.jsonl, results/r011_step_n500_s0/metrics.json

# checkpoint · qwen25-05b-instruct: no recipe recorded · configs/model.yaml; Apache 2.0

# dataset · gsm8k-test: no recipe recorded · Cobbe et al. 2021, MIT license

# per run: seed 0 · checker last_number · problems 500
# settings: name=Qwen/Qwen2.5-0.5B-Instruct revision=7ae557604adf67be50417f59c2c2f167def9a775 dtype=bfloat16 temperature=0.7 top_p=0.8 top_k=20 repetition_penalty=1.1 max_new_tokens=512 split=test subset_seed=2026
#   plain: prompt=plain
#   step_by_step: prompt=step_by_step
#   worked_examples: prompt=worked_examples
# command given later, not observed: python solve.py --prompt step_by_step --seed 0 --out results/r011_step_n500_s0 --problems 500

Qwen2.5-0.5B-Instruct 7ae557604adf67be50417f59c2c2f167def9a775

resourcecheckpoint · present · 9 runs read itqwen25-05b-instruct

What depends on this

read by 9 runs; 5 rules read them, under q1, q2

5 rules, by question

q1 · 4 rulesc1 c2 c3 c-ref
q2 · 1 rulec4

9 runs, by experiment

e1 · 9 runsplain, seed 0, answer-tag, problems 250; plain, seed 1, answer-tag, problems 250; plain, seed 2, answer-tag, problems 250; step by step, seed 0, answer-tag, problems 250; step by step, seed 1, answer-tag, problems 250; step by step, seed 2, answer-tag, problems 250; worked examples, seed 0, last-number, problems 250; plain, seed 0, last-number, problems 250; step by step, seed 0, last-number, problems 500

Source details

identity: version 7ae557604adf67be50417f59c2c2f167def9a775 (not exact)
where: hf:Qwen/Qwen2.5-0.5B-Instruct · recorded · local:/srv/rb/demo/cache/qwen2.5-0.5b-instruct · here

GSM8K test set

resourcedataset · reconstructible · 9 runs read itgsm8k-test

What depends on this

read by 9 runs; 5 rules read them, under q1, q2

5 rules, by question

q1 · 4 rulesc1 c2 c3 c-ref
q2 · 1 rulec4

9 runs, by experiment

e1 · 9 runsplain, seed 0, answer-tag, problems 250; plain, seed 1, answer-tag, problems 250; plain, seed 2, answer-tag, problems 250; step by step, seed 0, answer-tag, problems 250; step by step, seed 1, answer-tag, problems 250; step by step, seed 2, answer-tag, problems 250; worked examples, seed 0, last-number, problems 250; plain, seed 0, last-number, problems 250; step by step, seed 0, last-number, problems 500

Source details

how to get it again

Cobbe et al. 2021, MIT license

identity: sha256 3730d312f6e3
where: local:data/gsm8k/test.jsonl · gone · local:data/gsm8k/test.jsonl · gone

Open callsYour call › q1 › c1

Open callYour callStill open · c1

Step by step scores at least 5 points above the plain prompt

Not met: step by step is 3.60 points higher than the plain baseline (+0.62 to +6.58), short of the 5 points the rule asks for.

From 3 seeds (6 runs) · the whole finding

Nothing is recorded on it yet.

Record a call

Accept
```
rb decide c1 accept --why '‹why›'
```
The record will say: accepted ‹day›: ‹why›
Reject
```
rb decide c1 reject --why '‹why›'
```
The record will say: rejected ‹day›: ‹why›
Investigate
```
rb decide c1 investigate --why '‹why›'
```
The record will say: investigating since ‹day›: ‹why›

It is run with rb in a terminal; rb records the call and the reason.

Run it in your terminal; rb records the call and your reason.

Open callsYour call › q2 › c4

Open callYour callStill open · c4

The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from

Not reproduced: worked examples is 20.80 points higher than step by step (no interval yet), not within 0.05 points of the 22.9 points.

Too little evidence

From 1 seed (2 runs · 2 runs not counted) · the whole finding

Nothing is recorded on it yet.

Source details

cited: logs/agent.log:720 Worked examples against step by step: +22.9 points.

Record a call

Accept
```
rb decide c4 accept --why '‹why›'
```
The record will say: accepted ‹day›: ‹why›
Reject
```
rb decide c4 reject --why '‹why›'
```
The record will say: rejected ‹day›: ‹why›
Investigate
```
rb decide c4 investigate --why '‹why›'
```
The record will say: investigating since ‹day›: ‹why›

It is run with rb in a terminal; rb records the call and the reason.

Run it in your terminal; rb records the call and your reason.

Open callsYour call › q1 › c-ref

Open callYour callStill open · c-ref

The plain prompt reaches the published 49.6 on GSM8K

Not reproduced: plain baseline: 13.07% (11.00% to 15.14%), not within 0.05% of the cited 49.6%.

From 3 runs · the whole finding

Nothing is recorded on it yet.

Source details

cited: README.md:12 Qwen2.5-0.5B-Instruct scores 49.6 on GSM8K

Record a call

Accept
```
rb decide c-ref accept --why '‹why›'
```
The record will say: accepted ‹day›: ‹why›
Reject
```
rb decide c-ref reject --why '‹why›'
```
The record will say: rejected ‹day›: ‹why›
Investigate
```
rb decide c-ref investigate --why '‹why›'
```
The record will say: investigating since ‹day›: ‹why›

It is run with rb in a terminal; rb records the call and the reason.

Run it in your terminal; rb records the call and your reason.

plain

variantbaselinee1

plain

cell	Accuracy
answer-tag, problems 250	13.07% n=3
last-number, problems 250	43.20% n=1

the runs behind each mean · 9 runs

plain, seed 0, answer-tag, problems 250
plain, seed 1, answer-tag, problems 250
plain, seed 2, answer-tag, problems 250
step by step, seed 0, answer-tag, problems 250
step by step, seed 1, answer-tag, problems 250
step by step, seed 2, answer-tag, problems 250
worked examples, seed 0, last-number, problems 250
plain, seed 0, last-number, problems 250
step by step, seed 0, last-number, problems 500

step by step

variantcandidatee1

step-by-step

cell	Accuracy
answer-tag, problems 250	16.67% n=3
last-number, problems 500	42.20% n=1

the runs behind each mean · 9 runs

plain, seed 0, answer-tag, problems 250
plain, seed 1, answer-tag, problems 250
plain, seed 2, answer-tag, problems 250
step by step, seed 0, answer-tag, problems 250
step by step, seed 1, answer-tag, problems 250
step by step, seed 2, answer-tag, problems 250
worked examples, seed 0, last-number, problems 250
plain, seed 0, last-number, problems 250
step by step, seed 0, last-number, problems 500

worked examples

variantcandidatee1

worked-examples

cell	Accuracy
last-number, problems 250	39.60% n=1

the runs behind each mean · 9 runs

plain, seed 0, answer-tag, problems 250
plain, seed 1, answer-tag, problems 250
plain, seed 2, answer-tag, problems 250
step by step, seed 0, answer-tag, problems 250
step by step, seed 1, answer-tag, problems 250
step by step, seed 2, answer-tag, problems 250
worked examples, seed 0, last-number, problems 250
plain, seed 0, last-number, problems 250
step by step, seed 0, last-number, problems 500

notes.md · line 9

cited file

The cited lines

8  
9▸ 2026-10-16: Next: write up the prompt comparison for Friday's reading group.

read at commit 330c6ff

Cited by

A note on the project · the project's notes

Source details

file: notes.md
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: 9fe3d835f00157601a2def6538914e5da088b78ac108927380551cb965464f5d

README.md · line 3

cited file

The cited lines

2  
3▸ Does telling a small AI model to think step by step make it solve more grade-school math problems?
4

read at commit 330c6ff

Cited by

Experiment Prompt style on 250 GSM8K test problems: plain, step by step, worked examples e1

Source details

file: README.md
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: 7029e0be441faa31bb434075899b69a46f0c607ff2e05ff232b06a971ad19e63

README.md · line 7

cited file

The cited lines

6  
7▸ - Do worked examples help more than "think step by step"?
8  - How many problems do we need before the difference is real?

read at commit 330c6ff

Cited by

Experiment Prompt style on 250 GSM8K test problems: plain, step by step, worked examples e1

Source details

file: README.md
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: 7029e0be441faa31bb434075899b69a46f0c607ff2e05ff232b06a971ad19e63

logs/agent.log · line 719

cited file

The citation

Not quoted on a shared page · rb open shows these lines on the record's own machine.

Cited by

Rule The claim that step by step is 3.6 points above plain on the seed averages is checked here c2

Source details

file: logs/agent.log
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156

logs/agent.log · line 557

cited file

The citation

Not quoted on a shared page · rb open shows these lines on the record's own machine.

Cited by

Rule The claim that the step-by-step gain over plain holds on every seed is checked here c3

Source details

file: logs/agent.log
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156

logs/agent.log · line 720

cited file

The citation

Not quoted on a shared page · rb open shows these lines on the record's own machine.

Cited by

Rule The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from c4

Source details

file: logs/agent.log
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156

README.md · line 12

cited file

The cited lines

11  
12▸ Reference: Qwen2.5-0.5B-Instruct scores 49.6 on GSM8K (Qwen Team, "Qwen2.5 Technical Report", arXiv:2412.15115, table 10, the instruction-tuned models, on the full test set of 1319 problems).
13

read at commit 330c6ff

Cited by

Rule The plain prompt reaches the published 49.6 on GSM8K c-ref

Source details

file: README.md
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: 7029e0be441faa31bb434075899b69a46f0c607ff2e05ff232b06a971ad19e63

logs/agent.log · lines 581-584

cited file

The citation

Not quoted on a shared page · rb open shows these lines on the record's own machine.

Cited by

A note on e1

Source details

file: logs/agent.log
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156

logs/agent.log · line 669

cited file

The citation

Not quoted on a shared page · rb open shows these lines on the record's own machine.

Cited by

A note on e1

Source details

file: logs/agent.log
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156

results/r004_plain_s0/config.yaml · line 1

cited file

The cited lines

1▸ command: python solve.py --prompt plain --seed 0 --out results/r004_plain_s0
2  commit: 32c91d6f13af48510004050b7d83eb32d7ec2030

read at commit 330c6ff

Cited by

Only rb's own lines cite it here, such as the history or a recipe.

Source details

file: results/r004_plain_s0/config.yaml
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: a0cc679da96d571f6e6f0611625771ad866f104f5b7b81f8c21965a995da4052

results/r005_plain_s1/config.yaml · line 1

cited file

The cited lines

1▸ command: python solve.py --prompt plain --seed 1 --out results/r005_plain_s1
2  commit: 32c91d6f13af48510004050b7d83eb32d7ec2030

read at commit 330c6ff

Cited by

Only rb's own lines cite it here, such as the history or a recipe.

Source details

file: results/r005_plain_s1/config.yaml
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: 3355a2f8dcbe609ee212654e5b506bd739f119c99a1d716efb4b0fd5d8e887ea

results/r006_plain_s2/config.yaml · line 1

cited file

The cited lines

1▸ command: python solve.py --prompt plain --seed 2 --out results/r006_plain_s2
2  commit: 32c91d6f13af48510004050b7d83eb32d7ec2030

read at commit 330c6ff

Cited by

Only rb's own lines cite it here, such as the history or a recipe.

Source details

file: results/r006_plain_s2/config.yaml
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: c89e51d5968879e36aafe688b44014df32d8c43322a1c6778bc009138648747f

results/r007_step_s0/config.yaml · line 1

cited file

The cited lines

1▸ command: python solve.py --prompt step_by_step --seed 0 --out results/r007_step_s0
2  commit: 32c91d6f13af48510004050b7d83eb32d7ec2030

read at commit 330c6ff

Cited by

Only rb's own lines cite it here, such as the history or a recipe.

Source details

file: results/r007_step_s0/config.yaml
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: e954f4eaf2d43c7c6802fcf172d2b88c0682c96393238e3f38bc68d90935b718

results/r008_step_s1/config.yaml · line 1

cited file

The cited lines

1▸ command: python solve.py --prompt step_by_step --seed 1 --out results/r008_step_s1
2  commit: 32c91d6f13af48510004050b7d83eb32d7ec2030

read at commit 330c6ff

Cited by

Only rb's own lines cite it here, such as the history or a recipe.

Source details

file: results/r008_step_s1/config.yaml
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: 45294fd56d8b8cd4d5ef7aa77fdc3517ecadd5ea461f9d9e2ca6c431a12ef916

results/r009_step_s2/config.yaml · line 1

cited file

The cited lines

1▸ command: python solve.py --prompt step_by_step --seed 2 --out results/r009_step_s2
2  commit: 32c91d6f13af48510004050b7d83eb32d7ec2030

read at commit 330c6ff

Cited by

Only rb's own lines cite it here, such as the history or a recipe.

Source details

file: results/r009_step_s2/config.yaml
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: 32bbacd93509437ec63cd1f703c7a1948b55e6b3ba0c1567c26a017d0dc162ec

results/r010_worked_s0/config.yaml · line 1

cited file

The cited lines

1▸ command: python solve.py --prompt worked_examples --seed 0 --out results/r010_worked_s0
2  commit: 7265bd171c0966de034bcf2b6281935bc43f781a

read at commit 330c6ff

Cited by

Only rb's own lines cite it here, such as the history or a recipe.

Source details

file: results/r010_worked_s0/config.yaml
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: 9d357d5e71d0e11f0a117f9a2293ea731ae6456ef25dbca91999b251f7041b84

results/r012_plain_s0/config.yaml · line 1

cited file

The cited lines

1▸ command: python solve.py --prompt plain --seed 0 --out results/r012_plain_s0
2  commit: 9ba0f86037dc66c0dbee79728b3d0f18e304773d

read at commit 330c6ff

Cited by

Only rb's own lines cite it here, such as the history or a recipe.

Source details

file: results/r012_plain_s0/config.yaml
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: 9850b8180856b2a6e3f9c85d76861b7c7019040d8a9e132cc0c655d092fa12a2

notes.md · line 7

cited file

The cited lines

6  
7▸ 2026-10-13: Our rule: step by step helps if it scores at least 5 points above the plain prompt.
8

read at commit 330c6ff

Cited by

Only rb's own lines cite it here, such as the history or a recipe.

Source details

file: notes.md
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: 9fe3d835f00157601a2def6538914e5da088b78ac108927380551cb965464f5d

logs/agent.log · lines 582-585

cited file

The citation

Not quoted on a shared page · rb open shows these lines on the record's own machine.

Cited by

Only rb's own lines cite it here, such as the history or a recipe.

Source details

file: logs/agent.log
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156

logs/agent.log · lines 653-655

cited file

The citation

Not quoted on a shared page · rb open shows these lines on the record's own machine.

Cited by

Only rb's own lines cite it here, such as the history or a recipe.

Source details

file: logs/agent.log
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156

README.md · line 9

cited file

The cited lines

8  - How many problems do we need before the difference is real?
9▸ 
10  The model is Qwen2.5-0.5B-Instruct (Apache 2.0), run on CPU. The problems are GSM8K's test set (Cobbe et al. 2021, MIT license).

read at commit 330c6ff

Cited by

Only rb's own lines cite it here, such as the history or a recipe.

Source details

file: README.md
commit: 330c6ffaf0c2edadf246ef0967bc98d232cb3156
sha256: 7029e0be441faa31bb434075899b69a46f0c607ff2e05ff232b06a971ad19e63

Also open in the record (3)

Rules written after their results (1).

Step by step scores at least 5 points above the plain prompt c1

details

Rules written after their results or data were seen

Questions with nothing filed under them (1)

q3 is open and nothing is filed under it: How many problems do we need before the difference is real? q3

Runs whose command was given later, not seen (1).

1 run was recorded after it ran; nothing rb saw shows the command that made it step_by_step · seed 0 · checker last_number · problems 500

details

Commands given after the runs

About this record

Research record think-step-by-step-a-small-model-on-grade-school-math · rb 0.6.1

Kept with rb since 23 Oct 2026. All 9 runs were recorded afterwards, not as they happened.

What the words on this page mean

open (question): no call has closed it. A call on the question closes it: answered (accept), dropped (reject), parked, stopped, pivoted or shipped; investigate and continue keep it open. A finding or an interpretation does not make that call.
note (on a question): recorded words on what meeting a rule would mean for the question. A note counts only while its rule is met; rb computes the finding and never asserts the inverse.
Computed comparison: rb's reading of a declared reversal between two rules: both numbers and intervals stay visible, and it makes no call on the question.
headline: rb's one line on a question's evidence, the first that applies: No rule tests this question yet (rules are filed here but none is linked to it), Evidence needs review (an integrity problem), No evidence recorded yet, Measured differences; uncertainty remains (an interval includes 0), Evidence includes point estimates only (a reading has no interval), else Evidence available for review.
rule written before the results: the rule was declared before the evidence it reads existed: a test planned in advance.
rule written after the results: the rule or its question link was declared after the evidence existed or after the data was seen: never read as a test planned in advance.
no interval: one estimate with no interval, so how much it could vary is unknown. rb gives one from 3 units, and none across fixed design cells.
Uncertainty remains: unresolved: the interval includes 0.
Fixed design cells: the reading spans fixed design choices: each cell is read on its own, with how many meet and their range, and no sampling interval across cells.
Reversal observed; repeatability uncertain: the two sides of a declared reversal measured opposite directions, but not both with intervals clear of 0.
rb's interval: 95%, from 3 units: a Student t interval when the arms pair up by their unit, Welch (unpaired) when they never share one.
straddles: the interval crosses the rule's threshold: the rule word holds on the estimate, not across the whole interval.
rule: a threshold or comparison declared for a question; commands call it a criterion.
can't tell yet: there are results, but they don't settle the rule: it needs an interval, more units, or the metric's direction.
result: gain or loss (the interval clear of 0, read in the metric's direction), unresolved (the interval includes 0), equivalent (inside a declared margin), no interval (one estimate, no interval yet), measured (a level), higher or lower (the metric has no direction), not tested.
timing: declared before the evidence (before its runs, when only its baseline's runs came first: timing is read on the subject's runs); after the evidence existed; after the data was seen; order unknown.
units: the independent units a reading rests on. A repeat checks a run and adds no unit; a reanalysis adds none.
checks: problems with what a finding depends on (a stale source, an undeclared difference, a record edited outside rb). ok when there are none. Not the repository's own checks.
estimate meets the rule: met on the estimate, not across the whole interval; never drawn green.
verified: the named file, run or database row states this setting's value. It does not mean a run used it. provisional: given, not checked.
open (experiment): planned: an experiment nothing tests yet. open: evidence and no call. accepted, rejected, investigating, parked, stopped, pivoted, shipped: what the latest call on it made it.
locked: the spec and rules are locked; a later change is recorded as an amendment.
not locked: rules and settings can still change; a freeze locks them.
not tested: no results for this rule yet: no recorded run reads this comparison
measured: one arm on its own, with its interval, so neither better nor worse
command (as given): --command records what produced the numbers. rb did not run it.
not confirmed: it waits to be confirmed with rb in a terminal; until then it reads as a note, never as a call made or an answer.
reconstructible: present, changed, remote, reconstructible or lost: reconstructible means no copy is recorded, but how to get it again is.
gain: the interval is clear of 0 on the better side.
loss: the interval is clear of 0 on the worse side.
unresolved: the interval includes 0, which is not evidence of no difference.
equivalent: the interval lies inside the margin an equivalence rule declares.
reproduced: a rule from a cited source or from the record this one was cloned from, met by the runs recorded here.
not reproduced: a rule from a cited source or from the record this one was cloned from, not met by the runs recorded here.
paired difference: each unit's subject minus its reference, averaged over the units.
cited: a number someone else stated (a paper's table, say), checked against the source's own words.
margin: how far the value the rule judges (the estimate, an interval bound or the worst unit) sits from its threshold: positive inside it, negative outside. Not a declared equivalence or noninferiority margin.
question link: the question a rule's finding bears on, with the recorded words on what meeting the rule would mean for it.
identity: how exactly a resource is named: exact (the hash of its bytes), version (a release) or name only.
amended: changed after the rules and settings were locked; the change, its reason and what it changed from are kept.

Think step by step: a small model on grade-school math

How far to trust it⚠

Needs judgmentNeeds your judgmentStill open

Experiment designs not confirmed (1)

Research questions

Recent calls and changes

Reproduce this project

Research

Does telling a small AI model to think step by step make it solve more grade-school math problems?

Its rules (4)

No note on these rules yet.

Open callsYour call (2)

Experiments behind this question

Sub-question

Do worked examples help more than "think step by step"?

Its rules (1)

No note on these rules yet.

Open callsYour call (1)

Experiments behind this question

How many problems do we need before the difference is real?

Experiments behind this question

Rules (5)

Prompt style on 250 GSM8K test problems: plain, step by step, worked examples

Experiments

Prompt style on 250 GSM8K test problems: plain, step by step, worked examples

Comparison

Accuracy (%)

Runs (9)

Its rules (5)

Reproduce

What it takes to run it again

Recipe, from what the record holds

Next step

Runs

Prompt style on 250 GSM8K test problems: plain, step by step, worked examples

Resources

History

Raw log

Open callsYour call (3)

Also open in the record (3) →

Step by step scores at least 5 points above the plain prompt

What rb measured

The rule as written

Runs it rests on 3 seeds from 6 runs

What depends on this

Record a call

Still open

The claim that step by step is 3.6 points above plain on the seed averages is checked here

What rb measured

The rule as written

Runs it rests on 3 seeds from 6 runs

What depends on this

The claim that the step-by-step gain over plain holds on every seed is checked here

What rb measured

The rule as written

Runs it rests on 3 seeds from 6 runs

What depends on this

The claim that worked examples are 22.9 points above step by step is checked here, on the runs it was computed from

What rb measured

The rule as written

Runs it rests on 1 seed from 2 runs · 2 runs not counted

What depends on this

Record a call

Still open

The plain prompt reaches the published 49.6 on GSM8K

What rb measured

The rule as written

Runs it rests on 3 runs

What depends on this

Record a call

Still open

plain, seed 0, answer-tag, problems 250

Counts toward 4 rules

Results

plain, seed 1, answer-tag, problems 250

Counts toward 4 rules

Results

plain, seed 2, answer-tag, problems 250

Counts toward 4 rules

Results

How far to trust it