The record's format

This page describes the format of the files in .rb/, where its JSON Schema lives, and rb’s vocabulary of record kinds.

Every file in .rb/ opens with its format, rb: 1 or rb: 2, and then its kind, so a file travels and validates on its own. Both formats are one JSON Schema (draft 2020-12), served at its $id, rabbitbrain.ai/schema/rb-v2.schema.json, and kept in the repository at schema/rb-v2.schema.json. Format 1’s own schema, rb-v1.schema.json, stays as it was.

From the command line:

# the whole format, for any JSON Schema validator
rb schema all > rb-v2.schema.json
# one kind: project, question, experiment, metric, criterion, run, analysis, resource or decision
rb schema run
# a finding's shape: computed on every read, never stored
rb schema finding
# what every command prints with --json
rb schema envelope
# read one record file and check it against the format
rb show .rb/experiments/int8-ptq/runs/<id>.yaml

Rabbit Brain YAML

The files are YAML kept to a strict subset, so every YAML parser reads a record the same way: mappings, lists, text, numbers, true, false, null and comments, and nothing else. A plain value that YAML 1.1 and YAML 1.2 read differently is refused, with the spelling to write instead: yes, no, on, off, dates, numbers with leading zeros, and 1e-4 without a dot. rb fmt rewrites files in their canonical form, comments kept.

A comment or a reformat is not an edit: rb compares a record’s content, never its text, so you can annotate the files.

Format 1 and format 2

Format 1 is fixed: a later rb that adds a field writes that file in format 2, and an older rb says to upgrade rather than read part of it.

Format 2 adds four kinds of rule to a criterion and nothing else: a count (over: count, with count and of), a best of N (over: best, with of), a relative change (measure: relative) and a cited difference of means (measure: means). Criteria says how each reads.

• rb 0.8.0 writes a criterion’s file in format 2 only while it holds one of the four, and writes every other file in format 1, as rb 0.7 writes it. It reads both.
• A file that says rb: 1 and holds one of the four is refused (E_STATE_CORRUPT): format 1 never held one.
• A retracted rule keeps the rule it was, so its file stays in format 2.
• rb export --with-record writes record_format: 2 in the bundle’s manifest when a criterion’s file is in format 2, else 1.

rb 0.7.0 and 0.7.1 read format 1 only. They read a record with no format-2 file, and write to it as before. Where a file is format 2, rb doctor names it (“is format 2; this rb reads format 1: it was written by a newer rb”), rb status counts such files, and both say to upgrade (pip install --upgrade rabbit-brain). They read every other file and fail every --fail-on gate. They write nothing to the record, and refuse to clone a bundle that holds such a file. Upgrade everyone who shares a record together.

The vocabulary

rb schema ir prints every kind of record, what it means, the command that writes it, and the source rb requires:

$ rb schema ir
rb's vocabulary: every record kind, what it means, the command that writes it, and the sources rb requires.

question     what the work is trying to find out
             a tree: a sub-question names its parent, and an experiment may be filed under several; where it stands is read from what is below it, never a child's negative as its own
             write: rb question add "<what are you trying to find out?>" --parent <q>
             provenance: wording as asked
experiment   what is tested
             its purpose or hypothesis, one baseline and variants with roles, varies naming the intended differences; planned until its first run; an undeclared difference between variants is a confound
             write: rb experiment add "<title>" --hypothesis "<h>" --baseline <b> --candidate <c> --varies <s>
             provenance: declared, then checked against its settings
variant      one arm of an experiment
             baseline, candidate, control or ablation; its own settings override the shared ones
             write: rb variant add <exp> <name> --role <r>
             provenance: role declared
setting      what the experiment changes or depends on
             a value with where it comes from; population.* declares the counting structure and never blocks
             write: rb spec set <exp> <name> <value> --source file#key
             provenance: REQUIRED: a source rb can read, or --unknown, or a vouch (rb decide <exp>/<name> accept)
criterion    what would count on one comparison
             a rule over one comparison (subject, reference, metric, estimand), with a role (primary, secondary, exploratory, gate), declared by someone at a time before the evidence it reads; explicit question links may keep the author's verbatim interpretation when met, never an answer inferred by rb
             write: rb criterion add "<c>" -e <exp> -q <question> --metric <m> --subject <a> --reference <b> --better
             provenance: declared by whoever wrote it, with the user's words when an agent gives them; question links snapshot the question and retain author, time, reason, session and revisions; REQUIRED for cited numbers: --source --quote --locator, checked against the file
run          one execution, and what it produced
             results with a receipt: actor, chat session, the agent (its name, version, provider and model), time, repository state, command, environment, the resources read and made, when its output existed; the same output recorded twice is one run; a repeat checks a run and adds no unit; a wrong run is retracted (whole, or --scope for named measurements), never deleted
             write: rb run add <exp> --from <file> --set seed=N --input <resource> --config <resolved>
             provenance: REQUIRED: the receipt is taken when it is recorded; --from and --config are read and hashed; typed numbers are marked typed
analysis     a computation over saved outputs
             the comparisons it produced, each with its estimate and interval, the unit it resampled, and where it came from; a reanalysis is an analysis, never a run; a later one supersedes it with a reason, and its readings stay as history
             write: rb analysis add <exp> --compare "a/b:m=est[lo,hi]" --method "<m>" --unit "<u>" --units N --source <file>
             provenance: REQUIRED: computed from an output file rb hashed, or reported from a document rb checked, or marked typed
comparison   subject against reference on a metric
             the unit of state: an estimate, its interval when there is one, what it resampled; changing the metric, the reference or the estimand is a different comparison
             write: read by a criterion; produced by an analysis or by rb from the runs
             provenance: an analysis, or rb's reading of the runs
result       a number a run reports
             named <variant>.<metric>, change.<metric> (candidate minus baseline) or plain; compared across variants in rb compare
             write: NAME=VALUE on rb run add, or --from <file>
             provenance: the run's receipt
resource     a dataset, checkpoint, code, environment or artifact
             identity (name, version, hash), locations (any provider), and how to get it again; availability is computed: present, changed, remote, reconstructible or lost; a run records what it found when it read or made one
             write: rb resource add <id> --type dataset --name "<n>" --url <u>
             provenance: declared; a local file is hashed when recorded
metric       what is measured, by name
             unit, direction, aliases; a result naming an alias is read through it
             write: rb metric add <m> --unit <u> --minimize
             provenance: declared
finding      a criterion read on axes, computed
             result, rule, timing, units, integrity, review and history, computed from the runs and analyses on every read, never stored; no single word stands for it
             write: read it: rb show <criterion>
             provenance: derived
decision     a call, with what it was made on
             accept, reject or investigate a finding, an experiment, a question or a setting; continue, park, stop, pivot or ship a line of work; read against the readings it was made on, with when to revisit; one recorded in an agent session is proposed until the user confirms it (rb confirm), and a past one (--past) is Retrospective
             write: rb decide <subject> <action> -m "..." --revisit "..."
             provenance: REQUIRED: who made it and why, with the chat session when known and the user's words verbatim when an agent acts on them; for a past call, when it was made and the document that states it
note         words about an experiment or the project
             an assumption, a caveat, context; kept verbatim with who wrote it; changes no finding
             write: rb note add "<text>" -e <exp>
             provenance: text as written
relation     how one experiment stands to another
             reproduces, extends, follows_from or compares_with an experiment here, a uid, or a published manifest; structure, not evidence
             write: rb experiment relate <exp> reproduces <target>
             provenance: declared