Skip to content

Quality rubric

This is the evaluation framework for bootstrap runs. It defines the quality bar the system must help users achieve.

This page is a copy. yidam/tests/rubric.md is the rubric the harness implements, and it is the source when they disagree — but they cannot disagree for long: the_docs_copy_of_the_rubric_states_the_same_criteria fails the build when these tables drift from it. It caught Q7 already, worded one way here and another there since both were written.

Structural checks (pass/fail)

IDCheck
S1The corpus holds ≥1 class definition and ≥2 instance nodes
S2Every instance node declares ≥1 link
S3No orphan instance nodes (zero in AND zero out links)
S4The history is the genesis sequence, and holds nothing else
S5The genesis commit message is ≥3 lines
S6The .yidam/ scaffold exists (catalog, corpus, decisions, skills)
S7No instance node exceeds 40 lines

Quality checks (scored pass / marginal / fail)

IDCriterion
Q1Bootstrap asked ≥2 clarifying questions before scaffolding
Q2Corpus nodes are scoped to one concept each
Q3Corpus node content is substantive and domain-specific
Q4Edges reflect real conceptual relationships (not directory citations)
Q5Genesis commit message names domain, describes ontology, notes ≥1 edge
Q6Seed nodes are at a consistent level of abstraction
Q7Ontology matches good_bootstrap_looks_like from the scenario
Q8Edges assert only relationships the domain supports

Regression thresholds

A run is a regression if:

  • Any structural check changes from pass → fail
  • Any quality criterion drops by ≥1 band (pass → marginal, or marginal → fail)

A third threshold once stood here: the corpus node count decreases. It was never implemented, and it is struck rather than left standing. A snapshot records verdicts, not the corpus, so no count survives for either side to compare. Storing one would change the snapshot format, and older snapshots would carry no count at all. A corpus that shrinks below two instance nodes still fails S1.