RFC-0043 — A figure published twice, computed once — an inline REGEN block and the count generator
- Status: Implemented
- Commands:
count - Track: I32
- Relates to:
- RFC-0018 (the query surface, which this adopts verbatim as the generator's argument rather than coining a second way to name a set)
- RFC-0035 (class extent — a
coverage:declaration is a claim about how many there should be; this is a claim about how many there are) - RFC-0001 (the report contract —
countis a report with a--format jsonshape like the rest)
- Versioning layers touched: template
v0.8.0→v0.9.0(the REGEN marker format gains a form, which VERSIONING.md's Layer 1 table calls major, and §"Migration" argues it is right to) / SDK parity (scan_markersandupdate_regenin all three SDKs, thegraph.dfymodel, nine new fixtures) /yidam-coreminor (a new public field onMarker::Regen's sibling type; no existing signature changes) / CLI surface (one new command, one new generator). No MCP contract change, no bootstrap-protocol change, no migration of corpus content. - Downstream reference case:
matt-huffman'scrates/corpus/tests/roadmap.rsanddemi-moore'scrates/demi-feed/tests/readme.rs— two repositories that built the same gate independently, and each of whose header states the limit this RFC removes.
Summary
A number that states a fact about the corpus is published in two places: in the graph, where it
is computed, and in a sentence, where it is remembered. The remembered copy drifts. Three
derived repositories have now built a gate against it, none of the gates can reach a number in
the middle of a sentence, and two of them say so in their own headers. This RFC adds
<!-- REGEN: yidam count <query> -->44<!-- /REGEN --> — an RFC-0018 query, a number, and the
existing regen --check contract — so that the figure in the sentence is the figure in the
graph by construction rather than by assertion.
Getting there costs a marker-format change, because the form the sentence needs is inline and the marker contract has never had one. It turns out that writing one today is not merely unsupported: it is silently destructive, and it is destructive because the two functions that read a REGEN block disagree about what one is.
Problem
A figure published twice drifts, and three repositories have built the same gate
demi-moore's crates/demi-feed/tests/readme.rs opens with the inventory:
The front page of the domain computer said "twelve crates" against fourteen, "the seven connectors" against nine, and "13 invariant checks" against thirty-one — and the two crates it had never heard of were missing from its table as well. Nothing was careless; there was simply no gate.
matt-huffman's crates/corpus/tests/roadmap.rs opens with the same finding and adds the
measurement of what its own gate could not see:
This file gates an enumerated list of figures: the headline counts, the position row, the tenures row, the resolution heading, the crates, the links. It does not and cannot gate an arbitrary number in a narrative section, because nothing tells it which numbers those are. So when
3,104went stale to3,111in the roadmap's committee-rosters section, this gate could not have caught it — it did not fail, it was never pointed at it.
Its conclusion is this RFC's summary, written by the repository that needed it: "The general fix is not more assertions here; it is that a figure worth publishing twice should be computed once."
#1071 names a third, ohio-136th-assembly-hb-96, with a fourth symptom — prose saying "43
districts" against a graph holding 44.
The gate a repository can build is pointed by hand, and prose is where it cannot point
demi-moore's header names the gap in yidam's own terms:
regen-checkcannot cover this: it compares REGEN blocks against what the yidam CLI regenerates, and these numbers sit in hand-written prose above the block.
That is exactly right, and it is a fact about the marker format rather than about regen. A
REGEN block is line-shaped. Its open tag must begin a line, its close tag must be the whole of
another, and update_regen wrote a body
bracketed by newlines. A block can therefore hold a table row, a list, a paragraph or a
whole section — and cannot hold the word 44 between has and districts.
Writing the form anyway is destructive, and silent
This is the part that was not known when #1071 was filed. Run at 1dc8063, against
Ohio has <!-- REGEN: yidam count district -->44<!-- /REGEN --> districts.:
-
scan_markersreturns no marker at all. Its open-tag test is anchored to the start of a line, and this one does not start one:(in
markers.rs, at1dc8063):let Some(rest) = trimmed.strip_prefix("<!-- REGEN:") else { i += 1; continue; }; -
update_regendoes find it — it searches bytes, not lines — and rewrites it into the block form, so the sentence comes back asOhio has <!-- REGEN: … -->\n99\n<!-- /REGEN --> districts.
Move the same block to the start of a line and it gets worse. scan_markers reads the open tag,
fails to find a --> at the end of the line, takes the multi-line branch, and runs forward
looking for a line that ends in --> — which the next block's open tag does. The result is one
marker whose command is yidam count district -->44<!-- /REGEN --> districts., whose content is
the following block's body, and whose extent is reported as well-formed: malformed is
empty. A block was consumed and nothing said so.
unclaimed_in would then report the garbage command as a generator that does not exist, which is
the one visible symptom, and it would report it about the wrong thing.
Two scanners, two answers
The bug above is not two bugs. scan_markers was line-anchored and
update_regen byte-anchored, and nothing held
them to each other. Every existing document happens to satisfy both, so the
disagreement has cost nothing; the first document that does not is the one this RFC wants to
write.
The published contract already promised the thing that would have caught it. Of
parse_markers, yidam/sdks/README.md said: "Return them in document order with
their spans." No span was returned. Marker::Regen carries a command and a content string and
nothing that locates them, which is why a second, independent locator had to exist for the
writer.
The model has that second locator and does not have the first. RegenSpan computes a block's
four offsets by byte search, and
RegenSpanFindsEveryBlock proves it finds
every block that exists:
lemma RegenSpanFindsEveryBlock(text: string, command: string)
ensures HasRegenFor(text, command) <==> RegenSpan(text, command).Some?
That lemma is about the writer's notion of a block. scan_markers' notion is modelled nowhere,
so the proof obligation that would have caught this — the two agree — has never been
statable, let alone discharged.
Since #1097 the model has one locator, as the code does. RegenSpan is now a selection
over RegenScan, so the lemma above says that the selection misses no block the scan reads.
The two agree is the definition, and needs no proof.
count cannot be registered the way the other fourteen are
Every generator in GENERATORS writes a block at a path
it knows, under a command it spells as a literal. count cannot: its blocks are wherever a
document put them, and its command carries the query, so the set of (file, command) pairs is a
property of the tree and not of the source. Three things assume otherwise.
every_generator_in_the_crate_is_listedrequires everyupdate_file_regencall site to name its generator with a"yidam …"literal, so that the guard can see which block each one writes. A computed command has no literal.update_regenmatches its open tag by prefix, soyidam count districtfinds and overwrites<!-- REGEN: yidam count district[party=R] -->. This is #1058's defect with a much larger surface: today two generator names collide only if somebody chose them badly, and with arguments the collision is a property of the query text.update_regenrewrites the first match only. Two blocks in one document asking the same question — "44 districts … of those 44 districts" — leave the second stale forever, and--checkwill not report it, because a block nothing writes is a block nothing records.
Proposal
The block
Ohio has <!-- REGEN: yidam count district -->44<!-- /REGEN --> districts.
The command is yidam count and the rest of the open tag is an RFC-0018 query. The body is the
decimal count of nodes the query matched, and nothing else. yidam count refreshes every such
block in the tracked tree; yidam regen runs it with the other fourteen; yidam regen --check
reports a stale one with the remedy it reports for every other block.
The argument is a query, not a second way to name a set
#1071 proposes class=district. This RFC takes the query language instead, for three reasons.
It already exists and is already typechecked: query::check resolves the class against the
ontology, resolves each predicate's property against the class, and refuses an ordering on a type
that has no order. A class= mini-language would re-answer all of that, worse, in a second
place.
It reaches what the gates actually count. roadmap.rs binds four figures, and exactly one of
them is a class count. The others are the distinct targets of an edge — every tenure some
position is recorded-during — the distinct values of a property ref, and a set difference.
The first is position -recorded-during-> tenure and the count is the distinct terminal nodes,
which is what QueryReport::matched already means. class= can express none of them.
And it makes the open questions a corpus can ask of itself one surface rather than two.
question[status=open], legislation[enacted<2024], district[population?] are all countable
the day this lands, because RFC-0018 and RFC-0040 already defined them.
An inline block is one the document wrote inline, and it stays that way
The form is not a flag on the call. It is read off the block:
A REGEN block whose body contains no newline is written back without newlines, unless the content being written has a newline of its own. Every other block is written back as it is today.
<!-- REGEN: a -->44<!-- /REGEN --> has the body 44 and stays on its line.
<!-- REGEN: a --><!-- /REGEN --> has the body "" and stays on its line, which is what a
hand-authored empty block looks like. <!-- REGEN: a -->\n<!-- /REGEN --> — the shape
update_regen writes when it clears a section — has the body "\n" and keeps the block form,
so the empty-new-content fixture is unchanged and so is ClearingASectionLeavesNoBlankLine.
In the model this is two conditions in
RegenBody, which takes the block's form
as a second argument and returns the content unwrapped only when the block was inline and the
content will fit there.
The unless is not a hedge, and it was not in the first draft of this section. A value holding
a newline cannot be written on one line; writing it there anyway leaves a document whose body no
longer matches the form it declares, and the next run reads that body as block form and
rewrites it — so regen --check reports drift it caused itself. Deciding on the content as well
as on the block is what keeps a second call a no-op. The idempotency clause of UpdateRegenSpec
is what says so: with the newline test removed the model stops verifying, which is how the
condition was found.
The block form has a condition of its own, added by #1137. It is read by line, so it can be
written only where the block has its line to itself. An inline block inside a sentence, given a
value with a newline, fits neither form and is left as it is. Written in block form anyway, its
open tag would become prose, or its close tag would not stand alone and the next run would read
on to a later block's close tag. In the model this is
Holds.
The rule has the property that matters for a document people edit: the author decides, once, by writing the block, and no later regeneration second-guesses them.
One scanner
scan_markers learns the inline form — an open tag whose --> is followed, on the same line,
by the block's own <!-- /REGEN --> closes there, and the scan resumes after it, so a line may
carry more than one block — and it records each block's extent as it goes, which is the reader's
counterpart to RegenSpan and the honest version of what sdks/README.md already promises.
An extent is an offset into the text, and an offset is not a parity contract. Rust counts
bytes, JavaScript counts UTF-16 code units and Python counts code points, so a fixture asserting
open = 41 would assert three different things about one document the moment it held a
character outside ASCII — which is the trap find_reachable's sort order already fell into and
parity/README.md already warns about. The extents are therefore each language's own, in that
language's own unit, and no fixture asserts one. What the fixtures hold is behaviour: the marker
sequence parse_markers returns, and the string update_regen writes.
update_regen is then respecified over those extents rather than over a second search of its
own. Three things follow, and all three are defects closing rather than features opening:
- It matches a command exactly, where it matched a prefix.
yidam count districtno longer findsyidam count district[party=R]. - It rewrites every block with that command, where it rewrote the first.
- It and
scan_markerscan no longer disagree about what a block is, because there is one answer.
(1) and (2) are behaviour changes to a proved function, and §"Migration" states what they cost.
(3) landed in the code first. When this RFC shipped, RegenSpan in the model was still a
prefix search of its own that stopped at the first hit. On a document holding a block for ab,
the model, asked for a, answered with that block where update_regen answers with nothing.
#1097 closed that. RegenSpan is now FirstWithCommand over a modelled RegenScan, compared
by equality. UpdateRegen rewrites every block the scan names.
TheLocatorComparesTheWholeCommand
is the same document, proved with the code's answer. The cost was UpdateRegenSpec's re-scan
clause, as predicted. Its induction, ReadBack, shows that each rewritten block reads back
where it was written. The count clause the axiom lost also returned, stated over the scan:
Commands(RegenScan(result)) == Commands(RegenScan(text)).
The model's scan was not tied to lines at first. The code requires a block-form open tag to
start its line and the close tag to stand alone on its own line; the model did not, and a proved
witness stated the smallest document where the two disagreed. #1137 closed that. RegenScan
now reads a line at a time, following scan_markers' block-form branch, and the witness became
an agreement:
ACloseTagMustStandAlone is the same
document, and both sides now read no block in it. markers.rs pins the code's answer, so a
change to either side reddens one of them. A second witness,
AnOpenTagMidSentenceIsProse, covers the
other half of the rule: an open tag mid-sentence with no close tag beside it is prose, and the
scan reads the block on the next line.
The proof did not carry over unchanged. A block-form body may hold a close tag mid-line, because
the scan stops only at one that stands alone. So a block the rewrite leaves untouched is not
re-read by ReadBack, which asks for a body with no close tag at all. It is re-read by a
separate lemma, NextBlockPrefix: two texts that agree through a block's close tag, and on that
tag standing alone, read the same block.
It was a debt, not a blocker — corrected from an earlier draft. This section first said the
model was not optional past count, on the reasoning that count's command carries a query, so
yidam count district is a prefix of yidam count district-at-large and the document above
becomes one a corpus can write by accident. The premise is right and the conclusion was not: the
hazard is a property of the old locator, and clause (3) is what removed it. update_regen
matches by equality over the one scan and writes every match, so on the implementation count
ships against, neither the clobber nor the unwritten second block is reachable —
a_query_that_prefixes_another_does_not_clobber_it and
two_blocks_asking_the_same_question_are_both_written in yidam/cli/tests/count.rs are those
two documents, run. What #1097 bought is a model that says so. It did not buy safety that
count was waiting on.
count is pull-shaped, and the guard says so
count reads the tracked markdown set — tracked::list, for the reason
unclaimed_blocks already gives at length — collects
every block whose command begins yidam count , runs each distinct query once, and writes each
block through the single write point.
every_generator_in_the_crate_is_listed is
amended, not exempted. Its named == call_sites clause exists so that the guard can see which
block each call site writes; a call site whose command is computed is one the guard genuinely
cannot read, and the honest amendment is to require that such a site name its generator's
prefix as a literal, and to assert that the prefix is itself a listed generator name. A site
that names neither still fails. The floor clause is unchanged.
Revised on implementation. The prefix a call site can spell is not "yidam count " but
format!("yidam count {}", argument) — the command has to be built to be passed, and a site
that spelled the bare prefix beside a computed command would be naming a string it does not
use. So generator_named_by reads a "yidam …"
literal containing a { as a prefix and truncates at the brace: "yidam count {}" names
count. The clause the RFC asked for is unchanged in effect — the site names its generator, and
the guard can attribute the block — and the literal it reads is now one the code actually
evaluates rather than one kept beside it to be read.
The bare form is a command no generator writes
<!-- REGEN: yidam count --> names no query. No run can ever write it, and it is not count's
to guess at — but registering count in GENERATORS would have made
unclaimed_in match it whole and call it claimed.
The block would then have gone from correctly reported by #1062's gate to silently accepted,
which is the exact defect that gate exists to close, introduced by closing this one.
So a generator carrying an argument is claimed a second way and not the first.
PARAMETERISED lists count with its usage, and
claimable partitions the fifteen: fourteen are
matched whole, count is matched only as its name followed by a space and an argument. The
separator is the rule. Without it yidam counterexamples would be claimed by count, which is
#1058's collision wearing a different hat.
The verdict's wording changed with it. unclaimed used to read "name a generator that does not
exist" and then print a list — self-contradictory the moment that list contains count, since
the generator plainly exists and the block is still not one it writes. It now reads "name a
command no generator writes", and the list shows count <query> rather than count, so the
remedy for the bare block is visible in the message that reports it.
A query that does not typecheck
count refuses the block, names the file, the query and the diagnostic, and exits 1 — the
report-then-fail shape regen, doctor and query already have. It does not write a 0: a
query that cannot run has not counted zero of anything, and a gate whose failure mode is to
publish a plausible number is worse than no gate. The block keeps whatever it holds, and it
says which block and why. Every block is visited before the failure, so one bad query neither
hides the next nor costs the rest of the document its refresh.
It fails as an error, not as a verdict — including under --check. regen runs its
generators with .with_context(|| format!("running {name}"))?, so any generator returning Err
aborts the pass before a report is rendered; under --format json that means no RFC-0016 report
at all, only the anyhow chain on stderr. This is pre-existing structural behaviour rather than
something count introduces, but count is the first generator that can fail on corpus
content rather than on I/O, so it is the first time the path is reachable in ordinary use. A
machine consumer of regen --check --format json therefore has to read the exit code, not the
absent passed field. Widening the report to carry a third verdict is the open question below,
and this is the second reason to answer it.
What this does not touch
Number formatting. The body is decimal digits. roadmap.rs needed a thousands-grouping
helper and an English-number helper to bind one repository's prose, so both forms are real;
neither is in this RFC. See §"Open questions".
Counts that are not corpus counts. demi-moore's three drifted figures are crates,
connectors and invariant checks — facts about a workspace manifest and a Rust source tree, not
about .yidam/corpus/. A query cannot answer them and this RFC does not claim to. What it
removes is the corpus-shaped subset, which is the whole of #1071's case and part of the other
two.
<!-- TEMPLATE: … -->. The template marker is single-line already and gains nothing here.
The yidam prefix. A block belonging to another program is still not yidam's to judge, and
an inline block belonging to another program is still not yidam's to write. #1058's namespacing
question is untouched and stays open.
Migration & compatibility
A derived repository does nothing. No existing document changes, no existing block changes shape, no corpus content moves. Every generator writes exactly what it wrote.
Prose that shows the marker becomes prose that has one — measured. Twenty-eight lines across
the fourteen repositories on disk carry an open tag and a close tag on one line, and every one of
them is the same vendored document: yidam/sdks/README.md and its thirteen
.yidam/.vendor/ copies, describing the marker format inside a code fence. None of them begins
its line with the open tag, but the inline rule is not line-anchored — the mid-sentence case is
the point — so under this RFC they parse as blocks named cmd and command.
The harm is nil and is worth stating rather than assumed. unclaimed_in masks fenced code, so it
does not report them. They are well-formed, so malformed-regen-block has nothing to say. No
generator is named cmd or command, so nothing writes them. What changes is the marker
sequence parse_markers returns for one document that no generator regenerates. The template's
own copy is reworded so the count goes to zero going forward; the thirteen vendored copies fix
themselves on the next re-vendor, and cost nothing until then.
Why the template bump is major. VERSIONING.md's Layer 1 table calls a REGEN marker format
change major, and the instinct is to argue that this one is additive. The instinct is wrong, in
the direction that matters. The format is additive for documents and not for readers: a
document carrying an inline block, read by an SDK from before this change, is the destructive
case §"Problem" measures — silently, and against the reader's own repository. That is the
pattern where a new producer strands an old consumer, and the honest bump is the one that makes
a pin say so. Template v0.8.0 → v0.9.0, following the bootstrap layer's 0.3.0 → 0.4.0
precedent for a major in 0.x.
The two update_regen behaviour changes. Exact-command matching can only stop a write that
was landing on the wrong block, and no generator name in this repository is a prefix of another —
checked over generator_names, the fifteen are pairwise
non-prefixing. Writing every match instead of the first can only write a block that was stale.
Both are covered by new parity fixtures, and both are stated as postconditions in the model.
Alternatives considered
A test per figure, which is what three repositories did. It works and it does not scale: each
figure is pointed at by hand, an unpointed figure is invisible, and roadmap.rs measured its own
blind spot at one figure in a narrative section. The deeper objection is allen-county-ohio's,
which declined to pin node counts at all: "a test reading assert_eq!(marks, 129) fails on the
next phase that adds a site, and a gate that fails for the wrong reason gets switched off." That
objection is correct about tests and does not reach a REGEN block, because a block's remedy is
mechanical — yidam regen rewrites it, and the only thing CI asks is that the rewrite be
committed. The figure moving is not a failure; the figure moving without the sentence moving is.
class=district, as filed. A second way to name a set, unable to express three of the four
figures the reference gate binds. Declined in favour of RFC-0018.
A block-shaped count, no marker change. Ships in a day and answers table cells and list
items, which is where some of these figures live. It does not answer a number in a sentence,
which is where #1071's is, and it would leave the two-scanner disagreement in place for whoever
meets it next.
Line-anchoring the inline form. Requiring the open tag to begin its line gives zero false positives against the twenty-eight lines measured above — and forbids the sentence the RFC exists to write. Declined for the reason it was proposed: the position of the number in the sentence is not the author's to negotiate.
Masking fenced code inside scan_markers. The alternative to living with the twenty-eight.
Declined: mask_code is a CLI concern, the parser is a parity function in three languages, and
the measured harm does not pay for a markdown-aware scanner in each of them.
Open questions
Grouping and words. roadmap.rs carries grouped() for 3,104 and number_word() for
twelve, in one repository, for one document. A count generator that can write neither will be
adopted for the small figures and worked around for the large ones. The candidate is a modifier
on the command — yidam count district --grouped — and the question is whether the separator is
the author's (, versus . versus a space) and therefore a corpus declaration rather than a
flag. Not answered here; the two helpers are the evidence that it has to be answered.
What --check says about a block whose query stopped typechecking. Today stale and
unclaimed are the two verdicts and each has a remedy that clears it. A block whose class was
renamed out of the ontology is neither: yidam regen cannot fix it and the name is a real
generator. A third list, or unclaimed widened, or rename taught to rewrite a query — three
answers, none obviously right.
The implementation forces half an answer and leaves the other half open. count returns Err,
which regen propagates before rendering anything — so under --format json the run emits no
RFC-0016 report at all, and a machine consumer has to read the exit code rather than a missing
passed field. That is honest (the pass did not complete) but it is not a verdict, and it is
the shape every generator's I/O failure already had. Whichever of the three answers wins, it
also has to say whether a content failure stays an error or becomes a third list inside a report
that still renders.
When the model reaches the scan — settled by #1097 and #1137. RegenSpan is a selection
over a modelled RegenScan, and UpdateRegenSpec is proved over it. The scan is tied to lines
as the code's is; §"One scanner" names the witnesses. What the model still does not say is why
the code rejects a candidate: scan_markers reports CloseTagMissing and ClosedOnAnothersTag,
and the model's scan only skips the candidate.
Whether a block should be able to state its own expectation. <!-- REGEN: yidam count district -->44<!-- /REGEN --> publishes a number and asserts nothing about it. RFC-0035's
coverage: is the place a corpus says how many there should be, and a block that could say
expect>=44 would be a gate rather than a report. That is a different surface and probably
RFC-0035's, but it is the first thing a reader asks.