CLI reference
Every command yidam carries, grouped as yidam --help-all groups them. This page is the map;
yidam <command> --help is the detail. A gate holds the two surfaces equal: a command the
binary offers and this page omits fails cli_reference.rs.
yidam --help is a shorter listing: the fourteen commands a session usually needs. Everything
on this page is on --help-all.
So the roster below is the roster you have. It says nothing about the prose. A description here, or in a long help, is written by hand. It can fall behind the behaviour it describes, which is what #873 was.
Three conventions run through the whole surface.
A * means the command rewrites files in the repository it is run against. That was
previously visible only in each command's long help, where you had to already suspect it to go
looking. That is the wrong way round for a tool people point at a checkout
they only meant to inspect.
--format json is available on most commands and emits the machine-readable report contract
of RFC-0016. text is the default and is byte-stable. The editor
surface and CI both consume the JSON, so the prose is free to stay prose. Commands below that take
no options at all are marked (no flags).
--root <DIR> names the corpus to read. Every command that reads one takes it, including
the generators that refresh a README block. Without it yidam finds a repository with
git rev-parse --show-toplevel from wherever the process started, which makes the working
directory load-bearing. It takes the corpus directory or any directory inside one. The root
field of every JSON report says which one was resolved.
A named --root must hold a corpus. A directory with no .yidam/ is refused, not reported
empty. The working directory is not held to that. Run a report from nowhere and it still prints
what it found. Naming a directory is an assertion; being somewhere is not.
The commands that refuse it are the * ones that do more than refresh a README block. They
author a commit, write a build artifact, or name a repository to derive. Naming a corpus to
report on is one promise. Being run inside the one you are about to write to is another. It is
not a promise a reader should be able to make by passing a path. yidam <command> --help is
the answer for any single command.
Some commands need a build carrying the matching cargo feature. Those are marked, and
Installation has the table. yidam --version prints
the feature list of the binary in your hand.
Checks and gates
Read-only, and they exit nonzero on a problem — which is what makes them usable as CI steps.
| Command | What it answers |
|---|---|
doctor | Is this setup sound? One verdict per check, and a remedy where something is owed. --strict makes warnings fail. --only <id> reports only that check. See Troubleshooting |
graph-check | The graph gate: orphans, broken links, missing labels |
lint | Corpus quality checks against the baseline ratchet |
index-verify | Does an embedding provider reproduce this index's contract? --remote reads the contract from the vector index [index.remote] declares |
samudaya-audit | Inspect and validate samudaya/ seed files (no flags) |
derive check | Does each argument under [derive] paths hold to the corpus it cites? See below |
lint carries the most flags of any command, because it is the one with a ratchet:
| Flag | Effect |
|---|---|
--commits | Also check the git log against the commit vocabulary |
--range <RANGE> | Restrict --commits to a revision range, e.g. main..HEAD |
--explain | Print each check's rationale beside its findings |
--warn | Report findings but always exit 0 |
--bless | Rewrite .yidam/lint-baseline.yml from this run instead of gating on it |
--init-baseline | Write the baseline only if absent, then exit — safe to run unconditionally |
lint answers did this change make the corpus less clean? — not is the corpus clean? The
baseline is what makes that the question. — see
Configuration.
derive
A memo, dossier or finding carries a claim out of the repository. agent-conduct.md gives
that claim three rules, and derive check holds it to them
(RFC-0045). It reads every artifact under
[derive] paths:
claim: The dam's release schedule governs discharge below it.
reach: attributed
cites:
- node: reach/tailwater
span: Discharge tracks the release schedule
answers:
- node: reach/tailwater
refusal: The record does not say the dam caused the 2019 avulsion.
answer: This memo claims the schedule, not the avulsion.
Each span must sit in one paragraph of the node it names. Resolution, drift and a declared
tag: are checked as lint checks a node's own local citation.
The tier is computed. It is the weakest standing beneath the spans. public admits
[verified] only, attributed admits [inference] and stronger, and internal admits
anything. An artifact citing nothing is [open].
A declared refusal is owed an answer. A node declares one under refuses:. When an
artifact cites that paragraph, it names the refusal under answers: and says why the claim
survives it. Quoting the refusal itself owes nothing.
| Finding | Severity |
|---|---|
derive-unresolved, derive-span-drift, derive-span-crosses-paragraph, derive-tag-drift | error |
derive-beyond-reach, derive-no-reach | error |
derive-unanswered-refusal, derive-stale-answer | error |
derive-refusal-candidate | info |
The candidate is a cited paragraph that reads like a refusal and declares none. It is a
question for a person and never fails the check. No [derive] paths means nothing is read, and
the check passes.
The practice
Six commands, and none is a gate. Each exits zero however much is owed, however far a corpus has drifted, and however a contribution reads.
| Command | What it answers |
|---|---|
due | What is due? Four clocks read together — index staleness, catalog TTL, unanswered questions, phases in flight. --strict exits nonzero on a due clock |
cycle | Where is this repository in its loop, and what is the next act? Owed, in flight, blocked, next — composed from due, phases, lint and graph-check. --strict exits nonzero on owed or blocked |
hook <event> | The loop's Claude Code hooks. session-start prints the brief; stop says what a turn moved. Always exits zero |
kuten * | What is this corpus's practice aimed at? Writes the declaration into AGENTS.md. kuten check reads the history against it; kuten adopt <name> declares one |
practice * | What has this corpus practiced? Writes a sampled series of its conduct into PRACTICE.md, each point read against the kuten that point's own tree held. Regenerated, never authored |
score <range> | How did this session's work read? One row per declared criterion, with its evidence |
cycle gives the loop an object
Four surfaces each answered a quarter of where am I, and what next. due says what is owed,
phases says what is in flight, lint and graph-check say what is blocked. A person opening
a session runs none of them, or one.
cycle is those four read together, in four headed halves:
| Half | What it reads |
|---|---|
| owed | due's five clocks, through due's own reader |
| in flight | the unsettled inquiry refs yidam phases lists — active, never a settled phase or a standing position |
| blocked | what lint's baseline ratchet and graph-check would fail on today |
| next | one act per half, each carrying the finding, clock or declaration behind it |
It composes rather than computes. Every number is read from the command that already owns
it. So this report and its sources cannot come to disagree. That is what status did: it
reported 26 active phases against a true count of 1.
Blocked means what a gate would fail on, not what the corpus was forgiven. Take a corpus
carrying two hundred blessed error-severity findings. It is not blocked by them: it agreed to
them, and yidam lint is green over it.
What appears here is what lint itself gates on. Three things do. A finding introduced against
the baseline; an entry past its expiry; a stale entry the run no longer reproduces.
The order under next is the command's, not a kuten's. No kuten profile declares a phase
order. The phases slot is a set of four type names and a commit-share band. Nothing reads
it as a sequence.
What a kuten does declare is named as what it is. Two things are: the phase types a phase opened now would be named from. And the kind of question the practice presses toward.
The ranking itself is blocked, owed, in flight, practice. It is an argument the command makes, and it states it at the bottom of every report.
A repository holding no kuten gets the other three halves and is told that one needs a declaration. Holding none is a supported state.
Like due, it exits zero however much is owed. --strict exits nonzero when anything is owed
or blocked. That is the union of the two signals it composes. A scheduled job wants one signal
off the whole loop. It should not go quiet because the half that fired was the other one. It
drafts no commit.
A round remembers the last one
cycle --record writes this round to .yidam/loop/<branch>.json. That is local state, and the
directory ignores itself. cycle --since-last reports what arrived and what left since then. A
branch with no record is told so, which is not the same as nothing having moved.
cycle --brief is the report cut to a token budget, 500 by default. --budget changes it.
The hooks hand a session the loop
yidam hook is what .claude/settings.json runs, and a person rarely runs it by hand. The
scaffold installs it, and yidam migrate scaffold adds it to an older repository.
| Event | What it does |
|---|---|
session-start | Prints cycle --brief into the session's context and records the round. It runs after a compaction too |
stop | Records the round a turn ended on. When something moved, it tells the person what |
A hook never fails the session. It exits zero on every path. Outside a corpus, or on any failure, it prints nothing.
A quiet turn costs one git status. The round records a fingerprint of the checkout. A stop
that matches it reads nothing else.
A kuten declares what the work is for
A repository declares what it is about — in its ontology, its classes, its central question. It had no way to say what its work is for. A kuten is that declaration.
It is vendored with the prelude, and it carries a revision. A decision record in
.yidam/decisions/kuten.yml adopts it. yidam kuten writes it into AGENTS.md, so an agent
meets it at session start.
A kuten narrows the loop and may not widen the model. Five things it may not do:
- add a commit verb — it may declare a subset of the closed list and gloss it;
- add or alter a claim standing;
- contradict Articles I–VI;
- change the graph encoding;
- loosen a gate. It proposes no severity, and a profile that tries is refused.
A repository holding no kuten is a supported state. It reports as one.
Adopting one in a corpus that already exists
The bootstrap writes the decision record at genesis. A repository older than the layer has none. Re-vendor the prelude first, then adopt:
YIDAM_REF=v0.4.0 mise run yidam-vendor-update
yidam kuten adopt inquiry
adopt copies the revision from the vendored profile. Typing it by hand is the error the
revision exists to survive.
It also adds the AGENTS.md section when that file has none. The scaffold carrying that
section is deleted at genesis, so repositories predating the layer never received it.
Where there is no AGENTS.md at all, it writes none. Conjuring one reaches past what was
asked. It says so instead, and prints the section to paste. doctor's kuten-read check
warns until a file carries the block.
It refuses when a record already exists. Changing a kuten is a decide: commit carrying a
superseding record.
kuten check asks a question and never fails
It reads the declaration and the repository's own history. Then it says where they disagree —
you declared inquiry and have settled no phase in two hundred commits.
That is a question for a person, not a defect. due's argument applies verbatim: a corpus that
has drifted is owed a look. So the command writes nothing and exits zero. Anything that refuses
arrives through the policy layer, visible as
an override.
Vintage is never reported as divergence. Every consumer reads the kuten and the prelude
this repository vendored, not the current ones. A repository whose vendored GRAPH.md has no
phase verb has not stopped running phases. It never could, and the check says so.
A comparison across revisions is annotated, never silently made. The vendored profile may have moved past the revision the decision record names. The report says so, and the numbers stay readable.
It also counts what the artifact cites. A repository that declares
[object] paths gets a coupling section. It counts the
relative links between the corpus and the artifact, in both directions. It says how many
nodes, decisions and catalog entries the artifact cites. The JSON lists the uncited ones by id.
They are not findings, because no rule says a node must be cited. A dead link from the artifact
is a broken-object-link warning in yidam lint.
score reads a contribution, not a repository
The genesis rubric scores a repository's birth. It fires once. After that nothing said whether a session's work was any good. The gates said only whether it broke something, which is a floor and not a standard.
yidam score <range> reads one range against declared criteria. It reports one row each: a
number, and the commits or nodes that number came from. There is no overall score.
A single number over a range names somebody's session. So there is none. Each row is a reading a person can disagree with by looking at its evidence.
The range is required, as diff's is. A default range answers about a scope nobody chose.
The criteria come from the kuten's rubric slot. Where a repository holds no kuten, the
template's own criteria are used. The report says which, every time.
It also says why. The report names one of four states:
- the criteria are the corpus's own;
- the repository holds no kuten;
- the range ends before the adoption the repository now holds;
- a kuten is declared and no profile is vendored for it.
The last is a warning, the same one kuten check and doctor give.
--brief adds four questions a person answers. Nothing scores them.
It writes nothing and exits zero however it reads. A kuten binds nobody. A score that gated would be a gate decided by one. It refuses exactly once. A range spanning a kuten revision has no single set of criteria. It names both revisions and stops.
due is not doctor, and the difference is the point
doctor answers is this setup sound now. It is read under suspicion, and it exits nonzero on
what is wrong. due answers is it time, and it is read on a cadence. That might be a cron job,
a weekly ritual, or the start of a session.
A corpus with three expired sources is not unhealthy. It is owed. Nothing about it is broken,
no traversal will lie, and the gate is green. Fold that into doctor's warnings and it reads as a
problem. The repository is doing exactly what it is meant to do. The reader would learn to skip
the line. So the two reports are separate. due has its own verdicts: due, ok, undeclared,
declined, unmeasurable. It exits zero unless you pass --strict to ask for a signal.
Every interval is declared, and a clock nobody set never comes due
A clock is an age and an interval. The age is measured. The interval is always something the
corpus said about itself, never a number in the binary. The reasoning is
escalate_after's.
| Clock | Measures | Interval | Unit |
|---|---|---|---|
index | Corpus files changed since the index was built | [due] index_after | files |
catalog | How long since a source record was retrieved | [catalog] ttl_days, or an entry's own | days |
questions | How long a question has gone unanswered | [due] questions_after | corpus commits |
phases | How long a bounded inquiry has been in flight | [due] phases_after | days |
superseded | How long a node has cited a source that changed since it was last committed | [due] superseded_after | days |
The catalog clock reads the interval where it already lived
rather than restating it under [due]. A source's TTL is a statement about the source. Two places
to set one number means one of them is wrong.
A clock with no interval reports what it measured and is never due. Its remedy is the key that would set it. That is the state of every repository that has not opted in. It is the design, not a degraded mode.
A corpus may also decline a clock outright, under
[due.declined]. That is a different thing from never setting
one. The clock reads declined, still prints what it measured, and prints the decision record
behind it. The record is required, so a decline is something a corpus wrote down and can be
asked about.
Three of the five count days and two do not, which is deliberate. How long a question has gone
unanswered is a fact about the repository, so its clock is HEAD. A corpus that has not committed
has not ignored anything. A source's TTL, a phase's time in flight and a source's new version are facts about the world.
The world does not stop moving because nobody committed.
What discharges a clock
due reports; it does not act. Only one of the five clocks is discharged by something
propose can draft:
| Clock | What discharges it |
|---|---|
index | yidam index-build. A build, not a commit |
catalog | yidam propose, which already drafts an open: against each expired source |
questions | A person. Deciding a question is answered is a resolution event, and Article V confines those to a sangha |
phases | A person. Settling a phase or abandoning it is not a mechanical consequence of a finding |
superseded | A person re-reading the source, and a commit to each node it lists. Any commit touching the node counts |
Each clock names its own remedy in the report. The distinction is visible where it matters, not only here.
README blocks
Each of these rewrites its own <!-- REGEN: yidam <name> --> block in the repository's README.
That is their purpose, and it is why every one carries a *.
| Command | Block content |
|---|---|
regen * | Refresh every REGEN block in one pass. --check reports staleness and writes nothing; --commit authors the regen: commit for exactly the files that changed |
status * | Repository overview: nodes, open questions, catalog, index freshness, genesis |
open-questions * | Unresolved questions, newest first |
corpus-index * | Every corpus node by class, with label and link count |
catalog-audit * | Which catalog sources the corpus cites, and which it does not |
index-status * | Whether the vector index is present, and how stale against the corpus |
agents-index * | The domain agents in .yidam/agents/ (no flags) |
skills-index * | The domain skills in .yidam/skills/, each with its status: (no flags) |
crates-index * | The domain-computer crates in crates/, each with the capability that runs it (no flags) |
packages-index * | The domain-computer packages in packages/, each with the capability that runs it (no flags) |
bundle-status * | Freshness of .yidam/bundle.yiz against the corpus it was built from (no flags) |
gates * | The gate table: every run: step of every job in .github/workflows/ci.yml, in order (no flags) |
routes * | The reading routes in AGENTS.md, by occasion, from the vendored routes.yml (no flags) |
Four more generators are filed by what they report, not by the fact that they generate.
They are vault-status, decisions-log,
kuten and practice. yidam regen runs all sixteen.
gates reads the workflow and nothing else. A derived repository's README used to say in a
sentence which checks CI runs. The sentence could not follow the workflow. The table can.
A job that only runs once a layer exists says which path switches it on. A step that reports
without gating says so.
In a derived repository a stale REGEN block is a failing build. Run mise run regen before
committing; yidam regen --check is what CI runs.
A block naming a generator that does not exist is a failing build too. No generator
answers for it, so yidam regen never writes it and it keeps whatever it holds. A typo in a
marker name would otherwise turn a generated block into a hand-maintained one, silently.
yidam regen --check names the file, the command and every generator there is. Correct the
name, or delete the block.
Only yidam commands are judged. A block another program refreshes is left alone, and so is
one a page only shows inside a code fence.
The yidam prefix is reserved. A block marked yidam <name> belongs to yidam, and
yidam regen rewrites it whatever wrote it. Mark your own generator's blocks with your own
program's name: <!-- REGEN: my-tool <name> -->.
A gated block holds only what every checkout of the commit agrees on. The tree is all they share. So a block may not report anything read from outside it.
Phase counts broke that rule. They read branch refs, and which refs a checkout holds depends on
how it was fetched. For one commit there was then no count that passed everywhere. CI and a
fetched machine wanted different numbers. Pushing a phase branch also reddened this gate on every
open pull request. The counts moved to yidam phases and yidam status --format json, which are
reports rather than committed content.
Build artifacts broke the same rule, and for two releases nobody noticed. yidam status reported
whether .yidam/index/ existed. index-status and bundle-status reported what those artifacts
held. None of it is in a commit. So the gate passed or failed on what a machine had built. CI has
neither artifact and never saw it. The three blocks now hold a pointer. The measurements stay in
the commands: index-status, bundle-status, doctor and due.
Depth is the other half. genesis reads the repository's first commit, which a shallow clone does
not have. Git names the boundary commit a root instead. yidam regen refuses in a shallow clone
rather than writing that date. Check out with fetch-depth: 0, or run git fetch --unshallow.
These are the commands to be careful with against a checkout you only mean to read.
yidam status sounds read-only and is not — it rewrites the README block. yidam doctor is
the read-only overview.
The corpus and its history
| Command | What it does |
|---|---|
graph | The corpus graph: nodes, resolved edges, and the classes that license them |
neighbors <node> | One node's neighbourhood — the traversal serve --mcp performs. --depth |
retrieve <words> | Semantic search over corpus nodes — the retrieval serve --mcp performs. --k, --class, --corpora, --where |
query <query> | A typed path over the resolved graph |
pack <query> | A query's full answer filled to a token budget, with an account of what did not fit |
estimate <query> | What a query would cost before you run it |
count [query] * | How many nodes a query matches. With no query, refreshes every <!-- REGEN: yidam count <query> --> block in the tracked markdown set — the form regen runs (RFC-0043) |
diff <range> | Node and edge changes between two git refs |
check-diff [range] | What a code diff names that the ontology does not (RFC-0021). The ontology includes every installed dependency's classes. Defaults to the merge-base with main — this branch's work. An unmodelled-concept row may carry a nearest declared name: one whose --separated word shares a root of at least five characters with a word of the type's name, the two words' lengths within three. A word that ends three or more of the report's type names and appears nowhere else in them, such as error in FetchError, is the code's suffix and is not compared. More shared words win, then the longer root, then the name. It is a lead to check, not a match — no model reads either name |
log [range] | Commit history classified as testimony or pipeline work. --epistemic, --operational |
phases | Active inquiry phases — ma/*, rigpa/* and phase/* branches. State is active, interrupted, settled, rewritten or position. Source says which evidence decided it |
replay | Corpus health reconstructed across the repository's whole history. --every |
cohort <repo>… | What a set of derived repositories says about the prelude they inherited: per norm, how many kept it. --paths |
decisions-log * | Decision records in .yidam/decisions/, newest first. Writes the <!-- REGEN: yidam decisions-log --> block where that directory has a README carrying one (no flags) |
sangha | Electors, positions, and settled resolutions |
vocabulary | The closed commit vocabulary. --check <subject> tests a subject line before the commit exists |
rename <old> <new> * | Rename a node or a catalog entry (catalog/old), rewriting every edge into it. --dry-run |
migrate <sub> * | Change an ontology and every instance that adopted it, as one event. --dry-run |
propose * | Draft findings as proposed epistemic commits on a propose/<head> branch |
run [step] * | Invoke the stale capabilities declared in .yidam/capabilities.toml, in dependency order, and commit what each produced with a receipt. Named with a step, runs that step and everything it declares it comes after; with nothing, the whole manifest. --dry-run plans and writes nothing |
phase <sub> * | start opens a phase and snapshots what it begins from. run invokes its plan, recording each step. settle checks it produced outputs and drafts the merge — a person runs it |
cluster <sub> * | The same run on Argo Workflows, one pod per act. workflow generates the manifest. network-policy writes the egress fence. pin bundles the branch tip into a vault. step runs one capability against a bundle. land is the one command that moves a ref. admit says whether a run is owed. survey, ask and gather run a gather. status reports recent runs |
gather <name> * | Ask pinned peers the question in .yidam/gathers/<name>.toml. Their answers land as one cited ? node on a propose/* branch. --dry-run asks and writes nothing |
dispatch <question> * | Run the agent electors named by --seat, each as its seat declares. Each position lands on a propose/elector/* branch with a receipt. --dry-run runs them and writes nothing |
Retrieval, and the corpora it can reach
retrieve takes words, not a query expression. Reach for it before you know the corpus well
enough to write one. It is the same call serve --mcp answers, dispatched by tool name. What
it prints is what an agent sees.
Without a vector index it degrades to keyword search and says so. That arm ranks by BM25 over the nodes it scanned. With one it ranks by cosine similarity.
--where keeps only the rows whose declared date or number properties satisfy a predicate.
The predicate is the text inside a query step's [...]. It is the same grammar, the same
check and the same comparison. So the two commands below return the same tenures, and the
second returns them ranked:
yidam query 'tenure[began<=1893, ended>?1893]'
yidam retrieve 'courthouse' --where 'began<=1893, ended>?1893'
The predicate runs before the k cut. Without --class it narrows to the classes that declare
each property with an ordered type. A predicate that cannot be asked is rejected with query's
code. Over a local index built before typed property columns it is where-unindexed; run
yidam embed && yidam index-build. A remote index carries no properties and refuses it.
--corpora reaches the other corpora sharing the vector index [index.remote] names. This
corpus is always searched, and the named ones are added to it. Every result says which corpus
it came from: a yidam://<corpus>/node/<class>/<name> identifier, and a corpus field beside
it. A row from another corpus cannot be opened. Its text is all you get. That is why a row
the index had to cut says so.
yidam retrieve 'school funding' --corpora ohio-budget,allen-county
A corpus is named either by a nickname this repository declares or by its genesis hash:
[index.remote.corpora]
ohio-budget = "3f2a9c4d1b70"
The nickname is this repository's own. Nothing outside this file knows it, which is why a
result carries the hash. yidam index-push --dry-run lists the corpora an index holds, with a
row count each. It writes nothing, and it is where the hashes come from.
Only a shared remote index can answer a span. A local .yidam/index/ is one corpus by
construction. So a --corpora against one is not an error. It is not silently honoured
either. The local index answers, and the report reads scope local. That is what tells you
your request did not take effect.
The query language
query, pack and estimate share one language. A query is a class anchor and a sequence of
typed hops:
yidam query 'reach -measured-by-> gage'
yidam query 'concept~"hydropeaking" <-exhibits- reach'
Whitespace around a hop is required. -rel-> and <-rel- are single tokens, and that is what
lets a hyphenated relationship name be unambiguous. ~"…" is a similarity anchor. It opens on
--anchor-k entry nodes (default 1). An anchor is a starting point, not an answer.
Predicates
A step may be filtered by […], comma-separated. Commas are and; there is no or.
| Operator | Holds when | Declared types |
|---|---|---|
= | the value equals the operand, compared at the precision the operand was written — so observed_on=2026-08 matches every day in that month | any |
!= | it does not. On a list, of every element — or claim_tag: [open, verified] would satisfy claim_tag!=open | any |
~ | contiguous, case-insensitive substring of the value's serialized text | any |
< <= > >= | the date, or the number, orders that way | date and number only |
? | written as prop? — the node carries no value for it | any |
Ordering is date and number only, and asking for it elsewhere is refused rather than
answered. Comparing a string would compare text, ranking 10 before 9 and saying nothing
about it. yidam query 'reach[regulated<9]' comes back unordered-property with that reason.
A number compares exactly, and = on one is numeric. reach[length_km>9] returns the
reach measuring 10, and length_km=7.0 matches a stored 7, so <, = and > are
trichotomous. There is no precision rule: 7 is a point where 1893 is an interval. The
operand is a bare number. The unit is declared once on the class, as unit: km, and never
written in a value or a query.
A comparison runs at the precision the two sides share. began<1900-06-01 holds for a
began: 1893: 1893 is before 1900 whatever day it fell on. This is deliberately not ='s rule.
Ordering at the query's precision would silently drop every node the corpus knows only to the year.
? after an operator means an absent property satisfies it too. The standing rule is that an
absent property never matches, for any operator including !=. ? is how one predicate opts out
of it, where a reader can see it.
That is what makes the interval question expressible. The filter is a conjunction, and "ended after 1893 or never ended" is not:
# every tenure open at some point in 1893 — the line of office-holders
yidam query 'tenure[began<=1893, ended>?1893] -of-office-> office'
# questions nobody has closed
yidam query 'question[closed?]'
prop? and prop>?D are one character apart and mean different things. The first tests for
absence and takes no operand. The second is a comparison an absent property also satisfies.
| Flag | Applies to | Effect |
|---|---|---|
--select | query, estimate | Fields to project: node, class, label, description, body, properties.<name> |
--limit | query, estimate | Bounds the projection, not the traversal — the reported count is always the full one (default 50). With --paths, the number of paths printed (default 5) |
--budget | pack, estimate | Approximate token budget, 1 token ≈ 4 chars. pack is unbudgeted by default, because a default budget would silently truncate the first pack anybody builds |
--at <ref> | query | Answer as of a commit, reconstructed from git objects; the working tree is never touched |
--between <a..b> | query | Answer at every corpus-touching commit in the range, as a series |
--across | query | Query installed dependencies too. Every result says whose corpus it came from, and no hop crosses a corpus boundary |
--paths <from> <to> | query | Print the typed paths between two nodes, shortest first. Each one is a query that returns <to> when run. Takes no query |
--max-hops | query --paths | The longest path searched (default 4). A cut-off search names the nearest length that would connect |
migrate subcommands
A class definition cannot be corrected in place once the class contract gates. Editing it puts every instance in violation until each is fixed by hand. These do both halves together and write a record of what they touched.
| Subcommand | Changes |
|---|---|
class | Rename a class: its definition, its directory, and every edge that named it |
property | Rename a declared property on a class and on every instance carrying it |
retype | Change a declared property's type; requotes an instance value where only the quoting differs, refuses the rest |
value | Rename one value of a declared values: set, on the declaration and on every instance holding it |
edge | Point a declared relationship at a different class, at both ends |
references | Lift every reference written inside an evidence tag into the node's references: field |
findings | Lift every paragraph an earlier propose spliced into prose into a yidam: record |
routes | Put the yidam routes block into AGENTS.md, in place of its whole-file reading list |
scaffold | Mark the part of ci.yml and CLAUDE.md that a re-vendor updates |
locations | Rewrite every catalog location of kind doi or pmc as kind: identifier |
retype converts an instance value where the two types differ only in how the value is
written. A number is an unquoted YAML number. So retyping to number unquotes "24", and
retyping to string quotes a bare 24. Each is a repair property-type already names. A value
holding no number — about 24, ~24 — is refused, and the refusal says what it tried. A retype
into string on a property that declares values: is held to the set. An instance outside it
is reported and blocks. Widening the set or fixing the value is the author's call.
value renames one item of a closed set. The set is closed, so neither half can go first. It
rewrites the item on the declaration and the value on every instance holding it, in one event.
The quoting follows what was written. A value outside the set is refused. So is a rename onto a
value the set already holds, which would merge two.
references, findings and locations migrate data rather than the ontology over it.
A detail reading [verified — #362] names an issue no consumer can follow. The migration writes
issue/362 to the node. Where the detail was the reference and nothing else, the tag then
collapses to [verified].
Where the reference sits inside a sentence, the entry is written and the sentence is left
exactly as written. Excising #42 from #42 (agglomeration) would leave (agglomeration).
Nothing but an issue number is written unless the thing it names is on disk. So Auglaize/Mercer
and 6a/6b resolve to nothing and stay prose. Nothing is written that
reference-not-in-the-grammar would report. It is idempotent; run it with --dry-run first.
Applying it to the two corpora that write most of these tags left yidam lint reporting what it
reported before. One claim-tag-malformed finding went away per collapsed tag. That is 535 → 399
in one corpus and 534 → 459 in the other.
findings is the one-time lift that propose left behind. Before the record, a finding was an
English sentence appended to description:. It was found again by grepping the prose for its
own opening words. The paragraph's check and the finding's words become a record under the node's
yidam: key, and the paragraph goes.
A paragraph reworded past recognition stays prose. The reconstruction is exact — the marker, the check in brackets, and the fixed framing sentence — or it does not happen. A sentence an author rewrote is theirs, and guessing where the tool's words ended would edit somebody's argument. The report counts those rather than listing them as work to do.
Two numbers move for a corpus that runs it. The [open] claim counts drop, because a carried
question stops being counted among the claims the corpus makes. open-questions does not
change: the record is still an open question on the node. That asymmetry is the thing the
paragraph could not express. It is idempotent; run it with --dry-run first.
locations moves each catalog location of kind doi or pmc onto kind: identifier. The
value moves under its scheme unchanged:
- kind: doi - kind: identifier
value: 10.1167/tvst.8.5.14 → value: doi:10.1167/tvst.8.5.14
A value that already carries its scheme keeps it rather than gaining a second. The quoting
follows what was written. Run it with --dry-run first:
yidam migrate --dry-run locations # what it would rewrite, and what it leaves
yidam migrate locations
A value that is a URL is left alone and reported. https://doi.org/10.1/x holds a DOI.
Taking a link apart to find one is a reading of it, and that reading is yours. Rewrite it as
kind: url, or as kind: identifier with the id the link holds. A block-scalar value or a flow
mapping is reported too. So is a rewrite that does not read back as written. Each leftover is
still a catalog-location-malformed finding, and the report lists it.
propose is deliberately small
Three acts only. open records a finding's question against the node it is about. withdraw
deletes a node this corpus declared over-collected via [propose] withdraw_uncited_after.
close retires a question this command opened whose finding is gone.
Nothing merges itself and nothing synthesizes — no edge is drawn, no claim is re-tagged, no node
is authored. It writes git objects and one ref. The working tree, the index and HEAD are
untouched, so it is safe to run mid-edit. RFC-0020 has the
argument for why the surface is this small.
gather cites its peers and merges nothing
A gather asks every installed tonpa peer one question. You write it in .yidam/gathers/<name>.toml:
question = "What peak discharge has each gage recorded?"
query = "gage[state=VA]" # one step, in this corpus's own classes
answer = "peak_cfs" # the property whose value is the answer
key = "station" # optional: answers under one key are compared
lands_as = "question" # the local class the question node is written as
[peers.usgs]
classes = { gage = "station" }
properties = { station = "usgs_id", peak_cfs = "flood_peak", state = "state" }
Each [peers.<name>] says which of that peer's classes and properties mean yours. Nothing is matched by spelling.
A class or property the correspondence leaves out makes that peer unaligned, and it is not asked.
yidam gather <name> translates the query per peer and runs it against the pinned bundle.
Every answer becomes a cites: entry with the package, node, the bundle's commit, and a verbatim span.
It lands as one ? … node on propose/gather/<name>/<head>, as an open: commit.
Peers that give different answers under one key get a second ? node citing each answer.
Nothing is imported, no edge is drawn, and no claim is merged. Deciding between answers is yours.
Every peer is reported: answered, empty, unaligned, refused, missing or undeclared.
A path dependency is refused, because a working tree has no commit to cite.
A repeat over unchanged peers writes nothing and reports unchanged.
Each asked peer gets a v2 receipt at .yidam/runs/gather/<name>/<peer>.yml, in the first commit.
dispatch runs an agent seat and proposes what it wrote
An agent seat declares what runs it in .yidam/sangha/dispatch/<name>.toml, on its own ma/<name> branch:
model = "claude-opus-4-8" # must equal the row's Model
version = "1" # must equal the row's Version
run = ["sh", "elector.sh"] # the argv
reads = ["elector.sh", ".yidam/corpus/**"]
config = ["elector.sh"] # the files hashed into Config
yidam dispatch <question> --seat <name>… reads the declaration, the registry row and the corpus at each seat's tip.
A row must already record the model, version and config hash. A seat whose row does not is refused, and the report gives the hash to record.
A human seat is refused.
The elector runs in a scratch tree with $YIDAM_SEAT, $YIDAM_QUESTION, $YIDAM_POSITION, $YIDAM_MODEL, $YIDAM_MODEL_VERSION and $YIDAM_CONFIG set.
It may write .yidam/sangha/positions/<name>-<question>.md and its commitments file, and nothing else.
The position and a receipt land as one open: or revise: commit on propose/elector/<name>/<question>/<tip>.
The seat does not move until you take the commit: git merge --ff-only <proposal>, or git cherry-pick -S <proposal> if the row binds a Key.
A repeat over an unchanged tip reports unchanged. --dry-run runs the elector and writes no branch.
The report derives independence: over the seats that ran and never counts positions.
The command exits non-zero if any seat is refused or fails.
cohort reports about the prelude, not about a corpus
replay and kuten check answer about a corpus. Neither can see that a rule lost
everywhere. That is the question
docs/post-genesis-measurement.md had to ask by hand.
yidam cohort ../repo-a ../repo-b ../repo-c
The rows are norms and the columns are repositories. A rule every derivation fails is one finding about the rule, not five findings about five corpora.
Read-only: nothing is checked out and nothing is written. It is safe against repositories somebody is working in, and it exits zero however much lost.
Two controls, both printed rather than divided out. A repository whose vendored prelude does
not state a rule is reported as vintage, never as having broken it. Members are ordered by
authored commit count, the count beside every row. No rate is normalised by age.
A rule a repository never had the chance to break is unmeasurable, not kept. Every share is
quoted over the repositories the question could be asked of. Norms that held are reported
too: a negative result about coverage is the only durable record that coverage was checked.
Members are lettered rather than named. Most derived repositories are private, and this report
is written to be pasted somewhere. --paths opts back in.
One section is a measurement with no verdict per member: accretion. It fits the two
candidates RFC-0028 named for inquiry's retired classes bands. Each is replayed over every
member's history. A candidate any member leaves as it ages is printed as rejected.
The trailing window is read over the last 25, 50 and 100 authored commits.
RFC-0028 names no window, so the candidate is rejected only if it fails at all three.
Each member also carries upstream: how many commits cite an issue on the template's tracker.
It is a count. No norm is scored on it.
run is a plan, and only an operational commit advances your branch
.yidam/capabilities.toml declares what may run. Per entry: a kind, the argv to invoke, the
globs it reads and writes, and the commit verb it authors.
The kind is one of the three capability types the domain computer is built from —
calculator, connector, featurizer. All three are declarable and only a calculator
is invoked. A plan holding one of the other two is refused before any of it runs. The refusal
names the step, its kind, and why that kind is not invoked. A connector reaches a network and a
credential path. Feature engineering is declarable and not yet built.
[capability.travel-tier]
kind = "calculator"
run = ["sh", ".yidam/capabilities/travel-tier.sh"]
reads = [".yidam/corpus/**", ".yidam/capabilities/**"]
writes = [".yidam/computed/**"]
verb = "compute"
yidam run travel-tier checks the declared reads out of HEAD into a scratch directory. It
invokes the step there, with $YIDAM_IN, $YIDAM_OUT, $YIDAM_STEP and $YIDAM_INPUT_COMMIT
set. A step whose reads admit a corpus node gets a fifth, $YIDAM_GRAPH. It names a file
holding the corpus as yidam parsed it. Classes, labels, properties, and every link with its
target already resolved. So a calculator writes no parser of its own.
A list arrives one row per item, keyed <key>[<i>]. A record in a list adds one row per field,
keyed <key>[<i>].<field>.
That file is sliced to what the step reads. Its digest is part of the input state. Changing how
yidam resolves a link therefore makes every calculator's step stale. A step that reads no corpus
node is handed no $YIDAM_GRAPH. The name is removed from the environment rather than left empty,
so a script can test for it. What the step wrote lands as one commit on the current branch. A receipt lands with it, at
.yidam/runs/<step>.yml, naming the input commit and every digest.
Both declarations are load-bearing. The step sees what reads resolves to and nothing else. It
cannot quietly depend on a file it did not name. A step that writes outside writes is refused,
with nothing committed.
The verb decides the destination. An operational verb advances the current branch. An
epistemic verb — establish, revise, resolve — loads and runs just the same. But its
commit lands on propose/<head>, and the branch does not move. A person merges that branch to
accept it, or deletes it to reject it. So a run may open a question and may not decide one.
There is no route field and no policy key. A corpus that could write that permission for
itself could license its own runs to author establish: on its baseline.
It writes git objects and refs: the current branch, propose/<head>, or both when a plan holds
both kinds of verb. The working tree and the index are untouched, so it is safe to run mid-edit. It therefore leaves your checkout behind HEAD. The report says so, and names
the path-scoped git restore that syncs it. A re-run whose inputs have not moved writes no commit
at all.
A manifest is a plan. An entry's after names the steps it waits for. yidam run travel-tier
runs that step and everything it comes after, dependencies first. yidam run alone runs the
whole manifest. An after that forms a cycle is refused with the cycle named, and nothing runs.
A step may not come after an epistemic one. That step's output is on propose/<head>, not in
the tree a dependent is checked out from. So every epistemic step runs last, at the head the
branch steps leave.
A step whose input state matches its committed receipt is skipped, not invoked. The report names
each skipped step and why. A step that reads what the repository does not hold declares
ageing_days. It is re-run on that cadence even when its inputs have not moved. On the branch,
that re-run lands a commit even when the answer is the same. The commit records the check. An
epistemic re-run that reproduces its pending proposal lands nothing. The proposal already holds
that answer.
--dry-run resolves the plan and reports what is stale. It invokes nothing and writes nothing: no
commit, no ref, no receipt.
RFC-0026 has the argument for what a run may author. Its §6 has the argument for freshness.
The second arm: a calculator that is a typed function
run takes a second shape. A table naming one script declares a typed calculator
(RFC-0042).
[capability.class-of]
kind = "calculator"
run = { gluon = ".yidam/capabilities/class-of.glu", calls = 2000000 }
reads = [".yidam/corpus/**", ".yidam/capabilities/**"]
writes = [".yidam/computed/**"]
verb = "compute"
The script's entry point is declared Corpus -> Computed. It is applied to the resolved corpus in
this process. There is no scratch tree and no child process, so none of the five variables is set.
A calculator that cannot perform an effect needs no sandbox to stand in.
Corpus is a record of three fields. nodes is every corpus node its reads cover, with each
node's properties, its links and where each link resolves. classes is the ontology. signals is
what the calculators before it committed. It is read back from the .yidam/computed/ files its
reads cover. One row per node, in the shape the entry point returns.
That third field is what lets a pipeline's second stage be typed. A calculator whose input is
another capability's answer declares that file under reads, and is handed it. One that declares no
computed path is handed an empty signals. So is one in a corpus where nothing has been computed,
and a script cannot tell those apart.
A row's node is a reference — gage/canyon-outlet — and so is a node's id. A script that wants
both matches one against the other. The signals are not attached to the nodes, and the reason is the
declaration. A second stage may read a computed file and no corpus path. Its signals are then about
nodes that are not in nodes. Attaching them would mean the typed arm needed a wider declaration
than the shell arm to compute the same rule.
Computed is a record of two fields and both are required. signals is the table below, one row
per node. summary is what the calculator counted about the run as a whole. It is
written under a top-level summary:, not against any node. A calculator that summarizes nothing writes
summary = [] and the key is left out of the document. Gluon records are exact, so a script that
omits either field is refused as not a calculator.
It must still declare its own script under reads. A declaration that does not is refused by name,
because the script's digest belongs in the input state. That digest goes in the receipt too. So
editing a typed calculator makes its step stale, exactly as it does for the shell arm.
calls caps how many calls the script may make. Omitted, the binary's default applies. A
calculator that does not finish is refused rather than warned about.
The arm is outside the default build. The downloaded binary parses this declaration and plans it. It then declines the step by name, and says which feature would run it. A corpus that wants the downloaded binary to run it writes the sequence form.
What a run computes, and how it reaches a search
A calculator writes into .yidam/computed/, which is committed. Nothing read it until #1028. A
corpus could compute how far each assertion may travel, and commit the answer. It had no way to
ask a search for the nodes that may leave the repository.
A file there is read as a signal table when it says it is one. Saying so is a top-level
format_version: 1 and a signals: list. Every row of that list names a node.
format_version: 1
signals:
- node: gage/canyon-outlet
travels_as: open
downgraded: true
Every other key in a row is a signal about that node. yidam embed attaches it to that node's
embedding record as a signals object. The object is absent where there is none. A corpus that
computes nothing writes the records it wrote before.
A signal is deliberately not folded into the embedded text. A tier a query can filter on is worth more than one a sentence embedding is nudged by. Concatenating it would also move every vector already built. Index columns are a separate decision and are not made here.
Rows are keyed in the reference grammar and in nothing else. That is gage/canyon-outlet,
node/gage/canyon-outlet, or the absolute yidam://<corpus>/node/<path> form naming this corpus.
A revision pin is refused rather than ignored. A signal computed against a past commit is not a
signal about the node as it stands. A signal name is repository-wide. Two files claiming one name
is refused with both named, rather than resolved in favour of either.
A file with no format_version is listed and not read. A calculator whose output is a report
rather than a table stays legal. yidam doctor's computed check reports what is there, and what
no capability declares it writes. It also reports whether the answer still stands. That includes
the window right after a run, when the files are at HEAD and not in your checkout. yidam status --format json carries computed_files, computed_signals, computed_nodes and
computed_unowned.
A phase carries what it began from, and says so after an interruption
PHASES.md specifies a phase as a branch, a declared input state,
a body of agent work and a --no-ff merge. Nothing in a repository held any of it. yidam phases read a phase's state off its ref's namespace, so active meant a ref matching a
glob. A phase whose run died halfway through looked exactly like one opened this morning.
yidam phase start outcome-axis --type Investigation
git switch phase/outcome-axis
yidam phase run
yidam phase settle
start opens phase/<name> and commits the snapshot to .yidam/phases/<name>.yml. The
snapshot is the baseline commit, the digests of .yidam/capabilities.toml and
.yidam/config.toml, and the kuten revision. The declared --type is checked against the phase types the vendored kuten
declares. A repository holding no kuten records the type and validates it against nothing.
There is no default list, and inventing one would put a phase-type roster in this binary
(RFC-0028 §3).
The record is committed rather than written to the working tree, and that is what makes it
useful. yidam phases reads remote-tracking refs. A record living only on disk would answer
for the one branch you are standing on. The commit's verb is scaffold: — operational, so the
pipeline advanced and no understanding changed.
run resolves the manifest's plan and writes it to the record before the first step is
invoked. Each step's completion is committed as it happens. That order is the whole design. A
record that learned its plan on the way through could say which steps ran, and never which did
not. A step already recorded is not re-entered, so a run killed partway completes rather than
restarts.
The plan holds only the steps this binary can invoke. A kind = "connector" is refused by
design, so it is left out of the plan. So is any step that runs after a refused one. The
report names each under refused, with its reason. If no step can run, nothing is committed.
The commits move the branch and not your checkout. The run prints the git restore line that
takes them, whether it finished or stopped.
Until every step of the resolved plan is recorded, the phase reads interrupted. That is true
in yidam phases, in yidam status and in yidam due's phase clock. The state has no ref
shape — only the record knows it.
settle refreshes the REGEN blocks, checks the phase produced outputs and drafts the
merge subject. Without a flag it commits nothing. phase: is an epistemic verb. due
states who decides a phase is done: a person. Merging a phase, or abandoning it, is not a
mechanical consequence of a finding. So settle validates and hands back the commands to run.
settle --merge writes them, once you have decided. The regen: commit and the --no-ff
merge are authored by yidam phase, with you as committer. Then it switches to the baseline
and deletes the phase branch. --summary "<text>" replaces the drafted tally in the subject.
Nothing is pushed.
It refuses anything it would have to resolve. The baseline must be an ancestor of the phase.
The checkout must be the commit, apart from the blocks settle staged. The baseline must not
be checked out in another worktree. Each refusal says Nothing was merged. It never merges a
propose/* branch. RFC-0026 §2 and PHASES.md state the
licence.
The REGEN step is regen --check, then regen, then regen --check again. A stale block is
refreshed and staged. The printed sequence then opens with the regen: commit, because
git merge --no-ff refuses a dirty index. A block still stale after the refresh, or one no
generator writes, is not ready, with the remedy regen --check prints. Nothing is refreshed
from a checkout that differs from HEAD. A block generated from that tree would not hold
against the commit. settle names the files and the sync instead.
settle also asks one question of the template. Did the phase add files outside the laid-out
directories, and cite no issue on the template's tracker? Then it drafts a from:derived-repo
report and prints the gh issue create line. Nothing is filed. The laid-out directories are the
sections of guidelines/directories.md.
Like run and propose, all three write git objects and one ref, except settle --merge. They touch neither the
working tree nor the index, so they are safe mid-edit. They therefore leave your checkout
behind the branch. git reset --hard HEAD or a path-scoped git restore syncs it.
A phase opened by hand is not broken. git switch -c phase/<name> is PHASES.md's own
documented flow, and every phase in every repository predates this record. Those rows are still
listed, and their state is still read from the ref. The table says how many rows are an
inference. RFC-0028 §3 ranks the two sources rather than replacing one with the other. The ref
answers what is this ref; the record answers what happened in this run. Only the ref can say
whether a phase has landed on the baseline.
A run on a cluster is the same run, and the ref write is a credential
cluster-runs.md is the deployment guide. This is the command surface.
yidam cluster workflow --cron "0 6 * * *" > yidam.cronworkflow.yml
yidam cluster network-policy --executor 10.0.0.1/32 > deploy/corpus.netpol.yml
yidam cluster admit --remote git@host:corpus.git
yidam cluster pin --remote git@host:corpus.git --vault-url file:///var/yidam/vault --out pin.json
yidam cluster step travel-tier --bundle <digest> --vault-url file:///var/yidam/vault --out step.json
yidam cluster land --step-output @step.json --remote git@host:corpus.git --vault-url file:///var/yidam/vault
yidam cluster survey units --bundle <digest> --vault-url file:///var/yidam/vault --out survey.json
yidam cluster ask --ask '<one of survey.asks>' --vault-url file:///var/yidam/vault --out ask.json
yidam cluster gather units --bundle <digest> --asked '[…ask records…]' --vault-url file:///var/yidam/vault --out step.json
yidam cluster status --remote git@host:corpus.git
workflow reads .yidam/capabilities.toml and [cluster] in .yidam/config.toml, and
prints an Argo Workflow. With --cron it prints a CronWorkflow that admits itself. With
--on-push it also prints an Argo Events EventSource and Sensor. A push to the branch
then submits the run, which admits first. It
runs in a checkout and writes nothing. Every flag it takes overrides one [cluster] key.
--image names the container image every pod runs.
network-policy prints the NetworkPolicies that limit what each kind of pod reaches.
Every pod reaches DNS and the API server.
remote pods reach the remote. vault pods reach the vault and nothing else.
internet pods reach anywhere but the remote.
A policy matches addresses, so each place is a CIDR list from [cluster.egress].
Each flag replaces its key, and each may be repeated:
| Flag | Replaces | What it is |
|---|---|---|
--executor <CIDR> | [cluster.egress] executor | The API server's endpoint addresses. The generator refuses without them |
--remote-cidr <CIDR> | [cluster.egress] remote | Where the git remote is |
--vault-cidr <CIDR> | [cluster.egress] vault | Where an s3:// vault's endpoint is |
--sts-cidr <CIDR> | [cluster.egress] sts | Where STS is, for pods that assume a role |
--vault-url overrides the vault's url, as it does for workflow. Apply the output with the
corpus's overlay. cluster-runs.md says what each
address should be.
admit clones the branch and reads due's clocks, the manifest's staleness and the count
of open propose/* branches. Its record says admitted either way, and it exits zero either
way. A corpus with nothing owed is not a failure. The generated workflow submits no step when
admitted is false.
pin bundles refs/heads/<branch> and puts the bundle in the vault. The record names the
sha and the bundle's digest. Everything downstream reads the pin, never the remote.
step fetches a bundle, clones it into scratch and invokes one capability there. It is
run's loop body with the ref write removed. What it builds is a commit in a second bundle,
and its record carries {sha, class, verb, receipt, bundle}. The command has no --remote and
no --branch. That is the point of it, and a test holds it.
Its --image is the reference the pod started from. The receipt records it only when it pins a digest.
land reads a step record and verifies the bundle. It reads the class off the commit, not
off the record. Then it pushes the commit to main or to propose/<input>, with
--force-with-lease. If the branch moved, it rebuilds the commit on the new tip. That holds
only when nothing the step reads moved. Otherwise it refuses. Its last act is a fresh pin,
which the next step reads. With --git-auth github-app it mints an installation token from
--github-app-key and pushes over HTTPS. The token goes to git's environment and nowhere else.
--github-app-id names the App. --github-api is GitHub's REST API, https://api.github.com by default.
For GitHub Enterprise Server, pass https://<host>/api/v3.
survey, ask and gather run a gather, one pod per peer. None takes --remote.
survey plans at the pin and translates each peer's query there. Its asks list feeds the fan-out.
ask takes the peer's bundle from the vault, or fetches its lock url when the vault lacks it.
It checks the bytes against the lock, runs the query, and always puts a record in the vault.
Like step, it records --image in the peer's receipt only when it pins a digest.
gather holds every record to the plan. A peer with no record is refused, never dropped.
It builds the tree a local yidam gather would, and land pushes it to propose/gather/*.
The lander refuses a gather commit that is not open:, or that touches other files.
The generated workflow adds these four tasks per file in .yidam/gathers/.
The records are the pod contract, versioned by format_version. Under --format json
each sits under a record key in the report envelope.
status reports what the recent runs did. It reads the records Argo keeps on each
Workflow, through kubectl or from --workflows <file>. For each run it gives the admission
and its reason. It gives each step's outcome: landed, proposed, refused with the lander's
reason, or not reached. It gives each gather peer's outcome. With --remote, or [cluster] remote, it reads the branch tip and says whether each proposal is still open. A run whose
records are gone is unknown, with the reason. It never reads a log, and it commits nothing.
Under --format json it is a report, not a pod record.
--limit sets how many runs it shows, newest first; the default is 5.
--namespace overrides [cluster] namespace, and --context picks the kubeconfig context.
--corpus is the corpus's name in the cluster's objects. It defaults to the root directory's name.
Every pod that reads the vault takes the same vault flags:
| Flag | What it is |
|---|---|
--vault <NAME> | The vault's name, which decides which credential variables are read. Default default |
--vault-url <URL> | file:///path or s3://bucket/prefix |
--vault-region <REGION> | The S3 region, for an s3:// url |
--vault-endpoint <URL> | An S3-compatible endpoint, for a store that is not AWS |
--vault-path-style | Address the bucket in the path rather than the host |
The generated workflow writes these from the config, so a person rarely types them.
When land refuses, it still writes a record to --out, holding refused and the reason.
The pod still fails. That record is what status shows.
Index and embeddings
| Command | What it does |
|---|---|
embed * | Extract embedding text from corpus instances to .yidam/embeddings/. --no-catalog, --dry-run |
index-build * | Build the vector index as Arrow IPC (corpus.arrow), which retrieve and the web shell read. --model. Needs vector-read, which is in the default set |
index-push | Mirror .yidam/index/ into the vector bucket [index.remote] declares. --dry-run, --create. Needs vector-read, which is in the default set |
embed walks .yidam/catalog/ by default. In a real derived corpus the catalog was 51.3% of
the indexable text, against the corpus's 41.9%. Leaving it out had been a scope decision nobody
made on purpose.
embed --dry-run composes every record and writes none of them — not the record, not
.yidam/embeddings/, not the directory. It reports what a push would carry: the composed
text distribution, split by node and source. It names the largest row against the 40 KB
per-vector ceiling. It names the largest filterable half against the 2 KB one. Any row whose
text would be cut is listed. Run it before declaring a remote index. It is also how
RFC-0033 §8.3 was measured, over corpora nothing was
written into.
index-push is a mirror. It writes every row the local index holds, and deletes the ones this
corpus no longer has. A node removed from the corpus stops being findable. It never touches a
record that is not this corpus's.
Several corpora may share one index. A push reports the ones it found there — shares: <corpus> (<n> row(s)) — from the listing it already reads. With --dry-run that is a
read-only way to ask an index who is in it. It is how a [index.remote.corpora] table gets
written.
One index holds one vector space. An index carries a single embedding contract, and every push overwrites it. So a push into an index built with different embedding settings is refused. Two spaces in one index would let a spanning query rank one corpus's rows against the other's. The scores would be in the range a correct ranking has. Where the contract cannot be read at all — a write-only credential, a throttle — the push says so and proceeds.
The index carries typed properties, and the push does not forward them. index-build writes
a properties column: a JSON object of each row's declared date and number values. A row
with none holds null, and meta.json lists the columns. That is what retrieve --where reads.
The push leaves the column behind, deliberately. A remote index refuses a where rather than
answering it from metadata that only sometimes fits (RFC-0041).
Artifacts
Bytes a corpus rests on or produces are large, derived, or licensed — a fetched source, a built index. Git is the wrong place for all three. A vault holds them; the repository holds the record of which bytes. RFC-0023 states the constraint: a vault stores bytes, git stores the record of them, . So every pointer into a vault is a committed file. Losing a vault costs no knowledge claim, only the time to re-fetch.
| Command | What it does |
|---|---|
vault list | Every store this repository declares — its audience, what it holds, how many artifacts route to it |
vault put <path> * | Hash a file into the local cache; prints the content address on stdout |
vault get <sha256> * | From the cache, else from the vault. --out also writes a named copy |
vault path <sha256> | Where the artifact sits locally, or exit nonzero — so … || fetch works |
vault verify | Re-hash every cached artifact; exits nonzero if any is not what it claims |
vault push * | Upload what the corpus names and the vaults lack. --dry-run prints the exact string that would be signed; --artifact and --vault narrow; --index/--embeddings/--bundle send what this repository computed instead, and --commit authors the index: or bundle: commit for the lock entry it wrote |
vault pull * | Fetch what the corpus names and the cache lacks; --vault narrows; --index/--embeddings/--bundle fetch and unpack what .yidam/index.lock records |
vault status | Where each named artifact goes and where it is, grouped by store. --remote asks each vault — one HEAD per record, never a bucket listing |
vault gc | Report cached artifacts no committed file names; --yes deletes them |
vault materialize | Hardlink cached artifacts into .yidam/vault/<slug>/ under names a person can open; --entry narrows |
vault-status * | Writes the <!-- REGEN: yidam vault-status --> block in README.md. Committed files only — never the cache, never the network. yidam vault status, a hyphen apart, is the read-only report; yidam regen runs this one with the rest |
catalog-fetch [entry] * | Follow a catalog entry's declared address, cache the bytes under their digest, record them in artifacts:, and commit it as refresh:. --location narrows to one address; --bind name=value fills a url_template slot; --archive adds the nearest Wayback capture of each identifier whose pack allows it; --dry-run resolves and writes nothing. Exits nonzero if a publisher refused a location |
catalog-extract [entry] * | Take each reading an artifact lacks — a PDF's text, a source pack's extract, or a zip member its location lists under members: — file it in the cache, record it under the artifact in readings:, and commit it as extract:. --dry-run reports and reads nothing |
catalog-reconcile [entry] * | Rewrite a drifted used-by list to the citations, which are authoritative, and commit it as reconcile:. --dry-run reports and writes nothing |
source check | Hold every source pack and every prelude_sources pin to the pack format, offline, and run each transform over its fixtures. Exits nonzero on an error |
source list | Print each enabled pack, its schemes, and what its transport needs: YIDAM_CONTACT and any auth variables, each marked set or not. Never prints a value |
source search <pack> <query> | Ask the pack's [search] endpoint and print the identifiers it answers with. --limit caps them; --offline answers from the pack's fixtures. Writes nothing |
source add <scheme:id>… * | Resolve each identifier through its pack and commit a draft catalog entry, obtained: false, as catalog:. --fetch then runs catalog-fetch on it; --offline answers from fixtures; --dry-run writes nothing |
Following an address
A catalog entry has always carried a machine-shaped address. The gate has always checked its
syntax: catalog-location-malformed fails a url_template with no {…} slot. Until #720
nothing dereferenced one. url_template appeared four times in the CLI — twice in the type,
twice in that check.
catalog-fetch is the path between them. Four steps: address → bytes → cache → record →
commit.
$ yidam catalog-fetch local-registry --location 0
local-registry
location 0 — /repo/sources/registry-2026.csv
sha256:9f2c… (118 bytes)
push route: sources (s3://newsroom-sources/yidam)
a41f3c8 refresh: local-registry from sources/registry-2026.csv
The four location kinds are not equally followable, and the difference is reported rather than hidden:
| Kind | What a fetch does |
|---|---|
file | Reads it, relative to the repository root. No network, and no feature gate — this works in every build |
url | GETs it |
url_template | GETs it once every slot is bound. A partly-bound template is refused, naming the slots left |
address | Nothing. A records office is a real way to hold a source and not an endpoint; it is passed over in silence beside a location that can be followed, and explained when named by --location |
A template is refused rather than half-filled. A URL still carrying a literal {year} is one a
server answers with something. That something would then be committed as a digest this corpus
obtained.
$ yidam catalog-fetch local-registry --location 1
local-registry
location 1 skipped — `https://registry.example/…?year={year}` has a slot nothing bound:
`{year}`. Supply it — for example `--bind year=…`
Bindings are not inferred from the corpus. A gage node carrying properties.parameter: "00060"
looks like it could fill a slot of that name. It might. That has not been checked against more
than one corpus, so nothing guesses.
What lint holds a catalog entry to
Two checks read a catalog entry's shape. Both are Warn.
catalog-type-unknown reports a type: outside the closed set. The set is paper,
dataset, api, database, statute, report, standard, document and other. A word
outside it is reported, never mapped. The finding asks which of the four newer types it is.
primary gets one more clause: it is a standing, not a form. A primary source can be a statute,
a dataset or a report. Only someone who read it can choose, so no migration maps it.
catalog-location-malformed reports a location with no kind or no value. It also reports
a kind outside url, url_template, address, file and identifier. Several locations need a
description each. A value must match its kind:
| Kind | Refused |
|---|---|
url | A value that does not start http:// or https://, or that carries a # fragment |
url_template | A value with no {…} slot |
identifier | A value that is not scheme:local-id, has no local id, or is a URL |
Once the corpus enables a source pack, an identifier is also held to the packs. Its scheme must be
one an enabled pack declares, and its local id must match that scheme's pattern. When a template
pack declares the scheme, the finding names the pack to pin. A corpus that enables no pack is held
to the shape alone. See Source packs.
A fragment is never sent to the server, so catalog-fetch downloads the whole file. The finding
names that file. To read one file inside a zip, list it under members:. See
Read a file inside a zip.
kind: doi and kind: pmc were written before identifier existed. The finding for either names
yidam migrate locations, which rewrites it. See migrate subcommands.
Where a fetch stops
At the record, and at the local cache. Two boundaries, each closing a shortcut.
It never authors a node. That is establish:, which is epistemic. RFC-0026 confines those to a
person. A run authors operational commits directly. Every epistemic commit goes to a proposal
branch. refresh: and reconcile: are both operational, so both land on your current branch. The commit writer checks its own subject against classify_commit rather than trusting
its callers.
It never uploads. Bytes land in the machine-wide cache. Sending them is vault push, which
consults redistributable. That is a decision a person makes after looking at what arrived.
The record carries the digest, the size, the media type the server declared, the date, and the
location. On a first capture it omits two fields deliberately. vault: is left out because
routing is .yidam/config.toml's decision, and freezing it per record would make that routing
dead. redistributable: is left out because it is a licensing fact, and no HTTP 200 establishes
one.
On a re-capture of a location the entry already holds a record for, both are carried forward from that record. They are your statements, not the command's. Carrying one reproduces a decision you made about that source. Inventing one would assert a licence on the strength of a response code.
Only the latest record for that location is consulted. Change your answer and every edition after it inherits the new one. A record that says nothing carries nothing forward.
A re-capture with new bytes is a new version of the source. Every node citing the entry was
read against an earlier one. The report and the refresh: commit both name them, and yidam due
keeps naming each one until a commit touches it:
$ yidam catalog-fetch local-registry --location 0
local-registry
location 0 — /repo/sources/registry-2026.csv
sha256:4be1… (131 bytes)
push route: sources (s3://newsroom-sources/yidam)
c07d2e9 refresh: local-registry from sources/registry-2026.csv
a new version — 1 node(s) cite an earlier one, and `yidam due` holds them until each is re-read:
.yidam/corpus/record/hydrant-count.yml
"Of the source" means the same location and the same resource:. A location serving one file per
state, or an API asked at several endpoints, is many sources. Record which one each artifact is
with resource:. A new state's file is then not read as a new version of the others.
The refresh: commit body names each field it carried and the digest it came from. So a licence
appearing on a new record is reviewable as continued rather than established.
Without this, one licensing decision had to be re-entered by hand per edition. vault push
refused every re-capture of a source you had already cleared. A vault: none hold-back lapsed
the same way, routing new bytes to a store you kept the previous edition out of.
Re-running is free. A fetch that finds bytes the entry already records writes nothing and
commits nothing. That is what makes it safe to put on a ttl_days clock. A source that
changed appends a second record beside the first rather than replacing it. Overwriting would
delete the provenance of every claim resting on the older bytes.
Source packs
A source pack names a family of sources by identifier (RFC-0048). A catalog location
kind: identifier, value: doi:10.1167/tvst.8.5.14 names the scheme doi, and a pack declares
how that scheme resolves to an address. Source packs is the task page: pinning,
writing, every manifest key, and the transform contract. A pack is a directory:
<pack>/
pack.toml # schemes, resolution, transport, defaults, fixtures
entry.md # the body a new catalog entry starts from
transforms/*.glu # optional
fixtures/ # recorded responses
[pack]
name = "scholarly"
version = "0.1.0"
[scheme.doi]
pattern = '^10\.\d{4,9}/\S+$'
type = "paper"
resolve = { template = "https://api.crossref.org/works/{id}", media = "application/json" }
[transport]
contact = "required"
min_interval = "100ms"
[fixtures]
"doi:10.1167/tvst.8.5.14" = "crossref-tvst.json"
A template slot is {id} or a named group in the pattern. [fixtures] maps an identifier to
a file under fixtures/, and every file there must be named. A manifest refuses a key it does
not know.
A corpus writes its own packs in .yidam/sources/. It vendors others by pinning them in
.yidam/decisions/proposals.yml:
prelude_sources:
- scholarly@^0.1
- us-oh-legislature@^0.3 from github.com/<owner>/<corpus>@<40-hex commit>
mise run yidam-vendor-update copies each pinned pack into .yidam/.vendor/sources/. A pin
without from names a pack the template ships. One with from names a pack another
corpus wrote, at that commit. The copy records the pin and its origin under [vendored].
A range is ^X.Y, ~X.Y.Z, =X.Y.Z or a bare X.Y, with cargo's bounds. A commit holds one
version of a pack, so a from range is checked, never resolved. The update refuses a pin the
pack's version does not satisfy and copies nothing. An authored pack is read before a vendored
pack of the same name.
source check is the CI gate for an authored pack. It reads no network, and exits nonzero on an
error. --format json emits passed, packs and pack_findings; text is the default.
A scheme may name two transforms, describe and extract, as .glu files under
transforms/. They run in the calculator arm's closed prelude, over the response the scheme
resolves to. The binary parses that response as JSON, XML or CSV, by the scheme's media. A
describe returns a draft entry. An extract returns a reading of the artifact, which
catalog-extract records. source check admits each transform and runs it over the scheme's
fixtures. A then that reads describe.<key> must be answered by at least one fixture.
Transforms need the source-transforms feature. A build without it checks the rest of the
pack and says it did not check the transforms. catalog-extract reports each reading it could
not take as skipped.
A pack may declare one search endpoint. source search asks it:
[search]
scheme = "doi"
template = "https://api.crossref.org/works?query={query}&rows={limit}"
media = "application/json"
items = "message/items/*"
id = "DOI"
title = "title/0"
fixtures = { "retinal imaging" = "crossref-search.json" }
Each field matching items is one result, and * matches any one name. id and title are
paths below it, named the way a transform names them. A search is read by path, so it needs no
transform. JSON is read in every build. XML and CSV need source-transforms.
source add writes a draft entry for each identifier and commits each as catalog:. It
follows the scheme's then, and adds each identifier found as another location. Where the
build runs transforms, describe fills the title, type, date and description. The entry is
obtained: false, with the pack's ttl_days. Its body is the pack's entry.md, or a
template of eight prompts. source add never writes a node and never writes
obtained: true. It refuses an identifier the catalog already holds. If one identifier fails,
it writes nothing.
source search and source add ask a publisher the way catalog-fetch does, under the
pack's [transport], described below. A pack missing its contact or a credential is not
asked. --offline answers from the pack's recorded fixtures and asks nothing.
Fetching an identifier
catalog-fetch resolves a kind: identifier location through the enabled pack that declares
its scheme. It then fetches the address as that pack's [transport] says:
[transport]
contact = "required"
min_interval = "250ms"
auth = [
{ env = "COURTLISTENER_TOKEN", header = "Authorization", prefix = "Token " },
{ env = "BLS_KEY", query = "registrationkey" },
]
archive = "wayback"
- Contact. Every request sends
User-Agent: yidam-catalog/<version>, plus(+$YIDAM_CONTACT)when that variable is set. A pack withcontact = "required"is skipped without it, before any request. - Auth. Each credential is read from the named variable and sent in the header or query parameter the pack names. A bare name is refused, since it says nothing of where to send it. No credential reaches the report, the entry or the commit. A credentialed request follows redirects only on its own host.
- Spacing.
min_intervalholds between two requests to one host, across one run. - Atomic write. The body streams into a
.partialfile and is renamed once complete. - Refusals. A non-success status is recorded under
refusedon the entry, and the run exits nonzero. It is not retried. The entry's other locations are still fetched and recorded.
An identifier in a scheme no enabled pack declares is passed over, as before packs existed.
What the host will not do. It sends no browser User-Agent and takes no step around a 403. A
pack for a publisher that refuses automated clients says blocked = "<why>". Its identifiers are
refused with that reason. Record a manual export as a kind: file location instead. The same
holds for a publisher whose TLS chain does not verify: no pack can turn verification off.
Archive. wayback:<timestamp>/<url> resolves to the capture's id_ form, the original bytes.
catalog-fetch --archive looks up the nearest successful capture of each identifier whose pack
says archive = "wayback". It appends that capture to the entry's location: list as a
wayback: identifier. Its description carries the capture date, and it lands in the same
refresh: commit. It reads the
Wayback availability API and never asks the archive to capture anything. It needs an enabled
pack declaring the wayback scheme.
Reading a PDF
A quotation of a PDF is compared with a text reading of the PDF. Lint never extracts text
itself. Run catalog-extract once, on the machine that holds the PDF:
- Run
yidam vault pullif the cache lacks the PDF. - Run
yidam catalog-extract <entry>. - Run
yidam vault pushto share the reading with the other machines.
The command records the reading's digest, media type and extractor under the PDF's record, in
readings:. A record that has a reading keeps it. To take a new reading, delete that reading
and run it again. An entry written before readings: holds its reading under text:, which is
still read.
A scanned page has no text layer. The command refuses a reading with no letters or digits. It reports that PDF as skipped.
Reconciling a used-by list
catalog-audit reports the disagreement between an entry's hand-maintained used-by and the
nodes that actually cite it. It reconciles none of it. catalog-reconcile is the other half.
It can be mechanical because one of the two lists cannot be wrong. catalog-used-by-drift says
so of itself: the citations are authoritative — they cannot drift from the corpus, and a
hand-maintained list can. So this is a substitution, not a merge.
An entry declaring no used-by is left alone. Absence is not drift, and the gate stays silent
about it. Writing one would be a command deciding an entry should make a claim it never made.
An explicit used-by: [] is a different state, and both the gate and this command read it as
one. An empty sequence is something an author typed — it asserts that nothing cites the entry.
When nodes do cite it, that assertion is false, so catalog-used-by-drift reports it and
catalog-reconcile substitutes the citations for it. An entry genuinely cited by nothing
agrees with its own empty list and produces no finding.
Every artifact is named by the SHA-256 of its bytes, in lowercase hex. The cache is
machine-wide — $XDG_CACHE_HOME/yidam/vault, or YIDAM_VAULT_CACHE — so two repositories
citing the same source store it once. It is deliberately not partitioned by vault. A cache hit
answers do I have these bytes, never may I send them.
list, get, push, pull and status read .yidam/config.toml and need a repository.
put, path and verify touch only the cache and work anywhere.
[vault.default]
url = "file:///mnt/archive/yidam"
audience = "Who can read this store, and why that is acceptable."
audience is required and nothing can check it — it is .yidam/publishable's argument applied
to a store. What is enforced is that somebody wrote one.
A lone vault may be called anything and, if it declares no holds, takes everything. What a
name other than default gives up is the ambient AWS_* fallback below.
Naming vaults, and routing between them
A repository's own index and a licensed PDF it obtained have different readerships, and one
store cannot express both. Each vault declares what it holds:
[vault.default]
url = "s3://corpus-artifacts/yidam"
audience = "Anyone who can read this corpus. Derived output only."
holds = ["index", "embeddings", "bundle"]
[vault.sources]
url = "s3://licensed-sources/yidam"
audience = "The sangha. Documents obtained under a licence to read, not to host."
holds = ["catalog"]
The kinds are catalog, index, embeddings and bundle. Only catalog artifacts exist
today. The others are named so a corpus can declare their routes before #417 and #418 start
writing them. That avoids reorganising storage on the release that does.
one vault, no holds | it holds everything |
| two or more | every one declares holds — a vault claiming nothing would be a catch-all, and routing by default is how a licensed document reaches the public store |
| a kind claimed by two | refused, naming both |
| a kind claimed by none | refused when something of that kind needs a route, naming the kind |
a holds entry that is not a kind | refused — a typo claims nothing |
a record's own vault: | overrides the route its kind would take |
A kind nobody claims is refused at the artifact rather than when the config is read. The alternative would make the list of kinds a compatibility surface. Adding one in a later release would turn every multi-vault config red for a kind those corpora have none of.
--vault <name> on push, pull and status is a narrowing flag. It is useful where one
store is reachable from a runner and another is not. It never re-routes. An artifact routed to
sources is not pushed to default because somebody typed a flag. Moving an artifact between
stores is an edit to its record, in a commit. That is how every other assertion the repository
makes is done. Stores are opened lazily, one per vault that has work. A vault whose credentials
are absent does not block a push to one whose are present.
S3-compatible stores
[vault.default]
url = "s3://corpus-artifacts/yidam"
region = "us-east-1" # signing scope; MinIO wants one too
endpoint = "https://s3.example.net" # omit for AWS
path_style = true # defaults true when an endpoint is set
audience = "Anyone who can read this corpus."
Credentials come from the environment only — .yidam/config.toml is committed and must
never carry one:
| Variable | For |
|---|---|
YIDAM_VAULT_<NAME>_ACCESS_KEY_ID | that vault, always |
YIDAM_VAULT_<NAME>_SECRET_ACCESS_KEY | that vault, always |
YIDAM_VAULT_<NAME>_SESSION_TOKEN | temporary credentials |
YIDAM_VAULT_<NAME>_ROLE_ARN / YIDAM_VAULT_<NAME>_WEB_IDENTITY_TOKEN_FILE | that vault, assuming a role by web identity |
YIDAM_VAULT_<NAME>_ROLE_SESSION_NAME | optional, with a role |
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY | the vault named default, and no other |
AWS_ROLE_ARN / AWS_WEB_IDENTITY_TOKEN_FILE | the vault named default, and no other |
YIDAM_INDEX_ACCESS_KEY_ID / YIDAM_INDEX_SECRET_ACCESS_KEY / YIDAM_INDEX_SESSION_TOKEN | the remote vector index [index.remote] declares |
YIDAM_INDEX_ROLE_ARN / YIDAM_INDEX_WEB_IDENTITY_TOKEN_FILE / YIDAM_INDEX_ROLE_SESSION_NAME | the remote index, assuming a role by web identity |
A vault's own variables are tried first: its role, then its keys. Setting both is refused.
Then, for default only, the AWS_* keys, then AWS_ROLE_ARN.
The remote index follows the same order under YIDAM_INDEX_. It may fall back to AWS_*, as
default does. A corpus declares at most one remote index, so there is no second store to
confuse it with.
A role is exchanged with STS's AssumeRoleWithWebIdentity, as IRSA on EKS sets it up. The
token file is read on each exchange, and credentials are renewed before they expire. STS is
AWS_ENDPOINT_URL_STS when set, else the regional endpoint for AWS_REGION. The file is a
path the platform projects, never a key, so it counts as the environment.
The AWS_* asymmetry is deliberate. An ordinary AWS environment is plausibly already configured for
the store a repository publishes its own output to. A second vault exists because its readership
differs. That is the only reason to declare one. Letting it inherit whatever happens to be
exported is the failure the boundary was drawn to prevent.
A single PUT caps at 5 GiB, and multipart upload is not built. Over the cap the upload is
refused with a message that says so. It does not fail at the server as EntityTooLarge.
The artifacts this repository computes
.yidam/index/ is built by yidam index-build, and nothing keeps it in git. Building one is
minutes of embedding, so the vault carries it from the machine that built it to every other:
yidam vault push --index --commit # on a machine that has one; commits .yidam/index.lock
yidam vault pull --index # anywhere else; unpacks into .yidam/index/
--embeddings and --bundle do the same for .yidam/embeddings/ and .yidam/bundle.yiz.
These flags are an either/or, not an addition. vault push alone sends what the catalog
names; vault push --index sends the index and nothing else. An index is hundreds of megabytes.
--index quietly also uploading a corpus of papers would be a surprise in the direction nobody
wants.
An index is a directory and a vault stores one object. So a directory is packed into a single
archive and hashed as a whole. A corpus.arrow from one build beside a meta.json from another
is a corrupt index that nothing would notice. The archive is deterministic (sorted entries, zeroed
mtimes), so pushing an unchanged index does nothing. A .yiz is already one object and is stored
verbatim.
.yidam/index.lock
Committed, and the only thing that travels through git:
format_version = 1
[index]
sha256 = "9f2c8e…"
bytes = 41943040
vault = "default"
It names the store as well as the hash. A pull reads the store from there, not by
re-deriving it from holds. A holds edit made after the push would otherwise send the pull to
somewhere the bytes are not. That is a mutable ref wearing a lock file's clothes — the one thing
this design exists to avoid.
What push --index refuses
An index is not a file that happens to sit in .yidam/index/. It is a re-encoding of
everything walked to build it, and each row carries the node's text verbatim. So the question
is not whether the index may leave but whether everything it was derived from may:
| Artifact | Derived from |
|---|---|
index, embeddings | .yidam/corpus, .yidam/catalog |
bundle | those, plus .yidam/skills, .yidam/decisions |
A path .yidam/private-paths declares private that intersects one of those — in either
direction — refuses the push, naming the path. This is the rule
sadhana/github/workflows/release.yml already applies to a bundle, for the reason it gives:
the artifact outlives the access. A declared directory holding only a README.md or a
.gitkeep is intent rather than material and does not refuse.
Unlike a catalog artifact, a repository's own output is pushed by default. There is no
redistributable to set, because there is no third party whose licence it could be.
Reclaiming space, and opening a file
yidam vault gc reports cached artifacts that no committed file names. The live set is exactly
computable, because every pointer into a vault is a committed file. It deletes nothing until
--yes.
Read the list before you pass it. The cache is machine-wide and shared by every yidam
repository on this machine. An artifact another one names looks exactly like an orphan from here.
Usually deleting one costs a re-fetch. The exception is an artifact recorded vault: none, which
is in a cache and nowhere else by decision. For that one the cache is the only copy.
yidam vault materialize hardlinks cached artifacts to .yidam/vault/<entry slug>/<slug>.<ext>.
Content addressing is right for storage and useless for opening. Nobody wants to hand a colleague
9f2c8e… with no extension. The extension comes from the record's media_type, and an unlisted
type becomes .bin rather than a guess. A hardlink shares the bytes; a copy is the fallback when
the cache is on another filesystem.
It refuses to write until .yidam/vault/ is ignored. It asks git check-ignore rather than
grepping .gitignore. A repository ignoring it some other way is not reported as broken. A
licensed document in a tracked path is the leak push refuses, arriving through git add -A
instead.
What push refuses
vault push is the first egress channel yidam itself opens. An artifact must clear two
independent checks, and neither implies the other:
.yidam/private-paths— about this repository. An artifact whose record sits under a declared path is never uploaded, whatever its licence says. Same rule the release workflow applies to a bundle, for the reason it gives: the artifact outlives the access.redistributable— about the source. A catalog artifact is not pushed unless its record saysredistributable: true. A default of "upload unless told otherwise" would make the first push anybody runs a redistribution nobody chose. A catalog is full of papers.
Both checks are policy, not code in this
binary. disclose/record covers an artifact the catalog names, disclose/derived one this
repository computed. What they say is unchanged. A repository may state its own rule in
.yidam/policy/, and yidam policy check names it if it has.
Refusals are grouped by the store they were headed for, each under that store's own audience.
The reader learns what they were about to publish to, and with several vaults, which boundary
held. --artifact and --vault narrow what is sent and never bypass either check. A digest the
corpus does not record is refused, because it carries no redistributable and no path to check.
yidam doctor warns when two vaults resolve to the same credentials. That is legal, since one
account can own two buckets. It is also what a half-finished isolation setup looks like. The two
are indistinguishable from outside, so it reports the shape and lets the reader decide.
The rules this repository writes about itself
A gate's refusals are rules. A rule compiled into this binary is one the corpus it governs cannot
argue with. RFC-0024 makes the rule a committed file instead: git stores the rule, and yidam
evaluates it. The rules are Rego,
evaluated in-process — there is no daemon, no sidecar, and no network.
| Command | What it does |
|---|---|
policy check | Compile every rule; report each decision and whether it is inherited or this repository's own. Exits nonzero if a rule names a builtin this build does not carry |
policy eval --decision <name> | Ask one decision about one situation. Reads the input as JSON from --input <file> or stdin; --explain names the rule that fired |
policy gate <name> | Ask a decision about this repository and exit nonzero if it refuses. The form a workflow calls |
policy test | Run every test_* rule in every *_test.rego |
check, eval and test need no repository. The default policy is compiled in. Somebody
working out why a push was refused can ask without a checkout. gate is the exception. It builds
its input from the working tree, and there is nothing to answer about without one.
--format json emits the report contract on all four. It did not until #893 — the fields
arrived with no envelope around them, against RFC-0024's own stated dependency on RFC-0001.
If you read policy test's passed, it is now passing. This contract reserves passed
for a gate verdict, which is a boolean.
The decisions
The first family is disclosure — what this repository may let leave.
| Decision | Asks |
|---|---|
disclose/at_rest | May this material sit in this repository, given whether it is public? |
disclose/record | May these bytes be uploaded, given what their catalog record says? |
disclose/derived | May this computed artifact — an index, an embedding set, a bundle — be uploaded, given what it was built from? |
disclose/record and disclose/derived are separate because their evidence is. A record-bearing
artifact is judged by what its record says. A computed one has no record, so it is judged by what
it encodes. Neither consults whether the repository is private, and at_rest does — the artifact
outlives the access.
Routing is not a policy decision. Which vault an artifact goes to is vault list's question. Whether it may go at all is this one. The two are kept apart
because they fail differently. A route is edited casually by somebody reorganising storage. A
licence is not something that edit is allowed to undo.
Overriding a rule
Write .yidam/policy/<name>.rego declaring the same package. That rule then decides —
including by being more permissive than the default. RFC-0024 records the argument for and
against that.
An override is never silent. policy check names it and the file it came from. policy test runs
the inherited cases against your rule, and reports which expectations it no longer meets:
$ yidam policy test
ok test_no_overlap_permits
changed test_a_private_path_beats_a_licence
changed test_silence_is_not_a_licence
10 passed, 0 failed, 5 changed by an override
Those are not failures; a repository is entitled to decide. They are the list to read before
concluding the override says what you meant. policy check compares text. Whether a rule is
more permissive than the one it replaced is a question about every possible input. Nothing here
claims to have answered it. The changed list is the closest thing to an answer.
A repository's own *_test.rego failing is a failure, and it exits nonzero.
An override is also reported by yidam lint as policy-override at Info. It therefore reaches
the JSON report and the editor. yidam doctor reports it too, and is where a policy that does not
compile is caught, as a fail. Neither gates: the repository decided. What neither permits is
deciding quietly.
Export
| Command | What it does |
|---|---|
export * | Export the domain model. --format, --out, --list, --force, --prefix, --allow-breaking, --with-corpus |
bundle * | Alias for export --format bundle, kept for compatibility (no flags) |
schema * | Emit JSON Schema for the corpus shapes into .yidam/schemas/. A file it did not write is left alone, and the run refuses; --force replaces it. --settings prints the editor yaml.schemas mapping instead |
yidam export --list reports each format and its implementation status in your build. That is
the reliable answer, because two of the nine are feature-gated:
| Format | Produces | Needs |
|---|---|---|
bundle | .yidam/bundle.yiz — corpus, ontology, skills, decisions, index | default |
web | Feeds for the browser shell. --webllm-model names the chat panel's model | default |
graphml | GraphML for graph tools | default |
llms | A flattened corpus for a context window, on stdout unless --out names a file. --token-budget | default |
rdf | Turtle and JSON-LD. --rdf-format picks one | export-graph |
sqlite | SQLite + sqlite-vec | export-sqlite |
warehouse | warehouse/: the ontology as a BigQuery contract (contract.json and migrations under ddl/bigquery/), plus one load script per table for each feed under .yidam/warehouse/feeds/ (feeds/<feed>/<table>.sql). The first export writes 0001_init.sql. Each later one diffs against the contract.json it left. An additive change writes the next numbered migration, and no change writes none. A breaking change (a removed, renamed or retyped column, a removed value, a tightened or newly required column) is refused; --allow-breaking writes it to ddl/bigquery/breaking/<n>.sql for review. A feed that does not fit the contract is refused, and classes no feed maps are listed. A file edited since the last export is refused; --force overwrites it, except a migration, whose edit it records and keeps. A file it did not write is left alone. --with-corpus adds the feed corpus: each table's rows as feeds/corpus/<table>.ndjson with a BigQuery schema file, its .sql, and a load.sh that stages and merges them. A row's id is the node's reference. A node the contract cannot place is refused, naming it. Without the flag, no instance data is written. See RFC-0049 | default |
textql | textql/: the same contract as Markdown and CSV for a TextQL Ontology folder. It writes a README.md navigation table, warehouse.md (provenance columns, feeds, date precision), one classes/<class>.md per class and one vocab/<class>__<property>.csv per value set. It writes no ANA.md; it prints the row to add there, under --prefix (default: the output folder's name). The ownership rules are warehouse's, and --force works the same way | default |
A directory that is not a corpus is refused, not exported empty. export, bundle and
schema check for .yidam/ first. schema --settings is the exception: it prints a
compiled-in mapping and reads no corpus.
Serving the domain computer
| Command | What it does |
|---|---|
serve --mcp | MCP over stdio — the agent surface. See Connecting an agent |
serve --mcp --http | The same server over HTTP, for a client that takes a URL rather than spawning a process. --bind (loopback by default; an address off loopback with no token prints a startup warning naming every tool it serves), --port, --allow-origin, --token-file (or YIDAM_SERVE_TOKEN) to require a bearer token. --bundle serves a .yiz read-only. /healthz and /readyz need no token |
serve --lsp | LSP over stdio — the editor surface. See Editor setup |
record | What serve --mcp was asked, read from the record [serve] record keeps. Reads only, and exits 0 |
record --fold | Commits the record's new lines into .yidam/consumption.json as one refresh: commit. Commits nothing when nothing is new |
--root <DIR> matters most here, and serve is where it started (#421). A client
configures a command line, not a working directory. Without the flag the corpus depends on
wherever the client happened to spawn the process. That is what
Connecting an agent used to document a sh -c 'cd … && exec …' workaround for. A directory that is not in a corpus is refused, not served
empty.
Both transports are in the default build, and so is vector-read. It upgrades MCP's
retrieve from keyword to semantic search and adds nothing else. A build without it still
serves every other tool. It says degraded on the calls where the difference shows.
serve --mcp writes nothing unless the corpus says to. With [serve] record = true in
.yidam/config.toml it appends one line per tools/call to .yidam/record/calls.jsonl. The line
names the tool, a digest of its arguments, the row count and the latency. It also names whether
retrieval was degraded, and the corpus commit it answered from. Never the query text.
.yidam/record/ must be gitignored and the server refuses to start until it is. See
Configuration.
yidam record reads that file. It prints the calls per tool, then one sentence per
question the record answers. Those are the empty answers, degraded retrieval, act-tier calls,
and tools never called. It also names the corpus nodes no recorded retrieve returned. A missing file is reported as nothing was recorded. That is not the same
as nothing was asked, and the command says which one it found. It never gates: the file is
gitignored, so a fresh clone or CI runner does not have it.
Measuring the corpus
| Command | What it does |
|---|---|
bench | The committed goal set: anchored traversal against flat retrieval. --budget, --scaling |
--scaling measures the arms that are functions of N, over generated corpora rather than this
repository's own. It needs no index. The flat arm is constant in N, and is excluded by argument
rather than by omission.
Deriving and maintaining a repository
| Command | What it does |
|---|---|
init * | Write the smallest runnable corpus into the repository you are standing in. --class (repeatable) |
clone <target> * | Copy the template into a new directory and git init it. Target must not exist |
overlay <target> * | Add yidam infrastructure to an existing git repo. --backfill, --backfill-ref |
backfill * | Write a decision record for each epistemic commit in history. --since |
tonpa <sub> * | Manage bundle dependencies in .yidam/tonpa/. Needs tonpa (a default) |
init is the one of these that runs anywhere. clone and overlay copy the template out of
the yidam checkout the shell is standing in. In your own project neither of them does anything.
That was the whole of #1035.
init reads no template. It writes a class file per --class, one example node in each, and a
catalog entry. --class defaults to concept and observation. The tree it leaves passes
graph-check and lint as written. Every placeholder says what to replace it with. Each node
carries an [open] claim, so the first open-questions answers with something.
Two things it deliberately does not do. It does not run git init. A corpus is a git history
with a gate over it. Creating one where you are standing is a larger thing to do unasked.
init refuses and names the one line instead. It does not commit either. The first commit's
subject is a genesis: naming your domain. export and bundle read it back later, and only
you know the domain.
It refuses a tree that already holds a .yidam/, the way clone and overlay do. To add a
class to a corpus that exists, write its .ont.yml beside the others.
first-corpus-by-hand.md explains the shape init writes.
clone copies everything except the top-level paths NOT_INHERITED names in cmd/clone.rs.
Two of them are docs/ and examples/. The documentation here describes yidam itself. An example
is a whole foreign corpus that a new repository should not be born holding. The rest is machinery
this repository runs against itself. Also excluded: its installer, release and render scripts, its
packaging and its .github/. So are its plugin manifest, its editor launch configuration and its
demo shell.
A derived repository can neither run nor update any of it (#807).
overlay refuses a target that already has a .yidam/. It adds yidam/, sadhana/,
BOOTSTRAP.md, mise.yidam.toml and the .yidam.toml pin, without touching the repository's own
content. It also adds samudaya/, but only when this checkout holds seeds of its own.
Each subtree it copies is what git tracks under it. That is the answer clone already gives for
the whole template. A file this checkout does not track is not part of yidam. Build output, install
prefixes and editor caches therefore have no route into a repository that already holds your work.
Both commands need a git checkout of yidam as their source. Neither falls back to walking the
directory when it cannot read a tracked set.
backfill's classification is heuristic — it reads leading verbs. It does not extract
corpus nodes, and the records it writes are a starting point for a person, not testimony.
tonpa subcommands
| Subcommand | Effect |
|---|---|
add | Add a dependency, fetch it, and update the lock file. ../sibling@<sha> pins a path dependency to a commit |
install | Install everything declared in .yidam/tonpa.toml |
remove | Remove a dependency and delete its installed files |
list | Installed packages with node counts and model info |
status | Per-dependency: installed / missing / stale |
verify | Check installed bundle hashes against the lock file |
update | Re-fetch one or all dependencies to the latest bundle. A pinned path moves to the sibling's HEAD |
Sharing a derivation covers publishing a .yiz and consuming one,
and what a cross-corpus citation is and is not.