Sharing a derivation
How one derived corpus is published, how another consumes it, and what changes — and pointedly does not change — once it has.
A yidam corpus is built by one repository's sangha, under one ontology, with one revision
history. Composition is what makes it useful to anyone else: a .yiz bundle leaves the
repository that built it and is read somewhere that did not.
That crossing is the whole subject of this document. Everything below is either a mechanism for making it happen or a limit on what it is allowed to mean.
Producer and consumer, side by side
| Producer | Consumer | |
|---|---|---|
| Declares intent | .yidam/publishable | .yidam/tonpa.toml |
| Runs | tag v* → release.yml | yidam tonpa add, install, update |
| Produces | bundle.yiz on a GitHub release | .yidam/tonpa/<name>/, .yidam/tonpa/tonpa.lock |
| Gate | the publish guard, then graph-check | yidam tonpa verify |
What a bundle is
A .yiz file is a gzipped tar archive. It is a corpus, laid out the way the repository
that produced it lays out its own. So the code that reads an installed dependency is the code
that reads local instances. A change to instance parsing cannot apply to one and not the other.
manifest.yml provenance and counts
corpus/<name>.ont.yml class schemas
corpus/<class>/<name>.yml instances
skills/<name>.md skills
decisions/<name>.yml decision records
index/corpus.md rendered instance table
index/graph.md graph integrity report
index/decisions.md rendered decisions table
index/skills.md rendered skills table
index/corpus.arrow Arrow IPC, if an index was present
index/meta.json vector index metadata, if an index was present
index/embed.config.json embedding reproducibility contract, if an index was present
manifest.yml carries bundle_version, commit, genesis, genesis_hash,
generated_at, domain, and counts of classes, instances, skills and decisions,
plus vector_index_model — which is null when the bundle carries no index.
Two of those name the first commit, and they answer different questions. genesis is its
ISO date, so it can order two bundles. genesis_hash is its full SHA. It is the only field
that says which corpus this is. Two corpora created on one day share a date. No two share a
root commit. A name is chosen and can be changed; this cannot.
genesis_hash is null when the corpus has no first commit of its own. It is absent
from any bundle built before the field existed. Two cases give null. A tree with no commits
at all. And a corpus that is a directory inside a repository, rather than one itself. Git
answers such a directory with the enclosing repository. That repository's genesis names
something else, and is shared by every corpus nested in it. Absence means the corpus is unidentified. It never means
zero, and a reader must not substitute a placeholder. One constant would name every such
bundle alike. That is the confusion the field exists to prevent.
Note what is not in the list: no sangha/, no catalog/, no git history. A bundle
carries the corpus's claims, not the record of who settled them or what they were checked
against. That absence is load-bearing and the section on citation returns to it.
The version contract
bundle_version is "1". It is incremented only on a breaking change: removing a
field, renaming a field, changing a field's type, or removing a file from the archive
layout. Adding a field or a new archive entry is not breaking.
genesis_hash is the first field added under that rule, and it shows where the rule stops.
Old readers are safe: they ignore what they do not recognise. New readers are not, because
they meet bundles written before the field existed. So the burden is theirs — every reader
of a manifest field must have an answer for its absence.
The obligation this places on a consumer is the usual one, and it is not optional: ignore unknown fields and unknown archive paths. A consumer that fails on an unrecognized entry converts every additive change into a breaking one. The version number then stops meaning anything.
The authority for this is cmd/bundle.rs's
render_bundle doc comment, which is where the layout is actually defined. If this document
and that comment disagree, the comment is right and this document is stale.
Publish a bundle
1. Opt in, in the repository
Create .yidam/publishable. Say in it who may read this corpus and why that is
acceptable.
The file is not a security control — anyone who can push a tag can add a file. It is a
statement of intent that lives in the repository and outlasts the person who made it.
Without it, git tag v1 && git push --tags would publish an entire corpus, and "I did not
know the tag did that" is not something anyone can take back after the first download.
2. Tag
sadhana/github/workflows/release.yml runs on v*. It also runs on workflow_dispatch,
which builds and guards without publishing — use it. A publishing workflow nobody has ever
run is a workflow that gets debugged for the first time on the day it matters.
The workflow fails closed at every step. A check that cannot establish that publishing is safe refuses to publish.
.yidam/publishablemust exist.- No declared-private material may enter the bundle. See below.
yidam graph-checkmust pass. Publishing a corpus that fails its own gate ships broken edges to every consumer, where they read as the producer's claims.yidam export --format bundle, attached to the release asbundle.yiz— that exact name, because that is what the consumer resolves.
The privacy guard, and why it is not ci.yml's
ci.yml already has a privacy job. The release guard is stricter, and the difference is the
point:
ci.ymlprotects material that sits in the repository, and permits declared-private material as long as the repository is private. That is the right rule for a checkout and the wrong one for an artifact. The artifact outlives the access.
So .yidam/private-paths is read again at release time, against the directories that actually
enter a bundle. A match fails the release whether or not the repository is private. Private
material inside a published bundle is one download away from anywhere.
Three of those directories the archive carries as files: .yidam/corpus, .yidam/skills,
.yidam/decisions. The fourth is .yidam/catalog, and it is not obvious — no catalog file
is in a bundle at all. A bundle carries index/corpus.arrow, and that index has a text
column. The embedding step composes that text from the catalog as well as the corpus. So a
private catalog entry's prose ships inside the archive while no catalog file does. It was
missing from the guard until issue #443, for exactly that reason: the omission looked correct.
The list is duplicated in the workflow, and a test compares it against
vault::derived_sources(Derived::Bundle) — the same question asked by yidam vault push --bundle. Two lists of one fact, each pinned only by itself, is how one of them silently stops
matching; that is what happened here.
An index is opt-in, by omission
export --format bundle includes a vector index only when one is already built and
committed, so publishing never builds one.
A corpus is useful to a consumer without an index. Requiring the ML stack in order to publish
would put sharing behind a toolchain nobody needs to read Markdown.
Consume a bundle
tonpa is in the CLI's default feature set, so every published binary carries it.
Fetched dependencies
yidam tonpa add org/repo # → releases/latest/download/bundle.yiz
yidam tonpa add org/repo@v1.2.0 # → releases/download/v1.2.0/bundle.yiz
yidam tonpa add https://…/bundle.yiz
add writes the declaration to .yidam/tonpa.toml, fetches the bundle, records its
sha256 in .yidam/tonpa/tonpa.lock, and unpacks it to .yidam/tonpa/<name>/. Commit
both files. The unpacked directory is a build product; the declaration and the lock are
the record.
| Command | Answers |
|---|---|
tonpa status | Is each declared dependency installed, locked, and intact? |
tonpa list | What is installed — node counts, embedding model, genesis date |
tonpa verify | Do the installed bundles still hash to what the lock says? Nonzero if not |
tonpa install | Bring the tree into agreement with the lock, fetching at the pinned hash |
tonpa update [name] | Re-fetch at the latest bundle and move the pin |
tonpa remove <name> | Drop the declaration, the lock entry, and the files |
The distinction that matters is install versus update. install fetches at the hash
in the lock; update fetches whatever is published now and rewrites the lock. A CI job
runs install and verify. A person decides to update.
A fetched dependency is stale by construction: it is whatever was published, until someone updates it. That is not a defect. It is the only property that makes a consumer's build reproducible when the producer's corpus is under active revision.
Path dependencies
A sibling repository, read where it sits:
yidam tonpa add ../sibling-corpus
Nothing is fetched, hashed, or locked — hashing a working tree that changes under you records
nothing. tonpa status reports it as [linked] and unpinned. install skips it, and
update declines it and says why.
This is the only form that supports a development loop. An edit in the producer is visible in the consumer without cutting a release. That is how you find out whether the ontologies actually meet, before either side commits to a version.
A name declared as a path dependency wins over an unpacked bundle of the same name. The path form is the one someone is actively editing. Silently preferring a stale unpacked copy would make an edit appear to have no effect. That is the one failure a development loop must not have.
Pinned path dependencies
A sibling repository, read at a commit:
yidam tonpa add ../sibling-corpus@6ca345d
This is for siblings with no releases that you still need to read reproducibly. add resolves
the commit to its full SHA and records it as rev in .yidam/tonpa.toml. It reads the corpus
out of the sibling's git objects at that commit. The working tree is never read. The corpus is
written to .yidam/tonpa/<name>/, where a fetched bundle would unpack.
The lock records the commit as the entry's identity. Its sha256 is a digest of the corpus
files, not of an archive. verify and doctor check that digest against the files on disk.
| Command | On a pinned path |
|---|---|
tonpa install | Writes the corpus back from the sibling at the locked commit |
tonpa update [name] | Moves the pin to the sibling's HEAD, in the declaration and the lock |
tonpa status | [ok], [not installed], [hash mismatch], or [pin moved] |
install needs the sibling checkout on this machine. A missing sibling fails that dependency
and not the others.
What changes once a dependency is installed
Deliberately narrow: readable and searchable, never an edge target.
Keyword retrieval spans dependencies. serve --mcp's retrieve returns foreign nodes
alongside local ones whenever it is answering from keyword search — which is every call in
the default build, and every call in any build over a corpus with no index. Each result
carries an origin field: the package name for a foreign node, and null for a local one.
null rather than absent, so a client testing the key never has to distinguish "this node
is local" from "this server is old and never said."
Vector retrieval does not. yidam embed reads .yidam/corpus/ and .yidam/catalog/
and nothing else, so an index holds this repository's own nodes, and a vector result carries
no origin at all. On a composed corpus, degraded: true is therefore the more complete
search — the one arm that sees the whole of what is installed.
It carries an id, and this page said otherwise for three contract versions. That was fixed
in #425. The vector arm resolves each row through the same find_node get_node uses. What
it does not carry is origin. That is the claim that still holds.
That is a gap, not a rule. Everything else in this section is an argued boundary — a foreign node may be read and never cited — and that boundary holds under either retrieval path. Nothing in it implies search should get narrower when an index is present.
The history is plain enough: both arms shipped together, and dependency-spanning was added
to only one of them, by the change that made an installed bundle readable at all. That
change's own opening line was that tonpa fetched a corpus and nothing read it — "not
serve --mcp, not graph, not neighbors, not the index". It repaired the first of those
and left the last. So until the embedding step learns about .yidam/tonpa/, a
dependency-aware agent should know which arm it is talking to. The handshake's
retrieve.vector says, before the first query.
A shared vector index is a different reach, and it is not this one. Several corpora may
push to one remote index, and retrieve's corpora argument searches across them (#835).
That is not dependency spanning. A corpus sharing an index is not installed here. Its rows
carry a corpus field and an identifier, and get_node can fetch neither. Two reaches, two
sets of fields. The gap above is unchanged: the embedding step still does not read
.yidam/tonpa/.
Foreign ids are qualified. A dependency's node is pkg::class/name, and get_node
accepts that form. An id a client is shown and cannot then fetch is a worse affordance than
not surfacing the node at all.
Its path points into the dependency — .yidam/tonpa/<pkg>/corpus/<class>/<name>.yml,
never into the local corpus. An agent that opens the file gets someone else's file.
Nothing else reads a dependency. Not graph, not neighbors, not graph-check, not
lint, not status. Traversal does not cross the boundary, and no local edge resolves to a
foreign node. Reports stay local, and that is a correctness property rather than a
limitation: a status or lint that silently counted another repository's nodes would make
every corpus metric in every derived repository meaningless.
Both halves are in the same build.
serve --mcpused to need--features indexwhiletonpawas in the light default set, so the binary that could fetch a dependency was not the binary that could read one. It is now — and note which way round that landed: keyword retrieval is the arm that spans dependencies, and semantic retrieval answers over the local index only. The default build carries both, so a corpus with an index trades that reach for semantic ranking, and one without keeps it. mcp-server.md says what an agent should do withorigin.
Worked example.
examples/journalism/declaresexamples/property/as a path dependency and asks its pre-publication question across both — see the investigative journalism walkthrough. The newsroom's open question is who owns the terminal site; the title research that would answer it has an unresolved gap of its own, and the finding stays[open]because a local claim may not rest on a foreign node.yidam/cli/tests/example_corpus.rsgates it.
The epistemic status of a cross-corpus citation
The graph model says an edge is a claim, and the constitution has articles about who may assert one. A citation into a corpus with a different ontology, its own electors, and its own revision history is not the same object as a local edge — and it does not become one by arriving as a side effect of a package manager.
So the boundary above is not an implementation stage. It is the answer:
- A foreign node may be read and retrieved. It is evidence an agent can consult.
- A foreign node may not be an edge target. A local claim cannot rest on it structurally.
- Foreign nodes carry their own links, for display. They resolve within their own corpus.
graph-check needs a rule for what happens when the far side moves or a pinned bundle goes
stale — and under this boundary it needs none, because nothing local depends on the far side
holding still. That is most of why the boundary is where it is.
Recall what a bundle omits: no sangha, no elector register, no resolution history. A consumer receives a producer's conclusions with none of the apparatus that made them accountable. A claim tag that means "this elector body checked this" in the producing repository means "someone's elector body checked something" here.
cites-external — a genuine cross-corpus edge — is deferred to its own argued change.
It wants an RFC and a rule for the far side moving. It does not want to be shipped because a
fetch already worked.
Working in a repository that has dependencies
Guidance for an agent; see also
prelude/guidelines/agent-conduct.md.
Check origin on every retrieval result — and know when there is none to check. A
result with a non-null origin is not this corpus's claim: it was settled by another
sangha, under another ontology, and this repository's constitution never governed it. A
result with no origin key at all is a vector result, which means the search never looked
outside this corpus; absent is not the same as null, and treating it as such is how a
foreign node would get read as a local one. get_node answers with origin either way.
You may reason from a foreign node. You may not cite it as though it were local. Put what you took from it in a local node, in this corpus's own terms, tagged at this corpus's own standard — and say in prose where it came from. That local node is the thing this sangha becomes accountable for.
A foreign node's claim tag is the producer's tag, not yours. [verified] there means
that corpus's electors accepted that provenance. It does not transfer. The rule that a
derived assertion travels only as far as the weakest claim beneath it applies across the
boundary too, and the boundary is a place where "weakest" is genuinely unknown — nothing in
the bundle tells you how that corpus adjudicates.
Two ontologies agreeing on a word is not agreement. Classes are named per-corpus. A
concept/risk in a dependency and a concept/risk here are two nodes that share a string.
Treat a shared name as a question worth investigating, not as an identity.
A stale dependency is a normal state, not a finding. It is pinned. If its currency
matters to a conclusion, read tonpa list for the genesis date and the pinned commit, and
say in the node which pin you read.
Related
- what-yidam-is.md — the graph model, and why an edge is a claim
- information-architecture.md — the corpus layout a bundle mirrors
- web-interface.md — the other bundle contract, for the web shell
- constitutional-governance.md — who may assert an edge
- post-genesis-measurement.md — why a report counting foreign nodes would be meaningless