RFC-0007 — The Python SDK index/feature layer
- Status: Implemented
- Track: I7
- Relates to: RFC-0002 (node-model unification), RFC-0005 (MCP tool contract), RFC-0006 (correctness reconciliation), RFC-0003 (feature-gated builds)
- Versioning layers touched: SDK + parity (
yidam-coreminor bump; theembed_configandcompose_embed_textparity runners) - Downstream reference case: Project BOSC (watermark-directory)
Status 2026-09-26 (#1031): two asks of three ship, the third is retracted, and this now reads
Implementedon an amended scope. The scope is amended in this block and nowhere else — the sections below are the document as filed, and rewriting them would erase what was actually proposed.Ask 1, the runtime-verifiable embed contract, shipped and is good.
embed_config.rsandyidam index-verify(cmd/index_verify.rs) exist, theverificationblock exists, and the silent-drift failure #129 described is now detectable — #536 later exercised exactly that path.Ask 2, one canonical text assembly, shipped — in a different shape than the Proposal names. It is
compose_embed_text, a parity function in all three SDKs held to twelve fixtures inyidam/sdks/parity/fixtures/compose_embed_text/. Two departures from the Proposal, both deliberate:
- It is not "mirroring BOSC's proven shape (label · description · class · salient-meta)". The class stays a column, as the Rust reference always had it: folded into the text it puts one identical phrase in every vector of a class, which pulls the class toward a point and distinguishes no member of it from any other.
_meta_bitsis answered by the ontology, not by a curated list. The judgement BOSC was making by hand — which fields carry meaning — is now two declarations a class writes.prose: true(#746) says which keys and properties a node's substance is in, andretrievable: true(#717) says which identifiers and codes belong in an embedding without being prose: a gage'sparameter: "00060"is what a query is typed in and is not somethingnode-too-longshould count.compose_embed_texttakes both lists and unions them. So the answer to "which fields" is per corpus and written down in that corpus, rather than curated once in whichever consumer got there first.It is one assembler and not the whole embedder. Composing the text is the half that needs no model to check, and it is the half the two implementations disagreed on. Turning that text into a vector stays where the weights are.
Ask 3,
build_index/update_index/query/index_statusin Python, is retracted — not deferred. It was filed to spare the next consumer a 285-line re-implementation, and the measurement since says the next consumer is not arriving: across 180 derived-repo sessions and 62,048yidam-bearing shell invocations,index-buildwas run 0 times andretrieve0 times, and 0 of 16 corpora hold a local.yidam/index/. A second implementation of an index nobody builds is not a cost avoided, it is a second thing to keep in step withembed.config.json. The reconciliation subtlety below is unresolved because nothing now depends on resolving it; it is written down so the day a Python index is wanted the question is waiting rather than rediscovered. See Open questions for each.
sdks/README.mdwas the other half of this change, and is trimmed with it. It typed the fullfeatures/index/pipelinesurface under "the machine learning layer" and assigned Python ownership of it — "embed_node/embed_corpus— the only place raw text → vectors happens", "Rust can query but Python builds". UnderAcceptedthat read as a plan; retracting ask 3 without trimming it would leave it a claim. The line citations in the Summary and Problem below point into the untrimmed README and are as-of-filing.This header has been wrong once before, in the other direction. It read
Implementedon the strength of ask 1 alone until #718 corrected it toAcceptedon 2026-09-17. That correction is why this one names each ask and its evidence rather than moving a word: a status is a claim about three deliverables, and a document that states it without stating them can be wrong for a year with every gate green.
Summary
yidam-core's Python README promises a machine-learning layer — features, index,
pipeline — and calls Python "where nodes become vectors" (sdks/README.md:456). None of it
exists. The package exports corpus, git, markers and nothing else (__init__.py:1-3), with
dependencies = [] (pyproject.toml:10). Because the specced surface is vapor, the first
consumer built its own: BOSC's watermark.site.yidam_index is a 285-line LanceDB +
sentence-transformers index that re-derives embed_node's text assembly, build_index, and
query, and had to invent a reconciliation story with its own /ask embeddings from
scratch. This RFC proposes implementing the specced surface — honoring RFC-0006's embed
reproducibility contract so a Python-built index is interchangeable with the Rust one — and
confronts the one place they cannot be: yidam's canonical weights are quantized ONNX, which
sentence-transformers cannot load.
Problem
The spec is complete; the code is empty. sdks/README.md:480-527 fully types
embed_node(node, model, include_claims), embed_corpus(graph, model, batch_size),
EmbeddingSet, build_index, update_index, query, index_status, and sync_index. It
even assigns ownership: "embed_node / embed_corpus — the only place raw text → vectors
happens" and "Rust can query but Python builds" (sdks/README.md:537-542). Yet
yidam_core/__init__.py exports only the three parity modules, and pyproject.toml declares
zero runtime dependencies. There is no features.py, no index.py, no pipeline.py. A
downstream author who reads the README and then imports the package finds a hole exactly where
the README is most confident.
The concrete cost is a 285-line re-implementation. BOSC needed a semantic index over its
corpus mirror to serve serve --mcp (RFC-0005). With nothing to inherit, it wrote
watermark.site.yidam_index:
node_text(yidam_index.py:93-99) is a hand-rolledembed_node— it assembleslabel · description · node_class · <salient meta>, and_meta_bits(yidam_index.py:54-90) curates which fields carry meaning (kind, roles, relationship, hypothesis, tags, aliases) and which are noise to drop (site,scope,lei,uei). This is precisely the "how do you turn a node into a string" judgmentembed_nodewas specced to own.YidamVectorIndex.build/.query(yidam_index.py:169-232) re-derivebuild_indexandqueryover LanceDB, cosine metric, batched embedding.- The whole reconciliation narrative in the module docstring (
yidam_index.py:9-24) — a table of three vector surfaces that must share how they embed but not what they index — is work the SDK forced onto the consumer because it shipped no opinion about embedding at all.
Two design constraints bound any implementation. First, sdks/README.md:626-627: "Do not
load the full corpus into memory to answer a query. That is what the index layer exists to
prevent." Second, the embed-reproducibility doctrine (embed_config.rs:15-17): a consumer that
cannot embed with the index's exact settings "must degrade to keyword search, not embed with
different settings." An empty Python layer satisfies neither — and BOSC, absent the contract,
silently violated the second (see the reconciliation section).
Proposal
Implement yidam_core.features, yidam_core.index, and yidam_core.pipeline to write the
same durable artifacts the Rust path writes, so the two are interchangeable.
embed_node / embed_corpus — one canonical text assembly. The Rust reference composes a
node's embed text in embed.rs:20-43 (compose_text): label, then description, then
Related: <link-target file-stems>., with class kept as a separate column, not in the
embedded text. BOSC's node_text (yidam_index.py:93-99) instead folds class into the text
and uses curated meta bits rather than link stems. These are two different embed-text functions
over the same corpus; even with identical weights they would produce different vectors. So the
first deliverable is a single canonical assembler — mirroring BOSC's proven shape (label ·
description · class · salient-meta, dropping structural provenance) — surfaced as a parity
function so Rust compose_text and Python embed_node are held to a shared TOML fixture (per
RFC-0001's report-fixture discipline and RFC-0006's parity harness). include_claims
(sdks/README.md:484) presupposes the SDK's Markdown CorpusNode (title/claims/links),
while both real embedders run on the YAML instance model (label/description/links); which
node type embed_node accepts is inseparable from RFC-0002 and must be settled there
first — this RFC assumes the unified node.
build_index / update_index / query / index_status over LanceDB. build_index must
write the Rust path's artifact set, not a private one. The Rust index_build
(index_build.rs:44-206) creates a LanceDB table named corpus (index_build.rs:19) with
schema (path, class, label, text, vector<FixedSizeList<Float32>>), then exports three durable
files: index/corpus.arrow (Arrow IPC, index_build.rs:166), index/meta.json (model, dim,
node_count, indexed_commit, index_build.rs:185-188), and index/embed.config.json
(index_build.rs:196-199). BOSC writes none of these — it uses a table named yidam_nodes
(yidam_index.py:46), a richer schema, and no sidecars. The Python build_index must emit the
canonical table name + core columns (additive columns like uri/claim_tag are fine) and
all three sidecars, so an index built by yidam_core.index.build_index is byte-compatible with
one built by yidam index-build — both consumable by serve --mcp (RFC-0005) and by the
export_web / export_sqlite exporters, which copy vectors and never re-embed. query reads
embed.config.json to embed the query with the index's settings; index_status returns
meta.json plus a staleness verdict against the corpus HEAD commit.
sync_index — the stale-detection pipeline. sync_index(corpus_root, index_path, model, force) (sdks/README.md:520-527) is the high-level pass the README reserves for Python:
compare meta.json.indexed_commit against the corpus's current commit and per-node content
hashes, embed only the drifted nodes, update_index, and return a SyncResult
(added/updated/removed/unchanged counts). BOSC's index has no incremental path — it always
rebuilds whole (yidam_index.py:169-174) because its mirror is small; sync_index is the
generalization it skipped.
The dependency story. These modules move yidam-core off dependencies = [], but the
parity primitives must stay installable with zero native deps. Keep the split the package
already uses for sentence-transformers (pyproject.toml:14-16) and widen it into an
index extra:
[project.optional-dependencies]
index = ["sentence-transformers>=3", "lancedb>=0.13", "numpy>=1.26"]
import yidam_core.corpus stays dependency-free; import yidam_core.index raises a crisp
"install yidam-core[index]" if the extra is absent. This preserves the README's own layering
— corpus/git/markers are the parity core, features/index/pipeline are the ML layer
(sdks/README.md:454-463).
The reconciliation subtlety
yidam's canonical index is fastembed with quantized ONNX weights: embed.config.json
records model_file: onnx/model_quantized.onnx (embed_config.rs:28-32), and that field is
load-bearing — "quantized and fp32 exports of the same model differ by ~1e-3 per element, far
beyond retrieval-safe tolerance." sentence-transformers loads fp32 and cannot load the
Xenova quantized weights. So a Python-built index is not automatically in the Rust index's
vector space. This is not hypothetical: the parity suite already anticipates it — parity/README.md:76-78
says "a runtime that cannot load the exact weights in input.model_file declares its measured
drift in a [known_delta.<runtime>] section." BOSC embeds with sentence-transformers/all-MiniLM-L6-v2
fp32 via get_provider (yidam_index.py:41,262) and never reconciled against the quantized
space — a silent violation of the degrade-not-re-embed doctrine that nothing surfaced.
This RFC does not paper over it. Two honest paths, and the layer must pick one per build, not silently:
- Target the same weights. Have
embed_nodeload the quantized ONNX (viaoptimum/onnxruntime, not fp32 sentence-transformers) so the Python index lands in the Rust space andembed.config.jsonmatches byte-for-byte. Then a Python and a Rust index are truly interchangeable. - Declare a distinct space. If the Python build uses fp32, it writes an
embed.config.jsonwhosemodel_file/fastembed_model_enumreflect that, and RFC-0006'sindex-verifysurfaces the mismatch instead of letting a query silently retrieve against the wrong space. The Python parity runner (python/tests/parity/test_embed_config.py,parity/README.md:74) records the measured fp32-vs-quantized drift as its[known_delta.python]tolerance.
Either way the invariant holds: no index claims compatibility it does not have. query
refuses to run a query embedded under one embed.config.json against a table built under
another.
Migration & compatibility
This is an additive SDK-layer change: yidam-core gains three modules and an index extra —
existing importers of corpus/git/markers are untouched, so a minor bump (0.1 → 0.2).
No template or bootstrap version moves.
BOSC migrates watermark.site.yidam_index onto yidam_core.index by deletion, not rewrite:
node_text → the canonical embed_node; YidamVectorIndex.build/.query → build_index /
query; build_yidam_index (yidam_index.py:245-272) becomes a thin call into
sync_index. What BOSC keeps is its integration-specific reconciliation — the three-surface
table (yidam_index.py:9-24) that keeps this index distinct from data/cache/lancedb/ and the
/ask feed. The SDK owns how to embed and index; BOSC keeps owning what it points at
get_provider. If BOSC adopts path (1) above it gains Rust-index interchange for free; if it
stays on fp32 (path 2) it inherits the index-verify signal it never had, closing the live
silent-drift gap.
Alternatives considered
- Python shells out to the Rust binary for embeddings (FFI or subprocess). Honors "Rust is
always the reference" (
sdks/README.md:633) and sidesteps the quantized-weights problem entirely — one embedder, one space. But it makesyidam-core[index]depend on a built, unpublished Rust binary (RFC-0003), which defeats "no API call required, embeddings live in Python" (sdks/README.md:531,537-542). Reasonable as a fallback embedder, not the default. - Python never builds indexes; only queries them. fastembed is Rust-native and quantization
is a Rust strength; let Rust own
build_indexand give Python onlyquery+index_status. This contradicts the explicit spec ("Rust can query but Python builds",sdks/README.md:539) but is the most drift-safe option — worth weighing against how badly Python-native building is actually wanted. - Trim the README instead of implementing it. Delete
sdks/README.md:480-542and admit the ML layer is out of scope. Cheapest, and honest about current reality — but it strands every future consumer in BOSC's position of re-deriving the layer, which is the cost this RFC set exists to eliminate.
Open questions
Answered 2026-09-26 (#1031). Each is kept with its answer rather than deleted: a question that turns out not to matter is worth the same sentence as one that does, and "not pursued, because —" is the part a later reader cannot reconstruct.
-
Should the canonical embed-text assembler be a parity function, or a lower-tier convention? A parity function. The argument in the question was the right one — it is the exact seam where two implementations already diverged — and a convention is what the seam already had. It is
compose_embed_text, in the table inparity/README.md, implemented in all three SDKs and held to twelve fixtures.The question said "ninth parity function", and that count was stale when it was written. The table is the count now, deliberately: a number beside a list is a second copy of the list's length with nothing keeping it honest.
One boundary the fixture draws that the question did not anticipate: resolving which fields a class declared is not in the surface. Walking
<class>.ont.ymlanduniversal.ymlis a question about a corpus on disk and the CLI answers it; what all three SDKs must agree on is what a node composes to once it is answered. -
Path (1) vs (2) of the reconciliation — quantized ONNX from Python, or fp32 with a declared
[known_delta]? Not pursued, because ask 3 is retracted and nothing now builds an index from Python. The section stands as written: the constraint is real, the two paths are still the two paths, andindex-verify— which the question said this depended on — landed and is what surfaces a mismatch. The invariant it protects is unchanged and is enforced: no index claims compatibility it does not have. The day a second embedder exists, this is the first question it has to answer, and it is answerable then with a measurement rather than now with a guess. -
embed_nodetakes aCorpusNodein the spec but both real embedders run on the YAML instance model. Settled, as the question said it would be: downstream of RFC-0002, which landed. There is one node model, andcompose_embed_texttakes aCorpusInstance.include_claimswent with the Markdown model and has no successor — what a node says is the declared prose, and whether a claim is in it is the class'sprose:declaration to make. -
Does
update_indexneed true incremental LanceDB upserts, or is rebuild-whole the right default? Not pursued, with ask 3. Rebuild-whole is whatyidam index-builddoes and, at 0 invocations across the measured population, no corpus has yet made the cost of it visible. The question to ask first is not incremental-versus-whole but whether an index is being built at all.