RFC-0049 — The ontology is the schema a pipeline loads, and the semantic layer reads (warehouse contract and the textql export)
- Status: Draft
- Track: I36
- Relates to:
- RFC-0018 (the ontology as the bound on a typed traversal, which this extends to the bound on a table)
- RFC-0040, RFC-0044 (the
numbertype and the declared value set, the two declarations a column type and a check constraint are read from) - RFC-0032 (the reference grammar, whose reference is the
ida corpus row carries) - RFC-0026 (a fetch writes bytes and never a node, the boundary §3 keeps: yidam generates the SQL a pipeline runs and runs none of it)
- RFC-0014 (node rename, the operation a class or property rename does not yet have, and why §6 refuses one)
- Versioning layers touched: tooling only. Two new
exportformats,warehouseandtextql; one new file kind a corpus may write,.yidam/warehouse/feeds/<feed>.yml; one new report field,warehouse, inreport.schema.json. No MCP change, no SDK parity change, no template change beyond a guideline paragraph indirectories.md. - Downstream reference case: a corpus modelling GCP and AWS cloud resources whose instances live in the providers' inventories, not in the corpus. The corpus is where the model was worked out. A BigQuery warehouse is where the facts will live. TextQL's Ana, at the owner's workplace, is where the questions will be asked. The corpus cannot reach the workplace and the workplace cannot reach the corpus.
Summary
A corpus's ontology is the most worked-over artefact a derived repository has: classes with
descriptions, typed properties, declared value sets, edges with directions. Today it can
leave the repository as RDF (export --format rdf), as GraphML, or as prose. None of those
is something a data pipeline can load into, or something a semantic layer over a warehouse
can query.
This RFC makes the ontology the schema of record for a warehouse, and proposes three outputs. Each is generated from one in-memory contract derived from .ont.yml:
export --format warehouse. This produces DDL for the contract tables, starting with BigQuery, plus a machine-readablecontract.jsonand, for each feed a corpus declares, the SQL that moves a landing table into a contract table.- Feeds. A feed is a small, source-agnostic mapping file the corpus writes. It says which landing table, which columns or JSON paths, and which id feed each class and edge. The SQL it generates is run by whatever orchestrates the pipeline, such as Airflow; yidam runs none of it.
export --format textql. This produces a folder of Markdown, CSV and.tqlfiles laid out for TextQL's git-synced Ontology repository. The.tqlfiles query the contract tables by construction, so there is nothing to bind at the other end.
The corpus's own instances can travel the same route, behind a flag, as one more feed named corpus.
Problem
1.1 What a corpus has, and what a warehouse needs
| A warehouse table needs | The ontology already declares | Where |
|---|---|---|
| a name | the class stem | yidam/cli/src/corpus/class.rs:74-81 |
| typed columns | properties[].type: string, text, ref, number, date, claim, quotation | yidam/cli/src/cmd/lint/checks.rs:1469-1476 |
NOT NULL | properties[].required (#301) | yidam/cli/src/cmd/graph.rs:84 |
| an allowed-values constraint | properties[].values (RFC-0044) | docs/rfcs/0044-declared-value-set.md |
| join paths | edges[] with relationship, target, direction | yidam/cli/src/cmd/graph.rs:110-119 |
| a key | — | not declared; §2.2 |
| cardinality | — | not declared; §2.3 |
| an aggregation | — | not declared; §5.3 |
Five of the eight are already declared and checked by lint. The other three are where this RFC has to choose a default, and each choice below fails closed.
1.2 Why not RDF
export --format rdf already emits owl:Class and the edges as object properties
(yidam/cli/src/cmd/export_rdf.rs:260). That is the right artefact for a triple store and
the wrong one for a warehouse. No pipeline tool reads OWL to decide a table's columns, and no
semantic layer over SQL reads Turtle. The contract here is a second rendering of the same
model, beside RDF, not a replacement for it.
1.3 Why not a connector
The obvious alternative is a yidam connector that pulls inventory from AWS and GCP. It is refused (§Alternatives) for the reason RFC-0026 refuses a fetch that writes a node. The pipeline already exists at the reference case's workplace (Airbyte, Airflow) and so do the landing tables: Cloud Asset Inventory exports to BigQuery natively. What is missing is the mapping from landing tables into the shape the ontology describes. That mapping is declarative, it is where the corpus's knowledge applies, and it is what §3 proposes.
What TextQL's Ontology is
Read 2026-10-21 from docs.textql.com. This surface is a product's, and it moves. Re-read before implementing §5 and date the reading in the PR.
- Everything is a file in a git repository. It syncs both ways with GitHub, GitLab, CodeCommit or any git remote. A commit to the connected branch appears in the Ontology within minutes, and edits approved in TextQL are pushed back as commits or as pull requests.
.mdholds definitions, rules and conventions..csvholds reference data..tqlholds SQL templates with typed parameters, which TextQL calls the semantic layer. A.tqlfile is compiled to SQL against a connector with no LLM in the SQL path.ANA.mdis loaded into every chat for every user. FolderREADME.mdfiles with "when asked about X → go to file Y" tables are how Ana finds files on demand.- The recommended
.tqllayout keeps every table name in oneschema.tql. Relation, dimension and filter modules import it. This RFC's output follows that layout.
The consequence for this design: delivering to TextQL is committing a folder to a git repository. yidam needs no TextQL credential, and makes no TextQL API call.
Proposal
2. The contract
One internal model, warehouse::Contract, is derived from the loaded classes. The two
renderers below read it and nothing else, so the DDL, the feed SQL and the .tql cannot disagree
about a column.
2.1 Classes become tables
| Ontology | BigQuery column | Note |
|---|---|---|
| — | id STRING NOT NULL | §2.2 |
| — | _origin STRING NOT NULL | the feed that wrote the row: corpus, or a feed's name |
| — | _loaded_at TIMESTAMP NOT NULL | set by the feed SQL |
| — | _source STRING | the catalog slug the row's facts came from, when known |
| — | _corpus_commit STRING | the commit a corpus row was exported from; null otherwise |
string, text, ref | STRING | ref holds an RFC-0032 reference |
string with values: | STRING + a vocab_<set> table | BigQuery does not enforce CHECK; §2.4 |
number | FLOAT64 | yidam's number is a finite double (numeric_value), and FLOAT64 keeps it exact |
date | <name> DATE + <name>_precision STRING | §2.5 |
claim, quotation | none | prose and standing, not facts; carried in the Markdown layer |
required: true | NOT NULL |
Identifiers are the name with every character outside [A-Za-z0-9_] turned to _. Class
and property stems are kebab-case, but a relationship is free text: a derived corpus writes
exposed at. Two names that collide after that (a-b, a_b) are a refusal naming both,
not a silent merge.
2.2 The key
The primary key is (id, _origin), which BigQuery records as PRIMARY KEY … NOT ENFORCED.
- In a
corpusrow,idis the node's reference. - In a feed row,
idis whatever the feed'sid:expression yields. For the reference case that is the resource's provider-native name: a CAInameor an AWS ARN, both globally unique.
Keying on the pair, not on id alone, lets the corpus's account of a resource and the
inventory's record of it coexist without either overwriting the other. Whether they should
then be joined is an open question (Q1).
2.3 Edges become link tables
Each declared (class, relationship) becomes edge_<class>__<relationship>(src_id, dst_id, _origin, _loaded_at, _source). The ontology declares no cardinality, and a link table is the
shape that is correct for every cardinality. A one-to-one edge stored this way costs a join.
A many-to-many edge stored as a foreign key would lose rows. The default is the one that is
never wrong.
direction: in edges are written once, on the class that authors them, as source_classes
already resolves (yidam/cli/src/cmd/graph.rs:105-109). They are not written twice.
2.4 Value sets
Each declared set becomes a vocab_<class>__<property> table, one row per value. The same values are also
written as a CSV for the TextQL layer. BigQuery has no enforced CHECK, so the constraint is
enforced where it can be:
- the feed SQL (§3.3) counts the rows outside the set and fails the job;
- the
.tqlfilterables (§5.2) accept only declared values.
2.5 Dates with a precision
lint accepts YYYY, YYYY-MM and YYYY-MM-DD
(yidam/cli/src/cmd/lint/checks.rs:1469-1476). A single DATE column would have to invent
a day. So a date becomes the interval's first day plus its precision: 1893 is stored as
1893-01-01 and year. That is lossless, it sorts, and a query that cares can widen the
bound. Feed rows from a source with full timestamps write day.
3. Feeds
3.1 The file
A corpus declares a feed at .yidam/warehouse/feeds/<feed>.yml. The format is
source-agnostic. Nothing in it names a provider, and the cloud-inventory case below is an
instance of it, not its scope.
feed: gcp-cai
description: Cloud Asset Inventory export, per-asset-type tables, daily.
mode: snapshot # snapshot: rows absent from landing are deleted for this _origin
# append: rows are upserted and never deleted
landing: "${project}.${landing_dataset}" # optional; `${landing}` below expands to it
parameters: # every other ${...}, with what it is
project: The GCP project holding the landing dataset.
landing_dataset: The dataset CAI exports into.
classes:
compute-instance:
from: "`${landing}.cai_compute_googleapis_com_Instance`"
where: "resource.data.status != 'TERMINATED'"
id: "name"
source: gcp-cloud-asset-inventory # a catalog slug, written to _source
columns:
zone: "resource.location"
machine_type: "resource.data.machineType"
created: "DATE(resource.data.creationTimestamp)"
edges:
attached-to:
unnest: "resource.data.networkInterfaces"
as: nic # names the element; CAI's interfaces have a `name` too
dst_id: "nic.network"
The expressions are authored SQL fragments, trusted the way .tql treats authored fragments.
They are reviewed in the corpus's own PRs, and the generator neither parses nor rewrites them.
The ${...} parameters are the only thing left for run time. They are template parameters,
not SQL, so a dataset name that differs between environments needs no regeneration. The one
the generator touches is ${landing}, which it expands to landing: when the feed declares
one. Without landing: it is a run-time parameter like the rest. Every other parameter is
listed under parameters:, and the script's header repeats that list.
A date mapped without its _precision column is written as day, the precision of a
source with full timestamps (§2.5).
3.2 What export checks before writing
Every check runs against the contract, so all of them run here, with no warehouse to hand:
- every mapped class, property and edge is declared in the ontology;
- every
required: trueproperty of a mapped class is mapped; - no column is mapped twice;
- every
source:is an existing catalog slug; - every
${...}parameter is listed.
Classes no feed maps are reported, not refused. Coverage is information the author wants; a gate on it would refuse the first feed anyone writes.
3.3 The SQL it generates
For each mapped class and each mapped edge, the export writes one BigQuery script, warehouse/feeds/<feed>/<table>.sql, named for the contract table it loads. The script:
CASTs each expression to the contract type, so a landing column that arrives asNUMERICor a JSON string still lands as the contract says;- fails the job if any value of a value-set property is outside the set, naming the value;
- fails the job if any
idoccurs twice in landing, naming it (MERGEwould fail anyway, without saying which row); MERGEs into the contract table on(id, _origin);- under
mode: snapshot, deletes this_origin's rows that are absent from landing.
Link tables get the same treatment, keyed on (src_id, dst_id, _origin). A row whose dst_id is null links to nothing, so it is dropped rather than refused. The scripts are plain SQL with a header naming the parameters.
They run anywhere BigQuery SQL runs: an Airflow BigQueryInsertJobOperator, a scheduled
query, bq query, or a dbt run-operation. yidam ships no DAG. A worked Airflow example belongs in the
guide (§7), not in the binary.
3.4 The corpus as a feed
--with-corpus adds the feed corpus. It writes each class's instances as NDJSON beside a BigQuery
schema file, together with a load script that runs bq load into a staging table and the same MERGE. The
corpus is one more origin, so it takes the same path and the same checks as any other.
As built (P5, #1400). Every class and link table gets
feeds/corpus/<table>.sql; a table holding rows also gets<table>.ndjsonand<table>.schema.json, andload.shstages each one withbq load --replaceinto_staging_corpus__<table>before running its script, which drops the staging table. A line carries every contract column but_originand_loaded_at, which theMERGEwrites, and a null is written rather than omitted.idisyidam://<genesis12>/node/<class>/<name>, the corpus named as the RDF export names it, so a corpus with no genesis of its own is refused rather than keyed under a shared constant._sourceis the slug of the one catalog entry the node links, and null when it links none or several. The mode issnapshot, and a table the corpus holds no rows for stages nothing: its script deletes the table'scorpusrows. A node is checked with lint's ownproperty-typeand declared-value predicates, plus the calendar, sinceDATErefuses the2024-02-31lint admits. A missing required value is refused too. Every refusal names the node, and nothing is written. Left out and counted, never refused: nodes of an undeclared class, property values the class does not declare, links whose relationship is no declared edge, and links to no node of the edge's target classes. A re-export removes a folder its removals empty, so dropping the flag leaves nofeeds/corpus/behind.
4. Ownership and re-export
The TextQL folder syncs both ways. So does any repository the warehouse tree is committed to. A re-export must not overwrite what someone changed at the other end.
- Every generated file carries a marker in a comment, which is the
yidam schemaprecedent (#1057). The vocab CSVs are the exception: CSV has no comment, and TextQL would read a marker line as a value. The manifest alone records them, and an edited CSV is still a refusal. .yidam-export.jsonrecords each file's hash at export time.- A file whose hash no longer matches is a refusal naming it, unless
--forceis given. - A file without the marker was not written by yidam, and is left alone.
schema.tql(§5.2) is written only when it is absent, so a work-side edit to a dataset name survives.
5. The textql export
5.1 Layout
<prefix>/ --prefix, so the folder sits beside the work repo's own
README.md navigation: "when asked about <class label> → classes/<class>.md"
warehouse.md the provenance columns, the feeds, _origin, date precision
classes/<class>.md description; a property table with column, type, required, values;
an edge table with link table and target; alignment IRIs; claims
and quotations as prose
vocab/<class>__<property>.csv value sets
schema.tql { <class> = { backing = sql"`<project>.<dataset>.<class>`" }, … }
relations/<class>.tql one column fragment per contract column
links/<class>__<relationship>.tql join fragment through the link table
views/<class>.tql semantic view: dimensions, filterables, metrics
It writes no ANA.md. That file is loaded into every chat in the organisation, so adding
to it is the work-side owner's call. The export prints the navigation row to add. --prefix
is where the folder sits in the TextQL repository, and it appears only in that row, because
every link inside the folder is relative. It defaults to the output folder's name.
5.2 What the .tql views expose
- Dimensions: every
string,ref,date, and value-set column, plus_origin. - Filterables: equality on value-set columns, accepting the declared values only;
gte/lteonnumberanddatecolumns; equality on_origin. - Joins: one per edge, through its link table, onto the target class's relation.
5.3 Metrics
v1 emits count per class and nothing else. The ontology declares no aggregation, and
inventing sum(size_gb) because a property is a number would put a definition into the
semantic layer that nobody wrote, and Ana follows definitions exactly. A metrics:
declaration on .ont.yml is Q3.
6. Schema evolution
warehouse/contract.json is committed with the export. The next export diffs the new contract against it:
| Change | Output |
|---|---|
| class, property, edge or value added | warehouse/ddl/bigquery/<n>_<slug>.sql with CREATE TABLE / ALTER TABLE ADD COLUMN / INSERT into a vocab table |
a property's required loosened | ALTER COLUMN DROP NOT NULL |
removal, rename, type change, required tightened, value removed | refused, naming each |
| a required property added to a class that already has a table | refused: BigQuery cannot add a NOT NULL column, and the rows already loaded have no value |
--allow-breaking writes the breaking statements into a separate
warehouse/ddl/bigquery/breaking/<n>.sql for a human to run. It never runs them, and never folds them into the additive
script.
A rename shows up as a removal plus an addition, because the ontology records no class or property rename (RFC-0014 renames nodes). That is the reason renames are refused rather than guessed.
As built (P6, #1401). The first export, with no
contract.jsonbeside it, writes0001_init.sql.<n>is one more than the highest four-digit prefix inddl/bigquery/andddl/bigquery/breaking/together, so a breaking file and the migration written with it share a number and the next export moves past both. A migration an earlier export wrote is history: a later export keeps it although it no longer produces it, refuses an edit to it, and under--forcerecords the edit rather than rewriting the file, since the edited script may be the one that ran. A refusal lists the additive changes too, since none of them is written either. Descriptions and a link table's targets are not compared. Acontract.jsonof anotherformat_versionis refused rather than diffed.
7. Documentation
- A guide page: From a corpus to a warehouse and a semantic layer.
- Two reference feeds under
examples/, both read-only against public schemas:- Cloud Asset Inventory per-asset-type export;
- AWS Config snapshot, delivered to S3 and loaded to BigQuery.
- An Airflow example that runs a feed's scripts after an Airbyte sync.
As built (P7, #1402). The guide is the walkthrough
corpus-to-warehouse, so its$ yidamblocks are re-run by the transcript gate like every other walkthrough's. Both reference feeds live in a new example,examples/cloud-inventory/, whose ontology gives them classes to map: an example's feeds have to pass the export's checks inevery_example_places_in_a_contract. The Airflow DAG is a fence on the guide. It guards each snapshot behind aBigQueryCheckOperatorthat its landing table is not empty, which is where the empty-landing deletion §3.3 cannot see is caught.
What is verified, and what is not
| Claim | Status |
|---|---|
TextQL's Ontology is a git-synced file tree of .md, .csv and .tql | read 2026-10-21; not run |
a .tql of the shape §5.2 describes passes TextQL's inspect and render | unverified — P0 |
CAI's per-asset-type export names tables <table>_<asset_type with non-alphanumerics as _>, with resource.data as nested RECORDs | read 2026-10-21; not run |
MERGE … WHEN NOT MATCHED BY SOURCE AND _origin = @origin THEN DELETE is valid BigQuery | unverified — P0 |
Airbyte's BigQuery destination lands number as NUMERIC | read on a page that does not say which destination version; §3.3's CAST makes it moot either way |
What this does not touch
- The corpus's model. No
.ont.ymlfield is added. Q2 and Q3 would each add one, and each is its own RFC. export --format sqlite. It holds the vector index and stays that way.- The MCP contract. No tool reads the warehouse.
- Running anything. yidam writes SQL and data files; it opens no connection to BigQuery or TextQL.
Migration & compatibility
Nothing existing changes. A corpus with no feeds/ directory exports a contract and the TextQL
folder. The new warehouse report field is additive and is added to report.schema.json with its reachability
case. The feature is pure text generation with no new dependency, so it ships in the default build.
Alternatives considered
- A yidam connector that pulls AWS and GCP inventory. Refused. The pipeline exists, the landing data exists, and a connector would make yidam the owner of credentials, rate limits and API drift for two clouds. That ownership is the scope the owner asked not to be narrowed to, and it is the part a feed file makes unnecessary.
- One
textqlformat carrying the DDL inside it. Refused. The contract is useful to a warehouse with no TextQL in front of it, and a second semantic layer (dbt's, Cube) should be a second renderer over the same contract, not a fork of this one. - Binding the
.tqlto existing work tables through a mapping file. This was the first design, before the owner said the tables do not exist yet. It is subsumed: a feed maps landing tables into contract tables, so the.tqlbinds to the contract and never to a landing table. - Foreign keys for edges. Refused, for §2.3's reason: the ontology declares no cardinality.
- Airbyte stream schemas generated from the contract. Deferred. Useful only to someone writing a custom Airbyte connector whose output is already contract-shaped.
contract.jsoncarries enough for that to be a later renderer.
Phases
| Phase | Work | Gate |
|---|---|---|
| P0 | Work-side, by hand, on one class: create the table from hand-written DDL, load CAI rows with a hand-written MERGE, push one .tql view and run inspect, render and execute. | The owner's run. Every unverified row above is settled here. |
| P1 | warehouse::Contract, --format warehouse DDL for BigQuery, contract.json, markers and the manifest. | |
| P2 | Feed files: parse, the §3.2 checks, and the §3.3 SQL. | |
| P3 | --format textql: Markdown, vocab CSVs and the navigation README. | Useful before P4: Ana reads Markdown. |
| P4 | --format textql: schema.tql, relations, links and views. | P0's .tql result. |
| P5 | --with-corpus. | |
| P6 | Schema evolution: additive migrations and the breaking-change refusal. | Before the first re-export anyone relies on. |
| P7 | Guide, two reference feeds, the Airflow example. |
Open questions
- Should the corpus's row and a feed's row for the same resource be joined? If both carry one provider-native
id, a view could put the corpus's account beside the inventory's facts. That needs a precedence rule per column, and the corpus would have to carry provider ids on its nodes. - Should
.ont.ymldeclare cardinality? Acardinality: oneon an edge would let §2.3 emit a column instead of a link table. It is not needed for correctness. - Should
.ont.ymldeclare metrics? Ametrics:block (name,aggregate,over,description) would make §5.3 emit real measures. It is the one place this RFC most wants a declaration it does not have. - Should dialects beyond BigQuery follow? The contract is dialect-free and the DDL and feed renderers are per dialect. Snowflake and Postgres are the obvious next two, and nobody has asked for either yet.
- Should a second semantic layer follow? A dbt
schema.ymlwith tests generated from the contract is the cheapest candidate.