Skip to content

Source packs

Fetch a catalog source by its identifier, through a pack that says how its publisher answers.

A catalog entry can name a source by identifier, such as doi:10.1167/tvst.8.5.14. A source pack says how one family of identifiers resolves to an address. It also says how to ask that publisher politely, and how to read its answers. One pack serves every corpus that pins it, so nobody writes a connector twice. RFC-0048 gives the reasoning.

The template ships two packs in yidam/sources/. A corpus can also write its own, or pin one another corpus wrote.

PackSchemesReads
scholarlydoi, europepmc, pmc, arxivCrossref metadata, Europe PMC full text, the arXiv API
archivewayback, ia, ia-file, github, wikipediaWayback Machine snapshots, Internet Archive items, GitHub files at a commit, Wikipedia revisions

One doi: reaches the full text when Europe PMC holds it open access. The DOI's describe names a europepmc: location for the same DOI. That one asks Europe PMC for the record, and names a pmc: location when the full text is open. A DOI Europe PMC does not index still gets the europepmc: location, which leads nowhere. scholarly requires YIDAM_CONTACT, and waits three seconds between requests to one host, as arXiv asks.

An ia: item names its OCR text as an ia-file: location, when it has one. archive is also what catalog-fetch --archive needs: it declares wayback. A github: file and a wikipedia: article are each pinned to one version when added.

Name a source by identifier

Write the location as kind: identifier, with a scheme:local-id value:

location:
  - kind: identifier
    value: doi:10.1167/tvst.8.5.14
  - kind: identifier
    value: pmc:PMC6753881

The scheme is the text before the first colon. catalog-fetch resolves it through the enabled pack that declares it. An identifier in a scheme no pack declares is passed over, and nothing fails.

Once a corpus enables any pack, catalog-location-malformed holds each identifier to the packs. It reports a scheme no enabled pack declares, and a local id the scheme's pattern refuses. For a scheme a template pack declares, the finding names that pack. A corpus with no packs sees no change.

An entry written with kind: doi or kind: pmc is reported by catalog-location-malformed. Rewrite them:

yidam migrate --dry-run locations
yidam migrate locations

A value that is a URL is reported and left alone. Rewrite it by hand.

Find and add a source

yidam source list prints each enabled pack and its schemes. It also says which of the transport's variables are set.

Find an identifier with the pack's search endpoint, then draft its entry:

yidam source search scholarly "visual acuity reading speed"
yidam source add doi:10.1167/tvst.8.5.14 --fetch

source add commits a draft .yidam/catalog/<slug>.md as catalog:. The draft is obtained: false, with the pack's entry.md as its body. --fetch then runs catalog-fetch on it, as a second commit. Neither command writes a node.

Use a pack another corpus wrote

Pin each pack under prelude_sources in .yidam/decisions/proposals.yml:

prelude_sources:
  - scholarly@^0.1
  - us-oh-legislature@^0.3 from github.com/<owner>/<corpus>@<40-hex commit>
  1. Write <pack>@<range> for a pack the template ships.
  2. Write <pack>@<range> from <repo>@<commit> for a pack another corpus wrote.
  3. Run mise run yidam-vendor-update.
  4. Run yidam source check to confirm each copy matches its pin.
  5. Commit the copies under .yidam/.vendor/sources/.

A range is ^X.Y, ~X.Y.Z, =X.Y.Z or a bare X.Y. The bounds are cargo's. A bare range means ^, as in Cargo.toml. The commit is the full 40-character SHA. A short one can come to name a different commit.

A commit holds one version of a pack. So a from range is checked against it, never resolved. The update writes a [vendored] table onto each copy. It records the pin, the origin and the commit.

What the update refuses

The update checks every pin before it touches anything. One refusal leaves the repository unchanged.

It refuses a pin when:

  • the pin is neither of the two forms above
  • the pin names no range
  • the pack name is not lowercase letters, digits and -, starting with a letter
  • the same pack is pinned twice
  • a from pin does not end in a 40-character commit
  • the commit cannot be checked out
  • that repository has no pack of that name
  • the pack's version does not satisfy the range
  • the named corpus only vendored the pack; pin the corpus that wrote it

No prelude_sources, or no proposals file, vendors no packs.

Write your own pack

Put a pack your corpus writes in .yidam/sources/<pack>/:

.yidam/sources/scholarly/
  pack.toml          # schemes, resolution, transport, defaults, fixtures
  entry.md           # the body a new catalog entry starts from
  transforms/*.glu   # optional describe and extract scripts
  fixtures/          # recorded responses, checked offline

An authored pack is read before a vendored pack of the same name. The vendored copy stays on disk, and source check reports it as shadowed.

A pack name is lowercase letters, digits and -, starting with a letter. The directory name and [pack] name must match, because a pin names the directory.

A complete manifest:

[pack]
name        = "scholarly"
version     = "0.1.0"
description = "Papers by DOI and PMCID"

[scheme.doi]
pattern  = '^10\.(?P<registrant>\d{4,9})/\S+$'
type     = "paper"
resolve  = { template = "https://api.crossref.org/works/{id}", media = "application/json" }
describe = "transforms/crossref.glu"
then     = [{ scheme = "pmc", from = "describe.pmcid" }]

[scheme.pmc]
pattern = '^PMC\d+$'
type    = "paper"
resolve = { template = "https://www.ebi.ac.uk/europepmc/webservices/rest/{id}/fullTextXML", media = "application/xml" }
extract = "transforms/europepmc-body.glu"

[transport]
contact      = "required"
min_interval = "100ms"
auth         = [{ env = "CROSSREF_TOKEN", header = "Crossref-Plus-API-Token", prefix = "Bearer " }]
archive      = "wayback"

[defaults]
ttl_days = 365

[fixtures]
"doi:10.1167/tvst.8.5.14" = "crossref-tvst.json"
"pmc:PMC6753881"          = "europepmc-pmc6753881.xml"

Manifest keys

Every table refuses a key it does not know. A misspelt key is an error, not a silent default.

[pack]

KeyRequiredValue
nameyesThe pack name. Must equal the directory name.
versionyesThe pack's own semver, MAJOR.MINOR.PATCH. No pre-release or build suffix.
descriptionnoOne line saying what the pack resolves.

[scheme.<name>]

One table per scheme. A scheme name is lowercase letters, digits and -, starting with a letter. http and https are refused.

KeyRequiredValue
patternyesA regular expression the local id must match, anchored ^…$. Its named groups are template slots.
typeyesThe catalog type an entry from this scheme takes. One of the catalog types below.
resolve.templateone of threeAn http(s) address. Slots are {id} and each named group of pattern. At least one slot.
resolve.listingone of threeAn http(s) page whose links name the files. Slots as for template. Needs match.
resolve.dcatone of threeA bare host. Its catalog is https://<host>/data.json.
resolve.matchwith listingA regular expression, anchored ^…$, that a listed name must match. Slots are bound escaped.
resolve.picknoonly, the default, or latest. latest compares runs of digits as numbers.
resolve.medianoThe media type the publisher answers in. Required when the scheme has a transform and fixtures. With dcat, a distribution must declare it.
describenotransforms/<name>.glu: publisher metadata to a draft entry.
extractnotransforms/<name>.glu: a fetched artifact to a reading.
thennoA list of { scheme, from }. Each names a scheme in this pack, and from = "describe.<key>".
pinnoA table that pins a mutable identifier to one version. See When an identifier names a moving target.
paginatenoA table that reads a paged answer to its end. See When a service answers in pages.

A then names the next identifier a source leads to, such as a DOI's PMCID. Its from reads an identifier the scheme's describe returns. source check holds each then to that. source add follows each then, and the identifiers it names become the entry's further locations.

The catalog types are paper, dataset, api, database, statute, report, standard, document and other.

[transport]

KeyDefaultValue
contact"optional""required" skips every fetch through this pack unless YIDAM_CONTACT is set.
min_intervalnoneThe least time between two requests to one host in a run: <n>ms or <n>s.
auth[]Credentials, each named by its environment variable. See below.
archivenone"wayback" lets catalog-fetch --archive pin a Wayback capture. The only value. It also needs an enabled pack declaring the wayback scheme.
blockednoneWhy this publisher cannot be fetched at all. Its identifiers are refused with this reason.

Every request sends User-Agent: yidam-catalog/<version>. Set YIDAM_CONTACT to an email address or URL, and it is appended as (+<contact>). A pack cannot set the User-Agent, and no credential may be sent in it.

Each auth entry takes one of two forms:

FormSends the credential
{ env = "VAR", header = "Name" }In that request header.
{ env = "VAR", header = "Name", prefix = "Token " }In that header, after the prefix.
{ env = "VAR", query = "param" }As that query parameter. Takes no prefix.

env is an uppercase variable name, never the value. A bare name such as auth = ["VAR"] is refused, because it does not say where the credential goes. No credential reaches the report, the entry or the commit.

Use blocked for a publisher that refuses automated clients. Fetch it by hand, and record the export as a kind: file location.

[defaults]

KeyValue
ttl_daysThe ttl_days a new entry from this pack starts with.

source add writes ttl_days into the draft entry it commits.

When no template reaches the file

Some publishers move their files. NOAA dates each file by its release, and CMS writes a new path each time. For these, set listing or dcat in place of template:

[scheme.climdiv]
pattern = '^[a-z]+$'
type    = "dataset"
resolve = { listing = "https://www.ncei.noaa.gov/pub/data/cirs/climdiv/", match = '^climdiv-{id}-v1\.0\.0-\d{8}$', pick = "latest" }

[scheme.cms]
pattern = '^[0-9a-f-]{36}$'
type    = "dataset"
resolve = { dcat = "data.cms.gov", media = "text/csv" }

A listing's candidates are its links on the page's own host. A catalog's are the downloadURLs of the one dataset whose identifier names the local id. match and pick then choose one name.

source add reads the page and writes the name into the identifier, after an @:

  - kind: identifier
    value: climdiv:pcpndv@climdiv-pcpndv-v1.0.0-20261001

This name is the pin. It is relative to the page's directory, or an absolute path on its host. catalog-fetch follows only the pin, and never reads the page. So an entry stays the file that was read. An identifier with no pin is skipped, with a message. Add the source again to read the page anew. source add finds the entry already written.

A lookup scheme has no describe. Its extract reads the pinned file.

A pack cannot set redistributable, and a pack that tries fails to load. Whether bytes may leave this machine is a licence for one source. Write redistributable: true on that artifact's record, in the entry.

When an identifier names a moving target

Some identifiers name whatever the publisher holds today. A branch moves, an article is edited, and the Wayback Machine keeps capturing a page. A location names one version, so such a scheme declares a pin table:

[scheme.wikipedia]
pattern = '^(?P<lang>[a-z][a-z-]*)/(?P<title>[^\s@#]+)@(?P<pin>\d+)$'
type    = "document"
resolve = { template = "https://{lang}.wikipedia.org/w/index.php?oldid={pin}&action=raw" }

[scheme.wikipedia.pin]
mutable = '^(?P<lang>[a-z][a-z-]*)/(?P<title>[^\s@#]+)$'
read    = "https://{lang}.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=ids&format=json&formatversion=2&titles={title}"
media   = "application/json"
value   = "query/pages/0/revisions/0/revid"
pinned  = "{lang}/{title}@{pin}"
KeyValue
mutableA regular expression, anchored ^…$, for the mutable local id. Its named groups are slots.
readAn http(s) address. Each slot is bound percent-encoded.
mediaThe media type read answers in.
valueThe path of the version in the answer, written as a [search] path.
pinnedThe pinned local id. Slots are mutable's groups, and {pin} for the value read.

The scheme's pattern admits only the pinned form. Its group (?P<pin>…) is the value read. source add wikipedia:en/Allen_County,_Ohio asks read once, and writes the pinned identifier:

  - kind: identifier
    value: wikipedia:en/Allen_County,_Ohio@1375157827

catalog-fetch refuses the mutable form, and says how to pin it. It never asks read. Two pins of one source are one source to source add.

When a service answers in pages

Some services answer a query with its first page, and HTTP 200. ArcGIS stops at the layer's maxRecordCount, and sets exceededTransferLimit: true. Socrata stops at 1,000 rows, and says nothing. A scheme for such a service declares paginate:

[scheme.arcgis]
pattern  = '^(?P<host>[a-z0-9.-]+)/(?P<layer>[A-Za-z0-9_/]+/\d+)$'
type     = "dataset"
resolve  = { template = "https://{host}/{layer}/query?where=1%3D1&outFields=*&f=json", media = "application/json" }
paginate = { offset = "resultOffset", items = "features/*", until = "!exceededTransferLimit" }
KeyRequiredValue
offsetyesThe query parameter that holds how many records were read so far.
itemsyesThe path of the record array in a page, ending in /*. * alone is a page that is an array.
untilnoA path in a page. The page is the last when it is true. A leading ! makes it last when it is not true.
limitwith sizeThe query parameter that asks for a page size.
sizewith limitThe page size to ask for.

A scheme needs until, or limit and size. Without until, a page shorter than size is the last. Socrata's form is { offset = "$offset", limit = "$limit", size = 1000, items = "*" }.

The offset grows by the records each page held, so a host's lower cap loses nothing. catalog-fetch joins the pages into one JSON document, and records one artifact. The last page's until key is kept, so the document does not say it was cut short. A single page is recorded as it arrived.

Without paginate, catalog-fetch refuses an answer with exceededTransferLimit: true. The refusal is a finding on the entry, and nothing is recorded.

[fixtures]

Map each identifier to a recorded response under fixtures/:

[fixtures]
"doi:10.1167/tvst.8.5.14" = "crossref-tvst.json"

The identifier's scheme must be one this pack declares. Its local id must match that scheme's pattern. Every file under fixtures/ must be named here. An unnamed file belongs to no scheme and is an error. Write each path relative to fixtures/, as crossref.json and not ./crossref.json.

For a listing or dcat scheme, record the page under the identifier without a pin. source check reads it, and the name it picks must resolve. Record the file itself under the pinned identifier. Only that fixture is extracted.

For a scheme with a pin, record what read answered under the mutable identifier. source check reads the value from it, and the identifier it pins must resolve.

[vendored]

Never write this table by hand. yidam-vendor-update writes it on each copy it makes. So does the bootstrap's vendor step, for each pack the bootstrap pinned.

KeyValue
pinThe prelude_sources entry this copy satisfied, verbatim.
fromThe repository it was copied from.
commitThe commit it was copied at.

An authored pack that carries [vendored] is an error. So is a vendored copy without one.

Write a transform

A transform is a pure gluon script under transforms/. A scheme names at most one of each kind.

TransformEntry pointUsed by
describeParsed -> EntryDraftsource add, and source check over the fixtures.
extractParsed -> Readingcatalog-extract, on an artifact fetched through this scheme.

The script is a single function value. The host parses the response, and the script never sees bytes.

What a transform receives

resolve.media chooses the parser. JSON, XML and CSV are read, including +json and +xml types. Any other media type is refused by name.

The host hands the script a flat list in document order:

type Parsed = { media_type : String, fields : Array Field }
type Field  = { parent : Int, name : String, path : String, value : Value }
type Value  = | Text String | Int Int | Number Float | Flag Bool | Empty | Unrepresentable
FormatHow it becomes fields
JSONEach object member and array item is a field. An array item's name is its index. The document itself is not a field.
XMLEach element is a field. An attribute is a child named @name. Text is a child named #text.
CSVThe first row is the header. Each row is a field named by its index, with a child per column.

parent is the index of the parent field, or -1 at the root. path joins every name from the root with /, such as message/title/0. A container's value is Empty. Its children follow it in the list. A JSON null and an empty CSV cell are also Empty. Whitespace between XML elements is dropped. Adjacent XML text joins into one #text field.

Select a field by its path. JSON object order is not something a transform can rely on.

What a transform returns

A describe returns an EntryDraft:

{ name : Option String, kind : Option String, date : Option String,
  description : Option String, identifiers : Array { key : String, value : String } }

Leave a field None when the response does not say. A None kind takes the scheme's type. Each identifiers key is what a then reads as describe.<key>.

An extract returns a Reading:

{ media_type : String, text : String }

An empty media_type or an empty text is refused, and no reading is recorded.

This describe takes a Crossref title and PMCID:

\p ->
    let same : String -> String -> Bool = \a b -> a == b
    let pick name acc f =
        match acc with
        | Some found -> Some found
        | None -> if same f.path name then Some f.value else None
    let text : String -> Option String = \name ->
        match array.foldable.foldl (pick name) None p.fields with
        | Some (Text s) -> Some s
        | _ -> None
    let ids =
        match text "message/pmcid" with
        | Some d -> [{ key = "pmcid", value = d }]
        | None -> []
    { name = text "message/title/0", kind = None, date = None, description = None, identifiers = ids }

The closed prelude

A transform runs in the calculator arm's closed prelude, as a calculator does. Its modules are already bound by name, so write no import!:

Bound modules
array, bool, byte, char, cmp, float, foldable, function
int, list, map, option, result, show, string

No file, process, environment, network or debug module is reachable. A script that invokes any macro is refused before it runs. A call budget stops a script that loops.

Build with transforms

Transforms need the source-transforms feature. The released binary does not include it.

cargo install --git https://github.com/goedelsoup/yidam --locked --features source-transforms yidam

--features full includes it too. A build without it still checks the rest of each pack. It reports one info finding per pack with transforms, saying they were not checked. catalog-extract reports an artifact that needs a transform as skipped.

Check a pack

Run the check after every change to a pack or a pin:

yidam source check
yidam source check --format json

--format is text (the default) or json. It reads only the repository, never the network.

It verifies that:

  • each pack.toml parses, with no unknown key
  • the name matches its directory, and the version is MAJOR.MINOR.PATCH
  • entry.md exists
  • each pattern is anchored and compiles
  • each resolve is one form, http(s), and every slot is bound
  • each match is anchored and compiles, and no address carries a #
  • each type is a catalog type
  • each transform path is transforms/<name>.glu and exists
  • each then names a declared scheme and reads describe.<key>
  • min_interval and every auth entry are well formed
  • every fixture is claimed by a scheme, matches its pattern, and exists
  • every recorded listing or catalog picks a name that resolves
  • each pin binds from mutable, writes {pin}, and its scheme's pattern has a pin group
  • every recorded pin read answers a value that pins a resolving identifier
  • each paginate names query parameters, an array path, and how to know the last page
  • no two enabled packs declare one scheme
  • every vendored pack has its pin, and every pin has its satisfying copy

With source-transforms, it also admits each transform and runs it over its scheme's fixtures. Each then key must be yielded for at least one fixture.

The JSON report has passed, packs and pack_findings. Read passed rather than counting errors. Each finding has severity, pack, path and message. Any error fails the check.

A corpus with no packs and no pins passes. It says so.

Run it in CI

The scaffolded workflow does not run it. Add a step after the binary is installed:

- name: Check source packs
  run: yidam source check

It exits nonzero on any error. Build with source-transforms there if your packs carry transforms.

Readings on an artifact

catalog-extract records each reading under the artifact it was taken from:

artifacts:
  - sha256: 9f2c8e…
    media_type: application/xml
    from: 0
    readings:
      - sha256: 41d0a7…
        media_type: text/plain
        by: scholarly@0.1.0/transforms/europepmc-body.glu@sha256:7be1…
FieldValue
sha256The reading's own digest, in the same vault cache as the artifact.
media_typeWhat the reading is. A quotation is compared with the first text/plain reading of the whole artifact.
memberThe file inside a zip this reading is. Absent for a reading of the whole artifact.
byWhat took it. A transform is <pack>@<version>/<path>@sha256:<script hash>. A PDF's is <crate> <version>. A zip member's is unzip.

A changed transform script is a new by, so catalog-extract takes a new reading beside the old. A reading already taken is never replaced.

A location names the bytes fetched, never a member inside them. So the location for a file inside a zip names the zip, and no pin or template carries a #.

Read a file inside a zip

Name the file under members:, beside the location's value:

location:
  - kind: url
    value: https://www2.census.gov/census_2010/04-Summary_File_1/Ohio/oh2010.sf1.zip
    members:
      - oh2010.sf1.prd.packinglist.txt

Fetch the zip, then run catalog-extract. It unpacks each named member into the cache and records it as a reading of the zip:

    readings:
      - sha256: 5e07b2…
        media_type: text/plain
        member: oh2010.sf1.prd.packinglist.txt
        by: unzip

The member's sha256 is its own, and its media_type comes from its extension. A member's text is never the zip's text reading.

A member path must stay inside the archive. Lint refuses .., a leading /, a drive letter and a backslash. Lint also reports a member written into the url after #. The fragment is never sent, so the fetch takes the whole zip. catalog-extract refuses a member over 1 GiB, a zip of more than 10,000 entries, zip64, and encryption. It reads stored and deflated members, and checks each one's CRC.

An entry written before readings: holds its PDF reading under text:. It is still read.

Run yidam vault push afterwards to share readings with other machines. See Reading a PDF for the PDF steps.

See also