README

Folio stores normalized/enriched dataset JSONL as portable SQLite archive files. It is the database, archive, and browsing layer for data that has already been normalized by dataset/import tooling.

FolioBuilder turns explicit JSONL inputs into a standalone folio SQLite file. The owning application selects inputs and manages its workflow. Consumers such as zm can use the Symfony/DataContracts stack when present, while Python/R/SQLite users can query the archive directly.

Required: survos/field-bundle, survos/data-contracts. Suggested for ingest/write workflows: survos/jsonl-bundle, survos/import-bundle.

Folios live under survos_folio.data_dir (default APP_DATA_DIR); an application's own DataPaths service, if any, wins. See docs/configuration.md for paths and the required multi-connection Doctrine setup. See docs/archive-metadata.md for the standalone archive metadata contract. See docs/publisher-registration.md for how a publishing app confirms and withdraws folios with Harvest (PublisherRegistrar). See docs/presentation-layer.md for the proposal to use folios as narrative institutional presentation packages.

Building and command cleanup

See Building folios for the service API and supported commands. Dataset-specific build, archive, translate and ingest CLI shortcuts are removed; no replacement shortcuts are installed in another bundle.

Archive Contract

A folio file stores canonical rows in item and self-describing metadata alongside them:

• schema_table and schema_property describe observed DTO types and fields in this archive.
• schema_property.stats stores field profile output from survos/jsonl-bundle's profiler.
• docs stores generated JSON/Markdown documentation for humans, report writers, and AI agents.
• generated dto_* SQLite views project JSON fields into query-friendly columns.
• term_set and term store standalone controlled vocabularies and facets.

The metadata snapshot describes actual observed data, not the entire DTO contract universe. DTO classes from survos/data-contracts annotate observed fields with labels/descriptions when available, but consumers do not need PHP code to understand an archived folio.

Search and Publication Notes

• FolioBuilder loads rows, snapshots observed schema/docs/views, and rebuilds the SQLite FTS5 table item_fts.
• Existing folios can rebuild search with bin/console folio:fts:rebuild <provider/dataset> --query="search terms".
• FolioArchiveService refreshes archive metadata before packaging.
• FTS tables are derived data. Published archive files may drop item_fts, VACUUM, compress, ship, then rebuild FTS on the consuming side.
• SQLite views and docs are also derived from persisted metadata, but they are intentionally lightweight and useful for standalone consumers.
• Vector search is intentionally deferred. When added, start with a hybrid SQLite design: FTS5/BM25 for exact keyword strength, sqlite-vec for semantic retrieval, and Reciprocal Rank Fusion to merge ranks without normalizing incompatible score scales. Reference: https://ceaksan.com/en/hybrid-search-fts5-vector-rrf

Direct SQLite Examples

select * from schema_table where kind = 'dto';
select * from schema_property where table_id = ? order by position;
select local_id, label, dto_type, dto_data, extras from item limit 20;
select * from dto_document limit 20;
select id, type, audience, body from docs order by position;

TODO

• Add fieldSet support to the api-grid spreadsheet view to avoid displaying every DTO field at once.
• Rebuild views/docs on restore, not only FTS, if the archive was packaged without them.

Catalog consumers and published shared folios

survos_folio.read_only: true opens FolioService contexts using SQLite mode=ro, disables schema auto-migration/folio-row creation and avoids WAL mode changes. Publish a compatible, completed, checkpointed folio before sharing; the reader will not upgrade it. Explicit archive restore and inflate commands still write their selected local targets.

archive_api_prefix defaults to /folio and controls folio:pull listing/fallback download URLs independently of this app's route_prefix or a remote /f browse prefix. Download URLs returned by the catalog remain authoritative. local_passthrough: true skips already-present shared folios even with --force.

PeriodicalCoverageService::compute() preserves periodical block/article/ad distinctions and returns counts plus per-issue details. It scans folio rows: run it offline when recording catalog summaries, never as a directory-page cache-miss fallback. zm's publisher command persists the compact versioned summary in its registry; Ink consumes that recorded API data.

Catalog UI

The former dataset-registry collection controller and grid are removed. Applications render published catalog entries using FolioCatalogClient; the demo shows this without a dataset database. Reader routes for individual folios remain available.

Prepare a queryable DuckDB companion

DuckDbFolioPreparer::prepare($folioPath, fts: true) derives a local .duckdb from an already restored .folio, without running Harvest ingestion. Restore archives with the existing FolioPullCommand / FolioArchiveService first. The optional target defaults to the sibling .duckdb path.

The preparer reuses DuckDbFolioExporter, then adds search_records via DuckDbFolioSearchProjection and optional persisted FTS via search-bundle's DuckDbFtsIndexer. Native DTO columns remain authoritative; this export still mirrors the archive's logical tables as well. It is not a lean export profile.

_duckdb_preparation records the shared build ID, schema/version signature and FTS choice. Matching files are reused. Changes rebuild under a per-file local lock into a staging file and replace the target only after a checkpoint. Failures preserve the previously published companion. force: true rebuilds regardless of its signature. No PDO connection remains open after preparation. Native PDO DuckDB 2.x is required; install the FTS extension beforehand when used. Older folios without _folio_build are rejected rather than assigned an invented identity. ORM access through FolioContext is unchanged.

Newspaper DuckDB geometry

Newspaper .dfolio export requires a rendition publisher callback. One published DuckDB .drend per paper holds every issue's page layouts/dimensions and nested coordinate patches; _renditions pins its URI, size and source build. Stable anchors, text and semantic grouping remain in .dfolio. Every source item/page must map to an issue. Non-newspaper exports retain the previous complete-table behavior.

DuckDbRenditionReader validates the namespace, issue and build before restoring exact geometry. DuckDbFolioRowMapper constructs the existing detached Row/Page model. Full DuckDbFolioComparator comparison of a newspaper pair requires an openRend callback and compares restored values, not counts alone. Remote readers attach both files read-only with HTTP ranges. S3 secrets are temporary in the reader's isolated in-memory database, including the internal WAL existence probe; no credentials are written into either snapshot.

Rendition connections are pooled by file/endpoint and invalidated by build ID, not opened again for each issue. Every issue-bearing query must include an explicit WHERE issue = ?; joins between blocks and crops must also match issue, since their IDs repeat across issues. No issue-specific temporary views are installed on shared connections. restore($rend, $items, $pages, $issueId) requires the issue explicitly; validateIssue() replaces the old scope() API. Consumers using the old views must migrate with this reader change. The shared browse view and comparator use explicit issue filters too. Without DuckDbBundle, the reader keeps a bounded two-file cache with the same build invalidation.

Typed ordering and grouping counts

Exports materialize link.ordinal as nullable BIGINT from the retained link metadata when the source has no typed ordinal. Ordering can use ORDER BY l.ordinal, i.local_id without parsing JSON. This is stable relationship ordering, not an intrinsic order assigned to every item. Original extras remain unchanged; a source's existing typed ordinal takes precedence.

Newspaper exports also materialize issue_group_role: issue/page, group ordinal and ID, numeric role ordinal, role, block ID, and start/end line offsets. This lets counts join issue_grouping, issue_page, issue_group, and role spans to rendition OCR without traversing grouping JSON. Empty newspaper projections are created too. Issues with grouping history retain that history in issue_extras and participate in the typed grouping projection.

Re-export and republish existing newspaper snapshots before deploying consumers that require these columns/tables. Rebuild from the existing source folio (or the normal scratch-folio build from normalized inputs), not by running ALTER against published S3 files. No new OCR, grouping algorithm, or source schema migration is required. Consumers must synchronize the new catalog revision.