# blockworx JSON formats Everything blockworx writes is JSON (`docs/single-author-playbook.md`, D13). There are three files, and they answer three different questions: | File | What it is | Who reads it | |---|---|---| | `revs/NNNNNN.json.gz` | the whole document at each rev, gzip — **the document** | the app, on load and on every history step; the fsck | | `manifest.jsonl` | one appended row per rev: who, when, where, what it named | the app, on load; `jq` and scripts, for audit; it is the changelog | | `review/*.jsonl` | reviewer comment sidecars (Phase 6, not yet written) | the app, in review mode | A `document.json` beside the revs — a pretty-printed copy of the head, kept so a `git diff` of a container read as the edits between two folds — was **retired** (`docs/retire-projection-playbook.md`): the manifest is the changelog, nothing ever read the projection's body, and a second representation of the document written after every edit could silently disagree with the first. No shape leaves a container as "the diagram": the JSON export, the JSON import and the old `.bwx.zip` *share bundle* — one file wearing a container's name — are gone ([Provenance](#provenance--what-is-left-of-exports)). A diagram travels as its `.bwx` container, and where a host cannot hand over a directory it travels as [that container zipped](#bwxzip--a-container-as-one-file). The authority is the code, in this order: `crates/doc/src/document.rs` and `block_model.rs` (the document model — serde's derives on `Document` and its entities *are* the codec, both directions; `docs/log-vs-snapshot.md` §8), `crates/store/src/revs.rs` (the rev encoding), `crates/store/src/manifest.rs` (the row), `crates/store/src/document_file.rs` (the two doors: the spanned parse diagnostic and the pretty-printed write), `crates/store/src/stamp.rs` (the stamp an export carries). ## The container A `.bwx` container is a directory (D1), laid out by `crates/store/src/container.rs`: ```text doc.bwx/ revs/ the whole document at every rev, compressed — THE document manifest.jsonl one appended row per rev: names and circumstances assets/ content-addressed artwork review/ reviewer comment sidecars (Phase 6) lock advisory single-writer lock (pid + since) .gitattributes never auto-merge the manifest ``` `revs/{head}` **is** the document, and `assets/` holds the bytes it references. Delete a rev file and that rev is gone — nothing can recompute it, because there is no log to fold. There has been no log since P5 (`docs/log-vs-snapshot.md` S3); containers written before it are **not migrated** (§13), and this build does not read one. ### `revs/` — the document at every rev `crates/store/src/revs.rs`, `docs/log-vs-snapshot.md` S4. One file per rev, `revs/{rev:06}.json.gz`: gzip at level 1 over the *compact* `serde_json` bytes of the `Document` at that rev — the same spelling `document_file::to_json` writes pretty-printed. Rev 0 is the empty document and is written nothing, so a container nothing was written into holds an empty directory. K = 1: every rev is whole, and there are no diffs to chase (§5.2, P6 deferred). ```console $ gzip -dc doc.bwx/revs/000012.json.gz | jq .blocks.b7 ``` **Why gzip, and why level 1.** One compressor everywhere: the browser writes the same bytes into its origin storage that the command line writes into a directory, so a container moves between them with nothing to translate — and zstd's C does not build for `wasm32-unknown-unknown`. `flate2` on its pure-Rust `miniz_oxide` backend does, with gzip framing so a rev is still readable from a shell. Level 1 measured at 3.7 ms for the 2.8 MB of JSON the biggest fixture in the tree folds to (123 KB at zstd −1, 292 KB here); level 2 saves a seventh of the bytes for twice the time, and the slowest machine this runs on is a tablet, where that doubling is a commit the hand can feel. Containers written when revs were zstd are carried over by `blockworx migrate ` (§ *Carrying a zstd container over*), which is the last thing in the tree that names zstd. **A rev carries no payloads.** `assets/` is the one home for bytes: a rev file goes out with its `assets` table stripped and comes back with every payload it *references* (block icons, images) read from `assets/` and checked against the hash that named it. Otherwise a 1.5 MB symbol would be written again into every rev standing after it. An **export** is the exception and embeds its payloads, because a file taken out of the container has to be a whole drawing on its own. **Undo reads them (S6).** A history step reads the rev the session stood on before the record it is taking back, adopts that document, and writes it again as a new rev of its own — so an undo is still a forward row with a rev, a kind and an `of`, and the history is an audit trail rather than a state dump. The step's own commit carries no ops: a rev was accepted by the fold when it was written, so restoring it needs no second fold and no inverse to build one from. Rev 0 is what an undo of the first edit adopts. One thing a session *accumulates* rather than holds does not fall across a step: the id marks, so a restored entity keeps its own id and no id is minted twice. Payloads are not carried, and do not need to be — they live in `assets/`, are never deleted, and come back the moment a rev references one again. Across a *restart* the marks are re-derived from the head document as `max + 1` (§5.1), so a document whose highest-numbered entity was deleted re-mints that id: the ids in an older row's `touched` list are therefore a grep index, not a promise about identity. A rev file lands, fsync'd, *before* the row that names it, and its payloads land before it — so nothing ever names bytes a crash could still take away. A write that does not land costs the session its lock, exactly as a failed append does. Save-as copies the rev files at or below its cut as files. ## `manifest.jsonl` — the row shape One row per rev — and one per tag — `write_all`'d with its newline and fsync'd *after* the rev file it names has landed. A manifest that does not end in a newline ends in a write that was never made durable: load drops that row and truncates back to the last whole line. Object keys are written by `serde_json` in struct order, but the bytes a row *hashes* over are its canonical re-encoding — keys sorted, no whitespace (`manifest::Row::canonical_bytes`). A tool that rewrites the file must reproduce that spelling or the chain breaks. ```jsonc { "rev": 2, // the rev this row names, 1-based "kind": "edit", // "edit" | {"undo":{"of":3}} | {"redo":{"of":4}} | "tag" | "untag" "wall_time": 1756000002000, // unix milliseconds "author": { "name": "Ada Lovelace", "id": null }, // D9: id is null until auth ships "label": "Moved it", // the gesture's own name "scope": ["b1"], // the block path the author was standing in "scope_names": ["Adder"], // the same path as they saw it spelled (§8.1); omitted at the root "camera": { "x": 40.0, "y": -12.5, "zoom": 2.0 }, // what they were looking at "touched": ["block b1"], // the entities the act named — the grep index "hash": "", "parent": "" } ``` A real row, from `crates/store/src/goldens/manifest.jsonl`: ```json {"author":{"id":null,"name":"Ada Lovelace"},"camera":{"x":40.0,"y":-12.5,"zoom":2.0},"hash":"731eb06d1318f09386570af5602888bf0954fe1fc38bb0c20280da7865979e22","kind":"edit","label":"Moved it","parent":"8d26307f0a16996a712b6749951fcbdac4c37fcf11fb5f54acf227c9bb22a4ec","rev":2,"scope":["b1"],"scope_names":["Adder"],"touched":["block b1"],"wall_time":1756000002000} ``` **The invariant, and the line to hold in review (§10.1): a row carries names and circumstances, never document values.** `b7` is a name; the camera, the author and the clock are circumstances of the act, like `wall_time` always was. `{"x": 12}` would be a value — and the moment one appears, the log is back. Nothing folds a row, so nothing here can drift out from under a later build the way a replayed op could. `scope` is the chain of block ids from the root down to the scope the edit was made in (empty for the document root), and `scope_names` is that same chain as its author saw it spelled — recorded beside the ids rather than resolved at read time, so a history row still reads its own path after the blocks along it have been renamed or deleted. It is what §8.1's scope line shows, elided from the left so the leaf survives a narrow panel. `camera` is that scope's own coordinate space, as world-space centre and zoom rather than a rect, so the view reproduces sensibly at any window size. Both are **recorded rather than derived**: the view the author set is evidence of what they were working on, where a region computed from the change is only a guess. A headless write — a seeded container, a command-line save — records the origin at unity, so a row's shape never depends on whether there was a window. Both are advisory: a bug mis-aims a camera rather than corrupting a document. `touched` is the entities the act named, in `EntityRef`'s narration spelling (`block b7`, `pin p3`, `asset `, `document`) — kind-qualified, because `b7` and `p7` both exist. It is capped at 64, with `"truncated": true` beyond it; the fallback for a select-all-move is a diff of the two revs, which is acceptable for an index and would not have been for a camera. An undo or redo row carries the names of the row it steps, so a step frames what it moved. Deletes are self-documenting: rev 12 names `b7`, rev 12's document does not hold `b7`, so it was removed. The values are recovered by diffing the two revs (§12.1) — the trade the plan made deliberately. `label` is minted at the seal, from the verb the gesture opened under plus the entity its ops turned out to touch (`src/edit/describe.rs`). It does **not** name the scope: `scope_names` carries that, and a label that appended it would say it twice. A step's label is its direction and the label of the row it moves (`Undo Moved it`). A **tag** row names a rev the manifest already holds rather than taking one of its own, and its `label` is the name. §8.1 gives a rev zero or more tags, so removal is its own kind — an `untag` row naming which one comes off — where the older encoding, a `tag` row with a blank label, could only mean "all of them". Neither kind reaches the trail: tagging is *about* history rather than part of it, so `Ctrl+Z` after it still takes back the last edit. `hash` and `parent` are D12's two integrity checks, and they cost what a hash of bytes somebody else wrote costs — never a re-serialization of a document. `parent` is the chain: a break means the *file* was rewritten, and load refuses to open writable, reports the offending line and column, and offers read-only at the last verified prefix. `hash` is the rev file's own blake3, and it is the one state digest this format has: `stamp::Stamp.state` is the same number. Only the **head** rev is hashed at open — one file — and an older one is `blockworx verify`'s business; a rev whose bytes are gone or wrong makes an undo to it refuse (`Refusal::Unreachable`) rather than showing the wrong document. A tag has no rev file of its own and stamps the head it was appended under, which is what makes a tag written against another history detectable. A row whose `parent` does not chain, whose `rev` is out of sequence, whose `of` names a rev no row before it holds, or whose head rev file will not witness its `hash`, opens the container **read-only at the last good prefix**, with a located report. ### Save-as copies rows and rev files Playbook R37, `crates/store/src/prefix.rs`. A session looking at rev N saves the history *through* rev N — the document as shown — and lands in the result, writable, at that rev; a session at the head copies everything. The rows are copied **verbatim**, never re-serialized, so the hash chain stays whole by construction and every column a re-serialization would have to invent comes with them: wall times, authors, cameras, edit/undo/redo kinds, tag rows. The rev files at or below the cut are copied as files, and the artwork those revs reference under its own content-addressed names. Only one thing is written rather than copied: the tag rows for revs inside the cut that were named *after* it — those cannot be copied, since their parent link names a line that is not there, so they are re-stated by whoever asked for the save. A scratch session, having no rows to copy, is seeded rev by rev (`Store::seeded`). ## The stamped export A document written *out* of a container — the shape a `.json` export has, and what a PDF's metadata carries. Written by `stamp::export_text`: the `Document` serialized as itself, pretty-printed by `document_file::to_json`, under a stamp saying where it came from: ```jsonc { "stamp": { "rev": 12, "state": "" }, "version": 3, "name": "CT Scanner", // omitted when empty "top": "b5", // the id of the root (sheet) block "blocks": { "b1": { /* … */ }, "b2": { /* … */ } }, // every table is "pins": { "p1": { /* … */ } }, // a map keyed by id, "routes": { "r1": { /* … */ } }, // in id order, and "route_labels": { "x1": { /* … */ } }, // omitted when empty "texts": { "t1": { /* … */ } }, "areas": { "a1": { /* … */ } }, "images": { "i1": { /* … */ } }, "assets": { "": { "svg": "" } } // payloads, keyed by content } ``` The file is **flat, as the document holds it** (`docs/log-vs-snapshot.md` §8, §12.8): seven entity tables at the top level, each a map from id to entity, and the hierarchy is in the entities' own `parent` and `owner` pointers rather than in nesting. "What is in `b7`" is a search for `"parent": "b7"` and `"owner": "b7"`; a third-party tool rebuilds the tree with one group-by pass. The gain is what F5 wanted: a diff of two saves shows exactly the changed entities, one per line, with no nesting churn or re-indentation. **It is write-only (D11).** Load reads the head rev file unconditionally and reads nothing here but `stamp`, which is the first field so a reader takes it off the front without modelling the body. `state` is the head rev file's own blake3 — the same number its manifest row carries, and the only state digest this format has. What the stamp is for: | Stamp | Meaning | What happens | |---|---|---| | the head rev, as that rev was written | fresh | nothing | | an earlier rev, or no file | stale | a `•` in the window title; the next save rewrites it | | a rev this history never wrote | hand-edited | a warning in the log and the chrome; the next save overwrites it | It is refreshed at three moments and no others: **Save**, a clean exit, and Save As. There is deliberately no idle timer. Editing this file and expecting the document to follow does not work and is not meant to (D7). There is no road back in: **Import** takes PNG and SVG images only (see [Provenance](#provenance--what-is-left-of-exports) below). ## `.bwx.zip` — a container as one file A browser has no directory to hand a user, so a container leaves the origin's private storage as a zip of itself and comes back into it the same way (`crates/store/src/transfer.rs`, written over `Storage` and so the same bytes on every host). Entry names are the container's own, so unzipping one gives back the directory this document describes: ``` engine.bwx.zip .gitattributes manifest.jsonl revs/000001.json.gz … assets/.png … ``` Three rules, and they are all this form has: - **The entries are stored, not deflated.** A rev is already gzip and a payload is already PNG or SVG; a second pass costs a tab a compression of the same bytes and saves the transfer almost nothing. - **`lock` does not travel.** It says who is writing a container *here*, and a copy someone else opens is not that container. - **An archive holding no `manifest.jsonl`, or naming an entry outside the layout, is refused** rather than unpacked — an import lays down a container the ordinary open then verifies, not a shape that looks like one. This is not the old share bundle below, which carried a lone document file and claimed to be a container. ## Provenance — what is left of exports **Removed on `ui-polish` (P5), breaking:** Export ▸ JSON of the view and of a selection, a history row's **Copy** and **Export rev…**, **Import** of a `.json` document, the clipboard paste of an export as one block (D19), and the `.bwx.zip` share bundle (**Open shared diagram…** and **Share…**). A lone document file is not a container now that the artwork lives beside the revs in `assets/`, so none of those doors said anything true. Importing another diagram into this one as a block is wanted, and is a placement action and a dialog flow of its own, not a file door. **Import** keeps PNG and SVG images; SVG, PNG and PDF exports stay. What is left is the stamp's `provenance` block, which names where a document came from (D19): ```jsonc { "stamp": { "rev": 23, "state": "", "provenance": { "document": "motor-controller", // the source document's name "rev": 23, // the stamp's own rev, always "author": "Ada Lovelace", // the exporting session's identity (D9) "tags": ["Initial Draft"] // optional: what D18 called that rev } }, "version": 3, …the document… } ``` A document written by hand carries none, and the field is skipped when absent — which is how an importer tells an excerpt of another document from a document. `provenance.rev` is always `stamp.rev`: the one constructor that builds a provenance (`Stamp::from`) takes the rev from the stamp it attaches to. Provenance is **advisory everywhere**. Nothing folds it and nothing verifies it. Two things still use it: 1. A stamped document named on the **command line** opens as a scratch session that remembers the provenance for its title block — `From: Rev 23 of motor-controller`, with the author on hover. Session-only: it is not written into the new history. Nothing in the app writes such a file any more (`stamp::export_text` is its one writer, reached from tests). 2. A **PDF** export names the document, rev and author it was printed from in its metadata, from the same `Stamp::from`. ### Names are ids (D11, and D10 reversed) `b` and `p` **are** the entity's id: a per-kind, document-global counter minted from 1 by the document's own allocator, derived at load as one past the highest the document has ever held and never stored. Blocks, pins, routes, texts, areas and images are all written in id order, which — counters being monotonic — is creation order. So moving a block changes that block's `x`/`y` and nothing else: no renumbering, no reordering, a diff a reviewer can read. Pin numbering is **document-global**, not per block: `p47` names one pin whichever block holds it. That is what collapses the two anchor spellings (see "The root is a scope") into one, and what keeps a pin's identity across a move between owners. The cost, taken deliberately: a document file is no longer a history-independent canonical form. The same diagram built in two different orders serializes to two different files. `Document`'s own `PartialEq` — over its content, never its position — is the equality oracle that replaced it. The round trip is a property of the model rather than of a bridge: a file is what `Document` serializes to, so `parse(serialize(doc)) == doc`. `crates/store/src/document_file.rs` gates it over every `fixtures/*.json` — comparing documents, not two renderings of them — plus a byte-stable golden (`crates/store/src/goldens/document.json`) for what the format *looks like*, which the property cannot pin. `BLOCKWORX_UPDATE_GOLDENS=1` rewrites both, and the fixtures with them, on a deliberate format change. ## The document model `crates/doc/src/block_model.rs` is the authority; the file is its serde spelling, and the types own it (§8): grid geometry nests as `{ "top_left": { "x", "y" }, "size": { "w", "h" } }`, the accent reads as its variant name (`"Accent3"`), a pin's boundary slot is `{ "side": "East", "offset": 2 }`. Anything at its default is omitted — a coordinate of `0`, `"Accent0"`, `"Input"`, `"West"`, `false`, an empty string, an empty table — so a file says what the document says and nothing it merely defaults to. This is `crates/store/src/goldens/document.json`, written by the code from `fixtures/root-scope.json`: ```json { "version": 3, "name": "root scope", "top": "b1", "blocks": { "b1": { "rect": { "top_left": { "y": 7 }, "size": { "w": 6, "h": 4 } }, "title": { "name": "source", "side": "Bottom" } }, "b2": { "rect": { "top_left": { "x": 27, "y": 7 }, "size": { "w": 6, "h": 4 } }, "title": { "name": "sink", "side": "Bottom" } } }, "pins": { "p1": { "owner": "b1", "name": "out", "rect": { "top_left": { "x": 19, "y": 7 }, "size": { "w": 5, "h": 2 } }, "slot": { "side": "East" }, "dir": "Output" }, "p2": { "owner": "b2", "name": "in", "rect": { "top_left": { "x": 27, "y": 7 }, "size": { "w": 5, "h": 2 } } } }, "routes": { "r1": { "name": "link", "waypoints": [ { "pos": { "x": 16, "y": 6 } } ], "from": "p1", "to": "p2" } }, "texts": { "t1": { "text": "beside them", "pos": { "x": 3, "y": 3 } } }, "areas": { "a1": { "rect": { "size": { "w": 4, "h": 4 } }, "title": { "name": "note" } } }, "images": { "i1": { "asset": "09a1d6ef9fbcf63a29bc869ac38290ca558be2e2c6c3b4bde15fcf13e2638bbd", "rect": { "top_left": { "x": 3355443200 }, "size": { "w": 1006632960, "h": 1006632960 } } } }, "assets": { "09a1d6ef9fbcf63a29bc869ac38290ca558be2e2c6c3b4bde15fcf13e2638bbd": { "svg": "" } } } ``` (The golden is one field per line; the example above is the same bytes re-wrapped for width.) The tables and their fields: | Table | Key | Fields (beyond the defaults omitted) | |---|---|---| | `blocks` | `b` | `parent` (a block id; absent = the document root), `rect`, `locked`, `role`, `icon { asset, rect }`, `title`, `type_label` | | `pins` | `p` | `owner`, `name`, `type_name`, `tag`, `tag_hidden`, `rect` (the port body, in the owner's interior), `slot { side, offset }` (the boundary slot), `dir` (`Input` \| `InOut` \| `Output`), `port_accent`, `flip_lr` | | `routes` | `r` | `owner`, `name`, `role`, `waypoints [ { pos, locked } ]`, `from`, `to` (pin ids) | | `route_labels` | `x` | `owner` (a route id), `pos` (offset along the route) | | `texts` | `t` | `owner`, `text`, `pos`, `role`, `width` (grid cells, set by resizing the box; absent = as wide as its text, up to the column cap) | | `areas` | `a` | `owner`, `rect`, `role`, `title` | | `images` | `i` | `owner`, `asset` (a hash), `rect` (screen coordinates) | | `assets` | the content hash | `{ "svg": "" }` or `{ "png": "" }` | A label (`title`, `type_label`, an area's `title`) is `{ "name", "side" (Top \| Center \| Bottom), "offset", "hidden" }`. **Screen coordinates are fixed-point integers.** An image's or icon's `rect` and a label's `offset` are `FracVal`s: the value times 2²⁴, written as the raw integer (`3355443200` above is 200 px). That is the type's own spelling, kept as-is at the format move; a decimal human-readable form is a type-level follow-up, not a format decision. There is no `children` list and no render order in the file: **z-order is id order** (`docs/log-vs-snapshot.md` §14.1). Creation order is paint order, and editing a block does not raise it above its neighbours. ### The root is a scope The document root holds wires and annotations exactly as a block does — a diagram drawn without ever descending into a block lives entirely there. In a flat file that costs nothing to spell: the root is `b0` (`Id::NULL`, never minted), so a block at the top level simply has no `parent` and a wire or note owned by the root has no `owner` — both are the omitted default. Pin ids are document-global, so a wire names its endpoints `"p9"` at the root and inside a block alike. A document's own boundary ports are pins owned by `b0`; they are written like any other pin, and an *insert* of such a document reports them rather than slotting them onto whatever block the paste lands in. Assets are keyed by their content hash and hold exactly one payload, `{ "svg": "" }` or `{ "png": "" }`. Equal images get equal keys in every document and an unrelated edit never renumbers one. Both spellings are text (SVG is its own source text; a PNG is standard-alphabet base64), and an **export** is deliberately **self-contained**: it never points at `assets/`, because it is the interchange form — one file that opens anywhere. Only a rev file, which is always read beside its container, references the asset directory. Routes store their **corner waypoints**, not their solved geometry: route ids, the edge list and crossings are reconstructed on load (straight legs drawn directly; the auto-router runs only for a leg that is non-colinear or crosses a block). Hand-adjusted geometry therefore survives a save/load unchanged. `version` guards the format: a document declaring a version newer than this build reads is refused rather than half-read. ## Carrying a zstd container over Revs were zstd until the encoding became one compressor everywhere. A row stamps the digest of the bytes its rev file holds, so re-compressing a rev changes what its row must say — which means the manifest is rewritten too, and this is a one-way door: ```console $ blockworx migrate doc.bwx doc.bwx: 12 revs rewritten as gzip, and the manifest re-stamped, in 0.3s ``` Each rev lands under its new name before the old one is removed, and the manifest goes down whole or not at all once every rev has, so a run cut short leaves a container the next run finishes. Every row keeps every column it carried — the wall time, the author, the camera, the kind, the label, the `touched` names — and gets the `hash` its rev now has and the `parent` that follows from it, which is the chain the same rows would have had if this build had written them. A manifest that does not already verify is refused rather than re-chained around its own break, and running the migration twice is running it once. `blockworx migrate` takes the container's lock, so close the editor on it first. It is native-only and goes with the release after the one that lands it. ## KDL is gone (D14, closed) JSON is the only document format this build reads or writes. The hand-rolled KDL parser and walker were deleted at the choreographer playbook's 7·11, along with the two callers that kept them alive — the tutorial level files and the `.kdl` migration door. A `.kdl` path on the command line, or a `.kdl` file handed to *Import*, is now refused with a message saying the format is retired. The old format is described, for the record only, in `docs/kdl-format.md`. Do not resurrect it.