commit d2170bfb377377ccc4fba23afd7c83a28c593707 parent bf4e4c2c0f011b7fee2f08e3917770ad2137b3be Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st> Date: Sat, 12 Sep 2026 02:10:58 -0400 plans: the composed fixture site, so the parallel slices diff instead of rebuild `plans/tools/compose-fixture-one-youtube-channel/` is compose-site.ts's output over the repo's smallest real corpus fixture, taken at this slice's tip: the hand-written `site.json` that FACTS.md's recipe leaves to the reader, the 16 files under `public/`, the 7 under `index/`, and `compose-diff.sh`. S2a, S2b, S2c, S3 and S0-pause all branch off this tip. Without a committed artifact each of them would rebuild its own baseline, and "the contract did not move" would rest on five people having typed the same recipe — including the same `site.json`, which is the input that decides whether every manifest pointer in corpus.json comes out absolute or root-relative. It sets `siteUrl` on purpose: a fixture without one would pass a reader that had quietly dropped the origin. `compose-diff.sh` normalises exactly two things and says why in the file: `generatedAt` in both its compact and pretty spellings, and the archive zip's own size — a zip's entry mtimes are the build clock, so two runs of identical input differ by a byte in the compressed stream, and `archives/manifest.json` records that size. Zips are compared by unpacked, normalised entries instead, which do diff clean. Nothing else is excused; a difference in the URL shapes is the failure this directory exists to catch. The README carries the one trap the recipe hides: `build-index.ts` caches a schema version in `$F/index.mdb`, so a second run into the same corpus root reports "0 built, 1 up to date" and copies nothing. Compose into a fresh dir. Verified by re-running the README's own commands into two fresh temp dirs: IDENTICAL modulo the build clock, 16 files and 7. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Diffstat:
26 files changed, 230 insertions(+), 0 deletions(-)
diff --git a/plans/tools/compose-fixture-one-youtube-channel/README.md b/plans/tools/compose-fixture-one-youtube-channel/README.md @@ -0,0 +1,82 @@ +# A composed site, small enough to diff by hand + +This is the output of `compose-site.ts` over the repo's smallest real corpus +fixture, captured at the tip of **one-core phase 2 slice S1** +(`one-core/phase-2-s1`, branched from `c7f7b90`). It exists so the parallel +slices — S2a, S2b, S2c, S3 and S0-pause — can prove "the contract did not move" +against a committed artifact instead of each rebuilding a baseline and hoping +the two recipes matched. + +`plans/FACTS.md` § "How to compose a fixture site offline" is the recipe; this +directory is one run of it plus the two things the recipe leaves to the reader: +the hand-written `site.json`, and a diff that knows what is allowed to change. + +## What is here + +| Path | What | +|---|---| +| `site.json` | The hand-written `sites/testsite/site.json`. The recipe's only authored input. | +| `public/` | 16 files — what a deploy serves. | +| `index/` | 7 files — the intermediate index the compose step copies from. | +| `compose-diff.sh` | `./compose-diff.sh <expected> <actual>` — a normalising diff. | + +## How it was produced + +The corpus is `editor/e2e/fixtures/test-transcripts/one-youtube-channel-with-data` +(one channel, one video, one VTT). Nothing here reads or writes the real +`transcripts/`. + +```sh +F=$(mktemp -d) # the corpus root +O=$(mktemp -d); mkdir -p $O/public $O/index + +cp -r editor/e2e/fixtures/test-transcripts/one-youtube-channel-with-data/channels $F/ +mkdir -p $F/sites/testsite +cp plans/tools/compose-fixture-one-youtube-channel/site.json $F/sites/testsite/site.json +echo '{}' > $F/settings.json + +cd common +env TRANSCRIPTS_DIR=$F SITES_DIR=$F/sites \ + EXPORT_PUBLIC_DIR=$O/public EXPORT_INDEX_DIR=$O/index \ + SETTINGS_FILE=$F/settings.json SITE_ID=testsite \ + pnpm exec tsx bin/build-index.ts +env TRANSCRIPTS_DIR=$F SITES_DIR=$F/sites \ + EXPORT_PUBLIC_DIR=$O/public EXPORT_INDEX_DIR=$O/index \ + SETTINGS_FILE=$F/settings.json SITE_ID=testsite \ + pnpm exec tsx bin/compose-site.ts +cd .. + +plans/tools/compose-fixture-one-youtube-channel/compose-diff.sh \ + plans/tools/compose-fixture-one-youtube-channel/public $O/public +plans/tools/compose-fixture-one-youtube-channel/compose-diff.sh \ + plans/tools/compose-fixture-one-youtube-channel/index $O/index +``` + +Both should print `IDENTICAL modulo the build clock`. Anything else is a real +change to the wire, and belongs in a commit that says so. + +**`build-index.ts` writes `$F/index.mdb` and caches a schema version in it.** +Compose the fixture into a fresh `$F` each time, or the second run reports +`0 built, 1 up to date` and copies nothing. + +## What the diff normalises, and why only this + +- **`"generatedAt": "<iso>"`**, in both the compact and the pretty-printed + spellings. It appears in `corpus.json`, `site.json`, the five manifests, and + the archive zip's `channel.json`. +- **The archive zip's own bytes.** A zip's entry mtimes are the build clock, so + two runs of the same input differ by a byte or two in the compressed stream — + and `archives/manifest.json` records that size. Zips are therefore compared by + their unpacked, `generatedAt`-normalised **entries**, which do diff clean, and + the `bytes` field is normalised with them. + +Nothing else is excused. In particular the URL shapes in `corpus.json` — the +absolute-with-`siteUrl` form this fixture exercises — are the thing phase 2's +`lib/archive/contract.ts` must reproduce, so a diff there is the failure this +directory exists to catch. + +## The site.json is deliberately minimal but NOT trivial + +It sets `siteUrl`, which is what makes every manifest pointer in `corpus.json` +absolute. A fixture without one would pass a reader that had quietly dropped the +origin. diff --git a/plans/tools/compose-fixture-one-youtube-channel/compose-diff.sh b/plans/tools/compose-fixture-one-youtube-channel/compose-diff.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# Diff two composed site outputs modulo the build clock. +# +# Normalised away, and only these: +# - "generatedAt": "<iso>" in every JSON (compact and pretty forms) +# - the archive zip's own bytes: entry mtimes are the build clock, so a zip +# built twice differs by a byte or two even when its ENTRIES are identical. +# Zips are therefore compared by unpacked, generatedAt-normalised contents, +# and archives/manifest.json's `bytes` (the zip's size) with them. +A=$1; B=$2 +norm() { sed -E -e 's/"generatedAt": *"[^"]*"/"generatedAt":"X"/g' "$1"; } +normarch() { sed -E -e 's/"generatedAt": *"[^"]*"/"generatedAt":"X"/g' -e 's/"bytes": *[0-9]+/"bytes":N/g' "$1"; } +rc=0 +( cd "$A" && find . -type f | sort ) > /tmp/fa.$$ +( cd "$B" && find . -type f | sort ) > /tmp/fb.$$ +if ! diff -q /tmp/fa.$$ /tmp/fb.$$ >/dev/null; then echo "FILE LIST DIFFERS"; diff /tmp/fa.$$ /tmp/fb.$$; rc=1; fi +while read -r f; do + if [[ "$f" == *.zip ]]; then + ta=$(mktemp -d); tb=$(mktemp -d) + unzip -qq -o "$A/$f" -d "$ta"; unzip -qq -o "$B/$f" -d "$tb" + while read -r e; do + if ! diff <(norm "$ta/$e") <(norm "$tb/$e") >/dev/null; then + echo "ZIP ENTRY DIFFERS: $f!$e"; rc=1 + fi + done < <(cd "$ta" && find . -type f | sort) + if ! diff <(cd "$ta" && find . -type f|sort) <(cd "$tb" && find . -type f|sort) >/dev/null; then + echo "ZIP ENTRY LIST DIFFERS: $f"; rc=1; fi + rm -rf "$ta" "$tb" + elif [[ "$f" == ./archives/manifest.json ]]; then + if ! diff <(normarch "$A/$f") <(normarch "$B/$f") >/dev/null; then + echo "DIFFERS: $f"; diff <(normarch "$A/$f") <(normarch "$B/$f") | head -8; rc=1; fi + elif ! diff <(norm "$A/$f") <(norm "$B/$f") >/dev/null; then + echo "DIFFERS: $f"; diff <(norm "$A/$f") <(norm "$B/$f") | head -8; rc=1 + fi +done < /tmp/fa.$$ +rm -f /tmp/fa.$$ /tmp/fb.$$ +[ $rc -eq 0 ] && echo "IDENTICAL modulo the build clock ($(cd "$A" && find . -type f | wc -l) files)" +exit $rc diff --git a/plans/tools/compose-fixture-one-youtube-channel/index/shared/transcripts/test-youtube/manifest.json b/plans/tools/compose-fixture-one-youtube-channel/index/shared/transcripts/test-youtube/manifest.json @@ -0,0 +1 @@ +{"version":1,"channelSlug":"test-youtube","pageCount":1,"maxPageBytes":8388608,"generatedAt":"2026-09-12T06:01:16.087Z","slugToPage":{"20240101_test1234567":0}} +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/index/shared/transcripts/test-youtube/page-0000.json b/plans/tools/compose-fixture-one-youtube-channel/index/shared/transcripts/test-youtube/page-0000.json @@ -0,0 +1 @@ +[{"slug":"test-youtube/20240101_test1234567","id":"20240101_test1234567","channelSlug":"test-youtube","title":"Synthetic Test Video","uploadDate":"20240101","duration":120,"channel":"Test YouTube Channel","description":"A synthetic test video for e2e fixtures.","tags":[],"isLivestream":false,"ageRestricted":false,"platform":"youtube","webpageUrl":"https://www.youtube.com/watch?v=20240101_test1234567","cues":[]}] +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/index/sites/testsite/digests/manifest.json b/plans/tools/compose-fixture-one-youtube-channel/index/sites/testsite/digests/manifest.json @@ -0,0 +1 @@ +{"version":1,"channels":[],"totalCount":0,"generatedAt":"2026-09-12T06:01:16.116Z","siteId":"testsite"} +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/index/sites/testsite/posts/manifest.json b/plans/tools/compose-fixture-one-youtube-channel/index/sites/testsite/posts/manifest.json @@ -0,0 +1 @@ +{"version":1,"channels":[],"totalCount":0,"generatedAt":"2026-09-12T06:01:16.115Z","siteId":"testsite"} +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/index/sites/testsite/subs/manifest.json b/plans/tools/compose-fixture-one-youtube-channel/index/sites/testsite/subs/manifest.json @@ -0,0 +1 @@ +{"version":4,"channels":[],"totalCount":0,"liveChatTotalCount":0,"generatedAt":"2026-09-12T06:01:16.115Z","groups":[{"id":"default","name":"All channels","selectedByDefault":true,"inline":true}],"defaultGroupId":"default","siteId":"testsite"} +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/index/sites/testsite/summaries/manifest.json b/plans/tools/compose-fixture-one-youtube-channel/index/sites/testsite/summaries/manifest.json @@ -0,0 +1 @@ +{"version":3,"totalCount":1,"pageSize":1000,"pageCount":1,"generatedAt":"2026-09-12T06:01:16.115Z","channels":[{"slug":"test-youtube","name":"Test YouTube Channel","count":1,"groupId":"default"}],"groups":[{"id":"default","name":"All channels","selectedByDefault":true,"inline":true}],"defaultGroupId":"default","siteId":"testsite"} +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/index/sites/testsite/summaries/page-0000.json b/plans/tools/compose-fixture-one-youtube-channel/index/sites/testsite/summaries/page-0000.json @@ -0,0 +1 @@ +[{"slug":"test-youtube/20240101_test1234567","id":"20240101_test1234567","channelSlug":"test-youtube","title":"Synthetic Test Video","uploadDate":"20240101","date":"2024-01-01","duration":"2:00","channel":"Test YouTube Channel","isLivestream":false,"ageRestricted":false,"isDeleted":false,"isUnlisted":false,"platform":"youtube","webpageUrl":"https://www.youtube.com/watch?v=20240101_test1234567"}] +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/_headers b/plans/tools/compose-fixture-one-youtube-channel/public/_headers @@ -0,0 +1,25 @@ +# Generated by compose-site.ts — do not edit by hand. +/site.json + Access-Control-Allow-Origin: * +/search-aliases.json + Access-Control-Allow-Origin: * +/summaries/* + Access-Control-Allow-Origin: * +/subs/* + Access-Control-Allow-Origin: * +/transcripts/* + Access-Control-Allow-Origin: * +/posts/* + Access-Control-Allow-Origin: * +/stats/* + Access-Control-Allow-Origin: * +/archives/* + Access-Control-Allow-Origin: * +/corpus.json + Access-Control-Allow-Origin: * +/llms.txt + Access-Control-Allow-Origin: * +/robots.txt + Access-Control-Allow-Origin: * +/sitemap.xml + Access-Control-Allow-Origin: * diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/archives/manifest.json b/plans/tools/compose-fixture-one-youtube-channel/public/archives/manifest.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "thresholdBytes": 26214400, + "entries": [ + { + "kind": "transcripts", + "scope": "test-youtube", + "filename": "test-youtube.zip", + "bytes": 1010, + "videoCount": 1 + } + ] +} diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/archives/test-youtube.zip b/plans/tools/compose-fixture-one-youtube-channel/public/archives/test-youtube.zip Binary files differ. diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/corpus.json b/plans/tools/compose-fixture-one-youtube-channel/public/corpus.json @@ -0,0 +1 @@ +{"spec":3,"kind":"site","generatedAt":"2026-09-12T06:01:16.115Z","generator":"Archilyzer (https://archilyzer.pages.dev)","site":{"id":"testsite","title":"Test Site","description":"A fixture archive composed from one YouTube channel.","url":"https://testsite.example"},"totals":{"channels":1,"videos":1},"channels":[{"slug":"test-youtube","name":"Test YouTube Channel","videoCount":1,"groupId":"default","manifests":{"transcripts":"https://testsite.example/transcripts/test-youtube/manifest.json","subs":"https://testsite.example/subs/test-youtube/manifest.json"}}],"shardScheme":{"description":"Transcripts are served as paginated JSON shards — there is no per-video file. To read one video's transcript: (1) GET the channel's transcripts manifest; (2) look up the video id in its `slugToPage` map to get a page number N; (3) GET page-<NNNN>.json (N zero-padded to 4 digits) and take the record whose `id` matches.","transcriptsManifest":"<channel.manifests.transcripts> -> { pageCount, slugToPage: { <videoId>: <pageNumber> } }","transcriptPage":"/transcripts/<slug>/page-<NNNN>.json -> array of { id, title, uploadDate, duration, channel, description, tags, webpageUrl, platform, cues: [{ start, end, text }] }","subsManifest":"<channel.manifests.subs> -> lighter list-view records under the same slugToPage scheme","summariesIndex":"/summaries/manifest.json + /summaries/page-<NNNN>.json -> cross-channel browse index","pageNumberFormat":"zero-padded to 4 digits, e.g. page 0 -> page-0000.json"},"useWithAi":"https://testsite.example/use-with-ai","bulkArchives":{"manifest":"https://testsite.example/archives/manifest.json","note":"Whole-channel transcript and live-chat archives for offline bulk ingestion."}} +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/digests/manifest.json b/plans/tools/compose-fixture-one-youtube-channel/public/digests/manifest.json @@ -0,0 +1 @@ +{"version":1,"channels":[],"totalCount":0,"generatedAt":"2026-09-12T06:01:16.116Z","siteId":"testsite"} +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/llms.txt b/plans/tools/compose-fixture-one-youtube-channel/public/llms.txt @@ -0,0 +1,15 @@ +# Test Site + +> A fixture archive composed from one YouTube channel. A machine-navigable archive of 1 transcripts across 1 channel(s). See corpus.json for the shard API. + +## Ask AI +- [Use with AI](https://testsite.example/use-with-ai): in-browser chat (bring your own API key) and MCP-server setup for Claude Code, Cursor, and other tools. + +## Corpus +- [corpus.json](https://testsite.example/corpus.json): machine-readable index — channels and how to fetch any transcript from the paginated JSON shards. +- [Bulk archives](https://testsite.example/downloads): whole-channel transcript and live-chat zips for offline ingestion. + +## Channels +- Test YouTube Channel — 1 transcripts: https://testsite.example/transcripts/test-youtube/manifest.json + +Generated by Archilyzer (https://archilyzer.pages.dev) diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/posts/manifest.json b/plans/tools/compose-fixture-one-youtube-channel/public/posts/manifest.json @@ -0,0 +1 @@ +{"version":1,"channels":[],"totalCount":0,"generatedAt":"2026-09-12T06:01:16.115Z","siteId":"testsite"} +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/robots.txt b/plans/tools/compose-fixture-one-youtube-channel/public/robots.txt @@ -0,0 +1,5 @@ +User-agent: * +Allow: / + +# AI / LLM corpus index: /llms.txt and /corpus.json +Sitemap: https://testsite.example/sitemap.xml diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/search-aliases.json b/plans/tools/compose-fixture-one-youtube-channel/public/search-aliases.json @@ -0,0 +1 @@ +{"aliases":[{"id":"loli","label":"loli","triggers":["loli","lolly","loly","lolli"],"suggestion":"\\blol(i|ly)","useRegex":true,"note":"AI transcription often expands this to “lolly”/“loly”."},{"id":"youtube","label":"YouTube","triggers":["youtube","yt"],"suggestion":"\\byou ?tube\\b|\\byt\\b","useRegex":true,"note":"Catches “you tube” and the “yt” abbreviation."}]} +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/site.json b/plans/tools/compose-fixture-one-youtube-channel/public/site.json @@ -0,0 +1 @@ +{"contract":1,"siteId":"testsite","siteTitle":"Test Site","siteDescription":"A fixture archive composed from one YouTube channel.","headerTitle":"Test Site","homeTagline":"Fixture corpus","siteUrl":"https://testsite.example","pwa":false,"socialLinks":[],"groups":[{"id":"default","name":"All channels","selectedByDefault":true,"inline":true}],"defaultGroupId":"default","channels":[{"slug":"test-youtube","name":"Test YouTube Channel","count":1,"groupId":"default"}],"generatedAt":"2026-09-12T06:01:16.115Z","summariesVersion":3} +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/sitemap.xml b/plans/tools/compose-fixture-one-youtube-channel/public/sitemap.xml @@ -0,0 +1,7 @@ +<?xml version="1.0" encoding="UTF-8"?> +<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> + <url><loc>https://testsite.example/</loc></url> + <url><loc>https://testsite.example/use-with-ai</loc></url> + <url><loc>https://testsite.example/changelog</loc></url> + <url><loc>https://testsite.example/downloads</loc></url> +</urlset> diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/subs/manifest.json b/plans/tools/compose-fixture-one-youtube-channel/public/subs/manifest.json @@ -0,0 +1 @@ +{"version":4,"channels":[],"totalCount":0,"liveChatTotalCount":0,"generatedAt":"2026-09-12T06:01:16.115Z","groups":[{"id":"default","name":"All channels","selectedByDefault":true,"inline":true}],"defaultGroupId":"default","siteId":"testsite"} +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/summaries/manifest.json b/plans/tools/compose-fixture-one-youtube-channel/public/summaries/manifest.json @@ -0,0 +1 @@ +{"version":3,"totalCount":1,"pageSize":1000,"pageCount":1,"generatedAt":"2026-09-12T06:01:16.115Z","channels":[{"slug":"test-youtube","name":"Test YouTube Channel","count":1,"groupId":"default"}],"groups":[{"id":"default","name":"All channels","selectedByDefault":true,"inline":true}],"defaultGroupId":"default","siteId":"testsite"} +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/summaries/page-0000.json b/plans/tools/compose-fixture-one-youtube-channel/public/summaries/page-0000.json @@ -0,0 +1 @@ +[{"slug":"test-youtube/20240101_test1234567","id":"20240101_test1234567","channelSlug":"test-youtube","title":"Synthetic Test Video","uploadDate":"20240101","date":"2024-01-01","duration":"2:00","channel":"Test YouTube Channel","isLivestream":false,"ageRestricted":false,"isDeleted":false,"isUnlisted":false,"platform":"youtube","webpageUrl":"https://www.youtube.com/watch?v=20240101_test1234567"}] +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/transcripts/test-youtube/manifest.json b/plans/tools/compose-fixture-one-youtube-channel/public/transcripts/test-youtube/manifest.json @@ -0,0 +1 @@ +{"version":1,"channelSlug":"test-youtube","pageCount":1,"maxPageBytes":8388608,"generatedAt":"2026-09-12T06:01:16.087Z","slugToPage":{"20240101_test1234567":0}} +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/public/transcripts/test-youtube/page-0000.json b/plans/tools/compose-fixture-one-youtube-channel/public/transcripts/test-youtube/page-0000.json @@ -0,0 +1 @@ +[{"slug":"test-youtube/20240101_test1234567","id":"20240101_test1234567","channelSlug":"test-youtube","title":"Synthetic Test Video","uploadDate":"20240101","duration":120,"channel":"Test YouTube Channel","description":"A synthetic test video for e2e fixtures.","tags":[],"isLivestream":false,"ageRestricted":false,"platform":"youtube","webpageUrl":"https://www.youtube.com/watch?v=20240101_test1234567","cues":[]}] +\ No newline at end of file diff --git a/plans/tools/compose-fixture-one-youtube-channel/site.json b/plans/tools/compose-fixture-one-youtube-channel/site.json @@ -0,0 +1,11 @@ +{ + "siteId": "testsite", + "siteTitle": "Test Site", + "siteDescription": "A fixture archive composed from one YouTube channel.", + "headerTitle": "Test Site", + "homeTagline": "Fixture corpus", + "groups": [{ "id": "main", "label": "Main" }], + "defaultGroupId": "main", + "channels": [{ "slug": "test-youtube", "groupId": "main" }], + "siteUrl": "https://testsite.example" +}