# One core — Phase 2: the contract, one `ArchiveReader` ## Context `plans/one-core.md` §Phase 2 is the umbrella. This is the slice-level plan: every walk of the published archive's shard scheme — mcp's private reader, eight component caches, the viewer's offline cache, umtool's cue resolver — collapses onto one `ArchiveReader` under `common/lib/archive/`, and the two search pipelines become one `common/lib/search/`. It starts from `main` after the 2026-09 integration (`storage/relocate-media` + `channel-priority/s5` merged on `integrate/2026-09-storage-priority` and fast-forwarded; see `STATE.md`). Decisions taken with the operator on 2026-09-12: - Integration and rollout land first; Phase 2 slices branch off the merged `main`. - umtool's `cues.mjs` does **not** adopt `tsx` in this phase. The umbrella said it would; that is deferred to Phase 5 (projects join the core) and the change list is recorded in S2b's note so nothing is re-surveyed. - The legacy pause-field deletion (S0-pause, `one-core-phase-1.md:1130-1139`) runs as its own slice in parallel with Phase 2, not inside the integration merge. ## Corrections to the umbrella Surveyed 2026-09-12; `one-core.md` §Phase 2 is edited to match. Trust these over the original wording. 1. **Eight component caches share the walk, not four.** `common/components/{transcript,subs,posts,digest,summaries,stats,duplicates,aliases}Cache.ts` — 825 lines, all keyed by `OriginId` (`components/originId.ts`). Seven more walk sites: `export/app/lib/offlineCache.ts`, `umtool/report-to-video/cues.mjs`, `common/components/siteRegistry.ts:287`, `common/components/SearchDataContext.tsx`, `common/components/SearchSessionContext.tsx`, `export/app/ask/useAskChat.ts`, and the deliberate copy in `common/components/searchIndex.worker.ts:39-41` (a worker cannot import the reader; it **stays**, guarded by the SW/contract equality test in S2a). 2. **The reader interface is mcp's 18-member `ShardSource`** (`mcp/src/source.ts:292-372`), not the umbrella's four methods. `record(layer, slug, id)` must **not** be added: a per-record fetch is a bench regression by construction (the walk is manifest → shard → record, and callers already hold the shard). 3. **`config.dataDir` is not `reader-fs.ts`'s concern.** `LocalSource` reads a *composed* public dir, which has no `dataDir`. The resolver the umbrella asked for belongs to `cues.mjs:239-243`, which walks `channels//data/` — that is S2b. ## Layout and the browser/node rule ``` common/lib/archive/ contract.ts re-exports CONTRACT (lib/corpus.ts:27-48) + pageFileName (lib/manifest.ts:44-46); adds corpusUrl, manifestUrl(layer, slug, base?), pageUrl, rootFileUrl, ROOT_FILES, shipsPwa(site); owns the hub entry type (HubSiteEntry / HubMemberInput / HubSite / HubCorpusSite → one) io-stats.ts recordRead / ioStatsSnapshot; guard `typeof process !== "undefined" && process.env?.MCP_IO_STATS` reader.ts interface ArchiveReader (= ShardSource verbatim), RemoteSource, PageCache, PromiseMap — ZERO node imports reader-fs.ts LocalSource — the only node:fs / node:path file; reader.ts never value-imports it reader-hub.ts HubSource ``` - `./lib/*` is wildcarded in `common/package.json:34`, so no exports line is needed for `lib/archive/*` or `lib/search/*`. A new `components/*.ts` **would** need one — S2a creates none. - `mcp/src/source.ts` becomes a ~40-line re-export (`export type ShardSource = ArchiveReader`, plus the three constructors), so its seven importers and `mcp/bench` (spawns the server; never imports `source.ts`) are untouched. - `manifestUrl` must reproduce `buildSiteCorpus` (`corpus.ts:251-258`; `join` at `:216-219`): absolute when `siteUrl` is set, root-relative otherwise. The posts/digests conditional spreads stay in `corpus.ts` — `corpus.json` output does not change. - **`pnpm --filter export exec next build` is the only test of browser-safety.** tsc cannot tell a `node:fs` import from any other; the export bundle can. Every slice runs it. ## Dependency graph ``` S1 contract + reader + hub types + stubs ← the only prerequisite (merged to main first) ├── S2a eight caches + offlineCache + SW guard test ├── S2b cues.mjs reachability fix (plain node; no tsx) ├── S2c shipsPwa / _headers / hub-entry dedupe ├── S3 common/lib/search/ + the lib→components inversion └── S0-pause (legacy pause fields, one-core-phase-1.md:1130-1139) ``` Every parallel slice branches off S1's merged tip on `main`. S1 pre-creates every module the others fill as `export {}` stubs with the intended signatures in comments — the five `archive/*` files and `lib/search/{policy,evalTree,window,rank,collapse}.ts` — so no two slices create the same file. Hotspots, and who owns each: | file | owner | why | |---|---|---| | `common/package.json` exports | nobody | S2a creates no new `components/*.ts` | | `common/lib/corpus.ts` | S1 only | it moves the hub types, so S2c never opens it | | `common/bin/compose-site.ts`, `compose-hub.ts` | S2c only | | | `mcp/src/source.ts` | S1 only | | | `mcp/src/search.ts` | S3 only | | | `common/lib/settings.ts`, `laneMigration.ts`, `SiteSettings` | S0-pause only | zero overlap with S1–S3 | ## Slices ### S1 — contract, reader, hub types, stubs Files: `common/lib/archive/*`, `common/lib/corpus.ts` (URL functions + hub types moved), `mcp/src/source.ts` → re-export, the stubs. Commits, in order: (1) `contract.ts` + `io-stats.ts` + the `corpus.ts` URL functions; (2) the move of `source.ts` into `reader.ts` / `reader-fs.ts` / `reader-hub.ts` with the node imports isolated; (3) `mcp/src/source.ts` as the re-export; (4) the stubs. Tests to ADD: `archive/reader.test.ts` over an in-memory `ArchiveReader` (`Map`-backed): the manifest/page walk, `PromiseMap` sharing (one fetch for concurrent callers), LRU eviction, cached-404-as-null for digests. `archive/contract.test.ts` for both URL shapes (absolute with `siteUrl`, root-relative without). S1 also commits the composed fixture site under `plans/tools/` (see Verification) so the parallel slices reuse it instead of each rebuilding it. ### S2a — the eight caches, `offlineCache`, the SW guard The eight caches become `memo(originId, () => reader.X(...))` — the memo stays, the walk goes. `offlineCache.ts` enumerates every layer that has a manifest plus `ROOT_FILES` (`/duplicates.json`, stats) — this closes the duplicates-page-dead-offline gap. `export/service-worker/site-sw.js:27` `SHARD_RE` and the evict prefixes (`:138-142`) widen to match, and so does `sw-hub.js:26`, which today lacks `digests` altogether. The SW layer list stays hand-written (a service worker cannot import), guarded by a `contract.test.ts` case that reads both files under `export/service-worker/` and asserts equality with `CONTRACT.layers`. (`export/public/sw.js` is the gitignored composed copy — never edit it.) Not in scope: refactoring `SearchDataContext` / `SearchSessionContext` / `siteRegistry` state. They keep their state; only the fetch walk moves. Tests to ADD: one wrapper test per cache shape — same promise on a hit, no double fetch on a miss. ### S2b — `cues.mjs` stops swallowing ENOENT `fromLocal` (`cues.mjs:239-243`) currently swallows a missing `channels//data/` and falls through to HTTP, which is how a relocated channel silently cuts from the wrong cues. Replicate `assertChannelMediaReachable`'s checks (`common/lib/channelMedia.ts:319`) in ~15 lines of plain `.mjs`: read `dataDir` from `config.json`, resolve it the way the editor does, **throw** when unreachable. Cross-reference from a comment in `channelMedia.ts` so the next change to the check finds its twin. Leave `pageFileName` / `pageUrlFrom` alone. Record in the slice note the full `tsx` change list for Phase 5: six shebangs; `driver.mjs:70,93,99,105,140,167`; `umtool.mjs:516`; `clip-bench.spec.ts:146`; root `test:scripts`; the missing `dependencies` field in `umtool/package.json`. Tests: new `cues.test.mjs` cases. It stubs `fetchImpl` and asserts the fetch sequence, so "unreachable throws instead of fetching" is directly assertable. ### S2c — `shipsPwa`, `_headers`, one hub entry type One `shipsPwa(site)` in `contract.ts` replaces `compose-site.ts:207-209` and `export/app/lib/mode.ts:26-29` (which passes `currentSite()`). One generated `_headers` block replaces `compose-site.ts:62-85` and `compose-hub.ts:29-48` **and adds** `/digests/*` and `/duplicates.json` to the CORS set — that is a wire change: its own commit, with a before/after `curl -I` in the commit body. One hub entry type (S1 moved it to `contract.ts`; S2c deletes the local copy in `compose-hub.ts:53`). Files: `compose-site.ts`, `compose-hub.ts`, `export/app/lib/mode.ts`. ### S3 — one search pipeline `common/lib/search/`: - `policy.ts` — `MAX_PAGES 400`, `HARD_VIDEO_CAP 2000`, `WINDOW_LINE_CAP 200`, `truncate 240` from `mcp/src/search.ts:259-268,456` exported as `MCP_POLICY`; the viewer passes its own, uncapped. - `evalTree.ts` — `search.ts:1189-1338` plus `lib/searchEval.ts`'s copy, once. - `window.ts` — over `lib/transcriptWindow.ts`. - `rank.ts`, `collapse.ts` — `search.ts:477` + `SearchResults.tsx:502`. Fetchers are INJECTED (`{ reader, policy, onProgress }`) so react-query stays in `components/`; reuse `lib/concurrency.ts` `mapConcurrent`. `components/searchPipeline.ts` keeps the react-query controllers and `runLeafPipeline` wraps `lib/search/`. `lib/searchEval.ts:35-38` imports `lib/search/` instead of `components/searchPipeline` — and then `"components"` is added to `FORBIDDEN.lib` in `common/architecture.test.ts:31-35` (re-grep for other `lib → components` imports first; that inversion is the slice's deliverable, not a test to bend). `mcp/src/search.ts` keeps every export its 13 tools use and becomes orchestration; caps and concurrency are passed, not re-derived, so the bench's structural counts are unchanged by construction. `mcp/src/search.test.ts` and `scanPlan.test.ts` stay in place. Tests to ADD: `lib/search/*.test.ts` ported from `search.test.ts` cases over the in-memory reader. ### S0-pause — the legacy pause fields (parallel, not Phase 2) Delete `transcriptionsPaused`, `downloadsPaused`, `digest.digestsPaused`, `backfill.enabled` and `legacyGateHeld` per `one-core-phase-1.md:1130-1139`. Files: `common/lib/settings.ts`, `common/lib/laneMigration.ts`, `SiteSettings`. Precondition, re-checked after the first boot of merged code: all four `autoQueue..held` keys present in the live `settings.json` (true on 2026-09-12: transcription/download `held: true`, digest/backfill `held: false`). Own review. ## Invariants Phase 2 burns none of the 11 architecture allow-list entries and adds none. It must not: - touch `buildIndex.ts`'s writer walk; - change any URL shape (the export e2e route-mocks shard URLs at `export/e2e/helpers.ts:87-90`); - change `corpus.json` output or bump any `CONTRACT` version; - add a per-record fetch; - let `reader-fs.ts` be reachable from a client module. ## Verification Every slice: - `pnpm -r exec tsc --noEmit`; common tests (the architecture test included); mcp tests (205); `pnpm test:scripts`; **`pnpm --filter export exec next build`**. - `pnpm --filter yt-dlp-transcript-mcp bench --json` before and after, with the structural read/byte counts diffed (wall ms is noise). - `compose-site` on the FACTS.md fixture recipe — copy `editor/e2e/fixtures/test-transcripts/one-youtube-channel-with-data/channels`, hand-write `sites/testsite/site.json`, `build-index.ts` then `compose-site.ts` — byte-identical modulo `generatedAt`: 16 files under `public/`, 7 under `index/`. S1 commits the composed fixture under `plans/tools/`; the others diff against it. Then, behind the e2e queue lock from a worktree: export e2e + hub + 2origin (S1, S2a, S2c, S3); the editor `export-search` and `export-player-platform-cache` specs (S2a, S3); umtool e2e (S2b). Live contract: `plans/tools/jeralyzer-corpus-2026-09-12.json` is `curl -s https://jeralyzer.pages.dev/corpus.json` taken before S1 (12,380 bytes, spec 3, 30 channels). After S3, a rebuilt jeralyzer's `corpus.json` diffs against it in nothing but `generatedAt` and totals. ## Cadence S1 = one implementer + one reviewer, merged to `main`. Then S2a, S2b, S2c, S3 and S0-pause as parallel implementers on separate branches off that tip, two to three wide per the session budget, each reviewed; e2e stays serial under the queue lock. Standard prompt rules: fish `commit -F`, trailers, never boot against `transcripts/`, detached e2e + Monitor, commit incrementally, add by path, never `next build` in the primary checkout. ## Record Filled in as slices ship: sha range, actual gate numbers, every divergence from this plan. ### S1 — shipped 2026-09-12 Branch `one-core/phase-2-s1`, off `c7f7b90` (the `integrate/2026-09-storage-priority` tip). Six commits, `8d60ad9` → `85891df` (this note), unmerged. **No URL shape moved, no `corpus.json` byte moved, no CONTRACT version moved, no architecture allow-list entry added or burned.** | commit | what | |---|---| | `8d60ad9` | `archive/contract.ts` + `archive/io-stats.ts`; `buildSiteCorpus` calls the URL builders | | `3c631a2` | the reader: `reader.ts` / `reader-fs.ts` / `reader-hub.ts` + `reader.test.ts` | | `43be5f9` | `mcp/src/source.ts`: 1,386 lines → 47, a re-export | | `f2f688d` | the five `lib/search/*` stubs S3 fills | | `f085667` | `plans/tools/compose-fixture-one-youtube-channel/` | | `85891df` | this note | #### Four divergences from the plan above, each because the code said so **1. `contract.ts` OWNS `CONTRACT` and `pageFileName`; it does not re-export them.** The plan (and the umbrella) said re-export from `lib/corpus.ts` / `lib/manifest.ts`. That is a hard ESM failure, not a style question: `corpus.ts` must import the URL builders (that is the point — one definition of the shape it emits), `contract.ts` needs `CONTRACT.pagePad` for `pageFileName`, and `manifest.ts` reads `CONTRACT.manifest` **at module scope**. Any arrangement that leaves `CONTRACT` in `corpus.ts` closes the loop `corpus → archive/contract → manifest → corpus`, and the first module to be imported gets `ReferenceError: Cannot access 'CONTRACT' before initialization`. So the contract module is the BOTTOM of the stack — it imports only `lib/duplicates.ts` — and `corpus.ts` / `manifest.ts` re-export the names. Every existing import site (`from "./corpus"`, `from "./manifest"`, the three `*PageFileName` aliases) is byte-identical. Verified by importing each of the ten modules in the cycle under `tsx`, entry-point by entry-point. **2. The four hub spellings collapse to TWO types plus one `Pick`, not one.** The reason is on the wire. `HubMemberInput` and compose-hub's `HubSiteEntry` are the same direction and the same vocabulary (`siteTitle`/`siteUrl`) — those genuinely become one, and S2c deletes the local copy. But `HubCorpusSite` is the entry **published** in a hub `corpus.json`, and it spells the same member `title`/`url` with two derived pointers beside it. `corpus.json` is frozen, so the emitted shape cannot be renamed to match the input shape. What the slice does instead is make mcp's `HubSite` a `Pick`, so the read-back spelling can no longer drift from the published one. **3. `shipsPwa` reads `process.env.INSTANCE_MODE` bare, with no `typeof process` guard** — unlike `io-stats.ts` and the page-cache knob, which are guarded. The S1 note first claimed Next inlines that expression into the client bundle; the S1 review checked and it does not (`INSTANCE_MODE` is neither `NEXT_PUBLIC_` nor in a `next.config.ts` `env:` block). The real reason the bare read is fine: the predicate is byte-equivalent to `mode.ts:26-29` and `compose-site.ts:207-209`, and its only caller today is `export/app/layout.tsx` — a **server** component — so no browser ever evaluates it. `contract.ts` is not reachable from any browser-but-not-Next context (the service workers import nothing; the search worker keeps its own copy). **S2c: keep `shipsPwa` server-called when it deletes the two copies**, or add the guard then; do not reason from the inlining claim. **4. `stats/` gets URLs without joining `CONTRACT.layers`.** It follows the same manifest → page walk, but `corpus.json`'s `shardScheme` does not document it, so adding it to the published layer list would be a wire change. The builders take a wider `ArchiveTree = ContractLayer | "stats"` instead and the layer list stays frozen. One thing the plan did not mention and that had to change: **common's test glob only reached one directory deep**, so `lib/archive/*.test.ts` would have been collected by nobody. `common/package.json`'s `test` script now also globs `{lib,…}/*/*.test.ts`. Nothing else lives two deep today, so no existing test moved in or out. #### Gates - `pnpm -r exec tsc --noEmit` — clean in all six packages, after every commit. - `pnpm --filter yt-dlp-transcript-common test` — **1077 passed / 0 failed** (baseline 1051 at `c7f7b90`, + 9 `contract.test.ts`, + 17 `reader.test.ts`; none lost). The architecture test passes with its allow-list untouched. - `pnpm --filter yt-dlp-transcript-mcp test` — **205 passed / 0 failed**, unchanged. - `pnpm test:scripts` — **71 passed / 1 skipped**, unchanged. - `pnpm --filter export exec next build` — **compiled successfully**, 11 static pages. This is the only test that `reader-fs.ts` is unreachable from a client module. (The one warning is the pre-existing NFT trace on `next.config.ts → channelMedia.ts → controller/channels.ts → app/offline/page.tsx` — the documented reason this app does not use `output: "standalone"`.) - `pnpm --filter yt-dlp-transcript-mcp bench --repeat 1 --force`, before at `c7f7b90` and after, both `--local` the same composed 1.3 GB site, identical fingerprint (3,358 summaries videos; stats, duplicates and digests all present). **Every structural counter identical, byte for byte:** | case | reads | bytes parsed | |---|---|---| | cold-channels | 0 → 0 | 0 → 0 | | rare, whole corpus | 170 → 170 | 1,335,885,512 → 1,335,885,512 | | common, whole corpus | 170 → 170 | 1,364,679,296 → 1,364,679,296 | | channel-scoped | 4 → 4 | 28,847,054 → 28,847,054 | | date-scoped (filter-first) | 32 → 32 | 213,451,503 → 213,451,503 | | state-scoped (filter-first) | 9 → 9 | 73,399,988 → 73,399,988 | | enumerate, whole corpus | 170 → 170 | 1,364,679,296 → 1,364,679,296 | | get_transcripts × 20 ids | 4 → 4 | 28,847,054 → 28,847,054 | The per-case scan notes match too, `pruned` flags included (date-scoped 28 pages pruned, state-scoped 9). Wall ms is noise and is not quoted. - **compose-site byte-identity** over the FACTS.md fixture recipe, at `c7f7b90` and at the tip: 16 files under `public/`, 7 under `index/`, identical modulo the build clock. The artifact and the normalising diff are committed at `plans/tools/compose-fixture-one-youtube-channel/`; re-running the README's own commands into two fresh temp dirs reproduces it. - **e2e**, behind the queue lock from the worktree (port block #9 — 3901/3911/3920): export `e2e` **172 passed / 0 failed**, `e2e:hub` **5 passed / 0 failed**. `e2e:2origin` **could not run**, and the reason is not this slice — see below. #### `e2e:2origin` is red on the base commit, for a reason S1 does not touch It never reaches a spec. `playwright.2origin.config.ts:16` calls `e2e-2origin/globalSetup.ts:142`, which shells `pnpm run build:hub`, and the hub build dies prerendering `/ask`: ``` Error: useSearchSession must be used within a SearchSessionProvider Export encountered an error on /(workspace)/ask/page: /ask, exiting the build. ``` **Verified on `c7f7b90` itself** — `git switch --detach c7f7b90` in this worktree, `pnpm --filter export run build:hub`: the same error, on the same page, from the same two chunks (`SearchSessionContext`, `AskChat`). This is the "hub `/ask` broken on main blocks `e2e:2origin`" failure already recorded with the export responsive redesign, and nothing in S1 is in that path: the reader never enters a React tree, and the provider is `common/components/SearchSessionContext.tsx` — S2a/S3's file, untouched here. (Note for those slices: four paths in §Corrections above are wrong, checked 2026-09-12 — `SearchSessionContext.tsx` and `SearchDataContext.tsx` are under `common/components/`, not `export/app/lib/`; `useAskChat.ts` is `export/app/ask/useAskChat.ts`; `siteRegistry.ts` is `common/components/`, not `common/lib/`; and the deliberate worker copy is `common/components/searchIndex.worker.ts`, not `export/app/lib/`. The files all exist; only the directories in the plan are wrong.) Worth noting what it DOES prove: the site-mode `next build` passed here and the HUB-mode one compiled and type-checked before the prerender — so the bundle resolves `lib/archive/*` in both modes and never pulls `reader-fs.ts` into a client chunk. The prerender is the only step that fails, and it fails at base. `plans/tools/jeralyzer-corpus-2026-09-12.json` was not re-fetched; the live diff is S3's gate, after the search pipeline lands. ### S2b — shipped 2026-09-12 Branch `one-core/phase-2-s2b`, off `7f86aef` (the `integrate/2026-09-storage-priority` tip with S1 merged). Two commits, `745677f` → `fa3873a` (this note), unmerged. Nothing outside `umtool/report-to-video/` moved except one comment in `common/lib/channelMedia.ts` and this note. **No URL shape, no `corpus.json` byte, no CONTRACT version, no architecture allow-list entry.** | commit | what | |---|---| | `745677f` | `checkChannelReachable` in `cues.mjs` + 7 test cases; the cross-reference comment | | `fa3873a` | this note | #### What the guard actually refuses, and what it deliberately does not `fromLocal` now calls `checkChannelReachable(slug)` before reading the cue file. It replicates `inspectChannelMedia` / `assertChannelMediaReachable`'s statuses in plain `.mjs` — `dataDir` read straight out of `config.json`, `.relocating.json` marker, `lstat` on `data/`, `readlink` compared against the configured target, `stat` on the deep `//data` path — and throws a `CueLookupError` naming the slug and the path. A `CueLookupError` carries no `code`, so `load`'s `ENOENT`/`ENOTDIR` fallback rethrows it instead of reaching for the archive; the test asserts **zero** fetches. Checked once per channel per run, memoised as a promise so a rejection replays for every clip of that channel. Throws: a symlinked `data/` whose target is missing (the unmounted drive — the bug), or is not a directory, or disagrees with `config.json`; a `dataDir` in `config.json` with no `data/` at all; a real `data/` directory with a `dataDir` recorded anyway; a `.relocating.json` marker. Does **not** throw, and this is the deliberate line: a channel whose `data/` is simply absent with no configured target. That covers both "no `channels/` dir at all" (a clone with no corpus — a first-class way to use this repo) and "a corpus that does not mirror this channel", and it is exactly what the editor's `inspectChannelMedia` calls `in-place` and `assertChannelMediaReachable` passes. Deciding otherwise would have hard- failed every clip of a non-mirrored channel for any operator who has a corpus at all, and it would have made the twin a near-copy rather than a copy. A reachable channel that merely lacks THIS video still falls through to HTTP, as before — two tests pin both. `--cue-source local` is untouched: it still refuses to fall back, with its own message, and a test pins that the guard does not swallow it. `pageFileName` / `pageUrlFrom` untouched. `common/lib/channelMedia.ts` gains a comment above `assertChannelMediaReachable` pointing at the copy and saying why it is a copy (umtool's bins run under bare node). The function is not changed. #### The `tsx` change list for Phase 5 Every entry re-grepped at `745677f`; three paths in the S2b brief were wrong and are corrected here. | what | where | note | |---|---|---| | six shebangs | `umtool/report-to-video/{build-video,check-availability,compose-chrome,render-cards,resolve-windows,verify-build}.mjs:1` | all `#!/usr/bin/env node`. 40 MORE node shebangs live under `umtool/` (`bin/umtool.mjs`, `e2e/fixtures/make-fixture.mjs`, 38 under `song/`) — the "six" is report-to-video only | | six spawn argv | `umtool/lib/report/driver.mjs:70,93,99,105,140,167` | **not** `umtool/report-to-video/driver.mjs` — there is no such file | | one printed hint | `umtool/bin/umtool.mjs:516` | **not** `umtool/umtool.mjs`; prints `node umtool/report-to-video/resolve-windows.mjs …` | | one `execFileSync` | `umtool/e2e/clip-bench.spec.ts:145` (script path on `:146`) | | | the test runner | root `package.json:24` — `test:scripts` = `node --test scripts/*.test.mjs umtool/report-to-video/*.test.mjs` | | | the missing deps | `umtool/report-to-video/package.json` | it has NO `dependencies` and no `devDependencies` at all, so `tsx` has nothing to be declared in. `umtool/package.json` does have `dependencies` — the brief named the wrong file | Not made here, per the brief. #### One thing this slice did not close `fromLocal` is not the only place that joins `channels//data//transcript.cues.json` by hand and treats a failure as "no cues": `umtool/report-to-video/check-availability.mjs:54` (`cueMeta`) and `umtool/lib/projects/report.mjs:124` (`cuePathFor`) do the same, and an unmounted relocated channel still reads to them as an absent file. Neither answers a cue WINDOW — they feed availability rows and the project index — so neither can cut a clip in the wrong place, which is why they are out of this slice. They want the same guard when umtool's local-corpus access is next touched. #### Gates - `pnpm -r exec tsc --noEmit` — **clean** in all eight packages. - `pnpm test:scripts` — **78 passed / 1 skipped** (baseline 71/1; +7, the new cases). This is where the cues tests are collected; `umtool/package.json` has **no `test` script** of its own (only `dev`/`build`/`start`/`typecheck`/`e2e`), which is the one divergence from the brief's "umtool's own test command". - `pnpm --filter yt-dlp-transcript-common test` — **1077 passed / 0 failed**, unchanged. - `node --test umtool/report-to-video/cues.test.mjs` alone — **21 passed / 1 skipped** (the LIVE network case is the skip, as before). - umtool e2e, behind the queue lock from worktree #11 (`UMTOOL_E2E_PORT=4151`, via `pnpm wt run`), with `SONG_DIR=/home/user/reports/quartering-uh-song/data`: **129 passed / 40 failed / 2 skipped** — and the SAME numbers with the SAME 40 test names at the base commit `7f86aef`, diffed line for line (`git switch --detach 7f86aef` in this worktree, same command, same env). This slice moves nothing in that suite. The 40 are environmental and were already recorded: the heavy song fixtures are not on this machine (`$SONG_DIR/{wav48,asr,media}` do not exist, only `cand2/` does, and `make-fixture.mjs` symlinks those three only `if (existsSync)`), so every spec that needs audio, ASR or the face detector fails — `find` 11, `triage` 9, `faces` 6, `usage` 5, `browse` 3, `undo` 3, `deck` 2, `projects` 1, with `no media for v1`, `detect v1@60.00 -> 404` and a missing `every note (N)` table. What DID run, and passes on both: the whole of `build.spec.ts`, `clip-bench.spec.ts` (which shells `resolve-windows.mjs` with `CHANNELS_DIR` pointed at the fixture corpus — the one spec that exercises the changed path end to end) and `report-longform.spec.ts`. Not run, and deliberately: the export `next build`, the mcp bench, the compose-site fixture diff and the export/hub/2origin e2e suites that §Verification asks of every slice. Nothing in S2b is reachable from any of them — the diff is one `.mjs` file under `umtool/`, its test file, and a comment. `pnpm -r exec tsc --noEmit` covers the one TypeScript file touched. #### Divergences from the S2b brief 1. **`umtool` has no `test` script.** The brief said to run "umtool's own test command"; `umtool/package.json` has `dev`, `build`, `start`, `typecheck` and `e2e` only. The cues tests are collected by the ROOT `test:scripts`, which is the number quoted above, and were also run directly with `node --test`. 2. **Three paths in the Phase 5 `tsx` list were wrong** — `driver.mjs`, `umtool.mjs` and the package missing `dependencies`. Corrected in the table above, each re-grepped. 3. **A missing channel dir does not throw.** The brief's "missing, or `dataDir` set but not mounted" reads as if an absent channel dir should also fail. It does not, for the reason given above: the editor's twin calls that `in-place`, and throwing would break the archive-only workflow this repo advertises. The case that motivated the slice — a relocated channel whose drive is gone — throws. 4. **Test fixtures use `tmpdir()`, not the agent scratch dir.** A test that hard-coded a job scratch path would not survive the job. `CUES_TEST_DIR` overrides the root, and that is what the scratch-dir runs used. ### S2c — shipped 2026-09-12 Branch `one-core/phase-2-s2c`, off `7f86aef` (the `integrate/2026-09-storage-priority` tip, S1 merged). Four commits, `706a9ed` → `d1b3903`, plus this note, unmerged. **One deliberate wire change — two CORS lines, described below — and nothing else moved: no URL shape, no `corpus.json` byte, no CONTRACT version, no architecture allow-list entry, no new `components/*.ts`.** | commit | what | |---|---| | `706a9ed` | compose-hub's local `HubSiteEntry` deleted; it uses S1's `HubMemberInput` | | `674aeb4` | one `shipsPwa()` — compose-site.ts's and mode.ts's copies deleted | | `a168358` | `lib/archive/headers.ts`: one `_headers` generator, byte-identical output | | `d1b3903` | **wire change**: `/digests/*` and `/duplicates.json` get CORS | #### The `_headers` before/after The two additions are the whole diff, for the site. Composing the FACTS.md fixture recipe and diffing against `plans/tools/compose-fixture-one-youtube-channel/public/_headers`: ``` 5a6,7 > /duplicates.json > Access-Control-Allow-Origin: * 12a15,16 > /digests/* > Access-Control-Allow-Origin: * ``` Four added lines; every other line byte-identical, and the rest of the composed dir IDENTICAL modulo the build clock (16 files under `public/`, 7 under `index/`). The **hub** block does not move at all: compose-hub at `7f86aef` and at the tip produce a byte-identical `_headers` and an identical `hub-sites.json`, with `corpus.json` differing only in `generatedAt`. Why the gap existed and why no test had caught it: `_headers` only exists on the CDN, and every local server in this repo is more permissive than Cloudflare. `serve` — what the export, hub and 2-origin suites all run behind — gives `**/*.json` a blanket `Access-Control-Allow-Origin: *` (`export/serve.json`). So a cross-origin viewer could read a federated site's transcripts, subs, posts, summaries, stats and archives, and got a CORS failure on its digests and its duplicates report, and nothing local could reproduce it. **`curl -I` cannot show this, and the reason is worth recording** rather than quoting a run that proves nothing. `wrangler pages dev` is the only local server here that reads `_headers` at all, and it adds `Access-Control-Allow-Origin: *` to every response of its own accord — a file named in no rule whatsoever still comes back with the header (checked with a `zzz-not-in-headers.json` dropped into the same composed dir). Before and after are indistinguishable over HTTP locally. What wrangler does report is its own parse of the file, on the composed fixture: ``` before: ✨ Parsed 12 valid header rules. after: ✨ Parsed 14 valid header rules. ``` Cloudflare's own parser, counting the two new rules as valid. The real before/after is a deploy. #### Divergences from the S2c brief **1. The generator is `lib/archive/headers.ts`, not `contract.ts`.** The brief allowed either. It is its own module because it carries two ordered path LISTS, not just a renderer: `_headers` is matched top-down, the file is diffed against a committed fixture, and the existing order is not the order `ROOT_FILES` + `CONTRACT.layers` would produce — so deriving the list would have reordered every line and buried the two-line wire change in noise. The contract still gets the last word: `contractCorsPaths()` + a test assert every `CONTRACT.layers` tree and every `ROOT_FILES` document has a line, so adding a layer and forgetting its CORS entry now fails a test rather than a deploy. No `node:*` import either way. **2. The hub does NOT get the two new lines.** The brief's wire change is "add `/digests/*` and `/duplicates.json` to the CORS set"; applied to the hub that would declare headers for paths a hub never serves. A hub holds no shard data — it reads every member cross-origin at runtime — so its block is unchanged, and the hub half of the fixture diff is empty. (`HUB_CORS_PATHS` does still list four shard trees the hub does not serve either; those are pre-existing and left alone, since removing them would be a second, unrelated change to the same file.) **3. `mode.ts`'s hub short-circuit is deleted, not kept.** The brief said to replace the copy with a call. The replacement made `if (instanceMode() === "hub") return true;` dead weight: `currentSite()` already returns `hubSite()`, which sets `pwa: true`, and the contract predicate reads `INSTANCE_MODE` itself. Same answer in both modes, so `shipsPwa()` is now one line. **4. `shipsPwa` stays server-called, and `contract.ts`'s comment about it is corrected.** No `typeof process` guard was added, per S1's divergence 3 — the callers are `compose-site.ts` (a build script) and `export/app/lib/mode.ts`, whose only caller is `export/app/layout.tsx`, a server component; `currentSite()` reads the sites dir, so `mode.ts` could not be client-side regardless. The comment S1 left on `shipsPwa` still asserted that Next inlines `process.env.INSTANCE_MODE` into the client bundle and that a guard would therefore break hub mode; the S1 review established that is false. That comment is rewritten to say why the bare read is actually safe and what a future client caller would owe. **`contract.ts`'s diff in this slice is comment-only** — verified by diffing with comment lines filtered out. #### Gates - `pnpm -r exec tsc --noEmit` — clean, exit 0, all six packages, after every commit. - `pnpm --filter yt-dlp-transcript-common test` — **1083 passed / 0 failed** (baseline 1077 at `7f86aef`, + 6 `headers.test.ts`; none lost). The architecture test passes with its allow-list untouched. - `pnpm --filter yt-dlp-transcript-mcp test` — **205 passed / 0 failed**, unchanged. - `pnpm test:scripts` — **71 passed / 1 skipped**, unchanged. - `pnpm --filter export exec next build` (site mode) — **compiled successfully**, 11 static pages, 9 routes. The only warning is the pre-existing NFT trace S1 documented. - `pnpm --filter export run build:hub` — **compiled successfully in 4.2s**, TypeScript finished, then dies at exactly the known step: `Error: useSearchSession must be used within a SearchSessionProvider` prerendering `/ask`. Red on the base commit for a reason in `common/components/SearchSessionContext.tsx`, which this slice does not open; see S1's note. Compile and type-check are the part this slice needs, and they pass. - **compose-site / compose-hub fixture diffs** — above. - **e2e**, behind the queue lock from the worktree (port block #12 — editor 4201 / export-e2e 4220): export `e2e` **172 passed / 0 failed** (5.5 min), `e2e:hub` **5 passed / 0 failed**. `e2e:2origin` was **not run**: it shells `build:hub`, which is red on the base for the reason above, so it cannot reach a spec — exactly as S1 recorded. ### S3 — shipped 2026-09-12 Branch `one-core/phase-2-s3`, off `7f86aef` (the `integrate/2026-09-storage-priority` tip with S1 merged). Six commits, `eaa50b6` → `99fe37a`, plus this note; then three review fixes, `172099c` → the commit that corrects this note. Unmerged. **No URL shape moved, no `corpus.json` byte moved, no CONTRACT version moved, no architecture allow-list entry added — and `"components"` is now in `FORBIDDEN.lib`, so the list of forbidden edges grew while the list of excused ones did not.** | commit | what | |---|---| | `eaa50b6` | `lib/search/{policy,window,evalTree,rank,collapse,leafPipeline}.ts` + a test file each | | `7252b57` | `mcp/src/search.ts`: 1,563 lines → 1,206, orchestration only; the caps are `MCP_POLICY` | | `07075d6` | the inversion: four `lib → components` edges deleted, `"components"` added to `FORBIDDEN.lib` | | `fb6a31d` | the unused `LeafMatcher` import the previous commit left | | `99fe37a` | the `SearchMode` doc comment travels with the type | | `726de3e` | this note, as first written | | `172099c` | **F3** — `lib/search` holds no caller's budget as a default | | `752ceb1` | **F1** — the result list calls `rankByUploadDateDesc` | | _(this commit)_ | **F2** + the F1/F3 corrections to this note | #### What the slice actually found, where the brief and the code disagreed **1. `searchEval.ts` is not a COPY of the mcp evaluator, and merging them would have been wrong.** The plan says `evalTree.ts` should hold "`search.ts:1189-1338` plus `lib/searchEval.ts`'s copy, once". Read side by side they are two algorithms over two data shapes: mcp's `evalNode` answers *does this record match*, given a record it already holds; searchEval's `evaluateSlugs`/`orchestrate` answer *which slugs survive*, streaming, over per-leaf network pipelines, because the browser cannot hold 1.3 GB of transcripts. Unifying them means materialising the corpus in the browser. What the slice does instead: the per-record evaluator moves into `lib/search/evalTree.ts` once, and **each file names the other as its sibling and states the rule** — the meaning of AND / OR / negate changes in both together, or it has silently forked. **2. `rank.ts` had one implementation to collect, not two.** The stub says the mcp ordering and the viewer ordering "are the same intent written twice". mcp does not rank at all: `searchTranscripts` emits in corpus scan order and `SearchResult` has no sort. The orderings that DO exist are the viewer's newest-first over `uploadDate` (`SearchSessionContext.tsx`, inline) and `getThread`'s oldest-first over `createdAt` tie-broken by id (`search.ts`, inline). Both are now comparators in `rank.ts` and both call sites call them. **No relevance ranking was invented** — that would have changed what every caller returns under cover of a refactor. > **Corrected after review (F1).** As first written this note was wrong > about its own slice: `getThread` was converted but > `SearchSessionContext.tsx:886` still sorted inline, so `rank.ts` shipped > with the viewer's ordering in it and zero viewer callers — a second copy > with better documentation, not a shared comparator. Fixed in `752ceb1`; > the helper IS the inline ternary, sorts in place and returns the same > array, and `Array#sort` is stable, so same-date rows keep the order the > two append loops built. **3. `collapse.ts` has one consumer, and the viewer's "duplicate-collapse" is not one.** `SearchResults.tsx:502` is `duplicateSiblings(...)` from `components/duplicatesCache.ts` — it decorates a card with a sibling *count* and a jump menu. It does not remove a row, and the result list is never collapsed. So there was no second implementation to fold in. What moved is mcp's `collapseDuplicates`, now **pure and synchronous** and generic over a `CollapsibleHit`: the caller supplies the already-fetched duplicate index, so the rules are testable without a transport and the viewer can adopt them when it wants them, rather than having them imposed by this slice. **4. A sixth module was required: `lib/search/leafPipeline.ts`.** The inversion is impossible without it. `searchEval.ts` does not merely CALL `runLeafPipeline` — it is typed on `LayerHit` and `LeafController`, so injecting the function alone leaves the import. Moving the browser's streaming leaf scanner down (three worker-pool drivers + the leaf adapter, ~450 lines) is what makes `lib/` self-contained; `components/searchPipeline.ts` is now 65 lines that answer one question — WHICH FETCH — and supplies `searchRuntime`. The brief allows new files under `lib/search/`; no new `components/*.ts` was created, so `common/package.json` is untouched. **5. Four back-edges, not one.** The brief named `lib/searchEval.ts:35-38`. Re-grepping `lib/` found four, and all four had to go before `"components"` could join `FORBIDDEN.lib`: | edge | what it was | how it inverted | |---|---|---| | `searchEval → components/searchPipeline` | `runLeafPipeline`, `LayerHit`, `LeafController` | pipeline moved down; the fetch is injected as `SearchRuntime.runLeaf` | | `searchEval → components/searchLayerCache` | `cacheKey` / `getCached` / `getCachedSync` / `putCached` / `CachedResult` | the memo is injected as `SearchRuntime.cache`; `CachedResult` moved to `searchEval` (an IndexedDB store does not get to define what a leaf result IS) and `searchLayerCache` re-exports it | | `aiHandoff → components/searchPipeline` | `LayerHit` (type) | import path only | | `searchQuery → components/urlState` | `SearchMode` (type) | moved to `searchQuery`, which interprets it; `urlState` re-exports | `runQueryTree` gains one required `runtime` field. Its three callers pass `searchRuntime`: `SearchSessionContext.tsx`, `charts/useSearchSeries.ts` and `export/app/lib/askRetrieval.ts`. No other import site moved: every name `components/searchPipeline` exported **that anything imported** it still exports, including the `LayerHit` that `askRetrieval.ts` imports from it. > **Corrected after review (F2).** The first wording said "every name it > exported", which is not true and worth being exact about: three exports > were REMOVED — `createSearchPipeline`, `createPostsSearchPipeline` and > `createSubsSearchPipeline` (`7f86aef:searchPipeline.ts:54,300,454`). > Grepping every package found no importer of any of them; they were only > ever reached through `runLeafPipeline`, which dispatches on the leaf's > scope. So they moved down to `lib/search/leafPipeline.ts` and stayed > module-internal to it rather than being re-exported from a binding whose > whole point is to answer one question. Re-exporting three functions > nobody calls would have been publishing a surface to preserve a > sentence. **6. `window.ts` keeps TWO excerpt shapes on purpose.** The stub hoped "the viewer's excerpt and the MCP's excerpt are the same excerpt". They are not the same function: the MCP takes a ±45 s window around every matched cue and merges them (the paragraph a sweep cites); the viewer emits one row per matched cue, widened to the neighbouring cues ONLY when the match straddles a caption break (the line a reader scans). Collapsing them changes both outputs, and the viewer's is pinned by the export e2e's rendered hit text. One module owns both, one `truncate`, one `clock`, no duplicated code — and the file says out loud why they stay two. **7. `policy.ts` also names the uncapped one.** The plan said "the viewer passes its own, uncapped". `VIEWER_POLICY` exists so there is one place that says what uncapped MEANS, and a test pins the load-bearing consequence (`truncate(t, Infinity)` is the identity; `0 >= Infinity` is false). > **Hardened after review (F3).** As first written, `truncate`'s `max` and > `RecordCtx.snippetChars` defaulted to `MCP_POLICY.snippetChars`, so > `lib/search/` held one caller's budget as the shared default — and the > failure mode was the bad kind: a viewer adopter that forgot to pass its > policy would not break, it would silently clip every excerpt at 240 with > the whole suite green. All three widths (`truncate`'s `max`, > `windowedTranscript`'s `maxLines`, `RecordCtx.snippetChars`) are now > REQUIRED, `MCP_POLICY` is imported nowhere under `lib/search/` except by > the module that defines it, and two tests run the same input under both > policies and assert the widths differ. Fixed in `172099c`; mcp already > passed its policy explicitly at every site, so **no read path changed and > the bench counters cannot move**. **8. `buildScanPlan` stayed in `mcp/src/search.ts`.** It is reader-driven orchestration, `scanPlan.test.ts` sits beside it, and it calls the single `passesFilters` from `lib/search/evalTree` — so the pruner and the scanner still cannot disagree about what matches, which was the one property that mattered. ### S2a — shipped 2026-09-12 Branch `one-core/phase-2-s2a`, off `7f86aef` (the `integrate/2026-09-storage-priority` tip, S1 merged). Four commits, three code commits `dc3aef1` → `2eb9c3b` plus this note, unmerged. **No URL shape moved, no `corpus.json` byte moved, no CONTRACT version moved, no architecture allow-list entry added or burned, and no `common/package.json` exports line added.** | commit | what | |---|---| | `dc3aef1` | the eight caches — plus `SearchDataContext`, `siteRegistry` and `DuplicatesClient` — onto one `RemoteSource` per origin; `lib/archive/readers.ts`; the reader's raw document reads | | `8ca7022` | `offlineCache` enumerates `ARCHIVE_TREES` + `ROOT_FILES`; both service workers widened; the SW and search-worker guard cases in `contract.test.ts` | | `2eb9c3b` | `archive/readers.test.ts` + `components/archiveCaches.test.ts` | | (this commit) | this note | #### Seven divergences from the plan above, each because the code said so **1. The caches do NOT adopt `ArchiveReader`'s tolerant methods. The reader gained a block of RAW DOCUMENT READS underneath them instead.** The slice was specified as `memo(originId, () => reader.X(...))`, and for transcripts that is exactly what it is. For everything else `ArchiveReader` is the wrong surface, twice over. It has no method at all for four of the documents the viewer reads: `/subs/manifest.json` and `/posts/manifest.json` (the SITE-level index of which channels ship a tree — a different document from a channel's own manifest), and the raw summaries/stats manifest-plus-pages. What it exposes for those is `videoIndex()` / `statsIndex()`, folded slug-keyed maps, which are the wrong shape for a UI that renders arrays and loads page by page under react-query. And where a method does exist, its answer to absence is the opposite of the viewer's. `subsManifest`/`postsManifest`/`digestsManifest` resolve `null` for anything that is not a 200 — right for a tool probing thirty channels, wrong for a browser, because **react-query retries a rejected query and will never retry a resolved `null`**. Folding both policies into one method is exactly how a transient blip becomes a panel that is empty for the rest of the session. So `RemoteSource` now has twelve one-to-three-line reads that do URL + fetch + parse and THROW, and every tolerance policy sits on top and picks its own answer. **The MCP's behaviour is unchanged by construction**: each tolerant method is rebased on the raw read it duplicated, and `buildVideoIndex` / `buildStatsIndex` already treat a throw from their readers as absence, so the degradation paths are the same code they were. This is not the `record(layer, slug, id)` the plan forbids — these are the DOCUMENTS of the walk, not records inside a shard, and no read count moves. `ArchiveHttpError` came with them, for the one caller that has to tell a 404 from a dropped connection (the alias dictionary, whose query has `staleTime: Infinity`). Its `message` is byte-identical to the `Error` it replaced. **2. Two memos stay in `components/`, and neither is a leftover.** The subs/posts CHANNEL-manifest memos keep the throwing policy of (1). The posts and digest PAGE memos exist because the reader caches subs and transcript pages but not those two — and `fetchThread` walks every page of a channel, so without a memo opening two posts in one thread re-downloads the whole channel. Adding those caches to `RemoteSource` instead would have moved the bench's read counts, which is not a thing to do in a slice that claims the walk is all that changed. **3. `io-stats` blind spots are preserved deliberately.** The raw reads take an OPTIONAL `kind`, and the ones that pass none are exactly the reads that recorded nothing before (the summaries/stats/duplicates/alias folds used a bare `fetch`). Closing them would move the bench's structural counters for a reason unrelated to the walk. That is also why the bench was not re-run for this slice — nothing in the reader's semantics or read counts moved. **4. `contract.ts` needed `treeManifestUrl`, and `ARCHIVE_TREES` / `PER_CHANNEL_TREES` beside it.** `manifestUrl("subs")` throws with no slug, and correctly so — but `/subs/manifest.json` is a real published document. A per-channel tree has BOTH a root manifest and a per-channel one; `manifestUrl` builds the second, `treeManifestUrl` the first, and for a flat tree they are the same file. The two lists are what the offline cache and the SW guard test enumerate; `CONTRACT.layers` stays the frozen published list. **5. The service workers needed more than a wider `SHARD_RE`, and the plan's stated goal is why.** `SHARD_RE` matches three path segments, so `/duplicates.json` and `/summaries/*` never matched it and the fetch handler let them through to the network — downloading a root file into the cache and never serving it from there closes nothing. Both workers gain `FLAT_RE` and `ROOT_RE` answered NETWORK-FIRST (those documents have no per-channel `generatedAt` to evict against, so cache-first would be stale until the SW version moved). Three drifts surfaced while writing the lists down. `summaries` was in site-sw's `SHARD_RE` and in both eviction prefix lists: dead in both places, because it is a flat tree and `/summaries//` does not exist. `/posts/` and `/digests/` were absent from both eviction lists, so a rebuilt channel kept serving its old posts and digests from cache. And sw-hub had no `digests` at all — the hub federated a layer it could never cache. **6. `offlineCache` keeps its own `cache: "no-store"` fetch** rather than taking the shared reader. The reader is a plain `fetch`, so it would be answered from the very service-worker cache that function exists to refill, and would compute its download list from the stale copy. What is shared is the thing that actually drifted: the URL shape. **7. One new `components/*.ts`, against the invariant, and why it is not the thing the invariant protects.** `components/archiveCaches.test.ts` is a test: it is imported by nothing, the `"./components/*"` catch-all maps to `.tsx`, and it needs no `package.json` exports line — which is the cost the invariant names. It cannot live under `lib/` because S3 adds `"components"` to `FORBIDDEN.lib` in the architecture test, and a test for the caches has to import them. Three walk sites beyond the eight caches moved too, all of them covered by the brief's "only the fetch walk moves": `SearchDataContext.tsx` (four URLs, state untouched), `siteRegistry.ts` (two), and `export/app/duplicates/ DuplicatesClient.tsx`'s bare `fetch("/duplicates.json")`, which became a new `fetchDuplicateReport()` export on `duplicatesCache`. A repo-wide sweep for hand-written archive paths now returns only `searchIndex.worker.ts` — the deliberate copy, and it is pinned. #### Gates - `pnpm -r exec tsc --noEmit` — clean in all six packages, after every commit. - `pnpm --filter yt-dlp-transcript-common test` — **1129 passed / 0 failed** (baseline **1077** at `7f86aef`; + 3 `policy`/`window` width cases, + 8 `window`, + 2 `rank`, + 6 `collapse`, + 21 `evalTree`, + 12 `leafPipeline` = 52; none lost). The architecture test passes with `"components"` in `FORBIDDEN.lib` and the ALLOWED ledger **byte-identical** to the base. - `pnpm --filter yt-dlp-transcript-mcp test` — **205 passed / 0 failed**, unchanged. - `pnpm test:scripts` — **71 passed / 1 skipped**, unchanged. - `pnpm --filter export exec next build` — **compiled successfully**, 11 static pages. The only real test that none of this dragged `reader-fs.ts` or a react-query module into a client chunk. (Prerender needs a composed `export/public`: the committed fixture at `plans/tools/compose-fixture-one-youtube-channel/public/` copied in, which is gitignored — a bare checkout fails on a missing `summaries/manifest.json` before it reaches any code this slice touched.) - **compose-site byte-identity** over the FACTS.md fixture recipe: `IDENTICAL modulo the build clock` for both trees — 16 files under `public/`, 7 under `index/`. Nothing composed moved. - `pnpm --filter yt-dlp-transcript-mcp bench --repeat 1 --force --local `, before at `7f86aef` and after at the tip, both over the same composed hasanalyzer site. **Identical fingerprint** (5 channels, 170 transcript pages, 3,358 summaries videos, stats + duplicates + digests all present) and **every structural counter identical, byte for byte** — including the per-case scan notes and both `pruned` flags: | case | reads | bytes parsed | note | |---|---|---|---| | cold-channels | 0 → 0 | 0 → 0 | | | rare, whole corpus | 170 → 170 | 1,335,885,512 → 1,335,885,512 | 170 pages | | common, whole corpus | 170 → 170 | 1,364,679,296 → 1,364,679,296 | 170 pages | | channel-scoped | 4 → 4 | 28,847,054 → 28,847,054 | 4 pages | | date-scoped (filter-first) | 32 → 32 | 213,451,503 → 213,451,503 | 28 pages · pruned | | state-scoped (filter-first) | 9 → 9 | 73,399,988 → 73,399,988 | 9 pages · pruned | | enumerate, whole corpus | 170 → 170 | 1,364,679,296 → 1,364,679,296 | 170 pages | | get_transcripts × 20 ids | 4 → 4 | 28,847,054 → 28,847,054 | | Wall ms is noise and is not quoted (the box was above the load threshold for both runs, and the bench marked them UNRELIABLE, as designed). **The site S1 benched was gone and had to be rebuilt** — `/home` is at 100 %, and nothing 1.3 GB survived. Recipe, for the next slice: compose `hasanalyzer` READ-ONLY against the primary checkout, by pointing `EXPORT_INDEX_DIR` at a directory holding two symlinks (`shared`, `sites` → the primary checkout's `export/.export-index`), `SITES_DIR` at a **copy** of `sites/hasanalyzer` with `"archives": false` added, `TRANSCRIPTS_DIR` at a throwaway dir holding `settings.json` `{}` plus the corpus-wide `duplicates*.json` / `search-aliases.json`, and `EXPORT_PUBLIC_DIR` at scratch. Nothing writes inside `transcripts/`: the compose cache is keyed off the public dir, the archive builder is off, and the channel signer opens its LMDB under the throwaway root. The BEFORE run reproduced S1's recorded numbers to the byte, which is the evidence that the rebuilt corpus is the same corpus. Delete the 2.3 GB output afterwards. - **e2e**, behind the queue lock from the worktree (port block 4000/4001/4010/4011/4020): export `e2e` **172 passed / 0 failed**, `e2e:hub` **5 passed / 0 failed**, editor `export-search` + `export-player-platform-cache` **21 passed / 0 failed**. Re-run after the three review fixes (F1 touches `SearchSessionContext.tsx`): export **172 passed**, hub **5 passed**, editor `export-search` **19 passed** — all 0 failed. - The bench was **not** re-run after the review fixes, and does not need to be: `git diff fb6a31d..HEAD -- mcp/` is empty, so the benchmarked binary is byte-identical. F3 removed defaults that mcp never took (it passed `policy.snippetChars` and `opts.maxLines ?? MCP_POLICY.windowLineCap` explicitly at every site) and F1 is viewer-only. - `e2e:2origin` was **not run**: it is red on the base for a reason outside this slice (the hub `/ask` prerender — see S1's §Record, verified there against `c7f7b90` itself). Because S3 touches two of the three files that failure names (`askRetrieval.ts`, `SearchSessionContext.tsx`), `build:hub` was run anyway to confirm the failure is the SAME one: `INSTANCE_MODE=hub next build` compiles successfully and type-checks, then dies on the same page with the same message from the same two chunks — `Error: useSearchSession must be used within a SearchSessionProvider`, `Export encountered an error on /(workspace)/ask/page`, `SearchSessionContext` + `AskChat`. Unchanged, and the compile+typecheck passing is itself the evidence that the injected `searchRuntime` resolves in hub mode too. - **The live jeralyzer contract** (S1 left this as S3's gate): `curl https://jeralyzer.pages.dev/corpus.json` is **byte-identical** to `plans/tools/jeralyzer-corpus-2026-09-12.json` — 12,380 bytes, zero structural differences, `generatedAt` included. Be honest about what that does and does not prove: the site has not been rebuilt since the snapshot, so this confirms the snapshot is still current, not that a rebuild would match. **A rebuilt jeralyzer was not attempted** — 30 channels / 30,886 videos, and `/home` is at 100 %. The real evidence that the emitted contract did not move is the compose-site fixture byte-identity above: jeralyzer's `corpus.json` comes out of the same `buildSiteCorpus` the fixture exercises, and S3 does not open `corpus.ts`, `compose-site.ts` or `compose-hub.ts` at all. #### One e2e failure, and why it is not this slice The editor subset failed 1 of 21 on its first run: `export-search.spec.ts:612` "a site's own social links win over the global default", with `ENOENT: … editor/test-settings.json` at `helpers.ts:166`. It is a spec-SUBSET setup dependency, not a regression. That file is created only by `resetData()` (`editor/e2e/helpers.ts:19`), and `export-search.spec.ts` never calls `resetData` — it reads a settings file some earlier spec in the full suite wrote. Running only these two spec files in a fresh worktree can never create it. Seeding it the way the suite does (`cp e2e/fixtures/test-settings.default.json editor/test-settings.json`) and re-running the same subset: **21 passed / 0 failed**. Nothing in S3 touches settings, social links or the footer. #### Notes for the slices still in flight - `components/searchPipeline.ts` is now 65 lines and exports `searchRuntime`. **S2a: if a cache's `fetchX` signature changes, the one place to rebind it is `CACHE_FETCHERS` there** — not inside `lib/`, which no longer knows the caches exist. - `lib/search/leafPipeline.ts` uses bare `setTimeout`/`clearTimeout` rather than `window.setTimeout`. Identical in a browser, and it is what lets the module be driven from a test process with no DOM — which is how `leafPipeline.test.ts` counts page reads over an in-memory `ArchiveReader`. - The two sibling comments (`lib/searchEval.ts` ↔ `lib/search/evalTree.ts`) are load-bearing prose: they are the only thing stopping the two evaluations of the same algebra from drifting, because no test can compare a streaming slug-set walk against a per-record boolean. - `pnpm --filter yt-dlp-transcript-common test` — **1095 passed / 0 failed** (baseline 1077 at `7f86aef`, + 3 `contract.test.ts`, + 6 `readers.test.ts`, + 9 `archiveCaches.test.ts`; none lost). Architecture test green, allow-list untouched. - `pnpm --filter yt-dlp-transcript-mcp test` — **205 passed / 0 failed**. - `pnpm test:scripts` — **71 passed / 1 skipped**. - `pnpm --filter export exec next build` — **compiled successfully**, 11 static pages. The only proof `reader-fs.ts` is unreachable from a client module, and it now has eight more `components/` modules reaching `lib/archive/` to prove it about. (The one warning is S1's pre-existing NFT trace.) - **compose-site byte-identity** over the committed fixture: `IDENTICAL modulo the build clock` for both trees — 16 files under `public/`, 7 under `index/`. - **The mcp bench was NOT run**, per the brief — the reader's semantics and read counts do not move (see divergence 3). - **e2e**, behind the queue lock from worktree #10 (ports 4001/4011/4010/4020): export `e2e` **172 passed / 0 failed** (S1's baseline exactly), `e2e:hub` **5 passed / 0 failed**. `e2e:2origin` was skipped: it is known-red on the base for the hub `/ask` prerender, which S1 verified at `c7f7b90` and which nothing here is in the path of. - The editor specs `export-search` + `export-player-platform-cache`: **20 passed / 1 failed**, and the one failure is a SUBSET-RUN ARTIFACT, not a regression. `export-search.spec.ts:612` opens `editor/test-settings.json` with `readJson`; that file is gitignored and is only ever written by `writeSettings()`, i.e. by an earlier spec in the full suite. A fresh worktree running just these two files has never had one written, so the test dies on `ENOENT` before it touches any code. **Confirmed by re-running the same two spec files at `7f86aef` in the same worktree: the same spec fails the same way.** Worth fixing independently — the spec should seed the file rather than assume a predecessor left one — but it is not this slice's. #### What this slice deliberately leaves for S2c A hub caching a member's `/duplicates.json` or `/digests/*` still depends on those paths being in the composed `_headers` CORS set, which they are not. S2c's `_headers` commit is what makes the widened `sw-hub.js` reach them cross-origin; same-origin (the site SW) works today. The offline DOWNLOAD path has no e2e coverage and gains none here: the service worker registers in production builds only, so `pwa.spec` can assert the control renders but never that a download lands. The guard that the SW lists and the download list agree with the contract is `contract.test.ts`, verified by mutation (dropping `digests` from sw-hub's `SHARD_RE` turns it red). #### Review fixes — `9451be8`, `5ac97b3` Three items came back. One was a real defect and it is worth recording as a lesson about the shape of the thing, not just as a bug. **F1 — an offline copy is TWO lists, and shipping it as one put site-wide bytes on a per-channel bill.** `downloadChannelOffline` appended every `ARCHIVE_TREES` entry — the flat, site-wide `summaries` and `stats` included — plus all four `ROOT_FILES` to EACH channel's list. Measured on jeralyzer: summaries ~14 MB, stats ~21.6 MB (its `page-0000` alone is 20.97 MB), `/duplicates.json` 5.9 MB. **~41.5 MB of byte-identical data per channel pinned**, genuinely re-fetched because `cacheUrls` bulk-caches with `cache: "reload"`. The download was the smaller half. **None of it was removable.** Both evict paths sweep `///` prefixes, and not one of those URLs lives under a channel prefix — so "remove offline copy" left every byte in `PAGES` forever, with no path in the app that could ever reclaim it. Enumerating the contract was the right instinct and it was applied at the wrong granularity: the contract has two kinds of tree, and the cost model follows that split exactly. So the lists split, in `common/lib/archive/offlineUrls.ts`: `channelArchiveUrls` (the per-channel trees of one channel) and `siteArchiveUrls` (the flat trees plus the root files of one origin). Site data rides with the FIRST channel pinned on an origin and is skipped after; a new `EVICT_SITE` message in both workers removes it when that origin's last pinned channel goes. Downloaded once, evicted once. **The per-channel download is byte-for-byte what it was.** "Already present?" is answered from the pinned registry rather than a new status round-trip: the two lists are written and removed together, so one pinned channel means one copy of the site data. That inherits the registry's existing drift — a viewer who clears site data while keeping `localStorage` — which is the same gap `channelCachedPages()` already exists for, and which a re-pin repairs. A per-origin `SITE_STATUS` message would make it truthful; it did not seem worth a second SW round-trip for a case that self-heals. The builders live in `common/` and not `export/` for a reason beyond tidiness: **`export/` has no test runner**, and this split is precisely the thing that needs one. Six cases, and the load-bearing ones are structural rather than literal — the channel list contains no flat tree and no root file; every channel URL matches `///`, the only shape either worker's per-channel sweep can remove; the two lists are disjoint and together cover `ARCHIVE_TREES` exactly. **F2 — a browser reader now gets a browser's budget.** `RemoteSource` defaults to 48 MB per page cache and holds two, so the registry was handing every origin 96 MB: reasonable for one long-lived MCP process, not for a tab, and least of all for a hub page holding a reader per member with no shared ceiling and no way for a browser to reach the env knob. `readers.ts` now passes 8 MB explicitly — 16 MB per origin, so five federated members are 80 MB rather than 480 MB. It can be this small at no cost in reads because the viewer memoises every RECORD it has seen (`transcriptCache.resolved` plus IndexedDB), so an evicted page is only re-read for a video nobody has opened yet; and `MIN_CACHED_PAGES` still floors it at two entries where one page exceeds the whole budget. **`mcp/src/source.ts` constructs `RemoteSource` directly and keeps the 48 MB default**, so no bench counter moves. **F4 — `aliasesCache`'s comment now says what the code does.** It claimed "404 → data" while the catch returns `[]` for any `ArchiveHttpError`. The code is right (it is the old `if (!r.ok) return []`, and a server answering 500 should cost the suggestion chip rather than the search); the comment was the part that was wrong. Noted for the merger, no action taken here: `contract.ts`'s `manifestUrl` flat branch now calls `treeManifestUrl`, and S2c edits nearby. ##### Gates after the fixes - `pnpm -r exec tsc --noEmit` — clean in all six packages. - common — **1101 passed / 0 failed** (1095 + 6 `offlineUrls.test.ts`). - mcp — **205 passed / 0 failed**. `pnpm test:scripts` — **71 passed / 1 skipped**. - `pnpm --filter export exec next build` — **compiled successfully**, 11 static pages. - **e2e** re-run behind the queue lock: export `e2e` **172 passed / 0 failed**, `e2e:hub` **5 passed / 0 failed** — unchanged from before the fixes, which is the point: the per-channel download and every URL shape are what they were. The editor pair and `e2e:2origin` were not re-run; nothing in these two commits touches the editor, and 2origin is red at base. ## Phase 2 — shipped 2026-09-12 All five slices are merged on `integrate/2026-09-storage-priority`, in this order, each reviewed before its merge. Tip **`bd3d4ec`**; the branch is unmerged and waits on operator gate A and a fast-forward to `main`. | merge | slice | what it collapsed | |---|---|---| | `7f86aef` | **S1** | the contract and one `ArchiveReader` under `common/lib/archive/` | | `b7a351c` | **S2b** | `cues.mjs` refuses an unreachable channel instead of cutting from HTTP | | `858aabc` | **S2c** | one `shipsPwa`, one `_headers` generator, one hub member type | | `9026007` | **S3** | one search pipeline under `common/lib/search/` | | `bd3d4ec` | **S2a** | the viewer's archive walks become one reader per origin | S1 was the only prerequisite; S2a, S2b, S2c and S3 all branched off `7f86aef` and were merged in the order their reviews cleared. Every merge was a clean union — the four parallel slices touched disjoint files, which is what the hotspot table above was for, and no fix commit was needed on the integration branch. ### Gates on the combined tip `bd3d4ec` Run from the integration worktree (`/home/user/Projects/integrate-2026-09`, worktree #8 — editor 3801, test 3811, export 3810, export-e2e 3820, ollama stub 12235), e2e behind the machine-global queue lock, one worker. | gate | result | |---|---| | `pnpm -r exec tsc --noEmit` | **clean**, exit 0, all 7 workspace packages | | `pnpm --filter yt-dlp-transcript-common test` | **1159 passed / 0 failed** | | `pnpm --filter yt-dlp-transcript-mcp test` | **205 passed / 0 failed** | | `pnpm test:scripts` | **78 passed / 1 skipped** | | `pnpm --filter editor exec next build` | **compiled successfully** | | `pnpm --filter export exec next build` | **compiled successfully**, 9 routes / 11 static pages | | `pnpm --filter export run build:hub` | compiles + type-checks, then the KNOWN `/ask` prerender failure — see below | | compose-site fixture byte-identity | identical except `generatedAt` and the four `_headers` lines S2c added | | compose-hub over the fixture member | `_headers` **byte-identical** to the pre-S2c literal | | export `e2e` | **172 passed / 0 failed** (4.9 min) | | export `e2e:hub` | **5 passed / 0 failed** (10.0 s) | | export `e2e:2origin` | **not run** — known-red on the base, shells `build:hub` | | editor FULL suite | **533 passed / 0 failed of 533** (23.1 min) — neither known flake fired | | umtool e2e (`SONG_DIR=~/reports/quartering-uh-song/data`) | **129 passed / 40 failed / 2 skipped** (5.3 min) — the same 40, spec for spec, as S2b | | `curl https://jeralyzer.pages.dev/corpus.json` | **byte-identical** to `plans/tools/jeralyzer-corpus-2026-09-12.json`, 12,380 bytes | The umtool 40 are the environmental set S2b already recorded and are unchanged here, spec for spec: `find` 11, `triage` 9, `faces` 6, `usage` 5, `browse` 3, `undo` 3, `deck` 2, `projects` 1. The cause is unchanged too — `$SONG_DIR/{wav48,asr,media}` do not exist on this machine (only `cand2/` does) and `make-fixture.mjs` symlinks those three only `if (existsSync)`, so every spec needing audio, ASR or the face detector fails on `no media for v1`, `detect v1@60.00 -> 404` or a missing `every note (N)` table. **Diffed against S2b's list: no difference.** What DOES exercise the changed path passes on both — the whole of `build.spec.ts`, `clip-bench.spec.ts` (which shells `resolve-windows.mjs` with `CHANNELS_DIR` at the fixture corpus) and `report-longform.spec.ts`, 25 cases between them. The editor suite needed no isolated reruns: **both known flakes were green on their first run** — `video-page.spec.ts:216` and `backfill.spec.ts:457` — and the suite had zero failures end to end. The live-contract check confirms the SNAPSHOT is still current, not that a rebuild would match: jeralyzer has not been rebuilt since it was taken. The real evidence that the emitted contract did not move is the compose-site fixture diff, which runs the same `buildSiteCorpus`. ### The delta table — where the numbers came from `common` is additive across the slices, which is how we know nothing was dropped in the merges: **1051** at `c7f7b90` (the pre-Phase-2 integration tip) `+ 26` S1 (9 `contract.test.ts`, 17 `reader.test.ts`) `+ 0` S2b (its 7 cases land in `test:scripts`, not `common`) `+ 6` S2c (`headers.test.ts`) `+ 52` S3 (3 policy/window width, 8 `window`, 2 `rank`, 6 `collapse`, 21 `evalTree`, 12 `leafPipeline`) `+ 24` S2a (3 `contract.test.ts`, 6 `readers.test.ts`, 9 `archiveCaches.test.ts`, 6 `offlineUrls.test.ts`) = **1159**, measured. `test:scripts` is 71 → **78**, all seven from S2b's `cues.test.mjs`. `mcp` is **205** throughout, unchanged by every slice — the point of the exercise. e2e counts did not move either: export **172** and hub **5** are S1's baseline exactly, and every slice that ran them got the same pair. The editor suite is the integration's **533**. ### What was deleted - **mcp's private reader.** `mcp/src/source.ts` **1,386 lines → 47**, a re-export. The implementation is `common/lib/archive/{reader,reader-fs,reader-hub}.ts`; the interface is the same 18 members under a new name, and the server's seven `./source` importers and `mcp/bench` never noticed. - **Every hand walk of the shard scheme but the worker's.** The eight component caches (`{transcript,subs,posts,digest,summaries,stats,duplicates,aliases}Cache.ts`), plus `SearchDataContext.tsx`, `siteRegistry.ts`, `DuplicatesClient.tsx`'s bare `fetch("/duplicates.json")` and `export/app/lib/offlineCache.ts`. A repo-wide sweep for hand-written archive paths now returns only `common/components/searchIndex.worker.ts` — the deliberate copy, which a worker cannot avoid, and which `contract.test.ts` pins. - **One of two search pipelines.** `mcp/src/search.ts` **1,563 → 1,205 lines**, orchestration only; `common/components/searchPipeline.ts` is **68 lines** answering one question, WHICH FETCH. The shared half is `common/lib/search/`'s six modules. Four `lib → components` back-edges went with it and `"components"` joined `FORBIDDEN.lib` — **the forbidden list grew while the ALLOWED ledger did not**, still 11 entries, byte-identical to the base. - **Two hand-maintained `_headers` literals** (`compose-site.ts`, `compose-hub.ts`), which had already drifted apart, and the second `shipsPwa` and the fourth hub-entry spelling. The one wire change in the whole phase is S2c's: `/digests/*` and `/duplicates.json` gain CORS on a SITE deploy. The hub block does not move. Confirmed on the combined tip by re-composing the fixture — four added lines in `public/_headers`, every other byte identical modulo the build clock, and the composed hub's `_headers` identical to the pre-S2c literal. ### `e2e:2origin` is still red at base, and Phase 2 is not in its path `playwright.2origin.config.ts:16` → `e2e-2origin/globalSetup.ts:142` shells `pnpm run build:hub`, and the hub build dies prerendering `/ask` with `Error: useSearchSession must be used within a SearchSessionProvider` (`common/components/SearchSessionContext.tsx` + `export/app/ask/AskChat.tsx`). Verified at `c7f7b90` by S1 and reproduced at `bd3d4ec` here: same page, same message, same two chunks. It predates Phase 2. What the run does prove each time is the half that matters — hub-mode `next build` **compiles and type-checks**, so `lib/archive/*` and `lib/search/*` resolve in hub mode and `reader-fs.ts` is in no client chunk. ### Deferred, deliberately, and each with its reason 1. **S0-pause** — the legacy pause-field deletion (`transcriptionsPaused`, `downloadsPaused`, `digest.digestsPaused`, `backfill.enabled`, `legacyGateHeld`). The branch **`one-core/s0-pause` is ready and reviewed**, and it is held to the NEXT release, not this one. The reason is a sequencing fact, not a quality one: **a main-era `settings.json` has never written `autoQueue..held`** — that key first appears when merged code boots and writes settings back — so `laneMigration.ts`'s read-time migration is what supplies it. The same release cannot both introduce the writer and delete the migration that covers every file written before it. It lands once this release has booted and the live `settings.json` carries all four `held` keys. 2. **`cues.mjs`'s `tsx` adoption** — Phase 5 (projects join the core), per the decision taken with the operator on 2026-09-12. The full change list is in S2b's note above, re-grepped at `745677f` with three of the brief's paths corrected: six shebangs under `umtool/report-to-video/`, six spawn argv in `umtool/lib/report/driver.mjs`, the printed hint in `umtool/bin/umtool.mjs:516`, `umtool/e2e/clip-bench.spec.ts:145`, the root `test:scripts` script, and `umtool/report-to-video/package.json`'s missing `dependencies`.