Archilyzer · Source

archilyzer

Archilyzer
git clone https://archilyzer.pages.dev/source/archilyzer.git
Log | Files | Refs | README | LICENSE

commit 53f44d1945185c80a0c66e0a3b2f3027bc6e6817
parent 8a49d9059dc15d98b21d33903bc16644f4f01da8
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date:   Sat, 12 Sep 2026 13:40:19 -0400

plans: one-core Phase 2 shipped, and the release-candidate gates on the merged tip

All five Phase 2 slices are merged on integrate/2026-09-storage-priority, in
the order their reviews cleared: ea2d3d0 S1, 1fce539 S2b, d6aa384 S2c,
d0e83fe S3, 8a49d90 S2a. S1 was the only prerequisite; the other four branched
off its merged tip, touched disjoint files, and merged as clean unions with no
fix commit needed on the integration branch.

The gates, all measured on 8a49d90 from the integration worktree (#8), e2e
behind the machine-global queue lock, one worker:

  tsc --noEmit          clean, all 7 workspace packages
  common                1159 passed / 0 failed
  mcp                   205 passed / 0 failed
  test:scripts          78 passed / 1 skipped
  editor next build     compiled successfully
  export next build     compiled successfully, 9 routes / 11 static pages
  export build:hub      compiles + type-checks, then the known /ask prerender
  compose-site fixture  identical but generatedAt and S2c's four _headers lines
  compose-hub _headers  byte-identical to the pre-S2c literal
  export e2e            172 passed / 0 failed
  export e2e:hub        5 passed / 0 failed
  editor FULL suite     533 passed / 0 failed, 23.1 min
  umtool e2e            129 passed / 40 failed / 2 skipped
  jeralyzer corpus.json byte-identical, 12,380 bytes

common is additive across the slices, which is how we know nothing was dropped
in the merges: 1051 + 26 S1 + 0 S2b + 6 S2c + 52 S3 + 24 S2a = 1159, measured.
The editor suite needed no isolated reruns — both known flakes were green on
their first run. umtool's 40 failures are the environmental set S2b recorded,
unchanged spec for spec (find 11, triage 9, faces 6, usage 5, browse 3, undo 3,
deck 2, projects 1): the heavy song fixtures are not on this machine.
e2e:2origin was not run; it shells build:hub, whose /ask prerender is red on
the base and predates Phase 2.

Deleted by the phase: mcp's private reader (1,386 lines to 47), every hand walk
of the shard scheme but searchIndex.worker.ts's deliberate copy, and one of two
search pipelines (mcp/src/search.ts 1,563 to 1,205; components/searchPipeline.ts
to 68). "components" joined FORBIDDEN.lib while the ALLOWED ledger stayed at
eleven entries, byte-identical to the base.

Two items are deferred on purpose. S0-pause (branch one-core/s0-pause, ready and
reviewed) is held to the NEXT release: a main-era settings.json has never written
autoQueue.<lane>.held, so the same release cannot both introduce the writer and
delete the read-time migration that covers every file written before it. And
cues.mjs's tsx adoption is Phase 5, with the change list in S2b's note.

STATE.md's head is rewritten to say both shipments sit on one branch pending
gate A and the fast-forward; the operator runbook and the merge-review notes are
unchanged. FACTS.md gains the Phase 2 seams, and one-core.md marks the phase
shipped.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Diffstat:
Mplans/FACTS.md | 214+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Mplans/STATE.md | 74+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------
Mplans/one-core-phase-2.md | 130+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Mplans/one-core.md | 11+++++++++++
4 files changed, 414 insertions(+), 15 deletions(-)

diff --git a/plans/FACTS.md b/plans/FACTS.md @@ -3786,3 +3786,217 @@ Actions; priority replaced the Sync column with Tier, so the fixed count moved b one column the selection added. `editor/app/channels/page.tsx` feeds both halves: a row carries `media:` AND `priority:`, and the table takes `defaultMediaRoot` AND `sites`/`focusLabel`. + +## one-core Phase 2 (verified 2026-09-12, branch `integrate/2026-09-storage-priority`) + +Five slices merged onto the integration tip in order — `7f86aef` S1, `b7a351c` S2b, +`858aabc` S2c, `9026007` S3, `bd3d4ec` S2a. The seams below are what Phase 3 lands on; +every path was re-read at `bd3d4ec`. + +### `common/lib/archive/` — eight modules, and what each owns + +| file | owns | +|---|---| +| `contract.ts` | `CONTRACT`, `pageFileName`, `ARCHIVE_TREES`, `PER_CHANNEL_TREES`, `ROOT_FILES`, the URL builders (`corpusUrl`, `manifestUrl`, `treeManifestUrl`, `pageUrl`, `rootFileUrl`), `shipsPwa`, and the hub entry types (`HubMemberInput`, `HubCorpusSite`, `HubSite`) | +| `io-stats.ts` | `recordRead` / `ioStatsSnapshot` / `ioStatsEnabled`, guarded `typeof process !== "undefined" && process.env?.MCP_IO_STATS === "1"` | +| `reader.ts` | the `ArchiveReader` interface, `RemoteSource`, `PageCache`, `ArchiveHttpError`, `buildDuplicateIndex`. **Zero `node:*` imports** | +| `reader-fs.ts` | `LocalSource` — the only `node:fs` / `node:path` file; `reader.ts` never value-imports it | +| `reader-hub.ts` | `HubSource` — federation across member origins | +| `readers.ts` | the browser's reader REGISTRY: one `RemoteSource` per origin (`readerFor`), `channelRef`, `resetReaders` | +| `headers.ts` | the `_headers` renderer plus `SITE_CORS_PATHS` / `HUB_CORS_PATHS` / `contractCorsPaths()` | +| `offlineUrls.ts` | `channelArchiveUrls` (one channel's per-channel trees) and `siteArchiveUrls` (the flat trees + root files of one origin) | + +### `contract.ts` OWNS `CONTRACT` and `pageFileName`; `corpus.ts` / `manifest.ts` re-export + +Not a style choice — the other arrangement is a hard ESM failure. `corpus.ts` imports the +URL builders, `contract.ts` needs `CONTRACT.pagePad` for `pageFileName`, and `manifest.ts` +reads `CONTRACT.manifest` **at module scope**. Leaving `CONTRACT` in `corpus.ts` closes the +loop `corpus → archive/contract → manifest → corpus` and the first module imported throws +`ReferenceError: Cannot access 'CONTRACT' before initialization`. So `contract.ts` is the +BOTTOM of the stack (it imports only `lib/duplicates.ts`), and every existing import site +(`from "./corpus"`, `from "./manifest"`, the three `*PageFileName` aliases) is byte-identical. + +`stats/` gets URL builders without joining `CONTRACT.layers`: `corpus.json`'s `shardScheme` +does not document it, so the builders take a wider `ArchiveTree = ContractLayer | "stats"` +and the published layer list stays frozen. + +### `ArchiveReader` is mcp's 18-member `ShardSource`, verbatim, and has no `record()` + +`label`, `listChannels`, `transcriptsManifest`, `transcriptPage`, `loadAliases`, +`loadGroups`, `publicOrigin`, `subsManifest`, `subsPage`, `postsManifest`, `postsPage`, +`availabilityMap`, `videoIndex?`, `pageConcurrency?`, `duplicateIndex?`, `statsIndex?`, +`digestsManifest?`, `digestPage?`. **`record(layer, slug, id)` must not be added**: the walk +is manifest → shard → record and callers already hold the shard, so a per-record fetch is a +bench regression by construction. + +`mcp/src/source.ts` is 47 lines of re-export (it was 1,386), so this server's seven +`./source` importers and `mcp/bench` are untouched. + +### The raw throwing reads are on `RemoteSource` only + +Twelve one-to-three-line URL + fetch + parse reads that **throw** — +`readCorpus`, `readAliasConfig`, `readDuplicates`, `readSummariesManifest`, +`readSummariesPage`, `readStatsManifest`, `readStatsPage`, `readSubsSiteManifest`, +`readPostsSiteManifest`, `readChannelSubsManifest`, `readChannelPostsManifest`, +`readChannelDigestsManifest` — with every tolerance policy sitting on top. The reason is +react-query: it retries a REJECTED query and will never retry a resolved `null`, so folding +the browser's policy and the MCP's `null`-for-anything-not-200 into one method is how a +transient blip becomes a panel that stays empty for the session. `ArchiveHttpError` came with +them for the one caller that must tell a 404 from a dropped connection (the alias dictionary, +`staleTime: Infinity`); its `message` is byte-identical to the `Error` it replaced. + +The tolerant methods are rebased on the raw read they duplicated, so the MCP's behaviour is +unchanged by construction and no read count moves. The `io-stats` blind spots are preserved +deliberately: the raw reads take an OPTIONAL `kind`, and the ones passing none are exactly the +reads that recorded nothing before. + +### The browser's page-cache budget is 8 MB per cache, set in `readers.ts` + +`RemoteSource` defaults to `DEFAULT_PAGE_CACHE_MB = 48` (`reader.ts`, env +`TRANSCRIPT_MCP_PAGE_CACHE_MB`) and holds TWO page caches, so an origin costs 96 MB — right +for one long-lived MCP process, wrong for a tab and emphatically wrong for a hub page holding +a reader per member. `readers.ts` passes `BROWSER_PAGE_CACHE_BYTES = 8 MB` explicitly, so an +origin is 16 MB and five federated members are 80 MB rather than 480 MB. It can be this small +at no cost in reads because the viewer memoises every RECORD it has seen +(`transcriptCache.resolved` + IndexedDB), and `MIN_CACHED_PAGES = 2` still floors it where one +page exceeds the whole budget. **`mcp/src/source.ts` constructs `RemoteSource` directly and +keeps the 48 MB default**, which is why no bench counter moves. + +Two memos stay in `components/` on purpose: the subs/posts CHANNEL-manifest memos (they keep +the throwing policy above), and the posts and digest PAGE memos (the reader caches subs and +transcript pages but not those two, and `fetchThread` walks every page of a channel). + +### `common/lib/search/` — six modules, and lib holds no caller's policy default + +`policy.ts` (`MCP_POLICY` and `VIEWER_POLICY`), `evalTree.ts` (the per-record boolean +evaluator), `window.ts` (both excerpt shapes — the MCP's merged ±45 s paragraph and the +viewer's per-cue row — one `truncate`, one `clock`), `rank.ts` (the two comparators that +exist: newest-first by `uploadDate`, oldest-first by `createdAt` tie-broken by id), +`collapse.ts` (pure, synchronous, generic over `CollapsibleHit`), `leafPipeline.ts` (the +browser's streaming leaf scanner, three worker-pool drivers + the leaf adapter). + +**The rule: `lib/search/` holds no caller's budget as a default.** All three widths +(`truncate`'s `max`, `windowedTranscript`'s `maxLines`, `RecordCtx.snippetChars`) are +REQUIRED, and no non-test module under `lib/search/` imports `MCP_POLICY` at all except +`policy.ts`, which defines it (`maxPages` 400, `hardVideoCap` 2000, `windowLineCap` 200, +`snippetChars` 240; `VIEWER_POLICY` spells the uncapped one). + +A default would have failed the bad way: an adopter that forgot its policy would +silently clip every excerpt at 240 with the whole suite green. + +`lib/searchEval.ts` and `lib/search/evalTree.ts` are SIBLINGS, not a copy and its original: +one answers *which slugs survive* (streaming, over per-leaf network pipelines) and one answers +*does this record match*. Each names the other in a comment and states the rule — the meaning +of AND / OR / negate changes in both together. No test can compare a streaming slug-set walk +against a per-record boolean, so the prose is the only guard. + +`components/searchPipeline.ts` is 68 lines: it answers one question — WHICH FETCH — and +supplies `searchRuntime`. `mcp/src/search.ts` is 1,205 lines of orchestration (it was 1,563); +`buildScanPlan` stayed there and calls the single `passesFilters` from `lib/search/evalTree`, +so the pruner and the scanner cannot disagree about what matches. + +### `FORBIDDEN.lib` now includes `components`, and the allow-list is still 11 + +`common/architecture.test.ts`: `lib: ["controller", "jobs", "components"]`. Four back-edges +had to go first — `searchEval → components/searchPipeline`, `searchEval → +components/searchLayerCache`, `aiHandoff → components/searchPipeline` (a type), +`searchQuery → components/urlState` (a type). All four inverted: the pipeline moved down, the +fetch and the memo are injected as `SearchRuntime.runLeaf` / `SearchRuntime.cache`, and +`CachedResult` / `SearchMode` moved to the modules that interpret them (the old homes +re-export). **The forbidden list grew while the ALLOWED ledger did not** — still the same 11 +entries, byte-identical to the base. + +### The two service workers' path lists are pinned to the contract by `contract.test.ts` + +`export/service-worker/site-sw.js` and `export/service-worker/sw-hub.js` each hand-write +three regexes — a service worker can import nothing — and `common/lib/archive/contract.test.ts` +reads both files and asserts the alternations equal the contract: `SHARD_RE` ∪ `FLAT_RE` = +`ARCHIVE_TREES`, `SHARD_RE` = `PER_CHANNEL_TREES`, `ROOT_RE` = `ROOT_FILES`. Verified by +mutation (dropping `digests` from sw-hub's `SHARD_RE` turns it red). The same test pins +`common/components/searchIndex.worker.ts`'s deliberate `pageFileName` copy — a worker cannot +import the reader, so that one walk stays, guarded. + +`SHARD_RE` matches three path segments, so `/duplicates.json` and `/summaries/*` never matched +it: both workers gained `FLAT_RE` and `ROOT_RE`, answered NETWORK-FIRST (those documents carry +no per-channel `generatedAt` to evict against). Three drifts were closed with them: `summaries` +was dead in site-sw's `SHARD_RE` and in both eviction prefix lists; `/posts/` and `/digests/` +were absent from both eviction lists, so a rebuilt channel served stale posts and digests +forever; and sw-hub had no `digests` at all. + +**An offline copy is TWO lists.** `offlineUrls.ts` splits them because the cost model does: +appending the flat trees and root files to EACH channel pinned ~41.5 MB of byte-identical data +per channel on jeralyzer (summaries ~14 MB, stats ~21.6 MB, `/duplicates.json` 5.9 MB), and +NONE of it was removable — both evict paths sweep `/<tree>/<slug>/` prefixes and not one of +those URLs lives under a channel prefix. Site data now rides with the FIRST channel pinned on +an origin, and an `EVICT_SITE` message in both workers removes it when the last pinned channel +of that origin goes. + +`offlineCache.ts` keeps its own `cache: "no-store"` fetch rather than taking the shared reader: +the reader is a plain `fetch` and would be answered from the very service-worker cache that +function exists to refill. + +### `_headers` is generated by `headers.ts`; the site gets two CORS lines the hub does not + +One renderer, two ordered path lists. `SITE_CORS_PATHS` gained `/duplicates.json` and +`/digests/*` — the wire change of Phase 2, and the only one. `HUB_CORS_PATHS` is UNCHANGED: +a hub holds no shard data (it reads every member cross-origin at runtime), so declaring +headers for paths it never serves would be noise. `contractCorsPaths()` plus +`headers.test.ts` assert every `CONTRACT.layers` tree and every `ROOT_FILES` document has a +line, so adding a layer and forgetting its CORS entry fails a test rather than a deploy. + +The lists are ORDERED data, not derived: `_headers` is matched top-down by Cloudflare and the +file is diffed against a committed fixture, so deriving the order from `ROOT_FILES` + +`CONTRACT.layers` would have reordered every line. + +**`curl -I` cannot show this gap locally and never could.** `serve` (what the export, hub and +2-origin suites all run behind) gives `**/*.json` a blanket `Access-Control-Allow-Origin: *` +via `export/serve.json`, and `wrangler pages dev` — the only local server that reads `_headers` +at all — adds the header to every response of its own accord. What wrangler does report is its +own parse: **12 valid header rules before, 14 after**. The real before/after is a deploy. + +### `shipsPwa` is server-called only + +`common/lib/archive/contract.ts:183`. It reads `process.env.INSTANCE_MODE` BARE, with no +`typeof process` guard, unlike `io-stats.ts` and the page-cache knob. That is safe only +because of who calls it: `common/bin/compose-site.ts` (a build script) and +`export/app/lib/mode.ts`, whose only caller is `export/app/layout.tsx` — a server component — +and whose `currentSite()` reads the sites dir anyway. `contract.ts` is not reachable from any +browser-but-not-Next context (the service workers import nothing; the search worker keeps its +own copy). **A future CLIENT caller owes the guard.** Do not reason from the earlier claim +that Next inlines `INSTANCE_MODE` into the client bundle — it does not (`INSTANCE_MODE` is +neither `NEXT_PUBLIC_` nor in a `next.config.ts` `env:` block), and that claim has been +corrected twice. + +### `cues.mjs` carries a reachability twin of `assertChannelMediaReachable` + +`umtool/report-to-video/cues.mjs`'s `checkChannelReachable` replicates +`inspectChannelMedia` / `assertChannelMediaReachable` (`common/lib/channelMedia.ts:320-327`, +which carries the cross-reference comment) in plain `.mjs`, because umtool's bins run under +bare node with no `tsx`. It reads `dataDir` from `config.json`, checks the `.relocating.json` +marker, `lstat`s `data/`, compares `readlink` against the configured target, and `stat`s the +deep `<root>/<slug>/data` path — then **throws** a `CueLookupError`, which carries no `code`, +so `load`'s `ENOENT`/`ENOTDIR` fallback rethrows it instead of reaching for the archive. Zero +fetches, asserted. + +Where the line is drawn, deliberately: **a channel whose `data/` is simply absent with no +configured target does NOT throw.** That is `in-place` to the editor's twin, and it is the +archive-only workflow this repo advertises (a clone with no corpus at all). What throws is the +case that motivated the slice — a relocated channel whose drive is gone. + +Not closed, and wanting the same guard when umtool's local-corpus access is next touched: +`umtool/report-to-video/check-availability.mjs:53` (`cueMeta`) and +`umtool/lib/projects/report.mjs:121` (`cuePathFor`) still join +`channels/<slug>/data/<id>/transcript.cues.json` by hand and read an unmounted channel as an +absent file. Neither answers a cue WINDOW, so neither can cut a clip in the wrong place. + +### `e2e:2origin` is known-red on the base and Phase 2 did not touch it + +`playwright.2origin.config.ts:16` → `e2e-2origin/globalSetup.ts:142` shells +`pnpm run build:hub`, and the hub build dies prerendering `/ask`: +`Error: useSearchSession must be used within a SearchSessionProvider`, from +`common/components/SearchSessionContext.tsx` + `export/app/ask/AskChat.tsx`. **Verified on +`c7f7b90` itself** (the pre-Phase-2 integration tip) and again at `bd3d4ec`: same page, same +message, same two chunks. It predates Phase 2 and is the same failure already recorded with +the export responsive redesign. What the hub build DOES prove each time it is run: it compiles +and type-checks, so the bundle resolves `lib/archive/*` and `lib/search/*` in hub mode too and +never pulls `reader-fs.ts` into a client chunk. diff --git a/plans/STATE.md b/plans/STATE.md @@ -3,15 +3,39 @@ The working memory for the local-AI derived-corpus work. Rewritten at the end of every session, before context is cleared. See [`README.md`](README.md) for the protocol. -**Last updated:** 2026-09-12 — **both interlude shipments are merged** on branch -`integrate/2026-09-storage-priority`: `relocate-channel-media` (from `storage/relocate-media`) -and `channel-priority` (from `channel-priority/s5`), in that order, off `e74f005`. The branch -is unmerged and waits on a fast-forward to `main`; the two entries, the gate numbers actually -observed on the merged tip (editor **533/533**, `common` **1051/1051**), and the one operator -runbook they now share are below, under the Phase 1 record they build on. The two resolutions -that are not a union of both sides — `listChannelMeta`'s return shape and the pair of bulk -bars at the foot of `ChannelsTable` — are pinned in -[`FACTS.md`](FACTS.md#the-storage--priority-integration-verified-2026-09-12--where-the-two-branches-collide). +**Last updated:** 2026-09-12 — **the interlude shipments and one-core Phase 2 are all merged +on one branch**, `integrate/2026-09-storage-priority`, tip **`bd3d4ec`**, unmerged. Off +`e74f005`: `relocate-channel-media` (from `storage/relocate-media`), then `channel-priority` +(from `channel-priority/s5`), then Phase 2's five slices in the order their reviews cleared — +`7f86aef` S1, `b7a351c` S2b, `858aabc` S2c, `9026007` S3, `bd3d4ec` S2a. The branch waits on +**operator gate A**, then a fast-forward to `main`. The two resolutions that are not a union +of both sides — `listChannelMeta`'s return shape and the pair of bulk bars at the foot of +`ChannelsTable` — are pinned in +[`FACTS.md`](FACTS.md#the-storage--priority-integration-verified-2026-09-12--where-the-two-branches-collide); +Phase 2's seams are in +[`FACTS.md`](FACTS.md#one-core-phase-2-verified-2026-09-12-branch-integrate2026-09-storage-priority) +and the slice-level record is +[`one-core-phase-2.md`](one-core-phase-2.md#phase-2--shipped-2026-09-12). + +**The release-candidate gates on `bd3d4ec`, measured 2026-09-12 from the integration worktree +(`/home/user/Projects/integrate-2026-09`, worktree #8 — editor 3801, test 3811, export 3810, +export-e2e 3820), e2e behind the machine-global queue lock, one worker:** `tsc --noEmit` +clean in all 7 workspace packages; `common` **1159/1159**; `mcp` **205/205**; +`pnpm test:scripts` **78 passed / 1 skipped**; editor and export `next build` clean; the +compose-site fixture byte-identical except `generatedAt` and the four `_headers` lines S2c's +CORS change added, with the composed hub's `_headers` byte-identical to the pre-S2c literal; +export e2e **172/172**, hub **5/5**, the editor suite **533/533** in 23.1 min with neither +known flake firing, and umtool **129 passed / 40 failed / 2 skipped** — the same 40 +environmental failures S2b measured, spec for spec; and +`curl https://jeralyzer.pages.dev/corpus.json` byte-identical to +`plans/tools/jeralyzer-corpus-2026-09-12.json` (12,380 bytes) — which says the snapshot is +still current, not that a rebuild would match, because jeralyzer has not been rebuilt. +`e2e:2origin` was not run: it shells `build:hub`, whose `/ask` prerender is red on the base +and predates Phase 2. + +**The operator runbook below is unchanged and still the sequence to follow** — the +channel-priority migration first, with the editor stopped, then the platter mount, then the +saved-video store, then the channels. Nothing in Phase 2 touches it. **Previously:** 2026-09-08 — **one-core Phase 1 shipped** on branch `one-core/phase-1` (`7f294df` → `81a663f` plus a docs commit, 36 commits, not merged): **dispatch is one scheduler, and the lane is the @@ -296,12 +320,32 @@ not unreachable ones (`channels/actions.ts:536-543`); an unmounted channel is re `syncAction` by the `needsMedia` guard and surfaces as a failed sync, not a skip reason. Pre-existing on the relocate branch. -**Next:** one-core Phase 2 (the contract: one `ArchiveReader`), five slices, planned in -[`one-core-phase-2.md`](one-core-phase-2.md) — S1 is the only prerequisite, then S2a / S2b / -S2c / S3 and the legacy pause-field deletion in parallel off its merged tip. Read -`common/architecture.test.ts`'s allow-list first — it is the shortest accurate statement of -what is still tangled, it shrank by one across phase 1, and no slice added an entry, relocate -included. +**Next:** **operator gate A** — the digest lane's runner drives a production pass and comes +back after a restart, run from `one-core/phase-1-gate-a` (`0438a72`) per the runbook in +[`one-core-phase-1.md`](one-core-phase-1.md)'s "Operator gate A" section. It gates the merge, +not the authorship, which is why that branch exists. Then **fast-forward +`integrate/2026-09-storage-priority` to `main`** and run the **O2–O9 rollout**: the +channel-priority migration with the editor stopped, the platter mount, the saved-video store +rsync, and the channel moves through the UI — the four numbered steps above, in that order, +with the migration first because it is the only one that must happen before this code ever +boots. + +**Then S0-pause lands, in the release AFTER this one.** The branch `one-core/s0-pause` +(`2aeb358`) is ready and reviewed — it deletes `transcriptionsPaused`, `downloadsPaused`, +`digest.digestsPaused`, `backfill.enabled` and `legacyGateHeld`. It is held back for a +sequencing reason, not a quality one: a main-era `settings.json` has never carried +`autoQueue.<lane>.held`, that key first appears when merged code writes settings back, and the +same release cannot both introduce the writer and delete the read-time migration that covers +every file written before it. An install taking it straight from an older release loses its +pauses — transcription, downloads and digests come up running and the backfill lane comes up +held. Boot this release once, confirm all four `held` keys are in the live `settings.json`, +then take it. + +**Then one-core Phase 3** — views in the core, editor as shell, four slices +([`one-core.md`](one-core.md) §Phase 3). Read `common/architecture.test.ts`'s allow-list +first: it is the shortest accurate statement of what is still tangled. Phase 2 left it at +**eleven entries, byte-identical to the base**, while ADDING `"components"` to +`FORBIDDEN.lib` — the forbidden list grew and the debt ledger did not. **Previously:** 2026-09-07 — **one-core Phase 0 shipped** on branch `one-core/phase-0` (`df5eb48` → `1691c4f`, six commits, not merged): **guardrails and dead weight**, every item a diff --git a/plans/one-core-phase-2.md b/plans/one-core-phase-2.md @@ -1133,3 +1133,133 @@ branch now calls `treeManifestUrl`, and S2c edits nearby. the point: the per-channel download and every URL shape are what they were. The editor pair and `e2e:2origin` were not re-run; nothing in these two commits touches the editor, and 2origin is red at base. + +## Phase 2 — shipped 2026-09-12 + +All five slices are merged on `integrate/2026-09-storage-priority`, in this order, each +reviewed before its merge. Tip **`bd3d4ec`**; the branch is unmerged and waits on operator +gate A and a fast-forward to `main`. + +| merge | slice | what it collapsed | +|---|---|---| +| `7f86aef` | **S1** | the contract and one `ArchiveReader` under `common/lib/archive/` | +| `b7a351c` | **S2b** | `cues.mjs` refuses an unreachable channel instead of cutting from HTTP | +| `858aabc` | **S2c** | one `shipsPwa`, one `_headers` generator, one hub member type | +| `9026007` | **S3** | one search pipeline under `common/lib/search/` | +| `bd3d4ec` | **S2a** | the viewer's archive walks become one reader per origin | + +S1 was the only prerequisite; S2a, S2b, S2c and S3 all branched off `7f86aef` and were +merged in the order their reviews cleared. Every merge was a clean union — the four parallel +slices touched disjoint files, which is what the hotspot table above was for, and no fix +commit was needed on the integration branch. + +### Gates on the combined tip `bd3d4ec` + +Run from the integration worktree (`/home/user/Projects/integrate-2026-09`, worktree #8 — +editor 3801, test 3811, export 3810, export-e2e 3820, ollama stub 12235), e2e behind the +machine-global queue lock, one worker. + +| gate | result | +|---|---| +| `pnpm -r exec tsc --noEmit` | **clean**, exit 0, all 7 workspace packages | +| `pnpm --filter yt-dlp-transcript-common test` | **1159 passed / 0 failed** | +| `pnpm --filter yt-dlp-transcript-mcp test` | **205 passed / 0 failed** | +| `pnpm test:scripts` | **78 passed / 1 skipped** | +| `pnpm --filter editor exec next build` | **compiled successfully** | +| `pnpm --filter export exec next build` | **compiled successfully**, 9 routes / 11 static pages | +| `pnpm --filter export run build:hub` | compiles + type-checks, then the KNOWN `/ask` prerender failure — see below | +| compose-site fixture byte-identity | identical except `generatedAt` and the four `_headers` lines S2c added | +| compose-hub over the fixture member | `_headers` **byte-identical** to the pre-S2c literal | +| export `e2e` | **172 passed / 0 failed** (4.9 min) | +| export `e2e:hub` | **5 passed / 0 failed** (10.0 s) | +| export `e2e:2origin` | **not run** — known-red on the base, shells `build:hub` | +| editor FULL suite | **533 passed / 0 failed of 533** (23.1 min) — neither known flake fired | +| umtool e2e (`SONG_DIR=~/reports/quartering-uh-song/data`) | **129 passed / 40 failed / 2 skipped** (5.3 min) — the same 40, spec for spec, as S2b | +| `curl https://jeralyzer.pages.dev/corpus.json` | **byte-identical** to `plans/tools/jeralyzer-corpus-2026-09-12.json`, 12,380 bytes | + +The umtool 40 are the environmental set S2b already recorded and are unchanged here, spec for +spec: `find` 11, `triage` 9, `faces` 6, `usage` 5, `browse` 3, `undo` 3, `deck` 2, +`projects` 1. The cause is unchanged too — `$SONG_DIR/{wav48,asr,media}` do not exist on this +machine (only `cand2/` does) and `make-fixture.mjs` symlinks those three only `if (existsSync)`, +so every spec needing audio, ASR or the face detector fails on `no media for v1`, +`detect v1@60.00 -> 404` or a missing `every note (N)` table. **Diffed against S2b's list: no +difference.** What DOES exercise the changed path passes on both — the whole of +`build.spec.ts`, `clip-bench.spec.ts` (which shells `resolve-windows.mjs` with `CHANNELS_DIR` +at the fixture corpus) and `report-longform.spec.ts`, 25 cases between them. + +The editor suite needed no isolated reruns: **both known flakes were green on their first +run** — `video-page.spec.ts:216` and `backfill.spec.ts:457` — and the suite had zero failures +end to end. + +The live-contract check confirms the SNAPSHOT is still current, not that a rebuild would +match: jeralyzer has not been rebuilt since it was taken. The real evidence that the emitted +contract did not move is the compose-site fixture diff, which runs the same `buildSiteCorpus`. + +### The delta table — where the numbers came from + +`common` is additive across the slices, which is how we know nothing was dropped in the +merges: **1051** at `c7f7b90` (the pre-Phase-2 integration tip) +`+ 26` S1 (9 `contract.test.ts`, 17 `reader.test.ts`) +`+ 0` S2b (its 7 cases land in `test:scripts`, not `common`) +`+ 6` S2c (`headers.test.ts`) +`+ 52` S3 (3 policy/window width, 8 `window`, 2 `rank`, 6 `collapse`, 21 `evalTree`, 12 `leafPipeline`) +`+ 24` S2a (3 `contract.test.ts`, 6 `readers.test.ts`, 9 `archiveCaches.test.ts`, 6 `offlineUrls.test.ts`) += **1159**, measured. `test:scripts` is 71 → **78**, all seven from S2b's `cues.test.mjs`. +`mcp` is **205** throughout, unchanged by every slice — the point of the exercise. + +e2e counts did not move either: export **172** and hub **5** are S1's baseline exactly, and +every slice that ran them got the same pair. The editor suite is the integration's **533**. + +### What was deleted + +- **mcp's private reader.** `mcp/src/source.ts` **1,386 lines → 47**, a re-export. The + implementation is `common/lib/archive/{reader,reader-fs,reader-hub}.ts`; the interface is + the same 18 members under a new name, and the server's seven `./source` importers and + `mcp/bench` never noticed. +- **Every hand walk of the shard scheme but the worker's.** The eight component caches + (`{transcript,subs,posts,digest,summaries,stats,duplicates,aliases}Cache.ts`), plus + `SearchDataContext.tsx`, `siteRegistry.ts`, `DuplicatesClient.tsx`'s bare + `fetch("/duplicates.json")` and `export/app/lib/offlineCache.ts`. A repo-wide sweep for + hand-written archive paths now returns only `common/components/searchIndex.worker.ts` — the + deliberate copy, which a worker cannot avoid, and which `contract.test.ts` pins. +- **One of two search pipelines.** `mcp/src/search.ts` **1,563 → 1,205 lines**, orchestration + only; `common/components/searchPipeline.ts` is **68 lines** answering one question, WHICH + FETCH. The shared half is `common/lib/search/`'s six modules. Four `lib → components` + back-edges went with it and `"components"` joined `FORBIDDEN.lib` — **the forbidden list + grew while the ALLOWED ledger did not**, still 11 entries, byte-identical to the base. +- **Two hand-maintained `_headers` literals** (`compose-site.ts`, `compose-hub.ts`), which had + already drifted apart, and the second `shipsPwa` and the fourth hub-entry spelling. + +The one wire change in the whole phase is S2c's: `/digests/*` and `/duplicates.json` gain +CORS on a SITE deploy. The hub block does not move. Confirmed on the combined tip by +re-composing the fixture — four added lines in `public/_headers`, every other byte identical +modulo the build clock, and the composed hub's `_headers` identical to the pre-S2c literal. + +### `e2e:2origin` is still red at base, and Phase 2 is not in its path + +`playwright.2origin.config.ts:16` → `e2e-2origin/globalSetup.ts:142` shells +`pnpm run build:hub`, and the hub build dies prerendering `/ask` with +`Error: useSearchSession must be used within a SearchSessionProvider` +(`common/components/SearchSessionContext.tsx` + `export/app/ask/AskChat.tsx`). Verified at +`c7f7b90` by S1 and reproduced at `bd3d4ec` here: same page, same message, same two chunks. +It predates Phase 2. What the run does prove each time is the half that matters — hub-mode +`next build` **compiles and type-checks**, so `lib/archive/*` and `lib/search/*` resolve in +hub mode and `reader-fs.ts` is in no client chunk. + +### Deferred, deliberately, and each with its reason + +1. **S0-pause** — the legacy pause-field deletion (`transcriptionsPaused`, `downloadsPaused`, + `digest.digestsPaused`, `backfill.enabled`, `legacyGateHeld`). The branch + **`one-core/s0-pause` is ready and reviewed**, and it is held to the NEXT release, not + this one. The reason is a sequencing fact, not a quality one: **a main-era `settings.json` + has never written `autoQueue.<lane>.held`** — that key first appears when merged code + boots and writes settings back — so `laneMigration.ts`'s read-time migration is what + supplies it. The same release cannot both introduce the writer and delete the migration + that covers every file written before it. It lands once this release has booted and the + live `settings.json` carries all four `held` keys. +2. **`cues.mjs`'s `tsx` adoption** — Phase 5 (projects join the core), per the decision taken + with the operator on 2026-09-12. The full change list is in S2b's note above, re-grepped + at `745677f` with three of the brief's paths corrected: six shebangs under + `umtool/report-to-video/`, six spawn argv in `umtool/lib/report/driver.mjs`, the printed + hint in `umtool/bin/umtool.mjs:516`, `umtool/e2e/clip-bench.spec.ts:145`, the root + `test:scripts` script, and `umtool/report-to-video/package.json`'s missing `dependencies`. diff --git a/plans/one-core.md b/plans/one-core.md @@ -260,6 +260,17 @@ so it never edits the lane trees beside Phase 1. ### Phase 2 — The contract: one `ArchiveReader` (5 slices) +**SHIPPED 2026-09-12** on branch `integrate/2026-09-storage-priority`, tip **`bd3d4ec`**, +unmerged — all five slices, merged in the order their reviews cleared (`7f86aef` S1, +`b7a351c` S2b, `858aabc` S2c, `9026007` S3, `bd3d4ec` S2a). The record — every slice's sha +range, every divergence and every gate number on the combined tip — is +[`one-core-phase-2.md`](one-core-phase-2.md#phase-2--shipped-2026-09-12); the anchors are +[`FACTS.md`](FACTS.md#one-core-phase-2-verified-2026-09-12-branch-integrate2026-09-storage-priority). +**One wire change in the whole phase** — `/digests/*` and `/duplicates.json` gain CORS on a +site deploy — and the architecture test's allow-list is still eleven while `"components"` +joined `FORBIDDEN.lib`. S0-pause is deferred to the next release and `cues.mjs`'s `tsx` +adoption to Phase 5; both reasons are in the record. + Slice plan: [`one-core-phase-2.md`](one-core-phase-2.md). Corrected 2026-09-12 after a survey; the original wording is kept in git.