### As it went, part two — the big three (2026-10-02, 11:42–12:53) Asked for by the operator ("do the big three migration"); re-planned once the dry run put the three channels' text at 33.7 GB against 77 GB free on `/home`: the saved-video store (209 GB on the SSD, 202 GB of it nuxanor-kick's kept containers; the platter 1.2 TB free) is moved first, then the three are migrated. Both offline in one editor-stopped window, the store through the same controller the Storage card's Move button enqueues (`relocateSavedVideos`, run under tsx from `common/`). | step | result | |---|---| | dry run `migrate-tier --all --include-large` (editor up) | rekietalaw 8.73 GB text / 1,348 links; the-quartering-rumble 10.16 GB / 175; omnibased 14.84 GB / 318 | | saved-video store move out, 11:45 (editor stopped) | 46 files, 208.46 GB; STOPPED at 12:19 with 6.8 GB copied — the platter wrote at 2 MB/s (20 requests in flight, the drive 100 % busy); the marker stays at phase `copy`, the partial copy stays under `/saved-videos` | | `migrate-tier --all --include-large`, 12:19–12:34 | all three migrated in 860 s; `/home` 77 GB → 43 GB free; no platter writes (the rename) | | editor restart 12:34 (`Dt4zuo5uJcPKZ9fKk0L-f`, unchanged) | 200 in 2 s; boot cancelled 9 stale queued metas, closed 4 ghosts | | live proof | `/`, `/storage`, `/channels` in 0.23–0.31 s; one video page per channel 0.14–0.46 s; `transcript.live_chat.json` (rekietalaw) and `audio.mp3` (the other two) stream through their links (206); Refresh report on all three — snapshots rewritten 12:51–12:53 with `totalTextBytes` | | the platter's write speed, idle | `dd` 1 GiB direct: 226 s, 4.7 MB/s (reads were 55 MB/s during the migration); only a file manager held the volume open | The omnibased snapshot of 12:53 carries `totalMediaBytes: null`: the write test was holding the platter at that moment and the snapshot's media walk gave up as designed; the next Refresh fills it. Nothing on the retired layout remains. The marker clear was refused by the permission layer; the operator cleared it from the Storage page's saved-video card at 12:58 (the store reads "In place", writers free). The store move is deferred until the platter writes at a sane rate; a new Move reuses the partial copy. The ten-channel scripts gained `r17-big3-only.sh` and `tmp-move-store.ts` (run from `common/`). # Release 17 — the media tier: big files move, hot text stays on the SSD Written 2026-10-01 in plan mode against `main` `03be31b5` (releases 13–16 live). Rules: `plans/tools/implementer-rules.md` — one Opus implementer per slice in its own worktree, one read-only Opus review, the parent merges `--no-ff` on a clean tree; an implementer's commits carry its own model's `Co-Authored-By` (T1's were rewritten to the session's at the parent's request before merge; the rule was corrected 2026-10-02 after RL declined to do the same — the tree is identical either way); records state rulings never reasons; no identifier ending `the refused parent-suffix`; counts-only privacy greps before every merge; changelog bullets checked by eye; the homepage build gate is `build:nodata`, never `run build`; the corpus link is for `next build` only. The live editor on `:3001` is restarted ONCE, after T3 merges and BEFORE the migration runs (the scripts live in `~/reports/release-15/scripts/r15-{build,restart,smoke}.sh`; the guard sha there must be updated). ## Context A "moved" channel today moves its whole `data/` directory to another drive (an absolute symlink `channels//data → //data`, `config.dataDir`). That puts the hot text — transcripts, cues, `metadata.info.json`, every sidecar — on the slow platter, so every non-media operation on a relocated channel (index and stats builds, the video page, digests, normalize) waits on it, and a platter stall holds the whole channel. The big files alone belong on the slow drive and the text on the SSD; the same split for umtool (manifests stay, renders go). 13 of 76 channels are relocated whole today and are migrated by a one-off script. A channel export/import bundle is a LATER slice built on this release's file classifier — not part of this one. **Measured 2026-10-01** (the 13 relocated channels, MB): media (`audio.*`, `source-media.*`) 323,000; everything else 91,008 — of which raw `transcript.live_chat.json` 33,457 (rekietalaw 20,958, friday-night-tights 7,460, kirsche 4,706) — and `clips/` 15,408 (nuxanor-kick 15,317). `/home`: 1.5 TB, 123 GB free, gate floor 5 GB, resume margin 2 GB. The index never opens a media file (presence by name from one `readdir`; every `stat` is a sidecar). yt-dlp writes cwd-relative into `data//` and its postprocessor, like the app's transcode, `rename()`s a temp file OVER the final name (a symlink there would be replaced by a real file). The saved-video store (`saved-video.json` pointer, EXDEV-safe move) is the existing per-big-object pattern; umtool's `SONG_DIR` is the existing symlink-root pattern. ## Step 0 — the editor dashboard does not load (diagnosed 2026-10-01, read-only; fixed first) **Symptom:** `/`, `/channels`, `/jobs` give no response in 40 s; `/settings` answers in 1 s; `/api/pulse` 200; `/api/jobs/active` and `/api/auto-queue/status` never answer. **Cause (measured):** the editor's main thread is pegged (~97 % of a core, 485 ticks/5 s; process 435 % across V8 threads; the Platter idle, no errors in the log). `generateChannelSnapshot` (`common/controller/channelSnapshot.ts`) runs ON THE MAIN THREAD with no yielding, driven by `common/jobs/snapshotScheduler.ts:192` as `refresh-report` jobs; two live ones (omnibased 3,260 videos, the-quartering) have parsed thousands of ~500 KB `metadata.info.json` files for over an hour, starving every request that needs the loop. The dashboard's 3 s poll of the auto-queue status (no memo, no single-flight in `common/views/autoQueueStatus.ts`; 96 s cold this afternoon) piles up behind it. Three more "running" `refresh-report` metas (the-quartering-rumble ×2 at 03:21/03:29Z, the-quartering 07:14Z) predate the 12:54 restart: ghosts of the killed process (`common/jobs/shutdownCancel.ts:20` "graceful shutdown only"), never finalized — and slice RM's `channelWriters` reads running jobs by slug, so a move of the-quartering-rumble would refuse forever. **0a — done 2026-10-01 16:18:** the editor was restarted (`r15-restart.sh --force`; the build on disk was current, `Xfavvt2XajdNzOvvn0NtL`); `/` answered 200 after 3 s. The scheduler re-runs the two snapshots; `/` watched for ten minutes after (result in the Record). **0b — slice D0 `r17/dashboard-answers`** (small; beside T1; editor-side, one restart): 1. snapshot generation yields: chunk the per-video loop (`setImmediate` every N videos) so the loop serves requests between chunks — or run it in a worker thread; `refresh-report` concurrency 1 and dedup per slug in `snapshotScheduler.ts`; 2. `autoQueueStatus` view: single-flight + a short memo (≈ 3 s) so N pollers cost one digest; 3. boot finalizes stale `running`/`queued` metas from a dead process as `interrupted` (`registry.ts`, `editor/instrumentation.ts` already notes "anything a previous process left queued was queued before this instant"); `channelWriters` ignores them; 4. e2e: poll `/` and `/jobs` every 2 s while a large fixture snapshot regenerates — every response under 5 s; a ghost `running` meta from a fake dead pid is `interrupted` at boot and does not block a move. Gates: common + editor unit, `jobs`/`channels`/`channel-storage` specs, the capped editor build. ## Rulings (2026-10-01; do not re-open) - **Tier now, bundle later.** The classifier is a shared `common/lib` module the bundle slice reuses. - **Text = hot = SSD; media = cold = platter.** `transcript.live_chat.json` is COLD (media tier): read once by `normalizeLiveChat`, which derives the small `live_chat.cues.json` (hot). `clips/` stays on the SSD (a cache, age-evicted) — not tiered. - **The whole-directory layout is retired** after the migration; `config.dataDir` is tolerated one release as `legacy` (held) and removed later. - **umtool:** render scratch (`out/`) goes to a media root by default; deliverables (`clips/`, `share-*/`, final mp4s) move per project by a switch "like channels"; manifests, `revisions/`, the caches and the cue cache stay put. - **Migration:** one-off script, smallest text first, a stop before the three big-text channels (omnibased, rekietalaw, the-quartering-rumble) with the free space reported; the operator decides. - **A move holds only the channel's media writers**; a digest may run during a move. ## The model (A′) ``` channels// config.json mediaDir: "//media" (absent = in place) data/ REAL directory on the SSD, always: text, sidecars, clips/, scratch /audio.mp3 -> ../../media//audio.mp3 (RELATIVE link, one per big file) /transcript.live_chat.json -> ../../media//transcript.live_chat.json /transcript.json, metadata.info.json, live_chat.cues.json, clips/… (real files) media ONE absolute symlink -> //media (relocated) | a REAL directory on the SSD (tiered in place) | absent (classic: real files in data//) //media// (the bytes) ``` Why A′ over absolute per-file links or a pointer sidecar: readers keep opening `data//` by path (no reader changes; the index unchanged); the platter path exists in ONE link + ONE config key per channel, so slice RM's mover — marker, writers hold, copy, mirror (`--delete` toward the copy only), verify, Reconcile, re-point, rename, delete — carries over by renaming `data`→`media`; relative links survive `reconcileVideoDirs`' renames and a channel rename; move back makes `media/` a real SSD dir and every link stays valid (no "untier"). A classic channel keeps real files until its first move, whose preflight tiers it in seconds (same-filesystem `rename` + `symlink`); e2e fixtures need no layout change. ### 1. Classifier — `common/lib/mediaTier.ts` (pure, no fs; exported for the bundle slice) - `classifyEntry(name) → "media" | "text" | "scratch"`, BY NAME never by size, over `mediaFiles.ts`'s anchored predicates: media = `isRealAudioFile(name) || isSourceMediaFile(name) || name === LIVE_CHAT_FILENAME` (`transcript.live_chat.json`); scratch = `isPartAudioFile`, `/^audio\.tmp-\d+\./`, `/^\.audio\..*\.parakeet$/`, `/\.part-Frag\d+$/`, `/\.temp\./`, `/\.part\.(good|testing)$/`; text = everything else. `CLIPS_DIR_NAME` is never tiered. - `isTierable(name)` = `isRealAudioFile(name) || name === LIVE_CHAT_FILENAME` — narrower than "media": `source-media.*` stays real (`persistSourceVideo` would `rename` the LINK into the saved-video store) and `audio.*.part` stays real (yt-dlp's resumable partial). - `classifyVideoDir(entries) → { media, text, scratch }`. ### 2. Hook + helpers — `common/lib/mediaTier-server.ts` (fs, no controller import) - `channelMediaLink(paths, slug)` = `channels//media`; `relocatedMediaDir(root, slug)` = `//media` (replaces `relocatedDataDir`, suffix fixed); `tierLinkTarget(id, name)` = `../../media//`. - `tierMediaFile(videoDir, name) → "tiered" | "left" | "already"`: lstat a link or missing → already; `channels//media` not a directory (classic channel, or ENOENT through a dangling link when the platter is unmounted) → "left" (the file stays real, readers unaffected, the next sweep tiers it); `mkdir(media/)` NON-recursive (a bare mountpoint is never filled); `rename` → EXDEV → `copyFileAtomic` + rm (the `savedVideo-server.ts:70-80` pattern); `symlink(relative)`. Never throws into a download. - `tierVideoDir(videoDir)`; `tierChannelMedia(paths, slug, { since?, createMediaDir? })` (the mover's preflight passes `createMediaDir: true`, making `media/` a real dir when neither link nor dir exists). - `removeMediaFile(videoDir, name)` (readlink → rm target, rm link); `removeVideoDirMedia(videoDir)`. - **Every media-finalising site calls the hook** (one line each): `common/controller/transcode.ts:55` after `rename(tmp, out)` (covers `downloadOneManaged.ts:278`, `audioCheckedDownload.ts:769`, `videoActions.ts:128`); `downloadOneManaged.ts` `tierVideoDir(videoDir)` before both `writeDownloadOutcome` calls (`:1001`, `:1629` — after `reconcileVideoDirs`, covers yt-dlp's own `-x` and the legacy path); `runYtdlp.ts` batch runs (sync, download-from-playlist, download-missing, retry) → `tierChannelMedia(paths, slug, { since: runStartedAt })` after the child returns; `normalizeLiveChat` → `tierMediaFile(videoDir, LIVE_CHAT_FILENAME)` after writing the cues (and the download path's live-chat write). `persistSourceVideo`/`unpersistSavedVideo` unchanged; `fetch-window` → `clips/` no hook. - **Every media deleter derefs** (today each would orphan the platter file): `cleanAudioFromTranscribed.ts:158`, `cleanExtraAudioFormats.ts:75`, `removeWrongFormatAudio.ts:73`, `videoActions.ts:598`, `:490-503` (`deleteFile`), `deleteVideoDirAction` (+ `removeVideoDirMedia`), `backfillReacquire.ts:506`. Grep gate: `git grep -n "remove(path.join(.*videoDir\|rm(path.join(videoDir" common editor` hits only `mediaTier-server.ts`. - One reader changes: `editor/app/channels/[slug]/videos/[id]/videoActions.ts` `loadVideoDir` filters `isFile()` on dirents → `isFile() || isSymbolicLink()`, media entries stat'd through `onDrive(mediaDir)`. `measureTree` keeps `isFile()` (over `data/` it measures SSD text; over `media/` the real bytes). ### 3. `config.json` - `mediaDir?: string` (`channelConfig.ts`, `channelConfigSchema.ts`, CHANNEL.md row): "Where this channel's big files live when relocated: `channels//media` is a symlink to it, `//media`. Absent = in place. Written only by relocate / re-point / the tier migration." - `dataDir` stays parseable one release, documented RETIRED: a channel carrying it (or whose `data/` is a link) is `legacy` and held until `archilyzer storage migrate-tier ` runs. (`parseChannelConfig` is allow-list; dropping the key would make `patchChannelConfig` erase it on the first stamp → `inconsistent`.) ### 4. Statuses, guards, lanes, health (`common/lib/channelMedia.ts`, `channelMediaHold.ts`) - `ChannelMediaStatus` gains `"legacy"`; the others now describe `channels//media` + `mediaDir`: marker → `in-transition`; `data/` a link or `dataDir` set → `legacy` ("its media layout is the retired whole-directory one — run archilyzer storage migrate-tier"); no media link/dir and no `mediaDir` → `in-place`; link ≠ `mediaDir` or dir while `mediaDir` → `inconsistent`; `mediaDir` + link agree → `stat(target)` via `onDrive(mediaDir)` → `ok | unreachable | stalled`. - `ChannelMediaLocation`: `dataDir` (always the real text dir), `relocated`, `target` (= mediaDir), new `mediaLink`, `text: { dir, readable }`. `channelMediaStall` keys on `mediaDir`; `storageHealth.ts:529` `rootOfUnknownPath` strips `//media`. - Two guards: `assertChannelMediaReachable` (unchanged: `ok | in-place` pass — jobs that open the big file) and new `assertChannelTextReadable` (passes for all but `legacy`/unreadable `data/`). `isMediaHeld` (+ `legacy`), new `isTextHeld = status === "legacy"`, `HELD_REASON.legacy`. - `jobKinds.ts` `needsMedia` keeps its name with the doctrine "opens or writes the BIG file". Flip to false: `auto-digest`, `digest-channel-local/remote`, `digest-share-cluster`, `normalize-transcripts`, `purge-superseded-auto-subs`, `fetch-window`, `evict-clips`, `metadata-scan`, `download-missing-subs`, `check-availability`, `quick-availability-check`, `check-maybe-missing`, `check-kept-deleted`. Stay true: every download kind, `sync`, `import-one`, `redownload-*`, `retry-bucket`, `persist-kept`, whisper/transcribe, `diarize-channel`, `backfill-channel`/`auto-backfill`, `scan-media*`, `clean-*`, `remove-wrong-format-audio`, `transcode-audio`, and `normalize-live-chat` (reads the cold raw file). - Walkers call the text guard: `buildIndex.ts` `scanSource`, `buildStats.ts:291` — **new rule: the index and stats builds read the text tier only and are never held by the media tier** — `channelSnapshot.ts:744`, `operationBatch.ts:1580`, `normalizeAll.ts:63`, `shardActions.ts:88`. Lanes (`autoRunner.ts:370-386,641`): `isMediaHeld` for transcription/download/backfill, `isTextHeld` for digest. - `onDrive` keyed by FILE KIND: media reads wrap `onDrive(config.mediaDir, …)`, text reads pass no drive. Change: `channels.ts:108`, `channelSnapshot.ts:753` (sidecar stats direct; one `onDrive(mediaDir)` per video for its media entries; `DriveNotAnswering` → that channel's media bytes `null`, snapshot still written), `recencyIndex.ts:258`, `videos/page.tsx:153`, `videos/[id]/page.tsx:96,201`, the file route (`classifyEntry(name) === "media"` → `onDrive(mediaDir, stat)`; **ENOENT on a path whose `lstat` is a symlink → 503 "drive not answering/unmounted" with `retry-after`, not 404**). Stall budget: text never takes a slot. `storageWatch.ts` auto-pause stays one tier per channel. - `channelWriters(slug, { mediaOnly })`: jobs filtered by `kindNeedsMedia`, lane units from lanes ≠ digest; the move's `liveWriters`/`previewRelocation`/`assertNoWritersUnderMarker`/`channelMediaBusyReason` pass it. ### 5. The mover — `relocateChannelMedia.ts` over `media/` - Marker `.relocating.json` unchanged shape, `target = //media`, optional `scope: "media" | "tier-migration"` (readers ignore unknown keys; `cues.mjs`' twin too). - `moveOut` preflight, after the writers check and before `measureTree`: `tierChannelMedia(paths, slug, { createMediaDir: true })`; then today's phases with `dataDir`→`channelMediaLink`, parked names `media.relocated-` / `media.incoming`, `live: channelMediaLink` for `assertMirrorDirection`, `patchChannelConfig(…, { mediaDir })`. Preview tiers a classic channel the same way (same-fs, idempotent) and reports "N files tiered first". - `moveBack` identical, ending with `media/` a real SSD dir and `mediaDir` unset. Space check = `media/` bytes + margin. `relocationRootProblem`/`assertRelocationRootPresent`/`rootOfRelocatedMediaDir`. - `repointStorageLocation` (`storageLocations.ts:698-770`): link `media`, key `mediaDir`; `channelsOnLocation` keys on `mediaDir`; `locationOfDataDir` keeps its name (pure prefix test). `renameChannel.ts:125-180` and `deleteChannel` (`channels.ts:590` → `rm(config.mediaDir)`) follow. ### 6. Surfaces - Storage panel (`StorageStage.tsx`): two rows under one heading — `aria-label="media path"` (kept; `target ?? "/media (in place)"`), new `"text path"` (always in place), `"text bytes"`, `"media bytes"` (media tier only), `"free on media volume"` kept. Buttons keep their names (`Move media`, `Preview`, `Resume move`, `Reconcile and resume`, `Clear marker`); the hold sentence gains "its text stays readable". `MediaLocationBadge` keeps every label and adds `legacy: "Media layout retired"` (danger). - Snapshot (`channelSnapshot.ts:877-890`): `totalTextBytes`, `totalMediaBytes` (media tier only), `totalClipsBytes` (no longer inside media); an old snapshot without `totalTextBytes` reads as unknown. `/storage` (`buildStorage.ts`, `views/storage.ts`): location rows show media-tier bytes; the internal row "text N GB + clips N GB on the corpus volume, plus media of in-place channels"; `legacy` counts as unreachable with "(n to migrate)". Rack chip and `mediaHold` unchanged. - umtool twin `umtool/report-to-video/cues.mjs:285-360`: refuse `legacy` (data link or `dataDir`); refuse a marker only when `scope === "tier-migration"`. ## The one-off migration — `common/bin/migrate-media-tier.ts` Wired as `archilyzer storage migrate-tier |--all [--dry-run] [--reclaim] [--order smallest]` (`archilyzer.ts:268` `script([...])` beside `migrate channel-priority`). Editor STOPPED (it cannot see the in-process registry; refuses on a marker and says so). Dry run writes nothing; idempotent; resumable from the marker's phase. Per channel with `config.dataDir = D` (`//data`): | phase | action | |---|---| | preflight | status must be `ok`; `assertRelocationRootPresent(root)`; walk `D` with `classifyVideoDir` → counts/bytes by kind; space: `textBytes + resumeMarginGB ≤ free(channelsDir) − 5 GB floor` | | `copy` | marker `{target: //media, direction: "out", phase: "copy", scope: "tier-migration"}`; NUL list of text + scratch entries (classifier, not globs); `rsync -a --partial --info=progress2 --from0 --files-from=$T/.txt D/ channels//data.incoming/`; verify `rsync -a --dry-run --itemize-changes --from0 --files-from=… D/ data.incoming/` empty AND per-kind counts/bytes equal; then for every `` and every `isTierable` name: `mkdir data.incoming/`, `symlink("../../media//", …)` (EEXIST = already) | | `swap` | platter: `rename(//data, //media)`; SSD: `symlink(//media, channels//media)`; `unlink(channels//data)`; `rename(data.incoming, data)`; `patchChannelConfig(slug, { mediaDir }, { unset: ["dataDir"] })` — each step observes the disk (`linkOrDirState`) first | | `reclaim` | only `--reclaim`: delete from `//media//` every entry that is not `isTierable` (the platter's text copies and scratch), drop empty dirs; without it the duplicates stay until `migrate-tier --reclaim` | | done | clear the marker; print bytes by kind and links made | No index rebuild: `rsync -a` keeps the text mtimes and the index stats only sidecars. The snapshot's new byte fields come on the next Refresh report. Order `--order smallest`: realcandaceo (6 MB text), nuxanor-kick, omnimirror, leaflit-rumble, cornbreadman, shondo-vods, piratesoftware, kirsche, friday-night-tights, HasanAbiVODs3 (≈ 36 GB of SSD with live chat cold → ~16 GB; plus 15 GB clips for nuxanor-kick) — then STOP and report free space before omnibased (15.2 GB), rekietalaw (8.9 GB with chat cold), the-quartering-rumble (10.3 GB). Run `purge-superseded-auto-subs` on omnibased/HasanAbiVODs3 first (≈ 7.6 GB of `en-orig.vtt`). Live proof: realcandaceo (seconds) → Refresh report, one video page, one short Transcribe (the hook on a relocated channel), `/storage`, `df`; then nuxanor-kick; then `--all`. ## umtool - `umtool/lib/paths.mjs`: `MEDIA_ROOT = resolve(/* turbopackIgnore: true */ process.env.UMTOOL_MEDIA_DIR ?? REPORTS_ROOT)`, `MEDIA_TIERED`, `mediaMirror(abs)` = the project-relative path under `MEDIA_ROOT`; `MEDIA_ROOT` joins `READ_ROOTS` (never `WRITE_ROOTS`). `CACHE_DIR` decoupled from `SONG_DATA`: `UMTOOL_CACHE_DIR ?? $XDG_CACHE_HOME/archilyzer/umtool`; the e2e env sets it to `${FIXTURE}/cache` (`playwright.config.ts:62,73`, `projects.spec.ts:318`, `report-longform.spec.ts:21`, `make-fixture.mjs`); rollout runs `umtool index` once; `umtool doctor` reports both roots and a leftover old cache. - `umtool/lib/report/storage.mjs`: `ensureOutDir(projectDir)` (real dir → it; link → it; absent + tiered → `mkdir(mediaMirror(projectDir)/out)` + `symlink`; absent + untiered → `mkdir`), `moveDirToMedia(projectDir, name)` (rsync copy, mirror `--delete --info=del`, verify, `rename(name → name.moved-)`, symlink, rm parked; idempotent) and `moveDirToLocal`. Called at `driver.mjs:74,87`, `build-video.mjs:2615` (`outRoot` default), `export.mjs:71`. A dangling `out` link makes `mkdir -p out/clips-raw` fail loudly (nothing materialised). tmp-then-rename sites (`cut.mjs:98-99`, clip route `:52-55`) untouched: tmp and final share a directory either way. `umtool storage move-out |--all` moves existing `out/` trees (≈ 18 of the 20 GB in `~/reports`). - Deliverables: `video.manifest.json` gains `"storage": { "deliverables": "local" | "media" }` (absent = local), written only through `lib/report/manifest.mjs`'s writer; `umtool storage deliverables --to media|local` and a bench button "Move deliverables" move `clips/` and every `share-*/` with the movers (only when nothing is cutting/sharing), then set the field; `cut.mjs:96` and `deliver.mjs:362` create their dir through `deliverableDir(project, name)` (links when `media`); relative references (`clips/.mp4`) stay valid through the link. `kinds.mjs` `SKIP_DIRS` adds `clips`, plus a `SKIP_PREFIXES = ["share-"]` so an unmounted media drive is never descended. Final mp4s travel with `out/`. `umtool check` learns the two values. ## Slices (record: this file; `editor/CHANGELOG.md` + `umtool`'s changelog `[Unreleased]`) | slice | branch | owns | after | gates beyond the common set | |---|---|---|---|---| | **D0** dashboard answers | `r17/dashboard-answers` | `common/controller/channelSnapshot.ts` (the yielding loop only — T1 owns its guard/bytes changes, so D0 lands first and T1 merges main), `common/jobs/snapshotScheduler.ts`, `common/views/autoQueueStatus.ts`, `common/jobs/registry.ts` (stale-meta finalisation), `editor/instrumentation.ts`, `common/controller/channelWriters.ts` (ignore interrupted), a new e2e spec | — | see Step 0b | | **T1** classifier + model + guards | `r17/media-tier-model` | `common/lib/mediaTier.ts` + `mediaTier-server.ts` (+ tests), `channelConfig*.ts` + CHANNEL.md (`mediaDir`, retired `dataDir`), `channelMedia.ts` (`legacy`, `media` link, two guards), `channelMediaHold.ts`, `storageHealth.ts:529`, `jobKinds.ts` flips, `autoRunner.ts` lanes, the walkers' text guard (`buildIndex`, `buildStats`, `channelSnapshot` + byte fields, `operationBatch`, `normalizeAll`, `shardActions`), `channelWriters.ts` `mediaOnly`, the hook call sites (transcode, downloadOneManaged, runYtdlp, normalizeLiveChat, backfillReacquire), every deleter, `recencyIndex.ts`, `channels.ts`, `cues.mjs` twin | — | classifier table incl. `audio.en-orig.vtt`, `audio.tmp-2760235.mp3`, `source-media.temp.mp4`, `transcript.live_chat.json`; hook tier/already/left, EXDEV (injected `rename` throwing EXDEV), dangling `media` link creates nothing, non-recursive mkdir; deleters deref; `channelMedia.test.ts` + legacy cases; `jobKinds` flips pinned; "a stalled media location does not hold the index/stats/snapshot, holds the transcription lane, lets the digest lane run" | | **T2** mover + surfaces | `r17/media-tier-mover` | `relocateChannelMedia.ts` (over `media/`, preflight tier), `relocateDir.ts` (names), `storageLocations.ts` (lib + controller: re-point, rollups), `renameChannel.ts`, `deleteChannel`, editor `StorageStage.tsx`, `storageActions.ts`, `relocationJob.ts`, `bulkStorageActions.ts`, `MediaLocationBadge.tsx`, `channelRow.ts`, `videos/[id]/page.tsx` + `videoActions.ts` (`loadVideoDir`, file-kind `onDrive`), the file route 503, `videos/page.tsx`, `[slug]/page.tsx`, `buildStorage.ts`, `views/storage.ts`, `storageWatch.ts` | T1 | `relocateChannelMedia.test.ts` rebased on `media/` (classic → tiered → moved; interrupt at each phase + resume; reconcile; back leaves a real `media/`); `renameChannel`, `storageLocations`, `channels` tests; e2e `channel-storage`, `storage-locations`, `channels-storage-columns`, `bulk-actions`, `maybe-missing`, `video-page`, `fetch-window` (+ the new cases under Verification) | | **T3** migration + records | `r17/media-tier-migrate` | `common/bin/migrate-media-tier.ts`, `archilyzer.ts` wiring, fixture tests (tmp "platter"), FACTS "A channel's media is tiered", AGENTS.md's six things → seven, SETTINGS.md/CHANNEL.md regen, the release record, changelog | T1, T2 | dry run; resume from each phase; idempotent rerun; `--reclaim`; refusal on a marker; the free-space stop | | **U1** umtool roots + `out/` | `r17/umtool-media-root` | `paths.mjs` (`MEDIA_ROOT`, `CACHE_DIR`), `lib/report/storage.mjs`, `driver.mjs`, `build-video.mjs:2615`, `export.mjs`, `kinds.mjs`, `umtool doctor`, `umtool storage move-out`, the e2e env | — (∥ T1) | `test:scripts` (+ mover tests), `next-build-trace.test.mjs`, the capped umtool build with the corpus linked, umtool e2e | | **U2** deliverables switch | `r17/umtool-deliverables` | manifest `storage` field, `deliverableDir`, `cut.mjs:96`, `deliver.mjs:362`, `umtool storage deliverables`, bench "Move deliverables", `umtool check` | U1 | umtool unit + e2e: cut and share through a linked `clips/` | | **XP** X posts are private (operator-requested, beside the media tier) | `r17/x-posts-private` | `common/lib/postsVisibility.ts` (new) + test, `settingsSchema.ts` + `social/xCookieSource.ts` (`social.x.visibility`) + SETTINGS.md, `siteSchema.ts` + `site.ts` (`audience`, `isListedSite`, `resolveHubUrl`) + SITE.md, `buildIndex.ts` (the per-site loop only), `bin/compose-site.ts` + an integration test, `lib/corpus.ts`, `lib/builtExport.ts`, `publish/build.ts` + tests, `controller/poolSummary.ts` + test, `docker/publish-site.sh`, `export/app/offline/page.tsx`, editor `settings/{xSessionActions.ts,page.tsx,components/{XSessionSection,XPostsVisibilityControl}.tsx}`, `sites/{actions.ts,components/SiteForm.tsx,lib/{buildAction,deployAction}.ts}`, e2e `x-session`, `sites-crud`, export `x-posts-private` | — (∥ all) | the compose integration test (public vs private site, flip back, an X-only public site); deploy refusals before wrangler and before the upload | | **RL** a subtitle 429 does not fail a download; the pace adapts (operator-requested, beside the media tier) | `r17/rate-limit-adapts` | `common/jobs/{platformBackoff,downloadBackoff,unitOutcome,autoQueueState}.ts`, `lib/availability.ts`, `ytdlp/{platformArgs.mjs,channelArgs.ts}`, `downloadOneManaged.ts` (subtitle handling + pace only), `autoRunner.ts` (the download lane's gap, hold, probe), `runYtdlp.ts` (pace into `runManagedDownloads`, download-missing-subs), `checkAvailability.ts` (pace only), `settingsSchema.ts` `pacing` + SETTINGS.md, `RunnerOperationView.tsx` + `dispatch.ts`, `views/activeJobs.ts`, `pipelineActions.ts` + `videoActions.ts` (refusals, the subtitle line), `bin/doctor.ts`, `editor/e2e` | — (∥ D0, T1, U1) | unit: classifier, `applyUnitOutcome`, `channelExtraArgs` pace, the lane gap; e2e `rate-limit.spec` + `auto-queue`, `lane-runner`, `channel-priority`, `video-page` | Order: 0a → D0 ∥ T1 ∥ U1 → T2 ∥ U2 → T3 → parent: records, ONE editor rebuild + restart, umtool rebuild + restart (the restart is the operator's: the permission layer refuses the `0.0.0.0` bind) → the migration (editor stopped for it) → live proof → records/STATE/memory. AGENTS.md's seventh non-optional thing: **"A media file in `data//` may be a relative symlink into `channels//media/`. Remove one with `removeMediaFile`, never `rm`/`remove`; tier one with the hook after every media finalisation, never by hand; a dirent `isFile()` filter over a video dir hides it."** The `.relocating.json` bullet gains `scope`; the `clips/` bullet gains "on the SSD, never tiered". ## Verification - Unit as the table says; plus the health split end to end: a stalled/absent media location → index, stats and snapshot run; digest lane runs; transcription lane skips; `whisper-video` refused; `normalize-transcripts` runs; `normalize-live-chat` refused. - e2e (`channel-storage.spec.ts` rebased): after Move media, `data//audio.mp3` is a link and `transcript.live_chat.json` is a link; the videos list and the file route serve the transcript AND stream `audio.mp3` through the link; a renamed-away media root (the `seedDriveChannel` + rename trick from `buildStats.test.ts`) keeps the video page's text readable, the file route answers 503 for `audio.mp3`, the rack shows "media held", the digest lane still picks the channel; Move back; a `legacy` fixture shows "Media layout retired" and a media job's refusal names `migrate-tier`. `migrate-tier` on a fixture: dry run, run, rerun, `--reclaim`, kill between `copy` and `swap`, resume. - Live: realcandaceo → nuxanor-kick → `--all --order smallest` with the stop at the big three; after each channel: Refresh report, one video page, one short Transcribe, `/storage`, `df`. ## Assumptions the operator can overturn | assumed | alternative | |---|---| | A′: relative per-file links into `channels//media/`, `media/` is the unit of relocation | absolute per-file links (no `media/` tree): a per-video mover for out and back, O(files) rewrites on re-point/rename | | `dataDir` kept one release as `legacy` (held) | drop it now; the migration reads the raw file (`migrate-channel-priority.ts` pattern); risk: a stamp before migration erases the record | | a move holds media writers only; digests run during a move | hold every writer as RM does today (no `channelWriters` change) | | the `media` link is `channels//media` (visible) | `.media` (hidden) | | platter reclaim is opt-in (`--reclaim`) | reclaim in the same run after verify | | the move's preview tiers a classic channel (same-fs renames) | preview only reports "N files to tier"; the job tiers | | umtool `out/` is a directory link made by the first writer; existing `out/` stay until `move-out` | resolve `out/` through a function (no link): ~20 join sites + `deckPreviewFile`'s base change | | auto-pause stays one tier per channel | a media stall pauses only the media lanes (per-lane `autoPaused`) | | `UMTOOL_MEDIA_DIR` unset by default (behaviour unchanged until set) | default to `settings.storage.locations[platter].root + "/umtool"` (umtool reads no settings today) | ## Open questions (settled during T1/U1; the answer is recorded in the slice's section) 1. What `import-one` writes (its controller was not found; `pipelineActions.ts:487` enqueues it) — assumed a media writer. 2. The four `*.mp4` at umtool project roots — deliverables or inputs? Left untouched. 3. `reconcileVideoDirs` on a tiered dir: links move with the files; `media//` keeps its old name (valid, untidy) — a follow-up renames it. 4. Whether the Platter's media root should be created as `//media` beside today's `//data` (the migration renames one to the other) — yes as planned; confirm no other tool hardcodes `//data`. ## Follow-ups carried over (not scheduled; keep) - "Load more results" never resumes a leaf that settled at its cap (`runQueryTree`'s `setHitLimit` reaches running leaves only) — release 16 CK R-I1; wants a spec. - `/changelog/` overflows a 390 px phone (long inline code in `export/CHANGELOG.md:103`); wrap `` in the changelog renderer, then drop the `test.fail` in `export/e2e/responsive.spec.ts`. - `export-search.spec` "Advanced reset does not touch filter checkboxes": the "Deleted" checkbox locator also matches a result card's "Select … for AI" box — `exact: true`. - `social-channel.spec` fetch-posts case runs 22–26 s of a 30 s budget — `test.slow()` or a 60 s timeout. - The JSX entity/whitespace hazard sweep: 22 texts in 19 files (FACTS). - The editor's and export's build traces list dot-directories; `REQUIRED_TOKENS` lacks `--chart-other`; export `WorkspaceView` `splitOn` one-paint flash. - The build-only "Build all" through the rebuilt build image when the machine is idle. - Docker for others: no pull-able image; needs a GitHub home first; the version isn't tied to `release cut`. - X section by-hand check (release 16 XL live). - `channel-rename.spec`'s refresh race; RM L5: a cancelled job whose function never returns holds the channel's move until a restart (offer Force release on a cancelled row). - XL L2/I-items: Firefox store discovery differs from gallery-dl's; the profile side's "logged in" check accepts any X domain. - The `en` track → 0 cues bug (index prefers `en` over `en-orig`; some `en` VTTs parse to 0 cues). - A channel export/import **bundle** built on `mediaTier.ts`'s classifier — the slice after this release. ## Slice XP — the ruling (2026-10-01) - **Every X post is hidden from the public, for now; the data is kept, and stays readable by the MCP and umtool for the operator's own questions and tasks.** Fetching is not changed by this slice. - **A setting, `social.x.visibility`: `"public"` (default) | `"private"`**, beside `social.x.cookieSource`, chosen on `/settings` in the X account session section as "Where X posts appear", with one sentence saying what private means and that sites already published change on their next build and deploy. `"private"`: every X channel's posts (`sourceKind: "social"`, `platform: "twitter"`) are left out of every PUBLIC site build and built only into PRIVATE sites. Nothing on disk changes; flipping back is a rebuild. - **A site audience, `site.json` `audience`: `"public"` (default, absent) | `"private"`**, on the site's form with a sentence. A private site is **never deployed** — every deploy path refuses it with a sentence naming the audience, before any upload, where the release 13 W3 wrong-site guard runs; a build-only still works — and **never listed**: no `hubUrl`, in no homepage or hub listing. Its `corpus.json` says `"audience": "private"`. - **One predicate, `postsVisibleTo(site, channelConfig, settings)`**, pure and tested in `common/lib`, called from the index build and from compose. The Search in row's Posts toggle keeps working from what the build shipped (a public site whose only posts were X posts has no posts corpus and no Posts toggle) — verified, not special-cased. The MCP needs no change; umtool's report pipeline is checked for where it reads posts. - Not in scope: stopping fetches; a per-platform toggle for Bluesky; deleting anything; editing `transcripts/**` (the rollout — a private site holding every channel, the setting flipped, the public sites rebuilt — is the parent's, through the editor's own writers). ## Slice RL — the ruling (2026-10-01) Operator-requested, beside the media tier. **Measured 2026-10-01 (read-only, live corpus):** every one of the 78 HTTP 429 lines in 24 h of job logs is `Unable to download video subtitles for 'en': HTTP Error 429` on YouTube; none on the watch page, the player API, the m3u8 or a listing; no `sync` or `import-one` log carries a real 429. The YouTube backoff reads `fails: 80`, consecutive since the evening of 2026-09-30; the lane retries one unit every ~30 min and each burns another subtitle 429. Twelve videos of one channel rotate through the 6 h deferral and fail again each time; `redownload-archive` jobs on other videos of the same channel succeeded in between. Each failing unit had already picked its formats and died on the subtitle file. The 2026-09-25 postmortem (`plans/youtube-lane-pacing.md`) found the same: the timedtext endpoint 429s per video, not IP-wide. Today's flags: `--sleep-requests 1` for YouTube, `-t sleep` on the primary spawn, no `--sleep-subtitles` override; yt-dlp re-extracts once after a subtitle 429 then fails; the app never retries a `rate_limit`. 1. **A subtitle 429 never fails a download.** When yt-dlp fails only on the subtitle file, the unit downloads the media anyway and succeeds — one spawn where yt-dlp's own `--ignore-errors` allows it, else a second spawn without subtitles. `classifyDownloadFailure` gains `subs_rate_limit` for "the subtitle fetch is the only failure". It is recorded per video as a subtitle deferral (beside `videoDeferrals`, through `downloadBackoff.ts`'s write-through), does NOT touch the platform backoff and does NOT defer the video's media. The deferred subtitles are picked up by `download-missing-subs`; the transcription lane sees "downloaded, no transcript". After 3 subtitle deferrals the video's subtitles are left alone for 7 days (shown on the video page with the count and the date; a manual fetch still works). `plans/youtube-lane-pacing.md` gets a dated note pointing here. 2. **The pace adapts per platform.** A persisted per-platform `platformPace`: `--sleep-requests` starts at the platform's static value, doubles on every platform-level `rate_limit` (never on `subs_rate_limit`) up to a cap, and decays one step toward the base after every N clean units. It feeds `channelExtraArgs` (every yt-dlp call for the platform) and the lane's gap: the download lane honours `sleepBetweenDownloadsSeconds` AND adds the adaptive pace. Settings in a new `pacing` block (cap 16 s, decay 5 units, hold threshold, probe interval). YouTube's primary spawn gets `--sleep-subtitles `. 3. **A block is not a burst.** A platform whose backoff has failed at the cap `holdAfterFailsAtCap` times in a row (3) is HELD: one probe unit per `holdProbeMinutes` (60) instead of one every 30 min; a clean probe clears the hold and the backoff. Persisted beside the backoff. A manual Sync/download on a held platform is refused with a sentence naming the hold and the next probe. 4. **Visible.** The download lane page's "Rate-limit cooldown" region shows per platform the cooldown or hold (since, fails, next try/probe), the current pace, and the videos whose subtitles are deferred (with counts); a rack chip for a held platform only if it is a few lines; `archilyzer doctor` warns on a platform in cooldown/hold and on the pace; the idle reasons gain `held` and `subs-deferred` where the dispatch text names them. Not in scope: yt-dlp's player client or cookies for subtitles (an open question for the operator: whether the timedtext 429 for these twelve videos is a PO-token/client matter — not probed by hand); the sync scheduler's per-channel backoff; Rumble/Odysee specifics beyond the shared code. ## Record ### Slice U1, as shipped — umtool's render scratch goes to a media root (2026-10-01) Branch `r17/umtool-media-root` off `main` `7f4901f1`, `main` `90bd8384` (the deck/posts-room merge) merged in mid-slice, worktree `~/Projects/homepage-social-visible` (`pnpm wt list` block #11: editor 4101, test 4111, export 4110), one Opus implementer. Scratch files `U1-*` in the job's `tmp`. The ruling is the plan's: render scratch (`out/`) goes to a media root by default; deliverables move per project by a switch (slice U2); manifests, `revisions/`, the caches and the cue cache stay put. **What it does.** - **Two roots, one new knob.** `umtool/lib/paths.mjs`: `MEDIA_ROOT = UMTOOL_MEDIA_DIR || REPORTS_ROOT`, `MEDIA_TIERED` (they differ), `mediaMirror(abs, roots?)` (a path under `REPORTS_ROOT` → the same relative path under `MEDIA_ROOT`, null outside; pure). `MEDIA_ROOT` joins `READ_ROOTS`, never `WRITE_ROOTS`. Unset, nothing changes: `out/` is a directory in the project, and `READ_ROOTS` dedupes it away. Every path op carries `turbopackIgnore`. - **The cache leaves `SONG_DATA`.** `CACHE_DIR = UMTOOL_CACHE_DIR || $XDG_CACHE_HOME/archilyzer/umtool` (an empty `XDG_CACHE_HOME` is unset, as `common/lib/paths.ts` reads it; default `~/.cache`). `INDEX_DIR`, `MIX_CACHE`, the posters, loudness and clip audio follow it. `OLD_CACHE_DIR` (`/.cache/umtool`) is named only for the doctor. Nothing is migrated: the index is rebuilt by `umtool index` and everything else is remade on demand. The cue cache (`REPORT_CACHE_DIR`, `report-to-video/cues.mjs`) is untouched. - **`umtool/lib/report/storage.mjs`** (new; modelled on `common/controller/relocateDir.ts`, not importing it): - `ensureOutDir(projectDir, roots?)`: a real `out/` → kept; a link to a directory → kept; a **dangling link → refused** ("… is a link to …, which is not there — is the media drive mounted? Nothing was written, and nothing was created in its place."); absent and tiered → `mkdir -p /out` and an absolute `symlink`; absent and not tiered → `mkdir` as before. The media root itself is **stat'd and never created** (`mediaRootProblem`: missing, not a directory, or inside/around `REPORTS_ROOT`). A project outside `REPORTS_ROOT` is never tiered. EEXIST from a concurrent first writer is accepted when the winner resolves. - `ensureWriteDir(dir)`: a directory a pipeline step writes into; when it is a project's `out` or up to four levels under one, that `out` goes through `ensureOutDir` first, then `mkdir -p`. - `moveDirToMedia(projectDir, name, opts)` / `moveDirToLocal(...)`: by NAME (`out` now; `clips`, `share-*` for U2). Copy (`rsync -a --partial`), mirror toward the copy only (`-a --delete --info=del`; refused when source and copy contain one another), verify (`--dry-run --itemize-changes --delete` empty, one more mirror pass on a difference, a second refuses; equal counts/bytes), then park (`.moved-`), link, delete the parked copy. Space check on the destination's volume (bytes + 1 GB). Every state is dispatched on the disk, so a cut run is finished by running it again: a link to the mirror → `already` (a leftover parked copy removed); absent with one parked copy → link and delete it; the reverse uses `.incoming`, and after the rename deletes the media copy and every directory above it the move left empty, never the root. `dryRun` measures and changes nothing. - `outDirState`, `pathState`, `measureTree` for readers and the CLI. - **Call sites.** `build-video.mjs` (`outRoot`, before any fetch), `check-availability.mjs` (its one write), `render-cards.mjs` (CLI `--out`), `compose-chrome.mjs` (its `out/` base), `lib/report/onscreen.mjs` (`deckStill`'s scratch) all make `out/` through `ensureWriteDir`. `lib/report/export.mjs` reads only: it now says "out/ is a link to …, which is not there — is the media drive mounted?" instead of "no build" when the link dangles. tmp-then-rename sites (`cut.mjs`, the clip route) are untouched. - **The walk.** `kinds.mjs` `SKIP_DIRS` adds `clips` (`out` was already there) and `SKIP_PREFIXES = ["share-"]`, read through `skipsDir(name)` by `walk.mjs`, so the project walk never stats a link into a drive that is not there. The mix picker (`lib/media.ts`) follows a project's `out` link when it points INTO the media root (so a tiered deliverable stays in the picker under its project) and does not walk `MEDIA_ROOT` as a root of its own (it would list every tiered file twice). - **CLI.** `umtool doctor` adds `roots` (JSON) / a "roots" block: reports, media (tiered or "= reports"), cache (and whether an index exists), and the old cache with its size while it is there; it exits 1 when the media root is set and missing (the tools' `ok` keeps its meaning). `umtool storage []` lists every project's `out` (dir, link, DANGLING, none); `umtool storage move-out|move-back |--all [--dry-run] [--json]` runs the movers, one line per project and a total; move-out without `UMTOOL_MEDIA_DIR` refuses once. - **e2e env.** The app server and the specs' CLIs get `UMTOOL_CACHE_DIR=/cache` (`playwright.config.ts`, `projects.spec.ts`, `report-longform.spec.ts`; the index-deletion spec now removes `cache/index`), so no run writes `~/.cache`. `make-fixture.mjs` adds `storage-fixture` (a cached window, buildable offline), `storage-fresh-fixture` (no `out/`) and the media root `umtool/.e2e-song-media/`, a sibling of the fixture (inside it would be inside `REPORTS_ROOT`, which is refused), reset every run; `.gitignore` and umtool's trace excludes name it. `storage.spec.ts` (new, 5): move-out (dry run first; the index's state and facts unchanged through the link; again → already); **a build the app runs writes through the link and leaves it a link** (the app has no `UMTOOL_MEDIA_DIR` at all); the root renamed away → `storage` says dangling, `check-availability` refuses with the drive sentence on the moved project and with "is not there" on the fresh one, no `out` made, the root not recreated, `doctor` exits 1; the first writer of the fresh project makes the link; move-back → a real `out/`, the project's mirror gone, the root and the other project's mirror kept. **Commits** | Commit | What | |---|---| | `65a3d146` | `umtool:` `MEDIA_ROOT`, `MEDIA_TIERED`, `mediaMirror`; `MEDIA_ROOT` in `READ_ROOTS`; `CACHE_DIR` from `UMTOOL_CACHE_DIR` / `XDG_CACHE_HOME`; `OLD_CACHE_DIR` | | `47d6d1b4` | `umtool:` `lib/report/storage.mjs`; the five writers through `ensureWriteDir`; export's dangling sentence; `clips` + `share-*` skips; the picker follows `out` links into the media root; `doctor` roots; `umtool storage` | | `4c2cd0c7` | merge of `main` `90bd8384` (the deck/posts-room branch: `build-video.mjs`, `make-fixture.mjs` and more) — one conflict, `export.mjs`'s imports, both kept | | `cb08e57a` | `umtool:` `storage.test.mjs` (20); the e2e cache in the fixture; the storage fixtures, media root and `storage.spec.ts`; `umtool storage ` status | | `efef56b3` | `umtool:` `docs/folders.md` (`MEDIA_ROOT`, `CACHE_DIR`), `docs/cli.md`; two `[Unreleased]` bullets in `editor/CHANGELOG.md` | | this commit | `plans:` this section | #### Gates (logs `$T/U1-*`) - **tsc** (all workspaces) clean at `47d6d1b4`, at the merge `4c2cd0c7` and at `cb08e57a`. - **common:** 2,484/2,484 (300 s, under load). **Editor unit:** 109/109. Neither touched; run on the merged tree. - **test:scripts:** 390 tests (the merged `main`'s 370 + `storage.test.mjs`'s 20): 386 passed, 1 skipped, 3 failed, then 385/2/3 on a rerun — the three are `queue-lock.test.mjs` timing cases, a different three each time, at a load average of 27–47 (other implementers' suites and builds); `node --test scripts/queue-lock.test.mjs` alone: **11/11**. `storage.test.mjs` **20/20**. `next-build-trace.test.mjs` is in it: **10/10** after each capped build below (its second skip on the rerun is the staleness rule: the baseline run's `git checkout` of `main`'s umtool, below, gave the modules new mtimes after the build). - **The capped umtool build with the corpus linked** (`ln -sT /transcripts transcripts`, 76 channels visible through it; `systemd-run --scope -p MemoryMax=5G -p MemorySwapMax=0`, `timeout -s KILL 240`, the link removed after), at `cb08e57a`: **exit 0, 53 s, 0.83 GB peak**; and once more with `UMTOOL_MEDIA_DIR` set to a scratch directory: **exit 0, 53 s, 0.83 GB**. The two builds' `.nft.json` entries (39,658 each, every route) are **identical** (`diff` empty); none names `transcripts`, the scratch media root or `.e2e-song`. (The worktree carries an old `transcripts/` directory — an `index.mdb` — which `ln -sT` refuses to replace: the script sets it aside for the build and puts it back. A first attempt that did not was stopped before it counted.) The capped editor build was not run: no editor code changed (only `editor/CHANGELOG.md`). - **Numbers tool:** none. - **umtool e2e** (`SONG_DIR=~/reports/quartering-uh-song/data pnpm --filter umtool run e2e …` from the worktree root; the fixture found song data, `cand2`, `wav48` and `media`, no `asr`, no face detector, so `find.spec`'s 14 skip): | Run | At | Specs | Result | |---|---|---|---| | 1 | `efef56b3` | `storage`, `projects`, `report-longform`, `dashboard` | 37 passed, 6 failed, 10.3 min — `storage.spec` **5/5**; the six (`dashboard` ×3, `projects` ×3) are `page.goto: net::ERR_ABORTED` and 30 s timeouts at a load average of 47, and all six pass in run 2 | | 2 | `efef56b3` | the full suite (21 files) | **214 passed**, 17 failed, 14 skipped, 13.8 min of tests (48 min with 34 min in the queue) — `faces` ×4 (`/api/face/detect` 503: no detector here), `triage` ×9 (no `asr` here), `browse:241`, `mix:166`, `mix:201`, `usage:112` | | 3 | `main` `90bd8384`'s umtool, checked out into the worktree and restored after | `browse`, `faces`, `mix`, `triage`, `usage` | 48 passed, 15 failed, 7.2 min — the same `faces` ×4 and `triage` ×9, plus `browse:15`/`:34` (30 s timeouts) | | 4 | `efef56b3` | the same five | 49 passed, 14 failed, 3.2 min — `faces` ×4 and `triage` ×9 as on `main`; `browse:241`, `mix:166`, `mix:201` pass; `usage:112` fails again | | 5 | `efef56b3` | `usage` | **7 passed**, 0 failed, 19.6 s | So against `main` on this machine: the `faces` and `triage` failures are the machine's (both missing capabilities fail rather than skip — on `main` too); `mix:166`/`:201` and `browse:241` fail only after the whole suite (the corpus window an earlier spec fetched for `vid1` wins the picker's lookup), and pass in isolation on both; `usage:112` ("confirming the drop writes it through") failed twice when it ran right after the failing `triage` specs on this branch, passed once in that position on `main`, and passes alone — the verdict path it drives reads no cache and no `out/`. Left to the reviewer as an order/timing question, not changed. #### Found and left - **Open question 2 — the four `*.mp4` near the project roots** (measured in `~/reports`, depth ≤ 2, outside any `out/`): `kirsche-pippa/latest-contact-2026-06-20.mp4` (7.4 MB) is a cited clip fetched through the MCP's `fetch_clip`, with its `.provenance.json` beside it — the sweep report's evidence, a deliverable of a project that has no `out/`; `quartering-uh-song/jer-metalslug-bg.mp4` (51.5 MB) and `quartering-uh-song/pokemon-no-music-recording.mp4` (3.1 MB) are song-project INPUTS (a song spec's `background.path` names such a file relative to a media root, `song/spec.mjs`); `clips/ tim-pool-…mp4` (23.6 MB) is a loose cut at the reports root, in no project. None is render scratch: all four are left untouched, and none is under U2's `clips/` or `share-*/`. - **What move-out would move today:** `umtool storage move-out --all --dry-run` against `~/reports` (a scratch media root): **10 projects, 10.6 GB** (quartering-diet 5.0 GB, ferret-rescue 1.5 GB, quartering-employee-count 1.2 GB, elfpire-eva 1.1 GB, …) — less than the plan's "≈ 18 of the 20 GB": the rest of `~/reports` is song data and loose files, not project `out/`s. - **Rollout, once:** set `UMTOOL_MEDIA_DIR` (the live umtool's environment) to a directory that exists on the media drive, outside `~/reports`; restart umtool; run `umtool index` (the index is rebuilt under `~/.cache/archilyzer/umtool`; until then everything works, slower); `umtool doctor` shows the roots and the old cache (8.7 MB here), which can then be deleted; `umtool storage move-out --all` (when nothing is building) moves the existing `out/`s. - **A dangling `out` reads as "no build" to the summary readers** (`lib/projects/report.mjs`'s `stat0(out)`, `readAvailability`): only `export` and the writers say "is the media drive mounted?". `umtool storage` and `umtool doctor` name it. `umtool check` learning it is U2's (plan: "`umtool check` learns the two values"). - **For U2:** `lib/report/deliver.mjs` `listBatches` filters `isDirectory()` on the project's dirents, so a `share-*` that is a link would vanish from it; `sharedIdsIn`'s walk likewise does not follow a link. The movers take any one-segment name and return `{ state, src, dest|from, bytes, files }`. - **A CLI move cannot see the app's jobs** (they live in its memory): the verify refuses when the tree keeps changing, but a write in the instant between the verify and the park would be deleted with the parked copy. The CLI says "run when nothing is building"; U2's bench button runs in the app and can check. - **The worktree's stray `transcripts/`** (an `index.mdb` from 2026-09-28) is the trap the rules describe; left in place. #### Deviations from the plan - `ensureOutDir` is not called in `driver.mjs`: its step builders are synchronous, are unit-tested with a fake project directory, and only build argv; the call is in the scripts those steps run (`build-video.mjs`, `check-availability.mjs`) through `ensureWriteDir`, which also covers a hand-run script and the three other writers the plan did not list (`render-cards.mjs`, `compose-chrome.mjs`, `onscreen.mjs`). - `export.mjs` writes nothing under `out/`, so it does not create it; it reports a dangling link instead. - `umtool storage move-back` and the plain `umtool storage []` listing were added beside `move-out`: the e2e needs the way back, and an operator needs to see which projects moved. - `lib/media.ts` (not in the plan) follows `out` links into the media root and skips the root as its own: without it every moved deliverable fell out of the mix picker. `[Unreleased]` (`editor/CHANGELOG.md`): "umtool can keep each report's render folder on a media drive." and "umtool's cache moves to `~/.cache/archilyzer/umtool`". #### Review (SHIP AFTER FIXES) and the fixes | Finding | Fix | |---|---| | F1 — the e2e app and the spec CLIs spread the shell's environment, so a shell exporting `UMTOOL_MEDIA_DIR` would put every fixture build's `out/` on the real media drive | `683e0a0f`: `UMTOOL_MEDIA_DIR=` (empty = unset under `\|\|`) in the webServer command; `UMTOOL_MEDIA_DIR: ""` in the `projects`, `report-longform` and `dashboard` CLI envs (`dashboard`'s doctor also gets the fixture cache); `storage.spec` keeps its own | | L1 — `doctor --json`'s `ok` was the tools' verdict while the exit status also counted the roots | `683e0a0f`: `ok = tools && roots`, `toolsOk` = the tools alone, `roots.ok` kept | | L2 — a CLI move cannot see the app's jobs | `683e0a0f`: `move-out`/`move-back` (one project or `--all`) skip, as `busy` with the pids, any project a running pipeline script (`build-video`, `check-availability`, `render-cards`, `compose-chrome`, `verify-build`, `fetch-via-editor`, `resolve-windows`, `cut-from-cache`, `share-batch`) names on its command line (`/proc/*/cmdline`; none elsewhere). It does not see the app's in-process deck previews; U2's in-app button can ask the app's jobs | | L3 — `dropMediaCopy` deleted any target inside "the media root", which untiered is the reports root | `a6926e17`: deletes only the project's own mirror, only when tiered and only when that is what came home; anything else is left and returned as `mediaCopyLeft` (the CLI prints "left in place … remove it by hand once checked"). A resumed move-back (the link already gone) reports the mirror, never deletes it | | L4 — a move-back cut after its rename orphaned the media copy silently | `a6926e17`: "already" reports the project's mirror as `mediaCopyLeft` while it exists | | L5 — a cut move's leftovers were not a guard | `a6926e17`: while `out.moved-*` or `out.incoming` exists, `ensureOutDir` (absent `out`) and both movers' directory branches refuse, naming the leftover and the move that finishes it; move-out refuses a lone `.incoming`, move-back a parked copy. `683e0a0f`: `folders.md` — the media root is a directory inside the drive, never the mountpoint; the leftovers rule | | N1 — the `paths.mjs` comment named the wrong reason for `READ_ROOTS` | `683e0a0f`: it names `/api/mix/{media,track}` and the lexical write check | | N2 — the walk did not skip `*.moved-*`/`*.incoming` | `683e0a0f`: `skipsDir` does | | N3 — `isMediaLink` realpathed every link it met | `683e0a0f`, `45cf9946`: only an entry named `out` (U2 adds its names to `MEDIA_LINKS`) | | N4 — the changelog bullet | `683e0a0f`: "in umtool's environment (restart umtool after setting it), to a directory inside that drive" | | N5 — an empty mirror for a project that does not exist | `a6926e17`: `ensureOutDir` checks the project directory first | `683e0a0f` left `report-to-video`'s tsc red (the `isMediaLink` parameter type); `45cf9946` restored it. **Re-review (SHIP AFTER FIXES, no further round):** - R1 — a real `out/` beside an `out.moved-*`/`out.incoming` sent each move to the other, which refused again → `4862ec4c`: both movers say both exist, that the leftover holds the moved data, to keep one and remove the other by hand, then run the move; `folders.md` says the same; the leftovers unit case asserts it for both movers. - R2 — `ensureOutDir`'s project check used `lstat`, refusing a project directory that is itself a link → `4862ec4c`: `stat`; a unit case links a project in. - N6 — the walk's leftover skip matched any `*.incoming`/`*.moved-*` folder → `4862ec4c`: only `out`, `clips` or `share-*` followed by one. - Gates: umtool and `report-to-video` tsc clean, all workspaces clean; `storage.test.mjs` 24/24; `test:scripts` 394: 392 passed, 2 skipped (LIVE, and `next-build-trace`'s staleness skip — `storage.mjs` changed after the last build; its path ops gained one `stat`, with `turbopackIgnore`), 0 failed. **Gates after the fixes:** tsc clean at `45cf9946`. `storage.test.mjs` 24/24 (+4: the untiered hand-made link, a tiered foreign link, the leftovers guard both ways, the ghost project; the resumed-move-back case now expects the mirror reported and kept). `test:scripts` **394: 393 passed, 1 skipped (LIVE), 0 failed**, `next-build-trace` passing against a fresh build. Capped umtool build with the corpus linked (76 channels; the fixes touch `storage.mjs`'s path ops): exit 0, 39 s, 0.83 GB. umtool e2e at `45cf9946`: `storage`, `projects`, `report-longform`, `dashboard` — **42 passed**, 1 failed, 2.3 min (after 45 min in the queue; `storage.spec` 5/5) — the one is `dashboard:14`, the run's first test, a 30 s timeout on the cold first page; `dashboard.spec.ts` alone right after: **7 passed**, 0 failed, 42 s. **Still left (follow-ups):** the project summary (`lib/projects/report.mjs`) stats `out/.mp4` and reads `out/availability.json` through the link, so a **stalled** media drive blocks those reads, libuv's threadpool and the project list, and `availability.json` — small hot text — now lives on the media tier; a dangling link still reads as "no build" there (`umtool check` learning it is U2's). The `usage.spec:112` order question (after the `triage` specs, at load) is for a quiet-machine run of `triage.spec.ts usage.spec.ts` at integration. If a song project ever grows an `out/` and is moved, a mix render into it lands on the media root through the link (the write check is lexical; that matches the semantics). ### Slice U2, as shipped — umtool's deliverables switch (2026-10-01) Branch `r17/umtool-deliverables` off `main` `bb877f93` (slice U1 merged), `main` `1d5c33bf` (slice XP) merged in before the gates, worktree `~/Projects/r12-source-mirror` (editor 5101, test 5111, export 5110), one Opus implementer. Scratch files `U2-*` in the job's `tmp`. The ruling is the plan's: deliverables (`clips/`, `share-*/`, final mp4s) move per project by a switch "like channels"; manifests, `revisions/`, the caches and the cue cache stay put; final mp4s travel with `out/` (U1). **What it does.** - **The switch.** `video.manifest.json` gains `"storage": { "deliverables": "local" | "media" }` (absent = local). Its one writer is `updateStorage` in `lib/report/manifest.mjs` (the manifest lock, tmp + rename, the mtime token; any other key under `storage` kept; a no-op is not rewritten, so no open bench page's token goes stale). Its one caller is `moveDeliverables`, after every directory has moved. - **`lib/report/storage.mjs`.** `ensureOutDir`'s body is generalised to a name (`ensureProjectDir`; `ensureOutDir` keeps its behaviour and sentences). New: `DELIVERABLES_MODES`, `isDeliverableName` (`clips` or `share-`, never a move's leftover), `deliverablesModeOf` / `deliverablesMode` (a plain JSON read, so the module stays free of the writer's imports), **`deliverableDir(project, name)`** — an existing directory or link is used as it is; absent and local → a directory; absent and media → `mkdir -p /` + an absolute link, the media root stat'd and never created; refused, creating nothing, on a dangling link, a cut move's leftover (the sentence names `umtool storage deliverables --to …`), a bad value, a project outside the reports root, or `"media"` in a process with no `UMTOOL_MEDIA_DIR` (never silently local: a batch on the wrong drive is a split nobody chose); `deliverableNames` (clips first, every `share-*` dir or link, and a name present only as a leftover so a move finishes it); `deliverablesState`; `deliverablesProblems(state, name)` (the sentences that stop a write: a dangling link, a leftover, and for a directory not there yet a switch this process cannot honour); **`moveDeliverables(project, to, { writeMode })`** — U1's movers over each name, then the switch only when none failed; idempotent; a cut move resumes; `--dry-run` measures. `finishCommand` names the right command in the leftover sentence (`out` keeps U1's). - **The writers.** `cut.mjs` makes `clips/` through `deliverableDir` (a refusal is `reason: "storage"`, before any ffmpeg; tmp and final still share the directory). `buildShareBatch` makes `share-/` through it, and first refuses on `deliverablesProblems` — a batch that cannot read an earlier batch would ship its clips again. A batch name shaped like a leftover (`x.incoming`, `x.moved-…`) is refused by `share-batch.mjs` and the route. - **The readers.** `listBatches` lists a `share-*` link like a directory, a dangling one as `dangling` with no ids, and no leftover; `sharedIdsIn`'s walk follows a link to a directory (six levels at most). `deliverStateOf` carries `storage` (the state, `cutBlocked`, `shareBlocked`). The mix picker (`lib/media.ts`) follows `clips` and `share-*` links into the media root as it does `out`. `kinds.mjs` needed nothing: U1's `SKIP_DIRS`/`SKIP_PREFIXES` already keep the walk out of both. - **`umtool check`** (through `reportDecisions`): `storage-unreachable`, **blocking** — an `out`, `clips` or `share-*` link whose target is gone ("is the media drive mounted?"); `storage-mismatch`, open — a cut move's leftover (with the command that finishes it), a directory while the switch says media, a link while it says local, or media with no `UMTOOL_MEDIA_DIR` in this process; a value that is neither is `manifest-invalid` on `storage.deliverables`, blocking. - **CLI.** `umtool storage deliverables --to media|local [--dry-run] [--json]`: one line per directory and the switch's before → after; exit 1 on any failure (the switch then stays). Refused while a pipeline process works in the project (`lib/report/busy.mjs` after the review, H1): a script argument under the project directory (U1's scan, how the build steps name a project), a `--project` value equal to the project's id or name or resolving to its directory (how the app runs a cut and a share batch; whole values, never a prefix), and the report's own `apply-manifest.py` and `build.py` by working directory. Not seen: a hand-run command that is none of these, another machine. From the bench the move is itself a job, so the app's one-job-at-a-time rule keeps its cuts and batches out as well. `umtool storage []` lists each report's deliverables and switch. - **Bench.** The deliver panel says where the deliverables are (`data-deliverables` = `local` | `media` | `invalid`; one `data-deliverable=""` with `data-deliverable-state` per directory) and gains **"Move deliverables to media|local"** (`data-action="deliver-move"`, `data-move-to`), the `move` action of `/api/report/deliver`, a job (`driver.mjs` `moveDeliverablesSteps` runs the CLI). It is disabled, the reason in `data-move-reason`, while a job of the project runs ("… is running — move when it has finished"), while a deliverable link dangles, and toward media when the app has no `UMTOOL_MEDIA_DIR`. Cut and share are disabled with `deliverablesProblems`' sentences (`data-deliver-blocked`), and the route refuses them with 409. - **e2e.** `deliverables-fixture` (d01/d02 cuttable from a cached window; d03 cut and shipped in `share-first`) and `deliverables.spec.ts` (5, serial): the CLI move (dry run first; links; the switch set; again → already, not rewritten; `check` silent); the app (no media root) cuts THROUGH the `clips/` link while the move button is disabled with "is running"; the app refuses a new batch (button disabled with the reason, route 409, nothing made) and the CLI with the media root makes `share-second` as a link, reading `share-first` through its own (d03 not shipped again); the root unplugged → `check` exits 1 with `storage-unreachable` on all three links, the panel and the route refuse, nothing made, the root not recreated; the bench's button brings everything home (the app, untiered, leaves the media side and says "left in place"), the switch reads local, and the button then offers media, disabled, naming `UMTOOL_MEDIA_DIR`. **Commits** | Commit | What | |---|---| | `a61b60e2` | `umtool:` the switch — `updateStorage`, `deliverableDir` and the deliverables helpers in `storage.mjs`, `cut.mjs`/`buildShareBatch` through it, `listBatches`/`sharedIdsIn` follow links, `umtool check`'s storage decisions, `umtool storage deliverables`, the busy scan's `apply-manifest.py`/`build.py`, the picker; `deliverables.test.mjs` (14) | | `ec7c84f3` | `umtool:` "Move deliverables" on the bench, the route's `move` action and refusals, `deliverables-fixture` + `deliverables.spec.ts`, `docs/folders.md` + `docs/cli.md`, the `[Unreleased]` bullet | | `eb9bb64e` | merge of `main` `1d5c33bf` (slice XP) — clean, nothing under `umtool/` | | `875dbff1` | `umtool:` the spec's batch count includes the id its LIST.md says was already shared (run 1's one failure) | | `f316fd46` | `plans:` this section; FACTS "umtool's deliverables switch" | | `3ff84ad5` | `umtool:` review H1 (the busy scan by `--project`, `lib/report/busy.mjs`) and N1; +4 unit cases | | `f52f1eb1` | `umtool:` review N2 (the route's 409 for a move during a cut) | | `1d176e2a` | `plans:` the review subsection, the corrected busy-scan claim | | `ca30be18` | merge of `main` `fa24a57f` (slice T1, the deck's finale-dip branch) — clean; `driver.mjs` and `make-fixture.mjs` merged without conflict, both sides kept | | this commit | `plans:` the gates after the merge | #### Gates (logs `$T/U2-*`) - **tsc** (all workspaces) clean at `ec7c84f3` and on the merged tree. - **common:** 2,501/2,501 on the merged tree (no common code touched). **Editor unit:** skipped — no editor code touched (only `editor/CHANGELOG.md`). - **test:scripts:** 408 tests: 406 passed, 1 skipped (LIVE), 1 failed — `queue-lock.test.mjs`'s "QUEUE_LOCK_HELD passes straight through" (waited 1,065 ms at a load average of 16–22, the timing case U1 met); `node --test scripts/queue-lock.test.mjs` alone right after: **11/11**. `deliverables.test.mjs` **14/14**, `storage.test.mjs` 24/24, `next-build-trace.test.mjs` **10/10** against the fresh build below. - **The capped umtool build with the corpus linked** (worktree `transcripts/` set aside, `ln -sT`, 77 channels visible, `systemd-run --scope -p MemoryMax=5G -p MemorySwapMax=0`, `timeout -s KILL 240`, the link removed and the directory put back), on the merged tree: **exit 0, 32 s, 0.82 GB**; with `UMTOOL_MEDIA_DIR` set to a scratch directory: **exit 0, 35 s, 0.82 GB**. `.nft.json` entries 39,809 each, **identical** (`diff` empty), none naming `transcripts`, the scratch media root or `.e2e-song`. - **Numbers tool:** none. - **umtool e2e** (`SONG_DIR=~/reports/quartering-uh-song/data`, `UMTOOL_MEDIA_DIR=` in the shell, `pnpm --filter umtool run e2e …` from the worktree root, queued; lists in `$T/U2-specs*.txt`): | Run | At | Specs | Result | |---|---|---|---| | 1 | `eb9bb64e` | `storage`, `deliverables`, `deliver`, `clip-bench`, `report-fetch-via-editor`, `projects` | 91 passed, 1 failed, 2 did not run (serial), 7.7 min of tests (21 min with the queue and the fixture build) — `deliverables:143` expected the batch count `(2)`; the panel said `(3)`, which is right (fixed in `875dbff1`) | | 2 | `875dbff1` | `deliverables` | **5 passed**, 0 failed, 2.1 min of tests (21 min with the queue) | `projects.spec` is the one extra beyond the prompt's list: it reads the decisions inbox, which gained two kinds. #### Found and left - **The app's decisions index is signed without the deliverables** (`reportSignature`: the manifest, `out`, `availability.json`, `revisions`): a drive unmounted under an unchanged manifest leaves a cached inbox row set stale until something else changes. `umtool check` computes afresh. Signing `clips`/`share-*` would stat through the links on every index read — on a stalled drive, the hang U1 left for `out`. Left with U1's note. - **A dangling `clips/` still reads as "nothing cut" to `deliverStateOf`'s counts** (the panel's `need-cut` and per-section numbers); the panel now says why beside them and every write refuses. - **The app moves home without a media root of its own** (as in the e2e): the media copy is left in place and named, as U1's L3 rule says. The real app runs with `UMTOOL_MEDIA_DIR` set, where the project's own mirror is removed. - **U1's two items for this slice** are done: `listBatches`/`sharedIdsIn` follow a `share-*` link; `umtool check` reports a dangling `out/` (and `clips/`, `share-*`). #### Deviations from the plan - The plan's files `report-to-video/{cut,deliver,check}.mjs` are `lib/report/cut.mjs`, `lib/report/deliver.mjs` and `bin/umtool.mjs`'s `check` (through `lib/projects/report.mjs`'s `reportDecisions`, so `umtool check` and the decisions inbox say the same). - The movers' loop and the switch are one function in `storage.mjs` (`moveDeliverables`) taking the writer as `writeMode`, so `storage.mjs` — imported by the app's routes — does not import the manifest writer's dependencies. - The bench move is the deliver route's fifth action and a job, not a route of its own: the job registry is what keeps a cut or a batch from running under it. - `lib/media.ts` (the mix picker) and `driver.mjs` (the step builder) are touched beyond the owned list, one function each. - A FACTS section was added ("umtool's deliverables switch"); no existing fact changed. `[Unreleased]` (`editor/CHANGELOG.md`): "umtool can move a report's cut clips and share batches to the media drive, one project at a time." #### Review (SHIP AFTER FIXES) and the fixes | Finding | Fix | |---|---| | H1 (HIGH) — the busy scan counted a pipeline script only when an argument was the absolute project path, but the app runs `cut-from-cache.mjs` and `share-batch.mjs` as `--project `: a shell-run move did not see the app's cuts and batches, the two writers it exists to wait for; the docs, FACTS, this record and the CLI comment claimed it did | `3ff84ad5`: the scan moves to `lib/report/busy.mjs` (imported by the CLI only) and also counts a script whose `--project` value equals the project's id or name, or resolves against the process's cwd to the project directory — whole values, never a prefix; `move-out`/`move-back` get it too. Tests: `namesProject`'s cases, a dummy process carrying `cut-from-cache.mjs --project ` is seen and `-other` is not, the CLI refuses as `busy` with its pid and moves once only `-other` runs. The claim is corrected in the CLI comments, `docs/folders.md`, `docs/cli.md`, FACTS and the "What it does" bullet above: what it sees, and that a hand-run command that is none of these scripts, and another machine, are not seen | | N1 — a link under an absent key (absent means local) was not flagged, only under an explicit `"local"` | `3ff84ad5`: a `clips`/`share-*` link into the media root under no key is `storage-mismatch` too (a hand-made link elsewhere under no key is not); a unit case runs `umtool check` on no key, `local`, `media` | | N2 — the route's 409 for a move while a job runs was asserted nowhere | `f52f1eb1`: `deliverables.spec`'s cut test posts a `move` during the cut and expects 409 "a job is already running" | | L1 — the decisions inbox and the deliver panel stat through every `out`/`clips`/`share-*` link, so a stalled media drive blocks them (U1's class) | left, a follow-up (below) | **Gates after the fixes:** umtool tsc clean. `deliverables.test.mjs` **18/18** (+4), `storage.test.mjs` 24/24. umtool e2e at `f52f1eb1`, `deliverables` + `storage`: **10 passed**, 0 failed, 1.2 min of tests (10 min with the queue). **Gates after merging `main` `fa24a57f`** (`ca30be18`; log `$T/U2-regate.log`): umtool tsc clean; `deliverables.test.mjs` + `storage.test.mjs` + `driver.test.mjs` **45/45**; the capped umtool build with the corpus linked (77 channels; the worktree's `transcripts/` set aside and restored): **exit 0, 30 s, 0.80 GB**, no trace entry naming `transcripts/` or `.e2e-song`, `next-build-trace` 10/10; umtool e2e `deliverables` + `storage`: **10 passed**, 0 failed, 41 s of tests (6 min with the queue). **Follow-up (L1):** with a stalled (not absent) media drive, the inbox's decisions pass and the deliver panel block on the stat through each link, as U1's project summary does for `out`. ### Slice XP, as shipped — X posts are private (2026-10-01) Branch `r17/x-posts-private` off `main` `90bd8384`, worktree `~/Projects/r13-lows-export` (editor 5501, test 5511, export 5510), one Opus implementer, beside the media-tier slices. Scratch files `XP-*` in the job's `tmp`. The ruling is above ("Slice XP — the ruling"). **The listing side is release 14 slice HS's.** HS shipped `site.json` `listed` (absent = listed), the one predicate `isListedSite`, and every listing that reads it: the homepage summary, `channel-sites.json`, the pooled stats, the hub's `hub-sites.json` (and so its `corpus.json` and `llms.txt`), every footer, and the form's **List on the Archilyzer homepage and hub** checkbox. This slice adds no listing plumbing of its own: `isListedSite` gains one clause — a private site is never listed, whatever `listed` says — and the checkbox stays as it is. What is new is the deploy refusal, the private site's empty `hubUrl` and its `corpus.json` word, and "private content is built only into private sites". **What it does.** - **`social.x.visibility`: `"public"` (default, absent) | `"private"`**, beside `social.x.cookieSource` (`XSocialSettings` and `sanitizeSocial` in `common/social/xCookieSource.ts`, the social block's home; its doc in `settingsSchema.ts`; SETTINGS.md regenerated). On `/settings`, at the foot of the X account session section, **Where X posts appear** (`XPostsVisibilityControl.tsx`): "Public — every site that has the channel" | "Private — private sites only", with: "Private leaves every X channel and its posts out of every public site and builds them only into sites whose audience is private, which are never deployed; nothing is deleted and fetching goes on. Sites already published change on their next build and deploy." Written by `setXPostsVisibilityAction` through `saveSettings`; "public" is written as no key. - **`site.json` `audience`: `"public"` (default, absent) | `"private"`** (`siteSchema.ts`, only `"private"` written, `isPrivateSite` the one predicate; SITE.md regenerated). The site form has an **Audience** select with a sentence; `saveSiteAction` keeps only `"private"`. - **The rule, `common/lib/postsVisibility.ts`** (pure, tested): `postsVisibleTo(site, config, settings)` — an X channel (`sourceKind: "social"`, `platform: "twitter"`) goes only to a private site while the setting is private; every other channel is untouched — and `publishedMemberSlugs`, a site's members narrowed by it. **A public site leaves the X channel out whole**: posts are all a social channel holds, so without them it would be an empty checkbox and a name in `corpus.json`. Narrowed by the same call in `buildIndex`'s **per-site loop** (the summaries, subs, posts and digests manifests; the site fingerprint gains `withheld`, only when non-empty, so flipping the setting or the audience rebuilds the site and nothing else moves) and in `compose-site` (every shared tree it copies, so a tree a public site shipped before is pruned). The shared posts tree and the LMDB posts sub-DB stay corpus-wide. `channel-sites.json` maps a channel only to the sites whose build carries it (`channelSitesOf` takes the narrowing) and the export's `/offline` page lists the same members. - **A private build says so and belongs under no hub**: `corpus.json`'s `site.audience` is `"private"`; `resolveHubUrl` gives a private site no `hubUrl` (absent from its `site.json` and `corpus.json`). - **Never deployed.** `lib/builtExport.ts`: `siteDeployProblem(site)` — `Site "x" is private (audience: private): it is built for reading on this machine and is never deployed. Build it without deploying, or set its audience to public on its Settings tab` — `builtAudienceProblem(outDir)` (a bundle whose `corpus.json` says private, so a site switched back to public cannot ship its private build) and `deployAudienceProblem`, both. Asked first, before any upload, where release 13 W3's `builtBundleProblem` is asked: `runDeployIntoLog` (the last word before wrangler), `runDockerDeployAllPhase` (skipped with the sentence, not failed, so Build & deploy all is not red while a private site exists — the site is still built), `deploySite` (`archilyzer deploy site`, before the R2 upload), the editor's `deployExportAction` and `buildAndDeployAction` (before a job exists; Build & deploy asks again before its upload), the host Build & deploy all fallback (`basicBuildAndDeployAll`, skipped before the upload), and `docker/publish-site.sh` (before building, from `site.json`, and over the built `corpus.json` before publishing). The ops routes `build-deploy` and `deploy-site` answer the actions' sentence (400). **Build** still builds a private site. - **The Search in row's Posts toggle**: no code changed. `hasPostsCorpus` is "the site posts manifest lists a channel", so a public site whose only posts were X posts ships `channels: []` and no Posts box — pinned by the integration test (no `postScheme` either) and the export spec. **Where the rule lives — one deviation.** The prompt named the social-channel branch of `scanSource` (`buildIndex.ts:333-341`). That branch is corpus-wide: it feeds the shared posts tree that every site, and the MCP over a private build, read, so it must keep building X posts. The rule sits in the per-site loop (`buildIndex.ts` ~1943 and the fingerprint), a different hunk from T1's `scanSource` guard. **Found on the way, fixed here.** - **`saveSettings` merges one level deep, so a patch `{ social: { x } }` replaced the whole X block**: choosing a login source would have erased the visibility, and the reverse. Both X actions now write over the block as it is (`socialXPatch`); the e2e case pins both directions. - **compose-site trusted its per-site cache after ANOTHER site's compose.** `public/` is one directory every site composes into in turn (the basic build); the cache is per site, and a stage whose source had not changed was skipped. So composing site B, then site A again with no new data, shipped B's summaries — its whole channel list — as A's: a public site composed after a private one holding every channel would have listed the private site's channels. The site's own stages (summaries, stats, duplicates) are now trusted only when `public/site.json` names the site; `site.json` is cleared at the start of a compose and written at its end, so a compose cut short leaves nothing to trust. The per-channel trees keep their signatures (copies of the shared trees, the same bytes whichever site copied them). Docker builds have a per-site `public/` and were not affected. - **compose never ships a posts tree the site's posts manifest does not list** (the index build's word), so a channel config compose fails to read — read as visible — does not ship X posts. **The MCP and umtool.** - **The MCP needs no change**: it reads a composed export (`mcp/src/sources.ts`, `--local ` | `TRANSCRIPT_LOCAL_DIR`). A private site is composed into a directory of its own, without touching `export/public`, from the checkout root: ```sh pnpm archilyzer index # or any editor build: the index build writes the per-site manifests BUILD_ARCHIVES=0 EXPORT_PUBLIC_DIR="$HOME/archives/" \ EXPORT_INDEX_DIR="$PWD/export/.export-index" pnpm archilyzer compose site claude mcp remove archilyzer -s local # the name must be free; use the scope it was added in claude mcp add archilyzer \ --env ARCHILYZER_EDITOR_URL=http://localhost:3001 \ --env WORKER_TOKEN=… \ -- pnpm --silent -C "$PWD" archilyzer mcp --local "$HOME/archives/" ``` The two `--env` lines are what `fetch_clip` needs (the editor's own `WORKER_TOKEN`, from `editor/.env`); the name stays `archilyzer` for `/ask` and `/sweep` (AGENTS.md). (`EXPORT_PUBLIC_DIR` must be absolute — the command runs in `common/`; the compose cache lands beside it, in `$HOME/archives/.compose-cache/`.) The site's editor **Build** works too: it composes into `export/public` and builds into `export/out`. **The current registration, `--local /export/public`, reads whatever site was composed there last**: after a public site's build it has no X posts, after the private site's it has them. - **umtool is unaffected**: the report pipeline's posts (the deck's posts room) are carried whole in the report manifest — `posts[]` with `platform`, `date`, `text`, `url` (`validatePosts`, `umtool/report-to-video/deck.mjs`), "added by editing the manifest" — and its cues come from the local corpus or the archive `provenance.siteOrigin` names (videos only). It reads no public build's posts. A sweep that looks posts up for a manifest goes through the MCP, pointed at the private build as above. **Commits** | Commit | What | |---|---| | `3ea76853` | `common:` `postsVisibility.ts` + test; `social.x.visibility`; `site.json` `audience` (`isListedSite`, `resolveHubUrl`); the per-site loop and compose narrowed; `corpus.json` `audience`; the deploy refusals (`builtExport`, `publish/build`) + tests; `channel-sites.json`; `/offline`; `publish-site.sh`; SETTINGS.md, SITE.md; the compose integration test | | `0195d52a` | `editor:` Where X posts appear; the X block written whole; the Audience select; the deploy actions refuse a private site | | `75ccbef3` | `editor(e2e), export(e2e):` `x-session` and `sites-crud` cases; `export/e2e/x-posts-private.spec.ts` | | `30c14aa0` | `common:` compose never ships a posts tree the index withheld; the cache trusted only over the site's own last compose | | `63002de3` | `common:` the per-channel trees keep their signatures across sites | | `4ed7d410` | `plans:` this section, the ruling, the slices row; FACTS; the editor changelog | | `6466a68e` | `common:` the hub carries no site's data; `builtHubProblem` refuses one that does (review HIGH 1) | | `385e4eb1` | `common:` a compose over a stale index lists no withheld channel; the comments (LOW 3, NIT 7) | | `8e448fde` | `editor:` the changelog (LOW 5) | | `da5c2912` | `plans:` the review, its record, the rollout steps and the gates after it | | `837d630a` | merge of `main` `bb877f93` (slice U1: umtool, `.gitignore`, and the two shared records — both sides kept, U1's section before this one). Re-gated on the merged tree: tsc clean; common **2,501/2,501**; export unit 98/98; homepage unit 23/23; editor unit 109/109; test:scripts 392 passed, 0 failed, 2 skipped (394). The merge touched no file the editor, export or hub e2e lists cover (umtool only), so they were not re-run | | this commit | `plans:` the merge in this table | #### Gates (logs `$T/XP-*.log`) - **tsc** (all workspaces) clean before every commit; last at `63002de3`'s tree (`XP-tsc5.log`). - **common:** **2,499/2,499** at `63002de3`, 161 s (`main`'s count + the new `postsVisibility.test.ts` 7, `compose-site.postsVisibility.test.ts` 4, `build.test.ts` +3, `poolSummary.test.ts` +1); 2,498 at `75ccbef3`. `archilyzer docs files --check` and `settings example --check` exit 0. **Editor unit:** 109/109. **Export unit:** 98/98. **Homepage unit:** 23/23. **mcp:** 271/271 (no change there). - **test:scripts:** 367 passed, 1 failed, 2 skipped of 370 (`XP-scripts2.log`; the first run had 2 failed). The failures are `scripts/queue-lock.test.mjs`'s "prints a banner naming the holder while waiting" (and once "serves waiters in arrival order"): timing cases (200 ms and 150 ms staggers) run at a load average of 23–30 while other suites held the e2e queue; alone, the banner case still fails under that load (10/11). This slice does not touch `scripts/` (`git diff 90bd8384 -- scripts/` is empty). - **Builds** at `75ccbef3` (the later commits touch only `compose-site`, a bin no Next app bundles): the capped editor build with the corpus linked (`ln -sT`, `systemd-run --scope -p MemoryMax=6G`, the link removed after) exit 0, 172 s; `pnpm --filter export exec next build` exit 0, 83 s (the `export/public` links made, 0 dangling); `pnpm --filter homepage run build:nodata` exit 0, 47 s. - **Numbers tool:** none. **Privacy gate:** `git diff main --name-only | xargs grep -lc …` names one file, `plans/FACTS.md`, whose 3 matches are all on `main` already; 0 in this slice's added lines. | Run | At | Specs | Result | |---|---|---|---| | 1 (editor) | `75ccbef3` | `sites-crud`, `settings`, `x-session`, `forms-keep-input`, `deploy-page`, `site-publish-preview`, `sites-homepage` | **62 passed**, 0 failed, 4.0 min (after 35 min in the queue) | | 2 (export) | `63002de3` | `x-posts-private` (new, 2), `search-in`, `posts-search` | **20 passed**, 0 failed, 3.0 min (after 7 min in the queue) | New cases: `x-session` "where X posts appear persists, beside the login source and without it" (the whole-block write both ways); `sites-crud` "a private site saves its audience, and every deploy refuses it before any job" (the form, the file, `build-deploy` and `deploy-site` answering the sentence with 400, back to public with the key gone); export `x-posts-private` — the posts manifest a public site built with X private ships (no channel: no Posts box, no X post for "kappa") and the one a private site ships (the Posts box, both X posts). **The export suite's data is route-mocked, never built**, so the build rule itself is proved by `common/bin/compose-site.postsVisibility.test.ts` through the real `buildIndex` and compose: a public and a private site over one video, one X and one Bluesky channel; the flip back; a public site whose only posts were X posts (empty manifest, no `postScheme`); a config compose cannot read; a public site composed after the private one. #### Found and left - **An X channel's manifest-only transcripts tree** (pageCount 0) would still be copied into a public site when compose fails to read the channel's config: a directory named for the channel, holding no content. The posts tree, the posts manifest, the channel list and `corpus.json` follow the index build and do not carry it. - The `/sites` list does not mark a private site; its form and every deploy refusal do. - **The posts reconcile ships only the channels the site's posts manifest lists, on every build**: a social channel with 0 posts no longer gets a posts folder. Nothing advertised that folder (no posts manifest entry, no `corpus.json` posts link), so nothing reads the difference. - **A private site changes the homepage's and the hub's totals.** A private site is unlisted, and `channelsOnlyOnUnlistedSites` keeps a channel only unlisted sites expose out of every public total. A private site holding every channel turns every pool-only channel (on no public site) into "only on unlisted sites", so the homepage's and the hub's instance-wide totals drop by those channels at the next homepage and hub build. Channels a public site also has are unaffected. - **Cloudflare Pages keeps old builds reachable.** A production redeploy replaces what the production URL serves, nothing else: every earlier deployment stays live at its own `..pages.dev`, and a preview alias (`..pages.dev`) keeps its last build. The review found `tags-exclude.anilyzer.pages.dev/posts/manifest.json` still listing an X channel. So after the rollout, X posts are gone from the production URLs only; whether to delete the old deployments is the operator's call (rollout step 5). - `queue-lock.test.mjs`'s two timing cases failed under the machine's load (below); not this slice's file. #### Decisions the operator could overturn | What I did | The alternative | |---|---| | A public site leaves an X channel out WHOLE (posts, posts manifest, channel list, `site.json`, `corpus.json`, `channel-sites.json`) | Keep the channel listed with no posts: an empty checkbox and a name in `corpus.json` | | Build & deploy all builds a private site and SKIPS its deploy, with the sentence | Fail its deploy: the run goes red every time while a private site exists | | A bundle whose `corpus.json` says private is refused even when the site is public now | Trust the site's current audience only (a stale private build could ship) | | `docker/publish-site.sh` refuses a private site (the `site` service is the host's public face) | Let it publish locally behind Caddy | | `social.x.visibility` lives in `xCookieSource.ts` with the rest of the social block | A module of its own | | The hub's `search-aliases.json` is the global dictionary (it was whichever site composed last) | Ship none: hub-wide search in a reader (`reader-hub.ts`) would have no aliases | #### Rollout for this slice (the parent's, through the editor's own writers) 1. Rebuild and restart the editor (the setting, the Audience field, the deploy refusals, the compose and hub fixes). 2. Create the private site (Sites → New, **Audience: Private**, every channel), then set **Where X posts appear: Private** on `/settings`. 3. Rebuild and deploy every public site that has an X channel (each by name; Build & deploy all skips the private site's deploy and says why). 4. **Rebuild and deploy the hub** — the live hub serves the last-built site's data, X posts included, until it is rebuilt from this branch (the review's HIGH 1). Then the homepage, for the totals. 5. **Old deployments** (the operator decides): every earlier production deployment and every preview alias of a site that had X posts still serves them. To remove them: the Cloudflare dashboard → Workers & Pages → the project → Deployments → a deployment's ⋯ menu → Delete deployment (a preview alias's branch deployments are listed there too); or `pnpm dlx wrangler pages deployment list --project-name ` then `pnpm dlx wrangler pages deployment delete --project-name ` (check `--help` for the force flag an aliased deployment needs). The current production deployment cannot be deleted, and needs none. 6. Build the private site (its **Build**, or the compose-to-a-directory commands above) and point the MCP at it. #### Review **Verdict: SHIP AFTER FIXES** (`XP-review.md` in the job's scratch): two Highs, four Lows, three nits. | Finding | Where | |---|---| | HIGH 1: the hub bundle carried the last-composed site's data trees (the live hub serves jeralyzer's posts manifest, two X channels), through no gate | `6466a68e`: `compose-hub` removes every per-site entry from `public/` first (`SITE_ONLY_PUBLIC_ENTRIES`: summaries, transcripts, subs, posts, digests, stats, archives, `site.json`, `tags.json`, `duplicates.json`, `search-aliases.json`, `chart-templates.json`, `sitemap.xml`; a worktree link by the link only) and writes the global alias dictionary as the hub's; `builtHubProblem` refuses a hub bundle that still carries a data tree, so Deploy hub refuses one. Tests: `compose-hub.test.ts` +1, `builtExport.test.ts` extended. Rollout step 4 | | HIGH 2: Pages preview aliases and old deployments keep serving X posts | Record: "Found and left" and rollout step 5 (the operator decides; the dashboard and wrangler paths) | | LOW 3: a compose over an index built before the flip listed the X channel's name and count | `385e4eb1`: the served posts manifest and summaries manifest are narrowed to the published members (`site.json`, `corpus.json` follow); a narrowed summaries copy is not trusted by the next compose. Test: "a compose over an index built before the setting flipped lists no X channel anywhere" | | LOW 4: the MCP line lost `fetch_clip`'s env, and `add` fails over a registered name | This commit: `claude mcp remove archilyzer -s local` first, the two `--env` lines kept (`WORKER_TOKEN=…`) | | LOW 5: the changelog missed the compose-cache fix and the restart notes | `8e448fde`: a bullet for the compose-cache fix, one for the hub, and the restart/redeploy notes | | LOW 6: a private site holding every channel moves the homepage's and hub's totals | Record: "Found and left" | | NIT 7: the deploy-all comment said the bundle check runs first | `385e4eb1`: the comment says the audience check runs first and what that means for a private wrong-site bundle | | NIT 8: the posts reconcile drops a 0-post social channel's folder | Record: "Found and left" | | NIT 9: `/sites` does not mark a private site | Already listed as left | #### Gates after the review - **tsc** (all workspaces) clean at `385e4eb1`'s tree (`XP-tsc6.log`). - **common:** 2,500 passed, 1 failed of 2,501 (`XP-common4.log`; +2 since the first gates: the hub clearing and the stale-index compose). The one failure is `relocateChannelMedia.test.ts`'s "reconcile: an extra and a changed file on the destination are settled" (its diff listing counted `./` as changed — a directory mtime); 3/3 alone, and the file is not this slice's. **Export unit:** 98/98. **Homepage unit:** 23/23. **Editor unit:** 109/109. - **No rebuild:** the fixes touch two bins (`compose-hub`, `compose-site`), `builtExport.ts` and comments in `publish/build.ts`; no Next app bundles a changed module beyond `builtExport` (the editor's hub action reads `builtHubProblem`, a pure function, tsc-checked). | Run | At | Specs | Result | |---|---|---|---| | 3 (editor) | `8e448fde` | the run-1 list | **62 passed**, 0 failed, 3.0 min (after 45 min in the queue) | | 4 (export) | `8e448fde` | the run-2 list | **20 passed**, 0 failed, 1.1 min (after 50 min in the queue) | | 5 (hub) | `8e448fde` | `e2e:hub`, the whole suite | **39 passed**, 0 failed, 1.6 min (after 12 min in the queue) | The hub suite runs `next dev` in hub mode over `export/public` and never composes, so it shows the hub app is unchanged; the clearing itself is pinned by `compose-hub.test.ts`. ### Slice D0, as shipped — the dashboard answers while a snapshot regenerates (2026-10-01) Branch `r17/dashboard-answers` off `main` `7f4901f1`, worktree `~/Projects/r13-lows-editor` (editor 5401, test 5411, export 5410 — `pnpm wt list`'s block #24), one Opus implementer. Scratch files `D0-*` in the job's `tmp`. The plan is Step 0b above. **What was found before building** (the corpus only read; one video dir's text copied to scratch for the profile). - **The walk did yield — on every file read.** Each video's unit awaits its reads, so the loop turned between them; what the walk does ON the loop is parse. A CPU profile of `generateChannelSnapshot` over 300 copies of one omnibased video dir's text (metadata 0.62 MB, cues 0.62 MB, two 2.9 MB VTTs): `readNormalizedTranscript` (the cues parse, `readTranscriptCoverage`) 1,037 ms self, `readWebpageUrl` 791 ms self, everything else in the walk under 70 ms. `readWebpageUrl` is the reconcile pass at the walk's start (`reconcileVideoDirs.ts`): it reads and parses EVERY `metadata.info.json` for its `webpage_url`, one at a time, on every regeneration. - **One walk alone does not starve the pages.** Built and served by `next start` (loopback, a scratch corpus of one 2,000-video channel, load average 24), with this slice: the regeneration took 26 s, and `/` answered in 0.28–1.6 s and `/jobs` in 0.18–0.81 s, polled every 2 s throughout. Under `next dev` at a load average of 32–35 the same walk took 118 s and the answers ran 0.3–28 s; with `main`'s five files swapped back, 2.0–12.5 s. On this machine, under the dev server, one regeneration does not separate the two, and an isolated tsx micro-benchmark (800 such dirs, three interleaved pairs, load 25–35) put the chunked walk and `main`'s inside each other's noise (loop delay max 300–860 ms either way). - **So the hour-long outage needed more than one walk**, and the live metas show the rest: two walks side by side (the empty queue key), the omnibased one on the platter through `onDrive` (four slots per location, so every page read of that drive queued behind the walk's units), two regenerations of omnibased four seconds apart in the restarted process (`01M3WHY8…`, `01M3WHYC…` — two passes racing between the registry check and the record's registration), and the auto-queue status poll recomputing behind all of it. D0 closes the three it owns: one queue, a per-slug dedup that covers the enqueue in flight, and the status memo. The platter half is T1's (text reads leave `onDrive`). - **The ghosts never held a move.** `channelWriters` reads this process's registry only, never a meta, and the registry is in memory: a meta a dead process left `running` was never a writer. `/jobs` listed such a job as `archived` (`listJobs`: a non-terminal meta not in the registry). Harmless to a move, misleading on `/jobs`; the boot pass now closes them. - **What a SIGTERM'd process leaves behind today** (`shutdownCancel.ts`, "graceful shutdown only"): the reaper cancels every live job and re-raises the signal at once, without waiting. A queued job keeps `queued` on disk on purpose (the boot pass settles it). A running managed job's terminal meta is written only when its function returns (`streamCommand.ts`, the `.finally`); `generateChannelSnapshot` takes no signal, so a regeneration never returns before the exit, and its meta stays `running`. A SIGKILL leaves the same, and orphans children. **What it does.** - **The walk yields between chunks.** `generateChannelSnapshot` maps its video dirs through `mapInYieldingChunks` — chunks of `SNAPSHOT_YIELD_EVERY` (32: two full waves of the 16-wide limit), one `setImmediate` between chunks. The unit body is untouched: the diff in that file is the helper, the constant and the call's two lines, so T1's merge is trivial. - **One regeneration at a time, once per channel.** `snapshotScheduler.ts` gains `REFRESH_REPORT_QUEUE` (`"refresh-report"`: a non-empty key is the registry's existing concurrency-1 serialization, so no new mechanism) and `startRefreshReport(paths, slug)`, the one entry point: a slug with a `refresh-report` queued, running or being enqueued (a `starting` set on the scheduler's global state, held across `runManagedFunction`'s awaits) is answered `info` with `REFRESH_REPORT_ACTIVE`. The debounced pass and **Update all reports** (`refreshAllChannelSnapshotsAction`, `editor/app/channels/actions.ts` — not in the slice's file list and owned by no other slice) both go through it, so they dedup against each other; the action still reports a skip as "already running". The job body is the scheduler's (it bumps `generation`, which the action's copy did not). - **The auto-queue status poll shares one fold.** `common/views/autoQueueStatus.ts` gains `singleFlightMemo` (concurrent callers share the computation in flight; a landed value is reused for `AUTO_QUEUE_STATUS_MEMO_MS` = 3 s from when it LANDED; a rejection is not memoized; `clear()` detaches an in-flight computation) and `autoQueueStatusMemo()` on `globalThis`. `buildAutoQueueStatusPayload` is unchanged. The shell (`editor/app/operations/status.ts`) memoizes the snapshot-derived half only — the channel briefs and the four lanes' `computeLeafPending` — and reads the priority view, the state document, settings, the pool and the runners fresh. The e2e reset route drops the memo with the other singletons. - **Boot closes the running metas a dead process left.** A meta now records `pid` (`jobMeta.ts`, owned by no slice). `bootQueuedJobs.ts` gains `settleRunningJobMetas`, run from `instrumentation.ts` on every boot (idle and test server too: it re-queues nothing, and it does not wait for the storage pass): a `running` meta from before this boot, not in the registry, whose writer is gone is closed `cancelled` with `cancelReason` "interrupted: the process running it stopped before it finished", `endedAt` = its log's mtime (the last moment it is known to have run), else now. **The status:** `cancelled` + `cancelReason` already is this file's terminal state for "the server went down under it", and every reader (listJobs, `/jobs`, the job page's "Cancelled because", Retry) handles it — so there is no new `interrupted` status. **The dead-process test** (`writerIsGone`): metas carried no pid or host, and "predates this boot" alone is not reliable here, because `archilyzer run` (`bin/run-operation.ts`) runs jobs offline into the same `.jobs/`. Gone = no pid (written before this release), or this process's own pid (a container's editor restarts as the same pid; this process's own metas are already excluded by `bootedAt` and `isLive`), or `kill(pid, 0)` → ESRCH (EPERM counts as alive). A live pid is left alone. The queued pass now applies the same check. `channelWriters.ts` gains a comment saying why a ghost cannot reach it, and a test pins it. - **e2e** `dashboard-answers.spec.ts` (two cases) and `/api/test/settle-running-metas` (POST, guarded by `E2E_TEST_ROUTES`), which runs the running pass on demand: the e2e server boots once per suite. (a) Two 600-video channels (one hardlinked ~400 KB `metadata.info.json` per dir, a Rumble archive so each file is parsed twice) regenerated by Update all reports through the ops API while `/` and `/jobs` are polled every 2 s: both reports land, the second job started when the first had ended (read from their records), and every answer is inside the budget. (b) A `running` refresh-report meta with a dead pid is closed as interrupted, one with a live pid (the test runner's) is left `running`, the job page says "interrupted", and the channel's Move media completes. **Commits** | Commit | What | |---|---| | `8ce3b24d` | `common:` the chunked walk; `REFRESH_REPORT_QUEUE`, `startRefreshReport`, the `starting` set; `snapshotScheduler.test.ts` (2); Update all reports through it | | `56efcd74` | `common, editor:` `singleFlightMemo` + `autoQueueStatusMemo` + 5 tests; the shell memoizes the snapshot half; the reset route drops it | | `36babcad` | `common, editor:` meta `pid`; `writerIsGone`, `processIsAlive`, `settleRunningJobMetas`; the queued pass's writer check; boot wiring; 4 boot tests + 1 `channelWriters` test | | `fef9ebc2` | `editor(e2e):` `dashboard-answers.spec.ts`; `/api/test/settle-running-metas` | | `fe1676c3` | `editor(e2e):` the spec reworked after run 1 (two 600-video channels, the serial check, the idle-floored budget) | | `3c90b8e9` | `plans:` this section; the editor changelog | #### Gates (logs `$T/D0-*.log`) - **tsc** (all workspaces) clean at every commit. - **common:** **2,496/2,496** — new: `snapshotScheduler.test.ts` 2, `autoQueueStatus.test.ts` +5, `bootQueuedJobs.test.ts` +4, `channelWriters.test.ts` +1. **Editor unit:** 109/109. **test:scripts:** run 1: 301 passed, 1 failed, 2 skipped (304); rerun: 300 passed, 2 failed, 2 skipped. Every failure is in `queue-lock.test.mjs` ("prints a banner naming the holder while waiting", "serves waiters in arrival order (FIFO)"), timing cases run at load averages of 16–30; the file run alone passes 11/11, and the slice touches nothing under `scripts/`. - **Build:** the capped editor build with the corpus linked (`ln -sT`, `systemd-run --scope -p MemoryMax=6G`, the link removed after): exit 0, 123 s, at `fef9ebc2`. - **e2e** (`jobs`, `channels`, `channel-storage`, `dashboard-answers`; `next dev`): - run 1 at `fef9ebc2`: **23 passed, 2 failed, 16.7 min**. `dashboard-answers` (a) timed out at 5 min: 2,000 videos under `next dev` at a load average of ~30 did not finish (the scratch-server measurement above: 118 s for the walk alone, every answer slower). `channel-storage` "relocate a channel's media to another root, and move it back" — the run's first test, on a cold dev server — timed out at 90 s on its first request to the file route. - run 2 at `fe1676c3`: **25 passed, 0 failed, 3.7 min** (34 min with the queue wait). (a) took 23.8 s: nine polls of each page while the two regenerations ran, `/` 0.14–1.9 s and `/jobs` 0.34–1.6 s (budget 5 s); (b) 7.8 s. The relocate case passed — run 1's failure was the cold first test. - **Numbers tool:** none. **Deviations from the plan, one sentence each.** - The chunk is 32, not 25, so every wave of the 16-wide limit is full (25 left each chunk's second wave nine wide). - The memo covers the shell's snapshot-derived half, not the whole payload: memoizing it all would hold a lane's hold, runner and picks up to 3 s behind a click, and specs read them straight after one. - The running-meta pass is in `bootQueuedJobs.ts` beside the queued one, not in `registry.ts`, which is in memory and holds no metas. - "Interrupted" is `cancelled` + `cancelReason` (the existing terminal state for a job the server went down under), not a new status. - `channelWriters.ts` gets a comment and a test, no code: a ghost cannot reach it. - The e2e budget is 5 s or three times the slowest idle answer measured just before, whichever is longer: the dev server on this shared machine at load 30 takes 2–4 s a page with nothing regenerating, and the spec pins starvation, not the machine's speed. - The spec regenerates two 600-video channels, not one large one: 2,000 did not finish in five minutes under the dev server here, and two prove the serial queue from the jobs' records. **Found and left.** - The reconcile pass parses every `metadata.info.json` in full, on every regeneration, for one field — the largest single cost of a walk. Reading only the head, or skipping a dir whose name is already canonical, is a follow-up (`reconcileVideoDirs.ts` is not D0's). - `generateChannelSnapshot` takes no abort signal, so a Cancel or a graceful shutdown cannot stop a walk; the chunk boundary is now the natural place to check one — a follow-up. - A meta whose pid has since been reused by an unrelated live process stays `running` on `/jobs` until that process exits; a start-time check (`/proc//stat`) would close it. Not worth it today. - `editor/app/jobs/[id]/page.tsx` and `JobRow.tsx` describe the cancel reason as the one "a restart left queued"; it now also covers a dead process's running job. Comments only, left. **Changelog** (`editor/CHANGELOG.md` `[Unreleased]`): three bullets — the dashboard and `/jobs` answer while reports regenerate (one at a time, deduped; Update all reports takes the sum), the operations pages share one pending count (up to 3 s old; holds, runners and picks fresh), and jobs a stopped editor left "running" are closed at start. **Rollout note.** No `archilyzer run` may be in flight across the first editor restart after this ships: every meta written before it has no `pid`, so the boot pass reads its writer as gone and would close a still-running offline job's meta as interrupted (the live job rewrites it at its next meta write, but `/jobs` offers Retry meanwhile). The same holds across pid namespaces — an editor in a container judging a pid a host-side `archilyzer run` wrote, or the reverse, sees ESRCH. The release's rollout stops the editor for the migration anyway. #### Review (SHIP AFTER FIXES) and the fixes | Finding | Fix | |---|---| | H1 — the memo served pending counts folded from settings (lane policy and tree, `channelPriority`) up to 3 s stale against a fresh tree; the payload-reading specs were not run | `5b236f8c`: `singleFlightMemo.get(key, compute)`; the shell keys it on `JSON.stringify([settings.autoQueue, settings.channelPriority])`, time the only other expiry; only the latest-started computation stores (an old key landing late never overwrites); comments say what can be one poll late (a rewritten snapshot, the runner's in-flight set) and what is fresh; +1 test. The status-reading specs ran in the full suite below | | L2 — a change during a RUNNING regeneration was dropped until the next change (as on main) | `616a9ce2`: dedup only against a queued or starting regeneration (`isRefreshReportPending`), so a running walk gets one queued successor and no second; fire()'s trailing-edge comment is now true; +1 test | | L3 — the per-channel Refresh report walked in the request, outside the queue | `5509ccf4`: `refreshChannelSnapshotAction` (Refresh report, ops `{slug}`, e2e `generateReport`) starts a job through `startRefreshReport` and awaits it — or polls the one already queued for the channel — and returns the walk's error sentence (`onError`) as its `{error}`; the ops route's comment follows | | L4 — a queued refresh-report holds the Storage panel's Move (`mediaBusy.ts` counts queued jobs) | Not changed in D0, as ruled: T1's `mediaOnly` (refresh-report does not need media) clears it when T2's courtesy check passes it; to confirm at the T1/T2 merges | | L3 follow-on — `generateReport` (85 specs) now queues a job, which `/jobs` may draw before its default filter hides it | `7ba8cd41`: `bulk-actions.spec` "queues no job" leaves `refresh-report` rows out; `dashboard-answers` compiles the ops route (a 400 on `{}`) before measuring — its first compile held `/` 15 s once under load | | L5 — nothing pinned the yield; the e2e floor could grow without bound | `fea5b203`: `snapshotYield.test.ts` — a `setImmediate` probe queued during the first chunk runs before the second chunk's first unit (it fails with the yield removed, checked); 2 tests. The e2e budget is `min(max(5 s, 3 × slowest idle), 15 s)`, and the spec header says it pins the serial queue and gross starvation, not the yield | | L6 — the first boot after this ships closes a live pre-slice `archilyzer run`'s meta | The rollout note above | | N7 — `REFRESH_REPORT_QUEUE` beside the other queue keys | `616a9ce2`: defined in `lib/queueKeys.ts`, re-exported by the scheduler | | N8 — `operations/[id]/page.tsx`'s census comment claimed the payload's listing | `5b236f8c`: says it is the same listing only on a memo miss | | N9 — `requestCache.ts` said "no exception" | `5b236f8c`: one paragraph naming the memo and its bound | | N10 — `channelWriters.test.ts` conflicts with T1 (both append) | Keep both, at the merge | | the record and changelog | this commit: this subsection, the rollout note; the changelog's first two bullets say Refresh report waits its turn, a change during a regeneration queues one more, and a rules/focus/priority edit recounts at once | **Gates after the fixes.** tsc clean at every commit. **common 2,500/2,500** (+4: memo key 1, the successor 1, the yield 2). **Editor unit 109/109.** **e2e** — L3 makes every `generateReport` (85 specs) a queued job, so the whole editor suite rather than the two lists (it contains both), in three runs on a machine that ran out of memory (15 GB used, swap 19/19 GB; a parakeet transcription of the live editor beside several suites): - run 3, the full suite (702 tests) at `fea5b203`: stopped at 374 passed, 8 failed, ~50 min, when pages began to crash (`page.goto: Page crashed`). Two failures were D0's and are fixed in `7ba8cd41` (`bulk-actions` "queues no job"; `dashboard-answers` (a): one `/` of 14.9 s at the moment the ops route compiled, every other answer ≤ 4.6 s). The other six passed in run 4. - run 4 at `7ba8cd41` (run 3's failures plus every spec file from `new-channel-onboarding` on, and both lists — 69 files): **295 passed, 154 failed, 35.1 min** (54 with the queue). Every spec in both lists passed — `jobs` 2, `channels` 8, `channel-storage` 13, `dashboard-answers` 2, `focus-banner` 3, `auto-queue` 24, `lane-runner` 5, `channel-priority` 9, `operation-settings` 7, `backfill` 20, `channel-work` 11, `ops-api` 23 — except `view-route`, which ran after a machine-wide OOM kill at 22:41 took the test server (the kernel log names it, among browser tabs, the desktop session and the live editor's transcriber): every test from the 300th on failed in under 2 s against a dead server. Before it, four failures in `site-scope` (2), `sites-crud` and `social-channel` (the known 22–26 s fetch-posts case). - run 5 at `7ba8cd41`, the 25 spec files with a failure in run 4 (`view-route` included): **183 passed, 0 failed, 12.2 min** (13 with the queue). So every editor spec has passed with the fixes in, across runs 3–5, and both lists in full. **test:scripts** was not re-run: the fixes touch nothing under `scripts/`. #### Re-review (SHIP AFTER FIXES) and the fixes | Finding | Fix | |---|---| | R1 — Refresh report and both ops forms waited for the whole serial queue, with no bound and no feedback (the CLI's fetch gives up at 300 s; the button dropped the action's result) | `5287e8c2`: `requestRefreshReport` (start, or find the queued job) and `waitForRefreshReport` (up to `REFRESH_REPORT_WAIT_MS` = 15 s → done, failed, or waiting with the regenerations ahead), `refreshReportWaitNotice`; `e491ec14`: the action returns `{ notice }` ("Queued behind N report regenerations — the report updates when it finishes (job …).") past the bound, `RefreshSnapshotButton` draws an error and the notice, `InlineActionButton` shows the notice neutrally, and Update all reports answers once queued; `7ae184bd`: ops `{ slug }` returns `{ ok, jobId, started }` (404 for an unknown channel) and `{ all }` returns the ids at once, as `_lib.ts` rules — `--wait` follows them | | R2 — the "already queued" branch reported success when that job failed, or when the slug was mid-enqueue | `5287e8c2`: the wait reads any job's end — a failure returns the job log's `[error]` sentence, whoever started it; a slug mid-enqueue is waited for (2 s) until its id exists; +2 tests | | R3 — `started.stream` left open | `5287e8c2`: `requestRefreshReport` cancels every stream it starts | | R4 — run 4's count | this commit: 154 failed, not 68 (the log's tally) | | the changelog | this commit: bullet 1 says what a long wait shows | **Gates after the re-review fixes.** tsc clean at every commit; **common 2,502/2,502** (+2); **editor unit 109/109**; e2e `dashboard-answers`, `channels`, `ops-api`, `channel-work` (only these, as asked): run 6 at `7ae184bd`: **44 passed, 0 failed, 4.7 min** (dashboard-answers (a): 30 polls of each page while the two regenerations ran, every answer ≤ 3.1 s). **Merge of `main` `1d5c33bf`** (U1, export 0.11.1, XP) at `52446024`: two conflicts, both appends — this file keeps U1's and XP's record sections before D0's under `## Record`, and `editor/CHANGELOG.md` keeps every `[Unreleased]` bullet (main's, then D0's). Re-gated on the merged tree: tsc clean; **common 2,519/2,519**; **editor unit 109/109**; e2e `dashboard-answers`, `channels`, `jobs`, `channel-storage` (run 7): **25 passed, 0 failed, 6.4 min** (23 with the queue wait; every dashboard answer ≤ 3.7 s). ### Slice T1, as shipped — the classifier, the media link and the two guards (2026-10-01) Branch `r17/media-tier-model` off `main` `7f4901f1`, worktree `~/Projects/plans-export-header-first-search` (editor 4201, test 4211, export 4210 — `pnpm wt list`'s block #12), one Opus implementer, beside D0 and U1. Scratch files `T1-*` in the job's `tmp`. The plan is "The model (A′)" §§1–4 above plus the `channelWriters` option and the umtool twin; the mover and every editor surface are T2's. **What it does.** - **The classifier** (`common/lib/mediaTier.ts`, pure): `classifyEntry(name)` is `media` / `text` / `scratch` by name over `mediaFiles.ts`'s anchored predicates; `isTierable` is narrower (`audio.` and `transcript.live_chat.json` — never `source-media.*`, never a partial); `classifyVideoDir`. The table test pins 32 names, the four the slice table names among them. - **The hook and the deleter** (`common/lib/mediaTier-server.ts`): `tierMediaFile` moves a finished big file into `channels//media//` and leaves the relative link `../../media//` — `tiered | left | already`; never throws; a classic channel, a dangling or stalled `media` link, a full disk leave the file real; the per-video `mkdir` is non-recursive; EXDEV copies atomically; the link replaces the name by a rename (no window with nothing there); a failed same-disk move is undone. `tierVideoDir`, `tierChannelMedia({ since, createMediaDir })`, `removeMediaFile` (derefs only inside the channel's own `media//`), `removeVideoDirMedia`, `channelMediaLink`, `relocatedMediaDir`, `tierLinkTarget`. - **Every media finalisation calls it:** `transcodeAudio` after its rename (so the app extraction, the audio-checked download and the video page's Transcode), `downloadOneManaged` before both download-outcome writes, `runYtdlp`'s five media-writing modes after the child returns (`since` the run's start, in a `finally`), `normalizeLiveChat` after it writes the cues. **Every deleter derefs:** the three cleanup sweeps, `purgeSupersededAutoSubs`, `backfillReacquire`, the app extraction's source discard, the audio-checked source discard, `fixIncompleteTranscript` (both), and the video page's file delete, bulk audio removal and directory delete (`removeVideoDirMedia` first). Grep gate `git grep -n "remove(path.join(.*videoDir\|rm(path.join(videoDir" common editor`: **0 hits** (the module's own `rm`s take a joined path). - **`config.json`:** `mediaDir` (CHANNEL.md regenerated by `bin/file-schemas-docs.ts`); `dataDir` documented RETIRED and still parsed. - **`inspectChannelMedia`** describes `channels//media` + `mediaDir`, returns `mediaLink` and `text: { dir, readable }`, and has a seventh status, **`legacy`** (a `data` link or a recorded `dataDir`, answered from the corpus disk alone, text unreadable, its detail naming `archilyzer storage migrate-tier `). The marker parses an optional `scope` (`media` | `tier-migration`). Memo key: slug + `mediaDir` + `dataDir`. `assertChannelTextReadable` beside the media guard; `isTextHeld`, `HELD_REASON.legacy`; `channelMediaStall` keys on `mediaDir` (else the retired `dataDir`), `channelTextStall` on the retired `dataDir` only; `rootOfUnknownPath` strips `//media`. - **Which guard holds what.** The index and stats builds, the snapshot, `normalizeAllTranscripts`, `saveShardConfigAction`, clip eviction and the digest batch ask the text tier (`text.readable` / `assertChannelTextReadable`); the digest lane is held by `isTextHeld` or unreadable text, the other three by `isMediaHeld` (`isChannelHeldForLane`), and the pick→run marker backstop lets the digest lane through a media move; the backfill batch and `normalizeAllLiveChat` keep the media guard. Fourteen kinds flip `needsMedia` → `needsText` (the plan's list), and `runManagedFunction` asks the text guard for them, at enqueue and at start. `channelWriters(slug, { mediaOnly })` keeps `kindNeedsMedia` jobs and the non-digest lanes' units (no caller yet; T2's mover passes it). - **`onDrive` by file kind.** The snapshot reads the text directly; a video's tiered links are statted together as one `onDrive(mediaDir)` call, only while the media is `ok`/`in-place`; a drive that does not answer leaves `totalMediaBytes` and `totalAudioBytes` ABSENT and the snapshot is still written. Byte fields: `totalMediaBytes` (tierable names), `totalTextBytes` (new), `totalClipsBytes` (no longer inside media). `readChannelStat` and the recency tail reads pass a drive only for a legacy channel. - **umtool's twin** (`report-to-video/cues.mjs`) is now the twin of the TEXT guard: it refuses a legacy channel, mounted or not, and a marker only when its `scope` is `tier-migration`; relocated media is read past. **Deviations from the plan** (one sentence each): 1. `JobKindMeta.needsText` (+ `kindNeedsText`) is new: a kind flipped off `needsMedia` would otherwise run unguarded on a legacy channel whose `data/` link dangles and read it as empty. 2. The tier link carries the file's times (`lutimes`) and `normalizeLiveChat`/`isLiveChatCuesFresh` `lstat` the raw replay: the raw is media now, and the index's freshness check would otherwise reach the media drive per video. 3. `normalizeLiveChat` tiers only when it wrote the cues (not on "fresh"). As first shipped this was recorded as "an export build's normalize pass moves nothing", which was not true (`archiveLiveChat` called it, so a build with stale cues tiered — and read — the raw); since the review `archiveLiveChat` passes `tier: false`, so a build moves nothing. It still READS a stale raw replay through the link with no channel guard (T2: treat the corpus-wide live-chat passes as media readers). 4. `normalizeAllLiveChat` (corpus-wide, no slug for `runManagedFunction`) skips a channel whose media is not reachable — the plan's "`normalize-live-chat` refused". 5. `evictClipWindows` asks the text tier (`clips/` is never tiered) — its kind is in the flip list. 6. The classifier's scratch also takes `*.part`, `*.ytdl` and the hook's own `..tierlink-`. 7. `relocatedDataDir` stays exported (the retired shape) for the mover, the re-point, the rename and `storageActions.ts` until T2 replaces it with `relocatedMediaDir`. 8. `MediaLocationBadge.tsx` (T2's) gained the two `legacy` table entries the plan names (`Media layout retired`), because its two `Record` tables fail tsc without them. 9. The `isChannelHeldForLane` helper is new, so the lane decision is tested over real inspect answers. 10. `markerHoldsText`: a scope-less marker whose target is the retired `//data` shape (the old mover, until T2 writes `scope`) holds the text too, in the TS guard, the digest-lane backstop and the cues twin — the plan's "refuse only `tier-migration`" would let a digest write into a `data/` the old mover is copying. 11. The hook writes nothing while a `.relocating.json` stands on the channel (the file stays real), a backstop for a writer that started before a move. **Skipped until T2 rebases them** (28, each `{ skip: T1_SKIP }` with the reason string; they build the retired layout and expect it to read `ok`): `relocateChannelMedia.test.ts` 13 — "out: copies, links, records the target, keeps mtimes and reclaims the source", "abort from an onLog hook leaves the source intact, and the rerun completes", "back: restores a real directory, clears the config and reclaims the target", "out @ swap: crash before the rename — …", "out @ swap: crash after the config write — …", "out @ reclaim: the rerun sweeps every parked copy …", "back: an inconsistent channel is refused, …", "back: an unreachable channel is refused", "out @ swap: a directory timestamp is settled …", "out @ swap: a file the target is missing is mirrored …", "a resume with a stale extra dir on the destination completes", "reconcile: an extra and a changed file …", "reconcile: a marker past the copy phase resumes, …"; `renameChannel.test.ts` 1 — "rename re-points a convention-shaped relocated media dir"; `storageLocations.test.ts` 7 — "channelsOnLocation buckets ok / unreachable / moving …", "re-point rewrites both channels' links and configs, …", "re-point refuses a target that has no media …", "a failure on the second channel rolls the first one back", "re-point refuses a busy channel and names it", "a rerun after a crash finishes the channels that were left", "a channel killed between its symlink and its config write is resumed, …"; `storageWatch.test.ts` 7 (reason "release 17 T2 rebases the storage watch on mediaDir"; `storageWatch.ts` still reads `config.dataDir`) — "a channel whose target is gone is auto-paused, …", "the drive coming back restores the tier it overwrote", "write: false reports the transition …", "a drive that blips for one pass is never paused", "the restore needs only one good pass", "one missed probe stalls the location; …", "the stall clears only after two clean probes in a row". **Re-premised (this slice's own guards, not skipped):** `channelMedia.test.ts` (the media link; 7 legacy / text cases), `buildIndex.test.ts` (the hold now comes from an unreadable text tier — the channel put on the retired layout with its drive away — plus case (j): an unmounted MEDIA drive does not hold the index; the write spy no longer counts a symlink's target as a written path), `buildStats.test.ts` ((i) likewise, (i2) new), `storageStall.test.ts` (a stalled media drive: counts, recency and the snapshot are read with no call on it; the legacy cases keep the old gates; M3 is now "a tiered file's stat never answers → the snapshot is written, its media bytes unknown"), `channelSnapshot.test.ts`, `evictClipWindows.test.ts`, `doctor.test.ts`, `run-operation.test.ts`, and umtool's `cues.test.mjs` (6 cases). **Open questions, answered.** 1. **What `import-one` writes:** `importVideoAction` (`editor/app/channels/[slug]/pipelineActions.ts:465`) runs `downloadOneManaged` for one URL — media, subtitles, metadata, the archive line, then the roster. A media writer: it keeps `needsMedia`, and its media are tiered by `downloadOneManaged`'s hook. 4. **Who hardcodes `//data`:** only the mover family — `relocatedDataDir` and its callers (`relocateChannelMedia.ts`, `renameChannel.ts`, `storageLocations.ts`'s re-point, `storageActions.ts`'s marker check), `deleteChannel`'s `/` reclaim and `lib/storageLocations.ts`'s and `savedVideoStore.ts`'s comments — all T2's; and `storageHealth.ts`'s `rootOfUnknownPath`, which now takes both suffixes. Nothing in umtool, mcp, docker or `scripts/`. So `//media` beside `//data` is as planned. **`isFile()` over a video dir, the census** (a dirent `isFile()` is false for a link): `channelSnapshot.ts`'s byte loop — rewritten (`lstat`, links statted through the watchdog); `channelSnapshot.ts`'s `dirFileBytes`, `evictClipWindows.ts:145` and `clipWindow-server.ts:49` — over `clips/`, never tiered, fine; `downloadOneManaged.ts:430` (`discardPrefetchDir`) — a tiered link keeps the directory, the safe direction; `relocateDir.ts:70` (`measureTree`) — keeps `isFile()` by the plan. **For T2:** `videos/[id]/page.tsx:67` (`loadVideoDir`) and `videos/page.tsx:61` hide a tiered link today — on a tiered channel the video page and the videos list would not show the audio until T2 lands. **Found and left.** - For T2: `views/storage.ts`'s "N of it is fetched clip windows" and `storageLocations.ts`'s `mediaBytes`/`clipsBytes` rollups still treat clips as part of `totalMediaBytes`; `channelRow.ts` and `freeUpSelection.ts` read `totalMediaBytes`, now the media tier alone. The old mover still records `dataDir`, so a Move media on this branch alone produces a `legacy` channel. - For T3: the migration's links should carry each file's times (`lutimes`, as the hook does), or every migrated live chat's cues read as stale and the next index build re-parses the raw from the media drive; `buildIndex` re-parses a stale raw replay directly (no watchdog) and skips the track when it cannot. - `generateChannelSnapshot` passes a null config (a `config.json` with no valid `handling`) to the guard, which then reads no relocation — as before this slice. - `removeVideoDirMedia` cannot clear `media//` while the media drive is unmounted (its `lstat` fails); the video page's delete then leaves those bytes behind. **Commits** | Commit | What | |---|---| | `1d228a79` | `common:` the classifier (`mediaTier.ts`) and the hook/deleter (`mediaTier-server.ts`) + tests | | `8335f151` | `common:` the model — `mediaDir` (CHANNEL.md regenerated), `legacy`, the text guard, `isTextHeld`, the builds/snapshot/normalize/shards/eviction/digest batch on the text tier, the snapshot's three byte fields, `needsText` and the fourteen flips, `mediaOnly`; tests re-premised; the 28 T2 skips | | `85180cd6` | `common, editor:` every media finalisation tiers (transcode, both outcome writes, the batch modes, live-chat normalize), every deleter derefs, the link carries the file's mtime | | `05275d86` | `common, umtool:` `isChannelHeldForLane` + its test, `normalizeAllLiveChat`'s media guard, the flips pinned, the text-guard job test, the call-site hook tests, the cues twin as a text guard | | `d56050ff` | `common, umtool:` `markerHoldsText` (the old mover's scope-less `…/data` marker holds the text), the hook writes nothing under a marker | | this commit | `plans:` this section, FACTS ("Release 17 slice T1"), the editor changelog | #### Gates (logs `$T/T1-*.log`) - **tsc** (all workspaces) clean at every commit; last at `d56050ff`. - **common:** at `05275d86` **2,529 passed, 0 failed, 28 skipped** (2,557; `main`'s 2,528-test run had 49 failures on this branch's first pass, all re-premised or skipped as above). At `d56050ff` 2,530 passed, 1 failed, 28 skipped: the failure is `storageHealth.test.ts`'s "M4: a healthy 64-wide walk … at half the budget" timing case under a machine load of 22–28 (other sessions); the file passes 36/36 twice in isolation right after. New tests: `mediaTier.test.ts` 35, `mediaTier-server.test.ts` 16, `mediaTierHooks.test.ts` 4, `channelMedia.test.ts` 23 (rewritten), plus cases in `channelWriters`, `autoRunner`, `streamCommand`, `jobKinds`, `buildIndex` (j), `buildStats` (i2), `storageStall`, `channelSnapshot`, `evictClipWindows`, `storageHealth`. - **Editor unit:** 109/109. **test:scripts:** 304 passed, 2 skipped (306) — a first run had the two `queue-lock.test.mjs` timing cases fail under load; the rerun is clean. **umtool `cues.test.mjs`:** 23 passed, 1 skipped (LIVE). - **Build:** the capped editor build (`systemd-run --scope -p MemoryMax=6G`, `next build`): exit 0, 172 s, at `05275d86`. - **e2e** (from the worktree root, `$T/T1-specs.txt`: maybe-missing, video-page, cleanup-holds, cleanup-actionable, auto-queue, digest, jobs-channel, channel-storage) at `05275d86`: **78 passed, 5 failed, 8.9 min** (after 33.6 min in the queue). The 5 are all `channel-storage.spec.ts`, all the retired layout reading `legacy`, as expected until T2 rebases the mover: "relocate a channel's media to another root, and move it back" (:80), "the /channels bulk move queues one job per channel and skips the rest" (:250), "a bulk move puts every job on one queue and skips a channel with nothing to move" (:391), "the Storage panel moves to a location picked by name" (:484), "Sync all skips a channel whose media drive is not mounted" (:1091 — now says `media legacy: … run archilyzer storage migrate-tier test-youtube`). Not re-run after `d56050ff` (unit-covered; a marker rule the specs do not reach). - **Privacy gate:** 0 added lines carry the user or host name (`git diff 7f4901f1`, counts only; the one file the whole-file grep names is `plans/FACTS.md`, with the same count as on `main`). No identifier ending in the suffix the publish gate refuses. - **Numbers tool:** none. #### Review (SHIP AFTER FIXES, 12 findings) and the fixes Every commit over `main..HEAD` was rewritten (`git filter-branch --msg-filter`, worktree only) so its trailer is the session's `Co-Authored-By: Claude Fable 5.1 ` + `Claude-Session` line (finding 12): tip `f3f23792` → `001e9de7`; the shas in the table above are the rewritten ones (`828dc1a2` → `98ef1772` is the first `plans:` commit). | Finding | Fix | |---|---| | H1 the index stats (and may read) a tiered live chat through its link | `263c8b17`: the scan's `subsMs` loop `lstat`s a tierable track; a stale-cues fallback on a tiered raw reads through `onDrive(mediaDir)` only while the media is `ok`/`in-place`, else keeps the cues the last build held; `6b795c67`: an EXDEV tier gives the copy the file's times (`stat` and `lstat` agree). New gate `buildIndex.test.ts` (k): a STALLED media drive (the location marked, every call through a link onto it hanging) with a tiered live chat — the index and the stats build complete, nothing held, the live-chat cues kept, no call on the drive. A mutation (a following `stat` in `subsMs`) makes it hang. | | L2 crash window between the two renames | `6b795c67`: the bytes are placed by hard link (same disk) or atomic copy and renamed into the tier, then ONE rename swaps the name — it always resolves; `tierVideoDir` removes a dead process's stray `.tierlink-`. Tests: one inode after a same-disk tier; a stray healed. | | L3 `removeMediaFile` derefs only into `media//` | `6b795c67`: any id under the channel's own `media/` (never out of it); the target's dir is dropped when it empties. Test. | | L4 the export build tiers the raw live chat | `95c9a82c`: `normalizeLiveChat({ tier: false })` from `archiveLiveChat`; deviation 3 corrected above (the build still READS a stale raw — for T2). | | L5 a move that starts mid-hook (for T2) | `6b795c67`: the marker is asked again after the bytes land and before the name changes; the tier's copy is removed if one stands. | | L6 a video-dir delete on an unmounted or stalled media tier | `95c9a82c`: `deleteOneVideoDir` refuses unless the channel's media is `ok`/`in-place`, and removes the tier's side through `onDrive(mediaDir)`, refusing with the drive's sentence when it does not answer — before the text is touched. | | L7 gates at the tip | re-gated below. | | N8 an in-place `media/` takes a watchdog slot | `6b795c67`: a real `media/` is answered from the corpus disk. | | N9 `totalTextBytes` counts containers and partials | `95c9a82c`: documented as everything on the corpus disk but `clips/` and the tier. | | N10 a shard save runs the tier sweep | `95c9a82c`: skipped when `saveShardOnly`. | | L11 records | `001e9de7`: FACTS and the changelog bullet say what the index does with a tiered live chat; the bullet says "Needs a rebuild and restart of the editor." and names the refused video delete. | **Gates at `001e9de7`** (logs `$T/T1-regate2.log`, `$T/T1-e2e-2.log`): tsc clean; common **2,535 passed, 0 failed, 28 skipped** (2,563); editor unit 109/109; test:scripts 304 passed, 2 skipped; umtool `cues.test.mjs` 23 passed, 1 skipped; capped editor build exit 0, 101 s; e2e (the same 8 specs) **78 passed, 5 failed, 8.2 min** (after 39 min in the queue) — the same five `channel-storage.spec.ts` cases (:80, :250, :391, :484, :1091), the old mover's layout reading `legacy`, nothing else red. Privacy: 0 added lines carry the user or host name. **Re-review: SHIP**, with R1 and R2 folded in before the merge. R1 (`15acd7c7`): the index read a whole raw live chat under the watchdog's one-stat budget, and a timeout was never retried. Ruling: it probes the drive with one `stat` through `onDrive(mediaDir)` and then reads the raw directly, and a track that is neither read nor kept is stored with `subsMs: null`, so the next build retries it (case (l), mutation-checked). R2 (`ba24a313`): the classifier calls the hook's media-side temp (`..tiering-`) scratch, and `tierVideoDir` sweeps a dead process's temp from `media//` while the tier answers. Gates at `ba24a313`: `buildIndex.test.ts` + the tier tests 72/72; tsc clean; common 2,538 passed, 0 failed, 28 skipped (2,566). **Merge of `main` `173dd42b`** (D0 on U1, export 0.11.1, XP) at `ffbfbd63`: three conflicts, all appends — `channelWriters.test.ts` keeps both new tests (T1's `mediaOnly`, D0's ghost meta), `editor/CHANGELOG.md` keeps every `[Unreleased]` bullet (main's, then T1's), and this file keeps U1, XP, D0, then T1 under `## Record`; `channelSnapshot.ts` (D0's `mapInYieldingChunks` around T1's per-video body), `channelWriters.ts` and `buildIndex.ts` (XP's per-site posts predicate) merged cleanly. Re-gated on the merged tree (`$T/T1-regate4.log`, `$T/T1-e2e-3.log`): tsc clean; **common 2,573 passed, 0 failed, 28 skipped** (2,601); **editor unit 109/109**; **test:scripts 394 passed, 2 skipped** (two runs at a machine load of 32–35 failed `queue-lock.test.mjs`'s timing cases, 1 then 2; that file passed 11/11 three times alone and the full suite was clean at load 10 — this branch does not touch `scripts/`); capped editor build exit 0, 92 s; **e2e (the 8 specs) 78 passed, 5 failed, 8.2 min** — the same five `channel-storage.spec.ts` cases (:80, :250, :391, :484, :1091), the old mover's layout reading `legacy`, nothing else red. ### Slice RL, as shipped — a subtitle 429 does not fail a download, and the pace adapts (2026-10-01) Branch `r17/rate-limit-adapts` off `main` `90bd8384`, worktree `~/Projects/r11-runner-lows` (editor 4801, test 4811, export 4810 — `pnpm wt list`'s block #18), one Opus implementer. Scratch files `RL-*` in the job's `tmp`. `main` moved under the slice: it was merged at `1d5c33bf` (U1, XP), at `a395aaa1` (D0, T1) before the final gates, and at `0a62bf74` (U2) after them. The ruling is above ("Slice RL — the ruling"). **What yt-dlp does, verified offline.** yt-dlp 2026.08.19 (the editable install the editor runs) was run against a localhost HTTP server whose subtitle URL answers 429 — no YouTube request, per the operator's rule. `--ignore-errors` (`ignoreerrors: True`; `--no-abort-on-error` is `'only_download'` and still raises) reports `WARNING: Unable to download video subtitles for 'en': HTTP Error 429: Too Many Requests`, exits 0, still fires `--print after_video:…`, and — when the run is not `--skip-download` — goes on to download the media. Without it, a `--load-info-json` run that hits the subtitle error prints `The info failed to download: … trying with URL …` and re-extracts from the URL: the "one internal re-extraction" the evidence saw. So the ONE-spawn shape is yt-dlp's own, and the log keeps the line for the classifier. **What it does.** - **A subtitle 429 is not a failure of the download.** `classifyDownloadFailure` returns the new `subs_rate_limit` when a subtitle-429 line is present and every `ERROR:` line is a subtitle line (no soft block, no bot check). The youtube-handling primary (subtitles only, `--skip-download`) now carries `--ignore-errors` and `--sleep-subtitles max(5, pace)` after `-t sleep`: a subtitle 429 exits 0 with no transcript, and attempt 3 — the no-subs media pass, `--no-write-subs --no-write-auto-subs` appended after the channel's own args — downloads the audio, exactly as for a video with no captions (three spawns, as before for such a video). The record is a SUCCESS carrying `failureClass: "subs_rate_limit"` (the one class set on a success). Any other primary that died on its subtitles alone (a channel whose own args ask for subtitles) is run once more as `primary-without-subs` (a new `DownloadAttemptKind`). - **Only the subtitles are deferred.** `applyUnitOutcome` (lane) and `runManagedDownloads` (batches, through `recordSubtitleDeferral`) record `subtitleDeferrals[id] = {count, lastAt, until, channelSlug}`: 6 h after a strike, 7 days from the third. The platform backoff, hold and pace are not touched, the video is retired like any success, and it is not added to `videoDeferrals`. **Download missing subs** skips a video inside its window (named in the prefilter line with its count and date), records a subtitle 429 and goes on to the next video whatever `abortOnError` says, and clears the deferral when the subtitles come down (`runChildAndStream` now returns its stderr tail and attaches it to the thrown error). A download from the video page never reads it. - **The pace adapts.** `platformPace[pf] = {sleepRequestsSeconds, baseSeconds, cleanUnits}`: absent at the static value (`PLATFORM_ARGS`' `--sleep-requests`, 1 s for YouTube and Rumble, 0 for a platform with none); every platform-level `rate_limit` doubles it (from 0 to 1) up to `pacing.sleepRequestsCapSeconds` (16); every `pacing.decayAfterCleanUnits` (5) clean units halve it, and an entry back at its base is deleted. A `network` failure and a `subs_rate_limit` never move it. `channelExtraArgs` reads it synchronously through `livePlatformPaceSeconds` — the shared state when a runner holds it, else the pace the last read or write of the file saw, else the static value — so every spawn against the platform (listing, prefetch, primary, availability, metadata scan, clip, the new-channel probe through `pacedPlatformArgs`) uses it; `withSleepRequests` only raises, and the channel's own `ytdlpExtraArgs` still win. A manual 429 (`recordDownloadBackoff(pf, paths, failureClass)`) escalates the same way. - **The lane waits between units.** `platformNextStartAt` in the runner = settle + `downloadGapMs(sleepBetweenDownloadsSeconds, pace, base)`: the setting the lane used to ignore, plus the pace above its base. `runManagedDownloads` sleeps the same sum. - **A block is not a burst.** `FAILS_TO_REACH_CAP` = 6; a backoff that has failed at the cap `pacing.holdAfterFailsAtCap` (3) times in a row — `fails` 8 — holds the platform: `platformHolds[pf] = {since, probeAt}`, the backoff's `until` = now + `pacing.holdProbeMinutes` (60, no jitter), so the lane's existing cooldown gate lets one probe unit through per interval. A failed probe re-arms it; only a clean unit (`transcribed`, no subtitle 429) clears the hold and the backoff together. A held platform's backoff entry is never pruned. A manual Sync, any download mode but Store playlist, a metadata scan, and the video page's fetch-window / full-source fetch are refused with `heldPlatformSentence`: `youtube is held: its rate limit outlasted the cooldown cap (8 failures in a row, held since 14:02 UTC). Auto-download probes it once every 60 min — the next probe is in 42 min. Sync will run once a probe comes back clean.` (scheduled syncs go through the same action and are refused alike). - **Visible.** The download lane's `role="region"` "Rate-limit cooldown" gains `
    ` (since, failures, next probe, pace), `"Request pace"` (seconds between requests and the base) and `"Deferred subtitles"` (link, count, time left; "left alone" from the third); "Platforms in cooldown" now lists only unheld platforms and shows the pace too; `formatCooldown` gains a days arm. The view (`autoQueueStatus.ts`) carries `cooldowns[].hold`, `pace[]` and `subtitleDeferred[]`. Idle reasons gain `held` ("every pending platform is held after repeated rate limits — one probe at a time") and `paced` ("every pending platform is pausing between downloads"), in both exhaustive switches. The video page draws one `role="note"` `aria-label="subtitle deferral"` line for a video with a deferral, read from a new `GET /api/channels//videos//subtitle-deferral`. `archilyzer doctor` has a **download pacing** section: a cooldown, a hold or a raised pace warns (never fails), deferred subtitles are a note. - **Settings:** a `pacing` block (`sleepRequestsCapSeconds` 16 [1–120], `decayAfterCleanUnits` 5 [1–1000], `holdAfterFailsAtCap` 3 [1–100], `holdProbeMinutes` 60 [1–1440]); SETTINGS.md and settings.json.example regenerated; `sleepBetweenDownloadsSeconds`' description says the lane honours it now. | sha | what | |---|---| | `93462d22` | common: `subs_rate_limit` (`availability.ts`), the youtube primary's `--ignore-errors` + `--sleep-subtitles`, attempt 3 on a subtitle 429, `primary-without-subs`, the record's class; `platformBackoff.ts` pace/hold/subtitle-deferral seams + coercion; `autoQueueState.ts` three maps + `livePlatformPaceSeconds`; `unitOutcome.ts`; `downloadBackoff.ts` one write-through + `recordSubtitleDeferral`/`clearSubtitleDeferral`/`heldPlatformRefusal`; `platformArgs.mjs`/`channelArgs.ts` the pace; `autoRunner.ts` gap, hold idle, `held`/`paced`; `runYtdlp.ts` batch gap + deferral, download-missing-subs; `pacing` settings + SETTINGS.md; unit tests | | `8a40c91e` | editor + doctor: the region's three lists, the hold refusals (`pipelineActions.ts`, `videoActions.ts`), the video page's line, `autoQueueStatus` fields, `doctor` section, `checkAvailability`'s paced probe, a manual Sync's backoff passes its class | | `49b3d087` | e2e: `rate-limit.spec.ts` (3 tests); the fake's `dl429` obeys `--ignore-errors`, new `wp429` (watch-page 429 at the prefetch); `pacing.spec` T2 moves to `wp429`; `rumble-sweep.spec`'s paged walk paces at 2 s; the video-page line moved out of `cards/` | | `2ede037c` | plans: the ruling, FACTS, the dated note on `youtube-lane-pacing.md`, two `[Unreleased]` bullets | | `8c76ec09` | merge `main` `1d5c33bf` (U1, XP): both changelog sides, both rulings and slice rows kept | | `a8e9c241` | editor: the video page's line reads `GET /api/channels//videos//subtitle-deferral` instead of a mount-time server action (Next runs a page's server actions one at a time, so it sat in front of the first click — e2e run 2) | | `5e967e18` | merge `main` `a395aaa1` (D0, T1): both changelog sides kept; no code conflict (T1's tier hooks and lane holds sit clear of this slice's hunks) | | `5725e53a` | merge `main` `0a62bf74` (U2): clean; U2 touches umtool, the changelog and the plans only | | *(this commit)* | this record | **Gates** (worktree root). tsc (`pnpm -r --no-bail --workspace-concurrency=1 exec tsc --noEmit`) clean before every commit (a stale `editor/.next/dev` and `export/.next/dev` from the worktree's earlier use were removed first). Before the merge of `main`: common **2513/2513**, editor unit **109/109**, test:scripts **368 pass + 2 skip**, capped editor build ok (78 s). At the first merged tip `8c76ec09`: common **2530/2530** (29 of them new: `unitOutcome` +6, `platformBackoff` +5, `channelArgs` +4, `downloadBackoff` +4, `availability` +3, `subtitleRateLimit` +2 (new), `managedDownloadsSleep` +2, `autoQueueState` +1, `autoQueueStatus` +1, `doctor` +1), editor unit **109/109**, test:scripts **392 pass + 2 skip**, capped editor build ok (compiled in 24.0 s, 73 s). `settings example --check` clean. At the final tip `5e967e18` (after D0 and T1): tsc clean; common **2630 tests, 2602 pass, 0 fail, 28 skipped** (the 28 are `main`'s — T1's `relocateChannelMedia` cases skipped until T2, "release 17 T2 rebases the mover on media/"); editor unit **109/109**; test:scripts **394 pass + 2 skip**; capped editor build ok (compiled in 23.3 s, 56 s; the new route listed). EDITOR e2e, `$T/RL-specs.txt` = `rate-limit.spec.ts pacing.spec.ts auto-queue.spec.ts lane-runner.spec.ts channel-priority.spec.ts video-page.spec.ts rumble-sweep.spec.ts no-subs-fallback.spec.ts queues.spec.ts`: - run 1 (`RL-e2e1.log`, `49b3d087`): **74 passed, 1 failed, 13 min** (4 in the queue). The three `rate-limit.spec` tests, `pacing.spec` (T2 1.0 min) and `rumble-sweep.spec` passed. The failure was `auto-queue.spec:623` "Drain completes when an auto-transcribe unit is parked behind a busy worker" — the 30 s test budget ran out while polling `/api/auto-queue/status` (a transcription-lane test; nothing in this slice runs on that lane). - run 2 (`RL-e2e2.log`, `8c76ec09`): **74 passed, 1 failed, 26 min** (≈13 in the queue). The Drain test passed; `video-page.spec:280` "Mark untranscribable…" timed out: its click's server action queued behind the deferral line's mount-time action. Fixed in `a8e9c241`. - run 3 (`RL-e2e3.log`, `a8e9c241`, `rate-limit.spec.ts video-page.spec.ts`): **23 passed, 0 failed, 9 min** (≈8 in the queue). - run 4 (`RL-e2e4.log`, `5e967e18`, the full list): **71 passed, 4 failed, 8 min** — the four video-page text-preview tests (`:325`, `:367`, `:384`, `:472`), each a 5 s / 30 s timeout on a preview fetch, while a common test run of this implementer's was loading the machine beside it. - run 5 (`RL-e2e5.log`, `5e967e18`, `video-page.spec.ts` alone, nothing beside it): **20 passed, 0 failed, 1 min**. - run 6 (`RL-e2e6.log`, `5e967e18`, the full list, nothing beside it): **75 passed, 0 failed, 5 min** (no queue wait). After the U2 merge (`5725e53a`, umtool-only code): tsc clean; test:scripts **461 pass + 2 skip** — in two of four runs one or two `scripts/queue-lock.test.mjs` cases ("serves waiters in arrival order (FIFO)", "prints a banner naming the holder while waiting") failed while other slices' e2e runs held the machine-wide lock; a run with the lock free passes. The editor build, common, editor unit and e2e were not rerun: U2 changed no `common/` or `editor/` code. **Numbers: none.** `.auto-queue/state.json` is outside both numbers tools. **`platformPace: {}`, `platformHolds: {}` and `subtitleDeferrals: {}` appear on all four lanes of `state.json` at the first persist after the rollout boot**, and `pacing` appears in `settings.json` at the next save; an older build drops the three keys on its next write, so a rollback is safe. Privacy gate: `git diff main --name-only | xargs grep -lc "$(whoami)\|$(hostname)"` prints `plans/FACTS.md` only, for three lines `main` already carries (3477, 5033, 5430 at `0a62bf74`); the slice's added lines carry none. **Deviations, one sentence each.** - `subs-deferred` is not an idle reason: a subtitle deferral never idles the lane (the media is done), so it is the region's "Deferred subtitles" list instead; `paced` was added for the new gap, which would otherwise read as "capped". - The gap adds the pace ABOVE its base, not the whole pace: the base is already paid between the requests of every spawn, and adding it again would slow every lane unit and batch by a second with nothing rate-limited. - The lane's gap reads the global `sleepBetweenDownloadsSeconds`; a channel's own override still applies to its batch runs only. - No rack chip: the rack has no per-lane or per-platform chip to say "held: rate-limited" on, and threading the platform state into every row is more than a few lines in `channelRow.ts`, which is T2's. - Files touched beyond the list, each by a small hunk: `lib/downloadOutcome.ts` (the attempt kind, the class's doc), `lib/settingsDocs.ts` (the `pacing` table), `views/autoQueueStatus.ts` (three fields in `buildKind`, clear of D0's memo), `VideoPanel.tsx` + a new `SubtitleDeferralLine.tsx` beside it and its new GET route (the video page's line; `videoActions.ts` keeps only the refusals), the fake yt-dlp, `pacing.spec.ts` and `rumble-sweep.spec.ts` (their expectations follow the ruling). **Found and left.** - Subtitle deferrals are written by the lane, batch downloads and download-missing-subs; the video page's own download, the re-acquire backfill and import neither record nor clear one (a manual fetch never reads it either). - The `pacing` keys are in `settings.json` only, not on the `/settings` form. - The runner merges only `platformBackoff` from disk mid-run, as before; pace and holds written from outside go through the live object (`mutateDownloadState`), and a disk-only write while no runner lives is read at the runner's start. - **Rollout:** the live backoff reads `fails: 80` (the subtitle 429s this slice stops counting). Unless the boot finds it lapsed for over 30 min (then it is pruned, as before), the first real platform-level failure after the restart holds YouTube at once. **Clear hold** on `/operations/download` is the way out (it drops the hold, the backoff and the pace and says so in a `clear-platform-hold` job log); a clean lane unit, or a clean manual Sync once the probe is due, clears it too. The changelog bullet says the same. - **Open question for the operator** (not probed): whether YouTube's timedtext 429 for these twelve videos is a PO-token / player-client matter (`--extractor-args youtube:player_client=…`). #### Review (2026-10-01): SHIP AFTER FIXES → fixes | finding | fix | |---|---| | **H1** — under `--ignore-errors` EVERY subtitle failure exits 0 (a 403/404/5xx, a failed `live_chat` replay, an OSError writing the file) and ended `ok`, archived, with no transcript and no media, and counted as a clean lane unit | `5de4e385`: `hasNonRateLimitSubtitleFailure` (`lib/availability.ts`); a youtube primary that exited 0 with such a WARNING is a failed attempt, as before — `lastSucceeded` false, no media pass, no archive line, its error and class from the tail (a 403 → `network`, a `live_chat` 404 → `unknown`). Only a subtitle 429 takes the media path. Unit cases for a 403 and a `live_chat` failure (`subtitleRateLimit.test.ts`). FACTS' `--ignore-errors` bullet says "any subtitle failure" | | **H2** — a hold ended only with a lane probe; with the lane off or nothing pending it was permanent, Sync and scheduled syncs stayed refused, the page dropped the platform once the probe was overdue, and the refusal said "next probe in 1 min" for ever | `f003e29e`: `heldPlatformRefusal` refuses only while held AND before the probe time (as `fetchWindowAction` already did); `runYtdlp`'s new `onPlatformClean` (wired in `pipelineActions.ts`) and a clean metadata scan call `recordPlatformClean`, which settles the platform like a clean probe (backoff and hold clear) and logs it; `clearPlatformHold` drops hold + backoff + pace. The status view keeps a held platform whatever its probe time. `c2cc1228`: "Platforms held" says "probe overdue — the lane is off" (or "probe due — it waits for a pending video") and each row has **Clear hold**, run as a one-step `clear-platform-hold` job on queue `pacing:` whose log says what was cleared. e2e: an overdue hold with the lane off is shown, a Sync is not refused, and its clean run lifts the hold; Clear hold empties hold, backoff and pace | | L1 — a held probe whose media came down but whose subtitles 429'd did not lift the hold | `f003e29e`: it lifts the hold and the backoff, and does not count toward the pace's easing (`countForDecay: false`) | | L2 — the pace only eased on lane units | `f003e29e`: **the one time-based rule** — a raised pace eases one step per hour with no rate limit (`steppedAt`, `PACE_TIME_DECAY_MS`); readers (the args builder, the view, the doctor) apply it through `effectivePaceSeconds`, writers through `decayPaceByTime` | | L3 — Download missing subs ran back to back | `f003e29e`: it waits `downloadGapMs(sleepBetweenDownloadsSeconds, pace, base)` between videos | | L4 — the `fails: 80` rollout hazard was in the record only | the changelog bullet and the record's rollout bullet both say it, and that Clear hold is the way out | | L5 — a hold reached through network failures read as a rate limit | `f003e29e`: the hold carries `rateLimited`; the sentence, the list and the doctor say "failing (network errors)" for such a hold | | N9 — a WARNING webpage 429 beside a subtitle 429 read as subtitles-only | `5de4e385`: every 429 / too-many-requests line must be a subtitle line, and no non-429 subtitle failure may be present | | N10 — "YouTube's subtitles" on surfaces that match any platform | `c2cc1228`: the list heading and the video-page line say "the subtitles" | One more fix commit, `7e59c61f`: Clear hold's sentence lived inside the region it empties, so a clear that left nothing to list unmounted it; it now lives on the lane view (found by run 7). **Re-gate** (tip `7e59c61f`; the ruled list `rate-limit.spec video-page.spec auto-queue.spec lane-runner.spec`, `$T/RL-specs-fix.txt`). tsc clean before every commit. common **2638 tests, 2610 pass, 0 fail, 28 skipped** (the 28 are `main`'s T1 mover cases; 8 new: `subtitleRateLimit` +2, `unitOutcome` +3 and one rewritten, `downloadBackoff` +2, `availability` +1, `autoQueueStatus` +1). Editor unit **109/109**. EDITOR e2e: - run 7 (`RL-e2e7.log`, `c2cc1228`): **53 passed, 1 failed, 4 min** — the new Clear hold test (the sentence unmounted with the region), fixed in `7e59c61f`; - run 8 (`RL-e2e8.log`, `7e59c61f`, `rate-limit.spec` alone): **5 passed, 0 failed, 4 min** (≈3.5 in the queue); - run 9 (`RL-e2e9.log`, `7e59c61f`, the ruled list): **54 passed, 0 failed, 2 min**. test:scripts, the build and the full suite were not rerun, as ruled (the fixes touch no `scripts/` or umtool file; the new client code is covered by tsc and the e2e above). **Re-review (SHIP) → the cheap items before the merge.** | finding | fix | |---|---| | R1 — a subtitle 429 beside another subtitle failure (`en` 429 + `live_chat` 404) still classed `rate_limit` | `9c01dea2`: `classifyDownloadFailure` drops the subtitle-429 lines before the generic rate-limit test, so the class is the other failure's (`unknown` for the 404, `network` for a 403); a real platform 429 beside them still wins | | R3 — a manual run that asked the source nothing counted as clean | `9c01dea2`: `runYtdlpMode` counts requests (`requestCounter`, shared through every `{...opts}` copy: the listing, each per-video download, each raw spawn) and calls `onPlatformClean` only when it is above 0 — download-missing with nothing to fetch, or download-missing-subs with every video deferred, no longer lifts a due hold (`platformClean.test.ts`, 2 cases) | | R4 — `clear-platform-hold` was not a declared kind | `9c01dea2`: declared in `jobKinds.ts` ("Clear rate-limit hold"; not drainable, not replayable, custom queue, neither `needsMedia` nor `needsText`) | | R5 — "probe due — it waits for a pending video" on a paused lane | `8a59dce6`: a running lane whose gate is held, or idle `lane-held` / `downloads-paused` / `snoozed` / `disk-gate`, says "probe overdue — the lane is paused" | **Gates before the merge** (tip `8a59dce6`): tsc clean; common **2641 tests, 2613 pass, 0 fail, 28 skipped** (main's T1 mover cases); editor unit **109/109**; capped editor build ok (compiled in 17.7 s, 62 s; the fix pass added a `"use server"` module); EDITOR e2e `pacing.spec rumble-sweep.spec no-subs-fallback.spec rate-limit.spec` (`$T/RL-specs-merge.txt`): run 10 (`RL-e2e10.log`): **12 passed, 0 failed, 2 min**. **Left, recorded only (follow-ups).** - The video page's own download, the re-acquire backfill and import neither record nor clear a subtitle deferral, so the page's "Left alone … until" note can be stale after a page download fetched the subtitles (review item 7). - `pacingPlatformKey` reads `detectPlatform(url)` while the base pace reads `channelPlatform(config)` (`config.platform` first); they can disagree for a channel whose `platform` differs from its URL (N11). `pacing` has no `/settings` form. - umtool's own spawns (`check-availability.mjs`, `build-video.mjs` through `platformArgsForUrl`) run in a separate process and stay at the fixed pace: "every spawn" means every spawn of the editor and the CLI (N12). - R2: a persistent per-video subtitle 403 is `network`, which escalates the backoff without deferring the video, so one video can walk a platform into a hold (pre-RL behaviour plus the hold; a clean manual Sync once the probe is due, or Clear hold, gets out). Follow-up: defer the video when the only failure is a subtitle line. - An H1 failure records `ytdlpExitCode: 0` with an `error` (the one attempt that does; the job log line says why). ### Slice T2, as shipped — the mover over `media/`, and the surfaces (2026-10-02) Branch `r17/media-tier-mover` off `main` `a395aaa1` (U1, XP, D0 and T1 merged), worktree `~/Projects/r12-paths-fix` (editor 5001, test 5011, export 5010), one Opus implementer, beside U2 and RL. Scratch files `T2-*` in the job's `tmp`. The plan is "The model (A′)" §5 and §6, the `loadVideoDir` change in §2, the `onDrive`-by-file-kind editor changes in §4, and the T2 row; T1's "left for T2" items are below. **What it does.** - **The mover carries `channels//media`, never `data/`** (`relocateChannelMedia.ts`). Out: the job's first step asks the writers, then a classic channel is tiered in place (`tierChannelMedia(paths, slug, { createMediaDir: true })` — same-disk renames and links; only in the copy phase: a resumed run's marker stands and the hook writes nothing under one), then `media/` is measured, copied to `//media`, mirrored, verified, parked as `media.relocated-`, replaced by ONE absolute link, and `config.mediaDir` written. Back: the target → `media.incoming` → the link swapped for a real `media/` on the corpus disk, `mediaDir` unset, the target reclaimed; not one per-file link is touched in either direction. The marker keeps its shape plus `scope: "media"` (`readDirMarker` now keeps a `scope` it reads, so a resume rewrites what it found). The space check is `media/`'s bytes plus the margin. Refused with nothing touched: a `legacy` channel (`… cannot be moved: its media layout is the retired whole-directory one — run archilyzer storage migrate-tier .`), out, back and in the preview; a marker whose `scope` is `tier-migration`. `relocationRootProblem`, `assertRelocationRootPresent` and `rootOfRelocatedMediaDir` (renamed from `…DataDir`) work on `//media`; `relocateDir.ts`'s names follow. The result carries `tiered`; the job's done line says "(N file(s) tiered first)". - **The preview tiers an in-place classic channel too** (with a `data/`, no marker, no `mediaDir`) and reports `tieredFirst`, shown as "N file(s) tiered first"; a second preview tiers nothing. - **A move is held by the channel's media writers only.** `channelMediaWriters` (new, in the mover) is `channelWriters` minus every job whose kind is not `needsMedia` — except `relocate-channel-media`, the move itself — and minus the digest lane's units. The job's first step and the ask under the marker, the preview, and the editor's `channelMediaBusyReason(slug, what, { mediaOnly: true })` for Move, Move back, Resume, Reconcile, Clear marker, the bulk move and the panel's blocked line ask it; rename and delete still ask every writer. - **Locations key on `mediaDir`** (a legacy channel on its retired `dataDir`): `channelsOnLocation` (new `textBytes`, `unknownTextBytes`, `legacy`), `channelVolumeOf(config, …)`, the storage watch (one auto-pause per channel, as before), the re-point (rewrites the `media` link and `mediaDir`; since the review a legacy channel on the same location is re-pointed the retired way — see "Review"), the rename (a convention-shaped `//media` moves and the `media` link is re-pointed; the per-file links are relative and move with the channel dir; since the review a legacy channel's rename is refused), `deleteChannel` (removes `mediaDir`, and a legacy channel's `dataDir`). - **`/storage`**: a location row's figure is its channels' media tier, with no clips; the internal row is in-place media + every channel's text + every channel's clip windows (`internal.corpus`, folded in `buildStorage.ts`), with the breakdown "text N + clips N on the corpus volume, plus N media of in-place channels" (`aria-label="location tier bytes"`); a report with no `totalTextBytes` counts unmeasured, once per channel; `legacy` counts as unreachable with "(n to migrate)". The page's and the re-point's prose name `mediaDir`. SETTINGS.md regenerated (one field doc names `config.mediaDir`). - **The Storage panel**: two rows under one heading — `media path` (the target, or `/media (in place)`) with `media bytes` (`totalMediaBytes`, the tier), and `text path` with `text bytes` (`totalTextBytes`); `free on media volume` kept; the buttons keep their names; the hold sentence says "Its text stays readable: the video pages, the index and the digests go on." (or that the text is held too, when it is); the stale-marker paragraph says the media lanes skip the channel. A legacy channel's Move back is refused with the migrate-tier sentence (`move back refused`). Resume refuses a tier-migration marker. The badge's `legacy` entry is "Media layout retired" (danger; T1 added it for tsc). The Configure form's read-only line shows `mediaDir`, a legacy `dataDir` with the command, or "In the channel directory (media/)". - **The two video pages read the text on the corpus disk**: they gate on `channelTextStall` (a legacy channel only), and list a directory through one loader, `loadVideoDirFiles(videoDir, mediaDir)` (`editor/app/channels/[slug]/lib/videoDirFiles.ts`, replacing both pages' `isFile()` loaders): links are listed, a link's time is its `lstat`, the media links' sizes come from ONE `onDrive(mediaDir)` call, and an unreachable one is `size: null` ("— (media drive not reachable)"). The file route chooses the drive by `classifyEntry(name)` and answers **503 with `retry-after: 15`** for a link whose target is not there. A file delete refuses a tiered file whose drive is away (removing the link alone would orphan the bytes). **T1's "left for T2", answered.** - `loadVideoDir` and `videos/page.tsx` filtered `isFile()`: replaced by `loadVideoDirFiles` (above). - The `/storage` rollups counted clips inside media bytes: a location row has none now; the internal row holds every channel's clips and text. - The mover writes `scope: "media"` on every marker. - `relocatedDataDir` stays exported, documented as the retired shape (the migration and the tests build it); nothing moves a channel to it. - The hook's marker re-check before the name changes: verified in `mediaTier-server.ts` (`markerStands` is asked again after the bytes land; a move that began meanwhile leaves the file real). Corpus-wide callers carry no slug, so the hook asks the marker beside the `media` it is writing into — as T1 shipped it; nothing to add. - `archiveLiveChat` read a stale raw replay with no channel guard. Ruling taken here: the build asks the channel's media once (`inspectChannelMedia`, fresh); while it is not `ok`/`in-place` it reads no raw replay and stages the cues already on disk (the freshness check `lstat`s the link), logging how many are older than their replay; a video with no cues yet is left out of that build. Not a refusal: a build never fails over a media drive. **Small edits outside the T2 row** (each needed by the surfaces above): `common/lib/channelMedia.ts` (the `relocatedDataDir` comment), `common/lib/mediaTier-server.ts` and `common/lib/savedVideoStore.ts` (comments naming the retired shape), `common/jobs/jobKinds.ts` (one comment), `common/lib/ storageLocations.ts` (comments and one field doc), `editor/app/channels/lib/mediaBusy.ts` (`mediaOnly`), `editor/app/channels/[slug]/videos/[id]/components/cards/{videoFiles.ts,FilesList.tsx}` (`size: null`), `editor/app/channels/components/{ChannelForm.tsx,ChannelVolumeBar.tsx}`, `editor/app/{page.tsx,channels/ page.tsx}` (`channelVolumeOf(config)`), `editor/app/storage/{page.tsx,actions.ts,components/ StorageLocationsTable.tsx}` (prose, the tier line), `common/controller/archiveLiveChat.ts` (T1's item 4). No helper was added to T1's `mediaTier-server.ts` or `channelMedia.ts`. **Deviations from the plan** (one sentence each): 1. `channelMediaWriters` lives in the mover rather than as T1's `channelWriters(…, { mediaOnly })` alone: `relocate-channel-media` is not `needsMedia`, so `mediaOnly` dropped a running move and the panel would have offered Clear marker, and the preview a second move, over it. 2. The preview tiers only an in-place channel (no `mediaDir`): a relocated channel's tier is on the far drive, and a preview copies nothing there. 3. Locations, the volume column, the watch and `deleteChannel` place a legacy channel by its retired `dataDir`, so `/storage` can count it "(n to migrate)" on the drive it is actually on. 4. The containment check resolves the deepest EXISTING ancestor of the target (it used `realpath` or a lexical path): `//media` usually does not exist yet, so a `` level linked back into the channel dir was caught only by the mirror's direction check, after the preflight had tiered. 5. "The digest lane still picks the channel" is pinned in e2e as a digest JOB that runs on a channel whose media drive is away (the lane's decision is T1's `isChannelHeldForLane`, pinned in its unit tests): the fixture has no digest-lane work without a policy and a model. 6. Two new e2e cases rather than more steps in the main one: the legacy channel (badge, panel refusal, Sync all's skip naming `migrate-tier`) and the digest. **Commits** | Commit | What | |---|---| | `6407b697` | `common:` the mover over `media/` (tier first, `media.*` parked names, `scope`, `mediaDir`, legacy and tier-migration refused), `relocateDir.ts` names, the re-point, rename, `deleteChannel`, the watch and the rollups on `mediaDir`, `/storage`'s tiers and "(n to migrate)", `channelVolumeOf(config)`; the 28 T1 skips rebased | | `c93048a7` | `editor:` the Storage panel's two rows, the badge, Resume/bulk/job wording, `loadVideoDirFiles`, the file route's drive by kind and its 503, the guarded file delete, the Configure line | | `438911c5` | `common:` the live-chat archive reads no raw replay while the media is away | | `e4cab18e` | `common, editor:` `channelMediaWriters` — a move is held by the media writers only; the `/storage` prose | | `f93caa69` | `editor:` e2e rebased on the media tier, two new cases | | `8d9d11f9` | `editor:` the loader's comment names no corpus walker (`noCorpusWalkInRenderPaths`) | | `12339ae2` | `common, editor:` the re-point refuses a legacy channel naming `migrate-tier`; the reconcile spec's changed file differs in length | | `46f6b51e` | merge `main` `0a62bf74` (U2, the deck's finale) — clean | | this commit | `plans:` this section, FACTS "Release 17 slice T2", the editor changelog | #### Gates (logs `$T/T2-*.log`) - **tsc** (all workspaces) clean at every commit and on the merged tree. - **common:** before the merge **2,608 passed, 1 failed, 0 skipped** (2,609) — the failure was this slice's (`noCorpusWalkInRenderPaths`: a comment named the walker), fixed in `8d9d11f9`; on the merged tree **2,610 passed, 0 failed, 0 skipped** (2,610). **The 28 tests T1 skipped all run** (none skipped anywhere): `relocateChannelMedia.test.ts` 13, `renameChannel.test.ts` 1, `storageLocations.test.ts` 7, `storageWatch.test.ts` 7 (T1's record lists 7 there; 28 in all). New: `relocateChannelMedia.test.ts` 40 → 44 (a channel tiered in place moves without tiering; a legacy channel refused out, back and in the preview; a tier-migration marker refused; `channelMediaWriters`) plus the rebased preview case (tiers, idempotent) and back case (a real `media/`, the link untouched, the next move tiers nothing); `storageLocations.test.ts` 14 → 16 (a legacy channel counted unreachable and to migrate with the tier sums; the re-point refuses it); `channels.test.ts` 10 → 11 (a legacy delete); `views/storage.test.ts` 16 → 18; `channelRow.test.ts` rebased. - **Editor unit:** 109/109 (before and after the merge). - **test:scripts:** before the merge 394 passed, 2 skipped (396); on the merged tree 461 passed, 2 skipped (463 — U2 and the deck's tests came with `main`). - **Build:** the capped editor build (`systemd-run --scope -p MemoryMax=6G`): exit 0, 39 s before the merge, 53 s on the merged tree. - **e2e** (from the worktree root, `$T/T2-specs.txt`: channel-storage, storage-locations, channels-storage-columns, bulk-actions, maybe-missing, video-page, fetch-window, channel-rename): run 1 (before the merge) **67 passed, 1 failed, 4.9 min** — the reconcile case's "changed" file had the same size as the source's and was written in the same second, so rsync's quick check called it unchanged (a fixture fault, fixed in `12339ae2`); run 2, `channel-storage.spec.ts` alone, **15 passed, 0 failed, 2.1 min**; run 3 on the merged tree (`46f6b51e` + this record) **68 passed, 0 failed, 3.9 min** (after a few minutes in the queue behind another worktree's suite). - **Privacy gate:** 0 added lines carry the user or host name (`git diff main`, counts only; the one file the whole-file grep names is `plans/FACTS.md`, with the same count as on `main`). No identifier ends in the refused parent suffix. - **Numbers tool:** none. #### Found and left - `HELD_REASON.inconsistent` (`lib/channelMediaHold.ts`, T1's) still says "its data link and its config disagree"; since release 17 it is the media link. Wording only. - A move interrupted DURING its preflight's tiering leaves no marker; the rerun tiers the rest. A file a writer finishes while a marker stands stays real in `data/` (on the corpus disk) until the next hook sweep after the move tiers it onto the far drive — by design (the hook writes nothing under a marker). - `removeVideoDirMedia` still cannot clear `media//` while the drive is unmounted (T1's note); the video page's file delete now refuses that case for one file, and the directory delete already did. - For T3: the migration writes `mediaDir` and unsets `dataDir`; everything in this slice reads a channel with `mediaDir` and no `dataDir` as relocated on the new layout, and a channel with both as legacy. #### Review (SHIP AFTER FIXES) and the fixes The review (`$T/T2-review.md`) found one HIGH, three LOWs and five NITs. Rulings: the live-chat archive publishes the cues on disk while a channel's media is away and never drops the channel; legacy channels stay placed by their retired `dataDir`; the digest job stands for the digest lane in e2e. | Finding | Fix | |---|---| | H1 a legacy channel refused the re-point of its whole location | `22bdf3d7`: the preflight lists a legacy channel (its `data/` a link) with the others and `RepointPreflight.legacy` names it; the job re-points it the retired way in the same ledger and rollback — its `data` link and `dataDir` to `relocatedDataDir(newRoot, slug)` after checking that tree exists — so `dataDir` stays `//data` for T3. A recorded `dataDir` over a real `data/` is refused by name. Tests: one legacy and one `mediaDir` channel on one location, both re-pointed; a legacy tree missing under the new root is named and nothing moves. | | L2 a legacy channel's rename left `//data` | `b9de2318`: refused before anything moves, with the `migrate-tier` sentence. Test. | | L3 Clear marker removed a tier-migration marker | `0e53239a`: `tierMigrationRefusal` is exported and Clear marker, Resume/Reconcile, the preview and the job give its one sentence. | | L4 videos with a raw replay and no cues dropped silently | `ec3a7e59`: counted in the build log ("N video(s) with a raw replay and no cues left out of this build"); the channel is never dropped. | | N5 the job tiered before checking the destination's identity | `0e53239a`: `assertRelocationRootPresent` runs before the preflight tier (and again before the mkdir). | | N6 legacy channels counted "unmeasured" on the corpus row | `250c02fa`: left out of the corpus volume's sums and named "legacy (n to migrate)" in its breakdown. Test. | | N7 the drive was chosen by `classifyEntry === "media"` | `7069e5b8`: a file is on the media tier only when its `lstat` is a link and its name `isTierable` (`source-media.*` never is), in the file route, the file delete and the directory loader. | | N8 a dangling link on an in-place `media/` answered 503 forever | `7069e5b8`: 503 only when the channel's media is relocated (`mediaDir`); otherwise 404, and the file delete removes the dangling link. | | N9 the preview did not refuse a tier-migration marker | `0e53239a`: it does, and tiers nothing. Test. | **Gates after the fixes** (logs `$T/T2-regate.log`, `$T/T2-e2e-4.log`, `$T/T2-regate2.log`, `$T/T2-e2e-5.log`): - Before the merge, at `250c02fa`: tsc clean; the four mover/storage test files (`relocateChannelMedia`, `storageLocations`, `renameChannel`, `storageWatch`) 88/88; common 2,613 passed, 0 failed, 0 skipped; editor unit 109/109; e2e (`channel-storage`, `storage-locations`, `channel-rename`, `video-page`, `$T/T2-specs-review.txt`) **46 passed, 0 failed, 3.5 min**. - **Merge of `main` `ce9ec612`** (slice RL and a plans commit) at `a001f17e`: two conflicts, both appends — `editor/CHANGELOG.md` keeps every `[Unreleased]` bullet, this file keeps U1, U2, XP, D0, T1, RL, then T2 under `## Record`. On the merged tree: tsc clean; **common 2,653 passed, 0 failed, 0 skipped** (2,653); editor unit 109/109; e2e (the same four specs) **46 passed, 0 failed, 3.3 min**. - Not re-run: the capped build and test:scripts (no package, route or script surface changed by the fixes; tsc covers the editor's types). Privacy: 0 added lines carry the user or host name. - From this point a commit's `Co-Authored-By` names the model that wrote it (Opus); the earlier commits are not rewritten. **Re-review (SHIP AFTER FIXES, no further round):** - R1 the file route and the file delete `lstat`ed before asking the text stall → `bb2c259f`: `channelTextStall` first (no I/O), and no `lstat` on a channel with `dataDir` (a legacy channel's `data/` is a link onto the retired drive and holds no tiered links). - NIT the mixed re-point rollback → `37fe4644`: media channel first then the legacy one fails, and the reverse; both restored, no `media` link invented, settings written 0 times (`storageLocations.test.ts` 19/19). - R2 gates at `37fe4644`: tsc clean; **common 2,655 passed, 0 failed, 0 skipped**; capped editor build exit 0, 44 s; e2e (`channels-storage-columns`, `bulk-actions`, `maybe-missing`, `fetch-window`) **22 passed, 0 failed, 1.7 min**. ### Slice T3, as shipped — the one-off migration, and the records (2026-10-02) Branch `r17/media-tier-migrate` off `main` `b6d9c1e2` (U1, U2, XP, D0, T1, RL and T2 merged), worktree `~/Projects/r13-build-image` (editor 5201, test 5211, export 5210), one Opus implementer. Scratch files `T3-*` in the job's `tmp`. The plan is "The one-off migration" above and the T3 row; T1 left the `lutimes` rule, T2 left `relocatedDataDir`, the `//data` invariant and `tierMigrationRefusal`. **What it does.** - **`archilyzer storage migrate-tier …|--all [--order smallest] [--include-large] [--dry-run] [--reclaim]`** (`common/bin/migrate-media-tier.ts`, one `script([...])` row in `archilyzer.ts` beside `migrate channel-priority`) brings a `legacy` channel — `data/` an absolute link to `//data`, `config.dataDir` — onto the media tier: its text is copied home and its big files stay on the drive. - **Editor stopped.** A real run refuses (exit 2, nothing read further) while `/api/pulse` answers — any HTTP answer, or a connection that does not answer within 3 s — or while `.jobs/` holds a `running`/`queued` meta whose `pid` is alive and is not its own (a pid-less `running` meta written after the machine booted counts too; meta files older than the boot are not read). `migrate channel-priority` checked nothing; this is new. A dry run only notes it. - **Per channel**, exactly the plan's phases, each refusing before it writes: the plan reads the corpus disk (a marker whose `scope` is not `tier-migration` is refused naming the Storage panel; `dataDir` must agree with the `data` link and be `//data`; `mediaDir` must be absent); preflight — the old tree is a directory, `assertRelocationRootPresent(root)`, `//media` and `channels//media` absent, one walk classifying every video dir (`inventoryTree`: a tierable REGULAR file stays, everything else is listed and measured as `text | clips | scratch | source`), the space rule `copy − already in data.incoming + margin ≤ free(channelsDir) − minFreeDiskGB` (margin 0 with the gate off, the mover's rule); `copy` — the marker, the NUL list (`channels//.tier-migration.files`), `rsync -a -r --from0 --files-from=… --partial --info=progress2` into `data.incoming/` with the mover's decile progress lines, the verify (the same list's `--dry-run --itemize-changes` empty, a `.d..t` directory-time line not counted, AND per-kind files and bytes equal), then one relative link per tierable file with `lutimes` to its times (EEXIST = already when it is the same link); `swap` — the platter rename, the `media` link, the old `data` link unlinked, `data.incoming → data`, `patchChannelConfig(slug, { mediaDir }, { unset: ["dataDir"] })`, every step `linkOrDirState` first; `reclaim` (only `--reclaim`, or a reclaim marker being resumed) — every entry of `//media//` that is not a tierable regular file goes, an emptied dir goes; done — the marker cleared, the channel must then inspect `ok`, and the bytes by kind, links made and media left on the drive are printed. - **Resumable and idempotent.** A `tier-migration` marker is resumed from its phase (`copy` redoes the copy and verify — rsync sends only what is missing — and finds its links; `swap` and `reclaim` finish); a channel already on the new layout is `already` and writes nothing (with `--reclaim` it reclaims). A dry run writes nothing at all — no marker, list, directory or config — and says what it would do. - **`--all`** = every legacy channel plus any carrying a `tier-migration` marker (resumed first), smallest copy first, with the three big-text channels (`LARGE_TEXT_CHANNELS`) deferred: the run ends with "STOPPED before the big-text channels: …", the corpus disk's free space, each one's copy bytes, and "Continue with `archilyzer storage migrate-tier ` one at a time, or `archilyzer storage migrate-tier --all --include-large`" (exit 0). A real run stops at the first channel it cannot finish (exit 1, "run --all again after fixing it"); a dry run reports every channel and subtracts the space the earlier ones would take. - **`clips/`** is `text` by the classifier (`classifyEntry("clips")`, pinned) and is carried to the corpus disk with the text, its own kind in the counts. **Deviations from the plan** (one sentence each): 1. The copy carries `source-media.*` (media but never tierable) with the text, as kind `source`: the plan's "text + scratch" would have left it on the platter with no link, and `--reclaim` would then delete it. 2. rsync takes `-r` explicitly: `--files-from` turns off `-a`'s recursion, and `clips/` and a scratch dir are directories to carry whole. 3. "Status must be `ok`" is read for the retired layout (which `inspectChannelMedia` answers `legacy`, never `ok`, without touching the drive): `dataDir` and the link agree, have the fixed shape, and the old tree is a directory. 4. A dry run does not refuse a running editor, it notes it: it reads only, and the live dry run may come before the restart. 5. "A running meta newer than the last boot": nothing on disk records the editor's boot, so the rule is a live writer pid (or a pid-less `running` meta after the machine's boot). 6. The continuation after the stop is `--all --include-large` or per slug; `--order smallest` is the only order and `--all`'s default (any other value is a usage error). 7. A reclaim writes its own marker phase (`reclaim`, `scope: "tier-migration"`) so a killed reclaim resumes, and `--all --reclaim` also reclaims every channel already on the new layout (`mediaDir`, no `dataDir`) — one the mover moved has only tierable files there, so nothing is taken. 8. The NUL list lives beside the marker in the channel dir (the tool runs on the operator's machine, with no job scratch dir), and is removed after the swap. No helper was added to `mediaTier-server.ts`; the migration uses `relocatedDataDir`, `relocatedMediaDir`, `tierLinkTarget`, `channelMediaLink`, `linkOrDirState`, `rsyncTree`, `makeProgressSink`, the marker writers, `assertRelocationRootPresent`, `rootOfRelocatedMediaDir` and `patchChannelConfig` as they are. **Records.** `AGENTS.md`: the section is now "A channel's media may live on another drive" (the `media` link, `mediaDir`, `dataDir` retired and `legacy` until `migrate-tier`); "Six things" → seven — the seventh exactly as "Order" above states it, the `.relocating.json` bullet gains `scope`, the `clips/` bullet "on the SSD, never tiered", and the first two bullets name the `media` link and the text guard. `plans/FACTS.md`: "A channel's media is tiered — the model, the hook, the guards, the migration". SETTINGS.md, CHANNEL.md and ENVIRONMENT.md: no schema text changed (`docs files --check`, `settings example --check`, `docs env --check` all clean), so nothing regenerated. The `[Unreleased]` bullet in `editor/CHANGELOG.md`. **Commits** | Commit | What | |---|---| | `38a12ea3` | `common:` `archilyzer storage migrate-tier` — the migration, the editor check, `--all` with the stop, the CLI row; `bin/migrate-media-tier.test.ts` (12) | | `587b0946` | `docs:` AGENTS.md's media section and the seven things; the changelog bullet | | this commit | `plans:` this section, FACTS "A channel's media is tiered" | #### Gates (logs `$T/T3-*.log`) - **tsc** (all workspaces) clean at `38a12ea3` (the later commits change no code). - **common:** **2,667 passed, 0 failed, 0 skipped** (2,655 at T2's tip + this slice's 12). - **The migration test** (`bin/migrate-media-tier.test.ts`, real rsync, a tmp "platter" beside a corpus, a legacy channel with text, `audio.mp3`, `audio.opus`, a raw live chat, `clips/`, `source-media.mp4`, `audio.m4a.part`, `audio.tmp-1234.mp3` and a media-less video): **12/12** — `clips/` is text; a dry run writes nothing (a byte/mtime/link snapshot of the corpus and the platter is unchanged) and counts 3 tierable, 1 clip, 1 source, 2 scratch; a run migrates (every tierable file a relative link with the file's mtime by `lstat`, readable through it, its bytes on the platter; `source-media.*`, `.part` and the temp stay real with their mtimes; `clips/` carried; no marker, list or `data.incoming` left; `inspectChannelMedia` `ok`, text readable) and the rerun writes nothing; `--reclaim` in the same run and later (`already`, 0 taken, nothing written), and after a plain run (dry count = real count); a media move's marker refused with and without a `scope`, nothing touched; a running editor refuses a real run (exit 2, nothing touched) and a dry run notes it; `editorRunningReason` (refused port, an answer, a timeout, a live pid, a dead pid, a pid-less meta, a meta from before the boot); a kill at each of the eight steps from `copied` to `config-written` — the channel reads `in-transition` with its text held and `--all` would pick it — and the rerun resumes from `copy` or `swap` and ends migrated; the free-space stop (6 GB free, a 5 GB floor and a 2 GB margin: refused, nothing written; 8 GB: migrated); `--all` takes the smaller channel first and stops before `omnibased` with the free space and `--all --include-large`, which then migrates it; and an index built over the classic layout, the channel put on the retired layout and migrated: the live chat's cues read fresh and the next `buildIndex` reports **0 added, 0 changed, 0 removed**, none held. - **Editor unit:** 109/109 (no editor code touched). **test:scripts:** not run — no file it covers (`scripts/`, umtool) was touched. **Capped editor build:** not needed — no editor code touched. - **CLI smoke** against a scratch corpus in `$T`: usage via `archilyzer storage migrate-tier --help`; `--all --dry-run` notes the running editor and finds no channel; `--all` refuses (exit 2) naming the editor's `/api/pulse` answer; no slug and no `--all`, and `--order biggest`, are usage errors (exit 2). The probe's one GET reached the live editor's `/api/pulse` (read-only); nothing else was pointed at the primary checkout. - **e2e:** none named for this slice. - **Privacy gate:** 0 added lines carry the user or host name (`git diff main`, counts only; the one file the whole-file grep names is `plans/FACTS.md`, with the same count as on `main`). No identifier ends in the refused parent suffix. - **Numbers tool:** none. The live dry run is the parent's. #### Found and left - `lib/envVars.ts`'s `ARCHILYZER_EDITOR_URL.readBy` and CHANNEL.md's `dataDir` row were stale; both fixed in the review round (below). - The dry run (and `--all`'s ordering) walks every legacy channel's old tree on its drive, one `lstat` per entry, and the stop walks the big three to print what each would copy; a real run walks a channel again in its own preflight. Read-only, but minutes on a platter. (Since the review the stop no longer walks the big three a second time.) - The space rule counts `clips/` (nuxanor-kick's 15 GB) as text, as the plan measured it. #### Operator notes - **`purge-superseded-auto-subs` runs AFTER the migration, not before** (the plan said before): since T1 the kind is `needsText` and a `legacy` channel's text is refused, so the editor refuses the purge on omnibased and HasanAbiVODs3 until they are migrated. omnibased's ≈ 7.6 GB of `en-orig.vtt` lands on the corpus disk first and is purged there. - **`transcripts/channels` is a Syncthing folder (paused).** Syncthing syncs a symlink as a link; after the migration the 13 channels' text (≈ 64 GB with clips) is a real tree inside that folder, where today it is behind a `data` link. Nothing moves while the folder is paused; unpausing it would send that text to the peers. - `--reclaim` after the editor has run on the migrated channels for a while: until then the drive's text copy is a second copy. #### Review (SHIP AFTER FIXES) and the fixes The review (`$T/T3-review.md`) found four LOWs and five NITs. Rulings: `--all --reclaim` is limited to channels this tool migrated; a reclaim deletes only what has a same-size twin on the corpus disk; dead `*.temp.*` stay out of the copy; a pulse that does not answer within 3 s counts as a running editor (as built). | Finding | Fix | |---|---| | L1 `--all --reclaim` walked every channel on the media tier | `beaa6a44`: the swap leaves `channels//.tier-migration.reclaim` (`RECLAIM_NOTE`); `--all --reclaim` takes only channels carrying it, a named `--reclaim` without it is refused ("migrate-tier did not migrate it, or its reclaim is done"), a reclaim removes it. Tests: a Storage-panel-moved channel is not walked by `--all --reclaim` and is refused by name; a second `--reclaim` refuses and touches nothing; a reclaim killed after its deletes resumes from its marker and removes the note. | | L2 a later `--reclaim` deleted without asking whether the corpus disk still has the file | `beaa6a44`: an entry goes only when one `lstat` finds its twin at `channels//data//` (a directory for a directory, else the same size); everything else is kept and listed (`reclaimKept`, and a "kept N entries" line). Test: a removed and a rewritten text file are kept on the platter, the rest taken. | | L3 the purge cannot run first | Operator note above and in the changelog bullet. | | L4 dead `*.temp.*` carried to the corpus disk (15 GB on nuxanor-kick) | `beaa6a44`: `isLeftOnPlatter` (scratch by the classifier AND `*.temp.*`) is neither copied nor linked; it stays on the platter and `--reclaim` deletes it; the walk reports it ("dead postprocessor temps left there unlinked"). `96191600`: `/^\.syncthing\..*\.tmp$/` joins the classifier's scratch patterns (+ the table row). Tests: the fixture carries `source-media.temp.mp4` (left, then reclaimed) and `.syncthing.audio.mp3.tmp` (scratch, carried). | | NIT AGENTS.md's corpus table | `e01db98a`: "a big file in `data//` may be a relative link into `media/`, which may be an absolute SYMLINK to another drive (and a legacy channel's whole `data/` is one, until migrated)". | | NIT `ARCHILYZER_EDITOR_URL.readBy` | `e01db98a`: names `common/bin/migrate-media-tier.ts`; ENVIRONMENT.md regenerated. | | NIT the `dataDir` field doc | `e01db98a`: "Written only by the re-point of a storage location …; removed by that migration" (`lib/channelConfig.ts`); CHANNEL.md regenerated. | | NIT the stop re-walked the big three | `beaa6a44`: the stop reports the sizes the ordering walk measured (the big three are measured once, in the ordering pass, whether or not `--include-large`). | | NIT apparent bytes vs block slack | Left: the 2 GB margin covers it. | **Gates after the fixes** (logs `$T/T3-test4.log`, `$T/T3-tsc2.log`, `$T/T3-common2.log`): tsc (all workspaces) clean; the migration test **15/15** (12 + 3 new); the classifier test 37/37; `docs files --check`, `docs env --check`, `settings example --check` clean after the regeneration; **common 2,671 passed, 0 failed, 0 skipped** (2,667 + the three new migration cases + the classifier's Syncthing row). Not re-run: editor unit, test:scripts, the editor build (no editor, script or umtool file touched). Privacy: 0 added lines carry the user or host name. **Re-review (2026-10-02), one sentence the record left unsaid:** `--reclaim`'s twin check accepts a same-size file with different bytes as a twin — an accepted risk, since the migration's rsync verified the copy and a later same-size rewrite on the SSD is the editor's newer version — and matches a directory by name only (`clips/` is a cache; scratch dirs). ## Rollout `main` `a6155bf2` = all eight slices (D0, T1, T2, T3, U1, U2, XP, RL) + the deck session's two branches. Steps below in order; each live result is written back here as it lands. Scripts: `~/reports/release-17/scripts/r17-{build,restart,smoke,umtool-build,umtool-restart}.sh` (release 15's shapes; `MERGE` re-stamped to T3's merge). Logs in `~/reports/release-17/tmp/`. 0. Pre: `main` contains T3's merge; the tree is clean; no `archilyzer run` in flight (D0: its no-pid meta would be closed as interrupted at the first boot). 1. Editor: `r17-build.sh` (capped) → `r17-restart.sh --force` → `r17-smoke.sh`. First boot: the running metas a dead process left are closed as interrupted; the auto-queue state gains `platformPace`, `platformHolds`, `subtitleDeferrals`; YouTube's backoff is at `fails: 80`, so the first real platform failure after boot HOLDS YouTube at once — "Clear hold" on the download lane page is the way out; `/` and `/jobs` must answer while the scheduler re-runs the two big snapshots (D0's proof). 2. umtool: create `/umtool` (the media root is never created by umtool); `r17-umtool-build.sh`; restart with `UMTOOL_MEDIA_DIR=/umtool` in the environment (operator: the 0.0.0.0 bind); `umtool index` once (the cache moved to `~/.cache/archilyzer/umtool`); `umtool doctor` (roots + the leftover 8.7 MB old cache); `umtool storage move-out --all --dry-run` then `move-out --all` while nothing builds (≈ 10.6 GB, 10 projects). 3. Migration (editor STOPPED): `archilyzer storage migrate-tier realcandaceo --dry-run` → run → `nuxanor-kick` → `--all --order smallest` (stops before omnibased, rekietalaw, the-quartering-rumble and prints the free space); start the editor; live proof per migrated channel: Refresh report, one video page, one short Transcribe (the hook on a relocated channel), `/storage`, `df`. The big three: the operator decides. `purge-superseded-auto-subs` cannot run on a legacy channel (held since T1), so it runs AFTER each migration — omnibased's 7.6 GB of `en-orig.vtt` lands on the SSD first and is purged then. `*.temp.*` postprocessor leftovers (15 GB on nuxanor-kick) are not copied; `--reclaim` deletes them from the platter later. Projection: all 13 channels copy ≈ 50 GB of text to `/home` (99 GB free; floor 5, margin 2). `transcripts/channels` is a Syncthing folder, currently PAUSED: after migration the 13 channels' text is a real tree inside it, so unpausing would sync it to the peers — the operator decides before unpausing. `--reclaim` later to drop the platter's text copies. 4. X posts private (XP): in the new editor — a private site (id `research`, audience private, every channel), Settings → X account session → "Where X posts appear" = private; rebuild + deploy hasanalyzer, anilyzer, jeralyzer (operator); REBUILD + DEPLOY THE HUB (it serves X posts today; a hub bundle built before this release is refused by Deploy hub); old Cloudflare deployments/preview aliases still serve X posts — the operator decides whether to delete them (dashboard: Workers & Pages → project → Deployments → ⋯ → Delete; or wrangler pages deployment list/delete); compose the private site to its own dir and re-register the MCP: `claude mcp remove archilyzer -s local` then `claude mcp add archilyzer --env ARCHILYZER_EDITOR_URL=http://localhost:3001 --env WORKER_TOKEN=… -- pnpm --silent -C "$PWD" archilyzer mcp --local `. 5. Records: STATE.md, FACTS (done by slices), memory, the HTML runbook (`~/reports/release-17/RUNBOOK.html`). ### As it went (2026-10-02, 02:30–03:20) | step | result | |---|---| | editor build (`r17-build.sh`, capped) | OK in 40 s, `Xfavvt2XajdNzOvvn0NtL` → `Dt4zuo5uJcPKZ9fKk0L-f` | | editor restart 02:33 | 200 in 3 s; boot closed 8 ghost metas (6 refresh-report, 1 auto-transcribe, 1 auto-download); smoke `SMOKE_FAIL=0` | | umtool build + `test:scripts` | OK; 462 passed, 0 failed, 1 skipped | | umtool restart 02:40 with `UMTOOL_MEDIA_DIR=/umtool` | 200 in 2 s; `umtool index` run; `umtool doctor` reports both roots and the 8.7 MB old cache | | `umtool storage move-out --all` | 10 projects, 12.3 GB moved to the media root | | dry run `migrate-tier realcandaceo` (editor up) | 21.7 MB text, 435 links to make | | migration window (`r17-migrate.sh`, editor stopped 02:40–02:51) | realcandaceo in 30 s; `--all` migrated omnimirror, leaflit-rumble, cornbreadman, shondo-vods, friday-night-tights, piratesoftware, kirsche, nuxanor-kick, HasanAbiVODs3 in 657 s and STOPPED before rekietalaw, the-quartering-rumble, omnibased; `/home` 99 → 81 GB free; editor restarted 02:51 | | live proof | a migrated video page 0.19 s; its `audio.mp3` and `transcript.json` stream through the links (206); `/storage` says "legacy (3 to migrate)"; a migrated snapshot carries `totalTextBytes`/`totalMediaBytes`; three report regenerations at once (incl. the-quartering) while `/` and `/jobs` answered in 1.4–2.1 s; links counted on every migrated channel, none dangling | | the download lane after boot | YouTube backoff `fails: 6` at 03:18, no hold, no subtitle deferrals | A regeneration of omnibased was refused before the migration with the `migrate-tier` sentence: the three big-text channels stay held until the operator migrates them. The operator's steps (the big three, X posts private incl. the hub rebuild and deploy, the MCP line, the Syncthing folder) are in `~/reports/release-17/RUNBOOK.html`.