Archilyzer · Source

archilyzer

Archilyzer
git clone https://archilyzer.pages.dev/source/archilyzer.git
Log | Files | Refs | README | LICENSE

commit 1939ef917c4888eb13c68427ca6fcc4834819256
parent 7f7d14015a7acf7432aec0d1d9784b9f5d35d933
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date:   Fri, 11 Sep 2026 11:00:02 -0400

plans: relocate-channel-media refreshed at 7f7d140 and placed first, ahead of one-core phase 2

/home is at 100 % (6.9 G free); sda1/sdb1 platters are present and unmounted. The 2026-08-24
design (symlinked data/, config.dataDir as the record, a marker, guards, an rsync job) stands;
every anchor is re-verified at 7f7d140: 74 inline "data" joins across 53 files, four guard
sites (streamCommand via needsMedia on JobKindMeta, autoRunner buildChannelWork, the snapshot
swallow, the operationBatch swallow) plus the shardActions bypass. New: a settings-level cold
root and a bulk move from /channels (slice 3), and rollout Step 0 — the 130 GB saved-video
store moves today by a whole-directory symlink, because every pointer's dir is absolute and
nothing walks the store root.

Roadmap: STATE.md "Next" puts this plan first; one-core.md gains an Interlude between Phase 1
and Phase 2 and a note that ArchiveReader should read config.dataDir when it exists.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Diffstat:
Mplans/STATE.md | 6+++++-
Mplans/one-core.md | 12++++++++++++
Mplans/relocate-channel-media.md | 509++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------------
3 files changed, 376 insertions(+), 151 deletions(-)

diff --git a/plans/STATE.md b/plans/STATE.md @@ -155,7 +155,11 @@ assertion, and it was right: writing the two new snapshot entries doubled `/chan and Transcribe coverage, because that page passes the external ids into `buildOperationBands` and `addRegistryEntry` folded them on top of `addExternalBands`. Fixed in `d8754d2`. -**Next:** phase 2 (the contract: one `ArchiveReader`), 3 slices. Read +**Next:** [`relocate-channel-media.md`](relocate-channel-media.md) FIRST, 3 slices — +`/home` is at **100 %, 6.9 G free**, and the mechanism (a symlinked `data/` plus a +`config.dataDir` record and four guards) is orthogonal to phase 2, so it neither waits on +nor complicates the contract work. Then phase 2 (the contract: one `ArchiveReader`), 3 +slices. Read `common/architecture.test.ts`'s allow-list first — it is the shortest accurate statement of what is still tangled, it shrank by one across phase 1, and no slice added an entry. diff --git a/plans/one-core.md b/plans/one-core.md @@ -240,6 +240,16 @@ planners, ten settings fields with their sanitizers and forms, and it is the loop everything else folds into. Kept verbatim: the seven "must not be lost" invariants. +### Interlude — relocate channel media (between Phase 1 and Phase 2) + +Not a phase of this plan, but it runs before Phase 2 and its design is in +[`relocate-channel-media.md`](relocate-channel-media.md). `/home` is full (100 %, 6.9 G free) +and a channel's `data/` needs to be able to live on another drive. The mechanism is a +symlinked `channels/<slug>/data`, a `config.dataDir` record written only by the job, and four +guards so an unmounted drive never reads as "nothing downloaded". It touches dispatch +(`streamCommand`, `autoRunner`, `operationBatch`) but changes no dispatch semantics, and it is +independent of the contract work below. + ### Phase 2 — The contract: one `ArchiveReader` (3 slices) 1. **`common/lib/archive/`**: `contract.ts` (re-exports the `CONTRACT` object from phase 0 @@ -248,6 +258,8 @@ invariants. `shard(layer, slug, n)`, `record(layer, slug, id)` — and the HTTP transport, browser-safe), `reader-fs.ts` (node), `reader-hub.ts` (federation). The implementation is `mcp/src/source.ts` moved, caches included; mcp's `ShardSource` becomes a re-export. + `reader-fs.ts` should take `config.dataDir` into account where it exists — that is option A + of the interlude's plan, a resolver layered on top of the symlink rather than instead of it. 2. **Every reader uses it.** `common/components/{transcript,subs,posts,digest}Cache.ts` become thin memo wrappers over the HTTP reader; `export/app/lib/offlineCache.ts` keys its cache off `contract.ts` paths (this also fixes the open `duplicates.json` offline gap); diff --git a/plans/relocate-channel-media.md b/plans/relocate-channel-media.md @@ -1,69 +1,77 @@ # Relocate a channel's media to another drive -**Ask:** designate a channel's downloaded videos (`transcripts/channels/<slug>/data/`) to live -in another directory or drive, so a disk-heavy, mostly-redundant channel like `omnimirror` -(131 GB of 396 GB in `channels/`, 718 mp3s) moves off the SSD onto the larger platter drive. +**Ask (widened 2026-09-11):** hold large amounts of data on other hard drives, especially +channels that are not being prioritized. The unit is a channel's downloaded media +(`transcripts/channels/<slug>/data/`), which moves to another directory or drive while every +reader keeps working. -**Box today:** `/home` (nvme, 1.5 T) is at **94 %, 87 G free**. The platters `sda1` (466 G, -ext4) and `sdb1` (1.8 T, ext4) are present but **not mounted** (`lsblk`). Mounting one is a -prerequisite of the rollout, not of the code. +**Box today (measured 2026-09-11):** `/home` (nvme, 1.5 T, holds `transcripts/`) is at +**100 %, 6.9 G free**. `transcripts/` is **596.8 GB** — `channels/` 448.7 GB, +`saved-videos/` 130.5 GB, `index.mdb` 13 GB. The platters `sda1` (466 G, ext4) and `sdb1` +(1.8 T, ext4) are present and **still unmounted** (`lsblk`). Mounting one is a prerequisite +of the rollout, not of the code. + +**Branch:** this work lands on a new branch **`storage/relocate-media` off `61eae05`** +(`one-core/phase-1` is unmerged and this builds on it). **No slice moves any data.** The +rollout is the operator's, through the UI, after the platter is mounted. --- -## What the tree says (verified 2026-08-24) +## What the tree says (re-verified 2026-09-11 at `61eae05`) + +The 2026-08-24 survey was verified at `46ade6d`. Everything below has been re-checked; every +line number that moved is corrected here. -| Fact | Anchor | +| Fact | Anchor at `61eae05` | |---|---| -| There is **no** `channelDir()`/`videoDir()`/`dataDir()` helper. `"data"` is joined inline at **~74 non-test call sites**, most with only `(paths, slug)` in scope — no config. | `common/ytdlp/runYtdlp.ts:563,1048,1331,1445`, `downloadOneManaged.ts:435…`, `controller/channelSnapshot.ts:555`, `channels.ts:156,175,230`, `buildIndex.ts:300,1113`, `autoRunner.ts:936`, `editor/app/channels/[slug]/videos/page.tsx:137`, … | -| yt-dlp output templates are **relative** (`-o data/%(id)s/…`) and the download dir is set purely by `cwd: channelDir`. No `-P`/`--paths`. | `runYtdlp.ts:256-262, 339, 1653`; `downloadOneManaged.ts:312` | -| `archive`, `playlist`, `config.json`, `snapshot.json`, `roster.json`, `failed-transcriptions` all live in the channel dir, **not** in `data/`. The archive is appended by the app, not by `--download-archive`. | `downloadOneManaged.ts:417` | -| **Zero** symlink-aware code (`lstat`/`readlink`/`realpath`/`symlink`) in `common/` and `editor/`. Every reader goes through `path.join(channelDir, "data", …)` and would follow a symlink transparently. | grep | -| Channel listing filters `isDirectory()` on `channelsDir` entries — a symlinked **channel dir** would vanish. Only `data/` may be a link. | `channels.ts:285`, `buildIndex.ts:284`, `site.ts:380` | -| Snapshot swallows a missing `data/` as "no videos": `readdir(dataDir).catch(() => [])`. An **unmounted** drive would therefore read as *everything undownloaded*. | `channelSnapshot.ts:576` | -| `deleteChannel` is `rm(dir, {recursive})` — on a symlinked `data/` that removes the **link**, orphaning the media on the other drive. | `channels.ts:398-405` | -| `renameChannel` renames the channel dir (a symlink inside moves fine) and already has an EXDEV message for the saved-video store. | `renameChannel.ts:80-97` | -| LMDB index stores ids + mtimes, **no paths**. A move that preserves mtimes (`rsync -a`) needs no reindex. | `buildIndex.ts:181,462` | -| Low-disk gate is hard-wired to `paths.transcriptsDir`; `checkDiskSpaceFor(dir)` exists but nobody passes a dir; the hysteresis latch is one global. | `common/lib/diskSpace.ts:57-61, 207` | -| `export/` never reads `data/`; `buildIndex` reads captions/metadata only. | agent survey §7 | -| **Precedent for exactly this shape:** the saved-video store — `ChannelConfig.savedVideosDir` per-channel override, `savedVideoRoot()`/`savedVideoDir()` resolver, `moveFileCrossDevice()` (rename → EXDEV → copy+atomic rename), rsync mirror job with `execa` + `cancelSignal`, real-rsync unit tests. | `channelConfig.ts:96-101`, `savedVideo.ts:52-61`, `savedVideo-server.ts:66-82`, `controller/backupSavedVideos.ts`, `backupSavedVideos.test.ts:30` | -| Job launch template: `runManagedFunction({kind, queueKey, paths, fn(onLog, signal)})`; a kind is one entry in `JOB_KINDS`. | `editor/app/saved-videos/backupActions.ts`, `common/jobs/jobKinds.ts` | -| Rename refuses while the channel has running/queued jobs — the guard to copy. | `editor/app/channels/actions.ts:463-473` | -| `/cleanup` and channel stats are per-file `stat` through the data path — unaffected by a link. | `channelSnapshot.ts:647`, `editor/app/cleanup/lib/loadCleanup.ts:31` | +| There is still **no** `channelDir()`/`videoDir()`/`dataDir()` helper. `"data"` is joined inline at **exactly 74 non-test call sites across 53 files** — 57 in `common/`, 14 in `editor/`, 3 in `umtool/`, 0 in `export/` and `mcp/`. Most sites have only `(paths, slug)` in scope, no config. | `runYtdlp.ts:563,1048,1331,1445`; `downloadOneManaged.ts:435,514,650,866`; `channelSnapshot.ts:602`; `channels.ts:177,196,251`; `buildIndex.ts:298,1111`; `autoRunner.ts:1422`; `operationBatch.ts:958,1580`; … | +| yt-dlp output templates are still **relative** (`-o data/%(id)s/…`) and the download dir is set purely by `cwd`. No `-P`/`--paths`. | `runYtdlp.ts:256-262` (`OUTPUT_ARGS`), `:339` (`cwd: root`), `:1648` (`runChildAndStream(cwd)`); `downloadOneManaged.ts:307` | +| `archive`, `playlist`, `config.json`, `snapshot.json`, `roster.json`, `failed-transcriptions` all live in the channel dir, **not** in `data/`. The archive is appended by the app. | `downloadOneManaged.ts:417` | +| **Zero** symlink-aware code (`lstat`/`readlink`/`realpath`/`symlink`) in `common/`, `editor/` or `export/`. Re-confirmed at `61eae05`: the grep returns nothing. Every reader goes through `path.join(channelDir, "data", …)` and follows a symlink transparently. | grep | +| Channel listing filters `isDirectory()` on `channelsDir` entries — a symlinked **channel dir** would vanish. Only `data/` may be a link. | `channels.ts:246`, `channels.ts:363` (`listChannelSlugs`), `buildIndex.ts:282`, `common/lib/site.ts:380` | +| Snapshot swallows a missing `data/` as "no videos": `readdir(dataDir).catch(() => [])`. An unmounted drive reads as *everything undownloaded*. | `channelSnapshot.ts:602` (join), `:623` (the swallow) | +| **`runOperationBatch` has the same swallow**, and it is new since the last survey: `readdir(dataDir).catch(() => [])` decides the candidate list for both operation lanes. | `operationBatch.ts:1578-1580` | +| `deleteChannel` is `rm(dir, {recursive})` — on a symlinked `data/` that removes the **link**, orphaning the media. | `channels.ts:476-482` | +| `renameChannel` renames the channel dir (a symlink inside moves fine) and already has an EXDEV message plus a rollback for the saved-video store. | `renameChannel.ts:80` (rename), `:92-95` (store move, rollback, EXDEV) | +| Rename refuses while the channel has running/queued jobs — the guard to copy. **Moved** from `actions.ts:463-473`. | `editor/app/channels/actions.ts:541-552` | +| LMDB index stores ids + mtimes, **no paths**. A move preserving mtimes (`rsync -a`) needs no reindex. | `buildIndex.ts:460-461` (`mtimes` sub-DB), `:566,796` | +| Low-disk gate is still hard-wired to `paths.transcriptsDir` and the hysteresis latch is still **one global boolean**. `checkDiskSpaceFor(dir, settings)` and `getFreeBytes(dir)` exist and take a dir; `diskGate` does not. | `diskSpace.ts:13,37,57-61,181,207,225,238,244` | +| `diskGate` callers (**all moved or new** since the last survey). | `runYtdlp.ts:876`; `autoRunner.ts:1059` (was `:654`); `backfillReacquire.ts:166` (was `:79`); `persistKept.ts:106`; `pipelineActions.ts:49`; `persistActions.ts:32`; `fixIncompleteTranscript.ts:78`; `videoActions.ts:266,301`; **new:** `editor/app/jobs/active/buildActiveJobs.ts:347` (observe). `downloadOneManaged.ts` is **not** a caller — the old table implied it was. | +| `/cleanup` is now **entirely snapshot-driven** — zero disk reads — so it is a downstream consumer of the snapshot guard, not a site of its own. The old anchor `editor/app/cleanup/lib/loadCleanup.ts:31` no longer exists. | `editor/app/cleanup/lib/loadCleanup.ts:10-41` | +| `export/` never reads `data/`; `buildIndex` reads captions/metadata only. | still true | +| **Precedent for exactly this shape:** the saved-video store — per-channel override, resolver, rsync mirror job with `execa` + `cancelSignal`, real-rsync unit tests against `paths.rsyncBin`. | `channelConfig.ts:97-101`; `savedVideo.ts:52-67`; `savedVideo-server.ts:55-82`; `backupSavedVideos.ts:76-90`; `backupSavedVideos.test.ts:29-40`; `paths.ts:117,226` | +| Job launch template: `runManagedFunction({kind, queueKey, paths, channelSlug?, videoId?, fn})`; a kind is one entry in `JOB_KINDS` with shape `{kind, label?, drainable, replayable, queueKeyStrategy, defaultTier?}`. 40 kinds today. | `common/jobs/streamCommand.ts:264`, `:97-104` (opts), `:110-127` (`makeJob`); `common/jobs/jobKinds.ts:22-40` | +| **`architecture.test.ts` forbids `lib -> controller`, `lib -> jobs`, `jobs -> controller`, `components -> {controller,jobs,ytdlp}`. The allow-list can only SHRINK — a new entry fails the build.** | `common/architecture.test.ts:31-35` (`FORBIDDEN`), `:24-26` (the rule) | +| `common/package.json` exports `"./lib/*": "./lib/*.ts"` and `"./controller/*"` by wildcard — **a new module needs no exports-map edit.** | `common/package.json:34-36` | +| `ChannelForm.tsx`'s "Saved-video store dir" field is **unmoved**. | `editor/app/channels/components/ChannelForm.tsx:641-647` | +| The channel page renders stage components under `[slug]/components/stages/` with `[slug]/components/flow/` for the overview; `ChannelFormClient`, `RenameChannelForm`, `DeleteChannelForm` are composed in `page.tsx`. The old flat `components/` layout is gone. | `editor/app/channels/[slug]/page.tsx:53-67` | --- ## Decision: symlink is the mechanism, `config.json` is the record, a guard makes it safe -Two ways to do this: - -**A. Resolver migration** — add `ChannelConfig.dataDir` and a `channelDataDir(paths, slug, config)` -helper, then rewrite all ~74 call sites (and add `-P`/absolute `-o` for yt-dlp). Correct in -principle, but most sites don't have the config in hand, the runners touch production data -unattended, and **one missed site silently reads an empty dir** — the auto-runner would then -"re-download" 131 GB onto the SSD. High blast radius for a feature that moves bytes. - -**B. Symlink** — the job copies `data/` to the other drive, verifies, swaps in -`channels/<slug>/data -> <root>/<slug>/data`, and records the target in `config.json`. -Every existing reader, yt-dlp's `cwd`-relative writes, the LMDB index and the export build -keep working with **no call-site changes**, because the on-disk contract -(`channelDir/data/<id>/…`) is preserved. The known hazards are three and are each addressed -below: an unmounted drive (guard), `deleteChannel` orphaning the target (fix), and the disk -gate measuring the wrong volume (thread the dir). +**Unchanged, and not reopened.** **A. Resolver migration** (a `channelDataDir()` helper plus a +rewrite of all 74 call sites and absolute `-o` for yt-dlp) is correct in principle, but most +sites don't have the config in hand, the runners touch production data unattended, and **one +missed site silently reads an empty dir** — the auto-runner would then "re-download" 130 GB +onto a full SSD. **B. Symlink** — the job copies `data/` out, verifies, swaps in +`channels/<slug>/data -> <root>/<slug>/data` and records the target in `config.json` — keeps +every reader, yt-dlp's `cwd`-relative writes, the LMDB index and the export build working with +**no call-site changes**, because the on-disk contract (`channelDir/data/<id>/…`) is preserved. -**Go with B.** It is also exactly how an operator would do it by hand today, minus the -verification, the guard and the UI. A can be layered on later if a second mechanism is ever -needed (a networked store, say); the config field designed here is the same one A would use. +**Go with B.** One-core **Phase 2's `ArchiveReader`** (`plans/one-core.md` §Phase 2) is where +a resolver would later live, and `config.dataDir` — the field designed here — is exactly the +field it would read. Option A is a layer on top of B, not a replacement for it. ### Layout - Target: `<root>/<slug>/data/` — `<root>` is any directory the operator picks (e.g. `/mnt/platter/archilyzer-media`); the `<slug>/data` suffix is fixed so one root can - hold many channels and the shape mirrors the saved-video store (`<root>/<slug>/<videoId>`). + hold many channels and the shape mirrors the saved-video store. - `channels/<slug>/data` becomes an **absolute** symlink to that target. - `config.json` gains `dataDir: "<root>/<slug>/data"` (the link target, stored resolved). - Blank/absent = in place. Written **only by the relocate job on success** (same rule as the - sweep console: the record is the output of the action, never a free-text field that can - drift from disk). + Blank/absent = in place. Written **only by the relocate job on success** — never a + free-text field that can drift from disk. - Marker while a move is in flight: `channels/<slug>/.relocating.json` `{ target, direction: "out"|"back", startedAt, phase }`. Removed on completion. Its presence means "media is in transition" to every guard below, and lets an interrupted job resume. @@ -82,153 +90,354 @@ export type ChannelMediaLocation = { status: "in-place" | "ok" | "unreachable" | "in-transition" | "inconsistent"; detail?: string; // "target /mnt/… does not exist (drive not mounted?)", … }; -export async function inspectChannelMedia(paths, slug, config): Promise<ChannelMediaLocation> -export async function assertChannelMediaReachable(paths, slug, config): Promise<void> // throws ChannelMediaUnreachableError +export async function inspectChannelMedia(paths, slug, config?): Promise<ChannelMediaLocation> +export async function assertChannelMediaReachable(paths, slug, config?): Promise<void> export function relocatedDataDir(root: string, slug: string): string // <root>/<slug>/data +export class ChannelMediaUnreachableError extends Error {} ``` `inspect` = `lstat(data)` (is it a link? does that agree with `config.dataDir`?) + `stat(target)` (exists, is a directory) + `.relocating.json` presence. Because the link points at a *deep* -path, an unmounted root gives `ENOENT` — an empty mountpoint dir is not mistaken for the -media. Pure and cheap: two stats. Unit-tested with tmp dirs. - -### 2. Guard the readers that would otherwise see "nothing downloaded" - -- `channelSnapshot.ts:576` — before the `readdir(dataDir)`, if `config.dataDir` is set and - `inspect().status !== "ok"`, **throw `ChannelMediaUnreachableError`** instead of computing a - snapshot over an empty dir. The scheduler already treats a failed refresh as "keep the last - snapshot.json" (`snapshotScheduler.ts:97,205`); verify that path logs the reason. -- `autoRunner.ts` — skip a channel whose media is not `ok` (log once per state change), so - unattended download/transcribe/clean never runs against a dangling link. -- `undownloadedVideos.ts:28` and the per-channel manual actions - (`pipelineActions.ts`, `whisperActions.ts`, `persistActions.ts`, `bulkVideoActions.ts`, - `videoActions.ts`) — call `assertChannelMediaReachable` at the top; the error becomes the - action's `{ ok: false, error }`. -- Channel page + dashboard: a banner/badge from `inspect()` — "Media relocated to `<root>` - (131 GB) · reachable" / "**unreachable** — mount the drive". Cheap, render-safe (two stats). +path, an unmounted root gives `ENOENT` — an empty mountpoint is not mistaken for the media. +Two stats; render-safe. + +**It lives in `lib/` and so may not import `controller/` (`architecture.test.ts:31-35`), which +is where `readChannelConfig` is (`channels.ts:148`).** The `config` argument is therefore +optional, and when it is absent `channelMedia.ts` reads `<channelDir>/config.json` itself and +pulls only `dataDir` out of it — a four-line JSON read, no controller import, no allow-list +entry. Callers that already hold a config pass it and pay nothing. **Do not add an entry to +`ALLOWED` to get green; the test fails on additions by design.** + +Unit-tested with tmp dirs. + +### 2. Guards — the post-Phase-1 dispatch makes this four sites, not a file list + +The 2026-08-24 plan listed six action files to patch individually. Phase 1 changed the +dispatch, and the enumeration below is from reading the code, not from assumption. + +**What reaches a channel's media, and through what:** + +- **`runManagedFunction` (`common/jobs/streamCommand.ts:264`) is the funnel for every + job-shaped action.** 53 call sites; **35 already pass `channelSlug`** — `whisperActions` 10, + `videoActions` 6, `availabilityActions` 3, `pipelineActions`/`socialActions`/ + `incompleteTranscriptActions`/`operationJobs`/`workerServer` 2 each, `digestActions`/ + `normalizeActions`/`persistActions`/`channels/actions.ts`/`snapshotScheduler`/`autoRunner` + (the download unit) 1 each. The 18 without a slug are the lane runners, `/review`, + saved-video backup and the site build/deploy — none of them per-channel media work. +- **`runOperationBatch` (`operationBatch.ts:1544`) is per-channel** and swallows a missing dir + at `:1580`. Callers: `operationJobs.ts:103,236` and `digestActions.ts:118` — all three + already inside a `runManagedFunction`. +- **`runOperationUnit` (`operationBatch.ts:952`)** is reached from `runOperationBatch:1770` + **and directly from `autoRunner.ts:1262`** — so the digest and backfill **lane runners do + not go through `runOperationBatch`**. That is the one path a batch-level guard misses. +- **The four lane runners** are one cross-channel job each (`autoRunner.ts:1597`, + `queueKey: ""`, **no `channelSlug`**). They enumerate at `listChannelMeta` (`:288`), project + work at `buildChannelWork` (`:299`), and launch transcription/download units at `launchUnit` + (`:1431`). +- **`bulkVideoActions.ts` bypasses nothing** — all ten of its actions delegate to + `whisperActions` / `pipelineActions` / `incompleteTranscriptActions` / `videoActions` + (`:12-23`), every one a `runManagedFunction` site. **`shardActions.ts:87` is the one true + bypass**: it calls `runYtdlp` directly, with no job record. + +**So: four guard sites.** + +1. **`common/jobs/streamCommand.ts:264`** — in `runManagedFunction`, when `opts.channelSlug` + is set **and** the kind declares it touches media, `assertChannelMediaReachable` before the + job record is made; on failure return `{ ok: false, error }` without enqueuing. The + declaration is a new **`needsMedia?: boolean`** on `JobKindMeta` (`jobKinds.ts:22-40`) so a + bookkeeping kind (normalize, availability check, store playlist, clear markers) is not + refused for a drive it never reads. `jobs/` may import `lib/` — no back-edge. +2. **`common/controller/autoRunner.ts:299`** — in `buildChannelWork`, skip a channel whose + media is not `ok`, for all four lanes at once, with a **once-per-state-change** log line. + This is the runners' only channel-level chokepoint and it covers the direct + `runOperationUnit` call at `:1262`. **A skip is a skip: the lane keeps running other + channels. It is never a lane stop, and it is not a hold** — "a zero limit is a hold, never + a stop" (`pauseGates.ts`) is untouched by this and must stay untouched. +3. **`common/controller/channelSnapshot.ts:623`** — before the `readdir`, throw + `ChannelMediaUnreachableError` rather than computing a snapshot over an empty dir. The + scheduler already keeps the last `snapshot.json` on a failed refresh + (`snapshotScheduler.ts:205-212`); verify that path logs the reason rather than swallowing + it. Everything downstream of the snapshot — `/cleanup`, the channel bands, the lanes' work + lists — is protected by this one guard. +4. **`common/controller/operationBatch.ts:1578`** — assert before the `readdir`. Defence in + depth today (all three callers are already behind guard 1), correctness tomorrow for any + in-process caller that is not. + +Plus the one bypass: **`editor/app/channels/[slug]/shardActions.ts:87`** gets an explicit +`assertChannelMediaReachable` at the top of the `download-missing` branch. + +**Surfacing:** channel page and `/channels` render a badge from `inspect()` — "Media relocated +to `<root>` · reachable" / "**unreachable** — mount the drive". ### 3. `common/controller/relocateChannelMedia.ts` (new) — the move, both directions `relocateChannelMedia({ paths, slug, direction: "out", root, onLog, signal })`: -1. **Preflight** (all throw with an operator-readable message): channel exists and is not - social; no `.relocating.json` from a *different* target; `root` exists, is a directory, - is writable; target `<root>/<slug>/data` is absent **or** a partial from an earlier aborted - run (resumable); free space on `root` ≥ bytes in `data/` + `resumeMarginGB` (walk with - `stat`, like the snapshot does; `getFreeBytes(root)` from `diskSpace.ts`); source `data/` - is a real directory (not already a link — moving root→root is `direction:"out"` from the - current *target*, handled by resolving the real source first). +1. **Preflight** (all throw operator-readable messages): channel exists and is not social; no + `.relocating.json` for a *different* target; `root` exists, is a directory, is writable; + target `<root>/<slug>/data` is absent **or** a resumable partial; free space on `root` ≥ + bytes in `data/` + `resumeMarginGB` (`getFreeBytes(root)`, `diskSpace.ts:13`); source + `data/` is a real directory, not already a link. 2. **Write the marker**, phase `copy`. -3. **Copy**: `rsync -a --partial --info=progress2 <src>/ <target>/` via `execa(paths.rsyncBin, …, { cancelSignal: signal })`, - exactly as `backupSavedVideos.ts:78-90`. `-a` preserves mtimes → no LMDB reindex. Abort - leaves the source untouched and a resumable target. +3. **Copy**: `rsync -a --partial --info=progress2 <src>/ <target>/` via + `execa(paths.rsyncBin, …, { cancelSignal: signal })`, exactly as + `backupSavedVideos.ts:76-90`. `-a` preserves mtimes → no LMDB reindex. An abort leaves the + source untouched and the target resumable. 4. **Verify**: `rsync -a --dry-run --itemize-changes` must print nothing; then file count and - byte sum must match on both sides. Any drift → fail, source untouched, marker stays - (rerun resumes at step 3). + byte sum must match both sides. Any drift → fail, source untouched, marker stays. 5. **Swap** (phase `swap`): `rename(data, data.relocated-<ts>)` (same device, atomic) → - `symlink(target, data)` → `writeChannelConfig({ ...config, dataDir: target })`. -6. **Reclaim** (phase `reclaim`): `rm -r data.relocated-<ts>`; remove marker; log bytes - moved and free space on the source volume now. + `symlink(target, data)` → `writeChannelConfig({ ...config, dataDir: target })` + (`channels.ts:452`). +6. **Reclaim** (phase `reclaim`): `rm -r data.relocated-<ts>`; remove marker; log bytes moved + and free space on the source volume now. `direction: "back"`: copy `target/` → `channels/<slug>/data.incoming/`, verify, `unlink(data)`, -`rename(data.incoming, data)`, clear `dataDir`, `rm -r target`, remove marker. Same code, -reversed endpoints. +`rename(data.incoming, data)`, clear `dataDir`, `rm -r target`, remove marker. **Crash recovery** is the marker's `phase`: a rerun with a matching marker re-verifies and -continues from `swap`/`reclaim`; `inspect()` reports `in-transition` until it finishes. The -sibling `data.relocated-*` / `data.incoming` names are deliberate so nothing else ever -mistakes them for the live dir. +continues from `swap`/`reclaim`; `inspect()` reports `in-transition` until it finishes. -### 4. Job kind + action +### 4. Job kind + actions - `common/jobs/jobKinds.ts`: `"relocate-channel-media"` — label "Relocate channel media", - `drainable: false`, `replayable: false`, `queueKeyStrategy: "custom"`. Queue key = - `channelQueueKey(slug)` (`common/lib/queueKeys.ts:34`) so it serializes with the channel's - own jobs. + `drainable: false`, `replayable: false`, `queueKeyStrategy: "custom"`, `needsMedia: false` + (it *is* the thing that fixes an unreachable channel; guard 1 must not refuse it). Queue key + = `channelQueueKey(slug)` (`queueKeys.ts:34`) so it serializes with the channel's own jobs. - `editor/app/channels/[slug]/storageActions.ts` (new, `"use server"`): - - `previewRelocationAction(slug, root)` → `{ bytesToMove, freeOnRoot, freeOnSource, sameDevice, existingPartial, error? }` (`statfs` both sides; `sameDevice` via `stat().dev` — a warning, not a block). + - `previewRelocationAction(slug, root)` → `{ bytesToMove, freeOnRoot, freeOnSource, sameDevice, existingPartial, error? }`. - `relocateChannelMediaAction(slug, root)` / `moveChannelMediaBackAction(slug)` — the - rename-style active-jobs guard (`actions.ts:463-473`), then `runManagedFunction`. -- `deleteChannel` (`channels.ts:398`): if `config.dataDir` is set, `rm -r` the target first - (and the `<root>/<slug>` parent if it is then empty), then the channel dir. Refuse when - a `.relocating.json` marker is present. -- `renameChannel.ts`: after the channel-dir rename, if `dataDir` matches + rename-style active-jobs guard (copy `actions.ts:541-552`), then `runManagedFunction`. +- `deleteChannel` (`channels.ts:476`): if `config.dataDir` is set, `rm -r` the target first + (and `<root>/<slug>` if then empty), then the channel dir. Refuse when a `.relocating.json` + marker is present. +- `renameChannel.ts`: after the channel-dir rename (`:80`), if `dataDir` matches `<root>/<oldSlug>/data`, rename `<root>/<oldSlug>` → `<root>/<newSlug>`, re-create the - symlink, rewrite `dataDir`; roll back the dir rename on failure (same rollback shape as - `:94`). A target that doesn't follow the convention is left alone (still works — the link is - absolute). + symlink, rewrite `dataDir`; roll back the dir rename on failure (same shape as `:92-95`). A + target not following the convention is left alone — the link is absolute and still works. ### 5. Disk gate measures the right volume `diskGate(paths, settings, { mode, dir? })` — `dir` defaults to `paths.transcriptsDir`. Thread the channel's `data` path from the callers that write media for a known channel: -`runYtdlp.ts:876`, `downloadOneManaged.ts` (root is in scope), `autoRunner.ts:654`, -`backfillReacquire.ts:79`, `pipelineActions.ts:48`, `fixIncompleteTranscript.ts:78`, -`videoActions.ts:266,301`. The latch (`gateLatched`, `diskSpace.ts`) becomes a -`Map<dir, boolean>` so a full SSD does not pause downloads that land on the platter and vice -versa. `persistKept`/saved-video callers keep their current dir. Tests in `diskSpace.test.ts` -extend the existing hysteresis cases with two dirs. +`runYtdlp.ts:876`, `autoRunner.ts:1059`, `backfillReacquire.ts:166`, `pipelineActions.ts:49`, +`fixIncompleteTranscript.ts:78`, `videoActions.ts:266,301`. The global latch +(`diskSpace.ts:181` `gateLatched`, read at `:230,238`, cleared at `:244`) becomes a +`Map<dir, boolean>` so a full SSD does not pause downloads landing on the platter, and vice +versa. `persistKept.ts:106` and `buildActiveJobs.ts:347` (observe) keep their current dir. +`diskSpace.test.ts` extends the hysteresis cases to two dirs. ### 6. UI -- **Channel page** (`editor/app/channels/[slug]/components/`): a "Storage" panel — - location (in place / target path), status from `inspect()`, media bytes (from the loaded - snapshot, no new walk), free space on the current volume; a "Move media to…" form - (root path input → live preview from `previewRelocationAction` → confirm button that - starts the job), and "Move back in place" when relocated. Disabled with a reason while the - channel has jobs or a marker is present. -- **`ChannelForm.tsx`** (`:641-647`, next to "Saved-video store dir"): a read-only line - "Media location: <path> — change from the Storage panel". Not an input (see Layout). -- **Channels list / dashboard**: a small "relocated" badge; red when unreachable. -- **`/cleanup`**: nothing to change; the ledger already reads snapshots. Optionally show the - channel's volume next to its reclaimable bytes. +A "Storage" stage alongside the existing ones (`[slug]/components/stages/`, composed in +`page.tsx:53-67`): location, status from `inspect()`, media bytes (from the loaded snapshot — +no new walk), free space on the current volume, a "Move media to…" form (root input prefilled +from the settings default, live preview, confirm) and "Move back in place" when relocated — +disabled with a reason while the channel has jobs or a marker is present. In +`ChannelForm.tsx:641-647`, next to "Saved-video store dir", a **read-only** line "Media +location: `<path>` — change from the Storage panel"; not an input. A small "relocated" badge on +the channel page and in `ChannelsTable.tsx`, red when unreachable. ### 7. Docs - `AGENTS.md` corpus table: `data/` may be a symlink to `config.dataDir`; never symlink a - whole channel dir (`isDirectory()` filters); the marker file name. + whole channel dir (`isDirectory()` filters at `channels.ts:246,363`); the marker file name. - `RUNNING_IN_DOCKER.md`: the corpus is a named volume, so a relocated target must be - bind-mounted into the container **at the same absolute path** (e.g. - `- /mnt/platter/archilyzer-media:/mnt/platter/archilyzer-media`), otherwise the guard reports - the channel unreachable (by design — it does not fall back to empty). -- `WORKTREES.md` / shard notes: syncing `channels/` with `rsync -a` copies the link as a link; - use `--copy-links` (or sync the root too) when a shard must carry the media. Executors that - need no corpus are unaffected. -- `editor/CHANGELOG.md` entry. + bind-mounted **at the same absolute path** — otherwise the guard reports the channel + unreachable, by design, rather than falling back to empty. +- `WORKTREES.md`: `rsync -a` of `channels/` copies the link as a link; use `--copy-links` when + a shard must carry the media. `editor/CHANGELOG.md` gets an entry. + +### 8. Cold root (settings default) + bulk move — the widened ask + +Minimal, one field and one action. **No tiers, and no coupling to auto-queue weights** — a +relocated channel is not thereby deprioritized, and a deprioritized channel is not thereby +relocated. Coupling the two ("auto-relocate anything below weight N") is a possible later +step and is deliberately not designed here. + +- **`storage.mediaRoot`** on `SiteSettings` (`common/lib/settings.ts:82`), with a + `sanitizeStorage()` beside the existing sanitizers (`:735-1135`): trimmed string, blank = + none, no validation of existence at sanitize time (the drive may be unmounted when settings + are read). Rendered on the settings page (`editor/app/settings/components/SettingsForm.tsx`) + as "Default media root", with the same "blank uses none" hint shape as + `savedVideosDir`. It **prefills** the per-channel Storage panel's root input and nothing + more — it is never read by the relocate controller, which always takes an explicit `root`. +- **Bulk "Move media to <root>" on `/channels`.** `ChannelsTable.tsx` has no per-row selection + today (its only checkbox, `:224-232`, is the "Group by section" toggle), so the slice adds + one: a per-row checkbox plus a selection bar, modelled on the video-list selection bar. The + action (`editor/app/channels/bulkStorageActions.ts`) **enqueues one `relocate-channel-media` + job per selected channel** on that channel's `channelQueueKey` — the job queue serializes + them, there is no batch controller. Each job runs its own preview and space check **at run + time**, not at enqueue time, so a root that fills partway through refuses the remainder + cleanly. A channel already relocated to that root is skipped, not failed. The action returns + a summary (`{ enqueued, skipped, errors }`) rendered in the selection bar. --- ## Tests -- **Unit (`common/`, vitest, real rsync like `backupSavedVideos.test.ts`)**: - `channelMedia.test.ts` (in-place / ok / dangling target / marker present / link disagrees - with config); `relocateChannelMedia.test.ts` — out: link + config + mtimes preserved + - source gone; abort mid-copy (abort the signal from an `onLog` hook) leaves source intact and - target resumable; rerun completes; verify-failure keeps source; back: restores a real dir - and clears config; `deleteChannel` removes the target; `renameChannel` re-points a - convention-shaped target and rolls back on failure; `channelSnapshot` throws on unreachable - rather than writing an empty snapshot; `diskGate` per-dir latch. -- **e2e (one spec, `editor/e2e/channel-storage.spec.ts`)**: relocate the fixture channel to a - tmp root from the Storage panel, wait for the job, assert the badge, the videos list still - lists videos and a transcript file route still serves; move back. Runs under the global e2e - queue (`E2E_QUEUE`) — expect to wait. -- `pnpm exec tsc --noEmit` in `common/` and `editor/`; `pnpm -C common test`. -- **Offline dry-run on the real corpus** (no second editor — `instrumentation.ts` arms the - runners): a `tsx` script calling `previewRelocation`/`inspectChannelMedia` for `omnimirror` - against a mounted root, per the offline-controller-run rule. - -## Rollout for `omnimirror` - -1. Mount `sdb1` (1.8 T ext4 platter) at e.g. `/mnt/platter` with `nofail` in fstab; create - `/mnt/platter/archilyzer-media` owned by `user`. -2. Editor → `omnimirror` → Storage → Move media to `/mnt/platter/archilyzer-media`. Preview - should show ~131 GB to move; the copy is one platter write, ~15–25 min. The channel's jobs - are blocked for the duration; everything else keeps running. -3. Afterwards `/home` gains ~131 GB; the channel's next sync downloads straight onto the - platter through the link, and the disk gate watches that volume for it. +- **Unit (`common/`, `node:test` + tsx, real rsync like `backupSavedVideos.test.ts:29-40`)**: + `lib/channelMedia.test.ts` (in-place / ok / dangling target / marker present / link disagrees + with config / config read without a passed config object); + `controller/relocateChannelMedia.test.ts` (out: link + config + mtimes preserved + source + gone; abort mid-copy leaves source intact and target resumable; rerun completes; + verify-failure keeps source; back: restores a real dir and clears config); + `controller/channels.test.ts` (`deleteChannel` removes the target and refuses under a marker; + `renameChannel` re-points a convention-shaped target and rolls back on failure); + `controller/channelSnapshot.test.ts` (throws on unreachable rather than writing an empty + snapshot); `lib/diskSpace.test.ts` (per-dir latch, two dirs, existing hysteresis cases + unchanged); `architecture.test.ts` must stay green **with no new `ALLOWED` entry**. +- **e2e**: `editor/e2e/channel-storage.spec.ts` (slices 2 and 3). From a **worktree**, with a + composed fixture site copied into `export/public` so the export webServer answers 200 + (`plans/STATE.md`); the primary checkout is blocked by the untracked `editor/content` + symlink. Runs behind the machine-global queue lock — **run detached**, ~25-32 min for the + full suite. +- `pnpm exec tsc --noEmit` in `common/` and in `editor/`; `pnpm --filter yt-dlp-transcript-common test`. +- **Offline dry-run on the real corpus** (never a second editor — `instrumentation.ts` arms + the runners): a `tsx` script calling `inspectChannelMedia` / the preview for `omnimirror` + against a mounted root. + +--- + +## Slices + +Three mergeable units on `storage/relocate-media`, each with its own gate. + +### Slice 1 — core (no UI, no job) + +**Files:** `common/lib/channelMedia.ts` (new), `common/controller/relocateChannelMedia.ts` +(new), `common/jobs/streamCommand.ts`, `common/jobs/jobKinds.ts` (`needsMedia`), +`common/controller/autoRunner.ts`, `common/controller/channelSnapshot.ts`, +`common/controller/operationBatch.ts`, `common/controller/channels.ts` (`deleteChannel`), +`common/controller/renameChannel.ts`, `common/lib/diskSpace.ts` + its callers, and the six +unit test files. + +**Commits:** (1) `channelMedia.ts` + its test; (2) the four guards + `needsMedia`; +(3) `relocateChannelMedia.ts` + its test; (4) `deleteChannel`/`renameChannel` + tests; +(5) per-dir disk gate + test. + +**Gate:** `pnpm --filter yt-dlp-transcript-common exec tsc --noEmit` · `pnpm --filter yt-dlp-transcript-common test` +(includes `architecture.test.ts`, which must be green with no added `ALLOWED` entry) · +`pnpm --filter yt-dlp-transcript-editor exec tsc --noEmit`. + +### Slice 2 — editor: the job, the panel, the docs + +**Files:** `editor/app/channels/[slug]/storageActions.ts` (new), a `StorageStage` under +`[slug]/components/stages/` wired in `[slug]/page.tsx`, `ChannelForm.tsx:641-647`, +`ChannelsTable.tsx` (badge), `editor/app/channels/[slug]/shardActions.ts` (the one bypass), +`AGENTS.md`, `RUNNING_IN_DOCKER.md`, `WORKTREES.md`, `editor/CHANGELOG.md`, +`editor/e2e/channel-storage.spec.ts`. + +**Commits:** (1) job kind wiring + `storageActions.ts`; (2) the Storage panel + badge + +ChannelForm line; (3) the shardActions guard; (4) docs + changelog; (5) the e2e spec. + +**Gate:** both `tsc --noEmit` · the common suite · `pnpm e2e -- channel-storage.spec.ts` +from a worktree, detached. Cases: relocate the fixture channel to a tmp root from the panel, +wait for the job, assert the badge, assert the videos list still lists videos and a transcript +file route still serves, move back. + +### Slice 3 — cold root + bulk move + +**Files:** `common/lib/settings.ts` (`storage.mediaRoot` + `sanitizeStorage`), +`editor/app/settings/components/SettingsForm.tsx` + its action, +`editor/app/channels/components/ChannelsTable.tsx` (per-row selection + bar), +`editor/app/channels/bulkStorageActions.ts` (new), one more e2e case. + +**Commits:** (1) the settings field + sanitizer + form; (2) selection in `ChannelsTable`; +(3) `bulkStorageActions.ts`; (4) the e2e case. + +**Gate:** both `tsc --noEmit` · the common suite (new `sanitizeStorage` cases in +`settings.test.ts`) · `pnpm e2e -- channel-storage.spec.ts` from a worktree, detached. The +added case: set the default root, select two fixture channels, Move, assert two jobs enqueued +on two queue keys and both badges flip. + +--- + +## Rollout — the operator's, after the platter is mounted + +`/home` is at **100 %, 6.9 G free**. Nothing below needs free space on `/home`: every copy +goes **out** to the platter, and it is the swap that frees the source. The job path writes +essentially nothing to `/home` during a copy — the job log under `transcripts/jobs/` and the +`.relocating.json` marker, both kilobytes. The one real risk of a 100 %-full source is that an +*unrelated* writer (a download, the LMDB index) fails mid-run; that is a reason to do step 0 +first, not a reason to change the design. + +### Step 0 — the saved-video store, by hand, today (no code, frees ~130 GB) + +`transcripts/saved-videos` is **130.5 GB**, all of it one channel — `nuxanor-kick`, 29 +containers, still being written to. + +**`SavedVideoPointer.dir` is an ABSOLUTE store dir** (`common/lib/savedVideo.ts:30-44`; a real +pointer under `channels/nuxanor-kick/data/<id>/saved-video.json` carries +`/home/user/Projects/yt-dlp-transcript-browser/transcripts/saved-videos/<slug>/<id>`), and +`savedVideoPath()` (`:71-72`) joins `pointer.dir` with `pointer.file`. **So re-pointing +`SAVED_VIDEOS_DIR` or a channel's `savedVideosDir` does NOT move the store** — `savedVideoDir()` +(`:61-67`) recomputes from the root only for *new* persists and for `rewriteSavedVideoDir` +(`savedVideo-server.ts:55-61`, which the rename uses). It would split future persists from the +29 existing pointers, which keep resolving to the old path. + +A **whole-directory symlink** keeps every absolute pointer valid with zero code — the same +mechanism as the channel design. **Nothing walks the store root**: `resolveSavedVideo`, +`unpersistSavedVideo`, `dropSavedVideo` and `pruneEmptyStoreDirs` are all driven off +`pointer.dir` (`savedVideo-server.ts:86,146,157,163,168`); `pruneSavedVideos.ts:51` and +`savedVideoInventory.ts:29,57` walk `channelsDir` and the per-channel `data/` dirs reading +pointers, never the store; `backupSavedVideos.ts:154-166` walks the **destination** root, not +the store. And there is still **zero** `lstat`/`readlink`/`realpath`/`symlink` anywhere in +`common/`, `editor/` or `export/` — re-confirmed at `61eae05`, saved-video paths included. A +symlinked store root is transparent to all of it. + +Runbook (the platter takes the copy, so this works from a 100 %-full `/home`): + +1. Stop the editor — the persist job writes into the store. +2. `rsync -a --partial --info=progress2 transcripts/saved-videos/ /mnt/platter/archilyzer-saved-videos/` +3. `rsync -a --dry-run --itemize-changes transcripts/saved-videos/ /mnt/platter/archilyzer-saved-videos/` must print nothing. +4. `mv transcripts/saved-videos transcripts/saved-videos.moved-2026-09-11` +5. `ln -s /mnt/platter/archilyzer-saved-videos transcripts/saved-videos` +6. Restart; confirm one container plays and the persisted-video panel reads its size. +7. `rm -r transcripts/saved-videos.moved-2026-09-11` — **this is the step that frees the 130 GB.** + +Do this before any slice ships. It is why it is step 0. + +### Step 1 — mount the platter + +Mount `sdb1` (1.8 T ext4) at `/mnt/platter` with `nofail` in fstab; create +`/mnt/platter/archilyzer-media` and `/mnt/platter/archilyzer-saved-videos`, owned by `user`. + +### Step 2 — channels, largest first, through the UI + +Per-channel `data/` sizes, measured 2026-09-11 (`channels/` totals 448.7 GB across 68 +channels; the top ten are 302 GB of it): + +| Channel | GB | +|---|---:| +| `omnimirror` | 130.3 | +| `shondo-vods` | 42.4 | +| `rekietalaw` | 29.6 | +| `angryjoeshow` | 22.6 | +| `cornbreadman` | 21.6 | +| `HasanAbiVODs3` | 17.4 | +| `omnibased` | 17.0 | +| `nuxanor-kick` | 16.9 | +| `chibi-reviews` | 13.1 | +| `kirsche` | 10.8 | + +(Next: `the-quartering-rumble` 10.7.) **These are sizes, not priorities — the operator picks +the order**, largest deprioritized channel first. `omnimirror` alone is 130 GB and is the +obvious first move; it is also mostly-redundant mirror content, which is what makes it the +cheapest thing to put on a slow drive. + +For each: Editor → channel → Storage → Move media to `/mnt/platter/archilyzer-media` (or select +several on `/channels` and use the bulk action, once slice 3 ships). The preview shows the +bytes; the copy is one platter write. The channel's own jobs are blocked for the duration; +everything else keeps running. Afterwards the channel's next sync downloads straight onto the +platter through the link, and the disk gate watches **that** volume for it. + +--- ## Out of scope (deliberately) - Per-video or per-file relocation (the saved-video store already covers "keep the big - container elsewhere"); a named global list of storage roots (one drive today — a free-text - root plus preview is enough); making shards symlink-aware; network stores. + container elsewhere"); a named global list of storage roots beyond the one default; + making shards symlink-aware; network stores. +- Any coupling between relocation and auto-queue weights or lane priorities. Noted in §8 as a + possible later step; not designed, not built. +- Moving `index.mdb` (13 GB) — it is hot and small relative to the media.