commit a587e2abcc7e3147a0147095cf32bdbebc4b7ca5
parent 7cddd7cc34b26acf31395bef85b2bf808ab9198e
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Thu, 17 Sep 2026 13:34:27 -0400
plans: storage locations — named, refreshable, re-pointable media places
Approved 2026-09-17. Five slices: S0 popover stacking fix, S1 runner hardening
(omnimirror), S2 entity + probe + migration, S3 /storage page + re-point job, S4
channel surfaces + docs. Facts pinned to 7cddd7c.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
1 file changed, 191 insertions(+), 0 deletions(-)
diff --git a/plans/storage-locations.md b/plans/storage-locations.md
@@ -0,0 +1,191 @@
+# Storage locations: named, refreshable, re-pointable places a channel's media lives
+
+Status: **approved 2026-09-17**, facts pinned to `main` @ `6b4f25b`. Slices ship on
+`storage/locations-s{0,1,2,3,4}` branches; this file is updated as each lands.
+
+## Context
+
+The rollout puts big channels on the platter (`sdb1`, ext4, UUID
+`09b598d2-b765-40bd-9466-06d8e16cb119`) and forgets about them until later. Today that
+disk is udisks-automounted at `/run/media/user/<uuid>` with no fstab entry, and a relocated
+channel's `config.dataDir` is an absolute path. If the disk comes up somewhere else, or not
+at all, every channel on it reads `unreachable` and the only remedy is `ssh` and hand edits.
+The operator asked (2026-09-17) for **location entities in the editor** that check whether
+they are available (refreshable) and can be re-pointed to a different path.
+
+Decisions taken with the operator 2026-09-17: a new **`/storage`** page (twelfth nav entry,
+Machine group); re-point is **manual, one click, with a per-location opt-in `autoRepoint`**;
+the saved-video store is **not** a location in this sub-phase.
+
+The omnimirror incident (2026-09-13) is folded in as slice S1 because it is the same failure
+family: the relocate copy raced an in-flight auto-digest batch (its units make no job record,
+and the batch's reachability guard runs once at start), a sidecar written mid-copy left one
+directory mtime differing, `verifyCopy` refused, and the Storage panel offered no way to
+resume although the controller resumes from a marker.
+
+**Operator step today, before any of this ships** (verified on disk: the platter copy is
+byte-complete; drift is one directory timestamp `data/v4p31nz/`): `/jobs` idle for
+omnimirror → channel page → Storage → **Clear marker** → **Move media** to the same root
+(preview says "partial copy … resumes"; rsync fixes one mtime; swap and reclaim free 131 GB).
+
+## Verified facts (2026-09-17, `main` @ `6b4f25b`)
+
+- `common/lib/channelMedia.ts` `inspectChannelMedia` (2 stats + 1 JSON read) is the per-channel
+ truth; guards skip `unreachable`. Unchanged by this plan.
+- Zero channels are relocated in production; `settings.storage.mediaRoot` is the only stored
+ root (`common/lib/settings.ts:853-882`; sanitizer never checks existence, drops relative).
+- `ChannelConfig` round-trips are whitelisted (`channelConfig.ts:225-279`): a new config field
+ would need the type + parser. **This plan adds no config field**: a channel is on location
+ L iff `config.dataDir` is under `L.root + "/"`.
+- Subprocesses go through `execa`; lib may use it (`lib/digestApps.ts:304`, `lib/git.ts`), but
+ nothing reachable from a `"use client"` file may (next build fails on `node:child_process`).
+ Bins live on `Paths` (`paths.ts:117 rsyncBin`, env `RSYNC_BIN`).
+- `getFreeBytes` (`lib/diskSpace.ts:29-43`) walks up on ENOENT: an unmounted path reports the
+ parent volume. Only call it for an `available` location.
+- Host tooling: `findmnt`, `lsblk`, `blkid`, `udisksctl` present; `findmnt -J -T <path>`
+ gives `{source,target,fstype,uuid,label}`; `findmnt -rn -S UUID=<u> -o TARGET` gives the
+ mountpoint; `findmnt --fstab -S UUID=<u>` exits 1 (not in fstab); `/dev/disk/by-uuid/<u>`
+ exists. In Docker block devices are invisible and the root must be identity-bind-mounted
+ (`RUNNING_IN_DOCKER.md:379-399`) — every identity probe **fails open** to "unknown".
+- Re-point precedent: `common/controller/renameChannel.ts:139-201` unlinks, symlinks and
+ rewrites `dataDir` with a replay ledger. `relocateChannelMedia.ts` itself refuses a link
+ that points elsewhere (`:692-695`) — re-point is new code on the rename pattern.
+- Resume precedent: `relocateChannelMedia` resumes a same-direction marker (`:450,:507-513`);
+ `relocationJob.ts:59` already logs "(resumed an interrupted move)". The UI never calls it
+ (`StorageStage.tsx`: `canMoveOut = !location.relocated`; marker → only "Clear marker").
+- Auto-lane units are visible: `getAutoRunnerStatus(kind).inFlight[]` carries `channelSlug`
+ (`autoRunner.ts:282-292, 1553-1558`); `storageActions.ts:65-78 activeJobsRefusal` only
+ looks at the registry, so it could not see the digest unit that raced omnimirror.
+- Nav: `editor/app/lib/nav.ts` `NAV_GROUPS`, Machine = Jobs/Workers/Cleanup/Saved videos/
+ Settings; `nav.test.ts:19-22` asserts eleven; `plans/editor-operations-ia.md:70-76` states
+ "fold, do not add" — decision 1 is a recorded exception. There is no `measure-nav` nav
+ counter (`editor/scripts/measure-nav.mjs` times routes; it does not count).
+- e2e fakes are env-injected bins (`editor/package.json:8-9`, `e2e/fixtures/bin/fake-*.mjs`);
+ `channel-storage.spec.ts` (393 lines) fills `aria-label="destination root"` and asserts
+ `getByLabel(/^media location:/)`.
+
+## Design
+
+### Entity (settings)
+
+`StorageSettings = { locations: StorageLocation[]; defaultLocationId: string }`
+```ts
+type StorageLocation = {
+ id: string; // /^[a-z0-9][a-z0-9-]{0,63}$/, unique
+ label: string; // blank → id
+ root: string; // absolute, trailing "/" stripped, never existence-checked
+ autoRepoint: boolean; // opt-in: re-point without asking when safe
+ volume?: { uuid: string; fstype?: string; label?: string; mountpoint: string; relPath: string };
+}; // identity learned at the last successful probe; root === join(mountpoint, relPath)
+```
+Availability is **never persisted** (refresh must not churn `settings.json` → pulse rev).
+`mediaRoot` migrates on read (`migrateMediaRootToLocations`, laneMigration rules: only when
+`locations === undefined`, identity otherwise, blank → empty list, absolute → one location
+`{ id: "default", label: "Default", root }` as default) and leaves the schema; the settings
+form field becomes a link to `/storage`. Rollback: an older binary sanitizes to
+`{ mediaRoot: "" }` — one string lost. Runbook: `cp settings.json settings.json.pre-storage-locations`.
+
+### Probe (`common/lib/storageVolumes.ts`, server-only)
+
+`probeLocation(loc, bins) → { status, identity?, candidateRoot?, warning?, freeBytes? }`
+- `stat(root)` is a dir → `available`; `findmnt -J -T root` (3 s, `reject:false`) → identity or unknown; `freeBytes` only here.
+- root missing + `volume.uuid`: `findmnt -rn -S UUID=… -o TARGET` non-empty → `mounted-elsewhere`, `candidateRoot = join(target, relPath)`; else `lstat /dev/disk/by-uuid/<uuid>` → `unmounted`; else `absent`.
+- root missing, no uuid → `missing`.
+- `warning` when available and mountpoint under `/run/media/` or `/media/`, or not in fstab: "automount — may not be present at boot; add `UUID=<u> /mnt/platter <fstype> nofail 0 2`, or rely on re-point".
+`mountByUuid(uuid, bins)`: `udisksctl mount -b /dev/disk/by-uuid/<uuid>` (15 s, `reject:false`), offered only when `unmounted` and the binary resolves (`--version` probe memoised, as `digestApps.ts:301-313`).
+Bins on `Paths`: `findmntBin` (`FINDMNT_BIN`), `udisksctlBin` (`UDISKSCTL_BIN`).
+
+### Layers
+
+| module | layer | role |
+|---|---|---|
+| `common/lib/storageLocations.ts` | lib, pure | types, `locationOfDataDir(dataDir, locations)` (longest root wins), `migrateMediaRootToLocations`, `defaultLocationRoot` — the ONLY storage module a `"use client"` file may import (types) |
+| `common/lib/storageVolumes.ts` | lib, execa | `probeLocation`, `mountByUuid` |
+| `common/controller/storageLocations.ts` | controller | `channelsOnLocation` (per-channel `inspectChannelMedia` roll-up), `preflightRepoint`, `repointStorageLocation` (job body), `maybeAutoRepoint`, `runStorageBootPass`, 10 s probe memo |
+| `common/views/storage.ts` | views, pure | `buildStorageRows({ locations, defaultLocationId, probes, rollups, registry, now })` — the Phase 3 pattern; the shell does I/O |
+
+No new `ALLOWED` entry; no config field; no route path change.
+
+### The re-point job (`repoint-storage-location`)
+
+`jobKinds.ts` after `relocate-channel-media`: `drainable:false, replayable:false, queueKeyStrategy:"custom", needsMedia:false`; `queueKey = relocationQueueKey()` (serialises with moves); added to `snapshotScheduler.ts NO_REGEN_KINDS`. One at a time.
+- **Preflight** (nothing written): location exists; `newRoot` absolute dir, `!== root`; identity — recorded uuid and probed uuid both known and different → refuse "different disk"; every channel on the old root: link (not real dir), no marker, not busy, `<newRoot>/<slug>/data` is a dir (refuse listing all missing), `relocationRootProblem` per slug.
+- **Per channel, sequential**, renameChannel ledger `{slug, oldTarget, unlinked, relinked, configWritten}`: unlink → symlink(newTarget) → `writeChannelConfig({...fresh, dataDir: newTarget})`; on throw replay in reverse with `.catch(()=>{})`, rethrow naming slug + step. Restored state is the pre-job `unreachable`, never `inconsistent`.
+- **Then the location**: `writeSettings` with `root = newRoot`, `volume` from the probe; in the ledger too.
+- **Idempotent rerun**: already-moved channels are no longer "on" the old root; a rerun finishes the rest or does the settings write alone.
+- **autoRepoint**: `refreshLocationAction` and the boot pass call `maybeAutoRepoint` when `loc.autoRepoint && status === "mounted-elsewhere" && preflight ok`; idle boot (`ARCHILYZER_IDLE_BOOT`) probes but never enqueues.
+
+### Surfaces
+
+- `/storage` (server component, `force-dynamic`): `editor/app/storage/{page.tsx, buildStorage.ts (shell), actions.ts, lib/repointJob.ts, components/StorageLocationsTable.tsx, components/LocationForm.tsx}`. Rows: label, root, status badge (`aria-label="location status"`), identity, channels (`n ok / n unreachable / n moving`, `aria-label="location channels"`), free bytes when available, last probe, warning, actions **Refresh / Re-point to `<candidate>` / Mount / Edit / Delete** (delete refused while any channel's `dataDir` is under the root). Wrapper `aria-label="storage locations"`, row `aria-label="storage location: <id>"`, form fields `location id/label/root/auto re-point`, add button `add storage location`, outputs `Re-point output`.
+- Nav: `{ href: "/storage", label: "Storage", icon: HardDrive, keywords: "drive disk platter mount volume media location relocate cold" }` after Cleanup; `nav.test.ts` eleven → twelve; `nav.ts:27-30` header and `editor-operations-ia.md:70-76` record the exception (a location is a Machine noun, not a fold of Cleanup).
+- `editor/instrumentation.ts`: lazy-import `runStorageBootPass({ enqueue: !idle })` inside `try`, `void`ed.
+- `MediaLocationBadge`: `locationLabel?` → "on Platter" / "on Platter — unreachable"; tables never probe (not render-safe); projection at `channels/page.tsx:217-222` and the dashboard row builder via `locationOfDataDir`.
+- `StorageStage`: destination `<select aria-label="destination location">` (default = `defaultLocationId`) + `__custom` revealing today's `destination root` input; the page passes the channel's location probe; **new "Resume move"** (`resumeRelocationAction(slug)`: marker present, not busy, root = grandparent of `marker.target`, refuse if `marker.target !== relocatedDataDir(root, slug)`), offered under the same condition as Clear marker.
+- `ChannelSelectionDeck`: `<select aria-label="bulk media location">`; `bulkRelocateChannelMediaAction(slugs, locationId)`.
+- Docs: `RUNNING_IN_DOCKER.md` §another drive (identity unknown in a container; re-point by path is the whole story); `AGENTS.md` corpus table one sentence; `editor/CHANGELOG.md`.
+
+## Slices and dependency graph
+
+**S0 — one-commit bug fix, ships first, on its own.** Operator report 2026-09-17: on
+`/channels` a row's Advanced menu draws under later rows. Cause, verified:
+`ChannelsTable.tsx:607-609` puts `opacity-60` on the `<tr>` (excluded from build, or tier
+Paused — omnimirror is Paused, the "Media moving" badge is a coincidence). Opacity < 1
+creates a stacking context on the row, so `ChannelTierSelect.tsx:188`'s `absolute z-30`
+popover is confined to it and every later row paints over it. Fix: dim per cell, not per
+row — pass `dim` to the row's cells and apply `opacity-60` on every `<td>` EXCEPT the Tier
+cell (the one that hosts the popover), or dim by `text-muted-foreground` alone. Comment
+the reason at the site. e2e: `channels-*` specs stay green; add one assertion that a
+Paused row's Advanced popover is visible (`toBeInViewport` + a click on an inner control
+succeeds) in the spec that already opens Advanced.
+
+```
+S1 runner hardening (omnimirror) ──┐
+S2 entity + probe + migration ──────┼──> S3 /storage page + actions + job + nav ──> S4 channel surfaces + docs
+```
+S1 ∥ S2 (no shared file). S3 needs S2. S4 needs S3.
+
+- **S1** — (a) `operationBatch.ts next()` (`:1689-1717`, before `classifyOperationUnit`): `readRelocationMarker` → log + `return null` (ends the run cleanly); `autoRunner.ts run(picked)` (`:1636-1650`, before the lane branch): marker → `outcome: "skipped"` (the `finally`'s `markCompleted` retires the unit for the session — state the trade-off in code). (b) `verifyCopy` (`relocateChannelMedia.ts:337-370`): when every drift line matches `/^\.d\.\.t/`, one more `rsync -a` pass and re-verify, `retried` flag, content drift still throws. (c) new `editor/app/channels/lib/mediaBusy.ts` `channelMediaBusyReason(slug)` = registry running/queued + `LANES.flatMap(k => getAutoRunnerStatus(k).inFlight).filter(u => u.channelSlug === slug)`; used by `storageActions.ts`, `bulkStorageActions.ts`, `[slug]/page.tsx` `blockedReason`.
+- **S2** — `lib/storageLocations.ts`, `lib/storageVolumes.ts`, `settings.ts` (type, defaults, sanitizer, migration at the `:1475` hook applied to the parsed block), `paths.ts` bins, `storageSettings.test.ts` rewritten, `SettingsForm.tsx:134-139` + `settings/actions.ts` (`:58-61,:160-165,:262-264`) drop `mediaRoot`, `bulkStorageActions.ts:80` / `[slug]/page.tsx:553` / `channels/page.tsx:358` read `defaultLocationRoot(settings.storage)` so S2 ships without UI change, `channel-storage.spec.ts:226,327` settings block updated.
+- **S3** — controller, views, job kind + NO_REGEN, `/storage` files, nav + test + IA note, instrumentation boot pass, `editor/package.json` `dev:test`/`start:test` gain `FINDMNT_BIN`/`UDISKSCTL_BIN` fakes (`e2e/fixtures/bin/fake-findmnt.mjs`, `fake-udisksctl.mjs`, driven by a control file `test-transcripts/.fake-findmnt.json`), `e2e/storage-locations.spec.ts`.
+- **S4** — badge, `StorageStage` (select + Resume move), `storageActions.ts` (destination by location id, `resumeRelocationAction`), deck + table + page props, `channel-storage.spec.ts` selectors, docs, changelog.
+
+## Invariants
+
+- No data moves in any slice; re-point rewrites links and `dataDir` only. `inspectChannelMedia`, its statuses, and the umtool twin (`cues.mjs checkChannelReachable`) unchanged.
+- Refresh never writes availability; it writes identity only when it changed.
+- Every subprocess: `execa`, timeout, `reject:false`, fail-open to "unknown" — a correctly bind-mounted container channel is never declared unreachable by a probe.
+- No `execa`-importing module reachable from a `"use client"` file (proved by `next build`).
+- Allow-list stays 10; `views/` rules hold; no route retired, no wire shape changed.
+
+## Tests to add
+
+- S1 (`relocateChannelMedia.test.ts`): dir-mtime-only drift verifies after one retry; extra file still refuses "The source has NOT been touched"; a marker written from the first unit's log ends `runOperationBatch` with the stop line and no second dispatch.
+- S2: `storageSettings.test.ts` (sanitizer rules, migration rules 1–3, idempotence, `defaultLocationId` fallback, stale `mediaRoot` beside `locations` ignored); `storageLocations.test.ts` (`locationOfDataDir` longest root, trailing slash, no match); `storageVolumes.test.ts` with a fake findmnt script in the tmp dir (five statuses, warning, timeout, missing binary → unknown).
+- S3: `controller/storageLocations.test.ts` (rollup; re-point happy path over two channels + settings; refuse missing target naming the slug; rollback of channel 1 when channel 2's symlink fails via a read-only channel dir; refuse busy; crash-rerun finishes); `views/storage.test.ts`; e2e `storage-locations.spec.ts` (list/status/channels/delete-refused/add; fake-findmnt mounted-elsewhere → Refresh → Re-point → `readlink` + `config.dataDir` + transcript route 200).
+- S4: `channel-storage.spec.ts` (select `cold`, badge `on Cold`, bulk select) + new Resume-move case (seeded `.relocating.json` phase copy → "resumed an interrupted move", marker gone).
+- `nav.test.ts` at twelve; `architecture.test.ts` green.
+
+## Verification
+
+Per slice, from a worktree: `pnpm -r exec tsc --noEmit`; `pnpm --filter yt-dlp-transcript-common test`; `pnpm -C editor exec tsx --test "app/**/*.test.ts"`; `pnpm --filter editor exec next build`; e2e detached behind the queue lock — S1/S2 `channel-storage`; S3 `+ storage-locations`; S4 both + `channels-*`; full 533+ on the merged tip. Offline against the real host (never a second editor): `tsx -e` calling `probeLocation` on the platter's current automount → expect `available` with identity uuid `09b598d2-…` and the fstab warning; on `/mnt/platter/archilyzer-media` → `mounted-elsewhere` with candidate `/run/media/user/<uuid>/archilyzer-media`.
+
+## Risks
+
+- Twelve nav entries contradicts the IA rule; recorded as an exception. Fallback if reconsidered: a `/cleanup` tab.
+- `udisksctl` under a service session may be polkit-denied: surface stderr, never retry.
+- S1(a) leaves a one-unit race window; (b) absorbs the mtime echo; "Resume move" absorbs the rest; (c) turns the case into a refusal with a reason.
+- Nested roots are allowed; longest match wins — say so in the sanitizer.
+- Follow-up, not here: the saved-video store as a location (absolute pointers need a rewrite).
+
+## Cadence
+
+Plan file → `plans/storage-locations.md` (first commit). Fable reviews. S0 first, alone.
+S1 and S2 as two parallel Opus implementers on `storage/locations-s1` / `-s2`; S3 then S4
+serially on the merged tip; full suite on the final tip. Standing rules: commit small, add
+by path, never boot against `transcripts/`, e2e detached + Monitor, report contract with
+shas, gate outputs, divergences.
+
+## Status log
+
+- 2026-09-17 — plan committed on `main`.
