archilyzer

commit e856583a10f0727f5eb2f6db9205d3bdf0ebcf79
parent 1ceb88a6e23c2d2696a1051db04439e10e15e8bd
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date:   Fri,  2 Oct 2026 02:14:07 -0400

plans: slice T3's record — the migration as shipped, its gates; FACTS "A channel's media is tiered"

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Diffstat:
M
plans/FACTS.md
 | 
54
++++++++++++++++++++++++++++++++++++++++++++++++++++++
M
plans/release-17.md
 | 
135
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

2 files changed, 189 insertions(+), 0 deletions(-)
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -3983,6 +3983,60 @@ Supersedes, where they differ, the mover and storage-location facts above (the w
 - **`noCorpusWalkInRenderPaths.test.ts` greps render-path files for the walkers' NAMES**, comments
   included: naming `measureTree` in a comment of a file a page imports fails it.
 
+### A channel's media is tiered — the model, the hook, the guards, the migration (release 17, verified 2026-10-02)
+
+The whole of release 17's storage model in one place; the T1 and T2 sections above carry the detail
+and the measurements.
+- **The model.** `channels/<slug>/data/` is ALWAYS a real directory on the corpus disk and holds the
+  text: transcripts, cues, `metadata.info.json`, every sidecar, `clips/` (never tiered), scratch, and
+  `source-media.*` (the saved-video store renames it, so it is never a link). A tierable file
+  (`isTierable`: `audio.<ext>`, `transcript.live_chat.json`) may be a RELATIVE link
+  `data/<id>/<name> -> ../../media/<id>/<name>` carrying the file's times. `channels/<slug>/media` is a
+  real dir (tiered in place), ONE absolute link to `<root>/<slug>/media` with `config.mediaDir`
+  (relocated), or absent (classic). Classification is BY NAME (`lib/mediaTier.ts`, pure).
+- **The hook** (`lib/mediaTier-server.ts`): `tierMediaFile`/`tierVideoDir`/`tierChannelMedia` after every
+  media finalisation, never throwing; `removeMediaFile`/`removeVideoDirMedia` for every delete. It writes
+  nothing while a marker stands.
+- **The guards** (`lib/channelMedia.ts`): `assertChannelMediaReachable` for a kind with `needsMedia`,
+  `assertChannelTextReadable` for one with `needsText`; statuses `in-place | ok | unreachable |
+  in-transition | inconsistent | stalled | legacy`. A `media`-scoped marker holds media writers only; a
+  `tier-migration` marker holds the text too.
+- **The mover** (`controller/relocateChannelMedia.ts`) moves `media/` only, tiering a classic channel
+  first; it refuses a `legacy` channel and a `tier-migration` marker with `tierMigrationRefusal`.
+- **`legacy`** = `config.dataDir` recorded or `data/` a link: the pre-release-17 whole-directory move.
+  T2's re-point and rename keep `dataDir` = `<root>/<slug>/data` (rename refuses a legacy channel), so the
+  migration may derive everything from that shape.
+- **The migration** (`common/bin/migrate-media-tier.ts`, `archilyzer storage migrate-tier
+  <slug>…|--all`, release 17 slice T3). Editor STOPPED: a real run refuses while
+  `<ARCHILYZER_EDITOR_URL>/api/pulse` answers (a timeout counts as running) or a `running`/`queued` meta in
+  `.jobs/` belongs to a live pid that is not its own (a pid-less `running` meta written after the
+  machine booted counts too; files older than the boot are not read); a dry run only notes it and writes
+  nothing. Per channel: the plan reads the corpus disk (config, `data` link, marker) — a marker whose
+  `scope` is not `tier-migration` is refused, `dataDir` must be `<root>/<slug>/data` and agree with the
+  link, `mediaDir` must be absent; preflight: the old tree is a directory, `assertRelocationRootPresent`,
+  `<root>/<slug>/media` and `channels/<slug>/media` absent, one walk classifying every video dir's entries
+  (a tierable REGULAR file stays; everything else is listed), the space rule `copy bytes − bytes already
+  in data.incoming + margin ≤ free(channelsDir) − minFreeDiskGB` (margin 0 when the gate is off, the
+  mover's rule); copy: marker `{target: <root>/<slug>/media, direction: "out", phase: "copy", scope:
+  "tier-migration"}`, the NUL list at `channels/<slug>/.tier-migration.files`, `rsync -a -r --from0
+  --files-from=… --partial --info=progress2` into `channels/<slug>/data.incoming/` (`-r` explicitly:
+  `--files-from` turns off `-a`'s recursion and `clips/` is a directory), verify = the same list's
+  `--dry-run --itemize-changes` empty (a `.d..t` directory-time line is not content) AND per-kind
+  (`text | clips | scratch | source`) files and bytes equal, then a link per tierable file
+  (EEXIST = already, if it is the same link) with `lutimes` to the file's times; marker → `swap`;
+  swap: platter rename `data → media`, the `media` link, unlink the old `data` link, rename
+  `data.incoming → data`, `patchChannelConfig(slug, { mediaDir }, { unset: ["dataDir"] })` — each step
+  `linkOrDirState`s first, so a rerun after a kill at any point finishes; `--reclaim` (marker phase
+  `reclaim`): every entry of `<root>/<slug>/media/<id>/` that is not a tierable regular file goes, an
+  emptied dir goes; done: marker cleared, the channel must then inspect `ok`. A channel on the new layout
+  is a no-op (and takes `--reclaim` on its own). `--all` = every legacy channel plus any carrying a
+  `tier-migration` marker (those first), smallest copy first, the three big-text channels
+  (`LARGE_TEXT_CHANNELS`: omnibased, rekietalaw, the-quartering-rumble) deferred with the free space and
+  each one's copy bytes printed unless `--include-large`; a real run stops at the first channel it cannot
+  finish, a dry run reports every channel and projects the space the earlier ones would take. No index
+  rebuild: `buildIndex` over a migrated fixture reports 0 added, 0 changed, 0 removed
+  (`bin/migrate-media-tier.test.ts`).
+
 ## Channel priority (verified 2026-09-11) — one tier per channel, four compiled trees
 
 Branch `channel-priority/s5`, off S0's `28bfee3`, merging `s1`–`s4` and closing the twelve
diff --git a/plans/release-17.md b/plans/release-17.md
@@ -1913,4 +1913,139 @@ stay placed by their retired `dataDir`; the digest job stands for the digest lan
   exit 0, 44 s; e2e (`channels-storage-columns`, `bulk-actions`, `maybe-missing`, `fetch-window`)
   **22 passed, 0 failed, 1.7 min**.
 
+### Slice T3, as shipped — the one-off migration, and the records (2026-10-02)
+
+Branch `r17/media-tier-migrate` off `main` `b6d9c1e2` (U1, U2, XP, D0, T1, RL and T2 merged), worktree
+`~/Projects/r13-build-image` (editor 5201, test 5211, export 5210), one Opus implementer. Scratch files
+`T3-*` in the job's `tmp`. The plan is "The one-off migration" above and the T3 row; T1 left the
+`lutimes` rule, T2 left `relocatedDataDir`, the `<root>/<slug>/data` invariant and `tierMigrationRefusal`.
+
+**What it does.**
+- **`archilyzer storage migrate-tier <slug>…|--all [--order smallest] [--include-large] [--dry-run]
+  [--reclaim]`** (`common/bin/migrate-media-tier.ts`, one `script([...])` row in `archilyzer.ts` beside
+  `migrate channel-priority`) brings a `legacy` channel — `data/` an absolute link to `<root>/<slug>/data`,
+  `config.dataDir` — onto the media tier: its text is copied home and its big files stay on the drive.
+- **Editor stopped.** A real run refuses (exit 2, nothing read further) while `<ARCHILYZER_EDITOR_URL,
+  default http://localhost:3001>/api/pulse` answers — any HTTP answer, or a connection that does not
+  answer within 3 s — or while `.jobs/` holds a `running`/`queued` meta whose `pid` is alive and is not
+  its own (a pid-less `running` meta written after the machine booted counts too; meta files older than
+  the boot are not read). `migrate channel-priority` checked nothing; this is new. A dry run only notes it.
+- **Per channel**, exactly the plan's phases, each refusing before it writes: the plan reads the corpus
+  disk (a marker whose `scope` is not `tier-migration` is refused naming the Storage panel; `dataDir` must
+  agree with the `data` link and be `<root>/<slug>/data`; `mediaDir` must be absent); preflight — the old
+  tree is a directory, `assertRelocationRootPresent(root)`, `<root>/<slug>/media` and
+  `channels/<slug>/media` absent, one walk classifying every video dir (`inventoryTree`: a tierable
+  REGULAR file stays, everything else is listed and measured as `text | clips | scratch | source`), the
+  space rule `copy − already in data.incoming + margin ≤ free(channelsDir) − minFreeDiskGB` (margin 0 with
+  the gate off, the mover's rule); `copy` — the marker, the NUL list (`channels/<slug>/.tier-migration.files`),
+  `rsync -a -r --from0 --files-from=… --partial --info=progress2` into `data.incoming/` with the mover's
+  decile progress lines, the verify (the same list's `--dry-run --itemize-changes` empty, a `.d..t`
+  directory-time line not counted, AND per-kind files and bytes equal), then one relative link per
+  tierable file with `lutimes` to its times (EEXIST = already when it is the same link); `swap` — the
+  platter rename, the `media` link, the old `data` link unlinked, `data.incoming → data`,
+  `patchChannelConfig(slug, { mediaDir }, { unset: ["dataDir"] })`, every step `linkOrDirState` first;
+  `reclaim` (only `--reclaim`, or a reclaim marker being resumed) — every entry of
+  `<root>/<slug>/media/<id>/` that is not a tierable regular file goes, an emptied dir goes; done — the
+  marker cleared, the channel must then inspect `ok`, and the bytes by kind, links made and media left on
+  the drive are printed.
+- **Resumable and idempotent.** A `tier-migration` marker is resumed from its phase (`copy` redoes the
+  copy and verify — rsync sends only what is missing — and finds its links; `swap` and `reclaim` finish);
+  a channel already on the new layout is `already` and writes nothing (with `--reclaim` it reclaims). A
+  dry run writes nothing at all — no marker, list, directory or config — and says what it would do.
+- **`--all`** = every legacy channel plus any carrying a `tier-migration` marker (resumed first), smallest
+  copy first, with the three big-text channels (`LARGE_TEXT_CHANNELS`) deferred: the run ends with "STOPPED
+  before the big-text channels: …", the corpus disk's free space, each one's copy bytes, and "Continue with
+  `archilyzer storage migrate-tier <slug>` one at a time, or `archilyzer storage migrate-tier --all
+  --include-large`" (exit 0). A real run stops at the first channel it cannot finish (exit 1, "run --all
+  again after fixing it"); a dry run reports every channel and subtracts the space the earlier ones would
+  take.
+- **`clips/`** is `text` by the classifier (`classifyEntry("clips")`, pinned) and is carried to the corpus
+  disk with the text, its own kind in the counts.
+
+**Deviations from the plan** (one sentence each):
+1. The copy carries `source-media.*` (media but never tierable) with the text, as kind `source`: the plan's
+   "text + scratch" would have left it on the platter with no link, and `--reclaim` would then delete it.
+2. rsync takes `-r` explicitly: `--files-from` turns off `-a`'s recursion, and `clips/` and a scratch dir
+   are directories to carry whole.
+3. "Status must be `ok`" is read for the retired layout (which `inspectChannelMedia` answers `legacy`,
+   never `ok`, without touching the drive): `dataDir` and the link agree, have the fixed shape, and the old
+   tree is a directory.
+4. A dry run does not refuse a running editor, it notes it: it reads only, and the live dry run may come
+   before the restart.
+5. "A running meta newer than the last boot": nothing on disk records the editor's boot, so the rule is a
+   live writer pid (or a pid-less `running` meta after the machine's boot).
+6. The continuation after the stop is `--all --include-large` or per slug; `--order smallest` is the only
+   order and `--all`'s default (any other value is a usage error).
+7. A reclaim writes its own marker phase (`reclaim`, `scope: "tier-migration"`) so a killed reclaim resumes,
+   and `--all --reclaim` also reclaims every channel already on the new layout (`mediaDir`, no `dataDir`) —
+   one the mover moved has only tierable files there, so nothing is taken.
+8. The NUL list lives beside the marker in the channel dir (the tool runs on the operator's machine, with
+   no job scratch dir), and is removed after the swap.
+
+No helper was added to `mediaTier-server.ts`; the migration uses `relocatedDataDir`, `relocatedMediaDir`,
+`tierLinkTarget`, `channelMediaLink`, `linkOrDirState`, `rsyncTree`, `makeProgressSink`, the marker
+writers, `assertRelocationRootPresent`, `rootOfRelocatedMediaDir` and `patchChannelConfig` as they are.
+
+**Records.** `AGENTS.md`: the section is now "A channel's media may live on another drive" (the `media`
+link, `mediaDir`, `dataDir` retired and `legacy` until `migrate-tier`); "Six things" → seven — the seventh
+exactly as "Order" above states it, the `.relocating.json` bullet gains `scope`, the `clips/` bullet "on
+the SSD, never tiered", and the first two bullets name the `media` link and the text guard. `plans/FACTS.md`:
+"A channel's media is tiered — the model, the hook, the guards, the migration". SETTINGS.md, CHANNEL.md and
+ENVIRONMENT.md: no schema text changed (`docs files --check`, `settings example --check`, `docs env
+--check` all clean), so nothing regenerated. The `[Unreleased]` bullet in `editor/CHANGELOG.md`.
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `38a12ea3` | `common:` `archilyzer storage migrate-tier` — the migration, the editor check, `--all` with the stop, the CLI row; `bin/migrate-media-tier.test.ts` (12) |
+| `587b0946` | `docs:` AGENTS.md's media section and the seven things; the changelog bullet |
+| this commit | `plans:` this section, FACTS "A channel's media is tiered" |
+
+#### Gates (logs `$T/T3-*.log`)
+
+- **tsc** (all workspaces) clean at `38a12ea3` (the later commits change no code).
+- **common:** **2,667 passed, 0 failed, 0 skipped** (2,655 at T2's tip + this slice's 12).
+- **The migration test** (`bin/migrate-media-tier.test.ts`, real rsync, a tmp "platter" beside a corpus, a
+  legacy channel with text, `audio.mp3`, `audio.opus`, a raw live chat, `clips/`, `source-media.mp4`,
+  `audio.m4a.part`, `audio.tmp-1234.mp3` and a media-less video): **12/12** — `clips/` is text; a dry run
+  writes nothing (a byte/mtime/link snapshot of the corpus and the platter is unchanged) and counts 3
+  tierable, 1 clip, 1 source, 2 scratch; a run migrates (every tierable file a relative link with the file's
+  mtime by `lstat`, readable through it, its bytes on the platter; `source-media.*`, `.part` and the temp
+  stay real with their mtimes; `clips/` carried; no marker, list or `data.incoming` left; `inspectChannelMedia`
+  `ok`, text readable) and the rerun writes nothing; `--reclaim` in the same run and later (`already`, 0 taken,
+  nothing written), and after a plain run (dry count = real count); a media move's marker refused with and
+  without a `scope`, nothing touched; a running editor refuses a real run (exit 2, nothing touched) and a dry
+  run notes it; `editorRunningReason` (refused port, an answer, a timeout, a live pid, a dead pid, a pid-less
+  meta, a meta from before the boot); a kill at each of the eight steps from `copied` to `config-written` —
+  the channel reads `in-transition` with its text held and `--all` would pick it — and the rerun resumes
+  from `copy` or `swap` and ends migrated; the free-space stop (6 GB free, a 5 GB floor and a 2 GB margin:
+  refused, nothing written; 8 GB: migrated); `--all` takes the smaller channel first and stops before
+  `omnibased` with the free space and `--all --include-large`, which then migrates it; and an index built
+  over the classic layout, the channel put on the retired layout and migrated: the live chat's cues read
+  fresh and the next `buildIndex` reports **0 added, 0 changed, 0 removed**, none held.
+- **Editor unit:** 109/109 (no editor code touched). **test:scripts:** not run — no file it covers
+  (`scripts/`, umtool) was touched. **Capped editor build:** not needed — no editor code touched.
+- **CLI smoke** against a scratch corpus in `$T`: usage via `archilyzer storage migrate-tier --help`;
+  `--all --dry-run` notes the running editor and finds no channel; `--all` refuses (exit 2) naming the
+  editor's `/api/pulse` answer; no slug and no `--all`, and `--order biggest`, are usage errors (exit 2). The
+  probe's one GET reached the live editor's `/api/pulse` (read-only); nothing else was pointed at the
+  primary checkout.
+- **e2e:** none named for this slice.
+- **Privacy gate:** 0 added lines carry the user or host name (`git diff main`, counts only; the one file the
+  whole-file grep names is `plans/FACTS.md`, with the same count as on `main`). No identifier ends in the
+  refused parent suffix.
+- **Numbers tool:** none. The live dry run is the parent's.
+
+#### Found and left
+
+- `lib/envVars.ts` (not this slice's): `ARCHILYZER_EDITOR_URL`'s `readBy` does not list the migration, which
+  now reads it; ENVIRONMENT.md is generated from it.
+- CHANNEL.md's `dataDir` row (`channelConfigSchema.ts`, T1's) says it is "never written by anything but that
+  migration"; T2's re-point also rewrites a legacy channel's `dataDir` the retired way.
+- The dry run (and `--all`'s ordering) walks every legacy channel's old tree on its drive, one `lstat` per
+  entry, and the stop walks the big three to print what each would copy; a real run walks a channel again in
+  its own preflight. Read-only, but minutes on a platter.
+- The space rule counts `clips/` (nuxanor-kick's 15 GB) as text, as the plan measured it.
+
 ## Rollout