commit 1ceb88a6e23c2d2696a1051db04439e10e15e8bd
parent f698dc0f88b04633856e3d694b88054ee89a6a8b
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Fri, 2 Oct 2026 02:13:19 -0400
docs: AGENTS.md's media section for the tier (seven things), and the migrate-tier changelog bullet
The data/ symlink paragraph now describes the media link and mediaDir, with
dataDir retired (legacy until migrate-tier); the six things become seven: the
marker's scope, clips/ on the SSD and never tiered, and a media file in
data/<id>/ may be a relative link (removeMediaFile, the hook, isFile hides it).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
2 files changed, 39 insertions(+), 20 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
@@ -263,32 +263,46 @@ changed by `patchChannelConfig` (`common/controller/channels.ts`), never by spre
config read earlier; a per-video sidecar is declared once with `sidecar()`
(`common/lib/sidecar-server.ts`), which refuses a `transcript.<x>.<y>` name.
-## A channel's `data/` may live on another drive
-
-`channels/<slug>/data` can be an **absolute symlink** to `<root>/<slug>/data` on
-another disk, with `config.dataDir` recording the target. The editor's Storage panel
-(channel page → Storage) moves it — to one of the locations configured on `/storage`,
-or to a root typed by hand; nothing else writes that field. **A channel is not tagged
-with its location**: it is on location L iff its `config.dataDir` is under `L.root`,
-which is why re-pointing a location rewrites only links and `dataDir`. The on-disk contract
-`channelDir/data/<id>/…` is unchanged, so **no reader needs to know** — yt-dlp's
-cwd-relative writes, the LMDB index (it stores mtimes, and `rsync -a` preserves them)
-and the export build all keep working with no call-site changes.
-
-Six things that are not optional:
+## A channel's media may live on another drive
+
+A channel's text — transcripts, cues, `metadata.info.json`, every sidecar, `clips/` — is
+always in a REAL `channels/<slug>/data/` on the corpus disk. Only its big files move
+(release 17, the media tier; `common/lib/mediaTier.ts` says which, by name): each becomes a
+RELATIVE link `data/<id>/<name> -> ../../media/<id>/<name>`, and `channels/<slug>/media` is a
+real directory on the corpus disk, or ONE absolute symlink to `<root>/<slug>/media` on
+another disk with `config.mediaDir` recording the target, or absent (a classic channel, its
+big files still real in `data/<id>/`). The editor's Storage panel (channel page → Storage)
+moves `media/` — to one of the locations configured on `/storage`, or to a root typed by
+hand; nothing else writes `mediaDir` but the re-point, the rename and the one-off
+migration. **A channel is not tagged with its location**: it is on location L iff its
+`config.mediaDir` is under `L.root`, which is why re-pointing a location rewrites only the
+`media` links and `mediaDir`. The on-disk contract `channelDir/data/<id>/…` is unchanged,
+so **no reader needs to know** — yt-dlp's cwd-relative writes, the LMDB index (it stores
+mtimes; it `lstat`s a tierable name, and a tier link carries its file's times) and the
+export build all keep working with no call-site changes. **`config.dataDir` is RETIRED**:
+a channel that still carries it, or whose `data/` is itself a link (the pre-release-17
+whole-directory move), is `legacy` — held by every guard, text included — until
+`archilyzer storage migrate-tier <slug>` (editor stopped) brings its text home.
+
+Seven things that are not optional:
- **Never symlink a whole channel dir.** Channel listing filters `isDirectory()` on
`channelsDir` entries (`channels.ts:246,363`), so a symlinked `<slug>/` vanishes from
- the corpus. Only `data/` may be a link.
+ the corpus. Only `media` may be a link (and a legacy channel's `data/`, until it is
+ migrated).
- **An unmounted drive is not an empty channel.** Every enumerator swallows ENOENT on
`data/` as "no videos", which to a runner means *everything is undownloaded*.
`common/lib/channelMedia.ts` is the one module that can tell the two apart;
- `inspectChannelMedia` / `assertChannelMediaReachable` are what the guards call, and a
- job kind declares `needsMedia` in `common/jobs/jobKinds.ts` to be covered by the one
- in `runManagedFunction`. If you add a path that reads `data/`, guard it there.
+ `inspectChannelMedia` / `assertChannelMediaReachable` (a job that opens a big file) /
+ `assertChannelTextReadable` (a reader of the text) are what the guards call, and a job
+ kind declares `needsMedia` or `needsText` in `common/jobs/jobKinds.ts` to be covered by
+ the one in `runManagedFunction`. If you add a path that reads `data/`, guard it there.
- **`channels/<slug>/.relocating.json`** is the in-flight marker. Its presence means
"media is in transition" to every guard and lets an interrupted move resume from its
- `phase`. `deleteChannel` and `renameChannel` refuse while it exists. The saved-video
+ `phase`. Its `scope` says what is moving: `"media"` (the Storage panel's move — the
+ text stays readable, only media writers are held) or `"tier-migration"` (the one-off
+ migration, which rebuilds `data/` — the text is held too, and only `migrate-tier`
+ resumes or clears it). `deleteChannel` and `renameChannel` refuse while it exists. The saved-video
store has its own, one level up: **`transcripts/.relocating-saved-videos.json`**, the
same `{target, direction, startedAt, phase}` shape. Its reader and
`assertSavedVideosStoreWritable` live in `common/lib/savedVideoStore.ts` — in *lib*
@@ -307,10 +321,14 @@ Six things that are not optional:
(`controller/relocateChannelMedia.ts`) stats the root and, when it belongs to a location
carrying a `volume.uuid`, requires the probe's identity to match. It runs from the
preview AND immediately before each copy phase's mkdir.
-- **`data/<id>/clips/` is a cache, and `evictClipWindows` is the only thing that prunes
- it — BY AGE.** Nothing in the editor can know whether a umtool report still cites a
+- **`data/<id>/clips/` is a cache, on the SSD, never tiered, and `evictClipWindows` is
+ the only thing that prunes it — BY AGE.** Nothing in the editor can know whether a umtool report still cites a
window (the manifests are in a umtool project), so there is no reference count and every
surface says so. An evicted window is re-fetchable: the cost is a fetch, not data.
+- **A media file in `data/<id>/` may be a relative symlink into `channels/<slug>/media/`.
+ Remove one with `removeMediaFile`, never `rm`/`remove`; tier one with the hook after
+ every media finalisation, never by hand; a dirent `isFile()` filter over a video dir
+ hides it.**
## The live instances
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -35,6 +35,7 @@
- **A channel's text stays on the fast disk when its media moves, so a slow or unplugged media drive no longer holds its transcripts.** A channel's big files — the audio and the raw live-chat replay — can now live in the channel's own `media` folder, on this disk or another, while its transcripts, cues, metadata and every other small file stay in `data/` where they always were; each big file that moves leaves a small link behind, so everything that opens it by name still finds it. New downloads, transcodes and live-chat normalizes put their big files there as they finish, and every cleanup that deletes audio removes the file the link points to, not just the link. What that changes when a media drive is stalled, unplugged or mid-move: **the index and stats builds never wait on it or are held by it** (a live chat whose transcript cues are out of date keeps the cues the last build read until the drive answers), the channel's report still refreshes (its media size reads as unknown until the drive answers), **digests keep running — even during a move of that channel's media** — and so do normalize, the availability checks, the metadata scan and clip eviction. Transcription, downloads, the backfill lane and anything else that opens the audio are held as before. **A channel moved the old way — its whole `data/` on the other drive — is now shown as "Media layout retired" and held by everything, the builds and digests included, until `archilyzer storage migrate-tier <channel>` brings its text home;** every refusal says so. Deleting a video from its page is refused while its channel's media drive is not reachable, so its audio is never left behind on the drive. Needs a rebuild and restart of the editor.
- **Moving a channel's media now moves only its big files.** The Storage panel's **Move media** copies the channel's audio and raw live-chat replays to `<root>/<channel>/media` on the destination and leaves its transcripts, metadata and every other small file in `data/` on this disk; a channel whose audio is still in `data/` has it put into the channel's `media` folder on this disk first, and the preview says how many files ("2 file(s) tiered first"). **Move back in place** brings the `media` folder home; the links in `data/` are not touched either way. The panel shows the media path and the text path, each with its size. While a channel's media is held — moving, or on a drive that is unplugged or not answering — its text stays readable: the video page and the videos list open as usual and list the audio with its size unknown, the transcript opens, the audio answers "not reachable, try again" (503) instead of "not found", and a digest may run, during a move too; only the jobs and lanes that open the audio wait. A move, its preview and **Resume move** refuse a channel still on the retired whole-directory layout and name `archilyzer storage migrate-tier`; `/storage` counts such channels as unreachable "(n to migrate)". On `/storage` a location's figure is now the media on it, and the corpus volume's row adds every channel's text and clip windows ("text … + clips … on the corpus volume, plus … media of in-place channels"). Re-pointing a location, renaming a channel and deleting one follow the `media` folder; a re-point also carries a channel not yet migrated (its whole `data/` link), so a drive holding both kinds still follows its disk, while renaming such a channel is refused until it is migrated, and **Clear marker** never removes an interrupted migration's marker. Needs a rebuild and restart of the editor.
- **An export build no longer reads a raw live-chat replay from a media drive that is unplugged, not answering or mid-move.** It publishes the live-chat cues already on disk for that channel, and its log says how many are older than their replay and how many videos with a replay and no cues yet were left out of that build.
+- **`archilyzer storage migrate-tier` brings a channel moved the old way onto the media tier: its text comes home to this disk, its big files stay on the drive.** Run it with the editor **stopped** — a real run refuses while the editor answers on its port (`ARCHILYZER_EDITOR_URL`, default `http://localhost:3001`) or while a job on disk belongs to a live process; a dry run only notes it. `archilyzer storage migrate-tier <channel>` does one channel, `--all` every channel still shown as "Media layout retired". Start with `--dry-run`: it writes nothing and prints, per channel, how much text, clip windows, scratch and source video would be copied to this disk, how many audio files and live-chat replays stay on the drive, and whether it fits. A real run copies everything but the audio and the raw live-chat replays into the channel's `data/` on this disk, checks the copy file by file and kind by kind, links each audio file and replay back under its old name with its own time, renames the drive's `<root>/<channel>/data` to `<root>/<channel>/media`, and records `mediaDir` in the channel's `config.json` in place of `dataDir`; then it prints the bytes copied by kind and the links made. Nothing needs re-indexing: every file keeps its time. It needs the copy plus the resume margin free above the disk gate's floor on this disk, and refuses before writing anything when that is not there. `--all` takes the smallest channels first and **stops before omnibased, rekietalaw and the-quartering-rumble**, printing the free space and what each of them would copy; go on with the channel's name, or `--all --include-large`. A run that is stopped or fails partway carries on where it left off when run again, and a channel already done is left alone. The drive keeps its copy of the text until you run it again with `--reclaim`, which deletes those copies and keeps the media.
- **A video whose YouTube subtitles answer "Too Many Requests" (HTTP 429) is downloaded anyway, and YouTube is not put in a cooldown for it.** YouTube refuses a subtitle file per video while the video itself downloads fine; every one of the day's 429s on 2026-10-01 was a subtitle fetch, and each failed its download, put all of YouTube in a cooldown that reached 30 minutes, and deferred the video for 6 hours to fail the same way again. Now that refusal is noted and the download goes on to the audio, as for a video with no captions, so the transcription lane transcribes it; nothing platform-wide is backed off. The video's subtitles are deferred: **Download missing subs** skips them for 6 hours, and from the third time they are refused, for 7 days; a download from the video's own page still fetches them. The download lane's page lists them under **Deferred subtitles**, and the video page says how many times and when. **Download missing subs** also goes on to the next video when one video's subtitles are refused, instead of stopping. Any other subtitle failure (a 403, a missing file, a chat replay that fails) still fails the download, as before. Needs a rebuild and restart of the editor.
- **The download pace adapts to rate limits, a rate limit that outlasts the cooldown holds the platform, and the auto-download lane waits between downloads.** Every yt-dlp run against a platform now waits its platform's current pace between requests: 1 second for YouTube and Rumble, doubled by each real rate limit (up to 16 seconds) and eased back one step after every 5 clean downloads, and one step for every hour with no rate limit; a subtitle-only refusal never raises it. When a platform has failed three times in a row at the 30-minute cooldown, it is **held**: auto-download tries it once an hour instead of every 30 minutes, and until that try is due a manual **Sync**, download or metadata scan on it is refused with a sentence giving its time. A clean try lifts the hold — the lane's, or a manual Sync, download or scan once the try is due, which is how a hold ends while auto-download is off or has nothing to fetch on that platform — and so does a try whose video came down although its subtitles were refused. The lane page keeps a held platform listed until then (saying when the lane is off), with a **Clear hold** button that drops the hold, the cooldown and the raised pace at once and says so in a job log. The auto-download lane now waits **Sleep between downloads** between two downloads on one platform, as a channel's batch downloads always did, plus whatever the pace was raised by; batch downloads add that too. The lane page's **Rate-limit cooldown** box shows held platforms, the raised paces and the deferred subtitles, the lane says when it is idle because a platform is held or it is pausing between downloads, and `archilyzer doctor` warns about a platform in a cooldown or held, and about a raised pace. **Download missing subs** also waits that gap between videos. The four numbers are the new `pacing` block in `settings.json` (SETTINGS.md). **After the restart, YouTube may be held at its first real failure:** its cooldown count from before the update (the subtitle refusals) still stands, so one failure puts it straight past the cap — **Clear hold** on the download lane's page resets it, and a clean download does too. Needs a rebuild and restart of the editor.
