commit ab9de46aad86c183c1fc368a2c45ead5b735cce1
parent 85ab3b6e85c4c8a03619f0d8810d831f3d501f8c
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Fri, 2 Oct 2026 02:28:58 -0400
plans: slice T3's review — L1-L4 and the NITs to their commits, the operator notes (purge after the migration; the Syncthing folder), the gates after the fixes; the changelog bullet and FACTS follow
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
3 files changed, 55 insertions(+), 13 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -35,7 +35,7 @@
- **A channel's text stays on the fast disk when its media moves, so a slow or unplugged media drive no longer holds its transcripts.** A channel's big files — the audio and the raw live-chat replay — can now live in the channel's own `media` folder, on this disk or another, while its transcripts, cues, metadata and every other small file stay in `data/` where they always were; each big file that moves leaves a small link behind, so everything that opens it by name still finds it. New downloads, transcodes and live-chat normalizes put their big files there as they finish, and every cleanup that deletes audio removes the file the link points to, not just the link. What that changes when a media drive is stalled, unplugged or mid-move: **the index and stats builds never wait on it or are held by it** (a live chat whose transcript cues are out of date keeps the cues the last build read until the drive answers), the channel's report still refreshes (its media size reads as unknown until the drive answers), **digests keep running — even during a move of that channel's media** — and so do normalize, the availability checks, the metadata scan and clip eviction. Transcription, downloads, the backfill lane and anything else that opens the audio are held as before. **A channel moved the old way — its whole `data/` on the other drive — is now shown as "Media layout retired" and held by everything, the builds and digests included, until `archilyzer storage migrate-tier <channel>` brings its text home;** every refusal says so. Deleting a video from its page is refused while its channel's media drive is not reachable, so its audio is never left behind on the drive. Needs a rebuild and restart of the editor.
- **Moving a channel's media now moves only its big files.** The Storage panel's **Move media** copies the channel's audio and raw live-chat replays to `<root>/<channel>/media` on the destination and leaves its transcripts, metadata and every other small file in `data/` on this disk; a channel whose audio is still in `data/` has it put into the channel's `media` folder on this disk first, and the preview says how many files ("2 file(s) tiered first"). **Move back in place** brings the `media` folder home; the links in `data/` are not touched either way. The panel shows the media path and the text path, each with its size. While a channel's media is held — moving, or on a drive that is unplugged or not answering — its text stays readable: the video page and the videos list open as usual and list the audio with its size unknown, the transcript opens, the audio answers "not reachable, try again" (503) instead of "not found", and a digest may run, during a move too; only the jobs and lanes that open the audio wait. A move, its preview and **Resume move** refuse a channel still on the retired whole-directory layout and name `archilyzer storage migrate-tier`; `/storage` counts such channels as unreachable "(n to migrate)". On `/storage` a location's figure is now the media on it, and the corpus volume's row adds every channel's text and clip windows ("text … + clips … on the corpus volume, plus … media of in-place channels"). Re-pointing a location, renaming a channel and deleting one follow the `media` folder; a re-point also carries a channel not yet migrated (its whole `data/` link), so a drive holding both kinds still follows its disk, while renaming such a channel is refused until it is migrated, and **Clear marker** never removes an interrupted migration's marker. Needs a rebuild and restart of the editor.
- **An export build no longer reads a raw live-chat replay from a media drive that is unplugged, not answering or mid-move.** It publishes the live-chat cues already on disk for that channel, and its log says how many are older than their replay and how many videos with a replay and no cues yet were left out of that build.
-- **`archilyzer storage migrate-tier` brings a channel moved the old way onto the media tier: its text comes home to this disk, its big files stay on the drive.** Run it with the editor **stopped** — a real run refuses while the editor answers on its port (`ARCHILYZER_EDITOR_URL`, default `http://localhost:3001`) or while a job on disk belongs to a live process; a dry run only notes it. `archilyzer storage migrate-tier <channel>` does one channel, `--all` every channel still shown as "Media layout retired". Start with `--dry-run`: it writes nothing and prints, per channel, how much text, clip windows, scratch and source video would be copied to this disk, how many audio files and live-chat replays stay on the drive, and whether it fits. A real run copies everything but the audio and the raw live-chat replays into the channel's `data/` on this disk, checks the copy file by file and kind by kind, links each audio file and replay back under its old name with its own time, renames the drive's `<root>/<channel>/data` to `<root>/<channel>/media`, and records `mediaDir` in the channel's `config.json` in place of `dataDir`; then it prints the bytes copied by kind and the links made. Nothing needs re-indexing: every file keeps its time. It needs the copy plus the resume margin free above the disk gate's floor on this disk, and refuses before writing anything when that is not there. `--all` takes the smallest channels first and **stops before omnibased, rekietalaw and the-quartering-rumble**, printing the free space and what each of them would copy; go on with the channel's name, or `--all --include-large`. A run that is stopped or fails partway carries on where it left off when run again, and a channel already done is left alone. The drive keeps its copy of the text until you run it again with `--reclaim`, which deletes those copies and keeps the media.
+- **`archilyzer storage migrate-tier` brings a channel moved the old way onto the media tier: its text comes home to this disk, its big files stay on the drive.** Run it with the editor **stopped** — a real run refuses while the editor answers on its port (`ARCHILYZER_EDITOR_URL`, default `http://localhost:3001`) or while a job on disk belongs to a live process; a dry run only notes it. `archilyzer storage migrate-tier <channel>` does one channel, `--all` every channel still shown as "Media layout retired". Start with `--dry-run`: it writes nothing and prints, per channel, how much text, clip windows, scratch and source video would be copied to this disk, how many audio files and live-chat replays stay on the drive, and whether it fits. A real run copies everything but the audio, the raw live-chat replays and any dead `*.temp.*` left by a download's postprocessor into the channel's `data/` on this disk, checks the copy file by file and kind by kind, links each audio file and replay back under its old name with its own time, renames the drive's `<root>/<channel>/data` to `<root>/<channel>/media`, and records `mediaDir` in the channel's `config.json` in place of `dataDir`; then it prints the bytes copied by kind and the links made. Nothing needs re-indexing: every file keeps its time. It needs the copy plus the resume margin free above the disk gate's floor on this disk, and refuses before writing anything when that is not there. `--all` takes the smallest channels first and **stops before omnibased, rekietalaw and the-quartering-rumble**, printing the free space and what each of them would copy; go on with the channel's name, or `--all --include-large`. A run that is stopped or fails partway carries on where it left off when run again, and a channel already done is left alone. The drive keeps its copy of the text, and those dead temps, until you run it again with `--reclaim`, which deletes them and keeps the media. `--reclaim` takes only channels this command migrated, and deletes a copy only where the same file, at the same size, is in the channel's `data/` on this disk; anything else stays on the drive and is listed. Leave it until the editor has run on the migrated channels for a while: until then the drive's copy is a second one. **Superseded auto-subtitles are purged after the migration, not before:** a channel on the retired layout refuses `purge-superseded-auto-subs` like every text job, so a channel's superseded `en-orig` subtitles (7.6 GB on omnibased) are copied to this disk first and purged from there.
- **A video whose YouTube subtitles answer "Too Many Requests" (HTTP 429) is downloaded anyway, and YouTube is not put in a cooldown for it.** YouTube refuses a subtitle file per video while the video itself downloads fine; every one of the day's 429s on 2026-10-01 was a subtitle fetch, and each failed its download, put all of YouTube in a cooldown that reached 30 minutes, and deferred the video for 6 hours to fail the same way again. Now that refusal is noted and the download goes on to the audio, as for a video with no captions, so the transcription lane transcribes it; nothing platform-wide is backed off. The video's subtitles are deferred: **Download missing subs** skips them for 6 hours, and from the third time they are refused, for 7 days; a download from the video's own page still fetches them. The download lane's page lists them under **Deferred subtitles**, and the video page says how many times and when. **Download missing subs** also goes on to the next video when one video's subtitles are refused, instead of stopping. Any other subtitle failure (a 403, a missing file, a chat replay that fails) still fails the download, as before. Needs a rebuild and restart of the editor.
- **The download pace adapts to rate limits, a rate limit that outlasts the cooldown holds the platform, and the auto-download lane waits between downloads.** Every yt-dlp run against a platform now waits its platform's current pace between requests: 1 second for YouTube and Rumble, doubled by each real rate limit (up to 16 seconds) and eased back one step after every 5 clean downloads, and one step for every hour with no rate limit; a subtitle-only refusal never raises it. When a platform has failed three times in a row at the 30-minute cooldown, it is **held**: auto-download tries it once an hour instead of every 30 minutes, and until that try is due a manual **Sync**, download or metadata scan on it is refused with a sentence giving its time. A clean try lifts the hold — the lane's, or a manual Sync, download or scan once the try is due, which is how a hold ends while auto-download is off or has nothing to fetch on that platform — and so does a try whose video came down although its subtitles were refused. The lane page keeps a held platform listed until then (saying when the lane is off), with a **Clear hold** button that drops the hold, the cooldown and the raised pace at once and says so in a job log. The auto-download lane now waits **Sleep between downloads** between two downloads on one platform, as a channel's batch downloads always did, plus whatever the pace was raised by; batch downloads add that too. The lane page's **Rate-limit cooldown** box shows held platforms, the raised paces and the deferred subtitles, the lane says when it is idle because a platform is held or it is pausing between downloads, and `archilyzer doctor` warns about a platform in a cooldown or held, and about a raised pace. **Download missing subs** also waits that gap between videos. The four numbers are the new `pacing` block in `settings.json` (SETTINGS.md). **After the restart, YouTube may be held at its first real failure:** its cooldown count from before the update (the subtitle refusals) still stands, so one failure puts it straight past the cap — **Clear hold** on the download lane's page resets it, and a clean download does too. Needs a rebuild and restart of the editor.
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -4015,7 +4015,8 @@ and the measurements.
`scope` is not `tier-migration` is refused, `dataDir` must be `<root>/<slug>/data` and agree with the
link, `mediaDir` must be absent; preflight: the old tree is a directory, `assertRelocationRootPresent`,
`<root>/<slug>/media` and `channels/<slug>/media` absent, one walk classifying every video dir's entries
- (a tierable REGULAR file stays; everything else is listed), the space rule `copy bytes − bytes already
+ (a tierable REGULAR file stays; a postprocessor's dead `*.temp.*` — `isLeftOnPlatter` — stays too,
+ unlinked; everything else is listed), the space rule `copy bytes − bytes already
in data.incoming + margin ≤ free(channelsDir) − minFreeDiskGB` (margin 0 when the gate is off, the
mover's rule); copy: marker `{target: <root>/<slug>/media, direction: "out", phase: "copy", scope:
"tier-migration"}`, the NUL list at `channels/<slug>/.tier-migration.files`, `rsync -a -r --from0
@@ -4026,16 +4027,20 @@ and the measurements.
(EEXIST = already, if it is the same link) with `lutimes` to the file's times; marker → `swap`;
swap: platter rename `data → media`, the `media` link, unlink the old `data` link, rename
`data.incoming → data`, `patchChannelConfig(slug, { mediaDir }, { unset: ["dataDir"] })` — each step
- `linkOrDirState`s first, so a rerun after a kill at any point finishes; `--reclaim` (marker phase
- `reclaim`): every entry of `<root>/<slug>/media/<id>/` that is not a tierable regular file goes, an
- emptied dir goes; done: marker cleared, the channel must then inspect `ok`. A channel on the new layout
- is a no-op (and takes `--reclaim` on its own). `--all` = every legacy channel plus any carrying a
+ `linkOrDirState`s first, so a rerun after a kill at any point finishes; the swap leaves
+ `channels/<slug>/.tier-migration.reclaim` (the reclaim note); `--reclaim` (marker phase `reclaim`, only
+ on a channel carrying the note — a named slug without it is refused, `--all --reclaim` takes only those):
+ in `<root>/<slug>/media/<id>/` a tierable regular file stays, a dead `*.temp.*` goes, and any other entry
+ goes only when one `lstat` finds its twin at `channels/<slug>/data/<id>/<name>` (a dir for a dir, else the
+ same size) — the rest is kept and listed; an emptied dir goes; the note is removed. Done: marker cleared,
+ the channel must then inspect `ok`. A channel on the new layout is a no-op. `--all` = every legacy channel plus any carrying a
`tier-migration` marker (those first), smallest copy first, the three big-text channels
(`LARGE_TEXT_CHANNELS`: omnibased, rekietalaw, the-quartering-rumble) deferred with the free space and
- each one's copy bytes printed unless `--include-large`; a real run stops at the first channel it cannot
+ each one's copy bytes (from the ordering walk) printed unless `--include-large`; a real run stops at the first channel it cannot
finish, a dry run reports every channel and projects the space the earlier ones would take. No index
rebuild: `buildIndex` over a migrated fixture reports 0 added, 0 changed, 0 removed
- (`bin/migrate-media-tier.test.ts`).
+ (`bin/migrate-media-tier.test.ts`). `purge-superseded-auto-subs` is `needsText`, so it is refused on a
+ legacy channel and runs AFTER the migration. `.syncthing.<name>.tmp` is scratch (`lib/mediaTier.ts`).
## Channel priority (verified 2026-09-11) — one tier per channel, four compiled trees
diff --git a/plans/release-17.md b/plans/release-17.md
@@ -2039,13 +2039,50 @@ ENVIRONMENT.md: no schema text changed (`docs files --check`, `settings example
#### Found and left
-- `lib/envVars.ts` (not this slice's): `ARCHILYZER_EDITOR_URL`'s `readBy` does not list the migration, which
- now reads it; ENVIRONMENT.md is generated from it.
-- CHANNEL.md's `dataDir` row (`channelConfigSchema.ts`, T1's) says it is "never written by anything but that
- migration"; T2's re-point also rewrites a legacy channel's `dataDir` the retired way.
+- `lib/envVars.ts`'s `ARCHILYZER_EDITOR_URL.readBy` and CHANNEL.md's `dataDir` row were stale; both fixed in
+ the review round (below).
- The dry run (and `--all`'s ordering) walks every legacy channel's old tree on its drive, one `lstat` per
entry, and the stop walks the big three to print what each would copy; a real run walks a channel again in
- its own preflight. Read-only, but minutes on a platter.
+ its own preflight. Read-only, but minutes on a platter. (Since the review the stop no longer walks the big
+ three a second time.)
- The space rule counts `clips/` (nuxanor-kick's 15 GB) as text, as the plan measured it.
+#### Operator notes
+
+- **`purge-superseded-auto-subs` runs AFTER the migration, not before** (the plan said before): since T1
+ the kind is `needsText` and a `legacy` channel's text is refused, so the editor refuses the purge on
+ omnibased and HasanAbiVODs3 until they are migrated. omnibased's ≈ 7.6 GB of `en-orig.vtt` lands on the
+ corpus disk first and is purged there.
+- **`transcripts/channels` is a Syncthing folder (paused).** Syncthing syncs a symlink as a link; after the
+ migration the 13 channels' text (≈ 64 GB with clips) is a real tree inside that folder, where today it is
+ behind a `data` link. Nothing moves while the folder is paused; unpausing it would send that text to the
+ peers.
+- `--reclaim` after the editor has run on the migrated channels for a while: until then the drive's text
+ copy is a second copy.
+
+#### Review (SHIP AFTER FIXES) and the fixes
+
+The review (`$T/T3-review.md`) found four LOWs and five NITs. Rulings: `--all --reclaim` is limited to
+channels this tool migrated; a reclaim deletes only what has a same-size twin on the corpus disk; dead
+`*.temp.*` stay out of the copy; a pulse that does not answer within 3 s counts as a running editor (as
+built).
+
+| Finding | Fix |
+|---|---|
+| L1 `--all --reclaim` walked every channel on the media tier | `beaa6a44`: the swap leaves `channels/<slug>/.tier-migration.reclaim` (`RECLAIM_NOTE`); `--all --reclaim` takes only channels carrying it, a named `--reclaim` without it is refused ("migrate-tier did not migrate it, or its reclaim is done"), a reclaim removes it. Tests: a Storage-panel-moved channel is not walked by `--all --reclaim` and is refused by name; a second `--reclaim` refuses and touches nothing; a reclaim killed after its deletes resumes from its marker and removes the note. |
+| L2 a later `--reclaim` deleted without asking whether the corpus disk still has the file | `beaa6a44`: an entry goes only when one `lstat` finds its twin at `channels/<slug>/data/<id>/<name>` (a directory for a directory, else the same size); everything else is kept and listed (`reclaimKept`, and a "kept N entries" line). Test: a removed and a rewritten text file are kept on the platter, the rest taken. |
+| L3 the purge cannot run first | Operator note above and in the changelog bullet. |
+| L4 dead `*.temp.*` carried to the corpus disk (15 GB on nuxanor-kick) | `beaa6a44`: `isLeftOnPlatter` (scratch by the classifier AND `*.temp.*`) is neither copied nor linked; it stays on the platter and `--reclaim` deletes it; the walk reports it ("dead postprocessor temps left there unlinked"). `96191600`: `/^\.syncthing\..*\.tmp$/` joins the classifier's scratch patterns (+ the table row). Tests: the fixture carries `source-media.temp.mp4` (left, then reclaimed) and `.syncthing.audio.mp3.tmp` (scratch, carried). |
+| NIT AGENTS.md's corpus table | `e01db98a`: "a big file in `data/<id>/` may be a relative link into `media/`, which may be an absolute SYMLINK to another drive (and a legacy channel's whole `data/` is one, until migrated)". |
+| NIT `ARCHILYZER_EDITOR_URL.readBy` | `e01db98a`: names `common/bin/migrate-media-tier.ts`; ENVIRONMENT.md regenerated. |
+| NIT the `dataDir` field doc | `e01db98a`: "Written only by the re-point of a storage location …; removed by that migration" (`lib/channelConfig.ts`); CHANNEL.md regenerated. |
+| NIT the stop re-walked the big three | `beaa6a44`: the stop reports the sizes the ordering walk measured (the big three are measured once, in the ordering pass, whether or not `--include-large`). |
+| NIT apparent bytes vs block slack | Left: the 2 GB margin covers it. |
+
+**Gates after the fixes** (logs `$T/T3-test4.log`, `$T/T3-tsc2.log`, `$T/T3-common2.log`): tsc (all
+workspaces) clean; the migration test **15/15** (12 + 3 new); the classifier test 37/37; `docs files
+--check`, `docs env --check`, `settings example --check` clean after the regeneration; **common 2,671 passed, 0 failed, 0 skipped** (2,667 + the three new
+migration cases + the classifier's Syncthing row). Not re-run: editor unit, test:scripts, the editor build
+(no editor, script or umtool file touched). Privacy: 0 added lines carry the user or host name.
+
## Rollout
