commit f3a760706632ac7f8627943cb7192725e7f74c4c
parent 0f6cafceac44e64be36430cb1856677b1c13df29
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Thu, 1 Oct 2026 23:33:40 -0400
plans: slice RL — the ruling, the FACTS entry, the dated note on the 2026-09-25 postmortem, the changelog bullets
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
4 files changed, 110 insertions(+), 0 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -20,6 +20,8 @@
- **A form whose save is refused keeps what you typed.** Every editor form put its plain fields back to the stored values when its save was refused — a site's ID rejected, a page size out of range, a slug already taken — so everything typed had to be typed again. A refused save now leaves every field as you left it, beside the reason: **Settings**; a site's form (new and existing); the hub's config on `/sites`; **Cut release**; a channel's form (new and **Configure**), **Rename** and **Delete**; a video's **Delete directory**; **Drive health timing** on `/storage`; the backup config on `/saved-videos`; the sync operation's controls; the **Digest**, **Diarization**, **Speaker attribution** and **Speaker work lane** settings; and the worker list on `/workers`. A save that succeeds behaves as before, with one difference you may notice: a drop-down, and a checkbox or choice that the page tracks as you change it (a cadence, a worker's **Enabled**, a social link's **Keep in header**, a site membership, a site's accent), now shows what was saved. A form's own drop-downs used to go back to what the page had loaded with until a reload, and a second save from the same page sent that old choice again; the others went back until the page next refreshed itself (every 5 seconds by default).
- **A media move no longer starts over a job that is writing into the channel, holds the channel's writers while it runs, and makes its copy match the source before it verifies — so a transcription or a download during a move cannot fail it.** A move that has waited its turn behind other moves now checks again when it starts: if a job is running on the channel, or an auto-queue lane is working on one of its videos, it stops at once and says which ("a transcription of abc123 is running (Transcribe all, job …) — wait for it or cancel it"), with nothing copied — and a job you have just cancelled counts until it has actually stopped ("is stopping … — wait for it to stop"); **Preview** says the same, and the Storage panel's blocked message now names the job too. While a move's marker stands, the channel is held: every lane skips it, and every job that reads or writes its media (single-video transcriptions, downloads and transcodes and the availability checks now included) refuses to start, including one that was already queued when the move began. The rack shows a **media held** chip in the channel's Tier cell and the Storage panel says "Held: its media is moving"; both go when the move finishes or its marker is cleared. The copy is now followed by a pass that makes the destination copy match the source — files the source no longer has are removed from the copy, never from the source — so a file written or deleted during the copy (a transcriber's scratch folder, say) no longer fails the check, and **Resume move** finishes a move whose copy holds such leftovers. Every file removed from a copy is listed in the move's log, and **Preview** says so when a copy from an earlier attempt is already there. If the source keeps changing, the move stops and lists what differs: extra on the destination, missing there, or changed. A new **Reconcile and resume** button beside **Resume move** lists those differences, makes the copy match and finishes the move, so no file has to be deleted by hand. The saved-video store's move does the same matching and the same check before it starts. Needs a rebuild and restart of the editor.
- **Connecting an X account opens your own browser, and the X fetchers can use your everyday browser's X login instead.** **Settings → X account session → Connect X account** used to open Playwright's bundled Chromium with its automation signals on (the "controlled by automated test software" bar, `navigator.webdriver`): Google's sign-in refused it and X's own login form stalled in it. It now opens your Chromium or Chrome when one is installed (`ARCHILYZER_X_BROWSER` names another; Playwright's bundled Chromium otherwise), without those signals. Google's sign-in may still refuse an embedded browser; X's password login is the reliable path. A new **Login source** choice (`social.x.cookieSource` in `settings.json`) says where the X fetchers' login comes from: **Browser login** hands gallery-dl `--cookies-from-browser` with your `cookiesFromBrowser` on every fetch, so the login lasts as long as you stay logged in to x.com in that browser and no window is needed; **Connected profile** is the session broker, as before. Left on **Automatic**, it is the browser login when `cookiesFromBrowser` is set and no profile is connected, and the profile otherwise. **Check** says which source is in use, whether an X login is visible in it and when it was last used (the browser's cookies are read from a private copy, never written; this reads Firefox's, and gallery-dl reads Chromium's itself). Needs a rebuild and restart of the editor.
+- **A video whose YouTube subtitles answer "Too Many Requests" (HTTP 429) is downloaded anyway, and YouTube is not put in a cooldown for it.** YouTube refuses a subtitle file per video while the video itself downloads fine; every one of the day's 429s on 2026-10-01 was a subtitle fetch, and each failed its download, put all of YouTube in a cooldown that reached 30 minutes, and deferred the video for 6 hours to fail the same way again. Now the subtitle failure is noted and the download goes on to the audio, as for a video with no captions, so the transcription lane transcribes it; nothing platform-wide is backed off. The video's subtitles are deferred: **Download missing subs** skips them for 6 hours, and from the third time they are refused, for 7 days; a download from the video's own page still fetches them. The download lane's page lists them under **Deferred subtitles**, and the video page says how many times and when. **Download missing subs** also goes on to the next video when one video's subtitles are refused, instead of stopping. Needs a rebuild and restart of the editor.
+- **The download pace adapts to rate limits, a rate limit that outlasts the cooldown holds the platform, and the auto-download lane waits between downloads.** Every yt-dlp run against a platform now waits its platform's current pace between requests: 1 second for YouTube and Rumble, doubled by each real rate limit (up to 16 seconds) and eased back one step after every 5 clean downloads; a subtitle-only refusal never raises it. When a platform has failed three times in a row at the 30-minute cooldown, it is **held**: auto-download tries it once an hour instead of every 30 minutes, a clean try lifts the hold, and a manual **Sync**, download or metadata scan on it is refused with a sentence giving the next try's time. The auto-download lane now waits **Sleep between downloads** between two downloads on one platform, as a channel's batch downloads always did, plus whatever the pace was raised by; batch downloads add that too. The lane page's **Rate-limit cooldown** box shows held platforms, the raised paces and the deferred subtitles, the lane says when it is idle because a platform is held or it is pausing between downloads, and `archilyzer doctor` warns about a platform in a cooldown or held, and about a raised pace. The four numbers are the new `pacing` block in `settings.json` (SETTINGS.md). Needs a rebuild and restart of the editor.
## [0.11.0] - 2026-09-30
- **Transcripts that arrived after a video was first seen are counted.** The stats behind the homepage, the hub and every site's charts were cached per video and refreshed only when the video's metadata changed, so a transcript that came later — a Whisper run days after the download, or a video downloaded after the last index build — never reached them, and a video with YouTube captions alone had no transcription date. Counts and charts were low; the homepage could show a site with 0 transcripts, 0 channels and 0 hours while it served its videos. A stat is now also redone whenever the index re-reads the video, every transcript has a date, and a captioned video is dated by when its captions arrived rather than by a later Normalize run, so its place on "Transcribed over time" can move. **After updating, rebuild and restart the editor before anything else:** until then, **Build stats dataset** runs the old code and would undo the new stats, while a site, hub or homepage build already runs the new code — and the first stats build of any kind re-reads every video once (about 10–30 minutes on a large archive; it can be stopped and picks up where it stopped). Then build the index, the stats, the homepage, the hub, and the sites.
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -6350,6 +6350,57 @@ Line numbers are `plans/FACTS.md` lines at `e172749b`, before this record's in-p
read by nothing. So a "no-op" site save can move bytes; `jq -S` plus the `order` fills is the
expected diff, not a bug.
+## Release 17 — slice RL: a subtitle 429 is not a platform failure; the pace adapts (verified 2026-10-01)
+
+- **YouTube's 429s are its subtitle (timedtext) fetch, per video.** Measured 2026-10-01 on the
+ live corpus: all 78 HTTP 429 lines in 24 h of job logs were `Unable to download video subtitles
+ for 'en': HTTP Error 429`; none on the watch page, player API, m3u8 or a listing; the backoff
+ read `fails: 80`. The 2026-09-25 postmortem (`plans/youtube-lane-pacing.md`) found the same.
+- **`subs_rate_limit`** is a `DownloadFailureClass` (`lib/availability.ts`): `classifyDownloadFailure`
+ returns it when a subtitle-429 line is present and every `ERROR:` line is a subtitle line (no
+ soft block, no bot check) — `isSubtitleRateLimitOnly` / `hasSubtitleRateLimit`. It is set on a
+ **SUCCESS** record (`status: ok…`, `failureClass: "subs_rate_limit"`), the one class that is.
+- **yt-dlp's `--ignore-errors` turns a subtitle failure into a WARNING and carries on**
+ (`YoutubeDL._write_subtitles`: `ignoreerrors is True` → `report_warning`; `'only_download'`, i.e.
+ `--no-abort-on-error`, still raises). Without it, a `--load-info-json` run that fails re-extracts
+ from the URL once (`download_with_info_file` "The info failed to download … trying with URL") —
+ the "one internal re-extraction" the evidence saw. The youtube-handling primary
+ (`downloadOneManaged.ts` `youtubeHandlingArgs`) carries `--ignore-errors` and
+ `--sleep-subtitles max(5, pace)` after `-t sleep`; a subtitle 429 there exits 0, no transcript
+ lands, and attempt 3 (the no-subs media pass, `--no-write-subs --no-write-auto-subs` appended)
+ fetches the audio. Any other primary that died on its subtitles alone is re-run once as
+ `primary-without-subs` (`DownloadAttemptKind`).
+- **The pacing memory** is three maps beside `platformBackoff`/`videoDeferrals` on every lane of
+ `.auto-queue/state.json` (download only in practice; `{}` coerced for an older file):
+ `platformPace {sleepRequestsSeconds, baseSeconds, cleanUnits}` (absent = at the static value),
+ `platformHolds {since, probeAt}`, `subtitleDeferrals {count, lastAt, until, channelSlug}`. The
+ pure seam is `jobs/platformBackoff.ts` (`escalatePlatform`, `settlePlatformClean`,
+ `prunePlatformPacing`, `deferSubtitles`, `downloadGapMs`, `heldPlatformSentence`) and
+ `jobs/unitOutcome.ts`; every writer outside the runner goes through `jobs/downloadBackoff.ts`'s
+ one write-through (`recordDownloadBackoff(pf, paths, failureClass)`, `recordSubtitleDeferral`,
+ `clearSubtitleDeferral`, `heldPlatformRefusal`).
+- **The pace reaches every spawn synchronously**: `channelExtraArgs` reads
+ `livePlatformPaceSeconds(pf)` (`jobs/autoQueueState.ts`) — the shared object when a runner holds
+ it, else the pace the last read/write of the file saw (`holder.lastPace`; the status poll reads
+ the file), else the static value. `withSleepRequests` only RAISES `--sleep-requests`; the
+ channel's own `ytdlpExtraArgs` still come last and win. The pace key is
+ `detectPlatform(url) ?? "unknown"`, the cooldown's key.
+- **The hold**: `FAILS_TO_REACH_CAP` = 6 (60 s doubling reaches the 30-min cap at the sixth);
+ `failsAtCap(fails) = max(0, fails − 5)`. At `pacing.holdAfterFailsAtCap` the backoff's `until`
+ becomes `now + holdProbeMinutes` (no jitter) and `platformHolds[pf].probeAt` mirrors it; a held
+ platform's backoff entry is never pruned. Only a `transcribed` unit with no subtitle 429 clears
+ it. A manual Sync / download / metadata scan / fetch-window on a held platform is refused with
+ `heldPlatformSentence` (`<pf> is held: … the next probe is in N min. <What> will run once a probe
+ comes back clean.`).
+- **The download lane now waits between units**: `platformNextStartAt` (in memory) =
+ settle + `downloadGapMs(settings.sleepBetweenDownloadsSeconds, pace, base)` — the setting plus
+ the pace ABOVE its base (the base is already paid inside each spawn). `runManagedDownloads`
+ sleeps the same sum. Idle reasons `held` and `paced` join `cooldown`/`deferred`.
+- **download-missing-subs** skips videos inside a subtitle deferral (logged in the prefilter line),
+ records a subtitle 429 and goes on to the next video whatever `abortOnError` says, and clears a
+ deferral when the subtitles come down. `runChildAndStream` returns its stderr tail (and attaches
+ it to the thrown error as `stderrTail`).
+
## Release 7 — slices Y, C, K (verified 2026-09-25, `main` @ `bb3dbb4c`)
- **A rate-limited video is deferred, per video, for 6 h.** `common/jobs/platformBackoff.ts`:
diff --git a/plans/release-17.md b/plans/release-17.md
@@ -263,6 +263,7 @@ one short Transcribe (the hook on a relocated channel), `/storage`, `df`; then n
| **T3** migration + records | `r17/media-tier-migrate` | `common/bin/migrate-media-tier.ts`, `archilyzer.ts` wiring, fixture tests (tmp "platter"), FACTS "A channel's media is tiered", AGENTS.md's six things → seven, SETTINGS.md/CHANNEL.md regen, the release record, changelog | T1, T2 | dry run; resume from each phase; idempotent rerun; `--reclaim`; refusal on a marker; the free-space stop |
| **U1** umtool roots + `out/` | `r17/umtool-media-root` | `paths.mjs` (`MEDIA_ROOT`, `CACHE_DIR`), `lib/report/storage.mjs`, `driver.mjs`, `build-video.mjs:2615`, `export.mjs`, `kinds.mjs`, `umtool doctor`, `umtool storage move-out`, the e2e env | — (∥ T1) | `test:scripts` (+ mover tests), `next-build-trace.test.mjs`, the capped umtool build with the corpus linked, umtool e2e |
| **U2** deliverables switch | `r17/umtool-deliverables` | manifest `storage` field, `deliverableDir`, `cut.mjs:96`, `deliver.mjs:362`, `umtool storage deliverables`, bench "Move deliverables", `umtool check` | U1 | umtool unit + e2e: cut and share through a linked `clips/` |
+| **RL** a subtitle 429 does not fail a download; the pace adapts (operator-requested, beside the media tier) | `r17/rate-limit-adapts` | `common/jobs/{platformBackoff,downloadBackoff,unitOutcome,autoQueueState}.ts`, `lib/availability.ts`, `ytdlp/{platformArgs.mjs,channelArgs.ts}`, `downloadOneManaged.ts` (subtitle handling + pace only), `autoRunner.ts` (the download lane's gap, hold, probe), `runYtdlp.ts` (pace into `runManagedDownloads`, download-missing-subs), `checkAvailability.ts` (pace only), `settingsSchema.ts` `pacing` + SETTINGS.md, `RunnerOperationView.tsx` + `dispatch.ts`, `views/activeJobs.ts`, `pipelineActions.ts` + `videoActions.ts` (refusals, the subtitle line), `bin/doctor.ts`, `editor/e2e` | — (∥ D0, T1, U1) | unit: classifier, `applyUnitOutcome`, `channelExtraArgs` pace, the lane gap; e2e `rate-limit.spec` + `auto-queue`, `lane-runner`, `channel-priority`, `video-page` |
Order: 0a → D0 ∥ T1 ∥ U1 → T2 ∥ U2 → T3 → parent: records, ONE editor rebuild + restart, umtool rebuild +
restart (the restart is the operator's: the permission layer refuses the `0.0.0.0` bind) → the migration
@@ -321,6 +322,53 @@ hand; a dirent `isFile()` filter over a video dir hides it."** The `.relocating.
- The `en` track → 0 cues bug (index prefers `en` over `en-orig`; some `en` VTTs parse to 0 cues).
- A channel export/import **bundle** built on `mediaTier.ts`'s classifier — the slice after this release.
+## Slice RL — the ruling (2026-10-01)
+
+Operator-requested, beside the media tier. **Measured 2026-10-01 (read-only, live corpus):** every
+one of the 78 HTTP 429 lines in 24 h of job logs is `Unable to download video subtitles for 'en':
+HTTP Error 429` on YouTube; none on the watch page, the player API, the m3u8 or a listing; no
+`sync` or `import-one` log carries a real 429. The YouTube backoff reads `fails: 80`, consecutive
+since the evening of 2026-09-30; the lane retries one unit every ~30 min and each burns another
+subtitle 429. Twelve videos of one channel rotate through the 6 h deferral and fail again each
+time; `redownload-archive` jobs on other videos of the same channel succeeded in between. Each
+failing unit had already picked its formats and died on the subtitle file. The 2026-09-25
+postmortem (`plans/youtube-lane-pacing.md`) found the same: the timedtext endpoint 429s per video,
+not IP-wide. Today's flags: `--sleep-requests 1` for YouTube, `-t sleep` on the primary spawn, no
+`--sleep-subtitles` override; yt-dlp re-extracts once after a subtitle 429 then fails; the app
+never retries a `rate_limit`.
+
+1. **A subtitle 429 never fails a download.** When yt-dlp fails only on the subtitle file, the
+ unit downloads the media anyway and succeeds — one spawn where yt-dlp's own `--ignore-errors`
+ allows it, else a second spawn without subtitles. `classifyDownloadFailure` gains
+ `subs_rate_limit` for "the subtitle fetch is the only failure". It is recorded per video as a
+ subtitle deferral (beside `videoDeferrals`, through `downloadBackoff.ts`'s write-through), does
+ NOT touch the platform backoff and does NOT defer the video's media. The deferred subtitles are
+ picked up by `download-missing-subs`; the transcription lane sees "downloaded, no transcript".
+ After 3 subtitle deferrals the video's subtitles are left alone for 7 days (shown on the video
+ page with the count and the date; a manual fetch still works). `plans/youtube-lane-pacing.md`
+ gets a dated note pointing here.
+2. **The pace adapts per platform.** A persisted per-platform `platformPace`: `--sleep-requests`
+ starts at the platform's static value, doubles on every platform-level `rate_limit` (never on
+ `subs_rate_limit`) up to a cap, and decays one step toward the base after every N clean units.
+ It feeds `channelExtraArgs` (every yt-dlp call for the platform) and the lane's gap: the
+ download lane honours `sleepBetweenDownloadsSeconds` AND adds the adaptive pace. Settings in a
+ new `pacing` block (cap 16 s, decay 5 units, hold threshold, probe interval). YouTube's primary
+ spawn gets `--sleep-subtitles <pace>`.
+3. **A block is not a burst.** A platform whose backoff has failed at the cap
+ `holdAfterFailsAtCap` times in a row (3) is HELD: one probe unit per `holdProbeMinutes` (60)
+ instead of one every 30 min; a clean probe clears the hold and the backoff. Persisted beside the
+ backoff. A manual Sync/download on a held platform is refused with a sentence naming the hold
+ and the next probe.
+4. **Visible.** The download lane page's "Rate-limit cooldown" region shows per platform the
+ cooldown or hold (since, fails, next try/probe), the current pace, and the videos whose
+ subtitles are deferred (with counts); a rack chip for a held platform only if it is a few lines;
+ `archilyzer doctor` warns on a platform in cooldown/hold and on the pace; the idle reasons gain
+ `held` and `subs-deferred` where the dispatch text names them.
+
+Not in scope: yt-dlp's player client or cookies for subtitles (an open question for the operator:
+whether the timedtext 429 for these twelve videos is a PO-token/client matter — not probed by
+hand); the sync scheduler's per-channel backoff; Rumble/Odysee specifics beyond the shared code.
+
## Record
## Rollout
diff --git a/plans/youtube-lane-pacing.md b/plans/youtube-lane-pacing.md
@@ -1,3 +1,12 @@
+> **Corrected again 2026-10-01 by `plans/release-17.md` "## Slice RL — the ruling" (shipped as
+> "### Slice RL, as shipped" there).** A subtitle 429 — every YouTube 429 this file measured, and
+> all 78 of 2026-10-01's — no longer backs the platform off or defers the video: it is its own
+> class, `subs_rate_limit`, the download goes on to the media, and only the video's subtitles are
+> deferred. The "next lever" named below, honouring `sleepBetweenDownloadsSeconds` in the lane, is
+> built (plus an adaptive per-platform `--sleep-requests`), and a rate limit that outlasts the
+> cooldown cap now holds the platform to one probe at a time. The finding "pacing between videos
+> could not have prevented" a timedtext 429 stands.
+
# Plan: one YouTube video must not keep the whole platform in a 429 cooldown
> **Corrected by `plans/release-7.md` "## Slice Y" and shipped as "### Slice Y, as shipped"
