commit 6bd4e5bd804199cfe223f16b96dacc2adde1581e
parent eecdeb502e207c011f4942f1f9dcdea487a46f65
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Fri, 2 Oct 2026 00:47:16 -0400
plans: slice RL, as shipped — the record, the gates, the deviations, the open question
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
| M | plans/release-17.md | | | 165 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
1 file changed, 165 insertions(+), 0 deletions(-)
diff --git a/plans/release-17.md b/plans/release-17.md
@@ -1479,4 +1479,169 @@ the full suite was clean at load 10 — this branch does not touch `scripts/`);
92 s; **e2e (the 8 specs) 78 passed, 5 failed, 8.2 min** — the same five `channel-storage.spec.ts` cases
(:80, :250, :391, :484, :1091), the old mover's layout reading `legacy`, nothing else red.
+### Slice RL, as shipped — a subtitle 429 does not fail a download, and the pace adapts (2026-10-01)
+
+Branch `r17/rate-limit-adapts` off `main` `90bd8384`, worktree `~/Projects/r11-runner-lows` (editor 4801,
+test 4811, export 4810 — `pnpm wt list`'s block #18), one Opus implementer. Scratch files `RL-*` in the
+job's `tmp`. `main` moved under the slice: it was merged at `1d5c33bf` (U1, XP), at `a395aaa1` (D0, T1) before the
+final gates, and at `0a62bf74` (U2) after them.
+The ruling is above ("Slice RL — the ruling").
+
+**What yt-dlp does, verified offline.** yt-dlp 2026.08.19 (the editable install the editor runs) was run
+against a localhost HTTP server whose subtitle URL answers 429 — no YouTube request, per the operator's
+rule. `--ignore-errors` (`ignoreerrors: True`; `--no-abort-on-error` is `'only_download'` and still raises)
+reports `WARNING: Unable to download video subtitles for 'en': HTTP Error 429: Too Many Requests`, exits
+0, still fires `--print after_video:…`, and — when the run is not `--skip-download` — goes on to download
+the media. Without it, a `--load-info-json` run that hits the subtitle error prints `The info failed to
+download: … trying with URL …` and re-extracts from the URL: the "one internal re-extraction" the
+evidence saw. So the ONE-spawn shape is yt-dlp's own, and the log keeps the line for the classifier.
+
+**What it does.**
+- **A subtitle 429 is not a failure of the download.** `classifyDownloadFailure` returns the new
+ `subs_rate_limit` when a subtitle-429 line is present and every `ERROR:` line is a subtitle line (no
+ soft block, no bot check). The youtube-handling primary (subtitles only, `--skip-download`) now carries
+ `--ignore-errors` and `--sleep-subtitles max(5, pace)` after `-t sleep`: a subtitle 429 exits 0 with
+ no transcript, and attempt 3 — the no-subs media pass, `--no-write-subs --no-write-auto-subs` appended
+ after the channel's own args — downloads the audio, exactly as for a video with no captions (three
+ spawns, as before for such a video). The record is a SUCCESS carrying `failureClass:
+ "subs_rate_limit"` (the one class set on a success). Any other primary that died on its subtitles
+ alone (a channel whose own args ask for subtitles) is run once more as `primary-without-subs`
+ (a new `DownloadAttemptKind`).
+- **Only the subtitles are deferred.** `applyUnitOutcome` (lane) and `runManagedDownloads` (batches,
+ through `recordSubtitleDeferral`) record `subtitleDeferrals[id] = {count, lastAt, until,
+ channelSlug}`: 6 h after a strike, 7 days from the third. The platform backoff, hold and pace are
+ not touched, the video is retired like any success, and it is not added to `videoDeferrals`.
+ **Download missing subs** skips a video inside its window (named in the prefilter line with its
+ count and date), records a subtitle 429 and goes on to the next video whatever `abortOnError` says,
+ and clears the deferral when the subtitles come down (`runChildAndStream` now returns its stderr
+ tail and attaches it to the thrown error). A download from the video page never reads it.
+- **The pace adapts.** `platformPace[pf] = {sleepRequestsSeconds, baseSeconds, cleanUnits}`: absent at
+ the static value (`PLATFORM_ARGS`' `--sleep-requests`, 1 s for YouTube and Rumble, 0 for a platform
+ with none); every platform-level `rate_limit` doubles it (from 0 to 1) up to
+ `pacing.sleepRequestsCapSeconds` (16); every `pacing.decayAfterCleanUnits` (5) clean units halve it,
+ and an entry back at its base is deleted. A `network` failure and a `subs_rate_limit` never move it.
+ `channelExtraArgs` reads it synchronously through `livePlatformPaceSeconds` — the shared state when a
+ runner holds it, else the pace the last read or write of the file saw, else the static value — so
+ every spawn against the platform (listing, prefetch, primary, availability, metadata scan, clip, the
+ new-channel probe through `pacedPlatformArgs`) uses it; `withSleepRequests` only raises, and the
+ channel's own `ytdlpExtraArgs` still win. A manual 429 (`recordDownloadBackoff(pf, paths,
+ failureClass)`) escalates the same way.
+- **The lane waits between units.** `platformNextStartAt` in the runner = settle +
+ `downloadGapMs(sleepBetweenDownloadsSeconds, pace, base)`: the setting the lane used to ignore,
+ plus the pace above its base. `runManagedDownloads` sleeps the same sum.
+- **A block is not a burst.** `FAILS_TO_REACH_CAP` = 6; a backoff that has failed at the cap
+ `pacing.holdAfterFailsAtCap` (3) times in a row — `fails` 8 — holds the platform:
+ `platformHolds[pf] = {since, probeAt}`, the backoff's `until` = now + `pacing.holdProbeMinutes`
+ (60, no jitter), so the lane's existing cooldown gate lets one probe unit through per interval. A
+ failed probe re-arms it; only a clean unit (`transcribed`, no subtitle 429) clears the hold and the
+ backoff together. A held platform's backoff entry is never pruned. A manual Sync, any download mode
+ but Store playlist, a metadata scan, and the video page's fetch-window / full-source fetch are
+ refused with `heldPlatformSentence`: `youtube is held: its rate limit outlasted the cooldown cap (8
+ failures in a row, held since 14:02 UTC). Auto-download probes it once every 60 min — the next probe
+ is in 42 min. Sync will run once a probe comes back clean.` (scheduled syncs go through the same
+ action and are refused alike).
+- **Visible.** The download lane's `role="region"` "Rate-limit cooldown" gains `<ul aria-label="Platforms
+ held">` (since, failures, next probe, pace), `"Request pace"` (seconds between requests and the base)
+ and `"Deferred subtitles"` (link, count, time left; "left alone" from the third); "Platforms in
+ cooldown" now lists only unheld platforms and shows the pace too; `formatCooldown` gains a days arm.
+ The view (`autoQueueStatus.ts`) carries `cooldowns[].hold`, `pace[]` and `subtitleDeferred[]`. Idle
+ reasons gain `held` ("every pending platform is held after repeated rate limits — one probe at a
+ time") and `paced` ("every pending platform is pausing between downloads"), in both exhaustive
+ switches. The video page draws one `role="note"` `aria-label="subtitle deferral"` line for a video
+ with a deferral, read from a new `GET /api/channels/<slug>/videos/<id>/subtitle-deferral`. `archilyzer doctor` has a **download pacing** section: a cooldown, a hold or a
+ raised pace warns (never fails), deferred subtitles are a note.
+- **Settings:** a `pacing` block (`sleepRequestsCapSeconds` 16 [1–120], `decayAfterCleanUnits` 5
+ [1–1000], `holdAfterFailsAtCap` 3 [1–100], `holdProbeMinutes` 60 [1–1440]); SETTINGS.md and
+ settings.json.example regenerated; `sleepBetweenDownloadsSeconds`' description says the lane honours
+ it now.
+
+| sha | what |
+|---|---|
+| `93462d22` | common: `subs_rate_limit` (`availability.ts`), the youtube primary's `--ignore-errors` + `--sleep-subtitles`, attempt 3 on a subtitle 429, `primary-without-subs`, the record's class; `platformBackoff.ts` pace/hold/subtitle-deferral seams + coercion; `autoQueueState.ts` three maps + `livePlatformPaceSeconds`; `unitOutcome.ts`; `downloadBackoff.ts` one write-through + `recordSubtitleDeferral`/`clearSubtitleDeferral`/`heldPlatformRefusal`; `platformArgs.mjs`/`channelArgs.ts` the pace; `autoRunner.ts` gap, hold idle, `held`/`paced`; `runYtdlp.ts` batch gap + deferral, download-missing-subs; `pacing` settings + SETTINGS.md; unit tests |
+| `8a40c91e` | editor + doctor: the region's three lists, the hold refusals (`pipelineActions.ts`, `videoActions.ts`), the video page's line, `autoQueueStatus` fields, `doctor` section, `checkAvailability`'s paced probe, a manual Sync's backoff passes its class |
+| `49b3d087` | e2e: `rate-limit.spec.ts` (3 tests); the fake's `dl429` obeys `--ignore-errors`, new `wp429` (watch-page 429 at the prefetch); `pacing.spec` T2 moves to `wp429`; `rumble-sweep.spec`'s paged walk paces at 2 s; the video-page line moved out of `cards/` |
+| `2ede037c` | plans: the ruling, FACTS, the dated note on `youtube-lane-pacing.md`, two `[Unreleased]` bullets |
+| `8c76ec09` | merge `main` `1d5c33bf` (U1, XP): both changelog sides, both rulings and slice rows kept |
+| `a8e9c241` | editor: the video page's line reads `GET /api/channels/<slug>/videos/<id>/subtitle-deferral` instead of a mount-time server action (Next runs a page's server actions one at a time, so it sat in front of the first click — e2e run 2) |
+| `5e967e18` | merge `main` `a395aaa1` (D0, T1): both changelog sides kept; no code conflict (T1's tier hooks and lane holds sit clear of this slice's hunks) |
+| `5725e53a` | merge `main` `0a62bf74` (U2): clean; U2 touches umtool, the changelog and the plans only |
+| *(this commit)* | this record |
+
+**Gates** (worktree root). tsc (`pnpm -r --no-bail --workspace-concurrency=1 exec tsc --noEmit`) clean
+before every commit (a stale `editor/.next/dev` and `export/.next/dev` from the worktree's earlier use
+were removed first). Before the merge of `main`: common **2513/2513**, editor unit **109/109**,
+test:scripts **368 pass + 2 skip**, capped editor build ok (78 s). At the first merged tip `8c76ec09`: common
+**2530/2530** (29 of them new: `unitOutcome` +6, `platformBackoff` +5, `channelArgs` +4,
+`downloadBackoff` +4, `availability` +3, `subtitleRateLimit` +2 (new), `managedDownloadsSleep` +2,
+`autoQueueState` +1, `autoQueueStatus` +1, `doctor` +1), editor unit **109/109**, test:scripts
+**392 pass + 2 skip**, capped editor build ok (compiled in 24.0 s, 73 s). `settings example --check`
+clean. At the final tip `5e967e18` (after D0 and T1): tsc clean; common **2630 tests, 2602 pass, 0
+fail, 28 skipped** (the 28 are `main`'s — T1's `relocateChannelMedia` cases skipped until T2, "release 17
+T2 rebases the mover on media/"); editor unit **109/109**; test:scripts **394 pass + 2 skip**; capped
+editor build ok (compiled in 23.3 s, 56 s; the new route listed). EDITOR e2e, `$T/RL-specs.txt` = `rate-limit.spec.ts pacing.spec.ts auto-queue.spec.ts
+lane-runner.spec.ts channel-priority.spec.ts video-page.spec.ts rumble-sweep.spec.ts
+no-subs-fallback.spec.ts queues.spec.ts`:
+- run 1 (`RL-e2e1.log`, `49b3d087`): **74 passed, 1 failed, 13 min** (4 in the queue). The three
+ `rate-limit.spec` tests, `pacing.spec` (T2 1.0 min) and `rumble-sweep.spec` passed. The failure was
+ `auto-queue.spec:623` "Drain completes when an auto-transcribe unit is parked behind a busy worker"
+ — the 30 s test budget ran out while polling `/api/auto-queue/status` (a transcription-lane test;
+ nothing in this slice runs on that lane).
+- run 2 (`RL-e2e2.log`, `8c76ec09`): **74 passed, 1 failed, 26 min** (≈13 in the queue). The Drain
+ test passed; `video-page.spec:280` "Mark untranscribable…" timed out: its click's server action
+ queued behind the deferral line's mount-time action. Fixed in `a8e9c241`.
+- run 3 (`RL-e2e3.log`, `a8e9c241`, `rate-limit.spec.ts video-page.spec.ts`): **23 passed, 0 failed,
+ 9 min** (≈8 in the queue).
+- run 4 (`RL-e2e4.log`, `5e967e18`, the full list): **71 passed, 4 failed, 8 min** — the four
+ video-page text-preview tests (`:325`, `:367`, `:384`, `:472`), each a 5 s / 30 s timeout on a
+ preview fetch, while a common test run of this implementer's was loading the machine beside it.
+- run 5 (`RL-e2e5.log`, `5e967e18`, `video-page.spec.ts` alone, nothing beside it): **20 passed,
+ 0 failed, 1 min**.
+- run 6 (`RL-e2e6.log`, `5e967e18`, the full list, nothing beside it): **75 passed, 0 failed, 5 min**
+ (no queue wait).
+
+After the U2 merge (`5725e53a`, umtool-only code): tsc clean; test:scripts **461 pass + 2 skip** —
+in two of four runs one or two `scripts/queue-lock.test.mjs` cases ("serves waiters in arrival order
+(FIFO)", "prints a banner naming the holder while waiting") failed while other slices' e2e runs held
+the machine-wide lock; a run with the lock free passes. The editor build, common, editor unit and e2e
+were not rerun: U2 changed no `common/` or `editor/` code.
+
+**Numbers: none.** `.auto-queue/state.json` is outside both numbers tools. **`platformPace: {}`,
+`platformHolds: {}` and `subtitleDeferrals: {}` appear on all four lanes of `state.json` at the first
+persist after the rollout boot**, and `pacing` appears in `settings.json` at the next save; an older
+build drops the three keys on its next write, so a rollback is safe. Privacy gate: `git diff main
+--name-only | xargs grep -lc "$(whoami)\|$(hostname)"` prints `plans/FACTS.md` only, for three lines
+`main` already carries (3477, 5033, 5430 at `0a62bf74`); the slice's added lines carry none.
+
+**Deviations, one sentence each.**
+- `subs-deferred` is not an idle reason: a subtitle deferral never idles the lane (the media is done),
+ so it is the region's "Deferred subtitles" list instead; `paced` was added for the new gap, which
+ would otherwise read as "capped".
+- The gap adds the pace ABOVE its base, not the whole pace: the base is already paid between the
+ requests of every spawn, and adding it again would slow every lane unit and batch by a second with
+ nothing rate-limited.
+- The lane's gap reads the global `sleepBetweenDownloadsSeconds`; a channel's own override still
+ applies to its batch runs only.
+- No rack chip: the rack has no per-lane or per-platform chip to say "held: rate-limited" on, and
+ threading the platform state into every row is more than a few lines in `channelRow.ts`, which is
+ T2's.
+- Files touched beyond the list, each by a small hunk: `lib/downloadOutcome.ts` (the attempt kind,
+ the class's doc), `lib/settingsDocs.ts` (the `pacing` table), `views/autoQueueStatus.ts` (three
+ fields in `buildKind`, clear of D0's memo), `VideoPanel.tsx` + a new `SubtitleDeferralLine.tsx`
+ beside it and its new GET route (the video page's line; `videoActions.ts` keeps only the refusals), the fake yt-dlp, `pacing.spec.ts` and `rumble-sweep.spec.ts`
+ (their expectations follow the ruling).
+
+**Found and left.**
+- Subtitle deferrals are written by the lane, batch downloads and download-missing-subs; the video
+ page's own download, the re-acquire backfill and import neither record nor clear one (a manual
+ fetch never reads it either).
+- The `pacing` keys are in `settings.json` only, not on the `/settings` form.
+- The runner merges only `platformBackoff` from disk mid-run, as before; pace and holds written from
+ outside go through the live object (`mutateDownloadState`), and a disk-only write while no runner
+ lives is read at the runner's start.
+- At the rollout the live backoff reads `fails: 80`: unless the boot finds it lapsed for over 30 min
+ (then it is pruned, as before), the first platform-level failure after the boot holds YouTube at
+ once; a subtitle 429 no longer counts, and the first clean unit clears the backoff.
+- **Open question for the operator** (not probed): whether YouTube's timedtext 429 for these twelve
+ videos is a PO-token / player-client matter (`--extractor-args youtube:player_client=…`).
+
## Rollout
