commit dcb8d285c45939b1615ab8b2480d9cfd67e369a3
parent 11e6bb3368f51544f341f5250df2f9dfa536b51d
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Fri, 25 Sep 2026 11:25:48 -0400
plans: release 7 slice Y as shipped — YouTube lane pacing record + changelog
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
3 files changed, 81 insertions(+), 0 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,6 +1,9 @@
# Changelog
## [Unreleased]
+- **Auto-download no longer retries the same rate-limited video over and over; it moves on to the next one.** When a download answered HTTP 429, the runner paused the whole platform for a while and then picked the same video again, because it was still first in the queue. Each retry doubled the pause, up to 30 minutes. On 2026-09-24 one YouTube Short was retried 12 times this way and kept YouTube paused all evening. A YouTube 429 comes from the subtitle fetch for one video, not from the whole site. Now a rate-limited video is also **deferred for 6 hours**: auto-download skips it, so when the pause ends the runner takes the next video. The pause still grows only when *different* videos keep hitting the limit. Deferrals are kept in `.auto-queue/state.json` beside the platform cooldowns, so a restart does not retry the video early. The log line reads `… (attempt 1). <id> deferred 6h; next video after cooldown.` A manual Sync or *download missing* ignores deferrals and still fetches the video. When every video left is deferred, the runner reports that it is idle for that reason: "every pending video was rate-limited recently and is deferred".
+- **The cooldown strip on `/operations/download` also lists deferred videos.** It is now a region named *Rate-limit cooldown*, with a *Platforms in cooldown* list (unchanged) and a *Deferred videos* list. Each deferred video links to its page and shows how long it has left (`alpha/a1 — 5h 59m left`). The strip appears when either list has something in it. Times over an hour now read `5h 59m` instead of `359m 58s`.
+- **A video that keeps failing its metadata scan is no longer rescanned at every runner start.** When a scan hit the same error again (members-only, for example), the error's timestamp was not updated. The one-day rest that timestamp controls therefore ran out once and never started again, and each runner start rescanned all of them: 142 members-only videos on one channel, with cookies. The same error seen again after a day now updates the timestamp, so the video waits another day.
- **A video the server answers with HTTP 410 Gone is recorded as removed, not as an error.** Rumble answers a taken-down video with `HTTP Error 410: Gone`; the availability check read that as a generic error (one Rekieta Law Rumble video has said "error" since 2026-08-21), and a download that hit it could stop the batch. It now reads as removed, like "Video unavailable" does, so the check says so and a download skips that one video and carries on. Existing records change the next time the video is checked.
- **umtool's report videos can fetch Rumble clips again.** The clip fetch and the source availability check in `umtool/report-to-video` ran yt-dlp without the browser fingerprint Rumble now requires, so every Rumble clip failed with 403 and every Rumble source looked missing. They now pass the same Rumble arguments as the editor, from the same single table.
- **A transcript pulled back from a remote worker is written safely.** It used to be written straight onto `transcript.json`, so a crash part-way through left a truncated transcript; it now goes through the editor's one atomic write (temp file, then rename), like every other file the editor writes.
diff --git a/plans/release-7.md b/plans/release-7.md
@@ -314,4 +314,73 @@ incomplete; the LM chat-only tier (operator config).
## Record
+### Slice Y, as shipped — YouTube lane pacing (2026-09-25)
+
+Branch `one-core/r7-pacing` off `main` `6ee1d336`. `main` did not move during the slice. Every
+spec anchor was checked on `6ee1d336` before its edit, and all of them held at `0032ed8a` line
+numbers. Items 1 to 8 were built as written. Item 9 (`sleepBetweenDownloadsSeconds`) stays out.
+Item 10: no numbers were taken. The slice fixes two things. First, a rate-limited video is
+**deferred for 6 h** as well as backing its platform off, so when the cooldown lapses the runner
+picks the next video instead of re-picking the same one. The deferral is persisted beside
+`platformBackoff` so a restart honours it. Second, an identical metadata-scan error now refreshes
+its `at` once it is a cooldown old, so members-only ids stop being re-scanned at every runner start.
+
+| sha | what |
+|---|---|
+| `65dca073` | items 1 + 2. `platformBackoff.ts` appends `VideoDeferral {until, channelSlug}`, `VideoDeferralState`, `VIDEO_RATE_LIMIT_DEFER_MS` (6 h), `deferVideo`, `isVideoDeferred`, `pruneDeferred` (drops `until <= now`; there is no retention, since a deferral has no escalation memory) and `coerceVideoDeferrals` (mirrors `coercePlatformBackoff`). `nextBackoff` is untouched. `AutoQueueKindState.videoDeferrals` is added to the empty state, to `coerceKindState` (a missing or corrupt key becomes `{}`) and to the write trim. `platformBackoff.test` +5, `autoQueueState.test` +2; the existing "lane coerces to empty" deepEqual gained the key |
+| `71c755c8` | item 3. New `common/jobs/unitOutcome.ts` `applyUnitOutcome(state, unit, now, rand?) → {markCompleted, line}`. `rate_limit` runs `nextBackoff` + `deferVideo` and logs `Auto-download: <pf> rate_limit — backing off <s>s (attempt <n>). <id> deferred 6h; next video after cooldown.` `network` backs off with today's line and no deferral. `transcribed` calls `clearBackoff`. A unit with no platform (a non-download lane) touches neither map, as before. Every branch prunes. The block at `autoRunner.ts:1941-1963` is now the call + `onLog(line)` + `markCompleted` (still skipped on an operation lane), and `persist()` is unchanged. `unitOutcome.test.ts` has 7 cases, including "`fails` climbs only across distinct ids" |
+| `a9af6a0a` | item 4. The boot prune adds `pruneDeferred`. After the platform gate, the download branch drops pending ids with a live deferral and sets `anyDeferred` only when it actually dropped one. The idle reason is `anyCooling ? "cooldown" : anyDeferred ? "deferred" : "capped"`. `AutoRunnerIdleReason` gains `"deferred"`. `dispatch.ts` `idleReasonText` gains "every pending video was rate-limited recently and is deferred" |
+| `cb041841` | item 5. `metadataScanStore.ts` `upsertMetadataScan` also rewrites an identical error when `Date.parse(now) - Date.parse(prev.at) >= METADATA_SCAN_ERROR_COOLDOWN_MS`. The flush passes `new Date().toISOString()` as `now` (`ytdlp/metadataScan.ts:374-379`). `metadataScanStore.test` +2: 25 h later rewrites `at` and `updatedAt`; 1 h later leaves the file's mtime alone |
+| `04f5c9be` | items 6 + 7. `autoQueueStatus.ts` adds `VideoDeferralView {videoId, channelSlug, untilMs}` and `AutoQueueKindStatus.deferred` (live deferrals only, soonest first). `RunnerOperationView.tsx` draws the strip when there are cooldowns OR deferrals. The strip is `role="region"` `aria-label="Rate-limit cooldown"`. Under the existing heading is `<ul aria-label="Platforms in cooldown">` (items unchanged, drawn only when a platform cools). Then, when there are deferrals, the heading "Deferred videos — skipped by auto-download until:" and `<ul aria-label="Deferred videos">`, whose items read `alpha/a1 — 5h 59m left` with the id a `Link` to the video page, as `NextUp` does. **The region is a `<div role="region">`, not the `<section>` the spec wrote**: the strip sits inside the lane's own `<section>`, and the file's structural contract (`:30-32`) forbids a nested one because every `locator("section", {has})` in the suite would match two ancestors. The accessible role and name are the same. `formatCooldown` gains an hours arm (`5h 59m`, or `6h` when the minutes are zero). Its only caller is this strip, and platform cooldowns cap at ~33 min, so their text does not change |
+| `17dc70c0` | item 8. The fake's `dl429` sentinel is in `modeYoutubeSingleUrlManaged` only: stderr `ERROR: [youtube] <url>: Unable to download video subtitles for 'en': HTTP Error 429: Too Many Requests`, exit 1. The prefetch branch still succeeds, which is the real two-spawn shape. New `editor/e2e/pacing.spec.ts`. T1 seeds `download.videoDeferrals.a1` and checks: picks `["a2"]`, idle `deferred`, `deferred == [a1]`, and the region + `Deferred videos` list containing `alpha/a1` with no `Platforms in cooldown` list. T2 checks: `state.json` shows `videoDeferrals.dl429vid1` (slug `alpha`, > 5 h left) and `platformBackoff.youtube.fails === 1`; then, after the real ~60 s cooldown, picks become `["dl429vid1","a2"]` (`setTimeout(150_000)`) |
+| `ffe71f42` | e2e fix. T1's idle-sentence assertion hit two elements (the rail and the lane both draw the sentence, a strict-mode violation), so it now asserts `.first()` |
+| *(this commit)* | this record, `[Unreleased]` bullets, and the correction note at the top of `plans/youtube-lane-pacing.md` |
+
+**Gates** (worktree root, on `ffe71f42`). tsc (`pnpm -r --no-bail --workspace-concurrency=1 exec
+tsc --noEmit`) was clean before every commit. common **1770/1770** = 1754 + 5 (`platformBackoff`)
++ 2 (`autoQueueState`) + 7 (`unitOutcome`) + 2 (`metadataScanStore`). Editor unit **72/72**.
+test:scripts **159 pass + 1 skip**. mcp **219/219**. `pnpm --filter editor exec next build` ok
+(compiled in 27.6 s). `pnpm --filter export exec next build` ok (10.5 s; no dangling links under
+`export/public`). EDITOR e2e, `$T/y-specs.txt` =
+`pacing.spec fetch-window.spec queues.spec rumble-sweep.spec auto-queue.spec lane-runner.spec`
+(all six exist):
+- run 0 (`y-e2e0-misscoped.log`): **aborted at test 6 of 642**. The brief's bare names
+ (`pacing …`) are Playwright path regexes, and `pacing` matches the worktree path
+ `one-core-r7-pacing/`, so every spec was selected. Killed along with its orphan servers on
+ :3711/:3710 and its ollama stub; the list was rewritten with `.spec` suffixes. **A worktree whose
+ path contains a spec's name needs the suffixed form.**
+- run 1 (`y-e2e1.log`): **44 passed, 1 failed, 3.9 min**. T1 failed on its last assertion, the
+ strict-mode double match fixed in `ffe71f42`. T2 passed (1.1 min).
+- run 2 (`y-e2e2.log`, `pacing.spec` alone): **2 passed, 0 failed, 1.4 min**.
+- run 3 (`y-e2e3.log`, the full list on `ffe71f42`): **45 passed, 0 failed, 4.3 min**. No run
+ waited in the queue.
+
+**Numbers: none.** `.auto-queue/state.json` is outside both numbers tools (settings: 1,353 paths;
+files: config/site/sidecars), and no `settings.json`, `site.json` or `config.json` key changed.
+**`videoDeferrals: {}` will appear on all four lanes of `state.json` at the first persist after
+the rollout boot.** That is the expected change outside the md5 baseline (rollout step 4 records
+`.download|keys` before). An older build drops the key on its next write, so rollback is safe.
+
+**Found and left.**
+- **Manual Sync and *download missing* do not consult deferrals**, by design. A manual retry of a
+ deferred video still runs, and if it succeeds it clears the platform cooldown. It does not
+ clear the video's deferral: the runner retires a fetched id anyway, because it leaves
+ `undownloadedIds` at the next snapshot. A manual 429 goes through `recordDownloadBackoff` (a
+ platform cooldown only, no deferral). That read-modify-write now carries `videoDeferrals`
+ through, since `readAutoQueueState` coerces it.
+- **The runner does not merge deferrals from disk mid-run** the way it merges `platformBackoff`.
+ Only the runner writes deferrals, so there is nothing outside it to merge. A deferral
+ hand-seeded into `state.json` takes effect at the next runner start, which is what T1 does.
+- **`common/views/activeJobs.ts` `autoIdleNote` has no `"deferred"` case** (it is not
+ exhaustive: `default: null`). A runner idling `deferred` therefore shows no note on the
+ `/jobs` row. The runner page and the rail do say it. The file is not Y's; the one-line fix is
+ `case "deferred": return "every pending video deferred after a rate limit";`.
+- `common/views/autoQueueStatus.test.ts` has no case for `deferred` (the file is not on Y's list).
+ The filter and sort are covered by T1 through the real route.
+- The plan file's step 3 (a `plans/FACTS.md` entry: the lane defers a rate-limited video; the
+ cooldown escalates across distinct videos; YouTube 429s are per-video timedtext) is the
+ parent's to write.
+- **Commit trailers** name `Claude Opus 5.5 (1M context)`, as the release-6 implementers did.
+ `implementer-rules.md` still names Fable 5.1.
+
## Rollout
diff --git a/plans/youtube-lane-pacing.md b/plans/youtube-lane-pacing.md
@@ -1,5 +1,14 @@
# Plan: one YouTube video must not keep the whole platform in a 429 cooldown
+> **Corrected by `plans/release-7.md` "## Slice Y" and shipped as "### Slice Y, as shipped"
+> there (2026-09-25).** The deferral is PERSISTED as `videoDeferrals` beside `platformBackoff`
+> in `.auto-queue/state.json`, not in memory (a rollout restart would otherwise re-hit the video
+> at `fails+1`). The outcome logic moved to a pure seam, `common/jobs/unitOutcome.ts`, because
+> `runLoop` is not exported. An all-deferred lane idles with its own reason, `deferred`, not
+> `cooldown`. The page lists deferred videos. The identical-scan-error `at` refresh is also in
+> the slice. `sleepBetweenDownloadsSeconds` stays out. The steps and tests below are the
+> original plan; read the release-7 record for what shipped.
+
**Found 2026-09-25**, from STATE.md: "YouTube held a 429 cooldown across three attempts (18:43,
19:13, 19:41)", and the owed md5-sweep sync was refused all three times. Verified read-only on
`main` `93dcb532`, the live `settings.json` and the retained `transcripts/.jobs/*` (meta
