commit 607987760fea9b28cf9e94737ff76b187e1bad2b
parent e99effb6bd1d8876325d484478afb94b1ae41480
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Thu, 24 Sep 2026 22:38:11 -0400
plans: release 5 slice R — review fixes in the record (commit table, gates, retry cadence, umtool follow-up)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
1 file changed, 27 insertions(+), 8 deletions(-)
diff --git a/plans/release-5.md b/plans/release-5.md
@@ -144,7 +144,7 @@ STATE. umtool: rebuild only if `git diff --stat <live>..<new> -- umtool` is non-
### Slice R, as shipped — Rumble: impersonation everywhere, paced sweeps, incomplete ≠ failed (2026-09-24)
-Branch `one-core/r5-rumble` off `main` `f4da04a9`, four commits, unmerged. Every Rumble request
+Branch `one-core/r5-rumble` off `main` `f4da04a9`, seven commits (four, then three after review), unmerged. Every Rumble request
403s at Cloudflare without a browser TLS fingerprint (yt-dlp #17496), and the
`the-quartering-rumble` full sweep 429'd at page 155 and failed the whole sync every time for 44
days. There is now one arg builder with a platform table, and a 429 part-way through a full
@@ -155,7 +155,10 @@ enumeration is *incomplete*: cooldown recorded, one log line, the paged walk run
| `bca929a1` | `common/ytdlp/channelArgs.ts`: `PLATFORM_ARGS` (`rumble: --impersonate chrome --sleep-requests 1`, comment names #17496 and the 2026-09-24 probe), `platformArgs(platform)`, `channelPlatform(config)` = `config.platform ?? detectPlatform(config.url)`; `channelExtraArgs` = cookies → platform args → `ytdlpExtraArgs` (override last, wins). `configArgs` (runYtdlp) delegates to it; the metadata scan's and `checkAvailability`'s inline copies call it (`checkAvailability` falls back to `cookieArgs` only when the channel has no config); `probeChannelMeta` gets `platformArgs(detectPlatform(url))` — `detectPlatform("https://rumble.com/c/…")` is `"rumble"` (host suffix). `channelArgs.test.ts` (5) |
| `0a4b9999` | `EnumerationIncompleteError {platform, pagesReached, count}` + `lastListingPage(stderr)` (last `Downloading page N` — the real wording is `[RumbleChannel] TheQuartering: Downloading page 155`, job `01M3AVGZC5EZ9GCD04R9QW6NWX`). `enumeratePlaylistUrls` classifies a non-0/101 exit's buffered stderr with `classifyDownloadFailure`; `rate_limit` → the typed error (message still starts `yt-dlp exited with code N`), anything else → the plain error as before. `syncFullSweep` catches it before `acceptEnumeration`, calls `opts.onPlatformBackoff?.("rate_limit")` (= `recordDownloadBackoff(platform, paths)`, wired by `pipelineActions.ts:215`, the same helper and `nextBackoff` schedule a download's 429 uses), logs `Full sweep incomplete: 429 at page N of the listing, M entries — not a listing; next syncs are paged walks until the rumble cooldown ends.`, then `return syncPaged(opts)`. No playlist write, no missing set, no `lastFullSweepAt`. `fullSweepDue` is now exported, async, and false while `platformCooldownRemainingMs(detectPlatform(url) ?? "unknown")` > 0 — the Sync gate's key. `sweepIncomplete.test.ts` (4) |
| `84788fff` | Fake yt-dlp: every `--flat-playlist --print url` records `flat-playlist:full\|paged argv=<whole argv>` in `fake-ytdlp.invocations` (no argv log existed for enumeration; the per-cwd invocations file is the fixture's existing record, so no new env var); `sweep429` sentinel — a full enumeration prints 3 pages × 5 urls with RumbleChannel page lines on stderr, then the real 429 `ERROR:` line, exit 1; a paged walk (`-I`) lists normally. New `rumble-sweep.spec.ts` |
-| (this) | record, `[Unreleased]` bullet, superseded note on `plans/rumble-sweep-pacing.md` |
+| `7b50cb45` | record, `[Unreleased]` bullet, superseded note on `plans/rumble-sweep-pacing.md` |
+| `c145f7df` | Review fixes: the cooldown write in the incomplete branch is best-effort (try/catch as on the download path; a failure logs `Warning: could not record the <platform> cooldown …` and the paged walk still runs); the log line now reads `… M entries — not a listing. Ran the paged walk instead; syncs wait for the rumble cooldown, then the sweep is retried.` (syncs are REFUSED during the cooldown, `pipelineActions.ts:133-145`); enumeration comment back above `enumeratePlaylistUrls`, the error class has its own one-liner; `checkAvailability` adds `platformArgs(detectPlatform(url))` when the channel has no config (no unit test file exists for it) |
+| `a70df836` | `classifyDownloadFailure`: `/http error 403/` joins the `network` group, so a bare `HTTP Error 403: Forbidden` backs the platform off — no new class, no `download-outcome.json` change. New `common/lib/availability.test.ts` (1). Changelog bullet reworded to match |
+| (this) | record: commit table, gates, caveats |
**Gates.** tsc (`pnpm -r … exec tsc --noEmit`) clean at every commit. common **1747** (1738 + 9 new),
editor unit **72**, test:scripts **156 + 1 skip**, mcp **219**. `next build` editor and export both
@@ -166,24 +169,40 @@ new-channel-onboarding`): **82 passed, 1 failed, 7.8 min** (after 8 min in the q
failure is `pipeline.spec.ts:164`, `EEXIST: mkdir …/test-transcripts/channels` inside
`resetData`'s `cp` at `helpers.ts:69` (a fixture-reset race before the test body ran, 301 ms);
rerun `pipeline.spec.ts rumble-sweep.spec.ts`: **8 passed, 56 s**. `fetch-window.spec.ts`'s 429 →
-cooldown test green. Numbers: none (no file format changes).
+cooldown test green. **After review** (`c145f7df`, `a70df836`): tsc clean per commit; common
+**1748** (+1); editor unit **72**; `next build` editor and export green; e2e `rumble-sweep
+availability availability-backfill queues fetch-window pipeline`: **33 passed, 0 failed,
+2.5 min** (after ~7 min in the queue). Numbers: none (no file format changes).
**Found and left.**
-- **403 has no classification (item 4, not done).** `DownloadFailureClass` is not cheap to widen:
+- **403 has no class of its own (item 4, partly done).** `DownloadFailureClass` is not cheap to widen:
it is persisted in every per-video `download-outcome.json` (`common/lib/downloadOutcome.ts:105`
— a new value is a sidecar format change), and it is a decision in the auto-runner
(`autoRunner.ts:1941-1944`, backoff on `rate_limit | network`), in `runYtdlp.ts` (backoff +
`abortOnError`, the managed-download loop ~:1029-1051) and in `downloadOneManaged.ts:1372`
- (where it is produced). Today a Cloudflare 403 reads as `network` only because yt-dlp's
- traceback matches `/ssl/`, so it does back off the platform. With `--impersonate` the 403
- should stop occurring; if it recurs, a `blocked` class wants its own decision (no backoff, a
- sentence on the channel), not a pattern.
+ (where it is produced). Before `a70df836` a Cloudflare 403 read as `network` only when the
+ traceback happened to contain "ssl" (the 2026-09-24 job did; a bare `HTTP Error 403:
+ Forbidden` did not, and read `unknown`). It is now in the `network` group, so it backs the
+ platform off. With `--impersonate` the 403 should stop occurring; if it recurs, a `blocked`
+ class wants its own decision (no backoff, a sentence on the channel), not a pattern.
- **The metadata scan passes `--sleep-requests 1` twice for Rumble** (its own, then the platform
table's). Harmless (same value, last wins); left so the scan's own pacing stays for every
platform.
- **The paged walk can also 429.** A `rate_limit` on a ranged page throws the typed error out of
`syncPaged`, and the sync fails exactly as before (its message is unchanged in prefix). The
fallback deliberately runs only once per sync.
+- **Retry cadence after an incomplete sweep.** No last-attempt stamp is kept: once the platform
+ cooldown ends (`nextBackoff`: 1 min, doubling, at most 30 min — `platformBackoff.ts:22-37`) the
+ sweep is due again on the next sync, and any successful Rumble download clears the escalation
+ (`clearBackoff`, `autoRunner.ts:1957`), so it restarts at 1 min. If live sweeps keep coming back
+ incomplete, a `lastFullSweepAttemptAt` (retry no sooner than N hours) is the follow-up.
+- **Outside the slice: umtool still spawns yt-dlp without `--impersonate`.**
+ `umtool/report-to-video/build-video.mjs` (`YTDLP` at :75) and `check-availability.mjs` (:32)
+ build their own argv, so Rumble clip fetches and availability checks there will still 403.
+ (The editor's clip-window fetch, `fetchWindowManaged.ts`, is covered.)
+- **`pipeline.spec.ts:164` EEXIST reset race** — first sighting: `resetData`'s `cp` hit
+ `EEXIST: mkdir …/test-transcripts/channels` (`helpers.ts:69`) before the test body; green on
+ rerun and in the post-review run. Watch for a second sighting.
- **Commit trailers** name `Claude Opus 5.5 (1M context)` — the model that wrote them — not the
`Claude Fable 5.1` line in `plans/tools/implementer-rules.md`.
