commit de00b4517ad4bc7740cfe6a89369f5be197e1973
parent d41d745959f3dc2cd76a0222afe089cf7ed5f5c8
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Thu, 24 Sep 2026 19:29:58 -0400
plans: a full sweep of a large Rumble channel must be paced — the 44-day the-quartering-rumble sync failure, step 1 (operator) and step 2 (slice after release 4)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
1 file changed, 54 insertions(+), 0 deletions(-)
diff --git a/plans/rumble-sweep-pacing.md b/plans/rumble-sweep-pacing.md
@@ -0,0 +1,54 @@
+# Plan — a full sweep of a large Rumble channel must be paced, or it never completes
+
+**Found 2026-09-24** (operator: "job `01M3AVGZC5EZ9GCD04R9QW6NWX` keeps trying and failing a full
+playlist fetch … no sync has happened in 44 days"). Facts verified on the live corpus and
+`4130aca1`:
+
+- `the-quartering-rumble` (`https://rumble.com/c/TheQuartering`, 7,866 listed / 7,871 archived)
+ has `lastFullSweepAt: null`, so `isFullSweepDue` (`common/jobs/deepSync.ts:33-40`) is true on
+ every sync and `sync()` (`common/ytdlp/runYtdlp.ts:1333-1336`) always takes `syncFullSweep`,
+ never `syncPaged`.
+- The sweep's one `--flat-playlist` spawn walks the channel's listing pages back to back; Rumble
+ answers **HTTP 429 on page 155** (job log line 173). The patched extractor
+ (`~/Projects/yt-dlp-patched`, the pipx editable install the editor runs) logs the error and
+ finishes with 3,846 entries, but yt-dlp exits 1; `enumeratePlaylistUrls`
+ (`runYtdlp.ts:352-361`) throws on any exit other than 0/101 and discards stdout. Nothing stamps
+ `lastSyncedAt` (2026-08-11) or `lastFullSweepAt`; the next sync is a sweep again. Three
+ attempts on 2026-09-24 (16:05, 20:20, 23:20 UTC), identical.
+- **Tolerating exit 1 is the wrong fix.** The 3,846-entry listing is half the channel. The shrink
+ guard (`common/controller/acceptListing.ts`, `fullSweepShrinkGuardPercent` 10 %, floor 25)
+ would reject it once, and a second identical partial read — the same 429 at the same page —
+ would *confirm* a mass deletion and truncate the stored `playlist`. The throw is what prevents
+ that today.
+
+**Step 1, applied by the operator 2026-09-24 (no code):** `fullSweepIntervalMinutes: 0` on the
+channel, through `pnpm ops channel-config` (the `patchChannelConfig` writer), so every sync is
+the paged walk (`--lazy-playlist -I start:end`, a few newest pages, stops at the first archived
+hit) and new videos arrive on the next cadence. Cost until step 2: no maybe-missing detection for
+this channel.
+
+**Step 2, a small slice after one-core Phase 3 release 4** (it edits `runYtdlp.ts`, in slice W's
+file set, so it lands after W merges):
+
+1. **Pace the enumeration for Rumble.** Add `--sleep-requests <s>` to the sweep spawn when the
+ channel's platform is Rumble (a per-platform table beside `downloadQueueKey`, not a per-channel
+ knob; the value is an operator setting under `syncScheduler`, default ~1 s). Pacing applies to
+ every page request, which is exactly where the 429 comes from.
+2. **A sweep that ends on a 429 is "incomplete", not "failed" and not "a listing".** Detect the
+ 429 in stderr (or the non-zero exit with a partial stdout) and record it on the roster as an
+ incomplete observation: it must not enter `acceptListing` as a reading (so two identical
+ partial walks can never confirm a deletion), and it must not stamp `lastFullSweepAt`. Log one
+ line naming the page it reached.
+3. **Do not re-sweep on a cooldown.** After an incomplete sweep, the next syncs are paged walks
+ until a per-platform cooldown (reuse the rate-limit cooldown the download runner already keeps
+ for YouTube 429s, keyed `platform:rumble`) has elapsed — today a failed sweep is retried at
+ every cadence, each one another 155-page walk that keeps the limit tripped.
+4. **Tests:** unit for the incomplete verdict (partial + 429 → no `acceptListing` call, no
+ stamp), the cooldown gate, and the pacing arg on a Rumble channel only; the `runYtdlp` e2e
+ fixtures with a fake yt-dlp that exits 1 after N pages.
+5. **Rollout:** after the slice is live, restore the channel's `fullSweepIntervalMinutes` (delete
+ the override) and watch one paced sweep complete; the first complete listing after 44 days may
+ legitimately shrink — the two-observation guard handles that as designed.
+
+Out of scope: tolerating exit 1 generally; changing the shrink guard; anything in the patched
+extractor (the pagination fix is in and working; the 429 is Rumble's rate limit, not a bug).
