commit 9a61334a986324dae398051625c8f92aed5d8122
parent abecb56d465dc943012863df725931e1a650d577
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Thu, 24 Sep 2026 21:46:49 -0400
plans: release 5 — record file, durable implementer rules, STATE pointer
Two parallel slices off abecb56d: R (Rumble impersonation + paced sweeps +
incomplete-not-failed) and X (per-site transcriptDownloads off), merged R → X,
one editor restart at the end. Where release-5.md corrects rumble-sweep-pacing.md
or site-exports-off.md, release-5.md wins.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
3 files changed, 234 insertions(+), 0 deletions(-)
diff --git a/plans/STATE.md b/plans/STATE.md
@@ -9,6 +9,10 @@ prelude rolls it out. The one-sync md5 sweep from the release-3 rollout is still
YouTube held a 429 cooldown across three attempts (18:43, 19:13, 19:41). See the
[rollout record](one-core-phase-3.md#rollout-2026-09-24--9ab10d77-live-on-3001).
+**Release 5 in flight (2026-09-24, late):** slices R (Rumble) and X (exports-off) cut off
+`8b7f8924` as `one-core/r5-rumble` and `one-core/r5-exports`; plan, record and rollout in
+[`release-5.md`](release-5.md); rules in [`tools/implementer-rules.md`](tools/implementer-rules.md).
+
**Last updated:** 2026-09-24 (evening). **Release 4 is on `main` @ `e172749b`, and one-core
Phase 3 is COMPLETE. Phase 4 is next.** The prelude was plans only: `4130aca1` (the rollout
record) and `bbad0977` (the owed sync sweep). The three plan files `77f63356`, `42caf3cc` and
diff --git a/plans/release-5.md b/plans/release-5.md
@@ -0,0 +1,143 @@
+# Release 5 — Rumble un-broken, visitor exports off per site, then ONE rollout
+
+Cut 2026-09-24 (late evening) off `main` `8b7f8924` (code `e172749b`, one-core Phase 3 complete).
+Two parallel slices with no file overlap, merged **R → X**, then one editor restart at the END
+(release 3 and 4 rolled the previous release out as a prelude; release 5 rolls out last because R
+is what un-breaks Rumble syncing and the operator should not wait a release for it). The owed
+one-sync md5 sweep rides along with that rollout. Site rebuilds (X) are separate from the restart.
+
+Where this file corrects `plans/rumble-sweep-pacing.md` or `plans/site-exports-off.md`, this file
+wins and the slice's record commit updates that file. Implementer rules:
+[`tools/implementer-rules.md`](tools/implementer-rules.md).
+
+## Context
+
+- The live editor on :3001 still serves the `9ab10d77` build (`BUILD_ID` `XsKaA_drqdAbTguxUGVsn`).
+ Release 4 is merged, not rolled out. `git diff --stat 9ab10d77..8b7f8924 -- umtool` is empty.
+- Rumble is broken two ways. (1) Every Rumble download 403s at `https://rumble.com/embedJS/u3/`
+ (Cloudflare fingerprinting; upstream yt-dlp #17496, open since 2026-08-20); the editor never
+ passes `--impersonate`, and the pipx venv (editable install of `~/Projects/yt-dlp-patched`, the
+ binary the editor runs) has curl_cffi targets. The operator ran the probe 2026-09-24 —
+ `yt-dlp --impersonate chrome --skip-download --print title <rumble url>` printed a title.
+ (2) A full sweep of `the-quartering-rumble` (7,866 listed) 429s at listing page 155, yt-dlp exits
+ 1, the sync discards the partial listing (correctly) and never stamps `lastFullSweepAt`, so every
+ sync was a sweep → 44 days without sync. The operator set `fullSweepIntervalMinutes: 0` on that
+ channel as a stopgap.
+- Exports-off: the operator's five published sites drop their visitor export surfaces; the
+ capability stays a per-site option, default ON, for the OSS release.
+- Owed, unchanged: the one-sync md5 sweep; 188 orphan `*.tmp-*` under `transcripts/`; Anilyzer
+ production deploy; five sites to corpus spec 4; LM chat-only tier; `transcribeOne.ts:173`.
+
+## Slice R — Rumble: impersonation everywhere, paced sweeps, incomplete ≠ failed
+
+Branch `one-core/r5-rumble`. There is no single `configArgs`: four arg-builders drift today and
+`probeChannelMeta` has none.
+
+| Builder | Where | Spawns |
+|---|---|---|
+| `configArgs` | `common/ytdlp/runYtdlp.ts:222-227` | `enumeratePlaylistUrls` :336 (sweep + paged walk + `fetchFlatPlaylistUrls`), `downloadSubsForUrl` :1097, `downloadOneAudio` :1269 |
+| `channelExtraArgs` | `common/ytdlp/channelArgs.ts:13` | `downloadOneManaged.ts` :460,654,991,1035,1103,1240 (the per-video managed download), `fetchWindowManaged.ts:201` |
+| inline | `common/ytdlp/metadataScan.ts:411-412` | metadata scan |
+| inline | `common/controller/checkAvailability.ts:210-218` (`extraArgs` :120) | quick availability check |
+| none | `common/ytdlp/runYtdlp.ts:401-448` `probeChannelMeta` | new-channel URL probe |
+
+1. **One table, one builder.** `common/ytdlp/channelArgs.ts`: `PLATFORM_ARGS: Partial<Record<
+ Platform, readonly string[]>>` = `{ rumble: ["--impersonate", "chrome", "--sleep-requests", "1"] }`
+ with a comment naming #17496 and the 2026-09-24 probe; `platformArgs(platform: Platform | null)`;
+ `channelExtraArgs(config)` prepends `platformArgs(config.platform ?? detectPlatform(config.url))`
+ (`common/lib/platform.ts:36-54`, `Platform` :5-22) before `ytdlpExtraArgs` so a channel override
+ still wins. `configArgs` delegates to it (cookies first, as now); the two inline copies call it;
+ `probeChannelMeta` gets `platformArgs(detectPlatform(url))` (a Rumble channel cannot even be
+ created today — the probe 403s). Precedent: `resolveDownloadFormatSelector`
+ (`common/ytdlp/downloadFormat.ts:41-58`). No new setting: the table is code; per-channel
+ `ytdlpExtraArgs` remains the tweak surface.
+2. **A 429 mid-enumeration is "incomplete", not "failed" and not a listing.** `enumeratePlaylistUrls`
+ (`runYtdlp.ts:319-368`): on exit ≠ 0/101 classify the captured stderr with
+ `classifyDownloadFailure` (`common/lib/availability.ts:231-263`; `rate_limit` matches `http error
+ 429`); if `rate_limit` → throw a typed `EnumerationIncompleteError {platform, pagesReached, count}`
+ (page from the last `Downloading page N` line). `syncFullSweep` (`:1544-1727`; `:1557` is the
+ enumeration call) catches it BEFORE `acceptEnumeration`: records the backoff for the platform
+ via `recordDownloadBackoff` (`common/jobs/downloadBackoff.ts:31-42`; `platformBackoff` is
+ platform-keyed, `common/jobs/platformBackoff.ts`, set by the runner `autoRunner.ts:1936-1953`,
+ read by the Sync gate `pipelineActions.ts:126-142`), logs one line ("sweep incomplete: 429 at
+ page N of the listing, M entries — not a listing; next syncs are paged walks until the rumble
+ cooldown ends"), then RUNS `syncPaged` for this sync instead of failing, and never stamps
+ `lastFullSweepAt` (`touchLastFullSweep` is already inside `if (decision.accept)`). Exit-1-with-
+ other-stderr keeps throwing as today. Tolerating exit 1 generally stays out.
+3. **No re-sweep while the platform cools down:** `fullSweepDue` (`runYtdlp.ts:1339-1350`) also
+ returns false when `platformCooldownRemainingMs` (`downloadBackoff.ts:19-27`) > 0 for the
+ channel's platform.
+4. **403 classification, small:** `classifyDownloadFailure` has no 403 pattern (the 2026-09-24 job
+ read "aborted (network)" only because the traceback matched `/ssl/`). 403 from Cloudflare is a
+ fingerprint block, not a rate: add a `"blocked"` pattern group only if the type is cheap to
+ thread; otherwise record it and leave.
+5. **Tests.** Unit: `channelArgs.test.ts` (rumble gets the four args before channel args; youtube
+ gets none; override order), `enumeratePlaylistUrls` incomplete classification, `fullSweepDue`
+ under cooldown. Fake yt-dlp (`editor/e2e/fixtures/bin/fake-ytdlp.mjs`; scenarios are URL
+ sentinels, `ratelimit` → 429 + exit 1; no "exit 1 after N pages" mode): add a `sweep429`
+ sentinel that prints N pages of urls then the 429 and exits 1, and have the fixture record
+ argv so a spec can assert `--impersonate chrome` for a `rumble.com` URL. e2e: a Rumble-URL
+ channel whose sweep hits `sweep429` → job succeeds as a paged walk, log has the "incomplete"
+ line, `.auto-queue/state.json` has `platformBackoff.rumble`, `lastFullSweepAt` absent, the Sync
+ button then reads "rumble is in a rate-limit cooldown" (`queues.spec.ts:38-73` pattern);
+ `fetch-window.spec.ts:273-296` (429 → cooldown) stays green.
+6. **Rollout notes:** after R is live, remove `fullSweepIntervalMinutes: 0` from
+ `the-quartering-rumble` and watch one paced sweep complete; the first accepted listing after 44
+ days may legitimately shrink — the two-observation guard handles it.
+
+Numbers: none (no file format changes).
+
+## Slice X — visitor exports off, per site
+
+Branch `one-core/r5-exports`. The editor does NOT mount `TranscriptModal`/`PlayerProvider` — the
+gate is export-only. There is no `pnpm ops site-config`: `site.json` is written only by the
+Settings form (`SiteForm.tsx` → `saveSiteAction` → `writeSite`). Five sites mount the modal.
+
+1. **Schema key `transcriptDownloads`** (boolean, absent = on), the `archives` four-site pattern in
+ `common/lib/siteSchema.ts` (type, docs, zod opt-out, write-if-non-default). Regenerate `SITE.md`
+ with `pnpm --filter yt-dlp-transcript-common exec tsx bin/file-schemas-docs.ts` (`--check` is
+ pinned by `fileSchemaDocs.test.ts`).
+2. **Form:** `SiteForm.tsx` checkbox beside `archives` — label "Per-video transcript downloads
+ (Download menu, Copy Markdown, Copy download command)"; `actions.ts` reads it like `archives`.
+3. **The gate:** `PlayerProvider({children, features?})` takes `features: { transcriptDownloads:
+ boolean }` (default all on) into its context; `TranscriptModal` hides the three controls when
+ off (Copy download command, the Download dropdown, Copy MD). Mount sites pass
+ `features={{ transcriptDownloads: currentSite().transcriptDownloads !== false }}`:
+ `export/app/(workspace)/SiteWorkspace.tsx`, `export/app/duplicates/page.tsx`; hub pages
+ `HubHome.tsx` and `AskHub.tsx` read the hub's own `_homepage` site.json the same way (the hub is a
+ published surface and follows the same key; default on).
+4. **Tests.** Export e2e: a fixture site with `transcriptDownloads: false` → the three buttons
+ absent (by accessible name), default → present; an `archives: false` fixture → no `/downloads`
+ link, no zip manifest (new coverage); `phase3-files-numbers.ts` still reports zero unknown keys
+ on both sides. Editor e2e: the site form round-trips the new checkbox.
+5. **Rollout (after merge and the editor restart):** for each of jeralyzer, rekietalyzer,
+ hasanalyzer, anilyzer, bonnellyzer: Settings form → untick archives and transcript downloads →
+ save → build-site + deploy-site. Verify per site: `/downloads` absent, transcript modal has no
+ Download/Copy MD/Copy download command, `/corpus.json` and one `page-0001.json` still 200.
+ `use-with-ai` and `llms.txt` already gate their archives prose.
+6. Out of scope: gating the machine contract; "Copy share link"; umtool clip fetching; pruning the
+ archives R2 volume.
+
+Numbers: `plans/tools/phase3-files-numbers.ts` before/after — the new key must not appear as unknown.
+
+## Rollout at the end of release 5 (ONE restart)
+
+Template: `plans/one-core-phase-3.md` "Rollout 2026-09-24 — `9ab10d77` live on :3001". Steps:
+(1) offline round-trips `phase3-settings-numbers.ts live=settings.json` and `phase3-files-numbers.ts`
+over live sites/configs; (2) md5 baseline of `settings.json`, 6 `site.json`, 71 `config.json`;
+(3) detached `pnpm --filter editor build` into the live `.next`; (4) ONE restart: TERM the `pnpm run
+start` + `next-server` whose cwd is `<repo>/editor`, wait :3001 free, `setsid nohup pnpm run start
+-H 0.0.0.0`, poll `/` and `/tags`; (5) smoke = the 8 old/new view pairs stripped of live fields,
+`/api/view/<bogus>` + `invalidate-cache` 404, `/api/widget/presets` 200, `?rev=` `changed:false`,
+pages 200 incl. `/channels` (rack + group Transcribe enabled with a count), `/jobs`,
+`/operations/diarization`, one channel, one video, `BUILD_ID` changed, no `ZodError`; (6) md5 sweeps
+after boot, after ONE sync of ONE channel (a Rumble channel proves R live), after one form save;
+(7) `the-quartering-rumble`: remove the sweep override and watch one paced sweep; (8) record +
+STATE. umtool: rebuild only if `git diff --stat <live>..<new> -- umtool` is non-empty.
+
+## Then — Phase 4
+
+`plans/one-core.md:460-495` is the spec; deferred follow-ups are listed in
+`one-core-phase-3.md` "Next — Phase 4".
+
+## Record
diff --git a/plans/tools/implementer-rules.md b/plans/tools/implementer-rules.md
@@ -0,0 +1,87 @@
+# Common rules for every release implementer (one-core, release 5 onward)
+
+You are one Opus implementer for ONE slice of one release in the repo `yt-dlp-transcript-browser`.
+You work in your OWN git worktree (path given in the slice prompt), on your own branch, with your
+own port block (`pnpm wt list` from the primary shows it). Other implementers work in sibling
+worktrees at the same time on files that do not overlap yours; the shared files are only
+`editor/CHANGELOG.md` and the release's record file (named in the prompt).
+
+Read `AGENTS.md` (the repo's instructions) FIRST, then `plans/STATE.md` and `plans/FACTS.md`
+(verified facts, trust them over re-deriving), then the previous release's "as shipped" records
+(the prompt names them) — your record must have the same shape. `node_modules/next/dist/docs/` is
+the Next.js reference for this version.
+
+## Non-negotiables
+
+- **Scratch dir:** the job's `$CLAUDE_JOB_DIR/tmp` (`$T`, path in the prompt). Never `/tmp`.
+ Prefix your files with your slice letter.
+- **Detached, then waited on in short foreground loops — never polled with `run_in_background`,
+ never parked on a Monitor.** Anything over a minute (builds, e2e, numbers tools) is launched
+ `setsid nohup sh script.sh > $T/<x>.log 2>&1 < /dev/null &` with the script ending in
+ `echo exit=$?`. Then wait INSIDE foreground Bash calls of at most ~100 s each:
+ `for i in $(seq 1 18); do grep -q 'exit=' $T/<x>.log && break; sleep 5; done; tail -3 $T/<x>.log`
+ — repeat the call until `exit=` appears. A subagent parked on the `Monitor` tool is never woken
+ (Monitor is for the parent session only), and `run_in_background` is killed at ~2 min here.
+ If your session is cut and resumed, re-read the logs and continue from where the logs say you
+ are — never redo a run whose result is on disk.
+- **e2e is queued machine-wide** (`AGENTS.md`): `pnpm e2e` from YOUR WORKTREE ROOT waits for the
+ lock (`waiting for the e2e queue — held by …` is normal; another suite can hold it ~25–36 min).
+ Before launching, check no orphan holds your worktree's ports (`ss -ltnp | grep :<port>`). Write
+ the exact spec list to a file and pass it: `pnpm e2e $(cat $T/<x>-specs.txt)` — from the
+ worktree root, which runs the EDITOR suite; a spec name that does not exist is dropped and
+ recorded. The EXPORT suite is `node scripts/worktree.mjs run -- pnpm --filter export run e2e`
+ (also from the worktree root, also queued). Record `N passed, M failed, time` from the log for
+ every run.
+- **Worktree e2e needs `export/public`:** the gitignored entries under `export/public` must be
+ symlinked from the primary checkout per path (under `sh`, not fish/zsh):
+ `for p in $(git -C /home/user/Projects/yt-dlp-transcript-browser/export status --ignored --short public | awk '{print $2}'); do p=${p%/}; ln -sfn "/home/user/Projects/yt-dlp-transcript-browser/export/$p" "<worktree>/export/$p"; done`
+ — check for dangling links (`find <worktree>/export/public -xtype l`) before every export build
+ or export-related e2e, because the live editor's build-deploy job regenerates the primary's.
+- **Export build gate is `pnpm --filter export exec next build`** — NOT `run build`, which runs
+ `build:data` into a worktree-local `transcripts/`.
+- **tsc per commit:** `pnpm -r --no-bail --workspace-concurrency=1 exec tsc --noEmit` from the
+ worktree root must be clean before every commit (worktrees have no stale `.next/dev/types`).
+- **Never boot a second editor against the real corpus.** Your worktree's e2e uses its own
+ fixture; never point anything at `<primary>/transcripts` except the read-only numbers tools,
+ which copy to scratch. Never hand-edit `transcripts/**`. Never restart the live :3001 editor.
+- **Never stage** `settings.json.pre-priority-*` or anything under `test-results/`.
+- **Commits** are small, each tsc-green, message in the repo's voice (`channels: …`, `common: …`,
+ `plans: …`), and every commit message ends with:
+ ```
+ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
+ ```
+ Commit incrementally (a session limit can cut you mid-slice; work on disk and in commits
+ survives, work in your context does not). Do NOT push. Do NOT merge into `main` — the parent
+ merges. A LATER slice merges `main` and re-gates once an earlier slice has landed.
+- **Labels are contracts.** Every accessible name / test id the e2e specs use survives unchanged
+ unless the slice prompt says otherwise. Grep before renaming anything.
+- **Tailwind v4, no config:** class strings must be complete literals (never `z-${n}`).
+- **Do not touch files owned by another slice** (listed in your prompt). If you find you need to,
+ stop and say so in your report instead.
+- **Record + changelog:** your last code commit is followed by a `plans:` commit adding a
+ "### Slice <X>, as shipped" section to the release's record file (before its "## Rollout"
+ heading) in the shape of the previous release's records: what/why, the commit table (sha → one
+ line), gates with numbers (tsc, common tests, editor unit, test:scripts, mcp, builds, e2e per
+ run with counts and minutes, numbers tool), what was found and left, and `[Unreleased]` bullets
+ in `editor/CHANGELOG.md`. If the slice's plan file was corrected by the release plan, the same
+ commit updates that plan file.
+- **Report** to `$T/<x>-report.md` (path in the prompt): commit shas, gates with numbers, anything
+ not done and why, questions for the reviewer. End your final turn with `SLICE_DONE <branch tip>`
+ or `BLOCKED: <reason>`.
+
+## Gates, per slice (all from the worktree root)
+
+- `pnpm -r --no-bail --workspace-concurrency=1 exec tsc --noEmit` — clean.
+- `pnpm --filter yt-dlp-transcript-common test` (1738 at release 4; record the new count);
+ editor unit `pnpm exec tsx --test "app/**/*.test.ts"` run in `editor/` (72); `pnpm run
+ test:scripts` (156 + 1 skip); mcp `pnpm --filter yt-dlp-transcript-mcp test` (219) — check
+ `package.json` for the exact script names before running.
+- `pnpm --filter editor exec next build` and `pnpm --filter export exec next build`.
+- The slice's e2e spec list (in the prompt), detached and waited on as above.
+- The numbers tool the prompt names, diff-empty (or "none", stated).
+
+## Reviews
+
+Reviews are a separate Opus agent, read-only, written to a file the parent names, with a verdict
+line `SHIP | SHIP AFTER FIXES | BLOCK`. Fixes go back to the implementer by message; the reviewer
+re-reads the fix commits.
