commit 2b49aba10eef4ed31c5e30c7d81747206506d66c
parent a7f65f2f4e8d739e02fc9c7778f41e1f8343a31e
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Fri, 25 Sep 2026 12:57:55 -0400
plans: release 7 slice K record — pnpm ops keep-videos
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
2 files changed, 68 insertions(+), 0 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,6 +1,7 @@
# Changelog
## [Unreleased]
+- **New ops action: `pnpm ops keep-videos` marks every video of a channel whose title or description matches a pattern as "do not clean".** It sets the same marker as the video page's *Do not clean* toggle, so the clean sweep, extra-format cleanup, wrong-format removal, the superseded-subs purge and saved-video eviction all leave those videos alone. The body is `{"slug", "match", "fields"?, "note"?, "dryRun"?}`. `match` is matched the way a channel's download filter *include* is: a case-insensitive regex over title + description. `fields: ["title"]` or `["description"]` narrows it to one half, and `dryRun: true` reports without writing. Videos that already carry the marker are counted and left as they are. The marker lives in the video's folder, so a match that was never downloaded is listed under `notDownloaded` and no folder is created for it. Run `download-missing` on those ids, then run `keep-videos` again. On a new channel, run `metadata-scan` first: a video with no scanned title cannot match, and the reply counts those as `unscanned`.
- **Auto-download no longer retries the same rate-limited video over and over; it moves on to the next one.** When a download answered HTTP 429, the runner paused the whole platform for a while and then picked the same video again, because it was still first in the queue. Each retry doubled the pause, up to 30 minutes. On 2026-09-24 one YouTube Short was retried 12 times this way and kept YouTube paused all evening. A YouTube 429 comes from the subtitle fetch for one video, not from the whole site. Now a rate-limited video is also **deferred for 6 hours**: auto-download skips it, so when the pause ends the runner takes the next video. The pause still grows only when *different* videos keep hitting the limit. Deferrals are kept in `.auto-queue/state.json` beside the platform cooldowns, so a restart does not retry the video early. The log line reads `… (attempt 1). <id> deferred 6h; next video after cooldown.` A manual Sync or *download missing* ignores deferrals and still fetches the video. When every video left is deferred, the runner reports that it is idle for that reason: "every pending video was rate-limited recently and is deferred".
- **The cooldown strip on `/operations/download` also lists deferred videos.** It is now a region named *Rate-limit cooldown*, with a *Platforms in cooldown* list (unchanged) and a *Deferred videos* list. Each deferred video links to its page and shows how long it has left (`alpha/a1 — 5h 59m left`). The strip appears when either list has something in it. Times over an hour now read `5h 59m` instead of `359m 58s`.
- **A video that keeps failing its metadata scan is no longer rescanned at every runner start.** When a scan hit the same error again (members-only, for example), the error's timestamp was not updated. The one-day rest that timestamp controls therefore ran out once and never started again, and each runner start rescanned all of them: 142 members-only videos on one channel, with cookies. The same error seen again after a day now updates the timestamp, so the video waits another day.
diff --git a/plans/release-7.md b/plans/release-7.md
@@ -542,4 +542,71 @@ Homepage: **15 passed, 7 skipped, 25 s**.
- **Commit trailers** name `Claude Opus 5.5 (1M context)`, as in releases 5 and 6.
+### Slice K, as shipped — `pnpm ops keep-videos` (2026-09-25)
+
+Branch `one-core/r7-keep` off `main` `3049be43`. The operator's ask: "I've just added the Paramount
+Tactical channel — mark anything that includes TheQuartering in title or description as 'keep the
+video', with an ops command." "Keep the video" is the existing per-video **do-not-clean** marker
+(`data/<id>/do-not-clean.json`, `setDoNotClean`). The clean sweep, extra-format cleanup,
+wrong-format removal, the superseded-subs purge and saved-video eviction already honour it. Before
+this slice there was only the per-video toggle: no bulk form and no ops action. The slice adds the
+loop around the marker and no new kind of protection. Two facts shaped it:
+- **Text lives in two places only.** A downloaded video's `metadata.info.json` is read first,
+ then the channel's `metadata-scan.json` entry. An id known only from `playlist` / `roster.json`
+ has no text and is counted (`unscanned`), not guessed.
+- **`setDoNotClean` does not mkdir, and nothing here creates `data/<id>/`.** A matched video with
+ no dir is reported in `notDownloaded`. Creating the dir would break the scan store's invariant,
+ and every enumerator reads a dir as "fetched".
+
+"Matches" is the download filter's matcher. The pattern is compiled by `compileDownloadFilter({include})`
+and tested by `classifyAgainstFilter` over `downloadFilterText` (title + "\n" + description, with
+the description capped at 2 KB as the filter caps it), after `downloadFilterPatternProblem` (the
+form's ReDoS guard). There is no second matcher. `fields` narrows the subject by passing the
+left-out field as `""`. **The media guard runs first** (`assertChannelMediaReachable`):
+`listChannelVideoIds` swallows ENOENT, so on an unmounted relocated channel every downloaded match
+would otherwise read as `notDownloaded`.
+
+| sha | what |
+|---|---|
+| `8f9b0fc6` | `common/controller/keepVideosMatching.ts`: `keepVideosMatching({paths, channelSlug, pattern, fields?, note?, dryRun?})` returns `{pattern, fields, considered, matched: [{id, title, downloaded, alreadyKept, marked}], marked, alreadyKept, notDownloaded, unscanned, noMetadata, dryRun}`, and `KeepVideosError` covers a bad or unsafe pattern, a bad field and an unknown channel. The default note is `keep-videos: matched /<pattern>/i`. `noMetadata` is **additive to the brief's shape**: a data dir with no `metadata.info.json` and no scan entry has a dir but no text, so it is counted rather than silently skipped. `.test.ts` has 6 cases: title + description-only + case-insensitive; already-kept counted with its mtime and note unchanged; scan-only goes to `notDownloaded` with no dir created and `unscanned` = 2; dry run; `fields:["title"]`; the typed error for `(`, a nested quantifier, an unknown field and an unknown channel |
+| `22ec333b` | `keepVideosAction` in `videoActions.ts`, beside `toggleDoNotCleanAction`. It maps `KeepVideosError` / `ChannelMediaUnreachableError` to `{ok: false, error}`. When `marked > 0` it calls `revalidatePath` for each marked video and the channel page, and `requestChannelSnapshot` **once** |
+| `f11c5a72` | `editor/app/api/ops/keep-videos/route.ts`, an adapter with keys `slug, match, fields, note, dryRun` (`reqSlug`; `fields` must be a non-empty array of `"title"`/`"description"`) that returns `{ok: true, ...result}`. `scripts/archilyzer-ops.mjs` ACTIONS gains `keep-videos`, and `archilyzer-ops.test.mjs` +1. `ops-api.spec.ts` +1 covers: a seeded `20240102_keepme12345` titled "Reacting to THEQUARTERING" plus a scan-only `scanonly123` with a description hit. The dry run writes nothing; the real call gives `marked: 1` and `notDownloaded: ["scanonly123"]`, puts the marker only under the matching id, and creates no dir for the scan-only id. An unknown key returns 400 and `(` returns 400 |
+| `73be3573` | `RUNNING_IN_DOCKER.md`, "Driving the editor without a browser": an example line and a bullet for the two-step reality (`metadata-scan` first, `notDownloaded` then `download-missing`, then re-run) |
+| *(this commit)* | this record and the `[Unreleased]` bullet |
+
+**Gates** (worktree root, on `73be3573`). tsc (`pnpm -r --no-bail --workspace-concurrency=1 exec
+tsc --noEmit`) was clean on the full tree before the commits were made. The four commits are
+additive, in dependency order. common **1795/1795** = 1789 + 6 (`keepVideosMatching`). Editor unit
+**72/72**. test:scripts **161 pass + 1 skip** (160 + 1). mcp **219/219**. `pnpm --filter editor
+exec next build` ok, and `.next/server/app/api/ops/keep-videos` was emitted. The export build was
+not run, because the slice touches no file under `export/` and no `common/` module it imports.
+EDITOR e2e, `$T/k-specs.txt` = `ops-api.spec.ts` (`k-e2e1.log`): **21 passed, 0 failed, 58.9 s**,
+after a 4 m 58 s queue wait behind the release's final suites. All heavy steps were started only
+at ≥ 3 GB available memory, as the brief required.
+
+**Numbers: none.** No `settings.json`, `site.json` or `config.json` key changed. The action writes
+only per-video `do-not-clean.json` sidecars, and only when someone runs it.
+
+**Not merged with `main` `211d4666`.** The coordinator asked for `git merge main` before the final
+gates (a test-only commit in the deploy-hub region of `ops-api.spec.ts`). The merge was **refused by
+the session's permission classifier**, so the branch is still on `3049be43`. `git merge-tree
+--write-tree HEAD main` reports a **clean** merge. The parent's merge takes it as is, and the
+`ops-api.spec` run above does not include `211d4666`'s edit.
+
+**Found and left.**
+- **The rule-shaped alternative is not built.** This is a one-shot action: a video downloaded
+ *after* the run is not marked. A persistent per-channel "keep filter" (say `keepFilter.include`
+ in `config.json`, evaluated at download time and by the cleaners) would keep future matches too.
+ That is a schema change (CHANNEL.md, numbers), so it is out of scope here. Until then, re-run
+ `keep-videos` after new downloads. It is idempotent: already-kept videos are counted, not
+ rewritten.
+- There is no UI surface. The action exists for a future bulk bar or channel-page control.
+- `keepVideosMatching.ts` has its own 12-line `readPlaylistIds`. The same helper is duplicated
+ privately in `ytdlp/metadataScan.ts` and `controller/recencyIndex.ts`, and neither is exported.
+ A third copy was cheaper than reshaping two files outside this slice's ownership.
+- `unscanned` compares playlist/roster ids with dir names. A legacy dir named `YYYYMMDD_<id>` does
+ not equal its playlist id, so such a channel over-counts `unscanned`. The count is advisory.
+- **Commit trailers** name `Claude Opus 5.5 (1M context)`, as the release-6 and release-7
+ implementers did.
+
## Rollout
