commit e34d473336a9c0c5dd10da54ed9a38c411192bda
parent 4495d56a049dd94c04716bdf4f5b65d79a8fe6d9
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 22 Sep 2026 17:03:55 -0400
plans: STATE for 2026-09-22 — the release, and two notes that expired
Rewritten head per README: the curated-tags rollout (including the three seed
patterns that were wrong against the real corpus, which is the part worth
carrying forward), `storage/debts-1`, `feat/filtered-channel-followups`, and
this release's three branches with per-item shas, the measured gates, the
housekeeping, and the four post-release operator steps in order.
Two standing notes are retired because following them would now waste time:
- The `editor/content` e2e blocker is deleted, not annotated. The symlink is
gone and the path is gitignored, so `pnpm e2e` runs in the primary checkout
and the "add by path" discipline is no longer load-bearing. Leaving a
blocker in place after it lifts costs every future session a worktree.
- The four-legacy-pause-fields bullet moves to done. Its warning was heeded —
`defaultHeldFor` makes backfill default `true`, so the lane gate B fences is
off twice over — and the invariant is asserted through `sanitizeAutoQueue`,
where a reader hits it, rather than through the default in isolation.
Decisions added: s0-pause ships now rather than waiting on the live file,
2origin was fixed rather than skipped (a red run there is a regression now),
hub tag federation is out, and only merged branches were deleted. Commands
updated to the measured numbers, with the `run e2e*` scripts named instead of
`exec playwright test` — the scripts are what take the machine-global lock.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
| M | plans/STATE.md | | | 221 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------- |
1 file changed, 200 insertions(+), 21 deletions(-)
diff --git a/plans/STATE.md b/plans/STATE.md
@@ -3,9 +3,148 @@
The working memory for the local-AI derived-corpus work. Rewritten at the end of every
session, before context is cleared. See [`README.md`](README.md) for the protocol.
-**Last updated:** 2026-09-20 — **storage locations S5 + S6 shipped** on branch
+**Last updated:** 2026-09-22 — **the release *curated-tags follow-ups + debts sweep* is on
+`main` @ `766e0873`.** Base `4ac8ceda`; three branches merged in the order
+`1a011d96` alone → `tags/rules-and-ops` (`9837f066`) → `export/tags-followups` (`cb254919`)
+→ `editor/debts` (`766e0873`). Anchors for every seam are in
+[`FACTS.md`](FACTS.md#the-2026-09-22-release--curated-tags-follow-ups--debts-sweep-main--766e0873).
+
+**Final full-suite run on 766e0873: PENDING-FINAL-GATES**
+
+### Curated per-video tags — shipped, reviewed, deployed to Anilyzer
+
+The plan is [`curated-tags.md`](curated-tags.md); it is now a shipped design, not work in
+flight. `transcripts/tags.json` is the authoritative vocabulary AND every assignment, written
+through the ONE path `applyTagAssignments` (`common/lib/curatedTagsStore.ts`) by all three
+writers (editor UI, `pnpm ops`, umtool); the MCP gained a read-only `list_tags` and a `tags`
+filter and no write tool. The record field is **`curatedTags`**, never `tags`. S1.1 froze the
+model (`d8c9d252`), then S1 ‖ S2 ‖ S3; review fixes `86cbb616`, `468f3a31`; the two
+re-apply bugs `edf0e204` (it held the event loop for minutes) and `8db8a3be` (the
+assignment-only pass destructured a bare key).
+
+**Rollout, 2026-09-21 → 09-22.** Seeded through `pnpm ops tags --file` (never by hand — the
+file carries provenance). First derivation: examined 77,224, re-derived 831 in 623 s; 103
+transcript pages rewritten, 906 unchanged. **All three seed patterns were then wrong against
+the real corpus and were corrected** — the full before/after is in `curated-tags.md`'s rollout
+log, and the shape of the error is worth carrying: a bare `elfpire` matched a friends-list
+boilerplate on MommaOcco (200 false hits), `@PapaElfpire` in chat, and "Jason Angelfire" /
+"channelfireball" / "RekietaLawLive" in ASR. **ASR never spells "Elfpire"**, so the caption
+rule is a misspelling alternation with `\b` on both ends.
+
+**Anilyzer is live with Eva tags** (`4ac8ceda`), reviewed, with 21 suppressions. Two bugs the
+live data found were fixed there. Every other site stays at corpus spec 3 until the operator
+chooses to rebuild it — a pre-spec-4 site serves no `/tags.json`, which the MCP reports as
+empty-with-a-warning rather than as a silent miss.
+
+**And a rule can exclude channels** (`1a011d96`): `channelsExclude` is applied AFTER
+`channels` and **the exclusion wins**. It is part of `hashCuratedRules`, which is the trap —
+see the owed steps below.
+
+### `storage/debts-1` — merged 2026-09-21 (`7589c50c` → `e109b6ac`, fast-forward)
+
+The three things S5/S6 named and left out; two of them are now done and the plan file says so.
+`assertRelocationRootPresent` (`1fccabac`) — a move can no longer `mkdir` its destination
+under an absent mount, checked from the preview AND immediately before each copy phase, on the
+channel mover and the saved-video mover both. A `/review` section for auto-paused channels
+(`bbc1899e`) — `common/views/review.ts` `autoPausedRows`, and a channel a PERSON paused is not
+a row. **Only the per-unit reachability re-check in the two lane runners is still open**, still
+deliberately deferred to the five-minute storage watch.
+Around them: `clips/` counted in `totalMediaBytes` (`7589c50c`) so the platter's bytes stop
+lying, clip-window eviction BY AGE (`4416bb58`, `e4f9fa1f`), the relocate space check freed
+from how much RAM is free (`1271e21f`), a fan-out that says which jobs it started
+(`94b24fdf`), the full-source path (`fab61a49`), and the umtool suite SKIPPING rather than
+going red without the song data (`1ab07623` → `007c12e5`).
+Anchors: [`FACTS.md`](FACTS.md#storage-debts-verified-2026-09-21-branch-storagedebts-1).
+
+### `feat/filtered-channel-followups` — merged 2026-09-21 (`337477d5`)
+
+Nine commits, `f142f7d9` → `c442cb60`, behind the per-channel download filter. A rejection
+takes its own prefetch directory with it; the metadata scan is DISPATCHED by the download
+lane, not waited for, and goes in the pick log like any other unit; a filtered-out livestream
+gets a third answer rather than settling forever; `videoHasAnyArtifact` turned out to be
+`isVideoDownloaded` under a second name. Review (`5c02e7da`) found the discard was a
+deny-list and four things it let through.
+Anchors: [`FACTS.md`](FACTS.md#filtered-channel-follow-ups-verified-2026-09-21-branch-featfiltered-channel-followups).
+
+### The release itself — what each branch landed
+
+**A · `tags/rules-and-ops` → `9837f066`.** `1a011d96` (merged alone, first) `channelsExclude`.
+`dcd065e7` a site layer carries NO rules — dropped on read AND on write, so a hand-written
+site rule is removed from the file on the next save. `d653b504` `--wait` survives a failed
+poll: the log route is POLLED (`{content, nextOffset, status, …}`), a poll failure is not a
+job failure, there is an `/api/jobs/active` fallback, probe failures are bounded, a
+`--wait-timeout`, and `followJob` is importable with injected `fetch`/`sleep`. `52f5c64c`
+both build routes take `siteId` OR `siteIds` via `reqSiteIds`, and `build-deploy` fans out
+returning `{ok, jobs, skipped, jobId?}`. `6214eeb9` `wt rm` finds the directory `wt add` made
+(`worktreeDirFor`, one function for both verbs). `729f64ef` a measure-nav comment. `bad9ea43`
+the four-lane status poll reads the corpus ONCE. `d0df901b` the `typeof process` guard in
+`contract.ts`. `94750a1b` review fixes — a bad site id is 400 *before* any job, and a
+recovered poll answering `queued`/`running` is not an outcome.
+
+**B · `export/tags-followups` → `cb254919`.** `e942dc7f` tag chip groups fold (`<details>`
+per group, local state, `"n selected"` on the summary). `ce5379f2` `SearchResults`' leaf
+sections read `SCOPE_LABELS` — one scope-label table, no second copy anywhere.
+`f40c4257` + `7a38b60b` hub `/ask` prerenders, and the fix is that `AskHub` needs the WHOLE
+provider stack (`PlayerProvider` → `MultiSiteDataProvider` → `SearchSessionProvider`), never
+an opt-out; `2e13c0ab` its spec, `export/e2e-hub/ask.spec.ts`. `befd2fbf` MCP: a tag filter
+takes posts OUT of the search, reported in `postsScanned.skippedForTagFilter` and in a footer
+sentence, because posts carry no curated tags and the export UI does the same.
+
+**C · `editor/debts` → `766e0873`.** `94c965d7` + `674c976b` the channel Storage panel evicts
+that channel's clip windows (`ClipWindowsCard` with a `slug`, disabled by the panel's
+`blockedReason`, by-age caveat on screen). `d23883f7` rename and delete refuse with
+`channelMediaBusyReason`. `1b3d4837` Sync all skips a channel whose media drive is not
+mounted. `1cace934` worker auth's 503-when-unset asserted. `1e0b10a9` `.gitignore`
+`/editor/content`. **S0-pause landed** — `98fa001a` the four legacy pause fields and their
+migration deleted (`migrateHeldToLanes` gone), `27b6be0f` `held` defaults per lane via
+`defaultHeldFor` (backfill `true`, the others `false`), `ad20ffd8` the editor readers gone,
+`7d2e889a` `settingsWrite.test.ts`. Review fixes `4d2fa0ca` + `70c2443a` (**every**
+`/api/test/*` route 404s unless `EDITOR_TEST_ROUTES=1`), `8ad4ea77` (the `held` default
+asserted through `sanitizeAutoQueue`), `67f17b30`, `307c9b81`.
+
+### Gates measured for this release
+
+`tsc --noEmit` clean in `common`, `editor`, `export`, `mcp`. Unit: common **1604** (was
+1592), mcp **219** (was 216), `pnpm run test:scripts` **151 passed + 1 deliberate LIVE skip**
+(was 138). Per-branch e2e: A `tags.spec.ts ops-api.spec.ts` **24/24**; B export **188/188**,
+hub **6/6**, `e2e:2origin` **3/3**; C 20 specs **184/184**.
+
+**`e2e:2origin` is GREEN for the first time since hub `/ask` existed.** The FACTS entry
+calling it known-red on the base is closed — a red run there is a regression now, not a
+known failure to skip past.
+
+### Housekeeping
+
+Worktrees `storage-locations-s0..s4` and `s0-pause` removed. Branches deleted:
+`storage/locations-s0..s4`, `storage/debts-1`, `feat/filtered-channel-followups`,
+`one-core/phase-1`, the seven `umtool/*`, `one-core/s0-pause`. **Kept:**
+`parked/download-filter-preview`, `p1-e2e`, `worktree-duplicates-page`, `diet-series`. The
+three release worktrees and branches (`tags/rules-and-ops`, `export/tags-followups`,
+`editor/debts`) go once the final gates are in. Nothing unmerged was deleted.
+
+### Owed — the post-release operator steps, in order
+
+1. **Re-send each Eva tag definition** via `pnpm ops tags --file`, with
+ `rules[].channels: []` and `channelsExclude: ["elfpire-eva", "elfpire-eva-x"]`.
+2. **`pnpm ops build-index --wait`.** Expect ONE full re-derive of the whole corpus, ~80 s,
+ and the log line `curated tags: rules <hash8> (changed)`. This is not the exclusion doing
+ work — `channelsExclude` is part of `hashCuratedRules`, so merely shipping the field
+ changes the hash for every existing rule set and the first build after the deploy walks
+ every record. It happens once. Do not go looking for a bug.
+3. **`pnpm ops build-deploy --json '{"siteId":"anilyzer"}' --wait`**, then verify:
+ `curl https://anilyzer.pages.dev/tags.json | jq '.tags[].count'` and
+ `list_tags {source:"remote:https://anilyzer.pages.dev"}` against the allow-list build
+ (**96 / 282 / 56**). Suppress any new false positive in `/tags` → Preview.
+4. **Rebuild and restart the editor and umtool on this `main`.** Other sites stay at corpus
+ spec 3 until the operator chooses.
+
+**Out of scope, decided:** hub tag federation. A tag filter works per site; the hub does not
+aggregate `/tags.json` across members.
+
+**Previously:** 2026-09-20 — **storage locations S5 + S6 shipped** on branch
`storage/locations-s5-s6` (worktree `/home/user/Projects/storage-locations-s5-s6`, off `main`
-@ `60183f0`, unmerged). The operator's two asks that evening — *"disk space and current storage
+@ `60183f0`; **merged since — the "unmerged" below is stale, and the three things it names
+as left out are down to one, see the 2026-09-22 block above**). The operator's two asks that evening — *"disk space and current storage
volume as columns on the channels menu, and let me filter by volume"* and *"progress on
relocate jobs since rsync gives progress"* — plus the four things around them: per-channel
media bytes in the snapshot (`totalMediaBytes`), the corpus volume as a first-class `/storage`
@@ -106,20 +245,17 @@ scopes migrating into lane trees, both `enabled: false`.
`backfill.allowRedownload` and the channel scope. As configured today it would re-fetch
audio for ~66,540 videos. The migration left the lane disabled; the tree's channel leaves
are the scope from now on.
-- **The four legacy pause fields go only once the live `settings.json` carries all four
- `held` keys** — which happens on its first write through the editor. And when
- `sanitizePolicy` finally defaults `held`, **backfill's must default `true`**:
- `sanitizeBackfill` defaults `enabled: false` and `legacyGateHeld` reads that field
- INVERTED, so a `held: false` default would silently un-hold the very lane Gate B fences.
-
-**BLOCKER FOR ANY EDITOR e2e IN THE PRIMARY CHECKOUT, and it is not ours.**
-`editor/content` is an untracked **symlink to `/home/user/Projects/recipe-content`** (made
-2026-08-31). Tailwind's source detection follows it and Turbopack **panics** compiling
-`editor/app/globals.css`, so `pnpm e2e` dies before the first spec. Reproduced on `main` at
-`5ffb0e1`, so it is environmental. Every Phase 1 slice ran its suite from a `git worktree`
-instead — which needs one extra step, a composed fixture site copied into its `export/public`
-so the export webServer answers 200 instead of 500. **`git add -A` stages that symlink**, and
-a commit made that way carries the panic into every worktree of it: add by path.
+- ~~**The four legacy pause fields go only once the live `settings.json` carries all four
+ `held` keys**~~ — **DONE 2026-09-22 (S0-pause, in `editor/debts`).** The four fields
+ (`transcriptionsPaused`, `downloadsPaused`, `digest.digestsPaused`, and the inverted
+ `backfill.enabled`) are deleted from `SiteSettings`, from every sanitizer and from
+ `writeSettings`' merge literal, so a file that still spells one loses it on the next write;
+ `legacyGateHeld` and `migrateHeldToLanes` are gone. The warning this bullet carried was
+ heeded: `defaultHeldFor(lane)` is `lane === "backfill"`, so **backfill defaults `true`**,
+ and the invariant is asserted where a reader actually hits it — through `sanitizeAutoQueue`
+ in `common/jobs/autoQueuePolicy.test.ts`, not through a unit test of the default itself.
+ The backfill lane is now off twice over on a fresh install, unarmed AND held, which is gate
+ B kept as two deliberate acts.
**All four e2e flakes are fixed** (`plans/deflake-e2e.md`, `a32612a` → `fae99f1`), and the
first three's class is worth remembering: **a debounced write survives a test boundary
@@ -1300,6 +1436,35 @@ parallel task), Phase 0 transcription benchmark (a separate bottleneck).
## Decisions log
+**S0-pause ships in this release rather than waiting for the live file (2026-09-22).** The
+standing condition was "the four legacy pause fields go only once the live `settings.json`
+carries all four `held` keys". Waiting bought nothing a default could not: `sanitizeAutoQueue`
+supplies `held` from `defaultHeldFor(lane)`, which reproduces exactly the reading the retired
+fields gave a file that named no gate — false for the three paused-flags, **true for backfill**,
+because `backfill.enabled` was inverted. A settings.json written before this release is
+therefore read correctly on the first load, and loses the dead keys on its first write. The
+alternative — a fifth release carrying a migration for a field nobody may ever have written —
+was the more expensive of the two.
+
+**`e2e:2origin` was FIXED, not skipped (2026-09-22).** It had been known-red since the export
+responsive redesign, and the cheap option each time was to keep the note and move on. The
+cause was `AskHub` prerendering outside its providers, and the fix is the WHOLE provider stack
+(`PlayerProvider` → `MultiSiteDataProvider` → `SearchSessionProvider`), never
+`force-dynamic` or `ssr: false`: opting the page out of prerendering would have hidden a real
+missing-provider bug behind a build flag and cost the hub its static `/ask`. The suite is 3/3
+and the FACTS entry calling it known-red is closed — **a red run there is a regression now.**
+
+**Hub tag federation is out of scope (2026-09-22).** A tag filter works per site. The hub
+does not aggregate members' `/tags.json`, and a chip row is built from the site's own
+published file. Revisit only if someone actually wants to filter across members; the shape of
+the answer (whose counts, whose suppressions) is not obvious and nothing needs it today.
+
+**Only merged branches were deleted in the housekeeping pass (2026-09-22).** Every branch
+removed is an ancestor of `main`. `parked/download-filter-preview`, `p1-e2e`,
+`worktree-duplicates-page` and `diet-series` are kept precisely because they are not, and the
+three release branches wait for the final gates. A deleted unmerged branch is the one cleanup
+mistake that is not recoverable from the repo alone.
+
**Diarization is 2–3× SLOWER than the transcription it follows, so it does not run
inline (2026-08-07).** Measured, not assumed. GPU transcription came free from the
corpus's own `transcribe-outcome.json` records — **3,602 videos, 10,790 audio-hours,
@@ -1656,21 +1821,35 @@ playwright `webServer` with `OLLAMA_URL` pointed at it in **both** `dev:test` an
## Commands known to pass
-Established against this branch:
+Numbers measured on `main` @ `766e0873` (2026-09-22) unless noted:
```bash
-pnpm --filter yt-dlp-transcript-common run test # 356 tests (was 344)
+pnpm --filter yt-dlp-transcript-common run test # 1604 tests (was 1592)
+pnpm --filter yt-dlp-transcript-mcp run test # 219 tests (was 216)
+pnpm run test:scripts # 151 passed + 1 deliberate LIVE skip (was 138)
pnpm --filter editor build
-pnpm e2e # default dev mode; E2E_MODE=start serves a stale build
-pnpm --filter export exec playwright test # 150 tests, separate suite on :3020
+pnpm e2e # THE EDITOR SUITE, whatever spec you append — ~24 min
+pnpm --filter export run e2e # 188 tests, separate suite on :3020
+pnpm --filter export run e2e:hub # 6/6, :3041
+pnpm --filter export run e2e:2origin # 3/3 — GREEN since 2026-09-22, no longer known-red
```
-Run `tsc` per package — the brace form is mangled by fish:
+Use the `run e2e*` scripts, not `exec playwright test`: each one wraps `queue-lock.mjs`, which
+takes the machine-global lock and declares its ports. Bypassing it is how two suites end up
+driving each other's servers.
+
+**`pnpm e2e` runs in the primary checkout again** (2026-09-22): the `editor/content` symlink
+that panicked Turbopack is gone and `/editor/content` is in `.gitignore`, so `git add -A`
+cannot re-stage it. A fresh worktree still needs a composed fixture site copied into its
+`export/public` or the export webServer 500s — that was always a separate problem.
+
+Run `tsc` per package — the brace form is mangled by fish. All four are clean:
```bash
cd common && pnpm exec tsc --noEmit
cd editor && pnpm exec tsc --noEmit
cd export && pnpm exec tsc --noEmit
+cd mcp && pnpm exec tsc --noEmit
```
A `SCHEMA_VERSION` bump means a FULL index rebuild: 77k videos, **~35 min** and it re-reads
