commit 77f579b3e9fef356fb6d9c4e7e1c3fd1798a69ab
parent ff5be52a3f2ef254c8cfb6fd8dbc778e5ac2edde
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Wed, 26 Aug 2026 18:39:13 -0400
plans: unified-ops step 1 planned
Diffstat:
1 file changed, 277 insertions(+), 0 deletions(-)
diff --git a/plans/unified-ops-step-1.md b/plans/unified-ops-step-1.md
@@ -0,0 +1,277 @@
+# Unified-ops step 1 — the digest work list has one definition
+
+## Context
+
+`plans/unified-operations-model.md` step 1: "Collapse the counters. `snapshot.backfill.digest`
+replaces `noDigest`." It is the step Phase C skipped, `plans/STATE.md` "Recommended next" #2,
+the piece `plans/editor-ia-slice-3.md` named as what follows, and **the dependency IA slice 4
+(`/actionable` dissolves) is waiting on**. Slice 4 wants what a flat id list cannot carry — the
+per-state split (`blocked`, `deferred`, `partial`) and the coverage pair (`eligible`, `present`)
+— which is exactly what `snapshot.backfill.digest` holds and `buckets.noDigest` does not.
+
+**Every fact below was verified against the tree at `389ccea` (clean) and the real corpus,
+read-only, on 2026-08-26.** The verification changed the shape of the step:
+
+1. **The migration the fallback exists for is already over.** `digestWorkOf()`
+ (`common/controller/channelSnapshot.ts:480-527`) reads `snapshot.backfill.digest` first and
+ falls back to `buckets.noDigest` for snapshots that predate the registry entry. Its header
+ (`:446-463`, "THE FALLBACK IS NOT DEFENSIVENESS, IT IS THE MIGRATION … three generations in
+ play") and `buildBands.test.ts:116` ("Do NOT stop reading `buckets.noDigest` until every
+ snapshot has regenerated") both describe a world that no longer exists. Measured:
+ **68 of 68** `transcripts/channels/*/snapshot.json` carry `backfill.digest` (and `eligible`);
+ `generatedAt` ranges 2026-08-11 → 2026-08-26. The `"bucket"` branch is dead code against
+ production. STATE.md's "STEP 4 IS NOT DONE AND CANNOT BE DONE FROM HERE" (`:472-477`) is
+ stale — the live editor was restarted on the new build at some point after 08-10 and the
+ snapshot scheduler did the rest.
+2. **So no rendered number moves.** Every count already flows through `digestWorkOf` and gets
+ the registry answer: dashboard/`NeedsWorkPanel`/`ChannelsTable` via `actionableNoDigestCount`
+ (`loadActionable.ts:144-146`), the widget sync coverage (`api/widget/sync/route.ts:220-235`
+ — `eligible` is already a number corpus-wide, already `eligible − blocked`), the channel
+ station (`channelFlow.ts:249`), the stage card (`stageStatus.ts:416`, `[slug]/page.tsx:423`),
+ the channel list column (`channelGroupSections.ts:171`), the bands (`buildBands.ts:185`).
+ The "published coverage percentage will change" warning in step 1's text already happened
+ silently between 08-11 and 08-26. The export site never reads `snapshot.json` at all
+ (`corpus.json`'s `digestCount` comes from `channelDigestStats`, written by the build) —
+ nothing published changes, `CORPUS_SPEC_VERSION` stays 3.
+3. **The bucket and the registry entry are two definitions of "digested", and they disagree by
+ 11,777 videos today.** Corpus sum of `buckets.noDigest` = 59,159; sum of
+ `backfill.digest.ids` = 47,382. On 65 of 68 channels the delta is exactly `deferred`
+ (registry gates on a current `cues.json`; the bucket does not); the other three
+ (`HasanAbiVODs3`, `chibi-reviews`, `destiny`) are videos with a fresh digest but no current
+ `cues.json`, which the registry calls `deferred` and the bucket calls done. The bucket also
+ has no transcript gate (it silently omits untranscribed videos — `stageStatus.ts:435-440`
+ records "how a channel of untranscribed videos read as fully digested") and no `partial`.
+ `digestTarget.ts:5` says the two "MUST agree"; they cannot, structurally. **Deleting the
+ bucket is what makes there be one definition** — the registry's `state()`, which is also
+ what the runner dispatches from.
+4. **Four "digest is special" branches exist only because of the bucket**, and all four go:
+ `buildBands.ts:180-190`, `sweepPreview.ts:88` (`reachableIdsFor`) and `:114-128`
+ (`countsFor`), `sweepRecency.ts:71`. Each is `if (id === DIGEST_OPERATION_ID) …
+ digestWorkOf(snapshot)` guarding the case "no registry entry", which after this step is the
+ same answer for digest as for diarization: no entry → no work known → coverage unknown.
+5. **No regeneration is needed**, and none should be scripted. Old snapshots keep a stray
+ `noDigest` key until their next routine regen; no reader validates bucket keys
+ (`normalizeBuckets` picks the ones it knows; there is no snapshot schema version). The docs
+ filed the transcode band "with step 1" because both would need one regen of 66 snapshots —
+ that coupling is now void, see Out of scope. Do NOT write a `common/bin/regen-snapshots.ts`:
+ an offline writer would race the live editor's `snapshotScheduler` on the same files, and
+ there is nothing it would need to do.
+
+**A corpus finding to surface, not to fix here:** corpus-wide `deferred` is **16,156** in the
+current snapshots (`the-quartering` 3,762, `destiny` 3,869, `chibi-reviews` 2,699 …), against
+**1** measured on 2026-08-10 after the normalize pass (FACTS "Phase D step 1", `deferred`
+1,987 → 1). Either new `handling: "youtube"` downloads have accumulated with no `cues.json`
+(nothing automatic runs `normalizeTranscript` for a subtitle-downloading channel —
+FACTS:2001) or something regressed the cues gate. It is work the Digest card's **Normalize
+transcripts** button exists for, and it is invisible to a reader of the old bucket. Record it in
+STATE.md as an open question for the operator; do not run normalize from this slice.
+
+## Step 0 — the plan on disk
+
+Write this file verbatim to `plans/unified-ops-step-1.md` and commit it alone:
+`plans: unified-ops step 1 planned`. (Tree is clean at `389ccea`; nothing else goes in.)
+
+## Order: three commits
+
+1. **The deletion** — `buckets.noDigest` stops being written, typed, defaulted or read; the
+ `"bucket"` branch and `source` leave `digestWorkOf`; tests follow.
+2. **De-specialising digest** — the four `id === DIGEST_OPERATION_ID` fallbacks go; digest is a
+ plain registry entry to the band builder and the sweep planner.
+3. **Docs** — FACTS, STATE, the model file, the IA doc, CHANGELOG, memory.
+
+tsc in all six packages + `pnpm -C common test` + editor units after each. e2e once after
+commit 2, **detached** (see memory `e2e-run-detached`; the queue lock is serial and a Bash
+call caps at 10 min): `setsid nohup … pnpm e2e -- backfill.spec.ts digest.spec.ts
+actionable.spec.ts channel-groups.spec.ts channel-line.spec.ts` plus whichever spec asserts the
+dashboard/LaneDeck "digested" figure or `/api/widget/sync` (grep `digested\|widget/sync` under
+`editor/e2e/` first and add what you find). Edit nothing while it runs.
+
+## Commit 1 — the deletion
+
+### `common/controller/channelSnapshot.ts` (the writer)
+
+- Delete the `noDigest: string[]` field and its 12-line comment (`:195-206`) from
+ `ChannelSnapshot["buckets"]`. Fix the sibling comment at `:217` ("the sidecar is already
+ loaded here for noDigest" → it is loaded for `digestEngines` and `digestWarnings`).
+- Delete the accumulator (`:807`), the write (`:1225`), and the freshness test + push
+ (`:1064-1073`). **Keep** the `digestEngines` counting above it (`:1050-1062`) and the
+ `digestWarnings` push below it (`:1077-1082`); reword that block's comment ("a video that
+ failed outright is ALSO in noDigest" → "is also reachable work in `backfill.digest`").
+- Delete `digestTarget` (`:812-825`, the `resolveDigestTarget` call and its 12-line comment)
+ and the now-unused imports `isSectionFresh` (`:28`) and `resolveDigestTarget` (`:40`).
+ Confirm with tsc that nothing else in the file reads `digestTarget` — the census says only
+ `:1070-1071` do; the comment at `:608` ("exactly as digestTarget is above") must be reworded.
+ The property that comment was protecting — the stage's count and the batch runner's target
+ cannot disagree — is preserved by construction: the registry entry is folded from
+ `kind.state()` against `kind.resolveTarget(...)` (`:626-634`), the same identity the runner
+ uses.
+- Rewrite `digestWorkOf` (`:446-527`): drop the `bucket` branch and the `source` field from
+ `DigestWork`. Keep the shape (`ids / reachable / blocked / deferred / partial / present /
+ eligible`) — nine callers consume it and it is the editor's one adapter from an
+ `OperationSnapshotEntry` to the digest surfaces. Keep the no-entry return exactly
+ (`present: null, eligible: null`, everything else 0/[]): "unknown, not zero coverage" is the
+ test at `channelSnapshot.test.ts:131` and the property `api/widget/sync/route.ts:49-56`
+ builds on. Replace the header comment: the migration is over; date and count it (68/68,
+ 2026-08-26); state that a snapshot with no entry is a channel whose first snapshot has not
+ been written, not an old one.
+- Comments elsewhere in the file naming the bucket (`:95`, `:449-468`) — reword or delete.
+
+### The editor's defaulting layer and comments
+
+- `editor/app/channels/[slug]/lib/stageStatus.ts:45` — delete `noDigest: raw?.noDigest ?? []`
+ from `normalizeBuckets` (it enumerates every key; tsc will insist). Reword `:406`, `:459`.
+- Comment-only references to reword so they name the entry, not the bucket:
+ `common/controller/digestTarget.ts:5` ("to fill the noDigest bucket … MUST agree" — now: the
+ registry's `resolveTarget` is the one identity; the snapshot has no second one),
+ `common/controller/digestBatch.ts:160`, `common/lib/operations.ts:1117` ("Same rule as the
+ snapshot's noDigest bucket" — there is no second rule now; say that this entry IS the
+ definition), `common/lib/digest-server.ts:85` (already wrong — `hasDigest` feeds
+ `digestEngines`/`digestCount`, not the bucket), `editor/app/actionable/lib/loadActionable.ts:138-146`
+ (drop the last sentence about the fallback), `editor/app/api/widget/actionable/route.ts:16-24`
+ (keep the refusal and the 99.87% — the step's text says the refusal must survive, and it
+ does: `:44` filters on `undownloaded || untranscribed` only — just drop "rather than the
+ `noDigest` bucket"), `editor/app/api/widget/sync/route.ts:51-58` (keep the null rule; replace
+ "when the snapshots have all been regenerated" with the real remaining case), and the
+ one-liners at `channelGroupSections.ts:168`, `channelFlow.ts:246`, `videoRows.ts:44`,
+ `SpeakersStage.tsx:20`, `[slug]/page.tsx:419`, `common/lib/operations.ts:35`,
+ `channelSnapshot.ts:90-95` — these cite `noDigest` as the *precedent* for "four states, not
+ two"; keep the precedent, name it as history ("the old `noDigest` bucket").
+- **Do NOT rename the view-model fields** `DashboardChannel.noDigest` (`dashboard/types.ts:21`),
+ `WidgetActionableChannel.noDigest` (`api/widget/actionable/route.ts:24` — a wire field the
+ widget reads), `actionableNoDigestCount`, or the `noDigestIds` prop on `DigestStage`. They
+ name "videos with no current digest", which is still what they hold. Out of scope.
+
+### Tests (commit 1)
+
+- `common/controller/channelSnapshot.test.ts:100-113` — the "prefers the registry entry over
+ the legacy bucket" test: drop the bucket from the fixture, rename to say it reads the entry.
+ `:115-129` ("falls back to noDigest") — **delete**; its property is gone on purpose. `:131+`
+ ("reports UNKNOWN, not zero coverage") — keep; extend its input list with
+ `{ buckets: { noDigest: ["a"] } as never }` **only if** it can be typed without a cast on
+ the public signature — otherwise skip; the point is a stray on-disk key is ignored, which
+ tsc already guarantees by the key not existing.
+- `editor/app/channels/[slug]/lib/channelFlow.test.ts:272-283`,
+ `editor/app/channels/lib/channelGroupSections.test.ts:320-334` — feed
+ `backfill: { digest: { …counts, ids: ["a","b","c"], eligible: 3 } }` instead of the bucket
+ (there is a `foldBackfillEntry` / `emptyOperationCounts` to build one); assertions unchanged
+ (`reachable === 3`, `through === null`, `digest.total === 3`).
+- `editor/app/components/pipelines/buildBands.test.ts:114-130` — replace the fallback test
+ with its successor: a snapshot with **no** `backfill.digest` contributes `reachable 0`,
+ `eligible null`, `present null` (an unfilled outline, never 0 %). Keep the comment's intent —
+ "a channel with no entry must not read 'all digested'" — with the new mechanism.
+- `common/lib/operations.test.ts:1137-1138` — fix the comment's cross-reference.
+- `editor/e2e/backfill.spec.ts:855-878` (the "(N) THE REGRESSION THIS STEP'S DESIGN EXISTS TO
+ PREVENT" spec, which writes a fresh snapshot) — add one line:
+ `expect(snapshot.buckets).not.toHaveProperty("noDigest")`. This is the only place the
+ writer's output is asserted end-to-end, and without it the deletion is pinned by nothing.
+
+Commit message: `snapshot: the digest work list has one definition`. The body carries the
+68/68 measurement, the 59,159 vs 47,382 figure and why the delta is `deferred`.
+
+## Commit 2 — digest is not special to the readers
+
+- `editor/app/components/pipelines/buildBands.ts:180-190` — delete the `if (id ===
+ DIGEST_OPERATION_ID && !snapshot.backfill?.[id])` detour; `addRegistryEntry` already returns
+ on a missing entry and latches `eligible`/`present` to null through `sumOrNull`. Drop the
+ `digestWorkOf` and (if unused) `DIGEST_OPERATION_ID` imports. Check the file header's
+ purity rule (`:44-45`, `noCorpusWalkInRenderPaths.test.ts`) still holds — removing an import
+ can only help it.
+- `common/controller/sweepPreview.ts:80-90` (`reachableIdsFor`) and `:110-128` (`countsFor`) —
+ delete both digest branches; `countsFor` returns `null` for any operation with no entry,
+ which is what the caller already handles for diarization. Drop the now-unused imports.
+- `common/controller/sweepRecency.ts:71` (and the imports at `:32-33`) — same deletion. This
+ file reads as binary to `grep` (memory `grep-misses-sweeprecency`): use `grep -a` or trust
+ tsc; do not conclude from a silent grep that it is untouched.
+- After this, `digestWorkOf` has **no callers in `common/`** — it is an editor adapter.
+ Leave it in `channelSnapshot.ts` (moving it is churn without a deletion); say so in its
+ header.
+- Tests: `common/controller/sweepPreview.test.ts` / `sweepRecency.test.ts` — find any case
+ that feeds a `noDigest`-only snapshot to exercise the fallback (the census found none by
+ name, but check for fixtures with `buckets:` and a digest id) and turn it into "no entry →
+ null / []". `buildBands.test.ts` was already handled in commit 1.
+
+Commit message: `operations: digest is a registry entry to every reader`.
+
+## Commit 3 — docs and memory
+
+- `plans/unified-operations-model.md` — step 1 → **DONE**, three sentences: the fallback's
+ migration had already completed on disk (68/68); the number the step warned would change had
+ already changed; what was actually deleted (the bucket, `source`, four special-cases). Fix
+ the header's "designed, not built" (STATE already calls it stale by three steps — now four).
+- `plans/FACTS.md` — new dated section "Verified 2026-08-26 — unified-ops step 1": the
+ measurements (68/68 snapshots; sum bucket 59,159 vs entry 47,382; delta = `deferred` on
+ 65/68; the three exceptions and why; corpus `deferred` 16,156 vs 1 on 08-10); the surfaces
+ and what each reads now; `DigestWork` without `source`; the four deleted branches; "no
+ snapshot schema version exists — per-field optionality is the convention, `presentOperationWork`
+ is the precedent". Also strike the "44 of 66 snapshots" line (`:1988`) as superseded.
+- `plans/STATE.md` — header line; a dated entry with the decisions (no regen; no CLI; the
+ view-model names kept; transcode band deliberately not taken and why); "Recommended next":
+ #2 done → slice 4 is now unblocked (say so), and the **`deferred` finding** as a new
+ operator question. Correct the stale "STEP 4 IS NOT DONE" paragraph in place with a dated
+ note rather than deleting it.
+- `plans/editor-operations-ia.md` — slice 4's "Depends on unified-ops step 1" → satisfied
+ (dated); the Transcode row (`:60`): its band no longer waits on step 1 — it waits on its own
+ snapshot-shape change (below).
+- `editor/CHANGELOG.md` [Unreleased] — operator-facing: `snapshot.json` no longer carries
+ `buckets.noDigest`; the digest work list is the operation registry's, the same one the sweep
+ runs from; no counts on any page change.
+- Memory (`~/.claude/projects/…/memory/`): update `slice-3-chosen-next.md` → step 1 shipped,
+ slice 4 unblocked; index line to match.
+
+Commit message: `plans: unified-ops step 1 shipped`.
+
+## Verification
+
+1. After each code commit: `pnpm -C <pkg> exec tsc --noEmit` for `common editor export
+ homepage umtool mcp`; `pnpm -C common test` (829 → expect one fewer: the deleted fallback
+ test); editor units `pnpm -C editor exec tsx --test "app/**/*.test.ts"` (85 → ±1).
+2. Grep gates after commit 2, over `common/ editor/` (source, not plans): `noDigest` → only
+ the view-model identifiers listed above plus history comments; `source: "bucket"` /
+ `source: "registry"` → 0; `DIGEST_OPERATION_ID` in `buildBands.ts`, `sweepPreview.ts`,
+ `sweepRecency.ts` (with `-a`) → 0 unless a non-fallback use remains; `resolveDigestTarget`
+ imported in `channelSnapshot.ts` → 0.
+3. e2e as above, detached, once. Known flakes from the slice-3 run (`auto-subs-replace.spec.ts:394`,
+ `cadence-ui.spec.ts:212` timeouts) are not this slice's — retry individually.
+4. **Read-only corpus check, no editor boot** (AGENTS.md: never boot a second editor against
+ `transcripts/`): with `node -e`/`jq` over `transcripts/channels/*/snapshot.json`, print the
+ count carrying `backfill.digest` (expect 68/68) and the sum of `backfill.digest.ids`
+ (expect 47,382 ± whatever the scheduler regenerated since) — this is the number every page
+ was already showing, and the figure for the FACTS entry. Do not write under `transcripts/`.
+5. Manual, on the e2e fixture (`PORT=3021 pnpm dev:test`): a channel page's Digest stage
+ heading `Generate digests (N)` and the dashboard `◆ N` badge still render; `/operations/digest`
+ band draws with a filled coverage outline on a channel with an entry; kill the server,
+ remove `editor/test-transcripts` and `editor/test-settings.json`, `git status` clean.
+
+## Out of scope
+
+- **The transcode band.** Filed "with step 1" in three docs only because both would need one
+ snapshot regeneration; step 1 needs none. It is its own snapshot-shape change — the writer
+ recording a transcode population (`appliesTo(config)` and the kept-media denominator beside
+ `buckets.untranscoded`) so `EXTERNAL_BAND_IDS` can go to three — and its own plan. Say so in
+ the IA doc's Transcode row.
+- Renaming `noDigest` view-model fields / props / the widget wire field. `digestWorkOf` →
+ a generic `operationWorkOf`. Slice 4 itself. Running normalize over the 16,156 `deferred`.
+ A snapshot schema version. Any regen tooling.
+
+## Handoff — the cadence from here
+
+On approval, Fable does not implement. It:
+
+1. Spawns one implementation agent — `subagent_type: "general-purpose"`, `model: "opus"` —
+ with: the path of this plan (`plans/unified-ops-step-1.md` after step 0, which the agent
+ itself commits), the repo conventions that matter (AGENTS.md is auto-loaded; e2e detached;
+ commit messages via `git commit -F <file under $CLAUDE_JOB_DIR/tmp>` because fish mangles
+ backticks; never boot against `transcripts/`; tmp files under `$CLAUDE_JOB_DIR/tmp`), and
+ the report contract: **commit shas with one line each; the exact verification outputs
+ (tsc per package, test pass/fail counts, e2e pass/fail per spec with any retry); every place
+ the implementation diverged from the plan and why; anything it left undone; and the
+ read-only corpus figures it measured.** Its final message is the review input, not the
+ user's summary.
+2. Reviews on return: `git log --oneline 389ccea..`, `git show --stat` per commit, reads the
+ hunks in `channelSnapshot.ts`, `digestWorkOf`, `buildBands.ts`, `sweepPreview.ts`, and the
+ test rewrites; re-runs the grep gates and `pnpm -C common test` + editor units itself
+ (not the e2e); checks the FACTS/STATE entries against the measurements. Anything wrong goes
+ back to the same agent by `SendMessage` with its context intact, not a fresh spawn.
+3. Reports to the user in the job-report shape, and records this cadence as a `feedback`
+ memory (Fable plans; an Opus general-purpose agent implements and reports; Fable reviews)
+ so the next iteration starts the same way without being told.
