commit e8b29c4bb75afe561ab4a3cfd23fc22418ffe933
parent 05a6e47378b245380e86a7afed8af8f2e9552289
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Sun, 9 Aug 2026 17:13:42 -0400
Write down the model A-C were aiming at
Phase D, designed only. The diagnosis in one line: backfillSweep.ts
says in its own header that it is a clone of digestSweep.ts, which is
what happens when the first implementation's shape is not reusable --
and the third would be written for the same reason.
The design is that neither half gets reimplemented. autoQueuePolicy is
already an HTB/SWRR tree with nesting, three modes, per-node maxWorkers
and persisted fairness; the backfill lane's whole configurability is
four scalars. So operations become SELECTABLE IN THE TREE. The cheap
path already exists: snapshot.backfill[kindId].ids is the reachable work
list, which is exactly what a policy leaf already consumes from
snapshot.buckets.
Also records the migration order, because every step moves live numbers
on a 78,000-video corpus and they must not be done at once -- step 1 is
the noDigest collapse Phase C deliberately skipped -- and the list of
properties that must survive it, each of which was paid for once
already: a zero limit is a hold not a stop, re-derive from disk every
pull, one job per channel, the `never` default on the dispatch switch,
distinct queue keys as the only concurrency mechanism, and the two
numbers that are never summed.
STATE.md rewritten for the session. settings.backfill.enabled restored
to true (verified byte-identical to the pre-session backup); it is
gitignored and was never committed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Diffstat:
2 files changed, 194 insertions(+), 1 deletion(-)
diff --git a/plans/STATE.md b/plans/STATE.md
@@ -3,7 +3,67 @@
The working memory for the local-AI derived-corpus work. Rewritten at the end of every
session, before context is cleared. See [`README.md`](README.md) for the protocol.
-**Last updated:** 2026-08-08 (latest) — **the OOM wall is gone, and it was removed rather
+**Last updated:** 2026-08-09 — **the media-derived jobs got one catalog, declared
+dependencies and one disk rule.** Same branch `feat/diarization-oom-wall`, three commits,
+all verified. Phase D is designed only, in
+[`unified-operations-model.md`](unified-operations-model.md).
+
+**A: THE UNATTENDED DOWNLOADER HAD NO DISK CHECK AT ALL** (`1f82296`). Every download a
+person starts by clicking has preflighted for a long time; `autoRunner` — the one path that
+dispatches for days unobserved — contained zero references to disk. The disk was at 98%
+(29 GB) when this started. There is now one shared `diskGate` with **hysteresis** (resume
+needs `minFreeDiskGB + resumeMarginGB`, default margin 2 GB, or the first resumed download
+drops back under the floor and the pipeline flaps) and a **reason string**. Three modes, and
+the split is the point: `enforce` reads+writes the latch (unattended loops), `observe` reads
+it (UI polls — so a dashboard cannot show green while the pipeline is held), `manual`
+ignores it (the operator is standing right there). Four more byte-writing holes closed:
+`downloadVideoPipelineAction` (whose sibling already called the `lowDiskError` defined in
+the same file), `persistKept` per-item rather than once, the truncated-audio re-fetch checked
+BEFORE it deletes the stub, and `backfillBatch` where `diskFloorHit` was set and then
+ignored. Derived sidecars stay ungated on purpose — kilobytes, and holding them costs days.
+The dashboard's red "downloads paused" could only ever mean the manual toggle; it is two
+instruments now. **The new auto-runner spec was checked against a disabled gate before being
+believed — it goes red with 2 picks dispatched.**
+
+**B: `blocked` IS NOT `missing-input`** (`67d2ff2`). `attribution-diarized` waits on
+diarization.json and reported that as `missing-input` — which means one thing to the rest of
+the system: *the media is gone, re-acquire it*. So ~73,000 videos sat in the re-acquire
+population, and with `allowRedownload` on the lane would have spent a download each fetching
+AUDIO, which can never satisfy that wait, then deleted it again. `blocked` follows the
+`deferred` precedent exactly: counted, never summed into `reachableBackfillWork`, never
+dispatched, immune to `force` — and, the actual fix, unaffected by `allowRedownload`.
+`dependsOn` is declared and `resolveBackfillKinds` topologically sorts by it, **stably**,
+because attribution-text must stay last (the better lane has to reach a diarized video first
+or the text lane spends ~30 model calls on a record it must not write). Two existing
+assertions pinned the old behaviour and now pin the fix.
+
+**C: DIGEST IS IN THE CATALOG, ON ITS OWN LANE** (`f9c15d5`). Nothing about digest generation
+is rewritten — the entry calls `resolveDigestTarget` / `isCuesJsonFresh` / `isSectionFresh` /
+`digestVideo`, including the shared-from-a-duplicate-cluster rule, because two definitions of
+"digested" is the exact failure it exists to prevent. Two things are now true that were not:
+the transcript dependency is **declared** (`dependsOn: ["transcription"]`, so a video with no
+transcript reports `blocked` and names what it waits for — PLAN.md:146 stated this in prose
+and nothing enforced it), and each operation **declares its lane**. The obvious
+"unification" — everything on `BACKFILL_QUEUE` — would be a REGRESSION: `registry.ts` runs
+each key at concurrency 1, so it would make the GPU digest lane wait on CPU diarization
+across a multi-week sweep. `laneBackfillKinds` filters on the queue key so `backfillBatch`
+can never dispatch digest and can never drop its guards. **`backfill.spec.ts` now fails if
+the two lanes ever serialize.** `resolveTarget` became async and channel-scoped (a digest's
+identity includes the hash of the channel context note), still resolved once per run.
+
+**DELIBERATELY NOT DONE, and it is step 1 of Phase D:** collapsing `noDigest` into
+`snapshot.backfill.digest`, and merging digest into the backfill indicators. Both change live
+coverage numbers on a 78,000-video corpus, and `blocked` additionally removes untranscribed
+videos from the digest count — real improvements, but not ones this branch could verify.
+
+**Coordination note (again):** `settings.backfill.enabled` was set to false while
+`backfillKinds.ts` / `backfillBatch.ts` were edited, and **restored to true afterwards**.
+A backfill sweep was running at the time (`sweeping: true`). Free disk moved from 29 GB to
+61 GB during the session — something outside this work freed ~32 GB.
+
+---
+
+**Previous session:** 2026-08-08 — **the OOM wall is gone, and it was removed rather
than worked around.** Branch `feat/diarization-oom-wall`, four parts, all verified.
**THE WALL: windowed diarization.** 6 of 10 videos over 6 hours were being killed by the
diff --git a/plans/unified-operations-model.md b/plans/unified-operations-model.md
@@ -0,0 +1,133 @@
+# The unified rule model for media-derived work
+
+**Status: designed, not built.** Phases A–C shipped on `feat/diarization-oom-wall`
+(`1f82296`, `67d2ff2`, `f9c15d5`). This file is the target they aim at, written down so the
+next person does not have to re-derive it, and so the shortcuts taken in A–C are legible as
+shortcuts rather than as decisions.
+
+## The problem, stated once
+
+There are **four schedulers** for what is really one kind of work:
+
+| scheduler | what it dispatches | how it is configured |
+| --- | --- | --- |
+| `autoRunner` (×2 kinds) | transcription, download | `autoQueuePolicy` — a real HTB/SWRR tree |
+| `backfillSweep` → `backfillBatch` | diarization, attribution ×2 | four scalars |
+| `digestSweep` → `digestBatch` | digest ×2 lanes | its own settings block |
+| `/scheduler` sync ticks | channel syncs | cron-ish |
+
+`backfillSweep.ts` states in its own header that it is a clone of `digestSweep.ts`. That is
+the whole diagnosis: the second implementation was written because the first one's shape was
+not reusable, and the third would be written for the same reason.
+
+## The one-sentence design
+
+**The policy tree is the rule engine; the operation registry is the catalog.**
+
+Neither half gets reimplemented. `autoQueuePolicy.ts` is already an HTB analog — arbitrary
+group nesting, `strict` / `round-robin` / `weighted-fair`, per-node `maxWorkers` with
+fall-through, leaf matching on channel / platform / bucket, and persisted SWRR fairness
+(`current-weight` per node id, nginx's algorithm). The backfill lane's entire configurability
+is four scalars. So convergence means **backfill and digest operations become selectable in
+the policy tree**, not that the tree gets rebuilt for them.
+
+The cheap path already exists. `channelSnapshot` writes
+
+```
+snapshot.backfill[kindId] = { missing, stale, missingInput, deferred, blocked, ids }
+```
+
+and `ids` is *already* the reachable work list — precisely what a policy leaf consumes today
+from `snapshot.buckets[name]`. A leaf gains an `operation` selector alongside `bucket`, and
+`buildPendingByLeaf` reads from `snapshot.backfill[op].ids` instead of
+`snapshot.buckets[bucket]`. That is a small change to one function.
+
+## What Phases A–C already put in place
+
+- **`BackfillLane { queueKey, contendsFor }`** on every registered operation
+ (`common/lib/backfillKinds.ts`). This is the arbiter's input: `queueKey` is the concurrency
+ domain, `contendsFor` is the scarce resource. `laneYieldsToTranscription()` is the first
+ rule derived from it, and `digestBatch` already consults it rather than re-testing the app
+ id.
+- **`dependsOn` + topological ordering** (`orderByDependencies`), stable so that
+ declaration-order decisions survive.
+- **`blocked`** — a state meaning "waiting on an operation this system produces", distinct
+ from `missing-input` ("the media is gone"). Counted, never reachable, never dispatched,
+ immune to `allowRedownload`.
+- **`operationCatalog()` / `EXTERNAL_OPERATIONS`** — transcription and download are
+ registered as descriptors so the dependency graph is complete.
+- **One disk gate** (`diskGate`, three modes) that every byte-writing path shares.
+
+## Backfill stops being a lane and becomes a posture
+
+This is the generalization of the "auto-backfill as a passive job that stays behind new
+updates" idea. Today it is hard-coded into one lane: `backfillLimit()` treats `weight: 0` as
+idle-only. Make it a property any node can carry:
+
+```ts
+type AutoQueueNode = {
+ // ...existing id / match / weight / maxWorkers / mode / children
+ operation?: string; // an id from operationCatalog()
+ posture?: "new-first" | "passive"; // passive == today's weight:0 idle-only yield
+ order?: "newest" | "oldest" | "cheapest" | "heaviest";
+};
+```
+
+- **`posture: "passive"`** makes today's idle-only behaviour available to *any* operation
+ rather than being welded into one runner. A passive node takes slots only when nothing
+ ahead of it is working.
+- **`order`** makes two hardcoded decisions expressible instead of buried: digest's
+ shortest-first (`cheapest`) and both sweeps' heaviest-channel-first (`heaviest`). Today
+ changing either means editing a controller.
+
+## The migration, in the order it should be done
+
+Each step is independently shippable and independently revertable. **Do not do them in one
+change** — every one of them moves live numbers on a 78,000-video corpus.
+
+1. **Collapse the counters.** `snapshot.backfill.digest` replaces `noDigest`. This is the
+ step Phase C deliberately skipped: the digest kind's `state()` already computes it
+ (including the shared-from-duplicate rule), so this is a rewiring, not new logic. Watch
+ for: `/api/widget/actionable` refuses to filter on `noDigest` because during the backfill
+ it is 99.87% of the corpus — the replacement must keep that refusal, and note that
+ `blocked` now removes untranscribed videos from the count, which will change the
+ published coverage percentage.
+2. **A leaf can name an operation.** `buildPendingByLeaf` reads `snapshot.backfill[op].ids`.
+ At this point the tree can *express* backfill work without anything dispatching it.
+3. **One arbiter per lane, not per subsystem.** A single runner reads the tree, groups
+ selected work by `lane.queueKey`, and dispatches one job per key. This is where
+ `digestSweep` and `backfillSweep` finally merge — they are the same loop with different
+ constants.
+4. **Move the guards onto declared lane rules.** The blocker is that `digestBatch.limit()`
+ carries guards no scalar share can express: `digestsPaused`, `yieldToTranscription` plus
+ its CPU-worker carve-out, `spendCapUsd` on the metered lane, the `remoteEnabled`
+ fail-fast, the engine `probe()` fail-fast, and duplicate-cluster sharing. Each needs a
+ home in the lane rule *before* step 3 can carry digest. `contendsFor` is the first of
+ these and already exists.
+5. **One pause model.** `settings.backfill.enabled`, `digest.digestsPaused`,
+ `transcriptionsPaused` and `downloadsPaused` become node state on the tree. Keep the
+ property all four already have and that makes them safe: a pause returns `limit() === 0`,
+ which `runPool` idle-waits on, so a hold is never a stop and never re-derives anything.
+6. **Retire `sweepEnabled` / `sweepKinds` / `sweepChannels` / `backfill.weight`** into the
+ tree, last, once nothing reads them.
+
+## What must not be lost
+
+Collected here because each one was paid for once already:
+
+- **Reachable and needs-media are never summed.** They differ by ~91× on this corpus.
+ `deferred` and `blocked` are never summed into either.
+- **A zero limit is a hold, not a stop.** `runPool` idle-waits; returning `null` from
+ `next()` ends the job. Every pause in the repo depends on this distinction.
+- **Re-derive eligibility from disk on every pull**, never from a frozen array with a cursor.
+ It is what makes a restart, a concurrent lane, and a video that became eligible mid-run all
+ simply invisible.
+- **One job per channel, never per video.** The registry keeps 100 records and the log 500;
+ 77,000 jobs evict the history of the run that made them.
+- **The `never` default on the dispatch switch.** `candidateAction` is the one branch that can
+ hand an engine something it should not have. Adding a state must stay a compile error.
+- **Distinct queue keys are the only concurrency mechanism.** `registry.ts` submits every
+ non-empty key at concurrency 1. Sharing a key between a GPU and a CPU operation makes them
+ take turns; `backfill.spec.ts` has a test that fails if digest and diarization ever do.
+- **Derived sidecars are not disk-gated.** They are kilobytes; holding them frees nothing and
+ costs days.
