# The unified rule model for media-derived work

**Status: partly built — steps 1, 2, 3 and 4 are DONE; 5 is HALF done; 6 is open.** Phases A–C shipped
on `feat/diarization-oom-wall` (`1f82296`, `67d2ff2`, `f9c15d5`). This file is the target they
aim at, written down so the next person does not have to re-derive it, and so the shortcuts
taken in A–C are legible as shortcuts rather than as decisions. The step list below is the
current record; each DONE step says what actually shipped.

## The problem, stated once

There are **four schedulers** for what is really one kind of work:

| scheduler | what it dispatches | how it is configured |
| --- | --- | --- |
| `autoRunner` (×2 kinds) | transcription, download | `autoQueuePolicy` — a real HTB/SWRR tree |
| `backfillSweep` → `backfillBatch` | diarization, attribution ×2 | four scalars |
| `digestSweep` → `digestBatch` | digest ×2 lanes | its own settings block |
| `/scheduler` sync ticks | channel syncs | cron-ish |

`backfillSweep.ts` states in its own header that it is a clone of `digestSweep.ts`. That is
the whole diagnosis: the second implementation was written because the first one's shape was
not reusable, and the third would be written for the same reason.

## The one-sentence design

**The policy tree is the rule engine; the operation registry is the catalog.**

Neither half gets reimplemented. `autoQueuePolicy.ts` is already an HTB analog — arbitrary
group nesting, `strict` / `round-robin` / `weighted-fair`, per-node `maxWorkers` with
fall-through, leaf matching on channel / platform / bucket, and persisted SWRR fairness
(`current-weight` per node id, nginx's algorithm). The backfill lane's entire configurability
is four scalars. So convergence means **backfill and digest operations become selectable in
the policy tree**, not that the tree gets rebuilt for them.

The cheap path already exists. `channelSnapshot` writes

```
snapshot.backfill[kindId] = { missing, stale, missingInput, deferred, blocked, ids }
```

and `ids` is *already* the reachable work list — precisely what a policy leaf consumes today
from `snapshot.buckets[name]`. A leaf gains an `operation` selector alongside `bucket`, and
`buildPendingByLeaf` reads from `snapshot.backfill[op].ids` instead of
`snapshot.buckets[bucket]`. That is a small change to one function.

## What Phases A–C already put in place

- **`BackfillLane { queueKey, contendsFor }`** (now `Lane`) on every registered operation
  (`common/lib/backfillKinds.ts`, now `operations.ts`). This is the arbiter's input: `queueKey` is the concurrency
  domain, `contendsFor` is the scarce resource. `laneYieldsToTranscription()` is the first
  rule derived from it, and `digestBatch` already consults it rather than re-testing the app
  id.
- **`dependsOn` + topological ordering** (`orderByDependencies`), stable so that
  declaration-order decisions survive.
- **`blocked`** — a state meaning "waiting on an operation this system produces", distinct
  from `missing-input` ("the media is gone"). Counted, never reachable, never dispatched,
  immune to `allowRedownload`.
- **`operationCatalog()` / `EXTERNAL_OPERATIONS`** — transcription and download are
  registered as descriptors so the dependency graph is complete.
- **One disk gate** (`diskGate`, three modes) that every byte-writing path shares.

## Backfill stops being a lane and becomes a posture

This is the generalization of the "auto-backfill as a passive job that stays behind new
updates" idea. Today it is hard-coded into one lane: `backfillLimit()` treats `weight: 0` as
idle-only. Make it a property any node can carry:

```ts
type AutoQueueNode = {
  // ...existing id / match / weight / maxWorkers / mode / children
  operation?: string;                 // an id from operationCatalog()
  posture?: "new-first" | "passive";  // passive == today's weight:0 idle-only yield
  order?: "newest" | "oldest" | "cheapest" | "heaviest";
};
```

- **`posture: "passive"`** makes today's idle-only behaviour available to *any* operation
  rather than being welded into one runner. A passive node takes slots only when nothing
  ahead of it is working.
- **`order`** makes two hardcoded decisions expressible instead of buried: digest's
  shortest-first (`cheapest`) and both sweeps' heaviest-channel-first (`heaviest`). Today
  changing either means editing a controller.

## The migration, in the order it should be done

Each step is independently shippable and independently revertable. **Do not do them in one
change** — every one of them moves live numbers on a 78,000-video corpus.

1. **Collapse the counters.** — **DONE 2026-08-26** (`efb0cf9`, `00b1c8a`; plan
   [`unified-ops-step-1.md`](unified-ops-step-1.md)). `snapshot.backfill.digest` is the only
   digest work list: `buckets.noDigest` is no longer written, typed, defaulted or read, and
   `DigestWork.source` and the four `id === DIGEST_OPERATION_ID` fallbacks
   (`buildBands.ts`, `sweepPreview.ts` ×2, `sweepRecency.ts`) went with it. It was a
   **deletion, not a rewiring**: the fallback's migration had already completed on disk —
   68 of 68 snapshots carried `backfill.digest` when this was measured — so the bucket branch
   was reading nothing and **no rendered number moved**. The coverage percentage this step
   warned would change had already changed, silently, between 08-11 and 08-26; the two
   counters were 11,777 videos apart (bucket 59,159 vs entry 47,382), the delta being
   `deferred` on 65 of 68 channels. `/api/widget/actionable`'s refusal to filter on the count
   survives, unchanged: it filters on `undownloaded || untranscribed` only.
2. **A leaf can name an operation.** — **DONE.** `AutoQueueMatch.operation` beside `bucket`,
   drawing from a separate `ChannelWork.operations` id space, with the claim key widened to
   `${operation}\0${id}` so digest and diarization cannot steal each other's work. The tree
   can *express* operation work; nothing dispatches it yet.
3. **One arbiter per lane, not per subsystem.** — **DONE** (`common/controller/arbiter.ts`).
   One long-lived job on `queueKey: ""` — it WAITS on jobs needing the real keys, so holding
   one would deadlock against its own work. It reads both policy trees, plans one unit per
   (operation, channel) in leaf-priority order, and dispatches ONE UNIT PER LANE per pass,
   concurrently, reserving by `lane.queueKey` and never inventing one. It starts the
   per-channel jobs that already exist, so a pass is inspectable with the existing tools.
   It REFUSES to start beside an armed sweep and says which one — two dispatchers on one
   lane would start the same channel twice. It is deliberately NOT persisted across a
   restart; that comes with step 6. `digestSweep` and `backfillSweep` are untouched and
   still own dispatch until then.
4. **Move the guards onto declared lane rules.** — **DONE.** `common/controller/laneGuards.ts`
   holds all six, in the two shapes a dispatcher can act on: a **preflight** answered before
   any pool exists (`remoteEnabled`, the engine `probe()` — both still throw, because a
   configuration problem will not resolve itself and parking on it would idle forever while
   looking busy) and a **gate** answered per pull (`digestsPaused`, the yield with its
   CPU-worker carve-out, `spendCapUsd`), which returns a HOLD and never a stop.
   `laneSharesDuplicates()` covers cluster sharing. `digestBatch` calls all of them, so
   behaviour is unchanged and there is one definition rather than two. 8 unit tests; the
   yield branch stays covered end to end because it reads live pool state.
5. **One pause model — HALF DONE** (editor IA slice 7, `c924f73` → `685cba3`). What shipped
   is one DEFINITION and one WRITER, not one storage location: `common/lib/pauseGates.ts`'s
   `isGateHeld` / `withGateHeld` are the only place the four fields' polarity is known
   (`backfill.enabled` is inverted), one `pauseLaneAction` / `resumeLaneAction` pair replaces
   eight actions, and one control draws every lane's gate — including, for the first time, on
   the runner pages. The property that makes all four safe is unchanged and now stated in one
   header: a pause returns `limit() === 0`, which `runPool` idle-waits on, so a hold is never
   a stop and never re-derives anything.

   What is NOT done: the four settings fields are still four settings fields, not node state
   on the tree. That migration is step 6's, and it is now cheap — every reader and every
   writer among the controls goes through the two functions above, so the storage can move
   behind them without touching a surface. The `LaneSettingsForm` checkbox stays a second
   writer of `backfill.enabled` on purpose: one field, two places to set it, and they cannot
   drift.
6. **Retire `sweepEnabled` / `sweepKinds` / `sweepChannels` / `backfill.weight`** into the
   tree, last, once nothing reads them.

## What must not be lost

Collected here because each one was paid for once already:

- **Reachable and needs-media are never summed.** They differ by ~91× on this corpus.
  `deferred` and `blocked` are never summed into either.
- **A zero limit is a hold, not a stop.** `runPool` idle-waits; returning `null` from
  `next()` ends the job. Every pause in the repo depends on this distinction.
- **Re-derive eligibility from disk on every pull**, never from a frozen array with a cursor.
  It is what makes a restart, a concurrent lane, and a video that became eligible mid-run all
  simply invisible.
- **One job per channel, never per video.** The registry keeps 100 records and the log 500;
  77,000 jobs evict the history of the run that made them.
- **The `never` default on the dispatch switch.** `candidateAction` is the one branch that can
  hand an engine something it should not have. Adding a state must stay a compile error.
- **Distinct queue keys are the only concurrency mechanism.** `registry.ts` submits every
  non-empty key at concurrency 1. Sharing a key between a GPU and a CPU operation makes them
  take turns; `backfill.spec.ts` has a test that fails if digest and diarization ever do.
- **Derived sidecars are not disk-gated.** They are kilobytes; holding them frees nothing and
  costs days.