commit 1724aca9ef6b4607aa6d750ec27c145170f2e4d4
parent 62a1de45c519ad207b8c56ecb3c4d9570d412174
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Mon, 7 Sep 2026 18:48:55 -0400
plans: one-core Phase 1 — four lanes, one runner
The slice-level plan for Phase 1. Three surveys on 2026-09-07 (dispatch
backend, editor surfaces, live config and job history) changed the umbrella's
wording in one way: the arbiter never ran — zero records among 1,305 production
jobs — and it cannot dispatch the two lanes that carry 100% of live dispatch.
So the auto-queue runner generalizes to four lanes, and the arbiter and both
sweeps are deleted rather than extended.
Five slices, each one reviewable diff that deletes something, with the numbers
checked offline against the live corpus before and after.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
1 file changed, 254 insertions(+), 0 deletions(-)
diff --git a/plans/one-core-phase-1.md b/plans/one-core-phase-1.md
@@ -0,0 +1,254 @@
+# One core — Phase 1: four lanes, one runner
+
+## Context
+
+`plans/one-core.md` (committed on `one-core/phase-0`, seven commits, unmerged, e2e 511/511)
+is the umbrella. This is the slice-level plan for **Phase 1 — Dispatch: one scheduler**:
+finish `plans/unified-operations-model.md` steps 5 and 6 and delete what they retire, so one
+dispatcher owns every lane of media-derived work.
+
+Three surveys on 2026-09-07 (dispatch backend, editor surfaces, live config and job history)
+changed the umbrella's Phase 1 wording in one way, agreed with the operator:
+
+- **The arbiter never ran.** Zero `operations-arbiter` records among 1,305 production jobs,
+ no `.arbiter/` state, no boot resume, no fairness memory, no owner map, and it cannot
+ dispatch download or transcription — the two lanes carrying 100% of live dispatch
+ (`controller/operationLane.ts:22-30` resolves registry entries only). The auto-queue
+ runner (`controller/autoRunner.ts`) is the proven loop: persisted picks, boot resume,
+ worker slots, platform backoff, an e2e-pinned console.
+- **So the runner generalizes, and the arbiter and both sweeps are deleted.** `AutoQueueKind`
+ widens from `{transcription, download}` to the four lanes `lib/pauseGates.ts:41` already
+ names: `+ digest`, `+ backfill`. Each lane gets a policy tree, a runner and the same
+ console. Download and transcription dispatch is untouched.
+- **Pause moves to `autoQueue[lane].held`**, behind the unchanged `isGateHeld` /
+ `withGateHeld` API, with the four legacy fields read as a migration for one release.
+
+**The model in one sentence:** the *lane* is the dispatch noun (a queue key, a tree, a
+runner, a pause, a console), the *operation* is the work noun (a registry entry with a
+state per video), and a leaf in a lane's tree names channels and, where a lane carries
+more than one operation, an operation.
+
+## Facts that shape the slices
+
+- Live `settings.json`: two trees (`autoQueue.transcription`, `.download`), both roots
+ `strict` with ten channel leaves + a catch-all; no leaf names an operation; both sweeps
+ disarmed; `transcriptionsPaused: true`; `backfill.allowRedownload: true` with
+ `reach: "corpus"` and empty `sweepChannels`. Arming the backfill lane as configured would
+ re-fetch audio for ~66,540 videos (`STATE.md` runbook item iii, unresolved).
+- `.auto-queue/state.json` (2026-09-02): 29 download picks, 26 from the catch-all leaf;
+ `currentWeights` empty on both lanes (strict mode never writes them).
+- Snapshot `backfill` map keys are identical on all 68 channels:
+ `attribution-diarized, attribution-text, diarization, digest`; `ids.length == missing +
+ stale` everywhere (pinned `channelSnapshot.test.ts:28`). Baseline sums: digest missing
+ 55,956 / stale 267 / deferred 15 / blocked 1,722; diarization missing 424 / missingInput
+ 75,591; attribution-text missing 77,888; attribution-diarized missing 1,881 / blocked 76,015.
+ Download's list is `snapshot.undownloadedIds` (top level, playlist order; sum 9) plus
+ `buckets.partialDownloads`; transcription's is `buckets.downloadedNoTranscript` (881) +
+ `failedListed` (+ opt-in `downloadedAutoSubsOnly`). **There is no `buckets.undownloaded`
+ or `buckets.untranscribed`** — the umbrella's step 5 named fields that do not exist.
+- `EXTERNAL_OPERATIONS` (`lib/operations.ts:1321-1377`) are descriptors only: no `state`,
+ `run`, `inputs`, `outputs`. The four registry `Operation`s have all of them.
+- `backfillBatch.ts` does not import `laneGuards`; it re-implements the yield inline and
+ keys the GPU idle-only rule on the *presence* of `laneFor` (`:796`, the FACTS.md trap).
+ `digestBatch.ts` uses the declared guards (`laneGuards.digestGate`, keyed on `contendsFor`).
+- Runner console contract: `RunnerOperationView.tsx:20-35` `<section data-lane={kind}>`,
+ ~1,200 e2e lines depend on it. No UI exists for `match.operation`
+ (`LadderRung.tsx:390-497` offers channel / platform / all).
+- e2e cannot run in the primary checkout while the untracked `editor/content` symlink
+ exists; run it from a worktree as Phase 0 did.
+
+## The slices
+
+Each is one reviewable diff that deletes something, committed with its measurements, on a
+branch off `one-core/phase-0` (or `main` once that merges). Live numbers are checked with
+offline `tsx` over the real corpus, never a second editor.
+
+### 1.1 — Four lanes in the model (no behaviour change)
+
+- `lib/autoQueueTypes.ts:138`: `AutoQueueKind = "transcription" | "download" | "digest" |
+ "backfill"`, exported as `LANES`; `lib/pauseGates.ts:41` `PauseLane` becomes an alias of
+ it. Grep the ~20 files spelling the two-member union (`jobs/autoQueuePolicy.ts:105-152`,
+ `editor/app/api/auto-queue/control/route.ts:17`, `jobs/active/buildActiveJobs.ts:493`).
+- `AutoQueueSettings` gains `digest` and `backfill` policies. `defaultAutoQueue()` /
+ `sanitizeAutoQueue()` (`jobs/autoQueuePolicy.ts:521,554`) default them `enabled: false`,
+ root `strict` with one `{type:"all"}` leaf. `AutoQueueOrder` gains `"cheapest"` (digest's
+ shortest-first, comparator supplied by the runner from `readDurationFast`, memoized like
+ recency keys); the digest lane defaults to it so today's ordering survives.
+- `bucketsForKind` / `optInBucketsForKind` / `selectableBucketsForKind` (`:104-121`)
+ return `[]` for the new lanes. New `operationsForLane(lane, settings)` in
+ `lib/operations.ts`: enabled catalog operations whose `(op.laneFor?.(settings) ??
+ op.lane).queueKey` is the lane's queue (`DIGEST_LOCAL_QUEUE | DIGEST_REMOTE_QUEUE` for
+ digest, `BACKFILL_QUEUE` for backfill), in `orderByDependencies` order — a leaf naming no
+ operation draws all of them, the way a bucket-less leaf draws the default union.
+- `retainLeaves(pending, root, "buckets" | "operations")` (`:225`) loses its mode: a
+ leaf is retained when it draws anything from the lane's projection.
+- `jobs/autoQueueState.ts`: state keyed by all four lanes; missing lanes coerce to empty
+ (extend `autoQueueState.test.ts`).
+- Editor: `LadderRung.tsx` `LeafControls` — an *operation* select, shown only for lanes
+ with more than one operation (backfill); the bucket select hidden for lanes with none.
+ `/api/auto-queue/status` `buildKind` (`operations/status.ts:76-108`) iterates `LANES`.
+- `startAutoRunnersIfEnabled` still starts only lanes that have an executor (the two).
+- Update `plans/one-core.md` Phase 1 to this design in the same commit.
+- Tests: sanitizer defaults, `operationsForLane` per lane and per `digest.remoteEnabled`,
+ `pauseGates.test.ts` `pauseLaneFor` unchanged. **Numbers: none move.**
+
+### 1.2 — The runner runs operations (digest and backfill lanes live; sweeps untouched)
+
+- New `controller/operationBatch.ts` — the per-unit executor and the per-lane limit,
+ extracted from `digestBatch.ts:152-606` and `backfillBatch.ts:255-868`:
+ - `runOperationUnit({op, videoId, channel, settings, signal, force})`: re-derive from
+ disk (`readVideoFiles` + `op.state()`), `candidateAction` with its `never` default
+ (`backfillBatch.ts:192-238`, generalized; digest's inline `allFresh` becomes the
+ `fresh` arm), `op.run()`, the per-video overrides (`reacquireMediaFor` + `cleanup()` in
+ `finally` for backfill kinds, `laneSharesDuplicates` canonical share for digest),
+ `applyResult` on the remote-unit path.
+ - `laneLimit(lane, settings, live)`: digest = `digestPreflight` + `digestGate` +
+ concurrency + `remoteTerm` (`digestBatch.ts:514-559`); backfill = share + `remoteTerm`
+ + `unitTerm` (`backfillBatch.ts:727-845`) with the GPU yield through
+ `laneGuards.laneYieldsToTranscription(laneForOperation(op))` — keyed on `contendsFor`.
+ The `laneFor`-presence check at `:796` is deleted; the comment at
+ `operations.ts:438-445` and the FACTS.md entry are rewritten to say so.
+ - `countOperationWork(lane, channel)` replaces the `countMissingDigests` /
+ `countBackfillWork` twins (snapshot first, walk fallback for un-snapshotted channels).
+- `controller/autoRunner.ts`: `run` (`:823`) branches on lane — transcription and download
+ unchanged; digest and backfill call `runOperationUnit`. `limit` (`:594-616`) calls
+ `laneLimit` for the new lanes. `next` (`:621`) draws `ch.operations` for them;
+ `buildChannelWork` (`:210`) already projects `snap.backfill[op].ids` at `:249-261` — stop
+ zeroing those leaves. Runner job progress uses the existing `digests` / `backfills`
+ metrics. **The runner for lane X refuses to start while sweep X is armed, and says which**
+ (`arbiterBlockedReason` logic, `arbiter.ts:194-203`, moved and renamed
+ `laneBlockedReason`).
+- `runDigestChannelJob` / `runBackfillChannelJob` (the channel and video pages' manual Run)
+ keep their job kinds and call `operationBatch` inside their own pool, so there is one
+ executor. `digestBatch.ts` and `backfillBatch.ts` are deleted; `backfillBatch.test.ts`
+ becomes `operationBatch.test.ts` (`candidateAction` exhaustiveness, `laneLimit` cases).
+- Editor: `/operations/[id]` (`page.tsx:161`) chooses the runner console off the operation's
+ lane, not `op.runner` (which stays `undefined` for sync, per the IA doc); every operation
+ page with a lane renders `RunnerOperationView` with `data-lane` = the lane id, beside the
+ sweep section for the length of this slice. `useOperationsStatus` polls all four.
+- e2e: new `lane-runner.spec.ts` on the fixture corpus — the digest lane starts, dispatches
+ one unit through the Ollama stub, appears as a job, holds on pause, stops, and comes back
+ after `startAutoRunnersIfEnabled`; the backfill lane dispatches diarization then
+ attribution-diarized in dependency order through `fake-diarize`; digest and diarization
+ run concurrently on distinct queue keys (the `backfill.spec.ts:615` invariant, moved).
+- **Operator gate A, at the end of this slice:** enable `autoQueue.digest` in production
+ with one channel leaf, let the runner drive a pass, restart the editor, confirm it
+ resumes. Record units dispatched, the snapshot's `digest.missing` delta and, if a
+ transcription is started beside it, the observed yield. This is the "has run in
+ production once" precondition every retirement below stands on.
+
+### 1.3 — Sweeps and arbiter retire
+
+- Arming becomes tree authoring: `startDigestSweepAction(channels)` and
+ `startBackfillSweepAction(kinds, channels)` (`editor/app/jobs/actions.ts:160-251`) become
+ one `armLaneAction(lane, {operations, channels})` that writes `autoQueue[lane].root`
+ (a strict group of channel leaves, each carrying `operation` when scoped) and sets
+ `enabled: true`; disarm is `enabled: false`. `NextUp` is the plan (`computeLeafPending`);
+ the "also runs on …" line (`SweepScope.tsx:122-155`, via `workerMatches`) moves into
+ `LaneHeader`.
+- Settings migration on read in `getSettings`: `digest.sweepEnabled/sweepChannels/
+ recencyOrder/recencyReach` and `backfill.sweepEnabled/sweepKinds/sweepChannels/order/
+ reach/weight` populate `autoQueue.digest` / `autoQueue.backfill` when those blocks are
+ absent (`sweepEnabled:false` → `enabled:false`; channels → leaves; order/reach → policy
+ order/reach; `weight` is retired — the backfill lane yields on `contendsFor` like digest,
+ and `maxWorkers` is its share). Then the ten fields, their sanitizers, the
+ `LaneSettingsForm.tsx:80-86` weight field and `settingsActions.ts:215` are deleted. The
+ migration must never *enable* a lane the sweep flag did not.
+- Delete, backend: `controller/digestSweep.ts`, `backfillSweep.ts`, `sweepPreview.ts`,
+ `sweepRecency.ts`, `lib/sweepPlan.ts`, `arbiter.ts`, `arbiterWork.ts`; job kinds
+ `digest-sweep`, `backfill-sweep`, `operations-arbiter` (`jobs/jobKinds.ts` +
+ `jobKinds.test.ts`); `instrumentation.ts:109-126` resume hooks; `sweepPreview.test.ts`,
+ `arbiter.test.ts` (its leaf-order case already lives in `autoQueuePolicy.test.ts:729-806`).
+ `controller/digestPlan.buildDigestSweepPlan` stays (chunk-priced ETA for `bin/digest-plan`).
+- Delete, editor: `SweepLane.tsx`, `SweepScope.tsx`, `SweepPlan.tsx`,
+ `OperationDetail.tsx:117-123,157-223`, `ArbiterBar.tsx`, `OperationsBoard.tsx:52`,
+ `operations/lanes.ts` sweep half and `ArbiterStatus` (`:74-81,229-263,315-370`),
+ `railStates.ts:48-92`, `operations/actions.ts:264-330` (`saveLaneOrderAction`, arbiter
+ start/stop), `laneState.ts` `feedRunning` axis + `sweepLaneNote`, `MonitorWidget`
+ `BackfillStrip` `sweeping`, `buildActiveJobs.ts:436-450` and
+ `api/widget/sync/route.ts:297-304` sweep flags, `stageStatus.ts:126` job-kind map entry,
+ `[id]/page.tsx:53-63,190-215` `JOB_KINDS_BY_LANE`; `/api/test/resume-backfill-sweep` →
+ `/api/test/resume-lane?lane=`.
+- e2e: `backfill.spec.ts` seven sweep tests rewritten to the lane runner (arm = enable with
+ scope, persisted as a tree; restart via `resume-lane`); `auto-queue.spec.ts` three
+ arbiter tests deleted; `widget.spec.ts:776,787` and `laneState.test.ts` to runner state;
+ `operation-settings.spec.ts:104,214` to the surviving `LaneSettingsForm` fields.
+- The retired routes rule: every deleted action's URL and aria-label either survives on
+ the runner console or is asserted gone in the spec that pinned it.
+
+### 1.4 — Pause is lane state
+
+- `AutoQueuePolicy.held?: boolean`. `isGateHeld(settings, lane)` reads
+ `settings.autoQueue[lane].held`, falling back to the legacy field
+ (`transcriptionsPaused`, `downloadsPaused`, `digest.digestsPaused`, `!backfill.enabled`)
+ when undefined; `withGateHeld` writes the new key only; `getSettings` copies legacy → new
+ on read so the next write persists it. All 13 callers are unchanged. The transcription
+ asymmetry stays: the node value is intent, `workerPool.isPaused()` is live state
+ (`instrumentation.ts:71-76` re-applies from the new key).
+- After one release (or immediately once the live `settings.json` carries all four `held`
+ keys): delete the four legacy fields, their sanitizers, `settings/actions.ts:222-223`'s
+ spread, and rewrite `LaneSettingsForm.tsx:68-73` to write `held` through `withGateHeld`
+ (still a deliberate second writer of the same key).
+- Fixtures and specs that read the legacy fields from disk: `auto-queue.spec.ts:1199-1206`,
+ `dashboard.spec.ts:79`, `backfill.spec.ts:796,835`, `test-settings.default.json`.
+ `pauseGates.test.ts`'s polarity test becomes "held is a hold, never a stop" at the runner:
+ `limit()` returns 0 and `runPool` idle-waits.
+
+### 1.5 — One work list per lane in the snapshot
+
+- `generateChannelSnapshot` (`channelSnapshot.ts:524`) writes `backfill.download` and
+ `backfill.transcription` entries: `ids` = the union `defaultBucketsForPolicy` draws
+ (`undownloadedIds ∪ partialDownloads`; `downloadedNoTranscript ∪ failedListed`),
+ `missing = ids.length`, `missingInput = noTranscript` for transcription, `eligible` as
+ the others. `EXTERNAL_OPERATIONS` carry their declared `lane`, so `laneForOperation`
+ resolves it (`laneForOperation.test.ts:80` flips from "still has no lane" to "resolves
+ its declared lane"); `EXTERNAL_BAND_IDS`'s special case in `buildBands.ts` goes. The
+ `ChannelSnapshot.backfill` type is aliased `operations`; the disk key is unchanged.
+- `autoRunner.buildChannelWork` reads `snap.backfill[op].ids` for every lane's default
+ draw; `bucket` leaves keep working for sub-lists (retry buckets are a real UI).
+- Proof: a unit test on the fixture corpora that `next()` picks the same first N units
+ under the bucket projection and the operation projection; an offline `tsx` script over
+ the 68 live snapshots asserting the in-memory fold of the new entries equals the
+ bucket union (no snapshot is regenerated — that writes into `transcripts/`).
+
+## Operator gates
+
+- **A — before 1.3 lands:** the digest lane runner has driven a production pass and
+ resumed after a restart (end of 1.2).
+- **B — before the backfill lane is ever enabled in production:** decide
+ `backfill.allowRedownload` and the channel scope. As configured today it re-fetches audio
+ for ~66,540 `missingInput` videos, channel-major, with no inter-download sleep. The
+ 1.3 migration keeps the lane disabled; the tree's channel leaves are the scope from then on.
+
+## What must not be lost
+
+`unified-operations-model.md` "What must not be lost" — reachable ≠ needs-media (never
+summed, ~91× apart); a zero limit is a hold, never a stop (`runPool` idle-waits, `null` from
+`next()` ends the job); re-derive eligibility from disk on every pull; one job per channel
+for the manual verbs, and the runner's in-process units for the lanes; the `never` default on
+`candidateAction`; distinct queue keys are the only concurrency mechanism (`backfill.spec.ts:615`
+moves, does not vanish); derived sidecars are not disk-gated. Plus: `laneFor` presence stops
+being load-bearing in 1.2 and the test that pinned it changes in the same commit; the runner
+console's `<section data-lane>` contract is the surface every lane now shares; the
+`transcriptionsPaused` intent-vs-pool asymmetry; `undownloadedIds` is playlist order and
+must stay unsorted.
+
+## Verification
+
+Per slice: `tsc --noEmit` in common, editor, export, mcp, homepage, umtool;
+`pnpm --filter yt-dlp-transcript-common test` (architecture test included — 1.3 removes
+`lib/sweepPlan.ts → controller` and `jobs/*→ controller` entries from the allow-list rather
+than adding any); full editor e2e from a worktree behind the queue lock, detached, watched
+with Monitor.
+
+Numbers do not move: an offline `tsx` script over the 68 live snapshots and `settings.json`
+dumps, before and after each slice, every `backfill[op]` count, `undownloadedIds.length`,
+the per-lane `computeLeafPending` totals, and each lane's `held`/`enabled`; the diff is
+empty or explained in the slice note. After 1.3, the same script asserts the migrated
+`autoQueue.digest` / `.backfill` blocks are `enabled: false` with the expected leaves.
+
+Run the thing: gate A's production pass (units, sidecars, restart resume); after 1.3 an
+arm-from-the-console on the fixture corpus produces the same tree the migration would; after
+1.4 a pause from the dashboard, the widget and the runner page all flip one `held` key and
+the runner idles without stopping; after 1.5 `/channels` bands for download and
+transcription draw from the operation entries with no rendered number changed.
