commit ac2e1e09a4c95e9466c5cdfc71fad1fec11e85bf
parent 30a5ac502d0f68b127873fae6f23c4b59c7ca8bd
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Mon, 7 Sep 2026 19:03:45 -0400
common: four lanes in the model, and one of them stops being a filter
Slice 1.1 of plans/one-core-phase-1.md. `AutoQueueKind` widens from the two
runner kinds to the four lanes `lib/pauseGates.ts` already named, exported as
`LANES`; `PauseLane` becomes an alias of it, so the two id spaces cannot drift.
The digest and backfill lanes get a policy, a tree, a status entry and a state
block — and no executor. Nothing dispatches off them, no live number moves.
`retainLeaves` is DELETED, and that is the shape of the change rather than a
tidy-up. It existed because one tree could hold leaves belonging to two
dispatchers, so each had to zero the other's after the draw — a second pass that
had to be remembered at three call sites, and handing a digest candidate to
whisper if it was not. Now the lane IS the dispatcher and the draw is
lane-scoped: `bucketsForKind` returns [] for the operation lanes, the new
`operationsForLane(lane, settings)` says what a lane may draw from
`snapshot.backfill[op].ids` (off the queue key, and off `laneFor` before `lane`,
so `digest.remoteEnabled` does not empty the digest lane), and `buildChannelWork`
projects only the intersection. A leaf asking for anything else finds no list
and comes back empty by construction.
The two new lanes default `enabled: false`, root `strict` with one `{type:"all"}`
leaf — a runner lane's absent root still means the empty tree it has always
meant. `AutoQueueOrder` gains "cheapest" and the digest lane defaults to it; the
comparator arrives with the runner in 1.2, and until then the order select
offers the three date-keyed orders it always has.
Editor: the status payload is keyed by LANES, the ladder's leaf gains an
operation select (drawn only for a lane carrying more than one) and hides the
bucket select for a lane carrying none. `startAutoRunnersIfEnabled` still names
the two lanes that have an executor.
Numbers: none move. `plans/tools/phase1-numbers.ts` over the live 68 snapshots
and settings.json diffs empty before and after.
common 888 tests, mcp 205, `next build` clean, tsc clean in six packages, editor e2e
511 passed / 0 failed / 24.4 min from a worktree of this commit.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
22 files changed, 742 insertions(+), 211 deletions(-)
diff --git a/common/controller/arbiter.ts b/common/controller/arbiter.ts
@@ -41,7 +41,6 @@ import { drainStream } from "../jobs/drainStream";
import {
buildPendingByLeaf,
flattenLeaves,
- retainLeaves,
type AutoQueueGroup,
type ChannelWork,
} from "../jobs/autoQueuePolicy";
@@ -129,11 +128,12 @@ export function planArbiterUnits(
// only, and a bucket leaf with no explicit bucket would otherwise claim ids
// out of a union it is about to be filtered out of anyway. Claiming still
// dedups correctly because the claim key carries the operation.
- const pending = retainLeaves(
- buildPendingByLeaf(root, channels, []),
- root,
- "operations",
- );
+ //
+ // No second filtering pass: buildArbiterChannelWork projects `buckets: {}`,
+ // so a bucket leaf finds no list here whatever it names, and the loop below
+ // skips every leaf with no `match.operation` anyway. retainLeaves was doing
+ // both of those a third time.
+ const pending = buildPendingByLeaf(root, channels, []);
for (const leaf of flattenLeaves(root)) {
const operation = leaf.match.operation;
if (!operation) continue;
diff --git a/common/controller/autoRunner.ts b/common/controller/autoRunner.ts
@@ -22,11 +22,11 @@ import {
type WorkPick,
buildPendingByLeaf,
flattenLeaves,
- retainLeaves,
selectNextWork,
defaultBucketsForPolicy,
selectableBucketsForKind,
} from "../jobs/autoQueuePolicy";
+import { operationsForLane } from "../lib/operations";
import {
type RecencyKey,
buildRecencyKeys,
@@ -274,6 +274,29 @@ export function operationsNamedBy(root: AutoQueueGroup): string[] {
return [...out];
}
+// WHAT THIS LANE MAY PROJECT: the operations its tree names, INTERSECTED with
+// the operations the lane actually dispatches.
+//
+// The intersection is the safety rail retainLeaves used to be. A leaf naming
+// `digest` inside the auto-transcribe tree is legal to write and always has
+// been; before the lanes existed it was projected and then zeroed by a second
+// pass, and forgetting that pass handed a digest candidate to whisper. Now it
+// is never projected at all, so the leaf draws nothing and the wrong engine
+// cannot be reached from the wrong tree.
+//
+// It stays an INTERSECTION rather than "everything the lane dispatches" because
+// projection costs real memory: one channel's `digest.ids` is up to 11,329
+// strings, and this runs on a three-second status poll. Slice 1.2's `next()`
+// widens it to the lane's default draw for the lanes that have an executor.
+function laneOperationIds(kind: AutoQueueKind, root: AutoQueueGroup): string[] {
+ const named = operationsNamedBy(root);
+ if (named.length === 0) return [];
+ const dispatched = new Set(
+ operationsForLane(kind, getSettings()).map((op) => op.id),
+ );
+ return named.filter((id) => dispatched.has(id));
+}
+
// The policy's `order`, defaulted for settings files written before the field
// existed. One place, so the runner and the status page can never disagree about
// what an absent field means.
@@ -380,7 +403,7 @@ export async function computeLeafPending(
paths,
kind,
meta,
- operationsNamedBy(policy.root),
+ laneOperationIds(kind, policy.root),
);
const { compare, keys } = await recencyOrdering(
kind,
@@ -389,19 +412,15 @@ export async function computeLeafPending(
meta,
channels,
);
- // BUCKET LEAVES ONLY. A leaf naming an operation belongs to the arbiter, and
- // counting its work here would report a digest backlog as a transcription
- // one; DISPATCHING it would hand a digest candidate to whisper. See
- // retainLeaves.
- const pending = retainLeaves(
- buildPendingByLeaf(
- policy.root,
- channels,
- defaultBucketsForPolicy(kind, policy),
- compare ? { compare } : undefined,
- ),
+ // THE LANE'S OWN PROJECTION IS THE FILTER. `channels` above carries only the
+ // buckets and operations THIS lane dispatches (see laneOperationIds), so a
+ // leaf naming anything else finds no list and comes back empty — the zero
+ // retainLeaves used to apply afterwards, reached by construction.
+ const pending = buildPendingByLeaf(
policy.root,
- "buckets",
+ channels,
+ defaultBucketsForPolicy(kind, policy),
+ compare ? { compare } : undefined,
);
// A video already in flight is not "next up" — drop the live runner's set
// before asking the policy, exactly as next() does.
@@ -699,7 +718,7 @@ async function runLoop(
paths,
kind,
metaCache,
- operationsNamedBy(policy.root),
+ laneOperationIds(kind, policy.root),
);
// Re-derived each iteration from the freshly-read policy, like `enabled`, so
// toggling the replace-auto-captions lane (or the ordering) takes effect
@@ -712,18 +731,14 @@ async function runLoop(
metaCache,
channels,
);
- // BUCKET LEAVES ONLY — see the note in computeLeafPending. This is the
- // dispatch path, so it is the one where getting it wrong runs the wrong
+ // THE LANE'S OWN PROJECTION IS THE FILTER — see computeLeafPending. This is
+ // the dispatch path, so it is the one where getting it wrong runs the wrong
// engine on the wrong video.
- const pending = retainLeaves(
- buildPendingByLeaf(
- policy.root,
- channels,
- defaultBucketsForPolicy(kind, policy),
- compare ? { compare } : undefined,
- ),
+ const pending = buildPendingByLeaf(
policy.root,
- "buckets",
+ channels,
+ defaultBucketsForPolicy(kind, policy),
+ compare ? { compare } : undefined,
);
const exclude = new Set<string>([...live.inFlight.keys(), ...completed]);
removeIds(pending, exclude);
@@ -1168,6 +1183,13 @@ export async function startAutoRunner(
// Start every enabled runner. Called from the editor instrumentation hook at
// server startup.
+//
+// NAMED, NOT ITERATED OVER LANES, and that is the whole safety story of slice
+// 1.1. The digest and backfill lanes exist in the model — a policy, a tree, a
+// gate, a status entry — and have no executor: `run()` below knows how to
+// transcribe and how to download and nothing else. Iterating LANES here would
+// start two runners that would pick a video and then not know what to do with
+// it. They join this list in slice 1.2, when the executor does.
export async function startAutoRunnersIfEnabled(
paths: Paths = getPaths(),
): Promise<void> {
diff --git a/common/jobs/autoQueuePolicy.test.ts b/common/jobs/autoQueuePolicy.test.ts
@@ -2,17 +2,21 @@ import { test } from "node:test";
import assert from "node:assert/strict";
import {
type AutoQueueGroup,
+ type AutoQueueLeaf,
type AutoQueueRuntime,
type ChannelWork,
buildPendingByLeaf,
bucketsForKind,
defaultAutoQueue,
defaultBucketsForPolicy,
+ isGroup,
+ optInBucketsForKind,
selectableBucketsForKind,
emptyAutoQueueRuntime,
flattenLeaves,
policyDrawsBucket,
sanitizeAutoQueue,
+ sanitizeAutoQueueOrder,
selectNextWork,
} from "./autoQueuePolicy";
@@ -523,6 +527,81 @@ test("a leaf can target an opt-in bucket without the runner-wide switch", () =>
assert.deepEqual(pending.rest, ["c1", "h1"]);
});
+// --- the four lanes ---------------------------------------------------------
+
+test("sanitizeAutoQueue: every lane is keyed, even ones no file names", () => {
+ // Every settings.json in existence carries exactly two lanes.
+ const s = sanitizeAutoQueue({
+ transcription: { enabled: true },
+ download: { enabled: true },
+ });
+ assert.deepEqual(Object.keys(s).sort(), [
+ "backfill",
+ "digest",
+ "download",
+ "transcription",
+ ]);
+});
+
+test("sanitizeAutoQueue: the new lanes default off, strict, one catch-all leaf", () => {
+ for (const lane of ["digest", "backfill"] as const) {
+ const p = sanitizeAutoQueue({})[lane];
+ assert.equal(p.enabled, false, `${lane} must default OFF`);
+ assert.equal(p.root.mode, "strict");
+ assert.equal(p.root.children.length, 1);
+ const leaf = p.root.children[0];
+ assert.ok(!isGroup(leaf));
+ assert.deepEqual((leaf as AutoQueueLeaf).match, { type: "all" });
+ }
+});
+
+test("sanitizeAutoQueue: the runner lanes keep their EMPTY default root", () => {
+ // A tree that dispatches nothing is what an absent runner policy has always
+ // meant, and the new lanes' catch-all leaf must not leak into it.
+ for (const lane of ["transcription", "download"] as const) {
+ assert.equal(sanitizeAutoQueue({})[lane].root.children.length, 0);
+ assert.equal(defaultAutoQueue()[lane].root.children.length, 0);
+ }
+});
+
+test("sanitizeAutoQueue: the digest lane defaults to cheapest, the rest to listed", () => {
+ const s = sanitizeAutoQueue({});
+ assert.equal(s.digest.order, "cheapest");
+ assert.equal(s.backfill.order, "listed");
+ assert.equal(s.transcription.order, "listed");
+ assert.equal(s.download.order, "listed");
+ // An explicit stored order still wins over the lane default.
+ assert.equal(sanitizeAutoQueue({ digest: { order: "newest" } }).digest.order, "newest");
+ // And a garbage one falls back to "listed", not to the lane default —
+ // sanitizeAutoQueueOrder is the one coercion and it does not know lanes.
+ assert.equal(sanitizeAutoQueue({ digest: { order: "soonest" } }).digest.order, "listed");
+});
+
+test("sanitizeAutoQueueOrder accepts cheapest", () => {
+ assert.equal(sanitizeAutoQueueOrder("cheapest"), "cheapest");
+ assert.equal(sanitizeAutoQueueOrder("cheap"), "listed");
+});
+
+test("sanitizeAutoQueue: a stored root on a new lane replaces the default leaf", () => {
+ const s = sanitizeAutoQueue({
+ digest: { enabled: true, root: { mode: "strict", children: [] } },
+ });
+ assert.equal(s.digest.enabled, true);
+ assert.equal(s.digest.root.children.length, 0);
+});
+
+test("bucketsForKind: the operation lanes have no buckets", () => {
+ for (const lane of ["digest", "backfill"] as const) {
+ assert.deepEqual([...bucketsForKind(lane)], []);
+ assert.deepEqual([...optInBucketsForKind(lane)], []);
+ assert.deepEqual([...selectableBucketsForKind(lane)], []);
+ assert.deepEqual(
+ [...defaultBucketsForPolicy(lane, { replaceAutoSubs: true })],
+ [],
+ );
+ }
+});
+
test("sanitizeAutoQueue defaults replaceAutoSubs to false", () => {
assert.equal(defaultAutoQueue().transcription.replaceAutoSubs, false);
assert.equal(sanitizeAutoQueue({}).download.replaceAutoSubs, false);
diff --git a/common/jobs/autoQueuePolicy.ts b/common/jobs/autoQueuePolicy.ts
@@ -5,6 +5,7 @@ import type { Platform } from "../lib/platform";
// here so every existing `from "../jobs/autoQueuePolicy"` import still resolves.
import type {
AutoQueueGroup,
+ AutoQueueKind,
AutoQueueLeaf,
AutoQueueMatch,
AutoQueueMatchType,
@@ -15,9 +16,10 @@ import type {
AutoQueueReach,
AutoQueueSettings,
} from "../lib/autoQueueTypes";
-import { isGroup } from "../lib/autoQueueTypes";
+import { LANES, isGroup } from "../lib/autoQueueTypes";
export type {
AutoQueueGroup,
+ AutoQueueKind,
AutoQueueLeaf,
AutoQueueMatch,
AutoQueueMatchType,
@@ -28,7 +30,7 @@ export type {
AutoQueueReach,
AutoQueueSettings,
};
-export { isGroup };
+export { LANES, isGroup };
// Pure, side-effect-free policy engine for the automatic priority queue. It
@@ -56,6 +58,7 @@ export const AUTO_QUEUE_ORDERS: ReadonlyArray<AutoQueueOrder> = [
"listed",
"newest",
"oldest",
+ "cheapest",
];
@@ -75,7 +78,9 @@ export function sanitizeAutoQueueReach(value: unknown): AutoQueueReach {
// backfill.order) and three copies of this line would eventually disagree about
// what an absent field means.
export function sanitizeAutoQueueOrder(value: unknown): AutoQueueOrder {
- return value === "newest" || value === "oldest" ? value : "listed";
+ return value === "newest" || value === "oldest" || value === "cheapest"
+ ? value
+ : "listed";
}
@@ -99,27 +104,33 @@ export const DOWNLOAD_BUCKETS = ["partialDownloads", "undownloadedIds"] as const
export const TRANSCRIBE_OPT_IN_BUCKETS = ["downloadedAutoSubsOnly"] as const;
export const DOWNLOAD_OPT_IN_BUCKETS = ["autoSubsOnly"] as const;
-// Local kind type — do NOT import AutoQueueKind from autoQueueState.ts, which
-// already imports from this module (the reverse edge would be a cycle).
-export function bucketsForKind(
- kind: "transcription" | "download",
-): readonly string[] {
- return kind === "transcription" ? TRANSCRIBE_BUCKETS : DOWNLOAD_BUCKETS;
+// NO BUCKETS IS A REAL ANSWER, not a gap. The digest and backfill lanes draw
+// from operations, never from snapshot buckets — `snapshot.backfill[op].ids` is
+// their work list — so an empty array here is what says "this lane has no
+// bucket picker and no bucket projection", and every caller already handles an
+// empty list the way it handles a leaf that matches nothing.
+//
+// (The type is AutoQueueKind now that it lives in lib/autoQueueTypes.ts, which
+// this module already imports. The old note here warned against importing it
+// from autoQueueState.ts, which imports FROM this module — that cycle is gone
+// with the type.)
+export function bucketsForKind(kind: AutoQueueKind): readonly string[] {
+ if (kind === "transcription") return TRANSCRIBE_BUCKETS;
+ if (kind === "download") return DOWNLOAD_BUCKETS;
+ return [];
}
-export function optInBucketsForKind(
- kind: "transcription" | "download",
-): readonly string[] {
- return kind === "transcription"
- ? TRANSCRIBE_OPT_IN_BUCKETS
- : DOWNLOAD_OPT_IN_BUCKETS;
+export function optInBucketsForKind(kind: AutoQueueKind): readonly string[] {
+ if (kind === "transcription") return TRANSCRIBE_OPT_IN_BUCKETS;
+ if (kind === "download") return DOWNLOAD_OPT_IN_BUCKETS;
+ return [];
}
// Everything a leaf may be pointed at for this kind: the default union plus the
// opt-in buckets. Backs the editor's bucket dropdown AND the runner's snapshot
// projection — a leaf can only find ids in a bucket the runner projected.
export function selectableBucketsForKind(
- kind: "transcription" | "download",
+ kind: AutoQueueKind,
): readonly string[] {
return [...bucketsForKind(kind), ...optInBucketsForKind(kind)];
}
@@ -128,7 +139,7 @@ export function selectableBucketsForKind(
// bucketsForKind unless the policy opted into auto-caption replacement, which
// appends the opt-in buckets at the tail so real work always drains first.
export function defaultBucketsForPolicy(
- kind: "transcription" | "download",
+ kind: AutoQueueKind,
policy: Pick<AutoQueuePolicy, "replaceAutoSubs">,
): readonly string[] {
return policy.replaceAutoSubs
@@ -149,7 +160,7 @@ export function defaultBucketsForPolicy(
// Pure, like everything else here: the caller supplies the channel's slug and
// platform, never a snapshot.
export function policyDrawsBucket(
- kind: "transcription" | "download",
+ kind: AutoQueueKind,
policy: Pick<AutoQueuePolicy, "replaceAutoSubs" | "root">,
channel: Pick<ChannelWork, "slug" | "platform">,
bucket: string,
@@ -207,37 +218,20 @@ export type ChannelWork = {
operations?: Record<string, string[]>;
};
-// Split a pending map by WHO DISPATCHES IT.
+// `retainLeaves(pending, root, "buckets" | "operations")` USED TO LIVE HERE, and
+// its deletion is the point of the four-lane model rather than a tidy-up.
//
-// A leaf naming an operation is claimed by the arbiter; a leaf naming a bucket
-// (or nothing) is claimed by the runner whose tree it sits in. They can share a
-// tree — an operator may reasonably want "diarize cornbreadman before you
-// transcribe anything else" expressed in one priority list — and without this
-// they SHARE A DISPATCHER, which means the auto-transcribe runner picks a
-// digest candidate off an operation leaf and hands it to whisper.
+// It existed because one tree could hold leaves belonging to two dispatchers —
+// a bucket leaf the runner owned and an operation leaf the arbiter owned — and
+// each had to zero the other's before selecting, or auto-transcribe would hand
+// a digest candidate to whisper. That was a filter applied AFTER the draw.
//
-// So each dispatcher zeroes the leaves that are not its own before selecting.
-// Zeroed rather than removed: selectNextWork walks the TREE, and a leaf with an
-// empty list is skipped by the same code path that skips a leaf with no work,
-// so nothing new has to understand this. A leaf missing from the map entirely
-// would instead read as `undefined` at `pending[node.id]?.length`, which is the
-// same answer by luck rather than by construction.
-export function retainLeaves(
- pending: Record<string, string[]>,
- root: AutoQueueNode,
- want: "buckets" | "operations",
-): Record<string, string[]> {
- const out: Record<string, string[]> = {};
- const keep = new Set(
- flattenLeaves(root)
- .filter((l) => (want === "operations") === Boolean(l.match.operation))
- .map((l) => l.id),
- );
- for (const [id, ids] of Object.entries(pending)) {
- out[id] = keep.has(id) ? ids : [];
- }
- return out;
-}
+// Now the LANE is the dispatcher, and the draw itself is lane-scoped: a lane
+// projects only the buckets bucketsForKind gives it and only the operations
+// operationsForLane gives it (see controller/autoRunner.ts buildChannelWork).
+// A leaf asking for anything else finds no list to draw from and comes back
+// empty on its own — the same zero, reached by construction instead of by a
+// second pass that had to be remembered at three call sites.
// Pre-order (priority-order) flatten of every leaf in the tree.
export function flattenLeaves(node: AutoQueueNode): AutoQueueLeaf[] {
@@ -507,22 +501,60 @@ function sanitizeRoot(value: unknown, seen: Set<string>): AutoQueueGroup {
return { id: node.id, mode: "strict", weight: 1, maxWorkers: null, children: [] };
}
-export function defaultAutoQueuePolicy(): AutoQueuePolicy {
+function emptyRoot(): AutoQueueGroup {
+ return { id: "root", mode: "strict", weight: 1, maxWorkers: null, children: [] };
+}
+
+// THE DEFAULT TREE FOR A LANE, and the two answers are different on purpose.
+//
+// The runner lanes default to an EMPTY root: they have shipped that way since
+// the auto-queue existed, an empty tree dispatches nothing, and a settings file
+// that omits a root must keep meaning exactly that.
+//
+// The digest and backfill lanes default to one catch-all leaf, because their
+// work list is an operation's `ids` and a lane with no leaf at all could never
+// draw it. The leaf is inert while `enabled` is false — which is how they
+// default, and what keeps gate B (never enable the backfill lane against
+// ~66,540 missingInput videos by accident) a decision an operator still has to
+// take.
+function defaultRootFor(lane: AutoQueueKind): AutoQueueGroup {
+ if (lane === "transcription" || lane === "download") return emptyRoot();
+ return {
+ id: "root",
+ mode: "strict",
+ weight: 1,
+ maxWorkers: null,
+ children: [{ id: "all", match: { type: "all" }, weight: 1, maxWorkers: null }],
+ };
+}
+
+// The digest lane's historical ordering is SHORTEST-FIRST, and it is not
+// cosmetic: a 12-minute video is one chunk and a four-hour stream is thirty, so
+// draining the cheap end first is what makes a multi-week sweep show progress.
+// Defaulting the lane to "cheapest" is how that survives the move from the
+// sweep to the tree. The comparator arrives with the runner (slice 1.2); until
+// then next() has none for this order and falls back to today's.
+function defaultOrderFor(lane: AutoQueueKind): AutoQueueOrder {
+ return lane === "digest" ? "cheapest" : "listed";
+}
+
+export function defaultAutoQueuePolicy(
+ lane: AutoQueueKind = "transcription",
+): AutoQueuePolicy {
return {
enabled: false,
maxWorkers: null,
replaceAutoSubs: false,
- order: "listed",
+ order: defaultOrderFor(lane),
snoozeUntil: null,
- root: { id: "root", mode: "strict", weight: 1, maxWorkers: null, children: [] },
+ root: defaultRootFor(lane),
};
}
export function defaultAutoQueue(): AutoQueueSettings {
- return {
- transcription: defaultAutoQueuePolicy(),
- download: defaultAutoQueuePolicy(),
- };
+ return Object.fromEntries(
+ LANES.map((lane) => [lane, defaultAutoQueuePolicy(lane)]),
+ ) as AutoQueueSettings;
}
// A snooze that has already lapsed is not a snooze: normalizing it to null here
@@ -535,7 +567,16 @@ function sanitizeSnooze(value: unknown): number | null {
return at > Date.now() ? at : null;
}
-function sanitizePolicy(value: unknown): AutoQueuePolicy {
+// A LANE THAT IS NOT IN THE FILE IS THE LANE'S DEFAULT, not an empty object.
+//
+// Every settings.json in existence carries exactly two lanes, so the digest and
+// backfill blocks arrive `undefined` on every read until something writes them.
+// Coercing that to `sanitizePolicy({})` would give them an empty root — a lane
+// that can never draw anything even once an operator enables it — so the whole
+// default policy is the fallback, and only the fields the file actually names
+// override it.
+function sanitizePolicy(value: unknown, lane: AutoQueueKind): AutoQueuePolicy {
+ if (value == null) return defaultAutoQueuePolicy(lane);
const r = (value ?? {}) as Record<string, unknown>;
const seen = new Set<string>();
return {
@@ -544,18 +585,23 @@ function sanitizePolicy(value: unknown): AutoQueuePolicy {
// Opt-in only: anything but an explicit `true` (including a missing field on
// a pre-existing settings.json) leaves the lane off.
replaceAutoSubs: r.replaceAutoSubs === true,
- // Anything unrecognised (including a missing field) means today's behaviour.
- order: sanitizeAutoQueueOrder(r.order),
+ // Anything unrecognised (including a missing field) means the lane's own
+ // default — "listed" for the runner lanes, "cheapest" for digest.
+ order: r.order === undefined
+ ? defaultOrderFor(lane)
+ : sanitizeAutoQueueOrder(r.order),
snoozeUntil: sanitizeSnooze(r.snoozeUntil),
- root: sanitizeRoot(r.root, seen),
+ root:
+ r.root === undefined
+ ? defaultRootFor(lane)
+ : sanitizeRoot(r.root, seen),
};
}
export function sanitizeAutoQueue(value: unknown): AutoQueueSettings {
if (!value || typeof value !== "object") return defaultAutoQueue();
const r = value as Record<string, unknown>;
- return {
- transcription: sanitizePolicy(r.transcription),
- download: sanitizePolicy(r.download),
- };
+ return Object.fromEntries(
+ LANES.map((lane) => [lane, sanitizePolicy(r[lane], lane)]),
+ ) as AutoQueueSettings;
}
diff --git a/common/jobs/autoQueueState.test.ts b/common/jobs/autoQueueState.test.ts
@@ -69,3 +69,60 @@ test("corrupt platformBackoff field coerces to empty without throwing", async ()
assert.deepEqual(back.download.platformBackoff, {});
});
});
+
+test("every lane is keyed, and a state file written before a lane existed coerces to empty", async () => {
+ await withPaths(async (paths) => {
+ // Exactly what `.auto-queue/state.json` holds in production today: the two
+ // runner lanes and nothing else.
+ // Write a valid state first so the .auto-queue dir exists.
+ await writeAutoQueueState(paths, emptyAutoQueueState());
+ await writeFile(
+ paths.autoQueueStateFile,
+ JSON.stringify({
+ transcription: {
+ runtime: { currentWeights: { "node-1": 3 } },
+ picks: [],
+ platformBackoff: {},
+ },
+ download: { runtime: { currentWeights: {} }, picks: [], platformBackoff: {} },
+ }),
+ );
+
+ const back = await readAutoQueueState(paths);
+ assert.deepEqual(Object.keys(back).sort(), [
+ "backfill",
+ "digest",
+ "download",
+ "transcription",
+ ]);
+ // The lanes it does not name are EMPTY, not undefined: the status poll
+ // reads state[lane].picks for every lane the console draws.
+ assert.deepEqual(back.digest, {
+ runtime: { currentWeights: {} },
+ picks: [],
+ platformBackoff: {},
+ });
+ assert.deepEqual(back.backfill.picks, []);
+ // And the lanes it does name are untouched.
+ assert.deepEqual(back.transcription.runtime.currentWeights, { "node-1": 3 });
+ });
+});
+
+test("a lane missing from an in-memory state object still writes", async () => {
+ await withPaths(async (paths) => {
+ // Defensive, like every coercion in this module: a hand-built state (or one
+ // deserialized from an older build) must not make the persist throw and
+ // lose the fairness memory of the lanes that ARE there.
+ const partial = { transcription: emptyAutoQueueState().transcription } as ReturnType<
+ typeof emptyAutoQueueState
+ >;
+ await writeAutoQueueState(paths, partial);
+ const back = await readAutoQueueState(paths);
+ assert.deepEqual(Object.keys(back).sort(), [
+ "backfill",
+ "digest",
+ "download",
+ "transcription",
+ ]);
+ });
+});
diff --git a/common/jobs/autoQueueState.ts b/common/jobs/autoQueueState.ts
@@ -23,7 +23,7 @@ import {
// Defined in the model layer (lib/ may not import jobs/); re-exported here so
// every existing `from "./autoQueueState"` import still resolves.
export type { AutoQueueKind } from "../lib/autoQueueTypes";
-import type { AutoQueueKind } from "../lib/autoQueueTypes";
+import { LANES, type AutoQueueKind } from "../lib/autoQueueTypes";
// One recorded grant, newest-first, for the status panel.
export type AutoQueuePick = {
@@ -43,10 +43,12 @@ export type AutoQueueKindState = {
platformBackoff: PlatformBackoffState;
};
-export type AutoQueueState = {
- transcription: AutoQueueKindState;
- download: AutoQueueKindState;
-};
+// KEYED BY EVERY LANE, including the two with no executor yet. A lane whose
+// block is missing from the file coerces to empty state rather than to
+// `undefined`: `state[kind].picks` is read on the status poll for every lane the
+// console draws, and a lane added to LANES must not be able to make that throw
+// against a state file written before it existed.
+export type AutoQueueState = Record<AutoQueueKind, AutoQueueKindState>;
export const AUTO_QUEUE_PICK_LOG_LIMIT = 50;
@@ -55,10 +57,9 @@ export function emptyAutoQueueKindState(): AutoQueueKindState {
}
export function emptyAutoQueueState(): AutoQueueState {
- return {
- transcription: emptyAutoQueueKindState(),
- download: emptyAutoQueueKindState(),
- };
+ return Object.fromEntries(
+ LANES.map((lane) => [lane, emptyAutoQueueKindState()]),
+ ) as AutoQueueState;
}
function coerceRuntime(value: unknown): AutoQueueRuntime {
@@ -123,10 +124,9 @@ export async function readAutoQueueState(paths: Paths): Promise<AutoQueueState>
}
if (!raw || typeof raw !== "object") return emptyAutoQueueState();
const r = raw as Record<string, unknown>;
- return {
- transcription: coerceKindState(r.transcription),
- download: coerceKindState(r.download),
- };
+ return Object.fromEntries(
+ LANES.map((lane) => [lane, coerceKindState(r[lane])]),
+ ) as AutoQueueState;
}
// Write the state atomically (tmp + rename), creating the .auto-queue dir on
@@ -140,10 +140,9 @@ export async function writeAutoQueueState(
picks: k.picks.slice(0, AUTO_QUEUE_PICK_LOG_LIMIT),
platformBackoff: k.platformBackoff ?? {},
});
- const out: AutoQueueState = {
- transcription: trim(state.transcription),
- download: trim(state.download),
- };
+ const out = Object.fromEntries(
+ LANES.map((lane) => [lane, trim(state[lane] ?? emptyAutoQueueKindState())]),
+ ) as AutoQueueState;
await mkdir(path.dirname(paths.autoQueueStateFile), { recursive: true });
const tmp = `${paths.autoQueueStateFile}.tmp-${process.pid}`;
await writeFile(tmp, JSON.stringify(out, null, 2) + "\n");
diff --git a/common/lib/autoQueueTypes.ts b/common/lib/autoQueueTypes.ts
@@ -25,7 +25,13 @@ export type AutoQueueMatchType = "channel" | "platform" | "all";
//
// This deliberately does NOT reorder RULES: the tree is what expresses
// priority. A newest-first archive is one catch-all rule with order "newest".
-export type AutoQueueOrder = "listed" | "newest" | "oldest";
+//
+// "cheapest" is the digest lane's historical shortest-first: the comparator is
+// duration-keyed rather than date-keyed, and like the recency ones it is
+// SUPPLIED BY THE RUNNER (the engine never reads a duration off disk). A lane
+// whose runner supplies no comparator for it falls back to today's order —
+// which is what makes it safe to name here before anything computes it.
+export type AutoQueueOrder = "listed" | "newest" | "oldest" | "cheapest";
// How far an order REACHES.
//
@@ -125,14 +131,31 @@ export type AutoQueuePolicy = {
root: AutoQueueGroup;
};
-export type AutoQueueSettings = {
- transcription: AutoQueuePolicy;
- download: AutoQueuePolicy;
-};
+export type AutoQueueSettings = Record<AutoQueueKind, AutoQueuePolicy>;
+
+// --- Lanes ------------------------------------------------------------------
-// --- Runner kinds -----------------------------------------------------------
+// THE FOUR LANES. A lane is the DISPATCH noun: a queue key, a policy tree, a
+// runner, a pause gate and a console. The OPERATION is the work noun — a
+// registry entry with a state per video — and a leaf in a lane's tree names
+// channels and, where a lane carries more than one operation, an operation.
+//
+// It widened from the two runner kinds to the four names lib/pauseGates.ts
+// already listed. The two id spaces were always the same space; they are now
+// one type, and PauseLane is an alias.
+//
+// Here rather than in jobs/autoQueueState.ts (which persists their fairness
+// memory) for the same reason as the tree above: pauseGates.ts and
+// operations.ts name a lane without running one.
+//
+// ORDER IS LOAD-BEARING ONLY AS A DEFAULT: the two runner lanes come first so
+// anything that iterates LANES for display keeps the order the console had
+// before there were four of them.
+export const LANES = [
+ "transcription",
+ "download",
+ "digest",
+ "backfill",
+] as const;
-// The two auto-queue runners. Here rather than in jobs/autoQueueState.ts (which
-// persists their fairness memory) for the same reason as the tree above:
-// pauseGates.ts and operations.ts name a kind without running one.
-export type AutoQueueKind = "transcription" | "download";
+export type AutoQueueKind = (typeof LANES)[number];
diff --git a/common/lib/operations.test.ts b/common/lib/operations.test.ts
@@ -10,6 +10,7 @@ import {
resolveBackfillLaneOperations,
orderByDependencies,
operationCatalog,
+ operationsForLane,
operationCostBasis,
operationGroup,
operationLabel,
@@ -37,7 +38,11 @@ import {
import { CUES_FILE_VERSION } from "../controller/normalizeTranscript";
import { attributeOneVideo } from "../controller/attributeOne";
import { DIGEST_FILENAME, OLLAMA_DIGEST_APP_ID } from "./digest";
-import { BACKFILL_QUEUE, DIGEST_LOCAL_QUEUE } from "./queueKeys";
+import {
+ BACKFILL_QUEUE,
+ DIGEST_LOCAL_QUEUE,
+ DIGEST_REMOTE_QUEUE,
+} from "./queueKeys";
import {
ATTRIBUTION_FILENAME,
ATTRIBUTION_PROMPT_VERSION,
@@ -1585,3 +1590,60 @@ test("presentOperationWork says UNKNOWN rather than zero on an old snapshot", ()
7,
);
});
+
+// --- operationsForLane ------------------------------------------------------
+//
+// The registry's half of the four-lane model: which operations a LANE draws.
+// Off the QUEUE KEY and off laneFor() before lane, because digest's key follows
+// the engine it is configured with.
+
+function settingsWithBackfillLane(): SiteSettings {
+ const d = settingsWithDiarization();
+ const a = settingsWithAttribution();
+ return { ...d, attribution: a.attribution };
+}
+
+test("operationsForLane: the runner lanes draw no operations yet", () => {
+ // Not an opinion about the future: download and transcription are external
+ // entries with no state(), so there is no snapshot.backfill[op].ids for a
+ // leaf to draw. Slice 1.5 writes those entries.
+ const s = settingsWithBackfillLane();
+ assert.deepEqual(operationsForLane("transcription", s), []);
+ assert.deepEqual(operationsForLane("download", s), []);
+});
+
+test("operationsForLane: the backfill lane is its three kinds, in dependency order", () => {
+ const s = settingsWithBackfillLane();
+ assert.deepEqual(
+ operationsForLane("backfill", s).map((o) => o.id),
+ ["diarization", "attribution-diarized", "attribution-text"],
+ );
+ // And it is EMPTY when the features are off — a lane with nothing enabled
+ // draws nothing rather than drawing a disabled kind's backlog.
+ assert.deepEqual(operationsForLane("backfill", defaultSiteSettings()), []);
+});
+
+test("operationsForLane: digest stays on the digest lane on BOTH its queues", () => {
+ // THE laneFor CASE. `digest.remoteEnabled` moves the operation from
+ // DIGEST_LOCAL_QUEUE to DIGEST_REMOTE_QUEUE. Both are the digest lane, so the
+ // answer must not change — a queue-key map naming only the local key would
+ // empty the lane the moment the metered engine was switched on.
+ const local = defaultSiteSettings();
+ assert.equal(digestKind.laneFor!(local).queueKey, DIGEST_LOCAL_QUEUE);
+ assert.deepEqual(
+ operationsForLane("digest", local).map((o) => o.id),
+ ["digest"],
+ );
+
+ const remote: SiteSettings = {
+ ...local,
+ digest: { ...local.digest, remoteEnabled: true },
+ };
+ assert.equal(digestKind.laneFor!(remote).queueKey, DIGEST_REMOTE_QUEUE);
+ assert.deepEqual(
+ operationsForLane("digest", remote).map((o) => o.id),
+ ["digest"],
+ );
+ // And the backfill lane never picks it up on either engine.
+ assert.deepEqual(operationsForLane("backfill", remote).map((o) => o.id), []);
+});
diff --git a/common/lib/operations.ts b/common/lib/operations.ts
@@ -1640,6 +1640,43 @@ export function backfillLaneEntriesOf<T>(
return backfillLaneOperationEntriesOf(backfill).map(([, entry]) => entry);
}
+// THE OPERATIONS A LANE DISPATCHES, in dependency order.
+//
+// The other half of the four-lane model: `bucketsForKind` says what a lane's
+// leaves may draw from the snapshot's buckets, and this says what they may draw
+// from its operation entries. A leaf naming no operation draws ALL of them, the
+// way a bucket-less leaf draws the default bucket union.
+//
+// OFF THE QUEUE KEY, and off `laneFor` before `lane`, for the same reason
+// pauseGates.ts asks `runner` first: the queue key is what actually serializes
+// the work, and digest's key depends on the engine it is configured with. With
+// `digest.remoteEnabled` on, the digest operation moves from DIGEST_LOCAL_QUEUE
+// to DIGEST_REMOTE_QUEUE — both of which are the digest LANE, which is why both
+// are named here rather than one.
+//
+// The two runner lanes get [] today. They are not opinions about the future:
+// download and transcription are EXTERNAL_OPERATIONS with no `state`, so there
+// is no `snapshot.backfill[op].ids` for a leaf to draw. Slice 1.5 writes those
+// entries, and this function starts answering for them then with no caller
+// changing.
+export function operationsForLane(
+ lane: AutoQueueKind,
+ settings: SiteSettings,
+): Operation[] {
+ const keys =
+ lane === "digest"
+ ? [DIGEST_LOCAL_QUEUE, DIGEST_REMOTE_QUEUE]
+ : lane === "backfill"
+ ? [BACKFILL_QUEUE]
+ : [];
+ if (keys.length === 0) return [];
+ return orderByDependencies(
+ allOperations(settings).filter((op) =>
+ keys.includes((op.laneFor?.(settings) ?? op.lane).queueKey),
+ ),
+ );
+}
+
// Resolve a caller-supplied list of kind ids against the registry. An empty or
// absent list means "every enabled lane kind" — the sweep's scope default.
// Unknown ids are dropped rather than throwing: a settings file may name a kind
diff --git a/common/lib/pauseGates.ts b/common/lib/pauseGates.ts
@@ -38,12 +38,12 @@ import {
// The other three lanes have no live counterpart: their flag IS the gate, read
// at dispatch, which is why they need no boot hook.
-export type PauseLane = "transcription" | "download" | "digest" | "backfill";
-
-// The union of the two id spaces that already exist: AutoQueueKind (the runners)
-// and the editor's SweepLaneId (the sweeps). Asserted here for the first;
-// operations/lanes.ts asserts the second, where SweepLaneId lives.
-type _RunnerLanesArePauseLanes = AutoQueueKind extends PauseLane ? true : never;
+// A PAUSE LANE IS A LANE. This was a hand-written union of four names while
+// AutoQueueKind was a hand-written union of two; the four lanes in the model
+// closed that gap, so this is now an alias and the two id spaces cannot drift.
+// The name survives because thirteen call sites read as "which lane's gate",
+// and because a gate is what this file is about.
+export type PauseLane = AutoQueueKind;
// WHICH LANE'S GATE HOLDS THIS OPERATION — the answer to "the operator is on
// /operations/diarization and wants to hold it".
diff --git a/editor/app/operations/[id]/page.tsx b/editor/app/operations/[id]/page.tsx
@@ -4,9 +4,11 @@ import { getPaths } from "yt-dlp-transcript-common/lib/paths";
import { listChannelConfigs } from "yt-dlp-transcript-common/controller/channels";
import { PLATFORM_VALUES } from "yt-dlp-transcript-common/lib/platform";
import { selectableBucketsForKind } from "yt-dlp-transcript-common/jobs/autoQueuePolicy";
+import { LANES } from "yt-dlp-transcript-common/lib/autoQueueTypes";
import {
operationCatalog,
operationLabel,
+ operationsForLane,
type OperationSettingsBlock,
} from "yt-dlp-transcript-common/lib/operations";
import { getSettings, type SiteSettings } from "yt-dlp-transcript-common/lib/settings";
@@ -37,14 +39,18 @@ import { getActionableSummary } from "../../lib/actionable/loadActionable";
export const dynamic = "force-dynamic";
-// Buckets each runner can draw from — derived from the policy engine's single
+// Buckets each lane can draw from — derived from the policy engine's single
// source of truth (selectableBucketsForKind) so the picker can't drift from the
// runner. Includes the opt-in auto-caption buckets: a leaf that names one gets
// per-channel opt-in without flipping the runner-wide switch.
-const BUCKETS_BY_KIND = {
- transcription: [...selectableBucketsForKind("transcription")],
- download: [...selectableBucketsForKind("download")],
-};
+//
+// EMPTY IS A REAL ENTRY. The digest and backfill lanes draw from operations,
+// never from buckets, so their lists are `[]` and LeafControls draws no bucket
+// select at all — which is the honest UI for a lane where a bucket name would
+// mean nothing.
+const BUCKETS_BY_KIND = Object.fromEntries(
+ LANES.map((lane) => [lane, [...selectableBucketsForKind(lane)]]),
+) as Record<AutoQueueKind, string[]>;
// The job kinds that ARE this operation, for the running-jobs list. Only the
// sweep-fed operations need one: a runner lane lists its own units in flight,
@@ -195,6 +201,20 @@ export default async function OperationPage({
// their pages beside the pause and sweep they also share. The digest lane has
// no equivalent — its lane facts (`sweepEnabled`, order and reach) live in the
// digest block and the OrderReach control.
+ // THE OPERATIONS EACH LANE DISPATCHES, as ids, for the ladder's operation
+ // select. Server-side because it is a settings-dependent registry read
+ // (`digest.remoteEnabled` moves the digest operation between two queue keys
+ // that are both the digest lane), and the ladder is a client component.
+ //
+ // A lane with 0 or 1 operations draws no select: there is nothing to choose,
+ // and an operation leaf and a bucket-less leaf claim the same work.
+ const operationsByLane = Object.fromEntries(
+ LANES.map((lane) => [
+ lane,
+ operationsForLane(lane, settings).map((o) => o.id),
+ ]),
+ ) as Record<AutoQueueKind, string[]>;
+
const laneSettings =
laneId === "backfill" ? (
<LaneSettingsForm key="backfill-lane" initial={settings.backfill} />
@@ -258,6 +278,7 @@ export default async function OperationPage({
channels={channelOptions}
platforms={[...PLATFORM_VALUES]}
bucketsByKind={BUCKETS_BY_KIND}
+ operationsByLane={operationsByLane}
activeJobs={activeJobs}
runnerKind={runnerKind}
laneId={laneId}
diff --git a/editor/app/operations/components/ClaimLadder.tsx b/editor/app/operations/components/ClaimLadder.tsx
@@ -15,6 +15,7 @@ export function ClaimLadder({
channels,
platforms,
buckets,
+ operations,
data,
ops,
}: {
@@ -22,6 +23,7 @@ export function ClaimLadder({
channels: Channel[];
platforms: string[];
buckets: string[];
+ operations: string[];
data: RungData;
ops: RungOps;
}) {
@@ -40,6 +42,7 @@ export function ClaimLadder({
channels={channels}
platforms={platforms}
buckets={buckets}
+ operations={operations}
data={data}
ops={ops}
/>
diff --git a/editor/app/operations/components/LadderRung.tsx b/editor/app/operations/components/LadderRung.tsx
@@ -71,7 +71,13 @@ export function LadderRung(props: {
siblingCount: number;
channels: Channel[];
platforms: string[];
+ // What THIS LANE projects, and both lists can legitimately be empty. A lane
+ // with no buckets (digest, backfill) draws no bucket select; a lane with no
+ // operations, or exactly one, draws no operation select — with one there is
+ // nothing to choose, and naming it claims exactly what a bucket-less leaf
+ // already claims.
buckets: string[];
+ operations: string[];
data: RungData;
ops: RungOps;
}) {
@@ -393,10 +399,11 @@ function LeafControls(props: {
channels: Channel[];
platforms: string[];
buckets: string[];
+ operations: string[];
data: RungData;
ops: RungOps;
}) {
- const { leaf, channels, platforms, buckets, data, ops } = props;
+ const { leaf, channels, platforms, buckets, operations, data, ops } = props;
const type = leaf.match.type;
const ordinal = data.leafIds.indexOf(leaf.id) + 1;
return (
@@ -465,33 +472,78 @@ function LeafControls(props: {
</select>
)}
- <label className="flex items-center gap-1 text-xs text-muted-foreground">
- bucket
- {/* The option TEXT stays the raw bucket name: it is how an operator
- names a bucket to the runner, and one spec asserts exactly one
- option named `downloadedAutoSubsOnly` exists page-wide. The
- plain-language gloss goes in the sentence below instead. */}
- <select
- value={leaf.match.bucket ?? ""}
- aria-label="rule bucket"
- onChange={(e) =>
- ops.update(leaf.id, (n) => {
- const match = { ...(n as AutoQueueLeaf).match };
- if (e.target.value) match.bucket = e.target.value;
- else delete match.bucket;
- return { ...n, match };
- })
- }
- className={SELECT_CLASS}
- >
- <option value="">all buckets (default)</option>
- {buckets.map((b) => (
- <option key={b} value={b}>
- {b}
- </option>
- ))}
- </select>
- </label>
+ {/* THE OPERATION SELECT, for a lane that carries more than one.
+ `operation` and `bucket` are two id spaces and a leaf may name at most
+ one (sanitizeMatch enforces it, operation winning), so setting either
+ one clears the other here rather than letting the sanitizer decide
+ silently on save.
+
+ Native <select>, like every other control on this ladder: Radix
+ portals its options and unmounts them while closed, which would take
+ the page-wide option counts the e2e suite asserts to zero. */}
+ {operations.length > 1 && (
+ <label className="flex items-center gap-1 text-xs text-muted-foreground">
+ operation
+ <select
+ value={leaf.match.operation ?? ""}
+ aria-label="rule operation"
+ onChange={(e) =>
+ ops.update(leaf.id, (n) => {
+ const match = { ...(n as AutoQueueLeaf).match };
+ if (e.target.value) {
+ match.operation = e.target.value;
+ delete match.bucket;
+ } else {
+ delete match.operation;
+ }
+ return { ...n, match };
+ })
+ }
+ className={SELECT_CLASS}
+ >
+ <option value="">all operations (default)</option>
+ {operations.map((o) => (
+ <option key={o} value={o}>
+ {o}
+ </option>
+ ))}
+ </select>
+ </label>
+ )}
+
+ {buckets.length > 0 && (
+ <label className="flex items-center gap-1 text-xs text-muted-foreground">
+ bucket
+ {/* The option TEXT stays the raw bucket name: it is how an operator
+ names a bucket to the runner, and one spec asserts exactly one
+ option named `downloadedAutoSubsOnly` exists page-wide. The
+ plain-language gloss goes in the sentence below instead. */}
+ <select
+ value={leaf.match.bucket ?? ""}
+ aria-label="rule bucket"
+ onChange={(e) =>
+ ops.update(leaf.id, (n) => {
+ const match = { ...(n as AutoQueueLeaf).match };
+ if (e.target.value) {
+ match.bucket = e.target.value;
+ // The other half of the operation select's rule: at most one of
+ // the two id spaces, decided here rather than by the sanitizer.
+ delete match.operation;
+ } else delete match.bucket;
+ return { ...n, match };
+ })
+ }
+ className={SELECT_CLASS}
+ >
+ <option value="">all buckets (default)</option>
+ {buckets.map((b) => (
+ <option key={b} value={b}>
+ {b}
+ </option>
+ ))}
+ </select>
+ </label>
+ )}
<span className="text-xs text-muted-foreground">
{leafSentence(leaf, channels)}
diff --git a/editor/app/operations/components/OperationDetail.tsx b/editor/app/operations/components/OperationDetail.tsx
@@ -30,6 +30,7 @@ export function OperationDetail({
channels,
platforms,
bucketsByKind,
+ operationsByLane,
activeJobs,
runnerKind,
laneId,
@@ -45,6 +46,11 @@ export function OperationDetail({
channels: Channel[];
platforms: string[];
bucketsByKind: Record<AutoQueueKind, string[]>;
+ // The operation ids each lane dispatches, resolved on the SERVER off the
+ // registry (it is settings-dependent — `digest.remoteEnabled` moves the digest
+ // operation between two queue keys that are both the digest lane). The ladder
+ // offers a leaf an operation only for a lane carrying more than one.
+ operationsByLane: Record<AutoQueueKind, string[]>;
// Jobs on this operation's queues, SSR-rendered. Empty renders nothing at
// all (RunningJobsList returns null), so a quiet page carries no second
// in-flight list beside the lane's own.
@@ -111,6 +117,7 @@ export function OperationDetail({
channels={channels}
platforms={platforms}
buckets={bucketsByKind[runnerKind]}
+ operations={operationsByLane[runnerKind]}
onRefresh={refresh}
/>
</>
diff --git a/editor/app/operations/components/OrderReach.tsx b/editor/app/operations/components/OrderReach.tsx
@@ -5,7 +5,7 @@ import type {
AutoQueueOrder,
AutoQueueReach,
} from "yt-dlp-transcript-common/jobs/autoQueuePolicy";
-import { ORDER_LABEL, SELECT_CLASS } from "./dispatch";
+import { ORDER_LABEL, RECENCY_ORDERS, SELECT_CLASS } from "./dispatch";
// ORDER AND REACH, at the foot of the lane where the decision lives.
//
@@ -38,6 +38,11 @@ export function OrderReach({
// ("video order for auto-transcribe"). Optional because the sweep lanes have
// no such contract and the visible <label> already names the control.
orderAriaLabel,
+ // WHICH ORDERS THIS LANE OFFERS. Defaults to the three date-keyed ones, which
+ // is every order that existed before the lanes did — so no console gains an
+ // option it has no comparator for. A lane whose runner supplies one passes
+ // its own list.
+ orders = RECENCY_ORDERS,
onChange,
}: {
order: AutoQueueOrder;
@@ -50,6 +55,7 @@ export function OrderReach({
tradeoff: string | null;
busy: boolean;
orderAriaLabel?: string;
+ orders?: ReadonlyArray<AutoQueueOrder>;
onChange: (next: { order: AutoQueueOrder; reach: AutoQueueReach }) => void;
}) {
const orderId = useId();
@@ -79,7 +85,7 @@ export function OrderReach({
})
}
>
- {(Object.keys(ORDER_LABEL) as AutoQueueOrder[]).map((o) => (
+ {orders.map((o) => (
<option key={o} value={o}>
{ORDER_LABEL[o]}
</option>
diff --git a/editor/app/operations/components/PolicyTreeEditor.tsx b/editor/app/operations/components/PolicyTreeEditor.tsx
@@ -151,12 +151,14 @@ export function PolicyTreeEditor({
channels,
platforms,
buckets,
+ operations,
}: {
kind: AutoQueueKind;
status: AutoQueueKindStatus;
channels: Channel[];
platforms: string[];
buckets: string[];
+ operations: string[];
}) {
const [form, setForm] = useState<Form>(() => formOf(status));
// The last value we know is on disk. Everything dirty-related is a comparison
@@ -268,6 +270,7 @@ export function PolicyTreeEditor({
channels={channels}
platforms={platforms}
buckets={buckets}
+ operations={operations}
ops={ops}
data={{
pendingByLeaf: status.pendingByLeaf,
diff --git a/editor/app/operations/components/RunnerOperationView.tsx b/editor/app/operations/components/RunnerOperationView.tsx
@@ -40,6 +40,7 @@ export function RunnerOperationView({
channels,
platforms,
buckets,
+ operations,
onRefresh,
}: {
kind: AutoQueueKind;
@@ -48,6 +49,9 @@ export function RunnerOperationView({
channels: Channel[];
platforms: string[];
buckets: string[];
+ // The operation ids this lane dispatches. Empty for the two runner lanes
+ // today; the ladder draws a select only when there is more than one.
+ operations: string[];
onRefresh: () => Promise<void>;
}) {
const [busy, setBusy] = useState(false);
@@ -121,6 +125,7 @@ export function RunnerOperationView({
channels={channels}
platforms={platforms}
buckets={buckets}
+ operations={operations}
/>
<RecentPicks status={status} />
diff --git a/editor/app/operations/components/dispatch.ts b/editor/app/operations/components/dispatch.ts
@@ -73,13 +73,28 @@ export const ORDER_LABEL: Record<AutoQueueOrder, string> = {
listed: "Listed order",
newest: "Newest first",
oldest: "Oldest first",
+ // The digest lane's shortest-first. Named by DURATION, not by date, which is
+ // why it is not a third arm of the recency axis.
+ cheapest: "Shortest first",
};
+// THE ORDERS A DATE-KEYED LANE OFFERS — the three this control has always
+// offered. "cheapest" is deliberately not here: its comparator is supplied by
+// the lane's runner (slice 1.2), and a lane whose runner has none falls back to
+// "listed", so offering it would be a control that silently does nothing.
+export const RECENCY_ORDERS: ReadonlyArray<AutoQueueOrder> = [
+ "listed",
+ "newest",
+ "oldest",
+];
+
export function orderTradeoff(
kind: AutoQueueKind,
order: AutoQueueOrder,
): string | null {
- if (order === "listed") return null;
+ // Neither of these reorders by upload date, so neither costs a retry or a
+ // partial download its head start.
+ if (order === "listed" || order === "cheapest") return null;
return kind === "download"
? "Partial downloads lose their head start, so a half-finished download can wait behind fresh work."
: "Retries lose their head start, so a failed video can wait behind fresh work.";
diff --git a/editor/app/operations/lanes.ts b/editor/app/operations/lanes.ts
@@ -47,10 +47,12 @@ import {
// have is the same four things every lane has — a state, a reason, work in
// flight, and an order — and this is where those are read.
//
-// NO CLAIM LADDER IS RENDERED FOR THEM, deliberately. A policy tree for these
-// lanes does not exist yet (settings.autoQueue has exactly two keys), and
-// drawing an empty ladder would say "no rules match" where the truth is "rules
-// are not how this lane is dispatched". It gets one when the arbiter does.
+// NO CLAIM LADDER IS RENDERED FOR THEM YET. `settings.autoQueue` carries all
+// four lanes since slice 1.1, so the tree now exists — but nothing dispatches
+// off it until the runner does (slice 1.2), and drawing a ladder over a tree no
+// dispatcher reads would say "these rules decide the order" where the truth is
+// still "the sweep does". The ladder arrives with the runner, and this whole
+// file goes with the sweeps in slice 1.3.
export type SweepLaneId = "digest" | "backfill";
diff --git a/editor/app/operations/status.ts b/editor/app/operations/status.ts
@@ -12,6 +12,7 @@ import {
type AutoQueuePick,
readAutoQueueState,
} from "yt-dlp-transcript-common/jobs/autoQueueState";
+import { LANES } from "yt-dlp-transcript-common/lib/autoQueueTypes";
import type { AutoQueuePolicy } from "yt-dlp-transcript-common/jobs/autoQueuePolicy";
import { getWorkerPool } from "yt-dlp-transcript-common/jobs/workerPool";
import { isGateHeld } from "yt-dlp-transcript-common/lib/pauseGates";
@@ -63,13 +64,19 @@ export type AutoQueueKindStatus = {
held: boolean;
};
-export type AutoQueueStatusPayload = {
- transcription: AutoQueueKindStatus;
- download: AutoQueueKindStatus;
- // The other two pipelines, plus the comparison rail's bands. On the SAME
- // payload as the runners rather than a second endpoint, because the rail's
- // whole purpose is that four lanes are read together — two polls would let
- // the rail and the focused lane disagree about the same moment.
+export type AutoQueueStatusPayload = Record<
+ AutoQueueKind,
+ AutoQueueKindStatus
+> & {
+ // The sweep half of the digest and backfill pipelines, plus the comparison
+ // rail's bands. On the SAME payload as the runners rather than a second
+ // endpoint, because the rail's whole purpose is that four lanes are read
+ // together — two polls would let the rail and the focused lane disagree about
+ // the same moment.
+ //
+ // Every lane now has a `status[lane]` entry too. The two are not duplicates:
+ // this is what the SWEEPS are doing (armed, scoped, planned), and the entry
+ // is what the lane's TREE says. Slice 1.3 deletes this half.
lanes: AutoQueueLanesPayload;
};
@@ -100,18 +107,28 @@ async function buildKind(kind: AutoQueueKind): Promise<AutoQueueKindStatus> {
nextUp: pending.nextUp,
picks: state[kind].picks,
cooldowns,
+ // THE ASYMMETRY IS TRANSCRIPTION'S ALONE, and it is not a special case for
+ // "the first lane": its hold is LIVE on the worker pool, while every other
+ // lane's flag IS its gate. See lib/pauseGates.ts.
held:
kind === "transcription"
? getWorkerPool().isPaused()
- : isGateHeld(getSettings(), "download"),
+ : isGateHeld(getSettings(), kind),
};
}
+// ONE ENTRY PER LANE, off LANES rather than off two hand-written names — so a
+// lane added to the model appears on this payload with no edit here, which is
+// the whole point of the widened type.
export async function buildAutoQueueStatusPayload(): Promise<AutoQueueStatusPayload> {
- const [transcription, download, lanes] = await Promise.all([
- buildKind("transcription"),
- buildKind("download"),
+ const [kinds, lanes] = await Promise.all([
+ Promise.all(LANES.map((lane) => buildKind(lane))),
buildAutoQueueLanes(),
]);
- return { transcription, download, lanes };
+ return {
+ ...(Object.fromEntries(
+ LANES.map((lane, i) => [lane, kinds[i]]),
+ ) as Record<AutoQueueKind, AutoQueueKindStatus>),
+ lanes,
+ };
}
diff --git a/plans/one-core-phase-1.md b/plans/one-core-phase-1.md
@@ -252,3 +252,63 @@ arm-from-the-console on the fixture corpus produces the same tree the migration
1.4 a pause from the dashboard, the widget and the runner page all flip one `held` key and
the runner idles without stopping; after 1.5 `/channels` bands for download and
transcription draw from the operation entries with no rendered number changed.
+
+## 1.1, as shipped
+
+`7f294df` (this plan) → `f762944` (the numbers script) → this commit (the slice), on
+`one-core/phase-1` off `af2a360`. **Numbers: the before/after diff of
+`plans/tools/phase1-numbers.ts` over the live corpus is EMPTY.**
+
+Verification: `tsc --noEmit` clean in common, editor, export, mcp, homepage and umtool;
+`yt-dlp-transcript-common` 888 tests (876 before, +12 here), `yt-dlp-transcript-mcp` 205;
+`next build` clean; full editor e2e from a `p1-e2e` worktree of the slice commit —
+**511 passed, 0 failed, 24.4 min**, one worker behind the queue lock, matching Phase 0's
+511/511. The worktree needed the same seed Phase 0 recorded: `export/public` holds only
+the tracked static assets, so `build-index` + `compose-site` over a fixture corpus turns
+the export webServer's 500 into a 200.
+
+**The one design change: `retainLeaves` is deleted, not de-moded.** The plan said it
+"loses its mode: a leaf is retained when it draws anything from the lane's projection" —
+and that sentence, taken literally, describes the identity function. `pending[leaf]` IS
+what the leaf drew, so a post-hoc filter over it can only re-apply a rule the draw already
+applied. So the rule moved INTO the draw: `buildChannelWork` projects only
+`selectableBucketsForKind(lane)` and only `operationsNamedBy(root) ∩ operationsForLane(lane)`,
+and a leaf naming anything else finds no list. Verified equivalent at all three old call
+sites — the two runner ones (`operationsForLane` is `[]` for both, so an operation leaf
+projects nothing, exactly what the `"buckets"` mode zeroed) and the arbiter's, where
+`buildArbiterChannelWork` already projects `buckets: {}` and `planArbiterUnits` already
+skips every leaf with no `match.operation`.
+
+Shipped as planned: `LANES` + the four-member `AutoQueueKind` (`PauseLane` an alias);
+`digest`/`backfill` policies defaulting `enabled:false`, root `strict`, one `{type:"all"}`
+leaf, with an ABSENT lane block falling back to the whole default policy rather than to
+`sanitizePolicy({})` — a runner lane's absent root still means the empty tree it has
+always meant; `AutoQueueOrder` gains `"cheapest"` with the digest lane defaulting to it;
+`bucketsForKind`/`optInBucketsForKind`/`selectableBucketsForKind` return `[]` for the
+operation lanes; `operationsForLane(lane, settings)` in `lib/operations.ts`, off the queue
+key and off `laneFor` before `lane`; `autoQueueState` keyed by `LANES` with missing lanes
+coerced to empty on read AND on write; `/api/auto-queue/status` `buildKind` iterates
+`LANES`; `LeafControls` gains an operation select (drawn only above one operation) and
+hides the bucket select for a lane with none; `startAutoRunnersIfEnabled` still names the
+two lanes that have an executor.
+
+Two smaller divergences, both to keep "no UI text changes for the two existing lanes"
+true:
+
+- **`OrderReach` gained an `orders` prop** defaulting to the three date-keyed orders.
+ `Object.keys(ORDER_LABEL)` would have put a fourth option on every existing order
+ select, for an order whose comparator does not exist until 1.2. The digest lane passes
+ its own list when the runner supplies the comparator.
+- **`/api/auto-queue/control` still refuses a lane that is not `transcription` or
+ `download`.** Widening it would expose a Start button for a lane the runner cannot
+ execute. It widens in 1.2 with `run()`.
+
+**For 1.2.** `buildChannelWork`'s `operations` argument is now `laneOperationIds(kind,
+root)` — an INTERSECTION, so the digest lane projects nothing while its tree is the
+default catch-all leaf. `next()` drawing the lane's default operation union is the change
+that makes the lane live, and it is also where the projection cost lands: one channel's
+`digest.ids` is up to 11,329 strings and the status payload now folds four lanes on a
+three-second poll, so widen the projection in `run()`'s loop rather than in
+`computeLeafPending` if the two need to differ. `AutoQueueStatusPayload` is
+`Record<AutoQueueKind, AutoQueueKindStatus> & { lanes }`, so `data.digest` /
+`data.backfill` are already on the wire.
diff --git a/plans/one-core.md b/plans/one-core.md
@@ -181,41 +181,56 @@ Makes the later deletions safe and cheap.
four manifest versions, `SUMMARIES_PAGE_SIZE`, layer names) — the three files re-export
from it, nothing changes on the wire.
-### Phase 1 — Dispatch: one scheduler (4–5 slices)
+### Phase 1 — Dispatch: one scheduler (5 slices)
Finishes `unified-operations-model.md` steps 5 and 6 and deletes what they retire. This is
the phase with live numbers on a 78,000-video corpus; every slice is measured
-before/after with offline `tsx` over the real corpus, never a second editor.
-
-1. **Persist the arbiter** (`.arbiter/state.json`, same shape as
- `jobs/autoQueueState.ts`), and let `instrumentation.ts` resume it. Precondition named by
- the design doc: the arbiter has run in production once — this slice is the run.
-2. **Pause is node state** (step 5's remainder). `lib/pauseGates.ts` keeps `isGateHeld` /
- `withGateHeld` as the only API; storage moves from four settings fields to `held` on the
- tree root that dispatches the operation. The four fields stay readable for one release
- as a migration read, then are deleted with their sanitizers.
-3. **Retire `sweepEnabled` / `sweepKinds` / `sweepChannels` / `backfill.weight`** into tree
- leaves carrying `operation`, `posture: "passive"`, `order` (the shape the design doc
- already specifies). `lib/sweepPlan.ts` becomes the one planner;
- `controller/sweepPreview.ts` and `sweepRecency.ts` fold into it (recency stays cached
- through `controller/recencyIndex.ts`).
-4. **Delete `digestSweep.ts`, `backfillSweep.ts`, and `autoRunner.ts`'s dispatch loop.** The
- arbiter dispatches download and transcription too (they are catalogued operations
- already — `EXTERNAL_OPERATIONS`). `arbiterWork.buildChannelWork` is the one projection;
- autoRunner's copy goes. `digestBatch.ts` + `backfillBatch.ts` → one `operationBatch.ts`
- parameterised by the descriptor (`inputs / outputs / applyResult` already exist per
- kind). The GPU idle-only rule keys off `posture`, not off the presence of `laneFor`
- (FACTS.md records that trap; the test that pins it moves with the rule).
-5. **One work list in the snapshot.** Download and transcription gain `state()` in the
- registry so `snapshot.backfill[op]` (typed as `operations`, disk key unchanged) covers
- every operation. A slice-long assertion that `buckets.undownloaded ≡ operations.download.ids`
- and `buckets.untranscribed ≡ operations.transcription.ids` on all 68 snapshots runs
- before the bucket read path is deleted; the storage/cleanup buckets
- (`wrongFormatAudio`, cleanup sizes) remain buckets — they are not operations.
-
-Deleted by the end: two sweeps, one runner loop, one batch, two planners, four settings
-fields with their sanitizers and forms, `SweepLane`/`ArbiterBar`'s sweep-vs-arbiter
-distinction on `/operations`. Kept verbatim: the seven "must not be lost" invariants.
+before/after with offline `tsx` over the real corpus (`plans/tools/phase1-numbers.ts`),
+never a second editor.
+
+**Three surveys on 2026-09-07 changed one word of the plan below: the arbiter is not
+persisted, it is DELETED.** Zero `operations-arbiter` records among 1,305 production jobs,
+no `.arbiter/` state, no boot resume, and it cannot dispatch download or transcription —
+the two lanes carrying 100% of live dispatch. The auto-queue runner is the proven loop
+(persisted picks, boot resume, worker slots, platform backoff, an e2e-pinned console), so
+the RUNNER generalizes to four lanes and the arbiter goes with the sweeps.
+
+The slice-level plan, with the line numbers and the measurements, is
+[`one-core-phase-1.md`](one-core-phase-1.md).
+
+1. **1.1 — Four lanes in the model.** `AutoQueueKind` widens to `transcription |
+ download | digest | backfill` and is exported as `LANES`; `PauseLane` becomes an alias;
+ the two new lanes get a policy, a tree, a status entry and a state block. No dispatch,
+ no behaviour, no number moves.
+2. **1.2 — The runner runs operations.** `digestBatch.ts` + `backfillBatch.ts` → one
+ `controller/operationBatch.ts` parameterised by the descriptor; `autoRunner.run`
+ branches on lane; the GPU idle-only rule keys off `contendsFor`, not off the presence
+ of `laneFor` (FACTS.md records that trap; the test that pins it moves with the rule).
+ Ends with the digest lane driving a production pass — the "has run in production once"
+ precondition every retirement below stands on.
+3. **1.3 — Sweeps and arbiter retire.** Arming becomes tree authoring: one
+ `armLaneAction(lane, {operations, channels})` writes `autoQueue[lane].root`. The ten
+ `sweepEnabled` / `sweepKinds` / `sweepChannels` / `order` / `reach` / `weight` fields
+ migrate on read, then are deleted with their sanitizers and forms. `digestSweep.ts`,
+ `backfillSweep.ts`, `sweepPreview.ts`, `sweepRecency.ts`, `lib/sweepPlan.ts`,
+ `arbiter.ts` and `arbiterWork.ts` go.
+4. **1.4 — Pause is lane state.** `AutoQueuePolicy.held`, behind the unchanged
+ `isGateHeld` / `withGateHeld` API, with the four legacy fields read as a migration for
+ one release. All 13 callers unchanged; the transcription intent-vs-pool asymmetry stays.
+5. **1.5 — One work list per lane in the snapshot.** `generateChannelSnapshot` writes
+ `backfill.download` and `backfill.transcription` entries whose `ids` are the union
+ `defaultBucketsForPolicy` draws today, so `snapshot.backfill[op]` (typed `operations`,
+ disk key unchanged) covers every operation. **There is no `buckets.undownloaded` or
+ `buckets.untranscribed`** — the earlier draft of this step named fields that do not
+ exist; download's list is `snapshot.undownloadedIds` ∪ `buckets.partialDownloads` and
+ transcription's is `downloadedNoTranscript` ∪ `failedListed`. The storage/cleanup
+ buckets (`wrongFormatAudio`, cleanup sizes) remain buckets — they are not operations.
+
+Deleted by the end: two sweeps, the arbiter and its work projection, one batch, two
+planners, ten settings fields with their sanitizers and forms, and
+`SweepLane`/`ArbiterBar` from `/operations`. `autoRunner.ts` is GENERALIZED, not deleted —
+it is the loop everything else folds into. Kept verbatim: the seven "must not be lost"
+invariants.
### Phase 2 — The contract: one `ArchiveReader` (3 slices)
