commit 3ee0932d282f44f3b2eecfd7ce23c97672c5c59c
parent ffdaa5fc8dcf8e43e111b1961b9cc2a254994ee4
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Mon, 7 Sep 2026 22:40:02 -0400
plans+docs: the numbers script measures four lanes, and the prose stops naming the sweeps
Slice 1.3, fifth commit. `plans/tools/phase1-numbers.ts` prints all four lanes
off the sanitized, MIGRATED settings rather than the keys the file happens to
spell, plus each lane's leaf shapes — because the tree is what a sweep's scope
migrated INTO, and `enabled: false` alone would not say what it was armed on.
The file's own lanes are still printed FIRST, in the order they were printed
before, and the migrated ones appended. That is what keeps the before/after diff
pure addition: 0 lines removed, 18 added, and the two new sections read
`digest enabled = false` / `backfill enabled = false` with one `{"type":"all"}`
leaf apiece — which is the migration's whole claim about this corpus.
`digest.sweepEnabled` and `backfill.sweepEnabled` are read off the RAW FILE now,
since the settings type no longer has them: they are the migration's INPUT, and
the lines stay byte-identical until an operator cleans the retired keys out.
One new guard: library diagnostics are dropped from the output. Pricing the
digest lane's candidates makes the recency index log a `[recency] dating N of M`
line whose numbers move between runs on the same corpus, and one
nondeterministic line in a file whose entire purpose is to be compared byte for
byte would make every future slice's evidence unreadable.
AGENTS.md and RUNNING_IN_DOCKER.md said `ARCHILYZER_IDLE_BOOT=1` holds back "the
digest/backfill sweeps"; it holds back four lane runners. `plans/FACTS.md`
retires its sweep and arbiter anchors — kept as RETIRED entries rather than
deleted, because the reasoning outlived the code: the empty queue key for a job
that waits on jobs, the five populations that are never summed (now on
`OperationBand`), and `tsc` per package before calling a session green.
The changelog entry says what an operator has to know: their armed scope is
carried into the lane's rules, no lane is switched on that was not, and Reach,
the digest order field and the Resource share are retired rather than migrated —
each with the reason.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
5 files changed, 119 insertions(+), 67 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
@@ -152,10 +152,10 @@ Two things the image cannot bake, and the reasons matter:
carrying one enabled worker, because `defaults()` returns `workers: []` and zero
workers means auto-transcribe silently does nothing.
-`ARCHILYZER_IDLE_BOOT=1` boots the editor without arming the heartbeat, the
-auto-queue runners or the digest/backfill sweeps (`common/lib/idleBoot.ts`) — for
-pointing a fresh container at a corpus whose stored policies would otherwise resume
-GPU-weeks of work. The shutdown reaper stays armed regardless.
+`ARCHILYZER_IDLE_BOOT=1` boots the editor without arming the heartbeat or any of
+the four auto-queue lane runners (`common/lib/idleBoot.ts`) — for pointing a fresh
+container at a corpus whose stored policies would otherwise resume GPU-weeks of
+work. The shutdown reaper stays armed regardless.
Inside a container the multi-site build pipeline has no `docker` binary and falls
back to the serial host build it already handles. Do not try to make
diff --git a/RUNNING_IN_DOCKER.md b/RUNNING_IN_DOCKER.md
@@ -226,11 +226,12 @@ docker compose exec editor yt-dlp -U
### Booting without resuming work
-`editor/instrumentation.ts` arms the sync heartbeat, both auto-queue runners and
-the digest and backfill sweeps on every boot. That is right for a host install,
-where a restart interrupts work you own. It is wrong the first time you point a
-container at somebody else's corpus: its stored policies may say "sweep", and a
-corpus-wide digest sweep is GPU-*weeks*.
+`editor/instrumentation.ts` arms the sync heartbeat and every enabled auto-queue
+lane runner — transcription, download, digest and backfill — on every boot. That
+is right for a host install, where a restart interrupts work you own. It is wrong
+the first time you point a container at somebody else's corpus: its stored
+policies may have the digest lane switched on, and a corpus-wide digest pass is
+GPU-*weeks*.
```sh
ARCHILYZER_IDLE_BOOT=1
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,6 +1,7 @@
# Changelog
## [Unreleased]
+- **Every pipeline is dispatched by one thing now: its lane’s runner. The two corpus sweeps and the arbiter are gone.** Digest and Speaker work were driven by a *sweep* — a corpus walk armed by its own switch, with its own scope, its own order and its own console — while Download and Transcription were driven by the auto-queue runner, with rules, a claim ladder, a next-up and a pick log. Two mechanisms, two vocabularies, two sets of bugs. There is one: **each of the four lanes has a runner, a rule list, and Start / Drain / Stop beside its pause**, on the operation’s own page. Arming a corpus pass is switching the lane on; scoping it to particular channels or operations is a *rule*, written the same way auto-transcribe’s have been written since it shipped. The dashboard and the widget keep a one-click switch per lane — **Run every channel** / **Stop the lane** where they said *Sweep every channel* / *Stop sweeping* — and the scope lives on the lane’s page, where you can see what it would do next. **Your armed scope is carried over, and no lane is switched on that was not.** The ten settings fields the sweeps used (`digest.sweepEnabled`, `sweepChannels`, `recencyOrder`, `recencyReach`; `backfill.sweepEnabled`, `sweepKinds`, `sweepChannels`, `order`, `reach`, `weight`) are read once and written into the lane’s rules the first time the editor starts: a sweep armed on three channels becomes three rules, an unscoped one becomes a single *every channel* rule, and a disarmed sweep becomes a switched-off lane. What is retired rather than migrated: **Reach**, because a rule already orders every video it claims across every channel — which rule goes first is the rule list’s job; the digest **order**, whose real meaning was always *newest day first, shortest video within a day* and which the lane spells as **Shortest first** (pick *Newest first* there if you want the date order alone); and the backfill lane’s **Resource share**, which was one number answering two different questions. A lane now stands aside for transcription when it would actually compete for the graphics card, and keeps its slots when it would not — so speaker-naming over an LLM endpoint no longer parks itself behind a transcription it was not competing with. **The arbiter, which never ran a single unit in production, is deleted**; the runner is what dispatches an operation-named rule. **Nothing on disk changes**, and the retired keys are left in `settings.json` — harmless, ignored, and yours to delete.
- **The transcode operation is gone — it never fired.** A channel page had a *Transcode* stage, `/operations/transcode` had a "no console here" panel, `/cleanup` offered "Clear failed transcodings", and the video list drew a third status dot — all for a re-encode step built against two failures that never happened in production: in 68 channels, no snapshot has ever listed a video as missing its target format, no `failed-transcodings` file has ever held an id, and only four channels even met the stage's gate. Transcription never needed it — a video whose audio is in another format transcribes from that file. What stayed is everything that was never the operation's: the download path still re-encodes what it extracts itself, the video page still offers **Transcode audio.\<ext\> → \<fmt\>** per file, and both audio-format sweeps on the Cleanup stage and `/cleanup` are unchanged (gated on the channel having an `audioFormat`, which is what they compare against). The snapshot bucket behind the sweep is `wrongFormatAudio` now — its operator-facing name — and old reports keep their stray key until their next refresh. A `?stage=transcode` bookmark opens the channel overview. **Nothing on disk changes.** Also: the Pool's running-jobs list names the eight kinds its buttons enqueue, and the site's Search aliases tab no longer carries a "no site selected" branch that could not run.
- **A site has tabs, and the family has one page.** Charts, Search aliases, Deploy, Build and Homepage were five sidebar entries beside *Sites*, three of them reading the site from a `?site=` parameter the sidebar picker had to seed, one of them (Build) about no site at all, and one (Homepage) about the family's own hub. A site is one thing now: **`/sites/<id>` is Settings · Charts · Search aliases · Publish**, the site named in the path, the picker following it (and Dashboard and Channels following the picker). **`/sites` is the family page**: the list, then *Release notes*, *Build all sites* with the Basic/Docker mode, the *Hub*, and the *Pool* — the corpus-wide index, stats, sidecar and archive jobs — folded under a disclosure. Search aliases keep both sections on the site's tab: the global dictionary and the site's overrides. Every button, label and log is unchanged; "Select a specific site from the sidebar" is gone because a site's page always has one. The five routes redirect — a `?site=<id>` bookmark lands on that site's tab (the query rides along), `?site=__all__` and the bare routes on `/sites`; a bookmark to a deleted site 404s there exactly as `/sites/<id>` does. The Sites group is one entry; the nav is **eleven**, the IA doc's end state. **Nothing on disk changes.**
- **Jobs is one list.** `/jobs`, `/jobs/active` and `/jobs/queue` were three pages over three shapes — the directory listing, the registry's running work with its progress bars, and the scheduler's slots with their stuck reasons — and the same job was drawn three ways or hidden by one page's filter. `/jobs` is one table now: **one row per job**, the live head first (running, then queued in queue order with *next in line* / *2nd in line*, then anything that finished in the last half-minute) and the paged history below it, refreshed as before. A running row carries its per-task progress bars, its ETA and its *Transcripts: n / m* inside the Status cell; a queued row its Promote / ↑ / ↓; a stuck slot its `stuck · reason` badge, the last line of its log and **Force-release**, with **Reap stuck** on the health line above (active queues · running · queued · stuck · workers) and the lane strip above that. The kind/status/search filters apply to every row, live ones included. **The scheduler's drift check rides the live payload now**: every surface that draws it — this page, the dashboard, the monitor widget, `/api/jobs/active` — frees a running slot whose record is finished or gone the moment it sees one, after drawing it once. The channel, video, build and operation pages' *Active jobs* cards come off the same builder, so they show the progress bars they used to drop. `/jobs/active` and `/jobs/queue` redirect; the sidebar's *Active* entry and its running-count badge are gone (*Jobs* still counts running + queued); `/api/jobs/active` never moved. **Nothing on disk changes.**
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -820,8 +820,8 @@ anything with a settings surface gets one live run and one assertion.
| Channel-work sections | `editor/app/components/channelWork/sections.tsx` | `SectionConfig` + `channelWorkSections()` + `sectionsFor(op)`; rendered by `ChannelWorkTable.tsx` (a SERVER component — `primaryAction` is a function). Eight sections: three on `/operations/download`, two on `/operations/transcription`, one on `/operations/digest`, two (`operation: null`) on `/cleanup`. Counters live in `editor/app/lib/actionable/loadActionable.ts`; the review half is `editor/app/review/lib/loadReview.ts`. Unit-tested in `sections.test.ts`. |
| Widget sync payload | `editor/app/api/widget/sync/route.ts:15-26` | Comment at `:10-14` states it deliberately stays "a handful of scalars". `buildWidgetSyncPayload()` (`:30`) is exported for SSR seeding. |
| Pipeline band | `editor/app/components/dashboard/PipelineBand.tsx:63-114` | Pure props, no fetching. `<Instrument dotClass=…>` encodes state color. |
-| Operations board | `editor/app/operations/page.tsx` | `/operations`. Rail + arbiter + sync row, SSR-seeded, polls `/api/auto-queue/status` every 3 s. `data-board="operations"` carries `data-hydrated`. |
-| One operation | `editor/app/operations/[id]/page.tsx` | `/operations/<id>`, **routed off `operationCatalog()`** — an unknown id is `notFound()`, a new registry entry needs no route work. Runner ids (`download`, `transcription`) render `RunnerOperationView`; everything else renders `SweepLane` for its lane inside `<section data-lane="digest"|"backfill">`. |
+| Operations board | `editor/app/operations/page.tsx` | `/operations`. Rail + sync row, SSR-seeded, polls `/api/auto-queue/status` every 3 s. `data-board="operations"` carries `data-hydrated`. (The arbiter bar was here until slice 1.3.) |
+| One operation | `editor/app/operations/[id]/page.tsx` | `/operations/<id>`, **routed off `operationCatalog()`** — an unknown id is `notFound()`, a new registry entry needs no route work. Every operation with a lane (`pauseLaneFor`) renders `RunnerOperationView` inside `<section data-lane="transcription"\|"download"\|"digest"\|"backfill">`; since slice 1.3 that is the only lane section on the page. |
| Retired editor routes | `editor/next.config.ts` `redirects()` | `/auto-queue` → `/operations` and `/actionable` → `/operations`, both **temporary (307)**, not permanent — a 308 on a self-hosted admin surface is a support call with no remedy. Query strings pass through. The API paths `/api/auto-queue/{status,control}` and `/api/widget/actionable` did **not** move. |
---
@@ -1972,8 +1972,9 @@ its own header promising the same derivation as the batch. It now folds through
**`backfillSweep.ts` and `backfillBatch.ts` contained LITERAL NUL BYTES** (2 and 1
respectively), used correctly as join/key separators. This is why `grep` treated both as
binary, and why the scope comparison at `backfillSweep.ts:382` was repeatedly misread as
-`join(" ")` — it was already joining on a NUL. Both now use the `\u0000` escape;
-`digestSweep.ts` always did, and neither file needs `grep -a` any more.
+`join(" ")` — it was already joining on a NUL. Both used the `\u0000` escape by the end.
+**All three files are deleted** (slice 1.3 / 1.2); the lesson survives them, and
+`sweepRecency.ts` — which still had one — is gone too.
**Corpus census, 2026-08-09** (read-only walk of all 68 channel dirs, 20 s):
@@ -2051,21 +2052,23 @@ legitimately pending for digest AND diarization; an id-keyed set lets whichever
steal the other's work. Every pre-existing tree names no operation, so its keys are `"\0"+id`
and the dedup is byte-identical to before.
-**The arbiter runs on `queueKey: ""`** (`common/controller/arbiter.ts:395`) — a long-lived job that WAITS on jobs needing
-the real keys must not hold one or it deadlocks against its own work. It reserves by
-`lane.queueKey`, never invents one, dispatches one unit per lane per pass, one job per CHANNEL
-(77,000 per-video jobs would evict the registry's 100 records), and **refuses to start beside an
-armed sweep** (`common/controller/arbiter.ts:205`) because two dispatchers on one lane start the same channel twice.
-
-**`SweepKindCounts` carries five populations that are never summed** (`common/lib/sweepPlan.ts:37`): `reachable`,
-`missingInput`, `blocked`, `deferred`, plus `eligible` / `present` as `number | null` — null
-poisons a sum deliberately, because a partial denominator smaller than its numerator is a worse
-lie than "unknowable". Reachable and needs-media differ by ~91× on this corpus.
-
-**A test fixture of that type must carry all five** — `common/controller/sweepPreview.test.ts:258` had three literals with the
-old four fields after `dee7500` widened the type; `tsc` failed on `main` while `node --test`
-stayed 821/821, because the test runner does not typecheck. Run `tsc` per package before
-calling a session green.
+**RETIRED 2026-09-07 (slice 1.3): the arbiter and both sweeps are deleted.** What the two
+entries that stood here said is preserved because the reasoning outlived the code. The arbiter
+ran on `queueKey: ""` — a long-lived job that WAITS on jobs needing the real keys must not hold
+one or it deadlocks against its own work; `controller/autoRunner.ts` runs on the same empty key
+for the same reason. It refused to start beside an armed sweep, because two dispatchers on one
+lane start the same channel twice; there is one dispatcher per lane now and that rule
+(`laneBlockedReason`) is gone with the flag it read. `SweepKindCounts` (`lib/sweepPlan.ts`)
+carried five populations that are never summed — `reachable`, `missingInput`, `blocked`,
+`deferred`, plus `eligible` / `present` as `number | null`, because a partial denominator
+smaller than its numerator is a worse lie than "unknowable". That type is gone; **the rule
+survives on `OperationBand`** (`editor/app/components/pipelines/band.ts:27`), which the rail
+draws. Reachable and needs-media still differ by ~91× on this corpus.
+
+**Run `tsc` per package before calling a session green.** The lesson that used to be attached
+to `sweepPreview.test.ts`: three fixtures there kept the old four fields after `dee7500`
+widened the type, `tsc` failed on `main`, and `node --test` stayed 821/821 — the test runner
+does not typecheck.
**Worker tags route work through one rule** (`common/lib/workers.ts:136`): untagged matches everything, tagged
matches on intersection, a requirement lists acceptable qualifications. The pool's
@@ -2116,30 +2119,31 @@ the guard: `operationsForLane("backfill")` admits BACKFILL_QUEUE only, so digest
reach the backfill run's operation array even when asked for by name. `operations.test.ts`
pins it.
-**`laneForOperation` lives in `common/controller/operationLane.ts`**, alone, because the arbiter
-plans units against it and `operationJobs.ts` reserves a queue key with it — while the arbiter
-calls `operationJobs`. In `arbiter.ts` that was an import cycle.
+**`laneForOperation` lives in `common/controller/operationLane.ts`**, alone, because it started
+as the arbiter's: the arbiter planned units against it and `operationJobs.ts` reserves a queue
+key with it, while the arbiter called `operationJobs`. In `arbiter.ts` that was an import
+cycle. The arbiter retired in slice 1.3; the module stays where it is.
It asks `getBackfillKind`, **NOT `operationCatalog()`**, and that is load-bearing:
-`getBackfillKind("download")` is `undefined`, so download and transcription get no lane and
-`planArbiterUnits` skips them. A catalog lookup would hand them a lane and the arbiter would
-start a backfill channel job for work no backfill kind can do.
+`getBackfillKind("download")` is `undefined`, so download and transcription get no lane. A
+catalog lookup would hand them one, and a caller would reserve the backfill queue for work no
+backfill operation can do.
It also **does not consult `enabled`** and never has. A switched-off kind still resolves a lane;
-what keeps it out of the arbiter is the `enabled` set in `runArbiterPass`, which projects an
-empty id list. Teaching the resolver to return null would look like a tightening and would be a
+what keeps it out of dispatch is `operationsForLane`, which projects only the operations
+`allOperations` returns — so it arrives with an empty id list. Teaching the resolver to return null would look like a tightening and would be a
second, redundant gate that label-only callers would read as "no lane at all".
`common/controller/laneForOperation.test.ts` pins the boundary.
-**`common/controller/operationJobs.ts` is the ONE per-channel runner path** — a hand-clicked run,
-a swept one and an arbiter-dispatched one are the same function. `runBackfillChannelJob`,
+**`common/controller/operationJobs.ts` is the ONE per-channel runner path** — a hand-clicked run
+and a group-button one are the same function (a swept one and an arbiter-dispatched one were
+too, until both dispatchers retired in slice 1.3). `runBackfillChannelJob`,
`runDigestChannelJob`, and `runOperationChannelJob(operation, …)` over both. Since slice 1.2
all three drive `controller/operationBatch.ts`, which is also what the auto-queue runner's
digest and backfill lanes dispatch through — one executor, two callers.
-**They do NOT drain.** They return the `StreamActionResult` and the three callers that want
-sequencing (`backfillSweep`, `digestSweep`, `arbiter`) `await drainStream(result.stream)`
-themselves. A runner that drained would consume the stream the editor's `StreamActionLog`
+**They do NOT drain.** They return the `StreamActionResult` and a caller that wants sequencing
+`await drainStream(result.stream)`s itself. A runner that drained would consume the stream the editor's `StreamActionLog`
renders as a live log, and every stage card's log would come up dead. This is the single most
likely thing to be "fixed" back.
@@ -2167,12 +2171,13 @@ kind id / are a scheduler key and are correctly named.
**`ExternalOperation.runner?: AutoQueueKind`** names the auto-queue runner that dispatches an
operation. `dispatch` cannot answer it: download and transcription are `external` WITH a runner,
-a future `transcode` would be `external` with none. `/operations/<id>` reads it instead of
-testing the id — and reads `sweepLaneIdFor(op)` (off `lane.queueKey`, in
-`editor/app/operations/lanes.ts`) for the sweep lane, which returns **null** for an operation
-neither sweep runs. Null renders a "no console here" panel; the id-shaped predecessor would have
-rendered the backfill sweep's console — with a live Start button arming a corpus-scale sweep —
-under a `transcode` heading.
+a future `transcode` would be `external` with none. **Since slice 1.2 `/operations/<id>` chooses
+its console off `pauseLaneFor(op.id)`, not off `runner`** — `runner` is `undefined` for every
+registry operation, so reading it would leave the digest and backfill lanes without one — and
+`pauseLaneFor` asks `runner` FIRST, so an external entry sharing a queue with no runner of its
+own still gets `null`. Null renders a "no console here" panel; an id-shaped predecessor would
+have rendered the backfill lane's console — with a live Start button — under a `transcode`
+heading. (`sweepLaneIdFor` did this job for the sweep panel and is deleted with it.)
## Verified 2026-08-26 — the vocabulary pass (`BackfillKind` → `Operation`)
@@ -2843,10 +2848,10 @@ went from seven ids to eight, and this is the whole blast radius:
| `settings/page.tsx` worker tags | every id | moved to `workers/page.tsx`, filtered `scope === "video"` |
| `channels/page.tsx` `pipelineColumns` | Map lookup; `ids` = `EXTERNAL_BAND_IDS` + enabled kinds | unchanged — sync is in neither list, and its group is not in `OPERATION_GROUP_ORDER` |
| `channels/[slug]/lib/channelFlow.ts` | Map lookup over `buildChannelBands` ids | unchanged, same reason |
-| `operations/lanes.ts` `sweepLaneIdFor` | queue-key switch | `null` for sync |
+| `lib/pauseGates.ts` `pauseLaneFor` | `runner` first, then the queue key | `null` for sync (was `operations/lanes.ts` `sweepLaneIdFor`, deleted in slice 1.3) |
| `components/pipelines/buildBands.ts` | `EXTERNAL_BAND_IDS` + operation ids | unchanged — **no sync band, deliberately** |
| `controller/videoOperations.ts` | walks `OPERATIONS` | unchanged; sync is `scope: "channel"`, so a per-video reader has nothing to ask it |
-| `allOperations` / `backfillLaneOperations` / `countBackfillWork` / arbiter / `SweepScope` | `OPERATIONS` | untouched — the registry, not the catalog |
+| `allOperations` / `backfillLaneOperations` / `countOperationWork` / `operationsForLane` | `OPERATIONS` | untouched — the registry, not the catalog |
**`pauseGates.test.ts` pins the catalog's LENGTH**, which is what makes a new entry a failing
test rather than a silent omission: `assert.equal(ids.length, 8)` plus an expected lane for
diff --git a/plans/tools/phase1-numbers.ts b/plans/tools/phase1-numbers.ts
@@ -17,12 +17,23 @@
// offline; instrumentation.ts is not loaded, so no runner, sweep or scheduler
// is armed against the live corpus.
//
-// LANE COVERAGE IS DRIVEN BY THE FILE, NOT BY THE CODE. The lane sections come
-// from the keys actually present in settings.json's `autoQueue` object, not
-// from LANES — otherwise slice 1.1, which only ADDS two lanes to the in-memory
-// default, would print two new zero sections and the before/after diff would
-// be noise instead of evidence. The pause flags are the four PauseLane values,
-// which have all existed since pauseGates.ts landed.
+// LANE COVERAGE WAS DRIVEN BY THE FILE UNTIL SLICE 1.3, and now it is driven by
+// the SANITIZED, MIGRATED settings — all four lanes. The reason it was the file
+// is that slice 1.1 only ADDED two lanes to the in-memory default, so printing
+// them would have been noise rather than evidence. Slice 1.3 migrates the two
+// sweeps onto those lanes, which makes them real config with a real provenance,
+// and the whole claim of the slice is what they migrated TO.
+//
+// THE FILE'S LANES ARE STILL PRINTED FIRST, in the order they were printed
+// before, and the migrated ones are appended. That is deliberate: it keeps every
+// pre-existing line byte-identical and in place, so the before/after diff is
+// PURE ADDITION and can be read at a glance.
+//
+// The pause flags are the four PauseLane values, which have all existed since
+// pauseGates.ts landed. `digest.sweepEnabled` and `backfill.sweepEnabled` are
+// read off the RAW FILE now rather than off the settings type, which no longer
+// has them: they are the migration's INPUT, and the line stays until an operator
+// cleans the retired keys out of settings.json.
//
// Usage, from anywhere in the repo:
// pnpm --filter yt-dlp-transcript-common exec tsx ../plans/tools/phase1-numbers.ts
@@ -36,7 +47,8 @@ import {
readChannelSnapshot,
} from "../../common/controller/channels";
import { computeLeafPending } from "../../common/controller/autoRunner";
-import type { AutoQueueKind } from "../../common/lib/autoQueueTypes";
+import { LANES, type AutoQueueKind } from "../../common/lib/autoQueueTypes";
+import { flattenLeaves } from "../../common/jobs/autoQueuePolicy";
// Every count field on OperationSnapshotEntry, in a fixed order. `partial` is
// here alongside the four the plan names because it is part of OperationCounts
@@ -81,9 +93,33 @@ function printRow(prefix: string, row: Row): void {
}
}
+// THE OUTPUT IS A DIFF, so nothing but this script may write to it.
+//
+// `computeLeafPending` on the digest lane prices its candidates, and the recency
+// index logs a `[recency] dating N of M undated candidates …` line whose numbers
+// move between runs on the same corpus. One nondeterministic line in a file
+// whose entire purpose is to be compared byte for byte would make every future
+// slice's evidence unreadable, so diagnostics from common/ are dropped here
+// rather than the log being deleted from a library that is right to have it.
+function silenceLibraryLogs(): void {
+ const real = console.log.bind(console);
+ console.log = (...args: unknown[]) => {
+ if (typeof args[0] === "string" && args[0].startsWith("[")) return;
+ real(...(args as []));
+ };
+}
+
async function main(): Promise<void> {
+ silenceLibraryLogs();
const paths = getPaths();
const settings = getSettings();
+ // The raw file, for the two lines below that name keys the settings TYPE no
+ // longer has, and for "which lanes does the file itself spell".
+ const raw = JSON.parse(await readFile(paths.settingsFile, "utf8")) as {
+ autoQueue?: Record<string, unknown>;
+ digest?: Record<string, unknown>;
+ backfill?: Record<string, unknown>;
+ };
console.log("# one-core phase 1 — live numbers");
console.log(`# settingsFile: ${paths.settingsFile}`);
@@ -96,25 +132,26 @@ async function main(): Promise<void> {
console.log(`held ${lane} = ${isGateHeld(settings, lane)}`);
}
console.log(`digest.digestsPaused = ${settings.digest.digestsPaused}`);
- console.log(`digest.sweepEnabled = ${settings.digest.sweepEnabled}`);
+ console.log(`digest.sweepEnabled = ${raw.digest?.sweepEnabled === true}`);
console.log(`digest.remoteEnabled = ${settings.digest.remoteEnabled}`);
console.log(`backfill.enabled = ${settings.backfill.enabled}`);
- console.log(`backfill.sweepEnabled = ${settings.backfill.sweepEnabled}`);
+ console.log(`backfill.sweepEnabled = ${raw.backfill?.sweepEnabled === true}`);
console.log(`transcriptionsPaused = ${settings.transcriptionsPaused}`);
console.log(`downloadsPaused = ${settings.downloadsPaused}`);
console.log("");
- // --- lanes, as the file spells them --------------------------------------
- const raw = JSON.parse(await readFile(paths.settingsFile, "utf8")) as {
- autoQueue?: Record<string, unknown>;
- };
+ // --- lanes ---------------------------------------------------------------
+ // The file's own lanes first, in the order this script has always printed
+ // them, then whatever the migration filled in. Additions only.
const fileLanes = Object.keys(raw.autoQueue ?? {}).sort();
+ const laneOrder: AutoQueueKind[] = [
+ ...(fileLanes as AutoQueueKind[]),
+ ...LANES.filter((l) => !fileLanes.includes(l)),
+ ];
console.log("## lanes (from settings.json autoQueue)");
- for (const lane of fileLanes) {
- const policy = (settings.autoQueue as Record<string, unknown>)[lane] as
- | { enabled: boolean; order?: string; maxWorkers: number | null; replaceAutoSubs?: boolean }
- | undefined;
+ for (const lane of laneOrder) {
+ const policy = settings.autoQueue[lane];
if (!policy) {
console.log(`${lane} MISSING FROM SANITIZED SETTINGS`);
continue;
@@ -123,11 +160,19 @@ async function main(): Promise<void> {
console.log(`${lane} order = ${policy.order ?? "listed"}`);
console.log(`${lane} maxWorkers = ${policy.maxWorkers}`);
console.log(`${lane} replaceAutoSubs = ${policy.replaceAutoSubs === true}`);
+ // THE LEAF SHAPES, on one line, because the tree is what a sweep's scope
+ // migrated INTO and "enabled: false" alone would not say what it was armed
+ // on. Matches only: ids and weights are the sanitizer's business.
+ console.log(
+ `${lane} leaves = ${JSON.stringify(
+ flattenLeaves(policy.root).map((l) => l.match),
+ )}`,
+ );
}
console.log("");
console.log("## computeLeafPending (per lane, per leaf, sorted by leaf id)");
- for (const lane of fileLanes) {
+ for (const lane of laneOrder) {
const pending = await computeLeafPending(lane as AutoQueueKind, paths);
let total = 0;
for (const leafId of Object.keys(pending.counts).sort()) {
