commit 5aaeec9382a2492196141dbdbe2a4948a212f663 parent a35e901abbf219ff0ce44600f0d8e9464c530ec4 Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st> Date: Fri, 9 Oct 2026 11:52:43 -0400 ops: the read side — settings, storage, sites, workers, lanes, scheduler, cleanup; the lane control route needs the token Release 19 slice A3. GET /api/ops/{settings[?key], storage, sites, workers, auto-queue, scheduler, cleanup/<slug>} answer what each page already builds (getSettings, buildStorage, listSites, buildWorkersPayload, buildAutoQueueStatusPayload, buildSchedulerStatusPayload, and a new loadCleanupRow — loadCleanupSummary's row for one channel), behind the token, through one readRoute door (_read.ts) that refuses unknown query keys and redacts every secret-named string (a remote worker's token). The CLI reads them as `get settings [<key>]`, `get storage|sites|workers|auto-queue|scheduler`, `get cleanup <slug>`. /api/auto-queue/control was the one route that starts and stops lanes with no gate. It answers opsAuth now. Its one UI caller, the /operations lane header's Start/Drain/Stop, calls startAutoQueueAction / stopAutoQueueAction and a new drainAutoQueueAction instead (server actions carry Next's origin check; a page holds no token). The six e2e specs that drive the route send the test token. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Diffstat:
23 files changed, 507 insertions(+), 11 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md @@ -3,6 +3,8 @@ ## [Unreleased] - **`pnpm ops` finds the editor's token itself.** When `WORKER_TOKEN` or `ARCHILYZER_EDITOR_URL` is not set, it reads them from `editor/.env.local` and `editor/.env` of the checkout the script is in — wherever it is run from — and, in a linked worktree, which has no `editor/.env`, from the main worktree's; nothing else is taken from those files, and a variable already set wins. A 401 or 503 now says where the token came from (never what it is). `--wait` asks `GET /api/ops/job/<id>` (behind the token) whether a job it can no longer follow is still there, instead of the UI's live-jobs view; a job the editor has forgotten since (100 later jobs, or a restart) is reported with the status it ended with — it used to print `archived` and exit 1 for a job that finished `done` — and while a job waits, `--wait` prints its place in the queue whenever it changes. `GET /api/ops/job/<id>[?tail=N]` answers one job: its kind, channel, status, times and exit code, from its record or its sidecar; where it waits (`queue: {key, position, queued, head}`); and with `tail` its log's last N lines. Needs a restart of the editor. - **The /jobs list and its buttons over `pnpm ops`.** `pnpm ops get jobs` lists jobs newest first — `--active` every queued and running job in queue order with its place, `--failed` the failed ones, `--kind` and `--slug` to narrow either, `--limit` (50, at most 500); a filtered list looks through the newest 2000 jobs. `pnpm ops get job <id> --tail [N]` is one job with the last N lines of its log. `pnpm ops job cancel|drain|promote|force-release|retry <id>…` presses that row's button for each id (`POST /api/ops/job {"verb", "ids"}`) — the same server action, so the same refusals — and answers each id in `results`; an id that could not be acted on (unknown, or a promote of a job already at the front) makes the answer `ok: false`, named, without stopping the others. `job retry-failed` is the page's **Retry all**, and `--wait` follows the jobs a retry started; `job wait <id>…` follows jobs already running, printing each one's place in its queue while it waits. Retry all now returns the new jobs' ids beside its count. Needs a restart of the editor. +- **What the editor's pages show, over `pnpm ops`.** `get settings [<key>]` is settings.json as the editor reads it — migrated and defaulted — or one top-level key of it; `get storage` the /storage page (each location, mounted or not, its free space and tiers); `get sites` every site's id, title, public URL, Pages project, audience, whether it is listed, publish policy and channels; `get workers`, `get auto-queue` and `get scheduler` the payloads /workers, the four lanes and /operations/sync poll; `get cleanup <slug>` one channel's /cleanup row — what each sweep would reclaim, what holds the rest, the failed-transcriptions count. All behind the token, and no secret leaves through any of them: a remote worker's token reads `<redacted>`. Needs a restart of the editor. +- **`/api/auto-queue/control` needs the ops token.** It started, stopped and drained lanes for anything that could reach the editor, a form posted from another page in the operator's own browser included. A script now sends `Authorization: Bearer $WORKER_TOKEN` (or uses `pnpm ops lane {"lane", "action"}`, the same three verbs); the Start, Drain and Stop buttons on /operations call server actions and are unchanged to use. Needs a restart of the editor. - **A curated tag can exist on some sites only.** A tag's new **Sites** field on /tags (`sites` in `transcripts/tags.json`; `pnpm ops tags` takes it in a define) names the sites it exists on. Its rules then fire, and its pins apply, only to videos on those sites' channels, and every other site drops it from its records, its counts and its `/tags.json` — where **Hidden** only hid the chip. Empty is every site, as before. Setting it, or changing the channels of those sites, re-derives the corpus's tags once at the next index update. The Eva tags are what this is for: they belong on Anilyzer alone. - **The publish lane.** Publishing can run itself: turn it on at **/operations/publish** (the runner's Start, Drain and Stop, the hold, and the lane's settings; or `publish.enabled` in settings) and the lane checks every `checkEveryMinutes` (10) whether the index is stale; when it is — and its last update is at least `refreshEveryMinutes` (360) old — it updates it, then builds every site whose channels changed or whose data the new index moved, one stage at a time on the `publish` queue. What it may do with a site is the site's own — the **Publish policy** on the site's settings form, `site.json` `publish.auto` —: `off` (the default: left alone), `build`, `preview` (built and deployed to the preview branch `publish.previewBranch`) or `production`; the hub and the homepage have `publish.hub` and `publish.homepage`. A private site is only ever built, and a site needs its Cloudflare Pages project before it may deploy. Hold the lane and the stage running finishes and no next one starts; quiet hours (`publish.quietHours`) do the same; Drain finishes the stage and ends the runner. The lane never forces a stage: a stage that finds its target current does nothing. On /jobs every stage of one run reads `run <id> · <target>`, and a stage still queued when the editor restarts is cancelled, never re-queued — the lane works out again what is stale from what is on disk. `archilyzer publish now` runs the same plan from the command line, one stage after another in its own process. - **One index for every site.** The index is updated once and every site, the hub and the homepage are built from it; `archilyzer publish status` says, per site, whether its build is current — "stale: 3 channels changed (a, b, c)" as soon as a download, transcription or digest on one of its channels finishes, before any index runs; "stale: data changed" once the index has run and the site's data moved; "stale: config changed" after its site.json, tags or aliases changed — and whether what is deployed is that build, with a build made by older code marked "code newer" but not stale. diff --git a/editor/app/api/auto-queue/control/route.ts b/editor/app/api/auto-queue/control/route.ts @@ -7,14 +7,22 @@ import { } from "yt-dlp-transcript-common/controller/autoRunner"; import { LANES } from "yt-dlp-transcript-common/lib/autoQueueTypes"; import type { AutoQueueKind } from "yt-dlp-transcript-common/jobs/autoQueueState"; +import { opsAuth } from "../../ops/_lib"; export const dynamic = "force-dynamic"; // Start or stop an auto-queue runner without a server restart. The runner only // auto-starts at boot (editor/instrumentation.ts) when enabled, so the save flow -// and this endpoint are how an operator (or an e2e test) brings a runner up after -// flipping the enable toggle. The editor admin surface is otherwise -// unauthenticated (trusted self-host), consistent with the rest of the app. +// and this endpoint are how a script (or an e2e test) brings a runner up after +// flipping the enable toggle. +// +// BEHIND THE OPS TOKEN SINCE RELEASE 19 (A3). It was the one route that starts +// and stops lanes with no gate at all — reachable by anything that could reach +// the editor, and by a page in the operator's own browser posting a form at it +// (the CSRF reasoning in api/test/_guard.ts). The /operations Start, Drain and +// Stop buttons no longer call it: they are server actions now +// (operations/actions.ts), which Next checks for origin. `pnpm ops lane +// {"lane", "action"}` is the same three verbs for a caller with the token. // // Body: { kind: <one of LANES>, action: "start" | "stop" | "drain" }. // "stop" hard-cancels (aborts in-flight); "drain" lets in-flight units finish. @@ -24,6 +32,8 @@ export const dynamic = "force-dynamic"; // what to do with it is worse than no button. They have one now, and the check // is against LANES rather than two names so it cannot be forgotten again. export async function POST(req: Request) { + const denied = opsAuth(req); + if (denied) return denied; let body: { kind?: unknown; action?: unknown }; try { body = (await req.json()) as typeof body; diff --git a/editor/app/api/ops/_read.test.ts b/editor/app/api/ops/_read.test.ts @@ -0,0 +1,146 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { writeFile } from "node:fs/promises"; +import path from "node:path"; +import { callGet, callPost, setupOpsCorpus } from "./_testCorpus"; +import { redactSecrets } from "./_read"; + +// Run with: +// pnpm -C editor exec tsx --test "app/api/ops/_read.test.ts" +// +// The read side (release 19, A3): `get settings|storage|sites|workers| +// auto-queue|scheduler|cleanup <slug>` against a temp corpus — each behind the +// token, each the payload its page already builds, no secret in any of them — +// and the lane control route's new gate. + +const corpus = await setupOpsCorpus("one-youtube-channel-with-data"); +await corpus.writeSettings({ + minFreeDiskGB: 0, + workers: [ + { + id: "lan-box", + name: "LAN box", + kind: "remote", + enabled: false, + priority: 1, + remote: { baseUrl: "http://lan-box.local:3001", token: "never-printed" }, + }, + ], +}); +await corpus.writeSite("demo-site", { + siteUrl: "https://demo.example", + channels: [{ slug: "test-youtube", groupId: "default" }], +}); +await writeFile( + path.join(corpus.transcripts, "channels", "test-youtube", "snapshot.json"), + JSON.stringify({ + generatedAt: "2026-10-01T00:00:00.000Z", + buckets: { downloadedNoTranscript: ["a", "b"], downloadedAutoSubsOnly: [] }, + cleanupBytes: { transcribedWithAudio: 1000, multipleAudioFormats: 10, foreignAudio: 5 }, + }), +); + +type Get = Parameters<typeof callGet>[0]; +const route = async (name: string) => (await import(`./${name}/route`)).GET as Get; +test.after(() => corpus.cleanup()); + +test("every read route is behind the token", async () => { + for (const name of ["settings", "storage", "sites", "workers", "auto-queue", "scheduler"]) { + const res = await callGet(await route(name), undefined, {}, {}); + assert.equal(res.status, 401, name); + } + const cleanup = await callGet(await route("cleanup/[slug]"), undefined, { slug: "test-youtube" }, {}); + assert.equal(cleanup.status, 401); +}); + +test("get settings is the editor's read, with every secret redacted; ?key= answers one block", async () => { + const get = await route("settings"); + const all = await callGet(get); + assert.equal(all.status, 200); + const settings = all.body.settings as Record<string, unknown>; + assert.equal(settings.minFreeDiskGB, 0); + // A default the file never wrote is there: this is getSettings, not the file. + assert.ok("autoQueue" in settings); + assert.ok(!JSON.stringify(all.body).includes("never-printed")); + const workers = settings.workers as { id: string; remote?: { token?: string } }[]; + assert.equal(workers.find((w) => w.id === "lan-box")?.remote?.token, "<redacted>"); + + const one = await callGet(get, "http://localhost/api/ops/settings?key=minFreeDiskGB"); + assert.deepEqual(one.body, { ok: true, key: "minFreeDiskGB", value: 0 }); + const nope = await callGet(get, "http://localhost/api/ops/settings?key=minFreeDisk"); + assert.equal(nope.status, 400); + assert.match(String(nope.body.error), /no settings key "minFreeDisk" — known: .*minFreeDiskGB/); + const stray = await callGet(get, "http://localhost/api/ops/settings?block=x"); + assert.equal(stray.status, 400); + assert.match(String(stray.body.error), /unknown query key\(s\): block — accepted: key/); +}); + +test("redactSecrets replaces secret-named strings at any depth and leaves the rest", () => { + assert.deepEqual( + redactSecrets({ a: { apiKey: "k", token: "", n: 1 }, list: [{ password: "p", name: "x" }], maxTokens: 5 }), + { a: { apiKey: "<redacted>", token: "", n: 1 }, list: [{ password: "<redacted>", name: "x" }], maxTokens: 5 }, + ); +}); + +test("get sites lists each site from its site.json", async () => { + const res = await callGet(await route("sites")); + assert.equal(res.status, 200); + assert.deepEqual(res.body.sites, [ + { + siteId: "demo-site", + title: "demo-site", + siteUrl: "https://demo.example", + cloudflareProject: null, + audience: "public", + listed: true, + search: true, + publish: null, + channels: ["test-youtube"], + }, + ]); +}); + +test("get cleanup <slug> is the channel's ledger row, counts for the id lists", async () => { + const get = await route("cleanup/[slug]"); + const res = await callGet(get, undefined, { slug: "test-youtube" }); + assert.equal(res.status, 200); + assert.equal(res.body.slug, "test-youtube"); + assert.equal(res.body.transcribedBytes, 1000); + assert.equal(res.body.extraFormatsBytes, 10); + assert.equal(res.body.foreignBytes, 5); + assert.equal(res.body.included, true); + assert.equal(res.body.measured, false); + assert.equal(res.body.downloadedNoTranscript, 2); + assert.equal(res.body.reportedAt, "2026-10-01T00:00:00.000Z"); + assert.equal(res.body.channel, undefined); + assert.equal(res.body.snapshot, undefined); + const missing = await callGet(get, undefined, { slug: "no-such" }); + assert.equal(missing.status, 404); + const bad = await callGet(get, undefined, { slug: ".escape" }); + assert.equal(bad.status, 400); +}); + +test("get workers, auto-queue, scheduler and storage answer their pages' payloads", async () => { + const workers = await callGet(await route("workers")); + assert.equal(workers.status, 200, JSON.stringify(workers.body)); + assert.ok(!JSON.stringify(workers.body).includes("never-printed")); + const lanes = await callGet(await route("auto-queue")); + assert.equal(lanes.status, 200, JSON.stringify(lanes.body)); + assert.ok("transcription" in (lanes.body.autoQueue as Record<string, unknown>)); + const scheduler = await callGet(await route("scheduler")); + assert.equal(scheduler.status, 200, JSON.stringify(scheduler.body)); + const storage = await callGet(await route("storage")); + assert.equal(storage.status, 200, JSON.stringify(storage.body)); +}); + +test("/api/auto-queue/control is behind the ops token now", async () => { + const { POST } = await import("../auto-queue/control/route"); + const anon = await callPost(POST, { kind: "download", action: "stop" }, {}); + assert.equal(anon.status, 401); + const wrong = await callPost(POST, { kind: "download", action: "stop" }, { authorization: "Bearer nope" }); + assert.equal(wrong.status, 401); + // With the token it is the route it always was. + const bad = await callPost(POST, { kind: "transcode", action: "stop" }); + assert.equal(bad.status, 400); + assert.match(String(bad.body.error), /kind must be one of/); +}); diff --git a/editor/app/api/ops/_read.ts b/editor/app/api/ops/_read.ts @@ -0,0 +1,55 @@ +import { NextResponse } from "next/server"; +import { opsAuth, opsFail } from "./_lib"; + +// THE READ SIDE'S DOOR (release 19, A3): `GET /api/ops/<noun>` routes that +// answer what an existing builder already builds for a page — settings, the +// storage locations, the sites, the workers, the lanes, the sync scheduler, a +// channel's cleanup row — behind the token. A route here is the gate, the query +// check and one call; it shapes nothing a page does not already show. +// +// UNKNOWN QUERY KEYS ARE A 400, as unknown body keys are on the write side: a +// misspelled `?key=` would otherwise answer the whole document and look like +// an answer to the question asked. +export async function readRoute( + request: Request, + allowedQuery: readonly string[], + run: (q: URLSearchParams) => Promise<NextResponse | Record<string, unknown>>, +): Promise<NextResponse> { + const denied = opsAuth(request); + if (denied) return denied; + const q = new URL(request.url).searchParams; + const unknown = [...q.keys()].filter((k) => !allowedQuery.includes(k)); + if (unknown.length) { + return opsFail( + `unknown query key(s): ${unknown.join(", ")} — accepted: ${ + allowedQuery.length ? allowedQuery.join(", ") : "none" + }`, + ); + } + try { + const out = await run(q); + return out instanceof NextResponse ? out : NextResponse.json({ ok: true, ...out }); + } catch (e) { + return opsFail((e as Error).message, 500); + } +} + +// A SECRET NEVER LEAVES OVER THIS SURFACE, token or not. A remote worker's +// outbound `token` lives in settings.json beside everything else; the ops token +// lets a caller run the editor, not read the credentials of the machines it +// talks to. Any string under a key that names a secret is replaced by +// "<redacted>" ("" stays "", so "unset" still reads as unset). +const SECRET_KEY = /token|secret|password|passwd|api[-_]?key|credential/i; + +export function redactSecrets<T>(value: T): T { + if (Array.isArray(value)) return value.map(redactSecrets) as T; + if (typeof value !== "object" || value === null) return value; + const out: Record<string, unknown> = {}; + for (const [k, v] of Object.entries(value as Record<string, unknown>)) { + out[k] = + SECRET_KEY.test(k) && typeof v === "string" && v !== "" + ? "<redacted>" + : redactSecrets(v); + } + return out as T; +} diff --git a/editor/app/api/ops/_testCorpus.ts b/editor/app/api/ops/_testCorpus.ts @@ -58,6 +58,15 @@ export async function setupOpsCorpus(fixture: string | null): Promise<OpsCorpus> FFPROBE_BIN: path.join(BIN, "fake-ffprobe.mjs"), WHISPER_BIN: path.join(BIN, "fake-whisper.mjs"), WHISPER_MODEL: "/dev/null", + CHOUGH_BIN: path.join(BIN, "fake-chough.mjs"), + CHOUGH_MODEL: "/dev/null", + PARAKEET_STITCH_BIN: path.join(BIN, "fake-parakeet-stitch.mjs"), + PARAKEET_CLI: "/dev/null", + PARAKEET_MODEL: "/dev/null", + DIARIZE_BIN: path.join(BIN, "fake-diarize.mjs"), + CLAUDE_BIN: path.join(BIN, "fake-claude.mjs"), + FINDMNT_BIN: path.join(BIN, "fake-findmnt.mjs"), + UDISKSCTL_BIN: path.join(BIN, "fake-udisksctl.mjs"), WRANGLER_BIN: path.join(BIN, "fake-wrangler.mjs"), EXPORT_NEXT_BIN: path.join(BIN, "fake-next.mjs"), ARCHILYZER_BRANCH: "main", diff --git a/editor/app/api/ops/auto-queue/route.ts b/editor/app/api/ops/auto-queue/route.ts @@ -0,0 +1,16 @@ +import { buildAutoQueueStatusPayload } from "../../../operations/status"; +import { readRoute } from "../_read"; + +export const dynamic = "force-dynamic"; + +// GET /api/ops/auto-queue +// +// The four lanes' status — the /operations payload (`buildAutoQueueStatusPayload`, +// the same one /api/auto-queue/status polls): per lane its runner (running, +// draining, stopped), its policy (enabled, held, workers, order), cooldowns, +// what it picked last and what is pending. +export async function GET(request: Request) { + return readRoute(request, [], async () => ({ + autoQueue: await buildAutoQueueStatusPayload(), + })); +} diff --git a/editor/app/api/ops/cleanup/[slug]/route.ts b/editor/app/api/ops/cleanup/[slug]/route.ts @@ -0,0 +1,52 @@ +import { getPaths } from "yt-dlp-transcript-common/lib/paths"; +import { + isValidChannelSlug, + readChannelConfig, + readChannelSnapshot, +} from "yt-dlp-transcript-common/controller/channels"; +import { loadCleanupRow } from "../../../../cleanup/lib/loadCleanup"; +import { opsFail } from "../../_lib"; +import { readRoute } from "../../_read"; + +export const dynamic = "force-dynamic"; + +// GET /api/ops/cleanup/<slug> +// +// One channel's row of the /cleanup ledger (`loadCleanupRow`), off its report: +// what each sweep would reclaim — `transcribedBytes` (audio of videos that have +// a transcript: "Clean audio"), `extraFormatsBytes`, `foreignBytes` (audio not +// in the target format) — which OVERLAP and are never summed; what holds the +// rest (`heldBytes`/`heldCounts`: no transcript, keep-latest, do-not-clean, +// awaiting diarization); whether the channel is in the ledger's total +// (`included`); and the failed-transcriptions list's length. `measured: false` +// means the report predates the accounting: its zeros are unknowns. +// +// The sweeps themselves are `POST /api/ops/cleanup`. +export async function GET( + request: Request, + { params }: { params: Promise<{ slug: string }> }, +) { + const { slug } = await params; + return readRoute(request, [], async () => { + if (!isValidChannelSlug(slug)) { + return opsFail(`"${slug}" is not a valid channel slug`); + } + const paths = getPaths(); + const config = await readChannelConfig(paths, slug); + if (!config) return opsFail(`Channel "${slug}" not found`, 404); + const snapshot = await readChannelSnapshot(paths, slug); + const row = await loadCleanupRow(paths, { slug, config, snapshot }); + if (!row) return opsFail(`Channel "${slug}" not found`, 404); + const { channel: _c, snapshot: _s, transcribeIds, autoSubsIds, ...rest } = row; + void _c; + void _s; + return { + slug, + reportedAt: snapshot?.generatedAt ?? null, + ...rest, + // Counts, not the id lists: a bucket can hold thousands of ids. + downloadedNoTranscript: transcribeIds.length, + downloadedAutoSubsOnly: autoSubsIds.length, + }; + }); +} diff --git a/editor/app/api/ops/scheduler/route.ts b/editor/app/api/ops/scheduler/route.ts @@ -0,0 +1,16 @@ +import { buildSchedulerStatusPayload } from "../../../scheduler/status"; +import { readRoute } from "../_read"; + +export const dynamic = "force-dynamic"; + +// GET /api/ops/scheduler +// +// The sync scheduler — the /operations/sync payload +// (`buildSchedulerStatusPayload`, the same one /api/scheduler/status polls): +// each channel's resolved schedule and next due time, the recent tick log, and +// the effective scheduler settings. +export async function GET(request: Request) { + return readRoute(request, [], async () => ({ + scheduler: await buildSchedulerStatusPayload(), + })); +} diff --git a/editor/app/api/ops/settings/route.ts b/editor/app/api/ops/settings/route.ts @@ -0,0 +1,23 @@ +import { getSettings } from "yt-dlp-transcript-common/lib/settings"; +import { opsFail } from "../_lib"; +import { readRoute, redactSecrets } from "../_read"; + +export const dynamic = "force-dynamic"; + +// GET /api/ops/settings[?key=<top-level key>] +// +// settings.json as the editor reads it — through getSettings, so migrated and +// defaulted, exactly the values every action sees (not the raw file: a key the +// file leaves out answers with its default). `?key=autoQueue` answers one block. +// Secrets are redacted (`_read.ts`). SETTINGS.md is the key table. +export async function GET(request: Request) { + return readRoute(request, ["key"], async (q) => { + const settings = redactSecrets(getSettings()) as unknown as Record<string, unknown>; + const key = q.get("key"); + if (key === null) return { settings }; + if (!Object.hasOwn(settings, key)) { + return opsFail(`no settings key "${key}" — known: ${Object.keys(settings).sort().join(", ")}`); + } + return { key, value: settings[key] }; + }); +} diff --git a/editor/app/api/ops/sites/route.ts b/editor/app/api/ops/sites/route.ts @@ -0,0 +1,28 @@ +import { getPaths } from "yt-dlp-transcript-common/lib/paths"; +import { listSites } from "yt-dlp-transcript-common/lib/site"; +import { isListedSite, isPrivateSite } from "yt-dlp-transcript-common/lib/siteSchema"; +import { readRoute } from "../_read"; + +export const dynamic = "force-dynamic"; + +// GET /api/ops/sites +// +// Every site, one row each, from its site.json: id, title, public URL +// (`siteUrl`), Pages project, audience, whether the family lists it, whether it +// publishes search, its publish policy and the channels it carries. The deep +// status of a site's builds and deploys is `get publish`. +export async function GET(request: Request) { + return readRoute(request, [], async () => ({ + sites: listSites(getPaths()).map((s) => ({ + siteId: s.siteId, + title: s.siteTitle, + siteUrl: s.siteUrl ?? null, + cloudflareProject: s.cloudflareProject ?? null, + audience: isPrivateSite(s) ? "private" : "public", + listed: isListedSite(s), + search: s.search !== false, + publish: s.publish ?? null, + channels: s.channels.map((c) => c.slug), + })), + })); +} diff --git a/editor/app/api/ops/storage/route.ts b/editor/app/api/ops/storage/route.ts @@ -0,0 +1,15 @@ +import { buildStorage } from "../../../storage/buildStorage"; +import { readRoute } from "../_read"; + +export const dynamic = "force-dynamic"; + +// GET /api/ops/storage +// +// The /storage page's payload (`buildStorage`): every storage location with +// whether it is mounted, its free space, its channels and the bytes of each +// tier they hold, the corpus volume's text and clip tiers, and the saved-video +// store. Probes are the page's own (memoised 10 s); sizes come off the +// channels' reports, never a walk. +export async function GET(request: Request) { + return readRoute(request, [], async () => ({ storage: await buildStorage() })); +} diff --git a/editor/app/api/ops/workers/route.ts b/editor/app/api/ops/workers/route.ts @@ -0,0 +1,16 @@ +import { buildWorkersPayload } from "../../../workers/buildWorkers"; +import { readRoute, redactSecrets } from "../_read"; + +export const dynamic = "force-dynamic"; + +// GET /api/ops/workers +// +// The /workers page's live payload (`buildWorkersPayload`, the same one +// /api/workers polls): every configured transcription worker — enabled or not, +// busy or idle, what it is running — and whether the pool is paused. Behind the +// token, with any remote worker's credential redacted. +export async function GET(request: Request) { + return readRoute(request, [], async () => ({ + workers: redactSecrets(buildWorkersPayload()), + })); +} diff --git a/editor/app/cleanup/lib/loadCleanup.ts b/editor/app/cleanup/lib/loadCleanup.ts @@ -124,6 +124,19 @@ export function heldTotalOf(held: HeldAudio): number { ); } +// ONE channel's ledger row — the row loadCleanupSummary builds for it, from the +// same two reads (its brief, its failed-transcriptions list). For +// `GET /api/ops/cleanup/<slug>`: asking about one channel must not pay for +// every channel's snapshot. Null when the channel does not exist. +export async function loadCleanupRow( + paths: Paths, + brief: ChannelBrief | null, +): Promise<CleanupRow | null> { + if (!brief) return null; + const failedT = await loadFailedTranscriptions(paths, brief.slug); + return rowOf(brief, brief.snapshot, failedT.length); +} + export async function loadCleanupSummary( paths: Paths, ): Promise<CleanupSummary> { diff --git a/editor/app/operations/actions.ts b/editor/app/operations/actions.ts @@ -13,6 +13,7 @@ import { } from "yt-dlp-transcript-common/lib/pauseGates"; import { getWorkerPool } from "yt-dlp-transcript-common/jobs/workerPool"; import { + drainAutoRunner, startAutoRunner, stopAutoRunner, } from "yt-dlp-transcript-common/controller/autoRunner"; @@ -120,8 +121,9 @@ export async function saveAutoQueueAction( return { ok: true }; } -// Explicit start/stop for the Start/Stop buttons (also reachable as -// /api/auto-queue/control for e2e). Start is a no-op when the policy is disabled. +// Explicit start/stop/drain for the lane header's Start, Stop and Drain buttons +// (also reachable, behind the ops token, as /api/auto-queue/control and as +// `pnpm ops lane`). Start is a no-op when the policy is disabled. export async function startAutoQueueAction( kind: AutoQueueKind, ): Promise<SaveResult> { @@ -141,6 +143,15 @@ export async function stopAutoQueueAction( return { ok: true }; } +// Drain: the unit in flight finishes, no next one starts, the runner ends. +export async function drainAutoQueueAction( + kind: AutoQueueKind, +): Promise<SaveResult> { + drainAutoRunner(kind); + revalidateOperations(); + return { ok: true }; +} + // Idle a runner until `untilMs` (epoch ms) without stopping it, or wake it now // with null. Deliberately NOT part of saveAutoQueueAction: snoozing is a // one-click operational act, and routing it through the policy form would mean a diff --git a/editor/app/operations/components/RunnerOperationView.tsx b/editor/app/operations/components/RunnerOperationView.tsx @@ -18,6 +18,11 @@ import { PolicyTreeEditor } from "./PolicyTreeEditor"; import { SnoozeControl } from "./SnoozeControl"; import { type Channel, formatClock, formatCooldown, leafOrder } from "./dispatch"; import { clearPlatformHoldAction } from "../pacingActions"; +import { + drainAutoQueueAction, + startAutoQueueAction, + stopAutoQueueAction, +} from "../actions"; // The idle reasons that mean "paused or gated", not "waiting for work". const LANE_PAUSE_REASONS: ReadonlySet<string> = new Set([ @@ -85,11 +90,13 @@ export function RunnerOperationView({ async (action: "start" | "stop" | "drain") => { setBusy(true); try { - await fetch("/api/auto-queue/control", { - method: "POST", - headers: { "content-type": "application/json" }, - body: JSON.stringify({ kind, action }), - }); + // Server actions, not /api/auto-queue/control: that route is behind + // the ops token since release 19, and a page holds no token. + await (action === "start" + ? startAutoQueueAction(kind) + : action === "stop" + ? stopAutoQueueAction(kind) + : drainAutoQueueAction(kind)); await onRefresh(); } finally { setBusy(false); diff --git a/editor/e2e/auto-queue.spec.ts b/editor/e2e/auto-queue.spec.ts @@ -153,6 +153,8 @@ async function startRunner( kind: "transcription" | "download" = "transcription", ) { const res = await request.post(`${baseUrl}/api/auto-queue/control`, { + // Behind the ops token since release 19 (A3): it starts and stops lanes. + headers: { authorization: "Bearer test-worker-token" }, data: { kind, action: "start" }, }); expect(res.ok()).toBeTruthy(); @@ -164,6 +166,8 @@ async function stopRunner( kind: "transcription" | "download" = "transcription", ) { await request.post(`${baseUrl}/api/auto-queue/control`, { + // Behind the ops token since release 19 (A3): it starts and stops lanes. + headers: { authorization: "Bearer test-worker-token" }, data: { kind, action: "stop" }, }); } @@ -534,6 +538,8 @@ test("/jobs: the runner shows in its own labeled section, not Other", async ({ // Start the perpetual runner; it stays "running" (active) even once idle. await page.request.post(`${baseUrl}/api/auto-queue/control`, { + // Behind the ops token since release 19 (A3): it starts and stops lanes. + headers: { authorization: "Bearer test-worker-token" }, data: { kind: "transcription", action: "start" }, }); diff --git a/editor/e2e/auto-subs-replace.spec.ts b/editor/e2e/auto-subs-replace.spec.ts @@ -370,6 +370,8 @@ test("the auto-transcribe runner transcribes over auto-captions when opted in", }); const started = await request.post(`${baseUrl}/api/auto-queue/control`, { + // Behind the ops token since release 19 (A3): it starts and stops lanes. + headers: { authorization: "Bearer test-worker-token" }, data: { kind: "transcription", action: "start" }, }); expect(started.ok()).toBeTruthy(); @@ -386,6 +388,8 @@ test("the auto-transcribe runner transcribes over auto-captions when opted in", ); } finally { await request.post(`${baseUrl}/api/auto-queue/control`, { + // Behind the ops token since release 19 (A3): it starts and stops lanes. + headers: { authorization: "Bearer test-worker-token" }, data: { kind: "transcription", action: "stop" }, }); } diff --git a/editor/e2e/disk-space.spec.ts b/editor/e2e/disk-space.spec.ts @@ -152,6 +152,8 @@ test("the unattended auto-download runner idles below the floor", async ({ }); const start = await request.post(`${baseUrl}/api/auto-queue/control`, { + // Behind the ops token since release 19 (A3): it starts and stops lanes. + headers: { authorization: "Bearer test-worker-token" }, data: { kind: "download", action: "start" }, }); expect(start.ok()).toBeTruthy(); @@ -173,6 +175,8 @@ test("the unattended auto-download runner idles below the floor", async ({ ).toBe(false); } finally { await request.post(`${baseUrl}/api/auto-queue/control`, { + // Behind the ops token since release 19 (A3): it starts and stops lanes. + headers: { authorization: "Bearer test-worker-token" }, data: { kind: "download", action: "stop" }, }); } diff --git a/editor/e2e/lane-runner.spec.ts b/editor/e2e/lane-runner.spec.ts @@ -169,6 +169,8 @@ async function control( action: "start" | "stop" | "drain", ): Promise<{ started?: boolean; blocked?: string }> { const res = await request.post(`${baseUrl}/api/auto-queue/control`, { + // Behind the ops token since release 19 (A3): it starts and stops lanes. + headers: { authorization: "Bearer test-worker-token" }, data: { kind, action }, }); expect(res.ok()).toBeTruthy(); diff --git a/editor/e2e/pacing.spec.ts b/editor/e2e/pacing.spec.ts @@ -117,6 +117,8 @@ async function control( action: "start" | "stop", ) { const res = await request.post(`${baseUrl}/api/auto-queue/control`, { + // Behind the ops token since release 19 (A3): it starts and stops lanes. + headers: { authorization: "Bearer test-worker-token" }, data: { kind: "download", action }, }); if (action === "start") expect(res.ok()).toBeTruthy(); diff --git a/editor/e2e/rate-limit.spec.ts b/editor/e2e/rate-limit.spec.ts @@ -146,6 +146,8 @@ async function control( action: "start" | "stop", ) { const res = await request.post(`${baseUrl}/api/auto-queue/control`, { + // Behind the ops token since release 19 (A3): it starts and stops lanes. + headers: { authorization: "Bearer test-worker-token" }, data: { kind: "download", action }, }); if (action === "start") expect(res.ok()).toBeTruthy(); diff --git a/scripts/archilyzer-ops.mjs b/scripts/archilyzer-ops.mjs @@ -136,11 +136,39 @@ const GETTERS = { const s = q.toString(); return `/api/ops/jobs${s ? `?${s}` : ""}`; }, + // THE READ SIDE (release 19, A3): what each page draws, behind the token. + // settings.json as the editor reads it (migrated, defaulted, secrets + // redacted), or one top-level block of it. + settings: (key) => + key ? `/api/ops/settings?key=${encodeURIComponent(key)}` : "/api/ops/settings", + // The /storage page: each location, mounted or not, its free space and tiers. + storage: () => "/api/ops/storage", + // Every site: id, title, siteUrl, Pages project, audience, policy, channels. + sites: () => "/api/ops/sites", + // The transcription workers and what each is running. + workers: () => "/api/ops/workers", + // The four lanes: runner, policy, cooldowns, picks, pending. + "auto-queue": () => "/api/ops/auto-queue", + // The sync scheduler: each channel's schedule, the tick log. + scheduler: () => "/api/ops/scheduler", + // One channel's cleanup row: what each sweep would reclaim, what holds the rest. + cleanup: (slug) => `/api/ops/cleanup/${encodeURIComponent(slug)}`, }; // Nouns whose read takes no argument. `get channel` without a slug is a // mistake; `get tags` without one is the whole vocabulary. -const GET_ARG_OPTIONAL = new Set(["tags", "channels", "publish", "jobs"]); +const GET_ARG_OPTIONAL = new Set([ + "tags", + "channels", + "publish", + "jobs", + "settings", + "storage", + "sites", + "workers", + "auto-queue", + "scheduler", +]); // The flags each noun takes beyond the shared ones; any other is refused. const GET_FLAGS = { @@ -535,6 +563,9 @@ export function usage() { " pnpm ops job cancel|drain|promote|force-release|retry <id>... [--wait]", " pnpm ops job retry-failed [--wait]", " pnpm ops job wait <id>... [--wait-timeout <seconds>]", + " pnpm ops get settings [<key>]", + " pnpm ops get storage | sites | workers | auto-queue | scheduler", + " pnpm ops get cleanup <slug>", " pnpm ops list", "", `Actions: ${ACTIONS.join(", ")}`, @@ -562,6 +593,17 @@ export function usage() { " the answer ok: false without stopping the rest. --wait follows the jobs a", " retry started. job wait <id>... follows jobs already running.", "", + "get settings [<key>] is settings.json as the editor reads it (migrated and", + " defaulted; a secret is \"<redacted>\"), or one top-level key of it.", + "get storage is /storage: each location, mounted or not, free space, tiers.", + "get sites lists every site: id, title, siteUrl, Pages project, audience,", + " listed, search, publish policy, channels. (`get publish` is the status.)", + "get workers, get auto-queue and get scheduler are /workers, the four lanes", + " of /operations, and /operations/sync — the payloads those pages poll.", + "get cleanup <slug> is one channel's /cleanup row: what each sweep would", + " reclaim (they overlap — never add them), what holds the rest, and the", + " failed-transcriptions count. measured: false means unknown, not zero.", + "", 'publish runs publish stages on the editor\'s publish queue, one at a time,', ' under one run id: {"verb": …}. "index" updates the index; "build" builds', ' "siteId"/"siteIds" (forced; the index first when stale; "runner":', diff --git a/scripts/archilyzer-ops.test.mjs b/scripts/archilyzer-ops.test.mjs @@ -746,3 +746,18 @@ test("usage names the job reads and verbs", () => { assert.match(usage(), /pnpm ops get jobs \[--active \| --failed\]/); assert.match(usage(), /job <verb> <id>\.\.\. is a \/jobs row's button/); }); + +// THE READ SIDE (A3). +test("the read-side nouns are GETs on their routes, named in the usage", () => { + assert.equal(parseArgs(["get", "settings"]).path, "/api/ops/settings"); + assert.equal(parseArgs(["get", "settings", "autoQueue"]).path, "/api/ops/settings?key=autoQueue"); + for (const noun of ["storage", "sites", "workers", "auto-queue", "scheduler"]) { + const p = parseArgs(["get", noun]); + assert.equal(p.method, "GET", noun); + assert.equal(p.path, `/api/ops/${noun}`, noun); + } + assert.equal(parseArgs(["get", "cleanup", "the-quartering"]).path, "/api/ops/cleanup/the-quartering"); + assert.match(parseArgs(["get", "cleanup"]).error, /needs an argument/); + assert.match(usage(), /pnpm ops get settings \[<key>\]/); + assert.match(usage(), /get cleanup <slug> is one channel's \/cleanup row/); +});