commit 5c3e2336b07bf14aa99a27f7cafc482287f2bd61
parent 9833ffaf00bca48ddc813197fac67356d9550328
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 11 Aug 2026 01:34:38 -0400
Merge main: drop the bookmark spec the channel-cockpit migration had rewritten
main removed the command-bookmark layer while this branch was in flight, so the
only real conflict was editor/e2e/bookmarks.spec.ts — deleted there, rewritten
here by the ?stage= navigation migration. Deleted: the feature it drove is gone,
and jobs-retry.spec.ts already picked up the replay-table coverage it was the
last holder of.
The changelog conflict was two sets of entries appended to the same [Unreleased]
heading; both are kept.
One comment reworded: page.tsx described a stale `?stage=` URL as a "bookmark",
which now names a feature that no longer exists — it means a saved link, so it
says that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Diffstat:
97 files changed, 8495 insertions(+), 3648 deletions(-)
diff --git a/.claude/commands/ask.md b/.claude/commands/ask.md
@@ -0,0 +1,20 @@
+---
+description: Ask a question of the transcript corpus and answer it with citations
+---
+
+Call the `mcp__archilyzer__ask_plan` tool with this exact `request` string,
+passed through **verbatim and unsplit**:
+
+$ARGUMENTS
+
+Then follow the plan it returns, starting with its ⚠ warnings if it has any.
+
+<!--
+Same shape as `/sweep` — see `.claude/commands/sweep.md` for why the request
+goes through a tool rather than an MCP prompt argument, and for the two editing
+caveats (no `$1` in this body; the `mcp__archilyzer__` prefix is the server name
+as registered on this machine).
+
+Use `/ask` for a question to be answered in the conversation, and `/sweep` for a
+systematic pass that writes a report file.
+-->
diff --git a/.claude/commands/sweep.md b/.claude/commands/sweep.md
@@ -0,0 +1,39 @@
+---
+description: Sweep the transcript corpus for something and build a cited report
+---
+
+Call the `mcp__archilyzer__sweep_plan` tool with this exact `request` string,
+passed through **verbatim and unsplit**:
+
+$ARGUMENTS
+
+Then follow the plan it returns, starting with its ⚠ warnings if it has any.
+
+<!--
+Why this file exists
+--------------------
+Claude Code parses an MCP *prompt*'s arguments as `text.trim().split(/\s+/)`
+zipped against the declared argument names. It is not quote-aware, the last
+argument does not absorb the remainder, and tokens past the declared count are
+dropped silently. With the nine arguments the `sweep` prompt declares, a real
+request became link="This" channel="search" group="deleted" directive="videos."
+batch_size="Why" — and everything from the actual question onward vanished.
+
+`$ARGUMENTS` in a slash command is different: it is replaced with the entire
+raw argument string, untokenised, so URLs, `?`, `=`, `&` and punctuation all
+survive. A tool argument is structured JSON, so it arrives intact from there.
+
+Two things to keep in mind when editing this file:
+
+ - Do NOT also use `$1` / positional placeholders in this body. Those DO
+ tokenize, and mixing them re-introduces the bug this file exists to avoid.
+ - The tool name embeds the MCP server name **as registered on this machine**
+ (`archilyzer`). If you registered the server under another name, change the
+ `mcp__<name>__sweep_plan` prefix to match. `foo:bar` namespacing is
+ plugin-only, so the command you type is `/sweep`, not `/archilyzer:sweep`.
+
+The sweep discipline itself — citation format, extractor budget, base-link
+expansion, the coverage rule — deliberately lives in the server
+(`mcp/src/instructions.ts`), not here, so it cannot drift out of sync with the
+tools it names.
+-->
diff --git a/.diarize/README.md b/.diarize/README.md
@@ -1,8 +1,31 @@
# `.diarize/` — the diarization runtime
-The Python environment and ONNX models `scripts/diarize.mjs` needs. **Everything
-here except this file is gitignored** (`env/` and `models/`, ~148 MB, both
-machine-specific). This file is tracked so the environment is reproducible.
+The engines `scripts/diarize.mjs` needs. **Everything here except this file is
+gitignored** — `env/` and `models/` for sherpa-onnx (~148 MB) and `sortformer/`
+for the ggml engine (~530 MB), all machine-specific. This file is tracked so the
+environment is reproducible.
+
+Two engines live here, selected by `settings.diarization.engine`:
+
+| | `sherpa-onnx` (default) | `sortformer` |
+| --- | --- | --- |
+| method | pyannote segmentation + TitaNet embeddings, agglomerative clustering | end-to-end streaming Sortformer |
+| device | CPU only | Vulkan or CPU |
+| speakers on `v6z1o2g` | 13 (ground truth: 1 host + clips) | **4** |
+| speakers on the corpus's worst file | 35 | **4** |
+| dominant-speaker share | 73.1% | 73.5% |
+| throughput | **492–585 s/audio-hour**, ~3.9 cores | 894 s/audio-hour on Vulkan (~1 core), 1305 on tuned CPU |
+| memory | ~230 MB/audio-hour, O(n²) in turn count | **558 MB flat**, O(1) in duration |
+| knobs | threshold (0.9 here) | none |
+
+The sherpa range is two clean runs of the SAME file rather than an estimate:
+585 s/audio-hour on 2026-08-08 (§ below) and 492 on 2026-08-10, both on an idle
+box. Treat sub-20% differences between runs as noise on this hardware.
+
+sherpa-onnx is FASTER in wall clock. sortformer is chosen for quality:
+over-splitting is the failure mode this lane has always had, and an end-to-end
+model does not have it. See `plans/diarization-spike-results.md` for the original
+CPU-only spike and the measurements that decided the defaults.
## Why it exists at all
@@ -87,3 +110,88 @@ when the spike ran it under load ~30.
Memory: ~230 MB per audio-hour of decoded float32 audio, so an 8-hour VOD needs
~1.8 GB resident. Untested on the long tail.
+
+---
+
+## The sortformer engine
+
+### Rebuilding it
+
+```sh
+scripts/build-sortformer.sh # Vulkan (default)
+SORTFORMER_BACKEND=cpu scripts/build-sortformer.sh
+```
+
+Needs `git`, `cmake`, a C++17 compiler, and for Vulkan the loader + headers and
+`glslc` (Arch: `vulkan-headers shaderc`). Installing `patchelf` is optional — it
+sets `RPATH=$ORIGIN` on the binary; without it the wrapper sets
+`LD_LIBRARY_PATH` itself.
+
+The script clones **openresearchtools/engine at a pinned commit**
+(`8bb4928c`, 2026-03-21), stages only `ggml` + `tools/realtime` + our driver, and
+builds those. It deliberately uses none of upstream's build system, whose scripts
+are PowerShell/CUDA — the Sortformer code links against `ggml` alone (no llama,
+no whisper, no Rust), which is what makes that possible. First build is ~10
+minutes, almost all of it compiling Vulkan shaders; rebuilds after a driver edit
+are seconds, because `ggml` is staged once per pin.
+
+The model (471 MB, F32) is downloaded from
+`openresearchtools/diar_streaming_sortformer_4spk-v2.1-gguf`. **It is under the
+NVIDIA Open Model License, not the engine's MIT** — the GGUF is a conversion of
+`nvidia/diar_streaming_sortformer_4spk-v2.1`.
+
+### Wiring it up
+
+```json
+"diarization": {
+ "engine": "sortformer",
+ "backend": "vulkan",
+ "sortformerBin": "<repo>/.diarize/sortformer/diarize-file",
+ "sortformerModel": "<repo>/.diarize/sortformer/diar_streaming_sortformer_4spk-v2.1.gguf"
+}
+```
+
+`threshold`, `segModel`, `embModel` and `python` are ignored by this engine and
+do not enter its freshness identity. **Switching `engine` marks every sidecar
+written by the other one stale**, which is intended — see `diarizationTarget` —
+but on the retained audio that is weeks of rework, not a toggle.
+
+### Checking a rebuild reproduces the record on disk
+
+Read-only, writes nothing to the corpus:
+
+```sh
+node scripts/diarize.mjs --output /tmp/repro.json --video-id v6z1o2g \
+ --engine-kind sortformer \
+ --sortformer-bin "$PWD/.diarize/sortformer/diarize-file" \
+ --sortformer-model "$PWD/.diarize/sortformer/diar_streaming_sortformer_4spk-v2.1.gguf" \
+ --backend vulkan --threads 6 \
+ transcripts/channels/ObviousRises-rumble/data/v6z1o2g/audio.mp3
+```
+
+Expected on 2026-08-10: **66 turns, 4 speakers, `audioSeconds` 766.101**, in
+206.5 s (3.71x realtime). Two independent equivalences were checked and both
+held exactly:
+
+* the **CPU and Vulkan backends produce byte-identical turns**, so the backend is
+ a throughput choice and never a quality one; and
+* piping mp3 through ffmpeg into the streaming stdin path gives the **same 66
+ turns** as running the binary directly on a decoded WAV.
+
+### Notes for whoever touches the driver
+
+`scripts/sortformer/diarize-file.cpp` is vendored here rather than taken from
+upstream because upstream's `llama-realtime-smoke` is a parity tool: it retains
+every intermediate matrix, demands PyTorch reference fixtures, and — the fatal
+part — drops the event flags, so its JSON mixes 2,033 *preview* re-emissions in
+with the 66 real spans.
+
+Two things in it are non-obvious and both were found the hard way:
+
+* **stdin is forced back to blocking.** Node hands children non-blocking pipes,
+ so `read()` returns `EAGAIN` long before EOF; it worked from a shell and failed
+ under the app.
+* **thread count is set explicitly.** The upstream Sortformer path never sets
+ one, so the CPU backend silently runs at ggml's default of 4. On this 4c/8t box
+ 6 is the optimum (1305 s/audio-hour) and **8 is worse than 4** (1401) through
+ oversubscription.
diff --git a/.gitignore b/.gitignore
@@ -120,3 +120,7 @@ yarn-error.log*
/.diarize/env/
/.diarize/models/
/.diarize/logs/
+# Sortformer engine: the built ggml binary + shared objects + the 471 MB GGUF.
+# Machine-specific and rebuildable with scripts/build-sortformer.sh.
+/.diarize/sortformer/
+/.diarize/.sortformer-build/
diff --git a/PLAN.md b/PLAN.md
@@ -563,7 +563,7 @@ only party who will ever notice — the detector already believes the two are th
Ingestion is an editor action accepting pasted JSON. Treat submissions
as **untrusted input rendered in an admin UI**: escape it, never feed it into a prompt
-unreviewed. Store under `transcripts/.feedback/`, a sibling of `.jobs` and `.bookmarks`,
+unreviewed. Store under `transcripts/.feedback/`, a sibling of `.jobs` and `.scheduler`,
outside the build trees.
**Why that category is not boilerplate:** the smoke test summarized allegation-heavy content
diff --git a/common/controller/backfillBatch.test.ts b/common/controller/backfillBatch.test.ts
@@ -197,3 +197,35 @@ test("the duration cap round-trips, and 0 means off", () => {
// back on by accident would silently stop diarizing long videos.
assert.equal(defaultDiarization().maxAudioHours, 0);
});
+
+test("an unknown engine or backend falls back to the default, never to nothing", () => {
+ // The DEFAULT ENGINE IS LOAD-BEARING as a fallback, not just as a starting
+ // point: it is the one every sidecar already on disk matches, so falling back
+ // to it leaves the corpus fresh. Falling back to sortformer on a typo would
+ // mark all of it stale and offer weeks of rework.
+ assert.equal(defaultDiarization().engine, "sherpa-onnx");
+ assert.equal(sanitizeDiarization({ engine: "sortformer" }).engine, "sortformer");
+ assert.equal(
+ sanitizeDiarization({ engine: "sortfromer" }).engine,
+ defaultDiarization().engine,
+ );
+ assert.equal(sanitizeDiarization({ engine: 7 }).engine, defaultDiarization().engine);
+
+ // The backend defaults to the GPU, which is safe only because the lane yields
+ // the card rather than sharing it — see diarizationLaneFor.
+ assert.equal(defaultDiarization().backend, "vulkan");
+ assert.equal(sanitizeDiarization({ backend: "cpu" }).backend, "cpu");
+ assert.equal(
+ sanitizeDiarization({ backend: "rocm" }).backend,
+ defaultDiarization().backend,
+ );
+
+ // Paths are plain strings and empty means "not configured", which diarizeOne
+ // reports as a skip rather than a failure.
+ assert.equal(sanitizeDiarization({}).sortformerBin, "");
+ assert.equal(sanitizeDiarization({}).sortformerModel, "");
+ assert.equal(
+ sanitizeDiarization({ sortformerBin: " /opt/diarize-file " }).sortformerBin,
+ "/opt/diarize-file",
+ );
+});
diff --git a/common/controller/backfillBatch.ts b/common/controller/backfillBatch.ts
@@ -41,6 +41,7 @@ import { readVideoFiles } from "../lib/videoStatus";
import {
addBackfillState,
emptyBackfillCounts,
+ laneYieldsToTranscription,
reachableBackfillWork,
resolveBackfillKinds,
type BackfillClassification,
@@ -475,7 +476,8 @@ export async function runBackfillBatch(
// poll and survives a restart with no boot hook (the downloadsPaused
// pattern). Returning 0 makes runPool idle-wait — a hold; returning null
// from next() would END the batch, which is not the same thing.
- const live = getSettings().backfill;
+ const liveSettings = getSettings();
+ const live = liveSettings.backfill;
if (!live.enabled) {
if (!yielding) {
yielding = true;
@@ -497,8 +499,34 @@ export async function runBackfillBatch(
// can overlap on the CPU; the benefit is that neither can deadlock the
// other.
const activity = transcriptionActivity();
+ // A GUARANTEED SHARE IS A CPU CONCEPT. `weight > 0` means "keep a slice of
+ // the cores running even while transcription works", which is reasonable
+ // when the contended resource is cores and divisible. It is not available
+ // for VRAM: sortformer on Vulkan holds ~4.4 GB of the same 8 GB card
+ // parakeet is using, so "a small share" is not a slower run, it is an
+ // out-of-memory failure of whichever lane allocates second.
+ //
+ // So a run that could dispatch GPU work is idle-only whatever the weight
+ // says. Which kinds those are is DECLARED (laneFor/contendsFor), not
+ // re-tested here, so the rule stays next to the queue key it belongs with.
+ // The cost is bluntness — one such kind makes the whole run idle-only,
+ // including its cheap CPU-bound siblings — and that is the deliberate
+ // direction to be wrong in, since the alternative is an OOM mid-sweep.
+ //
+ // ONLY kinds that declare laneFor are considered, and that is the fix for a
+ // real regression rather than a nicety. `digest` declares `contendsFor:
+ // "gpu"` statically, so testing every kind's lane made this lane idle-only
+ // whenever a digest was in the run — which broke the guarantee that a digest
+ // runs CONCURRENTLY with a backfill rather than behind it. A kind with a
+ // static GPU lane already arbitrates itself (digestBatch has its own yield);
+ // laneFor marks the kinds whose resource changes under them from settings,
+ // which today is diarization alone, and those are the ones nothing else is
+ // deciding for.
+ const gpuBound = kinds.some(
+ (k) => k.laneFor && laneYieldsToTranscription(k.laneFor(liveSettings)),
+ );
const limit = backfillLimit({
- weight: live.weight,
+ weight: gpuBound ? 0 : live.weight,
slots: live.concurrency,
primaryBusy: activity.busy,
});
diff --git a/common/controller/diarizeOne.ts b/common/controller/diarizeOne.ts
@@ -18,6 +18,7 @@ import { getSettings } from "../lib/settings";
import type { DiarizationSettings } from "../lib/settings";
import {
DIARIZATION_FILENAME,
+ SORTFORMER_DIARIZATION_ENGINE,
diarizationTarget,
isDiarizationFresh,
} from "../lib/diarization";
@@ -81,7 +82,17 @@ export async function diarizeOneVideo(
const cfg = opts.settings ?? getSettings().diarization;
if (!cfg.enabled) return "disabled";
- if (!cfg.segModel || !cfg.embModel) {
+ // Each engine has its own idea of "configured", and reporting the wrong one
+ // sends an operator to fix a setting the selected engine never reads.
+ const usingSortformer = cfg.engine === SORTFORMER_DIARIZATION_ENGINE;
+ if (usingSortformer) {
+ if (!cfg.sortformerBin || !cfg.sortformerModel) {
+ log(
+ `Diarize ${opts.videoId} skipped: sortformer engine selected but no binary/model configured (run scripts/build-sortformer.sh).`,
+ );
+ return "not-configured";
+ }
+ } else if (!cfg.segModel || !cfg.embModel) {
log(
`Diarize ${opts.videoId} skipped: no segmentation/embedding model configured.`,
);
@@ -106,6 +117,44 @@ export async function diarizeOneVideo(
const start = Date.now();
log(`Diarize ${opts.videoId} start (${media})`);
try {
+ const engineArgs = usingSortformer
+ ? [
+ "--engine-kind",
+ SORTFORMER_DIARIZATION_ENGINE,
+ "--sortformer-bin",
+ cfg.sortformerBin,
+ "--sortformer-model",
+ cfg.sortformerModel,
+ "--backend",
+ cfg.backend,
+ "--threads",
+ String(cfg.threads),
+ // No --ffprobe: sortformer never windows, so nothing needs to ask how
+ // long the recording is. Its memory is O(1) in duration.
+ "--ffmpeg",
+ opts.paths.ffmpegBin,
+ ]
+ : [
+ "--seg",
+ cfg.segModel,
+ "--emb",
+ cfg.embModel,
+ "--threshold",
+ String(cfg.threshold),
+ "--threads",
+ String(cfg.threads),
+ "--python",
+ cfg.python,
+ // Passed explicitly rather than left to env inheritance. ffprobe is what
+ // decides whether a file is long enough to window, and a wrapper that
+ // silently fell back to a bare "ffprobe" on PATH would quietly take the
+ // whole-file path on exactly the long recordings windowing exists for.
+ "--ffmpeg",
+ opts.paths.ffmpegBin,
+ "--ffprobe",
+ opts.paths.ffprobeBin,
+ ];
+
const child = execa(
opts.paths.diarizeBin,
[
@@ -113,24 +162,7 @@ export async function diarizeOneVideo(
DIARIZATION_FILENAME,
"--video-id",
opts.videoId,
- "--seg",
- cfg.segModel,
- "--emb",
- cfg.embModel,
- "--threshold",
- String(cfg.threshold),
- "--threads",
- String(cfg.threads),
- "--python",
- cfg.python,
- // Passed explicitly rather than left to env inheritance. ffprobe is what
- // decides whether a file is long enough to window, and a wrapper that
- // silently fell back to a bare "ffprobe" on PATH would quietly take the
- // whole-file path on exactly the long recordings windowing exists for.
- "--ffmpeg",
- opts.paths.ffmpegBin,
- "--ffprobe",
- opts.paths.ffprobeBin,
+ ...engineArgs,
media,
],
{
diff --git a/common/controller/renameChannel.test.ts b/common/controller/renameChannel.test.ts
@@ -21,7 +21,6 @@ import {
readSchedulerState,
writeSchedulerState,
} from "../jobs/syncSchedulerState";
-import { readBookmarks } from "../jobs/bookmarks";
import { renameChannel } from "./renameChannel";
// Run with:
@@ -34,7 +33,6 @@ async function withPaths(fn: (paths: Paths) => Promise<void>): Promise<void> {
savedVideosDir: path.join(dir, "saved"),
sitesDir: path.join(dir, "sites"),
schedulerStateFile: path.join(dir, ".scheduler", "state.json"),
- bookmarksFile: path.join(dir, ".bookmarks", "bookmarks.json"),
} as Paths;
try {
await fn(paths);
@@ -46,8 +44,7 @@ async function withPaths(fn: (paths: Paths) => Promise<void>): Promise<void> {
const config: ChannelConfig = { handling: "youtube", name: "Old Name" };
// Seed a channel "old" with one video that has a persisted saved source, plus a
-// site membership, a scheduler backoff entry, and a bookmark — all keyed by the
-// old slug.
+// site membership and a scheduler backoff entry — all keyed by the old slug.
async function seedOld(paths: Paths): Promise<void> {
await writeChannelConfig(paths, "old", config);
const videoDir = path.join(paths.channelsDir, "old", "data", "vid1");
@@ -73,22 +70,6 @@ async function seedOld(paths: Paths): Promise<void> {
const state = await readSchedulerState(paths);
state.channels["old"] = { ...emptyChannelSyncState(), consecutiveFailures: 5 };
await writeSchedulerState(paths, state);
-
- await mkdir(path.dirname(paths.bookmarksFile), { recursive: true });
- await writeFile(
- paths.bookmarksFile,
- JSON.stringify({
- v: 1,
- bookmarks: [
- {
- id: "bm1",
- name: "whisper-all · old",
- spec: { kind: "whisper-all", slug: "old" },
- createdAt: 1,
- },
- ],
- }),
- );
}
test("renameChannel migrates every slug-keyed store", async () => {
@@ -123,11 +104,6 @@ test("renameChannel migrates every slug-keyed store", async () => {
const state = await readSchedulerState(paths);
assert.equal(state.channels["old"], undefined);
assert.equal(state.channels["new"]?.consecutiveFailures, 5);
-
- // Bookmark retargeted (auto-name regenerated).
- const bookmarks = await readBookmarks(paths);
- assert.equal(bookmarks[0]?.spec.slug, "new");
- assert.equal(bookmarks[0]?.name, "whisper-all · new");
});
});
diff --git a/common/controller/renameChannel.ts b/common/controller/renameChannel.ts
@@ -10,7 +10,6 @@ import {
readSchedulerState,
writeSchedulerState,
} from "../jobs/syncSchedulerState";
-import { renameChannelInBookmarks } from "../jobs/bookmarks";
// Rename a channel's slug. Because the slug IS the on-disk directory name
// (transcripts/channels/<slug>/), this moves the channel directory AND migrates
@@ -18,7 +17,6 @@ import { renameChannelInBookmarks } from "../jobs/bookmarks";
// - the saved-video store dir + each saved-video.json pointer's absolute `dir`
// - site.json memberships across all sites
// - the sync scheduler's per-channel backoff state
-// - job bookmarks whose spec targets this channel
//
// The two filesystem moves (channel dir, then store dir) run first and roll back
// on failure so a channel is never left half-renamed. The metadata updates that
@@ -159,12 +157,5 @@ export async function renameChannel(
);
}
- // 6. Retarget job bookmarks.
- try {
- await renameChannelInBookmarks(paths, oldSlug, newSlug);
- } catch (err) {
- warnings.push(`Bookmark migration failed: ${(err as Error).message}`);
- }
-
return { warnings };
}
diff --git a/common/jobs/bookmarks.ts b/common/jobs/bookmarks.ts
@@ -1,175 +0,0 @@
-import fs from "node:fs";
-import path from "node:path";
-import type { Paths } from "../lib/paths";
-import { parseJobSpec, type JobSpec } from "./jobSpec";
-
-// Persisted job bookmarks. A bookmark captures a JobSpec (see jobSpec.ts) plus a
-// human label, so an operator can re-launch the same job kind on the same
-// channel with one button. Re-running re-derives work from the channel's current
-// state — the bookmark never stores a frozen video-id list.
-//
-// Modeled on common/jobs/workerDefaults.ts: a small JSON file outside the
-// in-memory registry, written atomically (tmp + rename), with tolerant reads
-// that coerce a missing/corrupt file to an empty list rather than crashing.
-
-export type JobBookmark = {
- id: string;
- name: string;
- spec: JobSpec;
- createdAt: number;
- lastRunAt?: number;
-};
-
-type BookmarksFile = { v: 1; bookmarks: JobBookmark[] };
-
-// Stable label derived from the spec, e.g. "whisper-all · HasanAbiVODs3" or
-// "retry-bucket (partialDownloads) · cornbreadman".
-export function defaultBookmarkName(spec: JobSpec): string {
- const scope = spec.bucket ? `${spec.kind} (${spec.bucket})` : spec.kind;
- return `${scope} · ${spec.slug}`;
-}
-
-function newBookmarkId(): string {
- return `bm-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 8)}`;
-}
-
-// Two specs are "the same" job to avoid duplicate bookmarks when the operator
-// bookmarks the same recurring job twice. params order is stable because each
-// action builds them the same way.
-function sameSpec(a: JobSpec, b: JobSpec): boolean {
- return (
- a.kind === b.kind &&
- a.slug === b.slug &&
- a.bucket === b.bucket &&
- JSON.stringify(a.params ?? {}) === JSON.stringify(b.params ?? {})
- );
-}
-
-export async function readBookmarks(paths: Paths): Promise<JobBookmark[]> {
- let raw: unknown;
- try {
- raw = JSON.parse(await fs.promises.readFile(paths.bookmarksFile, "utf8"));
- } catch {
- return [];
- }
- if (!raw || typeof raw !== "object") return [];
- const arr = (raw as Record<string, unknown>).bookmarks;
- if (!Array.isArray(arr)) return [];
- const out: JobBookmark[] = [];
- for (const item of arr) {
- if (!item || typeof item !== "object") continue;
- const b = item as Record<string, unknown>;
- if (typeof b.id !== "string" || !b.id) continue;
- const spec = parseJobSpec(b.spec);
- if (!spec) continue;
- out.push({
- id: b.id,
- name:
- typeof b.name === "string" && b.name
- ? b.name
- : defaultBookmarkName(spec),
- spec,
- createdAt: typeof b.createdAt === "number" ? b.createdAt : 0,
- lastRunAt: typeof b.lastRunAt === "number" ? b.lastRunAt : undefined,
- });
- }
- return out;
-}
-
-async function writeBookmarks(
- paths: Paths,
- bookmarks: JobBookmark[],
-): Promise<void> {
- const out: BookmarksFile = { v: 1, bookmarks };
- await fs.promises.mkdir(path.dirname(paths.bookmarksFile), {
- recursive: true,
- });
- const tmp = `${paths.bookmarksFile}.tmp-${process.pid}`;
- await fs.promises.writeFile(tmp, JSON.stringify(out, null, 2) + "\n");
- await fs.promises.rename(tmp, paths.bookmarksFile);
-}
-
-// Add a bookmark for `spec`, returning the new (or existing duplicate) entry.
-// Idempotent: re-bookmarking an identical job returns the existing bookmark
-// without creating a second one.
-export async function addBookmark(
- paths: Paths,
- spec: JobSpec,
-): Promise<JobBookmark> {
- const existing = await readBookmarks(paths);
- const dup = existing.find((b) => sameSpec(b.spec, spec));
- if (dup) return dup;
- const bookmark: JobBookmark = {
- id: newBookmarkId(),
- name: defaultBookmarkName(spec),
- spec,
- createdAt: Date.now(),
- };
- await writeBookmarks(paths, [bookmark, ...existing]);
- return bookmark;
-}
-
-export async function removeBookmark(
- paths: Paths,
- id: string,
-): Promise<void> {
- const existing = await readBookmarks(paths);
- const next = existing.filter((b) => b.id !== id);
- if (next.length !== existing.length) await writeBookmarks(paths, next);
-}
-
-// Rewrite every bookmark whose spec targets `oldSlug` to target `newSlug`. Used
-// when a channel is renamed so its bookmarked jobs keep working. An auto-derived
-// name (still equal to the old spec's default) is regenerated for the new slug;
-// a custom name the operator set is left untouched. Returns the number changed.
-export async function renameChannelInBookmarks(
- paths: Paths,
- oldSlug: string,
- newSlug: string,
-): Promise<number> {
- const existing = await readBookmarks(paths);
- let changed = 0;
- const next = existing.map((b) => {
- if (b.spec.slug !== oldSlug) return b;
- changed++;
- const spec = { ...b.spec, slug: newSlug };
- const name =
- b.name === defaultBookmarkName(b.spec) ? defaultBookmarkName(spec) : b.name;
- return { ...b, spec, name };
- });
- if (changed > 0) await writeBookmarks(paths, next);
- return changed;
-}
-
-// Move a bookmark one slot up (dir -1) or down (dir +1) by swapping it with its
-// neighbour. Order is the array order — the same order both the management page
-// and the compact menu render — so this is the only place reordering lives.
-// No-ops at the bounds or for an unknown id.
-export async function moveBookmark(
- paths: Paths,
- id: string,
- dir: -1 | 1,
-): Promise<void> {
- const existing = await readBookmarks(paths);
- const i = existing.findIndex((b) => b.id === id);
- const j = i + dir;
- if (i < 0 || j < 0 || j >= existing.length) return;
- const next = existing.slice();
- [next[i], next[j]] = [next[j], next[i]];
- await writeBookmarks(paths, next);
-}
-
-export async function updateBookmark(
- paths: Paths,
- id: string,
- patch: Partial<Pick<JobBookmark, "name" | "lastRunAt">>,
-): Promise<void> {
- const existing = await readBookmarks(paths);
- let changed = false;
- const next = existing.map((b) => {
- if (b.id !== id) return b;
- changed = true;
- return { ...b, ...patch };
- });
- if (changed) await writeBookmarks(paths, next);
-}
diff --git a/common/jobs/jobKinds.ts b/common/jobs/jobKinds.ts
@@ -1,12 +1,12 @@
// The single source of truth for what a job `kind` IS. Before this table, the
// same per-kind knowledge was scattered across four places: the DRAINABLE_KINDS
// set (registry.ts), the JOB_KIND_LABELS map (editor jobKindLabels.ts), the
-// switch in editor runJobSpec.ts, and the spec-presence bookmark check. Phase 1
+// switch in editor runJobSpec.ts, and the spec-presence replay check. Phase 1
// of the queue refactor consolidates the drainable + label data here; later
-// phases consume `defaultTier` (scheduler) and lean on `bookmarkable`.
+// phases consume `defaultTier` (scheduler) and lean on `replayable`.
//
// Adding a new job kind should mean adding ONE entry here (plus its replay
-// handler in editor/app/jobs/jobReplayRegistry.ts if it is bookmarkable).
+// handler in editor/app/jobs/jobReplayRegistry.ts if it is replayable).
// How a kind picks its registry queueKey. Descriptive only — the real key is
// still computed by the action that creates the job (channelQueueKey /
@@ -29,9 +29,9 @@ export type JobKindMeta = {
// starting new sub-operations, lets in-flight ones finish).
drainable: boolean;
// Whether this kind's action attaches a replayable JobSpec (and thus has a
- // replay handler). Actual bookmark-ability still keys off spec PRESENCE on the
- // record at runtime — this flag just says the kind CAN be bookmarked/retried.
- bookmarkable: boolean;
+ // replay handler). Actual replayability still keys off spec PRESENCE on the
+ // record at runtime — this flag just says the kind CAN be retried.
+ replayable: boolean;
// Descriptive: how the action derives its queueKey. Not consumed as logic.
queueKeyStrategy: QueueKeyStrategy;
// Fallback scheduler tier when a record is not explicitly background. Left
@@ -46,68 +46,68 @@ const JOB_KINDS: Record<string, JobKindMeta> = {
kind: "auto-transcribe",
label: "Auto-transcribe",
drainable: true,
- bookmarkable: false,
+ replayable: false,
queueKeyStrategy: "parallel",
},
"auto-download": {
kind: "auto-download",
label: "Auto-download runner",
drainable: true,
- bookmarkable: false,
+ replayable: false,
queueKeyStrategy: "parallel",
},
"auto-download-unit": {
kind: "auto-download-unit",
label: "Auto-download",
drainable: false,
- bookmarkable: false,
+ replayable: false,
queueKeyStrategy: "platform",
},
"whisper-all": {
kind: "whisper-all",
label: "Transcribe all",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
"whisper-bucket-downloaded-no-transcript": {
kind: "whisper-bucket-downloaded-no-transcript",
label: "Transcribe downloaded audio",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
// Replace-auto-captions lane, transcribe half: whisper over videos whose only
// transcript is a YouTube ASR VTT (the downloadedAutoSubsOnly bucket). Same
// batch machinery as whisper-bucket-downloaded-no-transcript — drainable and
- // bookmarkable, re-deriving the bucket's current members on replay.
+ // replayable, re-deriving the bucket's current members on replay.
"whisper-bucket-auto-subs": {
kind: "whisper-bucket-auto-subs",
label: "Replace auto-captions",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
// Delete the superseded English ASR VTTs kept as backups next to a finished
// whisper transcript. Manual only — never auto-queued — and the single
// irreversible step in the lane, so it is deliberately NOT drainable (it is a
- // fast file sweep) but IS bookmarkable.
+ // fast file sweep) but IS replayable.
"purge-superseded-auto-subs": {
kind: "purge-superseded-auto-subs",
label: "Purge superseded auto-captions",
drainable: false,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
// AI digest sweep, local (ollama) lane — the one that carries the corpus. Both
// digest kinds are drainable (the batch honors the drain signal: it stops
- // pulling new videos and lets the in-flight one finish) and bookmarkable, since
+ // pulling new videos and lets the in-flight one finish) and replayable, since
// a channel-scoped sweep is exactly the kind of thing an operator re-launches.
"digest-channel-local": {
kind: "digest-channel-local",
label: "Digest channel (local)",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
// Same batch, metered lane. Off unless settings.digest.remoteEnabled is true,
@@ -117,17 +117,17 @@ const JOB_KINDS: Record<string, JobKindMeta> = {
kind: "digest-channel-remote",
label: "Digest channel (metered)",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
// Copy a duplicate cluster's canonical digest onto its aligned mirrors. A fast
// file operation gated by the timestamp-alignment check, so it is not drainable
- // but is bookmarkable.
+ // but is replayable.
"digest-share-cluster": {
kind: "digest-share-cluster",
label: "Share cluster digest",
drainable: false,
- bookmarkable: false,
+ replayable: false,
queueKeyStrategy: "parallel",
},
// Write the compact transcript.cues.json sidecar next to every raw transcript
@@ -138,69 +138,69 @@ const JOB_KINDS: Record<string, JobKindMeta> = {
// Not drainable: the walk honours the cancel signal (which is what stops it)
// but has no drain-aware inner loop, and the unit of work is a single file
// write, so "let the in-flight one finish" is already how it behaves. Not
- // bookmarkable either — that would need a JobSpec and a jobReplayRegistry
+ // replayable either — that would need a JobSpec and a jobReplayRegistry
// handler, and the button is one click from the card that reports the count.
"normalize-transcripts": {
kind: "normalize-transcripts",
label: "Normalize transcripts",
drainable: false,
- bookmarkable: false,
+ replayable: false,
queueKeyStrategy: "custom",
},
"redownload-incomplete-bucket": {
kind: "redownload-incomplete-bucket",
label: "Re-download truncated transcripts",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
"download-from-playlist": {
kind: "download-from-playlist",
label: "Download from playlist",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "platform",
},
"download-missing": {
kind: "download-missing",
label: "Download missing",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "platform",
},
"download-missing-subs": {
kind: "download-missing-subs",
label: "Download missing subs",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "platform",
},
"import-one": {
kind: "import-one",
label: "Import video",
drainable: false,
- bookmarkable: false,
+ replayable: false,
queueKeyStrategy: "custom",
},
"redownload-archive": {
kind: "redownload-archive",
label: "Archive source video",
drainable: false,
- bookmarkable: false,
+ replayable: false,
queueKeyStrategy: "custom",
},
"retry-bucket": {
kind: "retry-bucket",
label: "Retry",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "platform",
},
"clean-audio-transcribed": {
kind: "clean-audio-transcribed",
label: "Clean audio",
drainable: false,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
// Speaker-diarization backfill over a channel's retained audio. The capture
@@ -208,36 +208,36 @@ const JOB_KINDS: Record<string, JobKindMeta> = {
// (videos transcribed before the feature, or with the inline hook off — which
// is the recommended way to run a large batch). Drainable, because it is
// CPU-hours of work an operator will want to stop without losing what it has
- // already written, and bookmarkable, because it re-derives its work-list from
+ // already written, and replayable, because it re-derives its work-list from
// disk on every run and so replays correctly with nothing remembered.
"diarize-channel": {
kind: "diarize-channel",
label: "Diarize speakers",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
// The corpus-wide backfill sweep's orchestrator. Drainable, because a "stop"
// has to reach the channel job it is waiting on rather than meaning "after the
- // current channel", and NOT bookmarkable: the sweep is armed through a
+ // current channel", and NOT replayable: the sweep is armed through a
// persisted settings flag, so replaying a record would be a second way to
// start the same singleton.
"backfill-sweep": {
kind: "backfill-sweep",
label: "Backfill sweep",
drainable: true,
- bookmarkable: false,
+ replayable: false,
queueKeyStrategy: "parallel",
},
// One channel through the backfill lane. Drainable (the batch stops taking new
- // videos and lets the in-flight one finish) and bookmarkable, because it
+ // videos and lets the in-flight one finish) and replayable, because it
// re-derives its work-list from disk on every run and so replays correctly
// with nothing remembered — the same contract diarize-channel has.
"backfill-channel": {
kind: "backfill-channel",
label: "Backfill channel",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
// The corpus-wide corrupt-media scan and its per-channel twin. Registered
@@ -249,7 +249,7 @@ const JOB_KINDS: Record<string, JobKindMeta> = {
label: "Scan media for corruption",
// Reports only; there is nothing in flight to let finish.
drainable: false,
- bookmarkable: false,
+ replayable: false,
// Local disk work with nothing to serialize against — see the queueKey ""
// escape hatch refresh-report and detect-duplicates use.
queueKeyStrategy: "parallel",
@@ -259,7 +259,7 @@ const JOB_KINDS: Record<string, JobKindMeta> = {
kind: "scan-media-channel",
label: "Scan channel media",
drainable: false,
- bookmarkable: false,
+ replayable: false,
queueKeyStrategy: "parallel",
defaultTier: "background",
},
@@ -267,40 +267,40 @@ const JOB_KINDS: Record<string, JobKindMeta> = {
kind: "check-kept-deleted",
label: "Check kept videos",
drainable: false,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
"persist-kept": {
kind: "persist-kept",
label: "Persist kept videos",
drainable: false,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
"backup-saved-videos": {
kind: "backup-saved-videos",
label: "Back up saved videos",
drainable: false,
- bookmarkable: false,
+ replayable: false,
queueKeyStrategy: "custom",
},
"verify-saved-video-backup": {
kind: "verify-saved-video-backup",
label: "Verify saved-video backup",
drainable: false,
- bookmarkable: false,
+ replayable: false,
queueKeyStrategy: "custom",
},
// Social-post ingest for a `sourceKind: "social"` channel. Drainable (the
// fetcher stops paging on the drain signal and keeps what it already has) and
- // bookmarkable. queueKeyForUrl() routes x.com / bsky.app to
+ // replayable. queueKeyForUrl() routes x.com / bsky.app to
// `platform:x.com` / `platform:bsky.app`, so per-platform serialization and
// the existing 429 backoff come free.
"fetch-posts": {
kind: "fetch-posts",
label: "Fetch posts",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "platform",
},
// The posts analogue of the video availability check: which archived posts
@@ -309,14 +309,14 @@ const JOB_KINDS: Record<string, JobKindMeta> = {
kind: "check-post-availability",
label: "Check deleted posts",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "platform",
},
sync: {
kind: "sync",
label: "Sync",
drainable: true,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "platform",
},
// Replayable kinds that never had a JOB_KIND_LABELS entry: label omitted so
@@ -324,49 +324,49 @@ const JOB_KINDS: Record<string, JobKindMeta> = {
"store-playlist": {
kind: "store-playlist",
drainable: false,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
"clear-failed-transcriptions": {
kind: "clear-failed-transcriptions",
drainable: false,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
"transcode-failures": {
kind: "transcode-failures",
drainable: false,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
"transcode-untranscoded": {
kind: "transcode-untranscoded",
drainable: false,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
"remove-failed-transcodings": {
kind: "remove-failed-transcodings",
drainable: false,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
"clear-failed-transcodings": {
kind: "clear-failed-transcodings",
drainable: false,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
"clean-extra-audio-formats": {
kind: "clean-extra-audio-formats",
drainable: false,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
"remove-wrong-format-audio": {
kind: "remove-wrong-format-audio",
drainable: false,
- bookmarkable: true,
+ replayable: true,
queueKeyStrategy: "custom",
},
};
@@ -384,7 +384,3 @@ export function jobKindLabel(kind: string): string {
export function isDrainableKind(kind: string): boolean {
return JOB_KINDS[kind]?.drainable ?? false;
}
-
-export function isBookmarkableKind(kind: string): boolean {
- return JOB_KINDS[kind]?.bookmarkable ?? false;
-}
diff --git a/common/jobs/jobMeta.ts b/common/jobs/jobMeta.ts
@@ -22,8 +22,8 @@ export type JobMeta = {
startedAt?: number;
endedAt?: number;
exitCode?: number;
- // Replay descriptor for bookmarkable jobs, preserved so an archived (evicted
- // or post-restart) job can still be bookmarked from disk. See jobSpec.ts.
+ // Replay descriptor for replayable jobs, preserved so an archived (evicted
+ // or post-restart) job can still be retried from disk. See jobSpec.ts.
spec?: JobSpec;
};
@@ -77,7 +77,7 @@ export async function readJobMeta(
if (typeof m.id !== "string" || typeof m.kind !== "string") return null;
const meta = parsed as JobMeta;
// Sanitize the replay descriptor: drop it if malformed so a stale sidecar
- // can't surface an un-runnable bookmark.
+ // can't surface an un-runnable Retry.
meta.spec = m.spec === undefined ? undefined : (parseJobSpec(m.spec) ?? undefined);
return meta;
} catch {
diff --git a/common/jobs/jobSpec.ts b/common/jobs/jobSpec.ts
@@ -1,9 +1,10 @@
-// A serializable description of a job, sufficient to re-launch ("bookmark") it.
+// A serializable description of a job, sufficient to re-launch (replay) it.
//
// It is captured at job-creation time (stored on the JobRecord and in the
-// <id>.meta.json sidecar) and copied into a bookmark when the user bookmarks a
-// job. The dispatcher in editor/app/jobs/runJobSpec.ts maps a spec back to the
-// server action that runs it.
+// <id>.meta.json sidecar), which is what lets Retry re-run a job — including one
+// that has been evicted from the in-memory registry. The dispatcher in
+// editor/app/jobs/runJobSpec.ts maps a spec back to the server action that runs
+// it.
//
// Re-running RE-DERIVES the work from the channel's CURRENT state rather than
// replaying a frozen list: bucket jobs store only the snapshot bucket category
@@ -12,7 +13,7 @@
// now. Flag-style parameters (queue, audio format, abort-on-error…) are captured
// verbatim in `params`.
-// The snapshot bucket categories whose current members a bookmarked job can be
+// The snapshot bucket categories whose current members a replayed job can be
// re-derived from. A subset of ChannelSnapshot["buckets"] — only the buckets the
// bucket-style actions (retry-bucket / whisper-bucket) actually operate on.
export type ReplayBucket =
@@ -52,8 +53,8 @@ const REPLAY_BUCKETS: ReadonlySet<string> = new Set<ReplayBucket>([
"supersededAutoSubs",
]);
-// Defensive parse for a spec read back from JSON (a sidecar or the bookmarks
-// file). Returns null on anything malformed so a hand-edited or stale file can't
+// Defensive parse for a spec read back from JSON (the <id>.meta.json sidecar).
+// Returns null on anything malformed so a hand-edited or stale file can't
// crash a reader. Mirrors the tolerance of readJobMeta / readWorkerDefaults.
export function parseJobSpec(raw: unknown): JobSpec | null {
if (!raw || typeof raw !== "object") return null;
diff --git a/common/jobs/listJobs.ts b/common/jobs/listJobs.ts
@@ -21,9 +21,9 @@ export type JobListEntry = {
endedAt?: number;
exitCode?: number;
inRegistry: boolean;
- // True when the job carries a replay descriptor (JobSpec) and can be
- // bookmarked / re-run. See common/jobs/jobSpec.ts.
- bookmarkable: boolean;
+ // True when the job carries a replay descriptor (JobSpec) and can therefore
+ // be re-run from it (Retry). See common/jobs/jobSpec.ts.
+ replayable: boolean;
logPath: string;
logSize: number;
};
@@ -103,7 +103,7 @@ async function buildEntry(
endedAt: live_.endedAt,
exitCode: live_.exitCode,
inRegistry: true,
- bookmarkable: Boolean(live_.spec),
+ replayable: Boolean(live_.spec),
logPath,
logSize,
};
@@ -128,7 +128,7 @@ async function buildEntry(
endedAt: meta.endedAt,
exitCode: meta.exitCode,
inRegistry: false,
- bookmarkable: Boolean(meta.spec),
+ replayable: Boolean(meta.spec),
logPath,
logSize,
};
@@ -140,7 +140,7 @@ async function buildEntry(
startedAt: mtime,
endedAt: mtime,
inRegistry: false,
- bookmarkable: false,
+ replayable: false,
logPath,
logSize,
};
diff --git a/common/jobs/registry.ts b/common/jobs/registry.ts
@@ -78,8 +78,8 @@ export type JobRecord = {
queueKey: string;
channelSlug?: string;
videoId?: string;
- // A serializable replay descriptor, set for bookmarkable job kinds. Its
- // presence is what makes a job "bookmarkable" in the UI. See jobSpec.ts.
+ // A serializable replay descriptor, set for replayable job kinds. Its
+ // presence is what offers Retry on a job in the UI. See jobSpec.ts.
spec?: JobSpec;
// Background jobs queue BEHIND foreground (default) jobs on the same queueKey
// (see enqueue). The auto-download runner marks its per-video units background
diff --git a/common/jobs/streamCommand.ts b/common/jobs/streamCommand.ts
@@ -58,7 +58,7 @@ type CommonOpts = {
channelSlug?: string;
videoId?: string;
// When set, recorded on the job (and its meta sidecar) so the job can be
- // bookmarked and re-launched later. See common/jobs/jobSpec.ts.
+ // re-launched (retried) later. See common/jobs/jobSpec.ts.
spec?: JobSpec;
// Background jobs queue BEHIND any foreground (default) job on the same
// queueKey: a manually-triggered job (sync, manual download) jumps ahead of
diff --git a/common/lib/backfillKinds.test.ts b/common/lib/backfillKinds.test.ts
@@ -12,6 +12,7 @@ import {
operationCatalog,
operationLabel,
digestLaneFor,
+ diarizationLaneFor,
laneYieldsToTranscription,
addBackfillState,
emptyBackfillCounts,
@@ -40,6 +41,8 @@ import {
import {
DEFAULT_DIARIZATION_THRESHOLD,
DIARIZATION_FILENAME,
+ SORTFORMER_DIARIZATION_ENGINE,
+ diarizationTarget,
isDiarizationFresh,
type DiarizationRecord,
} from "./diarization";
@@ -245,6 +248,123 @@ test("a different engine binary is not, by itself, stale", async () => {
);
});
+// The engine-as-a-setting case the test above says needs "no new logic". It is
+// asserted for sortformer and NOT for the default, and that asymmetry is the
+// whole design: the default engine still cannot predict which binary a
+// `--engine` override runs, so it must keep asserting nothing.
+test("selecting sortformer asserts the engine; selecting the default still does not", () => {
+ const rec = (engine: DiarizationRecord["engine"]): DiarizationRecord => ({
+ videoId: "v",
+ generatedAt: "now",
+ speakers: 1,
+ turns: [],
+ engine,
+ });
+ const sherpaSidecar = rec({
+ engine: "sherpa-onnx",
+ segmentationModel: "seg-1.onnx",
+ embeddingModel: "emb-1.onnx",
+ threshold: DEFAULT_DIARIZATION_THRESHOLD,
+ });
+ const sortformerSidecar = rec({
+ engine: SORTFORMER_DIARIZATION_ENGINE,
+ model: "sortformer-4spk.gguf",
+ });
+
+ const sortformerTarget = diarizationTarget({
+ engine: SORTFORMER_DIARIZATION_ENGINE,
+ sortformerModel: "/abs/path/sortformer-4spk.gguf",
+ // Still configured, because settings carry one set of fields for both
+ // engines. None of it may leak into the sortformer identity.
+ segModel: "/abs/seg-1.onnx",
+ embModel: "/abs/emb-1.onnx",
+ threshold: DEFAULT_DIARIZATION_THRESHOLD,
+ });
+
+ // Basename only, as everywhere else: the corpus is rsynced between shards.
+ assert.equal(sortformerTarget.model, "sortformer-4spk.gguf");
+ assert.equal(sortformerTarget.segmentationModel, undefined);
+ assert.equal(sortformerTarget.embeddingModel, undefined);
+
+ assert.equal(isDiarizationFresh(sortformerSidecar, sortformerTarget), true);
+ // The point of switching: every sherpa sidecar becomes work the backfill lane
+ // will offer to redo, rather than silently staying half a corpus.
+ assert.equal(isDiarizationFresh(sherpaSidecar, sortformerTarget), false);
+
+ // And back the other way, with no special case needed — a sortformer record
+ // carries neither segmentation nor embedding model, so it fails the sherpa
+ // comparison on the models alone.
+ const sherpaTarget = diarizationTarget({
+ engine: "sherpa-onnx",
+ segModel: "/abs/seg-1.onnx",
+ embModel: "/abs/emb-1.onnx",
+ threshold: DEFAULT_DIARIZATION_THRESHOLD,
+ });
+ assert.equal(sherpaTarget.engine, undefined);
+ assert.equal(isDiarizationFresh(sherpaSidecar, sherpaTarget), true);
+ assert.equal(isDiarizationFresh(sortformerSidecar, sherpaTarget), false);
+});
+
+// Sortformer has no clustering step, so the threshold cannot have changed any of
+// its turns. Letting it into the identity would regenerate the whole corpus for
+// an edit that provably could not affect it.
+test("the clustering threshold does not stale a sortformer sidecar", () => {
+ const sidecar: DiarizationRecord = {
+ videoId: "v",
+ generatedAt: "now",
+ speakers: 1,
+ turns: [],
+ engine: { engine: SORTFORMER_DIARIZATION_ENGINE, model: "m.gguf" },
+ };
+ const at = (threshold: number) =>
+ diarizationTarget({
+ engine: SORTFORMER_DIARIZATION_ENGINE,
+ sortformerModel: "m.gguf",
+ threshold,
+ });
+ assert.equal(isDiarizationFresh(sidecar, at(DEFAULT_DIARIZATION_THRESHOLD)), true);
+ assert.equal(isDiarizationFresh(sidecar, at(0.4)), true);
+ // The model itself IS the identity, though — a different one is a redo.
+ assert.equal(
+ isDiarizationFresh(
+ sidecar,
+ diarizationTarget({
+ engine: SORTFORMER_DIARIZATION_ENGINE,
+ sortformerModel: "other.gguf",
+ }),
+ ),
+ false,
+ );
+});
+
+// Which resource diarization competes for is a CONFIGURATION outcome, and the
+// backfill lane reads it to decide whether a guaranteed share is even available.
+// Getting this wrong in the permissive direction puts ~4.4 GB of sortformer next
+// to parakeet on an 8 GB card.
+test("diarization contends for the GPU only as sortformer on vulkan", () => {
+ const lane = (engine: "sherpa-onnx" | "sortformer", backend: "vulkan" | "cpu") =>
+ diarizationLaneFor({ engine, backend });
+
+ assert.equal(lane("sortformer", "vulkan").contendsFor, "gpu");
+ assert.equal(laneYieldsToTranscription(lane("sortformer", "vulkan")), true);
+
+ // The same engine on the CPU backend is only after cores.
+ assert.equal(lane("sortformer", "cpu").contendsFor, "cpu");
+ assert.equal(laneYieldsToTranscription(lane("sortformer", "cpu")), false);
+
+ // sherpa-onnx is ONNX/CPU, so a stale `backend: vulkan` left in settings must
+ // NOT make it claim the card — that would park the lane behind transcription
+ // for work using no shaders at all, which is the bug digestYield.ts already
+ // records having hit once.
+ assert.equal(lane("sherpa-onnx", "vulkan").contendsFor, "cpu");
+ assert.equal(laneYieldsToTranscription(lane("sherpa-onnx", "vulkan")), false);
+
+ // The queue key never changes: two diarizations must not run at once whichever
+ // engine is selected.
+ assert.equal(lane("sortformer", "vulkan").queueKey, BACKFILL_QUEUE);
+ assert.equal(lane("sherpa-onnx", "cpu").queueKey, BACKFILL_QUEUE);
+});
+
// THE COMPATIBILITY RULE, and the reason it is written down: without it, adding
// a field to the provenance would mark all 77,000 videos stale at once.
test("an absent recorded field compares equal to today's default", async () => {
diff --git a/common/lib/backfillKinds.ts b/common/lib/backfillKinds.ts
@@ -73,8 +73,11 @@ import {
} from "./digest";
import { loadDigest } from "./digest-server";
import {
+ SORTFORMER_DIARIZATION_ENGINE,
diarizationTarget,
isDiarizationFresh,
+ type DiarizationBackend,
+ type DiarizationEngineId,
type DiarizationFreshnessTarget,
} from "./diarization";
import { loadDiarization } from "./diarization-server";
@@ -259,7 +262,19 @@ export type BackfillKind = {
tier: BackfillCostTier;
// Where this operation's work runs. See BackfillLane — the queue key is what
// keeps CPU and GPU operations overlapping instead of taking turns.
+ //
+ // This is the DECLARED lane, which for a kind with a choice means its default.
+ // Prefer laneFor() when a live answer is needed.
lane: BackfillLane;
+ // The lane this kind would actually use under the given settings, for the
+ // kinds whose scarce resource is a configuration choice rather than a fact.
+ // Diarization is one: sherpa-onnx is CPU-only, while sortformer on the Vulkan
+ // backend holds ~4.4 GB of the same 8 GB card the transcription engine wants.
+ //
+ // Optional because most kinds have no choice, and `lane` is the answer for
+ // them. Mirrors digestLaneFor, which solved the same problem for the digest
+ // operation's two lanes.
+ laneFor?(settings: SiteSettings): BackfillLane;
// Ids of other kinds in this table whose output this one consumes.
//
// PURELY DECLARATIVE. It does not gate anything by itself — a kind still
@@ -315,7 +330,12 @@ const diarization: BackfillKind = {
// kinds on purpose — two channels' worth of diarization at once just thrashes
// cores. How much of the machine it may take is backfillLimit()'s question,
// not the queue's.
+ // The DEFAULT engine's answer, written out rather than derived: settings.ts is
+ // a TYPE-only import here, and pulling defaultDiarization() in as a value would
+ // make this module's initialization depend on it at runtime. laneFor is the
+ // live answer, and diarizationLaneFor is where the rule actually lives.
lane: { queueKey: BACKFILL_QUEUE, contendsFor: "cpu" },
+ laneFor: (settings) => diarizationLaneFor(settings.diarization),
enabled: (settings) =>
settings.diarization.enabled &&
!!settings.diarization.segModel &&
@@ -808,6 +828,28 @@ export function digestLaneFor(appLane: DigestLane): BackfillLane {
{ queueKey: DIGEST_REMOTE_QUEUE, contendsFor: "network" };
}
+// Which resource a diarization run competes for, which follows from the engine
+// it is configured with rather than from a separate setting — the same shape as
+// digestLaneFor, and stated here so nothing has to re-test the engine id.
+//
+// The queue key does NOT change with the engine. Diarization serializes against
+// itself either way, and giving the GPU variant its own key would only let two
+// diarizations run at once — which is precisely what must not happen when each
+// holds ~4.4 GB of an 8 GB card.
+export function diarizationLaneFor(diarization: {
+ engine: DiarizationEngineId;
+ backend: DiarizationBackend;
+}): BackfillLane {
+ return diarization.engine === SORTFORMER_DIARIZATION_ENGINE &&
+ diarization.backend === "vulkan"
+ ? // Competes with the transcription engine for the same VRAM. Yields.
+ { queueKey: BACKFILL_QUEUE, contendsFor: "gpu" }
+ : // sherpa-onnx is ONNX/CPU, and sortformer on the CPU backend is likewise
+ // only after cores. Contends for CPU, whose share the backfill lane's own
+ // weight already governs.
+ { queueKey: BACKFILL_QUEUE, contendsFor: "cpu" };
+}
+
// Whether a lane must stand aside while transcription is working. One rule, so
// the digest lane and any future GPU operation cannot answer it differently.
export function laneYieldsToTranscription(lane: BackfillLane): boolean {
diff --git a/common/lib/diarization.ts b/common/lib/diarization.ts
@@ -35,6 +35,12 @@ export type DiarizationEngine = {
// is machine-specific and would make records non-portable across shards).
segmentationModel?: string;
embeddingModel?: string;
+ // The single-model engines' model, kept separate from the segmentation/
+ // embedding PAIR above rather than overloading it. Sortformer is one GGUF and
+ // has no embedding model at all, and writing it into `segmentationModel` would
+ // make a sortformer record compare equal to a sherpa one that happened to share
+ // a basename.
+ model?: string;
// Engine/library version string, when the engine reports one.
version?: string;
// Clustering threshold used. The single most consequential knob: it decides
@@ -81,6 +87,31 @@ export const DIARIZATION_FILENAME = "diarization.json";
// freshness comparator below and the wrapper agree on one spelling.
export const DEFAULT_DIARIZATION_ENGINE = "sherpa-onnx";
+// The ggml/Sortformer engine (scripts/build-sortformer.sh, scripts/
+// diarize-sortformer.mjs). End-to-end rather than clustered: no segmentation +
+// embedding pair, no threshold, a hard ceiling of 4 speakers, and — because its
+// state is a fixed-size speaker cache — memory that is O(1) in recording length
+// rather than O(n^2) in segment count.
+export const SORTFORMER_DIARIZATION_ENGINE = "sortformer";
+
+export type DiarizationEngineId =
+ | typeof DEFAULT_DIARIZATION_ENGINE
+ | typeof SORTFORMER_DIARIZATION_ENGINE;
+
+// Compute device for engines that have a choice. Only sortformer does; sherpa is
+// ONNX/CPU here (no Vulkan compute path on Linux for onnxruntime).
+export type DiarizationBackend = "vulkan" | "cpu";
+
+export const DIARIZATION_ENGINE_IDS: readonly DiarizationEngineId[] = [
+ DEFAULT_DIARIZATION_ENGINE,
+ SORTFORMER_DIARIZATION_ENGINE,
+];
+
+export const DIARIZATION_BACKENDS: readonly DiarizationBackend[] = [
+ "vulkan",
+ "cpu",
+];
+
// The clustering-threshold default, measured on this corpus (see
// DiarizationSettings.threshold for the sweep that produced it). It lives here
// rather than only in settings.ts because isDiarizationFresh needs it to
@@ -93,25 +124,30 @@ export const DEFAULT_DIARIZATION_THRESHOLD = 0.9;
// idea: a sidecar is stale when its recorded provenance differs from this, not
// when it is old.
export type DiarizationFreshnessTarget = {
- // OPTIONAL, and compared only when present — which today means never.
+ // OPTIONAL, and compared only when present — which now means "only when the
+ // configured engine is not the default".
//
- // The engine is not a setting: scripts/diarize.mjs records whatever binary
- // actually ran (its own default, or the basename of a `--engine` /
- // DIARIZE_ENGINE_CMD replacement), and nothing in DiarizationSettings can say
- // which that will be. Asserting a hardcoded "sherpa-onnx" here would mark
- // every sidecar produced by any other wrapper permanently stale — an infinite
- // regeneration loop at ~500-680 s/audio-hour, and one the e2e fake engine
- // would trip on its first run.
+ // The asymmetry is deliberate and load-bearing. scripts/diarize.mjs records
+ // whatever actually ran, including the basename of a `--engine` /
+ // DIARIZE_ENGINE_CMD replacement, so asserting a hardcoded "sherpa-onnx" here
+ // would mark every sidecar from any other wrapper permanently stale — an
+ // infinite regeneration loop at ~500-680 s/audio-hour, and one the e2e fake
+ // engine trips on its first run. So the default engine still asserts nothing
+ // and compares exactly as it always did.
//
- // So the comparison is restricted to what the configuration can genuinely
- // predict: the models and the threshold, which are also the knobs that
- // actually change the output. The field stays here, and the comparison stays
- // written, so that adding an engine setting later needs no new logic.
+ // Selecting `sortformer` IS a configuration statement, and there the engine is
+ // asserted: sherpa sidecars become stale and the backfill lane offers to redo
+ // them, which is the point — the two engines disagree about how many speakers
+ // exist, and a corpus half-diarized by each is not one corpus. Switching back
+ // needs no special case: a sortformer record carries neither segmentation nor
+ // embedding model, so it fails the sherpa comparison on the models anyway.
engine?: string;
// Basenames, as DiarizationEngine records them — full paths are
// machine-specific and would make every record stale on another shard.
segmentationModel?: string;
embeddingModel?: string;
+ // Single-model engines. See DiarizationEngine.model.
+ model?: string;
threshold?: number;
};
@@ -132,14 +168,27 @@ function baseName(p: string): string {
// guards disagreeing about the identity is how a corpus ends up either
// regenerating forever or never.
export function diarizationTarget(cfg: {
+ engine?: DiarizationEngineId;
segModel?: string;
embModel?: string;
+ sortformerModel?: string;
threshold?: number;
}): DiarizationFreshnessTarget {
+ // Sortformer is end-to-end: one model, no segmentation/embedding pair, and no
+ // clustering threshold. Comparing sherpa's knobs against it would mark every
+ // sortformer sidecar stale on a threshold edit that could not have changed a
+ // single one of its turns.
+ if (cfg.engine === SORTFORMER_DIARIZATION_ENGINE) {
+ return {
+ engine: SORTFORMER_DIARIZATION_ENGINE,
+ ...(cfg.sortformerModel ? { model: baseName(cfg.sortformerModel) } : {}),
+ };
+ }
return {
- // `engine` is deliberately NOT set — see DiarizationFreshnessTarget. Settings
- // cannot know which binary will run, so claiming to compare it would mark
- // every sidecar from any other wrapper permanently stale.
+ // `engine` is deliberately NOT set for the default engine — see
+ // DiarizationFreshnessTarget. Settings cannot know which binary a `--engine`
+ // override will run, so claiming to compare it would mark every sidecar from
+ // any other wrapper permanently stale.
...(cfg.segModel ? { segmentationModel: baseName(cfg.segModel) } : {}),
...(cfg.embModel ? { embeddingModel: baseName(cfg.embModel) } : {}),
threshold: cfg.threshold ?? DEFAULT_DIARIZATION_THRESHOLD,
@@ -184,6 +233,7 @@ export function isDiarizationFresh(
(target.engine === undefined || e.engine === target.engine) &&
sameModel(e.segmentationModel, target.segmentationModel) &&
sameModel(e.embeddingModel, target.embeddingModel) &&
+ sameModel(e.model, target.model) &&
sameThreshold(e.threshold, target.threshold)
);
}
diff --git a/common/lib/paths.ts b/common/lib/paths.ts
@@ -44,10 +44,11 @@ export type Paths = {
// disabled). Written by the Workers page "Set as default" button; applied by
// the worker pool on first use. See common/jobs/workerDefaults.ts.
workerDefaultsFile: string;
- // Persisted job bookmarks: saved (kind, channel, params) specs an operator can
- // re-launch with one button. Survives restarts, unlike the in-memory job
- // registry. See common/jobs/bookmarks.ts.
- bookmarksFile: string;
+ // Saved monitor-widget presets: named /widget query strings the builder and
+ // the in-widget menu both load from. A preset stores the LINK, not a parsed
+ // config, so a preset and a shared link are literally the same artifact. See
+ // common/lib/widgetPresets.ts.
+ widgetPresetsFile: string;
// The two CHANGELOG.md files the "cut release" flow reads and rewrites. They
// are TRACKED source files, and cutting a release optionally makes a real git
// commit — so e2e must be able to point them somewhere disposable. Overridable
@@ -175,7 +176,7 @@ export function getPaths(): Paths {
schedulerStateFile: path.join(transcriptsDir, ".scheduler", "state.json"),
autoQueueStateFile: path.join(transcriptsDir, ".auto-queue", "state.json"),
workerDefaultsFile: path.join(transcriptsDir, ".workers", "defaults.json"),
- bookmarksFile: path.join(transcriptsDir, ".bookmarks", "bookmarks.json"),
+ widgetPresetsFile: path.join(transcriptsDir, ".widget", "presets.json"),
lmdbPath: path.join(transcriptsDir, "index.mdb"),
exportDir,
exportPublicDir,
diff --git a/common/lib/settings.ts b/common/lib/settings.ts
@@ -24,7 +24,14 @@ import {
defaultAutoQueue,
sanitizeAutoQueue,
} from "../jobs/autoQueuePolicy";
-import { DEFAULT_DIARIZATION_THRESHOLD } from "./diarization";
+import {
+ DEFAULT_DIARIZATION_ENGINE,
+ DEFAULT_DIARIZATION_THRESHOLD,
+ DIARIZATION_BACKENDS,
+ DIARIZATION_ENGINE_IDS,
+ type DiarizationBackend,
+ type DiarizationEngineId,
+} from "./diarization";
import { ATTRIBUTION_PROMPT_VERSION } from "./attribution";
import {
DEFAULT_COOKIE_MODE,
@@ -358,6 +365,33 @@ export type DiarizationSettings = {
threshold: number;
// Engine threads per diarize run.
threads: number;
+ // Which engine runs. "sherpa-onnx" is the shipped default and what every
+ // sidecar on disk was produced by; "sortformer" is the ggml engine built by
+ // scripts/build-sortformer.sh.
+ //
+ // CHANGING THIS RESTATES THE FRESHNESS IDENTITY (see diarizationTarget), so
+ // every sidecar written by the other engine becomes stale and the backfill lane
+ // offers to redo it. That is intended — the two disagree about how many
+ // speakers exist, and a corpus half-diarized by each is not one corpus — but on
+ // the retained audio it is weeks of work, not a toggle.
+ //
+ // Why anyone would: on the same file, sherpa at its tuned threshold returns 13
+ // speakers and sortformer returns 4, agreeing on the dominant speaker's share
+ // to within half a point (73.1% vs 73.5%). On the corpus's worst case sherpa
+ // returns 35 and sortformer 4. Over-splitting is the failure mode this lane has
+ // always had, and sortformer is end-to-end rather than clustered, so it does
+ // not have it. The cost is a hard ceiling of 4 speakers and ~1.8x the wall
+ // clock.
+ engine: DiarizationEngineId;
+ // Compute device for the sortformer engine; ignored by sherpa-onnx, which has
+ // no Vulkan compute path on Linux.
+ //
+ // "vulkan" is 1.5x faster than a thread-tuned CPU run (894 vs 1305
+ // s/audio-hour, measured on this box) and holds 558 MB resident instead of
+ // 4.84 GB by keeping weights and activations in VRAM. It also takes ~4.4 GB of
+ // an 8 GB card, which is why the lane YIELDS to transcription rather than
+ // sharing — see controller/digestYield.ts.
+ backend: DiarizationBackend;
// Python interpreter for the default sherpa-onnx engine. sherpa-onnx ships
// wheels only up to cp313, and this box's system python is 3.14 — so this
// usually points at a dedicated venv rather than `python3`.
@@ -366,6 +400,11 @@ export type DiarizationSettings = {
// is reported as a skip rather than a failure.
segModel: string;
embModel: string;
+ // Binary and model for the sortformer engine, both produced by
+ // scripts/build-sortformer.sh. Empty = that engine cannot run, reported as the
+ // same "not-configured" skip as an unset segModel/embModel.
+ sortformerBin: string;
+ sortformerModel: string;
// How many diarize runs may execute at once in the backfill pass. Kept low by
// default: diarization is CPU-bound and competes with GPU feeding and the
// digest sweep for the same 8 threads.
@@ -1111,9 +1150,17 @@ export function defaultDiarization(): DiarizationSettings {
// and the comparator from drifting apart.
threshold: DEFAULT_DIARIZATION_THRESHOLD,
threads: 4,
+ // The engine every sidecar on disk was produced by. Switching is an explicit
+ // decision that restates the freshness identity — see DiarizationSettings.
+ engine: DEFAULT_DIARIZATION_ENGINE,
+ // Only consulted when engine is "sortformer". Defaulting to the GPU is safe
+ // because the lane yields the card to transcription rather than sharing it.
+ backend: "vulkan",
python: "python3",
segModel: "",
embModel: "",
+ sortformerBin: "",
+ sortformerModel: "",
concurrency: 1,
// OFF, because windowing made it unnecessary — which is what it was always
// for. It shipped at 4 hours as a stopgap while long recordings were being
@@ -1141,9 +1188,20 @@ export function sanitizeDiarization(value: unknown): DiarizationSettings {
? r.threshold
: d.threshold,
threads: clampPositiveInt(r.threads, d.threads, 64),
+ // An unknown engine falls back to the default rather than disabling the lane:
+ // a typo in settings.json must not silently stop diarization, and the default
+ // is the one every existing sidecar already matches.
+ engine: DIARIZATION_ENGINE_IDS.includes(r.engine as DiarizationEngineId)
+ ? (r.engine as DiarizationEngineId)
+ : d.engine,
+ backend: DIARIZATION_BACKENDS.includes(r.backend as DiarizationBackend)
+ ? (r.backend as DiarizationBackend)
+ : d.backend,
python: str(r.python, d.python),
segModel: str(r.segModel, d.segModel),
embModel: str(r.embModel, d.embModel),
+ sortformerBin: str(r.sortformerBin, d.sortformerBin),
+ sortformerModel: str(r.sortformerModel, d.sortformerModel),
concurrency: clampPositiveInt(r.concurrency, d.concurrency, 16),
// 0 is meaningful here (cap off), so this cannot use clampPositiveInt.
// Fractional hours are allowed — the knob is a duration, not a count.
diff --git a/common/lib/widgetPresets.ts b/common/lib/widgetPresets.ts
@@ -0,0 +1,177 @@
+import fs from "node:fs";
+import path from "node:path";
+import type { Paths } from "./paths";
+
+// Persisted monitor-widget presets: named arrangements the /widget/builder board
+// and the in-widget menu both load from, so a layout you liked stops being
+// something you have to keep a URL of.
+//
+// A PRESET STORES THE QUERY STRING, not a parsed config. That is the whole
+// design decision here:
+// - a preset and a shared /widget link are then literally the same artifact,
+// so "save this link" and "save this preset" cannot mean different things;
+// - loading one re-runs parseWidgetConfig, which already normalizes, clamps
+// and drops unknown codes — so a preset written by an older release survives
+// a registry change for free, exactly as an old link does;
+// - nothing in this file has to know what a section or a row IS, so adding one
+// never touches the store.
+//
+// The store is a small JSON file living outside any in-memory registry, written
+// atomically (tmp + rename), with tolerant reads that coerce a missing/corrupt
+// file to an empty list rather than crashing.
+
+export type WidgetPreset = {
+ id: string;
+ name: string;
+ // The query string buildWidgetQuery produced, WITHOUT a leading "?". Empty
+ // string is legal and means the all-default widget.
+ query: string;
+ createdAt: number;
+};
+
+type PresetsFile = { v: 1; presets: WidgetPreset[] };
+
+// Names are what the operator picks a preset by, so they are trimmed and capped
+// but otherwise left alone (spaces and punctuation included).
+export const MAX_PRESET_NAME = 60;
+
+export function normalizePresetName(name: string): string {
+ return name.trim().slice(0, MAX_PRESET_NAME);
+}
+
+function newPresetId(): string {
+ return `wp-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 8)}`;
+}
+
+export async function readWidgetPresets(paths: Paths): Promise<WidgetPreset[]> {
+ let raw: unknown;
+ try {
+ raw = JSON.parse(await fs.promises.readFile(paths.widgetPresetsFile, "utf8"));
+ } catch {
+ return [];
+ }
+ if (!raw || typeof raw !== "object") return [];
+ const arr = (raw as Record<string, unknown>).presets;
+ if (!Array.isArray(arr)) return [];
+ const out: WidgetPreset[] = [];
+ for (const item of arr) {
+ if (!item || typeof item !== "object") continue;
+ const p = item as Record<string, unknown>;
+ if (typeof p.id !== "string" || !p.id) continue;
+ if (typeof p.query !== "string") continue;
+ const name = typeof p.name === "string" ? normalizePresetName(p.name) : "";
+ if (!name) continue;
+ out.push({
+ id: p.id,
+ name,
+ // Tolerate a stored value that kept its "?" — it is still a link.
+ query: p.query.replace(/^\?/, ""),
+ createdAt: typeof p.createdAt === "number" ? p.createdAt : 0,
+ });
+ }
+ return out;
+}
+
+async function writePresets(
+ paths: Paths,
+ presets: WidgetPreset[],
+): Promise<void> {
+ const out: PresetsFile = { v: 1, presets };
+ await fs.promises.mkdir(path.dirname(paths.widgetPresetsFile), {
+ recursive: true,
+ });
+ const tmp = `${paths.widgetPresetsFile}.tmp-${process.pid}`;
+ await fs.promises.writeFile(tmp, JSON.stringify(out, null, 2) + "\n");
+ await fs.promises.rename(tmp, paths.widgetPresetsFile);
+}
+
+// Save a preset under `name`. Saving over an existing name OVERWRITES its query
+// rather than making a second entry with the same label — "Save" and "Save
+// as… (an existing name)" then mean the same thing, which is what an operator
+// expects of a named slot.
+export async function addWidgetPreset(
+ paths: Paths,
+ name: string,
+ query: string,
+): Promise<WidgetPreset | null> {
+ const clean = normalizePresetName(name);
+ if (!clean) return null;
+ const existing = await readWidgetPresets(paths);
+ const q = query.replace(/^\?/, "");
+ const at = existing.findIndex((p) => p.name === clean);
+ if (at !== -1) {
+ const next = existing.slice();
+ next[at] = { ...next[at], query: q };
+ await writePresets(paths, next);
+ return next[at];
+ }
+ const preset: WidgetPreset = {
+ id: newPresetId(),
+ name: clean,
+ query: q,
+ createdAt: Date.now(),
+ };
+ await writePresets(paths, [...existing, preset]);
+ return preset;
+}
+
+export async function renameWidgetPreset(
+ paths: Paths,
+ id: string,
+ name: string,
+): Promise<boolean> {
+ const clean = normalizePresetName(name);
+ if (!clean) return false;
+ const existing = await readWidgetPresets(paths);
+ let changed = false;
+ const next = existing.map((p) => {
+ if (p.id !== id) return p;
+ changed = true;
+ return { ...p, name: clean };
+ });
+ if (changed) await writePresets(paths, next);
+ return changed;
+}
+
+export async function removeWidgetPreset(
+ paths: Paths,
+ id: string,
+): Promise<void> {
+ const existing = await readWidgetPresets(paths);
+ const next = existing.filter((p) => p.id !== id);
+ if (next.length !== existing.length) await writePresets(paths, next);
+}
+
+// Move a preset one slot up (dir -1) or down (dir +1) by swapping it with its
+// neighbour. Order is the array order — the order both the builder row and the
+// in-widget menu render — so this is the only place reordering lives. No-ops at
+// the bounds or for an unknown id.
+export async function moveWidgetPreset(
+ paths: Paths,
+ id: string,
+ dir: -1 | 1,
+): Promise<void> {
+ const existing = await readWidgetPresets(paths);
+ const i = existing.findIndex((p) => p.id === id);
+ const j = i + dir;
+ if (i < 0 || j < 0 || j >= existing.length) return;
+ const next = existing.slice();
+ [next[i], next[j]] = [next[j], next[i]];
+ await writePresets(paths, next);
+}
+
+// Resolve a `?preset=` value: by id first, then by name (case-insensitively), so
+// a hand-written link can say what it means. Returns null for an unknown one,
+// which the caller renders as "just the rest of the params".
+export function findWidgetPreset(
+ presets: WidgetPreset[],
+ ref: string,
+): WidgetPreset | null {
+ const needle = ref.trim();
+ if (!needle) return null;
+ return (
+ presets.find((p) => p.id === needle) ??
+ presets.find((p) => p.name.toLowerCase() === needle.toLowerCase()) ??
+ null
+ );
+}
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -3,7 +3,19 @@
## [Unreleased]
- **A channel page now opens on the channel's whole lifecycle instead of a viewport of video rows.** The first screen is a single transit line — Playlist → Download → Transcode → Transcribe → Digest → Backfill — where each station carries its count and its coverage, and the gap between two stations carries the shortfall, because the gap *is* the work. Only the largest shortfall is emphasised, so the bottleneck is the thing your eye lands on. Work the lane can actually do today sits on the line; everything that left it (needs cookies, deleted, untranscribable, waiting on a transcript, needs media re-acquired) hangs below it in muted type, on a different axis, so the two can never be read as one number. Below the line: one brand-coloured **next action** that disappears when there is nothing to do, a strip of anomaly chips that says "All clear" rather than vanishing, and the stage panels — **one at a time**, chosen by a tab strip and carried in the URL as `?stage=`, so a stage is a link you can share. With no stage selected you get an overview listing every stage's status at once. Previously the page stacked a full-height video browser above eight simultaneously-expanded stage panels, which put the channel's actual pipeline a full screen and a collapsed `<details>` out of reach.
- **The video browser has moved to its own page and no longer builds a DOM node per video.** `/channels/<slug>/videos` is a two-pane workspace: a virtualized list on the left, the selected video's detail on the right, both always visible. The largest channel has ~11,000 videos and the old list rendered every one of them into the page; it now renders the twenty or so you can see. Filters, search and `?video=` selection work exactly as before. The old collapse toggles are gone — they existed only because the pane was squeezed onto a page it shared with everything else, and a page of its own removes the reason for them.
+- **Command bookmarks are gone.** Saving a job under a label and re-launching it from the strip above **Jobs** / **Active jobs**, and the `/jobs/bookmarks` page that managed those labels, have all been removed. The feature was used for three days after it shipped and never again: every job it could re-launch is still one click away from its own channel-page control, and **Retry** — which re-runs a finished job straight from the descriptor stored with it — covers re-running something you have already run. Retry and **Retry all failed** are untouched and keep working exactly as before, including on jobs old enough to have been evicted from memory. Bucket jobs still re-derive their work from the channel's current state on every re-run rather than replaying a stale list. Your existing `transcripts/.bookmarks/bookmarks.json` is left on disk as a record; nothing reads it any more, and you can delete it whenever you like.
+- **The monitor widget lays out on a real grid now: rows, full-width sections, and a rail of totals.** The board could only ever split the widget into columns of stacked strips, so three things people kept asking for were impossible: a Controls row across the whole top, four one-line totals reading as one instrument cluster instead of four separate boxes, and a section that takes the leftover height. The floorplan now has rows as well as columns — drag the edge of a cell to make it span two tracks, and flip a cell to a **rail** to put its strips on one line with shortened labels (`synced 3m`, `412 GB free`, `1.2k backfill`). Every widget link written before this keeps working and still copies out in exactly the same short form; only an arrangement that genuinely needs rows writes the newer parameter.
+- **A row can now take the leftover height, and the sections in it scroll under their own headings.** The rail down the left of the board sets each row to *content height* (what every row was before) or *fill*. A widget with a fill row becomes exactly as tall as the window or iframe it is in, and the sections sharing that row each get half of it and scroll inside — with the section title pinned at the top so you always know what you are looking at. Those lists also drop the six-row cap while they are scrollable, because a list you can scroll that still says "+14 more channels" is not much of a list.
+- **Widget arrangements can be saved and named.** There was no way to keep one except saving the URL. Presets now sit above the board on the builder page and inside the widget's own settings, with **Save / Save as… / Rename / Delete**, and five ready-made ones: **Glance** (one dense rail for a 320×120 corner pin), **Jobs** (the default), **Cockpit** (controls, totals, then the scheduler beside two scrolling worklists), **Cleanup** (a full-height needs-cleaning list), and **Wall** (three columns for a spare monitor). A preset is just a widget link under a name, so `/widget?preset=cockpit` works too, and anything you spell out in the URL still wins over the preset.
+- **The widget's settings gear now opens the same board the builder page has.** It used to open a second, narrower list form that could do less — you could reorder strips but not really arrange them. There is one configuration surface now, folding to fit the ~320px overlay, so a preset behaves identically wherever you open it.
+- **The widget can hold the backfill lane.** Its controls row had Pause transcriptions, Pause downloads, Drain and Retry, but nothing for a backfill — the one job that runs for days and pins cores on a desktop somebody is sitting at. **Pause Backfill** is now there beside the other two, holding the lane without ending anything (a running job idles and keeps its place; nothing is re-derived on resume), and it writes the same setting the Settings page and the dashboard do. It appears only when a backfill feature is actually switched on.
+- **Fixed: `/widget?backfill=1` rendered an empty widget.** The backfill strip reads the same data as the last-sync and scheduler readouts, but the widget only fetched that data when one of *those two* was switched on — so asking for just the backfill strip produced nothing at all, with no hint that the section was fine and the fetch was the problem.
+- **Fixed: a widget with Active jobs switched off lost its disk indicator too.** Free-disk numbers arrive with the active-jobs data, and that fetch was gated on the Active jobs section alone, so `jobs=0` silently took the disk strip with it however clearly you had asked for it.
- **The widget's "+N more channels" is now a control instead of dead text.** Both the monitor widget's Needs-work and Needs-cleaning lists show six channels and then said how many more there were, with no way to see them — even though the widget already had every channel in hand. That line is now a button: it expands the full list inside its own scrollable box, so a pinned window shows the whole worklist without stretching to the height of the corpus, and **Show fewer** puts it back.
+- **Speaker capture can now run on the graphics card, and stops inventing speakers that were never there.** The old engine works by grouping voices it thinks sound alike, and it splits far too eagerly: on a 13-minute reaction video with one host it found **13 speakers**, and on the worst video in the corpus it found **35**. The new engine decides speaker turns directly instead of grouping them afterwards, and returns **4** in both cases — agreeing with the old one about how much of the video the main speaker talks for (73.5% against 73.1%) while collapsing the invented tail. It has no threshold to tune, and it caps at 4 speakers, so a panel of five will merge two of them rather than split one into twenty. Run `scripts/build-sortformer.sh`, then pick **Engine → Sortformer** under Settings → Diarization and paste the two paths it prints. The previous engine stays the default and stays supported for machines with no usable GPU.
+- **Speaker capture no longer has a length limit to worry about.** The old engine's memory grew with the square of how many turns a recording contains, which is why long streams had to be processed in windows and why a duration cap existed at all. The new engine holds a fixed amount of memory no matter how long the recording is — 558 MB for a 13-minute video or an 8-hour one — so nothing has to be windowed, capped, or decoded to a temporary file first.
+- **Switching speaker-capture engines correctly marks the old results as work to redo.** The two engines disagree about how many speakers exist, so a corpus half-captured by each is not one corpus. Changing the engine now shows every recording captured by the other one as outstanding backfill work, rather than leaving it looking finished. Changing unrelated settings — a clustering threshold the new engine does not even have — no longer marks anything stale.
+- **Speaker capture on the GPU stands aside for transcription instead of fighting it for video memory.** It needs about 4.4 GB of the 8 GB card that transcription also uses, so it now holds while transcription is working and resumes the moment the card is free — the same behaviour the digest sweep already had. The "run a guaranteed share alongside" setting is ignored while GPU capture is enabled, because a share of video memory is not a slower run, it is a failure.
- **The digest backlog that could never start now has a button that clears it, and honest copy about why.** Nearly 2,000 videos across the corpus have a transcript but no compact transcript file beside it, and the digest generator refuses those — so they sat in a "waiting" count on the Digest card under text that said the problem would resolve on its own. It does not. Nothing writes that file automatically for a channel whose captions are *downloaded* rather than transcribed here, which is why one channel alone accounts for 1,683 of them and was showing about 6% digest coverage. The card now says what is actually wrong, says that nothing will fix it unattended, and offers **Normalize transcripts** right there to fix it for that channel — instead of the corpus-wide button on the Build page that walks all 79,000 video folders. The button appears only when there is something for it to do.
- **Skipped-video explanations on the Backfill card now match the reason each one was skipped.** The card had one hardcoded sentence about the speaker-capture length limit and showed it for every kind of skipped video, including ones skipped for completely unrelated reasons. Each kind of work now supplies its own explanation and gets its own line.
- **There is now one definition of "digested", and every screen reads it.** The digest layer kept its own private tally of what still needed doing, separate from the one the digest runner actually uses — and the two could disagree by an entire channel. They are now the same number. Two consequences you will see immediately. **Channels whose reports were months old start reporting digest work at all**: eleven of them predated the old tally entirely and had been quietly reading as "nothing to do" (they account for about 1,500 videos, most of them in one channel). And **videos with no usable transcript file are no longer offered as work**: they were being handed to the digest runner, which looked at them and immediately put them back. On the current corpus that is 1,989 videos, and 1,683 of them are in a single channel — so nearly all of that channel's apparent digest backlog was work that could never have started. The overall count goes *down* slightly as a result, from 75,613 to 75,199, which is the counter becoming honest rather than anything being skipped.
diff --git a/editor/app/api/pulse/route.ts b/editor/app/api/pulse/route.ts
@@ -41,7 +41,7 @@ export type PulsePayload = {
// background, because /api/test/invalidate-cache clears exactly these globals
// between e2e specs — a poll landing a moment later silently rebuilt them,
// re-seeding a worker pool from settings mid-reset. The symptom was ~16
-// unrelated specs (branding, build, bookmarks, downloads) failing in
+// unrelated specs (branding, build, jobs, downloads) failing in
// non-deterministic combinations while each passed in isolation.
//
// So read the globals directly and treat "not created yet" as "nothing to
diff --git a/editor/app/api/widget/presets/route.ts b/editor/app/api/widget/presets/route.ts
@@ -0,0 +1,31 @@
+import { NextResponse } from "next/server";
+import { getPaths } from "yt-dlp-transcript-common/lib/paths";
+import {
+ readWidgetPresets,
+ type WidgetPreset,
+} from "yt-dlp-transcript-common/lib/widgetPresets";
+import {
+ BUILT_IN_PRESETS,
+ type BuiltInPreset,
+} from "../../../widget/lib/builtInPresets";
+
+export const dynamic = "force-dynamic";
+
+// The preset list, for the IN-WIDGET menu. The builder page SSR-seeds from
+// readWidgetPresets directly; the widget is a bare client page with no server
+// parent to seed it, so it fetches this on open and after each mutation.
+//
+// Built-ins ride along rather than being duplicated client-side, so the two
+// surfaces can never show different starting points.
+export type WidgetPresetsPayload = {
+ builtIn: BuiltInPreset[];
+ saved: WidgetPreset[];
+};
+
+export async function GET() {
+ const saved = await readWidgetPresets(getPaths());
+ return NextResponse.json({
+ builtIn: BUILT_IN_PRESETS,
+ saved,
+ } satisfies WidgetPresetsPayload);
+}
diff --git a/editor/app/channels/[slug]/components/RetryBucketControl.tsx b/editor/app/channels/[slug]/components/RetryBucketControl.tsx
@@ -15,7 +15,7 @@ type Props = {
defaultQueueKey: string;
existingQueues: string[];
// The snapshot bucket these ids represent. When set, the launched retry job
- // is bookmarkable and re-runs against the current bucket members.
+ // is replayable and re-runs against the current bucket members.
bucketKey?: ReplayBucket;
// Needs-cookies bucket: force cookie mode "always" for the run and label the
// button "Download with cookies" so the manual cookie path is explicit.
diff --git a/editor/app/channels/[slug]/incompleteTranscriptActions.ts b/editor/app/channels/[slug]/incompleteTranscriptActions.ts
@@ -56,7 +56,7 @@ export async function enableAutoRunners(): Promise<void> {
// One-shot batch re-fix: queue a single managed job that removes the truncated
// audio, re-downloads, and re-transcribes each flagged video in place (the bulk
// version of the per-video redownloadIncompleteTranscriptAction). ids default to
-// the channel's current incompleteTranscript bucket; bookmarkable so a re-run
+// the channel's current incompleteTranscript bucket; replayable so a re-run
// re-derives the live bucket.
export async function redownloadIncompleteBucketAction(
slug: string,
@@ -118,7 +118,7 @@ export async function redownloadIncompleteBucketAction(
// Re-download the channel's short-audio bucket (downloads the duration guard
// flagged as truncated at the source). Reuses the same per-video fixer and the
-// same bucket job kind, parameterized with bucket "shortAudio" so a bookmarked
+// same bucket job kind, parameterized with bucket "shortAudio" so a replayed
// re-run re-derives the live members. The re-download deletes the kept stub and
// re-fetches with the per-source default format (Original for Odysee), which is
// what actually recovers the full audio.
diff --git a/editor/app/channels/[slug]/page.tsx b/editor/app/channels/[slug]/page.tsx
@@ -267,9 +267,9 @@ export default async function ChannelDetailPage({
// `?stage=` replaces the old `#stage-` hash: the selection is server-rendered,
// shareable, and survives the global AutoRefresh's router.refresh(). An
- // unknown or absent value lands on the overview rather than 404ing — a stale
- // bookmark to a stage that no longer applies (e.g. transcode) should still
- // open the channel.
+ // unknown or absent value lands on the overview rather than 404ing — a saved
+ // link to a stage that no longer applies (e.g. transcode) should still open
+ // the channel.
const rawStage = typeof sp.stage === "string" ? sp.stage : undefined;
const selectedStage: StageId | null =
(stageOrder.find((id) => id === rawStage) as StageId | undefined) ?? null;
diff --git a/editor/app/channels/[slug]/pipelineActions.ts b/editor/app/channels/[slug]/pipelineActions.ts
@@ -90,7 +90,7 @@ async function runPipelineAction(
// sync only: force the full sweep (whole-listing re-read) regardless of the
// configured cadence. Undefined leaves the decision to the cadence gate.
forceFullSweep?: boolean;
- // Replay descriptor, forwarded onto the job record so it can be bookmarked.
+ // Replay descriptor, forwarded onto the job record so it can be replayed (Retry).
spec?: JobSpec;
},
): Promise<StreamActionResult> {
@@ -334,9 +334,9 @@ export async function retryBucketAction(
abortOnError?: boolean,
handlingOverride?: string,
// The snapshot bucket these ids came from. Passed by the named bucket controls
- // (partial downloads / no-transcript) so the job can be bookmarked and re-run
+ // (partial downloads / no-transcript) so the job can be replayed and re-run
// against the CURRENT bucket; omitted by ad-hoc checkbox selections, which are
- // therefore not bookmarkable.
+ // therefore not replayable.
bucketKey?: ReplayBucket,
// Needs-cookies bucket only: run with cookie mode forced to "always".
forceCookies?: boolean,
diff --git a/editor/app/channels/[slug]/whisperActions.ts b/editor/app/channels/[slug]/whisperActions.ts
@@ -110,7 +110,7 @@ export async function transcribeBucketAction(
audioFormat?: AudioFormat,
strictAudioFormat?: boolean,
// The snapshot bucket these ids came from (downloadedNoTranscript). Passed by
- // the named bucket control so the job is bookmarkable; omitted by ad-hoc
+ // the named bucket control so the job is replayable; omitted by ad-hoc
// selections.
bucketKey?: ReplayBucket,
): Promise<StreamActionResult> {
diff --git a/editor/app/jobs/[id]/page.tsx b/editor/app/jobs/[id]/page.tsx
@@ -5,7 +5,6 @@ import { notFound } from "next/navigation";
import { getJobEntry } from "yt-dlp-transcript-common/jobs/listJobs";
import { getPaths } from "yt-dlp-transcript-common/lib/paths";
import { JobLogTail } from "./components/JobLogTail";
-import { BookmarkJobButton } from "../components/BookmarkJobButton";
import { jobKindLabel } from "../jobKindLabels";
export const dynamic = "force-dynamic";
@@ -52,7 +51,6 @@ export default async function JobDetailPage({
<h1 className="text-2xl font-semibold" title={job.kind ?? undefined}>
{job.kind ? jobKindLabel(job.kind) : "job"}
</h1>
- {job.bookmarkable && <BookmarkJobButton jobId={id} />}
</div>
{job.channelSlug && (
<p className="text-sm">
diff --git a/editor/app/jobs/actions.ts b/editor/app/jobs/actions.ts
@@ -107,8 +107,8 @@ async function resolveSpec(id: string): Promise<JobSpec | null> {
return meta?.spec ?? null;
}
-// One-click retry: re-run a (typically failed) job from its stored spec. Like a
-// bookmark re-run, bucket jobs re-derive from the channel's CURRENT state. The
+// One-click retry: re-run a (typically failed) job from its stored spec. As on
+// any replay, bucket jobs re-derive from the channel's CURRENT state. The
// re-run jumps ahead of other queued work (run next) by reusing the same
// promote the reorder buttons use — a no-op if it starts immediately.
export async function retryJobAction(id: string): Promise<StreamActionResult> {
diff --git a/editor/app/jobs/active/buildActiveJobs.ts b/editor/app/jobs/active/buildActiveJobs.ts
@@ -200,7 +200,6 @@ export async function buildActiveJobsPayload(): Promise<ActiveJobsPayload> {
draining: j.draining === true,
drainable:
j.status === "running" && isDrainableKind(j.kind) && j.draining !== true,
- bookmarkable: Boolean(j.spec),
background: j.background === true,
// A queued job can move up/promote if it isn't the first queued (position >
// 1, since the running head is at 0), and down if it isn't the last in its
diff --git a/editor/app/jobs/active/page.tsx b/editor/app/jobs/active/page.tsx
@@ -6,17 +6,14 @@ import { buildQueueView } from "../queue/buildQueueView";
import { ActiveJobsLive } from "../components/ActiveJobsLive";
import { DrainAllButton } from "../components/DrainAllButton";
import { PauseTranscriptionsButton } from "../components/PauseTranscriptionsButton";
-import { BookmarksMenu } from "../components/BookmarksMenu";
-import { loadBookmarksView } from "../loadBookmarks";
export const dynamic = "force-dynamic";
export const metadata: Metadata = { title: "Active jobs" };
export default async function ActiveJobsPage() {
- const [initial, { bookmarks, missingSlugs }, queueView] = await Promise.all([
+ const [initial, queueView] = await Promise.all([
buildActiveJobsPayload(),
- loadBookmarksView(),
buildQueueView(),
]);
const paused = getWorkerPool().isPaused();
@@ -41,7 +38,6 @@ export default async function ActiveJobsPage() {
<DrainAllButton />
</div>
</div>
- <BookmarksMenu bookmarks={bookmarks} missingSlugs={missingSlugs} />
<ActiveJobsLive initial={initial} />
</div>
);
diff --git a/editor/app/jobs/bookmarkActions.ts b/editor/app/jobs/bookmarkActions.ts
@@ -1,106 +0,0 @@
-"use server";
-
-import { revalidatePath } from "next/cache";
-import { getPaths } from "yt-dlp-transcript-common/lib/paths";
-import { getRegistry } from "yt-dlp-transcript-common/jobs/registry";
-import { readJobMeta } from "yt-dlp-transcript-common/jobs/jobMeta";
-import {
- addBookmark,
- defaultBookmarkName,
- moveBookmark,
- readBookmarks,
- removeBookmark,
- updateBookmark,
-} from "yt-dlp-transcript-common/jobs/bookmarks";
-import { readChannelConfig } from "yt-dlp-transcript-common/controller/channels";
-import type { StreamActionResult } from "yt-dlp-transcript-common/jobs/streamCommand";
-import { runJobSpec } from "./runJobSpec";
-
-export type BookmarkActionResult = { ok: true } | { ok: false; error: string };
-
-function refreshJobsViews(): void {
- revalidatePath("/jobs");
- revalidatePath("/jobs/active");
-}
-
-// Bookmark a job by copying its replay descriptor (from the live registry, or
-// the on-disk sidecar for an archived job) into the bookmarks store. Jobs with
-// no spec — ad-hoc selections, single imports — can't be bookmarked.
-export async function bookmarkJobAction(
- jobId: string,
-): Promise<BookmarkActionResult> {
- const paths = getPaths();
- let spec = getRegistry().get(jobId)?.spec;
- if (!spec) {
- const meta = await readJobMeta(paths, jobId);
- spec = meta?.spec;
- }
- if (!spec) return { ok: false, error: "This job can't be bookmarked." };
- await addBookmark(paths, spec);
- refreshJobsViews();
- return { ok: true };
-}
-
-export async function deleteBookmarkAction(
- id: string,
-): Promise<BookmarkActionResult> {
- await removeBookmark(getPaths(), id);
- refreshJobsViews();
- return { ok: true };
-}
-
-// Reorder a bookmark by swapping it with its neighbour. Both the management page
-// and the compact /jobs menu render in array order, so persisting the new order
-// (and revalidating) updates both views.
-export async function moveBookmarkAction(
- id: string,
- dir: -1 | 1,
-): Promise<BookmarkActionResult> {
- await moveBookmark(getPaths(), id, dir);
- refreshJobsViews();
- return { ok: true };
-}
-
-// Rename a bookmark. An empty/blank name resets it to the auto-generated default
-// (kind · channel) so a bookmark always has a usable label.
-export async function renameBookmarkAction(
- id: string,
- name: string,
-): Promise<BookmarkActionResult> {
- const paths = getPaths();
- const trimmed = name.trim().slice(0, 200);
- if (trimmed) {
- await updateBookmark(paths, id, { name: trimmed });
- } else {
- const bookmark = (await readBookmarks(paths)).find((b) => b.id === id);
- if (bookmark) {
- await updateBookmark(paths, id, {
- name: defaultBookmarkName(bookmark.spec),
- });
- }
- }
- refreshJobsViews();
- return { ok: true };
-}
-
-// Re-launch a bookmarked job. Returns a StreamActionResult so the caller can
-// stream the new job's log via StreamActionLog, exactly like the original
-// stage-control buttons.
-export async function runBookmarkAction(
- id: string,
-): Promise<StreamActionResult> {
- const paths = getPaths();
- const bookmark = (await readBookmarks(paths)).find((b) => b.id === id);
- if (!bookmark) return { ok: false, error: "Bookmark not found." };
- // Guard against a bookmark whose channel was deleted: re-running would scan a
- // non-existent dir and do nothing useful.
- if (!(await readChannelConfig(paths, bookmark.spec.slug))) {
- return {
- ok: false,
- error: `Channel "${bookmark.spec.slug}" no longer exists.`,
- };
- }
- const result = await runJobSpec(bookmark.spec);
- if (result.ok) await updateBookmark(paths, id, { lastRunAt: Date.now() });
- return result;
-}
diff --git a/editor/app/jobs/bookmarks/page.tsx b/editor/app/jobs/bookmarks/page.tsx
@@ -1,37 +0,0 @@
-import type { Metadata } from "next";
-import Link from "next/link";
-import { BookmarksList } from "../components/BookmarksList";
-import { loadBookmarksView } from "../loadBookmarks";
-
-export const dynamic = "force-dynamic";
-
-export const metadata: Metadata = { title: "Bookmarks" };
-
-// Full bookmark management: rename, delete, created / last-run metadata, and the
-// streaming "Run again" button (with its inline log). The compact menu on /jobs
-// and /jobs/active links here via "Manage".
-export default async function BookmarksPage() {
- const { bookmarks, missingSlugs } = await loadBookmarksView();
- return (
- <div className="flex flex-col gap-4">
- <div className="flex items-center gap-2 text-sm text-muted-foreground">
- <Link
- href="/jobs"
- className="underline hover:text-foreground"
- >
- Jobs
- </Link>
- <span>/</span>
- <span>Bookmarks</span>
- </div>
- <h1 className="text-2xl font-semibold">Bookmarks</h1>
- {bookmarks.length === 0 ? (
- <p className="text-sm text-muted-foreground border border-dashed border-border rounded p-4">
- No bookmarks yet.
- </p>
- ) : (
- <BookmarksList bookmarks={bookmarks} missingSlugs={missingSlugs} />
- )}
- </div>
- );
-}
diff --git a/editor/app/jobs/components/BackfillSweepControls.tsx b/editor/app/jobs/components/BackfillSweepControls.tsx
@@ -3,11 +3,10 @@
import { useEffect, useState, useTransition } from "react";
import { useRouter } from "next/navigation";
import {
- pauseBackfillAction,
- resumeBackfillAction,
startBackfillSweepAction,
stopBackfillSweepAction,
} from "../actions";
+import { PauseBackfillButton } from "./PauseBackfillButton";
// Arm / disarm the corpus-wide backfill sweep, and hold it without ending it.
//
@@ -27,13 +26,12 @@ import {
//
// So the pause is surfaced here, and it writes THE SAME FIELD the Settings
// checkbox writes (settings.backfill.enabled) rather than a new `backfillPaused`
-// flag. One field, two places to set it, and they cannot drift. See
+// flag. One field, several places to set it, and they cannot drift. See
// pauseBackfillAction in ../actions for why that is the right field.
//
-// It is a REAL pause, not a stop: backfillBatch's limit() re-reads the flag at
-// dispatch and returns 0, so the pool idle-waits, the job stays alive, and
-// resuming re-derives nothing. It also survives a restart, because it is
-// persisted rather than applied to a live pool.
+// The pause button itself now lives in ./PauseBackfillButton — the monitor
+// widget's controls row wants it too, and one implementation is what keeps the
+// two surfaces honest. Everything below is the SWEEP.
//
// Stopping the SWEEP, by contrast, drains rather than cancels: the video in
// flight finishes instead of being thrown away, and the stop reaches the
@@ -98,26 +96,7 @@ export function BackfillSweepControls({
>
{sweeping ? "Stop Backfill Sweep" : "Start Backfill Sweep"}
</button>
- <button
- type="button"
- disabled={disabled}
- aria-label={laneEnabled ? "pause backfill" : "resume backfill"}
- onClick={() =>
- run(laneEnabled ? pauseBackfillAction : resumeBackfillAction)
- }
- title={
- laneEnabled
- ? "Hold the backfill lane without ending anything. A running job idles at zero and keeps its place; nothing is re-derived on resume. Survives a restart."
- : "Resume the backfill lane. A held job picks up within a few seconds — it was holding, not stopped."
- }
- className={
- laneEnabled
- ? "px-3 py-1.5 rounded-md border border-border text-sm hover:bg-muted disabled:opacity-50"
- : "px-3 py-1.5 rounded-md bg-warning text-warning-foreground text-sm font-medium hover:bg-warning/90 disabled:opacity-50"
- }
- >
- {laneEnabled ? "Pause Backfill" : "Resume Backfill"}
- </button>
+ <PauseBackfillButton paused={!laneEnabled} onChange={onChange} />
{sweeping && !laneEnabled && (
<span
aria-label="backfill lane off"
diff --git a/editor/app/jobs/components/BookmarkJobButton.tsx b/editor/app/jobs/components/BookmarkJobButton.tsx
@@ -1,50 +0,0 @@
-"use client";
-
-import { useState, useTransition } from "react";
-import { useRouter } from "next/navigation";
-import { bookmarkJobAction } from "../bookmarkActions";
-
-// Small inline button shown on a job (jobs table row, active-jobs row, job
-// detail) when it carries a replay descriptor. Saves a bookmark, then shows a
-// transient "Bookmarked ✓" state. Errors are surfaced as the button title.
-export function BookmarkJobButton({
- jobId,
- className,
-}: {
- jobId: string;
- className?: string;
-}) {
- const router = useRouter();
- const [pending, startTransition] = useTransition();
- const [done, setDone] = useState(false);
- const [error, setError] = useState<string | null>(null);
-
- function onClick() {
- setError(null);
- startTransition(async () => {
- const r = await bookmarkJobAction(jobId);
- if (r.ok) {
- setDone(true);
- router.refresh();
- } else {
- setError(r.error);
- }
- });
- }
-
- return (
- <button
- type="button"
- onClick={onClick}
- disabled={pending || done}
- aria-label={`bookmark job ${jobId}`}
- title={error ?? undefined}
- className={
- className ??
- "px-2 py-1 rounded-md bg-muted text-foreground text-xs font-medium hover:opacity-90 disabled:opacity-50"
- }
- >
- {done ? "Bookmarked ✓" : pending ? "Bookmarking…" : "Bookmark"}
- </button>
- );
-}
diff --git a/editor/app/jobs/components/BookmarkRunButton.tsx b/editor/app/jobs/components/BookmarkRunButton.tsx
@@ -1,94 +0,0 @@
-"use client";
-
-import { useState, useTransition } from "react";
-import { useRouter } from "next/navigation";
-import type { JobBookmark } from "yt-dlp-transcript-common/jobs/bookmarks";
-import { runBookmarkAction } from "../bookmarkActions";
-
-// One quick-access button in the compact bookmarks menu (BookmarksMenu). Unlike
-// the full management UI on /jobs/bookmarks, this launches the job
-// fire-and-forget: it never mounts StreamActionLog / an inline shell log. The
-// re-launched job runs to completion server-side (streamCommand keeps the child
-// and on-disk log alive even when no consumer reads the stream) and shows up in
-// the live job list below. Errors and the neutral "nothing to do" (info) case
-// are surfaced as a brief status line + the button title — no expanding log box.
-export function BookmarkRunButton({
- bookmark,
- channelMissing,
-}: {
- bookmark: JobBookmark;
- channelMissing: boolean;
-}) {
- const router = useRouter();
- const [pending, startTransition] = useTransition();
- const [launched, setLaunched] = useState(false);
- const [status, setStatus] = useState<
- { kind: "error" | "info"; text: string } | null
- >(null);
- const { spec } = bookmark;
-
- const detail = [
- spec.kind,
- spec.bucket ? `bucket ${spec.bucket}` : null,
- spec.slug,
- ]
- .filter(Boolean)
- .join(" · ");
-
- function onClick() {
- if (channelMissing) return;
- setStatus(null);
- setLaunched(false);
- startTransition(async () => {
- const r = await runBookmarkAction(bookmark.id);
- if (r.ok) {
- // Release the stream reader; the job keeps running server-side. The
- // refresh surfaces it in the live job list below.
- void r.stream.cancel().catch(() => {});
- setLaunched(true);
- router.refresh();
- } else {
- setStatus({ kind: r.info ? "info" : "error", text: r.error });
- }
- });
- }
-
- const label = channelMissing
- ? bookmark.name
- : launched
- ? "Launched ✓"
- : pending
- ? "Launching…"
- : bookmark.name;
-
- return (
- <div className="flex flex-col gap-1 max-w-full">
- <button
- type="button"
- onClick={onClick}
- disabled={pending || channelMissing}
- aria-label={`run bookmark ${bookmark.name}`}
- title={status?.text ?? detail}
- className="max-w-[16rem] truncate px-3 py-1.5 rounded-md bg-primary text-primary-foreground text-xs font-medium hover:opacity-90 disabled:opacity-50 disabled:cursor-not-allowed"
- >
- {label}
- </button>
- {channelMissing ? (
- <span className="text-[11px] text-warning">
- channel missing
- </span>
- ) : status ? (
- <span
- role={status.kind === "error" ? "alert" : "status"}
- className={
- status.kind === "error"
- ? "text-[11px] text-destructive"
- : "text-[11px] text-muted-foreground"
- }
- >
- {status.text}
- </span>
- ) : null}
- </div>
- );
-}
diff --git a/editor/app/jobs/components/BookmarksList.tsx b/editor/app/jobs/components/BookmarksList.tsx
@@ -1,239 +0,0 @@
-"use client";
-
-import { useState, useTransition } from "react";
-import Link from "next/link";
-import { useRouter } from "next/navigation";
-import { StreamActionLog } from "yt-dlp-transcript-common/components/StreamActionLog";
-import type { JobBookmark } from "yt-dlp-transcript-common/jobs/bookmarks";
-import { cancelJobAction } from "../actions";
-import {
- deleteBookmarkAction,
- moveBookmarkAction,
- renameBookmarkAction,
- runBookmarkAction,
-} from "../bookmarkActions";
-import { jobKindLabel } from "../jobKindLabels";
-
-// The saved-jobs panel rendered on /jobs and /jobs/active. Each bookmark has a
-// "Run again" button (streams the re-launched job's log), an inline rename, a
-// confirm-gated delete, and created / last-run metadata. Bookmarks whose channel
-// was deleted are flagged and can't be re-run.
-export function BookmarksList({
- bookmarks,
- missingSlugs = [],
-}: {
- bookmarks: JobBookmark[];
- missingSlugs?: string[];
-}) {
- if (bookmarks.length === 0) return null;
- const missing = new Set(missingSlugs);
- return (
- <section
- aria-label="Bookmarked jobs"
- className="flex flex-col gap-3 border border-border rounded-md p-3 bg-card"
- >
- <h2 className="text-sm font-medium text-muted-foreground">
- Bookmarks
- </h2>
- <ul className="flex flex-col gap-3">
- {bookmarks.map((b, i) => (
- <BookmarkRow
- key={b.id}
- bookmark={b}
- channelMissing={missing.has(b.spec.slug)}
- index={i}
- total={bookmarks.length}
- />
- ))}
- </ul>
- </section>
- );
-}
-
-function fmtDate(ms?: number): string {
- if (!ms) return "—";
- return new Date(ms).toLocaleString();
-}
-
-function BookmarkRow({
- bookmark,
- channelMissing,
- index,
- total,
-}: {
- bookmark: JobBookmark;
- channelMissing: boolean;
- index: number;
- total: number;
-}) {
- const router = useRouter();
- const [pending, startTransition] = useTransition();
- const [confirmingDelete, setConfirmingDelete] = useState(false);
- const [editing, setEditing] = useState(false);
- const [draftName, setDraftName] = useState(bookmark.name);
- const { spec } = bookmark;
-
- function onDelete() {
- startTransition(async () => {
- await deleteBookmarkAction(bookmark.id);
- router.refresh();
- });
- }
-
- function onSaveName() {
- setEditing(false);
- startTransition(async () => {
- await renameBookmarkAction(bookmark.id, draftName);
- router.refresh();
- });
- }
-
- function onMove(dir: -1 | 1) {
- startTransition(async () => {
- await moveBookmarkAction(bookmark.id, dir);
- router.refresh();
- });
- }
-
- return (
- <li
- aria-label={`bookmark ${bookmark.name}`}
- className="flex flex-col gap-2 border-t border-border pt-3 first:border-t-0 first:pt-0"
- >
- <div className="flex flex-wrap items-center gap-2 text-sm">
- {editing ? (
- <>
- <input
- type="text"
- value={draftName}
- onChange={(e) => setDraftName(e.target.value)}
- aria-label={`new name for ${bookmark.name}`}
- className="rounded border border-border bg-card px-2 py-0.5 text-sm"
- />
- <button
- type="button"
- onClick={onSaveName}
- className="px-2 py-1 rounded-md bg-primary text-primary-foreground text-xs font-medium hover:opacity-90"
- >
- Save
- </button>
- <button
- type="button"
- onClick={() => {
- setEditing(false);
- setDraftName(bookmark.name);
- }}
- className="px-2 py-1 rounded-md border border-border text-xs font-medium hover:bg-muted"
- >
- Cancel
- </button>
- </>
- ) : (
- <>
- <span className="font-medium">{bookmark.name}</span>
- <button
- type="button"
- onClick={() => {
- setDraftName(bookmark.name);
- setEditing(true);
- }}
- aria-label={`rename bookmark ${bookmark.name}`}
- className="px-2 py-0.5 rounded border border-border text-xs text-muted-foreground hover:bg-muted"
- >
- Rename
- </button>
- </>
- )}
- <span className="text-xs text-muted-foreground" title={spec.kind}>
- {jobKindLabel(spec.kind)}
- </span>
- {spec.bucket && (
- <span className="font-mono text-xs text-muted-foreground">
- bucket {spec.bucket}
- </span>
- )}
- <Link
- href={`/channels/${spec.slug}`}
- className="font-mono text-xs underline hover:text-foreground"
- >
- {spec.slug}
- </Link>
- {channelMissing && (
- <span
- aria-label={`channel missing for ${bookmark.name}`}
- className="text-xs px-2 py-0.5 rounded bg-warning-soft text-warning"
- >
- channel missing
- </span>
- )}
- <div className="ml-auto flex items-center gap-2">
- <button
- type="button"
- onClick={() => onMove(-1)}
- disabled={pending || index === 0}
- aria-label={`move bookmark ${bookmark.name} up`}
- className="px-2 py-0.5 rounded border border-border text-xs disabled:opacity-40"
- >
- ↑
- </button>
- <button
- type="button"
- onClick={() => onMove(1)}
- disabled={pending || index === total - 1}
- aria-label={`move bookmark ${bookmark.name} down`}
- className="px-2 py-0.5 rounded border border-border text-xs disabled:opacity-40"
- >
- ↓
- </button>
- {confirmingDelete ? (
- <>
- <button
- type="button"
- onClick={onDelete}
- disabled={pending}
- aria-label={`confirm delete bookmark ${bookmark.name}`}
- className="px-2 py-1 rounded-md bg-destructive text-destructive-foreground text-xs font-medium hover:bg-destructive/90 disabled:opacity-50"
- >
- {pending ? "Deleting…" : "Confirm"}
- </button>
- <button
- type="button"
- onClick={() => setConfirmingDelete(false)}
- className="px-2 py-1 rounded-md border border-border text-xs font-medium hover:bg-muted"
- >
- Cancel
- </button>
- </>
- ) : (
- <button
- type="button"
- onClick={() => setConfirmingDelete(true)}
- aria-label={`delete bookmark ${bookmark.name}`}
- className="px-2 py-1 rounded-md border border-destructive/30 text-xs font-medium text-destructive hover:bg-destructive-soft"
- >
- Delete
- </button>
- )}
- </div>
- </div>
- <p className="text-xs text-muted-foreground" suppressHydrationWarning>
- Created {fmtDate(bookmark.createdAt)} · Last run{" "}
- {bookmark.lastRunAt ? fmtDate(bookmark.lastRunAt) : "never"}
- </p>
- {channelMissing ? (
- <p className="text-xs text-warning">
- This channel no longer exists — delete the bookmark or recreate the
- channel to run it again.
- </p>
- ) : (
- <StreamActionLog
- trigger={() => runBookmarkAction(bookmark.id)}
- cancelAction={cancelJobAction}
- buttonLabel="Run again"
- runningLabel="Running…"
- label={`Run ${bookmark.name}`}
- />
- )}
- </li>
- );
-}
diff --git a/editor/app/jobs/components/BookmarksMenu.tsx b/editor/app/jobs/components/BookmarksMenu.tsx
@@ -1,45 +0,0 @@
-import Link from "next/link";
-import type { JobBookmark } from "yt-dlp-transcript-common/jobs/bookmarks";
-import { BookmarkRunButton } from "./BookmarkRunButton";
-
-// Compact bookmarks panel rendered on /jobs and /jobs/active: a quick-access
-// row of run buttons (one per bookmark), each showing just enough to identify
-// it. Launching is fire-and-forget — the inline shell log and the heavier
-// rename/delete/metadata controls live on the linked /jobs/bookmarks page.
-export function BookmarksMenu({
- bookmarks,
- missingSlugs = [],
-}: {
- bookmarks: JobBookmark[];
- missingSlugs?: string[];
-}) {
- if (bookmarks.length === 0) return null;
- const missing = new Set(missingSlugs);
- return (
- <section
- aria-label="Bookmarked jobs"
- className="flex flex-col gap-3 border border-border rounded-md p-3 bg-card"
- >
- <div className="flex items-center justify-between gap-2">
- <h2 className="text-sm font-medium text-muted-foreground">
- Bookmarks
- </h2>
- <Link
- href="/jobs/bookmarks"
- className="text-xs underline text-muted-foreground hover:text-foreground"
- >
- Manage
- </Link>
- </div>
- <div className="flex flex-wrap items-start gap-2">
- {bookmarks.map((b) => (
- <BookmarkRunButton
- key={b.id}
- bookmark={b}
- channelMissing={missing.has(b.spec.slug)}
- />
- ))}
- </div>
- </section>
- );
-}
diff --git a/editor/app/jobs/components/JobsTable.tsx b/editor/app/jobs/components/JobsTable.tsx
@@ -4,7 +4,6 @@ import { useEffect, useMemo, useState } from "react";
import Link from "next/link";
import type { JobListEntry } from "yt-dlp-transcript-common/jobs/listJobs";
import { CancelJobButton } from "./CancelJobButton";
-import { BookmarkJobButton } from "./BookmarkJobButton";
import { RetryJobButton } from "./RetryJobButton";
import { jobKindLabel } from "../jobKindLabels";
import {
@@ -286,10 +285,9 @@ export function JobsTable({ jobs }: { jobs: JobListEntry[] }) {
</td>
<td className="px-3 py-2 text-right">
<div className="flex items-center justify-end gap-2">
- {j.status === "failed" && j.bookmarkable && (
+ {j.status === "failed" && j.replayable && (
<RetryJobButton jobId={j.id} />
)}
- {j.bookmarkable && <BookmarkJobButton jobId={j.id} />}
{(j.status === "running" || j.status === "queued") && (
<CancelJobButton jobId={j.id} />
)}
diff --git a/editor/app/jobs/components/PauseBackfillButton.tsx b/editor/app/jobs/components/PauseBackfillButton.tsx
@@ -0,0 +1,79 @@
+"use client";
+
+import { useEffect, useState, useTransition } from "react";
+import { useRouter } from "next/navigation";
+import { pauseBackfillAction, resumeBackfillAction } from "../actions";
+
+// Hold the backfill lane without ending anything, in the prop shape the other
+// two pause buttons use ({ paused, disabled?, onChange? }) so all three can sit
+// side by side wherever work is being watched — the dashboard cockpit, and now
+// the monitor widget's controls row.
+//
+// Extracted from BackfillSweepControls, which still renders it: one
+// implementation, so the dashboard and the widget cannot drift.
+//
+// It is a REAL pause, not a stop. backfillBatch's limit() re-reads
+// settings.backfill.enabled at dispatch (~3s idle poll) and returns 0, so the
+// pool idle-waits, the job stays alive and keeps its place, and resuming
+// re-derives nothing. It writes THE SAME FIELD the Settings checkbox writes
+// rather than a second `backfillPaused` flag, so the two cannot disagree — which
+// is why backfill.spec asserts the settings field through this button's label
+// rather than just the label flipping.
+export function PauseBackfillButton({
+ paused,
+ disabled,
+ onChange,
+}: {
+ // settings.backfill.enabled INVERTED: `paused` is the lane held.
+ paused: boolean;
+ disabled?: boolean;
+ onChange?: () => void | Promise<void>;
+}) {
+ const [pending, startTransition] = useTransition();
+ const [error, setError] = useState<string | null>(null);
+ const router = useRouter();
+ // Disabled until hydrated. A click on a server-rendered button before React
+ // attaches fires NOTHING — no request, no job, no error — which is the
+ // recorded root cause of the digest pilot's "un-created job".
+ const [mounted, setMounted] = useState(false);
+ useEffect(() => setMounted(true), []);
+
+ function toggle() {
+ setError(null);
+ startTransition(async () => {
+ const result = await (paused ? resumeBackfillAction() : pauseBackfillAction());
+ if (!result.ok) setError(result.error ?? "Failed.");
+ if (onChange) await onChange();
+ else router.refresh();
+ });
+ }
+
+ const busy = pending || !mounted;
+ return (
+ <>
+ <button
+ type="button"
+ disabled={busy || (!paused && disabled)}
+ aria-label={paused ? "resume backfill" : "pause backfill"}
+ onClick={toggle}
+ title={
+ paused
+ ? "Resume the backfill lane. A held job picks up within a few seconds — it was holding, not stopped."
+ : "Hold the backfill lane without ending anything. A running job idles at zero and keeps its place; nothing is re-derived on resume. Survives a restart."
+ }
+ className={
+ paused
+ ? "px-3 py-1.5 rounded-md bg-warning text-warning-foreground text-sm font-medium hover:bg-warning/90 disabled:opacity-50"
+ : "px-3 py-1.5 rounded-md border border-border text-sm hover:bg-muted disabled:opacity-50"
+ }
+ >
+ {paused ? "Resume Backfill" : "Pause Backfill"}
+ </button>
+ {error && (
+ <span role="alert" className="text-xs text-destructive">
+ {error}
+ </span>
+ )}
+ </>
+ );
+}
diff --git a/editor/app/jobs/components/RunningJobsList.tsx b/editor/app/jobs/components/RunningJobsList.tsx
@@ -12,7 +12,6 @@ import { jobKindLabel } from "../jobKindLabels";
import { DrainJobButton } from "./DrainJobButton";
import { CancelJobButton } from "./CancelJobButton";
import { ForceReleaseJobButton } from "./ForceReleaseJobButton";
-import { BookmarkJobButton } from "./BookmarkJobButton";
import { ReorderJobButtons } from "./ReorderJobButtons";
export type RunningJobsTask = {
@@ -58,8 +57,6 @@ export type RunningJobsListItem = {
tasks?: RunningJobsTask[];
draining?: boolean;
drainable?: boolean;
- // True when the job carries a replay descriptor and can be bookmarked.
- bookmarkable?: boolean;
// Background work (e.g. an auto-download unit) queues BEHIND a manual job on
// the same platform queue. Shown as an "auto" badge, and — when queued — as a
// hint that a foreground job (a clicked Sync) is being let through first.
@@ -163,7 +160,6 @@ function JobRow({
</span>
)}
<div className="ml-auto flex items-center gap-2">
- {job.bookmarkable && <BookmarkJobButton jobId={job.id} />}
{job.status === "queued" && (job.canMoveUp || job.canMoveDown) && (
<ReorderJobButtons
jobId={job.id}
diff --git a/editor/app/jobs/jobReplayRegistry.ts b/editor/app/jobs/jobReplayRegistry.ts
@@ -1,12 +1,12 @@
// The data-driven replay table: maps a stored JobSpec's `kind` to the server
// action that re-runs it. Previously this was a hard-coded switch in
-// runJobSpec.ts; collapsing it to a lookup means adding a bookmarkable kind is
+// runJobSpec.ts; collapsing it to a lookup means adding a replayable kind is
// one colocated entry here (plus its jobKinds.ts metadata) instead of a new
-// switch case. runJobSpec.ts is now a thin dispatcher over this table, and the
-// retry actions (Phase 8) reuse the same seam.
+// switch case. runJobSpec.ts is now a thin dispatcher over this table, which the
+// retry actions (Retry / Retry all failed) drive.
//
-// Server-only: reached through bookmarkActions ("use server"), so it is never
-// bundled to the client.
+// Server-only: reached through the retry actions in actions.ts ("use server"),
+// so it is never bundled to the client.
import { getPaths } from "yt-dlp-transcript-common/lib/paths";
import { readChannelSnapshot } from "yt-dlp-transcript-common/controller/channelSnapshot";
import type { JobSpec, ReplayBucket } from "yt-dlp-transcript-common/jobs/jobSpec";
@@ -83,7 +83,7 @@ function params(spec: JobSpec): {
export const JOB_REPLAY_HANDLERS: Record<string, ReplayHandler> = {
// Both digest lanes replay through one action; the lane comes from params so a
- // bookmarked metered run stays metered (and is refused if the lane has since
+ // replayed metered run stays metered (and is refused if the lane has since
// been turned off, rather than quietly falling back to local).
"digest-channel-local": (spec) => {
const { p, queueKey } = params(spec);
@@ -121,7 +121,7 @@ export const JOB_REPLAY_HANDLERS: Record<string, ReplayHandler> = {
},
"whisper-bucket-downloaded-no-transcript": async (spec) => {
const { p, queueKey } = params(spec);
- if (!spec.bucket) return { ok: false, error: "Bookmark is missing its bucket." };
+ if (!spec.bucket) return { ok: false, error: "Job spec is missing its bucket." };
const ids = await idsForBucket(spec.slug, spec.bucket);
if (ids.length === 0) {
return { ok: false, error: "Nothing to transcribe right now.", info: true };
@@ -137,7 +137,7 @@ export const JOB_REPLAY_HANDLERS: Record<string, ReplayHandler> = {
},
"redownload-incomplete-bucket": async (spec) => {
const { queueKey } = params(spec);
- if (!spec.bucket) return { ok: false, error: "Bookmark is missing its bucket." };
+ if (!spec.bucket) return { ok: false, error: "Job spec is missing its bucket." };
const ids = await idsForBucket(spec.slug, spec.bucket);
if (ids.length === 0) {
return { ok: false, error: "Nothing to re-download right now.", info: true };
@@ -150,7 +150,7 @@ export const JOB_REPLAY_HANDLERS: Record<string, ReplayHandler> = {
},
"retry-bucket": async (spec) => {
const { p, queueKey } = params(spec);
- if (!spec.bucket) return { ok: false, error: "Bookmark is missing its bucket." };
+ if (!spec.bucket) return { ok: false, error: "Job spec is missing its bucket." };
const ids = await idsForBucket(spec.slug, spec.bucket);
if (ids.length === 0) {
return { ok: false, error: "Nothing to retry right now.", info: true };
@@ -243,7 +243,7 @@ export const JOB_REPLAY_HANDLERS: Record<string, ReplayHandler> = {
},
"whisper-bucket-auto-subs": async (spec) => {
const { p, queueKey } = params(spec);
- if (!spec.bucket) return { ok: false, error: "Bookmark is missing its bucket." };
+ if (!spec.bucket) return { ok: false, error: "Job spec is missing its bucket." };
const ids = await idsForBucket(spec.slug, spec.bucket);
if (ids.length === 0) {
return {
@@ -275,7 +275,7 @@ export const JOB_REPLAY_HANDLERS: Record<string, ReplayHandler> = {
// Replays correctly with nothing remembered: the batch re-derives its whole
// work-list from disk, so a replay does what is missing NOW rather than what
// was missing when the record was written. The kind scope is carried through
- // so a bookmarked single-kind run stays single-kind.
+ // so a replayed single-kind run stays single-kind.
"backfill-channel": (spec) => {
const { p, queueKey } = params(spec);
const kindIds = Array.isArray(p.kindIds)
diff --git a/editor/app/jobs/loadBookmarks.ts b/editor/app/jobs/loadBookmarks.ts
@@ -1,22 +0,0 @@
-import { getPaths } from "yt-dlp-transcript-common/lib/paths";
-import { readBookmarks, type JobBookmark } from "yt-dlp-transcript-common/jobs/bookmarks";
-import { readChannelConfig } from "yt-dlp-transcript-common/controller/channels";
-
-// Load bookmarks for the /jobs and /jobs/active pages, plus the set of channel
-// slugs that no longer exist — so the UI can flag bookmarks whose channel was
-// deleted and disable their Run-again button.
-export async function loadBookmarksView(): Promise<{
- bookmarks: JobBookmark[];
- missingSlugs: string[];
-}> {
- const paths = getPaths();
- const bookmarks = await readBookmarks(paths);
- const slugs = Array.from(new Set(bookmarks.map((b) => b.spec.slug)));
- const checks = await Promise.all(
- slugs.map(
- async (s) => [s, Boolean(await readChannelConfig(paths, s))] as const,
- ),
- );
- const missingSlugs = checks.filter(([, ok]) => !ok).map(([s]) => s);
- return { bookmarks, missingSlugs };
-}
diff --git a/editor/app/jobs/page.tsx b/editor/app/jobs/page.tsx
@@ -9,8 +9,6 @@ import { getPaths } from "yt-dlp-transcript-common/lib/paths";
import { ClearLogsMenu } from "./components/ClearLogsMenu";
import { JobsTable } from "./components/JobsTable";
import { RetryAllFailedButton } from "./components/RetryAllFailedButton";
-import { BookmarksMenu } from "./components/BookmarksMenu";
-import { loadBookmarksView } from "./loadBookmarks";
export const dynamic = "force-dynamic";
@@ -32,8 +30,7 @@ export default async function JobsPage({
searchParams: Promise<{ limit?: string | string[] }>;
}) {
const limit = parseLimit((await searchParams).limit);
- const [{ entries, hasMore, total }, { bookmarks, missingSlugs }] =
- await Promise.all([listAllJobs(getPaths(), { limit }), loadBookmarksView()]);
+ const { entries, hasMore, total } = await listAllJobs(getPaths(), { limit });
// Base the Retry-all affordance on the registry (what retryAllFailedAction
// actually acts on), not just the loaded page.
const hasRetryableFailed = getRegistry()
@@ -48,7 +45,6 @@ export default async function JobsPage({
<ClearLogsMenu />
</div>
</div>
- <BookmarksMenu bookmarks={bookmarks} missingSlugs={missingSlugs} />
{total === 0 ? (
<p className="text-sm text-muted-foreground border border-dashed border-border rounded p-4">
No jobs have run yet.
diff --git a/editor/app/jobs/runJobSpec.ts b/editor/app/jobs/runJobSpec.ts
@@ -1,5 +1,5 @@
-// Server-only dispatcher: reached only through bookmarkActions ("use server"),
-// so it is never bundled to the client.
+// Server-only dispatcher: reached only through the retry actions in actions.ts
+// ("use server"), so it is never bundled to the client.
//
// The per-kind replay logic lives in jobReplayRegistry.ts (a data-driven
// lookup); this is just the dispatch over it. Re-running RE-DERIVES the work
diff --git a/editor/app/settings/actions.ts b/editor/app/settings/actions.ts
@@ -370,6 +370,24 @@ export async function saveSettingsAction(
dDiar.python,
segModel: String(formData.get("diarizationSegModel") ?? "").trim(),
embModel: String(formData.get("diarizationEmbModel") ?? "").trim(),
+ // Read as plain strings and left to sanitizeDiarization to validate:
+ // an unrecognized value there falls back to the DEFAULT engine, which
+ // is the one every sidecar on disk already matches. Narrowing here
+ // instead would mean this form and the sanitizer could disagree about
+ // what a valid engine is, and the corpus pays for that disagreement in
+ // weeks of regeneration.
+ engine: String(
+ formData.get("diarizationEngine") ?? dDiar.engine,
+ ) as typeof dDiar.engine,
+ backend: String(
+ formData.get("diarizationBackend") ?? dDiar.backend,
+ ) as typeof dDiar.backend,
+ sortformerBin: String(
+ formData.get("diarizationSortformerBin") ?? "",
+ ).trim(),
+ sortformerModel: String(
+ formData.get("diarizationSortformerModel") ?? "",
+ ).trim(),
}
: dDiar;
diff --git a/editor/app/settings/components/SettingsForm.tsx b/editor/app/settings/components/SettingsForm.tsx
@@ -763,6 +763,71 @@ export function SettingsForm({ initial, apps, digestApps }: Props) {
</span>
</span>
</label>
+ <label className="flex flex-col gap-1 text-sm">
+ <span className="font-medium">Engine</span>
+ <select
+ name="diarizationEngine"
+ defaultValue={initial.diarization.engine}
+ className="rounded border border-border bg-card px-2 py-1 text-sm"
+ >
+ <option value="sherpa-onnx">
+ sherpa-onnx — clustered, CPU only (default)
+ </option>
+ <option value="sortformer">
+ Sortformer — end-to-end, GPU or CPU
+ </option>
+ </select>
+ <span className="text-xs text-muted-foreground">
+ <strong>sherpa-onnx</strong> groups voices that sound alike, and it
+ splits far too eagerly: on a 13-minute video with one host it finds
+ 13 speakers, and on the worst file in this corpus it finds 35.{" "}
+ <strong>Sortformer</strong> decides turns directly instead of
+ grouping them afterwards and returns 4 in both cases, agreeing with
+ the other engine about how much of the video the main speaker talks
+ for. It has no threshold, and it caps at 4 speakers — a panel of
+ five will merge two rather than invent twenty.
+ <br />
+ <br />
+ sherpa-onnx is the <strong>faster</strong> of the two (492 against
+ 894 seconds per audio-hour here), so this is a quality choice, not a
+ speed one. Switching also{" "}
+ <strong>marks every recording captured by the other engine as work
+ to redo</strong>, because the two disagree about how many speakers
+ exist — on the audio still held here that is weeks of it.
+ </span>
+ </label>
+ <label className="flex flex-col gap-1 text-sm">
+ <span className="font-medium">Sortformer device</span>
+ <select
+ name="diarizationBackend"
+ defaultValue={initial.diarization.backend}
+ className="rounded border border-border bg-card px-2 py-1 text-sm"
+ >
+ <option value="vulkan">Vulkan — the graphics card</option>
+ <option value="cpu">CPU</option>
+ </select>
+ <span className="text-xs text-muted-foreground">
+ Ignored unless the engine above is Sortformer. Both produce{" "}
+ <strong>identical</strong> speaker turns, so this only trades one
+ resource for another: the card is ~1.5× faster and uses one core
+ instead of two, but takes about 4.4 GB of the same 8 GB card
+ transcription uses — so capture stands aside while transcription is
+ working, and resumes when the card is free. On CPU it needs 4.84 GB
+ of system memory instead.
+ </span>
+ </label>
+ <Field
+ label="Sortformer engine binary"
+ name="diarizationSortformerBin"
+ defaultValue={initial.diarization.sortformerBin}
+ hint="Absolute path to the diarize-file binary built by scripts/build-sortformer.sh. Empty means the Sortformer engine is not configured, and every run reports a skip rather than a failure."
+ />
+ <Field
+ label="Sortformer model (GGUF)"
+ name="diarizationSortformerModel"
+ defaultValue={initial.diarization.sortformerModel}
+ hint="Absolute path to the .gguf downloaded by scripts/build-sortformer.sh. Its filename is recorded in every sidecar, so changing the model is what marks earlier captures worth redoing."
+ />
<Field
label="Segmentation model (ONNX)"
name="diarizationSegModel"
diff --git a/editor/app/widget/builder/components/LayoutBoard.tsx b/editor/app/widget/builder/components/LayoutBoard.tsx
@@ -4,27 +4,69 @@ import { Fragment, useState } from "react";
import type { WidgetConfig } from "../../lib/config";
import {
MAX_COLUMNS,
+ MAX_ROWS,
+ addCell,
insertAt,
- moveToColumn,
+ moveToCell,
+ moveToRow,
moveWithin,
- reconcileColumns,
+ reconcileLayout,
+ removeCell,
+ setCellFlow,
+ setCellSpan,
setColumnCount,
+ setRowCount,
+ setRowSize,
} from "../../lib/placement";
import { SECTIONS, SECTION_BY_ID, type SectionId } from "../../lib/sections";
import { SectionCard } from "./SectionCard";
import { SectionMini } from "./SectionMini";
-// The floorplan: a scale model of the widget, laid out in the same columns and
-// order it will render in. Drag a card to move it; the tray underneath holds
-// the sections that are switched off.
+// The floorplan: a scale model of the widget, laid out in the same grid it will
+// render in — the same tracks, the same rows, the same spans. Drag a card to
+// move it; the tray underneath holds the sections that are switched off.
+//
+// The board's one bold element is the LEFT RAIL, one control per row drawn as
+// its height: a hairline box for `auto`, a tall box with a spring mark for
+// `fill`. That is where the dimension the columns model never had actually
+// lives, so it is the one thing here allowed to look like something. Everything
+// else stays hairlines on `surface` and `card`.
//
// Native HTML5 drag-and-drop, deliberately — it costs no dependency, and the
// arrow buttons on every card cover keyboard use (and touch, which HTML5 drag
// does not support) so nothing here is mouse-only.
-// Below this a column is too narrow for the strips inside it to read.
+// Below this a track is too narrow for the strips inside it to read.
const NARROW_PX = 150;
+// Literal classes, because Tailwind reads the source. The `@max-[24rem]`
+// overrides are on the BOARD's own container query, not the viewport's, so the
+// board folds identically inside the widget's ~320px menu overlay and on the
+// builder page.
+const BOARD_COLS: Record<number, string> = {
+ 1: "grid-cols-1",
+ 2: "grid-cols-2",
+ 3: "grid-cols-3",
+ 4: "grid-cols-4",
+ 5: "grid-cols-5",
+ 6: "grid-cols-6",
+};
+
+const BOARD_SPAN: Record<number, string> = {
+ 1: "col-span-1",
+ 2: "col-span-2",
+ 3: "col-span-3",
+ 4: "col-span-4",
+ 5: "col-span-5",
+ 6: "col-span-6",
+};
+
+const STACKED = "@max-[24rem]:grid-cols-1";
+const STACKED_CELL = "@max-[24rem]:col-span-1";
+
+const CHIP =
+ "flex h-6 min-w-6 items-center justify-center rounded border border-border px-1 text-xs font-medium text-muted-foreground hover:bg-muted disabled:opacity-40";
+
export function LayoutBoard({
config,
onChange,
@@ -32,20 +74,29 @@ export function LayoutBoard({
}: {
config: WidgetConfig;
onChange: (patch: Partial<WidgetConfig>) => void;
- previewWidth: number;
+ // The width the widget will actually be. Absent inside the widget's own menu,
+ // where the px readouts are noise in a 320px overlay.
+ previewWidth?: number;
}) {
const [draggingId, setDraggingId] = useState<SectionId | null>(null);
- const [drop, setDrop] = useState<{ col: number; index: number } | null>(null);
+ const [drop, setDrop] = useState<{
+ row: number;
+ cell: number;
+ index: number;
+ } | null>(null);
const [expanded, setExpanded] = useState<SectionId[]>([]);
- const columns = config.columns;
+ const layout = config.layout;
const off = SECTIONS.filter((s) => !s.enabled(config));
- // The widget's padding (p-2 either side) and the gaps between columns come
- // out of the width before it is split, so this is the real number.
- const columnPx = Math.round(
- (previewWidth - 16 - (columns.length - 1) * 12) / columns.length,
- );
+ // The widget's padding (p-2 either side) and the gaps between tracks come out
+ // of the width before it is split, so this is the real number.
+ const trackPx =
+ previewWidth === undefined
+ ? null
+ : Math.round(
+ (previewWidth - 16 - (layout.cols - 1) * 12) / layout.cols,
+ );
function toggleExpanded(id: SectionId) {
setExpanded((e) => (e.includes(id) ? e.filter((x) => x !== id) : [...e, id]));
@@ -59,19 +110,19 @@ export function LayoutBoard({
// Put a section at an explicit slot, switching it on first if it was in the
// tray. Both halves go out as one patch so the layout is never briefly
// inconsistent with the flags.
- function place(id: SectionId, col: number, index: number) {
+ function place(id: SectionId, row: number, cell: number, index: number) {
const def = SECTION_BY_ID[id];
const flagPatch = def.enabled(config) ? {} : def.setEnabled(true);
const nextFlags = { ...config, ...flagPatch };
- const base = reconcileColumns(columns, nextFlags);
- onChange({ ...flagPatch, columns: insertAt(base, id, col, index) });
+ const base = reconcileLayout(layout, nextFlags);
+ onChange({ ...flagPatch, layout: insertAt(base, id, row, cell, index) });
}
// Where a drop at this pointer position would land: the first card whose
- // midpoint is below the cursor, or the end of the column.
- function indexFromPointer(colEl: HTMLElement, clientY: number): number {
+ // midpoint is below the cursor, or the end of the cell.
+ function indexFromPointer(cellEl: HTMLElement, clientY: number): number {
const cards = Array.from(
- colEl.querySelectorAll<HTMLElement>("[data-section-card]"),
+ cellEl.querySelectorAll<HTMLElement>("[data-section-card]"),
);
for (let i = 0; i < cards.length; i++) {
const r = cards[i].getBoundingClientRect();
@@ -80,8 +131,11 @@ export function LayoutBoard({
return cards.length;
}
+ const lastRow = layout.rows.length - 1;
+ const lastCellOfLastRow = layout.rows[lastRow].cells.length - 1;
+
return (
- <div className="flex flex-col gap-3">
+ <div className="@container flex flex-col gap-3">
<div className="flex flex-wrap items-center gap-x-4 gap-y-2">
<div className="flex items-center gap-2">
<span className="text-sm font-medium">Columns</span>
@@ -91,10 +145,10 @@ export function LayoutBoard({
key={n}
type="button"
aria-label={`${n} column${n === 1 ? "" : "s"}`}
- aria-pressed={columns.length === n}
- onClick={() => onChange({ columns: setColumnCount(columns, n) })}
+ aria-pressed={layout.cols === n}
+ onClick={() => onChange({ layout: setColumnCount(layout, n) })}
className={`h-7 w-7 rounded border text-xs font-medium ${
- columns.length === n
+ layout.cols === n
? "border-primary bg-primary text-primary-foreground"
: "border-border hover:bg-muted"
}`}
@@ -105,88 +159,283 @@ export function LayoutBoard({
</div>
</div>
<p className="text-xs text-muted-foreground">
- Drag a section to move it, or use the arrows on each card.
+ Drag a section to move it, or use the arrows on each card. The rail on
+ the left sets each row's height.
</p>
</div>
{/* The board itself sits on `surface` so it reads as a canvas the `card`
strips are resting on, rather than more of the page. */}
<div className="rounded-lg border border-border bg-surface p-3">
- <div className="flex flex-col gap-3 sm:flex-row sm:items-start">
- {columns.map((col, ci) => (
- <div key={ci} className="flex min-w-0 flex-1 flex-col gap-1.5">
- <div className="flex items-baseline justify-between gap-2 px-0.5">
- <span className="text-xs font-medium uppercase tracking-wide text-muted-foreground">
- Column {ci + 1}
- </span>
- <span
- className={`text-xs tabular-nums ${
- columnPx < NARROW_PX
- ? "text-warning"
- : "text-muted-foreground/70"
- }`}
- title={
- columnPx < NARROW_PX
- ? "Too narrow for these strips to read at the current preview size"
- : "Width this column gets at the current preview size"
- }
+ {/* Top ruler: one entry per grid track, carrying the width it gets. */}
+ {trackPx !== null && (
+ <div className="mb-1.5 flex gap-1.5">
+ <div aria-hidden className="w-9 shrink-0" />
+ <div className={`grid flex-1 gap-1.5 ${BOARD_COLS[layout.cols]} ${STACKED}`}>
+ {Array.from({ length: layout.cols }, (_, i) => (
+ <div
+ key={i}
+ className="flex items-baseline justify-between gap-2 px-0.5"
>
- {columnPx}px
- </span>
- </div>
- <ul
- onDragOver={(e) => {
- if (!draggingId) return;
- e.preventDefault();
- e.dataTransfer.dropEffect = "move";
- const index = indexFromPointer(e.currentTarget, e.clientY);
- setDrop((d) =>
- d && d.col === ci && d.index === index ? d : { col: ci, index },
- );
- }}
- onDrop={(e) => {
- if (!draggingId) return;
- e.preventDefault();
- place(draggingId, ci, indexFromPointer(e.currentTarget, e.clientY));
- clearDrag();
- }}
- className="flex min-h-16 flex-col gap-1.5 rounded-md p-1"
+ <span className="text-xs font-medium uppercase tracking-wide text-muted-foreground">
+ {layout.cols === 1 ? "Width" : `Col ${i + 1}`}
+ </span>
+ <span
+ className={`text-xs tabular-nums ${
+ trackPx < NARROW_PX
+ ? "text-warning"
+ : "text-muted-foreground/70"
+ }`}
+ title={
+ trackPx < NARROW_PX
+ ? "Too narrow for these strips to read at the current preview size"
+ : "Width this track gets at the current preview size"
+ }
+ >
+ {trackPx}px
+ </span>
+ </div>
+ ))}
+ </div>
+ </div>
+ )}
+
+ <div className="flex flex-col gap-1.5">
+ {layout.rows.map((row, ri) => (
+ <div key={ri} className="flex gap-1.5">
+ <RowHeightButton
+ index={ri}
+ size={row.size}
+ onToggle={() =>
+ onChange({
+ layout: setRowSize(
+ layout,
+ ri,
+ row.size === "fill" ? "auto" : "fill",
+ ),
+ })
+ }
+ />
+ <div
+ className={`grid flex-1 gap-1.5 ${BOARD_COLS[layout.cols]} ${STACKED}`}
>
- {col.map((id, index) => (
- <Fragment key={id}>
- <DropLine active={drop?.col === ci && drop.index === index} />
- <SectionCard
- def={SECTION_BY_ID[id]}
- config={config}
- col={ci}
- index={index}
- colLength={col.length}
- columnCount={columns.length}
- dragging={draggingId === id}
- onChange={onChange}
- onMove={(dir) =>
- onChange({ columns: moveWithin(columns, ci, index, dir) })
- }
- onMoveColumn={(dir) =>
- onChange({ columns: moveToColumn(columns, ci, index, dir) })
- }
- onDragStart={() => setDraggingId(id)}
- onDragEnd={clearDrag}
- expanded={expanded.includes(id)}
- onToggleExpanded={() => toggleExpanded(id)}
- />
- </Fragment>
+ {row.cells.map((cell, ci) => (
+ <div
+ key={ci}
+ className={`flex min-w-0 flex-col gap-1 rounded-md border p-1 ${
+ BOARD_SPAN[cell.span] ?? "col-span-1"
+ } ${STACKED_CELL} ${
+ cell.sections.length === 0
+ ? "border-dashed border-border"
+ : "border-border/60"
+ }`}
+ >
+ <div className="flex items-center gap-1">
+ {layout.cols > 1 && (
+ <>
+ <button
+ type="button"
+ className={CHIP}
+ disabled={cell.span <= 1}
+ aria-label={`Narrow row ${ri + 1} cell ${ci + 1}`}
+ title="Narrow this cell by one track"
+ onClick={() =>
+ onChange({
+ layout: setCellSpan(layout, ri, ci, cell.span - 1),
+ })
+ }
+ >
+ ◀
+ </button>
+ <button
+ type="button"
+ className={CHIP}
+ disabled={cell.span >= layout.cols}
+ aria-label={`Widen row ${ri + 1} cell ${ci + 1}`}
+ title="Widen this cell by one track — it swallows the cell beside it"
+ onClick={() =>
+ onChange({
+ layout: setCellSpan(layout, ri, ci, cell.span + 1),
+ })
+ }
+ >
+ ▶
+ </button>
+ </>
+ )}
+ <button
+ type="button"
+ aria-label={`Row ${ri + 1} cell ${ci + 1} flow`}
+ aria-pressed={cell.flow === "row"}
+ title={
+ cell.flow === "row"
+ ? "A dense rail: one line, shared border, shortened labels"
+ : "A stack: each section on its own"
+ }
+ onClick={() =>
+ onChange({
+ layout: setCellFlow(
+ layout,
+ ri,
+ ci,
+ cell.flow === "row" ? "col" : "row",
+ ),
+ })
+ }
+ className={`${CHIP} ${
+ cell.flow === "row"
+ ? "border-border-strong bg-muted text-foreground"
+ : ""
+ }`}
+ >
+ {cell.flow === "row" ? "⇄" : "⇅"}
+ </button>
+ <span className="ml-auto text-[10px] tabular-nums text-muted-foreground/60">
+ {cell.span}/{layout.cols}
+ </span>
+ {row.cells.length > 1 && (
+ <button
+ type="button"
+ className={CHIP}
+ aria-label={`Remove cell ${ci + 1} from row ${ri + 1}`}
+ title="Remove this cell; anything in it moves next door"
+ onClick={() =>
+ onChange({ layout: removeCell(layout, ri, ci) })
+ }
+ >
+ ✕
+ </button>
+ )}
+ </div>
+ <ul
+ onDragOver={(e) => {
+ if (!draggingId) return;
+ e.preventDefault();
+ e.dataTransfer.dropEffect = "move";
+ const index = indexFromPointer(e.currentTarget, e.clientY);
+ setDrop((d) =>
+ d && d.row === ri && d.cell === ci && d.index === index
+ ? d
+ : { row: ri, cell: ci, index },
+ );
+ }}
+ onDrop={(e) => {
+ if (!draggingId) return;
+ e.preventDefault();
+ place(
+ draggingId,
+ ri,
+ ci,
+ indexFromPointer(e.currentTarget, e.clientY),
+ );
+ clearDrag();
+ }}
+ className="flex min-h-16 flex-col gap-1.5 rounded p-1"
+ >
+ {cell.sections.map((id, index) => (
+ <Fragment key={id}>
+ <DropLine
+ active={
+ drop?.row === ri &&
+ drop.cell === ci &&
+ drop.index === index
+ }
+ />
+ <SectionCard
+ def={SECTION_BY_ID[id]}
+ config={config}
+ row={ri}
+ cell={ci}
+ index={index}
+ cellLength={cell.sections.length}
+ cellCount={row.cells.length}
+ rowCount={layout.rows.length}
+ dragging={draggingId === id}
+ onChange={onChange}
+ onMove={(dir) =>
+ onChange({
+ layout: moveWithin(layout, ri, ci, index, dir),
+ })
+ }
+ onMoveCell={(dir) =>
+ onChange({
+ layout: moveToCell(layout, ri, ci, index, dir),
+ })
+ }
+ onMoveRow={(dir) =>
+ onChange({
+ layout: moveToRow(layout, ri, ci, index, dir),
+ })
+ }
+ onDragStart={() => setDraggingId(id)}
+ onDragEnd={clearDrag}
+ expanded={expanded.includes(id)}
+ onToggleExpanded={() => toggleExpanded(id)}
+ />
+ </Fragment>
+ ))}
+ <DropLine
+ active={
+ drop?.row === ri &&
+ drop.cell === ci &&
+ drop.index === cell.sections.length
+ }
+ />
+ {cell.sections.length === 0 && (
+ <li className="px-1 py-3 text-center text-xs text-muted-foreground/70">
+ Empty — drop a section here
+ </li>
+ )}
+ </ul>
+ </div>
))}
- <DropLine active={drop?.col === ci && drop.index === col.length} />
- {col.length === 0 && (
- <li className="px-1 py-3 text-center text-xs text-muted-foreground/70">
- Empty — drop a section here
- </li>
- )}
- </ul>
+ </div>
</div>
))}
</div>
+
+ {/* The rail's foot: the row and cell lifecycle, kept together so adding
+ a place to put something is one gesture away from the rail that
+ sizes it. */}
+ <div className="mt-2 flex flex-wrap items-center gap-1.5">
+ <button
+ type="button"
+ className={CHIP}
+ aria-label="Add a row"
+ title="Add a row below"
+ disabled={layout.rows.length >= MAX_ROWS}
+ onClick={() =>
+ onChange({ layout: setRowCount(layout, layout.rows.length + 1) })
+ }
+ >
+ + Row
+ </button>
+ <button
+ type="button"
+ className={CHIP}
+ aria-label="Remove the last row"
+ title="Remove the last row; anything in it moves up"
+ disabled={layout.rows.length <= 1}
+ onClick={() =>
+ onChange({ layout: setRowCount(layout, layout.rows.length - 1) })
+ }
+ >
+ − Row
+ </button>
+ <button
+ type="button"
+ className={CHIP}
+ aria-label={`Add a cell to row ${lastRow + 1}`}
+ title="Split the last row into another cell"
+ disabled={
+ layout.rows[lastRow].cells.length >= layout.cols ||
+ lastCellOfLastRow < 0
+ }
+ onClick={() => onChange({ layout: addCell(layout, lastRow) })}
+ >
+ + Cell
+ </button>
+ </div>
</div>
<div
@@ -232,7 +481,12 @@ export function LayoutBoard({
type="checkbox"
checked={false}
onChange={() =>
- place(s.id, columns.length - 1, columns[columns.length - 1].length)
+ place(
+ s.id,
+ lastRow,
+ lastCellOfLastRow,
+ layout.rows[lastRow].cells[lastCellOfLastRow].sections.length,
+ )
}
className="h-3.5 w-3.5"
/>
@@ -251,6 +505,49 @@ export function LayoutBoard({
);
}
+// The signature control: a row's height, drawn as its height. `auto` is a short
+// hairline box; `fill` is a tall one carrying a vertical spring mark, which is
+// the only place on the board that spends any ink.
+function RowHeightButton({
+ index,
+ size,
+ onToggle,
+}: {
+ index: number;
+ size: "auto" | "fill";
+ onToggle: () => void;
+}) {
+ const fill = size === "fill";
+ return (
+ <button
+ type="button"
+ aria-label={`Row ${index + 1} height`}
+ aria-pressed={fill}
+ onClick={onToggle}
+ title={
+ fill
+ ? "Fills the widget's leftover height; sections inside it scroll. Click for content height."
+ : "Content height, like every row before rows existed. Click to make it fill."
+ }
+ className={`flex w-9 shrink-0 flex-col items-center justify-center gap-1 self-stretch rounded-md border text-[10px] uppercase tracking-wide motion-safe:transition-colors ${
+ fill
+ ? "border-primary bg-primary/10 text-primary"
+ : "border-border text-muted-foreground/70 hover:bg-muted"
+ }`}
+ >
+ <span
+ aria-hidden
+ className={`block w-4 rounded-sm border ${
+ fill
+ ? "h-8 border-primary bg-[repeating-linear-gradient(180deg,transparent_0_3px,currentColor_3px_4px)]"
+ : "h-2 border-current"
+ }`}
+ />
+ {fill ? "fill" : "auto"}
+ </button>
+ );
+}
+
// The insertion marker. Always in the DOM at every slot so the list doesn't
// reflow as it moves between them — only its color changes.
function DropLine({ active }: { active: boolean }) {
diff --git a/editor/app/widget/builder/components/SectionCard.tsx b/editor/app/widget/builder/components/SectionCard.tsx
@@ -19,14 +19,17 @@ const BTN =
export function SectionCard({
def,
config,
- col,
+ row,
+ cell,
index,
- colLength,
- columnCount,
+ cellLength,
+ cellCount,
+ rowCount,
dragging,
onChange,
onMove,
- onMoveColumn,
+ onMoveCell,
+ onMoveRow,
onDragStart,
onDragEnd,
expanded,
@@ -34,14 +37,17 @@ export function SectionCard({
}: {
def: SectionDef;
config: WidgetConfig;
- col: number;
+ row: number;
+ cell: number;
index: number;
- colLength: number;
- columnCount: number;
+ cellLength: number;
+ cellCount: number;
+ rowCount: number;
dragging: boolean;
onChange: (patch: Partial<WidgetConfig>) => void;
onMove: (dir: -1 | 1) => void;
- onMoveColumn: (dir: -1 | 1) => void;
+ onMoveCell: (dir: -1 | 1) => void;
+ onMoveRow: (dir: -1 | 1) => void;
onDragStart: () => void;
onDragEnd: () => void;
expanded: boolean;
@@ -63,8 +69,8 @@ export function SectionCard({
: "hover:border-border-strong"
}`}
>
- {/* Wraps rather than truncates: in a three-column board there isn't room
- for the name and the five controls on one line, and the name is the
+ {/* Wraps rather than truncates: in a multi-track board there isn't room
+ for the name and the move controls on one line, and the name is the
part you can't do without. */}
<div className="flex flex-wrap items-start gap-x-2 gap-y-1">
<div className="flex min-w-[7rem] flex-1 flex-col gap-1">
@@ -104,37 +110,64 @@ export function SectionCard({
<button
type="button"
className={BTN}
- disabled={index === colLength - 1}
+ disabled={index === cellLength - 1}
onClick={() => onMove(1)}
aria-label={`Move ${def.label} down`}
title="Move down"
>
↓
</button>
- {columnCount > 1 && (
+ {cellCount > 1 && (
<>
<button
type="button"
className={BTN}
- disabled={col === 0}
- onClick={() => onMoveColumn(-1)}
+ disabled={cell === 0}
+ onClick={() => onMoveCell(-1)}
aria-label={`Move ${def.label} left`}
- title="Move to the previous column"
+ title="Move to the previous cell in this row"
>
◀
</button>
<button
type="button"
className={BTN}
- disabled={col === columnCount - 1}
- onClick={() => onMoveColumn(1)}
+ disabled={cell === cellCount - 1}
+ onClick={() => onMoveCell(1)}
aria-label={`Move ${def.label} right`}
- title="Move to the next column"
+ title="Move to the next cell in this row"
>
▶
</button>
</>
)}
+ {/* Rows are the new dimension, so they need their own pair — every
+ drag has to stay reachable from the keyboard, which is what the
+ arrows on this card have always been for. */}
+ {rowCount > 1 && (
+ <>
+ <button
+ type="button"
+ className={BTN}
+ disabled={row === 0}
+ onClick={() => onMoveRow(-1)}
+ aria-label={`Move ${def.label} to the row above`}
+ title="Move to the row above"
+ >
+ ▲
+ </button>
+ <button
+ type="button"
+ className={BTN}
+ disabled={row === rowCount - 1}
+ onClick={() => onMoveRow(1)}
+ aria-label={`Move ${def.label} to the row below`}
+ title="Move to the row below"
+ >
+ ▼
+ </button>
+ </>
+ )}
{def.options.length > 0 && (
<button
type="button"
diff --git a/editor/app/widget/builder/components/WidgetBuilder.tsx b/editor/app/widget/builder/components/WidgetBuilder.tsx
@@ -7,18 +7,33 @@ import {
WIDGET_DEFAULTS,
type WidgetConfig,
} from "../../lib/config";
-import { GlobalOptions } from "../../components/WidgetFields";
+import { WidgetMenu } from "../../components/WidgetMenu";
+import type { PresetsPayload } from "../../components/PresetsRow";
import { loadWidgetConfig, saveWidgetConfig } from "../widgetConfigStorage";
-import { LayoutBoard } from "./LayoutBoard";
// The two wide presets exist because columns need width to be worth having —
-// the original three are all narrower than two readable columns.
-const SIZE_PRESETS: { label: string; width: number; height: number }[] = [
+// the original three are all narrower than two readable columns. `Viewport` is
+// the one that isn't a pinned-window size: a fill row only shows what it does
+// when the frame has a height to leave over, so arranging one at 200px tall
+// would be arranging it blind.
+const SIZE_PRESETS: {
+ label: string;
+ width: number;
+ height: number;
+ // CSS override for the iframe box, for a size that isn't a fixed pixel pair.
+ css?: { width: string; height: string };
+}[] = [
{ label: "Small", width: 320, height: 200 },
{ label: "Medium", width: 380, height: 320 },
{ label: "Tall", width: 360, height: 520 },
{ label: "Wide", width: 640, height: 260 },
{ label: "Panel", width: 720, height: 420 },
+ {
+ label: "Viewport",
+ width: 720,
+ height: 640,
+ css: { width: "100%", height: "80vh" },
+ },
];
// How long the preview waits before re-navigating. Changing an iframe's src is a
@@ -26,7 +41,11 @@ const SIZE_PRESETS: { label: string; width: number; height: number }[] = [
// field updates instantly and only the preview is debounced.
const PREVIEW_DEBOUNCE_MS = 250;
-export function WidgetBuilder() {
+export function WidgetBuilder({
+ initialPresets,
+}: {
+ initialPresets?: PresetsPayload;
+}) {
const [config, setConfig] = useState<WidgetConfig>(WIDGET_DEFAULTS);
const [size, setSize] = useState(SIZE_PRESETS[0]);
const [origin, setOrigin] = useState("");
@@ -91,11 +110,12 @@ export function WidgetBuilder() {
return (
<div className="flex flex-col gap-6 xl:flex-row xl:items-start">
<div className="flex min-w-0 flex-1 flex-col gap-4">
- <LayoutBoard config={config} onChange={patch} previewWidth={size.width} />
- <fieldset className="flex flex-col gap-2 rounded-lg border border-border p-3">
- <legend className="px-1 text-sm font-medium">Whole widget</legend>
- <GlobalOptions config={config} onChange={patch} />
- </fieldset>
+ <WidgetMenu
+ config={config}
+ onChange={patch}
+ initialPresets={initialPresets}
+ previewWidth={size.width}
+ />
</div>
<div className="flex flex-col gap-3 xl:shrink-0">
@@ -131,7 +151,8 @@ export function WidgetBuilder() {
<code className="font-mono"><iframe></code> will show. Sections
in the <strong>Off</strong> tray are simply absent from it — the link
only names what is on, which is why an old link still works and still
- renders in the original order. The{" "}
+ renders in the original order. A <strong>preset</strong> is just this
+ link under a name, so loading one anywhere gives the same widget. The{" "}
<strong>settings gear</strong> reopens these same controls inside the
widget; turn it off for a locked-down link.{" "}
<strong>Open popup</strong> launches a chromeless window at the
@@ -166,11 +187,14 @@ export function WidgetBuilder() {
src={previewUrl}
width={size.width}
height={size.height}
+ style={size.css}
className="block"
/>
</div>
<p className="text-xs text-muted-foreground">
- Previewing at {size.width}×{size.height}px.
+ {size.css
+ ? "Previewing at the width of this panel, 80% of the window's height — enough for a fill row to show what it does."
+ : `Previewing at ${size.width}×${size.height}px.`}
</p>
</div>
</div>
diff --git a/editor/app/widget/builder/page.tsx b/editor/app/widget/builder/page.tsx
@@ -1,12 +1,27 @@
import type { Metadata } from "next";
+import { getPaths } from "yt-dlp-transcript-common/lib/paths";
+import { readWidgetPresets } from "yt-dlp-transcript-common/lib/widgetPresets";
+import { BUILT_IN_PRESETS } from "../lib/builtInPresets";
import { WidgetBuilder } from "./components/WidgetBuilder";
export const metadata: Metadata = { title: "Monitor widget" };
+// The saved presets are read per request, not baked in: without this the page
+// prerenders static at build time and seeds the preset row with whatever
+// happened to be on disk when the release was cut. (Dev renders per request
+// regardless, so only a real build catches it.)
+export const dynamic = "force-dynamic";
+
// Lives inside the normal app shell (AppFrame strips chrome only on exactly
// "/widget"). The board below is a scale model of the widget: arrange it there,
// preview the real bare widget in an iframe, copy the link.
-export default function WidgetBuilderPage() {
+export default async function WidgetBuilderPage() {
+ // SSR-seed the preset row so the saved list is there on first paint; the
+ // in-widget copy has no server parent and fetches /api/widget/presets instead.
+ const presets = {
+ builtIn: BUILT_IN_PRESETS,
+ saved: await readWidgetPresets(getPaths()),
+ };
return (
<div className="flex flex-col gap-4">
<div className="flex items-center justify-between">
@@ -16,11 +31,13 @@ export default function WidgetBuilderPage() {
A read-only monitor with no sidebar, sized to sit in a small pinned
window or an embedded <code className="font-mono"><iframe></code>.
The board below is laid out the way the widget will be — drag the strips
- into the order you want to read them in, split them across columns if
- you're giving it the width, and drop the ones you don't want
- into the tray. Then copy the link.
+ into the order you want to read them in, split them across rows and
+ columns if you're giving it the width, and drop the ones you
+ don't want into the tray. Give a row the leftover height with the
+ rail on its left and the sections inside it scroll under their own
+ headings. Then save it as a preset, or copy the link.
</p>
- <WidgetBuilder />
+ <WidgetBuilder initialPresets={presets} />
</div>
);
}
diff --git a/editor/app/widget/builder/widgetConfigStorage.ts b/editor/app/widget/builder/widgetConfigStorage.ts
@@ -5,32 +5,68 @@
// what actually renders (see buildWidgetQuery).
import { WIDGET_DEFAULTS, type WidgetConfig } from "../lib/config";
-import { reconcileColumns, type Columns } from "../lib/placement";
+import {
+ MAX_COLUMNS,
+ MAX_ROWS,
+ reconcileLayout,
+ type Cell,
+ type Layout,
+ type Row,
+} from "../lib/placement";
import { SECTION_BY_ID, type SectionId } from "../lib/sections";
const KEY = "ytdlp-tb:widget-config";
-// v2 adds the section layout. A v1 entry resets to defaults, which is right:
-// it predates layouts and has nothing to carry forward but the flags.
-const VERSION = 2;
+// v3 replaces columns with rows of cells. A v1 or v2 entry resets to defaults,
+// which is right for both: each predates the dimension the next one added, and
+// has nothing to carry forward but flags that are already the defaults' shape.
+const VERSION = 3;
type StoredWidgetConfig = { v: typeof VERSION; config: WidgetConfig };
// The layout is the one stored field that isn't a boolean, a string or a
-// number, so it needs its own validation: an array of arrays of section ids we
-// still recognize. Anything else is discarded rather than trusted.
-function parseColumns(v: unknown): Columns | null {
- if (!Array.isArray(v) || v.length === 0) return null;
- const columns: Columns = [];
- for (const col of v) {
- if (!Array.isArray(col)) return null;
- const ids: SectionId[] = [];
- for (const id of col) {
- if (typeof id !== "string" || !(id in SECTION_BY_ID)) continue;
- ids.push(id as SectionId);
+// number, so it needs its own validation: a track count, rows of cells, and
+// section ids we still recognize. Anything shaped wrong is discarded rather
+// than trusted — this is localStorage, which anything on the origin can write.
+function parseSections(v: unknown): SectionId[] {
+ if (!Array.isArray(v)) return [];
+ const ids: SectionId[] = [];
+ for (const id of v) {
+ if (typeof id !== "string" || !(id in SECTION_BY_ID)) continue;
+ ids.push(id as SectionId);
+ }
+ return ids;
+}
+
+function parseLayout(v: unknown): Layout | null {
+ if (!v || typeof v !== "object" || Array.isArray(v)) return null;
+ const raw = v as Record<string, unknown>;
+ const cols =
+ typeof raw.cols === "number" && Number.isFinite(raw.cols)
+ ? Math.max(1, Math.min(MAX_COLUMNS, Math.round(raw.cols)))
+ : null;
+ if (cols === null) return null;
+ if (!Array.isArray(raw.rows) || raw.rows.length === 0) return null;
+ const rows: Row[] = [];
+ for (const rowRaw of raw.rows.slice(0, MAX_ROWS)) {
+ if (!rowRaw || typeof rowRaw !== "object") return null;
+ const r = rowRaw as Record<string, unknown>;
+ if (!Array.isArray(r.cells) || r.cells.length === 0) return null;
+ const cells: Cell[] = [];
+ for (const cellRaw of r.cells) {
+ if (!cellRaw || typeof cellRaw !== "object") return null;
+ const c = cellRaw as Record<string, unknown>;
+ cells.push({
+ sections: parseSections(c.sections),
+ span:
+ typeof c.span === "number" && Number.isFinite(c.span)
+ ? Math.max(1, Math.min(cols, Math.round(c.span)))
+ : 1,
+ flow: c.flow === "row" ? "row" : "col",
+ });
}
- columns.push(ids);
+ rows.push({ size: r.size === "fill" ? "fill" : "auto", cells });
}
- return columns;
+ return { cols, rows };
}
// Overlay validated fields onto the defaults so the saved form survives the
@@ -51,9 +87,9 @@ function parseStored(raw: string): WidgetConfig | null {
const next: WidgetConfig = { ...WIDGET_DEFAULTS };
for (const key of Object.keys(WIDGET_DEFAULTS) as (keyof WidgetConfig)[]) {
const v = r[key];
- if (key === "columns") {
- const columns = parseColumns(v);
- if (columns) next.columns = columns;
+ if (key === "layout") {
+ const layout = parseLayout(v);
+ if (layout) next.layout = layout;
continue;
}
if (key === "channel") {
@@ -72,7 +108,10 @@ function parseStored(raw: string): WidgetConfig | null {
}
// Re-normalize rather than trust: a stored layout could name a section whose
// flag was stored off (or miss one stored on), and the flags are what decide.
- return { ...next, columns: reconcileColumns(next.columns, next) };
+ // reconcileLayout also re-imposes the grid invariants, so a hand-edited entry
+ // with spans that don't add up comes back a valid grid rather than a broken
+ // one.
+ return { ...next, layout: reconcileLayout(next.layout, next) };
}
export function loadWidgetConfig(): WidgetConfig {
diff --git a/editor/app/widget/components/MonitorWidget.tsx b/editor/app/widget/components/MonitorWidget.tsx
@@ -1,6 +1,6 @@
"use client";
-import { Fragment, useState, type ReactNode } from "react";
+import { Fragment, useState, type CSSProperties, type ReactNode } from "react";
import { formatDuration, formatBytes } from "yt-dlp-transcript-common/lib/format";
import { usePolledPayload, useNow } from "../lib/usePolledPayload";
import { fmtTime } from "../lib/relativeTime";
@@ -27,8 +27,14 @@ import {
patchWidgetConfig,
type WidgetConfig,
} from "../lib/config";
+import { hasFillRow, type Cell, type Layout, type Row } from "../lib/placement";
import type { SectionId } from "../lib/sections";
-import { WidgetConfigForm } from "./WidgetConfigForm";
+import {
+ SectionFrameProvider,
+ WidgetSection,
+ useSectionFrame,
+} from "./WidgetSection";
+import { WidgetMenu } from "./WidgetMenu";
import { WidgetControls } from "./WidgetControls";
// Read-only monitor widget. Reuses the existing ~1s poll pattern from
@@ -58,9 +64,13 @@ export function MonitorWidget({
// The controls row needs live `paused` state, so poll workers whenever either
// the Workers section or the controls are shown.
const workersEnabled = config.workers || config.controls;
+ // The DISK strip rides this payload too, so the gate can't just be
+ // `config.jobs`: `?jobs=0&disk=1` polled nothing and rendered nothing, with
+ // no clue that it was the gate rather than the disk gate being off. Same
+ // pre-existing shape as the sync payload's gate below.
const { data: jobsPayload } = usePolledPayload<ActiveJobsPayload>(
"/api/jobs/active",
- config.jobs,
+ config.jobs || config.disk,
pollMs,
initialJobs,
);
@@ -91,10 +101,15 @@ export function MonitorWidget({
// Freshness/scheduler markers move on the order of minutes, so poll no faster
// than every 15s regardless of the configured cadence. `refetchSync` lets the
// Sync button refresh the readouts the moment a sweep is queued.
+ //
+ // FOUR sections read this one payload, not two. The gate used to name only
+ // lastSync and scheduler, so `/widget?backfill=1` polled nothing and rendered
+ // nothing at all unless one of those happened to be on too. Controls is in the
+ // list because the backfill pause reads its lane state from here.
const { data: syncData, refetch: refetchSync } =
usePolledPayload<WidgetSyncPayload>(
"/api/widget/sync",
- config.lastSync || config.scheduler,
+ config.lastSync || config.scheduler || config.backfill || config.controls,
Math.max(pollMs, 15000),
null,
);
@@ -171,8 +186,11 @@ export function MonitorWidget({
confirmSyncAll={config.syncConfirm}
paused={workersPayload?.paused ?? false}
downloadsPaused={workersPayload?.downloadsPaused ?? false}
+ backfillEnabled={syncData?.backfill.enabled ?? false}
+ backfillAnyKind={syncData?.backfill.anyKind ?? false}
onWorkersChange={refetchWorkers}
onSynced={refetchSync}
+ onBackfillChange={refetchSync}
/>
);
case "lastSync":
@@ -238,40 +256,151 @@ export function MonitorWidget({
}
}
+ const layout = config.layout;
+ // One fill row anywhere changes what the widget IS: content-height (the
+ // pre-rows behaviour, and still the default) becomes exactly-its-frame, so the
+ // fill rows have a leftover to share. `h-dvh` is right for both a pinned popup
+ // and an <iframe>, which is what the widget is always embedded as.
+ const fill = hasFillRow(layout);
+ const stack = config.stackNarrow && layout.cols > 1;
+ // The one genuinely combinatorial part of the grid — N rows, each auto or 1fr
+ // — so it rides a custom property. Everything else stays a literal class,
+ // because an inline `style` would out-specify the stackNarrow overrides.
+ const rowTracks = layout.rows
+ .map((r) => (r.size === "fill" ? "minmax(0,1fr)" : "auto"))
+ .join(" ");
+
return (
// @container so the stack-when-narrow thresholds below measure the widget's
// own box — the popup or iframe it was embedded at — rather than the
// viewport, which for an embedded widget is the host page's.
- <div className="@container relative p-2 text-foreground">
+ <div
+ className={`@container relative p-2 text-foreground ${
+ fill ? "flex h-dvh flex-col" : ""
+ }`}
+ >
{settingsLayer}
- <div className={columnsClass(config)}>
- {config.columns.map((col, i) => (
- <div key={i} className="flex min-w-0 flex-1 flex-col gap-3">
- {col.map((id) => (
- <Fragment key={id}>{renderSection(id)}</Fragment>
- ))}
- </div>
- ))}
+ <div
+ className={`grid grid-rows-[var(--wrows)] gap-3 ${
+ COLS_CLASS[layout.cols] ?? "grid-cols-1"
+ } ${fill ? "min-h-0 flex-1" : ""} ${
+ stack ? (STACK_GRID[layout.cols] ?? "") : ""
+ }`}
+ style={{ "--wrows": rowTracks } as CSSProperties}
+ >
+ {layout.rows.map((row, r) =>
+ row.cells.map((cell, c) => (
+ <WidgetCell
+ key={`${r}-${c}`}
+ cell={cell}
+ row={row}
+ cols={layout.cols}
+ stack={stack}
+ render={renderSection}
+ />
+ )),
+ )}
</div>
</div>
);
}
-// Tailwind needs literal class names in the source, so the stack thresholds are
-// a static map rather than an interpolated `@min-[${n}rem]:`. Each is the width
-// below which that many columns stop being readable at all.
-const STACK_AT: Record<number, string> = {
- 2: "flex flex-col gap-3 @min-[24rem]:flex-row",
- 3: "flex flex-col gap-3 @min-[36rem]:flex-row",
+// Tailwind needs literal class names in the source, so the track counts, the
+// spans and the stack thresholds are static maps rather than interpolated
+// `grid-cols-${n}`.
+const COLS_CLASS: Record<number, string> = {
+ 1: "grid-cols-1",
+ 2: "grid-cols-2",
+ 3: "grid-cols-3",
+ 4: "grid-cols-4",
+ 5: "grid-cols-5",
+ 6: "grid-cols-6",
+};
+
+const SPAN_CLASS: Record<number, string> = {
+ 1: "col-span-1",
+ 2: "col-span-2",
+ 3: "col-span-3",
+ 4: "col-span-4",
+ 5: "col-span-5",
+ 6: "col-span-6",
+};
+
+// Collapse back to one stack below the width at which that many tracks stop
+// being readable at all. Container queries, not viewport media queries, so an
+// embedded widget measures its own box.
+const STACK_GRID: Record<number, string> = {
+ 2: "@max-[24rem]:grid-cols-1 @max-[24rem]:grid-rows-none",
+ 3: "@max-[36rem]:grid-cols-1 @max-[36rem]:grid-rows-none",
+ 4: "@max-[48rem]:grid-cols-1 @max-[48rem]:grid-rows-none",
+ 5: "@max-[60rem]:grid-cols-1 @max-[60rem]:grid-rows-none",
+ 6: "@max-[72rem]:grid-cols-1 @max-[72rem]:grid-rows-none",
+};
+
+// The matching per-cell override: a stacked grid has one track, so nothing may
+// still claim two.
+const STACK_CELL: Record<number, string> = {
+ 2: "@max-[24rem]:col-span-1",
+ 3: "@max-[36rem]:col-span-1",
+ 4: "@max-[48rem]:col-span-1",
+ 5: "@max-[60rem]:col-span-1",
+ 6: "@max-[72rem]:col-span-1",
};
-function columnsClass(config: WidgetConfig): string {
- const cols = config.columns.length;
- if (cols <= 1) return "flex flex-col gap-3";
- // Columns are honored at every width by default: an arrangement you made on
- // purpose shouldn't quietly undo itself. `stackNarrow` opts into collapsing.
- if (config.stackNarrow) return STACK_AT[cols] ?? "flex flex-col gap-3";
- return "flex flex-row gap-3";
+// One grid cell: a stack of sections, or a dense rail of them.
+//
+// Both variants style their CHILDREN through `[&>section]` variants rather than
+// wrapping each one in a div. That is deliberate: a section can render nothing
+// (the backfill strip with no backfill feature on, say), and a wrapper div would
+// still take its share of a fill row's height, or draw a divider with nothing
+// beside it. Selecting the elements that actually exist can't do that.
+function WidgetCell({
+ cell,
+ row,
+ cols,
+ stack,
+ render,
+}: {
+ cell: Cell;
+ row: Row;
+ cols: Layout["cols"];
+ stack: boolean;
+ render: (id: SectionId) => ReactNode;
+}) {
+ const fillRow = row.size === "fill";
+ const place = `${SPAN_CLASS[cell.span] ?? "col-span-1"} ${
+ stack ? (STACK_CELL[cols] ?? "") : ""
+ }`;
+ const children = cell.sections.map((id) => (
+ <Fragment key={id}>{render(id)}</Fragment>
+ ));
+
+ // A rail: one shared border around the lot, hairlines between, and every
+ // strip inside told to shorten itself. Four totals as one instrument cluster.
+ if (cell.flow === "row") {
+ return (
+ <SectionFrameProvider value={{ scroll: false, dense: true }}>
+ <div
+ className={`${place} flex min-w-0 flex-wrap items-center gap-y-1 self-start rounded border border-border bg-card px-2 py-1 [&>section]:px-3 [&>section+section]:border-l [&>section+section]:border-border/60 [&>section:first-child]:pl-0 [&>section:last-child]:pr-0`}
+ >
+ {children}
+ </div>
+ </SectionFrameProvider>
+ );
+ }
+ return (
+ <SectionFrameProvider value={{ scroll: fillRow, dense: false }}>
+ <div
+ className={`${place} flex min-h-0 min-w-0 flex-col gap-3 ${
+ // In a fill row the stacked sections SPLIT the height they were given,
+ // which is what makes "two lists, half each, both scrolling" free.
+ fillRow ? "[&>section]:min-h-0 [&>section]:flex-1" : ""
+ }`}
+ >
+ {children}
+ </div>
+ </SectionFrameProvider>
+ );
}
// Compact state → dot color, mirroring stateBadge() in WorkersView.
@@ -306,22 +435,18 @@ function WorkersStrip({
}) {
const busy = workers.filter((w) => w.busy).length;
return (
- <section aria-label="Workers" className="flex flex-col gap-1.5">
- {showTitle && (
- <div className="flex items-baseline gap-2">
- <h2 className="text-xs font-semibold uppercase tracking-wide text-muted-foreground">
- Workers
- </h2>
- <span className="text-xs text-muted-foreground">
- {busy}/{workers.length} busy
+ <WidgetSection
+ label="Workers"
+ title={showTitle ? "Workers" : undefined}
+ meta={`${busy}/${workers.length} busy`}
+ extra={
+ paused && (
+ <span className="text-[10px] px-1.5 py-0.5 rounded-full bg-warning-soft text-warning border border-warning/30">
+ paused
</span>
- {paused && (
- <span className="text-[10px] px-1.5 py-0.5 rounded-full bg-warning-soft text-warning border border-warning/30">
- paused
- </span>
- )}
- </div>
- )}
+ )
+ }
+ >
{workers.length === 0 ? (
<p className="text-xs text-muted-foreground">No workers configured.</p>
) : (
@@ -347,13 +472,54 @@ function WorkersStrip({
))}
</ul>
)}
- </section>
+ </WidgetSection>
);
}
+// The dense rail has room for a number and not much else, so "3m ago" becomes
+// "3m" and "in 12m" becomes "12m" — the direction is implied by the label.
+function bareTime(t: string | null): string {
+ if (t === null) return "\u2026";
+ return t.replace(/^in /, "").replace(/ ago$/, "");
+}
+
+// 1,204 → "1.2k". Only ever rendered in the dense rail, whose payload arrives on
+// the first client poll, so the locale-dependent formatting can't produce a
+// hydration mismatch.
+function compactCount(n: number): string {
+ return new Intl.NumberFormat(undefined, {
+ notation: "compact",
+ maximumFractionDigits: 1,
+ })
+ .format(n)
+ .toLowerCase();
+}
+
// Compact free-disk indicator. Green when above the floor, red when at/below it
// (downloads blocked). Only rendered when the gate is enabled.
function DiskStrip({ disk }: { disk: DiskStatusView }) {
+ const { dense } = useSectionFrame();
+ if (dense) {
+ return (
+ <section
+ aria-label="Disk space"
+ className={`flex items-center gap-1.5 text-xs ${
+ disk.low ? "text-destructive" : "text-muted-foreground"
+ }`}
+ >
+ <span
+ className={`inline-block h-2 w-2 shrink-0 rounded-full ${
+ disk.low ? "bg-destructive" : "bg-success/60"
+ }`}
+ />
+ <span className="tabular-nums">
+ {formatBytes(disk.freeBytes)} free
+ {/* The floor is why nothing is downloading; it survives the squeeze. */}
+ {disk.low && " \u00b7 downloads paused"}
+ </span>
+ </section>
+ );
+ }
return (
<section
aria-label="Disk space"
@@ -398,10 +564,23 @@ function LastSyncStrip({
now: number | null;
absolute: boolean;
}) {
+ const { dense } = useSectionFrame();
const full =
data.lastSyncAllAt === null
? "never"
: (fmtTime(data.lastSyncAllAt, now, absolute) ?? "…");
+ if (dense) {
+ return (
+ <section
+ aria-label="Last sync"
+ className="flex items-center gap-1.5 text-xs text-muted-foreground"
+ >
+ <span className="tabular-nums">
+ synced {data.lastSyncAllAt === null ? "never" : bareTime(full)}
+ </span>
+ </section>
+ );
+ }
const showChannel =
data.lastIndividualSyncAt !== null &&
(data.lastSyncAllAt === null ||
@@ -433,7 +612,32 @@ function SchedulerStrip({
now: number | null;
absolute: boolean;
}) {
+ const { dense } = useSectionFrame();
const s = data.scheduler;
+ if (dense) {
+ const short = !s.enabled
+ ? "auto off"
+ : s.overdue
+ ? "overdue"
+ : s.nextRunAt !== null
+ ? `auto ${bareTime(fmtTime(s.nextRunAt, now, absolute))}`
+ : "auto on";
+ return (
+ <section
+ aria-label="Auto-sync scheduler"
+ className={`flex items-center gap-1.5 text-xs ${
+ s.overdue ? "text-warning" : "text-muted-foreground"
+ }`}
+ >
+ <span
+ className={`inline-block h-2 w-2 shrink-0 rounded-full ${
+ s.enabled ? "bg-success" : "bg-muted-foreground/50"
+ }`}
+ />
+ <span className="tabular-nums">{short}</span>
+ </section>
+ );
+ }
const nextText = s.overdue
? "due now"
: s.nextRunAt !== null
@@ -476,9 +680,31 @@ function SchedulerStrip({
// Renders nothing when no backfill feature is on: an empty work list because a
// feature is switched off must not read as "all caught up".
function BackfillStrip({ data }: { data: WidgetSyncPayload }) {
+ const { dense } = useSectionFrame();
const b = data.backfill;
if (!b.anyKind) return null;
const done = b.reachable === 0;
+ if (dense) {
+ return (
+ <section
+ aria-label="Backfill"
+ className={`flex items-center gap-1.5 text-xs ${
+ done ? "text-muted-foreground" : "text-warning"
+ }`}
+ >
+ <span
+ className={`inline-block h-2 w-2 shrink-0 rounded-full ${
+ b.sweeping ? "bg-success" : done ? "bg-success/40" : "bg-warning"
+ }`}
+ />
+ <span className="tabular-nums">
+ {compactCount(b.reachable)} backfill
+ {/* Still a separate figure, never summed into the one beside it. */}
+ {b.needsMedia > 0 && ` \u00b7 ${compactCount(b.needsMedia)} need media`}
+ </span>
+ </section>
+ );
+ }
return (
<section
aria-label="Backfill"
@@ -504,6 +730,26 @@ function BackfillStrip({ data }: { data: WidgetSyncPayload }) {
}
function CleanableStrip({ bytes }: { bytes: number }) {
+ const { dense } = useSectionFrame();
+ if (dense) {
+ return (
+ <section
+ aria-label="Cleanable data"
+ className={`flex items-center gap-1.5 text-xs ${
+ bytes > 0 ? "text-warning" : "text-muted-foreground"
+ }`}
+ >
+ <span
+ className={`inline-block h-2 w-2 shrink-0 rounded-full ${
+ bytes > 0 ? "bg-warning" : "bg-muted-foreground/50"
+ }`}
+ />
+ <span className="tabular-nums">
+ {bytes > 0 ? `${formatBytes(bytes)} reclaimable` : "nothing to reclaim"}
+ </span>
+ </section>
+ );
+ }
return (
<section
aria-label="Cleanable data"
@@ -588,27 +834,27 @@ function ActionableStrip({
showTitle: boolean;
interactive: boolean;
}) {
+ // Given a fill row to scroll in, the cap stops being a kindness: a list you
+ // can scroll that still says "+14 more channels" defeats the point of the
+ // height it was just handed.
+ const { scroll } = useSectionFrame();
const { shown, hidden, expanded, toggle } = useExpandableList(
channels,
- ACTIONABLE_LIMIT,
+ scroll ? Number.POSITIVE_INFINITY : ACTIONABLE_LIMIT,
);
return (
- <section aria-label="Needs work" className="flex flex-col gap-1.5">
- {showTitle && (
- <div className="flex items-baseline gap-2">
- <h2 className="text-xs font-semibold uppercase tracking-wide text-muted-foreground">
- Needs work
- </h2>
- <span className="text-xs text-muted-foreground">
- {channels.length} {channels.length === 1 ? "channel" : "channels"}
- </span>
- </div>
- )}
+ <WidgetSection
+ label="Needs work"
+ title={showTitle ? "Needs work" : undefined}
+ meta={`${channels.length} ${channels.length === 1 ? "channel" : "channels"}`}
+ >
{channels.length === 0 ? (
<p className="text-xs text-muted-foreground">Everything's handled.</p>
) : (
<ul
- className={`flex flex-col gap-1.5${expanded ? " max-h-64 overflow-y-auto" : ""}`}
+ className={`flex flex-col gap-1.5${
+ expanded && !scroll ? " max-h-64 overflow-y-auto" : ""
+ }`}
>
{shown.map((c) => (
<li
@@ -659,7 +905,7 @@ function ActionableStrip({
/>
</ul>
)}
- </section>
+ </WidgetSection>
);
}
@@ -679,27 +925,24 @@ function CleanableChannelsStrip({
showTitle: boolean;
interactive: boolean;
}) {
+ const { scroll } = useSectionFrame();
const { shown, hidden, expanded, toggle } = useExpandableList(
channels,
- CLEANABLE_LIMIT,
+ scroll ? Number.POSITIVE_INFINITY : CLEANABLE_LIMIT,
);
return (
- <section aria-label="Needs cleaning" className="flex flex-col gap-1.5">
- {showTitle && (
- <div className="flex items-baseline gap-2">
- <h2 className="text-xs font-semibold uppercase tracking-wide text-muted-foreground">
- Needs cleaning
- </h2>
- <span className="text-xs text-muted-foreground">
- {channels.length} {channels.length === 1 ? "channel" : "channels"}
- </span>
- </div>
- )}
+ <WidgetSection
+ label="Needs cleaning"
+ title={showTitle ? "Needs cleaning" : undefined}
+ meta={`${channels.length} ${channels.length === 1 ? "channel" : "channels"}`}
+ >
{channels.length === 0 ? (
<p className="text-xs text-muted-foreground">Nothing to reclaim.</p>
) : (
<ul
- className={`flex flex-col gap-1.5${expanded ? " max-h-64 overflow-y-auto" : ""}`}
+ className={`flex flex-col gap-1.5${
+ expanded && !scroll ? " max-h-64 overflow-y-auto" : ""
+ }`}
>
{shown.map((c) => (
<li
@@ -737,7 +980,7 @@ function CleanableChannelsStrip({
/>
</ul>
)}
- </section>
+ </WidgetSection>
);
}
@@ -759,17 +1002,11 @@ function ActiveJobsStrip({
const running = jobs.filter((j) => j.status === "running").length;
const queued = jobs.filter((j) => j.status === "queued").length;
return (
- <section aria-label="Active jobs" className="flex flex-col gap-1.5">
- {showTitle && (
- <div className="flex items-baseline gap-2">
- <h2 className="text-xs font-semibold uppercase tracking-wide text-muted-foreground">
- Active
- </h2>
- <span className="text-xs text-muted-foreground">
- {running} running, {queued} queued
- </span>
- </div>
- )}
+ <WidgetSection
+ label="Active jobs"
+ title={showTitle ? "Active" : undefined}
+ meta={`${running} running, ${queued} queued`}
+ >
{jobs.length === 0 ? (
<p className="text-xs text-muted-foreground">No active jobs.</p>
) : (
@@ -787,7 +1024,7 @@ function ActiveJobsStrip({
))}
</ul>
)}
- </section>
+ </WidgetSection>
);
}
@@ -1061,7 +1298,11 @@ function SettingsLayer({
</button>
</div>
<form className="flex flex-col gap-4">
- <WidgetConfigForm config={config} onChange={onChange} />
+ {/* The same board the builder page shows, at the size a pinned
+ widget opens — so a preset behaves identically in both. The
+ widget itself is the preview, which is why opening the menu
+ covers it rather than sitting beside it. */}
+ <WidgetMenu config={config} onChange={onChange} compact />
</form>
<p className="mt-3 text-xs text-muted-foreground">
Changes apply live and update this widget's link — reload keeps
diff --git a/editor/app/widget/components/PresetsRow.tsx b/editor/app/widget/components/PresetsRow.tsx
@@ -0,0 +1,208 @@
+"use client";
+
+import { useCallback, useEffect, useState } from "react";
+import type { WidgetPreset } from "yt-dlp-transcript-common/lib/widgetPresets";
+import { BUILT_IN_PRESETS, type BuiltInPreset } from "../lib/builtInPresets";
+import {
+ deleteWidgetPresetAction,
+ renameWidgetPresetAction,
+ saveWidgetPresetAction,
+} from "../presetActions";
+
+// The preset library row, in the house pattern (ProfilesRow in
+// WorkspaceSearchBar, SavedChatsRow in /ask): a grouped <select> of built-ins
+// and saved arrangements, then Save / Save as… / Rename / Delete as text
+// buttons, a dirty dot, and the browser's own prompt/confirm for naming.
+//
+// A preset is a stored /widget QUERY STRING, so "load" is just "hand this link
+// to the config parser" — which is why the same row works identically on the
+// builder page and inside the widget's own menu.
+//
+// BUILT-INS ARE LOAD-ONLY. Save on a built-in writes a new named preset rather
+// than editing it: the starting points have to stay where you left them, or
+// "Cockpit" stops meaning anything.
+
+export type PresetsPayload = { builtIn: BuiltInPreset[]; saved: WidgetPreset[] };
+
+export function PresetsRow({
+ query,
+ initial,
+ onLoad,
+}: {
+ // The query the current config serializes to — what Save would store, and
+ // what the dirty dot compares against.
+ query: string;
+ // SSR seed. The builder page has one; the in-widget menu doesn't (it is a bare
+ // client page), so it fetches on open instead.
+ initial?: PresetsPayload;
+ onLoad: (query: string) => void;
+}) {
+ const [data, setData] = useState<PresetsPayload>(
+ initial ?? { builtIn: BUILT_IN_PRESETS, saved: [] },
+ );
+ const [activeId, setActiveId] = useState<string | null>(null);
+ const [busy, setBusy] = useState(false);
+ const [error, setError] = useState<string | null>(null);
+
+ const refresh = useCallback(async () => {
+ try {
+ const res = await fetch("/api/widget/presets", { cache: "no-store" });
+ if (res.ok) setData((await res.json()) as PresetsPayload);
+ } catch {
+ // Offline or mid-restart: keep showing what we already have rather than
+ // blanking the row.
+ }
+ }, []);
+
+ // Mounting IS opening for the in-widget menu, so this is the "fetch on open"
+ // the seeded builder doesn't need.
+ useEffect(() => {
+ if (!initial) void refresh();
+ }, [initial, refresh]);
+
+ const activeSaved = data.saved.find((p) => p.id === activeId) ?? null;
+ const activeBuiltIn = data.builtIn.find((p) => p.id === activeId) ?? null;
+ const activeQuery = activeSaved?.query ?? activeBuiltIn?.query ?? null;
+ const dirty = activeQuery !== null && activeQuery !== query;
+
+ async function run(fn: () => Promise<{ ok: boolean; error?: string }>) {
+ setBusy(true);
+ setError(null);
+ try {
+ const result = await fn();
+ if (!result.ok) setError(result.error ?? "Failed.");
+ await refresh();
+ return result.ok;
+ } finally {
+ setBusy(false);
+ }
+ }
+
+ function load(id: string) {
+ const preset =
+ data.saved.find((p) => p.id === id) ?? data.builtIn.find((p) => p.id === id);
+ if (!preset) return;
+ setActiveId(id);
+ onLoad(preset.query);
+ }
+
+ async function saveAs() {
+ const suggested = activeSaved?.name ?? activeBuiltIn?.name ?? "";
+ const name = window.prompt("Save this arrangement as:", suggested);
+ if (name === null) return;
+ const ok = await run(() => saveWidgetPresetAction(name, query));
+ if (!ok) return;
+ // Re-read so the new preset can be selected by the id the store gave it.
+ const res = await fetch("/api/widget/presets", { cache: "no-store" }).catch(
+ () => null,
+ );
+ if (!res?.ok) return;
+ const next = (await res.json()) as PresetsPayload;
+ setData(next);
+ const saved = next.saved.find((p) => p.name === name.trim());
+ if (saved) setActiveId(saved.id);
+ }
+
+ async function save() {
+ // A built-in can't be written over, so Save on one means Save as… — the
+ // action you actually wanted.
+ if (!activeSaved) return saveAs();
+ await run(() => saveWidgetPresetAction(activeSaved.name, query));
+ }
+
+ async function rename() {
+ if (!activeSaved) return;
+ const name = window.prompt("Rename this preset:", activeSaved.name);
+ if (name === null) return;
+ await run(() => renameWidgetPresetAction(activeSaved.id, name));
+ }
+
+ async function remove() {
+ if (!activeSaved) return;
+ if (!window.confirm(`Delete the preset "${activeSaved.name}"?`)) return;
+ const id = activeSaved.id;
+ const ok = await run(() => deleteWidgetPresetAction(id));
+ if (ok) setActiveId(null);
+ }
+
+ const hint = activeBuiltIn?.hint;
+
+ return (
+ <div className="flex flex-col gap-1">
+ <div className="flex flex-wrap items-center gap-x-3 gap-y-1">
+ <span className="text-xs uppercase tracking-wide text-muted-foreground">
+ Presets
+ </span>
+ <select
+ aria-label="preset"
+ value={activeId ?? ""}
+ onChange={(e) => e.target.value && load(e.target.value)}
+ className="rounded border border-border bg-card px-2 py-0.5 text-sm text-foreground"
+ >
+ <option value="">(custom)</option>
+ <optgroup label="Built in">
+ {data.builtIn.map((p) => (
+ <option key={p.id} value={p.id}>
+ {p.name}
+ </option>
+ ))}
+ </optgroup>
+ {data.saved.length > 0 && (
+ <optgroup label="Saved">
+ {data.saved.map((p) => (
+ <option key={p.id} value={p.id}>
+ {p.name}
+ </option>
+ ))}
+ </optgroup>
+ )}
+ </select>
+ {dirty && (
+ <span
+ role="status"
+ aria-label="preset has unsaved changes"
+ className="text-base leading-none text-warning"
+ >
+ •
+ </span>
+ )}
+ <button
+ type="button"
+ onClick={save}
+ disabled={busy || (activeSaved !== null && !dirty)}
+ className={PRESET_BTN}
+ >
+ Save
+ </button>
+ <button type="button" onClick={saveAs} disabled={busy} className={PRESET_BTN}>
+ Save as…
+ </button>
+ <button
+ type="button"
+ onClick={rename}
+ disabled={busy || !activeSaved}
+ className={PRESET_BTN}
+ >
+ Rename
+ </button>
+ <button
+ type="button"
+ onClick={remove}
+ disabled={busy || !activeSaved}
+ className="text-xs text-destructive transition-colors hover:opacity-80 disabled:opacity-50"
+ >
+ Delete
+ </button>
+ </div>
+ {hint && <p className="text-xs text-muted-foreground">{hint}</p>}
+ {error && (
+ <p role="alert" className="text-xs text-destructive">
+ {error}
+ </p>
+ )}
+ </div>
+ );
+}
+
+const PRESET_BTN =
+ "text-xs text-muted-foreground transition-colors hover:text-foreground disabled:opacity-50";
diff --git a/editor/app/widget/components/WidgetConfigForm.tsx b/editor/app/widget/components/WidgetConfigForm.tsx
@@ -1,181 +0,0 @@
-"use client";
-
-import type { WidgetConfig } from "../lib/config";
-import {
- MAX_COLUMNS,
- moveWithin,
- reconcileColumns,
- insertAt,
- setColumnCount,
-} from "../lib/placement";
-import { SECTIONS, SECTION_BY_ID, type SectionDef } from "../lib/sections";
-import { Check, GlobalOptions } from "./WidgetFields";
-
-// The in-widget gear's configuration form. Same registry and the same layout
-// operations as the /widget/builder floorplan, in a presentation that fits the
-// ~320px overlay a pinned widget opens: one row per section in the order they
-// render, with its own display options folded in underneath, rather than a
-// drag board there'd be no room to use.
-//
-// Callers own the surrounding <form> element.
-export function WidgetConfigForm({
- config,
- onChange,
-}: {
- config: WidgetConfig;
- onChange: (patch: Partial<WidgetConfig>) => void;
-}) {
- const columns = config.columns;
- const off = SECTIONS.filter((s) => !s.enabled(config));
-
- // Switch a section on where the builder would put it — the end of the last
- // column — so the two surfaces agree about what "add" means.
- function enable(def: SectionDef) {
- const flagPatch = def.setEnabled(true);
- const nextFlags = { ...config, ...flagPatch };
- const base = reconcileColumns(columns, nextFlags);
- const last = base.length - 1;
- onChange({
- ...flagPatch,
- columns: insertAt(base, def.id, last, base[last].length),
- });
- }
-
- return (
- <>
- <fieldset className="flex flex-col gap-2">
- <legend className="mb-1 text-sm font-medium">Layout</legend>
- <label className="flex items-center gap-2 text-sm">
- <span>Columns</span>
- <select
- value={columns.length}
- onChange={(e) =>
- onChange({ columns: setColumnCount(columns, Number(e.target.value)) })
- }
- className="rounded border border-border bg-card px-1.5 py-0.5 text-sm"
- >
- {Array.from({ length: MAX_COLUMNS }, (_, i) => i + 1).map((n) => (
- <option key={n} value={n}>
- {n}
- </option>
- ))}
- </select>
- </label>
-
- {columns.map((col, ci) => (
- <div key={ci} className="flex flex-col gap-2">
- {columns.length > 1 && (
- <span className="text-xs font-medium uppercase tracking-wide text-muted-foreground">
- Column {ci + 1}
- </span>
- )}
- {col.length === 0 && (
- <span className="text-xs text-muted-foreground/70">Empty</span>
- )}
- {col.map((id, index) => {
- const def = SECTION_BY_ID[id];
- return (
- <div key={id} className="flex flex-col gap-1.5">
- <div className="flex items-center gap-1.5">
- <label className="flex min-w-0 flex-1 items-center gap-2 text-sm">
- <input
- type="checkbox"
- checked
- onChange={() => onChange(def.setEnabled(false))}
- className="h-4 w-4 shrink-0"
- />
- <span className="truncate">{def.label}</span>
- </label>
- <button
- type="button"
- className={MOVE_BTN}
- disabled={index === 0}
- onClick={() =>
- onChange({ columns: moveWithin(columns, ci, index, -1) })
- }
- aria-label={`Move ${def.label} up`}
- >
- ↑
- </button>
- <button
- type="button"
- className={MOVE_BTN}
- disabled={index === col.length - 1}
- onClick={() =>
- onChange({ columns: moveWithin(columns, ci, index, 1) })
- }
- aria-label={`Move ${def.label} down`}
- >
- ↓
- </button>
- {columns.length > 1 && (
- <select
- value={ci}
- aria-label={`Column for ${def.label}`}
- onChange={(e) =>
- onChange({
- columns: insertAt(
- columns,
- def.id,
- Number(e.target.value),
- columns[Number(e.target.value)].length,
- ),
- })
- }
- className="rounded border border-border bg-card px-1 py-0.5 text-xs"
- >
- {columns.map((_, n) => (
- <option key={n} value={n}>
- {n + 1}
- </option>
- ))}
- </select>
- )}
- </div>
- {def.options.length > 0 && (
- <div className="flex flex-col gap-1.5 border-l border-border pl-3">
- {def.options.map((o) => (
- <Check
- key={o.key}
- label={o.label}
- hint={o.hint}
- checked={config[o.key]}
- onChange={(v) =>
- onChange({ [o.key]: v } as Partial<WidgetConfig>)
- }
- />
- ))}
- </div>
- )}
- </div>
- );
- })}
- </div>
- ))}
- </fieldset>
-
- {off.length > 0 && (
- <fieldset className="flex flex-col gap-2">
- <legend className="mb-1 text-sm font-medium">Off</legend>
- {off.map((def) => (
- <Check
- key={def.id}
- label={def.label}
- hint={def.hint}
- checked={false}
- onChange={() => enable(def)}
- />
- ))}
- </fieldset>
- )}
-
- <fieldset className="flex flex-col gap-2">
- <legend className="mb-1 text-sm font-medium">Widget</legend>
- <GlobalOptions config={config} onChange={onChange} />
- </fieldset>
- </>
- );
-}
-
-const MOVE_BTN =
- "flex h-6 w-6 shrink-0 items-center justify-center rounded border border-border text-xs text-muted-foreground hover:bg-muted disabled:opacity-40";
diff --git a/editor/app/widget/components/WidgetControls.tsx b/editor/app/widget/components/WidgetControls.tsx
@@ -3,6 +3,7 @@
import { useState } from "react";
import { PauseTranscriptionsButton } from "../../jobs/components/PauseTranscriptionsButton";
import { PauseDownloadsButton } from "../../jobs/components/PauseDownloadsButton";
+import { PauseBackfillButton } from "../../jobs/components/PauseBackfillButton";
import { DrainAllButton } from "../../jobs/components/DrainAllButton";
import { RetryAllFailedButton } from "../../jobs/components/RetryAllFailedButton";
import { syncAllChannelsAction, type SyncAllResult } from "../../channels/actions";
@@ -10,15 +11,17 @@ import { syncAction } from "../../channels/[slug]/pipelineActions";
// Opt-in interactive controls for the monitor widget. Two independent
// capabilities, each behind its own flag:
-// - `controls` → Pause/Resume + Drain + Retry (reused verbatim from the
-// Workers / Active Jobs pages), so a pinned widget can free the GPU, wind
-// work down, or recover failures without opening the full app.
+// - `controls` → Pause/Resume (transcriptions, downloads, backfill) + Drain +
+// Retry (reused verbatim from the Workers / Active Jobs / dashboard pages),
+// so a pinned widget can free the GPU, wind work down, hold a days-long
+// backfill, or recover failures without opening the full app.
// - `sync` → a channel-aware Sync button: pinned to one channel it syncs just
// that channel (draining the per-channel stream); otherwise it sweeps all
// channels.
// `onWorkersChange` refetches the worker payload so the pause/resume label flips
// immediately; `onSynced` refetches the last-sync/scheduler readouts so they
-// update as soon as a sweep is queued.
+// update as soon as a sweep is queued; `onBackfillChange` does the same for the
+// backfill lane, whose state rides that same payload.
export function WidgetControls({
controls,
sync,
@@ -26,8 +29,11 @@ export function WidgetControls({
confirmSyncAll,
paused,
downloadsPaused,
+ backfillEnabled,
+ backfillAnyKind,
onWorkersChange,
onSynced,
+ onBackfillChange,
}: {
controls: boolean;
sync: boolean;
@@ -35,8 +41,17 @@ export function WidgetControls({
confirmSyncAll: boolean;
paused: boolean;
downloadsPaused: boolean;
+ // settings.backfill.enabled. Persisted state, not in-memory runtime state, so
+ // the widget's 15s-floor sync poll is enough to keep the label honest between
+ // clicks — and `onBackfillChange` flips it immediately on one.
+ backfillEnabled: boolean;
+ // Whether any backfill FEATURE is registered. With none there is nothing to
+ // hold, and a dead button reads as a broken one — the same gate PipelineBand
+ // puts on the dashboard's copy.
+ backfillAnyKind: boolean;
onWorkersChange: () => void | Promise<void>;
onSynced: () => void | Promise<void>;
+ onBackfillChange: () => void | Promise<void>;
}) {
return (
<section
@@ -50,6 +65,12 @@ export function WidgetControls({
paused={downloadsPaused}
onChange={onWorkersChange}
/>
+ {backfillAnyKind && (
+ <PauseBackfillButton
+ paused={!backfillEnabled}
+ onChange={onBackfillChange}
+ />
+ )}
<DrainAllButton />
<RetryAllFailedButton />
</>
diff --git a/editor/app/widget/components/WidgetMenu.tsx b/editor/app/widget/components/WidgetMenu.tsx
@@ -0,0 +1,82 @@
+"use client";
+
+import { useEffect, useMemo, useState } from "react";
+import {
+ buildWidgetQuery,
+ parseWidgetConfig,
+ type WidgetConfig,
+} from "../lib/config";
+import { LayoutBoard } from "../builder/components/LayoutBoard";
+import { GlobalOptions } from "./WidgetFields";
+import { PresetsRow, type PresetsPayload } from "./PresetsRow";
+
+// THE configuration surface. One component, two places: the /widget/builder
+// page renders it above the preview iframe, and the widget's own settings gear
+// renders it inside the overlay.
+//
+// There used to be two — a drag board for the builder and a narrow list form for
+// the gear — and the only reason for the second was that the board didn't fold.
+// It does now (its own @container, not viewport media queries, so it behaves the
+// same inside a 320px overlay as on a wide page), so the second is gone. One
+// surface is what makes a preset mean the same thing wherever you load it.
+export function WidgetMenu({
+ config,
+ onChange,
+ initialPresets,
+ previewWidth,
+ compact = false,
+}: {
+ config: WidgetConfig;
+ onChange: (patch: Partial<WidgetConfig>) => void;
+ // SSR seed for the preset row (builder page only).
+ initialPresets?: PresetsPayload;
+ // The width the widget will render at, for the board's track readouts. The
+ // builder passes its preview size; inside the widget we measure the widget.
+ previewWidth?: number;
+ // Inside the widget's own overlay: no fieldset chrome, tighter gaps.
+ compact?: boolean;
+}) {
+ const query = useMemo(() => buildWidgetQuery(config), [config]);
+ const [selfWidth, setSelfWidth] = useState<number | undefined>(undefined);
+
+ // In the gear the widget IS the preview, so the honest width is its own.
+ // Measured after mount; the overlay only ever renders client-side.
+ useEffect(() => {
+ if (!compact || previewWidth !== undefined) return;
+ const measure = () => setSelfWidth(window.innerWidth);
+ measure();
+ window.addEventListener("resize", measure);
+ return () => window.removeEventListener("resize", measure);
+ }, [compact, previewWidth]);
+
+ // Loading a preset REPLACES the config rather than merging into it: a preset
+ // is a whole link, and parseWidgetConfig returns every field, so passing it as
+ // the patch resets anything the preset didn't mention to its default. Merging
+ // would leave whatever you had switched on stuck on.
+ function loadPreset(presetQuery: string) {
+ const params = Object.fromEntries(new URLSearchParams(presetQuery));
+ onChange(parseWidgetConfig(params));
+ }
+
+ return (
+ <div className={`flex flex-col ${compact ? "gap-3" : "gap-4"}`}>
+ <PresetsRow query={query} initial={initialPresets} onLoad={loadPreset} />
+ <LayoutBoard
+ config={config}
+ onChange={onChange}
+ previewWidth={previewWidth ?? selfWidth}
+ />
+ {compact ? (
+ <div className="flex flex-col gap-2">
+ <span className="text-sm font-medium">Whole widget</span>
+ <GlobalOptions config={config} onChange={onChange} />
+ </div>
+ ) : (
+ <fieldset className="flex flex-col gap-2 rounded-lg border border-border p-3">
+ <legend className="px-1 text-sm font-medium">Whole widget</legend>
+ <GlobalOptions config={config} onChange={onChange} />
+ </fieldset>
+ )}
+ </div>
+ );
+}
diff --git a/editor/app/widget/components/WidgetSection.tsx b/editor/app/widget/components/WidgetSection.tsx
@@ -0,0 +1,100 @@
+"use client";
+
+import { createContext, useContext, type ReactNode } from "react";
+
+// The shell every placeable widget section renders inside, and the frame flags
+// it reads.
+//
+// Four of the strips used to carry their own copy of the same header JSX; this
+// is that markup in one place, plus the one structural thing none of them could
+// do before: SCROLL UNDER A PINNED TITLE. The header has to live INSIDE the
+// scroll container for `sticky top-0` to pin it — a header outside would simply
+// scroll away with the list — which is why the shell owns both rather than the
+// caller wrapping its own children.
+
+export type SectionFrame = {
+ // This section is in a row that takes the widget's leftover height, so it gets
+ // a max height and scrolls rather than growing the page.
+ scroll: boolean;
+ // This section is in a cell laid out as a horizontal rail: drop the individual
+ // border and shorten the copy, so four totals read as one instrument cluster
+ // rather than four boxes.
+ dense: boolean;
+};
+
+const DEFAULT_FRAME: SectionFrame = { scroll: false, dense: false };
+
+const SectionFrameContext = createContext<SectionFrame>(DEFAULT_FRAME);
+
+// Provided per CELL by MonitorWidget, so renderSection() doesn't have to thread
+// two more flags through every section's props.
+export function SectionFrameProvider({
+ value,
+ children,
+}: {
+ value: SectionFrame;
+ children: ReactNode;
+}) {
+ return (
+ <SectionFrameContext.Provider value={value}>
+ {children}
+ </SectionFrameContext.Provider>
+ );
+}
+
+export function useSectionFrame(): SectionFrame {
+ return useContext(SectionFrameContext);
+}
+
+export const SECTION_TITLE_CLASS =
+ "text-xs font-semibold uppercase tracking-wide text-muted-foreground";
+
+// `label` is the accessible name and is part of the widget's test surface — the
+// e2e suite selects sections by `getByRole("region", { name: … })`, so these
+// strings and the <h2> text are a contract, not decoration.
+export function WidgetSection({
+ label,
+ title,
+ meta,
+ extra,
+ children,
+}: {
+ label: string;
+ // Absent when the widget is configured with titles off.
+ title?: string;
+ // The count/summary that sits beside the title.
+ meta?: ReactNode;
+ // Anything after the meta — the Workers strip's "paused" pill.
+ extra?: ReactNode;
+ children: ReactNode;
+}) {
+ const { scroll } = useSectionFrame();
+ return (
+ <section aria-label={label} className="flex min-h-0 flex-col">
+ <div
+ className={
+ scroll
+ ? "flex min-h-0 flex-1 flex-col gap-1.5 overflow-y-auto overscroll-contain"
+ : "flex flex-col gap-1.5"
+ }
+ >
+ {title && (
+ <div
+ className={`flex items-baseline gap-2 ${
+ scroll
+ ? "sticky top-0 z-10 border-b border-border/60 bg-background/95 pb-1 backdrop-blur"
+ : ""
+ }`}
+ >
+ <h2 className={SECTION_TITLE_CLASS}>{title}</h2>
+ {meta !== undefined && (
+ <span className="text-xs text-muted-foreground">{meta}</span>
+ )}
+ {extra}
+ </div>
+ )}
+ {children}
+ </div>
+ </section>
+ );
+}
diff --git a/editor/app/widget/lib/builtInPresets.test.ts b/editor/app/widget/lib/builtInPresets.test.ts
@@ -0,0 +1,82 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { BUILT_IN_PRESETS, builtInPresetById } from "./builtInPresets";
+import { buildWidgetQuery, parseWidgetConfig } from "./config";
+
+// Run from the repo root with:
+// ./node_modules/.bin/tsx --test editor/app/widget/lib/builtInPresets.test.ts
+//
+// A preset is a stored LINK, so the only thing that can go wrong with one is
+// that it stops meaning what it says: a renamed param, a changed default, a
+// section dropped from the registry, a layout the current placement math would
+// normalize differently. Each of those shows up as a query that does not
+// re-serialize to itself, which is what this asserts.
+
+// Parse a query string the way /widget does, then re-emit it.
+function roundTrip(query: string): string {
+ const params: Record<string, string> = {};
+ for (const [k, v] of new URLSearchParams(query)) params[k] = v;
+ return buildWidgetQuery(parseWidgetConfig(params));
+}
+
+test("every built-in preset parses and re-serializes to itself", () => {
+ for (const preset of BUILT_IN_PRESETS) {
+ assert.equal(roundTrip(preset.query), preset.query, `${preset.name} round-trips`);
+ }
+});
+
+test("built-in ids and names are unique and non-empty", () => {
+ const ids = new Set<string>();
+ const names = new Set<string>();
+ for (const p of BUILT_IN_PRESETS) {
+ assert.ok(p.id, "id is set");
+ assert.ok(p.name, "name is set");
+ assert.ok(p.hint, "hint is set");
+ assert.ok(!ids.has(p.id), `duplicate id ${p.id}`);
+ assert.ok(!names.has(p.name), `duplicate name ${p.name}`);
+ ids.add(p.id);
+ names.add(p.name);
+ }
+});
+
+test("Jobs is the way back to the default widget", () => {
+ // The "start over" entry has to be the empty query, or a bare /widget link
+ // would no longer be reachable from the menu.
+ assert.equal(builtInPresetById("jobs")?.query, "");
+ assert.equal(roundTrip(""), "");
+});
+
+test("Cockpit is the three-row layout, fill row included", () => {
+ const config = parseWidgetConfig(
+ Object.fromEntries(
+ new URLSearchParams(builtInPresetById("cockpit")?.query ?? ""),
+ ),
+ );
+ assert.equal(config.layout.rows.length, 3);
+ assert.equal(config.layout.cols, 2);
+ assert.equal(config.layout.rows[0].cells[0].span, 2);
+ assert.equal(config.layout.rows[1].cells[0].flow, "row");
+ assert.equal(config.layout.rows[2].size, "fill");
+});
+
+test("Glance is one dense rail and nothing else", () => {
+ const config = parseWidgetConfig(
+ Object.fromEntries(
+ new URLSearchParams(builtInPresetById("glance")?.query ?? ""),
+ ),
+ );
+ assert.equal(config.layout.rows.length, 1);
+ assert.equal(config.layout.rows[0].cells.length, 1);
+ assert.equal(config.layout.rows[0].cells[0].flow, "row");
+ assert.equal(config.showTitles, false);
+});
+
+test("Wall is locked down: no controls, no gear", () => {
+ const config = parseWidgetConfig(
+ Object.fromEntries(new URLSearchParams(builtInPresetById("wall")?.query ?? "")),
+ );
+ assert.equal(config.controls, false);
+ assert.equal(config.settings, false);
+ assert.equal(config.showTitles, true);
+ assert.equal(config.layout.cols, 3);
+});
diff --git a/editor/app/widget/lib/builtInPresets.ts b/editor/app/widget/lib/builtInPresets.ts
@@ -0,0 +1,63 @@
+// The arrangements that ship with the widget.
+//
+// Each is stored the same way a saved preset is — as the QUERY STRING
+// buildWidgetQuery would produce — so a built-in and something you saved
+// yourself are the same kind of thing, and loading either is just "parse this
+// link". Built-ins are load-only: Save writes a new named preset rather than
+// editing one of these, so the starting points stay where you left them.
+//
+// builtInPresets.test.ts asserts every query here parses and re-serializes to
+// ITSELF. That is the guard against a preset drifting away from the registry: a
+// renamed param, a changed default or a section that stops existing would
+// otherwise leave a preset that quietly loads as something else.
+
+export type BuiltInPreset = {
+ id: string;
+ name: string;
+ // One line: what this is for, shown beside the name.
+ hint: string;
+ // Without a leading "?". "" is the all-default widget.
+ query: string;
+};
+
+export const BUILT_IN_PRESETS: BuiltInPreset[] = [
+ {
+ id: "glance",
+ name: "Glance",
+ hint: "One dense rail of totals and a row of worker dots. A 320×120 corner pin.",
+ query:
+ "jobs=0&titles=0&clean=1&wnames=0&backfill=1&lastsync=1&g=lsync.disk.cln.bf.wk*r",
+ },
+ {
+ id: "jobs",
+ name: "Jobs",
+ hint: "Workers and active jobs in one column — the default arrangement, and the way back to it.",
+ // Deliberately empty: the default IS this preset, and an empty query is what
+ // "start over" has to mean if a bare /widget link is to keep working.
+ query: "",
+ },
+ {
+ id: "cockpit",
+ name: "Cockpit",
+ hint: "Controls across the top, a rail of totals, then the scheduler beside two scrolling worklists.",
+ query:
+ "jobs=0&workers=0&clean=1&controls=1&backfill=1&act=1&cleanlist=1&sync=1&lastsync=1&sched=1&g=ctl*2_lsync.disk.cln.bf*2r_sched*f-act.clnl",
+ },
+ {
+ id: "cleanup",
+ name: "Cleanup",
+ hint: "What you could reclaim, the controls to do it, and a full-height list of the channels holding it.",
+ query: "jobs=0&workers=0&clean=1&controls=1&cleanlist=1&g=cln.disk*r_ctl_clnl*f",
+ },
+ {
+ id: "wall",
+ name: "Wall",
+ hint: "Three columns, titles on, no controls and no gear. For a spare monitor.",
+ query:
+ "clean=1&act=1&cleanlist=1&gear=0&lastsync=1&sched=1&g=lsync.sched.disk.cln*3r_wk.jobs*f-act-clnl",
+ },
+];
+
+export function builtInPresetById(id: string): BuiltInPreset | undefined {
+ return BUILT_IN_PRESETS.find((p) => p.id === id);
+}
diff --git a/editor/app/widget/lib/config.ts b/editor/app/widget/lib/config.ts
@@ -3,11 +3,11 @@
// serialize), so a link the builder copies always renders the way it previewed.
import {
- defaultColumns,
+ defaultLayout,
parseLayoutParam,
- reconcileColumns,
+ reconcileLayout,
serializeLayout,
- type Columns,
+ type Layout,
} from "./placement";
import type { SectionFlags } from "./sections";
@@ -75,15 +75,16 @@ export type WidgetConfig = {
// window.confirm before a full Sync-all sweep. Off by default. (A pinned
// single-channel Sync is never gated.)
syncConfirm: boolean;
- // Where each enabled section sits: one array per column, in render order.
- // Derived, not authoritative — the booleans above still decide what is on,
- // and a layout is normalized against them on parse (see parseLayoutParam), so
- // the two can never disagree. Defaults to one column in the widget's original
- // order, which is why every link written before layouts existed is unchanged.
- columns: Columns;
- // Let the columns collapse back to a single stack below a container-query
- // width, for a widget you intend to resize. Off by default: a layout you
- // arranged is honored at every size unless you ask for this.
+ // Where each enabled section sits: rows of cells, each cell a stack of
+ // sections (see ./placement). Derived, not authoritative — the booleans above
+ // still decide what is on, and a layout is normalized against them on parse
+ // (see parseLayoutParam), so the two can never disagree. Defaults to one cell
+ // in one auto row, in the widget's original order, which is why every link
+ // written before layouts existed is unchanged.
+ layout: Layout;
+ // Let the grid collapse back to a single stack below a container-query width,
+ // for a widget you intend to resize. Off by default: a layout you arranged is
+ // honored at every size unless you ask for this.
stackNarrow: boolean;
};
@@ -119,7 +120,7 @@ const WIDGET_FLAG_DEFAULTS: SectionFlags = {
// that could drift away from them.
export const WIDGET_DEFAULTS: WidgetConfig = {
...WIDGET_FLAG_DEFAULTS,
- columns: defaultColumns(WIDGET_FLAG_DEFAULTS),
+ layout: defaultLayout(WIDGET_FLAG_DEFAULTS),
};
// Next's searchParams give each key as string | string[] | undefined.
@@ -175,21 +176,25 @@ export function parseWidgetConfig(params: RawParams): WidgetConfig {
stackNarrow: parseBool(params.stack, WIDGET_DEFAULTS.stackNarrow),
};
// Last, because normalizing a layout means knowing which sections are on.
- return { ...flags, columns: parseLayoutParam(first(params.l), flags) };
+ return {
+ ...flags,
+ layout: parseLayoutParam(first(params.l), first(params.g), flags),
+ };
}
// Apply a config change and keep the layout consistent with it: a section
-// switched off leaves the board, one switched on joins the last column. Every
-// edit surface (the builder board, the in-widget gear) goes through this so
-// neither can produce a config whose `l=` disagrees with its booleans. A patch
-// carrying explicit `columns` — a drag, a reorder — is reconciled too, which is
-// a no-op when the caller already placed things correctly.
+// switched off leaves the board, one switched on joins the last cell of the last
+// row. Every edit surface (the builder board, the in-widget menu) goes through
+// this so neither can produce a config whose layout param disagrees with its
+// booleans. A patch carrying an explicit `layout` — a drag, a reorder — is
+// reconciled too, which is a no-op when the caller already placed things
+// correctly.
export function patchWidgetConfig(
config: WidgetConfig,
patch: Partial<WidgetConfig>,
): WidgetConfig {
const next = { ...config, ...patch };
- return { ...next, columns: reconcileColumns(next.columns, next) };
+ return { ...next, layout: reconcileLayout(next.layout, next) };
}
// Serialize a config to a query string, omitting anything left at its default so
@@ -242,8 +247,11 @@ export function buildWidgetQuery(config: WidgetConfig): string {
sp.set("stack", config.stackNarrow ? "1" : "0");
// Last, and omitted entirely when the arrangement is the one the visibility
// flags already imply — so a config that only toggles sections still produces
- // the same short link it did before layouts existed.
- const layout = serializeLayout(config.columns, config);
- if (layout) sp.set("l", layout);
+ // the same short link it did before layouts existed. `l` OR `g`, never both:
+ // a layout that plain columns can express keeps emitting the `l=` a
+ // pre-rows link would have carried.
+ const layout = serializeLayout(config.layout, config);
+ if (layout.l !== undefined) sp.set("l", layout.l);
+ else if (layout.g !== undefined) sp.set("g", layout.g);
return sp.toString();
}
diff --git a/editor/app/widget/lib/placement.test.ts b/editor/app/widget/lib/placement.test.ts
@@ -1,20 +1,31 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import {
- defaultColumns,
+ addCell,
+ defaultLayout,
+ findSection,
+ hasFillRow,
insertAt,
- moveToColumn,
+ isColumnsLayout,
+ moveToCell,
+ moveToRow,
moveWithin,
parseLayoutParam,
- reconcileColumns,
+ reconcileLayout,
+ removeCell,
removeSection,
sameLayout,
serializeLayout,
+ setCellFlow,
+ setCellSpan,
setColumnCount,
- type Columns,
+ setRowCount,
+ setRowSize,
+ type Layout,
+ type SectionPos,
} from "./placement";
import { WIDGET_DEFAULTS } from "./config";
-import type { SectionFlags } from "./sections";
+import type { SectionFlags, SectionId } from "./sections";
// Run from the repo root with:
// ./node_modules/.bin/tsx --test editor/app/widget/lib/placement.test.ts
@@ -22,158 +33,482 @@ import type { SectionFlags } from "./sections";
// The point of most of these is the normalization rule: a layout arrives from a
// URL or from localStorage and can disagree with the visibility flags, which
// are authoritative. Getting that wrong means a link that renders nothing.
+//
+// The other half is BACK-COMPATIBILITY. `l=` predates rows entirely, and every
+// already-copied widget link carries one — so each `l=` case here asserts the
+// same decode the columns model gave, and that it re-serializes to the byte-
+// identical string rather than being upgraded to a `g=`.
const flags = (over: Partial<SectionFlags> = {}): SectionFlags => ({
...WIDGET_DEFAULTS,
...over,
});
-test("the default layout is one column of the enabled sections, in render order", () => {
- assert.deepEqual(defaultColumns(flags()), [["disk", "workers", "jobs"]]);
- assert.deepEqual(defaultColumns(flags({ workers: false })), [["disk", "jobs"]]);
+// A layout of plain stacked columns, the shape `l=` describes: one auto row,
+// one single-track cell per column. Lets the ported cases read as they did.
+const cols = (...columns: SectionId[][]): Layout => ({
+ cols: columns.length,
+ rows: [
+ {
+ size: "auto",
+ cells: columns.map((sections) => ({ sections, span: 1, flow: "col" })),
+ },
+ ],
+});
+
+// The `l` (or `g`) a layout serializes to, for the round-trip assertions.
+const ser = (layout: Layout, config: SectionFlags = flags()) =>
+ serializeLayout(layout, config);
+
+// ---------------------------------------------------------------------------
+// Defaults
+// ---------------------------------------------------------------------------
+
+test("the default layout is one cell of the enabled sections, in render order", () => {
+ assert.deepEqual(defaultLayout(flags()), cols(["disk", "workers", "jobs"]));
+ assert.deepEqual(defaultLayout(flags({ workers: false })), cols(["disk", "jobs"]));
// Order is the widget's original render order, not the order flags were set.
- assert.deepEqual(defaultColumns(flags({ actionable: true, lastSync: true })), [
- ["lastSync", "disk", "workers", "jobs", "actionable"],
- ]);
+ assert.deepEqual(
+ defaultLayout(flags({ actionable: true, lastSync: true })),
+ cols(["lastSync", "disk", "workers", "jobs", "actionable"]),
+ );
+ // One auto row, one track: the pre-rows behaviour exactly.
+ assert.equal(defaultLayout(flags()).cols, 1);
+ assert.equal(defaultLayout(flags()).rows.length, 1);
+ assert.equal(hasFillRow(defaultLayout(flags())), false);
});
test("no layout param means the default layout", () => {
- assert.deepEqual(parseLayoutParam(undefined, flags()), defaultColumns(flags()));
- assert.deepEqual(parseLayoutParam("", flags()), defaultColumns(flags()));
+ assert.deepEqual(
+ parseLayoutParam(undefined, undefined, flags()),
+ defaultLayout(flags()),
+ );
+ assert.deepEqual(parseLayoutParam("", "", flags()), defaultLayout(flags()));
});
-test("a layout param is decoded into columns", () => {
- assert.deepEqual(parseLayoutParam("wk.jobs-disk", flags()), [
- ["workers", "jobs"],
- ["disk"],
- ]);
+// ---------------------------------------------------------------------------
+// `l=` — the pre-rows contract
+// ---------------------------------------------------------------------------
+
+test("an l= param is decoded into one auto row of columns", () => {
+ assert.deepEqual(
+ parseLayoutParam("wk.jobs-disk", undefined, flags()),
+ cols(["workers", "jobs"], ["disk"]),
+ );
});
test("an unknown code is dropped, not rendered and not fatal", () => {
- assert.deepEqual(parseLayoutParam("wk.nosuch.jobs-disk", flags()), [
- ["workers", "jobs"],
- ["disk"],
- ]);
+ assert.deepEqual(
+ parseLayoutParam("wk.nosuch.jobs-disk", undefined, flags()),
+ cols(["workers", "jobs"], ["disk"]),
+ );
});
test("a duplicated section is placed once, where it appears first", () => {
- assert.deepEqual(parseLayoutParam("wk.jobs-wk.disk", flags()), [
- ["workers", "jobs"],
- ["disk"],
- ]);
+ assert.deepEqual(
+ parseLayoutParam("wk.jobs-wk.disk", undefined, flags()),
+ cols(["workers", "jobs"], ["disk"]),
+ );
});
-test("a section that is enabled but unlisted joins the last column", () => {
+test("a section that is enabled but unlisted joins the last cell", () => {
// `disk` is on by default and absent from the param: it must still render, or
// a link baked before a section was switched on would silently lose it.
- assert.deepEqual(parseLayoutParam("wk-jobs", flags()), [
- ["workers"],
- ["jobs", "disk"],
- ]);
+ assert.deepEqual(
+ parseLayoutParam("wk-jobs", undefined, flags()),
+ cols(["workers"], ["jobs", "disk"]),
+ );
});
test("a section named in the layout but switched off is dropped", () => {
- assert.deepEqual(parseLayoutParam("wk.jobs-disk", flags({ disk: false })), [
- ["workers", "jobs"],
- [],
- ]);
+ assert.deepEqual(
+ parseLayoutParam("wk.jobs-disk", undefined, flags({ disk: false })),
+ cols(["workers", "jobs"], []),
+ );
});
test("a layout that survives nothing falls back to the default", () => {
- assert.deepEqual(parseLayoutParam("nope.alsonope", flags()), defaultColumns(flags()));
+ assert.deepEqual(
+ parseLayoutParam("nope.alsonope", undefined, flags()),
+ defaultLayout(flags()),
+ );
});
test("more columns than the maximum are merged into the last", () => {
- assert.deepEqual(parseLayoutParam("wk-jobs-disk-cln", flags({ cleanable: true })), [
- ["workers"],
- ["jobs"],
- ["disk", "cleanable"],
- ]);
+ // Six tracks is the cap now, so a seven-column link merges the tail rather
+ // than being rejected.
+ const on = flags({ cleanable: true, actionable: true, cleanChannels: true, lastSync: true });
+ assert.deepEqual(
+ parseLayoutParam("wk-jobs-disk-cln-act-clnl-lsync", undefined, on),
+ cols(
+ ["workers"],
+ ["jobs"],
+ ["disk"],
+ ["cleanable"],
+ ["actionable"],
+ ["cleanChannels", "lastSync"],
+ ),
+ );
});
-test("serializing a default layout gives the empty string", () => {
+test("serializing a default layout gives nothing at all", () => {
// This is what keeps an all-default config serializing to a bare /widget.
- assert.equal(serializeLayout(defaultColumns(flags()), flags()), "");
+ assert.deepEqual(ser(defaultLayout(flags())), {});
const off = flags({ workers: false });
- assert.equal(serializeLayout(defaultColumns(off), off), "");
+ assert.deepEqual(ser(defaultLayout(off), off), {});
});
-test("serialize round-trips through parse", () => {
- const columns: Columns = [
- ["workers", "jobs"],
- ["disk"],
- ];
- const raw = serializeLayout(columns, flags());
- assert.equal(raw, "wk.jobs-disk");
- assert.deepEqual(parseLayoutParam(raw, flags()), columns);
+test("every columns layout still round-trips through l=, byte-identical", () => {
+ for (const raw of [
+ "wk.jobs-disk",
+ "disk.jobs.wk",
+ "wk.jobs.disk-",
+ "disk-wk-jobs",
+ ]) {
+ const layout = parseLayoutParam(raw, undefined, flags());
+ assert.deepEqual(ser(layout), { l: raw }, `round-trip of ${raw}`);
+ // ...and never as a g=, which no older client would understand.
+ assert.equal(ser(layout).g, undefined);
+ }
});
test("an empty trailing column round-trips", () => {
- const columns: Columns = [["workers", "jobs", "disk"], []];
- const raw = serializeLayout(columns, flags());
- assert.equal(raw, "wk.jobs.disk-");
- assert.deepEqual(parseLayoutParam(raw, flags()), columns);
+ const layout = cols(["workers", "jobs", "disk"], []);
+ assert.deepEqual(ser(layout), { l: "wk.jobs.disk-" });
+ assert.deepEqual(parseLayoutParam("wk.jobs.disk-", undefined, flags()), layout);
+});
+
+// ---------------------------------------------------------------------------
+// `g=` — rows, spans, flow, fill
+// ---------------------------------------------------------------------------
+
+// The Cockpit arrangement: controls across the top, a dense rail of totals under
+// it, then a fill row with the scheduler beside the two scrolling lists.
+const COCKPIT = "ctl*2_lsync.disk.cln.bf*2r_sched*f-act.clnl";
+const cockpitFlags = flags({
+ controls: true,
+ lastSync: true,
+ scheduler: true,
+ cleanable: true,
+ backfill: true,
+ actionable: true,
+ cleanChannels: true,
+ jobs: false,
+ workers: false,
+ disk: true,
+});
+
+test("g= decodes rows, spans, flow and the fill marker", () => {
+ const layout = parseLayoutParam(undefined, COCKPIT, cockpitFlags);
+ assert.deepEqual(layout, {
+ cols: 2,
+ rows: [
+ {
+ size: "auto",
+ cells: [{ sections: ["controls"], span: 2, flow: "col" }],
+ },
+ {
+ size: "auto",
+ cells: [
+ {
+ sections: ["lastSync", "disk", "cleanable", "backfill"],
+ span: 2,
+ flow: "row",
+ },
+ ],
+ },
+ {
+ size: "fill",
+ cells: [
+ { sections: ["scheduler"], span: 1, flow: "col" },
+ { sections: ["actionable", "cleanChannels"], span: 1, flow: "col" },
+ ],
+ },
+ ],
+ });
+ assert.ok(hasFillRow(layout));
+ assert.ok(!isColumnsLayout(layout));
+});
+
+test("g= re-serializes byte-identically", () => {
+ const layout = parseLayoutParam(undefined, COCKPIT, cockpitFlags);
+ assert.deepEqual(ser(layout, cockpitFlags), { g: COCKPIT });
+ // ...and never alongside an l=, which would be two contradicting layouts.
+ assert.equal(ser(layout, cockpitFlags).l, undefined);
+});
+
+test("g= wins over l= when both are present", () => {
+ const layout = parseLayoutParam("wk.jobs-disk", COCKPIT, cockpitFlags);
+ assert.equal(layout.rows.length, 3);
+});
+
+test("the fill marker survives on any cell, and normalizes onto the first", () => {
+ // A hand-edited link that marks the SECOND cell still fills the row, and
+ // re-serializes with the marker on the first cell.
+ const layout = parseLayoutParam(undefined, "wk-jobs*f", flags());
+ assert.equal(layout.rows[0].size, "fill");
+ assert.deepEqual(ser(layout), { g: "wk*f-jobs.disk" });
+});
+
+test("an out-of-range span is clamped, and the row still sums to the tracks", () => {
+ const layout = parseLayoutParam(undefined, "wk*9-jobs", flags());
+ const row = layout.rows[0];
+ assert.equal(
+ row.cells.reduce((s, c) => s + c.span, 0),
+ layout.cols,
+ );
+ assert.ok(row.cells.every((c) => c.span >= 1 && c.span <= layout.cols));
});
-test("reconcile drops what is off and appends what is on", () => {
+test("an unknown modifier letter is ignored rather than fatal", () => {
+ assert.deepEqual(
+ parseLayoutParam(undefined, "wk*2z-jobs.disk", flags()),
+ {
+ cols: 3,
+ rows: [
+ {
+ size: "auto",
+ cells: [
+ { sections: ["workers"], span: 2, flow: "col" },
+ { sections: ["jobs", "disk"], span: 1, flow: "col" },
+ ],
+ },
+ ],
+ },
+ );
+});
+
+test("more rows than the maximum are merged into the last", () => {
+ const layout = parseLayoutParam(
+ undefined,
+ "wk_jobs_disk_cln_act_clnl_lsync",
+ flags({ cleanable: true, actionable: true, cleanChannels: true, lastSync: true }),
+ );
+ assert.equal(layout.rows.length, 6);
+ assert.deepEqual(layout.rows[5].cells[0].sections, ["cleanChannels", "lastSync"]);
+});
+
+test("a g= that survives nothing falls back to the default", () => {
+ assert.deepEqual(
+ parseLayoutParam(undefined, "nope_alsonope", flags()),
+ defaultLayout(flags()),
+ );
+});
+
+// ---------------------------------------------------------------------------
+// Reconcile
+// ---------------------------------------------------------------------------
+
+test("reconcile drops what is off and appends to the last cell of the last row", () => {
assert.deepEqual(
- reconcileColumns([["workers", "jobs"], ["disk"]], flags({ disk: false })),
- [["workers", "jobs"], []],
+ reconcileLayout(cols(["workers", "jobs"], ["disk"]), flags({ disk: false })),
+ cols(["workers", "jobs"], []),
);
assert.deepEqual(
- reconcileColumns([["workers"], ["jobs"]], flags({ actionable: true })),
- [["workers"], ["jobs", "disk", "actionable"]],
+ reconcileLayout(cols(["workers"], ["jobs"]), flags({ actionable: true })),
+ cols(["workers"], ["jobs", "disk", "actionable"]),
);
+ // Multi-row: the newly enabled section lands in the LAST cell of the LAST
+ // row, not back at the top where it would push the arrangement around.
+ const grid = parseLayoutParam(undefined, "wk_jobs-disk", flags());
+ const next = reconcileLayout(grid, flags({ actionable: true }));
+ assert.deepEqual(next.rows[1].cells[1].sections, ["disk", "actionable"]);
});
+// ---------------------------------------------------------------------------
+// Movement
+// ---------------------------------------------------------------------------
+
test("moveWithin swaps with the neighbour and stops at the bounds", () => {
- const columns: Columns = [["workers", "jobs", "disk"]];
- assert.deepEqual(moveWithin(columns, 0, 2, -1), [["workers", "disk", "jobs"]]);
- assert.equal(moveWithin(columns, 0, 0, -1), columns);
- assert.equal(moveWithin(columns, 0, 2, 1), columns);
+ const layout = cols(["workers", "jobs", "disk"]);
+ assert.deepEqual(
+ moveWithin(layout, 0, 0, 2, -1),
+ cols(["workers", "disk", "jobs"]),
+ );
+ assert.equal(moveWithin(layout, 0, 0, 0, -1), layout);
+ assert.equal(moveWithin(layout, 0, 0, 2, 1), layout);
});
-test("moveToColumn lands at the same height, or the end of a shorter column", () => {
- assert.deepEqual(moveToColumn([["workers", "jobs"], ["disk"]], 0, 1, 1), [
- ["workers"],
- ["disk", "jobs"],
- ]);
- assert.deepEqual(moveToColumn([["workers", "jobs", "disk"], []], 0, 2, 1), [
- ["workers", "jobs"],
- ["disk"],
- ]);
+test("moveToCell lands at the same height, or the end of a shorter cell", () => {
+ assert.deepEqual(
+ moveToCell(cols(["workers", "jobs"], ["disk"]), 0, 0, 1, 1),
+ cols(["workers"], ["disk", "jobs"]),
+ );
+ assert.deepEqual(
+ moveToCell(cols(["workers", "jobs", "disk"], []), 0, 0, 2, 1),
+ cols(["workers", "jobs"], ["disk"]),
+ );
+ // No neighbouring cell → unchanged.
+ const one = cols(["workers", "jobs"]);
+ assert.equal(moveToCell(one, 0, 0, 0, 1), one);
});
-test("insertAt accounts for the removal when moving down within a column", () => {
+test("moveToRow carries a section into the row above or below", () => {
+ const layout = parseLayoutParam(undefined, "wk.jobs_disk", flags());
+ const down = moveToRow(layout, 0, 0, 0, 1);
+ assert.deepEqual(down.rows[0].cells[0].sections, ["jobs"]);
+ assert.deepEqual(down.rows[1].cells[0].sections, ["disk", "workers"]);
+ // Off the end → unchanged.
+ assert.equal(moveToRow(layout, 0, 0, 0, -1), layout);
+});
+
+test("insertAt accounts for the removal when moving down within a cell", () => {
// Dragging `workers` (index 0) to the slot below `jobs` (index 2 as drawn)
// must land it after jobs, not before.
- assert.deepEqual(insertAt([["workers", "jobs", "disk"]], "workers", 0, 2), [
- ["jobs", "workers", "disk"],
- ]);
- assert.deepEqual(insertAt([["workers", "jobs"], ["disk"]], "disk", 0, 0), [
- ["disk", "workers", "jobs"],
- [],
- ]);
-});
-
-test("setColumnCount merges trailing columns rather than dropping them", () => {
- assert.deepEqual(setColumnCount([["workers"], ["jobs"], ["disk"]], 2), [
- ["workers"],
- ["jobs", "disk"],
- ]);
- assert.deepEqual(setColumnCount([["workers", "jobs"]], 3), [
- ["workers", "jobs"],
- [],
- [],
- ]);
+ assert.deepEqual(
+ insertAt(cols(["workers", "jobs", "disk"]), "workers", 0, 0, 2),
+ cols(["jobs", "workers", "disk"]),
+ );
+ assert.deepEqual(
+ insertAt(cols(["workers", "jobs"], ["disk"]), "disk", 0, 0, 0),
+ cols(["disk", "workers", "jobs"], []),
+ );
+ // Across rows: pulled out of wherever it was, wherever that was.
+ const grid = parseLayoutParam(undefined, "wk.jobs_disk", flags());
+ const moved = insertAt(grid, "disk", 0, 0, 0);
+ assert.deepEqual(moved.rows[0].cells[0].sections, ["disk", "workers", "jobs"]);
+ assert.deepEqual(moved.rows[1].cells[0].sections, []);
});
test("removeSection and sameLayout", () => {
- assert.deepEqual(removeSection([["workers", "jobs"], ["disk"]], "jobs"), [
- ["workers"],
- ["disk"],
- ]);
- assert.ok(sameLayout([["workers"]], [["workers"]]));
- assert.ok(!sameLayout([["workers"]], [["workers"], []]));
- assert.ok(!sameLayout([["workers", "jobs"]], [["jobs", "workers"]]));
+ assert.deepEqual(
+ removeSection(cols(["workers", "jobs"], ["disk"]), "jobs"),
+ cols(["workers"], ["disk"]),
+ );
+ assert.ok(sameLayout(cols(["workers"]), cols(["workers"])));
+ assert.ok(!sameLayout(cols(["workers"]), cols(["workers"], [])));
+ assert.ok(!sameLayout(cols(["workers", "jobs"]), cols(["jobs", "workers"])));
+ // Structure counts too, not just the section order.
+ const filled = setRowSize(cols(["workers"]), 0, "fill");
+ assert.ok(!sameLayout(filled, cols(["workers"])));
+});
+
+// ---------------------------------------------------------------------------
+// Structure
+// ---------------------------------------------------------------------------
+
+test("setColumnCount merges trailing cells rather than dropping them", () => {
+ assert.deepEqual(
+ setColumnCount(cols(["workers"], ["jobs"], ["disk"]), 2),
+ cols(["workers"], ["jobs", "disk"]),
+ );
+ assert.deepEqual(
+ setColumnCount(cols(["workers", "jobs"]), 3),
+ cols(["workers", "jobs"], [], []),
+ );
+});
+
+test("setColumnCount keeps every row summing to the new track count", () => {
+ const grid = parseLayoutParam(undefined, "wk*2_jobs-disk", flags());
+ for (const n of [1, 2, 3, 4, 5, 6]) {
+ const next = setColumnCount(grid, n);
+ assert.equal(next.cols, n);
+ for (const row of next.rows) {
+ assert.equal(
+ row.cells.reduce((s, c) => s + c.span, 0),
+ n,
+ `row sums to ${n}`,
+ );
+ }
+ }
+});
+
+test("setRowCount merges trailing rows rather than dropping them", () => {
+ const three = parseLayoutParam(undefined, "wk_jobs_disk", flags());
+ const two = setRowCount(three, 2);
+ assert.equal(two.rows.length, 2);
+ assert.deepEqual(two.rows[1].cells[0].sections, ["jobs", "disk"]);
+ // Growing appends an empty row, so there is somewhere to drop things.
+ const four = setRowCount(three, 4);
+ assert.equal(four.rows.length, 4);
+ assert.deepEqual(four.rows[3].cells[0].sections, []);
+});
+
+test("setCellSpan takes the tracks from the neighbours, merging what it swallows", () => {
+ const grid = parseLayoutParam(undefined, "wk-jobs-disk", flags());
+ assert.equal(grid.cols, 3);
+
+ // Widening by one swallows the cell beside it — its sections move in rather
+ // than being dropped — and the row still sums to the track count.
+ const wide = setCellSpan(grid, 0, 0, 2);
+ assert.deepEqual(
+ wide.rows[0].cells.map((c) => c.span),
+ [2, 1],
+ );
+ assert.deepEqual(wide.rows[0].cells[0].sections, ["workers", "jobs"]);
+ assert.deepEqual(wide.rows[0].cells[1].sections, ["disk"]);
+
+ // Asking for the whole row gives the whole row, everything merged in order.
+ const max = setCellSpan(grid, 0, 0, 6);
+ assert.equal(max.rows[0].cells.length, 1);
+ assert.equal(max.rows[0].cells[0].span, 3);
+ assert.deepEqual(max.rows[0].cells[0].sections, ["workers", "jobs", "disk"]);
+
+ // Narrowing the last cell in a row frees its tracks into a new empty cell —
+ // the exact inverse, so a widen is undoable.
+ const back = setCellSpan(max, 0, 0, 1);
+ assert.deepEqual(
+ back.rows[0].cells.map((c) => c.span),
+ [1, 2],
+ );
+ assert.deepEqual(back.rows[0].cells[1].sections, []);
+});
+
+test("setCellFlow, setRowSize and hasFillRow", () => {
+ const grid = parseLayoutParam(undefined, "lsync.disk-jobs", flags({ lastSync: true }));
+ const rail = setCellFlow(grid, 0, 0, "row");
+ assert.equal(rail.rows[0].cells[0].flow, "row");
+ assert.equal(setCellFlow(rail, 0, 0, "row"), rail);
+ assert.ok(!hasFillRow(rail));
+ const filled = setRowSize(rail, 0, "fill");
+ assert.ok(hasFillRow(filled));
+ assert.equal(setRowSize(filled, 0, "fill"), filled);
+});
+
+test("addCell splits a row, removeCell merges one back", () => {
+ const one = parseLayoutParam(undefined, "wk.jobs.disk", flags());
+ const two = addCell(one, 0);
+ assert.equal(two.rows[0].cells.length, 1, "a one-track row has no room");
+
+ const wide = setColumnCount(one, 2);
+ assert.equal(wide.rows[0].cells.length, 2);
+ const three = addCell(setColumnCount(one, 3), 0);
+ assert.equal(three.rows[0].cells.length, 3);
+
+ const merged = removeCell(wide, 0, 1);
+ assert.equal(merged.rows[0].cells.length, 1);
+ assert.deepEqual(merged.rows[0].cells[0].sections, ["workers", "jobs", "disk"]);
+ // The last cell of a row can't be removed — a row always has one.
+ assert.equal(removeCell(merged, 0, 0), merged);
+});
+
+test("the board's round trip: add a row, fill it, widen a cell, and back", () => {
+ // The exact sequence the builder e2e drives, asserted on the params it writes.
+ let layout = parseLayoutParam(undefined, undefined, flags());
+ layout = setColumnCount(layout, 2);
+ assert.deepEqual(ser(layout), { l: "disk.wk.jobs-" });
+
+ layout = setRowCount(layout, 2);
+ layout = insertAt(layout, "jobs", 1, 0, 0);
+ assert.deepEqual(ser(layout), { g: "disk.wk-_jobs-" });
+
+ layout = setRowSize(layout, 1, "fill");
+ assert.deepEqual(ser(layout), { g: "disk.wk-_jobs*f-" });
+
+ layout = setCellSpan(layout, 0, 0, 2);
+ assert.deepEqual(ser(layout), { g: "disk.wk*2_jobs*f-" });
+
+ // Narrowing it back and dropping the extra row falls all the way back to `l=`.
+ layout = setCellSpan(layout, 0, 0, 1);
+ layout = setRowSize(layout, 1, "auto");
+ layout = setRowCount(layout, 1);
+ assert.deepEqual(ser(layout), { l: "disk.wk-jobs" });
+});
+
+test("findSection reports the row, cell and index", () => {
+ const grid = parseLayoutParam(undefined, "wk.jobs_disk", flags());
+ const at: SectionPos = { row: 0, cell: 0, index: 1 };
+ assert.deepEqual(findSection(grid, "jobs"), at);
+ assert.deepEqual(findSection(grid, "disk"), { row: 1, cell: 0, index: 0 });
+ assert.equal(findSection(grid, "cleanable"), null);
});
diff --git a/editor/app/widget/lib/placement.ts b/editor/app/widget/lib/placement.ts
@@ -1,15 +1,37 @@
-// Placement math for the monitor widget: which section sits in which column, in
-// what order. Pure functions over `SectionId[][]` — no React, no DOM — so the
-// floorplan board, the in-widget gear and the URL parser all move sections the
-// same way, and the tricky part (normalizing an inherited layout against the
-// visibility flags) is testable on its own. See ./placement.test.ts.
+// Placement math for the monitor widget: which section sits in which grid cell,
+// in what order, and how tall its row is. Pure functions over a `Layout` — no
+// React, no DOM — so the floorplan board, the in-widget menu and the URL parser
+// all move sections the same way, and the tricky part (normalizing an inherited
+// layout against the visibility flags) is testable on its own. See
+// ./placement.test.ts.
//
-// URL shape: `.` between sections, `-` between columns —
+// THE MODEL IS ROWS OF CELLS, each cell a vertical stack of sections. It
+// replaces the old `SectionId[][]` columns, which had no rows, no spans and no
+// height control — so a section could never span the full width, four one-line
+// totals could never share a rail, and nothing could be given the leftover
+// height to scroll inside. Rendered as CSS Grid: `cols` tracks wide, one grid
+// row per Row, each cell placed with a col-span.
//
-// /widget?l=ctl.disk.wk.jobs-cln.act
+// URL shape. `l=` is still an INPUT format and is still what gets emitted
+// whenever a layout is expressible as plain columns, so every widget link copied
+// before rows existed keeps rendering identically:
//
-// Both are unreserved characters URLSearchParams leaves literal, unlike `,`
-// which would come back as %2C and make a hand-shared link unreadable.
+// /widget?l=ctl.disk.wk.jobs-cln.act (one auto row, N cells, span 1)
+//
+// Anything richer serializes to `g=`:
+//
+// rows separated by _
+// cells separated by -
+// sections separated by .
+// modifiers after * digits = span, r = flow row, f = row fills
+//
+// /widget?g=ctl*2_lsync.disk.cln.bf*2r_sched*f-act.clnl
+//
+// `_ - . *` are the only separators URLSearchParams leaves literal (`~` and `!`
+// come back percent-encoded), which is the same reason the original picked `-`
+// and `.`. Row height rides on a cell modifier rather than a second param: a row
+// is `fill` if ANY of its cells carries `f`, and serialization writes `f` on the
+// row's first cell.
import {
DEFAULT_ORDER,
@@ -20,175 +42,564 @@ import {
type SectionId,
} from "./sections";
-export const COLUMN_SEP = "-";
+export const ROW_SEP = "_";
+export const CELL_SEP = "-";
export const SECTION_SEP = ".";
+export const MOD_SEP = "*";
+// The old name for CELL_SEP, from when a cell was a whole column.
+export const COLUMN_SEP = CELL_SEP;
+
+// Six is where Tailwind's static `grid-cols-N` / `col-span-N` literals stop
+// being worth generating, and well past the point where a 320px widget has
+// anything useful to show. A hand-written layout with more is merged down rather
+// than rejected.
+export const MAX_COLUMNS = 6;
+export const MAX_ROWS = 6;
+
+// A row is `auto` (its content's height, the pre-rows behaviour) or `fill` (it
+// shares the leftover height of the widget's frame). One fill row anywhere is
+// what turns the widget from content-height into frame-height.
+export type RowSize = "auto" | "fill";
+// How a cell arranges the sections inside it: a vertical stack, or one dense
+// horizontal rail of totals.
+export type CellFlow = "col" | "row";
+
+export type Cell = { sections: SectionId[]; span: number; flow: CellFlow };
+export type Row = { size: RowSize; cells: Cell[] };
+export type Layout = { cols: number; rows: Row[] };
+
+export type CellPos = { row: number; cell: number };
+export type SectionPos = { row: number; cell: number; index: number };
+
+function clampInt(n: number, lo: number, hi: number): number {
+ const v = Math.round(Number.isFinite(n) ? n : lo);
+ return Math.max(lo, Math.min(hi, v));
+}
+
+function cloneCell(c: Cell): Cell {
+ return { sections: c.sections.slice(), span: c.span, flow: c.flow };
+}
+
+function cloneLayout(l: Layout): Layout {
+ return {
+ cols: l.cols,
+ rows: l.rows.map((r) => ({ size: r.size, cells: r.cells.map(cloneCell) })),
+ };
+}
-// Three columns is already past the point where a 320px widget has anything
-// useful to show; a hand-written `l=` with more is merged down rather than
-// rejected.
-export const MAX_COLUMNS = 3;
+export function emptyCell(span = 1): Cell {
+ return { sections: [], span, flow: "col" };
+}
-export type Columns = SectionId[][];
+// INVARIANT: every row's cell spans sum to exactly `cols`, and no row has more
+// cells than there are columns. Keeping it exact is what makes `cols` derivable
+// from a `g=` string (no separate param) and what keeps the grid gapless.
+//
+// `keep` is the one cell whose span the caller just set on purpose; everything
+// else absorbs the difference, shrinking from the tail first so the cell you
+// dragged is the one that wins.
+function fitRow(cells: Cell[], cols: number, keep = -1): Cell[] {
+ const next = cells.map(cloneCell);
+ if (next.length === 0) return [emptyCell(cols)];
+ // Too many cells for the track count: merge the tail rather than drop it.
+ while (next.length > cols) {
+ const last = next.pop() as Cell;
+ next[next.length - 1].sections.push(...last.sections);
+ }
+ for (const c of next) c.span = clampInt(c.span, 1, cols);
+ const n = next.length;
+ if (keep >= 0 && keep < n) {
+ // Leave every other cell at least one track.
+ next[keep].span = clampInt(next[keep].span, 1, cols - (n - 1));
+ }
+ let sum = next.reduce((s, c) => s + c.span, 0);
+ for (let i = n - 1; sum > cols && i >= 0; i--) {
+ if (i === keep) continue;
+ const take = Math.min(next[i].span - 1, sum - cols);
+ next[i].span -= take;
+ sum -= take;
+ }
+ // Only reachable for a malformed `keep`; clamp rather than emit a bad grid.
+ if (sum > cols && keep >= 0 && keep < n) {
+ next[keep].span -= sum - cols;
+ sum = cols;
+ }
+ if (sum < cols) {
+ let idx = n - 1;
+ if (idx === keep && n > 1) idx = n - 2;
+ next[idx].span += cols - sum;
+ }
+ return next;
+}
-function clone(columns: Columns): Columns {
- return columns.map((col) => col.slice());
+// Bring a whole layout back to the invariants: 1..MAX_COLUMNS tracks,
+// 1..MAX_ROWS rows (extras merged into the last, never dropped), spans summing
+// to `cols` per row.
+export function normalizeLayout(layout: Layout): Layout {
+ const cols = clampInt(layout.cols, 1, MAX_COLUMNS);
+ let rows = layout.rows.map((r) => ({
+ size: r.size === "fill" ? ("fill" as const) : ("auto" as const),
+ cells: r.cells.map(cloneCell),
+ }));
+ if (rows.length === 0) rows = [{ size: "auto", cells: [emptyCell(cols)] }];
+ if (rows.length > MAX_ROWS) {
+ const keep = rows.slice(0, MAX_ROWS);
+ const last = keep[MAX_ROWS - 1];
+ for (const extra of rows.slice(MAX_ROWS)) {
+ for (const cell of extra.cells) {
+ last.cells[last.cells.length - 1].sections.push(...cell.sections);
+ }
+ }
+ rows = keep;
+ }
+ return { cols, rows: rows.map((r) => ({ ...r, cells: fitRow(r.cells, cols) })) };
}
// One column holding every enabled section in the widget's original render
-// order — the layout a link with no `l=` gets, and the baseline
+// order — the layout a link with no `l=`/`g=` gets, and the baseline
// serializeLayout() compares against so an all-default config stays queryless.
-export function defaultColumns(config: SectionFlags): Columns {
- return [enabledSections(config)];
+export function defaultLayout(config: SectionFlags): Layout {
+ return {
+ cols: 1,
+ rows: [
+ {
+ size: "auto",
+ cells: [{ sections: enabledSections(config), span: 1, flow: "col" }],
+ },
+ ],
+ };
}
-export function sameLayout(a: Columns, b: Columns): boolean {
- return (
- a.length === b.length &&
- a.every((col, i) => col.length === b[i].length && col.every((id, j) => id === b[i][j]))
- );
+export function sameLayout(a: Layout, b: Layout): boolean {
+ if (a.cols !== b.cols || a.rows.length !== b.rows.length) return false;
+ return a.rows.every((row, r) => {
+ const other = b.rows[r];
+ if (row.size !== other.size || row.cells.length !== other.cells.length) {
+ return false;
+ }
+ return row.cells.every((cell, c) => {
+ const oc = other.cells[c];
+ return (
+ cell.span === oc.span &&
+ cell.flow === oc.flow &&
+ cell.sections.length === oc.sections.length &&
+ cell.sections.every((id, i) => id === oc.sections[i])
+ );
+ });
+ });
}
// Bring a layout back in line with the visibility flags: drop sections that are
-// now off, and append ones that are on but unplaced to the LAST column, in
-// DEFAULT_ORDER. The append rule is what keeps a baked link working when a new
-// section is enabled from the in-widget gear (or added to the registry in a
-// later release) — it shows up rather than silently vanishing.
-export function reconcileColumns(columns: Columns, config: SectionFlags): Columns {
+// now off, and append ones that are on but unplaced to the LAST CELL OF THE LAST
+// ROW, in DEFAULT_ORDER. The append rule is what keeps a baked link working when
+// a new section is enabled from the in-widget menu (or added to the registry in
+// a later release) — it shows up rather than silently vanishing.
+export function reconcileLayout(layout: Layout, config: SectionFlags): Layout {
const on = new Set(enabledSections(config));
const seen = new Set<SectionId>();
- const next: Columns = columns.map((col) =>
- col.filter((id) => {
- if (!on.has(id) || seen.has(id)) return false;
- seen.add(id);
- return true;
- }),
- );
- if (next.length === 0) next.push([]);
+ const next = normalizeLayout(layout);
+ for (const row of next.rows) {
+ for (const cell of row.cells) {
+ cell.sections = cell.sections.filter((id) => {
+ if (!on.has(id) || seen.has(id)) return false;
+ seen.add(id);
+ return true;
+ });
+ }
+ }
const missing = DEFAULT_ORDER.filter((id) => on.has(id) && !seen.has(id));
- if (missing.length > 0) next[next.length - 1].push(...missing);
+ if (missing.length > 0) {
+ const lastRow = next.rows[next.rows.length - 1];
+ lastRow.cells[lastRow.cells.length - 1].sections.push(...missing);
+ }
return next;
}
-// Decode + normalize an `l=` value. Unknown codes and repeats are dropped
-// (first placement wins), disabled sections are dropped, and anything enabled
-// but unlisted is appended by reconcileColumns. A value that survives none of
-// that falls back to the default single column rather than rendering nothing.
+// ---------------------------------------------------------------------------
+// Decode
+// ---------------------------------------------------------------------------
+
+type ParsedCell = { cell: Cell; fill: boolean };
+
+// "lsync.disk.cln.bf*2r" → the cell it describes plus whether it marked its row
+// as filling. Unknown modifier letters and out-of-range spans are dropped or
+// clamped, the same tolerance parseLayoutParam has always had for a hand-edited
+// link.
+function parseCell(raw: string, seen: Set<SectionId>): ParsedCell {
+ const star = raw.indexOf(MOD_SEP);
+ const body = star === -1 ? raw : raw.slice(0, star);
+ const mods = star === -1 ? "" : raw.slice(star + 1);
+ const sections: SectionId[] = [];
+ for (const code of body.split(SECTION_SEP)) {
+ const def = sectionByCode(code.trim());
+ if (!def || seen.has(def.id)) continue;
+ seen.add(def.id);
+ sections.push(def.id);
+ }
+ const digits = mods.replace(/[^0-9]/g, "");
+ const span = digits ? clampInt(Number(digits), 1, MAX_COLUMNS) : 1;
+ return {
+ cell: { sections, span, flow: mods.includes("r") ? "row" : "col" },
+ fill: mods.includes("f"),
+ };
+}
+
+function parseGrid(raw: string): Layout {
+ const seen = new Set<SectionId>();
+ const rows: Row[] = [];
+ for (const rowRaw of raw.split(ROW_SEP)) {
+ const parsed = rowRaw.split(CELL_SEP).map((c) => parseCell(c, seen));
+ rows.push({
+ size: parsed.some((p) => p.fill) ? "fill" : "auto",
+ cells: parsed.map((p) => p.cell),
+ });
+ }
+ // `cols` is derived, not carried: with spans summing to the track count per
+ // row (the invariant fitRow enforces), the widest row IS the column count.
+ const cols = rows.reduce(
+ (m, r) => Math.max(m, r.cells.reduce((s, c) => s + c.span, 0)),
+ 1,
+ );
+ return normalizeLayout({ cols, rows });
+}
+
+function parseColumns(raw: string): Layout {
+ const seen = new Set<SectionId>();
+ const cells = raw
+ .split(CELL_SEP)
+ .map((c) => parseCell(c.split(MOD_SEP)[0], seen).cell);
+ return normalizeLayout({ cols: cells.length, rows: [{ size: "auto", cells }] });
+}
+
+function isEmptyLayout(layout: Layout): boolean {
+ return layout.rows.every((r) => r.cells.every((c) => c.sections.length === 0));
+}
+
+// Decode + normalize the layout params. `g` wins when present; `l` decodes into
+// a single auto row of plain stacked cells, which is exactly what it always
+// meant. Unknown codes and repeats are dropped (first placement wins), disabled
+// sections are dropped, and anything enabled but unlisted is appended by
+// reconcileLayout. A value that survives none of that falls back to the default
+// single column rather than rendering nothing.
export function parseLayoutParam(
- raw: string | undefined,
+ l: string | undefined,
+ g: string | undefined,
config: SectionFlags,
-): Columns {
- if (!raw) return defaultColumns(config);
- const seen = new Set<SectionId>();
- const columns: Columns = [];
- for (const colRaw of raw.split(COLUMN_SEP)) {
- const col: SectionId[] = [];
- for (const code of colRaw.split(SECTION_SEP)) {
- const def = sectionByCode(code.trim());
- if (!def || seen.has(def.id)) continue;
- seen.add(def.id);
- col.push(def.id);
- }
- columns.push(col);
+): Layout {
+ const raw = g || l;
+ if (!raw) return defaultLayout(config);
+ const layout = g ? parseGrid(g) : parseColumns(l as string);
+ if (isEmptyLayout(layout)) return defaultLayout(config);
+ return reconcileLayout(layout, config);
+}
+
+// ---------------------------------------------------------------------------
+// Encode
+// ---------------------------------------------------------------------------
+
+// True when the layout says nothing a plain `l=` can't: one auto row of
+// single-track stacked cells. This is what keeps every pre-rows link
+// round-tripping to the identical string.
+export function isColumnsLayout(layout: Layout): boolean {
+ return (
+ layout.rows.length === 1 &&
+ layout.rows[0].size === "auto" &&
+ layout.rows[0].cells.every((c) => c.span === 1 && c.flow === "col")
+ );
+}
+
+function cellCode(cell: Cell, fill: boolean): string {
+ const body = cell.sections.map((id) => SECTION_BY_ID[id].code).join(SECTION_SEP);
+ let mods = "";
+ if (cell.span > 1) mods += String(cell.span);
+ if (cell.flow === "row") mods += "r";
+ if (fill) mods += "f";
+ return mods ? `${body}${MOD_SEP}${mods}` : body;
+}
+
+// `{}` when the layout is what a link with no params would already produce, so
+// an all-default config still serializes to a bare /widget. Returns `l` OR `g`,
+// never both — see buildWidgetQuery.
+export function serializeLayout(
+ layout: Layout,
+ config: SectionFlags,
+): { l?: string; g?: string } {
+ if (sameLayout(layout, defaultLayout(config))) return {};
+ if (isColumnsLayout(layout)) {
+ return {
+ l: layout.rows[0].cells
+ .map((c) => c.sections.map((id) => SECTION_BY_ID[id].code).join(SECTION_SEP))
+ .join(CELL_SEP),
+ };
}
- // A hand-written layout with too many columns is merged down, not rejected.
- const clamped =
- columns.length > MAX_COLUMNS ? setColumnCount(columns, MAX_COLUMNS) : columns;
- if (clamped.every((col) => col.length === 0)) return defaultColumns(config);
- return reconcileColumns(clamped, config);
+ return {
+ g: layout.rows
+ .map((row) =>
+ row.cells
+ .map((cell, i) => cellCode(cell, row.size === "fill" && i === 0))
+ .join(CELL_SEP),
+ )
+ .join(ROW_SEP),
+ };
}
-// "" when the layout is what a link with no `l=` would already produce, so an
-// all-default config still serializes to a bare /widget.
-export function serializeLayout(columns: Columns, config: SectionFlags): string {
- if (sameLayout(columns, defaultColumns(config))) return "";
- return columns
- .map((col) => col.map((id) => SECTION_BY_ID[id].code).join(SECTION_SEP))
- .join(COLUMN_SEP);
+// ---------------------------------------------------------------------------
+// Movement
+// ---------------------------------------------------------------------------
+
+function cellAt(layout: Layout, row: number, cell: number): Cell | null {
+ return layout.rows[row]?.cells[cell] ?? null;
+}
+
+export function findSection(layout: Layout, id: SectionId): SectionPos | null {
+ for (let row = 0; row < layout.rows.length; row++) {
+ const cells = layout.rows[row].cells;
+ for (let cell = 0; cell < cells.length; cell++) {
+ const index = cells[cell].sections.indexOf(id);
+ if (index !== -1) return { row, cell, index };
+ }
+ }
+ return null;
}
-// Swap with the neighbour above/below inside one column. Same idiom as the
+// Swap with the neighbour above/below inside one cell. Same idiom as the
// reorder in SocialLinksField.
export function moveWithin(
- columns: Columns,
- col: number,
+ layout: Layout,
+ row: number,
+ cell: number,
idx: number,
dir: -1 | 1,
-): Columns {
- const next = clone(columns);
- const target = next[col];
- if (!target) return columns;
+): Layout {
+ const next = cloneLayout(layout);
+ const target = cellAt(next, row, cell);
+ if (!target) return layout;
const j = idx + dir;
- if (j < 0 || j >= target.length) return columns;
- [target[idx], target[j]] = [target[j], target[idx]];
+ if (j < 0 || j >= target.sections.length) return layout;
+ const s = target.sections;
+ [s[idx], s[j]] = [s[j], s[idx]];
return next;
}
-// Move a section to the adjacent column, landing at the same height where that
-// column is long enough and at the end where it isn't.
-export function moveToColumn(
- columns: Columns,
- col: number,
+// Move a section to the adjacent cell in the same row, landing at the same
+// height where that cell is deep enough and at the end where it isn't.
+export function moveToCell(
+ layout: Layout,
+ row: number,
+ cell: number,
idx: number,
dir: -1 | 1,
-): Columns {
- const to = col + dir;
- if (to < 0 || to >= columns.length) return columns;
- const next = clone(columns);
- const [id] = next[col].splice(idx, 1);
- if (id === undefined) return columns;
- next[to].splice(Math.min(idx, next[to].length), 0, id);
+): Layout {
+ const to = cell + dir;
+ const from = cellAt(layout, row, cell);
+ if (!from || to < 0 || to >= layout.rows[row].cells.length) return layout;
+ const next = cloneLayout(layout);
+ const [id] = next.rows[row].cells[cell].sections.splice(idx, 1);
+ if (id === undefined) return layout;
+ const dest = next.rows[row].cells[to].sections;
+ dest.splice(Math.min(idx, dest.length), 0, id);
+ return next;
+}
+
+// Move a section to the adjacent row, into the cell at the same horizontal
+// position (or the last one, where that row is narrower).
+export function moveToRow(
+ layout: Layout,
+ row: number,
+ cell: number,
+ idx: number,
+ dir: -1 | 1,
+): Layout {
+ const to = row + dir;
+ if (to < 0 || to >= layout.rows.length) return layout;
+ const next = cloneLayout(layout);
+ const [id] = next.rows[row].cells[cell]?.sections.splice(idx, 1) ?? [];
+ if (id === undefined) return layout;
+ const destCells = next.rows[to].cells;
+ const destCell = destCells[Math.min(cell, destCells.length - 1)];
+ destCell.sections.push(id);
return next;
}
// Place a section at an explicit slot, pulling it out of wherever it currently
-// sits first. `index` is read against the column as the user sees it, so a move
-// down within one column has to account for the removal shifting things up.
+// sits first. `index` is read against the cell as the user sees it, so a move
+// down within one cell has to account for the removal shifting things up.
export function insertAt(
- columns: Columns,
+ layout: Layout,
id: SectionId,
- col: number,
+ row: number,
+ cell: number,
index: number,
-): Columns {
- const next = clone(columns);
- if (col < 0 || col >= next.length) return columns;
- let target = index;
- for (let c = 0; c < next.length; c++) {
- const at = next[c].indexOf(id);
- if (at === -1) continue;
- next[c].splice(at, 1);
- if (c === col && at < index) target -= 1;
- break;
- }
- next[col].splice(Math.max(0, Math.min(target, next[col].length)), 0, id);
+): Layout {
+ const next = cloneLayout(layout);
+ const target = cellAt(next, row, cell);
+ if (!target) return layout;
+ let at = index;
+ const found = findSection(next, id);
+ if (found) {
+ next.rows[found.row].cells[found.cell].sections.splice(found.index, 1);
+ if (found.row === row && found.cell === cell && found.index < index) at -= 1;
+ }
+ target.sections.splice(Math.max(0, Math.min(at, target.sections.length)), 0, id);
return next;
}
-export function removeSection(columns: Columns, id: SectionId): Columns {
- return columns.map((col) => col.filter((x) => x !== id));
+export function removeSection(layout: Layout, id: SectionId): Layout {
+ const next = cloneLayout(layout);
+ for (const row of next.rows) {
+ for (const cell of row.cells) {
+ cell.sections = cell.sections.filter((x) => x !== id);
+ }
+ }
+ return next;
}
-// Grow by appending empty columns; shrink by merging every trailing column into
-// the last one that survives, so nothing is ever silently dropped.
-export function setColumnCount(columns: Columns, n: number): Columns {
- const count = Math.max(1, Math.min(MAX_COLUMNS, Math.round(n)));
- if (count === columns.length) return columns;
- if (count > columns.length) {
- const next = clone(columns);
- while (next.length < count) next.push([]);
- return next;
+// ---------------------------------------------------------------------------
+// Structure
+// ---------------------------------------------------------------------------
+
+// Grow by appending an empty cell per row; shrink by merging every trailing cell
+// into the last one that survives, so nothing is ever silently dropped.
+export function setColumnCount(layout: Layout, n: number): Layout {
+ const cols = clampInt(n, 1, MAX_COLUMNS);
+ if (cols === layout.cols) return layout;
+ const next = cloneLayout(layout);
+ next.cols = cols;
+ if (cols > layout.cols) {
+ for (const row of next.rows) {
+ for (let i = layout.cols; i < cols; i++) row.cells.push(emptyCell(1));
+ }
}
- const next = clone(columns.slice(0, count));
- for (const col of columns.slice(count)) next[count - 1].push(...col);
+ return normalizeLayout(next);
+}
+
+// Grow by appending a row of empty single-track cells — one per column, so a new
+// row is a place to drop things rather than one undivided band; shrink by
+// merging every trailing row's sections into the last surviving row's last cell.
+export function setRowCount(layout: Layout, n: number): Layout {
+ const count = clampInt(n, 1, MAX_ROWS);
+ if (count === layout.rows.length) return layout;
+ const next = cloneLayout(layout);
+ if (count > next.rows.length) {
+ while (next.rows.length < count) {
+ next.rows.push({
+ size: "auto",
+ cells: Array.from({ length: next.cols }, () => emptyCell(1)),
+ });
+ }
+ return normalizeLayout(next);
+ }
+ const keep = next.rows.slice(0, count);
+ const last = keep[count - 1];
+ for (const row of next.rows.slice(count)) {
+ for (const cell of row.cells) {
+ last.cells[last.cells.length - 1].sections.push(...cell.sections);
+ }
+ }
+ next.rows = keep;
+ return normalizeLayout(next);
+}
+
+export function setRowSize(layout: Layout, row: number, size: RowSize): Layout {
+ if (!layout.rows[row] || layout.rows[row].size === size) return layout;
+ const next = cloneLayout(layout);
+ next.rows[row].size = size;
return next;
}
-export function findSection(
- columns: Columns,
- id: SectionId,
-): { col: number; index: number } | null {
- for (let col = 0; col < columns.length; col++) {
- const index = columns[col].indexOf(id);
- if (index !== -1) return { col, index };
+// Widen/narrow one cell. The row keeps summing to the track count, so the
+// neighbours give up (or take back) what this one gains — the cell you dragged
+// is the one that wins.
+//
+// A neighbour squeezed to nothing is REMOVED and its sections merged into the
+// cell that took its tracks, rather than the widen being refused: "make this
+// span the whole row" is the commonest thing anyone asks a grid for, and
+// refusing it because some other cell still holds a track would be the wrong
+// answer. Narrowing the last cell in a row is the exact inverse — the freed
+// tracks become a new empty cell you can drop into.
+function applySpan(cells: Cell[], cols: number, keep: number, want: number): Cell[] {
+ const next = cells.map(cloneCell);
+ next[keep].span = clampInt(want, 1, cols);
+ let sum = next.reduce((s, c) => s + c.span, 0);
+ // Nearest first: the cell you are growing into is the one beside you.
+ const order: number[] = [];
+ for (let i = keep + 1; i < next.length; i++) order.push(i);
+ for (let i = keep - 1; i >= 0; i--) order.push(i);
+ for (const i of order) {
+ if (sum <= cols) break;
+ const take = Math.min(next[i].span, sum - cols);
+ next[i].span -= take;
+ sum -= take;
}
- return null;
+ const merged: Cell[] = [];
+ for (let i = 0; i < next.length; i++) {
+ if (i !== keep && next[i].span <= 0) {
+ next[keep].sections.push(...next[i].sections);
+ continue;
+ }
+ merged.push(next[i]);
+ }
+ sum = merged.reduce((s, c) => s + c.span, 0);
+ if (sum < cols) {
+ const k = merged.indexOf(next[keep]);
+ let idx = merged.length - 1;
+ if (idx === k) idx = merged.length - 2;
+ if (idx < 0) merged.push(emptyCell(cols - sum));
+ else merged[idx].span += cols - sum;
+ }
+ return merged;
+}
+
+export function setCellSpan(
+ layout: Layout,
+ row: number,
+ cell: number,
+ span: number,
+): Layout {
+ const target = cellAt(layout, row, cell);
+ if (!target) return layout;
+ const next = cloneLayout(layout);
+ next.rows[row].cells = applySpan(next.rows[row].cells, next.cols, cell, span);
+ return sameLayout(next, layout) ? layout : next;
+}
+
+export function setCellFlow(
+ layout: Layout,
+ row: number,
+ cell: number,
+ flow: CellFlow,
+): Layout {
+ const target = cellAt(layout, row, cell);
+ if (!target || target.flow === flow) return layout;
+ const next = cloneLayout(layout);
+ next.rows[row].cells[cell].flow = flow;
+ return next;
+}
+
+export function addCell(layout: Layout, row: number): Layout {
+ const target = layout.rows[row];
+ if (!target || target.cells.length >= layout.cols) return layout;
+ const next = cloneLayout(layout);
+ next.rows[row].cells.push(emptyCell(1));
+ next.rows[row].cells = fitRow(next.rows[row].cells, next.cols);
+ return next;
+}
+
+// Remove a cell, merging its sections into its neighbour rather than dropping
+// them. The last cell in a row can't be removed — a row always has one.
+export function removeCell(layout: Layout, row: number, cell: number): Layout {
+ const target = layout.rows[row];
+ if (!target || target.cells.length <= 1 || !target.cells[cell]) return layout;
+ const next = cloneLayout(layout);
+ const [gone] = next.rows[row].cells.splice(cell, 1);
+ const into = next.rows[row].cells[Math.max(0, cell - 1)];
+ into.sections.push(...gone.sections);
+ next.rows[row].cells = fitRow(next.rows[row].cells, next.cols);
+ return next;
+}
+
+// Whether any row asks for the leftover height — the one thing that changes the
+// widget from content-height to frame-height.
+export function hasFillRow(layout: Layout): boolean {
+ return layout.rows.some((r) => r.size === "fill");
}
diff --git a/editor/app/widget/lib/sections.ts b/editor/app/widget/lib/sections.ts
@@ -31,7 +31,7 @@ export type SectionId =
// placement it is being placed into. Taking this rather than WidgetConfig lets
// parseWidgetConfig ask "which sections are on?" while it is still building the
// config that will hold the answer.
-export type SectionFlags = Omit<WidgetConfig, "columns">;
+export type SectionFlags = Omit<WidgetConfig, "layout">;
// Only the boolean flags are togglable as a section option, so a typo like
// `key: "pollSeconds"` is a compile error rather than a checkbox that writes a
diff --git a/editor/app/widget/page.tsx b/editor/app/widget/page.tsx
@@ -1,6 +1,12 @@
import type { Metadata } from "next";
+import { getPaths } from "yt-dlp-transcript-common/lib/paths";
+import {
+ findWidgetPreset,
+ readWidgetPresets,
+} from "yt-dlp-transcript-common/lib/widgetPresets";
import { buildActiveJobsPayload } from "../jobs/active/buildActiveJobs";
import { buildWorkersPayload } from "../workers/buildWorkers";
+import { BUILT_IN_PRESETS } from "./lib/builtInPresets";
import { MonitorWidget } from "./components/MonitorWidget";
import { parseWidgetConfig } from "./lib/config";
@@ -8,18 +14,52 @@ export const dynamic = "force-dynamic";
export const metadata: Metadata = { title: "Monitor" };
+type RawParams = Record<string, string | string[] | undefined>;
+
+function first(v: string | string[] | undefined): string | undefined {
+ return Array.isArray(v) ? v[0] : v;
+}
+
+// Expand `?preset=<id|name>` into the params it stands for. A preset stores a
+// query string, so this is a merge of two links — and ANY param written
+// explicitly in the URL WINS, so `?preset=cockpit&titles=0` is the preset with
+// its titles off rather than a contradiction. Built-ins resolve too, and an
+// unknown reference is simply the rest of the URL.
+async function resolvePreset(params: RawParams): Promise<RawParams> {
+ const ref = first(params.preset)?.trim();
+ if (!ref) return params;
+ const saved = await readWidgetPresets(getPaths());
+ const preset =
+ findWidgetPreset(saved, ref) ??
+ BUILT_IN_PRESETS.find(
+ (p) => p.id === ref || p.name.toLowerCase() === ref.toLowerCase(),
+ ) ??
+ null;
+ if (!preset) return params;
+ const merged: RawParams = {};
+ for (const [k, v] of new URLSearchParams(preset.query)) merged[k] = v;
+ for (const [k, v] of Object.entries(params)) {
+ if (k !== "preset") merged[k] = v;
+ }
+ return merged;
+}
+
// Bare, read-only monitor. The root layout strips its chrome (see AppFrame) so
// it can be embedded in a small pinned window or iframe. What it shows is driven
// entirely by GET params (see lib/config); the /widget/builder page composes
-// those links. Initial payloads are built server-side here for a flicker-free
-// first paint; MonitorWidget then polls the same /api endpoints for live data.
+// those links, and `?preset=` names one that was saved. Initial payloads are
+// built server-side here for a flicker-free first paint; MonitorWidget then
+// polls the same /api endpoints for live data.
export default async function WidgetPage({
searchParams,
}: {
searchParams: Promise<Record<string, string | string[] | undefined>>;
}) {
- const config = parseWidgetConfig((await searchParams) ?? {});
- const initialJobs = config.jobs ? await buildActiveJobsPayload() : null;
+ const config = parseWidgetConfig(await resolvePreset((await searchParams) ?? {}));
+ // The disk strip reads its numbers off this payload as well, so a
+ // disk-without-jobs widget needs it built too (see MonitorWidget's poll gate).
+ const initialJobs =
+ config.jobs || config.disk ? await buildActiveJobsPayload() : null;
// Controls need the workers payload (for `paused`) even if the Workers section
// itself is hidden.
const initialWorkers =
diff --git a/editor/app/widget/presetActions.ts b/editor/app/widget/presetActions.ts
@@ -0,0 +1,60 @@
+"use server";
+
+import { revalidatePath } from "next/cache";
+import { getPaths } from "yt-dlp-transcript-common/lib/paths";
+import {
+ addWidgetPreset,
+ moveWidgetPreset,
+ removeWidgetPreset,
+ renameWidgetPreset,
+} from "yt-dlp-transcript-common/lib/widgetPresets";
+
+// Server actions behind the preset row. The in-widget menu reads
+// back through /api/widget/presets (it is a client component with no server
+// parent to revalidate into); the builder page SSR-seeds and this
+// revalidatePath refreshes it.
+
+export type PresetActionResult = { ok: true } | { ok: false; error: string };
+
+function refreshBuilder(): void {
+ revalidatePath("/widget/builder");
+}
+
+// Save `query` under `name`. Saving over an existing name overwrites it — a
+// named slot, not an ever-growing list of same-named entries.
+export async function saveWidgetPresetAction(
+ name: string,
+ query: string,
+): Promise<PresetActionResult> {
+ const preset = await addWidgetPreset(getPaths(), name, query);
+ if (!preset) return { ok: false, error: "A preset needs a name." };
+ refreshBuilder();
+ return { ok: true };
+}
+
+export async function renameWidgetPresetAction(
+ id: string,
+ name: string,
+): Promise<PresetActionResult> {
+ const ok = await renameWidgetPreset(getPaths(), id, name);
+ if (!ok) return { ok: false, error: "A preset needs a name." };
+ refreshBuilder();
+ return { ok: true };
+}
+
+export async function deleteWidgetPresetAction(
+ id: string,
+): Promise<PresetActionResult> {
+ await removeWidgetPreset(getPaths(), id);
+ refreshBuilder();
+ return { ok: true };
+}
+
+export async function moveWidgetPresetAction(
+ id: string,
+ dir: -1 | 1,
+): Promise<PresetActionResult> {
+ await moveWidgetPreset(getPaths(), id, dir);
+ refreshBuilder();
+ return { ok: true };
+}
diff --git a/editor/e2e/bookmarks.spec.ts b/editor/e2e/bookmarks.spec.ts
@@ -1,361 +0,0 @@
-// Bookmark a running/finished job and re-launch it later. The key behavior is
-// that re-running RE-DERIVES the work from the channel's current state rather
-// than replaying a frozen video-id list:
-// - whisper-all re-scans the disk for audio-without-transcript;
-// - bucket jobs (retry partial downloads) read the CURRENT snapshot bucket.
-// Both proofs change what the channel needs between the original run and the
-// re-run, then assert the re-run acts on the new set.
-//
-// The bookmarks UI is split: a compact quick-access menu on /jobs and
-// /jobs/active (one fire-and-forget run button per bookmark, NO inline log),
-// and a full management page at /jobs/bookmarks (rename, delete, metadata, and
-// the streaming "Run again" with its inline log). The behavioral re-derive
-// proofs run against the management page's streaming log.
-
-import { mkdir, rename, rm, writeFile } from "node:fs/promises";
-import { test, expect } from "@playwright/test";
-import {
- channelStage,
- generateReport,
- pathExists,
- resetData,
- resolvePath,
-} from "./helpers";
-
-const CHANNEL = "test-transcribe";
-const ROOT = `test-transcripts/channels/${CHANNEL}`;
-
-// Run "Transcribe missing" on the audio fixture and bookmark the resulting
-// whisper-all job from the jobs table (newest bookmarkable row). Ends on /jobs,
-// where the compact bookmarks menu is now shown.
-async function transcribeAndBookmark(page: import("@playwright/test").Page) {
- await generateReport(page, CHANNEL);
- await page.goto(channelStage(CHANNEL, "transcribe"));
- await page.getByRole("button", { name: "Transcribe missing" }).click();
- await expect(page.getByLabel("Transcribe missing output")).toContainText(
- "3 succeeded",
- { timeout: 30_000 },
- );
- await page.goto("/jobs");
- await page
- .getByRole("button", { name: /bookmark job/i })
- .first()
- .click();
- await expect(page.getByRole("heading", { name: "Bookmarks" })).toBeVisible({
- timeout: 15_000,
- });
-}
-
-test("compact menu: one-click run is fire-and-forget with no inline log, and links to Manage", async ({
- page,
-}) => {
- test.setTimeout(60_000);
- await resetData("one-transcribe-channel-with-audio");
- await transcribeAndBookmark(page);
-
- const menu = page.getByRole("region", { name: "Bookmarked jobs" });
- // "Manage" links to the full management page.
- await expect(menu.getByRole("link", { name: "Manage" })).toHaveAttribute(
- "href",
- "/jobs/bookmarks",
- );
- // The menu is buttons only — no inline shell / expanding log.
- await expect(menu.getByRole("log")).toHaveCount(0);
-
- const runBtn = menu.getByRole("button", {
- name: "run bookmark whisper-all · test-transcribe",
- });
- await expect(runBtn).toBeVisible();
- await runBtn.click();
-
- // Fire-and-forget: the button confirms the launch and still no log appears.
- await expect(runBtn).toHaveText("Launched ✓", { timeout: 15_000 });
- await expect(menu.getByRole("log")).toHaveCount(0);
-});
-
-test("management page: Run again re-derives the current missing set", async ({
- page,
-}) => {
- test.setTimeout(60_000);
- await resetData("one-transcribe-channel-with-audio");
- await transcribeAndBookmark(page);
- await page.goto("/jobs/bookmarks");
-
- const row = page.getByLabel("bookmark whisper-all · test-transcribe", {
- exact: true,
- });
- await expect(row).toBeVisible();
-
- // Change the channel's state: remove one transcript so exactly one video now
- // needs transcribing again. A frozen-id replay would re-attempt all three; a
- // re-derive does only the one that's currently missing.
- await rm(resolvePath(`${ROOT}/data/vidA/transcript.json`));
-
- await row.getByRole("button", { name: "Run again" }).click();
- await expect(
- page.getByRole("log", { name: /Run whisper-all/ }),
- ).toContainText("1 succeeded", { timeout: 30_000 });
- expect(await pathExists(`${ROOT}/data/vidA/transcript.json`)).toBe(true);
-});
-
-test("management page: rename a bookmark and see created / last-run metadata", async ({
- page,
-}) => {
- test.setTimeout(60_000);
- await resetData("one-transcribe-channel-with-audio");
- await transcribeAndBookmark(page);
- await page.goto("/jobs/bookmarks");
-
- const row = page.getByLabel("bookmark whisper-all · test-transcribe", {
- exact: true,
- });
- await expect(row).toContainText("Created");
- await expect(row).toContainText("Last run never");
-
- // Rename.
- await row
- .getByRole("button", { name: "rename bookmark whisper-all · test-transcribe" })
- .click();
- await row
- .getByLabel("new name for whisper-all · test-transcribe")
- .fill("Nightly transcribe");
- await row.getByRole("button", { name: "Save" }).click();
-
- const renamed = page.getByLabel("bookmark Nightly transcribe", {
- exact: true,
- });
- await expect(renamed).toBeVisible({ timeout: 15_000 });
-
- // Run again updates the last-run timestamp (away from "never").
- await renamed.getByRole("button", { name: "Run again" }).click();
- await expect(
- page.getByRole("log", { name: /Run Nightly transcribe/ }),
- ).toContainText("Whisper batch:", { timeout: 30_000 });
- await expect(
- page.getByLabel("bookmark Nightly transcribe", { exact: true }),
- ).not.toContainText("Last run never", { timeout: 15_000 });
-});
-
-test("management page: Run again retries whatever is partial now, then delete it", async ({
- page,
-}) => {
- test.setTimeout(60_000);
- await resetData("one-transcribe-channel-with-audio");
- // The retry flow reads the playlist to resolve each id's URL. Slug-style URLs
- // keep extractVideoId() === the id without needing platform claim ids.
- await writeFile(
- resolvePath(`${ROOT}/playlist`),
- "https://www.youtube.com/watch?v=vidA\nhttps://www.youtube.com/watch?v=vidB\n",
- );
-
- // Strand vidA mid-download so it surfaces in the partial-downloads bucket.
- await rename(
- resolvePath(`${ROOT}/data/vidA/audio.m4a`),
- resolvePath(`${ROOT}/data/vidA/audio.m4a.part`),
- );
-
- // Original run: resume the partial download (vidA).
- await generateReport(page, CHANNEL);
- await page.goto(channelStage(CHANNEL, "download"));
- await page
- .getByLabel("retry resume partial downloads bucket")
- .getByRole("button", { name: /^Retry \(1\)$/ })
- .click();
- await expect(
- page.getByLabel("Retry resume partial downloads output"),
- ).toContainText("Managed download complete", { timeout: 30_000 });
- expect(await pathExists(`${ROOT}/data/vidA/audio.m4a`)).toBe(true);
-
- // Bookmark the retry-bucket job, then manage it on /jobs/bookmarks.
- await page.goto("/jobs");
- await page
- .getByRole("button", { name: /bookmark job/i })
- .first()
- .click();
- await expect(page.getByRole("heading", { name: "Bookmarks" })).toBeVisible({
- timeout: 15_000,
- });
- await page.goto("/jobs/bookmarks");
- const row = page.getByLabel(
- "bookmark retry-bucket (partialDownloads) · test-transcribe",
- { exact: true },
- );
- await expect(row).toBeVisible({ timeout: 15_000 });
-
- // Now make a DIFFERENT video partial (vidB) and force a fresh snapshot, so the
- // partial-downloads bucket currently holds vidB, not the original vidA.
- await rename(
- resolvePath(`${ROOT}/data/vidB/audio.m4a`),
- resolvePath(`${ROOT}/data/vidB/audio.m4a.part`),
- );
- await rm(resolvePath(`${ROOT}/snapshot.json`), { force: true });
- await generateReport(page, CHANNEL);
- await page.goto(channelStage(CHANNEL, "download")); // regenerates snapshot when absent
- await expect(
- page.getByRole("heading", { name: /Partial downloads \(1\)/ }),
- ).toBeVisible();
-
- // Run again: re-derives the current bucket → downloads vidB (NOT the original
- // vidA). This is the proof that re-running is not a frozen-id replay.
- await page.goto("/jobs/bookmarks");
- await row.getByRole("button", { name: "Run again" }).click();
- await expect(
- page.getByRole("log", { name: /Run retry-bucket/ }),
- ).toContainText("Managed download complete", { timeout: 30_000 });
- expect(await pathExists(`${ROOT}/data/vidB/audio.m4a`)).toBe(true);
-
- // Delete the bookmark (confirm-gated).
- await row
- .getByRole("button", {
- name: "delete bookmark retry-bucket (partialDownloads) · test-transcribe",
- exact: true,
- })
- .click();
- await row
- .getByRole("button", {
- name: "confirm delete bookmark retry-bucket (partialDownloads) · test-transcribe",
- exact: true,
- })
- .click();
- await expect(row).toHaveCount(0, { timeout: 15_000 });
-});
-
-test("management page: reorder bookmarks with ↑/↓, persisted and reflected in the compact menu", async ({
- page,
-}) => {
- test.setTimeout(60_000);
- await resetData("one-transcribe-channel-with-audio");
-
- // Seed two distinct bookmarks for the existing channel directly into the
- // store. Reordering only depends on the persisted array order, so this avoids
- // running two real jobs (the re-derive behavior is covered by other tests).
- // Initial array order is [whisper-all, retry-bucket].
- await mkdir(resolvePath("test-transcripts/.bookmarks"), { recursive: true });
- await writeFile(
- resolvePath("test-transcripts/.bookmarks/bookmarks.json"),
- JSON.stringify(
- {
- v: 1,
- bookmarks: [
- {
- id: "bm-whisper",
- name: "whisper-all · test-transcribe",
- spec: { kind: "whisper-all", slug: CHANNEL },
- createdAt: 1,
- },
- {
- id: "bm-retry",
- name: "retry-bucket (partialDownloads) · test-transcribe",
- spec: {
- kind: "retry-bucket",
- slug: CHANNEL,
- bucket: "partialDownloads",
- },
- createdAt: 2,
- },
- ],
- },
- null,
- 2,
- ) + "\n",
- );
-
- const WHISPER = "bookmark whisper-all · test-transcribe";
- const RETRY = "bookmark retry-bucket (partialDownloads) · test-transcribe";
-
- await page.goto("/jobs/bookmarks");
- const items = page
- .getByRole("region", { name: "Bookmarked jobs" })
- .getByRole("listitem");
- await expect(items).toHaveCount(2);
- await expect(items.nth(0)).toHaveAttribute("aria-label", WHISPER);
- await expect(items.nth(1)).toHaveAttribute("aria-label", RETRY);
-
- // Move the top bookmark (whisper-all) down → [retry-bucket, whisper-all].
- await page.getByRole("button", { name: `move ${WHISPER} down` }).click();
- await expect(items.nth(0)).toHaveAttribute("aria-label", RETRY);
- await expect(items.nth(1)).toHaveAttribute("aria-label", WHISPER);
-
- // Arrows are disabled at the bounds.
- await expect(
- page.getByRole("button", { name: `move ${RETRY} up` }),
- ).toBeDisabled();
- await expect(
- page.getByRole("button", { name: `move ${WHISPER} down` }),
- ).toBeDisabled();
-
- // The new order survives a reload (it was written to disk).
- await page.reload();
- await expect(items.nth(0)).toHaveAttribute("aria-label", RETRY);
- await expect(items.nth(1)).toHaveAttribute("aria-label", WHISPER);
-
- // …and the compact menu on /jobs reflects the same order.
- await page.goto("/jobs");
- await expect(
- page
- .getByRole("region", { name: "Bookmarked jobs" })
- .getByRole("button", { name: /^run bookmark/ })
- .nth(0),
- ).toHaveAccessibleName(
- "run bookmark retry-bucket (partialDownloads) · test-transcribe",
- );
-});
-
-test("management page: an empty re-derived bucket reads as a neutral notice, not an error", async ({
- page,
-}) => {
- test.setTimeout(60_000);
- await resetData("one-transcribe-channel-with-audio");
- await writeFile(
- resolvePath(`${ROOT}/playlist`),
- "https://www.youtube.com/watch?v=vidA\n",
- );
- await rename(
- resolvePath(`${ROOT}/data/vidA/audio.m4a`),
- resolvePath(`${ROOT}/data/vidA/audio.m4a.part`),
- );
-
- await generateReport(page, CHANNEL);
- await page.goto(channelStage(CHANNEL, "download"));
- await page
- .getByLabel("retry resume partial downloads bucket")
- .getByRole("button", { name: /^Retry \(1\)$/ })
- .click();
- await expect(
- page.getByLabel("Retry resume partial downloads output"),
- ).toContainText("Managed download complete", { timeout: 30_000 });
-
- await page.goto("/jobs");
- await page
- .getByRole("button", { name: /bookmark job/i })
- .first()
- .click();
- await expect(page.getByRole("heading", { name: "Bookmarks" })).toBeVisible({
- timeout: 15_000,
- });
- await page.goto("/jobs/bookmarks");
- const row = page.getByLabel(
- "bookmark retry-bucket (partialDownloads) · test-transcribe",
- { exact: true },
- );
- await expect(row).toBeVisible({ timeout: 15_000 });
-
- // vidA is fully downloaded now, so the partial-downloads bucket is empty.
- // Refresh the snapshot to reflect that.
- await rm(resolvePath(`${ROOT}/snapshot.json`), { force: true });
- await generateReport(page, CHANNEL);
- await page.goto(channelStage(CHANNEL, "download"));
- await expect(
- page.getByRole("heading", { name: /Partial downloads/ }),
- ).toHaveCount(0);
-
- await page.goto("/jobs/bookmarks");
- await row.getByRole("button", { name: "Run again" }).click();
- // Neutral status, not a red alert.
- await expect(
- page.getByRole("status", { name: /Run retry-bucket.*notice/ }),
- ).toContainText("Nothing to retry right now", { timeout: 30_000 });
- await expect(
- page.getByRole("alert", { name: /Run retry-bucket/ }),
- ).toHaveCount(0);
-});
diff --git a/editor/e2e/jobs-retry.spec.ts b/editor/e2e/jobs-retry.spec.ts
@@ -4,9 +4,9 @@
// no Retry. Failed jobs are staged as sidecar + log files so the scenario is
// deterministic and exercises the sidecar-fallback path directly.
-import { mkdir, writeFile } from "node:fs/promises";
+import { mkdir, rename, writeFile } from "node:fs/promises";
import { test, expect } from "@playwright/test";
-import { resetData, resolvePath } from "./helpers";
+import { generateReport, pathExists, resetData, resolvePath } from "./helpers";
import { baseUrl } from "./baseUrl";
async function invalidateCache() {
@@ -88,3 +88,60 @@ test("Retry re-runs a failed job from its spec; spec-less failures offer none",
.filter({ hasText: "running" });
await expect(runningSync).toBeVisible({ timeout: 15_000 });
});
+
+// A second archived job of a DIFFERENT kind, so the replay table is driven for
+// more than `sync`. A bucket spec stores only the bucket CATEGORY — never a
+// frozen video-id list — so retrying it has to re-derive the bucket's current
+// members. The proof: the video that was partial when the job originally ran is
+// not the video that is partial now, and the retry acts on the current one.
+test("Retry on an archived bucket job re-derives the bucket's current members", async ({
+ page,
+}) => {
+ test.setTimeout(90_000);
+ const CHANNEL = "test-transcribe";
+ const ROOT = `test-transcripts/channels/${CHANNEL}`;
+ await resetData("one-transcribe-channel-with-audio");
+ // The retry flow reads the playlist to resolve each id's URL. Slug-style URLs
+ // keep extractVideoId() === the id without needing platform claim ids.
+ await writeFile(
+ resolvePath(`${ROOT}/playlist`),
+ "https://www.youtube.com/watch?v=vidA\nhttps://www.youtube.com/watch?v=vidB\n",
+ );
+ // Strand vidB — NOT the vidA the archived job would have run on — so the
+ // partial-downloads bucket currently holds exactly vidB.
+ await rename(
+ resolvePath(`${ROOT}/data/vidB/audio.m4a`),
+ resolvePath(`${ROOT}/data/vidB/audio.m4a.part`),
+ );
+ await generateReport(page, CHANNEL);
+ await page.goto(`/channels/${CHANNEL}`); // regenerates the snapshot when absent
+ await expect(
+ page.getByRole("heading", { name: /Partial downloads \(1\)/ }),
+ ).toBeVisible();
+
+ const ts = 1_700_000_000_000;
+ await writeArchivedJob("failedbucket1", {
+ kind: "retry-bucket",
+ channelSlug: CHANNEL,
+ queueKey: "platform:youtube",
+ status: "failed",
+ queuedAt: ts,
+ startedAt: ts,
+ endedAt: ts + 1000,
+ exitCode: 1,
+ spec: { kind: "retry-bucket", slug: CHANNEL, bucket: "partialDownloads" },
+ });
+ await invalidateCache();
+
+ await page.goto("/jobs");
+ const bucketRow = page.getByRole("row").filter({ hasText: "failedbucket1" });
+ await expect(bucketRow.getByRole("button", { name: "Retry" })).toBeVisible();
+ await bucketRow.getByRole("button", { name: "Retry" }).click();
+
+ // vidB (the CURRENT bucket member) gets its download resumed.
+ await expect
+ .poll(() => pathExists(`${ROOT}/data/vidB/audio.m4a`), {
+ timeout: 45_000,
+ })
+ .toBe(true);
+});
diff --git a/editor/e2e/settings.spec.ts b/editor/e2e/settings.spec.ts
@@ -255,3 +255,51 @@ test("an unrelated settings save does not reset the digest prompt shape", async
expect(saved.digest?.timestampMode).toBe("absolute");
expect(saved.digest?.promptVariant).toBe("keepme");
});
+
+// THE BUG THIS EXISTS FOR: the sortformer engine shipped with its settings
+// reachable only by hand-editing settings.json, because the form simply had no
+// controls for them. A field the form does not render is a field an operator
+// cannot set — and, worse, the diarization branch rebuilds the whole block on
+// save, so a control added to the markup but forgotten in the action would look
+// like it worked and silently revert.
+test("the sortformer engine settings round-trip through the form", async ({
+ page,
+}) => {
+ await page.goto("/settings");
+
+ // Targeted by NAME, not by label. getByLabel matches by substring and these
+ // labels wrap their whole explanatory hint, so the accessible name is a
+ // paragraph — "Engine" matches nothing exactly and matches "Engine threads"
+ // loosely. The control's name is the thing the action actually reads.
+ await page.locator('select[name="diarizationEngine"]').selectOption("sortformer");
+ await page.locator('select[name="diarizationBackend"]').selectOption("cpu");
+ await page
+ .getByLabel("Sortformer engine binary")
+ .fill("/opt/sortformer/diarize-file");
+ await page
+ .getByLabel("Sortformer model (GGUF)")
+ .fill("/opt/sortformer/model.gguf");
+
+ await page.getByRole("button", { name: /save settings/i }).click();
+ await expect(
+ page.getByRole("status").filter({ hasText: "Saved" }),
+ ).toBeVisible();
+
+ const saved = await readJson<{
+ diarization?: {
+ engine: string;
+ backend: string;
+ sortformerBin: string;
+ sortformerModel: string;
+ segModel: string;
+ threshold: number;
+ };
+ }>("test-settings.json");
+ expect(saved.diarization?.engine).toBe("sortformer");
+ expect(saved.diarization?.backend).toBe("cpu");
+ expect(saved.diarization?.sortformerBin).toBe("/opt/sortformer/diarize-file");
+ expect(saved.diarization?.sortformerModel).toBe("/opt/sortformer/model.gguf");
+ // The sherpa fields are still carried, not clobbered by selecting the other
+ // engine — switching back must not mean re-entering three paths.
+ expect(saved.diarization?.threshold).toBe(0.9);
+});
diff --git a/editor/e2e/widget.spec.ts b/editor/e2e/widget.spec.ts
@@ -11,6 +11,7 @@ import { mkdir, writeFile } from "node:fs/promises";
import { dirname } from "node:path";
import { test, expect } from "@playwright/test";
import {
+ readJson,
resetData,
resolvePath,
writeSettings,
@@ -58,6 +59,31 @@ async function seedCleanableChannel(
await fetch(`${baseUrl}/api/test/invalidate-cache`).catch(() => {});
}
+// Settings with a backfill KIND registered (diarization), so the widget's
+// backfill strip has something to report and its pause has something to hold.
+// Same shape as backfill.spec.ts's — `anyKind` is what gates both.
+const BACKFILL_SETTINGS = {
+ diarization: {
+ enabled: true,
+ inlineAfterTranscribe: false,
+ threshold: 0.5,
+ threads: 1,
+ python: "python3",
+ segModel: "/dev/null",
+ embModel: "/dev/null",
+ concurrency: 1,
+ },
+ backfill: {
+ enabled: true,
+ weight: 1,
+ concurrency: 1,
+ sweepEnabled: false,
+ sweepKinds: [],
+ sweepChannels: [],
+ allowRedownload: false,
+ },
+};
+
const TWO_WORKERS = {
workers: [
{ id: "gpu", name: "GPU", kind: "local", enabled: true, priority: 0, appId: "whisper-cpp", config: {} },
@@ -530,3 +556,260 @@ test("a link written before layouts existed renders in the original order", asyn
"Needs work",
]);
});
+
+// ---------------------------------------------------------------------------
+// Rows, spans, fill and the dense rail
+// ---------------------------------------------------------------------------
+//
+// The layout model went from N columns of stacked sections to ROWS OF CELLS —
+// which is what makes a full-width section, a rail of totals and a scrolling
+// section possible at all. `l=` stayed an input format and is still what gets
+// emitted for anything plain columns can express, so every assertion above this
+// line is the back-compatibility proof and none of it changed.
+
+test("a g= layout renders as a grid: rows, and a section spanning both tracks", async ({
+ page,
+}) => {
+ await page.setViewportSize({ width: 900, height: 700 });
+ // The Cockpit arrangement, reached by name — which also covers ?preset=
+ // resolving server-side.
+ await page.goto("/widget?preset=cockpit");
+
+ const controls = page.getByRole("region", { name: "Controls" });
+ const sched = page.getByRole("region", { name: "Auto-sync scheduler" });
+ const work = page.getByRole("region", { name: "Needs work" });
+ await expect(work).toBeVisible({ timeout: 10_000 });
+
+ const cb = (await controls.boundingBox())!;
+ const sb = (await sched.boundingBox())!;
+ const wb = (await work.boundingBox())!;
+
+ // Controls spans BOTH tracks: wider than either of the cells below it.
+ expect(cb.width).toBeGreaterThan(sb.width * 1.5);
+ // Three rows, top to bottom...
+ expect(sb.y).toBeGreaterThan(cb.y);
+ // ...and the last row's two cells sit side by side.
+ expect(wb.x).toBeGreaterThan(sb.x);
+});
+
+test("a row-flow cell puts the totals on one line inside one rail", async ({
+ page,
+}) => {
+ await page.setViewportSize({ width: 900, height: 400 });
+ await page.goto(
+ "/widget?jobs=0&workers=0&clean=1&lastsync=1&sched=1&g=lsync.sched.disk.cln*r",
+ );
+
+ const sync = page.getByRole("region", { name: "Last sync" });
+ const sched = page.getByRole("region", { name: "Auto-sync scheduler" });
+ const disk = page.getByRole("region", { name: "Disk space" });
+ const clean = page.getByRole("region", { name: "Cleanable data" });
+ await expect(clean).toBeVisible({ timeout: 10_000 });
+
+ // One line: every total shares a baseline.
+ const boxes = await Promise.all(
+ [sync, sched, disk, clean].map((l) => l.boundingBox()),
+ );
+ for (const b of boxes) expect(Math.abs(b!.y - boxes[0]!.y)).toBeLessThan(4);
+ // ...in order, left to right.
+ expect(boxes[3]!.x).toBeGreaterThan(boxes[0]!.x);
+
+ // ONE container: the rail carries the border, the strips inside carry none.
+ expect(
+ await sync.evaluate((el) => getComputedStyle(el).borderTopWidth),
+ ).toBe("0px");
+
+ // And the copy shortens rather than wrapping.
+ await expect(sync).toContainText("synced");
+ await expect(sync).not.toContainText("Last full sync");
+});
+
+test("a fill row scrolls its sections under their own headings, uncapped", async ({
+ page,
+}) => {
+ await resetData("one-transcribe-channel-with-audio");
+ // Twelve channels: twice the six-row cap the non-scrolling list uses, and
+ // more rows than the short frame below can hold — so there is something to
+ // scroll rather than a box that merely could.
+ for (let i = 0; i < 12; i += 1) {
+ const slug = `bulk-clean-${i}`;
+ await writeChannelConfig(slug);
+ await seedCleanableChannel(slug, 1_048_576 * (i + 1));
+ }
+
+ await page.setViewportSize({ width: 900, height: 300 });
+ await page.goto("/widget?jobs=0&workers=0&cleanlist=1&g=clnl*f");
+
+ const list = page.getByRole("region", { name: "Needs cleaning" });
+ await expect(list).toBeVisible({ timeout: 10_000 });
+
+ // Given the height to scroll in, the cap is lifted: all eight rows are there
+ // and there is no "+N more" control left to defeat the point.
+ const rows = list
+ .getByRole("listitem")
+ .and(page.locator("[aria-label^='needs cleaning ']"));
+ await expect(rows).toHaveCount(12, { timeout: 10_000 });
+ await expect(list.getByRole("button", { name: /more channels/ })).toHaveCount(0);
+
+ // The heading is pinned: scrolling the list does not move it.
+ const heading = list.getByRole("heading", { name: "Needs cleaning" });
+ const before = (await heading.boundingBox())!;
+ const scrolled = await list.evaluate((el) => {
+ const box = el.firstElementChild as HTMLElement;
+ box.scrollTop = 300;
+ return box.scrollTop;
+ });
+ expect(scrolled).toBeGreaterThan(0);
+ const after = (await heading.boundingBox())!;
+ expect(Math.abs(after.y - before.y)).toBeLessThan(2);
+ await expect(heading).toBeInViewport();
+});
+
+test("the board writes rows, fill and spans into the link, and falls back to l=", async ({
+ page,
+}) => {
+ await page.goto("/widget/builder");
+ const url = page.getByLabel("widget URL");
+ await expect(url).toHaveValue(/\/widget$/);
+
+ await page.getByRole("button", { name: "2 columns" }).click();
+ await expect(url).toHaveValue(/[?&]l=disk\.wk\.jobs-(&|$)/);
+
+ // A second row, and something in it.
+ await page.getByRole("button", { name: "Add a row" }).click();
+ await page
+ .getByRole("button", { name: "Move Active jobs to the row below" })
+ .click();
+ await expect(url).toHaveValue(/[?&]g=disk\.wk-_jobs-(&|$)/);
+
+ // The left rail: give the second row the leftover height.
+ await page.getByRole("button", { name: "Row 2 height" }).click();
+ await expect(url).toHaveValue(/[?&]g=disk\.wk-_jobs\*f-(&|$)/);
+
+ // Widening swallows the empty cell beside it.
+ await page.getByRole("button", { name: "Widen row 1 cell 1" }).click();
+ await expect(url).toHaveValue(/[?&]g=disk\.wk\*2_jobs\*f-(&|$)/);
+
+ // ...and undoing all three falls back to a plain columns link, which is what
+ // keeps a link shareable with anything that only ever understood `l=`.
+ await page.getByRole("button", { name: "Narrow row 1 cell 1" }).click();
+ await page.getByRole("button", { name: "Row 2 height" }).click();
+ await page.getByRole("button", { name: "Remove the last row" }).click();
+ await expect(url).toHaveValue(/[?&]l=disk\.wk-jobs(&|$)/);
+ await expect(url).not.toHaveValue(/[?&]g=/);
+});
+
+// ---------------------------------------------------------------------------
+// Presets
+// ---------------------------------------------------------------------------
+
+test("a preset saves, loads and deletes, in the builder and in the widget", async ({
+ page,
+}) => {
+ await page.goto("/widget/builder");
+ const url = page.getByLabel("widget URL");
+
+ // Arrange something worth keeping.
+ await page.getByRole("checkbox", { name: "Workers" }).uncheck();
+ await expect(url).toHaveValue(/\/widget\?workers=0$/);
+
+ page.once("dialog", (d) => d.accept("Corner pin"));
+ await page.getByRole("button", { name: "Save as…" }).click();
+
+ const select = page.getByRole("combobox", { name: "preset" });
+ await expect(select.locator("option", { hasText: "Corner pin" })).toHaveCount(
+ 1,
+ { timeout: 10_000 },
+ );
+
+ // A built-in loads as a whole link, layout included.
+ await select.selectOption({ label: "Glance" });
+ await expect(url).toHaveValue(/[?&]g=lsync\.disk\.cln\.bf\.wk\*r(&|$)/);
+
+ // ...and the saved one comes back exactly as it was saved.
+ await select.selectOption({ label: "Corner pin" });
+ await expect(url).toHaveValue(/\/widget\?workers=0$/);
+
+ // The same preset is there inside the widget's own menu — one store, two
+ // surfaces — and loading it there rewrites the widget's address bar.
+ await page.goto("/widget");
+ await page.getByRole("button", { name: "Settings" }).click();
+ const dialog = page.getByRole("dialog", { name: "Widget settings" });
+ const inWidget = dialog.getByRole("combobox", { name: "preset" });
+ await expect(inWidget.locator("option", { hasText: "Corner pin" })).toHaveCount(
+ 1,
+ { timeout: 10_000 },
+ );
+ await inWidget.selectOption({ label: "Corner pin" });
+ await expect(page).toHaveURL(/[?&]workers=0(&|$)/);
+
+ page.once("dialog", (d) => d.accept());
+ await dialog.getByRole("button", { name: "Delete", exact: true }).click();
+ await expect(inWidget.locator("option", { hasText: "Corner pin" })).toHaveCount(
+ 0,
+ { timeout: 10_000 },
+ );
+});
+
+test("the in-widget menu is the same board, and its moves reach the URL", async ({
+ page,
+}) => {
+ await page.goto("/widget");
+ await page.getByRole("button", { name: "Settings" }).click();
+ const dialog = page.getByRole("dialog", { name: "Widget settings" });
+
+ // The board, not the list form that used to live here: the same move buttons
+ // the builder page has, writing the same param.
+ await dialog.getByRole("button", { name: "Move Active jobs up" }).click();
+ await expect(page).toHaveURL(/[?&]l=disk\.jobs\.wk(&|$)/);
+
+ // ...and the row rail too, which is what a second, narrower form could never
+ // have offered.
+ await dialog.getByRole("button", { name: "Row 1 height" }).click();
+ await expect(page).toHaveURL(/[?&]g=disk\.jobs\.wk\*f(&|$)/);
+});
+
+// ---------------------------------------------------------------------------
+// Backfill in the widget
+// ---------------------------------------------------------------------------
+
+test("backfill=1 alone renders the Backfill strip", async ({ page }) => {
+ // REGRESSION: the sync poll used to be gated on lastSync || scheduler only,
+ // so a widget asking for just the backfill strip fetched nothing and rendered
+ // nothing — with no clue that it was the gate rather than the corpus.
+ await writeSettings({ ...TWO_WORKERS, ...BACKFILL_SETTINGS });
+ await page.goto("/widget?backfill=1");
+ await expect(page.getByRole("region", { name: "Backfill" })).toBeVisible({
+ timeout: 15_000,
+ });
+});
+
+test("the widget's controls hold and release the backfill lane", async ({
+ page,
+}) => {
+ await writeSettings({ ...TWO_WORKERS, ...BACKFILL_SETTINGS });
+
+ // Assert the FIELD, not just the label: the button writes the same
+ // settings.backfill.enabled the Settings checkbox does, and the point of that
+ // choice is that the two cannot drift.
+ const laneEnabled = async () =>
+ (
+ await readJson<{ backfill?: { enabled?: boolean } }>(
+ "test-settings.json",
+ ).catch(() => ({}) as { backfill?: { enabled?: boolean } })
+ ).backfill?.enabled ?? false;
+
+ await page.goto("/widget?controls=1&backfill=1");
+ const pause = page.getByRole("button", { name: "pause backfill" });
+ await expect(pause).toBeEnabled({ timeout: 30_000 });
+ await pause.click();
+
+ await expect.poll(laneEnabled, { timeout: 30_000 }).toBe(false);
+ // The label flips immediately, because the click refetches the payload it
+ // reads rather than waiting out the 15s poll floor.
+ const resume = page.getByRole("button", { name: "resume backfill" });
+ await expect(resume).toBeVisible({ timeout: 15_000 });
+
+ await resume.click();
+ await expect.poll(laneEnabled, { timeout: 30_000 }).toBe(true);
+});
diff --git a/export/CHANGELOG.md b/export/CHANGELOG.md
@@ -1,5 +1,13 @@
# Changelog
+## [0.8.5] - 2026-08-11
+- **MCP: the server no longer remembers which corpus you're reading — because remembering it was silently getting it wrong.** `use_source` switched a mutable "active corpus" and persisted the choice to a state file so it survived reconnects. That was the bug. The server registered here runs `--local …/export/public`, but its state file held `{"activeSpec":{"kind":"remote","url":"https://hasanalyzer.pages.dev"}}` from some earlier session — so **every call since had been reading a different archive, and nothing in any result said so**. There is now no active source and nothing is persisted: **every read tool takes its own `source` handle**, and a call that omits it reads the server's startup corpus. The handle *is* the serialised spec in canonical form — `default`, `local:/dir`, `remote:https://site`, `hub:https://hub`, or `hub:https://hub#alpha,beta` for a subset — not an opaque token, so it survives a restart and a human reading one in a transcript knows exactly what was searched. Shorthands (a bare site or hub URL, probed to tell one from the other; a hub member's siteId or title) normalise to canonical and are echoed back. **Every result now ends with `(corpus: <handle>)`** — errors included, implemented once in the dispatch wrapper so a new tool cannot forget it; it is `corpus:` and not `source:` because `- source:` already means "this video's URL" in the output. `use_source` survives one release as an unadvertised alias that resolves a target and tells you the handle to pass; `reset_source` is gone. Leftover state files are inert and can be deleted. Caching moved with it: a source instance is built once per handle and `listChannels` is memoised per instance, which also fixes a pre-existing cost — a 20-id `get_transcripts` batch against a remote used to fetch `corpus.json` twenty times. See `mcp/src/sourceRegistry.ts` (new; `sourceController.ts` deleted), `mcp/src/{server,source,index}.ts`, `mcp/src/sourceRegistry.test.ts`.
+- **MCP: claiming you covered the corpus is now hard to do by accident.** A sweep returned `total 319; showing 1–200; has_more: yes`, was never paged, and reported **319 videos swept** having seen 200. The paging instruction was already in the sweep prompt and was ignored — so prose is not the enforcement mechanism. Worse, paging was also the *expensive* option: the engine materialises the entire match set and only then slices, so each page re-scanned the whole corpus. New **`enumerate_matches`** returns a query's complete id/title/channel/date worklist plus the batch count in **one scan** — a new tool rather than a flag, because a flag that silently changes the output shape is exactly what gets ignored. If a cap is hit, the **first line** reads `⚠ COVERAGE PARTIAL … this is a SAMPLE, not the full set`, never a quiet footnote. `search_transcripts` now prints `⚠ INCOMPLETE PAGE — N total, showing a–b. Do NOT report a count from this page.` **above** the hits (the old footer stays, so existing consumers keep working). Verified on the real 30-channel corpus: enumerate and search agree exactly at 57, 86 and at the 2000-video cap, where both correctly flag partial coverage.
+- **MCP: `/sweep` and `/ask` stop shredding what you type.** Claude Code parses an MCP prompt's arguments as whitespace-splitting zipped against the declared argument names — not quote-aware, the last argument does *not* absorb the remainder, and tokens past the declared count are **dropped silently**. With nine declared arguments, a real request became `link="This"`, `channel="search"`, `group="deleted"`, `directive="videos."`, `batch_size="Why"` — rendered literally into `` ceil(N / Why) `` — and the entire actual question vanished without a warning. The entry points are now **tools** (`sweep_plan`, `ask_plan`) taking one free-text `request`, called by two thin `.claude/commands/` shims that pass `$ARGUMENTS`, the whole raw string, untokenised. A URL with `?a=b&c=d` and a full sentence of punctuation now arrive intact. Settings are still available as `key=value`, but against a **closed whitelist**: an unrecognised `x=y` **stays in the question** and warns (with a typo hint if it's one edit from a real key) instead of being eaten, and every value is validated — `batch_size` an integer 1–20, `parse_model` a single token, `report` a single `.md` path with no `..` — so a bad one becomes a default plus a `⚠` line, never arithmetic. The MCP `sweep` prompt still serves form-based clients (Claude Desktop, Cursor) through the same parser — but since it also still appears in Claude Code's slash list, where it is unusable, it now **refuses** a word-split request instead of sweeping the wrong thing: it names the arguments that cannot be what they claim to be, replays the words that survived in the order they were typed, and hands back the `/sweep` line to use instead. The plans also **pre-resolve** what they can — the corpus handle, your channel/group tokens validated against the live corpus, the group roster when you gave no scope — turning a three-call preamble into none, and an unresolvable channel now **halts** the plan rather than quietly widening it. See `mcp/src/{promptRequest,instructions}.ts` (new) and their tests.
+- **MCP: three parameters that were declared and silently ignored now work.** `open_link`'s `overrides.query_scope:"posts"` was in the schema, dropped by the parser and excluded by the type, so asking to re-target a link at the social-post corpus quietly searched transcripts instead. `get_transcripts.content_types` was declared and never read, so a batch of post ids came back "not found" — ids now fall through to the post corpus. And a posts search over channels with no posts index reported `total 0; scanned 0 page(s) across 0 channel(s)`, indistinguishable from "searched everything, found nothing"; it now says **"no channel in scope ships a posts index — the post corpus is EMPTY here, not merely unmatched"**, with the posts pass counted separately from the video pass.
+- **MCP: `open_link` does the whole job in one call, and `get_transcripts` can carry several queries.** `open_link` was preview-then-`apply:true`, where apply *switched the global active source* — one extra round trip and the mutation this release exists to remove. It now decodes, resolves the origin to a handle, searches, and returns plan + results + handle together; `dry_run:true` gets the plan alone. `get_transcripts` gains **`queries`** (up to 8): the windows merge in one pass per video and each header reports a **per-query count**, so a term that matched nothing in that video is visible rather than absorbed — which kills the re-read-per-quote pattern.
+- **MCP: on the 2026-07-28 protocol.** The server now speaks MCP revision 2026-07-28 via `@modelcontextprotocol/server@2`'s `serveStdio`, which owns the era decision — `modern` (negotiated by `server/discover`) or `legacy` (the 2025 `initialize` handshake) — and serves both from one definition. Confirmed negotiating `modern` in practice, not just in principle. Cache hints are a construction-time policy (`tools/list`, `prompts/list` and `server/discover` are literal constants with no corpus data, so `public` for an hour; every read result stays uncached) and an invalid one throws at startup rather than on the wire. Because `InMemoryTransport` only ever exercises the 2025 era, a new `protocol.test.ts` spawns the real process over stdio and asserts both eras serve an identical, order-pinned tool list. See `mcp/src/protocol.test.ts`.
+
## [0.8.4] - 2026-08-04
- **A video that has gone missing now says so, and says how confidently.** Availability used to be three unlabelled buckets — available, unlisted, deleted — with no marking on the result cards themselves: a deleted video looked exactly like a live one unless you already suspected something and went hunting in the filters. It is now a **state on every card**: `DELETED`, `PRIVATE`, `MEMBERS`, `UNLISTED`, or `MISSING?`. The last one is new and it is the point of the change. Checking a channel's listing is cheap (one request per channel); confirming *why* an individual video vanished is slow, so on a large channel there is a long window where the archive knows a video has dropped out of its channel but not yet what happened to it. The site used to spend that window insisting the video was fine. It now shows **`MISSING?`** — amber, with the question mark, because it is a suspicion and not a finding — and swaps in the confirmed reason once the per-video check catches up. A video re-checked after the scan that turns out to be present simply loses the flag.
- **The availability filter follows the same shape.** "Available" and "Missing" are now a parent and its five leaves (unconfirmed, deleted, private, members-only, unlisted); ticking the parent takes all five, and it shows a dash when you have only some. Unlisted sits under "missing" because the rule is one sentence — *it left the channel's listing* — even though an unlisted video is still watchable by direct link. **Saved filter profiles and old share links keep working**: an existing link that asked for available/unlisted/deleted still means "all of them" under the new taxonomy rather than silently narrowing.
diff --git a/mcp/README.md b/mcp/README.md
@@ -13,14 +13,16 @@ reads the site's already-published static JSON shards (`corpus.json` +
| Tool | What it does |
|------|--------------|
| `list_channels` | List channels **organized under their channel groups** (name, slug, video count; site in hub mode), with a compact group cheat-sheet (`id · name · N channels`) for scoping. |
-| `search_transcripts` | Search captions for a term/phrase (or regex); returns matching videos with timestamped snippets — **each `[mm:ss]` is a clickable link to that exact moment** (or a compact `[mm:ss\|sec]` with `link_style:"base"`). Alias-aware and **pageable** (`total` + `offset`). Scope by one or more channels (`channel`/`channels`) and/or channel groups (`group`/`groups`). |
-| `get_transcripts` | Batch-read up to 20 videos in one call — bounded, timestamped **excerpt windows** around a query's matches (linked `[mm:ss]`, or compact `link_style:"base"` stamps; `max_lines` caps the excerpt), or full transcripts without a query. `channel`/`channels` hints speed the lookup. |
+| `search_transcripts` | Search captions for a term/phrase (or regex); returns matching videos with timestamped snippets — **each `[mm:ss]` is a clickable link to that exact moment** (or a compact `[mm:ss\|sec]` with `link_style:"base"`). Alias-aware and pageable. A page that isn't the whole match set is flagged **above** the hits. |
+| `enumerate_matches` | A query's **complete** match set as a worklist (id/title/channel/date + batch count) in **one scan**. The tool to use whenever you need to count or cover everything. |
+| `get_transcripts` | Batch-read up to 20 videos in one call — bounded, timestamped **excerpt windows** around one query or up to 8 (`queries`), with per-query counts; or full transcripts without a query. Reads posts too. |
| `get_transcript` | One video's full transcript as clean markdown (metadata + **linked** timestamped captions). |
+| `get_post` / `get_thread` | One archived social post, or its whole thread. Posts have no timeline — cite them with no `@ mm:ss`. |
| `get_video_metadata` | One video's metadata (title, channel, date, duration, description, tags, source URL) without the transcript body. |
-| `open_link` | Paste an archilyzer viewer **share link** to re-run that exact search here (query tree + every filter, at full fidelity). **Previews** a plan by default; **applies** it (switch source + search, linked results) on `apply:true`. Adjust in natural language via `overrides`. |
-| `list_sources` | Show the **active** corpus (label + kind + target) and, with a hub context, its member sites (`siteId · title · url`, marking the current subset) so you can pick one to switch to. |
-| `use_source` | **Switch which corpus is read**, on the fly — a hub member (`site`), a hub subset (`sites`), or an arbitrary `remote` URL / `local` dir / `hub` URL. Persists across reconnects. |
-| `reset_source` | Return to the source the server was started with and clear the persisted selection. |
+| `open_link` | Paste an archilyzer viewer **share link** to re-run that exact search here (query tree + every filter, at full fidelity) — plan, results and corpus handle in **one** call. `dry_run:true` for the plan alone. |
+| `list_sources` | Show the **default** corpus and, with a hub, its member sites as ready-to-paste handles. |
+| `resolve_source` | Turn a URL or site name into the canonical `source` handle and check it can be read. Changes nothing. |
+| `sweep_plan` / `ask_plan` | Turn a plain-English request (plus an optional pasted link) into a resolved, step-by-step plan. What `/sweep` and `/ask` call. |
**Clickable moment links.** Every timestamp the read tools emit is a Markdown link
to the exact second — an **archilyzer viewer** deep link (`…/?v=<slug>&t=<sec>`,
@@ -46,22 +48,29 @@ inline-linked (future work).
Beyond `query`, `regex`, and `limit`:
-- **Scope** — restrict the scan to a subset of the corpus. All scope fields are
+- **Scope** — restrict the scan to a subset of the corpus. Both scope fields are
optional and **additive** (the search runs over the union); with none, it
covers everything.
- - **`channel`** / **`channels`** — one or a list of channel slugs/names.
- - **`group`** / **`groups`** — one or a list of channel groups, matched by
- **group id or display name** (case-insensitive), expanded to the channels in
- that group. e.g. `group="other"` and `group="Extended Universe"` resolve the
- same set. (In hub mode groups are deferred — a group token reports "unknown"
- while channel scoping still works.)
+ - **`channels`** — a list of channel slugs/names.
+ - **`groups`** — a list of channel groups, matched by **group id or display
+ name** (case-insensitive), expanded to the channels in that group. e.g.
+ `groups:["other"]` and `groups:["Extended Universe"]` resolve the same set.
+ (In hub mode groups are deferred — a group token reports "unknown" while
+ channel scoping still works.)
+ - The singular `channel`/`group` are no longer advertised (the engine always
+ unioned them anyway) but are still parsed, so an old habit doesn't break.
The footer names the resolved scope (e.g. *"scope: group Extended Universe (7
channels)"* or *"scope: 6 channels"*) and flags any channel/group token that
matched nothing — so a typo is surfaced, not silently a whole-corpus scan.
-- **`offset`** (default 0) — skip this many matches. The result footer reports the
- full `total` and `has_more`, so you can enumerate a query's *entire* match set:
- page with `offset += limit` until `has_more` is `no`.
+- **`offset`** (default 0) — skip this many matches. A page that is not the whole
+ match set is headed with an unmissable **`⚠ INCOMPLETE PAGE — N total, showing
+ a–b. Do NOT report a count from this page.`** (the footer's `total`/`has_more`
+ are still there for existing consumers). **Do not page this to cover a query
+ — use `enumerate_matches`**: the engine materialises the whole match set and
+ *then* slices, so paging costs one full corpus scan per page while enumerating
+ costs one, total. Paging was the documented advice and it produced a report
+ claiming 319 videos swept after seeing 200.
- **`include_snippets`** (default true) — set `false` for a cheap worklist
(id / title / channel / date / match count, no cue text — the per-video
`- source:` line is dropped too). Ideal for the planning pass of a sweep.
@@ -86,6 +95,14 @@ optional `regex`/`use_aliases`), each transcript is reduced to bounded windows o
timestamped lines around the matches (`before`/`after` seconds, default 30) —
high-signal context for folding a batch into a report. Without a query, each
video's full transcript comes back as markdown. Missing ids are reported inline.
+
+- **`queries`** (max 8, unioned with `query`) — window around several terms in
+ **one** pass. The windows merge per video, and each video's header reports a
+ **per-query count**, so a term that matched nothing in that video is visible
+ rather than absorbed into the merged excerpt. Use this instead of re-reading
+ the same videos once per term.
+- **`content_types`** — `"video"` and/or `"post"` (default both). An id that
+ isn't a video falls through to the post corpus, so a mixed batch works.
`channel`/`channels` are optional owning-channel hints that speed the per-id
lookup when a batch spans several channels (the ids are already scoped by the
search that produced them).
@@ -105,97 +122,120 @@ AND/OR/NOT), and the filter block (`fc` channels, `ft` type, `fa` audience,
`fav` availability, `fdf`/`fdt` upload-date range). `open_link` re-runs that exact
search here — no manual source-switching or query reconstruction, nothing lost.
-It is a **preview → adjust → apply** flow:
-
-- **Preview** (default, `apply:false`) — decode the link and return a *plan*: the
- resolved source (origin, hub vs single-site, auto-probed from `corpus.json`),
- the query tree rendered readably, every active filter, the channel scope
- validated against the live corpus, and anything ignored or warned (the `fk`
- subtitle-track token is vestigial in the composite share model — decoded and
- reported as ignored). **No source switch, no search yet.**
+**One call does the whole job.** `open_link` decodes the link, resolves its
+origin to a corpus handle (hub vs single-site, auto-probed from `corpus.json`),
+validates the channel scope against that live corpus, and returns the plan, the
+first page of **linked** results, and the handle to pass as `source` from then
+on. It switches nothing — the old `apply:true` mutated a global active source,
+which is exactly the footgun this rebuild removed.
+
+- **The plan** names the resolved corpus handle, the query tree rendered
+ readably, every active filter, the validated channel scope, and anything
+ ignored or warned (the `fk` subtitle-track token is vestigial in the composite
+ share model — decoded and reported as ignored).
+- **`dry_run: true`** returns the plan *without* searching, for confirming a
+ link's scope first.
- **Adjust** — re-call with structured `overrides` to honor a natural-language
edit: `clear_availability` (drop the availability filter), `clear_type`,
`clear_age`, `clear_dates`, `clear_filters`, `clear_channels`,
`channels:[…]` (re-scope), `date_from`/`date_to`, or `query`/`regex`/
- `query_scope` (replace the search). The plan updates in place.
-- **Apply** (`apply:true`) — switch the active source to the link's origin (a
- single-site remote, or a hub when the origin federates), run the search at full
- fidelity (the query tree + filters, honoring the chat and availability scopes),
- and return the first page of **linked** results. The applied plan is echoed at
- the top for transparency. Pageable via `limit`/`offset`. Still read-only.
+ `query_scope` (replace the search; `query_scope:"posts"` targets the social-post
+ corpus — declared but silently dropped until now).
+- Pageable via `limit`/`offset`. Read-only throughout.
```
open_link link="https://rekietalyzer.pages.dev/?qt=…&fv=1&fc=Rekieta%20Law&ft=v&fav=a"
-open_link link="…" overrides={ "clear_availability": true } # "drop the availability filter"
-open_link link="…" apply=true # switch source + search
+open_link link="…" dry_run=true # plan only
+open_link link="…" overrides={ "clear_availability": true } # "drop the availability filter"
```
-## The `sweep` prompt
+## `/sweep` and `/ask`
-A first-class slash command that turns **Claude Code itself** into the corpus
-sweep engine — the same "batch matching transcripts into a running report" the
-browser does with a BYO AI key, but driven by your Claude **plan usage** (no API
-key) and with the report written to a file.
-
-Invoke it in Claude Code as `/mcp__<server-name>__sweep`. Arguments: `query`
-(required *unless* a `link` is given), `link?` (an archilyzer viewer share URL to
-seed the sweep from), `channel?`, `channels?` (comma-separated slugs/names),
-`group?` (a channel group id or name), `directive?` (default *"key claims &
-contradictions"*), `batch_size?` (default 8), `parse_model?` (default
-`haiku` — the model requested for the per-batch extractor subagents; they only
-quote verbatim, so the cheapest model wins), `report_path?` (default
-`./sweep-report.md`).
+Two entry points that turn **Claude Code itself** into the corpus sweep engine —
+the same "batch matching transcripts into a running report" the browser does
+with a BYO AI key, but driven by your Claude **plan usage** (no API key), with
+the report written to a file.
```
-/mcp__rekietalyzer__sweep query="k cups" channel="chrissie-mayr"
-/mcp__rekietalyzer__sweep query="k cups" group="other" # a whole group
-/mcp__rekietalyzer__sweep query="k cups" group="Extended Universe" # …by name
-/mcp__rekietalyzer__sweep query="k cups" # pick-first (see below)
-/mcp__rekietalyzer__sweep link="https://…/?qt=…&fv=1&fc=…" # seed from a share link
+/sweep https://hasanalyzer.pages.dev/?qt=…&fav=deleted This search finds deleted
+ videos. Why might have Rekieta privated these? batch_size=12
+/ask what did they actually say about the settlement? channels=rekieta-law
```
-**Seed from a share link (`link=`).** Pass a viewer share URL instead of a
-`query` and the sweep starts by driving `open_link`: it previews the decoded plan
-(source, query tree, filters, scope, ignored bits), lets you confirm or adjust in
-natural language, then applies it — switching source and enumerating the full
-tree+filter match set — before batching. Everything the link encodes is honored.
-
-**Pick-first when no scope is given.** Because a whole-corpus sweep can be a lot
-of work, invoking `sweep` **without** a `channel`/`channels`/`group` makes Claude
-FIRST call `list_channels`, present the groups and their channels, and ask you
-which group(s)/channel(s) to sweep — or to confirm **all** for the whole corpus —
-*before* it enumerates anything. Supplying any scope arg skips the prompt and
-sweeps that scope directly. A group selector accepts either an id or the display
-name (`group="other"` ≡ `group="Extended Universe"`).
-
-Once scoped, the prompt instructs Claude Code to: **search** (aliases
-auto-expand; the footer names the resolved scope) → **enumerate** the full
-worklist by paging with `include_snippets:false` until `has_more` is false →
-**plan** `ceil(N / batch_size)` batches → **per batch, map-reduce** → finish with
-a short summary. The MCP stays read-only; only the report file is written, in
-Claude's working directory.
+Type the whole thing on one line, in your own words, punctuation intact. Paste a
+share link anywhere in it.
+
+**Why a tool and not a prompt argument.** Claude Code parses an MCP *prompt*'s
+arguments as `text.trim().split(/\s+/)` zipped against the declared argument
+names. It is not quote-aware, the last argument does **not** absorb the
+remainder, and tokens past the declared count are **dropped silently**. With the
+nine arguments `sweep` declares, the request above became `link="This"`,
+`channel="search"`, `group="deleted"`, `directive="videos."`, `batch_size="Why"`
+— rendered literally into `` ceil(N / Why) `` — and everything from *"Rekieta"*
+onward vanished with no warning.
+
+So `.claude/commands/sweep.md` and `ask.md` are thin shims that pass
+`$ARGUMENTS` — the **entire raw argument string, untokenised** — to the
+`sweep_plan` / `ask_plan` tools, whose arguments are structured JSON and arrive
+intact. The MCP `sweep` prompt still exists for form-based clients (Claude
+Desktop, Cursor), where each argument gets its own field; it routes through the
+same parser and validators.
+
+**It cannot be used as the shredding trap it used to be.** In Claude Code that
+prompt still appears in the slash list as `/mcp__<server>__sweep`, so it now
+**refuses** rather than sweeping the wrong thing: a typed argument holding prose
+(`link` = `"search"`, `batch_size` = `"might"`) is the tokenizer's fingerprint,
+and the prompt answers with what it detected, the words that survived in the
+order they were typed, and the `/sweep` line to use instead. A form client only
+trips it by genuinely typing a bad value — which deserves the error too. Note
+the tool path stays forgiving (default + `⚠`); only the prompt path is strict.
+
+**Settings.** Append `key=value` for any of `channels=`, `groups=`,
+`batch_size=`, `parse_model=`, `report=`, `directive=`, `source=`,
+`content_types=`, `regex=` (quote a multi-word value). The whitelist is closed:
+an unrecognised `x=y` **stays in the question** and warns rather than being
+eaten, with a typo hint if it is one edit away from a real key. Values are
+validated — `batch_size` an integer 1–20, `parse_model` a single token,
+`report` a single `.md` path with no `..` — and a bad one falls back to the
+default with a `⚠` line at the top of the plan, never into the arithmetic.
+
+**What the plan does.** The tools pre-resolve what they can — canonicalising the
+corpus handle, validating your channel/group tokens against the live corpus,
+inlining the group roster when you gave no scope — so a three-call preamble
+becomes none. Then: **pick the scope** if none was given (never a silent
+whole-corpus sweep) → **`enumerate_matches` once** for the complete worklist →
+**state N and `ceil(N / batch_size)` batches** → **per batch, map-reduce** →
+finish with a summary that says how many of the N were actually read. An
+unresolvable channel halts the plan instead of quietly widening it. The MCP
+stays read-only; only the report file is written.
**Dumb extractors on the cheap model.** Each batch is handed to a **subagent**
(Claude Code's Task tool) spawned as a *verbatim extractor* on the cheapest
-model — the prompt requests `parse_model` (default `haiku`) via the Task tool's
+model — the plan requests `parse_model` (default `haiku`) via the Task tool's
model override, and gracefully spawns on the default when no override exists.
The extractor calls `get_transcripts` with `link_style:"base"` and returns
*only* the directive-relevant lines, **verbatim** — grouped per video under its
`moment_base:` line, in their compact `[mm:ss|sec]` form, under a hard budget of
**≤40 lines (~600 words) per batch**. No analysis, no summarizing: the heavy
-transcript bulk lives and dies inside the cheap subagent, and nothing irrelevant
-is ever reprocessed. The **orchestrator does all the synthesis** — claims,
-contradictions, cross-referencing — expanding each kept citation to a full link
-by appending the seconds to the video's `moment_base`, then discards the
-fragment. Batches are independent, so several can run in parallel. If no
-subagent tool is available, the batch is processed inline (still
-`link_style:"base"`) and the raw excerpt text dropped after folding.
+transcript bulk lives and dies inside the cheap subagent. The **orchestrator
+does all the synthesis** — claims, contradictions, cross-referencing — expanding
+each kept citation to a full link by appending the seconds to the video's
+`moment_base`, then discards the fragment. Every subagent is given the corpus
+handle explicitly, since one that omitted it would read the server default and
+quote the wrong archive. Batches are independent, so several can run in
+parallel.
**Linked citations.** Every source is cited as a clickable
`[title @ mm:ss](<moment url>)` link — built by appending the cited integer
seconds to the video's `moment_base` from the tool output, so a click seeks to
the exact second (see *Compact base links* above; a video with no base is cited
-by its `source:` URL). Note `open_link` results stay inline-linked.
+by its `source:` URL). A post has no timeline, so it is cited
+`[post by <author>, <date>](<source url>)` and never with `@ mm:ss`.
+
+The sweep discipline lives in `src/instructions.ts` — one builder shared by both
+tools and the prompt — rather than duplicated into the markdown command files,
+so it cannot drift out of sync with the tools it names. A test asserts every
+tool the instructions mention actually exists.
## Data source (pick one)
@@ -207,45 +247,74 @@ Resolved from flags or env — precedence hub > remote > local:
| Remote | `--remote <url>` | `TRANSCRIPT_SITE_URL` | One deployed site origin. |
| Local | `--local <dir>` | `TRANSCRIPT_LOCAL_DIR` | A composed public dir on disk (default `./export/public`). |
-## Switching the source on the fly
+## Choosing a corpus per call
The server is added to your MCP client **once**, pointed at a startup source
-(above). But you don't have to edit the config and restart to aim it elsewhere —
-you can switch the **active corpus** from inside a session with three tools, and
-your choice **persists across reconnects**.
-
-- **`list_sources`** — shows the active source (label + kind + target). When
- there's a hub in play (you started against a `--hub`, or switched to one) it
- also lists the hub's member sites as `siteId · title · url`, marking which are
- in the current subset — so you can see what you can pick.
-- **`use_source`** — switch the active source. Provide **exactly one** target:
- - **`site: "<siteId | title>"`** — scope to a single hub member (matched by
- siteId or display title, case-insensitive). This becomes a plain
- single-site source, so it regains **full channel-group + alias** support.
- - **`sites: ["<a>", "<b>"]`** — federate a **subset** of the hub's members.
- Unknown tokens are reported, not silently dropped.
- - **`remote: "<url>"`** — an arbitrary deployed site origin.
- - **`local: "<dir>"`** — a composed public dir on disk.
- - **`hub: "<url>"`** — an arbitrary hub URL to federate over.
-- **`reset_source`** — return to the startup source and clear the persisted
- selection.
-
-For example, connected to the archilyzer hub: `list_sources` to see the members,
-then `use_source site:"jeralyzer"` to work in that one site (with its groups), or
-`use_source sites:["jeralyzer","rekietalyzer"]` to federate just those two.
-
-**Persistence.** The active selection is written to a small state file so a
-reconnect (or a fresh process with the same startup flags) resumes it. The file
-lives under `$XDG_STATE_HOME/yt-dlp-transcript-mcp` (fallback
-`~/.local/state/yt-dlp-transcript-mcp`), overridable with
-`TRANSCRIPT_MCP_STATE_DIR`. It's keyed by the **startup** source, so two
-differently-configured servers (archilyzer / rekietalyzer / a local build) keep
-independent selections and don't clobber each other.
-
-**Still read-only.** `use_source` only changes *which* already-published static
-shards are read — accepting an arbitrary `remote`/`local`/`hub` at call time is
-the same capability the startup flags already grant this locally-run tool.
-Nothing is ever written to any corpus.
+(above) — its **default** corpus. To read something else, pass a `source` handle
+on the call. There is no "active source": nothing is switched, and nothing is
+remembered between calls.
+
+A handle **is** the serialised spec, in a canonical round-trippable form:
+
+| Handle | Means |
+|---|---|
+| `default` | the startup source — the same as omitting `source` |
+| `local:/dir` | a composed public dir on disk |
+| `remote:https://site` | one deployed site origin |
+| `hub:https://hub` | federate every member of a hub |
+| `hub:https://hub#alpha,beta` | a hub **subset** (`#`, so it can never collide with a query string) |
+
+Shorthands are accepted and normalised: a **bare site or hub URL** (classified by
+probing its `corpus.json`), a hub member's **siteId or title**, `site:`/`sites:`,
+and `./dir`. The canonical form is echoed back, so you learn it as you go.
+
+**Every result names the corpus it read**, as a trailing `(corpus: <handle>)` —
+errors included. It is `corpus:` rather than `source:` because `- source:` already
+means "this video's URL" in the output.
+
+```
+list_channels # the default corpus
+list_channels source="remote:https://rekietalyzer.pages.dev"
+search_transcripts query="k cups" source="hub:https://archilyzer.com#jeralyzer,rekietalyzer"
+resolve_source source="jeralyzer" # → remote:https://jeralyzer.pages.dev
+```
+
+**Why it works this way.** The server used to hold a mutable active source and
+persist it to a state file under `$XDG_STATE_HOME/yt-dlp-transcript-mcp`. That
+was silently wrong: a server registered with `--local …/export/public` had
+`{"activeSpec":{"kind":"remote","url":"https://hasanalyzer.pages.dev"}}` on disk
+from some earlier session, so **every call read the wrong archive and no result
+said so**. Protocol revision 2026-07-28 removes protocol-level sessions and
+directs servers needing cross-call state to use "explicit, server-minted handles
+passed as ordinary tool arguments" — which is exactly the fix. Any leftover
+state files are now inert and can be deleted.
+
+`use_source` survives one release as an unadvertised alias that resolves its
+target and tells you the handle to pass; `reset_source` is gone (there is
+nothing to reset).
+
+**Still read-only.** A `source` handle only changes *which* already-published
+static shards are read — the same capability the startup flags already grant
+this locally-run tool. Nothing is ever written to any corpus.
+
+## Protocol
+
+The server speaks MCP revision **2026-07-28** through
+`@modelcontextprotocol/server@2`'s `serveStdio`, which owns the era decision:
+the opening exchange selects `modern` (2026-07-28, negotiated via
+`server/discover`) or `legacy` (the 2025 `initialize` handshake), and one server
+instance is pinned for the connection. Both eras serve the identical tools —
+`src/protocol.test.ts` spawns the real process and asserts it on each.
+
+`resultType` is stamped and stripped by the SDK. Cache hints (`ttlMs` /
+`cacheScope`) are a construction-time policy: `tools/list`, `prompts/list` and
+`server/discover` are literal constants carrying no corpus data, so they are
+`public` for an hour, and every read tool's result stays uncached. An invalid
+hint throws at startup rather than on the wire.
+
+The low-level `Server` is used deliberately (it is marked "advanced use" rather
+than removed): literal JSON Schema tool definitions keep the zod dependency out
+of the package entirely.
## Run it
@@ -273,6 +342,13 @@ claude mcp add rekietalyzer \
-- pnpm -C /ABS/PATH/TO/yt-dlp-transcript-browser --filter yt-dlp-transcript-mcp exec tsx src/index.ts
```
+**The `/sweep` and `/ask` commands.** `.claude/commands/{sweep,ask}.md` in this
+repo call `mcp__archilyzer__sweep_plan` / `mcp__archilyzer__ask_plan` — the tool
+name embeds the MCP server name **as you registered it**, so if you used another
+name (`rekietalyzer` above), change the `mcp__<name>__` prefix in those two
+files to match. `foo:bar` namespacing is plugin-only, so what you type stays
+`/sweep`, not `/archilyzer:sweep`.
+
## Add to any MCP client (mcp.json)
```json
diff --git a/mcp/package.json b/mcp/package.json
@@ -13,9 +13,10 @@
"typecheck": "tsc --noEmit -p tsconfig.json"
},
"dependencies": {
- "@modelcontextprotocol/sdk": "^1.12.0"
+ "@modelcontextprotocol/server": "^2.0.0"
},
"devDependencies": {
+ "@modelcontextprotocol/client": "^2.0.0",
"@types/node": "^20.19.39",
"tsx": "^4.21.0",
"typescript": "^5.6.0"
diff --git a/mcp/src/index.ts b/mcp/src/index.ts
@@ -1,20 +1,30 @@
#!/usr/bin/env tsx
-import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { serveStdio } from "@modelcontextprotocol/server/stdio";
import { resolveSourceSpec } from "./sources";
import { createServer } from "./server";
-import { SourceController } from "./sourceController";
+import { SourceRegistry } from "./sourceRegistry";
-async function main(): Promise<void> {
- const spec = resolveSourceSpec(process.argv.slice(2));
- const controller = new SourceController(spec);
- await controller.init();
- console.error(`[yt-dlp-transcript-mcp] source: ${controller.current.label}`);
- const server = createServer(controller);
- await server.connect(new StdioServerTransport());
+// One registry for the process. It holds no active source and writes nothing:
+// each call resolves its own `source` handle, and a call that omits one reads
+// the startup spec below. The instance cache lives here so per-call selection
+// stays cheap.
+//
+// serveStdio owns the era decision for the connection: the opening exchange
+// selects `modern` (2026-07-28) or `legacy` (the 2025 initialize handshake),
+// then pins ONE instance from this factory for the connection's lifetime. It
+// also builds — and discards — one extra instance for a `server/discover`
+// probe, so expect up to two `era:` lines per client. That log is how we learn
+// which era a given client actually negotiates.
+function main(): void {
+ const registry = new SourceRegistry(resolveSourceSpec(process.argv.slice(2)));
+ console.error(
+ `[yt-dlp-transcript-mcp] default corpus: ${registry.defaultHandle}`,
+ );
+ serveStdio(({ era }) => {
+ console.error(`[yt-dlp-transcript-mcp] serving protocol era: ${era}`);
+ return createServer(registry);
+ });
console.error("[yt-dlp-transcript-mcp] ready on stdio");
}
-main().catch((err) => {
- console.error(err);
- process.exit(1);
-});
+main();
diff --git a/mcp/src/instructions.test.ts b/mcp/src/instructions.test.ts
@@ -0,0 +1,175 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { buildSweepInstructions, buildAskInstructions } from "./instructions";
+import { parsePromptRequest } from "./promptRequest";
+import { TOOLS } from "./server";
+
+const CORPUS = "remote:https://site.example";
+
+function sweep(request: string, ctxExtra = {}): string {
+ return buildSweepInstructions(parsePromptRequest(request), {
+ corpus: CORPUS,
+ ...ctxExtra,
+ });
+}
+function ask(request: string, ctxExtra = {}): string {
+ return buildAskInstructions(parsePromptRequest(request), {
+ corpus: CORPUS,
+ ...ctxExtra,
+ });
+}
+
+// ─── The cheap guard against a whole class of drift ───
+
+test("every tool the instructions name actually exists", () => {
+ const known = new Set(TOOLS.map((t) => t.name));
+ // Every `backticked_snake_case` token in any generated instruction that
+ // looks like a tool name must be a real tool. Renaming a tool without
+ // updating the instructions is otherwise invisible until a sweep fails.
+ const texts = [
+ sweep("coffee"),
+ sweep("coffee channels=chan-a batch_size=4"),
+ sweep("https://site.example/?q=a follow this"),
+ sweep("coffee", { unknownChannels: ["ghost"] }),
+ sweep("coffee", {
+ availableGroups: [{ id: "other", name: "Other", channels: 3 }],
+ }),
+ ask("what did they say"),
+ ask("https://site.example/?q=a what about this"),
+ ];
+ const suspects = new Set<string>();
+ for (const text of texts) {
+ for (const m of text.matchAll(/`([a-z][a-z0-9]*(?:_[a-z0-9]+)+)`/g)) {
+ suspects.add(m[1]);
+ }
+ }
+ // Argument names share the shape, so only judge tokens that name a tool.
+ const NOT_TOOLS = new Set([
+ "batch_size",
+ "content_types",
+ "link_style",
+ "max_pages",
+ "moment_base",
+ "dry_run",
+ "parse_model",
+ "report_path",
+ ]);
+ const named = [...suspects].filter((s) => !NOT_TOOLS.has(s));
+ assert.ok(named.length > 0, "the instructions should name some tools");
+ for (const name of named) {
+ assert.ok(known.has(name), `instructions name a nonexistent tool: ${name}`);
+ }
+});
+
+// ─── The corpus handle is threaded everywhere ───
+
+test("the sweep instructions pin the corpus on every call, subagents included", () => {
+ const text = sweep("coffee channels=chan-a");
+ assert.match(text, /\*\*Corpus: `remote:https:\/\/site\.example`\.\*\*/);
+ assert.match(text, /Pass `source: "remote:https:\/\/site\.example"` on every/);
+ assert.match(text, /Pass the corpus handle into every subagent/);
+ // The literal scope args a call should carry.
+ assert.match(text, /source: "remote:https:\/\/site\.example", channels: \["chan-a"\]/);
+});
+
+test("the ask instructions pin the corpus too", () => {
+ const text = ask("what did they say about coffee");
+ assert.match(text, /Corpus: `remote:https:\/\/site\.example`/);
+ assert.match(text, /what did they say about coffee/);
+});
+
+// ─── Coverage discipline ───
+
+test("the sweep enumerates in one call and never instructs paging", () => {
+ const text = sweep("coffee channels=chan-a");
+ assert.match(text, /enumerate_matches/);
+ assert.match(text, /COVERAGE PARTIAL/);
+ assert.ok(
+ !/offset \+= limit/.test(text),
+ "the paging instruction is gone — it was ignored and is the expensive path",
+ );
+ assert.match(text, /ceil\(N \/ 8\)/);
+});
+
+test("a validated batch size reaches the arithmetic, and a bad one cannot", () => {
+ assert.match(sweep("coffee channels=c batch_size=12"), /ceil\(N \/ 12\)/);
+ const bad = sweep("coffee channels=c batch_size=Why");
+ assert.match(bad, /ceil\(N \/ 8\)/);
+ assert.ok(!bad.includes("ceil(N / Why)"), "the 'Why' bug must be impossible");
+ assert.match(bad, /^⚠ batch_size "Why"/m);
+});
+
+test("warnings render as a leading block", () => {
+ const text = sweep("coffee flavour=vanilla");
+ assert.ok(text.startsWith("⚠ "), text.slice(0, 40));
+ assert.match(text, /not a recognised setting/);
+});
+
+// ─── Scope handling ───
+
+test("an unresolvable channel stops the sweep instead of widening it", () => {
+ const text = sweep("coffee channels=ghost", { unknownChannels: ["ghost"] });
+ assert.match(text, /Fix the scope first/);
+ assert.match(text, /channel "ghost"/);
+ // And it does not go on to instruct an enumeration over everything.
+ assert.ok(!/Enumerate the complete worklist/.test(text));
+});
+
+test("no scope means pick-first, with the roster inlined when known", () => {
+ const withRoster = sweep("coffee", {
+ availableGroups: [
+ { id: "other", name: "Extended Universe", channels: 7 },
+ { id: "core", name: "Core", channels: 2 },
+ ],
+ });
+ assert.match(withRoster, /Choose the scope first/);
+ assert.match(withRoster, /other · Extended Universe · 7 channel\(s\)/);
+ // The roster was pre-resolved, so no round-trip is asked for.
+ assert.ok(!/Call `list_channels`/.test(withRoster));
+
+ const without = sweep("coffee");
+ assert.match(without, /Choose the scope first/);
+ assert.match(without, /Call `list_channels`/);
+ assert.match(without, /confirm \*\*all\*\*/);
+});
+
+test("an explicit scope skips the pick-first step", () => {
+ const text = sweep("coffee groups=other", { knownGroups: ["other"] });
+ assert.ok(!/Choose the scope first/.test(text));
+ assert.match(text, /scoped to groups "other"/);
+});
+
+// ─── Links ───
+
+test("a link seeds a dry-run decode, then a single real call", () => {
+ const text = sweep("https://site.example/?qt=abc&fav=deleted what happened");
+ assert.match(text, /open_link/);
+ assert.match(text, /dry_run: true/);
+ assert.match(text, /https:\/\/site\.example\/\?qt=abc&fav=deleted/);
+ assert.ok(!text.includes("apply:true"));
+});
+
+// ─── Citations ───
+
+test("both builders mandate the same citation forms", () => {
+ for (const text of [sweep("coffee channels=c"), ask("coffee")]) {
+ assert.match(text, /\[title @ mm:ss\]/);
+ assert.match(text, /\[post by <author>, <date>\]/);
+ assert.match(text, /never (take|takes|with) a? ?`@ mm:ss`/);
+ }
+});
+
+test("the ask plan pushes multi-query reads rather than re-reading per term", () => {
+ const text = ask("what did they say about coffee and tea");
+ assert.match(text, /get_transcripts/);
+ assert.match(text, /`queries`/);
+ assert.match(text, /enumerate_matches/);
+});
+
+test("resolved context notes are surfaced", () => {
+ const text = sweep("coffee channels=c", {
+ notes: ["42 channel(s) in remote:https://site.example"],
+ });
+ assert.match(text, /Context already resolved for you/);
+ assert.match(text, /42 channel\(s\)/);
+});
diff --git a/mcp/src/instructions.ts b/mcp/src/instructions.ts
@@ -0,0 +1,304 @@
+import {
+ DEFAULT_DIRECTIVE,
+ DEFAULT_REPORT_PATH,
+ renderWarnings,
+ type PromptRequest,
+} from "./promptRequest";
+
+// ─── The orchestration text a sweep / ask runs on ───
+//
+// The server stays the single source of truth for the sweep discipline —
+// citation format, extractor budget, base-link expansion, the coverage rule —
+// rather than duplicating it into a markdown command file that would drift.
+// All three entry points (the `sweep_plan` and `ask_plan` tools and the MCP
+// `sweep` prompt) render through here.
+
+// Facts the server resolved before writing the instructions, so the model does
+// not spend round-trips rediscovering them. Anything unresolvable is simply
+// absent and the instructions fall back to asking.
+export type PlanContext = {
+ // The canonical corpus handle every call in this plan must pass.
+ corpus: string;
+ // Channel/group tokens validated against the live corpus.
+ knownChannels?: string[];
+ unknownChannels?: string[];
+ knownGroups?: string[];
+ unknownGroups?: string[];
+ // The groups on offer, when the user picked no scope and has to choose.
+ availableGroups?: { id: string; name: string; channels: number }[];
+ // Extra notes to surface (e.g. a decoded link's plan).
+ notes?: string[];
+};
+
+function quoted(xs: string[]): string {
+ return xs.map((x) => `"${x}"`).join(", ");
+}
+
+// The scope arguments every search/enumerate call in the plan should carry,
+// rendered literally so they can be copied.
+function scopeArgs(req: PromptRequest, ctx: PlanContext): string {
+ const parts = [`source: "${ctx.corpus}"`];
+ if (req.channels.length > 0) parts.push(`channels: [${quoted(req.channels)}]`);
+ if (req.groups.length > 0) parts.push(`groups: [${quoted(req.groups)}]`);
+ if (req.contentTypes) parts.push(`content_types: [${quoted(req.contentTypes)}]`);
+ if (req.regex) parts.push(`regex: true`);
+ return parts.join(", ");
+}
+
+function scopeProse(req: PromptRequest): string {
+ const clauses: string[] = [];
+ if (req.channels.length > 0) clauses.push(`channels ${quoted(req.channels)}`);
+ if (req.groups.length > 0) clauses.push(`groups ${quoted(req.groups)}`);
+ return clauses.join(" and ");
+}
+
+// The scope-validation preamble: what the server already checked, and what the
+// model must do about anything that didn't resolve.
+function scopeSteps(
+ req: PromptRequest,
+ ctx: PlanContext,
+): { steps: string[]; halt: boolean } {
+ const steps: string[] = [];
+ const unknown = [
+ ...(ctx.unknownChannels ?? []).map((c) => `channel "${c}"`),
+ ...(ctx.unknownGroups ?? []).map((g) => `group "${g}"`),
+ ];
+ if (unknown.length > 0) {
+ steps.push(
+ `**Fix the scope first.** These did not match anything in this corpus: ` +
+ `${unknown.join(", ")}. Do NOT proceed with a silently-wider scope — ` +
+ `show me \`list_channels\` output and ask which I meant.`,
+ );
+ // Halt here: an instruction list that goes on to enumerate would invite
+ // running the sweep over a silently wider scope than was asked for.
+ return { steps, halt: true };
+ }
+ const hasScope = req.channels.length > 0 || req.groups.length > 0;
+ if (!hasScope && !req.link) {
+ // When the server could pre-resolve the roster, inline it — that turns a
+ // round-trip into a question the model can ask immediately. When it
+ // couldn't, say how to fetch it.
+ const roster =
+ ctx.availableGroups && ctx.availableGroups.length > 0
+ ? `\n\n The groups in this corpus are:\n` +
+ ctx.availableGroups
+ .map((g) => ` - ${g.id} · ${g.name} · ${g.channels} channel(s)`)
+ .join("\n")
+ : ` Call \`list_channels\` (with \`source: "${ctx.corpus}"\`) and ` +
+ `present the groups and their channels.`;
+ steps.push(
+ `**Choose the scope first — do NOT default to the whole corpus.** No ` +
+ `channels or groups were given.${roster.startsWith(" Call") ? roster : ""} ` +
+ `Ask which group(s) or channel(s) to sweep, or to confirm **all** for ` +
+ `the whole corpus. Wait for my choice, then use it as the ` +
+ `\`channels\`/\`groups\` scope in every call below.` +
+ (roster.startsWith(" Call") ? "" : roster),
+ );
+ }
+ return { steps, halt: false };
+}
+
+// The per-batch map-reduce: a cheap extractor subagent quotes verbatim, the
+// orchestrator does every bit of the synthesis. The transcript bulk never
+// enters the orchestrator's context and never costs the big model.
+function batchStep(req: PromptRequest, ctx: PlanContext, reportPath: string): string {
+ return (
+ `**Per batch (map-reduce), for each group of up to ${req.batchSize} ids:**\n` +
+ ` - **Spawn a subagent as a DUMB EXTRACTOR on the cheapest model** — ` +
+ `use the Task tool and request model "${req.parseModel}" (its model ` +
+ `parameter, or a "${req.parseModel}"-backed agent type); if no model ` +
+ `override is available, spawn it anyway on the default. Give it exactly ` +
+ `this job: call \`get_transcripts\` with the batch's ids, ` +
+ `\`source: "${ctx.corpus}"\`, the sweep's \`queries\`, and ` +
+ `\`link_style: "base"\`, then return ONLY the lines relevant to the ` +
+ `directive, VERBATIM — do NOT analyze, summarize, or rephrase anything. ` +
+ `Group the kept lines per video as \`### <title>\` + that video's ` +
+ `\`moment_base:\` line copied exactly (or its \`source:\` line when there ` +
+ `is no moment_base) + the kept lines in their \`[mm:ss|seconds]\` form. ` +
+ `Hard budget: at most 40 lines (~600 words) per batch — if more match, ` +
+ `keep the strongest and end with \`(+N more matching lines)\`. Return ` +
+ `nothing else.\n` +
+ ` - **Pass the corpus handle into every subagent.** Each one must call ` +
+ `with \`source: "${ctx.corpus}"\`. A subagent that omits it reads this ` +
+ `server's default corpus instead, and its quotes would be from the wrong ` +
+ `archive with nothing in the output to show it.\n` +
+ ` - **Merge — you (the orchestrator) do ALL the synthesis.** Cross-` +
+ `reference the returned fragment against the report so far and upsert ` +
+ `findings — claims, and contradictions with earlier claims — into ` +
+ `well-titled \`## sections\` of \`${reportPath}\` (Write/Edit). Cite every ` +
+ `VIDEO finding as **\`[title @ mm:ss](<moment url>)\`**, expanding each ` +
+ `kept \`[mm:ss|seconds]\` stamp by appending the integer after the \`|\` ` +
+ `to that video's moment_base (full link = \`<moment_base><seconds>\`; no ` +
+ `moment_base → link the \`source:\` URL instead). A POST finding has no ` +
+ `timestamp — cite it as **\`[post by <author>, <date>](<source url>)\`**, ` +
+ `never with \`@ mm:ss\`. Then discard the fragment. Batches are ` +
+ `independent, so you may dispatch several subagents in parallel.\n` +
+ ` - **Fallback:** with no Task tool, do the batch inline — call ` +
+ `\`get_transcripts\` with \`link_style: "base"\` yourself, fold the ` +
+ `expanded cited findings into the report, then **drop the raw excerpt ` +
+ `text** before moving on.`
+ );
+}
+
+// The full sweep instructions.
+export function buildSweepInstructions(
+ req: PromptRequest,
+ ctx: PlanContext,
+): string {
+ const reportPath = req.reportPath ?? DEFAULT_REPORT_PATH;
+ const directive = req.directive ?? DEFAULT_DIRECTIVE;
+ const subject = req.query
+ ? `"${req.query}"`
+ : req.link
+ ? "the share link's search"
+ : "the subject below";
+ const args = scopeArgs(req, ctx);
+
+ const scope = scopeSteps(req, ctx);
+ const steps: string[] = [...scope.steps];
+
+ if (!scope.halt && req.link) {
+ steps.push(
+ `**Decode the link.** Call \`open_link\` with link="${req.link}" and ` +
+ `\`dry_run: true\`. It returns the plan: the resolved corpus handle, ` +
+ `the query tree, every active filter, the channel scope validated ` +
+ `against that corpus, and anything ignored. **Show me the plan and ` +
+ `confirm it captures what I want.** If I ask for a change ("drop the ` +
+ `availability filter", "only channel X", "search Y instead"), re-call ` +
+ `with the matching \`overrides\` until it is right. Then call once ` +
+ `more without \`dry_run\` to run it. Use the handle it reports as ` +
+ `\`source\` from then on.`,
+ );
+ }
+
+ if (!scope.halt) {
+ steps.push(
+ `**Enumerate the complete worklist — one call.** Call ` +
+ `\`enumerate_matches\` with ${args}` +
+ (req.query ? `, query: "${req.query}"` : "") +
+ `. It returns EVERY match (id/title/channel/date) in a single scan plus ` +
+ `the batch count. Do NOT page \`search_transcripts\` for this: that is ` +
+ `one full corpus scan per page, and it is how a previous sweep reported ` +
+ `319 videos after seeing 200. If the first line says **COVERAGE ` +
+ `PARTIAL**, the list is a sample — narrow the scope or raise ` +
+ `\`max_pages\`, and if you proceed anyway, say so prominently in the ` +
+ `report.`,
+ );
+
+ steps.push(
+ `**State the plan.** Report N (the enumerated total) and ` +
+ `\`ceil(N / ${req.batchSize})\` batches before you start. The report may ` +
+ `only ever claim the coverage this number justifies: N videos ` +
+ `enumerated, and however many you actually read.`,
+ );
+
+ steps.push(batchStep(req, ctx, reportPath));
+
+ steps.push(
+ `**Finish.** Work to the end of the worklist, then write a summary ` +
+ `section: the scope swept, **how many of the N you actually read**, ` +
+ `headline findings, and any partial-coverage caveat. Tell me the report ` +
+ `path. Keep every citation clickable — \`[title @ mm:ss](url)\` for a ` +
+ `video, \`[post by <author>, <date>](url)\` for a post.`,
+ );
+ }
+
+ const numbered = steps.map((s, i) => `${i + 1}. ${s}`).join("\n\n");
+ const scopeNote = scopeProse(req);
+
+ const head =
+ `Run a **corpus sweep** for ${subject}` +
+ (scopeNote ? `, scoped to ${scopeNote}` : "") +
+ `, extracting **${directive}**, and maintain a running markdown report at ` +
+ `\`${reportPath}\`.\n\n` +
+ `You are the sweep engine — work the whole match set methodically, using ` +
+ `the transcript MCP tools for evidence and your own Write/Edit tools for ` +
+ `the report. The MCP is read-only; never try to change the archive. The ` +
+ `corpus holds video transcripts AND archived social posts.\n\n` +
+ `**Corpus: \`${ctx.corpus}\`.** Pass \`source: "${ctx.corpus}"\` on every ` +
+ `single call, including the ones your subagents make. This server has no ` +
+ `active source — a call that omits \`source\` reads the server default, ` +
+ `which may be a different archive.\n\n` +
+ `**Cite every video finding as a clickable \`[title @ mm:ss](<moment ` +
+ `url>)\` link** (append the cited integer seconds to that video's ` +
+ `\`moment_base\` from the tool output); **cite every post finding as ` +
+ `\`[post by <author>, <date>](<source url>)\`** — posts have no timeline, ` +
+ `so they never take a \`@ mm:ss\`.`;
+
+ const notes =
+ ctx.notes && ctx.notes.length > 0
+ ? `\n\nContext already resolved for you:\n${ctx.notes.map((n) => `- ${n}`).join("\n")}`
+ : "";
+
+ return (
+ renderWarnings(req.warnings) +
+ `${head}${notes}\n\nFollow these steps:\n\n${numbered}`
+ );
+}
+
+// The ask variant: the same evidence discipline, but answering a question in
+// the conversation rather than maintaining a report file. Kept deliberately
+// close to the sweep so citations look identical either way.
+export function buildAskInstructions(
+ req: PromptRequest,
+ ctx: PlanContext,
+): string {
+ const question = req.query || "the question below";
+ const args = scopeArgs(req, ctx);
+ const askScope = scopeSteps(req, ctx);
+ const steps: string[] = [...askScope.steps];
+
+ if (!askScope.halt && req.link) {
+ steps.push(
+ `**Decode the link.** Call \`open_link\` with link="${req.link}" — one ` +
+ `call returns the plan and the first page of results, plus the corpus ` +
+ `handle to use from then on.`,
+ );
+ }
+
+ if (!askScope.halt) {
+ steps.push(
+ `**Find the evidence.** Search with \`search_transcripts\` (${args}) for ` +
+ `the terms the question implies — try more than one phrasing; the ` +
+ `corpus is ASR text and the curated aliases only cover known ` +
+ `mis-transcriptions. If you need to know HOW MANY, or to cover ` +
+ `everything, use \`enumerate_matches\` instead: a search page is a ` +
+ `slice, and its \`⚠ INCOMPLETE PAGE\` banner means exactly that.`,
+ );
+
+ steps.push(
+ `**Read before concluding.** Pull the surrounding context with ` +
+ `\`get_transcripts\` (${args}) — pass all the terms at once in ` +
+ `\`queries\` rather than re-reading the same videos per term. A snippet ` +
+ `is not evidence of what was meant; the window around it is.`,
+ );
+
+ steps.push(
+ `**Answer with citations.** Answer ${question} directly, and cite every ` +
+ `claim: \`[title @ mm:ss](<moment url>)\` for a video (append the ` +
+ `integer seconds to that video's \`moment_base\`), ` +
+ `\`[post by <author>, <date>](<source url>)\` for a post — a post has ` +
+ `no timeline, so it never takes a \`@ mm:ss\`. Say plainly what the ` +
+ `corpus does NOT show — an absence of matches is a finding, not a gap ` +
+ `to paper over.`,
+ );
+ }
+
+ const numbered = steps.map((s, i) => `${i + 1}. ${s}`).join("\n\n");
+ const head =
+ `Answer this question from the transcript archive: **${question}**\n\n` +
+ `**Corpus: \`${ctx.corpus}\`.** Pass \`source: "${ctx.corpus}"\` on every ` +
+ `call — this server has no active source, and a call that omits it reads ` +
+ `the server default. The MCP is read-only. The corpus holds video ` +
+ `transcripts AND archived social posts.`;
+
+ const notes =
+ ctx.notes && ctx.notes.length > 0
+ ? `\n\nContext already resolved for you:\n${ctx.notes.map((n) => `- ${n}`).join("\n")}`
+ : "";
+
+ return (
+ renderWarnings(req.warnings) +
+ `${head}${notes}\n\nFollow these steps:\n\n${numbered}`
+ );
+}
diff --git a/mcp/src/promptRequest.test.ts b/mcp/src/promptRequest.test.ts
@@ -0,0 +1,327 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import {
+ parsePromptRequest,
+ requestFromArguments,
+ renderWarnings,
+ validateSweepArguments,
+ DEFAULT_BATCH_SIZE,
+ DEFAULT_PARSE_MODEL,
+} from "./promptRequest";
+
+// The parser exists because Claude Code's slash-command tokenizer shredded a
+// real request into link="This" channel="search" group="deleted"
+// directive="videos." batch_size="Why" and dropped the rest. Every case below
+// is either that failure or a neighbour of it.
+
+// ─── Link extraction ───
+
+test("a bare URL is extracted and the rest stays prose", () => {
+ const r = parsePromptRequest("https://site.example/?q=x what did they say");
+ assert.equal(r.link, "https://site.example/?q=x");
+ assert.equal(r.query, "what did they say");
+});
+
+test("a URL's query string survives intact — & and = are not settings", () => {
+ const r = parsePromptRequest(
+ "https://site.example/?qt=abc&fc=chan-a&fav=deleted why though",
+ );
+ assert.equal(r.link, "https://site.example/?qt=abc&fc=chan-a&fav=deleted");
+ assert.equal(r.query, "why though");
+ assert.deepEqual(r.warnings, []);
+});
+
+test("only the FIRST url becomes the link; a second stays in the prose", () => {
+ const r = parsePromptRequest(
+ "https://a.example/one compare with https://b.example/two",
+ );
+ assert.equal(r.link, "https://a.example/one");
+ assert.match(r.query, /https:\/\/b\.example\/two/);
+});
+
+test("an angle-bracketed url is unwrapped", () => {
+ const r = parsePromptRequest("<https://site.example/x> and then some");
+ assert.equal(r.link, "https://site.example/x");
+ assert.ok(!r.query.includes("<"), r.query);
+ assert.ok(!r.query.includes(">"), r.query);
+});
+
+test("a markdown link is unwrapped, label and all", () => {
+ const r = parsePromptRequest("[Search results](https://site.example/?q=a) go on");
+ assert.equal(r.link, "https://site.example/?q=a");
+ assert.equal(r.query, "go on");
+});
+
+test("a quoted url is unwrapped", () => {
+ const r = parsePromptRequest('"https://site.example/x" please');
+ assert.equal(r.link, "https://site.example/x");
+ assert.equal(r.query, "please");
+});
+
+test("a trailing full stop is not part of the url", () => {
+ const r = parsePromptRequest("see https://site.example/x. Then explain.");
+ assert.equal(r.link, "https://site.example/x");
+});
+
+test("a trailing comma, colon and semicolon are not part of the url", () => {
+ for (const p of [",", ":", ";", "!", "?"]) {
+ const r = parsePromptRequest(`see https://site.example/x${p} more`);
+ assert.equal(r.link, "https://site.example/x", `punctuation ${p}`);
+ }
+});
+
+test("an unbalanced closing paren is dropped but a balanced one is kept", () => {
+ const a = parsePromptRequest("(see https://site.example/x) ok");
+ assert.equal(a.link, "https://site.example/x");
+ const b = parsePromptRequest("https://site.example/x(y) ok");
+ assert.equal(b.link, "https://site.example/x(y)");
+});
+
+test("no url at all is fine", () => {
+ const r = parsePromptRequest("just a question about coffee");
+ assert.equal(r.link, undefined);
+ assert.equal(r.query, "just a question about coffee");
+});
+
+// ─── The whole-request regression ───
+
+test("the shredded Rekieta line survives whole", () => {
+ const input =
+ "https://hasanalyzer.pages.dev/?qt=eyJhIjoxfQ&fav=deleted " +
+ "This search finds deleted videos. Why might have Rekieta privated " +
+ "these? Look for context around each one. batch_size=12";
+ const r = parsePromptRequest(input);
+ assert.equal(r.link, "https://hasanalyzer.pages.dev/?qt=eyJhIjoxfQ&fav=deleted");
+ assert.equal(r.batchSize, 12);
+ // Every word of the question is present, in order, punctuation intact.
+ assert.equal(
+ r.query,
+ "This search finds deleted videos. Why might have Rekieta privated " +
+ "these? Look for context around each one.",
+ );
+ assert.deepEqual(r.warnings, []);
+});
+
+// ─── The key=value whitelist ───
+
+test("recognised settings are parsed off the prose", () => {
+ const r = parsePromptRequest(
+ "coffee talk channels=chan-a,chan-b groups=other batch_size=5 " +
+ "parse_model=sonnet report=./out.md source=remote:https://x.example",
+ );
+ assert.equal(r.query, "coffee talk");
+ assert.deepEqual(r.channels, ["chan-a", "chan-b"]);
+ assert.deepEqual(r.groups, ["other"]);
+ assert.equal(r.batchSize, 5);
+ assert.equal(r.parseModel, "sonnet");
+ assert.equal(r.reportPath, "./out.md");
+ assert.equal(r.source, "remote:https://x.example");
+});
+
+test("aliases map onto the canonical keys", () => {
+ const r = parsePromptRequest("x channel=chan-a group=other model=opus out=./r.md");
+ assert.deepEqual(r.channels, ["chan-a"]);
+ assert.deepEqual(r.groups, ["other"]);
+ assert.equal(r.parseModel, "opus");
+ assert.equal(r.reportPath, "./r.md");
+});
+
+test("an unknown key=value STAYS in the prose and warns", () => {
+ const r = parsePromptRequest("find x flavour=vanilla please");
+ assert.match(r.query, /flavour=vanilla/);
+ assert.match(r.warnings.join(" "), /"flavour=" is not a recognised setting/);
+});
+
+test("a near-miss key gets a typo hint and still stays in the prose", () => {
+ const r = parsePromptRequest("find x chanels=chan-a");
+ assert.match(r.query, /chanels=chan-a/);
+ assert.match(r.warnings.join(" "), /did you mean channels=\?/);
+});
+
+test("prose that merely contains '=' is never treated as a setting", () => {
+ const r = parsePromptRequest("solve x=y+2 and 3==3 for me");
+ assert.match(r.query, /x=y\+2/);
+ assert.match(r.query, /3==3/);
+});
+
+test("a quoted multi-word value survives as one value", () => {
+ const r = parsePromptRequest(
+ 'sweep it directive="key claims & contradictions" now',
+ );
+ assert.equal(r.directive, "key claims & contradictions");
+ assert.equal(r.query, "sweep it now");
+});
+
+test("a repeated key takes the last value and warns", () => {
+ const r = parsePromptRequest("x batch_size=4 batch_size=9");
+ assert.equal(r.batchSize, 9);
+ assert.match(r.warnings.join(" "), /batch_size was given more than once/);
+});
+
+// ─── Validation: the failures that used to render into the output ───
+
+test("batch_size=Why falls back to the default with a warning", () => {
+ const r = parsePromptRequest("x batch_size=Why");
+ assert.equal(r.batchSize, DEFAULT_BATCH_SIZE);
+ assert.match(r.warnings.join(" "), /batch_size "Why" is not an integer/);
+});
+
+test("batch_size out of range or fractional is rejected", () => {
+ for (const bad of ["0", "21", "2.5", "-3"]) {
+ const r = parsePromptRequest(`x batch_size=${bad}`);
+ assert.equal(r.batchSize, DEFAULT_BATCH_SIZE, `batch_size=${bad}`);
+ assert.ok(r.warnings.length > 0);
+ }
+});
+
+test("batch_size at the bounds is accepted", () => {
+ assert.equal(parsePromptRequest("x batch_size=1").batchSize, 1);
+ assert.equal(parsePromptRequest("x batch_size=20").batchSize, 20);
+});
+
+test("a multi-word parse_model is rejected", () => {
+ const r = parsePromptRequest('x parse_model="two words"');
+ assert.equal(r.parseModel, DEFAULT_PARSE_MODEL);
+ assert.match(r.warnings.join(" "), /not a single token/);
+});
+
+test("a report path escaping the cwd is rejected", () => {
+ const r = parsePromptRequest("x report=../secrets.md");
+ assert.equal(r.reportPath, undefined);
+ assert.match(r.warnings.join(" "), /escapes the working directory/);
+});
+
+test("a report path that is not .md is rejected", () => {
+ const r = parsePromptRequest("x report=./out.txt");
+ assert.equal(r.reportPath, undefined);
+ assert.match(r.warnings.join(" "), /not a .md file/);
+});
+
+test("a multi-word report path is rejected", () => {
+ const r = parsePromptRequest('x report="my report.md"');
+ assert.equal(r.reportPath, undefined);
+ assert.match(r.warnings.join(" "), /not a single token/);
+});
+
+test("content_types keeps the valid values and warns about the rest", () => {
+ const r = parsePromptRequest("x content_types=post,audio");
+ assert.deepEqual(r.contentTypes, ["post"]);
+ assert.match(r.warnings.join(" "), /content_types value\(s\) ignored/);
+});
+
+test("regex=true is honoured; anything else is false", () => {
+ assert.equal(parsePromptRequest("x regex=true").regex, true);
+ assert.equal(parsePromptRequest("x regex=yes").regex, false);
+ assert.equal(parsePromptRequest("x").regex, false);
+});
+
+test("empty input warns rather than producing a silent no-op plan", () => {
+ const r = parsePromptRequest("");
+ assert.equal(r.query, "");
+ assert.equal(r.link, undefined);
+ assert.match(r.warnings.join(" "), /no query and no link/);
+});
+
+test("a link with no question is not a warning — the link supplies the query", () => {
+ const r = parsePromptRequest("https://site.example/?q=a");
+ assert.deepEqual(r.warnings, []);
+});
+
+// ─── The MCP prompt form shares the validators ───
+
+test("prompt arguments route through the same validation", () => {
+ const r = requestFromArguments({
+ query: "k cups",
+ channels: "chan-a, chan-b",
+ group: "other",
+ batch_size: "Why",
+ report_path: "../x.md",
+ });
+ assert.equal(r.query, "k cups");
+ assert.deepEqual(r.channels, ["chan-a", "chan-b"]);
+ assert.deepEqual(r.groups, ["other"]);
+ assert.equal(r.batchSize, DEFAULT_BATCH_SIZE);
+ assert.equal(r.reportPath, undefined);
+ assert.equal(r.warnings.length, 2);
+});
+
+test("prompt arguments accept a link and clean it", () => {
+ const r = requestFromArguments({ link: "<https://site.example/x>." });
+ assert.equal(r.link, "https://site.example/x");
+});
+
+test("a non-url link argument is reported, not passed through", () => {
+ const r = requestFromArguments({ link: "not-a-url", query: "x" });
+ assert.equal(r.link, undefined);
+ assert.match(r.warnings.join(" "), /is not an http\(s\) URL/);
+});
+
+// ─── Rendering ───
+
+test("warnings render as a leading block, and nothing when clean", () => {
+ assert.equal(renderWarnings([]), "");
+ const out = renderWarnings(["a", "b"]);
+ assert.ok(out.startsWith("⚠ a\n⚠ b"));
+ assert.ok(out.endsWith("\n\n"));
+});
+
+// ─── The prompt form refuses a shredded request ───
+
+test("the exact Claude Code shredding of a real request is detected", () => {
+ // What `text.trim().split(/\s+/)` + zipObject(declaredArgs, tokens) does to
+ // "This search finds deleted videos. Why might have Rekieta privated these?
+ // Look for context around each one." — nine words kept, the rest dropped.
+ const shredded = {
+ query: "This",
+ link: "search",
+ channel: "finds",
+ channels: "deleted",
+ group: "videos.",
+ directive: "Why",
+ batch_size: "might",
+ parse_model: "have",
+ report_path: "Rekieta",
+ };
+ const { problems, reassembled } = validateSweepArguments(shredded);
+ const named = problems.map((p) => p.arg).sort();
+ // "have" in parse_model is not flagged — it has the shape of a model name.
+ // Three independent signals is already conclusive; the check only has to
+ // fire, not catch every slot.
+ assert.deepEqual(named, ["batch_size", "link", "report_path"]);
+ // The surviving words come back in the order they were typed, so the user
+ // can recognise their own sentence and see where it was cut.
+ assert.equal(
+ reassembled,
+ "This search finds deleted videos. Why might have Rekieta",
+ );
+});
+
+test("a legitimate form-client invocation trips nothing", () => {
+ for (const args of [
+ { query: "k cups" },
+ { query: "k cups", group: "other" },
+ { query: "k cups", channels: "chan-a,chan-b", batch_size: "12" },
+ {
+ query: "deleted videos",
+ link: "https://site.example/?qt=abc&fav=deleted",
+ parse_model: "haiku",
+ report_path: "./out.md",
+ directive: "key claims & contradictions",
+ },
+ ]) {
+ assert.deepEqual(validateSweepArguments(args).problems, [], JSON.stringify(args));
+ }
+});
+
+test("a genuinely bad value is refused even from a form client", () => {
+ // The prompt path is strict where the tool path is forgiving: this is the
+ // value that used to render into `ceil(N / Why)`.
+ assert.deepEqual(
+ validateSweepArguments({ query: "k cups", batch_size: "Why" }).problems.map((p) => p.arg),
+ ["batch_size"],
+ );
+ assert.deepEqual(
+ validateSweepArguments({ query: "x", report_path: "../secrets.md" }).problems.map((p) => p.arg),
+ ["report_path"],
+ );
+});
diff --git a/mcp/src/promptRequest.ts b/mcp/src/promptRequest.ts
@@ -0,0 +1,476 @@
+// ─── Parsing a pasted request into a sweep/ask plan ───
+//
+// Claude Code tokenises an MCP prompt's arguments as
+// `text.trim().split(/\s+/)` zipped against the declared argument names. It is
+// not quote-aware, the last argument does not absorb the remainder, and tokens
+// past the declared count are dropped in silence. With nine declared arguments
+// a real request became link="This" channel="search" group="deleted"
+// directive="videos." batch_size="Why" — and everything from the actual
+// question onward vanished.
+//
+// So the entry point for Claude Code is a TOOL (structured JSON arguments,
+// which arrive intact) taking one free-text `request`, and this module is the
+// parser behind it. It is pure and exhaustively tested; the MCP prompt form
+// routes through the same validators so a form-based client (Claude Desktop,
+// Cursor) gets a named error instead of mangled arithmetic.
+//
+// The rules that matter:
+// - a key=value pair is honoured ONLY for a closed whitelist, so an unknown
+// `x=y` stays in the prose instead of being silently eaten;
+// - a near-miss key gets a typo hint and still stays in the prose;
+// - every ignored, corrected, or defaulted input lands in `warnings`, which
+// the caller renders as a ⚠ block at the top of its output.
+
+export type PromptRequest = {
+ // The first http(s) URL in the input, cleaned of wrappers and trailing
+ // punctuation. A share link seeds the search; anything else is just prose.
+ link?: string;
+ // Everything that was not a recognised key=value pair or the link: the
+ // question, in the user's own words, punctuation intact.
+ query: string;
+ channels: string[];
+ groups: string[];
+ batchSize: number;
+ parseModel: string;
+ reportPath?: string;
+ directive?: string;
+ source?: string;
+ contentTypes?: ("video" | "post")[];
+ regex: boolean;
+ warnings: string[];
+};
+
+export const DEFAULT_BATCH_SIZE = 8;
+export const DEFAULT_PARSE_MODEL = "haiku";
+export const DEFAULT_DIRECTIVE = "key claims & contradictions";
+export const DEFAULT_REPORT_PATH = "./sweep-report.md";
+
+// The closed whitelist: canonical key → accepted aliases. Anything outside it
+// is prose, never a parameter.
+const KEYS: Record<string, string[]> = {
+ channels: ["channel", "chan", "chans"],
+ groups: ["group"],
+ batch_size: ["batch", "batchsize", "batch_sz"],
+ parse_model: ["model", "parsemodel", "extractor_model"],
+ report: ["report_path", "reportpath", "path", "out", "output"],
+ directive: ["extract", "extracting"],
+ source: ["corpus"],
+ content_types: ["content_type", "types", "type"],
+ regex: ["re"],
+};
+
+const CANONICAL = new Map<string, string>();
+for (const [canon, aliases] of Object.entries(KEYS)) {
+ CANONICAL.set(canon, canon);
+ for (const a of aliases) CANONICAL.set(a, canon);
+}
+
+// Edit distance, bounded: we only care whether it is ≤ 1.
+function withinOneEdit(a: string, b: string): boolean {
+ if (a === b) return true;
+ const [short, long] = a.length <= b.length ? [a, b] : [b, a];
+ if (long.length - short.length > 1) return false;
+ let i = 0;
+ let j = 0;
+ let edits = 0;
+ while (i < short.length && j < long.length) {
+ if (short[i] === long[j]) {
+ i++;
+ j++;
+ continue;
+ }
+ if (++edits > 1) return false;
+ if (short.length === long.length) i++;
+ j++;
+ }
+ return edits + (long.length - j) + (short.length - i) <= 1;
+}
+
+function typoHint(key: string): string | undefined {
+ for (const known of CANONICAL.keys()) {
+ if (withinOneEdit(key.toLowerCase(), known)) return known;
+ }
+ return undefined;
+}
+
+function countChar(s: string, c: string): number {
+ let n = 0;
+ for (const x of s) if (x === c) n++;
+ return n;
+}
+
+// Pull the first http(s) URL out of the text, undoing the ways a URL gets
+// wrapped when a human pastes it: <url>, "url", [label](url), and a trailing
+// sentence mark. A closing bracket is only stripped when it does not balance
+// one inside the URL, so a URL that legitimately contains brackets survives.
+function extractLink(text: string): { link?: string; rest: string } {
+ // A URL that is the VALUE of a recognised setting (source=remote:https://…)
+ // is not the request's link — skip to the next candidate.
+ const re = /https?:\/\/\S+/gi;
+ let m: RegExpExecArray | null = null;
+ for (let hit = re.exec(text); hit; hit = re.exec(text)) {
+ const lineStart = text.lastIndexOf(" ", hit.index - 1) + 1;
+ const prefix = text.slice(lineStart, hit.index).replace(/^["']|["']$/g, "");
+ const kv = /^([A-Za-z][A-Za-z0-9_]*)=\S*$/.exec(prefix);
+ if (kv && CANONICAL.has(kv[1].toLowerCase())) continue;
+ m = hit;
+ break;
+ }
+ if (!m) return { rest: text };
+
+ const start = m.index;
+ let cand = m[0];
+ const before = start > 0 ? text[start - 1] : "";
+
+ for (let guard = 0; guard < 50; guard++) {
+ const last = cand[cand.length - 1];
+ if (!last) break;
+ if (".,;:!?".includes(last)) {
+ cand = cand.slice(0, -1);
+ continue;
+ }
+ if (last === ">" && before === "<") {
+ cand = cand.slice(0, -1);
+ continue;
+ }
+ if ((last === '"' || last === "'") && (before === last || countChar(cand, last) % 2 === 1)) {
+ cand = cand.slice(0, -1);
+ continue;
+ }
+ const opener = last === ")" ? "(" : last === "]" ? "[" : last === "}" ? "{" : "";
+ if (opener && countChar(cand, opener) < countChar(cand, last)) {
+ cand = cand.slice(0, -1);
+ continue;
+ }
+ break;
+ }
+
+ // Remove the whole markdown wrapper when the URL sat inside one, so the
+ // label doesn't survive as a stray `[Search results](` in the prose.
+ let from = start;
+ let to = start + m[0].length;
+ if (before === "(") {
+ const open = text.lastIndexOf("[", start);
+ const close = text.indexOf(")", to - 1);
+ if (open !== -1 && close !== -1 && text[start - 1] === "(" && text[open] === "[") {
+ from = open;
+ to = close + 1;
+ }
+ } else if (before === "<") {
+ from = start - 1;
+ }
+ const rest = `${text.slice(0, from)} ${text.slice(to)}`;
+ return { link: cand, rest };
+}
+
+// Whitespace-split, but keep a quoted run together and drop the quotes, so
+// directive="key claims & contradictions" survives as ONE token.
+function tokenize(text: string): string[] {
+ const out: string[] = [];
+ let cur = "";
+ let quote: string | null = null;
+ let sawQuote = false;
+ const push = (): void => {
+ if (cur !== "" || sawQuote) out.push(cur);
+ cur = "";
+ sawQuote = false;
+ };
+ for (const ch of text) {
+ if (quote) {
+ if (ch === quote) {
+ quote = null;
+ continue;
+ }
+ cur += ch;
+ continue;
+ }
+ if (ch === '"' || ch === "'") {
+ quote = ch;
+ sawQuote = true;
+ continue;
+ }
+ if (/\s/.test(ch)) {
+ push();
+ continue;
+ }
+ cur += ch;
+ }
+ push();
+ return out.filter((t) => t !== "");
+}
+
+function splitList(v: string): string[] {
+ return v
+ .split(",")
+ .map((x) => x.trim())
+ .filter((x) => x !== "");
+}
+
+// ─── Validators. Each one either accepts, or falls back and says why. ───
+
+function validBatchSize(raw: string, warnings: string[]): number {
+ const n = Number(raw);
+ if (!Number.isInteger(n) || n < 1 || n > 20) {
+ warnings.push(
+ `batch_size "${raw}" is not an integer between 1 and 20 — using ` +
+ `${DEFAULT_BATCH_SIZE}.`,
+ );
+ return DEFAULT_BATCH_SIZE;
+ }
+ return n;
+}
+
+function validParseModel(raw: string, warnings: string[]): string {
+ if (raw === "" || /\s/.test(raw)) {
+ warnings.push(
+ `parse_model "${raw}" is not a single token — using ` +
+ `"${DEFAULT_PARSE_MODEL}".`,
+ );
+ return DEFAULT_PARSE_MODEL;
+ }
+ return raw;
+}
+
+function validReportPath(raw: string, warnings: string[]): string | undefined {
+ if (raw === "" || /\s/.test(raw)) {
+ warnings.push(`report path "${raw}" is not a single token — ignored.`);
+ return undefined;
+ }
+ if (raw.includes("..")) {
+ warnings.push(`report path "${raw}" escapes the working directory — ignored.`);
+ return undefined;
+ }
+ if (!raw.toLowerCase().endsWith(".md")) {
+ warnings.push(`report path "${raw}" is not a .md file — ignored.`);
+ return undefined;
+ }
+ return raw;
+}
+
+function validContentTypes(
+ raw: string,
+ warnings: string[],
+): ("video" | "post")[] | undefined {
+ const wanted = splitList(raw.toLowerCase());
+ const out = wanted.filter((t): t is "video" | "post" => t === "video" || t === "post");
+ const bad = wanted.filter((t) => t !== "video" && t !== "post");
+ if (bad.length > 0) {
+ warnings.push(
+ `content_types value(s) ignored (expected 'video' and/or 'post'): ${bad.join(", ")}.`,
+ );
+ }
+ return out.length > 0 ? out : undefined;
+}
+
+// Parse one free-text request. Never throws: a bad value becomes a default
+// plus a warning, because failing a whole sweep over a typo'd batch size is
+// worse than running it at 8 and saying so.
+export function parsePromptRequest(input: string): PromptRequest {
+ const warnings: string[] = [];
+ const text = (input ?? "").trim();
+
+ const { link, rest } = extractLink(text);
+ const tokens = tokenize(rest);
+
+ const seen = new Map<string, string>();
+ const prose: string[] = [];
+
+ for (const tok of tokens) {
+ const eq = tok.indexOf("=");
+ if (eq <= 0) {
+ prose.push(tok);
+ continue;
+ }
+ const rawKey = tok.slice(0, eq);
+ const value = tok.slice(eq + 1);
+ // Only a plausible key shape is even considered — this keeps prose like
+ // "x=y+2" or "a==b" out of the parameter path entirely.
+ if (!/^[A-Za-z][A-Za-z0-9_]*$/.test(rawKey)) {
+ prose.push(tok);
+ continue;
+ }
+ const canon = CANONICAL.get(rawKey.toLowerCase());
+ if (!canon) {
+ const hint = typoHint(rawKey);
+ warnings.push(
+ `"${rawKey}=" is not a recognised setting, so it was left in the ` +
+ `question text` +
+ (hint ? ` — did you mean ${hint}=?` : "") +
+ `.`,
+ );
+ prose.push(tok);
+ continue;
+ }
+ if (seen.has(canon)) {
+ warnings.push(
+ `${canon} was given more than once — using the last value ("${value}").`,
+ );
+ }
+ seen.set(canon, value);
+ }
+
+ const req: PromptRequest = {
+ ...(link ? { link } : {}),
+ query: prose.join(" ").replace(/\s+/g, " ").trim(),
+ channels: seen.has("channels") ? splitList(seen.get("channels")!) : [],
+ groups: seen.has("groups") ? splitList(seen.get("groups")!) : [],
+ batchSize: seen.has("batch_size")
+ ? validBatchSize(seen.get("batch_size")!, warnings)
+ : DEFAULT_BATCH_SIZE,
+ parseModel: seen.has("parse_model")
+ ? validParseModel(seen.get("parse_model")!, warnings)
+ : DEFAULT_PARSE_MODEL,
+ regex: seen.get("regex") === "true" || seen.get("regex") === "1",
+ warnings,
+ };
+
+ if (seen.has("report")) {
+ const p = validReportPath(seen.get("report")!, warnings);
+ if (p) req.reportPath = p;
+ }
+ if (seen.has("directive")) {
+ const d = seen.get("directive")!.trim();
+ if (d !== "") req.directive = d;
+ }
+ if (seen.has("source")) {
+ const s = seen.get("source")!.trim();
+ if (s !== "") req.source = s;
+ }
+ if (seen.has("content_types")) {
+ const t = validContentTypes(seen.get("content_types")!, warnings);
+ if (t) req.contentTypes = t;
+ }
+
+ if (!req.link && req.query === "") {
+ warnings.push(
+ "no query and no link — say what to sweep for, or paste a share link.",
+ );
+ }
+ return req;
+}
+
+// The `sweep` prompt's declared argument names, IN ORDER. Claude Code zips the
+// user's whitespace-split words against exactly this list, so the order is also
+// the recipe for reassembling what they typed when it has shredded a request.
+export const SWEEP_PROMPT_ARGS = [
+ "query",
+ "link",
+ "channel",
+ "channels",
+ "group",
+ "directive",
+ "batch_size",
+ "parse_model",
+ "report_path",
+] as const;
+
+export type ArgumentProblem = { arg: string; value: string; why: string };
+
+// Hard validation for the PROMPT form. The tool path is forgiving (a bad value
+// becomes a default plus a ⚠, because failing a whole sweep over a typo is
+// worse than running it at 8 and saying so). The prompt path must not be: a
+// typed argument holding prose is the fingerprint of Claude Code's tokenizer
+// having word-split the request, and continuing would run a sweep for
+// something the user never asked for.
+export function validateSweepArguments(args: Record<string, unknown>): {
+ problems: ArgumentProblem[];
+ reassembled: string;
+} {
+ const str = (k: string): string | undefined => {
+ const v = args[k];
+ return typeof v === "string" && v.trim() !== "" ? v.trim() : undefined;
+ };
+ const problems: ArgumentProblem[] = [];
+ const bad = (arg: string, value: string, why: string): void => {
+ problems.push({ arg, value, why });
+ };
+
+ const link = str("link");
+ if (link && !/^<?https?:\/\//i.test(link)) {
+ bad("link", link, "is not an http(s) URL");
+ }
+ const batch = str("batch_size");
+ if (batch) {
+ const n = Number(batch);
+ if (!Number.isInteger(n) || n < 1 || n > 20) {
+ bad("batch_size", batch, "is not an integer between 1 and 20");
+ }
+ }
+ const report = str("report_path") ?? str("report");
+ if (report && (!report.toLowerCase().endsWith(".md") || report.includes(".."))) {
+ bad("report_path", report, "is not a safe .md path");
+ }
+ const model = str("parse_model");
+ if (model && !/^[A-Za-z0-9._-]+$/.test(model)) {
+ bad("parse_model", model, "is not a model name");
+ }
+
+ // Whatever words did land, back in the order they were typed.
+ const reassembled = SWEEP_PROMPT_ARGS.map((a) => str(a) ?? "")
+ .filter((v) => v !== "")
+ .join(" ");
+ return { problems, reassembled };
+}
+
+// The MCP prompt's declared arguments, routed through the SAME validators, so
+// a form-based client gets an error naming the offending argument rather than
+// a rendered `ceil(N / Why)`.
+export function requestFromArguments(
+ args: Record<string, unknown>,
+): PromptRequest {
+ const warnings: string[] = [];
+ const str = (k: string): string | undefined => {
+ const v = args[k];
+ return typeof v === "string" && v.trim() !== "" ? v.trim() : undefined;
+ };
+
+ const rawLink = str("link");
+ const link = rawLink ? extractLink(rawLink).link : undefined;
+ if (rawLink && !link) {
+ warnings.push(`link "${rawLink}" is not an http(s) URL — ignored.`);
+ }
+
+ const channels = [
+ ...(str("channel") ? [str("channel")!] : []),
+ ...(str("channels") ? splitList(str("channels")!) : []),
+ ];
+ const groups = [
+ ...(str("group") ? [str("group")!] : []),
+ ...(str("groups") ? splitList(str("groups")!) : []),
+ ];
+
+ const batchRaw = str("batch_size");
+ const modelRaw = str("parse_model");
+ const reportRaw = str("report_path") ?? str("report");
+ const typesRaw = str("content_types");
+
+ const req: PromptRequest = {
+ ...(link ? { link } : {}),
+ query: str("query") ?? "",
+ channels,
+ groups,
+ batchSize: batchRaw ? validBatchSize(batchRaw, warnings) : DEFAULT_BATCH_SIZE,
+ parseModel: modelRaw
+ ? validParseModel(modelRaw, warnings)
+ : DEFAULT_PARSE_MODEL,
+ regex: args.regex === true || str("regex") === "true",
+ warnings,
+ };
+ if (reportRaw) {
+ const p = validReportPath(reportRaw, warnings);
+ if (p) req.reportPath = p;
+ }
+ if (str("directive")) req.directive = str("directive");
+ if (str("source")) req.source = str("source");
+ if (typesRaw) {
+ const t = validContentTypes(typesRaw, warnings);
+ if (t) req.contentTypes = t;
+ }
+ return req;
+}
+
+// The ⚠ block a caller puts at the top of its output. Empty when clean.
+export function renderWarnings(warnings: string[]): string {
+ if (warnings.length === 0) return "";
+ return `⚠ ${warnings.join("\n⚠ ")}\n\n`;
+}
diff --git a/mcp/src/protocol.test.ts b/mcp/src/protocol.test.ts
@@ -0,0 +1,212 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { mkdtemp, mkdir, writeFile, readdir, rm } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { Client } from "@modelcontextprotocol/client";
+import {
+ StdioClientTransport,
+ getDefaultEnvironment,
+} from "@modelcontextprotocol/client/stdio";
+import { transcriptPageFileName } from "yt-dlp-transcript-common/lib/manifest";
+
+// ─── Real-process protocol coverage ───
+//
+// Every other server test links a client and server over InMemoryTransport,
+// which only ever exercises the 2025 ("legacy") era — the era decision lives in
+// the serving entry (serveStdio), which an in-memory pair never runs. So a
+// 2026-only regression would be invisible to the ~100 tests next door.
+//
+// This file is the guard: it spawns the real `src/index.ts` over stdio and
+// drives it twice — once negotiating the modern (2026-07-28) era, once on the
+// plain legacy handshake — asserting both still serve the same tools. Keep it
+// unconditional and in the default `test` script.
+
+const HERE = path.dirname(fileURLToPath(import.meta.url));
+const ENTRY = path.join(HERE, "index.ts");
+const TSX = path.join(HERE, "..", "node_modules", ".bin", "tsx");
+
+// The exact advertised tool list, in order. `tools/list` is cached (see
+// CACHE_HINTS) and its order is part of what clients see, so pin it: a rename
+// or an accidental reshuffle has to be a deliberate edit here.
+const EXPECTED_TOOLS = [
+ "list_channels",
+ "search_transcripts",
+ "enumerate_matches",
+ "get_transcript",
+ "get_post",
+ "get_thread",
+ "get_transcripts",
+ "get_video_metadata",
+ "list_sources",
+ "resolve_source",
+ "open_link",
+ "sweep_plan",
+ "ask_plan",
+];
+
+// A minimal composed public dir: one channel, two videos, real shard layout.
+async function writeFixture(): Promise<string> {
+ const dir = await mkdtemp(path.join(os.tmpdir(), "mcp-protocol-"));
+ await writeFile(
+ path.join(dir, "corpus.json"),
+ JSON.stringify({
+ channels: [{ slug: "chan-a", name: "Channel A", videoCount: 2 }],
+ }),
+ );
+ const chDir = path.join(dir, "transcripts", "chan-a");
+ await mkdir(chDir, { recursive: true });
+ await writeFile(
+ path.join(chDir, "manifest.json"),
+ JSON.stringify({
+ version: 1,
+ channelSlug: "chan-a",
+ pageCount: 1,
+ maxPageBytes: 1000,
+ generatedAt: "2026-01-01",
+ slugToPage: { a1: 0, a2: 0 },
+ }),
+ );
+ const rec = (id: string, title: string, text: string) => ({
+ slug: `chan-a/${id}`,
+ id,
+ channelSlug: "chan-a",
+ title,
+ uploadDate: "20240101",
+ duration: 300,
+ channel: "Channel A",
+ description: "",
+ tags: [],
+ isLivestream: false,
+ ageRestricted: false,
+ platform: "youtube",
+ webpageUrl: `https://example.test/${id}`,
+ cues: [{ start: 12, end: 15, text }],
+ });
+ await writeFile(
+ path.join(chDir, transcriptPageFileName(0)),
+ JSON.stringify([
+ rec("a1", "Coffee one", "i love coffee"),
+ rec("a2", "Tea two", "i love tea"),
+ ]),
+ );
+ return dir;
+}
+
+type Session = {
+ client: Client;
+ stateDir: string;
+ close: () => Promise<void>;
+};
+
+// Spawn the real server over stdio against `corpusDir`, connecting with the
+// given negotiation mode. Each session gets its own state dir so the assertion
+// that nothing is persisted is meaningful (and so a stale state file on the
+// developer's machine can never leak into the test).
+async function connect(
+ corpusDir: string,
+ versionNegotiation?: { mode: "auto" | "legacy" },
+): Promise<Session> {
+ const stateDir = await mkdtemp(path.join(os.tmpdir(), "mcp-state-"));
+ const transport = new StdioClientTransport({
+ command: TSX,
+ args: [ENTRY, "--local", corpusDir],
+ // The old controller keyed a state file off this; nothing reads it now,
+ // and the assertion below is that the dir stays empty regardless.
+ env: { ...getDefaultEnvironment(), TRANSCRIPT_MCP_STATE_DIR: stateDir },
+ stderr: "pipe",
+ });
+ const client = new Client(
+ { name: "protocol-test", version: "0" },
+ { capabilities: {}, ...(versionNegotiation ? { versionNegotiation } : {}) },
+ );
+ await client.connect(transport);
+ return {
+ client,
+ stateDir,
+ close: async () => {
+ await client.close();
+ await rm(stateDir, { recursive: true, force: true });
+ },
+ };
+}
+
+function firstText(res: unknown): string {
+ const content = (res as { content: { type: string; text: string }[] }).content;
+ return content.map((c) => c.text).join("\n");
+}
+
+test("stdio: the modern (2026-07-28) era negotiates and serves the tools", async (t) => {
+ const dir = await writeFixture();
+ t.after(() => rm(dir, { recursive: true, force: true }));
+ const s = await connect(dir, { mode: "auto" });
+ t.after(() => s.close());
+
+ assert.equal(
+ s.client.getProtocolEra(),
+ "modern",
+ "server/discover should select the modern era over stdio",
+ );
+
+ const tools = await s.client.listTools();
+ assert.deepEqual(tools.tools.map((x) => x.name), EXPECTED_TOOLS);
+
+ const res = await s.client.callTool({
+ name: "search_transcripts",
+ arguments: { query: "coffee" },
+ });
+ const out = firstText(res);
+ assert.match(out, /Coffee one/);
+ assert.match(out, /total 1 match/);
+});
+
+test("stdio: the legacy (2025) handshake still serves the same tools", async (t) => {
+ const dir = await writeFixture();
+ t.after(() => rm(dir, { recursive: true, force: true }));
+ // No versionNegotiation at all — the SDK's default posture, and what a
+ // client that has never heard of 2026-07-28 sends.
+ const s = await connect(dir);
+ t.after(() => s.close());
+
+ assert.equal(s.client.getProtocolEra(), "legacy");
+
+ const tools = await s.client.listTools();
+ assert.deepEqual(tools.tools.map((x) => x.name), EXPECTED_TOOLS);
+
+ const res = await s.client.callTool({
+ name: "search_transcripts",
+ arguments: { query: "tea" },
+ });
+ assert.match(firstText(res), /Tea two/);
+});
+
+test("stdio: prompts are served on both eras", async (t) => {
+ const dir = await writeFixture();
+ t.after(() => rm(dir, { recursive: true, force: true }));
+ const modern = await connect(dir, { mode: "auto" });
+ t.after(() => modern.close());
+ const legacy = await connect(dir, { mode: "legacy" });
+ t.after(() => legacy.close());
+
+ for (const s of [modern, legacy]) {
+ const prompts = await s.client.listPrompts();
+ assert.deepEqual(prompts.prompts.map((p) => p.name), ["sweep"]);
+ }
+});
+
+test("stdio: a read-only session writes no state file", async (t) => {
+ const dir = await writeFixture();
+ t.after(() => rm(dir, { recursive: true, force: true }));
+ const s = await connect(dir, { mode: "auto" });
+ t.after(() => s.close());
+
+ await s.client.callTool({ name: "list_channels", arguments: {} });
+ await s.client.callTool({
+ name: "search_transcripts",
+ arguments: { query: "coffee" },
+ });
+
+ const left = await readdir(s.stateDir).catch(() => [] as string[]);
+ assert.deepEqual(left, [], "the server must not persist anything");
+});
diff --git a/mcp/src/search.test.ts b/mcp/src/search.test.ts
@@ -1,7 +1,8 @@
import { test } from "node:test";
import assert from "node:assert/strict";
-import { Client } from "@modelcontextprotocol/sdk/client/index.js";
-import { InMemoryTransport } from "@modelcontextprotocol/sdk/inMemory.js";
+// Client AND InMemoryTransport must come from the SAME package: each bundles
+// its own copy with private state, so a pair split across packages never links.
+import { Client, InMemoryTransport } from "@modelcontextprotocol/client";
import type {
ChannelTranscriptsManifest,
ChannelSubsManifest,
@@ -33,6 +34,7 @@ import {
type SearchFilters,
} from "./search";
import { createServer } from "./server";
+import { SourceRegistry } from "./sourceRegistry";
import {
MISSING_STATES,
VIDEO_STATES,
@@ -849,8 +851,8 @@ test("server: the sweep prompt reflects a supplied group scope", async () => {
arguments: { query: "k cups", group: "other" },
});
const text = (got.messages[0].content as { text: string }).text;
- assert.match(text, /scoped to group "other"/);
- assert.match(text, /group "other"/);
+ assert.match(text, /scoped to groups "other"/);
+ assert.match(text, /groups: \["other"\]/);
await client.close();
});
@@ -892,7 +894,9 @@ test("server: the sweep prompt accepts a link= seed and drives open_link", async
});
const text = (got.messages[0].content as { text: string }).text;
assert.match(text, /open_link/);
- assert.match(text, /apply:true/);
+ // One call now does the whole job; dry_run is the opt-in preview.
+ assert.match(text, /dry_run: true/);
+ assert.ok(!text.includes("apply:true"), "the two-phase apply is gone");
assert.match(text, /https:\/\/site\.example\/\?q=coffee/);
// A link-only sweep needs no query argument.
assert.match(text, /share link/);
@@ -1133,3 +1137,339 @@ test("server: search_transcripts renders a post hit without a moment link", asyn
assert.doesNotMatch(out, /moment_base/);
await client.close();
});
+
+// ─── Server-level coverage for the single-record reads and open_link ───
+//
+// These three had no server-level test at all, and open_link's default just
+// changed from "preview" to "decode and search in one call" — exactly the kind
+// of change that needs a test underneath it before it lands.
+
+// A registry whose every spec resolves to the same StubSource, so a share
+// link's origin can be resolved without a live fetch.
+function stubRegistry(): SourceRegistry {
+ return new SourceRegistry(
+ { kind: "remote", url: "https://site.example" },
+ {
+ build: () => new StubSource(),
+ probe: async () => "remote" as const,
+ },
+ );
+}
+
+async function connectRegistry(): Promise<Client> {
+ const server = createServer(stubRegistry());
+ const [ct, st] = InMemoryTransport.createLinkedPair();
+ const client = new Client({ name: "test", version: "0" }, { capabilities: {} });
+ await Promise.all([server.connect(st), client.connect(ct)]);
+ return client;
+}
+
+test("server: get_transcript returns the full transcript as markdown", async () => {
+ const client = await connectClient(new StubSource());
+ const out = firstText(
+ await client.callTool({
+ name: "get_transcript",
+ arguments: { video_id: "a1" },
+ }),
+ );
+ assert.match(out, /Coffee one/);
+ assert.match(out, /i love coffee/);
+ assert.match(out, /\(corpus: /, "every result names its corpus");
+ await client.close();
+});
+
+test("server: get_transcript reports a missing id as an error", async () => {
+ const client = await connectClient(new StubSource());
+ const res = await client.callTool({
+ name: "get_transcript",
+ arguments: { video_id: "nope" },
+ });
+ assert.equal((res as { isError?: boolean }).isError, true);
+ assert.match(firstText(res), /video not found: nope/);
+ await client.close();
+});
+
+test("server: get_video_metadata returns metadata without the cue bodies", async () => {
+ const client = await connectClient(new StubSource());
+ const out = firstText(
+ await client.callTool({
+ name: "get_video_metadata",
+ arguments: { video_id: "a1" },
+ }),
+ );
+ const parsed = JSON.parse(out.slice(0, out.lastIndexOf("}") + 1));
+ assert.equal(parsed.id, "a1");
+ assert.equal(parsed.title, "Coffee one");
+ assert.equal(parsed.channelName, "Channel A");
+ assert.equal(parsed.cueCount, 1);
+ assert.equal(parsed.cues, undefined, "the transcript body is not included");
+ await client.close();
+});
+
+test("server: get_video_metadata reports a missing id as an error", async () => {
+ const client = await connectClient(new StubSource());
+ const res = await client.callTool({
+ name: "get_video_metadata",
+ arguments: { video_id: "nope" },
+ });
+ assert.equal((res as { isError?: boolean }).isError, true);
+ assert.match(firstText(res), /video not found: nope/);
+ await client.close();
+});
+
+test("server: open_link decodes AND searches in one call, returning the handle", async () => {
+ const client = await connectRegistry();
+ const out = firstText(
+ await client.callTool({
+ name: "open_link",
+ arguments: { link: "https://site.example/?q=coffee" },
+ }),
+ );
+ // The plan…
+ assert.match(out, /## Share link/);
+ assert.match(out, /Corpus handle: `remote:https:\/\/site\.example`/);
+ // …and the results, from the same call.
+ assert.match(out, /Coffee one/);
+ assert.match(out, /video\(s\) matched/);
+ await client.close();
+});
+
+test("server: open_link dry_run returns the plan and runs no search", async () => {
+ const client = await connectRegistry();
+ const out = firstText(
+ await client.callTool({
+ name: "open_link",
+ arguments: { link: "https://site.example/?q=coffee", dry_run: true },
+ }),
+ );
+ assert.match(out, /Share-link plan \(dry run\)/);
+ assert.match(out, /no search was performed/);
+ assert.ok(!out.includes("video(s) matched"), "nothing was searched");
+ await client.close();
+});
+
+test("server: open_link honours overrides", async () => {
+ const client = await connectRegistry();
+ const out = firstText(
+ await client.callTool({
+ name: "open_link",
+ arguments: {
+ link: "https://site.example/?q=coffee",
+ overrides: { query: "k cups" },
+ dry_run: true,
+ },
+ }),
+ );
+ assert.match(out, /query replaced with "k cups"/);
+ await client.close();
+});
+
+test("server: open_link accepts a posts query_scope override", async () => {
+ const client = await connectRegistry();
+ const out = firstText(
+ await client.callTool({
+ name: "open_link",
+ arguments: {
+ link: "https://site.example/?q=coffee",
+ overrides: { query: "bluesky", query_scope: "posts" },
+ dry_run: true,
+ },
+ }),
+ );
+ // Declared in the schema but dropped by parseOverrides until now, so a
+ // caller asking for the post corpus silently got transcripts.
+ assert.match(out, /posts/);
+ await client.close();
+});
+
+test("server: open_link rejects a link that is not a URL", async () => {
+ const client = await connectRegistry();
+ const res = await client.callTool({
+ name: "open_link",
+ arguments: { link: "not a url" },
+ });
+ assert.equal((res as { isError?: boolean }).isError, true);
+ await client.close();
+});
+
+// ─── W4: coverage is mechanical, and silently-ignored parameters are not ───
+
+test("server: enumerate_matches returns the whole set in one call", async () => {
+ const client = await connectClient(new StubSource());
+ const enumerated = firstText(
+ await client.callTool({
+ name: "enumerate_matches",
+ arguments: { query: "coffee", batch_size: 2 },
+ }),
+ );
+ // The complete worklist, with the batch arithmetic already done.
+ assert.match(enumerated, /the complete worklist/);
+ assert.match(enumerated, /complete set: yes/);
+ assert.match(enumerated, /batch\(es\) of 2/);
+ assert.match(enumerated, /- a1 \| video \| Channel A \|/);
+ assert.ok(!enumerated.includes("i love coffee"), "no snippet bodies");
+
+ // And its total agrees with what search reports — the invariant a sweep
+ // relies on when it claims coverage.
+ const searched = firstText(
+ await client.callTool({
+ name: "search_transcripts",
+ arguments: { query: "coffee", limit: 1 },
+ }),
+ );
+ const enumTotal = /(\d+) match\(es\) for "coffee"/.exec(enumerated)?.[1];
+ const searchTotal = /total (\d+) match/.exec(searched)?.[1];
+ assert.equal(enumTotal, searchTotal);
+ await client.close();
+});
+
+test("server: enumerate_matches with no matches says so plainly", async () => {
+ const client = await connectClient(new StubSource());
+ const out = firstText(
+ await client.callTool({
+ name: "enumerate_matches",
+ arguments: { query: "zzzznotathing" },
+ }),
+ );
+ assert.match(out, /No matches for "zzzznotathing"/);
+ assert.match(out, /complete set: yes/);
+ await client.close();
+});
+
+test("server: enumerate_matches shouts when a cap made coverage partial", async () => {
+ const client = await connectClient(new StubSource());
+ const out = firstText(
+ await client.callTool({
+ name: "enumerate_matches",
+ arguments: { query: "coffee", max_pages: 1 },
+ }),
+ );
+ // The first line, not a footer note.
+ assert.ok(out.startsWith("⚠ COVERAGE PARTIAL"), out.slice(0, 60));
+ assert.match(out, /are a SAMPLE, not the full set/);
+ assert.match(out, /a PARTIAL worklist/);
+ assert.ok(!out.includes("the complete worklist"));
+ assert.match(out, /complete set: NO — capped/);
+ await client.close();
+});
+
+test("server: an incomplete search page warns ABOVE the hits", async () => {
+ const client = await connectClient(new StubSource());
+ const out = firstText(
+ await client.callTool({
+ name: "search_transcripts",
+ arguments: { query: "coffee", limit: 2 },
+ }),
+ );
+ assert.ok(out.startsWith("⚠ INCOMPLETE PAGE"), out.slice(0, 60));
+ assert.match(out, /Do NOT report a count/);
+ assert.match(out, /enumerate_matches/);
+ // The old footer is still there, so existing consumers keep working.
+ assert.match(out, /has_more: yes/);
+ await client.close();
+});
+
+test("server: a complete search page carries no banner", async () => {
+ const client = await connectClient(new StubSource());
+ const out = firstText(
+ await client.callTool({
+ name: "search_transcripts",
+ arguments: { query: "coffee", limit: 50 },
+ }),
+ );
+ assert.ok(!out.includes("INCOMPLETE PAGE"));
+ assert.match(out, /has_more: no/);
+ await client.close();
+});
+
+test("server: an empty posts corpus is named as empty, not as zero pages", async () => {
+ const client = await connectClient(new StubSource());
+ // chan-a is video-only, so a posts-scoped search there has no index at all.
+ const out = firstText(
+ await client.callTool({
+ name: "search_transcripts",
+ arguments: {
+ query: "anything",
+ channels: ["chan-a"],
+ content_types: ["post"],
+ },
+ }),
+ );
+ assert.match(out, /the post corpus is EMPTY here, not merely unmatched/);
+ await client.close();
+});
+
+test("server: a posts corpus that WAS searched reports its own scan counts", async () => {
+ const client = await connectClient(new StubSource());
+ const out = firstText(
+ await client.callTool({
+ name: "search_transcripts",
+ arguments: {
+ query: "zephyrpost",
+ channels: ["chan-b"],
+ content_types: ["post"],
+ },
+ }),
+ );
+ assert.match(out, /posts: scanned \d+ page\(s\) across 1 posting channel\(s\)/);
+ assert.ok(!out.includes("EMPTY here"));
+ await client.close();
+});
+
+test("server: get_transcripts queries[] merges windows and counts each query", async () => {
+ const client = await connectClient(new StubSource());
+ const out = firstText(
+ await client.callTool({
+ name: "get_transcripts",
+ arguments: { video_ids: ["a1"], queries: ["coffee", "zzzznotathing"] },
+ }),
+ );
+ // Both queries are reported per video, so the one that matched nothing is
+ // visible rather than silently absorbed into the merged excerpt.
+ assert.match(out, /"coffee": 1/);
+ assert.match(out, /"zzzznotathing": 0/);
+ assert.match(out, /i love coffee/);
+ assert.match(out, /queries: "coffee", "zzzznotathing"/);
+ await client.close();
+});
+
+test("server: get_transcripts content_types reaches the post corpus", async () => {
+ const client = await connectClient(new StubSource());
+ const out = firstText(
+ await client.callTool({
+ name: "get_transcripts",
+ arguments: { video_ids: ["p1"] },
+ }),
+ );
+ // p1 is a post id; the declared content_types was never read before, so
+ // this used to come back "not found".
+ assert.match(out, /zephyrpost/);
+ assert.ok(!out.includes("not found: p1"));
+ await client.close();
+});
+
+test("server: get_transcripts video-only scope does not fall through to posts", async () => {
+ const client = await connectClient(new StubSource());
+ const out = firstText(
+ await client.callTool({
+ name: "get_transcripts",
+ arguments: { video_ids: ["p1"], content_types: ["video"] },
+ }),
+ );
+ assert.match(out, /not found: p1/);
+ await client.close();
+});
+
+test("server: the unadvertised channel/group singulars are still parsed", async () => {
+ const client = await connectClient(new StubSource());
+ // Dropped from the advertised schema in favour of the plural forms, but a
+ // stored habit (or an old transcript) must not hard-fail.
+ const out = firstText(
+ await client.callTool({
+ name: "search_transcripts",
+ arguments: { query: "coffee", channel: "chan-a" },
+ }),
+ );
+ assert.match(out, /scope: 1 channel/);
+ await client.close();
+});
diff --git a/mcp/src/search.ts b/mcp/src/search.ts
@@ -191,6 +191,12 @@ export type SearchResult = {
// expansion it applied). Empty for a plain or explicit-regex search.
firedAliases: SearchAlias[];
scanned: { channels: number; pages: number };
+ // The posts pass, counted separately. Without this a posts-only search over a
+ // corpus where no channel ships a posts index reports "scanned 0 page(s)
+ // across 0 channel(s)" — indistinguishable from "searched everything, found
+ // nothing". `channels` counts the channels that actually HAVE a posts index,
+ // so 0 with `requested` true means the post corpus is empty here.
+ postsScanned: { requested: boolean; channels: number; pages: number };
// Coverage is partial — the page cap (MAX_PAGES) or the video cap
// (HARD_VIDEO_CAP) was reached before the corpus was fully scanned.
truncated: boolean;
@@ -333,6 +339,10 @@ export async function searchTranscripts(
const contentTypes = opts.contentTypes ?? [...ALL_CONTENT_TYPES];
const wantVideos = contentTypes.includes("video");
const wantPosts = contentTypes.includes("post");
+ // Counted apart from the video pass so "no posts index anywhere in scope" is
+ // distinguishable from "searched the posts and found nothing".
+ let postChannelsScanned = 0;
+ let postPagesScanned = 0;
outer: for (const ch of channels) {
if (!wantVideos) break;
@@ -406,6 +416,7 @@ export async function searchTranscripts(
continue;
}
if (!pm) continue; // not a social channel
+ postChannelsScanned++;
for (let page = 0; page < pm.pageCount; page++) {
if (pagesScanned >= maxPages) {
truncated = true;
@@ -418,6 +429,7 @@ export async function searchTranscripts(
continue;
}
pagesScanned++;
+ postPagesScanned++;
for (const post of posts) {
if (!match(post.text)) continue;
all.push({
@@ -460,6 +472,11 @@ export async function searchTranscripts(
hasMore: offset + limit < total,
firedAliases,
scanned: { channels: channelsScanned, pages: pagesScanned },
+ postsScanned: {
+ requested: wantPosts,
+ channels: postChannelsScanned,
+ pages: postPagesScanned,
+ },
truncated,
selection: {
all: selection.all,
diff --git a/mcp/src/server.ts b/mcp/src/server.ts
@@ -1,10 +1,9 @@
-import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import {
- CallToolRequestSchema,
- ListToolsRequestSchema,
- ListPromptsRequestSchema,
- GetPromptRequestSchema,
-} from "@modelcontextprotocol/sdk/types.js";
+ Server,
+ type Prompt,
+ type ServerOptions,
+ type Tool,
+} from "@modelcontextprotocol/server";
import { transcriptToMarkdown } from "yt-dlp-transcript-common/lib/transcriptToMarkdown";
import { formatDate } from "yt-dlp-transcript-common/lib/format";
import { momentUrl, momentBaseUrl } from "yt-dlp-transcript-common/lib/momentUrl";
@@ -16,15 +15,9 @@ import {
FALLBACK_GROUP,
type ChannelGroup,
} from "yt-dlp-transcript-common/lib/channelGroups";
-import {
- HubSource,
- RemoteSource,
- type ChannelRef,
- type HubSite,
- type ShardSource,
-} from "./source";
+import type { ChannelRef, HubSite, ShardSource } from "./source";
import type { SourceSpec } from "./sources";
-import type { SourceController } from "./sourceController";
+import { SourceRegistry, type ResolvedSource } from "./sourceRegistry";
import type { Post } from "yt-dlp-transcript-common/lib/posts";
import {
searchTranscripts,
@@ -40,6 +33,18 @@ import {
type ScopedSnippet,
} from "./search";
import {
+ parsePromptRequest,
+ requestFromArguments,
+ validateSweepArguments,
+ DEFAULT_REPORT_PATH,
+ type PromptRequest,
+} from "./promptRequest";
+import {
+ buildSweepInstructions,
+ buildAskInstructions,
+ type PlanContext,
+} from "./instructions";
+import {
decodeShareLink,
applyLinkOverrides,
renderQueryTree,
@@ -130,7 +135,42 @@ function errorText(s: string): ToolResult {
return { content: [{ type: "text", text: s }], isError: true };
}
-const TOOLS = [
+// Cache hints for the 2026-07-28 revision's cacheable results (`ttlMs` /
+// `cacheScope`). Our three cacheable operations — the tool list, the prompt
+// list, and the `server/discover` descriptor the SDK builds from them — are
+// literal constants in this file: identical for every caller, unchanged for the
+// life of the process, and carrying no corpus data (the corpus is chosen
+// per-call now, not per-connection). So they are safely `public` and worth an
+// hour. Everything else — every read tool's result — stays uncached by default.
+// Invalid values throw a RangeError in the Server constructor, so a typo here
+// fails at startup rather than on the wire.
+const CACHE_HINTS: NonNullable<ServerOptions["cacheHints"]> = {
+ "tools/list": { ttlMs: 3_600_000, cacheScope: "public" },
+ "prompts/list": { ttlMs: 3_600_000, cacheScope: "public" },
+ "server/discover": { ttlMs: 3_600_000, cacheScope: "public" },
+};
+
+// The `source` argument every read tool accepts. One shared constant spliced
+// into each literal schema, so adding a tool can't accidentally omit it and
+// `additionalProperties: false` still holds everywhere.
+const SOURCE_ARG = {
+ source: {
+ type: "string",
+ description:
+ "Which corpus to read, as a handle: 'default' (this server's startup " +
+ "corpus — the same as omitting it), 'local:<dir>', 'remote:<site url>', " +
+ "'hub:<hub url>', or 'hub:<hub url>#<siteA,siteB>' for a hub subset. A " +
+ "bare site URL, or a hub member's siteId/title, also works and is " +
+ "normalised. There is no active source and nothing persists between " +
+ "calls: every call reads exactly what it asks for, and every result " +
+ "ends with the canonical handle it actually read, as '(corpus: …)'.",
+ },
+} as const;
+
+// The advertised tool list. Deliberately a hand-written literal array in a
+// fixed order — never generated from a Map or Object.keys — so `tools/list` is
+// byte-stable across processes and safely cacheable (see CACHE_HINTS).
+export const TOOLS: Tool[] = [
{
name: "list_channels",
description:
@@ -140,7 +180,20 @@ const TOOLS = [
"compact list of the groups (id · name · channel count) and whether each " +
"is selected by default — so you can pick a group or channels to scope a " +
"search or sweep to.",
- inputSchema: { type: "object", properties: {}, additionalProperties: false },
+ inputSchema: {
+ type: "object",
+ properties: {
+ ...SOURCE_ARG,
+ refresh: {
+ type: "boolean",
+ description:
+ "Re-read the channel list instead of using this process's cached " +
+ "copy (default false). Only needed if the corpus was rebuilt while " +
+ "the server was running.",
+ },
+ },
+ additionalProperties: false,
+ },
},
{
name: "search_transcripts",
@@ -165,30 +218,22 @@ const TOOLS = [
inputSchema: {
type: "object",
properties: {
+ ...SOURCE_ARG,
query: { type: "string", description: "Term, phrase, or regex to find." },
- channel: {
- type: "string",
- description: "Optional channel slug or name to restrict the search to.",
- },
channels: {
type: "array",
items: { type: "string" },
description:
"Optional list of channel slugs/names to restrict the search to " +
- "(union with channel/group/groups).",
- },
- group: {
- type: "string",
- description:
- "Optional channel group (by id or display name, e.g. 'other' or " +
- "'Extended Universe') to expand to its member channels.",
+ "(union with groups). A single channel is just a one-element list.",
},
groups: {
type: "array",
items: { type: "string" },
description:
"Optional list of channel groups (ids or names) to expand to their " +
- "member channels (union with the other scope fields).",
+ "member channels (union with channels). A single group is just a " +
+ "one-element list.",
},
regex: {
type: "boolean",
@@ -250,6 +295,70 @@ const TOOLS = [
},
},
{
+ name: "enumerate_matches",
+ description:
+ "Get a query's COMPLETE match set as a worklist, in one call — every " +
+ "matching id/title/channel/date, no snippets, plus the batch count for a " +
+ "sweep. Use this, not paged search_transcripts, whenever you need to " +
+ "cover or count everything: the engine materialises the whole match set " +
+ "before slicing, so enumerating is ONE scan where paging the same query " +
+ "would be one full scan per page. If a cap is hit the FIRST line says " +
+ "coverage is partial and the list is a sample — a result without that " +
+ "line is the complete set, and is the only basis on which you may state " +
+ "a total or claim full coverage. Same scoping and alias expansion as " +
+ "search_transcripts.",
+ inputSchema: {
+ type: "object",
+ properties: {
+ ...SOURCE_ARG,
+ query: { type: "string", description: "Term, phrase, or regex to find." },
+ channels: {
+ type: "array",
+ items: { type: "string" },
+ description: "Optional channel slugs/names to restrict the scan to.",
+ },
+ groups: {
+ type: "array",
+ items: { type: "string" },
+ description: "Optional channel groups (ids or names) to restrict to.",
+ },
+ regex: {
+ type: "boolean",
+ description:
+ "Treat query as a case-insensitive regex (default false). " +
+ "Disables alias expansion.",
+ },
+ use_aliases: {
+ type: "boolean",
+ description:
+ "Expand the query via curated aliases (default true; ignored with " +
+ "regex).",
+ },
+ content_types: {
+ type: "array",
+ items: { type: "string", enum: ["video", "post"] },
+ description:
+ "Which corpora to enumerate: 'video' and/or 'post'. Defaults to " +
+ "BOTH.",
+ },
+ batch_size: {
+ type: "number",
+ description:
+ "Videos per batch, used only to report the batch count (default " +
+ "8).",
+ },
+ max_pages: {
+ type: "number",
+ description:
+ "Max shard pages to scan before stopping (default 400). Reaching " +
+ "it makes coverage partial, which is reported loudly.",
+ },
+ },
+ required: ["query"],
+ additionalProperties: false,
+ },
+ },
+ {
name: "get_transcript",
description:
"Fetch one video's full transcript as clean markdown (metadata header + " +
@@ -258,6 +367,7 @@ const TOOLS = [
inputSchema: {
type: "object",
properties: {
+ ...SOURCE_ARG,
video_id: { type: "string", description: "The video id." },
channel: { type: "string", description: "Optional owning channel slug/name." },
timestamps: {
@@ -278,6 +388,7 @@ const TOOLS = [
inputSchema: {
type: "object",
properties: {
+ ...SOURCE_ARG,
post_id: { type: "string", description: "The post id (tweet id / atproto rkey)." },
channel: {
type: "string",
@@ -297,6 +408,7 @@ const TOOLS = [
inputSchema: {
type: "object",
properties: {
+ ...SOURCE_ARG,
post_id: {
type: "string",
description: "Any post id in the thread (root or a reply).",
@@ -322,6 +434,7 @@ const TOOLS = [
inputSchema: {
type: "object",
properties: {
+ ...SOURCE_ARG,
video_ids: {
type: "array",
items: { type: "string" },
@@ -333,6 +446,16 @@ const TOOLS = [
"Optional term/phrase/regex to window around. When given, only " +
"excerpts around matches are returned instead of full transcripts.",
},
+ queries: {
+ type: "array",
+ items: { type: "string" },
+ description:
+ "Several terms to window around in ONE pass (max 8; unioned with " +
+ "query). Windows for all of them merge per video, and each " +
+ "video's header reports the per-query match count — so a query " +
+ "that matched nothing is visible. Use this instead of re-reading " +
+ "the same videos once per term.",
+ },
regex: {
type: "boolean",
description:
@@ -407,6 +530,7 @@ const TOOLS = [
inputSchema: {
type: "object",
properties: {
+ ...SOURCE_ARG,
video_id: { type: "string", description: "The video id." },
channel: { type: "string", description: "Optional owning channel slug/name." },
},
@@ -417,92 +541,61 @@ const TOOLS = [
{
name: "list_sources",
description:
- "Show the ACTIVE corpus this server is currently reading (label + kind + " +
- "target). When there is a hub context, also lists the hub's member sites " +
- "(siteId · title · url), marking which are in the current subset — so you " +
- "can pick a site or a subset to switch to with use_source. You can also " +
- "switch to an arbitrary remote site URL, local dir, or hub URL. The chosen " +
- "source persists across reconnects. Strictly read-only either way.",
- inputSchema: { type: "object", properties: {}, additionalProperties: false },
- },
- {
- name: "use_source",
- description:
- "Switch which corpus this server reads — on the fly, and the choice " +
- "persists across reconnects. Provide EXACTLY ONE target: `site` (a hub " +
- "member by siteId or title — becomes a single-site source with full group/" +
- "alias support), `sites` (a list of hub members by siteId/title — a " +
- "federated subset of the hub), `remote` (an arbitrary deployed site " +
- "origin URL), `local` (a composed public dir on disk), or `hub` (an " +
- "arbitrary hub URL to federate). Unknown site tokens are reported, not " +
- "silently dropped. Still strictly read-only — this only changes which " +
- "already-published static shards are read; nothing is written to any corpus.",
+ "Show the corpora this server can read: its DEFAULT corpus (the one used " +
+ "when a call omits `source`) and, when that default is a hub, the hub's " +
+ "member sites (siteId · title · url) as ready-to-paste handles. There is " +
+ "no 'active' source to change — every read tool takes its own `source` " +
+ "handle, and a call that omits it reads the default. Strictly read-only.",
inputSchema: {
type: "object",
- properties: {
- site: {
- type: "string",
- description:
- "A hub member site to scope to, by siteId or title " +
- "(case-insensitive). Switches to a single-site remote source.",
- },
- sites: {
- type: "array",
- items: { type: "string" },
- description:
- "A list of hub member sites (siteId or title) to federate as a " +
- "subset of the hub.",
- },
- remote: {
- type: "string",
- description: "An arbitrary deployed site origin URL to read.",
- },
- local: {
- type: "string",
- description: "A composed public dir on disk to read.",
- },
- hub: {
- type: "string",
- description: "An arbitrary hub URL to federate over every member.",
- },
- },
+ properties: { ...SOURCE_ARG },
additionalProperties: false,
},
},
{
- name: "reset_source",
+ name: "resolve_source",
description:
- "Return to the source this server was started with (its --hub/--remote/" +
- "--local flag or env) and clear the persisted selection.",
- inputSchema: { type: "object", properties: {}, additionalProperties: false },
+ "Turn a corpus reference into the canonical handle to pass as `source`, " +
+ "and check it can actually be read. Accepts a handle, a bare site or hub " +
+ "URL (auto-detected), or a hub member's siteId/title. Returns the " +
+ "canonical handle plus a channel/group count. It changes NOTHING: there " +
+ "is no active source, so this only tells you what to pass — every read " +
+ "tool still needs the handle in its own `source` argument.",
+ inputSchema: {
+ type: "object",
+ properties: { ...SOURCE_ARG },
+ required: ["source"],
+ additionalProperties: false,
+ },
},
{
name: "open_link",
description:
"Paste an archilyzer viewer **share link** (origin + a `qt=` query tree + " +
- "`fc/ft/fa/fav/fdf/fdt` filters) to re-run that exact search here — at full " +
- "fidelity, with clickable timestamped result links. Two-phase: by default " +
- "(apply:false) it PREVIEWS — decodes the link and returns a plan (resolved " +
- "source, the query tree rendered readably, every active filter, the channel " +
- "scope validated against the live corpus, and anything ignored, e.g. the " +
- "vestigial `fk` tracks) WITHOUT switching source or searching. Adjust in " +
- "natural language by re-calling with `overrides` (e.g. clear_availability " +
- "to drop the availability filter, channels to re-scope, query to replace " +
- "the search). When the plan looks right, call again with apply:true: it " +
- "switches the active source to the link's origin (hub or single-site, " +
- "auto-detected) and returns the first page of linked results. Read-only.",
+ "`fc/ft/fa/fav/fdf/fdt` filters) to re-run that exact search here — at " +
+ "full fidelity, with clickable timestamped result links. ONE call does " +
+ "the whole job: it decodes the link, resolves its origin to a source " +
+ "handle (hub or single-site, auto-detected), validates the channel scope " +
+ "against that live corpus, and returns the plan, the first page of " +
+ "results, and the handle — pass that handle as `source` on the follow-up " +
+ "calls. Adjust in natural language by re-calling with `overrides` (e.g. " +
+ "clear_availability to drop the availability filter, channels to " +
+ "re-scope, query to replace the search). Set dry_run:true to see the plan " +
+ "without searching. Read-only: nothing about this server changes.",
inputSchema: {
type: "object",
properties: {
+ ...SOURCE_ARG,
link: {
type: "string",
description: "The archilyzer viewer share URL to decode and run.",
},
- apply: {
+ dry_run: {
type: "boolean",
description:
- "false (default) previews the plan only; true switches source and " +
- "runs the search.",
+ "true returns the decoded plan WITHOUT running the search — for " +
+ "confirming a link's scope and filters first. Default false: " +
+ "decode and search in one call.",
},
overrides: {
type: "object",
@@ -574,115 +667,179 @@ const TOOLS = [
},
limit: {
type: "number",
- description: "Results per page when apply:true (default 20).",
+ description: "Results per page (default 20).",
},
offset: {
type: "number",
- description: "Skip this many matches when apply:true (default 0).",
+ description: "Skip this many matches (default 0).",
},
},
required: ["link"],
additionalProperties: false,
},
},
-];
-
-// The slice of SourceController the server needs. A bare ShardSource is wrapped
-// in a fixed, non-persisting implementation so the existing tools (and tests
-// that pass a plain source) keep working while switching is simply disabled.
-export interface SourceControllerLike {
- current: ShardSource;
- activeSpec?: SourceSpec;
- switchTo(spec: SourceSpec): Promise<void>;
- reset(): Promise<void>;
- listHubSites(): Promise<HubSite[]>;
- hubUrl(): string | undefined;
-}
-
-// Wrap a bare ShardSource so the server always talks to a controller. Switching
-// is disabled (the server was pointed at one fixed source), but list_sources
-// still reports it and a hub source can still enumerate its members.
-function fixedController(source: ShardSource): SourceControllerLike {
- return {
- current: source,
- activeSpec: undefined,
- async switchTo(): Promise<void> {
- throw new Error(
- "this server is pinned to a single source; source switching is disabled",
- );
- },
- async reset(): Promise<void> {
- // nothing persisted, nothing to return to
- },
- async listHubSites(): Promise<HubSite[]> {
- return source instanceof HubSource ? source.listSites() : [];
+ {
+ name: "sweep_plan",
+ description:
+ "Turn a request written in plain English — optionally with a pasted " +
+ "share link — into the step-by-step plan for a corpus sweep, with the " +
+ "scope, corpus handle and batch count already resolved. Call this when " +
+ "the user asks to sweep, survey, or systematically go through the " +
+ "archive. It only RETURNS instructions: it reads nothing, writes " +
+ "nothing, and starts nothing. Following the plan it returns is a long " +
+ "job that writes a report file with your own Write/Edit tools, so run " +
+ "it when that is what was asked for, and show the plan's ⚠ warnings.",
+ inputSchema: {
+ type: "object",
+ properties: {
+ ...SOURCE_ARG,
+ request: {
+ type: "string",
+ description:
+ "The whole request, verbatim and unsplit: a share link and/or " +
+ "what to sweep for, in the user's own words, punctuation intact. " +
+ "Recognised settings may be appended as key=value — channels=, " +
+ "groups=, batch_size=, parse_model=, report=, directive=, " +
+ "source=, content_types=, regex= (quote a multi-word value). " +
+ "Anything else stays part of the question.",
+ },
+ },
+ required: ["request"],
+ additionalProperties: false,
},
- hubUrl(): string | undefined {
- return source instanceof HubSource ? source.hubBase : undefined;
+ },
+ {
+ name: "ask_plan",
+ description:
+ "Turn a plain-English question about the archive — optionally with a " +
+ "pasted share link — into a step-by-step evidence plan: which searches " +
+ "to run, against which corpus handle, and how to cite the answer. Use " +
+ "it for a question to be answered in conversation (sweep_plan is for a " +
+ "systematic pass that writes a report). It only RETURNS instructions: " +
+ "it reads nothing and writes nothing.",
+ inputSchema: {
+ type: "object",
+ properties: {
+ ...SOURCE_ARG,
+ request: {
+ type: "string",
+ description:
+ "The whole question, verbatim and unsplit, punctuation intact, " +
+ "plus any share link. The same key=value settings as sweep_plan " +
+ "are recognised; anything else stays part of the question.",
+ },
+ },
+ required: ["request"],
+ additionalProperties: false,
},
- };
+ },
+];
+
+// Append the corpus actually read to a result, so no answer can be silently
+// about the wrong archive. Deliberately `(corpus: …)` and not `source:` —
+// `- source:` already means "this video's URL" throughout the output.
+//
+// This lives in the dispatch wrapper, applied to every result including
+// errors: a per-handler echo is one that a new tool forgets.
+function withCorpus(result: ToolResult, resolved: ResolvedSource): ToolResult {
+ const note = [`corpus: ${resolved.handle}`, ...resolved.notes].join("; ");
+ const content = [...result.content];
+ const last = content[content.length - 1];
+ if (last && last.type === "text") {
+ content[content.length - 1] = { ...last, text: `${last.text}\n\n(${note})` };
+ } else {
+ content.push({ type: "text", text: `(${note})` });
+ }
+ return { ...result, content };
}
-// Build a configured MCP server over a data source or a SourceController. The
+// Build a configured MCP server over a data source or a SourceRegistry. The
// core read tools work for local / remote / hub sources — only the ShardSource
-// differs — and the source can be switched at runtime via use_source when a
-// real controller is supplied.
-export function createServer(sourceOrController: ShardSource | SourceController): Server {
- const controller: SourceControllerLike =
- "current" in sourceOrController
- ? sourceOrController
- : fixedController(sourceOrController);
+// differs — and which one a call reads is decided per call by its `source`
+// argument, resolved through the registry. A bare ShardSource is wrapped in a
+// one-source registry, so `createServer(someSource)` still works.
+export function createServer(
+ sourceOrRegistry: ShardSource | SourceRegistry,
+): Server {
+ const registry =
+ sourceOrRegistry instanceof SourceRegistry
+ ? sourceOrRegistry
+ : SourceRegistry.forSource(sourceOrRegistry);
const server = new Server(
{ name: "yt-dlp-transcript-mcp", version: "0.1.0" },
- { capabilities: { tools: {}, prompts: {} } },
+ { capabilities: { tools: {}, prompts: {} }, cacheHints: CACHE_HINTS },
);
- server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: TOOLS }));
+ server.setRequestHandler("tools/list", async () => ({ tools: TOOLS }));
- server.setRequestHandler(ListPromptsRequestSchema, async () => ({
+ server.setRequestHandler("prompts/list", async () => ({
prompts: PROMPTS,
}));
- server.setRequestHandler(GetPromptRequestSchema, async (req) => {
+ server.setRequestHandler("prompts/get", async (req) => {
if (req.params.name !== "sweep") {
throw new Error(`unknown prompt: ${req.params.name}`);
}
return buildSweepPrompt((req.params.arguments ?? {}) as Record<string, unknown>);
});
- server.setRequestHandler(CallToolRequestSchema, async (req) => {
+ server.setRequestHandler("tools/call", async (req) => {
const name = req.params.name;
const args = (req.params.arguments ?? {}) as Record<string, unknown>;
- const source = controller.current;
+
+ // Resolving the handle is the FIRST thing every call does, so a typo'd or
+ // unreachable corpus fails as itself rather than as an empty result set.
+ let resolved: ResolvedSource;
try {
- switch (name) {
- case "list_channels":
- return await handleListChannels(source);
- case "search_transcripts":
- return await handleSearch(source, args);
- case "get_transcript":
- return await handleGetTranscript(source, args);
- case "get_transcripts":
- return await handleGetTranscripts(source, args);
- case "get_post":
- return await handleGetPost(source, args);
- case "get_thread":
- return await handleGetThread(source, args);
- case "get_video_metadata":
- return await handleGetMetadata(source, args);
- case "list_sources":
- return await handleListSources(controller);
- case "use_source":
- return await handleUseSource(controller, args);
- case "reset_source":
- return await handleResetSource(controller);
- case "open_link":
- return await handleOpenLink(controller, args);
- default:
- return errorText(`unknown tool: ${name}`);
- }
+ resolved = await registry.resolve(args.source);
+ } catch (e) {
+ return errorText(`${name}: ${(e as Error).message}`);
+ }
+ const source = resolved.source;
+
+ try {
+ const result = await (async (): Promise<ToolResult> => {
+ switch (name) {
+ case "list_channels":
+ return handleListChannels(source, args);
+ case "search_transcripts":
+ return handleSearch(source, args);
+ case "enumerate_matches":
+ return handleEnumerateMatches(source, args);
+ case "get_transcript":
+ return handleGetTranscript(source, args);
+ case "get_transcripts":
+ return handleGetTranscripts(source, args);
+ case "get_post":
+ return handleGetPost(source, args);
+ case "get_thread":
+ return handleGetThread(source, args);
+ case "get_video_metadata":
+ return handleGetMetadata(source, args);
+ case "list_sources":
+ return handleListSources(registry, resolved);
+ case "resolve_source":
+ return handleResolveSource(resolved);
+ // Unadvertised one-release alias. It no longer switches anything —
+ // it resolves, and says to pass the handle per call instead.
+ case "use_source":
+ return handleLegacyUseSource(registry, args);
+ case "open_link":
+ return handleOpenLink(registry, resolved, args);
+ case "sweep_plan":
+ return handlePlan("sweep", registry, resolved, args);
+ case "ask_plan":
+ return handlePlan("ask", registry, resolved, args);
+ default:
+ return errorText(`unknown tool: ${name}`);
+ }
+ })();
+ return withCorpus(result, resolved);
} catch (e) {
- return errorText(`${name} failed: ${(e as Error).message}`);
+ return withCorpus(
+ errorText(`${name} failed: ${(e as Error).message}`),
+ resolved,
+ );
}
});
@@ -702,8 +859,11 @@ function channelLine(c: ChannelRef): string {
// a group selector for a scoped search/sweep. When the source has no groups
// (absent manifest, or hub mode) every channel folds under the fallback group
// and the Groups line is omitted.
-async function handleListChannels(source: ShardSource): Promise<ToolResult> {
- const channels = await source.listChannels();
+async function handleListChannels(
+ source: ShardSource,
+ args: Record<string, unknown>,
+): Promise<ToolResult> {
+ const channels = await source.listChannels({ refresh: args.refresh === true });
if (channels.length === 0) return text(`No channels found in ${source.label}.`);
const { groups, defaultGroupId } = await source.loadGroups();
@@ -784,6 +944,7 @@ async function handleSearch(
const aliasNote = describeFiredAliases(result.firedAliases);
const scopeNote = describeScope(result.selection);
+ const postsNote = describePostsPass(result);
const rangeStart = result.total === 0 ? 0 : result.offset + 1;
const rangeEnd = result.offset + result.hits.length;
const footer =
@@ -794,15 +955,21 @@ async function handleSearch(
? "; coverage PARTIAL — scan hit the page/video cap"
: "") +
(scopeNote ? `; ${scopeNote}` : "") +
+ (postsNote ? `; ${postsNote}` : "") +
(aliasNote ? `; ${aliasNote}` : "") +
")";
+ // The warning goes ABOVE the hits, not only in the footer. The footer said
+ // "has_more: yes" on the run that motivated this and it was read straight
+ // past — a report then claimed 319 videos swept from a 200-hit page.
+ const banner = incompletePageBanner(result, "search_transcripts");
+
if (result.hits.length === 0) {
const head =
result.total === 0
? `No matches for "${query}".`
: `No matches in this page (offset ${result.offset} is past the ${result.total} total).`;
- return text(`${head}${footer}`);
+ return text(`${banner}${head}${footer}`);
}
const blocks = result.hits.map((h) => {
// A POST has no timeline: no moment link, no [m:ss] stamps. Render it as a
@@ -846,7 +1013,120 @@ async function handleSearch(
? "post(s)"
: "result(s) (videos + posts)";
return text(
- `${result.total} ${label} matching "${query}":\n\n${blocks.join("\n\n")}${footer}${baseNote}`,
+ `${banner}${result.total} ${label} matching "${query}":\n\n` +
+ `${blocks.join("\n\n")}${footer}${baseNote}`,
+ );
+}
+
+// The unmissable header for a page that is not the whole match set. Rendered
+// before the hits so it cannot be scrolled past, and phrased as an instruction
+// because the failure mode it exists to stop is reporting a count from one
+// page. `enumerate_matches` is named because it is the cheap fix: the engine
+// already materialises the full set, so enumerating is ONE scan where paging
+// this query would be ceil(total/limit) full scans.
+function incompletePageBanner(
+ result: { total: number; offset: number; limit: number; hasMore: boolean },
+ tool: string,
+): string {
+ if (!result.hasMore) return "";
+ const shown = `${result.offset + 1}–${Math.min(result.offset + result.limit, result.total)}`;
+ return (
+ `⚠ INCOMPLETE PAGE — ${result.total} total, showing ${shown}. Do NOT ` +
+ `report a count or "all of them" from this page. Call ` +
+ `\`enumerate_matches\` for the full worklist in one scan` +
+ (tool === "search_transcripts" ? "" : ` (this ${tool} page is a slice)`) +
+ `.\n\n`
+ );
+}
+
+// A note distinguishing "the posts corpus was searched and matched nothing"
+// from "there is no posts corpus here at all". The second used to render as
+// `scanned 0 page(s) across 0 channel(s)`, which reads like nothing ran.
+function describePostsPass(result: SearchResult): string {
+ const p = result.postsScanned;
+ if (!p.requested) return "";
+ if (p.channels === 0) {
+ return (
+ "posts: no channel in scope ships a posts index — the post corpus is " +
+ "EMPTY here, not merely unmatched"
+ );
+ }
+ return `posts: scanned ${p.pages} page(s) across ${p.channels} posting channel(s)`;
+}
+
+// The full match set as a worklist, in a single scan. This exists because the
+// paging instruction was in the sweep prompt all along and got ignored — prose
+// cannot be the enforcement mechanism, so the cheap, correct thing is also the
+// one-call thing. A cap being hit is stated on the FIRST line, never only in a
+// footer.
+async function handleEnumerateMatches(
+ source: ShardSource,
+ args: Record<string, unknown>,
+): Promise<ToolResult> {
+ const query = String(args.query ?? "").trim();
+ if (!query) return errorText("query is required");
+
+ const batchSizeArg =
+ typeof args.batch_size === "number" ? Math.floor(args.batch_size) : 8;
+ const batchSize = batchSizeArg >= 1 ? batchSizeArg : 8;
+
+ const result = await searchTranscripts(source, {
+ query,
+ // The singulars stay parsed even though they are no longer advertised.
+ channel: typeof args.channel === "string" ? args.channel : undefined,
+ channels: strArray(args.channels),
+ group: typeof args.group === "string" ? args.group : undefined,
+ groups: strArray(args.groups),
+ regex: args.regex === true,
+ useAliases: args.use_aliases !== false,
+ maxPages: typeof args.max_pages === "number" ? args.max_pages : undefined,
+ contentTypes: parseContentTypes(args.content_types),
+ includeSnippets: false,
+ offset: 0,
+ // The whole set: searchTranscripts collects up to HARD_VIDEO_CAP and then
+ // slices, so asking for every collected match costs nothing extra.
+ limit: Number.MAX_SAFE_INTEGER,
+ });
+
+ const partial = result.truncated;
+ const head = partial
+ ? `⚠ COVERAGE PARTIAL — the scan hit its page/video cap before the corpus ` +
+ `was exhausted. The ${result.total} match(es) below are a SAMPLE, not ` +
+ `the full set: the true total is higher. Narrow the scope (channels/` +
+ `groups) or raise max_pages to enumerate exhaustively, and say so in any ` +
+ `report built on this.\n\n`
+ : "";
+
+ const rows = result.hits.map((h) => {
+ const kind = h.contentType === "post" ? "post" : "video";
+ return (
+ `- ${h.videoId} | ${kind} | ${h.channelName} | ` +
+ `${formatDate(h.uploadDate)} | ${h.matches} match(es) | ${h.title}`
+ );
+ });
+
+ const batches = Math.ceil(result.total / batchSize);
+ const scopeNote = describeScope(result.selection);
+ const aliasNote = describeFiredAliases(result.firedAliases);
+ const postsNote = describePostsPass(result);
+
+ const footer =
+ `\n\n(complete set: ${partial ? "NO — capped" : "yes"}; ` +
+ `${result.total} match(es); ${batches} batch(es) of ${batchSize}; ` +
+ `scanned ${result.scanned.pages} page(s) across ` +
+ `${result.scanned.channels} channel(s)` +
+ (scopeNote ? `; ${scopeNote}` : "") +
+ (postsNote ? `; ${postsNote}` : "") +
+ (aliasNote ? `; ${aliasNote}` : "") +
+ ")";
+
+ if (result.total === 0) {
+ return text(`${head}No matches for "${query}".${footer}`);
+ }
+ return text(
+ `${head}${result.total} match(es) for "${query}" — ` +
+ `${partial ? "a PARTIAL worklist" : "the complete worklist"}:\n\n` +
+ `${rows.join("\n")}${footer}`,
);
}
@@ -906,24 +1186,51 @@ async function handleGetTranscripts(
const dropped = ids.length > CAP ? ids.length - CAP : 0;
const batch = ids.slice(0, CAP);
- const query = typeof args.query === "string" ? args.query.trim() : "";
+ // One query or several. `queries` kills the refetch-per-quote pattern: the
+ // windows for every query merge in ONE pass per video (mergeSnippets), and
+ // each header reports a per-query count — so a query that matched nothing
+ // becomes visible instead of vanishing into a merged excerpt.
+ const QUERY_CAP = 8;
+ const queryList = [
+ ...(typeof args.query === "string" && args.query.trim() !== ""
+ ? [args.query.trim()]
+ : []),
+ ...(strArray(args.queries) ?? []),
+ ];
+ const droppedQueries =
+ queryList.length > QUERY_CAP ? queryList.length - QUERY_CAP : 0;
+ const queries = queryList.slice(0, QUERY_CAP);
+
const channelHint = [
...(typeof args.channel === "string" ? [args.channel] : []),
...(strArray(args.channels) ?? []),
];
const timestamps = args.timestamps !== false;
-
- // Build the (alias-aware) matcher once for the whole batch when windowing.
- let matcher: ReturnType<typeof buildMatcher> | null = null;
- if (query) {
+ const types = parseContentTypes(args.content_types);
+ const wantVideos = types.includes("video");
+ const wantPosts = types.includes("post");
+
+ // Build the (alias-aware) matchers once for the whole batch when windowing:
+ // one per query for the counts, plus their OR for the window pass.
+ const firedAliases: SearchAlias[] = [];
+ let perQuery: { query: string; match: (t: string) => boolean }[] = [];
+ let matcher: { match: (t: string) => boolean } | null = null;
+ if (queries.length > 0) {
const useAliases = args.use_aliases !== false && args.regex !== true;
const aliases = useAliases ? await source.loadAliases() : [];
- matcher = buildMatcher({
- query,
- regex: args.regex === true,
- useAliases,
- aliases,
+ perQuery = queries.map((q) => {
+ const built = buildMatcher({
+ query: q,
+ regex: args.regex === true,
+ useAliases,
+ aliases,
+ });
+ for (const a of built.firedAliases) {
+ if (!firedAliases.some((x) => x.id === a.id)) firedAliases.push(a);
+ }
+ return { query: q, match: built.match };
});
+ matcher = { match: (t: string) => perQuery.some((m) => m.match(t)) };
}
const before = typeof args.before === "number" ? args.before : 30;
@@ -937,8 +1244,16 @@ async function handleGetTranscripts(
const blocks: string[] = [];
const missing: string[] = [];
for (const id of batch) {
- const found = await findVideo(source, id, channelHint);
+ const found = wantVideos ? await findVideo(source, id, channelHint) : null;
if (!found) {
+ // content_types was declared on this tool and never read, so a batch of
+ // post ids silently came back "not found". Fall through to the post
+ // corpus for anything the transcript lookup didn't resolve.
+ const post = wantPosts ? await findPost(source, id, channelHint) : null;
+ if (post) {
+ blocks.push(postToMarkdown(post.post));
+ continue;
+ }
missing.push(id);
continue;
}
@@ -974,10 +1289,24 @@ async function handleGetTranscripts(
stamp,
maxLines,
});
+ // Per-query counts, so a query contributing nothing to this video is
+ // visible rather than absorbed into the merged window — that visibility
+ // is the real diagnostic value of batching several queries.
+ const cues = record.cues ?? [];
+ const counts = perQuery.map((m) => ({
+ query: m.query,
+ n: cues.reduce((acc, c) => acc + (m.match(c.text) ? 1 : 0), 0),
+ }));
+ const countNote =
+ counts.length > 1
+ ? counts.map((c) => `"${c.query}": ${c.n}`).join(", ")
+ : `${matchCount} matching line(s)`;
const body =
matchCount === 0
- ? "_(no lines matched the query in this transcript)_"
- : `_(${matchCount} matching line(s), windowed)_\n${lines.join("\n")}`;
+ ? `_(no lines matched ${
+ counts.length > 1 ? "any query" : "the query"
+ } in this transcript${counts.length > 1 ? ` — ${countNote}` : ""})_`
+ : `_(${countNote}, windowed)_\n${lines.join("\n")}`;
blocks.push(`${head}\n\n${body}`);
} else {
const md = transcriptToMarkdown(
@@ -1000,12 +1329,14 @@ async function handleGetTranscripts(
}
const notes: string[] = [];
- if (query) {
- const aliasNote = describeFiredAliases(matcher?.firedAliases ?? []);
- if (aliasNote) notes.push(aliasNote);
- }
+ if (queries.length > 1) notes.push(`queries: ${queries.map((q) => `"${q}"`).join(", ")}`);
+ const aliasNote = describeFiredAliases(firedAliases);
+ if (aliasNote) notes.push(aliasNote);
if (missing.length > 0) notes.push(`not found: ${missing.join(", ")}`);
if (dropped > 0) notes.push(`${dropped} extra id(s) beyond the 20-cap dropped`);
+ if (droppedQueries > 0) {
+ notes.push(`${droppedQueries} quer(ies) beyond the ${QUERY_CAP}-cap dropped`);
+ }
if (base) notes.push(BASE_EXPANSION_NOTE);
const footer = notes.length > 0 ? `\n\n(${notes.join("; ")})` : "";
@@ -1141,10 +1472,10 @@ async function handleGetMetadata(
);
}
-// ─── Source switching: list_sources / use_source / reset_source ───
+// ─── Source discovery: list_sources / resolve_source ───
// A one-line human description of a source spec's target (kind + where it
-// points), for the list_sources / use_source reports.
+// points), for the list_sources / resolve_source reports.
function describeSpec(spec: SourceSpec | undefined): string {
if (!spec) return "(pinned source)";
switch (spec.kind) {
@@ -1159,194 +1490,145 @@ function describeSpec(spec: SourceSpec | undefined): string {
}
}
-// A cheap post-switch summary: channel count, and group count when the new
-// source actually ships channel groups. Tolerant of a source that can't be
-// reached (reports the switch anyway).
-async function sourceSummary(source: ShardSource): Promise<string> {
- try {
- const channels = await source.listChannels();
- const { groups } = await source.loadGroups();
- const groupNote =
- groups.length > 0 ? `, ${groups.length} group(s)` : "";
- return `${channels.length} channel(s)${groupNote}`;
- } catch (e) {
- return `(could not read the new source: ${(e as Error).message})`;
- }
-}
-
-// Report the active source and, when a hub is in play, its member sites so a
-// human can pick one (or a subset) to switch to.
+// Report the corpus this call read (the default when `source` was omitted) and,
+// when a hub is in play, its member sites as ready-to-paste handles. There is
+// no active source to report — only a default and whatever a caller names.
async function handleListSources(
- controller: SourceControllerLike,
+ registry: SourceRegistry,
+ resolved: ResolvedSource,
): Promise<ToolResult> {
+ const isDefault = resolved.handle === registry.defaultHandle;
const lines: string[] = [
- `Active source: ${controller.current.label}`,
- ` target: ${describeSpec(controller.activeSpec)}`,
+ `Corpus read by this call: ${resolved.label}`,
+ ` handle: ${resolved.handle}` + (isDefault ? " (the default)" : ""),
+ ` target: ${describeSpec(resolved.spec)}`,
];
+ if (!isDefault) {
+ lines.push(
+ ` default (used when a call omits source): ${registry.defaultHandle}`,
+ );
+ }
+ const hubUrl =
+ resolved.spec.kind === "hub" ? resolved.spec.url : registry.hubUrl();
let sites: HubSite[] = [];
- try {
- sites = await controller.listHubSites();
- } catch (e) {
- lines.push(`\n(could not list hub members: ${(e as Error).message})`);
+ if (hubUrl) {
+ try {
+ sites = await registry.listHubSites(hubUrl);
+ } catch (e) {
+ lines.push(`\n(could not list hub members: ${(e as Error).message})`);
+ }
}
if (sites.length > 0) {
- // Which siteIds are in the current subset (only when the active source is a
- // hub scoped to a subset); otherwise every member is "in scope".
- const spec = controller.activeSpec;
+ // When the resolved corpus is a hub subset, mark who is actually in it.
+ const spec = resolved.spec;
const subset =
- spec && spec.kind === "hub" && spec.sites && spec.sites.length > 0
+ spec.kind === "hub" && spec.sites && spec.sites.length > 0
? new Set(spec.sites)
: undefined;
- const rows = sites.map((s) => {
- const inScope = !subset || subset.has(s.siteId);
- const mark = subset ? (inScope ? "✓ " : " ") : "";
- return ` - ${mark}${s.siteId} · ${s.title} · ${s.url}`;
+ const rows = sites.map((site) => {
+ const mark = subset ? (subset.has(site.siteId) ? "✓ " : " ") : "";
+ return (
+ ` - ${mark}${site.siteId} · ${site.title} · ${site.url}\n` +
+ ` source: remote:${site.url.replace(/\/+$/, "")}`
+ );
});
lines.push(
`\nHub member sites (${sites.length})` +
- (subset ? ` — ✓ = in the current subset` : "") +
+ (subset ? ` — ✓ = in the subset this call read` : "") +
`:\n${rows.join("\n")}`,
);
+ lines.push(
+ `\nA subset of them: source:"hub:${hubUrl}#${sites
+ .slice(0, 2)
+ .map((x) => x.siteId)
+ .join(",")}".`,
+ );
}
lines.push(
- `\nSwitch with use_source: site:"<siteId|title>" (one member), ` +
- `sites:["<a>","<b>"] (a subset), or an arbitrary remote:"<url>" / ` +
- `local:"<dir>" / hub:"<url>". reset_source returns to the startup source. ` +
- `Read-only throughout.`,
+ `\nPass a corpus per call, in each tool's own \`source\` argument — ` +
+ `'default', 'local:<dir>', 'remote:<url>', 'hub:<url>', or ` +
+ `'hub:<url>#<siteA,siteB>'. Nothing is switched or remembered: a call ` +
+ `without \`source\` always reads the default. resolve_source turns a ` +
+ `loose reference into the handle to pass. Read-only throughout.`,
);
return text(lines.join("\n"));
}
-// Switch the active source. Exactly one target family is accepted; a hub member
-// token (site/sites) is resolved against the hub's member list.
-async function handleUseSource(
- controller: SourceControllerLike,
- args: Record<string, unknown>,
+// Resolve a reference to its canonical handle and verify the corpus can be
+// read. Mutates nothing — the point is to hand back a string to pass as
+// `source`, not to select anything.
+async function handleResolveSource(
+ resolved: ResolvedSource,
): Promise<ToolResult> {
- const site = typeof args.site === "string" ? args.site.trim() : "";
- const sites = strArray(args.sites);
- const remote = typeof args.remote === "string" ? args.remote.trim() : "";
- const local = typeof args.local === "string" ? args.local.trim() : "";
- const hub = typeof args.hub === "string" ? args.hub.trim() : "";
-
- const families = [
- site ? "site" : "",
- sites ? "sites" : "",
- remote ? "remote" : "",
- local ? "local" : "",
- hub ? "hub" : "",
- ].filter((f) => f !== "");
- if (families.length === 0) {
- return errorText(
- "use_source needs exactly one target: site, sites, remote, local, or hub.",
+ const lines = [
+ `Handle: ${resolved.handle}`,
+ ` label: ${resolved.label}`,
+ ` target: ${describeSpec(resolved.spec)}`,
+ ];
+ for (const n of resolved.notes) lines.push(` note: ${n}`);
+
+ try {
+ const channels = await resolved.source.listChannels();
+ const { groups } = await resolved.source.loadGroups();
+ lines.push(
+ ` reachable: yes — ${channels.length} channel(s)` +
+ (groups.length > 0 ? `, ${groups.length} group(s)` : ""),
);
- }
- if (families.length > 1) {
+ } catch (e) {
return errorText(
- `use_source takes exactly one target; got ${families.join(", ")}.`,
+ `${lines.join("\n")}\n reachable: NO — ${(e as Error).message}`,
);
}
- let spec: SourceSpec;
- if (remote) {
- spec = { kind: "remote", url: remote };
- } else if (local) {
- spec = { kind: "local", dir: local };
- } else if (hub) {
- spec = { kind: "hub", url: hub };
- } else {
- // site / sites — resolve against the hub member list.
- const hubUrl = controller.hubUrl();
- if (!hubUrl) {
- return errorText(
- "no hub context to resolve a site token against — start the server " +
- "against a hub, or first switch with hub:\"<url>\" (or use remote/local).",
- );
- }
- let members: HubSite[];
- try {
- members = await controller.listHubSites();
- } catch (e) {
- return errorText(`could not read hub members: ${(e as Error).message}`);
- }
- const resolve = (token: string): HubSite | undefined => {
- const t = token.toLowerCase();
- return members.find(
- (m) => m.siteId.toLowerCase() === t || m.title.toLowerCase() === t,
- );
- };
-
- if (site) {
- const member = resolve(site);
- if (!member) {
- return errorText(
- `unknown hub site "${site}". Known: ` +
- members.map((m) => m.siteId).join(", "),
- );
- }
- // A single member becomes a plain remote source → full group/alias support.
- spec = { kind: "remote", url: member.url };
- } else {
- const resolved: string[] = [];
- const unknown: string[] = [];
- for (const token of sites!) {
- const member = resolve(token);
- if (member) resolved.push(member.siteId);
- else unknown.push(token);
- }
- if (resolved.length === 0) {
- return errorText(
- `no known hub sites in [${sites!.join(", ")}]. Known: ` +
- members.map((m) => m.siteId).join(", "),
- );
- }
- spec = { kind: "hub", url: hubUrl, sites: resolved };
- if (unknown.length > 0) {
- // Switch to the resolvable subset but surface the typos.
- try {
- await controller.switchTo(spec);
- } catch (e) {
- return errorText(`use_source failed: ${(e as Error).message}`);
- }
- const summary = await sourceSummary(controller.current);
- return text(
- `Switched to ${controller.current.label} — ${summary}.\n` +
- ` target: ${describeSpec(spec)}\n` +
- `Unknown site token(s) skipped: ${unknown.join(", ")}.`,
- );
- }
- }
- }
-
- try {
- await controller.switchTo(spec);
- } catch (e) {
- return errorText(`use_source failed: ${(e as Error).message}`);
- }
- const summary = await sourceSummary(controller.current);
- return text(
- `Switched to ${controller.current.label} — ${summary}.\n` +
- ` target: ${describeSpec(spec)}`,
+ lines.push(
+ ``,
+ `Nothing was switched. Pass source:"${resolved.handle}" on each call ` +
+ `that should read this corpus.`,
);
+ return text(lines.join("\n"));
}
-// Return to the startup source and clear the persisted selection.
-async function handleResetSource(
- controller: SourceControllerLike,
+// The old switching tool, kept one release as an unadvertised alias so a habit
+// (or a stored transcript) doesn't hard-fail. It resolves its target and hands
+// back the handle; it cannot switch anything, because there is nothing to
+// switch.
+async function handleLegacyUseSource(
+ registry: SourceRegistry,
+ args: Record<string, unknown>,
): Promise<ToolResult> {
+ const sites = strArray(args.sites);
+ const token =
+ (typeof args.site === "string" && args.site.trim()) ||
+ (typeof args.remote === "string" && `remote:${args.remote.trim()}`) ||
+ (typeof args.local === "string" && `local:${args.local.trim()}`) ||
+ (typeof args.hub === "string" && `hub:${args.hub.trim()}`) ||
+ (sites ? `sites:${sites.join(",")}` : "");
+ if (!token) {
+ return errorText(
+ "use_source is gone: there is no active source to switch. Pass the " +
+ "corpus per call in each tool's `source` argument instead, and use " +
+ "resolve_source to turn a URL or site name into a handle.",
+ );
+ }
+
+ let target: ResolvedSource;
try {
- await controller.reset();
+ target = await registry.resolve(token);
} catch (e) {
- return errorText(`reset_source failed: ${(e as Error).message}`);
+ return errorText(`use_source: ${(e as Error).message}`);
}
- const summary = await sourceSummary(controller.current);
return text(
- `Reset to the startup source ${controller.current.label} — ${summary}.\n` +
- ` target: ${describeSpec(controller.activeSpec)}`,
+ `use_source no longer switches anything — this server holds no active ` +
+ `source (a persisted one silently sent every call to the wrong corpus, ` +
+ `so it was removed).\n\n` +
+ `That target's handle is: ${target.handle}\n` +
+ ` target: ${describeSpec(target.spec)}\n` +
+ (target.notes.length > 0 ? ` note: ${target.notes.join("; ")}\n` : "") +
+ `\nPass source:"${target.handle}" on each call that should read it.`,
);
}
@@ -1376,6 +1658,11 @@ function parseOverrides(raw: unknown): LinkOverrides | undefined {
queryScope:
scope === "transcripts" ||
scope === "chat" ||
+ // "posts" was declared in the tool schema and dropped here, so a caller
+ // asking to re-target a link at the post corpus silently got transcripts.
+ // Everything downstream (LayerScope, runSearchSpec's posts pass) already
+ // supported it.
+ scope === "posts" ||
scope === "metadata" ||
scope === "description" ||
scope === "tags"
@@ -1386,28 +1673,6 @@ function parseOverrides(raw: unknown): LinkOverrides | undefined {
type OriginProbe = { kind: "hub" | "remote"; siteCount?: number; channelCount?: number };
-// Probe an origin's corpus.json to classify it as a federated hub (has a
-// `sites[]` array) or a single site (has `channels[]`). Transient — no source is
-// committed. Defaults to single-site when the probe can't be read.
-async function probeOrigin(origin: string): Promise<OriginProbe> {
- try {
- const res = await fetch(`${origin}/corpus.json`);
- if (res.ok) {
- const j = (await res.json()) as {
- sites?: unknown[];
- channels?: unknown[];
- };
- if (Array.isArray(j.sites)) return { kind: "hub", siteCount: j.sites.length };
- if (Array.isArray(j.channels)) {
- return { kind: "remote", channelCount: j.channels.length };
- }
- }
- } catch {
- // unreachable — fall through to the single-site default
- }
- return { kind: "remote" };
-}
-
type ChannelScope = {
channels: ChannelRef[];
all: boolean;
@@ -1441,20 +1706,22 @@ async function resolveLinkChannels(
return { channels, all: false, matched, unknown };
}
-// Render the human-readable preview/echo plan for a decoded link.
+// Render the human-readable plan for a decoded link.
function buildLinkPlan(
decoded: DecodedLink,
probe: OriginProbe,
scope: ChannelScope,
- applied: boolean,
+ handle: string,
+ searched: boolean,
): string {
const lines: string[] = [];
- lines.push(applied ? "## Applied share link" : "## Share-link preview");
+ lines.push(searched ? "## Share link" : "## Share-link plan (dry run)");
const kindNote =
probe.kind === "hub"
? `hub (${probe.siteCount ?? "?"} member site(s))`
: `single site${probe.channelCount != null ? ` (${probe.channelCount} channel(s))` : ""}`;
lines.push(`- Source: ${decoded.origin} — ${kindNote}`);
+ lines.push(`- Corpus handle: \`${handle}\` — pass this as \`source\` on the follow-up calls`);
lines.push(`- Query: ${renderQueryTree(decoded.tree)} _(from ${decoded.querySource})_`);
if (scope.all) {
@@ -1483,11 +1750,11 @@ function buildLinkPlan(
for (const w of decoded.warnings) lines.push(`- Note: ${w}`);
- if (!applied) {
+ if (!searched) {
lines.push(
``,
- `No search run yet. Call again with apply:true to switch source and ` +
- `search, or pass overrides to adjust first.`,
+ `Dry run — no search was performed. Re-call without dry_run to search, ` +
+ `or pass overrides to adjust the plan first.`,
);
}
return lines.join("\n");
@@ -1527,7 +1794,8 @@ function renderSpecResults(source: ShardSource, hits: SpecHit[]): string {
}
async function handleOpenLink(
- controller: SourceControllerLike,
+ registry: SourceRegistry,
+ callerCorpus: ResolvedSource,
args: Record<string, unknown>,
): Promise<ToolResult> {
const linkStr = typeof args.link === "string" ? args.link.trim() : "";
@@ -1541,47 +1809,43 @@ async function handleOpenLink(
}
decoded = applyLinkOverrides(decoded, parseOverrides(args.overrides));
- const apply = args.apply === true;
- const probe = await probeOrigin(decoded.origin);
-
- if (!apply) {
- // Preview: resolve channels against a transient source; do not commit.
- const preview: ShardSource =
- probe.kind === "hub"
- ? new HubSource(decoded.origin)
- : new RemoteSource(decoded.origin);
- let scope: ChannelScope;
+ // The link's ORIGIN decides the corpus, not the caller's `source` — a share
+ // link is a pointer to a specific deployment. An explicit `source` still
+ // wins, for the case where the same corpus is reachable at another handle.
+ const explicitSource =
+ typeof args.source === "string" && args.source.trim() !== "";
+ let target: ResolvedSource;
+ if (explicitSource) {
+ target = callerCorpus;
+ } else {
try {
- scope = await resolveLinkChannels(preview, decoded);
+ target = await registry.resolve(decoded.origin);
} catch (e) {
return errorText(
- `could not read the corpus at ${decoded.origin}: ${(e as Error).message}`,
+ `could not resolve the link's origin ${decoded.origin}: ${(e as Error).message}`,
);
}
- return text(buildLinkPlan(decoded, probe, scope, false));
}
-
- // Apply: commit the source (reusing the controller), then search.
- const spec: SourceSpec =
- probe.kind === "hub"
- ? { kind: "hub", url: decoded.origin }
- : { kind: "remote", url: decoded.origin };
- try {
- await controller.switchTo(spec);
- } catch (e) {
- return errorText(`open_link could not switch source: ${(e as Error).message}`);
- }
- const source = controller.current;
+ const source = target.source;
+ const probe: OriginProbe =
+ target.spec.kind === "hub" ? { kind: "hub" } : { kind: "remote" };
let scope: ChannelScope;
try {
scope = await resolveLinkChannels(source, decoded);
} catch (e) {
return errorText(
- `switched to ${source.label} but could not read its corpus: ${(e as Error).message}`,
+ `could not read the corpus at ${target.handle}: ${(e as Error).message}`,
);
}
- const plan = buildLinkPlan(decoded, probe, scope, true);
+ probe.channelCount = scope.all ? scope.channels.length : undefined;
+
+ // dry_run stops here: the plan, with no search. The default is to do the
+ // whole job in this one call — the old two-phase apply:true cost a round
+ // trip and, worse, mutated a global.
+ const dryRun = args.dry_run === true;
+ const plan = buildLinkPlan(decoded, probe, scope, target.handle, !dryRun);
+ if (dryRun) return text(plan);
const limit = typeof args.limit === "number" ? args.limit : 20;
const offset = typeof args.offset === "number" ? args.offset : 0;
@@ -1614,7 +1878,118 @@ async function handleOpenLink(
: `No matches in this page (offset ${result.offset} is past the ${result.total} total).`
: `${result.total} video(s) matched:\n\n${renderSpecResults(source, result.hits)}`;
- return text(`${plan}\n\n---\n\n${body}${footer}`);
+ const banner = incompletePageBanner(result, "open_link");
+ return text(`${plan}\n\n---\n\n${banner}${body}${footer}`);
+}
+
+// ─── sweep_plan / ask_plan: a pasted request in, a resolved plan out ───
+
+// Pre-resolve the facts the model would otherwise burn round-trips on:
+// canonicalise the corpus, validate the channel/group tokens against the live
+// channel list, and offer the group roster when no scope was given. Tolerant —
+// an unreadable corpus just means fewer resolved facts, not a failed plan.
+async function buildPlanContext(
+ resolved: ResolvedSource,
+ req: PromptRequest,
+): Promise<PlanContext> {
+ const ctx: PlanContext = { corpus: resolved.handle, notes: [...resolved.notes] };
+ let channels: ChannelRef[];
+ let groups: ChannelGroup[];
+ try {
+ channels = await resolved.source.listChannels();
+ groups = (await resolved.source.loadGroups()).groups;
+ } catch (e) {
+ ctx.notes = [
+ ...(ctx.notes ?? []),
+ `could not read ${resolved.handle} to validate the scope: ${(e as Error).message}`,
+ ];
+ return ctx;
+ }
+
+ const known = (token: string): boolean => {
+ const want = token.toLowerCase();
+ return channels.some(
+ (c) =>
+ c.slug.toLowerCase() === want ||
+ c.key.toLowerCase() === want ||
+ c.name.toLowerCase() === want,
+ );
+ };
+ ctx.knownChannels = req.channels.filter(known);
+ ctx.unknownChannels = req.channels.filter((c) => !known(c));
+
+ const knownGroup = (token: string): boolean => {
+ const want = token.toLowerCase();
+ return groups.some(
+ (g) =>
+ g.id.toLowerCase() === want ||
+ (g.name.trim() !== "" && g.name.toLowerCase() === want),
+ );
+ };
+ ctx.knownGroups = req.groups.filter(knownGroup);
+ ctx.unknownGroups = req.groups.filter((g) => !knownGroup(g));
+
+ if (req.channels.length === 0 && req.groups.length === 0 && groups.length > 0) {
+ const counts = new Map<string, number>();
+ for (const c of channels) {
+ const gid = resolveChannelGroupId(
+ c.groupId,
+ groups,
+ (await resolved.source.loadGroups()).defaultGroupId,
+ );
+ counts.set(gid, (counts.get(gid) ?? 0) + 1);
+ }
+ ctx.availableGroups = sortGroups(groups).map((g) => ({
+ id: g.id,
+ name: g.name || g.id,
+ channels: counts.get(g.id) ?? 0,
+ }));
+ }
+ ctx.notes = [
+ ...(ctx.notes ?? []),
+ `${channels.length} channel(s) in ${resolved.handle}`,
+ ];
+ return ctx;
+}
+
+// Both plan tools. The request arrives as ONE structured JSON string, so a URL
+// with `?a=b&c=d` and a full sentence of punctuation survive intact — which is
+// the whole reason these are tools and not prompt arguments.
+async function handlePlan(
+ kind: "sweep" | "ask",
+ registry: SourceRegistry,
+ callerCorpus: ResolvedSource,
+ args: Record<string, unknown>,
+): Promise<ToolResult> {
+ const raw = typeof args.request === "string" ? args.request : "";
+ if (raw.trim() === "") {
+ return errorText(
+ `${kind}_plan needs a request: what to ${kind === "sweep" ? "sweep for" : "ask"}, ` +
+ `in plain English, optionally with a share link.`,
+ );
+ }
+ const req = parsePromptRequest(raw);
+
+ // A `source=` inside the request text wins over the tool's own `source`
+ // argument, since it is the more specific statement of intent.
+ let corpus = callerCorpus;
+ if (req.source) {
+ try {
+ corpus = await registry.resolve(req.source);
+ } catch (e) {
+ req.warnings.push(
+ `source=${req.source} could not be resolved (${(e as Error).message}) — ` +
+ `using ${callerCorpus.handle}.`,
+ );
+ }
+ }
+
+ const ctx = await buildPlanContext(corpus, req);
+ return text(
+ kind === "sweep"
+ ? buildSweepInstructions(req, ctx)
+ : buildAskInstructions(req, ctx),
+ );
}
// ─── Prompts: the first-class `sweep` entry point ───
@@ -1623,15 +1998,20 @@ async function handleOpenLink(
// windowed transcripts, and fold findings into a markdown report it maintains
// with its own Write/Edit tools. The MCP stays read-only; the report is a file
// in Claude's cwd. Mirrors the browser's accumulationSystemPrompt discipline.
-const PROMPTS = [
+const PROMPTS: Prompt[] = [
{
name: "sweep",
description:
- "Sweep a query across the corpus (or a chosen group/channels): enumerate " +
- "every matching video, batch-read the transcripts, and fold cited, " +
- "cross-referenced findings into a markdown report — driven by Claude Code " +
- "on plan usage, no API key. With no scope arg it lists the channel groups " +
- "and asks you to pick a group/channels (or confirm 'all') before sweeping.",
+ "IN CLAUDE CODE, USE /sweep INSTEAD — this form's arguments get " +
+ "word-split by the slash-command tokenizer and everything past the " +
+ "ninth word is dropped; it will refuse rather than sweep the wrong " +
+ "thing. This form is for clients that give each argument its own field " +
+ "(Claude Desktop, Cursor). Sweep a query across the corpus (or a chosen " +
+ "group/channels): enumerate every matching video, batch-read the " +
+ "transcripts, and fold cited, cross-referenced findings into a markdown " +
+ "report — on plan usage, no API key. With no scope arg it lists the " +
+ "channel groups and asks you to pick a group/channels (or confirm " +
+ "'all') before sweeping.",
arguments: [
{
name: "query",
@@ -1694,181 +2074,79 @@ const PROMPTS = [
},
];
-function argStr(args: Record<string, unknown>, key: string): string | undefined {
- const v = args[key];
- return typeof v === "string" && v.trim() !== "" ? v.trim() : undefined;
-}
+// The MCP prompt form. Its declared arguments go through the SAME parser and
+// validators as the tools, so a form-based client (Claude Desktop, Cursor —
+// where each argument gets its own field and multi-word values are fine) gets
+// the identical instructions, and a bad value gets a named warning instead of
+// being rendered into the text.
+function buildSweepPrompt(args: Record<string, unknown>): {
+ description: string;
+ messages: { role: "user"; content: { type: "text"; text: string } }[];
+} {
+ // Refuse a shredded request rather than sweeping for something nobody asked
+ // for. A typed argument holding prose ("link" = "search", "batch_size" =
+ // "Why") is the fingerprint of Claude Code's slash-command tokenizer, which
+ // whitespace-splits this prompt's arguments and drops the overflow. A
+ // form-based client, where each argument has its own field, only trips this
+ // by genuinely typing a bad value — which also deserves an error.
+ const { problems, reassembled } = validateSweepArguments(args);
+ if (problems.length > 0) {
+ const named = problems
+ .map((p) => ` - \`${p.arg}\` = "${p.value}" — ${p.why}`)
+ .join("\n");
+ return {
+ description: "sweep: the request did not arrive intact",
+ messages: [
+ {
+ role: "user" as const,
+ content: {
+ type: "text" as const,
+ text:
+ `Do NOT run a sweep. Tell me, briefly, that the request did not ` +
+ `arrive intact, and show me this:\n\n` +
+ `The \`sweep\` prompt received values that cannot be what they ` +
+ `claim to be:\n${named}\n\n` +
+ `In Claude Code this almost always means the slash-command ` +
+ `tokenizer word-split the line: it splits on whitespace, zips ` +
+ `the words onto this prompt's nine declared arguments in order, ` +
+ `and **silently drops everything past the ninth word**. It is ` +
+ `not quote-aware.\n\n` +
+ (reassembled
+ ? `The words that survived, in order, were:\n\n ${reassembled}\n\n` +
+ `Anything after them was discarded.\n\n`
+ : "") +
+ `**Use \`/sweep\` instead** — it passes the whole line through ` +
+ `untouched:\n\n` +
+ ` /sweep ${reassembled || "<link and/or what to sweep for, in your own words>"}\n\n` +
+ `(\`/ask\` is the same thing for a question answered in the ` +
+ `conversation rather than a report file. If \`/sweep\` is not in ` +
+ `the slash list, the commands live in \`.claude/commands/\` and ` +
+ `are picked up when a session starts — restart the session; ` +
+ `restarting the MCP server only reloads its tools.)`,
+ },
+ },
+ ],
+ };
+ }
-function buildSweepPrompt(args: Record<string, unknown>) {
- const query = argStr(args, "query");
- const link = argStr(args, "link");
- if (!query && !link) {
+ const req = requestFromArguments(args);
+ if (!req.query && !req.link) {
throw new Error("sweep requires a query argument (or a link)");
}
- const channel = argStr(args, "channel");
- const group = argStr(args, "group");
- const channelsRaw = argStr(args, "channels");
- const channels = channelsRaw
- ? channelsRaw.split(",").map((s) => s.trim()).filter((s) => s !== "")
- : [];
- const directive = argStr(args, "directive") ?? "key claims & contradictions";
- const batchSize = argStr(args, "batch_size") ?? "8";
- const parseModel = argStr(args, "parse_model") ?? "haiku";
- const reportPath = argStr(args, "report_path") ?? "./sweep-report.md";
- const subject = query ? `"${query}"` : "the share link's search";
-
- // Human-readable scope clauses + the literal search_transcripts scope args to
- // pass. When none is given (and there's no link), the sweep must pick-first
- // (list + ask) rather than silently scanning the whole corpus.
- const scopeClauses: string[] = [];
- if (channel) scopeClauses.push(`channel "${channel}"`);
- if (channels.length > 0)
- scopeClauses.push(`channels [${channels.map((c) => `"${c}"`).join(", ")}]`);
- if (group) scopeClauses.push(`group "${group}"`);
- const hasScope = scopeClauses.length > 0;
- const scopeArgsText = scopeClauses.join(" and ");
-
- const introScope = link
- ? ` seeded from a share link (its source, query tree, and filters — see step 1)`
- : hasScope
- ? ` scoped to ${scopeArgsText}`
- : " over a scope you will confirm with me first (see step 1)";
-
- const steps: string[] = [];
-
- // ── Discovery / enumeration differs for a link-seeded sweep vs a query one ──
- if (link) {
- steps.push(
- `**Decode & confirm the link.** Call \`open_link\` with ` +
- `link="${link}" (preview mode — no apply). It returns a plan: the ` +
- `resolved source (origin, hub or single-site), the query tree, every ` +
- `active filter, the channel scope validated against the corpus, and any ` +
- `ignored bits (e.g. the vestigial \`fk\` tracks). **Show me the plan and ` +
- `confirm it captures what I want.** If I ask for a change ("drop the ` +
- `availability filter", "only channel X", "search Y instead"), re-call ` +
- `\`open_link\` with the matching \`overrides\` until the plan is right.`,
- );
- steps.push(
- `**Apply & enumerate.** Call \`open_link\` again with the confirmed ` +
- `arguments plus apply:true — this switches the active source to the ` +
- `link's origin and runs the search (the full query tree + filters, at ` +
- `fidelity). Page it with a rising \`offset\` (offset += limit) until ` +
- `\`has_more\` is no to collect the whole worklist of video ids; note the ` +
- `\`total\`. Results already carry linked \`[mm:ss](url)\` timestamps and ` +
- `the applied plan is echoed at the top — record the source, query, and ` +
- `filters in the report. If coverage is PARTIAL, say so.`,
- );
- } else {
- if (!hasScope) {
- steps.push(
- `**Choose the scope first — do NOT default to the whole corpus.** No ` +
- `channel/channels/group was supplied. Call \`list_channels\`, present ` +
- `the channel groups and their channels to me, and ask which group(s) ` +
- `or channel(s) to sweep — or to confirm **all** for the whole corpus. ` +
- `Wait for my choice before enumerating anything. Only sweep everything ` +
- `if I explicitly choose "all". Use my choice as the ` +
- `\`channel\`/\`channels\`/\`group\` scope in every ` +
- `\`search_transcripts\` call below.`,
- );
- }
- steps.push(
- `**Search.** Call \`search_transcripts\` with query "${query}"` +
- (hasScope
- ? ` and ${scopeArgsText}`
- : ` and the scope I chose in step 1`) +
- `. Curated aliases auto-expand the query — the footer reports which ` +
- `fired (e.g. mis-transcribed spellings) and names the resolved scope ` +
- `(and warns about any channel/group token that matched nothing — fix a ` +
- `typo before continuing). Treat the *expanded* match set as your target ` +
- `and mention the expansion and the scope in the report.`,
- );
- steps.push(
- `**Enumerate the full worklist.** Page the complete set with ` +
- `\`include_snippets: false\` and a rising \`offset\` (offset += limit) ` +
- `until \`has_more\` is false — this gives you every id/title/channel/date ` +
- `cheaply. Note the \`total\`. If the footer says coverage is PARTIAL ` +
- `(page/video cap), say so in the report — the sweep is then a sample, ` +
- `not exhaustive.`,
- );
- }
-
- steps.push(
- `**Plan.** With N total matches and a batch size of ${batchSize}, that is ` +
- `\`ceil(N / ${batchSize})\` batches. State the plan (N and the batch ` +
- `count) before you start.`,
- );
-
- // ── Feature 2: dumb-extractor-per-batch map-reduce on the cheapest model —
- // the extractor only quotes verbatim base-form excerpts, so the heavy
- // transcript text never reaches the orchestrator (and never costs the big
- // model). Feature 1: linked citations, expanded from moment_base + seconds.
- steps.push(
- `**Per batch (map-reduce), for each group of up to ${batchSize} video ids:**\n` +
- ` - **Spawn a subagent as a DUMB EXTRACTOR on the cheapest model** — ` +
- `use the Task tool and request model "${parseModel}" (its model ` +
- `parameter, or a "${parseModel}"-backed agent type); if no model ` +
- `override is available, spawn it anyway on the default. Give it exactly ` +
- `this job: call \`get_transcripts\` with the batch's ids, the query, and ` +
- `\`link_style: "base"\`, then return ONLY the lines relevant to the ` +
- `directive, VERBATIM — do NOT analyze, summarize, or rephrase anything. ` +
- `Group the kept lines per video as \`### <title>\` + that video's ` +
- `\`moment_base:\` line copied exactly (or its \`source:\` line when ` +
- `there is no moment_base) + the kept lines in their \`[mm:ss|seconds]\` ` +
- `form. Hard budget: at most 40 lines (~600 words) per batch — if more ` +
- `match, keep the strongest and end with \`(+N more matching lines)\`. ` +
- `Return nothing else — the raw transcript bulk stays inside the ` +
- `subagent and never enters your context.\n` +
- ` - **Merge — you (the orchestrator) do ALL the synthesis.** Cross-` +
- `reference the returned fragment against the report so far and upsert ` +
- `findings — claims, and contradictions with earlier claims — into ` +
- `well-titled \`## sections\` of \`${reportPath}\` (Write/Edit). Cite ` +
- `every VIDEO finding as **\`[title @ mm:ss](<moment url>)\`**, expanding each ` +
- `kept \`[mm:ss|seconds]\` stamp by appending the integer after the ` +
- `\`|\` to that video's moment_base (full link = ` +
- `\`<moment_base><seconds>\`; no moment_base → link the \`source:\` URL ` +
- `instead). A POST finding has no timestamp — cite it as ` +
- `**\`[post by <author>, <date>](<source url>)\`** instead, never with ` +
- `\`@ mm:ss\`. Then discard the fragment. Batches are independent, so you ` +
- `may dispatch several subagents in parallel.\n` +
- ` - **Fallback:** if no subagent/Task tool is available, do the batch ` +
- `inline — call \`get_transcripts\` with \`link_style: "base"\` yourself, ` +
- `fold the expanded cited findings into the report, then **drop the raw ` +
- `excerpt text** before moving on (don't carry it forward).`,
- );
-
- steps.push(
- `**Finish.** Repeat to the end of the worklist, then write a short summary ` +
- `section (the scope swept, how many videos covered, headline findings, any ` +
- `partial-coverage caveat) and tell me the report path. Keep every citation ` +
- `a clickable link — \`[title @ mm:ss](url)\` for a video, ` +
- `\`[post by <author>, <date>](url)\` for a post.`,
- );
-
- const numbered = steps
- .map((s, i) => `${i + 1}. ${s}`)
- .join("\n\n");
-
- const text =
- `Run a **corpus sweep** for ${subject}${introScope}, extracting ` +
- `**${directive}**, and maintain a running markdown report at ` +
- `\`${reportPath}\`. You are the sweep engine — work through the whole match ` +
- `set methodically, using the transcript MCP tools for evidence and your own ` +
- `Write/Edit tools for the report. The MCP is read-only; never try to change ` +
- `the archive. The corpus holds video transcripts AND archived social posts. ` +
- `**Cite every video finding as a clickable ` +
- `\`[title @ mm:ss](<moment url>)\` link** (build each moment URL by ` +
- `appending the cited integer seconds to that video's \`moment_base\` from ` +
- `the tool output); **cite every post finding as ` +
- `\`[post by <author>, <date>](<source url>)\`** — posts have no timeline, ` +
- `so they never take a \`@ mm:ss\`.\n\n` +
- `Follow these steps:\n\n${numbered}`;
-
+ // The prompt form has no live corpus to resolve against — it is rendered
+ // before any tool call — so the plan names the server default and the model
+ // resolves the rest as step 1.
+ const ctx: PlanContext = { corpus: "default" };
+ const subject = req.query ? `"${req.query}"` : "the share link's search";
return {
- description: `Corpus sweep for ${subject} → ${reportPath}`,
+ description: `Corpus sweep for ${subject} → ${req.reportPath ?? DEFAULT_REPORT_PATH}`,
messages: [
{
role: "user" as const,
- content: { type: "text" as const, text },
+ content: {
+ type: "text" as const,
+ text: buildSweepInstructions(req, ctx),
+ },
},
],
};
diff --git a/mcp/src/shareLink.ts b/mcp/src/shareLink.ts
@@ -147,7 +147,13 @@ export type LinkOverrides = {
// Replace the whole query with a single leaf (query + optional regex/scope).
query?: string;
regex?: boolean;
- queryScope?: "transcripts" | "chat" | "metadata" | "description" | "tags";
+ queryScope?:
+ | "transcripts"
+ | "chat"
+ | "posts"
+ | "metadata"
+ | "description"
+ | "tags";
};
const KEEP_ALL_FILTERS: SearchFilters = {
diff --git a/mcp/src/source.ts b/mcp/src/source.ts
@@ -123,7 +123,12 @@ function parseGroupsManifest(raw: unknown): ChannelGroups {
// a page of full transcript records.
export interface ShardSource {
readonly label: string;
- listChannels(): Promise<ChannelRef[]>;
+ // The channel list, memoised per source instance. searchTranscripts,
+ // findVideo and findPost all call it, so a 20-id get_transcripts batch used
+ // to cost 20 corpus.json fetches over HTTP. The memo is the promise, so
+ // concurrent callers share one fetch. Pass `refresh` to drop it and re-read —
+ // an explicit staleness escape hatch, deliberately not a TTL.
+ listChannels(opts?: { refresh?: boolean }): Promise<ChannelRef[]>;
transcriptsManifest(ch: ChannelRef): Promise<ChannelTranscriptsManifest>;
transcriptPage(ch: ChannelRef, page: number): Promise<TranscriptDetail[]>;
// The site's shipped curated search aliases (the same /search-aliases.json the
@@ -301,7 +306,18 @@ export class LocalSource implements ShardSource {
return this.groups;
}
- async listChannels(): Promise<ChannelRef[]> {
+ private channelList?: Promise<ChannelRef[]>;
+
+ listChannels(opts: { refresh?: boolean } = {}): Promise<ChannelRef[]> {
+ if (opts.refresh) this.channelList = undefined;
+ this.channelList ??= this.readChannels().catch((e: unknown) => {
+ this.channelList = undefined; // don't memoise a failure
+ throw e;
+ });
+ return this.channelList;
+ }
+
+ private async readChannels(): Promise<ChannelRef[]> {
try {
const raw = await readFile(path.join(this.dir, "corpus.json"), "utf8");
const corpus = JSON.parse(raw) as SiteCorpusJson;
@@ -455,7 +471,18 @@ export class RemoteSource implements ShardSource {
return (await res.json()) as T;
}
- async listChannels(): Promise<ChannelRef[]> {
+ private channelList?: Promise<ChannelRef[]>;
+
+ listChannels(opts: { refresh?: boolean } = {}): Promise<ChannelRef[]> {
+ if (opts.refresh) this.channelList = undefined;
+ this.channelList ??= this.readChannels().catch((e: unknown) => {
+ this.channelList = undefined; // don't memoise a failure
+ throw e;
+ });
+ return this.channelList;
+ }
+
+ private async readChannels(): Promise<ChannelRef[]> {
const corpus = await this.getJson<SiteCorpusJson>("/corpus.json");
return (corpus.channels ?? []).map((c) => ({
key: c.slug,
@@ -544,7 +571,21 @@ export class HubSource implements ShardSource {
return m;
}
- async listChannels(): Promise<ChannelRef[]> {
+ private channelList?: Promise<ChannelRef[]>;
+
+ listChannels(opts: { refresh?: boolean } = {}): Promise<ChannelRef[]> {
+ if (opts.refresh) this.channelList = undefined;
+ this.channelList ??= this.readChannels().catch((e: unknown) => {
+ this.channelList = undefined; // don't memoise a failure
+ throw e;
+ });
+ return this.channelList;
+ }
+
+ // Populating `members` must stay INSIDE the memoised call: memberFor()
+ // depends on it, so a memo that skipped this would leave every
+ // transcriptPage/postsPage dispatch throwing "unknown hub member site".
+ private async readChannels(): Promise<ChannelRef[]> {
const sites = (await this.listSites()).filter(
(s) => !this.allowSiteIds || this.allowSiteIds.has(s.siteId),
);
diff --git a/mcp/src/sourceController.test.ts b/mcp/src/sourceController.test.ts
@@ -1,281 +0,0 @@
-import { test } from "node:test";
-import assert from "node:assert/strict";
-import { mkdtemp, readFile, rm } from "node:fs/promises";
-import os from "node:os";
-import path from "node:path";
-import { Client } from "@modelcontextprotocol/sdk/client/index.js";
-import { InMemoryTransport } from "@modelcontextprotocol/sdk/inMemory.js";
-import type { ChannelTranscriptsManifest } from "yt-dlp-transcript-common/lib/manifest";
-import type { TranscriptDetail } from "yt-dlp-transcript-common/lib/transcripts";
-import type { SearchAlias } from "yt-dlp-transcript-common/lib/searchAliases";
-import type {
- ChannelGroups,
- ChannelRef,
- HubSite,
- ShardSource,
- VideoAvailability,
-} from "./source";
-import type { SourceSpec } from "./sources";
-import { SourceController } from "./sourceController";
-import { createServer } from "./server";
-
-// ─── A build factory over in-memory stub sources (no HTTP) ───
-//
-// Each spec maps to a labelled stub. Hub specs additionally expose listSites()
-// returning a fixed two-member roster, so `site`/`sites` resolution and the
-// list_sources member listing can be exercised with no network.
-
-const HUB_SITES: HubSite[] = [
- { siteId: "alpha", title: "Alpha Site", url: "https://alpha.example" },
- { siteId: "beta", title: "Beta Site", url: "https://beta.example" },
-];
-
-function chanRef(key: string, name: string): ChannelRef {
- return { key, slug: key, name };
-}
-
-// A minimal ShardSource whose only interesting behaviour is a distinct label
-// and a channel count, plus (for hub specs) listSites().
-class FakeSource implements ShardSource {
- readonly label: string;
- readonly listSites?: () => Promise<HubSite[]>;
- constructor(
- label: string,
- private channels: ChannelRef[],
- isHub: boolean,
- ) {
- this.label = label;
- if (isHub) this.listSites = async () => HUB_SITES;
- }
- async loadAliases(): Promise<SearchAlias[]> {
- return [];
- }
- async loadGroups(): Promise<ChannelGroups> {
- return { groups: [], defaultGroupId: "default" };
- }
- async listChannels(): Promise<ChannelRef[]> {
- return this.channels;
- }
- async transcriptsManifest(): Promise<ChannelTranscriptsManifest> {
- throw new Error("not used in these tests");
- }
- async transcriptPage(): Promise<TranscriptDetail[]> {
- return [];
- }
- publicOrigin(): string | null {
- return null;
- }
- async subsManifest(): Promise<null> {
- return null;
- }
- async subsPage(): Promise<[]> {
- return [];
- }
- async postsManifest(): Promise<null> {
- return null;
- }
- async postsPage(): Promise<[]> {
- return [];
- }
- async availabilityMap(): Promise<Map<string, VideoAvailability>> {
- return new Map();
- }
-}
-
-function labelFor(spec: SourceSpec): string {
- switch (spec.kind) {
- case "hub":
- return spec.sites && spec.sites.length > 0
- ? `hub:${spec.url} (${spec.sites.length} site(s))`
- : `hub:${spec.url}`;
- case "remote":
- return `remote:${spec.url}`;
- case "local":
- return `local:${spec.dir}`;
- }
-}
-
-// The injected factory: FakeSource per spec, hub specs get listSites().
-function fakeBuild(spec: SourceSpec): ShardSource {
- const channels =
- spec.kind === "hub"
- ? [chanRef("h1", "Hub Chan 1"), chanRef("h2", "Hub Chan 2")]
- : [chanRef("c1", "Chan 1")];
- return new FakeSource(labelFor(spec), channels, spec.kind === "hub");
-}
-
-async function tempStateFile(): Promise<{ file: string; dir: string }> {
- const dir = await mkdtemp(path.join(os.tmpdir(), "mcp-src-state-"));
- return { file: path.join(dir, "state.json"), dir };
-}
-
-function newController(startup: SourceSpec, stateFile: string): SourceController {
- return new SourceController(startup, { build: fakeBuild, stateFile });
-}
-
-// ─── Client helpers ───
-
-async function connect(controller: SourceController): Promise<Client> {
- const server = createServer(controller);
- const [ct, st] = InMemoryTransport.createLinkedPair();
- const client = new Client({ name: "test", version: "0" }, { capabilities: {} });
- await Promise.all([server.connect(st), client.connect(ct)]);
- return client;
-}
-
-function firstText(res: unknown): string {
- const content = (res as { content: { type: string; text: string }[] }).content;
- return content.map((c) => c.text).join("\n");
-}
-
-const HUB_SPEC: SourceSpec = { kind: "hub", url: "https://hub.example" };
-
-// ─── (a) list_sources shows the active source and hub members ───
-
-test("list_sources shows the active source and, with a hub, its members", async () => {
- const { file, dir } = await tempStateFile();
- const controller = newController(HUB_SPEC, file);
- await controller.init();
- const client = await connect(controller);
-
- const out = firstText(await client.callTool({ name: "list_sources", arguments: {} }));
- assert.match(out, /Active source: hub:https:\/\/hub\.example/);
- assert.match(out, /alpha · Alpha Site · https:\/\/alpha\.example/);
- assert.match(out, /beta · Beta Site · https:\/\/beta\.example/);
-
- await client.close();
- await rm(dir, { recursive: true, force: true });
-});
-
-// ─── (b) use_source remote swaps; reset returns to startup ───
-
-test("use_source remote swaps the active source; reset_source returns to startup", async () => {
- const { file, dir } = await tempStateFile();
- const controller = newController(HUB_SPEC, file);
- await controller.init();
- const client = await connect(controller);
-
- const sw = firstText(
- await client.callTool({
- name: "use_source",
- arguments: { remote: "https://solo.example" },
- }),
- );
- assert.match(sw, /Switched to remote:https:\/\/solo\.example/);
- // A following list_channels reflects the new (remote → single) source.
- const chans = firstText(await client.callTool({ name: "list_channels", arguments: {} }));
- assert.match(chans, /remote:https:\/\/solo\.example/);
- assert.match(chans, /Chan 1/);
- assert.ok(!chans.includes("Hub Chan"), "no longer the hub's channels");
-
- const reset = firstText(await client.callTool({ name: "reset_source", arguments: {} }));
- assert.match(reset, /Reset to the startup source hub:https:\/\/hub\.example/);
- const back = firstText(await client.callTool({ name: "list_channels", arguments: {} }));
- assert.match(back, /Hub Chan 1/);
-
- await client.close();
- await rm(dir, { recursive: true, force: true });
-});
-
-// ─── (c) use_source site resolves a hub member; unknown is reported ───
-
-test("use_source site resolves a hub member to a single remote source", async () => {
- const { file, dir } = await tempStateFile();
- const controller = newController(HUB_SPEC, file);
- await controller.init();
- const client = await connect(controller);
-
- // By title (case-insensitive) → the member's remote origin.
- const byTitle = firstText(
- await client.callTool({ name: "use_source", arguments: { site: "alpha site" } }),
- );
- assert.match(byTitle, /Switched to remote:https:\/\/alpha\.example/);
- assert.equal(controller.activeSpec.kind, "remote");
-
- const unknown = firstText(
- await client.callTool({ name: "use_source", arguments: { site: "gamma" } }),
- );
- assert.match(unknown, /unknown hub site "gamma"/);
-
- await client.close();
- await rm(dir, { recursive: true, force: true });
-});
-
-// ─── (d) use_source sites builds a subset federation ───
-
-test("use_source sites builds a hub subset federation and reports unknown tokens", async () => {
- const { file, dir } = await tempStateFile();
- const controller = newController(HUB_SPEC, file);
- await controller.init();
- const client = await connect(controller);
-
- const out = firstText(
- await client.callTool({
- name: "use_source",
- arguments: { sites: ["alpha", "beta", "ghost"] },
- }),
- );
- assert.match(out, /Switched to hub:https:\/\/hub\.example \(2 site\(s\)\)/);
- assert.match(out, /Unknown site token\(s\) skipped: ghost/);
- assert.equal(controller.activeSpec.kind, "hub");
- assert.deepEqual(
- controller.activeSpec.kind === "hub" ? controller.activeSpec.sites : null,
- ["alpha", "beta"],
- );
-
- await client.close();
- await rm(dir, { recursive: true, force: true });
-});
-
-// ─── (e) persistence across a fresh controller; reset clears it ───
-
-test("a switched source persists to a fresh controller; reset clears the file", async () => {
- const { file, dir } = await tempStateFile();
- const first = newController(HUB_SPEC, file);
- await first.init();
- await first.switchTo({ kind: "remote", url: "https://persisted.example" });
-
- // The state file exists and names the switched spec.
- const raw = JSON.parse(await readFile(file, "utf8"));
- assert.equal(raw.activeSpec.url, "https://persisted.example");
- assert.equal(raw.hubRef, "https://hub.example", "hub context is remembered");
-
- // A brand-new controller over the same startup + state file resumes it.
- const second = newController(HUB_SPEC, file);
- await second.init();
- assert.equal(second.current.label, "remote:https://persisted.example");
- assert.equal(second.activeSpec.kind, "remote");
- // hubRef survives, so site/sites still resolve.
- assert.equal(second.hubUrl(), "https://hub.example");
-
- // reset clears persistence; a third controller starts from the startup spec.
- await second.reset();
- await assert.rejects(readFile(file, "utf8"), "state file deleted");
- const third = newController(HUB_SPEC, file);
- await third.init();
- assert.equal(third.current.label, "hub:https://hub.example");
-
- await rm(dir, { recursive: true, force: true });
-});
-
-// ─── (f) a bare ShardSource still drives createServer (switching disabled) ───
-
-test("createServer accepts a bare ShardSource; list_sources works, use_source is disabled", async () => {
- const bare = new FakeSource("local:/tmp/x", [chanRef("c1", "Chan 1")], false);
- const server = createServer(bare);
- const [ct, st] = InMemoryTransport.createLinkedPair();
- const client = new Client({ name: "test", version: "0" }, { capabilities: {} });
- await Promise.all([server.connect(st), client.connect(ct)]);
-
- const listed = firstText(await client.callTool({ name: "list_sources", arguments: {} }));
- assert.match(listed, /Active source: local:\/tmp\/x/);
-
- const res = await client.callTool({
- name: "use_source",
- arguments: { remote: "https://nope.example" },
- });
- assert.equal((res as { isError?: boolean }).isError, true);
- assert.match(firstText(res), /pinned to a single source/);
-
- await client.close();
-});
diff --git a/mcp/src/sourceController.ts b/mcp/src/sourceController.ts
@@ -1,140 +0,0 @@
-import { mkdir, readFile, writeFile, rm } from "node:fs/promises";
-import os from "node:os";
-import path from "node:path";
-import { HubSource, type HubSite, type ShardSource } from "./source";
-import { buildSource, type SourceSpec } from "./sources";
-
-// What we persist to the state file: the active source spec plus a remembered
-// hub URL (so `site`/`sites` tokens still resolve after switching to a single
-// member or an arbitrary target). Kept minimal and forward-tolerant.
-type PersistedState = {
- activeSpec: SourceSpec;
- hubRef?: string;
-};
-
-// The directory selections are persisted under. TRANSCRIPT_MCP_STATE_DIR wins;
-// else $XDG_STATE_HOME/yt-dlp-transcript-mcp (fallback ~/.local/state/…).
-function stateDir(): string {
- const override = process.env.TRANSCRIPT_MCP_STATE_DIR;
- if (override && override.trim() !== "") return override;
- const xdg = process.env.XDG_STATE_HOME;
- const base =
- xdg && xdg.trim() !== "" ? xdg : path.join(os.homedir(), ".local", "state");
- return path.join(base, "yt-dlp-transcript-mcp");
-}
-
-// A filesystem-safe slug of a source label, used to key the state file so two
-// differently-configured servers (archilyzer / rekietalyzer / a local build)
-// keep independent selections and don't clobber each other.
-function slugify(label: string): string {
- return (
- label
- .toLowerCase()
- .replace(/[^a-z0-9]+/g, "-")
- .replace(/^-+|-+$/g, "") || "default"
- );
-}
-
-// A hub URL a `site`/`sites` token can resolve against: the remembered hubRef,
-// or the active spec's url when the active source is itself a hub.
-function hubUrlFrom(activeSpec: SourceSpec, hubRef?: string): string | undefined {
- if (hubRef) return hubRef;
- if (activeSpec.kind === "hub") return activeSpec.url;
- return undefined;
-}
-
-// Holds the mutable "active source" for the server and persists the selection
-// so it survives reconnects. The `build` factory is injectable so tests can
-// swap in stub sources with no HTTP. Everything the server does per call reads
-// `controller.current`.
-export class SourceController {
- current: ShardSource;
- activeSpec: SourceSpec;
- hubRef?: string;
- readonly startupSpec: SourceSpec;
- readonly stateFile: string;
- private build: (spec: SourceSpec) => ShardSource;
-
- constructor(
- startupSpec: SourceSpec,
- opts: {
- build?: (spec: SourceSpec) => ShardSource;
- stateFile?: string;
- } = {},
- ) {
- this.startupSpec = startupSpec;
- this.build = opts.build ?? buildSource;
- this.activeSpec = startupSpec;
- this.hubRef = startupSpec.kind === "hub" ? startupSpec.url : undefined;
- this.current = this.build(startupSpec);
- this.stateFile =
- opts.stateFile ?? path.join(stateDir(), `${slugify(this.current.label)}.json`);
- }
-
- // Adopt a persisted selection if one exists and parses (persist across
- // reconnects); otherwise stay on the startup source. Never throws — a
- // missing/corrupt state file just means "start fresh".
- async init(): Promise<void> {
- try {
- const raw = await readFile(this.stateFile, "utf8");
- const state = JSON.parse(raw) as PersistedState;
- if (state && state.activeSpec && typeof state.activeSpec.kind === "string") {
- this.activeSpec = state.activeSpec;
- this.hubRef = state.hubRef ?? this.hubRef;
- this.current = this.build(state.activeSpec);
- }
- } catch {
- // no/invalid state file — keep the startup source
- }
- }
-
- // Switch the active source, remembering a hub URL when the target is a hub,
- // and persist the new selection.
- async switchTo(spec: SourceSpec): Promise<void> {
- this.current = this.build(spec);
- this.activeSpec = spec;
- if (spec.kind === "hub") this.hubRef = spec.url;
- await this.persist();
- }
-
- // Return to the startup source and clear the persisted selection.
- async reset(): Promise<void> {
- this.activeSpec = this.startupSpec;
- this.hubRef =
- this.startupSpec.kind === "hub" ? this.startupSpec.url : undefined;
- this.current = this.build(this.startupSpec);
- try {
- await rm(this.stateFile, { force: true });
- } catch {
- // best-effort — nothing to clean up
- }
- }
-
- private async persist(): Promise<void> {
- const state: PersistedState = {
- activeSpec: this.activeSpec,
- hubRef: this.hubRef,
- };
- await mkdir(path.dirname(this.stateFile), { recursive: true });
- await writeFile(this.stateFile, JSON.stringify(state, null, 2), "utf8");
- }
-
- // The hub URL a `site`/`sites` token resolves against (remembered hubRef, or
- // the active spec's url when it's a hub), or undefined when there's no hub
- // context at all.
- hubUrl(): string | undefined {
- return hubUrlFrom(this.activeSpec, this.hubRef);
- }
-
- // List the member sites of the hub context (unfiltered), or [] when there is
- // no hub in play. Built through the same factory so tests can stub it.
- async listHubSites(): Promise<HubSite[]> {
- const url = this.hubUrl();
- if (!url) return [];
- const hub = this.build({ kind: "hub", url });
- if (hub instanceof HubSource) return hub.listSites();
- // A stubbed factory may return a non-HubSource that still exposes listSites.
- const maybe = hub as unknown as { listSites?: () => Promise<HubSite[]> };
- return typeof maybe.listSites === "function" ? maybe.listSites() : [];
- }
-}
diff --git a/mcp/src/sourceRegistry.test.ts b/mcp/src/sourceRegistry.test.ts
@@ -0,0 +1,468 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { readdir } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { Client, InMemoryTransport } from "@modelcontextprotocol/client";
+import type { ChannelTranscriptsManifest } from "yt-dlp-transcript-common/lib/manifest";
+import type { TranscriptDetail } from "yt-dlp-transcript-common/lib/transcripts";
+import type { SearchAlias } from "yt-dlp-transcript-common/lib/searchAliases";
+import type {
+ ChannelGroups,
+ ChannelRef,
+ HubSite,
+ ShardSource,
+ VideoAvailability,
+} from "./source";
+import type { SourceSpec } from "./sources";
+import { SourceRegistry, handleFor } from "./sourceRegistry";
+import { createServer } from "./server";
+
+// ─── A build factory over in-memory stub sources (no HTTP) ───
+
+const HUB_SITES: HubSite[] = [
+ { siteId: "alpha", title: "Alpha Site", url: "https://alpha.example" },
+ { siteId: "beta", title: "Beta Site", url: "https://beta.example" },
+];
+
+function chanRef(key: string, name: string): ChannelRef {
+ return { key, slug: key, name };
+}
+
+class FakeSource implements ShardSource {
+ readonly label: string;
+ readonly listSites?: () => Promise<HubSite[]>;
+ constructor(
+ label: string,
+ private channels: ChannelRef[],
+ isHub: boolean,
+ ) {
+ this.label = label;
+ if (isHub) this.listSites = async () => HUB_SITES;
+ }
+ async loadAliases(): Promise<SearchAlias[]> {
+ return [];
+ }
+ async loadGroups(): Promise<ChannelGroups> {
+ return { groups: [], defaultGroupId: "default" };
+ }
+ async listChannels(): Promise<ChannelRef[]> {
+ return this.channels;
+ }
+ async transcriptsManifest(): Promise<ChannelTranscriptsManifest> {
+ throw new Error("not used in these tests");
+ }
+ async transcriptPage(): Promise<TranscriptDetail[]> {
+ return [];
+ }
+ publicOrigin(): string | null {
+ return null;
+ }
+ async subsManifest(): Promise<null> {
+ return null;
+ }
+ async subsPage(): Promise<[]> {
+ return [];
+ }
+ async postsManifest(): Promise<null> {
+ return null;
+ }
+ async postsPage(): Promise<[]> {
+ return [];
+ }
+ async availabilityMap(): Promise<Map<string, VideoAvailability>> {
+ return new Map();
+ }
+}
+
+function labelFor(spec: SourceSpec): string {
+ switch (spec.kind) {
+ case "hub":
+ return spec.sites && spec.sites.length > 0
+ ? `hub:${spec.url} (${spec.sites.length} site(s))`
+ : `hub:${spec.url}`;
+ case "remote":
+ return `remote:${spec.url}`;
+ case "local":
+ return `local:${spec.dir}`;
+ }
+}
+
+// The injected factory, counting builds so instance reuse is observable.
+function spyBuild(): {
+ build: (spec: SourceSpec) => ShardSource;
+ calls: string[];
+} {
+ const calls: string[] = [];
+ return {
+ calls,
+ build(spec: SourceSpec): ShardSource {
+ calls.push(handleFor(spec));
+ const channels =
+ spec.kind === "hub"
+ ? [chanRef("h1", "Hub Chan 1"), chanRef("h2", "Hub Chan 2")]
+ : spec.kind === "remote"
+ ? [chanRef("r1", "Remote Chan 1")]
+ : [chanRef("c1", "Chan 1")];
+ return new FakeSource(labelFor(spec), channels, spec.kind === "hub");
+ },
+ };
+}
+
+const HUB_SPEC: SourceSpec = { kind: "hub", url: "https://hub.example" };
+
+async function connect(registry: SourceRegistry): Promise<Client> {
+ const server = createServer(registry);
+ const [ct, st] = InMemoryTransport.createLinkedPair();
+ const client = new Client({ name: "test", version: "0" }, { capabilities: {} });
+ await Promise.all([server.connect(st), client.connect(ct)]);
+ return client;
+}
+
+function firstText(res: unknown): string {
+ const content = (res as { content: { type: string; text: string }[] }).content;
+ return content.map((c) => c.text).join("\n");
+}
+
+// ─── Handles ───
+
+test("a handle is the serialised spec and round-trips through resolve", async () => {
+ const { build } = spyBuild();
+ const reg = new SourceRegistry(HUB_SPEC, { build });
+
+ const cases: [string, string][] = [
+ ["local:/srv/site", "local:/srv/site"],
+ ["remote:https://x.example", "remote:https://x.example"],
+ // A trailing slash is meaningless and must not fork the instance cache.
+ ["remote:https://x.example/", "remote:https://x.example"],
+ ["hub:https://hub.example", "hub:https://hub.example"],
+ ["hub:https://hub.example#alpha,beta", "hub:https://hub.example#alpha,beta"],
+ ];
+ for (const [token, expected] of cases) {
+ const r = await reg.resolve(token);
+ assert.equal(r.handle, expected, `${token} → ${expected}`);
+ // Round-trip: feeding the canonical handle back yields the same handle.
+ assert.equal((await reg.resolve(r.handle)).handle, expected);
+ }
+});
+
+test("an absent or 'default' source resolves to the startup spec", async () => {
+ const { build } = spyBuild();
+ const reg = new SourceRegistry(HUB_SPEC, { build });
+ for (const token of [undefined, "", " ", "default", "DEFAULT"]) {
+ assert.equal((await reg.resolve(token)).handle, "hub:https://hub.example");
+ }
+});
+
+test("shorthands normalise to canonical and say so", async () => {
+ const { build } = spyBuild();
+ const reg = new SourceRegistry(HUB_SPEC, { build });
+
+ // A hub member by title, case-insensitively → that member's own origin.
+ const one = await reg.resolve("alpha site");
+ assert.equal(one.handle, "remote:https://alpha.example");
+ assert.match(one.notes.join(" "), /resolved to its origin/);
+
+ // Several members → a hub subset.
+ const subset = await reg.resolve("sites:alpha,beta");
+ assert.equal(subset.handle, "hub:https://hub.example#alpha,beta");
+
+ // Unknown tokens are reported, not silently dropped.
+ const partial = await reg.resolve("sites:alpha,beta,ghost");
+ assert.equal(partial.handle, "hub:https://hub.example#alpha,beta");
+ assert.match(partial.notes.join(" "), /unknown site token\(s\) skipped: ghost/);
+
+ // A path is a local dir.
+ assert.equal((await reg.resolve("/srv/x")).handle, "local:/srv/x");
+ assert.equal((await reg.resolve("./out")).handle, "local:./out");
+});
+
+test("an unresolvable source throws rather than silently reading the default", async () => {
+ const { build } = spyBuild();
+ // No hub context, so a bare word has nothing to resolve against.
+ const reg = new SourceRegistry({ kind: "local", dir: "/srv/x" }, { build });
+ await assert.rejects(reg.resolve("nonsense"), /unrecognised source/);
+ await assert.rejects(reg.resolve("site:alpha"), /no hub context/);
+});
+
+test("every unknown hub site token is an error, not an empty corpus", async () => {
+ const { build } = spyBuild();
+ const reg = new SourceRegistry(HUB_SPEC, { build });
+ await assert.rejects(reg.resolve("sites:ghost,phantom"), /unknown hub site/);
+});
+
+// ─── Caching ───
+
+test("one instance per handle is built, and reused across calls", async () => {
+ const { build, calls } = spyBuild();
+ const reg = new SourceRegistry(HUB_SPEC, { build });
+
+ const a = await reg.resolve("remote:https://x.example");
+ const b = await reg.resolve("remote:https://x.example/");
+ const c = await reg.resolve("remote:https://x.example");
+ assert.equal(a.source, b.source, "trailing slash shares the instance");
+ assert.equal(a.source, c.source);
+ assert.deepEqual(
+ calls.filter((h) => h === "remote:https://x.example").length,
+ 1,
+ "built exactly once",
+ );
+});
+
+test("the hub roster is fetched once per hub url", async () => {
+ let listSitesCalls = 0;
+ const reg = new SourceRegistry(HUB_SPEC, {
+ build(spec) {
+ const src = new FakeSource(labelFor(spec), [], spec.kind === "hub");
+ if (spec.kind === "hub") {
+ Object.defineProperty(src, "listSites", {
+ value: async () => {
+ listSitesCalls++;
+ return HUB_SITES;
+ },
+ });
+ }
+ return src;
+ },
+ });
+ await reg.resolve("alpha");
+ await reg.resolve("beta");
+ await reg.resolve("sites:alpha,beta");
+ assert.equal(listSitesCalls, 1);
+});
+
+// ─── The regression persistence caused ───
+
+test("a per-call source never leaks into the next call", async () => {
+ const { build } = spyBuild();
+ const reg = new SourceRegistry({ kind: "local", dir: "/srv/site" }, { build });
+ const client = await connect(reg);
+
+ // 1. default → the local corpus
+ const first = firstText(
+ await client.callTool({ name: "list_channels", arguments: {} }),
+ );
+ assert.match(first, /Chan 1/);
+ assert.match(first, /\(corpus: local:\/srv\/site\)/);
+
+ // 2. an explicit remote → that corpus, and it says so
+ const second = firstText(
+ await client.callTool({
+ name: "list_channels",
+ arguments: { source: "remote:https://elsewhere.example" },
+ }),
+ );
+ assert.match(second, /Remote Chan 1/);
+ assert.match(second, /\(corpus: remote:https:\/\/elsewhere\.example\)/);
+
+ // 3. default again → BACK on local. This is the exact regression the
+ // persisted active source caused: every later call silently read the
+ // switched-to corpus, and no result said so.
+ const third = firstText(
+ await client.callTool({ name: "list_channels", arguments: {} }),
+ );
+ assert.match(third, /Chan 1/);
+ assert.ok(!third.includes("Remote Chan"), "call 2 must not affect call 3");
+ assert.match(third, /\(corpus: local:\/srv\/site\)/);
+
+ await client.close();
+});
+
+test("every result — including an error — names the corpus it read", async () => {
+ const { build } = spyBuild();
+ const reg = new SourceRegistry({ kind: "local", dir: "/srv/site" }, { build });
+ const client = await connect(reg);
+
+ const ok = await client.callTool({ name: "list_channels", arguments: {} });
+ assert.match(firstText(ok), /\(corpus: local:\/srv\/site\)$/);
+
+ const bad = await client.callTool({
+ name: "search_transcripts",
+ arguments: { query: "" },
+ });
+ assert.equal((bad as { isError?: boolean }).isError, true);
+ assert.match(firstText(bad), /\(corpus: local:\/srv\/site\)$/);
+
+ await client.close();
+});
+
+test("an unresolvable source fails the call by name", async () => {
+ const { build } = spyBuild();
+ const reg = new SourceRegistry({ kind: "local", dir: "/srv/site" }, { build });
+ const client = await connect(reg);
+ const res = await client.callTool({
+ name: "list_channels",
+ arguments: { source: "gibberish" },
+ });
+ assert.equal((res as { isError?: boolean }).isError, true);
+ assert.match(firstText(res), /unrecognised source "gibberish"/);
+ await client.close();
+});
+
+// ─── The tools ───
+
+test("list_sources reports the default and the hub roster as handles", async () => {
+ const { build } = spyBuild();
+ const reg = new SourceRegistry(HUB_SPEC, { build });
+ const client = await connect(reg);
+
+ const out = firstText(
+ await client.callTool({ name: "list_sources", arguments: {} }),
+ );
+ assert.match(out, /handle: hub:https:\/\/hub\.example \(the default\)/);
+ assert.match(out, /alpha · Alpha Site · https:\/\/alpha\.example/);
+ assert.match(out, /source: remote:https:\/\/alpha\.example/);
+ assert.match(out, /hub:https:\/\/hub\.example#alpha,beta/);
+
+ await client.close();
+});
+
+test("resolve_source returns a handle and verifies reachability, changing nothing", async () => {
+ const { build } = spyBuild();
+ const reg = new SourceRegistry(HUB_SPEC, { build });
+ const client = await connect(reg);
+
+ const out = firstText(
+ await client.callTool({
+ name: "resolve_source",
+ arguments: { source: "alpha" },
+ }),
+ );
+ assert.match(out, /Handle: remote:https:\/\/alpha\.example/);
+ assert.match(out, /reachable: yes — 1 channel/);
+ assert.match(out, /Nothing was switched/);
+
+ // And the default is untouched.
+ const after = firstText(
+ await client.callTool({ name: "list_channels", arguments: {} }),
+ );
+ assert.match(after, /Hub Chan 1/);
+
+ await client.close();
+});
+
+test("resolve_source reports an unreachable corpus as an error", async () => {
+ const reg = new SourceRegistry(HUB_SPEC, {
+ build(spec) {
+ const src = new FakeSource(labelFor(spec), [], spec.kind === "hub");
+ if (spec.kind === "remote") {
+ Object.defineProperty(src, "listChannels", {
+ value: async () => {
+ throw new Error("ECONNREFUSED");
+ },
+ });
+ }
+ return src;
+ },
+ });
+ const client = await connect(reg);
+ const res = await client.callTool({
+ name: "resolve_source",
+ arguments: { source: "remote:https://down.example" },
+ });
+ assert.equal((res as { isError?: boolean }).isError, true);
+ assert.match(firstText(res), /reachable: NO — ECONNREFUSED/);
+ await client.close();
+});
+
+test("use_source survives as an alias that resolves instead of switching", async () => {
+ const { build } = spyBuild();
+ const reg = new SourceRegistry(HUB_SPEC, { build });
+ const client = await connect(reg);
+
+ const out = firstText(
+ await client.callTool({
+ name: "use_source",
+ arguments: { remote: "https://solo.example" },
+ }),
+ );
+ assert.match(out, /no longer switches anything/);
+ assert.match(out, /remote:https:\/\/solo\.example/);
+
+ // The default is unchanged — the whole point.
+ const after = firstText(
+ await client.callTool({ name: "list_channels", arguments: {} }),
+ );
+ assert.match(after, /Hub Chan 1/);
+
+ await client.close();
+});
+
+test("use_source and reset_source are no longer advertised", async () => {
+ const { build } = spyBuild();
+ const client = await connect(new SourceRegistry(HUB_SPEC, { build }));
+ const names = (await client.listTools()).tools.map((t) => t.name);
+ assert.ok(!names.includes("use_source"), "unadvertised alias");
+ assert.ok(!names.includes("reset_source"), "deleted outright");
+ assert.ok(names.includes("resolve_source"));
+ await client.close();
+});
+
+test("reset_source is gone", async () => {
+ const { build } = spyBuild();
+ const client = await connect(new SourceRegistry(HUB_SPEC, { build }));
+ const res = await client.callTool({ name: "reset_source", arguments: {} });
+ assert.equal((res as { isError?: boolean }).isError, true);
+ assert.match(firstText(res), /unknown tool: reset_source/);
+ await client.close();
+});
+
+// ─── No filesystem, at all ───
+
+test("resolving and reading writes no state file anywhere", async () => {
+ const { build } = spyBuild();
+ const reg = new SourceRegistry(HUB_SPEC, { build });
+ const client = await connect(reg);
+
+ await client.callTool({ name: "list_channels", arguments: {} });
+ await client.callTool({
+ name: "list_channels",
+ arguments: { source: "remote:https://x.example" },
+ });
+ await client.callTool({
+ name: "resolve_source",
+ arguments: { source: "alpha" },
+ });
+ await client.callTool({
+ name: "use_source",
+ arguments: { remote: "https://y.example" },
+ });
+
+ // The controller used to write here, keyed by a slug of the source label.
+ // Nothing in the registry path can create it: there is no fs import at all.
+ const stateDir = path.join(
+ process.env.XDG_STATE_HOME || path.join(os.homedir(), ".local", "state"),
+ "yt-dlp-transcript-mcp",
+ );
+ const before = await readdir(stateDir).catch(() => null);
+ // If the dir exists it is a leftover from the old build; what matters is
+ // that this run added nothing, so compare against a second read.
+ const after = await readdir(stateDir).catch(() => null);
+ assert.deepEqual(after, before);
+
+ await client.close();
+});
+
+// ─── A bare ShardSource still drives createServer ───
+
+test("createServer accepts a bare ShardSource as its default corpus", async () => {
+ const bare = new FakeSource("local:/tmp/x", [chanRef("c1", "Chan 1")], false);
+ const server = createServer(bare);
+ const [ct, st] = InMemoryTransport.createLinkedPair();
+ const client = new Client({ name: "test", version: "0" }, { capabilities: {} });
+ await Promise.all([server.connect(st), client.connect(ct)]);
+
+ const listed = firstText(
+ await client.callTool({ name: "list_sources", arguments: {} }),
+ );
+ assert.match(listed, /Corpus read by this call: local:\/tmp\/x/);
+ assert.match(listed, /handle: local:\/tmp\/x \(the default\)/);
+
+ const chans = firstText(
+ await client.callTool({ name: "list_channels", arguments: {} }),
+ );
+ assert.match(chans, /Chan 1/);
+ assert.match(chans, /\(corpus: local:\/tmp\/x\)/);
+
+ await client.close();
+});
diff --git a/mcp/src/sourceRegistry.ts b/mcp/src/sourceRegistry.ts
@@ -0,0 +1,317 @@
+import { HubSource, type HubSite, type ShardSource } from "./source";
+import { buildSource, type SourceSpec } from "./sources";
+
+// ─── Explicit, server-minted source handles ───
+//
+// The 2026-07-28 revision removes protocol-level sessions and steers servers
+// that need cross-call state towards "explicit, server-minted handles passed as
+// ordinary tool arguments". This module is that: there is no active source and
+// nothing is persisted — every read tool takes an optional `source` handle and
+// each call resolves it independently.
+//
+// The handle is not an opaque token into a table. It IS the serialised spec, in
+// a canonical round-trippable form:
+//
+// default the CLI/env startup spec
+// local:/dir a composed public dir on disk
+// remote:https://site one deployed site origin
+// hub:https://hub federate every member of a hub
+// hub:https://hub#alpha,beta a hub subset ('#', so it can never collide
+// with a query string in the url)
+//
+// Opaque tokens would die with the process and mean nothing in a transcript;
+// these survive a restart, and a human reading `(corpus: remote:https://…)` in
+// a footer knows exactly what was searched.
+
+export type ResolvedSource = {
+ // The canonical handle — what results echo and what a caller passes back.
+ handle: string;
+ spec: SourceSpec;
+ source: ShardSource;
+ // The live source's own human label (a hub's includes its subset size).
+ label: string;
+ // Anything worth saying about how the token was interpreted: a shorthand
+ // that was normalised, a site token that resolved against the hub roster.
+ notes: string[];
+};
+
+// The canonical handle for a spec. Round-trips through parseHandle.
+export function handleFor(spec: SourceSpec): string {
+ switch (spec.kind) {
+ case "local":
+ return `local:${spec.dir}`;
+ case "remote":
+ return `remote:${spec.url}`;
+ case "hub":
+ return spec.sites && spec.sites.length > 0
+ ? `hub:${spec.url}#${spec.sites.join(",")}`
+ : `hub:${spec.url}`;
+ }
+}
+
+// Trailing slashes are meaningless to every source and would otherwise split
+// the instance cache ("remote:https://x/" vs "remote:https://x").
+function trimUrl(u: string): string {
+ return u.trim().replace(/\/+$/, "");
+}
+
+function splitList(s: string): string[] {
+ return s
+ .split(",")
+ .map((x) => x.trim())
+ .filter((x) => x !== "");
+}
+
+// Parse an explicitly-prefixed handle. Returns undefined for anything that
+// isn't one of the canonical forms — the caller then tries the shorthands.
+function parseHandle(token: string): SourceSpec | undefined {
+ const local = /^local:(.+)$/s.exec(token);
+ if (local) return { kind: "local", dir: local[1].trim() };
+
+ const remote = /^remote:(.+)$/s.exec(token);
+ if (remote) return { kind: "remote", url: trimUrl(remote[1]) };
+
+ const hub = /^hub:(.+)$/s.exec(token);
+ if (hub) {
+ const rest = hub[1].trim();
+ const hash = rest.indexOf("#");
+ if (hash === -1) return { kind: "hub", url: trimUrl(rest) };
+ const sites = splitList(rest.slice(hash + 1));
+ const url = trimUrl(rest.slice(0, hash));
+ return sites.length > 0 ? { kind: "hub", url, sites } : { kind: "hub", url };
+ }
+ return undefined;
+}
+
+type OriginKind = "hub" | "remote";
+
+// Classify an origin by its corpus.json: a federated hub ships `sites[]`, a
+// single site ships `channels[]`. Unreachable/unparseable falls back to a
+// single site, which is the safe guess (a hub that can't be read federates
+// nothing anyway).
+async function probeOriginKind(origin: string): Promise<OriginKind> {
+ try {
+ const res = await fetch(`${origin}/corpus.json`);
+ if (res.ok) {
+ const j = (await res.json()) as { sites?: unknown[]; channels?: unknown[] };
+ if (Array.isArray(j.sites)) return "hub";
+ }
+ } catch {
+ // unreachable — treat it as a single site
+ }
+ return "remote";
+}
+
+// Resolves source handles to live sources, caching one instance per canonical
+// handle for the life of the process. Holds no "current" source: `resolve()` is
+// a pure function of its argument plus the startup spec, so two calls in the
+// same session can read two different corpora and neither can surprise the
+// other.
+export class SourceRegistry {
+ readonly defaultSpec: SourceSpec;
+ readonly defaultHandle: string;
+ private build: (spec: SourceSpec) => ShardSource;
+ private probe: (origin: string) => Promise<OriginKind>;
+ private instances = new Map<string, ShardSource>();
+ private rosters = new Map<string, Promise<HubSite[]>>();
+
+ constructor(
+ defaultSpec: SourceSpec,
+ opts: {
+ build?: (spec: SourceSpec) => ShardSource;
+ // Injectable so tests can resolve a bare origin without a live fetch.
+ probe?: (origin: string) => Promise<OriginKind>;
+ } = {},
+ ) {
+ this.defaultSpec = defaultSpec;
+ this.defaultHandle = handleFor(defaultSpec);
+ this.build = opts.build ?? buildSource;
+ this.probe = opts.probe ?? probeOriginKind;
+ }
+
+ // Wrap an already-built ShardSource as a one-source registry: `default`
+ // resolves to that exact instance. Lets `createServer(someSource)` keep
+ // working unchanged — the tests' in-memory stubs, and any caller that has a
+ // source but no spec.
+ static forSource(source: ShardSource): SourceRegistry {
+ const spec = parseHandle(source.label) ?? {
+ kind: "local" as const,
+ dir: source.label,
+ };
+ const reg = new SourceRegistry(spec);
+ reg.instances.set(reg.defaultHandle, source);
+ // A pinned instance may not match what `build` would produce for its spec
+ // (a stub, or a hub whose label carries its subset size), so pin the label
+ // too — the handle is what callers pass back, the label is what humans read.
+ reg.pinnedLabel = source.label;
+ return reg;
+ }
+
+ private pinnedLabel?: string;
+
+ // The live source for a canonical handle, built once and reused. This is the
+ // cache that makes per-call source selection affordable: without it every
+ // call would rebuild (and so re-fetch every corpus.json).
+ private instanceFor(handle: string, spec: SourceSpec): ShardSource {
+ const hit = this.instances.get(handle);
+ if (hit) return hit;
+ const built = this.build(spec);
+ this.instances.set(handle, built);
+ return built;
+ }
+
+ // The hub a bare site token resolves against: the startup spec when it is a
+ // hub. (There is no remembered hub — a token for some other hub has to name
+ // it, e.g. `hub:https://other#alpha`.)
+ hubUrl(): string | undefined {
+ return this.defaultSpec.kind === "hub" ? this.defaultSpec.url : undefined;
+ }
+
+ // A hub's member roster, fetched once per hub url. Cached as the promise so
+ // concurrent resolutions share one fetch.
+ listHubSites(hubUrl?: string): Promise<HubSite[]> {
+ const url = hubUrl ?? this.hubUrl();
+ if (!url) return Promise.resolve([]);
+ const hit = this.rosters.get(url);
+ if (hit) return hit;
+ const p = (async () => {
+ const src = this.instanceFor(handleFor({ kind: "hub", url }), {
+ kind: "hub",
+ url,
+ });
+ if (src instanceof HubSource) return src.listSites();
+ // A stubbed factory may return a non-HubSource that still lists sites.
+ const maybe = src as unknown as { listSites?: () => Promise<HubSite[]> };
+ return typeof maybe.listSites === "function" ? maybe.listSites() : [];
+ })().catch((e: unknown) => {
+ // Don't cache a failure — a transient network blip shouldn't poison the
+ // roster for the life of the process.
+ this.rosters.delete(url);
+ throw e;
+ });
+ this.rosters.set(url, p);
+ return p;
+ }
+
+ // Resolve a `source` argument to a live source. An absent/blank token — or
+ // the literal "default" — is the startup spec. Everything else is either a
+ // canonical handle or one of the shorthands, which are normalised to
+ // canonical and echoed back so the model learns the canonical form.
+ //
+ // Throws only when a token names something that cannot exist (an unknown hub
+ // member). Reachability is NOT checked here — a read tool surfaces that as
+ // its own failure, and `resolve_source` checks it deliberately.
+ async resolve(token?: unknown): Promise<ResolvedSource> {
+ const raw = typeof token === "string" ? token.trim() : "";
+ const notes: string[] = [];
+
+ if (raw === "" || raw.toLowerCase() === "default") {
+ return this.finish(this.defaultSpec, notes);
+ }
+
+ const direct = parseHandle(raw);
+ if (direct) return this.finish(direct, notes);
+
+ // ── shorthands ──
+
+ // A bare origin: probe corpus.json to tell a hub from a single site.
+ if (/^https?:\/\//i.test(raw)) {
+ const url = trimUrl(raw);
+ const kind = await this.probe(url);
+ notes.push(
+ `"${raw}" probed as a ${kind === "hub" ? "federated hub" : "single site"}`,
+ );
+ return this.finish(
+ kind === "hub" ? { kind: "hub", url } : { kind: "remote", url },
+ notes,
+ );
+ }
+
+ // `site:<token>` / `sites:<a,b>` — hub members, by siteId or title.
+ const site = /^site:(.+)$/s.exec(raw);
+ const sites = /^sites:(.+)$/s.exec(raw);
+ if (site || sites) {
+ const tokens = site ? [site[1].trim()] : splitList(sites![1]);
+ return this.finish(await this.resolveSiteTokens(tokens, notes), notes);
+ }
+
+ // An explicit path — anything that looks like a directory.
+ if (raw.startsWith("/") || raw.startsWith("./") || raw.startsWith("../")) {
+ return this.finish({ kind: "local", dir: raw }, notes);
+ }
+
+ // A bare word: a hub member's siteId or title, when there is a hub to ask.
+ if (this.hubUrl()) {
+ return this.finish(await this.resolveSiteTokens([raw], notes), notes);
+ }
+
+ throw new Error(
+ `unrecognised source "${raw}". Use a handle — default, local:<dir>, ` +
+ `remote:<url>, hub:<url>, or hub:<url>#<siteA,siteB> — or a bare ` +
+ `site URL.`,
+ );
+ }
+
+ // Resolve hub-member tokens (siteId or title, case-insensitive) to a spec.
+ // One member becomes a plain remote source, which keeps full group/alias
+ // support; several become a hub subset.
+ private async resolveSiteTokens(
+ tokens: string[],
+ notes: string[],
+ ): Promise<SourceSpec> {
+ const hubUrl = this.hubUrl();
+ if (!hubUrl) {
+ throw new Error(
+ `no hub context to resolve the site token(s) [${tokens.join(", ")}] ` +
+ `against — this server was not started against a hub. Name one ` +
+ `explicitly: hub:<url>#${tokens.join(",")}`,
+ );
+ }
+ const members = await this.listHubSites(hubUrl);
+ const find = (t: string): HubSite | undefined => {
+ const want = t.toLowerCase();
+ return members.find(
+ (m) => m.siteId.toLowerCase() === want || m.title.toLowerCase() === want,
+ );
+ };
+
+ const resolved: HubSite[] = [];
+ const unknown: string[] = [];
+ for (const t of tokens) {
+ const m = find(t);
+ if (m) resolved.push(m);
+ else unknown.push(t);
+ }
+ if (resolved.length === 0) {
+ throw new Error(
+ `unknown hub site(s): ${unknown.join(", ")}. Known: ` +
+ members.map((m) => m.siteId).join(", "),
+ );
+ }
+ if (unknown.length > 0) {
+ notes.push(`unknown site token(s) skipped: ${unknown.join(", ")}`);
+ }
+ if (resolved.length === 1) {
+ notes.push(
+ `site "${resolved[0].siteId}" resolved to its origin ${resolved[0].url}`,
+ );
+ return { kind: "remote", url: trimUrl(resolved[0].url) };
+ }
+ return { kind: "hub", url: hubUrl, sites: resolved.map((m) => m.siteId) };
+ }
+
+ private finish(spec: SourceSpec, notes: string[]): ResolvedSource {
+ const handle = handleFor(spec);
+ const source = this.instanceFor(handle, spec);
+ return {
+ handle,
+ spec,
+ source,
+ label:
+ handle === this.defaultHandle && this.pinnedLabel
+ ? this.pinnedLabel
+ : source.label,
+ notes,
+ };
+ }
+}
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
@@ -285,10 +285,13 @@ importers:
mcp:
dependencies:
- '@modelcontextprotocol/sdk':
- specifier: ^1.12.0
- version: 1.29.0(zod@4.3.6)
+ '@modelcontextprotocol/server':
+ specifier: ^2.0.0
+ version: 2.0.0
devDependencies:
+ '@modelcontextprotocol/client':
+ specifier: ^2.0.0
+ version: 2.0.0
'@types/node':
specifier: ^20.19.39
version: 20.19.39
@@ -675,12 +678,6 @@ packages:
'@harperfast/extended-iterable@1.0.3':
resolution: {integrity: sha512-sSAYhQca3rDWtQUHSAPeO7axFIUJOI6hn1gjRC5APVE1a90tuyT8f5WIgRsFhhWA7htNkju2veB9eWL6YHi/Lw==}
- '@hono/node-server@1.19.14':
- resolution: {integrity: sha512-GwtvgtXxnWsucXvbQXkRgqksiH2Qed37H9xHZocE5sA3N8O8O8/8FA3uclQXxXVzc9XBZuEOMK7+r02FmSpHtw==}
- engines: {node: '>=18.14.1'}
- peerDependencies:
- hono: ^4
-
'@humanfs/core@0.19.2':
resolution: {integrity: sha512-UhXNm+CFMWcbChXywFwkmhqjs3PRCmcSa/hfBgLIb7oQ5HNb1wS0icWsGtSAUNgefHeI+eBrA8I1fxmbHsGdvA==}
engines: {node: '>=18.18.0'}
@@ -905,15 +902,17 @@ packages:
cpu: [x64]
os: [win32]
- '@modelcontextprotocol/sdk@1.29.0':
- resolution: {integrity: sha512-zo37mZA9hJWpULgkRpowewez1y6ML5GsXJPY8FI0tBBCd77HEvza4jDqRKOXgHNn867PVGCyTdzqpz0izu5ZjQ==}
- engines: {node: '>=18'}
- peerDependencies:
- '@cfworker/json-schema': ^4.1.1
- zod: ^3.25 || ^4.0
- peerDependenciesMeta:
- '@cfworker/json-schema':
- optional: true
+ '@modelcontextprotocol/client@2.0.0':
+ resolution: {integrity: sha512-8f1OghQ2rjzIOfqgUCP+8GiUWqRs89njoWLNqAe8kWmDePv3s1fZXseej+QXemssEuuOvLLmLO/kqM3IQHtISw==}
+ engines: {node: '>=20'}
+
+ '@modelcontextprotocol/core@2.0.0':
+ resolution: {integrity: sha512-pJCEwGG7Lfr/+PQp9ZTwKXNeO5wzbfKL7H3MYpCorM4oFBoQrdjnBgEoqG+RjhsvS1FKrDbKux+M1HhlnGWqcA==}
+ engines: {node: '>=20'}
+
+ '@modelcontextprotocol/server@2.0.0':
+ resolution: {integrity: sha512-YhHWdHfpFMQfd0prsEnxKeS3Qz3ytIGmsS0sth4KDjnacIT7hxk6hXHkJ9KysxlkvTM+WZAtQbbcUhdoP4Hvtw==}
+ engines: {node: '>=20'}
'@msgpackr-extract/msgpackr-extract-darwin-arm64@3.0.3':
resolution: {integrity: sha512-QZHtlVgbAdy2zAqNA9Gu1UpIuI8Xvsd1v8ic6B2pZmeFnFcMWiPLfWXh7TVw4eGEZ/C9TH281KwhVoeQUKbyjw==}
@@ -2092,10 +2091,6 @@ packages:
'@zeit/schemas@2.36.0':
resolution: {integrity: sha512-7kjMwcChYEzMKjeex9ZFXkt1AyNov9R5HZtjBKVsmVpw7pa7ZtlCGvCBC2vnnXctaYN+aRI61HjIqeetZW5ROg==}
- accepts@2.0.0:
- resolution: {integrity: sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng==}
- engines: {node: '>= 0.6'}
-
acorn-jsx@5.3.2:
resolution: {integrity: sha512-rq9s+JNhf0IChjtDXxllJ7g41oZk5SlXtp0LHwyA5cejwn7vKmKp4pPri6YEePv2PU65sAsegbXtIinmDFDXgQ==}
peerDependencies:
@@ -2106,14 +2101,6 @@ packages:
engines: {node: '>=0.4.0'}
hasBin: true
- ajv-formats@3.0.1:
- resolution: {integrity: sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ==}
- peerDependencies:
- ajv: ^8.0.0
- peerDependenciesMeta:
- ajv:
- optional: true
-
ajv@6.15.0:
resolution: {integrity: sha512-fgFx7Hfoq60ytK2c7DhnF8jIvzYgOMxfugjLOSMHjLIPgenqa7S7oaagATUq99mV6IYvN2tRmC0wnTYX6iPbMw==}
@@ -2222,10 +2209,6 @@ packages:
engines: {node: '>=6.0.0'}
hasBin: true
- body-parser@2.3.0:
- resolution: {integrity: sha512-2cGmJupaNgg+QUwVLAucDuWuoMZ6EX9iHDRswZ5lsNYEmwPaRknMPCLZz07yTzVq/83p4o/wzbDZbBrTvGGTIw==}
- engines: {node: '>=18'}
-
bowser@2.14.1:
resolution: {integrity: sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg==}
@@ -2341,33 +2324,9 @@ packages:
resolution: {integrity: sha512-kRGRZw3bLlFISDBgwTSA1TMBFN6J6GWDeubmDE3AF+3+yXL8hTWv8r5rkLbqYXY4RjPk/EzHnClI3zQf1cFmHA==}
engines: {node: '>= 0.6'}
- content-disposition@1.1.0:
- resolution: {integrity: sha512-5jRCH9Z/+DRP7rkvY83B+yGIGX96OYdJmzngqnw2SBSxqCFPd0w2km3s5iawpGX8krnwSGmF0FW5Nhr0Hfai3g==}
- engines: {node: '>=18'}
-
- content-type@1.0.5:
- resolution: {integrity: sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==}
- engines: {node: '>= 0.6'}
-
- content-type@2.0.0:
- resolution: {integrity: sha512-j/O/d7GcZCyNl7/hwZAb606rzqkyvaDctLmckbxLzHvFBzTJHuGEdodATcP3yIRoDrLHkIATJuvzbFlp/ki2cQ==}
- engines: {node: '>=18'}
-
convert-source-map@2.0.0:
resolution: {integrity: sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==}
- cookie-signature@1.2.2:
- resolution: {integrity: sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg==}
- engines: {node: '>=6.6.0'}
-
- cookie@0.7.2:
- resolution: {integrity: sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==}
- engines: {node: '>= 0.6'}
-
- cors@2.8.6:
- resolution: {integrity: sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw==}
- engines: {node: '>= 0.10'}
-
cross-spawn@7.0.6:
resolution: {integrity: sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==}
engines: {node: '>= 8'}
@@ -2481,10 +2440,6 @@ packages:
resolution: {integrity: sha512-8QmQKqEASLd5nx0U1B1okLElbUuuttJ/AnYmRXbbbGDWh6uS208EjD4Xqq/I9wK7u0v6O08XhTWnt5XtEbR6Dg==}
engines: {node: '>= 0.4'}
- depd@2.0.0:
- resolution: {integrity: sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==}
- engines: {node: '>= 0.8'}
-
detect-libc@2.1.2:
resolution: {integrity: sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==}
engines: {node: '>=8'}
@@ -2506,9 +2461,6 @@ packages:
eastasianwidth@0.2.0:
resolution: {integrity: sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA==}
- ee-first@1.1.1:
- resolution: {integrity: sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==}
-
electron-to-chromium@1.5.344:
resolution: {integrity: sha512-4MxfbmNDm+KPh066EZy+eUnkcDPcZ35wNmOWzFuh/ijvHsve6kbLTLURy88uCNK5FbpN+yk2nQY6BYh1GEt+wg==}
@@ -2518,10 +2470,6 @@ packages:
emoji-regex@9.2.2:
resolution: {integrity: sha512-L18DaJsXSUk2+42pv8mLs5jJT2hqFkFE4j21wOmgbUqsZ2hL72NsUU785g9RXgo3s0ZNgVl42TiHp3ZtOv/Vyg==}
- encodeurl@2.0.0:
- resolution: {integrity: sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==}
- engines: {node: '>= 0.8'}
-
enhanced-resolve@5.21.0:
resolution: {integrity: sha512-otxSQPw4lkOZWkHpB3zaEQs6gWYEsmX4xQF68ElXC/TWvGxGMSGOvoNbaLXm6/cS/fSfHtsEdw90y20PCd+sCA==}
engines: {node: '>=10.13.0'}
@@ -2567,9 +2515,6 @@ packages:
resolution: {integrity: sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==}
engines: {node: '>=6'}
- escape-html@1.0.3:
- resolution: {integrity: sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==}
-
escape-string-regexp@4.0.0:
resolution: {integrity: sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA==}
engines: {node: '>=10'}
@@ -2698,10 +2643,6 @@ packages:
resolution: {integrity: sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g==}
engines: {node: '>=0.10.0'}
- etag@1.8.1:
- resolution: {integrity: sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==}
- engines: {node: '>= 0.6'}
-
eventemitter3@4.0.7:
resolution: {integrity: sha512-8guHBZCwKnFhYdHr2ysuRWErTwhoN2X8XELRlrRwpmfeY2jjuUN4taQMsULKUVo1K4DvZl+0pgfyoysHxvmvEw==}
@@ -2725,16 +2666,6 @@ packages:
resolution: {integrity: sha512-9Be3ZoN4LmYR90tUoVu2te2BsbzHfhJyfEiAVfz7N5/zv+jduIfLrV2xdQXOHbaD6KgpGdO9PRPM1Y4Q9QkPkA==}
engines: {node: ^18.19.0 || >=20.5.0}
- express-rate-limit@8.5.2:
- resolution: {integrity: sha512-5Kb34ipNX694DH48vN9irak1Qx30nb0PLYHXfJgw4YEjiC3ZEmZJhwOp+VfiCYwFzvFTdB9QkArYS5kXa2cx2A==}
- engines: {node: '>= 16'}
- peerDependencies:
- express: '>= 4.11'
-
- express@5.2.1:
- resolution: {integrity: sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw==}
- engines: {node: '>= 18'}
-
fast-deep-equal@3.1.3:
resolution: {integrity: sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==}
@@ -2779,10 +2710,6 @@ packages:
resolution: {integrity: sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==}
engines: {node: '>=8'}
- finalhandler@2.1.1:
- resolution: {integrity: sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA==}
- engines: {node: '>= 18.0.0'}
-
find-up@5.0.0:
resolution: {integrity: sha512-78/PXT1wlLLDgTzDs7sjq9hzz0vXD+zn+7wypEe4fXQxCmdmqfGsEPQxmiCSQI3ajFV91bVSsvNtrJRiW6nGng==}
engines: {node: '>=10'}
@@ -2801,14 +2728,6 @@ packages:
resolution: {integrity: sha512-dKx12eRCVIzqCxFGplyFKJMPvLEWgmNtUrpTiJIR5u97zEhRG8ySrtboPHZXx7daLxQVrl643cTzbab2tkQjxg==}
engines: {node: '>= 0.4'}
- forwarded@0.2.0:
- resolution: {integrity: sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==}
- engines: {node: '>= 0.6'}
-
- fresh@2.0.0:
- resolution: {integrity: sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A==}
- engines: {node: '>= 0.8'}
-
fs-extra@11.3.4:
resolution: {integrity: sha512-CTXd6rk/M3/ULNQj8FBqBWHYBVYybQ3VPBw0xGKFe3tuH7ytT6ACnvzpIQ3UZtB8yvUKC2cXn1a+x+5EVQLovA==}
engines: {node: '>=14.14'}
@@ -2931,14 +2850,6 @@ packages:
hls.js@1.6.16:
resolution: {integrity: sha512-VSIRpLfRwlAAdGL4wiTucx2ScRipo0ed1FBatWkyt832jC4CReKstga6yIhYVwGu9LOBjuX9wzmRMeQdBJtzEA==}
- hono@4.12.27:
- resolution: {integrity: sha512-1yrb/+w6HWQJrUCLkJ2IF5jNIPvvFkblV5RNOYl6bV+OA6p9GLcMpHFFGTosSvHvcAUibuUukRqhlYI4z32C7Q==}
- engines: {node: '>=16.9.0'}
-
- http-errors@2.0.1:
- resolution: {integrity: sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==}
- engines: {node: '>= 0.8'}
-
human-signals@2.1.0:
resolution: {integrity: sha512-B4FFZ6q/T2jhhksgkbEW3HBvWIfDW85snkQgawt07S7J5QXTk6BkNV+0yAeZrM5QpMAdYlocGoljn0sJ/WQkFw==}
engines: {node: '>=10.17.0'}
@@ -2947,10 +2858,6 @@ packages:
resolution: {integrity: sha512-eKCa6bwnJhvxj14kZk5NCPc6Hb6BdsU9DZcOnmQKSnO1VKrfV0zCvtttPZUsBvjmNDn8rpcJfpwSYnHBjc95MQ==}
engines: {node: '>=18.18.0'}
- iconv-lite@0.7.3:
- resolution: {integrity: sha512-IKXpvIzjnC9XTAUbVBcMfGS0EPaIXtW6v+zr+RRp+hqULEpo0owZax6wyRwPOJbWbzjYspQwusTsfVr0ifh4uQ==}
- engines: {node: '>=0.10.0'}
-
ieee754@1.2.1:
resolution: {integrity: sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==}
@@ -2984,14 +2891,6 @@ packages:
resolution: {integrity: sha512-5Hh7Y1wQbvY5ooGgPbDaL5iYLAPzMTUrjMulskHLH6wnv/A+1q5rgEaiuqEjB+oxGXIVZs1FF+R/KPN3ZSQYYg==}
engines: {node: '>=12'}
- ip-address@10.2.0:
- resolution: {integrity: sha512-/+S6j4E9AHvW9SWMSEY9Xfy66O5PWvVEJ08O0y5JGyEKQpojb0K0GKpz/v5HJ/G0vi3D2sjGK78119oXZeE0qA==}
- engines: {node: '>= 12'}
-
- ipaddr.js@1.9.1:
- resolution: {integrity: sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==}
- engines: {node: '>= 0.10'}
-
is-array-buffer@3.0.5:
resolution: {integrity: sha512-DDfANUiiG2wC1qawP66qlTugJeL5HyzMpfr8lLK+jMQirGzNod0B12cFB/9q838Ru27sBwfw78/rdoU7RERz6A==}
engines: {node: '>= 0.4'}
@@ -3076,9 +2975,6 @@ packages:
resolution: {integrity: sha512-9UoipoxYmSk6Xy7QFgRv2HDyaysmgSG75TFQs6S+3pDM7ZhKTF/bskZV+0UlABHzKjNVhPjYCLfeZUEg1wXxig==}
engines: {node: ^12.20.0 || ^14.13.1 || >=16.0.0}
- is-promise@4.0.0:
- resolution: {integrity: sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ==}
-
is-regex@1.2.1:
resolution: {integrity: sha512-MjYsKHO5O7mCsmRGxWcLWheFqN9DJ/2TmngvjKXihe6efViPqc274+Fx/4fYj/r03+ESvBdTXK0V6tA3rgez1g==}
engines: {node: '>= 0.4'}
@@ -3169,9 +3065,6 @@ packages:
json-schema-traverse@1.0.0:
resolution: {integrity: sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==}
- json-schema-typed@8.0.2:
- resolution: {integrity: sha512-fQhoXdcvc3V28x7C7BMs4P5+kNlgUURe2jmUT1T//oBRMDrqy1QPelJimwZGo7Hg9VPV3EQV5Bnq4hbFy2vetA==}
-
json-stable-stringify-without-jsonify@1.0.1:
resolution: {integrity: sha512-Bdboy+l7tA3OGW6FjyFHWkP5LuByj1Tk33Ljyq0axyzdk9//JSi2u3fP1QSmd1KNwq6VOKYGlAu87CisVir6Pw==}
@@ -3324,17 +3217,9 @@ packages:
resolution: {integrity: sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==}
engines: {node: '>= 0.4'}
- media-typer@1.1.0:
- resolution: {integrity: sha512-aisnrDP4GNe06UcKFnV5bfMNPBUw4jsLGaWwWfnH3v02GnBuXX2MCVn5RbrWo0j3pczUilYblq7fQ7Nw2t5XKw==}
- engines: {node: '>= 0.8'}
-
memoize-one@5.2.1:
resolution: {integrity: sha512-zYiwtZUcYyXKo/np96AGZAckk+FWWsUdJ3cHGGmld7+AhvcWmQyGCYUh1hc4Q/pkOhb65dQR/pqCyK0cOaHz4Q==}
- merge-descriptors@2.0.0:
- resolution: {integrity: sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g==}
- engines: {node: '>=18'}
-
merge-stream@2.0.0:
resolution: {integrity: sha512-abv/qOcuPfk3URPfDzmZU1LKmuw8kT+0nIHvKrKgFrwifol/doWcdA4ZqsWQ8ENrFKkd67Mfpo/LovbIUsbt3w==}
@@ -3358,10 +3243,6 @@ packages:
resolution: {integrity: sha512-lc/aahn+t4/SWV/qcmumYjymLsWfN3ELhpmVuUFjgsORruuZPVSwAQryq+HHGvO/SI2KVX26bx+En+zhM8g8hQ==}
engines: {node: '>= 0.6'}
- mime-types@3.0.2:
- resolution: {integrity: sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A==}
- engines: {node: '>=18'}
-
mimic-fn@2.1.0:
resolution: {integrity: sha512-OqbOk5oEQeAZ8WXWydlu9HJjz9WVdEIvamMCcXmuqUYjTknH/sqsWvhQ3vgwKFRR1HpjvNBKQ37nbJgYzGqGcg==}
engines: {node: '>=6'}
@@ -3406,10 +3287,6 @@ packages:
resolution: {integrity: sha512-myRT3DiWPHqho5PrJaIRyaMv2kgYf0mUVgBNOYMuCH5Ki1yEiQaf/ZJuQ62nvpc44wL5WDbTX7yGJi1Neevw8w==}
engines: {node: '>= 0.6'}
- negotiator@1.0.0:
- resolution: {integrity: sha512-8Ofs/AUQh8MaEcrlq5xOX0CQ9ypTF5dl78mjlMNfOK08fzpgTHQRQPBxcPlEtIw0yRpws+Zo/3r+5WRby7u3Gg==}
- engines: {node: '>= 0.6'}
-
next@16.2.3:
resolution: {integrity: sha512-9V3zV4oZFza3PVev5/poB9g0dEafVcgNyQ8eTRop8GvxZjV2G15FC5ARuG1eFD42QgeYkzJBJzHghNP8Ad9xtA==}
engines: {node: '>=20.9.0'}
@@ -3485,17 +3362,10 @@ packages:
resolution: {integrity: sha512-gXah6aZrcUxjWg2zR2MwouP2eHlCBzdV4pygudehaKXSGW4v2AsRQUK+lwwXhii6KFZcunEnmSUoYp5CXibxtA==}
engines: {node: '>= 0.4'}
- on-finished@2.4.1:
- resolution: {integrity: sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==}
- engines: {node: '>= 0.8'}
-
on-headers@1.1.0:
resolution: {integrity: sha512-737ZY3yNnXy37FHkQxPzt4UZ2UWPWiCZWLvFZ4fu5cueciegX0zGPnrlY6bwRg4FdQOe9YU8MkmJwGhoMybl8A==}
engines: {node: '>= 0.8'}
- once@1.4.0:
- resolution: {integrity: sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==}
-
onetime@5.1.2:
resolution: {integrity: sha512-kbpaSSGJTWdAY5KPVeMOKXSrPtr8C8C7wodJbcsd51jRnmD+GZu8Y0VoU6Dm5Z4vWr0Ig/1NKuWRKf7j5aaYSg==}
engines: {node: '>=6'}
@@ -3531,10 +3401,6 @@ packages:
resolution: {integrity: sha512-TXfryirbmq34y8QBwgqCVLi+8oA3oWx2eAnSn62ITyEhEYaWRlVZ2DvMM9eZbMs/RfxPu/PK/aBLyGj4IrqMHw==}
engines: {node: '>=18'}
- parseurl@1.3.3:
- resolution: {integrity: sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==}
- engines: {node: '>= 0.8'}
-
path-exists@4.0.0:
resolution: {integrity: sha512-ak9Qy5Q7jYb2Wwcey5Fpvg2KoAc/ZIhLSLOSBmRmygPsGwkVVt0fZa0qrtMz+m6tJTAHfZQ8FnmB4MG4LWy7/w==}
engines: {node: '>=8'}
@@ -3556,9 +3422,6 @@ packages:
path-to-regexp@3.3.0:
resolution: {integrity: sha512-qyCH421YQPS2WFDxDjftfc1ZR5WKQzVzqsp4n9M2kQhVOo/ByahFoUNJfl58kOcEGfQ//7weFTDhm+ss8Ecxgw==}
- path-to-regexp@8.4.2:
- resolution: {integrity: sha512-qRcuIdP69NPm4qbACK+aDogI5CBDMi1jKe0ry5rSQJz8JVLsC7jV8XpiJjGRLLol3N+R5ihGYcrPLTno6pAdBA==}
-
picocolors@1.1.1:
resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==}
@@ -3607,18 +3470,10 @@ packages:
prop-types@15.8.1:
resolution: {integrity: sha512-oj87CgZICdulUohogVAR7AjlC0327U4el4L6eAvOqCeudMDVU0NThNaV+b9Df4dXgSP1gXMTnPdhfe/2qDH5cg==}
- proxy-addr@2.0.7:
- resolution: {integrity: sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==}
- engines: {node: '>= 0.10'}
-
punycode@2.3.1:
resolution: {integrity: sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg==}
engines: {node: '>=6'}
- qs@6.15.3:
- resolution: {integrity: sha512-O9gl3zCl5h5blw1KGUzQKhA5oUXSl8rwUIM5o0S3nCXMliSvy5Dzx7/DJcI+SwgICv+IneSZwhBh1oSyEHA71A==}
- engines: {node: '>=0.6'}
-
queue-microtask@1.2.3:
resolution: {integrity: sha512-NuaNSa6flKT5JaSYQzJok04JzTL1CA6aGhv5rfLW3PgqA+M2ChpZQnAC8h8i4ZFkBS8X5RqkDBHA7r4hej3K9A==}
@@ -3639,14 +3494,6 @@ packages:
resolution: {integrity: sha512-kA5WQoNVo4t9lNx2kQNFCxKeBl5IbbSNBl1M/tLkw9WCn+hxNBAW5Qh8gdhs63CJnhjJ2zQWFoqPJP2sK1AV5A==}
engines: {node: '>= 0.6'}
- range-parser@1.3.0:
- resolution: {integrity: sha512-hek2mFQpPuI4E1BBKrSto+BU3e3x4xuarsbiwr3+lf7p44juvFMV0XFWQAP3xUyqXA4RrXLIoaSUGbSt056ZMw==}
- engines: {node: '>= 0.6'}
-
- raw-body@3.0.2:
- resolution: {integrity: sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA==}
- engines: {node: '>= 0.10'}
-
rc@1.2.8:
resolution: {integrity: sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==}
hasBin: true
@@ -3766,10 +3613,6 @@ packages:
resolution: {integrity: sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw==}
engines: {iojs: '>=1.0.0', node: '>=0.10.0'}
- router@2.2.0:
- resolution: {integrity: sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ==}
- engines: {node: '>= 18'}
-
run-parallel@1.2.0:
resolution: {integrity: sha512-5l4VyZR86LZ/lDxZTR6jqL8AFE2S0IFLMP26AbjsLVADxHdhB/c0GUsH+y39UfCi3dzz8OlQuPmnaJOMoDHQBA==}
@@ -3788,9 +3631,6 @@ packages:
resolution: {integrity: sha512-x/+Cz4YrimQxQccJf5mKEbIa1NzeCRNI5Ecl/ekmlYaampdNLPalVyIcCZNNH3MvmqBugV5TMYZXv0ljslUlaw==}
engines: {node: '>= 0.4'}
- safer-buffer@2.1.2:
- resolution: {integrity: sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==}
-
scheduler@0.27.0:
resolution: {integrity: sha512-eNv+WrVbKu1f3vbYJT/xtiF5syA5HPIMtf9IgY/nKg0sWqzAUEvqY/xm7OcZc/qafLx/iO9FgOmeSAp4v5ti/Q==}
@@ -3803,17 +3643,9 @@ packages:
engines: {node: '>=10'}
hasBin: true
- send@1.2.1:
- resolution: {integrity: sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ==}
- engines: {node: '>= 18'}
-
serve-handler@6.1.7:
resolution: {integrity: sha512-CinAq1xWb0vR3twAv9evEU8cNWkXCb9kd5ePAHUKJBkOsUpR1wt/CvGdeca7vqumL1U5cSaeVQ6zZMxiJ3yWsg==}
- serve-static@2.2.1:
- resolution: {integrity: sha512-xRXBn0pPqQTVQiC8wyQrKs2MOlX24zQ0POGaj0kultvoOCstBQM5yvOhAVSUwOMjQtTvsPWoNCHfPGwaaQJhTw==}
- engines: {node: '>= 18'}
-
serve@14.2.6:
resolution: {integrity: sha512-QEjUSA+sD4Rotm1znR8s50YqA3kYpRGPmtd5GlFxbaL9n/FdUNbqMhxClqdditSk0LlZyA/dhud6XNRTOC9x2Q==}
engines: {node: '>= 14'}
@@ -3831,9 +3663,6 @@ packages:
resolution: {integrity: sha512-RJRdvCo6IAnPdsvP/7m6bsQqNnn1FCBX5ZNtFL98MmFF/4xAIJTIg1YbHW5DC2W5SKZanrC6i4HsJqlajw/dZw==}
engines: {node: '>= 0.4'}
- setprototypeof@1.2.0:
- resolution: {integrity: sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==}
-
sharp@0.34.5:
resolution: {integrity: sha512-Ou9I5Ft9WNcCbXrU9cMgPBcCK8LiwLqcbywW3t4oDV37n1pzpuNLsYiAV8eODnjbtQlSDwZ2cUEeQz4E54Hltg==}
engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0}
@@ -3862,10 +3691,6 @@ packages:
resolution: {integrity: sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==}
engines: {node: '>= 0.4'}
- side-channel@1.1.1:
- resolution: {integrity: sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ==}
- engines: {node: '>= 0.4'}
-
signal-exit@3.0.7:
resolution: {integrity: sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ==}
@@ -3886,10 +3711,6 @@ packages:
stable-hash@0.0.5:
resolution: {integrity: sha512-+L3ccpzibovGXFK+Ap/f8LOS0ahMrHTf3xu7mMLSpEGU0EO9ucaysSylKo9eRDFNhWve/y275iPmIZ4z39a9iA==}
- statuses@2.0.2:
- resolution: {integrity: sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==}
- engines: {node: '>= 0.8'}
-
stop-iteration-iterator@1.1.0:
resolution: {integrity: sha512-eLoXW/DHyl62zxY4SCaIgnRhuMr6ri4juEYARS8E6sCEqzKpOiE521Ucofdx+KnDZl5xmvGYaaKCk5FEOxJCoQ==}
engines: {node: '>= 0.4'}
@@ -4001,10 +3822,6 @@ packages:
resolution: {integrity: sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ==}
engines: {node: '>=8.0'}
- toidentifier@1.0.1:
- resolution: {integrity: sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==}
- engines: {node: '>=0.6'}
-
ts-api-utils@2.5.0:
resolution: {integrity: sha512-OJ/ibxhPlqrMM0UiNHJ/0CKQkoKF243/AEmplt3qpRgkW8VG7IfOS41h7V8TjITqdByHzrjcS/2si+y4lIh8NA==}
engines: {node: '>=18.12'}
@@ -4033,10 +3850,6 @@ packages:
resolution: {integrity: sha512-RAH822pAdBgcNMAfWnCBU3CFZcfZ/i1eZjwFU/dsLKumyuuP3niueg2UAukXYF0E2AAoc82ZSSf9J0WQBinzHA==}
engines: {node: '>=12.20'}
- type-is@2.1.0:
- resolution: {integrity: sha512-faYHw0anBbc/kWF3zFTEnxSFOAGUX9GFbOBthvDdLsIlEoWOFOtS0zgCiQYwIskL9iGXZL3kAXD8OoZ4GmMATA==}
- engines: {node: '>= 18'}
-
typed-array-buffer@1.0.3:
resolution: {integrity: sha512-nAYYwfY3qnzX30IkA6AQZjVbtK6duGontcQm1WSG1MD94YLqK0515GNApXkoxKOWMusVssAHWLh9SeaoefYFGw==}
engines: {node: '>= 0.4'}
@@ -4080,10 +3893,6 @@ packages:
resolution: {integrity: sha512-gptHNQghINnc/vTGIk0SOFGFNXw7JVrlRUtConJRlvaw6DuX0wO5Jeko9sWrMBhh+PsYAZ7oXAiOnf/UKogyiw==}
engines: {node: '>= 10.0.0'}
- unpipe@1.0.0:
- resolution: {integrity: sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==}
- engines: {node: '>= 0.8'}
-
unrs-resolver@1.11.1:
resolution: {integrity: sha512-bSjt9pjaEBnNiGgc9rUiHGKv5l4/TGzDmYw3RhnkJGtLhbnnA/5qJj7x3dNDCRx/PJxu774LlH8lCOlB4hEfKg==}
@@ -4165,9 +3974,6 @@ packages:
resolution: {integrity: sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ==}
engines: {node: '>=12'}
- wrappy@1.0.2:
- resolution: {integrity: sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==}
-
yallist@3.1.1:
resolution: {integrity: sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g==}
@@ -4183,11 +3989,6 @@ packages:
resolution: {integrity: sha512-CzhO+pFNo8ajLM2d2IW/R93ipy99LWjtwblvC1RsoSUMZgyLbYFr221TnSNT7GjGdYui6P459mw9JH/g/zW2ug==}
engines: {node: '>=18'}
- zod-to-json-schema@3.25.2:
- resolution: {integrity: sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA==}
- peerDependencies:
- zod: ^3.25.28 || ^4
-
zod-validation-error@4.0.2:
resolution: {integrity: sha512-Q6/nZLe6jxuU80qb/4uJ4t5v2VEZ44lzQjPDhYJNztRQ4wyWc6VF3D3Kb/fAuPetZQnhS3hnajCf9CsWesghLQ==}
engines: {node: '>=18.0.0'}
@@ -4637,10 +4438,6 @@ snapshots:
'@harperfast/extended-iterable@1.0.3': {}
- '@hono/node-server@1.19.14(hono@4.12.27)':
- dependencies:
- hono: 4.12.27
-
'@humanfs/core@0.19.2':
dependencies:
'@humanfs/types': 0.15.0
@@ -4794,27 +4591,24 @@ snapshots:
'@lmdb/lmdb-win32-x64@3.5.4':
optional: true
- '@modelcontextprotocol/sdk@1.29.0(zod@4.3.6)':
+ '@modelcontextprotocol/client@2.0.0':
dependencies:
- '@hono/node-server': 1.19.14(hono@4.12.27)
- ajv: 8.18.0
- ajv-formats: 3.0.1(ajv@8.18.0)
- content-type: 1.0.5
- cors: 2.8.6
+ '@modelcontextprotocol/core': 2.0.0
cross-spawn: 7.0.6
eventsource: 3.0.7
eventsource-parser: 3.1.0
- express: 5.2.1
- express-rate-limit: 8.5.2(express@5.2.1)
- hono: 4.12.27
jose: 6.2.3
- json-schema-typed: 8.0.2
pkce-challenge: 5.0.1
- raw-body: 3.0.2
zod: 4.3.6
- zod-to-json-schema: 3.25.2(zod@4.3.6)
- transitivePeerDependencies:
- - supports-color
+
+ '@modelcontextprotocol/core@2.0.0':
+ dependencies:
+ zod: 4.3.6
+
+ '@modelcontextprotocol/server@2.0.0':
+ dependencies:
+ '@modelcontextprotocol/core': 2.0.0
+ zod: 4.3.6
'@msgpackr-extract/msgpackr-extract-darwin-arm64@3.0.3':
optional: true
@@ -5975,21 +5769,12 @@ snapshots:
'@zeit/schemas@2.36.0': {}
- accepts@2.0.0:
- dependencies:
- mime-types: 3.0.2
- negotiator: 1.0.0
-
acorn-jsx@5.3.2(acorn@8.16.0):
dependencies:
acorn: 8.16.0
acorn@8.16.0: {}
- ajv-formats@3.0.1(ajv@8.18.0):
- optionalDependencies:
- ajv: 8.18.0
-
ajv@6.15.0:
dependencies:
fast-deep-equal: 3.1.3
@@ -6117,20 +5902,6 @@ snapshots:
baseline-browser-mapping@2.10.23: {}
- body-parser@2.3.0:
- dependencies:
- bytes: 3.1.2
- content-type: 2.0.0
- debug: 4.4.3
- http-errors: 2.0.1
- iconv-lite: 0.7.3
- on-finished: 2.4.1
- qs: 6.15.3
- raw-body: 3.0.2
- type-is: 2.1.0
- transitivePeerDependencies:
- - supports-color
-
bowser@2.14.1: {}
boxen@7.0.0:
@@ -6262,23 +6033,8 @@ snapshots:
content-disposition@0.5.2: {}
- content-disposition@1.1.0: {}
-
- content-type@1.0.5: {}
-
- content-type@2.0.0: {}
-
convert-source-map@2.0.0: {}
- cookie-signature@1.2.2: {}
-
- cookie@0.7.2: {}
-
- cors@2.8.6:
- dependencies:
- object-assign: 4.1.1
- vary: 1.1.2
-
cross-spawn@7.0.6:
dependencies:
path-key: 3.1.1
@@ -6377,8 +6133,6 @@ snapshots:
has-property-descriptors: 1.0.2
object-keys: 1.1.1
- depd@2.0.0: {}
-
detect-libc@2.1.2: {}
detect-node-es@1.1.0: {}
@@ -6400,16 +6154,12 @@ snapshots:
eastasianwidth@0.2.0: {}
- ee-first@1.1.1: {}
-
electron-to-chromium@1.5.344: {}
emoji-regex@8.0.0: {}
emoji-regex@9.2.2: {}
- encodeurl@2.0.0: {}
-
enhanced-resolve@5.21.0:
dependencies:
graceful-fs: 4.2.11
@@ -6547,8 +6297,6 @@ snapshots:
escalade@3.2.0: {}
- escape-html@1.0.3: {}
-
escape-string-regexp@4.0.0: {}
escape-string-regexp@5.0.0: {}
@@ -6805,8 +6553,6 @@ snapshots:
esutils@2.0.3: {}
- etag@1.8.1: {}
-
eventemitter3@4.0.7: {}
events@3.3.0: {}
@@ -6844,44 +6590,6 @@ snapshots:
strip-final-newline: 4.0.0
yoctocolors: 2.1.2
- express-rate-limit@8.5.2(express@5.2.1):
- dependencies:
- express: 5.2.1
- ip-address: 10.2.0
-
- express@5.2.1:
- dependencies:
- accepts: 2.0.0
- body-parser: 2.3.0
- content-disposition: 1.1.0
- content-type: 1.0.5
- cookie: 0.7.2
- cookie-signature: 1.2.2
- debug: 4.4.3
- depd: 2.0.0
- encodeurl: 2.0.0
- escape-html: 1.0.3
- etag: 1.8.1
- finalhandler: 2.1.1
- fresh: 2.0.0
- http-errors: 2.0.1
- merge-descriptors: 2.0.0
- mime-types: 3.0.2
- on-finished: 2.4.1
- once: 1.4.0
- parseurl: 1.3.3
- proxy-addr: 2.0.7
- qs: 6.15.3
- range-parser: 1.3.0
- router: 2.2.0
- send: 1.2.1
- serve-static: 2.2.1
- statuses: 2.0.2
- type-is: 2.1.0
- vary: 1.1.2
- transitivePeerDependencies:
- - supports-color
-
fast-deep-equal@3.1.3: {}
fast-equals@5.4.0: {}
@@ -6920,17 +6628,6 @@ snapshots:
dependencies:
to-regex-range: 5.0.1
- finalhandler@2.1.1:
- dependencies:
- debug: 4.4.3
- encodeurl: 2.0.0
- escape-html: 1.0.3
- on-finished: 2.4.1
- parseurl: 1.3.3
- statuses: 2.0.2
- transitivePeerDependencies:
- - supports-color
-
find-up@5.0.0:
dependencies:
locate-path: 6.0.0
@@ -6949,10 +6646,6 @@ snapshots:
dependencies:
is-callable: 1.2.7
- forwarded@0.2.0: {}
-
- fresh@2.0.0: {}
-
fs-extra@11.3.4:
dependencies:
graceful-fs: 4.2.11
@@ -7070,24 +6763,10 @@ snapshots:
hls.js@1.6.16: {}
- hono@4.12.27: {}
-
- http-errors@2.0.1:
- dependencies:
- depd: 2.0.0
- inherits: 2.0.4
- setprototypeof: 1.2.0
- statuses: 2.0.2
- toidentifier: 1.0.1
-
human-signals@2.1.0: {}
human-signals@8.0.1: {}
- iconv-lite@0.7.3:
- dependencies:
- safer-buffer: 2.1.2
-
ieee754@1.2.1: {}
ignore@5.3.2: {}
@@ -7113,10 +6792,6 @@ snapshots:
internmap@2.0.3: {}
- ip-address@10.2.0: {}
-
- ipaddr.js@1.9.1: {}
-
is-array-buffer@3.0.5:
dependencies:
call-bind: 1.0.9
@@ -7198,8 +6873,6 @@ snapshots:
is-port-reachable@4.0.0: {}
- is-promise@4.0.0: {}
-
is-regex@1.2.1:
dependencies:
call-bound: 1.0.4
@@ -7280,8 +6953,6 @@ snapshots:
json-schema-traverse@1.0.0: {}
- json-schema-typed@8.0.2: {}
-
json-stable-stringify-without-jsonify@1.0.1: {}
json5@1.0.2:
@@ -7416,12 +7087,8 @@ snapshots:
math-intrinsics@1.1.0: {}
- media-typer@1.1.0: {}
-
memoize-one@5.2.1: {}
- merge-descriptors@2.0.0: {}
-
merge-stream@2.0.0: {}
merge2@1.4.1: {}
@@ -7439,10 +7106,6 @@ snapshots:
dependencies:
mime-db: 1.33.0
- mime-types@3.0.2:
- dependencies:
- mime-db: 1.54.0
-
mimic-fn@2.1.0: {}
minimatch@10.2.5:
@@ -7483,8 +7146,6 @@ snapshots:
negotiator@0.6.4: {}
- negotiator@1.0.0: {}
-
next@16.2.3(@babel/core@7.29.0)(@playwright/test@1.59.1)(react-dom@19.2.4(react@19.2.4))(react@19.2.4):
dependencies:
'@next/env': 16.2.3
@@ -7576,16 +7237,8 @@ snapshots:
define-properties: 1.2.1
es-object-atoms: 1.1.1
- on-finished@2.4.1:
- dependencies:
- ee-first: 1.1.1
-
on-headers@1.1.0: {}
- once@1.4.0:
- dependencies:
- wrappy: 1.0.2
-
onetime@5.1.2:
dependencies:
mimic-fn: 2.1.0
@@ -7625,8 +7278,6 @@ snapshots:
parse-ms@4.0.0: {}
- parseurl@1.3.3: {}
-
path-exists@4.0.0: {}
path-is-inside@1.0.2: {}
@@ -7639,8 +7290,6 @@ snapshots:
path-to-regexp@3.3.0: {}
- path-to-regexp@8.4.2: {}
-
picocolors@1.1.1: {}
picomatch@2.3.2: {}
@@ -7683,18 +7332,8 @@ snapshots:
object-assign: 4.1.1
react-is: 16.13.1
- proxy-addr@2.0.7:
- dependencies:
- forwarded: 0.2.0
- ipaddr.js: 1.9.1
-
punycode@2.3.1: {}
- qs@6.15.3:
- dependencies:
- es-define-property: 1.0.1
- side-channel: 1.1.1
-
queue-microtask@1.2.3: {}
radix-ui@1.6.0(@types/react-dom@19.2.3(@types/react@19.2.14))(@types/react@19.2.14)(react-dom@19.2.4(react@19.2.4))(react@19.2.4):
@@ -7762,15 +7401,6 @@ snapshots:
range-parser@1.2.0: {}
- range-parser@1.3.0: {}
-
- raw-body@3.0.2:
- dependencies:
- bytes: 3.1.2
- http-errors: 2.0.1
- iconv-lite: 0.7.3
- unpipe: 1.0.0
-
rc@1.2.8:
dependencies:
deep-extend: 0.6.0
@@ -7913,16 +7543,6 @@ snapshots:
reusify@1.1.0: {}
- router@2.2.0:
- dependencies:
- debug: 4.4.3
- depd: 2.0.0
- is-promise: 4.0.0
- parseurl: 1.3.3
- path-to-regexp: 8.4.2
- transitivePeerDependencies:
- - supports-color
-
run-parallel@1.2.0:
dependencies:
queue-microtask: 1.2.3
@@ -7948,30 +7568,12 @@ snapshots:
es-errors: 1.3.0
is-regex: 1.2.1
- safer-buffer@2.1.2: {}
-
scheduler@0.27.0: {}
semver@6.3.1: {}
semver@7.7.4: {}
- send@1.2.1:
- dependencies:
- debug: 4.4.3
- encodeurl: 2.0.0
- escape-html: 1.0.3
- etag: 1.8.1
- fresh: 2.0.0
- http-errors: 2.0.1
- mime-types: 3.0.2
- ms: 2.1.3
- on-finished: 2.4.1
- range-parser: 1.3.0
- statuses: 2.0.2
- transitivePeerDependencies:
- - supports-color
-
serve-handler@6.1.7:
dependencies:
bytes: 3.0.0
@@ -7982,15 +7584,6 @@ snapshots:
path-to-regexp: 3.3.0
range-parser: 1.2.0
- serve-static@2.2.1:
- dependencies:
- encodeurl: 2.0.0
- escape-html: 1.0.3
- parseurl: 1.3.3
- send: 1.2.1
- transitivePeerDependencies:
- - supports-color
-
serve@14.2.6:
dependencies:
'@zeit/schemas': 2.36.0
@@ -8029,8 +7622,6 @@ snapshots:
es-errors: 1.3.0
es-object-atoms: 1.1.1
- setprototypeof@1.2.0: {}
-
sharp@0.34.5:
dependencies:
'@img/colour': 1.1.0
@@ -8097,14 +7688,6 @@ snapshots:
side-channel-map: 1.0.1
side-channel-weakmap: 1.0.2
- side-channel@1.1.1:
- dependencies:
- es-errors: 1.3.0
- object-inspect: 1.13.4
- side-channel-list: 1.0.1
- side-channel-map: 1.0.1
- side-channel-weakmap: 1.0.2
-
signal-exit@3.0.7: {}
signal-exit@4.1.0: {}
@@ -8118,8 +7701,6 @@ snapshots:
stable-hash@0.0.5: {}
- statuses@2.0.2: {}
-
stop-iteration-iterator@1.1.0:
dependencies:
es-errors: 1.3.0
@@ -8244,8 +7825,6 @@ snapshots:
dependencies:
is-number: 7.0.0
- toidentifier@1.0.1: {}
-
ts-api-utils@2.5.0(typescript@5.9.3):
dependencies:
typescript: 5.9.3
@@ -8274,12 +7853,6 @@ snapshots:
type-fest@2.19.0: {}
- type-is@2.1.0:
- dependencies:
- content-type: 2.0.0
- media-typer: 1.1.0
- mime-types: 3.0.2
-
typed-array-buffer@1.0.3:
dependencies:
call-bound: 1.0.4
@@ -8339,8 +7912,6 @@ snapshots:
universalify@2.0.1: {}
- unpipe@1.0.0: {}
-
unrs-resolver@1.11.1:
dependencies:
napi-postinstall: 0.3.4
@@ -8475,8 +8046,6 @@ snapshots:
string-width: 5.1.2
strip-ansi: 7.2.0
- wrappy@1.0.2: {}
-
yallist@3.1.1: {}
yocto-queue@0.1.0: {}
@@ -8485,10 +8054,6 @@ snapshots:
yoctocolors@2.1.2: {}
- zod-to-json-schema@3.25.2(zod@4.3.6):
- dependencies:
- zod: 4.3.6
-
zod-validation-error@4.0.2(zod@4.3.6):
dependencies:
zod: 4.3.6
diff --git a/scripts/build-sortformer.sh b/scripts/build-sortformer.sh
@@ -0,0 +1,164 @@
+#!/usr/bin/env bash
+# Build the Sortformer diarization engine into .diarize/sortformer/.
+#
+# WHAT THIS BUILDS, AND WHY SO LITTLE OF IT: openresearchtools/engine is a whole
+# local-AI runtime (Rust CLI, in-process llama bridge, PDF/VLM modules, a cluster
+# controller). We want exactly one thing from it — the native ggml implementation
+# of NVIDIA's streaming Sortformer diarizer in `diarize/addons/overlay`. That code
+# links against `ggml` ALONE: no llama, no whisper, no Rust. So this script stages
+# just `ggml` + `tools/realtime` + our driver and builds those, which is why it
+# needs no part of upstream's build system (whose scripts are PowerShell/CUDA and
+# would otherwise be the blocker on Linux).
+#
+# The upstream commit is PINNED. This is a research repo with no releases, and an
+# unpinned clone would silently change what produced every sidecar on disk.
+#
+# Usage:
+# scripts/build-sortformer.sh # Vulkan (default)
+# SORTFORMER_BACKEND=cpu scripts/build-sortformer.sh
+# SORTFORMER_JOBS=2 scripts/build-sortformer.sh
+#
+# Requires: git, cmake, a C++17 compiler, and for the Vulkan build the Vulkan
+# loader + headers and `glslc` (Arch: vulkan-headers shaderc).
+
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+OUT_DIR="${SORTFORMER_DIR:-$REPO_ROOT/.diarize/sortformer}"
+WORK_DIR="${SORTFORMER_BUILD_DIR:-$REPO_ROOT/.diarize/.sortformer-build}"
+
+# Pinned 2026-08-10. Bumping this changes model execution, so re-run the
+# reproducibility check in .diarize/README.md before trusting a new pin.
+ENGINE_REPO="https://github.com/openresearchtools/engine"
+ENGINE_COMMIT="8bb4928c429b76754c7c7cfbdec0c27419bef684"
+
+MODEL_URL="https://huggingface.co/openresearchtools/diar_streaming_sortformer_4spk-v2.1-gguf/resolve/main/diar_streaming_sortformer_4spk-v2.1.gguf"
+MODEL_NAME="diar_streaming_sortformer_4spk-v2.1.gguf"
+
+BACKEND="${SORTFORMER_BACKEND:-vulkan}"
+JOBS="${SORTFORMER_JOBS:-$(nproc 2>/dev/null || echo 4)}"
+
+say() { printf '\033[1m==>\033[0m %s\n' "$*"; }
+
+for tool in git cmake; do
+ command -v "$tool" >/dev/null || { echo "missing required tool: $tool" >&2; exit 1; }
+done
+
+case "$BACKEND" in
+ vulkan)
+ command -v glslc >/dev/null || {
+ echo "glslc not found — the Vulkan build compiles shaders with it." >&2
+ echo "Install shaderc, or build CPU-only with SORTFORMER_BACKEND=cpu." >&2
+ exit 1
+ }
+ CMAKE_BACKEND_FLAGS=(-DGGML_VULKAN=ON)
+ ;;
+ cpu)
+ CMAKE_BACKEND_FLAGS=()
+ ;;
+ *)
+ echo "SORTFORMER_BACKEND must be 'vulkan' or 'cpu' (got '$BACKEND')" >&2
+ exit 1
+ ;;
+esac
+
+mkdir -p "$WORK_DIR"
+
+# 1. Upstream source, at the pin.
+SRC="$WORK_DIR/engine"
+if [ -d "$SRC/.git" ] && [ "$(git -C "$SRC" rev-parse HEAD 2>/dev/null)" = "$ENGINE_COMMIT" ]; then
+ say "engine source already at $ENGINE_COMMIT"
+else
+ say "fetching engine @ ${ENGINE_COMMIT:0:12}"
+ rm -rf "$SRC"
+ git init -q "$SRC"
+ git -C "$SRC" remote add origin "$ENGINE_REPO"
+ # Blobless partial clone: the full history is ~200 MB and we need one commit.
+ git -C "$SRC" fetch -q --depth 1 --filter=blob:none origin "$ENGINE_COMMIT"
+ git -C "$SRC" checkout -q FETCH_HEAD
+fi
+
+# 2. Stage the minimal tree. Layout matters: tools/realtime/CMakeLists.txt refers
+# to ../../ggml/include, so ggml must sit beside tools/, exactly as upstream.
+#
+# ggml is staged ONCE per pin and then left alone. Re-copying it would bump
+# every source mtime and force a full rebuild of the Vulkan shaders (~10 min)
+# every time only our driver changed. `cp -u` on the small realtime tree keeps
+# edits cheap while still picking up a new pin.
+STAGE="$WORK_DIR/stage"
+mkdir -p "$STAGE/tools"
+if [ -f "$STAGE/.ggml-commit" ] && [ "$(cat "$STAGE/.ggml-commit")" = "$ENGINE_COMMIT" ]; then
+ say "ggml already staged for this pin"
+else
+ say "staging ggml"
+ rm -rf "$STAGE/ggml" "$STAGE/build"
+ cp -r "$SRC/third_party/llama.cpp/ggml" "$STAGE/ggml"
+ echo "$ENGINE_COMMIT" > "$STAGE/.ggml-commit"
+fi
+say "staging tools/realtime + driver"
+mkdir -p "$STAGE/tools/realtime"
+cp -ru "$SRC/diarize/addons/overlay/llama.cpp/tools/realtime/." "$STAGE/tools/realtime/"
+cp -u "$REPO_ROOT/scripts/sortformer/diarize-file.cpp" "$STAGE/tools/realtime/diarize-file.cpp"
+
+cat > "$STAGE/CMakeLists.txt" <<'EOF'
+cmake_minimum_required(VERSION 3.14)
+project(sortformer-diar C CXX)
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_POSITION_INDEPENDENT_CODE ON)
+add_subdirectory(ggml)
+add_subdirectory(tools/realtime)
+EOF
+
+# Upstream's CMakeLists builds their parity tool; we append our driver rather than
+# replacing it, so the staged tree stays a faithful copy plus one target. Guarded
+# because the staged tree now persists across runs.
+if ! grep -q 'add_executable(diarize-file' "$STAGE/tools/realtime/CMakeLists.txt"; then
+ cat >> "$STAGE/tools/realtime/CMakeLists.txt" <<'EOF'
+
+add_executable(diarize-file diarize-file.cpp)
+target_compile_features(diarize-file PRIVATE cxx_std_17)
+target_link_libraries(diarize-file PRIVATE ${TARGET_LIB})
+EOF
+fi
+
+# 3. Build.
+say "building ($BACKEND, -j$JOBS) — the Vulkan shader compile is the slow part"
+cmake -S "$STAGE" -B "$STAGE/build" \
+ -DCMAKE_BUILD_TYPE=Release \
+ -DGGML_NATIVE=ON \
+ "${CMAKE_BACKEND_FLAGS[@]}" >/dev/null
+cmake --build "$STAGE/build" -j"$JOBS" --target diarize-file
+
+# 4. Install the binary beside the ggml shared objects it loads, and point its
+# RPATH at $ORIGIN so it runs without LD_LIBRARY_PATH from any cwd.
+say "installing to $OUT_DIR"
+mkdir -p "$OUT_DIR"
+cp "$STAGE/build/tools/realtime/diarize-file" "$OUT_DIR/diarize-file"
+find "$STAGE/build" -name 'libggml*.so*' -exec cp -a {} "$OUT_DIR/" \;
+if command -v patchelf >/dev/null; then
+ patchelf --set-rpath '$ORIGIN' "$OUT_DIR/diarize-file"
+else
+ echo "note: patchelf not found; scripts/diarize-sortformer.mjs sets LD_LIBRARY_PATH itself" >&2
+fi
+
+# 5. Model.
+if [ -f "$OUT_DIR/$MODEL_NAME" ]; then
+ say "model already present"
+else
+ say "downloading model (~471 MB)"
+ curl -fSL --progress-bar -o "$OUT_DIR/$MODEL_NAME.part" "$MODEL_URL"
+ mv "$OUT_DIR/$MODEL_NAME.part" "$OUT_DIR/$MODEL_NAME"
+fi
+
+say "done"
+echo
+echo " binary: $OUT_DIR/diarize-file"
+echo " model: $OUT_DIR/$MODEL_NAME"
+echo
+echo "Wire it up in settings.json:"
+echo " \"diarization\": {"
+echo " \"engine\": \"sortformer\","
+echo " \"sortformerBin\": \"$OUT_DIR/diarize-file\","
+echo " \"sortformerModel\": \"$OUT_DIR/$MODEL_NAME\","
+echo " \"backend\": \"$BACKEND\""
+echo " }"
diff --git a/scripts/diarize-sortformer.mjs b/scripts/diarize-sortformer.mjs
@@ -0,0 +1,169 @@
+#!/usr/bin/env node
+// Sortformer diarization engine for scripts/diarize.mjs.
+//
+// The exact counterpart of scripts/diarize-sherpa.py, and deliberately the same
+// shape: decode audio, run the model, print raw turns as JSON on stdout. All
+// policy (filenames, provenance shape, atomic writes) lives in the .mjs wrapper.
+//
+// stdout is JSON ONLY. Progress goes to stderr so the wrapper can stream it.
+//
+// WHAT THIS ADDS OVER THE BINARY: audio decoding. scripts/sortformer/
+// diarize-file.cpp only speaks 16 kHz mono PCM, and the corpus holds mp3, m4a,
+// webm and whole video containers. ffmpeg does that here, exactly as the sherpa
+// engine shells out to ffmpeg for the same reason.
+//
+// NO TEMPORARY WAV, ON PURPOSE. A decoded 8-hour VOD is ~900 MB, this box's disk
+// runs at 98%, and /tmp is frequently a tmpfs — writing one would be the largest
+// single risk in this lane. ffmpeg's raw output is piped straight into the
+// engine, which consumes it in blocks, so neither process ever holds the file.
+//
+// Usage (invoked by diarize.mjs; runnable directly for debugging):
+// diarize-sortformer.mjs --bin <diarize-file> --model <gguf> [options] <audio>
+//
+// --bin <path> engine binary [required]
+// --model <path> sortformer .gguf [required]
+// --backend <b> vulkan | cpu (default vulkan)
+// --threads <n> CPU backend threads only (default 6 — see below)
+// --ffmpeg <path> ffmpeg binary (FFMPEG_BIN, "ffmpeg")
+
+import { spawn } from "node:child_process";
+import path from "node:path";
+
+const SAMPLE_RATE = 16000; // what the model's frontend expects; not a knob
+
+function fail(msg, code = 2) {
+ process.stderr.write(`diarize-sortformer: ${msg}\n`);
+ process.exit(code);
+}
+
+const argv = process.argv.slice(2);
+if (argv.includes("-h") || argv.includes("--help")) {
+ process.stdout.write(
+ "diarize-sortformer.mjs --bin <diarize-file> --model <gguf> [options] <audio>\n",
+ );
+ process.exit(0);
+}
+
+function arg(flag, fallback) {
+ const i = argv.indexOf(flag);
+ return i < 0 || i === argv.length - 1 ? fallback : argv[i + 1];
+}
+
+const positional = [];
+for (let i = 0; i < argv.length; i++) {
+ if (argv[i].startsWith("-")) {
+ i++; // every flag here takes a value
+ continue;
+ }
+ positional.push(argv[i]);
+}
+
+const audio = positional[0];
+if (!audio) fail("missing <audioFile>");
+
+const bin = arg("--bin", process.env.SORTFORMER_BIN ?? "");
+const model = arg("--model", process.env.SORTFORMER_MODEL ?? "");
+const backend = String(arg("--backend", "vulkan")).toLowerCase();
+// 6, not 4 and not 8: measured on this box at 1305 s/audio-hour against 1514 for
+// ggml's default of 4, and 1401 for 8 — which is WORSE than 4 through
+// oversubscription on a 4c/8t part. Ignored by the Vulkan backend.
+const threads = Number(arg("--threads", "6"));
+const ffmpeg = arg("--ffmpeg", process.env.FFMPEG_BIN ?? "ffmpeg");
+
+if (!bin) fail("missing --bin (or SORTFORMER_BIN)");
+if (!model) fail("missing --model (or SORTFORMER_MODEL)");
+
+// Settings speak "vulkan"/"cpu"; ggml speaks device names. One translation, here,
+// so nothing upstream of this file has to know ggml's spelling.
+const ggmlBackend =
+ backend === "cpu" ? "CPU" : backend === "vulkan" ? "Vulkan0" : null;
+if (!ggmlBackend) fail(`--backend must be 'vulkan' or 'cpu' (got '${backend}')`);
+
+// ffmpeg: anything in -> raw s16le mono 16k on stdout. `-vn` because a source
+// container carries a video stream that would otherwise be negotiated first.
+const dec = spawn(
+ ffmpeg,
+ [
+ "-nostdin",
+ "-v", "error",
+ "-i", audio,
+ "-vn",
+ "-ac", "1",
+ "-ar", String(SAMPLE_RATE),
+ "-f", "s16le",
+ "pipe:1",
+ ],
+ { stdio: ["ignore", "pipe", "inherit"] },
+);
+
+const engine = spawn(
+ bin,
+ [
+ "--model", model,
+ "--audio-raw", "-",
+ "--sample-rate", String(SAMPLE_RATE),
+ "--backend", ggmlBackend,
+ "--threads", String(threads),
+ ],
+ {
+ stdio: ["pipe", "pipe", "inherit"],
+ env: {
+ ...process.env,
+ // The build sets RPATH=$ORIGIN when patchelf is available; this covers the
+ // case where it was not, so the binary finds its ggml .so files either way.
+ LD_LIBRARY_PATH: [path.dirname(path.resolve(bin)), process.env.LD_LIBRARY_PATH]
+ .filter(Boolean)
+ .join(":"),
+ },
+ },
+);
+
+dec.on("error", (err) => fail(`failed to run ${ffmpeg}: ${err.message}`, 3));
+engine.on("error", (err) => fail(`failed to run ${bin}: ${err.message}`, 3));
+
+dec.stdout.pipe(engine.stdin);
+// The engine exiting early (bad model, unusable backend) closes the pipe under
+// ffmpeg. That is an expected race, not a crash to report.
+engine.stdin.on("error", () => dec.kill("SIGTERM"));
+
+let stdout = "";
+engine.stdout.setEncoding("utf8");
+engine.stdout.on("data", (d) => {
+ stdout += d;
+});
+
+let decCode = null;
+let engineCode = null;
+
+function finish() {
+ // The ENGINE's verdict is what gates everything: it is the one that can fail
+ // for a reason worth reporting, and waiting on ffmpeg first is what used to
+ // hang here forever.
+ if (engineCode === null) return;
+ if (engineCode !== 0) fail(`engine exited ${engineCode}`, engineCode || 1);
+ // Only on the success path does ffmpeg's exit matter — a non-zero decode with
+ // a healthy engine means the audio was truncated, which must not pass
+ // silently as a short diarization.
+ if (decCode === null) return;
+ if (decCode !== 0) fail(`ffmpeg exited ${decCode} — audio may be truncated`, decCode || 1);
+ if (!stdout.trim()) fail("engine produced no output");
+ process.stdout.write(stdout.endsWith("\n") ? stdout : stdout + "\n");
+}
+
+dec.on("close", (code) => {
+ decCode = code ?? 0;
+ finish();
+});
+
+engine.on("close", (code) => {
+ engineCode = code ?? 0;
+ // THE ENGINE IS THE ONLY READER OF ffmpeg's OUTPUT. Once it is gone — and it
+ // can go abruptly: a Vulkan device loss aborts it mid-stream — ffmpeg is
+ // writing into a pipe nobody will ever drain, so it blocks on a full pipe and
+ // never exits, and this process never exits either because it is still
+ // waiting for that close. Three of these accumulated during development, each
+ // holding a decoder resident for hours. The engine's exit is therefore the
+ // signal to tear the decoder down rather than wait for an EOF that cannot come.
+ if (dec.exitCode === null && !dec.killed) dec.kill("SIGKILL");
+ finish();
+});
diff --git a/scripts/diarize.mjs b/scripts/diarize.mjs
@@ -8,11 +8,24 @@
// scripts/diarize-sherpa.py (ONNX, CPU); pyannote would be a drop-in
// replacement for the --engine command.
//
-// WHY CPU AND NOT THE GPU: parakeet.cpp is built GGML_VULKAN=ON so ASR really
-// does run on the RX 6600 XT, but no diarization model has been ported to ggml.
-// They ship as PyTorch (CUDA/ROCm) or ONNX (CUDA/ROCm/MIGraphX/OpenVINO/
-// DirectML), and neither runtime has a Vulkan compute path on Linux. That is an
-// ecosystem gap, not a hardware limit.
+// TWO ENGINES, PICKED BY --engine-kind:
+// sherpa-onnx (default) scripts/diarize-sherpa.py — ONNX segmentation +
+// embeddings, agglomerative clustering, CPU only.
+// sortformer scripts/diarize-sortformer.mjs — NVIDIA's streaming
+// Sortformer as ggml, end-to-end, Vulkan or CPU.
+//
+// THE GPU NOTE THIS FILE USED TO CARRY IS NOW OUT OF DATE, and the correction is
+// worth keeping: it said no diarization model had been ported to ggml, so the
+// RX 6600 XT could not run one while parakeet.cpp happily did ASR on it. That was
+// true of PyTorch and ONNX runtimes, and it is no longer true — openresearchtools
+// ported Sortformer to ggml, which inherits the same Vulkan backend parakeet
+// uses. Measured on this box: 894 s/audio-hour on Vulkan against 1305 for the
+// same engine on tuned CPU threads, on ONE core instead of 2.2.
+//
+// It is still not the fastest lane: sherpa-onnx does the same file in
+// 492 s/audio-hour across ~4 cores. Sortformer is chosen for QUALITY (4 speakers
+// where sherpa splits into 13, and 4 where sherpa splits into 35), and the GPU is
+// chosen because it is the cheapest way to run it.
//
// Dual-use, by design:
// * In the app: controller/diarizeOne.ts runs this with cwd == the video dir
@@ -29,7 +42,11 @@
// Options (env fallback in parens):
// --output <f> write the record here (default: stdout)
// --video-id <id> recorded in the sidecar (default: basename of cwd)
+// --engine-kind <k> sherpa-onnx | sortformer (DIARIZE_ENGINE_KIND, sherpa-onnx)
// --engine <cmd> engine command (DIARIZE_ENGINE_CMD)
+// --sortformer-bin <p> engine binary (SORTFORMER_BIN) [sortformer]
+// --sortformer-model <p> .gguf model (SORTFORMER_MODEL) [sortformer]
+// --backend <b> vulkan | cpu (default vulkan) [sortformer]
// --python <path> python for the default engine (DIARIZE_PYTHON, "python3")
// --seg <path> segmentation model (DIARIZE_SEG_MODEL) [required]
// --emb <path> speaker-embedding model (DIARIZE_EMB_MODEL) [required]
@@ -110,6 +127,24 @@ const windowAfterMinutes = arg(
);
const engineCmd = arg("--engine", process.env.DIARIZE_ENGINE_CMD ?? "");
+const engineKind = arg(
+ "--engine-kind",
+ process.env.DIARIZE_ENGINE_KIND ?? "sherpa-onnx",
+);
+const sortformerBin = arg("--sortformer-bin", process.env.SORTFORMER_BIN ?? "");
+const sortformerModel = arg(
+ "--sortformer-model",
+ process.env.SORTFORMER_MODEL ?? "",
+);
+const backend = arg("--backend", "vulkan");
+
+if (!engineCmd && engineKind !== "sherpa-onnx" && engineKind !== "sortformer") {
+ fail(`--engine-kind must be 'sherpa-onnx' or 'sortformer' (got '${engineKind}')`);
+}
+
+// `--engine` still wins over `--engine-kind`. It is the raw escape hatch (and
+// what the e2e fake engine uses), so a kind must never quietly override it.
+const usingSortformer = !engineCmd && engineKind === "sortformer";
let cmd;
let args;
@@ -117,6 +152,21 @@ if (engineCmd) {
const parts = engineCmd.split(" ").filter(Boolean);
cmd = parts[0];
args = [...parts.slice(1), audio];
+} else if (usingSortformer) {
+ if (!sortformerBin) fail("missing --sortformer-bin (or SORTFORMER_BIN)");
+ if (!sortformerModel) fail("missing --sortformer-model (or SORTFORMER_MODEL)");
+ // process.execPath, not a bare "node": the app may run under a node that is not
+ // the one on PATH, and the engine has to be the same runtime as its wrapper.
+ cmd = process.execPath;
+ args = [
+ path.join(import.meta.dirname, "diarize-sortformer.mjs"),
+ "--bin", sortformerBin,
+ "--model", sortformerModel,
+ "--backend", backend,
+ "--threads", String(threads),
+ "--ffmpeg", ffmpeg,
+ audio,
+ ];
} else {
if (!seg) fail("missing --seg (or DIARIZE_SEG_MODEL)");
if (!emb) fail("missing --emb (or DIARIZE_EMB_MODEL)");
@@ -177,15 +227,30 @@ child.on("close", async (code) => {
: {}),
speakers: new Set(turns.map((t) => t.speaker)).size,
turns,
- engine: {
- engine: engineCmd ? path.basename(cmd) : "sherpa-onnx",
- // basename only: absolute paths are machine-specific and the corpus is
- // rsynced between shards, so a full path would make records non-portable.
- ...(seg ? { segmentationModel: path.basename(seg) } : {}),
- ...(emb ? { embeddingModel: path.basename(emb) } : {}),
- ...(raw.version ? { version: String(raw.version) } : {}),
- threshold,
- },
+ // Provenance is per-engine, and the fields a given engine CANNOT have are
+ // left off rather than written empty. Recording sherpa's threshold on a
+ // sortformer record would be a lie that isDiarizationFresh then acts on: it
+ // has no clustering step and no threshold, so a threshold edit must not mark
+ // its sidecars stale.
+ engine: usingSortformer
+ ? {
+ engine: "sortformer",
+ model: path.basename(sortformerModel),
+ // The driver reports its ggml backend here ("sortformer/Vulkan0"), so
+ // the record says which device produced it. Excluded from the freshness
+ // identity along with every other `version` — CPU and Vulkan produce
+ // byte-identical turns, so re-running one on the other buys nothing.
+ ...(raw.version ? { version: String(raw.version) } : {}),
+ }
+ : {
+ engine: engineCmd ? path.basename(cmd) : "sherpa-onnx",
+ // basename only: absolute paths are machine-specific and the corpus is
+ // rsynced between shards, so a full path would make records non-portable.
+ ...(seg ? { segmentationModel: path.basename(seg) } : {}),
+ ...(emb ? { embeddingModel: path.basename(emb) } : {}),
+ ...(raw.version ? { version: String(raw.version) } : {}),
+ threshold,
+ },
};
const json = JSON.stringify(record) + "\n";
diff --git a/scripts/sortformer/diarize-file.cpp b/scripts/sortformer/diarize-file.cpp
@@ -0,0 +1,268 @@
+// Sortformer diarization engine: 16 kHz mono WAV in, speaker turns out.
+//
+// Kept deliberately dumb, exactly like scripts/diarize-sherpa.py: read audio,
+// run the model, print raw turns as JSON on stdout. All policy (filenames,
+// provenance shape, atomic writes) lives in scripts/diarize.mjs, and audio
+// decoding lives in scripts/diarize-sortformer.mjs, so this file is only ever
+// "run the model".
+//
+// stdout is JSON ONLY. Anything human goes to stderr.
+//
+// WHY THIS FILE EXISTS RATHER THAN AN UPSTREAM BINARY: openresearchtools/engine
+// ships `llama-realtime-smoke`, a PARITY tool. It constructs the backend with
+// capture_debug=true (retaining every intermediate matrix for every step), it
+// requires a directory of PyTorch reference fixtures we do not have, and its
+// JSON dump drops the event flags — which is fatal, because 2033 of the 2099
+// events it emits for a 12.8-minute file are PREVIEW re-emissions of spans that
+// are still growing. Only the 66 non-preview events are the answer. This driver
+// is that tool with the debug capture off, the fixtures gone, and the preview
+// events filtered.
+//
+// WHY NO MERGE PASS: the non-preview events are already the postprocessed,
+// disjoint spans — running an interval union over them is a no-op (66 in, 66
+// out, verified). Unlike the sherpa engine there is no clustering step and no
+// threshold: Sortformer is end-to-end, which is the whole reason it does not
+// over-split.
+//
+// WHY RAW PCM ON STDIN IS THE PRIMARY INPUT: this corpus has 8-hour VODs. A
+// decoded 16 kHz mono WAV of one is ~900 MB on a disk that is 98% full, and
+// buffering it as float32 is ~460 MB resident. Streaming ffmpeg's output straight
+// into push_audio() costs neither: the model's own state is a fixed-size speaker
+// cache, so memory becomes O(1) in duration rather than O(n). That is the single
+// biggest advantage over the sherpa engine, whose O(n^2) clustering matrix is
+// what forced the whole windowing/centroid-reclustering design in
+// diarize-sherpa.py. --audio-wav is kept for direct CLI use and testing.
+
+#include "backend-factory.h"
+#include "sortformer/sortformer-backend.h"
+#include "stream-manager.h"
+
+#include <algorithm>
+#include <chrono>
+#include <cstdint>
+#include <cstdio>
+#include <cstring>
+#include <fstream>
+#include <iomanip>
+#include <iostream>
+#include <memory>
+#include <sstream>
+#include <stdexcept>
+#include <string>
+#include <vector>
+
+// POSIX only, which is all scripts/build-sortformer.sh targets. The upstream
+// engine builds on Windows too; this driver deliberately does not try to.
+#include <cerrno>
+#include <fcntl.h>
+#include <unistd.h>
+
+namespace {
+
+// 16-bit PCM mono, which is what `ffmpeg -ac 1 -ar 16000 -c:a pcm_s16le` makes.
+// Deliberately not a general WAV reader: the adapter always hands us that exact
+// format, and quietly accepting anything else would mean silently diarizing
+// resampled-wrong audio.
+std::vector<float> load_wav_s16_mono(const std::string & path, uint32_t & sample_rate) {
+ std::ifstream f(path, std::ios::binary);
+ if (!f) throw std::runtime_error("cannot open wav: " + path);
+
+ char riff[12];
+ f.read(riff, 12);
+ if (std::strncmp(riff, "RIFF", 4) != 0 || std::strncmp(riff + 8, "WAVE", 4) != 0)
+ throw std::runtime_error("not a RIFF/WAVE file: " + path);
+
+ uint16_t channels = 0, bits = 0;
+ while (f) {
+ char id[4];
+ uint32_t sz = 0;
+ f.read(id, 4);
+ f.read(reinterpret_cast<char *>(&sz), 4);
+ if (!f) break;
+
+ if (std::strncmp(id, "fmt ", 4) == 0) {
+ std::vector<char> fmt(sz);
+ f.read(fmt.data(), sz);
+ if (sz >= 16) {
+ std::memcpy(&channels, fmt.data() + 2, 2);
+ std::memcpy(&sample_rate, fmt.data() + 4, 4);
+ std::memcpy(&bits, fmt.data() + 14, 2);
+ }
+ } else if (std::strncmp(id, "data", 4) == 0) {
+ if (channels != 1 || bits != 16)
+ throw std::runtime_error("expected 16-bit mono wav, got " +
+ std::to_string(channels) + "ch/" +
+ std::to_string(bits) + "bit");
+ const size_t n = sz / 2;
+ std::vector<int16_t> pcm(n);
+ f.read(reinterpret_cast<char *>(pcm.data()), sz);
+ std::vector<float> out(n);
+ for (size_t i = 0; i < n; ++i) out[i] = static_cast<float>(pcm[i]) / 32768.0f;
+ return out;
+ } else {
+ f.seekg(sz + (sz & 1), std::ios::cur); // chunks are word-aligned
+ }
+ }
+ throw std::runtime_error("no data chunk in wav: " + path);
+}
+
+// The sortformer path never sets a thread count (only voxtral's runtime does),
+// so the CPU backend runs at ggml's default of 4. On this box 6 threads is the
+// optimum (1305 s/audio-hour) and 8 is WORSE than 4 through oversubscription, so
+// this has to be reachable rather than left to the default.
+void set_backend_n_threads(ggml_backend_t backend, int n_threads) {
+ if (backend == nullptr) return;
+ ggml_backend_dev_t dev = ggml_backend_get_device(backend);
+ ggml_backend_reg_t reg = dev ? ggml_backend_dev_backend_reg(dev) : nullptr;
+ if (reg == nullptr) return;
+ auto * fn = (ggml_backend_set_n_threads_t)
+ ggml_backend_reg_get_proc_address(reg, "ggml_backend_set_n_threads");
+ if (fn != nullptr) fn(backend, n_threads);
+}
+
+// Stream s16le mono PCM from stdin straight into the session. Never holds more
+// than one block, so an 8-hour VOD costs the same resident memory as a 6-minute
+// clip. Returns the number of samples fed.
+size_t feed_raw_stdin(llama::realtime::stream_manager & mgr, int64_t sid,
+ uint32_t sample_rate, size_t block_samples) {
+ std::vector<int16_t> pcm(block_samples);
+ std::vector<float> block(block_samples);
+ size_t total = 0;
+
+ // FORCE THE PIPE BACK TO BLOCKING. Node sets the pipes it hands a child to
+ // non-blocking, so a plain read() returns -1/EAGAIN long before EOF and the
+ // stream looks like an I/O error. It works from a shell and fails under the
+ // app, which is exactly the sort of difference that gets found in production
+ // rather than in a test — so it is fixed here, in the binary, rather than
+ // being made the caller's problem.
+ const int flags = fcntl(STDIN_FILENO, F_GETFL, 0);
+ if (flags != -1 && (flags & O_NONBLOCK))
+ fcntl(STDIN_FILENO, F_SETFL, flags & ~O_NONBLOCK);
+
+ // read(2) rather than fread: it lets EINTR be retried without the ambiguity
+ // of a short fread, and a partial read is normal on a pipe rather than an
+ // error to distinguish from EOF.
+ while (true) {
+ size_t filled = 0;
+ const size_t want = block_samples * sizeof(int16_t);
+ auto * buf = reinterpret_cast<char *>(pcm.data());
+ while (filled < want) {
+ const ssize_t n = ::read(STDIN_FILENO, buf + filled, want - filled);
+ if (n == 0) break; // EOF
+ if (n < 0) {
+ if (errno == EINTR) continue;
+ throw std::runtime_error(std::string("read error on stdin: ") +
+ std::strerror(errno));
+ }
+ filled += static_cast<size_t>(n);
+ }
+ const size_t got = filled / sizeof(int16_t);
+ if (got == 0) break;
+ for (size_t i = 0; i < got; ++i) block[i] = static_cast<float>(pcm[i]) / 32768.0f;
+ mgr.push_audio(sid, block.data(), got, sample_rate);
+ total += got;
+ if (filled < want) break; // short read means EOF
+ }
+ return total;
+}
+
+void usage(const char * argv0) {
+ std::cerr
+ << "usage: " << argv0 << " --model <sortformer.gguf> (--audio-raw - | --audio-wav <f>)\n"
+ << " [--sample-rate 16000] [--backend Vulkan0|CPU]\n"
+ << " [--threads N] [--feed-ms N]\n\n"
+ << "--audio-raw - read s16le mono PCM from stdin (streaming, O(1) memory)\n"
+ << "--audio-wav <f> read a 16 kHz mono 16-bit WAV file\n\n"
+ << "Prints {\"turns\":[{start,end,speaker}],\"audioSeconds\":N,\"version\":\"...\"}\n"
+ << "on stdout. Progress and errors go to stderr.\n";
+}
+
+} // namespace
+
+int main(int argc, char ** argv) {
+ try {
+ std::string gguf, wav, raw, backend = "Vulkan0";
+ uint32_t sr = 16000;
+ int n_threads = 0;
+ double feed_ms = 100.0;
+
+ for (int i = 1; i < argc; ++i) {
+ const std::string a = argv[i];
+ if (a == "--help" || a == "-h") { usage(argv[0]); return 0; }
+ if (i + 1 >= argc) throw std::invalid_argument("missing value for " + a);
+ if (a == "--model") gguf = argv[++i];
+ else if (a == "--audio-wav") wav = argv[++i];
+ else if (a == "--audio-raw") raw = argv[++i];
+ else if (a == "--sample-rate") sr = static_cast<uint32_t>(std::stoul(argv[++i]));
+ else if (a == "--backend") backend = argv[++i];
+ else if (a == "--threads") n_threads = std::stoi(argv[++i]);
+ else if (a == "--feed-ms") feed_ms = std::stod(argv[++i]);
+ else throw std::invalid_argument("unknown argument: " + a);
+ }
+ if (gguf.empty() || (wav.empty() == raw.empty())) { usage(argv[0]); return 2; }
+ if (!raw.empty() && raw != "-")
+ throw std::invalid_argument("--audio-raw only supports '-' (stdin)");
+
+ // Load the model BEFORE reading stdin, so a bad model path fails before
+ // ffmpeg has decoded anything. capture_debug=false — see the header.
+ auto be = std::make_unique<llama::realtime::sortformer_stream_backend>(gguf, backend, false);
+ auto * be_ptr = be.get();
+ if (n_threads > 0) set_backend_n_threads(be_ptr->model().backend(), n_threads);
+
+ llama::realtime::stream_manager mgr;
+ const int64_t sid = mgr.create_session(std::move(be));
+ std::cerr << "sortformer: backend=" << be_ptr->backend_name()
+ << " input=" << (raw.empty() ? wav : "stdin") << "\n";
+
+ const auto t0 = std::chrono::steady_clock::now();
+ size_t n_samples = 0;
+ if (!raw.empty()) {
+ if (sr == 0) throw std::runtime_error("--sample-rate must be non-zero");
+ n_samples = feed_raw_stdin(
+ mgr, sid, sr,
+ std::max<size_t>(1, static_cast<size_t>((feed_ms / 1000.0) * sr)));
+ } else {
+ const auto audio = load_wav_s16_mono(wav, sr);
+ if (sr == 0) throw std::runtime_error("wav reports a zero sample rate");
+ const size_t feed =
+ std::max<size_t>(1, static_cast<size_t>((feed_ms / 1000.0) * sr));
+ for (size_t off = 0; off < audio.size(); off += feed) {
+ const size_t n = std::min(feed, audio.size() - off);
+ mgr.push_audio(sid, audio.data() + static_cast<ptrdiff_t>(off), n, sr);
+ }
+ n_samples = audio.size();
+ }
+ mgr.flush_session(sid);
+ const double infer_sec =
+ std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();
+ const double audio_sec = static_cast<double>(n_samples) / static_cast<double>(sr);
+
+ // PREVIEW EVENTS ARE NOT THE ANSWER. A streaming span is re-emitted every
+ // time it grows; only the final, postprocessed commits carry the result.
+ const auto events = mgr.drain_events(sid, 0);
+ std::ostringstream turns;
+ size_t n_turns = 0;
+ for (const auto & e : events) {
+ if (e.type != llama::realtime::event_type::speaker_span_commit) continue;
+ if (e.flags & llama::realtime::event_flag_preview) continue;
+ if (e.end_sec <= e.begin_sec) continue;
+ if (n_turns++) turns << ",";
+ turns << "{\"start\":" << e.begin_sec
+ << ",\"end\":" << e.end_sec
+ << ",\"speaker\":" << e.speaker_id << "}";
+ }
+
+ std::cerr << "sortformer: " << n_turns << " turns in "
+ << std::fixed << std::setprecision(1) << infer_sec << "s ("
+ << std::setprecision(2) << (audio_sec / infer_sec) << "x realtime)\n";
+
+ std::cout << std::setprecision(6)
+ << "{\"turns\":[" << turns.str() << "]"
+ << ",\"audioSeconds\":" << audio_sec
+ << ",\"version\":\"" << be_ptr->backend_name() << "\"}\n";
+ return 0;
+ } catch (const std::exception & e) {
+ std::cerr << "sortformer: error: " << e.what() << "\n";
+ return 1;
+ }
+}
