Archilyzer · Source

archilyzer

Archilyzer
git clone https://archilyzer.pages.dev/source/archilyzer.git
Log | Files | Refs | README | LICENSE

commit d3f1d2532024e74a41ed34ac36418e1c6e36add8
parent 4a6fbf4758ed0c8ca3e55ea91cc5c8bacfa770cc
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date:   Tue, 18 Aug 2026 23:29:51 -0400

umtool: the docs, `umtool new`, and a Tailwind own-goal

Eleven more sheets under umtool/docs/, so the entry point for "you have been
asked for an umtool video" is a decision tree and three commands rather than a
reading of the source. Each ends with a "discovered by getting it wrong once"
section, matching the specs/ house style, and every one of those items is
something this work actually got wrong: the import cycle Turbopack sees and node
does not, the production path written out by hand that silently emptied the
fixture's inbox, forty true rows burying two blocking ones, the shadow
CHANNELS_DIR that made three confident findings false.

`umtool new <slug> --from <sweep-report.md>` scaffolds a project and writes an
EMPTY timeline on purpose. Deriving first-draft clips from a report's citations
would be easy -- the shape is regular -- and it is not done, which is the honest
position rather than a missing feature: a report records ONE second per citation,
a window needs a start and an end from the cue file, and matching a quote to its
cues is the actual work of authoring a cut. A generated timeline of guessed
windows would look finished and be wrong, and every clip would have to be opened
anyway. So it lists the citations it found as a CHECKLIST -- 26 of them from the
real hasan-bike report -- and leaves siteOrigin empty so `check` blocks until
somebody sets it.

And the Tailwind trap, twice, the second time by my own hand.

The first was a scratch NEXT_DIST_DIR that was not gitignored: Tailwind v4
auto-detects its sources, honours .gitignore and scans everything else, so it
read a binary turbopack cache, extracted `p-[var(-sM0or-Z)]` as a class
candidate, and every page 500d on a CSS parse error with nothing wrong in the
CSS. Fixed by widening .gitignore.

The second was docs/browse.md. Documenting the failure meant quoting the corrupt
candidate; Tailwind scans markdown; the trap re-created itself and took a whole
suite run down. So detection is now OFF -- `@import "tailwindcss" source(none)`
plus explicit `@source` for app, components and lib. A doc, a fixture, a test
artefact or a stray dist dir can no longer poison the stylesheet at all. Verified
the stylesheet is still complete: 81 arbitrary-value utilities, `text-[11px]` and
`border-[var(--color-sel)]` among them.

Left alone deliberately: scripts/report-to-video/*.mjs is being edited
concurrently by another job, which is building a rail feature ON TOP of this
work -- it uses the EMIT protocol and buildVideo() added in the first commit, and
every export those introduced survives. Only umtool/ is staged here.

e2e: 142 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Diffstat:
Mumtool/app/globals.css | 17++++++++++++++++-
Mumtool/bin/umtool.mjs | 140+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Mumtool/docs/README.md | 21++++++++++++++++-----
Aumtool/docs/authoring.md | 125+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Aumtool/docs/browse.md | 88+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Aumtool/docs/build.md | 109+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Aumtool/docs/cli.md | 89+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Aumtool/docs/clip-bench.md | 93+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Aumtool/docs/decisions.md | 80+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Aumtool/docs/e2e.md | 81+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Aumtool/docs/folders.md | 90+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Aumtool/docs/index.md | 75+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Aumtool/docs/mix-from-a-project.md | 84+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Aumtool/docs/projects.md | 115+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Mumtool/e2e/projects.spec.ts | 30++++++++++++++++++++++++++++++
15 files changed, 1231 insertions(+), 6 deletions(-)

diff --git a/umtool/app/globals.css b/umtool/app/globals.css @@ -1,4 +1,19 @@ -@import "tailwindcss"; +/* Tailwind v4 auto-detects its sources, and that is too broad here. + * + * It honours .gitignore but scans everything else -- including MARKDOWN. This + * file's own documentation (umtool/docs/browse.md) described the failure mode + * by quoting the corrupt class name it produces, Tailwind extracted that quote + * as a candidate, and every page 500d on a CSS parse error again. Documenting + * the trap re-created the trap. + * + * So detection is turned off and the three directories that actually hold + * class names are named. This makes the whole class of problem impossible: a + * scratch dist dir, a test artefact, a fixture or a doc can no longer poison + * the stylesheet. */ +@import "tailwindcss" source(none); +@source "../app"; +@source "../components"; +@source "../lib"; /* A dark, dense, keyboard-first bench. Nothing here is deployed; the only user is someone judging thousands of clips, or auditioning one transition, in a diff --git a/umtool/bin/umtool.mjs b/umtool/bin/umtool.mjs @@ -20,6 +20,7 @@ // umtool window <project> <clip> [--start S] [--end E] [--lock] [--lock-end] ... // umtool build <project> [--preset preview|fast|final] [--only ID] [--dry] // umtool index [--rebuild] [--prune] [--since MS] [--json] +// umtool new <slug> [--kind report-video] [--from <sweep-report.md>] import process from "node:process"; import { PROJECT_KINDS, @@ -32,6 +33,8 @@ import { summarise, } from "../lib/projects/core.mjs"; import { readClipDetail, readManifest } from "../lib/projects/report.mjs"; +import path from "node:path"; +import { mkdir, readFile, writeFile, stat } from "node:fs/promises"; import { updateClip } from "../lib/report/manifest.mjs"; import { buildSteps, PRESETS } from "../lib/report/driver.mjs"; import { openIndex, signRecord } from "../lib/projects/index-db.mjs"; @@ -287,6 +290,7 @@ function usage() { " umtool window <project> <clip> [--start S] [--end E] [--lock|--lock-end|…]", " umtool build <project> [--preset preview|fast|final] [--only ID]", " umtool index [--rebuild] [--prune] [--since MS] [--json]", + " umtool new <slug> [--kind report-video] [--from <sweep-report.md>]", "", `reading ${REPORTS_ROOT} (set REPORTS_DIR to move it)`, "", @@ -301,6 +305,7 @@ const COMMANDS = { window: cmdWindow, build: cmdBuild, index: cmdIndex, + new: cmdNew, show: cmdShow, check: cmdCheck, decisions: cmdDecisions, @@ -446,3 +451,138 @@ async function cmdIndex() { console.log("against the filesystem, so a stale record self-heals on the next load."); await ix.close(); } + + +// --------------------------------------------------------------------------- +// Scaffolding. +// --------------------------------------------------------------------------- + +/** + * A manifest skeleton, and deliberately an EMPTY timeline. + * + * It would be easy to derive first-draft clips from a report's citations: the + * shape is regular (`> "quote"` then `— [title @ h:mm:ss](…?v=slug%2Fid&t=sec)`). + * It is not done, and that is the honest position rather than a missing feature. + * A report records ONE second per citation; a window needs a start AND an end + * taken from transcript.cues.json, and matching a quote to its cues is the + * actual work of authoring a cut. A generated timeline of guessed windows would + * look finished and be wrong, and every clip would have to be opened anyway. + * + * So this writes what can be known -- the slug, the origin, the channel, the + * render block -- lists the citations it found as a checklist, and says what to + * do next. + */ +function skeleton(slug, title, provenance) { + return { + schemaVersion: 1, + slug, + title, + subtitle: "", + generatedOn: new Date().toISOString().slice(0, 10), + provenance: { + // The field that shipped broken TWICE. It is first, and it is empty rather + // than plausible, so `umtool check` blocks until somebody sets it. + siteOrigin: "", + channelSlug: "", + channel: "", + ...provenance, + }, + render: { + width: 1920, + height: 1080, + fps: 30, + audioRate: 48000, + audioChannels: 2, + maxHeightSource: 1080, + fontRegular: "/usr/share/fonts/TTF/FiraSans-Regular.ttf", + fontBold: "/usr/share/fonts/TTF/FiraSans-Bold.ttf", + palette: { bg: "#12100c", fg: "#f6f1e6", muted: "#a2957f", accent: "#c8752a", amber: "#ffc860" }, + transition: 0.4, + fetchPad: 3, + snapWindow: 1.6, + silenceMinDur: 0.09, + silenceRelDb: 6, + headerHeight: 56, + footerHeight: 0, + crf: 21, + preset: "slow", + qr: { scale: 4, quiet: 3, ecc: "M", margin: 28 }, + }, + timelineNodes: [], + timeline: [], + }; +} + +async function cmdNew() { + const slug = positional[0]; + if (!slug) die("usage: umtool new <slug> [--kind report-video] [--from <sweep-report.md>]"); + if (!/^[a-z0-9][a-z0-9-]*$/.test(slug)) { + die(`"${slug}" will not route — use lower-case letters, digits and dashes`); + } + const kind = val("--kind") ?? "report-video"; + if (kind !== "report-video") die(`only report-video can be scaffolded so far, not ${kind}`); + + const dir = path.join(REPORTS_ROOT, slug); + if (await stat(dir).then(() => true, () => false)) die(`${dir} already exists`); + + let title = slug.replace(/-/g, " "); + const citations = []; + const fromArg = val("--from"); + if (fromArg) { + const text = await readFile(fromArg, "utf8").catch(() => null); + if (text === null) die(`could not read ${fromArg}`); + title = text.match(/^#\s+(.+)$/m)?.[1]?.trim() ?? title; + // `?v=<channel>%2F<id>&t=<sec>` -- the shape the viewer's share links use. + for (const m of text.matchAll(/\]\([^)]*[?&]v=([^&)]+)&t=(\d+)/g)) { + const [chan, id] = decodeURIComponent(m[1]).split("/"); + citations.push({ channel: chan, video: id, second: Number(m[2]) }); + } + } + + const channels = [...new Set(citations.map((c) => c.channel))]; + const doc = skeleton(slug, title, channels.length === 1 ? { channelSlug: channels[0] } : {}); + + await mkdir(dir, { recursive: true }); + await writeFile(path.join(dir, "video.manifest.json"), JSON.stringify(doc, null, 2) + "\n", "utf8"); + + if (fromArg) { + await writeFile( + path.join(dir, "sweep-report.md"), + await readFile(fromArg, "utf8"), + "utf8", + ); + } + + const lines = [ + `# ${title}`, + "", + "What this cut argues, which sources it draws on, and anything cut short on", + "purpose (with why — that is what a `lock` in the manifest means).", + "", + "## Windows still to write", + "", + citations.length + ? "Each of these is ONE second from the report. A clip needs a start AND an" + + " end, read from the source's transcript.cues.json — that matching is the work." + : "No `?v=` citations were found, so there is nothing to work from yet.", + "", + ...citations.map( + (c, i) => `- [ ] c${String(i).padStart(2, "0")} ${c.channel}/${c.video} @ ${c.second}s`, + ), + ]; + await writeFile(path.join(dir, "README.md"), lines.join("\n") + "\n", "utf8"); + + if (json) return out({ ok: true, dir, citations: citations.length, channels }); + + console.log(`${dir}`); + console.log(` video.manifest.json an EMPTY timeline — see README.md`); + if (fromArg) console.log(` sweep-report.md copied from ${fromArg}`); + console.log(` README.md ${citations.length} citation(s) as a checklist`); + console.log(""); + console.log("Next, in order:"); + console.log(` 1. set provenance.siteOrigin — it is EMPTY, and \`check\` blocks until it is not.`); + console.log(` Two finished videos shipped with QR codes that resolve to nothing.`); + console.log(` 2. write the timeline (umtool/docs/authoring.md)`); + console.log(` 3. umtool check ${slug}`); + console.log(` 4. umtool build ${slug} --preset fast`); +} diff --git a/umtool/docs/README.md b/umtool/docs/README.md @@ -20,18 +20,29 @@ Work out which kind you are making first — the rest follows from it. The short path from a cited report to a built video: ```sh -# 1. write video.manifest.json beside the report (authoring.md — this is the work) +# 0. scaffold it (writes an EMPTY timeline and a citation checklist) +umtool new <slug> --from <sweep-report.md> + +# 1. write the timeline (authoring.md — this is the work, and nothing automates it) + # 2. is every source still fetchable, and is every citation wired up? -umtool check ~/reports/<slug> +umtool check <slug> + # 3. widen windows to whole sentences (dry first, then apply) node scripts/report-to-video/resolve-windows.mjs ~/reports/<slug>/video.manifest.json node scripts/report-to-video/resolve-windows.mjs ~/reports/<slug>/video.manifest.json --write -# 4. a fast pass to look at, then the real one + +# 4. bench any clip whose edges you are unsure of +# /browse/<slug>/clip/<id> + +# 5. build: a fast pass to watch, then the real one. +# The BUTTON on the project page runs it -- cancellation, the per-step +# timeouts and the process-group kill live in the server's job runner. +# `umtool build` prints the same chain if you would rather paste it. umtool build <slug> --preset fast -umtool build <slug> --preset final ``` -**Run step 2 before step 4, always.** It is a few seconds and it catches the two +**Run step 2 before step 5, always.** It is a few seconds and it catches the two defects that have already shipped in real videos: a manifest with no `siteOrigin` (19 QR codes encoding `undefined/?v=…`) and one pointing at `http://localhost:3000` (QR codes that resolve to nothing on anybody's phone). diff --git a/umtool/docs/authoring.md b/umtool/docs/authoring.md @@ -0,0 +1,125 @@ +# Authoring a report video + +From a cited sweep report to a built cut. Written for an agent; a person can +follow it too. + +## 0. Scaffold + +```sh +umtool new <slug> --from ~/reports/<sweep>/sweep-report.md +``` + +Writes the directory, a manifest skeleton with an **empty** timeline, and a +README listing every `?v=` citation as a checklist. + +## 1. Fill in provenance — `siteOrigin` FIRST + +```jsonc +"provenance": { + "siteOrigin": "https://jeralyzer.pages.dev", // the archive's REAL origin + "channelSlug": "the-quartering", // default channel for cue lookups + "channel": "TheQuartering" +} +``` + +**This is the field that shipped broken twice.** One manifest has no `siteOrigin` +at all — 19 QR codes encoding `undefined/?v=…` — and one has +`http://localhost:3000`, a finished video whose codes resolve to nothing on +anybody's phone. `umtool check` blocks until it is set to something real. + +## 2. Turn each citation into a WINDOW + +**This is the work, and it is the part nothing automates.** + +A report records **one** second per citation. A clip needs a start *and* an end, +and both come from the source's cue file: + +``` +transcripts/channels/<channel>/data/<video>/transcript.cues.json +``` + +For each citation: + +1. Open the cue file and find the cues around the cited second. +2. **Verify the quote is actually there, and that the speaker is who you think.** + Two standing traps: a first-person quote is often the host reading someone + else's tweet aloud or being sarcastic — check ±90 s of context; and ASR + garbles names (the corpus stores "Metokur" as "mr medicare"). +3. Take `start` from the first cue of the thought and `end` from the last. +4. Round to **2 dp**. + +```jsonc +{ "type": "clip", "id": "c04", "video": "uyz1_FIqIEk", + "start": 32980.24, "end": 32994.19, + "cite": 32989, + "quote": "the words this clip exists for", + "note": "why it is in the cut" } +``` + +**Array order is the cut.** There is no sort. + +Per-clip `channel` when the sources span mirrors. On Rumble, `video` must be the +**local directory slug**, not the site/MCP id — see [quirks.md](quirks.md). + +## 3. Check, before anything encodes + +```sh +umtool check <slug> +``` + +Fix everything blocking. It catches a dead origin, a missing `channelSlug`, +duplicate ids, a cue file that is not there, and a source the last preflight found +gone. + +## 4. Widen to sentences + +```sh +node scripts/report-to-video/resolve-windows.mjs <manifest> # dry +node scripts/report-to-video/resolve-windows.mjs <manifest> --write # apply +``` + +A cue boundary is a *line-wrap* boundary, so cutting there drops the lead-in that +makes a quote make sense. + +Then read the result. Where the widener made a clip worse — it swallowed +neighbouring audio, or it undid a deliberately short quote — set `lock` and say +why in the project README. Across the six real manifests, five are **100% locked**: +a human-chosen window usually *is* the truth. + +## 5. Bench each clip + +`/browse/<slug>/clip/<id>`. Look at the waveform and the cue rail, and act on +what it says: run a clip to the end of its sentence, or set `lockEnd` to +acknowledge that you meant to cut there. One 14-clip cut shipped with 8 clips +ending mid-thought. + +If it warns that the widener would revert your edit, take the offer to set the +matching lock. + +## 6. Fast pass, watch it, then final + +```sh +umtool build <slug> --preset fast # hard cuts, minutes +umtool build <slug> --preset final +``` + +Watch the fast pass end to end. It is the only way to find a clip that is +technically correct and editorially wrong. + +## The checklist + +- [ ] `siteOrigin` is the archive's real origin +- [ ] every `video` is the **local** directory slug +- [ ] every quote verified in context, not just found by search +- [ ] every window at 2 dp, `end` after `start` +- [ ] `umtool check` exits 0 +- [ ] windows resolved, or locked with a reason written down +- [ ] no clip ends mid-sentence unless `lockEnd` says so +- [ ] a fast pass watched all the way through + +## What a sweep will miss + +Worth knowing before you trust a citation list to be complete: a share-link +keyword is the **spine** of a sweep, not its coverage. Back it with substring, +clinical-vocabulary and symptom sweeps — one real cut's earliest and best clip +never says the search word at all. diff --git a/umtool/docs/browse.md b/umtool/docs/browse.md @@ -0,0 +1,88 @@ +# /browse — the project index + +Every project, of every kind, in one list. **Zero client JavaScript**: the filters +are links that change `searchParams`, and `?q=` is a plain GET form. Nothing here +hydrates, and `pnpm build` still reports the page as server-rendered. + +## The four filters + +| filter | param | values | +|---|---|---| +| kind / template | `?kind=`, `?template=` | registry ids | +| state | `?state=` | `draft` `windows` `fetched` `built` `shipped` `stale` | +| open decisions | `?open=blocking` \| `?open=1` | from the decision counts | +| text / recency | `?q=`, `?sort=name\|recent` | substring over the haystack | + +**Counts come from the unfiltered set.** A chip whose number changes when you +click a different chip moves under the cursor, and the whole point of a filter row +is to say how much is behind each one. An e2e spec asserts the chip's number +equals the number of cards rendered. + +`?q=` is a `<form method="get">` carrying the other filters as hidden inputs — the +`/browse/find` idiom — so searching does not throw away the filters you set, and +any filtered view is one pasteable URL. + +## The state vocabulary + +One vocabulary across every kind, not per-kind words, because the point of the +filter is to ask "what is half-done" without first asking "half-done at what". A +kind maps its own situation onto these; it does not invent a seventh. + +## The cards + +`components/projects/ProjectGrid.tsx` **contains no kind ids and no per-kind +branches**, and that is the design working rather than an omission: a kind's +summariser has already rendered its own facts (`19 clips · 4m46s · 17 sources`, +`3/4 cuts · 2 variants`) and its own flags. The grid lays out strings. + +A kind can attach `data-*` attributes to its card through `attrs`, which is how a +song keeps `data-missing` as an assertion surface without the grid knowing what a +cut is. + +The poster falls back: the deliverable → a built **segment** (which already +carries the chrome, so the card looks like the video mid-build) → a raw clip → +nothing. Served by `/api/browse/poster?project=…`, which takes **no +client-supplied `rel`** — the frame is the one the summariser chose. + +## Routing + +`app/browse/[...path]` resolves the **longest** path prefix that is a project and +hands the rest to that kind's view. `a/b` being a project must not stop `a/b/c` +from being one. + +- `[]` → the project page +- `["wide"]` on a song → the cut page, unchanged +- `["clip","c04"]` on a report video → the clip bench +- anything else → 404, rather than a page that silently drops half its URL + +The static tool pages (`/browse/decisions`, `find`, `sources`, `faces`, `trim`, +`at`) still win their routes, and a spec asserts each is 200. + +## Cost + +Listing never probes and never shells out — the rule `listSongs()` already +followed, extended to every kind. A summary is memoised against a signature of +mtimes and sizes, so it survives for as long as the project has not changed and is +discarded the moment it has. + +The `open` filter needs decision counts for every project, which means running +each kind's reducer. That is memoised the same way; the expensive input is cue +files, and their *derived* answers are cached against the file's own mtime. + +## Discovered by getting it wrong once + +**Tailwind's source detection is turned OFF, and the three real directories are +named in `app/globals.css`.** It has to be. Auto-detection honours `.gitignore` +but scans everything else, so a scratch `NEXT_DIST_DIR` got read, its binary +turbopack cache yielded a garbage class candidate, and every page 500d on a CSS +parse error with nothing wrong in the CSS. + +Then it happened a second time from **this file**: an earlier draft quoted the +corrupt candidate to explain the first failure, Tailwind scanned the markdown, +and the trap re-created itself. Hence `source(none)` plus explicit `@source` — +a doc, a fixture, a test artefact or a stray dist dir can no longer poison the +stylesheet at all. If you add a directory that holds class names, name it there. + +**`lmdb` must be in `serverExternalPackages`.** Bundled, Turbopack tries to +resolve its optional `moduleRequire('cbor-x')` and fails the whole module graph — +so every page importing `lib/projects` 500s naming a package nothing here uses. diff --git a/umtool/docs/build.md b/umtool/docs/build.md @@ -0,0 +1,109 @@ +# Building + +From the project page, or `umtool build <project>` to see the chain. + +## Four steps + +| # | step | why it is separate | +|---|---|---| +| 1 | **check every source is still fetchable** | `yt-dlp --simulate`, no bytes. See below. | +| 2 | **resolve windows (DRY)** | Applying is a separate, explicit action. | +| 3 | **build** | `--progress ndjson --continue-on-error` | +| 4 | **verify the file that came out** | A build can exit 0 and be wrong. | + +**Availability is a STEP, not a preamble somebody remembers to run.** It is the +one fact about a manifest that goes stale in *both* directions — a source can die +after the manifest is written, and one annotated "gone" can come back. It costs +seconds. Without it a dead source is discovered twenty minutes and a dozen +paid-for fetches into the build. + +**Resolve runs dry.** A widener silently rewriting windows somebody just set in +the bench is exactly the surprise `lock` exists to prevent, so the chain never +passes `--write` and applying is a second action. + +**Verify exists because success is not self-evident.** A concat that produced a +zero-length file, a chapter pass that dropped markers, a timeline that lost a clip +because `--continue-on-error` let it — each looks like success at the terminal and +like a finished video in a directory listing. `verify-build.mjs` checks duration > +0, chapters == timeline entries, and a length floor. + +## Presets + +- **preview one clip** — `--only <id> --no-xfade`, for after moving an edge. +- **fast pass** — hard cuts over the whole timeline. Minutes, not tens of minutes. + What you watch to check the argument. +- **final** — crossfades and chapters. The deliverable. + +## Timeouts + +Per step, not one number. The default is 15 minutes and exists to catch the +accidental hour-long job; a 19-clip crossfaded build legitimately runs 20 to 40, +so the build step asks for `max(15 min, clips × 90 s)`. Raising the default to fit +the build would remove the guard from everything else. + +## Cancelling is safe, and resuming is free + +Cancel kills the **process group**. `build-video.mjs` shells out through +`execFile`, so the thing actually burning CPU or holding a download open is a +*grandchild* — `child.kill()` reaps the node process and leaves it running, which +is the same failure the diarize backfill had. + +Every artefact is content-addressed: a fetched window by its window, a segment by +its clip id. Re-running skips whatever finished. **A cancelled build is a paused +one.** + +## Overwriting + +`build-video.mjs` always passes `-y`. An output **newer than its manifest** is +refused (409, `needsReplace`); with `replace=1` it is moved aside as +`out/<slug>.<YYYYMMDD-HHMM>.mp4` — the stamp shape `promote` already uses for a +demoted cut. A deliverable that cost an hour of network fetches is never destroyed +to make a new one. + +## One job, process-wide + +Two builds writing one `out/segments/` would interleave, and two in different +projects would still fight over yt-dlp's rate limits and the CPU. A second start +is a 409 naming what is running. + +## Progress + +`--progress ndjson` emits one JSON object per line: `start`, `card`, `clip`, +`fetch`, `snap`, `segment`, `entry-failed`, `concat`, `chapters`, `note`, `done`, +`error`. The UI renders one box per timeline entry from them, because "step 3 of +4, running" is not progress when step 3 is the twenty-minute one. + +The event set is exactly what was already being printed. Making it a *format* +switch is what stops a wording change from breaking the driver. + +## `--continue-on-error` + +A dead source at clip 14 of 19 otherwise throws away thirteen fetches already paid +for. With it, everything buildable is built — and the run then **refuses to +concatenate** and exits non-zero. A finished file quietly missing a citation looks +complete, which is worse than no file. + +## Why it is spawned, not imported + +A 40-minute chain of yt-dlp and ffmpeg inside a request handler has no +cancellation story, its `execFile` buffers live in the server's heap, and a +runaway grandchild outlives the request that started it. `buildVideo()` is +exported anyway, and `widen()` *is* imported — the bench needs the CLI's own +function, or the two would disagree about where a clip ends. + +`umtool build` **prints** the chain rather than running it, for the same reason: +cancellation, the timeouts and the group kill live in the server's job runner, and +a second runner would be a second, worse set of those. + +## Everything here is testable offline + +The e2e fixture writes stub `YTDLP_BIN` and `QRENCODE_BIN`. The stub reports one +id removed the way a deleted upload is, which gives `source-unavailable` a true +answer. A full 4-clip build — cache reuse, three stub fetches, QR overlay, concat, +chapters, verify — runs in **3 seconds with no network**. + +## See also + +[quirks.md](quirks.md) for the VP9 trap, `--ignore-config`, exit 101, the Rumble +HLS retry and the relative silence threshold — every one of which will bite a +build and none of which is guessable. diff --git a/umtool/docs/cli.md b/umtool/docs/cli.md @@ -0,0 +1,89 @@ +# umtool, from a terminal + +``` +pnpm --filter umtool exec umtool <command> +node umtool/bin/umtool.mjs <command> +``` + +The audience is an AI assistant working in this repo, which is why every command +takes `--json` and why `check` exits non-zero. + +It reads the **same** `lib/projects/*.mjs` the app does, so `umtool ls` and +`/browse` cannot disagree about what a project is, and `umtool check` and the +decisions inbox cannot disagree about what is wrong with one. + +## Commands + +| | | +|---|---| +| `ls [--kind --template --state --open --blocking --q --sort --json]` | the index, as text or JSON | +| `show <project> [--json]` | one project: summary, the cut, per-clip status, decisions | +| `check [<project>] [--json]` | **exit 1 on anything blocking** | +| `decisions [--json]` | the inbox | +| `folders [--json]` · `kinds [--json]` | the tree, the registry | +| `window <project> <clip> [--start S] [--end E] [--lock] [--lock-end] [--no-lock-end] [--note …]` | edit a window | +| `build <project> [--preset preview\|fast\|final] [--only ID]` | **prints** the chain | +| `index [--rebuild] [--prune] [--since MS] [--json]` | the cache | +| `new <slug> [--kind report-video] [--from <sweep-report.md>]` | scaffold | + +A project argument is an exact id, a directory, or a **unique** basename. Two +projects answering to one name is reported, never resolved by picking one. + +## Environment + +`REPORTS_DIR`, `SONG_REPORTS_DIR`, `SONG_DIR`, `CHANNELS_DIR`, `UMTOOL_INDEX_DIR` +— which is how it is tested against the e2e fixture. + +## `check` is the one to run before every build + +``` +$ umtool check +BLOCKING ferret-rescue manifest-invalid provenance.siteOrigin + `http://localhost:3000` — every QR in this cut resolves to nothing on anyone else's phone +BLOCKING quartering-employee-count manifest-invalid provenance.siteOrigin + missing — every QR in this cut encodes `undefined/?v=…` +OPEN quartering-walmart-shelves stale-build out/quartering-walmart-shelves.mp4 +12 project(s), 2 blocking, 1 open +$ echo $? +1 +``` + +Those are the two defects that shipped in finished videos. It is a few seconds in +front of a twenty-minute build, and it exits non-zero so a script can gate on it. + +## Two deliberate limits + +**`build` prints, it does not run.** Cancellation, per-step timeouts and the +process-group kill live in the server's job runner; a second runner here would be +a second, worse set of those. Use the button on the project page, or paste the +printed commands. + +**`check` cannot compute the song reducer.** Nine decision kinds are TypeScript +beside `readSong()`, the loudness cache and the accepted cover set. It reports how +many projects it only checked the routing of, and points at `/browse/decisions`. + +## `window` goes through the same writer the bench does + +2 dp, the CLI's own formatting, tmp+rename, one `.bak`. A second implementation is +how the two would start disagreeing about where a clip ends. + +``` +$ umtool window ferret-rescue c01 --start 43.12 --end 61.48 --lock-end +c01: 43.12–61.48 -> 43.12–61.48 + lockEnd +``` + +`--no-lock-end` **removes** the key rather than writing `false`: these manifests +are read by humans, and `"lockEnd": false` reads like a decision. + +## `new` writes an EMPTY timeline, on purpose + +It would be easy to derive first-draft clips from a report's citations — the shape +is regular. It is not done, and that is the honest position rather than a missing +feature: a report records **one** second per citation, a window needs a start and +an end taken from `transcript.cues.json`, and matching a quote to its cues is the +actual work. A generated timeline of guessed windows would look finished and be +wrong, and every clip would have to be opened anyway. + +So it writes what can be known, lists the citations it found as a **checklist**, +and leaves `siteOrigin` empty so `check` blocks until somebody sets it. diff --git a/umtool/docs/clip-bench.md b/umtool/docs/clip-bench.md @@ -0,0 +1,93 @@ +# The clip bench + +`/browse/<project>/clip/<id>`. + +"How much context does this clip need" used to be a loop of hand-editing JSON, +re-running two CLIs and watching an mp4. This is that loop in one place. + +## Absolute source seconds, everywhere + +The manifest's numbers, the cue file's, the QR's. The cached file's own start +(`fetchStart`) is the **only** relative number in the component, and it exists +solely to set `video.currentTime`. The moment those two are allowed to mix is the +moment a window is off by the pad and nobody can see why. + +## The preview is the cached file, served whole + +`out/clips-raw/<video>_<a>-<b>.mp4`, with byte ranges, and all windowing happens +in the browser. No ffmpeg per drag. + +**A 206 is not optional** — without one the `<video>` element will not seek in a +stream it did not fully download, and that is the entire interaction. The file is +immutable (its window is in its name), so it is cached hard and `analyseMedia`'s +envelope can never miss twice for the same window. + +`file` must be a member of the server's own scan of that clip's cached windows. +Never a path from the client. + +## A drag never downloads + +Dragging past the cached window **clamps** and offers a button. A handle that +silently starts a 12-second network fetch is a handle you stop trusting. + +The fetch runs the pipeline's own `--fetch-only` path, so the file lands named the +way the build expects, with the same format pin and the same Rumble HLS retry — +and containing-window reuse then makes that generous fetch **be** the build's +cache rather than a second one. + +Past the source's own duration the handle stops for good. + +## Three things the JSON cannot show you + +**Ends mid-sentence**, quoting the cue the cut lands inside — because "ends +mid-sentence" alone does not tell you what you are cutting off. One 14-clip cut +shipped with 8 clips ending mid-thought. `lockEnd` is the acknowledgement and +silences it. + +**This source has no punctuation**, when the ASR emitted no terminators at all. +Then *every* clip "ends mid-sentence" and the fact says nothing about the cut, so +the answer is `null` — cannot be known — rather than a confident `false`. Set +those edges by ear and lock them. + +**What the widener would do**, computed in-process because `widen()` is pure once +the cues are read. Moving an edge somewhere `resolve-windows` would not produce +means the next `--write` reverts it, so the bench offers to set the matching lock. +That is why five of six real manifests are 100% locked. "Run the widener and see" +stops being a leap of faith. + +## The cue rail + +Every cue in view, positioned by time. Inside the selection in full contrast, +outside dimmed — so "what am I cutting off" is *read*, not inferred from a +waveform. A `¶` marks a cue that closes a sentence, using the same regex +`resolve-windows.mjs` uses (imported, not re-written, so the rail and the widener +cannot disagree). Clicking a cue snaps the nearer edge to it. + +## Keyboard + +`[` `]` move the start · `,` `.` move the end · shift for 0.5 s instead of 0.05 · +`space` auditions the selection · `R` resets to the saved window. + +Audition happens on pointer-**up**, never during a drag — the Deck's rule, because +a sound restarting on every `pointermove` is unusable. + +## Saving + +`PUT /api/report/window` with `{project, clip, start, end, lock…, token}`. It +never sends a path and it cannot ask for an entry to move. + +See [report-video.md](report-video.md) for the four rules the writer keeps (2 dp, +the CLI's formatting, tmp+rename under a lock, an mtime token). A stale token is a +**409 with both values**, never a silent overwrite: the other writer is usually +somebody's judgement. + +## Discovered by getting it wrong once + +**The page and the inbox must answer the same question the same way.** `umtool +show` pilled four ferret-rescue clips "ends mid-sentence" while the decisions +inbox stayed silent about them, because the inbox had a punctuation gate and the +detail did not. The inbox was right. + +**The bench's prediction is testable, and is tested.** It says 3.00–6.00 widens to +3.00–9.00; saving 3.00–9.00 makes `resolve-windows` a no-op on that clip. That +round trip is the whole argument for the bench. diff --git a/umtool/docs/decisions.md b/umtool/docs/decisions.md @@ -0,0 +1,80 @@ +# Decisions + +One worklist, every project, every kind: `/browse/decisions`, or +`umtool decisions`. + +## Severity is EARNED + +| | | +|---|---| +| `blocking` | something downstream would **lie or die** if you acted on it | +| `open` | a real decision nobody has made | +| `info` | a true fact that is not a decision | + +An inbox that marks four missing cuts per song as blocking is an inbox nobody +opens twice. A song that never had a vertical is not waiting on you. + +## It is a REDUCER, and it must never measure + +Every call it makes is one a project page already makes. An inbox that shells out +to ffmpeg once per rendition, or to yt-dlp once per source, is an inbox that takes +a minute to open — which is the one thing it cannot afford to be. + +So loudness is read from the **cache**, and availability is read from whatever the +preflight last **wrote** (`out/availability.json`), never measured here. "Nobody +has ever run one" is itself something it can say. + +## The kinds + +Each registry entry owns its own vocabulary (`decisionKinds`), and the union is +assembled rather than hand-written — a closed union in one shared file would mean +every future kind editing it to say a word only it uses. + +**report-video** + +| kind | severity | trigger | +|---|---|---| +| `manifest-invalid` | blocking | missing or `localhost` `siteOrigin`, missing `channelSlug`, duplicate ids, `section` out of range | +| `clip-no-cues` | blocking | no `transcript.cues.json` — the build dies there | +| `clip-unfetchable` | blocking | the last preflight says the source is gone | +| `clip-mid-sentence` | open | the cut lands inside a cue that does not close a sentence, and `lockEnd` is unset | +| `window-overlap` | open/info | two clips from one source overlap | +| `no-punctuation` | info | a source's ASR has no terminators — one row per project | +| `stale-build` | open | the output is older than the manifest | +| `unbuilt` | info | never built. A normal state, not a decision | + +**song** — the nine the existing reducer emits: `unjudged-variant`, +`missing-cut`, `no-recipe`, `stale-recipe`, `spec-problem`, `no-plan`, +`unattributed`, `thumb-unaccepted`, `loudness`. + +**routing**, kind-independent — `shadowed-name` (blocking), `unroutable-name` +(info), `ambiguous-project` (blocking). + +## Adding one + +Extend the kind's `decisionKinds` and emit it from its `decisions(ctx, summary)`. +Nothing outside `lib/projects/` needs to change. + +## What the CLI can and cannot do + +`umtool check` computes every report-video decision and every routing one, and +exits 1 on anything blocking — which is what makes it usable as a gate before a +build. It **cannot** compute the song reducer: that is TypeScript beside +`readSong()`, the loudness cache and the accepted cover set, and a second +implementation is the thing this design exists to avoid having two of. It says +how many projects it only checked the routing of. + +## Discovered by getting it wrong once + +**Forty true rows are worse than one.** The first real run emitted a +`no-punctuation` row per *source* — forty-odd across six projects, all correct, +burying two blocking rows off the top of the list. One per project now. + +**Do not assume where a project reads its cues from.** The first run confidently +reported three sources of `quartering-flagging-takedowns` as having no cue file. +They cite deleted YouTube uploads, are cut from live Rumble mirrors, and build +against a **shadow `CHANNELS_DIR`** the project's own `make-shadow-channels.sh` +writes. A project now says which directory it reads — `provenance.channelsDir`, or +the `.shadow-channels` convention that already existed — and neither is a guess: +both are things the project wrote down. When the builder is present but unrun, the +decision says to run it rather than declaring the sources gone. diff --git a/umtool/docs/e2e.md b/umtool/docs/e2e.md @@ -0,0 +1,81 @@ +# The e2e suite + +```sh +pnpm --filter umtool e2e # everything +pnpm --filter umtool e2e projects.spec.ts # one file +``` + +A **"waiting for the e2e queue"** banner is normal, not a hang: the queue is +machine-global and one suite runs at a time. + +## The fixture + +`e2e/fixtures/make-fixture.mjs`, rebuilt on every run. The suite **never** runs +against the real song directory or the real reports tree — those hold thousands of +real human verdicts and six finished videos, and a spec that judged a clip or +rendered over a deliverable would be indistinguishable from a person doing it. + +**Every fixture item has a true answer.** That is the principle; the specifics: + +| | | +|---|---| +| `bg.mp4` | 220 Hz then 3000 Hz at the same loudness — only the brightness curve can see the change. A cue at 3.00 s. | +| `song.mp4` | 2 s of silence then a tone. `firstSound` at 2.00 s. | +| `vid1` cues | punctuated, with a run-on cue at 3–6 s | +| `vid2` cues | **no terminator anywhere** — the real degradation in this corpus | +| `gone1` cues | readable, but the stub yt-dlp reports the upload removed | +| `vid1_0.00-9.00.mp4` | tone/silence/tone with silences centred on 3.0 s and 6.0 s (verified in the file: 2.90–3.11, 5.92–6.11) | + +## The projects, and why each exists + +| | | +|---|---| +| `report-fixture` | read-only. 4 clips: c01 ends mid-sentence, c04 does too but sets `lockEnd`, c03's source has no punctuation | +| `bench-fixture` | the clip bench **writes** | +| `build-fixture` | the build **writes** | +| `gone-fixture` | its source is gone — the preflight must block it | +| `no-origin-fixture` / `localhost-fixture` | the two defects that shipped | +| `bike-fixture` | the third kind | +| `find/` | shadowed by a tool page | +| `deep/nested/solo-fixture` | a pass-through chain, for the collapse | + +**Three copies of one manifest is not duplication.** Sharing one project between +the read-only specs and the writing ones made the suite pass or fail depending on +which file playwright ran first — and the failure named the wrong thing entirely. + +## Stubs + +`YTDLP_BIN` and `QRENCODE_BIN` point at node scripts the fixture writes, so the +whole build chain runs **offline and deterministically**. They are node, not bash: +the yt-dlp stub does fractional arithmetic on `--download-sections *FROM-TO`, and +doing that in bash means awk, which means three layers of quoting inside a +generated file. It got mangled once. + +## Writing a spec here + +- Assert **relationships, not magic numbers**. "the chip's number equals the + number of cards" survives a new fixture project; `toHaveCount(6)` does not. +- If your spec writes, give it its own project. +- A project id is a **path**: `[data-project$='/alpha']`, not `[data-song=alpha]`. +- Some assertions read the **source**, not a page — that a kind id is not + special-cased outside the registry, that `RESERVED_BROWSE` matches the real + directory listing. Those are the ones that fail when the design is broken in a + way no rendering can show. + +## Known flake + +`undo.spec.ts:93` reds under load and passes in isolation. It is the documented +waveform-drag capture race, not a regression. + +## Gotchas + +**Run e2e in dev mode** (the default). `E2E_MODE=start` serves the last build, +which is stale for uncommitted source changes. + +**`pnpm build` is a separate check.** The suite runs in dev, which never +prerenders — a layout or client-component change can pass every spec and 500 in +production. Two real bugs in this feature were found that way and one only by a +page render. + +**Kill stray dev servers by port**, not with `pkill -f`: the pattern matches your +own shell's command line. diff --git a/umtool/docs/folders.md b/umtool/docs/folders.md @@ -0,0 +1,90 @@ +# Folders, and the roots + +## `REPORTS_ROOT` + +The tree every project hangs off. `REPORTS_DIR ?? ~/reports`, and in e2e it +defaults to `dirname(SONG_REPORTS_DIR)` so a fixture stays confined without a new +environment variable. + +`SONG_REPORTS` (the um-song deliverables) keeps its exact previous default and is +now a *subdirectory* of `REPORTS_ROOT` rather than the widest root there is. + +## The walk + +Two rules do almost all the work. + +**A PROJECT IS A LEAF.** Detection stops the descent. That is what keeps `out/` +— 1,210 files and 3.1 GB across `~/reports` — out of the walk entirely. Nothing +in the index ever sees a clip, a segment, a card PNG or a variant. + +**A FOLDER WITH NO PROJECT BENEATH IT DOES NOT EXIST.** That silently drops the +~40 loose test directories under `quartering-uh-song` — `alarm-tests`, +`chop-tests`, `run-visual-tests`, `sfx`, `pipeline` — with no denylist to +maintain and nothing to update when the 41st appears. + +Also: dotfiles, `node_modules`, `out`, `variants`, `plan`, `thumbs` and `data` +are never descended into; depth is capped at 4; symlinked directories *are* +followed, but every real path is visited once so a link to an ancestor terminates +instead of spinning. + +Measured on the real tree: **12 projects in 25 ms**, and the 3.1 GB never touched. + +## Collapsing + +A folder with no projects and exactly one child collapses **for display**: +`quartering-uh-song / videos` is one heading. + +**The URL is never collapsed.** `/browse/quartering-uh-song/videos/yoshi` stays +the one true address. A URL has to mean the same thing in six weeks, and a +display convenience does not get to decide what a link is. + +## Read roots vs write roots + +`resolveInRoots()` guarded what may be **opened** and what may be **rendered to**. +They were one list — so widening the read root to reach report videos would in +the same stroke have made every report's `out/` a legal render target. A 46 MB +deliverable that cost an hour of network fetches, one typo in `/api/mix/render` +away from being overwritten. + +``` +READ_ROOTS SONG_REPORTS, REPORTS_ROOT, SONG_DATA, SONG_SCRATCH +WRITE_ROOTS SONG_REPORTS, SONG_SCRATCH +``` + +Reports became readable and mixable. **Nothing new became writable.** A mix of a +report clip still lands in `SONG_REPORTS`, and a hand-typed path outside the +write set is refused exactly as before. + +`MIX_ROOTS` overrides the read set; `MIX_WRITE_ROOTS` overrides the write set. + +**`SONG_REPORTS` stays first in the read list.** It is a subdirectory of +`REPORTS_ROOT`, so whichever comes first decides every relative label — and +putting `REPORTS_ROOT` first would silently rewrite every existing +`videos/<song>/wide.mp4` into `quartering-uh-song/videos/<song>/wide.mp4`. +`labelFor` and `resolveInRoots` read the same ordered list, which is what keeps a +label a round trip. + +## Media listing + +`listMedia()` walks the **project** roots two levels deep — a report's deliverable +is at `<project>/out/<slug>.mp4` and a song's cut at `videos/<song>/<cut>.mp4`, +and neither was visible before. `SONG_DATA` and `SONG_SCRATCH` stay at one level: +a second level there is thousands of stats of clip fragments to find nothing +anybody would load. + +`segments`, `clips-raw`, `cards` and `qr` are excluded by name, or reaching one +level deeper would put ~260 intermediates into a picker that is already saturated. + +## Discovered by getting it wrong once + +**Relative paths bind to the first root, without stating.** `resolveInRoots` does +not touch the disk, so a relative path resolves against the first root it *could* +live under whether or not it is there. Survivable only because every path that +crosses the wire from a picker or a project link is **absolute** — a relative one +is a display label being handed back, and those came from `labelFor` against the +same ordered list. Keep it that way. + +**The picker's cap was saturated, and depth alone did not fix it.** Measured: +four of the six report deliverables still fell off the end of a 600-entry +newest-first list. Coverage had to become a property of the *enumeration* — each +project is asked for its own files, with its own small cap — not of the limit. diff --git a/umtool/docs/index.md b/umtool/docs/index.md @@ -0,0 +1,75 @@ +# The index + +`CACHE_DIR/index/projects.mdb`, LMDB, entirely optional. + +## Honest sizing + +At twelve projects this saves **50 to 150 ms** per load. It is not a speed fix +today and is not presented as one. What it buys: + +- **`--since`** — an agent asking what changed is a range read, not a diff of two + full scans. +- **recency as a range read**, for when the tree is 500 projects. The key is + `[MAX - mtimeMs, id]`, so ascending *is* newest-first. +- **decision counts without running the reducer** — the cost that grows fastest, + being the only part of a project read that touches megabytes of cue files. + +## The rule + +> **If a value exists only in the index, that is a bug.** + +It is in the code, because it is what keeps `lib/browse.ts`'s "No database. The +filesystem is the model." true. + +**FS first, index after, best effort**, in a swallowed try/catch. A crash between +the two leaves a signature that no longer matches, which the next read repairs. A +stale index self-heals and the user sees nothing but latency. + +Index-**first** could claim something the filesystem does not say. That is the one +failure this refuses. + +## Freshness signs INPUTS + +`sha1(schema + kind signature + mtimes + sizes)`. Never the produced record, which +would be circular; never bytes, which the export build already learned about. + +The schema folds into every signature so a bump invalidates everything — +deliberately **not** a generation counter, which would invalidate every project +whenever any one of them changed. + +Every read verifies. A record whose signature no longer matches is discarded, not +migrated. + +## Degradation + +A missing or unopenable store returns a **no-op** whose `get()` is `null` and +whose `put()` does nothing — copied in posture from +`common/lib/channelSignature.ts`. A fresh checkout, a deleted cache and a machine +without the native module all take that path, and everything still works. + +## Observing it + +It is deliberately almost invisible, so its health has to be surfaced on purpose: + +- the footer note on `/browse` — `index: 9/11 fresh` +- the `x-index` header on `/api/browse/projects` +- `umtool index` — records, schema, when it was built +- `umtool index --rebuild` / `--prune` / `--since <ms>` + +Three specs hold it to its contract: the header reports what it served, deleting +the `.mdb` produces byte-identical page data, and a manifest edited behind its +back is re-read rather than served stale. + +## Discovered by getting it wrong once + +**`lmdb` has to be a direct dependency of umtool.** The plan said to import it +through `common`, which already depends on it — but under pnpm's strict resolution +it does not resolve from umtool at all, and the CLI (plain node, no bundler) +cannot import a bridge written in TypeScript. pnpm dedupes it to the same store +entry anyway. + +**`lmdb` has to be in `serverExternalPackages`.** Bundled, Turbopack tries to +resolve its `moduleRequire('cbor-x')` — an *optional* dependency it only reaches +for an encoding nothing here uses — and fails the whole module graph. Every page +importing `lib/projects` then 500s naming a package that is not involved. Caught by +e2e, not by `tsc` and not by a build run before the wiring. diff --git a/umtool/docs/mix-from-a-project.md b/umtool/docs/mix-from-a-project.md @@ -0,0 +1,84 @@ +# Reaching /mix from a project + +Six of seven projects used to resolve to `null` in the mix bench: `MEDIA_ROOTS` +was the um-song subtree, so no report video's media was openable at all. + +## The link + +``` +/mix?body=<absolute>&start=<s>&end=<s>&from=<projectId>&clip=<clipId> +``` + +- **`body` is absolute**, like the picker's own `<option value>`. That is what + dodges the relative-binding hazard: a relative path is tried against each root + in order and never stats, so it binds to the first root it *could* live under. +- **`from` and `clip` are provenance** — the back-crumb and the header line. They + are deliberately not used to resolve media; that would be a second, divergent + resolver for what the first one already did. +- A deliverable link carries no window: the whole thing is the point. + +Resolved **server-side** in `app/mix/page.tsx`, which is why this needs no +`useSearchParams` and no Suspense boundary — that page already ran on the server, +it simply never read its own `searchParams`. + +## What a clip opens, and why + +The **widest cached raw window**, not the built segment. + +1. It exists as soon as the clip has been fetched once; segments only exist after + a build. +2. It has **no chrome burned in** — which is what a mix is looking at. +3. It is the file the bench already has peaks for. + +The built segment is the fallback, and the link says which it is. A clip with +neither renders **no link at all** rather than a dead one that 400s. + +## The prefill + +`start = clip.start - fetchStart`, `end = clip.end - fetchStart`, computed +server-side because the arithmetic is exact and known there; making the client do +it would be a second place to get it wrong. + +The bench then says *"this window came from c01 — the file itself runs +0.00–9.00s"*, so the material outside the window does not look unreachable. +Dragging past the end is harmless: `resolveMix` clamps `end` to the body's +duration, and `normaliseSpec` clamps `start` to ≥ 0. + +## Precedence + +**preset > the saved pair > blank.** A link that named a file and a window must +not lose to whatever the bench was last pointed at. The per-pair knobs — handover, +fade, gain, duck — *are* adopted, because those are things learned about that +pair. + +Arriving via a link does **not** write a session. A session is saved only on a +real render, so drive-by navigation cannot change what the next bare `/mix` visit +opens. + +## Refuse, never clamp + +A `body` outside the roots, or one with no audio track, produces a bench with no +preset and a **visible reason**. Silently opening a different file than the link +named is the one outcome worse than an error, and `lib/mix.ts` already takes this +line for the same reason. + +## The picker + +Grouped by project via `<optgroup>` — zero client JS, still a native select — with +a one-line text filter above it. + +Coverage is a property of the **enumeration**, not of the cap: each project is +asked for its own files with its own small cap, so one busy project cannot push +every other off the end. Measured before this: four of six report deliverables +fell off a 600-entry newest-first list, and per-song cuts never appeared at all. + +A report project whose only media is `out/clips-raw` correctly offers **nothing** — +those are intermediates, excluded by name. + +## Where a mix lands + +`SONG_REPORTS`, still. Reports became **readable**, not writable — see +[folders.md](folders.md). A mix of a report clip therefore lands in the um-song +deliverables directory, which is not ideal, and the trade was deliberate: the +alternative made every report's `out/` a legal render target, one typo away from +overwriting a 46 MB deliverable that cost an hour of fetches. diff --git a/umtool/docs/projects.md b/umtool/docs/projects.md @@ -0,0 +1,115 @@ +# Projects + +A **project** is a directory under `REPORTS_ROOT` that holds one piece of work. +umtool finds them by walking the tree; nothing registers itself and nothing had +to move on disk for this to exist. + +## Kind vs template + +A **kind** is what a thing *is*. A **template** is which configuration of that +kind it is. `um-song` is not a kind — it is the one template of the `song` kind +that exists so far, and the distinction is the whole point: the next thing this +tool has to hold (a supercut, a cover set, a vertical short) will be a new +template of an existing kind at least as often as a new kind. + +Three kinds ship: + +| kind | template | marker | what it is | +|---|---|---|---| +| `report-video` | `cited-timeline` | `video.manifest.json` | a sweep report said in the sources' own voices | +| `song` | `um-song` | `spec.json`, `verdicts.json`, or any cut | filler sounds playing a game tune | +| `sweep-report` | `sweep` | a `*sweep*.md` **and no manifest** | a report that is not yet a video | + +`sweep-report` earns its place on day one because `~/reports/hasan-bike` is one, +and because a kind with no decisions, no build and no rich read is the cheapest +possible proof that the registry is extensible. + +## Detection + +`project.json` (`{kind, template}`) wins outright, so a directory can always +declare itself. Otherwise every kind's `detect()` runs against the directory's +entry NAMES — no reads, no stats. + +**Two matches is an error, never a guess.** A directory that is two kinds is a +bug, and picking one would hide it; it gets an `ambiguous-project` decision +instead. + +## Identity: an id is a PATH + +A project's id is its POSIX path relative to `REPORTS_ROOT` — +`ferret-rescue`, or `quartering-uh-song/videos/yoshi`. Not the basename: bare +names collide across folders (a second `pokemon` is a matter of time) and the +path is what makes a link stable. + +A **bare basename still works as a URL**, because `/browse/yoshi` and +`/browse/yoshi/wide` are the addresses that exist in every decision href, every +spec, and whatever anybody has open. It resolves **only when unique** — two +projects sharing a name is precisely why an id is a path, so that case reports +the collision and offers both canonical URLs rather than picking one. + +## Two ways a project cannot be opened + +Both used to fail **silently**, which is worse than either. + +**`shadowed`** — the first path segment is a static page under `app/browse/` +(`decisions`, `faces`, `find`, `sources`, `trim`, `at`). A static segment beats a +dynamic one, so `/browse/find` renders the phrase console no matter what is on +disk. The project is still listed, with a `blocking` decision, and its link goes +to `/browse/at?path=…`. + +**`unroutable`** — the name fails `isSegment()` (a space is enough). It has no +address of its own; it is listed with an `info` decision and reached the same way. + +`RESERVED_BROWSE` is asserted by an e2e spec to equal the real directory listing +of `app/browse/`, so a tenth tool page cannot quietly make a project unreachable. + +## Adding a kind + +**A registry entry and one view. Nothing else.** + +1. Add an entry to `lib/projects/kinds.mjs`: `id`, `template`, `label`, `badge`, + `detect(names)`, `stages`, `decisionKinds`, and optionally `summarise`, + `signature`, `decisions`, `views`. +2. Add a component to `components/projects/` and a case in `ProjectView.tsx`. + +`e2e/projects.spec.ts` enforces this three ways, and they are the assertions that +fail when somebody special-cases a kind in a page — which no page test can see: + +- **no kind id is special-cased outside `lib/projects/` and + `components/projects/`** — a grep for lines carrying both a kind id and the + word `kind`; +- **`RESERVED_BROWSE` equals the real static pages**; +- **a kind injected through `UMTOOL_EXTRA_KINDS` reaches the index, the chips and + the CLI with no code edit at all**. + +## Where the code is + +| file | what | +|---|---| +| `lib/projects/kinds.mjs` | the registry. Plain ESM, no TypeScript | +| `lib/projects/walk.mjs` | the folder walk, routing, collapsing | +| `lib/projects/{report,song,sweep}.mjs` | per-kind reading | +| `lib/projects/core.mjs` | the CLI's assembled view | +| `lib/projects.ts` | the app's façade: caching, dispatch, the index | +| `lib/project-types.ts` | types only, **no `node:` import ever** | + +## Discovered by getting it wrong once + +**`lib/projects/kinds.mjs` is server-only**, despite being plain ESM. It imports +the per-kind modules and those read the disk, so importing it from a client +component drags `node:fs` into the browser bundle — which this repo has already +been bitten by: it passed `tsc --noEmit` and then 500'd every page. A client +component takes `lib/project-types.ts` and gets the rest as props. **`pnpm build`, +not typecheck, is what catches a regression here.** + +**Watch the import cycle.** `songIds()` lives in its own module +(`lib/projects/song-ids.mjs`) because putting it in `song.mjs` closes +`song → walk → kinds → song`. Plain node survives that; Turbopack evaluates +`kinds.mjs` while `song.mjs` is still initialising and every song page 500s with +"Cannot access 'CUT_NAMES' before initialization". `pnpm build` does not see it +either — nothing prerenders. A page render does. + +**Do not write a root path out by hand.** The song-decision dispatch had the +production path hard-coded, so under the e2e fixture — whose songs live elsewhere +— *no song decisions were produced at all* and the inbox looked clean because it +was empty. diff --git a/umtool/e2e/projects.spec.ts b/umtool/e2e/projects.spec.ts @@ -443,3 +443,33 @@ test("a project that changed on disk is re-read, not served stale", async ({ req const after = (await (await request.get("/api/browse/projects")).json()) as typeof before; expect(idOf(after, "reports/gone-fixture").facts).toContain("2 clips"); }); + +test("umtool new scaffolds a project that check immediately blocks", () => { + const dir = path.join(FIXTURE, "scaffold-root"); + rmSync(dir, { recursive: true, force: true }); + const env = { ...cliEnv, REPORTS_DIR: dir }; + const run = (args: string[]) => + execFileSync("node", ["bin/umtool.mjs", ...args], { cwd: UMTOOL, encoding: "utf8", env }); + + run(["new", "scaffolded", "--json"]); + const m = JSON.parse( + readFileSync(path.join(dir, "scaffolded", "video.manifest.json"), "utf8"), + ) as { timeline: unknown[]; provenance: { siteOrigin: string } }; + + // An EMPTY timeline on purpose. A report records ONE second per citation; a + // window needs a start and an end from the cue file, and generating guesses + // would look finished and be wrong. + expect(m.timeline).toHaveLength(0); + // And an empty siteOrigin, so the field that shipped broken twice cannot be + // left plausible-looking. + expect(m.provenance.siteOrigin).toBe(""); + + let code = 0; + try { + run(["check", "scaffolded"]); + } catch (e) { + code = (e as { status: number }).status; + } + expect(code).toBe(1); + rmSync(dir, { recursive: true, force: true }); +});