commit 44109436af680716625067efa9fa84aeab0b6a7c
parent 253305dfafb76663c43502d8f9780d96319491be
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Wed, 30 Sep 2026 20:18:10 -0400
Merge main (bbf1daf8) into r13/lows-editor — release 13 slice W1 brought to today's main
One conflict: editor/CHANGELOG.md, where the 0.11.0 cut renamed the old
[Unreleased]; W1's four bullets sit under a new [Unreleased] above [0.11.0].
doctor.ts/doctor.test.ts (DT, SG and HP changed them), HomepageBuildButtons.tsx
(the source-mirror note), editor/package.json (DS's start script) and FACTS
merged cleanly with both sides kept.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
341 files changed, 32834 insertions(+), 6763 deletions(-)
diff --git a/.dockerignore b/.dockerignore
@@ -17,9 +17,14 @@ export/public/summaries/
export/public/transcripts/
export/public/transcripts.tar.gz
export/public/transcripts.tar.xz
-# The source snapshot served by the project site — regenerated by
-# create-archives.sh on the host, never needed inside a build container.
+# The published source (`archilyzer source publish`, run by `archilyzer build
+# homepage`): the tarball, the git mirror and the raw tree (~66 MB), and the
+# skip key beside them, which holds the rules hash. All are regenerated on the
+# host; none is baked into an image, where a plain `next build` would ship a
+# copy that today's rules were never applied to.
homepage/public/downloads/
+homepage/public/source/
+homepage/.source-publish.json
# Build staging / caches / per-site outputs — bind-mounted at run time, never baked.
export/.export-index/
diff --git a/.gitignore b/.gitignore
@@ -46,7 +46,8 @@ yarn-error.log*
# nested transcripts repo
/transcripts
-# downloadable transcript archives (see the commented block in create-archives.sh)
+# downloadable transcript archives (an old whole-corpus format; ignored so a
+# stray copy is never committed)
/export/public/transcripts.tar.gz
/export/public/transcripts.tar.xz
@@ -90,10 +91,22 @@ yarn-error.log*
/homepage/public/channel-sites.json
/homepage/public/chart-templates.json
/homepage/public/homepage-summary.json
-# source snapshot tarball + its sidecar (regenerate with ./create-archives.sh).
+# source snapshot tarball + its sidecar (regenerated by `archilyzer source
+# publish`, which `archilyzer build homepage` runs).
# Note this does NOT cover /homepage/public/_headers — the `_headers` rule above
# is scoped to /export/public, so the homepage's stays tracked.
/homepage/public/downloads
+# The published source: the git mirror (archilyzer.git: packs, info/refs), the
+# raw tree and manifest.json, all written by `archilyzer source publish`
+# (common/publish/source.ts). Ignored for Tailwind as much as for git: Tailwind
+# v4 scans every file .gitignore does not exclude, and a pack file is binary —
+# the umtool `.next-*` failure below, where a scanned binary yields "class
+# names" that break the stylesheet. homepage/tsconfig.json excludes `public`
+# for the same reason: the raw tree is ~2,000 .ts/.tsx files of the whole repo.
+/homepage/public/source
+# The publish step's skip key (rules hash + source commit). Kept out of
+# public/ on purpose: a hash of the denylist is a guess-confirmation oracle.
+/homepage/.source-publish.json
# editor e2e fixtures and ephemeral state
/editor/test-transcripts/
@@ -128,6 +141,12 @@ yarn-error.log*
/homepage/out/
# the e2e's synthetic summary, and a killed run's temp copy (homepage/e2e/fixture-summary.ts)
/homepage/e2e/.e2e-summary.json*
+# the e2e's settings.json (social links only), and its temp copy (homepage/e2e/fixture-social.ts)
+/homepage/e2e/.e2e-settings.json*
+# the e2e's empty sites directory (no homepage.json; homepage/playwright.config.ts)
+/homepage/e2e/.e2e-sites/
+# the e2e's fixture publish for /source/ (homepage/e2e/fixture-source.ts)
+/homepage/e2e/.e2e-source/
# site config
/settings.json
diff --git a/AGENTS.md b/AGENTS.md
@@ -317,7 +317,7 @@ table is a convenience and will drift:
| `hasanalyzer` | Hasanalyzer | https://hasanalyzer.pages.dev |
| `anilyzer` | Anilyzer | https://anilyzer.pages.dev |
| `bonnellyzer` | Bonnellyzer | https://bonnellyzer.pages.dev |
-| `jasolyzer` | Jasolyzer | *(no `siteUrl` — not published)* |
+| `jasolyzer` | Jasolyzer | https://jasolyzer.pages.dev (launched 2026-09-28, exports off) |
Jeralyzer is the largest and most popular instance (30 channels / 30,886 videos at its
2026-08-07 build). The project's own site is `https://archilyzer.pages.dev`
@@ -339,6 +339,26 @@ The editor's `instrumentation.ts` arms the runners on boot, which resumes long-r
sweeps against production data. To exercise `common/` controllers over the real corpus,
run them offline with `tsx` instead of starting a server.
+# The source mirror
+
+`archilyzer source publish` (`common/publish/source.ts`), which `archilyzer build homepage` and
+the `/sites` Homepage jobs run, puts this repo on the project site read-only:
+- a fresh clone of `main`, scrubbed by git-filter-repo;
+- audited against a denylist, every object and every file;
+- published as a dumb-HTTP mirror at `/source/archilyzer.git/`, a raw tree and a tarball.
+
+**A refusal withdraws the previous publish** from `public/` and `out/`, and `deploy homepage`
+refuses a source it cannot vouch for.
+- **The operator's two files live OUTSIDE the repo** (`~/.config/archilyzer/source-scrub.txt`,
+ `source-denylist.txt`). **Never print, cat, quote, log or commit them** — they hold the private
+ strings the gate keeps off the site. Code reads them; you may count lines or check a mode.
+- **Never bypass the gate.** No denylist or scrub file of your own, no hand-edited
+ `homepage/.source-publish.json`. A refusal is fixed by an operator scrub rule, or by removing the
+ text from history.
+- **Never publish a path segment named `.git`:** wrangler's upload drops it silently.
+
+Everything else is in [PUBLISH.md](PUBLISH.md), "The source mirror (homepage)".
+
# Roadmap
Long-running work on the local-AI derived corpus is tracked in [PLAN.md](PLAN.md).
diff --git a/ENVIRONMENT.md b/ENVIRONMENT.md
@@ -12,7 +12,7 @@ The one override surface for where things live and which binary runs. Every one
| Variable | Default | What it does | Read by |
|---|---|---|---|
-| `TRANSCRIPTS_DIR` | `<repo>/transcripts` | The corpus: channels, sites, the LMDB index, job logs, the saved-video store. | common/lib/paths.ts (getPaths) |
+| `TRANSCRIPTS_DIR` | `<repo>/transcripts` | The corpus: channels, sites, the LMDB index, job logs, the saved-video store. umtool reads `<it>/channels` too, when its own `CHANNELS_DIR` is unset. | common/lib/paths.ts (getPaths) |
| `SAVED_VIDEOS_DIR` | `<TRANSCRIPTS_DIR>/saved-videos` | The persisted source-video store, when it should live on another disk. | common/lib/paths.ts (getPaths) |
| `SITES_DIR` | `<TRANSCRIPTS_DIR>/sites` | Per-site config (`<id>/site.json`, every key in [SITE.md](SITE.md)) and the homepage's `_homepage/`. | common/lib/paths.ts (getPaths) |
| `SETTINGS_FILE` | `<repo>/settings.json` | The settings file (every key in [SETTINGS.md](SETTINGS.md)). | common/lib/paths.ts (getPaths) |
@@ -39,6 +39,12 @@ The one override surface for where things live and which binary runs. Every one
| `GALLERY_DL_BIN` | `gallery-dl` on PATH | The X/Twitter post fetcher, for social channels. | common/lib/paths.ts (getPaths) |
| `OLLAMA_URL` | `http://127.0.0.1:11434` | The local ollama server, the local digest and attribution engine. | common/lib/paths.ts (getPaths) |
| `CLAUDE_BIN` | `claude` on PATH | The `claude` CLI, driving the opt-in metered digest lane. | common/lib/paths.ts (getPaths) |
+| `ARCHILYZER_CONFIG_DIR` | `~/.config/archilyzer` | The operator's private config dir, outside the repo: the two inputs of `archilyzer source publish` below. Never committed. | common/lib/paths.ts (getPaths) |
+| `SOURCE_SCRUB_FILE` | `<ARCHILYZER_CONFIG_DIR>/source-scrub.txt` | git-filter-repo `lhs==>rhs` rules applied to file contents AND commit messages when the source mirror is generated (`<home dir>==>/home/user` is built in and runs first). Every rule's left side is also denied. See [PUBLISH.md](PUBLISH.md). | common/lib/paths.ts (getPaths) |
+| `SOURCE_DENYLIST_FILE` | `<ARCHILYZER_CONFIG_DIR>/source-denylist.txt` | Literals the published source must never contain, one per line (`i:` = any case). One hit anywhere in the mirror, the tree or the tarball refuses the publish. | common/lib/paths.ts (getPaths) |
+| `ARCHILYZER_SOURCE_SCRATCH` | the OS temp dir | Where `source publish` makes its scratch clone and stage (removed afterwards unless `--keep-scratch`). | common/lib/paths.ts (getPaths) |
+| `XDG_CACHE_HOME` | `~/.cache` | The cache root: `source publish` keeps the history pages' render cache in `<it>/archilyzer/source-history/` (about 140 MB; never inside the checkout). | common/lib/paths.ts (getPaths) |
+| `STAGIT_BIN` | `stagit` on PATH, then `~/.local/bin/stagit` | stagit, which renders the source's history pages (`/source/git/`: the log and a page per commit with its diff). Optional: without it the source is published without them. See [PUBLISH.md](PUBLISH.md). | common/lib/paths.ts (getPaths) |
## Runtime
@@ -71,6 +77,9 @@ Tokens, credentials and knobs a running process reads. Most configuration is not
| `TRANSCRIPT_PLATFORM_LINKS` | off | `1` cites platform watch pages instead of the archive's own pages. | common/lib/archive/reader-fs.ts |
| `AUDIO_CHECK_RESUME_DURING_PROBE` | the channel's `audioCheck.resumeDuringProbe` | `1` or `true` resumes yt-dlp during the audio check's probe, anything else holds it, for a one-off comparison run; unset = the channel's setting. | common/ytdlp/audioCheckedDownload.ts |
| `AUDIO_CHECK_BACKOFF_FACTOR` | the built-in factor | The audio check's interval backoff factor, in (0, 1], for a one-off run. | common/ytdlp/audioCheckedDownload.ts |
+| `ARCHILYZER_STATS_ALLOW_DOWNGRADE` | off | `1` lets a stats build clear a stats cache that a NEWER build wrote, for a deliberate rollback. Unset, such a build refuses and names both versions. | common/controller/buildStats.ts |
+| `ARCHILYZER_INDEX_ALLOW_HELD` | off | `1` lets a FULL index rebuild (a schema change, or no index yet) proceed while a channel's media cannot be read; that channel stays out of the index until its media is back and the index is built again. Unset, such a build refuses and names each channel. | common/controller/buildIndex.ts |
+| `UV_THREADPOOL_SIZE` | `16` for the editor (`4` is Node's own) | Threads in Node's pool for filesystem calls. A call on a stalled drive holds one until the drive answers, so the editor starts with 16. It buys time for calls already in flight and isolates nothing: the storage health probe and its gate keep new calls off a stalled drive. | Node's libuv (set by editor/package.json `start` and docker/entrypoint.sh) |
| `MCP_IO_STATS` | off | `1` turns on per-call I/O accounting, for `mcp/bench`. | common/lib/archive/io-stats.ts |
| `ARCHILYZER_EDITOR_URL` | `http://localhost:3001` | Which editor `pnpm ops` and the MCP's `fetch_clip` talk to. | scripts/archilyzer-ops.mjs, mcp/src/fetchClip.ts, umtool |
| `ARCHILYZER_AGENT` | `cli` | Who is asking, recorded as the provenance of a curated-tag write through `pnpm ops`. | scripts/archilyzer-ops.mjs |
@@ -123,7 +132,7 @@ The publish pipeline sets these for a process it spawns. Listed so a reader know
| `INSTANCE_MODE` | a site | `hub` makes the export build the hub. Set by `archilyzer build hub`. | export/app/lib/mode.ts, common/lib/archive/contract.ts |
| `BUILD_ARCHIVES` | on | `0` skips archive-zip generation for one build (`--skip-archives`). | common/bin/compose-site.ts, common/bin/build-archives.ts |
| `ARCHIVES_READONLY` | off | `1` inside a docker-mode build container: materialize archives, never write the shared cache. | common/bin/compose-site.ts |
-| `HOMEPAGE_PUBLIC_DIR` | `<repo>/homepage/public` | Where `compose homepage` writes. | common/bin/compose-homepage.ts |
+| `HOMEPAGE_PUBLIC_DIR` | `<repo>/homepage/public` | Where `compose homepage` and `source publish` write. | common/bin/compose-homepage.ts, common/publish/source.ts |
## Docker
@@ -182,4 +191,6 @@ Read only by a test harness, a fake binary or a test-mode branch. Never set one
| `E2E_RETRIES` | `0` | Retries per shard (`--retries N` wins); 0 keeps a sharded run comparable to a serial one. | scripts/run-sharded-e2e.mjs |
| `E2E_IMAGE` | `yt-dlp-transcript-browser-e2e` | The sharded e2e run's image tag. | scripts/run-sharded-e2e.mjs |
| `E2E_SKIP_BUILD` | off | `1` reuses the sharded e2e image instead of rebuilding it (`--no-build`). | scripts/run-sharded-e2e.mjs |
+| `E2E_SOURCE_PUBLIC_DIR` | unset (the page reads `homepage/public`) | The fixture publish the homepage's e2e dev server reads the `/source/` page from while it holds a manifest; set by `homepage/playwright.config.ts`, written and removed by `homepage/e2e/source-history.spec.ts`, ignored by a production build. | homepage/app/lib/source.ts |
+| `E2E_EXPECT_SOURCE` | off (both states pass) | `1` makes the homepage suite's `/source/` specs fail on the empty state; a gate that ran `archilyzer source publish` first sets it. | homepage/e2e/source.spec.ts |
| `E2E_HOMEPAGE_SUMMARY_FILE` | `homepage/public/homepage-summary.json` | The synthetic summary the homepage's e2e dev server reads; set by `homepage/playwright.config.ts`, ignored by a production build. | homepage/app/lib/summary.ts |
diff --git a/PUBLISH.md b/PUBLISH.md
@@ -8,6 +8,7 @@ is in [ENVIRONMENT.md](ENVIRONMENT.md).
- [What gets published](#what-gets-published)
- [The three ways to drive it](#the-three-ways-to-drive-it)
- [Cloudflare Pages](#cloudflare-pages)
+- [The source mirror (homepage)](#the-source-mirror-homepage)
- [Preview deployments](#preview-deployments)
- [Download archives and R2](#download-archives-and-r2)
- [Securing downloads against cost-abuse](#securing-downloads-against-cost-abuse)
@@ -25,7 +26,7 @@ server-side code, servable by anything:
|---|---|---|---|
| A **site** | The export app over the channels one site selects: search, transcripts, charts, downloads. One corpus can publish several. | `export/out` (docker mode: `export/.export-builds/<siteId>/out`) | `transcripts/sites/<id>/site.json` ([SITE.md](SITE.md)) |
| The **hub** | The export app in hub mode: federated search across the family of sites. | `export/out` | `transcripts/sites/_homepage/homepage.json` |
-| The **homepage** | The project's own site (`homepage/`). | `homepage/out` | — |
+| The **homepage** | The project's own site (`homepage/`), with the source mirror, its raw tree, its history pages and the source tarball. | `homepage/out` | `~/.config/archilyzer/` (the source mirror's two operator files) |
The hub and the homepage are two different Pages projects: the hub deploys to the
project `homepage.json` names (e.g. `archilyzer-hub`) and is refused `archilyzer`,
@@ -50,7 +51,8 @@ entry points in `common/publish/build.ts`.
| Build, then deploy | *Build & deploy* | `build-deploy` | `build site <id>` then `deploy site <id>` |
| Build every site | /sites → **Build all sites** | — | `build all [--skip-archives]` |
| The hub | /sites → Hub → **Build hub** / **Deploy hub** | `build-hub`, `deploy-hub` | `build hub`, `deploy hub [--preview <branch>]` |
-| The homepage | /sites → Homepage → **Build homepage** (tick *Deploy after build*) / **Deploy homepage**, with an optional preview branch | `build-homepage` (`{"deploy":true}` to deploy after), `deploy-homepage` (`{"preview":"<branch>"}`) | `build homepage`, `deploy homepage [--preview <branch>]` |
+| The homepage | /sites → Homepage → **Build homepage** (tick *Deploy after build*) / **Deploy homepage**, with an optional preview branch | `build-homepage` (`{"deploy":true}` to deploy after), `deploy-homepage` (`{"preview":"<branch>"}`) | `build homepage [--no-source]`, `deploy homepage [--preview <branch>]` |
+| The source mirror alone | — (every homepage build runs it) | — | `source publish [--force] [--check] [--keep-scratch]`, `source audit [<git dir>]` |
`pnpm archilyzer <command>` is the short form of
`pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts <command>`;
@@ -86,6 +88,276 @@ fits inside Cloudflare's **free tier**.
preview path passes `--branch`; `deploy homepage` passes `--branch main`. If a
"production" deploy did not go live, check what branch the checkout is on.
+## The source mirror (homepage)
+
+The homepage carries the project's own source, read-only, as static files:
+
+> **Before ANY deploy of the homepage — a preview included — the denylist must hold
+> everything private.** A Pages preview is a public publication, and every Pages
+> deployment stays reachable at its own `<hash>.<project>.pages.dev` URL until that
+> deployment is deleted; a preview's branch alias is guessable (the plans that name
+> it ship in the mirror). If a deploy ever carries something private, DELETE that
+> deployment in the Pages dashboard: a newer deploy does not remove the old one.
+
+| Path on the site | What |
+|---|---|
+| `/source/` | The page: the clone command, the mirror head, the private `main` it reflects |
+| `/source/archilyzer.git/` | A git repository for git's **dumb HTTP** protocol: `git clone https://archilyzer.pages.dev/source/archilyzer.git` |
+| `/source/tree/` | Every tracked file of `main`, raw, with a generated `index.html` per directory |
+| `/source/git/` | The history: `log.html`, a page per commit with its diff (`commit/<sha>.html`), `refs.html`, `files.html` (an index into the raw tree), `atom.xml` and `tags.xml` — rendered by **stagit** when it is installed (below) |
+| `/downloads/archilyzer-source.tar.gz` + `snapshot.json` | The same tree without history |
+| `/source/manifest.json` | What was published, from which private commit, audited how |
+
+**`archilyzer source publish`** makes them, and **`archilyzer build homepage` runs it**
+between compose and `next build` — so do the editor's /sites homepage jobs and
+`pnpm ops build-homepage`, in the editor's own process and `PATH` (that process needs
+`~/.local/bin` on its `PATH` to find a pipx-installed `git filter-repo`; without it the
+step falls back to `pipx run`, which needs the network).
+
+**A refusal withdraws the source, everywhere it could ship from.** It fails the build
+before `next build`, and:
+
+- the step removes the LAST publish from `homepage/public` (manifest first, then the
+ mirror, the tree, the history pages, the tarball, `snapshot.json` and the skip key,
+ and the history's render cache) — it was audited
+ under rules that may not be today's, and the refusal is the best evidence that they
+ are not. Any outcome but success does this once the rules are loaded (an audit hit,
+ a limit, a missing tool, a cancel, a crash); `--check` writes nothing, this included;
+- the build removes the last BUILD's copy from `homepage/out` (`out/source`, the
+ tarball, `snapshot.json`);
+- **`deploy homepage` (and /sites → Deploy homepage) refuses** an `out/` holding a
+ source unless the skip key says that publish was made under today's rules and step
+ version, of today's `main`, by today's gitleaks, and is exactly the one in `out/`:
+ its mirror head, and a digest over every published file (the mirror, the tree, the
+ history pages, the manifest, the tarball, `snapshot.json` — sorted path, size and
+ sha256, recomputed over `out/`), so a mixed or edited `out/` refuses too: "run
+ `archilyzer build homepage` (it re-audits), then deploy". History pages that are
+ not the audited ones (edited, missing, or present when none were published) are
+ named on their own: "homepage/out's history pages (/source/git/) are not the ones
+ that were audited". An `out/` whose `/source` page shows the empty
+ state (`--no-source`) deploys as before; one with no `/source` page (a refused build)
+ does not.
+
+**After merging a change to this step, rebuild and restart the editor before any
+/sites Homepage job.** The editor runs its BUILT bundle: until it is rebuilt, its
+Build homepage job runs the old `buildHomepage` (without this step, or without the
+withdrawal) and its Deploy homepage job has no source check.
+
+`build homepage --no-source` (CLI only) removes the previously published source
+instead, because it was audited against the rules of its own day. A checkout with no
+git repository — the docker runtime, a tarball install — has nothing to mirror: the
+build says `no git repository here; nothing to mirror — the /source page will show its
+empty state`, removes any old publish and goes on; `source publish` run directly there
+exits 1 with the same sentence.
+
+What one publish does:
+
+1. A **fresh** `git clone --no-local --bare --single-branch --no-tags --branch main` of
+ the checkout's git *common* dir — so a worktree build mirrors the primary's `main`.
+ The private repository's history is never rewritten.
+2. **git-filter-repo** rewrites that copy: file contents (`--replace-text`) AND commit
+ messages (`--replace-message`) with the operator's scrub rules. Author and committer
+ identities are not rewritten — the gate still reads them.
+3. `git repack -a -d --max-pack-size=20m`, `pack-refs`, `update-server-info`: the
+ dumb-HTTP file set is `HEAD`, `refs/heads/main`, `packed-refs`, `info/refs`,
+ `objects/info/packs` and `objects/pack/*.{pack,idx}`, copied from an **allowlist** —
+ never `config` (it names the clone's origin path), `hooks/`, `filter-repo/` (the
+ PRIVATE commit ids) or `logs/`.
+4. **The gate** (`common/publish/sourceAudit.ts`): every object of the rewritten
+ mirror — blobs, commits and tags whole (messages and identity lines), trees by
+ entry name — then every staged file and path (the tarball decompressed), searched
+ for every denied literal; then gitleaks over the history, when it is installed.
+ **One hit and nothing is written** (and the last publish is withdrawn, above). The
+ report names a literal only by where it was written — `denylist line 3 (len 5)`,
+ `scrub line 2 lhs (len 11)`, `built-in home rule (len 11)` — never by any of its
+ characters, and a hit only by its object: kind, id, the blob's path in history, the
+ byte offset, and for a commit or tag the field (`author`, `committer`, `tagger`,
+ `message`). It prints no byte from the object, because what sits beside a denied
+ name (a surname, the rest of an address) is as private as the name, and every
+ refusal message and path it prints is masked (`[REDACTED]`), so a literal that spans
+ path components (`a/b`) is not printed by a refusal that names a tree path. It ends with
+ `add a rule to ~/.config/archilyzer/source-scrub.txt or drop the file from history,
+ then re-run.`
+5. The tree (`git archive` → `tar -x`, a page per directory; a tracked `index.html`,
+ a tracked `404.html` — Pages would serve it as HTML for every missing path below it —
+ or a symlink is a refusal) and the tarball (`git archive --format=tar.gz -9`). Then
+ the history pages (stagit, below), into the same stage — so the file sweep of step 4
+ reads every one of them too.
+6. Limits, then the install: link-safe (`common/bin/_publicFile.ts`), the manifest
+ removed first and written **last**, so a crash leaves the page's empty state,
+ never a manifest over a half-copied tree.
+
+An unchanged `main` with unchanged rules, step version, filter-repo and gitleaks, and
+published files that still match their digest, skips (`[source] up to date at …`;
+`--force` rebuilds). `--check` does everything but the
+install and writes nothing. `--keep-scratch` leaves the scratch clone
+(`ARCHILYZER_SOURCE_SCRATCH`, default the OS temp dir) for a look, minus
+`replace.txt` (the scrub rules), which is always deleted; a scratch root inside the
+checkout or the public dir is refused, since a kept scratch there could be committed
+or published. `archilyzer source audit [<git dir>]` runs the gate alone —
+over the published mirror by default, or over a clone: `archilyzer source audit
+<clone>/.git`.
+
+**The two operator files** live outside the repo, in
+`${ARCHILYZER_CONFIG_DIR:-~/.config/archilyzer}/` (`SOURCE_SCRUB_FILE` and
+`SOURCE_DENYLIST_FILE` override each path; [ENVIRONMENT.md](ENVIRONMENT.md)). They are
+never committed; keep them mode 600. A missing file is a refusal naming it.
+
+- `source-scrub.txt` — git-filter-repo `--replace-text` lines: `lhs==>rhs` (split at
+ the last `==>`), `literal:…`, `regex:…==>…`, `glob:…==>…`; a line with no `==>` is
+ replaced by `***REMOVED***`. Lines whose first non-blank character is `#` are
+ comments (the step drops them — filter-repo itself would treat one as a literal).
+ `<your home dir>==>/home/user` is built in and runs first (a trailing `/` on the home
+ dir is dropped). Rules apply in order. An empty left side is a refusal.
+- `source-denylist.txt` — one literal per line, `i:` in front for any ASCII case;
+ `#` comments.
+- In both, a UTF-8 byte-order mark and CRLF line endings are dropped first: either
+ would silently turn the first rule into one that matches nothing.
+- **Every literal scrub rule's left side is denied too** — its exact bytes, so a rule
+ that stopped matching because its text was mistyped refuses instead of leaking. A
+ NEW spelling in history (another case, another path) is caught only by the denylist:
+ deny the name itself, not just the paths it appears in.
+- The rules' hash (with the step's version) is the skip key and the deploy check, kept
+ in `homepage/.source-publish.json` — beside `public/`, never in it: a published hash
+ of the denylist would confirm a guess at it. For the same reason the manifest does
+ not say how many literals there are.
+
+**A known limit: compressed content is opaque.** The gate is a byte search, so it
+cannot see inside a ZIP member, a PNG's compressed text chunks, a PDF stream or a
+woff2 (the tarball is the exception: it is decompressed). And git-filter-repo never
+scrubs a blob with a NUL in its first 8 KiB, so a binary is audited, never scrubbed.
+At release 12 the review decompressed every such blob in the published history (a
+zip, PNGs and icons) and found no denied literal. A private string inside a binary
+must be removed from history by hand.
+
+**Tools.** `git filter-repo` (`pipx install git-filter-repo`; without it the step runs
+`pipx run --spec git-filter-repo==2.47.0`, which needs the network on first use, and
+without pipx it refuses with the install line). gitleaks is optional: without it the
+secret scan is skipped with a WARNING and the literal audit still runs. stagit is
+optional too (next section). `archilyzer doctor` has a "source publish" block: which
+filter-repo would run, stagit (its path, or not found, and the render cache's path and
+size), gitleaks, the two files (rule
+counts and modes, never contents) and the last publish.
+The mirror's ids are deterministic for a given filter-repo version; an upgrade that
+changes its rewriting changes every id (readers re-clone). The manifest records both
+tools.
+
+**The history pages (`/source/git/`) are stagit's.** [stagit](https://codemadness.org/stagit.html)
+(C over libgit2) renders the scrubbed mirror as static pages: the log, a page per commit
+with its diffstat and diff, the refs, and two Atom feeds. It is an operator-installed
+tool like git-filter-repo — never vendored, never committed. Install it once (libgit2
+and its headers are the one dependency):
+
+```sh
+git clone git://git.codemadness.org/stagit && make -C stagit && cp stagit/stagit ~/.local/bin/
+```
+
+The step finds it as `STAGIT_BIN`, else `stagit` on `PATH`, else `~/.local/bin/stagit`
+(the editor's process needs `~/.local/bin` on its `PATH` for a pipx `git filter-repo`;
+for stagit the fallback covers it). **Without it the source is published without the
+history**: one line says so and how to install it, the manifest has no `history`
+block, and `/source/` shows no History links. A render that fails, or history pages
+that would break the host's limits, are the same, with a WARNING — never a failed
+build, and the next build tries again (a stagit present and no history never skips).
+What the step does with it:
+
+- It runs after the mirror is built and its objects audited: `stagit -c <cache> -u
+ https://archilyzer.pages.dev/source/git/ <the scrubbed bare clone>` (past the cap,
+ `-l 10000` in place of `-c`, below). The clone is
+ named `archilyzer.git` (stagit names the repository after its directory); its
+ `description` is set to "Archilyzer" and its `url` to the clone URL, for stagit's
+ header. Neither file is published.
+- **Only an allowlist is published:** `log.html`, `files.html`, `refs.html`,
+ `atom.xml`, `tags.xml`, the page of each commit the log lists (`commit/<sha>.html`),
+ and a `style.css` the step writes from
+ `common/styles/tokens.css` (the homepage's two grounds, light and dark; diff
+ insertions in `--success`, deletions in `--destructive`). **stagit's per-file
+ pages (`file/…`) are not**: browsing is the raw tree, so every link into `file/` —
+ the Files index, the README and LICENSE links in the header, a diff's file names —
+ is rewritten to `../tree/<path>`.
+- Every `.html` page (never the feeds) gets exactly two additions: the homepage's
+ pre-paint theme script (the one `ThemeScript` emits, `lib/themeConfig.ts`), so a
+ page opens on the visitor's stored base or the homepage's dark default — without
+ JavaScript, `prefers-color-scheme` decides — and one line at the top, "Archilyzer ·
+ Source", linking to `/source/`. stagit's logo and favicon point at the site's
+ `/icons/icon-32.png`. A link to what is not published keeps its text and loses its
+ `href`: a diff of a file main no longer has (deleted or renamed since — stagit links
+ every diff side), and a commit with no page (past the cap, the oldest page's parent);
+ the raw tree's file list decides. Nothing else is changed; the pages are rewritten byte
+ for byte.
+- **The gate reads every page**: they are in the stage the file sweep reads, so a
+ denied literal in one refuses the publish, named by its path (`file
+ source/git/commit/<sha>.html (contents, byte N)`) and never by its bytes. The pages
+ are rendered from objects the object sweep already read, so a hit there means
+ something stagit or the step added.
+- The manifest's `history` block has the log's href, the commits with a page and the
+ commits in all (`commits`, `total`), the head, the file count and bytes, a sha256
+ over the pages, and stagit's identity (the sha256 of its binary; it has no version
+ flag). The skip key has stagit's identity and the pages' digest.
+- **The cap: the newest 10,000 commits** (`SOURCE_HISTORY_MAX_COMMITS`). Past it, stagit
+ runs with `-l 10000`: its log lists the newest 10,000 and ends "N more commits
+ remaining, fetch the repository", and `/source/` says "the latest 10,000 of M
+ commits". `-l` does not cap the pages — stagit still writes one for every commit —
+ so the step publishes the page of each commit the log lists and no other; and stagit
+ refuses `-c` with `-l`, so past the cap every publish computes 10,000 diffstats
+ (about 5 ms each here) where `-c` computes only the new ones. The 15,000-file drop
+ below stays, as the last resort.
+- **The render cache is `${XDG_CACHE_HOME:-~/.cache}/archilyzer/source-history/`**
+ (about 140 MB at 1,872 commits; made on first use; never inside the checkout or the
+ public dir — the step renders without it there, and says so). stagit keeps a commit
+ page once rendered, and `-c` keeps its log lines, so a publish renders only the new
+ commits. At 1,834 commits, stagit alone took 0.6 s with nothing new against 8 s for
+ every page; the whole history step (with the post-pass and the copy) 2 s against 8 s.
+ **But `-c`'s walk is in commit-date order and stops at the head it rendered last**, so
+ a `--no-ff` merge of commits older than that head leaves them out: the step's count
+ check sees the log come up short and renders every page again (about 8 s), with a line
+ that says so. With this repo's merges of parallel slices that is common, and correct.
+ The cache holds stagit's `-c` file, its output and a key. Its pages are kept only
+ under the same rules, step, filter-repo, stagit and header text, and after a run that
+ finished; its log lines only when they end at an ancestor of today's head. One
+ publish holds it at a time; a lock whose pid is not running, or over an hour old, is
+ stale and replaced (one line). A cache that cannot be made or written (EACCES, EROFS,
+ ENOSPC) is one line and a render without it, never a failed build. `--force` renders
+ every page again, `--check` never touches it (it renders in its own scratch), and a
+ refusal removes it. `archilyzer doctor` ends its stagit line with `cache: <path>,
+ <size>`.
+- **Its size, and the limit ahead.** At 1,872 commits (2026-09-30): 1,878 files,
+ 141.3 MB, the largest page 5.0 MB (stagit prints "Diff is too large, output
+ suppressed" past 1,000 files or 100,000 lines in one commit); the whole publish is
+ 4,396 files, and `homepage/out` 4,576. One file per commit counts against the step's
+ 15,000 (of Pages' 20,000); the cap keeps it at 10,006 at most. Should the rest of the
+ publish ever grow past 5,000 files, the history is dropped with a WARNING, never the
+ publish.
+
+**Cloudflare Pages, and the traps it sets.**
+
+- **Never a path segment named `.git`.** wrangler's upload ignore list drops `**/.git`
+ (and `**/node_modules`) *silently*, and Cloudflare's managed rules block `/.git/`
+ requests. The mirror is `archilyzer.git`, which passes both. Extensionless files
+ (`HEAD`, `info/refs`) upload and serve fine.
+- **The caps:** 20,000 files per deployment and 25 MiB per file. The step refuses at
+ **15,000 files** (the rest of the site needs room) and **24 MiB**; packs are split
+ at 20 MB.
+- **`_headers` makes the raw tree plain text.** wrangler's mime map would serve `.ts`
+ as `video/mp2t`, `.mjs` as JavaScript and `.html` as a page on this origin, so
+ `homepage/public/_headers` sets `/source/tree/*` to `text/plain; charset=utf-8` with
+ `nosniff` and `noindex`, then puts back `text/html` for the directory pages and the
+ real types of the few binaries (`.ttf`, `.m4a`, `.zip`, `.onnx`, `.svg` — the SVG
+ under a `default-src 'none'` CSP). `/source/git/*`, the history pages, is `noindex`
+ too (ruled), and keeps its own types. **Every matching rule applies, and a header a
+ later rule sets again is APPENDED** (`text/plain; charset=utf-8, text/html;
+ charset=utf-8`), so each of those overrides first detaches the tree's type with
+ `! Content-Type`. Only the ROOT `_headers` is read; the tree's own
+ copies are served as text. Locally, only `wrangler pages dev` honours `_headers` —
+ `next dev` and `serve` do not, and neither serves a directory's `index.html` at
+ `/<dir>/` the way Pages does.
+- **Tailwind and tsc must not see the mirror.** Tailwind v4 scans every file
+ `.gitignore` does not exclude (a binary pack yields "class names" that break the
+ stylesheet), and `homepage/tsconfig.json` includes `**/*.ts`: `/homepage/public/source`
+ is gitignored, and both `public` and `out` (where `next build` copies it) are excluded
+ from the tsconfig. Keep all three. `.dockerignore` leaves the mirror and the skip key
+ out of every image.
+
## Preview deployments
A **preview** is the same built bundle deployed to a branch that is not the Pages
diff --git a/README.md b/README.md
@@ -63,14 +63,14 @@ TRANSCRIPT_SITE_URL=https://jeralyzer.pages.dev \
```
There is nothing to compile — it runs from source through `tsx`. To register it with
-Claude Code:
+Claude Code, from the repo's root:
```bash
claude mcp add archilyzer \
--env TRANSCRIPT_SITE_URL=https://jeralyzer.pages.dev \
--env ARCHILYZER_EDITOR_URL=http://localhost:3001 \
--env WORKER_TOKEN=… \
- -- pnpm -C /ABS/PATH/TO/this/repo --filter yt-dlp-transcript-mcp exec tsx src/index.ts
+ -- pnpm -C "$PWD" --filter yt-dlp-transcript-mcp exec tsx src/index.ts
```
The two editor lines are optional: they let `fetch_clip` ask a local editor for clip
@@ -318,7 +318,8 @@ quietly sampling.
### Quickstart (no corpus required)
```bash
-pnpm install
+git clone https://archilyzer.pages.dev/source/archilyzer.git archilyzer # or the tarball on archilyzer.pages.dev/downloads/
+cd archilyzer && pnpm install
claude mcp add archilyzer \
--env TRANSCRIPT_SITE_URL=https://jeralyzer.pages.dev \
--env ARCHILYZER_EDITOR_URL=http://localhost:3001 \
@@ -477,13 +478,25 @@ Channels can sync automatically on a per-channel cadence via a cron heartbeat
Archilyzer is MIT licensed and deliberately **forge-neutral**: there is no `.github/`
directory, no provider-specific CI configuration, and nothing in the build that assumes
-a particular host. However you obtained this tree — a release archive, a mirror, or a
+a particular host. However you obtained this tree — a clone, a release archive, or a
fork on whichever platform — it is a complete, self-contained working copy.
-The project site publishes a dated source snapshot at
+**The canonical public copy is a read-only git mirror on the project site:**
+
+```sh
+git clone https://archilyzer.pages.dev/source/archilyzer.git
+```
+
+It is `main` only, with its whole history, served as static files over git's dumb
+HTTP protocol, and regenerated from the private repository by every homepage build
+(`archilyzer source publish`, which `archilyzer build homepage` runs). Its commit ids
+differ from the private repository's because machine paths are scrubbed on the way
+out; a gate refuses to publish anything that still carries a denied string. Every
+file is browsable raw at [archilyzer.pages.dev/source/tree/](https://archilyzer.pages.dev/source/tree/),
+and the same tree without history is a tarball at
[archilyzer.pages.dev/downloads](https://archilyzer.pages.dev/downloads/), with a
-checksum. `./create-archives.sh` regenerates one, writing a `snapshot.json` sidecar
-(commit, size, SHA-256) beside it.
+`snapshot.json` sidecar (commit, size, SHA-256). How it is built:
+[PUBLISH.md](PUBLISH.md#the-source-mirror-homepage).
See [CONTRIBUTING.md](CONTRIBUTING.md) to work on the code.
diff --git a/SETTINGS.md b/SETTINGS.md
@@ -449,6 +449,7 @@ Per entry — each entry spells its own values.
| `reason` | One reason today. A union so a second one has somewhere to go, and so a surface can say WHICH machine decided rather than "automatic". |
| `since` | ISO, for "auto-paused — media unreachable since <date>". |
| `previousTier` | The base tier the channel had before the machine paused it; what a restore puts back. Never `paused` (that would restore to paused — a no-op dressed as a restore). |
+| `cause` | `not-there` (the drive is unmounted or unplugged) or `not-answering` (it is there and does not answer: a stalled disk). Only what the words on /review, the rack and the channel page say. Optional: a record written before it existed reads as `not-there`. |
Default:
@@ -471,9 +472,10 @@ Per entry — each entry spells its own values.
| Key | Description |
|---|---|
-| `label` | Visible name, also the accessible label of the icon. |
+| `label` | The link's name: the icon's accessible name and its tooltip, never text beside it. Shown as text only in place of an icon that fails the check at render. |
| `url` | Link target: http(s), mailto: or a site-relative path. |
-| `svg` | Inline SVG markup. Normalized on save (width/height stripped, fill="currentColor", aria-hidden) and rejected when unsafe (script, foreignObject, event handlers, javascript: URLs) or when it has no viewBox. |
+| `svg` | Inline SVG markup: ONE well-formed `<svg>` element, checked when it is saved new or edited and again every time it is rendered (a link whose icon fails at render shows its label instead; `archilyzer doctor` names it). It may contain shapes, groups, defs, gradients, patterns, clip paths, masks, filters, text and animate/animateTransform/set — no script, style block, foreignObject, a, image, title, desc or any HTML element (a title or desc holding text only is removed); SVG presentation attributes plus aria-*, data-* and xmlns:* — no event handler (on…); a `style` attribute of presentation properties only; an href or url(…) only to an id inside the icon, written plainly; no CSS escape, comment or function that loads anything (image-set, image, cross-fade, element, src, paint, @import); ids plain names. A leading XML declaration, a DOCTYPE without an internal subset and comments are removed. Normalized on save: width/height stripped, aria-hidden added, a single-colour icon's fills made fill="currentColor" (an icon of two or more colours keeps them). A root with no viewBox but a numeric width W and height H (unitless or px) is given `viewBox="0 0 W H"`, so a file pasted as downloaded is accepted. A refused save names why; export from a drawing program with presentation attributes rather than a style block (in Inkscape, save as Plain SVG). |
+| `featured` | Keep this link in the header on small screens (the editor's "Keep in header on small screens"). A narrow header shows only the featured links (up to 4, the last 4 if more are marked; none marked → none, so the name has the room); a wide header shows every link, up to 4, the featured ones kept first, then the last of the rest. The footer shows every link. Written only when true. |
Default:
@@ -483,7 +485,7 @@ Default:
## `homepageUrl`
-Absolute public URL of the family hub (e.g. "https://archilyzer-hub.pages.dev"). Every export site links back to it ("the family" backlink) when set. Empty = no hub link rendered. Normalized to a trailing-slash-free http(s) URL.
+Absolute public URL of the family hub (e.g. "https://archilyzer-hub.pages.dev"). The default for every site's `hubUrl` (a site's own wins): published as `hubUrl` in the site's public `/site.json` and `/corpus.json`, so the hub can tell its member sites from arbitrary added origins. No page links to it (the header's Hub link was removed in release 14). Empty = none published. Normalized to a trailing-slash-free http(s) URL.
Default: `""`
@@ -520,6 +522,7 @@ Where a channel's downloaded media goes when it is relocated off the corpus disk
| `locations` | `[]` | The named storage locations a channel's media may be relocated to — one entry per root, each with an id, label, root, `autoRepoint` and the learned volume identity. Order is display order. Managed on /storage. |
| `defaultLocationId` | `""` | The location prefilled as the destination of a move. "" = no default. |
| `savedVideosLocationId` | absent | WHERE THE SAVED-VIDEO STORE IS, by location id. "" = in place, under the corpus at `paths.savedVideosDir`.<br><br>A RECORD OF WHAT IS ON DISK, never an intention — the same contract as a channel's `config.dataDir`. It is written by the move, on success, after the copy has verified and the symlink is in place; nothing else writes it, and a reader that disagrees with the disk trusts the disk. Optional so an older settings.json parses (and an older binary that drops it leaves a store that still works, because the symlink is what every reader follows). |
+| `health` | absent | THE DRIVE-HEALTH TIMINGS: how long a read may take before a drive counts as not answering, how often the health pass looks, how long its look may take, how many clean looks clear a stall, and how many reads may be on one drive at once. Edited on /storage (Drive health timing). Absent = every default, and only a value that differs from its default is written, so an untuned install follows a default changed later. See `storage.health` below. |
#### `storage.locations[]`
@@ -545,6 +548,16 @@ Per entry — each entry spells its own values.
| `mountpoint` | Where the volume was mounted at the last successful probe, and the path of the location's root RELATIVE to that mountpoint. Invariant: `root === join(mountpoint, relPath)`. Keeping the two halves is what lets a probe compute a candidate root when the volume reappears elsewhere. |
| `relPath` | The location root's path RELATIVE to `mountpoint` (see there). Invariant: `root === join(mountpoint, relPath)`. |
+#### `storage.health`
+
+| Key | Default | Description |
+|---|---|---|
+| `budgetMs` | `3000` | How long one read may take before the drive counts as not answering, in ms (default 3000, 500–60000). The watchdog's budget (`onDrive`, lib/storageHealth.ts) for one unit of work — a video directory's reads, a page's reads of one video: a unit that has not answered by then is refused, and marks its location not answering unless the disk's request counters show it still completing others (slow, not stalled). A read waiting for a slot is refused when nothing on the drive has returned for this long plus a quarter of it (at most 250 ms). Takes effect on the next read after a save. |
+| `passIntervalMs` | `15000` | How often the health pass reads each location's disk counters, in ms (default 15000, 5000–300000). A stall that starts between two passes is seen by the next, or at once by a page's read. A save on /storage re-arms the pass's timer at once; a hand edit, at the next pass. Two counter samples are compared only when at least min(10 s, this − 5 s) apart, a spacing never less than half of this. |
+| `probeTimeoutMs` | `3000` | How long the health pass waits, in ms (default 3000, 500–30000), for the child `stat` of a root where no disk can be named (a timeout counts as not answering) and for the `findmnt` that names a root's disk (a timeout names none that pass). Takes effect on the next pass. |
+| `clearAfterCleanPasses` | `2` | How many clean answers in a row clear a location marked not answering (default 2, 1–10). Each health pass is one answer, and so is a Refresh on /storage; a miss in between starts the count again. Takes effect on the next answer. |
+| `inFlightPerLocation` | `4` | How many reads through the watchdog may be on one location's drive at once (default 4, 1–8); the rest wait in the editor's own queue, so a stall mid-walk holds this many of Node's threads, not all of them. At most 8, half of `UV_THREADPOOL_SIZE` (16 in the editor's start script and the container), so one drive that stops answering cannot hold every thread. Takes effect on the next read. |
+
Default:
```json
diff --git a/SETUP.md b/SETUP.md
@@ -159,8 +159,9 @@ Caveats on native Windows:
- Building the **whisper.cpp / parakeet.cpp** transcription backends is Unix-oriented;
do transcription work under WSL2, or let the container do it (it ships a backend
already built).
-- `create-archives.sh` and the external-cron sync path (`pnpm sync:tick` from cron)
- assume a Unix shell — use WSL2 or Task Scheduler equivalents.
+- The external-cron sync path (`pnpm sync:tick` from cron) assumes a Unix shell — use
+ WSL2 or Task Scheduler equivalents. So does publishing the source mirror
+ (`archilyzer source publish`: git-filter-repo, `tar`).
---
@@ -168,8 +169,20 @@ Caveats on native Windows:
Archilyzer is MIT licensed and **forge-neutral** — there is no provider-specific CI
config or `.github/` directory, and nothing in the build assumes a particular host. Any
-complete copy of the tree works: a clone from wherever the project is mirrored, a fork
-of your own, or the dated source snapshot published by the project site:
+complete copy of the tree works. The canonical public copy is the read-only git mirror
+on the project site — `main` with its whole history:
+
+```sh
+git clone https://archilyzer.pages.dev/source/archilyzer.git archilyzer
+cd archilyzer
+```
+
+`git pull` updates it; nothing takes a push. Its commit ids differ from the private
+repository's (machine paths are scrubbed on the way out), and every homepage build
+regenerates it (`archilyzer source publish`; [PUBLISH.md](PUBLISH.md#the-source-mirror-homepage)).
+
+No git? The same tree without history is a tarball, with a `snapshot.json` sidecar
+(commit, size, SHA-256) beside it:
```sh
curl -LO https://archilyzer.pages.dev/downloads/archilyzer-source.tar.gz
@@ -178,9 +191,7 @@ cd archilyzer
```
The snapshot is a working tree at one commit — no history, no branches, no remote — so
-updating means fetching a newer one. Regenerate a snapshot yourself with
-`./create-archives.sh`, which also writes a `snapshot.json` sidecar (commit, size,
-SHA-256) beside it.
+updating means fetching a newer one.
Once you have a tree, from a clone or a snapshot alike:
diff --git a/SITE.md b/SITE.md
@@ -23,6 +23,7 @@ Regenerate this file with `pnpm --filter yt-dlp-transcript-common exec tsx bin/f
| [`cloudflareProject`](#cloudflareproject) | absent |
| [`accent`](#accent) | absent |
| [`siteUrl`](#siteurl) | absent |
+| [`listed`](#listed) | `true` |
| [`relatedSites`](#relatedsites) | `[]` |
| [`pwa`](#pwa) | `false` |
| [`archives`](#archives) | `true` |
@@ -77,9 +78,10 @@ Per entry — each entry spells its own values.
| Key | Description |
|---|---|
-| `label` | Visible name, also the accessible label of the icon. |
+| `label` | The link's name: the icon's accessible name and its tooltip, never text beside it. Shown as text only in place of an icon that fails the check at render. |
| `url` | Link target: http(s), mailto: or a site-relative path. |
-| `svg` | Inline SVG markup. Normalized on save (width/height stripped, fill="currentColor", aria-hidden) and rejected when unsafe (script, foreignObject, event handlers, javascript: URLs) or when it has no viewBox. |
+| `svg` | Inline SVG markup: ONE well-formed `<svg>` element, checked when it is saved new or edited and again every time it is rendered (a link whose icon fails at render shows its label instead; `archilyzer doctor` names it). It may contain shapes, groups, defs, gradients, patterns, clip paths, masks, filters, text and animate/animateTransform/set — no script, style block, foreignObject, a, image, title, desc or any HTML element (a title or desc holding text only is removed); SVG presentation attributes plus aria-*, data-* and xmlns:* — no event handler (on…); a `style` attribute of presentation properties only; an href or url(…) only to an id inside the icon, written plainly; no CSS escape, comment or function that loads anything (image-set, image, cross-fade, element, src, paint, @import); ids plain names. A leading XML declaration, a DOCTYPE without an internal subset and comments are removed. Normalized on save: width/height stripped, aria-hidden added, a single-colour icon's fills made fill="currentColor" (an icon of two or more colours keeps them). A root with no viewBox but a numeric width W and height H (unitless or px) is given `viewBox="0 0 W H"`, so a file pasted as downloaded is accepted. A refused save names why; export from a drawing program with presentation attributes rather than a style block (in Inkscape, save as Plain SVG). |
+| `featured` | Keep this link in the header on small screens (the editor's "Keep in header on small screens"). A narrow header shows only the featured links (up to 4, the last 4 if more are marked; none marked → none, so the name has the room); a wide header shows every link, up to 4, the featured ones kept first, then the last of the rest. The footer shows every link. Written only when true. |
## `groups`
@@ -146,7 +148,7 @@ Default: absent
## `accent`
-Per-site brand accent: a named accent id (`signal`, `brass`, `vermilion`, `violet`, `sakura`, `blue`, `green`) or a custom `"#rrggbb"`. It is the site's default accent — a reader can pick another. Absent = `signal`, the family default. A custom hex is darkened or lightened per base until it reaches 4.5:1. The public `/site.json` always carries a hex: an id is published as its on-dark value. Any other spelling is dropped.
+Per-site brand accent: a named accent id (`signal`, `brass`, `vermilion`, `violet`, `sakura`, `blue`, `green`) or a custom `"#rrggbb"`. It is the site's accent on every page; a reader does not pick one. Absent = `signal`, the family default. A custom hex is darkened or lightened per base until it reaches 4.5:1. The public `/site.json` always carries a hex: an id is published as its on-dark value. Any other spelling is dropped.
Default: absent
@@ -156,6 +158,12 @@ Absolute public URL of this site's deployment, e.g. `https://jeralyzer.pages.dev
Default: absent
+## `listed`
+
+Whether the family lists this site. Opt-OUT: absent/true = listed, only an explicit `false` is written. An unlisted site still builds and deploys as before, and its own pages are unchanged; it is left out of the homepage (cards, chart, `/stats`), the hub (members, federated search, `/corpus.json`, `/llms.txt`), every other site's footer, and the published `channel-sites.json` and pooled `stats/`. A channel only unlisted sites expose is in none of the family's public totals; a channel a listed site also exposes is credited to the listed one.
+
+Default: `true`
+
## `relatedSites`
Pulls specific siblings to the front of the footer's cross-site list, in named groups. Siblings not named here fall into a trailing "Other sites" group. Absent/empty = one flat list of every sibling.
@@ -207,6 +215,6 @@ Default: absent
## `hubUrl`
-Per-site override for the hub this site belongs under (the PWA it points visitors toward). Absent = the family default, `settings.json` `homepageUrl`. Surfaced on the public /site.json so a hub can tell member sites from arbitrary added origins.
+Per-site override for the hub this site belongs under. Absent = the family default, `settings.json` `homepageUrl`. Published on the public `/site.json` and `/corpus.json` so a hub can tell member sites from arbitrary added origins; the header does not link to it (release 14).
Default: absent
diff --git a/WORKTREES.md b/WORKTREES.md
@@ -116,7 +116,7 @@ one is not, it **aborts** rather than starting:
```
queue-lock: ABORTING — this suite's ports are already in use.
- PORT=3011 pid 2244791 cwd /home/user/Projects/yt-dlp-transcript-browser/editor
+ PORT=3011 pid 2244791 cwd /path/to/checkout/editor
next-server (v16.2.3)
kill 2244791
```
diff --git a/common/bin/_cli.test.ts b/common/bin/_cli.test.ts
@@ -314,6 +314,40 @@ test("release show says the latest release, its date and what is pending, and wh
assert.equal(formatAllLine([editor]), null);
});
+// --- archilyzer source (release 12 slice R) --------------------------------
+
+test("usage lists source publish, source audit and build homepage's --no-source", () => {
+ const u = usage(COMMANDS);
+ assert.match(u, /archilyzer source publish\s+\[--force\] \[--check\] \[--keep-scratch\] the scrubbed git mirror/);
+ assert.match(u, /archilyzer source audit\s+\[<git dir>\] the denied-literal gate/);
+ assert.match(u, /archilyzer build homepage\s+\[--no-source\] compose \+ source publish \+ next build/);
+});
+
+test("source publish takes its three booleans and nothing else; none swallows a word", () => {
+ const hit = resolveCommand(COMMANDS, ["source", "publish"]);
+ assert.deepEqual(hit?.command.path, ["source", "publish"]);
+ const parsed = parseArgv(["source", "publish", "--check", "--force", "--keep-scratch"], booleanFlags(COMMANDS));
+ assert.deepEqual(parsed, {
+ positionals: ["source", "publish"],
+ flags: { check: true, force: true, "keep-scratch": true },
+ });
+ assert.equal(argumentProblem(hit!.command, parsed.flags, []), null);
+ assert.match(argumentProblem(hit!.command, { preview: "x" }, [])!, /unknown flag --preview \(accepts --force, --check, --keep-scratch\)/);
+ assert.match(argumentProblem(hit!.command, {}, ["extra"])!, /unexpected argument "extra"/);
+});
+
+test("source audit takes one git dir; build homepage takes --no-source only", () => {
+ const audit = resolveCommand(COMMANDS, ["source", "audit", "/tmp/clone/.git"]);
+ assert.deepEqual(audit?.rest, ["/tmp/clone/.git"]);
+ assert.equal(argumentProblem(audit!.command, {}, audit!.rest), null);
+ assert.match(argumentProblem(audit!.command, {}, ["a", "b"])!, /unexpected argument "b"/);
+ const home = resolveCommand(COMMANDS, ["build", "homepage"]);
+ const parsed = parseArgv(["build", "homepage", "--no-source"], booleanFlags(COMMANDS));
+ assert.deepEqual(parsed, { positionals: ["build", "homepage"], flags: { "no-source": true } });
+ assert.equal(argumentProblem(home!.command, parsed.flags, []), null);
+ assert.match(argumentProblem(home!.command, { force: true }, [])!, /unknown flag --force \(accepts --no-source\)/);
+});
+
// ── passthrough commands (one-core Phase 4 slice 3) ─────────────────────────
test("a passthrough command gets every word after its path, verbatim and unchecked", async () => {
diff --git a/common/bin/archilyzer.ts b/common/bin/archilyzer.ts
@@ -137,15 +137,45 @@ export const COMMANDS: Command[] = [
},
{
path: ["build", "homepage"],
- usage: "compose + next build in homepage/ (reads the index as it stands)",
- run: async () => {
+ usage:
+ "[--no-source] compose + source publish + next build in homepage/ (reads the index as it stands); --no-source removes the published source instead",
+ flags: { "no-source": "boolean" },
+ run: async ({ flags }) => {
const { buildHomepage } = await import("../publish/build");
- const code = await buildHomepage({ signal: interrupted() });
+ const code = await buildHomepage({
+ signal: interrupted(),
+ skipSource: flags["no-source"] === true,
+ });
if (code !== 0) console.error(`build homepage: failed (exit ${code})`);
return code;
},
},
{
+ path: ["source", "publish"],
+ usage:
+ "[--force] [--check] [--keep-scratch] the scrubbed git mirror, raw tree, history pages (stagit, when installed) and tarball into homepage/public, behind the denied-literal gate (--check: audit and count, write nothing)",
+ flags: { force: "boolean", check: "boolean", "keep-scratch": "boolean" },
+ run: async ({ flags }) => {
+ const { publishSource } = await import("../publish/source");
+ return publishSource({
+ signal: interrupted(),
+ force: flags.force === true,
+ check: flags.check === true,
+ keepScratch: flags["keep-scratch"] === true,
+ });
+ },
+ },
+ {
+ path: ["source", "audit"],
+ usage:
+ "[<git dir>] the denied-literal gate (+ gitleaks) over a git dir; default the published homepage/public/source/archilyzer.git",
+ maxPositionals: 1,
+ run: async ({ positionals }) => {
+ const { auditSource } = await import("../publish/source");
+ return auditSource({ signal: interrupted(), gitDir: positionals[0] });
+ },
+ },
+ {
path: ["deploy", "site"],
usage:
"<id> [--preview <branch>] ship the site built in export/out to its Pages project (default id: SITE_ID)",
diff --git a/common/bin/build-index.ts b/common/bin/build-index.ts
@@ -3,10 +3,16 @@
// as `tsx bin/build-index.ts` (export's build:index script).
import { getPaths, type Paths } from "../lib/paths";
import { buildIndex } from "../controller/buildIndex";
+import { settingsFromFile } from "../lib/settings";
+import { applyHealthTimings } from "../lib/storageHealth";
import { runIfEntryPoint } from "./_cli";
export async function main(opts: { paths?: Paths } = {}): Promise<void> {
- await buildIndex({ paths: opts.paths ?? getPaths() });
+ const paths = opts.paths ?? getPaths();
+ // A CLI process has no health pass: the drive-health timings the build's
+ // watchdog runs on (settings.storage.health) are applied here, once.
+ applyHealthTimings(settingsFromFile(paths.settingsFile).storage.health);
+ await buildIndex({ paths });
}
runIfEntryPoint(import.meta.url, () => main());
diff --git a/common/bin/build-stats.ts b/common/bin/build-stats.ts
@@ -3,10 +3,16 @@
// build:stats script); `main` is exported for the archilyzer CLI.
import { getPaths, type Paths } from "../lib/paths";
import { buildStats } from "../controller/buildStats";
+import { settingsFromFile } from "../lib/settings";
+import { applyHealthTimings } from "../lib/storageHealth";
import { runIfEntryPoint } from "./_cli";
export async function main(opts: { paths?: Paths } = {}): Promise<void> {
- await buildStats({ paths: opts.paths ?? getPaths() });
+ const paths = opts.paths ?? getPaths();
+ // A CLI process has no health pass: the drive-health timings the build's
+ // watchdog runs on (settings.storage.health) are applied here, once.
+ applyHealthTimings(settingsFromFile(paths.settingsFile).storage.health);
+ await buildStats({ paths });
}
runIfEntryPoint(import.meta.url, () => main());
diff --git a/common/bin/compose-homepage.ts b/common/bin/compose-homepage.ts
@@ -8,6 +8,9 @@
// public/channel-sites.json <- channel slug -> [siteId, ...]
// public/homepage-summary.json <- small cross-site landing summary
//
+// None of the three names an unlisted site (site.json `listed: false`) or holds
+// a channel only unlisted sites expose (lib/siteSchema.ts isListedSite).
+//
// Requires build:index to have populated the cues LMDB first (the homepage
// prebuild chains it), same as the export pipeline.
@@ -15,6 +18,7 @@ import path from "node:path";
import { getPaths, type Paths } from "../lib/paths";
import { writeJsonAtomic as writeJsonAtomicShared } from "../lib/jsonFile-server";
import { buildPoolSummary } from "../controller/poolSummary";
+import { isListedSite } from "../lib/site";
import { runIfEntryPoint } from "./_cli";
// Where the homepage Next.js app serves static assets from. Overridable for e2e
@@ -45,7 +49,7 @@ export async function main(opts: { paths?: Paths } = {}): Promise<void> {
statsDir,
});
- // channel slug -> the ids of the content sites that expose it. Drives
+ // channel slug -> the ids of the listed content sites that expose it. Drives
// `groupBy: "site"` in the hub's dashboard (see channelSites.tsx).
await writeJsonAtomic(
path.join(publicDir, "channel-sites.json"),
@@ -60,8 +64,11 @@ export async function main(opts: { paths?: Paths } = {}): Promise<void> {
summary,
);
+ const unlisted = sites.filter((s) => !isListedSite(s)).length;
console.log(
- `compose-homepage: ${Object.keys(channelSites).length} channel(s) mapped across ${sites.length} site(s); ` +
+ `compose-homepage: ${Object.keys(channelSites).length} channel(s) mapped across ${sites.length - unlisted} listed site(s)` +
+ (unlisted > 0 ? ` (${unlisted} unlisted left out)` : "") +
+ "; " +
`summary covers ${summary.totals.transcripts} transcription(s) / ${summary.totals.downloads} download(s) ` +
`across ${summary.sites.length} public site(s) into ${publicDir}.`,
);
diff --git a/common/bin/compose-hub.test.ts b/common/bin/compose-hub.test.ts
@@ -130,3 +130,43 @@ test("in a worktree, compose-hub writes its own files and never through the link
rmSync(root, { recursive: true, force: true });
}
});
+
+// Release 14 slice HS: an unlisted site builds and deploys, and the hub does
+// not list it — not a member, so not in federated search, corpus.json or
+// llms.txt.
+test("an unlisted site is in none of the hub's files; a listed one is in each", async () => {
+ const root = mkdtempSync(path.join(tmpdir(), "compose-hub-"));
+ const log = console.log;
+ try {
+ const paths = fixturePaths(root);
+ const write = (id: string, site: Record<string, unknown>) => {
+ mkdirSync(path.join(paths.sitesDir, id), { recursive: true });
+ writeFileSync(path.join(paths.sitesDir, id, "site.json"), JSON.stringify(site));
+ };
+ write("fixture-shown", { siteTitle: "Shown Fixture", siteUrl: "https://fixture-shown.example" });
+ write("fixture-unlisted", {
+ siteTitle: "Unlisted Fixture",
+ siteUrl: "https://fixture-unlisted.example",
+ listed: false,
+ });
+ const lines: string[] = [];
+ console.log = (...a: unknown[]) => void lines.push(a.join(" "));
+ await main({ paths });
+ console.log = log;
+ const pool = JSON.parse(
+ readFileSync(path.join(paths.exportPublicDir, "hub-sites.json"), "utf8"),
+ ) as Array<{ siteId: string }>;
+ assert.deepEqual(pool.map((s) => s.siteId), ["fixture-shown"]);
+ for (const f of ["hub-sites.json", "corpus.json", "llms.txt"]) {
+ const text = readFileSync(path.join(paths.exportPublicDir, f), "utf8");
+ assert.ok(text.includes("fixture-shown.example"), `${f} lists the listed site`);
+ for (const needle of ["fixture-unlisted", "Unlisted Fixture"]) {
+ assert.ok(!text.includes(needle), `${f} names ${needle}`);
+ }
+ }
+ assert.match(lines.join("\n"), /compose-hub: 1 built-in pool site\(s\)/);
+ } finally {
+ console.log = log;
+ rmSync(root, { recursive: true, force: true });
+ }
+});
diff --git a/common/bin/compose-hub.ts b/common/bin/compose-hub.ts
@@ -4,7 +4,8 @@
// there is no SITE_ID and no per-site data: the hub is a federating shell that
// reads every archive cross-origin at runtime. It emits:
//
-// public/hub-sites.json <- the built-in trusted pool (listSites with a siteUrl)
+// public/hub-sites.json <- the built-in trusted pool (listSites with a siteUrl,
+// listed — site.json `listed`, isListedSite)
// public/hub-summary.json <- the official instances' numbers, the homepage's
// own (lib/hubSummary.ts) — OPTIONAL: skipped when
// there is no index to walk
@@ -19,7 +20,7 @@ import { existsSync } from "node:fs";
import { cp, rm, access } from "node:fs/promises";
import { getPaths, type Paths } from "../lib/paths";
import { accentHex } from "../lib/accent";
-import { listSites, resolveHubUrl } from "../lib/site";
+import { isListedSite, listSites, resolveHubUrl } from "../lib/site";
import { getHomepageConfig } from "../lib/homepage";
import { SITE_DESCRIPTOR_VERSION } from "../lib/siteDescriptor";
import {
@@ -79,7 +80,10 @@ export async function main(opts: { paths?: Paths } = {}): Promise<void> {
const paths = opts.paths ?? getPaths();
const publicDir = paths.exportPublicDir;
- // Built-in pool: every configured site that publishes a public URL. The entry
+ // Built-in pool: every configured site that publishes a public URL and is
+ // listed. An unlisted site (`listed: false`) still builds and deploys, but the
+ // hub does not list it: not a member, not in federated search, not in the
+ // hub's corpus.json or llms.txt (both are built from this list). The entry
// the hub registry loads at boot to seed its trusted built-in pool, and the
// input buildHubCorpus maps — ONE type for both (lib/archive/contract.ts
// HubMemberInput), where this file used to restate it as a local
@@ -88,7 +92,7 @@ export async function main(opts: { paths?: Paths } = {}): Promise<void> {
// a siteUrl can't be federated and is dropped — by both consumers.
const builtins: HubMemberInput[] = [];
for (const site of listSites(paths)) {
- if (!site.siteUrl) continue;
+ if (!site.siteUrl || !isListedSite(site)) continue;
builtins.push({
siteId: site.siteId,
siteTitle: site.siteTitle,
diff --git a/common/bin/compose-site.ts b/common/bin/compose-site.ts
@@ -179,7 +179,7 @@ async function emitAiFiles(paths: ReturnType<typeof getPaths>): Promise<void> {
// siteUrl; otherwise clear any stale copy from a previous build.
const sitemapPath = path.join(paths.exportPublicDir, "sitemap.xml");
if (descriptor.siteUrl) {
- const routes = ["/", "/use-with-ai", "/changelog"];
+ const routes = ["/", "/changelog"];
if (hasArchives) routes.push("/downloads");
if (await exists(path.join(paths.exportPublicDir, DUPLICATES_FILENAME))) {
routes.push("/duplicates");
diff --git a/common/bin/doctor.test.ts b/common/bin/doctor.test.ts
@@ -58,6 +58,11 @@ function checkout(): { root: string; bin: string; paths: Paths } {
parakeetBin: path.join(root, "scripts", "parakeet-stitch.mjs"),
parakeetModel: "",
parakeetCliBin: "parakeet-cli",
+ sourceScrubFile: path.join(root, ".config", "source-scrub.txt"),
+ sourceDenylistFile: path.join(root, ".config", "source-denylist.txt"),
+ // Outside the checkout, as the XDG cache is: the tests that compare the
+ // checkout's tree before and after never see it.
+ sourceHistoryCacheDir: path.join(TMP, `${path.basename(root)}-cache`, "archilyzer", "source-history"),
} as unknown as Paths;
return { root, bin, paths };
}
@@ -259,3 +264,132 @@ test("umtool's table and the port block are reported, never failed", async () =>
assert.match(r.checks.find((x) => x.id === "EDITOR_PORT")!.detail, /^3301 in use/);
assert.equal(r.ok, true);
});
+
+test("the source publish block: the tools, the operator files by count and mode (never their contents), the last publish — never a failure", async () => {
+ const c = checkout();
+ const deps = (tools: Awaited<ReturnType<NonNullable<Parameters<typeof collectDoctorReport>[0]["sourceTools"]>>>) => ({
+ env: { PATH: c.bin },
+ paths: c.paths,
+ nodeVersion: "22.0.0",
+ portBlock: async () => null,
+ portInUse: async () => false,
+ umtoolTools: async () => null,
+ sourceTools: async () => tools,
+ });
+ // A clone that never publishes: notes, not warnings.
+ let r = await collectDoctorReport(deps({ filterRepo: null, gitleaks: null, stagit: null }));
+ assert.equal(r.ok, true, renderDoctorReport(r));
+ for (const id of ["filter-repo", "gitleaks", "stagit", "scrub rules", "denylist", "published"]) {
+ assert.equal(status(r, id), "info", id);
+ }
+ assert.match(r.checks.find((x) => x.id === "filter-repo")!.detail, /install: `pipx install git-filter-repo`/);
+ // stagit, beside git-filter-repo: where it is, or not found and how to
+ // install it; then the render cache, where it is and how big.
+ assert.equal(
+ r.checks.find((x) => x.id === "stagit")!.detail,
+ `not found (PATH, ~/.local/bin) — the source is published without its history pages (/source/git/); install it once: git clone git://git.codemadness.org/stagit && make -C stagit && cp stagit/stagit ~/.local/bin/; cache: ${c.paths.sourceHistoryCacheDir}, none yet`,
+ );
+ const order = r.checks.filter((x) => x.section === "source publish").map((x) => x.id);
+ assert.equal(order.indexOf("stagit"), order.indexOf("filter-repo") + 1, "beside git-filter-repo");
+
+ // The operator's files exist (one readable by others), pipx only, a publish.
+ mkdirSync(path.dirname(c.paths.sourceScrubFile), { recursive: true });
+ writeFileSync(c.paths.sourceScrubFile, "# mine\nPLANTED-A==>x\n\nPLANTED-B==>y\n");
+ chmodSync(c.paths.sourceScrubFile, 0o600);
+ writeFileSync(c.paths.sourceDenylistFile, "PLANTED-SECRET\n");
+ chmodSync(c.paths.sourceDenylistFile, 0o644);
+ const pub = path.join(c.root, "homepage", "public", "source");
+ mkdirSync(pub, { recursive: true });
+ writeFileSync(path.join(pub, "manifest.json"), JSON.stringify({
+ version: 1, generatedAt: "2026-09-28T12:00:00.000Z", branch: "main",
+ sourceCommit: "1".repeat(40), mirrorHead: "2".repeat(40), subject: "s", files: 2400, bytes: 1,
+ mirror: { files: 9, bytes: 1, packs: 2 }, tree: { files: 1, dirs: 1, bytes: 1 },
+ tarball: { href: "/downloads/archilyzer-source.tar.gz", bytes: 1, sha256: "3".repeat(64) },
+ audit: { objects: 1, commits: 1, gitleaks: "clean" }, tools: {},
+ }));
+ const before = tree(c.root);
+ r = await collectDoctorReport(deps({ filterRepo: { via: "pipx", version: "1.15.0" }, gitleaks: { version: "8.28.0" }, stagit: "/opt/stagit/stagit" }));
+ assert.equal(r.ok, true, renderDoctorReport(r));
+ assert.equal(status(r, "filter-repo"), "warn");
+ assert.match(r.checks.find((x) => x.id === "filter-repo")!.detail, /pipx run --spec git-filter-repo==2\.47\.0/);
+ assert.equal(status(r, "gitleaks"), "ok");
+ assert.equal(status(r, "stagit"), "ok");
+ mkdirSync(path.join(c.paths.sourceHistoryCacheDir, "out"), { recursive: true });
+ writeFileSync(path.join(c.paths.sourceHistoryCacheDir, "out", "log.html"), Buffer.alloc(3 * 1024 * 1024));
+ const withCache = await collectDoctorReport(deps({ filterRepo: null, gitleaks: null, stagit: "/opt/stagit/stagit" }));
+ assert.equal(withCache.checks.find((x) => x.id === "stagit")!.detail, `/opt/stagit/stagit; cache: ${c.paths.sourceHistoryCacheDir}, 3.0 MB`);
+ assert.equal(status(r, "scrub rules"), "ok");
+ assert.match(r.checks.find((x) => x.id === "scrub rules")!.detail, /\(2 rules, mode 600\)$/);
+ assert.equal(status(r, "denylist"), "warn");
+ assert.match(r.checks.find((x) => x.id === "denylist")!.detail, /\(1 literal, mode 644\) — readable by others: chmod 600/);
+ assert.match(r.checks.find((x) => x.id === "published")!.detail, /^main 111111111111 as 222222222222, 2026-09-28T12:00:00\.000Z \(2400 files\)/);
+ const text = renderDoctorReport(r);
+ assert.match(text, /\nsource publish\n/);
+ assert.ok(!/PLANTED/.test(text), "the doctor never prints an operator file's contents");
+ assert.deepEqual(tree(c.root), before);
+
+ r = await collectDoctorReport(deps({ filterRepo: { via: "git", version: "a40bce548d2c" }, gitleaks: null, stagit: null }));
+ assert.equal(status(r, "filter-repo"), "ok");
+
+ // A STAGIT_BIN that names nothing: a warning, never a failure.
+ r = await collectDoctorReport({ ...deps({ filterRepo: null, gitleaks: null, stagit: null }), env: { PATH: c.bin, STAGIT_BIN: "/nowhere/stagit" } });
+ assert.equal(status(r, "stagit"), "warn");
+ assert.match(r.checks.find((x) => x.id === "stagit")!.detail, /^not found: STAGIT_BIN=\/nowhere\/stagit names no executable/);
+ assert.equal(r.ok, true);
+});
+
+// A stored icon the checker refuses is named by file and label with its
+// reason — never its markup — and the doctor still exits 0 (a WARN).
+test("social icons: a refused stored icon is named by file and label, not its markup", async () => {
+ const c = checkout();
+ const sitesDir = path.join(c.paths.transcriptsDir, "sites");
+ mkdirSync(path.join(sitesDir, "one"), { recursive: true });
+ const secret = "SECRET-MARKUP-XYZ";
+ writeFileSync(c.paths.settingsFile, JSON.stringify({
+ socialLinks: [
+ { label: "Good", url: "https://good.example", svg: `<svg viewBox="0 0 8 8"><path d="M0 0"/></svg>` },
+ { label: "Old", url: "https://old.example", svg: `<svg viewBox="0 0 8 8"><style>.${secret}{}</style></svg>` },
+ ],
+ }));
+ writeFileSync(path.join(sitesDir, "one", "site.json"), JSON.stringify({
+ socialLinks: [{ label: "Site icon", url: "https://s.example", svg: `<svg/onload="x()" viewBox="0 0 8 8"></svg>` }],
+ }));
+ const report = await collectDoctorReport({
+ env: {},
+ paths: { ...c.paths, sitesDir, homepageConfigFile: path.join(sitesDir, "_homepage", "homepage.json") },
+ probe: async (t) => ({ id: t.id, bin: t.bin ?? "", present: true, version: "1", neededBy: t.neededBy, required: t.required }) as never,
+ portInUse: async () => false,
+ portBlock: async () => null,
+ umtoolTools: async () => null,
+ sourceTools: async () => ({ filterRepo: null, gitleaks: null, stagit: null }),
+ });
+ const line = report.checks.find((x) => x.section === "social icons")!;
+ assert.equal(line.status, "warn");
+ assert.match(line.detail, /2 of 3 fail/);
+ assert.match(line.detail, /settings\.json: "Old" — it has an element an icon has no use for \(style\)/);
+ assert.match(line.detail, /site\.json: "Site icon" — it is not one well-formed <svg> element/);
+ assert.ok(!line.detail.includes(secret), "no markup");
+ assert.ok(report.ok, "a warning, not a failure");
+});
+
+// Release 15 slice DT left doctor's line to SG: the drive-health timings are
+// applied before any drive is inspected (as the index and stats bins apply
+// them), and the corpus block says which are in force.
+test("the drive-health timings: applied from settings.storage.health before the corpus is inspected, and printed", async () => {
+ const { healthTimings, applyHealthTimings } = await import("../lib/storageHealth");
+ const c = checkout();
+ mkdirSync(c.paths.channelsDir, { recursive: true });
+ let r = await run(c);
+ assert.equal(
+ r.checks.find((x) => x.id === "drive health")!.detail,
+ "a read may take 3 s, a check every 15 s (3 s each), a stall clears on a clean check twice in a row, 4 reads in flight per drive — the defaults",
+ );
+ writeFileSync(c.paths.settingsFile, JSON.stringify({ storage: { health: { budgetMs: 5000, clearAfterCleanPasses: 3 } } }));
+ r = await run(c);
+ assert.equal(
+ r.checks.find((x) => x.id === "drive health")!.detail,
+ "a read may take 5 s, a check every 15 s (3 s each), a stall clears on a clean check 3 times in a row, 4 reads in flight per drive — settings.storage.health",
+ );
+ assert.equal(healthTimings().budgetMs, 5000, "applied to this process");
+ applyHealthTimings();
+});
diff --git a/common/bin/doctor.ts b/common/bin/doctor.ts
@@ -70,6 +70,16 @@ export type DoctorDeps = {
portBlock?: () => Promise<{ label: string; ports: Record<string, string> } | null>;
// umtool's report-pipeline table, or null when there is no umtool here.
umtoolTools?: () => Promise<ToolSpec[] | null>;
+ // The source publish's tools. Default: probeSourceTools(env).
+ sourceTools?: () => Promise<SourceTools>;
+};
+
+// Which git-filter-repo `archilyzer source publish` would run, gitleaks, and
+// the stagit that renders the history pages (its path, or null).
+export type SourceTools = {
+ filterRepo: { via: "git"; version: string } | { via: "pipx"; version: string } | null;
+ gitleaks: { version: string } | null;
+ stagit: string | null;
};
const MIN_NODE = [20, 9, 0] as const; // next 16's engines field
@@ -101,6 +111,22 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
? "no path or binary overrides set (ENVIRONMENT.md lists them)"
: overrides.map((v) => `${v.name}=${env[v.name]}`).join(" "));
+ // The effective settings, read the way every process reads them (defaults
+ // when the file is absent). Read-only: the reader never writes. Read before
+ // the corpus, for the drive-health timings (release 15 slice DT): applied
+ // here, as the index and stats bins apply them, a CLI process's drive
+ // inspects run on the machine's timings instead of racing the defaults.
+ const { settingsFromFile } = await import("../lib/settings");
+ let settings: ReturnType<typeof settingsFromFile> | null = null;
+ let settingsError: Error | null = null;
+ try {
+ settings = settingsFromFile(paths.settingsFile);
+ } catch (err) {
+ settingsError = err as Error;
+ }
+ const { applyHealthTimings } = await import("../lib/storageHealth");
+ const timings = applyHealthTimings(settings?.storage.health);
+
// ── corpus ───────────────────────────────────────────────────────────────
const C = "corpus";
let channelSlugs: string[] = [];
@@ -113,7 +139,9 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
const unreachable: string[] = [];
const { inspectChannelMedia } = await import("../lib/channelMedia");
for (const slug of channelSlugs) {
- const loc = await inspectChannelMedia(paths, slug);
+ const loc = await inspectChannelMedia(paths, slug, undefined, {
+ fresh: true,
+ });
if (loc.status !== "ok" && loc.status !== "in-place") {
unreachable.push(`${slug}: ${loc.status} — ${loc.detail ?? ""}`.trim());
}
@@ -123,6 +151,12 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
} else if (channelSlugs.length > 0) {
add(C, "media", "ok", "every channel's data/ is reachable");
}
+ const { secondsText, clearRuleText } = await import("../lib/storageHealthTimings");
+ const tuned = Object.keys(settings?.storage.health ?? {}).length > 0;
+ add(C, "drive health", "info",
+ `a read may take ${secondsText(timings.budgetMs)}, a check every ${secondsText(timings.passIntervalMs)} ` +
+ `(${secondsText(timings.probeTimeoutMs)} each), a stall clears on a clean check ${clearRuleText(timings.clearAfterCleanPasses)}, ` +
+ `${timings.inFlightPerLocation} reads in flight per drive — ${tuned ? "settings.storage.health" : "the defaults"}`);
const index = statOrNull(paths.lmdbPath);
if (index) {
add(C, "index", "ok", `${paths.lmdbPath} (stat only; built ${index.mtime.toISOString().slice(0, 16).replace("T", " ")})`);
@@ -151,14 +185,25 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
`${paths.settingsFile} is not a JSON object${parsed instanceof Error ? ` (${parsed.message})` : ""} — every process silently reads it as the defaults`);
}
}
- // The effective settings, read the way every process reads them (defaults
- // when the file is absent). Read-only: the reader never writes.
- const { settingsFromFile } = await import("../lib/settings");
- let settings: ReturnType<typeof settingsFromFile> | null = null;
- try {
- settings = settingsFromFile(paths.settingsFile);
- } catch (err) {
- add(S, "schema", "fail", `settings do not load: ${(err as Error).message}`);
+ if (settingsError) add(S, "schema", "fail", `settings do not load: ${settingsError.message}`);
+
+ // ── social icons ─────────────────────────────────────────────────────────
+ // Every stored social link's icon — settings.json, each site.json,
+ // homepage.json — through the check a page runs before it inlines one
+ // (lib/socialSvg.ts). One that fails renders as its label on every page;
+ // editing its SVG in the editor is the fix. Named by file and label with the
+ // reason class — never the markup.
+ const SI = "social icons";
+ const icons = await storedSocialIcons(paths);
+ if (icons.checked === 0) {
+ add(SI, "stored icons", "info", "no stored social icons");
+ } else if (icons.refused.length === 0) {
+ add(SI, "stored icons", "ok",
+ `${icons.checked} icon${icons.checked === 1 ? "" : "s"} in ${icons.files} file${icons.files === 1 ? "" : "s"} pass the check`);
+ } else {
+ add(SI, "stored icons", "warn",
+ `${icons.refused.length} of ${icons.checked} fail the check and show as their label; edit each one's SVG:\n` +
+ icons.refused.map((r) => `${path.relative(root, r.file) || r.file}: "${r.label}" — ${r.problem}`).join("\n"));
}
// ── tools ────────────────────────────────────────────────────────────────
@@ -259,6 +304,65 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
}
}
+ // ── source publish ───────────────────────────────────────────────────────
+ // `archilyzer build homepage` runs it (common/publish/source.ts). Never a
+ // failure: a checkout that never publishes the homepage is not broken. A
+ // WARN is a machine that means to (its operator files exist) and cannot.
+ // The operator files are stat'd and their RULES COUNTED — their contents
+ // are never printed.
+ const SP = "source publish";
+ const src = await import("../publish/source");
+ const tools = await (deps.sourceTools ?? (() => probeSourceTools(env, paths)))();
+ const scrubFile = paths.sourceScrubFile;
+ const denylistFile = paths.sourceDenylistFile;
+ const intends = [scrubFile, denylistFile].some((f) => f && existsSync(f));
+ if (tools.filterRepo?.via === "git") {
+ add(SP, "filter-repo", "ok", `git filter-repo ${tools.filterRepo.version}`);
+ } else if (tools.filterRepo?.via === "pipx") {
+ add(SP, "filter-repo", intends ? "warn" : "info",
+ `not installed; \`pipx run --spec ${src.FILTER_REPO_PIPX_SPEC}\` fetches it at build time (network on first use; pipx ${tools.filterRepo.version}) — \`${src.FILTER_REPO_INSTALL}\` once removes that`);
+ } else {
+ add(SP, "filter-repo", intends ? "warn" : "info",
+ `neither git-filter-repo nor pipx — \`archilyzer build homepage\` refuses; install: \`${src.FILTER_REPO_INSTALL}\``);
+ }
+ // stagit renders the history pages (/source/git/); without it the publish
+ // goes on without them. A STAGIT_BIN that names nothing is a warning. The
+ // line ends with the render cache: where it is and how big (stat'd only).
+ const hist = await import("../publish/sourceHistory");
+ const cacheDir = paths.sourceHistoryCacheDir;
+ const cache = cacheDir ? `; cache: ${cacheDir}, ${await dirSizeText(cacheDir)}` : "";
+ if (tools.stagit) {
+ add(SP, "stagit", "ok", `${tools.stagit}${cache}`);
+ } else if (env.STAGIT_BIN) {
+ add(SP, "stagit", "warn",
+ `not found: STAGIT_BIN=${env.STAGIT_BIN} names no executable — the source is published without its history pages (/source/git/)${cache}`);
+ } else {
+ add(SP, "stagit", "info",
+ `not found (PATH, ~/.local/bin) — the source is published without its history pages (/source/git/); install it once: ${hist.STAGIT_INSTALL}${cache}`);
+ }
+ add(SP, "gitleaks", tools.gitleaks ? "ok" : "info",
+ tools.gitleaks ? `gitleaks ${tools.gitleaks.version}` : "absent — the gate skips the secret scan with a WARNING (the literal audit still runs)");
+ for (const [id, file, unit] of [
+ ["scrub rules", scrubFile, "rule"],
+ ["denylist", denylistFile, "literal"],
+ ] as const) {
+ const st = file ? statOrNull(file) : null;
+ if (!file || !st) {
+ add(SP, id, "info", `${file ?? "(unset)"} is missing — \`source publish\` (and so \`build homepage\`) refuses until it exists (PUBLISH.md, "The source mirror")`);
+ continue;
+ }
+ const count = (readOrNull(file) ?? "").split("\n").filter((l) => l.trim() && !l.trim().startsWith("#")).length;
+ const mode = st.mode & 0o777;
+ const open = (mode & 0o077) !== 0;
+ add(SP, id, open ? "warn" : "ok",
+ `${file} (${count} ${unit}${count === 1 ? "" : "s"}, mode ${mode.toString(8)})${open ? " — readable by others: chmod 600" : ""}`);
+ }
+ const publicDir = env.HOMEPAGE_PUBLIC_DIR || path.join(root, "homepage", "public");
+ const published = await src.readPublishedManifest(publicDir);
+ add(SP, "published", "info", published
+ ? `main ${published.sourceCommit.slice(0, 12)} as ${published.mirrorHead.slice(0, 12)}, ${published.generatedAt} (${published.files} files)`
+ : `nothing published in ${path.join(publicDir, "source")}`);
+
// ── ports ────────────────────────────────────────────────────────────────
const P = "ports";
const block = await (deps.portBlock ?? (() => worktreePortBlock(root)))();
@@ -307,6 +411,53 @@ export async function main(opts: { json?: boolean; env?: NodeJS.ProcessEnv } = {
// ── helpers ────────────────────────────────────────────────────────────────
+// The social links stored in settings.json, every sites/<id>/site.json and
+// sites/_homepage/homepage.json, each checked. Read-only; a missing or
+// unreadable file is skipped.
+async function storedSocialIcons(paths: Paths): Promise<{
+ files: number;
+ checked: number;
+ refused: { file: string; label: string; problem: string }[];
+}> {
+ const { parseSocialLinks } = await import("../lib/settingsSchema");
+ const { socialSvgProblem } = await import("../lib/socialSvg");
+ const candidates = [paths.settingsFile];
+ if (paths.sitesDir) {
+ try {
+ for (const e of await readdir(paths.sitesDir, { withFileTypes: true })) {
+ if (e.isDirectory() && !e.name.startsWith("_")) {
+ candidates.push(path.join(paths.sitesDir, e.name, "site.json"));
+ }
+ }
+ } catch {
+ /* no sites directory */
+ }
+ }
+ if (paths.homepageConfigFile) candidates.push(paths.homepageConfigFile);
+ let files = 0;
+ let checked = 0;
+ const refused: { file: string; label: string; problem: string }[] = [];
+ for (const file of candidates) {
+ const text = file ? readOrNull(file) : null;
+ if (text === null) continue;
+ let raw: unknown;
+ try {
+ raw = JSON.parse(text);
+ } catch {
+ continue;
+ }
+ const links = parseSocialLinks((raw as { socialLinks?: unknown } | null)?.socialLinks);
+ if (links.length === 0) continue;
+ files += 1;
+ for (const link of links) {
+ checked += 1;
+ const problem = socialSvgProblem(link.svg);
+ if (problem) refused.push({ file, label: link.label, problem });
+ }
+ }
+ return { files, checked, refused };
+}
+
function versionAtLeast(v: string, min: readonly [number, number, number]): boolean {
const parts = v.split(".").map((n) => Number.parseInt(n, 10) || 0);
for (let i = 0; i < 3; i++) {
@@ -401,6 +552,45 @@ async function worktreePortBlock(
}
}
+// What `source publish` would run, by version flags only: `git filter-repo
+// --version` answering 0 is an installed filter-repo; otherwise pipx's own
+// version (never `pipx run`, which would download). gitleaks likewise. stagit
+// has no version flag: it is looked up the way the publish looks it up.
+async function probeSourceTools(env: NodeJS.ProcessEnv, paths: Paths): Promise<SourceTools> {
+ const version = async (bin: string, args: string[]): Promise<string | null> => {
+ try {
+ const { stdout, stderr } = await execFileP(bin, args, { env, timeout: 10_000 });
+ return (stdout || stderr).trim().split("\n")[0] ?? "";
+ } catch {
+ return null;
+ }
+ };
+ const git = await version("git", ["filter-repo", "--version"]);
+ const pipx = git === null ? await version("pipx", ["--version"]) : null;
+ const leaks = await version("gitleaks", ["version"]);
+ const { resolveStagit } = await import("../publish/sourceHistory");
+ return {
+ filterRepo: git !== null ? { via: "git", version: git } : pipx !== null ? { via: "pipx", version: pipx } : null,
+ gitleaks: leaks !== null ? { version: leaks } : null,
+ stagit: resolveStagit(paths.stagitBin ?? "stagit", env),
+ };
+}
+
+// A directory's size, by stat alone ("none yet" when it is not there).
+async function dirSizeText(dir: string): Promise<string> {
+ if (!existsSync(dir)) return "none yet";
+ let bytes = 0;
+ const walk = async (d: string): Promise<void> => {
+ for (const ent of await readdir(d, { withFileTypes: true }).catch(() => [])) {
+ const p = path.join(d, ent.name);
+ if (ent.isDirectory()) await walk(p);
+ else if (ent.isFile()) bytes += statOrNull(p)?.size ?? 0;
+ }
+ };
+ await walk(dir);
+ return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
+}
+
// In use = something accepts a TCP connection on 127.0.0.1. Never binds.
function tcpPortInUse(port: number): Promise<boolean> {
return new Promise((resolve) => {
diff --git a/common/bin/gen-wordmark-metrics.py b/common/bin/gen-wordmark-metrics.py
@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""Write common/lib/wordmarkMetrics.ts: the advance of each Latin character
+Archivo maps, at the wordmark's two instances (wdth 118; wght 720 for the lead,
+380 for the suffix), measured from the vendored variable font
+(umtool/report-to-video/fonts/Archivo[wdth,wght].ttf, the same bytes the video
+lockup is outlined from).
+
+The export header reserves the wordmark's width from this table
+(common/lib/wordmarkWidth.ts), so whether a title fits beside the header's
+icons is decided by the display face's widths, before and after the web font
+loads: the fallback face is narrower, and a title that fitted in it would
+otherwise show and then vanish when Archivo arrived.
+
+ python3 common/bin/gen-wordmark-metrics.py # rewrite the module
+ python3 common/bin/gen-wordmark-metrics.py --stdout # print it
+
+Needs fontTools (pip install fonttools). The module records the fontTools
+version it was generated with.
+"""
+import hashlib
+import json
+import os
+import sys
+
+import fontTools
+from fontTools.ttLib import TTFont
+from fontTools.varLib import instancer
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+REPO = os.path.normpath(os.path.join(HERE, "..", ".."))
+FONT = "Archivo[wdth,wght].ttf"
+FONT_PATH = os.path.join(REPO, "umtool", "report-to-video", "fonts", FONT)
+OUT = os.path.join(REPO, "common", "lib", "wordmarkMetrics.ts")
+WEIGHTS = (720, 380)
+WDTH = 118
+# Basic Latin, Latin-1 and Latin Extended-A: what a site's title is written in.
+WANT = [cp for a, b in ((32, 126), (160, 383)) for cp in range(a, b + 1)]
+
+
+def runs(cps):
+ out = []
+ for cp in cps:
+ if out and cp == out[-1][1] + 1:
+ out[-1][1] = cp
+ else:
+ out.append([cp, cp])
+ return out
+
+
+def wrap(values, indent=" ", width=100):
+ lines, line = [], ""
+ for v in values:
+ s = f"{v},"
+ if line and len(indent) + len(line) + 1 + len(s) > width:
+ lines.append(indent + line)
+ line = s
+ else:
+ line = f"{line} {s}" if line else s
+ if line:
+ lines.append(indent + line)
+ return "\n".join(lines)
+
+
+def build():
+ raw = open(FONT_PATH, "rb").read()
+ sha = hashlib.sha256(raw).hexdigest()
+ base = TTFont(FONT_PATH)
+ upm = base["head"].unitsPerEm
+ mapped = base.getBestCmap()
+ cps = [cp for cp in WANT if cp in mapped]
+ adv = {}
+ for w in WEIGHTS:
+ inst = instancer.instantiateVariableFont(TTFont(FONT_PATH), {"wght": w, "wdth": WDTH})
+ cmap = inst.getBestCmap()
+ hmtx = inst["hmtx"].metrics
+ adv[w] = [hmtx[cmap[cp]][0] for cp in cps]
+ parts = [
+ "// GENERATED by common/bin/gen-wordmark-metrics.py -- do not edit; re-run it.",
+ "//",
+ f"// The advance, in font units, of each Latin character {FONT} maps, at the",
+ f"// wordmark's instances (wdth {WDTH}; wght {' and '.join(map(str, WEIGHTS))}):",
+ "// what lib/wordmarkWidth.ts reserves a title's width by.",
+ "export const WORDMARK_METRICS = {",
+ f" font: {json.dumps(FONT)},",
+ f" sha256: {json.dumps(sha)},",
+ f" fontTools: {json.dumps(fontTools.version)},",
+ f" unitsPerEm: {upm},",
+ f" wdth: {WDTH},",
+ " // [first, last] codepoint ranges; the advances below follow them in order.",
+ " ranges: [",
+ wrap([f"[{a}, {b}]" for a, b in runs(cps)], indent=" "),
+ " ] as ReadonlyArray<readonly [number, number]>,",
+ " advances: {",
+ ]
+ for w in WEIGHTS:
+ parts.append(f" {w}: [")
+ parts.append(wrap(adv[w]))
+ parts.append(" ],")
+ parts += [" } as Readonly<Record<720 | 380, readonly number[]>>,", "};", ""]
+ return "\n".join(parts)
+
+
+if __name__ == "__main__":
+ text = build()
+ if "--stdout" in sys.argv[1:]:
+ sys.stdout.write(text)
+ else:
+ with open(OUT, "w") as f:
+ f.write(text)
+ print(f"wrote {os.path.relpath(OUT)}")
diff --git a/common/components/BrandMark.test.ts b/common/components/BrandMark.test.ts
@@ -21,7 +21,7 @@ test("the ring's corner is the ground's: MARK_GROUND_RX of MARK_VIEWBOX", () =>
assert.equal(pct, 21.875);
assert.ok(BRAND_MARK_RING_CLASS.split(" ").includes(`rounded-[${pct}%]`));
// The ring is the dark variant only, 1px, spread with no blur or offset,
- // in the palette's ring colour; light and sepia get no shadow class at all.
+ // in the palette's ring colour; light gets no shadow class at all.
const shadows = BRAND_MARK_RING_CLASS.split(" ").filter((c) => c.includes("shadow"));
assert.deepEqual(shadows, ["dark:shadow-[0_0_0_1px_var(--mark-ring)]"]);
// The svg must not clip the ground a second time at that corner.
diff --git a/common/components/BrandMark.tsx b/common/components/BrandMark.tsx
@@ -11,7 +11,7 @@ export type BrandMarkPalette = Readonly<Record<MarkTone, string>>;
// clears it. So on dark — `.dark` on <html>, tokens.css's `@custom-variant
// dark` — the tile gets a 1px ring OUTSIDE it, in the palette's own dim
// (`--mark-ring`, set from `palette.dim` below), so the ring reads as part of
-// the mark. Light and sepia are untouched.
+// the mark. Light is untouched.
// - A box-shadow, so it takes no layout: the mark's size and the header's
// alignment are the same on every base.
// - `rounded-[21.875%]` is the ground's corner (MARK_GROUND_RX / MARK_VIEWBOX,
diff --git a/common/components/FiltersContainer.tsx b/common/components/FiltersContainer.tsx
@@ -28,9 +28,13 @@ import { useMediaQuery } from "../lib/useMediaQuery";
export function FiltersTrigger({
activeCount,
onApply,
+ applyDisabled = false,
}: {
activeCount: number;
onApply: () => void;
+ // Refused exactly when Search is ("Search in" has nothing ticked); the
+ // panel above it says which row to fix.
+ applyDisabled?: boolean;
}) {
const [open, setOpen] = useState(false);
// From sm the sheet comes in from the right like the menu; on a phone it
@@ -72,6 +76,7 @@ export function FiltersTrigger({
<Button
type="button"
className="w-full"
+ disabled={applyDisabled}
onClick={() => {
onApply();
setOpen(false);
diff --git a/common/components/FiltersPanel.tsx b/common/components/FiltersPanel.tsx
@@ -27,6 +27,7 @@ import { clearAdvanced } from "./exportAdvancedStorage";
import { ymdToInput, inputToYmd } from "../lib/ymd";
import {
useSearchSession,
+ SEARCH_IN_REFUSAL,
DEFAULT_MAX_HITS,
DEFAULT_FETCH_CONCURRENCY,
DEFAULT_FLUSH_INTERVAL_MS,
@@ -75,6 +76,13 @@ export default function FiltersPanel({
draftNov,
draftNop,
setDraftNop,
+ draftNotr,
+ setDraftNotr,
+ draftLc,
+ setDraftLc,
+ draftSearchInEmpty,
+ hasPostsCorpus,
+ hasSubs,
setDraftNov,
draftNol,
setDraftNol,
@@ -99,11 +107,6 @@ export default function FiltersPanel({
setHitLimit,
} = useSearchSession();
- // Only offer the Posts type toggle on a site that actually ships a posts
- // corpus, so a pure-video deployment's filter row is unchanged.
- const { postsManifest } = useSearchData();
- const hasPostsCorpus = (postsManifest?.channels.length ?? 0) > 0;
-
const toggleState = useCallback(
(state: VideoState, keep: boolean) => {
setDraftStates((prev) => {
@@ -184,6 +187,7 @@ export default function FiltersPanel({
onDelete={handleDeleteProfile}
onRevert={handleRevertProfile}
onShareCurrentSearch={handleShareCurrentSearch}
+ saveDisabled={draftSearchInEmpty}
/>
{channelOptions.length > 0 && (
@@ -407,6 +411,60 @@ export default function FiltersPanel({
</div>
)}
<TagChipRow />
+ {/* What a plain query reads — the default "transcripts" leaf, not a
+ leaf asked for by name in the builder. Posts is offered only
+ where the site ships posts and Live chat only where it has live
+ chat; with nothing ticked Search is refused and this row says so.
+ Type below still says which records are listed. */}
+ <div
+ className="flex flex-wrap items-center gap-x-3 gap-y-1"
+ data-testid="search-in-row"
+ >
+ <span className="text-xs uppercase tracking-wide text-muted-foreground">
+ Search in
+ </span>
+ <label className="flex items-center gap-1.5 select-none">
+ <Checkbox
+ checked={!draftNotr}
+ onCheckedChange={(value) => setDraftNotr(value !== true)}
+ />
+ Transcripts
+ </label>
+ {hasPostsCorpus && (
+ <label className="flex items-center gap-1.5 select-none">
+ <Checkbox
+ checked={!draftNop}
+ onCheckedChange={(value) => setDraftNop(value !== true)}
+ />
+ Posts
+ </label>
+ )}
+ {hasSubs && (
+ <label className="flex items-center gap-1.5 select-none">
+ <Checkbox
+ checked={draftLc}
+ onCheckedChange={(value) => setDraftLc(value === true)}
+ />
+ Live chat
+ </label>
+ )}
+ {/* Always mounted, empty unless Search is refused: a live region
+ is announced when its content changes, which several screen
+ readers miss for a region that appears with its text. A polite
+ live region rather than role="status": this panel is always
+ mounted from xl, and a page-wide second "status" would be a
+ second answer to `getByRole("status")` (modal-digest.spec).
+ The Search button is described by the bar's own copy of these
+ words (this panel may be a closed sheet). */}
+ <span
+ aria-live="polite"
+ aria-atomic="true"
+ data-testid="search-in-empty"
+ className={cn("text-xs text-warning", draftSearchInEmpty && "basis-full")}
+ >
+ {draftSearchInEmpty ? SEARCH_IN_REFUSAL : ""}
+ </span>
+ </div>
<div className="flex flex-wrap items-center gap-x-3 gap-y-1">
<span className="text-xs uppercase tracking-wide text-muted-foreground">
Type
@@ -425,19 +483,6 @@ export default function FiltersPanel({
/>
Livestreams
</label>
- {/* The social-post corpus is a third media kind, not a video
- sub-type — without its own toggle the video/livestream pair
- would silently drop every post. Only offered when the site
- actually ships posts. */}
- {hasPostsCorpus && (
- <label className="flex items-center gap-1.5 select-none">
- <Checkbox
- checked={!draftNop}
- onCheckedChange={(value) => setDraftNop(value !== true)}
- />
- Posts
- </label>
- )}
</div>
<div className="flex flex-wrap items-center gap-x-3 gap-y-1">
<span className="text-xs uppercase tracking-wide text-muted-foreground">
@@ -625,6 +670,7 @@ function ProfilesRow({
onDelete,
onRevert,
onShareCurrentSearch,
+ saveDisabled,
}: {
profileNames: string[];
activeProfileName: string | null;
@@ -637,6 +683,9 @@ function ProfilesRow({
onDelete: () => void;
onRevert: () => void;
onShareCurrentSearch: () => void;
+ // A save commits the draft, so it is refused when Search is ("Search in"
+ // has nothing ticked).
+ saveDisabled: boolean;
}) {
const showRevert = activeProfileName != null && profileDirty;
return (
@@ -675,7 +724,7 @@ function ProfilesRow({
variant="link"
size="sm"
onClick={onSave}
- disabled={!activeProfileName || !profileDirty}
+ disabled={!activeProfileName || !profileDirty || saveDisabled}
className="text-xs text-muted-foreground"
>
Save
@@ -685,6 +734,7 @@ function ProfilesRow({
variant="link"
size="sm"
onClick={onSaveAs}
+ disabled={saveDisabled}
className="text-xs text-muted-foreground"
>
Save as…
diff --git a/common/components/SearchBar.tsx b/common/components/SearchBar.tsx
@@ -18,7 +18,7 @@ import { Button } from "./ui/button";
import QueryBuilder, { compactLayerActions } from "./QueryBuilder";
import { FiltersTrigger } from "./FiltersContainer";
import { useMediaQuery } from "../lib/useMediaQuery";
-import { useSearchSession } from "./SearchSessionContext";
+import { SEARCH_IN_REFUSAL, useSearchSession } from "./SearchSessionContext";
export default function SearchBar({ nav }: { nav?: ReactNode }) {
const {
@@ -30,6 +30,7 @@ export default function SearchBar({ nav }: { nav?: ReactNode }) {
groupStates,
queryDirty,
filtersDirty,
+ searchedThisPageLife,
handleResetLayers,
layersResetDisabled,
hasSubs,
@@ -40,6 +41,9 @@ export default function SearchBar({ nav }: { nav?: ReactNode }) {
draftNov,
draftNol,
draftNop,
+ draftNotr,
+ draftLc,
+ draftSearchInEmpty,
draftNaa,
draftNar,
draftStates,
@@ -56,11 +60,15 @@ export default function SearchBar({ nav }: { nav?: ReactNode }) {
const compact = isCompactRoot(draftRoot);
const layers = compactLayerActions(draftRoot, setDraftRoot);
- // What the Filters chip's badge counts: anything the reader has narrowed.
+ // What the Filters chip's badge counts: anything the reader has narrowed —
+ // and the "Search in" row off its default, which can widen (Live chat) as
+ // well as narrow, because either way a plain query no longer reads what it
+ // reads by default and the panel that says so may be behind the chip.
const activeFilters = useMemo(() => {
let n = 0;
if (draftExcludedChannels.size > 0) n += 1;
- if (draftNov || draftNol || draftNop) n += 1;
+ if (draftNotr || draftNop || draftLc) n += 1;
+ if (draftNov || draftNol) n += 1;
if (draftNaa || draftNar) n += 1;
if (draftStates.size < MISSING_STATES.length + 1) n += 1;
if (draftDateFrom || draftDateTo) n += 1;
@@ -71,6 +79,8 @@ export default function SearchBar({ nav }: { nav?: ReactNode }) {
draftNov,
draftNol,
draftNop,
+ draftNotr,
+ draftLc,
draftNaa,
draftNar,
draftStates,
@@ -79,11 +89,27 @@ export default function SearchBar({ nav }: { nav?: ReactNode }) {
draftTags,
]);
+ // The line under the bar says what to do: after an edit that is not applied
+ // yet, and before the first Search of the page life, when the results area
+ // is empty and waits for it. Not while Search is refused ("Search in" has
+ // nothing ticked) — the panel's own line says what to fix.
+ const promptApply =
+ !draftSearchInEmpty && (queryDirty || filtersDirty || !searchedThisPageLife);
+ // The hint points at the row, so it goes once the box is ticked.
+ const chatHint = hasSubs && !draftLc;
+
const submit = (
<Button
type="submit"
data-testid="search-submit"
data-dirty={queryDirty || filtersDirty ? "true" : "false"}
+ // "Search in" with nothing ticked reads nothing; commitSearch refuses it
+ // too, for Enter and the sheet's Apply. The reason is its description,
+ // from the always-mounted copy below (not only a `title`, which neither
+ // touch nor a keyboard reaches).
+ disabled={draftSearchInEmpty}
+ aria-describedby="search-in-refusal"
+ title={draftSearchInEmpty ? SEARCH_IN_REFUSAL : undefined}
variant={queryDirty || filtersDirty ? "default" : "outline"}
className="shrink-0 h-11 sm:h-9"
>
@@ -149,6 +175,13 @@ export default function SearchBar({ nav }: { nav?: ReactNode }) {
{!compact && (
<div className="flex items-center gap-2 flex-wrap">{submit}</div>
)}
+ {/* The Search button's description. Here rather than pointing at the
+ Filters panel's line, which below xl lives in a sheet that is not
+ mounted while closed. Not a live region: the panel's line is the
+ one announced, where the box was unticked. */}
+ <span id="search-in-refusal" className="sr-only">
+ {draftSearchInEmpty ? SEARCH_IN_REFUSAL : ""}
+ </span>
{/* The chip row. Rendered OUTSIDE the `mounted` gate so the workspace
nav exists as plain anchors before hydration — a nav click lost to
@@ -193,6 +226,7 @@ export default function SearchBar({ nav }: { nav?: ReactNode }) {
<FiltersTrigger
activeCount={activeFilters}
onApply={commitSearch}
+ applyDisabled={draftSearchInEmpty}
/>
)}
{!inlineFilters && (
@@ -220,23 +254,24 @@ export default function SearchBar({ nav }: { nav?: ReactNode }) {
{/* One status line under the bar instead of two hints competing for room
inside it — and a SIBLING of the form, so the pinned block is never
more than the input row plus the chips. */}
- {(queryDirty || filtersDirty || hasSubs) && (
+ {(promptApply || chatHint) && (
<p className="flex flex-wrap items-center gap-x-3 gap-y-0.5 text-xs">
{/* The site's accent, not the warning hue: an unapplied edit is
the next step, not a fault. `--brand` clears 4.5:1 on every
base's page ground (lib/brand.ts MIN_ACCENT_CONTRAST). */}
- {(queryDirty || filtersDirty) && (
+ {promptApply && (
<span className="text-brand">
Press Enter or click Search to apply
</span>
)}
- {hasSubs && (
+ {chatHint && (
<span
className="text-muted-foreground"
title={`${liveChatTotalCount} videos have live chat`}
>
Live chat available on {liveChatTotalCount} video
- {liveChatTotalCount === 1 ? "" : "s"} — try scope: Live chat.
+ {liveChatTotalCount === 1 ? "" : "s"} — tick Live chat under
+ Search in.
</span>
)}
</p>
diff --git a/common/components/SearchDataContext.tsx b/common/components/SearchDataContext.tsx
@@ -58,6 +58,13 @@ export type SearchDataValue = {
// many). Merged across origins in hub mode. Null / empty channel list when
// this site ships no posts corpus.
postsManifest: PostsManifest | null;
+ // The two manifests above have answered (or failed), so a null or an empty
+ // one means "this site has none", not "not answered yet". The "Search in"
+ // row waits for it before it refuses a Search or runs one: until then a
+ // stored row cannot be told apart from an empty one. Hub mode: the merge
+ // covers the ready archives only, and an archive is ready only once both
+ // its manifests have settled, so it is `summariesReady`.
+ manifestsSettled: boolean;
// The channel selection model, resolved mode-appropriately.
channels: ChannelOption[];
groups: ChannelGroup[];
@@ -147,8 +154,17 @@ export function useSearchData(): SearchDataValue {
// share links, and saved profiles round-trip exactly as before.
export function SingleSiteDataProvider({ children }: { children: ReactNode }) {
const summariesState = useSummaries("");
- const subsManifest = useSubsManifest("").data ?? null;
- const postsManifest = usePostsManifest("").data ?? null;
+ const subsQuery = useSubsManifest("");
+ const postsQuery = usePostsManifest("");
+ const subsManifest = subsQuery.data ?? null;
+ const postsManifest = postsQuery.data ?? null;
+ // Settled at the first failure, not after the retry: a site with no live
+ // chat ships no subs manifest, and its 404 must not hold a search for the
+ // retry's second. A retry that does answer later still arrives (the row then
+ // re-runs), as a late manifest always did.
+ const manifestsSettled =
+ (subsQuery.data !== undefined || subsQuery.isError || subsQuery.failureCount > 0) &&
+ (postsQuery.data !== undefined || postsQuery.isError || postsQuery.failureCount > 0);
const aliases = useSearchAliases("");
const curatedTags = useCuratedTags("");
const manifest = summariesState.manifest;
@@ -192,6 +208,7 @@ export function SingleSiteDataProvider({ children }: { children: ReactNode }) {
summariesState,
subsManifest,
postsManifest,
+ manifestsSettled,
channels,
groups,
defaultGroupId,
@@ -204,6 +221,7 @@ export function SingleSiteDataProvider({ children }: { children: ReactNode }) {
summariesState,
subsManifest,
postsManifest,
+ manifestsSettled,
channels,
groups,
defaultGroupId,
@@ -815,6 +833,7 @@ export function MultiSiteDataProvider({
summariesState,
subsManifest,
postsManifest,
+ manifestsSettled: summariesState.summariesReady,
channels,
groups,
defaultGroupId,
diff --git a/common/components/SearchResults.tsx b/common/components/SearchResults.tsx
@@ -34,6 +34,7 @@ import { vodExpiry } from "../lib/vodExpiry";
import { Button } from "./ui/button";
import { LayerSwatch } from "./LayerSwatch";
import { SCOPE_LABELS } from "./QueryLeafView";
+import type { LayerScope } from "../lib/searchQuery";
import { ChartShapeControls } from "./charts/ChartShapeControls";
import { SearchChartPanel } from "./charts/SearchChartPanel";
import {
@@ -55,6 +56,7 @@ const CARD_HIT_CAP = 8;
export default function SearchResults() {
const {
+ searchedThisPageLife,
hasActiveQuery,
resultGroups,
totalHits,
@@ -112,6 +114,13 @@ export default function SearchResults() {
// results" and the footer sit underneath it and cannot be reached.
const selectionBar = view !== "chart" && resultGroups.length > 0 && selectedCount > 0;
+ // A clear screen until the visitor asks: no count, no controls, no chart and
+ // no listing — the bar, the page's intro and the footer, with the bar's
+ // "Press Enter or click Search" line saying what to do. An empty Search shows
+ // every video; a link with a query or a filter shows its results on load.
+ // `resultGroups` is still computed underneath.
+ if (!searchedThisPageLife) return null;
+
return (
<section
data-testid="results-section"
@@ -762,7 +771,15 @@ const ResultCard = memo(function ResultCard({
of them "Transcripts", which is where the hits did NOT come
from. */}
<span className="text-[10px] uppercase tracking-wide text-muted-foreground">
- {SCOPE_LABELS[leafInfo.scope]}
+ {
+ SCOPE_LABELS[
+ sectionScope(
+ leafInfo.scope,
+ buckets.get(leafId) ?? hits,
+ !!group.post,
+ )
+ ]
+ }
</span>
<span className="font-mono text-xs text-muted-foreground truncate">
{leafInfo.query}
@@ -910,9 +927,10 @@ const HitRow = memo(function HitRow({
{hit.scope === "metadata" ? "—" : formatSeconds(hit.start)}
</span>
)}
- {hit.track && hit.track !== "live_chat" && (
- <TrackBadge track={hit.track} />
- )}
+ {/* Every track wears its badge, live chat included: with "Search in"
+ a plain query's live-chat hits share a section with its transcript
+ hits, and the badge is what tells the two apart. */}
+ {hit.track && <TrackBadge track={hit.track} />}
<span className="flex-1 min-w-0 text-sm">
{highlight(hit.text, query, useRegex)}
</span>
@@ -922,6 +940,23 @@ const HitRow = memo(function HitRow({
});
HitRow.displayName = "HitRow";
+// What a section's bar names. Its leaf's scope — except that a plain
+// ("transcripts") leaf reads what "Search in" ticks, so its section on a post's
+// card holds post hits, and on a video whose only matches were in its live chat
+// holds chat hits; the bar names what is there rather than a kind that was not
+// read. Mixed transcript and chat hits keep "Transcripts", each chat hit
+// wearing its "live chat" badge.
+function sectionScope(
+ scope: LayerScope,
+ hits: ReadonlyArray<LayerHit>,
+ isPost: boolean,
+): LayerScope {
+ if (scope !== "transcripts") return scope;
+ if (isPost) return "posts";
+ if (hits.length > 0 && hits.every((h) => h.scope === "chat")) return "chat";
+ return scope;
+}
+
// Which social platform a post came from. Reuses the TrackBadge shape so post
// rows sit visually alongside chat rows rather than inventing a new idiom.
function PostBadge({ platform }: { platform: string }) {
diff --git a/common/components/SearchSessionContext.tsx b/common/components/SearchSessionContext.tsx
@@ -35,16 +35,21 @@ import {
type TreeProgress,
} from "../lib/searchEval";
import {
+ applySearchIn,
canonicalHash,
emptyRoot,
forEachLeaf,
isNodeActive,
parseRoot,
rootFromLegacy,
+ searchInReadsNothing,
+ searchInUnderTags,
stringifyRoot,
type GroupNode,
type LayerScope,
+ type SearchIn,
} from "../lib/searchQuery";
+import { foldSearchIn } from "../lib/search/searchIn";
import {
VIDEO_STATES,
summaryState,
@@ -52,6 +57,7 @@ import {
} from "../lib/availability";
import { useUrlParams, writeUrlParams } from "./urlState";
import {
+ FILTER_URL_KEYS_V1,
buildShareSearchParams,
hasShareV1,
parseShareV1,
@@ -178,6 +184,10 @@ export type LeafInfo = {
// Advanced-pipeline defaults, surfaced so the bar's Advanced-options controls
// (and their "Reset to defaults" button) share the exact same constants.
export const DEFAULT_MAX_HITS = 500;
+
+// The words the Filters panel's "Search in" row and the Search button's
+// description say while nothing the site offers is ticked.
+export const SEARCH_IN_REFUSAL = "Search in: pick at least one";
export const DEFAULT_FETCH_CONCURRENCY = 6;
export const DEFAULT_FLUSH_INTERVAL_MS = 120;
@@ -197,13 +207,23 @@ export type GroundingMode = "search" | "selection";
// export-filter parser.
const SELECTION_KEY = "ytdlp-tb:selection";
-// Whether a search has run in this page life: a commit, a profile load, or a
-// query arriving on the URL. Module state, so it survives client-side
-// navigation and resets on a reload or a new visit. A stored query is held
-// (see `runHeld`) only on the page life's FIRST load — the hub mounts a fresh
-// provider per route, and `/` → `/ask` must keep grounding in the search the
-// visitor just ran there.
+// Two facts about this page life, each with one job. Module state, so both
+// survive client-side navigation and reset on a reload or a new visit.
+//
+// `ranThisPageLife`: a search has run — a commit (Search, Enter, a Filters
+// Apply), a profile load, or an active query arriving on the URL. A stored
+// query is held (see `runHeld`) while it is unset: on the page life's first
+// load, and on any later mount while nothing has run. Once it is set a later
+// mount runs the stored query — the hub mounts a fresh provider per route, and
+// `/` → `/ask` must keep grounding in the search the visitor just ran there. A
+// filter-only link does not set it: it shows its results, but it ran no query.
+//
+// `askedThisPageLife`: the visitor has asked — a search ran, or a link carries
+// a query or a filter (`urlAsks`). The results area shows nothing until it is
+// set (`searchedThisPageLife`, its reactive copy): until the visitor asks, the
+// page is the bar, the intro and the footer.
let ranThisPageLife = false;
+let askedThisPageLife = false;
function loadSelection(): string[] {
if (typeof window === "undefined") return [];
@@ -257,6 +277,22 @@ function urlHasAnyFilterParam(): boolean {
return FILTER_URL_KEYS.some((k) => p.has(k));
}
+// A link that carries a query (`qt`, legacy `q` / `m`) or a filter (`tg`, the
+// share-link keys, the legacy filter keys) is the visitor asking: it shows its
+// results on load, as a Search would (`askedThisPageLife`). A video (`v`, `t`)
+// or a chart's shape (`view`, `cs`) alone is not a search.
+const ASKING_URL_KEYS: readonly string[] = [
+ "qt",
+ "q",
+ "tg",
+ ...FILTER_URL_KEYS,
+ ...FILTER_URL_KEYS_V1,
+];
+
+function urlAsks(params: URLSearchParams): boolean {
+ return ASKING_URL_KEYS.some((k) => params.has(k));
+}
+
// Resolve a stored snapshot's per-channel deltas (plus group defaults) into
// the concrete set of EXCLUDED channel names.
function snapshotToExcluded(
@@ -365,10 +401,17 @@ function useSearchSessionState() {
>(() => new Set());
const [committedNov, setCommittedNov] = useState(false);
const [committedNol, setCommittedNol] = useState(false);
- // Third media kind: social posts. Videos and livestreams are a binary split
- // (isLivestream ? nol : nov); a post is neither, so it needs its own toggle
- // or the existing two would silently drop the whole posts corpus.
+ // Posts, the "Search in" row's second box (it was a third media kind in the
+ // Type row until release 16): unticked, a plain query reads no posts. A leaf
+ // of scope "Posts" reads them whatever it says.
const [committedNop, setCommittedNop] = useState(false);
+ // The rest of the "Search in" row (release 16) beside `nop`: what a plain
+ // query — a "transcripts" leaf — reads. Stored as off-the-default booleans
+ // like the others: `notr` = Transcripts unticked, `lc` = Live chat ticked.
+ // The row is applied to the committed tree just before it runs
+ // (`applySearchIn`); the tree itself, its hash and its `qt=` do not change.
+ const [committedNotr, setCommittedNotr] = useState(false);
+ const [committedLc, setCommittedLc] = useState(false);
const [committedNaa, setCommittedNaa] = useState(false);
const [committedNar, setCommittedNar] = useState(false);
// Availability is a six-state enum (VideoState), not a set of "exclude X"
@@ -419,6 +462,8 @@ function useSearchSessionState() {
>(() => new Set());
const [draftNov, setDraftNov] = useState(false);
const [draftNop, setDraftNop] = useState(false);
+ const [draftNotr, setDraftNotr] = useState(false);
+ const [draftLc, setDraftLc] = useState(false);
const [draftNol, setDraftNol] = useState(false);
const [draftNaa, setDraftNaa] = useState(false);
const [draftNar, setDraftNar] = useState(false);
@@ -457,18 +502,14 @@ function useSearchSessionState() {
const [view, setView] = useState<"results" | "chart">("results");
const [chartShape, setChartShape] = useState<ChartShape | null>(null);
- // Tracks whether the user has committed at least once this session. Used
- // by the placeholder copy below — until the user has searched, we show a
- // "tip" hint rather than the "no matching videos" empty state.
- const [, setSearchExecuted] = useState<boolean>(() => {
- if (typeof window === "undefined") return true;
- const params = new URLSearchParams(window.location.search);
- return !(
- params.has("v") &&
- !params.has("qt") &&
- (params.get("q") ?? "") === ""
- );
- });
+ // The reactive copy of `askedThisPageLife`, which stays the source of truth
+ // across remounts: a mount later in the page life starts from it, and every
+ // place that sets it sets this too. False until the visitor asks, and the
+ // results area renders nothing until then. False on the server, where the
+ // module variable is never set, so the first client render matches.
+ const [searchedThisPageLife, setSearchedThisPageLife] = useState(
+ () => askedThisPageLife,
+ );
// Defer rendering of the QueryBuilder to the client. The builder's leaf
// IDs come from a module-scoped counter that's necessarily out of sync
@@ -485,6 +526,7 @@ function useSearchSessionState() {
summariesState,
subsManifest,
postsManifest,
+ manifestsSettled,
channels: channelModel,
groups: manifestGroups,
channelKeyOf,
@@ -500,13 +542,72 @@ function useSearchSessionState() {
// only load subs manifests when a chat leaf is actually in the tree.
const liveChatTotalCount = subsManifest?.liveChatTotalCount ?? 0;
const hasSubs = subsManifest !== null && liveChatTotalCount > 0;
+ const hasPostsCorpus = (postsManifest?.channels.length ?? 0) > 0;
+
+ // ─── Search in ────────────────────────────────────────────────────────────
+ // What the row ticks, as far as this site can honour it: Posts counts only
+ // where the site ships posts and Live chat only where it has live chat (the
+ // panel offers neither box otherwise), so a stored `lc` on a site without
+ // chat asks for no manifest that is not there.
+ const draftOffered = useMemo<SearchIn>(
+ () => ({
+ transcripts: !draftNotr,
+ posts: !draftNop && hasPostsCorpus,
+ chat: draftLc && hasSubs,
+ }),
+ [draftNotr, draftNop, draftLc, hasPostsCorpus, hasSubs],
+ );
+ // What the rewrite reads: the same, less a posts copy under a tag filter,
+ // which no post can pass (`searchInUnderTags`).
+ const committedSearchIn = useMemo<SearchIn>(
+ () =>
+ searchInUnderTags(
+ {
+ transcripts: !committedNotr,
+ posts: !committedNop && hasPostsCorpus,
+ chat: committedLc && hasSubs,
+ },
+ committedTags.size > 0,
+ ),
+ [committedNotr, committedNop, committedLc, hasPostsCorpus, hasSubs, committedTags],
+ );
+ const draftSearchIn = useMemo<SearchIn>(
+ () => searchInUnderTags(draftOffered, draftTags.size > 0),
+ [draftOffered, draftTags],
+ );
+ // Nothing ticked that this site offers: Search, Enter, Apply and the profile
+ // saves all refuse, and the panel says which row to fix. Only once the
+ // manifests have answered: until then Posts and Live chat read as not
+ // offered, and a stored row with Transcripts unticked would read as empty.
+ // The run waits for them too (below), so the decision is held, not guessed.
+ const draftSearchInEmpty =
+ manifestsSettled && searchInReadsNothing(draftOffered);
+ // The committed tree as it RUNS: every "transcripts" leaf rewritten to read
+ // the kinds ticked. `committedRoot` stays the tree the visitor built — the
+ // builder, the result cards' leaf sections and `qt=` read that one, and the
+ // run's progress is folded back onto it (`foldSearchIn`).
+ const committedTree = useMemo(
+ () => applySearchIn(committedRoot, committedSearchIn),
+ [committedRoot, committedSearchIn],
+ );
+ const committedTreeHash = useMemo(
+ () => canonicalHash(committedTree.root),
+ [committedTree],
+ );
+ const draftTreeRoot = useMemo(
+ () => applySearchIn(draftRoot, draftSearchIn).root,
+ [draftRoot, draftSearchIn],
+ );
+ // Read from the rewritten trees, so a plain query with Live chat ticked
+ // loads the subs manifests (and with Posts ticked, the posts manifests)
+ // exactly as a leaf asked for by name always has.
const draftHasChatLeaf = useMemo(
- () => anyLeafHasScope(draftRoot, "chat"),
- [draftRoot],
+ () => anyLeafHasScope(draftTreeRoot, "chat"),
+ [draftTreeRoot],
);
const committedHasChatLeaf = useMemo(
- () => anyLeafHasScope(committedRoot, "chat"),
- [committedRoot],
+ () => anyLeafHasScope(committedTree.root, "chat"),
+ [committedTree],
);
const needsChatManifests = draftHasChatLeaf || committedHasChatLeaf;
// Origin-qualified refs. In single-site mode the manifest slugs are bare, so
@@ -553,12 +654,12 @@ function useSearchSessionState() {
// manifests when a posts leaf is actually present, then flatten them into the
// set of post slugs the tree may match.
const draftHasPostsLeaf = useMemo(
- () => anyLeafHasScope(draftRoot, "posts"),
- [draftRoot],
+ () => anyLeafHasScope(draftTreeRoot, "posts"),
+ [draftTreeRoot],
);
const committedHasPostsLeaf = useMemo(
- () => anyLeafHasScope(committedRoot, "posts"),
- [committedRoot],
+ () => anyLeafHasScope(committedTree.root, "posts"),
+ [committedTree],
);
const needsPostsManifests = draftHasPostsLeaf || committedHasPostsLeaf;
const postsRefs = useMemo(
@@ -661,11 +762,12 @@ function useSearchSessionState() {
const pipelineRef = useRef<TreeController | null>(null);
const { openTranscript } = usePlayer();
- // Reset the live hit cap on each new committed query.
+ // Reset the live hit cap on each new committed query — including the same
+ // query read in other kinds ("Search in").
useEffect(() => {
setHitLimit(hitBatchSize);
// eslint-disable-next-line react-hooks/exhaustive-deps
- }, [committedHash]);
+ }, [committedHash, committedTreeHash]);
// ─── Channel-grouping setup ────────────────────────────────────────────────
// Selection identity is the stable channelKey from the provider's model. In
@@ -773,7 +875,7 @@ function useSearchSessionState() {
[committedTags],
);
- const filterKey = `${committedChannelsKey}|${committedNov ? 1 : 0}|${committedNop ? 1 : 0}|${committedNol ? 1 : 0}|${committedNaa ? 1 : 0}|${committedNar ? 1 : 0}|${committedStatesKey}|${committedDateFrom}|${committedDateTo}|${committedTagsKey}`;
+ const filterKey = `${committedChannelsKey}|${committedNov ? 1 : 0}|${committedNop ? 1 : 0}|${committedNotr ? 1 : 0}|${committedLc ? 1 : 0}|${committedNol ? 1 : 0}|${committedNaa ? 1 : 0}|${committedNar ? 1 : 0}|${committedStatesKey}|${committedDateFrom}|${committedDateTo}|${committedTagsKey}`;
// The scope universe is videos AND posts. The two namespaces are disjoint;
// searchEval partitions them per-leaf (see EvalCtx.postScopeSlugs) so a posts
@@ -788,12 +890,17 @@ function useSearchSessionState() {
// passesFilter gives an untagged video. See tagFilteredPostScope below:
// the DEFAULT scope is only half of it, because a `posts`-scope leaf
// carries its own slug set past this.
- if (!committedNop && committedTags.size === 0) {
+ //
+ // Not gated on `nop` (release 16, review M2): the Posts box governs the
+ // plain query's posts copy, through `applySearchIn`, and a leaf of scope
+ // "Posts" was asked for by name and reads posts whatever the row says.
+ // `postScopeSlugs` is empty unless some leaf reads posts.
+ if (committedTags.size === 0) {
for (const slug of postScopeSlugs) out.push(slug);
}
return out;
// eslint-disable-next-line react-hooks/exhaustive-deps
- }, [transcripts, filterKey, postScopeSlugs, committedNop]);
+ }, [transcripts, filterKey, postScopeSlugs]);
// The posts slug set as the PIPELINE sees it. A `posts`-scope leaf is fed
// from here and not from the global scope above, so gating only the global
@@ -809,8 +916,9 @@ function useSearchSessionState() {
[committedTags, postScopeSlugs],
);
- // False while a restored query is held (see `runHeld`): the results area
- // shows the browse listing under the restored filters, and nothing fetches.
+ // False while a restored query is held (see `runHeld`): nothing fetches, and
+ // the results area is empty until the visitor asks — or, when a filter-only
+ // link asked, it lists every video under the link's filters.
const hasActiveQuery = useMemo(
() => !runHeld && isNodeActive(committedRoot),
[committedRoot, runHeld],
@@ -830,11 +938,15 @@ function useSearchSessionState() {
return;
}
if (!transcripts) return;
+ // "Search in" counts Posts and Live chat only once the site's manifests
+ // say it has them; a run before that would read the wrong kinds (with
+ // Transcripts unticked, transcripts) and run again. Hold it.
+ if (!manifestsSettled) return;
if (needsChatManifests && !subsManifestReady) return;
if (needsPostsManifests && !postsManifestReady) return;
const controller = runQueryTree({
- root: committedRoot,
+ root: committedTree.root,
runtime: searchRuntime,
globalScope: globalScopeSlugs,
summaries: transcripts,
@@ -843,7 +955,9 @@ function useSearchSessionState() {
initialHitLimit: hitLimit,
concurrency: fetchConcurrency,
flushIntervalMs,
- emit: (p) => setTreeProgress(p),
+ // The copies "Search in" made report under ids of their own; fold them
+ // back under the leaf the visitor built before anything reads them.
+ emit: (p) => setTreeProgress(foldSearchIn(p, committedTree)),
});
pipelineRef.current = controller;
return () => {
@@ -857,8 +971,10 @@ function useSearchSessionState() {
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [
committedHash,
+ committedTreeHash,
hasActiveQuery,
transcripts,
+ manifestsSettled,
filterKey,
needsChatManifests,
subsManifestReady,
@@ -1024,7 +1140,9 @@ function useSearchSessionState() {
const indexLoading =
hasActiveQuery &&
- (!summariesReady || (needsChatManifests && !subsManifestReady));
+ (!summariesReady ||
+ !manifestsSettled ||
+ (needsChatManifests && !subsManifestReady));
// Browse mode still needs the summaries before it can list anything.
const summariesLoading = !summariesReady;
const searching = hasActiveQuery && totalToProcess > 0;
@@ -1036,6 +1154,8 @@ function useSearchSessionState() {
!sameSet(draftExcludedChannels, committedChannels) ||
draftNov !== committedNov ||
draftNop !== committedNop ||
+ draftNotr !== committedNotr ||
+ draftLc !== committedLc ||
draftNol !== committedNol ||
draftNaa !== committedNaa ||
draftNar !== committedNar ||
@@ -1052,6 +1172,11 @@ function useSearchSessionState() {
);
const snap: FilterSnapshot = { channels: deltas };
if (committedNov) snap.nov = true;
+ // `nop` was missing here, so a profile with Posts unticked always read as
+ // changed against the committed state it had just loaded.
+ if (committedNop) snap.nop = true;
+ if (committedNotr) snap.notr = true;
+ if (committedLc) snap.lc = true;
if (committedNol) snap.nol = true;
if (committedNaa) snap.naa = true;
if (committedNar) snap.nar = true;
@@ -1068,6 +1193,9 @@ function useSearchSessionState() {
channelOptions,
defaultSelectedChannels,
committedNov,
+ committedNop,
+ committedNotr,
+ committedLc,
committedNol,
committedNaa,
committedNar,
@@ -1088,6 +1216,8 @@ function useSearchSessionState() {
};
if (draftNov) snap.nov = true;
if (draftNop) snap.nop = true;
+ if (draftNotr) snap.notr = true;
+ if (draftLc) snap.lc = true;
if (draftNol) snap.nol = true;
if (draftNaa) snap.naa = true;
if (draftNar) snap.nar = true;
@@ -1103,6 +1233,8 @@ function useSearchSessionState() {
defaultSelectedChannels,
draftNov,
draftNop,
+ draftNotr,
+ draftLc,
draftNol,
draftNaa,
draftNar,
@@ -1117,6 +1249,8 @@ function useSearchSessionState() {
setCommittedExcludedChannels(new Set(draftExcludedChannels));
setCommittedNov(draftNov);
setCommittedNop(draftNop);
+ setCommittedNotr(draftNotr);
+ setCommittedLc(draftLc);
setCommittedNol(draftNol);
setCommittedNaa(draftNaa);
setCommittedNar(draftNar);
@@ -1131,6 +1265,8 @@ function useSearchSessionState() {
draftExcludedChannels,
draftNov,
draftNop,
+ draftNotr,
+ draftLc,
draftNol,
draftNaa,
draftNar,
@@ -1240,12 +1376,20 @@ function useSearchSessionState() {
let resolvedRoot = rootFromUrl;
// True only when the query came from the stored snapshot, never the URL.
let holdRestored = false;
+ // The stored profile / working snapshot.
+ const snapshot =
+ stored?.activeProfileName != null
+ ? (stored.profiles[stored.activeProfileName] ?? stored.working)
+ : (stored?.working ?? null);
+ // The "Search in" row always comes from here, whatever the URL carries: no
+ // link carries it (share-v1 has no key for it, by ruling), so a shared
+ // link reads with the visitor's own row. Before the row, `nop` was read
+ // from nowhere, and a stored Posts-unticked came back ticked on reload.
+ const initialNop = snapshot?.nop === true;
+ const initialNotr = snapshot?.notr === true;
+ const initialLc = snapshot?.lc === true;
if (!initialExcluded) {
// Fall back to stored profile / working snapshot.
- const snapshot =
- stored?.activeProfileName != null
- ? (stored.profiles[stored.activeProfileName] ?? stored.working)
- : (stored?.working ?? null);
const excluded = snapshotToExcluded(
snapshot,
channelOptions,
@@ -1273,13 +1417,19 @@ function useSearchSessionState() {
setDraftRoot(resolvedRoot);
setCommittedRoot(resolvedRoot);
if (holdRestored) setRunHeld(true);
- else setSearchExecuted(true);
- if (rootFromUrl && isNodeActive(rootFromUrl)) ranThisPageLife = true;
}
+ // After `holdRestored` is decided. Only an active URL query counts as a
+ // run; a filter-only link asks (its results show) without releasing a
+ // stored query on this mount or a later one.
+ if (rootFromUrl && isNodeActive(rootFromUrl)) ranThisPageLife = true;
+ if (ranThisPageLife || urlAsks(params)) askedThisPageLife = true;
+ setSearchedThisPageLife(askedThisPageLife);
setDraftExcludedChannels(initialExcluded);
setDraftNov(initialNov);
- setDraftNop(false);
+ setDraftNop(initialNop);
+ setDraftNotr(initialNotr);
+ setDraftLc(initialLc);
setDraftNol(initialNol);
setDraftNaa(initialNaa);
setDraftNar(initialNar);
@@ -1289,6 +1439,9 @@ function useSearchSessionState() {
setDraftTags(new Set(initialTags ?? []));
setCommittedExcludedChannels(initialExcluded);
setCommittedNov(initialNov);
+ setCommittedNop(initialNop);
+ setCommittedNotr(initialNotr);
+ setCommittedLc(initialLc);
setCommittedNol(initialNol);
setCommittedNaa(initialNaa);
setCommittedNar(initialNar);
@@ -1388,9 +1541,13 @@ function useSearchSessionState() {
}, [persistUiCollapse]);
const commitSearch = () => {
- setSearchExecuted(true);
+ // "Search in" with nothing ticked reads nothing: refused here, which is
+ // where Search, Enter and the sheet's Apply all arrive.
+ if (draftSearchInEmpty) return;
setRunHeld(false);
ranThisPageLife = true;
+ askedThisPageLife = true;
+ setSearchedThisPageLife(true);
setCommittedRoot(draftRoot);
const nextSnapshot = buildDraftSnapshot();
const current = loadStoredState() ?? emptyStoredState();
@@ -1435,6 +1592,8 @@ function useSearchSessionState() {
setDraftExcludedChannels(excluded);
setDraftNov(snapshot?.nov === true);
setDraftNop(snapshot?.nop === true);
+ setDraftNotr(snapshot?.notr === true);
+ setDraftLc(snapshot?.lc === true);
setDraftNol(snapshot?.nol === true);
setDraftNaa(snapshot?.naa === true);
setDraftNar(snapshot?.nar === true);
@@ -1455,6 +1614,8 @@ function useSearchSessionState() {
// Loading a profile commits it, which runs it — as it always has.
setRunHeld(false);
ranThisPageLife = true;
+ askedThisPageLife = true;
+ setSearchedThisPageLife(true);
applyDraftSnapshot(snapshot);
const excluded = snapshotToExcluded(
snapshot,
@@ -1463,6 +1624,11 @@ function useSearchSessionState() {
);
setCommittedExcludedChannels(excluded);
setCommittedNov(snapshot?.nov === true);
+ // `nop` was not committed here, so a loaded profile with Posts unticked
+ // kept searching posts until the next Search.
+ setCommittedNop(snapshot?.nop === true);
+ setCommittedNotr(snapshot?.notr === true);
+ setCommittedLc(snapshot?.lc === true);
setCommittedNol(snapshot?.nol === true);
setCommittedNaa(snapshot?.naa === true);
setCommittedNar(snapshot?.nar === true);
@@ -1516,6 +1682,8 @@ function useSearchSessionState() {
const handleSaveProfile = useCallback(() => {
if (!activeProfileName) return;
+ // A save commits the draft, so it refuses what Search refuses.
+ if (draftSearchInEmpty) return;
const snap = buildDraftSnapshot();
promoteDraftsToCommitted();
setProfiles((prev) => {
@@ -1528,6 +1696,7 @@ function useSearchSessionState() {
});
}, [
activeProfileName,
+ draftSearchInEmpty,
buildDraftSnapshot,
promoteDraftsToCommitted,
writeStorage,
@@ -1535,6 +1704,7 @@ function useSearchSessionState() {
const handleSaveAsProfile = useCallback(() => {
if (typeof window === "undefined") return;
+ if (draftSearchInEmpty) return;
const raw = window.prompt("Save current filters as a new profile named:");
if (raw == null) return;
const name = raw.trim();
@@ -1555,7 +1725,13 @@ function useSearchSessionState() {
return next;
});
setActiveProfileName(name);
- }, [profiles, buildDraftSnapshot, promoteDraftsToCommitted, writeStorage]);
+ }, [
+ profiles,
+ draftSearchInEmpty,
+ buildDraftSnapshot,
+ promoteDraftsToCommitted,
+ writeStorage,
+ ]);
const handleRenameProfile = useCallback(() => {
if (typeof window === "undefined") return;
@@ -2012,6 +2188,14 @@ function useSearchSessionState() {
setDraftNov,
draftNop,
setDraftNop,
+ // ── Search in (the rest of the row; Posts is `draftNop` above) ──
+ draftNotr,
+ setDraftNotr,
+ draftLc,
+ setDraftLc,
+ // Nothing the site offers is ticked: Search is disabled and refused.
+ draftSearchInEmpty,
+ hasPostsCorpus,
draftNol,
setDraftNol,
draftNaa,
@@ -2040,6 +2224,9 @@ function useSearchSessionState() {
hitBatchSize,
setHitLimit,
// ── Results ──
+ // False until the visitor asks in this page life; nothing in the results
+ // area renders until then (see `askedThisPageLife`).
+ searchedThisPageLife,
hasActiveQuery,
resultGroups,
totalHits,
diff --git a/common/components/SocialLinks.tsx b/common/components/SocialLinks.tsx
@@ -0,0 +1,109 @@
+import { useId } from "react";
+import type { SocialLink } from "../lib/settingsSchema";
+import {
+ headerSocialLinks,
+ safeSocialSvg,
+ scopeSvgIds,
+ sizeSocialSvg,
+ type HeaderWidth,
+} from "../lib/socialLinks";
+import { cn } from "../lib/utils";
+
+export type SocialLinksPlacement = "header" | "footer";
+
+// THE SOCIAL ROW, for any header or footer. The links are the operator's
+// (settings.json `socialLinks`, or a site's or the homepage's own list), each an
+// icon with its label as its accessible name — never text beside it.
+//
+// - `placement: "header"` shows the header's list for `width`
+// (headerSocialLinks): "wide", every link up to four (the `featured` ones
+// kept first); "narrow", only the `featured` ones. A header renders both,
+// each shown by CSS at its own widths; `"footer"` shows every link.
+// - Each link is a 36 px key around a 20 px glyph, 44 px under a coarse
+// pointer. The glyph is the link's colour (`--muted-foreground`, the
+// foreground on hover) when the icon is single-colour; an icon of two or
+// more colours keeps its own (normalizeSocialSvg).
+// - Focus: the ring colour's 2 px ring. In forced colours a box-shadow is not
+// drawn, so the browser's own focus outline is left in place there.
+// - The ids inside each inlined icon are scoped to this row and this link
+// (scopeSvgIds): a page inlines the same icon more than once, and a gradient
+// defined in a copy that is `display: none` would not paint in the others.
+// - An icon is inlined only if it passes the save-time check again
+// (safeSocialSvg). One that does not is not injected: the link shows its
+// label as text instead, so a bad file costs the icon, never the page.
+//
+// Server-safe and client-safe: no state, no effects; `useId` works in both.
+// Renders nothing when there is nothing to show, so a caller can drop the
+// column or the group around it on the same condition.
+// Complete literal class strings (Tailwind v4 scans them as written).
+// A key clips what it holds (`overflow-hidden`, `contain: paint`): an icon
+// paints inside its 36 px box and nowhere else. The focus ring is the key's
+// own box-shadow, outside that clip. A refused icon's text fallback is capped
+// at 10rem and ends in an ellipsis, its full label in `title`.
+const KEY =
+ "inline-flex size-9 shrink-0 items-center justify-center overflow-hidden [contain:paint] rounded-md text-muted-foreground transition-colors hover:bg-muted hover:text-foreground focus-visible:ring-2 focus-visible:ring-ring not-forced-colors:focus-visible:outline-none pointer-coarse:size-11 [&_svg]:size-5 [&_svg]:shrink-0";
+const TEXT_KEY =
+ "inline-block h-9 max-w-40 shrink-0 truncate rounded-md px-2 text-sm leading-9 text-muted-foreground transition-colors hover:bg-muted hover:text-foreground focus-visible:ring-2 focus-visible:ring-ring not-forced-colors:focus-visible:outline-none pointer-coarse:h-11 pointer-coarse:leading-[2.75rem]";
+
+export function SocialLinks({
+ links,
+ placement,
+ width = "wide",
+ className,
+}: {
+ links: readonly SocialLink[];
+ placement: SocialLinksPlacement;
+ // The header's list: which of its two widths this copy is.
+ width?: HeaderWidth;
+ className?: string;
+}) {
+ const scope = `sl${useId().replace(/[^A-Za-z0-9_-]/g, "")}`;
+ const shown = placement === "header" ? headerSocialLinks(links, width) : [...links];
+ if (shown.length === 0) return null;
+ return (
+ <ul
+ data-social-links={placement}
+ data-header-width={placement === "header" ? width : undefined}
+ className={cn(
+ "flex items-center list-none",
+ // The footer's column is left-aligned under its eyebrow: pull the first
+ // key out by its inset so the first glyph lines up with the text. The
+ // footer shows every link, so its row wraps rather than widen the page.
+ placement === "footer" && "flex-wrap -mx-2 pointer-coarse:-mx-3",
+ className,
+ )}
+ >
+ {shown.map((link, i) => {
+ const svg = safeSocialSvg(link.svg);
+ return (
+ <li key={`${link.url}-${i}`} className="flex">
+ {svg ? (
+ <a
+ href={link.url}
+ title={link.label}
+ aria-label={link.label}
+ target="_blank"
+ rel="noopener noreferrer"
+ className={KEY}
+ dangerouslySetInnerHTML={{
+ __html: sizeSocialSvg(scopeSvgIds(svg, `${scope}-${i}`)),
+ }}
+ />
+ ) : (
+ <a
+ href={link.url}
+ target="_blank"
+ rel="noopener noreferrer"
+ data-social-icon-refused=""
+ title={link.label}
+ className={TEXT_KEY}
+ >
+ {link.label}
+ </a>
+ )}
+ </li>
+ );
+ })}
+ </ul>
+ );
+}
diff --git a/common/components/SocialScroll.tsx b/common/components/SocialScroll.tsx
@@ -0,0 +1,52 @@
+"use client";
+
+import type { ReactNode } from "react";
+import { cn } from "../lib/utils";
+
+// THE SOCIAL ROW'S LAST RESORT: a box that scrolls the row sideways when it
+// still cannot fit (a header that has already made room). The row's END is in
+// view first, with no script and no change to the DOM or tab order: the box is
+// `direction: rtl`, whose first scroll position is its right edge, and the row
+// inside it is `ltr` and `w-max` (as wide as its links, so it overflows to the
+// left, where an rtl box can scroll). No scrollbar is drawn; touch, a
+// trackpad, shift + wheel and the keyboard still scroll it. When the row fits,
+// nothing about it changes.
+//
+// The one piece of script: a link that takes focus is scrolled into view
+// (`nearest`, inside the 4 px `scroll-padding`, so its ring shows). Chromium
+// does not bring a partly hidden link into view on focus inside an rtl box.
+//
+// `tabIndex={-1}`: Firefox makes a scroll container that overflows a tab stop
+// of its own, with no name. The links inside are the stops, and focus scrolls
+// them into view, so the box itself is taken out of the order.
+//
+// Put the row's own 4 px inline padding on the row (SocialLinks' className,
+// `px-1`) — the padding must be inside the scrolled content to be reachable —
+// and this box cancels it with `-mx-1`, so the key after the box (the header's
+// theme control) still meets the last link.
+export function SocialScroll({
+ children,
+ className,
+}: {
+ children: ReactNode;
+ className?: string;
+}) {
+ return (
+ <div
+ data-social-scroll=""
+ tabIndex={-1}
+ className={cn(
+ "min-w-0 overflow-x-auto [direction:rtl] [scrollbar-width:none] [&::-webkit-scrollbar]:hidden scroll-px-1 -mx-1 -my-1 py-1",
+ className,
+ )}
+ onFocus={(event) => {
+ const target = event.target as HTMLElement;
+ if (target !== event.currentTarget) {
+ target.scrollIntoView({ block: "nearest", inline: "nearest" });
+ }
+ }}
+ >
+ {children}
+ </div>
+ );
+}
diff --git a/common/components/ThemeMenu.tsx b/common/components/ThemeMenu.tsx
@@ -1,86 +0,0 @@
-"use client";
-
-import { Palette } from "lucide-react";
-import { useTheme } from "./ThemeProvider";
-import {
- THEME_BASES,
- accentOptions,
- isThemeAccent,
- isThemeBase,
-} from "./themeConfig";
-import {
- DropdownMenu,
- DropdownMenuContent,
- DropdownMenuLabel,
- DropdownMenuRadioGroup,
- DropdownMenuRadioItem,
- DropdownMenuSeparator,
- DropdownMenuTrigger,
-} from "./ui/dropdown-menu";
-import { cn } from "../lib/utils";
-
-// The theme picker: a palette dropdown with two radio groups — the BASE
-// (System / Light / Sepia / Dark) and the ACCENT (the seven named accents, the
-// site's own tagged "default"; a custom-hex site adds its colour first as
-// "Site colour"). Sits beside the quick base toggle (ThemeToggle). Picking only
-// changes attributes on <html>; tokens.css does the rest. The `menuitemradio`
-// names are an e2e contract. Must render inside a <ThemeProvider/>.
-export function ThemeMenu({ className }: { className?: string }) {
- const { base, accent, siteAccent, setBase, setAccent } = useTheme();
-
- return (
- <DropdownMenu>
- <DropdownMenuTrigger
- aria-label="Choose theme"
- title="Theme"
- className={cn(
- "inline-flex h-8 w-8 items-center justify-center rounded-md border border-border text-muted-foreground transition-colors hover:text-foreground outline-none focus-visible:ring-2 focus-visible:ring-ring",
- className,
- )}
- >
- <Palette className="h-4 w-4" aria-hidden="true" />
- </DropdownMenuTrigger>
- <DropdownMenuContent align="end" className="min-w-52">
- <DropdownMenuLabel>Base</DropdownMenuLabel>
- <DropdownMenuRadioGroup
- aria-label="Base"
- value={base}
- onValueChange={(v) => {
- if (isThemeBase(v)) setBase(v);
- }}
- >
- {THEME_BASES.map((b) => (
- <DropdownMenuRadioItem key={b.id} value={b.id}>
- {b.label}
- </DropdownMenuRadioItem>
- ))}
- </DropdownMenuRadioGroup>
- <DropdownMenuSeparator />
- <DropdownMenuLabel>Accent</DropdownMenuLabel>
- <DropdownMenuRadioGroup
- aria-label="Accent"
- value={accent}
- onValueChange={(v) => {
- if (isThemeAccent(v)) setAccent(v);
- }}
- >
- {accentOptions(siteAccent).map((o) => (
- <DropdownMenuRadioItem key={o.id} value={o.id}>
- <span
- aria-hidden="true"
- className="size-3 shrink-0 rounded-full ring-1 ring-inset ring-foreground/15"
- style={{ background: o.swatch }}
- />
- <span>{o.label}</span>
- {o.isSiteDefault && (
- <span className="ml-auto pl-3 font-mono text-[0.625rem] uppercase tracking-[0.12em] text-muted-foreground">
- default
- </span>
- )}
- </DropdownMenuRadioItem>
- ))}
- </DropdownMenuRadioGroup>
- </DropdownMenuContent>
- </DropdownMenu>
- );
-}
diff --git a/common/components/ThemeProvider.tsx b/common/components/ThemeProvider.tsx
@@ -9,33 +9,34 @@ import {
useMemo,
useState,
} from "react";
-import { BASE_GROUNDS, DEFAULT_ACCENT, isAccentId, type AccentId } from "../lib/brand";
+import { BASE_GROUNDS, DEFAULT_ACCENT } from "../lib/brand";
import {
- ACCENT_KEY,
BASE_KEY,
LEGACY_MODE_KEY,
LEGACY_THEME_KEY,
+ RETIRED_BASE,
isThemeBase,
migrateLegacy,
nextBase,
resolveBase,
+ storedBase,
type ResolvedBase,
type ThemeAccent,
type ThemeBase,
} from "./themeConfig";
-// Runtime theme controller: the reader's BASE (light | sepia | dark | system)
-// and ACCENT (one of the seven, defaulting to the site's own).
+// Runtime theme controller: the reader's BASE (light | dark | system). The
+// ACCENT is the site's own (`siteAccent`), never the reader's: a stored pick
+// from before is neither read nor removed.
//
// The pre-paint <ThemeScript/> colours the FIRST paint by setting attributes on
// <html> from localStorage before hydration. Hydration can reset <html>'s
-// attributes to what the server rendered — the className without `.dark`, and
-// the SITE's `data-accent` over a reader's pick — so on mount this provider
-// RE-ASSERTS the persisted, resolved theme (data-base, .dark, data-accent,
-// theme-color) in a useLayoutEffect, before paint. That is idempotent: it
-// recomputes what the script computed from the same storage, so it never
-// fights the script and never flashes. After that it writes to the DOM only on
-// a reader's change and on a live OS-preference change.
+// attributes to what the server rendered — the className without `.dark` — so
+// on mount this provider RE-ASSERTS the persisted, resolved theme (data-base,
+// .dark, data-accent, theme-color) in a useLayoutEffect, before paint. That is
+// idempotent: it recomputes what the script computed from the same storage, so
+// it never fights the script and never flashes. After that it writes to the
+// DOM only on a reader's change and on a live OS-preference change.
type ThemeContextValue = {
/** The reader's choice, "system" included. */
@@ -44,16 +45,12 @@ type ThemeContextValue = {
resolvedBase: ResolvedBase;
/** Whether the dark base is applied. */
isDark: boolean;
- /** The accent applied: the reader's pick, else the site's. */
+ /** The accent applied: the site's own ("custom" for a site with its own
+ * hex). */
accent: ThemeAccent;
- /** The site's own accent ("custom" for a site with its own hex). */
- siteAccent: ThemeAccent;
setBase: (b: ThemeBase) => void;
- /** system → light → sepia → dark → system. */
+ /** system → light → dark → system. */
cycleBase: () => void;
- /** Picking the site's own accent REMOVES the stored pick, so the reader
- * follows the site's default from then on. */
- setAccent: (a: ThemeAccent) => void;
};
const ThemeContext = createContext<ThemeContextValue | null>(null);
@@ -68,9 +65,10 @@ function systemPrefersDark(): boolean {
}
}
-// What storage holds, after the same one-time legacy migration the pre-paint
-// script runs (a no-op when the script already ran it).
-function readStored(): { base: ThemeBase | null; accent: AccentId | null } {
+// The stored base, after the same one-time migrations the pre-paint script
+// runs (a no-op when the script already ran them): the legacy keys, and a
+// stored RETIRED_BASE rewritten to "light".
+function readStoredBase(): ThemeBase | null {
try {
const s = window.localStorage;
let base = s.getItem(BASE_KEY);
@@ -87,13 +85,10 @@ function readStored(): { base: ThemeBase | null; accent: AccentId | null } {
s.removeItem(LEGACY_THEME_KEY);
s.removeItem(LEGACY_MODE_KEY);
}
- const accent = s.getItem(ACCENT_KEY);
- return {
- base: isThemeBase(base) ? base : null,
- accent: isAccentId(accent) ? accent : null,
- };
+ if (base === RETIRED_BASE) s.setItem(BASE_KEY, "light");
+ return storedBase(base);
} catch {
- return { base: null, accent: null };
+ return null;
}
}
@@ -107,10 +102,9 @@ function applyToDom(resolved: ResolvedBase, accent: ThemeAccent) {
}
}
-function store(key: string, value: string | null) {
+function store(key: string, value: string) {
try {
- if (value === null) window.localStorage.removeItem(key);
- else window.localStorage.setItem(key, value);
+ window.localStorage.setItem(key, value);
} catch {
/* storage disabled: the choice holds for this page only */
}
@@ -126,22 +120,20 @@ export function ThemeProvider({
children: React.ReactNode;
}) {
// Initial state MUST equal what the server rendered (the defaults), so
- // hydration matches; the persisted values are adopted just below.
+ // hydration matches; the persisted base is adopted just below.
const [base, setBaseState] = useState<ThemeBase>(defaultBase);
- const [accent, setAccentState] = useState<ThemeAccent>(siteAccent);
const [systemDark, setSystemDark] = useState(false);
const [adopted, setAdopted] = useState(false);
+ const accent = siteAccent;
// Adopt the persisted choice before paint. The state updates re-render
// synchronously, and the effect below then re-asserts it to <html>.
useLayoutEffect(() => {
- const stored = readStored();
- setBaseState(stored.base ?? defaultBase);
- setAccentState(stored.accent ?? siteAccent);
+ setBaseState(readStoredBase() ?? defaultBase);
setSystemDark(systemPrefersDark());
setAdopted(true);
- // Effectively once: the defaults are stable props from the layout.
- }, [defaultBase, siteAccent]);
+ // Effectively once: the default is a stable prop from the layout.
+ }, [defaultBase]);
// A LIVE OS preference: "system" follows a change made while the page is
// open (the pre-paint script can only read it once).
@@ -175,34 +167,16 @@ export function ThemeProvider({
const cycleBase = useCallback(() => setBase(nextBase(base)), [base, setBase]);
- const setAccent = useCallback(
- (a: ThemeAccent) => {
- if (a === siteAccent) {
- setAccentState(a);
- store(ACCENT_KEY, null);
- return;
- }
- // "custom" only ever means THIS site's own colour (above); a named
- // accent is the only thing a reader can store.
- if (!isAccentId(a)) return;
- setAccentState(a);
- store(ACCENT_KEY, a);
- },
- [siteAccent],
- );
-
const value = useMemo<ThemeContextValue>(
() => ({
base,
resolvedBase,
isDark: resolvedBase === "dark",
accent,
- siteAccent,
setBase,
cycleBase,
- setAccent,
}),
- [base, resolvedBase, accent, siteAccent, setBase, cycleBase, setAccent],
+ [base, resolvedBase, accent, setBase, cycleBase],
);
return <ThemeContext.Provider value={value}>{children}</ThemeContext.Provider>;
diff --git a/common/components/ThemeScript.tsx b/common/components/ThemeScript.tsx
@@ -2,24 +2,20 @@ import { buildThemeScript, type ThemeBase } from "./themeConfig";
// Server component (NOT "use client"): renders an inline <script> that runs
// synchronously BEFORE first paint, so there is no flash of the wrong theme.
-// From localStorage it migrates the retired theme/mode keys once, then sets
-// `data-base` (the resolved light | sepia | dark ground) and `.dark` on <html>,
-// `data-accent` when the reader stored an accent of their own, the browser
-// chrome colour, and — last — the `data-theme-ready` marker the e2e no-flash
-// tests wait for. The script's source and its order are
-// themeConfig.buildThemeScript (unit-tested in node:vm).
+// From localStorage it migrates the retired theme/mode keys once and a stored
+// retired base to "light", then sets `data-base` (the resolved light | dark
+// ground) and `.dark` on <html>, the browser chrome colour, and — last — the
+// `data-theme-ready` marker the e2e no-flash tests wait for. The script's
+// source and its order are themeConfig.buildThemeScript (unit-tested in
+// node:vm).
//
-// The site's OWN accent is not this script's business: the layout
-// server-renders it as `<html data-accent>`, and the script only replaces it
-// with a reader's valid stored pick. Works identically for static-export sites
-// and the server editor. Pair with `suppressHydrationWarning` on <html>, since
-// this mutates the element before React hydrates — and with ThemeProvider,
-// which re-asserts the same attributes after hydration.
-export function ThemeScript({
- defaultBase = "system",
-}: {
- defaultBase?: ThemeBase;
-}) {
+// The accent is not this script's business: the layout server-renders the
+// site's own as `<html data-accent>`, and a reader's stored pick from before
+// is not read. Works identically for static-export sites and the server
+// editor. Pair with `suppressHydrationWarning` on <html>, since this mutates
+// the element before React hydrates — and with ThemeProvider, which
+// re-asserts the same attributes after hydration.
+export function ThemeScript({ defaultBase = "system" }: { defaultBase?: ThemeBase }) {
return (
<script
dangerouslySetInnerHTML={{ __html: buildThemeScript({ defaultBase }) }}
diff --git a/common/components/ThemeToggle.tsx b/common/components/ThemeToggle.tsx
@@ -1,23 +1,37 @@
"use client";
-import { BookOpen, Monitor, Moon, Sun } from "lucide-react";
+import { Monitor, Moon, Sun } from "lucide-react";
import { useTheme } from "./ThemeProvider";
import { nextBase, type ThemeBase } from "./themeConfig";
import { cn } from "../lib/utils";
-// The quick base toggle: cycles system → light → sepia → dark → system. The
-// icon shows the base in force, the label names the next one ("Switch to …" —
-// the e2e specs find it by /switch to/i). The accent lives in ThemeMenu. Must
-// be rendered inside a <ThemeProvider/>.
-const ICON = { system: Monitor, light: Sun, sepia: BookOpen, dark: Moon } as const;
+// The theme control: cycles the base, system → light → dark → system
+// (themeConfig THEME_BASES). The icon shows the base in force, the label names
+// the next one ("Switch to …" — the e2e specs find it by /switch to/i). There
+// is no accent control: each site shows its own. Must be rendered inside a
+// <ThemeProvider/>.
+const ICON = { system: Monitor, light: Sun, dark: Moon } as const;
const LABEL: Record<ThemeBase, string> = {
system: "system",
light: "light",
- sepia: "sepia",
dark: "dark",
};
-export function ThemeToggle({ className }: { className?: string }) {
+// `variant="bare"` (the homepage): dressed as a social link's key
+// (SocialLinks.tsx) — a 36 px box (44 px under a coarse pointer), no border, the
+// muted foreground and the faint hover square, a 20 px glyph, the ring
+// colour's 2 px focus ring and, in forced colours, the browser's own outline —
+// so it reads as the last key of the row it follows. The default is unchanged.
+const BARE =
+ "inline-flex size-9 shrink-0 items-center justify-center rounded-md text-muted-foreground transition-colors hover:bg-muted hover:text-foreground focus-visible:ring-2 focus-visible:ring-ring not-forced-colors:focus-visible:outline-none pointer-coarse:size-11";
+
+export function ThemeToggle({
+ className,
+ variant = "default",
+}: {
+ className?: string;
+ variant?: "default" | "bare";
+}) {
const { base, cycleBase } = useTheme();
const Icon = ICON[base];
@@ -29,11 +43,13 @@ export function ThemeToggle({ className }: { className?: string }) {
title={`Theme: ${LABEL[base]}`}
data-theme-base={base}
className={cn(
- "inline-flex h-8 w-8 items-center justify-center rounded-md border border-[var(--border)] text-[var(--muted-foreground)] transition-colors hover:text-[var(--foreground)] hover:border-[var(--border-strong,var(--border))]",
+ variant === "bare"
+ ? BARE
+ : "inline-flex h-8 w-8 items-center justify-center rounded-md border border-[var(--border)] text-[var(--muted-foreground)] transition-colors hover:text-[var(--foreground)] hover:border-[var(--border-strong,var(--border))]",
className,
)}
>
- <Icon className="h-4 w-4" aria-hidden="true" />
+ <Icon className={variant === "bare" ? "size-5" : "h-4 w-4"} aria-hidden="true" />
</button>
);
}
diff --git a/common/components/Wordmark.tsx b/common/components/Wordmark.tsx
@@ -13,15 +13,21 @@ import { splitWordmark } from "../lib/brand";
//
// Size, leading and truncation come from `className`. The face is Archivo,
// the display face on every base (common/styles/fonts.ts `--font-display`,
-// with the `wdth` axis the stretch needs).
+// with the `wdth` axis the stretch needs). `leadStyle` is laid over the lead's
+// own style (the homepage's instance cards tint it in the site's accent).
export function Wordmark({
title,
lead,
className,
+ leadStyle,
+ style,
}: {
title: string;
lead?: string;
className?: string;
+ leadStyle?: React.CSSProperties;
+ // Laid over the outer span's own (the export header reserves its width).
+ style?: React.CSSProperties;
}) {
const parts = splitWordmark(title, lead);
return (
@@ -31,11 +37,12 @@ export function Wordmark({
style={{
fontFamily: "var(--font-display)",
fontStretch: "118%",
+ ...style,
}}
>
<span
data-wordmark-lead=""
- style={{ fontWeight: 720, color: "var(--foreground)" }}
+ style={{ fontWeight: 720, color: "var(--foreground)", ...leadStyle }}
>
{parts.lead}
</span>
diff --git a/common/components/charts/ChartView.tsx b/common/components/charts/ChartView.tsx
@@ -26,6 +26,7 @@ import {
import { useMediaQuery } from "../../lib/useMediaQuery";
import type { ChartData } from "../../lib/chartAggregate";
import { xAxisLabel, yAxisLabel, type ChartConfig } from "../../lib/chartConfig";
+import { BAR_GAP } from "./surfaceGap";
const AXIS_LABEL_STYLE = { fill: "var(--muted-foreground)", fontSize: 11 };
@@ -187,6 +188,7 @@ export function ChartView({
name={m.label}
fill={`var(--color-${m.id})`}
radius={2}
+ {...(config.type === "stackedBar" && meta.length > 1 ? BAR_GAP : {})}
stackId={config.type === "stackedBar" ? "a" : undefined}
/>
))}
diff --git a/common/components/charts/CrossSiteChart.tsx b/common/components/charts/CrossSiteChart.tsx
@@ -22,6 +22,7 @@ import { ChartMessage } from "./ChartMessage";
import type { HomepageSummary } from "../../lib/homepageSummary";
import { type ChartState } from "../../lib/homepageChart";
import { buildTimeSeries } from "../../lib/homepageChartData";
+import { BAR_GAP } from "./surfaceGap";
// The stacked renderer: area (cumulative/Growth) or bars, optionally 100%-share.
// Stacking shows per-series composition *and* the combined total at once — the
@@ -67,6 +68,9 @@ export function CrossSiteChart({
}
const isArea = state.chartType === "area";
+ // The marks spec's surface gap between touching bar segments
+ // (charts/surfaceGap.ts): only when there is more than one series to part.
+ const stacked = series.length > 1;
const isShare = state.valueMode === "share";
const stackOffset = isShare ? "expand" : undefined;
const showLegend = state.breakdown === "channel" && series.length > 1;
@@ -133,6 +137,7 @@ export function CrossSiteChart({
dataKey={s.id}
name={s.label}
fill={s.color}
+ {...(stacked ? BAR_GAP : {})}
stackId="a"
isAnimationActive={false}
/>
diff --git a/common/components/charts/surfaceGap.ts b/common/components/charts/surfaceGap.ts
@@ -0,0 +1,14 @@
+// THE SURFACE GAP (the marks spec) between touching BARS: the segments of a
+// stacked bar are parted by a 2 px gap in the colour behind the plot, never by
+// a line drawn around them. `--chart-gap` is that colour (tokens.css: the
+// chart surface, and the reader's Canvas in forced colours). Recharts draws in
+// CSS pixels, so a width here is a width on screen. A segment's stroke is
+// centred on its edge: 1 px inside each of two touching segments makes the
+// 2 px gap (the outer edges meet the surface, so nothing shows there).
+//
+// Stacked AREAS keep their series-coloured top edge (ruled after review,
+// 2026-09-28): a surface stroke along a band's top also runs along the stack's
+// upper edge and over any band under ~2 px, and on translucent fills it erased
+// small values and cut peaks. A gap there waits on opaque fills (a ruling), a
+// gap on inner boundaries only, and the growth chart's thin-band rule.
+export const BAR_GAP = { stroke: "var(--chart-gap)", strokeWidth: 2 } as const;
diff --git a/common/components/exportFilterStorage.test.ts b/common/components/exportFilterStorage.test.ts
@@ -0,0 +1,83 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import {
+ emptySnapshot,
+ emptyStoredState,
+ loadStoredState,
+ parseSnapshot,
+ saveStoredState,
+ snapshotsEqual,
+} from "./exportFilterStorage";
+
+// The "Search in" row's two new keys (release 16): `notr` (Transcripts
+// unticked) and `lc` (Live chat ticked), stored beside `nop` only when off the
+// default, in the working snapshot and in every profile.
+
+test("notr and lc survive a JSON round trip through the parser", () => {
+ const snap = { ...emptySnapshot(), notr: true, nop: true, lc: true };
+ const back = parseSnapshot(JSON.parse(JSON.stringify(snap)));
+ assert.ok(back);
+ assert.equal(back.notr, true);
+ assert.equal(back.nop, true);
+ assert.equal(back.lc, true);
+ assert.ok(snapshotsEqual(snap, back));
+});
+
+test("absent reads as the default, and the default is not written", () => {
+ const back = parseSnapshot({ channels: { included: [], excluded: [] } });
+ assert.ok(back);
+ assert.equal("notr" in back, false);
+ assert.equal("lc" in back, false);
+ // A profile saved before the row existed equals one that spells the
+ // defaults out.
+ assert.ok(snapshotsEqual(back, { ...emptySnapshot(), notr: false, lc: false }));
+});
+
+test("a value that is not a boolean is dropped, not coerced", () => {
+ const back = parseSnapshot({
+ channels: { included: [], excluded: [] },
+ notr: "yes",
+ lc: 1,
+ });
+ assert.ok(back);
+ assert.equal(back.notr, undefined);
+ assert.equal(back.lc, undefined);
+});
+
+test("snapshotsEqual tells each of the row's keys apart", () => {
+ const base = emptySnapshot();
+ assert.ok(!snapshotsEqual(base, { ...base, notr: true }));
+ assert.ok(!snapshotsEqual(base, { ...base, lc: true }));
+ assert.ok(!snapshotsEqual({ ...base, notr: true }, { ...base, lc: true }));
+});
+
+test("the working snapshot and a profile keep the row through storage", () => {
+ const store = new Map<string, string>();
+ const g = globalThis as { window?: unknown };
+ const before = g.window;
+ g.window = {
+ localStorage: {
+ getItem: (k: string) => store.get(k) ?? null,
+ setItem: (k: string, v: string) => void store.set(k, v),
+ removeItem: (k: string) => void store.delete(k),
+ },
+ };
+ try {
+ const state = emptyStoredState();
+ state.working = { ...emptySnapshot(), lc: true };
+ state.profiles = { chatOnly: { ...emptySnapshot(), notr: true, nop: true, lc: true } };
+ state.activeProfileName = "chatOnly";
+ saveStoredState(state);
+ const loaded = loadStoredState();
+ assert.ok(loaded);
+ assert.equal(loaded.working.lc, true);
+ assert.equal(loaded.working.notr, undefined);
+ assert.deepEqual(
+ [loaded.profiles.chatOnly.notr, loaded.profiles.chatOnly.nop, loaded.profiles.chatOnly.lc],
+ [true, true, true],
+ );
+ assert.equal(loaded.activeProfileName, "chatOnly");
+ } finally {
+ g.window = before;
+ }
+});
diff --git a/common/components/exportFilterStorage.ts b/common/components/exportFilterStorage.ts
@@ -26,9 +26,18 @@ const VERSION = 1;
export type FilterSnapshot = {
channels: { included: string[]; excluded: string[] };
nov?: boolean;
- // Exclude the social-post corpus — the third media kind beside videos and
- // livestreams (see SearchSessionContext.passesFilter).
+ // The "Search in" row (release 16): what a plain query — a "transcripts"
+ // leaf — reads. Each is stored only off its default, so a profile saved
+ // before the row existed reads transcripts and posts, as it always did.
+ // notr — Transcripts unticked (the transcript cues are not read)
+ // nop — Posts unticked (the social-post corpus; before the row this was
+ // the Posts box in the Type row, and it also emptied a leaf of
+ // scope "Posts" — since release 16 it does not: a leaf asked for by
+ // name reads what it names)
+ // lc — Live chat ticked (the live_chat track is read; off by default)
+ notr?: boolean;
nop?: boolean;
+ lc?: boolean;
nol?: boolean;
naa?: boolean;
nar?: boolean;
@@ -124,7 +133,8 @@ function isStringArray(v: unknown): v is string[] {
return Array.isArray(v) && v.every((x) => typeof x === "string");
}
-function parseSnapshot(raw: unknown): FilterSnapshot | null {
+// Exported for the round-trip test; the app reads through loadStoredState.
+export function parseSnapshot(raw: unknown): FilterSnapshot | null {
if (!raw || typeof raw !== "object") return null;
const r = raw as Record<string, unknown>;
const ch = r.channels as Record<string, unknown> | undefined;
@@ -140,6 +150,8 @@ function parseSnapshot(raw: unknown): FilterSnapshot | null {
// `nop` and `nu` were written by committedSnapshot but never read back here,
// so both were silently dropped on every reload.
if (typeof r.nop === "boolean") snap.nop = r.nop;
+ if (typeof r.notr === "boolean") snap.notr = r.notr;
+ if (typeof r.lc === "boolean") snap.lc = r.lc;
if (typeof r.nol === "boolean") snap.nol = r.nol;
if (typeof r.naa === "boolean") snap.naa = r.naa;
if (typeof r.nar === "boolean") snap.nar = r.nar;
@@ -257,6 +269,8 @@ export function snapshotsEqual(a: FilterSnapshot, b: FilterSnapshot): boolean {
}
if ((a.nov ?? false) !== (b.nov ?? false)) return false;
if ((a.nop ?? false) !== (b.nop ?? false)) return false;
+ if ((a.notr ?? false) !== (b.notr ?? false)) return false;
+ if ((a.lc ?? false) !== (b.lc ?? false)) return false;
if ((a.nol ?? false) !== (b.nol ?? false)) return false;
if ((a.naa ?? false) !== (b.naa ?? false)) return false;
if ((a.nar ?? false) !== (b.nar ?? false)) return false;
diff --git a/common/components/themeConfig.test.ts b/common/components/themeConfig.test.ts
@@ -1,25 +1,25 @@
// The pre-paint theme script, run for real: buildThemeScript's string executed
// in node:vm against a fake <html>, localStorage, matchMedia and theme-color
// metas, over the whole legacy-migration matrix. The expected values come from
-// the pure helpers (migrateLegacy, resolveBase), so the inline script and the
-// runtime provider cannot drift apart.
+// the pure helpers (migrateLegacy, storedBase, resolveBase), so the inline
+// script and the runtime provider cannot drift apart.
import { test } from "node:test";
import assert from "node:assert/strict";
import vm from "node:vm";
-import { ACCENT_IDS, BASE_GROUNDS, isAccentId } from "../lib/brand";
+import { ACCENT_IDS, BASE_GROUNDS } from "../lib/brand";
import {
ACCENT_KEY,
BASE_KEY,
LEGACY_MODE_KEY,
LEGACY_THEME_KEY,
+ RETIRED_BASE,
THEME_BASES,
buildThemeScript,
- isThemeBase,
- isThemeAccent,
migrateLegacy,
nextBase,
resolveBase,
+ storedBase,
type ThemeBase,
} from "./themeConfig";
@@ -28,6 +28,8 @@ type Env = {
prefersDark: boolean;
serverAccent?: string;
storageThrows?: boolean;
+ // Reads work, writes throw (a full or read-only storage).
+ writeThrows?: boolean;
matchMediaThrows?: boolean;
};
@@ -73,8 +75,8 @@ function run(defaultBase: ThemeBase, env: Env): Result {
? { getItem: fail, setItem: fail, removeItem: fail }
: {
getItem: (k: string) => (store.has(k) ? store.get(k)! : null),
- setItem: (k: string, v: string) => void store.set(k, String(v)),
- removeItem: (k: string) => void store.delete(k),
+ setItem: (k: string, v: string) => (env.writeThrows ? fail() : void store.set(k, String(v))),
+ removeItem: (k: string) => (env.writeThrows ? fail() : void store.delete(k)),
};
ctx.window = {
get localStorage() {
@@ -109,12 +111,13 @@ function run(defaultBase: ThemeBase, env: Env): Result {
const THEMES = [null, "base", "archive", "selenized", "swiss", "archilyzer", "terminal"];
const MODES = [null, "light", "dark", "system", "bogus"];
-const BASES = [null, "light", "sepia", "dark", "system", "bogus"];
+const BASES = [null, "light", RETIRED_BASE, "dark", "system", "bogus"];
const DEFAULTS: ThemeBase[] = ["system", "light", "dark"];
test("migrateLegacy follows the plan's table", () => {
assert.equal(migrateLegacy({ theme: null, mode: "light" }), "light");
- assert.equal(migrateLegacy({ theme: "archive", mode: "light" }), "sepia");
+ // The old "archive" paper theme was the retired third ground: light now.
+ assert.equal(migrateLegacy({ theme: "archive", mode: "light" }), "light");
assert.equal(migrateLegacy({ theme: "selenized", mode: "light" }), "light");
assert.equal(migrateLegacy({ theme: "archive", mode: "dark" }), "dark");
assert.equal(migrateLegacy({ theme: "selenized", mode: "dark" }), "dark");
@@ -140,18 +143,21 @@ test("the pre-paint script over the whole legacy × stored-base × default × OS
const label = JSON.stringify({ theme, mode, base, defaultBase, prefersDark });
// 1. Migration: only when no base is stored; legacy keys always go.
+ // 2. The retired base is rewritten to light, in storage too.
const migrated = base === null ? migrateLegacy({ theme, mode }) : null;
- const storedAfter = base ?? migrated;
+ const storedAfter = base === RETIRED_BASE ? "light" : (base ?? migrated);
assert.equal(r.store.get(BASE_KEY) ?? null, storedAfter, `${label}: stored base`);
assert.ok(!r.store.has(LEGACY_THEME_KEY), `${label}: legacy theme key kept`);
assert.ok(!r.store.has(LEGACY_MODE_KEY), `${label}: legacy mode key kept`);
- // 2–3. Validated base, resolved against the OS.
- const effective: ThemeBase = isThemeBase(storedAfter) ? storedAfter : defaultBase;
+ // 3–4. Validated base, resolved against the OS.
+ const effective: ThemeBase = storedBase(storedAfter) ?? defaultBase;
const resolved = resolveBase(effective, prefersDark);
assert.equal(r.attrs.get("data-base"), resolved, `${label}: data-base`);
assert.equal(r.dark, resolved === "dark", `${label}: .dark`);
+ assert.ok(resolved === "light" || resolved === "dark", `${label}: a third ground`);
+
// 5. Browser chrome follows the resolved ground.
assert.deepEqual(r.metas, [BASE_GROUNDS[resolved], BASE_GROUNDS[resolved]], `${label}: theme-color`);
@@ -163,16 +169,35 @@ test("the pre-paint script over the whole legacy × stored-base × default × OS
assert.equal(cases, THEMES.length * MODES.length * BASES.length * DEFAULTS.length * 2);
});
-test("data-accent comes only from a valid stored id; otherwise the server's stays", () => {
+test("a stored accent is ignored: the server's data-accent stays, and the key is left in place", () => {
for (const stored of [null, ...ACCENT_IDS, "custom", "#cc3366", "bogus", ""])
for (const serverAccent of [undefined, "brass", "custom"]) {
const r = run("system", { storage: { [ACCENT_KEY]: stored }, prefersDark: false, serverAccent });
- const expected = stored && isAccentId(stored) ? stored : serverAccent;
- assert.equal(r.attrs.get("data-accent"), expected, JSON.stringify({ stored, serverAccent }));
- // The script never writes the accent key.
- assert.equal(r.store.get(ACCENT_KEY) ?? null, stored);
+ const label = JSON.stringify({ stored, serverAccent });
+ assert.equal(r.attrs.get("data-accent"), serverAccent, label);
+ assert.ok(!r.log.includes("data-accent"), `${label}: the script wrote data-accent`);
+ assert.equal(r.store.get(ACCENT_KEY) ?? null, stored, `${label}: the stored accent moved`);
assert.equal(r.log.at(-1), "data-theme-ready");
}
+ assert.ok(!buildThemeScript().includes(ACCENT_KEY));
+});
+
+test("the retired base paints light even when storage will not take the rewrite", () => {
+ for (const defaultBase of DEFAULTS)
+ for (const prefersDark of [false, true]) {
+ const r = run(defaultBase, { storage: { [BASE_KEY]: RETIRED_BASE }, prefersDark, writeThrows: true });
+ assert.equal(r.attrs.get("data-base"), "light");
+ assert.equal(r.dark, false);
+ assert.equal(r.log.at(-1), "data-theme-ready");
+ }
+});
+
+test("storedBase: light, dark and system, the retired one as light, anything else null", () => {
+ assert.equal(storedBase("light"), "light");
+ assert.equal(storedBase("dark"), "dark");
+ assert.equal(storedBase("system"), "system");
+ assert.equal(storedBase(RETIRED_BASE), "light");
+ for (const v of [null, "", "bogus", "Light", 1]) assert.equal(storedBase(v), null);
});
test("storage that throws still paints the default base and sets the marker", () => {
@@ -199,21 +224,14 @@ test("an invalid defaultBase falls back to system", () => {
assert.equal(r.attrs.get("data-base"), "dark");
});
-test("the toggle cycles system → light → sepia → dark → system, in picker order", () => {
+test("the toggle cycles system → light → dark → system, in THEME_BASES order", () => {
const seen: ThemeBase[] = [];
let b: ThemeBase = "system";
- for (let i = 0; i < 4; i++) {
+ for (let i = 0; i < 3; i++) {
seen.push(b);
b = nextBase(b);
}
assert.equal(b, "system");
assert.deepEqual(seen, THEME_BASES.map((x) => x.id));
- assert.deepEqual(THEME_BASES.map((x) => x.label), ["System", "Light", "Sepia", "Dark"]);
-});
-
-test("isThemeAccent takes the seven ids and custom only", () => {
- for (const id of ACCENT_IDS) assert.ok(isThemeAccent(id));
- assert.ok(isThemeAccent("custom"));
- assert.ok(!isThemeAccent("#cc3366"));
- assert.ok(!isThemeAccent("Signal"));
+ assert.deepEqual(THEME_BASES.map((x) => x.label), ["System", "Light", "Dark"]);
});
diff --git a/common/components/themeConfig.ts b/common/components/themeConfig.ts
@@ -1,229 +1,4 @@
-// Shared, dependency-light theme constants, types and the pre-paint script's
-// source. Used by the pre-paint ThemeScript (server), the runtime ThemeProvider
-// (client), the pickers, the unit tests and the homepage e2e (REQUIRED_TOKENS).
-// Keys are namespaced under the existing `ytdlp-tb:*` localStorage convention.
-//
-// A READER THEME IS TWO INDEPENDENT CHOICES (plans/brand-and-themes.md):
-// • a BASE — a light, sepia or dark ground, or "system", which follows the
-// OS between light and dark. `html[data-base]` selects one of the three
-// token blocks in common/styles/tokens.css; `.dark` is on <html> iff the
-// resolved base is dark, so Tailwind's `dark:` utilities keep working and
-// sepia styles as a light ground.
-// • an ACCENT — one of the seven named accents in lib/brand.ts. Each site
-// defaults to its own (a server-rendered `html[data-accent]`); a reader's
-// pick is stored only while it differs from the site's.
-//
-// PURE: no React, no DOM at import time. lib/brand.ts is pure too.
-
-import {
- ACCENTS,
- ACCENT_IDS,
- BASE_GROUNDS,
- isAccentId,
- type AccentId,
-} from "../lib/brand";
-
-// The reader's base ("light" | "sepia" | "dark" | "system").
-export const BASE_KEY = "ytdlp-tb:base";
-// The reader's accent: an accent id, stored only while it differs from the
-// site's own. (Before the base × accent themes this key held a hex that
-// nothing in the UI wrote; a stored value that is not an accent id is ignored.)
-export const ACCENT_KEY = "ytdlp-tb:accent";
-// The retired theme-family and light/dark-mode keys. Read once by the
-// migration (migrateLegacy, and the same table inside the pre-paint script),
-// then deleted.
-export const LEGACY_THEME_KEY = "ytdlp-tb:theme";
-export const LEGACY_MODE_KEY = "ytdlp-tb:mode";
-
-// The three grounds a base resolves to, and the reader's choice (which adds
-// "system").
-export type ResolvedBase = "light" | "sepia" | "dark";
-export type ThemeBase = ResolvedBase | "system";
-
-// What `html[data-accent]` can carry: a named accent, or "custom" — a site
-// whose site.json accent is its own hex (the inline `--accent-custom-*` vars).
-// A reader can never STORE "custom": it only ever means "this site's colour".
-export type ThemeAccent = AccentId | "custom";
-
-// Picker order.
-export const THEME_BASES: ReadonlyArray<{ id: ThemeBase; label: string }> = [
- { id: "system", label: "System" },
- { id: "light", label: "Light" },
- { id: "sepia", label: "Sepia" },
- { id: "dark", label: "Dark" },
-];
-
-export function isThemeBase(v: unknown): v is ThemeBase {
- return v === "system" || v === "light" || v === "sepia" || v === "dark";
-}
-
-export function isThemeAccent(v: unknown): v is ThemeAccent {
- return v === "custom" || isAccentId(v);
-}
-
-// The accent choices a picker offers on a site whose own accent is
-// `siteAccent`: the seven named accents in brand order, each with its swatch
-// (`--swatch-<id>` is that accent's value on the base in force, so the dot
-// shows what the pick will paint) and a flag on the site's own. A custom-hex
-// site adds its colour first, as "Site colour".
-export type AccentOption = {
- id: ThemeAccent;
- label: string;
- swatch: string;
- isSiteDefault: boolean;
-};
-
-export function accentOptions(siteAccent: ThemeAccent): AccentOption[] {
- const named: AccentOption[] = ACCENT_IDS.map((id) => ({
- id,
- label: ACCENTS[id].name,
- swatch: `var(--swatch-${id})`,
- isSiteDefault: id === siteAccent,
- }));
- return siteAccent === "custom"
- ? [{ id: "custom", label: "Site colour", swatch: "var(--swatch-custom)", isSiteDefault: true }, ...named]
- : named;
-}
-
-// ThemeToggle's cycle: system → light → sepia → dark → system.
-export function nextBase(b: ThemeBase): ThemeBase {
- return b === "system" ? "light" : b === "light" ? "sepia" : b === "sepia" ? "dark" : "system";
-}
-
-// A choice resolved against the OS preference.
-export function resolveBase(b: ThemeBase, systemDark: boolean): ResolvedBase {
- return b === "system" ? (systemDark ? "dark" : "light") : b;
-}
-
-// Every colour token a base block in tokens.css must declare. The failure this
-// guards is silent: a base that omits a token inherits the light block's value
-// (`:root` always matches), so a half-declared palette looks "a bit off" rather
-// than broken, and only on the base nobody checked. The unit test
-// (themeTokens.test.ts) parses tokens.css against this list; the homepage e2e
-// reads every one off the computed style of each base.
-export const REQUIRED_TOKENS = [
- "--background",
- "--foreground",
- "--card",
- "--card-foreground",
- "--popover",
- "--popover-foreground",
- "--primary",
- "--primary-foreground",
- "--secondary",
- "--secondary-foreground",
- "--muted",
- "--muted-foreground",
- "--accent",
- "--accent-foreground",
- "--destructive",
- "--destructive-foreground",
- "--destructive-soft",
- "--border",
- "--border-strong",
- "--input",
- "--ring",
- "--surface",
- "--faint",
- "--panel",
- "--panel-2",
- "--success",
- "--success-foreground",
- "--success-soft",
- "--warning",
- "--warning-foreground",
- "--warning-soft",
- "--info",
- "--info-foreground",
- "--info-soft",
- "--brand",
- "--brand-strong",
- "--brand-soft",
- "--brand-ink",
- "--state-gone",
- "--state-gone-soft",
- "--chart-1",
- "--chart-2",
- "--chart-3",
- "--chart-4",
- "--chart-5",
- "--chart-6",
- "--chart-surface",
- "--chart-grid",
- "--chart-axis",
- "--chart-tooltip-bg",
-] as const;
-
-// The retired keys → a base, once. After it runs, both legacy keys are deleted
-// by the caller, whatever it returned.
-//
-// stored mode result
-// light light — or sepia when the old theme was "archive" (paper)
-// dark dark
-// system system
-// absent/other null: nothing is stored and the app's default base applies
-//
-// The old theme FAMILY is otherwise dropped: every family retired, and the
-// accent is the site's again.
-export function migrateLegacy({
- theme,
- mode,
-}: {
- theme: string | null;
- mode: string | null;
-}): ThemeBase | null {
- if (mode === "light") return theme === "archive" ? "sepia" : "light";
- if (mode === "dark") return "dark";
- if (mode === "system") return "system";
- return null;
-}
-
-// The pre-paint script, as the string ThemeScript.tsx inlines. It runs
-// synchronously before first paint, in this order:
-// 1. when no base is stored yet, migrate the legacy keys (migrateLegacy's
-// table); then delete both legacy keys;
-// 2. validate the stored base, falling back to `defaultBase`;
-// 3. set `data-base` to the RESOLVED ground and toggle `.dark`;
-// 4. set `data-accent` only from a valid stored accent id — otherwise the
-// server-rendered default (the site's own accent) stays;
-// 5. point every `meta[name=theme-color]` at the resolved ground;
-// 6. LAST: `data-theme-ready="1"`, the e2e no-flash marker, so it is present
-// only once every attribute above is set. e2e's reloads resolve on
-// navigation commit, possibly before this head script has run, and wait
-// for the marker instead of racing.
-// Storage may throw (privacy modes): each storage touch is guarded, so a
-// failure still paints the default base and still sets the marker.
-// themeConfig.test.ts runs this string in node:vm over the whole matrix.
-export function buildThemeScript({
- defaultBase = "system",
-}: { defaultBase?: ThemeBase } = {}): string {
- const fallback: ThemeBase = isThemeBase(defaultBase) ? defaultBase : "system";
- const q = JSON.stringify;
- return (
- "(function(){try{" +
- "var d=document.documentElement,b=null,a=null,s;" +
- "try{s=window.localStorage;" +
- `b=s.getItem(${q(BASE_KEY)});` +
- `var lt=s.getItem(${q(LEGACY_THEME_KEY)}),lm=s.getItem(${q(LEGACY_MODE_KEY)});` +
- "if(lt!==null||lm!==null){" +
- "if(b===null){" +
- "var m=lm==='light'?(lt==='archive'?'sepia':'light'):lm==='dark'?'dark':lm==='system'?'system':null;" +
- `if(m){s.setItem(${q(BASE_KEY)},m);b=m;}` +
- "}" +
- `s.removeItem(${q(LEGACY_THEME_KEY)});s.removeItem(${q(LEGACY_MODE_KEY)});` +
- "}" +
- `a=s.getItem(${q(ACCENT_KEY)});` +
- "}catch(e){}" +
- `if(b!=='light'&&b!=='sepia'&&b!=='dark'&&b!=='system')b=${q(fallback)};` +
- "var r=b;" +
- "if(b==='system'){r='light';try{if(window.matchMedia('(prefers-color-scheme: dark)').matches)r='dark';}catch(e){}}" +
- "d.setAttribute('data-base',r);" +
- "d.classList.toggle('dark',r==='dark');" +
- `if(a&&${q(ACCENT_IDS)}.indexOf(a)>=0)d.setAttribute('data-accent',a);` +
- `var g=${q(BASE_GROUNDS)}[r];` +
- "try{var ms=document.querySelectorAll('meta[name=\"theme-color\"]');for(var i=0;i<ms.length;i++)ms[i].setAttribute('content',g);}catch(e){}" +
- "d.setAttribute('data-theme-ready','1');" +
- "}catch(e){}})();"
- );
-}
-
+// The theme's constants, types and pre-paint script live in lib/themeConfig.ts
+// (the source publish reads the script too, and the publish layer may not
+// import components/). Re-exported here for the UI and its tests.
+export * from "../lib/themeConfig";
diff --git a/common/components/themeTokens.test.ts b/common/components/themeTokens.test.ts
@@ -69,16 +69,22 @@ function rule(selector: string): Rule {
const BASE_SELECTOR: Record<BaseGround, string> = {
light: ':root, html[data-base="light"]',
- sepia: 'html[data-base="sepia"]',
dark: 'html[data-base="dark"]',
};
-const ON: Record<BaseGround, "onLight" | "onSepia" | "onDark"> = {
+const ON: Record<BaseGround, "onLight" | "onDark"> = {
light: "onLight",
- sepia: "onSepia",
dark: "onDark",
};
+test("two bases: tokens.css has a block for light and dark and none for any other", () => {
+ const blocks = RULES.map((r) => r.selector.match(/html\[data-base="([^"]+)"\]/g) ?? [])
+ .flat()
+ .map((sel) => sel.slice('html[data-base="'.length, -2));
+ assert.deepEqual([...new Set(blocks)].sort(), ["dark", "light"]);
+ assert.deepEqual([...BASE_GROUND_IDS], ["light", "dark"]);
+});
+
test("every base declares every required token, color-scheme and every swatch", () => {
for (const base of BASE_GROUND_IDS) {
const r = rule(BASE_SELECTOR[base]);
@@ -92,7 +98,7 @@ test("every base declares every required token, color-scheme and every swatch",
}
});
-test("--base-light|sepia|dark are 1 on their own base and 0 on the others", () => {
+test("--base-light|dark are 1 on their own base and 0 on the other", () => {
// lib/siteColor.ts perBaseColor multiplies each base's channel by these, so
// exactly one may be 1 — two would add two colours' channels together.
for (const base of BASE_GROUND_IDS) {
@@ -142,7 +148,7 @@ test("each base defaults --brand to Signal, keeps --primary neutral and rings in
assert.equal(r.decls.get("--brand"), "var(--swatch-signal)");
assert.equal(r.decls.get("--ring"), "var(--brand)");
assert.doesNotMatch(r.decls.get("--primary") ?? "", /brand|swatch/);
- assert.equal(r.decls.get("--primary"), base === "dark" ? "#efe7d8" : base === "sepia" ? "#33281a" : "#202a31");
+ assert.equal(r.decls.get("--primary"), base === "dark" ? "#efe7d8" : "#202a31");
assert.equal(r.decls.get("--brand-ink"), base === "dark" ? BASE_GROUNDS.dark : "#ffffff");
assert.match(r.decls.get("--brand-strong") ?? "", base === "dark" ? /var\(--brand\).*white/ : /var\(--brand\).*black/);
assert.match(r.decls.get("--brand-soft") ?? "", base === "dark" ? /var\(--brand\) 16%/ : /var\(--brand\) 12%/);
@@ -271,5 +277,5 @@ test("status text reads at 4.5:1 on its own soft fill, on every base", () => {
}
}
}
- assert.equal(checks, 30); // 3 bases × 2 grounds × 5 pairs
+ assert.equal(checks, 20); // 2 bases × 2 grounds × 5 pairs
});
diff --git a/common/components/ui/alert.tsx b/common/components/ui/alert.tsx
@@ -5,7 +5,7 @@ import { cn } from "../../lib/utils"
// A token-driven Alert primitive replacing the family's ad-hoc role="alert" /
// role="status" notice divs. Each variant is a soft-filled, colored-border block
-// that recolors correctly on every base (light, sepia, dark) via the semantic
+// that recolors correctly on every base (light, dark) via the semantic
// tokens. Callers keep control of the a11y role (default "alert"; pass
// role="status" for non-urgent notices) so existing e2e querying by role stays
// green. An optional leading icon is supported via the grid layout below.
diff --git a/common/components/ui/badge.tsx b/common/components/ui/badge.tsx
@@ -21,7 +21,7 @@ const badgeVariants = cva(
// Soft-filled semantic tones — colorful but calm, legible on every
// base (token hues tuned as text-on-soft). The accent reads on its own
// tint only as --brand-strong: plain --brand on --brand-soft falls to
- // ~4:1 on the light and sepia grounds.
+ // ~4:1 on the light ground.
success:
"border-success/30 bg-success-soft text-success [a&]:hover:bg-success/15",
warning:
diff --git a/common/components/ui/sonner.tsx b/common/components/ui/sonner.tsx
@@ -13,8 +13,8 @@ import { useTheme } from "../ThemeProvider"
const Toaster = ({ ...props }: ToasterProps) => {
// Reuse the family theme controller rather than next-themes, so toast chrome
- // tracks the base in force: sepia is a light ground, and "system" is already
- // resolved (live) by the provider.
+ // tracks the base in force: "system" is already resolved (live) by the
+ // provider.
const { isDark } = useTheme()
return (
diff --git a/common/controller/buildIndex.test.ts b/common/controller/buildIndex.test.ts
@@ -0,0 +1,645 @@
+// Integration: the index build's HOLD, through the REAL buildIndex, over a temp
+// corpus.
+//
+// A channel's `data/` may be an absolute symlink to another drive (AGENTS.md,
+// "A channel's `data/` may live on another drive"). With that drive unmounted
+// the link dangles, and the index build used to read the channel as having no
+// videos: it removed every record the channel had, and the site built next
+// published the channel as gone. These cases pin the hold that replaced it:
+// the channel is not rescanned, and its records and shared pages are kept as
+// they are; a FULL rebuild with a channel held refuses unless
+// ARCHILYZER_INDEX_ALLOW_HELD is set; and all of it undoes itself when the
+// drive is back. The stats build's twin is buildStats.test.ts case (i).
+//
+// Run with: node_modules/.bin/tsx --test common/controller/buildIndex.test.ts
+
+import { after, test } from "node:test";
+import assert from "node:assert/strict";
+import { spawnSync } from "node:child_process";
+import { createRequire, syncBuiltinESMExports } from "node:module";
+import {
+ chmodSync,
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ readdirSync,
+ readFileSync,
+ renameSync,
+ rmSync,
+ statSync,
+ symlinkSync,
+ writeFileSync,
+} from "node:fs";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+// EVERY PATH getPaths() CAN RESOLVE TO A PLACE THIS FILE'S CODE MAY WRITE IS
+// PINNED UNDER ROOT before anything calls it (the buildStats.test.ts list). The
+// last case proves no write this file caused landed outside ROOT.
+const ROOT = mkdtempSync(path.join(tmpdir(), "build-index-"));
+const PINNED: Record<string, string> = {
+ TRANSCRIPTS_DIR: path.join(ROOT, "transcripts"),
+ SAVED_VIDEOS_DIR: path.join(ROOT, "saved-videos"),
+ SITES_DIR: path.join(ROOT, "transcripts", "sites"),
+ SETTINGS_FILE: path.join(ROOT, "settings.json"),
+ EXPORT_PUBLIC_DIR: path.join(ROOT, "public"),
+ EXPORT_INDEX_DIR: path.join(ROOT, ".export-index"),
+ EXPORT_BUILDS_DIR: path.join(ROOT, ".export-builds"),
+ EDITOR_CHANGELOG_FILE: path.join(ROOT, "editor-CHANGELOG.md"),
+ EXPORT_CHANGELOG_FILE: path.join(ROOT, "export-CHANGELOG.md"),
+ CHARTS_CONFIG_FILE: path.join(ROOT, "chart-templates.json"),
+ SEARCH_ALIASES_FILE: path.join(ROOT, "transcripts", "search-aliases.json"),
+ CURATED_TAGS_FILE: path.join(ROOT, "transcripts", "tags.json"),
+ ARCHILYZER_CONFIG_DIR: path.join(ROOT, "config"),
+ ARCHILYZER_SOURCE_SCRATCH: path.join(ROOT, "source-scratch"),
+};
+Object.assign(process.env, PINNED);
+delete process.env.ARCHILYZER_INDEX_ALLOW_HELD;
+after(() => rmSync(ROOT, { recursive: true, force: true }));
+
+const { getPaths } = await import("../lib/paths");
+const { buildIndex, INDEX_ALLOW_HELD_ENV } = await import("./buildIndex");
+const { writeGlobalTags } = await import("../lib/curatedTagsStore");
+const { META_PAGES_PENDING } = await import("./curatedTagsIndex");
+const { open } = await import("lmdb");
+
+const paths = getPaths();
+const CHANNEL = "test-channel";
+const DRIVE_CHANNEL = "drive-channel";
+const SITE = "testsite";
+const MEDIA = path.join(ROOT, "media");
+const AWAY = `${MEDIA}-away`;
+const COMMON = fileURLToPath(new URL("..", import.meta.url));
+
+// ── a write spy over the whole file ─────────────────────────────────────────
+// The buildStats.test.ts spy, writes only: node:fs and node:fs/promises, async,
+// sync and callback, synced into the named ESM imports the code under test
+// holds. The last case reads it.
+const writes: string[] = [];
+let afterStat: ((p: string) => void) | null = null;
+{
+ const req = createRequire(import.meta.url);
+ const fsCjs = req("node:fs") as Record<string, unknown>;
+ const fspCjs = req("node:fs/promises") as Record<string, unknown>;
+ const WRITES = ["writeFile", "appendFile", "rename", "mkdir", "rm", "rmdir", "unlink", "copyFile", "cp", "symlink", "link", "utimes", "truncate", "mkdtemp", "chmod"];
+ const TWO_PATHS = new Set(["rename", "copyFile", "cp", "symlink", "link"]);
+ const opensForWrite = (flags: unknown) =>
+ (typeof flags === "string" && /[wa+]/.test(flags)) ||
+ (typeof flags === "number" && (flags & 3) !== 0);
+ const asPath = (v: unknown) =>
+ typeof v === "string" ? v : v instanceof URL ? fileURLToPath(v) : Buffer.isBuffer(v) ? v.toString() : null;
+ const wrap = (mod: Record<string, unknown>, name: string, mode: "write" | "open") => {
+ const fn = mod[name];
+ if (typeof fn !== "function") return;
+ mod[name] = function (this: unknown, ...args: unknown[]) {
+ if (mode === "write" || opensForWrite(args[1])) {
+ const ps = TWO_PATHS.has(name.replace(/Sync$/, "")) ? [args[0], args[1]] : [args[0]];
+ for (const a of ps) {
+ const p = asPath(a);
+ if (p !== null) writes.push(path.resolve(p));
+ }
+ }
+ return (fn as (...a: unknown[]) => unknown).apply(this, args);
+ };
+ };
+ for (const n of WRITES) {
+ wrap(fspCjs, n, "write");
+ wrap(fsCjs, n, "write");
+ wrap(fsCjs, `${n}Sync`, "write");
+ }
+ wrap(fspCjs, "open", "open");
+ wrap(fsCjs, "open", "open");
+ wrap(fsCjs, "openSync", "open");
+ wrap(fsCjs, "createWriteStream", "write");
+ // Case (i)'s hook: the drive is lost DURING the scan's walk. node:fs/promises
+ // `stat` calls `afterStat` with each path it has just answered for.
+ const stat = fspCjs.stat as (...a: unknown[]) => Promise<unknown>;
+ fspCjs.stat = async function (this: unknown, ...args: unknown[]) {
+ const result = await stat.apply(this, args);
+ const p = asPath(args[0]);
+ if (p !== null) afterStat?.(path.resolve(p));
+ return result;
+ };
+ syncBuiltinESMExports();
+}
+
+// ── the corpus ──────────────────────────────────────────────────────────────
+const writeJson = (file: string, value: unknown) => {
+ mkdirSync(path.dirname(file), { recursive: true });
+ writeFileSync(file, JSON.stringify(value, null, 2));
+};
+const videoDir = (id: string, channel = CHANNEL) =>
+ path.join(paths.channelsDir, channel, "data", id);
+
+// YouTube's rolling-caption shape: parseVtt keeps only lines carrying inline
+// timing tags, so a plain cue would parse to nothing.
+const VTT =
+ "WEBVTT\nKind: captions\nLanguage: en\n\n" +
+ "00:00:00.000 --> 00:00:05.000 align:start position:0%\n" +
+ "First<00:00:01.000><c> caption</c><00:00:02.000><c> line.</c>\n\n" +
+ "00:01:00.000 --> 00:01:50.000 align:start position:0%\n" +
+ "Second<00:01:10.000><c> caption</c><00:01:20.000><c> line.</c>\n";
+
+// Metadata and English captions; `subs` adds a German track, which the index
+// publishes in the shared subs tree.
+function seedVideo(
+ id: string,
+ channel = CHANNEL,
+ opts: { title?: string; subs?: boolean; dir?: string } = {},
+): void {
+ const dir = opts.dir ?? videoDir(id, channel);
+ writeJson(path.join(dir, "metadata.info.json"), {
+ id,
+ title: opts.title ?? `Video ${id}`,
+ channel: channel,
+ upload_date: "20260601",
+ duration: 120,
+ description: "fixture",
+ webpage_url: `https://www.youtube.com/watch?v=${id}`,
+ extractor_key: "Youtube",
+ });
+ writeFileSync(path.join(dir, "transcript.en.vtt"), VTT);
+ if (opts.subs) writeFileSync(path.join(dir, "transcript.de.vtt"), VTT);
+}
+
+function writeChannel(slug: string, extra: Record<string, unknown> = {}): void {
+ writeJson(path.join(paths.channelsDir, slug, "config.json"), {
+ handling: "youtube",
+ name: slug,
+ url: `https://www.youtube.com/@${slug}/videos`,
+ ...extra,
+ });
+}
+
+// A fresh corpus, LMDB and export tree per test, so every count is exact. The
+// drive is a storage location, as /storage records it: a held channel is named
+// by its label, never by a path.
+function resetCorpus(channels: string[] = [CHANNEL, DRIVE_CHANNEL]): void {
+ for (const p of [paths.transcriptsDir, PINNED.EXPORT_INDEX_DIR, MEDIA, AWAY]) {
+ rmSync(p, { recursive: true, force: true });
+ }
+ mkdirSync(paths.transcriptsDir, { recursive: true });
+ writeFileSync(
+ paths.settingsFile,
+ JSON.stringify({
+ storage: { locations: [{ id: "usb", label: "USB drive", root: MEDIA, autoRepoint: false }] },
+ }),
+ );
+ writeChannel(CHANNEL);
+ writeJson(path.join(paths.sitesDir, SITE, "site.json"), {
+ siteId: SITE,
+ siteTitle: "Test Site",
+ siteDescription: "fixture",
+ headerTitle: "Test Site",
+ homeTagline: "",
+ socialLinks: [],
+ groups: [{ id: "default", name: "All channels", selectedByDefault: true }],
+ defaultGroupId: "default",
+ channels: channels.map((slug) => ({ slug, groupId: "default" })),
+ });
+}
+
+// A channel whose data/ is a relocated symlink, the way the editor's Storage
+// panel leaves it: channels/<slug>/data -> <MEDIA>/<slug>/data, with
+// config.dataDir recording the target. d1 carries a subtitle track.
+function seedDriveChannel(titles: Record<string, string> = {}): void {
+ const target = path.join(MEDIA, DRIVE_CHANNEL, "data");
+ mkdirSync(target, { recursive: true });
+ writeChannel(DRIVE_CHANNEL, { dataDir: target });
+ symlinkSync(target, path.join(paths.channelsDir, DRIVE_CHANNEL, "data"));
+ seedVideo("d1", DRIVE_CHANNEL, { subs: true, title: titles.d1 });
+ seedVideo("d2", DRIVE_CHANNEL, { title: titles.d2 });
+}
+
+// Unmount: the link now dangles, exactly as an absent USB drive leaves it.
+const unmount = () => renameSync(MEDIA, AWAY);
+const remount = () => renameSync(AWAY, MEDIA);
+
+async function runIndex(log: string[] = []) {
+ const res = await buildIndex({ paths, onLog: (s) => log.push(s) });
+ return { res, log };
+}
+
+// ── reading what the build left ─────────────────────────────────────────────
+// The index LMDB, opened the way buildIndex opens it, and closed again.
+function withIndex<T>(fn: (db: (name: string) => ReturnType<ReturnType<typeof open>["openDB"]>) => T): T {
+ const root = open({ path: paths.lmdbPath, maxDbs: 18, compression: true });
+ try {
+ return fn((name) => root.openDB({ name, encoding: "msgpack" }));
+ } finally {
+ root.close();
+ }
+}
+// Every indexed video, as "<channel>/<dir>".
+const indexed = () =>
+ withIndex((db) =>
+ [...db("mtimes").getKeys()].map((k) => (k as unknown as string[]).join("/")).sort(),
+ );
+const summaryCount = () => withIndex((db) => [...db("sums").getKeys()].length);
+const storedSchema = () => withIndex((db) => db("meta").get("schema") as number);
+const setStoredSchema = (v: number) => withIndex((db) => db("meta").putSync("schema", v));
+const pagesPending = () => withIndex((db) => db("meta").get(META_PAGES_PENDING));
+
+// A directory tree as {relative path: contents}, or null when it is not there.
+function tree(dir: string): Record<string, string> | null {
+ if (!existsSync(dir)) return null;
+ const out: Record<string, string> = {};
+ const walk = (d: string) => {
+ for (const e of readdirSync(d, { withFileTypes: true })) {
+ const p = path.join(d, e.name);
+ if (e.isDirectory()) walk(p);
+ else out[path.relative(dir, p)] = readFileSync(p, "utf8");
+ }
+ };
+ walk(dir);
+ return out;
+}
+const sharedTranscripts = (slug = DRIVE_CHANNEL) =>
+ tree(path.join(paths.exportSharedTranscriptsDir, slug));
+const sharedSubs = (slug = DRIVE_CHANNEL) => tree(path.join(paths.exportSharedSubsDir, slug));
+
+type Published = { id: string; channelSlug?: string; state?: string; curatedTags?: string[] };
+const siteDir = () => path.join(paths.exportSitesIndexDir, SITE);
+const summaries = (): Published[] =>
+ JSON.parse(readFileSync(path.join(siteDir(), "summaries", "page-0000.json"), "utf8"));
+const siteManifest = () =>
+ JSON.parse(readFileSync(path.join(siteDir(), "summaries", "manifest.json"), "utf8")) as {
+ channels: { slug: string; count: number }[];
+ };
+const siteSubsManifest = () =>
+ JSON.parse(readFileSync(path.join(siteDir(), "subs", "manifest.json"), "utf8")) as {
+ channels: { slug: string; videoCount: number }[];
+ };
+const stateOf = (id: string) => {
+ const r = summaries().find((s) => s.id === id);
+ assert.ok(r, `${id} is published`);
+ return r.state ?? "available";
+};
+const transcriptRecord = (id: string, slug = DRIVE_CHANNEL): Published => {
+ const page = JSON.parse(
+ readFileSync(path.join(paths.exportSharedTranscriptsDir, slug, "page-0000.json"), "utf8"),
+ ) as Published[];
+ const r = page.find((s) => s.id === id);
+ assert.ok(r, `${id} is on its channel's transcript page`);
+ return r;
+};
+
+const DRIVE_VIDEOS = [`${DRIVE_CHANNEL}/d1`, `${DRIVE_CHANNEL}/d2`];
+
+// ── the cases ───────────────────────────────────────────────────────────────
+
+test("(a) an unmounted drive: the channel's records, transcript pages and subs survive an incremental build", async () => {
+ resetCorpus();
+ seedVideo("local");
+ seedDriveChannel();
+ const mounted = await runIndex();
+ assert.deepEqual(mounted.res.heldChannels, []);
+ assert.deepEqual(indexed(), [...DRIVE_VIDEOS, `${CHANNEL}/local`]);
+ const pagesBefore = sharedTranscripts();
+ const subsBefore = sharedSubs();
+ assert.ok(pagesBefore?.["manifest.json"] && pagesBefore["page-0000.json"], "the drive's transcript pages");
+ assert.ok(subsBefore?.["manifest.json"], "the drive's subs pages");
+
+ unmount();
+ // Something else changed too, so the build rewrites the shared trees and
+ // the site: the hold has to survive a real build, not a no-op one.
+ seedVideo("local2");
+ const { res, log } = await runIndex();
+
+ assert.deepEqual(res.heldChannels, [DRIVE_CHANNEL]);
+ assert.equal(res.added, 1);
+ assert.equal(res.removed, 0, "not read as a channel with no videos");
+ assert.deepEqual(indexed(), [...DRIVE_VIDEOS, `${CHANNEL}/local`, `${CHANNEL}/local2`]);
+ assert.equal(summaryCount(), 4);
+ // Byte-identical, manifest included: its generatedAt shows no rewrite.
+ assert.deepEqual(sharedTranscripts(), pagesBefore);
+ assert.deepEqual(sharedSubs(), subsBefore);
+ // The site built from this index still publishes the channel.
+ for (const id of ["d1", "d2", "local", "local2"]) {
+ assert.ok(summaries().some((s) => s.id === id), `${id} still published`);
+ }
+ assert.equal(siteManifest().channels.find((c) => c.slug === DRIVE_CHANNEL)?.count, 2);
+ assert.equal(siteSubsManifest().channels.find((c) => c.slug === DRIVE_CHANNEL)?.videoCount, 1);
+
+ // Said, with the location's label and no path.
+ const line = log.find((l) => l.startsWith(`Channel ${DRIVE_CHANNEL}:`));
+ assert.ok(line, log.join("\n"));
+ assert.match(
+ line,
+ /its media is not reachable \(drive not mounted\?\), on location "USB drive"; its 2 indexed video\(s\) are kept as they are, not rescanned/,
+ );
+ assert.ok(!line.includes(ROOT), line);
+ assert.ok(
+ log.some((l) => l.startsWith("Diff: +1 added, ~0 changed, -0 removed, 2 total. Held: 1 channel(s), 2 video(s) kept.")),
+ log.join("\n"),
+ );
+ assert.ok(
+ log.some((l) => l.startsWith("Done in") && l.endsWith(`Held, their media not readable: ${DRIVE_CHANNEL}.`)),
+ log.join("\n"),
+ );
+});
+
+test("(b) a held channel's availability states are carried over, not re-read from the missing drive", async () => {
+ resetCorpus();
+ seedVideo("local");
+ seedDriveChannel();
+ // Both drive videos fell out of the channel's listing. d2 was confirmed
+ // public after that scan (available); d1 never was (maybe missing).
+ writeJson(path.join(paths.channelsDir, DRIVE_CHANNEL, "maybe-missing.json"), {
+ checkedAt: "2026-08-01T12:00:00.000Z",
+ freshPlaylistCount: 0,
+ ids: ["d1", "d2"],
+ });
+ writeJson(path.join(videoDir("d2", DRIVE_CHANNEL), "availability.json"), {
+ checkedAt: "2026-08-02T00:00:00.000Z",
+ availability: "public",
+ });
+ await runIndex();
+ assert.deepEqual([stateOf("d1"), stateOf("d2")], ["maybe_missing", "available"]);
+
+ unmount();
+ seedVideo("local2"); // so the site is rebuilt
+ const { res } = await runIndex();
+ assert.deepEqual(res.heldChannels, [DRIVE_CHANNEL]);
+ // Re-read from the missing drive, d2's confirmation would be gone and both
+ // would say "maybe missing"; skipped without the carry, d1's would vanish.
+ assert.deepEqual([stateOf("d1"), stateOf("d2")], ["maybe_missing", "available"]);
+ const videoState = withIndex((db) =>
+ Object.fromEntries([...db("videoState").getRange()].map(({ key, value }) => [String(key).replace("\x00", "/"), value])),
+ );
+ assert.deepEqual(videoState, { [`${DRIVE_CHANNEL}/d1`]: "maybe_missing" });
+});
+
+test("(c) a full rebuild with a channel held refuses without the override, and holds with it", async () => {
+ resetCorpus();
+ seedVideo("local");
+ seedDriveChannel();
+ await runIndex();
+ const current = storedSchema();
+ const pagesBefore = sharedTranscripts();
+ const subsBefore = sharedSubs();
+
+ unmount();
+ setStoredSchema(current - 1);
+ await assert.rejects(runIndex(), (err: Error) => {
+ assert.match(
+ err.message,
+ new RegExp(
+ `must be rebuilt in full \\(index schema ${current - 1} -> ${current}\\), but 1 channel\\(s\\) cannot be read: ` +
+ `drive-channel \\(its media is not reachable \\(drive not mounted\\?\\), on location "USB drive"\\)`,
+ ),
+ );
+ // Why, the ways out (mounting first), and the override by name; no path.
+ assert.match(err.message, /the next site build would publish them as gone/);
+ assert.match(
+ err.message,
+ /For each: mount its media and run this again; or repair or re-point its location on \/storage; or finish or clear its move .*; or, if it is gone for good, delete the channel or set excludeFromBuild/,
+ );
+ // The override by name, and where it is set for each way a build runs.
+ assert.match(
+ err.message,
+ new RegExp(
+ `set ${INDEX_ALLOW_HELD_ENV}=1 in the environment of the process that runs the build: ` +
+ `for the command line, the command's own \\(\`${INDEX_ALLOW_HELD_ENV}=1 pnpm archilyzer index\`\\); ` +
+ `for the editor's Build index job, or a site build started from the editor, the editor's own environment, which takes a restart of the editor\\.$`,
+ ),
+ );
+ assert.ok(!err.message.includes(ROOT), err.message);
+ return true;
+ });
+ // Refused before the clear: nothing touched.
+ assert.equal(storedSchema(), current - 1);
+ assert.deepEqual(indexed(), [...DRIVE_VIDEOS, `${CHANNEL}/local`]);
+ assert.deepEqual(sharedTranscripts(), pagesBefore);
+
+ // The CLI (the export's build:index, a site build's data phase) exits
+ // non-zero on it, and leaves the index as it was.
+ const cli = spawnSync(
+ path.join(COMMON, "node_modules", ".bin", "tsx"),
+ ["bin/archilyzer.ts", "index"],
+ { cwd: COMMON, env: { ...process.env }, encoding: "utf8" },
+ );
+ assert.notEqual(cli.status, 0, cli.stdout + cli.stderr);
+ assert.match(cli.stderr, /must be rebuilt in full/);
+ assert.equal(storedSchema(), current - 1);
+ assert.deepEqual(indexed(), [...DRIVE_VIDEOS, `${CHANNEL}/local`]);
+
+ // Overridden: the rebuild runs, the held channel's records go with the
+ // clear (they cannot be re-read), and its pages are left as they are.
+ process.env[INDEX_ALLOW_HELD_ENV] = "1";
+ try {
+ const { res, log } = await runIndex();
+ assert.deepEqual(res.heldChannels, [DRIVE_CHANNEL]);
+ assert.equal(storedSchema(), current);
+ assert.deepEqual(indexed(), [`${CHANNEL}/local`]);
+ assert.deepEqual(sharedTranscripts(), pagesBefore);
+ assert.deepEqual(sharedSubs(), subsBefore);
+ assert.ok(
+ log.some(
+ (l) =>
+ l.startsWith(`Channel ${DRIVE_CHANNEL}: its media is not reachable`) &&
+ l.includes(`held under ${INDEX_ALLOW_HELD_ENV}: this full rebuild cleared its index records`),
+ ),
+ log.join("\n"),
+ );
+ } finally {
+ delete process.env[INDEX_ALLOW_HELD_ENV];
+ }
+
+ // The drive back: an ordinary build takes the channel in again.
+ remount();
+ const back = await runIndex();
+ assert.deepEqual(back.res.heldChannels, []);
+ assert.equal(back.res.added, 2);
+ assert.deepEqual(indexed(), [...DRIVE_VIDEOS, `${CHANNEL}/local`]);
+ assert.equal(siteManifest().channels.find((c) => c.slug === DRIVE_CHANNEL)?.count, 2);
+});
+
+test("(d) a first build, with no index yet, is a full rebuild: it refuses with a channel held too", async () => {
+ resetCorpus();
+ seedVideo("local");
+ seedDriveChannel();
+ unmount();
+ await assert.rejects(runIndex(), /index schema <none> -> \d+\), but 1 channel\(s\) cannot be read: drive-channel/);
+ remount();
+ const { res } = await runIndex();
+ assert.deepEqual(res.heldChannels, []);
+ assert.deepEqual(indexed(), [...DRIVE_VIDEOS, `${CHANNEL}/local`]);
+});
+
+test("(e) the drive back: the held set is empty and what arrived meanwhile is indexed", async () => {
+ resetCorpus();
+ seedVideo("local");
+ seedDriveChannel();
+ await runIndex();
+ unmount();
+ const away = await runIndex();
+ assert.deepEqual(away.res.heldChannels, [DRIVE_CHANNEL]);
+ assert.equal(away.res.removed, 0);
+
+ // Downloaded onto the drive while it was elsewhere.
+ seedVideo("d3", DRIVE_CHANNEL, { dir: path.join(AWAY, DRIVE_CHANNEL, "data", "d3") });
+ remount();
+ const { res, log } = await runIndex();
+ assert.deepEqual(res.heldChannels, []);
+ assert.equal(res.added, 1);
+ assert.equal(res.changed, 0, "the kept records match the disk");
+ assert.equal(res.removed, 0);
+ assert.deepEqual(indexed(), [...DRIVE_VIDEOS, `${DRIVE_CHANNEL}/d3`, `${CHANNEL}/local`]);
+ assert.deepEqual(Object.keys(JSON.parse(sharedTranscripts()!["manifest.json"]).slugToPage).sort(), ["d1", "d2", "d3"]);
+ assert.ok(!log.some((l) => l.includes("Held")), log.join("\n"));
+});
+
+test("(f) a channel that is really empty is still emptied, not held", async () => {
+ resetCorpus(["emptied", "gone", CHANNEL]);
+ for (const slug of ["emptied", "gone"]) {
+ writeChannel(slug);
+ seedVideo("x1", slug);
+ seedVideo("x2", slug);
+ }
+ seedVideo("local");
+ await runIndex();
+ assert.equal(indexed().length, 5);
+
+ // Its media deleted: an empty data/ in place, and no data/ at all. Neither
+ // was relocated, so there is no drive to be missing.
+ for (const id of ["x1", "x2"]) rmSync(videoDir(id, "emptied"), { recursive: true });
+ rmSync(path.join(paths.channelsDir, "gone", "data"), { recursive: true });
+ const { res, log } = await runIndex();
+ assert.deepEqual(res.heldChannels, []);
+ assert.equal(res.removed, 4);
+ assert.deepEqual(indexed(), [`${CHANNEL}/local`]);
+ assert.deepEqual(JSON.parse(sharedTranscripts("emptied")!["manifest.json"]).slugToPage, {});
+ // The missing directory is said, not swallowed.
+ assert.ok(
+ log.includes("Channel gone: no data/ directory; indexed as a channel with no videos."),
+ log.join("\n"),
+ );
+});
+
+test("(g) a data directory that cannot be read holds its channel, and says why", async () => {
+ if (process.getuid?.() === 0) return; // root reads through a mode of 000
+ resetCorpus(["locked", "flaky", CHANNEL]);
+ for (const slug of ["locked", "flaky"]) {
+ writeChannel(slug);
+ seedVideo("x1", slug);
+ seedVideo("x2", slug);
+ }
+ await runIndex();
+ assert.equal(indexed().length, 4);
+
+ // The whole data/ unreadable; and one video dir unreadable mid-walk.
+ const locked = path.join(paths.channelsDir, "locked", "data");
+ const flaky = videoDir("x2", "flaky");
+ chmodSync(locked, 0o000);
+ chmodSync(flaky, 0o000);
+ try {
+ const { res, log } = await runIndex();
+ assert.deepEqual([...res.heldChannels].sort(), ["flaky", "locked"]);
+ assert.equal(res.removed, 0);
+ assert.equal(indexed().length, 4);
+ assert.ok(
+ log.some((l) => l.startsWith("Channel locked: its data directory could not be read (EACCES)")),
+ log.join("\n"),
+ );
+ assert.ok(
+ // One video's directory, said apart from the whole data/ above.
+ log.some((l) => l.startsWith("Channel flaky: a video in its data directory could not be read (EACCES)")),
+ log.join("\n"),
+ );
+ } finally {
+ chmodSync(locked, 0o755);
+ chmodSync(flaky, 0o755);
+ }
+ const { res } = await runIndex();
+ assert.deepEqual(res.heldChannels, []);
+ assert.equal(indexed().length, 4);
+});
+
+test("(h) a curated-tag change while a channel is held reaches its pages when the drive is back", async () => {
+ resetCorpus();
+ seedVideo("local");
+ seedDriveChannel({ d1: "Stream with Elfpire Eva" });
+ await runIndex();
+ assert.equal("curatedTags" in transcriptRecord("d1"), false);
+
+ unmount();
+ // A rule edit moves no mtime. It re-derives d1 in the index (no disk read),
+ // but d1's page is not rewritten while its channel is held.
+ writeGlobalTags(paths, {
+ version: 1,
+ tags: [
+ {
+ id: "eva-collab",
+ label: "Collab",
+ group: "eva",
+ groupLabel: "Eva",
+ order: 1,
+ rules: [{ id: "meta", kind: "metadata" as const, pattern: "elfpire", enabled: true }],
+ },
+ ],
+ assignments: {},
+ });
+ const pagesBefore = sharedTranscripts();
+ const { log } = await runIndex();
+ assert.deepEqual(sharedTranscripts(), pagesBefore);
+ assert.equal(pagesPending(), true, "the page debt is kept");
+ assert.ok(log.some((l) => l.startsWith("curated tags: the pages of 1 held channel(s) are not rewritten while held")), log.join("\n"));
+
+ remount();
+ const back = await runIndex();
+ assert.equal(back.res.added + back.res.changed + back.res.removed, 0, "no mtime moved");
+ assert.deepEqual(transcriptRecord("d1").curatedTags, ["eva-collab"]);
+ assert.equal(pagesPending(), false);
+});
+
+test("(i) the drive lost MID-WALK: the second look holds the channel instead of dropping the rest of it", async () => {
+ resetCorpus();
+ seedVideo("local");
+ seedDriveChannel();
+ for (const id of ["d3", "d4"]) seedVideo(id, DRIVE_CHANNEL);
+ const drive = [...DRIVE_VIDEOS, `${DRIVE_CHANNEL}/d3`, `${DRIVE_CHANNEL}/d4`];
+ await runIndex();
+ assert.deepEqual(indexed(), [...drive, `${CHANNEL}/local`]);
+ const pagesBefore = sharedTranscripts();
+ const subsBefore = sharedSubs();
+
+ // The first look before the walk finds the drive, and readdir lists all four
+ // videos. Then the drive goes, right after the walk's first metadata stat:
+ // every later stat in the channel is ENOENT, which on its own reads as "no
+ // metadata yet" and would drop the rest of the channel as gone.
+ let lost = false;
+ afterStat = (p) => {
+ if (lost || !p.endsWith(`${path.sep}metadata.info.json`)) return;
+ if (!p.includes(`${path.sep}${DRIVE_CHANNEL}${path.sep}data${path.sep}`)) return;
+ lost = true;
+ unmount();
+ };
+ seedVideo("local2"); // so the build is not a no-op
+ const { res, log } = await runIndex().finally(() => {
+ afterStat = null;
+ });
+ assert.ok(lost, "the drive went away inside the walk");
+ assert.deepEqual(res.heldChannels, [DRIVE_CHANNEL]);
+ assert.equal(res.removed, 0, log.join("\n"));
+ assert.equal(res.added, 1);
+ assert.deepEqual(indexed(), [...drive, `${CHANNEL}/local`, `${CHANNEL}/local2`]);
+ assert.deepEqual(sharedTranscripts(), pagesBefore);
+ assert.deepEqual(sharedSubs(), subsBefore);
+ assert.ok(
+ log.some((l) =>
+ l.startsWith(`Channel ${DRIVE_CHANNEL}: its media is not reachable (drive not mounted?), on location "USB drive"; its 4 indexed video(s) are kept`),
+ ),
+ log.join("\n"),
+ );
+});
+
+test("(z) no write this file caused landed outside its temp root", () => {
+ // LMDB writes natively, past the spy: its file must be under the root too.
+ assert.ok(paths.lmdbPath.startsWith(ROOT + path.sep), paths.lmdbPath);
+ assert.ok(statSync(ROOT).isDirectory());
+ const outside = writes.filter((p) => p !== ROOT && !p.startsWith(ROOT + path.sep));
+ assert.deepEqual(outside, []);
+ assert.ok(writes.length > 0, "the spy saw the writes");
+});
diff --git a/common/controller/buildIndex.ts b/common/controller/buildIndex.ts
@@ -8,6 +8,22 @@
//
// Per-channel config.json selects the transcript parser ("youtube" → VTT,
// "transcribe" → whisper.cpp JSON). Short-circuits when mtimes already match.
+//
+// A CHANNEL WHOSE MEDIA IS NOT REACHABLE is HELD, not emptied: a relocated
+// `data/` on an unmounted drive, one mid-relocation, a link and a config that
+// disagree (inspectChannelMedia), or a data dir that fails to read. It is not
+// rescanned; its index records are kept as they are and its shared page trees
+// (transcripts, subs, digests) are left as they are, so the next site build
+// still publishes it. Until this hold the scan read such a channel as having
+// no videos, removed every record it had, and the site built next published
+// the channel as gone. The stats build has the same hold (buildStats.ts); the
+// words are shared (lib/channelMediaHold.ts).
+//
+// A FULL REBUILD (a schema change, or no index yet) with a channel held
+// REFUSES: it clears every channel's records, and a held channel cannot be
+// re-read, so it would come out empty. ARCHILYZER_INDEX_ALLOW_HELD=1 lets it
+// proceed; the held channel is then out of the index until its media is back
+// and the index is built again.
import path from "node:path";
import { createHash } from "node:crypto";
@@ -64,6 +80,7 @@ import {
pageFileName,
} from "../lib/manifest";
import { getSettings } from "../lib/settings";
+import { INDEX_SCANNED_AT_KEY } from "../lib/stats";
import {
listSites,
siteDigestsDir,
@@ -77,6 +94,13 @@ import {
type ChannelHandling,
} from "../lib/channelConfig";
import { readChannelConfigFile } from "./channels";
+import { inspectChannelMedia } from "../lib/channelMedia";
+import {
+ HELD_WAYS_OUT,
+ describeHeld,
+ heldReason,
+ isMediaHeld,
+} from "../lib/channelMediaHold";
import { resolveChannelGroupId } from "../lib/channelGroups";
import type { Paths } from "../lib/paths";
import {
@@ -274,21 +298,32 @@ async function exists(p: string): Promise<boolean> {
}
}
+function errCode(err: unknown): string {
+ const code = (err as NodeJS.ErrnoException | null)?.code;
+ return typeof code === "string" ? code : String(err);
+}
+
+// `held` maps each channel whose media could not be read to why, in words with
+// no path in them. A held channel contributes no live entries; the caller keeps
+// its records and pages (see the file header).
async function scanSource(
channelsDir: string,
log: (msg: string) => void,
): Promise<{
live: LiveEntry[];
channels: Map<string, ChannelConfig>;
+ held: Map<string, string>;
}> {
+ const locations = getSettings().storage.locations;
const channels = new Map<string, ChannelConfig>();
const live: LiveEntry[] = [];
+ const held = new Map<string, string>();
let channelEntries: Dirent[];
try {
channelEntries = await readdir(channelsDir, { withFileTypes: true });
} catch {
// Fresh transcripts dir with no channels yet.
- return { live, channels };
+ return { live, channels, held };
}
for (const ch of channelEntries) {
if (!ch.isDirectory()) continue;
@@ -307,13 +342,35 @@ async function scanSource(
// and routing it through the video scan would only ever produce noise. Its
// posts tree is built from the JSONL shards further down.
if (isSocialChannel(cfg)) continue;
+ // An unmounted drive is not an empty channel (lib/channelMedia.ts): the
+ // readdir below would fail, and every record the channel has would be
+ // removed as gone.
+ const media = await inspectChannelMedia({ channelsDir }, ch.name, cfg, {
+ fresh: true,
+ });
+ if (isMediaHeld(media.status)) {
+ held.set(ch.name, heldReason(media, cfg.dataDir, locations));
+ continue;
+ }
const dataDir = path.join(channelDir, "data");
let videoEntries: Dirent[];
try {
videoEntries = await readdir(dataDir, { withFileTypes: true });
- } catch {
+ } catch (err) {
+ const code = errCode(err);
+ // No data/ at all on a channel whose media was never moved: it has
+ // downloaded nothing yet (or its media was deleted), and it IS empty.
+ if (code === "ENOENT" && media.status === "in-place") {
+ log(`Channel ${ch.name}: no data/ directory; indexed as a channel with no videos.`);
+ continue;
+ }
+ // Anything else — a relocated drive gone between the check and the
+ // read, a permission or I/O error — is a channel that could not be read.
+ held.set(ch.name, `its data directory could not be read (${code})`);
continue;
}
+ const channelLive: LiveEntry[] = [];
+ let readFailure: string | null = null;
for (const v of videoEntries) {
if (!v.isDirectory()) continue;
const videoDir = v.name;
@@ -322,7 +379,16 @@ async function scanSource(
let metaMs: number;
try {
metaMs = (await stat(metaPath)).mtimeMs;
- } catch {
+ } catch (err) {
+ // ENOENT is a video dir with no metadata yet (a download in flight, a
+ // partial one): skipped, as always. Any other error is the channel's
+ // media failing mid-scan; the channel is held below rather than read
+ // as missing this video and every one after it.
+ const code = errCode(err);
+ if (code !== "ENOENT" && code !== "ENOTDIR") {
+ readFailure = code;
+ break;
+ }
continue;
}
// Hybrid: a single channel may contain both YouTube auto-subs (.vtt)
@@ -372,7 +438,7 @@ async function scanSource(
// Sidecar absent — the common case (102 of ~76,000 videos have one).
}
}
- live.push({
+ channelLive.push({
channelSlug: ch.name,
handling: cfg.handling,
configName: cfg.name,
@@ -388,8 +454,26 @@ async function scanSource(
digestMs,
});
}
+ if (readFailure !== null) {
+ // One video, not the directory: said apart from the readdir failure
+ // above, so the log points at the right place. The whole channel is
+ // held all the same.
+ held.set(ch.name, `a video in its data directory could not be read (${readFailure})`);
+ continue;
+ }
+ // Asked again after the walk: a drive that went away DURING it leaves the
+ // videos after that point missing from this scan, which would remove them.
+ // Three syscalls a channel.
+ const after = await inspectChannelMedia({ channelsDir }, ch.name, cfg, {
+ fresh: true,
+ });
+ if (isMediaHeld(after.status)) {
+ held.set(ch.name, heldReason(after, cfg.dataDir, locations));
+ continue;
+ }
+ for (const e of channelLive) live.push(e);
}
- return { live, channels };
+ return { live, channels, held };
}
function pathKeyId(k: PathKey): string {
@@ -417,6 +501,9 @@ export type BuildIndexResult = {
changed: number;
removed: number;
shortCircuited: boolean;
+ // Channels whose media could not be read, so they were not rescanned: their
+ // index records and shared pages were kept as they were (see the header).
+ heldChannels: string[];
};
export type BuildIndexOptions = {
@@ -424,6 +511,17 @@ export type BuildIndexOptions = {
onLog?: (msg: string) => void;
};
+// Set to 1 (or true/yes/on) to let a FULL rebuild proceed with a channel held.
+// Declared in lib/envVars.ts.
+export const INDEX_ALLOW_HELD_ENV = "ARCHILYZER_INDEX_ALLOW_HELD";
+const TRUTHY = new Set(["1", "true", "yes", "on"]);
+function allowsHeldFullRebuild(
+ env: Record<string, string | undefined> = process.env,
+): boolean {
+ const raw = env.ARCHILYZER_INDEX_ALLOW_HELD;
+ return typeof raw === "string" && TRUTHY.has(raw.trim().toLowerCase());
+}
+
export async function buildIndex({
paths,
onLog,
@@ -534,6 +632,39 @@ export async function buildIndex({
const storedSchema = meta.get("schema") as number | undefined;
const schemaBumped = storedSchema !== SCHEMA_VERSION;
+
+ // Recorded as INDEX_SCANNED_AT_KEY only when this build completes, so
+ // buildStats can tell apart a video with no `mtimes` record: metadata newer
+ // than this is "not indexed yet"; older, and this build saw it and skipped it
+ // (no upload_date, or processing failed) or its channel was held.
+ //
+ // The scan reads only the source tree, so it runs BEFORE a schema clear: a
+ // full rebuild must know which channels it cannot read before it drops them.
+ const scanStartedAt = Date.now();
+ const {
+ live,
+ channels: channelConfigs,
+ held,
+ } = await scanSource(channelsDir, log);
+
+ // A full rebuild clears every channel's records, and a held channel cannot be
+ // re-read: it would come out of this build empty, and the site built next
+ // would publish it as gone. Refuse, unless told to go on without it.
+ const heldThroughClear = schemaBumped && held.size > 0;
+ if (heldThroughClear && !allowsHeldFullRebuild()) {
+ await root.close();
+ throw new Error(
+ `The index must be rebuilt in full (index schema ${storedSchema ?? "<none>"} -> ${SCHEMA_VERSION}), ` +
+ `but ${held.size} channel(s) cannot be read: ${describeHeld(held)}. ` +
+ `A full rebuild clears every channel's index records, so these would come out empty and the next site build would publish them as gone. ` +
+ `${HELD_WAYS_OUT} ` +
+ `To rebuild without them anyway (each stays out of the index until its media is back and the index is built again), set ${INDEX_ALLOW_HELD_ENV}=1 ` +
+ `in the environment of the process that runs the build: for the command line, the command's own ` +
+ `(\`${INDEX_ALLOW_HELD_ENV}=1 pnpm archilyzer index\`); for the editor's Build index job, or a site build started from the editor, ` +
+ `the editor's own environment, which takes a restart of the editor.`,
+ );
+ }
+
if (schemaBumped) {
log(
`Schema change (${storedSchema ?? "<none>"} -> ${SCHEMA_VERSION}); invalidating LMDB cache.`,
@@ -556,7 +687,6 @@ export async function buildIndex({
await meta.put("schema", SCHEMA_VERSION);
}
- const { live, channels: channelConfigs } = await scanSource(channelsDir, log);
const livePathIds = new Set<string>();
const liveByPathId = new Map<string, LiveEntry>();
for (const s of live) {
@@ -584,12 +714,29 @@ export async function buildIndex({
changed.push(s);
}
}
+ // A held channel has no live entries, and its records are not "gone": they
+ // are kept, and counted for the log.
+ const keptHeld = new Map<string, number>();
for (const { key, value } of mtimes.getRange()) {
const k = key as PathKey;
+ if (held.has(k[0])) {
+ keptHeld.set(k[0], (keptHeld.get(k[0]) ?? 0) + 1);
+ continue;
+ }
if (!livePathIds.has(pathKeyId(k))) {
removed.push({ pathKey: k, indexKey: (value as MtimeRecord).indexKey });
}
}
+ let keptHeldTotal = 0;
+ for (const [slug, why] of held) {
+ const kept = keptHeld.get(slug) ?? 0;
+ keptHeldTotal += kept;
+ log(
+ heldThroughClear
+ ? `Channel ${slug}: ${why}; held under ${INDEX_ALLOW_HELD_ENV}: this full rebuild cleared its index records, so it is out of the index until its media is back and the index is built again. Its transcript, subtitle and digest pages are left as they are.`
+ : `Channel ${slug}: ${why}; its ${kept} indexed video(s) are kept as they are, not rescanned, and its transcript, subtitle and digest pages are left as they are.`,
+ );
+ }
const anyMutations =
added.length > 0 || changed.length > 0 || removed.length > 0;
@@ -602,6 +749,8 @@ export async function buildIndex({
// from LMDB further below so they reflect current site config.
const sharedManifestsPresent = async (): Promise<boolean> => {
for (const channelSlug of channelConfigs.keys()) {
+ // A held channel's pages are not written this build either way.
+ if (held.has(channelSlug)) continue;
const mPath = path.join(transcriptsOutDir, channelSlug, "manifest.json");
const raw = await readFile(mPath, "utf8").catch(() => null);
if (!raw) return false;
@@ -632,7 +781,10 @@ export async function buildIndex({
const curatedFresh = new Set<string>();
log(
- `Diff: +${added.length} added, ~${changed.length} changed, -${removed.length} removed, ${live.length} total.`,
+ `Diff: +${added.length} added, ~${changed.length} changed, -${removed.length} removed, ${live.length} total.` +
+ (held.size > 0
+ ? ` Held: ${held.size} channel(s), ${keptHeldTotal} video(s) kept.`
+ : ""),
);
for (const channelSlug of channelConfigs.keys()) {
@@ -1050,6 +1202,10 @@ export async function buildIndex({
if (sharedNeedsBuild) {
for (const channelSlug of Array.from(channelConfigs.keys()).sort()) {
+ // A held channel's pages are left exactly as the last build wrote them:
+ // not rewritten, not pruned, and (below) not removed. After a full rebuild
+ // its records are gone, and a rewrite would publish it empty.
+ if (held.has(channelSlug)) continue;
const channelDir = path.join(transcriptsOutDir, channelSlug);
await mkdir(channelDir, { recursive: true });
@@ -1166,6 +1322,10 @@ export async function buildIndex({
// availability.json for just those ids is cheap and exact.
let maybeMissingCount = 0;
for (const slug of channelConfigs.keys()) {
+ // A held channel's availability.json files are on the media that cannot be
+ // read, and a missing one reads as "maybe missing": its states are carried
+ // over from the last build instead (below).
+ if (held.has(slug)) continue;
const record = await loadMaybeMissing(paths, slug);
if (!record?.ids.length) continue;
const scannedAtMs = Date.parse(record.checkedAt);
@@ -1193,6 +1353,25 @@ export async function buildIndex({
);
}
+ // A held channel keeps the states the last build published for it (the
+ // confirmed ones above come from its kept records; this adds the overlay's),
+ // read back before the wholesale rewrite below. Nothing to carry after a full
+ // rebuild: the clear took them, with the records they described.
+ if (held.size > 0) {
+ for (const { key, value } of videoState.getRange()) {
+ const id = key as string;
+ const cut = id.indexOf("\x00");
+ if (cut < 0) continue;
+ const slug = id.slice(0, cut);
+ if (!held.has(slug)) continue;
+ const rec = mtimes.get([slug, id.slice(cut + 1)]);
+ if (!rec) continue;
+ const st = value as VideoState;
+ stateByIndexKey.set(indexKeyId(rec.indexKey), st);
+ stateByPath.set(id, st);
+ }
+ }
+
// Publish the sparse map for buildStats, which runs after us against the same
// LMDB file and would otherwise have to re-read ~76k availability.json files
// to build the status chart. Rewritten wholesale each build: the map is small
@@ -1213,6 +1392,16 @@ export async function buildIndex({
if (sharedNeedsBuild) {
for (const channelSlug of Array.from(channelConfigs.keys()).sort()) {
+ // Held: its subs dir is left as it is, and its stats carried over so the
+ // site manifests still list it (none to carry after a full rebuild).
+ if (held.has(channelSlug)) {
+ const prev = channelStatsDb.get(channelSlug);
+ if (prev) {
+ channelStats.set(channelSlug, prev);
+ subsTotalCount += prev.videoCount;
+ }
+ continue;
+ }
const cfg = channelConfigs.get(channelSlug)!;
const subsChannelDir = path.join(subsOutDir, channelSlug);
const tracksInChannel = new Set<string>();
@@ -1325,7 +1514,7 @@ export async function buildIndex({
const subsChannelSlugSet = new Set(channelStats.keys());
for (const e of topSubsEntries) {
if (e.isDirectory()) {
- if (!subsChannelSlugSet.has(e.name)) {
+ if (!subsChannelSlugSet.has(e.name) && !held.has(e.name)) {
await rm(path.join(subsOutDir, e.name), {
recursive: true,
force: true,
@@ -1359,7 +1548,17 @@ export async function buildIndex({
// curated-tag page debt is settled. Deliberately AFTER the page build and not
// beside the hashes: an interrupt anywhere above must leave the flag standing
// so the next build rewrites the shards.
- clearCuratedPagesPending(meta);
+ //
+ // Except for a held channel's pages, which were not written: the re-apply
+ // pass re-derives its records in LMDB like any other, and its pages owe them.
+ // The flag stays, so the first build with its media back rewrites them.
+ if (held.size > 0 && curatedReapply.pagesPending) {
+ log(
+ `curated tags: the pages of ${held.size} held channel(s) are not rewritten while held; the re-derived tags reach them on the first build with their media back.`,
+ );
+ } else {
+ clearCuratedPagesPending(meta);
+ }
await meta.flushed;
// ---------------------------------------------------------------------------
@@ -1572,6 +1771,16 @@ export async function buildIndex({
if (sharedNeedsBuild) {
for (const channelSlug of Array.from(channelConfigs.keys()).sort()) {
+ // Held: as with subs, its digest dir is left as it is and its stats
+ // carried over.
+ if (held.has(channelSlug)) {
+ const prev = channelDigestStatsDb.get(channelSlug);
+ if (prev) {
+ channelDigestStats.set(channelSlug, prev);
+ digestTotalCount += prev.digestCount;
+ }
+ continue;
+ }
const cfg = channelConfigs.get(channelSlug)!;
const digestChannelDir = path.join(digestsOutDir, channelSlug);
let digestCount = 0;
@@ -1671,7 +1880,7 @@ export async function buildIndex({
}).catch(() => [] as Dirent[]);
for (const e of topDigestEntries) {
if (e.isDirectory()) {
- if (!channelDigestStats.has(e.name)) {
+ if (!channelDigestStats.has(e.name) && !held.has(e.name)) {
await rm(path.join(digestsOutDir, e.name), {
recursive: true,
force: true,
@@ -1997,13 +2206,17 @@ export async function buildIndex({
}
}
for (const k of staleFpKeys) meta.remove(k);
+ await meta.put(INDEX_SCANNED_AT_KEY, scanStartedAt);
await meta.flushed;
await root.close();
const durationMs = Date.now() - t0;
log(
- `Done in ${(durationMs / 1000).toFixed(2)}s. ${sites.length} site(s): ${sitesBuilt} built, ${sitesSkipped} up to date; ${live.length} transcripts in pool.`,
+ `Done in ${(durationMs / 1000).toFixed(2)}s. ${sites.length} site(s): ${sitesBuilt} built, ${sitesSkipped} up to date; ${live.length} transcripts in pool.` +
+ (held.size > 0
+ ? ` Held, their media not readable: ${[...held.keys()].join(", ")}.`
+ : ""),
);
return {
totalCount: aggregateSummaries,
@@ -2017,5 +2230,6 @@ export async function buildIndex({
changed: changed.length,
removed: removed.length,
shortCircuited: !sharedNeedsBuild && sitesBuilt === 0,
+ heldChannels: [...held.keys()],
};
}
diff --git a/common/controller/buildStats.test.ts b/common/controller/buildStats.test.ts
@@ -0,0 +1,662 @@
+// Integration: the stats cache, through the REAL buildIndex and buildStats, over
+// a temp corpus.
+//
+// The stats cache (`statsByPath`) used to be keyed on metadata.info.json's
+// mtime alone, while hasTranscript / cueCount / coverage / transcribedDate come
+// from the index and the transcript files. So a transcript that arrived after a
+// video was first seen never reached its stat, and a caption video (a VTT, no
+// transcript.json, no outcome sidecar) could never be dated at all. Measured on
+// a real corpus: one site served 1,889 videos and the homepage said 0
+// transcripts, 0 channels, 0 hours. These cases pin the fix: the key also holds
+// the index's own record for the video, and a transcript always has a date.
+//
+// Run with: node_modules/.bin/tsx --test common/controller/buildStats.test.ts
+
+import { after, test } from "node:test";
+import assert from "node:assert/strict";
+import { spawnSync } from "node:child_process";
+import { createRequire, syncBuiltinESMExports } from "node:module";
+import {
+ mkdirSync,
+ mkdtempSync,
+ readFileSync,
+ renameSync,
+ rmSync,
+ symlinkSync,
+ utimesSync,
+ writeFileSync,
+} from "node:fs";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+// EVERY PATH getPaths() CAN RESOLVE TO A PLACE THIS FILE'S CODE MAY WRITE IS
+// PINNED UNDER ROOT, before anything calls it (it is lazy and cached — the
+// maybeMissingBuild.test.ts pattern). From common/lib/paths.ts: TRANSCRIPTS_DIR
+// (the LMDB, channels, jobs), SAVED_VIDEOS_DIR, SITES_DIR, SETTINGS_FILE,
+// EXPORT_PUBLIC_DIR, EXPORT_INDEX_DIR, EXPORT_BUILDS_DIR, EDITOR_CHANGELOG_FILE,
+// EXPORT_CHANGELOG_FILE, CHARTS_CONFIG_FILE, SEARCH_ALIASES_FILE,
+// CURATED_TAGS_FILE, ARCHILYZER_CONFIG_DIR, ARCHILYZER_SOURCE_SCRATCH. The rest
+// of its variables name binaries and a URL, which nothing here runs. The last
+// test proves no write this file caused landed outside ROOT.
+const ROOT = mkdtempSync(path.join(tmpdir(), "build-stats-"));
+const PINNED: Record<string, string> = {
+ TRANSCRIPTS_DIR: path.join(ROOT, "transcripts"),
+ SAVED_VIDEOS_DIR: path.join(ROOT, "saved-videos"),
+ SITES_DIR: path.join(ROOT, "transcripts", "sites"),
+ SETTINGS_FILE: path.join(ROOT, "settings.json"),
+ EXPORT_PUBLIC_DIR: path.join(ROOT, "public"),
+ EXPORT_INDEX_DIR: path.join(ROOT, ".export-index"),
+ EXPORT_BUILDS_DIR: path.join(ROOT, ".export-builds"),
+ EDITOR_CHANGELOG_FILE: path.join(ROOT, "editor-CHANGELOG.md"),
+ EXPORT_CHANGELOG_FILE: path.join(ROOT, "export-CHANGELOG.md"),
+ CHARTS_CONFIG_FILE: path.join(ROOT, "chart-templates.json"),
+ SEARCH_ALIASES_FILE: path.join(ROOT, "transcripts", "search-aliases.json"),
+ CURATED_TAGS_FILE: path.join(ROOT, "transcripts", "tags.json"),
+ ARCHILYZER_CONFIG_DIR: path.join(ROOT, "config"),
+ ARCHILYZER_SOURCE_SCRATCH: path.join(ROOT, "source-scratch"),
+};
+Object.assign(process.env, PINNED);
+delete process.env.ARCHILYZER_STATS_ALLOW_DOWNGRADE;
+after(() => rmSync(ROOT, { recursive: true, force: true }));
+
+const { getPaths } = await import("../lib/paths");
+const { buildIndex } = await import("./buildIndex");
+const { buildStats, STATS_DOWNGRADE_ENV } = await import("./buildStats");
+const { normalizeTranscript } = await import("./normalizeTranscript");
+const { readStatsPages } = await import("./poolSummary");
+const { siteStatsDir } = await import("../lib/site");
+const { STATS_SCHEMA_VERSION } = await import("../lib/stats");
+const { open } = await import("lmdb");
+
+const paths = getPaths();
+const CHANNEL = "test-channel";
+const DRIVE_CHANNEL = "drive-channel";
+const SITE = "testsite";
+const POOL = path.join(ROOT, "pool-stats");
+const COMMON = fileURLToPath(new URL("..", import.meta.url));
+
+// ── an fs spy over the whole file ───────────────────────────────────────────
+// Wraps the node:fs and node:fs/promises functions on their CJS exports objects
+// and syncs them into the named ESM imports the code under test holds. Records
+// every call's path(s), and whether it writes. Case (e) reads the reads; the
+// last case reads the writes.
+type FsCall = { fn: string; p: string; write: boolean };
+const fsCalls: FsCall[] = [];
+{
+ const req = createRequire(import.meta.url);
+ const fsCjs = req("node:fs") as Record<string, unknown>;
+ const fspCjs = req("node:fs/promises") as Record<string, unknown>;
+ const READS = ["readFile", "stat", "lstat", "readdir", "readlink"];
+ const WRITES = ["writeFile", "appendFile", "rename", "mkdir", "rm", "rmdir", "unlink", "copyFile", "cp", "symlink", "link", "utimes", "truncate", "mkdtemp"];
+ const TWO_PATHS = new Set(["rename", "copyFile", "cp", "symlink", "link"]);
+ const opensForWrite = (flags: unknown) =>
+ (typeof flags === "string" && /[wa+]/.test(flags)) ||
+ (typeof flags === "number" && (flags & 3) !== 0);
+ const asPath = (v: unknown) =>
+ typeof v === "string" ? v : v instanceof URL ? fileURLToPath(v) : Buffer.isBuffer(v) ? v.toString() : null;
+ const wrap = (mod: Record<string, unknown>, name: string, write: boolean | "open") => {
+ const fn = mod[name];
+ if (typeof fn !== "function") return;
+ mod[name] = function (this: unknown, ...args: unknown[]) {
+ const isWrite = write === "open" ? opensForWrite(args[1]) : write;
+ const paths = TWO_PATHS.has(name.replace(/Sync$/, "")) ? [args[0], args[1]] : [args[0]];
+ for (const a of paths) {
+ const p = asPath(a);
+ if (p !== null) fsCalls.push({ fn: name, p: path.resolve(p), write: isWrite });
+ }
+ return (fn as (...a: unknown[]) => unknown).apply(this, args);
+ };
+ };
+ for (const n of READS) wrap(fspCjs, n, false);
+ for (const n of WRITES) {
+ wrap(fspCjs, n, true);
+ wrap(fsCjs, n, true);
+ wrap(fsCjs, `${n}Sync`, true);
+ }
+ wrap(fspCjs, "open", "open");
+ wrap(fsCjs, "open", "open");
+ wrap(fsCjs, "openSync", "open");
+ wrap(fsCjs, "createWriteStream", true);
+ syncBuiltinESMExports();
+}
+
+const at = (iso: string) => new Date(iso);
+const writeJson = (file: string, value: unknown) => {
+ mkdirSync(path.dirname(file), { recursive: true });
+ writeFileSync(file, JSON.stringify(value, null, 2));
+};
+const videoDir = (id: string, channel = CHANNEL) =>
+ path.join(paths.channelsDir, channel, "data", id);
+const touch = (file: string, iso: string) => utimesSync(file, at(iso), at(iso));
+
+// A fresh corpus (and a fresh LMDB) per test: every count below is exact.
+function resetCorpus(): void {
+ for (const p of [paths.transcriptsDir, PINNED.EXPORT_INDEX_DIR, POOL, path.join(ROOT, "media")]) {
+ rmSync(p, { recursive: true, force: true });
+ }
+ mkdirSync(paths.transcriptsDir, { recursive: true });
+ writeFileSync(paths.settingsFile, JSON.stringify({}));
+ writeJson(path.join(paths.channelsDir, CHANNEL, "config.json"), {
+ handling: "youtube",
+ name: "Test Channel",
+ url: "https://www.youtube.com/@example/videos",
+ });
+ writeJson(path.join(paths.sitesDir, SITE, "site.json"), {
+ siteId: SITE,
+ siteTitle: "Test Site",
+ siteDescription: "fixture",
+ headerTitle: "Test Site",
+ homeTagline: "",
+ socialLinks: [],
+ groups: [{ id: "default", name: "All channels", selectedByDefault: true }],
+ defaultGroupId: "default",
+ channels: [{ slug: CHANNEL, groupId: "default" }],
+ });
+}
+
+// Metadata only — what a download leaves before any transcript exists. With
+// `metaIso` null the file keeps its real mtime (now): "downloaded just now".
+function seedVideo(
+ id: string,
+ metaIso: string | null = "2026-07-11T11:00:00Z",
+ extra: Record<string, unknown> = {},
+ channel = CHANNEL,
+): void {
+ const file = path.join(videoDir(id, channel), "metadata.info.json");
+ writeJson(file, {
+ id,
+ title: `Video ${id}`,
+ channel: "Test Channel",
+ upload_date: "20260601",
+ duration: 120,
+ description: "fixture",
+ webpage_url: `https://www.youtube.com/watch?v=${id}`,
+ extractor_key: "Youtube",
+ ...extra,
+ });
+ if (metaIso) touch(file, metaIso);
+}
+
+// YouTube's own captions, as a youtube-handled download writes them. parseVtt
+// keeps only lines carrying inline timing tags (YouTube's rolling-caption
+// shape), so the fixture has them.
+function addCaptions(id: string, iso: string, channel = CHANNEL): void {
+ const file = path.join(videoDir(id, channel), "transcript.en.vtt");
+ writeFileSync(
+ file,
+ "WEBVTT\nKind: captions\nLanguage: en\n\n" +
+ "00:00:00.000 --> 00:00:05.000 align:start position:0%\n" +
+ "First<00:00:01.000><c> caption</c><00:00:02.000><c> line.</c>\n\n" +
+ "00:01:00.000 --> 00:01:50.000 align:start position:0%\n" +
+ "Second<00:01:10.000><c> caption</c><00:01:20.000><c> line.</c>\n",
+ );
+ touch(file, iso);
+}
+
+// A Whisper-family run: transcript.json, and (unless `outcome` is false) the
+// transcribe-outcome.json sidecar every run writes beside it.
+function addWhisper(id: string, iso: string, outcome = true): void {
+ const file = path.join(videoDir(id), "transcript.json");
+ writeJson(file, {
+ duration_seconds: 120,
+ chunks: 2,
+ text: "one two",
+ chunk_data: [
+ { start_time: 0, end_time: 5, text: "Spoken line one." },
+ { start_time: 60, end_time: 110, text: "Spoken line two." },
+ ],
+ });
+ touch(file, iso);
+ if (outcome) {
+ writeJson(path.join(videoDir(id), "transcribe-outcome.json"), {
+ videoId: id,
+ transcribedAt: iso,
+ });
+ }
+}
+
+type Stat = Awaited<ReturnType<typeof readStatsPages>>[number];
+
+async function runStats(log: string[] = []) {
+ const res = await buildStats({
+ paths,
+ onLog: (s) => log.push(s),
+ wholePoolStatsDir: POOL,
+ });
+ const byId = new Map<string, Stat>(
+ (await readStatsPages(POOL)).map((s) => [s.id, s]),
+ );
+ return { res, byId, log };
+}
+
+const runIndex = () => buildIndex({ paths, onLog: () => {} });
+
+function statOf(byId: Map<string, Stat>, id: string): Stat {
+ const s = byId.get(id);
+ assert.ok(s, `${id} has a stat`);
+ return s;
+}
+
+test("(a) a transcript that arrives after the stat was cached reaches it on the next run", async () => {
+ resetCorpus();
+ seedVideo("late");
+ await runIndex();
+ const first = await runStats();
+ assert.equal(statOf(first.byId, "late").hasTranscript, false);
+
+ // Whisper runs days later. metadata.info.json — the old key — is untouched.
+ addWhisper("late", "2026-07-17T05:11:14Z");
+ await runIndex();
+ const second = await runStats();
+ assert.equal(second.res.changed, 1, "the index's record moved, so the stat is redone");
+ const s = statOf(second.byId, "late");
+ assert.equal(s.hasTranscript, true);
+ assert.equal(s.cueCount, 2);
+ assert.equal(s.transcribedDate, "20260717");
+ // The MCP's "covers only N% — truncated" note reads this field. A stale
+ // record said 0 for a complete transcript.
+ assert.ok(s.coverage != null && s.coverage > 0.9, `coverage ${s.coverage}`);
+});
+
+test("(b) stats built before the index had the video heal after the index build", async () => {
+ resetCorpus();
+ seedVideo("early");
+ addCaptions("early", "2026-07-11T12:00:00Z");
+ // The pool composers run buildStats against the index as it stands; this
+ // video was downloaded after the last index build.
+ const first = await runStats();
+ assert.equal(statOf(first.byId, "early").hasTranscript, false);
+
+ await runIndex();
+ const second = await runStats();
+ assert.equal(second.res.changed, 1, "the key moved off NOT_INDEXED, so the stat is redone");
+ const s = statOf(second.byId, "early");
+ assert.equal(s.hasTranscript, true);
+ assert.equal(s.transcribedDate, "20260711");
+
+ // And the lag is said, not silent: counted in the result and logged. (No
+ // index build has completed yet, so it is "not indexed yet".)
+ assert.equal(first.res.notIndexedYet, 1);
+ assert.equal(first.res.notIndexable, 0);
+ assert.ok(
+ first.log.some((l) => l.startsWith("1 video(s) were downloaded after the last index build")),
+ first.log.join("\n"),
+ );
+ assert.equal(second.res.notIndexedYet, 0);
+});
+
+test("(c) a caption-only video is dated by its captions' arrival, not by a later Normalize", async () => {
+ resetCorpus();
+ // Captions only: no transcript.json, no outcome sidecar, never normalized.
+ seedVideo("vtt-only");
+ addCaptions("vtt-only", "2026-07-11T12:00:00Z");
+ // Captions that arrived the same day, normalized a month later.
+ seedVideo("normalized");
+ addCaptions("normalized", "2026-07-11T12:00:00Z");
+ const norm = await normalizeTranscript({
+ videoDir: videoDir("normalized"),
+ channelSlug: CHANNEL,
+ });
+ assert.equal(norm.status, "wrote");
+ touch(path.join(videoDir("normalized"), "transcript.cues.json"), "2026-08-10T13:44:00Z");
+
+ await runIndex();
+ const { byId } = await runStats();
+ for (const id of ["vtt-only", "normalized"]) {
+ const s = statOf(byId, id);
+ assert.equal(s.hasTranscript, true, id);
+ assert.equal(s.transcribedDate, "20260711", id);
+ }
+});
+
+test("(d) a Whisper video resolves exactly as before: the outcome sidecar, else transcript.json", async () => {
+ resetCorpus();
+ // Transcribed before the stats first saw it, with and without the sidecar.
+ seedVideo("with-outcome");
+ addWhisper("with-outcome", "2026-06-01T10:00:00Z");
+ seedVideo("no-outcome");
+ addWhisper("no-outcome", "2026-06-02T10:00:00Z", false);
+ // The sidecar wins over every mtime, even a caption file's.
+ seedVideo("hybrid");
+ addCaptions("hybrid", "2026-05-01T10:00:00Z");
+ addWhisper("hybrid", "2026-06-04T10:00:00Z");
+ // Transcribed AFTER the stats first saw it.
+ seedVideo("after");
+ await runIndex();
+ await runStats();
+ addWhisper("after", "2026-06-03T10:00:00Z");
+ await runIndex();
+ const { byId } = await runStats();
+
+ const dates = Object.fromEntries(
+ ["with-outcome", "no-outcome", "hybrid", "after"].map((id) => [
+ id,
+ statOf(byId, id).transcribedDate,
+ ]),
+ );
+ assert.deepEqual(dates, {
+ "with-outcome": "20260601",
+ "no-outcome": "20260602",
+ hybrid: "20260604",
+ after: "20260603",
+ });
+});
+
+test("(e) the key does not churn: a heal redoes one stat, then an unchanged run reads nothing per video", async () => {
+ resetCorpus();
+ seedVideo("w");
+ addWhisper("w", "2026-06-01T10:00:00Z");
+ seedVideo("c");
+ addCaptions("c", "2026-06-01T10:00:00Z");
+ seedVideo("m");
+ await runIndex();
+ const first = await runStats();
+ assert.equal(first.res.added, 3);
+
+ addWhisper("m", "2026-06-05T10:00:00Z");
+ await runIndex();
+ const heal = await runStats();
+ assert.equal(heal.res.added, 0);
+ assert.equal(heal.res.changed, 1, "only the video whose transcript arrived");
+
+ // An index build over an unchanged corpus rewrites nothing the key reads.
+ await runIndex();
+ const dataDir = path.join(paths.channelsDir, CHANNEL, "data");
+ const from = fsCalls.length;
+ const steady = await runStats();
+ assert.equal(steady.res.added, 0);
+ assert.equal(steady.res.changed, 0);
+ assert.equal(steady.res.notIndexedYet + steady.res.notIndexable, 0);
+ // The spy sees node:fs and node:fs/promises, async and sync.
+ const perVideo = fsCalls.slice(from).filter((c) => c.p.startsWith(dataDir + path.sep));
+ assert.deepEqual(
+ perVideo.map((c) => `${c.fn} ${path.relative(dataDir, c.p)}`).sort(),
+ [
+ `stat ${path.join("c", "metadata.info.json")}`,
+ `stat ${path.join("m", "metadata.info.json")}`,
+ `stat ${path.join("w", "metadata.info.json")}`,
+ ],
+ "the unchanged path stats each metadata file and touches nothing else in a video dir",
+ );
+});
+
+test("(f) cues are read under the index's own key, even when the metadata's upload date moved", async () => {
+ resetCorpus();
+ seedVideo("moved", "2026-07-11T11:00:00Z");
+ addCaptions("moved", "2026-07-11T12:00:00Z");
+ const norm = await normalizeTranscript({ videoDir: videoDir("moved"), channelSlug: CHANNEL });
+ assert.equal(norm.status, "wrote");
+ touch(path.join(videoDir("moved"), "transcript.cues.json"), "2026-08-10T13:44:00Z");
+ // The metadata is rewritten with another upload date (a stream's date
+ // settled) but stays older than the normalized cues, so buildIndex still
+ // trusts transcript.cues.json — and keys the cues by ITS upload date.
+ seedVideo("moved", "2026-07-12T11:00:00Z", { upload_date: "20260602" });
+ await runIndex();
+ const { byId } = await runStats();
+ const s = statOf(byId, "moved");
+ assert.equal(s.uploadDate, "20260602");
+ assert.equal(s.hasTranscript, true, "the cues are found under the key the index used");
+ assert.equal(s.cueCount, 2);
+});
+
+test("(g) a re-index for another reason that changes the cues redoes the stat", async () => {
+ resetCorpus();
+ seedVideo("drift");
+ addCaptions("drift", "2026-07-11T12:00:00Z");
+ await runIndex();
+ const first = await runStats();
+ assert.equal(statOf(first.byId, "drift").cueCount, 2);
+
+ // A Normalize run writes transcript.cues.json (here with one cue fewer than
+ // the raw parse). buildIndex does not re-index for that alone...
+ await normalizeTranscript({ videoDir: videoDir("drift"), channelSlug: CHANNEL });
+ const cuesPath = path.join(videoDir("drift"), "transcript.cues.json");
+ const doc = JSON.parse(readFileSync(cuesPath, "utf8")) as { cues: unknown[] };
+ doc.cues = doc.cues.slice(0, 1);
+ writeFileSync(cuesPath, JSON.stringify(doc));
+ // ...but an availability recheck makes it re-process the video, and then it
+ // reads the fresher cues.json. The transcript mtime did not move.
+ writeJson(path.join(videoDir("drift"), "availability.json"), {
+ checkedAt: "2026-08-20T00:00:00.000Z",
+ availability: "public",
+ });
+ await runIndex();
+ const second = await runStats();
+ assert.equal(second.res.changed, 1);
+ assert.equal(statOf(second.byId, "drift").cueCount, 1);
+});
+
+test("(h) a video the index skipped is not announced as pending on every run", async () => {
+ resetCorpus();
+ seedVideo("fine");
+ // No upload_date: buildIndex skips it (it needs one for the index key).
+ seedVideo("undated", "2026-07-11T11:00:00Z", { upload_date: undefined });
+ await runIndex();
+ // Downloaded after that index build.
+ seedVideo("fresh", null);
+ for (let run = 0; run < 2; run++) {
+ const { res, log } = await runStats();
+ assert.equal(res.notIndexedYet, 1, `run ${run}`);
+ assert.equal(res.notIndexable, 1, `run ${run}`);
+ assert.ok(log.some((l) => l.startsWith("1 video(s) were downloaded after the last index build")), log.join("\n"));
+ assert.ok(
+ log.some(
+ (l) =>
+ l.startsWith("1 video(s) are not in the index although they are older than its last build") &&
+ l.includes("or its channel's media was unreachable during that build"),
+ ),
+ log.join("\n"),
+ );
+ }
+ await runIndex();
+ const { res } = await runStats();
+ assert.equal(res.notIndexedYet, 0, "the next index build takes the fresh one");
+ assert.equal(res.notIndexable, 1, "the undated one stays, and is said as such");
+});
+
+// A second channel whose data/ is a relocated symlink, the way the editor's
+// Storage panel leaves it: channels/<slug>/data -> <root>/<slug>/data, with
+// config.dataDir recording the target.
+function seedDriveChannel(): { target: string } {
+ const target = path.join(ROOT, "media", DRIVE_CHANNEL, "data");
+ mkdirSync(target, { recursive: true });
+ writeJson(path.join(paths.channelsDir, DRIVE_CHANNEL, "config.json"), {
+ handling: "youtube",
+ name: "Drive Channel",
+ url: "https://www.youtube.com/@drive/videos",
+ dataDir: target,
+ });
+ symlinkSync(target, path.join(paths.channelsDir, DRIVE_CHANNEL, "data"));
+ for (const id of ["d1", "d2"]) {
+ seedVideo(id, "2026-07-11T11:00:00Z", {}, DRIVE_CHANNEL);
+ addCaptions(id, "2026-07-11T12:00:00Z", DRIVE_CHANNEL);
+ }
+ return { target };
+}
+
+test("(i) an unmounted media drive keeps its channel's stats; a cache clear refuses", async () => {
+ resetCorpus();
+ seedVideo("local");
+ seedDriveChannel();
+ await runIndex();
+ const mounted = await runStats();
+ assert.equal(statOf(mounted.byId, "d1").hasTranscript, true);
+
+ // The drive is a storage location, as /storage records it; the refusal names
+ // it by its label, never by a path.
+ const media = path.join(ROOT, "media");
+ writeFileSync(
+ paths.settingsFile,
+ JSON.stringify({ storage: { locations: [{ id: "usb", label: "USB drive", root: media, autoRepoint: false }] } }),
+ );
+ // Unmount: the link now dangles, exactly as an absent USB drive leaves it.
+ renameSync(media, `${media}-away`);
+ const log: string[] = [];
+ const away = await runStats(log);
+ assert.equal(away.res.removed, 0, "not read as a channel with no videos");
+ for (const id of ["d1", "d2", "local"]) assert.ok(away.byId.has(id), `${id} still published`);
+ assert.deepEqual(away.res.heldChannels, [DRIVE_CHANNEL]);
+ assert.equal(statOf(away.byId, "d1").hasTranscript, true);
+ assert.ok(
+ log.some((l) => l.startsWith(`Channel ${DRIVE_CHANNEL}: its media is not reachable`) && l.includes("its 2 cached stat(s) are kept")),
+ log.join("\n"),
+ );
+
+ // A schema change needs the whole cache rebuilt, which cannot include a
+ // channel it cannot read: refuse, and leave the cache as it is.
+ setStoredSchema(STATS_SCHEMA_VERSION - 1);
+ await assert.rejects(runStats(), (err: Error) => {
+ assert.match(
+ err.message,
+ /must be rebuilt .* cannot be read: drive-channel \(its media is not reachable \(drive not mounted\?\), on location "USB drive"\)/,
+ );
+ // The ways out, mounting first, and no path in the message.
+ assert.match(
+ err.message,
+ /For each: mount its media and run this again; or repair or re-point its location on \/storage; or finish or clear its move .*; or, if it is gone for good, delete the channel or set excludeFromBuild/,
+ );
+ assert.ok(!err.message.includes(ROOT), err.message);
+ return true;
+ });
+ assert.equal(readStoredSchema(), STATS_SCHEMA_VERSION - 1);
+ assert.equal(countStats(), 3);
+
+ renameSync(`${media}-away`, media);
+ const back = await runStats();
+ assert.deepEqual(back.res.heldChannels, []);
+ assert.equal(readStoredSchema(), STATS_SCHEMA_VERSION);
+ assert.equal(back.byId.size, 3);
+});
+
+// Direct access to the temp LMDB's stats cache, for the schema cases.
+function withDb<T>(fn: (dbs: { meta: ReturnType<ReturnType<typeof open>["openDB"]>; stats: ReturnType<ReturnType<typeof open>["openDB"]> }) => T): T {
+ const root = open({ path: paths.lmdbPath, maxDbs: 12, compression: true });
+ try {
+ return fn({
+ meta: root.openDB({ name: "statsMeta", encoding: "msgpack" }),
+ stats: root.openDB({ name: "statsByPath", encoding: "msgpack" }),
+ });
+ } finally {
+ root.close();
+ }
+}
+const setStoredSchema = (v: number) =>
+ withDb(({ meta }) => meta.putSync("schema", v));
+const readStoredSchema = () => withDb(({ meta }) => meta.get("schema"));
+const countStats = () => withDb(({ stats }) => [...stats.getKeys()].length);
+
+test("(j) the schema guard: an older cache is cleared, a newer one is refused unless overridden", async () => {
+ resetCorpus();
+ seedVideo("v1");
+ addCaptions("v1", "2026-07-11T12:00:00Z");
+ await runIndex();
+ await runStats();
+ assert.equal(countStats(), 1);
+
+ // Older: cleared and rebuilt, as every schema bump has always done.
+ setStoredSchema(STATS_SCHEMA_VERSION - 1);
+ const log: string[] = [];
+ const older = await runStats(log);
+ assert.ok(log.some((l) => l.includes("clearing stats cache")), log.join("\n"));
+ assert.equal(older.res.added, 1);
+ assert.equal(readStoredSchema(), STATS_SCHEMA_VERSION);
+
+ // Newer: refused, naming both versions and the override; nothing touched.
+ setStoredSchema(STATS_SCHEMA_VERSION + 1);
+ await assert.rejects(
+ runStats(),
+ new RegExp(
+ `newer build \\(stats schema ${STATS_SCHEMA_VERSION + 1}; this build's is ${STATS_SCHEMA_VERSION}\\).*${STATS_DOWNGRADE_ENV}=1`,
+ ),
+ );
+ assert.equal(readStoredSchema(), STATS_SCHEMA_VERSION + 1);
+ assert.equal(countStats(), 1);
+
+ // The CLI exits non-zero on it.
+ const cli = spawnSync(
+ path.join(COMMON, "node_modules", ".bin", "tsx"),
+ ["bin/archilyzer.ts", "build", "stats"],
+ { cwd: COMMON, env: { ...process.env }, encoding: "utf8" },
+ );
+ assert.notEqual(cli.status, 0, cli.stdout + cli.stderr);
+ assert.match(cli.stderr, /written by a newer build/);
+ assert.equal(readStoredSchema(), STATS_SCHEMA_VERSION + 1);
+ assert.equal(countStats(), 1);
+
+ // A deliberate rollback, overridden: cleared and rebuilt at this version.
+ process.env[STATS_DOWNGRADE_ENV] = "1";
+ try {
+ const rolled = await runStats();
+ assert.equal(rolled.res.added, 1);
+ assert.equal(readStoredSchema(), STATS_SCHEMA_VERSION);
+ } finally {
+ delete process.env[STATS_DOWNGRADE_ENV];
+ }
+});
+
+// Release 14 slice HS: the whole-pool bundle is published as the homepage's
+// `stats/`, and an unlisted site's content is in no public total.
+test("(k) the whole-pool bundle leaves out a channel only an unlisted site exposes; the site's own bundle keeps it", async () => {
+ resetCorpus();
+ const seedChannel = (slug: string, ids: string[]) => {
+ writeJson(path.join(paths.channelsDir, slug, "config.json"), {
+ handling: "youtube",
+ name: slug,
+ url: `https://www.youtube.com/@${slug}/videos`,
+ });
+ for (const id of ids) {
+ seedVideo(id, "2026-07-11T11:00:00Z", {}, slug);
+ addCaptions(id, "2026-07-11T12:00:00Z", slug);
+ }
+ };
+ seedVideo("listed-1");
+ seedChannel("unlisted-channel", ["u1", "u2"]);
+ seedChannel("shared-channel", ["s1"]);
+ seedChannel("pool-channel", ["p1"]);
+ // The listed site also exposes the shared channel; the unlisted site exposes
+ // its own channel and the shared one. The pool channel is on no site.
+ const siteFile = path.join(paths.sitesDir, SITE, "site.json");
+ const listed = JSON.parse(readFileSync(siteFile, "utf8"));
+ listed.channels.push({ slug: "shared-channel", groupId: "default" });
+ writeJson(siteFile, listed);
+ writeJson(path.join(paths.sitesDir, "fixture-unlisted", "site.json"), {
+ ...listed,
+ siteId: "fixture-unlisted",
+ siteTitle: "Unlisted",
+ headerTitle: "Unlisted",
+ siteUrl: "https://unlisted.example",
+ listed: false,
+ channels: [
+ { slug: "unlisted-channel", groupId: "default" },
+ { slug: "shared-channel", groupId: "default" },
+ ],
+ });
+ await runIndex();
+ const log: string[] = [];
+ const { byId } = await runStats(log);
+ assert.deepEqual([...byId.keys()].sort(), ["listed-1", "p1", "s1"]);
+ const manifest = JSON.parse(readFileSync(path.join(POOL, "manifest.json"), "utf8"));
+ assert.equal(manifest.totalCount, 3);
+ assert.deepEqual(
+ manifest.channels.map((c: { slug: string }) => c.slug),
+ ["pool-channel", "shared-channel", CHANNEL],
+ );
+ assert.ok(
+ log.includes("Stats whole-pool: 3 videos, 1 page(s); 1 channel(s) only unlisted sites expose left out."),
+ log.join("\n"),
+ );
+ // The unlisted site still builds as before: its own bundle has its videos.
+ const own = await readStatsPages(siteStatsDir(paths, "fixture-unlisted"));
+ assert.deepEqual(own.map((s) => s.id).sort(), ["s1", "u1", "u2"]);
+});
+
+test("(z) no write this file caused landed outside its temp root", () => {
+ // LMDB writes natively, past the spy: its file must be under the root too.
+ assert.ok(paths.lmdbPath.startsWith(ROOT + path.sep), paths.lmdbPath);
+ const outside = fsCalls.filter(
+ (c) => c.write && c.p !== ROOT && !c.p.startsWith(ROOT + path.sep),
+ );
+ assert.deepEqual(outside, []);
+ assert.ok(fsCalls.some((c) => c.write), "the spy saw the writes");
+});
diff --git a/common/controller/buildStats.ts b/common/controller/buildStats.ts
@@ -2,13 +2,34 @@
// page-NNNN}.json for the viewer charts feature. Engagement metrics
// (view/like/comment counts, follower count, categories, language) live in each
// video's metadata.info.json but are NOT carried by the search index, so this
-// reads the raw metadata. Incremental: per-video mtime state is kept in a
-// dedicated `statsByPath` LMDB sub-DB so re-runs only re-parse changed videos.
+// reads the raw metadata. Incremental: per-video state is kept in a dedicated
+// `statsByPath` LMDB sub-DB so re-runs only re-parse changed videos.
//
// Transcript presence + cue count are read from the index LMDB `cues` sub-DB,
// and each video's visibility from the `videoState` sub-DB — both populated by
// buildIndex, so this must run after build:index, which the export prebuild
-// guarantees by chaining build:index && build:stats.
+// guarantees by chaining build:index && build:stats. The pool composers
+// (compose-hub, compose-homepage) do NOT chain it: they read the index as it
+// stands, which the cache key below makes safe.
+//
+// THE CACHE KEY IS TWO THINGS: the metadata file's mtime AND the index's own
+// per-video record (buildIndex's `mtimes` entry, as indexSignature, or
+// NOT_INDEXED). Until schema 6 it was the metadata mtime alone, while
+// hasTranscript / cueCount / coverage / transcribedDate come from the index and
+// the transcript files — so a transcript that arrived after a video was first
+// seen (Whisper days later, or a stats run before build:index had the video)
+// never reached its stat, and a whole channel could publish as untranscribed.
+//
+// A CHANNEL WHOSE MEDIA IS NOT REACHABLE (a relocated `data/` on an unmounted
+// drive, or one mid-relocation) is not rescanned: its cached stats are kept as
+// they are, rather than read as a channel with no videos and removed. This is
+// the stats build's own guard — the job registry's `needsMedia` check is per
+// channel, and this build is pool-wide — so the editor job and the CLI share it.
+//
+// ONE STATS BUILD AT A TIME. Nothing here takes a lock: two runs at once are
+// harmless unless one of them clears the cache (a schema change) after the
+// other scanned, when the other can publish truncated pages. The editor's build
+// jobs share the "build" queue by default; the CLI is outside every queue.
import path from "node:path";
import { createHash } from "node:crypto";
@@ -34,12 +55,30 @@ import {
import type { VideoState } from "../lib/availability";
import { loadDownloadOutcome } from "../lib/downloadOutcome-server";
import { loadTranscribeOutcome } from "../lib/transcribeOutcome-server";
+import {
+ CUES_JSON_FILENAME,
+ pickIndexTranscript,
+ readVideoFiles,
+} from "../lib/videoStatus";
import type { VideoStatus } from "../lib/stats";
import type { ChannelConfig } from "../lib/channelConfig";
import { readChannelConfigFile } from "./channels";
+import { inspectChannelMedia } from "../lib/channelMedia";
+import {
+ HELD_WAYS_OUT,
+ describeHeld,
+ heldReason,
+ isMediaHeld,
+} from "../lib/channelMediaHold";
+import { getSettings } from "../lib/settings";
import type { Paths } from "../lib/paths";
-import { listSites, siteStatsDir } from "../lib/site";
import {
+ channelsOnlyOnUnlistedSites,
+ listSites,
+ siteStatsDir,
+} from "../lib/site";
+import {
+ INDEX_SCANNED_AT_KEY,
STATS_SCHEMA_VERSION,
STATS_MANIFEST_VERSION,
STATS_MAX_PAGE_BYTES,
@@ -51,7 +90,56 @@ import {
type IndexKey = [string, string, string];
type PathKey = [string, string];
-type StatsRecord = { metaMs: number; stat: VideoStat };
+
+// `idx` is the index's per-video record as this stat saw it (indexSignature).
+// Optional only because a record written before schema 6 has none; the schema
+// bump clears those, and a missing value compares unequal to every real one, so
+// such a record would be recomputed anyway.
+type StatsRecord = { metaMs: number; idx?: string; stat: VideoStat };
+
+// The part of buildIndex's `mtimes` record this cache reads: every input whose
+// change makes buildIndex re-process the video (and so rewrite its cues), and
+// the key it stored the cues under.
+type IndexRecord = {
+ metaMs: number;
+ transcriptMs: number | null;
+ subsMs?: number | null;
+ availabilityMs?: number | null;
+ digestMs?: number | null;
+ indexKey: IndexKey;
+};
+
+// buildIndex has no `mtimes` record for this video. See the two counts below.
+const NOT_INDEXED = "-";
+
+// The whole index record as one comparable string. Keying on all of it (not
+// the transcript mtime alone) means ANY re-index redoes the stat: a re-index
+// for another reason can read a fresher transcript.cues.json and change the cue
+// count, and a moved index key moves the cues. Numbers print as their shortest
+// round-trip form, so an unchanged record gives an identical string.
+function indexSignature(r: IndexRecord | undefined): string {
+ if (!r) return NOT_INDEXED;
+ const n = (v: number | null | undefined) => (v == null ? "" : String(v));
+ return [
+ n(r.metaMs),
+ n(r.transcriptMs),
+ n(r.subsMs),
+ n(r.availabilityMs),
+ n(r.digestMs),
+ ...r.indexKey,
+ ].join("\u0000");
+}
+
+// Set to 1 (or true/yes/on) to let an OLDER build clear a stats cache a newer
+// one wrote — a deliberate rollback. Declared in lib/envVars.ts.
+export const STATS_DOWNGRADE_ENV = "ARCHILYZER_STATS_ALLOW_DOWNGRADE";
+const TRUTHY = new Set(["1", "true", "yes", "on"]);
+function allowsStatsDowngrade(
+ env: Record<string, string | undefined> = process.env,
+): boolean {
+ const raw = env.ARCHILYZER_STATS_ALLOW_DOWNGRADE;
+ return typeof raw === "string" && TRUTHY.has(raw.trim().toLowerCase());
+}
type ScanEntry = {
channelSlug: string;
@@ -68,6 +156,16 @@ export type BuildStatsResult = {
added: number;
changed: number;
removed: number;
+ // Videos on disk with no index record, in two kinds. `notIndexedYet`: their
+ // metadata is newer than the last completed index build's scan — downloaded
+ // since — and the first stats run after the next index build redoes them.
+ // `notIndexable`: older than the last index build, which did not index them
+ // (no upload_date, a failure, or their channel's media unreachable during
+ // that build); a stats run alone will not change that.
+ notIndexedYet: number;
+ notIndexable: number;
+ // Channels whose media was not reachable, so their cached stats were kept.
+ heldChannels: string[];
pagesWritten: number;
shortCircuited: boolean;
durationMs: number;
@@ -77,11 +175,12 @@ export type BuildStatsOptions = {
paths: Paths;
onLog?: (msg: string) => void;
signal?: AbortSignal;
- // When set, also write an UNFILTERED whole-pool stats bundle (every non-
- // excluded channel) into this dir as {manifest,page-NNNN}.json. Used by the
- // Archilyzer hub (compose-homepage), whose cross-site charts need the full
- // dataset rather than any one site's filtered slice. Forces collection of the
- // full dataset even when no per-site bundle needs a rebuild.
+ // When set, also write a whole-pool stats bundle (every non-excluded
+ // channel, but for those only unlisted sites expose) into this dir as
+ // {manifest,page-NNNN}.json. Used by the Archilyzer hub (compose-homepage),
+ // whose cross-site charts need the full dataset rather than any one site's
+ // filtered slice. Forces collection of the full dataset even when no per-site
+ // bundle needs a rebuild.
wholePoolStatsDir?: string;
};
@@ -116,6 +215,22 @@ async function fileMtimeMs(p: string): Promise<number | null> {
// "content added over time" progress charts. Prefers the explicit outcome
// sidecars (reliable across the shard rsync model, where file mtimes drift);
// falls back to file mtimes for content added before the sidecars existed.
+//
+// A TRANSCRIPT ALWAYS HAS A DATE (schema 6). The fallbacks, in order:
+// 1. transcribe-outcome.json's `transcribedAt` — every Whisper run writes it;
+// 2. the mtime of the transcript the index takes its cues from
+// (pickIndexTranscript: transcript.json, else the caption VTT);
+// 3. the mtime of transcript.cues.json;
+// 4. downloadedDate, which always resolves.
+// A Whisper video resolves exactly as before: 1, else transcript.json's mtime,
+// which is what (2) picks whenever transcript.json exists (the one difference:
+// a sidecar whose `transcribedAt` will not parse used to leave no date, and now
+// falls through to 2). A CAPTION-handled
+// video is dated by when its captions ARRIVED — the VTT's mtime — not by a later
+// Normalize run: (3) used to be the only file that could date one, so a
+// caption video either had no date (never normalized) or took the Normalize
+// run's date (1,683 of one channel's, all on one day). The homepage fold, the
+// charts and the recent rail all need `hasTranscript` ⇒ `transcribedDate`.
async function resolveAcquisitionDates(
videoDir: string,
metaMs: number,
@@ -128,29 +243,38 @@ async function resolveAcquisitionDates(
let transcribedDate: string | null = null;
if (hasTranscript) {
const tr = await loadTranscribeOutcome(videoDir);
- if (tr?.transcribedAt) {
- transcribedDate = ymdFromIso(tr.transcribedAt);
- } else {
+ transcribedDate = tr?.transcribedAt ? ymdFromIso(tr.transcribedAt) : null;
+ if (transcribedDate === null) {
+ const picked = pickIndexTranscript(await readVideoFiles(videoDir));
const mtime =
- (await fileMtimeMs(path.join(videoDir, "transcript.json"))) ??
- (await fileMtimeMs(path.join(videoDir, "transcript.cues.json")));
- transcribedDate = mtime != null ? ymdFromMs(mtime) : null;
+ (picked ? await fileMtimeMs(path.join(videoDir, picked.filename)) : null) ??
+ (await fileMtimeMs(path.join(videoDir, CUES_JSON_FILENAME)));
+ transcribedDate = (mtime != null ? ymdFromMs(mtime) : null) ?? downloadedDate;
}
}
return { downloadedDate, transcribedDate };
}
+// `held` maps each channel whose media is not reachable to why, in words with
+// no path in them: it is not scanned, and the caller keeps its cached stats
+// (see the file header).
async function scanSource(
channelsDir: string,
log: (msg: string) => void,
-): Promise<{ entries: ScanEntry[]; channels: Map<string, ChannelConfig> }> {
+): Promise<{
+ entries: ScanEntry[];
+ channels: Map<string, ChannelConfig>;
+ held: Map<string, string>;
+}> {
+ const locations = getSettings().storage.locations;
const channels = new Map<string, ChannelConfig>();
const entries: ScanEntry[] = [];
+ const held = new Map<string, string>();
let channelEntries: Dirent[];
try {
channelEntries = await readdir(channelsDir, { withFileTypes: true });
} catch {
- return { entries, channels };
+ return { entries, channels, held };
}
for (const ch of channelEntries) {
if (!ch.isDirectory()) continue;
@@ -162,6 +286,15 @@ async function scanSource(
}
if (cfg.excludeFromBuild) continue;
channels.set(ch.name, cfg);
+ // An unmounted drive is not an empty channel (lib/channelMedia.ts): the
+ // readdir below would fail and every one of its stats would be removed.
+ const media = await inspectChannelMedia({ channelsDir }, ch.name, cfg, {
+ fresh: true,
+ });
+ if (isMediaHeld(media.status)) {
+ held.set(ch.name, heldReason(media, cfg.dataDir, locations));
+ continue;
+ }
const dataDir = path.join(channelDir, "data");
let videoEntries: Dirent[];
try {
@@ -187,7 +320,7 @@ async function scanSource(
});
}
}
- return { entries, channels };
+ return { entries, channels, held };
}
// Buffered byte-capped page writer. Stats records are small, so a page fits in
@@ -266,10 +399,49 @@ export async function buildStats({
encoding: "msgpack",
});
const meta = root.openDB<unknown, string>({ name: "statsMeta", encoding: "msgpack" });
+ // Read-only views of buildIndex's per-video `mtimes` record (keyed like
+ // statsByPath: the second half of this cache's key, and the key the video's
+ // cues were stored under) and of its `meta` (when its last scan began). One
+ // LMDB get per video, and no file I/O, on the unchanged path.
+ const indexMtimes = root.openDB<IndexRecord, PathKey>({
+ name: "mtimes",
+ encoding: "msgpack",
+ });
+ const indexMeta = root.openDB<unknown, string>({ name: "meta", encoding: "msgpack" });
+ // NEVER CLEAR A CACHE A NEWER BUILD WROTE. An old build running beside a
+ // new one (an editor not yet restarted onto the new code) would otherwise
+ // clear it, refill it the old way, and the next new run would clear it back:
+ // a full pass each time, and old-logic numbers published in between.
const storedSchema = meta.get("schema") as number | undefined;
+ if (
+ typeof storedSchema === "number" &&
+ storedSchema > STATS_SCHEMA_VERSION &&
+ !allowsStatsDowngrade()
+ ) {
+ await root.close();
+ throw new Error(
+ `The stats cache was written by a newer build (stats schema ${storedSchema}; this build's is ${STATS_SCHEMA_VERSION}). ` +
+ `Refusing to clear it: rebuild and restart onto the current code. ` +
+ `For a deliberate rollback, set ${STATS_DOWNGRADE_ENV}=1.`,
+ );
+ }
+
+ const { entries, channels, held } = await scanSource(paths.channelsDir, log);
+ log(`Scanned ${entries.length} videos across ${channels.size} channels.`);
+
const schemaBumped = storedSchema !== STATS_SCHEMA_VERSION;
if (schemaBumped) {
+ // A clear with a channel's media unreachable would drop that channel's
+ // stats for good (it cannot be rescanned), and the pages built from this
+ // run would publish it as empty. Refuse instead.
+ if (held.size > 0) {
+ await root.close();
+ throw new Error(
+ `The stats cache must be rebuilt (stats schema ${storedSchema ?? "<none>"} -> ${STATS_SCHEMA_VERSION}), ` +
+ `but ${held.size} channel(s) cannot be read: ${describeHeld(held)}. ${HELD_WAYS_OUT}`,
+ );
+ }
log(
`Stats schema change (${storedSchema ?? "<none>"} -> ${STATS_SCHEMA_VERSION}); clearing stats cache.`,
);
@@ -277,30 +449,65 @@ export async function buildStats({
await meta.put("schema", STATS_SCHEMA_VERSION);
}
- const { entries, channels } = await scanSource(paths.channelsDir, log);
- log(`Scanned ${entries.length} videos across ${channels.size} channels.`);
-
const liveIds = new Set(
entries.map((e) => pathKeyId([e.channelSlug, e.videoDir])),
);
- const toProcess: ScanEntry[] = [];
+ const scannedAt = indexMeta.get(INDEX_SCANNED_AT_KEY) as number | undefined;
+ const toProcess: { e: ScanEntry; idx: string; indexKey?: IndexKey }[] = [];
let added = 0;
let changed = 0;
+ let notIndexedYet = 0;
+ let notIndexable = 0;
for (const e of entries) {
- const prev = statsByPath.get([e.channelSlug, e.videoDir]);
+ const pk: PathKey = [e.channelSlug, e.videoDir];
+ const rec = indexMtimes.get(pk);
+ const idx = indexSignature(rec);
+ if (idx === NOT_INDEXED) {
+ if (typeof scannedAt !== "number" || e.metaMs > scannedAt) notIndexedYet++;
+ else notIndexable++;
+ }
+ const prev = statsByPath.get(pk);
if (!prev) {
added++;
- toProcess.push(e);
- } else if (prev.metaMs !== e.metaMs) {
+ toProcess.push({ e, idx, indexKey: rec?.indexKey });
+ } else if (prev.metaMs !== e.metaMs || prev.idx !== idx) {
+ // The second half is the fix for stats frozen at first sight: a
+ // transcript that arrives later makes buildIndex re-process the video
+ // (and a video first seen before the index had it moves off
+ // NOT_INDEXED), while the metadata file — the old key's only input — is
+ // never touched.
changed++;
- toProcess.push(e);
+ toProcess.push({ e, idx, indexKey: rec?.indexKey });
}
}
+ // Neither is a failure of this build, and both are said, so a published
+ // number that lags the disk has its reason in the log. Only the first kind
+ // resolves itself.
+ if (notIndexedYet > 0) {
+ log(
+ `${notIndexedYet} video(s) were downloaded after the last index build; their transcripts reach the stats on the first run after the next one.`,
+ );
+ }
+ if (notIndexable > 0) {
+ log(
+ `${notIndexable} video(s) are not in the index although they are older than its last build: it skipped them (no upload_date, or it failed on them: see that build's log), or its channel's media was unreachable during that build (run an index build with every drive mounted). Their stats show no transcript until then.`,
+ );
+ }
const removedKeys: PathKey[] = [];
+ const keptHeld = new Map<string, number>();
for (const { key } of statsByPath.getRange()) {
const k = key as PathKey;
+ if (held.has(k[0])) {
+ keptHeld.set(k[0], (keptHeld.get(k[0]) ?? 0) + 1);
+ continue;
+ }
if (!liveIds.has(pathKeyId(k))) removedKeys.push(k);
}
+ for (const [slug, why] of held) {
+ log(
+ `Channel ${slug}: ${why}; its ${keptHeld.get(slug) ?? 0} cached stat(s) are kept as they are, not rescanned.`,
+ );
+ }
const removed = removedKeys.length;
const BATCH = 200;
@@ -308,7 +515,7 @@ export async function buildStats({
signal?.throwIfAborted();
const slice = toProcess.slice(i, i + BATCH);
await Promise.all(
- slice.map(async (e) => {
+ slice.map(async ({ e, idx, indexKey: recordedKey }) => {
try {
const metaRaw = await readFile(e.metaPath, "utf8");
const parsedMeta = JSON.parse(metaRaw) as RawMetadata;
@@ -319,16 +526,22 @@ export async function buildStats({
e.configName,
);
if (!base.uploadDate) return;
- const indexKey: IndexKey = [base.uploadDate, e.channelSlug, base.id];
+ // The key buildIndex stored this video's cues under, when it has a
+ // record: its uploadDate comes from transcript.cues.json when that
+ // is fresh, which the metadata's can differ from. The computed key
+ // is only a fallback for a video the index does not have.
+ const indexKey: IndexKey =
+ recordedKey ?? [base.uploadDate, e.channelSlug, base.id];
const cueList = cues.get(indexKey);
const cueCount = cueList ? cueList.length : null;
const coverage = transcriptCoverage(cueList, base.duration).coverage;
// Placeholder. `status` is NOT a cached field any more: it is applied
// from buildIndex's `videoState` sub-DB at collection time below.
- // This record is keyed on metadata mtime alone, so caching a status
- // here meant a pure availability flip only reached the chart on the
- // next metadata touch or schema bump — a video deleted today did not
- // show as deleted today.
+ // This record's key does not see availability (metadata mtime and
+ // the index's transcript mtime only), so caching a status here meant
+ // a pure availability flip only reached the chart on the next
+ // metadata touch or schema bump — a video deleted today did not show
+ // as deleted today.
const status: VideoStatus = "available";
const hasTranscript = cueCount != null && cueCount > 0;
const { downloadedDate, transcribedDate } =
@@ -353,6 +566,7 @@ export async function buildStats({
);
await statsByPath.put([e.channelSlug, e.videoDir], {
metaMs: e.metaMs,
+ idx,
stat,
});
} catch (err) {
@@ -500,16 +714,25 @@ export async function buildStats({
log(`Stats site ${plan.site.siteId}: ${filtered.length} videos, ${pageCount} page(s).`);
}
- // Whole-pool bundle for the hub: every non-excluded channel, unfiltered. Built
+ // Whole-pool bundle for the hub: every non-excluded channel, except a channel
+ // only unlisted sites expose (site.json `listed: false`): the bundle is
+ // published as the homepage's `stats/`, and an unlisted site's content is in
+ // no public total. Its own per-site bundle above is built as before. Built
// from the same in-memory dataset so it stays consistent with the per-site
// bundles. Always rewritten when requested (stats records are small).
if (wholePoolStatsDir) {
+ const unlistedOnly = channelsOnlyOnUnlistedSites(sites);
+ const pooled =
+ unlistedOnly.size > 0
+ ? all.filter((s) => !unlistedOnly.has(s.channelSlug))
+ : all;
const { pageCount } = await writePages(
wholePoolStatsDir,
- all,
+ pooled,
STATS_MAX_PAGE_BYTES,
);
const channelEntries: StatsChannelEntry[] = [...channels.keys()]
+ .filter((slug) => !unlistedOnly.has(slug))
.map((slug) => ({
slug,
name: channels.get(slug)?.name ?? slug,
@@ -519,13 +742,18 @@ export async function buildStats({
const manifest: StatsManifest = {
version: STATS_MANIFEST_VERSION,
generatedAt: new Date().toISOString(),
- totalCount: all.length,
+ totalCount: pooled.length,
pageCount,
maxPageBytes: STATS_MAX_PAGE_BYTES,
channels: channelEntries,
};
await writeJsonAtomic(path.join(wholePoolStatsDir, "manifest.json"), manifest);
- log(`Stats whole-pool: ${all.length} videos, ${pageCount} page(s).`);
+ log(
+ `Stats whole-pool: ${pooled.length} videos, ${pageCount} page(s)` +
+ (unlistedOnly.size > 0
+ ? `; ${unlistedOnly.size} channel(s) only unlisted sites expose left out.`
+ : "."),
+ );
}
// Prune fingerprints for sites that no longer exist (their staging dirs are
@@ -555,6 +783,9 @@ export async function buildStats({
added,
changed,
removed,
+ notIndexedYet,
+ notIndexable,
+ heldChannels: [...held.keys()],
pagesWritten: aggregatePages,
shortCircuited: needBuild.length === 0,
durationMs: Date.now() - t0,
diff --git a/common/controller/channelSnapshot.ts b/common/controller/channelSnapshot.ts
@@ -22,6 +22,7 @@ import {
} from "../lib/availability";
import { resolveCookiePolicy } from "../lib/cookiePolicy";
import { assertChannelMediaReachable } from "../lib/channelMedia";
+import { isDriveNotAnswering, onDrive } from "../lib/storageHealth";
import { CLIPS_DIR_NAME } from "../lib/clipWindow";
import { getSettings } from "../lib/settings";
import {
@@ -737,6 +738,20 @@ export async function generateChannelSnapshot(
const config = await readChannelConfig(paths, slug);
await assertChannelMediaReachable(paths, slug, config);
+ // A CHANNEL ON ANOTHER DRIVE IS WALKED THROUGH THE WATCHDOG
+ // (lib/storageHealth.ts `onDrive`): the data/ listing, the keep-latest keys and
+ // each video directory's unit below. At most four of them wait on that drive
+ // at once — this walk runs in the editor's own process after every download
+ // or sync of the channel, sixteen wide, which is exactly while a long write is
+ // stressing the drive — and one that does not answer within the budget
+ // (`storage.health.budgetMs`, 3 s by default) throws, so the
+ // scheduler keeps the last good snapshot.json, as on any failed refresh.
+ // The reconcile pass just below is sequential (one read at a time) and is
+ // not raced.
+ const drive = config?.dataDir?.trim() || undefined;
+ const through = <T>(read: () => Promise<T>): Promise<T> =>
+ drive ? onDrive(drive, read) : read();
+
// Heal any video dir that drifted from the canonical id layout before we read
// data/* (best-effort; never fail snapshot generation on a reconcile error).
try {
@@ -753,7 +768,11 @@ export async function generateChannelSnapshot(
maybeMissingRecord,
roster,
] = await Promise.all([
- readdir(dataDir, { withFileTypes: true }).catch(() => [] as Dirent[]),
+ through(() => readdir(dataDir, { withFileTypes: true })).catch((err) => {
+ // A drive that did not answer is not an empty channel: rethrown.
+ if (isDriveNotAnswering(err)) throw err;
+ return [] as Dirent[];
+ }),
readPlaylistUrls(playlistPath),
readArchive(archivePath),
loadFailedTranscriptions(paths, slug),
@@ -769,6 +788,7 @@ export async function generateChannelSnapshot(
paths,
channelSlug: slug,
keepLatest: config?.keepLatest ?? 0,
+ through,
});
const videoDirNames = dirEntries
.filter((d) => d.isDirectory())
@@ -821,7 +841,8 @@ export async function generateChannelSnapshot(
const limit = pLimit(SNAPSHOT_VIDEO_CONCURRENCY);
const perVideo = await Promise.all(
videoDirNames.map((id) =>
- limit(async () => {
+ // One video directory's reads are one unit through the watchdog.
+ limit(() => through(async () => {
const dir = path.join(dataDir, id);
const files = await readVideoFiles(dir, { checkUntranscribable: true });
// EVERY FILE IN THE DIR, STATTED ONCE, feeding two numbers.
@@ -952,7 +973,7 @@ export async function generateChannelSnapshot(
vttProvenance,
digest,
};
- }),
+ })),
),
);
diff --git a/common/controller/channels.ts b/common/controller/channels.ts
@@ -16,7 +16,8 @@ import {
readVideoFiles,
} from "../lib/videoStatus";
import { loadDigest } from "../lib/digest-server";
-import { readRelocationMarker } from "../lib/channelMedia";
+import { channelMediaStall, readRelocationMarker } from "../lib/channelMedia";
+import { isDriveNotAnswering, onDrive } from "../lib/storageHealth";
// TYPE-ONLY, and it must stay that way: ./channelSnapshot imports
// readChannelConfig from this module, and it drags in the snapshot generator's
// whole dependency graph (lmdb, the archive reader, the digest layer). A value
@@ -87,39 +88,57 @@ async function hasDigestWithItems(videoDir: string): Promise<boolean> {
);
}
-async function countDataFiles(dataDir: string): Promise<{
+// `drive` is the channel's configured target (`config.dataDir`) when its media
+// is on another drive. Then every read goes through `onDrive`: none while that
+// location is stalled, at most `inFlightPerLocation` (4) in flight on it, and
+// one that has not answered within the budget (`storage.health.budgetMs`, 3 s
+// by default) marks it stalled — and the walk answers null ("the drive did
+// not answer") instead of counts. The rest of the walk is refused without a
+// call. An in-place channel's walk is on the corpus disk and is not wrapped.
+async function countDataFiles(
+ dataDir: string,
+ drive?: string,
+): Promise<{
videos: number;
transcripts: number;
downloads: number;
digests: number;
-}> {
+} | null> {
+ const through = <T>(call: () => Promise<T>): Promise<T> =>
+ drive ? onDrive(drive, call) : call();
let dirs: Dirent[];
try {
- dirs = await readdir(dataDir, { withFileTypes: true });
- } catch {
+ dirs = await through(() => readdir(dataDir, { withFileTypes: true }));
+ } catch (err) {
+ if (isDriveNotAnswering(err)) return null;
return { videos: 0, transcripts: 0, downloads: 0, digests: 0 };
}
const videoDirs = dirs.filter((d) => d.isDirectory());
// Bounded: the largest channel has 11,224 video dirs and this used to open
// them all at once.
- const flags = await mapConcurrent(
- videoDirs,
- VIDEO_READ_CONCURRENCY,
- async (d) => {
- const dir = path.join(dataDir, d.name);
- const files = await readVideoFiles(dir);
- return {
- transcript: isVideoTranscribed(files),
- download: isVideoDownloaded(files),
- // Only transcribed videos can carry a digest, so the sidecar read is
- // skipped for the rest — the same conditional per-video sidecar-read
- // pattern channelSnapshot.ts uses for coverage and VTT provenance.
- digest: isVideoTranscribed(files)
- ? await hasDigestWithItems(dir)
- : false,
- };
- },
- );
+ let flags: Array<{ transcript: boolean; download: boolean; digest: boolean }>;
+ try {
+ flags = await mapConcurrent(videoDirs, VIDEO_READ_CONCURRENCY, (d) =>
+ // One video directory's few reads are one call through the watchdog.
+ through(async () => {
+ const dir = path.join(dataDir, d.name);
+ const files = await readVideoFiles(dir);
+ return {
+ transcript: isVideoTranscribed(files),
+ download: isVideoDownloaded(files),
+ // Only transcribed videos can carry a digest, so the sidecar read is
+ // skipped for the rest — the same conditional per-video sidecar-read
+ // pattern channelSnapshot.ts uses for coverage and VTT provenance.
+ digest: isVideoTranscribed(files)
+ ? await hasDigestWithItems(dir)
+ : false,
+ };
+ }),
+ );
+ } catch (err) {
+ if (isDriveNotAnswering(err)) return null;
+ throw err;
+ }
let transcripts = 0;
let downloads = 0;
let digests = 0;
@@ -189,8 +208,18 @@ export async function readChannelStat(
): Promise<ChannelStat | null> {
const config = await readChannelConfig(paths, slug);
if (!config) return null;
+ // A WALK OF `data/` ON A DRIVE THAT IS NOT ANSWERING IS NOT STARTED. The
+ // one-second job-list poll asks this for every channel with a job listed, and
+ // on a stalled drive each readdir and stat in the walk would hold an I/O
+ // thread until the drive came back. No counts is what a caller already
+ // handles (the row draws no progress bar).
+ if (channelMediaStall(config)) return null;
const channelDir = path.join(paths.channelsDir, slug);
- const counts = await countDataFiles(path.join(channelDir, "data"));
+ const counts = await countDataFiles(
+ path.join(channelDir, "data"),
+ config.dataDir?.trim() || undefined,
+ );
+ if (!counts) return null;
return {
slug,
config,
@@ -265,7 +294,14 @@ export async function listChannelStatsFromDisk(
if (!config) continue;
const channelDir = path.join(paths.channelsDir, slug);
const dataDir = path.join(channelDir, "data");
- const counts = await countDataFiles(dataDir);
+ // A batch job's ground truth: no drive passed, so nothing is raced and the
+ // answer is never null.
+ const counts = (await countDataFiles(dataDir)) ?? {
+ videos: 0,
+ transcripts: 0,
+ downloads: 0,
+ digests: 0,
+ };
out.push({
slug,
config,
diff --git a/common/controller/evictClipWindows.ts b/common/controller/evictClipWindows.ts
@@ -96,7 +96,9 @@ async function evictChannel(
// fine. (The JOB also declares `needsMedia: true`, which covers a
// single-channel run before it starts; this covers the corpus-wide one,
// where there is no slug for that guard to check.)
- const media = await inspectChannelMedia(opts.paths, slug);
+ const media = await inspectChannelMedia(opts.paths, slug, undefined, {
+ fresh: true,
+ });
if (media.status !== "ok" && media.status !== "in-place") {
out.skipped.push(
`${slug}: media ${media.status}${media.detail ? ` (${media.detail})` : ""} — nothing was touched`,
diff --git a/common/controller/keptVideos.ts b/common/controller/keptVideos.ts
@@ -2,6 +2,10 @@ import path from "node:path";
import { readdir } from "node:fs/promises";
import type { Paths } from "../lib/paths";
import { loadRawMetadataFromDir } from "../lib/transcripts-server";
+import { mapConcurrent } from "../lib/concurrency";
+
+// Metadata reads in flight while keying a channel's videos by upload date.
+const KEY_READ_CONCURRENCY = 16;
// Rolling "keep-latest" window computation. Given a channel's keepLatest config,
// returns the ids of the newest N videos (by upload date). Used by both the
@@ -13,6 +17,11 @@ export type ComputeKeptOptions = {
paths: Paths;
channelSlug: string;
keepLatest: number;
+ // Runs each video's metadata read. The snapshot passes `onDrive` for a
+ // channel on another drive (lib/storageHealth.ts), which caps the reads in
+ // flight on that drive and gives up on one that does not answer. Default:
+ // the read itself.
+ through?: <T>(read: () => Promise<T>) => Promise<T>;
};
// List a channel's data-dir video ids (directories, skipping dotfiles). Mirrors
@@ -58,16 +67,21 @@ async function uploadKey(videoDir: string, id: string): Promise<string> {
async function keyedVideosNewestFirst(
paths: Paths,
channelSlug: string,
+ through: <T>(read: () => Promise<T>) => Promise<T> = (read) => read(),
): Promise<Array<{ id: string; key: string }>> {
- const ids = await listChannelVideoIds(paths, channelSlug);
+ const ids = await through(() => listChannelVideoIds(paths, channelSlug));
if (ids.length === 0) return [];
const dataDir = path.join(paths.channelsDir, channelSlug, "data");
- const keyed = await Promise.all(
- ids.map(async (id) => ({
- id,
- key: await uploadKey(path.join(dataDir, id), id),
- })),
- );
+ // BOUNDED, like every other corpus-shaped fan-out (lib/concurrency.ts): one
+ // metadata read per video, and the largest channel has eleven thousand. With
+ // a `through` of `onDrive`, at most four of these are on the drive at once
+ // and the rest wait in its queue, which refuses a waiting read only when
+ // nothing on the drive has returned for the watchdog's budget — never for
+ // the queue's depth alone.
+ const keyed = await mapConcurrent(ids, KEY_READ_CONCURRENCY, async (id) => ({
+ id,
+ key: await through(() => uploadKey(path.join(dataDir, id), id)),
+ }));
keyed.sort((a, b) =>
a.key === b.key ? b.id.localeCompare(a.id) : b.key.localeCompare(a.key),
);
@@ -82,9 +96,10 @@ export async function computeKeptVideoIds({
paths,
channelSlug,
keepLatest,
+ through,
}: ComputeKeptOptions): Promise<Set<string>> {
if (!Number.isFinite(keepLatest) || keepLatest <= 0) return new Set();
- const keyed = await keyedVideosNewestFirst(paths, channelSlug);
+ const keyed = await keyedVideosNewestFirst(paths, channelSlug, through);
return new Set(keyed.slice(0, Math.floor(keepLatest)).map((k) => k.id));
}
diff --git a/common/controller/poolSummary.test.ts b/common/controller/poolSummary.test.ts
@@ -0,0 +1,28 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { parseSite } from "../lib/siteSchema";
+import { channelSitesOf } from "./poolSummary";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// channelSitesOf is the published `channel-sites.json` (compose-homepage) and
+// the map the summary attributes channels by. Release 14 slice HS: an unlisted
+// site (site.json `listed: false`) is in neither.
+
+test("channel-sites.json names listed sites only; a channel only an unlisted site exposes is absent", () => {
+ const sites = [
+ parseSite("fixture-a", { channels: [{ slug: "a1" }, { slug: "shared" }] }),
+ parseSite("fixture-b", { channels: [{ slug: "b1" }, { slug: "shared" }] }),
+ parseSite("fixture-unlisted", {
+ listed: false,
+ channels: [{ slug: "shared" }, { slug: "own" }],
+ }),
+ ];
+ assert.deepEqual(channelSitesOf(sites), {
+ a1: ["fixture-a"],
+ shared: ["fixture-a", "fixture-b"],
+ b1: ["fixture-b"],
+ });
+ assert.ok(!JSON.stringify(channelSitesOf(sites)).includes("fixture-unlisted"));
+});
diff --git a/common/controller/poolSummary.ts b/common/controller/poolSummary.ts
@@ -11,7 +11,7 @@ import path from "node:path";
import { mkdir, readFile } from "node:fs/promises";
import type { Paths } from "../lib/paths";
import { buildStats } from "./buildStats";
-import { listSites, type Site } from "../lib/site";
+import { isListedSite, listSites, type Site } from "../lib/site";
import {
statsPageFileName,
type StatsManifest,
@@ -45,11 +45,14 @@ export async function readStatsPages(statsDir: string): Promise<VideoStat[]> {
return out;
}
-// channel slug -> the ids of the content sites that expose it. A channel on
-// multiple sites maps to all of them; a pool-only channel is simply absent.
-export function channelSitesOf(sites: Site[]): ChannelSitesMap {
+// channel slug -> the ids of the LISTED content sites that expose it: the
+// published `channel-sites.json`. A channel on multiple sites maps to all of
+// them; a pool-only channel is simply absent, and so is an unlisted site
+// (site.json `listed: false`) and a channel only unlisted sites expose.
+export function channelSitesOf(sites: readonly Site[]): ChannelSitesMap {
const channelSites: ChannelSitesMap = {};
for (const site of sites) {
+ if (!isListedSite(site)) continue;
for (const c of site.channels) {
(channelSites[c.slug] ??= []).push(site.siteId);
}
@@ -71,8 +74,9 @@ export async function buildPoolSummary(opts: {
}): Promise<PoolSummary> {
const { paths, statsDir } = opts;
await mkdir(statsDir, { recursive: true });
- // Whole-pool stats dataset (every non-excluded channel). buildStats also
- // refreshes the per-site bundles as a side effect, which is harmless.
+ // Whole-pool stats dataset (every non-excluded channel but those only
+ // unlisted sites expose). buildStats also refreshes the per-site bundles as a
+ // side effect, which is harmless.
await buildStats({ paths, wholePoolStatsDir: statsDir });
const sites = listSites(paths);
const channelSites = channelSitesOf(sites);
diff --git a/common/controller/recencyIndex.ts b/common/controller/recencyIndex.ts
@@ -7,6 +7,12 @@ import { mapConcurrent } from "../lib/concurrency";
import { extractVideoId } from "../lib/videoId";
import { uploadKeyFor } from "./keptVideos";
import type { AutoQueueOrder } from "../jobs/autoQueuePolicy";
+import type { ChannelConfig } from "../lib/channelConfig";
+import {
+ isDriveNotAnswering,
+ onDrive,
+ stalledLocationForPath,
+} from "../lib/storageHealth";
// Upload-date lookup for the auto-queue's "newest first" ordering.
//
@@ -189,11 +195,23 @@ async function readTailUploadDate(file: string): Promise<string | null> {
// Date the ids in `wanted` from their on-disk metadata. Removes each id it
// keys from `wanted`, like interpolateFromPlaylist.
+//
+// `drives` maps a channel whose media is on another drive to its configured
+// target. Its reads go through `onDrive`: none while that drive's location is
+// stalled (lib/storageHealth.ts) — each would hold an I/O thread until the drive
+// came back, 32 at a time — at most `inFlightPerLocation` (4) in flight on it,
+// and one that has not answered within the budget (3 s by default) marks it
+// stalled. An id not read for that reason is NOT
+// memoized as a miss: it falls through to layers 3 and 4 for now, and a later
+// refresh with the drive answering reads it.
+const NOT_READ = Symbol("not read: the drive is not answering");
+
async function datesFromMetadata(
paths: Paths,
owner: ReadonlyMap<string, string>,
wanted: Set<string>,
out: Map<string, RecencyKey>,
+ drives: ReadonlyMap<string, string> = new Map(),
): Promise<void> {
const todo: string[] = [];
for (const id of wanted) {
@@ -205,7 +223,10 @@ async function datesFromMetadata(
}
continue;
}
- if (!owner.has(id)) continue;
+ const slug = owner.get(id);
+ if (slug === undefined) continue;
+ const drive = drives.get(slug);
+ if (drive && stalledLocationForPath(drive)) continue;
todo.push(id);
if (todo.length >= TAIL_READS_PER_BUILD) {
if (!tailCapLogged) {
@@ -219,19 +240,31 @@ async function datesFromMetadata(
}
}
if (todo.length === 0) return;
- const dates = await mapConcurrent(todo, TAIL_READ_CONCURRENCY, (id) =>
- readTailUploadDate(
- path.join(
+ const dates = await mapConcurrent(
+ todo,
+ TAIL_READ_CONCURRENCY,
+ async (id): Promise<string | null | typeof NOT_READ> => {
+ const slug = owner.get(id) as string;
+ const file = path.join(
paths.channelsDir,
- owner.get(id) as string,
+ slug,
"data",
id,
"metadata.info.json",
- ),
- ),
+ );
+ const drive = drives.get(slug);
+ if (!drive) return readTailUploadDate(file);
+ try {
+ return await onDrive(drive, () => readTailUploadDate(file));
+ } catch (err) {
+ if (isDriveNotAnswering(err)) return NOT_READ;
+ throw err;
+ }
+ },
);
for (const [i, id] of todo.entries()) {
const date = dates[i];
+ if (date === NOT_READ) continue;
if (tailMemo.size < TAIL_MEMO_CAP) tailMemo.set(id, date);
if (!date) continue;
out.set(id, { key: date, estimated: false });
@@ -312,7 +345,12 @@ export function interpolateFromPlaylist(
export type BuildRecencyKeysArgs = {
paths: Paths;
// The channels whose work is in play — the runner's own channel-meta list.
- meta: ReadonlyArray<{ slug: string }>;
+ // The config, when the caller holds it, is what tells layer 2 which channels
+ // are on another drive (their reads go through the stall watchdog).
+ meta: ReadonlyArray<{
+ slug: string;
+ config?: Pick<ChannelConfig, "dataDir"> | null;
+ }>;
// Every video id the caller might sort. Ids outside this set are not keyed.
candidateIds: ReadonlySet<string>;
// videoId -> owning channel slug. Required for the tail-read layer, which has
@@ -468,7 +506,12 @@ export async function buildRecencyKeys({
// auto-transcribe, whose entire candidate set is by definition absent from the
// transcript index.
if (owner && missing.size > 0) {
- await datesFromMetadata(paths, owner, missing, out);
+ const drives = new Map<string, string>();
+ for (const m of meta) {
+ const dir = m.config?.dataDir?.trim();
+ if (dir) drives.set(m.slug, dir);
+ }
+ await datesFromMetadata(paths, owner, missing, out, drives);
}
// Layer 3: playlist interpolation, for videos with nothing on disk at all.
diff --git a/common/controller/relocateChannelMedia.ts b/common/controller/relocateChannelMedia.ts
@@ -38,9 +38,17 @@ import {
type VolumeBins,
} from "../lib/storageVolumes";
import { getFreeBytes } from "../lib/diskSpace";
+import {
+ NOT_ANSWERING,
+ isDriveNotAnswering,
+ onDrive,
+ sinceText,
+ stalledLocation,
+} from "../lib/storageHealth";
import { getSettings, type SiteSettings } from "../lib/settings";
import { formatBytes } from "../lib/format";
import {
+ forgetChannelMedia,
inspectChannelMedia,
relocatedDataDir,
relocationMarkerPath,
@@ -282,10 +290,30 @@ export async function relocationRootPresenceProblem(
const named = locationForRoot(r, storage.locations);
const where = named ? ` (location "${named.id}")` : "";
+ // A destination whose drive is not answering is refused WITHOUT the stat
+ // below: that stat would wait on the drive, and a move onto it would too.
+ const stall = named ? stalledLocation(named) : null;
+ if (stall) {
+ return (
+ `The destination root ${r}${where}: ${NOT_ANSWERING} ` +
+ `(${sinceText(stall.since)}). Wait for it to answer, or check the drive.`
+ );
+ }
+
+ // Through the watchdog: a stat that has not answered within the budget
+ // (`storage.health.budgetMs`, 3 s by default) marks the location stalled
+ // and refuses the same way.
let isDir = false;
try {
- isDir = (await stat(r)).isDirectory();
- } catch {
+ isDir = (await onDrive(named ?? r, () => stat(r))).isDirectory();
+ } catch (err) {
+ if (isDriveNotAnswering(err)) {
+ return (
+ `The destination root ${r}${where}: ${NOT_ANSWERING} ` +
+ `(${err.health ? sinceText(err.health.since) : "a stat of it did not answer"}). ` +
+ `Wait for it to answer, or check the drive.`
+ );
+ }
isDir = false;
}
if (!isDir) {
@@ -373,16 +401,23 @@ async function sweepParked(
// The channel's marker, at `channels/<slug>/.relocating.json`. The FILE is the
// contract — see relocateDir.ts — and these three are the channel's name for it.
+//
+// Each write and the clear also drop the channel from `inspectChannelMedia`'s
+// five-second page memo. Every phase change (copy → swap → reclaim) writes the
+// marker AFTER the link and the config it changes, so a page asks the disk
+// again the moment the move has done something it would see.
async function writeMarker(
paths: Paths,
slug: string,
marker: RelocationMarker,
): Promise<void> {
await writeDirMarker(relocationMarkerPath(paths, slug), marker);
+ forgetChannelMedia(slug);
}
async function clearMarker(paths: Paths, slug: string): Promise<void> {
await clearDirMarker(relocationMarkerPath(paths, slug));
+ forgetChannelMedia(slug);
}
async function readMarkerRaw(
@@ -601,7 +636,9 @@ async function moveOut(args: {
// every guard and exactly wrong here: the rerun that finishes an interrupted
// move is the one caller allowed to see it. A marker for a DIFFERENT target
// was already refused above.
- const location = await inspectChannelMedia(paths, slug, args.config);
+ const location = await inspectChannelMedia(paths, slug, args.config, {
+ fresh: true,
+ });
if (!args.resumed && location.status !== "in-place") {
throw new Error(
`Channel "${slug}" is not in a movable state: ${
@@ -838,7 +875,9 @@ async function moveBack(args: {
//
// `in-transition` is allowed because a marker is what a resume carries, and a
// rerun is the caller this precondition must not refuse.
- const location = await inspectChannelMedia(paths, slug, args.config);
+ const location = await inspectChannelMedia(paths, slug, args.config, {
+ fresh: true,
+ });
if (
!args.resumed &&
location.status !== "ok" &&
diff --git a/common/controller/relocateSavedVideos.ts b/common/controller/relocateSavedVideos.ts
@@ -20,6 +20,13 @@ import {
} from "../lib/settings";
import type { RelocationMarker, RelocationPhase } from "../lib/channelMedia";
import {
+ NOT_ANSWERING,
+ isDriveNotAnswering,
+ onDrive,
+ sinceText,
+ stalledLocation,
+} from "../lib/storageHealth";
+import {
relocatedSavedVideosDir,
savedVideosMarkerPath,
} from "../lib/savedVideoStore";
@@ -156,7 +163,33 @@ export async function inspectSavedVideosStore(
detail: `the store links to ${target}, but no storage location is recorded for it`,
};
}
- if (!(await isDirectory(target))) {
+ // The store's drive is not answering: reported without the stat below,
+ // which would wait on it (the /storage page asks on every render). The
+ // page skips the store's size walk for an unreachable store, too.
+ const stall = loc ? stalledLocation(loc) : null;
+ if (stall) {
+ return {
+ dir,
+ locationId,
+ target,
+ status: "unreachable",
+ detail: `${NOT_ANSWERING} (location "${stall.label}", ${sinceText(stall.since)})`,
+ };
+ }
+ let reachable: boolean;
+ try {
+ reachable = await onDrive(loc ?? target, () => isDirectory(target));
+ } catch (err) {
+ if (!isDriveNotAnswering(err)) throw err;
+ return {
+ dir,
+ locationId,
+ target,
+ status: "unreachable",
+ detail: err.message,
+ };
+ }
+ if (!reachable) {
return {
dir,
locationId,
diff --git a/common/controller/savedVideoInventory.ts b/common/controller/savedVideoInventory.ts
@@ -4,8 +4,13 @@ import type { Paths } from "../lib/paths";
import { mapConcurrent } from "../lib/concurrency";
import { loadSavedVideo } from "../lib/savedVideo-server";
import { savedVideoPath, type SavedVideoPointer } from "../lib/savedVideo";
+import {
+ isDriveNotAnswering,
+ onDrive,
+ stalledLocationForPath,
+} from "../lib/storageHealth";
-const { pathExists, readdir } = fs;
+const { pathExists, readdir, readlink } = fs;
// Channels are read a few at a time so the per-video fan-out inside each one
// still dominates; the product is the real ceiling on open descriptors.
@@ -37,12 +42,23 @@ async function listChannelSlugs(paths: Paths): Promise<string[]> {
// channel when channelSlug is given), by following the saved-video.json pointers
// in each data dir. The pointer's `dir` is absolute, so per-channel store
// overrides resolve correctly without consulting channel config here.
+//
+// `notAnswering`, for a PAGE: pass an array and every channel whose `data/`
+// links onto a storage location whose drive is not answering
+// (lib/storageHealth.ts) is skipped — its pointers are one read per video dir,
+// each of which would wait on the drive — and its slug is pushed there, so the
+// page can say which channels it did not read. The link is read, not followed:
+// it is on the corpus disk. A relocated channel's reads then go through
+// `onDrive`'s watchdog, so a drive that stops answering mid-list is skipped and
+// named the same way. Omitted (the backup job), nothing is skipped or raced.
export async function listSavedVideos({
paths,
channelSlug,
+ notAnswering,
}: {
paths: Paths;
channelSlug?: string;
+ notAnswering?: string[];
}): Promise<SavedVideoEntry[]> {
const slugs = channelSlug ? [channelSlug] : await listChannelSlugs(paths);
// One pointer read per video dir — 78,350 of them across the corpus. Done one
@@ -53,25 +69,43 @@ export async function listSavedVideos({
CHANNEL_CONCURRENCY,
async (slug): Promise<SavedVideoEntry[]> => {
const dataDir = path.join(paths.channelsDir, slug, "data");
- if (!(await pathExists(dataDir))) return [];
- const ids = await readdir(dataDir).catch(() => [] as string[]);
- const entries = await mapConcurrent(
- ids,
- VIDEO_CONCURRENCY,
- async (videoId): Promise<SavedVideoEntry | null> => {
- const videoDir = path.join(dataDir, videoId);
- const pointer = await loadSavedVideo(videoDir);
- if (!pointer) return null;
- return {
- slug,
- videoId,
- videoDir,
- storedPath: savedVideoPath(pointer),
- pointer,
- };
- },
- );
- return entries.filter((e): e is SavedVideoEntry => e !== null);
+ let drive = "";
+ if (notAnswering) {
+ drive = await readlink(dataDir).catch(() => "");
+ if (drive && stalledLocationForPath(drive)) {
+ notAnswering.push(slug);
+ return [];
+ }
+ }
+ const through = <T>(call: () => Promise<T>): Promise<T> =>
+ drive ? onDrive(drive, call) : call();
+ try {
+ if (!(await through(() => pathExists(dataDir)))) return [];
+ const ids = await through(() =>
+ readdir(dataDir).catch(() => [] as string[]),
+ );
+ const entries = await mapConcurrent(
+ ids,
+ VIDEO_CONCURRENCY,
+ async (videoId): Promise<SavedVideoEntry | null> => {
+ const videoDir = path.join(dataDir, videoId);
+ const pointer = await through(() => loadSavedVideo(videoDir));
+ if (!pointer) return null;
+ return {
+ slug,
+ videoId,
+ videoDir,
+ storedPath: savedVideoPath(pointer),
+ pointer,
+ };
+ },
+ );
+ return entries.filter((e): e is SavedVideoEntry => e !== null);
+ } catch (err) {
+ if (!notAnswering || !isDriveNotAnswering(err)) throw err;
+ notAnswering.push(slug);
+ return [];
+ }
},
);
const out = perChannel.flat();
@@ -91,6 +125,7 @@ export type SavedVideoTotals = {
export async function savedVideoTotals(opts: {
paths: Paths;
channelSlug?: string;
+ notAnswering?: string[];
}): Promise<SavedVideoTotals> {
const entries = await listSavedVideos(opts);
let bytes = 0;
diff --git a/common/controller/storageLocations.test.ts b/common/controller/storageLocations.test.ts
@@ -561,7 +561,7 @@ test("defaultStorage is still the empty list (the harness's assumption)", () =>
// The first version of this guard compared `stat(root).dev` with its parent's,
// and that is wrong for the only location on the production machine: a root is
// `join(mountpoint, relPath)`, `platter` is
-// `/run/media/user/<uuid>/archilyzer-media`, and a subdirectory is on the same
+// `/run/media/<user>/<uuid>/archilyzer-media`, and a subdirectory is on the same
// filesystem as its parent BY CONSTRUCTION. /channels reported "free space
// unknown" for a correctly mounted drive. A bind mount reads the same way.
diff --git a/common/controller/storageLocations.ts b/common/controller/storageLocations.ts
@@ -24,7 +24,16 @@ import {
type StorageLocationProbe,
type VolumeBins,
} from "../lib/storageVolumes";
-import { inspectChannelMedia, relocatedDataDir } from "../lib/channelMedia";
+import {
+ forgetChannelMedia,
+ inspectChannelMedia,
+ relocatedDataDir,
+} from "../lib/channelMedia";
+import {
+ isDriveNotAnswering,
+ onDrive,
+ stalledLocation,
+} from "../lib/storageHealth";
import { relocationQueueKey } from "../lib/queueKeys";
import {
runManagedFunction,
@@ -247,61 +256,86 @@ export async function volumeFreeBytes(opts: {
out[INTERNAL_LOCATION_ID] = Number.isFinite(corpus) ? corpus : undefined;
await Promise.all(
opts.locations.map(async (loc) => {
- const st = await stat(loc.root).catch(() => null);
- if (!st?.isDirectory()) {
+ // NOT ASKED WHILE ITS DRIVE IS NOT ANSWERING: each stat and the statfs
+ // below would hold an I/O thread until it did. "—", like unmounted. The
+ // calls that reach the drive go through `onDrive`'s watchdog; one that
+ // has not answered within the budget (3 s by default) marks the location
+ // stalled, and reads "—" too.
+ if (stalledLocation(loc)) {
out[loc.id] = undefined;
return;
}
- // THE DIRECTORY EXISTING IS NOT THE DRIVE BEING THERE, and this is the
- // half the stat above could not catch. An unmounted mountpoint is a real,
- // empty directory ON ITS PARENT'S FILESYSTEM — so `getFreeBytes` succeeds
- // and reports the parent volume's free space, which on this machine is
- // the disk the operator is trying to empty. The location would then read
- // "233 GB free" about a platter that is not plugged in.
- //
- // THE BOUNDARY IS TESTED AT THE MOUNTPOINT, NEVER AT THE ROOT, and the
- // first version of this got that wrong in the one way that matters on
- // this machine. A location's root is `join(mountpoint, relPath)` — the
- // production `platter` is `/run/media/user/<uuid>/archilyzer-media`, a
- // SUBDIRECTORY of the mountpoint — so the root and its parent are on the
- // same filesystem BY CONSTRUCTION whenever `relPath` is non-empty, and
- // comparing those two devices reported "free space unknown" for a
- // correctly mounted drive. A bind mount reads the same way.
- //
- // Crossing a mount changes the device number, so `stat(mountpoint).dev`
- // against `stat(dirname(mountpoint)).dev` is the honest question, and it
- // is two syscalls — TABLES NEVER PROBE (see the header) rules out asking
- // `probeLocation`, which is up to three subprocesses.
- //
- // Only asked of a location that has learned a `volume.uuid`: that field
- // is the assertion that the root is supposed to be on its own volume. A
- // location on a plain directory (never probed, a container, a
- // subdirectory of the system disk by design) shares its parent's device
- // legitimately, and withholding its free space would be wrong.
- const mountpoint = loc.volume?.uuid
- ? (loc.volume.mountpoint ?? "").trim()
- : "";
- // `/` is its own parent, so a volume mounted at the root has no boundary
- // to test and is trivially there — the process is reading from it.
- if (mountpoint && mountpoint !== path.dirname(mountpoint)) {
- const [atMount, aboveMount] = await Promise.all([
- stat(mountpoint).catch(() => null),
- stat(path.dirname(mountpoint)).catch(() => null),
- ]);
- // Gone entirely, or present as an ordinary directory on the parent
- // filesystem: either way nothing is mounted there.
- if (!atMount || (aboveMount && aboveMount.dev === atMount.dev)) {
- out[loc.id] = undefined;
- return;
- }
+ try {
+ out[loc.id] = await freeOnLocation(loc);
+ } catch (err) {
+ if (!isDriveNotAnswering(err)) throw err;
+ out[loc.id] = undefined;
}
- const free = await getFreeBytes(loc.root);
- out[loc.id] = Number.isFinite(free) ? free : undefined;
}),
);
return out;
}
+// One location's free space, for volumeFreeBytes. Throws DriveNotAnsweringError
+// when its drive does not answer.
+async function freeOnLocation(
+ loc: StorageLocation,
+): Promise<number | undefined> {
+ const orNull = <T>(p: Promise<T>): Promise<T | null> =>
+ p.catch((err) => {
+ if (isDriveNotAnswering(err)) throw err;
+ return null;
+ });
+ const st = await orNull(onDrive(loc, () => stat(loc.root)));
+ if (!st?.isDirectory()) return undefined;
+ // THE DIRECTORY EXISTING IS NOT THE DRIVE BEING THERE, and this is the
+ // half the stat above could not catch. An unmounted mountpoint is a real,
+ // empty directory ON ITS PARENT'S FILESYSTEM — so `getFreeBytes` succeeds
+ // and reports the parent volume's free space, which on this machine is
+ // the disk the operator is trying to empty. The location would then read
+ // "233 GB free" about a platter that is not plugged in.
+ //
+ // THE BOUNDARY IS TESTED AT THE MOUNTPOINT, NEVER AT THE ROOT, and the
+ // first version of this got that wrong in the one way that matters on
+ // this machine. A location's root is `join(mountpoint, relPath)` — the
+ // production `platter` is `/run/media/<user>/<uuid>/archilyzer-media`, a
+ // SUBDIRECTORY of the mountpoint — so the root and its parent are on the
+ // same filesystem BY CONSTRUCTION whenever `relPath` is non-empty, and
+ // comparing those two devices reported "free space unknown" for a
+ // correctly mounted drive. A bind mount reads the same way.
+ //
+ // Crossing a mount changes the device number, so `stat(mountpoint).dev`
+ // against `stat(dirname(mountpoint)).dev` is the honest question, and it
+ // is two syscalls — TABLES NEVER PROBE (see the header) rules out asking
+ // `probeLocation`, which is up to three subprocesses.
+ //
+ // Only asked of a location that has learned a `volume.uuid`: that field
+ // is the assertion that the root is supposed to be on its own volume. A
+ // location on a plain directory (never probed, a container, a
+ // subdirectory of the system disk by design) shares its parent's device
+ // legitimately, and withholding its free space would be wrong.
+ const mountpoint = loc.volume?.uuid
+ ? (loc.volume.mountpoint ?? "").trim()
+ : "";
+ // `/` is its own parent, so a volume mounted at the root has no boundary
+ // to test and is trivially there — the process is reading from it.
+ if (mountpoint && mountpoint !== path.dirname(mountpoint)) {
+ // The mountpoint is the drive's own top directory; its parent is on the
+ // filesystem above it, so only the first goes through the watchdog.
+ const [atMount, aboveMount] = await Promise.all([
+ orNull(onDrive(loc, () => stat(mountpoint))),
+ stat(path.dirname(mountpoint)).catch(() => null),
+ ]);
+ // Gone entirely, or present as an ordinary directory on the parent
+ // filesystem: either way nothing is mounted there.
+ if (!atMount || (aboveMount && aboveMount.dev === atMount.dev)) {
+ return undefined;
+ }
+ }
+ const free = await onDrive(loc, () => getFreeBytes(loc.root));
+ return Number.isFinite(free) ? free : undefined;
+}
+
// ---------------------------------------------------------------------------
// The probe memo
// ---------------------------------------------------------------------------
@@ -528,7 +562,9 @@ export async function preflightRepoint(opts: {
const missing: string[] = [];
for (const slug of slugs) {
const config = await readChannelConfig(opts.paths, slug);
- const media = await inspectChannelMedia(opts.paths, slug, config);
+ const media = await inspectChannelMedia(opts.paths, slug, config, {
+ fresh: true,
+ });
if (media.status === "in-transition") {
base.problems.push(
`${slug}: a media relocation is in flight or was interrupted ` +
@@ -795,6 +831,10 @@ export async function repointStorageLocation(opts: {
}
resetStorageProbeMemo();
+ // Every channel on it has a new link and a new dataDir: the page memo's keys
+ // already differ, and this drops the old answers rather than letting them
+ // age out.
+ forgetChannelMedia();
log(
`Done. "${loc.label}" is at ${pre.newRoot}; ${pre.channels.length} ` +
`channel(s) re-pointed.`,
diff --git a/common/controller/storageStall.test.ts b/common/controller/storageStall.test.ts
@@ -0,0 +1,544 @@
+// A STALLED DRIVE IS NOT ASKED: every page-and-poll path that would touch a
+// storage location's drive in-process asks the health state first
+// (lib/storageHealth.ts) and, on a stalled location, answers WITHOUT the call.
+//
+// No test stalls a real drive. The drive here is an ordinary temp directory
+// that answers every call at once; the health state is TOLD it is stalled.
+// Every node:fs and node:fs/promises call this file's code makes is recorded
+// (the buildIndex.test.ts spy, reads included), so a gate that let one call
+// through shows up as a recorded path under the drive — it cannot hide behind
+// a call that happened to answer quickly.
+//
+// Run with:
+// pnpm --filter yt-dlp-transcript-common exec tsx --test controller/storageStall.test.ts
+
+import { after, beforeEach, test } from "node:test";
+import assert from "node:assert/strict";
+import { createRequire, syncBuiltinESMExports } from "node:module";
+import {
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ rmSync,
+ symlinkSync,
+ writeFileSync,
+} from "node:fs";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import type { Paths } from "../lib/paths";
+import type { StorageLocation } from "../lib/storageLocations";
+import {
+ assertChannelMediaReachable,
+ ChannelMediaUnreachableError,
+ CHANNEL_MEDIA_MEMO_MS,
+ clearRelocationMarker,
+ forgetChannelMedia,
+ inspectChannelMedia,
+} from "../lib/channelMedia";
+import { HELD_REASON, isMediaHeld } from "../lib/channelMediaHold";
+import {
+ locationHealth,
+ recordLocationHealth,
+ registerLocationHealth,
+ resetStorageHealth,
+ setDriveCallBudget,
+} from "../lib/storageHealth";
+import {
+ probeLocation,
+ probeLocationMemo,
+ resetStorageProbeMemo,
+} from "../lib/storageVolumes";
+import { volumeFreeBytes } from "./storageLocations";
+import { readChannelStat } from "./channels";
+import { buildRecencyKeys, clearRecencyCache } from "./recencyIndex";
+import { relocationRootPresenceProblem } from "./relocateChannelMedia";
+import { generateChannelSnapshot } from "./channelSnapshot";
+import { inspectSavedVideosStore } from "./relocateSavedVideos";
+import { listSavedVideos } from "./savedVideoInventory";
+import type { SiteSettings } from "../lib/settings";
+
+// ── the spy ────────────────────────────────────────────────────────────────
+type Call = { fn: string; path: string };
+let calls: Call[] = [];
+// THE HANG: a promise-API call whose (function, path) matches never settles —
+// what a read blocked on a stalled drive looks like from here. Recorded like
+// any other call. No drive is involved.
+let hang: ((c: Call) => boolean) | null = null;
+{
+ const req = createRequire(import.meta.url);
+ const fsCjs = req("node:fs") as Record<string, unknown>;
+ const fspCjs = req("node:fs/promises") as Record<string, unknown>;
+ const NAMES = [
+ "access", "appendFile", "chmod", "copyFile", "cp", "lstat", "mkdir",
+ "open", "opendir", "readdir", "readFile", "readlink", "realpath", "rename",
+ "rm", "rmdir", "stat", "statfs", "symlink", "unlink", "utimes", "writeFile",
+ ];
+ const asPath = (v: unknown) =>
+ typeof v === "string"
+ ? v
+ : v instanceof URL
+ ? fileURLToPath(v)
+ : Buffer.isBuffer(v)
+ ? v.toString()
+ : null;
+ const wrap = (mod: Record<string, unknown>, name: string, promises = false) => {
+ const fn = mod[name];
+ if (typeof fn !== "function") return;
+ mod[name] = function (this: unknown, ...args: unknown[]) {
+ const p = asPath(args[0]);
+ if (p !== null) {
+ const call = { fn: name, path: path.resolve(p) };
+ calls.push(call);
+ if (promises && hang?.(call)) return new Promise(() => {});
+ }
+ return (fn as (...a: unknown[]) => unknown).apply(this, args);
+ };
+ };
+ for (const n of NAMES) {
+ wrap(fspCjs, n, true);
+ wrap(fsCjs, n);
+ wrap(fsCjs, `${n}Sync`);
+ }
+ wrap(fsCjs, "existsSync");
+ wrap(fsCjs, "createReadStream");
+ syncBuiltinESMExports();
+}
+
+// ── the corpus and the drive ───────────────────────────────────────────────
+const ROOT = mkdtempSync(path.join(tmpdir(), "ttb-stall-"));
+after(() => rmSync(ROOT, { recursive: true, force: true }));
+
+const CORPUS = path.join(ROOT, "corpus");
+const DRIVE = path.join(ROOT, "drive");
+const SLUG = "on-drive";
+const paths = {
+ transcriptsDir: CORPUS,
+ channelsDir: path.join(CORPUS, "channels"),
+ lmdbPath: path.join(CORPUS, "index.mdb"),
+ savedVideosDir: path.join(CORPUS, "saved-videos"),
+ jobsDir: path.join(CORPUS, "jobs"),
+ // A findmnt that leaves a mark if anything runs it.
+ findmntBin: path.join(ROOT, "findmnt-ran.sh"),
+ udisksctlBin: path.join(ROOT, "no-udisksctl"),
+} as unknown as Paths;
+const LOC: StorageLocation = {
+ id: "usb",
+ label: "USB drive",
+ root: DRIVE,
+ autoRepoint: false,
+};
+const TARGET = path.join(DRIVE, SLUG, "data");
+const LINK = path.join(paths.channelsDir, SLUG, "data");
+const CONFIG = { dataDir: TARGET };
+const FINDMNT_MARK = path.join(ROOT, "findmnt-ran");
+
+function seed(): void {
+ rmSync(CORPUS, { recursive: true, force: true });
+ rmSync(DRIVE, { recursive: true, force: true });
+ mkdirSync(path.join(TARGET, "vid1"), { recursive: true });
+ writeFileSync(
+ path.join(TARGET, "vid1", "metadata.info.json"),
+ JSON.stringify({ id: "vid1", upload_date: "20260601" }),
+ );
+ writeFileSync(path.join(TARGET, "vid1", "transcript.en.vtt"), "WEBVTT\n");
+ mkdirSync(path.join(paths.channelsDir, SLUG), { recursive: true });
+ writeFileSync(
+ path.join(paths.channelsDir, SLUG, "config.json"),
+ JSON.stringify({
+ handling: "youtube",
+ name: SLUG,
+ url: `https://www.youtube.com/@${SLUG}/videos`,
+ dataDir: TARGET,
+ }),
+ );
+ symlinkSync(TARGET, LINK);
+ writeFileSync(paths.findmntBin, `#!/bin/sh\ntouch ${FINDMNT_MARK}\nexit 1\n`, {
+ mode: 0o755,
+ });
+ rmSync(FINDMNT_MARK, { force: true });
+}
+
+// Anything that reaches the drive: a path under its root, or through the
+// channel's link (`data/` itself, followed, or anything below it).
+function onDrive(c: Call): boolean {
+ const under = (p: string, base: string) =>
+ p === base || p.startsWith(base + path.sep);
+ return under(c.path, DRIVE) || under(c.path, LINK);
+}
+
+function stall(): void {
+ recordLocationHealth(LOC, "stalled", { now: Date.now() });
+}
+
+beforeEach(() => {
+ hang = null;
+ setDriveCallBudget();
+ resetStorageHealth();
+ forgetChannelMedia();
+ resetStorageProbeMemo();
+ clearRecencyCache();
+ seed();
+ calls = [];
+});
+
+// ── inspectChannelMedia: the gate and the memo ─────────────────────────────
+
+const MARKER = path.join(paths.channelsDir, SLUG, ".relocating.json");
+
+test("inspect with the config in hand: on a stalled drive only the marker is read, on the corpus disk", async () => {
+ stall();
+ calls = [];
+ const media = await inspectChannelMedia(paths, SLUG, CONFIG);
+ assert.deepEqual(calls, [{ fn: "readFile", path: MARKER }]);
+ assert.deepEqual(calls.filter(onDrive), []);
+ assert.equal(media.status, "stalled");
+ assert.equal(media.target, TARGET);
+ assert.match(String(media.detail), /^drive not answering \(location "USB drive", since /);
+ // Held by both pool-wide builds, with a reason that names no path.
+ assert.equal(isMediaHeld(media.status), true);
+ assert.doesNotMatch(HELD_REASON.stalled, /\//);
+});
+
+test("inspect without the config reads config.json and nothing on the drive", async () => {
+ stall();
+ calls = [];
+ const media = await inspectChannelMedia(paths, SLUG);
+ assert.equal(media.status, "stalled");
+ assert.deepEqual(
+ calls.map((c) => [c.fn, path.relative(ROOT, c.path)]),
+ [
+ ["readFile", path.join("corpus", "channels", SLUG, "config.json")],
+ ["readFile", path.join("corpus", "channels", SLUG, ".relocating.json")],
+ ],
+ );
+});
+
+test("the marker first: a channel mid-move on a stalled drive reads in-transition", async () => {
+ writeFileSync(
+ MARKER,
+ JSON.stringify({ target: TARGET, direction: "back", startedAt: "", phase: "copy" }),
+ );
+ stall();
+ calls = [];
+ const media = await inspectChannelMedia(paths, SLUG, CONFIG);
+ assert.equal(media.status, "in-transition");
+ assert.deepEqual(calls.filter(onDrive), []);
+ // Remembered as a move: a second look within five seconds gives the move
+ // back, not the stall.
+ const again = await inspectChannelMedia(paths, SLUG, CONFIG);
+ assert.equal(again.status, "in-transition");
+});
+
+test("the start-of-work guard refuses a stalled channel without a call", async () => {
+ stall();
+ calls = [];
+ await assert.rejects(
+ () => assertChannelMediaReachable(paths, SLUG, CONFIG),
+ (err: unknown) =>
+ err instanceof ChannelMediaUnreachableError &&
+ err.status === "stalled" &&
+ /drive not answering/.test(err.message),
+ );
+ assert.deepEqual(calls.filter(onDrive), []);
+});
+
+test("the memo: two inspects within 5 s stat the drive once; fresh and age each ask again", async () => {
+ const t0 = 1_000_000;
+ const targetStats = () =>
+ calls.filter((c) => c.fn === "stat" && c.path === TARGET).length;
+ const a = await inspectChannelMedia(paths, SLUG, CONFIG, { now: t0 });
+ assert.equal(a.status, "ok");
+ assert.equal(targetStats(), 1);
+ const b = await inspectChannelMedia(paths, SLUG, CONFIG, { now: t0 + 4_999 });
+ assert.equal(b.status, "ok");
+ assert.equal(targetStats(), 1, "a second inspect inside five seconds is remembered");
+ await inspectChannelMedia(paths, SLUG, CONFIG, { now: t0 + 4_999, fresh: true });
+ assert.equal(targetStats(), 2, "fresh bypasses the memo");
+ await inspectChannelMedia(paths, SLUG, CONFIG, { now: t0 + CHANNEL_MEDIA_MEMO_MS });
+ assert.equal(targetStats(), 3, "an answer five seconds old is asked again");
+ // A fresh answer is not remembered: the memo still holds the one from t0+5000.
+ await inspectChannelMedia(paths, SLUG, CONFIG, { now: t0 + CHANNEL_MEDIA_MEMO_MS + 1 });
+ assert.equal(targetStats(), 3);
+});
+
+test("the memo is keyed by the configured target, and a mover's forget clears it", async () => {
+ const now = 2_000_000;
+ const targetStats = () =>
+ calls.filter((c) => c.fn === "stat" && c.path === TARGET).length;
+ await inspectChannelMedia(paths, SLUG, CONFIG, { now });
+ // Another configured target is another key.
+ const other = await inspectChannelMedia(paths, SLUG, { dataDir: path.join(DRIVE, "x", "data") }, { now });
+ assert.equal(other.status, "inconsistent");
+ forgetChannelMedia(SLUG);
+ await inspectChannelMedia(paths, SLUG, CONFIG, { now });
+ assert.equal(targetStats(), 2);
+ // clearRelocationMarker (the operator's last resort) forgets too.
+ await clearRelocationMarker(paths, SLUG);
+ await inspectChannelMedia(paths, SLUG, CONFIG, { now });
+ assert.equal(targetStats(), 3);
+});
+
+test("the gate is asked before the memo: a stall is seen with an ok remembered", async () => {
+ const now = 3_000_000;
+ assert.equal((await inspectChannelMedia(paths, SLUG, CONFIG, { now })).status, "ok");
+ stall();
+ calls = [];
+ const media = await inspectChannelMedia(paths, SLUG, CONFIG, { now: now + 1 });
+ assert.equal(media.status, "stalled");
+ assert.deepEqual(calls, []);
+});
+
+// ── the other gated callers ────────────────────────────────────────────────
+
+test("probeLocation and its memo: 'stalled', with no stat, no statfs and no findmnt", async () => {
+ // A remembered "available" first, so the memo's gate is the thing tested.
+ await probeLocationMemo(LOC, paths);
+ rmSync(FINDMNT_MARK, { force: true });
+ stall();
+ calls = [];
+ const direct = await probeLocation(LOC, paths);
+ const memo = await probeLocationMemo(LOC, paths);
+ assert.equal(direct.status, "stalled");
+ assert.equal(memo.status, "stalled");
+ assert.deepEqual(direct.identity, { known: false });
+ assert.equal(direct.freeBytes, undefined);
+ assert.deepEqual(calls.filter(onDrive), []);
+ assert.equal(existsSync(FINDMNT_MARK), false, "findmnt was not run");
+});
+
+test("volumeFreeBytes: a stalled location reads unknown, with no call on it", async () => {
+ stall();
+ calls = [];
+ const out = await volumeFreeBytes({ paths, locations: [LOC] });
+ assert.equal(out.usb, undefined);
+ assert.equal(typeof out.internal, "number");
+ assert.deepEqual(calls.filter(onDrive), []);
+});
+
+test("readChannelStat: no walk of data/ on a stalled drive", async () => {
+ const before = await readChannelStat(paths, SLUG);
+ assert.equal(before?.videoCount, 1);
+ stall();
+ calls = [];
+ assert.equal(await readChannelStat(paths, SLUG), null);
+ assert.deepEqual(calls.filter(onDrive), []);
+});
+
+test("recency: no tail read on a stalled drive, and no miss remembered for it", async () => {
+ const owner = new Map([["vid1", SLUG]]);
+ const meta = [{ slug: SLUG, config: CONFIG }];
+ const args = {
+ paths,
+ meta,
+ candidateIds: new Set(["vid1"]),
+ owner,
+ interpolate: false,
+ fresh: true,
+ };
+ stall();
+ calls = [];
+ const stalledKeys = await buildRecencyKeys(args);
+ assert.deepEqual(calls.filter(onDrive), []);
+ // Layer 4: an undatable id sorts oldest.
+ assert.deepEqual(stalledKeys.get("vid1"), { key: "", estimated: false });
+ // The drive answers again: the same id is read now, because the stall was
+ // not remembered as a miss.
+ resetStorageHealth();
+ const keys = await buildRecencyKeys(args);
+ assert.deepEqual(keys.get("vid1"), { key: "20260601", estimated: false });
+ assert.ok(calls.some((c) => c.fn === "open" && onDrive(c)));
+});
+
+test("a move onto a stalled location is refused without a stat of its root", async () => {
+ stall();
+ calls = [];
+ const problem = await relocationRootPresenceProblem(
+ DRIVE,
+ { locations: [LOC], defaultLocationId: "" },
+ paths,
+ );
+ assert.match(String(problem), /drive not answering/);
+ assert.match(String(problem), /location "usb"/);
+ assert.deepEqual(calls.filter(onDrive), []);
+});
+
+test("a snapshot refresh of a stalled channel throws before its walk", async () => {
+ stall();
+ calls = [];
+ await assert.rejects(
+ () => generateChannelSnapshot(paths, SLUG),
+ (err: unknown) =>
+ err instanceof ChannelMediaUnreachableError && err.status === "stalled",
+ );
+ assert.deepEqual(calls.filter(onDrive), []);
+});
+
+test("the saved-video store on a stalled drive reads unreachable, without a stat through its link", async () => {
+ const storeTarget = path.join(DRIVE, "saved-videos");
+ mkdirSync(storeTarget, { recursive: true });
+ symlinkSync(storeTarget, paths.savedVideosDir);
+ const settings = {
+ storage: { locations: [LOC], defaultLocationId: "", savedVideosLocationId: "usb" },
+ } as unknown as SiteSettings;
+ assert.equal((await inspectSavedVideosStore(paths, settings)).status, "ok");
+ stall();
+ calls = [];
+ const store = await inspectSavedVideosStore(paths, settings);
+ assert.equal(store.status, "unreachable");
+ assert.match(String(store.detail), /^drive not answering \(location "USB drive"/);
+ const followed = calls.filter(
+ (c) =>
+ onDrive(c) ||
+ (c.path.startsWith(paths.savedVideosDir) && !["lstat", "readlink"].includes(c.fn)),
+ );
+ assert.deepEqual(followed, []);
+});
+
+test("the saved-video inventory, for a page: a stalled channel is named, not read", async () => {
+ // A saved video on the drive, so a channel that WAS read has an entry.
+ // (savedVideoInventory reads through fs-extra, whose functions graceful-fs
+ // captured before this file's spy was installed, so this case proves the skip
+ // by what comes back, not by the spy.)
+ writeFileSync(
+ path.join(TARGET, "vid1", "saved-video.json"),
+ JSON.stringify({ storedAt: "", dir: "/store", file: "v.mp4", bytes: 7 }),
+ );
+ assert.equal((await listSavedVideos({ paths, notAnswering: [] })).length, 1);
+ stall();
+ const notAnswering: string[] = [];
+ assert.deepEqual(await listSavedVideos({ paths, notAnswering }), []);
+ assert.deepEqual(notAnswering, [SLUG]);
+ // Without the array (the backup job) nothing is skipped: it reads the drive.
+ assert.equal((await listSavedVideos({ paths })).length, 1);
+});
+
+// ── the watchdog: a call that does not answer in the budget ────────────────
+// The location is registered (the health pass does that in the editor) but
+// answering; the hang makes one call on the drive never settle. The budget is
+// shortened to 100 ms; lib/storageHealth.test.ts holds the 3 s default.
+
+function hangOnDrive(fns: string[]): void {
+ hang = (c) => fns.includes(c.fn) && onDrive(c);
+}
+
+async function watchdogCase(
+ fns: string[],
+ run: () => Promise<unknown>,
+): Promise<unknown> {
+ registerLocationHealth([LOC]);
+ setDriveCallBudget(100);
+ hangOnDrive(fns);
+ calls = [];
+ const started = Date.now();
+ const out = await run();
+ assert.ok(Date.now() - started < 2_000, "answered on the watchdog, not on the drive");
+ assert.equal(locationHealth("usb")?.state, "stalled", "the location is marked at once");
+ assert.match(String(locationHealth("usb")?.cause), /a read in the editor did not answer/);
+ return out;
+}
+
+test("watchdog: inspect's target stat never answers → stalled, marked, and nothing more is asked of the drive", async () => {
+ const media = (await watchdogCase(["stat"], () =>
+ inspectChannelMedia(paths, SLUG, CONFIG),
+ )) as { status: string; detail?: string };
+ assert.equal(media.status, "stalled");
+ assert.match(String(media.detail), /^drive not answering \(location "USB drive"/);
+ // The next inspect, the guard and the walk make no call on the drive.
+ calls = [];
+ assert.equal((await inspectChannelMedia(paths, SLUG, CONFIG)).status, "stalled");
+ await assert.rejects(() => assertChannelMediaReachable(paths, SLUG, CONFIG));
+ assert.equal(await readChannelStat(paths, SLUG), null);
+ assert.deepEqual(calls.filter(onDrive), []);
+ // Cleared (two clean answers), the drive is asked again.
+ recordLocationHealth(LOC, "ok");
+ recordLocationHealth(LOC, "ok");
+ hang = null;
+ assert.equal(
+ (await inspectChannelMedia(paths, SLUG, CONFIG, { fresh: true })).status,
+ "ok",
+ );
+});
+
+test("watchdog: a video directory's read never answers mid-walk → readChannelStat answers null", async () => {
+ for (const id of ["vid2", "vid3", "vid4", "vid5", "vid6"]) {
+ mkdirSync(path.join(TARGET, id), { recursive: true });
+ }
+ const out = await watchdogCase(["readdir"], async () => {
+ // The walk's first readdir (of data/ itself, through the link) answers;
+ // the video directories' do not.
+ hang = (c) =>
+ c.fn === "readdir" && c.path.startsWith(path.join(LINK, "vid"));
+ return readChannelStat(paths, SLUG);
+ });
+ assert.equal(out, null);
+ // At most four video directories were asked before the stall refused the rest.
+ const asked = calls.filter((c) => c.fn === "readdir" && c.path.startsWith(path.join(LINK, "vid")));
+ assert.ok(asked.length <= 4, `${asked.length} video dirs asked`);
+});
+
+test("watchdog: probeLocation's stat never answers → 'stalled'", async () => {
+ const probe = (await watchdogCase(["stat"], () => probeLocation(LOC, paths))) as {
+ status: string;
+ };
+ assert.equal(probe.status, "stalled");
+});
+
+test("watchdog: volumeFreeBytes' stat never answers → unknown", async () => {
+ const out = (await watchdogCase(["stat"], () =>
+ volumeFreeBytes({ paths, locations: [LOC] }),
+ )) as Record<string, number | undefined>;
+ assert.equal(out.usb, undefined);
+});
+
+test("watchdog: a recency tail read never answers → not dated, not remembered as a miss", async () => {
+ const args = {
+ paths,
+ meta: [{ slug: SLUG, config: CONFIG }],
+ candidateIds: new Set(["vid1"]),
+ owner: new Map([["vid1", SLUG]]),
+ interpolate: false,
+ fresh: true,
+ };
+ const keys = (await watchdogCase(["open"], () => buildRecencyKeys(args))) as Map<
+ string,
+ { key: string }
+ >;
+ assert.equal(keys.get("vid1")?.key, "");
+ resetStorageHealth();
+ hang = null;
+ assert.equal((await buildRecencyKeys(args)).get("vid1")?.key, "20260601");
+});
+
+test("watchdog: a move onto a root whose stat never answers is refused", async () => {
+ const problem = await watchdogCase(["stat"], () =>
+ relocationRootPresenceProblem(
+ DRIVE,
+ { locations: [LOC], defaultLocationId: "" },
+ paths,
+ ),
+ );
+ assert.match(String(problem), /drive not answering/);
+});
+
+test("watchdog (M3): the snapshot walk's video unit never answers → the refresh throws and writes no snapshot", async () => {
+ for (const id of ["vid2", "vid3", "vid4", "vid5", "vid6", "vid7"]) {
+ mkdirSync(path.join(TARGET, id), { recursive: true });
+ }
+ const snapshotFile = path.join(paths.channelsDir, SLUG, "snapshot.json");
+ await watchdogCase(["readdir"], async () => {
+ // data/ itself answers (the listing, the reconcile pass); the video
+ // directories do not.
+ hang = (c) => c.fn === "readdir" && c.path.startsWith(path.join(LINK, "vid"));
+ await assert.rejects(
+ () => generateChannelSnapshot(paths, SLUG),
+ (err: unknown) => err instanceof Error && err.name === "DriveNotAnsweringError",
+ );
+ });
+ assert.equal(existsSync(snapshotFile), false, "the last snapshot.json stands");
+ // At most four video directories reached the drive.
+ const asked = calls.filter(
+ (c) => c.fn === "readdir" && c.path.startsWith(path.join(LINK, "vid")),
+ );
+ assert.ok(asked.length <= 4, `${asked.length} video dirs asked`);
+});
diff --git a/common/controller/storageWatch.test.ts b/common/controller/storageWatch.test.ts
@@ -6,19 +6,39 @@ import path from "node:path";
import type { Paths } from "../lib/paths";
import type { SiteSettings } from "../lib/settings";
import {
+ autoPauseReasonOf,
compileLanes,
sanitizeChannelPriority,
} from "../lib/channelPriority";
import { LANES } from "../lib/autoQueueTypes";
import {
+ refreshLocationHealth,
resetStorageWatchSuspicion,
+ runStorageHealthPass,
runStorageWatchPass,
+ startStorageHealthWatch,
+ startStorageWatch,
+ stopStorageHealthWatch,
+ stopStorageWatch,
} from "./storageWatch";
+import { inspectChannelMedia } from "../lib/channelMedia";
+import {
+ applyHealthTimings,
+ healthTimings,
+ locationHealth,
+ resetStorageHealth,
+ type LocationHealthState,
+} from "../lib/storageHealth";
// THE CONFIRMATION COUNT IS MODULE STATE (see storageWatch.ts rule 3), so each
// case starts from a clean one — otherwise the second test inherits the first
-// test's suspicions and pauses on what should be its first pass.
-beforeEach(() => resetStorageWatchSuspicion());
+// test's suspicions and pauses on what should be its first pass. The health
+// state (lib/storageHealth.ts) is process state for the same reason.
+beforeEach(() => {
+ resetStorageWatchSuspicion();
+ resetStorageHealth();
+ applyHealthTimings();
+});
// Run with:
// pnpm --filter yt-dlp-transcript-common exec tsx --test controller/storageWatch.test.ts
@@ -375,3 +395,267 @@ test("the restore needs only one good pass", async () => {
assert.equal(h.writes, 1);
});
});
+
+// ---------------------------------------------------------------------------
+// The health pass (15 s): is the drive ANSWERING
+// ---------------------------------------------------------------------------
+//
+// The probe is injected: `probeLocationHealth` (a child `stat` raced against a
+// 3 s timer) has its own tests in lib/storageHealthProbe.test.ts. Here the
+// answers are scripted, one per pass.
+
+function scripted(answers: LocationHealthState[]) {
+ let i = 0;
+ return async () => answers[Math.min(i++, answers.length - 1)];
+}
+
+test("one missed probe stalls the location; pages then answer 'stalled' without asking", async () => {
+ await withTmp(async (h) => {
+ await seedRelocated(h, "slow", { targetExists: true });
+ const lines: string[] = [];
+ const r = await runStorageHealthPass({
+ io: h.io,
+ probe: scripted(["stalled"]),
+ log: (l) => lines.push(l),
+ });
+ assert.deepEqual(r.answers, { cold: "stalled" });
+ // Registered first (answering), so the miss is a transition from ok.
+ assert.deepEqual(r.transitions, [{ id: "cold", from: "ok", to: "stalled" }]);
+ assert.equal(locationHealth("cold")?.state, "stalled");
+ assert.match(lines.join("\n"), /"cold": drive not answering/);
+ // The target is there and would answer, but nothing asks it.
+ const media = await inspectChannelMedia(h.paths, "slow");
+ assert.equal(media.status, "stalled");
+ // The five-minute pass sees the location as down (its probe answers
+ // "stalled" without a stat) and suspects the channel, as for any outage.
+ const w = await runStorageWatchPass({ paths: h.paths, io: h.io, bins: h.paths });
+ assert.deepEqual(w.suspected, ["slow"]);
+ assert.equal(h.writes, 0);
+ });
+});
+
+test("the stall clears only after two clean probes in a row", async () => {
+ await withTmp(async (h) => {
+ await seedRelocated(h, "slow", { targetExists: true });
+ const probe = scripted(["stalled", "ok", "stalled", "ok", "ok"]);
+ const lines: string[] = [];
+ const pass = () =>
+ runStorageHealthPass({ io: h.io, probe, log: (l) => lines.push(l) });
+ await pass();
+ await pass(); // one clean answer
+ assert.equal(locationHealth("cold")?.state, "stalled");
+ await pass(); // missed again: the count starts over
+ await pass(); // one clean
+ assert.equal(locationHealth("cold")?.state, "stalled");
+ assert.equal((await inspectChannelMedia(h.paths, "slow")).status, "stalled");
+ const last = await pass(); // two clean in a row
+ assert.deepEqual(last.transitions, [{ id: "cold", from: "stalled", to: "ok" }]);
+ assert.match(lines.at(-1) ?? "", /"cold": answering again/);
+ assert.equal((await inspectChannelMedia(h.paths, "slow")).status, "ok");
+ });
+});
+
+test("a location no longer configured is forgotten", async () => {
+ await withTmp(async (h) => {
+ await runStorageHealthPass({ io: h.io, probe: scripted(["stalled"]) });
+ assert.equal(locationHealth("cold")?.state, "stalled");
+ await runStorageHealthPass({ locations: [], probe: scripted(["ok"]) });
+ assert.equal(locationHealth("cold"), undefined);
+ });
+});
+
+test("a probe that throws is 'could not ask': ok, never stalled", async () => {
+ await withTmp(async (h) => {
+ const r = await runStorageHealthPass({
+ io: h.io,
+ probe: async () => {
+ throw new Error("spawn failed");
+ },
+ });
+ assert.deepEqual(r.answers, { cold: "ok" });
+ });
+});
+
+test("the health pass is armed on its own, runs once at once, and stops; the five-minute watch arms no health pass", async () => {
+ await withTmp(async (h) => {
+ let asked = 0;
+ const armed = startStorageHealthWatch({
+ io: h.io,
+ probe: async () => {
+ asked += 1;
+ return "stalled";
+ },
+ log: () => {},
+ });
+ try {
+ assert.equal(armed, true);
+ // Armed once per process.
+ assert.equal(startStorageHealthWatch({ io: h.io }), false);
+ for (let i = 0; i < 50 && locationHealth("cold")?.state !== "stalled"; i++) {
+ await new Promise((r) => setTimeout(r, 10));
+ }
+ assert.equal(asked, 1);
+ assert.equal(locationHealth("cold")?.state, "stalled");
+ } finally {
+ stopStorageHealthWatch();
+ }
+ assert.equal(startStorageHealthWatch({ io: h.io, probe: async () => "ok" }), true);
+ stopStorageHealthWatch();
+ // The watch (below the idle gate) runs nothing at arm time and asks no drive.
+ resetStorageHealth();
+ assert.equal(startStorageWatch({ paths: h.paths, io: h.io, bins: h.paths, write: false }), true);
+ stopStorageWatch();
+ assert.equal(locationHealth("cold"), undefined);
+ });
+});
+
+test("a Refresh asks one location now: it counts as one answer, and prunes nothing", async () => {
+ await withTmp(async (h) => {
+ const other = { id: "other", label: "Other", root: "/elsewhere", autoRepoint: false };
+ await runStorageHealthPass({
+ locations: [h.io.read().storage.locations[0], other],
+ probe: scripted(["stalled"]),
+ });
+ const cold = h.io.read().storage.locations[0];
+ assert.equal(await refreshLocationHealth(cold, async () => "ok"), "ok");
+ // One clean answer is not two.
+ assert.equal(locationHealth("cold")?.state, "stalled");
+ assert.equal(locationHealth("other")?.state, "stalled");
+ await refreshLocationHealth(cold, async () => "ok");
+ assert.equal(locationHealth("cold")?.state, "ok");
+ assert.equal(locationHealth("other")?.state, "stalled");
+ });
+});
+
+test("the pass registers every location, records a verdict's detector, and a verdict with no answer changes nothing", async () => {
+ await withTmp(async (h) => {
+ const verdicts = [
+ { answer: null, detector: "counters" as const, device: "sdz1" },
+ {
+ answer: "stalled" as const,
+ detector: "counters" as const,
+ device: "sdz1",
+ cause: "its disk (sdz1) had 1 request(s) in flight and completed none in 15 s",
+ },
+ ];
+ let i = 0;
+ const probe = async () => verdicts[i++];
+ const lines: string[] = [];
+ const first = await runStorageHealthPass({ io: h.io, probe, log: (l) => lines.push(l) });
+ // Registered, answering, and the detector named — with no verdict yet.
+ assert.deepEqual(first.answers, {});
+ assert.deepEqual(first.transitions, []);
+ assert.equal(locationHealth("cold")?.state, "ok");
+ assert.equal(locationHealth("cold")?.detector, "counters");
+ const second = await runStorageHealthPass({ io: h.io, probe, log: (l) => lines.push(l) });
+ assert.deepEqual(second.transitions, [{ id: "cold", from: "ok", to: "stalled" }]);
+ assert.match(String(locationHealth("cold")?.cause), /its disk \(sdz1\)/);
+ assert.match(lines.join("\n"), /"cold": drive not answering — its disk \(sdz1\)/);
+ });
+});
+
+test("a counters verdict records its device (for the watchdog); a stat verdict forgets it", async () => {
+ await withTmp(async (h) => {
+ const verdicts = [
+ { answer: null, detector: "counters" as const, device: "sdz1" },
+ { answer: "ok" as const, detector: "stat" as const },
+ ];
+ let i = 0;
+ const probe = async () => verdicts[i++];
+ await runStorageHealthPass({ io: h.io, probe });
+ assert.equal(locationHealth("cold")?.device, "sdz1");
+ await runStorageHealthPass({ io: h.io, probe });
+ assert.equal(locationHealth("cold")?.device, undefined);
+ assert.equal(locationHealth("cold")?.detector, "stat");
+ });
+});
+
+test("a stall auto-pauses after two passes, and says the drive is not answering (not that it is not there)", async () => {
+ await withTmp(async (h) => {
+ await seedRelocated(h, "slow", { targetExists: true });
+ await runStorageHealthPass({ io: h.io, probe: async () => "stalled" });
+ await twoPasses(h);
+ const entry = h.io.read().channelPriority.channels.slow;
+ assert.equal(entry?.tier, "paused");
+ assert.equal(entry?.autoPaused?.cause, "not-answering");
+ const reason = autoPauseReasonOf(h.io.read().channelPriority, "slow");
+ assert.match(String(reason), /drive that is not answering/);
+ assert.match(String(reason), /when the drive answers again/);
+ // The sanitizer keeps the cause; a record without one reads as not there.
+ const kept = sanitizeChannelPriority(h.io.read().channelPriority);
+ assert.equal(kept.channels.slow?.autoPaused?.cause, "not-answering");
+ const old = sanitizeChannelPriority({
+ channels: {
+ a: { tier: "paused", autoPaused: { reason: "storage", since: "", previousTier: "low" } },
+ },
+ });
+ assert.match(String(autoPauseReasonOf(old, "a")), /drive that is not there/);
+ });
+});
+
+// ── the timings are settings (release 15 slice DT) ─────────────────────────
+
+test("DT: every pass applies the timings it reads — the clear count and the log line follow storage.health", async () => {
+ await withTmp(async (h) => {
+ await seedRelocated(h, "slow", { targetExists: true });
+ h.io.read().storage.health = { clearAfterCleanPasses: 3, budgetMs: 5_000 };
+ const probe = scripted(["stalled", "ok", "ok", "ok"]);
+ const lines: string[] = [];
+ const pass = () => runStorageHealthPass({ io: h.io, probe, log: (l) => lines.push(l) });
+ await pass();
+ assert.equal(healthTimings().budgetMs, 5_000, "applied before anything was asked");
+ assert.match(lines.join("\n"), /until it answers 3 times in a row/);
+ await pass();
+ await pass();
+ assert.equal(locationHealth("cold")?.state, "stalled", "two clean passes are not three");
+ await pass();
+ assert.equal(locationHealth("cold")?.state, "ok");
+ // A pass handed its locations reads no settings: the timings stay.
+ await runStorageHealthPass({ locations: h.io.read().storage.locations, probe: scripted(["ok"]) });
+ assert.equal(healthTimings().clearAfterCleanPasses, 3);
+ });
+});
+
+test("DT: a changed pass interval re-arms the armed pass; an explicit interval follows nothing", async () => {
+ await withTmp(async (h) => {
+ let asked = 0;
+ const lines: string[] = [];
+ startStorageHealthWatch({
+ io: h.io,
+ probe: async () => {
+ asked += 1;
+ return "ok";
+ },
+ log: (l) => lines.push(l),
+ });
+ try {
+ assert.equal(asked, 1, "one pass at once");
+ // The save on /storage: written to settings, then applied at once.
+ h.io.read().storage.health = { passIntervalMs: 5_000 };
+ applyHealthTimings(h.io.read().storage.health);
+ assert.match(lines.join("\n"), /health pass re-armed: every 5 s/);
+ // At the default 15 s nothing would run for another 15 s; re-armed at
+ // 5 s, the next pass comes within about 5 s.
+ const started = Date.now();
+ while (asked < 2 && Date.now() - started < 7_000) {
+ await new Promise((r) => setTimeout(r, 50));
+ }
+ assert.equal(asked, 2, `a second pass after ${Date.now() - started} ms`);
+ assert.ok(Date.now() - started >= 4_500);
+ } finally {
+ stopStorageHealthWatch();
+ }
+ // Stopped: a later change re-arms nothing.
+ const before = lines.length;
+ applyHealthTimings({ passIntervalMs: 20_000 });
+ assert.equal(lines.length, before);
+ // Armed with an explicit interval, a change is not followed.
+ startStorageHealthWatch({ io: h.io, probe: async () => "ok", intervalMs: 60_000, log: (l) => lines.push(l) });
+ try {
+ applyHealthTimings({ passIntervalMs: 6_000 });
+ assert.equal(lines.some((l) => /re-armed/.test(l) && /6 s/.test(l)), false);
+ } finally {
+ stopStorageHealthWatch();
+ }
+ });
+});
diff --git a/common/controller/storageWatch.ts b/common/controller/storageWatch.ts
@@ -14,13 +14,35 @@ import {
resolveFocusSlugs,
restoreAfterMedia,
sanitizeChannelPriority,
+ type AutoPauseCause,
type ChannelPriority,
} from "../lib/channelPriority";
import { LANES } from "../lib/autoQueueTypes";
import { siteChannelIndex } from "../lib/site";
import { inspectChannelMedia } from "../lib/channelMedia";
-import { locationOfDataDir } from "../lib/storageLocations";
-import type { VolumeBins } from "../lib/storageVolumes";
+import {
+ locationOfDataDir,
+ type StorageLocation,
+} from "../lib/storageLocations";
+import {
+ detectLocationHealth,
+ type HealthVerdict,
+ type LocationHealthProbe,
+ type VolumeBins,
+} from "../lib/storageVolumes";
+import {
+ NOT_ANSWERING,
+ applyHealthTimings,
+ healthTimings,
+ noteLocationDetector,
+ onPassIntervalChange,
+ pruneLocationHealth,
+ recordLocationHealth,
+ registerLocationHealth,
+ type HealthTransition,
+ type LocationHealthState,
+} from "../lib/storageHealth";
+import { clearRuleText, secondsText } from "../lib/storageHealthTimings";
import { listChannelConfigs } from "./channels";
import { maybeAutoRepoint, probeAllLocations } from "./storageLocations";
@@ -185,6 +207,7 @@ export async function runStorageWatchPass(
const configs = await listChannelConfigs(paths);
let model: ChannelPriority = settings.channelPriority;
+ const pauseCauses = new Map<string, AutoPauseCause>();
for (const { slug, config } of configs) {
const wasAutoPaused = Boolean(model.channels[slug]?.autoPaused);
@@ -208,14 +231,28 @@ export async function runStorageWatchPass(
const loc = locationOfDataDir(dataDir, locations);
const probe = loc ? probes[loc.id] : undefined;
const locationDown = Boolean(loc) && probe?.status !== "available";
- const media = await inspectChannelMedia(paths, slug, config);
+ // FRESH: this pass is the detector, and the page memo is not what it asks.
+ // A location the health probe found not answering reads `stalled` here
+ // without a call, and a stall is `down` like any other.
+ const media = await inspectChannelMedia(paths, slug, config, {
+ fresh: true,
+ });
// `in-transition` is NEVER a reason to pause: a marker means a move is
// running or was interrupted, and the relocate job is precisely the thing
// that would then be refused by the state it created.
+ // A stall is down too, on a location or not (a root typed by hand gets its
+ // `stalled` from the watchdog alone).
const down =
media.status === "in-transition"
? false
- : locationDown || media.status === "unreachable";
+ : locationDown ||
+ media.status === "unreachable" ||
+ media.status === "stalled";
+ // WHICH down it is, for the pause record's words (autoPauseReasonOf).
+ const cause: AutoPauseCause =
+ media.status === "stalled" || probe?.status === "stalled"
+ ? "not-answering"
+ : "not-there";
if (down && !wasAutoPaused) {
// ONE BAD READ IS A SUSPICION, TWO IN A ROW IS A FACT. See rule 3.
@@ -230,7 +267,8 @@ export async function runStorageWatchPass(
continue;
}
const before = model;
- model = autoPauseForMedia(model, slug);
+ model = autoPauseForMedia(model, slug, new Date(), cause);
+ pauseCauses.set(slug, cause);
// autoPauseForMedia no-ops on a channel the OPERATOR already paused —
// which is right, and means "nothing changed" is a normal outcome here.
if (model !== before) {
@@ -286,7 +324,9 @@ export async function runStorageWatchPass(
)
: stored;
let merged: ChannelPriority = base;
- for (const slug of out.paused) merged = autoPauseForMedia(merged, slug);
+ for (const slug of out.paused) {
+ merged = autoPauseForMedia(merged, slug, new Date(), pauseCauses.get(slug));
+ }
for (const slug of out.restored) merged = restoreAfterMedia(merged, slug);
merged = sanitizeChannelPriority(merged);
// AND THE TREES, in the same write. Two writes to one settings file race each
@@ -308,23 +348,189 @@ export async function runStorageWatchPass(
return out;
}
-// ---------------------------------------------------------------------------
-// The cadence
-// ---------------------------------------------------------------------------
-
// Five minutes. A drive does not come and go on a timescale a person would
// notice faster than that, and every pass is one findmnt per location plus two
// stats per relocated channel — cheap, but not free, and this runs for the life
// of the process.
export const STORAGE_WATCH_INTERVAL_MS = 5 * 60_000;
+// ---------------------------------------------------------------------------
+// The health pass: is each location's drive ANSWERING
+// ---------------------------------------------------------------------------
+//
+// A SECOND CADENCE, AND A MUCH SHORTER ONE. The pass above asks "is the disk
+// here" every five minutes and pauses on two misses, which is right for a
+// cable pulled out. It is no help for a drive that is here and stalled — an
+// SMR disk in a USB enclosure resetting under a long write — because every
+// in-process call on that drive waits for it, and four waits stop the editor
+// answering at all. So every `storage.health.passIntervalMs` (15 s by default)
+// this reads each location's block device counters in /sys
+// (`detectLocationHealth`; a child `stat` of the root only where no device can
+// be named) and records the answer in `lib/storageHealth.ts`, which every page
+// and poll consults before it touches a drive. One `stalled` answer marks a
+// location at once; `clearAfterCleanPasses` clean answers in a row (two by
+// default) clear it (the rules are that module's).
+//
+// EVERY PASS APPLIES THE TIMINGS from the settings it reads (`applyHealthTimings`),
+// so a value changed by hand takes effect within one pass, and a changed
+// interval re-arms the pass's own timer (below).
+//
+// READ-ONLY AND IN MEMORY. It writes no settings and pauses nothing: the pass
+// above sees a stalled location as down (its probe answers `stalled` without
+// asking) and pauses on its own cadence. So, like the boot probe, it is armed
+// ABOVE the idle gate (`startStorageHealthWatch`, from instrumentation): an idle
+// boot has no five-minute pass, but it has the health pass — without it nothing
+// registers the locations, and a stall the watchdog marks is never cleared.
+
+export type StorageHealthPassOpts = {
+ // Default: the configured locations, read from settings.
+ locations?: readonly StorageLocation[];
+ io?: { read: () => SiteSettings };
+ // findmnt, for naming each root's block device. Default: getPaths().
+ bins?: Pick<VolumeBins, "findmntBin">;
+ // Test seam. Default: `detectLocationHealth` — the block device's counters,
+ // or a child `stat` against `probeTimeoutMs` when no device can be named.
+ probe?: LocationHealthProbe;
+ now?: () => number;
+ log?: (line: string) => void;
+};
+
+export type StorageHealthPassResult = {
+ probed: number;
+ answers: Record<string, LocationHealthState>;
+ // Only the locations whose state changed.
+ transitions: HealthTransition[];
+};
+
+// A probe's answer as a verdict. A bare state names no detector; a probe that
+// threw is "could not ask": `ok`, never `stalled`.
+function asVerdict(answer: LocationHealthState | HealthVerdict): HealthVerdict {
+ return typeof answer === "string" ? { answer } : answer;
+}
+
+function statCause(): string {
+ return `a stat of its root did not answer within ${secondsText(healthTimings().probeTimeoutMs)}`;
+}
+
+// Record one verdict. A verdict with no answer records nothing but the
+// detector that gave it.
+function recordVerdict(
+ loc: StorageLocation,
+ verdict: HealthVerdict,
+ now: number,
+): HealthTransition | null {
+ // The counters' device, for the watchdog's slow-or-stalled check; a stat
+ // verdict forgets it.
+ const device =
+ verdict.detector === "counters"
+ ? verdict.device
+ : verdict.detector === "stat"
+ ? null
+ : undefined;
+ if (verdict.answer === null) {
+ if (verdict.detector) noteLocationDetector(loc.id, verdict.detector, device);
+ return null;
+ }
+ return recordLocationHealth(loc, verdict.answer, {
+ now,
+ cause: verdict.cause ?? statCause(),
+ ...(verdict.detector ? { detector: verdict.detector } : {}),
+ ...(device !== undefined ? { device } : {}),
+ });
+}
+
+export async function runStorageHealthPass(
+ opts: StorageHealthPassOpts = {},
+): Promise<StorageHealthPassResult> {
+ const log = opts.log ?? (() => {});
+ // The settings this pass runs on: its locations, and the drive-health
+ // timings, applied before anything is asked. A caller that hands in the
+ // locations reads no settings, and the timings stay as they were.
+ const storage = opts.locations ? null : (opts.io ?? DEFAULT_IO).read().storage;
+ if (storage) applyHealthTimings(storage.health);
+ const locations = opts.locations ?? storage?.locations ?? [];
+ pruneLocationHealth(locations.map((l) => l.id));
+ // Every configured location has an entry before anything is asked, so the
+ // watchdog (lib/storageHealth.ts `onDrive`) can find a channel's location
+ // even before the counters have given a first verdict.
+ registerLocationHealth(locations);
+ const bins = opts.bins ?? getPaths();
+ const probe: LocationHealthProbe =
+ opts.probe ?? ((loc) => detectLocationHealth(loc, bins));
+ // Every location at once: each answer is bounded by the probe's own timer,
+ // so the pass is too, and one stalled drive does not delay the others.
+ const answers = await Promise.all(
+ locations.map(async (loc) => {
+ const verdict = await probe(loc).then(asVerdict, (): HealthVerdict => ({
+ answer: "ok",
+ }));
+ return [loc, verdict] as const;
+ }),
+ );
+ const out: StorageHealthPassResult = {
+ probed: answers.length,
+ answers: {},
+ transitions: [],
+ };
+ const now = opts.now?.() ?? Date.now();
+ for (const [loc, verdict] of answers) {
+ if (verdict.answer !== null) out.answers[loc.id] = verdict.answer;
+ const t = recordVerdict(loc, verdict, now);
+ if (!t) continue;
+ out.transitions.push(t);
+ if (t.to === "stalled") {
+ log(
+ `[storage] "${loc.id}": ${NOT_ANSWERING} — ${verdict.cause ?? statCause()}; ` +
+ `pages and polls skip it until it answers ` +
+ `${clearRuleText(healthTimings().clearAfterCleanPasses)}`,
+ );
+ } else if (t.from === "stalled") {
+ log(`[storage] "${loc.id}": answering again (${t.to})`);
+ }
+ }
+ return out;
+}
+
+// ONE LOCATION, NOW: what /storage's Refresh asks before its own probe, so the
+// operator pressing it after doing something about the drive gets an answer
+// taken afterwards. It counts as one answer like any other — a stalled location
+// still needs two clean ones in a row — and the counters give none when their
+// last sample is under `minCounterIntervalMs()` old. Nothing is pruned.
+export async function refreshLocationHealth(
+ loc: StorageLocation,
+ probe?: LocationHealthProbe,
+): Promise<LocationHealthState | null> {
+ registerLocationHealth([loc]);
+ const ask: LocationHealthProbe =
+ probe ?? ((l) => detectLocationHealth(l, getPaths()));
+ const verdict = await ask(loc).then(asVerdict, (): HealthVerdict => ({
+ answer: "ok",
+ }));
+ recordVerdict(loc, verdict, Date.now());
+ return verdict.answer;
+}
+
+// ---------------------------------------------------------------------------
+// The cadence
+// ---------------------------------------------------------------------------
+
+// `storage.health.passIntervalMs`, fifteen seconds by default
+// (lib/storageHealth.ts says why). Each pass reads each location's device
+// counters in /sys (a findmnt only when the device is not known yet or its /sys
+// entry stopped reading; with no device, one short-lived `stat`).
+
// A per-module-copy singleton, deliberately left so: it is not a temp-file
// name (slice W folded every tmp + rename onto lib/jsonFile-server.ts, whose
// state is on globalThis), and its one caller is editor/instrumentation.ts,
-// so only one copy ever arms it.
+// so only one copy ever arms it. (The health STATE the second timer writes is
+// on globalThis — pages in another module copy read it.)
let timer: ReturnType<typeof setInterval> | null = null;
+let healthTimer: ReturnType<typeof setInterval> | null = null;
+let healthInFlight = false;
+let stopFollowingInterval: (() => void) | null = null;
-// ARMED ONCE PER PROCESS. `unref()` so it never holds the event loop open — a
+// THE FIVE-MINUTE PASS, ARMED ONCE PER PROCESS, below the idle gate: it writes
+// settings (an auto-pause). `unref()` so it never holds the event loop open — a
// CLI that imports a controller must still exit.
export function startStorageWatch(
opts: StorageWatchOpts & { intervalMs?: number } = {},
@@ -343,7 +549,70 @@ export function startStorageWatch(
}
export function stopStorageWatch(): void {
- if (!timer) return;
- clearInterval(timer);
+ if (timer) clearInterval(timer);
timer = null;
}
+
+// THE HEALTH PASS, ARMED ONCE PER PROCESS, above the idle gate: it is in
+// memory and writes nothing (see the section header). It also runs once at
+// once, so the locations are registered and a drive that is already stalled is
+// on its way to being known before the first page.
+//
+// ITS INTERVAL FOLLOWS THE SETTING. Armed at `healthTimings().passIntervalMs`,
+// and RE-ARMED whenever an applied change moves it — a save on /storage (at
+// once, from the page's module copy: the subscription is on globalThis), or a
+// hand edit (at the next pass, which applies what it read). An explicit
+// `intervalMs` (the tests') is fixed and follows nothing.
+export function startStorageHealthWatch(
+ opts: {
+ io?: { read: () => SiteSettings };
+ bins?: Pick<VolumeBins, "findmntBin">;
+ probe?: LocationHealthProbe;
+ intervalMs?: number;
+ log?: (line: string) => void;
+ } = {},
+): boolean {
+ if (healthTimer) return false;
+ const health = () => {
+ // One at a time: a pass is bounded by its timers, but a pass that overran
+ // the interval must not stack a second one on top of it.
+ if (healthInFlight) return;
+ healthInFlight = true;
+ void runStorageHealthPass({
+ io: opts.io,
+ bins: opts.bins,
+ probe: opts.probe,
+ log: opts.log,
+ })
+ .catch((err) => {
+ (opts.log ?? console.warn)(
+ `[storage] health pass failed: ${(err as Error).message}`,
+ );
+ })
+ .finally(() => {
+ healthInFlight = false;
+ });
+ };
+ const arm = (every: number) => {
+ if (healthTimer) clearInterval(healthTimer);
+ healthTimer = setInterval(health, every);
+ healthTimer.unref?.();
+ };
+ arm(opts.intervalMs ?? healthTimings().passIntervalMs);
+ if (opts.intervalMs === undefined) {
+ stopFollowingInterval = onPassIntervalChange((every) => {
+ if (!healthTimer) return;
+ (opts.log ?? console.log)(`[storage] health pass re-armed: every ${secondsText(every)}`);
+ arm(every);
+ });
+ }
+ health();
+ return true;
+}
+
+export function stopStorageHealthWatch(): void {
+ if (healthTimer) clearInterval(healthTimer);
+ healthTimer = null;
+ stopFollowingInterval?.();
+ stopFollowingInterval = null;
+}
diff --git a/common/lib/accent.test.ts b/common/lib/accent.test.ts
@@ -45,7 +45,6 @@ test("resolveAccent: a named accent reads its table row", () => {
assert.deepEqual(resolveAccent(id), {
id,
light: ACCENTS[id].onLight,
- sepia: ACCENTS[id].onSepia,
dark: ACCENTS[id].onDark,
});
}
@@ -58,7 +57,7 @@ test("resolveAccent: absent or malformed is the default accent (Signal)", () =>
}
});
-// Custom hexes chosen to need every kind of fit: too light for light/sepia,
+// Custom hexes chosen to need every kind of fit: too light for light,
// too dark for dark, already fine, and the extremes.
const CUSTOMS = [
"#cc3366", "#ffff00", "#00ffff", "#ffffff", "#000000", "#808080", "#1e90ff",
@@ -83,16 +82,16 @@ test("resolveAccent: a custom hex is fitted to 4.5:1 on every ground and with it
});
test("resolveAccent: a custom hex that already passes is kept exactly; others move the right way", () => {
- // #cc3366 is 4.57:1 on light (kept), 4.21 on sepia (darkened), 3.98 on dark
- // (lightened). The themes slice's site-branding spec expects #cc3366 on light.
+ // #cc3366 is 4.57:1 on light (kept) and 3.98 on dark (lightened). The
+ // themes slice's site-branding spec expects #cc3366 on light.
const r = resolveAccent("#CC3366");
assert.equal(r.light, "#cc3366");
- assert.notEqual(r.sepia, "#cc3366");
assert.notEqual(r.dark, "#cc3366");
const lum = (h: string) => contrastRatio(h, "#000000");
- assert.ok(lum(r.sepia) < lum("#cc3366"), "sepia is darker");
assert.ok(lum(r.dark) > lum("#cc3366"), "dark is lighter");
- // A dark-enough colour is kept on light and sepia, a light-enough on dark.
+ // The one that is too light for light is darkened there.
+ assert.ok(lum(resolveAccent("#ffff00").light) < lum("#ffff00"), "light is darker");
+ // A dark-enough colour is kept on light, a light-enough on dark.
assert.equal(resolveAccent("#101010").light, "#101010");
assert.equal(resolveAccent("#f0f0f0").dark, "#f0f0f0");
});
@@ -123,7 +122,6 @@ test("customAccentVars: only a custom hex carries inline vars", () => {
const r = resolveAccent("#cc3366");
assert.deepEqual(customAccentVars("#cc3366"), {
"--accent-custom-light": r.light,
- "--accent-custom-sepia": r.sepia,
"--accent-custom-dark": r.dark,
});
});
diff --git a/common/lib/accent.ts b/common/lib/accent.ts
@@ -51,7 +51,6 @@ export function parseAccentSetting(input: unknown): string | undefined {
export type ResolvedAccent = {
id: AccentId | "custom";
light: string;
- sepia: string;
dark: string;
};
@@ -68,7 +67,7 @@ function meetsRule(hex: string, base: BaseGround): boolean {
// Fit a custom hex to one base: unchanged when it already meets the rule
// (brand.ts MIN_ACCENT_CONTRAST against the ground AND the ink), otherwise
-// mixed toward black (light/sepia) or white (dark) in 1 % steps until it does.
+// mixed toward black (light) or white (dark) in 1 % steps until it does.
// Mixing keeps the hue; full black/white always passes, so this terminates.
function fitAccent(hex: string, base: BaseGround): string {
if (meetsRule(hex, base)) return hex;
@@ -93,12 +92,11 @@ export function resolveAccent(input: unknown): ResolvedAccent {
return {
id: "custom",
light: fitAccent(setting, "light"),
- sepia: fitAccent(setting, "sepia"),
dark: fitAccent(setting, "dark"),
};
}
const a = ACCENTS[setting && isAccentId(setting) ? setting : DEFAULT_ACCENT];
- return { id: a.id, light: a.onLight, sepia: a.onSepia, dark: a.onDark };
+ return { id: a.id, light: a.onLight, dark: a.onDark };
}
// The PUBLISHED accent: always a hex. An id becomes its on-dark value (the
diff --git a/common/lib/brand.test.ts b/common/lib/brand.test.ts
@@ -21,9 +21,8 @@ import {
} from "./brand";
import { PROJECT_NAME, PROJECT_WORDMARK_LEAD } from "./project";
-const ON: Record<BaseGround, "onLight" | "onSepia" | "onDark"> = {
+const ON: Record<BaseGround, "onLight" | "onDark"> = {
light: "onLight",
- sepia: "onSepia",
dark: "onDark",
};
@@ -31,7 +30,8 @@ test("accents: the table is the plan's, keyed and ordered by ACCENT_IDS", () =>
assert.deepEqual(Object.keys(ACCENTS), [...ACCENT_IDS]);
for (const id of ACCENT_IDS) {
assert.equal(ACCENTS[id].id, id);
- for (const k of ["onDark", "onLight", "onSepia"] as const) {
+ assert.deepEqual(Object.keys(ACCENTS[id]).sort(), ["id", "name", "onDark", "onLight"]);
+ for (const k of ["onDark", "onLight"] as const) {
assert.match(ACCENTS[id][k], /^#[0-9a-f]{6}$/, `${id}.${k}`);
}
}
@@ -39,13 +39,13 @@ test("accents: the table is the plan's, keyed and ordered by ACCENT_IDS", () =>
// The whole table, literally, against plans/brand-and-themes.md "Accents":
// a typo that still clears 4.5:1 would pass the contrast test below.
assert.deepEqual(ACCENTS, {
- signal: { id: "signal", name: "Signal", onDark: "#5fa8a0", onLight: "#2e7b73", onSepia: "#2b756e" },
- brass: { id: "brass", name: "Brass", onDark: "#e3b15c", onLight: "#95661a", onSepia: "#8e6119" },
- vermilion: { id: "vermilion", name: "Vermilion", onDark: "#ec7a52", onLight: "#b3431f", onSepia: "#b3431f" },
- violet: { id: "violet", name: "Violet", onDark: "#b49cf2", onLight: "#6a4bc4", onSepia: "#6a4bc4" },
- sakura: { id: "sakura", name: "Sakura", onDark: "#ee8fb5", onLight: "#a83a6a", onSepia: "#a83a6a" },
- blue: { id: "blue", name: "Blue", onDark: "#74a9f2", onLight: "#2d5fb8", onSepia: "#2d5fb8" },
- green: { id: "green", name: "Green", onDark: "#7cc46a", onLight: "#3f7a2c", onSepia: "#3d772b" },
+ signal: { id: "signal", name: "Signal", onDark: "#5fa8a0", onLight: "#2e7b73" },
+ brass: { id: "brass", name: "Brass", onDark: "#e3b15c", onLight: "#95661a" },
+ vermilion: { id: "vermilion", name: "Vermilion", onDark: "#ec7a52", onLight: "#b3431f" },
+ violet: { id: "violet", name: "Violet", onDark: "#b49cf2", onLight: "#6a4bc4" },
+ sakura: { id: "sakura", name: "Sakura", onDark: "#ee8fb5", onLight: "#a83a6a" },
+ blue: { id: "blue", name: "Blue", onDark: "#74a9f2", onLight: "#2d5fb8" },
+ green: { id: "green", name: "Green", onDark: "#7cc46a", onLight: "#3f7a2c" },
});
assert.equal(isAccentId("brass"), true);
assert.equal(isAccentId("Brass"), false);
@@ -53,8 +53,8 @@ test("accents: the table is the plan's, keyed and ordered by ACCENT_IDS", () =>
});
// THE CONTRAST RULE: every accent's value on every base reaches 4.5:1 against
-// that base's ground AND against the ink set on it (white on light/sepia, the
-// dark ground on dark). 7 accents × 3 bases × 2 = 42 checks.
+// that base's ground AND against the ink set on it (white on light, the dark
+// ground on dark). 7 accents × 2 bases × 2 = 28 checks.
test("accents: every value reaches 4.5:1 on its ground and with its ink", () => {
// The bar itself is pinned as a literal: comparing only against the
// imported constant would let an edit to brand.ts lower rule and test
@@ -71,9 +71,8 @@ test("accents: every value reaches 4.5:1 on its ground and with its ink", () =>
checks += 2;
}
}
- assert.equal(checks, 42);
+ assert.equal(checks, 28);
assert.equal(ACCENT_INK.light, "#ffffff");
- assert.equal(ACCENT_INK.sepia, "#ffffff");
assert.equal(ACCENT_INK.dark, BASE_GROUNDS.dark);
});
@@ -94,7 +93,7 @@ test("contrastRatio: the sRGB curve, at the 4.5:1 boundary on white", () => {
});
test("bases and icon palettes are the plan's", () => {
- assert.deepEqual(BASE_GROUNDS, { light: "#f3f6f7", sepia: "#f4ecd8", dark: "#0c0a08" });
+ assert.deepEqual(BASE_GROUNDS, { light: "#f3f6f7", dark: "#0c0a08" });
assert.deepEqual(ICON_PALETTES.archilyzer, { ground: "#151b20", dim: "#586977", lit: "#e7edf1" });
assert.deepEqual(childIconPalette(ACCENTS.brass.onDark), {
ground: "#0c0a08",
@@ -104,9 +103,8 @@ test("bases and icon palettes are the plan's", () => {
});
// The mark's two rules (brand.ts MIN_MARK_DIM_CONTRAST, MIN_MARK_LIT_OVER_DIM).
-// A child site can be lit by any named accent — its own, Signal when it has
-// none, or the one a reader picks, which the header mark follows — so the
-// child dim is held against all seven. The parent mark is lit in bone; the
+// A child site can be lit by any named accent — its own, or Signal when it
+// has none — so the child dim is held against all seven. The parent mark is lit in bone; the
// Media mark's Signal on the same slate is held in brandMedia.test.ts.
const LIT_OF: ReadonlyArray<[string, string, string]> = [
...ACCENT_IDS.map((id): [string, string, string] => [`child · ${id}`, ICON_PALETTES.child.dim, ACCENTS[id].onDark]),
diff --git a/common/lib/brand.ts b/common/lib/brand.ts
@@ -1,7 +1,7 @@
-// THE BRAND, AS DATA — the Found-line mark, the accent palette, the three base
+// THE BRAND, AS DATA — the Found-line mark, the accent palette, the two base
// grounds and the wordmark split. plans/brand-and-themes.md "The design" is the
// source of every value here; where the design canvas and the plan differ, the
-// plan wins (notably the contrast-corrected on-light / on-sepia accents).
+// plan wins (notably the contrast-corrected on-light accents).
//
// PURE: zero imports, no I/O, no framework types, no zod. It is safe in server
// components, `"use client"` trees (the editor's accent swatches), route
@@ -31,17 +31,16 @@ export type Accent = {
name: string;
onDark: string;
onLight: string;
- onSepia: string;
};
export const ACCENTS: Readonly<Record<AccentId, Accent>> = {
- signal: { id: "signal", name: "Signal", onDark: "#5fa8a0", onLight: "#2e7b73", onSepia: "#2b756e" },
- brass: { id: "brass", name: "Brass", onDark: "#e3b15c", onLight: "#95661a", onSepia: "#8e6119" },
- vermilion: { id: "vermilion", name: "Vermilion", onDark: "#ec7a52", onLight: "#b3431f", onSepia: "#b3431f" },
- violet: { id: "violet", name: "Violet", onDark: "#b49cf2", onLight: "#6a4bc4", onSepia: "#6a4bc4" },
- sakura: { id: "sakura", name: "Sakura", onDark: "#ee8fb5", onLight: "#a83a6a", onSepia: "#a83a6a" },
- blue: { id: "blue", name: "Blue", onDark: "#74a9f2", onLight: "#2d5fb8", onSepia: "#2d5fb8" },
- green: { id: "green", name: "Green", onDark: "#7cc46a", onLight: "#3f7a2c", onSepia: "#3d772b" },
+ signal: { id: "signal", name: "Signal", onDark: "#5fa8a0", onLight: "#2e7b73" },
+ brass: { id: "brass", name: "Brass", onDark: "#e3b15c", onLight: "#95661a" },
+ vermilion: { id: "vermilion", name: "Vermilion", onDark: "#ec7a52", onLight: "#b3431f" },
+ violet: { id: "violet", name: "Violet", onDark: "#b49cf2", onLight: "#6a4bc4" },
+ sakura: { id: "sakura", name: "Sakura", onDark: "#ee8fb5", onLight: "#a83a6a" },
+ blue: { id: "blue", name: "Blue", onDark: "#74a9f2", onLight: "#2d5fb8" },
+ green: { id: "green", name: "Green", onDark: "#7cc46a", onLight: "#3f7a2c" },
};
// An absent accent reads as this one — on a child site AND on the family's own
@@ -58,19 +57,17 @@ export function isAccentId(v: unknown): v is AccentId {
// against these, and the browser chrome colour is taken from them.
export const BASE_GROUNDS = {
light: "#f3f6f7",
- sepia: "#f4ecd8",
dark: "#0c0a08",
} as const;
export type BaseGround = keyof typeof BASE_GROUNDS;
-export const BASE_GROUND_IDS = ["light", "sepia", "dark"] as const satisfies ReadonlyArray<BaseGround>;
+export const BASE_GROUND_IDS = ["light", "dark"] as const satisfies ReadonlyArray<BaseGround>;
// The text set ON an accent fill (a filled button, a badge): white on the light
-// and sepia bases, the dark ground on the dark base.
+// base, the dark ground on the dark base.
export const ACCENT_INK: Readonly<Record<BaseGround, string>> = {
light: "#ffffff",
- sepia: "#ffffff",
dark: BASE_GROUNDS.dark,
};
diff --git a/common/lib/channelMedia.ts b/common/lib/channelMedia.ts
@@ -2,6 +2,16 @@ import path from "node:path";
import { lstat, readFile, readlink, rm, stat } from "node:fs/promises";
import type { Paths } from "./paths";
import type { ChannelConfig } from "./channelConfig";
+import {
+ NOT_ANSWERING,
+ healthTimings,
+ isDriveNotAnswering,
+ onDrive,
+ sinceText,
+ stalledLocationForPath,
+ type LocationHealth,
+} from "./storageHealth";
+import { secondsText } from "./storageHealthTimings";
// WHERE A CHANNEL'S MEDIA ACTUALLY IS, and whether it can be reached.
//
@@ -60,7 +70,12 @@ export type ChannelMediaStatus =
// A relocation is in flight (or was interrupted): the marker is present.
| "in-transition"
// Disk and config disagree, in either direction. Never guessed past.
- | "inconsistent";
+ | "inconsistent"
+ // Relocated onto a storage location whose drive is not answering
+ // (`lib/storageHealth.ts`). Answered from memory, WITHOUT a filesystem call:
+ // a call there would block one of the process's few I/O threads for as long
+ // as the drive takes to come back. Held and refused like `unreachable`.
+ | "stalled";
export type ChannelMediaLocation = {
// Always channelDir/data — the path every reader uses, relocated or not.
@@ -162,6 +177,7 @@ export async function clearRelocationMarker(
slug: string,
): Promise<void> {
await rm(relocationMarkerPath(paths, slug), { force: true });
+ forgetChannelMedia(slug);
}
// The `dataDir` field alone, read straight off config.json. Deliberately NOT
@@ -185,13 +201,124 @@ async function readConfiguredDataDir(
}
}
+// The configured target's drive is not answering: the answer, from memory. The
+// detail names the location and when it stopped, never a path — the badge's
+// title shows it, and /storage has the paths.
+export function stalledMediaLocation(
+ dataDir: string,
+ configured: string,
+ // Null when the drive is on no location the health state knows (a root typed
+ // by hand) and the watchdog found it not answering, or when the watchdog
+ // found it slow rather than stalled.
+ health: LocationHealth | null,
+ // The watchdog's own words, for a refusal that marked no location.
+ detail?: string,
+): ChannelMediaLocation {
+ return {
+ dataDir,
+ relocated: true,
+ target: configured,
+ status: "stalled",
+ detail: health
+ ? `${NOT_ANSWERING} (location "${health.label}", ${sinceText(health.since)})`
+ : (detail ??
+ `${NOT_ANSWERING} (a read did not answer within ${secondsText(healthTimings().budgetMs)})`),
+ };
+}
+
+// THE STALL, for a caller holding a parsed config: the stalled location the
+// channel's media is on, or null. No I/O — the question every page and poll
+// that reads a channel's `data/` asks before it does.
+export function channelMediaStall(
+ config: Pick<ChannelConfig, "dataDir"> | null | undefined,
+): LocationHealth | null {
+ const dir = config?.dataDir?.trim();
+ return dir ? stalledLocationForPath(dir) : null;
+}
+
+// ---------------------------------------------------------------------------
+// The memo
+// ---------------------------------------------------------------------------
+//
+// FIVE SECONDS, PER CHANNEL, KEYED BY SLUG AND THE CONFIGURED TARGET. The home
+// page, /channels and the auto-queue status poll (every three seconds, four
+// lanes) each inspect every channel; without this each of them costs three
+// syscalls a relocated channel, every time, on a drive that may be the slow
+// one. The key carries the configured `dataDir`, so a move that rewrites it is
+// a new key at once, and the movers clear the memo outright
+// (`forgetChannelMedia`) whenever a marker is written or removed.
+//
+// THE STALL GATE IS ASKED BEFORE A REMEMBERED ANSWER IS GIVEN, so a drive that
+// stops answering is seen on the next call even when an `ok` from four seconds
+// ago is remembered. A remembered `in-transition` is given as it is: the
+// marker is asked before the gate (below), and the movers forget the channel
+// whenever they write or remove it.
+//
+// `fresh: true` BYPASSES IT, and every caller that decides something from the
+// answer passes it: the start-of-work guard (`assertChannelMediaReachable`),
+// the movers, the index and stats builds, the storage watch, eviction and the
+// re-point preflight. The memo is for pages and polls.
+//
+// ONE MAP PER PROCESS (on `globalThis`), for the reason `storageHealth.ts`
+// gives: the movers that clear it run in one bundle layer and the pages that
+// read it in another.
+
+export const CHANNEL_MEDIA_MEMO_MS = 5_000;
+
+export type InspectOptions = {
+ // Skip the memo: take a fresh answer, and do not remember it.
+ fresh?: boolean;
+ // Test seam for the clock.
+ now?: number;
+};
+
+type MediaMemo = Map<string, { at: number; location: ChannelMediaLocation }>;
+
+declare global {
+ // eslint-disable-next-line no-var
+ var __yttChannelMediaMemo__: MediaMemo | undefined;
+}
+
+function mediaMemo(): MediaMemo {
+ if (!globalThis.__yttChannelMediaMemo__) {
+ globalThis.__yttChannelMediaMemo__ = new Map();
+ }
+ return globalThis.__yttChannelMediaMemo__;
+}
+
+// Forget what the memo holds: for one channel, or for every channel. The
+// movers call it whenever the disk changes under a channel (a marker written or
+// removed, a link swapped, a location re-pointed), so the next page sees it.
+export function forgetChannelMedia(slug?: string): void {
+ const memo = mediaMemo();
+ if (slug === undefined) {
+ memo.clear();
+ return;
+ }
+ for (const key of [...memo.keys()]) {
+ if (key.split("\u0000")[1] === slug) memo.delete(key);
+ }
+}
+
+function memoKey(paths: ChannelMediaPaths, slug: string, configured?: string): string {
+ return `${paths.channelsDir}\u0000${slug}\u0000${configured ?? ""}`;
+}
+
+// Past this many entries, expired ones are swept on insert. A corpus has tens
+// of channels; this only matters to a process that inspects many corpora.
+const MEMO_SWEEP_AT = 512;
+
// Two stats and (at most) one small JSON read. Render-safe: nothing here walks a
// directory, so calling it per channel on a listing page costs three syscalls a
-// row.
+// row — or none, for five seconds after the last answer (see the memo above).
+// On a stalled location the one call that reaches the drive (the target's
+// `stat`) is not made; the marker, the link and config.json are on the corpus
+// disk and are read as usual.
export async function inspectChannelMedia(
paths: ChannelMediaPaths,
slug: string,
config?: Pick<ChannelConfig, "dataDir"> | null,
+ opts: InspectOptions = {},
): Promise<ChannelMediaLocation> {
const dataDir = channelMediaDir(paths, slug);
const configured =
@@ -201,6 +328,42 @@ export async function inspectChannelMedia(
? config.dataDir.trim()
: undefined;
+ const now = opts.now ?? Date.now();
+ const key = memoKey(paths, slug, configured);
+ const memo = mediaMemo();
+ if (!opts.fresh) {
+ const hit = memo.get(key);
+ if (hit && now - hit.at < CHANNEL_MEDIA_MEMO_MS) {
+ if (hit.location.status !== "in-transition" && configured) {
+ const stall = stalledLocationForPath(configured);
+ if (stall) return stalledMediaLocation(dataDir, configured, stall);
+ }
+ return { ...hit.location };
+ }
+ }
+ const location = await inspectOnDisk(paths, slug, dataDir, configured);
+ // A stall is not remembered: the health state is already its memory.
+ if (!opts.fresh && location.status !== "stalled") {
+ if (memo.size >= MEMO_SWEEP_AT) {
+ for (const [k, v] of memo) {
+ if (now - v.at >= CHANNEL_MEDIA_MEMO_MS) memo.delete(k);
+ }
+ }
+ memo.set(key, { at: now, location });
+ }
+ return { ...location };
+}
+
+async function inspectOnDisk(
+ paths: ChannelMediaPaths,
+ slug: string,
+ dataDir: string,
+ configured: string | undefined,
+): Promise<ChannelMediaLocation> {
+ // THE MARKER FIRST. It is in the channel dir, on the corpus disk, so reading
+ // it costs the drive nothing — and a channel mid-move reads `in-transition`
+ // whatever its drive is doing, which is what a resumed move and every guard
+ // key off.
const marker = await readRelocationMarker(paths, slug);
if (marker) {
return {
@@ -215,6 +378,15 @@ export async function inspectChannelMedia(
};
}
+ // THE GATE: a channel whose configured target is on a location whose drive
+ // is not answering is answered from memory, before the link is looked at and
+ // before the target's stat, which would hold an I/O thread for as long as the
+ // drive takes.
+ if (configured) {
+ const stall = stalledLocationForPath(configured);
+ if (stall) return stalledMediaLocation(dataDir, configured, stall);
+ }
+
let link: Awaited<ReturnType<typeof lstat>> | null = null;
try {
link = await lstat(dataDir);
@@ -266,8 +438,13 @@ export async function inspectChannelMedia(
// The link points at a DEEP path (<root>/<slug>/data), so an unmounted root
// gives ENOENT here. An empty mountpoint can never be mistaken for the
// media, which is the whole reason the suffix is fixed.
+ //
+ // THE ONE CALL HERE THAT REACHES THE DRIVE, so it goes through the
+ // watchdog: not made while the location is stalled, and a stat that has
+ // not answered within the budget (`storage.health.budgetMs`, 3 s by
+ // default) marks it stalled and answers `stalled` now.
try {
- const st = await stat(configured);
+ const st = await onDrive(configured, () => stat(configured));
if (!st.isDirectory()) {
return {
dataDir,
@@ -277,7 +454,10 @@ export async function inspectChannelMedia(
detail: `${configured} exists but is not a directory`,
};
}
- } catch {
+ } catch (err) {
+ if (isDriveNotAnswering(err)) {
+ return stalledMediaLocation(dataDir, configured, err.health, err.message);
+ }
return {
dataDir,
relocated: true,
@@ -316,6 +496,10 @@ export async function inspectChannelMedia(
// "ok" and "in-place" pass; everything else throws. An in-transition or
// inconsistent channel is refused for the same reason an unreachable one is:
// the caller would otherwise read a half-populated or empty dir as the truth.
+// A stalled one is refused because the work would block on the drive.
+//
+// ALWAYS FRESH: this is the start-of-work guard, and a remembered "ok" from a
+// few seconds ago is not what a job about to read `data/` should be told.
//
// THIS CHECK HAS A TWIN. `checkChannelReachable` in
// `umtool/report-to-video/cues.mjs` repeats the same statuses in plain `.mjs`,
@@ -329,7 +513,9 @@ export async function assertChannelMediaReachable(
slug: string,
config?: Pick<ChannelConfig, "dataDir"> | null,
): Promise<ChannelMediaLocation> {
- const location = await inspectChannelMedia(paths, slug, config);
+ const location = await inspectChannelMedia(paths, slug, config, {
+ fresh: true,
+ });
if (location.status === "ok" || location.status === "in-place") {
return location;
}
diff --git a/common/lib/channelMediaHold.ts b/common/lib/channelMediaHold.ts
@@ -0,0 +1,62 @@
+import type {
+ ChannelMediaLocation,
+ ChannelMediaStatus,
+} from "./channelMedia";
+import {
+ locationLabelOfDataDir,
+ type StorageLocation,
+} from "./storageLocations";
+
+// THE HOLD, in the words both pool-wide builds use.
+//
+// The index build (controller/buildIndex.ts) and the stats build
+// (controller/buildStats.ts) each walk every channel's `data/`. A channel whose
+// media cannot be read — a relocated `data/` on an unmounted drive, a move in
+// progress, a link and a config that disagree — is HELD by both: not rescanned,
+// and what the last build knew of it kept, rather than read as a channel with
+// no videos and removed (lib/channelMedia.ts says why that reading is the
+// dangerous one). This module is only the shared vocabulary: which statuses
+// hold, why, in words with no path in them (/storage shows the paths), and the
+// ways out a refusal names. Each build decides for itself what "kept" means.
+//
+// Pure: no I/O. The caller asks inspectChannelMedia and passes the answer in.
+
+// "ok" and "in-place" are read; every other status holds, including any a later
+// inspectChannelMedia adds.
+export function isMediaHeld(status: ChannelMediaStatus): boolean {
+ return status !== "ok" && status !== "in-place";
+}
+
+// Why a channel is held, without the paths inspectChannelMedia's `detail`
+// carries.
+export const HELD_REASON: Record<ChannelMediaStatus, string> = {
+ unreachable: "its media is not reachable (drive not mounted?)",
+ "in-transition": "a move of its media is in progress or was interrupted",
+ inconsistent: "its data link and its config disagree",
+ stalled: "its drive is not answering (a stalled disk)",
+ ok: "reachable",
+ "in-place": "reachable",
+};
+
+// The reason, and the storage location's label when the channel's media is on
+// one. `dataDir` is the channel config's; the inspector's target stands in when
+// the config names none (a move in flight).
+export function heldReason(
+ media: Pick<ChannelMediaLocation, "status" | "target">,
+ dataDir: string | undefined,
+ locations: StorageLocation[],
+): string {
+ const label = locationLabelOfDataDir(dataDir ?? media.target, locations);
+ return `${HELD_REASON[media.status]}${label ? `, on location "${label}"` : ""}`;
+}
+
+// A held channel named in a refusal, as `slug (why)`, joined.
+export function describeHeld(held: Map<string, string>): string {
+ return [...held].map(([slug, why]) => `${slug} (${why})`).join("; ");
+}
+
+// What a refusal tells the operator to do, mounting first.
+export const HELD_WAYS_OUT =
+ `For each: mount its media and run this again; or repair or re-point its location on /storage; ` +
+ `or finish or clear its move (the channel's Storage panel); or, if it is gone for good, ` +
+ `delete the channel or set excludeFromBuild in its config.`;
diff --git a/common/lib/channelPriority.ts b/common/lib/channelPriority.ts
@@ -183,8 +183,15 @@ export type ChannelAutoPause = {
reason: "storage";
since: string;
previousTier: StoredChannelTier;
+ cause?: AutoPauseCause;
};
+// Which storage trouble paused it: the drive is not there (unmounted,
+// unplugged), or it is there and not answering (lib/storageHealth.ts). A record
+// with none was written before the second existed, when the first was the only
+// one.
+export type AutoPauseCause = "not-there" | "not-answering";
+
export const CHANNEL_AUTO_PAUSE_FIELD_DOCS: FieldDocs<ChannelAutoPause> = {
reason:
"One reason today. A union so a second one has somewhere to go, and so " +
@@ -195,6 +202,11 @@ export const CHANNEL_AUTO_PAUSE_FIELD_DOCS: FieldDocs<ChannelAutoPause> = {
"The base tier the channel had before the machine paused it; what a " +
"restore puts back. Never `paused` (that would restore to paused — a " +
"no-op dressed as a restore).",
+ cause:
+ "`not-there` (the drive is unmounted or unplugged) or `not-answering` (it " +
+ "is there and does not answer: a stalled disk). Only what the words on " +
+ "/review, the rack and the channel page say. Optional: a record written " +
+ "before it existed reads as `not-there`.",
};
// Each field is documented in CHANNEL_PRIORITY_FIELD_DOCS below (rendered into SETTINGS.md).
@@ -387,12 +399,16 @@ function sanitizeAutoPause(
reason: "storage",
since: typeof r.since === "string" ? r.since : "",
previousTier: previousTier === "paused" ? DEFAULT_CHANNEL_TIER : previousTier,
+ ...(r.cause === "not-there" || r.cause === "not-answering"
+ ? { cause: r.cause }
+ : {}),
};
}
// --- Auto-pause: the machine's own pause, and its undo ----------------------
-// PAUSE A CHANNEL BECAUSE ITS DRIVE IS NOT THERE, recording what to put back.
+// PAUSE A CHANNEL BECAUSE ITS DRIVE IS NOT THERE (OR NOT ANSWERING), recording
+// what to put back, and which of the two it was.
//
// A NO-OP IN TWO CASES, and both matter. Already auto-paused: a flapping drive
// must not overwrite `previousTier` with the `paused` it wrote last time, which
@@ -403,6 +419,7 @@ export function autoPauseForMedia(
model: ChannelPriority,
slug: string,
now: Date = new Date(),
+ cause?: AutoPauseCause,
): ChannelPriority {
const key = slug.trim();
if (!key) return model;
@@ -421,6 +438,7 @@ export function autoPauseForMedia(
reason: "storage",
since: now.toISOString(),
previousTier,
+ ...(cause ? { cause } : {}),
},
},
},
@@ -466,6 +484,12 @@ export function autoPauseReasonOf(
const auto = model.channels[slug]?.autoPaused;
if (!auto) return null;
const since = auto.since ? ` since ${auto.since.slice(0, 10)}` : "";
+ if (auto.cause === "not-answering") {
+ return (
+ `Auto-paused — its media is on a drive that is not answering${since}. ` +
+ `It returns to ${auto.previousTier} on its own when the drive answers again.`
+ );
+ }
return (
`Auto-paused — its media is on a drive that is not there${since}. ` +
`It returns to ${auto.previousTier} on its own when the drive is back.`
diff --git a/common/lib/corpus.test.ts b/common/lib/corpus.test.ts
@@ -9,7 +9,7 @@ import {
renderSitemapXml,
CORPUS_SPEC_VERSION,
} from "./corpus";
-import { PROJECT_GENERATOR } from "./project";
+import { AI_DOC_URL, PROJECT_GENERATOR } from "./project";
import type { PublicSiteDescriptor } from "./siteDescriptor";
// Run with:
@@ -159,6 +159,37 @@ test("both builders stamp the project generator, and llms.txt trails it", () =>
}
});
+test("Use with AI is the homepage's AI and MCP doc; the chat is the instance's /ask/", () => {
+ // Release 16 slice DX: no site or hub has a /use-with-ai page. corpus.json
+ // keeps the key and names the doc; llms.txt's Ask AI section is two lines.
+ const site = buildSiteCorpus(descriptor({ siteUrl: "https://demo.example" }), {
+ hasArchives: false,
+ });
+ const hub = buildHubCorpus(
+ [{ siteId: "a", siteTitle: "A", siteUrl: "https://a.example" }],
+ { hubTitle: "The Hub", hubUrl: "https://hub.example", generatedAt: "t" },
+ );
+ assert.equal(site.useWithAi, AI_DOC_URL);
+ assert.equal(hub.useWithAi, AI_DOC_URL);
+ assert.equal(AI_DOC_URL, "https://archilyzer.pages.dev/docs/ai-and-mcp/");
+
+ const siteLines = renderSiteLlmsTxt(site).split("\n");
+ const siteAt = siteLines.indexOf("## Ask AI");
+ assert.deepEqual(siteLines.slice(siteAt + 1, siteAt + 3), [
+ "- [Ask AI](https://demo.example/ask/): in-browser chat (bring your own API key).",
+ `- [Use with AI](${AI_DOC_URL}): MCP-server setup for Claude Code, Cursor, and other tools.`,
+ ]);
+ const hubLines = renderHubLlmsTxt(hub).split("\n");
+ const hubAt = hubLines.indexOf("## Ask AI");
+ assert.deepEqual(hubLines.slice(hubAt + 1, hubAt + 3), [
+ "- [Ask AI](https://hub.example/ask/): ask across the whole federation (bring your own API key).",
+ `- [Use with AI](${AI_DOC_URL}): wire up the MCP server.`,
+ ]);
+ for (const txt of [renderSiteLlmsTxt(site), renderHubLlmsTxt(hub)]) {
+ assert.doesNotMatch(txt, /use-with-ai/);
+ }
+});
+
test("the spec is 4, and `generator` is still not why", () => {
// Guard on the reasoning, not just the number: a bump announces a new
// FETCHABLE document. spec 4 is /tags.json. The informational credit string
@@ -207,8 +238,8 @@ test("renderRobotsTxt: sitemap line only with an absolute siteUrl", () => {
test("renderSitemapXml: one loc per route", () => {
const xml = renderSitemapXml({
siteUrl: "https://demo.example",
- routes: ["/", "/use-with-ai"],
+ routes: ["/", "/changelog"],
});
assert.match(xml, /<loc>https:\/\/demo\.example\/<\/loc>/);
- assert.match(xml, /<loc>https:\/\/demo\.example\/use-with-ai<\/loc>/);
+ assert.match(xml, /<loc>https:\/\/demo\.example\/changelog<\/loc>/);
});
diff --git a/common/lib/corpus.ts b/common/lib/corpus.ts
@@ -1,5 +1,5 @@
import type { PublicSiteDescriptor } from "./siteDescriptor";
-import { PROJECT_GENERATOR } from "./project";
+import { AI_DOC_URL, PROJECT_GENERATOR } from "./project";
import { TAGS_FILENAME } from "./curatedTags";
import {
CONTRACT,
@@ -190,7 +190,9 @@ export type SiteCorpus = {
tags?: { url: string; videoField: "curatedTags"; description: string };
// Present when this build ships bulk-download archives (whole-channel zips).
bulkArchives?: { manifest: string; note: string };
- // Pointer to the human page and BYO-key chat.
+ // Pointer to the human page on using the archive with AI: the homepage's AI
+ // and MCP doc (AI_DOC_URL; the site's own /use-with-ai page until release
+ // 16). The BYO-key chat is the site's /ask/.
useWithAi: string;
};
@@ -270,7 +272,7 @@ export function buildSiteCorpus(
totals: { channels: channels.length, videos },
channels,
shardScheme: SHARD_SCHEME,
- useWithAi: join(base, "/use-with-ai"),
+ useWithAi: AI_DOC_URL,
};
// Only advertise the post scheme when this site actually ships posts, so a
// pure-video site's corpus.json is unchanged apart from the spec bump.
@@ -337,7 +339,7 @@ export function buildHubCorpus(
"the whole federation, fetch each site's corpus.json and follow its " +
"shardScheme; results can be merged client-side.",
},
- useWithAi: join(opts.hubUrl, "/use-with-ai"),
+ useWithAi: AI_DOC_URL,
};
}
@@ -362,8 +364,11 @@ export function renderSiteLlmsTxt(corpus: SiteCorpus): string {
out.push("");
out.push("## Ask AI");
out.push(
- `- [Use with AI](${join(base, "/use-with-ai")}): in-browser chat (bring ` +
- `your own API key) and MCP-server setup for Claude Code, Cursor, and other tools.`,
+ `- [Ask AI](${join(base, "/ask/")}): in-browser chat (bring your own API key).`,
+ );
+ out.push(
+ `- [Use with AI](${AI_DOC_URL}): MCP-server setup for Claude Code, Cursor, ` +
+ `and other tools.`,
);
out.push("");
out.push("## Corpus");
@@ -426,9 +431,10 @@ export function renderHubLlmsTxt(corpus: HubCorpus): string {
out.push("");
out.push("## Ask AI");
out.push(
- `- [Use with AI](${join(base, "/use-with-ai")}): ask across the whole ` +
- `federation (bring your own API key) or wire up the MCP server.`,
+ `- [Ask AI](${join(base, "/ask/")}): ask across the whole federation ` +
+ `(bring your own API key).`,
);
+ out.push(`- [Use with AI](${AI_DOC_URL}): wire up the MCP server.`);
out.push("");
out.push("## Federation");
out.push(
diff --git a/common/lib/envVars.ts b/common/lib/envVars.ts
@@ -53,7 +53,7 @@ const paths = (name: string, def: string, doc: string): EnvVarDecl => ({
const DECLARED: EnvVarDecl[] = [
// ── paths: getPaths() ──────────────────────────────────────────────────
- paths("TRANSCRIPTS_DIR", "`<repo>/transcripts`", "The corpus: channels, sites, the LMDB index, job logs, the saved-video store."),
+ paths("TRANSCRIPTS_DIR", "`<repo>/transcripts`", "The corpus: channels, sites, the LMDB index, job logs, the saved-video store. umtool reads `<it>/channels` too, when its own `CHANNELS_DIR` is unset."),
paths("SAVED_VIDEOS_DIR", "`<TRANSCRIPTS_DIR>/saved-videos`", "The persisted source-video store, when it should live on another disk."),
paths("SITES_DIR", "`<TRANSCRIPTS_DIR>/sites`", "Per-site config (`<id>/site.json`, every key in [SITE.md](SITE.md)) and the homepage's `_homepage/`."),
paths("SETTINGS_FILE", "`<repo>/settings.json`", "The settings file (every key in [SETTINGS.md](SETTINGS.md))."),
@@ -80,6 +80,12 @@ const DECLARED: EnvVarDecl[] = [
paths("GALLERY_DL_BIN", "`gallery-dl` on PATH", "The X/Twitter post fetcher, for social channels."),
paths("OLLAMA_URL", "`http://127.0.0.1:11434`", "The local ollama server, the local digest and attribution engine."),
paths("CLAUDE_BIN", "`claude` on PATH", "The `claude` CLI, driving the opt-in metered digest lane."),
+ paths("ARCHILYZER_CONFIG_DIR", "`~/.config/archilyzer`", "The operator's private config dir, outside the repo: the two inputs of `archilyzer source publish` below. Never committed."),
+ paths("SOURCE_SCRUB_FILE", "`<ARCHILYZER_CONFIG_DIR>/source-scrub.txt`", "git-filter-repo `lhs==>rhs` rules applied to file contents AND commit messages when the source mirror is generated (`<home dir>==>/home/user` is built in and runs first). Every rule's left side is also denied. See [PUBLISH.md](PUBLISH.md)."),
+ paths("SOURCE_DENYLIST_FILE", "`<ARCHILYZER_CONFIG_DIR>/source-denylist.txt`", "Literals the published source must never contain, one per line (`i:` = any case). One hit anywhere in the mirror, the tree or the tarball refuses the publish."),
+ paths("ARCHILYZER_SOURCE_SCRATCH", "the OS temp dir", "Where `source publish` makes its scratch clone and stage (removed afterwards unless `--keep-scratch`)."),
+ paths("XDG_CACHE_HOME", "`~/.cache`", "The cache root: `source publish` keeps the history pages' render cache in `<it>/archilyzer/source-history/` (about 140 MB; never inside the checkout)."),
+ paths("STAGIT_BIN", "`stagit` on PATH, then `~/.local/bin/stagit`", "stagit, which renders the source's history pages (`/source/git/`: the log and a page per commit with its diff). Optional: without it the source is published without them. See [PUBLISH.md](PUBLISH.md)."),
// ── runtime ────────────────────────────────────────────────────────────
{ name: "WORKER_TOKEN", audience: "runtime", default: "unset (both surfaces off)", readBy: "common/lib/workerToken.ts, scripts/archilyzer-ops.mjs, mcp/src/fetchClip.ts", doc: "Bearer token for the remote-worker API and for `/api/ops/*` (`pnpm ops`, the MCP's `fetch_clip`). Set the same value on both ends." },
@@ -107,6 +113,9 @@ const DECLARED: EnvVarDecl[] = [
{ name: "TRANSCRIPT_PLATFORM_LINKS", audience: "runtime", default: "off", readBy: "common/lib/archive/reader-fs.ts", doc: "`1` cites platform watch pages instead of the archive's own pages." },
{ name: "AUDIO_CHECK_RESUME_DURING_PROBE", audience: "runtime", default: "the channel's `audioCheck.resumeDuringProbe`", readBy: "common/ytdlp/audioCheckedDownload.ts", doc: "`1` or `true` resumes yt-dlp during the audio check's probe, anything else holds it, for a one-off comparison run; unset = the channel's setting." },
{ name: "AUDIO_CHECK_BACKOFF_FACTOR", audience: "runtime", default: "the built-in factor", readBy: "common/ytdlp/audioCheckedDownload.ts", doc: "The audio check's interval backoff factor, in (0, 1], for a one-off run." },
+ { name: "ARCHILYZER_STATS_ALLOW_DOWNGRADE", audience: "runtime", default: "off", readBy: "common/controller/buildStats.ts", doc: "`1` lets a stats build clear a stats cache that a NEWER build wrote, for a deliberate rollback. Unset, such a build refuses and names both versions." },
+ { name: "ARCHILYZER_INDEX_ALLOW_HELD", audience: "runtime", default: "off", readBy: "common/controller/buildIndex.ts", doc: "`1` lets a FULL index rebuild (a schema change, or no index yet) proceed while a channel's media cannot be read; that channel stays out of the index until its media is back and the index is built again. Unset, such a build refuses and names each channel." },
+ { name: "UV_THREADPOOL_SIZE", audience: "runtime", default: "`16` for the editor (`4` is Node's own)", readBy: "Node's libuv (set by editor/package.json `start` and docker/entrypoint.sh)", doc: "Threads in Node's pool for filesystem calls. A call on a stalled drive holds one until the drive answers, so the editor starts with 16. It buys time for calls already in flight and isolates nothing: the storage health probe and its gate keep new calls off a stalled drive." },
{ name: "MCP_IO_STATS", audience: "runtime", default: "off", readBy: "common/lib/archive/io-stats.ts", doc: "`1` turns on per-call I/O accounting, for `mcp/bench`." },
{ name: "ARCHILYZER_EDITOR_URL", audience: "runtime", default: "`http://localhost:3001`", readBy: "scripts/archilyzer-ops.mjs, mcp/src/fetchClip.ts, umtool", doc: "Which editor `pnpm ops` and the MCP's `fetch_clip` talk to." },
{ name: "ARCHILYZER_AGENT", audience: "runtime", default: "`cli`", readBy: "scripts/archilyzer-ops.mjs", doc: "Who is asking, recorded as the provenance of a curated-tag write through `pnpm ops`." },
@@ -132,7 +141,7 @@ const DECLARED: EnvVarDecl[] = [
{ name: "INSTANCE_MODE", audience: "internal", default: "a site", readBy: "export/app/lib/mode.ts, common/lib/archive/contract.ts", doc: "`hub` makes the export build the hub. Set by `archilyzer build hub`." },
{ name: "BUILD_ARCHIVES", audience: "internal", default: "on", readBy: "common/bin/compose-site.ts, common/bin/build-archives.ts", doc: "`0` skips archive-zip generation for one build (`--skip-archives`)." },
{ name: "ARCHIVES_READONLY", audience: "internal", default: "off", readBy: "common/bin/compose-site.ts", doc: "`1` inside a docker-mode build container: materialize archives, never write the shared cache." },
- { name: "HOMEPAGE_PUBLIC_DIR", audience: "internal", default: "`<repo>/homepage/public`", readBy: "common/bin/compose-homepage.ts", doc: "Where `compose homepage` writes." },
+ { name: "HOMEPAGE_PUBLIC_DIR", audience: "internal", default: "`<repo>/homepage/public`", readBy: "common/bin/compose-homepage.ts, common/publish/source.ts", doc: "Where `compose homepage` and `source publish` write." },
// ── docker: the container's set ────────────────────────────────────────
{ name: "ARCHILYZER_TRANSCRIBER", audience: "docker", default: "baked per image target (`whisper-cpp` in `runtime`)", readBy: "docker/entrypoint.sh", doc: "`whisper-cpp` or `parakeet`: which worker the first boot seeds and which model it fetches." },
@@ -181,6 +190,8 @@ const DECLARED: EnvVarDecl[] = [
{ name: "E2E_RETRIES", audience: "test", default: "`0`", readBy: "scripts/run-sharded-e2e.mjs", doc: "Retries per shard (`--retries N` wins); 0 keeps a sharded run comparable to a serial one." },
{ name: "E2E_IMAGE", audience: "test", default: "`yt-dlp-transcript-browser-e2e`", readBy: "scripts/run-sharded-e2e.mjs", doc: "The sharded e2e run's image tag." },
{ name: "E2E_SKIP_BUILD", audience: "test", default: "off", readBy: "scripts/run-sharded-e2e.mjs", doc: "`1` reuses the sharded e2e image instead of rebuilding it (`--no-build`)." },
+ { name: "E2E_SOURCE_PUBLIC_DIR", audience: "test", default: "unset (the page reads `homepage/public`)", readBy: "homepage/app/lib/source.ts", doc: "The fixture publish the homepage's e2e dev server reads the `/source/` page from while it holds a manifest; set by `homepage/playwright.config.ts`, written and removed by `homepage/e2e/source-history.spec.ts`, ignored by a production build." },
+ { name: "E2E_EXPECT_SOURCE", audience: "test", default: "off (both states pass)", readBy: "homepage/e2e/source.spec.ts", doc: "`1` makes the homepage suite's `/source/` specs fail on the empty state; a gate that ran `archilyzer source publish` first sets it." },
{ name: "E2E_HOMEPAGE_SUMMARY_FILE", audience: "test", default: "`homepage/public/homepage-summary.json`", readBy: "homepage/app/lib/summary.ts", doc: "The synthetic summary the homepage's e2e dev server reads; set by `homepage/playwright.config.ts`, ignored by a production build." },
];
diff --git a/common/lib/homepage.test.ts b/common/lib/homepage.test.ts
@@ -1,6 +1,6 @@
import { test } from "node:test";
import assert from "node:assert/strict";
-import { mkdtemp, readFile } from "node:fs/promises";
+import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import type { Paths } from "./paths";
@@ -44,3 +44,17 @@ test("writeHomepageConfig persists transcriptDownloads only when false, and it r
disk = JSON.parse(await readFile(paths.homepageConfigFile, "utf8"));
assert.equal("transcriptDownloads" in disk, false);
});
+
+test("writeHomepageConfig keeps an unchanged stored icon the checker now refuses, and checks an edited one", async () => {
+ const paths = scratchPaths(await mkdtemp(path.join(os.tmpdir(), "homepage-")));
+ const old = { label: "Old", url: "https://old.example", svg: `<svg viewBox="0 0 8 8"><metadata/></svg>` };
+ await mkdir(path.dirname(paths.homepageConfigFile), { recursive: true });
+ await writeFile(paths.homepageConfigFile, JSON.stringify({ socialLinks: [old] }));
+ const base = getHomepageConfig(paths);
+ await writeHomepageConfig({ ...base, siteTitle: "Renamed" }, paths);
+ assert.deepEqual(JSON.parse(await readFile(paths.homepageConfigFile, "utf8")).socialLinks, [old]);
+ await assert.rejects(
+ writeHomepageConfig({ ...base, socialLinks: [{ ...old, svg: `<svg viewBox="0 0 8 8"><script/></svg>` }] }, paths),
+ /Social link "Old" has an invalid SVG: it has a script/,
+ );
+});
diff --git a/common/lib/homepage.ts b/common/lib/homepage.ts
@@ -4,11 +4,11 @@ import { getPaths, type Paths } from "./paths";
import { PROJECT_NAME, PROJECT_TAGLINE } from "./project";
import {
getSettings,
- normalizeSocialSvg,
parseSocialLinks,
type SiteSettings,
type SocialLink,
} from "./settings";
+import { socialLinksForSave } from "./socialLinks";
// The Archilyzer hub/homepage is a SINGLE, instance-level landing site (the
// `homepage` SSG package) that sits above the per-content sites. Unlike a Site,
@@ -108,16 +108,18 @@ export async function writeHomepageConfig(
config: HomepageConfig,
paths: Paths = getPaths(),
): Promise<void> {
+ // A link whose SVG is unchanged from homepage.json on disk is kept as it is;
+ // a new or edited one is checked (lib/socialLinks.ts socialLinksForSave).
let socialLinks: SocialLink[] | undefined;
if (config.socialLinks !== undefined) {
- socialLinks = [];
- for (const link of parseSocialLinks(config.socialLinks)) {
- const svg = normalizeSocialSvg(link.svg);
- if (svg === null) {
- throw new Error(`Social link "${link.label}" has an invalid SVG`);
- }
- socialLinks.push({ ...link, svg });
+ const r = socialLinksForSave(
+ parseSocialLinks(config.socialLinks),
+ getHomepageConfig(paths).socialLinks,
+ );
+ if ("refused" in r) {
+ throw new Error(`Social link "${r.refused.label}" has an invalid SVG: ${r.refused.problem}`);
}
+ socialLinks = r.links;
}
const merged: HomepageConfig = {
siteTitle: config.siteTitle,
diff --git a/common/lib/homepageSummary.test.ts b/common/lib/homepageSummary.test.ts
@@ -198,6 +198,46 @@ test("a site's accent is published as a hex: an id becomes its on-dark value", (
assert.ok(plain.sites.every((x) => !("accent" in x)));
});
+// Stats schema 6 guarantees a transcript a date, but a stats page written
+// before it could carry a whole channel of transcripts with none — and the fold
+// used to require the date for everything, so a site serving 1,889 videos
+// showed 0 transcripts, 0 channels, 0 hours. The date is only for the time axis.
+test("a transcript with no transcribedDate (a pre-schema-6 page) is counted, just not charted", () => {
+ const sites = [...SITES, site("gamma", ["g1"], "https://gamma.example")];
+ const undated = [
+ stat({ channelSlug: "g1", id: "u1", uploadDate: "20251101", duration: 3600, transcribedDate: null }),
+ stat({ channelSlug: "g1", id: "u2", uploadDate: "20260210", duration: 3600, transcribedDate: null }),
+ ];
+ const base = buildHomepageSummary(STATS, CHANNEL_SITES, sites, NOW);
+ const s = buildHomepageSummary([...STATS, ...undated], { ...CHANNEL_SITES, g1: ["gamma"] }, sites, NOW);
+
+ const gamma = s.sites.find((x) => x.siteId === "gamma");
+ assert.ok(gamma, "a site whose transcripts are all undated is still on the family page");
+ assert.deepEqual(
+ [gamma.channels, gamma.transcribed.total, gamma.hoursArchived, gamma.recordings],
+ [1, 2, 2, 2],
+ );
+ // No month to put them in: not this month, not the sparkline, not a series.
+ assert.equal(gamma.transcribed.thisMonth, 0);
+ assert.deepEqual(gamma.transcribed.last12, new Array(12).fill(0));
+ assert.equal(s.series.transcribed.month.bySite.gamma, undefined);
+ assert.deepEqual(s.series.transcribed.month.total, base.series.transcribed.month.total);
+ assert.equal(s.totals.transcribedThisMonth, base.totals.transcribedThisMonth);
+ assert.ok(!s.recent.some((r) => r.siteId === "gamma"), "the recent rail needs a date");
+ // Counted everywhere a count lives, and placed by UPLOAD month like any other.
+ assert.equal(s.totals.transcripts, base.totals.transcripts + 2);
+ assert.equal(s.totals.channels, base.totals.channels + 1);
+ assert.equal(s.official!.transcripts, base.official!.transcripts + 2);
+ assert.equal(s.official!.channels, base.official!.channels + 1);
+ assert.equal(s.monthly!.find((m) => m.month === "2025-11")!.bySite.gamma, 1);
+ assert.equal(s.monthly!.find((m) => m.month === "2026-02")!.bySite.gamma, 1);
+ const placed = s.monthly!.reduce(
+ (a, m) => a + Object.values(m.bySite).reduce((x, y) => x + y, 0),
+ 0,
+ );
+ assert.equal(placed + s.monthlyUnplaced!, s.official!.transcripts);
+});
+
test("a named accent also travels as its id; a custom hex and no accent carry none", () => {
const sites = [
{ ...site("alpha", ["a1", "a2"], "https://alpha.example"), accent: " Brass " },
@@ -214,3 +254,93 @@ test("a named accent also travels as its id; a custom hex and no accent carry no
const plain = buildHomepageSummary(STATS, CHANNEL_SITES, SITES, NOW);
assert.ok(plain.sites.every((x) => !("accentId" in x)));
});
+
+test("a site's wordmark lead travels when it is a proper prefix of the title; otherwise no key", () => {
+ const sites = [
+ { ...site("alpha", ["a1", "a2"], "https://alpha.example"), siteTitle: "Jeralyzer", wordmarkLead: " Jer " },
+ { ...site("beta", ["b1"], "https://beta.example"), siteTitle: "Anilyzer", wordmarkLead: "Anilyzer" },
+ ] as Site[];
+ const s = buildHomepageSummary(STATS, CHANNEL_SITES, sites, NOW);
+ assert.equal(s.sites.find((x) => x.siteId === "alpha")!.wordmarkLead, "Jer");
+ // The whole title is no split (lib/brand.ts wordmarkLeadFor): no key.
+ assert.ok(!("wordmarkLead" in s.sites.find((x) => x.siteId === "beta")!));
+ // Resolved against the title the card shows, case-sensitive.
+ const other = buildHomepageSummary(
+ STATS,
+ CHANNEL_SITES,
+ [{ ...sites[0], wordmarkLead: "jer" }] as Site[],
+ NOW,
+ );
+ assert.ok(!("wordmarkLead" in other.sites[0]));
+ // No lead configured: no key, as before.
+ const plain = buildHomepageSummary(STATS, CHANNEL_SITES, SITES, NOW);
+ assert.ok(plain.sites.every((x) => !("wordmarkLead" in x)));
+ assert.equal(s.version, HOMEPAGE_SUMMARY_VERSION);
+});
+
+// Release 14 slice HS: site.json `listed: false`. The site still builds and
+// deploys; the family's public pages do not list it, and no public total counts
+// the channels only it exposes.
+test("an unlisted site is in no array and no total; a channel it shares is the listed site's", () => {
+ const unlisted = { ...site("zeta", ["q1", "a1"], "https://zeta.example"), listed: false } as Site;
+ const sites = [...SITES, unlisted];
+ const channelSites = { ...CHANNEL_SITES, a1: ["alpha", "zeta"], q1: ["zeta"] };
+ const own = [
+ stat({ channelSlug: "q1", id: "q-1", uploadDate: "20251101", duration: 7200 }),
+ stat({ channelSlug: "q1", id: "q-2", uploadDate: "20260201", status: "deleted" }),
+ stat({ channelSlug: "q1", id: "q-3", hasTranscript: false, transcribedDate: null }),
+ ];
+ const s = buildHomepageSummary([...STATS, ...own], channelSites, sites, NOW);
+ const base = buildHomepageSummary(STATS, CHANNEL_SITES, SITES, NOW);
+ // Byte for byte the summary without the unlisted site: every array (sites,
+ // channels, series, recent, monthly) and every total (totals, official,
+ // availability, monthlyUnplaced).
+ assert.deepEqual(s, base);
+ const text = JSON.stringify(s);
+ for (const needle of ["zeta", "ZETA", "q1", "Q1", "q-1"]) {
+ assert.ok(!text.includes(needle), needle);
+ }
+ // The shared channel stays credited to alpha.
+ assert.equal(s.sites.find((x) => x.siteId === "alpha")!.channels, 2);
+ assert.equal(s.version, 6);
+});
+
+test("an unlisted site with no siteUrl, or with every channel shared, changes nothing either", () => {
+ const base = buildHomepageSummary(STATS, CHANNEL_SITES, SITES, NOW);
+ // No siteUrl: never public, and its own channel is still in no total (unlike
+ // a pool-only channel, which `totals` counts).
+ const urlless = { ...site("zeta", ["r1"]), listed: false } as Site;
+ assert.deepEqual(
+ buildHomepageSummary(
+ [...STATS, stat({ channelSlug: "r1", id: "r-1" })],
+ { ...CHANNEL_SITES, r1: ["zeta"] },
+ [...SITES, urlless],
+ NOW,
+ ),
+ base,
+ );
+ const sharedOnly = { ...site("zeta", ["a1", "b1"], "https://zeta.example"), listed: false } as Site;
+ assert.deepEqual(
+ buildHomepageSummary(STATS, { ...CHANNEL_SITES, a1: ["alpha", "zeta"], b1: ["beta", "zeta"] }, [...SITES, sharedOnly], NOW),
+ base,
+ );
+ // Listed explicitly is the default: the same summary as no key at all.
+ const listedTrue = SITES.map((x) => ({ ...x, listed: true })) as Site[];
+ assert.deepEqual(buildHomepageSummary(STATS, CHANNEL_SITES, listedTrue, NOW), base);
+});
+
+test("a shared channel is the listed site's even when the unlisted site's id sorts first", () => {
+ const base = buildHomepageSummary(STATS, CHANNEL_SITES, SITES, NOW);
+ // "aaa-hidden" < "beta": the primary pick sorts ids, so only the listed
+ // filter keeps b1 with beta.
+ const unlisted = { ...site("aaa-hidden", ["b1", "q1"], "https://aaa-hidden.example"), listed: false } as Site;
+ const s = buildHomepageSummary(
+ [...STATS, stat({ channelSlug: "q1", id: "q-1", uploadDate: "20251101" })],
+ { ...CHANNEL_SITES, b1: ["aaa-hidden", "beta"], q1: ["aaa-hidden"] },
+ [unlisted, ...SITES],
+ NOW,
+ );
+ assert.deepEqual(s, base);
+ assert.ok(s.recent.filter((r) => r.slug.startsWith("b1/")).every((r) => r.siteId === "beta"));
+ assert.ok(!JSON.stringify(s).includes("aaa-hidden"));
+});
diff --git a/common/lib/homepageSummary.ts b/common/lib/homepageSummary.ts
@@ -1,8 +1,9 @@
import type { Platform } from "./platform";
import type { VideoStat } from "./stats";
import type { Site } from "./site";
+import { channelsOnlyOnUnlistedSites, isListedSite } from "./siteSchema";
import { accentHex, accentIdOf } from "./accent";
-import type { AccentId } from "./brand";
+import { wordmarkLeadFor, type AccentId } from "./brand";
import { VIDEO_STATES, type VideoState } from "./availability";
// Pre-computed, lightweight cross-site summary for the hub (homepage) landing.
@@ -10,11 +11,15 @@ import { VIDEO_STATES, type VideoState } from "./availability";
// HTML, so the landing renders instantly without the browser fetching the
// multi-MB whole-pool stats dataset.
//
-// Scope: the chart "universe" is PUBLIC sites only (those with a siteUrl), and
-// every video is attributed to a single PRIMARY public site (the first, by
-// sorted id, exposing its channel) so the Site and Channel breakdowns partition
-// the same set and combined totals stay honest. The KPI `totals` are instance-
-// wide (count pool-only / URL-less content too) — a deliberate scope difference.
+// Scope: the chart "universe" is PUBLIC sites only (those with a siteUrl that
+// are listed — site.json `listed`, lib/siteSchema.ts isListedSite), and every
+// video is attributed to a single PRIMARY public site (the first, by sorted id,
+// exposing its channel) so the Site and Channel breakdowns partition the same
+// set and combined totals stay honest. The KPI `totals` (and `availability`)
+// are instance-wide (count pool-only / URL-less content too) — a deliberate
+// scope difference — EXCEPT a channel only unlisted sites expose, which no
+// part of the summary counts (channelsOnlyOnUnlistedSites). An unlisted site is
+// in no array here; a channel it shares with a listed site is the listed one's.
//
// Two metrics are pre-binned at two granularities; Cumulative and Share (100%)
// are derived client-side from these, so no extra precompute is needed.
@@ -30,8 +35,15 @@ import { VIDEO_STATES, type VideoState } from "./availability";
// the same reason: a v4 summary on disk must still render (the numbers hide).
//
// Still v5: the per-site `accentId` (release 10) is additive and optional in
-// the same way — a summary without it paints its sites' hex, as before.
-export const HOMEPAGE_SUMMARY_VERSION = 5;
+// the same way — a summary without it paints its sites' hex, as before. So is
+// the per-site `wordmarkLead` (release 14): a summary without it shows each
+// card's title plain, as before. Nothing reads this number to accept a file.
+//
+// v6 (release 14): an unlisted site (site.json `listed: false`) is in no array,
+// and a channel only unlisted sites expose is in no total — `totals` and
+// `availability` included. No field was added or removed; the number says the
+// totals' scope moved.
+export const HOMEPAGE_SUMMARY_VERSION = 6;
// Day buckets are capped to this many trailing days so the embedded summary stays
// small regardless of archive age (daily detail is only useful recently).
@@ -61,6 +73,9 @@ export type MetricSeries = {
};
export type SiteMetricStat = {
+ // For `transcribed`, also counts transcripts with no transcribedDate (a stats
+ // page from before stats schema 6) — they have no month, so they are in
+ // `total` and in neither `thisMonth` nor `last12`.
total: number;
thisMonth: number;
last12: number[]; // last 12 month buckets (zero-padded left), for the sparkline
@@ -90,6 +105,13 @@ export type HomepageSummarySite = {
// base (`var(--swatch-<id>)`, lib/siteColor.ts). Optional, additive
// (release 10): absent for a custom hex, no accent, or an older summary.
accentId?: AccentId;
+ // The site's wordmark lead ("Jer" of "Jeralyzer"): site.json `wordmarkLead`
+ // resolved against `siteTitle` by lib/brand.ts wordmarkLeadFor, the resolver
+ // the sites' own header and site.json use — a proper prefix of the title, or
+ // absent. The homepage's card sets the title as the site's wordmark with it.
+ // Optional, additive (release 14): absent for a site with no lead, or an
+ // older summary.
+ wordmarkLead?: string;
};
export type HomepageChannelMeta = { slug: string; name: string };
@@ -154,7 +176,8 @@ export type HomepageOfficialTotals = {
export type HomepageSummary = {
version: number;
generatedAt: string; // ISO timestamp
- // Instance-wide headline numbers (count everything, not just public sites).
+ // Instance-wide headline numbers (count everything, not just public sites —
+ // but never a channel only unlisted sites expose).
totals: {
transcripts: number;
downloads: number;
@@ -322,6 +345,12 @@ function siteStatFrom(
return { total, thisMonth, last12 };
}
+// A site's transcribed card counts its undated transcripts too. They have no
+// month, so they join `total` only — never `thisMonth` or the sparkline.
+function withUndated(stat: SiteMetricStat, undated: number): SiteMetricStat {
+ return undated > 0 ? { ...stat, total: stat.total + undated } : stat;
+}
+
export function buildHomepageSummary(
stats: readonly VideoStat[],
channelSites: ChannelSitesMap,
@@ -333,12 +362,21 @@ export function buildHomepageSummary(
const nowWeek = weekOf(nowDay.replace(/-/g, ""))!;
const dayFloor = isoDate(new Date(now.getTime() - (DAY_WINDOW - 1) * 86400000));
- // Public sites only (need a link target + a stable place on the chart).
- const publicSites = sites.filter((s): s is Site & { siteUrl: string } => !!s.siteUrl);
+ // Public sites only (need a link target + a stable place on the chart), and
+ // only the listed ones.
+ const publicSites = sites.filter(
+ (s): s is Site & { siteUrl: string } => !!s.siteUrl && isListedSite(s),
+ );
const siteById = new Map(publicSites.map((s) => [s.siteId, s]));
+ // An unlisted site's own channels: counted nowhere below, totals included.
+ const unlistedOnly = channelsOnlyOnUnlistedSites(sites);
+ const inScope = unlistedOnly.size > 0
+ ? stats.filter((s) => !unlistedOnly.has(s.channelSlug))
+ : stats;
// channel slug -> primary public site (first by sorted id). Channels with no
- // public site are out of the chart universe entirely.
+ // public site are out of the chart universe entirely. Over listed sites only,
+ // so a channel an unlisted site shares with a listed one is the listed one's.
const primarySiteOf = new Map<string, string>();
for (const slug of Object.keys(channelSites)) {
const primary = [...channelSites[slug]]
@@ -351,7 +389,7 @@ export function buildHomepageSummary(
const channelName = new Map<string, string>();
const transcribedItems: Attributed[] = [];
const downloadedItems: Attributed[] = [];
- // Instance-wide KPI accumulators (count everything, not just public).
+ // Instance-wide KPI accumulators (count everything in scope, not just public).
let transcripts = 0;
let downloads = 0;
let hoursSeconds = 0;
@@ -362,11 +400,14 @@ export function buildHomepageSummary(
// seconds, gone. `transcripts` is counted HERE, in the same pass that places
// or leaves unplaced each transcript, so `placed + unplaced =
// official.transcripts` holds even for a `transcribedDate` that bucketize
- // drops (a future or malformed month).
+ // drops (a future or malformed month). `undated` counts the transcripts with
+ // no `transcribedDate` at all (a stats page from before stats schema 6): no
+ // transcribed bucket can hold them, so they join the card's total directly.
type SiteAcc = {
channels: Set<string>;
recordings: number;
transcripts: number;
+ undated: number;
seconds: number;
gone: number;
};
@@ -376,7 +417,7 @@ export function buildHomepageSummary(
if (!a)
siteAcc.set(
id,
- (a = { channels: new Set(), recordings: 0, transcripts: 0, seconds: 0, gone: 0 }),
+ (a = { channels: new Set(), recordings: 0, transcripts: 0, undated: 0, seconds: 0, gone: 0 }),
);
return a;
};
@@ -384,13 +425,21 @@ export function buildHomepageSummary(
const uploadItems: { month: string; siteId: string }[] = [];
let monthlyUnplaced = 0;
- for (const s of stats) {
- const hasTx = s.hasTranscript && !!s.transcribedDate;
+ for (const s of inScope) {
+ // A transcript COUNTS whether or not it carries a date: transcripts,
+ // channels, hours, upload-month placement. Only the transcribed time series
+ // (and "this month", and the recent rail) need `transcribedDate`. buildStats
+ // guarantees one since stats schema 6, but a page written before that could
+ // hold a whole channel of transcripts with none — and requiring the date
+ // here made a site serving 1,889 videos show 0 transcripts, 0 channels and
+ // 0 hours.
+ const hasTx = s.hasTranscript;
+ const dated = hasTx && !!s.transcribedDate;
if (hasTx) {
transcripts += 1;
channelSet.add(s.channelSlug);
hoursSeconds += s.duration > 0 ? s.duration : 0;
- if (monthOf(s.transcribedDate) === nowMonth) transcribedThisMonth += 1;
+ if (dated && monthOf(s.transcribedDate) === nowMonth) transcribedThisMonth += 1;
}
if (s.downloadedDate) {
downloads += 1;
@@ -410,7 +459,11 @@ export function buildHomepageSummary(
const um = uploadMonthOf(s.uploadDate);
if (um && um < nowMonth) uploadItems.push({ month: um, siteId });
else monthlyUnplaced += 1;
- transcribedItems.push({ date: s.transcribedDate as string, channelSlug: s.channelSlug, siteId });
+ if (dated) {
+ transcribedItems.push({ date: s.transcribedDate as string, channelSlug: s.channelSlug, siteId });
+ } else {
+ acc.undated += 1;
+ }
}
if (s.downloadedDate) {
downloadedItems.push({ date: s.downloadedDate, channelSlug: s.channelSlug, siteId });
@@ -433,7 +486,10 @@ export function buildHomepageSummary(
siteTitle: s.siteTitle,
siteDescription: s.siteDescription,
siteUrl: s.siteUrl,
- transcribed: siteStatFrom(series.transcribed.month, s.siteId, nowMonth),
+ transcribed: withUndated(
+ siteStatFrom(series.transcribed.month, s.siteId, nowMonth),
+ siteAcc.get(s.siteId)?.undated ?? 0,
+ ),
downloaded: siteStatFrom(series.downloaded.month, s.siteId, nowMonth),
channels: siteAcc.get(s.siteId)?.channels.size ?? 0,
recordings: siteAcc.get(s.siteId)?.recordings ?? 0,
@@ -443,6 +499,9 @@ export function buildHomepageSummary(
// travels as its id too.
...(accentHex(s.accent) ? { accent: accentHex(s.accent) } : {}),
...(accentIdOf(s.accent) ? { accentId: accentIdOf(s.accent) } : {}),
+ ...(wordmarkLeadFor(s.siteTitle, s.wordmarkLead)
+ ? { wordmarkLead: wordmarkLeadFor(s.siteTitle, s.wordmarkLead) }
+ : {}),
}))
// Keep a public site only if it has any activity in either metric.
.filter((s) => s.transcribed.total > 0 || s.downloaded.total > 0)
@@ -457,7 +516,7 @@ export function buildHomepageSummary(
.sort((a, b) => a.name.localeCompare(b.name));
// Recent feed (public universe), attributed to the primary site for its link.
- const recent: HomepageRecentItem[] = stats
+ const recent: HomepageRecentItem[] = inScope
.filter((s): s is VideoStat & { transcribedDate: string } =>
Boolean(s.hasTranscript && s.transcribedDate && primarySiteOf.get(s.channelSlug)),
)
@@ -483,16 +542,17 @@ export function buildHomepageSummary(
};
});
- // State census over every record, matching `totals`' instance-wide scope
- // (pool-only channels included) rather than the charts' public-site universe.
+ // State census over every in-scope record, matching `totals`' instance-wide
+ // scope (pool-only channels included, a channel only unlisted sites expose
+ // not) rather than the charts' public-site universe.
// Seeded with every state at zero so the shape is stable across corpora.
const byState = Object.fromEntries(
VIDEO_STATES.map((s) => [s, 0]),
) as Record<VideoState, number>;
- for (const s of stats) byState[s.status] = (byState[s.status] ?? 0) + 1;
+ for (const s of inScope) byState[s.status] = (byState[s.status] ?? 0) + 1;
const availability: HomepageAvailability = {
byState,
- counted: stats.length,
+ counted: inScope.length,
};
// Monthly back-catalogue series, keyed by the sites that survived the
diff --git a/common/lib/normalizeSocialSvg.test.ts b/common/lib/normalizeSocialSvg.test.ts
@@ -1,6 +1,9 @@
import { test } from "node:test";
import assert from "node:assert/strict";
-import { normalizeSocialSvg } from "./settingsSchema";
+import { normalizeSocialSvg, socialSvgProblem } from "./settingsSchema";
+import { SVG_PROBLEM } from "./socialSvg";
+import { scopeSvgIds, sizeSocialSvg } from "./socialLinks";
+import { ADVERSARIAL, LOADS_ELSEWHERE, REAL_SHAPES } from "./socialSvg.vectors";
// normalizeSocialSvg runs on every save of a social link (Settings, a site's
// form, the homepage config). It used to theme only the ROOT <svg>'s fill, so a
@@ -190,3 +193,382 @@ test("still refuses what is unsafe to inline", () => {
assert.equal(normalizeSocialSvg(`<svg viewBox="0 0 1 1" onload="x()"></svg>`), null);
assert.equal(normalizeSocialSvg(`<svg><path fill="white"/></svg>`), null, "no viewBox");
});
+
+// A ROOT WITH A SIZE AND NO viewBox. A vendor's file often carries only its
+// width and height; it used to be refused as pasted, and the operator had to
+// add a viewBox by hand. Its size now becomes `viewBox="0 0 W H"` before the
+// size is stripped. A synthetic two-colour disc with a letter, shaped like such
+// a file: a dark offset disc, a light disc with a dark outline, a dark glyph,
+// `fill="none"` on the root.
+const SIZED_DISC =
+ `<svg xmlns="http://www.w3.org/2000/svg" width="81" height="81" fill="none">` +
+ `<circle cx="44" cy="43" r="37" fill="#1a1a1a"/>` +
+ `<circle cx="39" cy="39" r="38" fill="#f4c542" stroke="#1a1a1a" stroke-width="1.5"/>` +
+ `<path fill="#1a1a1a" d="M30 22h20v8H38v6h10v8H38v14h-8z"/></svg>`;
+
+test("a disc pasted with only its size: accepted, its size made the viewBox, both colours kept", () => {
+ const out = normalized(SIZED_DISC);
+ assert.ok(
+ out.startsWith('<svg aria-hidden="true" viewBox="0 0 81 81" xmlns="http://www.w3.org/2000/svg" fill="none">'),
+ out.slice(0, 120),
+ );
+ assert.ok(!/\s(width|height)\s*=\s*"81"/.test(out.slice(0, out.indexOf(">"))), "size stripped");
+ assert.deepEqual(fills(out), ["none", "#1a1a1a", "#f4c542", "#1a1a1a"]);
+ // Nothing below the root moved.
+ assert.equal(out.slice(out.indexOf(">")), SIZED_DISC.slice(SIZED_DISC.indexOf(">")));
+});
+
+test("a size in px or unitless, either quote, decimals: the viewBox is the numbers", () => {
+ const box = (open: string) => {
+ const out = normalized(`${open}<path fill="#fff" d="M0 0"/></svg>`);
+ return /\sviewBox="([^"]*)"/.exec(out)?.[1];
+ };
+ assert.equal(box(`<svg width="24" height="24">`), "0 0 24 24");
+ assert.equal(box(`<svg width="24px" height='16PX'>`), "0 0 24 16");
+ assert.equal(box(`<svg height=" 12.5 " width="30">`), "0 0 30 12.5");
+ assert.equal(box(`<svg width="048" height=".5">`), "0 0 48 0.5");
+ // stroke-width on the root is not a width.
+ assert.equal(box(`<svg stroke-width="2" width="10" height="20">`), "0 0 10 20");
+ // A size spelled inside another attribute's value is not the root's size.
+ assert.equal(
+ normalizeSocialSvg(`<svg data-a=' width="7" height="9"'><path d="M0 0"/></svg>`),
+ null,
+ );
+});
+
+test("a size that is not a pixel size is still refused", () => {
+ for (const open of [
+ `<svg width="100%" height="100%">`,
+ `<svg width="2em" height="2em">`,
+ `<svg width="24">`,
+ `<svg height="24">`,
+ `<svg width="0" height="24">`,
+ `<svg width="24" height="0">`,
+ `<svg width="" height="24">`,
+ `<svg width="-24" height="24">`,
+ `<svg width="auto" height="24">`,
+ `<svg stroke-width="2">`,
+ ]) {
+ assert.equal(normalizeSocialSvg(`${open}<path d="M0 0"/></svg>`), null, open);
+ assert.equal(socialSvgProblem(`${open}<path d="M0 0"/></svg>`), SVG_PROBLEM.viewBox, open);
+ }
+});
+
+test("a viewBox already there is never replaced by the size", () => {
+ const raw = `<svg width="81" height="81" viewBox="0 0 24 24"><path fill="#fff" d="M0 0"/></svg>`;
+ assert.equal(
+ normalized(raw),
+ `<svg aria-hidden="true" fill="currentColor" viewBox="0 0 24 24"><path fill="currentColor" d="M0 0"/></svg>`,
+ );
+ // A single-quoted viewBox was refused before this change and still is: the
+ // size never adds a second one.
+ assert.equal(
+ normalizeSocialSvg(`<svg width="8" height="8" viewBox='0 0 8 8'><path d="M0 0"/></svg>`),
+ null,
+ );
+});
+
+test("idempotent with a synthesized viewBox", () => {
+ const once = normalized(SIZED_DISC);
+ assert.equal(normalized(once), once);
+ const sized = normalized(`<svg width="24" height="24"><path fill="#fff" d="M0 0"/></svg>`);
+ assert.equal(normalized(sized), sized);
+});
+
+// ── WHAT AN ICON MAY CONTAIN ────────────────────────────────────────────────
+// The review's five inputs, each accepted by the old text denylist and each
+// running script in Chromium, then one case per class the allowlist refuses.
+
+const REVIEW_BYPASSES: Record<string, string> = {
+ slash_handler: `<svg/onload="window.__x=1" viewBox="0 0 8 8"><path d="M0 0"/></svg>`,
+ deletion_join: `<svg viewBox="0 0 8 8" o width="1"nload="window.__x=1"><path d="M0 0"/></svg>`,
+ image_slash: `<svg viewBox="0 0 8 8"><image href="x:"/onerror="window.__x=1"/></svg>`,
+ charref_js: `<svg viewBox="0 0 8 8"><a href="javascript:window.__x=1"><rect width="8" height="8"/></a></svg>`,
+ breakout_img: `<svg viewBox="0 0 8 8"><img src="x:"/onerror="window.__x=1"></svg>`,
+};
+
+test("the review's five bypasses are refused", () => {
+ for (const [name, raw] of Object.entries(REVIEW_BYPASSES)) {
+ assert.equal(normalizeSocialSvg(raw), null, name);
+ assert.ok(socialSvgProblem(raw), name);
+ }
+});
+
+const OK = (inner: string, open = `<svg viewBox="0 0 8 8">`) => `${open}${inner}</svg>`;
+
+test("an event handler is refused wherever an attribute can start", () => {
+ for (const raw of [
+ `<svg viewBox="0 0 8 8" onload="x()"><path d="M0 0"/></svg>`,
+ `<svg viewBox="0 0 8 8"\tonload="x()"><path d="M0 0"/></svg>`,
+ `<svg viewBox="0 0 8 8"\nONLOAD="x()"><path d="M0 0"/></svg>`,
+ `<svg viewBox="0 0 8 8"\fonclick="x()"><path d="M0 0"/></svg>`,
+ OK(`<path d="M0 0" onmouseover="x()"/>`),
+ OK(`<animate attributeName="onclick" to="x()"/>`),
+ ]) {
+ assert.equal(socialSvgProblem(raw), SVG_PROBLEM.handler, raw);
+ }
+ // A `/` between attributes (which HTML takes as a separator) is not markup
+ // this reads.
+ assert.equal(socialSvgProblem(`<svg/onload="x()" viewBox="0 0 8 8"></svg>`), SVG_PROBLEM.markup);
+});
+
+test("a script, or a link that runs one, in any spelling", () => {
+ for (const raw of [
+ OK(`<script>x()</script>`),
+ OK(`<rect fill="url(#a)" style="fill:javascript:x()"/>`),
+ OK(`<rect data-x="javascript:x()"/>`),
+ OK(`<rect data-x="java	script:x()"/>`),
+ OK(`<rect data-x="java script:x()"/>`),
+ OK(`<rect data-x="javascript:x()"/>`),
+ OK(`<rect data-x="VBScript:x()"/>`),
+ ]) {
+ assert.equal(socialSvgProblem(raw), SVG_PROBLEM.script, raw);
+ }
+});
+
+test("a link to anything but a fragment of this icon", () => {
+ for (const raw of [
+ OK(`<use href="https://x.example/i.svg#a"/>`),
+ OK(`<use xlink:href="data:image/svg+xml,<svg/>"/>`),
+ OK(`<use href="#a""/>`),
+ OK(`<linearGradient id="g" href="//x.example/g"/>`),
+ OK(`<rect fill="url(https://x.example/p.svg#a)"/>`),
+ OK(`<rect style="fill:url(data:image/png;base64,AAAA)"/>`),
+ OK(`<rect mask="url( '//x.example' )"/>`),
+ OK(`<set attributeName="href" to="#a"/>`),
+ ]) {
+ assert.equal(socialSvgProblem(raw), SVG_PROBLEM.external, raw);
+ }
+ // A fragment, in each spelling a reference takes, is fine.
+ for (const raw of [
+ OK(`<defs><linearGradient id="g"/></defs><use href="#g"/><use xlink:href="#g"/>`),
+ OK(`<rect fill="url(#g)" style="fill:url('#g')" mask="url("#g")"/>`),
+ ]) {
+ assert.equal(socialSvgProblem(raw), null, raw);
+ }
+});
+
+test("elements an icon has no use for, named", () => {
+ for (const [inner, name] of [
+ [`<foreignObject><div/></foreignObject>`, "foreignObject"],
+ [`<style>@import "x";</style>`, "style"],
+ [`<a href="#a"><rect/></a>`, "a"],
+ [`<image href="#a"/>`, "image"],
+ [`<iframe/>`, "iframe"],
+ [`<object/>`, "object"],
+ [`<embed/>`, "embed"],
+ [`<audio/>`, "audio"],
+ [`<video/>`, "video"],
+ [`<img/>`, "img"],
+ [`<p>x</p>`, "p"],
+ [`<sodipodi:namedview/>`, "sodipodi:namedview"],
+ ] as const) {
+ assert.equal(socialSvgProblem(OK(inner)), SVG_PROBLEM.element(name), inner);
+ }
+});
+
+test("only a drawing program's leftovers say how to export: a style, metadata, its own namespace", () => {
+ const HINT = /presentation attributes rather than a style block \(in Inkscape, save as Plain SVG\)$/;
+ for (const inner of [
+ `<style>path{fill:red}</style>`,
+ `<metadata><x/></metadata>`,
+ `<sodipodi:namedview/>`,
+ `<rect inkscape:label="x"/>`,
+ `<rect style="position:fixed"/>`,
+ ]) {
+ assert.match(socialSvgProblem(OK(inner)) ?? "", HINT, inner);
+ }
+ for (const inner of [
+ `<a href="#a"><rect/></a>`,
+ `<image href="#a"/>`,
+ `<title><path/></title>`,
+ `<foreignObject/>`,
+ `<rect src="x"/>`,
+ `<rect formaction="x"/>`,
+ ]) {
+ const problem = socialSvgProblem(OK(inner));
+ assert.ok(problem, inner);
+ assert.doesNotMatch(problem, /presentation attributes|Plain SVG/, inner);
+ }
+});
+
+test("attributes an icon has no use for, named; a style with an escape or an import", () => {
+ assert.equal(socialSvgProblem(OK(`<rect src="x"/>`)), SVG_PROBLEM.attribute("src"));
+ assert.equal(socialSvgProblem(OK(`<rect formaction="x"/>`)), SVG_PROBLEM.attribute("formaction"));
+ assert.equal(socialSvgProblem(OK(`<rect inkscape:label="x"/>`)), SVG_PROBLEM.attribute("inkscape:label"));
+ assert.equal(socialSvgProblem(OK(`<rect style="fill:u\\72l(x)"/>`)), SVG_PROBLEM.style);
+ assert.equal(socialSvgProblem(OK(`<rect style="@import 'x'"/>`)), SVG_PROBLEM.external);
+});
+
+test("markup that is not one well-formed <svg>: declarations, CDATA, instructions, strays", () => {
+ for (const raw of [
+ OK(`<![CDATA[x]]>`),
+ OK(`<!ENTITY x "y">`),
+ OK(`<?php x ?>`),
+ OK(`<path d="M0 0">`), // never closed
+ OK(`</g>`), // closed, never opened
+ `<svg viewBox="0 0 8 8"></svg><svg viewBox="0 0 8 8"></svg>`,
+ `<svg viewBox="0 0 8 8"></svg>text<svg></svg>`,
+ OK(`<path d=M0/>`), // an unquoted value
+ OK(`<path d="M0 0"/ >`),
+ ]) {
+ assert.equal(socialSvgProblem(raw), SVG_PROBLEM.markup, raw);
+ }
+});
+
+test("comments are removed first; an XML declaration and a plain DOCTYPE at the start are too", () => {
+ const out = normalized(
+ `<?xml version="1.0" encoding="UTF-8"?>\n<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">\n` +
+ `<!-- Generator: a drawing app --><svg viewBox="0 0 8 8"><!-- a --><path d="M0 0"/></svg>`,
+ );
+ assert.equal(out, `<svg aria-hidden="true" fill="currentColor" viewBox="0 0 8 8"><path d="M0 0"/></svg>`);
+ // A comment cannot hide a handler: what is left after removal is checked.
+ assert.equal(socialSvgProblem(`<svg viewBox="0 0 8 8" on<!-- -->load="x()"></svg>`), SVG_PROBLEM.handler);
+ // A DOCTYPE with an internal subset is not removed, and is refused with
+ // what to do about it.
+ assert.equal(socialSvgProblem(`<!DOCTYPE svg [<!ENTITY x "y">]><svg viewBox="0 0 8 8"></svg>`), SVG_PROBLEM.doctype);
+});
+
+test("an id a rendered copy cannot prefix safely is refused", () => {
+ for (const id of ["->", "-!>", "1a", "a b", "a&b", ""]) {
+ assert.equal(socialSvgProblem(OK(`<path id="${id}" d="M0 0"/>`)), SVG_PROBLEM.id, id);
+ }
+ assert.equal(socialSvgProblem(OK(`<path id="a.b:c-d_1" d="M0 0"/>`)), null);
+});
+
+// The shapes an operator's icons take keep passing: a gradient body with a
+// solid part, a single path drawn white, a two-colour disc.
+test("the shapes real icons take are accepted", () => {
+ const gradient =
+ `<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><defs>` +
+ `<linearGradient id="grad_1" x1="0" y1="0" x2="0" y2="1" gradientUnits="objectBoundingBox">` +
+ `<stop offset="0" stop-color="#e0a030"/><stop offset="1" style="stop-color:#b04020"/></linearGradient></defs>` +
+ `<path fill="url(#grad_1)" style="fill:url(#grad_1)" d="M12 2 20 20H4z"/><circle fill="#333" cx="12" cy="15" r="2"/></svg>`;
+ const single = `<svg viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M2 2h20v20H2z" fill="white"/></svg>`;
+ for (const raw of [gradient, single, SIZED_DISC]) {
+ assert.equal(socialSvgProblem(raw), null, raw.slice(0, 60));
+ const out = normalized(raw);
+ assert.equal(socialSvgProblem(out), null);
+ }
+});
+
+// ── THE RE-REVIEW (R1, R2, R5, R6, the style properties) ────────────────────
+
+test("R1: a title or desc with text only is removed; with a child element it is refused", () => {
+ assert.equal(
+ normalized(OK(`<title>A name</title><desc/><path d="M0 0"/>`)),
+ `<svg aria-hidden="true" fill="currentColor" viewBox="0 0 8 8"><path d="M0 0"/></svg>`,
+ );
+ for (const raw of [
+ OK(`<title><path d="M0 0"/></title>`),
+ OK(`<desc><g></g></desc>`),
+ OK(`<title><svg viewBox="0 0 1 1"><title>t</title></svg></title>`),
+ ]) {
+ assert.equal(socialSvgProblem(raw), SVG_PROBLEM.element(raw.includes("<desc>") ? "desc" : "title"), raw);
+ }
+});
+
+test("R2: the eight inputs that loaded from another origin are refused", () => {
+ for (const name of LOADS_ELSEWHERE) {
+ assert.equal(normalizeSocialSvg(ADVERSARIAL[name]), null, name);
+ }
+ for (const raw of [
+ OK(`<rect fill="\\75 rl(#a)"/>`),
+ OK(`<rect style="fill:url/**/(#a)"/>`),
+ OK(`<rect mask="cross-fade(url(#a), url(#b))"/>`),
+ OK(`<rect mask="-webkit-cross-fade(url(#a), url(#b))"/>`),
+ OK(`<rect style="fill:element(#a)"/>`),
+ OK(`<rect style="fill:-moz-element(#a)"/>`),
+ OK(`<rect style="fill:image(#a)"/>`),
+ OK(`<rect style="fill:src(#a)"/>`),
+ OK(`<rect style="fill:paint(x)"/>`),
+ OK(`<rect style="fill:expression(x)"/>`),
+ OK(`<rect><set attributeName="fill" to="image-set('${"https://x.example"}/a.png' 1x)"/></rect>`),
+ ]) {
+ assert.ok(socialSvgProblem(raw), raw);
+ }
+});
+
+test("a style holds presentation properties only", () => {
+ for (const decl of ["position:fixed", "inset:0", "z-index:9", "top:0", "width:100vw", "background:red", "cursor:pointer", "transform:scale(9)"]) {
+ assert.equal(socialSvgProblem(OK(`<rect style="${decl}"/>`)), SVG_PROBLEM.style, decl);
+ }
+ assert.equal(
+ socialSvgProblem(OK(`<rect style="fill:#fff; stroke:#000;stroke-width:2;opacity:.5;paint-order:stroke;display:inline"/>`)),
+ null,
+ );
+});
+
+test("R5: an animation targets nothing named href and no handler, in any spelling", () => {
+ for (const attr of ["x:href", " HREF ", "xlink:href", "href"]) {
+ assert.equal(socialSvgProblem(OK(`<set attributeName="${attr}" to="#a"/>`)), SVG_PROBLEM.external, attr);
+ }
+ for (const attr of ["onclick", "xlink:onclick", "x:onload"]) {
+ assert.equal(socialSvgProblem(OK(`<set attributeName="${attr}" to="1"/>`)), SVG_PROBLEM.handler, attr);
+ }
+});
+
+test("R6: a reference the render cannot scope is refused: an encoded or padded fragment", () => {
+ for (const raw of [
+ OK(`<defs><g id="a"/></defs><use href="#a"/>`),
+ OK(`<defs><g id="a"/></defs><use href=" #a "/>`),
+ OK(`<defs><g id="a"/></defs><rect fill="url(#a)"/>`),
+ OK(`<defs><g id="a"/></defs><rect fill="url(#a)"/>`),
+ ]) {
+ assert.equal(socialSvgProblem(raw), SVG_PROBLEM.external, raw);
+ }
+ // …and each accepted spelling is one scopeSvgIds rewrites.
+ const ok = normalized(OK(`<defs><g id="a"/></defs><use href="#a"/><rect fill="url(#a)" mask="url("#a")" style="fill:url('#a')"/>`));
+ const scoped = scopeSvgIds(ok, "s");
+ assert.ok(!/#a\b/.test(scoped.replace(/#s-a/g, "")), scoped);
+});
+
+test("the shapes real icons take pass, stay stable and survive the render", () => {
+ for (const [name, raw] of Object.entries(REAL_SHAPES)) {
+ const out = normalizeSocialSvg(raw);
+ assert.ok(out, `${name}: ${socialSvgProblem(raw)}`);
+ assert.equal(normalizeSocialSvg(out), out, `${name} idempotent`);
+ assert.ok(normalizeSocialSvg(sizeSocialSvg(scopeSvgIds(out, "sl_S_1_-0"))), `${name} rendered`);
+ }
+ assert.ok(normalized(REAL_SHAPES.gradient_outlined).includes('viewBox="-3 -3 76.3 80.8"'));
+});
+
+// The review's whole battery: an input is refused, or its output is stable,
+// survives the render's scoping and sizing, and — the structural half of R1's
+// invariant — holds only SVG elements the HTML parser keeps in foreign content
+// (no title, desc, foreignObject or HTML element can reach a page). The
+// browser half (a real parser, #after outside the icon) is the homepage e2e's
+// svg-vectors.spec.ts: the repo has no HTML parser to run it here.
+const FOREIGN_SAFE = new Set([
+ "svg", "g", "defs", "symbol", "use", "path", "rect", "circle", "ellipse", "line",
+ "polyline", "polygon", "text", "tspan", "lineargradient", "radialgradient", "stop",
+ "pattern", "clippath", "mask", "filter", "feblend", "fecolormatrix",
+ "fecomponenttransfer", "fecomposite", "fedropshadow", "feflood", "fefunca", "fefuncb",
+ "fefuncg", "fefuncr", "fegaussianblur", "femerge", "femergenode", "femorphology",
+ "feoffset", "animate", "animatetransform", "set",
+]);
+
+test("every adversarial input is refused, or accepted in a form that stays inside its <svg>", () => {
+ let accepted = 0;
+ for (const [name, raw] of Object.entries(ADVERSARIAL)) {
+ const out = normalizeSocialSvg(raw);
+ if (out === null) {
+ assert.ok(socialSvgProblem(raw), name);
+ continue;
+ }
+ accepted += 1;
+ assert.equal(normalizeSocialSvg(out), out, `${name} idempotent`);
+ const rendered = sizeSocialSvg(scopeSvgIds(out, "sl_S_1_-0"));
+ assert.ok(normalizeSocialSvg(rendered), `${name} rendered`);
+ // Tag names outside attribute values (a quoted `<img>` is a value, and inert).
+ const markup = rendered.replace(/"[^"]*"|'[^']*'/g, '""');
+ for (const m of markup.matchAll(/<\/?([A-Za-z][\w:.-]*)/g)) {
+ assert.ok(FOREIGN_SAFE.has(m[1].toLowerCase()), `${name}: <${m[1]}>`);
+ }
+ assert.ok(!/<!|<\?/.test(markup), `${name}: markup`);
+ }
+ assert.ok(accepted > 10, `${accepted} accepted`);
+ for (const name of ["title_child_el", "desc_child_el", "title_title", "title_svg_title", "overlay_style", ...LOADS_ELSEWHERE]) {
+ assert.equal(normalizeSocialSvg(ADVERSARIAL[name]), null, name);
+ }
+});
diff --git a/common/lib/paths.ts b/common/lib/paths.ts
@@ -158,79 +158,114 @@ export type Paths = {
// settings.digest.remoteEnabled). Not bundled; install it separately and point
// CLAUDE_BIN at it if it isn't on PATH.
claudeBin: string;
+ // The operator's PRIVATE config dir, outside the repo (~/.config/archilyzer
+ // by default). Holds the two inputs of `archilyzer source publish`
+ // (common/publish/source.ts), which are never committed:
+ // sourceScrubFile — git-filter-repo `lhs==>rhs` rules for the mirror
+ // sourceDenylistFile — literals the published source must never contain
+ configDir: string;
+ sourceScrubFile: string;
+ sourceDenylistFile: string;
+ // Where `source publish` makes its scratch clone (removed afterwards).
+ sourceScratchDir: string;
+ // The history pages' render cache (publish/sourceHistory.ts): the XDG
+ // cache dir's archilyzer/source-history — ~/.cache unless XDG_CACHE_HOME
+ // says otherwise (an empty one is unset, as the XDG spec has it). Never
+ // inside the checkout or the public dir: the step renders without it there.
+ sourceHistoryCacheDir: string;
+ // stagit, which renders the source's history pages (/source/git/). An
+ // operator-installed tool, never vendored: `stagit` on PATH, then
+ // ~/.local/bin/stagit (publish/sourceHistory.ts resolveStagit). Without it
+ // the publish goes on without the history pages.
+ stagitBin: string;
};
let cached: Paths | null = null;
+// Every path in getPaths() is built on the repo root, which findMonorepoRoot()
+// finds by walking up from `process.cwd()`. Turbopack evaluates `process.cwd()`
+// statically, and a path op on a value derived from it becomes an asset
+// reference — to every file under it when the path is a directory
+// (plans/FACTS.md, "A path joined from `process.cwd()` …"). So every join on
+// such a value goes through this one opted-out call. Nothing changes at run
+// time. The comment sits before a named first argument, not a spread: that is
+// the form Turbopack's own advice shows.
+function under(first: string, ...rest: string[]): string {
+ return path.join(/* turbopackIgnore: true */ first, ...rest);
+}
+
export function getPaths(): Paths {
if (cached) return cached;
const monorepoRoot = findMonorepoRoot();
const transcriptsDir =
- process.env.TRANSCRIPTS_DIR ?? path.join(monorepoRoot, "transcripts");
- const exportDir = path.join(monorepoRoot, "export");
+ process.env.TRANSCRIPTS_DIR ?? under(monorepoRoot, "transcripts");
+ const exportDir = under(monorepoRoot, "export");
const exportPublicDir =
- process.env.EXPORT_PUBLIC_DIR ?? path.join(exportDir, "public");
+ process.env.EXPORT_PUBLIC_DIR ?? under(exportDir, "public");
// Staging sibling of the served public dir (so it lands inside the test data
// root when EXPORT_PUBLIC_DIR is overridden). Not served; composed into
// exportPublicDir per site by the build:site step.
const exportIndexDir =
process.env.EXPORT_INDEX_DIR ??
- path.join(path.dirname(exportPublicDir), ".export-index");
- const exportSharedDir = path.join(exportIndexDir, "shared");
- const sitesDir = process.env.SITES_DIR ?? path.join(transcriptsDir, "sites");
- const homepageDir = path.join(sitesDir, "_homepage");
+ under(path.dirname(/* turbopackIgnore: true */ exportPublicDir), ".export-index");
+ const exportSharedDir = under(exportIndexDir, "shared");
+ const sitesDir = process.env.SITES_DIR ?? under(transcriptsDir, "sites");
+ const homepageDir = under(sitesDir, "_homepage");
+ const configDir =
+ process.env.ARCHILYZER_CONFIG_DIR ??
+ path.join(os.homedir(), ".config", "archilyzer");
cached = {
monorepoRoot,
transcriptsDir,
- channelsDir: path.join(transcriptsDir, "channels"),
+ channelsDir: under(transcriptsDir, "channels"),
savedVideosDir:
- process.env.SAVED_VIDEOS_DIR ?? path.join(transcriptsDir, "saved-videos"),
+ process.env.SAVED_VIDEOS_DIR ?? under(transcriptsDir, "saved-videos"),
sitesDir,
homepageDir,
- homepageConfigFile: path.join(homepageDir, "homepage.json"),
- homepageChartTemplatesFile: path.join(homepageDir, "chart-templates.json"),
- jobsDir: path.join(transcriptsDir, ".jobs"),
- workerScratchDir: path.join(transcriptsDir, ".worker-scratch"),
- schedulerStateFile: path.join(transcriptsDir, ".scheduler", "state.json"),
- autoQueueStateFile: path.join(transcriptsDir, ".auto-queue", "state.json"),
- workerDefaultsFile: path.join(transcriptsDir, ".workers", "defaults.json"),
- widgetPresetsFile: path.join(transcriptsDir, ".widget", "presets.json"),
- lmdbPath: path.join(transcriptsDir, "index.mdb"),
+ homepageConfigFile: under(homepageDir, "homepage.json"),
+ homepageChartTemplatesFile: under(homepageDir, "chart-templates.json"),
+ jobsDir: under(transcriptsDir, ".jobs"),
+ workerScratchDir: under(transcriptsDir, ".worker-scratch"),
+ schedulerStateFile: under(transcriptsDir, ".scheduler", "state.json"),
+ autoQueueStateFile: under(transcriptsDir, ".auto-queue", "state.json"),
+ workerDefaultsFile: under(transcriptsDir, ".workers", "defaults.json"),
+ widgetPresetsFile: under(transcriptsDir, ".widget", "presets.json"),
+ lmdbPath: under(transcriptsDir, "index.mdb"),
exportDir,
exportPublicDir,
- exportSummariesDir: path.join(exportPublicDir, "summaries"),
- exportTranscriptsDir: path.join(exportPublicDir, "transcripts"),
- exportSubsDir: path.join(exportPublicDir, "subs"),
- exportPostsDir: path.join(exportPublicDir, "posts"),
- exportDigestsDir: path.join(exportPublicDir, "digests"),
- exportStatsDir: path.join(exportPublicDir, "stats"),
+ exportSummariesDir: under(exportPublicDir, "summaries"),
+ exportTranscriptsDir: under(exportPublicDir, "transcripts"),
+ exportSubsDir: under(exportPublicDir, "subs"),
+ exportPostsDir: under(exportPublicDir, "posts"),
+ exportDigestsDir: under(exportPublicDir, "digests"),
+ exportStatsDir: under(exportPublicDir, "stats"),
exportIndexDir,
exportSharedDir,
- exportSharedTranscriptsDir: path.join(exportSharedDir, "transcripts"),
- exportSharedSubsDir: path.join(exportSharedDir, "subs"),
- exportSharedPostsDir: path.join(exportSharedDir, "posts"),
- exportSharedDigestsDir: path.join(exportSharedDir, "digests"),
- exportSitesIndexDir: path.join(exportIndexDir, "sites"),
+ exportSharedTranscriptsDir: under(exportSharedDir, "transcripts"),
+ exportSharedSubsDir: under(exportSharedDir, "subs"),
+ exportSharedPostsDir: under(exportSharedDir, "posts"),
+ exportSharedDigestsDir: under(exportSharedDir, "digests"),
+ exportSitesIndexDir: under(exportIndexDir, "sites"),
exportBuildsDir:
process.env.EXPORT_BUILDS_DIR ??
- path.join(path.dirname(exportPublicDir), ".export-builds"),
+ under(path.dirname(/* turbopackIgnore: true */ exportPublicDir), ".export-builds"),
settingsFile:
- process.env.SETTINGS_FILE ?? path.join(monorepoRoot, "settings.json"),
+ process.env.SETTINGS_FILE ?? under(monorepoRoot, "settings.json"),
editorChangelogFile:
process.env.EDITOR_CHANGELOG_FILE ??
- path.join(monorepoRoot, "editor", "CHANGELOG.md"),
+ under(monorepoRoot, "editor", "CHANGELOG.md"),
exportChangelogFile:
process.env.EXPORT_CHANGELOG_FILE ??
- path.join(exportDir, "CHANGELOG.md"),
+ under(exportDir, "CHANGELOG.md"),
chartsConfigFile:
process.env.CHARTS_CONFIG_FILE ??
- path.join(monorepoRoot, "chart-templates.json"),
+ under(monorepoRoot, "chart-templates.json"),
globalAliasesFile:
process.env.SEARCH_ALIASES_FILE ??
- path.join(transcriptsDir, "search-aliases.json"),
+ under(transcriptsDir, "search-aliases.json"),
globalTagsFile:
process.env.CURATED_TAGS_FILE ??
- path.join(transcriptsDir, TAGS_FILENAME),
+ under(transcriptsDir, TAGS_FILENAME),
ytdlpBin: process.env.YTDLP_BIN ?? "yt-dlp",
whisperBin: process.env.WHISPER_BIN ?? "whisper-cli",
whisperModel:
@@ -250,7 +285,7 @@ export function getPaths(): Paths {
galleryDlBin: process.env.GALLERY_DL_BIN ?? "gallery-dl",
parakeetBin:
process.env.PARAKEET_STITCH_BIN ??
- path.join(monorepoRoot, "scripts", "parakeet-stitch.mjs"),
+ under(monorepoRoot, "scripts", "parakeet-stitch.mjs"),
parakeetCliBin: process.env.PARAKEET_CLI ?? "parakeet-cli",
parakeetModel: process.env.PARAKEET_MODEL ?? "",
// Speaker-diarization wrapper, same shape as parakeetBin: a script we own,
@@ -258,23 +293,41 @@ export function getPaths(): Paths {
// today, pyannote later) without touching any caller.
diarizeBin:
process.env.DIARIZE_BIN ??
- path.join(monorepoRoot, "scripts", "diarize.mjs"),
+ under(monorepoRoot, "scripts", "diarize.mjs"),
ollamaUrl: (process.env.OLLAMA_URL ?? "http://127.0.0.1:11434").replace(
/\/+$/,
"",
),
claudeBin: process.env.CLAUDE_BIN ?? "claude",
+ configDir,
+ sourceScrubFile:
+ process.env.SOURCE_SCRUB_FILE ?? path.join(configDir, "source-scrub.txt"),
+ sourceDenylistFile:
+ process.env.SOURCE_DENYLIST_FILE ??
+ path.join(configDir, "source-denylist.txt"),
+ sourceScratchDir: process.env.ARCHILYZER_SOURCE_SCRATCH ?? os.tmpdir(),
+ sourceHistoryCacheDir: path.join(
+ process.env.XDG_CACHE_HOME || path.join(os.homedir(), ".cache"),
+ "archilyzer",
+ "source-history",
+ ),
+ stagitBin: process.env.STAGIT_BIN ?? "stagit",
};
return cached;
}
+// Walks up from the working directory to the workspace root; falls back to
+// the working directory itself (an app's own directory). Every path op on
+// these cwd-derived values opts out of Turbopack's tracing, so what the
+// fallback evaluates to at build time never becomes an asset reference.
function findMonorepoRoot(): string {
- let dir = process.cwd();
+ const start = path.resolve(/* turbopackIgnore: true */ process.cwd());
+ let dir = start;
for (let i = 0; i < 8; i++) {
- if (fs.existsSync(path.join(dir, "pnpm-workspace.yaml"))) return dir;
- const parent = path.dirname(dir);
+ if (fs.existsSync(path.join(/* turbopackIgnore: true */ dir, "pnpm-workspace.yaml"))) return dir;
+ const parent = path.dirname(/* turbopackIgnore: true */ dir);
if (parent === dir) break;
dir = parent;
}
- return process.cwd();
+ return start;
}
diff --git a/common/lib/ports.test.ts b/common/lib/ports.test.ts
@@ -23,6 +23,11 @@ import {
const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../..");
const PACKAGES = ["", "common", "editor", "export", "homepage", "mcp", "umtool", "umtool/report-to-video"];
+// A script's `${NAME:-N}` that is a number and NOT a port, named one by one so
+// every other spelled default is still read as a port: the editor's `start`
+// gives Node's thread pool 16 threads (envVars.ts, UV_THREADPOOL_SIZE).
+const NUMERIC_NOT_PORTS = new Set(["UV_THREADPOOL_SIZE"]);
+
function scripts(dir: string): Record<string, string> {
const file = path.join(ROOT, dir, "package.json");
return (JSON.parse(readFileSync(file, "utf8")).scripts ?? {}) as Record<string, string>;
@@ -63,6 +68,7 @@ function found(): Found[] {
for (const [script, line] of Object.entries(scripts(pkg))) {
const where = `${pkg || "."}/package.json "${script}"`;
for (const m of line.matchAll(/\$\{([A-Z0-9_]+):-(\d+)\}/g)) {
+ if (NUMERIC_NOT_PORTS.has(m[1])) continue;
hits.push({ where, name: m[1], port: Number(m[2]) });
}
for (const m of line.matchAll(/--ports\s+(\S+)/g)) {
diff --git a/common/lib/project.ts b/common/lib/project.ts
@@ -32,12 +32,24 @@ export const PROJECT_WORDMARK_LEAD = "Archi";
// valid. Only then is changing this string a cosmetic follow-up.
export const PROJECT_URL = "https://archilyzer.pages.dev";
+// The homepage's Official Instances section, where every archive's header
+// links in place of the old sites dropdown (homepage/app/page.tsx,
+// `id="instances"`).
+export const INSTANCES_URL = `${PROJECT_URL}/#instances`;
+
export const PROJECT_TAGLINE = "Self-hosted, searchable video-transcript archives.";
-// Where a visitor gets the source. A dated snapshot tarball — there is no
-// public git repository (see homepage/app/downloads).
+// Where a visitor gets the source as a tarball. The canonical public copy is
+// the read-only git mirror on the same site (lib/sourceManifest.ts CLONE_URL,
+// homepage/app/source); this is its no-git alternative.
export const PROJECT_DOWNLOADS_URL = `${PROJECT_URL}/downloads/`;
+// The homepage's AI and MCP doc (homepage/content/docs/ai-and-mcp.md): the
+// published contract, the MCP server and its setup, in one place. Every
+// archive's "Use with AI" link goes here, in the same tab (release 16 slice
+// DX), as do corpus.json's `useWithAi` and llms.txt's line of that name.
+export const AI_DOC_URL = `${PROJECT_URL}/docs/ai-and-mcp/`;
+
// The `generator` string stamped into corpus.json and llms.txt, so anything
// that reads an archive machine-side can find the software that built it.
// Shape mirrors the HTML <meta name="generator"> convention.
diff --git a/common/lib/search/leafPipeline.ts b/common/lib/search/leafPipeline.ts
@@ -194,9 +194,20 @@ export function createSearchPipeline(
for (let i = 0; i < needed; i++) worker();
};
+ // `worker()` counts itself in synchronously, before its first await, so
+ // right after ensureWorkers a zero here means none is running or will.
+ const settleIfIdle = () => {
+ if (activeWorkers === 0 && !done) finalize();
+ };
+
// Emit initial snapshot synchronously so the UI clears previous results.
pushUpdate();
ensureWorkers();
+ // No worker started (nothing to scan, or the cap is already met): nobody is
+ // left to call finalize, so settle here. Without it an empty scope — which
+ // "Search in" makes from a plain query, e.g. a posts copy under a tag filter
+ // — never reported done and the search read "searching" for ever.
+ settleIfIdle();
return {
cancel() {
@@ -216,6 +227,7 @@ export function createSearchPipeline(
pushUpdate();
}
ensureWorkers();
+ settleIfIdle();
},
};
}
@@ -325,8 +337,15 @@ export function createPostsSearchPipeline(
for (let i = 0; i < needed; i++) worker();
};
+ // `worker()` counts itself in synchronously, before its first await, so
+ // right after ensureWorkers a zero here means none is running or will.
+ const settleIfIdle = () => {
+ if (activeWorkers === 0 && !done) finalize();
+ };
+
pushUpdate();
ensureWorkers();
+ settleIfIdle(); // see createSearchPipeline
return {
cancel() {
@@ -345,6 +364,7 @@ export function createPostsSearchPipeline(
pushUpdate();
}
ensureWorkers();
+ settleIfIdle();
},
};
}
@@ -503,8 +523,15 @@ export function createSubsSearchPipeline(
for (let i = 0; i < needed; i++) worker();
};
+ // `worker()` counts itself in synchronously, before its first await, so
+ // right after ensureWorkers a zero here means none is running or will.
+ const settleIfIdle = () => {
+ if (activeWorkers === 0 && !done) finalize();
+ };
+
pushUpdate();
ensureWorkers();
+ settleIfIdle(); // see createSearchPipeline
return {
cancel() {
@@ -523,6 +550,7 @@ export function createSubsSearchPipeline(
pushUpdate();
}
ensureWorkers();
+ settleIfIdle();
},
};
}
diff --git a/common/lib/search/searchIn.test.ts b/common/lib/search/searchIn.test.ts
@@ -0,0 +1,347 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import {
+ applySearchIn,
+ newGroup,
+ newLeaf,
+ type GroupNode,
+ type SearchIn,
+} from "../searchQuery";
+import {
+ runQueryTree,
+ type LayerCache,
+ type TreeProgress,
+} from "../searchEval";
+import type { TranscriptDetail } from "../transcripts";
+import type { SubsDetail } from "../subs";
+import type { Post } from "../posts";
+import {
+ runLeafPipeline,
+ type LayerHit,
+ type LeafController,
+ type LeafFetchers,
+ type LeafProgress,
+ type LeafRunner,
+} from "./leafPipeline";
+import { foldSearchIn } from "./searchIn";
+
+// The "Search in" fold, and the rewrite run end to end through the real tree
+// evaluator with a fake leaf runner: a transcripts leaf reads the kinds ticked,
+// and what the visitor sees is filed under the leaf they built.
+
+// A tiny corpus. v2 is the only video with live chat; p1 is a post.
+const TEXT: Record<string, Record<string, string>> = {
+ transcripts: { v1: "an alpha line", v2: "a beta line", v3: "a gamma line" },
+ chat: { v2: "@u: alpha in chat" },
+ posts: { p1: "a post about alpha" },
+};
+
+const noCache: LayerCache = {
+ key: (a, b) => `${a}__${b}`,
+ getSync: () => null,
+ get: async () => null,
+ put: () => {},
+};
+
+// Matches `leaf.query` against the text for the leaf's scope, one hit per
+// matching slug, and reports it the way runLeafPipeline does (a chat hit
+// carries its track).
+const fakeRunLeaf: LeafRunner = (opts) => {
+ const { leaf, scopeSlugs, emit } = opts;
+ const slugs = new Set<string>();
+ const hits = new Map<string, LayerHit[]>();
+ for (const slug of scopeSlugs) {
+ const text = TEXT[leaf.scope]?.[slug];
+ if (!text || !text.includes(leaf.query)) continue;
+ slugs.add(slug);
+ if (leaf.contributeHits) {
+ hits.set(slug, [
+ {
+ leafId: leaf.id,
+ scope: leaf.scope,
+ ...(leaf.scope === "chat" ? { track: "live_chat" } : {}),
+ start: 1,
+ text,
+ },
+ ]);
+ }
+ }
+ let resolve!: (r: { slugs: Set<string>; hits: Map<string, LayerHit[]> }) => void;
+ const done = new Promise<{ slugs: Set<string>; hits: Map<string, LayerHit[]> }>(
+ (r) => (resolve = r),
+ );
+ queueMicrotask(() => {
+ emit({
+ slugs,
+ hits,
+ totalHits: [...hits.values()].reduce((n, l) => n + l.length, 0),
+ processed: scopeSlugs.length,
+ totalToProcess: scopeSlugs.length,
+ capped: false,
+ done: true,
+ });
+ resolve({ slugs, hits });
+ });
+ const ctrl: LeafController = { cancel() {}, setHitLimit() {}, done };
+ return ctrl;
+};
+
+function run(root: GroupNode, s: SearchIn): Promise<TreeProgress> {
+ const rewritten = applySearchIn(root, s);
+ return new Promise((resolve) => {
+ runQueryTree({
+ root: rewritten.root,
+ runtime: { runLeaf: fakeRunLeaf, cache: noCache },
+ // The session's global scope: the videos, and the posts when Posts is on.
+ globalScope: ["v1", "v2", "v3", ...(s.posts ? ["p1"] : [])],
+ summaries: [],
+ chatScopeSlugs: new Set(["v2"]),
+ postScopeSlugs: new Set(["p1"]),
+ initialHitLimit: 500,
+ concurrency: 2,
+ flushIntervalMs: 10,
+ emit: (p) => {
+ if (p.done) resolve(foldSearchIn(p, rewritten));
+ },
+ });
+ });
+}
+
+const ALPHA = () => newGroup({ id: "root", children: [newLeaf({ id: "a", query: "alpha" })] });
+
+test("Transcripts on: the transcript cues only (today's plain query)", async () => {
+ const p = await run(ALPHA(), { transcripts: true, posts: false, chat: false });
+ assert.deepEqual([...p.slugs].sort(), ["v1"]);
+});
+
+test("Transcripts + Posts (the default): videos and the post, in one result set", async () => {
+ const p = await run(ALPHA(), { transcripts: true, posts: true, chat: false });
+ assert.deepEqual([...p.slugs].sort(), ["p1", "v1"]);
+ assert.equal(p.hits.get("p1")?.[0].scope, "posts");
+ assert.equal(p.hits.get("p1")?.[0].leafId, "a", "filed under the visitor's leaf");
+});
+
+test("Live chat on: a chat hit lands in the same leaf, wearing its track", async () => {
+ const p = await run(ALPHA(), { transcripts: true, posts: false, chat: true });
+ assert.deepEqual([...p.slugs].sort(), ["v1", "v2"]);
+ const chat = p.hits.get("v2")?.[0];
+ assert.equal(chat?.leafId, "a");
+ assert.equal(chat?.scope, "chat");
+ assert.equal(chat?.track, "live_chat");
+ // The copies' states fold into the leaf's: no copy id is left for the
+ // builder to miss, and the count is the union.
+ assert.deepEqual([...p.leafStates.keys()], ["a"]);
+ assert.equal(p.leafStates.get("a")?.slugCount, 2);
+ assert.equal(p.leafStates.get("a")?.totalHits, 2);
+ assert.equal(p.leafStates.get("a")?.active, false);
+});
+
+test("Transcripts off + Live chat on: a word only in the cues returns nothing", async () => {
+ const off: SearchIn = { transcripts: false, posts: false, chat: true };
+ const onlyInCues = newGroup({ children: [newLeaf({ id: "b", query: "beta" })] });
+ assert.equal((await run(onlyInCues, off)).slugs.size, 0);
+ const p = await run(ALPHA(), off);
+ assert.deepEqual([...p.slugs], ["v2"]);
+});
+
+test("a leaf asked for by name ignores the row", async () => {
+ const chatByName = newGroup({
+ children: [newLeaf({ id: "c", query: "alpha", scope: "chat" })],
+ });
+ const p = await run(chatByName, { transcripts: true, posts: true, chat: false });
+ assert.deepEqual([...p.slugs], ["v2"]);
+});
+
+test("NOT reads NOT of the union, and its count is still what it matched", async () => {
+ const root = newGroup({
+ children: [newLeaf({ id: "a", query: "alpha", negate: true })],
+ });
+ const p = await run(root, { transcripts: true, posts: false, chat: true });
+ // alpha is in v1's cues and v2's chat, so only v3 is left.
+ assert.deepEqual([...p.slugs], ["v3"]);
+ assert.equal(p.leafStates.get("a")?.slugCount, 2);
+});
+
+test("the fold before every copy has started: the largest copy, still active", () => {
+ const { origin, unionOf } = applySearchIn(ALPHA(), {
+ transcripts: true,
+ posts: true,
+ chat: true,
+ });
+ const state = (slugCount: number) => ({
+ slugCount,
+ totalHits: slugCount,
+ processed: 1,
+ totalToProcess: 4,
+ capped: false,
+ cached: false,
+ active: false,
+ });
+ const p: TreeProgress = {
+ slugs: new Set(),
+ hits: new Map(),
+ leafStates: new Map([
+ ["a~transcripts", state(3)],
+ ["a~posts", state(1)],
+ ]),
+ // What the evaluator reports while a copy has no result: the whole scope.
+ groupStates: new Map([["a~in", { slugCount: 30_000 }]]),
+ done: false,
+ capped: false,
+ };
+ const folded = foldSearchIn(p, { origin, unionOf });
+ const a = folded.leafStates.get("a");
+ assert.equal(a?.slugCount, 3);
+ assert.equal(a?.active, true);
+ assert.equal(a?.totalHits, 4);
+ assert.equal(a?.processed, 2);
+ assert.equal(a?.totalToProcess, 8);
+});
+
+test("nothing rewritten: the fold hands back the same progress", () => {
+ const p: TreeProgress = {
+ slugs: new Set(["v1"]),
+ hits: new Map(),
+ leafStates: new Map(),
+ groupStates: new Map(),
+ done: true,
+ capped: false,
+ };
+ assert.equal(foldSearchIn(p, { origin: new Map(), unionOf: new Map() }), p);
+});
+
+// ─── An empty scope settles (review M1) ───
+// The rewrite makes copies whose scope can be empty from a plain query: a posts
+// copy under a tag filter or on a channel selection with no posts, a chat copy
+// where no video in scope has chat, a transcripts copy when the Type row keeps
+// no video. The drivers used to start no worker for zero slugs and so never
+// finalize — the query read "searching" for ever. These run the REAL drivers.
+
+const realFetchers: LeafFetchers = {
+ transcript: async (slug) =>
+ ({
+ slug,
+ cues: [{ start: 1, end: 2, text: TEXT.transcripts[slug] ?? "" }],
+ }) as unknown as TranscriptDetail,
+ subs: async (slug) =>
+ ({
+ slug,
+ tracks: { live_chat: [{ start: 1, end: 2, text: TEXT.chat[slug] ?? "" }] },
+ }) as unknown as SubsDetail,
+ post: async (slug) => ({ slug, text: TEXT.posts[slug] ?? "" }) as unknown as Post,
+};
+const realRunLeaf: LeafRunner = (opts) =>
+ runLeafPipeline({ ...opts, fetchers: realFetchers });
+
+function within<T>(p: Promise<T>, ms = 2_000): Promise<T> {
+ return Promise.race([
+ p,
+ new Promise<T>((_, reject) =>
+ setTimeout(() => reject(new Error(`not settled within ${ms} ms`)), ms),
+ ),
+ ]);
+}
+
+for (const scope of ["transcripts", "posts", "chat", "description"] as const) {
+ test(`the real ${scope} driver settles an empty scope, and again after a raised cap`, async () => {
+ const seen: LeafProgress[] = [];
+ const ctrl = runLeafPipeline({
+ leaf: newLeaf({ id: "x", query: "alpha", scope }),
+ scopeSlugs: [],
+ initialHitLimit: 10,
+ concurrency: 2,
+ flushIntervalMs: 5,
+ emit: (p) => seen.push(p),
+ fetchers: realFetchers,
+ });
+ const result = await within(ctrl.done);
+ assert.equal(result.slugs.size, 0);
+ assert.equal(seen.at(-1)?.done, true);
+ // "Show more" on it resumes nothing, and must settle again.
+ ctrl.setHitLimit(1_000);
+ assert.equal(seen.at(-1)?.done, true);
+ });
+}
+
+function runReal(
+ root: GroupNode,
+ s: SearchIn,
+ scopes: { chat: ReadonlySet<string> | null; posts: ReadonlySet<string> | null },
+ // Every folded progress, in order — what the session's readout sees.
+ seen: TreeProgress[] = [],
+): Promise<TreeProgress> {
+ const rewritten = applySearchIn(root, s);
+ return within(
+ new Promise((resolve) => {
+ runQueryTree({
+ root: rewritten.root,
+ runtime: { runLeaf: realRunLeaf, cache: noCache },
+ globalScope: ["v1", "v2", "v3"],
+ summaries: [],
+ chatScopeSlugs: scopes.chat,
+ postScopeSlugs: scopes.posts,
+ initialHitLimit: 500,
+ concurrency: 2,
+ flushIntervalMs: 5,
+ emit: (p) => {
+ const folded = foldSearchIn(p, rewritten);
+ seen.push(folded);
+ if (p.done) resolve(folded);
+ },
+ });
+ }),
+ );
+}
+
+test("a plain query whose posts copy has nothing to read finishes (posts scope null)", async () => {
+ // What the session passes under a tag filter: no posts in scope at all.
+ const seen: TreeProgress[] = [];
+ const p = await runReal(
+ ALPHA(),
+ { transcripts: true, posts: true, chat: false },
+ { chat: null, posts: null },
+ seen,
+ );
+ assert.deepEqual([...p.slugs], ["v1"]);
+ assert.equal(p.leafStates.get("a")?.active, false);
+ // The folded leaf's "searched N/M" only climbs (re-review R-L1): the empty
+ // copy reports 0 of 0, not its parent scope as done, so the readout does
+ // not start full and fall back as the transcripts copy streams.
+ let lastProcessed = 0;
+ let lastFraction = 0;
+ for (const q of seen) {
+ const a = q.leafStates.get("a");
+ if (!a || a.totalToProcess === 0) continue;
+ assert.ok(a.processed >= lastProcessed, `processed fell: ${lastProcessed} → ${a.processed}`);
+ const f = a.processed / a.totalToProcess;
+ assert.ok(f >= lastFraction, `searched fell: ${lastFraction} → ${f}`);
+ lastProcessed = a.processed;
+ lastFraction = f;
+ }
+ assert.equal(p.leafStates.get("a")?.processed, 3);
+ assert.equal(p.leafStates.get("a")?.totalToProcess, 3, "the empty copy adds nothing to the total");
+});
+
+test("… and with an empty posts set (a channel selection with no posts channel)", async () => {
+ const p = await runReal(ALPHA(), { transcripts: true, posts: true, chat: false }, {
+ chat: null,
+ posts: new Set(),
+ });
+ assert.deepEqual([...p.slugs], ["v1"]);
+ assert.equal(p.leafStates.get("a")?.active, false);
+});
+
+test("… and with Live chat ticked where no video in scope has chat", async () => {
+ const p = await runReal(ALPHA(), { transcripts: true, posts: false, chat: true }, {
+ chat: new Set(),
+ posts: null,
+ });
+ assert.deepEqual([...p.slugs], ["v1"]);
+ // Posts only, with no posts: nothing, and done.
+ const none = await runReal(ALPHA(), { transcripts: false, posts: true, chat: false }, {
+ chat: null,
+ posts: null,
+ });
+ assert.equal(none.slugs.size, 0);
+ assert.equal(none.leafStates.get("a")?.active, false);
+});
diff --git a/common/lib/search/searchIn.ts b/common/lib/search/searchIn.ts
@@ -0,0 +1,94 @@
+// The "Search in" fold — a run of the rewritten tree, read as the tree the
+// visitor built.
+//
+// `applySearchIn` (lib/searchQuery.ts) turns a "transcripts" leaf that reads
+// two or three kinds into an OR over one copy per kind, each copy with an id of
+// its own. The run reports under those ids; the builder, the result cards and
+// the session all know only the visitor's leaf. This fold files every copy's
+// hits and counts back under that leaf, so a chat hit lands in the same
+// section of the same video row as the transcript hits, wearing its track.
+//
+// Pure: it copies what it changes and returns the progress it was given when
+// the rewrite changed nothing.
+
+import type { SearchInTree } from "../searchQuery";
+import type { LeafState, TreeProgress } from "../searchEval";
+import type { LayerHit } from "./leafPipeline";
+
+export function foldSearchIn(
+ p: TreeProgress,
+ tree: Pick<SearchInTree, "origin" | "unionOf">,
+): TreeProgress {
+ const { origin, unionOf } = tree;
+ if (origin.size === 0) return p;
+
+ const hits = new Map<string, LayerHit[]>();
+ for (const [slug, list] of p.hits) {
+ let changed = false;
+ const out = list.map((h) => {
+ const to = origin.get(h.leafId);
+ if (to === undefined) return h;
+ changed = true;
+ return { ...h, leafId: to };
+ });
+ hits.set(slug, changed ? out : list);
+ }
+
+ // Each copy's state, grouped under the leaf it stands for. A copy that has
+ // not started yet has no state at all.
+ const copies = new Map<string, LeafState[]>();
+ const copiesExpected = new Map<string, number>();
+ for (const to of origin.values()) {
+ copiesExpected.set(to, (copiesExpected.get(to) ?? 0) + 1);
+ }
+ const leafStates = new Map<string, LeafState>();
+ for (const [id, state] of p.leafStates) {
+ const to = origin.get(id);
+ if (to === undefined) {
+ leafStates.set(id, state);
+ continue;
+ }
+ const list = copies.get(to) ?? [];
+ list.push(state);
+ copies.set(to, list);
+ }
+
+ for (const [to, states] of copies) {
+ const allStarted = states.length === (copiesExpected.get(to) ?? 0);
+ // The OR group's state is the union of what the copies matched, inside the
+ // same scope the leaf itself would have read — the count the leaf shows.
+ // Until every copy has started it is not: the evaluator reads a copy with
+ // no result yet as "everything in scope", so the union would briefly be the
+ // whole scope. Until then the largest copy is the count (a lower bound).
+ const groupId = unionOf.get(to);
+ const union = groupId ? p.groupStates.get(groupId) : undefined;
+ let slugCount = 0;
+ if (allStarted && union) slugCount = union.slugCount;
+ else for (const s of states) slugCount = Math.max(slugCount, s.slugCount);
+ let totalHits = 0;
+ let processed = 0;
+ let totalToProcess = 0;
+ let capped = false;
+ let cached = true;
+ let active = !allStarted;
+ for (const s of states) {
+ totalHits += s.totalHits;
+ processed += s.processed;
+ totalToProcess += s.totalToProcess;
+ capped ||= s.capped;
+ cached &&= s.cached;
+ active ||= s.active;
+ }
+ leafStates.set(to, {
+ slugCount,
+ totalHits,
+ processed,
+ totalToProcess,
+ capped,
+ cached,
+ active,
+ });
+ }
+
+ return { ...p, hits, leafStates };
+}
diff --git a/common/lib/searchEval.ts b/common/lib/searchEval.ts
@@ -345,16 +345,30 @@ async function runLeaf(
// Fully-network leaves (transcripts / chat). Try the layer cache first.
const scopeArr = Array.from(effectiveScope);
+ // Nothing to read — a posts leaf on a site or selection with no posts, a
+ // chat leaf where no video in scope has chat, a video leaf over posts only.
+ // "Search in" makes such leaves from a plain query, so settle it here, at
+ // once and without a cache lookup: matched nothing, done. (The drivers
+ // settle an empty scope too; this skips the cache round trip.)
+ if (scopeArr.length === 0) {
+ return applyCached(
+ leaf,
+ 0,
+ { slugs: new Set(), hits: new Map() },
+ ctx,
+ /*cached*/ false,
+ );
+ }
const scopeHash = hashSlugs(scopeArr);
const key = ctx.runtime.cache.key(canonicalHash(leaf), scopeHash);
const cachedSync = ctx.runtime.cache.getSync(key);
if (cachedSync) {
- return applyCached(leaf, parentScope, cachedSync, ctx, /*cached*/ true);
+ return applyCached(leaf, scopeArr.length, cachedSync, ctx, /*cached*/ true);
}
const cached = await ctx.runtime.cache.get(key);
if (ctx.cancelled) return new Set();
if (cached) {
- return applyCached(leaf, parentScope, cached, ctx, /*cached*/ true);
+ return applyCached(leaf, scopeArr.length, cached, ctx, /*cached*/ true);
}
// Cache miss — kick off a real pipeline. Pure-filter leaves (no hit
@@ -414,9 +428,16 @@ async function runLeaf(
return new Set(result.slugs);
}
+// `scopeSize` is the size of the scope the leaf READ — its effective scope,
+// not its parent's — so its progress is "N of N" for the slugs it covered. The
+// parent scope over-counted: a chat leaf narrowed to the videos with chat, or
+// a "Search in" copy, reported the whole parent as processed, and the folded
+// plain leaf's "searched N/M" started full and then fell back as its other
+// copies streamed. An empty scope reports 0 of 0, which the session's
+// progress readout skips.
function applyCached(
leaf: LeafNode,
- parentScope: Set<string>,
+ scopeSize: number,
cached: CachedResult,
ctx: EvalCtx,
isCacheHit: boolean,
@@ -438,8 +459,8 @@ function applyCached(
setLeafState(ctx, leaf.id, {
slugCount: slugs.size,
totalHits: countHits(hits),
- processed: parentScope.size,
- totalToProcess: parentScope.size,
+ processed: scopeSize,
+ totalToProcess: scopeSize,
capped: false,
cached: isCacheHit,
active: false,
diff --git a/common/lib/searchQuery.test.ts b/common/lib/searchQuery.test.ts
@@ -0,0 +1,201 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import {
+ SEARCH_IN_DEFAULT,
+ applySearchIn,
+ canonicalHash,
+ isGroup,
+ isLeaf,
+ newGroup,
+ newLeaf,
+ searchInReadsNothing,
+ searchInUnderTags,
+ stringifyRoot,
+ type GroupNode,
+ type LeafNode,
+ type SearchIn,
+} from "./searchQuery";
+
+// The "Search in" rewrite (release 16, slice CK): what a "transcripts" leaf
+// reads is the row's business, a leaf asked for by name is not.
+
+const T_ONLY: SearchIn = { transcripts: true, posts: false, chat: false };
+
+function tree(...children: GroupNode["children"]): GroupNode {
+ return newGroup({ id: "root", children });
+}
+
+test("the default row with no posts corpus changes nothing — the very same root", () => {
+ const root = tree(newLeaf({ id: "a", query: "alpha" }));
+ const r = applySearchIn(root, T_ONLY);
+ assert.equal(r.root, root);
+ assert.equal(r.origin.size, 0);
+ assert.equal(r.unionOf.size, 0);
+});
+
+test("the ruling's default is Transcripts and Posts on, Live chat off", () => {
+ assert.deepEqual(SEARCH_IN_DEFAULT, {
+ transcripts: true,
+ posts: true,
+ chat: false,
+ });
+});
+
+test("Transcripts + Live chat: the leaf becomes OR(transcripts, chat) under derived ids", () => {
+ const root = tree(newLeaf({ id: "a", query: "alpha" }));
+ const r = applySearchIn(root, { transcripts: true, posts: false, chat: true });
+ const [or] = r.root.children;
+ assert.ok(isGroup(or));
+ assert.equal(or.id, "a~in");
+ assert.equal(or.op, "OR");
+ assert.equal(or.negate, false);
+ assert.deepEqual(
+ or.children.map((c) => [c.id, (c as LeafNode).scope, (c as LeafNode).query]),
+ [
+ ["a~transcripts", "transcripts", "alpha"],
+ ["a~chat", "chat", "alpha"],
+ ],
+ );
+ assert.deepEqual(
+ [...r.origin],
+ [
+ ["a~transcripts", "a"],
+ ["a~chat", "a"],
+ ],
+ );
+ assert.deepEqual([...r.unionOf], [["a", "a~in"]]);
+ // The root keeps its id; nothing is mutated.
+ assert.equal(r.root.id, "root");
+ assert.equal(root.children[0].id, "a");
+});
+
+test("Live chat only: the same leaf, scope chat, same id — nothing to fold", () => {
+ const root = tree(newLeaf({ id: "a", query: "alpha", useRegex: true }));
+ const r = applySearchIn(root, { transcripts: false, posts: false, chat: true });
+ const [leaf] = r.root.children;
+ assert.ok(isLeaf(leaf));
+ assert.equal(leaf.id, "a");
+ assert.equal(leaf.scope, "chat");
+ assert.equal(leaf.useRegex, true);
+ assert.equal(r.origin.size, 0);
+ assert.equal(r.unionOf.size, 0);
+});
+
+test("Posts only: the same leaf, scope posts", () => {
+ const root = tree(newLeaf({ id: "a", query: "alpha" }));
+ const r = applySearchIn(root, { transcripts: false, posts: true, chat: false });
+ const [leaf] = r.root.children;
+ assert.ok(isLeaf(leaf));
+ assert.equal(leaf.scope, "posts");
+ assert.equal(leaf.id, "a");
+});
+
+test("all three: one copy per kind, in a fixed order", () => {
+ const root = tree(newLeaf({ id: "a", query: "alpha" }));
+ const r = applySearchIn(root, { transcripts: true, posts: true, chat: true });
+ const [or] = r.root.children;
+ assert.ok(isGroup(or));
+ assert.deepEqual(
+ or.children.map((c) => (c as LeafNode).scope),
+ ["transcripts", "posts", "chat"],
+ );
+ assert.equal(r.origin.size, 3);
+});
+
+test("nothing ticked: the tree is left as it is (it reads its transcripts)", () => {
+ const root = tree(newLeaf({ id: "a", query: "alpha" }));
+ const nothing = { transcripts: false, posts: false, chat: false };
+ assert.ok(searchInReadsNothing(nothing));
+ assert.ok(!searchInReadsNothing(T_ONLY));
+ assert.ok(!searchInReadsNothing({ transcripts: false, posts: false, chat: true }));
+ assert.equal(applySearchIn(root, nothing).root, root);
+});
+
+test("a leaf asked for by name, and an empty leaf, are not the row's business", () => {
+ const byName = ["chat", "posts", "metadata", "description", "tags"] as const;
+ const root = tree(
+ ...byName.map((scope) => newLeaf({ id: scope, query: "alpha", scope })),
+ newLeaf({ id: "empty", query: " " }),
+ );
+ const r = applySearchIn(root, { transcripts: false, posts: true, chat: true });
+ assert.equal(r.root, root, "no leaf changed, so the root is the same object");
+});
+
+test("a negated leaf reads NOT of the union: the OR sits inside a negated AND", () => {
+ const root = tree(
+ newLeaf({ id: "a", query: "alpha", negate: true, contributeHits: false }),
+ );
+ const r = applySearchIn(root, { transcripts: true, posts: false, chat: true });
+ const [not] = r.root.children;
+ assert.ok(isGroup(not));
+ assert.equal(not.id, "a~not");
+ assert.equal(not.op, "AND");
+ assert.equal(not.negate, true);
+ const [or] = not.children;
+ assert.ok(isGroup(or));
+ assert.equal(or.id, "a~in");
+ assert.equal(or.negate, false, "the union itself is never negated");
+ for (const c of or.children) {
+ assert.ok(isLeaf(c));
+ assert.equal(c.negate, false);
+ assert.equal(c.contributeHits, false, "the copy keeps contributeHits");
+ }
+ assert.deepEqual([...r.unionOf], [["a", "a~in"]]);
+});
+
+test("nested: only the transcripts leaf is rewritten; untouched subtrees keep identity", () => {
+ const untouched = newGroup({
+ id: "g2",
+ op: "OR",
+ children: [newLeaf({ id: "m", query: "title", scope: "metadata" })],
+ });
+ const inner = newGroup({
+ id: "g1",
+ op: "AND",
+ children: [newLeaf({ id: "t", query: "beta" })],
+ });
+ const root = tree(untouched, inner);
+ const before = stringifyRoot(root);
+ const hash = canonicalHash(root);
+ const r = applySearchIn(root, { transcripts: true, posts: true, chat: false });
+ assert.equal(r.root.children[0], untouched);
+ const g1 = r.root.children[1];
+ assert.ok(isGroup(g1));
+ assert.equal(g1.id, "g1");
+ assert.equal(g1.children[0].id, "t~in");
+ // The visitor's tree — its URL form and its hash — is untouched.
+ assert.equal(stringifyRoot(root), before);
+ assert.equal(canonicalHash(root), hash);
+});
+
+test("two transcripts leaves each get their own copies", () => {
+ const root = tree(
+ newLeaf({ id: "a", query: "alpha" }),
+ newLeaf({ id: "b", query: "beta" }),
+ );
+ const r = applySearchIn(root, { transcripts: true, posts: true, chat: false });
+ assert.deepEqual(
+ r.root.children.map((c) => c.id),
+ ["a~in", "b~in"],
+ );
+ assert.deepEqual(
+ [...r.origin.values()].sort(),
+ ["a", "a", "b", "b"],
+ );
+});
+
+test("under a tag filter the posts copy is left out, unless posts are all the row reads", () => {
+ const all = { transcripts: true, posts: true, chat: true };
+ assert.equal(searchInUnderTags(all, false), all, "no tag filter: unchanged");
+ assert.deepEqual(searchInUnderTags(all, true), { ...all, posts: false });
+ assert.deepEqual(
+ searchInUnderTags({ transcripts: false, posts: true, chat: true }, true),
+ { transcripts: false, posts: false, chat: true },
+ );
+ // Posts alone stay: the copy reads an empty scope (no results), rather than
+ // the row reading nothing and the leaf falling back to its transcripts.
+ const postsOnly = { transcripts: false, posts: true, chat: false };
+ assert.equal(searchInUnderTags(postsOnly, true), postsOnly);
+ const noPosts = { transcripts: true, posts: false, chat: false };
+ assert.equal(searchInUnderTags(noPosts, true), noPosts);
+});
diff --git a/common/lib/searchQuery.ts b/common/lib/searchQuery.ts
@@ -472,3 +472,121 @@ export function forEachLeaf(
}
for (const c of root.children) forEachLeaf(c, fn);
}
+
+// ─── "Search in" ───
+// The Filters panel's "Search in" row says what the DEFAULT leaf reads: a leaf
+// whose scope is "transcripts" (what `emptyRoot` and every plain query make)
+// reads the transcript cues, the posts corpus and the live-chat track, each
+// when its box is ticked. A leaf asked for BY NAME in the builder ("Live chat",
+// "Posts", "Title / channel", …) is not the row's business and is left as it
+// is. The row is not part of the tree: the tree the visitor built, the `qt=`
+// it writes and the canonical hash stay what they were, and this rewrite is
+// applied to the committed tree just before it runs.
+//
+// `posts` and `chat` are what the row ticks AND the site ships — the caller
+// folds "Posts is ticked but this site has no posts" to false, so a rewrite
+// never asks for a corpus that is not there.
+export type SearchIn = { transcripts: boolean; posts: boolean; chat: boolean };
+
+// Transcripts and posts on, live chat off: the ruling's default.
+export const SEARCH_IN_DEFAULT: Readonly<SearchIn> = {
+ transcripts: true,
+ posts: true,
+ chat: false,
+};
+
+export function searchInReadsNothing(s: SearchIn): boolean {
+ return !s.transcripts && !s.posts && !s.chat;
+}
+
+// Under a curated-tag filter no post can match — a post carries no curated
+// tags — so a posts copy would read an empty scope and cost a posts-manifest
+// load for nothing. It is left out, unless posts are all the row reads: then
+// it stays (and reads nothing, which is the answer) so that the leaf does not
+// fall back to reading its transcripts.
+export function searchInUnderTags(s: SearchIn, tagFilterOn: boolean): SearchIn {
+ if (!tagFilterOn || !s.posts || (!s.transcripts && !s.chat)) return s;
+ return { ...s, posts: false };
+}
+
+export type SearchInTree = {
+ root: GroupNode;
+ // A per-kind copy's leaf id → the id of the "transcripts" leaf it stands
+ // for. The result fold (`lib/search/searchIn.ts`) reads it to file the
+ // copies' hits and counts back under the leaf the visitor sees.
+ origin: ReadonlyMap<string, string>;
+ // The "transcripts" leaf's id → the OR group over its copies, whose group
+ // state is the union the leaf's count should read. Only for a leaf that
+ // reads two or three kinds (one kind is the same leaf with another scope).
+ unionOf: ReadonlyMap<string, string>;
+};
+
+// The kinds in the order their copies are made. The order is cosmetic (an OR's
+// children run in parallel and its hash sorts them); fixed so a test can name it.
+const SEARCH_IN_KINDS = ["transcripts", "posts", "chat"] as const;
+
+// Rewrite every ACTIVE "transcripts" leaf to read what the row ticks:
+// transcripts only → the leaf, unchanged (the same object)
+// one other kind only → the leaf with that scope; same id, same negate
+// two or three kinds → an OR group over one copy per kind, each copy with
+// the leaf's `contributeHits`. A negated leaf is NOT
+// of the union: the OR goes inside a negated
+// one-child AND, so the OR's own state is always the
+// union (the count the leaf shows) whatever `negate`.
+// nothing → the leaf, unchanged. The UI never commits it (the
+// Search button, Enter, Apply and the profile saves
+// all refuse); a tree that arrives with nothing ticked
+// some other way reads its transcripts rather than
+// silently matching nothing.
+// Copies get derived ids (`<id>~posts`, `<id>~in`, …) because every leaf in a
+// run keys its own results by id. An unchanged subtree keeps its identity, so
+// the default row returns the very root it was given.
+export function applySearchIn(root: GroupNode, s: SearchIn): SearchInTree {
+ const origin = new Map<string, string>();
+ const unionOf = new Map<string, string>();
+ const kinds = SEARCH_IN_KINDS.filter((k) => s[k]);
+ if (kinds.length === 0 || (kinds.length === 1 && kinds[0] === "transcripts")) {
+ return { root, origin, unionOf };
+ }
+ const out = rewriteSearchIn(root, kinds, origin, unionOf);
+ return { root: out as GroupNode, origin, unionOf };
+}
+
+function rewriteSearchIn(
+ node: QueryNode,
+ kinds: ReadonlyArray<LayerScope>,
+ origin: Map<string, string>,
+ unionOf: Map<string, string>,
+): QueryNode {
+ if (isLeaf(node)) {
+ if (node.scope !== "transcripts" || !isLeafActive(node)) return node;
+ if (kinds.length === 1) return { ...node, scope: kinds[0] };
+ const union: GroupNode = {
+ kind: "group",
+ id: `${node.id}~in`,
+ op: "OR",
+ negate: false,
+ children: kinds.map((scope) => {
+ const id = `${node.id}~${scope}`;
+ origin.set(id, node.id);
+ return { ...node, id, scope, negate: false };
+ }),
+ };
+ unionOf.set(node.id, union.id);
+ if (!node.negate) return union;
+ return {
+ kind: "group",
+ id: `${node.id}~not`,
+ op: "AND",
+ negate: true,
+ children: [union],
+ };
+ }
+ let changed = false;
+ const children = node.children.map((c) => {
+ const r = rewriteSearchIn(c, kinds, origin, unionOf);
+ if (r !== c) changed = true;
+ return r;
+ });
+ return changed ? { ...node, children } : node;
+}
diff --git a/common/lib/settings.ts b/common/lib/settings.ts
@@ -38,10 +38,10 @@ import {
} from "./workers";
import { migrateSweepsToLanes } from "./laneMigration";
import { migrateMediaRootToLocations } from "./storageLocations";
+import { socialLinksForSave } from "./socialLinks";
import {
clampParallelTranscriptions,
defaultStorage,
- normalizeSocialSvg,
parseSocialLinks,
sanitizeTranscriptionApps,
siteSettingsSchema,
@@ -220,19 +220,18 @@ function deriveWorkerShadow(next: SiteSettings): SiteSettings {
return { ...next, workers, transcriptionApp, transcriptionApps };
}
-// Every social link's SVG normalized for inline use, or a THROW naming the
-// first one that is not safe to inline. The schema's own `parseSocialLinks`
-// only checks shape; this is the write-side half.
-function validatedSocialLinks(value: unknown): SocialLink[] {
- const out: SocialLink[] = [];
- for (const link of parseSocialLinks(value)) {
- const svg = normalizeSocialSvg(link.svg);
- if (svg === null) {
- throw new Error(`Social link "${link.label}" has an invalid SVG`);
- }
- out.push({ ...link, svg });
+// The social links to store: each new or edited SVG normalized for inline use,
+// or a THROW naming the first that is refused; a link whose SVG is unchanged
+// from the file is kept as it is (lib/socialLinks.ts socialLinksForSave). The
+// schema's own `parseSocialLinks` only checks shape; this is the write-side
+// half.
+function validatedSocialLinks(value: unknown, file: string): SocialLink[] {
+ const stored = parseSocialLinks(rawObject(readRawSettings(file)).socialLinks);
+ const r = socialLinksForSave(parseSocialLinks(value), stored);
+ if ("refused" in r) {
+ throw new Error(`Social link "${r.refused.label}" has an invalid SVG: ${r.refused.problem}`);
}
- return out;
+ return r.links;
}
export async function writeSettings(next: SiteSettings): Promise<void> {
@@ -242,10 +241,10 @@ export async function writeSettings(next: SiteSettings): Promise<void> {
// (`transcriptionsPaused`, `downloadsPaused`, `digest.digestsPaused`, the
// inverted `backfill.enabled`) loses them on this write. The gate is
// `autoQueue[lane].held` and nothing else — see lib/pauseGates.ts.
+ const file = getPaths().settingsFile;
const merged = siteSettingsSchema.parse({
...deriveWorkerShadow(next),
- socialLinks: validatedSocialLinks(next.socialLinks),
+ socialLinks: validatedSocialLinks(next.socialLinks, file),
});
- const file = getPaths().settingsFile;
await writeJsonAtomic(file, merged);
}
diff --git a/common/lib/settingsDocs.ts b/common/lib/settingsDocs.ts
@@ -48,6 +48,10 @@ import {
STORAGE_SETTINGS_FIELD_DOCS,
STORAGE_VOLUME_FIELD_DOCS,
} from "./storageLocations";
+import {
+ HEALTH_TIMING_DEFAULTS,
+ STORAGE_HEALTH_SETTINGS_FIELD_DOCS,
+} from "./storageHealthTimings";
// `workers` IS LEFT OUT OF THE EXAMPLE, and that is the one place the example
// is not the literal default object. Its default is `[]`, and a settings.json
@@ -172,6 +176,13 @@ export function blockTables(d: SiteSettings): Partial<Record<keyof SiteSettings,
},
{ path: "storage.locations[]", docs: STORAGE_LOCATION_FIELD_DOCS },
{ path: "storage.locations[].volume", docs: STORAGE_VOLUME_FIELD_DOCS },
+ // Absent from the default block (only a tuned value is written), so the
+ // Default column is each timing's default, not the block's.
+ {
+ path: "storage.health",
+ docs: STORAGE_HEALTH_SETTINGS_FIELD_DOCS,
+ defaults: fromObject(HEALTH_TIMING_DEFAULTS),
+ },
],
buildPipeline: [
{
diff --git a/common/lib/settingsSchema.ts b/common/lib/settingsSchema.ts
@@ -59,6 +59,7 @@ import {
type StorageSettings,
type StorageVolume,
} from "./storageLocations";
+import { sanitizeStorageHealth } from "./storageHealthTimings";
import {
DEFAULT_DIARIZATION_ENGINE,
DEFAULT_DIARIZATION_THRESHOLD,
@@ -551,18 +552,45 @@ export type SocialLink = {
label: string;
url: string;
svg: string;
+ // Stored only when true. What a header does with it: lib/socialLinks.ts.
+ featured?: boolean;
};
export const SOCIAL_LINK_FIELD_DOCS: FieldDocs<SocialLink> = {
label:
- "Visible name, also the accessible label of the icon.",
+ "The link's name: the icon's accessible name and its tooltip, never " +
+ "text beside it. Shown as text only in place of an icon that fails the " +
+ "check at render.",
url:
"Link target: http(s), mailto: or a site-relative path.",
svg:
- "Inline SVG markup. Normalized on save (width/height stripped, " +
- "fill=\"currentColor\", aria-hidden) and rejected when unsafe (script, " +
- "foreignObject, event handlers, javascript: URLs) or when it has no " +
- "viewBox.",
+ "Inline SVG markup: ONE well-formed `<svg>` element, checked when it is " +
+ "saved new or edited and again every time it is rendered (a link whose " +
+ "icon fails at render shows its label instead; `archilyzer doctor` names " +
+ "it). It may contain shapes, groups, defs, gradients, patterns, clip " +
+ "paths, masks, filters, text and animate/animateTransform/set — no " +
+ "script, style block, foreignObject, a, image, title, desc or any HTML " +
+ "element (a title or desc holding text only is removed); SVG " +
+ "presentation attributes plus aria-*, data-* and xmlns:* — no event " +
+ "handler (on…); a `style` attribute of presentation properties only; an " +
+ "href or url(…) only to an id inside the icon, written plainly; no CSS " +
+ "escape, comment or function that loads anything (image-set, image, " +
+ "cross-fade, element, src, paint, @import); ids plain names. A leading " +
+ "XML declaration, a DOCTYPE without an internal subset and comments are " +
+ "removed. Normalized on save: width/height stripped, aria-hidden added, " +
+ "a single-colour icon's fills made fill=\"currentColor\" (an icon of two " +
+ "or more colours keeps them). A root with no viewBox but a numeric width " +
+ "W and height H (unitless or px) is given `viewBox=\"0 0 W H\"`, so a " +
+ "file pasted as downloaded is accepted. A refused save names why; export " +
+ "from a drawing program with presentation attributes rather than a style " +
+ "block (in Inkscape, save as Plain SVG).",
+ featured:
+ "Keep this link in the header on small screens (the editor's \"Keep in " +
+ "header on small screens\"). A narrow header shows only the featured " +
+ "links (up to 4, the last 4 if more are marked; none marked → none, so " +
+ "the name has the room); a wide header shows every link, up to 4, the " +
+ "featured ones kept first, then the last of the rest. The footer shows " +
+ "every link. Written only when true.",
};
// Each field is documented in ARCHIVE_STORAGE_SETTINGS_FIELD_DOCS below (rendered into SETTINGS.md).
@@ -951,10 +979,15 @@ export function sanitizeStorage(value: unknown): StorageSettings {
const savedVideosLocationId = locations.some((l) => l.id === savedWanted)
? savedWanted
: "";
+ // THE DRIVE-HEALTH TIMINGS: each clamped into its range, and kept only where
+ // it differs from its default; a block with nothing left is not written
+ // (lib/storageHealthTimings.ts).
+ const health = sanitizeStorageHealth(r.health);
return {
locations,
defaultLocationId,
...(savedVideosLocationId ? { savedVideosLocationId } : {}),
+ ...(Object.keys(health).length > 0 ? { health } : {}),
};
}
@@ -1281,143 +1314,23 @@ export function parseSocialLinks(input: unknown): SocialLink[] {
const svg = typeof r.svg === "string" ? r.svg : "";
if (!label || !url || !svg) continue;
if (!SOCIAL_URL_RE.test(url)) continue;
- out.push({ label, url, svg });
+ // `featured` only when it is exactly `true`: absent and false read the
+ // same, and a file that never marked a link parses as it always did.
+ out.push({ label, url, svg, ...(r.featured === true ? { featured: true } : {}) });
}
return out;
}
-// A paint that is not a colour: no paint (`none`, `transparent`), a paint
-// server (a gradient or pattern, `url(#…)`, with or without a fallback), or
-// what the element takes from its parent already. Never counted, never swapped.
-const KEPT_PAINT = /^(?:none|transparent|currentcolor|inherit)$|^url\(/i;
-
-// One solid colour however it is spelled, so `#FFF`, `#ffffff`, `white` and
-// `rgb(255, 255, 255)` count as ONE colour of an icon, not four.
-function paintKey(value: string): string {
- const v = value.trim().toLowerCase().replace(/\s+/g, "");
- if (v === "white") return "#ffffff";
- if (v === "black") return "#000000";
- const short = /^#([0-9a-f])([0-9a-f])([0-9a-f])$/.exec(v);
- if (short) return `#${short[1]}${short[1]}${short[2]}${short[2]}${short[3]}${short[3]}`;
- const rgb = /^rgb\((\d{1,3}),(\d{1,3}),(\d{1,3})\)$/.exec(v);
- if (rgb) {
- return `#${rgb
- .slice(1, 4)
- .map((n) => Math.min(255, Number(n)).toString(16).padStart(2, "0"))
- .join("")}`;
- }
- return v;
-}
-
-// Every fill of one tag, mapped: its `fill` attribute and any `fill:`
-// declaration in its `style` (which beats the attribute). `fill-rule`,
-// `fill-opacity`, strokes and gradient stops are not fills and are left alone.
-function mapTagFills(tag: string, paint: (value: string) => string): string {
- return tag
- .replace(
- /(\sfill\s*=\s*)(?:"([^"]*)"|'([^']*)')/gi,
- (_m, pre: string, dq?: string, sq?: string) =>
- dq !== undefined ? `${pre}"${paint(dq)}"` : `${pre}'${paint(sq ?? "")}'`,
- )
- .replace(
- /(\sstyle\s*=\s*)(?:"([^"]*)"|'([^']*)')/gi,
- (_m, pre: string, dq?: string, sq?: string) => {
- const css = (dq ?? sq ?? "").replace(
- /(^|;)(\s*fill\s*:\s*)([^;]*)/gi,
- (_d, lead: string, prop: string, v: string) => `${lead}${prop}${paint(v)}`,
- );
- return dq !== undefined ? `${pre}"${css}"` : `${pre}'${css}'`;
- },
- );
-}
+// The icon's SVG — what it may contain, and its normalized form — is
+// lib/socialSvg.ts (pure, so the render path runs it too). Re-exported here for
+// the importers that reach it through lib/settings.
+export { normalizeSocialSvg, socialSvgProblem } from "./socialSvg";
-// The children, tag by tag. Passed over whole, never read or changed:
-// - a <mask> (its white and black say how much shows through, not what
-// colour) and a <clipPath> (a clip never paints: only its shape counts —
-// Figma exports almost every icon as a path clipped by a
-// `<clipPath><rect fill="white"/></clipPath>`, release 11, O2b) —
-// self-closing first, so an empty `<mask …/>` or `<clipPath …/>` cannot
-// swallow everything up to the next closing tag;
-// - an animation tag, whose `fill="freeze"` / `fill="remove"` is timing.
-const CHILD_TAG =
- /<(?:mask|clipPath)\b[^>]*\/>|<mask\b[\s\S]*?<\/mask\s*>|<clipPath\b[\s\S]*?<\/clipPath\s*>|<[a-zA-Z][^>]*>/gi;
-const PASSED_OVER = /^<(?:mask|clipPath|animate\w*|set)\b/i;
-
-function mapChildFills(body: string, paint: (value: string) => string): string {
- return body.replace(CHILD_TAG, (m) => (PASSED_OVER.test(m) ? m : mapTagFills(m, paint)));
-}
-
-// Normalize an admin-provided SVG snippet for inline use in the export
-// footer. Returns null on anything that looks unsafe or unrenderable.
-// Steps: trim, allowlist-check, strip width/height, force fill="currentColor"
-// + aria-hidden on the root <svg>. Requires a viewBox so the icon scales.
-//
-// A SINGLE-COLOUR icon follows the theme: when its fills — the root's and its
-// children's together, outside a <mask> or <clipPath> — hold at most one solid colour, each
-// becomes `currentColor`, the footer link's colour. Drawn for one background,
-// such an icon vanishes on another (X's official logo is a `<path
-// fill="white">`: 1.11:1 on the light footer). An icon of TWO or more colours
-// draws its shape with them (YouTube's mark is a red rounded rectangle with a
-// white play triangle; flattened, it is a blank rectangle), carries its own
-// contrast, and keeps every colour as pasted. Paints that are not colours
-// (`none`, `url(…)`, `inherit`) are never counted or changed. Idempotent: a
-// themed icon has no solid colour left.
-export function normalizeSocialSvg(raw: string): string | null {
- if (typeof raw !== "string") return null;
- const trimmed = raw.trim();
- if (!trimmed.startsWith("<svg") || !trimmed.endsWith("</svg>")) return null;
- if (/<script\b/i.test(trimmed)) return null;
- if (/<foreignObject\b/i.test(trimmed)) return null;
- if (/<iframe\b/i.test(trimmed)) return null;
- if (/javascript:/i.test(trimmed)) return null;
- if (/\son[a-z]+\s*=/i.test(trimmed)) return null;
- if (/<\?|<!ENTITY/i.test(trimmed)) return null;
-
- const openEnd = trimmed.indexOf(">");
- if (openEnd < 0) return null;
- let opening = trimmed.slice(0, openEnd);
- let body = trimmed.slice(openEnd);
-
- if (!/\sviewBox\s*=\s*"/i.test(opening)) return null;
-
- opening = opening.replace(/\s(width|height)\s*=\s*"[^"]*"/gi, "");
- opening = opening.replace(/\s(width|height)\s*=\s*'[^']*'/gi, "");
-
- const colours = new Set<string>();
- const count = (v: string) => {
- if (!KEPT_PAINT.test(v.trim())) colours.add(paintKey(v));
- return v;
- };
- mapTagFills(opening, count);
- mapChildFills(body, count);
- if (colours.size <= 1) {
- const themed = (v: string) => (KEPT_PAINT.test(v.trim()) ? v : "currentColor");
- opening = mapTagFills(opening, themed);
- body = mapChildFills(body, themed);
- }
-
- if (!/\sfill\s*=/i.test(opening)) {
- opening = opening.replace(/^<svg/i, '<svg fill="currentColor"');
- }
- if (!/\saria-hidden\s*=/i.test(opening)) {
- opening = opening.replace(/^<svg/i, '<svg aria-hidden="true"');
- }
- return opening + body;
-}
-
-// normalizeSocialSvg() deliberately STRIPS width/height so the icon scales to its
-// wrapper. The cost is that a viewBox-only <svg> has no intrinsic size, so before
-// the stylesheet loads on a static host it paints at the replaced-element default
-// (huge) — the "flash of giant social icons" FOUC. sizeSocialSvg() re-injects an
-// intrinsic pixel size at RENDER time (existing site.json files already have the
-// attributes stripped, so this must run on read, not just on write). The size is
-// an *attribute*, not inline style, so a wrapper's `w-*`/`h-*` utilities still win
-// once CSS loads — it only governs the pre-CSS first paint.
-export function sizeSocialSvg(svg: string, px = 20): string {
- if (typeof svg !== "string") return svg;
- if (/^<svg[^>]*\swidth\s*=/i.test(svg)) return svg; // already sized
- return svg.replace(/^<svg\b/i, `<svg width="${px}" height="${px}"`);
-}
+// The render-time half (sizing, id scoping, the header's selection, the
+// render-time check) lives in lib/socialLinks.ts, which imports only the pure
+// socialSvg.ts, so a client tree can use it. Re-exported here for the importers
+// that reach it through lib/settings.
+export { sizeSocialSvg } from "./socialLinks";
export function clampSleepBetweenDownloadsSeconds(value: unknown): number {
const n =
@@ -1595,7 +1508,7 @@ export const siteSettingsSchema = z.object({
"Default social links applied to every site that doesn't define its own. A site inherits these unless its site.json carries an explicit `socialLinks` array — see Site.socialLinks / resolveSocialLinks in common/lib/site.ts. The one presentation field that lives globally so a shared footer doesn't have to be repeated per site.",
),
homepageUrl: settingsField((v): string => normalizeHomepageUrl(v)).describe(
- "Absolute public URL of the family hub (e.g. \"https://archilyzer-hub.pages.dev\"). Every export site links back to it (\"the family\" backlink) when set. Empty = no hub link rendered. Normalized to a trailing-slash-free http(s) URL.",
+ "Absolute public URL of the family hub (e.g. \"https://archilyzer-hub.pages.dev\"). The default for every site's `hubUrl` (a site's own wins): published as `hubUrl` in the site's public `/site.json` and `/corpus.json`, so the hub can tell its member sites from arbitrary added origins. No page links to it (the header's Hub link was removed in release 14). Empty = none published. Normalized to a trailing-slash-free http(s) URL.",
),
savedVideoBackup: settingsField((v): SavedVideoBackupSettings => sanitizeSavedVideoBackup(v)).describe(
"Backup configuration for the saved-video store (Phase 4 of the video-persistence feature). When enabled with a destination, the store is mirrored there (additively, no deletes) with a per-backup manifest, and the sync scheduler runs the backup on the configured cadence. See common/controller/backupSavedVideos.ts.",
diff --git a/common/lib/site.ts b/common/lib/site.ts
@@ -6,13 +6,14 @@ import { TAGS_FILENAME } from "./curatedTags";
import type { SiteChannelIndex } from "./channelPriority";
import {
getSettings,
- normalizeSocialSvg,
parseSocialLinks,
type SiteSettings,
type SocialLink,
} from "./settings";
+import { socialLinksForSave } from "./socialLinks";
import { readJsonFileSync, writeJsonAtomic } from "./jsonFile-server";
import {
+ isListedSite,
isValidSiteId,
parseSite,
parseSiteUrl,
@@ -138,7 +139,9 @@ export type CrossSiteLink = { siteId: string; title: string; url: string };
export type CrossSiteGroup = { label?: string; sites: CrossSiteLink[] };
// Resolve the footer's cross-site list for `current` against the full pool.
-// Siblings that lack a siteUrl (or are `current`) are not linkable and dropped.
+// Siblings that lack a siteUrl (or are `current`) are not linkable and dropped,
+// and so is an unlisted sibling (`listed: false`, isListedSite) — even one a
+// featured group names. An unlisted `current` still lists its siblings.
// `current.relatedSites` groups render first, in order, each filtered to known
// linkable ids (unknown/used/self skipped, empty groups dropped). Every still-
// unused sibling lands in a trailing remainder group — unlabeled when there
@@ -149,7 +152,7 @@ export function resolveRelatedSites(
): CrossSiteGroup[] {
const byId = new Map<string, CrossSiteLink>();
for (const s of all) {
- if (s.siteId === current.siteId || !s.siteUrl) continue;
+ if (s.siteId === current.siteId || !s.siteUrl || !isListedSite(s)) continue;
byId.set(s.siteId, { siteId: s.siteId, title: s.siteTitle, url: s.siteUrl });
}
const used = new Set<string>();
@@ -211,6 +214,16 @@ export function defaultSiteId(paths: Paths = getPaths()): string | null {
return ids.length === 1 ? ids[0] : null;
}
+// The social links a JSON file on disk holds now (shape-checked only), or none.
+function storedSocialLinks(file: string): SocialLink[] {
+ try {
+ const raw = JSON.parse(fs.readFileSync(file, "utf8")) as { socialLinks?: unknown };
+ return parseSocialLinks(raw?.socialLinks);
+ } catch {
+ return [];
+ }
+}
+
export async function writeSite(
site: Site,
paths: Paths = getPaths(),
@@ -230,16 +243,18 @@ export async function writeSite(
}
// undefined socialLinks = inherit the global default; only validate/persist a
// key when the site explicitly overrides (an array, even empty).
+ // A link whose SVG is unchanged from the file on disk is kept as it is; a new
+ // or edited one is checked (lib/socialLinks.ts socialLinksForSave).
let socialLinks: SocialLink[] | undefined;
if (site.socialLinks !== undefined) {
- socialLinks = [];
- for (const link of parseSocialLinks(site.socialLinks)) {
- const svg = normalizeSocialSvg(link.svg);
- if (svg === null) {
- throw new Error(`Social link "${link.label}" has an invalid SVG`);
- }
- socialLinks.push({ ...link, svg });
+ const r = socialLinksForSave(
+ parseSocialLinks(site.socialLinks),
+ storedSocialLinks(siteConfigFile(paths, site.siteId)),
+ );
+ if ("refused" in r) {
+ throw new Error(`Social link "${r.refused.label}" has an invalid SVG: ${r.refused.problem}`);
}
+ socialLinks = r.links;
}
await writeJsonAtomic(
siteConfigFile(paths, site.siteId),
diff --git a/common/lib/siteColor.test.ts b/common/lib/siteColor.test.ts
@@ -6,10 +6,10 @@ import { resolveAccent } from "./accent";
import { ACCENTS, ACCENT_IDS, BASE_GROUNDS, BASE_GROUND_IDS, contrastRatio, type BaseGround } from "./brand";
// What the browser paints for a perBaseColor value on `base`: tokens.css sets
-// `--base-<base>` to 1 and the other two to 0 (`null`: no token sheet at all,
+// `--base-<base>` to 1 and the other to 0 (`null`: no token sheet at all,
// so every var() takes its fallback). Returns "#rrggbb".
function paint(css: string, base: BaseGround | null): string {
- const flags = css.replace(/var\(--base-(light|sepia|dark), ([01])\)/g, (_, b, fallback) =>
+ const flags = css.replace(/var\(--base-(light|dark), ([01])\)/g, (_, b, fallback) =>
base === null ? fallback : b === base ? "1" : "0",
);
const channels = [...flags.matchAll(/calc\(([^()]*)\)/g)].map((m) =>
@@ -38,7 +38,7 @@ test("siteColor: a named accent is its per-base swatch, whatever hex rides along
test("siteColor: a custom hex is fitted to each base, as the site's own pages fit it", () => {
// A pale hex: 1.43:1 on the light ground as published, so the old card
- // painted it nearly invisible on light and sepia.
+ // painted it nearly invisible on light.
const pale = "#f4c2d7";
const fitted = resolveAccent(pale);
const css = siteColor({ accent: "#F4C2D7" }, seriesColor(0));
@@ -49,8 +49,8 @@ test("siteColor: a custom hex is fitted to each base, as the site's own pages fi
}
// Fitted, not as published, where the ground needs it; kept where it reads.
assert.deepEqual(
- { light: paint(css, "light"), sepia: paint(css, "sepia"), dark: paint(css, "dark") },
- { light: "#846974", sepia: "#7c636e", dark: pale },
+ { light: paint(css, "light"), dark: paint(css, "dark") },
+ { light: "#846974", dark: pale },
);
// A hex that already reads on a ground is kept there exactly (#cc3366 on
// light), and the chart colour is never used for one.
@@ -59,7 +59,7 @@ test("siteColor: a custom hex is fitted to each base, as the site's own pages fi
});
test("perBaseColor: one value, each base's colour; the light one with no token sheet", () => {
- const v = { light: "#010203", sepia: "#a0b0c0", dark: "#ffeedd" };
+ const v = { light: "#010203", dark: "#ffeedd" };
const css = perBaseColor(v);
for (const base of BASE_GROUND_IDS) assert.equal(paint(css, base), v[base]);
assert.equal(paint(css, null), v.light);
@@ -68,11 +68,11 @@ test("perBaseColor: one value, each base's colour; the light one with no token s
});
test("perBaseColor: anything but a #rrggbb per base throws, never paints rgb(NaN …)", () => {
- const ok = { light: "#010203", sepia: "#a0b0c0", dark: "#ffeedd" };
+ const ok = { light: "#010203", dark: "#ffeedd" };
for (const bad of ["#fff", "red", "", "var(--brand)", "#12345g", "#1234567"]) {
- assert.throws(() => perBaseColor({ ...ok, sepia: bad }), RangeError, bad);
+ assert.throws(() => perBaseColor({ ...ok, dark: bad }), RangeError, bad);
}
- assert.throws(() => perBaseColor({ light: "#010203" } as never), /dark|sepia/);
+ assert.throws(() => perBaseColor({ light: "#010203" } as never), /dark/);
});
test("fittedHex: a published hex fitted per base; anything else is undefined", () => {
diff --git a/common/lib/siteColor.ts b/common/lib/siteColor.ts
@@ -86,18 +86,18 @@ export function siteChartColors(sites: readonly { accentId?: string }[]): string
return slot.map((k) => seriesColor(k));
}
-// ONE CSS colour that is `values.light` on the light base, `values.sepia` on
-// sepia and `values.dark` on dark — for a colour a component knows and the
-// token sheet cannot (a custom hex is per site). tokens.css sets
-// `--base-light|sepia|dark` to 1 on their own base and 0 on the others, so each
-// channel is a calc() over the three and the browser resolves it to a plain
-// rgb() for whichever base is in force, switching with it. It goes anywhere a
-// colour goes (a background, a border, color-mix()); the fallbacks paint the
-// light value on a page with no token sheet, as `:root` does.
+// ONE CSS colour that is `values.light` on the light base and `values.dark` on
+// dark — for a colour a component knows and the token sheet cannot (a custom
+// hex is per site). tokens.css sets `--base-light|dark` to 1 on their own base
+// and 0 on the other, so each channel is a calc() over the two and the browser
+// resolves it to a plain rgb() for whichever base is in force, switching with
+// it. It goes anywhere a colour goes (a background, a border, color-mix());
+// the fallbacks paint the light value on a page with no token sheet, as
+// `:root` does.
//
// The value is CSS only: never parse it or compare it as a hex. Each value
// must be a `#rrggbb` (resolveAccent's output); anything else THROWS rather
-// than paint `rgb(NaN …)`, which a browser drops without a word. A fourth
+// than paint `rgb(NaN …)`, which a browser drops without a word. A third
// base needs its own flag in tokens.css (themeTokens.test.ts holds exactly one
// 1 per base, over the same BASE_GROUND_IDS this walks).
const RRGGBB = /^#[0-9a-f]{6}$/i;
@@ -124,8 +124,17 @@ export function fittedHex(accent: unknown): string | undefined {
return hex ? perBaseColor(resolveAccent(hex)) : undefined;
}
+// A site's OWN accent as a CSS colour on the base in force — a named accent's
+// swatch, or a custom hex fitted to each base — or undefined for a site with
+// none. Both are text colours: ≥ 4.5:1 on each ground, and above 4:1 on the
+// homepage's card surface, whose instance cards tint the wordmark's lead with
+// it.
+export function siteAccentColor(site: SiteColorSource): string | undefined {
+ if (isAccentId(site.accentId)) return `var(--swatch-${site.accentId})`;
+ return fittedHex(site.accent);
+}
+
// A site's mark colour; `chart` is its siteChartColors entry.
export function siteColor(site: SiteColorSource, chart: string): string {
- if (isAccentId(site.accentId)) return `var(--swatch-${site.accentId})`;
- return fittedHex(site.accent) ?? chart;
+ return siteAccentColor(site) ?? chart;
}
diff --git a/common/lib/siteSchema.test.ts b/common/lib/siteSchema.test.ts
@@ -9,12 +9,14 @@ import type { z } from "zod";
import {
SITE_FIELD_DOCS,
SITE_KEYS,
+ channelsOnlyOnUnlistedSites,
+ isListedSite,
parseSite,
siteFieldsSchema,
siteToDisk,
type Site,
} from "./siteSchema";
-import { getSite, siteConfigFile, writeSite } from "./site";
+import { getSite, resolveRelatedSites, siteConfigFile, writeSite } from "./site";
import type { Paths } from "./paths";
const HERE = path.dirname(fileURLToPath(import.meta.url));
@@ -60,6 +62,7 @@ test("empty, null, [] and a number all read as the defaults, every key emitted",
assert.equal(want.duplicates, true);
assert.equal(want.transcriptDownloads, true);
assert.equal(want.pwa, false);
+ assert.equal(want.listed, true);
assert.deepEqual(want.relatedSites, []);
assert.equal(want.socialLinks, undefined);
assert.ok("socialLinks" in want);
@@ -231,6 +234,7 @@ function fixtures(): Array<[string, unknown]> {
channels: [{ slug: "c1", groupId: "a", order: 3 }, { slug: "c2", groupId: "q" }],
cloudflareProject: "p",
siteUrl: "https://s.example//",
+ listed: false,
relatedSites: [{ siteIds: ["x", "x", "BAD"] }, { label: " ", siteIds: [] }],
pwa: true,
archives: false,
@@ -285,6 +289,59 @@ test("writeSite throws on no groups, a default outside the groups, and an unsafe
assert.equal(fs.existsSync(siteConfigFile(paths, "s")), false);
});
+test("listed: absent reads listed, only an explicit false unlists, and only false is written", async () => {
+ for (const v of [undefined, true, 0, "false", null]) {
+ assert.equal(parseSite("s", { listed: v }).listed, true, String(v));
+ assert.equal("listed" in siteToDisk(parseSite("s", { listed: v })), false, String(v));
+ }
+ assert.equal(parseSite("s", { listed: false }).listed, false);
+ // `true` in a caller's Site is the default, so it is not written either.
+ assert.equal("listed" in siteToDisk({ ...parseSite("s", {}), listed: true }), false);
+
+ // Through the real writer and reader: false survives, absent reads listed.
+ const paths = scratchPaths(await mkdtemp(path.join(os.tmpdir(), "site-")));
+ const hidden = parseSite("s", { siteTitle: "Hidden", siteUrl: "https://h.example", listed: false });
+ await writeSite(hidden, paths);
+ assert.equal(JSON.parse(await readFile(siteConfigFile(paths, "s"), "utf8")).listed, false);
+ assert.deepEqual(getSite("s", paths), hidden);
+ await writeSite({ ...hidden, listed: true }, paths);
+ const disk = JSON.parse(await readFile(siteConfigFile(paths, "s"), "utf8"));
+ assert.equal("listed" in disk, false);
+ assert.equal(getSite("s", paths).listed, true);
+});
+
+test("isListedSite is the key's default; channelsOnlyOnUnlistedSites keeps a shared channel with the listed site", () => {
+ assert.equal(isListedSite({}), true);
+ assert.equal(isListedSite({ listed: true }), true);
+ assert.equal(isListedSite({ listed: false }), false);
+ const sites = [
+ parseSite("shown", { channels: [{ slug: "shared" }, { slug: "mine" }] }),
+ parseSite("hidden", { listed: false, channels: [{ slug: "shared" }, { slug: "secret" }] }),
+ parseSite("hidden2", { listed: false, channels: [{ slug: "secret" }, { slug: "secret2" }] }),
+ ];
+ assert.deepEqual([...channelsOnlyOnUnlistedSites(sites)].sort(), ["secret", "secret2"]);
+ assert.deepEqual([...channelsOnlyOnUnlistedSites([sites[0]])], []);
+});
+
+test("the footer never links an unlisted sibling, and an unlisted site's own footer still lists the rest", () => {
+ const current = parseSite("cur", {
+ siteUrl: "https://cur.example",
+ relatedSites: [{ label: "Friends", siteIds: ["hidden", "shown"] }],
+ });
+ const shown = parseSite("shown", { siteTitle: "Shown", siteUrl: "https://shown.example" });
+ const hidden = parseSite("hidden", { siteTitle: "Hidden", siteUrl: "https://hidden.example", listed: false });
+ const other = parseSite("other", { siteTitle: "Other", siteUrl: "https://other.example" });
+ const ids = (groups: ReturnType<typeof resolveRelatedSites>) =>
+ groups.flatMap((g) => g.sites.map((s) => s.siteId));
+ // Named in a featured group or not, the unlisted site is not linked.
+ assert.deepEqual(ids(resolveRelatedSites(current, [current, shown, hidden, other])), ["shown", "other"]);
+ // The unlisted site is still built as before: its footer lists its siblings.
+ assert.deepEqual(
+ ids(resolveRelatedSites({ ...hidden, relatedSites: [] }, [current, shown, hidden, other])),
+ ["cur", "shown", "other"],
+ );
+});
+
test("writeSite → getSite round-trips, and the file holds only non-defaults", async () => {
const paths = scratchPaths(await mkdtemp(path.join(os.tmpdir(), "site-")));
const site = parseSite("s", { siteTitle: "Mine", archives: false });
@@ -295,4 +352,5 @@ test("writeSite → getSite round-trips, and the file holds only non-defaults",
assert.equal("duplicates" in disk, false);
assert.equal("transcriptDownloads" in disk, false);
assert.equal("pwa" in disk, false);
+ assert.equal("listed" in disk, false);
});
diff --git a/common/lib/siteSchema.ts b/common/lib/siteSchema.ts
@@ -84,6 +84,7 @@ export type Site = {
cloudflareProject?: string;
accent?: string;
siteUrl?: string;
+ listed?: boolean;
relatedSites?: RelatedSiteGroup[];
pwa?: boolean;
archives?: boolean;
@@ -113,9 +114,11 @@ export const SITE_FIELD_DOCS: FieldDocs<Site> = {
cloudflareProject:
"Cloudflare Pages project name this site deploys to (`wrangler pages deploy out --project-name <cloudflareProject>`). Trimmed; blank = none.",
accent:
- 'Per-site brand accent: a named accent id (`signal`, `brass`, `vermilion`, `violet`, `sakura`, `blue`, `green`) or a custom `"#rrggbb"`. It is the site\'s default accent — a reader can pick another. Absent = `signal`, the family default. A custom hex is darkened or lightened per base until it reaches 4.5:1. The public `/site.json` always carries a hex: an id is published as its on-dark value. Any other spelling is dropped.',
+ 'Per-site brand accent: a named accent id (`signal`, `brass`, `vermilion`, `violet`, `sakura`, `blue`, `green`) or a custom `"#rrggbb"`. It is the site\'s accent on every page; a reader does not pick one. Absent = `signal`, the family default. A custom hex is darkened or lightened per base until it reaches 4.5:1. The public `/site.json` always carries a hex: an id is published as its on-dark value. Any other spelling is dropped.',
siteUrl:
"Absolute public URL of this site's deployment, e.g. `https://jeralyzer.pages.dev` (trimmed, trailing slashes removed; anything not absolute http(s) is dropped). Drives the cross-site footer: a site with no siteUrl is omitted from every other site's list.",
+ listed:
+ "Whether the family lists this site. Opt-OUT: absent/true = listed, only an explicit `false` is written. An unlisted site still builds and deploys as before, and its own pages are unchanged; it is left out of the homepage (cards, chart, `/stats`), the hub (members, federated search, `/corpus.json`, `/llms.txt`), every other site's footer, and the published `channel-sites.json` and pooled `stats/`. A channel only unlisted sites expose is in none of the family's public totals; a channel a listed site also exposes is credited to the listed one.",
relatedSites:
"Pulls specific siblings to the front of the footer's cross-site list, in named groups. Siblings not named here fall into a trailing \"Other sites\" group. Absent/empty = one flat list of every sibling.",
pwa:
@@ -129,7 +132,7 @@ export const SITE_FIELD_DOCS: FieldDocs<Site> = {
archiveMaxBytes:
"Per-site served-file size cap in bytes: any archive larger is dropped from what is served and flagged in the manifest, so a capped host (Cloudflare Pages: 25 MB) will not reject the deploy. 0 = no cap. Absent = the global default. Negative or non-numeric values are dropped.",
hubUrl:
- "Per-site override for the hub this site belongs under (the PWA it points visitors toward). Absent = the family default, `settings.json` `homepageUrl`. Surfaced on the public /site.json so a hub can tell member sites from arbitrary added origins.",
+ "Per-site override for the hub this site belongs under. Absent = the family default, `settings.json` `homepageUrl`. Published on the public `/site.json` and `/corpus.json` so a hub can tell member sites from arbitrary added origins; the header does not link to it (release 14).",
};
// siteId shares the group-id grammar: lowercase slug, used as a directory name.
@@ -139,6 +142,36 @@ export function isValidSiteId(id: unknown): id is string {
return typeof id === "string" && SITE_ID_RE.test(id);
}
+// THE ONE PREDICATE for `listed` (site.json's opt-out; absent = listed). Every
+// public output that enumerates the family's sites filters through it: the
+// homepage summary (lib/homepageSummary.ts), channel-sites.json and the pooled
+// stats (controller/poolSummary.ts, controller/buildStats.ts), the hub's
+// member list (bin/compose-hub.ts) and the footer's siblings
+// (lib/site.ts resolveRelatedSites). The editor's own pages list every site.
+// Here, beside the key, and exported from lib/site like isValidSiteId, so the
+// pure summary builder can use it without importing file I/O.
+export function isListedSite(site: Pick<Site, "listed">): boolean {
+ return site.listed !== false;
+}
+
+// The channels whose content belongs to unlisted sites alone: exposed by at
+// least one site, and by no listed one. No public total counts them. A channel
+// a listed site also exposes is not here (it is credited to the listed site),
+// and a channel no site exposes (pool-only) is not here either — the family's
+// instance-wide totals have always counted it.
+export function channelsOnlyOnUnlistedSites(
+ sites: readonly Pick<Site, "listed" | "channels">[],
+): Set<string> {
+ const onListed = new Set<string>();
+ const onUnlisted = new Set<string>();
+ for (const site of sites) {
+ const into = isListedSite(site) ? onListed : onUnlisted;
+ for (const c of site.channels) into.add(c.slug);
+ }
+ for (const slug of onListed) onUnlisted.delete(slug);
+ return onUnlisted;
+}
+
export const SITE_DEFAULT_TITLE = "Transcript Browser";
export const SITE_DEFAULT_DESCRIPTION = "Browse and search video transcripts";
@@ -246,6 +279,8 @@ export const siteFieldsSchema = z.object({
).describe(d.cloudflareProject),
accent: settingsField(parseAccentSetting).describe(d.accent),
siteUrl: settingsField(parseSiteUrl).describe(d.siteUrl),
+ // Opt-out: only an explicit false unlists. Absent/true stays listed.
+ listed: settingsField((v): boolean => v !== false).describe(d.listed),
relatedSites: settingsField(parseRelatedSites).describe(d.relatedSites),
pwa: settingsField((v): boolean => v === true).describe(d.pwa),
// Opt-out: only an explicit false disables. Absent/true stays on.
@@ -335,6 +370,8 @@ export function siteToDisk(site: Site): Site {
: {}),
...(accent ? { accent } : {}),
...(siteUrl ? { siteUrl } : {}),
+ // Listed is the default: only the opt-out is persisted.
+ ...(site.listed === false ? { listed: false } : {}),
...(relatedSites.length > 0 ? { relatedSites } : {}),
...(site.pwa ? { pwa: true } : {}),
// Persist only the non-default: archives is on unless explicitly disabled.
diff --git a/common/lib/socialLinks.test.ts b/common/lib/socialLinks.test.ts
@@ -0,0 +1,191 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import {
+ HEADER_SOCIAL_LINKS_MAX,
+ headerSocialLinks,
+ safeSocialSvg,
+ scopeSvgIds,
+ sizeSocialSvg,
+ socialLinksForSave,
+} from "./socialLinks";
+import { normalizeSocialSvg, parseSocialLinks, type SocialLink } from "./settingsSchema";
+
+const SVG = `<svg aria-hidden="true" fill="currentColor" viewBox="0 0 24 24"><path d="M1 1h2"/></svg>`;
+const link = (label: string, featured?: boolean): SocialLink => ({
+ label,
+ url: `https://${label.toLowerCase()}.example`,
+ svg: SVG,
+ ...(featured ? { featured: true } : {}),
+});
+const labels = (links: SocialLink[]) => links.map((l) => l.label);
+
+test("the header bound is four", () => {
+ assert.equal(HEADER_SOCIAL_LINKS_MAX, 4);
+});
+
+test("wide: four or fewer links → all of them, in order, marked or not", () => {
+ assert.deepEqual(headerSocialLinks([], "wide"), []);
+ assert.deepEqual(labels(headerSocialLinks([link("A")], "wide")), ["A"]);
+ assert.deepEqual(
+ labels(headerSocialLinks([link("A"), link("B", true), link("C"), link("D")], "wide")),
+ ["A", "B", "C", "D"],
+ );
+});
+
+test("wide: six links, none marked → the last four", () => {
+ const six = ["A", "B", "C", "D", "E", "F"].map((l) => link(l));
+ assert.deepEqual(labels(headerSocialLinks(six, "wide")), ["C", "D", "E", "F"]);
+});
+
+test("wide: six links, some marked → the marked kept first, the last of the rest after, in configured order", () => {
+ const six = [link("A", true), link("B"), link("C"), link("D", true), link("E"), link("F")];
+ assert.deepEqual(labels(headerSocialLinks(six, "wide")), ["A", "D", "E", "F"]);
+ const one = [link("A", true), link("B"), link("C"), link("D"), link("E"), link("F")];
+ assert.deepEqual(labels(headerSocialLinks(one, "wide")), ["A", "D", "E", "F"]);
+});
+
+test("wide: more than four marked → the last four marked", () => {
+ const six = ["A", "B", "C", "D", "E", "F"].map((l) => link(l, l !== "F"));
+ assert.deepEqual(labels(headerSocialLinks(six, "wide")), ["B", "C", "D", "E"]);
+});
+
+test("narrow: none marked → none; the name wins and the footer has them all", () => {
+ assert.deepEqual(headerSocialLinks([], "narrow"), []);
+ assert.deepEqual(headerSocialLinks([link("A"), link("B")], "narrow"), []);
+ const six = ["A", "B", "C", "D", "E", "F"].map((l) => link(l));
+ assert.deepEqual(headerSocialLinks(six, "narrow"), []);
+});
+
+test("narrow: the marked ones only, in order; more than four marked → the last four", () => {
+ assert.deepEqual(labels(headerSocialLinks([link("A"), link("B"), link("C", true)], "narrow")), ["C"]);
+ const six = [link("A"), link("B", true), link("C"), link("D", true), link("E"), link("F")];
+ assert.deepEqual(labels(headerSocialLinks(six, "narrow")), ["B", "D"]);
+ const many = ["A", "B", "C", "D", "E", "F"].map((l) => link(l, l !== "F"));
+ assert.deepEqual(labels(headerSocialLinks(many, "narrow")), ["B", "C", "D", "E"]);
+});
+
+test("header: the input is not changed", () => {
+ const six = ["A", "B", "C", "D", "E", "F"].map((l) => link(l, l === "B"));
+ headerSocialLinks(six, "wide");
+ headerSocialLinks(six, "narrow");
+ assert.equal(six.length, 6);
+ assert.deepEqual(labels(six), ["A", "B", "C", "D", "E", "F"]);
+});
+
+test("parseSocialLinks: featured is kept only when exactly true", () => {
+ const parsed = parseSocialLinks([
+ { label: "A", url: "https://a.example", svg: SVG, featured: true },
+ { label: "B", url: "https://b.example", svg: SVG, featured: false },
+ { label: "C", url: "https://c.example", svg: SVG, featured: "yes" },
+ { label: "D", url: "https://d.example", svg: SVG },
+ ]);
+ assert.deepEqual(parsed[0], { label: "A", url: "https://a.example", svg: SVG, featured: true });
+ for (const l of parsed.slice(1)) assert.ok(!("featured" in l), l.label);
+ // An old file — no link marked — parses exactly as it did before the key.
+ const old = [{ label: "A", url: "https://a.example", svg: SVG }];
+ assert.deepEqual(parseSocialLinks(old), old);
+ assert.deepEqual(JSON.parse(JSON.stringify(parseSocialLinks(old))), old);
+});
+
+test("sizeSocialSvg: an intrinsic size on an unsized root, never a second one", () => {
+ assert.ok(sizeSocialSvg(SVG).startsWith('<svg width="20" height="20" aria-hidden'));
+ const sized = `<svg width="8" height="8" viewBox="0 0 8 8"></svg>`;
+ assert.equal(sizeSocialSvg(sized), sized);
+ assert.ok(sizeSocialSvg(SVG, 36).startsWith('<svg width="36" height="36"'));
+});
+
+// A gradient, a clip and a mask, referenced every way an icon references one.
+const REFS =
+ `<svg id="root" viewBox="0 0 8 8"><defs>` +
+ `<linearGradient id="g"><stop offset="0" style="stop-color:#D7DF23"/></linearGradient>` +
+ `<linearGradient id="g2" xlink:href="#g"/>` +
+ `<clipPath id='c'><rect/></clipPath><mask id="m"><rect fill="white"/></mask></defs>` +
+ `<path fill="url(#g)" style="fill:url( '#g2' )" clip-path="url(#c)" mask="url(#m)"/>` +
+ `<use href="#g2"/><a href="#top"/><path fill="url(#gx)"/></svg>`;
+
+test("scopeSvgIds: every id and every reference to one gains the scope", () => {
+ const out = scopeSvgIds(REFS, "s1-0");
+ for (const id of ["root", "g", "g2", "m"]) assert.ok(out.includes(`id="s1-0-${id}"`), id);
+ assert.ok(out.includes(`id='s1-0-c'`));
+ assert.ok(out.includes(`fill="url(#s1-0-g)"`));
+ assert.ok(out.includes(`style="fill:url('#s1-0-g2')"`));
+ assert.ok(out.includes(`clip-path="url(#s1-0-c)"`));
+ assert.ok(out.includes(`mask="url(#s1-0-m)"`));
+ assert.ok(out.includes(`xlink:href="#s1-0-g"`));
+ assert.ok(out.includes(`<use href="#s1-0-g2"/>`));
+ // Not an id of this icon: left alone. So is a colour that looks like one.
+ assert.ok(out.includes(`<a href="#top"/>`));
+ assert.ok(out.includes(`fill="url(#gx)"`));
+ assert.ok(out.includes(`stop-color:#D7DF23`));
+});
+
+test("scopeSvgIds: two scopes of one icon share no id; no ids → unchanged", () => {
+ const ids = (svg: string) => [...svg.matchAll(/\sid=["']([^"']+)["']/g)].map((m) => m[1]);
+ const a = ids(scopeSvgIds(REFS, "a"));
+ const b = ids(scopeSvgIds(REFS, "b"));
+ assert.equal(a.length, 5);
+ assert.ok(a.every((id) => !b.includes(id)));
+ assert.equal(scopeSvgIds(SVG, "a"), SVG);
+});
+
+test("scopeSvgIds: entity-quoted and case-varied references are scoped with their id", () => {
+ const svg =
+ `<svg viewBox="0 0 8 8"><defs><linearGradient ID="g"/></defs>` +
+ `<rect mask="url("#g")" fill="url('#g')"/><use HREF="#g"/></svg>`;
+ const out = scopeSvgIds(svg, "s");
+ assert.ok(out.includes(`ID="s-g"`));
+ assert.ok(out.includes(`mask="url("#s-g")"`));
+ assert.ok(out.includes(`fill="url('#s-g')"`));
+ assert.ok(out.includes(`HREF="#s-g"`));
+});
+
+test("scopeSvgIds: an id that is not a plain name is left alone, so no comment can close early", () => {
+ for (const id of ["->", "-!>"]) {
+ const svg = `<svg viewBox="0 0 8 8"><!-- <path id="${id}"/> <image href="x"/> --></svg>`;
+ assert.equal(scopeSvgIds(svg, "s"), svg, id);
+ }
+});
+
+// Every icon the checker accepts, scoped as a page renders it, still passes the
+// checker: the render never makes markup the save would have refused.
+test("a scoped and sized icon still passes the checker", () => {
+ for (const raw of [
+ `<svg viewBox="0 0 24 24"><defs><linearGradient id="g_1"><stop offset="0" stop-color="#e0a030"/></linearGradient></defs><path fill="url(#g_1)" style="fill:url("#g_1")" d="M0 0"/><use href="#g_1"/></svg>`,
+ `<svg viewBox="0 0 8 8"><clipPath id="c.1"><rect/></clipPath><g clip-path="url(#c.1)"><path fill="#fff" d="M0 0"/></g></svg>`,
+ ]) {
+ const stored = normalizeSocialSvg(raw);
+ assert.ok(stored, raw.slice(0, 60));
+ const rendered = sizeSocialSvg(scopeSvgIds(stored, "sl_S_1_-0"));
+ assert.equal(normalizeSocialSvg(rendered)?.includes("sl_S_1_-0-"), true);
+ }
+});
+
+test("safeSocialSvg: a stored icon that fails the check is not rendered", () => {
+ assert.equal(safeSocialSvg(SVG), SVG);
+ for (const bad of [
+ `<svg/onload="window.__x=1" viewBox="0 0 8 8"><path d="M0 0"/></svg>`,
+ `<svg viewBox="0 0 8 8"><a href="javascript:x()"><rect/></a></svg>`,
+ `<svg viewBox="0 0 8 8"><script>x()</script></svg>`,
+ "not an svg",
+ 42,
+ undefined,
+ ]) {
+ assert.equal(safeSocialSvg(bad), null, String(bad).slice(0, 40));
+ }
+});
+
+test("socialLinksForSave: an unchanged stored icon is kept as it is; a new or edited one is checked", () => {
+ const refusedByNow = `<svg viewBox="0 0 8 8"><style>*{}</style></svg>`;
+ const stored: SocialLink[] = [{ label: "Old", url: "https://old.example", svg: refusedByNow }];
+ // Unchanged (a label or `featured` may change): kept byte-identical.
+ const same = socialLinksForSave([{ ...stored[0], label: "Renamed", featured: true }], stored);
+ assert.deepEqual(same, { links: [{ ...stored[0], label: "Renamed", featured: true }] });
+ // Edited to something refused: the label and the reason, never the markup.
+ const bad = socialLinksForSave([{ ...stored[0], svg: `<svg viewBox="0 0 8 8" onload="x()"></svg>` }], stored);
+ assert.ok("refused" in bad && bad.refused.label === "Old" && /event handler/.test(bad.refused.problem));
+ // New and good: normalized.
+ const fresh = socialLinksForSave([{ label: "New", url: "https://new.example", svg: `<svg viewBox="0 0 8 8"><path d="M0 0"/></svg>` }], stored);
+ assert.ok("links" in fresh && fresh.links[0].svg.startsWith('<svg aria-hidden="true" fill="currentColor"'));
+ // Nothing stored: everything is checked.
+ assert.ok("refused" in socialLinksForSave(stored, undefined));
+});
diff --git a/common/lib/socialLinks.ts b/common/lib/socialLinks.ts
@@ -0,0 +1,133 @@
+// THE SOCIAL ROW'S RENDER-TIME RULES — pure (a type, and the pure icon checker
+// socialSvg.ts), so the shared component (common/components/SocialLinks.tsx) is
+// safe in a server or a client tree. The stored shape is in settingsSchema.ts
+// (`SocialLink`, `parseSocialLinks`).
+
+import type { SocialLink } from "./settingsSchema";
+import { normalizeSocialSvg, socialSvgProblem, SVG_ID_RE } from "./socialSvg";
+
+// A header holds at most this many social links. The footer always holds all.
+export const HEADER_SOCIAL_LINKS_MAX = 4;
+
+// The header's two widths (the ruling of 2026-09-28): a header keeps the
+// site's NAME on a narrow screen and shows only the links marked for it; the
+// rest are in the footer, which always shows every link.
+export type HeaderWidth = "wide" | "narrow";
+
+// Which links a header shows at `width`, in their configured order:
+// - wide: every link, up to HEADER_SOCIAL_LINKS_MAX; with more configured,
+// the `featured` ones are kept first, then the last of the rest fill the
+// row;
+// - narrow: the `featured` ones only, up to HEADER_SOCIAL_LINKS_MAX; none
+// marked → none (the name wins; the footer has them all).
+// Where a bound cuts, the LAST ones are kept: the newest link an operator adds
+// goes at the end of the list, and is the one they most want seen.
+export function headerSocialLinks<T extends Pick<SocialLink, "featured">>(
+ links: readonly T[],
+ width: HeaderWidth,
+): T[] {
+ const featured = links.filter((link) => link.featured === true).slice(-HEADER_SOCIAL_LINKS_MAX);
+ if (width === "narrow") return featured;
+ if (links.length <= HEADER_SOCIAL_LINKS_MAX) return [...links];
+ const room = HEADER_SOCIAL_LINKS_MAX - featured.length;
+ const rest = room > 0 ? links.filter((link) => !featured.includes(link)).slice(-room) : [];
+ const keep = new Set<T>([...featured, ...rest]);
+ return links.filter((link) => keep.has(link));
+}
+
+// normalizeSocialSvg() deliberately STRIPS width/height so the icon scales to its
+// wrapper. The cost is that a viewBox-only <svg> has no intrinsic size, so before
+// the stylesheet loads on a static host it paints at the replaced-element default
+// (huge) — the "flash of giant social icons" FOUC. sizeSocialSvg() re-injects an
+// intrinsic pixel size at RENDER time (existing site.json files already have the
+// attributes stripped, so this must run on read, not just on write). The size is
+// an *attribute*, not inline style, so a wrapper's `w-*`/`h-*` utilities still win
+// once CSS loads — it only governs the pre-CSS first paint.
+export function sizeSocialSvg(svg: string, px = 20): string {
+ if (typeof svg !== "string") return svg;
+ if (/^<svg[^>]*\swidth\s*=/i.test(svg)) return svg; // already sized
+ return svg.replace(/^<svg\b/i, `<svg width="${px}" height="${px}"`);
+}
+
+// THE READ PATH. A stored icon is inlined only if it passes the save-time
+// check AGAIN (socialSvg.ts): a file edited by hand, written by an older build,
+// or read from another checkout never reaches a page unchecked. The result is
+// the normalized SVG, or null — and a caller then shows the link's label as
+// text instead of an icon. Every inlining goes through here.
+export function safeSocialSvg(svg: unknown): string | null {
+ return typeof svg === "string" ? normalizeSocialSvg(svg) : null;
+}
+
+// THE WRITE PATH, for every writer of a social-link list (settings.json, a
+// site.json, homepage.json, and the editor's two forms). A link whose `svg` is
+// byte-identical to a link already stored is kept exactly as it is — whatever
+// the checker now thinks of it — so a save that did not touch the links (a
+// lane pause, a priority, a title) never fails on an icon an older build
+// stored; the render re-checks it and shows the label if it fails, and
+// `archilyzer doctor` names it. Every new or edited icon is normalized, or the
+// save is refused with the link's label and the reason.
+export function socialLinksForSave(
+ next: readonly SocialLink[],
+ stored: readonly SocialLink[] | undefined,
+): { links: SocialLink[] } | { refused: { label: string; problem: string } } {
+ const kept = new Set((stored ?? []).map((l) => l.svg));
+ const links: SocialLink[] = [];
+ for (const link of next) {
+ if (kept.has(link.svg)) {
+ links.push({ ...link });
+ continue;
+ }
+ const svg = normalizeSocialSvg(link.svg);
+ if (svg === null) {
+ return { refused: { label: link.label, problem: socialSvgProblem(link.svg) ?? "it is refused" } };
+ }
+ links.push({ ...link, svg });
+ }
+ return { links };
+}
+
+const ID_ATTR = /\sid\s*=\s*(?:"([^"]+)"|'([^']+)')/gi;
+
+function escapeRegExp(s: string): string {
+ return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+}
+
+// Every `id` inside one inlined icon, and every reference to one (`url(#…)` in
+// an attribute or a style, `href="#…"`, `xlink:href="#…"`), prefixed with
+// `scope`. The same icon is inlined more than once on a page (a header and a
+// footer, and a header renders one row per breakpoint), and an id resolves to
+// the FIRST element that carries it: when that copy is `display: none`, a
+// gradient, mask or clip defined in it does not paint, and every visible copy
+// that points at it loses that part of the icon. Scoping each copy makes each
+// self-contained. The stored SVG is not changed; this runs at render.
+export function scopeSvgIds(svg: string, scope: string): string {
+ if (typeof svg !== "string") return svg;
+ // Only a plain name is prefixed (the checker refuses any other id), so the
+ // prefix can never complete a `-->` or change the markup's structure.
+ const ids = new Set<string>();
+ for (const m of svg.matchAll(ID_ATTR)) {
+ const id = m[1] ?? m[2];
+ if (SVG_ID_RE.test(id)) ids.add(id);
+ }
+ if (ids.size === 0) return svg;
+ // Longest first, so an id that is a prefix of another never wins its match.
+ const alt = [...ids]
+ .sort((a, b) => b.length - a.length)
+ .map(escapeRegExp)
+ .join("|");
+ // A reference's quote may be a character reference (`url("#a")`).
+ const Q = `["']|"|'|"|'|"|'`;
+ return svg
+ .replace(
+ new RegExp(`(\\sid\\s*=\\s*)(["'])(${alt})\\2`, "gi"),
+ (_m, pre: string, q: string, id: string) => `${pre}${q}${scope}-${id}${q}`,
+ )
+ .replace(
+ new RegExp(`url\\(\\s*(${Q})?#(${alt})\\1\\s*\\)`, "g"),
+ (_m, q: string | undefined, id: string) => `url(${q ?? ""}#${scope}-${id}${q ?? ""})`,
+ )
+ .replace(
+ new RegExp(`(\\s(?:xlink:)?href\\s*=\\s*)(["'])#(${alt})\\2`, "gi"),
+ (_m, pre: string, q: string, id: string) => `${pre}${q}#${scope}-${id}${q}`,
+ );
+}
diff --git a/common/lib/socialSvg.ts b/common/lib/socialSvg.ts
@@ -0,0 +1,485 @@
+// THE SOCIAL ICON'S SVG: what one may contain, and its normalized form.
+//
+// Pure, no imports: settingsSchema.ts (every save of a social link — Settings,
+// a site's form, the homepage config) and socialLinks.ts (every render) both run
+// it, so a stored icon is checked again each time it is inlined into a page.
+//
+// AN ALLOWLIST, TOKENIZED. An icon is inlined into every header and footer, and
+// an operator pastes it from anywhere, so a text denylist is not enough: `/`
+// separates attributes as well as whitespace does, a character reference spells
+// `javascript:`, and a transform can join two fragments into a handler. The
+// input is read once, tag by tag, and refused unless:
+// - it is ONE well-formed <svg> element: every tag either self-closes or is
+// closed in order, attributes are separated by HTML whitespace and quoted,
+// and there is no other markup (no <!…> declaration, CDATA or processing
+// instruction; an XML declaration and a DOCTYPE with no internal subset at
+// the very start, and every comment, are removed first);
+// - every element is on ELEMENTS (shapes, groups, gradients, clips, masks,
+// filters, text, and the three animation elements) — so no script,
+// foreignObject, style, a, image, iframe, object, embed, audio or video,
+// none of the HTML elements that break out of SVG, and no title or desc:
+// those two are HTML integration points, where the HTML parser reads a
+// child as HTML and can leave the icon unclosed around the rest of the
+// page. A title or desc holding text only is removed before the check
+// (the link's aria-label names the icon; the root is aria-hidden); one
+// with a child element is refused;
+// - every attribute is on ATTRIBUTES (the SVG presentation, geometry, filter
+// and animation set, plus aria-*, data-* and xmlns:*), and none is an event
+// handler (a name starting with "on");
+// - after decoding character references (numeric and named, with or without
+// the `;`) and dropping the whitespace a browser ignores in a URL, no value
+// holds `javascript:` or `vbscript:`, a backslash (a CSS escape), a CSS
+// comment, `@import`, `expression(`, or a function that can load
+// something (`image-set(`, `-webkit-image-set(`, `image(`, `cross-fade(`,
+// `-webkit-cross-fade(`, `element(`, `-moz-element(`, `src(`, `paint(`);
+// every `url(…)` points at a fragment of this icon, spelled plainly (a
+// quote may be a character reference); an `href` / `xlink:href` is a
+// plain fragment (`#id`, no reference, no space) — so every reference is
+// one scopeSvgIds rewrites; a `style` holds only presentation properties
+// (STYLE_PROPERTIES); an animation never targets anything named `href` or
+// a handler. The same rules cover an animation's to/from/values/by;
+// - every id is a plain name (`^[A-Za-z_][\w.:-]*$`), so a rendered copy can
+// prefix it safely (socialLinks.ts scopeSvgIds).
+// The OUTPUT of the normalization below is checked again the same way, so no
+// transform can assemble what the input check refused.
+
+const ELEMENTS = new Set(
+ [
+ "svg", "g", "defs", "symbol", "use",
+ "path", "rect", "circle", "ellipse", "line", "polyline", "polygon",
+ "text", "tspan",
+ "linearGradient", "radialGradient", "stop", "pattern", "clipPath", "mask",
+ "filter", "feBlend", "feColorMatrix", "feComponentTransfer", "feComposite",
+ "feDropShadow", "feFlood", "feFuncA", "feFuncB", "feFuncG", "feFuncR",
+ "feGaussianBlur", "feMerge", "feMergeNode", "feMorphology", "feOffset",
+ "animate", "animateTransform", "set",
+ ].map((e) => e.toLowerCase()),
+);
+
+const ANIMATION = new Set(["animate", "animatetransform", "set"]);
+
+const ATTRIBUTES = new Set(
+ [
+ // core
+ "id", "class", "style", "lang", "xml:lang", "xml:space", "xmlns", "version",
+ "baseProfile", "role", "focusable",
+ // geometry
+ "viewBox", "preserveAspectRatio", "width", "height", "x", "y", "x1", "y1",
+ "x2", "y2", "cx", "cy", "r", "rx", "ry", "fx", "fy", "fr", "d", "points",
+ "pathLength", "transform", "transform-origin",
+ // paint and presentation
+ "fill", "fill-opacity", "fill-rule", "clip-rule", "clip-path", "clipPathUnits",
+ "mask", "maskUnits", "maskContentUnits", "filter", "filterUnits",
+ "primitiveUnits", "stroke", "stroke-width", "stroke-linecap",
+ "stroke-linejoin", "stroke-miterlimit", "stroke-dasharray",
+ "stroke-dashoffset", "stroke-opacity", "opacity", "color", "display",
+ "visibility", "overflow", "shape-rendering", "text-rendering",
+ "image-rendering", "color-interpolation", "color-interpolation-filters",
+ "color-rendering", "vector-effect", "paint-order", "mix-blend-mode",
+ "isolation", "enable-background",
+ // gradients and patterns
+ "stop-color", "stop-opacity", "offset", "gradientUnits", "gradientTransform",
+ "spreadMethod", "patternUnits", "patternContentUnits", "patternTransform",
+ // text
+ "font-family", "font-size", "font-weight", "font-style", "font-variant",
+ "font-stretch", "text-anchor", "dominant-baseline", "alignment-baseline",
+ "baseline-shift", "letter-spacing", "word-spacing", "text-decoration",
+ "writing-mode", "dx", "dy", "rotate", "textLength", "lengthAdjust",
+ // filter primitives
+ "in", "in2", "result", "stdDeviation", "flood-color", "flood-opacity",
+ "lighting-color", "operator", "k1", "k2", "k3", "k4", "mode", "values",
+ "type", "tableValues", "slope", "intercept", "amplitude", "exponent",
+ "radius", "edgeMode",
+ // animation
+ "attributeName", "attributeType", "from", "to", "by", "dur", "begin", "end",
+ "repeatCount", "repeatDur", "calcMode", "keyTimes", "keySplines",
+ "additive", "accumulate", "restart", "min", "max",
+ // references — a fragment of this icon only (checked below)
+ "href", "xlink:href",
+ ].map((a) => a.toLowerCase()),
+);
+
+const ATTRIBUTE_PATTERNS = [/^aria-[a-z-]+$/, /^data-[a-z0-9_.-]+$/, /^xmlns:[a-z][a-z0-9_.-]*$/];
+
+// An id a rendered copy can prefix (socialLinks.ts): a plain name.
+export const SVG_ID_RE = /^[A-Za-z_][\w.:-]*$/;
+const FRAGMENT_RE = /^#[A-Za-z_][\w.:-]*$/;
+
+// The reasons a pasted icon is refused, by class. The editor shows them after
+// the link's label, as text (React escapes it). None echoes the markup: a tag
+// or attribute NAME is reduced to letters, digits and `_.:-` and cut to 40
+// characters before it is named. Only the refusals a drawing program's export
+// causes say how to export an acceptable file: a style (a `<style>` block or a
+// `style` attribute), `<metadata>`, and an element or attribute in the
+// program's own namespace (`inkscape:`, `sodipodi:`, …); an old DOCTYPE has a
+// sentence of its own. An `<a>`, an `<image>`, a `<title>` with markup or a
+// `src` is not something an export setting removes, so it gets no hint.
+const nameOf = (name: string) => name.replace(/[^A-Za-z0-9_.:-]/g, "").slice(0, 40) || "?";
+const EXPORT_HINT =
+ "export it with presentation attributes rather than a style block (in Inkscape, save as Plain SVG)";
+const fromDrawingProgram = (name: string) => /^(style|metadata)$/i.test(name) || name.includes(":");
+const withHint = (name: string) => (fromDrawingProgram(name) ? `; ${EXPORT_HINT}` : "");
+export const SVG_PROBLEM = {
+ markup: "it is not one well-formed <svg> element",
+ doctype: "it has a DOCTYPE with an internal subset; export it again without one, or delete the DOCTYPE",
+ handler: "it has an event handler attribute",
+ script: "it has a script, or a link that runs one",
+ external: "it links to something outside the icon",
+ style: `it has a style an icon cannot use; ${EXPORT_HINT}`,
+ id: "it has an id an icon cannot use",
+ viewBox: "it has no viewBox, and no numeric width and height to make one from",
+ element: (name: string) => `it has an element an icon has no use for (${nameOf(name)})${withHint(name)}`,
+ attribute: (name: string) => `it has an attribute an icon has no use for (${nameOf(name)})${withHint(name)}`,
+} as const;
+
+// The only properties a `style` attribute may set: presentation, never layout
+// or position (an icon that paints outside its key, or over the page, is
+// refused).
+const STYLE_PROPERTIES = new Set([
+ "fill", "stroke", "stop-color", "stop-opacity", "opacity", "fill-opacity",
+ "stroke-opacity", "stroke-width", "stroke-linecap", "stroke-linejoin",
+ "fill-rule", "clip-rule", "display", "visibility", "paint-order",
+]);
+
+// CSS functions that can load something, whatever attribute or style holds them.
+const LOADING_FUNCTIONS =
+ /(?:-webkit-)?image-set\(|(?:^|[^a-z-])image\(|(?:-webkit-)?cross-fade\(|(?:-moz-)?element\(|(?:^|[^a-z-])src\(|paint\(|@import|expression\(/i;
+
+// A url(…) as scopeSvgIds rewrites it: a plain fragment, its quote literal or a
+// character reference.
+const RAW_URL_FRAGMENT = /url\(\s*(["']|"|'|"|'|"|')?#[A-Za-z_][\w.:-]*\1\s*\)/gi;
+
+// HTML whitespace, the only attribute separator this reads (a `/` between
+// attributes, which HTML also takes, is refused as markup).
+const WS = "[\\t\\n\\f\\r ]";
+const TAG_RE = new RegExp(
+ `<(\\/?)([A-Za-z][A-Za-z0-9_.:-]*)((?:${WS}+[^\\t\\n\\f\\r "'<>\\/=]+(?:${WS}*=${WS}*(?:"[^"]*"|'[^']*'))?)*)(${WS}*)(\\/?)>`,
+ "y",
+);
+const ATTR_RE = new RegExp(
+ `(${WS}+)([^\\t\\n\\f\\r "'<>\\/=]+)(?:${WS}*=${WS}*(?:"([^"]*)"|'([^']*)'))?`,
+ "g",
+);
+
+type Attr = { name: string; value: string; raw: string };
+type Tag = {
+ close: boolean;
+ name: string;
+ attrs: Attr[];
+ trailing: string; // the whitespace before `>` / `/>`
+ selfClose: boolean;
+ start: number;
+ end: number; // index just past `>`
+};
+
+// Character references a browser decodes in an attribute value, so that the
+// checks see what the browser will: numeric (decimal, hex) and named, with or
+// without the `;`. Named ones are matched case-insensitively — decoding MORE
+// than a browser would only makes the checks stricter.
+const NAMED_REFS: Record<string, string> = {
+ amp: "&", lt: "<", gt: ">", quot: '"', apos: "'", colon: ":", tab: "\t",
+ newline: "\n", nbsp: "\u00a0", lpar: "(", rpar: ")", sol: "/", bsol: "\\",
+ num: "#", period: ".", excl: "!", semi: ";", comma: ",", equals: "=",
+ plus: "+", dollar: "$", percnt: "%", ast: "*", lowbar: "_", hyphen: "-",
+ dash: "-", quest: "?", commat: "@", lsqb: "[", rsqb: "]", lcub: "{",
+ rcub: "}", verbar: "|", grave: "`", hat: "^",
+};
+
+export function decodeCharRefs(value: string): string {
+ return value.replace(
+ /&(#[xX][0-9a-fA-F]+|#[0-9]+|[A-Za-z][A-Za-z0-9]*);?/g,
+ (m, ref: string) => {
+ if (ref[0] === "#") {
+ const hex = ref[1] === "x" || ref[1] === "X";
+ const cp = hex ? parseInt(ref.slice(2), 16) : parseInt(ref.slice(1), 10);
+ return Number.isFinite(cp) && cp > 0 && cp <= 0x10ffff ? String.fromCodePoint(cp) : "\ufffd";
+ }
+ return NAMED_REFS[ref.toLowerCase()] ?? m;
+ },
+ );
+}
+
+// Everything a browser ignores inside a URL, and case.
+const squash = (v: string) => v.replace(/[\u0000-\u0020\u007f-\u00a0]+/g, "");
+
+function attrProblem(element: string, a: Attr): string | null {
+ const name = a.name.toLowerCase();
+ if (name.startsWith("on")) return SVG_PROBLEM.handler;
+ if (!ATTRIBUTES.has(name) && !ATTRIBUTE_PATTERNS.some((p) => p.test(name))) {
+ return SVG_PROBLEM.attribute(a.name);
+ }
+ const decoded = decodeCharRefs(a.value);
+ const tight = squash(decoded);
+ const lower = tight.toLowerCase();
+ if (lower.includes("javascript:") || lower.includes("vbscript:") || lower.includes("livescript:")) {
+ return SVG_PROBLEM.script;
+ }
+ // A CSS escape can spell `url(` so no check sees it; a comment can split a
+ // function name. Neither has a place in an icon's value.
+ if (decoded.includes("\\") || decoded.includes("/*")) {
+ return name === "style" ? SVG_PROBLEM.style : SVG_PROBLEM.external;
+ }
+ if (LOADING_FUNCTIONS.test(tight)) return SVG_PROBLEM.external;
+ if (name === "href" || name === "xlink:href") {
+ // Plain, as written: `#a` or ` #a ` would name an id the render's
+ // scoping cannot see.
+ if (!FRAGMENT_RE.test(a.value)) return SVG_PROBLEM.external;
+ }
+ // Every url(…), wherever it is (a fill, a clip, a style): a fragment here,
+ // and every one of them spelled so scopeSvgIds rewrites it.
+ const urls = [...tight.matchAll(/url\(/gi)];
+ for (const m of urls) {
+ const rest = tight.slice(m.index);
+ if (!/^url\((["']?)#[A-Za-z_][\w.:-]*\1\)/i.test(rest)) return SVG_PROBLEM.external;
+ }
+ if (urls.length > 0 && [...a.value.matchAll(RAW_URL_FRAGMENT)].length !== urls.length) {
+ return SVG_PROBLEM.external;
+ }
+ if (name === "style") {
+ for (const decl of decoded.split(";")) {
+ if (!decl.trim()) continue;
+ const colon = decl.indexOf(":");
+ const prop = (colon < 0 ? decl : decl.slice(0, colon)).trim().toLowerCase();
+ if (colon < 0 || !STYLE_PROPERTIES.has(prop)) return SVG_PROBLEM.style;
+ }
+ }
+ if (name === "id" && !SVG_ID_RE.test(a.value)) return SVG_PROBLEM.id;
+ if (ANIMATION.has(element) && name === "attributename") {
+ const target = lower.trim();
+ if (target.includes("href")) return SVG_PROBLEM.external;
+ if (/(^|:)on/.test(target)) return SVG_PROBLEM.handler;
+ }
+ return null;
+}
+
+// One pass over the markup: its tags, or the first problem.
+function scan(src: string): { tags: Tag[] } | { problem: string } {
+ const tags: Tag[] = [];
+ const stack: string[] = [];
+ let i = 0;
+ while (i < src.length) {
+ const lt = src.indexOf("<", i);
+ if (lt < 0) {
+ // Only whitespace may follow the root's closing tag.
+ return stack.length === 0 && tags.length > 0 && !src.slice(i).trim()
+ ? { tags }
+ : { problem: SVG_PROBLEM.markup };
+ }
+ if (stack.length === 0 && tags.length > 0) return { problem: SVG_PROBLEM.markup };
+ if (stack.length === 0 && src.slice(i, lt).trim()) return { problem: SVG_PROBLEM.markup };
+ if (src.startsWith("<!", lt) || src.startsWith("<?", lt)) return { problem: SVG_PROBLEM.markup };
+ TAG_RE.lastIndex = lt;
+ const m = TAG_RE.exec(src);
+ if (!m) return { problem: SVG_PROBLEM.markup };
+ const [whole, slash, name, attrText, trailing, selfSlash] = m;
+ const close = slash === "/";
+ const lname = name.toLowerCase();
+ if (close && (attrText || selfSlash)) return { problem: SVG_PROBLEM.markup };
+ if (!close) {
+ if (lname === "script") return { problem: SVG_PROBLEM.script };
+ if (!ELEMENTS.has(lname)) return { problem: SVG_PROBLEM.element(name) };
+ if (tags.length === 0 && lname !== "svg") return { problem: SVG_PROBLEM.markup };
+ }
+ const attrs: Attr[] = [];
+ for (const a of attrText.matchAll(ATTR_RE)) {
+ attrs.push({ name: a[2], value: a[3] ?? a[4] ?? "", raw: a[0] });
+ }
+ for (const a of attrs) {
+ const p = attrProblem(lname, a);
+ if (p) return { problem: p };
+ }
+ if (close) {
+ if (stack.pop() !== lname) return { problem: SVG_PROBLEM.markup };
+ } else if (selfSlash !== "/") {
+ stack.push(lname);
+ }
+ tags.push({
+ close,
+ name,
+ attrs,
+ trailing,
+ selfClose: selfSlash === "/",
+ start: lt,
+ end: lt + whole.length,
+ });
+ i = lt + whole.length;
+ }
+ return stack.length === 0 && tags.length > 0 ? { tags } : { problem: SVG_PROBLEM.markup };
+}
+
+// What a pasted icon looks like before it is read: trimmed, with an XML
+// declaration and a DOCTYPE (with no internal subset) removed from the very
+// start, and every comment removed.
+function prepare(raw: string): string {
+ return raw
+ .trim()
+ .replace(/^<\?xml\b[^>]*\?>\s*/i, "")
+ .replace(/^<!DOCTYPE\s+svg\b[^>[]*>\s*/i, "")
+ .replace(/<!--[\s\S]*?-->/g, "")
+ .replace(TEXT_ONLY_TITLE, "")
+ .trim();
+}
+
+// A <title> or <desc> holding text only (or nothing), removed before the
+// check: see the header. One with a child element stays, and is refused.
+const TEXT_ONLY_TITLE = /<(title|desc)\b[^<>]*?(?:\/>|>[^<]*<\/\1\s*>)/gi;
+
+// A paint that is not a colour: no paint (`none`, `transparent`), a paint
+// server (a gradient or pattern, `url(#…)`, with or without a fallback), or
+// what the element takes from its parent already. Never counted, never swapped.
+const KEPT_PAINT = /^(?:none|transparent|currentcolor|inherit)$|^url\(/i;
+
+// One solid colour however it is spelled, so `#FFF`, `#ffffff`, `white` and
+// `rgb(255, 255, 255)` count as ONE colour of an icon, not four.
+function paintKey(value: string): string {
+ const v = value.trim().toLowerCase().replace(/\s+/g, "");
+ if (v === "white") return "#ffffff";
+ if (v === "black") return "#000000";
+ const short = /^#([0-9a-f])([0-9a-f])([0-9a-f])$/.exec(v);
+ if (short) return `#${short[1]}${short[1]}${short[2]}${short[2]}${short[3]}${short[3]}`;
+ const rgb = /^rgb\((\d{1,3}),(\d{1,3}),(\d{1,3})\)$/.exec(v);
+ if (rgb) {
+ return `#${rgb
+ .slice(1, 4)
+ .map((n) => Math.min(255, Number(n)).toString(16).padStart(2, "0"))
+ .join("")}`;
+ }
+ return v;
+}
+
+// Every fill of one tag, mapped: its `fill` attribute and any `fill:`
+// declaration in its `style` (which beats the attribute). `fill-rule`,
+// `fill-opacity`, strokes and gradient stops are not fills and are left alone.
+function mapTagFills(tag: string, paint: (value: string) => string): string {
+ return tag
+ .replace(
+ /(\sfill\s*=\s*)(?:"([^"]*)"|'([^']*)')/gi,
+ (_m, pre: string, dq?: string, sq?: string) =>
+ dq !== undefined ? `${pre}"${paint(dq)}"` : `${pre}'${paint(sq ?? "")}'`,
+ )
+ .replace(
+ /(\sstyle\s*=\s*)(?:"([^"]*)"|'([^']*)')/gi,
+ (_m, pre: string, dq?: string, sq?: string) => {
+ const css = (dq ?? sq ?? "").replace(
+ /(^|;)(\s*fill\s*:\s*)([^;]*)/gi,
+ (_d, lead: string, prop: string, v: string) => `${lead}${prop}${paint(v)}`,
+ );
+ return dq !== undefined ? `${pre}"${css}"` : `${pre}'${css}'`;
+ },
+ );
+}
+
+// The children, tag by tag. Passed over whole, never read or changed:
+// - a <mask> (its white and black say how much shows through, not what
+// colour) and a <clipPath> (a clip never paints: only its shape counts —
+// Figma exports almost every icon as a path clipped by a
+// `<clipPath><rect fill="white"/></clipPath>`, release 11, O2b) —
+// self-closing first, so an empty `<mask …/>` or `<clipPath …/>` cannot
+// swallow everything up to the next closing tag;
+// - an animation tag, whose `fill="freeze"` / `fill="remove"` is timing.
+const CHILD_TAG =
+ /<(?:mask|clipPath)\b[^>]*\/>|<mask\b[\s\S]*?<\/mask\s*>|<clipPath\b[\s\S]*?<\/clipPath\s*>|<[a-zA-Z][^>]*>/gi;
+const PASSED_OVER = /^<(?:mask|clipPath|animate\w*|set)\b/i;
+
+function mapChildFills(body: string, paint: (value: string) => string): string {
+ return body.replace(CHILD_TAG, (m) => (PASSED_OVER.test(m) ? m : mapTagFills(m, paint)));
+}
+
+// A root's own size as a viewBox — `0 0 W H` — when its `width` and `height`
+// attributes are both positive numbers, unitless or in px. Anything else (a
+// percentage, `em`, a missing or zero side) is no size.
+const SVG_LENGTH_PX = /^\s*(\d+(?:\.\d+)?|\.\d+)\s*(?:px)?\s*$/i;
+
+function viewBoxFromSize(attrs: readonly Attr[]): string | null {
+ const side = (name: "width" | "height"): string | null => {
+ const a = attrs.find((x) => x.name.toLowerCase() === name);
+ const len = a ? SVG_LENGTH_PX.exec(a.value) : null;
+ const n = len ? Number(len[1]) : NaN;
+ return Number.isFinite(n) && n > 0 ? String(n) : null;
+ };
+ const w = side("width");
+ const h = side("height");
+ return w && h ? `0 0 ${w} ${h}` : null;
+}
+
+// Why a pasted icon cannot be stored, as one of SVG_PROBLEM's sentences, or
+// null when normalizeSocialSvg accepts it.
+export function socialSvgProblem(raw: unknown): string | null {
+ if (typeof raw !== "string") return SVG_PROBLEM.markup;
+ return normalize(raw).problem;
+}
+
+// Normalize an admin-provided SVG snippet for inline use in a header or a
+// footer. Returns null on anything the checks above refuse, or that has no
+// viewBox to scale by. Steps: read and check (above); give a root with no
+// viewBox, but a numeric width and height (unitless or px), `viewBox="0 0 W
+// H"` from them — a vendor's file pasted as downloaded often carries only its
+// size — then strip width/height; theme a single-colour icon; add
+// aria-hidden; check the result again.
+//
+// A SINGLE-COLOUR icon follows the theme: when its fills — the root's and its
+// children's together, outside a <mask> or <clipPath> — hold at most one solid
+// colour, each becomes `currentColor`, the link's colour. Drawn for one
+// background, such an icon vanishes on another (X's official logo is a `<path
+// fill="white">`: 1.11:1 on the light footer). An icon of TWO or more colours
+// draws its shape with them (YouTube's mark is a red rounded rectangle with a
+// white play triangle; flattened, it is a blank rectangle), carries its own
+// contrast, and keeps every colour as pasted. Paints that are not colours
+// (`none`, `url(…)`, `inherit`) are never counted or changed. Idempotent: a
+// themed icon has no solid colour left.
+export function normalizeSocialSvg(raw: string): string | null {
+ if (typeof raw !== "string") return null;
+ return normalize(raw).svg;
+}
+
+function normalize(raw: string): { svg: string | null; problem: string | null } {
+ const refuse = (problem: string) => ({ svg: null, problem });
+ const src = prepare(raw);
+ if (/^<!DOCTYPE\b/i.test(src)) return refuse(SVG_PROBLEM.doctype);
+ if (!src.startsWith("<svg") || !src.endsWith("</svg>")) return refuse(SVG_PROBLEM.markup);
+ const scanned = scan(src);
+ if ("problem" in scanned) return refuse(scanned.problem);
+ const root = scanned.tags[0];
+
+ // The root's opening tag, rebuilt from its own attributes (each exactly as
+ // written), so a size read from inside another attribute's value can never
+ // count, and stripping width/height removes those attributes and nothing else.
+ let attrs = root.attrs;
+ if (!attrs.some((a) => a.name.toLowerCase() === "viewbox")) {
+ const box = viewBoxFromSize(attrs);
+ if (box) attrs = [{ name: "viewBox", value: box, raw: ` viewBox="${box}"` }, ...attrs];
+ }
+ // A viewBox in single quotes has always been refused; it still is.
+ if (!attrs.some((a) => a.name.toLowerCase() === "viewbox" && /^\s+viewBox\s*=\s*"/i.test(a.raw))) {
+ return refuse(SVG_PROBLEM.viewBox);
+ }
+ attrs = attrs.filter((a) => !/^(width|height)$/i.test(a.name));
+ let opening = `<${root.name}${attrs.map((a) => a.raw).join("")}${root.trailing}`;
+ let body = src.slice(root.end - 1); // from the root's `>`
+
+ const colours = new Set<string>();
+ const count = (v: string) => {
+ if (!KEPT_PAINT.test(v.trim())) colours.add(paintKey(v));
+ return v;
+ };
+ mapTagFills(opening, count);
+ mapChildFills(body, count);
+ if (colours.size <= 1) {
+ const themed = (v: string) => (KEPT_PAINT.test(v.trim()) ? v : "currentColor");
+ opening = mapTagFills(opening, themed);
+ body = mapChildFills(body, themed);
+ }
+
+ if (!/\sfill\s*=/i.test(opening)) {
+ opening = opening.replace(/^<svg/i, '<svg fill="currentColor"');
+ }
+ if (!/\saria-hidden\s*=/i.test(opening)) {
+ opening = opening.replace(/^<svg/i, '<svg aria-hidden="true"');
+ }
+ const out = opening + body;
+ // The result is read and checked again: no transform above may assemble
+ // what the input check refused.
+ const again = scan(out);
+ if ("problem" in again) return refuse(again.problem);
+ return { svg: out, problem: null };
+}
diff --git a/common/lib/socialSvg.vectors.ts b/common/lib/socialSvg.vectors.ts
@@ -0,0 +1,140 @@
+// TEST VECTORS for the social icon checker (socialSvg.ts) and its render path —
+// test data only, imported by common/lib/normalizeSocialSvg.test.ts and the
+// homepage e2e (svg-vectors.spec.ts, social.spec.ts). Synthetic throughout.
+//
+// ADVERSARIAL: the review's battery (release 14, slice HP): every input the
+// checker must either refuse or render inertly — no script, no request to
+// another origin, and the page parsed around the icon unchanged.
+// LOADS_ELSEWHERE: the eight the first allowlist accepted that made Chromium
+// fetch from another origin (R2); all refused now.
+// REAL_SHAPES: the shapes an operator's pasted icons take; each must pass, or
+// that icon turns into a text label on every site.
+
+const V = `viewBox="0 0 8 8"`;
+const W = (inner: string, open = `<svg ${V}>`) => `${open}${inner}</svg>`;
+export const EVIL = "https://evil.example";
+
+export const ADVERSARIAL: Record<string, string> = {
+ // the five originals
+ slash_handler: `<svg/onload="window.__x=1" ${V}><path d="M0 0"/></svg>`,
+ deletion_join: `<svg ${V} o width="1"nload="window.__x=1"><path d="M0 0"/></svg>`,
+ image_slash: W(`<image href="x:"/onerror="window.__x=1"/>`),
+ charref_js: W(`<a href="javascript:window.__x=1"><rect width="8" height="8"/></a>`),
+ breakout_img: W(`<img src="x:"/onerror="window.__x=1">`),
+ // tokenizer
+ unterminated_tag: `<svg ${V}><path d="M0 0"</svg>`,
+ unterminated_quote: `<svg ${V}><path d="M0 0/></svg>`,
+ attr_no_value: `<svg ${V} focusable><path d="M0 0"/></svg>`,
+ handler_no_value: `<svg ${V} onload><path d="M0 0"/></svg>`,
+ dup_href_first_ok: W(`<defs><linearGradient id="g"/></defs><use href="#g" href="${EVIL}/x.svg#g"/>`),
+ dup_href_first_bad: W(`<defs><linearGradient id="g"/></defs><use href="${EVIL}/x.svg#g" href="#g"/>`),
+ dup_fill: W(`<rect fill="url(#a)" fill="url(${EVIL}/p.svg#a)"/>`),
+ upper_xlink: W(`<use XLINK:HREF="javascript:window.__x=1"/>`),
+ upper_xlink_frag: W(`<defs><g id="a"/></defs><use XLINK:HREF="#a"/>`),
+ xml_base: `<svg ${V} xml:base="${EVIL}/"><use href="#a"/></svg>`,
+ xmlns_redef: `<svg ${V} xmlns:xlink="${EVIL}/ns" xmlns:foo="http://www.w3.org/1999/xlink"><defs><g id="a"/></defs><use foo:href="${EVIL}/x"/></svg>`,
+ xmlns_js: `<svg ${V} xmlns:xlink="javascript:window.__x=1"><path d="M0 0"/></svg>`,
+ nul_in_name: `<svg ${V} on\u0000load="window.__x=1"><path d="M0 0"/></svg>`,
+ nul_in_value: `<svg ${V} data-x="a\u0000b"><path d="M0 0"/></svg>`,
+ vt_separator: `<svg ${V}\u000bonload="window.__x=1"><path d="M0 0"/></svg>`,
+ cr_separator: `<svg ${V}\ronload="window.__x=1"><path d="M0 0"/></svg>`,
+ no_ws_between_attrs: `<svg ${V} data-a="1"onload="window.__x=1"><path d="M0 0"/></svg>`,
+ backtick_value: `<svg ${V} data-x=\`a\`><path d="M0 0"/></svg>`,
+ unquoted_value: `<svg ${V} data-x=a><path d="M0 0"/></svg>`,
+ gt_in_dq_value: `<svg ${V} data-x="><img src=x onerror=window.__x=1>"><path d="M0 0"/></svg>`,
+ gt_in_sq_value: `<svg ${V} data-x='"><img src=x onerror=window.__x=1>'><path d="M0 0"/></svg>`,
+ sq_value_with_fill: `<svg ${V} data-x=' fill="#fff" style="fill:#000"'><path fill="#f00" d="M0 0"/></svg>`,
+ // entities
+ ent_leading_zeros: W(`<rect data-x="javascript:window.__x=1"/>`),
+ ent_hex_nosemi: W(`<rect data-x="ڪvascript:x"/>`),
+ ent_colon_tab_nl: W(`<rect data-x="java	scr
ipt:x"/>`),
+ ent_colon_nosemi: W(`<rect data-x="javascript&colonx"/>`),
+ href_ent_hash: W(`<defs><g id="a"/></defs><use href="#a"/>`),
+ href_ws_frag: W(`<defs><g id="a"/></defs><use href=" #a "/>`),
+ // css url forms
+ style_url_escape: W(`<rect style="fill:\\75 rl(${EVIL}/p.svg#a)"/>`),
+ style_comment_url: W(`<rect style="fill:url/**/(${EVIL}/p.svg#a)"/>`),
+ pres_url_escape_fill: W(`<rect width="8" height="8" fill="\\75 rl(${EVIL}/fill.svg#a)"/>`),
+ pres_url_escape_filter: W(`<rect width="8" height="8" filter="\\75 rl(${EVIL}/filter.svg#a)"/>`),
+ pres_url_escape_mask: W(`<rect width="8" height="8" mask="\\75 rl(${EVIL}/mask.svg#a)"/>`),
+ pres_url_escape_clip: W(`<rect width="8" height="8" clip-path="\\75 rl(${EVIL}/clip.svg#a)"/>`),
+ pres_mask_imageset: W(`<rect width="8" height="8" mask="image-set('${EVIL}/mask-is.png' 1x)"/>`),
+ style_bg_imageset_root: `<svg ${V} style="background-image:image-set('${EVIL}/bg-is.png' 1x)"><path d="M0 0"/></svg>`,
+ style_mask_imageset: W(`<rect width="8" height="8" style="mask-image:image-set('${EVIL}/mimg-is.png' 1x)"/>`),
+ style_cursor_imageset: `<svg ${V} style="cursor:image-set('${EVIL}/cur-is.png' 1x),auto"><path d="M0 0"/></svg>`,
+ style_webkit_imageset: `<svg ${V} style="background-image:-webkit-image-set('${EVIL}/wk-is.png' 1x)"><path d="M0 0"/></svg>`,
+ anim_fill_escape: W(`<rect width="8" height="8"><set attributeName="fill" to="\\75 rl(${EVIL}/anim.svg#a)"/></rect>`),
+ anim_mask_imageset: W(`<rect width="8" height="8"><set attributeName="mask" to="image-set('${EVIL}/anim-is.png' 1x)"/></rect>`),
+ anim_style: W(`<rect width="8" height="8"><set attributeName="style" to="background:red"/></rect>`),
+ // references
+ use_external: W(`<use href="${EVIL}/x.svg#a"/>`),
+ use_data: W(`<use href="data:image/svg+xml,%3Csvg%3E"/>`),
+ feimage: W(`<filter id="f"><feImage href="${EVIL}/i.png"/></filter>`),
+ anchor: W(`<a href="#a"><rect/></a>`),
+ anim_href: W(`<set attributeName="href" to="javascript:x"/>`),
+ anim_href_ws: W(`<set attributeName=" HREF " to="#a"/>`),
+ anim_xlink_prefix: `<svg ${V} xmlns:x="http://www.w3.org/1999/xlink"><defs><g id="a"/></defs><use href="#a"><set attributeName="x:href" to="${EVIL}/x.svg#a" begin="0s"/></use></svg>`,
+ anim_onclick: W(`<set attributeName="onclick" to="window.__x=1"/>`),
+ anim_xlink_onclick: W(`<rect width="8" height="8"><set attributeName="xlink:onclick" to="window.__x=1"/></rect>`),
+ // elements
+ style_el: W(`<style>*{}</style>`),
+ nested_svg: W(`<svg viewBox="0 0 1 1"><path d="M0 0"/></svg>`),
+ title_markup_text: W(`<title><img src=x onerror=window.__x=1></title>`),
+ title_child_el: W(`<title><path d="M0 0"/></title>`),
+ desc_child_el: W(`<desc><g></g></desc>`),
+ title_title: W(`<title><title>x</title></title>`),
+ title_svg_title: W(`<title><svg viewBox="0 0 1 1"><title>t</title></svg></title>`),
+ math: W(`<math><mi>x</mi></math>`),
+ table: W(`<table><tr><td>x</td></tr></table>`),
+ after_root_html: `<svg ${V}></svg><img src=x onerror=window.__x=1>`,
+ after_root_ws: `<svg ${V}><path d="M0 0"/></svg>\n `,
+ doctype_subset: `<!DOCTYPE svg [<!ENTITY x "y">]><svg ${V}></svg>`,
+ doctype_plain: `<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "x"><svg ${V}><path d="M0 0"/></svg>`,
+ doctype_html: `<!DOCTYPE html><svg ${V}><path d="M0 0"/></svg>`,
+ xmldecl_gt: `<?xml version="1.0" encoding="a>b"?><svg ${V}><path d="M0 0"/></svg>`,
+ comment_join: `<svg ${V}><scr<!-- -->ipt>window.__x=1</script></svg>`,
+ comment_bang_close: `<svg ${V}><!-- a --!><img src=x onerror=window.__x=1> --><path d="M0 0"/></svg>`,
+ comment_abrupt: `<svg ${V}><!--><img src=x onerror=window.__x=1>--><path d="M0 0"/></svg>`,
+ cdata: W(`<![CDATA[<img src=x onerror=window.__x=1>]]>`),
+ pi: W(`<?x y?>`),
+ upper_close: `<svg ${V}><path d="M0 0"/></SVG>`,
+ close_ws: `<svg ${V}><path d="M0 0"/></svg >`,
+ close_attr: `<svg ${V}><path d="M0 0"/></svg foo="1">`,
+ svg_ns_script: W(`<svg:script>window.__x=1</svg:script>`),
+ // layout / redress (not script)
+ overlay_style: `<svg ${V} style="position:fixed;inset:0;width:100vw;height:100vh;z-index:2147483647"><path d="M0 0"/></svg>`,
+ class_redress: `<svg ${V} class="fixed inset-0 z-50"><path d="M0 0"/></svg>`,
+ id_clobber: W(`<g id="__next_f"/>`),
+ role_aria: `<svg ${V} aria-hidden="false" role="img" aria-label="Pay here"><path d="M0 0"/></svg>`,
+};
+
+export const LOADS_ELSEWHERE: readonly string[] = [
+ "pres_url_escape_mask",
+ "pres_url_escape_clip",
+ "pres_mask_imageset",
+ "style_bg_imageset_root",
+ "style_mask_imageset",
+ "style_cursor_imageset",
+ "style_webkit_imageset",
+ "anim_mask_imageset",
+];
+
+export const REAL_SHAPES: Record<string, string> = {
+ // A gradient body with a dark outline under it, the root as a drawing
+ // program writes it.
+ gradient_outlined:
+ `<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:svg="http://www.w3.org/2000/svg" x="0px" y="0px" viewBox="-3 -3 76.3 80.8" enable-background="new 0 0 198.7 74.8" xml:space="preserve">` +
+ `<defs><linearGradient id="grad_1" gradientUnits="userSpaceOnUse" x1="35" y1="0" x2="35" y2="75" gradientTransform="matrix(1,0,0,-1,0,75)">` +
+ `<stop offset="0" style="stop-color:#E0A030"/><stop offset="1" style="stop-color:#1C9A5B"/></linearGradient></defs>` +
+ `<path fill="url(#grad_1)" style="fill:url(#grad_1)" stroke="#161c21" stroke-width="6" stroke-linejoin="round" paint-order="stroke" d="M38 75C50 64 70 50 70 30 70 12 55 0 38 0S5 12 5 30c0 20 21 34 33 45z"/>` +
+ `<circle fill="#333333" cx="31" cy="10" r="2.1"/></svg>`,
+ // One path in the link's colour, `fill="none"` on the root.
+ single_path:
+ `<svg viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M2 2h20v20H2z" fill="currentColor"/></svg>`,
+ // A two-colour disc with a stroke on the disc and a decimal viewBox.
+ disc_stroked:
+ `<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 80.5 80.5" fill="none">` +
+ `<circle cx="42" cy="42" r="36" fill="#1a1a1a"/>` +
+ `<circle cx="38.5" cy="38.5" r="37" fill="#f4c542" stroke="#1a1a1a" stroke-width="1.557"/>` +
+ `<path fill="#1a1a1a" d="M30 22h20v8H38v6h10v8H38v14h-8z"/></svg>`,
+};
diff --git a/common/lib/sourceManifest.test.ts b/common/lib/sourceManifest.test.ts
@@ -0,0 +1,89 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { parseSourceManifest, TARBALL_HREF } from "./sourceManifest";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// The homepage believes a manifest only through parseSourceManifest, and the
+// /source/ page reads its numbers during `next build`: a malformed or
+// foreign one must be the empty state (null), never a crash (review L10).
+
+const good = () => ({
+ version: 1,
+ generatedAt: "2026-09-28T12:00:00.000Z",
+ branch: "main",
+ sourceCommit: "1".repeat(40),
+ mirrorHead: "2".repeat(40),
+ subject: "s",
+ files: 10,
+ bytes: 100,
+ mirror: { files: 9, bytes: 80, packs: 2 },
+ tree: { files: 5, dirs: 2, bytes: 20 },
+ tarball: { href: TARBALL_HREF, bytes: 7, sha256: "3".repeat(64) },
+ cloneUrl: "x",
+ treeHref: "/source/tree/",
+ audit: { objects: 3, commits: 1, gitleaks: "clean" },
+ tools: { git: "2", filterRepo: "x" },
+});
+
+test("a well-formed version-1 manifest parses", () => {
+ assert.deepEqual(parseSourceManifest(good()), good());
+});
+
+test("a wrong version, a bad id or sha, or any number the page reads that is not one is null", () => {
+ const broken: Array<[string, (m: ReturnType<typeof good>) => unknown]> = [
+ ["version 2", (m) => ({ ...m, version: 2 })],
+ ["short mirror head", (m) => ({ ...m, mirrorHead: "abc" })],
+ ["upper-case sha", (m) => ({ ...m, tarball: { ...m.tarball, sha256: "A".repeat(64) } })],
+ ["no tree", (m) => ({ ...m, tree: undefined })],
+ ["tree.files a string", (m) => ({ ...m, tree: { ...m.tree, files: "5" } })],
+ ["mirror.packs missing", (m) => ({ ...m, mirror: { files: 1, bytes: 1 } })],
+ ["mirror.bytes NaN", (m) => ({ ...m, mirror: { ...m.mirror, bytes: Number.NaN } })],
+ ["negative files", (m) => ({ ...m, files: -1 })],
+ ["tarball.bytes missing", (m) => ({ ...m, tarball: { href: TARBALL_HREF, sha256: "3".repeat(64) } })],
+ ["unparsable date", (m) => ({ ...m, generatedAt: "yesterday" })],
+ ["no audit", (m) => ({ ...m, audit: null })],
+ ];
+ for (const [what, make] of broken) assert.equal(parseSourceManifest(make(good())), null, what);
+ for (const junk of [null, 1, "x", [], {}]) assert.equal(parseSourceManifest(junk), null, JSON.stringify(junk));
+});
+
+// Release 15 slice SG: the history block is optional — absent is a manifest
+// without History links — and all-or-nothing when present.
+const history = () => ({
+ href: "/source/git/log.html",
+ commits: 3,
+ total: 3,
+ head: "2".repeat(40),
+ files: 9,
+ bytes: 1234,
+ sha256: "4".repeat(64),
+ tool: "stagit (sha256 0123456789ab)",
+});
+
+test("a manifest with a history block parses; without one it is the same manifest", () => {
+ const withHistory = { ...good(), history: history() };
+ assert.deepEqual(parseSourceManifest(withHistory), withHistory);
+ const capped = { ...good(), history: { ...history(), total: 12_345 } };
+ assert.deepEqual(parseSourceManifest(capped), capped, "the latest 3 of 12,345");
+ assert.equal(parseSourceManifest(good())?.history, undefined);
+});
+
+test("a history block that is present but malformed makes the manifest untrusted", () => {
+ const broken: Array<[string, (h: ReturnType<typeof history>) => unknown]> = [
+ ["null", () => null],
+ ["commits a string", (h) => ({ ...h, commits: "3" })],
+ ["no total", (h) => ({ ...h, total: undefined })],
+ ["fewer in all than with a page", (h) => ({ ...h, total: 2 })],
+ ["files negative", (h) => ({ ...h, files: -1 })],
+ ["bytes missing", (h) => ({ ...h, bytes: undefined })],
+ ["short head", (h) => ({ ...h, head: "abc" })],
+ ["sha256 not hex", (h) => ({ ...h, sha256: "z".repeat(64) })],
+ ["no href", (h) => ({ ...h, href: undefined })],
+ ["tool a number", (h) => ({ ...h, tool: 1 })],
+ ];
+ for (const [what, make] of broken) {
+ assert.equal(parseSourceManifest({ ...good(), history: make(history()) }), null, what);
+ }
+});
diff --git a/common/lib/sourceManifest.ts b/common/lib/sourceManifest.ts
@@ -0,0 +1,150 @@
+// The published source's contract: where `archilyzer source publish`
+// (common/publish/source.ts) puts things on the project site, and the shape of
+// the manifest it writes LAST, beside them (public/source/manifest.json). The
+// homepage's /source/ page (homepage/app/lib/source.ts) reads the same shape.
+//
+// A leaf: it imports only lib/project.ts (itself import-free), so the homepage
+// can pull it into a server component without dragging node: modules along.
+//
+// The mirror is a DUMB-HTTP git repository: static files only (HEAD,
+// info/refs, objects/info/packs, the packs), which `git clone` reads with no
+// server-side git at all. Its directory is `archilyzer.git`, never a path
+// segment named `.git` — wrangler's upload ignore list drops `**/.git`
+// silently, and Cloudflare's managed rules block `/.git/` paths.
+
+import { PROJECT_URL } from "./project";
+
+export const SOURCE_MANIFEST_VERSION = 1;
+
+/** The mirror's directory under /source/. */
+export const MIRROR_DIR = "archilyzer.git";
+
+/** What a reader types: `git clone <CLONE_URL>`. */
+export const CLONE_URL = `${PROJECT_URL}/source/${MIRROR_DIR}`;
+
+/** The raw tree's generated index. */
+export const TREE_HREF = "/source/tree/";
+
+/**
+ * The history pages' directory under /source/: stagit's rendering of the
+ * mirror (publish/sourceHistory.ts) — the log, a page per commit with its
+ * diff, the refs, the files index (into the raw tree) and two Atom feeds.
+ */
+export const HISTORY_DIR = "git";
+export const HISTORY_LOG_HREF = `/source/${HISTORY_DIR}/log.html`;
+export const HISTORY_REFS_HREF = `/source/${HISTORY_DIR}/refs.html`;
+export const HISTORY_ATOM_HREF = `/source/${HISTORY_DIR}/atom.xml`;
+
+/** The manifest's public path. */
+export const SOURCE_MANIFEST_HREF = "/source/manifest.json";
+
+/**
+ * The tarball's public path — the one the Downloads page has always linked
+ * (homepage/app/lib/snapshot.ts SNAPSHOT_HREF). Stable by design: the date and
+ * commit live in snapshot.json beside it.
+ */
+export const TARBALL_HREF = "/downloads/archilyzer-source.tar.gz";
+
+export type SourceManifest = {
+ version: typeof SOURCE_MANIFEST_VERSION;
+ generatedAt: string;
+ branch: "main";
+ // The PRIVATE repository's main, which the mirror reflects. Its history is
+ // never rewritten; the mirror is generated from a fresh clone of it.
+ sourceCommit: string;
+ // The mirror's main: a different id for the same history, because paths
+ // were scrubbed on the way out.
+ mirrorHead: string;
+ // The mirror head's subject line (audited like every other commit).
+ subject: string;
+ // Everything else the step published: the mirror, the tree and its
+ // indexes, the tarball and snapshot.json (not this manifest itself).
+ files: number;
+ bytes: number;
+ mirror: { files: number; bytes: number; packs: number };
+ // The tracked files of main, extracted; `files`/`bytes` do not count the
+ // generated index.html pages, `dirs` counts the directories (each has one).
+ tree: { files: number; dirs: number; bytes: number };
+ tarball: { href: string; bytes: number; sha256: string };
+ cloneUrl: string;
+ treeHref: string;
+ // What the gate read: how many git objects (commits among them), and
+ // whether gitleaks ran. NOT how many literals it searched for: `plans/`
+ // publishes which ones the plan put in the files, so the count would say
+ // whether the operator added private ones.
+ audit: {
+ objects: number;
+ commits: number;
+ gitleaks: "clean" | "skipped";
+ };
+ // The tools that made it (a filter-repo upgrade may change mirrorHead).
+ tools: { git: string; filterRepo: string };
+ // The history pages, when stagit rendered them (absent: no stagit on the
+ // publishing machine, or a render that failed — the page then shows no
+ // History links).
+ history?: SourceHistory;
+};
+
+export type SourceHistory = {
+ // The log page (HISTORY_LOG_HREF).
+ href: string;
+ // How many commits have a page (the newest ones: the log lists exactly
+ // these), of `total`, every commit of the mirror's main. Equal until main
+ // passes the cap (publish/sourceHistory.ts SOURCE_HISTORY_MAX_COMMITS).
+ // `head` is the one the log starts at (the manifest's mirrorHead).
+ commits: number;
+ total: number;
+ head: string;
+ // Every file under /source/git/ (the pages, the two feeds, style.css): how
+ // many, their bytes, and one sha256 over them — each file's path, size and
+ // sha256, in sorted path order — which the deploy check recomputes over out/.
+ files: number;
+ bytes: number;
+ sha256: string;
+ // The renderer: "stagit (sha256 <first 12 of its binary's>)".
+ tool: string;
+};
+
+const HEX40 = /^[0-9a-f]{40}$/;
+const HEX64 = /^[0-9a-f]{64}$/;
+
+/**
+ * The manifest, or null when `value` is not one this code can trust: version
+ * 1, 40-hex commit ids, a 64-hex tarball sha, and a finite number wherever a
+ * page reads one. A malformed or foreign manifest is the page's empty state,
+ * never a crash in `next build`. The homepage additionally requires the files
+ * it describes to be present (homepage/app/lib/source.ts).
+ */
+export function parseSourceManifest(value: unknown): SourceManifest | null {
+ if (!value || typeof value !== "object") return null;
+ const m = value as Partial<SourceManifest>;
+ const num = (x: unknown) => typeof x === "number" && Number.isFinite(x) && x >= 0;
+ const obj = (x: unknown): x is Record<string, unknown> => !!x && typeof x === "object";
+ if (m.version !== SOURCE_MANIFEST_VERSION) return null;
+ if (typeof m.mirrorHead !== "string" || !HEX40.test(m.mirrorHead)) return null;
+ if (typeof m.sourceCommit !== "string" || !HEX40.test(m.sourceCommit)) return null;
+ if (typeof m.subject !== "string" || typeof m.generatedAt !== "string") return null;
+ if (Number.isNaN(Date.parse(m.generatedAt))) return null;
+ if (!num(m.files) || !num(m.bytes)) return null;
+ if (!obj(m.tarball) || typeof m.tarball.sha256 !== "string" || !HEX64.test(m.tarball.sha256)) {
+ return null;
+ }
+ if (!num(m.tarball.bytes) || typeof m.tarball.href !== "string") return null;
+ if (!obj(m.mirror) || !num(m.mirror.files) || !num(m.mirror.bytes) || !num(m.mirror.packs)) {
+ return null;
+ }
+ if (!obj(m.tree) || !num(m.tree.files) || !num(m.tree.dirs) || !num(m.tree.bytes)) return null;
+ if (!obj(m.audit)) return null;
+ // Optional, and all-or-nothing: a history block that is present but
+ // malformed makes the whole manifest untrusted, like any other bad field.
+ if (m.history !== undefined) {
+ const h = m.history as Partial<SourceHistory> | null;
+ if (!obj(h)) return null;
+ if (!num(h.commits) || !num(h.total) || !num(h.files) || !num(h.bytes)) return null;
+ if ((h.total as number) < (h.commits as number)) return null;
+ if (typeof h.head !== "string" || !HEX40.test(h.head)) return null;
+ if (typeof h.sha256 !== "string" || !HEX64.test(h.sha256)) return null;
+ if (typeof h.href !== "string" || typeof h.tool !== "string") return null;
+ }
+ return m as SourceManifest;
+}
diff --git a/common/lib/stats.ts b/common/lib/stats.ts
@@ -3,10 +3,21 @@ import { pageFileName } from "./manifest";
import type { VideoState } from "./availability";
// Bumping this invalidates the LMDB `statsByPath` incremental cache and forces
-// a full re-extraction (e.g. when a new field is added below).
-export const STATS_SCHEMA_VERSION = 5;
+// a full re-extraction (e.g. when a new field is added below). It versions the
+// CACHE, not the published pages: STATS_MANIFEST_VERSION is theirs.
+// 6 — the cache key gained the index's transcript record, and a transcript
+// always has a `transcribedDate` (caption videos: the VTT's arrival).
+// The page shape did not change.
+export const STATS_SCHEMA_VERSION = 6;
export const STATS_MANIFEST_VERSION = 1;
+// A key in buildIndex's `meta` sub-DB (not this cache's): the ms timestamp the
+// last COMPLETED index build's scan began at. buildIndex writes it; buildStats
+// reads it to tell a video downloaded since that scan from one the scan saw and
+// did not index. Here rather than in buildIndex.ts so buildStats need not load
+// the index builder. Additive: the index schema did not move.
+export const INDEX_SCANNED_AT_KEY = "scannedAt";
+
// Visibility of a video on its source platform. One type with the viewer's, so
// the status chart and the search filter can never drift apart; see VideoState
// in lib/availability for the taxonomy (available + the five missing leaves).
@@ -38,6 +49,10 @@ export type VideoStat = {
// "content added over time" progress charts; the time X-axis can bin on any of
// these date fields.
downloadedDate: string | null;
+ // Non-null whenever `hasTranscript` is, since schema 6 (a caption video takes
+ // its captions' arrival). A page written by an older build can still carry a
+ // transcript with a null date: readers count it and only leave it off a time
+ // axis (homepageSummary does exactly that).
transcribedDate: string | null;
timestamp: number | null; // unix seconds
duration: number; // seconds
diff --git a/common/lib/storageHealth.test.ts b/common/lib/storageHealth.test.ts
@@ -0,0 +1,683 @@
+import { beforeEach, test } from "node:test";
+import assert from "node:assert/strict";
+import {
+ DriveNotAnsweringError,
+ allLocationHealth,
+ applyHealthTimings,
+ driveCallsInFlight,
+ healthTimings,
+ isDriveNotAnswering,
+ noteLocationDetector,
+ onDrive,
+ onPassIntervalChange,
+ registerLocationHealth,
+ setCounterReader,
+ setDriveCallBudget,
+ locationHealth,
+ notAnsweringText,
+ pruneLocationHealth,
+ recordLocationHealth,
+ resetStorageHealth,
+ sinceText,
+ stalledLocation,
+ stalledLocationForPath,
+} from "./storageHealth";
+import { HEALTH_TIMING_DEFAULTS } from "./storageHealthTimings";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common exec tsx --test lib/storageHealth.test.ts
+//
+// The rules the health state keeps (storageHealth.ts's header): one miss stalls
+// at once, two clean answers in a row clear it, a miss in between starts the
+// count again, and a new root starts the location over.
+
+beforeEach(() => {
+ resetStorageHealth();
+ applyHealthTimings();
+ setDriveCallBudget();
+ setCounterReader(undefined);
+});
+
+const USB = { id: "usb", label: "USB drive", root: "/mnt/usb/media" };
+
+test("one missed probe marks the location stalled at once", () => {
+ assert.equal(recordLocationHealth(USB, "ok", { now: 1_000 })?.to, "ok");
+ const t = recordLocationHealth(USB, "stalled", { now: 2_000, cause: "no answer" });
+ assert.deepEqual(t, { id: "usb", from: "ok", to: "stalled" });
+ const h = locationHealth("usb");
+ assert.equal(h?.state, "stalled");
+ assert.equal(h?.since, 2_000);
+ assert.equal(h?.cause, "no answer");
+});
+
+test("a location first seen stalled is stalled", () => {
+ const t = recordLocationHealth(USB, "stalled", { now: 5 });
+ assert.deepEqual(t, { id: "usb", from: null, to: "stalled" });
+ assert.equal(stalledLocation(USB)?.since, 5);
+});
+
+test("one clean probe after a stall does not clear it; two in a row do", () => {
+ assert.equal(HEALTH_TIMING_DEFAULTS.clearAfterCleanPasses, 2);
+ assert.equal(healthTimings().clearAfterCleanPasses, 2);
+ recordLocationHealth(USB, "ok", { now: 0 });
+ recordLocationHealth(USB, "stalled", { now: 10 });
+ assert.equal(recordLocationHealth(USB, "ok", { now: 20 }), null);
+ assert.equal(locationHealth("usb")?.state, "stalled");
+ // `since` stays the stall's start while it lasts.
+ assert.equal(locationHealth("usb")?.since, 10);
+ const t = recordLocationHealth(USB, "ok", { now: 30 });
+ assert.deepEqual(t, { id: "usb", from: "stalled", to: "ok" });
+ const h = locationHealth("usb");
+ assert.equal(h?.state, "ok");
+ assert.equal(h?.since, 30);
+ assert.equal(h?.cause, undefined);
+});
+
+test("a miss between two clean probes starts the count again", () => {
+ recordLocationHealth(USB, "stalled", { now: 0 });
+ recordLocationHealth(USB, "ok", { now: 1 });
+ assert.equal(recordLocationHealth(USB, "stalled", { now: 2 }), null);
+ // The stall's start is not moved by a second miss.
+ assert.equal(locationHealth("usb")?.since, 0);
+ recordLocationHealth(USB, "ok", { now: 3 });
+ assert.equal(locationHealth("usb")?.state, "stalled");
+ recordLocationHealth(USB, "ok", { now: 4 });
+ assert.equal(locationHealth("usb")?.state, "ok");
+});
+
+test("an absent answer is a clean one: an unplugged drive answers at once", () => {
+ recordLocationHealth(USB, "stalled", { now: 0 });
+ recordLocationHealth(USB, "absent", { now: 1 });
+ const t = recordLocationHealth(USB, "absent", { now: 2 });
+ assert.deepEqual(t, { id: "usb", from: "stalled", to: "absent" });
+ assert.equal(stalledLocation(USB), null);
+});
+
+test("a re-pointed root starts the location over", () => {
+ recordLocationHealth(USB, "stalled", { now: 0 });
+ const moved = { ...USB, root: "/mnt/elsewhere/media" };
+ // The stall belonged to the old root: the new root's lookup is not stalled,
+ // and neither is the old one's (the entry now describes the new root).
+ assert.equal(stalledLocation(moved), null);
+ const t = recordLocationHealth(moved, "ok", { now: 1 });
+ assert.deepEqual(t, { id: "usb", from: null, to: "ok" });
+ assert.equal(locationHealth("usb")?.root, "/mnt/elsewhere/media");
+ assert.equal(stalledLocation(USB), null);
+});
+
+test("the path gate matches like locationOfDataDir: longest root, strictly under", () => {
+ const parent = { id: "parent", label: "Parent", root: "/mnt/p" };
+ const child = { id: "child", label: "Child", root: "/mnt/p/archive" };
+ recordLocationHealth(parent, "stalled", { now: 0 });
+ recordLocationHealth(child, "ok", { now: 0 });
+ // A channel on the nested location answers for that location, not its parent.
+ assert.equal(stalledLocationForPath("/mnt/p/archive/chan/data"), null);
+ assert.equal(stalledLocationForPath("/mnt/p/chan/data")?.id, "parent");
+ // The root itself is not "under" it, and an unrelated path is on nothing.
+ assert.equal(stalledLocationForPath("/mnt/p"), null);
+ assert.equal(stalledLocationForPath("/elsewhere/chan/data"), null);
+ assert.equal(stalledLocationForPath(""), null);
+});
+
+test("pruning drops locations no longer configured", () => {
+ recordLocationHealth(USB, "stalled", { now: 0 });
+ recordLocationHealth({ id: "b", label: "B", root: "/b" }, "ok", { now: 0 });
+ pruneLocationHealth(["b"]);
+ assert.deepEqual(Object.keys(allLocationHealth()), ["b"]);
+ assert.equal(stalledLocationForPath("/mnt/usb/media/x/data"), null);
+});
+
+test("the state is one map per process, on globalThis", () => {
+ recordLocationHealth(USB, "stalled", { now: 0 });
+ // A second module copy reads the same object (the house pattern).
+ assert.equal(
+ globalThis.__yttStorageHealth__?.byId.get("usb")?.state,
+ "stalled",
+ );
+});
+
+test("the words: since a time today, or a date and time", () => {
+ const now = new Date(2026, 8, 29, 12, 0).getTime();
+ const today = new Date(2026, 8, 29, 11, 35).getTime();
+ const yesterday = new Date(2026, 8, 28, 23, 5).getTime();
+ assert.equal(sinceText(today, now), "since 11:35");
+ assert.equal(sinceText(yesterday, now), "since 2026-09-28 23:05");
+ assert.equal(notAnsweringText({ since: today }, now), "not answering since 11:35");
+});
+
+// ── the watchdog (`onDrive`) ────────────────────────────────────────────────
+// A call that never answers is a promise that never settles: exactly what a
+// read blocked on a stalled drive looks like from here. No drive is involved.
+
+const never = () => new Promise<never>(() => {});
+
+function deferred<T>() {
+ let resolve!: (v: T) => void;
+ let reject!: (e: Error) => void;
+ const promise = new Promise<T>((res, rej) => {
+ resolve = res;
+ reject = rej;
+ });
+ return { promise, resolve, reject };
+}
+
+test("a call that answers in time passes through, value or error, and frees its slot", async () => {
+ registerLocationHealth([USB]);
+ assert.equal(await onDrive("/mnt/usb/media/ch/data", async () => 42), 42);
+ await assert.rejects(
+ () => onDrive(USB, async () => {
+ throw new Error("ENOENT");
+ }),
+ /ENOENT/,
+ );
+ assert.equal(driveCallsInFlight("usb"), 0);
+ assert.equal(locationHealth("usb")?.state, "ok");
+});
+
+test("a call that never answers: stalled on the timer, the location marked at once, the call left to settle", async () => {
+ registerLocationHealth([USB], 0);
+ setDriveCallBudget(80);
+ const late = deferred<string>();
+ const started = Date.now();
+ await assert.rejects(
+ () => onDrive("/mnt/usb/media/ch/data", () => late.promise),
+ (err: unknown) =>
+ err instanceof DriveNotAnsweringError &&
+ isDriveNotAnswering(err) &&
+ err.health?.id === "usb" &&
+ /^drive not answering \(location "USB drive", since /.test(err.message),
+ );
+ assert.ok(Date.now() - started < 1_000);
+ const h = locationHealth("usb");
+ assert.equal(h?.state, "stalled");
+ assert.ok((h?.since ?? 0) >= started, "since is now, not the entry's first sighting");
+ assert.match(String(h?.cause), /a read in the editor did not answer within 0.08 s/);
+ // The slot is held until the call really returns.
+ assert.equal(driveCallsInFlight("usb"), 1);
+ late.resolve("finally");
+ await new Promise((r) => setImmediate(r));
+ assert.equal(driveCallsInFlight("usb"), 0);
+});
+
+test("a stalled location is refused without the call being made", async () => {
+ registerLocationHealth([USB]);
+ recordLocationHealth(USB, "stalled");
+ let made = 0;
+ await assert.rejects(
+ () => onDrive("/mnt/usb/media/ch/data", async () => ++made),
+ DriveNotAnsweringError,
+ );
+ await assert.rejects(() => onDrive(USB, async () => ++made), DriveNotAnsweringError);
+ assert.equal(made, 0);
+});
+
+test("at most four calls in flight on a location; the rest wait, and are refused without a call when it stalls", async () => {
+ assert.equal(HEALTH_TIMING_DEFAULTS.inFlightPerLocation, 4);
+ registerLocationHealth([USB]);
+ setDriveCallBudget(80);
+ let made = 0;
+ const calls = Array.from({ length: 7 }, () =>
+ onDrive(USB, () => {
+ made += 1;
+ return never();
+ }).then(
+ () => "answered",
+ (err: Error) => err.name,
+ ),
+ );
+ await new Promise((r) => setImmediate(r));
+ assert.equal(made, 4, "four in flight, three waiting");
+ const outcomes = await Promise.all(calls);
+ assert.deepEqual(outcomes, Array(7).fill("DriveNotAnsweringError"));
+ assert.equal(made, 4, "the three that waited were refused without a call");
+ assert.equal(locationHealth("usb")?.state, "stalled");
+});
+
+test("a waiting call runs when a slot frees, if the location is still answering", async () => {
+ registerLocationHealth([USB]);
+ const gates = Array.from({ length: 4 }, () => deferred<number>());
+ const first = gates.map((g) => onDrive(USB, () => g.promise));
+ let fifth = false;
+ const waiting = onDrive(USB, async () => {
+ fifth = true;
+ return 5;
+ });
+ await new Promise((r) => setImmediate(r));
+ assert.equal(fifth, false);
+ gates[0].resolve(1);
+ assert.equal(await waiting, 5);
+ for (const g of gates.slice(1)) g.resolve(0);
+ await Promise.all(first);
+ assert.equal(driveCallsInFlight("usb"), 0);
+});
+
+test("a path on no known location is raced and capped by its root, and names no location", async () => {
+ setDriveCallBudget(80);
+ let made = 0;
+ // Two channels under one hand-typed root share its four slots.
+ const calls = [
+ ...Array.from({ length: 3 }, () => "/hand/typed/chan-a/data"),
+ ...Array.from({ length: 3 }, () => "/hand/typed/chan-b/data"),
+ ].map((p) =>
+ onDrive(p, () => {
+ made += 1;
+ return never();
+ }).then(
+ () => "answered",
+ (err: DriveNotAnsweringError) => (err.health === null ? "refused" : "marked"),
+ ),
+ );
+ await new Promise((r) => setImmediate(r));
+ assert.equal(made, 4, "one root, four slots");
+ assert.deepEqual(await Promise.all(calls), Array(6).fill("refused"));
+ assert.equal(made, 4);
+ assert.deepEqual(allLocationHealth(), {});
+ // Every slot is held by a call given up on: the next is refused at once.
+ const started = Date.now();
+ await assert.rejects(() => onDrive("/hand/typed/chan-c/data", async () => ++made));
+ assert.ok(Date.now() - started < 50);
+ assert.equal(made, 4);
+});
+
+test("a probe of another root under a location's id has its own slots and does not rewrite that location", async () => {
+ registerLocationHealth([USB]);
+ setDriveCallBudget(50);
+ // Four candidate probes that never answer: they hold the CANDIDATE root's
+ // four slots, not the location's.
+ for (let i = 0; i < 4; i++) {
+ await assert.rejects(
+ () => onDrive({ ...USB, root: "/mnt/candidate" }, never),
+ DriveNotAnsweringError,
+ );
+ }
+ assert.equal(locationHealth("usb")?.root, "/mnt/usb/media");
+ assert.equal(locationHealth("usb")?.state, "ok");
+ assert.equal(driveCallsInFlight("usb"), 0);
+ // The real location's calls run as if nothing happened.
+ assert.deepEqual(
+ await Promise.all([1, 2, 3, 4, 5].map((n) => onDrive(USB, async () => n))),
+ [1, 2, 3, 4, 5],
+ );
+});
+
+test("M1: every slot held by a call given up on — a new call is refused within the budget, after the pass cleared the location", async () => {
+ registerLocationHealth([USB]);
+ setDriveCallBudget(80);
+ const hung = Array.from({ length: 4 }, () => onDrive(USB, never).catch(() => "gave up"));
+ assert.deepEqual(await Promise.all(hung), Array(4).fill("gave up"));
+ assert.equal(locationHealth("usb")?.state, "stalled");
+ // Two clean answers from the pass (in a reset loop's good moment).
+ recordLocationHealth(USB, "ok");
+ recordLocationHealth(USB, "ok");
+ assert.equal(locationHealth("usb")?.state, "ok");
+ let made = false;
+ const started = Date.now();
+ await assert.rejects(
+ () => onDrive(USB, async () => {
+ made = true;
+ }),
+ (err: unknown) => err instanceof DriveNotAnsweringError && err.health?.id === "usb",
+ );
+ assert.ok(Date.now() - started < 80, "refused at once, not after a wait");
+ assert.equal(made, false);
+ // And the location is marked again: none of the four has returned.
+ assert.equal(locationHealth("usb")?.state, "stalled");
+ assert.match(String(locationHealth("usb")?.cause), /4 reads on it have not answered/);
+});
+
+test("M1: every transition to stalled refuses the waiting calls at once, whoever decided it", async () => {
+ registerLocationHealth([USB]);
+ setDriveCallBudget(5_000);
+ const gates = Array.from({ length: 4 }, () => deferred<number>());
+ const inFlight = gates.map((g) => onDrive(USB, () => g.promise));
+ const waiting = onDrive(USB, async () => 5).then(
+ () => "ran",
+ (err: Error) => err.name,
+ );
+ await new Promise((r) => setImmediate(r));
+ const started = Date.now();
+ // The pass (not the watchdog) finds the drive stalled.
+ recordLocationHealth(USB, "stalled");
+ assert.equal(await waiting, "DriveNotAnsweringError");
+ assert.ok(Date.now() - started < 1_000);
+ for (const g of gates) g.resolve(0);
+ await Promise.all(inFlight);
+});
+
+test("L6: a call that times out while its disk is still completing requests is slow — refused, the location not marked", async () => {
+ registerLocationHealth([USB]);
+ noteLocationDetector("usb", "counters", "sdz1");
+ let completed = 100;
+ setCounterReader((device) => {
+ assert.equal(device, "sdz1");
+ completed += 7;
+ return { completed, inFlight: 2 };
+ });
+ setDriveCallBudget(60);
+ // Four slow calls and one waiting behind them.
+ const slow = Array.from({ length: 4 }, () =>
+ onDrive(USB, never).then(
+ () => "answered",
+ (err: Error) => err.message,
+ ),
+ );
+ const waiter = onDrive(USB, async () => 1).then(
+ () => "ran",
+ (err: Error) => err.message,
+ );
+ const outcomes = await Promise.all(slow);
+ for (const o of outcomes) assert.match(o, /^drive slow/);
+ assert.equal(locationHealth("usb")?.state, "ok", "slow is not stalled");
+ // Nothing on the drive returns: the waiter's wait runs out — refused, still
+ // without marking.
+ assert.match(await waiter, /nothing on it answered for 0.06 s while a read waited/);
+ assert.equal(locationHealth("usb")?.state, "ok");
+ // With the counters standing still, the same timeout marks it.
+ setCounterReader(() => ({ completed: 500, inFlight: 2 }));
+ resetStorageHealth();
+ registerLocationHealth([USB]);
+ noteLocationDetector("usb", "counters", "sdz1");
+ setCounterReader(() => ({ completed: 500, inFlight: 2 }));
+ await assert.rejects(() => onDrive(USB, never), DriveNotAnsweringError);
+ assert.equal(locationHealth("usb")?.state, "stalled");
+});
+
+test("the default budget is 3 s", async () => {
+ assert.equal(HEALTH_TIMING_DEFAULTS.budgetMs, 3_000);
+ assert.equal(healthTimings().budgetMs, 3_000);
+ registerLocationHealth([USB]);
+ const started = Date.now();
+ await assert.rejects(() => onDrive(USB, never), DriveNotAnsweringError);
+ const took = Date.now() - started;
+ assert.ok(took >= 3_000 && took < 4_500, `answered after ${took} ms`);
+});
+
+test("registering locations creates entries without an answer, and a moved root starts over", () => {
+ registerLocationHealth([USB], 5);
+ assert.equal(locationHealth("usb")?.state, "ok");
+ recordLocationHealth(USB, "stalled", { now: 6 });
+ registerLocationHealth([USB], 7);
+ assert.equal(locationHealth("usb")?.state, "stalled", "registering is not an answer");
+ registerLocationHealth([{ ...USB, root: "/mnt/new" }], 8);
+ assert.equal(locationHealth("usb")?.state, "ok");
+ assert.equal(locationHealth("usb")?.root, "/mnt/new");
+});
+
+// ── M4: the wait's deadline follows progress ───────────────────────────────
+
+const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
+
+test("M4: a healthy 64-wide walk of units at half the budget — no refusals, nothing marked", async () => {
+ registerLocationHealth([USB]);
+ setDriveCallBudget(100);
+ let answered = 0;
+ const started = Date.now();
+ const outcomes = await Promise.all(
+ Array.from({ length: 64 }, () =>
+ onDrive("/mnt/usb/media/ch/data", async () => {
+ await sleep(50);
+ answered += 1;
+ return "ok";
+ }).catch((err: Error) => err.message),
+ ),
+ );
+ // Sixteen rounds of 50 ms: the last call waited about 750 ms, far past the
+ // 100 ms budget, and was not refused — the drive kept answering.
+ assert.ok(Date.now() - started >= 700, `took ${Date.now() - started} ms`);
+ assert.deepEqual(outcomes, Array(64).fill("ok"));
+ assert.equal(answered, 64);
+ assert.equal(locationHealth("usb")?.state, "ok");
+ assert.equal(driveCallsInFlight("usb"), 0);
+});
+
+test("M4: the same deep queue on a hand-typed root — no refusals either", async () => {
+ setDriveCallBudget(100);
+ const outcomes = await Promise.all(
+ Array.from({ length: 32 }, () =>
+ onDrive("/hand/typed/ch/data", async () => {
+ await sleep(50);
+ return "ok";
+ }).catch((err: Error) => err.message),
+ ),
+ );
+ assert.deepEqual(outcomes, Array(32).fill("ok"));
+});
+
+test("M4: a queue behind four hung calls is still refused within the budget", async () => {
+ setDriveCallBudget(100);
+ // A hand-typed root: nothing marks it, so the refusal is the timeouts'.
+ const hung = Array.from({ length: 4 }, () =>
+ onDrive("/hand/typed/ch/data", never).catch(() => "gave up"),
+ );
+ const started = Date.now();
+ const waiters = Array.from({ length: 6 }, () =>
+ onDrive("/hand/typed/ch/data", async () => "ran").catch((err: Error) => err.name),
+ );
+ assert.deepEqual(await Promise.all(waiters), Array(6).fill("DriveNotAnsweringError"));
+ assert.ok(Date.now() - started < 400, `refused after ${Date.now() - started} ms`);
+ await Promise.all(hung);
+ // And on a configured location, the timeouts mark it and refuse the queue.
+ registerLocationHealth([USB]);
+ const hungHere = Array.from({ length: 4 }, () => onDrive(USB, never).catch(() => "gave up"));
+ const t0 = Date.now();
+ const waitersHere = Array.from({ length: 6 }, () =>
+ onDrive(USB, async () => "ran").catch((err: Error) => err.name),
+ );
+ assert.deepEqual(await Promise.all(waitersHere), Array(6).fill("DriveNotAnsweringError"));
+ assert.ok(Date.now() - t0 < 400);
+ assert.equal(locationHealth("usb")?.state, "stalled");
+ await Promise.all(hungHere);
+});
+
+test("M4: a call that returns late frees its slot for the calls waiting behind it", async () => {
+ registerLocationHealth([USB]);
+ noteLocationDetector("usb", "counters", "sdz1");
+ // The counters keep completing, so the late calls are slow, not stalled.
+ let completed = 0;
+ setCounterReader(() => ({ completed: (completed += 5), inFlight: 3 }));
+ setDriveCallBudget(80);
+ // Four calls that take 150 ms: refused as slow at 80 ms, returning at 150.
+ const slow = Array.from({ length: 4 }, () =>
+ onDrive(USB, () => sleep(150)).then(
+ () => "answered",
+ (err: Error) => err.message,
+ ),
+ );
+ // Four more queue at 70 ms, deadline 70 + 80 + 20: the returns at 150 ms
+ // come first and hand them the slots.
+ await sleep(70);
+ const behind = Array.from({ length: 4 }, () =>
+ onDrive(USB, async () => "ran").catch((err: Error) => err.message),
+ );
+ for (const o of await Promise.all(slow)) assert.match(o, /^drive slow/);
+ assert.deepEqual(await Promise.all(behind), Array(4).fill("ran"));
+ assert.equal(locationHealth("usb")?.state, "ok");
+ // The first return can feed all four waiters; let the other three slow
+ // calls return too, so their releases do not land in the next test's state.
+ await sleep(120);
+ assert.equal(driveCallsInFlight("usb"), 0);
+});
+
+// ── the overdue refusal: slow or stalled, by the counters ──────────────────
+
+test("every slot held by a slow unit on a disk still completing: the next call is refused, nothing marked", async () => {
+ registerLocationHealth([USB]);
+ noteLocationDetector("usb", "counters", "sdz1");
+ let completed = 0;
+ setCounterReader(() => ({ completed: (completed += 3), inFlight: 4 }));
+ setDriveCallBudget(60);
+ // Four units slower than the budget (they return long after): each is
+ // refused as slow at its timeout, and holds its slot, overdue.
+ const slow = Array.from({ length: 4 }, () =>
+ onDrive(USB, never).catch((err: Error) => err.message),
+ );
+ for (const o of await Promise.all(slow)) assert.match(o, /^drive slow/);
+ let made = false;
+ const started = Date.now();
+ await assert.rejects(
+ () => onDrive(USB, async () => {
+ made = true;
+ }),
+ (err: unknown) =>
+ err instanceof DriveNotAnsweringError &&
+ err.health === null &&
+ /^drive slow \(4 reads on it are past 0.06 s/.test(err.message),
+ );
+ assert.ok(Date.now() - started < 60, "refused at once");
+ assert.equal(made, false);
+ assert.equal(locationHealth("usb")?.state, "ok", "slow, not stalled");
+});
+
+test("every slot held by an overdue call on a disk completing nothing: the next call is refused and marks it", async () => {
+ registerLocationHealth([USB]);
+ noteLocationDetector("usb", "counters", "sdz1");
+ setCounterReader(() => ({ completed: 500, inFlight: 4 }));
+ setDriveCallBudget(60);
+ const hung = Array.from({ length: 4 }, () => onDrive(USB, never).catch(() => "gave up"));
+ await Promise.all(hung);
+ assert.equal(locationHealth("usb")?.state, "stalled");
+ // The pass clears it; the four have still not returned.
+ recordLocationHealth(USB, "ok");
+ recordLocationHealth(USB, "ok");
+ await assert.rejects(
+ () => onDrive(USB, async () => 1),
+ (err: unknown) => err instanceof DriveNotAnsweringError && err.health?.id === "usb",
+ );
+ assert.equal(locationHealth("usb")?.state, "stalled");
+ assert.match(String(locationHealth("usb")?.cause), /4 reads on it have not answered/);
+});
+
+// ── the timings are settings (release 15 slice DT) ─────────────────────────
+// `applyHealthTimings` takes the stored `settings.storage.health` block, as the
+// health pass and /storage's save hand it over; `healthTimings()` is what every
+// number above is read through.
+
+test("DT: the applied budget feeds the watchdog — a unit slower than it is refused, and the same unit passes on the default", async () => {
+ registerLocationHealth([USB]);
+ // The settings' floor, 500 ms (a stored 200 clamps to it; storageHealthTimings.test.ts).
+ applyHealthTimings({ budgetMs: 200 });
+ assert.equal(healthTimings().budgetMs, 500);
+ const unit = () => sleep(700).then(() => "done");
+ await assert.rejects(() => onDrive(USB, unit), DriveNotAnsweringError);
+ assert.equal(locationHealth("usb")?.state, "stalled");
+ assert.match(String(locationHealth("usb")?.cause), /did not answer within 0.5 s/);
+ // Back on the defaults (3 s), with the location clear, the same unit answers.
+ await sleep(250);
+ applyHealthTimings();
+ resetStorageHealth();
+ registerLocationHealth([USB]);
+ assert.equal(await onDrive(USB, unit), "done");
+ assert.equal(locationHealth("usb")?.state, "ok");
+});
+
+test("DT: the applied cap — with inFlightPerLocation 2, the third call waits for a slot", async () => {
+ registerLocationHealth([USB]);
+ applyHealthTimings({ inFlightPerLocation: 2 });
+ const gates = Array.from({ length: 2 }, () => deferred<number>());
+ const first = gates.map((g) => onDrive(USB, () => g.promise));
+ let third = false;
+ const waiting = onDrive(USB, async () => {
+ third = true;
+ return 3;
+ });
+ await new Promise((r) => setImmediate(r));
+ assert.equal(driveCallsInFlight("usb"), 2);
+ assert.equal(third, false, "the third call is queued, not on the drive");
+ gates[0].resolve(1);
+ assert.equal(await waiting, 3);
+ gates[1].resolve(2);
+ await Promise.all(first);
+ assert.equal(driveCallsInFlight("usb"), 0);
+});
+
+test("DT: a raised cap admits the calls already waiting; a lowered one is reached as calls return", async () => {
+ registerLocationHealth([USB]);
+ applyHealthTimings({ inFlightPerLocation: 1 });
+ const gates = Array.from({ length: 3 }, () => deferred<number>());
+ let made = 0;
+ const calls = gates.map((g) =>
+ onDrive(USB, () => {
+ made += 1;
+ return g.promise;
+ }),
+ );
+ await new Promise((r) => setImmediate(r));
+ assert.equal(made, 1);
+ // Raised to 3 on a save: the two waiting take the new slots now.
+ applyHealthTimings({ inFlightPerLocation: 3 });
+ await new Promise((r) => setImmediate(r));
+ assert.equal(made, 3);
+ assert.equal(driveCallsInFlight("usb"), 3);
+ // Lowered to 1 with three in flight: a fourth call waits, and a return gives
+ // its slot back rather than handing it on while more than one is in flight.
+ applyHealthTimings({ inFlightPerLocation: 1 });
+ let fourth = false;
+ const late = onDrive(USB, async () => {
+ fourth = true;
+ return 4;
+ });
+ gates[0].resolve(0);
+ gates[1].resolve(0);
+ await new Promise((r) => setImmediate(r));
+ assert.equal(fourth, false, "two returns bring three down to one: no slot for the fourth yet");
+ assert.equal(driveCallsInFlight("usb"), 1);
+ gates[2].resolve(0);
+ assert.equal(await late, 4);
+ await Promise.all(calls);
+ assert.equal(driveCallsInFlight("usb"), 0);
+});
+
+test("DT: the applied clear count — three clean answers with clearAfterCleanPasses 3, one with 1", () => {
+ applyHealthTimings({ clearAfterCleanPasses: 3 });
+ recordLocationHealth(USB, "stalled", { now: 0 });
+ recordLocationHealth(USB, "ok", { now: 1 });
+ recordLocationHealth(USB, "ok", { now: 2 });
+ assert.equal(locationHealth("usb")?.state, "stalled", "two are not three");
+ assert.deepEqual(recordLocationHealth(USB, "ok", { now: 3 }), {
+ id: "usb",
+ from: "stalled",
+ to: "ok",
+ });
+ applyHealthTimings({ clearAfterCleanPasses: 1 });
+ recordLocationHealth(USB, "stalled", { now: 4 });
+ recordLocationHealth(USB, "ok", { now: 5 });
+ assert.equal(locationHealth("usb")?.state, "ok");
+});
+
+test("DT: a changed pass interval is told to the subscribers; the same one, or another timing, is not", () => {
+ const heard: number[] = [];
+ const stop = onPassIntervalChange((ms) => heard.push(ms));
+ try {
+ applyHealthTimings({ passIntervalMs: 30_000 });
+ applyHealthTimings({ passIntervalMs: 30_000, budgetMs: 5_000 });
+ applyHealthTimings();
+ assert.deepEqual(heard, [30_000, 15_000]);
+ } finally {
+ stop();
+ }
+ applyHealthTimings({ passIntervalMs: 60_000 });
+ assert.deepEqual(heard, [30_000, 15_000], "unsubscribed");
+});
+
+test("DT: the test seam's budget wins over the applied one, and the timings survive a reset", () => {
+ applyHealthTimings({ budgetMs: 8_000, inFlightPerLocation: 6 });
+ setDriveCallBudget(80);
+ assert.equal(healthTimings().budgetMs, 80);
+ setDriveCallBudget();
+ assert.equal(healthTimings().budgetMs, 8_000);
+ resetStorageHealth();
+ assert.equal(healthTimings().inFlightPerLocation, 6, "configuration, not health");
+});
+
+test("DT review L4: a timeout on no known location names the budget the call ran against", async () => {
+ setDriveCallBudget(80);
+ const refused = onDrive("/hand/typed/ch/data", never).then(
+ () => "answered",
+ (err: Error) => err.message,
+ );
+ // Changed once the call is out (it reads its budget after taking a slot):
+ // its words keep the budget it was given.
+ await new Promise((r) => setImmediate(r));
+ setDriveCallBudget(5_000);
+ assert.equal(await refused, "drive not answering (a read did not answer within 0.08 s)");
+});
diff --git a/common/lib/storageHealth.ts b/common/lib/storageHealth.ts
@@ -0,0 +1,845 @@
+import { locationOfDataDir, type StorageLocation } from "./storageLocations";
+import {
+ HEALTH_TIMING_DEFAULTS,
+ resolveHealthTimings,
+ secondsText,
+ type HealthTimings,
+ type StorageHealthSettings,
+} from "./storageHealthTimings";
+
+// IS A STORAGE LOCATION'S DRIVE ANSWERING RIGHT NOW — the in-memory answer every
+// page and poll asks before it touches the drive.
+//
+// A drive can be mounted and still not answer. An SMR disk in a USB enclosure
+// under a long write stalls, the enclosure resets, and every filesystem call
+// that has to reach the disk blocks for about 30 seconds. Node runs those calls
+// on libuv's thread pool (four threads by default), so four of them block the
+// whole editor: no page, no poll and no job log answers until the disk does.
+// "Not mounted" does not describe that (a `stat` there does not fail, it hangs),
+// and no in-process call can find it out without paying the hang itself.
+//
+// SO SOMETHING THAT CANNOT HANG ASKS, AND THIS MODULE REMEMBERS WHAT IT SAID.
+// Two detectors write here. The health pass (`controller/storageWatch.ts`,
+// every `passIntervalMs`) reads each location's block device counters in /sys,
+// which never touch the drive (`detectLocationHealth` in `storageVolumes.ts`; a
+// child `stat` of the root raced against `probeTimeoutMs` only where no device
+// can be named). And `onDrive` below races every in-process call the gate
+// covers against `budgetMs` and marks the location the moment one does not
+// answer. The numbers are `settings.storage.health`, read through
+// `healthTimings()` below (by default a pass every 15 s, a 3 s probe and a 3 s
+// budget). Everything that would
+// touch the drive in-process asks this state first and, on a stalled location,
+// answers without the call: `inspectChannelMedia` reports `stalled`, the
+// free-space column reads "—", the probe reads "Not answering", the recency
+// layer skips the tail read.
+//
+// THE RULES, which are what keep a flaky drive from flapping the UI:
+// - ONE `stalled` answer marks the location `stalled` at once. A drive that
+// did not answer will not answer the next page either, and every page that
+// asks costs a thread.
+// - `clearAfterCleanPasses` (TWO by default) consecutive clean answers clear
+// it. A clean answer is anything else: the counters moving or idle, or a
+// child `stat` answering in time whether the root was there (`ok`) or not
+// (`absent`: an unmounted drive answers ENOENT at once, and that is a
+// different problem, which `inspectChannelMedia` already reports). One
+// clean answer in the middle of a reset loop is not recovery.
+// - A ROOT CHANGE (a re-point) starts the location over: the old root's stall
+// says nothing about the new one.
+//
+// IN MEMORY ONLY. A stall is a fact about this minute, not about the corpus;
+// persisting it would outlive the reset loop that caused it. A restarted
+// process starts with no stall, and the first pass (at arm time, on an idle
+// boot too) registers the locations; the watchdog re-learns a stall the moment
+// a page reaches the drive.
+//
+// THE TIMINGS ARE SETTINGS (`settings.storage.health`, release 15 slice DT),
+// held here as the process last applied them: the health pass applies them
+// from the settings it reads on every pass, and /storage's save applies them at
+// once. A process with no pass (a CLI) runs on the defaults unless its entry
+// applies them (the index and stats bins do). `healthTimings()` is the one
+// accessor. This module still reads no file.
+//
+// ONE MAP PER PROCESS, NOT PER MODULE COPY. The watch that probes is armed from
+// `editor/instrumentation.ts`, and the pages that read are another bundle
+// layer; Next can load this module once for each. The map lives on
+// `globalThis`, the house pattern (`lib/jsonFile-server.ts`, `jobs/registry.ts`),
+// so the writer and every reader see one map.
+//
+// PURE of I/O, and deliberately without execa, so `lib/channelMedia.ts` can ask
+// it without pulling a subprocess module into everything that imports that.
+
+// One probe's answer, and a location's state.
+export type LocationHealthState =
+ // The root answered in time and is a directory.
+ | "ok"
+ // The root did not answer within the probe's budget.
+ | "stalled"
+ // The root answered in time, and is not a directory (not mounted, or gone).
+ | "absent";
+
+// What decided a location's state. `counters`: the block device's own request
+// counters (`/sys/class/block/<dev>/stat`), which never touch the drive;
+// `stat`: a child `stat` of the root, when no device could be named (a
+// container, no findmnt, no /sys entry). A stall marked by `onDrive`'s watchdog
+// keeps the detector the last pass used; its `cause` says what did not answer.
+export type HealthDetector = "counters" | "stat";
+
+export type LocationHealth = {
+ id: string;
+ label: string;
+ root: string;
+ state: LocationHealthState;
+ detector?: HealthDetector;
+ // When the current state began, ms since epoch.
+ since: number;
+ // When the last probe (or observation) was recorded.
+ checkedAt: number;
+ // Consecutive clean answers since the location was marked stalled. It clears
+ // at `clearAfterCleanPasses`.
+ cleanStreak: number;
+ // What did not answer, for a stalled location: the probe's own words.
+ cause?: string;
+ // The block device the counters detector reads for this location, when it
+ // could name one. The watchdog reads its counters too (see `onDrive`).
+ device?: string;
+};
+
+// THE DEFAULTS, and why they are what they are (lib/storageHealthTimings.ts
+// holds them, with their ranges):
+// - `probeTimeoutMs` 3 s: a `stat` of a directory on a healthy disk answers in
+// microseconds; three seconds is a thousand times that, and short enough
+// that a page asking during a stall has not waited long.
+// - `passIntervalMs` 15 s: short, because the gate is only as current as the
+// last answer — a stall that began just after a pass costs every page that
+// touches the drive until the next one.
+// - `clearAfterCleanPasses` 2: one clean answer in a reset loop is not
+// recovery.
+
+// The one wording of the state, for every surface that shows it.
+export const NOT_ANSWERING = "drive not answering";
+
+// A call waiting for a slot. `resolve` answers whether it took the slot (a
+// waiter whose own wait already timed out does not); `rearm` restarts its
+// deadline, which a call returning on its key does (see `acquireSlot`).
+type Waiter = {
+ resolve: () => boolean;
+ reject: (err: Error) => void;
+ rearm: () => void;
+};
+
+// A call the watchdog gave up on that has not returned yet: when it began,
+// and the device's counters then (null when the counters detector has named
+// no device for its location).
+type OverdueCall = { startedAt: number; before: BlockStatSample | null };
+
+type HealthState = {
+ byId: Map<string, LocationHealth>;
+ // `onDrive`'s bookkeeping, by slot key (see `resolveWhere`): calls in
+ // flight, the ones among them the watchdog has already given up on, and
+ // the calls waiting for a slot.
+ inFlight?: Map<string, number>;
+ overdue?: Map<string, OverdueCall[]>;
+ waiters?: Map<string, Waiter[]>;
+ // The timings as last applied (`applyHealthTimings`); absent = the defaults.
+ timings?: HealthTimings;
+ // Told when the pass interval changes, so the armed pass re-arms its timer
+ // (controller/storageWatch.ts). On globalThis like the rest: the save that
+ // changes it runs in a page's module copy, the pass in instrumentation's.
+ intervalListeners?: Set<(passIntervalMs: number) => void>;
+ // Test seam: the watchdog's budget, below the settings' 500 ms floor.
+ budgetMs?: number;
+ // Reads a block device's counters, synchronously and without touching the
+ // drive (set by `storageVolumes.ts`, which owns /sys). Absent: no check.
+ readCounters?: (device: string) => BlockStatSample | null;
+};
+
+declare global {
+ // eslint-disable-next-line no-var
+ var __yttStorageHealth__: HealthState | undefined;
+}
+
+type FilledState = HealthState &
+ Required<Pick<HealthState, "inFlight" | "overdue" | "waiters" | "intervalListeners">>;
+
+function healthState(): FilledState {
+ if (!globalThis.__yttStorageHealth__) {
+ globalThis.__yttStorageHealth__ = { byId: new Map() };
+ }
+ const s = globalThis.__yttStorageHealth__;
+ // Filled lazily: a dev server's hot reload keeps an object made by an older
+ // copy of this module.
+ s.inFlight ??= new Map();
+ s.overdue ??= new Map();
+ s.waiters ??= new Map();
+ s.intervalListeners ??= new Set();
+ return s as FilledState;
+}
+
+// THE ONE ACCESSOR of the drive-health timings: `settings.storage.health` as
+// this process last applied it, every absent key its default. Read at the
+// moment a number is needed (a call's timer, a slot, an answer's count, a
+// pass's probe), so an applied change takes effect on the next of each.
+export function healthTimings(): HealthTimings {
+ const s = healthState();
+ const t = s.timings ?? HEALTH_TIMING_DEFAULTS;
+ return s.budgetMs === undefined ? t : { ...t, budgetMs: s.budgetMs };
+}
+
+// Apply `settings.storage.health` (the stored block; absent = every default).
+// The health pass calls this with the settings it reads on every pass, and
+// /storage's save calls it at once. A changed pass interval is told to the
+// armed pass (`onPassIntervalChange`); a raised cap lets calls already waiting
+// for a slot take the new ones.
+export function applyHealthTimings(stored?: StorageHealthSettings): HealthTimings {
+ const s = healthState();
+ const before = healthTimings();
+ s.timings = resolveHealthTimings(stored);
+ const after = healthTimings();
+ if (after.inFlightPerLocation > before.inFlightPerLocation) {
+ for (const key of [...s.waiters.keys()]) admitWaiters(key);
+ }
+ if (after.passIntervalMs !== before.passIntervalMs) {
+ for (const fn of [...s.intervalListeners]) {
+ try {
+ fn(after.passIntervalMs);
+ } catch {
+ /* a listener that throws must not stop a save */
+ }
+ }
+ }
+ return after;
+}
+
+// Subscribe to pass-interval changes; returns the unsubscribe.
+export function onPassIntervalChange(fn: (passIntervalMs: number) => void): () => void {
+ const set = healthState().intervalListeners;
+ set.add(fn);
+ return () => {
+ set.delete(fn);
+ };
+}
+
+// Test seam, and the escape hatch for a process that wants to forget. Calls
+// still waiting for a slot are released to run. The applied timings are
+// configuration, not health, and are kept.
+export function resetStorageHealth(): void {
+ const s = healthState();
+ s.byId.clear();
+ s.inFlight.clear();
+ s.overdue.clear();
+ for (const q of s.waiters.values()) for (const w of q) w.resolve();
+ s.waiters.clear();
+}
+
+// `storageVolumes.ts` registers its /sys reader here (see `onDrive`, L6 in
+// the record: a call that times out while its disk is still completing
+// requests is slow, not stalled). A test passes its own, or undefined.
+export function setCounterReader(
+ read: ((device: string) => BlockStatSample | null) | undefined,
+): void {
+ healthState().readCounters = read;
+}
+
+export type HealthTransition = {
+ id: string;
+ from: LocationHealthState | null;
+ to: LocationHealthState;
+};
+
+// Record one answer about one location. Returns the transition when the state
+// changed, null when it did not.
+export function recordLocationHealth(
+ loc: Pick<StorageLocation, "id" | "label" | "root">,
+ answer: LocationHealthState,
+ opts: {
+ now?: number;
+ cause?: string;
+ detector?: HealthDetector;
+ // The counters' device; `null` forgets it (the stat detector answered).
+ device?: string | null;
+ } = {},
+): HealthTransition | null {
+ const t = recordAnswer(loc, answer, opts);
+ // EVERY TRANSITION TO STALLED refuses the calls waiting for a slot on the
+ // location, whoever decided it (the pass, the watchdog, a Refresh): they
+ // would otherwise wait for a slot held by a call the drive is not answering.
+ if (t?.to === "stalled") {
+ refuseWaiters(slotKeyOfLocation(loc.id), stalledLocation(loc));
+ }
+ return t;
+}
+
+function recordAnswer(
+ loc: Pick<StorageLocation, "id" | "label" | "root">,
+ answer: LocationHealthState,
+ opts: {
+ now?: number;
+ cause?: string;
+ detector?: HealthDetector;
+ device?: string | null;
+ },
+): HealthTransition | null {
+ const now = opts.now ?? Date.now();
+ const map = healthState().byId;
+ const prev = map.get(loc.id);
+ const label = loc.label || loc.id;
+ // First sighting, or the root moved under the same id: start over.
+ if (!prev || prev.root !== loc.root) {
+ map.set(loc.id, {
+ id: loc.id,
+ label,
+ root: loc.root,
+ state: answer,
+ ...(opts.detector ? { detector: opts.detector } : {}),
+ ...(opts.device ? { device: opts.device } : {}),
+ since: now,
+ checkedAt: now,
+ cleanStreak: 0,
+ ...(answer === "stalled" && opts.cause ? { cause: opts.cause } : {}),
+ });
+ return { id: loc.id, from: null, to: answer };
+ }
+ prev.label = label;
+ prev.checkedAt = now;
+ if (opts.detector) prev.detector = opts.detector;
+ if (opts.device === null) delete prev.device;
+ else if (opts.device) prev.device = opts.device;
+ if (answer === "stalled") {
+ prev.cleanStreak = 0;
+ if (prev.state === "stalled") return null;
+ const from = prev.state;
+ prev.state = "stalled";
+ prev.since = now;
+ if (opts.cause) prev.cause = opts.cause;
+ return { id: loc.id, from, to: "stalled" };
+ }
+ if (prev.state === "stalled") {
+ prev.cleanStreak += 1;
+ if (prev.cleanStreak < healthTimings().clearAfterCleanPasses) return null;
+ prev.state = answer;
+ prev.since = now;
+ prev.cleanStreak = 0;
+ delete prev.cause;
+ return { id: loc.id, from: "stalled", to: answer };
+ }
+ if (prev.state === answer) return null;
+ const from = prev.state;
+ prev.state = answer;
+ prev.since = now;
+ return { id: loc.id, from, to: answer };
+}
+
+// Make sure every configured location has an entry, without recording an
+// answer: a new location starts `ok`, and a known one whose root moved starts
+// over. The watchdog below can only mark a location it can find, and it finds
+// a channel's location among these entries.
+export function registerLocationHealth(
+ locations: ReadonlyArray<Pick<StorageLocation, "id" | "label" | "root">>,
+ now: number = Date.now(),
+): void {
+ const map = healthState().byId;
+ for (const loc of locations) {
+ const prev = map.get(loc.id);
+ if (prev && prev.root === loc.root) {
+ prev.label = loc.label || loc.id;
+ continue;
+ }
+ recordLocationHealth(loc, "ok", { now });
+ }
+}
+
+// Which detector decided a location's last answer, when it gave none (the
+// counters' first sample has nothing to compare with).
+export function noteLocationDetector(
+ id: string,
+ detector: HealthDetector,
+ device?: string | null,
+): void {
+ const h = healthState().byId.get(id);
+ if (!h) return;
+ h.detector = detector;
+ if (device === null) delete h.device;
+ else if (device) h.device = device;
+}
+
+// Drop every location that is no longer configured.
+export function pruneLocationHealth(liveIds: Iterable<string>): void {
+ const keep = new Set(liveIds);
+ const map = healthState().byId;
+ for (const id of [...map.keys()]) if (!keep.has(id)) map.delete(id);
+}
+
+export function locationHealth(id: string): LocationHealth | undefined {
+ const h = healthState().byId.get(id);
+ return h ? { ...h } : undefined;
+}
+
+// Every location's last answer, by id. A copy: the caller may keep it.
+export function allLocationHealth(): Record<string, LocationHealth> {
+ const out: Record<string, LocationHealth> = {};
+ for (const [id, h] of healthState().byId) out[id] = { ...h };
+ return out;
+}
+
+// THE GATE. The stalled location `p` is on, or null. `p` is a channel's
+// `dataDir` (or anything under a location's root); the match is the one
+// `locationOfDataDir` makes, longest root first, so a channel on a nested
+// location answers for that location and not its parent.
+export function stalledLocationForPath(p: string): LocationHealth | null {
+ const map = healthState().byId;
+ if (map.size === 0 || !p) return null;
+ const entries = [...map.values()];
+ const loc = locationOfDataDir(
+ p,
+ entries.map((h) => ({ id: h.id, label: h.label, root: h.root, autoRepoint: false })),
+ );
+ if (!loc) return null;
+ const h = map.get(loc.id);
+ return h && h.state === "stalled" ? { ...h } : null;
+}
+
+// The location with this id, when it is stalled AND still at this root. The
+// gate for a caller that holds a location rather than a channel path
+// (`probeLocation`, `volumeFreeBytes`).
+export function stalledLocation(
+ loc: Pick<StorageLocation, "id" | "root">,
+): LocationHealth | null {
+ const h = healthState().byId.get(loc.id);
+ return h && h.state === "stalled" && h.root === loc.root ? { ...h } : null;
+}
+
+// "since 11:35" in the server's clock, or "since 2026-09-28 11:35" when the
+// stall began on another day than `now`.
+export function sinceText(since: number, now: number = Date.now()): string {
+ const d = new Date(since);
+ const n = new Date(now);
+ const pad = (x: number) => String(x).padStart(2, "0");
+ const hm = `${pad(d.getHours())}:${pad(d.getMinutes())}`;
+ const sameDay =
+ d.getFullYear() === n.getFullYear() &&
+ d.getMonth() === n.getMonth() &&
+ d.getDate() === n.getDate();
+ return sameDay
+ ? `since ${hm}`
+ : `since ${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())} ${hm}`;
+}
+
+// "not answering since 11:35" — the one line /storage, the /channels volume bar
+// and the channel's Storage panel show for a stalled location.
+export function notAnsweringText(h: Pick<LocationHealth, "since">, now?: number): string {
+ return `not answering ${sinceText(h.since, now)}`;
+}
+
+// ---------------------------------------------------------------------------
+// The watchdog: every call the gate covers, raced against the budget
+// ---------------------------------------------------------------------------
+//
+// THE DETECTOR THAT CANNOT BE FOOLED BY A CACHE. The pass reads the block
+// device's counters (or, with no device, a child `stat`), and a stall that
+// starts between two passes is not seen by it. A page or a poll that actually
+// reaches the drive is: `onDrive` runs the call against a timer of
+// `settings.storage.health.budgetMs` (3 s by default; `healthTimings()`), and a
+// call that has not answered by then marks its location `stalled` at once
+// (since now) and throws `DriveNotAnsweringError`. The call itself is left to
+// settle on its own: its thread is held until the drive answers, which is the
+// stated limit.
+//
+// THE BUDGET COVERS A WHOLE UNIT OF WORK. A caller sends a unit through as one
+// call (a video directory's few reads, a page's reads of one video), so a slow
+// drive that is still answering can be marked by one long unit. Unless its
+// disk is visibly still completing requests: on a timeout, when the counters
+// detector has named the location's device, its counters are read (from /sys,
+// never the drive) and compared with a reading taken when the call began; if
+// requests completed meanwhile the drive is slow, not stalled, and only this
+// call is refused.
+//
+// AT MOST `inFlightPerLocation` (4 by default) CALLS PER SLOT KEY ARE IN
+// FLIGHT. A walk of
+// `data/` fans out 64 wide, and a stall mid-walk would otherwise put all 64 in
+// libuv's queue before the watchdog fired. The rest wait in a queue of our own:
+// - a transition to `stalled` (from any detector) refuses them at once;
+// - a wait is refused (without marking anything) only when no call on its key
+// has returned for the budget plus a small grace: every call that returns
+// restarts every waiter's deadline, so a deep queue on a drive that is busy
+// but answering waits as long as it takes;
+// - when every slot is held by a call the watchdog already gave up on, a new
+// call is refused at once, and the location marked stalled again unless
+// the disk has been completing requests since the oldest of those calls
+// began (slow, not stalled — the same test as a timeout's): none of those
+// calls has returned, whatever the last pass said.
+// A slot is released when its call really returns, not when the watchdog gave
+// up on it. So on one location at most `inFlightPerLocation` threads wait on
+// its drive for the calls that come through here — every page and poll path,
+// and the snapshot walk. A job's own reads that do not come through here are
+// not capped. A cap lowered while calls are in flight is reached as they
+// return (a returning call gives its slot back instead of handing it on); a
+// cap raised lets waiting calls take the new slots at once.
+//
+// THE SLOT KEY: a configured location's id; for a probe of another root under a
+// location's id, that root; for a path on no configured location (a root typed
+// by hand), the root the path is under (`<root>/<slug>/data` → `<root>`). Only a
+// configured location can be marked.
+//
+// Do not nest `onDrive` for one key: the inner call would wait for a slot the
+// outer one holds.
+
+// Test seam: shorten (or restore, with no argument) the watchdog's budget,
+// past the settings' 500 ms floor. It wins over the applied timings.
+export function setDriveCallBudget(ms?: number): void {
+ healthState().budgetMs = ms;
+}
+
+function driveCallBudget(): number {
+ return healthTimings().budgetMs;
+}
+
+export class DriveNotAnsweringError extends Error {
+ // The stalled location, when a known one was marked.
+ readonly health: LocationHealth | null;
+ constructor(health: LocationHealth | null, detail?: string) {
+ super(
+ detail ??
+ (health
+ ? `${NOT_ANSWERING} (location "${health.label}", ${sinceText(health.since)})`
+ : `${NOT_ANSWERING} (a read did not answer within ${secondsText(driveCallBudget())})`),
+ );
+ this.name = "DriveNotAnsweringError";
+ this.health = health;
+ }
+}
+
+export function isDriveNotAnswering(err: unknown): err is DriveNotAnsweringError {
+ return err instanceof Error && err.name === "DriveNotAnsweringError";
+}
+
+type Where = string | Pick<StorageLocation, "id" | "label" | "root">;
+
+type Resolved = {
+ key: string;
+ // The location to mark, when marking is right.
+ loc: Pick<StorageLocation, "id" | "label" | "root"> | null;
+};
+
+function slotKeyOfLocation(id: string): string {
+ return `loc:${id}`;
+}
+
+// The root a path on no configured location is under: `<root>/<slug>/data`
+// (relocatedDataDir's shape) gives `<root>`; anything else is its own key.
+function rootOfUnknownPath(p: string): string {
+ const clean = p.replace(/\/+$/, "");
+ const parts = clean.split("/");
+ return parts.length > 2 && parts[parts.length - 1] === "data"
+ ? parts.slice(0, -2).join("/") || "/"
+ : clean;
+}
+
+function resolveWhere(where: Where): Resolved {
+ const map = healthState().byId;
+ if (typeof where !== "string") {
+ const entry = map.get(where.id);
+ // A probe of a candidate root under a location's id is its own key, and
+ // marks nothing: racing it is right, rewriting that location is not.
+ if (entry && entry.root !== where.root) {
+ return { key: `root:${where.root}`, loc: null };
+ }
+ return { key: slotKeyOfLocation(where.id), loc: where };
+ }
+ const found =
+ map.size > 0 && where
+ ? locationOfDataDir(
+ where,
+ [...map.values()].map((h) => ({
+ id: h.id,
+ label: h.label,
+ root: h.root,
+ autoRepoint: false,
+ })),
+ )
+ : null;
+ if (found) return { key: slotKeyOfLocation(found.id), loc: found };
+ return { key: `root:${rootOfUnknownPath(where)}`, loc: null };
+}
+
+function refuseWaiters(key: string, health: LocationHealth | null): void {
+ const s = healthState();
+ const q = s.waiters.get(key) ?? [];
+ s.waiters.delete(key);
+ for (const w of q) w.reject(new DriveNotAnsweringError(health));
+}
+
+// Take a slot on `key`, or wait for one (raced against the budget), or be
+// refused at once when every slot is held by an overdue call.
+async function acquireSlot(r: Resolved): Promise<void> {
+ const s = healthState();
+ const cap = healthTimings().inFlightPerLocation;
+ const n = s.inFlight.get(r.key) ?? 0;
+ if (n < cap) {
+ s.inFlight.set(r.key, n + 1);
+ return;
+ }
+ const overdue = s.overdue.get(r.key) ?? [];
+ if (overdue.length >= cap) {
+ // EVERY SLOT IS HELD BY A CALL THE WATCHDOG GAVE UP ON: refused at once.
+ // Marked stalled only when the disk has not been completing requests
+ // since the oldest of them began — the watchdog's own slow-or-stalled test
+ // (see `onDrive`): four slow reads on a busy disk are not a stall.
+ const oldest = overdue.reduce((a, b) => (b.startedAt < a.startedAt ? b : a));
+ const now = readDeviceCounters(r.loc);
+ if (oldest.before && now && now.completed > oldest.before.completed) {
+ throw new DriveNotAnsweringError(
+ null,
+ `drive slow (${overdue.length} reads on it are past ` +
+ `${secondsText(driveCallBudget())}, while its disk is still completing others)`,
+ );
+ }
+ let health: LocationHealth | null = null;
+ if (r.loc) {
+ recordLocationHealth(r.loc, "stalled", {
+ cause: `${overdue.length} reads on it have not answered`,
+ });
+ health = stalledLocation(r.loc);
+ }
+ throw new DriveNotAnsweringError(health);
+ }
+ const budget = driveCallBudget();
+ // THE WAIT'S DEADLINE FOLLOWS PROGRESS, NOT THE QUEUE. A waiter is refused
+ // only when NO call on its key has returned for a full budget — "nothing on
+ // this drive answered for 3 s", the condition the watchdog exists for.
+ // Every call that returns on the key (in time or late) re-arms every waiter
+ // behind it, so a deep queue on a drive that is busy but answering (a
+ // 64-wide walk of units that each take a second) is never refused for its
+ // depth alone. Timed from when the call queued, it was: a waiter at depth d
+ // waits about d/4 units, whatever the drive is doing.
+ //
+ // PLUS A SMALL GRACE. A waiter's timer is created when it queues — before
+ // the race timers of calls that took their slots in the same tick — so with
+ // equal deadlines the waiters would give up a moment before the calls they
+ // wait behind, and be refused without the mark those calls are about to
+ // make. The grace lets them time out first.
+ const grace = Math.min(250, Math.round(budget / 4));
+ await new Promise<void>((resolve, reject) => {
+ let settled = false;
+ let timer: ReturnType<typeof setTimeout> | undefined;
+ const giveUp = () => {
+ if (settled) return;
+ settled = true;
+ const q = s.waiters.get(r.key);
+ if (q) {
+ const i = q.indexOf(waiter);
+ if (i >= 0) q.splice(i, 1);
+ }
+ reject(
+ new DriveNotAnsweringError(
+ r.loc ? stalledLocation(r.loc) : null,
+ `${NOT_ANSWERING} (nothing on it answered for ${secondsText(budget)} while a read waited)`,
+ ),
+ );
+ };
+ const arm = () => {
+ if (timer) clearTimeout(timer);
+ timer = setTimeout(giveUp, budget + grace);
+ };
+ const waiter: Waiter = {
+ resolve: () => {
+ if (settled) return false;
+ settled = true;
+ if (timer) clearTimeout(timer);
+ resolve();
+ return true;
+ },
+ reject: (err) => {
+ if (settled) return;
+ settled = true;
+ if (timer) clearTimeout(timer);
+ reject(err);
+ },
+ rearm: () => {
+ if (!settled) arm();
+ },
+ };
+ arm();
+ const q = s.waiters.get(r.key) ?? [];
+ q.push(waiter);
+ s.waiters.set(r.key, q);
+ });
+ // Resolved by a release that handed its slot over: the count is unchanged.
+}
+
+// A slot is released when its call returns (or, on a refusal after a wait,
+// without a call). A return is progress: every waiter on the key has its
+// deadline restarted, then the slot goes to the first waiter still waiting —
+// unless the cap was lowered below what is in flight, when it is given back.
+function releaseSlot(key: string): void {
+ const s = healthState();
+ const q = s.waiters.get(key);
+ if (q) for (const w of q) w.rearm();
+ const n = s.inFlight.get(key) ?? 1;
+ if (n <= healthTimings().inFlightPerLocation) {
+ while (q && q.length > 0) {
+ const next = q.shift() as Waiter;
+ if (next.resolve()) return;
+ }
+ }
+ s.inFlight.set(key, Math.max(0, n - 1));
+}
+
+// A raised cap: the calls already waiting on `key` take the new slots now,
+// rather than one at a time as calls return.
+function admitWaiters(key: string): void {
+ const s = healthState();
+ const q = s.waiters.get(key);
+ const cap = healthTimings().inFlightPerLocation;
+ while (q && q.length > 0 && (s.inFlight.get(key) ?? 0) < cap) {
+ const next = q.shift() as Waiter;
+ if (next.resolve()) s.inFlight.set(key, (s.inFlight.get(key) ?? 0) + 1);
+ }
+}
+
+// How many calls are in flight on a location through `onDrive` (for tests and
+// for a reader that wants to say so).
+export function driveCallsInFlight(id: string): number {
+ return healthState().inFlight.get(slotKeyOfLocation(id)) ?? 0;
+}
+
+function readDeviceCounters(loc: Resolved["loc"]): BlockStatSample | null {
+ if (!loc) return null;
+ const s = healthState();
+ const device = s.byId.get(loc.id)?.device;
+ if (!device || !s.readCounters) return null;
+ try {
+ return s.readCounters(device);
+ } catch {
+ return null;
+ }
+}
+
+// Run `call` against the drive `where` is on: refused at once when that
+// location is stalled, queued behind `inFlightPerLocation` calls already in
+// flight on its key, and raced against `budgetMs` (`healthTimings()`). Throws
+// DriveNotAnsweringError for a refusal or a timeout; any other error is the
+// call's own.
+export async function onDrive<T>(where: Where, call: () => Promise<T>): Promise<T> {
+ const r = resolveWhere(where);
+ const refused = r.loc ? stalledLocation(r.loc) : null;
+ if (refused) throw new DriveNotAnsweringError(refused);
+ await acquireSlot(r);
+ // Stalled while this call waited: refused, and the slot passed on.
+ const late = r.loc ? stalledLocation(r.loc) : null;
+ if (late) {
+ releaseSlot(r.key);
+ throw new DriveNotAnsweringError(late);
+ }
+ const s = healthState();
+ let released = false;
+ const release = () => {
+ if (released) return;
+ released = true;
+ releaseSlot(r.key);
+ };
+ const startedAt = Date.now();
+ const before = readDeviceCounters(r.loc);
+ let pending: Promise<T>;
+ try {
+ pending = call();
+ } catch (err) {
+ release();
+ throw err;
+ }
+ const TIMED_OUT = Symbol("timed out");
+ // The budget this call runs against, read as it starts.
+ const budget = driveCallBudget();
+ let timer: ReturnType<typeof setTimeout> | undefined;
+ // Not unref'd: it is cleared the moment the call answers, and while the call
+ // is outstanding the timer is what must fire.
+ const timeout = new Promise<typeof TIMED_OUT>((resolve) => {
+ timer = setTimeout(() => resolve(TIMED_OUT), budget);
+ });
+ let answer: T | typeof TIMED_OUT;
+ try {
+ answer = await Promise.race([pending, timeout]);
+ } catch (err) {
+ release();
+ throw err;
+ } finally {
+ if (timer) clearTimeout(timer);
+ }
+ if (answer !== TIMED_OUT) {
+ release();
+ return answer;
+ }
+ // The call is still waiting on the drive. Its slot stays held, and counted
+ // overdue, until it returns; nothing awaits it.
+ const entry: OverdueCall = { startedAt, before };
+ s.overdue.set(r.key, [...(s.overdue.get(r.key) ?? []), entry]);
+ const done = () => {
+ const left = (s.overdue.get(r.key) ?? []).filter((e) => e !== entry);
+ if (left.length > 0) s.overdue.set(r.key, left);
+ else s.overdue.delete(r.key);
+ release();
+ };
+ pending.then(done, done);
+ const after = readDeviceCounters(r.loc);
+ if (before && after && after.completed > before.completed) {
+ // SLOW, NOT STALLED: the disk completed requests while this call waited.
+ throw new DriveNotAnsweringError(
+ null,
+ `drive slow (a read did not answer within ${secondsText(budget)}, ` +
+ `while its disk was still completing others)`,
+ );
+ }
+ if (r.loc) {
+ recordLocationHealth(r.loc, "stalled", {
+ cause: `a read in the editor did not answer within ${secondsText(budget)}`,
+ });
+ throw new DriveNotAnsweringError(stalledLocation(r.loc));
+ }
+ refuseWaiters(r.key, null);
+ // The budget this call ran against, not the one in force now: a save during
+ // the call must not rewrite what it was given.
+ throw new DriveNotAnsweringError(
+ null,
+ `${NOT_ANSWERING} (a read did not answer within ${secondsText(budget)})`,
+ );
+}
+
+// ---------------------------------------------------------------------------
+// The block device's counters
+// ---------------------------------------------------------------------------
+//
+// `/sys/class/block/<dev>/stat` is the kernel's own count of a device's
+// requests (Documentation/block/stat.rst): field 1 reads completed, 5 writes
+// completed, 9 requests in flight now. Reading it never touches the drive. A
+// drive that is merely slow, even one grinding through a long write, keeps
+// completing requests; one in a reset loop has requests in flight and
+// completes none. So, between two samples a pass apart:
+//
+// stalled ⇔ in flight at both samples AND nothing completed between (reads,
+// writes, discards, flushes)
+// ok ⇔ anything else (nothing in flight at one of them, or completions
+// moved)
+//
+// and the health rules above turn one `stalled` into a stall and two `ok`s in
+// a row into its end.
+
+export type BlockStatSample = { completed: number; inFlight: number };
+
+// Completed: reads (field 1) + writes (5), and, on kernels that count them,
+// discards (12) and flushes (16) — in_flight counts those too, so a long flush
+// alone (an SMR drive emptying its media cache) must not read as nothing
+// completing.
+export function parseBlockStat(line: string): BlockStatSample | null {
+ const f = line.trim().split(/\s+/).map(Number);
+ if (f.length < 9 || f.slice(0, 9).some((n) => !Number.isFinite(n))) return null;
+ const opt = (i: number) => (Number.isFinite(f[i]) ? f[i] : 0);
+ return { completed: f[0] + f[4] + opt(11) + opt(15), inFlight: f[8] };
+}
+
+export function countersVerdict(
+ prev: BlockStatSample,
+ cur: BlockStatSample,
+): "ok" | "stalled" {
+ return prev.inFlight > 0 && cur.inFlight > 0 && cur.completed === prev.completed
+ ? "stalled"
+ : "ok";
+}
diff --git a/common/lib/storageHealthCounters.test.ts b/common/lib/storageHealthCounters.test.ts
@@ -0,0 +1,260 @@
+import { beforeEach, test } from "node:test";
+import assert from "node:assert/strict";
+import path from "node:path";
+import { tmpdir } from "node:os";
+import { chmod, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
+import {
+ blockDeviceName,
+ detectLocationHealth,
+ minCounterIntervalMs,
+ resetHealthDetector,
+} from "./storageVolumes";
+import { applyHealthTimings, countersVerdict, parseBlockStat } from "./storageHealth";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common exec tsx --test lib/storageHealthCounters.test.ts
+//
+// THE COUNTERS DETECTOR. No test stalls a real drive: findmnt is a fake that
+// names a device, and /sys/class/block is a temp directory whose `stat` files
+// the test writes — a stalled device is one whose in-flight count stays up
+// while its completions stand still.
+
+beforeEach(() => {
+ resetHealthDetector();
+ applyHealthTimings();
+});
+
+// A real line from this machine's /sys/class/block/<dev>/stat (17 fields).
+const LINE =
+ "368126412 135952867 36463191157 80435430 29670281 46781260 4001255365 167469727 0 44151394 253218297 877589 0 861936568 4100450 578669 1212688";
+
+test("the stat line: reads + writes (+ discards + flushes) completed, and requests in flight", () => {
+ // Fields 1, 5, 12 and 16 completed; field 9 in flight.
+ assert.deepEqual(parseBlockStat(LINE), {
+ completed: 368126412 + 29670281 + 877589 + 578669,
+ inFlight: 0,
+ });
+ // A long flush alone moves completions: not a stall.
+ const before = parseBlockStat("10 0 0 0 5 0 0 0 1 0 0 0 0 0 0 7 0");
+ const after = parseBlockStat("10 0 0 0 5 0 0 0 1 0 0 0 0 0 0 8 0");
+ assert.equal(countersVerdict(before!, after!), "ok");
+ // An 11-field line from an older kernel parses the same way.
+ assert.deepEqual(parseBlockStat("10 0 0 0 5 0 0 0 3 0 0\n"), { completed: 15, inFlight: 3 });
+ assert.equal(parseBlockStat(""), null);
+ assert.equal(parseBlockStat("1 2 3"), null);
+ assert.equal(parseBlockStat("a b c d e f g h i"), null);
+});
+
+test("the verdict over a pair of samples", () => {
+ const s = (completed: number, inFlight: number) => ({ completed, inFlight });
+ // In flight at both, nothing completed between: stalled.
+ assert.equal(countersVerdict(s(100, 2), s(100, 5)), "stalled");
+ // Completions moved: slow, perhaps, but answering.
+ assert.equal(countersVerdict(s(100, 2), s(101, 2)), "ok");
+ // Nothing in flight at either end: idle.
+ assert.equal(countersVerdict(s(100, 0), s(100, 0)), "ok");
+ assert.equal(countersVerdict(s(100, 0), s(100, 4)), "ok");
+ assert.equal(countersVerdict(s(100, 4), s(100, 0)), "ok");
+});
+
+test("a mount source's device name: a partition, a bind or subvolume suffix, nothing that is not /dev", async () => {
+ assert.equal(await blockDeviceName("/dev/no-such-disk1"), "no-such-disk1");
+ assert.equal(await blockDeviceName("/dev/no-such-disk2[/@home]"), "no-such-disk2");
+ assert.equal(await blockDeviceName("tmpfs"), null);
+ assert.equal(await blockDeviceName("server:/export"), null);
+ assert.equal(await blockDeviceName(""), null);
+});
+
+// ── the detector end to end, with a fake findmnt and a fake /sys ────────────
+
+const FAKE_FINDMNT = `#!/usr/bin/env node
+import { readFileSync } from "node:fs";
+import path from "node:path";
+const c = JSON.parse(readFileSync(path.join(import.meta.dirname, "control.json"), "utf8"));
+if (c.sleepMs) Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, c.sleepMs);
+if (c.exit) process.exit(c.exit);
+process.stdout.write(JSON.stringify({ filesystems: [{ source: c.source, uuid: c.uuid ?? null }] }) + "\\n");
+`;
+
+type H = {
+ root: string;
+ bins: { findmntBin: string };
+ sys: string;
+ control: (c: Record<string, unknown>) => Promise<void>;
+ counters: (device: string, completed: number, inFlight: number) => Promise<void>;
+};
+
+async function withHarness(fn: (h: H) => Promise<void>): Promise<void> {
+ const dir = await mkdtemp(path.join(tmpdir(), "ttb-counters-"));
+ try {
+ const bin = path.join(dir, "fake-findmnt.mjs");
+ await writeFile(bin, FAKE_FINDMNT);
+ await chmod(bin, 0o755);
+ const root = path.join(dir, "media");
+ await mkdir(root);
+ const sys = path.join(dir, "sys-block");
+ await fn({
+ root,
+ bins: { findmntBin: bin },
+ sys,
+ control: (c) => writeFile(path.join(dir, "control.json"), JSON.stringify(c)),
+ counters: async (device, completed, inFlight) => {
+ await mkdir(path.join(sys, device), { recursive: true });
+ // Reads `completed`, writes 0, in flight `inFlight`, the rest zero.
+ await writeFile(
+ path.join(sys, device, "stat"),
+ `${completed} 0 0 0 0 0 0 0 ${inFlight} 0 0 0 0 0 0 0 0\n`,
+ );
+ },
+ });
+ } finally {
+ await rm(dir, { recursive: true, force: true });
+ }
+}
+
+const loc = (root: string, uuid?: string) => ({
+ id: "usb",
+ root,
+ ...(uuid ? { volume: { uuid, mountpoint: root, relPath: "" } } : {}),
+});
+
+test("counters: first sample no verdict; stuck → stalled; two clean samples; each pass apart", async () => {
+ await withHarness(async (h) => {
+ await h.control({ source: "/dev/fakedisk1", uuid: "u-1" });
+ const at = (i: number) => ({ sysBlockDir: h.sys, now: i * 15_000 });
+ await h.counters("fakedisk1", 100, 2);
+ assert.deepEqual(await detectLocationHealth(loc(h.root, "u-1"), h.bins, at(1)), {
+ answer: null,
+ detector: "counters",
+ device: "fakedisk1",
+ });
+ // Still two in flight, nothing completed in 15 s.
+ await h.counters("fakedisk1", 100, 2);
+ const stuck = await detectLocationHealth(loc(h.root, "u-1"), h.bins, at(2));
+ assert.equal(stuck.answer, "stalled");
+ assert.equal(stuck.detector, "counters");
+ assert.match(String(stuck.cause), /^its disk \(fakedisk1\) had 2 request\(s\) in flight and completed none in 15 s$/);
+ // It drains: nothing in flight.
+ await h.counters("fakedisk1", 140, 0);
+ assert.equal((await detectLocationHealth(loc(h.root, "u-1"), h.bins, at(3))).answer, "ok");
+ // Busy and moving: still ok.
+ await h.counters("fakedisk1", 190, 3);
+ assert.equal((await detectLocationHealth(loc(h.root, "u-1"), h.bins, at(4))).answer, "ok");
+ });
+});
+
+test("counters: a second sample sooner than the minimum interval gives no verdict and keeps the first", async () => {
+ await withHarness(async (h) => {
+ await h.control({ source: "/dev/fakedisk1" });
+ await h.counters("fakedisk1", 100, 2);
+ await detectLocationHealth(loc(h.root), h.bins, { sysBlockDir: h.sys, now: 0 });
+ const soon = await detectLocationHealth(loc(h.root), h.bins, {
+ sysBlockDir: h.sys,
+ now: minCounterIntervalMs() - 1,
+ });
+ assert.equal(minCounterIntervalMs(), 10_000);
+ assert.equal(soon.answer, null);
+ // Compared with the FIRST sample, not the refused one.
+ const later = await detectLocationHealth(loc(h.root), h.bins, {
+ sysBlockDir: h.sys,
+ now: 15_000,
+ });
+ assert.equal(later.answer, "stalled");
+ });
+});
+
+test("counters: the minimum spacing follows the pass interval (storage.health.passIntervalMs)", async () => {
+ await withHarness(async (h) => {
+ await h.control({ source: "/dev/fakedisk1" });
+ await h.counters("fakedisk1", 100, 2);
+ // A pass every 8 s: samples 4 s apart are compared (8 − 5 = 3 s, floored
+ // at half the interval), which the default 15 s pass would refuse.
+ applyHealthTimings({ passIntervalMs: 8_000 });
+ assert.equal(minCounterIntervalMs(), 4_000);
+ await detectLocationHealth(loc(h.root), h.bins, { sysBlockDir: h.sys, now: 0 });
+ const early = await detectLocationHealth(loc(h.root), h.bins, {
+ sysBlockDir: h.sys,
+ now: 3_999,
+ });
+ assert.equal(early.answer, null);
+ const due = await detectLocationHealth(loc(h.root), h.bins, {
+ sysBlockDir: h.sys,
+ now: 4_000,
+ });
+ assert.equal(due.answer, "stalled");
+ // A pass every 5 minutes: 10 s, as at the default.
+ applyHealthTimings({ passIntervalMs: 300_000 });
+ assert.equal(minCounterIntervalMs(), 10_000);
+ });
+});
+
+test("no device → the child stat, and the verdict says so", async () => {
+ await withHarness(async (h) => {
+ const opts = { sysBlockDir: h.sys, now: 0 };
+ // A tmpfs or network source names no block device.
+ await h.control({ source: "tmpfs" });
+ assert.deepEqual(await detectLocationHealth(loc(h.root), h.bins, opts), {
+ answer: "ok",
+ detector: "stat",
+ });
+ // findmnt cannot say (no such path, a container without it).
+ await h.control({ exit: 1 });
+ assert.equal((await detectLocationHealth(loc(h.root), h.bins, opts)).detector, "stat");
+ assert.equal(
+ (await detectLocationHealth(loc(path.join(h.root, "gone")), h.bins, opts)).answer,
+ "absent",
+ );
+ // A device with no /sys entry.
+ await h.control({ source: "/dev/nodisk9" });
+ assert.equal((await detectLocationHealth(loc(h.root), h.bins, opts)).detector, "stat");
+ // A root whose filesystem is not the location's recorded volume.
+ await h.counters("fakedisk1", 1, 0);
+ await h.control({ source: "/dev/fakedisk1", uuid: "someone-else" });
+ assert.equal(
+ (await detectLocationHealth(loc(h.root, "u-1"), h.bins, opts)).detector,
+ "stat",
+ );
+ // No findmnt binary at all.
+ assert.equal(
+ (
+ await detectLocationHealth(
+ loc(h.root),
+ { findmntBin: path.join(h.root, "no-findmnt") },
+ opts,
+ )
+ ).detector,
+ "stat",
+ );
+ });
+});
+
+test("a device already named is read without findmnt; findmnt again only when its /sys entry stops reading", async () => {
+ await withHarness(async (h) => {
+ await h.control({ source: "/dev/fakedisk1" });
+ await h.counters("fakedisk1", 100, 2);
+ await detectLocationHealth(loc(h.root), h.bins, { sysBlockDir: h.sys, now: 0 });
+ // findmnt now hangs: it is not asked, the known device is read.
+ await h.control({ sleepMs: 5_000, source: "/dev/fakedisk1" });
+ const started = Date.now();
+ const v = await detectLocationHealth(loc(h.root), h.bins, {
+ sysBlockDir: h.sys,
+ now: 15_000,
+ timeoutMs: 300,
+ });
+ assert.ok(Date.now() - started < 250, "no findmnt was run");
+ assert.equal(v.detector, "counters");
+ assert.equal(v.answer, "stalled");
+ // The device is gone from /sys (replugged under another name): findmnt is
+ // asked again — here it does not answer, so no device, and the child stat.
+ await rm(path.join(h.sys, "fakedisk1"), { recursive: true });
+ const again = await detectLocationHealth(loc(h.root), h.bins, {
+ sysBlockDir: h.sys,
+ now: 30_000,
+ timeoutMs: 300,
+ });
+ assert.equal(again.detector, "stat");
+ // The detector's samples are on globalThis (the pass and a Refresh run in
+ // different module copies and compare against one previous sample).
+ assert.ok(globalThis.__yttHealthDetector__);
+ });
+});
diff --git a/common/lib/storageHealthProbe.test.ts b/common/lib/storageHealthProbe.test.ts
@@ -0,0 +1,128 @@
+import test from "node:test";
+import assert from "node:assert/strict";
+import path from "node:path";
+import { tmpdir } from "node:os";
+import { chmod, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
+import { probeLocationHealth } from "./storageVolumes";
+import { HEALTH_TIMING_DEFAULTS } from "./storageHealthTimings";
+import { applyHealthTimings } from "./storageHealth";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common exec tsx --test lib/storageHealthProbe.test.ts
+//
+// THE HEALTH PROBE: `stat` on a location's root as a CHILD PROCESS, raced
+// against a timer. No test stalls a real drive: a stalled `stat` is a fake
+// binary that never answers (a child blocked in the kernel looks the same from
+// here — it does not exit), and the case that matters is that the answer
+// arrives on the timer while the child is still running.
+
+async function withDir(fn: (dir: string) => Promise<void>): Promise<void> {
+ const dir = await mkdtemp(path.join(tmpdir(), "ttb-health-"));
+ try {
+ await fn(dir);
+ } finally {
+ await rm(dir, { recursive: true, force: true });
+ }
+}
+
+// A fake `stat`: sleeps `sleepMs` (a blocking sleep, so it is simply not
+// answering), then prints `out` and exits `code`.
+async function fakeStat(
+ dir: string,
+ opts: { sleepMs?: number; out?: string; code?: number },
+): Promise<string> {
+ const bin = path.join(dir, "fake-stat.mjs");
+ await writeFile(
+ bin,
+ `#!/usr/bin/env node
+if (${opts.sleepMs ?? 0} > 0) {
+ Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ${opts.sleepMs ?? 0});
+}
+process.stdout.write(${JSON.stringify(opts.out ?? "")} + "\\n");
+process.exit(${opts.code ?? 0});
+`,
+ );
+ await chmod(bin, 0o755);
+ return bin;
+}
+
+test("the real stat: a directory is ok, a missing path and a file are absent", async () => {
+ await withDir(async (dir) => {
+ const root = path.join(dir, "media");
+ await mkdir(root);
+ await writeFile(path.join(dir, "a-file"), "x");
+ assert.equal(await probeLocationHealth({ root }), "ok");
+ assert.equal(await probeLocationHealth({ root: path.join(dir, "gone") }), "absent");
+ assert.equal(await probeLocationHealth({ root: path.join(dir, "a-file") }), "absent");
+ assert.equal(await probeLocationHealth({ root: " " }), "absent");
+ });
+});
+
+test("a child that never answers is 'stalled' on the timer, without waiting for it", async () => {
+ await withDir(async (dir) => {
+ // Twenty seconds asleep: if the probe waited for the child, this test would
+ // take that long.
+ const statBin = await fakeStat(dir, { sleepMs: 20_000, out: "directory" });
+ const started = Date.now();
+ const answer = await probeLocationHealth(
+ { root: dir },
+ { statBin, timeoutMs: 400 },
+ );
+ const took = Date.now() - started;
+ assert.equal(answer, "stalled");
+ assert.ok(took >= 400, `answered after ${took} ms, before the timer`);
+ assert.ok(took < 3_000, `answered after ${took} ms`);
+ });
+});
+
+test("the default budget is 3 s", async () => {
+ assert.equal(HEALTH_TIMING_DEFAULTS.probeTimeoutMs, 3_000);
+ await withDir(async (dir) => {
+ const statBin = await fakeStat(dir, { sleepMs: 20_000, out: "directory" });
+ const started = Date.now();
+ assert.equal(await probeLocationHealth({ root: dir }, { statBin }), "stalled");
+ const took = Date.now() - started;
+ assert.ok(took >= 3_000 && took < 6_000, `answered after ${took} ms`);
+ });
+});
+
+test("the budget is storage.health.probeTimeoutMs when the caller names none", async () => {
+ applyHealthTimings({ probeTimeoutMs: 500 });
+ try {
+ await withDir(async (dir) => {
+ const statBin = await fakeStat(dir, { sleepMs: 20_000, out: "directory" });
+ const started = Date.now();
+ assert.equal(await probeLocationHealth({ root: dir }, { statBin }), "stalled");
+ const took = Date.now() - started;
+ assert.ok(took >= 500 && took < 2_500, `answered after ${took} ms`);
+ });
+ } finally {
+ applyHealthTimings();
+ }
+});
+
+test("an answer inside the budget is taken as given", async () => {
+ await withDir(async (dir) => {
+ const slowDir = await fakeStat(dir, { sleepMs: 100, out: "directory" });
+ assert.equal(
+ await probeLocationHealth({ root: dir }, { statBin: slowDir, timeoutMs: 2_500 }),
+ "ok",
+ );
+ const aFile = await fakeStat(dir, { out: "regular file" });
+ assert.equal(await probeLocationHealth({ root: dir }, { statBin: aFile }), "absent");
+ const missing = await fakeStat(dir, { code: 1 });
+ assert.equal(await probeLocationHealth({ root: dir }, { statBin: missing }), "absent");
+ });
+});
+
+test("a stat that cannot be started fails open: ok, never stalled", async () => {
+ await withDir(async (dir) => {
+ assert.equal(
+ await probeLocationHealth(
+ { root: dir },
+ { statBin: path.join(dir, "no-such-stat") },
+ ),
+ "ok",
+ );
+ });
+});
diff --git a/common/lib/storageHealthTimings.test.ts b/common/lib/storageHealthTimings.test.ts
@@ -0,0 +1,138 @@
+// THE DRIVE-HEALTH TIMINGS AS SETTINGS: `settings.storage.health`.
+//
+// Run with:
+// pnpm --filter yt-dlp-transcript-common exec tsx --test lib/storageHealthTimings.test.ts
+//
+// The claims: absent is every default; a read clamps an out-of-range value into
+// its range (the schema's rule: a read never throws); only a value that differs
+// from its default is kept, so an untuned file has no `health` key and a save
+// writes none; the block survives a save and a read, and the mediaRoot
+// migration. Its own file for the SETTINGS SEAM (settingsWrite.test.ts says
+// why): SETTINGS_FILE is set before the settings module is imported.
+
+import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
+import { rm } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { after, test } from "node:test";
+import assert from "node:assert/strict";
+import {
+ HEALTH_TIMING_BOUNDS,
+ HEALTH_TIMING_DEFAULTS,
+ HEALTH_TIMING_KEYS,
+ clearRuleText,
+ counterSampleMinimumMs,
+ resolveHealthTimings,
+ sanitizeStorageHealth,
+ secondsText,
+} from "./storageHealthTimings";
+
+const ROOT = mkdtempSync(path.join(os.tmpdir(), "health-timings-"));
+process.env.TRANSCRIPTS_DIR = ROOT;
+process.env.SETTINGS_FILE = path.join(ROOT, "settings.json");
+
+const { getSettings, writeSettings, siteSettingsSchema, defaultSiteSettings } =
+ await import("./settings");
+
+after(() => rm(ROOT, { recursive: true, force: true }));
+
+const onDisk = () =>
+ JSON.parse(readFileSync(process.env.SETTINGS_FILE!, "utf8")) as {
+ storage: Record<string, unknown>;
+ };
+
+test("the defaults are the constants they replace", () => {
+ assert.deepEqual(HEALTH_TIMING_DEFAULTS, {
+ budgetMs: 3_000,
+ passIntervalMs: 15_000,
+ probeTimeoutMs: 3_000,
+ clearAfterCleanPasses: 2,
+ inFlightPerLocation: 4,
+ });
+ assert.deepEqual([...HEALTH_TIMING_KEYS].sort(), Object.keys(HEALTH_TIMING_DEFAULTS).sort());
+ for (const key of HEALTH_TIMING_KEYS) {
+ const { min, max } = HEALTH_TIMING_BOUNDS[key];
+ const d = HEALTH_TIMING_DEFAULTS[key];
+ assert.ok(min <= d && d <= max, `${key}'s default is in its range`);
+ }
+});
+
+test("absent is every default: no block, an empty block, junk", () => {
+ for (const raw of [undefined, null, {}, [], "3000", 7]) {
+ assert.deepEqual(sanitizeStorageHealth(raw), {});
+ assert.deepEqual(resolveHealthTimings(raw as never), HEALTH_TIMING_DEFAULTS);
+ }
+ assert.equal(getSettings().storage.health, undefined, "no file: no block");
+ assert.equal(defaultSiteSettings().storage.health, undefined);
+});
+
+test("a read clamps into the range, rounds, drops non-numbers and keeps only what differs from the default", () => {
+ assert.deepEqual(
+ sanitizeStorageHealth({
+ budgetMs: 200, // below 500
+ passIntervalMs: 900_000, // above 300000
+ probeTimeoutMs: "4000", // a string is not a number
+ clearAfterCleanPasses: 2, // the default
+ inFlightPerLocation: 7.6,
+ unknown: 1,
+ }),
+ { budgetMs: 500, passIntervalMs: 300_000, inFlightPerLocation: 8 },
+ );
+ // A value that clamps ONTO its default is the default, and is not kept.
+ assert.deepEqual(sanitizeStorageHealth({ clearAfterCleanPasses: 2.2 }), {});
+ assert.deepEqual(resolveHealthTimings({ budgetMs: 200 }), {
+ ...HEALTH_TIMING_DEFAULTS,
+ budgetMs: 500,
+ });
+});
+
+test("a settings.json round trip: a tuned value is written, read back, and a default is not written", async () => {
+ const base = defaultSiteSettings();
+ await writeSettings({
+ ...base,
+ storage: { ...base.storage, health: { budgetMs: 4_000, clearAfterCleanPasses: 2 } },
+ });
+ assert.deepEqual(onDisk().storage.health, { budgetMs: 4_000 });
+ assert.deepEqual(getSettings().storage.health, { budgetMs: 4_000 });
+ // Every value back to its default: the key is not written at all.
+ await writeSettings({ ...base, storage: { ...base.storage, health: { budgetMs: 3_000 } } });
+ assert.equal("health" in onDisk().storage, false);
+ // A hand-edited file out of range reads clamped.
+ writeFileSync(
+ process.env.SETTINGS_FILE!,
+ JSON.stringify({ storage: { locations: [], health: { inFlightPerLocation: 99 } } }),
+ );
+ assert.deepEqual(getSettings().storage.health, { inFlightPerLocation: 8 });
+});
+
+test("the timings survive the mediaRoot migration (a block with no locations)", () => {
+ const parsed = siteSettingsSchema.parse({
+ storage: { health: { passIntervalMs: 30_000 } },
+ });
+ assert.deepEqual(parsed.storage.health, { passIntervalMs: 30_000 });
+ writeFileSync(
+ process.env.SETTINGS_FILE!,
+ JSON.stringify({ storage: { mediaRoot: "/mnt/cold", health: { budgetMs: 6_000 } } }),
+ );
+ const s = getSettings();
+ assert.deepEqual(s.storage.locations.map((l) => l.root), ["/mnt/cold"]);
+ assert.deepEqual(s.storage.health, { budgetMs: 6_000 });
+});
+
+test("the counters' sample spacing: min(10 s, interval − 5 s), floored at half the interval", () => {
+ assert.equal(counterSampleMinimumMs(15_000), 10_000, "the default, as it was");
+ assert.equal(counterSampleMinimumMs(300_000), 10_000);
+ assert.equal(counterSampleMinimumMs(12_000), 7_000);
+ assert.equal(counterSampleMinimumMs(10_000), 5_000);
+ assert.equal(counterSampleMinimumMs(8_000), 4_000, "the floor: 3 s would be under half");
+ assert.equal(counterSampleMinimumMs(5_000), 2_500, "never 0 at the minimum interval");
+});
+
+test("the words", () => {
+ assert.equal(secondsText(3_000), "3 s");
+ assert.equal(secondsText(2_500), "2.5 s");
+ assert.equal(secondsText(80), "0.08 s");
+ assert.equal(clearRuleText(1), "once");
+ assert.equal(clearRuleText(2), "twice in a row");
+ assert.equal(clearRuleText(5), "5 times in a row");
+});
diff --git a/common/lib/storageHealthTimings.ts b/common/lib/storageHealthTimings.ts
@@ -0,0 +1,166 @@
+import type { FieldDocs } from "./fieldDocs";
+
+// THE DRIVE-HEALTH TIMINGS — `settings.storage.health` — and their defaults.
+//
+// The health gate (lib/storageHealth.ts) decides that a drive is not answering
+// from a handful of numbers: how long one read may take, how often the health
+// pass looks at the drive, how long that look may take, how many clean looks in
+// a row clear a stall, and how many reads may wait on one drive at once. They
+// were constants. Under heavy external-disk churn a stall can be misjudged, and
+// the ruling (release 15, slice DT) is that the operator then tunes the numbers
+// on /storage rather than the code. Today's constants are the defaults.
+//
+// THE STORED BLOCK HOLDS ONLY WHAT DIFFERS FROM A DEFAULT. Every key is
+// optional, an absent key is its default, and `sanitizeStorageHealth` drops a
+// value equal to its default — so a settings.json that never tuned anything has
+// no `health` key at all, and a default changed in a later release reaches it.
+//
+// READS CLAMP, THE FORM REFUSES. A hand-edited value outside its range is
+// clamped into it on read (the schema's rule: a read never throws); the /storage
+// form refuses one with a sentence instead, because a save that quietly stored
+// another number than the one typed reads as a form that did not listen.
+//
+// PURE, NO IMPORTS BUT A TYPE: the /storage form (a "use client" file) imports
+// the defaults, the ranges and the words from here.
+
+export type StorageHealthSettings = {
+ budgetMs?: number;
+ passIntervalMs?: number;
+ probeTimeoutMs?: number;
+ clearAfterCleanPasses?: number;
+ inFlightPerLocation?: number;
+};
+
+// Every timing, resolved: what the health module runs on.
+export type HealthTimings = Required<StorageHealthSettings>;
+
+export type HealthTimingKey = keyof HealthTimings;
+
+// Today's constants (release 15 slice DS), now the defaults.
+export const HEALTH_TIMING_DEFAULTS: Readonly<HealthTimings> = Object.freeze({
+ budgetMs: 3_000,
+ passIntervalMs: 15_000,
+ probeTimeoutMs: 3_000,
+ clearAfterCleanPasses: 2,
+ inFlightPerLocation: 4,
+});
+
+export const HEALTH_TIMING_BOUNDS: Readonly<
+ Record<HealthTimingKey, { min: number; max: number }>
+> = Object.freeze({
+ budgetMs: { min: 500, max: 60_000 },
+ passIntervalMs: { min: 5_000, max: 300_000 },
+ probeTimeoutMs: { min: 500, max: 30_000 },
+ clearAfterCleanPasses: { min: 1, max: 10 },
+ // At most HALF the editor's 16 file-access threads (`UV_THREADPOOL_SIZE` in
+ // `start` and the container): one drive that stops answering holds this
+ // many of them, and at 16 it would hold every one — the outage the cap
+ // exists to bound.
+ inFlightPerLocation: { min: 1, max: 8 },
+});
+
+// In the order the form shows them.
+export const HEALTH_TIMING_KEYS: readonly HealthTimingKey[] = [
+ "budgetMs",
+ "passIntervalMs",
+ "probeTimeoutMs",
+ "clearAfterCleanPasses",
+ "inFlightPerLocation",
+];
+
+// A whole number in range, or undefined (not a finite number).
+export function clampHealthTiming(key: HealthTimingKey, value: unknown): number | undefined {
+ if (typeof value !== "number" || !Number.isFinite(value)) return undefined;
+ const { min, max } = HEALTH_TIMING_BOUNDS[key];
+ return Math.min(max, Math.max(min, Math.round(value)));
+}
+
+// Coerce a raw `settings.storage.health` into the stored block: each value a
+// number clamped into its range, and only when it differs from its default.
+// Anything else — a string, a key the block does not name — is dropped.
+export function sanitizeStorageHealth(value: unknown): StorageHealthSettings {
+ if (!value || typeof value !== "object" || Array.isArray(value)) return {};
+ const r = value as Record<string, unknown>;
+ const out: StorageHealthSettings = {};
+ for (const key of HEALTH_TIMING_KEYS) {
+ const n = clampHealthTiming(key, r[key]);
+ if (n !== undefined && n !== HEALTH_TIMING_DEFAULTS[key]) out[key] = n;
+ }
+ return out;
+}
+
+// The stored block with every absent key filled from the defaults.
+export function resolveHealthTimings(stored?: StorageHealthSettings): HealthTimings {
+ const clean = sanitizeStorageHealth(stored);
+ return { ...HEALTH_TIMING_DEFAULTS, ...clean };
+}
+
+// THE COUNTERS' SAMPLE SPACING, derived from the pass interval. Two samples of a
+// disk's request counters closer than this are not compared: a healthy drive can
+// have a request in flight at two instants a moment apart without completing
+// one (release 15 DS, "kept at review"). The ruling: min(10 s, interval − 5 s),
+// so consecutive passes always compare — which is 10 s at the default 15 s, as
+// it was. Below a 10 s interval that formula falls toward nothing (0 at the 5 s
+// minimum, where a /storage Refresh just after a pass would compare two samples
+// taken milliseconds apart), so it is floored at half the interval.
+export function counterSampleMinimumMs(passIntervalMs: number): number {
+ return Math.min(10_000, Math.max(passIntervalMs - 5_000, Math.round(passIntervalMs / 2)));
+}
+
+// "3 s", "2.5 s", "0.08 s": a millisecond figure in the seconds every surface
+// uses.
+export function secondsText(ms: number): string {
+ return `${ms / 1000} s`;
+}
+
+// How many clean answers clear a stall, in words: "once", "twice in a row",
+// "3 times in a row".
+export function clearRuleText(n: number): string {
+ return n === 1 ? "once" : n === 2 ? "twice in a row" : `${n} times in a row`;
+}
+
+// What each field means, in the operator's words: the /storage form's hint line.
+export const HEALTH_TIMING_HINTS: Readonly<Record<HealthTimingKey, string>> = Object.freeze({
+ budgetMs:
+ "How long one read may take before the drive counts as not answering. Takes effect on the next read.",
+ passIntervalMs:
+ "How often each drive is checked, from its disk's own request counters. A save re-arms the check at once.",
+ probeTimeoutMs:
+ "How long one check may wait for the drive's root where no disk can be named (and for naming the disk).",
+ clearAfterCleanPasses:
+ "How many clean checks in a row it takes before a drive marked not answering is used again.",
+ inFlightPerLocation:
+ "How many reads may be on one drive at once; the rest wait their turn, and are refused if it stops answering. At most 8, half the editor's 16 file-access threads: a drive that stops answering holds this many of them.",
+});
+
+// SETTINGS.md's `storage.health` table.
+export const STORAGE_HEALTH_SETTINGS_FIELD_DOCS: FieldDocs<StorageHealthSettings> = {
+ budgetMs:
+ "How long one read may take before the drive counts as not answering, in ms (default 3000, " +
+ "500–60000). The watchdog's budget (`onDrive`, lib/storageHealth.ts) for one unit of work — a " +
+ "video directory's reads, a page's reads of one video: a unit that has not answered by then is " +
+ "refused, and marks its location not answering unless the disk's request counters show it still " +
+ "completing others (slow, not stalled). A read waiting for a slot is refused when nothing on the " +
+ "drive has returned for this long plus a quarter of it (at most 250 ms). Takes effect on the next " +
+ "read after a save.",
+ passIntervalMs:
+ "How often the health pass reads each location's disk counters, in ms (default 15000, " +
+ "5000–300000). A stall that starts between two passes is seen by the next, or at once by a page's " +
+ "read. A save on /storage re-arms the pass's timer at once; a hand edit, at the next pass. Two " +
+ "counter samples are compared only when at least min(10 s, this − 5 s) apart, a spacing never " +
+ "less than half of this.",
+ probeTimeoutMs:
+ "How long the health pass waits, in ms (default 3000, 500–30000), for the child `stat` of a root " +
+ "where no disk can be named (a timeout counts as not answering) and for the `findmnt` that names a " +
+ "root's disk (a timeout names none that pass). Takes effect on the next pass.",
+ clearAfterCleanPasses:
+ "How many clean answers in a row clear a location marked not answering (default 2, 1–10). Each " +
+ "health pass is one answer, and so is a Refresh on /storage; a miss in between starts the count " +
+ "again. Takes effect on the next answer.",
+ inFlightPerLocation:
+ "How many reads through the watchdog may be on one location's drive at once (default 4, 1–8); the " +
+ "rest wait in the editor's own queue, so a stall mid-walk holds this many of Node's threads, not " +
+ "all of them. At most 8, half of `UV_THREADPOOL_SIZE` (16 in the editor's start script and the " +
+ "container), so one drive that stops answering cannot hold every thread. Takes effect on the next " +
+ "read.",
+};
diff --git a/common/lib/storageLocations.ts b/common/lib/storageLocations.ts
@@ -1,5 +1,6 @@
import path from "node:path";
import type { FieldDocs } from "./fieldDocs";
+import type { StorageHealthSettings } from "./storageHealthTimings";
// STORAGE LOCATIONS — the named places a channel's media may live.
//
@@ -95,6 +96,7 @@ export type StorageSettings = {
locations: StorageLocation[];
defaultLocationId: string;
savedVideosLocationId?: string;
+ health?: StorageHealthSettings;
};
export const STORAGE_SETTINGS_FIELD_DOCS: FieldDocs<StorageSettings> = {
@@ -112,6 +114,14 @@ export const STORAGE_SETTINGS_FIELD_DOCS: FieldDocs<StorageSettings> = {
"Optional so an older settings.json parses (and an older binary that " +
"drops it leaves a store that still works, because the symlink is what " +
"every reader follows).",
+ health:
+ "THE DRIVE-HEALTH TIMINGS: how long a read may take before a drive " +
+ "counts as not answering, how often the health pass looks, how long " +
+ "its look may take, how many clean looks clear a stall, and how many " +
+ "reads may be on one drive at once. Edited on /storage (Drive health " +
+ "timing). Absent = every default, and only a value that differs from " +
+ "its default is written, so an untuned install follows a default " +
+ "changed later. See `storage.health` below.",
};
// Strip trailing slashes so "/mnt/platter/" and "/mnt/platter" are one root.
@@ -214,9 +224,12 @@ export function migrateMediaRootToLocations(raw: unknown): unknown {
if (!raw || typeof raw !== "object" || Array.isArray(raw)) return raw;
const r = raw as Record<string, unknown>;
if (r.locations !== undefined) return raw;
+ // The drive-health timings are not about the locations: a block that spells
+ // them and no `locations` (a hand edit) keeps them through the migration.
+ const health = r.health !== undefined ? { health: r.health } : {};
const mediaRoot = typeof r.mediaRoot === "string" ? r.mediaRoot.trim() : "";
if (mediaRoot === "" || !path.isAbsolute(mediaRoot)) {
- return { locations: [], defaultLocationId: "" };
+ return { locations: [], defaultLocationId: "", ...health };
}
return {
locations: [
@@ -228,5 +241,6 @@ export function migrateMediaRootToLocations(raw: unknown): unknown {
},
],
defaultLocationId: "default",
+ ...health,
};
}
diff --git a/common/lib/storageVolumes.test.ts b/common/lib/storageVolumes.test.ts
@@ -139,7 +139,7 @@ test("available: an automounted root warns, and names its fstab line", async ()
await h.control({
// udisks mounts under /run/media/<user>/<uuid>. The mountpoint is what
// gives it away; the fstab query is not even made.
- identity: { target: "/run/media/user/" + UUID, fstype: "ext4", uuid: UUID },
+ identity: { target: "/run/media/operator/" + UUID, fstype: "ext4", uuid: UUID },
fstab: `UUID=${UUID}`,
});
const p = await probeLocation(loc(root), h.bins, {
@@ -169,7 +169,7 @@ test("available: a mount with no fstab entry warns too", async () => {
test("mounted-elsewhere: the recorded uuid is up under another mountpoint", async () => {
await withHarness(async (h) => {
- await h.control({ uuidTarget: "/run/media/user/" + UUID });
+ await h.control({ uuidTarget: "/run/media/operator/" + UUID });
const p = await probeLocation(
loc(path.join(h.dir, "gone", "archilyzer-media"), true),
h.bins,
@@ -178,10 +178,10 @@ test("mounted-elsewhere: the recorded uuid is up under another mountpoint", asyn
assert.equal(p.status, "mounted-elsewhere");
assert.equal(
p.candidateRoot,
- `/run/media/user/${UUID}/archilyzer-media`,
+ `/run/media/operator/${UUID}/archilyzer-media`,
);
// The sighting is live enough to write back as the location's volume.
- assert.equal(p.identity.known && p.identity.mountpoint, `/run/media/user/${UUID}`);
+ assert.equal(p.identity.known && p.identity.mountpoint, `/run/media/operator/${UUID}`);
assert.equal(p.identity.known && p.identity.fstype, "ext4");
assert.equal(p.freeBytes, undefined);
});
diff --git a/common/lib/storageVolumes.ts b/common/lib/storageVolumes.ts
@@ -1,9 +1,23 @@
import path from "node:path";
-import { lstat, stat } from "node:fs/promises";
+import { readFileSync } from "node:fs";
+import { lstat, readFile, realpath, stat } from "node:fs/promises";
import { execa } from "execa";
import type { Paths } from "./paths";
import { getFreeBytes } from "./diskSpace";
import type { StorageLocation, StorageVolume } from "./storageLocations";
+import {
+ countersVerdict,
+ healthTimings,
+ isDriveNotAnswering,
+ onDrive,
+ parseBlockStat,
+ setCounterReader,
+ stalledLocation,
+ type BlockStatSample,
+ type HealthDetector,
+ type LocationHealthState,
+} from "./storageHealth";
+import { counterSampleMinimumMs, secondsText } from "./storageHealthTimings";
// STORAGE VOLUME PROBES — is this location's disk here, and if not, where?
//
@@ -33,7 +47,11 @@ export type StorageLocationStatus =
| "absent"
// The root is not there and we have no identity to look for — nothing to say
// beyond "that path does not exist".
- | "missing";
+ | "missing"
+ // The drive did not answer the last health probe (`lib/storageHealth.ts`), so
+ // this probe did not ask: an in-process `stat` there would hold one of the
+ // process's I/O threads until the drive answered. No identity, no free space.
+ | "stalled";
// `known: false` is the fail-open answer and is NOT a problem report: it means
// the probe could not ask (no findmnt, a container, a timeout), not that the
@@ -226,13 +244,26 @@ export async function probeLocation(
const timeoutMs = opts.findmntTimeoutMs ?? FINDMNT_TIMEOUT_MS;
const root = loc.root.trim();
+ // A DRIVE THAT IS NOT ANSWERING IS NOT ASKED. The `stat` and `statfs` below
+ // run in-process, and on a stalled disk each holds an I/O thread until the
+ // drive comes back; the health pass (below) already asked without touching
+ // it. Both go through `onDrive`'s watchdog, too: one that has not answered
+ // within the budget (`storage.health.budgetMs`, 3 s by default) marks the
+ // location stalled and this probe answers so.
+ const stalledProbe: StorageLocationProbe = {
+ status: "stalled",
+ identity: { known: false },
+ };
+ if (stalledLocation(loc)) return stalledProbe;
+
// AVAILABILITY IS `stat`, AND ONLY `stat`. A root that is a directory is
// available even when every identity probe below fails — see the header.
let isDir = false;
if (root !== "") {
try {
- isDir = (await stat(root)).isDirectory();
- } catch {
+ isDir = (await onDrive(loc, () => stat(root))).isDirectory();
+ } catch (err) {
+ if (isDriveNotAnswering(err)) return stalledProbe;
isDir = false;
}
}
@@ -245,7 +276,13 @@ export async function probeLocation(
// `null` and every byte formatter downstream gets a surprise. An
// unmeasurable root reports no free space at all, which is the honest
// answer and the one the field is already optional for.
- const free = await getFreeBytes(root);
+ let free: number;
+ try {
+ free = await onDrive(loc, () => getFreeBytes(root));
+ } catch (err) {
+ if (isDriveNotAnswering(err)) return stalledProbe;
+ throw err;
+ }
// Two ways a mount will not be there after a reboot: udisks put it under
// /run/media (or /media) because a human plugged it in, or there is no
// fstab entry naming its UUID. The fstab call is skipped when the
@@ -304,6 +341,327 @@ export async function probeLocation(
}
}
+// ---------------------------------------------------------------------------
+// The health probe: is the drive ANSWERING, asked from a child process
+// ---------------------------------------------------------------------------
+//
+// `probeLocation` answers "is the disk here" with an in-process `stat`, which is
+// right until the disk is here and not answering: then that `stat` does not
+// fail, it waits — for as long as the drive takes, on one of the four threads
+// libuv runs every filesystem call on. A child process waiting in the kernel
+// holds none of them. So this runs `stat` on the root as a SUBPROCESS and races
+// it against a timer, and the timer's answer is `stalled`.
+//
+// THE TIMER WINS, AND NOTHING WAITS FOR THE CHILD. A process in uninterruptible
+// I/O cannot be killed until the I/O returns, so awaiting its exit (which is
+// what execa's own `timeout` does) would hand the wait straight back to us.
+// The child is sent SIGKILL, which it takes when the drive lets it, and its
+// promise settles into a handler nobody awaits.
+//
+// FAILS OPEN, like everything else here: a `stat` that could not be started
+// (no binary) is "could not ask", which is `ok`, never `stalled`. Only a child
+// that started and did not answer in time is a stall.
+
+export type HealthProbeOptions = {
+ timeoutMs?: number;
+ // Test seam: the binary to run (it is called as `<bin> -L -c %F -- <root>`).
+ statBin?: string;
+};
+
+// One pass's answer about one location. `answer: null` is no verdict (the
+// counters' first sample, or a second one taken too soon after the last).
+export type HealthVerdict = {
+ answer: LocationHealthState | null;
+ detector?: HealthDetector;
+ // For a `stalled` answer: what did not answer, in words with no path.
+ cause?: string;
+ // The block device the counters were read from.
+ device?: string;
+};
+
+// What the health pass asks about a location. A bare state is a verdict with
+// no detector named (the tests' scripted probes).
+export type LocationHealthProbe = (
+ loc: Pick<StorageLocation, "id" | "label" | "root" | "volume">,
+) => Promise<LocationHealthState | HealthVerdict>;
+
+export async function probeLocationHealth(
+ loc: Pick<StorageLocation, "root">,
+ opts: HealthProbeOptions = {},
+): Promise<LocationHealthState> {
+ const root = loc.root.trim();
+ if (root === "") return "absent";
+ const timeoutMs = opts.timeoutMs ?? healthTimings().probeTimeoutMs;
+ let child: ReturnType<typeof execa>;
+ try {
+ child = execa(opts.statBin ?? "stat", ["-L", "-c", "%F", "--", root], {
+ buffer: true,
+ reject: false,
+ stdin: "ignore",
+ });
+ } catch {
+ return "ok";
+ }
+ const answered: Promise<LocationHealthState> = child.then(
+ (res) => {
+ // No exit code: the binary never ran (or was killed after the race was
+ // already decided). Could not ask.
+ if (typeof res.exitCode !== "number") return "ok";
+ if (res.exitCode !== 0) return "absent";
+ const out = typeof res.stdout === "string" ? res.stdout.trim() : "";
+ return out === "directory" ? "ok" : "absent";
+ },
+ () => "ok",
+ );
+ let timer: ReturnType<typeof setTimeout> | undefined;
+ const timedOut = new Promise<LocationHealthState>((resolve) => {
+ timer = setTimeout(() => resolve("stalled"), timeoutMs);
+ timer.unref?.();
+ });
+ const answer = await Promise.race([answered, timedOut]);
+ if (timer) clearTimeout(timer);
+ if (answer === "stalled") {
+ try {
+ child.kill("SIGKILL");
+ } catch {
+ /* already gone */
+ }
+ }
+ return answer;
+}
+
+// ---------------------------------------------------------------------------
+// The counters detector: the block device's own request counters
+// ---------------------------------------------------------------------------
+//
+// THE STAT PROBE ABOVE CAN BE ANSWERED FROM THE KERNEL'S CACHE. A root's inode
+// is cached whenever anything has used the drive lately, so a child `stat` of
+// it answers in microseconds while the reads that actually reach the device
+// wait out a reset loop. The block device's counters (lib/storageHealth.ts,
+// `countersVerdict`) are the device's own account of what it has done, and
+// reading them touches only /sys. So the health pass asks this first:
+//
+// 1. the root's device: `findmnt -J -T <root> -o SOURCE,UUID` as a child
+// raced against `probeTimeoutMs` (3 s by default; a findmnt stuck
+// resolving the root holds nothing of ours), the `[subvolume]` suffix a
+// bind or btrfs mount adds taken off,
+// `/dev/mapper/<x>` resolved to its `dm-N`, then the basename. A partition
+// and a mapper device both have `/sys/class/block/<name>/stat`. Asked only
+// when the root has no device yet or its device's /sys entry cannot be read
+// (a drive replugged under another name): a findmnt per pass would leave
+// one child stuck per pass during a long stall. One that times out names
+// none this pass; one whose UUID is not the location's recorded one names
+// none (the root is then a directory on some other filesystem).
+// 2. that device's `stat` line, compared with the previous pass's sample for
+// the location (same device, at least `minCounterIntervalMs()` earlier).
+//
+// NO DEVICE (a container, no findmnt, a network or tmpfs mount, no /sys entry)
+// FALLS BACK TO THE CHILD `stat`, and the verdict says which detector answered.
+
+export type DetectorOptions = HealthProbeOptions & {
+ // Test seams: where /sys/class/block is, and the clock.
+ sysBlockDir?: string;
+ now?: number;
+};
+
+export const SYS_BLOCK_DIR = "/sys/class/block";
+// Two samples closer than this are not compared: a healthy drive can have a
+// request in flight at two instants a moment apart without completing one.
+// Derived from the pass interval (`counterSampleMinimumMs`): 10 s at the
+// default 15 s, so a /storage Refresh just after a pass gives no verdict.
+export function minCounterIntervalMs(): number {
+ return counterSampleMinimumMs(healthTimings().passIntervalMs);
+}
+
+type CounterSample = BlockStatSample & { device: string; at: number };
+
+type DetectorState = {
+ samples: Map<string, CounterSample>;
+ deviceByRoot: Map<string, string>;
+};
+
+declare global {
+ // eslint-disable-next-line no-var
+ var __yttHealthDetector__: DetectorState | undefined;
+}
+
+// ON globalThis, the house pattern: the health pass runs in instrumentation's
+// module copy and /storage's Refresh in a page's, and they must compare
+// against the same previous sample.
+function detectorState(): DetectorState {
+ globalThis.__yttHealthDetector__ ??= {
+ samples: new Map(),
+ deviceByRoot: new Map(),
+ };
+ return globalThis.__yttHealthDetector__;
+}
+
+export function resetHealthDetector(): void {
+ detectorState().samples.clear();
+ detectorState().deviceByRoot.clear();
+}
+
+// THE WATCHDOG'S READING of a device's counters (lib/storageHealth.ts,
+// `onDrive`): synchronous, so it does not wait behind the thread pool it is
+// judging, and cheap — a /sys read is answered by the kernel from memory and
+// never reaches the drive.
+function readCountersNow(device: string): BlockStatSample | null {
+ try {
+ return parseBlockStat(readFileSync(path.join(SYS_BLOCK_DIR, device, "stat"), "utf8"));
+ } catch {
+ return null;
+ }
+}
+setCounterReader(readCountersNow);
+
+type Raced = { answered: false } | ({ answered: true } & Run);
+
+// `run`, but raced against a timer that nobody waits past: a child stuck in
+// the kernel is sent SIGKILL and left to exit when it can.
+async function runRaced(bin: string, args: string[], timeoutMs: number): Promise<Raced> {
+ let child: ReturnType<typeof execa>;
+ try {
+ child = execa(bin, args, { buffer: true, reject: false, stdin: "ignore" });
+ } catch {
+ return { answered: true, ok: false, exitCode: undefined, stdout: "", stderr: "" };
+ }
+ const answered: Promise<Raced> = child.then(
+ (res) => {
+ const exitCode = typeof res.exitCode === "number" ? res.exitCode : undefined;
+ return {
+ answered: true as const,
+ ok: exitCode === 0,
+ exitCode,
+ stdout: typeof res.stdout === "string" ? res.stdout : "",
+ stderr: typeof res.stderr === "string" ? res.stderr : "",
+ };
+ },
+ () => ({ answered: true as const, ok: false, exitCode: undefined, stdout: "", stderr: "" }),
+ );
+ let timer: ReturnType<typeof setTimeout> | undefined;
+ const timedOut = new Promise<Raced>((resolve) => {
+ timer = setTimeout(() => resolve({ answered: false }), timeoutMs);
+ });
+ const out = await Promise.race([answered, timedOut]);
+ if (timer) clearTimeout(timer);
+ if (!out.answered) {
+ try {
+ child.kill("SIGKILL");
+ } catch {
+ /* already gone */
+ }
+ }
+ return out;
+}
+
+// The block device name under /sys/class/block for a mount SOURCE, or null.
+export async function blockDeviceName(source: string): Promise<string | null> {
+ const bare = source.replace(/\[.*\]$/, "").trim();
+ if (!bare.startsWith("/dev/")) return null;
+ // /dev/mapper/<x> and /dev/disk/by-*/<x> are links to the kernel's name.
+ // Resolving them reads /dev, never the drive.
+ const resolved = await realpath(bare).catch(() => bare);
+ const name = path.basename(resolved);
+ return name && name !== "dev" ? name : null;
+}
+
+async function blockDeviceOfRoot(
+ loc: Pick<StorageLocation, "root" | "volume">,
+ bins: Pick<VolumeBins, "findmntBin">,
+ timeoutMs: number,
+): Promise<{ device: string | null; timedOut: boolean }> {
+ const res = await runRaced(
+ bins.findmntBin,
+ ["-J", "-T", loc.root, "-o", "SOURCE,UUID"],
+ timeoutMs,
+ );
+ if (!res.answered) return { device: null, timedOut: true };
+ if (!res.ok) return { device: null, timedOut: false };
+ let fs0: Record<string, unknown> | undefined;
+ try {
+ fs0 = (JSON.parse(res.stdout) as { filesystems?: Record<string, unknown>[] })
+ ?.filesystems?.[0];
+ } catch {
+ return { device: null, timedOut: false };
+ }
+ const source = typeof fs0?.source === "string" ? fs0.source : "";
+ const uuid = typeof fs0?.uuid === "string" ? fs0.uuid : "";
+ const recorded = loc.volume?.uuid?.trim() ?? "";
+ if (recorded && uuid && uuid !== recorded) return { device: null, timedOut: false };
+ return { device: await blockDeviceName(source), timedOut: false };
+}
+
+async function readBlockStat(
+ device: string,
+ sysBlockDir: string,
+): Promise<BlockStatSample | null> {
+ try {
+ return parseBlockStat(await readFile(path.join(sysBlockDir, device, "stat"), "utf8"));
+ } catch {
+ return null;
+ }
+}
+
+// One location's verdict for the health pass: the counters when its device can
+// be named and read, the child `stat` otherwise.
+export async function detectLocationHealth(
+ loc: Pick<StorageLocation, "id" | "root" | "volume">,
+ bins: Pick<VolumeBins, "findmntBin">,
+ opts: DetectorOptions = {},
+): Promise<HealthVerdict> {
+ const now = opts.now ?? Date.now();
+ const timeoutMs = opts.timeoutMs ?? healthTimings().probeTimeoutMs;
+ const sysBlockDir = opts.sysBlockDir ?? SYS_BLOCK_DIR;
+ const detector = detectorState();
+ // The device this root was last known on, if its /sys entry still reads;
+ // findmnt only when there is none (see step 1 above).
+ let device: string | null = detector.deviceByRoot.get(loc.root) ?? null;
+ let sample = device ? await readBlockStat(device, sysBlockDir) : null;
+ if (!sample) {
+ detector.deviceByRoot.delete(loc.root);
+ const mapped = loc.root.trim()
+ ? await blockDeviceOfRoot(loc, bins, timeoutMs)
+ : { device: null, timedOut: false };
+ device = mapped.device;
+ sample = device ? await readBlockStat(device, sysBlockDir) : null;
+ }
+ if (device) {
+ if (sample) {
+ detector.deviceByRoot.set(loc.root, device);
+ const prev = detector.samples.get(loc.id);
+ if (prev && prev.device === device && now - prev.at < minCounterIntervalMs()) {
+ return { answer: null, detector: "counters", device };
+ }
+ detector.samples.set(loc.id, { ...sample, device, at: now });
+ if (!prev || prev.device !== device) {
+ return { answer: null, detector: "counters", device };
+ }
+ const answer = countersVerdict(prev, sample);
+ return {
+ answer,
+ detector: "counters",
+ device,
+ ...(answer === "stalled"
+ ? {
+ cause:
+ `its disk (${device}) had ${sample.inFlight} request(s) in flight and ` +
+ `completed none in ${Math.round((now - prev.at) / 1000)} s`,
+ }
+ : {}),
+ };
+ }
+ }
+ detector.samples.delete(loc.id);
+ const answer = await probeLocationHealth(loc, opts);
+ return {
+ answer,
+ detector: "stat",
+ ...(answer === "stalled"
+ ? { cause: `a stat of its root did not answer within ${secondsText(timeoutMs)}` }
+ : {}),
+ };
+}
+
// udisksctl availability, memoised per binary path. Same shape as a digest
// app's probe (`digestApps.ts` claudeCode.probe): `--version`, reject:false,
// short timeout. Memoised because /storage asks once per render and the answer
@@ -356,7 +714,7 @@ export async function mountByUuid(
const said = res.stderr.trim() || res.stdout.trim();
return { ok: false, error: said || `udisksctl mount failed for ${uuid}` };
}
- // "Mounted /dev/sdb1 at /run/media/user/<uuid>." — the trailing period is
+ // "Mounted /dev/sdb1 at /run/media/<user>/<uuid>." — the trailing period is
// part of the message and not part of the path.
const m = /\bat\s+(.+?)\.?\s*$/m.exec(res.stdout.trim());
return { ok: true, mountpoint: m ? m[1] : undefined };
@@ -401,6 +759,12 @@ export async function probeLocationMemo(
opts: ProbeOptions & { refresh?: boolean; now?: number } = {},
): Promise<MemoizedProbe> {
const now = opts.now ?? Date.now();
+ // The stall is asked before the memo, so a remembered "available" from a few
+ // seconds ago does not outlive the drive's answer. Not remembered either:
+ // the health probe's state is already the memory.
+ if (stalledLocation(loc)) {
+ return { status: "stalled", identity: { known: false }, probedAt: now };
+ }
const hit = probeMemo.get(loc.id);
if (
!opts.refresh &&
diff --git a/common/lib/themeConfig.ts b/common/lib/themeConfig.ts
@@ -0,0 +1,217 @@
+// Shared, dependency-light theme constants, types and the pre-paint script's
+// source. Used by the pre-paint ThemeScript (server), the runtime ThemeProvider
+// (client), the toggle, the unit tests and the homepage e2e (REQUIRED_TOKENS) —
+// and by the source publish, which puts the homepage's pre-paint script on
+// every history page (publish/sourceHistory.ts). That last reader is why this
+// module lives in lib/: the publish layer may not import components/, which
+// re-exports it (components/themeConfig.ts) for the UI's importers.
+// Keys are namespaced under the existing `ytdlp-tb:*` localStorage convention.
+//
+// THE THEME (plans/brand-and-themes.md; two grounds since release 14, T1):
+// • the reader's BASE — the light or the dark ground, or "system", which
+// follows the OS between them. `html[data-base]` selects one of the two
+// token blocks in common/styles/tokens.css; `.dark` is on <html> iff the
+// resolved base is dark, so Tailwind's `dark:` utilities keep working. A
+// stored RETIRED_BASE (the third ground, retired in release 14) is read
+// as, and rewritten to, "light".
+// • the ACCENT — one of the seven named accents in lib/brand.ts, or a
+// site's own hex. It is the SITE's (a server-rendered `html[data-accent]`),
+// never the reader's: a reader's stored pick from before is ignored, and
+// left in storage.
+//
+// PURE: no React, no DOM at import time. lib/brand.ts is pure too.
+
+import { BASE_GROUNDS, type AccentId } from "./brand";
+
+// The reader's base ("light" | "dark" | "system"; RETIRED_BASE, from before
+// release 14, is migrated to "light").
+export const BASE_KEY = "ytdlp-tb:base";
+// A reader's accent pick from before release 14. Nothing reads it and nothing
+// deletes it: each app paints its own accent.
+export const ACCENT_KEY = "ytdlp-tb:accent";
+// The retired theme-family and light/dark-mode keys. Read once by the
+// migration (migrateLegacy, and the same table inside the pre-paint script),
+// then deleted.
+export const LEGACY_THEME_KEY = "ytdlp-tb:theme";
+export const LEGACY_MODE_KEY = "ytdlp-tb:mode";
+
+// The two grounds a base resolves to, and the reader's choice (which adds
+// "system").
+export type ResolvedBase = "light" | "dark";
+export type ThemeBase = ResolvedBase | "system";
+
+// The third ground, retired in release 14: a stored base of this value is
+// light. The one place its name is spelled.
+export const RETIRED_BASE = "sepia";
+
+// What `html[data-accent]` can carry: a named accent, or "custom" — a site
+// whose site.json accent is its own hex (the inline `--accent-custom-*` vars).
+export type ThemeAccent = AccentId | "custom";
+
+// The toggle's cycle order, and every base a reader can store.
+export const THEME_BASES: ReadonlyArray<{ id: ThemeBase; label: string }> = [
+ { id: "system", label: "System" },
+ { id: "light", label: "Light" },
+ { id: "dark", label: "Dark" },
+];
+
+// The project site's base for a reader who has stored none: it opens on the
+// dark ground (homepage/app/layout.tsx passes it to ThemeScript and
+// ThemeProvider). The source's history pages run the same pre-paint script
+// with it, so they open where the homepage does.
+export const HOMEPAGE_DEFAULT_BASE: ThemeBase = "dark";
+
+export function isThemeBase(v: unknown): v is ThemeBase {
+ return v === "system" || v === "light" || v === "dark";
+}
+
+// What a stored base means: a base, RETIRED_BASE → "light", anything else null
+// (the app's default applies).
+export function storedBase(v: unknown): ThemeBase | null {
+ if (v === RETIRED_BASE) return "light";
+ return isThemeBase(v) ? v : null;
+}
+
+// ThemeToggle's cycle, THEME_BASES in order: system → light → dark → system.
+export function nextBase(b: ThemeBase): ThemeBase {
+ const i = THEME_BASES.findIndex((t) => t.id === b);
+ return THEME_BASES[(i + 1) % THEME_BASES.length].id;
+}
+
+// A choice resolved against the OS preference.
+export function resolveBase(b: ThemeBase, systemDark: boolean): ResolvedBase {
+ return b === "system" ? (systemDark ? "dark" : "light") : b;
+}
+
+// Every colour token a base block in tokens.css must declare. The failure this
+// guards is silent: a base that omits a token inherits the light block's value
+// (`:root` always matches), so a half-declared palette looks "a bit off" rather
+// than broken, and only on the base nobody checked. The unit test
+// (themeTokens.test.ts) parses tokens.css against this list; the homepage e2e
+// reads every one off the computed style of each base.
+export const REQUIRED_TOKENS = [
+ "--background",
+ "--foreground",
+ "--card",
+ "--card-foreground",
+ "--popover",
+ "--popover-foreground",
+ "--primary",
+ "--primary-foreground",
+ "--secondary",
+ "--secondary-foreground",
+ "--muted",
+ "--muted-foreground",
+ "--accent",
+ "--accent-foreground",
+ "--destructive",
+ "--destructive-foreground",
+ "--destructive-soft",
+ "--border",
+ "--border-strong",
+ "--input",
+ "--ring",
+ "--surface",
+ "--faint",
+ "--panel",
+ "--panel-2",
+ "--success",
+ "--success-foreground",
+ "--success-soft",
+ "--warning",
+ "--warning-foreground",
+ "--warning-soft",
+ "--info",
+ "--info-foreground",
+ "--info-soft",
+ "--brand",
+ "--brand-strong",
+ "--brand-soft",
+ "--brand-ink",
+ "--state-gone",
+ "--state-gone-soft",
+ "--chart-1",
+ "--chart-2",
+ "--chart-3",
+ "--chart-4",
+ "--chart-5",
+ "--chart-6",
+ "--chart-surface",
+ "--chart-grid",
+ "--chart-axis",
+ "--chart-tooltip-bg",
+] as const;
+
+// The retired keys → a base, once. After it runs, both legacy keys are deleted
+// by the caller, whatever it returned.
+//
+// stored mode result
+// light light (the old "archive" paper theme too: it was the
+// third ground until that was retired)
+// dark dark
+// system system
+// absent/other null: nothing is stored and the app's default base applies
+//
+// The old theme FAMILY is otherwise dropped: every family retired, and the
+// accent is the site's.
+export function migrateLegacy({
+ mode,
+}: {
+ theme: string | null;
+ mode: string | null;
+}): ThemeBase | null {
+ if (mode === "light") return "light";
+ if (mode === "dark") return "dark";
+ if (mode === "system") return "system";
+ return null;
+}
+
+// The pre-paint script, as the string ThemeScript.tsx inlines. It runs
+// synchronously before first paint, in this order:
+// 1. when no base is stored yet, migrate the legacy keys (migrateLegacy's
+// table); then delete both legacy keys;
+// 2. a stored RETIRED_BASE becomes "light", in storage too (once);
+// 3. validate the stored base, falling back to `defaultBase`;
+// 4. set `data-base` to the RESOLVED ground and toggle `.dark`;
+// 5. point every `meta[name=theme-color]` at the resolved ground;
+// 6. LAST: `data-theme-ready="1"`, the e2e no-flash marker, so it is present
+// only once every attribute above is set. e2e's reloads resolve on
+// navigation commit, possibly before this head script has run, and wait
+// for the marker instead of racing.
+// The accent is not read: `html[data-accent]` is the server's (the site's
+// own), and a stored ACCENT_KEY stays where it is, unread.
+// Storage may throw (privacy modes): each storage touch is guarded, so a
+// failure still paints the default base and still sets the marker.
+// themeConfig.test.ts runs this string in node:vm over the whole matrix.
+export function buildThemeScript({
+ defaultBase = "system",
+}: { defaultBase?: ThemeBase } = {}): string {
+ const fallback: ThemeBase = isThemeBase(defaultBase) ? defaultBase : "system";
+ const q = JSON.stringify;
+ return (
+ "(function(){try{" +
+ "var d=document.documentElement,b=null,s;" +
+ "try{s=window.localStorage;" +
+ `b=s.getItem(${q(BASE_KEY)});` +
+ `var lt=s.getItem(${q(LEGACY_THEME_KEY)}),lm=s.getItem(${q(LEGACY_MODE_KEY)});` +
+ "if(lt!==null||lm!==null){" +
+ "if(b===null){" +
+ "var m=lm==='light'?'light':lm==='dark'?'dark':lm==='system'?'system':null;" +
+ `if(m){s.setItem(${q(BASE_KEY)},m);b=m;}` +
+ "}" +
+ `s.removeItem(${q(LEGACY_THEME_KEY)});s.removeItem(${q(LEGACY_MODE_KEY)});` +
+ "}" +
+ `if(b===${q(RETIRED_BASE)}){b='light';s.setItem(${q(BASE_KEY)},'light');}` +
+ "}catch(e){}" +
+ `if(b===${q(RETIRED_BASE)})b='light';` +
+ `if(b!=='light'&&b!=='dark'&&b!=='system')b=${q(fallback)};` +
+ "var r=b;" +
+ "if(b==='system'){r='light';try{if(window.matchMedia('(prefers-color-scheme: dark)').matches)r='dark';}catch(e){}}" +
+ "d.setAttribute('data-base',r);" +
+ "d.classList.toggle('dark',r==='dark');" +
+ `var g=${q(BASE_GROUNDS)}[r];` +
+ "try{var ms=document.querySelectorAll('meta[name=\"theme-color\"]');for(var i=0;i<ms.length;i++)ms[i].setAttribute('content',g);}catch(e){}" +
+ "d.setAttribute('data-theme-ready','1');" +
+ "}catch(e){}})();"
+ );
+}
diff --git a/common/lib/wordmarkMetrics.ts b/common/lib/wordmarkMetrics.ts
@@ -0,0 +1,58 @@
+// GENERATED by common/bin/gen-wordmark-metrics.py -- do not edit; re-run it.
+//
+// The advance, in font units, of each Latin character Archivo[wdth,wght].ttf maps, at the
+// wordmark's instances (wdth 118; wght 720 and 380):
+// what lib/wordmarkWidth.ts reserves a title's width by.
+export const WORDMARK_METRICS = {
+ font: "Archivo[wdth,wght].ttf",
+ sha256: "0e094a7d3c7c4c25cf1310c4b30014f1dae9332220b1c2c88f4fa996f0b05053",
+ fontTools: "4.65.0",
+ unitsPerEm: 1000,
+ wdth: 118,
+ // [first, last] codepoint ranges; the advances below follow them in order.
+ ranges: [
+ [32, 126], [160, 383],
+ ] as ReadonlyArray<readonly [number, number]>,
+ advances: {
+ 720: [
+ 279, 329, 473, 708, 653, 1074, 911, 253, 362, 362, 424, 704, 325, 393, 325, 305, 724, 663,
+ 722, 727, 715, 726, 728, 679, 737, 728, 330, 333, 704, 704, 704, 674, 1188, 867, 855, 886,
+ 880, 814, 747, 955, 917, 345, 687, 875, 697, 1061, 916, 954, 810, 954, 873, 810, 780, 900,
+ 833, 1137, 867, 839, 794, 346, 305, 346, 704, 608, 280, 711, 710, 702, 710, 715, 421, 710,
+ 700, 290, 288, 673, 290, 1078, 700, 723, 710, 710, 439, 653, 441, 700, 645, 966, 685, 645,
+ 612, 361, 283, 361, 704, 279, 329, 712, 721, 617, 711, 281, 719, 385, 811, 479, 603, 704, 393,
+ 811, 365, 400, 704, 430, 430, 280, 703, 686, 394, 266, 430, 466, 603, 1024, 1023, 1024, 674,
+ 867, 867, 867, 867, 867, 867, 1199, 886, 814, 814, 814, 814, 345, 345, 345, 345, 880, 916,
+ 954, 954, 954, 954, 954, 704, 954, 900, 900, 900, 900, 839, 824, 731, 711, 711, 711, 711, 711,
+ 711, 1140, 702, 715, 715, 715, 715, 290, 290, 290, 290, 726, 700, 723, 723, 723, 723, 723,
+ 704, 723, 700, 700, 700, 700, 645, 710, 645, 867, 711, 867, 711, 867, 711, 886, 702, 886, 702,
+ 886, 702, 886, 702, 880, 710, 880, 710, 814, 715, 814, 715, 814, 715, 814, 715, 814, 715, 955,
+ 710, 955, 710, 955, 710, 955, 710, 917, 700, 917, 700, 345, 290, 345, 290, 345, 290, 345, 290,
+ 345, 290, 1032, 579, 687, 288, 875, 673, 673, 697, 290, 697, 290, 697, 290, 697, 290, 697,
+ 290, 916, 700, 916, 700, 916, 700, 700, 916, 700, 954, 723, 954, 723, 954, 723, 1448, 1184,
+ 873, 439, 873, 439, 873, 439, 810, 653, 810, 653, 810, 653, 810, 653, 780, 441, 780, 441, 780,
+ 441, 900, 700, 900, 700, 900, 700, 900, 700, 900, 700, 900, 700, 1137, 966, 839, 645, 839,
+ 794, 612, 794, 612, 794, 612, 421,
+ ],
+ 380: [
+ 300, 278, 353, 683, 592, 1025, 809, 183, 336, 336, 424, 682, 277, 393, 277, 301, 692, 605,
+ 680, 690, 655, 688, 691, 628, 698, 691, 281, 281, 682, 682, 682, 629, 1207, 827, 827, 889,
+ 879, 816, 733, 954, 885, 306, 636, 808, 639, 1014, 886, 958, 792, 958, 852, 798, 733, 869,
+ 777, 1087, 840, 775, 763, 293, 301, 293, 682, 568, 222, 661, 660, 639, 660, 665, 352, 657,
+ 647, 241, 239, 606, 241, 1023, 647, 673, 660, 660, 380, 592, 372, 646, 576, 831, 601, 576,
+ 574, 318, 280, 318, 682, 300, 278, 660, 685, 578, 655, 280, 674, 325, 824, 455, 572, 682, 393,
+ 824, 355, 400, 682, 390, 390, 222, 652, 651, 394, 235, 390, 440, 572, 989, 988, 989, 629, 827,
+ 827, 827, 827, 827, 827, 1215, 889, 816, 816, 816, 816, 306, 306, 306, 306, 879, 886, 958,
+ 958, 958, 958, 958, 682, 958, 869, 869, 869, 869, 775, 798, 708, 661, 661, 661, 661, 661, 661,
+ 1096, 639, 665, 665, 665, 665, 241, 241, 241, 241, 672, 647, 673, 673, 673, 673, 673, 682,
+ 673, 646, 646, 646, 646, 576, 658, 576, 827, 661, 827, 661, 827, 661, 889, 639, 889, 639, 889,
+ 639, 889, 639, 879, 660, 879, 660, 816, 665, 816, 665, 816, 665, 816, 665, 816, 665, 954, 657,
+ 954, 657, 954, 657, 954, 657, 885, 647, 885, 647, 306, 241, 306, 241, 306, 241, 306, 241, 306,
+ 241, 942, 480, 636, 239, 808, 606, 606, 639, 241, 639, 241, 639, 241, 622, 241, 639, 241, 886,
+ 647, 886, 647, 886, 647, 647, 886, 646, 958, 673, 958, 673, 958, 673, 1496, 1134, 852, 380,
+ 852, 380, 852, 380, 798, 592, 798, 592, 798, 592, 798, 592, 733, 372, 733, 372, 733, 372, 869,
+ 646, 869, 646, 869, 646, 869, 646, 869, 646, 869, 646, 1087, 831, 775, 576, 775, 763, 574,
+ 763, 574, 763, 574, 352,
+ ],
+ } as Readonly<Record<720 | 380, readonly number[]>>,
+};
diff --git a/common/lib/wordmarkWidth.test.ts b/common/lib/wordmarkWidth.test.ts
@@ -0,0 +1,46 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { createHash } from "node:crypto";
+import fs from "node:fs";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { WORDMARK_METRICS } from "./wordmarkMetrics";
+import { wordmarkWidthEm } from "./wordmarkWidth";
+
+// Run with: pnpm --filter yt-dlp-transcript-common test
+
+const FONT = path.resolve(
+ path.dirname(fileURLToPath(import.meta.url)),
+ "..",
+ "..",
+ "umtool",
+ "report-to-video",
+ "fonts",
+ WORDMARK_METRICS.font,
+);
+
+test("the table was generated from the vendored Archivo", () => {
+ const sha = createHash("sha256").update(fs.readFileSync(FONT)).digest("hex");
+ assert.equal(WORDMARK_METRICS.sha256, sha, "re-run common/bin/gen-wordmark-metrics.py");
+ const size = WORDMARK_METRICS.ranges.reduce((n, [a, b]) => n + b - a + 1, 0);
+ assert.equal(WORDMARK_METRICS.advances[720].length, size);
+ assert.equal(WORDMARK_METRICS.advances[380].length, size);
+});
+
+test("the lead is set heavier, so wider, than the same letters as suffix", () => {
+ const heavy = wordmarkWidthEm("Wwww", "Wwww");
+ const light = wordmarkWidthEm("xWwww", "x") - wordmarkWidthEm("x", "x");
+ assert.ok(heavy > light, `${heavy} vs ${light}`);
+});
+
+test("longer titles are wider; tracking narrows by its value per character", () => {
+ assert.ok(wordmarkWidthEm("Rekietalyzer", "Rekieta") > wordmarkWidthEm("Jeralyzer", "Jer"));
+ const plain = wordmarkWidthEm("Archilyzer", "Archi");
+ const tracked = wordmarkWidthEm("Archilyzer", "Archi", -0.01);
+ assert.ok(Math.abs(plain - tracked - 0.1 * 1.03) < 0.002, `${plain} → ${tracked}`);
+});
+
+test("a character the table does not hold counts wide, not zero", () => {
+ const latin = wordmarkWidthEm("Ab", "Ab");
+ assert.ok(wordmarkWidthEm("Ab中", "Ab") - latin > 1.2);
+});
diff --git a/common/lib/wordmarkWidth.ts b/common/lib/wordmarkWidth.ts
@@ -0,0 +1,44 @@
+import { splitWordmark } from "./brand";
+import { WORDMARK_METRICS } from "./wordmarkMetrics";
+
+// A wordmark's width as the DISPLAY FACE sets it, in em of its font size —
+// so a layout can reserve it before the web font has loaded. The export
+// header's wordmark drops its text when the text does not fit beside the
+// icons (export/app/components/Header.tsx); measured in the fallback face,
+// which is narrower, a title could fit, show, and then vanish when Archivo
+// arrived. Reserving this width makes the decision the same either way.
+//
+// The lead is weight 720 and the suffix 380 (common/components/Wordmark.tsx),
+// both at wdth 118; each character is its advance from the vendored font
+// (lib/wordmarkMetrics.ts, generated) plus `letterSpacingEm`, kerning left
+// out, and MARGIN more: the web font Google serves renders 0.2–2.4 % wider
+// than the vendored file's advances sum to (measured in Chromium). The header
+// sets this as the wordmark's MIN-width, not its width: where a platform
+// renders the text wider still, the box grows with it (and the text drops a
+// little sooner) rather than clipping its last letter. A character the table
+// does not hold counts as UNMAPPED_EM, wider than any it does.
+
+const UNMAPPED_EM = 1.25;
+const MARGIN = 1.03;
+
+function advanceEm(cp: number, wght: 720 | 380): number {
+ const { ranges, advances, unitsPerEm } = WORDMARK_METRICS;
+ let offset = 0;
+ for (const [first, last] of ranges) {
+ if (cp >= first && cp <= last) return advances[wght][offset + cp - first] / unitsPerEm;
+ offset += last - first + 1;
+ }
+ return UNMAPPED_EM;
+}
+
+export function wordmarkWidthEm(title: string, lead: string | undefined, letterSpacingEm = 0): number {
+ const parts = splitWordmark(title, lead);
+ let em = 0;
+ for (const [text, wght] of [
+ [parts.lead, 720],
+ [parts.suffix, 380],
+ ] as const) {
+ for (const ch of text) em += advanceEm(ch.codePointAt(0)!, wght) + letterSpacingEm;
+ }
+ return Math.round(em * MARGIN * 1000) / 1000;
+}
diff --git a/common/publish/__fixtures__/fakeStagit.ts b/common/publish/__fixtures__/fakeStagit.ts
@@ -0,0 +1,82 @@
+import { chmodSync, mkdirSync, writeFileSync } from "node:fs";
+import path from "node:path";
+
+// A stand-in for stagit, for the source step's tests (sourceHistory.test.ts,
+// source.test.ts; the real one is tested where it is installed). It writes
+// what stagit writes, where stagit writes it (its CWD), the way stagit.c does:
+// - `-c <file>`: walks from HEAD down to the commit the file names, writes a
+// log line and a page for each NEW commit, appends the file's old lines,
+// and rewrites the file (the head, then every line);
+// - `-l <n>`: a log line for the newest n and "<m> more commits remaining,
+// fetch the repository"; a page for EVERY commit that has none (-l does
+// not cap the pages);
+// - neither: a line and a page for every commit;
+// - `-c` with `-l`: stagit's usage error;
+// - and every run: files.html, refs.html, the two feeds (atom.xml carries
+// the -u base), a per-file page, a leftover in the CWD. A commit page is
+// never rewritten once it exists.
+// Each call is appended to `log` as `cache=… limit=… base=… repo=<dir name>`.
+// `exit` fails at once (with a stderr line); `extra` goes into every commit
+// page; `pad` bytes are added to the head's page.
+export function writeFakeStagit(
+ file: string,
+ log: string,
+ o: { exit?: number; extra?: string; pad?: number } = {},
+): string {
+ mkdirSync(path.dirname(file), { recursive: true });
+ writeFileSync(
+ file,
+ `#!/bin/sh
+cache=""; base=""; repo=""; limit=""
+while [ $# -gt 0 ]; do
+ case "$1" in
+ -c) cache="$2"; shift 2;;
+ -l) limit="$2"; shift 2;;
+ -u) base="$2"; shift 2;;
+ *) repo="$1"; shift;;
+ esac
+done
+echo "cache=$cache limit=$limit base=$base repo=$(basename "$repo")" >> '${log}'
+${o.exit ? `echo 'stagit: something broke' >&2; exit ${o.exit}` : ""}
+if [ -n "$cache" ] && [ -n "$limit" ]; then echo 'usage: stagit [-c cachefile | -l commits] [-u baseurl] repodir' >&2; exit 1; fi
+mkdir -p commit file
+tip=$(git --git-dir "$repo" rev-parse HEAD)
+last=""
+if [ -n "$cache" ] && [ -f "$cache" ]; then last=$(head -n 1 "$cache"); fi
+: > lines.tmp
+n=0; rem=0
+for c in $(git --git-dir "$repo" rev-list HEAD); do
+ [ "$c" = "$last" ] && break
+ if [ -n "$limit" ] && [ "$n" -ge "$limit" ]; then
+ rem=$((rem + 1))
+ [ -f "commit/$c.html" ] && continue
+ else
+ n=$((n + 1))
+ echo "<tr><td><a href=\\"commit/$c.html\\">$c</a></td></tr>" >> lines.tmp
+ fi
+ [ -f "commit/$c.html" ] || printf '<html>\\n<head>\\n</head>\\n<body>\\n<a href="../file/README.md.html">README</a> ${o.extra ?? ""}\\n</body>\\n</html>\\n' > "commit/$c.html"
+done
+if [ -n "$cache" ]; then
+ if [ -n "$last" ]; then tail -n +2 "$cache" >> lines.tmp; fi
+ { echo "$tip"; cat lines.tmp; } > "$cache"
+fi
+if [ "$rem" -gt 0 ]; then echo "<tr><td></td><td colspan=\\"5\\">$rem more commits remaining, fetch the repository</td></tr>" >> lines.tmp; fi
+{
+ printf '<html>\\n<head>\\n</head>\\n<body>\\n<span class="desc">%s</span> %s <a href="file/README.md.html">README</a> <img src="logo.png" />\\n<table id="log">\\n' "$(cat "$repo/description")" "$(cat "$repo/url")"
+ cat lines.tmp
+ printf '</table>\\n</body>\\n</html>\\n'
+} > log.html
+rm -f lines.tmp
+printf '<html>\\n<head>\\n</head>\\n<body>\\n<a href="file/app/%%5Bslug%%5D/page.tsx.html">x</a>\\n</body>\\n</html>\\n' > files.html
+printf '<html>\\n<head>\\n</head>\\n<body>\\n</body>\\n</html>\\n' > refs.html
+printf '<feed>%s file/README.md.html</feed>\\n' "$base" > atom.xml
+printf '<feed/>\\n' > tags.xml
+printf 'x\\n' > file/README.md.html
+touch cache.XXXXleftover
+${o.pad ? `head -c ${o.pad} /dev/zero >> "commit/$tip.html"` : ""}
+exit 0
+`,
+ );
+ chmodSync(file, 0o755);
+ return file;
+}
diff --git a/common/publish/build.test.ts b/common/publish/build.test.ts
@@ -1,8 +1,12 @@
import { test } from "node:test";
import assert from "node:assert/strict";
+import { chmodSync, existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
import type { Paths } from "../lib/paths";
import {
HOMEPAGE_PAGES_PROJECT,
+ buildHomepage,
buildHubSteps,
buildSiteSteps,
hubProjectProblem,
@@ -143,6 +147,119 @@ test("homepageOutDir is homepage/out of the checkout", () => {
assert.equal(homepageOutDir({ monorepoRoot: "/repo" } as Paths), "/repo/homepage/out");
});
+// buildHomepage's order — compose, the source step, `next build` — over a temp
+// homepage/ whose `compose` is `true` and whose `next` only says it ran.
+test("buildHomepage: the source step sits between compose and next build; a refusal stops the build; --no-source clears instead", async () => {
+ const root = mkdtempSync(path.join(os.tmpdir(), "build-homepage-"));
+ try {
+ const home = path.join(root, "homepage");
+ mkdirSync(path.join(home, "node_modules", ".bin"), { recursive: true });
+ writeFileSync(
+ path.join(home, "package.json"),
+ JSON.stringify({ name: "fake-homepage", private: true, scripts: { compose: "echo COMPOSED" } }),
+ );
+ const next = path.join(home, "node_modules", ".bin", "next");
+ writeFileSync(next, "#!/bin/sh\necho NEXT RAN\n");
+ chmodSync(next, 0o755);
+ const fakePaths = { monorepoRoot: root } as Paths;
+ // The last good build's source, in out/ (M1: a refusal must take it out).
+ const out = path.join(home, "out");
+ const plantOut = () => {
+ mkdirSync(path.join(out, "source", "archilyzer.git"), { recursive: true });
+ mkdirSync(path.join(out, "downloads"), { recursive: true });
+ writeFileSync(path.join(out, "source", "manifest.json"), "{}");
+ writeFileSync(path.join(out, "downloads", "archilyzer-source.tar.gz"), "old");
+ writeFileSync(path.join(out, "downloads", "snapshot.json"), "{}");
+ writeFileSync(path.join(out, "downloads", "index.html"), "<p>the page</p>");
+ };
+
+ const run = async (o: Parameters<typeof buildHomepage>[0]) => {
+ const logs: string[] = [];
+ const code = await buildHomepage({ paths: fakePaths, onLog: (l) => logs.push(l), ...o });
+ return { code, logs: logs.join("\n") };
+ };
+ const calls: string[] = [];
+
+ // A refusal (1) fails the build before next build runs, and takes the
+ // last build's source out of homepage/out, so a deploy-only ships none.
+ plantOut();
+ let r = await run({
+ publishSource: async (p) => {
+ calls.push(`publish:${p.paths === fakePaths}:${p.noRepository}`);
+ return 1;
+ },
+ });
+ assert.equal(r.code, 1);
+ assert.match(r.logs, /COMPOSED/);
+ assert.doesNotMatch(r.logs, /NEXT RAN/);
+ assert.deepEqual(calls, ["publish:true:empty"], "a checkout with no git builds with the empty state");
+ assert.ok(!existsSync(path.join(out, "source")), "out/source is withdrawn");
+ assert.ok(!existsSync(path.join(out, "downloads", "archilyzer-source.tar.gz")));
+ assert.ok(!existsSync(path.join(out, "downloads", "snapshot.json")));
+ assert.ok(existsSync(path.join(out, "downloads", "index.html")), "the page itself stays");
+ assert.match(r.logs, /removed from homepage\/out too/);
+
+ // Published: next build follows.
+ r = await run({ publishSource: async () => (calls.push("publish"), 0) });
+ assert.equal(r.code, 0, r.logs);
+ assert.ok(r.logs.indexOf("COMPOSED") < r.logs.indexOf("NEXT RAN"));
+
+ // --no-source: the publish is never called; the clear is.
+ calls.length = 0;
+ r = await run({
+ skipSource: true,
+ publishSource: async () => {
+ throw new Error("--no-source must not publish");
+ },
+ clearSource: async () => {
+ calls.push("clear");
+ },
+ });
+ assert.equal(r.code, 0, r.logs);
+ assert.deepEqual(calls, ["clear"]);
+ assert.match(r.logs, /NEXT RAN/);
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+// M1: a deploy-only ships homepage/out as the last build left it. Its source
+// ships only with a record that it was audited under today's rules (the
+// positive case is source.test.ts' round trip, over publishedSourceProblem).
+test("deployHomepage refuses an out/ whose source has no record of the rules it was audited under", async () => {
+ const root = mkdtempSync(path.join(os.tmpdir(), "deploy-homepage-"));
+ try {
+ const out = path.join(root, "homepage", "out");
+ mkdirSync(path.join(out, "source"), { recursive: true });
+ writeFileSync(path.join(out, "index.html"), "<p>home</p>");
+ writeFileSync(path.join(out, "source", "index.html"), "<p>the /source page</p>");
+ writeFileSync(
+ path.join(out, "source", "manifest.json"),
+ JSON.stringify({
+ version: 1, generatedAt: "2026-09-28T12:00:00.000Z", branch: "main", sourceCommit: "1".repeat(40),
+ mirrorHead: "2".repeat(40), subject: "s", files: 1, bytes: 1, mirror: { files: 1, bytes: 1, packs: 1 },
+ tree: { files: 1, dirs: 1, bytes: 1 }, tarball: { href: "/downloads/archilyzer-source.tar.gz", bytes: 1, sha256: "3".repeat(64) },
+ audit: { objects: 1, commits: 1, gitleaks: "clean" }, tools: {},
+ }),
+ );
+ const p = { monorepoRoot: root } as Paths;
+ await assert.rejects(
+ deployHomepage({ paths: p, previewBranch: "r12-source" }),
+ /homepage\/out's source has no record of the rules it was audited under — run `archilyzer build homepage`/,
+ );
+ // A half-removed source (no manifest) refuses too…
+ rmSync(path.join(out, "source", "manifest.json"));
+ mkdirSync(path.join(out, "source", "archilyzer.git"));
+ await assert.rejects(deployHomepage({ paths: p }), /without a valid manifest — run `archilyzer build homepage`/);
+ // …and so does a build whose source step refused (buildHomepage took
+ // out/source away, the page with it).
+ rmSync(path.join(out, "source"), { recursive: true });
+ await assert.rejects(deployHomepage({ paths: p }), /has no \/source page \(its source step refused/);
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
test("deployHomepage refuses a bad preview branch before it looks for a build", async () => {
const noBuild = { monorepoRoot: "/nonexistent-repo" } as Paths;
await assert.rejects(
diff --git a/common/publish/build.ts b/common/publish/build.ts
@@ -969,11 +969,48 @@ export async function composeHomepage(opts: PublishOpts = {}): Promise<number> {
]);
}
-/** composeHomepage, then `next build` in homepage/ (→ homepage/out). */
-export async function buildHomepage(opts: PublishOpts = {}): Promise<number> {
+/**
+ * composeHomepage, then `archilyzer source publish` (the git mirror, the raw
+ * tree and the tarball into homepage/public — source.ts), then `next build` in
+ * homepage/ (→ homepage/out).
+ *
+ * A source REFUSAL fails the build before `next build`, and withdraws the
+ * source twice over: the step itself removes the last publish from
+ * homepage/public, and this removes the last BUILD's copy from homepage/out
+ * (`out/source`, the tarball, `snapshot.json`), so a deploy-only cannot ship
+ * a source today's rules were never applied to (deployHomepage checks too).
+ * The audit report is in the log.
+ *
+ * `skipSource` (the CLI's `--no-source`) REMOVES the published source instead:
+ * a copy left from an earlier publish was audited against the rules of ITS
+ * day, so a build that skips the gate ships none — the pages show their empty
+ * states. A checkout with no git repository (the docker runtime, a tarball
+ * install) builds with the empty state too. `publishSource` / `clearSource`
+ * are the test's seams.
+ */
+export async function buildHomepage(
+ opts: PublishOpts & {
+ skipSource?: boolean;
+ publishSource?: (o: PublishOpts & { noRepository?: "refuse" | "empty" }) => Promise<number>;
+ clearSource?: (o: PublishOpts) => Promise<void>;
+ } = {},
+): Promise<number> {
const { paths, onLog, signal } = resolved(opts);
const code = await composeHomepage({ paths, onLog, signal });
if (code !== 0) return code;
+ // Loaded lazily: the source step pulls nothing the other publish entry
+ // points need, and it imports this module's types.
+ if (opts.skipSource) {
+ const clear = opts.clearSource ?? (await import("./source")).clearPublishedSource;
+ await clear({ paths, onLog, signal });
+ } else {
+ const publish = opts.publishSource ?? (await import("./source")).publishSource;
+ const sourceCode = await publish({ paths, onLog, signal, noRepository: "empty" });
+ if (sourceCode !== 0 || signal.aborted) {
+ await withdrawBuiltSource(paths, onLog);
+ return sourceCode || 1;
+ }
+ }
return runSteps(onLog, signal, [
{
command: "pnpm",
@@ -984,6 +1021,22 @@ export async function buildHomepage(opts: PublishOpts = {}): Promise<number> {
]);
}
+// The last build's copy of the source, out of homepage/out: a refused source
+// step must not leave it for a deploy-only to ship.
+async function withdrawBuiltSource(paths: Paths, onLog: (line: string) => void): Promise<void> {
+ const out = homepageOutDir(paths);
+ const had =
+ existsSync(path.join(out, "source", "manifest.json")) ||
+ existsSync(path.join(out, "downloads", "archilyzer-source.tar.gz"));
+ await rm(path.join(out, "source", "manifest.json"), { force: true });
+ await rm(path.join(out, "source"), { recursive: true, force: true });
+ await rm(path.join(out, "downloads", "archilyzer-source.tar.gz"), { force: true });
+ await rm(path.join(out, "downloads", "snapshot.json"), { force: true });
+ if (had) {
+ onLog("[source] the last build's source was removed from homepage/out too, so a deploy-only ships none.\n");
+ }
+}
+
/**
* The wrangler argv (after `pnpm dlx`) for a homepage deploy of `outDir`: the
* production branch `main` — what homepage/package.json's hardcoded `deploy`
@@ -1021,6 +1074,10 @@ export async function deployHomepage(
if (!existsSync(path.join(outDir, "index.html"))) {
throw new Error("homepage/out holds no build — run archilyzer build homepage first");
}
+ // The source in out/ ships only if it was audited under TODAY's rules, of
+ // today's main (source.ts). An out/ with no source deploys as before.
+ const sourceProblem = await (await import("./source")).publishedSourceProblem(paths, outDir);
+ if (sourceProblem) throw new Error(sourceProblem);
if (branch) onLog(`=== Deploy homepage (preview "${branch}") ===\n`);
const code = await runPagesDeployIntoLog(onLog, signal, {
outDir,
diff --git a/common/publish/source.test.ts b/common/publish/source.test.ts
@@ -0,0 +1,949 @@
+import { test, after, before } from "node:test";
+import assert from "node:assert/strict";
+import { createHash } from "node:crypto";
+import { execFileSync } from "node:child_process";
+import {
+ chmodSync,
+ cpSync,
+ existsSync,
+ lstatSync,
+ mkdirSync,
+ mkdtempSync,
+ readdirSync,
+ readFileSync,
+ rmSync,
+ statSync,
+ symlinkSync,
+ writeFileSync,
+} from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import type { Paths } from "../lib/paths";
+import { CLONE_URL, MIRROR_DIR, TARBALL_HREF, TREE_HREF } from "../lib/sourceManifest";
+import {
+ MAX_FILES,
+ MAX_FILE_BYTES,
+ NO_REPOSITORY,
+ SOURCE_STEP_VERSION,
+ SourceRefusal,
+ clearPublishedSource,
+ limitProblem,
+ loadSourceRules,
+ parseScrubRules,
+ publishSource,
+ publishedSourceProblem,
+ gitleaksIdentity,
+ historyCacheFor,
+ historyDigest,
+ resolveFilterRepo,
+ rulesHashOf,
+ scratchRootProblem,
+ sourceDigest,
+ type SourcePublishOpts,
+} from "./source";
+import { writeFakeStagit } from "./__fixtures__/fakeStagit";
+import { HISTORY_BACK_LINK } from "./sourceHistory";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// `archilyzer source publish` (source.ts) over temp repos. The two tests that
+// rewrite history need git-filter-repo; they SKIP when resolveFilterRepo()
+// cannot find one (say so in a gate: the release gate requires they RAN).
+// Every repo, public dir and scratch dir is under the OS temp dir, and the git
+// variables a hook or a wrapper might export are cleared first.
+for (const key of ["GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE", "GIT_PREFIX"]) {
+ delete process.env[key];
+}
+
+const TMP = mkdtempSync(path.join(os.tmpdir(), "source-publish-"));
+after(() => rmSync(TMP, { recursive: true, force: true }));
+
+// Planted, never real: the home dir the built-in rule scrubs, and the paths.
+const HOME = "/home/not-a-real-user";
+const PLANTED = "plantedhome";
+const SECRET = "plantedsecret";
+
+let filterRepoProblem: string | null = null;
+before(async () => {
+ try {
+ await resolveFilterRepo({});
+ } catch (err) {
+ filterRepoProblem = (err as Error).message;
+ }
+});
+
+let n = 0;
+function dir(name: string): string {
+ const d = path.join(TMP, `${name}-${n++}`);
+ mkdirSync(d, { recursive: true });
+ return d;
+}
+
+function gitIn(cwd: string, ...args: string[]): string {
+ return execFileSync("git", args, { cwd, stdio: "pipe" }).toString().trim();
+}
+
+// A repo with `main` of three commits: a planted path in one blob and one
+// message, and the names the tree pages must encode.
+function sourceRepo(extra: Record<string, string> = {}): string {
+ const repo = dir("src");
+ gitIn(repo, "init", "-q", "-b", "main");
+ gitIn(repo, "config", "user.name", "source test");
+ gitIn(repo, "config", "user.email", "source@example.invalid");
+ gitIn(repo, "config", "commit.gpgsign", "false");
+ const put = (files: Record<string, string>, message: string) => {
+ for (const [f, text] of Object.entries(files)) {
+ mkdirSync(path.dirname(path.join(repo, f)), { recursive: true });
+ writeFileSync(path.join(repo, f), text);
+ }
+ gitIn(repo, "add", "-A");
+ gitIn(repo, "commit", "-q", "-m", message);
+ };
+ put(
+ {
+ "README.md": "hello\n",
+ "app/[slug]/page.tsx": "export default 1;\n",
+ "fonts/Archivo[wdth,wght].ttf": "not really a font\n",
+ },
+ "first",
+ );
+ put({ "notes.txt": `data lives at /srv/${PLANTED}/x\n`, ...extra }, "add notes");
+ put({ "README.md": "hello again\n" }, `moved from /srv/${PLANTED}`);
+ return repo;
+}
+
+function operatorFiles(scrub: string, deny: string): { scrubFile: string; denylistFile: string } {
+ const d = dir("config");
+ writeFileSync(path.join(d, "source-scrub.txt"), scrub);
+ writeFileSync(path.join(d, "source-denylist.txt"), deny);
+ return { scrubFile: path.join(d, "source-scrub.txt"), denylistFile: path.join(d, "source-denylist.txt") };
+}
+
+// A checkout of its own (never TMP itself, which holds the scratch root: the
+// step refuses a scratch dir inside the checkout).
+function opts(repo: string, files: { scrubFile: string; denylistFile: string }, logs: string[], extra: Partial<SourcePublishOpts> = {}): SourcePublishOpts {
+ return {
+ paths: { monorepoRoot: dir("checkout") } as Paths,
+ sourceRepo: path.join(repo, ".git"),
+ publicDir: path.join(dir("site"), "public"),
+ scratchRoot: path.join(TMP, "scratch"),
+ gitleaks: null,
+ // No history pages unless a test gives a stagit (the machine may have one).
+ stagit: null,
+ homeDir: HOME,
+ onLog: (l) => logs.push(l),
+ now: () => new Date("2026-09-28T12:00:00.000Z"),
+ ...files,
+ ...extra,
+ };
+}
+
+const sha256 = (file: string) => createHash("sha256").update(readFileSync(file)).digest("hex");
+
+// A manifest the page would believe, for the tests that fake a publish.
+function fakeManifest(sourceCommit: string, mirrorHead: string, sha = "b".repeat(64)): string {
+ return JSON.stringify({
+ version: 1, generatedAt: "2026-09-28T12:00:00.000Z", branch: "main", sourceCommit, mirrorHead, subject: "s",
+ files: 1, bytes: 1, mirror: { files: 1, bytes: 1, packs: 1 }, tree: { files: 1, dirs: 1, bytes: 1 },
+ tarball: { href: TARBALL_HREF, bytes: 7, sha256: sha }, cloneUrl: CLONE_URL, treeHref: TREE_HREF,
+ audit: { objects: 1, commits: 1, gitleaks: "skipped" }, tools: { git: "x", filterRepo: "x" },
+ });
+}
+
+test("scrub rules: the home-dir rule first, comments and blanks dropped, every literal left side denied", () => {
+ const rules = parseScrubRules(
+ ["# a comment", "", `/srv/${PLANTED}==>/home/user`, " # indented comment", "literal:abc==>x", "regex:a+b==>c", "glob:*.x==>y", "bare-line", "a==>b==>c", "\r"].join("\n"),
+ HOME,
+ );
+ assert.deepEqual(rules.lines, [
+ `${HOME}==>/home/user`,
+ `/srv/${PLANTED}==>/home/user`,
+ "literal:abc==>x",
+ "regex:a+b==>c",
+ "glob:*.x==>y",
+ "bare-line",
+ "a==>b==>c",
+ ]);
+ // filter-repo splits at the LAST ==>; regex and glob rules deny nothing.
+ // Each denial is named by where it was written (the report's label).
+ assert.deepEqual(rules.denied, [
+ { text: HOME, from: "built-in home rule" },
+ { text: `/srv/${PLANTED}`, from: "scrub line 3 lhs" },
+ { text: "abc", from: "scrub line 5 lhs" },
+ { text: "bare-line", from: "scrub line 8 lhs" },
+ { text: "a==>b", from: "scrub line 9 lhs" },
+ ]);
+ // A home dir of `/`, or one that IS the replacement, gets no built-in rule;
+ // a trailing slash is dropped, or the rule would match nothing.
+ assert.deepEqual(parseScrubRules("", "/").lines, []);
+ assert.deepEqual(parseScrubRules("", "/home/user").lines, []);
+ assert.deepEqual(parseScrubRules("", `${HOME}/`).lines, [`${HOME}==>/home/user`]);
+});
+
+test("scrub rules: a byte-order mark and CRLF are dropped; an empty left side refuses (review L2)", () => {
+ // The reviewer's reproduction: a BOM made the first rule — and its implied
+ // denial — `\uFEFF/srv/…`, which matches nothing.
+ const bom = parseScrubRules(`\uFEFF/srv/${PLANTED}==>/home/user\r\nx==>y\r\n`, HOME);
+ assert.deepEqual(bom.lines, [`${HOME}==>/home/user`, `/srv/${PLANTED}==>/home/user`, "x==>y"]);
+ assert.deepEqual(bom.denied.map((d) => d.text), [HOME, `/srv/${PLANTED}`, "x"]);
+ for (const line of ["==>user", "literal:==>user", "regex:==>user", "glob:"]) {
+ assert.throws(
+ () => parseScrubRules(`ok==>fine\n${line}\n`, HOME),
+ (e) => e instanceof SourceRefusal && /scrub line 2 has an empty left side/.test(e.message),
+ line,
+ );
+ }
+});
+
+test("the operator's files: a missing one refuses by name; the denylist's i: and the implied left sides; the hash moves with them", async () => {
+ const d = dir("cfg");
+ await assert.rejects(
+ loadSourceRules({ scrubFile: path.join(d, "nope.txt"), denylistFile: path.join(d, "deny.txt"), homeDir: HOME }),
+ (e) => e instanceof SourceRefusal && /no scrub rules at .*nope\.txt — create it/.test(e.message),
+ );
+ writeFileSync(path.join(d, "scrub.txt"), `/srv/${PLANTED}==>/home/user\n`);
+ await assert.rejects(
+ loadSourceRules({ scrubFile: path.join(d, "scrub.txt"), denylistFile: path.join(d, "deny.txt"), homeDir: HOME }),
+ (e) => e instanceof SourceRefusal && /no denylist at .*deny\.txt/.test(e.message),
+ );
+ writeFileSync(path.join(d, "deny.txt"), `\uFEFF# mine\r\ni:${SECRET.toUpperCase()}\r\n`);
+ const a = await loadSourceRules({ scrubFile: path.join(d, "scrub.txt"), denylistFile: path.join(d, "deny.txt"), homeDir: HOME });
+ assert.deepEqual(
+ a.literals.map((l) => [l.bytes.toString(), l.ci, l.from]),
+ [[SECRET, true, "denylist line 2"], [HOME, false, "built-in home rule"], [`/srv/${PLANTED}`, false, "scrub line 1 lhs"]],
+ );
+ writeFileSync(path.join(d, "deny.txt"), `i:${SECRET}\nanother\n`);
+ const b = await loadSourceRules({ scrubFile: path.join(d, "scrub.txt"), denylistFile: path.join(d, "deny.txt"), homeDir: HOME });
+ assert.notEqual(a.rulesHash, b.rulesHash, "a new literal must defeat the skip");
+});
+
+test("the limits: 15,000 files and 24 MiB a file, inside Pages' 20,000 and 25 MiB", () => {
+ assert.equal(MAX_FILES, 15_000);
+ assert.equal(MAX_FILE_BYTES, 24 * 1024 * 1024);
+ assert.equal(limitProblem([{ rel: "a", bytes: MAX_FILE_BYTES }]), null);
+ assert.match(limitProblem([{ rel: "big.pack", bytes: MAX_FILE_BYTES + 1 }])!, /^big\.pack is 24\.0 MiB, over the step's limit of 24\.0 MiB/);
+ const many = Array.from({ length: MAX_FILES + 1 }, (_, i) => ({ rel: `f${i}`, bytes: 1 }));
+ assert.match(limitProblem(many)!, /^15001 files to publish, over the step's limit of 15000/);
+ assert.equal(limitProblem(many.slice(1)), null);
+});
+
+test("skip: an unchanged main with unchanged rules, tools and files does nothing; each part of the key defeats it; a refusal withdraws", async () => {
+ const repo = sourceRepo();
+ const files = operatorFiles(`/srv/${PLANTED}==>/home/user\n`, "");
+ const logs: string[] = [];
+ // filter-repo is `false`: reaching it would refuse, so a 0 proves the skip
+ // and a 1 proves the key did NOT match (review R2-L3: every case below goes
+ // through the skip, never `check: true`, which skips nothing).
+ const o = opts(repo, files, logs, { filterRepo: ["false"] });
+ const pub = o.publicDir!;
+ const sourceCommit = gitIn(repo, "rev-parse", "main");
+ const rules = await loadSourceRules({ ...files, homeDir: HOME });
+ const mirrorHead = "a".repeat(40);
+ const state = path.join(path.dirname(pub), ".source-publish.json");
+ const plant = async (tweak: Record<string, string> = {}) => {
+ mkdirSync(path.join(pub, "source", MIRROR_DIR, "info"), { recursive: true });
+ mkdirSync(path.join(pub, "downloads"), { recursive: true });
+ writeFileSync(path.join(pub, "source", MIRROR_DIR, "info", "refs"), "x\trefs/heads/main\n");
+ writeFileSync(path.join(pub, "downloads", path.basename(TARBALL_HREF)), "tarball");
+ writeFileSync(path.join(pub, "downloads", "snapshot.json"), "{}");
+ writeFileSync(path.join(pub, "source", "manifest.json"), fakeManifest(sourceCommit, mirrorHead));
+ const key = {
+ sourceCommit,
+ mirrorHead,
+ rulesHash: rules.rulesHash,
+ filterRepo: "false (given)",
+ gitleaks: "skipped",
+ contentDigest: await sourceDigest(pub),
+ stagit: "absent",
+ history: null,
+ };
+ writeFileSync(state, JSON.stringify({ ...key, ...tweak }));
+ };
+ const run = async () => {
+ logs.length = 0;
+ return publishSource(o);
+ };
+
+ await plant();
+ assert.equal(await run(), 0);
+ assert.match(logs.join("\n"), new RegExp(`up to date at ${sourceCommit.slice(0, 12)}; skipping`));
+
+ // Each part of the key, changed alone, reaches the rewrite (and the refusal
+ // withdraws the planted publish, so it is planted again each time).
+ for (const [what, tweak] of [
+ ["another filter-repo", { filterRepo: "git filter-repo 0.1" }],
+ ["another gitleaks", { gitleaks: "gitleaks 8.0 (abc)" }],
+ ["other files", { contentDigest: "0".repeat(64) }],
+ ["other rules", { rulesHash: "0".repeat(64) }],
+ ["another stagit", { stagit: "stagit (sha256 0123456789ab)" }],
+ ] as Array<[string, Record<string, string>]>) {
+ await plant(tweak);
+ assert.equal(await run(), 1, `${what} must not skip`);
+ assert.match(logs.join("\n"), /REFUSED: false --force --quiet exited 1/, what);
+ }
+ // A published file edited in place defeats the skip too.
+ await plant();
+ writeFileSync(path.join(pub, "downloads", "snapshot.json"), '{"edited":true}');
+ assert.equal(await run(), 1, "an edited public/ file must not skip");
+
+ // A changed rule refuses AND withdraws the last publish.
+ await plant();
+ writeFileSync(files.denylistFile, "a-new-literal\n");
+ assert.equal(await run(), 1, "the new rule reached the rewrite");
+ assert.match(logs.join("\n"), /REFUSED: false --force --quiet exited 1/);
+ assert.match(logs.join("\n"), /the previous publish was WITHDRAWN/);
+ assert.ok(!existsSync(path.join(pub, "source")), "the last publish is withdrawn");
+ assert.ok(!existsSync(path.join(pub, "downloads", path.basename(TARBALL_HREF))));
+ assert.ok(!existsSync(path.join(pub, "downloads", "snapshot.json")));
+ assert.ok(!existsSync(state));
+ assert.equal(readdirSync(o.scratchRoot!).length, 0, "the scratch dir is removed");
+});
+
+test("the rules hash moves with the step's version (review R2-L3), and loadSourceRules uses the current one", async () => {
+ const lines = ["a==>b"];
+ const literals = [{ bytes: Buffer.from("a"), ci: false }];
+ assert.notEqual(rulesHashOf(lines, literals, 1), rulesHashOf(lines, literals, 2));
+ assert.notEqual(rulesHashOf(lines, literals, 1), rulesHashOf(lines, [{ bytes: Buffer.from("a"), ci: true }], 1));
+ const files = operatorFiles("a==>b\n", "");
+ const rules = await loadSourceRules({ ...files, homeDir: "/" });
+ assert.equal(rules.rulesHash, rulesHashOf(["a==>b"], rules.literals, SOURCE_STEP_VERSION));
+ assert.ok(SOURCE_STEP_VERSION >= 4, "release 15 slice SG: the history pages");
+});
+
+test("gitleaksIdentity: skipped, absent, or the version line with the binary's sha256 (review R2-L4)", async () => {
+ const bin = dir("gl-bin");
+ assert.equal(await gitleaksIdentity(null, { PATH: bin }), "skipped");
+ assert.equal(await gitleaksIdentity("gitleaks", { PATH: bin }), "absent");
+ const fake = path.join(bin, "gitleaks");
+ writeFileSync(fake, "#!/bin/sh\necho 'version is set by build process'\n");
+ chmodSync(fake, 0o755);
+ const a = await gitleaksIdentity("gitleaks", { PATH: bin });
+ assert.match(a, /^version is set by build process \([0-9a-f]{12}\)$/);
+ // Same version line, another binary (an upgrade on a distribution that
+ // prints a constant line): another identity.
+ writeFileSync(fake, "#!/bin/sh\necho 'version is set by build process'\n# 8.30\n");
+ assert.notEqual(await gitleaksIdentity("gitleaks", { PATH: bin }), a);
+});
+
+test("the scratch root: refused inside the checkout or the public dir, through a symlink too (review L12)", async () => {
+ const checkout = dir("chk");
+ const pub = path.join(dir("site"), "public");
+ const places: Array<[string, string]> = [["the checkout", checkout], ["the public dir", pub]];
+ assert.match((await scratchRootProblem(path.join(checkout, "tmp", "x"), places))!, /inside the checkout/);
+ assert.match((await scratchRootProblem(path.join(pub, "s"), places))!, /inside the public dir/);
+ const link = path.join(dir("links"), "into-checkout");
+ symlinkSync(checkout, link);
+ assert.match((await scratchRootProblem(path.join(link, "new"), places))!, /inside the checkout/);
+ assert.equal(await scratchRootProblem(path.join(TMP, "scratch"), places), null);
+ // …and publishSource says so before making anything.
+ const repo = sourceRepo();
+ const logs: string[] = [];
+ const o = opts(repo, operatorFiles("", ""), logs, { filterRepo: ["false"] });
+ assert.equal(await publishSource({ ...o, scratchRoot: path.join(o.publicDir!, "scratch") }), 1);
+ assert.match(logs.join("\n"), /REFUSED: the scratch root .* is inside the public dir/);
+ assert.ok(!existsSync(path.join(o.publicDir!, "scratch")));
+});
+
+test("no git repository here (a docker runtime, a tarball install): the build gets the empty state, the CLI a sentence (review L8)", async () => {
+ const bare = dir("no-git");
+ const pub = path.join(dir("site"), "public");
+ mkdirSync(path.join(pub, "source"), { recursive: true });
+ writeFileSync(path.join(pub, "source", "manifest.json"), "{}");
+ const logs: string[] = [];
+ const o: SourcePublishOpts = {
+ paths: { monorepoRoot: bare } as Paths,
+ publicDir: pub,
+ onLog: (l) => logs.push(l),
+ // No operator files at all: nothing is read before the repository check.
+ scrubFile: path.join(bare, "none.txt"),
+ denylistFile: path.join(bare, "none.txt"),
+ };
+ assert.equal(await publishSource({ ...o, check: true }), 1);
+ assert.ok(existsSync(path.join(pub, "source", "manifest.json")), "--check writes nothing");
+ logs.length = 0;
+ assert.equal(await publishSource(o), 1, "the CLI refuses");
+ assert.equal(logs[0], `[source] ${NO_REPOSITORY}`);
+ assert.ok(!logs.join("\n").includes("fatal"), "no raw git error");
+ assert.ok(!existsSync(path.join(pub, "source")), "an old publish is removed");
+ mkdirSync(path.join(pub, "source"), { recursive: true });
+ writeFileSync(path.join(pub, "source", "manifest.json"), "{}");
+ logs.length = 0;
+ assert.equal(await publishSource({ ...o, noRepository: "empty" }), 0, "buildHomepage builds on");
+ assert.equal(logs[0], `[source] ${NO_REPOSITORY}`);
+ assert.ok(!existsSync(path.join(pub, "source")));
+});
+
+test("round trip: --check writes nothing; publish; a dumb clone of the mirror is main, scrubbed, from static files", async (t) => {
+ if (filterRepoProblem) return t.skip(`git-filter-repo unavailable: ${filterRepoProblem}`);
+ const repo = sourceRepo();
+ // A byte-order mark and CRLF endings, as some editors write them: the rule
+ // must still scrub, and its left side must still be denied (review L2).
+ const files = operatorFiles(`\uFEFF# the planted path\r\n/srv/${PLANTED}==>/home/user\r\n`, `i:${PLANTED}\r\n`);
+ const logs: string[] = [];
+ const o = opts(repo, files, logs);
+ const pub = o.publicDir!;
+ const site = path.dirname(pub);
+
+ assert.equal(await publishSource({ ...o, check: true }), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /check passed — would publish main .* nothing written/);
+ assert.ok(!existsSync(pub), "--check wrote nothing");
+ assert.ok(!existsSync(path.join(site, ".source-publish.json")));
+
+ logs.length = 0;
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /\[source\] audit clean: \d+ objects \(3 commits\), \d+ staged files against 3 denied literals; gitleaks skipped/);
+ assert.match(logs.join("\n"), /\[source\] published main [0-9a-f]{12} as [0-9a-f]{12}: \d+ files/);
+ const manifest = JSON.parse(readFileSync(path.join(pub, "source", "manifest.json"), "utf8"));
+ const sourceCommit = gitIn(repo, "rev-parse", "main");
+ assert.equal(manifest.version, 1);
+ assert.equal(manifest.branch, "main");
+ assert.equal(manifest.sourceCommit, sourceCommit);
+ assert.notEqual(manifest.mirrorHead, sourceCommit, "scrubbed history, different ids");
+ assert.equal(manifest.subject, "moved from /home/user");
+ assert.equal(manifest.cloneUrl, CLONE_URL);
+ assert.equal(manifest.treeHref, TREE_HREF);
+ assert.deepEqual(manifest.tree, { files: 4, dirs: 4, bytes: manifest.tree.bytes });
+ assert.ok(manifest.mirror.packs >= 1);
+ assert.equal(manifest.audit.commits, 3);
+ assert.equal(manifest.audit.gitleaks, "skipped");
+ assert.ok(!("rulesHash" in manifest), "the rules hash is never published");
+ assert.ok(!("literals" in manifest.audit), "nor how many literals there are (review L3)");
+ assert.ok(existsSync(path.join(site, ".source-publish.json")), "…it is kept beside public/");
+
+ // The mirror: the allowlist and the dumb-HTTP files.
+ const mirror = path.join(pub, "source", MIRROR_DIR);
+ for (const f of ["config", "hooks", "description", "filter-repo", "logs", "info/exclude"]) {
+ assert.ok(!existsSync(path.join(mirror, f)), `${f} is never published`);
+ }
+ assert.equal(readFileSync(path.join(mirror, "HEAD"), "utf8"), "ref: refs/heads/main\n");
+ assert.equal(readFileSync(path.join(mirror, "info", "refs"), "utf8"), `${manifest.mirrorHead}\trefs/heads/main\n`);
+ assert.match(readFileSync(path.join(mirror, "objects", "info", "packs"), "utf8"), /^P pack-[0-9a-f]+\.pack$/m);
+ // Only packs and their indexes: git 2.55 also writes .rev files, which the
+ // allowlist leaves behind.
+ const packDir = readdirSync(path.join(mirror, "objects", "pack"));
+ assert.ok(packDir.length >= 2);
+ for (const f of packDir) assert.match(f, /^pack-[0-9a-f]+\.(pack|idx)$/);
+
+ // EVERY object in the published packs, reachable or not — what a dumb-HTTP
+ // reader gets — read from a plain copy of the published files (review L9).
+ const copy = path.join(dir("copy"), "m.git");
+ cpSync(mirror, copy, { recursive: true });
+ const all = execFileSync("git", ["--git-dir", copy, "cat-file", "--batch-all-objects", "--batch"], { maxBuffer: 1 << 26 });
+ assert.equal(all.indexOf(PLANTED), -1, "no object in the packs holds the planted path");
+ assert.ok(all.includes(Buffer.from("/home/user")), "…which was scrubbed, not dropped");
+ assert.ok(!existsSync(path.join(pub, "source", "index.html")), "the /source/ page keeps its route");
+
+ // The clone.
+ const clone = path.join(dir("clone"), "c");
+ execFileSync("git", ["clone", "-q", `file://${mirror}`, clone], { stdio: "pipe" });
+ assert.equal(gitIn(clone, "rev-parse", "HEAD"), manifest.mirrorHead);
+ assert.equal(gitIn(clone, "log", "-1", "--format=%s"), "moved from /home/user");
+ assert.equal(readFileSync(path.join(clone, "notes.txt"), "utf8"), "data lives at /home/user/x\n");
+ const revs = gitIn(clone, "rev-list", "--all").split("\n");
+ assert.equal(revs.length, 3);
+ assert.throws(
+ () => execFileSync("git", ["grep", "-F", PLANTED, ...revs], { cwd: clone, stdio: "pipe" }),
+ (e: { status?: number }) => e.status === 1,
+ "no revision holds the planted path",
+ );
+ assert.ok(!gitIn(clone, "log", "--all", "--format=%B%an%ae").includes(PLANTED));
+
+ // The tarball and its sidecar agree with the manifest and the bytes.
+ const tarball = path.join(pub, "downloads", path.basename(TARBALL_HREF));
+ const snapshot = JSON.parse(readFileSync(path.join(pub, "downloads", "snapshot.json"), "utf8"));
+ assert.equal(snapshot.sha256, sha256(tarball));
+ assert.equal(snapshot.sha256, manifest.tarball.sha256);
+ assert.equal(snapshot.bytes, statSync(tarball).size);
+ assert.equal(snapshot.commit, manifest.mirrorHead);
+ assert.equal(snapshot.subject, "moved from /home/user");
+
+ // The tree pages.
+ const tree = path.join(pub, "source", "tree");
+ for (const d of ["", "app", "app/[slug]", "fonts"]) assert.ok(existsSync(path.join(tree, d, "index.html")), d);
+ assert.match(readFileSync(path.join(tree, "fonts", "index.html"), "utf8"), /href="Archivo%5Bwdth%2Cwght%5D\.ttf"/);
+ assert.equal(readFileSync(path.join(tree, "notes.txt"), "utf8"), "data lives at /home/user/x\n");
+ assert.equal(readdirSync(o.scratchRoot!).length, 0, "the scratch dir is removed");
+
+ // Nothing changed: the next publish skips.
+ logs.length = 0;
+ assert.equal(await publishSource(o), 0);
+ assert.match(logs.join("\n"), /up to date at/);
+
+ // The deploy check (M1): a build's out/ that is this publish deploys…
+ const out = path.join(dir("out"), "out");
+ const check = { ...files, homeDir: HOME, publicDir: pub, sourceRepo: path.join(repo, ".git"), gitleaks: null };
+ const checkPaths = o.paths!;
+ // A build's out/ always has the /source page; a --no-source one, nothing more.
+ mkdirSync(path.join(out, "source"), { recursive: true });
+ writeFileSync(path.join(out, "source", "index.html"), "<p>No source published in this build.</p>");
+ assert.equal(await publishedSourceProblem(checkPaths, out, check), null, "--no-source deploys as before");
+ cpSync(path.join(pub, "source"), path.join(out, "source"), { recursive: true });
+ cpSync(path.join(pub, "downloads"), path.join(out, "downloads"), { recursive: true });
+ assert.equal(await publishedSourceProblem(checkPaths, out, check), null);
+ // Review R2-L3: a state naming another mirror head, or another gitleaks,
+ // refuses — each by its own sentence (the digest below would refuse
+ // neither, since out/ is untouched).
+ const stateFile = path.join(site, ".source-publish.json");
+ const good = readFileSync(stateFile, "utf8");
+ const stateWith = (tweak: object) => writeFileSync(stateFile, JSON.stringify({ ...JSON.parse(good), ...tweak }));
+ stateWith({ mirrorHead: "f".repeat(40) });
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /is not the last publish \(ffffffffffff\)/);
+ stateWith({ gitleaks: "gitleaks 8.0 (abc)" });
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /scanned by another gitleaks/);
+ writeFileSync(stateFile, good);
+ // Review R2-L2: a mixed out/ — one tree file swapped, or one pack byte
+ // flipped — does not match the digest of what was audited.
+ const readme = path.join(out, "source", "tree", "README.md");
+ const readmeWas = readFileSync(readme);
+ writeFileSync(readme, "another publish's README\n");
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /not the ones that were audited/);
+ writeFileSync(readme, readmeWas);
+ const packDirOut = path.join(out, "source", MIRROR_DIR, "objects", "pack");
+ const pack = path.join(packDirOut, readdirSync(packDirOut).find((f) => f.endsWith(".pack"))!);
+ const packWas = readFileSync(pack);
+ chmodSync(pack, 0o644); // git writes packs read-only; the digest reads bytes, not modes
+ const flipped = Buffer.from(packWas);
+ flipped[Math.floor(flipped.length / 2)] ^= 0x01;
+ writeFileSync(pack, flipped);
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /not the ones that were audited/);
+ writeFileSync(pack, packWas);
+ assert.equal(await publishedSourceProblem(checkPaths, out, check), null, "restored, it deploys again");
+ // …a tampered tarball does not…
+ writeFileSync(path.join(out, "downloads", path.basename(TARBALL_HREF)), "other bytes");
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /tarball is not the one its manifest describes/);
+ cpSync(path.join(pub, "downloads"), path.join(out, "downloads"), { recursive: true });
+ // …nor one of an older main.
+ writeFileSync(path.join(repo, "later.txt"), "x\n");
+ gitIn(repo, "add", "-A");
+ gitIn(repo, "commit", "-q", "-m", "later");
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /main is now [0-9a-f]{12} — run `archilyzer build homepage`/);
+
+ // A REFUSAL WITHDRAWS THE PUBLISH (M1): deny a literal the mirror holds.
+ // The deploy check refuses first (other rules), then the publish refuses
+ // and removes public/source and the downloads.
+ writeFileSync(files.denylistFile, `i:${PLANTED}\nhello again\n`);
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /audited under other rules/);
+ logs.length = 0;
+ assert.equal(await publishSource(o), 1);
+ assert.match(logs.join("\n"), /AUDIT REFUSED/);
+ assert.match(logs.join("\n"), /denylist line 2 \(len 11\)/);
+ assert.ok(!logs.join("\n").includes("hello again"), "the report never prints the literal");
+ assert.ok(!existsSync(path.join(pub, "source")), "public/source is withdrawn");
+ assert.ok(!existsSync(tarball), "the tarball is withdrawn");
+ assert.ok(!existsSync(path.join(pub, "downloads", "snapshot.json")));
+ assert.ok(!existsSync(path.join(site, ".source-publish.json")));
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /no record of the rules/);
+});
+
+test("a denied literal no rule removes: refused, nothing written, the report never prints it", async (t) => {
+ if (filterRepoProblem) return t.skip(`git-filter-repo unavailable: ${filterRepoProblem}`);
+ const repo = sourceRepo({ "keys.txt": `the ${SECRET} is here\n` });
+ const files = operatorFiles(`/srv/${PLANTED}==>/home/user\n`, `${SECRET}\n`);
+ const logs: string[] = [];
+ const o = opts(repo, files, logs);
+ const downloads = path.join(o.publicDir!, "downloads");
+ mkdirSync(downloads, { recursive: true });
+ writeFileSync(path.join(downloads, "snapshot.json"), "yesterday's");
+ assert.equal(await publishSource({ ...o, keepScratch: true }), 1);
+ const report = logs.join("\n");
+ assert.ok(!report.includes(SECRET), report);
+ assert.ok(!report.includes("is here"), "no byte from beside the hit (review L4)");
+ assert.match(report, /AUDIT REFUSED: 1 hit in \d+ objects/);
+ assert.match(report, /denylist line 1 \(len 13\): 1 in blob/);
+ assert.match(report, /blob [0-9a-f]{12} keys\.txt \(byte 4\): denylist line 1 \(len 13\)/);
+ assert.match(report, /add a rule to .*source-scrub\.txt or drop the file from history, then re-run\./);
+ assert.ok(!existsSync(path.join(o.publicDir!, "source")), "no manifest, no mirror");
+ assert.ok(!existsSync(path.join(downloads, "snapshot.json")), "yesterday's snapshot is withdrawn too");
+ // --keep-scratch keeps the clone for a look, never the scrub rules.
+ const kept = /scratch kept at (\S+) \(replace\.txt, the scrub rules, deleted\)/.exec(report);
+ assert.ok(kept, report);
+ // The log tildifies with the real home dir, so a TMPDIR under it prints "~/…".
+ const keptDir = kept[1].replace(/^~(?=\/|$)/, os.homedir());
+ assert.ok(existsSync(path.join(keptDir, MIRROR_DIR)), "the clone is named as the mirror is (stagit names it after its directory)");
+ assert.ok(!existsSync(path.join(keptDir, "replace.txt")));
+ rmSync(keptDir, { recursive: true, force: true });
+});
+
+test("a refusal naming a tree path masks a literal that spans path components (review R2-L1)", async (t) => {
+ if (filterRepoProblem) return t.skip(`git-filter-repo unavailable: ${filterRepoProblem}`);
+ // `plant/secret` is invisible to the object walk (which reads entry names
+ // one at a time), so the first refusal to name it is the tree's own: a
+ // tracked symlink below it, then a tracked index.html.
+ const spanning = "plant/secret";
+ const repo = sourceRepo();
+ mkdirSync(path.join(repo, "plant", "secret"), { recursive: true });
+ symlinkSync("../../README.md", path.join(repo, "plant", "secret", "link"));
+ gitIn(repo, "add", "-A");
+ gitIn(repo, "commit", "-q", "-m", "a link");
+ const files = operatorFiles(`/srv/${PLANTED}==>/home/user\n`, `${spanning}\n`);
+ const logs: string[] = [];
+ assert.equal(await publishSource(opts(repo, files, logs)), 1);
+ let log = logs.join("\n");
+ assert.match(log, /REFUSED: the tree holds a symlink \(\[REDACTED\]\/link\)/);
+ assert.ok(!log.includes(spanning), log);
+
+ gitIn(repo, "rm", "-q", "plant/secret/link");
+ mkdirSync(path.join(repo, "plant", "secret"), { recursive: true });
+ writeFileSync(path.join(repo, "plant", "secret", "index.html"), "<p>x</p>");
+ gitIn(repo, "add", "-A");
+ gitIn(repo, "commit", "-q", "-m", "an index");
+ logs.length = 0;
+ assert.equal(await publishSource(opts(repo, files, logs)), 1);
+ log = logs.join("\n");
+ assert.match(log, /REFUSED: the tree already has \[REDACTED\]\/index\.html/);
+ assert.ok(!log.includes(spanning), log);
+});
+
+test("--no-source's clear: the manifest, mirror, tree, tarball and skip key go; a linked downloads/ goes as a link", async () => {
+ const site = dir("clear");
+ const pub = path.join(site, "public");
+ mkdirSync(path.join(pub, "source", MIRROR_DIR), { recursive: true });
+ writeFileSync(path.join(pub, "source", "manifest.json"), "{}");
+ const elsewhere = dir("elsewhere");
+ writeFileSync(path.join(elsewhere, "snapshot.json"), "the other checkout's");
+ symlinkSync(elsewhere, path.join(pub, "downloads"));
+ writeFileSync(path.join(site, ".source-publish.json"), "{}");
+ const logs: string[] = [];
+ await clearPublishedSource({ paths: {} as Paths, publicDir: pub, onLog: (l) => logs.push(l) });
+ assert.ok(!existsSync(path.join(pub, "source")));
+ assert.ok(!existsSync(path.join(site, ".source-publish.json")));
+ assert.equal(lstatSync(path.join(pub, "downloads"), { throwIfNoEntry: false }), undefined);
+ assert.equal(readFileSync(path.join(elsewhere, "snapshot.json"), "utf8"), "the other checkout's");
+ assert.match(logs.join("\n"), /previously published source .* was removed/);
+});
+
+// ── the history pages (release 15 slice SG) ─────────────────────────────────
+//
+// Over a fake stagit that writes what stagit writes (sourceHistory.test.ts
+// runs the real one), and with filter-repo replaced by `true` — no rewrite,
+// so these run on any machine; the history does not care what the rewrite did.
+
+const TOKENS_FILE = path.join(path.dirname(fileURLToPath(import.meta.url)), "..", "styles", "tokens.css");
+
+// The fake (__fixtures__/fakeStagit.ts) writes what stagit writes, with -c
+// and -l as stagit.c has them.
+function fakeStagit(log: string, o: { exit?: number; extra?: string; pad?: number } = {}): string {
+ return writeFakeStagit(path.join(dir("fake-stagit"), "stagit"), log, o);
+}
+
+// A render cache of its own, where XDG_CACHE_HOME would put it.
+const cacheDirIn = (root: string) => path.join(root, "archilyzer", "source-history");
+
+const stagitCalls = (log: string) => (existsSync(log) ? readFileSync(log, "utf8").trim().split("\n") : []);
+
+test("history: published with the source — the allowlist at /source/git/, the manifest's block, the skip key, incremental, and the deploy check names it", async () => {
+ const repo = sourceRepo();
+ const files = operatorFiles("", "");
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const cacheDir = cacheDirIn(dir("cache-home"));
+ const o = opts(repo, files, logs, { filterRepo: ["true"], stagit: fakeStagit(log), tokensFile: TOKENS_FILE, historyCacheDir: cacheDir });
+ const pub = o.publicDir!;
+ const site = path.dirname(pub);
+
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ let text = logs.join("\n");
+ assert.match(text, /\[source\] history: stagit \(sha256 [0-9a-f]{12}\) — 3 commits, 3 pages rendered; 9 files, [\d.]+ MB, the largest git\/\S+ [\d.]+ MB/);
+ assert.match(text, /\[source\] published main .* tree \d+ dirs, history 3 commits in 9 files\)/);
+ assert.deepEqual(stagitCalls(log), [
+ `cache=${path.join(cacheDir, "stagit.cache")} limit= base=https://archilyzer.pages.dev/source/git/ repo=${MIRROR_DIR}`,
+ ]);
+
+ const manifest = JSON.parse(readFileSync(path.join(pub, "source", "manifest.json"), "utf8"));
+ assert.deepEqual(Object.keys(manifest.history).sort(), ["bytes", "commits", "files", "head", "href", "sha256", "tool", "total"]);
+ assert.equal(manifest.history.href, "/source/git/log.html");
+ assert.equal(manifest.history.commits, 3);
+ assert.equal(manifest.history.total, 3);
+ assert.equal(manifest.history.head, manifest.mirrorHead);
+ assert.equal(manifest.history.files, 9);
+ assert.equal(manifest.history.sha256, await historyDigest(pub));
+ assert.match(manifest.history.tool, /^stagit \(sha256 [0-9a-f]{12}\)$/);
+ assert.ok(manifest.files >= 9, "the manifest's count includes the history");
+
+ const git = path.join(pub, "source", "git");
+ assert.deepEqual(readdirSync(git).sort(), ["atom.xml", "commit", "files.html", "log.html", "refs.html", "style.css", "tags.xml"]);
+ const logHtml = readFileSync(path.join(git, "log.html"), "utf8");
+ assert.ok(logHtml.includes(`<span class="desc">Archilyzer</span> ${CLONE_URL} `), "stagit's header: the name and the clone URL");
+ assert.ok(logHtml.includes(HISTORY_BACK_LINK));
+ assert.match(logHtml, /<script>\(function\(\)\{try\{var d=document\.documentElement/);
+ assert.match(logHtml, /href="\.\.\/tree\/README\.md"/);
+ // The feed is copied as it is: not even its file/ text is rewritten.
+ assert.equal(readFileSync(path.join(git, "atom.xml"), "utf8"), "<feed>https://archilyzer.pages.dev/source/git/ file/README.md.html</feed>\n");
+ assert.match(readFileSync(path.join(git, "style.css"), "utf8"), /html\[data-base="dark"\] \{/);
+ // The description and url files stagit read are never published.
+ for (const f of ["description", "url"]) assert.ok(!existsSync(path.join(pub, "source", MIRROR_DIR, f)), f);
+
+ const stateFile = path.join(site, ".source-publish.json");
+ const state = JSON.parse(readFileSync(stateFile, "utf8"));
+ assert.equal(state.stagit, manifest.history.tool);
+ assert.deepEqual(state.history, { files: 9, digest: manifest.history.sha256 });
+
+ // Unchanged: skipped, stagit not run.
+ logs.length = 0;
+ assert.equal(await publishSource(o), 0);
+ assert.match(logs.join("\n"), /up to date at/);
+ assert.equal(stagitCalls(log).length, 1);
+
+ // The deploy check: this publish in out/ deploys; a history page edited,
+ // or the history missing, is named.
+ const out = path.join(dir("out"), "out");
+ mkdirSync(path.join(out, "source"), { recursive: true });
+ writeFileSync(path.join(out, "source", "index.html"), "<p>the page</p>");
+ cpSync(path.join(pub, "source"), path.join(out, "source"), { recursive: true });
+ cpSync(path.join(pub, "downloads"), path.join(out, "downloads"), { recursive: true });
+ const check = { ...files, homeDir: HOME, publicDir: pub, sourceRepo: path.join(repo, ".git"), gitleaks: null, stagit: null };
+ assert.equal(await publishedSourceProblem(o.paths!, out, check), null);
+ const outLog = path.join(out, "source", "git", "log.html");
+ const was = readFileSync(outLog);
+ writeFileSync(outLog, Buffer.concat([was, Buffer.from(" ")]));
+ assert.match((await publishedSourceProblem(o.paths!, out, check))!, /history pages \(\/source\/git\/\) are not the ones that were audited — run `archilyzer build homepage`/);
+ rmSync(path.join(out, "source", "git"), { recursive: true });
+ assert.match((await publishedSourceProblem(o.paths!, out, check))!, /history pages \(\/source\/git\/\) are not the ones that were audited/);
+ cpSync(path.join(pub, "source", "git"), path.join(out, "source", "git"), { recursive: true });
+ assert.equal(await publishedSourceProblem(o.paths!, out, check), null, "restored, it deploys");
+
+ // A new commit on main: only its page is rendered, the rest from the cache.
+ writeFileSync(path.join(repo, "later.txt"), "x\n");
+ gitIn(repo, "add", "-A");
+ gitIn(repo, "commit", "-q", "-m", "later");
+ logs.length = 0;
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /— 4 commits, 1 page rendered \(the rest from the cache\); 10 files/);
+ // --force renders every page again.
+ logs.length = 0;
+ assert.equal(await publishSource({ ...o, force: true }), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /— 4 commits, 4 pages rendered; 10 files/);
+ // --check writes nothing — the cache included — and still renders (and
+ // sweeps) the pages in its own scratch.
+ const cacheBefore = readFileSync(path.join(cacheDir, "key.json"), "utf8");
+ logs.length = 0;
+ assert.equal(await publishSource({ ...o, check: true }), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /— 4 commits, 4 pages rendered;/);
+ assert.equal(stagitCalls(log).at(-1), `cache= limit= base=https://archilyzer.pages.dev/source/git/ repo=${MIRROR_DIR}`);
+ assert.equal(readFileSync(path.join(cacheDir, "key.json"), "utf8"), cacheBefore);
+});
+
+test("history: without stagit — one line with the install, nothing at /source/git/, no block in the manifest; installing it re-publishes", async () => {
+ const repo = sourceRepo();
+ const files = operatorFiles("", "");
+ const logs: string[] = [];
+ const o = opts(repo, files, logs, { filterRepo: ["true"], stagit: null, tokensFile: TOKENS_FILE, scratchRoot: dir("scratch-root") });
+ const pub = o.publicDir!;
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ const said = logs.filter((l) => /stagit/.test(l));
+ assert.deepEqual(said, [
+ "[source] stagit not found (not on PATH, not in ~/.local/bin) — publishing without the history pages (/source/git/); install it once: git clone git://git.codemadness.org/stagit && make -C stagit && cp stagit/stagit ~/.local/bin/",
+ ]);
+ assert.match(logs.join("\n"), /\[source\] published main .*, no history\)/);
+ const manifest = JSON.parse(readFileSync(path.join(pub, "source", "manifest.json"), "utf8"));
+ assert.ok(!("history" in manifest));
+ assert.ok(!existsSync(path.join(pub, "source", "git")));
+ const state = JSON.parse(readFileSync(path.join(path.dirname(pub), ".source-publish.json"), "utf8"));
+ assert.equal(state.stagit, "absent");
+ assert.equal(state.history, null);
+
+ logs.length = 0;
+ assert.equal(await publishSource(o), 0);
+ assert.match(logs.join("\n"), /up to date at/, "no stagit, nothing changed: skipped");
+
+ // stagit installed since: not skipped, and the history is published.
+ logs.length = 0;
+ const log = path.join(dir("stagit-log"), "calls");
+ assert.equal(await publishSource({ ...o, stagit: fakeStagit(log) }), 0, logs.join("\n"));
+ assert.ok(!/up to date/.test(logs.join("\n")));
+ assert.equal(JSON.parse(readFileSync(path.join(pub, "source", "manifest.json"), "utf8")).history.commits, 3);
+ assert.ok(existsSync(path.join(pub, "source", "git", "log.html")));
+});
+
+test("history: a denied literal in a history page refuses, names the page and never the literal, and withdraws the publish and the render cache", async () => {
+ const repo = sourceRepo();
+ const planted = "plantedinhistory";
+ const files = operatorFiles("", "");
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const cacheDir = cacheDirIn(dir("cache-home"));
+ // The literal is only ever in what stagit writes, never in the repository:
+ // what this proves is that the file gate reads source/git/**.
+ const o = opts(repo, files, logs, { filterRepo: ["true"], stagit: fakeStagit(log, { extra: `said ${planted} here` }), tokensFile: TOKENS_FILE, historyCacheDir: cacheDir });
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ assert.ok(existsSync(path.join(o.publicDir!, "source", "git", "log.html")));
+ assert.ok(existsSync(cacheDir));
+
+ // Deny it: the rules change, so every page is rendered again — and swept.
+ writeFileSync(files.denylistFile, `${planted}\n`);
+ logs.length = 0;
+ assert.equal(await publishSource(o), 1);
+ const report = logs.join("\n");
+ assert.match(report, /AUDIT REFUSED: 3 hits in \d+ objects \(3 commits\), \d+ staged files/);
+ assert.match(report, /denylist line 1 \(len 16\): 3 in files/);
+ assert.match(report, /\[source\] {3}file source\/git\/commit\/[0-9a-f]{40}\.html \(contents, byte \d+\): denylist line 1 \(len 16\)/);
+ assert.ok(!report.includes(planted), report);
+ assert.ok(!report.includes("said"), "no byte from beside the hit");
+ assert.match(report, /the previous publish was WITHDRAWN \(mirror, tree, history, tarball\)/);
+ assert.ok(!existsSync(path.join(o.publicDir!, "source")), "public/source, the history with it, is withdrawn");
+ assert.ok(!existsSync(cacheDir), "the render cache is removed too");
+});
+
+test("history: a stagit that fails, or a page over the host's limit, is a WARNING; the rest is published, and the next build tries again", async () => {
+ const repo = sourceRepo();
+ const files = operatorFiles("", "");
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const o = opts(repo, files, logs, { filterRepo: ["true"], stagit: fakeStagit(log, { exit: 3 }), tokensFile: TOKENS_FILE, scratchRoot: dir("scratch-root") });
+ const pub = o.publicDir!;
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /\[source\] WARNING: the history pages were not rendered: stagit exited 3: stagit: something broke — publishing without the history pages \(\/source\/git\/\)/);
+ assert.ok(existsSync(path.join(pub, "source", MIRROR_DIR, "info", "refs")), "the mirror is published");
+ assert.ok(!existsSync(path.join(pub, "source", "git")));
+ const state = JSON.parse(readFileSync(path.join(path.dirname(pub), ".source-publish.json"), "utf8"));
+ assert.match(state.stagit, /^stagit \(sha256/);
+ assert.equal(state.history, null);
+ logs.length = 0;
+ assert.equal(await publishSource(o), 0);
+ assert.ok(!/up to date/.test(logs.join("\n")), "a stagit present and no history: the next build tries again");
+ assert.equal(stagitCalls(log).length, 2);
+
+ // One page over the step's 24 MiB: the history is dropped, not the publish.
+ logs.length = 0;
+ assert.equal(await publishSource({ ...o, stagit: fakeStagit(log, { pad: MAX_FILE_BYTES }) }), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /\[source\] WARNING: git\/commit\/[0-9a-f]{40}\.html is 24\.0 MiB, over the step's limit of 24\.0 MiB \(Pages allows 25 MiB per file\) — publishing without the history pages/);
+ assert.ok(!existsSync(path.join(pub, "source", "git")));
+ assert.ok(!("history" in JSON.parse(readFileSync(path.join(pub, "source", "manifest.json"), "utf8"))));
+
+ // Unreadable design tokens: the same.
+ logs.length = 0;
+ assert.equal(await publishSource({ ...o, stagit: fakeStagit(log), tokensFile: path.join(dir("none"), "tokens.css") }), 0);
+ assert.match(logs.join("\n"), /\[source\] WARNING: the design tokens \(.*tokens\.css\) could not be read — publishing without the history pages/);
+});
+
+test("history: past the cap, the latest N of M — the manifest's commits and total, the log line, the summary; the pages of the newest only", async () => {
+ const repo = sourceRepo();
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const o = opts(repo, operatorFiles("", ""), logs, {
+ filterRepo: ["true"],
+ stagit: fakeStagit(log),
+ tokensFile: TOKENS_FILE,
+ historyCacheDir: cacheDirIn(dir("cache-home")),
+ maxHistoryCommits: 2,
+ });
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ const text = logs.join("\n");
+ assert.equal(stagitCalls(log).at(-1), `cache= limit=2 base=https://archilyzer.pages.dev/source/git/ repo=${MIRROR_DIR}`, "-l, and no -c");
+ assert.match(text, /\[source\] history: stagit \(sha256 [0-9a-f]{12}\) — the latest 2 of 3 commits, 3 pages rendered; 8 files/);
+ assert.match(text, /history the latest 2 of 3 commits in 8 files\)/);
+ const manifest = JSON.parse(readFileSync(path.join(o.publicDir!, "source", "manifest.json"), "utf8"));
+ assert.equal(manifest.history.commits, 2);
+ assert.equal(manifest.history.total, 3);
+ const newest = gitIn(repo, "rev-list", "--max-count=2", "main").split("\n");
+ const published = readdirSync(path.join(o.publicDir!, "source", "git", "commit")).sort();
+ assert.equal(published.length, 2);
+ // filter-repo is `true` here, so the mirror's ids are main's.
+ assert.deepEqual(published, newest.map((c) => `${c}.html`).sort());
+});
+
+test("history: the render cache is the XDG cache's archilyzer/source-history — none for --check, none inside the checkout or the public dir", async () => {
+ const home = dir("xdg");
+ const checkout = dir("chk");
+ const pub = path.join(dir("site"), "public");
+ const paths = { monorepoRoot: checkout, sourceHistoryCacheDir: cacheDirIn(home) } as Paths;
+ const logs: string[] = [];
+ const onLog = (l: string) => logs.push(l);
+ assert.equal(await historyCacheFor({}, paths, pub, onLog), cacheDirIn(home));
+ assert.equal(await historyCacheFor({ check: true }, paths, pub, onLog), null);
+ assert.equal(await historyCacheFor({ historyCacheDir: null }, paths, pub, onLog), null);
+ assert.equal(await historyCacheFor({ historyCacheDir: path.join(home, "other") }, paths, pub, onLog), path.join(home, "other"));
+ assert.deepEqual(logs, []);
+ assert.equal(await historyCacheFor({ historyCacheDir: path.join(checkout, ".cache", "h") }, paths, pub, onLog), null);
+ assert.match(logs.at(-1)!, /the render cache .* \(XDG_CACHE_HOME\) is inside the checkout; rendering without it/);
+ assert.equal(await historyCacheFor({ historyCacheDir: path.join(pub, "h") }, paths, pub, onLog), null);
+ assert.match(logs.at(-1)!, /is inside the public dir/);
+ // getPaths: XDG_CACHE_HOME, else ~/.cache (an empty one is unset).
+ const { getPaths } = await import("../lib/paths");
+ const real = getPaths().sourceHistoryCacheDir;
+ assert.ok(real.endsWith(path.join("archilyzer", "source-history")), real);
+ const want = process.env.XDG_CACHE_HOME || path.join(os.homedir(), ".cache");
+ assert.equal(real, path.join(want, "archilyzer", "source-history"));
+});
+
+test("history: an unusable render cache is one masked line and a render without it — the publish and its history go on (review L1, L2)", async () => {
+ const repo = sourceRepo();
+ // The cache's path carries a denied literal: the line naming it is masked,
+ // as every line of the step is.
+ const planted = "plantedcachedir";
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const root = path.join(dir("ro-cache"), planted);
+ mkdirSync(root, { recursive: true });
+ chmodSync(root, 0o555);
+ try {
+ const o = opts(repo, operatorFiles("", `${planted}\n`), logs, {
+ filterRepo: ["true"],
+ stagit: fakeStagit(log),
+ tokensFile: TOKENS_FILE,
+ historyCacheDir: path.join(root, "archilyzer", "source-history"),
+ });
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ const text = logs.join("\n");
+ assert.match(text, /\[source\] history: the render cache .*\[REDACTED\].* is unusable \(EACCES\); rendering without it/);
+ assert.ok(!text.includes(planted), "never the literal");
+ assert.equal(stagitCalls(log).at(-1), `cache= limit= base=https://archilyzer.pages.dev/source/git/ repo=${MIRROR_DIR}`);
+ assert.equal(JSON.parse(readFileSync(path.join(o.publicDir!, "source", "manifest.json"), "utf8")).history.commits, 3);
+ } finally {
+ chmodSync(root, 0o755);
+ }
+});
+
+test("history: pages that would take the publish over the step's file limit are dropped with a WARNING, never the publish (review L6)", async () => {
+ const repo = sourceRepo();
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const o = opts(repo, operatorFiles("", ""), logs, {
+ filterRepo: ["true"],
+ stagit: fakeStagit(log),
+ tokensFile: TOKENS_FILE,
+ historyCacheDir: null,
+ // Any limit the rest already fills: the history is what is dropped.
+ historyFileLimit: 1,
+ });
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /\[source\] WARNING: 9 history files would make \d+ files to publish, over the step's limit of 1 \(Pages allows 20,000 per deployment\) — publishing without the history pages \(\/source\/git\/\)/);
+ assert.ok(!existsSync(path.join(o.publicDir!, "source", "git")));
+ assert.ok(existsSync(path.join(o.publicDir!, "source", MIRROR_DIR, "info", "refs")), "the rest is published");
+ assert.ok(!("history" in JSON.parse(readFileSync(path.join(o.publicDir!, "source", "manifest.json"), "utf8"))));
+});
+
+test("history: a refusal with a render cache it cannot remove still exits 1 and withdraws the source; one masked line says so (re-review)", async () => {
+ const repo = sourceRepo();
+ const planted = "plantedinhistory";
+ const files = operatorFiles("", "");
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const cacheDir = cacheDirIn(dir("cache-home"));
+ const o = opts(repo, files, logs, {
+ filterRepo: ["true"],
+ stagit: fakeStagit(log, { extra: `said ${planted} here` }),
+ tokensFile: TOKENS_FILE,
+ historyCacheDir: cacheDir,
+ });
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ assert.ok(existsSync(path.join(cacheDir, "key.json")));
+ // The cache becomes read-only; then a rule denies what its pages say.
+ chmodSync(cacheDir, 0o555);
+ try {
+ writeFileSync(files.denylistFile, `${planted}\n`);
+ logs.length = 0;
+ assert.equal(await publishSource(o), 1, "the refusal's exit stands");
+ const text = logs.join("\n");
+ assert.match(text, /AUDIT REFUSED/);
+ assert.match(text, /the previous publish was WITHDRAWN/);
+ assert.match(text, /\[source\] the history's render cache .* could not be removed \(EACCES\); remove it by hand/);
+ assert.ok(!text.includes(planted));
+ assert.ok(!existsSync(path.join(o.publicDir!, "source")), "the source is withdrawn");
+ assert.ok(!existsSync(path.join(path.dirname(o.publicDir!), ".source-publish.json")));
+ } finally {
+ chmodSync(cacheDir, 0o755);
+ }
+});
diff --git a/common/publish/source.ts b/common/publish/source.ts
@@ -0,0 +1,1426 @@
+// `archilyzer source publish` — the repo on the project site, read-only.
+//
+// The private repository is never rewritten. Every publish makes a FRESH bare
+// clone of its `main`, rewrites that copy with git-filter-repo (the operator's
+// scrub rules over file contents AND commit messages), repacks it for git's
+// dumb-HTTP protocol, and publishes these under homepage/public:
+//
+// source/archilyzer.git/ a clonable mirror: HEAD, refs, info/refs,
+// objects/info/packs, the packs — static files only
+// source/tree/ the tracked files of main, raw, with an
+// index.html per directory (sourceTree.ts)
+// source/git/ the history: the log, a page per commit with its
+// diff, the refs, two Atom feeds — stagit's, when
+// it is installed (sourceHistory.ts)
+// downloads/ archilyzer-source.tar.gz + snapshot.json, the
+// tarball the Downloads page has always offered
+// source/manifest.json written LAST: what was published, from what
+//
+// THE GATE. Before anything is staged for the site, every object of the
+// rewritten mirror, and then every staged file, is searched for every denied
+// literal (sourceAudit.ts): the operator's denylist plus every scrub rule's
+// left side. One hit and nothing is written; the report names the literal by
+// number, never by its bytes.
+//
+// OPERATOR-PRIVATE INPUTS live outside the repo, in
+// `${ARCHILYZER_CONFIG_DIR ?? ~/.config/archilyzer}/`: source-scrub.txt
+// (git-filter-repo `lhs==>rhs` lines; `<home dir>==>/home/user` is always
+// applied first) and source-denylist.txt (one literal per line, `i:` = any
+// case). A missing file is a refusal naming it. Nothing here names a user.
+//
+// `buildHomepage` runs this between compose and `next build`, so `archilyzer
+// build homepage`, the runbooks' home scripts and the editor's /sites homepage
+// jobs all publish it; an unchanged main with unchanged rules skips.
+
+import { execFile } from "node:child_process";
+import { createHash } from "node:crypto";
+import { createReadStream, existsSync } from "node:fs";
+import { cp, lstat, mkdir, mkdtemp, readdir, readFile, realpath, rm, stat, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { runChildIntoLog } from "../jobs/runChild";
+import { copyPublicFile, ownDir, writePublicFile } from "../bin/_publicFile";
+import { getPaths, type Paths } from "../lib/paths";
+import { PROJECT_NAME, PROJECT_URL } from "../lib/project";
+import {
+ CLONE_URL,
+ HISTORY_DIR,
+ HISTORY_LOG_HREF,
+ MIRROR_DIR,
+ SOURCE_MANIFEST_VERSION,
+ TARBALL_HREF,
+ TREE_HREF,
+ parseSourceManifest,
+ type SourceHistory,
+ type SourceManifest,
+} from "../lib/sourceManifest";
+import {
+ HistoryProblem,
+ STAGIT_INSTALL,
+ dropHistoryCache,
+ SOURCE_HISTORY_MAX_COMMITS,
+ historyCacheKey,
+ historyStylesheet,
+ homepageThemeScript,
+ renderHistory,
+ resolveStagit,
+ stagitIdentity,
+ type HistoryRun,
+} from "./sourceHistory";
+import {
+ SourceRefusal,
+ auditBare,
+ auditFiles,
+ cleanGitEnv,
+ dedupeLiterals,
+ formatAuditReport,
+ operatorLines,
+ maskLiterals,
+ onPath,
+ parseDenylist,
+ tildify,
+ type Literal,
+} from "./sourceAudit";
+import { writeTreeIndexes } from "./sourceTree";
+// Type-only: build.ts imports THIS module lazily, and must not load it (or the
+// AWS SDK this would drag in) at import time.
+import type { PublishOpts } from "./build";
+
+export { SourceRefusal } from "./sourceAudit";
+
+/** The one branch mirrored (an operator decision: no other refs, no tags). */
+export const SOURCE_BRANCH = "main";
+
+/**
+ * The step's own version, hashed into the rules hash (the skip key and the
+ * deploy check). BUMP IT WHENEVER THE SCRUB OR THE AUDIT CHANGES — what the
+ * rules parse to, what the gate reads, what it refuses — so an unchanged
+ * `main` is re-published through the fixed step instead of skipped, and a
+ * `homepage/out` built by the old step refuses to deploy.
+ * 1 — release 12 slice R.
+ * 2 — release 12 slice R review: BOM/CRLF/trailing-slash parsing, empty
+ * left sides refused, `404.html` refused, no context bytes.
+ * 3 — the re-review: masked refusal messages, a digest of every published
+ * file and the gitleaks identity in the key.
+ * 4 — release 15 slice SG: the history pages (source/git/, stagit) are
+ * staged, swept by the file gate and bound by the digest; stagit's
+ * identity is in the key.
+ */
+export const SOURCE_STEP_VERSION = 4;
+
+/** What `pipx run` fetches when `git filter-repo` is not installed. */
+export const FILTER_REPO_PIPX_SPEC = "git-filter-repo==2.47.0";
+export const FILTER_REPO_INSTALL = "pipx install git-filter-repo";
+
+/** The home-directory rule's replacement (always the first scrub rule). */
+export const HOME_REPLACEMENT = "/home/user";
+
+// Cloudflare Pages allows 20,000 files per deployment and 25 MiB per file;
+// the step refuses well inside both, leaving the rest of the site its room.
+export const MAX_FILES = 15_000;
+export const MAX_FILE_BYTES = 24 * 1024 * 1024;
+
+// Packs are split at this size (under the per-file cap, with room to grow).
+const PACK_SIZE = "20m";
+
+const TARBALL_NAME = path.basename(TARBALL_HREF);
+
+export type SourcePublishOpts = PublishOpts & {
+ // Rebuild even when main and the rules are unchanged.
+ force?: boolean;
+ // Build, audit and count everything, then write NOTHING.
+ check?: boolean;
+ // Leave the scratch dir (the rewritten bare clone, the stage) for a look.
+ keepScratch?: boolean;
+ // The repository to mirror. Default: this checkout's git COMMON dir, so a
+ // worktree build mirrors the primary's main.
+ sourceRepo?: string;
+ // Default: HOMEPAGE_PUBLIC_DIR, else <repo>/homepage/public.
+ publicDir?: string;
+ scrubFile?: string;
+ denylistFile?: string;
+ // The filter-repo argv; default: resolveFilterRepo().
+ filterRepo?: string[] | null;
+ // The gitleaks binary; null skips the secret scan (tests). Default "gitleaks".
+ gitleaks?: string | null;
+ // The stagit binary; null publishes without the history pages, as a
+ // machine without stagit does. Default: resolveStagit(paths.stagitBin).
+ stagit?: string | null;
+ // The history's render cache; null renders without one. Default:
+ // paths.sourceHistoryCacheDir (~/.cache/archilyzer/source-history).
+ historyCacheDir?: string | null;
+ // The history's cap. Default: SOURCE_HISTORY_MAX_COMMITS (the tests' seam).
+ maxHistoryCommits?: number;
+ // The file limit the history's last-resort drop checks. Default: MAX_FILES
+ // (the tests' seam; step 14 always applies MAX_FILES).
+ historyFileLimit?: number;
+ // The design tokens the history pages' style.css is written from. Default:
+ // <repo>/common/styles/tokens.css.
+ tokensFile?: string;
+ // Where the scratch dir is made. Default: paths.sourceScratchDir.
+ scratchRoot?: string;
+ // The environment the children run in (PATH decides which tools). Default:
+ // process.env.
+ env?: NodeJS.ProcessEnv;
+ now?: () => Date;
+ // The home dir the built-in rule scrubs. Default: os.homedir().
+ homeDir?: string;
+ // A checkout with no git repository: "refuse" (the CLI: exit 1) or "empty"
+ // (buildHomepage: the /source page's empty state, exit 0). Either way it
+ // says NO_REPOSITORY and removes an old publish.
+ noRepository?: "refuse" | "empty";
+};
+
+// ── the operator's files ────────────────────────────────────────────────────
+
+export type ScrubRules = {
+ // replace.txt as filter-repo will read it: the built-in rule first, then
+ // the operator's rules in order (comments and blank lines dropped —
+ // filter-repo itself would treat a `#` line as a literal to replace).
+ lines: string[];
+ // Every rule's LITERAL left side: denied, exact case, by implication, with
+ // where it was written (the report's name for it).
+ denied: Array<{ text: string; from: string }>;
+};
+
+export const BUILT_IN_HOME_RULE = "built-in home rule";
+
+/**
+ * The scrub file's text as rules. A line is `lhs==>rhs` (split at the LAST
+ * `==>`, as filter-repo splits it), `literal:lhs==>rhs`, `regex:…==>…` or
+ * `glob:…==>…`; a line with no `==>` is replaced by filter-repo's
+ * `***REMOVED***`. Lines whose first non-blank character is `#` are comments.
+ * A byte-order mark and CRLF endings are dropped first (operatorLines), and
+ * the home dir loses a trailing `/`: either would make a rule — and the
+ * denial it implies — silently match nothing. An empty left side is a
+ * refusal, not a rule filter-repo would skip.
+ */
+export function parseScrubRules(text: string, homeDir: string): ScrubRules {
+ const lines: string[] = [];
+ const denied: ScrubRules["denied"] = [];
+ const home = homeDir.replace(/\/+$/, "");
+ // A home dir of `/` (a container user) would scrub every slash, and one
+ // that IS the replacement would deny the replacement itself.
+ if (home.length > 1 && home !== HOME_REPLACEMENT) {
+ lines.push(`${home}==>${HOME_REPLACEMENT}`);
+ denied.push({ text: home, from: BUILT_IN_HOME_RULE });
+ }
+ operatorLines(text).forEach((line, n) => {
+ const t = line.trim();
+ if (t === "" || t.startsWith("#")) return;
+ const i = line.lastIndexOf("==>");
+ let lhs = i === -1 ? line : line.slice(0, i);
+ const from = `scrub line ${n + 1} lhs`;
+ for (const prefix of ["regex:", "glob:", "literal:"]) {
+ if (lhs.startsWith(prefix)) {
+ if (lhs.length === prefix.length) {
+ throw new SourceRefusal(`scrub line ${n + 1} has an empty left side — it would match nothing and deny nothing`);
+ }
+ if (prefix === "literal:") lhs = lhs.slice(prefix.length);
+ else lhs = "";
+ break;
+ }
+ }
+ if (i === 0) {
+ throw new SourceRefusal(`scrub line ${n + 1} has an empty left side — it would match nothing and deny nothing`);
+ }
+ lines.push(line);
+ if (lhs) denied.push({ text: lhs, from });
+ });
+ return { lines, denied };
+}
+
+export type SourceRules = {
+ scrub: ScrubRules;
+ literals: Literal[];
+ // Changes whenever a rule or a literal does: part of the skip key. Never
+ // published (a hash of the denylist would confirm a guess at it).
+ rulesHash: string;
+};
+
+async function readOperatorFile(file: string, what: string, how: string): Promise<string> {
+ try {
+ return await readFile(file, "utf8");
+ } catch (err) {
+ if ((err as NodeJS.ErrnoException).code === "ENOENT") {
+ throw new SourceRefusal(`no ${what} at ${tildify(file)} — ${how}`);
+ }
+ throw err;
+ }
+}
+
+/** Both operator files, parsed, with the literal list the gate searches for. */
+export async function loadSourceRules(opts: {
+ scrubFile: string;
+ denylistFile: string;
+ homeDir?: string;
+}): Promise<SourceRules> {
+ const scrubText = await readOperatorFile(
+ opts.scrubFile,
+ "scrub rules",
+ "create it (git-filter-repo `lhs==>rhs` lines; the home-directory rule is built in, so it may be empty) or point SOURCE_SCRUB_FILE at one",
+ );
+ const denyText = await readOperatorFile(
+ opts.denylistFile,
+ "denylist",
+ "create it (one literal per line, `i:` for any case; every scrub rule's left side is denied too, so it may be empty) or point SOURCE_DENYLIST_FILE at one",
+ );
+ const scrub = parseScrubRules(scrubText, opts.homeDir ?? os.homedir());
+ const literals = dedupeLiterals([
+ ...parseDenylist(denyText),
+ ...scrub.denied.map((d) => ({ bytes: Buffer.from(d.text, "utf8"), ci: false, from: d.from })),
+ ]);
+ return { scrub, literals, rulesHash: rulesHashOf(scrub.lines, literals, SOURCE_STEP_VERSION) };
+}
+
+/**
+ * The rules hash: the scrub lines, every literal (and whether it is `i:`),
+ * and the step's version — so a fix to the step moves it like a new rule does.
+ */
+export function rulesHashOf(lines: readonly string[], literals: readonly Literal[], step: number): string {
+ return createHash("sha256")
+ .update(
+ JSON.stringify({
+ step,
+ rules: lines,
+ literals: literals.map((l) => `${l.ci ? "i" : "x"}:${l.bytes.toString("hex")}`),
+ }),
+ )
+ .digest("hex");
+}
+
+// ── children ────────────────────────────────────────────────────────────────
+
+type Ctx = {
+ onLog: (line: string) => void;
+ signal: AbortSignal;
+ env: NodeJS.ProcessEnv;
+ // Every echoed or quoted child line is masked with these once they are known.
+ literals: readonly Literal[];
+};
+
+class Cancelled extends Error {}
+
+/**
+ * One child through runChildIntoLog, with a timeout. Returns its combined
+ * output. A non-zero exit or a timeout is a refusal quoting its last lines
+ * (masked); a cancel throws Cancelled.
+ */
+async function run(
+ ctx: Ctx,
+ command: string,
+ args: string[],
+ o: { cwd: string; timeoutMs: number; echo?: boolean; allowFail?: boolean },
+): Promise<{ code: number; out: string[] }> {
+ const out: string[] = [];
+ const timeout = AbortSignal.timeout(o.timeoutMs);
+ const code = await runChildIntoLog(
+ (line) => {
+ out.push(line);
+ if (o.echo) ctx.onLog(`[source] ${maskLiterals(line, ctx.literals)}`);
+ },
+ AbortSignal.any([ctx.signal, timeout]),
+ { command, args, cwd: o.cwd, env: ctx.env },
+ );
+ if (ctx.signal.aborted) throw new Cancelled();
+ // `git --git-dir <path> repack …` is named by its verb, not the path.
+ const what = `${command} ${(args[0] === "--git-dir" ? args.slice(2, 3) : args.slice(0, 2)).join(" ")}`;
+ if (timeout.aborted) {
+ throw new SourceRefusal(`${what} timed out after ${Math.round(o.timeoutMs / 1000)} s`);
+ }
+ if (code !== 0 && !o.allowFail) {
+ const tail = out.slice(-3).map((l) => maskLiterals(l, ctx.literals)).join(" / ");
+ throw new SourceRefusal(`${what} exited ${code}${tail ? `: ${tail}` : ""}`);
+ }
+ return { code, out };
+}
+
+const lastLine = (out: string[]) => (out.filter((l) => l.trim()).at(-1) ?? "").trim();
+const OID = /^[0-9a-f]{40}(?:[0-9a-f]{24})?$/;
+
+async function revParse(ctx: Ctx, gitDir: string, rev: string): Promise<string> {
+ const { out } = await run(ctx, "git", ["--git-dir", gitDir, "rev-parse", "--verify", rev], {
+ cwd: gitDir,
+ timeoutMs: 30_000,
+ });
+ const oid = lastLine(out);
+ if (!OID.test(oid)) throw new SourceRefusal(`git rev-parse ${rev}: not an object id`);
+ return oid;
+}
+
+export type FilterRepoChoice = { argv: string[]; label: string; version: string };
+
+/**
+ * Which git-filter-repo runs: an installed `git filter-repo`, else `pipx run`
+ * of the pinned version (network on first use), else a refusal with the
+ * install line.
+ */
+export async function resolveFilterRepo(opts: {
+ env?: NodeJS.ProcessEnv;
+ signal?: AbortSignal;
+ onLog?: (line: string) => void;
+}): Promise<FilterRepoChoice> {
+ const ctx: Ctx = {
+ onLog: opts.onLog ?? (() => {}),
+ signal: opts.signal ?? new AbortController().signal,
+ env: cleanGitEnv(opts.env ?? process.env),
+ literals: [],
+ };
+ const cwd = os.tmpdir();
+ const installed = await run(ctx, "git", ["filter-repo", "--version"], {
+ cwd,
+ timeoutMs: 30_000,
+ allowFail: true,
+ });
+ if (installed.code === 0) {
+ return { argv: ["git", "filter-repo"], label: "git filter-repo", version: lastLine(installed.out) };
+ }
+ if (!onPath("pipx", ctx.env.PATH)) {
+ throw new SourceRefusal(
+ `git-filter-repo is not installed and pipx is not on PATH — install it once: \`${FILTER_REPO_INSTALL}\` (pipx comes from your OS's packages)`,
+ );
+ }
+ const argv = ["pipx", "run", "--spec", FILTER_REPO_PIPX_SPEC, "git-filter-repo"];
+ const viaPipx = await run(ctx, argv[0], [...argv.slice(1), "--version"], {
+ cwd,
+ timeoutMs: 300_000,
+ allowFail: true,
+ });
+ if (viaPipx.code !== 0) {
+ throw new SourceRefusal(
+ `\`pipx run --spec ${FILTER_REPO_PIPX_SPEC}\` failed (exit ${viaPipx.code}; it needs the network on first use) — install it once: \`${FILTER_REPO_INSTALL}\``,
+ );
+ }
+ return {
+ argv,
+ label: `pipx run --spec ${FILTER_REPO_PIPX_SPEC} git-filter-repo`,
+ version: lastLine(viaPipx.out),
+ };
+}
+
+// ── helpers ─────────────────────────────────────────────────────────────────
+
+function terminalLog(line: string): void {
+ process.stdout.write(line.endsWith("\n") ? line : `${line}\n`);
+}
+
+function sha256File(file: string): Promise<string> {
+ return new Promise((resolve, reject) => {
+ const h = createHash("sha256");
+ createReadStream(file)
+ .on("data", (c) => h.update(c))
+ .on("error", reject)
+ .on("end", () => resolve(h.digest("hex")));
+ });
+}
+
+async function walkFiles(dir: string): Promise<Array<{ rel: string; bytes: number }>> {
+ const out: Array<{ rel: string; bytes: number }> = [];
+ const walk = async (rel: string) => {
+ for (const ent of await readdir(path.join(dir, rel), { withFileTypes: true })) {
+ const r = rel ? `${rel}/${ent.name}` : ent.name;
+ if (ent.isDirectory()) await walk(r);
+ else out.push({ rel: r, bytes: (await stat(path.join(dir, r))).size });
+ }
+ };
+ await walk("");
+ return out;
+}
+
+const mb = (bytes: number) => (bytes / (1024 * 1024)).toFixed(1);
+
+/**
+ * Why `files` may not be published on Pages, as one sentence — or null. The
+ * step's limits sit inside the host's: 15,000 files of its 20,000 per
+ * deployment (the rest of the site needs room), 24 MiB of its 25 MiB per file.
+ */
+export function limitProblem(files: ReadonlyArray<{ rel: string; bytes: number }>): string | null {
+ if (files.length > MAX_FILES) {
+ return `${files.length} files to publish, over the step's limit of ${MAX_FILES} (Pages allows 20,000 per deployment)`;
+ }
+ const big = files.find((f) => f.bytes > MAX_FILE_BYTES);
+ if (big) {
+ return `${big.rel} is ${mb(big.bytes)} MiB, over the step's limit of ${mb(MAX_FILE_BYTES)} MiB (Pages allows 25 MiB per file)`;
+ }
+ return null;
+}
+
+/** Where publishSource writes, for a given checkout. */
+export function sourcePublicDir(paths: Paths, override?: string): string {
+ return override ?? process.env.HOMEPAGE_PUBLIC_DIR ?? path.join(paths.monorepoRoot, "homepage", "public");
+}
+
+// The skip key, kept BESIDE the public dir, never in it: it holds the rules
+// hash, and public/ is deployed. It is also what `deployHomepage` checks
+// homepage/out's source against (publishedSourceProblem).
+function statePath(publicDir: string): string {
+ return path.join(path.dirname(publicDir), ".source-publish.json");
+}
+
+type PublishState = {
+ sourceCommit: string;
+ mirrorHead: string;
+ // The rules, the literals and SOURCE_STEP_VERSION.
+ rulesHash: string;
+ // "<label> <version>": an upgrade may rewrite every id, so it re-publishes.
+ filterRepo: string;
+ // gitleaksIdentity(): a newer gitleaks may find what the old one did not,
+ // so it re-publishes, and an out/ it never scanned does not deploy.
+ gitleaks: string;
+ // sourceDigest() over every published file: an out/ (or public/) holding
+ // another publish's mirror or tree beside this manifest does not match.
+ contentDigest: string;
+ // stagitIdentity(): "absent", or which stagit rendered the history. An
+ // install (or an upgrade) re-publishes.
+ stagit: string;
+ // The history pages as audited (historyDigest), or null when none were
+ // published. A null beside a stagit that is present (a render that failed,
+ // pages over the limits) never skips: the next build tries again.
+ history: { files: number; digest: string } | null;
+};
+
+/**
+ * One digest over every file a publish puts on the site, under `root` laid
+ * out as public/ and out/ both are: `source/archilyzer.git/**`,
+ * `source/tree/**`, `source/git/**`, `source/manifest.json`,
+ * `downloads/archilyzer-source.tar.gz` and `downloads/snapshot.json`. Each
+ * file contributes its relative path, size and sha256 (streamed), in sorted
+ * path order; a symlink or a missing piece contributes a marker, so it can
+ * only differ. The /source PAGE and anything else Next writes beside them are
+ * not part of it.
+ */
+export async function sourceDigest(root: string): Promise<string> {
+ return digestOf(root, [
+ `source/${MIRROR_DIR}`,
+ "source/tree",
+ `source/${HISTORY_DIR}`,
+ "source/manifest.json",
+ `downloads/${TARBALL_NAME}`,
+ "downloads/snapshot.json",
+ ]);
+}
+
+/**
+ * The same digest over the history pages alone (`source/git/**`): the
+ * manifest's `history.sha256`, and what the deploy check recomputes over
+ * out/ to name a history that is not the audited one.
+ */
+export async function historyDigest(root: string): Promise<string> {
+ return digestOf(root, [`source/${HISTORY_DIR}`]);
+}
+
+async function digestOf(root: string, tops: readonly string[]): Promise<string> {
+ const entries: string[] = [];
+ const walk = async (rel: string): Promise<void> => {
+ const abs = path.join(root, rel);
+ const st = await lstat(abs).catch(() => null);
+ if (!st) {
+ entries.push(`${rel}\0missing`);
+ } else if (st.isSymbolicLink()) {
+ entries.push(`${rel}\0symlink`);
+ } else if (st.isDirectory()) {
+ for (const name of await readdir(abs)) await walk(`${rel}/${name}`);
+ } else {
+ entries.push(`${rel}\0${st.size}\0${await sha256File(abs)}`);
+ }
+ };
+ for (const top of tops) await walk(top);
+ entries.sort((a, b) => (a < b ? -1 : a > b ? 1 : 0));
+ const h = createHash("sha256");
+ for (const e of entries) h.update(`${e}\n`);
+ return h.digest("hex");
+}
+
+/**
+ * Which gitleaks the audit runs: "skipped" (none asked for), "absent" (not on
+ * PATH), else its version line and the sha256 of its binary — the version
+ * line alone is not enough (a distribution build can print the same line for
+ * every release).
+ */
+export async function gitleaksIdentity(bin: string | null, env: NodeJS.ProcessEnv): Promise<string> {
+ if (bin === null) return "skipped";
+ const found = onPath(bin, env.PATH);
+ if (!found) return "absent";
+ const version = await new Promise<string>((resolve) => {
+ execFile(found, ["version"], { env, timeout: 10_000 }, (_err, stdout, stderr) =>
+ resolve(String(stdout || stderr).trim().split("\n")[0] ?? ""),
+ );
+ });
+ const sha = await sha256File(await realpath(found));
+ return `${version || "unknown version"} (${sha.slice(0, 12)})`;
+}
+
+async function readJson(file: string): Promise<unknown> {
+ try {
+ return JSON.parse(await readFile(file, "utf8"));
+ } catch {
+ return null;
+ }
+}
+
+/** The last published manifest in `publicDir`, or null. */
+export async function readPublishedManifest(publicDir: string): Promise<SourceManifest | null> {
+ return parseSourceManifest(await readJson(path.join(publicDir, "source", "manifest.json")));
+}
+
+/** Said, and nothing mirrored, in a checkout without git history. */
+export const NO_REPOSITORY =
+ "no git repository here; nothing to mirror — the /source page will show its empty state";
+
+/**
+ * Remove a publish from `publicDir`: the manifest FIRST (the page never
+ * describes a half-removed tree), the skip key, the mirror, the tree and the
+ * history pages, the tarball and snapshot.json. Link-safe like the install: a
+ * linked `source/` or `downloads/` (a worktree's, into the primary) goes as a
+ * link, its target untouched. True when there was something to remove.
+ */
+export async function removePublishedSource(publicDir: string): Promise<boolean> {
+ const pubSource = path.join(publicDir, "source");
+ const pubDownloads = path.join(publicDir, "downloads");
+ const had =
+ existsSync(path.join(pubSource, "manifest.json")) ||
+ existsSync(path.join(pubSource, MIRROR_DIR)) ||
+ existsSync(path.join(pubSource, HISTORY_DIR)) ||
+ existsSync(path.join(pubDownloads, TARBALL_NAME));
+ await rm(path.join(pubSource, "manifest.json"), { force: true });
+ await rm(statePath(publicDir), { force: true });
+ // fs.rm reads the path with lstat: a linked public/source goes as a link.
+ await rm(pubSource, { recursive: true, force: true });
+ const linked = await lstat(pubDownloads).then((s) => s.isSymbolicLink(), () => false);
+ if (linked) {
+ await rm(pubDownloads, { force: true });
+ } else {
+ await rm(path.join(pubDownloads, "snapshot.json"), { force: true });
+ await rm(path.join(pubDownloads, TARBALL_NAME), { force: true });
+ }
+ return had;
+}
+
+// The checkout's git common dir, or null when there is no repository here (or
+// no git at all — a docker runtime, a tarball install).
+async function commonDir(ctx: Ctx, cwd: string): Promise<string | null> {
+ const r = await run(ctx, "git", ["rev-parse", "--path-format=absolute", "--git-common-dir"], {
+ cwd,
+ timeoutMs: 30_000,
+ allowFail: true,
+ });
+ if (r.code === 0) return lastLine(r.out);
+ if (!onPath("git", ctx.env.PATH) || r.out.some((l) => /not a git repository/i.test(l))) return null;
+ const tail = r.out.slice(-2).map((l) => maskLiterals(l, ctx.literals)).join(" / ");
+ throw new SourceRefusal(`git rev-parse exited ${r.code}${tail ? `: ${tail}` : ""}`);
+}
+
+// The real path of `p`, or of its deepest existing ancestor with the rest
+// appended: a scratch root may not exist yet, and a symlink must not hide
+// where it lands.
+async function landsAt(p: string): Promise<string> {
+ const abs = path.resolve(p);
+ let head = abs;
+ const rest: string[] = [];
+ for (;;) {
+ try {
+ return path.join(await realpath(head), ...rest);
+ } catch {
+ const up = path.dirname(head);
+ if (up === head) return abs;
+ rest.unshift(path.basename(head));
+ head = up;
+ }
+ }
+}
+
+const within = (child: string, parent: string) =>
+ child === parent || child.startsWith(parent.endsWith(path.sep) ? parent : parent + path.sep);
+
+/**
+ * Why `scratchRoot` may not hold the scratch dir, or null. Inside the checkout
+ * or the public dir, a kept scratch (`--keep-scratch`) could be committed or
+ * published — and it held replace.txt, the scrub rules in plain text.
+ */
+export async function scratchRootProblem(
+ scratchRoot: string,
+ places: ReadonlyArray<[string, string]>,
+): Promise<string | null> {
+ const what = await insideOf(scratchRoot, places);
+ return what
+ ? `the scratch root ${tildify(scratchRoot)} (ARCHILYZER_SOURCE_SCRATCH) is inside ${what}, where a kept scratch dir could be committed or published — point it outside`
+ : null;
+}
+
+// Which of `places` ([name, dir]) `p` lands inside, through symlinks, or null.
+async function insideOf(p: string, places: ReadonlyArray<[string, string]>): Promise<string | null> {
+ const at = await landsAt(p);
+ for (const [what, dir] of places) {
+ if (within(at, await landsAt(dir))) return what;
+ }
+ return null;
+}
+
+/**
+ * The history's render cache for this publish, or null: none for `--check`
+ * (it writes nothing outside its scratch), none when asked for none, and none
+ * — with a line saying so — when it would land inside the checkout or the
+ * public dir, where it could be committed or published.
+ */
+export async function historyCacheFor(
+ opts: Pick<SourcePublishOpts, "check" | "historyCacheDir">,
+ paths: Paths,
+ publicDir: string,
+ onLog: (line: string) => void,
+): Promise<string | null> {
+ if (opts.check) return null;
+ const dir = opts.historyCacheDir === undefined ? (paths.sourceHistoryCacheDir ?? null) : opts.historyCacheDir;
+ if (!dir) return null;
+ const what = await insideOf(dir, [
+ ["the checkout", paths.monorepoRoot],
+ ["the public dir", publicDir],
+ ]);
+ if (what) {
+ onLog(`[source] history: the render cache ${tildify(dir)} (XDG_CACHE_HOME) is inside ${what}; rendering without it`);
+ return null;
+ }
+ return dir;
+}
+
+// ── the step ────────────────────────────────────────────────────────────────
+
+/**
+ * Publish the source (see the header). Returns 0 when published, skipped or
+ * checked, 1 when refused or cancelled; the refusal's reason (and the audit's
+ * report) is in the log. Throws only on a bug or an I/O failure.
+ *
+ * A REFUSAL WITHDRAWS THE PREVIOUS PUBLISH. Once the rules are loaded, any
+ * outcome but success — an audit hit, a limit, a missing tool, a cancel, a
+ * crash — removes what the last publish left (removePublishedSource): it was
+ * audited under rules that may not be today's, and a refusal is the best
+ * evidence that they are not. `--check` writes nothing, this included.
+ *
+ * `noRepository: "empty"` (buildHomepage) turns a checkout with no git
+ * repository into an empty /source page and 0; the CLI's default refuses.
+ */
+export async function publishSource(opts: SourcePublishOpts = {}): Promise<number> {
+ const paths = opts.paths ?? getPaths();
+ const onLog = opts.onLog ?? terminalLog;
+ const signal = opts.signal ?? new AbortController().signal;
+ const ctx: Ctx = { onLog, signal, env: cleanGitEnv(opts.env ?? process.env), literals: [] };
+ const publicDir = sourcePublicDir(paths, opts.publicDir);
+ const progress = { rulesLoaded: false, historyCache: null as string | null };
+ const withdraw = async () => {
+ if (!progress.rulesLoaded || opts.check) return;
+ if (await removePublishedSource(publicDir)) {
+ onLog(
+ "[source] the previous publish was WITHDRAWN (mirror, tree, history, tarball): it was audited under rules that may not be today's. /source shows its empty state until a publish passes.",
+ );
+ }
+ // The history's render cache was rendered under those rules too. It is
+ // never why a withdrawal fails: one line, and the refusal stands.
+ if (progress.historyCache) {
+ try {
+ await dropHistoryCache(progress.historyCache);
+ } catch (err) {
+ const why = (err as NodeJS.ErrnoException)?.code ?? (err as Error)?.message ?? String(err);
+ onLog(
+ maskLiterals(
+ `[source] the history's render cache ${tildify(progress.historyCache)} could not be removed (${why}); remove it by hand`,
+ ctx.literals,
+ ),
+ );
+ }
+ }
+ };
+ let code: number;
+ try {
+ code = await publish(opts, paths, ctx, publicDir, progress);
+ } catch (err) {
+ if (err instanceof Cancelled || signal.aborted) {
+ onLog("[source] cancelled — nothing published");
+ code = 1;
+ } else if (err instanceof SourceRefusal) {
+ // A refusal may name a tree path or quote a child's stderr: masked, so
+ // a literal spanning path components (`a/b`) is never printed.
+ onLog(`[source] REFUSED: ${maskLiterals(err.message, ctx.literals)}`);
+ code = 1;
+ } else {
+ await withdraw().catch(() => {});
+ throw err;
+ }
+ }
+ if (code !== 0) await withdraw();
+ return code;
+}
+
+async function publish(
+ opts: SourcePublishOpts,
+ paths: Paths,
+ ctx: Ctx,
+ publicDir: string,
+ progress: { rulesLoaded: boolean; historyCache: string | null },
+): Promise<number> {
+ const { onLog } = ctx;
+ const started = Date.now();
+ const pubSource = path.join(publicDir, "source");
+ const pubDownloads = path.join(publicDir, "downloads");
+
+ // 1. The private main — or no repository at all (L8: the docker runtime,
+ // a tarball install), where nothing can be mirrored and nothing can leak.
+ const sourceRepo = opts.sourceRepo ?? (await commonDir(ctx, paths.monorepoRoot));
+ if (sourceRepo === null) {
+ onLog(`[source] ${NO_REPOSITORY}`);
+ if (opts.check) return 1;
+ if (await removePublishedSource(publicDir)) onLog("[source] the previous publish was removed.");
+ return opts.noRepository === "empty" ? 0 : 1;
+ }
+ const sourceCommit = await revParse(ctx, sourceRepo, `refs/heads/${SOURCE_BRANCH}^{commit}`);
+
+ // 2. The operator's rules. From here on, a failure withdraws the last publish.
+ const rules = await loadSourceRules({
+ scrubFile: opts.scrubFile ?? paths.sourceScrubFile,
+ denylistFile: opts.denylistFile ?? paths.sourceDenylistFile,
+ homeDir: opts.homeDir,
+ });
+ ctx.literals = rules.literals;
+ progress.rulesLoaded = true;
+
+ // 3. The tools (the filter-repo version is part of the skip key).
+ const filterRepo: FilterRepoChoice =
+ opts.filterRepo && opts.filterRepo.length > 0
+ ? { argv: opts.filterRepo, label: opts.filterRepo.join(" "), version: "(given)" }
+ : await resolveFilterRepo({ env: ctx.env, signal: ctx.signal });
+ const filterRepoId = `${filterRepo.label} ${filterRepo.version}`;
+ const gitleaksBin = opts.gitleaks === undefined ? "gitleaks" : opts.gitleaks;
+ const gitleaksId = await gitleaksIdentity(gitleaksBin, ctx.env);
+ // stagit is optional: absent, the source is published without its history.
+ const stagit =
+ opts.stagit === undefined
+ ? resolveStagit(paths.stagitBin ?? "stagit", ctx.env)
+ : opts.stagit && existsSync(opts.stagit)
+ ? opts.stagit
+ : null;
+ const stagitId = await stagitIdentity(stagit);
+
+ // 4. Nothing changed: skip. The key is main, the rules (with the step's
+ // version), the tools, and the published files themselves.
+ if (!opts.force && !opts.check) {
+ const manifest = await readPublishedManifest(publicDir);
+ const state = (await readJson(statePath(publicDir))) as Partial<PublishState> | null;
+ if (
+ manifest?.sourceCommit === sourceCommit &&
+ state?.sourceCommit === sourceCommit &&
+ state.rulesHash === rules.rulesHash &&
+ state.filterRepo === filterRepoId &&
+ state.gitleaks === gitleaksId &&
+ state.stagit === stagitId &&
+ (stagitId === "absent" ? state.history === null : !!state.history) &&
+ state.mirrorHead === manifest.mirrorHead &&
+ existsSync(path.join(pubSource, MIRROR_DIR, "info", "refs")) &&
+ existsSync(path.join(pubDownloads, TARBALL_NAME)) &&
+ state.contentDigest === (await sourceDigest(publicDir))
+ ) {
+ onLog(`[source] up to date at ${sourceCommit.slice(0, 12)}; skipping (--force to rebuild)`);
+ return 0;
+ }
+ }
+ if (existsSync(path.join(pubSource, "index.html"))) {
+ throw new SourceRefusal(
+ `${tildify(path.join(pubSource, "index.html"))} exists and would replace the /source/ page — remove it`,
+ );
+ }
+ const gitVersion = lastLine((await run(ctx, "git", ["--version"], { cwd: os.tmpdir(), timeoutMs: 30_000 })).out)
+ .replace(/^git version /, "");
+ onLog(`[source] main ${sourceCommit.slice(0, 12)}; git ${gitVersion}; ${filterRepoId}`);
+
+ const scratchRoot = opts.scratchRoot ?? paths.sourceScratchDir;
+ const badRoot = await scratchRootProblem(scratchRoot, [
+ ["the checkout", paths.monorepoRoot],
+ ["the public dir", publicDir],
+ ]);
+ if (badRoot) throw new SourceRefusal(badRoot);
+ // From here a refusal removes the history's render cache too.
+ const historyCache = await historyCacheFor(opts, paths, publicDir, (l) => onLog(maskLiterals(l, ctx.literals)));
+ progress.historyCache = historyCache;
+ await mkdir(scratchRoot, { recursive: true });
+ const scratch = await mkdtemp(path.join(scratchRoot, "archilyzer-source-"));
+ const replace = path.join(scratch, "replace.txt");
+ try {
+ // Named as the published mirror is: stagit names the repository after its
+ // directory ("archilyzer", the `.git` dropped).
+ const bare = path.join(scratch, MIRROR_DIR);
+
+ // 5. A fresh bare clone of main alone. --no-local: through upload-pack, not
+ // a copy of the object dir (which would carry every loose leftover).
+ await run(
+ ctx,
+ "git",
+ ["clone", "--no-local", "--bare", "--single-branch", "--no-tags", "--branch", SOURCE_BRANCH, "--quiet", sourceRepo, bare],
+ { cwd: scratch, timeoutMs: 120_000 },
+ );
+ await run(ctx, "git", ["--git-dir", bare, "remote", "remove", "origin"], { cwd: bare, timeoutMs: 30_000 });
+
+ // 6. The rewrite. --force: filter-repo wants a fresh clone with an origin.
+ await writeFile(replace, rules.scrub.lines.join("\n") + "\n", { mode: 0o600 });
+ onLog(`[source] rewriting history (${rules.scrub.lines.length} scrub rule${rules.scrub.lines.length === 1 ? "" : "s"})…`);
+ await run(
+ ctx,
+ filterRepo.argv[0],
+ [
+ ...filterRepo.argv.slice(1),
+ "--force",
+ "--quiet",
+ "--replace-refs",
+ "delete-no-add",
+ "--replace-text",
+ replace,
+ "--replace-message",
+ replace,
+ ],
+ { cwd: bare, timeoutMs: 900_000, echo: true },
+ );
+ // commit-map / ref-map hold the PRIVATE ids.
+ await rm(path.join(bare, "filter-repo"), { recursive: true, force: true });
+
+ // 7. Packed for dumb HTTP.
+ const g = (args: string[], timeoutMs = 60_000) =>
+ run(ctx, "git", ["--git-dir", bare, ...args], { cwd: bare, timeoutMs });
+ await g(["repack", "-a", "-d", "-q", `--max-pack-size=${PACK_SIZE}`], 300_000);
+ await g(["prune-packed"]);
+ await g(["pack-refs", "--all"]);
+ await g(["update-server-info"]);
+ const counts = (await g(["count-objects", "-v"])).out;
+ const loose = Number(/^count:\s*(\d+)/m.exec(counts.join("\n"))?.[1] ?? NaN);
+ if (loose !== 0) throw new SourceRefusal(`the repacked mirror still has ${loose} loose objects`);
+ const packsList = await readFile(path.join(bare, "objects", "info", "packs"), "utf8").catch(() => "");
+ const packs = packsList.split("\n").filter((l) => l.startsWith("P ")).length;
+ if (packs === 0) throw new SourceRefusal("the repacked mirror lists no packs in objects/info/packs");
+ const refs = (await g(["for-each-ref", "--format=%(refname)"])).out.filter((l) => l.trim());
+ if (refs.length !== 1 || refs[0] !== `refs/heads/${SOURCE_BRANCH}`) {
+ throw new SourceRefusal(`the mirror must hold refs/heads/${SOURCE_BRANCH} alone, and holds ${refs.length} refs`);
+ }
+ const head = (await readFile(path.join(bare, "HEAD"), "utf8")).trim();
+ if (head !== `ref: refs/heads/${SOURCE_BRANCH}`) {
+ throw new SourceRefusal(`the mirror's HEAD is not refs/heads/${SOURCE_BRANCH}`);
+ }
+
+ // 8. What it became.
+ const mirrorHead = await revParse(ctx, bare, `refs/heads/${SOURCE_BRANCH}`);
+ const subject = lastLine((await g(["log", "-1", "--format=%s", `refs/heads/${SOURCE_BRANCH}`])).out);
+
+ // 9. THE GATE, over every object.
+ onLog(`[source] auditing ${mirrorHead.slice(0, 12)} for ${rules.literals.length} denied literals…`);
+ const audit = await auditBare(bare, rules.literals, {
+ scratch,
+ onLog,
+ signal: ctx.signal,
+ gitleaks: gitleaksBin,
+ env: ctx.env,
+ });
+ const scrubFile = opts.scrubFile ?? paths.sourceScrubFile;
+ if (audit.hits.length > 0) {
+ for (const l of formatAuditReport(audit, rules.literals, { scrubFile })) onLog(l);
+ return 1;
+ }
+
+ const now = opts.now?.() ?? new Date();
+ const generatedAt = now.toISOString();
+ const stage = path.join(scratch, "stage");
+ const stageSource = path.join(stage, "source");
+ const stageMirror = path.join(stageSource, MIRROR_DIR);
+ const stageTree = path.join(stageSource, "tree");
+ const stageDownloads = path.join(stage, "downloads");
+ await mkdir(stageTree, { recursive: true });
+ await mkdir(stageDownloads, { recursive: true });
+
+ // 10. The raw tree, with its directory pages.
+ const treeTar = path.join(scratch, "tree.tar");
+ await g(["archive", "--format=tar", "-o", treeTar, `refs/heads/${SOURCE_BRANCH}`], 120_000);
+ await run(ctx, "tar", ["-xf", treeTar, "-C", stageTree], { cwd: scratch, timeoutMs: 120_000 });
+ await rm(treeTar, { force: true });
+ const tree = await writeTreeIndexes(stageTree, { mirrorHead, generatedAt });
+
+ // 11. The tarball and its sidecar (the Snapshot shape the Downloads page reads).
+ const tarball = path.join(stageDownloads, TARBALL_NAME);
+ await g(
+ ["archive", "--format=tar.gz", "-9", "--prefix=archilyzer/", "-o", tarball, `refs/heads/${SOURCE_BRANCH}`],
+ 120_000,
+ );
+ const tarBytes = (await stat(tarball)).size;
+ const tarSha = await sha256File(tarball);
+ const snapshotText =
+ JSON.stringify(
+ { generatedAt, commit: mirrorHead, subject, bytes: tarBytes, sha256: tarSha },
+ null,
+ 2,
+ ) + "\n";
+ await writeFile(path.join(stageDownloads, "snapshot.json"), snapshotText);
+
+ // 12. The mirror, from an ALLOWLIST: never config (it names the clone's
+ // origin path), hooks/, description, logs/, filter-repo/, *.rev, *.bitmap.
+ // refs/heads/<branch> is written beside packed-refs so the directory is a
+ // git dir to git itself too (a file:// clone, `source audit`): git wants a
+ // refs/ directory, and an empty one would not survive the deploy.
+ await mkdir(path.join(stageMirror, "info"), { recursive: true });
+ await mkdir(path.join(stageMirror, "objects", "info"), { recursive: true });
+ await mkdir(path.join(stageMirror, "objects", "pack"), { recursive: true });
+ await mkdir(path.join(stageMirror, "refs", "heads"), { recursive: true });
+ for (const f of ["HEAD", "packed-refs", "info/refs", "objects/info/packs"]) {
+ await cp(path.join(bare, f), path.join(stageMirror, f));
+ }
+ await writeFile(path.join(stageMirror, "refs", "heads", SOURCE_BRANCH), `${mirrorHead}\n`);
+ for (const f of await readdir(path.join(bare, "objects", "pack"))) {
+ if (/^pack-[0-9a-f]+\.(pack|idx)$/.test(f)) {
+ await cp(path.join(bare, "objects", "pack", f), path.join(stageMirror, "objects", "pack", f));
+ }
+ }
+ const mirrorFiles = await walkFiles(stageMirror);
+
+ // 12b. The history pages (stagit), rendered from the audited mirror into
+ // the stage, so the file gate (13) reads every page. Without stagit — or
+ // when the render fails, or would break the host's limits — one line says
+ // so and the rest is published without them: never a failed build.
+ const stageHistoryDir = path.join(stageSource, HISTORY_DIR);
+ let history: SourceHistory | null = null;
+ if (!stagit) {
+ const why = (paths.stagitBin ?? "stagit").includes("/")
+ ? "STAGIT_BIN names no executable"
+ : "not on PATH, not in ~/.local/bin";
+ onLog(
+ `[source] stagit not found (${why}) — publishing without the history pages (/source/${HISTORY_DIR}/); install it once: ${STAGIT_INSTALL}`,
+ );
+ } else {
+ const commits = Number(lastLine((await g(["rev-list", "--count", `refs/heads/${SOURCE_BRANCH}`])).out));
+ history = await stageHistoryPages({
+ ctx,
+ opts,
+ paths,
+ stagit,
+ stagitId,
+ bare,
+ mirrorHead,
+ commits,
+ scratch,
+ historyCache,
+ rulesHash: rules.rulesHash,
+ filterRepoId,
+ stage,
+ dest: stageHistoryDir,
+ // The raw tree's files, as tracked: links to any other become text.
+ treeFiles: new Set((await walkFiles(stageTree)).map((f) => f.rel)),
+ // Everything staged so far, and the manifest still to come.
+ stagedSoFar: (await walkFiles(stage)).length + 1,
+ });
+ }
+
+ // The manifest, staged with the rest so the file sweep reads it too. It
+ // carries no rules hash and no literal count: `plans/` publishes which
+ // literals the plan put in the files, so either would say whether the
+ // operator added private ones — and the hash would confirm a guess.
+ const staged = await walkFiles(stage);
+ const manifest: SourceManifest = {
+ version: SOURCE_MANIFEST_VERSION,
+ generatedAt,
+ branch: SOURCE_BRANCH,
+ sourceCommit,
+ mirrorHead,
+ subject,
+ files: staged.length,
+ bytes: staged.reduce((n, f) => n + f.bytes, 0),
+ mirror: {
+ files: mirrorFiles.length,
+ bytes: mirrorFiles.reduce((n, f) => n + f.bytes, 0),
+ packs,
+ },
+ tree,
+ tarball: { href: TARBALL_HREF, bytes: tarBytes, sha256: tarSha },
+ cloneUrl: CLONE_URL,
+ treeHref: TREE_HREF,
+ audit: {
+ objects: audit.objects,
+ commits: audit.commits,
+ gitleaks: audit.gitleaks === "clean" ? "clean" : "skipped",
+ },
+ tools: { git: gitVersion, filterRepo: filterRepoId },
+ ...(history ? { history } : {}),
+ };
+ const manifestText = JSON.stringify(manifest, null, 2) + "\n";
+ await writeFile(path.join(stageSource, "manifest.json"), manifestText);
+ // Every published file, as it will land: the state carries it, and the
+ // skip and the deploy check recompute it over public/ and out/.
+ const contentDigest = await sourceDigest(stage);
+
+ // 13. THE GATE, over every staged file and path (packs excepted: step 9
+ // read their objects).
+ await auditFiles(stage, rules.literals, { result: audit });
+ if (audit.hits.length > 0) {
+ for (const l of formatAuditReport(audit, rules.literals, { scrubFile })) onLog(l);
+ return 1;
+ }
+ for (const l of formatAuditReport(audit, rules.literals, { scrubFile })) onLog(l);
+
+ // 14. The host's limits.
+ const all = await walkFiles(stage);
+ const tooMuch = limitProblem(all);
+ if (tooMuch) throw new SourceRefusal(maskLiterals(tooMuch, rules.literals));
+ const totalBytes = all.reduce((n, f) => n + f.bytes, 0);
+ const summary =
+ `main ${sourceCommit.slice(0, 12)} as ${mirrorHead.slice(0, 12)}: ${all.length} files, ${mb(totalBytes)} MB ` +
+ `(mirror ${packs} pack${packs === 1 ? "" : "s"}, tree ${tree.dirs} dirs, ` +
+ `${history ? `history ${history.commits < history.total ? `the latest ${history.commits} of ${history.total}` : history.commits} commits in ${history.files} files` : "no history"}), ` +
+ `tarball ${mb(tarBytes)} MB sha256 ${tarSha.slice(0, 12)}`;
+
+ // 15. --check writes nothing.
+ if (opts.check) {
+ onLog(`[source] check passed — would publish ${summary}; nothing written (${elapsed(started)})`);
+ return 0;
+ }
+
+ // 16. Install, link-safe. The manifest goes first and comes back last: a
+ // crash mid-copy leaves the page's empty state, never a manifest over a
+ // half-written tree.
+ await ownDir(pubSource);
+ await rm(path.join(pubSource, "manifest.json"), { force: true });
+ await rm(path.join(pubSource, MIRROR_DIR), { recursive: true, force: true });
+ await rm(path.join(pubSource, "tree"), { recursive: true, force: true });
+ await rm(path.join(pubSource, HISTORY_DIR), { recursive: true, force: true });
+ await cp(stageMirror, path.join(pubSource, MIRROR_DIR), { recursive: true });
+ await cp(stageTree, path.join(pubSource, "tree"), { recursive: true });
+ if (history) await cp(stageHistoryDir, path.join(pubSource, HISTORY_DIR), { recursive: true });
+ await ownDir(pubDownloads);
+ await copyPublicFile(tarball, path.join(pubDownloads, TARBALL_NAME));
+ await writePublicFile(path.join(pubDownloads, "snapshot.json"), snapshotText);
+ const state: PublishState = {
+ sourceCommit,
+ mirrorHead,
+ rulesHash: rules.rulesHash,
+ filterRepo: filterRepoId,
+ gitleaks: gitleaksId,
+ contentDigest,
+ stagit: stagitId,
+ history: history ? { files: history.files, digest: history.sha256 } : null,
+ };
+ await writeFile(statePath(publicDir), JSON.stringify(state, null, 2) + "\n");
+ await writePublicFile(path.join(pubSource, "manifest.json"), manifestText);
+
+ // 17.
+ onLog(`[source] published ${summary} (${elapsed(started)})`);
+ return 0;
+ } finally {
+ if (opts.keepScratch) {
+ // The scrub rules in plain text never outlive the run.
+ await rm(replace, { force: true });
+ onLog(`[source] scratch kept at ${maskLiterals(tildify(scratch), ctx.literals)} (replace.txt, the scrub rules, deleted)`);
+ } else {
+ await rm(scratch, { recursive: true, force: true });
+ }
+ }
+}
+
+function elapsed(started: number): string {
+ return `${Math.round((Date.now() - started) / 1000)} s`;
+}
+
+/**
+ * Step 12b: render the history pages into `a.dest` (stage/source/git) and say
+ * what they are, or say why not and leave no `dest` — a WARNING, never a
+ * refusal: the tokens unreadable, stagit failing, or pages that would break
+ * the host's limits (the step's, which step 14 applies to the whole publish).
+ * A cancel still cancels.
+ */
+async function stageHistoryPages(a: {
+ ctx: Ctx;
+ opts: SourcePublishOpts;
+ paths: Paths;
+ stagit: string;
+ stagitId: string;
+ bare: string;
+ mirrorHead: string;
+ commits: number;
+ scratch: string;
+ historyCache: string | null;
+ rulesHash: string;
+ filterRepoId: string;
+ stage: string;
+ dest: string;
+ treeFiles: ReadonlySet<string>;
+ stagedSoFar: number;
+}): Promise<SourceHistory | null> {
+ const { ctx } = a;
+ const started = Date.now();
+ const where = `/source/${HISTORY_DIR}/`;
+ const without = async (why: string): Promise<null> => {
+ await rm(a.dest, { recursive: true, force: true });
+ ctx.onLog(`[source] WARNING: ${maskLiterals(why, ctx.literals)} — publishing without the history pages (${where})`);
+ return null;
+ };
+ const tokensFile =
+ a.opts.tokensFile ??
+ path.join(/* turbopackIgnore: true */ a.paths.monorepoRoot, "common", "styles", "tokens.css");
+ let stylesheet: string;
+ try {
+ stylesheet = historyStylesheet(await readFile(tokensFile, "utf8"));
+ } catch (err) {
+ if (err instanceof HistoryProblem) return without(err.message);
+ return without(`the design tokens (${tildify(tokensFile)}) could not be read`);
+ }
+ // stagit and git through the step's own runner (the environment, the
+ // cancel, the timeout); a failure or a timeout is the history's problem.
+ const child: HistoryRun = async (command, args, o) => {
+ try {
+ return await run(ctx, command, args, { ...o, allowFail: true });
+ } catch (err) {
+ if (err instanceof SourceRefusal) throw new HistoryProblem(err.message);
+ throw err;
+ }
+ };
+ const baseUrl = `${PROJECT_URL}${where}`;
+ let r;
+ try {
+ r = await renderHistory({
+ stagit: a.stagit,
+ gitDir: a.bare,
+ head: a.mirrorHead,
+ commits: a.commits,
+ maxCommits: a.opts.maxHistoryCommits ?? SOURCE_HISTORY_MAX_COMMITS,
+ dest: a.dest,
+ scratch: a.scratch,
+ cacheDir: a.historyCache,
+ cacheKey: historyCacheKey({
+ rulesHash: a.rulesHash,
+ filterRepo: a.filterRepoId,
+ stagit: a.stagitId,
+ description: PROJECT_NAME,
+ cloneUrl: CLONE_URL,
+ baseUrl,
+ }),
+ fresh: !!a.opts.force,
+ description: PROJECT_NAME,
+ cloneUrl: CLONE_URL,
+ baseUrl,
+ stylesheet,
+ themeScript: homepageThemeScript(),
+ treeFiles: a.treeFiles,
+ run: child,
+ // stagit's words (a failure's tail, a timeout's argv) can name the home
+ // dir: every line is masked, as every other child line of the step is.
+ onLog: (l) => ctx.onLog(maskLiterals(l, ctx.literals)),
+ });
+ } catch (err) {
+ if (err instanceof HistoryProblem) return without(`the history pages were not rendered: ${err.message}`);
+ throw err;
+ }
+ const fileLimit = a.opts.historyFileLimit ?? MAX_FILES;
+ if (a.stagedSoFar + r.files > fileLimit) {
+ return without(
+ `${r.files} history files would make ${a.stagedSoFar + r.files} files to publish, over the step's limit of ${fileLimit} (Pages allows 20,000 per deployment)`,
+ );
+ }
+ if (r.largest.bytes > MAX_FILE_BYTES) {
+ return without(
+ `${HISTORY_DIR}/${r.largest.rel} is ${mb(r.largest.bytes)} MiB, over the step's limit of ${mb(MAX_FILE_BYTES)} MiB (Pages allows 25 MiB per file)`,
+ );
+ }
+ const sha256 = await historyDigest(a.stage);
+ ctx.onLog(
+ `[source] history: ${a.stagitId} — ${r.shown < a.commits ? `the latest ${r.shown} of ${a.commits}` : a.commits} commits, ${r.rendered} page${r.rendered === 1 ? "" : "s"} rendered` +
+ `${r.cached ? " (the rest from the cache)" : ""}; ${r.files} files, ${mb(r.bytes)} MB, ` +
+ `the largest ${HISTORY_DIR}/${r.largest.rel} ${mb(r.largest.bytes)} MB (${elapsed(started)})`,
+ );
+ return {
+ href: HISTORY_LOG_HREF,
+ commits: r.shown,
+ total: a.commits,
+ head: a.mirrorHead,
+ files: r.files,
+ bytes: r.bytes,
+ sha256,
+ tool: a.stagitId,
+ };
+}
+
+/**
+ * `build homepage --no-source`: remove what an earlier publish left, because
+ * it was audited against the rules of ITS day. The pages then show their
+ * empty states.
+ */
+export async function clearPublishedSource(
+ opts: PublishOpts & { publicDir?: string } = {},
+): Promise<void> {
+ const paths = opts.paths ?? getPaths();
+ const onLog = opts.onLog ?? terminalLog;
+ const had = await removePublishedSource(sourcePublicDir(paths, opts.publicDir));
+ onLog(
+ had
+ ? "[notice] --no-source: the previously published source (mirror, tree, history, tarball) was removed — this build ships none.\n"
+ : "[notice] --no-source: no source published in this build.\n",
+ );
+}
+
+/**
+ * Why homepage/out's source may not be deployed, as one sentence — or null.
+ * `deployHomepage` asks before every deploy, production and preview alike: a
+ * deploy-only ships `out/` as the last build left it, and that build's source
+ * was audited under the rules of its day. It deploys only when the last
+ * publish (the skip key beside public/) was made under TODAY's rules and step
+ * (`rulesHash`), of TODAY's `main`, and is the one in `out/` (its mirror head,
+ * its tarball). An `out/` whose /source page shows the empty state — a
+ * `--no-source` build — deploys as it always has; one with no /source page at
+ * all (a refused build) does not.
+ */
+export async function publishedSourceProblem(
+ paths: Paths,
+ outDir: string,
+ opts: {
+ publicDir?: string;
+ scrubFile?: string;
+ denylistFile?: string;
+ homeDir?: string;
+ sourceRepo?: string;
+ env?: NodeJS.ProcessEnv;
+ // The gitleaks the audit would run (default "gitleaks"; null: none).
+ gitleaks?: string | null;
+ } = {},
+): Promise<string | null> {
+ const outSource = path.join(outDir, "source");
+ const outTarball = path.join(outDir, "downloads", TARBALL_NAME);
+ const rebuild = "run `archilyzer build homepage` (it re-audits), then deploy";
+ // The /source/ PAGE is always in a finished build (it renders its empty
+ // state without a publish). Without it, the build's source step refused —
+ // it took out/source away — or the build predates the page.
+ if (!existsSync(path.join(outSource, "index.html"))) {
+ return `homepage/out has no /source page (its source step refused, or the build predates it) — ${rebuild}`;
+ }
+ const artefacts = [
+ path.join(outSource, "manifest.json"),
+ path.join(outSource, MIRROR_DIR),
+ path.join(outSource, "tree"),
+ path.join(outSource, HISTORY_DIR),
+ outTarball,
+ ];
+ // A `--no-source` build: the page's empty state and nothing else.
+ if (!artefacts.some((a) => existsSync(a))) return null;
+ const manifest = parseSourceManifest(await readJson(path.join(outSource, "manifest.json")));
+ if (!manifest) return `homepage/out holds a source publish without a valid manifest — ${rebuild}`;
+ const state = (await readJson(statePath(sourcePublicDir(paths, opts.publicDir)))) as Partial<PublishState> | null;
+ if (!state?.rulesHash || !state.mirrorHead || !state.sourceCommit || !state.contentDigest || !state.gitleaks) {
+ return `homepage/out's source has no record of the rules it was audited under — ${rebuild}`;
+ }
+ let rules: SourceRules;
+ try {
+ rules = await loadSourceRules({
+ scrubFile: opts.scrubFile ?? paths.sourceScrubFile,
+ denylistFile: opts.denylistFile ?? paths.sourceDenylistFile,
+ homeDir: opts.homeDir,
+ });
+ } catch (err) {
+ if (err instanceof SourceRefusal) return `${err.message}; homepage/out's source cannot be checked`;
+ throw err;
+ }
+ if (state.rulesHash !== rules.rulesHash) {
+ return `homepage/out's source was audited under other rules (or an older version of the step) — ${rebuild}`;
+ }
+ if (state.mirrorHead !== manifest.mirrorHead) {
+ return `homepage/out's source (${manifest.mirrorHead.slice(0, 12)}) is not the last publish (${state.mirrorHead.slice(0, 12)}) — ${rebuild}`;
+ }
+ const ctx: Ctx = {
+ onLog: () => {},
+ signal: new AbortController().signal,
+ env: cleanGitEnv(opts.env ?? process.env),
+ literals: rules.literals,
+ };
+ let main: string | null = null;
+ try {
+ const repo = opts.sourceRepo ?? (await commonDir(ctx, paths.monorepoRoot));
+ main = repo ? await revParse(ctx, repo, `refs/heads/${SOURCE_BRANCH}^{commit}`) : null;
+ } catch (err) {
+ if (!(err instanceof SourceRefusal)) throw err;
+ }
+ if (main === null) return `homepage/out holds a source publish, and main cannot be read here — ${rebuild}`;
+ if (main !== state.sourceCommit || main !== manifest.sourceCommit) {
+ return `homepage/out's source mirrors main at ${manifest.sourceCommit.slice(0, 12)}, and main is now ${main.slice(0, 12)} — ${rebuild}`;
+ }
+ if (existsSync(outTarball) && (await sha256File(outTarball)) !== manifest.tarball.sha256) {
+ return `homepage/out's source tarball is not the one its manifest describes — ${rebuild}`;
+ }
+ const env = cleanGitEnv(opts.env ?? process.env);
+ const gitleaks = await gitleaksIdentity(opts.gitleaks === undefined ? "gitleaks" : opts.gitleaks, env);
+ if (state.gitleaks !== gitleaks) {
+ return `homepage/out's source was scanned by another gitleaks (${state.gitleaks}; this machine has ${gitleaks}) — ${rebuild}`;
+ }
+ // The history pages: exactly the audited ones, or none when none were
+ // published. The digest below binds them too; this names them.
+ const historyNow = existsSync(path.join(outSource, HISTORY_DIR)) ? await historyDigest(outDir) : null;
+ const historyThen = state.history?.digest ?? null;
+ if (historyNow !== historyThen || (manifest.history?.sha256 ?? null) !== historyThen) {
+ return `homepage/out's history pages (/source/${HISTORY_DIR}/) are not the ones that were audited — ${rebuild}`;
+ }
+ // Last, and the one that binds every byte: the mirror, the tree, the
+ // manifest and both downloads, against the digest of what was audited.
+ if ((await sourceDigest(outDir)) !== state.contentDigest) {
+ return `homepage/out's source files are not the ones that were audited (a mixed or edited out/) — ${rebuild}`;
+ }
+ return null;
+}
+
+// ── `archilyzer source audit` ───────────────────────────────────────────────
+
+/**
+ * The gate alone, over any git dir — by default the published mirror; the
+ * rollout runs it on a LIVE clone. 0 clean, 1 hits (the redacted report is
+ * logged) or refused.
+ */
+export async function auditSource(
+ opts: PublishOpts & {
+ gitDir?: string;
+ publicDir?: string;
+ scrubFile?: string;
+ denylistFile?: string;
+ gitleaks?: string | null;
+ scratchRoot?: string;
+ env?: NodeJS.ProcessEnv;
+ homeDir?: string;
+ } = {},
+): Promise<number> {
+ const paths = opts.paths ?? getPaths();
+ const onLog = opts.onLog ?? terminalLog;
+ const signal = opts.signal ?? new AbortController().signal;
+ const env = cleanGitEnv(opts.env ?? process.env);
+ const gitDir = path.resolve(
+ opts.gitDir ?? path.join(sourcePublicDir(paths, opts.publicDir), "source", MIRROR_DIR),
+ );
+ const scrubFile = opts.scrubFile ?? paths.sourceScrubFile;
+ let scratch: string | null = null;
+ let literals: readonly Literal[] = [];
+ try {
+ const rules = await loadSourceRules({
+ scrubFile,
+ denylistFile: opts.denylistFile ?? paths.sourceDenylistFile,
+ homeDir: opts.homeDir,
+ });
+ literals = rules.literals;
+ const ctx: Ctx = { onLog, signal, env, literals: rules.literals };
+ const head = await revParse(ctx, gitDir, "HEAD");
+ const scratchRoot = opts.scratchRoot ?? paths.sourceScratchDir;
+ await mkdir(scratchRoot, { recursive: true });
+ scratch = await mkdtemp(path.join(scratchRoot, "archilyzer-source-audit-"));
+ onLog(`[source] auditing ${maskLiterals(tildify(gitDir), literals)} (HEAD ${head.slice(0, 12)}) for ${rules.literals.length} denied literals…`);
+ const audit = await auditBare(gitDir, rules.literals, {
+ scratch,
+ onLog,
+ signal,
+ gitleaks: opts.gitleaks === undefined ? "gitleaks" : opts.gitleaks,
+ env,
+ });
+ for (const l of formatAuditReport(audit, rules.literals, { scrubFile })) onLog(l);
+ return audit.hits.length > 0 ? 1 : 0;
+ } catch (err) {
+ if (err instanceof Cancelled || signal.aborted) {
+ onLog("[source] cancelled");
+ return 1;
+ }
+ if (err instanceof SourceRefusal) {
+ onLog(`[source] REFUSED: ${maskLiterals(err.message, literals)}`);
+ return 1;
+ }
+ throw err;
+ } finally {
+ if (scratch) await rm(scratch, { recursive: true, force: true });
+ }
+}
diff --git a/common/publish/sourceAudit.test.ts b/common/publish/sourceAudit.test.ts
@@ -0,0 +1,195 @@
+import { test, after } from "node:test";
+import assert from "node:assert/strict";
+import { mkdirSync, mkdtempSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { gzipSync } from "node:zlib";
+import { execFileSync } from "node:child_process";
+import {
+ SourceRefusal,
+ auditBare,
+ auditFiles,
+ auditObjects,
+ formatAuditReport,
+ literalLabel,
+ maskLiterals,
+ objectField,
+ parseDenylist,
+ runGitleaks,
+ scanBuffer,
+ treeEntryNames,
+ type Literal,
+} from "./sourceAudit";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// The source mirror's gate (sourceAudit.ts): the literal list, the byte scan,
+// the redaction, the object walk over a real temp repo, the staged-file sweep
+// and gitleaks' absence. Every repo is built in the OS temp dir; the git
+// variables a hook or a wrapper might export are cleared first so nothing here
+// can reach the real repo.
+for (const key of ["GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE", "GIT_PREFIX"]) {
+ delete process.env[key];
+}
+
+const TMP = mkdtempSync(path.join(os.tmpdir(), "source-audit-"));
+after(() => rmSync(TMP, { recursive: true, force: true }));
+
+// The planted literal every test hunts for. Never a real name.
+const PLANTED = "plantedhome";
+const lits = (...xs: string[]): Literal[] =>
+ xs.map((x, i) => ({ bytes: Buffer.from(x), ci: false, from: `denylist line ${i + 1}` }));
+
+let n = 0;
+function repo(): string {
+ const dir = path.join(TMP, `r${n++}`);
+ mkdirSync(dir);
+ const git = (...args: string[]) => execFileSync("git", args, { cwd: dir, stdio: "pipe" });
+ git("init", "-q", "-b", "main");
+ git("config", "user.name", "audit test");
+ git("config", "user.email", "audit@example.invalid");
+ git("config", "commit.gpgsign", "false");
+ return dir;
+}
+function commit(dir: string, files: Record<string, string>, message: string, author?: string): void {
+ for (const [f, text] of Object.entries(files)) {
+ mkdirSync(path.dirname(path.join(dir, f)), { recursive: true });
+ writeFileSync(path.join(dir, f), text);
+ }
+ const git = (...args: string[]) => execFileSync("git", args, { cwd: dir, stdio: "pipe" });
+ git("add", "-A");
+ git("commit", "-q", "-m", message, ...(author ? ["--author", author] : []));
+}
+
+test("the denylist: one literal a line, i: for any case, comments and blanks skipped, duplicates once", () => {
+ const list = parseDenylist(`# a comment\n\n${PLANTED}\ni:MiXeD\n ${PLANTED} \r\ni:\n`);
+ assert.equal(list.length, 2);
+ assert.deepEqual(list[0], { bytes: Buffer.from(PLANTED), ci: false, from: "denylist line 3" });
+ assert.deepEqual(list[1], { bytes: Buffer.from("mixed"), ci: true, from: "denylist line 4" });
+ // Named by where it was written and its length — none of its characters.
+ assert.equal(literalLabel(list, 0), `denylist line 3 (len ${PLANTED.length})`);
+ assert.equal(literalLabel(list, 1), "denylist line 4 (len 5, any case)");
+});
+
+test("a byte-order mark and CRLF endings never become part of a literal", () => {
+ const list = parseDenylist(`\uFEFF${PLANTED}\r\ni:Other\r\n`);
+ assert.deepEqual(list.map((l) => l.bytes.toString()), [PLANTED, "other"]);
+ assert.equal(list[1].from, "denylist line 2");
+});
+
+test("scanBuffer finds every occurrence; an i: literal matches any ASCII case", () => {
+ const buf = Buffer.from(`a ${PLANTED} b PlantedHome c mixed MIXED`);
+ const hits = scanBuffer(buf, [...lits(PLANTED), ...parseDenylist("i:plantedHOME\ni:mixed")]);
+ assert.deepEqual(
+ hits.map((h) => [h.lit, h.offset]),
+ [[0, 2], [1, 2], [1, 16], [2, 30], [2, 36]],
+ );
+});
+
+test("maskLiterals merges overlapping occurrences into one mark and leaves the rest", () => {
+ const list = lits(PLANTED, "homeplanted");
+ assert.equal(maskLiterals(`/srv/${PLANTED}/x ${PLANTED}`, list), "/srv/[REDACTED]/x [REDACTED]");
+ assert.equal(maskLiterals(`a plantedhomeplanted b`, list), "a [REDACTED] b");
+ assert.equal(maskLiterals("nothing here", list), "nothing here");
+});
+
+test("objectField names a commit's header field or its message, never its bytes", () => {
+ const commit = Buffer.from(
+ "tree 1234\nparent 5678\nauthor A <a@x> 1 +0000\ncommitter C <c@x> 1 +0000\ngpgsig -----BEGIN\n sig line\n -----END\n\nthe message\n",
+ );
+ const at = (s: string) => commit.indexOf(s);
+ assert.equal(objectField(commit, at("a@x")), "author");
+ assert.equal(objectField(commit, at("c@x")), "committer");
+ assert.equal(objectField(commit, at("sig line")), "gpgsig");
+ assert.equal(objectField(commit, at("message")), "message");
+ assert.equal(objectField(commit, at("5678")), "parent");
+});
+
+test("a tree's entry names are what is scanned, not its binary ids", () => {
+ const id = Buffer.alloc(20, 0x70); // 'p' x20: would match "ppp" if ids were read
+ const tree = Buffer.concat([
+ Buffer.from("100644 a.txt\0"), id,
+ Buffer.from(`40000 ${PLANTED}\0`), id,
+ ]);
+ assert.equal(treeEntryNames(tree, 20).toString("latin1"), `a.txt\0${PLANTED}\0`);
+ assert.equal(scanBuffer(treeEntryNames(tree, 20), lits("ppp")).length, 0);
+});
+
+test("the object walk finds a literal in a blob, a commit message, an author line and a tree entry name", async () => {
+ const dir = repo();
+ commit(dir, { "README.md": "clean\n" }, "first");
+ commit(dir, { "notes.txt": `path /srv/${PLANTED}/data\n` }, "a blob carries it");
+ commit(dir, { "README.md": "clean 2\n" }, `the message says /srv/${PLANTED}`);
+ commit(dir, { "README.md": "clean 3\n" }, "an identity", `Planted <${PLANTED}@example.invalid>`);
+ commit(dir, { [`dir-${PLANTED}/x.txt`]: "x\n" }, "a name");
+ const result = await auditObjects(path.join(dir, ".git"), lits(PLANTED));
+ const kinds = result.hits.map((h) => `${h.kind}:${h.field ?? ""}`).sort();
+ // The message and the author line are one commit object each.
+ // The tree's hit is its second entry: README.md, dir-…, notes.txt.
+ assert.deepEqual(kinds, ["blob:", "commit:author", "commit:message", "tree:entry 2"]);
+ assert.equal(result.commits, 5);
+ const objects = execFileSync("git", ["count-objects", "-v"], { cwd: dir }).toString();
+ assert.equal(result.objects, Number(/^count: (\d+)/m.exec(objects)![1]), "every object was read");
+ // The report names the literal by where it was written, and prints no
+ // byte of any object: not the literal, not what sits beside it.
+ const report = formatAuditReport(result, lits(PLANTED), {
+ scrubFile: path.join(os.homedir(), ".config", "archilyzer", "source-scrub.txt"),
+ }).join("\n");
+ assert.ok(!report.includes(PLANTED), report);
+ for (const beside of ["/srv/", "/data", "example.invalid", "Planted <", "the message says"]) {
+ assert.ok(!report.includes(beside), `the report carries "${beside}" from an object`);
+ }
+ assert.match(report, /AUDIT REFUSED: 4 hits/);
+ assert.match(report, /denylist line 1 \(len 11\): 1 in blob, 2 in commits, 1 in tree/);
+ assert.match(report, /commit [0-9a-f]{12} \(author, byte \d+\): denylist line 1 \(len 11\)/);
+ assert.match(report, /commit [0-9a-f]{12} \(message, byte \d+\): denylist line 1 \(len 11\)/);
+ assert.match(report, /blob [0-9a-f]{12} \(byte 10\): denylist line 1 \(len 11\)/);
+ assert.match(report, /tree [0-9a-f]{12} \(entry 2\): denylist line 1 \(len 11\)/);
+ assert.match(report, /add a rule to ~\/\.config\/archilyzer\/source-scrub\.txt or drop the file from history, then re-run\.$/);
+});
+
+test("a clean repo audits clean, with gitleaks skipped when it is not on PATH", async () => {
+ const dir = repo();
+ commit(dir, { "a.txt": "nothing to see\n" }, "clean");
+ const logs: string[] = [];
+ const result = await auditBare(path.join(dir, ".git"), lits(PLANTED), {
+ scratch: TMP,
+ onLog: (l) => logs.push(l),
+ signal: new AbortController().signal,
+ gitleaks: "gitleaks-not-installed-here",
+ });
+ assert.equal(result.hits.length, 0);
+ assert.equal(result.gitleaks, "skipped");
+ assert.match(logs.join("\n"), /WARNING: gitleaks-not-installed-here is not on PATH — the secret scan is skipped/);
+ assert.match(formatAuditReport(result, lits(PLANTED), { scrubFile: "/x" })[0], /^\[source\] audit clean: \d+ objects \(1 commit\)/);
+ const skipped = await runGitleaks(path.join(dir, ".git"), {
+ bin: "gitleaks",
+ scratch: TMP,
+ literals: [],
+ onLog: () => {},
+ signal: new AbortController().signal,
+ env: { PATH: path.join(TMP, "empty-path") },
+ });
+ assert.equal(skipped.status, "skipped");
+});
+
+test("the staged-file sweep reads contents, names and gzip'd bytes decompressed, skips packs, refuses a symlink", async () => {
+ const dir = path.join(TMP, "stage");
+ mkdirSync(path.join(dir, "tree", `d-${PLANTED}`), { recursive: true });
+ writeFileSync(path.join(dir, "tree", "ok.txt"), "fine\n");
+ writeFileSync(path.join(dir, "tree", "bad.txt"), `x ${PLANTED} y\n`);
+ writeFileSync(path.join(dir, "tree", `d-${PLANTED}`, "f.txt"), "fine\n");
+ writeFileSync(path.join(dir, "t.tar.gz"), gzipSync(Buffer.from(`inside ${PLANTED}`)));
+ writeFileSync(path.join(dir, "pack-1.pack"), PLANTED); // the object walk's job
+ const result = await auditFiles(dir, lits(PLANTED));
+ assert.deepEqual(result.hits.map((h) => `${h.where} ${h.field}`).sort(), [
+ "t.tar.gz decompressed",
+ "tree/bad.txt contents",
+ "tree/d-[REDACTED] path",
+ "tree/d-[REDACTED]/f.txt path",
+ ]);
+ assert.equal(result.files, 5);
+ symlinkSync("/etc/hostname", path.join(dir, "tree", "link"));
+ await assert.rejects(auditFiles(dir, lits(PLANTED)), (err) => err instanceof SourceRefusal && /symlink/.test(err.message));
+});
diff --git a/common/publish/sourceAudit.ts b/common/publish/sourceAudit.ts
@@ -0,0 +1,656 @@
+// The source mirror's gate: does anything about to be published carry a
+// literal the operator denied? (`archilyzer source publish`, source.ts.)
+//
+// Three sweeps, all of them byte searches for the SAME literal list (the
+// denylist plus every scrub rule's left side — source.ts builds it):
+// - auditObjects: EVERY object in a git dir, read in one
+// `git cat-file --batch-all-objects --batch` stream — blobs whole, commits
+// and tags whole (message AND the author/committer/tagger lines), trees by
+// entry NAME. Reachable or not: a leftover of the private history in a
+// pack would be published with it, so it is read with it.
+// - auditFiles: every file staged beside the mirror (the raw tree, the index
+// pages, the tarball decompressed, the ref files, the manifest), and every
+// staged PATH.
+// - runGitleaks: the secret scanner over the mirror's history, when it is
+// installed (a WARNING, not a refusal, when it is not).
+//
+// THE REPORT NEVER PRINTS A LITERAL, AND NO BYTES OF THE OBJECT AROUND ONE. A
+// literal is named by where the operator wrote it — `denylist line 3 (len 5)`,
+// `scrub line 2 lhs (len 11)`, `built-in home rule (len 11)` — never by any of
+// its characters. A hit is named by its object (kind, id, the path when one
+// is known), the byte offset, and for a commit or tag the field it sits in
+// (author, committer, tagger, message): a context window would print what sits
+// NEXT to a denied name (a surname, the rest of an address), which is exactly
+// as private. Paths and child lines quoted in the log are masked
+// (`[REDACTED]`).
+
+import { spawn } from "node:child_process";
+import { existsSync, statSync, accessSync, constants } from "node:fs";
+import { lstat, readdir, readFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { gunzipSync } from "node:zlib";
+import { runChildIntoLog } from "../jobs/runChild";
+
+/** A refusal: the step stops, publishes nothing, and says why. */
+export class SourceRefusal extends Error {
+ constructor(message: string) {
+ super(message);
+ this.name = "SourceRefusal";
+ }
+}
+
+// ── literals ────────────────────────────────────────────────────────────────
+
+/**
+ * One denied literal. `ci` literals match ASCII case-insensitively: their
+ * `bytes` are stored folded, and are searched for in a folded copy. `from`
+ * says where the operator wrote it (`denylist line 3`), which is how a report
+ * names it.
+ */
+export type Literal = { bytes: Buffer; ci: boolean; from?: string };
+
+const LOWER_A = 0x61;
+const UPPER_A = 0x41;
+const UPPER_Z = 0x5a;
+
+/** A copy of `buf` with A–Z folded to a–z (every other byte as it is). */
+export function foldAscii(buf: Uint8Array): Buffer {
+ const out = Buffer.from(buf);
+ for (let i = 0; i < out.length; i++) {
+ const b = out[i];
+ if (b >= UPPER_A && b <= UPPER_Z) out[i] = b - UPPER_A + LOWER_A;
+ }
+ return out;
+}
+
+/**
+ * An operator file's text as lines: a UTF-8 byte-order mark at the start is
+ * dropped and CRLF endings become LF. An editor that writes either must not
+ * turn the first rule (and the denial it implies) into one that never matches.
+ */
+export function operatorLines(text: string): string[] {
+ return text.replace(/^\uFEFF/, "").split("\n").map((l) => l.replace(/\r$/, ""));
+}
+
+/**
+ * The denylist file: one literal per line. `i:` in front makes it ASCII
+ * case-insensitive. Blank lines and lines starting with `#` are skipped;
+ * surrounding whitespace is trimmed. Duplicates collapse (the first line
+ * names it).
+ */
+export function parseDenylist(text: string): Literal[] {
+ const out: Literal[] = [];
+ operatorLines(text).forEach((raw, i) => {
+ const line = raw.trim();
+ if (line === "" || line.startsWith("#")) return;
+ const from = `denylist line ${i + 1}`;
+ if (line.startsWith("i:")) {
+ const lit = line.slice(2).trim();
+ if (lit) out.push({ bytes: foldAscii(Buffer.from(lit, "utf8")), ci: true, from });
+ } else {
+ out.push({ bytes: Buffer.from(line, "utf8"), ci: false, from });
+ }
+ });
+ return dedupeLiterals(out);
+}
+
+export function dedupeLiterals(list: Literal[]): Literal[] {
+ const seen = new Set<string>();
+ const out: Literal[] = [];
+ for (const l of list) {
+ const key = `${l.ci ? "i" : "x"}:${l.bytes.toString("hex")}`;
+ if (seen.has(key) || l.bytes.length === 0) continue;
+ seen.add(key);
+ out.push(l);
+ }
+ return out;
+}
+
+/**
+ * How a literal is named in a report: where it was written and its length —
+ * never any of its characters (a first letter and a length all but spell a
+ * short first name).
+ */
+export function literalLabel(literals: readonly Literal[], index: number): string {
+ const l = literals[index];
+ return `${l.from ?? `literal ${index + 1}`} (len ${l.bytes.length}${l.ci ? ", any case" : ""})`;
+}
+
+// ── scanning ────────────────────────────────────────────────────────────────
+
+export type Hit = { lit: number; offset: number; length: number };
+
+/** Every occurrence of every literal in `buf`, by offset. */
+export function scanBuffer(buf: Buffer, literals: readonly Literal[]): Hit[] {
+ const hits: Hit[] = [];
+ let folded: Buffer | null = null;
+ literals.forEach((l, lit) => {
+ let hay = buf;
+ if (l.ci) {
+ folded ??= foldAscii(buf);
+ hay = folded;
+ }
+ let at = hay.indexOf(l.bytes);
+ while (at !== -1) {
+ hits.push({ lit, offset: at, length: l.bytes.length });
+ at = hay.indexOf(l.bytes, at + 1);
+ }
+ });
+ return hits.sort((a, b) => a.offset - b.offset || a.lit - b.lit);
+}
+
+/** `text` with every occurrence of every literal replaced by `[REDACTED]`. */
+export function maskLiterals(text: string, literals: readonly Literal[]): string {
+ const buf = Buffer.from(text, "utf8");
+ const hits = scanBuffer(buf, literals);
+ if (hits.length === 0) return text;
+ // Merge overlapping occurrences into runs, then splice one mark per run.
+ const runs: Array<[number, number]> = [];
+ for (const h of hits) {
+ const last = runs[runs.length - 1];
+ if (last && h.offset <= last[1]) last[1] = Math.max(last[1], h.offset + h.length);
+ else runs.push([h.offset, h.offset + h.length]);
+ }
+ const parts: Buffer[] = [];
+ let pos = 0;
+ for (const [a, b] of runs) {
+ parts.push(buf.subarray(pos, a), Buffer.from("[REDACTED]"));
+ pos = b;
+ }
+ parts.push(buf.subarray(pos));
+ return Buffer.concat(parts).toString("utf8");
+}
+
+// ── the audit's result ──────────────────────────────────────────────────────
+
+export type HitKind = "blob" | "commit" | "tree" | "tag" | "file" | "gitleaks";
+const KIND_ORDER: readonly HitKind[] = ["blob", "commit", "tree", "tag", "file", "gitleaks"];
+
+export type AuditHit = {
+ kind: HitKind;
+ // An object id (blob/commit/tree/tag) — with the blob's path in history
+ // once known — a path relative to the staged dir (file), or the finding
+ // (gitleaks). Paths are masked.
+ where: string;
+ // A literal's index, or -1 for a gitleaks finding.
+ lit: number;
+ // The byte offset of the hit in the object or file (-1 for gitleaks).
+ offset: number;
+ // Where in the object, without its bytes: a commit's or tag's header field
+ // (`author`, `committer`, `tagger`, …) or `message`; a tree's `entry N`.
+ field?: string;
+};
+
+export type AuditResult = {
+ literals: number;
+ objects: number;
+ commits: number;
+ files: number;
+ gitleaks: "clean" | "skipped" | "not run";
+ hits: AuditHit[];
+};
+
+// Only this many hits get a line of their own: a literal every commit carries
+// (the gate's own planted `Co-Authored-By`) would otherwise print thousands.
+const LISTED = 20;
+
+export function emptyAudit(literals: number): AuditResult {
+ return { literals, objects: 0, commits: 0, files: 0, gitleaks: "not run", hits: [] };
+}
+
+function record(
+ result: AuditResult,
+ kind: HitKind,
+ where: string,
+ hits: Hit[],
+ fieldOf?: (offset: number) => string,
+ // A tree's hits are offsets into its joined entry NAMES, not the object:
+ // the entry number says where, and the offset is left out.
+ withOffset = true,
+): void {
+ for (const h of hits) {
+ result.hits.push({
+ kind,
+ where,
+ lit: h.lit,
+ offset: withOffset ? h.offset : -1,
+ ...(fieldOf ? { field: fieldOf(h.offset) } : {}),
+ });
+ }
+}
+
+/**
+ * Which part of a commit or tag object `offset` falls in: `message` past the
+ * blank line that ends the headers, else the header line's keyword (`author`,
+ * `committer`, `tagger`, `tree`, `parent`, …) — a word git wrote, never the
+ * operator's bytes.
+ */
+export function objectField(data: Buffer, offset: number): string {
+ const end = data.indexOf("\n\n");
+ if (end !== -1 && offset > end) return "message";
+ let lineStart = data.lastIndexOf(0x0a, offset - 1) + 1;
+ // A continuation line (a signature, a mergetag) begins with a space: walk
+ // back to the header it continues.
+ while (lineStart > 0 && data[lineStart] === 0x20) {
+ lineStart = data.lastIndexOf(0x0a, lineStart - 2) + 1;
+ }
+ const sp = data.indexOf(0x20, lineStart);
+ const word = data.subarray(lineStart, sp === -1 ? lineStart : sp).toString("latin1");
+ return /^[a-z][a-z-]{0,15}$/.test(word) ? word : "header";
+}
+
+/** A tree hit's entry number (1-based) in the NUL-separated names buffer. */
+function treeEntry(names: Buffer, offset: number): string {
+ let n = 1;
+ for (let i = names.indexOf(0, 0); i !== -1 && i < offset; i = names.indexOf(0, i + 1)) n++;
+ return `entry ${n}`;
+}
+
+// ── the object walk ─────────────────────────────────────────────────────────
+
+/** The git environment a child must not inherit: it would point git elsewhere. */
+export function cleanGitEnv(env: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEnv {
+ const out: NodeJS.ProcessEnv = { ...env };
+ for (const k of Object.keys(out)) {
+ if (/^GIT_(DIR|WORK_TREE|INDEX_FILE|PREFIX|OBJECT_DIRECTORY|ALTERNATE_OBJECT_DIRECTORIES|COMMON_DIR|NAMESPACE|CEILING_DIRECTORIES|CONFIG|CONFIG_PARAMETERS|CONFIG_COUNT)$/.test(k)) {
+ out[k] = undefined;
+ }
+ }
+ return out;
+}
+
+/** A tree object's entry names, NUL-separated (a literal never holds a NUL). */
+export function treeEntryNames(data: Buffer, hashBytes: number): Buffer {
+ const names: Buffer[] = [];
+ let i = 0;
+ while (i < data.length) {
+ const sp = data.indexOf(0x20, i);
+ if (sp === -1) break;
+ const nul = data.indexOf(0x00, sp + 1);
+ if (nul === -1) break;
+ names.push(data.subarray(sp + 1, nul), Buffer.from([0]));
+ i = nul + 1 + hashBytes;
+ }
+ return Buffer.concat(names);
+}
+
+/**
+ * Read every object in `gitDir` once, scan it, and count. One child, one
+ * stream: `cat-file --batch` frames each object as `<oid> <type> <size>\n`,
+ * then the bytes, then `\n`.
+ */
+export async function auditObjects(
+ gitDir: string,
+ literals: readonly Literal[],
+ opts: { signal?: AbortSignal; env?: NodeJS.ProcessEnv; result?: AuditResult } = {},
+): Promise<AuditResult> {
+ const result = opts.result ?? emptyAudit(literals.length);
+ const child = spawn(
+ "git",
+ ["--git-dir", gitDir, "cat-file", "--batch-all-objects", "--unordered", "--batch"],
+ { stdio: ["ignore", "pipe", "pipe"], env: cleanGitEnv(opts.env), signal: opts.signal },
+ );
+ let stderr = "";
+ child.stderr.on("data", (c: Buffer) => {
+ if (stderr.length < 4000) stderr += c.toString("utf8");
+ });
+ const exited = new Promise<number>((resolve, reject) => {
+ child.once("error", reject);
+ child.once("close", (code) => resolve(code ?? 1));
+ });
+ exited.catch(() => {}); // awaited below; a throw in the loop must not orphan it
+
+ let state: "header" | "body" | "lf" = "header";
+ let headerParts: Buffer[] = [];
+ let oid = "";
+ let type = "";
+ let body = Buffer.alloc(0);
+ let filled = 0;
+
+ const onObject = () => {
+ result.objects++;
+ if (literals.length === 0) return;
+ if (type === "tree") {
+ const names = treeEntryNames(body, oid.length / 2);
+ const hits = scanBuffer(names, literals);
+ if (hits.length) record(result, "tree", oid, hits, (o) => treeEntry(names, o), false);
+ return;
+ }
+ const hits = scanBuffer(body, literals);
+ if (hits.length === 0) return;
+ if (type === "commit" || type === "tag") {
+ const data = body;
+ record(result, type, oid, hits, (o) => objectField(data, o));
+ } else {
+ record(result, "blob", oid, hits);
+ }
+ };
+
+ let drained = false;
+ try {
+ for await (const chunk of child.stdout as AsyncIterable<Buffer>) {
+ let i = 0;
+ // One pass per framing step; the three steps run in order inside one
+ // turn, so an empty object goes header -> body -> newline at once.
+ while (i < chunk.length) {
+ if (state === "header") {
+ const nl = chunk.indexOf(0x0a, i);
+ if (nl === -1) {
+ headerParts.push(chunk.subarray(i));
+ break;
+ }
+ headerParts.push(chunk.subarray(i, nl));
+ i = nl + 1;
+ const header = Buffer.concat(headerParts).toString("latin1");
+ headerParts = [];
+ const [o, t, size] = header.split(" ");
+ if (t === "missing" || size === undefined) {
+ throw new SourceRefusal(`git cat-file: unexpected header "${header.slice(0, 80)}"`);
+ }
+ oid = o;
+ type = t;
+ body = Buffer.allocUnsafe(Number(size));
+ filled = 0;
+ state = "body";
+ if (type === "commit") result.commits++;
+ }
+ if (state === "body") {
+ const n = Math.min(body.length - filled, chunk.length - i);
+ chunk.copy(body, filled, i, i + n);
+ filled += n;
+ i += n;
+ if (filled < body.length) break;
+ state = "lf";
+ }
+ if (state === "lf") {
+ if (i >= chunk.length) break;
+ i++; // the framing newline
+ onObject();
+ state = "header";
+ }
+ }
+ }
+ drained = true;
+ } finally {
+ // Only a loop that threw leaves a child to stop; a drained one is exiting.
+ if (!drained) child.kill();
+ }
+ const code = await exited;
+ if (code !== 0) {
+ throw new SourceRefusal(
+ `git cat-file over ${tildify(gitDir)} exited ${code}: ${stderr.trim().split("\n")[0] ?? ""}`,
+ );
+ }
+ if (state !== "header" || headerParts.length > 0) {
+ throw new SourceRefusal(`git cat-file over ${tildify(gitDir)} ended mid-object`);
+ }
+ return result;
+}
+
+/**
+ * Where each blob sits in history (the first path it was seen at), for the
+ * report. Only asked for when a blob hit, so a clean audit never pays for it.
+ */
+async function blobPaths(
+ gitDir: string,
+ wanted: Set<string>,
+ env?: NodeJS.ProcessEnv,
+): Promise<Map<string, string>> {
+ const out = new Map<string, string>();
+ const child = spawn("git", ["--git-dir", gitDir, "rev-list", "--objects", "--all"], {
+ stdio: ["ignore", "pipe", "ignore"],
+ env: cleanGitEnv(env),
+ });
+ let rest = "";
+ for await (const chunk of child.stdout as AsyncIterable<Buffer>) {
+ const lines = (rest + chunk.toString("utf8")).split("\n");
+ rest = lines.pop() ?? "";
+ for (const line of lines) {
+ const sp = line.indexOf(" ");
+ if (sp === -1) continue;
+ const o = line.slice(0, sp);
+ if (wanted.has(o) && !out.has(o)) out.set(o, line.slice(sp + 1));
+ }
+ }
+ await new Promise((r) => child.once("close", r));
+ return out;
+}
+
+// ── the staged files ────────────────────────────────────────────────────────
+
+/**
+ * Every file under `dir` except `skip` (the packs, which the object walk read
+ * decompressed), plus every path. A `.gz` is scanned decompressed — its
+ * compressed bytes would never match. A symlink is a hit of its own: nothing
+ * staged may point outside the stage.
+ */
+export async function auditFiles(
+ dir: string,
+ literals: readonly Literal[],
+ opts: { skip?: RegExp; result?: AuditResult } = {},
+): Promise<AuditResult> {
+ const skip = opts.skip ?? /\.(pack|idx)$/;
+ const result = opts.result ?? emptyAudit(literals.length);
+ const walk = async (rel: string): Promise<void> => {
+ const abs = path.join(dir, rel);
+ for (const ent of await readdir(abs, { withFileTypes: true })) {
+ const r = rel ? `${rel}/${ent.name}` : ent.name;
+ const nameHits = scanBuffer(Buffer.from(r, "utf8"), literals);
+ if (nameHits.length) record(result, "file", maskLiterals(r, literals), nameHits, () => "path");
+ if (ent.isSymbolicLink()) {
+ throw new SourceRefusal(`the stage holds a symlink (${maskLiterals(r, literals)}); nothing published may point outside it`);
+ }
+ if (ent.isDirectory()) {
+ await walk(r);
+ continue;
+ }
+ result.files++;
+ if (skip.test(ent.name)) continue;
+ let data = await readFile(path.join(abs, ent.name));
+ if (ent.name.endsWith(".gz")) data = gunzipSync(data);
+ const hits = scanBuffer(data, literals);
+ if (hits.length) {
+ record(result, "file", maskLiterals(r, literals), hits, () => (ent.name.endsWith(".gz") ? "decompressed" : "contents"));
+ }
+ }
+ };
+ if ((await lstat(dir)).isDirectory()) await walk("");
+ return result;
+}
+
+// ── gitleaks ────────────────────────────────────────────────────────────────
+
+/** A bare name resolved against PATH the way a spawn would, or null. */
+export function onPath(bin: string, envPath: string | undefined): string | null {
+ if (bin.includes("/")) return existsSync(bin) ? bin : null;
+ for (const d of (envPath ?? "").split(path.delimiter)) {
+ if (!d) continue;
+ const p = path.join(d, bin);
+ try {
+ accessSync(p, constants.X_OK);
+ if (statSync(p).isFile()) return p;
+ } catch {
+ /* not here */
+ }
+ }
+ return null;
+}
+
+type GitleaksFinding = { RuleID?: string; File?: string; Commit?: string };
+
+/**
+ * gitleaks over the mirror's history. Exit 0 is clean, 3 is findings (parsed
+ * from its JSON report), anything else is a refusal. Not installed: "skipped",
+ * with a WARNING line — the literal audit still ran.
+ */
+export async function runGitleaks(
+ gitDir: string,
+ opts: {
+ bin: string;
+ scratch: string;
+ literals: readonly Literal[];
+ onLog: (line: string) => void;
+ signal: AbortSignal;
+ env?: NodeJS.ProcessEnv;
+ timeoutMs?: number;
+ },
+): Promise<{ status: "clean" | "skipped"; hits: AuditHit[] }> {
+ const env = cleanGitEnv(opts.env ?? process.env);
+ if (!onPath(opts.bin, env.PATH)) {
+ opts.onLog(
+ `[source] WARNING: ${opts.bin} is not on PATH — the secret scan is skipped (the literal audit still ran). Install gitleaks to add it.`,
+ );
+ return { status: "skipped", hits: [] };
+ }
+ const report = path.join(opts.scratch, "gitleaks.json");
+ const timeout = AbortSignal.timeout(opts.timeoutMs ?? 300_000);
+ const code = await runChildIntoLog(
+ (line) => opts.onLog(`[gitleaks] ${maskLiterals(line, opts.literals)}`),
+ AbortSignal.any([opts.signal, timeout]),
+ {
+ command: opts.bin,
+ args: [
+ "git",
+ "--no-banner",
+ "--no-color",
+ "--redact",
+ "--exit-code",
+ "3",
+ "--report-format",
+ "json",
+ "--report-path",
+ report,
+ gitDir,
+ ],
+ // Its own ignore file is read from the cwd: the scratch dir has none.
+ cwd: opts.scratch,
+ env,
+ },
+ );
+ if (timeout.aborted) throw new SourceRefusal("gitleaks timed out");
+ if (code === 0) return { status: "clean", hits: [] };
+ if (code !== 3) throw new SourceRefusal(`gitleaks exited ${code}`);
+ let findings: GitleaksFinding[] = [];
+ try {
+ findings = JSON.parse(await readFile(report, "utf8")) as GitleaksFinding[];
+ } catch {
+ throw new SourceRefusal("gitleaks reported findings but its report did not parse");
+ }
+ return {
+ status: "clean",
+ hits: findings.map((f) => ({
+ kind: "gitleaks" as const,
+ where: maskLiterals(
+ `${f.RuleID ?? "?"} in ${f.File ?? "?"} @ ${(f.Commit ?? "").slice(0, 12)}`,
+ opts.literals,
+ ),
+ lit: -1,
+ offset: -1,
+ })),
+ };
+}
+
+// ── together ────────────────────────────────────────────────────────────────
+
+/**
+ * The gate over a git dir: the object walk, then (only when it is clean — a
+ * refusal is already certain otherwise) gitleaks. Blob hits are given their
+ * path in history.
+ */
+export async function auditBare(
+ gitDir: string,
+ literals: readonly Literal[],
+ opts: {
+ scratch: string;
+ onLog: (line: string) => void;
+ signal: AbortSignal;
+ gitleaks: string | null;
+ env?: NodeJS.ProcessEnv;
+ },
+): Promise<AuditResult> {
+ const result = await auditObjects(gitDir, literals, { signal: opts.signal, env: opts.env });
+ const blobs = new Set(result.hits.filter((h) => h.kind === "blob").map((h) => h.where));
+ if (blobs.size > 0) {
+ const where = await blobPaths(gitDir, blobs, opts.env);
+ for (const h of result.hits) {
+ const p = h.kind === "blob" ? where.get(h.where) : undefined;
+ if (p) h.where = `${h.where} ${maskLiterals(p, literals)}`;
+ }
+ }
+ if (result.hits.length > 0) return result;
+ if (opts.gitleaks === null) {
+ result.gitleaks = "skipped";
+ return result;
+ }
+ const g = await runGitleaks(gitDir, {
+ bin: opts.gitleaks,
+ scratch: opts.scratch,
+ literals,
+ onLog: opts.onLog,
+ signal: opts.signal,
+ env: opts.env,
+ });
+ result.gitleaks = g.status;
+ result.hits.push(...g.hits);
+ return result;
+}
+
+/** A path under the home directory as `~/…`: the home dir names the user. */
+export function tildify(p: string): string {
+ const home = os.homedir();
+ return p === home || p.startsWith(`${home}/`) ? `~${p.slice(home.length)}` : p;
+}
+
+/**
+ * The report, as lines. Clean: one line of counts. Hits: the counts per
+ * literal and kind, then the first hits — each by object, field and byte
+ * offset, never by its bytes — and what to do. No line carries a literal or
+ * anything read from beside one.
+ */
+export function formatAuditReport(
+ result: AuditResult,
+ literals: readonly Literal[],
+ opts: { scrubFile: string },
+): string[] {
+ const read =
+ `${result.objects.toLocaleString("en-US")} objects (${result.commits.toLocaleString("en-US")} commit${result.commits === 1 ? "" : "s"})` +
+ (result.files ? `, ${result.files.toLocaleString("en-US")} staged files` : "") +
+ ` against ${result.literals} denied literal${result.literals === 1 ? "" : "s"}; gitleaks ${result.gitleaks}`;
+ if (result.hits.length === 0) return [`[source] audit clean: ${read}`];
+ const lines = [`[source] AUDIT REFUSED: ${result.hits.length} hit${result.hits.length === 1 ? "" : "s"} in ${read}`];
+ const byLit = new Map<number, Map<HitKind, number>>();
+ for (const h of result.hits) {
+ const m = byLit.get(h.lit) ?? new Map<HitKind, number>();
+ m.set(h.kind, (m.get(h.kind) ?? 0) + 1);
+ byLit.set(h.lit, m);
+ }
+ for (const [lit, kinds] of [...byLit].sort((a, b) => a[0] - b[0])) {
+ const label = lit === -1 ? "gitleaks findings" : literalLabel(literals, lit);
+ const parts = KIND_ORDER.filter((k) => kinds.has(k)).map((k) => {
+ const n = kinds.get(k)!;
+ return `${n} in ${k}${n === 1 ? "" : "s"}`;
+ });
+ lines.push(`[source] ${label}: ${parts.join(", ")}`);
+ }
+ const shown = result.hits.slice(0, LISTED);
+ for (const h of shown) {
+ const where = h.kind === "file" || h.kind === "gitleaks" ? h.where : shortWhere(h.where);
+ const at = [h.field, h.offset >= 0 ? `byte ${h.offset}` : ""].filter(Boolean).join(", ");
+ const label = h.lit === -1 ? "" : `: ${literalLabel(literals, h.lit)}`;
+ lines.push(`[source] ${h.kind} ${where}${at ? ` (${at})` : ""}${label}`);
+ }
+ if (result.hits.length > shown.length) {
+ lines.push(`[source] … and ${result.hits.length - shown.length} more`);
+ }
+ lines.push(
+ `[source] add a rule to ${maskLiterals(tildify(opts.scrubFile), literals)} or drop the file from history, then re-run.`,
+ );
+ return lines;
+}
+
+// "<oid> <path>" → "<oid12> <path>"; a bare oid → oid12.
+function shortWhere(where: string): string {
+ const sp = where.indexOf(" ");
+ return sp === -1 ? where.slice(0, 12) : `${where.slice(0, 12)}${where.slice(sp)}`;
+}
diff --git a/common/publish/sourceHistory.test.ts b/common/publish/sourceHistory.test.ts
@@ -0,0 +1,632 @@
+import { test, after } from "node:test";
+import assert from "node:assert/strict";
+import { execFileSync, spawnSync } from "node:child_process";
+import {
+ chmodSync,
+ utimesSync,
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ readdirSync,
+ readFileSync,
+ rmSync,
+ writeFileSync,
+} from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { buildThemeScript, HOMEPAGE_DEFAULT_BASE } from "../lib/themeConfig";
+import { writeFakeStagit } from "./__fixtures__/fakeStagit";
+import {
+ HISTORY_BACK_LINK,
+ HISTORY_LOCK_STALE_MS,
+ HISTORY_TOKENS,
+ HistoryProblem,
+ SITE_ICON_HREF,
+ SOURCE_HISTORY_MAX_COMMITS,
+ loggedCommits,
+ historyStylesheet,
+ holdHistoryCache,
+ homepageThemeScript,
+ renderHistory,
+ resolveStagit,
+ rewriteHistoryPage,
+ stageHistory,
+ stagitIdentity,
+ tokenBlocks,
+ type HistoryRun,
+ type RenderHistoryOpts,
+} from "./sourceHistory";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// The history pages (sourceHistory.ts): the post-pass, the stylesheet from
+// the tokens, the binary's lookup, and the render with its cache — over a
+// fake stagit that writes what stagit writes, and over the real one when it
+// is installed (that test SKIPS without it; a gate says whether it ran).
+for (const key of ["GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE", "GIT_PREFIX"]) {
+ delete process.env[key];
+}
+
+const HERE = path.dirname(fileURLToPath(import.meta.url));
+const TOKENS = readFileSync(path.join(HERE, "..", "styles", "tokens.css"), "utf8");
+const TMP = mkdtempSync(path.join(os.tmpdir(), "source-history-"));
+after(() => rmSync(TMP, { recursive: true, force: true }));
+
+let n = 0;
+function dir(name: string): string {
+ const d = path.join(TMP, `${name}-${n++}`);
+ mkdirSync(d, { recursive: true });
+ return d;
+}
+
+// ── the post-pass ───────────────────────────────────────────────────────────
+
+// stagit's own markup, as it writes a commit page two levels down (relpath
+// "../"): the header, the diffstat's anchors, a diff header linking both
+// sides into file/, and a message that quotes an href as text.
+const COMMIT_PAGE = `<!DOCTYPE html>
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+<title>a subject - archilyzer - Archilyzer</title>
+<link rel="icon" type="image/png" href="../favicon.png" />
+<link rel="alternate" type="application/atom+xml" title="archilyzer.git Atom Feed" href="../atom.xml" />
+<link rel="stylesheet" type="text/css" href="../style.css" />
+</head>
+<body>
+<table><tr><td><a href="../../"><img src="../logo.png" alt="" width="32" height="32" /></a></td><td><h1>archilyzer</h1></td></tr><tr><td></td><td>
+<a href="../log.html">Log</a> | <a href="../files.html">Files</a> | <a href="../refs.html">Refs</a> | <a href="../file/README.md.html">README</a> | <a href="../file/LICENSE.html">LICENSE</a></td></tr></table>
+<hr/>
+<div id="content">
+<pre><b>commit</b> <a href="../commit/${"a".repeat(40)}.html">${"a".repeat(40)}</a>
+a message that says href="file/x.html" as text
+<b>diff --git a/<a id="h0" href="../file/app/%5Bslug%5D/page.tsx.html">app/[slug]/page.tsx</a> b/<a href="../file/app/%5Bslug%5D/page.tsx.html">app/[slug]/page.tsx</a></b>
+<a href="#h0-0-0" id="h0-0-0" class="i">+added
+</a></pre>
+</div>
+</body>
+</html>
+`;
+
+test("the post-pass: the theme script before </head>, the back link after <body>, file/ links to the raw tree, the logo and favicon to the site's icon — nothing else", () => {
+ const script = "var x=1;";
+ const out = rewriteHistoryPage(COMMIT_PAGE, script);
+ const expected = COMMIT_PAGE
+ .replace("</head>", `<script>${script}</script>\n</head>`)
+ .replace("<body>\n", `<body>\n${HISTORY_BACK_LINK}\n`)
+ .replace('href="../favicon.png"', `href="${SITE_ICON_HREF}"`)
+ .replace('src="../logo.png"', `src="${SITE_ICON_HREF}"`)
+ .replace('href="../file/README.md.html"', 'href="../../tree/README.md"')
+ .replace('href="../file/LICENSE.html"', 'href="../../tree/LICENSE"')
+ .replaceAll('href="../file/app/%5Bslug%5D/page.tsx.html"', 'href="../../tree/app/%5Bslug%5D/page.tsx"');
+ assert.equal(out, expected);
+ // Text that quotes an href is encoded by stagit, and left alone.
+ assert.match(out, /says href="file\/x\.html" as text/);
+ // The back link is the one line added to the body, and it is ASCII.
+ assert.equal(HISTORY_BACK_LINK, '<p class="archilyzer-source"><a href="/source/">Archilyzer · Source</a></p>');
+ assert.ok(!/[^\x00-\x7f]/.test(HISTORY_BACK_LINK));
+});
+
+test("the post-pass at the top level: files.html's links go to ../tree/<path>, as stagit encoded them", () => {
+ const files = `<html>\n<head>\n</head>\n<body>\n<tr><td>-rw-r--r--</td><td><a href="file/fonts/Archivo%5Bwdth%2Cwght%5D.ttf.html">fonts/Archivo[wdth,wght].ttf</a></td></tr>\n<a href="file/a.html.html">a.html</a>\n</body>\n</html>\n`;
+ const out = rewriteHistoryPage(files, "s()");
+ assert.match(out, /<a href="\.\.\/tree\/fonts\/Archivo%5Bwdth%2Cwght%5D\.ttf">/);
+ assert.match(out, /<a href="\.\.\/tree\/a\.html">a\.html<\/a>/, "a file that is itself .html keeps its name");
+ assert.ok(!out.includes('href="file/'));
+});
+
+test("the post-pass with what is published: a link to a file main no longer has, or to a commit with no page, becomes text (its id kept)", () => {
+ const sha = "a".repeat(40);
+ const gone = "b".repeat(40);
+ const page = `<html>\n<head>\n</head>\n<body>\n<a href="../commit/${sha}.html">${sha}</a> <b>parent</b> <a href="../commit/${gone}.html">${gone}</a>\n<b>diff --git a/<a id="h0" href="../file/old/name.txt.html">old/name.txt</a> b/<a href="../file/app/%5Bslug%5D/page.tsx.html">app/[slug]/page.tsx</a></b>\n<a href="../file/bad%zz.html">bad</a>\n</body>\n</html>\n`;
+ const published = {
+ tree: (p: string) => p === "app/[slug]/page.tsx",
+ commit: (c: string) => c === sha,
+ };
+ const out = rewriteHistoryPage(page, "t()", published);
+ assert.ok(out.includes(`<a href="../commit/${sha}.html">${sha}</a>`), "a published commit keeps its link");
+ assert.ok(out.includes(`<b>parent</b> <a>${gone}</a>`), "an unpublished commit is text");
+ assert.ok(out.includes(`a/<a id="h0">old/name.txt</a>`), "a file main no longer has is text, and keeps the diffstat's target");
+ assert.ok(out.includes(`b/<a href="../../tree/app/%5Bslug%5D/page.tsx">`), "a file in the tree keeps its (encoded) link");
+ assert.ok(out.includes(`<a>bad</a>`), "an encoding that does not decode is not linked");
+ // Without the set, nothing is dropped (the unit tests above).
+ assert.ok(rewriteHistoryPage(page, "t()").includes(`href="../../tree/old/name.txt"`));
+});
+
+test("the theme script is the homepage's own: its base, its string", () => {
+ assert.equal(HOMEPAGE_DEFAULT_BASE, "dark");
+ assert.equal(homepageThemeScript(), buildThemeScript({ defaultBase: HOMEPAGE_DEFAULT_BASE }));
+ assert.ok(!/[^\x00-\x7f]/.test(homepageThemeScript()), "ASCII, for the byte-for-byte rewrite");
+ // It must not be able to end its own element.
+ assert.throws(() => rewriteHistoryPage("<head></head>", "a</script><b>"));
+ // The homepage's layout passes the same constant to ThemeScript.
+ const layout = readFileSync(path.join(HERE, "..", "..", "homepage", "app", "layout.tsx"), "utf8");
+ assert.match(layout, /<ThemeScript defaultBase=\{HOMEPAGE_DEFAULT_BASE\} \/>/);
+});
+
+// ── the stylesheet ──────────────────────────────────────────────────────────
+
+test("style.css: the homepage's two grounds from tokens.css, by prefers-color-scheme and by data-base", () => {
+ const css = historyStylesheet(TOKENS);
+ const { light, dark } = tokenBlocks(TOKENS);
+ const block = (selector: RegExp) => {
+ const m = selector.exec(css);
+ assert.ok(m, String(selector));
+ return m[1];
+ };
+ const lightBlock = block(/:root,\nhtml\[data-base="light"\] \{\n([\s\S]*?)\n\}/);
+ const mediaBlock = block(/@media \(prefers-color-scheme: dark\) \{\n {2}:root:not\(\[data-base\]\) \{\n([\s\S]*?)\n {2}\}/);
+ const darkBlock = block(/\nhtml\[data-base="dark"\] \{\n([\s\S]*?)\n\}/);
+ for (const name of [...HISTORY_TOKENS, "color-scheme", "--swatch-signal"]) {
+ assert.ok(lightBlock.includes(` ${name}: ${light.get(name)};`), `light ${name}`);
+ assert.ok(darkBlock.includes(` ${name}: ${dark.get(name)};`), `dark ${name}`);
+ assert.ok(mediaBlock.includes(` ${name}: ${dark.get(name)};`), `no-JS dark ${name}`);
+ }
+ // Only what the rules read (and what that reaches through var()).
+ assert.ok(!lightBlock.includes("--chart-1"));
+ // Diff lines: insertions in --success, deletions in --destructive.
+ assert.match(css, /pre a\.i \{ color: var\(--success\); \}/);
+ assert.match(css, /pre a\.d \{ color: var\(--destructive\); \}/);
+ assert.match(css, /p\.archilyzer-source a \{/);
+});
+
+test("style.css follows the tokens: a changed value is the new value; a missing block or token is a HistoryProblem", () => {
+ const moved = TOKENS.replace(/(html\[data-base="dark"\] \{[\s\S]*?--background: )#[0-9a-f]+;/, "$1#010203;");
+ assert.notEqual(moved, TOKENS);
+ assert.match(historyStylesheet(moved), /html\[data-base="dark"\] \{[\s\S]*?--background: #010203;/);
+ assert.throws(() => historyStylesheet(":root { --background: #fff; }"), HistoryProblem);
+ const noInfo = TOKENS.replace(/--info: #[0-9a-f]+;/g, "");
+ assert.throws(() => historyStylesheet(noInfo), (e) => e instanceof HistoryProblem && /no --info/.test(e.message));
+});
+
+// ── the binary ──────────────────────────────────────────────────────────────
+
+function exe(file: string, body = "#!/bin/sh\nexit 0\n"): string {
+ mkdirSync(path.dirname(file), { recursive: true });
+ writeFileSync(file, body);
+ chmodSync(file, 0o755);
+ return file;
+}
+
+test("resolveStagit: STAGIT_BIN as a path, else stagit on PATH, else ~/.local/bin/stagit, else null", async () => {
+ const home = dir("home");
+ const bin = dir("bin");
+ const empty = dir("empty");
+ assert.equal(resolveStagit("stagit", { PATH: empty }, home), null);
+ const local = exe(path.join(home, ".local", "bin", "stagit"));
+ assert.equal(resolveStagit("stagit", { PATH: empty }, home), local);
+ const onPathBin = exe(path.join(bin, "stagit"));
+ assert.equal(resolveStagit("stagit", { PATH: `${empty}:${bin}` }, home), onPathBin, "PATH first");
+ const given = exe(path.join(dir("given"), "my-stagit"));
+ assert.equal(resolveStagit(given, { PATH: bin }, home), given);
+ assert.equal(resolveStagit(path.join(empty, "nope"), { PATH: bin }, home), null, "a path that is not there is nothing, not a PATH lookup");
+ writeFileSync(path.join(empty, "not-exec"), "x");
+ assert.equal(resolveStagit(path.join(empty, "not-exec"), { PATH: bin }, home), null);
+
+ assert.equal(await stagitIdentity(null), "absent");
+ const a = await stagitIdentity(given);
+ assert.match(a, /^stagit \(sha256 [0-9a-f]{12}\)$/);
+ exe(given, "#!/bin/sh\nexit 1\n");
+ assert.notEqual(await stagitIdentity(given), a, "another binary, another identity");
+ assert.ok(!a.includes(given), "never the path");
+});
+
+// ── the render, over a fake stagit ──────────────────────────────────────────
+
+function gitIn(cwd: string, ...args: string[]): string {
+ return execFileSync("git", args, { cwd, stdio: "pipe" }).toString().trim();
+}
+
+// A bare repository named archilyzer.git with `count` commits on main.
+function bareRepo(count: number): { gitDir: string; work: string } {
+ const work = dir("work");
+ gitIn(work, "init", "-q", "-b", "main");
+ gitIn(work, "config", "user.name", "history test");
+ gitIn(work, "config", "user.email", "history@example.invalid");
+ gitIn(work, "config", "commit.gpgsign", "false");
+ for (let i = 0; i < count; i++) addCommit(work, i);
+ const gitDir = path.join(dir("bare"), "archilyzer.git");
+ gitIn(TMP, "clone", "-q", "--bare", work, gitDir);
+ return { gitDir, work };
+}
+
+function addCommit(work: string, i: number): void {
+ writeFileSync(path.join(work, "README.md"), `hello ${i}\n`);
+ mkdirSync(path.join(work, "app", "[slug]"), { recursive: true });
+ writeFileSync(path.join(work, "app", "[slug]", "page.tsx"), `export default ${i};\n`);
+ gitIn(work, "add", "-A");
+ gitIn(work, "commit", "-q", "-m", `commit ${i}`);
+}
+
+// The fake (__fixtures__/fakeStagit.ts) writes what stagit writes, with -c
+// and -l as stagit.c has them; every call is logged to `log`.
+function fakeStagit(log: string, o: { exit?: number; extra?: string; pad?: number } = {}): string {
+ return writeFakeStagit(path.join(dir("fake"), "stagit"), log, o);
+}
+
+const run: HistoryRun = async (command, args, o) => {
+ const p = spawnSync(command, args, { cwd: o.cwd, encoding: "utf8" });
+ return { code: p.status ?? 1, out: `${p.stdout}${p.stderr}`.split("\n") };
+};
+
+function renderOpts(gitDir: string, stagit: string, extra: Partial<RenderHistoryOpts> = {}): RenderHistoryOpts & { logs: string[] } {
+ const logs: string[] = [];
+ const head = gitIn(TMP, "--git-dir", gitDir, "rev-parse", "main");
+ const commits = Number(gitIn(TMP, "--git-dir", gitDir, "rev-list", "--count", "main"));
+ return {
+ stagit,
+ gitDir,
+ head,
+ commits,
+ maxCommits: SOURCE_HISTORY_MAX_COMMITS,
+ dest: path.join(dir("stage"), "source", "git"),
+ scratch: dir("scratch"),
+ cacheDir: null,
+ cacheKey: "k1",
+ fresh: false,
+ description: "Archilyzer",
+ cloneUrl: "https://archilyzer.pages.dev/source/archilyzer.git",
+ baseUrl: "https://archilyzer.pages.dev/source/git/",
+ stylesheet: "/* css */\n",
+ themeScript: "t()",
+ run,
+ onLog: (l) => logs.push(l),
+ logs,
+ ...extra,
+ };
+}
+
+const calls = (log: string) => (existsSync(log) ? readFileSync(log, "utf8").trim().split("\n") : []);
+
+test("render: the allowlist of stagit's output, each page through the post-pass, the feeds as they are, style.css; never file/", async () => {
+ const { gitDir } = bareRepo(3);
+ const log = path.join(dir("log"), "calls");
+ const o = renderOpts(gitDir, fakeStagit(log));
+ const r = await renderHistory(o);
+ assert.deepEqual(calls(log), [`cache= limit= base=${o.baseUrl} repo=archilyzer.git`]);
+ assert.equal(r.files, 3 + 5 + 1);
+ assert.equal(r.rendered, 3);
+ assert.equal(r.shown, 3);
+ assert.equal(r.cached, false);
+ assert.deepEqual(readdirSync(o.dest).sort(), ["atom.xml", "commit", "files.html", "log.html", "refs.html", "style.css", "tags.xml"]);
+ assert.equal(readdirSync(path.join(o.dest, "commit")).length, 3);
+ const logHtml = readFileSync(path.join(o.dest, "log.html"), "utf8");
+ // stagit's header reads the description and the clone URL from the repo.
+ assert.match(logHtml, /<span class="desc">Archilyzer<\/span> https:\/\/archilyzer\.pages\.dev\/source\/archilyzer\.git /);
+ assert.ok(logHtml.includes(`<body>\n${HISTORY_BACK_LINK}\n`));
+ assert.ok(logHtml.includes("<script>t()</script>\n</head>"));
+ assert.match(logHtml, /href="\.\.\/tree\/README\.md"/);
+ assert.match(logHtml, new RegExp(`src="${SITE_ICON_HREF}"`));
+ assert.match(readFileSync(path.join(o.dest, "files.html"), "utf8"), /href="\.\.\/tree\/app\/%5Bslug%5D\/page\.tsx"/);
+ // The feeds are not pages: not one byte changes.
+ assert.equal(readFileSync(path.join(o.dest, "atom.xml"), "utf8"), `<feed>${o.baseUrl} file/README.md.html</feed>\n`);
+ assert.equal(readFileSync(path.join(o.dest, "style.css"), "utf8"), "/* css */\n");
+ // No cache: the render ran in the publish's own scratch.
+ assert.ok(existsSync(path.join(o.scratch, "history", "log.html")));
+ assert.ok(!existsSync(path.join(o.scratch, "history", "file")), "the per-file pages are dropped where they were rendered");
+});
+
+test("render: the page bytes are kept (a diff of a file that is not UTF-8), only the ASCII injections added", async () => {
+ const work = dir("bytes");
+ mkdirSync(path.join(work, "commit"));
+ const odd = Buffer.concat([Buffer.from("<html>\n<head>\n</head>\n<body>\n"), Buffer.from([0xff, 0xfe, 0x80]), Buffer.from("\n</body>\n</html>\n")]);
+ writeFileSync(path.join(work, "log.html"), odd);
+ for (const f of ["files.html", "refs.html"]) writeFileSync(path.join(work, f), "<html>\n<head>\n</head>\n<body>\n</body>\n</html>\n");
+ for (const f of ["atom.xml", "tags.xml"]) writeFileSync(path.join(work, f), Buffer.from([0x3c, 0xff, 0x3e]));
+ const dest = path.join(dir("dest"), "git");
+ await stageHistory(work, dest, { themeScript: "t()", stylesheet: "", commits: [] });
+ const out = readFileSync(path.join(dest, "log.html"));
+ assert.ok(out.includes(Buffer.from([0xff, 0xfe, 0x80])), "the odd bytes survive");
+ assert.ok(out.includes(Buffer.from(HISTORY_BACK_LINK)));
+ assert.deepEqual(readFileSync(path.join(dest, "atom.xml")), Buffer.from([0x3c, 0xff, 0x3e]));
+ await assert.rejects(stageHistory(work, dest, { themeScript: "é", stylesheet: "", commits: [] }), /ASCII/);
+ // A commit the log lists must have its page.
+ await assert.rejects(
+ stageHistory(work, dest, { themeScript: "t()", stylesheet: "", commits: ["a".repeat(40)] }),
+ (e) => e instanceof HistoryProblem && /the log lists aaaaaaaaaaaa, whose page is not there/.test(e.message),
+ );
+});
+
+test("render: a failing stagit, or one that leaves the wrong pages, is a HistoryProblem", async () => {
+ const { gitDir } = bareRepo(2);
+ const log = path.join(dir("log"), "calls");
+ await assert.rejects(
+ renderHistory(renderOpts(gitDir, fakeStagit(log, { exit: 3 }))),
+ (e) => e instanceof HistoryProblem && /stagit exited 3: stagit: something broke/.test(e.message),
+ );
+ await assert.rejects(
+ renderHistory(renderOpts(gitDir, fakeStagit(log), { commits: 5 })),
+ (e) => e instanceof HistoryProblem && /stagit's log lists 2 commits, not 5 \(5 in all, at most 10000\)/.test(e.message),
+ );
+});
+
+test("loggedCommits: the commits the log links, in order, once each", () => {
+ const a = "a".repeat(40);
+ const b = "b".repeat(40);
+ const log = `<tr><td><a href="commit/${a}.html">x</a></td></tr>\n<tr><td><a href="commit/${b}.html">y</a></td></tr>\n<a href="commit/${a}.html">again</a> says href="commit/${"c".repeat(40)}.html"`;
+ assert.deepEqual(loggedCommits(log), [a, b]);
+});
+
+test("the cap: past it, -l keeps the newest in the log, and only their pages are published", async () => {
+ const { gitDir, work } = bareRepo(5);
+ const newest = gitIn(TMP, "--git-dir", gitDir, "rev-list", "--max-count=3", "main").split("\n");
+ const log = path.join(dir("log"), "calls");
+ const stagit = fakeStagit(log);
+
+ // No cache: `-l 3`, no `-c`.
+ let o = renderOpts(gitDir, stagit, { maxCommits: 3 });
+ let r = await renderHistory(o);
+ assert.equal(calls(log).at(-1), `cache= limit=3 base=${o.baseUrl} repo=archilyzer.git`);
+ assert.equal(r.shown, 3);
+ assert.equal(r.files, 3 + 5 + 1);
+ assert.deepEqual(readdirSync(path.join(o.dest, "commit")).sort(), newest.map((c) => `${c}.html`).sort());
+ assert.equal(readdirSync(path.join(o.scratch, "history", "commit")).length, 5, "stagit wrote all five; three are published");
+ assert.match(readFileSync(path.join(o.dest, "log.html"), "utf8"), /2 more commits remaining, fetch the repository/);
+
+ // With the cache: still `-l 3` (stagit refuses -c with -l). A new commit:
+ // its page is the one rendered, and the oldest of the three drops out.
+ const cacheDir = path.join(dir("cache-home"), "archilyzer", "source-history");
+ o = renderOpts(gitDir, stagit, { maxCommits: 3, cacheDir });
+ r = await renderHistory(o);
+ assert.equal(calls(log).at(-1), `cache= limit=3 base=${o.baseUrl} repo=archilyzer.git`);
+ assert.equal(r.rendered, 5);
+ addCommit(work, 9);
+ gitIn(TMP, "--git-dir", gitDir, "fetch", "-q", work, "main:main");
+ const now = gitIn(TMP, "--git-dir", gitDir, "rev-list", "--max-count=3", "main").split("\n");
+ o = renderOpts(gitDir, stagit, { maxCommits: 3, cacheDir });
+ r = await renderHistory(o);
+ assert.equal(r.cached, true);
+ assert.equal(r.rendered, 1);
+ assert.equal(r.shown, 3);
+ assert.deepEqual(readdirSync(path.join(o.dest, "commit")).sort(), now.map((c) => `${c}.html`).sort());
+ assert.match(readFileSync(path.join(o.dest, "log.html"), "utf8"), /3 more commits remaining/);
+});
+
+test("the cache: -c in the cache dir, incremental the next time, the key and the log's ancestry checked, busy or cut-off locks handled", async () => {
+ const { gitDir, work } = bareRepo(2);
+ const log = path.join(dir("log"), "calls");
+ const stagit = fakeStagit(log);
+ const cacheDir = path.join(dir("cache-home"), "archilyzer", "source-history");
+ const cacheFile = path.join(cacheDir, "stagit.cache");
+ const cacheArgs = (o: RenderHistoryOpts) => `cache=${cacheFile} limit= base=${o.baseUrl} repo=archilyzer.git`;
+
+ // First: everything rendered, the cache made (recursively) and kept (key,
+ // cache file, output), unlocked; no per-file pages, no stagit leftovers.
+ let o = renderOpts(gitDir, stagit, { cacheDir });
+ let r = await renderHistory(o);
+ assert.equal(r.cached, false);
+ assert.equal(r.rendered, 2);
+ assert.equal(calls(log).at(-1), cacheArgs(o));
+ assert.deepEqual(readdirSync(cacheDir).sort(), ["key.json", "out", "stagit.cache"]);
+ assert.ok(!existsSync(path.join(cacheDir, "out", "file")));
+ assert.ok(!existsSync(path.join(o.dest, "cache.XXXXleftover")), "only the allowlist is published");
+
+ // A new commit: the cache is used, and only the new page is rendered.
+ addCommit(work, 2);
+ gitIn(TMP, "--git-dir", gitDir, "fetch", "-q", work, "main:main");
+ o = renderOpts(gitDir, stagit, { cacheDir });
+ r = await renderHistory(o);
+ assert.equal(r.cached, true);
+ assert.equal(r.rendered, 1);
+ assert.equal(r.files, 3 + 5 + 1);
+ assert.equal(loggedCommits(readFileSync(path.join(o.dest, "log.html"), "utf8")).length, 3, "the new line and the cached ones");
+
+ // Another key (other rules, another stagit…): every page rendered again.
+ const junk = path.join(cacheDir, "out", "commit", `${"f".repeat(40)}.html`);
+ writeFileSync(junk, "junk");
+ o = renderOpts(gitDir, stagit, { cacheDir, cacheKey: "k2" });
+ r = await renderHistory(o);
+ assert.equal(r.cached, false);
+ assert.equal(r.rendered, 3);
+ assert.ok(!existsSync(junk));
+
+ // Log lines ending at a commit that is not an ancestor of the head (a
+ // rewritten main): the cache file goes, the pages stay, the log is whole.
+ writeFileSync(cacheFile, `${"e".repeat(40)}\n<tr><td><a href="commit/${"e".repeat(40)}.html">x</a></td></tr>\n`);
+ o = renderOpts(gitDir, stagit, { cacheDir, cacheKey: "k2" });
+ r = await renderHistory(o);
+ assert.equal(r.cached, true);
+ assert.equal(r.rendered, 0);
+ assert.equal(loggedCommits(readFileSync(path.join(o.dest, "log.html"), "utf8")).length, 3);
+
+ // A page of a commit history does not have is never published.
+ writeFileSync(junk, "junk");
+ o = renderOpts(gitDir, stagit, { cacheDir, cacheKey: "k2" });
+ r = await renderHistory(o);
+ assert.equal(r.files, 3 + 5 + 1);
+ assert.ok(!existsSync(path.join(o.dest, "commit", `${"f".repeat(40)}.html`)));
+
+ // A listed commit's page gone from the cache: noticed, and every page is
+ // rendered again.
+ const head = gitIn(TMP, "--git-dir", gitDir, "rev-parse", "main");
+ const oldest = gitIn(TMP, "--git-dir", gitDir, "rev-list", "main").split("\n").at(-1)!;
+ assert.notEqual(oldest, head);
+ rmSync(path.join(cacheDir, "out", "commit", `${oldest}.html`));
+ o = renderOpts(gitDir, stagit, { cacheDir, cacheKey: "k2" });
+ r = await renderHistory(o);
+ assert.equal(r.cached, false);
+ assert.match(o.logs.join("\n"), new RegExp(`the cached render does not cover this head \\(the log lists ${oldest.slice(0, 12)}, whose page is not there\\) — stagit's -c stops at the last head it rendered, and a merge of older commits falls behind it; rendering every page again`));
+ assert.ok(existsSync(path.join(o.dest, "commit", `${oldest}.html`)));
+
+ // `fresh` (--force) renders every page, whatever the cache holds.
+ o = renderOpts(gitDir, stagit, { cacheDir, cacheKey: "k2", fresh: true });
+ r = await renderHistory(o);
+ assert.equal(r.cached, false);
+ assert.equal(r.rendered, 3);
+
+ // Busy: another live publish holds it (this process's pid stands in).
+ writeFileSync(path.join(cacheDir, "lock"), `${process.pid}\n`);
+ o = renderOpts(gitDir, stagit, { cacheDir, cacheKey: "k2" });
+ r = await renderHistory(o);
+ assert.equal(r.cached, false);
+ assert.equal(calls(log).at(-1), `cache= limit= base=${o.baseUrl} repo=archilyzer.git`, "rendered without -c");
+ assert.match(o.logs.join("\n"), /the render cache is in use by another publish; rendering without it/);
+ assert.ok(existsSync(path.join(cacheDir, "lock")), "the other holder's lock is left alone");
+
+ // Cut off: the holder is gone, so nothing in the cache is trusted.
+ const dead = spawnSync("sh", ["-c", "echo $$"], { encoding: "utf8" }).stdout.trim();
+ writeFileSync(path.join(cacheDir, "lock"), `${dead}\n`);
+ writeFileSync(junk, "junk");
+ assert.equal(await holdHistoryCache(cacheDir), true);
+ assert.ok(!existsSync(junk), "emptied");
+ assert.ok(!existsSync(path.join(cacheDir, "key.json")));
+ rmSync(path.join(cacheDir, "lock"));
+
+ // A render that fails empties the cache and releases it.
+ o = renderOpts(gitDir, fakeStagit(log, { exit: 2 }), { cacheDir, cacheKey: "k2" });
+ await assert.rejects(renderHistory(o), HistoryProblem);
+ assert.deepEqual(readdirSync(cacheDir), []);
+});
+
+test("a stale lock: its pid not running, or over an hour old whoever runs its pid now, is replaced with one line", async () => {
+ const cacheDir = path.join(dir("stale"), "archilyzer", "source-history");
+ mkdirSync(path.join(cacheDir, "out"), { recursive: true });
+ const lock = path.join(cacheDir, "lock");
+ const logs: string[] = [];
+ const onLog = (l: string) => {
+ logs.push(l);
+ };
+ // Live and fresh: busy.
+ writeFileSync(lock, `${process.pid}\n`);
+ assert.equal(await holdHistoryCache(cacheDir, onLog), false);
+ assert.deepEqual(logs, []);
+ // Live, but two hours old: stale.
+ const old = new Date(Date.now() - 2 * HISTORY_LOCK_STALE_MS);
+ utimesSync(lock, old, old);
+ assert.equal(await holdHistoryCache(cacheDir, onLog), true);
+ assert.match(logs.join("\n"), new RegExp(`a stale lock on the render cache \\(pid ${process.pid}, 120 minutes old\\) was replaced; the cache is rendered afresh`));
+ assert.ok(!existsSync(path.join(cacheDir, "out")), "emptied");
+ assert.equal(readFileSync(lock, "utf8"), `${process.pid}\n`, "and held");
+ rmSync(lock);
+ // Not running: stale, whatever its age.
+ const dead = spawnSync("sh", ["-c", "echo $$"], { encoding: "utf8" }).stdout.trim();
+ writeFileSync(lock, `${dead}\n`);
+ logs.length = 0;
+ assert.equal(await holdHistoryCache(cacheDir, onLog), true);
+ assert.match(logs.join("\n"), new RegExp(`\\(pid ${dead}, not running\\) was replaced`));
+});
+
+test("a render cache that cannot be written (a read-only dir, or one that cannot be made) is one line and a render without it", async () => {
+ const { gitDir } = bareRepo(2);
+ const log = path.join(dir("log"), "calls");
+ const stagit = fakeStagit(log);
+ const root = dir("ro");
+ const readOnly = path.join(root, "cache");
+ mkdirSync(readOnly);
+ chmodSync(readOnly, 0o555);
+ const parent = path.join(root, "parent");
+ mkdirSync(parent);
+ chmodSync(parent, 0o555);
+ try {
+ for (const cacheDir of [readOnly, path.join(parent, "archilyzer", "source-history")]) {
+ const o = renderOpts(gitDir, stagit, { cacheDir });
+ const r = await renderHistory(o);
+ assert.equal(r.cached, false);
+ assert.equal(r.shown, 2);
+ assert.equal(calls(log).at(-1), `cache= limit= base=${o.baseUrl} repo=archilyzer.git`, "rendered without -c");
+ assert.match(o.logs.join("\n"), /the render cache .* is unusable \(EACCES\); rendering without it/);
+ assert.equal(readdirSync(path.join(o.dest, "commit")).length, 2);
+ }
+ } finally {
+ chmodSync(readOnly, 0o755);
+ chmodSync(parent, 0o755);
+ }
+});
+
+test("a --no-ff merge of commits older than the cached head: stagit's -c misses them, the count notices, and every page is rendered again", async () => {
+ const { gitDir, work } = bareRepo(2);
+ const log = path.join(dir("log"), "calls");
+ const stagit = fakeStagit(log);
+ const cacheDir = path.join(dir("merge-cache"), "archilyzer", "source-history");
+ // A branch whose commits are dated before the head the cache will name.
+ const base = gitIn(work, "rev-parse", "HEAD~1");
+ gitIn(work, "checkout", "-q", "-b", "side", base);
+ for (const i of [7, 8]) {
+ writeFileSync(path.join(work, `side-${i}.txt`), `${i}\n`);
+ gitIn(work, "add", "-A");
+ execFileSync("git", ["commit", "-q", "-m", `side ${i}`], {
+ cwd: work,
+ stdio: "pipe",
+ env: { ...process.env, GIT_COMMITTER_DATE: `2001-01-0${i - 6}T00:00:00Z`, GIT_AUTHOR_DATE: `2001-01-0${i - 6}T00:00:00Z` },
+ });
+ }
+ gitIn(work, "checkout", "-q", "main");
+ let o = renderOpts(gitDir, stagit, { cacheDir });
+ await renderHistory(o);
+ gitIn(work, "merge", "-q", "--no-ff", "-m", "merge side", "side");
+ gitIn(TMP, "--git-dir", gitDir, "fetch", "-q", work, "main:main");
+ o = renderOpts(gitDir, stagit, { cacheDir });
+ const r = await renderHistory(o);
+ assert.equal(r.shown, 5);
+ assert.equal(r.cached, false);
+ assert.match(o.logs.join("\n"), /the cached render does not cover this head \(stagit's log lists 3 commits, not 5 .*\) — stagit's -c stops at the last head it rendered, and a merge of older commits falls behind it; rendering every page again/);
+ assert.equal(readdirSync(path.join(o.dest, "commit")).length, 5);
+});
+
+// ── the real stagit ─────────────────────────────────────────────────────────
+
+test("the real stagit: a page per commit, every link into file/ now into the tree, every relative link resolves", async (t) => {
+ const stagit = resolveStagit(process.env.STAGIT_BIN ?? "stagit", process.env);
+ if (!stagit) return t.skip("stagit is not installed (STAGIT_BIN, PATH, ~/.local/bin)");
+ const { gitDir, work } = bareRepo(1);
+ // A file that main no longer has: its diffs are published, its links not.
+ writeFileSync(path.join(work, "gone.txt"), "soon gone\n");
+ gitIn(work, "add", "-A");
+ gitIn(work, "commit", "-q", "-m", "add gone.txt");
+ gitIn(work, "rm", "-q", "gone.txt");
+ gitIn(work, "commit", "-q", "-m", "remove gone.txt");
+ gitIn(TMP, "--git-dir", gitDir, "fetch", "-q", work, "main:main");
+ const treeFiles = new Set(["README.md", "app/[slug]/page.tsx"]);
+ const o = renderOpts(gitDir, stagit, { cacheDir: path.join(dir("real-root"), "archilyzer", "source-history"), treeFiles });
+ const r = await renderHistory(o);
+ assert.equal(r.rendered, 3);
+ assert.equal(readdirSync(path.join(o.dest, "commit")).length, 3);
+ assert.ok(!existsSync(path.join(o.dest, "file")));
+ for (const page of ["log.html", "files.html", "refs.html", ...readdirSync(path.join(o.dest, "commit")).map((f) => `commit/${f}`)]) {
+ const html = readFileSync(path.join(o.dest, page), "utf8");
+ assert.ok(html.includes(`<body>\n${HISTORY_BACK_LINK}\n`), page);
+ assert.ok(html.includes("<script>t()</script>\n</head>"), page);
+ assert.ok(!/href="(?:\.\.\/)*file\//.test(html), `${page} links into file/`);
+ assert.ok(!/(?:logo|favicon)\.png"/.test(html), `${page} names stagit's images`);
+ for (const m of html.matchAll(/(?:href|src)="([^"#:]+)"/g)) {
+ const target = m[1];
+ if (target.startsWith("/")) continue; // the site's own: /source/, the icon
+ const resolved = path.posix
+ .normalize(path.posix.join(path.posix.dirname(`source/git/${page}`), target))
+ .replace(/\/$/, "");
+ if (resolved.startsWith("source/tree/")) {
+ // Into the raw tree: only a file main has (`gone.txt` is text).
+ assert.ok(treeFiles.has(decodeURIComponent(resolved.slice("source/tree/".length))), `${page}: ${target}`);
+ } else if (resolved.startsWith("source/git/")) {
+ assert.ok(existsSync(path.join(path.dirname(path.dirname(o.dest)), resolved)), `${page}: ${target} → ${resolved}`);
+ } else {
+ assert.equal(resolved, "source", `${page}: ${target}`); // the logo's link, to /source/
+ }
+ }
+ }
+ const pages = readdirSync(path.join(o.dest, "commit")).map((f) => readFileSync(path.join(o.dest, "commit", f), "utf8"));
+ assert.ok(pages.some((h) => /<a id="h0">gone\.txt<\/a>/.test(h)), "gone.txt's diff header is text, its #h0 target kept");
+ const logHtml = readFileSync(path.join(o.dest, "log.html"), "utf8");
+ assert.match(logHtml, /<title>Log - archilyzer - Archilyzer<\/title>/);
+ assert.match(logHtml, /git clone <a href="https:\/\/archilyzer\.pages\.dev\/source\/archilyzer\.git">/);
+ assert.match(readFileSync(path.join(o.dest, "atom.xml"), "utf8"), /href="https:\/\/archilyzer\.pages\.dev\/source\/git\/commit\/[0-9a-f]{40}\.html"/);
+
+ // The cap, by stagit itself: -l keeps the NEWEST in the log (and says how
+ // many more), writes a page for every commit, and only the listed are
+ // published.
+ const capped = renderOpts(gitDir, stagit, { maxCommits: 2, treeFiles });
+ const rc = await renderHistory(capped);
+ const newest = gitIn(TMP, "--git-dir", gitDir, "rev-list", "--max-count=2", "main").split("\n");
+ assert.equal(rc.shown, 2);
+ assert.deepEqual(loggedCommits(readFileSync(path.join(capped.dest, "log.html"), "utf8")), newest);
+ assert.deepEqual(readdirSync(path.join(capped.dest, "commit")).sort(), newest.map((c) => `${c}.html`).sort());
+ assert.equal(readdirSync(path.join(capped.scratch, "history", "commit")).length, 3);
+ assert.match(readFileSync(path.join(capped.dest, "log.html"), "utf8"), /1 more commits remaining, fetch the repository/);
+ // The oldest published page's parent has no page: its link is text.
+ const oldestShown = readFileSync(path.join(capped.dest, "commit", `${newest[1]}.html`), "utf8");
+ const parent = gitIn(TMP, "--git-dir", gitDir, "rev-parse", `${newest[1]}^`);
+ assert.ok(oldestShown.includes(`<b>parent</b> <a>${parent}</a>`), "the parent past the cap is text");
+ assert.ok(!oldestShown.includes(`commit/${parent}.html`));
+});
diff --git a/common/publish/sourceHistory.ts b/common/publish/sourceHistory.ts
@@ -0,0 +1,767 @@
+// The source's history pages: stagit's rendering of the scrubbed mirror,
+// published at /source/git/ by `archilyzer source publish` (source.ts), which
+// runs this after the mirror is built and its objects audited, and before the
+// staged files are audited — so every page goes through the same gate as the
+// mirror, the tree and the tarball.
+//
+// stagit (codemadness.org, C over libgit2) is an OPERATOR-INSTALLED tool, like
+// git-filter-repo: never vendored, never committed. It is found as STAGIT_BIN,
+// else `stagit` on PATH, else ~/.local/bin/stagit (resolveStagit). WITHOUT IT
+// THE PUBLISH GOES ON: one log line says so and how to install it, and the
+// source is published without /source/git/ (the /source/ page then shows no
+// History links). A render that fails, or pages over the host's limits, are
+// the same: a WARNING, and no history — never a failed build.
+//
+// WHAT IS PUBLISHED, from an allowlist of what stagit writes:
+// log.html, files.html, refs.html, atom.xml, tags.xml, commit/<sha>.html
+// and style.css, written here from common/styles/tokens.css. NOT stagit's
+// per-file pages (file/…): browsing is the raw tree at /source/tree/, so
+// every link into file/ is rewritten to it and file/ is never staged.
+//
+// THE POST-PASS (rewriteHistoryPage) touches every .html page and never the
+// two feeds. It adds exactly two things — the homepage's pre-paint theme
+// script (lib/themeConfig.ts buildThemeScript, the string the homepage's
+// ThemeScript emits), and one line at the top linking back to /source/ — and
+// points two kinds of stagit link somewhere that exists: `…file/<path>.html`
+// at the raw tree, and its logo.png / favicon.png at the site's own icon.
+// stagit's header (the name, the description, the clone line, Log | Files |
+// Refs) stays as stagit writes it.
+//
+// THE CAP. At most SOURCE_HISTORY_MAX_COMMITS commits, the newest, get a page:
+// past it stagit runs with `-l`, the log lists that many and says how many
+// more there are, and only the listed commits' pages are published (stagit
+// still writes one for every commit). The /source/ page then says "the latest
+// N of M". The 15,000-file drop in source.ts stays, as the last resort.
+//
+// THE CACHE. stagit's `-c <cachefile>` renders incrementally: it walks from
+// HEAD to the commit the cache names and keeps every commit page already in
+// its output directory (`-l` keeps them too; it cannot be combined with
+// `-c`). So the cache is a DIRECTORY kept between publishes —
+// `${XDG_CACHE_HOME:-~/.cache}/archilyzer/source-history/`
+// (paths.sourceHistoryCacheDir; never inside the checkout or the public dir)
+// holding the cache file, stagit's output and a key. Its pages are kept only
+// when the key matches (the scrub rules and step, filter-repo, stagit, the
+// header text) and the last run finished (the key is removed before a render
+// and written after it); its log lines only when the commit they end at is an
+// ancestor of today's head. One publish holds it at a time (`lock`, the
+// holder's pid); a second renders without it, and a lock whose pid is not
+// running or that is over an hour old is stale and replaced. A cache that
+// cannot be written (EACCES, EROFS, ENOSPC) is one line and a render without
+// it. `--force` renders it afresh, `--check` never touches it, and a refusal
+// removes it.
+//
+// LINKS TO WHAT IS NOT PUBLISHED become text: a diff's file that main no
+// longer has, a commit with no page (past the cap). The raw tree's file list
+// decides (`treeFiles`).
+
+import { createHash } from "node:crypto";
+import { accessSync, constants, existsSync, statSync } from "node:fs";
+import { cp, mkdir, readdir, readFile, realpath, rm, stat, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { PROJECT_NAME } from "../lib/project";
+import { buildThemeScript, HOMEPAGE_DEFAULT_BASE } from "../lib/themeConfig";
+import { onPath, tildify } from "./sourceAudit";
+
+/** How to install stagit, as the log line and the doctor say it. */
+export const STAGIT_INSTALL =
+ "git clone git://git.codemadness.org/stagit && make -C stagit && cp stagit/stagit ~/.local/bin/";
+
+/**
+ * The cap: at most this many commits — the newest — get a page and a log
+ * line (stagit `-l`); the log says how many more there are. One file per
+ * commit counts against the step's 15,000 (Pages' 20,000).
+ */
+export const SOURCE_HISTORY_MAX_COMMITS = 10_000;
+
+/** The site's own 32 px icon, which stagit's logo.png and favicon.png become. */
+export const SITE_ICON_HREF = "/icons/icon-32.png";
+
+/**
+ * The one line put at the top of every page. ASCII only (`·`): the
+ * pages are rewritten byte for byte (latin1 in, latin1 out), so anything
+ * injected must be the same bytes in every encoding.
+ */
+export const HISTORY_BACK_LINK = `<p class="archilyzer-source"><a href="/source/">${PROJECT_NAME} · Source</a></p>`;
+
+/** The pre-paint script the homepage emits (homepage/app/layout.tsx). */
+export function homepageThemeScript(): string {
+ return buildThemeScript({ defaultBase: HOMEPAGE_DEFAULT_BASE });
+}
+
+// What stagit writes that is published: the top-level pages and feeds, and
+// one page per commit.
+const TOP_FILES = ["log.html", "files.html", "refs.html", "atom.xml", "tags.xml"] as const;
+const COMMIT_PAGE = /^[0-9a-f]{40}\.html$/;
+
+/** Something about the history went wrong: a WARNING, and no history. */
+export class HistoryProblem extends Error {
+ constructor(message: string) {
+ super(message);
+ this.name = "HistoryProblem";
+ }
+}
+
+// ── the binary ──────────────────────────────────────────────────────────────
+
+function isExecutableFile(p: string): boolean {
+ try {
+ accessSync(p, constants.X_OK);
+ return statSync(p).isFile();
+ } catch {
+ return false;
+ }
+}
+
+/**
+ * The stagit binary to run, or null. `bin` is paths.stagitBin: STAGIT_BIN,
+ * else "stagit". A name with a slash is that file, or nothing. A bare name is
+ * looked up on PATH, then in ~/.local/bin (the editor's process may not have
+ * it on its PATH, where a hand-built tool is usually put).
+ */
+export function resolveStagit(bin: string, env: NodeJS.ProcessEnv, homeDir: string = os.homedir()): string | null {
+ if (bin.includes("/")) return isExecutableFile(bin) ? bin : null;
+ const found = onPath(bin, env.PATH);
+ if (found) return found;
+ const local = path.join(homeDir, ".local", "bin", bin);
+ return isExecutableFile(local) ? local : null;
+}
+
+/**
+ * Which stagit renders: "absent", else the first 12 hex of its binary's
+ * sha256 (stagit has no version flag). Part of the publish's skip key and of
+ * the cache's, and the manifest's `history.tool`. Never the path: the
+ * manifest is published, and a path under the home dir is a denied literal.
+ */
+export async function stagitIdentity(found: string | null): Promise<string> {
+ if (!found) return "absent";
+ const bytes = await readFile(await realpath(found));
+ return `stagit (sha256 ${createHash("sha256").update(bytes).digest("hex").slice(0, 12)})`;
+}
+
+// ── the stylesheet ──────────────────────────────────────────────────────────
+
+// The tokens the stylesheet reads, on each base. `--brand` is Signal on both
+// (the homepage's accent: the base blocks default to it), and every token a
+// value names through var() comes along.
+export const HISTORY_TOKENS = [
+ "--background",
+ "--surface",
+ "--foreground",
+ "--muted-foreground",
+ "--faint",
+ "--border-strong",
+ "--brand",
+ "--brand-soft",
+ "--info",
+ "--success",
+ "--destructive",
+] as const;
+
+type Decls = Map<string, string>;
+
+function declarations(body: string): Decls {
+ const out: Decls = new Map();
+ for (const part of body.split(";")) {
+ const i = part.indexOf(":");
+ if (i === -1) continue;
+ const name = part.slice(0, i).trim();
+ if (name.startsWith("--") || name === "color-scheme") out.set(name, part.slice(i + 1).trim());
+ }
+ return out;
+}
+
+/**
+ * The light and the dark base blocks of tokens.css, as declarations: the rule
+ * whose selector list holds `html[data-base="light"]`, and the one holding
+ * `html[data-base="dark"]`. A file without both is an error.
+ */
+export function tokenBlocks(tokensCss: string): { light: Decls; dark: Decls } {
+ const text = tokensCss.replace(/\/\*[\s\S]*?\*\//g, "");
+ let light: Decls | null = null;
+ let dark: Decls | null = null;
+ for (const m of text.matchAll(/([^{}]+)\{([^{}]*)\}/g)) {
+ const selectors = m[1].split(",").map((s) => s.trim());
+ if (selectors.includes('html[data-base="light"]')) light = declarations(m[2]);
+ else if (selectors.includes('html[data-base="dark"]')) dark = declarations(m[2]);
+ }
+ if (!light || !dark) throw new HistoryProblem("tokens.css has no light or no dark base block");
+ return { light, dark };
+}
+
+// `names` and every custom property their values reach through var(), in
+// the block's own order, as ` name: value;` lines.
+function closure(block: Decls, names: readonly string[], base: string): string {
+ const want = new Set<string>(["color-scheme"]);
+ const visit = (name: string) => {
+ if (want.has(name)) return;
+ const value = block.get(name);
+ if (value === undefined) throw new HistoryProblem(`tokens.css's ${base} block has no ${name}`);
+ want.add(name);
+ for (const ref of value.matchAll(/var\(\s*(--[\w-]+)/g)) visit(ref[1]);
+ };
+ for (const n of names) visit(n);
+ return [...block].filter(([k]) => want.has(k)).map(([k, v]) => ` ${k}: ${v};`).join("\n");
+}
+
+/**
+ * style.css for the history pages, from common/styles/tokens.css: the
+ * homepage's two grounds. Without the theme script (no JS) the pages follow
+ * `prefers-color-scheme`; with it, `html[data-base]` is the visitor's stored
+ * choice, or the homepage's default. The rules are stagit's own stylesheet,
+ * recoloured: links in the accent, diff insertions in --success and deletions
+ * in --destructive (each line also keeps its + or − sign).
+ */
+export function historyStylesheet(tokensCss: string): string {
+ const { light, dark } = tokenBlocks(tokensCss);
+ const lightVars = closure(light, HISTORY_TOKENS, "light");
+ const darkVars = closure(dark, HISTORY_TOKENS, "dark");
+ const indent = (s: string) => s.split("\n").map((l) => ` ${l}`).join("\n");
+ return `/* The source's history pages (stagit), styled by \`archilyzer source publish\`
+ from common/styles/tokens.css — the homepage's two grounds. Generated. */
+:root,
+html[data-base="light"] {
+${lightVars}
+}
+@media (prefers-color-scheme: dark) {
+ :root:not([data-base]) {
+${indent(darkVars)}
+ }
+}
+html[data-base="dark"] {
+${darkVars}
+}
+
+html { background: var(--background); }
+body {
+ margin: 0;
+ padding: 1rem;
+ background: var(--background);
+ color: var(--foreground);
+ font-family: ui-monospace, "IBM Plex Mono", SFMono-Regular, Menlo, Consolas, "Liberation Mono", monospace;
+ font-size: 0.875rem;
+ line-height: 1.5;
+}
+a { color: var(--brand); }
+a:hover { color: var(--foreground); }
+a:not([href]) { color: inherit; text-decoration: none; }
+p.archilyzer-source {
+ margin: 0 0 1rem;
+ font-family: system-ui, -apple-system, "Segoe UI", sans-serif;
+ font-size: 0.8125rem;
+}
+p.archilyzer-source a { color: var(--muted-foreground); text-decoration: none; }
+p.archilyzer-source a:hover { color: var(--foreground); text-decoration: underline; }
+h1, h2, h3, h4, h5, h6 { font-size: 1em; margin: 0; }
+tr.url a { overflow-wrap: anywhere; }
+img, h1, h2 { vertical-align: middle; }
+img { border: 0; }
+a:target { background-color: var(--brand-soft); }
+a.d, a.h, a.i, a.line { text-decoration: none; }
+#blob a { color: var(--faint); }
+#blob a:hover { color: var(--brand); text-decoration: none; }
+table thead td { font-weight: bold; }
+table td { padding: 0 0.4em; }
+#content { overflow-x: auto; }
+#content table td { vertical-align: top; white-space: nowrap; }
+#branches tr:hover td,
+#tags tr:hover td,
+#index tr:hover td,
+#log tr:hover td,
+#files tr:hover td { background-color: var(--surface); }
+#index tr td:nth-child(2),
+#tags tr td:nth-child(3),
+#branches tr td:nth-child(3),
+#log tr td:nth-child(2) { white-space: normal; }
+td.num { text-align: right; }
+.desc { color: var(--muted-foreground); }
+hr { border: 0; border-top: 1px solid var(--border-strong); height: 1px; }
+pre { font-family: inherit; }
+pre a.h { color: var(--info); }
+.A,
+span.i,
+pre a.i { color: var(--success); }
+.D,
+span.d,
+pre a.d { color: var(--destructive); }
+pre a.h:hover,
+pre a.i:hover,
+pre a.d:hover { text-decoration: none; }
+`;
+}
+
+// ── the post-pass ───────────────────────────────────────────────────────────
+
+/**
+ * What is published beside the pages, for the post-pass to link to only what
+ * is there: a path of the raw tree (decoded, as tracked), a commit with a page.
+ */
+export type PublishedSet = {
+ tree: (path: string) => boolean;
+ commit: (sha: string) => boolean;
+};
+
+// stagit's percent-encoding undone; null when it is not valid.
+function decodedPath(p: string): string | null {
+ try {
+ return decodeURIComponent(p);
+ } catch {
+ return null;
+ }
+}
+
+/**
+ * One stagit page, as published. Adds the theme script before `</head>` and
+ * HISTORY_BACK_LINK after `<body>`; points every `href="…file/<path>.html"`
+ * (the Files index, the header's README and LICENSE, a diff's file names) at
+ * the raw tree — `../tree/<path>` from the same depth, the path as stagit
+ * encoded it — and stagit's logo.png and favicon.png at the site's icon.
+ *
+ * With `published`, a link to what is not published loses its `href` and
+ * stays as text (an `<a>` with no `href`, its `id` kept — a diff header is the
+ * diffstat's `#h<n>` target): a file no longer in main (a diff of a deleted or
+ * renamed file), and a commit with no page (past the cap, the oldest page's
+ * parent). Nothing else changes. Page text cannot fake an `href="…"`: stagit
+ * encodes every `"` it prints from the repository as `"`.
+ */
+export function rewriteHistoryPage(html: string, themeScript: string, published?: PublishedSet): string {
+ if (/<\/script/i.test(themeScript)) throw new Error("the theme script may not close its own element");
+ let out = html;
+ const head = out.indexOf("</head>");
+ if (head !== -1) out = `${out.slice(0, head)}<script>${themeScript}</script>\n${out.slice(head)}`;
+ const body = /<body[^>]*>\n?/.exec(out);
+ if (body) {
+ const at = body.index + body[0].length;
+ out = `${out.slice(0, at)}${HISTORY_BACK_LINK}\n${out.slice(at)}`;
+ }
+ out = out
+ .replace(/ href="((?:\.\.\/)*)file\/([^"]*)\.html"/g, (_m, up: string, p: string) => {
+ if (published) {
+ const tracked = decodedPath(p);
+ if (tracked === null || !published.tree(tracked)) return "";
+ }
+ return ` href="${up}../tree/${p}"`;
+ })
+ .replace(/src="(?:\.\.\/)*logo\.png"/g, `src="${SITE_ICON_HREF}"`)
+ .replace(/href="(?:\.\.\/)*favicon\.png"/g, `href="${SITE_ICON_HREF}"`);
+ if (published) {
+ out = out.replace(/ href="((?:\.\.\/)*)commit\/([0-9a-f]{40})\.html"/g, (m, _up: string, sha: string) =>
+ published.commit(sha) ? m : "",
+ );
+ }
+ return out;
+}
+
+/**
+ * The commits the log lists, newest first: every `href="commit/<sha>.html"`
+ * in log.html (stagit writes one per log line; a commit's own text cannot
+ * fake one, since stagit encodes every `"` it prints as `"`). These, and
+ * only these, have their pages published.
+ */
+export function loggedCommits(logHtml: string): string[] {
+ const seen = new Set<string>();
+ for (const m of logHtml.matchAll(/<a href="commit\/([0-9a-f]{40})\.html">/g)) seen.add(m[1]);
+ return [...seen];
+}
+
+/**
+ * Copy the allowlist of stagit's output from `work` into `dest` — the
+ * top-level pages and feeds, and the page of each commit in `commits` (the
+ * ones the log lists; any other page in `work` stays there) — each page
+ * through the post-pass (byte for byte otherwise: read and written as latin1,
+ * so a diff of a file that is not UTF-8 keeps its bytes), then style.css.
+ * The feeds are copied as they are. A listed page that is not there is a
+ * HistoryProblem.
+ */
+export async function stageHistory(
+ work: string,
+ dest: string,
+ o: {
+ themeScript: string;
+ stylesheet: string;
+ commits: readonly string[];
+ // The raw tree's files (as tracked); absent, no tree link is dropped.
+ treeFiles?: ReadonlySet<string>;
+ },
+): Promise<{ files: number; bytes: number; largest: { rel: string; bytes: number } }> {
+ if (/[^\x00-\x7f]/.test(o.themeScript)) throw new Error("the theme script must be ASCII");
+ const rels: string[] = [];
+ for (const f of TOP_FILES) {
+ if (!existsSync(path.join(work, f))) throw new HistoryProblem(`stagit wrote no ${f}`);
+ rels.push(f);
+ }
+ for (const sha of o.commits) {
+ const rel = `commit/${sha}.html`;
+ if (!COMMIT_PAGE.test(`${sha}.html`)) throw new HistoryProblem(`the log names ${sha.slice(0, 40)}, not a commit id`);
+ if (!existsSync(path.join(work, rel))) throw new HistoryProblem(`the log lists ${sha.slice(0, 12)}, whose page is not there`);
+ rels.push(rel);
+ }
+ const pages = new Set(o.commits);
+ const treeFiles = o.treeFiles;
+ const published: PublishedSet | undefined = treeFiles
+ ? { tree: (p) => treeFiles.has(p), commit: (sha) => pages.has(sha) }
+ : undefined;
+ await mkdir(path.join(dest, "commit"), { recursive: true });
+ let bytes = 0;
+ let largest = { rel: "", bytes: -1 };
+ const note = (rel: string, n: number) => {
+ bytes += n;
+ if (n > largest.bytes) largest = { rel, bytes: n };
+ };
+ for (const rel of rels) {
+ const src = path.join(work, rel);
+ const dst = path.join(dest, rel);
+ if (rel.endsWith(".html")) {
+ const text = rewriteHistoryPage((await readFile(src)).toString("latin1"), o.themeScript, published);
+ const buf = Buffer.from(text, "latin1");
+ await writeFile(dst, buf);
+ note(rel, buf.length);
+ } else {
+ await cp(src, dst);
+ note(rel, (await stat(dst)).size);
+ }
+ }
+ await writeFile(path.join(dest, "style.css"), o.stylesheet);
+ note("style.css", Buffer.byteLength(o.stylesheet));
+ return { files: rels.length + 1, bytes, largest };
+}
+
+// ── the render, with its cache ──────────────────────────────────────────────
+
+export type HistoryRun = (
+ command: string,
+ args: string[],
+ o: { cwd: string; timeoutMs: number },
+) => Promise<{ code: number; out: string[] }>;
+
+export type RenderHistoryOpts = {
+ stagit: string;
+ // The scrubbed bare clone — a directory named archilyzer.git (stagit names
+ // the repository after it) — at `head`, with `commits` commits in all.
+ gitDir: string;
+ head: string;
+ commits: number;
+ // The cap: at most this many commits (the newest) get a page and a log
+ // line. SOURCE_HISTORY_MAX_COMMITS, but for the tests.
+ maxCommits: number;
+ // Where the published copy goes (stage/source/git).
+ dest: string;
+ // The publish's own scratch dir: the render's directory when there is no
+ // cache to use.
+ scratch: string;
+ // The kept cache directory, or null (a `--check`, or a cache dir the step
+ // may not use).
+ cacheDir: string | null;
+ // Changes whenever pages rendered before must not be kept.
+ cacheKey: string;
+ // `--force`: render every page again.
+ fresh: boolean;
+ // stagit's header: the description line and the clone URL.
+ description: string;
+ cloneUrl: string;
+ // The Atom feeds' absolute base (`<site>/source/git/`).
+ baseUrl: string;
+ stylesheet: string;
+ themeScript: string;
+ // The raw tree's files, as tracked: a page's link to a file not among them
+ // (deleted or renamed since) becomes text. Absent: every link is kept.
+ treeFiles?: ReadonlySet<string>;
+ // A child, run with the publish's environment and cancel signal.
+ run: HistoryRun;
+ onLog: (line: string) => void;
+};
+
+export type RenderedHistory = {
+ files: number;
+ bytes: number;
+ largest: { rel: string; bytes: number };
+ // The commits with a page — min(commits, maxCommits), the newest.
+ shown: number;
+ // Pages stagit wrote this run (all of them without a usable cache).
+ rendered: number;
+ cached: boolean;
+};
+
+const pidAlive = (pid: number): boolean => {
+ try {
+ process.kill(pid, 0);
+ return true;
+ } catch (err) {
+ return (err as NodeJS.ErrnoException).code === "EPERM";
+ }
+};
+
+/**
+ * A lock this old is stale whoever holds its pid now: a publish holds the
+ * cache for one render (stagit's timeout is 10 minutes), and a pid is reused.
+ */
+export const HISTORY_LOCK_STALE_MS = 60 * 60 * 1000;
+
+/**
+ * Hold the cache directory, or say it is busy. The directory is made on the
+ * way (recursively). The lock names its holder's pid. A lock is stale when
+ * that pid is not running OR the lock is older than HISTORY_LOCK_STALE_MS: a
+ * render was cut off, so nothing in the directory is trusted — it is emptied
+ * and taken, with one line. An I/O error (an unwritable or full cache dir)
+ * is thrown: renderHistory then renders without the cache.
+ */
+export async function holdHistoryCache(
+ dir: string,
+ onLog: (line: string) => void = () => {},
+ now: () => number = Date.now,
+): Promise<boolean> {
+ await mkdir(dir, { recursive: true, mode: 0o700 });
+ const lock = path.join(dir, "lock");
+ for (let attempt = 0; attempt < 2; attempt++) {
+ try {
+ await writeFile(lock, `${process.pid}\n`, { flag: "wx" });
+ return true;
+ } catch (err) {
+ if ((err as NodeJS.ErrnoException).code !== "EEXIST") throw err;
+ const pid = Number((await readFile(lock, "utf8").catch(() => "")).trim());
+ const mtime = (await stat(lock).catch(() => null))?.mtimeMs ?? now();
+ const age = now() - mtime;
+ const running = Number.isInteger(pid) && pid > 0 && pidAlive(pid);
+ if (running && age < HISTORY_LOCK_STALE_MS) return false;
+ onLog(
+ `[source] history: a stale lock on the render cache (pid ${pid > 0 ? pid : "unknown"}, ` +
+ `${running ? `${Math.round(age / 60_000)} minutes old` : "not running"}) was replaced; the cache is rendered afresh`,
+ );
+ await emptyDir(dir);
+ }
+ }
+ return false;
+}
+
+async function emptyDir(dir: string): Promise<void> {
+ for (const f of await readdir(dir).catch(() => [] as string[])) {
+ await rm(path.join(dir, f), { recursive: true, force: true });
+ }
+}
+
+async function releaseHistoryCache(dir: string): Promise<void> {
+ await rm(path.join(dir, "lock"), { force: true });
+}
+
+/**
+ * Remove the cache (a refusal: it was rendered under rules that may not be
+ * today's). Left alone while another publish holds it.
+ */
+export async function dropHistoryCache(dir: string): Promise<void> {
+ if (!existsSync(dir)) return;
+ if (await holdHistoryCache(dir)) await rm(dir, { recursive: true, force: true });
+}
+
+async function countCommitPages(work: string): Promise<number> {
+ const names = await readdir(path.join(work, "commit")).catch(() => [] as string[]);
+ return names.filter((f) => COMMIT_PAGE.test(f)).length;
+}
+
+/**
+ * Render the history into `o.dest`. Throws HistoryProblem when stagit fails
+ * or its output is not what it should be; the caller publishes without the
+ * history then.
+ *
+ * THE CAP. Up to `maxCommits` commits, stagit runs with `-c` (its log lines
+ * cached: a publish renders only the new commits). Past it, with `-l
+ * <maxCommits>`: the log lists the newest `maxCommits` and says how many more
+ * there are ("N more commits remaining, fetch the repository"). stagit refuses
+ * `-c` with `-l`, and `-l` still writes a page for EVERY commit — so what is
+ * published is the page of each commit the log lists, and no other. The
+ * pages already in the cache's directory are kept by stagit either way.
+ */
+export async function renderHistory(o: RenderHistoryOpts): Promise<RenderedHistory> {
+ // stagit's header reads both from the repository directory. Neither is
+ // published as a file (the mirror is staged from an allowlist). No newline
+ // after the description: stagit keeps it, in the <title> and the header.
+ await writeFile(path.join(o.gitDir, "description"), o.description);
+ await writeFile(path.join(o.gitDir, "url"), `${o.cloneUrl}\n`);
+
+ const withoutCache = async (): Promise<RenderedHistory> => ({
+ ...(await renderOnce(o, path.join(o.scratch, "history"), null)),
+ cached: false,
+ });
+ const dir = o.cacheDir;
+ if (!dir) return withoutCache();
+ // A cache that cannot be used (unwritable, read-only, full) is one line and
+ // a render without it: never a failed build.
+ const unusable = (err: unknown) =>
+ o.onLog(`[source] history: the render cache ${tildify(dir)} is unusable (${ioCode(err)}); rendering without it`);
+ let held: boolean;
+ try {
+ held = await holdHistoryCache(dir, o.onLog);
+ } catch (err) {
+ if (!isIoError(err)) throw err;
+ unusable(err);
+ return withoutCache();
+ }
+ if (!held) {
+ o.onLog("[source] history: the render cache is in use by another publish; rendering without it");
+ return withoutCache();
+ }
+ try {
+ const keyFile = path.join(dir, "key.json");
+ const cacheFile = path.join(dir, "stagit.cache");
+ const work = path.join(dir, "out");
+ let usable: boolean;
+ try {
+ usable = !o.fresh && (await cacheUsable(o, keyFile, work));
+ if (!usable) await emptyCache(dir);
+ else await dropStaleLogCache(o, dir, cacheFile);
+ // The key goes before the render and comes back after it: a render cut
+ // off leaves none, and the next publish starts over.
+ await rm(keyFile, { force: true });
+ } catch (err) {
+ if (!isIoError(err)) throw err;
+ unusable(err);
+ return await withoutCache();
+ }
+ try {
+ const r = await renderOnce(o, work, cacheFile);
+ await writeKey(o, keyFile);
+ return { ...r, cached: usable };
+ } catch (err) {
+ if (!(err instanceof HistoryProblem) || !usable) throw err;
+ // Not a fault: stagit's -c walk is in commit-date order and stops at the
+ // head it rendered last, so the commits of a merge that are older than
+ // that head are left out, and the log comes up short. Once more, every
+ // page.
+ o.onLog(
+ `[source] history: the cached render does not cover this head (${err.message}) — stagit's -c stops at the last head it rendered, and a merge of older commits falls behind it; rendering every page again`,
+ );
+ try {
+ await emptyCache(dir);
+ } catch (ioErr) {
+ if (!isIoError(ioErr)) throw ioErr;
+ unusable(ioErr);
+ return await withoutCache();
+ }
+ const r = await renderOnce(o, work, cacheFile);
+ await writeKey(o, keyFile);
+ return { ...r, cached: false };
+ }
+ } catch (err) {
+ await emptyCache(dir).catch(() => {});
+ throw err;
+ } finally {
+ await releaseHistoryCache(dir).catch(() => {});
+ }
+}
+
+// An I/O error from the file system (it carries an errno code), as opposed to
+// the history's own problems, a cancel, or a bug.
+function isIoError(err: unknown): err is NodeJS.ErrnoException {
+ return err instanceof Error && !(err instanceof HistoryProblem) && typeof (err as NodeJS.ErrnoException).code === "string";
+}
+
+function ioCode(err: unknown): string {
+ return (err as NodeJS.ErrnoException)?.code ?? "an I/O error";
+}
+
+// The key, after a render that finished. A key that cannot be written is no
+// key: the next publish renders every page again.
+async function writeKey(o: RenderHistoryOpts, keyFile: string): Promise<void> {
+ try {
+ await writeFile(keyFile, JSON.stringify({ key: o.cacheKey }) + "\n");
+ } catch (err) {
+ if (!isIoError(err)) throw err;
+ o.onLog(`[source] history: the render cache's key could not be written (${ioCode(err)}); the next publish renders every page again`);
+ }
+}
+
+// Everything in the cache but its lock.
+async function emptyCache(dir: string): Promise<void> {
+ for (const f of await readdir(dir).catch(() => [] as string[])) {
+ if (f !== "lock") await rm(path.join(dir, f), { recursive: true, force: true });
+ }
+}
+
+// The cache's pages can be kept: the same key, and a last run that finished.
+// (A page is its commit's, by id; pages of commits no longer in history are
+// never published, since only the log's commits are.)
+async function cacheUsable(o: RenderHistoryOpts, keyFile: string, work: string): Promise<boolean> {
+ let key: unknown = null;
+ try {
+ key = (JSON.parse(await readFile(keyFile, "utf8")) as { key?: unknown }).key;
+ } catch {
+ return false;
+ }
+ return key === o.cacheKey && existsSync(path.join(work, "log.html"));
+}
+
+// stagit's `-c` file holds the log lines down to the commit it names, and
+// stagit appends them to the new ones: when that commit is not an ancestor
+// of today's head (a main rewritten since), the lines are of another history,
+// and the file goes (the pages stay).
+async function dropStaleLogCache(o: RenderHistoryOpts, dir: string, cacheFile: string): Promise<void> {
+ if (!existsSync(cacheFile)) return;
+ const last = (await readFile(cacheFile, "utf8").catch(() => "")).split("\n")[0].trim();
+ const ancestor =
+ /^[0-9a-f]{40}$/.test(last) &&
+ (await o.run("git", ["--git-dir", o.gitDir, "merge-base", "--is-ancestor", last, o.head], {
+ cwd: dir,
+ timeoutMs: 30_000,
+ })).code === 0;
+ if (!ancestor) await rm(cacheFile, { force: true });
+}
+
+async function renderOnce(
+ o: RenderHistoryOpts,
+ work: string,
+ cacheFile: string | null,
+): Promise<Omit<RenderedHistory, "cached">> {
+ try {
+ await mkdir(work, { recursive: true });
+ } catch (err) {
+ if (!isIoError(err)) throw err;
+ throw new HistoryProblem(`the render directory cannot be made (${ioCode(err)})`);
+ }
+ const before = await countCommitPages(work);
+ const capped = o.commits > o.maxCommits;
+ const shown = Math.min(o.commits, o.maxCommits);
+ const args = [
+ ...(capped ? ["-l", String(o.maxCommits)] : cacheFile ? ["-c", cacheFile] : []),
+ "-u",
+ o.baseUrl,
+ o.gitDir,
+ ];
+ const r = await o.run(o.stagit, args, { cwd: work, timeoutMs: 600_000 });
+ // The per-file pages are never published, and stagit writes them all again
+ // on every run: none is kept.
+ await rm(path.join(work, "file"), { recursive: true, force: true }).catch(() => {});
+ if (r.code !== 0) {
+ const tail = r.out.filter((l) => l.trim()).slice(-2).join(" / ");
+ throw new HistoryProblem(`stagit exited ${r.code}${tail ? `: ${tail}` : ""}`);
+ }
+ const logPath = path.join(work, "log.html");
+ if (!existsSync(logPath)) throw new HistoryProblem("stagit wrote no log.html");
+ const commits = loggedCommits((await readFile(logPath)).toString("latin1"));
+ if (commits.length !== shown) {
+ throw new HistoryProblem(`stagit's log lists ${commits.length} commits, not ${shown} (${o.commits} in all, at most ${o.maxCommits})`);
+ }
+ if (commits[0] !== o.head) throw new HistoryProblem(`stagit's log does not start at the head`);
+ const staged = await stageHistory(work, o.dest, {
+ themeScript: o.themeScript,
+ stylesheet: o.stylesheet,
+ commits,
+ treeFiles: o.treeFiles,
+ });
+ const after = await countCommitPages(work);
+ return { ...staged, shown, rendered: cacheFile ? after - before : after };
+}
+
+/**
+ * The cache's key: whatever, changed, makes a page rendered before wrong — the
+ * rules (with the step's version) and filter-repo that made the ids, the
+ * stagit that wrote the pages, and the header text every page carries.
+ */
+export function historyCacheKey(parts: {
+ rulesHash: string;
+ filterRepo: string;
+ stagit: string;
+ description: string;
+ cloneUrl: string;
+ baseUrl: string;
+}): string {
+ return createHash("sha256").update(JSON.stringify(parts)).digest("hex");
+}
diff --git a/common/publish/sourceTree.test.ts b/common/publish/sourceTree.test.ts
@@ -0,0 +1,96 @@
+import { test, after } from "node:test";
+import assert from "node:assert/strict";
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { SourceRefusal } from "./sourceAudit";
+import { escapeHtml, hrefFor, renderTreeIndex, sortEntries, writeTreeIndexes } from "./sourceTree";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// The raw tree's directory pages (sourceTree.ts): hrefs a static host can
+// resolve for the repo's real bracketed names, escaped labels, the order,
+// the relative breadcrumbs, and the refusals.
+
+const TMP = mkdtempSync(path.join(os.tmpdir(), "source-tree-"));
+after(() => rmSync(TMP, { recursive: true, force: true }));
+
+const META = { mirrorHead: "0123456789abcdef0123456789abcdef01234567", generatedAt: "2026-09-28T12:00:00.000Z" };
+
+test("hrefs are URL-encoded, a directory's with a trailing slash — the repo's brackets included", () => {
+ assert.equal(hrefFor({ name: "[slug]", dir: true }), "%5Bslug%5D/");
+ assert.equal(hrefFor({ name: "Archivo[wdth,wght].ttf", dir: false }), "Archivo%5Bwdth%2Cwght%5D.ttf");
+ assert.equal(hrefFor({ name: "a b#c?.md", dir: false }), "a%20b%23c%3F.md");
+ assert.equal(hrefFor({ name: "page.tsx", dir: false }), "page.tsx");
+});
+
+test("names are escaped wherever they are printed", () => {
+ assert.equal(escapeHtml(`<a href="x">&'</a>`), "<a href="x">&'</a>");
+ const html = renderTreeIndex({
+ relDir: "x",
+ entries: [{ name: "<script>.ts", dir: false, bytes: 3 }],
+ ...META,
+ });
+ assert.ok(!html.includes("<script>.ts"), "a name never becomes markup");
+ assert.match(html, /href="%3Cscript%3E\.ts"><script>\.ts<\/a>/);
+});
+
+test("directories first, then files, each by name; sizes with raw bytes in the title", () => {
+ const sorted = sortEntries([
+ { name: "b.ts", dir: false, bytes: 1 },
+ { name: "z", dir: true, bytes: 0 },
+ { name: "a.ts", dir: false, bytes: 2048 },
+ { name: "B", dir: true, bytes: 0 },
+ ]);
+ assert.deepEqual(sorted.map((e) => e.name), ["B", "z", "a.ts", "b.ts"]);
+ const html = renderTreeIndex({ relDir: "", entries: sorted, ...META });
+ assert.match(html, /<span title="2048 bytes">2\.0 KB<\/span>/);
+ assert.match(html, /<meta name="robots" content="noindex">/);
+ assert.match(html, /main @ 0123456789ab · generated 2026-09-28T12:00:00\.000Z · <a href="\/source\/">clone<\/a>/);
+});
+
+test("breadcrumbs hop up relatively; the root has no `..` row", () => {
+ const root = renderTreeIndex({ relDir: "", entries: [], ...META });
+ assert.ok(!root.includes(`href="../"`));
+ assert.match(root, /<nav><strong>archilyzer<\/strong><\/nav>/);
+ const deep = renderTreeIndex({ relDir: "homepage/app/[slug]", entries: [], ...META });
+ assert.match(
+ deep,
+ /<nav><a href="\.\.\/\.\.\/\.\.\/">archilyzer<\/a><span class="sep">\/<\/span><a href="\.\.\/\.\.\/">homepage<\/a><span class="sep">\/<\/span><a href="\.\.\/">app<\/a><span class="sep">\/<\/span><strong>\[slug\]<\/strong><\/nav>/,
+ );
+ assert.match(deep, /<tr><td class="name"><a href="\.\.\/">\.\.<\/a>/);
+});
+
+test("writeTreeIndexes pages every directory and counts the tree, refusing a tracked index.html, a 404.html or a symlink", async () => {
+ const tree = path.join(TMP, "t1");
+ mkdirSync(path.join(tree, "app", "[slug]"), { recursive: true });
+ writeFileSync(path.join(tree, "README.md"), "hello\n");
+ writeFileSync(path.join(tree, "app", "[slug]", "page.tsx"), "x");
+ const totals = await writeTreeIndexes(tree, META);
+ assert.deepEqual(totals, { files: 2, dirs: 3, bytes: 7 });
+ for (const d of ["", "app", "app/[slug]"]) assert.ok(existsSync(path.join(tree, d, "index.html")), d);
+ const appPage = readFileSync(path.join(tree, "app", "index.html"), "utf8");
+ assert.match(appPage, /href="%5Bslug%5D\/">\[slug\]\/<\/a>/);
+ assert.ok(!appPage.includes(`>index.html<`), "a page does not list itself");
+
+ const withIndex = path.join(TMP, "t2");
+ mkdirSync(path.join(withIndex, "docs"), { recursive: true });
+ writeFileSync(path.join(withIndex, "docs", "index.html"), "<p>tracked</p>");
+ await assert.rejects(writeTreeIndexes(withIndex, META), (e) => e instanceof SourceRefusal && /docs\/index\.html/.test(e.message));
+
+ // A tracked 404.html anywhere: Pages would serve it, as HTML on this
+ // origin, for every missing path below its directory.
+ const with404 = path.join(TMP, "t4");
+ mkdirSync(path.join(with404, "docs", "deep"), { recursive: true });
+ writeFileSync(path.join(with404, "docs", "deep", "404.html"), "<script>x</script>");
+ await assert.rejects(
+ writeTreeIndexes(with404, META),
+ (e) => e instanceof SourceRefusal && /docs\/deep\/404\.html; Pages would serve it, as HTML/.test(e.message),
+ );
+
+ const withLink = path.join(TMP, "t3");
+ mkdirSync(withLink);
+ symlinkSync("/etc/hostname", path.join(withLink, "leak"));
+ await assert.rejects(writeTreeIndexes(withLink, META), (e) => e instanceof SourceRefusal && /symlink/.test(e.message));
+});
diff --git a/common/publish/sourceTree.ts b/common/publish/sourceTree.ts
@@ -0,0 +1,186 @@
+// The raw tree's directory pages: one dependency-free index.html per directory
+// of the extracted `main`, so /source/tree/ is browsable on a static host.
+//
+// The FILES under the tree are served as text/plain whatever their extension
+// (homepage/public/_headers, `/source/tree/*`); only these generated pages are
+// HTML. Every name is escaped for HTML and every href is URL-encoded — the tree
+// holds Next's dynamic-route directories (`[slug]`) and variable fonts
+// (`Archivo[wdth,wght].ttf`), whose brackets a browser would otherwise send
+// raw.
+
+import { readdir, stat, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { SourceRefusal } from "./sourceAudit";
+
+export type TreeEntry = { name: string; dir: boolean; bytes: number };
+
+export type TreeMeta = { mirrorHead: string; generatedAt: string };
+
+/** The href of one entry, relative to its directory's page. */
+export function hrefFor(entry: Pick<TreeEntry, "name" | "dir">): string {
+ return encodeURIComponent(entry.name) + (entry.dir ? "/" : "");
+}
+
+export function escapeHtml(s: string): string {
+ return s
+ .replace(/&/g, "&")
+ .replace(/</g, "<")
+ .replace(/>/g, ">")
+ .replace(/"/g, """)
+ .replace(/'/g, "'");
+}
+
+export function formatTreeBytes(bytes: number): string {
+ if (bytes < 1024) return `${bytes} B`;
+ if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`;
+ return `${(bytes / (1024 * 1024)).toFixed(2)} MB`;
+}
+
+/** Directories first, then files, each by name (code-unit order: stable everywhere). */
+export function sortEntries(entries: readonly TreeEntry[]): TreeEntry[] {
+ return [...entries].sort((a, b) =>
+ a.dir !== b.dir ? (a.dir ? -1 : 1) : a.name < b.name ? -1 : a.name > b.name ? 1 : 0,
+ );
+}
+
+const CSS = `
+:root { color-scheme: light dark; --fg: #1d1b18; --muted: #6b665e; --bg: #faf8f4;
+ --rule: #e4dfd6; --link: #7a5a1c; --hover: #f1ede5; }
+@media (prefers-color-scheme: dark) {
+ :root { --fg: #ece8e1; --muted: #a39d93; --bg: #161512; --rule: #2e2b26;
+ --link: #d9b46a; --hover: #221f1b; }
+}
+* { box-sizing: border-box; }
+body { margin: 0; background: var(--bg); color: var(--fg);
+ font: 14px/1.5 ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; }
+main { max-width: 64rem; margin: 0 auto; padding: 1.5rem 1rem 3rem; }
+nav { font-size: 1rem; margin-bottom: 1rem; word-break: break-all; }
+nav .sep { color: var(--muted); padding: 0 0.25rem; }
+a { color: var(--link); text-decoration: none; }
+a:hover { text-decoration: underline; }
+table { width: 100%; border-collapse: collapse; }
+td { padding: 0.3rem 0.5rem; border-top: 1px solid var(--rule); }
+tr:hover td { background: var(--hover); }
+td.size { text-align: right; color: var(--muted); white-space: nowrap; width: 1%; }
+td.name { word-break: break-all; }
+footer { margin-top: 1.5rem; color: var(--muted); font-size: 0.8125rem; }
+`.trim();
+
+/**
+ * One directory's page. `relDir` is the directory relative to the tree root
+ * ("" for the root); every link on the page is relative, so it works under
+ * /source/tree/ on any host and from a directory's `index.html` alike.
+ */
+export function renderTreeIndex(opts: {
+ relDir: string;
+ entries: readonly TreeEntry[];
+ mirrorHead: string;
+ generatedAt: string;
+}): string {
+ const segs = opts.relDir ? opts.relDir.split("/") : [];
+ const depth = segs.length;
+ const up = (n: number) => (n === 0 ? "./" : "../".repeat(n));
+ const crumbs: string[] = [
+ depth === 0
+ ? `<strong>archilyzer</strong>`
+ : `<a href="${up(depth)}">archilyzer</a>`,
+ ];
+ segs.forEach((seg, i) => {
+ const last = i === depth - 1;
+ crumbs.push(
+ last
+ ? `<strong>${escapeHtml(seg)}</strong>`
+ : `<a href="${up(depth - 1 - i)}">${escapeHtml(seg)}</a>`,
+ );
+ });
+ const rows: string[] = [];
+ if (depth > 0) {
+ rows.push(`<tr><td class="name"><a href="../">..</a></td><td class="size"></td></tr>`);
+ }
+ for (const e of sortEntries(opts.entries)) {
+ const label = escapeHtml(e.name) + (e.dir ? "/" : "");
+ const size = e.dir
+ ? ""
+ : `<span title="${e.bytes} bytes">${formatTreeBytes(e.bytes)}</span>`;
+ rows.push(
+ `<tr><td class="name"><a href="${escapeHtml(hrefFor(e))}">${label}</a></td><td class="size">${size}</td></tr>`,
+ );
+ }
+ const title = depth === 0 ? "archilyzer" : `archilyzer/${opts.relDir}`;
+ return [
+ "<!doctype html>",
+ `<html lang="en">`,
+ "<head>",
+ `<meta charset="utf-8">`,
+ `<meta name="viewport" content="width=device-width, initial-scale=1">`,
+ `<meta name="robots" content="noindex">`,
+ `<title>${escapeHtml(title)} · source</title>`,
+ `<style>${CSS}</style>`,
+ "</head>",
+ "<body>",
+ "<main>",
+ `<nav>${crumbs.join(`<span class="sep">/</span>`)}</nav>`,
+ `<table>`,
+ ...rows,
+ `</table>`,
+ `<footer>main @ ${escapeHtml(opts.mirrorHead.slice(0, 12))} · generated ${escapeHtml(opts.generatedAt)} · <a href="/source/">clone</a></footer>`,
+ "</main>",
+ "</body>",
+ "</html>",
+ "",
+ ].join("\n");
+}
+
+/**
+ * Write an index.html into every directory under `treeDir`, the root
+ * included. REFUSES when a directory already holds an `index.html` (a tracked
+ * one would be overwritten, or served in the index's place) or a `404.html`
+ * (Pages would serve it as HTML for any missing path below it), or when
+ * anything is a symlink. Returns the tree's own files and bytes (not the pages) and how
+ * many directories got a page.
+ */
+export async function writeTreeIndexes(
+ treeDir: string,
+ meta: TreeMeta,
+): Promise<{ files: number; dirs: number; bytes: number }> {
+ const totals = { files: 0, dirs: 0, bytes: 0 };
+ const walk = async (rel: string): Promise<void> => {
+ const abs = rel ? path.join(treeDir, rel) : treeDir;
+ const ents = await readdir(abs, { withFileTypes: true });
+ const entries: TreeEntry[] = [];
+ for (const ent of ents) {
+ const r = rel ? `${rel}/${ent.name}` : ent.name;
+ if (ent.isSymbolicLink()) {
+ throw new SourceRefusal(`the tree holds a symlink (${r}); the raw tree publishes files only`);
+ }
+ if (ent.isDirectory()) {
+ entries.push({ name: ent.name, dir: true, bytes: 0 });
+ continue;
+ }
+ if (ent.name === "index.html") {
+ throw new SourceRefusal(
+ `the tree already has ${r}; its directory page would replace it — rename the file or publish without the raw tree`,
+ );
+ }
+ // Pages answers a missing path with the NEAREST 404.html, as HTML (the
+ // directory-page rule): a tracked one would run on this origin.
+ if (ent.name === "404.html") {
+ throw new SourceRefusal(
+ `the tree has ${r}; Pages would serve it, as HTML on this origin, for every missing path below it — rename the file or publish without the raw tree`,
+ );
+ }
+ const { size } = await stat(path.join(abs, ent.name));
+ entries.push({ name: ent.name, dir: false, bytes: size });
+ totals.files++;
+ totals.bytes += size;
+ }
+ await writeFile(
+ path.join(abs, "index.html"),
+ renderTreeIndex({ relDir: rel, entries, ...meta }),
+ );
+ totals.dirs++;
+ for (const e of entries) if (e.dir) await walk(rel ? `${rel}/${e.name}` : e.name);
+ };
+ await walk("");
+ return totals;
+}
diff --git a/common/styles/tokens.css b/common/styles/tokens.css
@@ -5,20 +5,20 @@
@import "../../common/styles/tokens.css";
A reader theme is TWO independent choices (plans/brand-and-themes.md):
- • `html[data-base="light|sepia|dark"]` selects the BASE — the ground, text,
+ • `html[data-base="light|dark"]` selects the BASE — the ground, text,
lines, status and chart colours. "System" is not a block: the pre-paint
ThemeScript resolves it to light or dark before first paint.
• `html[data-accent="<id>|custom"]` selects the ACCENT — `--brand`. Each
- site server-renders its own; a reader may pick another.
+ site server-renders its own, and a reader does not pick one.
`html.dark` is set iff the resolved base is dark. The `@custom-variant dark`
below makes Tailwind's `dark:` utilities follow that class (not the OS media
- query), so sepia styles as a light ground.
+ query).
`:root` always matches <html>, so the light block below is also the
fallback for a page no script has touched (no-JS, the editor before its
- script runs). The sepia and dark blocks are `html[data-base=…]` (0,1,1) and
- win over it; the accent rules come AFTER all three at the same specificity,
- so a `data-accent` wins `--brand` on every base.
+ script runs). The dark block is `html[data-base=…]` (0,1,1) and wins over
+ it; the accent rules come AFTER both at the same specificity, so a
+ `data-accent` wins `--brand` on every base.
TOKEN NAMING:
• The shadcn "new-york" contract owns the unprefixed names: --primary,
@@ -30,13 +30,13 @@
--brand-ink, and --brand-mark, the header mark's on-dark value) so it
never collides with shadcn's neutral `--accent`.
• `--swatch-<id>` is each named accent's value ON THIS BASE (lib/brand.ts
- ACCENTS — themeTokens.test.ts keeps the two equal), so a picker can
- show the colour a choice will actually paint.
- • `--base-light`, `--base-sepia`, `--base-dark` are 1 on their own base
- and 0 on the others. They let a colour that differs per base but is
- known only to a component be ONE CSS value: each channel a calc() over
- the three (lib/siteColor.ts perBaseColor — a custom-hex site's card on
- the homepage and the hub, fitted to each ground).
+ ACCENTS — themeTokens.test.ts keeps the two equal), so a component
+ can name an accent's colour on the ground in force.
+ • `--base-light` and `--base-dark` are 1 on their own base and 0 on the
+ other. They let a colour that differs per base but is known only to a
+ component be ONE CSS value: each channel a calc() over the two
+ (lib/siteColor.ts perBaseColor — a custom-hex site's card on the
+ homepage and the hub, fitted to each ground).
========================================================================== */
@import "tw-animate-css";
@@ -215,7 +215,6 @@ html[data-base="light"] {
/* Which base this is, as numbers (the header's note; lib/siteColor.ts). */
--base-light: 1;
- --base-sepia: 0;
--base-dark: 0;
--brand: var(--swatch-signal);
@@ -243,7 +242,7 @@ html[data-base="light"] {
chart-1 is a blue, not Signal (ΔE 17.3 from it).
chart-6 (release 11) is a rust, Vermilion's family: the sixth site's
colour. The only hue family that clears every pair with chart-1..5 on
- all three bases is the red-browns (OKLCH h ≈ 15–60); this one's worst
+ both bases is the red-browns (OKLCH h ≈ 15–60); this one's worst
pair with them is CVD ΔE 12.5 deutan and normal 16.6, 8.1:1 on the chart
surface and 7.4:1 on the ground. It sits near --state-gone (ΔE 8.4
normal), which is only ever labelled text, never a chart mark. */
@@ -256,95 +255,19 @@ html[data-base="light"] {
--chart-surface: #ffffff;
--chart-grid: rgba(22, 28, 33, 0.09);
--chart-axis: #55646e;
+ /* OTHER: the homepage growth chart's band for the sites it groups
+ (homepage/app/lib/growthGaps.ts), a near-neutral slate at the foot of
+ the palette's lightness band (OKLCH L 0.43, C 0.03). 7.36:1 on the
+ ground, 7.99:1 on the chart surface. Against each slot it can sit on
+ (the dataviz validator, normal / worst CVD): blue 21.9 / 21.4, green
+ 17.4 / 15.2, violet 16.2 / 13.4, amber 22.1 / 18.2, magenta 24.5 /
+ 9.1, rust 14.1 / 10.5 — every CVD pair clears 8; rust is the one
+ normal pair under 15. */
+ --chart-other: #3e545c;
--chart-tooltip-bg: #ffffff;
}
/* ---------------------------------------------------------------------------
- SEPIA — warm paper for long reading. New with the base × accent themes; its
- status, panel and chart colours derive from the retired archive "paper"
- block, darkened for this darker ground (every text colour ≥ 4.5:1 on it).
- --------------------------------------------------------------------------- */
-html[data-base="sepia"] {
- color-scheme: light;
-
- --background: #f4ecd8;
- --surface: #ece2ca;
- --foreground: #33281a;
- --card: #faf4e6;
- --card-foreground: #33281a;
- --popover: #faf4e6;
- --popover-foreground: #33281a;
- --primary: #33281a;
- --primary-foreground: #f4ecd8;
- --secondary: #e9dec3;
- --secondary-foreground: #33281a;
- --muted: #ece2ca;
- --muted-foreground: #6b5c43;
- --accent: #e4d7b8;
- --accent-foreground: #33281a;
- --destructive: #b3261e;
- --destructive-foreground: #ffffff;
- --destructive-soft: rgba(179, 38, 30, 0.12);
- --border: #dccdaa;
- --border-strong: #c4b187;
- --input: #d3c29c;
- --ring: var(--brand);
- --faint: #857a64;
- --panel: rgba(250, 244, 230, 0.72);
- --panel-2: rgba(236, 226, 202, 0.72);
-
- /* Each deep enough to read at 4.5:1 on its OWN soft fill over the ground
- and the card, as the @theme note promises (themeTokens.test.ts). */
- --success: #256829;
- --success-foreground: #ffffff;
- --success-soft: rgba(37, 104, 41, 0.14);
- --warning: #8c4c00;
- --warning-foreground: #ffffff;
- --warning-soft: rgba(140, 76, 0, 0.14);
- --info: #1858bc;
- --info-foreground: #ffffff;
- --info-soft: rgba(24, 88, 188, 0.12);
-
- /* Each named accent's on-sepia value (lib/brand.ts ACCENTS.onSepia). */
- --swatch-signal: #2b756e;
- --swatch-brass: #8e6119;
- --swatch-vermilion: #b3431f;
- --swatch-violet: #6a4bc4;
- --swatch-sakura: #a83a6a;
- --swatch-blue: #2d5fb8;
- --swatch-green: #3d772b;
- --swatch-custom: var(--accent-custom-sepia, var(--swatch-signal));
-
- /* Which base this is, as numbers (the header's note; lib/siteColor.ts). */
- --base-light: 0;
- --base-sepia: 1;
- --base-dark: 0;
-
- --brand: var(--swatch-signal);
- --brand-strong: color-mix(in oklab, var(--brand) 78%, black);
- --brand-soft: color-mix(in srgb, var(--brand) 12%, transparent);
- --brand-ink: #ffffff;
-
- --state-gone: #a3392a;
- --state-gone-soft: rgba(163, 57, 42, 0.12);
-
- /* Same hue order as light, stepped for the paper surface. Adjacent pairs
- pass every check; all pairs pass with green ↔ amber in the CVD floor
- band (6.1). chart-3 vs --state-gone: ΔE 23.7 normal. chart-6, the
- rust, is light's: worst pair CVD 9.1 deutan, normal 16.6. */
- --chart-1: #3574d6;
- --chart-2: #2a7d4f;
- --chart-3: #5e3aa8;
- --chart-4: #a8741a;
- --chart-5: #bb4585;
- --chart-6: #823c10;
- --chart-surface: #faf4e6;
- --chart-grid: rgba(51, 40, 26, 0.09);
- --chart-axis: #6b5c43;
- --chart-tooltip-bg: #fffaf0;
-}
-
-/* ---------------------------------------------------------------------------
DARK — warm ink. The former archive "ink" face (the homepage's archive room),
with a neutral primary, and the --destructive-soft and --state-gone(-soft)
it never declared.
@@ -400,7 +323,6 @@ html[data-base="dark"] {
/* Which base this is, as numbers (the header's note; lib/siteColor.ts). */
--base-light: 0;
- --base-sepia: 0;
--base-dark: 1;
--brand: var(--swatch-signal);
@@ -425,6 +347,14 @@ html[data-base="dark"] {
--chart-surface: #16110a;
--chart-grid: rgba(233, 220, 197, 0.08);
--chart-axis: #8a8170;
+ /* OTHER, on this base: a near-neutral grey (OKLCH L 0.49, C 0.01),
+ the dimmest chart mark. 3.22:1 on the ground, 3.06:1 on the chart
+ surface (the slots: 3.37–6.46:1 on the ground). Against each slot (normal / worst CVD): blue 17.8 / 17.9,
+ green 19.2 / 15.7, violet 23.4 / 21.5, amber 20.3 / 18.1, magenta
+ 19.5 / 7.3, rust 13.5 / 9.3 — magenta in the CVD 6–8 floor band
+ (legal with the legend, the table and Other's place on top), rust the
+ one normal pair under 15. */
+ --chart-other: #62625c;
--chart-tooltip-bg: #1b150d;
}
@@ -464,9 +394,24 @@ html[data-accent="green"] {
--brand-mark: #7cc46a;
}
/* A site whose site.json accent is its own hex: the layout renders
- `data-accent="custom"` plus the fitted `--accent-custom-light|sepia|dark`
+ `data-accent="custom"` plus the fitted `--accent-custom-light|dark`
inline on <html>. */
html[data-accent="custom"] {
--brand: var(--swatch-custom);
--brand-mark: var(--accent-custom-dark, #5fa8a0);
}
+
+/* ---------------------------------------------------------------------------
+ THE CHART GAP — the colour that parts touching chart marks, the marks
+ spec's surface gap (common/components/charts/surfaceGap.ts): the chart
+ surface on every base (a chart card is `--card`, the same value), and the
+ reader's Canvas in forced colours.
+ --------------------------------------------------------------------------- */
+:root {
+ --chart-gap: var(--chart-surface);
+}
+@media (forced-colors: active) {
+ :root {
+ --chart-gap: Canvas;
+ }
+}
diff --git a/common/testing/chartPixels.ts b/common/testing/chartPixels.ts
@@ -0,0 +1,110 @@
+// WHAT A CHART PAINTED, for the e2e specs: a screenshot of the chart read back
+// as pixels in the page (the browser's own PNG decoder, an <img> drawn to a
+// canvas), so a test checks the rendered geometry — where the stack's top
+// is, which colours show — rather than what a style says a stroke is.
+//
+// No Playwright import (common/ does not depend on it): the page and the
+// locator are typed by the two methods this uses.
+
+// eslint-disable-next-line @typescript-eslint/no-explicit-any
+type Shooter = { screenshot(opts?: any): Promise<Buffer> };
+// eslint-disable-next-line @typescript-eslint/no-explicit-any
+type Evaluator = { evaluate(fn: any, arg: any): Promise<any> };
+
+export type Rgb = [number, number, number];
+
+// "rgb(1, 2, 3)" / "rgba(1, 2, 3, 0.5)" → [1, 2, 3].
+export function rgbOf(css: string): Rgb {
+ const m = css.match(/[\d.]+/g);
+ if (!m || m.length < 3) throw new Error(`not an rgb() colour: ${css}`);
+ return [Number(m[0]), Number(m[1]), Number(m[2])];
+}
+
+// `alpha` of `fg` over `bg`, as the browser composites a translucent fill.
+export function over(fg: Rgb, alpha: number, bg: Rgb): Rgb {
+ return [0, 1, 2].map((c) => Math.round(fg[c] * alpha + bg[c] * (1 - alpha))) as Rgb;
+}
+
+export type Painted = {
+ // Per requested column: the first row, from the top, of a run of `run` rows
+ // that each differ from `ground` by more than `tol` in some channel; null
+ // when the column is all ground.
+ tops: (number | null)[];
+ // Per requested colour: how many pixels are within `tol` of it, and in how
+ // many distinct columns.
+ counts: { pixels: number; columns: number }[];
+ width: number;
+ height: number;
+};
+
+export async function painted(
+ page: Evaluator,
+ target: Shooter,
+ opts: { columns: number[]; ground: Rgb; colours: Rgb[]; tol?: number; run?: number },
+): Promise<Painted> {
+ const png = await target.screenshot({ scale: "css", animations: "disabled" });
+ return page.evaluate(
+ async ({
+ b64,
+ columns,
+ ground,
+ colours,
+ tol,
+ run,
+ }: {
+ b64: string;
+ columns: number[];
+ ground: Rgb;
+ colours: Rgb[];
+ tol: number;
+ run: number;
+ }) => {
+ const img = new Image();
+ img.src = `data:image/png;base64,${b64}`;
+ await img.decode();
+ const c = document.createElement("canvas");
+ c.width = img.naturalWidth;
+ c.height = img.naturalHeight;
+ const ctx = c.getContext("2d")!;
+ ctx.drawImage(img, 0, 0);
+ const { data, width, height } = ctx.getImageData(0, 0, c.width, c.height);
+ const at = (x: number, y: number) => (y * width + x) * 4;
+ const differs = (x: number, y: number) => {
+ const p = at(x, y);
+ return [0, 1, 2].some((k) => Math.abs(data[p + k] - ground[k]) > tol);
+ };
+ const tops = columns.map((cx) => {
+ const x = Math.min(width - 1, Math.max(0, Math.round(cx)));
+ for (let y = 0; y + run <= height; y++) {
+ let solid = true;
+ for (let d = 0; d < run && solid; d++) solid = differs(x, y + d);
+ if (solid) return y;
+ }
+ return null;
+ });
+ const counts = colours.map((col) => {
+ let pixels = 0;
+ const cols = new Set<number>();
+ for (let y = 0; y < height; y++) {
+ for (let x = 0; x < width; x++) {
+ const p = at(x, y);
+ if ([0, 1, 2].every((k) => Math.abs(data[p + k] - col[k]) <= tol)) {
+ pixels++;
+ cols.add(x);
+ }
+ }
+ }
+ return { pixels, columns: cols.size };
+ });
+ return { tops, counts, width, height };
+ },
+ {
+ b64: png.toString("base64"),
+ columns: opts.columns,
+ ground: opts.ground,
+ colours: opts.colours,
+ tol: opts.tol ?? 12,
+ run: opts.run ?? 3,
+ },
+ );
+}
diff --git a/common/views/laneState.ts b/common/views/laneState.ts
@@ -45,8 +45,8 @@ export function deriveLaneState({
}
// No new palette. These map onto the station tones the channel line already
-// uses (see pipeline/tone.ts): a bespoke hue here would be wrong on all three
-// bases (light, sepia, dark) at once. Deliberately NOT a second copy
+// uses (see pipeline/tone.ts): a bespoke hue here would be wrong on both
+// bases (light, dark) at once. Deliberately NOT a second copy
// of those maps — flow/OverviewPanel already made one, and three would be a
// guarantee they drift.
export const LANE_DOT: Record<LaneState, string> = {
diff --git a/common/views/pipeline/stageStatus.ts b/common/views/pipeline/stageStatus.ts
@@ -601,7 +601,8 @@ export function computeStageStatuses(
mediaStatus === "in-transition"
? "running"
: mediaStatus === "unreachable" ||
- mediaStatus === "inconsistent"
+ mediaStatus === "inconsistent" ||
+ mediaStatus === "stalled"
? "danger"
: "neutral",
};
diff --git a/common/views/pipeline/tone.ts b/common/views/pipeline/tone.ts
@@ -1,7 +1,7 @@
import type { StageTone } from "./stageStatus";
// The line invents no colours. Every value below is one of the semantic tokens
-// the repo already carries across the three bases, light, sepia and dark
+// the repo already carries across the two bases, light and dark
// (common/styles/tokens.css); a bespoke hue here would be wrong on every base
// at once.
diff --git a/common/views/storage.test.ts b/common/views/storage.test.ts
@@ -60,7 +60,7 @@ test("an available location reports identity, free bytes, counts and a fresh pro
uuid: "09b5",
fstype: "ext4",
label: "platter",
- mountpoint: "/run/media/user/09b5",
+ mountpoint: "/run/media/operator/09b5",
relPath: "archilyzer-media",
},
freeBytes: 1024,
@@ -68,7 +68,7 @@ test("an available location reports identity, free bytes, counts and a fresh pro
probedAt: NOW - 4_000,
};
const { rows } = buildStorageRows({
- locations: [loc("cold", "/run/media/user/09b5/archilyzer-media")],
+ locations: [loc("cold", "/run/media/operator/09b5/archilyzer-media")],
defaultLocationId: "cold",
probes: { cold: probe },
rollups: { cold: rollup({ locationId: "cold", total: 3, ok: 2, unreachable: 1 }) },
@@ -81,7 +81,7 @@ test("an available location reports identity, free bytes, counts and a fresh pro
assert.equal(row.isDefault, true);
assert.equal(
row.identity,
- "platter · ext4 · UUID 09b5 · at /run/media/user/09b5",
+ "platter · ext4 · UUID 09b5 · at /run/media/operator/09b5",
);
assert.equal(row.freeBytes, 1024);
assert.equal(row.lastProbeAgeMs, 4_000);
@@ -124,10 +124,10 @@ test("mounted-elsewhere offers Re-point named after the candidate root", () => {
identity: {
known: true,
uuid: "09b5",
- mountpoint: "/run/media/user/09b5",
+ mountpoint: "/run/media/operator/09b5",
relPath: "media",
},
- candidateRoot: "/run/media/user/09b5/media",
+ candidateRoot: "/run/media/operator/09b5/media",
probedAt: NOW,
},
},
@@ -137,11 +137,11 @@ test("mounted-elsewhere offers Re-point named after the candidate root", () => {
});
const row = rows[0];
assert.equal(row.statusLabel, "Mounted elsewhere");
- assert.equal(row.candidateRoot, "/run/media/user/09b5/media");
+ assert.equal(row.candidateRoot, "/run/media/operator/09b5/media");
const repoint = action(row, "repoint");
assert.equal(repoint.offered, true);
- assert.equal(repoint.label, "Re-point to /run/media/user/09b5/media");
- assert.equal(repoint.newRoot, "/run/media/user/09b5/media");
+ assert.equal(repoint.label, "Re-point to /run/media/operator/09b5/media");
+ assert.equal(repoint.newRoot, "/run/media/operator/09b5/media");
// Two channels live there, so Delete is refused with the count and the root.
const del = action(row, "delete");
assert.equal(del.offered, false);
@@ -399,3 +399,36 @@ test("a location row carries the clips share of its bytes", () => {
// A SUBSET, not a sibling: the clips are already inside `bytes`.
assert.equal(row?.bytes, 10_000_000_000);
});
+
+test("a location whose drive is not answering reads so, and says since when", () => {
+ const { rows } = buildStorageRows({
+ locations: [loc("cold", "/mnt/cold")],
+ defaultLocationId: "",
+ probes: {
+ cold: { status: "stalled", identity: { known: false }, probedAt: NOW },
+ },
+ notAnswering: { cold: "not answering since 11:35 — a stat of its root did not answer within 3 s." },
+ rollups: { cold: rollup({ locationId: "cold", total: 2, unreachable: 2 }) },
+ registry: NO_JOBS,
+ now: NOW,
+ });
+ const row = rows[0];
+ assert.equal(row.status, "stalled");
+ assert.equal(row.statusLabel, "Not answering");
+ assert.match(String(row.notAnswering), /^not answering since 11:35/);
+ assert.equal(row.freeBytes, undefined);
+ // Neither re-point nor mount is offered for a drive that is here and stalled.
+ assert.equal(action(row, "repoint").offered, false);
+ assert.match(String(action(row, "repoint").withheld), /not answering/);
+ assert.equal(action(row, "mount").offered, false);
+ // A row with no entry says nothing.
+ const quiet = buildStorageRows({
+ locations: [loc("cold", "/mnt/cold")],
+ defaultLocationId: "",
+ probes: {},
+ rollups: {},
+ registry: NO_JOBS,
+ now: NOW,
+ }).rows[0];
+ assert.equal(quiet.notAnswering, undefined);
+});
diff --git a/common/views/storage.ts b/common/views/storage.ts
@@ -103,6 +103,9 @@ export type StorageRow = {
freeBytes?: number;
// How long ago the probe behind this row was taken, in ms.
lastProbeAgeMs: number;
+ // "not answering since 11:35 — …", while the health probe finds this
+ // location's drive not answering (lib/storageHealth.ts). Absent otherwise.
+ notAnswering?: string;
warning?: string;
candidateRoot?: string;
// Why every action on this row is withheld, or null. One re-point runs at a
@@ -201,6 +204,10 @@ export type StorageRowsInputs = {
// omitted, because a row that vanishes when a probe fails is worse than one
// that says it does not know.
probes: Record<string, MemoizedProbe | undefined>;
+ // By location id: the sentence for a location whose drive is not answering.
+ // Worded by the shell (lib/storageHealth.ts's `notAnsweringText`), because
+ // this module takes types only from lib/.
+ notAnswering?: Record<string, string | undefined>;
rollups: Record<string, LocationRollup | undefined>;
registry: RegistryReader;
udisksctlAvailable?: boolean;
@@ -217,6 +224,7 @@ export const STORAGE_STATUS_LABEL: Record<StorageLocationStatus, string> = {
unmounted: "Not mounted",
absent: "Not attached",
missing: "Missing",
+ stalled: "Not answering",
};
export const REPOINT_JOB_KIND = "repoint-storage-location";
@@ -386,6 +394,9 @@ export function buildStorageRows(i: StorageRowsInputs): StorageRowsPayload {
clipsText: storageClipsText(clipsBytes),
...(probe?.freeBytes !== undefined ? { freeBytes: probe.freeBytes } : {}),
lastProbeAgeMs: probe ? Math.max(0, i.now - probe.probedAt) : 0,
+ ...(i.notAnswering?.[loc.id]
+ ? { notAnswering: i.notAnswering[loc.id] }
+ : {}),
...(probe?.warning ? { warning: probe.warning } : {}),
...(candidateRoot ? { candidateRoot } : {}),
busy,
diff --git a/create-archives.sh b/create-archives.sh
@@ -1,62 +0,0 @@
-#!/bin/bash
-# Build the downloadable source snapshot served at /downloads/ on the project
-# site, plus a sidecar of facts about it.
-#
-# WHY A SNAPSHOT AND NOT A REPOSITORY. There is no public git remote, by choice.
-# `git archive` of `main` produces the working tree at one commit: no history, no
-# branches, no remote, nothing to `git pull`. The /downloads/ page says exactly
-# that rather than implying a clone.
-#
-# WHY THE SIDECAR. The filename is deliberately STABLE — a dated filename would
-# break every link the moment a new snapshot shipped. So the date, commit, size
-# and checksum live in snapshot.json beside it, and the page states the truth it
-# reads there instead of hardcoding facts that go stale. No sidecar → the page
-# renders an explanation and NO download link, rather than a link that lies.
-#
-# Safe to publish: `git archive` only includes TRACKED files, so the corpus
-# (transcripts/, gitignored), settings.json (untracked; only .example is
-# tracked) and node_modules are all structurally excluded.
-set -euo pipefail
-
-cd "$(dirname "$0")"
-
-REF="${1:-main}"
-OUT_DIR="homepage/public/downloads"
-TARBALL="$OUT_DIR/archilyzer-source.tar.gz"
-SIDECAR="$OUT_DIR/snapshot.json"
-
-mkdir -p "$OUT_DIR"
-
-# --prefix so the tarball unpacks into `archilyzer/` rather than spilling into
-# the current directory. The docs tell people to `cd archilyzer` afterwards.
-git archive -9 --format tar.gz --prefix=archilyzer/ -o "$TARBALL" "$REF"
-
-BYTES=$(stat -c %s "$TARBALL" 2>/dev/null || stat -f %z "$TARBALL")
-SHA=$(sha256sum "$TARBALL" | cut -d' ' -f1)
-COMMIT=$(git rev-parse "$REF")
-SUBJECT=$(git log -1 --format=%s "$REF")
-GENERATED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
-
-# node rather than a heredoc so the commit subject is JSON-escaped properly —
-# it is arbitrary human text and routinely contains quotes.
-GENERATED_AT="$GENERATED_AT" COMMIT="$COMMIT" SUBJECT="$SUBJECT" \
-BYTES="$BYTES" SHA="$SHA" node -e '
- const fs = require("fs");
- fs.writeFileSync(process.argv[1], JSON.stringify({
- generatedAt: process.env.GENERATED_AT,
- commit: process.env.COMMIT,
- subject: process.env.SUBJECT,
- bytes: Number(process.env.BYTES),
- sha256: process.env.SHA,
- }, null, 2) + "\n");
-' "$SIDECAR"
-
-echo "create-archives: $TARBALL ($(( BYTES / 1024 )) KiB) at ${COMMIT:0:12}"
-
-# Cloudflare Pages rejects any single asset over 25 MB. The snapshot is an order
-# of magnitude under that, but say so loudly if that ever stops being true —
-# silently shipping an asset Pages will refuse is a deploy that "succeeds" and
-# then 404s.
-if [ "$BYTES" -gt 26214400 ]; then
- echo "create-archives: WARNING — $TARBALL exceeds Cloudflare Pages' 25 MB per-file limit." >&2
-fi
diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
@@ -222,6 +222,12 @@ editor)
log "idle boot: schedulers, auto-queue runners and sweeps stay STOPPED"
fi
cd /repo/editor
+ # Sixteen threads for Node's filesystem pool instead of four. A call on a
+ # drive that has stalled holds its thread until the drive answers; with four,
+ # four such calls stop the editor answering at all. More threads buy time for
+ # calls already in flight — they isolate nothing (the storage health probe
+ # and its gate are what keep new calls off a stalled drive).
+ export UV_THREADPOOL_SIZE="${UV_THREADPOOL_SIZE:-16}"
exec "$(next_bin /repo/editor)" start --port "${EDITOR_PORT:-3001}"
;;
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -6,6 +6,24 @@
- **`/jobs` names the hub's and the homepage's jobs.** They show as **Build hub**, **Deploy hub**, **Build & deploy hub**, **Build homepage**, **Deploy homepage** and **Build & deploy homepage**, not as `build-hub`, `build-homepage` and so on.
- **A job cancelled before it started now stays cancelled.** Its record on disk kept saying "queued", so a restart could put a job you had just cancelled back in its queue, and a clip fetch cancelled while waiting could be reported as still queued. Jobs still waiting when the editor shuts down are handled as before: the next start settles or re-queues them.
+## [0.11.0] - 2026-09-30
+- **Transcripts that arrived after a video was first seen are counted.** The stats behind the homepage, the hub and every site's charts were cached per video and refreshed only when the video's metadata changed, so a transcript that came later — a Whisper run days after the download, or a video downloaded after the last index build — never reached them, and a video with YouTube captions alone had no transcription date. Counts and charts were low; the homepage could show a site with 0 transcripts, 0 channels and 0 hours while it served its videos. A stat is now also redone whenever the index re-reads the video, every transcript has a date, and a captioned video is dated by when its captions arrived rather than by a later Normalize run, so its place on "Transcribed over time" can move. **After updating, rebuild and restart the editor before anything else:** until then, **Build stats dataset** runs the old code and would undo the new stats, while a site, hub or homepage build already runs the new code — and the first stats build of any kind re-reads every video once (about 10–30 minutes on a large archive; it can be stopped and picks up where it stopped). Then build the index, the stats, the homepage, the hub, and the sites.
+- **A stats build keeps the stats of a channel whose drive is not mounted, and will not undo a newer version's stats.** A channel whose media is on a drive that is not mounted (or is being moved) is left as it was instead of being read as a channel with no videos; a stats rebuild that has to start over refuses until the drive is back. A stats build refuses to clear stats written by a newer version of the editor; set `ARCHILYZER_STATS_ALLOW_DOWNGRADE=1` to roll back on purpose. Its log also says apart how many videos were downloaded since the last index build (they catch up after the next one) and how many the index skipped (no upload date, or it failed on them).
+- **An index build keeps a channel whose drive is not mounted, instead of dropping it from the sites.** **Build index**, a site build's data phase and `archilyzer index` read a channel whose media is on a drive that is not mounted (or is being moved, or whose link and config disagree) as a channel with no videos: they removed its videos from the index, and the next site build published the channel as gone. Such a channel is now left as the last build had it — its videos stay in the index, its pages stay as they were, and the sites built next still list it — and the log names it, with its storage location: one line per channel, ` Held: N channel(s), K video(s) kept.` at the end of the `Diff:` line, and the channels again on the last line. A data folder that fails to read is held the same way, and a channel with no data folder at all is said in the log instead of passed over. An index rebuild that has to start over (after an update that changes the index's format, or with no index yet) refuses while any channel is held and says which; mount the drive first, or set `ARCHILYZER_INDEX_ALLOW_HELD=1` to rebuild without that channel until its drive is back and the index is built again — on the command for a command-line build (`ARCHILYZER_INDEX_ALLOW_HELD=1 pnpm archilyzer index`), or in the editor's own environment, with a restart, for **Build index** and the site builds started from the editor.
+- **umtool's build no longer lists its e2e test data, the e2e server's build folder or `.env.local` among a route's files.** The clip-audio route named its cache files in a way the bundler read as a pattern reaching into umtool's hidden folders, so its list of files took in the e2e fixture (where the tests link the song data), the e2e dev server's build folder and the env file: 1,704 of its 2,167 entries. It now lists what the other routes list (463). Those folders and env files are also excluded from every route's list, and `pnpm test:scripts` reads the last umtool build's lists back and fails on any such entry. A checkout whose umtool build predates its code (this change included) skips that check, saying so, until umtool is rebuilt (`pnpm --filter umtool exec next build`). Nothing changes when umtool runs.
+- **A drive that stops answering no longer stops the editor answering.** When a storage location's drive is mounted but not answering (an SMR disk in a USB enclosure resetting under a long write), every page and poll that touched it waited on it, and a few such waits froze the whole editor until the drive came back. Every 15 seconds the editor now reads each location's disk activity counters from the kernel, which never waits on the drive: a disk with requests waiting and none finished since the last look is marked **Not answering**, and the mark comes off after two looks in a row find it working. Where no disk can be named (in a container, say) it asks the drive from a separate process with a 3-second limit instead. Any page or poll that reads the drive also gives up after 3 seconds and marks it the same way, and no more than four such reads wait on one drive at a time. While it is marked, the editor's pages and polls do not read that drive: `/storage` shows the location as **Not answering** with the time it stopped and how it is watched (**Refresh** asks the drive again), the `/channels` volume chip reads "not answering since HH:MM" and its channels' badges "not answering", their videos list, video pages and Cleanup stage say so instead of reading the drive, `/saved-videos` names the channels it did not read, and the index and stats builds keep those channels as they do for an unmounted drive, and a channel that is in the middle of a move still shows as moving. Jobs for those channels are refused until the drive answers, and a channel paused automatically for it says the drive is not answering rather than not there. The drive check keeps running on an editor started with `ARCHILYZER_IDLE_BOOT`. Pages and polls also reuse each channel's media check for 5 seconds. The editor's `start` script and the container now give Node 16 threads for file access instead of 4 (`UV_THREADPOOL_SIZE`); that buys time for reads already waiting on a drive, and a read that was already waiting when the drive stalled still waits until the drive answers.
+- **Building the homepage now publishes the source: a read-only git mirror, its raw tree and a fresh tarball, behind a gate.** `archilyzer build homepage`, the `/sites` Homepage jobs and `pnpm ops build-homepage` run `archilyzer source publish` between compose and `next build`. It makes a fresh clone of the private `main` (the repository itself is never rewritten), rewrites that copy with git-filter-repo using your scrub rules (file contents and commit messages; your home directory becomes `/home/user` without a rule), and publishes it under `homepage/public` for `git clone https://archilyzer.pages.dev/source/archilyzer.git`, beside `/source/tree/` and the Downloads tarball. Before anything is written, every object of the rewritten history and every file about to be published is searched for every string you have denied; **one hit refuses the build**, and its log names the string only by where you wrote it (`denylist line 3 (len 5)`) and each hit by its object, field and byte offset — never a byte of the object. **A refusal withdraws the source**: the last publish is removed from `homepage/public` and the last build's copy from `homepage/out`, and **Deploy homepage refuses** a build whose source was not audited under today's rules and today's `main` ("run `archilyzer build homepage`, then deploy"). The rules live outside the repo, in `~/.config/archilyzer/source-scrub.txt` and `source-denylist.txt` (`ARCHILYZER_CONFIG_DIR`, `SOURCE_SCRUB_FILE`, `SOURCE_DENYLIST_FILE`); **without them the build refuses**, naming the missing file. **Put everything private in the denylist before any deploy, a preview included**: previews are public, and every deployment stays reachable at its own address until you delete it. Install git-filter-repo once (`pipx install git-filter-repo`; the editor's process needs `~/.local/bin` on its `PATH` to find it) — without it the build fetches it through `pipx run`, which needs the network — and gitleaks if you want its secret scan too. An unchanged `main` with unchanged rules is skipped, so a rebuild costs about 20 seconds only when something moved. A checkout with no git repository (the docker image, a tarball install) builds with the /source page's empty state. `archilyzer source publish --check` audits without writing, `archilyzer source audit <clone>/.git` checks any clone, `archilyzer build homepage --no-source` removes the published source instead, and `archilyzer doctor` reports the tools, the two files (rule counts and permissions, never their contents) and the last publish. `create-archives.sh` is gone. See PUBLISH.md, "The source mirror (homepage)".
+- **umtool reads the corpus from its checkout (or `TRANSCRIPTS_DIR`), and the song project's data defaults to `~/.local/share/archilyzer/song`.** If yours is elsewhere, link it there before restarting umtool: `mkdir -p ~/.local/share/archilyzer && ln -s <where the data is> ~/.local/share/archilyzer/song` (the data stays where it is). With no `CHANNELS_DIR`, umtool reads the corpus at `$TRANSCRIPTS_DIR/channels`, else the checkout's own `transcripts/channels`; it used to fall back to an absolute path that existed on one machine only. The song project's videos default to `~/reports/quartering-uh-song/videos`; `SONG_DIR` and `VIDEO_ROOT` still win. The song project's tracked manifests record their paths relative to the song folders, and the twenty one-off `umtool/song/*.sh` run logs, which only ever ran on the machine that wrote them, are gone.
+- **umtool's production build no longer reads the corpus folder.** Since umtool began finding the corpus from its checkout (the bullet above), `next build` treated the checkout's whole `transcripts/channels` as files to bundle. On a real archive it ran out of memory and was killed, so umtool could not be rebuilt. The build now ignores that folder and finishes in about 25 s at under 1 GB, the same as a checkout with no corpus. Nothing changes when umtool runs.
+- **A social icon pasted with only a width and height is accepted, and each social link can be kept in the header on small screens.** The social-link editors (Settings, a site's form) refused an SVG with no `viewBox`, so a vendor's logo file as downloaded, which often carries only its size, was refused. On save, a root with a numeric width and height (unitless or px) and no viewBox is now given `viewBox="0 0 W H"`; a percentage, `em`, or a missing or zero side is still refused. Each link has a **Keep in header on small screens** checkbox, stored as `featured: true` only when checked, with the hint "On small screens the header shows only these; the rest stay in the footer.": a narrow header (under 520 px) shows only the checked links, none when none is checked; a wide one shows every link, up to four, the checked ones kept first (the homepage's, every site's and the hub's headers read it). A file with neither is read and rendered as before. `SETTINGS.md` and `SITE.md` describe `featured`.
+- **A social icon is checked by what it may contain, when it is saved new or edited and every time it is shown, and a refused one says why.** An icon must be one well-formed `<svg>` of shapes, groups, gradients, clips, masks, filters, text and simple animation, with SVG presentation attributes: no script, `style` block, `foreignObject`, link, embedded image, `title`/`desc` with anything but text (text-only ones are removed), or HTML element; no event handler, however it is written; a `style` attribute of presentation properties only; a reference only to something inside the icon, written plainly; and nothing that could load from elsewhere (a CSS escape or comment, `image-set(`, `image(`, `cross-fade(`, `element(`, `src(`, `paint(`, `@import`). Comments, a leading XML declaration and a plain DOCTYPE are removed. A refused save ends with the reason ("… has an invalid SVG: it has an event handler attribute.", "… it links to something outside the icon.") and never repeats the markup; for a drawing program's file it says to export it with presentation attributes rather than a style block (in Inkscape, save as Plain SVG). **Upgrading:** an icon an older build stored is kept as it is when a save does not change it — a pause, a priority or a title still saves — but a page shows it as its label until its SVG is replaced; `archilyzer doctor`'s new "social icons" line names every stored icon that fails, by file and label, with the reason.
+- **Two grounds, Light and Dark, and no accent picker in the header.** The editor's header keeps its theme toggle, which cycles System, Light and Dark; the theme menu (Base and Accent) is gone, and the editor wears its own accent, Signal. A stored choice of the retired third ground loads as Light and is rewritten once; a stored accent is not read and is left in storage. A site's accent is still set in its form; the form's hint no longer says a reader can pick another.
+- **The hub URL hints say what the setting does now.** Settings' **Family hub URL** and a site's **Hub URL** no longer promise a Hub link in the header (it was removed): the value is published as `hubUrl` in each site's `/site.json` and `/corpus.json`, so the hub can tell its member sites. `SETTINGS.md` and `SITE.md` say the same.
+- **A site can be left off the homepage and the hub.** A site's settings have a new checkbox, **List on the Archilyzer homepage and hub**, on by default (`listed` in `site.json`; only `false` is written). Turned off, the site still builds and deploys at its own URL as before, but the homepage has no card, chart series, `/stats` entry or recent item for it; the hub does not list it as a member, search it, or name it in its `corpus.json` and `llms.txt`; no other site's footer links it; and `channel-sites.json` and the homepage's `stats/` leave it out. A channel only unlisted sites carry is in none of the published totals, the homepage's headline numbers included; a channel a listed site also carries is counted under the listed site. The editor's own pages still show every site. It takes effect at the next homepage, hub and site builds.
+- **The sidebar's site picker shows your site from the first paint.** It used to show "All sites" on every page and then jump to the site you had picked, and Dashboard and Channels came up in your site only after a `?site=` had been added to the address. The picked site is now kept in a cookie that the editor reads before it draws a page, so the picker, Dashboard and Channels open in it at once, and the address is left alone. A link that carries `?site=<id>` still opens that page in that site, without changing the one you picked; picking a site on such a page drops the `?site=` from the address. On a site's own pages (Charts, Publish, …) the picker still follows the page, and opening one still makes that site the picked one. The first time you open the editor after updating, a site picked before is moved into the cookie; the picker may show "All sites" for a moment that once. A site picked in one tab reaches the editor's other open tabs without a reload. **New channel** starts with the picked site ticked under its sites (or the site of a `?site=` link), including when it is opened from the editor's own links. Each editor keeps its own pick, as before, when several run on one machine on different ports.
+- **The drive check's timings can be changed on `/storage`.** The numbers the editor decides a drive is "not answering" by were fixed: a read may take 3 seconds, each drive is checked every 15 seconds (a check that asks the drive from a separate process waits up to 3 seconds), two clean checks in a row put a drive back in use, and at most four reads wait on one drive at a time. They are now **Drive health timing**, a collapsed block at the foot of `/storage`, with those numbers as the defaults — for when a drive that is busy but working is marked not answering, or a stalled one is not. An empty field is its default; a number outside a field's range is refused, with the range. A save takes effect at once: the next read, the next check, and a new check interval re-times the checks. `settings.json` keeps only the values that differ from a default, under `storage.health` (see `SETTINGS.md`), so an editor that never changes them follows the defaults; `archilyzer index` and the stats build read them too. The messages that said "3 s", "every 15 s" or "twice in a row" now say the numbers in force.
+- **Building the homepage publishes the source's history too: every commit of main with its diff, at `/source/git/`, when stagit is installed.** `archilyzer build homepage`, the `/sites` Homepage jobs and `pnpm ops build-homepage` render the scrubbed mirror with stagit into a log, a page per commit, the refs and two Atom feeds; each page carries the homepage's grounds and one line back to `/source/`, its Files page links into the raw tree, and every page goes through the same gate as the mirror (a denied literal in one refuses the publish). stagit is installed once, outside the repository: `git clone git://git.codemadness.org/stagit && make -C stagit && cp stagit/stagit ~/.local/bin/` (or point `STAGIT_BIN` at it). Without it the build goes on without the history and says so in one line; a render that fails, or pages that would pass the host's limits, are a warning, never a failed build. `archilyzer doctor` shows where stagit is, beside git-filter-repo. The newest 10,000 commits have pages; past that the log says how many more there are. The first build after this change publishes the source again (the step's version is 4). A render cache of about 140 MB is kept in `~/.cache/archilyzer/source-history` (or under `XDG_CACHE_HOME`), so later builds render only the new commits; `archilyzer doctor` shows where it is and its size.
+
## [0.10.0] - 2026-09-28
- **The homepage can be built and deployed from `/sites`.** Under a new **Homepage** section, after Hub, there is **Build homepage** (tick **Deploy after build** to ship it in the same job, only if the build succeeds) and **Deploy homepage**, which ships the build already in `homepage/out`. A **Preview branch** box beside them sends either deploy to a Cloudflare Pages preview of the `archilyzer` project instead of production, and shows the preview's address as you type; a name Cloudflare would refuse or rewrite, or `main`, greys the deploy buttons out and says why. A line under the buttons says what a deploy would ship: when `homepage/out` was built (or that it holds no build yet), and where it goes, with the live URL. Deploy homepage with nothing built is refused before any job starts. The homepage reads the search index as it stands, so run **Build index** first when its numbers should move. The jobs run the same code as `archilyzer build homepage` / `deploy homepage`, and show on `/jobs` as `build-homepage`, `deploy-homepage` and `build-deploy-homepage`. The Hub section no longer describes the homepage.
- **`pnpm ops build-homepage` and `pnpm ops deploy-homepage`.** The same two jobs over HTTP: `build-homepage` takes `{"deploy": true}` to deploy after a successful build, and both take `{"preview": "<branch>"}` for a preview (`build-homepage` only with `deploy`). `deploy-homepage` answers with the preview's address, and refuses a bad preview name or a missing build before any job starts.
@@ -86,7 +104,7 @@
- **A channel's Storage panel has one verb.** *Move media to…* and *Move back in place* were two sections with two buttons whose availability was the inverse of each other — one decision split across two controls. The corpus volume is now a destination in the same select; picking it moves the media back. While the media is on a location it is the only destination offered, because a move straight from one location to another is refused by the mover itself.
- **The saved-video store can be moved to another drive.** It is the one large thing in the corpus that belongs to no channel, so no channel move could ever reach it. `/storage` now shows it with its size and where it is, and moves it onto a location — and back — by exactly the mechanism a channel uses: the copy is verified before the source is touched, a symlink is left behind, and every reader keeps working unchanged. An interrupted move can be **resumed** rather than restarted, and its marker cleared if it cannot.
- **A drive that disappears now disables and flags the channels on it, and un-does that when it comes back.** Every guard in the app refuses work on unreachable media at the moment the work starts; none of them was a *detector*, so a channel whose drive fell off a cable sat there being refused with nothing anywhere saying why. A five-minute pass now notices, **pauses those channels and marks them** — the tier column shows a *storage* badge with the reason — and restores each one to the tier it had when the drive returns. It never claims a channel you paused yourself, and changing a tier by hand permanently takes it out of the machine's hands. It writes only when something has actually changed, and never arms at all under `ARCHILYZER_IDLE_BOOT`.
-- **The drives a channel's media lives on are named places now, and one click re-points them.** The cold root used to be a single string typed into Settings, and a relocated channel's `config.dataDir` an absolute path — so when the platter was automounted at `/run/media/user/<uuid>` and came back somewhere else, every channel on it read *unreachable* and the only remedy was SSH and hand edits. **`/storage`** (twelfth entry, under Machine) lists each media root as a **location** with a name, a status and the channels on it: `Available`, `Not mounted`, `Not attached`, or **`Mounted elsewhere`** — which is the one that matters, because it means the disk is here under a different mountpoint, and the row then offers **Re-point**, which rewrites every channel's `data/` symlink and `config.dataDir` and **moves no bytes at all**. There is a **Refresh** per row (a probe is `findmnt`, memoised for ten seconds, and it never writes availability to `settings.json`), a **Mount** for an attached-but-unmounted volume, a per-location **auto re-point** opt-in for operators who would rather it just happened, and a boot pass that checks every location as the editor starts. Identity is the volume's filesystem **UUID**, learned at the last successful probe — in a container there are no block devices to learn it from, every probe **fails open to "unknown", and re-point by path is the whole story** (see RUNNING_IN_DOCKER.md). The old `settings.storage.mediaRoot` **migrates on read** into a one-entry list called *Default*; the Settings field is now a link to the page. Everywhere a move starts, the destination is a **name picked from a list** rather than a path retyped per channel: the channel's Storage panel has a **destination select** showing each drive's current state (with *Another root…* keeping the free-text box), and `/channels`' selection deck has the same select for a whole batch — and what reaches the server is the **id**, never the root, so a page rendered before a re-point cannot aim a batch at a root that has since moved. The badge on every channel row says **`on Platter`** instead of sixty columns of absolute path, or **`on Platter — unreachable`** when the drive is not there. **And an interrupted move can be finished.** The controller has always resumed a half-done copy; nothing in the editor could reach it, so the only offered way out of a killed rsync was *Clear marker* and a full re-copy — which for the incident behind this work meant re-copying 131 GB that was already correctly on the far side. The panel now offers **Resume move** beside it, and the same release closes the three ways that incident happened: the relocate copy raced an auto-queue digest unit that made **no job record**, so "is this channel busy" now asks the lanes as well as the registry, and a lane that finds a relocation marker stops instead of writing into a directory being copied; a sidecar written mid-copy left one directory timestamp differing and `verifyCopy` refused the whole 131 GB, so a drift that is *only* directory mtimes now gets one more `rsync -a` pass and a re-verify (content drift still refuses, and still says the source has not been touched). Also fixed: on `/channels` a dimmed row's **Advanced menu drew underneath the rows below it** — `opacity` on a `<tr>` makes a stacking context, so the row is dimmed cell by cell now, and the cell hosting the popover is left alone.
+- **The drives a channel's media lives on are named places now, and one click re-points them.** The cold root used to be a single string typed into Settings, and a relocated channel's `config.dataDir` an absolute path — so when the platter was automounted at `/run/media/<user>/<uuid>` and came back somewhere else, every channel on it read *unreachable* and the only remedy was SSH and hand edits. **`/storage`** (twelfth entry, under Machine) lists each media root as a **location** with a name, a status and the channels on it: `Available`, `Not mounted`, `Not attached`, or **`Mounted elsewhere`** — which is the one that matters, because it means the disk is here under a different mountpoint, and the row then offers **Re-point**, which rewrites every channel's `data/` symlink and `config.dataDir` and **moves no bytes at all**. There is a **Refresh** per row (a probe is `findmnt`, memoised for ten seconds, and it never writes availability to `settings.json`), a **Mount** for an attached-but-unmounted volume, a per-location **auto re-point** opt-in for operators who would rather it just happened, and a boot pass that checks every location as the editor starts. Identity is the volume's filesystem **UUID**, learned at the last successful probe — in a container there are no block devices to learn it from, every probe **fails open to "unknown", and re-point by path is the whole story** (see RUNNING_IN_DOCKER.md). The old `settings.storage.mediaRoot` **migrates on read** into a one-entry list called *Default*; the Settings field is now a link to the page. Everywhere a move starts, the destination is a **name picked from a list** rather than a path retyped per channel: the channel's Storage panel has a **destination select** showing each drive's current state (with *Another root…* keeping the free-text box), and `/channels`' selection deck has the same select for a whole batch — and what reaches the server is the **id**, never the root, so a page rendered before a re-point cannot aim a batch at a root that has since moved. The badge on every channel row says **`on Platter`** instead of sixty columns of absolute path, or **`on Platter — unreachable`** when the drive is not there. **And an interrupted move can be finished.** The controller has always resumed a half-done copy; nothing in the editor could reach it, so the only offered way out of a killed rsync was *Clear marker* and a full re-copy — which for the incident behind this work meant re-copying 131 GB that was already correctly on the far side. The panel now offers **Resume move** beside it, and the same release closes the three ways that incident happened: the relocate copy raced an auto-queue digest unit that made **no job record**, so "is this channel busy" now asks the lanes as well as the registry, and a lane that finds a relocation marker stops instead of writing into a directory being copied; a sidecar written mid-copy left one directory timestamp differing and `verifyCopy` refused the whole 131 GB, so a drift that is *only* directory mtimes now gets one more `rsync -a` pass and a re-verify (content drift still refuses, and still says the source has not been touched). Also fixed: on `/channels` a dimmed row's **Advanced menu drew underneath the rows below it** — `opacity` on a `<tr>` makes a stacking context, so the row is dimmed cell by cell now, and the cell hosting the popover is left alone.
- **The operations poll reads the auto-queue's state file once instead of four times.** Every payload the editor draws — the jobs head, the workers grid, the operations board, the sync schedule, the widget's tiles, the pulse token — used to be computed by a function that did its own reading, so each of the four lanes on `/operations` opened `.auto-queue/state.json` for itself: four parses of the same document every three seconds, on a page whose four lanes were always reading one document. Those builders are pure functions in the shared core now — they are handed the settings, the registry, the scheduler, the pool, the clock and their readings, and they cannot reach disk or construct a singleton, which a layer test enforces rather than a comment asking nicely. The reading happens once, at the edge, and is shared. **Nothing moved that you can see**: same pages, same URLs, same JSON on every endpoint, same numbers — the difference is that each payload now has unit tests of its own (the console's cooldown filter, the pulse token's sensitivity, the workers grid's task grouping), where previously the only way to test one was to render the page that showed it.
- **`/channels` is a rack now, with one selection deck and a meter bridge.** The page had two selection bars for one selection — a floating one for Tier and Focus, and a second block below sixty-seven rows for Move media, both saying "N selected" and both offering Clear. There is **one deck**: it docks under the table when you tick a row, carries **Tier**, **Focus** and **Media** side by side, and unmounts when you untick. The destination root lives in its own box beside the button (the button used to carry it in its label, where it truncated to *Move media to…* and you could not read where the files were going). **The table stops spilling off the screen.** It lives in one scroll region: the column headers pin to its top, the checkbox and slug cells pin to its left, and a section's name pins under the headers — so the identity column and the meter bridge header stay on screen while sixteen columns scroll sideways. The six pipeline columns read as **one block** rather than six loose dashes: a shared *Pipeline* eyebrow, a surface behind them, a rule at each end. **Rows are 41 px instead of ~90.** The tier cell is one line, and being held by a focus is a small **held** chip rather than the same orange sentence repeated on sixty-one rows — the sentence is stated once, with a count, on the focus line above the table, and each chip still carries the full reason for a screen reader and on hover. Opening a row's *Advanced* overlays the rows below instead of pushing them down. **The page's caveat is at the top.** The note saying every number here is read from each channel's last report, and how old the oldest one is, used to be the last thing on the page in 11 px type under a floating bar; it is the subtitle beside the title now, with the channel count. The band legend and the Names·A / Names·T explainer moved above the table too, beside *Group by section*. In the header, *Sync every channel*, *Full sweep every channel* and *Update all reports* are outlines under an **Every channel** eyebrow that says what they sweep, and **New channel** is the only filled button. Nothing on disk moved and no control changed its name.
- **A channel's media can live on another drive.** A channel page has a **Storage** panel: where its media actually is, how much audio is on disk, how much room is free on the volume holding it, and **Move media to…** — give it a directory on another disk, press *Preview* to see the bytes and the free space there, and the move copies, **verifies**, and only then swaps `data/` for a link to the new location and records it. **Move back in place** reverses it. The source is never touched until the copy has verified, so a cancelled or crashed move leaves everything where it was and the partial copy resumable; re-running finishes it. Nothing else changes: every page, every job, yt-dlp and the search index read the channel exactly as before, because the path they use is unchanged. **The point is what happens when the drive is not mounted.** `data/` reads as empty then, and an empty `data/` means "nothing has been downloaded" to the download runner — an instruction to re-fetch the entire channel onto the disk that was too full to hold it. So an unreachable channel is **refused rather than guessed at**: its media jobs will not start, the four lane runners skip it (and keep running every other channel — this is not a lane stop), its report will not regenerate over an empty directory, and a red **Media unreachable** badge names the path on `/channels`, on the dashboard and on the channel itself. A relocated-and-reachable channel gets a neutral badge saying where; a channel in place gets none. The low-disk floor now measures **the volume the bytes are actually going to** rather than always the corpus disk, and holds each volume separately — a full SSD no longer pauses downloads landing on the platter. The **Media location** line on a channel's Configure form is read-only on purpose: it is a record of what is on disk, written only by a move that succeeded. The cold drive is typed **once**: **Settings → Default media root** seeds the root box in every channel's Storage panel, and `/channels` rows can now be ticked — select several and **Move media to…** queues one job per channel on that channel's own queue, so they serialize instead of fanning out, each one running its own space check at run time rather than at enqueue time (a root that fills partway through refuses the remainder cleanly, and a channel already on that root is skipped rather than failed). The default is a default and nothing more: it is never read by the move itself, which always takes an explicit root, and a relocated channel is not thereby deprioritized. **Nothing moves on its own, and nothing on disk changes until you move a channel.** *(Superseded above: **Settings → Default media root** is gone — the roots are named locations on `/storage` now, and every destination is picked from that list by name rather than typed.)*
diff --git a/editor/app/api/channels/[slug]/videos/[id]/files/[name]/route.ts b/editor/app/api/channels/[slug]/videos/[id]/files/[name]/route.ts
@@ -4,6 +4,13 @@ import { stat } from "node:fs/promises";
import { NextResponse } from "next/server";
import { getPaths } from "yt-dlp-transcript-common/lib/paths";
import { makeSafeController } from "yt-dlp-transcript-common/lib/safeStreamController";
+import { readChannelConfig } from "yt-dlp-transcript-common/controller/channels";
+import { channelMediaStall } from "yt-dlp-transcript-common/lib/channelMedia";
+import {
+ NOT_ANSWERING,
+ isDriveNotAnswering,
+ onDrive,
+} from "yt-dlp-transcript-common/lib/storageHealth";
export const dynamic = "force-dynamic";
@@ -123,10 +130,24 @@ export async function GET(
return NextResponse.json({ error: "Forbidden" }, { status: 403 });
}
+ // A drive that is not answering is not asked: the stat and the stream would
+ // each wait on it. 503, because it is a state that passes. The stat goes
+ // through the watchdog, so a drive that stops answering now is a 503 too;
+ // the stream that follows a stat that answered is not raced.
+ const notAnswering = () =>
+ NextResponse.json(
+ { error: `Media not read: ${NOT_ANSWERING}.` },
+ { status: 503, headers: { "retry-after": "15" } },
+ );
+ const channelConfig = await readChannelConfig(paths, slug);
+ if (channelMediaStall(channelConfig)) return notAnswering();
+ const drive = channelConfig?.dataDir?.trim();
+
let stats;
try {
- stats = await stat(fullPath);
- } catch {
+ stats = await (drive ? onDrive(drive, () => stat(fullPath)) : stat(fullPath));
+ } catch (err) {
+ if (isDriveNotAnswering(err)) return notAnswering();
return NextResponse.json({ error: "Not found" }, { status: 404 });
}
if (!stats.isFile()) {
diff --git a/editor/app/api/test/invalidate-cache/route.ts b/editor/app/api/test/invalidate-cache/route.ts
@@ -4,6 +4,8 @@ import { resetSnapshotScheduler } from "yt-dlp-transcript-common/jobs/snapshotSc
import { resetChannelSnapshotMemo } from "yt-dlp-transcript-common/controller/channels";
import { resetStorageProbeMemo } from "yt-dlp-transcript-common/controller/storageLocations";
import { resetVideoTitleMemo } from "yt-dlp-transcript-common/controller/videoTitles";
+import { forgetChannelMedia } from "yt-dlp-transcript-common/lib/channelMedia";
+import { resetStorageHealth } from "yt-dlp-transcript-common/lib/storageHealth";
import { testRouteDenied } from "../_guard";
export const dynamic = "force-dynamic";
@@ -112,6 +114,12 @@ function invalidate() {
// testInfo.outputPath varies by accident rather than on purpose; cleared here
// so it is on purpose.
resetStorageProbeMemo();
+ // And the two memories a stalled drive lives in: inspectChannelMedia's
+ // five-second answers (keyed by channels dir, slug and the configured target,
+ // which a reset fixture reproduces exactly) and each location's health, which
+ // a previous spec's location id would otherwise carry into this one.
+ forgetChannelMedia();
+ resetStorageHealth();
// And the video list's metadata.info.json title memo. It is keyed by the
// channel's data/ mtime, which resetData() changes by recreating the dir — but
// a spec that rewrites a title in place inside one mtime tick would otherwise
diff --git a/editor/app/channels/[slug]/components/MediaNotAnswering.tsx b/editor/app/channels/[slug]/components/MediaNotAnswering.tsx
@@ -0,0 +1,72 @@
+import Link from "next/link";
+import {
+ healthTimings,
+ notAnsweringText,
+ type LocationHealth,
+} from "yt-dlp-transcript-common/lib/storageHealth";
+import { secondsText } from "yt-dlp-transcript-common/lib/storageHealthTimings";
+
+// WHAT A PAGE THAT READS A CHANNEL'S `data/` SHOWS WHILE ITS DRIVE IS NOT
+// ANSWERING, instead of reading it.
+//
+// The videos list and the video page read the channel's media directory on
+// every render — a readdir, a stat per file, a metadata head read per title. On
+// a drive that is mounted and not answering (lib/storageHealth.ts) each of those
+// calls waits for the drive, on one of the few threads every page and poll in
+// this process shares. So the page asks the health state first, and on a
+// stalled drive it says so and reads nothing. The rest of the channel (its
+// overview, its report, its settings) comes off the corpus disk and still
+// renders.
+//
+// Server-only: it takes the health entry from the page, which read it from
+// memory — or from the DriveNotAnsweringError a page's read got from `onDrive`'s
+// watchdog.
+export function MediaNotAnswering({
+ slug,
+ stall,
+ what,
+}: {
+ slug: string;
+ // The stalled location, or null when the drive is on no location the health
+ // state knows and a read of it did not answer within the budget
+ // (`storage.health.budgetMs`).
+ stall: LocationHealth | null;
+ // What would have been shown: "The video list", "This video".
+ what: string;
+}) {
+ const timings = healthTimings();
+ return (
+ <div className="flex flex-col gap-3">
+ <p
+ role="status"
+ aria-label="media not answering"
+ className="rounded border border-destructive/50 bg-destructive/5 px-3 py-2 text-sm text-destructive"
+ >
+ {stall ? (
+ <>
+ {what} reads this channel's media, which is on “
+ {stall.label}” — a drive that is {notAnsweringText(stall)}.
+ Nothing is read from it until it answers again (it is checked every{" "}
+ {secondsText(timings.passIntervalMs)}).
+ </>
+ ) : (
+ <>
+ {what} reads this channel's media, and a read of its drive did
+ not answer within {secondsText(timings.budgetMs)}. Nothing more is
+ read from it on this page.
+ </>
+ )}
+ </p>
+ <p className="text-sm text-muted-foreground">
+ <Link href={`/channels/${slug}`} className="underline hover:text-foreground">
+ The channel
+ </Link>{" "}
+ still shows its report, and{" "}
+ <Link href="/storage" className="underline hover:text-foreground">
+ Storage
+ </Link>{" "}
+ shows the drive.
+ </p>
+ </div>
+ );
+}
diff --git a/editor/app/channels/[slug]/components/stages/StorageStage.tsx b/editor/app/channels/[slug]/components/stages/StorageStage.tsx
@@ -93,7 +93,8 @@ type Props = {
clipsBytes: number | null;
// Free space on the volume the media is on RIGHT NOW — the platter for a
// relocated channel, the corpus disk otherwise.
- freeBytes: number;
+ // Null when the drive did not answer the statfs ("—").
+ freeBytes: number | null;
volumeDir: string;
// Why both buttons are off, or null when they are live. Running/queued jobs
// for this channel, or a relocation marker left by an interrupted move.
@@ -151,7 +152,7 @@ export function StorageStage({
</dd>
<dt className="text-muted-foreground">Free on that volume</dt>
<dd aria-label="free on media volume">
- {formatBytes(freeBytes)}{" "}
+ {freeBytes === null ? "—" : formatBytes(freeBytes)}{" "}
<span className="text-xs text-muted-foreground font-mono">
({volumeDir})
</span>
@@ -214,7 +215,8 @@ export function StorageStage({
// a reason, one click earlier.
unvouched={
location.status === "inconsistent" ||
- location.status === "unreachable"
+ location.status === "unreachable" ||
+ location.status === "stalled"
? `This channel's media location is ${location.status}: ${
location.detail ?? "disk and config do not agree"
} Moving back would delete the relocated copy, so it is refused until the location reads "relocated · reachable".`
diff --git a/editor/app/channels/[slug]/page.tsx b/editor/app/channels/[slug]/page.tsx
@@ -40,7 +40,15 @@ import {
type ShardOp,
} from "yt-dlp-transcript-common/controller/shard";
import { getPaths } from "yt-dlp-transcript-common/lib/paths";
-import { inspectChannelMedia } from "yt-dlp-transcript-common/lib/channelMedia";
+import {
+ channelMediaStall,
+ inspectChannelMedia,
+} from "yt-dlp-transcript-common/lib/channelMedia";
+import { MediaNotAnswering } from "./components/MediaNotAnswering";
+import {
+ isDriveNotAnswering,
+ onDrive,
+} from "yt-dlp-transcript-common/lib/storageHealth";
import { getFreeBytes } from "yt-dlp-transcript-common/lib/diskSpace";
import {
platformQueueKey,
@@ -515,12 +523,34 @@ export default async function ChannelDetailPage({
/>
);
case "cleanup": {
+ // The saved-video summary below reads a pointer in every video dir, and
+ // this stage's actions all act on the media: on a drive that is not
+ // answering the stage says so instead (see MediaNotAnswering).
+ const stall = channelMediaStall(config);
+ if (stall) {
+ return (
+ <MediaNotAnswering slug={slug} stall={stall} what="The Cleanup stage" />
+ );
+ }
// Saved-video store summary for this channel + whether backups are
- // configured, for the Retention & persistence section.
+ // configured, for the Retention & persistence section. Its reads go
+ // through the watchdog (`notAnswering`): a drive that stops answering
+ // on the way is named there, and the stage says so.
+ const notAnswering: string[] = [];
const savedTotals = await savedVideoTotals({
paths,
channelSlug: slug,
+ notAnswering,
});
+ if (notAnswering.length > 0) {
+ return (
+ <MediaNotAnswering
+ slug={slug}
+ stall={channelMediaStall(config)}
+ what="The Cleanup stage"
+ />
+ );
+ }
return (
<CleanupStage
slug={slug}
@@ -586,7 +616,19 @@ export default async function ChannelDetailPage({
// marker — reading the file again here was a second read of the same
// bytes that could disagree with the status rendered beside it.
const marker = media.marker ?? null;
- const freeBytes = await getFreeBytes(volumeDir);
+ // A statfs of the channel's drive goes through the watchdog
+ // (lib/storageHealth.ts): refused on a stalled location, given up on
+ // after the budget (3 s by default), and read "—" either way.
+ let freeBytes: number | null;
+ try {
+ freeBytes =
+ volumeDir === media.target && media.target
+ ? await onDrive(media.target, () => getFreeBytes(volumeDir))
+ : await getFreeBytes(volumeDir);
+ } catch (err) {
+ if (!isDriveNotAnswering(err)) throw err;
+ freeBytes = null;
+ }
// The SAME two conditions storageActions.ts refuses on, stated here as
// prose so the button is off with a reason rather than off and silent —
// and stated in the action too, because a disabled button is a courtesy
diff --git a/editor/app/channels/[slug]/videos/[id]/page.tsx b/editor/app/channels/[slug]/videos/[id]/page.tsx
@@ -42,6 +42,12 @@ import { TagsPanel } from "./components/TagsPanel";
import { MetadataHistoryDetails } from "./components/MetadataHistoryDetails";
import { loadVideoTags } from "./lib/videoTags";
import { loadVideoOperationPanels } from "./lib/videoOperationPanels";
+import { channelMediaStall } from "yt-dlp-transcript-common/lib/channelMedia";
+import {
+ isDriveNotAnswering,
+ onDrive,
+} from "yt-dlp-transcript-common/lib/storageHealth";
+import { MediaNotAnswering } from "../../components/MediaNotAnswering";
export const dynamic = "force-dynamic";
@@ -80,8 +86,19 @@ export async function generateMetadata({
params: Promise<{ slug: string; id: string }>;
}): Promise<Metadata> {
const { slug, id } = await params;
- const meta = await loadMeta(slug, id);
- const subject = meta.title ?? id;
+ // The title is read off the drive; a drive that is not answering is not
+ // asked, and one that does not answer within the budget
+ // (`storage.health.budgetMs`, 3 s by default) is given up on.
+ const drive = (await readChannelConfig(getPaths(), slug))?.dataDir?.trim();
+ let subject = id;
+ try {
+ const meta = await (drive
+ ? onDrive(drive, () => loadMeta(slug, id))
+ : loadMeta(slug, id));
+ subject = meta.title ?? id;
+ } catch (err) {
+ if (!isDriveNotAnswering(err)) throw err;
+ }
return { title: `${subject} — Video — ${slug}` };
}
@@ -93,63 +110,117 @@ export default async function VideoDetailPage({
const { slug, id } = await params;
const config = await readChannelConfig(getPaths(), slug);
if (!config) notFound();
- const dirData = await loadVideoDir(slug, id);
- const meta = await loadMeta(slug, id);
- const videoDir = path.join(getPaths().channelsDir, slug, "data", id);
- const downloadOutcome = await loadDownloadOutcome(videoDir);
- const availabilityRecord = await loadAvailability(videoDir);
- const availabilityHistory = availabilityRecord?.history ?? [];
- // Every rewrite of metadata.info.json that changed its bytes (release 10
- // slice N). One small bounded file, read once; null → nothing drawn.
- const metadataHistory = metadataHistoryView(
- await loadMetadataHistory(videoDir),
- Date.now(),
- );
- const doNotClean = await isDoNotClean(videoDir);
- const excludedFromTruncatedCheck =
- await isExcludedFromTruncatedCheck(videoDir);
- const savedVideo = await loadSavedVideo(videoDir);
- // The windows another tool asked this editor to fetch. One readdir of
- // data/<id>/clips/ plus a stat per file — and no per-CHANNEL count anywhere,
- // because that would be a walk of every video dir to draw one number.
- const clipWindows = await listClipWindows(videoDir);
- // Where each subtitle track came from, so the panel can say "YouTube
- // auto-captions" vs "manual captions" — and offer to replace the former with a
- // transcript of our own. One 4 KB head read per VTT (see subtitleProvenance).
- const vttProvenance: Record<string, SubtitleProvenance> = {};
- for (const f of dirData.files) {
- if (!isTranscriptVtt(f.name)) continue;
- vttProvenance[f.name] = await resolveVttProvenance(videoDir, f.name);
+ // Everything below reads the video's directory on the channel's drive; a
+ // drive that is not answering is not read. See MediaNotAnswering.
+ const stall = channelMediaStall(config);
+ if (stall) {
+ return <MediaNotAnswering slug={slug} stall={stall} what="This video's page" />;
}
- const cov = await readTranscriptCoverage(videoDir);
- const coverage = cov
- ? {
- lastCueEnd: cov.cov.lastCueEnd,
- duration: cov.cov.duration,
- coverage: cov.cov.coverage,
- incomplete:
- !excludedFromTruncatedCheck &&
- isIncompleteTranscript(cov.cov, {
- isLivestream: cov.isLivestream,
- }),
- }
- : null;
+ // EVERY READ BELOW IS OF THIS VIDEO'S DIRECTORY, on the channel's drive when
+ // it is relocated, so they go through the watchdog as one unit: a drive that
+ // has not answered them within the budget (3 s by default) is marked stalled
+ // and the page says so.
+ const loadAll = async () => {
+ const dirData = await loadVideoDir(slug, id);
+ const meta = await loadMeta(slug, id);
+ const videoDir = path.join(getPaths().channelsDir, slug, "data", id);
+ const downloadOutcome = await loadDownloadOutcome(videoDir);
+ const availabilityRecord = await loadAvailability(videoDir);
+ const availabilityHistory = availabilityRecord?.history ?? [];
+ // Every rewrite of metadata.info.json that changed its bytes (release 10
+ // slice N). One small bounded file, read once; null → nothing drawn.
+ const metadataHistory = metadataHistoryView(
+ await loadMetadataHistory(videoDir),
+ Date.now(),
+ );
+ const doNotClean = await isDoNotClean(videoDir);
+ const excludedFromTruncatedCheck =
+ await isExcludedFromTruncatedCheck(videoDir);
+ const savedVideo = await loadSavedVideo(videoDir);
+ // The windows another tool asked this editor to fetch. One readdir of
+ // data/<id>/clips/ plus a stat per file — and no per-CHANNEL count anywhere,
+ // because that would be a walk of every video dir to draw one number.
+ const clipWindows = await listClipWindows(videoDir);
+ // Where each subtitle track came from, so the panel can say "YouTube
+ // auto-captions" vs "manual captions" — and offer to replace the former with a
+ // transcript of our own. One 4 KB head read per VTT (see subtitleProvenance).
+ const vttProvenance: Record<string, SubtitleProvenance> = {};
+ for (const f of dirData.files) {
+ if (!isTranscriptVtt(f.name)) continue;
+ vttProvenance[f.name] = await resolveVttProvenance(videoDir, f.name);
+ }
+ const cov = await readTranscriptCoverage(videoDir);
+ const coverage = cov
+ ? {
+ lastCueEnd: cov.cov.lastCueEnd,
+ duration: cov.cov.duration,
+ coverage: cov.cov.coverage,
+ incomplete:
+ !excludedFromTruncatedCheck &&
+ isIncompleteTranscript(cov.cov, {
+ isLivestream: cov.isLivestream,
+ }),
+ }
+ : null;
- // ONE PANEL PER REGISTRY OPERATION, each carrying the state that entry's own
- // state() reports. The page used to re-derive the digest's freshness here —
- // its own target resolution and its own per-section fold beside the
- // registry's — which is how a page and a work list end up describing the same
- // disk differently.
- const panels = await loadVideoOperationPanels({
- paths: getPaths(),
- channelSlug: slug,
- videoId: id,
- settings: getSettings(),
- });
+ // ONE PANEL PER REGISTRY OPERATION, each carrying the state that entry's own
+ // state() reports. The page used to re-derive the digest's freshness here —
+ // its own target resolution and its own per-section fold beside the
+ // registry's — which is how a page and a work list end up describing the same
+ // disk differently.
+ const panels = await loadVideoOperationPanels({
+ paths: getPaths(),
+ channelSlug: slug,
+ videoId: id,
+ settings: getSettings(),
+ });
- // The curated tags on this video, with each one's provenance. Reads tags.json
- // plus (for rule hits) ONE key out of the transcript index — not a scan.
- const tagsView = await loadVideoTags(slug, id);
+ // The curated tags on this video, with each one's provenance. Reads tags.json
+ // plus (for rule hits) ONE key out of the transcript index — not a scan.
+ const tagsView = await loadVideoTags(slug, id);
+ return {
+ dirData,
+ meta,
+ videoDir,
+ downloadOutcome,
+ availabilityHistory,
+ metadataHistory,
+ doNotClean,
+ excludedFromTruncatedCheck,
+ savedVideo,
+ clipWindows,
+ vttProvenance,
+ coverage,
+ panels,
+ tagsView,
+ };
+ };
+ const drive = config.dataDir?.trim();
+ let loaded: Awaited<ReturnType<typeof loadAll>>;
+ try {
+ loaded = await (drive ? onDrive(drive, loadAll) : loadAll());
+ } catch (err) {
+ if (!isDriveNotAnswering(err)) throw err;
+ return (
+ <MediaNotAnswering slug={slug} stall={err.health} what="This video's page" />
+ );
+ }
+ const {
+ dirData,
+ meta,
+ videoDir,
+ downloadOutcome,
+ availabilityHistory,
+ metadataHistory,
+ doNotClean,
+ excludedFromTruncatedCheck,
+ savedVideo,
+ clipWindows,
+ vttProvenance,
+ coverage,
+ panels,
+ tagsView,
+ } = loaded;
const registry = getRegistry();
const existingQueues = registry.activeQueueNames();
diff --git a/editor/app/channels/[slug]/videos/page.tsx b/editor/app/channels/[slug]/videos/page.tsx
@@ -36,6 +36,12 @@ import { computeVideoRows, readDataDirVideoIds } from "../lib/videoRowsServer";
import { normalizeBuckets } from "yt-dlp-transcript-common/views/pipeline/stageStatus";
import { attachCuratedTags } from "../lib/videoTagRows";
import { readChannelVideoTitles } from "yt-dlp-transcript-common/controller/videoTitles";
+import { channelMediaStall } from "yt-dlp-transcript-common/lib/channelMedia";
+import {
+ isDriveNotAnswering,
+ onDrive,
+} from "yt-dlp-transcript-common/lib/storageHealth";
+import { MediaNotAnswering } from "../components/MediaNotAnswering";
export const dynamic = "force-dynamic";
@@ -100,6 +106,13 @@ export default async function ChannelVideosPage({
// A social channel has posts, not videos — there is no data directory to list
// and nothing here would render. 404 rather than an empty workspace.
if (isSocialChannel(config)) notFound();
+ // THE LIST IS READ OFF THE DRIVE (a readdir of data/, a head read per title,
+ // the selected video's files), so a drive that is not answering is not read:
+ // the page says so instead. See MediaNotAnswering.
+ const stall = channelMediaStall(config);
+ if (stall) {
+ return <MediaNotAnswering slug={slug} stall={stall} what="The video list" />;
+ }
const registry = getRegistry();
const existingQueues = registry.activeQueueNames();
@@ -131,14 +144,31 @@ export default async function ChannelVideosPage({
);
const channelDataDir = path.join(paths.channelsDir, slug, "data");
- const channelDataDirIds = await readDataDirVideoIds(channelDataDir);
- // What each video is called: one key range over the transcript index, one
- // read of the channel's metadata-scan.json, and a head read of
- // metadata.info.json only for what those two did not name. Measured at
- // ~80 ms for a synthetic 5,000-id channel (plans/release-8.md, slice V).
- const titles = await readChannelVideoTitles(paths, slug, [
- ...new Set([...channelDataDirIds, ...(snapshot.undownloadedIds ?? [])]),
- ]);
+ // THE READS OF THE DRIVE go through the watchdog when the channel is
+ // relocated: a drive that has not answered them within the budget
+ // (`storage.health.budgetMs`, 3 s by default) is marked stalled and the page
+ // says so instead (see MediaNotAnswering).
+ const drive = config.dataDir?.trim();
+ const onMedia = <T,>(call: () => Promise<T>): Promise<T> =>
+ drive ? onDrive(drive, call) : call();
+ let channelDataDirIds: string[];
+ let titles: Awaited<ReturnType<typeof readChannelVideoTitles>>;
+ try {
+ [channelDataDirIds, titles] = await onMedia(async () => {
+ const ids = await readDataDirVideoIds(channelDataDir);
+ // What each video is called: one key range over the transcript index,
+ // one read of the channel's metadata-scan.json, and a head read of
+ // metadata.info.json only for what those two did not name. Measured at
+ // ~80 ms for a synthetic 5,000-id channel (plans/release-8.md, slice V).
+ const named = await readChannelVideoTitles(paths, slug, [
+ ...new Set([...ids, ...(snapshot.undownloadedIds ?? [])]),
+ ]);
+ return [ids, named] as const;
+ });
+ } catch (err) {
+ if (!isDriveNotAnswering(err)) throw err;
+ return <MediaNotAnswering slug={slug} stall={err.health} what="The video list" />;
+ }
const rows = computeVideoRows({
channelDataDirIds,
snapshot,
@@ -188,8 +218,9 @@ export default async function ChannelVideosPage({
);
if (selectedVideoId) {
const videoDir = path.join(channelDataDir, selectedVideoId);
- const [dirData, title, outcome, availabilityRecord, cov, excludedTrunc] =
- await Promise.all([
+ let loadedVideo;
+ try {
+ loadedVideo = await onMedia(() => Promise.all([
loadVideoDir(channelDataDir, selectedVideoId),
// Already read for the list — the title map covers every row.
Promise.resolve(titles.get(selectedVideoId)?.title ?? null),
@@ -197,7 +228,13 @@ export default async function ChannelVideosPage({
loadAvailability(videoDir),
readTranscriptCoverage(videoDir),
isExcludedFromTruncatedCheck(videoDir),
- ]);
+ ]));
+ } catch (err) {
+ if (!isDriveNotAnswering(err)) throw err;
+ return <MediaNotAnswering slug={slug} stall={err.health} what="The video list" />;
+ }
+ const [dirData, title, outcome, availabilityRecord, cov, excludedTrunc] =
+ loadedVideo;
const coverage = cov
? {
lastCueEnd: cov.cov.lastCueEnd,
diff --git a/editor/app/channels/components/ChannelForm.tsx b/editor/app/channels/components/ChannelForm.tsx
@@ -96,7 +96,8 @@ type Props = {
sites: SiteMembershipOption[];
// Edit mode: the channel's current site memberships.
initialMemberships?: InitialMembership[];
- // Create mode: the active site (from ?site=) to pre-check.
+ // Create mode: the active site, which starts checked (ChannelFormClient: a
+ // `?site=` link's, else the stored selection).
activeSiteId?: string | null;
};
diff --git a/editor/app/channels/components/ChannelFormClient.tsx b/editor/app/channels/components/ChannelFormClient.tsx
@@ -1,10 +1,12 @@
"use client";
-import { useActionState, useEffect, useState } from "react";
+import { useActionState } from "react";
+import { useSearchParams } from "next/navigation";
import { ChannelForm } from "./ChannelForm";
import type { ChannelConfig } from "yt-dlp-transcript-common/lib/channelConfig";
import type { ActionResult } from "../actions";
-import { ALL_SITES } from "../../lib/activeSite";
+import { resolveActiveSiteFrom } from "../../lib/activeSite";
+import { useSiteScope } from "../../components/SiteScopeProvider";
import type {
InitialMembership,
SiteMembershipOption,
@@ -32,21 +34,18 @@ export function ChannelFormClient({
action,
undefined,
);
- // The active site is mirrored into the `?site=` URL param by SiteScopeSelect.
- // Read it here (avoiding useSearchParams so this form needn't be wrapped in a
- // Suspense boundary) so a newly created channel pre-checks that site in the
- // Sites membership section. Only meaningful in create mode — an existing
- // channel's memberships come from initialMemberships.
- const [activeSite, setActiveSite] = useState<string | null>(null);
- useEffect(() => {
- try {
- setActiveSite(new URLSearchParams(window.location.search).get("site"));
- } catch {
- /* ignore */
- }
- }, []);
- const activeSiteId =
- !initial && activeSite && activeSite !== ALL_SITES ? activeSite : null;
+ // A new channel starts checked on the active site, as the picker resolves it
+ // off a site's pages: a `?site=` link's, else the stored selection (the
+ // cookie, through SiteScopeProvider). Both are known on the server and on the
+ // first client render, so the box is checked in the first paint. Create mode
+ // only: an existing channel's memberships come from initialMemberships.
+ // (useSearchParams needs no Suspense here: both pages that render this form
+ // are dynamic.)
+ const { stored, siteIds } = useSiteScope();
+ const linkSite = useSearchParams().get("site");
+ const activeSiteId = initial
+ ? null
+ : resolveActiveSiteFrom([linkSite, stored], siteIds).siteId;
return (
<ChannelForm
action={formAction}
diff --git a/editor/app/channels/components/ChannelVolumeBar.tsx b/editor/app/channels/components/ChannelVolumeBar.tsx
@@ -21,10 +21,9 @@ import { LOCATION_FILTER_PARAM } from "yt-dlp-transcript-common/views/storage";
// per-row focus sentence made before the focus bar took it. So it lives here,
// one chip per volume, above the rack.
//
-// THE CHIP IS THE FILTER, and it is a LINK. `?location=` is a URL param beside
-// `?site=` for the reason that one is: the page is a server component, the
-// filter changes what the server sends, and a link is shareable — /storage
-// links straight to a filtered list. Client state would also lose the race with
+// THE CHIP IS THE FILTER, and it is a LINK. `?location=` is a URL param because
+// the page is a server component, the filter changes what the server sends,
+// and a link is shareable — /storage links straight to a filtered list. Client state would also lose the race with
// the global AutoRefresh's router.refresh(), which is why the sort is the one
// thing here that stays local.
@@ -42,6 +41,12 @@ export type ChannelVolume = {
// statfs of the root, or undefined when the root is not there (an unmounted
// drive). Never inferred from a parent — see volumeFreeBytes.
freeBytes?: number;
+ // "not answering since 11:35", while the health probe finds the drive not
+ // answering (its free space is then not asked either).
+ notAnswering?: string;
+ // With `notAnswering`: how many clean checks clear it, in words ("twice in a
+ // row"; settings.storage.health.clearAfterCleanPasses).
+ clears?: string;
};
export function ChannelVolumeBar({
@@ -62,8 +67,9 @@ export function ChannelVolumeBar({
const pathname = usePathname();
const params = useSearchParams();
// THE SITE SCOPE SURVIVES THE VOLUME FILTER. They are two independent
- // questions ("whose channels" and "which disk") and a chip that silently
- // dropped ?site= would answer one by discarding the other.
+ // questions ("whose channels" and "which disk"). The stored scope is a cookie
+ // and not in the URL; a `?site=` link's scope is, and a chip that silently
+ // dropped it would answer one question by discarding the other.
const href = (id: string | null): string => {
const next = new URLSearchParams(params.toString());
if (id === null) next.delete(LOCATION_FILTER_PARAM);
@@ -96,10 +102,14 @@ export function ChannelVolumeBar({
detail={
`${v.channels} ch · ${formatBytes(v.bytes)}` +
(v.unmeasured > 0 ? ` +${v.unmeasured}?` : "") +
- ` · ${v.freeBytes === undefined ? "free —" : `${formatBytes(v.freeBytes)} free`}`
+ (v.notAnswering
+ ? ` · ${v.notAnswering}`
+ : ` · ${v.freeBytes === undefined ? "free —" : `${formatBytes(v.freeBytes)} free`}`)
}
title={
- v.freeBytes === undefined
+ v.notAnswering
+ ? `${v.label}: the drive is ${v.notAnswering}. Pages and polls do not touch it until it answers ${v.clears ?? "again"}; its channels read "not answering".`
+ : v.freeBytes === undefined
? `${v.label}: the root is not there — an unmounted drive reports no free space rather than its parent's.`
: `${v.label}: ${v.channels} channel(s) hold ${formatBytes(v.bytes)}${
v.unmeasured > 0
diff --git a/editor/app/channels/components/SiteMembershipsSection.tsx b/editor/app/channels/components/SiteMembershipsSection.tsx
@@ -1,6 +1,6 @@
"use client";
-import { useEffect, useState } from "react";
+import { useState } from "react";
// The channel form's per-site membership picker: every configured site with a
// checkbox (member or not) plus a group dropdown, including a "+ New group…"
@@ -28,8 +28,9 @@ type Props = {
sites: SiteMembershipOption[];
// Edit mode: the channel's current memberships (pre-checked).
initialMemberships?: InitialMembership[];
- // Create mode: the active site from `?site=` to pre-check. Arrives via a
- // mount effect in ChannelFormClient, before any user interaction.
+ // Create mode: the active site, which starts checked. ChannelFormClient
+ // resolves it on its first render (a `?site=` link's, else the stored
+ // selection), so it is part of the initial state below.
activeSiteId?: string | null;
};
@@ -39,28 +40,28 @@ export function SiteMembershipsSection({
activeSiteId,
}: Props) {
// Presence in the map = checked. groupId "" = the site's default group.
- const [selected, setSelected] = useState<Map<string, Row>>(
- () =>
- new Map(
- (initialMemberships ?? []).map((m) => [
- m.siteId,
- { groupId: m.groupId ?? "", newGroupName: "" },
- ]),
- ),
- );
-
- // Create-mode pre-check of the active site (mirrors the old hidden
- // `activeSite` field's behavior). Fires before user interaction, so no
- // clobber guard beyond "already checked" is needed.
- useEffect(() => {
- if (!activeSiteId || !sites.some((s) => s.siteId === activeSiteId)) return;
- setSelected((prev) => {
- if (prev.has(activeSiteId)) return prev;
- const next = new Map(prev);
- next.set(activeSiteId, { groupId: "", newGroupName: "" });
- return next;
- });
- }, [activeSiteId, sites]);
+ //
+ // Create mode starts with the active site checked (mirrors the old hidden
+ // `activeSite` field's behavior). It is INITIAL state, not an effect: the
+ // server renders the box checked, so there is no unchecked first paint, and a
+ // later refresh (the pulse, or a site picked in another tab) cannot re-check a
+ // box the user has cleared.
+ const [selected, setSelected] = useState<Map<string, Row>>(() => {
+ const rows = new Map(
+ (initialMemberships ?? []).map((m) => [
+ m.siteId,
+ { groupId: m.groupId ?? "", newGroupName: "" },
+ ]),
+ );
+ if (
+ activeSiteId &&
+ !rows.has(activeSiteId) &&
+ sites.some((s) => s.siteId === activeSiteId)
+ ) {
+ rows.set(activeSiteId, { groupId: "", newGroupName: "" });
+ }
+ return rows;
+ });
if (sites.length === 0) {
// No hidden field at all: the actions skip membership reconciliation.
diff --git a/editor/app/channels/page.tsx b/editor/app/channels/page.tsx
@@ -8,6 +8,12 @@ import {
import { getPaths } from "yt-dlp-transcript-common/lib/paths";
import { inspectChannelMedia } from "yt-dlp-transcript-common/lib/channelMedia";
import {
+ healthTimings,
+ notAnsweringText,
+ stalledLocation,
+} from "yt-dlp-transcript-common/lib/storageHealth";
+import { clearRuleText } from "yt-dlp-transcript-common/lib/storageHealthTimings";
+import {
getSite,
listSiteIds,
listSites,
@@ -52,7 +58,7 @@ import {
} from "yt-dlp-transcript-common/views/channelGroupSections";
import { SyncAllChannelsButton } from "./components/SyncAllChannelsButton";
import { RefreshAllReportsButton } from "./components/RefreshAllReportsButton";
-import { resolveActiveSite } from "../lib/activeSite";
+import { readActiveSite } from "../lib/activeSiteServer";
export const dynamic = "force-dynamic";
@@ -163,7 +169,8 @@ export default async function ChannelsPage({
const paths = getPaths();
const settings = getSettings();
const { site, location: locationParam, sort: sortParam } = await searchParams;
- const active = resolveActiveSite(site, listSiteIds(paths));
+ // The active site: a `?site=` link's for this request, else the cookie's.
+ const active = await readActiveSite(site, listSiteIds(paths));
// Counts come from each channel's last snapshot, not a corpus walk. One read
// serves both the table and the freshness footer below.
const briefs = await listChannelBriefs(paths);
@@ -220,6 +227,15 @@ export default async function ChannelsPage({
// TWO SYSCALLS PER VOLUME, NOT A PROBE. See volumeFreeBytes: this table draws
// 71 rows on every auto-refresh and the rule is that tables never shell out.
const freeByVolume = await volumeFreeBytes({ paths, locations });
+ // A DRIVE THAT IS NOT ANSWERING, said on its chip. From memory — the health
+ // pass (the block device's counters, every `storage.health.passIntervalMs`)
+ // or the watchdog on a read is what found it (lib/storageHealth.ts); the
+ // table asks nothing.
+ const notAnsweringByVolume: Record<string, string | undefined> = {};
+ for (const loc of locations) {
+ const stall = stalledLocation(loc);
+ if (stall) notAnsweringByVolume[loc.id] = notAnsweringText(stall);
+ }
// ONE ROW PER CHANNEL, off the shared builder (common/views/channelRow.ts):
// the dashboard and the operation pages build theirs the same way. The view
// carries no `config` — this table is a client component, and a channel
@@ -292,6 +308,12 @@ export default async function ChannelsPage({
bytes: measured.reduce((sum, c) => sum + (c.mediaBytes ?? 0), 0),
unmeasured: rows.length - measured.length,
freeBytes: freeByVolume[id],
+ ...(notAnsweringByVolume[id]
+ ? {
+ notAnswering: notAnsweringByVolume[id],
+ clears: clearRuleText(healthTimings().clearAfterCleanPasses),
+ }
+ : {}),
};
})
// The unnamed-root chip only exists when something is actually on one.
diff --git a/editor/app/components/MediaLocationBadge.tsx b/editor/app/components/MediaLocationBadge.tsx
@@ -10,18 +10,19 @@ import type { ChannelRowMedia } from "yt-dlp-transcript-common/views/channelRow"
// the filesystem; the inspect() call that produces the location happens on the
// server, once per row, and only its result travels.
//
-// WHAT THE FIVE STATUSES LOOK LIKE, and why there are only three appearances:
+// WHAT THE SIX STATUSES LOOK LIKE, and why there are only three appearances:
//
// in-place → NOTHING. The overwhelming majority of channels are in place,
// and a badge on every row saying "normal" is noise that makes
// the two that matter harder to see, not easier.
// ok → neutral. Relocated and reachable is a fact worth stating (the
// bytes are not on the corpus disk) but it is not a problem.
-// everything → red. unreachable, in-transition and inconsistent are all
-// else "do not trust what this channel's dirs say right now": the
+// everything → red. unreachable, in-transition, inconsistent and stalled are
+// else all "do not trust what this channel's dirs say right now": the
// first because the drive is not mounted, the second because a
// move is half-done, the third because disk and config disagree
-// and nothing here is willing to guess which one is right.
+// and nothing here is willing to guess which one is right, the
+// fourth because the drive is mounted and not answering.
//
// The `detail` string is the operator's prose from inspect() — the drive path,
// the phase, the disagreement — and it goes on `title` so a row badge carries
@@ -52,6 +53,7 @@ const LABELS: Record<ChannelMediaStatus, string | null> = {
unreachable: "Media unreachable",
"in-transition": "Media moving",
inconsistent: "Media inconsistent",
+ stalled: "Media not answering",
};
// The one-word state, for the compact rendering of a NAMED location: "on
@@ -63,6 +65,7 @@ const SHORT_STATUS: Record<ChannelMediaStatus, string | null> = {
unreachable: "unreachable",
"in-transition": "moving",
inconsistent: "inconsistent",
+ stalled: "not answering",
};
// Null means "draw nothing" — an in-place channel, or no location at all (a
diff --git a/editor/app/components/SiteScopeProvider.tsx b/editor/app/components/SiteScopeProvider.tsx
@@ -0,0 +1,224 @@
+"use client";
+
+import {
+ createContext,
+ useCallback,
+ useContext,
+ useEffect,
+ useMemo,
+ useRef,
+ useState,
+ type ReactNode,
+} from "react";
+import { usePathname, useRouter } from "next/navigation";
+import {
+ ACTIVE_SITE_CHANNEL,
+ ACTIVE_SITE_KEY,
+ isStorableActiveSite,
+ siteIdFromPathname,
+} from "../lib/activeSite";
+import { setActiveSiteAction } from "../lib/activeSiteActions";
+
+// THE SITE SCOPE, supplied once by the root layout (see app/lib/activeSite.ts).
+//
+// `activeSite` is the cookie's selection, resolved by the layout's
+// readActiveSite() on the server, so the server and the first client render
+// agree and the picker paints the stored site at once. `stored` below starts
+// from it and moves ahead of it optimistically when a choice is made; when the
+// server's value changes (the re-render a cookie write triggers, or a refresh
+// after another tab chose), it follows that.
+//
+// OTHER TABS. The cookie is shared by every tab of this origin, but a tab's
+// `stored` comes from its root layout, which a client-side navigation does not
+// re-render: a site picked in one tab left the others showing the old one over
+// pages that read the new one. So every successful write is announced on a
+// BroadcastChannel (ACTIVE_SITE_CHANNEL; per origin, so per port, like the
+// cookie's name), and the other tabs answer with router.refresh(): the layout
+// and the page re-render with the cookie as it is now, and the tab's router
+// cache is dropped. A channel does not deliver a message to the instance that
+// posted it, so a tab does not refresh for its own write. A browser without
+// BroadcastChannel keeps each tab's value until its next full load.
+//
+// Two effects, and neither rewrites a URL:
+// - visiting a site's own page (/sites/<id>/…) records that site, so Dashboard
+// and Channels follow — the path already shows it, so there is no flash;
+// - the one-time MIGRATION: a visitor with the old localStorage key and no
+// cookie gets it copied into the cookie, then the key is removed. That is the
+// only localStorage read, and the one paint it may cost is on the first visit
+// after the update.
+// Both only WRITE the cookie (`record`); the value comes back through the
+// server's re-render. They run as the page hydrates, and a state change there
+// would re-render the picker before React replays a choice made on the
+// server-rendered select before hydration: the re-render resets the select to
+// its old value, and the replayed change reads that instead of the choice.
+type SiteScope = {
+ siteIds: string[];
+ // The stored selection: a site id or "__all__". The picker resolves it
+ // against the path and a `?site=` param; it is not the effective scope alone.
+ stored: string;
+ // Store a choice: shown at once, written through setActiveSiteAction. Resolves
+ // once the cookie is set (and the server has re-rendered with it) to true, or
+ // to false with the previous selection put back when the write failed.
+ choose: (value: string) => Promise<boolean>;
+};
+
+const SiteScopeContext = createContext<SiteScope | null>(null);
+
+export function SiteScopeProvider({
+ activeSite,
+ fromCookie,
+ siteIds,
+ children,
+}: {
+ activeSite: string;
+ // Whether the request carried the cookie at all (the migration's trigger).
+ fromCookie: boolean;
+ siteIds: string[];
+ children: ReactNode;
+}) {
+ const [stored, setStored] = useState(activeSite);
+ // Follow the server's value when IT changes — not on every render, so an
+ // optimistic choice is not undone by a refresh that raced the cookie write —
+ // and not while a write is in flight: two quick choices re-render twice, and
+ // the first one's render must not paint over the second choice.
+ const [seen, setSeen] = useState(activeSite);
+ const [writing, setWriting] = useState(0);
+ if (seen !== activeSite) {
+ setSeen(activeSite);
+ if (writing === 0) setStored(activeSite);
+ }
+ // What `stored` is now, for the effects below without making them re-run on
+ // it. Synced after commit (declared first, so it runs before them), and moved
+ // at once by `choose`.
+ const storedRef = useRef(stored);
+ useEffect(() => {
+ storedRef.current = stored;
+ }, [stored]);
+
+ // The other tabs (see above): subscribe, and keep the one instance to post on.
+ const router = useRouter();
+ const channel = useRef<BroadcastChannel | null>(null);
+ useEffect(() => {
+ if (typeof BroadcastChannel === "undefined") return;
+ const ch = new BroadcastChannel(ACTIVE_SITE_CHANNEL);
+ channel.current = ch;
+ ch.onmessage = () => router.refresh();
+ return () => {
+ ch.close();
+ if (channel.current === ch) channel.current = null;
+ };
+ }, [router]);
+ const announce = useCallback((value: string) => {
+ try {
+ channel.current?.postMessage(value);
+ } catch {
+ /* closed while unmounting: nothing to tell */
+ }
+ }, []);
+
+ const choose = useCallback(async (value: string): Promise<boolean> => {
+ const previous = storedRef.current;
+ storedRef.current = value;
+ setStored(value);
+ setWriting((n) => n + 1);
+ let ok = false;
+ try {
+ ok = await setActiveSiteAction(value);
+ } catch {
+ ok = false;
+ }
+ setWriting((n) => n - 1);
+ if (ok) {
+ announce(value);
+ } else {
+ storedRef.current = previous;
+ setStored(previous);
+ }
+ return ok;
+ }, [announce]);
+
+ // Write a value the page did not choose — no optimistic state (see above).
+ const record = useCallback(async (value: string): Promise<boolean> => {
+ let ok = false;
+ try {
+ ok = await setActiveSiteAction(value);
+ } catch {
+ ok = false;
+ }
+ if (ok) announce(value);
+ return ok;
+ }, [announce]);
+
+ // Visiting a site's own page records it: once per arrival at that site's
+ // pages (a re-run of the effect — StrictMode, Fast Refresh — does not write
+ // again), and not when the store already holds it.
+ const pathname = usePathname();
+ const pathSite = siteIdFromPathname(pathname)?.siteId ?? null;
+ const known = pathSite !== null && siteIds.includes(pathSite);
+ const recordedFor = useRef<string | null>(null);
+ useEffect(() => {
+ if (!known || pathSite === null) {
+ recordedFor.current = null;
+ return;
+ }
+ if (recordedFor.current === pathSite) return;
+ recordedFor.current = pathSite;
+ if (pathSite === storedRef.current) return;
+ void record(pathSite);
+ }, [known, pathSite, record]);
+
+ // The one-time move from localStorage. Guarded by a ref, not by deps: it is
+ // about this page load, and StrictMode's second mount must not repeat it.
+ const migrated = useRef(false);
+ useEffect(() => {
+ if (migrated.current) return;
+ migrated.current = true;
+ const forget = () => {
+ try {
+ window.localStorage.removeItem(ACTIVE_SITE_KEY);
+ } catch {
+ /* storage unavailable: nothing to forget */
+ }
+ };
+ // A cookie already decides, and on a site's page the path does (the effect
+ // above records it): the old key has nothing left to say.
+ if (fromCookie || known) {
+ forget();
+ return;
+ }
+ let legacy: string | null = null;
+ try {
+ legacy = window.localStorage.getItem(ACTIVE_SITE_KEY);
+ } catch {
+ /* storage unavailable */
+ }
+ if (!isStorableActiveSite(legacy)) {
+ forget();
+ return;
+ }
+ // Removed only once the cookie holds it, so a failed write loses nothing.
+ void record(legacy).then((ok) => {
+ if (ok) forget();
+ });
+ // Once per page load, on purpose (see the ref).
+ // eslint-disable-next-line react-hooks/exhaustive-deps
+ }, []);
+
+ const value = useMemo(
+ () => ({ siteIds, stored, choose }),
+ [siteIds, stored, choose],
+ );
+ return (
+ <SiteScopeContext.Provider value={value}>
+ {children}
+ </SiteScopeContext.Provider>
+ );
+}
+
+export function useSiteScope(): SiteScope {
+ const scope = useContext(SiteScopeContext);
+ if (!scope) {
+ throw new Error("useSiteScope() outside SiteScopeProvider (app/layout.tsx)");
+ }
+ return scope;
+}
diff --git a/editor/app/components/SiteScopeSelect.tsx b/editor/app/components/SiteScopeSelect.tsx
@@ -1,84 +1,62 @@
"use client";
-import { useEffect } from "react";
+import { useState } from "react";
import { usePathname, useRouter, useSearchParams } from "next/navigation";
import {
- ACTIVE_SITE_KEY,
ALL_SITES,
- resolveActiveSite,
+ resolveActiveSiteFrom,
siteIdFromPathname,
+ withoutSiteParam,
} from "../lib/activeSite";
+import { useSiteScope } from "./SiteScopeProvider";
export type SiteScopeOption = { siteId: string; siteTitle: string };
-// The single global site selector shown in the sidebar. Persists the choice in
-// localStorage (the source of truth) and mirrors it into the `?site=` URL param
-// so the server-rendered scoped pages (Dashboard, Channels) can read it from
-// their `searchParams`. On a site's own pages (`/sites/<id>/…`) the path IS the
-// selection: the picker shows it, writes it to storage so Dashboard and Channels
-// follow, and changing it navigates to the same tab of the other site. See
+// The single global site selector shown in the sidebar. The stored selection is
+// a cookie, read by the root layout and supplied by SiteScopeProvider, so this
+// renders the right value on the FIRST paint, on the server and on every
+// navigation — it has no effect that rewrites the URL and reads no storage. See
// app/lib/activeSite.ts.
+//
+// What it shows, first match wins: the choice just made on this URL (until the
+// navigation it starts lands), the site a /sites/<id>/… path names, a valid
+// `?site=` on the URL (a link's scope for that page), the stored selection.
export function SiteScopeSelect({ sites }: { sites: SiteScopeOption[] }) {
const router = useRouter();
const pathname = usePathname();
const searchParams = useSearchParams();
- const siteIds = sites.map((s) => s.siteId);
- const urlValue = searchParams.get("site");
+ const { siteIds, stored, choose } = useSiteScope();
+ const search = searchParams.toString();
+ const param = searchParams.get("site");
const onSite = siteIdFromPathname(pathname);
- const resolved = resolveActiveSite(onSite?.siteId ?? urlValue, siteIds);
+ // A controlled <select> snaps back to its value when the change does not
+ // re-render it with the new one. Where the path or the param outranks the
+ // stored value, the choice is held here for the URL it was made on, so the
+ // select shows it until the navigation it starts replaces that URL. Once the
+ // URL has moved it is dropped, so coming back to that URL later (Back, or a
+ // link) shows what the URL says, not an old choice.
+ const urlKey = `${pathname}?${search}`;
+ const [pending, setPending] = useState<{ urlKey: string; value: string } | null>(
+ null,
+ );
+ if (pending !== null && pending.urlKey !== urlKey) setPending(null);
+ const held = pending?.urlKey === urlKey ? pending.value : null;
+ const resolved = resolveActiveSiteFrom(
+ [held, onSite?.siteId, param, stored],
+ siteIds,
+ );
const multi = sites.length > 1;
- // The /sites CRUD pages manage every site and never read ?site=; seeding it
- // there would only let the mount-effect replace() below clobber an in-flight
- // push to /sites/<id> (a link click on the list), bouncing the user back to
- // /sites?site=<id>. Only the scoped server pages (Dashboard, Channels) consume
- // the param, so restrict the seed to non-/sites routes. On `/sites/<id>/…` the
- // picker reads the path instead (below) and never seeds either.
- const seedsSiteParam = !pathname.startsWith("/sites");
-
- function setParam(value: string) {
- const params = new URLSearchParams(searchParams.toString());
- params.set("site", value);
- router.replace(`${pathname}?${params.toString()}`, { scroll: false });
- }
- // Keep URL and localStorage reconciled. An explicit, valid URL param wins (and
- // is written back to localStorage); otherwise seed the URL from the stored
- // preference so server components pick up the active site on navigation.
- useEffect(() => {
- if (siteIds.length === 0) return;
- // The path wins over the param: on a site's own pages it IS the selection,
- // and writing it to storage is what makes Dashboard and Channels follow.
- const chosen = onSite?.siteId ?? urlValue;
- const chosenIsValid =
- chosen === ALL_SITES || (!!chosen && siteIds.includes(chosen));
- if (chosenIsValid) {
- try {
- window.localStorage.setItem(ACTIVE_SITE_KEY, chosen as string);
- } catch {
- /* ignore */
- }
- return;
- }
- if (!seedsSiteParam) return;
- let stored: string | null = null;
- try {
- stored = window.localStorage.getItem(ACTIVE_SITE_KEY);
- } catch {
- /* ignore */
- }
- setParam(resolveActiveSite(stored, siteIds).value);
- // eslint-disable-next-line react-hooks/exhaustive-deps
- }, [urlValue, pathname, siteIds.join(",")]);
-
- function onChange(value: string) {
- try {
- window.localStorage.setItem(ACTIVE_SITE_KEY, value);
- } catch {
- /* ignore */
- }
+ async function onChange(value: string) {
if (onSite) {
- // The site is the path here, so changing it is a navigation, not a param:
- // the same tab of the other site, or the family page for "All sites".
+ // The site is the path here, so changing it is a navigation: the same tab
+ // of the other site, or the family page for "All sites". The cookie is
+ // written FIRST, so a page opened right after the URL moves reads it.
+ setPending({ urlKey, value });
+ if (!(await choose(value))) {
+ setPending(null);
+ return;
+ }
router.push(
value === ALL_SITES
? "/sites"
@@ -86,7 +64,20 @@ export function SiteScopeSelect({ sites }: { sites: SiteScopeOption[] }) {
);
return;
}
- setParam(value);
+ if (param === null) {
+ // The common case: the stored value is what shows, and the cookie write
+ // re-renders this page (Dashboard, Channels) in the new scope.
+ await choose(value);
+ return;
+ }
+ // A `?site=` link's page: the choice replaces the link's scope, so the
+ // param goes — after the cookie is written, so the new URL renders with it.
+ setPending({ urlKey, value });
+ if (!(await choose(value))) {
+ setPending(null);
+ return;
+ }
+ router.replace(withoutSiteParam(pathname, search), { scroll: false });
}
if (sites.length === 0) {
@@ -102,7 +93,7 @@ export function SiteScopeSelect({ sites }: { sites: SiteScopeOption[] }) {
<span className="text-xs uppercase tracking-wide text-muted-foreground">site</span>
<select
value={resolved.value}
- onChange={(e) => onChange(e.target.value)}
+ onChange={(e) => void onChange(e.target.value)}
aria-label="Active site"
className="rounded border border-border bg-card px-2 py-1 text-sm"
>
diff --git a/editor/app/components/SocialLinksField.tsx b/editor/app/components/SocialLinksField.tsx
@@ -1,11 +1,17 @@
"use client";
import type { SocialLink } from "yt-dlp-transcript-common/lib/settings";
+import { socialLinksJson, type SocialRow } from "./socialLinksJson";
-export type SocialRow = { label: string; url: string; svg: string };
+export type { SocialRow };
export function toSocialRow(s: SocialLink): SocialRow {
- return { label: s.label, url: s.url, svg: s.svg };
+ return {
+ label: s.label,
+ url: s.url,
+ svg: s.svg,
+ ...(s.featured ? { featured: true } : {}),
+ };
}
type Props = {
@@ -32,13 +38,7 @@ export function SocialLinksField({ value, onChange, name }: Props) {
const update = (idx: number, patch: Partial<SocialRow>) =>
onChange(value.map((row, i) => (i === idx ? { ...row, ...patch } : row)));
- const json = JSON.stringify(
- value.map((s) => ({
- label: s.label.trim(),
- url: s.url.trim(),
- svg: s.svg.trim(),
- })),
- );
+ const json = socialLinksJson(value);
return (
<>
@@ -96,6 +96,23 @@ export function SocialLinksField({ value, onChange, name }: Props) {
placeholder='<svg viewBox="0 0 24 24"><path d="…"/></svg>'
className="rounded border border-border bg-card px-2 py-1 text-xs font-mono"
/>
+ {/* On a small screen a header shows only the links checked here
+ (`featured`); a wide one shows every link, up to four, these
+ kept first (common/lib/socialLinks.ts headerSocialLinks). */}
+ <div className="flex flex-wrap items-center gap-x-3 gap-y-1">
+ <label className="flex items-center gap-2 text-sm">
+ <input
+ type="checkbox"
+ checked={s.featured === true}
+ onChange={(e) => update(idx, { featured: e.target.checked })}
+ className="accent-brand"
+ />
+ Keep in header on small screens
+ </label>
+ <span className="text-xs text-muted-foreground">
+ On small screens the header shows only these; the rest stay in the footer.
+ </span>
+ </div>
</div>
))}
<div>
diff --git a/editor/app/components/socialLinksJson.test.ts b/editor/app/components/socialLinksJson.test.ts
@@ -0,0 +1,23 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { parseSocialLinks } from "yt-dlp-transcript-common/lib/settingsSchema";
+import { socialLinksJson } from "./socialLinksJson";
+
+const SVG = `<svg viewBox="0 0 24 24"><path d="M0 0"/></svg>`;
+
+test("Keep in header on small screens: a checked row posts featured: true, an unchecked one no key", () => {
+ const posted = JSON.parse(
+ socialLinksJson([
+ { label: " A ", url: " https://a.example ", svg: ` ${SVG} `, featured: true },
+ { label: "B", url: "https://b.example", svg: SVG, featured: false },
+ { label: "C", url: "https://c.example", svg: SVG },
+ ]),
+ );
+ assert.deepEqual(posted, [
+ { label: "A", url: "https://a.example", svg: SVG, featured: true },
+ { label: "B", url: "https://b.example", svg: SVG },
+ { label: "C", url: "https://c.example", svg: SVG },
+ ]);
+ // …and the actions' parser keeps exactly that.
+ assert.deepEqual(parseSocialLinks(posted), posted);
+});
diff --git a/editor/app/components/socialLinksJson.ts b/editor/app/components/socialLinksJson.ts
@@ -0,0 +1,23 @@
+// What the social-links editor posts: its rows, trimmed, as the JSON both the
+// settings and the site actions parse with parseSocialLinks. Pure, so it has a
+// unit test (socialLinksJson.test.ts) without a DOM.
+
+export type SocialRow = {
+ label: string;
+ url: string;
+ svg: string;
+ // "Keep in header on small screens". Posted only when checked, the way the
+ // schema stores it.
+ featured?: boolean;
+};
+
+export function socialLinksJson(rows: readonly SocialRow[]): string {
+ return JSON.stringify(
+ rows.map((s) => ({
+ label: s.label.trim(),
+ url: s.url.trim(),
+ svg: s.svg.trim(),
+ ...(s.featured ? { featured: true } : {}),
+ })),
+ );
+}
diff --git a/editor/app/globals.css b/editor/app/globals.css
@@ -2,7 +2,7 @@
@import "../../common/styles/tokens.css";
@source "../../common/components";
-/* Design tokens, the `dark` variant, the three bases (light / sepia / dark) and
+/* Design tokens, the `dark` variant, the two bases (light / dark) and
the accents live in common/styles/tokens.css; the faces in
common/styles/fonts.ts. The editor follows the operator's system base with
the Signal accent, as a compact, command-first cockpit (shell + dashboard +
diff --git a/editor/app/layout.tsx b/editor/app/layout.tsx
@@ -11,7 +11,6 @@ import { fontVars } from "yt-dlp-transcript-common/styles/fonts";
import { ThemeScript } from "yt-dlp-transcript-common/components/ThemeScript";
import { ThemeProvider } from "yt-dlp-transcript-common/components/ThemeProvider";
import { ThemeToggle } from "yt-dlp-transcript-common/components/ThemeToggle";
-import { ThemeMenu } from "yt-dlp-transcript-common/components/ThemeMenu";
import { BrandMark } from "yt-dlp-transcript-common/components/BrandMark";
import { ICON_PALETTES } from "yt-dlp-transcript-common/lib/brand";
import { AppFrame } from "./components/AppFrame";
@@ -19,6 +18,8 @@ import { AutoRefresh } from "./components/AutoRefresh";
import { CommandPalette } from "./components/CommandPalette";
import { ChangelogNavLink } from "./components/ChangelogNavLink";
import { SiteScopeSelect } from "./components/SiteScopeSelect";
+import { SiteScopeProvider } from "./components/SiteScopeProvider";
+import { readActiveSite } from "./lib/activeSiteServer";
import { Toaster } from "yt-dlp-transcript-common/components/ui/sonner";
import { NAV_GROUPS, type NavLink } from "./lib/nav";
import { CleanableBadge } from "./components/CleanableBadge";
@@ -63,6 +64,12 @@ export default async function RootLayout({
siteId: s.siteId,
siteTitle: s.siteTitle,
}));
+ const siteIds = sites.map((s) => s.siteId);
+ // THE SITE SCOPE, read once per request from its cookie and supplied to the
+ // picker (and anything else that asks) by SiteScopeProvider. A layout has no
+ // searchParams, so a `?site=` link is applied by the picker from the URL and
+ // by the scoped pages from their own searchParams. See app/lib/activeSite.ts.
+ const scope = await readActiveSite(null, siteIds);
const navItemClass =
"px-2.5 py-1.5 rounded-md text-foreground/80 hover:text-foreground hover:bg-muted whitespace-nowrap flex items-center gap-2 transition-colors";
const renderLink = (link: NavLink) => {
@@ -129,6 +136,11 @@ export default async function RootLayout({
<body className="min-h-full flex flex-col md:flex-row bg-background text-foreground">
<ThemeScript />
<ThemeProvider>
+ <SiteScopeProvider
+ activeSite={scope.value}
+ fromCookie={scope.stored !== null}
+ siteIds={siteIds}
+ >
<AppFrame
sidebar={
<aside className="md:w-56 md:shrink-0 md:sticky md:top-0 md:self-start md:h-screen md:overflow-y-auto border-b md:border-b-0 md:border-r border-border bg-card flex flex-col">
@@ -153,10 +165,14 @@ export default async function RootLayout({
</div>
</div>
<div className="flex items-center gap-1.5 shrink-0">
- <ThemeMenu />
<ThemeToggle />
</div>
</div>
+ {/* The picker reads useSearchParams() (a `?site=` link). This
+ layout reads a cookie, so every page renders per request and
+ the hook does not suspend on the server: the fallback is never
+ what the first paint shows. The boundary stays for the
+ production build's missing-Suspense check. */}
<Suspense fallback={null}>
<SiteScopeSelect sites={sites} />
</Suspense>
@@ -198,6 +214,7 @@ export default async function RootLayout({
>
{children}
</AppFrame>
+ </SiteScopeProvider>
</ThemeProvider>
</body>
</html>
diff --git a/editor/app/lib/activeSite.test.ts b/editor/app/lib/activeSite.test.ts
@@ -1,6 +1,14 @@
import { test } from "node:test";
import assert from "node:assert/strict";
-import { siteIdFromPathname } from "./activeSite";
+import {
+ ACTIVE_SITE_COOKIE,
+ ALL_SITES,
+ activeSiteCookieName,
+ isStorableActiveSite,
+ resolveActiveSiteFrom,
+ siteIdFromPathname,
+ withoutSiteParam,
+} from "./activeSite";
// Run with: pnpm -C editor exec tsx --test "app/**/*.test.ts"
//
@@ -37,3 +45,78 @@ test("routes outside /sites name no site", () => {
test("a deeper path than a tab names no site", () => {
assert.equal(siteIdFromPathname("/sites/a/b/c"), null);
});
+
+// ── The cookie (release 15 slice SS) ─────────────────────────────────────────
+//
+// The store is a cookie read on the server; these are its pure rules: the name a
+// request uses, what may be written, and which value wins.
+
+test("the cookie name carries the port, so two editors on one host keep one each", () => {
+ assert.equal(activeSiteCookieName("localhost:3001"), `${ACTIVE_SITE_COOKIE}-3001`);
+ assert.equal(activeSiteCookieName("127.0.0.1:3411"), `${ACTIVE_SITE_COOKIE}-3411`);
+ assert.equal(activeSiteCookieName("[::1]:3101"), `${ACTIVE_SITE_COOKIE}-3101`);
+});
+
+test("a host with no port, or no host, uses the base name", () => {
+ assert.equal(activeSiteCookieName("editor.example.org"), ACTIVE_SITE_COOKIE);
+ assert.equal(activeSiteCookieName(null), ACTIVE_SITE_COOKIE);
+ assert.equal(activeSiteCookieName(undefined), ACTIVE_SITE_COOKIE);
+ assert.equal(activeSiteCookieName("[::1]"), ACTIVE_SITE_COOKIE);
+});
+
+test("a site id or the all-sites sentinel may be stored", () => {
+ assert.equal(isStorableActiveSite("alpha"), true);
+ assert.equal(isStorableActiveSite("a-1"), true);
+ assert.equal(isStorableActiveSite(ALL_SITES), true);
+});
+
+test("anything else is refused: the value goes into a header", () => {
+ for (const bad of [
+ "",
+ "Alpha",
+ "-alpha",
+ "a b",
+ "a;b",
+ "a=b",
+ "a\r\nSet-Cookie: x=y",
+ "_homepage",
+ "a".repeat(129),
+ null,
+ undefined,
+ 42,
+ ]) {
+ assert.equal(isStorableActiveSite(bad), false, JSON.stringify(bad));
+ }
+ assert.equal(isStorableActiveSite("a".repeat(128)), true);
+});
+
+test("the first candidate naming a configured site wins", () => {
+ const ids = ["alpha", "beta"];
+ // A `?site=` link beats the cookie.
+ assert.equal(resolveActiveSiteFrom(["beta", "alpha"], ids).value, "beta");
+ // No param: the cookie.
+ assert.equal(resolveActiveSiteFrom([null, "alpha"], ids).value, "alpha");
+ assert.equal(resolveActiveSiteFrom([undefined, "alpha"], ids).siteId, "alpha");
+ // The sentinel is a real choice, not a miss.
+ assert.equal(resolveActiveSiteFrom([ALL_SITES, "alpha"], ids).isAll, true);
+});
+
+test("a param naming no site falls through to the cookie", () => {
+ const ids = ["alpha", "beta"];
+ assert.equal(resolveActiveSiteFrom(["gone", "beta"], ids).value, "beta");
+});
+
+test("nothing valid resolves to the default: the lone site, else all sites", () => {
+ assert.equal(resolveActiveSiteFrom([null, "gone"], ["alpha", "beta"]).value, ALL_SITES);
+ assert.equal(resolveActiveSiteFrom([null, null], ["solo"]).value, "solo");
+ assert.equal(resolveActiveSiteFrom([], []).value, ALL_SITES);
+});
+
+test("dropping the site param keeps every other param", () => {
+ assert.equal(withoutSiteParam("/channels", "site=alpha"), "/channels");
+ assert.equal(
+ withoutSiteParam("/channels", "site=alpha&location=internal&sort=size"),
+ "/channels?location=internal&sort=size",
+ );
+ assert.equal(withoutSiteParam("/", ""), "/");
+});
diff --git a/editor/app/lib/activeSite.ts b/editor/app/lib/activeSite.ts
@@ -1,25 +1,73 @@
// Shared notion of "the site I'm working on" for the editor's site-scoped views
-// (Dashboard, Channels). The selection is persisted client-side in localStorage
-// by SiteScopeSelect, but those pages are server components, so the selection is
-// mirrored into the URL `?site=` query param and read here from the page's
-// `searchParams`. This module has NO server-only imports so it can be shared by
-// the client selector too.
+// (Dashboard, Channels) and the sidebar's "Active site" picker. This module has
+// NO server-only imports so the client picker and provider can share it.
//
-// ON A SITE'S OWN PAGES the site is the PATH, not the param: /sites/<id>/<tab>
-// names it, the tab reads `params.siteId`, and the picker reads the same path
-// through `siteIdFromPathname` so the two never disagree.
+// THE STORE IS A COOKIE (release 15 slice SS). The root layout reads it once per
+// request (`readActiveSite()`, app/lib/activeSiteServer.ts) and hands it to
+// `SiteScopeProvider`; Dashboard and Channels read it through the same helper.
+// So the server renders the picker, and the scoped pages, with the stored
+// selection on the first paint — nothing is reconciled after it. It is written
+// only by `setActiveSiteAction` (app/lib/activeSiteActions.ts).
+//
+// A `?site=<id>` LINK still works: a valid param governs that one request (the
+// server pages read it from `searchParams`, the picker from the URL), and is not
+// written to the cookie. The editor's own navigation no longer appends it.
+//
+// ON A SITE'S OWN PAGES the site is the PATH, not the cookie or the param:
+// /sites/<id>/<tab> names it, the tab reads `params.siteId`, and the picker
+// reads the same path through `siteIdFromPathname` so the two never disagree.
+// Visiting one records that site in the cookie, so Dashboard and Channels follow.
-// Sentinel param value meaning "all sites" (full channel pool). A bare string so
-// it can never collide with a real siteId (which matches SITE_ID_RE).
+// Sentinel value meaning "all sites" (full channel pool). A bare string so it
+// can never collide with a real siteId (which matches SITE_ID_RE).
export const ALL_SITES = "__all__";
-// localStorage key the selector reads/writes.
+// The localStorage key the picker used before the cookie. Read ONCE, by
+// SiteScopeProvider's migration, when a visitor has it and no cookie; then
+// removed. Nothing else reads it.
export const ACTIVE_SITE_KEY = "activeSite";
+// The cookie's base name. The name a request uses carries the port it was made
+// to (`activeSiteCookieName`), because a cookie is shared by every port on a
+// host while localStorage was per origin: two editors on one machine (the live
+// one and a worktree's) keep a selection each, as they did.
+export const ACTIVE_SITE_COOKIE = "archilyzer-active-site";
+
+// The BroadcastChannel a successful write is announced on, so the editor's
+// other tabs refresh (SiteScopeProvider). A channel is per origin, so each port
+// has its own, like the cookie's name.
+export const ACTIVE_SITE_CHANNEL = "archilyzer-active-site";
+
+// One year, in seconds (the cookie's `maxAge`).
+export const ACTIVE_SITE_COOKIE_MAX_AGE = 60 * 60 * 24 * 365;
+
+// "localhost:3001" → "archilyzer-active-site-3001"; a host with no port (the
+// default port, behind a proxy) or no host at all → the base name.
+export function activeSiteCookieName(host: string | null | undefined): string {
+ const port = host ? /:(\d+)$/.exec(host)?.[1] : undefined;
+ return port ? `${ACTIVE_SITE_COOKIE}-${port}` : ACTIVE_SITE_COOKIE;
+}
+
+// SITE_ID_RE re-spelled (common/lib/siteSchema.ts imports zod and is
+// server-only), with a length cap because the value goes into a header.
+const SITE_ID_SHAPE = /^[a-z0-9][a-z0-9-]*$/;
+const STORED_MAX_LENGTH = 128;
+
+// What may be stored: ALL_SITES or anything shaped like a site id. Whether the
+// id names a site is decided when it is read (`resolveActiveSite`), so a site
+// deleted since resolves as a missing selection does.
+export function isStorableActiveSite(value: unknown): value is string {
+ return (
+ typeof value === "string" &&
+ (value === ALL_SITES ||
+ (value.length <= STORED_MAX_LENGTH && SITE_ID_SHAPE.test(value)))
+ );
+}
+
export type ResolvedActiveSite = {
// All configured site ids (passed in by the caller).
siteIds: string[];
- // The canonical param value for the resolved selection: a siteId or ALL_SITES.
+ // The canonical value for the resolved selection: a siteId or ALL_SITES.
value: string;
// True when the selection spans every site (full pool).
isAll: boolean;
@@ -27,7 +75,7 @@ export type ResolvedActiveSite = {
siteId: string | null;
};
-// Resolve the raw `searchParams.site` value against the configured site ids.
+// Resolve one raw value (a param, a cookie) against the configured site ids.
// - a valid siteId -> that site
// - ALL_SITES -> all sites (full pool)
// - missing/invalid (default) -> the lone site if exactly one, else all sites
@@ -54,10 +102,23 @@ export function resolveActiveSite(
return all;
}
+// THE PRECEDENCE, in one place: the first candidate that names a configured
+// site (or ALL_SITES) wins; when none does, the default above. The server
+// passes [?site=, cookie]; the picker passes [its pending choice, the path's
+// site, ?site=, the stored value].
+export function resolveActiveSiteFrom(
+ candidates: ReadonlyArray<string | null | undefined>,
+ siteIds: string[],
+): ResolvedActiveSite {
+ const chosen = candidates.find(
+ (c) => c === ALL_SITES || (!!c && siteIds.includes(c)),
+ );
+ return resolveActiveSite(chosen, siteIds);
+}
+
// The routes under /sites where the path names the site. `new` is /sites/new,
// a static route that beats [siteId] and is not a site. The id pattern is
-// SITE_ID_RE re-spelled: common/lib/site.ts imports node:fs and this module
-// must stay importable from the client picker.
+// SITE_ID_RE re-spelled, as above.
const SITE_PATH_RE = /^\/sites\/([a-z0-9][a-z0-9-]*)(?:\/([a-z-]+))?\/?$/;
export type SitePath = { siteId: string; segment: string | null };
@@ -69,3 +130,12 @@ export function siteIdFromPathname(pathname: string): SitePath | null {
if (!m || m[1] === "new") return null;
return { siteId: m[1], segment: m[2] ?? null };
}
+
+// The same URL with its `site` param dropped (every other param kept, in
+// order): where the picker goes when a choice replaces a `?site=` link's scope.
+export function withoutSiteParam(pathname: string, search: string): string {
+ const params = new URLSearchParams(search);
+ params.delete("site");
+ const q = params.toString();
+ return q ? `${pathname}?${q}` : pathname;
+}
diff --git a/editor/app/lib/activeSiteActions.ts b/editor/app/lib/activeSiteActions.ts
@@ -0,0 +1,31 @@
+"use server";
+
+import { cookies } from "next/headers";
+import {
+ ACTIVE_SITE_COOKIE_MAX_AGE,
+ isStorableActiveSite,
+} from "./activeSite";
+import { activeSiteCookieForRequest } from "./activeSiteServer";
+
+// THE ONE WRITER of the active-site cookie (see ./activeSite.ts). Called by the
+// picker's onChange, by SiteScopeProvider when a site's own page is visited, and
+// once by its localStorage migration. Setting a cookie in a server action makes
+// Next re-render the current page and its layouts with the new value, so the
+// root layout hands the provider the new selection and Dashboard or Channels
+// re-scope, with no refresh of our own.
+//
+// The value is only SHAPE-checked (a site id or "__all__"): whether it names a
+// configured site is decided at every read, so a site deleted later resolves as
+// no selection. Returns false, writing nothing, for anything else.
+export async function setActiveSiteAction(value: string): Promise<boolean> {
+ if (!isStorableActiveSite(value)) return false;
+ const name = await activeSiteCookieForRequest();
+ (await cookies()).set(name, value, {
+ path: "/",
+ sameSite: "lax",
+ maxAge: ACTIVE_SITE_COOKIE_MAX_AGE,
+ // Only the server reads it: the page gets it through the layout.
+ httpOnly: true,
+ });
+ return true;
+}
diff --git a/editor/app/lib/activeSiteServer.ts b/editor/app/lib/activeSiteServer.ts
@@ -0,0 +1,35 @@
+import "server-only";
+import { cookies, headers } from "next/headers";
+import { getPaths } from "yt-dlp-transcript-common/lib/paths";
+import { listSiteIds } from "yt-dlp-transcript-common/lib/site";
+import {
+ activeSiteCookieName,
+ resolveActiveSiteFrom,
+ type ResolvedActiveSite,
+} from "./activeSite";
+
+// THE ONE READ of the active site on the server (see ./activeSite.ts). The root
+// layout calls it with no param (a layout has no searchParams) and hands the
+// result to SiteScopeProvider; Dashboard and Channels call it with their
+// `searchParams.site`, which wins for that request when it names a site or
+// "__all__". Nothing is written here: a server component cannot set a cookie,
+// and only setActiveSiteAction does.
+export type ActiveSiteRead = ResolvedActiveSite & {
+ // The cookie's raw value, or null when the request carried none — which is
+ // what tells the provider a visitor may still have the old localStorage key.
+ stored: string | null;
+};
+
+// The request's own cookie name (per port; see activeSiteCookieName).
+export async function activeSiteCookieForRequest(): Promise<string> {
+ return activeSiteCookieName((await headers()).get("host"));
+}
+
+export async function readActiveSite(
+ param?: string | null,
+ siteIds: string[] = listSiteIds(getPaths()),
+): Promise<ActiveSiteRead> {
+ const name = await activeSiteCookieForRequest();
+ const stored = (await cookies()).get(name)?.value ?? null;
+ return { ...resolveActiveSiteFrom([param, stored], siteIds), stored };
+}
diff --git a/editor/app/page.tsx b/editor/app/page.tsx
@@ -13,7 +13,7 @@ import {
getActionableSummary,
widgetActionableRows,
} from "./lib/actionable/loadActionable";
-import { resolveActiveSite } from "./lib/activeSite";
+import { readActiveSite } from "./lib/activeSiteServer";
import { buildActiveJobsPayload } from "./jobs/active/buildActiveJobs";
import { buildWorkersPayload } from "./workers/buildWorkers";
import { buildWidgetSyncPayload } from "yt-dlp-transcript-common/views/widgetSync";
@@ -52,8 +52,9 @@ export default async function Dashboard({
searchParams: Promise<{ site?: string }>;
}) {
const paths = getPaths();
+ // The active site: a `?site=` link's for this request, else the cookie's.
const { site } = await searchParams;
- const active = resolveActiveSite(site, listSiteIds(paths));
+ const active = await readActiveSite(site, listSiteIds(paths));
const summary = await getActionableSummary(paths);
// Scope the channel-derived data to the active site's membership. Under "all
diff --git a/editor/app/saved-videos/page.tsx b/editor/app/saved-videos/page.tsx
@@ -22,7 +22,11 @@ export default async function SavedVideosPage() {
const paths = getPaths();
const settings = getSettings();
const backup = settings.savedVideoBackup;
- const entries = await listSavedVideos({ paths });
+ // Channels whose drive is not answering are not read (one pointer read per
+ // video dir, each of which would wait on it); the page names them.
+ const notAnswering: string[] = [];
+ const entries = await listSavedVideos({ paths, notAnswering });
+ notAnswering.sort();
const scheduler = await readSchedulerState(paths);
const byChannel = new Map<string, ChannelSummary>();
@@ -96,6 +100,21 @@ export default async function SavedVideosPage() {
<section aria-label="per-channel saved videos" className="flex flex-col gap-2">
<h2 className="text-base font-semibold">By channel</h2>
+ {notAnswering.length > 0 && (
+ <p
+ role="status"
+ aria-label="saved videos not read"
+ className="rounded border border-destructive/50 bg-destructive/5 px-3 py-2 text-sm text-destructive"
+ >
+ Not read, because the drive their media is on is not answering:{" "}
+ {notAnswering.join(", ")}. Their saved videos are not in the counts
+ above until it answers again (see{" "}
+ <Link href="/storage" className="underline">
+ Storage
+ </Link>
+ ).
+ </p>
+ )}
{channels.length === 0 ? (
<p className="text-sm text-muted-foreground">
No saved videos yet. Set a channel's keep-latest window, then run
diff --git a/editor/app/settings/actions.ts b/editor/app/settings/actions.ts
@@ -3,6 +3,7 @@
import { revalidatePath } from "next/cache";
import {
AUTO_REFRESH_INTERVAL_MAX_SECONDS,
+ getSettings,
AUTO_REFRESH_INTERVAL_MIN_SECONDS,
DEFAULT_REPORT_DEBOUNCE_PRESET,
defaultBuildPipeline,
@@ -10,7 +11,6 @@ import {
MIN_FREE_DISK_GB_MAX,
RESUME_MARGIN_GB_DEFAULT,
RESUME_MARGIN_GB_MAX,
- normalizeSocialSvg,
parseSocialLinks,
PARALLEL_TRANSCRIPTIONS_DEFAULT,
SLEEP_BETWEEN_DOWNLOADS_MAX_SECONDS,
@@ -20,6 +20,7 @@ import {
type SocialLink,
} from "yt-dlp-transcript-common/lib/settings";
import { saveSettings } from "./saveSettings";
+import { socialLinksForSave } from "yt-dlp-transcript-common/lib/socialLinks";
import {
DEFAULT_COOKIE_MODE,
isCookieMode,
@@ -166,17 +167,16 @@ export async function saveSettingsAction(
"Each social link needs a label, URL (http(s)://, mailto:, or /), and SVG.",
};
}
- const socialLinks: SocialLink[] = [];
- for (const link of socialParsed) {
- const svg = normalizeSocialSvg(link.svg);
- if (svg === null) {
- return {
- ok: false,
- error: `Social link "${link.label}" has an invalid SVG.`,
- };
- }
- socialLinks.push({ ...link, svg });
+ // A link whose SVG is unchanged from settings.json is kept as it is; a new
+ // or edited one is checked (lib/socialLinks.ts socialLinksForSave).
+ const checked = socialLinksForSave(socialParsed, getSettings().socialLinks);
+ if ("refused" in checked) {
+ return {
+ ok: false,
+ error: `Social link "${checked.refused.label}" has an invalid SVG: ${checked.refused.problem}.`,
+ };
}
+ const socialLinks: SocialLink[] = checked.links;
// Build pipeline. Values are clamped/coerced by sanitizeBuildPipeline inside
// the settings schema on save, so we only read the form here (NaN/blank → default).
diff --git a/editor/app/settings/components/SettingsForm.tsx b/editor/app/settings/components/SettingsForm.tsx
@@ -45,7 +45,7 @@ export function SettingsForm({ initial }: Props) {
name="homepageUrl"
defaultValue={initial.homepageUrl}
type="url"
- hint="Absolute URL of the family hub (e.g. https://archilyzer-hub.pages.dev). Every export site links back to it. Leave blank for no hub link."
+ hint="Absolute URL of the family hub (e.g. https://archilyzer-hub.pages.dev). Each site that names no hub of its own publishes it in its /site.json and /corpus.json, so the hub can tell its member sites; no page links to it. Leave blank to publish none."
/>
<Field
label="Max transcript page bytes"
diff --git a/editor/app/settings/saveSettings.test.ts b/editor/app/settings/saveSettings.test.ts
@@ -62,6 +62,28 @@ test("an object nested inside a block replaces, it is not merged", () => {
assert.deepEqual(out.channelPriority.channels, { other: { tier: "paused" } });
});
+// A LOCATION WRITE KEEPS THE DRIVE-HEALTH TIMINGS. The /storage location
+// actions patch `storage` with `{ locations, defaultLocationId }` only; the
+// one-level merge is what keeps `storage.health` (and `savedVideosLocationId`,
+// which an add once erased before slice 4a).
+test("a storage patch of the locations keeps storage.health", () => {
+ const base = defaultSiteSettings();
+ base.storage = {
+ ...base.storage,
+ savedVideosLocationId: "cold",
+ health: { budgetMs: 4_000, inFlightPerLocation: 2 },
+ };
+ const out = mergeSettingsPatch(base, {
+ storage: {
+ locations: [{ id: "cold", label: "Cold", root: "/mnt/cold", autoRepoint: false }],
+ defaultLocationId: "cold",
+ },
+ });
+ assert.deepEqual(out.storage.health, { budgetMs: 4_000, inFlightPerLocation: 2 });
+ assert.equal(out.storage.savedVideosLocationId, "cold");
+ assert.equal(out.storage.locations.length, 1);
+});
+
test("saveSettings writes the merged result and touches nothing else", async () => {
writeFileSync(
process.env.SETTINGS_FILE!,
@@ -76,3 +98,39 @@ test("saveSettings writes the merged result and touches nothing else", async ()
const onDisk = JSON.parse(readFileSync(process.env.SETTINGS_FILE!, "utf8"));
assert.equal(onDisk.adminTitle, "Kept");
});
+
+// A STORED ICON THE CHECKER NOW REFUSES does not block an unrelated save: an
+// older build stored it, and a lane pause or a title must still save. The link
+// is written back byte-identical; only a NEW or EDITED icon is checked.
+test("an icon stored by an older build survives unrelated saves; editing it is checked", async () => {
+ const old = `<svg aria-hidden="true" fill="currentColor" viewBox="0 0 8 8" inkscape:version="1.0"><defs><style>.a{fill:#f00}</style></defs><metadata/><path class="a" d="M0 0"/></svg>`;
+ const good = `<svg viewBox="0 0 8 8"><path d="M0 0"/></svg>`;
+ const stored = { label: "Old", url: "https://old.example", svg: old };
+ writeFileSync(process.env.SETTINGS_FILE!, JSON.stringify({ socialLinks: [stored] }));
+ const linkOnDisk = () =>
+ (JSON.parse(readFileSync(process.env.SETTINGS_FILE!, "utf8")) as { socialLinks: unknown[] }).socialLinks;
+
+ await saveSettings({ adminTitle: "Renamed" });
+ await saveSettings({ minFreeDiskGB: 2 });
+ const s = getSettings();
+ // A lane pause, the way the editor's pause control writes it.
+ const lane = Object.keys(s.autoQueue)[0] as keyof typeof s.autoQueue;
+ await saveSettings({ autoQueue: { ...s.autoQueue, [lane]: { ...s.autoQueue[lane], held: true } } });
+ await saveSettings({
+ channelPriority: { ...s.channelPriority, channels: { some: { tier: "paused" } } },
+ });
+ assert.deepEqual(linkOnDisk(), [stored], "byte-identical after four unrelated saves");
+
+ // Edited to something still refused: the save is refused, with the reason.
+ await assert.rejects(
+ saveSettings({ socialLinks: [{ ...stored, svg: `<svg viewBox="0 0 8 8"><style>*{}</style></svg>` }] }),
+ /Social link "Old" has an invalid SVG: it has an element an icon has no use for \(style\)/,
+ );
+ assert.deepEqual(linkOnDisk(), [stored]);
+
+ // Edited to a good icon: saved, normalized.
+ await saveSettings({ socialLinks: [{ ...stored, svg: good }] });
+ assert.deepEqual(linkOnDisk(), [
+ { ...stored, svg: `<svg aria-hidden="true" fill="currentColor" viewBox="0 0 8 8"><path d="M0 0"/></svg>` },
+ ]);
+});
diff --git a/editor/app/sites/actions.ts b/editor/app/sites/actions.ts
@@ -11,6 +11,7 @@ import {
wordmarkLeadFor,
} from "yt-dlp-transcript-common/lib/brand";
import {
+ getSite,
writeSite,
deleteSite,
isValidSiteId,
@@ -20,7 +21,6 @@ import {
type Site,
} from "yt-dlp-transcript-common/lib/site";
import {
- normalizeSocialSvg,
parseSocialLinks,
type SocialLink,
} from "yt-dlp-transcript-common/lib/settings";
@@ -29,6 +29,7 @@ import {
resolveDefaultGroupId,
} from "yt-dlp-transcript-common/lib/channelGroups";
import { migrateToSites } from "yt-dlp-transcript-common/controller/migrateToSites";
+import { socialLinksForSave } from "yt-dlp-transcript-common/lib/socialLinks";
export type SaveResult = { ok: true; siteId: string } | { ok: false; error: string };
@@ -91,6 +92,9 @@ export async function saveSiteAction(
};
}
const siteUrl = parseSiteUrl(siteUrlRaw);
+ // Listed on the homepage and hub by default: the same opt-out idiom as
+ // archives below (an unchecked box sends no key → persisted as false).
+ const listed = formData.get("listed") === "on";
// Hub parent (per-site override of the family default) + PWA opt-in.
const hubUrlRaw = String(formData.get("hubUrl") ?? "").trim();
@@ -176,17 +180,22 @@ export async function saveSiteAction(
"Each social link needs a label, URL (http(s)://, mailto:, or /), and SVG.",
};
}
- socialLinks = [];
- for (const link of socialParsed) {
- const svg = normalizeSocialSvg(link.svg);
- if (svg === null) {
- return {
- ok: false,
- error: `Social link "${link.label}" has an invalid SVG.`,
- };
- }
- socialLinks.push({ ...link, svg });
+ // A link whose SVG is unchanged from the site's file is kept as it is; a
+ // new or edited one is checked (lib/socialLinks.ts socialLinksForSave).
+ let stored: SocialLink[] = [];
+ try {
+ stored = getSite(siteId, getPaths()).socialLinks ?? [];
+ } catch {
+ stored = [];
+ }
+ const checked = socialLinksForSave(socialParsed, stored);
+ if ("refused" in checked) {
+ return {
+ ok: false,
+ error: `Social link "${checked.refused.label}" has an invalid SVG: ${checked.refused.problem}.`,
+ };
}
+ socialLinks = checked.links;
}
let channelsInput: unknown;
@@ -211,6 +220,8 @@ export async function saveSiteAction(
...(accent ? { accent } : {}),
...(cloudflareProject ? { cloudflareProject } : {}),
...(siteUrl ? { siteUrl } : {}),
+ // The Site is rebuilt from the form: a key missing here is dropped on save.
+ ...(listed ? {} : { listed: false }),
...(hubUrl ? { hubUrl } : {}),
...(pwa ? { pwa: true } : {}),
...(archives ? {} : { archives: false }),
diff --git a/editor/app/sites/components/HomepageBuildButtons.tsx b/editor/app/sites/components/HomepageBuildButtons.tsx
@@ -183,6 +183,13 @@ export function HomepageBuildButtons({ project, builtAt }: Props) {
The homepage reads the search index as it stands: run Build index
(under Pool jobs, below) first when its numbers should move.
</p>
+ <p className="text-xs text-muted-foreground">
+ Build homepage also publishes the source mirror (<code>archilyzer
+ source publish</code>) and refuses if a denied string survives the
+ scrub; a refusal removes the last published source, from
+ homepage/out too. Deploy homepage refuses a build whose source was
+ not audited under today’s rules.
+ </p>
{lane && (
<JobLane
key={lane.key}
diff --git a/editor/app/sites/components/SiteForm.tsx b/editor/app/sites/components/SiteForm.tsx
@@ -253,9 +253,9 @@ export function SiteForm({ initial, channels, allSites, isNew }: Props) {
<fieldset className="flex flex-col gap-2 text-sm">
<legend className="font-medium">Brand accent</legend>
<p className="text-xs text-muted-foreground">
- This site's default accent (its icon and first paint); a reader
- can pick another. A custom colour is darkened or lightened on each
- base (light, sepia, dark) until it reads at 4.5:1.
+ This site's accent (its icon and every page). A custom colour
+ is darkened or lightened on each base (light, dark) until it reads
+ at 4.5:1.
</p>
<div className="flex flex-wrap gap-x-4 gap-y-2">
{ACCENT_IDS.map((id) => (
@@ -327,11 +327,27 @@ export function SiteForm({ initial, channels, allSites, isNew }: Props) {
defaultValue={initial.siteUrl ?? ""}
hint="Absolute URL this site is served at (e.g. https://jeralyzer.com). Used so other sites can link to it in their footer. Leave blank to omit this site from cross-site lists."
/>
+ <label className="flex items-center gap-2 text-sm">
+ <input
+ type="checkbox"
+ name="listed"
+ defaultChecked={initial.listed !== false}
+ className="accent-brand"
+ />
+ List on the Archilyzer homepage and hub
+ </label>
+ <p className="-mt-2 text-xs text-muted-foreground">
+ On by default. Turn off to leave this site out of the homepage (its
+ card, chart and stats), the hub (its members, search, corpus.json and
+ llms.txt) and every other site's footer, and to count its own
+ channels in none of the published totals. The site still builds and
+ deploys as before, at its own URL.
+ </p>
<Field
label="Hub URL"
name="hubUrl"
defaultValue={initial.hubUrl ?? ""}
- hint="The hub this site belongs under (e.g. https://archilyzer-hub.pages.dev). Shows a 'Hub' backlink and lets the hub recognize this site as a member. Leave blank to inherit the family default from Settings."
+ hint="The hub this site belongs under (e.g. https://archilyzer-hub.pages.dev), published in this site's /site.json and /corpus.json so the hub can tell it is a member; the header does not link to it. Leave blank to inherit the family default from Settings."
/>
<label className="flex items-center gap-2 text-sm">
<input
diff --git a/editor/app/storage/actions.ts b/editor/app/storage/actions.ts
@@ -33,8 +33,22 @@ import { enqueueRepointJob } from "./lib/repointJob";
import { enqueueSavedVideosRelocation } from "./lib/savedVideosJob";
import { enqueueEvictClipWindows } from "./lib/evictClipsJob";
import { savedVideosStoreBusyReason } from "./lib/storeBusy";
+import { refreshLocationHealth } from "yt-dlp-transcript-common/controller/storageWatch";
+import { forgetChannelMedia } from "yt-dlp-transcript-common/lib/channelMedia";
+import {
+ applyHealthTimings,
+ healthTimings,
+ locationHealth,
+ notAnsweringText,
+} from "yt-dlp-transcript-common/lib/storageHealth";
+import {
+ clearRuleText,
+ secondsText,
+} from "yt-dlp-transcript-common/lib/storageHealthTimings";
+import { parseHealthTimingsForm } from "./lib/healthTimingsForm";
-// THE SIX THINGS AN OPERATOR MAY DO TO A STORAGE LOCATION.
+// THE SIX THINGS AN OPERATOR MAY DO TO A STORAGE LOCATION (and, at the end,
+// the drive-health timings every location is judged by).
//
// Five of them are small settings writes or one subprocess and run INLINE:
// their whole output is a sentence, and a queued job with a log would be a
@@ -244,6 +258,27 @@ export async function refreshStorageLocationAction(
const settings = getSettings();
const location = settings.storage.locations.find((l) => l.id === id);
if (!location) return { ok: false, error: `There is no storage location "${id}".` };
+ // IS IT ANSWERING, asked first and without touching the drive (the block
+ // device's counters in /sys, or a child `stat` raced against
+ // `storage.health.probeTimeoutMs` where no device can be named): the probe
+ // below runs in-process, and on a stalled drive it is not asked at all. The
+ // counters give no answer within 10 s (by default) of the pass's last
+ // sample. The channels' remembered answers go too — the operator has just
+ // done something about the drive.
+ await refreshLocationHealth(location);
+ forgetChannelMedia();
+ const health = locationHealth(location.id);
+ if (health?.state === "stalled") {
+ revalidateStorage();
+ const timings = healthTimings();
+ return {
+ ok: true,
+ note:
+ `${location.label}: ${notAnsweringText(health)} — ${health.cause ?? "its root did not answer"}. ` +
+ `Pages skip this drive until it answers ${clearRuleText(timings.clearAfterCleanPasses)} ` +
+ `(checked every ${secondsText(timings.passIntervalMs)}).`,
+ };
+ }
const probe = await probeLocationMemo(location, paths, { refresh: true });
const wrote = await recordProbedIdentity({ locationId: id, probe });
@@ -451,3 +486,45 @@ export async function evictClipWindowsAction(opts: {
...(opts.dryRun ? { dryRun: true } : {}),
});
}
+
+// ---------------------------------------------------------------------------
+// The drive-health timings
+// ---------------------------------------------------------------------------
+
+export type HealthTimingsResult = { ok: true; note: string } | { ok: false; error: string };
+
+// SAVE `settings.storage.health` FROM THE /storage FORM (HealthTimingForm).
+//
+// Parsed by `parseHealthTimingsForm`: an empty field is the default and is not
+// written, and a value out of range is REFUSED with a sentence (the schema
+// would clamp it; a save that stored another number than the one typed would
+// read as a form that did not listen). Then written through the one settings
+// writer, as a patch of the storage block, and APPLIED AT ONCE to this
+// process's health state (`applyHealthTimings`, on globalThis): the next read
+// runs on the new budget and cap, the next answer on the new clear count, the
+// next pass on the new probe timeout, and a new pass interval re-arms the
+// pass's timer now. The pass would apply them anyway, at its next run.
+export async function saveHealthTimingsAction(
+ _prev: HealthTimingsResult | undefined,
+ formData: FormData,
+): Promise<HealthTimingsResult> {
+ const parsed = parseHealthTimingsForm(formData);
+ if (!parsed.ok) return parsed;
+ const settings = getSettings();
+ try {
+ await saveSettings({ storage: { ...settings.storage, health: parsed.health } });
+ } catch (e) {
+ return { ok: false, error: (e as Error).message };
+ }
+ const t = applyHealthTimings(getSettings().storage.health);
+ revalidatePath("/storage");
+ return {
+ ok: true,
+ note:
+ `Saved. A read may take ${secondsText(t.budgetMs)}; drives are checked every ` +
+ `${secondsText(t.passIntervalMs)} (a check waits up to ${secondsText(t.probeTimeoutMs)}); ` +
+ `a drive marked not answering is used again after ` +
+ `${t.clearAfterCleanPasses === 1 ? "one clean check" : `${t.clearAfterCleanPasses} clean checks in a row`}; ` +
+ `${t.inFlightPerLocation} read(s) at once per drive.`,
+ };
+}
diff --git a/editor/app/storage/buildStorage.ts b/editor/app/storage/buildStorage.ts
@@ -3,6 +3,14 @@ import { getSettings } from "yt-dlp-transcript-common/lib/settings";
import { getRegistry } from "yt-dlp-transcript-common/jobs/registry";
import { getFreeBytes } from "yt-dlp-transcript-common/lib/diskSpace";
import { udisksctlAvailable } from "yt-dlp-transcript-common/lib/storageVolumes";
+import {
+ healthTimings,
+ isDriveNotAnswering,
+ notAnsweringText,
+ onDrive,
+ stalledLocation,
+} from "yt-dlp-transcript-common/lib/storageHealth";
+import { clearRuleText } from "yt-dlp-transcript-common/lib/storageHealthTimings";
import { listChannelBriefs } from "yt-dlp-transcript-common/controller/channels";
import {
channelsOnLocation,
@@ -81,11 +89,41 @@ export async function buildStorage(): Promise<StorageRowsPayload> {
// situation the operator opened the page to understand. The store's SIZE is
// the only thing cached; its location, status and marker are read fresh every
// render, because those are the safety facts.
+ // THE DRIVES THAT ARE NOT ANSWERING, in words. From memory: the health pass
+ // (the block device's counters, every `storage.health.passIntervalMs`) or the
+ // watchdog on a page's read is what found it (lib/storageHealth.ts).
+ const now = Date.now();
+ const clears = clearRuleText(healthTimings().clearAfterCleanPasses);
+ const notAnswering: Record<string, string> = {};
+ for (const loc of locations) {
+ const stall = stalledLocation(loc);
+ if (stall) {
+ const watched =
+ stall.detector === "counters"
+ ? " Watched through its disk's request counters."
+ : stall.detector === "stat"
+ ? " Watched with a stat of its root (no disk could be named here)."
+ : "";
+ notAnswering[loc.id] =
+ `${notAnsweringText(stall, now)} — ${stall.cause ?? "its root did not answer"}. ` +
+ `Pages and polls skip this drive until it answers ${clears}.${watched}`;
+ }
+ }
const store = await inspectSavedVideosStore(paths, settings);
- const storeMeasured =
- store.status === "unreachable" || store.status === "in-transition"
- ? { bytes: 0, files: 0 }
- : await measureTreeCached(paths.savedVideosDir);
+ // A store on a location walks that location's drive: through the watchdog,
+ // so a drive that stops answering mid-walk leaves the size at 0 (the store's
+ // status line says why on the next render) instead of holding the page.
+ const storeLocation = locations.find((l) => l.id === store.locationId);
+ let storeMeasured = { bytes: 0, files: 0 };
+ if (store.status !== "unreachable" && store.status !== "in-transition") {
+ try {
+ storeMeasured = await (storeLocation
+ ? onDrive(storeLocation, () => measureTreeCached(paths.savedVideosDir))
+ : measureTreeCached(paths.savedVideosDir));
+ } catch (err) {
+ if (!isDriveNotAnswering(err)) throw err;
+ }
+ }
return buildStorageRows({
locations,
savedVideos: {
@@ -106,9 +144,10 @@ export async function buildStorage(): Promise<StorageRowsPayload> {
},
defaultLocationId: settings.storage.defaultLocationId,
probes,
+ notAnswering,
rollups,
registry: getRegistry(),
udisksctlAvailable: udisksctl,
- now: Date.now(),
+ now,
});
}
diff --git a/editor/app/storage/components/HealthTimingForm.tsx b/editor/app/storage/components/HealthTimingForm.tsx
@@ -0,0 +1,119 @@
+"use client";
+
+import { useActionState } from "react";
+import {
+ HEALTH_TIMING_BOUNDS,
+ HEALTH_TIMING_DEFAULTS,
+ HEALTH_TIMING_HINTS,
+ type StorageHealthSettings,
+} from "yt-dlp-transcript-common/lib/storageHealthTimings";
+import { HEALTH_TIMING_FIELDS } from "../lib/healthTimingsForm";
+import { saveHealthTimingsAction, type HealthTimingsResult } from "../actions";
+
+// THE DRIVE HEALTH TIMING — `settings.storage.health`, the five numbers the
+// editor decides "this drive is mounted and not answering" by
+// (lib/storageHealth.ts). The ruling (release 15, slice DT): when a stall is
+// misjudged under heavy external-disk churn, the operator tunes these rather
+// than the code.
+//
+// COLLAPSED, AND LAST. The defaults suit a healthy disk and most operators will
+// never open it; the locations are what the page is for.
+//
+// A FIELD LEFT EMPTY IS THE DEFAULT, which is its placeholder. A field holds a
+// value only where one was saved, so clearing it goes back to the default (and
+// nothing is written for it). The server refuses a value out of range with a
+// sentence, rather than storing another number than the one typed.
+//
+// TEXT INPUTS, NOT `type="number"`: a number input's own validation blocks the
+// submit on "3.5" with a browser bubble instead of the sentence the action
+// returns, and the action is the one validator.
+//
+// VALUES FROM THE TIMINGS MODULE ONLY (pure): this is a "use client" file.
+//
+// Accessible names: "drive health timing" (the block), the five fields'
+// (lib/healthTimingsForm.ts), "save timing", "timing saved", "timing error" —
+// none contains another (Playwright's getByLabel matches substrings).
+
+export function HealthTimingForm({ stored }: { stored: StorageHealthSettings }) {
+ const [state, formAction, pending] = useActionState<
+ HealthTimingsResult | undefined,
+ FormData
+ >(saveHealthTimingsAction, undefined);
+ const tuned = HEALTH_TIMING_FIELDS.filter((f) => stored[f.key] !== undefined).length;
+
+ return (
+ <details
+ aria-label="drive health timing"
+ className="rounded-xl border border-border bg-card px-4 py-3"
+ >
+ <summary className="cursor-pointer text-base font-semibold">
+ Drive health timing
+ <span className="ml-2 text-sm font-normal text-muted-foreground">
+ {tuned === 0 ? "defaults" : `${tuned} changed from the default`}
+ </span>
+ </summary>
+ <div className="mt-3 flex flex-col gap-3">
+ <p className="text-sm text-muted-foreground max-w-3xl">
+ How the editor decides that a drive is mounted but not answering, and
+ stops reading it until it answers again. The defaults suit a disk in
+ good health; change them only when a busy drive is being called not
+ answering (or a stalled one is not). An empty field is its default.
+ </p>
+ <form action={formAction} className="flex flex-col gap-3">
+ <div className="grid gap-3 sm:grid-cols-2 lg:grid-cols-3">
+ {HEALTH_TIMING_FIELDS.map((f) => {
+ const { min, max } = HEALTH_TIMING_BOUNDS[f.key];
+ const unit = f.unit ? ` ${f.unit}` : "";
+ return (
+ <label key={f.key} className="flex flex-col gap-1 text-sm">
+ <span className="font-medium">
+ {f.label}
+ {f.unit && <span className="font-normal text-muted-foreground"> ({f.unit})</span>}
+ </span>
+ <input
+ type="text"
+ inputMode="numeric"
+ name={f.key}
+ aria-label={f.ariaLabel}
+ defaultValue={stored[f.key] === undefined ? "" : String(stored[f.key])}
+ placeholder={String(HEALTH_TIMING_DEFAULTS[f.key])}
+ className="rounded border border-border bg-background px-2 py-1 text-sm font-mono tabular-nums"
+ />
+ <span className="text-xs text-muted-foreground">
+ {HEALTH_TIMING_HINTS[f.key]} Default {HEALTH_TIMING_DEFAULTS[f.key]}
+ {unit}; {min}–{max}
+ {unit}.
+ </span>
+ </label>
+ );
+ })}
+ </div>
+ <div className="flex flex-wrap items-center gap-3">
+ <button
+ type="submit"
+ disabled={pending}
+ aria-label="save timing"
+ className="px-3 py-1.5 rounded-md border border-border text-sm font-medium disabled:opacity-50"
+ >
+ {pending ? "Saving…" : "Save timing"}
+ </button>
+ {state?.ok === true && (
+ <span role="status" aria-label="timing saved" className="text-sm">
+ {state.note}
+ </span>
+ )}
+ {state?.ok === false && (
+ <span
+ role="alert"
+ aria-label="timing error"
+ className="text-sm text-destructive"
+ >
+ {state.error}
+ </span>
+ )}
+ </div>
+ </form>
+ </div>
+ </details>
+ );
+}
diff --git a/editor/app/storage/components/StorageLocationsTable.tsx b/editor/app/storage/components/StorageLocationsTable.tsx
@@ -254,6 +254,15 @@ function LocationCard({
<dd aria-label="location last probe">{formatAge(row.lastProbeAgeMs)}</dd>
</dl>
+ {row.notAnswering && (
+ <p
+ role="status"
+ aria-label="location not answering"
+ className="text-xs rounded border border-destructive/50 bg-destructive/5 px-3 py-2 text-destructive"
+ >
+ {row.notAnswering}
+ </p>
+ )}
{row.warning && (
<p
role="status"
diff --git a/editor/app/storage/lib/healthTimingsForm.test.ts b/editor/app/storage/lib/healthTimingsForm.test.ts
@@ -0,0 +1,95 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import {
+ HEALTH_TIMING_DEFAULTS,
+ HEALTH_TIMING_KEYS,
+} from "yt-dlp-transcript-common/lib/storageHealthTimings";
+import { HEALTH_TIMING_FIELDS, parseHealthTimingsForm } from "./healthTimingsForm";
+
+// Run with: pnpm -C editor exec tsx --test "app/**/*.test.ts"
+//
+// THE /storage DRIVE HEALTH TIMING FORM'S PARSE: an empty field is the default
+// and is not written; a value equal to its default is not written either; a
+// whole number in range is; anything else is refused with a sentence naming
+// the field, and nothing is saved.
+
+function form(fields: Record<string, string>): FormData {
+ const f = new FormData();
+ for (const [k, v] of Object.entries(fields)) f.set(k, v);
+ return f;
+}
+
+test("every timing has one field, in the settings' order", () => {
+ assert.deepEqual(
+ HEALTH_TIMING_FIELDS.map((f) => f.key),
+ [...HEALTH_TIMING_KEYS],
+ );
+ const names = HEALTH_TIMING_FIELDS.map((f) => f.ariaLabel);
+ assert.equal(new Set(names).size, names.length);
+ // No accessible name contains another (Playwright's getByLabel is a
+ // substring match).
+ for (const a of names) {
+ for (const b of names) if (a !== b) assert.ok(!a.includes(b), `${a} contains ${b}`);
+ }
+});
+
+test("empty fields are the defaults: nothing is written", () => {
+ assert.deepEqual(parseHealthTimingsForm(form({})), { ok: true, health: {} });
+ assert.deepEqual(
+ parseHealthTimingsForm(form({ budgetMs: "", passIntervalMs: " " })),
+ { ok: true, health: {} },
+ );
+});
+
+test("a value in range is kept, trimmed; one equal to its default is not written", () => {
+ assert.deepEqual(
+ parseHealthTimingsForm(
+ form({
+ budgetMs: " 4000 ",
+ passIntervalMs: String(HEALTH_TIMING_DEFAULTS.passIntervalMs),
+ probeTimeoutMs: "500",
+ clearAfterCleanPasses: "3",
+ inFlightPerLocation: "8",
+ }),
+ ),
+ {
+ ok: true,
+ health: {
+ budgetMs: 4_000,
+ probeTimeoutMs: 500,
+ clearAfterCleanPasses: 3,
+ inFlightPerLocation: 8,
+ },
+ },
+ );
+});
+
+test("out of range is refused with the field, the range and the value — not clamped", () => {
+ assert.deepEqual(parseHealthTimingsForm(form({ budgetMs: "200" })), {
+ ok: false,
+ error: "Read budget must be between 500 and 60000 ms (got 200 ms).",
+ });
+ assert.deepEqual(parseHealthTimingsForm(form({ inFlightPerLocation: "9" })), {
+ ok: false,
+ error: "Reads at once per drive must be between 1 and 8 (got 9).",
+ });
+ assert.deepEqual(parseHealthTimingsForm(form({ clearAfterCleanPasses: "0" })), {
+ ok: false,
+ error: "Clean checks to clear must be between 1 and 10 (got 0).",
+ });
+});
+
+test("not a whole number is refused", () => {
+ for (const bad of ["3.5", "-1", "3e3", "abc", "0x10"]) {
+ const r = parseHealthTimingsForm(form({ passIntervalMs: bad }));
+ assert.equal(r.ok, false, bad);
+ if (!r.ok) {
+ assert.equal(r.error, `Health check interval must be a whole number of milliseconds (got "${bad}").`);
+ }
+ }
+ const count = parseHealthTimingsForm(form({ clearAfterCleanPasses: "two" }));
+ assert.deepEqual(count, {
+ ok: false,
+ error: 'Clean checks to clear must be a whole number (got "two").',
+ });
+});
diff --git a/editor/app/storage/lib/healthTimingsForm.ts b/editor/app/storage/lib/healthTimingsForm.ts
@@ -0,0 +1,91 @@
+import {
+ HEALTH_TIMING_BOUNDS,
+ HEALTH_TIMING_DEFAULTS,
+ type HealthTimingKey,
+ type StorageHealthSettings,
+} from "yt-dlp-transcript-common/lib/storageHealthTimings";
+
+// THE DRIVE HEALTH TIMING FORM ON /storage: its fields' names and units, and
+// the one parse of what was submitted. Pure, and client-safe (it imports only
+// the pure timings module), so the form draws its labels from here and the
+// server action parses with it.
+//
+// AN EMPTY FIELD IS THE DEFAULT. The inputs show each default as a placeholder
+// and hold a value only where the operator set one, so clearing a field puts
+// that timing back on its default — and nothing is written for it
+// (`sanitizeStorageHealth` keeps only what differs from a default).
+//
+// OUT OF RANGE IS REFUSED, NOT CLAMPED. The schema clamps a hand-edited value on
+// read (a read never throws); a save from this form says which field is out of
+// range and writes nothing, because storing another number than the one typed
+// reads as a form that did not listen.
+
+export type HealthTimingField = {
+ key: HealthTimingKey;
+ // The visible label, and the input's accessible name (a contract once an
+ // e2e spec names it).
+ label: string;
+ ariaLabel: string;
+ // "ms", or "" for a count.
+ unit: string;
+};
+
+export const HEALTH_TIMING_FIELDS: readonly HealthTimingField[] = [
+ { key: "budgetMs", label: "Read budget", ariaLabel: "read budget", unit: "ms" },
+ {
+ key: "passIntervalMs",
+ label: "Health check interval",
+ ariaLabel: "health check interval",
+ unit: "ms",
+ },
+ {
+ key: "probeTimeoutMs",
+ label: "Health check timeout",
+ ariaLabel: "health check timeout",
+ unit: "ms",
+ },
+ {
+ key: "clearAfterCleanPasses",
+ label: "Clean checks to clear",
+ ariaLabel: "clean checks to clear",
+ unit: "",
+ },
+ {
+ key: "inFlightPerLocation",
+ label: "Reads at once per drive",
+ ariaLabel: "reads at once per drive",
+ unit: "",
+ },
+];
+
+export type HealthTimingsParse =
+ | { ok: true; health: StorageHealthSettings }
+ | { ok: false; error: string };
+
+// What the form posted, as the stored block: each non-empty field a whole
+// number in its range, and only the ones that differ from their default.
+export function parseHealthTimingsForm(form: {
+ get(name: string): FormDataEntryValue | null;
+}): HealthTimingsParse {
+ const health: StorageHealthSettings = {};
+ for (const field of HEALTH_TIMING_FIELDS) {
+ const raw = form.get(field.key);
+ const text = typeof raw === "string" ? raw.trim() : "";
+ if (text === "") continue;
+ const unit = field.unit ? ` ${field.unit}` : "";
+ const n = Number(text);
+ if (!/^\d+$/.test(text) || !Number.isSafeInteger(n)) {
+ const what = field.unit === "ms" ? "a whole number of milliseconds" : "a whole number";
+ return { ok: false, error: `${field.label} must be ${what} (got "${text}").` };
+ }
+ const { min, max } = HEALTH_TIMING_BOUNDS[field.key];
+ if (n < min || n > max) {
+ return {
+ ok: false,
+ error: `${field.label} must be between ${min} and ${max}${unit} (got ${n}${unit}).`,
+ };
+ }
+ if (n !== HEALTH_TIMING_DEFAULTS[field.key]) health[field.key] = n;
+ }
+ return { ok: true, health };
+}
diff --git a/editor/app/storage/page.tsx b/editor/app/storage/page.tsx
@@ -1,7 +1,9 @@
import type { Metadata } from "next";
import Link from "next/link";
+import { getSettings } from "yt-dlp-transcript-common/lib/settings";
import { buildStorage } from "./buildStorage";
import { StorageLocationsTable } from "./components/StorageLocationsTable";
+import { HealthTimingForm } from "./components/HealthTimingForm";
// FORCE-DYNAMIC, and not as a formality. Every number on this page comes from a
// probe of the machine taken when the page was asked for — whether a disk is
@@ -33,6 +35,10 @@ export default async function StoragePage() {
</p>
<StorageLocationsTable payload={payload} />
+
+ {/* THE TIMINGS EVERY LOCATION IS JUDGED BY (settings.storage.health),
+ collapsed and last: a page-wide setting, not a fact about a row. */}
+ <HealthTimingForm stored={getSettings().storage.health ?? {}} />
</div>
);
}
diff --git a/editor/e2e/export-search.spec.ts b/editor/e2e/export-search.spec.ts
@@ -588,7 +588,7 @@ test.describe("export footer", () => {
const builtWith = footer.getByRole("link", { name: "Archilyzer" });
await expect(builtWith).toBeVisible();
await expect(builtWith).toHaveAttribute("href", PROJECT_URL);
- // Navigation, not a social link: same tab, matching the header's hub link.
+ // Navigation, not a social link: same tab, as the header's Archilyzer link.
await expect(builtWith).not.toHaveAttribute("target", "_blank");
// The accessible name is exactly the product, with "Built with" outside it.
await expect(footer).toContainText("Built with Archilyzer");
diff --git a/editor/e2e/helpers.ts b/editor/e2e/helpers.ts
@@ -152,6 +152,7 @@ export async function writeSite(
? { cloudflareProject: site.cloudflareProject }
: {}),
...(site.siteUrl ? { siteUrl: site.siteUrl } : {}),
+ ...(site.listed === false ? { listed: false } : {}),
...(site.relatedSites ? { relatedSites: site.relatedSites } : {}),
};
await writeFile(
diff --git a/editor/e2e/settings.spec.ts b/editor/e2e/settings.spec.ts
@@ -42,6 +42,7 @@ test("saves global default social links", async ({ page }) => {
await page
.getByPlaceholder(/<svg viewbox/i)
.fill('<svg viewBox="0 0 24 24"><path d="M0 0h24v24H0z"/></svg>');
+ await page.getByRole("checkbox", { name: "Keep in header on small screens" }).check();
await page.getByRole("button", { name: /save settings/i }).click();
await expect(
@@ -49,10 +50,12 @@ test("saves global default social links", async ({ page }) => {
).toBeVisible();
const saved = await readJson<{
- socialLinks?: { label: string; url: string; svg: string }[];
+ socialLinks?: { label: string; url: string; svg: string; featured?: boolean }[];
}>("test-settings.json");
expect(saved.socialLinks).toHaveLength(1);
expect(saved.socialLinks?.[0].label).toBe("GitHub");
+ // "Keep in header on small screens" is stored as featured: true.
+ expect(saved.socialLinks?.[0].featured).toBe(true);
// SVG was normalized on save (fill="currentColor" injected).
expect(saved.socialLinks?.[0].svg).toContain("currentColor");
});
diff --git a/editor/e2e/site-scope.spec.ts b/editor/e2e/site-scope.spec.ts
@@ -1,5 +1,7 @@
-import { test, expect } from "@playwright/test";
-import { readJson, resetData, writeSite } from "./helpers";
+import { test, expect, type Page } from "@playwright/test";
+import { activeSiteCookieName } from "../app/lib/activeSite";
+import { baseUrl } from "./baseUrl";
+import { readJson, resetData, writeSettings, writeSite } from "./helpers";
type SiteFile = { channels: { slug: string }[] };
@@ -31,15 +33,19 @@ test("selecting a site scopes the channels list and persists", async ({
await page.goto("/channels");
await page.getByLabel("Active site").selectOption("alpha");
- await expect(page).toHaveURL(/site=alpha/);
+ // The choice is a cookie (release 15 SS): the URL is left alone, where it
+ // used to gain ?site=alpha.
+ await expect(page).toHaveURL(/\/channels$/);
await expect(page.getByRole("link", { name: "slow-a" })).toBeVisible();
await expect(page.getByRole("link", { name: "slow-b" })).toHaveCount(0);
+ await expectNoSiteParam(page);
- // localStorage persists the choice: revisiting with no param re-applies it.
+ // The cookie persists the choice: revisiting with no param re-applies it.
await page.goto("/channels");
await expect(page.getByLabel("Active site")).toHaveValue("alpha");
- await expect(page).toHaveURL(/site=alpha/);
+ await expect(page).toHaveURL(/\/channels$/);
await expect(page.getByRole("link", { name: "slow-b" })).toHaveCount(0);
+ await expectNoSiteParam(page);
// Switching to beta flips the scope.
await page.getByLabel("Active site").selectOption("beta");
@@ -70,10 +76,12 @@ test("charts is a site's tab, and the picker follows the path", async ({
await expect(page.getByText(/author the default dashboard/i)).toContainText(
"beta",
);
- // The path wrote localStorage, so the scoped pages follow.
+ // The path wrote the cookie, so the scoped pages follow — with no ?site=
+ // seeded onto the URL any more (release 15 SS).
await page.goto("/channels");
await expect(page.getByLabel("Active site")).toHaveValue("beta");
- await expect(page).toHaveURL(/site=beta/);
+ await expect(page).toHaveURL(/\/channels$/);
+ await expectNoSiteParam(page);
// "All sites" is the family page.
await page.goto("/sites/beta/charts");
await page.getByLabel("Active site").selectOption("__all__");
@@ -138,3 +146,317 @@ test("creating a channel under a site adds it to that site's membership", async
await page.goto("/channels?site=alpha");
await expect(page.getByRole("link", { name: "gamma" })).toBeVisible();
});
+
+// ── No wrong paint (release 15 slice SS) ─────────────────────────────────────
+//
+// The picker used to render "All sites" from the URL on every navigation and
+// then snap to the stored site once an effect had read localStorage and
+// rewritten the URL. Playwright's auto-retrying `toHaveValue` CANNOT see that:
+// it polls until the value is right and passes, flash or no flash. So these
+// cases read the value ONCE, with no retry, right after `domcontentloaded`, and
+// a script installed before any page script records every value the select
+// ever had — at every DOM mutation and every animation frame, i.e. everything
+// that could have been painted — and "__all__" must never be among them.
+
+const PICKER = 'select[aria-label="Active site"]';
+// The cookie this test server's requests use (its name carries the port).
+const COOKIE = activeSiteCookieName(new URL(baseUrl).host);
+// How long a page is watched after it hydrates. There is nothing to wait FOR:
+// the assertion is that nothing happens, and the old snap landed within one
+// server round trip of hydration.
+const SETTLE_MS = 750;
+
+async function recordPickerValues(page: Page) {
+ await page.addInitScript((selector) => {
+ const seen: string[] = [];
+ (window as unknown as { __pickerValues: string[] }).__pickerValues = seen;
+ const read = () => {
+ const el = document.querySelector<HTMLSelectElement>(selector);
+ if (el && seen[seen.length - 1] !== el.value) seen.push(el.value);
+ };
+ new MutationObserver(read).observe(document, {
+ subtree: true,
+ childList: true,
+ attributes: true,
+ });
+ const frame = () => {
+ read();
+ requestAnimationFrame(frame);
+ };
+ requestAnimationFrame(frame);
+ }, PICKER);
+}
+
+function pickerValues(page: Page): Promise<string[]> {
+ return page.evaluate(
+ () =>
+ (window as unknown as { __pickerValues?: string[] }).__pickerValues ?? [],
+ );
+}
+
+// Hydrated = React has attached its props to the select. It is the one React
+// internal this spec peeks at; there is no public signal. Effects follow it.
+async function hydrated(page: Page) {
+ await page.waitForFunction((selector) => {
+ const el = document.querySelector(selector);
+ return !!el && Object.keys(el).some((k) => k.startsWith("__reactProps"));
+ }, PICKER);
+}
+
+// A hydration mismatch is logged by React in development; fail on any.
+function watchHydration(page: Page): string[] {
+ const seen: string[] = [];
+ page.on("console", (msg) => {
+ if (/hydrat/i.test(msg.text())) seen.push(msg.text());
+ });
+ page.on("pageerror", (err) => {
+ if (/hydrat/i.test(err.message)) seen.push(err.message);
+ });
+ return seen;
+}
+
+// No `?site=` on the URL once the page has hydrated and settled, read with no
+// retry. `toHaveURL` alone would pass on its first poll, before a `replace` from
+// an effect (how the old picker seeded the param) could land.
+async function expectNoSiteParam(page: Page) {
+ await hydrated(page);
+ await page.waitForTimeout(SETTLE_MS);
+ expect(new URL(page.url()).searchParams.has("site"), page.url()).toBe(false);
+}
+
+async function storedCookie(page: Page): Promise<string | undefined> {
+ const cookies = await page.context().cookies();
+ return cookies.find((c) => c.name === COOKIE)?.value;
+}
+
+// A full load, read at its first paint.
+async function firstPaint(page: Page, path: string): Promise<string> {
+ await page.goto(path, { waitUntil: "commit" });
+ await page.waitForLoadState("domcontentloaded");
+ return page.locator(PICKER).inputValue();
+}
+
+test("a stored site is the picker's first paint on every page, never All sites", async ({
+ page,
+}) => {
+ test.setTimeout(90_000);
+ await twoSites();
+ const hydration = watchHydration(page);
+ await recordPickerValues(page);
+ const picker = page.getByLabel("Active site");
+
+ // Store alpha the way a person does, and wait for it to land: the re-render
+ // that drops slow-b is the one the cookie write triggers.
+ await page.goto("/channels");
+ await picker.selectOption("alpha");
+ await expect(page.getByRole("link", { name: "slow-b" })).toHaveCount(0);
+ expect(await storedCookie(page)).toBe("alpha");
+
+ // Full loads: the server renders the stored site, and nothing moves it.
+ for (const path of ["/", "/channels", "/jobs", "/settings", "/sites/alpha", "/"]) {
+ expect(await firstPaint(page, path), `${path}: first paint`).toBe("alpha");
+ await hydrated(page);
+ await page.waitForTimeout(SETTLE_MS);
+ expect(await pickerValues(page), `${path}: every value`).toEqual(["alpha"]);
+ expect(new URL(page.url()).searchParams.has("site"), page.url()).toBe(false);
+ }
+
+ // Client-side navigations: the layout persists, the picker re-renders.
+ const nav = page.locator("aside nav");
+ for (const [name, url] of [
+ ["Channels", /\/channels$/],
+ [/^Jobs/, /\/jobs$/],
+ ["Settings", /\/settings$/],
+ ["Sites", /\/sites$/],
+ ["Dashboard", /\/$/],
+ ] as const) {
+ await nav.getByRole("link", { name, exact: typeof name === "string" }).click();
+ await expect(page).toHaveURL(url);
+ expect(await picker.inputValue(), `${String(name)}: first render`).toBe("alpha");
+ }
+ await page.waitForTimeout(SETTLE_MS);
+ expect(await pickerValues(page), "every value across the navigations").toEqual([
+ "alpha",
+ ]);
+ expect(new URL(page.url()).searchParams.has("site"), page.url()).toBe(false);
+
+ // Visiting another site's page records it; Dashboard then paints it at once.
+ expect(await firstPaint(page, "/sites/beta"), "/sites/beta: first paint").toBe(
+ "beta",
+ );
+ await expect.poll(() => storedCookie(page), { timeout: 15_000 }).toBe("beta");
+ expect(await firstPaint(page, "/"), "/ after /sites/beta").toBe("beta");
+ await hydrated(page);
+ await page.waitForTimeout(SETTLE_MS);
+ expect(await pickerValues(page)).toEqual(["beta"]);
+
+ expect(hydration, "hydration warnings").toEqual([]);
+});
+
+test("a ?site= link scopes its own page and is not stored; a choice there drops it", async ({
+ page,
+}) => {
+ test.setTimeout(60_000);
+ await twoSites();
+ const picker = page.getByLabel("Active site");
+ await page.goto("/channels");
+ await picker.selectOption("alpha");
+ await expect(page.getByRole("link", { name: "slow-b" })).toHaveCount(0);
+
+ // The link's scope, on the first paint, for this page only.
+ expect(await firstPaint(page, "/channels?site=beta")).toBe("beta");
+ await expect(page.getByRole("link", { name: "slow-b" })).toBeVisible();
+ await expect(page.getByRole("link", { name: "slow-a" })).toHaveCount(0);
+ expect(await storedCookie(page)).toBe("alpha");
+
+ // Not written: the next plain visit is the stored site's.
+ expect(await firstPaint(page, "/channels")).toBe("alpha");
+ await expect(page.getByRole("link", { name: "slow-b" })).toHaveCount(0);
+
+ // Choosing on a link's page replaces the link's scope: stored, and the param
+ // goes, so the page and the picker agree.
+ await page.goto("/channels?site=beta");
+ await hydrated(page);
+ await picker.selectOption("__all__");
+ await expect(page).toHaveURL(/\/channels$/);
+ await expect(picker).toHaveValue("__all__");
+ await expect(page.getByRole("link", { name: "slow-a" })).toBeVisible();
+ await expect(page.getByRole("link", { name: "slow-b" })).toBeVisible();
+ expect(await storedCookie(page)).toBe("__all__");
+ expect(await firstPaint(page, "/channels")).toBe("__all__");
+});
+
+test("a choice the old picker kept in localStorage moves to the cookie once", async ({
+ page,
+}) => {
+ test.setTimeout(60_000);
+ await twoSites();
+ const picker = page.getByLabel("Active site");
+ const legacy = () => page.evaluate(() => localStorage.getItem("activeSite"));
+
+ // Seed the old key on the editor's origin from a route that mounts no app.
+ await page.goto("/api/pulse");
+ await page.evaluate(() => localStorage.setItem("activeSite", "beta"));
+
+ // The one visit that may flash: the server had no cookie to render from.
+ await page.goto("/channels");
+ await expect(picker).toHaveValue("beta");
+ await expect(page.getByRole("link", { name: "slow-a" })).toHaveCount(0);
+ await expect.poll(() => storedCookie(page), { timeout: 15_000 }).toBe("beta");
+ await expect.poll(legacy, { timeout: 15_000 }).toBeNull();
+
+ // From then on it is the first paint.
+ expect(await firstPaint(page, "/")).toBe("beta");
+
+ // With a cookie, a leftover key is removed and never read.
+ await page.evaluate(() => localStorage.setItem("activeSite", "alpha"));
+ expect(await firstPaint(page, "/channels")).toBe("beta");
+ await expect.poll(legacy, { timeout: 15_000 }).toBeNull();
+ expect(await storedCookie(page)).toBe("beta");
+});
+
+test("Back to a site's page shows that site, not the choice made there", async ({
+ page,
+}) => {
+ await twoSites();
+ const picker = page.getByLabel("Active site");
+ await page.goto("/sites/alpha/charts");
+ await hydrated(page);
+ // The choice is held on the page it was made on until the push lands…
+ await picker.selectOption("beta");
+ await expect(page).toHaveURL(/\/sites\/beta\/charts$/);
+ // …and dropped once the URL has moved: back on alpha's page, the path rules.
+ await page.goBack();
+ await expect(page).toHaveURL(/\/sites\/alpha\/charts$/);
+ await expect(picker).toHaveValue("alpha");
+});
+
+// A site picked in one tab reaches the others. The cookie is shared, but a
+// tab's picker reads its root layout, which a client-side navigation does not
+// re-render; without the broadcast, tab A kept showing alpha over beta's pages
+// and skipped recording a visit to alpha's page.
+test("a site picked in another tab reaches this one", async ({ context }) => {
+ await twoSites();
+ // Passive refresh off: a tree refresh whenever the pulse moves would also
+ // re-read the layout, and only the broadcast may move tab A here.
+ await writeSettings({ autoRefreshIntervalSeconds: 0 });
+ const a = await context.newPage();
+ const pickerA = a.getByLabel("Active site");
+ await a.goto("/channels");
+ await pickerA.selectOption("alpha");
+ await expect(a.getByRole("link", { name: "slow-b" })).toHaveCount(0);
+ await a.goto("/settings");
+ await hydrated(a);
+
+ const b = await context.newPage();
+ await b.goto("/channels");
+ await b.getByLabel("Active site").selectOption("beta");
+ await expect(b.getByRole("link", { name: "slow-a" })).toHaveCount(0);
+ expect(await storedCookie(b)).toBe("beta");
+
+ // Tab A follows without a reload, and its next page agrees with its picker.
+ await expect(pickerA).toHaveValue("beta");
+ await a.locator("aside nav").getByRole("link", { name: "Channels", exact: true }).click();
+ await expect(a).toHaveURL(/\/channels$/);
+ await expect(pickerA).toHaveValue("beta");
+ await expect(a.getByRole("link", { name: "slow-b" })).toBeVisible();
+ await expect(a.getByRole("link", { name: "slow-a" })).toHaveCount(0);
+
+ // And a client-side visit to alpha's page is recorded again: tab A no longer
+ // believes alpha is stored.
+ await a.locator("aside nav").getByRole("link", { name: "Sites", exact: true }).click();
+ await expect(a).toHaveURL(/\/sites$/);
+ await a.locator('main a[href="/sites/alpha"]').click();
+ await expect(a).toHaveURL(/\/sites\/alpha$/);
+ await expect.poll(() => storedCookie(a), { timeout: 15_000 }).toBe("alpha");
+ // Tab B hears of that write too.
+ await expect(b.getByLabel("Active site")).toHaveValue("alpha");
+});
+
+// A new channel starts checked on the active site: the stored one, or a
+// `?site=` link's instead (not both). It is the form's initial state, so it is
+// in the first paint, read here with no retry, and a refresh later (a pick in
+// another tab) does not re-check a box the user cleared.
+test("a new channel starts checked on the stored site, or on a ?site= link's", async ({
+ context,
+}) => {
+ test.setTimeout(60_000);
+ await twoSites();
+ await writeSettings({ autoRefreshIntervalSeconds: 0 });
+ const a = await context.newPage();
+ await a.goto("/channels");
+ await a.getByLabel("Active site").selectOption("alpha");
+ await expect(a.getByRole("link", { name: "slow-b" })).toHaveCount(0);
+
+ const alpha = a.getByLabel("Include on Alpha");
+ const beta = a.getByLabel("Include on Beta");
+ await a.goto("/channels/new", { waitUntil: "commit" });
+ await a.waitForLoadState("domcontentloaded");
+ expect(await alpha.isChecked(), "stored site, first paint").toBe(true);
+ expect(await beta.isChecked()).toBe(false);
+
+ await a.goto("/channels/new?site=beta", { waitUntil: "commit" });
+ await a.waitForLoadState("domcontentloaded");
+ expect(await beta.isChecked(), "the link's site, first paint").toBe(true);
+ expect(await alpha.isChecked(), "not the stored one as well").toBe(false);
+
+ // Cleared by hand, it stays cleared when another tab's pick refreshes this one.
+ await a.goto("/channels/new");
+ await expect
+ .poll(() =>
+ alpha.evaluate((el) =>
+ Object.keys(el).some((k) => k.startsWith("__reactProps")),
+ ),
+ )
+ .toBe(true);
+ await alpha.click();
+ await expect(a.getByLabel("Group for Alpha")).toHaveCount(0);
+ const b = await context.newPage();
+ await b.goto("/channels");
+ await b.getByLabel("Active site").selectOption("beta");
+ await expect(b.getByRole("link", { name: "slow-a" })).toHaveCount(0);
+ await expect(a.getByLabel("Active site")).toHaveValue("beta");
+ await a.waitForTimeout(SETTLE_MS);
+ expect(await alpha.isChecked()).toBe(false);
+ expect(await beta.isChecked()).toBe(false);
+});
diff --git a/editor/e2e/sites-crud.spec.ts b/editor/e2e/sites-crud.spec.ts
@@ -345,6 +345,62 @@ test("archives + per-video transcript downloads opt-outs round-trip", async ({
await expect(downloads).toBeChecked();
});
+test("the listed opt-out round-trips, and a save of another field keeps it", async ({
+ page,
+}) => {
+ await resetData("empty");
+ // An invented fixture id: no real site is named in a test.
+ await writeSite("fixture-unlisted", {
+ siteTitle: "Unlisted Fixture",
+ siteUrl: "https://fixture-unlisted.example",
+ listed: false,
+ });
+
+ type ListedSiteFile = { siteTitle?: string; listed?: boolean };
+ const file = "test-transcripts/sites/fixture-unlisted/site.json";
+ const listed = page.getByRole("checkbox", {
+ name: "List on the Archilyzer homepage and hub",
+ });
+ const save = async () => {
+ await page.getByRole("button", { name: /save site/i }).click();
+ await expect(
+ page.getByRole("status").filter({ hasText: "Saved" }),
+ ).toBeVisible();
+ };
+
+ // `false` on disk opens unticked, and a save that changes only the title
+ // keeps it: the action rebuilds the site from the form.
+ await page.goto("/sites/fixture-unlisted");
+ await expect(listed).not.toBeChecked();
+ await page.getByLabel(/site title/i).fill("Unlisted Fixture Renamed");
+ await save();
+ await expect(async () => {
+ const site = await readJson<ListedSiteFile>(file);
+ expect(site.siteTitle).toBe("Unlisted Fixture Renamed");
+ expect(site.listed).toBe(false);
+ }).toPass({ timeout: 10_000 });
+
+ // Ticking it removes the key — listed is the default.
+ await page.goto("/sites/fixture-unlisted");
+ await expect(listed).not.toBeChecked();
+ await listed.check();
+ await save();
+ await expect(async () => {
+ const site = await readJson<ListedSiteFile>(file);
+ expect("listed" in site).toBe(false);
+ }).toPass({ timeout: 10_000 });
+
+ // And unticking writes the explicit false again.
+ await page.goto("/sites/fixture-unlisted");
+ await expect(listed).toBeChecked();
+ await listed.uncheck();
+ await save();
+ await expect(async () => {
+ const site = await readJson<ListedSiteFile>(file);
+ expect(site.listed).toBe(false);
+ }).toPass({ timeout: 10_000 });
+});
+
test("the hub form's per-video transcript downloads opt-out round-trips to homepage.json", async ({
page,
}) => {
diff --git a/editor/e2e/storage-locations.spec.ts b/editor/e2e/storage-locations.spec.ts
@@ -678,3 +678,85 @@ test("evicting at any age needs the tick as well as the preview", async ({
await page.getByLabel("clip eviction age").selectOption("0");
await expect(page.getByLabel("confirm evicting every window")).not.toBeChecked();
});
+
+// --- THE DRIVE HEALTH TIMING (release 15 slice DT) ---------------------------
+//
+// The five numbers the editor judges "mounted but not answering" by are
+// `settings.storage.health`, edited in a collapsed block at the foot of the
+// page. The claim: a value saved there is in settings.json and is what the page
+// shows on the next load; an empty field is the default and writes nothing; a
+// value out of range is refused with a sentence and writes nothing. (That the
+// saved numbers feed the watchdog is unit-tested in lib/storageHealth.test.ts:
+// no fixture here has a drive that stalls.)
+
+async function storedHealth(): Promise<Record<string, unknown> | undefined> {
+ const s = await readJson<{ storage?: { health?: Record<string, unknown> } }>(
+ "test-settings.json",
+ );
+ return s.storage?.health;
+}
+
+// React has attached its props to the form's button: a click now runs the
+// action through React rather than as a pre-hydration form post.
+async function timingFormHydrated(page: Page): Promise<void> {
+ await page.waitForFunction(() => {
+ const el = document.querySelector('[aria-label="save timing"]');
+ return !!el && Object.keys(el).some((k) => k.startsWith("__reactProps"));
+ });
+}
+
+test("the drive health timing saves to settings.json and reads back", async ({
+ page,
+}) => {
+ test.setTimeout(90_000);
+ await resetData("one-youtube-channel-with-data");
+ await writeSettings({ adminTitle: "Test Admin", minFreeDiskGB: 0 });
+ expect(await storedHealth()).toBeUndefined();
+
+ await page.goto("/storage");
+ const block = page.getByLabel("drive health timing");
+ const summary = block.locator("summary");
+ const budget = block.getByLabel("read budget");
+ // Opened after hydration, so React never meets a `<details open>` it did
+ // not render.
+ await timingFormHydrated(page);
+ // COLLAPSED: the defaults suit a healthy disk.
+ await expect(summary).toContainText("defaults");
+ await expect(budget).toBeHidden();
+ await summary.click();
+ await expect(budget).toBeVisible();
+ // An empty field is its default, which is its placeholder.
+ await expect(budget).toHaveValue("");
+ await expect(budget).toHaveAttribute("placeholder", "3000");
+ await expect(block.getByLabel("reads at once per drive")).toHaveAttribute(
+ "placeholder",
+ "4",
+ );
+
+ await budget.fill("4000");
+ await block.getByLabel("save timing").click();
+ await expect(block.getByLabel("timing saved")).toContainText("A read may take 4 s");
+ // ONLY WHAT DIFFERS FROM A DEFAULT IS WRITTEN: the four empty fields are not.
+ expect(await storedHealth()).toEqual({ budgetMs: 4000 });
+
+ await page.reload();
+ await timingFormHydrated(page);
+ await expect(summary).toContainText("1 changed from the default");
+ await summary.click();
+ await expect(budget).toHaveValue("4000");
+ await expect(block.getByLabel("health check interval")).toHaveValue("");
+
+ // OUT OF RANGE IS REFUSED, not clamped, and nothing is written.
+ await budget.fill("200");
+ await block.getByLabel("save timing").click();
+ await expect(block.getByLabel("timing error")).toHaveText(
+ "Read budget must be between 500 and 60000 ms (got 200 ms).",
+ );
+ expect(await storedHealth()).toEqual({ budgetMs: 4000 });
+
+ // Emptied, it is the default again, and the block is gone from the file.
+ await budget.fill("");
+ await block.getByLabel("save timing").click();
+ await expect(block.getByLabel("timing saved")).toContainText("A read may take 3 s");
+ expect(await storedHealth()).toBeUndefined();
+});
diff --git a/editor/e2e/theme.spec.ts b/editor/e2e/theme.spec.ts
@@ -1,5 +1,6 @@
import { test, expect, type Page } from "@playwright/test";
import { resetData } from "./helpers";
+import { RETIRED_BASE } from "../../common/components/themeConfig";
// Regression: refreshing the editor must honor the persisted theme. The
// pre-paint <ThemeScript> sets `data-base` and `.dark` on <html>, but <html> is
@@ -9,7 +10,8 @@ import { resetData } from "./helpers";
// persisted base on mount.
//
// And the one-time migration: the retired `ytdlp-tb:theme` / `ytdlp-tb:mode`
-// keys become a base (archive + light → sepia) and are deleted.
+// keys become a base (archive + light → light) and are deleted; a stored
+// retired third ground becomes light.
const BASE_KEY = "ytdlp-tb:base";
const LEGACY_THEME_KEY = "ytdlp-tb:theme";
const LEGACY_MODE_KEY = "ytdlp-tb:mode";
@@ -55,14 +57,46 @@ test("explicit light base loads light", async ({ page }) => {
await expect(html).toHaveAttribute("data-base", "light");
});
-test("sepia loads as a light ground (no .dark) and survives a reload", async ({
- page,
-}) => {
- await seed(page, { [BASE_KEY]: "sepia" });
- await page.reload();
- const html = page.locator("html");
- await expect(html).toHaveAttribute("data-base", "sepia");
- await expect(html).not.toHaveClass(/(^|\s)dark(\s|$)/);
+// The retired third ground: a reader who chose it gets Light, before first
+// paint, with no other ground on the way, and the stored value becomes
+// "light" once. Every value `data-base` holds is recorded by a
+// MutationObserver installed before the page's first script.
+test.describe("a stored retired base, with the OS dark", () => {
+ test.use({ colorScheme: "dark" });
+
+ test("renders Light with no other ground on the way, and is rewritten", async ({ page }) => {
+ await page.addInitScript(
+ ([key, retired]) => {
+ try {
+ localStorage.setItem(key, retired);
+ } catch {}
+ const held: (string | null)[] = [];
+ (window as unknown as { __bases: (string | null)[] }).__bases = held;
+ new MutationObserver((records) => {
+ for (const r of records) if (r.target === document.documentElement) held.push(r.oldValue);
+ }).observe(document, {
+ subtree: true,
+ attributes: true,
+ attributeFilter: ["data-base"],
+ attributeOldValue: true,
+ });
+ },
+ [BASE_KEY, RETIRED_BASE],
+ );
+ await page.goto("/", { waitUntil: "commit" });
+ await page.waitForFunction(() => document.documentElement?.dataset.themeReady === "1");
+ await page.waitForLoadState("load");
+ await page.waitForTimeout(300);
+ const held = await page.evaluate(() => [
+ ...(window as unknown as { __bases: (string | null)[] }).__bases,
+ document.documentElement.getAttribute("data-base"),
+ ]);
+ expect(held.filter((v) => v !== null && v !== "light"), JSON.stringify(held)).toEqual([]);
+ expect(held.at(-1)).toBe("light");
+ const html = page.locator("html");
+ await expect(html).not.toHaveClass(/(^|\s)dark(\s|$)/);
+ expect(await storage(page)).toEqual(["light", null, null]);
+ });
});
test.describe("system base with OS dark", () => {
@@ -89,13 +123,13 @@ test("migration: a stored selenized + dark becomes the dark base; the old keys g
expect(await storage(page)).toEqual(["dark", null, null]);
});
-test("migration: a stored archive + light (the paper look) becomes sepia", async ({
+test("migration: a stored archive + light (the paper look) becomes light", async ({
page,
}) => {
await seed(page, { [LEGACY_MODE_KEY]: "light", [LEGACY_THEME_KEY]: "archive" });
await page.reload();
const html = page.locator("html");
- await expect(html).toHaveAttribute("data-base", "sepia");
+ await expect(html).toHaveAttribute("data-base", "light");
await expect(html).not.toHaveClass(/(^|\s)dark(\s|$)/);
- expect(await storage(page)).toEqual(["sepia", null, null]);
+ expect(await storage(page)).toEqual(["light", null, null]);
});
diff --git a/editor/instrumentation.ts b/editor/instrumentation.ts
@@ -4,8 +4,10 @@
//
// See editor/app/scheduler/heartbeat.ts and SCHEDULED_SYNC.md.
//
-// Everything armed here except the shutdown reaper is skipped when the process
-// boots idle (ARCHILYZER_IDLE_BOOT) — see isIdleBoot below.
+// Everything armed here that starts or writes work is skipped when the process
+// boots idle (ARCHILYZER_IDLE_BOOT) — see isIdleBoot below. What stays armed
+// only stops work or only reads: the shutdown reaper, the persisted-pause
+// restore, the storage boot probe and the drive health pass.
//
// The one STATIC import in this file, and safe as one because idleBoot.ts
// imports nothing and touches no Node API: the Edge bundle's static Node-API
@@ -83,8 +85,26 @@ export async function register() {
/* a failed re-pause must not block server readiness */
}
- // ONE PASS OVER THE STORAGE LOCATIONS, and it runs on an IDLE BOOT TOO —
- // the only thing below the reaper that does.
+ // THE DRIVE HEALTH PASS, every `storage.health.passIntervalMs` (15 s by
+ // default; a save on /storage re-arms it) — and ON AN IDLE BOOT TOO, like the
+ // probe below. It reads each location's block device counters (never the
+ // drive) and keeps, in memory only, which drives are not answering; every
+ // page and poll asks it before touching a drive, and its watchdog marks a
+ // drive a page reached and got no answer from. It writes nothing and starts
+ // no work, and without it nothing would ever clear such a mark. The
+ // five-minute pass that may auto-pause channels is armed below the idle gate.
+ // See common/controller/storageWatch.ts and common/lib/storageHealth.ts.
+ try {
+ const { startStorageHealthWatch } = await import(
+ "yt-dlp-transcript-common/controller/storageWatch"
+ );
+ startStorageHealthWatch({ log: (line) => console.log(line) });
+ } catch {
+ /* a health pass that fails to arm must not block server readiness */
+ }
+
+ // ONE PASS OVER THE STORAGE LOCATIONS, and it runs on an IDLE BOOT TOO: it
+ // only reads, like the health pass above.
//
// The pass probes each location (is the disk here, and if not, where?) and,
// for a location the operator armed with `autoRepoint`, re-points it to
@@ -165,10 +185,11 @@ export async function register() {
// is the thing that looks, on a five-minute cadence, and auto-pauses (and
// later restores) the channels on a location that is not there.
//
- // BELOW THE IDLE GATE, deliberately, and unlike the boot probe above: this
- // one WRITES settings.channelPriority, and a container pointed at somebody
- // else's corpus for the first time has no business rewriting that corpus's
- // priority document. See common/controller/storageWatch.ts.
+ // BELOW THE IDLE GATE, deliberately, and unlike the boot probe and the
+ // health pass above: this one WRITES settings.channelPriority, and a
+ // container pointed at somebody else's corpus for the first time has no
+ // business rewriting that corpus's priority document. See
+ // common/controller/storageWatch.ts.
try {
const { startStorageWatch } = await import(
"yt-dlp-transcript-common/controller/storageWatch"
diff --git a/editor/next.config.ts b/editor/next.config.ts
@@ -80,7 +80,8 @@ const nextConfig: NextConfig = {
// site's tab: the value regex is SITE_ID_RE, anchored by Next, so
// ?site=__all__ (underscores) misses it and lands on the family page. The
// matched query is NOT stripped — /charts?site=a lands on
- // /sites/a/charts?site=a — which the tab ignores and the picker reconciles.
+ // /sites/a/charts?site=a — which the tab ignores, and the picker too: on a
+ // site's own pages the path is the selection (app/lib/activeSite.ts).
// Order matters: first match wins, so each `has` rule precedes its bare one.
//
// TEMPORARY, not permanent: a 308 is cached by the browser forever, and this
diff --git a/editor/package.json b/editor/package.json
@@ -8,7 +8,7 @@
"dev:test": "WORKER_TOKEN=test-worker-token TRANSCRIPTS_DIR=$(pwd)/test-transcripts EXPORT_PUBLIC_DIR=$(pwd)/test-transcripts/.export-public SETTINGS_FILE=$(pwd)/test-settings.json EDITOR_CHANGELOG_FILE=$(pwd)/test-changelog.md EXPORT_CHANGELOG_FILE=$(pwd)/test-export-changelog.md YTDLP_BIN=$(pwd)/e2e/fixtures/bin/fake-ytdlp.mjs GALLERY_DL_BIN=$(pwd)/e2e/fixtures/bin/fake-gallery-dl.mjs WHISPER_BIN=$(pwd)/e2e/fixtures/bin/fake-whisper.mjs WHISPER_MODEL=/dev/null CHOUGH_BIN=$(pwd)/e2e/fixtures/bin/fake-chough.mjs CHOUGH_MODEL=/dev/null PARAKEET_STITCH_BIN=$(pwd)/e2e/fixtures/bin/fake-parakeet-stitch.mjs PARAKEET_CLI=/dev/null PARAKEET_MODEL=/dev/null DIARIZE_BIN=$(pwd)/e2e/fixtures/bin/fake-diarize.mjs FFMPEG_BIN=$(pwd)/e2e/fixtures/bin/fake-ffmpeg.mjs FFPROBE_BIN=$(pwd)/e2e/fixtures/bin/fake-ffprobe.mjs OLLAMA_URL=http://127.0.0.1:${OLLAMA_STUB_PORT:-11435} CLAUDE_BIN=$(pwd)/e2e/fixtures/bin/fake-claude.mjs FINDMNT_BIN=$(pwd)/e2e/fixtures/bin/fake-findmnt.mjs UDISKSCTL_BIN=$(pwd)/e2e/fixtures/bin/fake-udisksctl.mjs next dev --port ${PORT:-3011}",
"start:test": "WORKER_TOKEN=test-worker-token TRANSCRIPTS_DIR=$(pwd)/test-transcripts EXPORT_PUBLIC_DIR=$(pwd)/test-transcripts/.export-public SETTINGS_FILE=$(pwd)/test-settings.json EDITOR_CHANGELOG_FILE=$(pwd)/test-changelog.md EXPORT_CHANGELOG_FILE=$(pwd)/test-export-changelog.md YTDLP_BIN=$(pwd)/e2e/fixtures/bin/fake-ytdlp.mjs GALLERY_DL_BIN=$(pwd)/e2e/fixtures/bin/fake-gallery-dl.mjs WHISPER_BIN=$(pwd)/e2e/fixtures/bin/fake-whisper.mjs WHISPER_MODEL=/dev/null CHOUGH_BIN=$(pwd)/e2e/fixtures/bin/fake-chough.mjs CHOUGH_MODEL=/dev/null PARAKEET_STITCH_BIN=$(pwd)/e2e/fixtures/bin/fake-parakeet-stitch.mjs PARAKEET_CLI=/dev/null PARAKEET_MODEL=/dev/null DIARIZE_BIN=$(pwd)/e2e/fixtures/bin/fake-diarize.mjs FFMPEG_BIN=$(pwd)/e2e/fixtures/bin/fake-ffmpeg.mjs FFPROBE_BIN=$(pwd)/e2e/fixtures/bin/fake-ffprobe.mjs OLLAMA_URL=http://127.0.0.1:${OLLAMA_STUB_PORT:-11435} CLAUDE_BIN=$(pwd)/e2e/fixtures/bin/fake-claude.mjs FINDMNT_BIN=$(pwd)/e2e/fixtures/bin/fake-findmnt.mjs UDISKSCTL_BIN=$(pwd)/e2e/fixtures/bin/fake-udisksctl.mjs next start --port ${PORT:-3011}",
"build": "next build",
- "start": "next start --port ${EDITOR_PORT:-3001}",
+ "start": "UV_THREADPOOL_SIZE=${UV_THREADPOOL_SIZE:-16} next start --port ${EDITOR_PORT:-3001}",
"lint": "eslint",
"test": "tsx --test \"app/**/*.test.ts\"",
"e2e": "node ../scripts/queue-lock.mjs --ports PORT:3011,EXPORT_PORT:3010,OLLAMA_STUB_PORT:11435 -- playwright test",
diff --git a/export/CHANGELOG.md b/export/CHANGELOG.md
@@ -1,5 +1,19 @@
# Changelog
+## [Unreleased]
+- **Use with AI goes to the Archilyzer site's AI and MCP doc; the page on each site is gone.** The header's, the slide-out menu's, the footer's and Ask AI's **Use with AI** keep their label and open https://archilyzer.pages.dev/docs/ai-and-mcp/ in the same tab, on every site and the hub, where one block says how to run Claude Code against any archive (the source, `pnpm install`, `claude mcp add archilyzer`, `/ask`). `/use-with-ai/` is no longer built. `corpus.json`'s `useWithAi` names the doc; `llms.txt`'s Ask AI section lists the site's `/ask/` chat and the doc; the sitemap drops `/use-with-ai`. Needs a rebuild and deploy of each site and the hub.
+- **A search with a layer that has nothing to read finishes.** A "Posts" layer under a tag chip, or a "Live chat" layer where no video in the selection has live chat, read "searched N/M…" for ever and never said "No matching videos."; it now finishes at once, having matched nothing. Needs a rebuild and deploy of each site and the hub.
+- **A search reads what the visitor ticks under "Search in": Transcripts, Posts and Live chat.** The Filters panel has a new row, **Search in**, beside Type. **Transcripts** and **Posts** are ticked by default and **Live chat** is not; Posts is offered only on a site that has posts, and Live chat only on a site with live chat. The row decides what a plain query reads: with Posts ticked, a plain query now finds posts as well as videos (before, a post was found only by a layer whose scope was "Posts"); with Live chat ticked, it finds live-chat messages too, shown in the same video's card beside the transcript hits, each marked "live chat"; with Transcripts unticked it reads no transcripts. A layer whose scope is picked by name in the query builder ("Live chat", "Posts", "Title / channel", …) reads what it names, whatever the row says. An empty query still lists every video the Type row keeps. With nothing ticked, Search and Apply filters are disabled and the row says "Search in: pick at least one". The Posts box moved here from the Type row, and unticking it no longer empties a layer whose scope is "Posts". Under a tag chip a plain query reads no posts, since a post carries no tags. The row is remembered, and saved with a profile; a shared link does not carry it, so it opens with the reader's own row. A live-chat hit now wears its "live chat" badge wherever it is shown, and the hint under the search bar says to tick Live chat under Search in. Posts unticked is now also remembered after a reload and restored with a profile, which it was not. Needs a rebuild and deploy of each site and the hub.
+
+## [0.11.0] - 2026-09-30
+- **The charts count every transcript, once the site is rebuilt.** A transcript that arrived after its video was first indexed was missing from the charts' transcript and cue counts and from "Transcribed over time", and a video with YouTube captions alone had no transcription date. Both are counted now, and a captioned video is dated by when its captions arrived.
+- **A social icon that fails the check is shown as its label, and every icon paints inside its box.** The footer inlines a social link's SVG only if it passes the same check a save runs (what an icon may contain is in `SITE.md`); otherwise the link shows its label as text, at most 10rem with an ellipsis. Each icon is clipped to its own box. Needs a rebuild and deploy of each site.
+- **A chart's stacked bars are separated by a 2 px gap in the chart card's colour.** A stacked bar's segments were drawn touching; they now have a 2 px gap in the card's colour between them, and in high-contrast mode the system's background colour. Stacked areas keep their line in each series' colour along the top, charts of one series, line charts and side-by-side bars are unchanged. Needs a rebuild and deploy of each site.
+- **Two grounds, Light and Dark, and each site in its own accent.** The third ground, the warm paper one, is gone: the header's toggle cycles System, Light and Dark. A reader who had chosen it gets Light, before the page first paints and with no other ground on the way, and the stored choice becomes Light (the old paper theme's `archive` + `light` too). The theme menu's accent picker is gone from the header and the slide-out menu: every page wears the site's own accent (`site.json` `accent`), and a reader's stored pick from before is not read and is left in storage. Needs a rebuild and deploy of each site.
+- **The header carries the operator's social links and one theme toggle, keeps the site's name on a small screen, and links to the Archilyzer home in place of the sites menu.** Every site's header and the hub's end with the social icons (the site's `socialLinks`, else `settings.json`'s) followed by the theme toggle, all 36 px keys (44 px on a touch screen) with a focus ring. From 520 px wide the header shows every link, up to four (with more, the ones marked **Keep in header on small screens** first, then the last of the rest); below 520 px it shows only the marked ones (none marked → none) and keeps the site's name beside them. The switch is 32.5rem, so at a larger text size it comes later. With one marked link, every current site's name shows in full from 360 px wide on a touch screen. The footer keeps every link, in the same keys (its icons were 20 px and turned the accent on hover; they now turn the text colour), and wraps them rather than widen the page. The **Sites** dropdown and the **Hub** link are gone from the header and the slide-out menu: in their place a link, **Archilyzer**, goes to the Archilyzer home's Official Instances, in the same tab (not on the hub, which lists them itself). **Changelog** moved from the header and the menu to the footer, after Use with AI. The nav and the Archilyzer link are inline from 1024 px wide; below that they are in the slide-out menu, which now holds only them. Only as last resorts, for a very long name on a phone, does the name drop (its mark stays; the same before and after the page's font has loaded, and never with its last letter cut off) and do the icons scroll sideways in their own box. A site's `hubUrl` still loads and is no longer shown. Needs a rebuild and deploy of each site.
+- **An unlisted site is not in the hub or in another site's footer.** The hub's members (`hub-sites.json`), and so its federated search, `corpus.json` and `llms.txt`, leave out a site whose `listed` is `false`; the hub's instance figures count none of the channels only it carries; and no other site's footer links it, even from a featured group. The unlisted site's own pages are unchanged.
+- **A clear screen until the first Search.** A plain visit to a site's search page, and to the hub's, shows the search bar, the page's intro and the footer: no count, no listing and no results controls, and the line under the bar, "Press Enter or click Search to apply", says what to do. Search with the box empty lists every video, as before. A link that carries a query or a filter (`qt=`, `q=`, `tg=`, a share link, the older filter keys) still shows its results on load. Within one visit the results stay: going to Ask AI or another page and coming back keeps them. A reload starts over, and shows results at once only when the address carries a query or a filter. A query restored from the last visit waits in the box until Search, over the clear screen or under a filter link's results, and it still waits after another page and Back. Needs a rebuild and deploy of each site and the hub.
+
## [0.10.0] - 2026-09-28
- **A video whose recheck failed shows as possibly missing rather than available.** When a video drops out of its channel's listing it is marked "Missing?" until a recheck says why. A recheck that could not reach the video — a blocked request or a network error — used to clear the mark as if the video had been found. It now leaves "Missing?" in place until a recheck actually reaches the video. Needs a rebuild and deploy of every export site.
- **The hub's Ask AI says when the hub has no archives.** On a hub whose list of archives is empty or could not be read, with none added in this browser, the question box read "Loading transcripts…" forever. It is now disabled, and one line under it says the hub has no archives yet, with a link to the front page, where one can be added.
diff --git a/export/app/(workspace)/ask/page.tsx b/export/app/(workspace)/ask/page.tsx
@@ -1,5 +1,5 @@
import type { Metadata } from "next";
-import Link from "next/link";
+import { AI_DOC_URL } from "yt-dlp-transcript-common/lib/project";
import { currentSite } from "../../lib/site";
import { instanceMode } from "../../lib/mode";
import AskHub from "../../ask/AskHub";
@@ -34,9 +34,9 @@ export default function AskPage() {
browser, sends the relevant excerpts to your chosen AI, and answers with
citations. Nothing is hosted here — your key and the requests stay
between your browser and the provider. See{" "}
- <Link href="/use-with-ai" className="text-brand hover:underline">
+ <a href={AI_DOC_URL} className="text-brand hover:underline">
Use with AI
- </Link>{" "}
+ </a>{" "}
for other ways to use {isHub ? "every archive on this hub" : "this archive"}.
</p>
</header>
diff --git a/export/app/changelog/page.tsx b/export/app/changelog/page.tsx
@@ -7,7 +7,7 @@ export const metadata: Metadata = { title: "Changelog" };
function loadChangelog(): string {
return readFileSync(
- path.join(process.cwd(), "CHANGELOG.md"),
+ path.join(/* turbopackIgnore: true */ process.cwd(), "CHANGELOG.md"),
"utf8",
);
}
diff --git a/export/app/components/Footer.tsx b/export/app/components/Footer.tsx
@@ -1,18 +1,17 @@
-import {
- getSettings,
- sizeSocialSvg,
-} from "yt-dlp-transcript-common/lib/settings";
+import { getSettings } from "yt-dlp-transcript-common/lib/settings";
import {
listSites,
resolveRelatedSites,
resolveSocialLinks,
} from "yt-dlp-transcript-common/lib/site";
import {
+ AI_DOC_URL,
PROJECT_NAME,
PROJECT_URL,
} from "yt-dlp-transcript-common/lib/project";
import { ICON_PALETTES } from "yt-dlp-transcript-common/lib/brand";
import { BrandMark } from "yt-dlp-transcript-common/components/BrandMark";
+import { SocialLinks } from "yt-dlp-transcript-common/components/SocialLinks";
import { currentSite } from "../lib/site";
import { instanceMode } from "../lib/mode";
import { hasArchives } from "../lib/archives";
@@ -63,22 +62,35 @@ export default function Footer() {
Offline
</a>
)}
- <span aria-hidden="true" className="text-muted-foreground/50">
- ·
- </span>
+ {/* The dot parts the downloads from the rest, so it shows only when
+ there is something before it. */}
+ {(hasArchives() || site.pwa) && (
+ <span aria-hidden="true" className="text-muted-foreground/50">
+ ·
+ </span>
+ )}
+ {/* The homepage's AI and MCP doc, in the same tab (release 16). */}
<a
- href="/use-with-ai"
+ href={AI_DOC_URL}
className="underline underline-offset-2 hover:text-foreground transition-colors"
>
Use with AI
</a>
+ {/* Changelog lives here, not in the header (release 14). */}
+ <a
+ href="/changelog"
+ className="underline underline-offset-2 hover:text-foreground transition-colors"
+ >
+ Changelog
+ </a>
</div>
<div className="flex flex-wrap items-center gap-x-4 gap-y-2">
{/* "Built with Archilyzer" — deliberately NOT gated on instanceMode()
(being identical on every deployment is the point), NOT dependent
on a configured hubUrl (that is the operator's family link, a
- different thing), and NOT target="_blank": the header's hub link
- navigates in the same tab, and only the social icons open new ones.
+ different thing), and NOT target="_blank": the header's Archilyzer
+ link navigates in the same tab, and only the social icons open new
+ ones.
"Built with" sits OUTSIDE the anchor so the accessible name is
exactly "Archilyzer" — and so does the parent mark before it,
which is decorative (aria-hidden) and the same on every site. */}
@@ -94,23 +106,9 @@ export default function Footer() {
</a>
</span>
</span>
- {socialLinks.length > 0 && (
- <ul className="flex items-center gap-3 list-none">
- {socialLinks.map((link, i) => (
- <li key={`${link.url}-${i}`}>
- <a
- href={link.url}
- title={link.label}
- aria-label={link.label}
- target="_blank"
- rel="noopener noreferrer"
- className="inline-block w-5 h-5 text-muted-foreground hover:text-brand transition-colors [&_svg]:w-full [&_svg]:h-full"
- dangerouslySetInnerHTML={{ __html: sizeSocialSvg(link.svg) }}
- />
- </li>
- ))}
- </ul>
- )}
+ {/* The operator's social links, every one of them (the header shows
+ at most four): the shared row, its keys and focus ring. */}
+ <SocialLinks links={socialLinks} placement="footer" />
</div>
</div>
{related.length > 0 && (
diff --git a/export/app/components/Header.tsx b/export/app/components/Header.tsx
@@ -1,67 +1,102 @@
import Link from "next/link";
-import { ArrowUpLeft } from "lucide-react";
import { getSettings } from "yt-dlp-transcript-common/lib/settings";
-import {
- listSites,
- resolveHubUrl,
- resolveRelatedSites,
-} from "yt-dlp-transcript-common/lib/site";
+import { resolveSocialLinks } from "yt-dlp-transcript-common/lib/site";
+import { headerSocialLinks } from "yt-dlp-transcript-common/lib/socialLinks";
+import { AI_DOC_URL, INSTANCES_URL } from "yt-dlp-transcript-common/lib/project";
import { ThemeToggle } from "yt-dlp-transcript-common/components/ThemeToggle";
-import { ThemeMenu } from "yt-dlp-transcript-common/components/ThemeMenu";
+import { SocialLinks } from "yt-dlp-transcript-common/components/SocialLinks";
+import { SocialScroll } from "yt-dlp-transcript-common/components/SocialScroll";
import { BrandMark } from "yt-dlp-transcript-common/components/BrandMark";
import { Wordmark } from "yt-dlp-transcript-common/components/Wordmark";
+import { wordmarkWidthEm } from "yt-dlp-transcript-common/lib/wordmarkWidth";
import { currentSite } from "../lib/site";
import { headerMarkPalette } from "../lib/brand";
import { instanceMode } from "../lib/mode";
import { hasArchives } from "../lib/archives";
import { hasDuplicates } from "../lib/duplicates";
-import SiblingSwitcher from "./SiblingSwitcher";
import MobileMenu from "./MobileMenu";
-// The export site's masthead: the Found-line mark + the split wordmark, calm
-// sans nav, and the cross-site "family" chrome (hub backlink + sibling
-// switcher). The mark's lit line follows the reader's accent (lib/brand.ts
-// headerMarkPalette); the hub wears the parent mark. The link's accessible name
-// is the header title exactly — the mark is decorative and the wordmark's two
-// spans join without a space. All cross-site data is resolved at build time
-// from the pool, so single-site installs simply render neither.
+// The export site's masthead (every site, and the hub): the Found-line mark +
+// the split wordmark, the nav, a link to the Archilyzer home's official
+// instances, and the homepage's group — the operator's social links
+// (common/components/SocialLinks.tsx) and the theme toggle
+// (common/components/ThemeToggle.tsx, `variant="bare"`) as the group's last
+// key, their boxes touching, every glyph 16 px from the next. The mark's lit
+// line follows the site's accent (lib/brand.ts headerMarkPalette); the hub
+// wears the parent mark. The brand link's accessible name is the header title
+// exactly — the mark is decorative and the wordmark's two spans join without a
+// space.
+//
+// ≥ lg brand · nav · Archilyzer · group
+// < lg brand · group · menu trigger (the nav, and the Archilyzer link, in
+// the slide-out menu). The nav, the link and a long title with four
+// icons do not fit a 768 px bar, and the nav gives way before any
+// icon scrolls.
+//
+// THE HEADER KEEPS THE SITE'S NAME ON A NARROW SCREEN (the ruling of
+// 2026-09-28): below the switch the social row holds only the links marked
+// `featured` (none marked → none; the footer always shows every link); from
+// it the row holds every link, up to four, the marked ones kept first
+// (headerSocialLinks, "narrow" and "wide"). Both rows are rendered and CSS
+// shows one; the other is display:none, so exactly one is focusable and in
+// the accessibility tree. The switch is 32.5rem — 520 px at the default text
+// size, where every link (up to four, 44 px keys), the toggle, the menu
+// button and the longest real site title fit the bar (release-14.md, the
+// measured table) — in rem so it moves with the reader's text size, as the
+// keys and the title do. Below it the bar's two gaps (name → row, toggle →
+// menu) are 8 px, not 12 (the ruling of 2026-09-29), so every real title
+// shows in full at 360 px with one marked link under touch; type, the
+// mark–name gap and the reserved width are unchanged. The
+// Archilyzer link replaces the old sites dropdown and the hub backlink; it is
+// absent on the hub, which lists the instances itself. Changelog is in the
+// footer.
+//
+// THE LAST RESORTS, now rare (a very long title on a phone): the wordmark's
+// TEXT drops and the mark stays, exactly when the text does not fit — whatever
+// the site's title. Titles differ in length ("Bonnellyzer", "Rekietalyzer"), so no width
+// threshold is written down: the text sits in a one-line box (`h-7`,
+// `overflow-hidden`, `flex-wrap`) behind a zero-width strut, and a flex item
+// that does not fit beside the strut wraps to the box's second line, which is
+// clipped. The text keeps a MINIMUM WIDTH reserved from the display face's
+// own metrics (lib/wordmarkWidth.ts, in em, so it scales with the reader's
+// text size): the decision is the same in the fallback face, which is
+// narrower, and after Archivo loads — a title never shows and then vanishes.
+// A minimum, not a width: where the text renders wider than the reservation,
+// its box grows with it instead of clipping the last letter. The text stays
+// in the DOM, so the link keeps its name.
+// Below `lg` the brand link takes the bar's free space (basis 0, grow 1) and
+// can shrink to the mark alone; only then does the group give way, and its
+// social row scrolls in SocialScroll's box, the last resort, its END shown
+// first, the toggle outside it. From `lg` the link is sized by its content,
+// so it shrinks first by weight (`shrink-[999]`): with a very long title the
+// text drops before any icon scrolls there too.
export default function Header() {
const site = currentSite();
- const settings = getSettings();
- // The hub this site points visitors toward: its own hubUrl override, else the
- // family default (resolveHubUrl). Only a plain site shows the backlink — the
- // hub itself never links to itself — and never when the parent is this very
- // deployment.
const isSite = instanceMode() === "site";
- const resolvedHub = resolveHubUrl(site, settings);
- const hubUrl =
- isSite && resolvedHub && resolvedHub !== site.siteUrl
- ? resolvedHub
- : undefined;
- // The sibling switcher is build-time family navigation — it belongs on a
- // single site, not on the hub (whose "family" is the runtime shelf).
- const related = isSite ? resolveRelatedSites(site, listSites()) : [];
+ const socialLinks = resolveSocialLinks(site, getSettings());
+ const narrow = headerSocialLinks(socialLinks, "narrow").length;
+ const wide = headerSocialLinks(socialLinks, "wide").length;
const showDownloads = hasArchives();
const showDuplicates = hasDuplicates();
- // The inline nav from md. Ask AI is new: the chat was only reachable from
+ // The inline nav from lg. Ask AI is new: the chat was only reachable from
// the workspace control or from Use with AI, which is not where anyone looks
// for it.
- const navLinks = [
+ // Use with AI is the homepage's AI and MCP doc (release 16 slice DX): off
+ // this site, so a plain <a> in the same tab, not a client navigation.
+ const navLinks: { href: string; label: string; external?: boolean }[] = [
{ href: "/", label: "Search" },
{ href: "/ask/", label: "Ask AI" },
...(showDuplicates ? [{ href: "/duplicates", label: "Duplicates" }] : []),
...(showDownloads ? [{ href: "/downloads", label: "Downloads" }] : []),
- { href: "/use-with-ai", label: "Use with AI" },
+ { href: AI_DOC_URL, label: "Use with AI", external: true },
];
- // The sheet also takes the two links the wide header keeps in its right
- // cluster or its footer: Changelog, and Offline on a PWA-shipping site —
- // gated exactly as Footer.tsx gates it, so the two never disagree about
- // whether this instance has an offline mode.
+ // The sheet also takes Offline on a PWA-shipping site — gated exactly as
+ // Footer.tsx gates it, so the two never disagree about whether this instance
+ // has an offline mode.
const menuLinks = [
...navLinks,
...(site.pwa ? [{ href: "/offline/", label: "Offline" }] : []),
- { href: "/changelog", label: "Changelog" },
];
return (
@@ -69,54 +104,81 @@ export default function Header() {
{/* min-h-14, not h-14, and never wrapping: a fixed-height flex-wrap row
put the overflow rows OUTSIDE the sticky header's background at phone
widths, and the page scrolled through them. */}
- <div className="max-w-6xl mx-auto px-4 sm:px-6 flex min-h-14 items-center gap-x-5 gap-y-1 flex-nowrap">
- <Link href="/" className="flex min-w-0 items-center gap-2.5">
+ <div className="max-w-6xl mx-auto px-4 sm:px-6 flex min-h-14 items-center gap-3 max-[32.5rem]:gap-2 lg:gap-5 flex-nowrap">
+ <Link
+ href="/"
+ data-header-brand=""
+ className="flex min-w-7 flex-1 basis-0 items-center lg:grow-0 lg:basis-auto lg:shrink-[999]"
+ >
<BrandMark palette={headerMarkPalette()} className="size-7 shrink-0" />
- <Wordmark
- title={site.headerTitle}
- lead={site.wordmarkLead}
- className="min-w-0 truncate text-[1.35rem] leading-none tracking-[-0.01em]"
- />
+ <span className="flex h-7 min-w-0 flex-1 flex-wrap items-center overflow-hidden">
+ <span aria-hidden="true" className="h-7 w-0" />
+ <Wordmark
+ title={site.headerTitle}
+ lead={site.wordmarkLead}
+ className="ml-2.5 shrink-0 whitespace-nowrap text-[1.35rem] leading-none tracking-[-0.01em]"
+ style={{ minWidth: `${wordmarkWidthEm(site.headerTitle, site.wordmarkLead, -0.01)}em` }}
+ />
+ </span>
</Link>
- <nav className="hidden md:flex items-center gap-4 text-sm font-medium">
- {navLinks.map((l) => (
- <Link
- key={l.href}
- href={l.href}
- className="text-foreground hover:text-brand transition-colors"
- >
- {l.label}
- </Link>
- ))}
- </nav>
-
- <div className="ml-auto flex items-center gap-2 sm:gap-3 shrink-0">
- <div className="hidden md:flex items-center gap-2 sm:gap-3">
- <SiblingSwitcher groups={related} />
- {hubUrl && (
+ <nav className="hidden lg:flex shrink-0 items-center gap-4 text-sm font-medium">
+ {navLinks.map((l) =>
+ l.external ? (
<a
- href={hubUrl}
- rel="noopener noreferrer"
- className="inline-flex items-center gap-1 font-mono text-xs uppercase tracking-[0.12em] text-muted-foreground hover:text-brand transition-colors"
+ key={l.href}
+ href={l.href}
+ className="text-foreground hover:text-brand transition-colors"
>
- <ArrowUpLeft className="size-3.5" aria-hidden="true" />
- Hub
+ {l.label}
</a>
- )}
- <Link
- href="/changelog"
- className="text-sm text-muted-foreground hover:text-foreground transition-colors"
+ ) : (
+ <Link
+ key={l.href}
+ href={l.href}
+ className="text-foreground hover:text-brand transition-colors"
+ >
+ {l.label}
+ </Link>
+ ),
+ )}
+ </nav>
+
+ <div className="flex min-w-0 items-center gap-5 lg:ml-auto">
+ {isSite && (
+ <a
+ href={INSTANCES_URL}
+ aria-label="Archilyzer — official instances"
+ className="hidden lg:inline shrink-0 text-sm text-muted-foreground hover:text-foreground transition-colors"
>
- Changelog
- </Link>
- <ThemeMenu />
+ Archilyzer
+ </a>
+ )}
+ <div data-header-group="" className="flex min-w-0 items-center">
+ {narrow > 0 && (
+ <SocialScroll className="min-[32.5rem]:hidden">
+ <SocialLinks
+ links={socialLinks}
+ placement="header"
+ width="narrow"
+ className="w-max px-1 [direction:ltr]"
+ />
+ </SocialScroll>
+ )}
+ {wide > 0 && (
+ <SocialScroll className="hidden min-[32.5rem]:block">
+ <SocialLinks
+ links={socialLinks}
+ placement="header"
+ width="wide"
+ className="w-max px-1 [direction:ltr]"
+ />
+ </SocialScroll>
+ )}
+ <ThemeToggle variant="bare" />
</div>
- {/* The one control that stays at every width: theme.spec resolves it
- by /switch to/i, and one tap to flip light/dark is worth a slot. */}
- <ThemeToggle />
- <MobileMenu links={menuLinks} sites={related} hubUrl={hubUrl} />
</div>
+ <MobileMenu links={menuLinks} instancesUrl={isSite ? INSTANCES_URL : undefined} />
</div>
</header>
);
diff --git a/export/app/components/MobileMenu.tsx b/export/app/components/MobileMenu.tsx
@@ -2,7 +2,7 @@
import { useState } from "react";
import Link from "next/link";
-import { ArrowUpLeft, MenuIcon } from "lucide-react";
+import { MenuIcon } from "lucide-react";
import { Button } from "yt-dlp-transcript-common/components/ui/button";
import {
Sheet,
@@ -11,39 +11,29 @@ import {
SheetTitle,
SheetTrigger,
} from "yt-dlp-transcript-common/components/ui/sheet";
-import { useTheme } from "yt-dlp-transcript-common/components/ThemeProvider";
-import {
- THEME_BASES,
- accentOptions,
- isThemeAccent,
- isThemeBase,
-} from "yt-dlp-transcript-common/components/themeConfig";
-import type { SwitcherGroup } from "./SiblingSwitcher";
-// The phone half of the masthead. Below `md` the header keeps only the brand,
-// the base toggle and this trigger; everything else the header offers — the
-// nav, the sibling sites, the hub backlink, the base and accent pickers —
-// moves in here, which is also how the Hub backlink and the sites list become
-// reachable on a phone at all (they were `hidden sm:*` and a dropdown too small
-// to hit).
+// The narrow half of the masthead: the NAV. Below `lg` the header's bar keeps
+// the brand, the social row, the theme toggle and this trigger; the nav's
+// links (five or more do not fit a phone's or a tablet's bar beside the group)
+// move in here, with the link to the Archilyzer home's official instances
+// after them on a site (the bar shows both inline from `lg`). Nothing else:
+// the theme is the bar's toggle, the sibling sites are the footer's, and
+// Changelog is in the footer.
//
-// Header is a server component, so the link groups arrive as plain serialisable
-// props. The base and accent lists are read from the client ThemeProvider and
-// rendered as NATIVE radio groups ("Base", "Accent"), not by nesting
-// ThemeMenu's DropdownMenu inside this dialog — one popover layer inside
-// another is a focus-trap fight, and the dropdown's `menuitemradio`s are a spec
-// hook that belongs to the md+ header.
+// Header is a server component, so the links arrive as plain serialisable
+// props.
export default function MobileMenu({
links,
- sites,
- hubUrl,
+ instancesUrl,
}: {
- links: { href: string; label: string }[];
- sites: SwitcherGroup[];
- hubUrl?: string;
+ // `external`: off this site (Use with AI, the homepage's AI and MCP doc), so
+ // a plain <a> in the same tab rather than a client navigation.
+ links: { href: string; label: string; external?: boolean }[];
+ // The Archilyzer home's Official Instances (common/lib/project.ts
+ // INSTANCES_URL); absent on the hub, which lists the instances itself.
+ instancesUrl?: string;
}) {
const [open, setOpen] = useState(false);
- const { base, accent, siteAccent, setBase, setAccent } = useTheme();
return (
<Sheet open={open} onOpenChange={setOpen}>
@@ -52,7 +42,7 @@ export default function MobileMenu({
variant="ghost"
size="icon"
aria-label="Open menu"
- className="md:hidden"
+ className="shrink-0 lg:hidden"
>
<MenuIcon aria-hidden="true" />
</Button>
@@ -72,141 +62,34 @@ export default function MobileMenu({
<nav className="mt-2 flex flex-col px-2">
{links.map((l) => (
<SheetClose asChild key={l.href}>
- <Link
- href={l.href}
- className="rounded-md px-2 py-3 text-base font-medium text-foreground transition-colors hover:bg-accent"
- >
- {l.label}
- </Link>
+ {l.external ? (
+ <a
+ href={l.href}
+ className="rounded-md px-2 py-3 text-base font-medium text-foreground transition-colors hover:bg-accent"
+ >
+ {l.label}
+ </a>
+ ) : (
+ <Link
+ href={l.href}
+ className="rounded-md px-2 py-3 text-base font-medium text-foreground transition-colors hover:bg-accent"
+ >
+ {l.label}
+ </Link>
+ )}
</SheetClose>
))}
- {hubUrl && (
+ {instancesUrl && (
<a
- href={hubUrl}
- rel="noopener noreferrer"
- className="flex items-center gap-1.5 rounded-md px-2 py-3 text-base font-medium text-muted-foreground transition-colors hover:bg-accent"
+ href={instancesUrl}
+ aria-label="Archilyzer — official instances"
+ className="rounded-md px-2 py-3 text-base font-medium text-muted-foreground transition-colors hover:bg-accent"
>
- <ArrowUpLeft className="size-4" aria-hidden="true" />
- Hub
+ Archilyzer
</a>
)}
</nav>
-
- {sites.length > 0 && (
- <div className="mt-4 px-2">
- <MenuHeading>Sites</MenuHeading>
- {sites.map((group, gi) => (
- <div key={group.label ?? `group-${gi}`} className="mt-1">
- {group.label && (
- <p className="px-2 py-1 font-mono text-[0.625rem] uppercase tracking-[0.16em] text-muted-foreground/70">
- {group.label}
- </p>
- )}
- <ul className="list-none">
- {group.sites.map((s) => (
- <li key={s.siteId}>
- <a
- href={s.url}
- rel="noopener noreferrer"
- className="block rounded-md px-2 py-2.5 text-sm text-foreground transition-colors hover:bg-accent"
- >
- {s.title}
- </a>
- </li>
- ))}
- </ul>
- </div>
- ))}
- </div>
- )}
-
- <div className="mt-4 px-2">
- <MenuHeading>Base</MenuHeading>
- <RadioList
- name="mobile-theme-base"
- label="Base"
- value={base}
- options={THEME_BASES}
- onPick={(v) => {
- if (isThemeBase(v)) setBase(v);
- }}
- />
- </div>
-
- <div className="mt-4 px-2">
- <MenuHeading>Accent</MenuHeading>
- <RadioList
- name="mobile-theme-accent"
- label="Accent"
- value={accent}
- options={accentOptions(siteAccent)}
- onPick={(v) => {
- if (isThemeAccent(v)) setAccent(v);
- }}
- />
- </div>
</SheetContent>
</Sheet>
);
}
-
-function MenuHeading({ children }: { children: React.ReactNode }) {
- return (
- <p className="px-2 font-mono text-xs uppercase tracking-[0.14em] text-muted-foreground">
- {children}
- </p>
- );
-}
-
-function RadioList({
- name,
- label,
- value,
- options,
- onPick,
-}: {
- name: string;
- label: string;
- value: string;
- options: ReadonlyArray<{
- id: string;
- label: string;
- // An accent's dot (a CSS colour) and whether it is the site's own.
- swatch?: string;
- isSiteDefault?: boolean;
- }>;
- onPick: (id: string) => void;
-}) {
- return (
- <div role="radiogroup" aria-label={label} className="mt-1 flex flex-col">
- {options.map((o) => (
- <label
- key={o.id}
- className="flex cursor-pointer items-center gap-2.5 rounded-md px-2 py-2.5 text-sm text-foreground transition-colors hover:bg-accent"
- >
- <input
- type="radio"
- name={name}
- value={o.id}
- checked={value === o.id}
- onChange={() => onPick(o.id)}
- className="size-4 accent-[var(--brand)]"
- />
- {o.swatch && (
- <span
- aria-hidden="true"
- className="size-3 shrink-0 rounded-full ring-1 ring-inset ring-foreground/15"
- style={{ background: o.swatch }}
- />
- )}
- {o.label}
- {o.isSiteDefault && (
- <span className="ml-auto font-mono text-[0.625rem] uppercase tracking-[0.12em] text-muted-foreground">
- default
- </span>
- )}
- </label>
- ))}
- </div>
- );
-}
diff --git a/export/app/components/SiblingSwitcher.tsx b/export/app/components/SiblingSwitcher.tsx
@@ -1,62 +0,0 @@
-"use client";
-
-import { ChevronDown } from "lucide-react";
-import { Button } from "yt-dlp-transcript-common/components/ui/button";
-import {
- DropdownMenu,
- DropdownMenuContent,
- DropdownMenuItem,
- DropdownMenuLabel,
- DropdownMenuSeparator,
- DropdownMenuTrigger,
-} from "yt-dlp-transcript-common/components/ui/dropdown-menu";
-
-// A sibling-site switcher for the export header: the "family" navigation across
-// the other public sites in the pool. Fed by the same resolveRelatedSites() data
-// the footer uses (groups of { siteId, title, url }), resolved server-side and
-// passed in as plain props so this stays a thin client shell over the kit.
-export type SwitcherSite = { siteId: string; title: string; url: string };
-export type SwitcherGroup = { label?: string; sites: SwitcherSite[] };
-
-export default function SiblingSwitcher({
- groups,
-}: {
- groups: SwitcherGroup[];
-}) {
- const total = groups.reduce((n, g) => n + g.sites.length, 0);
- if (total === 0) return null;
-
- return (
- <DropdownMenu>
- <DropdownMenuTrigger asChild>
- <Button
- variant="ghost"
- size="sm"
- className="gap-1 font-mono text-xs uppercase tracking-[0.12em] text-muted-foreground"
- >
- Sites
- <ChevronDown className="size-3.5" aria-hidden="true" />
- </Button>
- </DropdownMenuTrigger>
- <DropdownMenuContent align="end" className="min-w-48">
- {groups.map((group, gi) => (
- <div key={group.label ?? `group-${gi}`}>
- {gi > 0 && <DropdownMenuSeparator />}
- {group.label && (
- <DropdownMenuLabel className="font-mono text-[0.625rem] uppercase tracking-[0.16em] text-muted-foreground">
- {group.label}
- </DropdownMenuLabel>
- )}
- {group.sites.map((s) => (
- <DropdownMenuItem key={s.siteId} asChild>
- <a href={s.url} rel="noopener noreferrer">
- {s.title}
- </a>
- </DropdownMenuItem>
- ))}
- </div>
- ))}
- </DropdownMenuContent>
- </DropdownMenu>
- );
-}
diff --git a/export/app/components/hub/useHubSites.ts b/export/app/components/hub/useHubSites.ts
@@ -15,7 +15,7 @@
// each in its own accent or none: the hex its /site.json published (a named
// accent's on-dark value, or the site's own), FITTED to each base like any
// custom hex (lib/siteColor.ts fittedHex, release 11 O2b) — so a pale one reads
-// on light and sepia on its card, chip and result stripe. A value that is not
+// on light on its card, chip and result stripe. A value that is not
// a hex is dropped, never passed into a style.
//
// `listed` says the list is the hub's WHOLE list: `/hub-sites.json` has been
diff --git a/export/app/globals.css b/export/app/globals.css
@@ -2,11 +2,11 @@
@import "../../common/styles/tokens.css";
@source "../../common/components";
-/* Design tokens, the `dark` variant, the three bases (light / sepia / dark) and
+/* Design tokens, the `dark` variant, the two bases (light / dark) and
the accents live in common/styles/tokens.css; the faces in
common/styles/fonts.ts. A site opens on the reader's system base in its own
accent (site.json, rendered as `html[data-accent]`); the hub opens dark in
- Signal. The ThemeMenu picks any base and any accent. */
+ Signal. The header's toggle cycles the base; the accent is the site's. */
body {
background: var(--background);
diff --git a/export/app/layout.tsx b/export/app/layout.tsx
@@ -25,12 +25,13 @@ import "./globals.css";
// id, or a custom hex fitted to each ground). The hub is the TOOL, not an
// archive: it opens dark in Signal, the family's own accent, exactly as the
// homepage does (homepage/app/layout.tsx), so the two read as one product.
-// Either way the reader can pick any base and any accent (ThemeMenu).
+// Either way the reader cycles the base (ThemeToggle); the accent is the
+// site's, and a reader's stored pick from before is ignored.
// instanceMode() is server-only; this whole file is a server component.
function themeDefaults(): {
defaultBase: ThemeBase;
accent: ThemeAccent;
- // The inline `--accent-custom-light|sepia|dark` a custom-hex site needs
+ // The inline `--accent-custom-light|dark` a custom-hex site needs
// (tokens.css `--swatch-custom` reads them); null for a named accent.
accentVars: Record<string, string> | null;
} {
@@ -100,8 +101,8 @@ export default async function RootLayout({
}>) {
// The accent is baked onto <html> at prerender, so it is right on first
// paint before any script: `data-accent` selects the tokens.css rule, and a
- // custom hex adds its fitted per-base values inline. A reader's stored pick
- // replaces `data-accent` pre-paint (ThemeScript). The hub's dark base is
+ // custom hex adds its fitted per-base values inline. Nothing replaces it: a
+ // reader does not pick an accent. The hub's dark base is
// server-rendered too (`.dark` + data-base), as on the homepage; a site's
// "system" base cannot be resolved on the server, so it renders none and
// the pre-paint script sets it.
diff --git a/export/app/lib/brand.ts b/export/app/lib/brand.ts
@@ -14,10 +14,9 @@ export function iconPalette(): IconPalette {
: siteIconPalette(currentSite().accent);
}
-// The header mark's palette. On a site it is the icon's ink tile, but its lit
-// line follows the READER's accent — `--brand-mark` is the picked accent's
-// on-dark value (the tile is always ink) — while the favicon keeps the site's
-// default. tokens.css defines `--brand-mark` on every page (a Signal default on
+// The header mark's palette. On a site it is the icon's ink tile, its lit line
+// the accent in force — `--brand-mark` is that accent's on-dark value (the
+// tile is always ink), the same the favicon wears. tokens.css defines `--brand-mark` on every page (a Signal default on
// `:root`, then each `html[data-accent]` rule); the `--brand` fallback is only a
// guard. The hub's mark is the parent mark and does not follow.
export function headerMarkPalette(): BrandMarkPalette {
diff --git a/export/app/manifest.ts b/export/app/manifest.ts
@@ -23,7 +23,8 @@ export default function manifest(): MetadataRoute.Manifest {
// right shape for the player view.
// The splash is the icon's own ground, so the mark sits on its tile's
// colour; the installed app's chrome is the dark base's ground. Neither is
- // the accent: a reader may pick another, and the manifest cannot follow.
+ // the accent: the chrome is the ground a dark reader sees, whatever the
+ // site's accent.
background_color: iconPalette().ground,
theme_color: BASE_GROUNDS.dark,
icons: [
diff --git a/export/app/use-with-ai/page.tsx b/export/app/use-with-ai/page.tsx
@@ -1,139 +0,0 @@
-import type { Metadata } from "next";
-import Link from "next/link";
-import { currentSite } from "../lib/site";
-import { instanceMode } from "../lib/mode";
-import { hasArchives } from "../lib/archives";
-
-export const metadata: Metadata = { title: "Use with AI" };
-
-// A human-facing hub for the "bring your own AI" surface: the in-browser chat,
-// the machine-readable discovery files (llms.txt / corpus.json), and the MCP
-// server. Hub-aware — on a hub build it frames everything as federation-wide.
-// Fully static server component; no data fetching.
-export default function UseWithAiPage() {
- const site = currentSite();
- const isHub = instanceMode() === "hub";
- const base = site.siteUrl?.replace(/\/+$/, "") ?? "";
- const abs = (p: string) => (base ? `${base}${p}` : p);
- const scope = isHub ? "the whole federation" : "this archive";
- const envVar = isHub ? "TRANSCRIPT_HUB_URL" : "TRANSCRIPT_SITE_URL";
- const target = base || (isHub ? "https://your-hub.example" : "https://your-site.example");
-
- const mcpSnippet = `{
- "mcpServers": {
- "${site.siteId}": {
- "command": "pnpm",
- "args": [
- "-C", "/path/to/yt-dlp-transcript-browser",
- "--filter", "yt-dlp-transcript-mcp",
- "exec", "tsx", "src/index.ts"
- ],
- "env": { "${envVar}": "${target}" }
- }
- }
-}`;
-
- return (
- <div className="mx-auto flex max-w-3xl flex-col gap-10">
- <header className="flex flex-col gap-3 border-b border-border pb-6">
- <p className="font-mono text-xs uppercase tracking-[0.18em] text-brand">
- Use with AI · {site.headerTitle}
- </p>
- <h1 className="font-display text-3xl font-semibold leading-tight text-foreground">
- Ask an AI about {scope}
- </h1>
- <p className="max-w-prose text-sm text-muted-foreground">
- Nothing is hosted or paid for here — you bring your own AI. Chat in your
- browser with your own API key, or point a tool like Claude Code at the
- machine-readable index and let it browse the transcripts itself.
- </p>
- </header>
-
- <section className="flex flex-col gap-3">
- <h2 className="font-mono text-xs uppercase tracking-[0.14em] text-muted-foreground">
- Chat in your browser
- </h2>
- <div className="flex flex-col gap-3 rounded-lg border border-border bg-card/40 p-5">
- <p className="text-sm text-muted-foreground">
- A retrieval-augmented chat that searches the transcripts and answers
- with citations. It runs entirely in your browser and calls your own
- provider (Anthropic, OpenAI, or Google Gemini) with a key you supply —
- the key stays on your device and requests go straight to the provider.
- </p>
- <div>
- <Link
- href="/ask"
- className="inline-flex items-center gap-2 rounded-md bg-primary px-4 py-2 text-sm font-medium text-primary-foreground transition-colors hover:bg-brand-strong"
- >
- Open the chat →
- </Link>
- </div>
- </div>
- </section>
-
- <section className="flex flex-col gap-3">
- <h2 className="font-mono text-xs uppercase tracking-[0.14em] text-muted-foreground">
- For coding agents & LLM tools
- </h2>
- <div className="flex flex-col gap-3 rounded-lg border border-border bg-card/40 p-5">
- <p className="text-sm text-muted-foreground">
- {scope[0].toUpperCase() + scope.slice(1)} publishes a small, fixed set
- of discovery files. An agent (e.g. Claude Code via <code className="font-mono">WebFetch</code>)
- can read these and navigate every transcript without any per-video
- pages — the paginated shard scheme is documented inline.
- </p>
- <ul className="flex flex-col gap-2 text-sm">
- <li>
- <a href={abs("/llms.txt")} className="font-mono text-brand hover:underline">
- /llms.txt
- </a>
- <span className="text-muted-foreground"> — an LLM-readable overview and link map.</span>
- </li>
- <li>
- <a href={abs("/corpus.json")} className="font-mono text-brand hover:underline">
- /corpus.json
- </a>
- <span className="text-muted-foreground">
- {" "}— the channel index and exactly how to fetch any transcript
- from the JSON shards{isHub ? " across every member site" : ""}.
- </span>
- </li>
- </ul>
- </div>
- </section>
-
- <section className="flex flex-col gap-3">
- <h2 className="font-mono text-xs uppercase tracking-[0.14em] text-muted-foreground">
- MCP server
- </h2>
- <div className="flex flex-col gap-3 rounded-lg border border-border bg-card/40 p-5">
- <p className="text-sm text-muted-foreground">
- The repo ships an MCP server that exposes {scope} to Claude Code,
- Claude Desktop, Cursor, and other MCP clients as tools
- (<code className="font-mono">search_transcripts</code>,
- {" "}<code className="font-mono">get_transcript</code>, …). It reads the
- same static shards — over HTTP or from a local build
- {isHub ? ", federating every member site" : ""}. Add it to a client:
- </p>
- <pre className="overflow-x-auto rounded-md border border-border bg-muted/50 p-3 font-mono text-xs text-foreground">
- <code>{mcpSnippet}</code>
- </pre>
- <p className="text-xs text-muted-foreground/80">
- Setup details and the <code className="font-mono">claude mcp add</code>{" "}
- command are in <code className="font-mono">mcp/README.md</code>.
- </p>
- </div>
- </section>
-
- {hasArchives() && (
- <p className="text-xs text-muted-foreground/70">
- Ingesting in bulk instead? The{" "}
- <a href="/downloads" className="text-brand hover:underline">
- Downloads page
- </a>{" "}
- has whole-channel transcript zips.
- </p>
- )}
- </div>
- );
-}
diff --git a/export/e2e-hub/brand.spec.ts b/export/e2e-hub/brand.spec.ts
@@ -52,10 +52,10 @@ test("the hub header is the parent mark + Archi|lyzer", async ({ page }) => {
// The ring on dark (release 10, slice MR): on the dark base the tile has no
// edge, so every mark's tile gets a 1px ring outside it in its palette's dim,
// following the ground's corner (rx 112 of 512). A box-shadow, so the mark's
-// box is the same on every base; light and sepia draw none.
+// box is the same on every base; light draws none.
const BASE_KEY = "ytdlp-tb:base";
-async function onBase(page: Page, base: "light" | "sepia" | "dark") {
+async function onBase(page: Page, base: "light" | "dark") {
await page.evaluate(([k, b]) => localStorage.setItem(k, b), [BASE_KEY, base]);
await page.reload({ waitUntil: "commit" });
await page.waitForFunction(() => document.documentElement?.dataset.themeReady === "1");
@@ -79,7 +79,7 @@ async function markRing(mark: Locator) {
});
}
-test("on dark the hub's header and footer marks have a 1px ring in the slate's dim; on light and sepia none", async ({
+test("on dark the hub's header and footer marks have a 1px ring in the slate's dim; on light none", async ({
page,
}) => {
await page.route("**/hub-sites.json", (r) => fulfillJson(r, []));
@@ -93,7 +93,7 @@ test("on dark the hub's header and footer marks have a 1px ring in the slate's d
expect(await markRing(header)).toEqual({ ring, radius: "21.875%", size: [28, 28] });
expect(await markRing(footer)).toEqual({ ring, radius: "21.875%", size: [16, 16] });
- for (const base of ["light", "sepia"] as const) {
+ for (const base of ["light"] as const) {
await onBase(page, base);
expect(await markRing(header), base).toEqual({ ring: [], radius: "21.875%", size: [28, 28] });
expect(await markRing(footer), base).toEqual({ ring: [], radius: "21.875%", size: [16, 16] });
diff --git a/export/e2e-hub/federated-search.spec.ts b/export/e2e-hub/federated-search.spec.ts
@@ -4,6 +4,7 @@ import {
LIVE_CHAT_MISSING,
NO_ARCHIVES_IN_SCOPE,
} from "../app/ask/hubScopeCopy";
+import { showAll } from "../e2e/helpers";
// The hub's federated search, per archive: two official members (hub-sites.json)
// served by route mocks WITH CORS, each with one video. Proves the scope chips
@@ -26,6 +27,10 @@ import {
// later; a member whose live chat cannot be read says so on its chip
// (LIVE_CHAT_MISSING), with a Retry for the live chat alone that keeps its
// videos in the search while it runs.
+// Release 14 (S1): the results — the listing and the "N of M archives
+// answered" line with it — show only after the first Search of the page life,
+// so a spec that reads them presses Search first (`showAll`); the chips carry
+// each archive's state before that.
const ORIGIN_A = "http://localhost:4598";
const ORIGIN_B = "http://localhost:4599";
@@ -199,6 +204,14 @@ test.describe("hub federated search — scope, per-archive state, attribution",
"aria-pressed",
"true",
);
+ // Both archives are in, and nothing has been asked: no results area yet,
+ // and the bar says what to do.
+ await expect(page.getByTestId("results-summary")).toHaveCount(0);
+ await expect(resultFrom(page, A)).toHaveCount(0);
+ await expect(resultFrom(page, B)).toHaveCount(0);
+ await expect(page.getByText("Press Enter or click Search to apply")).toBeVisible();
+ await showAll(page);
+ await expect(page.getByTestId("results-summary")).toHaveText("All videos (2)");
// Browse listing: one card per archive, each naming its source in text.
const b = resultFrom(page, B);
await expect(b).toHaveCount(1);
@@ -216,6 +229,7 @@ test.describe("hub federated search — scope, per-archive state, attribution",
}) => {
const mocks = await setup(page);
await page.goto("/");
+ await showAll(page);
await expect(resultFrom(page, B)).toHaveCount(1);
await expect(resultFrom(page, A)).toHaveCount(1);
@@ -234,6 +248,7 @@ test.describe("hub federated search — scope, per-archive state, attribution",
await page.reload();
await expect(chip(page, A)).toHaveAttribute("data-status", "ready");
await expect(chip(page, B)).toHaveAttribute("data-status", "off");
+ await showAll(page);
await expect(resultFrom(page, A)).toHaveCount(1);
await expect(resultFrom(page, B)).toHaveCount(0);
expect(mocks.b.requests).toEqual([]);
@@ -251,6 +266,7 @@ test.describe("hub federated search — scope, per-archive state, attribution",
const mocks = await setup(page);
mocks.b.pages = "abort";
await page.goto("/");
+ await showAll(page);
await expect(chip(page, B)).toHaveAttribute("data-status", "failed");
await expect(chip(page, B)).toContainText("failed");
@@ -314,6 +330,7 @@ test.describe("hub federated search — scope, per-archive state, attribution",
mocks.b.pages = "abort";
mocks.c!.pages = "hold";
await page.goto("/");
+ await showAll(page);
await expect(chip(page, B)).toHaveAttribute("data-status", "failed");
await expect(chip(page, C)).toHaveAttribute("data-status", "loading");
@@ -335,6 +352,7 @@ test.describe("hub federated search — scope, per-archive state, attribution",
const mocks = await setup(page);
mocks.b.subs = "missing";
await page.goto("/");
+ await showAll(page);
// Ready means every feed of the archive is in, subs included. A 404 is an
// empty subs manifest, at once. It used to be an error, retried after ~1 s,
@@ -354,6 +372,7 @@ test.describe("hub federated search — scope, per-archive state, attribution",
// transcripts first) lists B then A. Neither sets an accent.
await setup(page, [A, B], { summary: [B, A] });
await page.goto("/");
+ await showAll(page);
const cards = page.getByTestId("shelf-spine");
await expect(cards).toHaveCount(2);
@@ -521,6 +540,7 @@ test.describe("hub federated search — scope, per-archive state, attribution",
const mocks = await setup(page);
mocks.b.subs = "error";
await page.goto("/");
+ await showAll(page);
// Live chat is the auxiliary layer: after one retry the error counts as
// settled, so Origin B is ready — its videos searched, without live chat —
@@ -544,6 +564,7 @@ test.describe("hub federated search — scope, per-archive state, attribution",
const mocks = await setup(page);
mocks.b.subs = "error";
await page.goto("/");
+ await showAll(page);
await expect(chip(page, B)).toHaveAttribute("data-status", "ready");
await expect(resultFrom(page, B)).toHaveCount(1);
diff --git a/export/e2e-hub/official-instances.spec.ts b/export/e2e-hub/official-instances.spec.ts
@@ -1,4 +1,7 @@
+import fs from "node:fs";
+import path from "node:path";
import { expect, test, type Page, type Route } from "@playwright/test";
+import { normalizeSocialSvg } from "../../common/lib/settingsSchema";
import { resolveAccent } from "../../common/lib/accent";
import { ACCENTS } from "../../common/lib/brand";
@@ -119,6 +122,69 @@ test.describe("hub official instances", () => {
).toHaveCount(0);
});
+ // The hub lists the instances itself: its header has the group (the theme
+ // toggle; the social row when there are links) and no link to the
+ // Archilyzer home's list, no sites menu and no hub link.
+ test("the hub's header has no Archilyzer link, sites menu or theme menu", async ({ page }) => {
+ await stubMember(page);
+ await page.route("**/hub-summary.json", (r) =>
+ r.fulfill({ status: 404, body: "" }),
+ );
+ await page.goto("/");
+ const banner = page.getByRole("banner");
+ await expect(banner.getByRole("button", { name: /^switch to /i })).toBeVisible();
+ await expect(banner.getByRole("link", { name: /official instances/i })).toHaveCount(0);
+ await expect(banner.getByRole("button", { name: /sites/i })).toHaveCount(0);
+ await expect(banner.getByRole("button", { name: "Choose theme" })).toHaveCount(0);
+ await expect(banner.getByRole("link", { name: "Changelog" })).toHaveCount(0);
+ });
+
+ // The hub's header follows the same ruling as every site's: on a narrow
+ // screen the name and only the marked link(s); from 520 px every link. The
+ // hub's links come from the settings file the suite serves; this test writes
+ // three (the last marked) and puts the file back. Both pointers: under
+ // touch the keys are 44 px, and the name still shows at 360 px.
+ for (const touch of [false, true]) {
+ test.describe(touch ? "under a coarse pointer" : "with a mouse", () => {
+ test.use({ hasTouch: touch });
+ test("the hub's header keeps its name at 360 and 390 px and shows only the marked link", async ({ page }) => {
+ const file = path.resolve(process.cwd(), "test-settings.hub.json");
+ const pristine = fs.readFileSync(file, "utf8");
+ try {
+ const icon = (d: string) => normalizeSocialSvg(`<svg viewBox="0 0 24 24"><path d="${d}"/></svg>`)!;
+ const links = [
+ { label: "Square", url: "https://square.example/", svg: icon("M4 4h16v16H4z") },
+ { label: "Bar", url: "https://bar.example/", svg: icon("M3 10h18v4H3z") },
+ { label: "Post", url: "https://post.example/", svg: icon("M10 2h4v20h-4z"), featured: true },
+ ];
+ fs.writeFileSync(file, JSON.stringify({ ...JSON.parse(pristine), socialLinks: links }, null, 2));
+ await stubMember(page);
+ await page.route("**/hub-summary.json", (r) => r.fulfill({ status: 404, body: "" }));
+ const banner = page.getByRole("banner");
+ const row = page.locator('header [data-social-links="header"]:visible');
+ for (const width of [360, 390]) {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ await expect(banner.getByRole("button", { name: /^switch to /i })).toBeVisible();
+ const text = await page.locator("header [data-header-brand] span.flex-wrap").evaluate(
+ (box) => (box.lastElementChild as HTMLElement).offsetTop < box.clientHeight - 1,
+ );
+ expect(text, `${width} px: the name`).toBe(true);
+ await expect(row.getByRole("link")).toHaveCount(1);
+ await expect(banner.getByRole("link", { name: "Post", exact: true })).toHaveCount(1);
+ await expect(banner.getByRole("link", { name: "Square", exact: true })).toHaveCount(0);
+ }
+ await page.setViewportSize({ width: 1280, height: 800 });
+ await page.goto("/");
+ await expect(row.getByRole("link")).toHaveCount(3);
+ await expect(page.locator('footer [data-social-links="footer"]').getByRole("link")).toHaveCount(3);
+ } finally {
+ fs.writeFileSync(file, pristine);
+ }
+ });
+ });
+ }
+
test("the hub has no link to ko-fi.com", async ({ page }) => {
await stubMember(page);
await page.route("**/hub-summary.json", (r) =>
@@ -179,20 +245,16 @@ test.describe("hub official instances", () => {
await expect(page.locator("html")).toHaveAttribute("data-base", "dark");
await expect.poll(colour).toBe(rgb(ACCENTS.brass.onDark));
- for (const base of ["light", "sepia"] as const) {
- await page.evaluate((b) => localStorage.setItem("ytdlp-tb:base", b), base);
- await page.reload();
- await expect(page.locator("html")).toHaveAttribute("data-base", base);
- await expect
- .poll(colour)
- .toBe(rgb(base === "light" ? ACCENTS.brass.onLight : ACCENTS.brass.onSepia));
- }
+ await page.evaluate(() => localStorage.setItem("ytdlp-tb:base", "light"));
+ await page.reload();
+ await expect(page.locator("html")).toHaveAttribute("data-base", "light");
+ await expect.poll(colour).toBe(rgb(ACCENTS.brass.onLight));
});
// Release 11 (O2b): an archive the VISITOR added wears the hex its
// /site.json published, fitted to the base in force — on its card and on its
// scope chip — like any custom hex. A pale one used to be painted as is:
- // ~1.4:1 on the light and sepia grounds.
+ // ~1.4:1 on the light ground.
test("an added archive's own hex is fitted to each base, on its card and its chip", async ({
page,
}) => {
@@ -240,13 +302,12 @@ test.describe("hub official instances", () => {
const fitted = resolveAccent(PALE);
// Fitted, not as published, where the ground needs it.
expect(fitted.light).not.toBe(PALE);
- expect(fitted.sepia).not.toBe(PALE);
await page.goto("/");
await expect(
page.getByRole("heading", { level: 2, name: "Archives You Added", exact: true }),
).toBeVisible();
- for (const base of ["dark", "light", "sepia"] as const) {
+ for (const base of ["dark", "light"] as const) {
if (base !== "dark") {
await page.evaluate((b) => localStorage.setItem("ytdlp-tb:base", b), base);
await page.reload();
diff --git a/export/e2e-hub/use-with-ai-link.spec.ts b/export/e2e-hub/use-with-ai-link.spec.ts
@@ -0,0 +1,48 @@
+import { expect, test, type Locator, type Route } from "@playwright/test";
+import { AI_DOC_URL } from "../../common/lib/project";
+
+// Release 16 slice DX, on the hub: no /use-with-ai page; the header's, the
+// footer's and Ask AI's "Use with AI" go to the homepage's AI and MCP doc
+// (AI_DOC_URL), in the same tab, as plain anchors — the same links a site has.
+
+async function fulfillJson(route: Route, body: unknown) {
+ await route.fulfill({
+ status: 200,
+ contentType: "application/json",
+ headers: { "access-control-allow-origin": "*" },
+ body: JSON.stringify(body),
+ });
+}
+
+async function expectDocLink(link: Locator) {
+ await expect(link).toHaveAttribute("href", AI_DOC_URL);
+ await expect(link).not.toHaveAttribute("target", /./);
+}
+
+test.beforeEach(async ({ page }) => {
+ // No members: the shelf stays empty and nothing reaches a real origin.
+ await page.route("**/hub-sites.json", (r) => fulfillJson(r, []));
+ await page.route("**/hub-summary.json", (r) => r.fulfill({ status: 404, body: "" }));
+});
+
+test("the hub's header and footer Use with AI go to the homepage doc", async ({ page }) => {
+ await page.goto("/");
+ await expectDocLink(
+ page.getByRole("banner").getByRole("link", { name: "Use with AI", exact: true }),
+ );
+ await expectDocLink(
+ page.getByRole("contentinfo").getByRole("link", { name: "Use with AI", exact: true }),
+ );
+});
+
+test("the hub's Ask AI links Use with AI to the homepage doc", async ({ page }) => {
+ await page.goto("/ask/");
+ await expectDocLink(
+ page.getByRole("main").getByRole("link", { name: "Use with AI", exact: true }),
+ );
+});
+
+test("the hub has no /use-with-ai page", async ({ page }) => {
+ const res = await page.request.get("/use-with-ai/");
+ expect(res.status()).toBe(404);
+});
diff --git a/export/e2e/brand.spec.ts b/export/e2e/brand.spec.ts
@@ -9,7 +9,7 @@ import { installRoutes } from "./helpers";
// site's accent. The fixture site (e2e/fixtures/sites/testsite/site.json) is
// headerTitle "Fixture Header", wordmarkLead "Fixture", accent #cc3366.
-// The computed fill of each part of a BrandMark, plus what the reader's accent
+// The computed fill of each part of a BrandMark, plus what the site's accent
// resolves to — so the lit line can be checked against the token it follows
// without pinning a colour the themes slice will change.
async function markFills(mark: Locator) {
@@ -59,7 +59,7 @@ test("the header link is the mark + the split wordmark, named by the header titl
await expect(mark).toBeVisible();
});
-test("the header mark is the ink tile, its lit line following the reader's accent", async ({
+test("the header mark is the ink tile, its lit line following the site's accent", async ({
page,
}) => {
await installRoutes(page);
@@ -75,10 +75,10 @@ test("the header mark is the ink tile, its lit line following the reader's accen
// The ring on dark (release 10, slice MR): on the dark base a site's ink tile
// IS the page, so every mark's tile gets a 1px ring outside it in its own
// palette's dim, following the ground's corner (rx 112 of 512). A box-shadow,
-// so the mark's box is the same on every base; light and sepia draw none.
+// so the mark's box is the same on every base; light draws none.
const BASE_KEY = "ytdlp-tb:base";
-async function onBase(page: Page, base: "light" | "sepia" | "dark") {
+async function onBase(page: Page, base: "light" | "dark") {
await page.evaluate(([k, b]) => localStorage.setItem(k, b), [BASE_KEY, base]);
await page.reload({ waitUntil: "commit" });
await page.waitForFunction(() => document.documentElement?.dataset.themeReady === "1");
@@ -102,7 +102,7 @@ async function markRing(mark: Locator) {
});
}
-test("on dark the header and footer marks' tiles have a 1px ring in their dim; on light and sepia none", async ({
+test("on dark the header and footer marks' tiles have a 1px ring in their dim; on light none", async ({
page,
}) => {
await installRoutes(page);
@@ -122,7 +122,7 @@ test("on dark the header and footer marks' tiles have a 1px ring in their dim; o
size: [16, 16],
});
- for (const base of ["light", "sepia"] as const) {
+ for (const base of ["light"] as const) {
await onBase(page, base);
expect(await markRing(header), base).toEqual({ ring: [], radius: "21.875%", size: [28, 28] });
expect(await markRing(footer), base).toEqual({ ring: [], radius: "21.875%", size: [16, 16] });
diff --git a/export/e2e/browse-all.spec.ts b/export/e2e/browse-all.spec.ts
@@ -5,11 +5,12 @@ import {
VIDEO_CHAT_SMALL,
VIDEO_TRANSCRIPT_ONLY,
} from "./fixtures/data";
-import { installRoutes } from "./helpers";
+import { installRoutes, showAll } from "./helpers";
// Browse-all e2e: with no search query active the results section lists every
-// video passing the committed filters (no hits). Typing a query narrows it;
-// clearing it returns to the full browse list.
+// video passing the committed filters (no hits) — once the visitor has pressed
+// Search; until then the page shows no results area at all (first-search.spec).
+// Typing a query narrows it; clearing it returns to the full browse list.
const TRANSCRIPT_ONLY_SLUG = `${CHANNEL_SLUG}/${VIDEO_TRANSCRIPT_ONLY}`;
const CHAT_SMALL_SLUG = `${CHANNEL_SLUG}/${VIDEO_CHAT_SMALL}`;
@@ -31,8 +32,20 @@ test.describe("browse all (no query)", () => {
await installRoutes(page);
});
- test("lists every video on load with no query", async ({ page }) => {
+ test("a clear screen on load, then every video on an empty Search", async ({
+ page,
+ }) => {
await page.goto("/");
+ await page.getByTestId("query-builder").waitFor();
+ // Give the session time to hydrate (it waits for the manifest) and to
+ // show a listing if it were going to.
+ await page.waitForTimeout(1_000);
+ await expect(page.getByTestId("results-section")).toHaveCount(0);
+ await expect(page.getByTestId("results-summary")).toHaveCount(0);
+ await expect(page.getByTestId("browse-hint")).toHaveCount(0);
+ await expect(page.locator("[data-card-header]")).toHaveCount(0);
+
+ await page.getByTestId("search-submit").click();
await expect(page.getByTestId("results-section")).toBeVisible();
await expect(page.getByTestId("results-summary")).toHaveText(
"All videos (3)",
@@ -43,6 +56,7 @@ test.describe("browse all (no query)", () => {
test("filters apply to the browse list on Search", async ({ page }) => {
await page.goto("/");
+ await showAll(page);
await expectResultSlugs(page, ALL_SLUGS);
// All fixture videos are non-livestream, so unchecking "Videos" excludes
@@ -62,6 +76,7 @@ test.describe("browse all (no query)", () => {
page,
}) => {
await page.goto("/");
+ await showAll(page);
await expectResultSlugs(page, ALL_SLUGS);
const leafInput = page.locator('input[data-testid^="leaf-query-"]').first();
diff --git a/export/e2e/charts.spec.ts b/export/e2e/charts.spec.ts
@@ -1,5 +1,6 @@
import { expect, test, type Page } from "@playwright/test";
-import { installChartRoutes, urlParams } from "./helpers";
+import { installChartRoutes, showAll, urlParams } from "./helpers";
+import { over, painted, rgbOf } from "../../common/testing/chartPixels";
// Charts are now a VIEW MODE of the search page: a "Results | Chart" toggle
// plots the current search/filters as a single chart. Stats, summaries and
@@ -16,6 +17,17 @@ async function runSearch(page: Page, term: string) {
await expect(page).toHaveURL(/[?&]qt=/);
}
+// The chart card's colour: what is behind the plot.
+function cardColour(page: Page) {
+ return page.locator(".recharts-surface").first().evaluate((el) => {
+ for (let n: Element | null = el; n; n = n.parentElement) {
+ const bg = getComputedStyle(n).backgroundColor;
+ if (bg !== "rgba(0, 0, 0, 0)" && bg !== "transparent") return bg;
+ }
+ return "";
+ });
+}
+
function chartTab(page: Page) {
return page.getByTestId("view-toggle").getByRole("button", { name: "Chart" });
}
@@ -62,6 +74,7 @@ test.describe("charts (search view mode)", () => {
test("Chart toggle with no query plots metadata", async ({ page }) => {
await page.goto("/");
+ await showAll(page);
await expect(page.getByTestId("results-summary")).toContainText("All videos");
await openChart(page);
await expect(page.locator(".recharts-surface")).toBeVisible({
@@ -94,6 +107,7 @@ test.describe("charts (search view mode)", () => {
page,
}) => {
await page.goto("/");
+ await showAll(page);
await expect(page.getByTestId("results-summary")).toContainText("All videos");
await openChart(page);
const opts = page.getByTestId("chart-options");
@@ -110,6 +124,7 @@ test.describe("charts (search view mode)", () => {
page,
}) => {
await page.goto("/");
+ await showAll(page);
await expect(page.getByTestId("results-summary")).toContainText("All videos");
await openChart(page);
const opts = page.getByTestId("chart-options");
@@ -138,6 +153,7 @@ test.describe("charts (search view mode)", () => {
page,
}) => {
await page.goto("/");
+ await showAll(page);
await expect(page.getByTestId("results-summary")).toContainText("All videos");
await openChart(page);
await expect(page.locator(".recharts-surface")).toBeVisible({
@@ -210,4 +226,110 @@ test.describe("charts (search view mode)", () => {
page.getByTestId("chart-options").locator("select").first(),
).toHaveValue("bar");
});
+ // THE SURFACE GAP (common/components/charts/surfaceGap.ts): the segments of
+ // a stacked bar are parted by a 2 px stroke in the colour behind the plot —
+ // the chart card — never by a line of their own. Stacked AREAS keep their
+ // series-coloured top edge (a surface stroke there erased small values and
+ // cut peaks).
+ test("stacked bars are parted by a gap in the card's colour; stacked areas keep their own edge", async ({
+ page,
+ }) => {
+ await page.goto("/");
+ await showAll(page);
+ await expect(page.getByTestId("results-summary")).toContainText("All videos");
+ await openChart(page);
+ const opts = page.getByTestId("chart-options");
+ await opts.locator('label:has-text("Chart type") select').selectOption("stackedBar");
+ await opts.locator('label:has-text("Group into series by") select').selectOption("mediaType");
+ const bars = page.locator(".recharts-bar-rectangle path");
+ await expect(bars.first()).toBeVisible({ timeout: 20_000 });
+ const surface = await cardColour(page);
+ const text = await page.evaluate(() => getComputedStyle(document.body).color);
+ expect(surface).not.toBe(text);
+ const strokes = (sel: string) =>
+ page.locator(sel).evaluateAll((els) =>
+ els.map((e) => [getComputedStyle(e).stroke, getComputedStyle(e).strokeWidth]),
+ );
+ const barStrokes = await strokes(".recharts-bar-rectangle path");
+ expect(barStrokes.length).toBeGreaterThan(0);
+ for (const s of barStrokes) expect(s).toEqual([surface, "2px"]);
+
+ await opts.locator('label:has-text("Chart type") select').selectOption("area");
+ await expect(page.locator(".recharts-area-curve").first()).toBeAttached({ timeout: 20_000 });
+ const edges = await page.locator(".recharts-area").evaluateAll((els) =>
+ els.map((g) => [
+ getComputedStyle(g.querySelector(".recharts-area-curve")!).stroke,
+ getComputedStyle(g.querySelector(".recharts-area-area")!).fill,
+ ]),
+ );
+ expect(edges.length).toBeGreaterThan(1);
+ for (const [stroke, fill] of edges) {
+ expect(stroke).toBe(fill);
+ expect(stroke).not.toBe(surface);
+ }
+ });
+
+ // THE DATA IS WHAT IS PAINTED. Read back from a screenshot: at each month
+ // the stack's topmost painted row is within 1 px of the value scale's y for
+ // the month's true total (three fixture videos, one per month, so a total of
+ // 1 each), and every series with data shows pixels of its own fill.
+ for (const width of [390, 1280]) {
+ test(`${width} px: a stacked area paints its true total and every band`, async ({ page }) => {
+ await page.setViewportSize({ width, height: 1000 });
+ await page.goto("/");
+ await showAll(page);
+ await expect(page.getByTestId("results-summary")).toContainText("All videos");
+ await openChart(page);
+ const opts = page.getByTestId("chart-options");
+ // Collapsed on a phone.
+ if (!(await opts.evaluate((el) => (el as HTMLDetailsElement).open))) {
+ await opts.locator("summary").click();
+ }
+ await opts.locator('label:has-text("Chart type") select').selectOption("area");
+ await opts.locator('label:has-text("Group into series by") select').selectOption("mediaType");
+ const svg = page.locator(".recharts-surface").first();
+ await expect(page.locator(".recharts-area")).toHaveCount(3, { timeout: 20_000 });
+ // No tooltip or active dot over the plot, and recharts' entry animation done.
+ await page.mouse.move(0, 0);
+ await expect(page.locator(".recharts-tooltip-wrapper")).toBeHidden();
+ await page.waitForTimeout(1_600);
+ const geo = await svg.evaluate((el) => {
+ const num = (t: Element) => Number((t.textContent ?? "").replace(/[^\d.-]/g, ""));
+ // The value scale: each tick's value, at its gridline's y (a tick
+ // label sits a pixel off its line).
+ const values = [...el.querySelectorAll(".recharts-yAxis .recharts-cartesian-axis-tick-value")]
+ .map(num)
+ .sort((p, q) => p - q);
+ const lines = [...el.querySelectorAll(".recharts-cartesian-grid-horizontal line")]
+ .map((l) => Number(l.getAttribute("y1")))
+ .sort((p, q) => q - p);
+ const yTicks = values.map((v, i) => ({ v, y: lines[i] }));
+ const xTicks = [...el.querySelectorAll(".recharts-xAxis .recharts-cartesian-axis-tick-value")].map((t) =>
+ Number(t.getAttribute("x")),
+ );
+ const fills = [...el.querySelectorAll(".recharts-area-area")].map((p) => {
+ const cs = getComputedStyle(p);
+ return { fill: cs.fill, opacity: Number(cs.fillOpacity) };
+ });
+ return { yTicks, xTicks, fills };
+ });
+ const [a, b] = [geo.yTicks[0], geo.yTicks[geo.yTicks.length - 1]];
+ const yOf = (v: number) => a.y + ((v - a.v) * (b.y - a.y)) / (b.v - a.v);
+ const ground = rgbOf(await cardColour(page));
+ const shot = await painted(page, svg, {
+ columns: geo.xTicks,
+ ground,
+ colours: geo.fills.map((f) => over(rgbOf(f.fill), f.opacity, ground)),
+ tol: 10,
+ });
+ expect(geo.xTicks.length).toBe(3);
+ for (const [i, top] of shot.tops.entries()) {
+ expect(top, `month ${i}: nothing painted`).not.toBeNull();
+ expect(Math.abs(top! - yOf(1)), `month ${i}: top ${top} vs ${yOf(1).toFixed(1)}`).toBeLessThanOrEqual(1);
+ }
+ for (const [i, c] of shot.counts.entries()) {
+ expect(c.columns, `series ${i}'s own colour`).toBeGreaterThan(0);
+ }
+ });
+ }
});
diff --git a/export/e2e/first-search.spec.ts b/export/e2e/first-search.spec.ts
@@ -0,0 +1,208 @@
+import { expect, test, type Page } from "@playwright/test";
+import {
+ CHANNEL,
+ CHANNEL_SLUG,
+ TAG_TOPIC,
+ VIDEO_CHAT_LARGE,
+ VIDEO_CHAT_SMALL,
+ VIDEO_TRANSCRIPT_ONLY,
+} from "./fixtures/data";
+import { installRoutes, installTagRoutes, openFilters, showAll } from "./helpers";
+
+// Release 14 (S1): a clear screen until the first Search. A plain visit shows
+// the search bar, the page's intro and the footer — no count, no controls, no
+// listing — and the bar's line says what to do. The first Search of the page
+// life (Enter, the button, a Filters Apply, a profile load) shows the results;
+// an empty one lists every video, exactly as before. A link that carries a
+// query (`qt=`) or a filter (`tg=`, `ch=`, a share link) is the visitor asking
+// and shows its results on load. The gate is once per page life: client-side
+// navigation keeps the listing, a reload clears it.
+
+const STORAGE_KEY = "ytdlp-tb:export-filters";
+const HINT = "Press Enter or click Search to apply";
+
+const ALPHA_TREE = { k: "g", o: "AND", c: [{ k: "l", q: "alpha", s: "transcripts" }] };
+const qt = (tree: unknown) => encodeURIComponent(JSON.stringify(tree));
+
+const leafInput = (page: Page) =>
+ page.locator('input[data-testid^="leaf-query-"]').first();
+
+const card = (page: Page, id: string) =>
+ page.locator(`[data-result-slug="${CHANNEL_SLUG}/${id}"]`);
+
+// The bar has mounted; then give the session time to hydrate (it waits for the
+// manifest) and to show a listing, were it going to.
+async function settle(page: Page) {
+ await page.getByTestId("query-builder").waitFor();
+ await page.waitForTimeout(1_000);
+}
+
+async function expectClearScreen(page: Page) {
+ await expect(page.getByTestId("results-section")).toHaveCount(0);
+ await expect(page.getByTestId("results-summary")).toHaveCount(0);
+ await expect(page.getByTestId("view-toggle")).toHaveCount(0);
+ await expect(page.getByTestId("selection-toolbar")).toHaveCount(0);
+ await expect(page.getByTestId("browse-hint")).toHaveCount(0);
+ await expect(page.locator("[data-card-header]")).toHaveCount(0);
+ await expect(page.getByText(HINT)).toBeVisible();
+}
+
+// A route outside the workspace, reached as a client navigation in the same
+// page life: the search session unmounts with the workspace and mounts again on
+// Back. Until release 16 the header's Use with AI link made that navigation;
+// it now leaves the site, and the footer's Changelog is a plain <a> (a new page
+// life), so the spec asks the app router itself — `window.next.router`, which
+// Next sets for debugging in development and production alike.
+async function leaveForChangelog(page: Page) {
+ await page.evaluate(() => {
+ (
+ window as unknown as { next: { router: { push(href: string): void } } }
+ ).next.router.push("/changelog/");
+ });
+ // A dev server compiles /changelog on its first visit.
+ await expect(page).toHaveURL(/\/changelog\/?$/, { timeout: 20_000 });
+ await expect(page.getByRole("heading", { level: 1, name: "Changelog" })).toBeVisible();
+ await expect(page.getByTestId("query-builder")).toHaveCount(0);
+}
+
+async function expectAllVideos(page: Page) {
+ await expect(page.getByTestId("results-summary")).toHaveText("All videos (3)");
+ await expect(page.getByTestId("browse-hint")).toBeVisible();
+ await expect(page.locator("[data-card-header]")).toHaveCount(3);
+}
+
+test.describe("a clear screen until the first Search", () => {
+ test.beforeEach(async ({ page }) => {
+ await installRoutes(page);
+ });
+
+ for (const { width, height } of [
+ { width: 1280, height: 800 },
+ { width: 390, height: 844 },
+ ]) {
+ test(`${width}×${height}: on load, the bar, the intro and the footer — no results`, async ({
+ page,
+ }) => {
+ await page.setViewportSize({ width, height });
+ await page.goto("/");
+ await settle(page);
+ await expectClearScreen(page);
+ // The page's intro stays: the transcript count.
+ await expect(page.getByRole("heading", { level: 1 })).toContainText("transcripts");
+ // The footer is on the first screen, whole.
+ await expect(page.getByRole("contentinfo")).toBeInViewport({ ratio: 1 });
+ });
+ }
+
+ test("Enter on an empty box shows every video", async ({ page }) => {
+ await page.goto("/");
+ await settle(page);
+ await expectClearScreen(page);
+ await leafInput(page).press("Enter");
+ await expectAllVideos(page);
+ // The line has done its job.
+ await expect(page.getByText(HINT)).toHaveCount(0);
+ });
+
+ test("the Search button shows every video", async ({ page }) => {
+ await page.goto("/");
+ await settle(page);
+ await page.getByTestId("search-submit").click();
+ await expectAllVideos(page);
+ });
+
+ test("changing a filter before the first Search does not show the listing", async ({
+ page,
+ }) => {
+ await page.goto("/");
+ await settle(page);
+ await openFilters(page);
+ await page.getByRole("checkbox", { name: "Videos" }).uncheck();
+ await expectClearScreen(page);
+ await page.getByTestId("search-submit").click();
+ await expect(page.getByTestId("results-summary")).toHaveText("All videos (0)");
+ });
+
+ test("/ → /ask → / keeps the listing", async ({ page }) => {
+ await page.goto("/");
+ await showAll(page);
+ await expectAllVideos(page);
+ const nav = page.getByTestId("workspace-nav");
+ await nav.getByRole("link", { name: "Chat" }).click();
+ // A dev server compiles /ask on its first visit.
+ await expect(page).toHaveURL(/\/ask\/?$/, { timeout: 20_000 });
+ await expect(page.getByPlaceholder(/Ask about the transcripts/)).toBeVisible();
+ await nav.getByRole("link", { name: "Search" }).click();
+ await expect(page).not.toHaveURL(/\/ask/);
+ await expectAllVideos(page);
+ });
+
+ test("leaving the search page and coming Back keeps the listing", async ({ page }) => {
+ await page.goto("/");
+ await showAll(page);
+ await expectAllVideos(page);
+ // A route outside the workspace: the search session unmounts with it.
+ await leaveForChangelog(page);
+ await page.goBack();
+ await expect(page).not.toHaveURL(/changelog/);
+ await expectAllVideos(page);
+ });
+
+ test("a reload clears it", async ({ page }) => {
+ await page.goto("/");
+ await showAll(page);
+ await expectAllVideos(page);
+ await page.reload();
+ await settle(page);
+ await expectClearScreen(page);
+ });
+
+ test("a qt= link shows its results on load", async ({ page }) => {
+ await page.goto(`/?qt=${qt(ALPHA_TREE)}`);
+ await expect(page.getByTestId("results-summary")).toContainText("Matching videos");
+ await expect(page.locator("[data-card-header]")).toHaveCount(3);
+ await expect(page.getByText(HINT)).toHaveCount(0);
+ });
+
+ test("a filter-only link shows its results on load", async ({ page }) => {
+ await installTagRoutes(page);
+ await page.goto(`/?tg=${TAG_TOPIC}`);
+ await expect(page.getByTestId("results-summary")).toHaveText("All videos (1)");
+ await expect(card(page, VIDEO_CHAT_LARGE)).toBeVisible();
+ await expect(card(page, VIDEO_CHAT_SMALL)).toHaveCount(0);
+
+ // A legacy channel link: this one leaves the only channel out.
+ await page.goto(`/?ch=${encodeURIComponent(CHANNEL)}`);
+ await expect(page.getByTestId("results-summary")).toHaveText("All videos (0)");
+ await expect(page.getByText("No videos match the current filters.")).toBeVisible();
+ });
+
+ test("a restored query shows the clear screen and the filled form", async ({ page }) => {
+ await page.goto("/");
+ await page.evaluate(
+ ({ key, value }) => window.localStorage.setItem(key, value),
+ {
+ key: STORAGE_KEY,
+ value: JSON.stringify({
+ v: 1,
+ working: {
+ channels: { included: [], excluded: [] },
+ nol: true,
+ query: JSON.stringify(ALPHA_TREE),
+ },
+ profiles: {},
+ activeProfileName: null,
+ }),
+ },
+ );
+ await page.goto("/");
+ await expect(leafInput(page)).toHaveValue("alpha");
+ await expect(page.getByTestId("search-submit")).toHaveAttribute("data-dirty", "true");
+ await settle(page);
+ await expectClearScreen(page);
+ // The visitor runs it.
+ await leafInput(page).press("Enter");
+ await expect(page.getByTestId("results-summary")).toContainText("Matching videos");
+ await expect(card(page, VIDEO_TRANSCRIPT_ONLY)).toBeVisible();
+ });
+});
diff --git a/export/e2e/fixtures/data.ts b/export/e2e/fixtures/data.ts
@@ -462,8 +462,12 @@ export function subsPage() {
}
// ─── Social-post corpus fixtures ───
-// "alpha" appears in BOTH the transcript cues and a post body, so a combined
-// (transcripts OR posts) query is provably returning results from both corpora.
+// The posts have words of their own ("kappa", "sigma", "omega"), in no video's
+// cues, title or chat. Since release 16 a plain query reads posts by default
+// ("Search in": Transcripts and Posts ticked), and when the posts said "alpha"
+// and "gamma" like every video's cues, every spec that searches those words
+// for its own reasons got two post cards it was not about. A query that reads
+// both corpora names a word from each (posts-search.spec, search-in.spec).
function makePost(
id: string,
@@ -523,17 +527,17 @@ export function channelPostsManifest() {
export function postsPage() {
return [
- makePost(POST_ROOT_ID, "a post about alpha things", {
+ makePost(POST_ROOT_ID, "a post about kappa things", {
links: ["https://example.com/linked"],
engagement: { likes: 12, reposts: 3, replies: 1 },
}),
- makePost(POST_REPLY_ID, "replying about alpha again", {
+ makePost(POST_REPLY_ID, "replying about kappa again", {
isReply: true,
threadId: POST_ROOT_ID,
replyTo: { platform: "bluesky", id: POST_ROOT_ID },
createdAt: "2026-02-03T11:00:00.000Z",
}),
- makePost(POST_DELETED_ID, "a deleted gamma post", {
+ makePost(POST_DELETED_ID, "a deleted sigma post", {
isDeleted: true,
availability: "deleted",
availabilityCheckedAt: "2026-02-04T00:00:00.000Z",
diff --git a/export/e2e/fixtures/sites/testsite/site.json b/export/e2e/fixtures/sites/testsite/site.json
@@ -10,7 +10,8 @@
{
"label": "GitHub",
"url": "https://github.com/example",
- "svg": "<svg aria-hidden=\"true\" fill=\"currentColor\" viewBox=\"0 0 24 24\"><path d=\"M1 1h2\"/></svg>"
+ "svg": "<svg aria-hidden=\"true\" fill=\"currentColor\" viewBox=\"0 0 24 24\"><path d=\"M1 1h2\"/></svg>",
+ "featured": true
}
],
"relatedSites": [{ "label": "Friends", "siteIds": ["othersite"] }],
diff --git a/export/e2e/header.spec.ts b/export/e2e/header.spec.ts
@@ -0,0 +1,586 @@
+import fs from "node:fs";
+import path from "node:path";
+import { test, expect, type Locator, type Page } from "@playwright/test";
+import { INSTANCES_URL } from "../../common/lib/project";
+import { normalizeSocialSvg } from "../../common/lib/settingsSchema";
+import { ADVERSARIAL, EVIL, LOADS_ELSEWHERE } from "../../common/lib/socialSvg.vectors";
+import { installRoutes } from "./helpers";
+
+// THE EXPORT HEADER (export/app/components/Header.tsx): brand · nav ·
+// Archilyzer · the group (the social row and the theme toggle) from `lg`;
+// brand · group · menu trigger below it. No sites dropdown, no hub link, no
+// theme menu; Changelog is in the footer. The header keeps the NAME on a
+// narrow screen: below 520 px its row holds only the links marked `featured`
+// (none marked → none), from 520 px every link up to four, the marked ones
+// kept first; the footer holds every link. Only as last resorts does the
+// wordmark's text drop (whenever it does not fit, whatever the title) and the
+// social row scroll in its box.
+//
+// The suite serves ONE site (SITE_ID=testsite) and currentSite() re-reads
+// site.json on every dev render, so each test rewrites the fixture's title and
+// social links and puts the file back after. The original rides along in
+// `_e2ePristine`, which playwright.config.ts restores if a run dies mid-test.
+// The icons are synthetic, put through the save path's normalizer.
+
+const SITE_FILE = path.resolve(process.cwd(), "e2e", "fixtures", "sites", "testsite", "site.json");
+let pristine = "";
+
+test.describe.configure({ mode: "serial" });
+
+test.beforeAll(() => {
+ pristine = fs.readFileSync(SITE_FILE, "utf8");
+});
+
+test.afterEach(() => {
+ fs.writeFileSync(SITE_FILE, pristine);
+});
+
+type RawLink = { label: string; url: string; svg: string; featured?: boolean };
+
+function writeSite(patch: { headerTitle?: string; wordmarkLead?: string; socialLinks?: RawLink[] }) {
+ const site = JSON.parse(pristine) as Record<string, unknown>;
+ fs.writeFileSync(
+ SITE_FILE,
+ `${JSON.stringify({ ...site, ...patch, _e2ePristine: pristine }, null, 2)}\n`,
+ );
+}
+
+const icon = (d: string) => `<svg viewBox="0 0 24 24"><path d="${d}"/></svg>`;
+const link = (label: string, d: string, featured = false): RawLink => ({
+ label,
+ url: `https://${label.toLowerCase()}.example/fixture`,
+ svg: normalizeSocialSvg(icon(d))!,
+ ...(featured ? { featured: true } : {}),
+});
+const SIX = [
+ link("Square", "M4 4h16v16H4z"),
+ link("Triangle", "M12 3l10 18H2z"),
+ link("Bar", "M3 10h18v4H3z"),
+ link("Diamond", "M12 2l10 10-10 10L2 12z"),
+ link("Post", "M10 2h4v20h-4z"),
+ link("Chevron", "M4 8l8 8 8-8-3-3-5 5-5-5z"),
+];
+const TITLES = {
+ short: { headerTitle: "Shortlyzer", wordmarkLead: "Short" },
+ long: { headerTitle: "Longestfixturealyzer", wordmarkLead: "Longestfixture" },
+} as const;
+
+const banner = (page: Page) => page.getByRole("banner");
+const brand = (page: Page) => page.locator("header [data-header-brand]");
+// The row at the current width: a narrow and a wide copy are both rendered and
+// CSS shows one.
+const headerRow = (page: Page) => page.locator('header [data-social-links="header"]:visible');
+const footerRow = (page: Page) => page.locator('footer [data-social-links="footer"]');
+const marked = (links: RawLink[]) => links.map((l) => ({ ...l, featured: true }));
+const lastMarked = (links: RawLink[]) => links.map((l, i) => (i === links.length - 1 ? { ...l, featured: true } : l));
+const labelsOf = (loc: Locator) =>
+ loc.getByRole("link").evaluateAll((els) => els.map((e) => e.getAttribute("aria-label")));
+const toggle = (page: Page) => banner(page).getByRole("button", { name: /^switch to /i });
+
+async function open(page: Page, width: number) {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ await expect(toggle(page)).toBeVisible();
+}
+
+// What the narrow header is doing: is the wordmark's text on its visible line,
+// does the row's box scroll, does anything overflow.
+const layout = (page: Page) =>
+ page.evaluate(() => {
+ const box = document.querySelector("header [data-header-brand] span.flex-wrap") as HTMLElement;
+ const text = box.lastElementChild as HTMLElement;
+ const scroll = ([...document.querySelectorAll("header [data-social-scroll]")] as HTMLElement[]).find(
+ (e) => e.offsetParent !== null,
+ );
+ const row = document.querySelector("header > div") as HTMLElement;
+ return {
+ text: text.offsetTop < box.clientHeight - 1,
+ scrolls: scroll ? scroll.scrollWidth > scroll.clientWidth + 1 : false,
+ page: document.documentElement.scrollWidth - document.documentElement.clientWidth,
+ row: row.scrollWidth - row.clientWidth,
+ };
+ });
+
+for (const touch of [false, true]) {
+ test.describe(touch ? "under a coarse pointer" : "with a mouse", () => {
+ test.use({ hasTouch: touch });
+ for (const [kind, title] of Object.entries(TITLES)) {
+ test(`${kind} title, 0–4 marked links, 280–430 px: nothing scrolls the page, only the marked shown, the text drops before the row scrolls`, async ({
+ page,
+ }) => {
+ test.setTimeout(150_000);
+ await installRoutes(page);
+ for (const n of [0, 1, 2, 3, 4]) {
+ // Four links, the last n marked.
+ writeSite({ ...title, socialLinks: SIX.slice(0, 4).map((l, i) => (i >= 4 - n ? { ...l, featured: true } : l)) });
+ for (const w of [280, 300, 320, 340, 360, 375, 390, 414, 430]) {
+ await open(page, w);
+ const at = `${kind} title, ${n} marked, ${w} px`;
+ await expect(headerRow(page).getByRole("link"), at).toHaveCount(n);
+ const l = await layout(page);
+ if (n === 0) expect(l.scrolls, `${at}: a row with no links`).toBe(false);
+ expect(l.page, `${at}: the page scrolls sideways`).toBeLessThanOrEqual(0);
+ expect(l.row, `${at}: the header row overflows`).toBeLessThanOrEqual(0);
+ if (l.scrolls) expect(l.text, `${at}: the row scrolls while the text shows`).toBe(false);
+ await expect(brand(page).locator("svg[data-brand-mark]"), at).toBeInViewport({ ratio: 1 });
+ await expect(toggle(page), at).toBeInViewport({ ratio: 1 });
+ // The name is the title whether or not the text shows.
+ await expect(brand(page), at).toHaveAccessibleName(title.headerTitle);
+ }
+ }
+ });
+ }
+ });
+}
+
+// From lg the brand link is sized by its content; with a very long title it
+// must shrink first (`shrink-[999]`), so its text drops before any icon
+// scrolls — as below lg.
+test.describe("from lg, under a coarse pointer, a very long title", () => {
+ test.use({ hasTouch: true });
+ test("1024–1050 px: the text drops before the icons scroll, and nothing scrolls the page", async ({
+ page,
+ }) => {
+ await installRoutes(page);
+ writeSite({
+ headerTitle: "Longestfixturearchivealyzer",
+ wordmarkLead: "Longestfixturearchive",
+ socialLinks: lastMarked(SIX.slice(0, 4)),
+ });
+ await open(page, 1024);
+ for (let w = 1024; w <= 1050; w += 2) {
+ await page.setViewportSize({ width: w, height: 800 });
+ const l = await layout(page);
+ expect(l.scrolls, `${w} px: the icons scroll`).toBe(false);
+ expect(l.page, `${w} px: the page scrolls sideways`).toBeLessThanOrEqual(0);
+ expect(l.row, `${w} px: the header row overflows`).toBeLessThanOrEqual(0);
+ }
+ });
+});
+
+// At twice the text size the header's fixed parts (the mark, the toggle, the
+// menu trigger) are twice as wide; the social row's box takes up the rest and
+// scrolls, so the header itself never widens the page. (The page's own
+// content below is not this header's.)
+for (const touch of [false, true]) {
+ test.describe(touch ? "at 200 % text, under a coarse pointer" : "at 200 % text, with a mouse", () => {
+ test.use({ hasTouch: touch });
+ test("320 and 360 px: the header fits, every key in reach", async ({ page }) => {
+ await installRoutes(page);
+ writeSite({ ...TITLES.long, socialLinks: marked(SIX.slice(0, 4)) });
+ await page.addInitScript(() => {
+ document.addEventListener("DOMContentLoaded", () => {
+ document.documentElement.style.fontSize = "200%";
+ });
+ });
+ for (const w of [320, 360]) {
+ await open(page, w);
+ await expect(page.locator("html")).toHaveCSS("font-size", "32px");
+ const header = await page.locator("header").first().evaluate((el) => ({
+ over: el.scrollWidth - el.clientWidth,
+ right: Math.max(...[...el.querySelectorAll("a, button")].filter((e) => (e as HTMLElement).offsetParent && !e.closest("[data-social-scroll]")).map((e) => e.getBoundingClientRect().right)),
+ }));
+ expect(header.over, `${w} px: the header overflows`).toBeLessThanOrEqual(0);
+ expect(header.right, `${w} px: a control past the edge`).toBeLessThanOrEqual(w);
+ await expect(toggle(page)).toBeInViewport({ ratio: 1 });
+ await expect(page.getByRole("button", { name: "Open menu" })).toBeInViewport({ ratio: 1 });
+ // The last key is at the box's end, in view, wherever the box has room
+ // for one (at 320 px under touch the mark, the 88 px toggle and the
+ // 72 px menu button leave it none: the phone header is the operator's
+ // to rule on).
+ const room = await page.locator("header [data-social-scroll]:visible").evaluate((el) => el.clientWidth);
+ if (room >= (touch ? 88 : 72)) {
+ await expect(headerRow(page).getByRole("link").last()).toBeInViewport();
+ }
+ expect(w === 360 || !touch ? room : 1, `${w} px: the row's box`).toBeGreaterThan(0);
+ }
+ });
+ });
+}
+
+// The switch between the narrow and the wide row is 32.5rem, so it moves with
+// the reader's own text size (the browser's default font size, which rem in a
+// media query follows): at 32 px it is 1040 px. A page that sets its own root
+// size moves the page's rem but not a media query's; the export's wrap needs
+// no count, so there too the row never scrolls while the name shows.
+async function browserTextSize(page: Page, px: number) {
+ const cdp = await page.context().newCDPSession(page);
+ await cdp.send("Page.enable");
+ await cdp.send("Page.setFontSizes", { fontSizes: { standard: px } });
+}
+for (const touch of [false, true]) {
+ test.describe(touch ? "at 200 % text, the switch, under a coarse pointer" : "at 200 % text, the switch, with a mouse", () => {
+ test.use({ hasTouch: touch });
+ test("the browser's text size at 32 px: the narrow row until 1040 px; the page's root at 200 %: the switch stays at 520 px; the row never scrolls while the name shows", async ({
+ page,
+ }) => {
+ test.setTimeout(90_000);
+ await installRoutes(page);
+ writeSite({ ...LONGEST, socialLinks: lastMarked(SIX.slice(0, 4)) });
+ const four = SIX.slice(0, 4).map((l) => l.label);
+ await browserTextSize(page, 32);
+ for (const w of [600, 767, 1039, 1040, 1300]) {
+ await open(page, w);
+ await expect(page.locator("html")).toHaveCSS("font-size", "32px");
+ const at = `browser text 32 px, ${w} px`;
+ expect(await labelsOf(headerRow(page)), at).toEqual(w < 1040 ? ["Diamond"] : four);
+ const l = await layout(page);
+ if (l.scrolls) expect(l.text, `${at}: the row scrolls while the name shows`).toBe(false);
+ expect(l.row, `${at}: the header row overflows`).toBeLessThanOrEqual(0);
+ // 767 px at 32 px text is 383 px at the default size: past every real
+ // title's full-name width.
+ if (w >= 767) expect(l.text, `${at}: the name`).toBe(true);
+ }
+ await browserTextSize(page, 16);
+ await page.addInitScript(() => {
+ document.addEventListener("DOMContentLoaded", () => {
+ document.documentElement.style.fontSize = "200%";
+ });
+ });
+ for (const w of [519, 520, 600, 680, 767]) {
+ await open(page, w);
+ await expect(page.locator("html")).toHaveCSS("font-size", "32px");
+ const at = `root 200 %, ${w} px`;
+ expect(await labelsOf(headerRow(page)), at).toEqual(w < 520 ? ["Diamond"] : four);
+ const l = await layout(page);
+ if (l.scrolls) expect(l.text, `${at}: the row scrolls while the name shows`).toBe(false);
+ expect(l.row, `${at}: the header row overflows`).toBeLessThanOrEqual(0);
+ }
+ });
+ });
+}
+
+// The footer shows every link: its row wraps rather than widen the page.
+test.describe("under a coarse pointer, eight links", () => {
+ test.use({ hasTouch: true });
+ test("320 px: the footer's row wraps, and nothing scrolls the page", async ({ page }) => {
+ await installRoutes(page);
+ const eight = [...SIX, link("Ring", "M12 3a9 9 0 1 1 0 18 9 9 0 0 1 0-18z"), link("Cross", "M10 3h4v7h7v4h-7v7h-4v-7H3v-4h7z")];
+ writeSite({ ...TITLES.short, socialLinks: eight });
+ await open(page, 320);
+ const row = page.locator('footer [data-social-links="footer"]');
+ await expect(row.getByRole("link")).toHaveCount(8);
+ const boxes = await row.getByRole("link").evaluateAll((els) => els.map((e) => e.getBoundingClientRect()).map((r) => ({ right: r.right, top: r.top })));
+ for (const b of boxes) expect(b.right).toBeLessThanOrEqual(320);
+ expect(new Set(boxes.map((b) => Math.round(b.top))).size, "one row of eight").toBeGreaterThan(1);
+ expect((await layout(page)).page).toBeLessThanOrEqual(0);
+ });
+});
+
+// The title's fit does not wait on the web font: its width is reserved from
+// the display face's metrics (lib/wordmarkWidth.ts), so in the fallback face
+// (narrower) and in Archivo the text is shown, or not, at the same widths.
+test("the title shows at the same widths in the fallback face and in Archivo", async ({ page }) => {
+ test.setTimeout(120_000);
+ await installRoutes(page);
+ const widths = [360, 375, 390, 412, 430];
+ const fits = async () => {
+ const out: boolean[] = [];
+ for (const w of widths) {
+ await open(page, w);
+ await page.evaluate(() => document.fonts.ready);
+ out.push((await layout(page)).text);
+ }
+ return out;
+ };
+ const archivo = () =>
+ page.evaluate(() => [...document.fonts].filter((f) => /Archivo/i.test(f.family)).map((f) => f.status));
+ for (const title of [TITLES.short, { headerTitle: "Mediumfixturealyzer", wordmarkLead: "Mediumfixture" }, TITLES.long]) {
+ writeSite({ ...title, socialLinks: lastMarked(SIX.slice(0, 3)) });
+ await page.route(/\.woff2(\?|$)/, (r) => r.abort());
+ const fallback = await fits();
+ expect(await archivo(), "the web font is blocked").not.toContain("loaded");
+ await page.unroute(/\.woff2(\?|$)/);
+ const loaded = await fits();
+ expect(await archivo(), "the web font loaded").toContain("loaded");
+ expect(loaded, `${title.headerTitle} at ${widths.join(", ")} px`).toEqual(fallback);
+ }
+});
+
+// The reservation is a MINIMUM width: where the text renders wider than the
+// display face's metrics say (here forced wider by letter-spacing), its box
+// grows with it and the text drops sooner; its last letter is never clipped.
+test("a wordmark that renders wider than its reservation widens its box and is never clipped", async ({ page }) => {
+ await installRoutes(page);
+ writeSite({ headerTitle: "Rekietalyzer", wordmarkLead: "Rekieta", socialLinks: lastMarked(SIX.slice(0, 4)) });
+ await open(page, 519);
+ await page.addStyleTag({ content: "header [data-wordmark] { letter-spacing: 0.08em !important; }" });
+ await page.evaluate(() => document.fonts.ready);
+ let shown = 0;
+ let hidden = 0;
+ for (let w = 519; w >= 280; w--) {
+ await page.setViewportSize({ width: w, height: 800 });
+ const r = await page.evaluate(() => {
+ const box = document.querySelector("header [data-header-brand] span.flex-wrap") as HTMLElement;
+ const wm = box.lastElementChild as HTMLElement;
+ return {
+ shown: wm.offsetTop < box.clientHeight - 1,
+ textRight: (wm.lastElementChild as HTMLElement).getBoundingClientRect().right,
+ boxRight: box.getBoundingClientRect().right,
+ over: wm.scrollWidth - wm.clientWidth,
+ };
+ });
+ if (!r.shown) {
+ hidden++;
+ continue;
+ }
+ shown++;
+ expect(r.over, `${w} px: the text overflows its box`).toBeLessThanOrEqual(1);
+ expect(r.textRight, `${w} px: the last letter is clipped`).toBeLessThanOrEqual(r.boxRight + 0.5);
+ }
+ expect(shown, "widths with the name").toBeGreaterThan(0);
+ expect(hidden, "widths without it").toBeGreaterThan(0);
+});
+
+// THE RULING: on a narrow screen the header keeps the NAME and shows only the
+// marked link(s), in the accessibility tree too; focus goes brand → the marked
+// link → the toggle → the menu. Every real site title shows in full at 360 px
+// with one marked link, even under touch: below 520 px the bar's two gaps are
+// 8 px. Rekietalyzer and Hasanalyzer are the longest.
+const REAL_TITLES = [
+ { headerTitle: "Jeralyzer", wordmarkLead: "Jer" },
+ { headerTitle: "Anilyzer", wordmarkLead: "Ani" },
+ { headerTitle: "Bonnellyzer", wordmarkLead: "Bonnell" },
+ { headerTitle: "Hasanalyzer", wordmarkLead: "Hasan" },
+ { headerTitle: "Rekietalyzer", wordmarkLead: "Rekieta" },
+ { headerTitle: "Jasolyzer", wordmarkLead: "Jaso" },
+] as const;
+const LONGEST = REAL_TITLES[4];
+const NARROW_TITLES = {
+ short: TITLES.short,
+ Bonnellyzer: REAL_TITLES[2],
+ Hasanalyzer: REAL_TITLES[3],
+ Rekietalyzer: LONGEST,
+};
+
+test.describe("the narrow header, touch, every real title", () => {
+ test.use({ hasTouch: true });
+ test("360 px, one marked link: the full name, the link, and nothing overflows", async ({ page }) => {
+ await installRoutes(page);
+ for (const title of REAL_TITLES) {
+ writeSite({ ...title, socialLinks: lastMarked(SIX.slice(0, 4)) });
+ await open(page, 360);
+ await page.evaluate(() => document.fonts.ready);
+ const at = `${title.headerTitle} at 360 px`;
+ const l = await layout(page);
+ expect(l.text, `${at}: the name`).toBe(true);
+ expect(l.scrolls, `${at}: the row scrolls`).toBe(false);
+ expect(l.page, `${at}: the page scrolls sideways`).toBeLessThanOrEqual(0);
+ expect(l.row, `${at}: the header row overflows`).toBeLessThanOrEqual(0);
+ expect(await labelsOf(headerRow(page)), at).toEqual(["Diamond"]);
+ await expect(page.getByRole("button", { name: "Open menu" }), at).toBeInViewport({ ratio: 1 });
+ }
+ });
+});
+
+for (const touch of [false, true]) {
+ test.describe(touch ? "the narrow header, touch" : "the narrow header, a mouse", () => {
+ test.use({ hasTouch: touch });
+ for (const [kind, title] of Object.entries(NARROW_TITLES)) {
+ test(`${kind} title at 360 and 390 px: the full name, the marked link alone, focus brand → link → toggle → menu`, async ({
+ page,
+ }) => {
+ await installRoutes(page);
+ writeSite({ ...title, socialLinks: lastMarked(SIX.slice(0, 4)) });
+ for (const w of [360, 390]) {
+ await open(page, w);
+ expect((await layout(page)).text, `${w} px: the name`).toBe(true);
+ expect(await labelsOf(headerRow(page))).toEqual(["Diamond"]);
+ for (const other of ["Square", "Triangle", "Bar"]) {
+ await expect(banner(page).getByRole("link", { name: other, exact: true })).toHaveCount(0);
+ }
+ expect(await labelsOf(footerRow(page))).toEqual(SIX.slice(0, 4).map((l) => l.label));
+ const order: string[] = [];
+ for (let i = 0; i < 4; i++) {
+ await page.keyboard.press("Tab");
+ order.push(
+ await page.evaluate(
+ () => document.activeElement?.getAttribute("aria-label") ?? document.activeElement?.textContent?.trim() ?? "",
+ ),
+ );
+ }
+ expect(order[0], "the brand").toBe(title.headerTitle);
+ expect(order[1]).toBe("Diamond");
+ expect(order[2]).toMatch(/^Switch to /);
+ expect(order[3]).toBe("Open menu");
+ }
+ });
+ }
+ test("none marked: the narrow header has no social link, and the name shows", async ({ page }) => {
+ await installRoutes(page);
+ writeSite({ ...LONGEST, socialLinks: SIX.slice(0, 4) });
+ for (const w of [320, 360, 390]) {
+ await open(page, w);
+ expect((await layout(page)).text, `${w} px: the name`).toBe(true);
+ await expect(banner(page).locator('[data-social-links="header"]:visible')).toHaveCount(0);
+ expect(await labelsOf(footerRow(page))).toEqual(SIX.slice(0, 4).map((l) => l.label));
+ }
+ });
+ });
+}
+
+test("from 520 px every link shows, up to four; the footer keeps every link; nothing scrolls the page from 280 to 1400 px", async ({
+ page,
+}) => {
+ await installRoutes(page);
+ writeSite({ ...LONGEST, socialLinks: lastMarked(SIX.slice(0, 4)) });
+ for (const w of [280, 320, 360, 390, 430, 519, 520, 640, 768, 1023, 1024, 1280, 1400]) {
+ await open(page, w);
+ expect(await labelsOf(headerRow(page)), `${w} px`).toEqual(
+ w < 520 ? ["Diamond"] : SIX.slice(0, 4).map((l) => l.label),
+ );
+ expect(await labelsOf(footerRow(page))).toEqual(SIX.slice(0, 4).map((l) => l.label));
+ expect((await layout(page)).page, `${w} px: the page scrolls sideways`).toBeLessThanOrEqual(0);
+ if (w >= 520) expect((await layout(page)).text, `${w} px: the name`).toBe(true);
+ }
+});
+
+test("a short title shows in full on a phone with one marked link; a very long one gives way", async ({ page }) => {
+ await installRoutes(page);
+ writeSite({ ...TITLES.short, socialLinks: lastMarked(SIX.slice(0, 1)) });
+ await open(page, 390);
+ expect((await layout(page)).text).toBe(true);
+ writeSite({ ...TITLES.long, socialLinks: marked(SIX.slice(0, 4)) });
+ await open(page, 390);
+ expect(await layout(page)).toMatchObject({ text: false, scrolls: false });
+ await open(page, 1280);
+ expect((await layout(page)).text).toBe(true);
+});
+
+for (const width of [390, 1280]) {
+ test(`${width} px: the group's keys are 36 px, the toggle is the last, their boxes touch`, async ({
+ page,
+ }) => {
+ await installRoutes(page);
+ writeSite({ ...TITLES.short, socialLinks: marked(SIX.slice(0, 3)) });
+ await open(page, width);
+ const keys: Locator[] = [...(await headerRow(page).getByRole("link").all()), toggle(page)];
+ expect(keys).toHaveLength(4);
+ const boxes = await Promise.all(keys.map((k) => k.boundingBox()));
+ for (const b of boxes) expect([b!.width, b!.height]).toEqual([36, 36]);
+ for (let i = 1; i < boxes.length; i++) {
+ expect(Math.abs(boxes[i]!.x - (boxes[i - 1]!.x + boxes[i - 1]!.width))).toBeLessThanOrEqual(0.5);
+ }
+ });
+}
+
+test.describe("under a coarse pointer", () => {
+ test.use({ hasTouch: true });
+ test("the keys and the toggle are 44 px", async ({ page }) => {
+ await installRoutes(page);
+ writeSite({ ...TITLES.short, socialLinks: marked(SIX.slice(0, 2)) });
+ await open(page, 390);
+ const keys = [...(await headerRow(page).getByRole("link").all()), toggle(page)];
+ expect(keys).toHaveLength(3);
+ for (const k of keys) {
+ const b = (await k.boundingBox())!;
+ expect([b.width, b.height]).toEqual([44, 44]);
+ }
+ });
+});
+
+test("keyboard focus draws the ring on a social key", async ({ page }) => {
+ await installRoutes(page);
+ writeSite({ ...TITLES.short, socialLinks: SIX.slice(0, 2) });
+ await open(page, 1280);
+ const first = headerRow(page).getByRole("link").first();
+ for (let i = 0; i < 30; i++) {
+ await page.keyboard.press("Tab");
+ if (await first.evaluate((el) => el === document.activeElement)) break;
+ }
+ const s = await first.evaluate((el) => {
+ const cs = getComputedStyle(el);
+ return { outline: cs.outlineStyle, shadow: cs.boxShadow };
+ });
+ expect(s.outline).toBe("none");
+ expect(s.shadow).toMatch(/0px 0px 0px 2px/);
+});
+
+test("the Archilyzer link goes to the official instances; no sites menu, hub link or theme menu", async ({
+ page,
+}) => {
+ await installRoutes(page);
+ await open(page, 1280);
+ const archilyzer = banner(page).getByRole("link", {
+ name: "Archilyzer — official instances",
+ exact: true,
+ });
+ await expect(archilyzer).toBeVisible();
+ await expect(archilyzer).toHaveText("Archilyzer");
+ await expect(archilyzer).toHaveAttribute("href", INSTANCES_URL);
+ await expect(archilyzer).not.toHaveAttribute("target", /.+/);
+ await expect(banner(page).getByRole("button", { name: /sites/i })).toHaveCount(0);
+ await expect(page.getByRole("button", { name: "Choose theme" })).toHaveCount(0);
+ await expect(banner(page).getByRole("link", { name: "Hub", exact: true })).toHaveCount(0);
+ // Below lg it is in the menu instead (responsive.spec), not the bar.
+ await open(page, 390);
+ await expect(archilyzer).toBeHidden();
+});
+
+test("Changelog is in the footer, not the header", async ({ page }) => {
+ await installRoutes(page);
+ await open(page, 1280);
+ await expect(banner(page).getByRole("link", { name: "Changelog" })).toHaveCount(0);
+ const footer = page.locator("footer");
+ const changelog = footer.getByRole("link", { name: "Changelog", exact: true });
+ await expect(changelog).toHaveAttribute("href", "/changelog");
+ // After Use with AI, in the same row.
+ const order = await footer
+ .getByRole("link")
+ .evaluateAll((els) => els.map((e) => e.textContent?.trim() ?? ""));
+ expect(order.indexOf("Changelog")).toBe(order.indexOf("Use with AI") + 1);
+});
+
+test("six links: wide shows the last four, narrow none; two marked: wide keeps them first, narrow shows them alone", async ({
+ page,
+}) => {
+ await installRoutes(page);
+ writeSite({ ...TITLES.short, socialLinks: SIX });
+ await open(page, 1280);
+ expect(await labelsOf(headerRow(page))).toEqual(SIX.slice(2).map((l) => l.label));
+ expect(await labelsOf(footerRow(page))).toEqual(SIX.map((l) => l.label));
+ await open(page, 390);
+ await expect(headerRow(page)).toHaveCount(0);
+ const featured = SIX.map((l) =>
+ l.label === "Square" || l.label === "Post" ? { ...l, featured: true } : l,
+ );
+ writeSite({ ...TITLES.short, socialLinks: featured });
+ await open(page, 1280);
+ expect(await labelsOf(headerRow(page))).toEqual(["Square", "Diamond", "Post", "Chevron"]);
+ await open(page, 390);
+ expect(await labelsOf(headerRow(page))).toEqual(["Square", "Post"]);
+});
+
+// A hand-edited site.json can hold anything. The header (and the footer)
+// inline an icon only if it passes the save path's check again; the rest show
+// their label, run nothing and fetch nothing from anywhere else.
+test("a hostile icon in a site's socialLinks is its label: nothing runs, nothing is fetched elsewhere", async ({
+ page,
+}) => {
+ await installRoutes(page);
+ const hostile: RawLink[] = [
+ { label: "Hostile 1", url: "https://h1.example", svg: `<svg/onload="window.__hdrHostile=1" viewBox="0 0 8 8"><path d="M0 0"/></svg>` },
+ { label: "Hostile 2", url: "https://h2.example", svg: `<svg viewBox="0 0 8 8"><img src="x:" onerror="window.__hdrHostile=1"></svg>` },
+ ...LOADS_ELSEWHERE.slice(0, 2).map((name, i) => ({
+ label: `Hostile ${i + 3}`,
+ url: `https://h${i + 3}.example`,
+ svg: ADVERSARIAL[name],
+ })),
+ ];
+ writeSite({ ...TITLES.short, socialLinks: hostile });
+ const elsewhere: string[] = [];
+ page.on("request", (r) => {
+ const u = new URL(r.url());
+ if (u.protocol.startsWith("http") && u.hostname !== "localhost" && u.hostname !== "127.0.0.1") {
+ elsewhere.push(r.url());
+ }
+ });
+ await open(page, 1280);
+ await page.waitForLoadState("networkidle");
+ for (const l of hostile) {
+ await expect(headerRow(page).getByRole("link", { name: l.label, exact: true })).toHaveText(l.label);
+ }
+ expect(await page.evaluate(() => (window as unknown as { __hdrHostile?: number }).__hdrHostile)).toBeUndefined();
+ expect(elsewhere.filter((u) => u.startsWith(EVIL)), "requests to the icons' origin").toEqual([]);
+ expect(elsewhere, "requests to any other origin").toEqual([]);
+});
diff --git a/export/e2e/helpers.ts b/export/e2e/helpers.ts
@@ -156,6 +156,17 @@ export async function openFilters(page: Page) {
await profiles.waitFor();
}
+// The search page shows nothing under the bar until the first Search of the
+// page life (a link that carries a query or a filter shows its results on
+// load). A spec that wants the listing of every video presses Search, as a
+// visitor does. It waits for the query builder first, which renders only once
+// the bar has mounted, so the click is never lost to the pre-hydration window.
+export async function showAll(page: Page) {
+ await page.getByTestId("query-builder").waitFor();
+ await page.getByTestId("search-submit").click();
+ await page.getByTestId("results-summary").waitFor();
+}
+
export async function urlParams(page: Page): Promise<URLSearchParams> {
const search = await page.evaluate(() => window.location.search);
return new URLSearchParams(search);
diff --git a/export/e2e/posts-search.spec.ts b/export/e2e/posts-search.spec.ts
@@ -15,6 +15,11 @@ import { installRoutes } from "./helpers";
// The social-post corpus as a PARALLEL dataset to video transcripts: one
// search, one result set, with a toggle-able mode. Seeded through `?qt=`
// (the composite query tree) exactly like query-tree.spec.ts.
+//
+// The posts say "kappa" (two of them), "sigma" (the deleted one) and "omega";
+// every video's cues say "alpha". Since release 16 a plain query — a
+// "transcripts" leaf — reads posts too while Posts is ticked under Search in
+// (search-in.spec.ts); a leaf whose scope is "Posts" reads them by name.
type SLeaf = {
k: "l";
@@ -63,7 +68,7 @@ test.describe("social-post corpus — search", () => {
const tree: SGroup = {
k: "g",
o: "AND",
- c: [{ k: "l", q: "alpha", s: "posts" }],
+ c: [{ k: "l", q: "kappa", s: "posts" }],
};
await page.goto(`/?qt=${qt(tree)}`);
await expectResultSlugs(page, [POST_ROOT_SLUG, POST_REPLY_SLUG]);
@@ -84,7 +89,7 @@ test.describe("social-post corpus — search", () => {
o: "OR",
c: [
{ k: "l", q: "alpha", s: "transcripts" },
- { k: "l", q: "alpha", s: "posts" },
+ { k: "l", q: "kappa", s: "posts" },
],
};
await page.goto(`/?qt=${qt(tree)}`);
@@ -101,7 +106,7 @@ test.describe("social-post corpus — search", () => {
const tree: SGroup = {
k: "g",
o: "AND",
- c: [{ k: "l", q: "alpha", s: "posts" }],
+ c: [{ k: "l", q: "kappa", s: "posts" }],
};
await page.goto(`/?qt=${qt(tree)}`);
await expectResultSlugs(page, [POST_ROOT_SLUG, POST_REPLY_SLUG]);
@@ -133,7 +138,7 @@ test.describe("social-post corpus — search", () => {
const tree: SGroup = {
k: "g",
o: "AND",
- c: [{ k: "l", q: "alpha", s: "posts" }],
+ c: [{ k: "l", q: "kappa", s: "posts" }],
};
await page.goto(`/?qt=${qt(tree)}`);
await expectResultSlugs(page, [POST_ROOT_SLUG, POST_REPLY_SLUG]);
@@ -147,7 +152,7 @@ test.describe("social-post corpus — search", () => {
await expect(modal).toBeVisible();
// Appears twice by design: as the primary post and again in its thread.
await expect(
- modal.getByText("a post about alpha things").first(),
+ modal.getByText("a post about kappa things").first(),
).toBeVisible();
// Thread context: the reply is archived under the same threadId.
await expect(modal.getByText(/Thread \(2 posts\)/)).toBeVisible();
@@ -157,13 +162,13 @@ test.describe("social-post corpus — search", () => {
).toBeVisible();
});
- test("the Posts type toggle switches the corpus off", async ({ page }) => {
+ test("the Posts box leaves a \"Posts\" leaf reading posts", async ({ page }) => {
const tree: SGroup = {
k: "g",
o: "OR",
c: [
{ k: "l", q: "alpha", s: "transcripts" },
- { k: "l", q: "alpha", s: "posts" },
+ { k: "l", q: "kappa", s: "posts" },
],
};
await page.goto(`/?qt=${qt(tree)}`);
@@ -175,7 +180,9 @@ test.describe("social-post corpus — search", () => {
POST_REPLY_SLUG,
]);
- // Posts are a third media kind beside Videos / Livestreams.
+ // The Posts box sits under Search in (release 16; it was in the Type row,
+ // same key `nop`) and says what a plain query reads. A leaf whose scope is
+ // "Posts" was asked for by name, and still reads its two posts unticked.
await page.getByRole("checkbox", { name: "Posts" }).uncheck();
await page.getByTestId("search-submit").click();
@@ -183,6 +190,8 @@ test.describe("social-post corpus — search", () => {
TRANSCRIPT_ONLY_SLUG,
CHAT_SMALL_SLUG,
CHAT_LARGE_SLUG,
+ POST_ROOT_SLUG,
+ POST_REPLY_SLUG,
]);
});
});
@@ -197,7 +206,7 @@ test("a deleted post is flagged in results and in the modal", async ({ page }) =
o: "AND",
// A term unique to the deleted fixture, so the other specs' expected
// result sets stay untouched.
- c: [{ k: "l", q: "gamma", s: "posts" }],
+ c: [{ k: "l", q: "sigma", s: "posts" }],
};
await page.goto(`/?qt=${qt(tree)}`);
const deletedSlug = `${POST_CHANNEL_SLUG}/${POST_DELETED_ID}`;
diff --git a/export/e2e/responsive.spec.ts b/export/e2e/responsive.spec.ts
@@ -1,6 +1,7 @@
import { expect, test, type Page } from "@playwright/test";
import { CHANNEL_SLUG, VIDEO_TRANSCRIPT_ONLY } from "./fixtures/data";
-import { expectModalOpen, installRoutes } from "./helpers";
+import { expectModalOpen, installRoutes, showAll } from "./helpers";
+import { INSTANCES_URL } from "../../common/lib/project";
// The phone. Every other spec in this suite runs at the project's 1440×1200
// desktop viewport, which is exactly why the export site could ship a header
@@ -73,33 +74,49 @@ test.describe("phone layout", () => {
"/ask/",
"/downloads/",
"/duplicates/",
- "/use-with-ai/",
+ "/changelog/",
]) {
test(`no horizontal overflow on ${route}`, async ({ page }) => {
+ // KNOWN, and expected to fail until fixed: /changelog/ (checked here
+ // since release 16, in place of the removed /use-with-ai/) overflows a
+ // 390 px phone by ~600 px. Long inline `code` in the released entries
+ // (file lists like `export/app/ask/{MessageBubble,…}.tsx/ts`) has no
+ // break opportunity. When the changelog wraps them, this flips red:
+ // delete the line.
+ test.fail(route === "/changelog/", "the changelog's long inline code overflows a phone");
await page.goto(route);
await page.waitForLoadState("networkidle");
await expectNoHorizontalOverflow(page);
});
}
- test("the header menu carries the nav that the wide header shows inline", async ({
+ test("the header menu carries the nav that the wide header shows inline, and nothing else", async ({
page,
}) => {
await page.goto("/");
+ // The theme is the bar's toggle, beside the trigger.
+ await expect(page.getByRole("banner").getByRole("button", { name: /^switch to /i })).toBeVisible();
await page.getByRole("button", { name: "Open menu" }).click();
const menu = page.getByRole("dialog");
await expect(menu.getByRole("link", { name: "Ask AI" })).toBeVisible();
await expect(menu.getByRole("link", { name: "Search" })).toBeVisible();
- // The theme picker is a dropdown in the wide header; here it is two plain
- // radio lists (Base, Accent), because a popover inside a dialog is a
- // focus-trap fight.
- await expect(menu.getByRole("radio", { name: "Sepia" })).toBeVisible();
+ // The Archilyzer home's official instances, where the sites list was.
+ await expect(
+ menu.getByRole("link", { name: "Archilyzer — official instances", exact: true }),
+ ).toHaveAttribute("href", INSTANCES_URL);
+ // No theme radios; Changelog, the sibling sites and the hub are not here.
+ await expect(menu.getByRole("radiogroup")).toHaveCount(0);
+ await expect(menu.getByRole("radio")).toHaveCount(0);
+ await expect(menu.getByRole("link", { name: "Changelog" })).toHaveCount(0);
+ await expect(menu.getByRole("link", { name: "Hub" })).toHaveCount(0);
+ await expect(menu.getByText("Sites", { exact: true })).toHaveCount(0);
});
test("the filters sheet applies a filter and the results change", async ({
page,
}) => {
await page.goto("/");
+ await showAll(page);
const summary = page.getByTestId("results-summary");
await expect(summary).toContainText("All videos (3)");
@@ -137,6 +154,7 @@ test.describe("phone layout", () => {
page,
}) => {
await page.goto("/");
+ await showAll(page);
const toolbar = page.getByTestId("selection-toolbar");
// Nothing selected: it is an ordinary row above the cards, as the 1440
// specs see it.
diff --git a/export/e2e/restore-no-refire.spec.ts b/export/e2e/restore-no-refire.spec.ts
@@ -1,5 +1,6 @@
import { expect, test, type Page } from "@playwright/test";
-import { installRoutes } from "./helpers";
+import { TAG_TOPIC } from "./fixtures/data";
+import { installRoutes, installTagRoutes } from "./helpers";
// Release 8 slice E. The search session restored from localStorage loads the
// query and the filters but does NOT run the search on first load: a restored
@@ -7,6 +8,10 @@ import { installRoutes } from "./helpers";
// shards (up to 8 MB each on a cold device) for a search the visitor had not
// asked for this time. The visitor runs it — Search / Enter. A query on the
// URL (`qt=`) is a shared link, i.e. the visitor asking, and still runs.
+// Release 14: until then the page shows no results area at all — the held
+// query no longer sits over a browse listing. A link that carries only a
+// filter (`tg=`, `ch=`) shows its results but runs no query, so a stored one
+// stays held on that load AND on every later mount in the same visit.
const STORAGE_KEY = "ytdlp-tb:export-filters";
const SHARD = /\/transcripts\/[^/]+\/page-\d+\.json$/;
@@ -51,6 +56,31 @@ async function seedStoredSearch(page: Page) {
);
}
+// A route outside the workspace unmounts the search session; coming back
+// mounts it again in the same page life. Until release 16 the header's Use
+// with AI link made that client navigation; it now leaves the site, and the
+// footer's Changelog is a plain <a> (a new page life), so the spec asks the app
+// router itself — `window.next.router`, which Next sets for debugging in
+// development and production alike.
+async function leaveForChangelog(page: Page) {
+ await page.evaluate(() => {
+ (
+ window as unknown as { next: { router: { push(href: string): void } } }
+ ).next.router.push("/changelog/");
+ });
+ await expect(page).toHaveURL(/\/changelog\/?$/, { timeout: 20_000 });
+ await expect(page.getByRole("heading", { level: 1, name: "Changelog" })).toBeVisible();
+ await expect(page.getByTestId("query-builder")).toHaveCount(0);
+}
+
+async function expectHeld(page: Page) {
+ await expect(leafInput(page)).toHaveValue("alpha");
+ await expect(page.getByTestId("search-submit")).toHaveAttribute(
+ "data-dirty",
+ "true",
+ );
+}
+
test.describe("restored search waits for the visitor", () => {
test.beforeEach(async ({ page }) => {
await installRoutes(page);
@@ -78,14 +108,14 @@ test.describe("restored search waits for the visitor", () => {
await expect(
page.getByText("Press Enter or click Search to apply"),
).toBeVisible();
- // The results area is the browse listing under the restored filters.
- await expect(page.getByTestId("results-summary")).toHaveText(
- "All videos (3)",
- );
- await expect(page.getByTestId("browse-hint")).toBeVisible();
// Give a would-be pipeline every chance to start.
await page.waitForTimeout(1_500);
expect(shards.n).toBe(0);
+ // Nothing has been asked in this page life, so there is no results area
+ // at all: no count, no listing under the restored filters.
+ await expect(page.getByTestId("results-summary")).toHaveCount(0);
+ await expect(page.getByTestId("browse-hint")).toHaveCount(0);
+ await expect(page.locator("[data-card-header]")).toHaveCount(0);
// The visitor runs it.
const fetched = page.waitForRequest(
@@ -122,4 +152,54 @@ test.describe("restored search waits for the visitor", () => {
"false",
);
});
+
+ test("a filter-only link shows its results and leaves a stored query held, then and after Back", async ({
+ page,
+ }) => {
+ await installTagRoutes(page);
+ const shards = countShards(page);
+ await seedStoredSearch(page);
+
+ shards.n = 0;
+ await page.goto(`/?tg=${TAG_TOPIC}`);
+ // The link asked: its results show, under its filter…
+ await expect(page.getByTestId("results-summary")).toHaveText("All videos (1)");
+ // …and the stored query is in the box, held.
+ await expectHeld(page);
+
+ await leaveForChangelog(page);
+ await page.goBack();
+ await expect(page).not.toHaveURL(/changelog/);
+ // A second mount in the same visit: still held, still the listing.
+ await expectHeld(page);
+ await expect(page.getByTestId("results-summary")).toHaveText("All videos (1)");
+ await page.waitForTimeout(1_500);
+ expect(shards.n).toBe(0);
+ });
+
+ test("a filter-only link, then the header's Search link: the stored query stays held", async ({
+ page,
+ }) => {
+ const shards = countShards(page);
+ await seedStoredSearch(page);
+
+ shards.n = 0;
+ // Under a filter link the stored session is not read at all.
+ await page.goto(`/?ch=${encodeURIComponent("Nobody")}`);
+ await expect(page.getByTestId("results-summary")).toContainText("All videos");
+ await expect(leafInput(page)).toHaveValue("");
+
+ await leaveForChangelog(page);
+ await page
+ .getByRole("banner")
+ .getByRole("link", { name: "Search", exact: true })
+ .click();
+ await expect(page).not.toHaveURL(/changelog/, { timeout: 20_000 });
+ // The plain search page reads the stored session: the query is held, and
+ // the listing shows because the visitor asked earlier in this visit.
+ await expectHeld(page);
+ await expect(page.getByTestId("results-summary")).toHaveText("All videos (3)");
+ await page.waitForTimeout(1_500);
+ expect(shards.n).toBe(0);
+ });
});
diff --git a/export/e2e/search-in.spec.ts b/export/e2e/search-in.spec.ts
@@ -0,0 +1,327 @@
+import { expect, test, type Page } from "@playwright/test";
+import {
+ CHANNEL_SLUG,
+ TAG_COLLAB,
+ POST_CHANNEL_SLUG,
+ POST_REPLY_ID,
+ POST_ROOT_ID,
+ VIDEO_CHAT_LARGE,
+ VIDEO_CHAT_SMALL,
+ VIDEO_TRANSCRIPT_ONLY,
+} from "./fixtures/data";
+import { installRoutes, installTagRoutes, openFilters, showAll } from "./helpers";
+
+// Release 16, slice CK: the Filters panel's "Search in" row — Transcripts,
+// Posts, Live chat — says what a plain query (a "transcripts" leaf) reads.
+// Transcripts and Posts are on by default, Live chat off; a leaf asked for by
+// name in the builder is not the row's business; an empty query still lists
+// what the Type row says.
+//
+// The fixture: every video's cues say "… — alpha line" (and beta, gamma); two
+// posts say "kappa", which no video does; only VIDEO_CHAT_SMALL's live chat
+// says "message".
+
+const STORAGE_KEY = "ytdlp-tb:export-filters";
+
+const TRANSCRIPT_ONLY_SLUG = `${CHANNEL_SLUG}/${VIDEO_TRANSCRIPT_ONLY}`;
+const CHAT_SMALL_SLUG = `${CHANNEL_SLUG}/${VIDEO_CHAT_SMALL}`;
+const CHAT_LARGE_SLUG = `${CHANNEL_SLUG}/${VIDEO_CHAT_LARGE}`;
+const VIDEO_SLUGS = [TRANSCRIPT_ONLY_SLUG, CHAT_SMALL_SLUG, CHAT_LARGE_SLUG];
+const POST_SLUGS = [
+ `${POST_CHANNEL_SLUG}/${POST_ROOT_ID}`,
+ `${POST_CHANNEL_SLUG}/${POST_REPLY_ID}`,
+];
+
+const row = (page: Page) => page.getByTestId("search-in-row");
+const box = (page: Page, name: "Transcripts" | "Posts" | "Live chat") =>
+ row(page).getByRole("checkbox", { name, exact: true });
+const leafInput = (page: Page) =>
+ page.locator('input[data-testid^="leaf-query-"]').first();
+
+async function expectResultSlugs(page: Page, slugs: string[]) {
+ const cards = page.locator("[data-card-header]");
+ await expect(async () => {
+ const got = await cards.evaluateAll((els) =>
+ els.map((e) => e.getAttribute("data-result-slug") ?? ""),
+ );
+ expect(got.slice().sort()).toEqual(slugs.slice().sort());
+ }).toPass({ timeout: 15_000 });
+}
+
+async function search(page: Page, q: string) {
+ await leafInput(page).fill(q);
+ await page.getByTestId("search-submit").click();
+}
+
+// The search has finished: the progress line has lost its ellipsis.
+async function expectFinished(page: Page) {
+ await expect(page.getByText(/^searched \d+\/\d+$/)).toBeVisible({ timeout: 15_000 });
+}
+
+test.describe("Search in", () => {
+ test.beforeEach(async ({ page }) => {
+ await installRoutes(page);
+ });
+
+ test("by default Transcripts and Posts are ticked and Live chat is not", async ({
+ page,
+ }) => {
+ await page.goto("/");
+ await openFilters(page);
+ await expect(box(page, "Transcripts")).toBeChecked();
+ await expect(box(page, "Posts")).toBeChecked();
+ await expect(box(page, "Live chat")).not.toBeChecked();
+ // Posts moved here from the Type row: one box of that name on the page.
+ await expect(page.getByRole("checkbox", { name: "Posts", exact: true })).toHaveCount(1);
+ await expect(page.getByRole("checkbox", { name: "Videos" })).toBeVisible();
+ await expect(page.getByRole("checkbox", { name: "Livestreams" })).toBeVisible();
+ // The bar's hint points at the row, and goes once the box is ticked.
+ const hint = page.getByText(/Live chat available on 2 videos — tick Live chat under Search in/);
+ await expect(hint).toBeVisible();
+ await box(page, "Live chat").check();
+ await expect(hint).toHaveCount(0);
+ // The refusal line is always there, and empty unless Search is refused.
+ await expect(page.getByTestId("search-in-empty")).toBeEmpty();
+ });
+
+ test("a plain query reads posts by default; unticking Posts drops them", async ({
+ page,
+ }) => {
+ await page.goto("/");
+ await search(page, "kappa");
+ await expectResultSlugs(page, POST_SLUGS);
+ // A post's section says what it holds, not the leaf's "Transcripts".
+ await expect(
+ page.locator(`[data-result-slug="${POST_SLUGS[0]}"] [data-leaf-section]`),
+ ).toContainText("Posts");
+
+ await openFilters(page);
+ await box(page, "Posts").uncheck();
+ await page.getByTestId("search-submit").click();
+ await expect(page.getByTestId("results-summary")).toHaveText("Matching videos (0)");
+ });
+
+ test("one plain leaf reads both corpora into one result set", async ({ page }) => {
+ // A regex leaf of scope "transcripts" — the default leaf — with no posts
+ // leaf beside it: the videos by their cues, the posts by their bodies.
+ const tree = {
+ k: "g",
+ o: "AND",
+ c: [{ k: "l", q: "alpha|kappa", s: "transcripts", r: 1 }],
+ };
+ await page.goto(`/?qt=${encodeURIComponent(JSON.stringify(tree))}`);
+ await expectResultSlugs(page, [...VIDEO_SLUGS, ...POST_SLUGS]);
+ await expect(page.getByTestId("results-summary")).toHaveText(
+ "Matching videos (5 videos, 5 hits)",
+ );
+ });
+
+ test("ticking Live chat reads the chat: hits appear, badged live chat", async ({
+ page,
+ }) => {
+ await page.goto("/");
+ // "message" is only in VIDEO_CHAT_SMALL's live chat: by default, nothing.
+ await search(page, "message");
+ await expect(page.getByTestId("results-summary")).toHaveText("Matching videos (0)");
+
+ await openFilters(page);
+ await box(page, "Live chat").check();
+ await page.getByTestId("search-submit").click();
+ await expectResultSlugs(page, [CHAT_SMALL_SLUG]);
+ await expect(page.getByTestId("results-summary")).toHaveText(
+ "Matching videos (1 video, 30 hits)",
+ );
+ const card = page.locator(`[data-result-slug="${CHAT_SMALL_SLUG}"]`);
+ // Every hit shown wears the track badge; the section says Live chat.
+ const hits = card.locator("[data-leaf-section] + ul > li");
+ const badges = card.getByText("live chat", { exact: true });
+ await expect(badges.first()).toBeVisible();
+ expect(await badges.count()).toBe(await hits.count());
+ await expect(card.locator("[data-leaf-section]")).toContainText("Live chat");
+ // The URL carries the query the visitor built, not the rewrite.
+ const qt = new URL(page.url()).searchParams.get("qt") ?? "";
+ expect(JSON.parse(qt)).toEqual({
+ k: "g",
+ o: "AND",
+ c: [{ k: "l", q: "message", s: "transcripts" }],
+ });
+ });
+
+ test("with Transcripts off and Live chat on, a word only in the cues finds nothing", async ({
+ page,
+ }) => {
+ await page.goto("/");
+ await openFilters(page);
+ await box(page, "Transcripts").uncheck();
+ await box(page, "Live chat").check();
+
+ // "line" is in every video's cues and nowhere else.
+ await search(page, "line");
+ await expect(page.getByTestId("results-summary")).toHaveText("Matching videos (0)");
+ await expect(page.locator("[data-card-header]")).toHaveCount(0);
+
+ await search(page, "message");
+ await expectResultSlugs(page, [CHAT_SMALL_SLUG]);
+
+ // With Transcripts back on, "line" finds all three again.
+ await box(page, "Transcripts").check();
+ await search(page, "line");
+ await expectResultSlugs(page, VIDEO_SLUGS);
+ });
+
+ test("with nothing ticked Search is refused and the row says so", async ({
+ page,
+ }) => {
+ await page.goto("/");
+ await openFilters(page);
+ await leafInput(page).fill("kappa");
+ await box(page, "Transcripts").uncheck();
+ await box(page, "Posts").uncheck();
+ // Live chat is off by default: nothing is ticked now.
+ const submit = page.getByTestId("search-submit");
+ const empty = page.getByTestId("search-in-empty");
+ await expect(submit).toBeDisabled();
+ await expect(empty).toHaveText("Search in: pick at least one");
+ await expect(empty).toHaveAttribute("aria-live", "polite");
+ // The disabled button says why, to a screen reader as well as on hover.
+ await expect(submit).toHaveAccessibleDescription("Search in: pick at least one");
+ // A save commits the draft, so Save as… is refused the same way.
+ await expect(page.getByRole("button", { name: "Save as…" })).toBeDisabled();
+ await expect(page.getByText("Press Enter or click Search to apply")).toHaveCount(0);
+ // Enter does not commit it either.
+ await leafInput(page).press("Enter");
+ await page.waitForTimeout(500);
+ expect(new URL(page.url()).searchParams.get("qt")).toBeNull();
+ await expect(page.getByTestId("results-summary")).toHaveCount(0);
+
+ await box(page, "Live chat").check();
+ await expect(submit).toBeEnabled();
+ await expect(empty).toBeEmpty();
+ await expect(submit).toHaveAccessibleDescription("");
+ await expect(page.getByRole("button", { name: "Save as…" })).toBeEnabled();
+ await box(page, "Live chat").uncheck();
+ await expect(submit).toBeDisabled();
+ await box(page, "Posts").check();
+ await expect(submit).toBeEnabled();
+ await submit.click();
+ await expectResultSlugs(page, POST_SLUGS);
+ });
+
+ test("the row survives a reload and a saved profile", async ({ page }) => {
+ await page.goto("/");
+ await openFilters(page);
+ await box(page, "Transcripts").uncheck();
+ await box(page, "Live chat").check();
+ await search(page, "message");
+ await expectResultSlugs(page, [CHAT_SMALL_SLUG]);
+
+ await page.reload();
+ await openFilters(page);
+ await expect(box(page, "Transcripts")).not.toBeChecked();
+ await expect(box(page, "Posts")).toBeChecked();
+ await expect(box(page, "Live chat")).toBeChecked();
+
+ // Save the row as a profile.
+ page.once("dialog", (d) => d.accept("chat only"));
+ await page.getByRole("button", { name: "Save as…" }).click();
+ await expect(page.getByTestId("profile-select")).toHaveValue("chat only");
+ await expect(page.getByTestId("profile-dirty-dot")).toHaveCount(0);
+
+ // Back to the default row and Search: the commit leaves the profile.
+ await box(page, "Transcripts").check();
+ await box(page, "Live chat").uncheck();
+ await page.getByTestId("search-submit").click();
+ await expect(page.getByTestId("profile-select")).toHaveValue("");
+
+ await page.reload();
+ await openFilters(page);
+ await expect(box(page, "Transcripts")).toBeChecked();
+ await expect(box(page, "Live chat")).not.toBeChecked();
+
+ // Loading the profile brings its row back, and it does not read as changed.
+ await page.getByTestId("profile-select").selectOption("chat only");
+ await expect(box(page, "Transcripts")).not.toBeChecked();
+ await expect(box(page, "Posts")).toBeChecked();
+ await expect(box(page, "Live chat")).toBeChecked();
+ await expect(page.getByTestId("profile-dirty-dot")).toHaveCount(0);
+ await expectResultSlugs(page, [CHAT_SMALL_SLUG]);
+
+ // Stored off the default only: `notr` and `lc`, and no `nop`.
+ const stored = await page.evaluate(
+ (key) => JSON.parse(window.localStorage.getItem(key) ?? "null"),
+ STORAGE_KEY,
+ );
+ const profile = stored.profiles["chat only"];
+ expect(profile.notr).toBe(true);
+ expect(profile.lc).toBe(true);
+ expect("nop" in profile).toBe(false);
+ });
+
+ test("an empty query with Transcripts off still lists every video", async ({
+ page,
+ }) => {
+ await page.goto("/");
+ await openFilters(page);
+ await box(page, "Transcripts").uncheck();
+ await showAll(page);
+ await expect(page.getByTestId("results-summary")).toHaveText("All videos (3)");
+ await expectResultSlugs(page, VIDEO_SLUGS);
+ });
+
+ test("a leaf asked for by name is not changed by the row", async ({ page }) => {
+ // A "Live chat" leaf reads the chat with the row at its default (Live
+ // chat unticked)…
+ const chatLeaf = { k: "g", o: "AND", c: [{ k: "l", q: "message", s: "chat" }] };
+ await page.goto(`/?qt=${encodeURIComponent(JSON.stringify(chatLeaf))}`);
+ await expectResultSlugs(page, [CHAT_SMALL_SLUG]);
+ await openFilters(page);
+ await expect(box(page, "Live chat")).not.toBeChecked();
+
+ // …and a "Posts" leaf reads posts with Transcripts and Posts unticked
+ // (Live chat ticked, so the row reads something and can be committed).
+ await box(page, "Transcripts").uncheck();
+ await box(page, "Posts").uncheck();
+ await box(page, "Live chat").check();
+ await page.getByTestId("search-submit").click();
+ const postsLeaf = { k: "g", o: "AND", c: [{ k: "l", q: "kappa", s: "posts" }] };
+ await page.goto(`/?qt=${encodeURIComponent(JSON.stringify(postsLeaf))}`);
+ await openFilters(page);
+ await expect(box(page, "Transcripts")).not.toBeChecked();
+ await expect(box(page, "Posts")).not.toBeChecked();
+ await expectResultSlugs(page, POST_SLUGS);
+ });
+
+ test("a plain query with a tag chip on finishes, with its results or none", async ({
+ page,
+ }) => {
+ // Posts carry no curated tags, so under a chip the plain query reads no
+ // posts; before the fix the posts copy it read had an empty scope and the
+ // search never finished.
+ await installTagRoutes(page);
+ await page.goto("/");
+ await openFilters(page);
+ await page.locator(`[data-testid="tag-chip"][data-tag-id="${TAG_COLLAB}"]`).click();
+ await search(page, "alpha");
+ await expectResultSlugs(page, [CHAT_SMALL_SLUG, CHAT_LARGE_SLUG]);
+ await expectFinished(page);
+
+ // Posts alone under a chip: nothing can match, and it says so.
+ await box(page, "Transcripts").uncheck();
+ await search(page, "kappa");
+ await expect(page.getByText("No matching videos.")).toBeVisible({ timeout: 15_000 });
+ await expectFinished(page);
+ });
+
+ test("with no video type kept, a plain query still reads the posts, and finishes", async ({
+ page,
+ }) => {
+ // The transcripts copy's scope is then empty (the Type row keeps no video).
+ await page.goto("/");
+ await openFilters(page);
+ await page.getByRole("checkbox", { name: "Videos" }).uncheck();
+ await page.getByRole("checkbox", { name: "Livestreams" }).uncheck();
+ await search(page, "kappa");
+ await expectResultSlugs(page, POST_SLUGS);
+ await expectFinished(page);
+ });
+});
diff --git a/export/e2e/site-branding.spec.ts b/export/e2e/site-branding.spec.ts
@@ -26,7 +26,7 @@ test("footer renders the site's social links", async ({ page }) => {
await installRoutes(page);
await page.goto("/");
await expect(
- page.getByRole("link", { name: "GitHub" }),
+ page.locator("footer").getByRole("link", { name: "GitHub" }),
).toHaveAttribute("href", "https://github.com/example");
});
diff --git a/export/e2e/tag-chips.spec.ts b/export/e2e/tag-chips.spec.ts
@@ -1,5 +1,5 @@
import { expect, test, type Page } from "@playwright/test";
-import { installRoutes, installTagRoutes } from "./helpers";
+import { installRoutes, installTagRoutes, showAll } from "./helpers";
import {
CHANNEL_SLUG,
TAG_COLLAB,
@@ -146,6 +146,7 @@ test.describe("curated tag chips", () => {
test("a tagged card shows its tags; an untagged one shows none", async ({
page,
}) => {
+ await showAll(page);
await expect(card(page, VIDEO_CHAT_LARGE).getByTestId("card-tag")).toHaveCount(2);
await expect(
card(page, VIDEO_CHAT_LARGE).locator('[data-testid="card-tag"][data-tag-id="' + TAG_TOPIC + '"]'),
@@ -161,6 +162,7 @@ test.describe("curated tag chips", () => {
page,
}) => {
// All three videos before any tag filter.
+ await showAll(page);
await expect(card(page, VIDEO_TRANSCRIPT_ONLY)).toBeVisible();
await chip(page, TAG_TOPIC).click();
@@ -237,10 +239,10 @@ test.describe("curated tag chips", () => {
// the assertion because the header is where it showed: N videos and a
// hit count that no card on the page accounts for.
//
- // "chat" matches all three video titles; "alpha" matches two posts.
+ // "chat" matches all three video titles; "kappa" matches two posts.
const tree = qt([
{ q: "chat", s: "metadata" },
- { q: "alpha", s: "posts" },
+ { q: "kappa", s: "posts" },
]);
await page.goto(`/?qt=${tree}&tg=${TAG_COLLAB}`);
await waitForHydration(page);
@@ -261,7 +263,7 @@ test.describe("curated tag chips", () => {
// tag filter, never by this change.
const tree = qt([
{ q: "chat", s: "metadata" },
- { q: "alpha", s: "posts" },
+ { q: "kappa", s: "posts" },
]);
await page.goto(`/?qt=${tree}`);
await waitForHydration(page);
@@ -378,6 +380,7 @@ test.describe("curated tag chips", () => {
await installRoutes(page);
await page.goto("/");
await waitForHydration(page);
+ await showAll(page);
await expect(card(page, VIDEO_TRANSCRIPT_ONLY)).toBeVisible();
await expect(chipRow(page)).toHaveCount(0);
diff --git a/export/e2e/theme-accent.spec.ts b/export/e2e/theme-accent.spec.ts
@@ -1,17 +1,19 @@
+import fs from "node:fs";
+import path from "node:path";
import { test, expect, type Page } from "@playwright/test";
import { ACCENTS, MIN_ACCENT_CONTRAST, contrastRatio } from "../../common/lib/brand";
import { resolveAccent } from "../../common/lib/accent";
import { installRoutes } from "./helpers";
-// The ThemeMenu's two radio groups. A site opens in its OWN accent (the
-// fixture's site.json sets the custom hex #cc3366, so the layout renders
-// data-accent="custom" and the menu offers "Site colour" first, tagged
-// "default"). A reader's pick of a named accent persists and is applied before
-// paint; picking the site's own again REMOVES the stored key, so the reader
-// follows the site from then on. The base group sits in the same menu.
+// A site shows its OWN accent, and a reader does not pick one. The fixture's
+// site.json sets the custom hex #cc3366, so the layout renders
+// data-accent="custom" with the hex fitted to each ground inline; `--brand`
+// follows the base in force. A reader's stored pick from an earlier build is
+// ignored — before paint and after hydration — and left where it is.
const ACCENT_KEY = "ytdlp-tb:accent";
-const FIXTURE = resolveAccent("#cc3366"); // {id: "custom", light, sepia, dark}
+const BASE_KEY = "ytdlp-tb:base";
+const FIXTURE = resolveAccent("#cc3366"); // {id: "custom", light, dark}
async function state(page: Page) {
return page.evaluate((key) => {
@@ -27,136 +29,82 @@ async function state(page: Page) {
}, ACCENT_KEY);
}
-// Open the menu, absorbing a pre-hydration lost click on the trigger.
-async function openMenu(page: Page) {
- const trigger = page.getByRole("button", { name: "Choose theme" });
- const menu = page.getByRole("menu");
- await expect(async () => {
- if (!(await menu.isVisible())) await trigger.click();
- await expect(menu).toBeVisible({ timeout: 1_000 });
- }).toPass({ timeout: 10_000 });
- return menu;
-}
-
-// An item's accessible name is its label, plus "default" on the site's own
-// accent (the visible tag) — so match the label at the start.
-async function pick(page: Page, label: string) {
- const menu = await openMenu(page);
- await menu.getByRole("menuitemradio", { name: new RegExp(`^${label}\\b`) }).click();
- await expect(menu).toBeHidden();
+async function onBase(page: Page, base: "light" | "dark") {
+ await page.evaluate(([k, b]) => localStorage.setItem(k, b), [BASE_KEY, base]);
+ await page.reload({ waitUntil: "commit" });
+ await page.waitForFunction(() => document.documentElement?.dataset.themeReady === "1");
+ await page.waitForLoadState("load");
}
-test("the menu offers four bases and the site's colour first, then the seven accents", async ({
- page,
-}) => {
- await page.emulateMedia({ colorScheme: "light" });
- await page.goto("/");
- const menu = await openMenu(page);
-
- const base = menu.getByRole("group", { name: "Base" });
- for (const name of ["System", "Light", "Sepia", "Dark"]) {
- await expect(base.getByRole("menuitemradio", { name, exact: true })).toBeVisible();
- }
- await expect(base.getByRole("menuitemradio", { name: "System", exact: true })).toHaveAttribute(
- "aria-checked",
- "true",
- );
-
- const accent = menu.getByRole("group", { name: "Accent" });
- const items = accent.getByRole("menuitemradio");
- await expect(items).toHaveCount(8);
- // The custom-hex site's own colour comes first, checked, with its tag.
- await expect(items.first()).toHaveText(/Site colour\s*default/);
- await expect(items.first()).toHaveAttribute("aria-checked", "true");
- for (const a of Object.values(ACCENTS)) {
- await expect(accent.getByRole("menuitemradio", { name: a.name, exact: true })).toBeVisible();
- }
-});
-
-test("pick Violet: it persists before paint, and --brand is violet on each base", async ({
+test("a site wears its own accent, fitted to each base, and offers no accent picker", async ({
page,
}) => {
await page.emulateMedia({ colorScheme: "light" });
await page.goto("/");
await expect.poll(() => state(page)).toMatchObject({
accent: "custom",
+ base: "light",
stored: null,
brand: FIXTURE.light,
});
+ await expect(page.getByRole("button", { name: "Choose theme" })).toHaveCount(0);
+ await expect(page.getByRole("menuitemradio")).toHaveCount(0);
- await pick(page, "Violet");
- await expect.poll(() => state(page)).toMatchObject({
- accent: "violet",
- stored: "violet",
- base: "light",
- brand: ACCENTS.violet.onLight,
- });
-
- // The reload applies the pick on the first commit, before React hydrates —
- // and hydration does not put the site's own accent back.
- await page.reload({ waitUntil: "commit" });
- await page.waitForFunction(() => document.documentElement?.dataset.themeReady === "1");
- expect((await state(page)).accent).toBe("violet");
- await page.waitForLoadState("load");
- await expect.poll(() => state(page)).toMatchObject({ accent: "violet", brand: ACCENTS.violet.onLight });
-
- // Sepia, from the same menu: the ground changes and the accent keeps its
- // on-sepia value.
- await pick(page, "Sepia");
- await expect.poll(() => state(page)).toMatchObject({
- base: "sepia",
- accent: "violet",
- background: "#f4ecd8",
- brand: ACCENTS.violet.onSepia,
- });
- expect(await page.evaluate(() => localStorage.getItem("ytdlp-tb:base"))).toBe("sepia");
-
- await pick(page, "Dark");
+ await onBase(page, "dark");
await expect.poll(() => state(page)).toMatchObject({
+ accent: "custom",
base: "dark",
background: "#0c0a08",
- brand: ACCENTS.violet.onDark,
+ brand: FIXTURE.dark,
});
});
-test("picking the site's own colour again removes the stored accent", async ({
+// Every value `data-accent` ever holds is recorded by a MutationObserver
+// installed before the page's first script, so a flash between two samples
+// cannot pass.
+test("a stored accent from before is ignored, with no flash, and left in place", async ({
page,
}) => {
await page.emulateMedia({ colorScheme: "light" });
- await page.goto("/");
- await pick(page, "Brass");
- await expect.poll(() => state(page)).toMatchObject({ accent: "brass", stored: "brass" });
-
- await pick(page, "Site colour");
- await expect.poll(() => state(page)).toMatchObject({
- accent: "custom",
- stored: null,
- brand: FIXTURE.light,
- });
-
- // With nothing stored, the site's colour is fitted to each ground.
- await pick(page, "Sepia");
- await expect.poll(() => state(page)).toMatchObject({ base: "sepia", brand: FIXTURE.sepia });
- await pick(page, "Dark");
- await expect.poll(() => state(page)).toMatchObject({ base: "dark", brand: FIXTURE.dark });
+ await page.addInitScript((key) => {
+ try {
+ localStorage.setItem(key, "violet");
+ } catch {}
+ const held: (string | null)[] = [];
+ (window as unknown as { __accents: (string | null)[] }).__accents = held;
+ new MutationObserver((records) => {
+ for (const r of records) if (r.target === document.documentElement) held.push(r.oldValue);
+ }).observe(document, {
+ subtree: true,
+ attributes: true,
+ attributeFilter: ["data-accent"],
+ attributeOldValue: true,
+ });
+ }, ACCENT_KEY);
+ await page.goto("/", { waitUntil: "commit" });
+ await page.waitForFunction(() => document.documentElement?.dataset.themeReady === "1");
+ expect((await state(page)).accent).toBe("custom");
+ await page.waitForLoadState("load");
+ await page.waitForTimeout(300);
+ const held = await page.evaluate(() => [
+ ...(window as unknown as { __accents: (string | null)[] }).__accents,
+ document.documentElement.getAttribute("data-accent"),
+ ]);
+ expect(held.filter((v) => v !== null && v !== "custom"), JSON.stringify(held)).toEqual([]);
+ expect(await state(page)).toMatchObject({ accent: "custom", stored: "violet", brand: FIXTURE.light });
});
// The search bar's "Press Enter or click Search to apply" wears the site's
// accent (release 10). It was `text-warning`, the same yellow on every site.
-// It follows the reader's accent and base like any `text-brand`, and stays
-// readable on the page ground it sits on.
-test("the unapplied-search hint wears the accent, readable on each base", async ({
+// It follows the base like any `text-brand`, and stays readable on the page
+// ground it sits on.
+test("the unapplied-search hint wears the site's accent, readable on each base", async ({
page,
}) => {
await installRoutes(page);
await page.emulateMedia({ colorScheme: "light" });
await page.goto("/");
const hint = page.getByText("Press Enter or click Search to apply");
- // Typing is an unapplied edit (the input mounts after hydration).
- await page.locator('input[data-testid^="leaf-query-"]').first().fill("alpha");
- await expect(hint).toBeVisible();
- await expect(hint).toHaveClass(/(^|\s)text-brand(\s|$)/);
- await expect(hint).not.toHaveClass(/text-warning/);
// The hint's colour and the ground under it, as "#rrggbb".
const paint = () =>
@@ -173,19 +121,65 @@ test("the unapplied-search hint wears the accent, readable on each base", async
};
});
const expectAccent = async (value: string) => {
+ // Typing is an unapplied edit (the input mounts after hydration).
+ await page.locator('input[data-testid^="leaf-query-"]').first().fill("alpha");
+ await expect(hint).toBeVisible();
+ await expect(hint).toHaveClass(/(^|\s)text-brand(\s|$)/);
+ await expect(hint).not.toHaveClass(/text-warning/);
await expect.poll(async () => (await paint()).color).toBe(value);
const { color, ground } = await paint();
expect(contrastRatio(color, ground)).toBeGreaterThanOrEqual(MIN_ACCENT_CONTRAST);
};
- // The fixture site's own (custom) colour, fitted to the light ground…
+ // The fixture site's own (custom) colour, fitted to each ground.
await expectAccent(FIXTURE.light);
- // …a reader's pick…
- await pick(page, "Brass");
- await expectAccent(ACCENTS.brass.onLight);
- // …and each base's own value of it.
- await pick(page, "Sepia");
- await expectAccent(ACCENTS.brass.onSepia);
- await pick(page, "Dark");
- await expectAccent(ACCENTS.brass.onDark);
+ await onBase(page, "dark");
+ await expectAccent(FIXTURE.dark);
+});
+
+// A site with a NAMED accent (the fixture's is a custom hex): its accent's own
+// value on each base, and a reader's stored pick from before does not override
+// it. The fixture site is rewritten for the test and put back after; the
+// original rides along in `_e2ePristine`, which playwright.config.ts restores
+// if a run dies mid-test.
+test.describe("a site with a named accent", () => {
+ const SITE_FILE = path.resolve(process.cwd(), "e2e", "fixtures", "sites", "testsite", "site.json");
+ let pristine = "";
+ test.describe.configure({ mode: "serial" });
+ test.beforeAll(() => {
+ pristine = fs.readFileSync(SITE_FILE, "utf8");
+ });
+ test.afterEach(() => {
+ fs.writeFileSync(SITE_FILE, pristine);
+ });
+
+ test("wears its accent's value on Light and on Dark, and a stored pick does not override it", async ({
+ page,
+ }) => {
+ const site = JSON.parse(pristine) as Record<string, unknown>;
+ fs.writeFileSync(
+ SITE_FILE,
+ `${JSON.stringify({ ...site, accent: "violet", _e2ePristine: pristine }, null, 2)}\n`,
+ );
+ await page.addInitScript((key) => {
+ try {
+ localStorage.setItem(key, "brass");
+ } catch {}
+ }, ACCENT_KEY);
+ await page.emulateMedia({ colorScheme: "light" });
+ await page.goto("/");
+ await expect.poll(() => state(page)).toMatchObject({
+ accent: "violet",
+ base: "light",
+ stored: "brass",
+ brand: ACCENTS.violet.onLight,
+ });
+ await onBase(page, "dark");
+ await expect.poll(() => state(page)).toMatchObject({
+ accent: "violet",
+ base: "dark",
+ stored: "brass",
+ brand: ACCENTS.violet.onDark,
+ });
+ });
});
diff --git a/export/e2e/theme.spec.ts b/export/e2e/theme.spec.ts
@@ -1,8 +1,9 @@
import { test, expect, type Locator, type Page } from "@playwright/test";
+import { RETIRED_BASE } from "../../common/components/themeConfig";
// A published site opens on the reader's SYSTEM base (tokens.css resolves it
// to the light or dark block) and follows the OS live. The base toggle cycles
-// system → light → sepia → dark → system; every choice persists across a
+// system → light → dark → system; every choice persists across a
// reload and is applied by the pre-paint ThemeScript before hydration — the
// reload waits on `data-theme-ready`, which the script sets last.
@@ -59,7 +60,7 @@ test("a site opens on the system base and follows the OS live", async ({
await expect.poll(async () => (await state(page)).base).toBe("light");
});
-test("the toggle cycles the four bases; each persists across a reload with no flash", async ({
+test("the toggle cycles the three bases; each persists across a reload with no flash", async ({
page,
}) => {
await page.emulateMedia({ colorScheme: "light" });
@@ -68,8 +69,7 @@ test("the toggle cycles the four bases; each persists across a reload with no fl
await expect(toggle).toBeVisible();
const steps = [
- { stored: "light", base: "light", dark: false, background: "#f3f6f7", next: "Switch to sepia" },
- { stored: "sepia", base: "sepia", dark: false, background: "#f4ecd8", next: "Switch to dark" },
+ { stored: "light", base: "light", dark: false, background: "#f3f6f7", next: "Switch to dark" },
{ stored: "dark", base: "dark", dark: true, background: "#0c0a08", next: "Switch to system" },
// Back to system: stored as "system", resolved against the (light) OS.
{ stored: "system", base: "light", dark: false, background: "#f3f6f7", next: "Switch to light" },
@@ -110,6 +110,49 @@ test("the retired theme/mode keys migrate once, before paint", async ({
theme: localStorage.getItem("ytdlp-tb:theme"),
mode: localStorage.getItem("ytdlp-tb:mode"),
}));
- // archive + light was the paper look: it becomes sepia, and the old keys go.
- expect(s).toEqual({ base: "sepia", stored: "sepia", theme: null, mode: null });
+ // archive + light was the paper look, the retired third ground: it becomes
+ // light, and the old keys go.
+ expect(s).toEqual({ base: "light", stored: "light", theme: null, mode: null });
+});
+
+// The retired third ground: a reader who chose it gets Light, before first
+// paint, with no other ground on the way, and the stored value becomes
+// "light" once. Every value `data-base` holds is recorded by a
+// MutationObserver installed before the page's first script.
+test("a stored retired base renders Light, with no other ground on the way, and is rewritten", async ({
+ page,
+}) => {
+ await page.emulateMedia({ colorScheme: "dark" });
+ await page.addInitScript(
+ ([key, retired]) => {
+ try {
+ localStorage.setItem(key, retired);
+ } catch {}
+ const held: (string | null)[] = [];
+ (window as unknown as { __bases: (string | null)[] }).__bases = held;
+ new MutationObserver((records) => {
+ for (const r of records) if (r.target === document.documentElement) held.push(r.oldValue);
+ }).observe(document, {
+ subtree: true,
+ attributes: true,
+ attributeFilter: ["data-base"],
+ attributeOldValue: true,
+ });
+ },
+ [BASE_KEY, RETIRED_BASE],
+ );
+ await page.goto("/", { waitUntil: "commit" });
+ await page.waitForFunction(() => document.documentElement?.dataset.themeReady === "1");
+ expect(await page.evaluate(() => document.documentElement.getAttribute("data-base"))).toBe("light");
+ await page.waitForLoadState("load");
+ await page.waitForTimeout(300);
+ const held = await page.evaluate(() => [
+ ...(window as unknown as { __bases: (string | null)[] }).__bases,
+ document.documentElement.getAttribute("data-base"),
+ ]);
+ // None (a site's server markup has no base: it is the reader's system), then
+ // light — never the retired ground, never dark from the OS.
+ expect(held.filter((v) => v !== null && v !== "light"), JSON.stringify(held)).toEqual([]);
+ expect(held.at(-1)).toBe("light");
+ expect(await state(page)).toMatchObject({ base: "light", dark: false, stored: "light" });
});
diff --git a/export/e2e/use-with-ai-link.spec.ts b/export/e2e/use-with-ai-link.spec.ts
@@ -0,0 +1,49 @@
+import { expect, test, type Locator } from "@playwright/test";
+import { AI_DOC_URL } from "../../common/lib/project";
+import { installRoutes } from "./helpers";
+
+// Release 16 slice DX: a site has no /use-with-ai page. Its "Use with AI"
+// links — the header's nav (inline from `lg`, in the slide-out menu below it),
+// the footer's and the one on Ask AI — go to the homepage's AI and MCP doc
+// (AI_DOC_URL), in the same tab, as plain anchors: the doc is on another
+// origin, so there is no client navigation to make. The label is unchanged.
+
+async function expectDocLink(link: Locator) {
+ await expect(link).toHaveAttribute("href", AI_DOC_URL);
+ await expect(link).not.toHaveAttribute("target", /./);
+}
+
+test.beforeEach(async ({ page }) => {
+ await installRoutes(page);
+});
+
+test("the header's and the footer's Use with AI go to the homepage doc", async ({ page }) => {
+ await page.goto("/");
+ await expectDocLink(
+ page.getByRole("banner").getByRole("link", { name: "Use with AI", exact: true }),
+ );
+ await expectDocLink(
+ page.getByRole("contentinfo").getByRole("link", { name: "Use with AI", exact: true }),
+ );
+});
+
+test("the slide-out menu's Use with AI goes to the homepage doc", async ({ page }) => {
+ await page.setViewportSize({ width: 390, height: 844 });
+ await page.goto("/");
+ await page.getByRole("button", { name: "Open menu" }).click();
+ await expectDocLink(
+ page.getByRole("dialog").getByRole("link", { name: "Use with AI", exact: true }),
+ );
+});
+
+test("Ask AI's Use with AI goes to the homepage doc", async ({ page }) => {
+ await page.goto("/ask/");
+ await expectDocLink(
+ page.getByRole("main").getByRole("link", { name: "Use with AI", exact: true }),
+ );
+});
+
+test("the page is gone", async ({ page }) => {
+ const res = await page.request.get("/use-with-ai/");
+ expect(res.status()).toBe(404);
+});
diff --git a/export/e2e/workspace-shell.spec.ts b/export/e2e/workspace-shell.spec.ts
@@ -5,7 +5,7 @@ import {
VIDEO_CHAT_SMALL,
VIDEO_TRANSCRIPT_ONLY,
} from "./fixtures/data";
-import { installRoutes } from "./helpers";
+import { installRoutes, showAll } from "./helpers";
// The unified workspace shell: `/` (results) and `/ask` (chat) share one layout,
// so the persistent search bar and its committed search survive navigation
@@ -61,9 +61,10 @@ test.describe("workspace shell", () => {
"aria-current",
"page",
);
- // Wait for the results pane to hydrate (browse mode lists every video) before
- // clicking — a client Link click lost to a pre-hydration window would leave us
- // stranded on `/`.
+ // Wait for the results pane to hydrate (an empty Search lists every video)
+ // before clicking — a client Link click lost to a pre-hydration window would
+ // leave us stranded on `/`.
+ await showAll(page);
await expect(page.locator("[data-card-header]").first()).toBeVisible();
await nav.getByRole("link", { name: "Chat" }).click();
await expect(page.getByPlaceholder(/Ask about the transcripts/)).toBeVisible();
diff --git a/export/playwright.config.ts b/export/playwright.config.ts
@@ -28,11 +28,16 @@ buildFixtureSettings(TEST_SETTINGS_FILE);
// fixture site for one test and restores it afterwards. A run killed inside that
// test leaves the key behind, and every modal spec before it would then fail on
// the missing controls — so strip it here, before any spec runs.
+// header.spec rewrites the fixture's title and social links for a test and
+// carries the original file in `_e2ePristine` (parseSite drops the key): a run
+// killed inside it is put back here the same way.
const FIXTURE_SITE = path.join(TEST_SITES_DIR, "testsite", "site.json");
{
const raw = fs.readFileSync(FIXTURE_SITE, "utf8");
const site = JSON.parse(raw) as Record<string, unknown>;
- if ("transcriptDownloads" in site) {
+ if (typeof site._e2ePristine === "string") {
+ fs.writeFileSync(FIXTURE_SITE, site._e2ePristine);
+ } else if ("transcriptDownloads" in site) {
delete site.transcriptDownloads;
fs.writeFileSync(FIXTURE_SITE, `${JSON.stringify(site, null, 2)}\n`);
}
diff --git a/homepage/CHANGELOG.md b/homepage/CHANGELOG.md
@@ -1,23 +1,39 @@
# Homepage Changelog
## [Unreleased]
+- **The AI and MCP doc has a Ten-minute setup.** Right after the MCP server's introduction, one block runs Claude Code against a published archive, the Jeralyzer as the example: clone the source (or unpack the tarball on Downloads), `pnpm install`, `claude mcp add archilyzer`, start `claude` and try `/ask`; then what it needs, why the server must be registered as `archilyzer` (the shipped `/ask` and `/sweep` call `mcp__archilyzer__…`), the two optional editor lines for `fetch_clip`, `TRANSCRIPT_HUB_URL`, where the `mcp.json` form for other clients is, and WSL2 on Windows. "What it can do" is a heading of its own after it. Every archive's **Use with AI** link now lands on this page.
+- **`/source/` links the source's history.** A History block — how many commits (past 10,000, "the latest 10,000 of N"), the newest one (linking to its page), and links to the Log, the Refs and the Atom feed — shows when the build published the history pages (`/source/git/`, rendered by stagit); without them there is no History block. The history pages open on the homepage's ground (the reader's stored choice, else Dark; without JavaScript, the system's), start with one line back to `/source/`, and their Files page is an index into the raw tree. The e2e shows the page with and without a history from a fixture publish (`E2E_SOURCE_PUBLIC_DIR`, never read by a production build), and walks the real pages when the checkout has published them.
+- **The growth chart draws its smallest instances together as Other.** Two or more instances that each hold under 5% of the chart's total are one band, **Other**, on top of the stack, in a near-neutral grey of its own (`--chart-other`: 7.36:1 on the Light ground, 3.22:1 on the Dark one, and apart from every instance colour for colour-blind readers); an instance at exactly 5% keeps its band, and a single one under 5% is not grouped. The other instances keep their bands and their colours. The legend lists them and Other; the caption says what Other is and the chart's description names the instances in it; every month's hover title and the Numbers by year table still name every instance. At this release's numbers Hasanalyzer, Rekietalyzer and Jasolyzer are Other. The instance cards and `/stats` are unchanged. The e2e fixture's fifth site transcribes 4 a day rather than 5, so two of its six sites are grouped.
+- **An unlisted site is not on the homepage.** A site whose settings turn off **List on the Archilyzer homepage and hub** (`listed: false`) has no Official Instances card, chart series, `/stats` entry or recent item, is not in `channel-sites.json` or `stats/`, and the channels only it carries count in none of the numbers, the headline totals included. The summary's version is 6. The e2e fixture has a seventh, unlisted site that no page names.
+- **`/#instances` goes straight to Official Instances.** The section carries `id="instances"`, clear of the sticky header, and every archive's header now links there (`INSTANCES_URL` in `common/lib/project.ts`). With no sites the link lands on the top of the page.
+- **The social links are in the header beside the theme toggle, and on a small screen the header keeps the name.** The operator's social icons (`homepage.json`'s, else `settings.json`'s) sit in the header's bar as well as in the footer's Elsewhere column, followed by the theme toggle, all spaced alike. From 768 px wide the bar is wordmark, nav, icons, toggle; below 768 px it is wordmark, icons, toggle, and the nav has the rule below to itself, where its four links fit. From 520 px wide the header shows every link, up to four (with more, the ones marked **Keep in header on small screens** first, then the last of the rest); below 520 px it shows only the marked ones (none marked → none; the switch is 32.5rem, so at a larger text size it comes later), and the wordmark keeps its full name: "Archilyzer" shows from 292 px wide on a touch screen with one marked link. The footer always shows every link. Only as last resorts, on a screen narrower still or at a much larger text size, does the wordmark's text drop (its mark stays) and do the icons scroll sideways in their own box, the last one in view first. Each is an icon named by its label, with no text beside it.
+- **One theme toggle in place of the two theme buttons.** The header's theme menu (Base and Accent) and its base toggle are one toggle, the last of the header's icons, that cycles the ground (System, Light, Dark) and is named for the next one. There is no accent to pick: the homepage's is its own, Signal, and an accent stored by an earlier build is ignored here, with no flash.
+- **Two grounds, Light and Dark.** The third ground, the warm paper one, is gone: the toggle cycles System, Light and Dark. A reader who had chosen it gets Light, before the page first paints and with no other ground on the way, and the stored choice becomes Light.
+- **Changelog is in the footer only.** The header's nav is Docs, Source, Downloads and Stats; the footer's Sections list and the 404 page keep Changelog.
+- **Each Official Instances card names its site with the site's wordmark.** The first part of the name is set heavy in the site's own accent and the rest light, as the site's own header sets it (Jer·alyzer, Hasan·alyzer, …), at the card title's size; the accent is fitted to the ground in force and reads above 4:1 on the card on Light and Dark. A site with no configured lead, or a summary built before this, shows its title plain as before. `homepage-summary.json` gains an optional `wordmarkLead` per site (still version 5).
+- **The growth chart's bands are separated by a 2 px gap in the ground's colour, where both bands can spare it.** The gap runs along a band's upper edge where another band sits on it, the same 2 px at every width (it was a 1.25 px line in the page colour). It is drawn only where both bands keep at least a pixel of their own colour, measured at right angles to the edge at the narrowest width the chart is drawn at: on a steep month, and beside the thinnest instances, the bands touch instead, so no band is ever covered, and the gap shows in stretches rather than all along. In high-contrast mode the gap is the system's background colour. The gridlines are unchanged. The `/stats` charts' stacked bars get the same 2 px gap in the chart panel's colour; their stacked areas keep their coloured top lines.
+- **Larger social links, with a focus ring.** Each icon, in the header and the footer, is a 36 px target around its 20 px glyph (44 px on a touch screen), in the muted text colour and the text colour on hover; the footer's were 20 px, in the faint colour, with no ring. Keyboard focus draws a 2 px ring in the accent, and in high-contrast mode the browser's own focus outline. An icon of two or more colours keeps its colours, and every icon paints inside its own box. A stored icon that fails the check a save runs is shown as its label (at most 10rem, with an ellipsis) instead.
+- **Tab no longer stops on the header's scrolling boxes in Firefox.** When the social icons' box or the nav's rule under the header overflows (a screen under about 300 px, or a large text size), Firefox made it a tab stop with no name of its own; the icons and links inside are the stops now, and each scrolls into view as it takes focus.
+- **The e2e no longer reads the checkout's `settings.json` or `homepage.json`.** Its dev server reads `e2e/.e2e-settings.json` (`SETTINGS_FILE`), written by `e2e/fixture-social.ts`: three synthetic icons (a gradient with an outline, one colour, and a two-colour disc pasted with only its size), put through the same check a save runs; and `SITES_DIR` points at an empty directory. `e2e/social.spec.ts` covers the header and footer rows (at every width, with 1, 3 and 4 links, both pointers; the scroll fallback; hostile stored icons that must neither run nor fetch), `e2e/toggle.spec.ts` the theme toggle and that a stored accent is ignored, `e2e/svg-vectors.spec.ts` that every accepted icon stays inside its `<svg>` in a real parse, `e2e/growth-chart.spec.ts` the chart's gaps and that it paints its true peak and every band, and `e2e/instance-wordmark.spec.ts` the cards' names; specs change the ground through one helper, `chooseTheme` (`e2e/helpers.ts`).
+- **A site's card counts every transcript, and never shows 0 channels while it serves recordings.** A transcript that arrived after its video was first indexed, or a video with YouTube captions alone, could be left out of the family's numbers: one site served 1,889 recordings and its card said 0 transcripts, 0 channels and 0 hours. Such transcripts are counted now — in the card, the family totals and the archive-growth chart — and one with no transcription date is left off only what is placed by that date: the charts by transcription date, "this month" and the recent list. The official-instance figures on the hub move with them.
+- **The source is on the site, with its history: `/source/`.** A new **Source** page (and nav entry) gives `git clone https://archilyzer.pages.dev/source/archilyzer.git`, a read-only mirror of the main branch regenerated with every deploy, with its head, the private commit it reflects, a link to browse every file raw at `/source/tree/`, and the tarball with its size and sha256. Commit ids differ from the private repository's, because machine paths are scrubbed on the way out, and the page says so. A build without a published source says "No source published in this build." instead of offering a clone. The Downloads tarball is now regenerated by every build (its commit is the mirror's), and the page points at the mirror for history. The docs that said there is no public repository (*Install*, the FAQ, *What is Archilyzer*) now say how to clone. Below `md` the header's nav drops to its own row, as it did below `sm`, because five labels no longer fit beside the wordmark. `_headers` serves the raw tree as plain text.
- **The docs' *Building several sites at once* page says what Build all does.** It called the container pipeline opt-in, turned on in the settings. Build all sites builds every site in parallel in containers whenever a container engine is available, and one after another when none is; there is nothing to switch on.
- **A single-colour social icon shows on every ground.** The footer's social icons are the operator's (`homepage.json`'s, else `settings.socialLinks`), normalized when they are saved (`normalizeSocialSvg`, release 11 slice O1). An icon drawn in one colour now takes the footer's colour throughout; before, a part that carried its own colour kept it, so X's official logo, which is white, was invisible on the Light ground. An icon of two or more colours, such as YouTube's red mark with its white triangle, keeps its colours as pasted. "No fill", gradients, masks, clip paths and animation timing are never changed, and a clip path's own colour does not count, so a one-colour icon exported from Figma follows the footer too. It applies when the settings are next saved, then needs a rebuild and deploy of the homepage.
- **In high-contrast mode the header mark's tile keeps its edge.** In Windows' high-contrast mode (forced colours) the reader's own background replaces the page on every ground and can be as dark as the slate tile, whose ring is only drawn on Dark. In that mode the tile gets a 1-pixel outline in the reader's text colour, on every ground, following its rounded corners. Nothing changes outside that mode.
-- **A sixth official instance has a chart colour of its own.** The growth chart, its legend and `/stats` had five validated colours, so a sixth site fell to a pink within a degree of the fifth's magenta. There is now a sixth, a rust (`--chart-6`: `#823c10` on Light and Sepia, `#a54a08` on Dark), which is Vermilion's hue family, so Jasolyzer's card and its layer will share a hue once it is published. It clears every pair with the other five on all three grounds for colour-blind readers (the dataviz validator, all pairs; worst CVD ΔE 9.1, normal 16.3). Any six sites now wear the six validated colours; `/stats`' sixth channel gets the rust too.
-- **A custom-hex accent reads on every ground.** An Official Instances card whose site sets its own hex painted it exactly as set, so a pale one was all but invisible on Light and Sepia. It is now fitted to each ground the way the site's own pages fit it (4.5:1, `resolveAccent`). No live site uses one.
+- **A sixth official instance has a chart colour of its own.** The growth chart, its legend and `/stats` had five validated colours, so a sixth site fell to a pink within a degree of the fifth's magenta. There is now a sixth, a rust (`--chart-6`: `#823c10` on Light, `#a54a08` on Dark), which is Vermilion's hue family, so Jasolyzer's card and its layer will share a hue once it is published. It clears every pair with the other five on each ground for colour-blind readers (the dataviz validator, all pairs; worst CVD ΔE 9.1, normal 16.3). Any six sites now wear the six validated colours; `/stats`' sixth channel gets the rust too.
+- **A custom-hex accent reads on every ground.** An Official Instances card whose site sets its own hex painted it exactly as set, so a pale one was all but invisible on Light. It is now fitted to each ground the way the site's own pages fit it (4.5:1, `resolveAccent`). No live site uses one.
- **The e2e no longer needs the operator's data.** It reads a synthetic summary built by the real summary builder (`e2e/fixture-summary.ts`, six sites, deterministic) instead of `public/homepage-summary.json`, so a fresh clone runs every spec instead of skipping eight. `e2e/fixture-accents.ts` is gone.
-- **On the Dark ground the header mark's tile has a thin outline.** Its slate tile now has a 1-pixel ring just outside it, following its rounded corners, in the colour of the mark's unlit lines, so the tile's edge shows against the dark page. Light and Sepia are unchanged, and so are the icons.
+- **On the Dark ground the header mark's tile has a thin outline.** Its slate tile now has a 1-pixel ring just outside it, following its rounded corners, in the colour of the mark's unlit lines, so the tile's edge shows against the dark page. Light is unchanged, and so are the icons.
- **The lines inside the Archilyzer mark are easier to see.** The header mark's and the icons' three unlit lines now read at 3:1 against the slate tile instead of 2:1.
-- **The homepage opens on the Dark ground in Signal, even with JavaScript off, and a reader can pick another ground or accent.**
- The theme menu has **Base** (System, Light, Sepia, Dark) and **Accent** (the seven named
- accents, Signal tagged *default*); the toggle cycles System → Light → Sepia → Dark. The old
- Archilyzer theme family is gone, and so are the other four: the dark ground is now the warm ink
- the archives used. Type is Archivo, IBM Plex Sans and IBM Plex Mono. The chart's third colour
- is a violet, well clear of the "gone" red, and the chart stacks its instances in palette order,
- so no two touching layers are hard to tell apart for a colour-blind reader. A stored theme from
- before carries over once. The docs' *Operate* page describes the accent and the reader's menu.
+- **The homepage opens on the Dark ground in Signal, even with JavaScript off, and a reader can pick another ground.**
+ The header's toggle cycles System → Light → Dark; the accent is the homepage's own,
+ Signal. The old Archilyzer theme family is gone, and so are the other four: the dark ground is
+ now the warm ink the archives used. Type is Archivo, IBM Plex Sans and IBM Plex Mono. The
+ chart's third colour is a violet, well clear of the "gone" red, and the chart stacks its
+ instances in palette order, so no two touching layers are hard to tell apart for a colour-blind
+ reader. A stored theme from before carries over once. The docs' *Operate* page describes the
+ accent.
- **The Found-line mark.** The header's CSS triangle is now the family's parent mark (bone
on slate) and "ARCHILYZER" is the split wordmark "Archi|lyzer" — heavy lead, light
suffix, no longer tracked uppercase; the link is still named "Archilyzer home". The
diff --git a/homepage/app/changelog/page.tsx b/homepage/app/changelog/page.tsx
@@ -22,7 +22,7 @@ export const metadata: Metadata = {
function loadChangelog(): string | null {
try {
return readFileSync(
- path.join(process.cwd(), "..", "export", "CHANGELOG.md"),
+ path.join(/* turbopackIgnore: true */ process.cwd(), "..", "export", "CHANGELOG.md"),
"utf8",
);
} catch {
diff --git a/homepage/app/components/ArchiveCards.tsx b/homepage/app/components/ArchiveCards.tsx
@@ -1,5 +1,10 @@
import type { HomepageSummarySite } from "yt-dlp-transcript-common/lib/homepageSummary";
-import { siteChartColors, siteColor } from "yt-dlp-transcript-common/lib/siteColor";
+import {
+ siteAccentColor,
+ siteChartColors,
+ siteColor,
+} from "yt-dlp-transcript-common/lib/siteColor";
+import { Wordmark } from "yt-dlp-transcript-common/components/Wordmark";
// The official instances: one card per public archive, its title linking out,
// its own numbers underneath. No description line under the title — the
@@ -10,7 +15,7 @@ import { siteChartColors, siteColor } from "yt-dlp-transcript-common/lib/siteCol
// siteColor() — a named accent as `var(--swatch-<id>)`, its value on the base
// in force, from the summary's `accentId`; a custom hex fitted to each base as
// the site's own pages fit it (release 11: a pale hex published as is was
-// ~1.4:1 on light and sepia); a site with no accent its chart colour. The
+// ~1.4:1 on light); a site with no accent its chart colour. The
// hub's official cards wear the same (lib/hubSummary.ts officialInstances).
//
// A CARD STILL READS AS ITS LAYER'S LEGEND, BY HUE. The growth chart draws
@@ -38,6 +43,28 @@ function Figure({ value, unit }: { value: number | undefined; unit: string }) {
);
}
+// THE NAME IS THE SITE'S WORDMARK when the summary carries its lead: the lead
+// heavy, the rest light (common/components/Wordmark.tsx, as the site's own
+// header sets it), at the card title's size. The lead is TINTED in the site's
+// own accent (siteAccentColor: a named accent's swatch, a custom hex fitted to
+// the base in force — above 4:1 on the card on every base); a site with no
+// accent keeps the foreground. The two spans are adjacent inline text, so the
+// link's accessible name stays the plain title. With no lead (none configured,
+// or a summary from before release 14) the title is plain, as it was. In forced
+// colours the tint is the system's text colour, like any text.
+function SiteName({ site }: { site: HomepageSummarySite }) {
+ if (!site.wordmarkLead) return <>{site.siteTitle}</>;
+ const accent = siteAccentColor(site);
+ return (
+ <Wordmark
+ title={site.siteTitle}
+ lead={site.wordmarkLead}
+ className="tracking-[-0.01em]"
+ leadStyle={accent ? { color: accent } : undefined}
+ />
+ );
+}
+
export function ArchiveCards({ sites }: { sites: HomepageSummarySite[] }) {
if (sites.length === 0) return null;
const chart = siteChartColors(sites);
@@ -60,7 +87,7 @@ export function ArchiveCards({ sites }: { sites: HomepageSummarySite[] }) {
rel="noopener noreferrer"
className="underline decoration-transparent underline-offset-4 transition-colors hover:decoration-[var(--border-strong)]"
>
- {site.siteTitle}
+ <SiteName site={site} />
<span aria-hidden="true" className="ml-1.5 text-sm text-[var(--faint)]">↗</span>
</a>
</h3>
diff --git a/homepage/app/components/ArchiveGrowthChart.tsx b/homepage/app/components/ArchiveGrowthChart.tsx
@@ -3,7 +3,16 @@ import type {
HomepageSummarySite,
} from "yt-dlp-transcript-common/lib/homepageSummary";
import { monthLabel } from "yt-dlp-transcript-common/lib/homepageChart";
-import { siteChartColors } from "yt-dlp-transcript-common/lib/siteColor";
+import {
+ FOLD_PERCENT,
+ OTHER_LABEL,
+ PLOT_SIZES,
+ gapSegments,
+ growthLayers,
+ growthStack,
+ layerColors,
+ runsOf,
+} from "../lib/growthGaps";
// The front page's showpiece: every official instance's back catalogue as
// stacked strata, one month per step, from the oldest upload to the last
@@ -35,44 +44,50 @@ import { siteChartColors } from "yt-dlp-transcript-common/lib/siteColor";
// The legend above the plot wears the same colours. The layers stack in the
// summary's order (STACK_ORDER, unchanged).
//
+// OTHER (release 14, slice CF; lib/growthGaps.ts). Two or more sites each
+// under 5 % of the chart's total fold into ONE band, "Other", on top of the
+// stack. The kept sites keep their slots — siteChartColors runs over EVERY
+// site, so a fold never repaints one — and Other wears its own near-neutral
+// grey, `--chart-other` (tokens.css: Light #3e545c, Dark #62625c; 7.36 /
+// 3.22:1 on the ground, 7.99 / 3.06:1 on the chart surface), not the axis
+// labels' colour. The legend shows the kept sites and Other; the hover title
+// and the table name every site, and the image's label and the caption say
+// what Other holds. A grey is below the validator's chroma floor by
+// definition (it is the de-emphasis role, not a categorical slot); against
+// each slot it can sit on, Other is (normal ΔE / worst of protan and deutan,
+// light · dark):
+// blue 21.9 / 21.4 · 17.8 / 17.9 — the only neighbour in today's data
+// (Bonnellyzer, the top kept site in every month Other has data);
+// green 17.4 / 15.2 · 19.2 / 15.7; violet 16.2 / 13.4 · 23.4 / 21.5;
+// amber 22.1 / 18.2 · 20.3 / 18.1; magenta 24.5 / 9.1 · 19.5 / 7.3;
+// rust 14.1 / 10.5 · 13.5 / 9.3.
+// Every CVD pair clears the floor (6), and on Light the target (8); Dark's
+// magenta is in the 6–8 band and rust is under the normal-vision 15 on both
+// bases: there the gap between the bands, Other's place on top, the legend
+// and the table carry it.
+//
// VALIDATED (the dataviz skill's validator, each base's values on its
// --chart-surface and on the page ground), with the operator's accents
// (Jeralyzer Brass, Anilyzer Sakura, Bonnellyzer Blue, Hasanalyzer Violet,
// Rekietalyzer Green; Jasolyzer Vermilion once it is published):
// • adjacent — the stack — in today's order (Jeralyzer, Anilyzer,
-// Bonnellyzer, Hasanalyzer, Rekietalyzer): PASS on every base, worst CVD
-// ΔE 13.1 / 11.1 / 11.0 and normal 17.7 / 15.3 / 16.7 (light / sepia /
-// dark; today's mapping — release 10's slice MC, re-run in release 11);
+// Bonnellyzer, Hasanalyzer, Rekietalyzer): PASS on both bases, worst CVD
+// ΔE 13.1 / 11.0 and normal 17.7 / 16.7 (light / dark; today's mapping —
+// release 10's slice MC, re-run in release 11);
// with Jasolyzer at any place in it: PASS, worst CVD
-// 12.5 / 9.1 / 11.0, normal 16.6 / 15.3 / 16.3 (release 11);
+// 12.5 / 11.0, normal 16.6 / 16.3 (release 11);
// • all pairs, all six: the palette's own borderline in any order — CVD 6.2
-// / 6.1 / 6.9 (green↔amber, green↔magenta: the 6–8 floor band, legal with
-// the legend, the surface-gap edges and the table), normal ≥ 15.3. The
-// sixth slot adds no pair under the target: its worst is CVD 9.1 (sepia,
-// against green), normal 16.3 (dark, against amber).
+// / 6.9 (green↔amber, green↔magenta: the 6–8 floor band, legal with the
+// legend, the surface-gap edges and the table), normal ≥ 15.3. The sixth
+// slot adds no pair under the target (measured on three bases, before the
+// third was retired): its worst normal pair is 16.3 (dark, against
+// amber).
// The accents' own values fail as a chart palette (Brass↔Vermilion ΔE 1.0
// deutan, Blue↔Violet 8.5 normal), which is why the chart wears their hue
// families rather than the accents.
const W = 1000;
const H = 300;
-const STACK_ORDER = [0, 1, 2, 3, 4];
-
-function stackOrder(n: number): number[] {
- const head = STACK_ORDER.filter((i) => i < n);
- const tail = Array.from({ length: Math.max(0, n - 5) }, (_, k) => k + 5);
- return [...head, ...tail];
-}
-
-// A clean tick step giving three or four gridlines under `max`.
-function niceStep(max: number): number {
- const raw = max / 3.5;
- const pow = 10 ** Math.floor(Math.log10(raw));
- for (const m of [1, 2, 2.5, 5, 10]) {
- if (m * pow >= raw) return m * pow;
- }
- return 10 * pow;
-}
export function ArchiveGrowthChart({
months,
@@ -84,39 +99,48 @@ export function ArchiveGrowthChart({
const n = months.length;
if (n === 0 || sites.length === 0) return null;
- const totals = months.map((m) =>
- sites.reduce((a, s) => a + (m.bySite[s.siteId] ?? 0), 0),
- );
- const peak = Math.max(...totals);
+ // The bands, bottom-up: the kept sites, then Other when two or more fold.
+ const {
+ totals,
+ peak,
+ yMax,
+ ticks,
+ layers: bands,
+ stack: stacked,
+ } = growthStack(months, sites, growthLayers(months, sites));
if (peak === 0) return null;
- const step = niceStep(peak);
- const yMax = Math.ceil(peak / step) * step;
- const ticks: number[] = [];
- for (let t = step; t <= yMax; t += step) ticks.push(t);
const x = (i: number) => (n === 1 ? W / 2 : (i / (n - 1)) * W);
const y = (v: number) => H - (v / yMax) * H;
const r = (v: number) => Math.round(v * 10) / 10;
- // One colour per site, in `sites` order (the legend's too).
- const colors = siteChartColors(sites);
+ // One colour per band, in stack order (the legend's too).
+ const colors = layerColors(bands, sites);
+ const folded = bands.find((b) => b.other)?.sites.map((si) => sites[si].siteTitle) ?? [];
- // Layers bottom-up, each with its lower and upper edge per month.
- const order = stackOrder(sites.length);
- const base = new Array<number>(n).fill(0);
- const layers = order.map((si) => {
- const site = sites[si];
- const lo = base.slice();
- const hi = months.map((m, i) => (base[i] += m.bySite[site.siteId] ?? 0));
+ const layers = stacked.map(({ lo, hi }, k) => {
const top = hi.map((v, i) => `${r(x(i))},${r(y(v))}`);
const bottom = lo.map((v, i) => `${r(x(i))},${r(y(v))}`).reverse();
return {
- site,
- color: colors[si],
+ key: bands[k].key,
+ label: bands[k].other ? OTHER_LABEL : sites[bands[k].sites[0]].siteTitle,
+ color: colors[k],
+ top,
area: `M${top.join("L")}L${bottom.join("L")}Z`,
- edge: `M${top.join("L")}`,
};
});
+ // The gaps, one set per plot height (lib/growthGaps.ts says where a gap is
+ // drawn, and why a thin band gets none).
+ const gapSets = PLOT_SIZES.map((size) => {
+ const segments = gapSegments(stacked, yMax, size);
+ const paths = layers.map((l, k) => ({
+ key: l.key,
+ d: runsOf(segments[k])
+ .map((run) => `M${[...run, run.at(-1)! + 1].map((i) => l.top[i]).join("L")}`)
+ .join(""),
+ }));
+ return { px: size.px, className: size.className, paths: paths.filter((p) => p.d) };
+ });
// Year ticks at each January. Every second year from sm up, every fourth on a
// phone; the ends are skipped so a label never hangs off the plot.
@@ -128,11 +152,18 @@ export function ArchiveGrowthChart({
const last = monthLabel(months[n - 1].month);
const peakIdx = totals.indexOf(peak);
const all = totals.reduce((a, b) => a + b, 0);
+ // The legend is hidden from assistive tech, so the image's label says what
+ // Other holds; the caption says what it is.
+ const otherNote = folded.length
+ ? `${folded.slice(0, -1).join(", ")} and ${folded.at(-1)}, each under ` +
+ `${FOLD_PERCENT}% of the total, are drawn together as ${OTHER_LABEL}.`
+ : "";
const label =
`Stacked area chart of ${all.toLocaleString()} transcripts by the month each ` +
`video was published, ${first} to ${last}, across ${sites.length} official ` +
`instances. The busiest month is ${monthLabel(months[peakIdx].month)}, ` +
- `with ${peak.toLocaleString()}.`;
+ `with ${peak.toLocaleString()}.` +
+ (otherNote ? ` ${otherNote}` : "");
// Per-year table for anyone who wants the numbers rather than the shape.
const byYear = new Map<string, Record<string, number>>();
@@ -145,10 +176,10 @@ export function ArchiveGrowthChart({
return (
<figure className="flex flex-col gap-4">
<ul className="flex flex-wrap gap-x-5 gap-y-2 list-none" aria-hidden="true">
- {sites.map((s, i) => (
- <li key={s.siteId} className="flex items-center gap-2 text-sm text-[var(--muted-foreground)]">
- <span className="h-2.5 w-2.5 shrink-0 rounded-[2px]" style={{ backgroundColor: colors[i] }} />
- {s.siteTitle}
+ {layers.map((l) => (
+ <li key={l.key} className="flex items-center gap-2 text-sm text-[var(--muted-foreground)]">
+ <span className="h-2.5 w-2.5 shrink-0 rounded-[2px]" style={{ backgroundColor: l.color }} />
+ {l.label}
</li>
))}
</ul>
@@ -178,23 +209,39 @@ export function ArchiveGrowthChart({
/>
))}
{layers.map((l) => (
- <path key={l.site.siteId} d={l.area} fill={l.color} />
+ <path key={l.key} d={l.area} fill={l.color} />
))}
- {/* The surface gap: each layer's upper edge is drawn in the page
- colour, so touching strata separate by negative space. */}
- {layers.map((l) => (
- <path
- key={`${l.site.siteId}-edge`}
- d={l.edge}
- fill="none"
- stroke="var(--background)"
- strokeWidth={1.25}
- strokeLinejoin="round"
- vectorEffect="non-scaling-stroke"
- />
+ {/* THE SURFACE GAP (the marks spec): touching bands are parted by
+ a 2 px gap in the colour behind the plot — the page ground, as
+ the chart sits on it — never by a line of their own. It runs
+ along each band's upper edge, centred, so each neighbour gives
+ 1 px. Where nothing sits on a band (the stack's top meets the
+ surface itself), or where either band is too thin, measured at
+ right angles to the edge at the narrowest plot of that height,
+ to give its pixel and keep one of its own colour (one set of
+ gaps per height, shown by its class; lib/growthGaps.ts), there
+ is no gap: thin bands touch rather than vanish. Colour and
+ width are
+ `.growth-gap` in globals.css (Canvas in forced colours), so
+ they follow the theme with no script. */}
+ {gapSets.map((set) => (
+ <g key={set.px} data-plot-height={set.px} className={set.className}>
+ {set.paths.map((p) => (
+ <path
+ key={p.key}
+ d={p.d}
+ className="growth-gap"
+ fill="none"
+ strokeLinejoin="round"
+ vectorEffect="non-scaling-stroke"
+ />
+ ))}
+ </g>
))}
{months.map((m, i) => {
const w = W / n;
+ // Every site with data that month, by name — a folded one too:
+ // the title and the table are where Other's sites are found.
const parts = sites
.map((s) => [s.siteTitle, m.bySite[s.siteId] ?? 0] as const)
.filter(([, v]) => v > 0)
@@ -246,6 +293,8 @@ export function ArchiveGrowthChart({
<figcaption className="text-sm text-[var(--muted-foreground)]">
Transcripts by the month each video was published, all official instances.
+ {folded.length > 0 &&
+ ` Instances under ${FOLD_PERCENT}% of the total are drawn together as ${OTHER_LABEL}.`}
</figcaption>
<details className="text-sm">
<summary className="cursor-pointer text-[var(--muted-foreground)] underline decoration-[var(--border-strong)] underline-offset-4 hover:text-[var(--foreground)]">
diff --git a/homepage/app/components/Fact.tsx b/homepage/app/components/Fact.tsx
@@ -0,0 +1,25 @@
+// One labelled fact in a page's fact list — the Downloads and Source pages'
+// rows. `testId` marks the value, for the specs that compare it with a file.
+export function Fact({
+ label,
+ children,
+ testId,
+}: {
+ label: string;
+ children: React.ReactNode;
+ testId?: string;
+}) {
+ return (
+ <div className="flex flex-col gap-1 border-t border-[var(--border)] py-3 sm:flex-row sm:gap-6">
+ <span className="label-machine sm:w-32 sm:shrink-0 sm:pt-0.5">
+ {label}
+ </span>
+ <span
+ data-testid={testId}
+ className="min-w-0 break-all text-sm text-[var(--muted-foreground)]"
+ >
+ {children}
+ </span>
+ </div>
+ );
+}
diff --git a/homepage/app/components/Footer.tsx b/homepage/app/components/Footer.tsx
@@ -1,15 +1,13 @@
import Link from "next/link";
-import {
- getSettings,
- sizeSocialSvg,
-} from "yt-dlp-transcript-common/lib/settings";
+import { getSettings } from "yt-dlp-transcript-common/lib/settings";
import { resolveHomepageSocialLinks } from "yt-dlp-transcript-common/lib/homepage";
+import { SocialLinks } from "yt-dlp-transcript-common/components/SocialLinks";
import {
PROJECT_NAME,
PROJECT_TAGLINE,
} from "yt-dlp-transcript-common/lib/project";
import { currentHomepage } from "../lib/homepage";
-import { NAV } from "../lib/nav";
+import { FOOTER_NAV } from "../lib/nav";
// The project site's footer. Note the identity split it embodies: the wordmark
// and tagline are PRODUCT strings (the same on every install), while the social
@@ -36,7 +34,7 @@ export default function Footer() {
<nav aria-label="Footer" className="flex flex-col gap-3">
<span className="label-machine">Sections</span>
<ul className="flex flex-col gap-2 list-none">
- {NAV.map((item) => (
+ {FOOTER_NAV.map((item) => (
<li key={item.href}>
<Link
href={item.href}
@@ -52,21 +50,7 @@ export default function Footer() {
{socialLinks.length > 0 && (
<div className="flex flex-col gap-3">
<span className="label-machine">Elsewhere</span>
- <ul className="flex items-center gap-4 list-none">
- {socialLinks.map((link, i) => (
- <li key={`${link.url}-${i}`}>
- <a
- href={link.url}
- title={link.label}
- aria-label={link.label}
- target="_blank"
- rel="noopener noreferrer"
- className="inline-block w-5 h-5 text-[var(--faint)] hover:text-[var(--foreground)] transition-colors [&_svg]:w-full [&_svg]:h-full"
- dangerouslySetInnerHTML={{ __html: sizeSocialSvg(link.svg) }}
- />
- </li>
- ))}
- </ul>
+ <SocialLinks links={socialLinks} placement="footer" />
</div>
)}
</div>
diff --git a/homepage/app/components/Header.tsx b/homepage/app/components/Header.tsx
@@ -7,13 +7,18 @@ import { ICON_PALETTES } from "yt-dlp-transcript-common/lib/brand";
import { BrandMark } from "yt-dlp-transcript-common/components/BrandMark";
import { Wordmark } from "yt-dlp-transcript-common/components/Wordmark";
import { ThemeToggle } from "yt-dlp-transcript-common/components/ThemeToggle";
-import { ThemeMenu } from "yt-dlp-transcript-common/components/ThemeMenu";
-import { NAV } from "../lib/nav";
+import { SocialLinks } from "yt-dlp-transcript-common/components/SocialLinks";
+import { SocialScroll } from "yt-dlp-transcript-common/components/SocialScroll";
+import { getSettings } from "yt-dlp-transcript-common/lib/settings";
+import { resolveHomepageSocialLinks } from "yt-dlp-transcript-common/lib/homepage";
+import { headerSocialLinks } from "yt-dlp-transcript-common/lib/socialLinks";
+import { currentHomepage } from "../lib/homepage";
+import { HEADER_NAV } from "../lib/nav";
function NavList({ className }: { className?: string }) {
return (
<ul className={`flex items-center list-none ${className ?? ""}`}>
- {NAV.map((item) => (
+ {HEADER_NAV.map((item) => (
<li key={item.href}>
<Link
href={item.href}
@@ -28,22 +33,82 @@ function NavList({ className }: { className?: string }) {
}
// The project site's header: a hairline bar with the wordmark and the four
-// destinations. It carried NO links at all before this — the page it sat above
-// was the whole site.
+// destinations (Changelog is in the footer only; lib/nav.ts). It carried NO
+// links at all before this — the page it sat above was the whole site.
//
// The wordmark is PRODUCT identity (common/lib/project.ts), not the operator's
// editable homepage config: this bar says what the software is called, and that
// is the same string on every install. The operator's own naming still governs
-// /stats/ and the social links in the footer.
+// /stats/ and the social links, which this bar carries as well as the footer.
+//
+// THE SOCIAL ROW AND THE THEME TOGGLE ARE ONE GROUP, in the bar at every width:
+// the operator's links (common/components/SocialLinks.tsx) and then the toggle
+// that cycles the base (common/components/ThemeToggle.tsx, `variant="bare"`),
+// dressed alike, their 36 px boxes touching, so every glyph is 16 px from the
+// next. There is no accent control: the homepage wears its own (layout.tsx).
+// THE HEADER KEEPS THE NAME ON A NARROW SCREEN (the ruling of 2026-09-28): below
+// the switch the row holds only the links marked `featured` (none marked →
+// none; the footer shows them all); from it the row holds every link, up to
+// four, the marked ones kept first (headerSocialLinks, "narrow" and "wide").
+// Both rows are rendered and CSS shows one; the other is display:none, so
+// exactly one is focusable and in the accessibility tree. The switch is
+// 32.5rem, the export's (520 px at the default text size, where every link,
+// the toggle and the longest real site title fit the export's bar, menu button
+// included; this bar is shorter), in rem so it moves with the reader's text
+// size as the keys and the wordmark do. With the wordmark
+// link at 148–152 px, the nav at 276 px and a key 36 px (44 px under a coarse
+// pointer):
+// ≥ md (768) wordmark · nav · group, the nav 32 px from the group's first
+// box (40 px from its first glyph).
+// < md wordmark · group, and the nav alone on the rule below.
+// THE LAST RESORTS, now rare: when the bar — a size container, `@container/bar`
+// — is still narrower than the full wordmark, a 12 px gap and the group the
+// bar shows, the wordmark's TEXT is hidden and the mark stays (the link keeps
+// its name, "Archilyzer home"); the class is chosen by the row's count from
+// WORDMARK_FITS — below the switch the narrow row's, from it the wide row's —
+// (the bar's content width in px at the default text size, 152 + 12 + (n + 1)
+// keys; the classes say it in rem):
+// links mouse (36 px keys) touch (44 px keys) viewport
+// 0 < 200 < 208 < 240 / < 248
+// 1 < 236 < 252 < 276 / < 292
+// 2 < 272 < 296 < 312 / < 336
+// 3 < 308 < 340 < 348 / < 380
+// 4 < 344 < 384 < 384 / < 424
+// And after that the row scrolls sideways inside the bar (SocialScroll), its
+// END shown first, the toggle outside it, the header never wider than the
+// screen; a link that takes focus is scrolled into view with its ring.
//
// The mark is the family's Found-line mark in its parent colours — bone on
// slate, no accent: the project spends no colour on its own chrome. The
// wordmark splits on the subject ("Archi" + "lyzer"); the link keeps its
// explicit name, "Archilyzer home".
+// The wordmark text's hiding classes, by how many links the header shows:
+// complete literal classes, one set per count (see the table above), in rem so
+// they scale with the reader's text size as the wordmark and the keys do. The
+// NARROW set applies below the switch and is keyed by the narrow row's count;
+// the WIDE set from the switch, keyed by the wide row's.
+const WORDMARK_FITS_NARROW: Record<number, string> = {
+ 0: "max-[32.5rem]:@max-[12.5rem]/bar:hidden pointer-coarse:max-[32.5rem]:@max-[13rem]/bar:hidden",
+ 1: "max-[32.5rem]:@max-[14.75rem]/bar:hidden pointer-coarse:max-[32.5rem]:@max-[15.75rem]/bar:hidden",
+ 2: "max-[32.5rem]:@max-[17rem]/bar:hidden pointer-coarse:max-[32.5rem]:@max-[18.5rem]/bar:hidden",
+ 3: "max-[32.5rem]:@max-[19.25rem]/bar:hidden pointer-coarse:max-[32.5rem]:@max-[21.25rem]/bar:hidden",
+ 4: "max-[32.5rem]:@max-[21.5rem]/bar:hidden pointer-coarse:max-[32.5rem]:@max-[24rem]/bar:hidden",
+};
+const WORDMARK_FITS_WIDE: Record<number, string> = {
+ 0: "min-[32.5rem]:@max-[12.5rem]/bar:hidden pointer-coarse:min-[32.5rem]:@max-[13rem]/bar:hidden",
+ 1: "min-[32.5rem]:@max-[14.75rem]/bar:hidden pointer-coarse:min-[32.5rem]:@max-[15.75rem]/bar:hidden",
+ 2: "min-[32.5rem]:@max-[17rem]/bar:hidden pointer-coarse:min-[32.5rem]:@max-[18.5rem]/bar:hidden",
+ 3: "min-[32.5rem]:@max-[19.25rem]/bar:hidden pointer-coarse:min-[32.5rem]:@max-[21.25rem]/bar:hidden",
+ 4: "min-[32.5rem]:@max-[21.5rem]/bar:hidden pointer-coarse:min-[32.5rem]:@max-[24rem]/bar:hidden",
+};
+
export default function Header() {
+ const socialLinks = resolveHomepageSocialLinks(currentHomepage(), getSettings());
+ const narrow = headerSocialLinks(socialLinks, "narrow").length;
+ const wide = headerSocialLinks(socialLinks, "wide").length;
return (
<header className="sticky top-0 z-20 border-b border-[var(--border)] bg-[var(--background)]/85 backdrop-blur-md">
- <div className="max-w-6xl mx-auto px-5 sm:px-6 flex h-14 items-center gap-6">
+ <div className="@container/bar max-w-6xl mx-auto px-5 sm:px-6 flex h-14 items-center gap-3 md:gap-6">
<Link
href="/"
className="flex items-center gap-2.5 shrink-0"
@@ -53,28 +118,53 @@ export default function Header() {
<Wordmark
title={PROJECT_NAME}
lead={PROJECT_WORDMARK_LEAD}
- className="text-[1.3rem] leading-none tracking-[-0.01em]"
+ className={`text-[1.3rem] leading-none tracking-[-0.01em] ${WORDMARK_FITS_NARROW[narrow] ?? WORDMARK_FITS_NARROW[4]} ${WORDMARK_FITS_WIDE[wide] ?? WORDMARK_FITS_WIDE[4]}`}
/>
</Link>
- <nav aria-label="Main" className="ml-auto hidden sm:block">
- <NavList className="gap-6" />
- </nav>
- <div className="ml-auto sm:ml-0 flex items-center gap-2">
- <ThemeMenu />
- <ThemeToggle />
+ <div className="ml-auto flex min-w-0 items-center gap-8">
+ <nav aria-label="Main" className="hidden md:block">
+ <NavList className="gap-6" />
+ </nav>
+ <div className="flex min-w-0 items-center">
+ {narrow > 0 && (
+ <SocialScroll className="min-[32.5rem]:hidden">
+ <SocialLinks
+ links={socialLinks}
+ placement="header"
+ width="narrow"
+ className="w-max px-1 [direction:ltr]"
+ />
+ </SocialScroll>
+ )}
+ {wide > 0 && (
+ <SocialScroll className="hidden min-[32.5rem]:block">
+ <SocialLinks
+ links={socialLinks}
+ placement="header"
+ width="wide"
+ className="w-max px-1 [direction:ltr]"
+ />
+ </SocialScroll>
+ )}
+ <ThemeToggle variant="bare" />
+ </div>
</div>
</div>
- {/* Below `sm` the bar has no room for four labels beside the wordmark and
- the theme controls, so the nav drops to its own scrollable rule rather
- than collapsing behind a menu button — four links do not earn a
- disclosure widget. Only one of the two is ever in the accessibility
- tree (the other is display:none), but they carry distinct labels so a
- test or a screen reader can never conflate them. */}
+ {/* Below `md` the nav drops to its own rule rather than collapsing behind
+ a menu button — four links do not earn a disclosure widget, and they
+ fit a 320 px rule. Only one of the two navs is ever in the
+ accessibility tree (the other is display:none), but they carry
+ distinct labels so a test or a screen reader can never conflate them.
+ `overflow-x-auto` only guards a larger text setting or a screen
+ under 320 px; nothing scrolls at the default size from 320 px.
+ `tabIndex={-1}`: Firefox makes a scroll container that overflows a
+ tab stop of its own; the links are the stops (as SocialScroll). */}
<nav
aria-label="Main, compact"
- className="sm:hidden border-t border-[var(--border)] overflow-x-auto"
+ tabIndex={-1}
+ className="md:hidden border-t border-[var(--border)] overflow-x-auto"
>
- <NavList className="gap-5 px-5 h-10" />
+ <NavList className="gap-5 px-5 sm:px-6 h-10" />
</nav>
</header>
);
diff --git a/homepage/app/downloads/page.tsx b/homepage/app/downloads/page.tsx
@@ -1,6 +1,7 @@
import Link from "next/link";
import type { Metadata } from "next";
import { PageShell, PageHeading } from "../components/PageShell";
+import { Fact } from "../components/Fact";
import { loadSnapshot, formatBytes, SNAPSHOT_HREF } from "../lib/snapshot";
export const metadata: Metadata = {
@@ -9,19 +10,6 @@ export const metadata: Metadata = {
"Get the Archilyzer source as a dated snapshot tarball — MIT licensed, with a checksum.",
};
-function Fact({ label, children }: { label: string; children: React.ReactNode }) {
- return (
- <div className="flex flex-col gap-1 border-t border-[var(--border)] py-3 sm:flex-row sm:gap-6">
- <span className="label-machine sm:w-32 sm:shrink-0 sm:pt-0.5">
- {label}
- </span>
- <span className="min-w-0 break-all text-sm text-[var(--muted-foreground)]">
- {children}
- </span>
- </div>
- );
-}
-
export default function DownloadsPage() {
const snapshot = loadSnapshot();
const date = snapshot
@@ -50,6 +38,16 @@ export default function DownloadsPage() {
branches, nothing to <code className="font-mono">git pull</code>.
Updating means downloading a newer snapshot.
</p>
+ <p className="leading-[1.7] text-[var(--muted-foreground)]">
+ The history is in the{" "}
+ <Link
+ href="/source/"
+ className="underline decoration-[var(--border-strong)] underline-offset-2 hover:text-[var(--foreground)]"
+ >
+ read-only git mirror
+ </Link>
+ , published by the same build as this tarball.
+ </p>
</div>
{snapshot ? (
@@ -99,8 +97,9 @@ export default function DownloadsPage() {
</p>
<p className="text-sm text-[var(--faint)] leading-relaxed">
The tarball and its sidecar are build artefacts, generated by{" "}
- <code className="font-mono">./create-archives.sh</code> and not
- committed. A site built without running it ships no download, and
+ <code className="font-mono">archilyzer source publish</code> (which{" "}
+ <code className="font-mono">archilyzer build homepage</code> runs)
+ and not committed. A site built without it ships no download, and
says so here rather than linking to a file that isn’t there.
</p>
</div>
diff --git a/homepage/app/globals.css b/homepage/app/globals.css
@@ -105,3 +105,17 @@ body {
.growth-hit:hover {
fill: var(--chart-grid);
}
+
+/* The growth chart's surface gap (ArchiveGrowthChart.tsx): 2 px along a
+ band's upper edge, in the colour behind the plot — the page ground — so
+ touching bands read apart by a gap, never by a line drawn around them.
+ Forced colours: the reader's Canvas. */
+.growth-gap {
+ stroke: var(--background);
+ stroke-width: 2px;
+}
+@media (forced-colors: active) {
+ .growth-gap {
+ stroke: Canvas;
+ }
+}
diff --git a/homepage/app/layout.tsx b/homepage/app/layout.tsx
@@ -2,6 +2,7 @@ import type { Metadata, Viewport } from "next";
import { fontVars } from "yt-dlp-transcript-common/styles/fonts";
import { ThemeScript } from "yt-dlp-transcript-common/components/ThemeScript";
import { ThemeProvider } from "yt-dlp-transcript-common/components/ThemeProvider";
+import { HOMEPAGE_DEFAULT_BASE } from "yt-dlp-transcript-common/lib/themeConfig";
import { BASE_GROUNDS, DEFAULT_ACCENT } from "yt-dlp-transcript-common/lib/brand";
import {
PROJECT_NAME,
@@ -71,9 +72,10 @@ export default function RootLayout({
>
<body className="min-h-full flex flex-col bg-[var(--background)] text-[var(--foreground)] font-sans selection:bg-[var(--brand-soft)] selection:text-[var(--foreground)]">
{/* The project's own site opens on the dark base in Signal, the
- family's accent; a reader can pick any other. */}
- <ThemeScript defaultBase="dark" />
- <ThemeProvider defaultBase="dark" siteAccent={DEFAULT_ACCENT}>
+ family's accent. A reader cycles the base with the header's toggle;
+ the accent is always Signal, whatever this origin's storage holds. */}
+ <ThemeScript defaultBase={HOMEPAGE_DEFAULT_BASE} />
+ <ThemeProvider defaultBase={HOMEPAGE_DEFAULT_BASE} siteAccent={DEFAULT_ACCENT}>
<Header />
<main className="flex-1 w-full">{children}</main>
<Footer />
diff --git a/homepage/app/lib/docs.ts b/homepage/app/lib/docs.ts
@@ -13,8 +13,11 @@ export { DOC_GROUP_ORDER };
// Build-time only. `output: "export"` means every one of these reads happens
// during `next build`; nothing here runs per request. Mirrors the
// readFileSync-from-cwd approach in export/app/changelog/page.tsx.
+//
+// Every path op on a cwd-derived path opts out of Turbopack's asset tracing
+// (plans/FACTS.md, "A path joined from `process.cwd()` …").
function docPath(entry: DocEntry): string {
- return path.join(process.cwd(), "content", "docs", entry.file);
+ return path.join(/* turbopackIgnore: true */ process.cwd(), "content", "docs", entry.file);
}
// The manifest entries whose file actually exists.
@@ -27,7 +30,7 @@ function docPath(entry: DocEntry): string {
export function listDocs(): DocEntry[] {
return DOCS.filter((entry) => {
try {
- return fs.statSync(docPath(entry)).isFile();
+ return fs.statSync(/* turbopackIgnore: true */ docPath(entry)).isFile();
} catch {
return false;
}
@@ -42,7 +45,7 @@ export function findDoc(slug: string): DocEntry | null {
// heading so it stands alone as a document; the page renders its own <h1> from
// the manifest, and two titles in a row reads as a mistake.
export function readDoc(entry: DocEntry): string {
- const raw = fs.readFileSync(docPath(entry), "utf8");
+ const raw = fs.readFileSync(/* turbopackIgnore: true */ docPath(entry), "utf8");
return raw.replace(/^?\s*#[^\n#][^\n]*\n+/, "");
}
diff --git a/homepage/app/lib/growthGaps.test.ts b/homepage/app/lib/growthGaps.test.ts
@@ -0,0 +1,289 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { siteChartColors } from "yt-dlp-transcript-common/lib/siteColor";
+import { buildFixtureSummary } from "../../e2e/fixture-summary";
+import {
+ GAP_PX,
+ MIN_KEEP_PX,
+ OTHER_COLOR,
+ OTHER_KEY,
+ PLOT_SIZES,
+ bandAbove,
+ foldedSites,
+ gapSegments,
+ growthLayers,
+ growthStack,
+ layerColors,
+ ownLayers,
+ type Band,
+ type PlotSize,
+} from "./growthGaps";
+
+// Run with:
+// pnpm --filter homepage test
+//
+// THE PROMISE: wherever the growth chart draws a gap, both bands it parts keep
+// at least MIN_KEEP_PX of their own colour, measured at right angles to the
+// edge, at the narrowest plot of each height — so no band is ever fully
+// covered. Checked here independently of gapSegments' own test: each band's
+// extent under a gap segment is the distance from its FAR edge's two points to
+// the gapped edge's line, in px, less every gap that touches the band there.
+
+type Pt = [number, number];
+
+function px(size: PlotSize, yMax: number, n: number) {
+ const dx = size.minWidth / Math.max(1, n - 1);
+ return (i: number, v: number): Pt => [i * dx, (v / yMax) * size.px];
+}
+
+// Distance from p to the line through a and b.
+function dist(p: Pt, a: Pt, b: Pt): number {
+ const [x, y] = p;
+ const [x1, y1] = a;
+ const [x2, y2] = b;
+ return Math.abs((y2 - y1) * x - (x2 - x1) * y + x2 * y1 - y2 * x1) / Math.hypot(x2 - x1, y2 - y1);
+}
+
+// Every band-segment a gap leaves with less than MIN_KEEP_PX of its colour.
+function covered(stack: readonly Band[], yMax: number, size: PlotSize): string[] {
+ const n = stack[0].hi.length;
+ const at = px(size, yMax, n);
+ const segs = gapSegments(stack, yMax, size).map((s) => new Set(s));
+ // Band j loses GAP_PX / 2 on its lower edge at segment i when the band
+ // beneath it has a gap there whose band above is j.
+ const gapBelow = (j: number, i: number) =>
+ stack.some((_, k) => k < j && segs[k].has(i) && bandAbove(stack, k, i) === j);
+ const bad: string[] = [];
+ stack.forEach((band, k) => {
+ for (const i of segs[k]) {
+ const a = at(i, band.hi[i]);
+ const b = at(i + 1, band.hi[i + 1]);
+ // The band below the edge: its lower edge's points.
+ const below = Math.min(dist(at(i, band.lo[i]), a, b), dist(at(i + 1, band.lo[i + 1]), a, b));
+ const keepBelow = below - GAP_PX / 2 - (gapBelow(k, i) ? GAP_PX / 2 : 0);
+ // The band above: its upper edge's points.
+ const m = bandAbove(stack, k, i)!;
+ const above = Math.min(dist(at(i, stack[m].hi[i]), a, b), dist(at(i + 1, stack[m].hi[i + 1]), a, b));
+ const keepAbove = above - GAP_PX / 2 - (segs[m].has(i) ? GAP_PX / 2 : 0);
+ if (keepBelow < MIN_KEEP_PX - 1e-9) bad.push(`${size.px}px band ${k} @${i}: ${keepBelow.toFixed(2)}`);
+ if (keepAbove < MIN_KEEP_PX - 1e-9) bad.push(`${size.px}px band ${m} @${i}: ${keepAbove.toFixed(2)}`);
+ }
+ });
+ return bad;
+}
+
+// A deterministic stand-in for the published chart's shape: one large band
+// with one-month spikes and dips, one medium band, and four slivers of 0–12
+// that start late.
+function slivers(): { stack: Band[]; yMax: number } {
+ let seed = 7;
+ const rnd = () => ((seed = (seed * 1103515245 + 12345) % 2 ** 31) / 2 ** 31);
+ const n = 204;
+ const months = Array.from({ length: n }, (_, i) => {
+ const big = i < 60 ? rnd() * 20 : 80 + i * 3 + (i % 17 === 0 ? 900 : 0) - (i % 23 === 0 ? 70 : 0);
+ const mid = i < 120 ? rnd() * 10 : 60 + rnd() * 120 + (i % 29 === 0 ? 600 : 0);
+ const s = (from: number) => (i < from ? 0 : Math.round(rnd() * 12));
+ return { bySite: { a: Math.max(0, big), b: mid, c: s(90), d: s(130), e: s(150), f: s(190) } };
+ });
+ const sites = ["a", "b", "c", "d", "e", "f"].map((siteId) => ({ siteId }));
+ const { stack, yMax } = growthStack(months, sites);
+ return { stack, yMax };
+}
+
+test("the fixture summary, as the chart draws it (Other on top): gaps are drawn, and no band is ever covered, at every height", () => {
+ const summary = buildFixtureSummary();
+ const months = summary.monthly ?? [];
+ const layers = growthLayers(months, summary.sites);
+ // The fixture's last two sites are under 5 %: they fold.
+ assert.deepEqual(foldedSites(months, summary.sites), [4, 5]);
+ assert.equal(layers.at(-1)!.key, OTHER_KEY);
+ const { stack, yMax } = growthStack(months, summary.sites, layers);
+ let drawn = 0;
+ let underOther = 0;
+ for (const size of PLOT_SIZES) {
+ const segs = gapSegments(stack, yMax, size);
+ drawn += segs.flat().length;
+ // The gap under Other parts it from the top kept site; nothing sits on
+ // Other, so its own upper edge has none.
+ underOther += segs[stack.length - 2].filter((i) => bandAbove(stack, stack.length - 2, i) === stack.length - 1).length;
+ assert.deepEqual(segs[stack.length - 1], [], `${size.px}px: a gap along Other's top`);
+ assert.deepEqual(covered(stack, yMax, size), [], `${size.px}px`);
+ }
+ assert.ok(drawn > 0, "no gap drawn at all");
+ assert.ok(underOther > 0, "no gap between the top kept site and Other");
+ // Unfolded, the same summary keeps its promise too.
+ const own = growthStack(months, summary.sites);
+ for (const size of PLOT_SIZES) assert.deepEqual(covered(own.stack, own.yMax, size), [], `unfolded ${size.px}px`);
+});
+
+test("slivers beside large bands, with one-month spikes: no band is ever covered", () => {
+ const { stack, yMax } = slivers();
+ for (const size of PLOT_SIZES) {
+ const bad = covered(stack, yMax, size);
+ assert.equal(bad.length, 0, bad.slice(0, 5).join("; "));
+ }
+});
+
+test("a one-month spike: its steep flanks get no gap at a phone's width, however tall the bands vertically", () => {
+ const n = 40;
+ const months = Array.from({ length: n }, (_, i) => ({
+ bySite: { a: i === 20 ? 3000 : 1000, b: 400 },
+ }));
+ const sites = [{ siteId: "a" }, { siteId: "b" }];
+ const { stack, yMax } = growthStack(months, sites);
+ const phone = gapSegments(stack, yMax, PLOT_SIZES[0])[0];
+ // b is 20 px tall vertically on the flanks, but about 1 px at right angles.
+ assert.ok(!phone.includes(19) && !phone.includes(20), `flank gaps: ${phone.join(",")}`);
+ // The flat stretches either side are parted.
+ assert.ok(phone.includes(5) && phone.includes(30), `flat gaps: ${phone.join(",")}`);
+ assert.deepEqual(covered(stack, yMax, PLOT_SIZES[0]), []);
+});
+
+test("no gap along the stack's top, and none under the run length", () => {
+ const n = 10;
+ const months = Array.from({ length: n }, (_, i) => ({ bySite: { a: 500, b: i === 4 ? 500 : 0 } }));
+ const { stack, yMax } = growthStack(months, [{ siteId: "a" }, { siteId: "b" }]);
+ for (const size of PLOT_SIZES) {
+ // b sits on a for one month only: a run of one (two segments) is dropped.
+ assert.deepEqual(gapSegments(stack, yMax, size), [[], []], `${size.px}px`);
+ }
+});
+
+// ── The fold ──────────────────────────────────────────────────────────────────
+//
+// A site under 5 % of the placed total (every site's transcripts over the whole
+// plotted range) folds into ONE Other band on top, when two or more do. The
+// range's sums are what count: `monthsOf` spreads each site's sum over three
+// months, so no single month decides.
+
+type Named = { siteId: string; siteTitle: string; accentId?: string };
+
+function sitesOf(ids: readonly string[], accents: readonly (string | undefined)[] = []): Named[] {
+ return ids.map((siteId, i) => ({ siteId, siteTitle: siteId.toUpperCase(), accentId: accents[i] }));
+}
+
+function monthsOf(sites: readonly Named[], sums: readonly number[]) {
+ // Thirds, the remainder in the last month.
+ return [0, 1, 2].map((k) => ({
+ bySite: Object.fromEntries(
+ sites.map((s, i) => [s.siteId, k < 2 ? Math.floor(sums[i] / 3) : sums[i] - 2 * Math.floor(sums[i] / 3)]),
+ ),
+ }));
+}
+
+test("the fold with today's proportions: the three largest keep their bands, the other three are one Other band on top", () => {
+ // The family's shares in the 2026-09-28 summary, in its order and with its
+ // accents: 42.03, 38.40, 9.18, 4.33, 3.79, 2.26 %.
+ const sites = sitesOf(["a", "b", "c", "d", "e", "f"], ["brass", "sakura", "blue", "violet", "green", "vermilion"]);
+ const sums = [4203, 3840, 918, 433, 379, 226];
+ const months = monthsOf(sites, sums);
+ assert.deepEqual(foldedSites(months, sites), [3, 4, 5]);
+ const layers = growthLayers(months, sites);
+ assert.deepEqual(
+ layers.map((l) => [l.key, l.sites, l.other]),
+ [
+ ["a", [0], false],
+ ["b", [1], false],
+ ["c", [2], false],
+ [OTHER_KEY, [3, 4, 5], true],
+ ],
+ );
+ // Other is the sum of its sites, on top: the stack's top is every month's
+ // total, as before the fold.
+ const { stack, totals } = growthStack(months, sites, layers);
+ assert.deepEqual(stack[3].hi.map((v, i) => v - stack[3].lo[i]), months.map((m) => m.bySite.d + m.bySite.e + m.bySite.f));
+ assert.deepEqual(stack[3].hi, totals);
+ // Colours: the kept sites their families' slots, Other the neutral grey.
+ assert.deepEqual(layerColors(layers, sites), ["var(--chart-4)", "var(--chart-5)", "var(--chart-1)", OTHER_COLOR]);
+ assert.equal(OTHER_COLOR, "var(--chart-other)");
+});
+
+test("the threshold's edge: exactly 5 % keeps its band, just under folds", () => {
+ const sites = sitesOf(["a", "b", "c", "d"]);
+ // b and c are 999 of 20,000 (4.995 %), d exactly 1,000 (5 %).
+ assert.deepEqual(foldedSites(monthsOf(sites, [17_002, 999, 999, 1_000]), sites), [1, 2]);
+ // Two sites at exactly 5 %: neither folds.
+ const three = sitesOf(["a", "b", "c"]);
+ assert.deepEqual(foldedSites(monthsOf(three, [18_000, 1_000, 1_000]), three), []);
+ assert.deepEqual(growthLayers(monthsOf(three, [18_000, 1_000, 1_000]), three), ownLayers(three));
+});
+
+test("a fold of one is no fold: a single site under 5 % keeps its band", () => {
+ const sites = sitesOf(["a", "b", "c"]);
+ const months = monthsOf(sites, [60, 36, 4]);
+ assert.deepEqual(foldedSites(months, sites), []);
+ assert.deepEqual(growthLayers(months, sites), ownLayers(sites));
+ assert.ok(growthLayers(months, sites).every((l) => !l.other));
+});
+
+test("no site under 5 %: every site keeps its band, as before the fold", () => {
+ const sites = sitesOf(["a", "b", "c"]);
+ const months = monthsOf(sites, [50, 30, 20]);
+ assert.deepEqual(foldedSites(months, sites), []);
+ const layers = growthLayers(months, sites);
+ assert.deepEqual(layers, ownLayers(sites));
+ // The same stack as growthStack's default.
+ assert.deepEqual(growthStack(months, sites, layers), growthStack(months, sites));
+});
+
+test("all but one under 5 %: one kept band and one Other band", () => {
+ const sites = sitesOf(["a", "b", "c", "d", "e", "f"]);
+ const months = monthsOf(sites, [80, 4, 4, 4, 4, 4]);
+ const layers = growthLayers(months, sites);
+ assert.deepEqual(
+ layers.map((l) => [l.key, l.sites]),
+ [
+ ["a", [0]],
+ [OTHER_KEY, [1, 2, 3, 4, 5]],
+ ],
+ );
+ const { stack, totals } = growthStack(months, sites, layers);
+ assert.equal(stack.length, 2);
+ assert.deepEqual(stack[1].hi, totals);
+});
+
+test("a site with nothing in the plotted range is under 5 %, and folds with another", () => {
+ const sites = sitesOf(["a", "b", "c"]);
+ assert.deepEqual(foldedSites(monthsOf(sites, [96, 4, 0]), sites), [1, 2]);
+ // Nothing plotted at all: nothing folds.
+ assert.deepEqual(foldedSites(monthsOf(sites, [0, 0, 0]), sites), []);
+});
+
+test("every site under 5 % (over twenty sites): every site folds, one Other band", () => {
+ const ids = Array.from({ length: 25 }, (_, i) => `s${i}`);
+ const sites = sitesOf(ids);
+ const layers = growthLayers(monthsOf(sites, ids.map(() => 40)), sites);
+ assert.deepEqual(
+ layers.map((l) => [l.key, l.sites.length]),
+ [[OTHER_KEY, 25]],
+ );
+});
+
+test("a kept site's colour is the one it wears unfolded, whoever folds", () => {
+ // No accents: each site's slot is its index's (siteChartColors), so a list
+ // of the kept sites alone would repaint d — the chart runs it over them all.
+ const sites = sitesOf(["a", "b", "c", "d"]);
+ const months = monthsOf(sites, [500, 20, 20, 460]);
+ const layers = growthLayers(months, sites);
+ assert.deepEqual(
+ layers.map((l) => l.key),
+ ["a", "d", OTHER_KEY],
+ );
+ const unfolded = layerColors(ownLayers(sites), sites);
+ assert.deepEqual(unfolded, siteChartColors(sites));
+ const colours = layerColors(layers, sites);
+ assert.deepEqual(colours, [unfolded[0], unfolded[3], OTHER_COLOR]);
+ assert.notEqual(colours[1], siteChartColors([sites[0], sites[3]])[1], "the kept list alone would repaint d");
+ // With accents, the same: each kept site keeps its family's slot.
+ const named = sitesOf(["a", "b", "c", "d", "e", "f"], ["brass", "sakura", "blue", "violet", "green", "vermilion"]);
+ const all = siteChartColors(named);
+ const nm = monthsOf(named, [4203, 3840, 918, 433, 379, 226]);
+ const nl = growthLayers(nm, named);
+ assert.deepEqual(
+ layerColors(nl, named).slice(0, -1),
+ nl.slice(0, -1).map((l) => all[l.sites[0]]),
+ );
+ // The fold never hands a kept site the grey, nor Other a site's slot.
+ assert.ok(!layerColors(nl, named).slice(0, -1).includes(OTHER_COLOR));
+ assert.ok(!all.includes(OTHER_COLOR));
+});
diff --git a/homepage/app/lib/growthGaps.ts b/homepage/app/lib/growthGaps.ts
@@ -0,0 +1,237 @@
+import { siteChartColors } from "yt-dlp-transcript-common/lib/siteColor";
+
+// THE GROWTH CHART'S LAYERS AND SURFACE GAPS, worked out (ArchiveGrowthChart.tsx
+// draws them). Pure: no React, no DOM, so a unit test can check every layer and
+// every segment.
+//
+// THE FOLD. A site under FOLD_PERCENT % of the placed total — every site's
+// transcripts over the whole plotted range, the sum the bands are placed from
+// — is drawn in ONE "Other" band on top of the stack, in the chart's neutral
+// grey (OTHER_COLOR). Exactly FOLD_PERCENT % keeps its band; the comparison is
+// on integers (100 × a site's sum < FOLD_PERCENT × the total), so the edge is
+// exact. A fold of one is no fold: with a single site under the line, every
+// site keeps its band. The kept sites keep their colours: each is its
+// siteChartColors slot over EVERY site, folded or not, so a site's colour never
+// changes because another one folded. The legend shows the kept sites and
+// Other; the hover title and the "Numbers by year" table still name every site.
+//
+// THE GAPS. The marks spec parts touching bands by a 2 px gap in the colour
+// behind the plot. The chart draws it centred on each band's upper edge, so
+// each of the two bands gives 1 px. A band too thin to give its pixel and keep
+// one of its own colour must touch its neighbour instead, or the gap erases
+// it. "Thin" is measured AT RIGHT ANGLES to the edge, where the stroke's 2 px
+// are measured: on a steep edge a band's perpendicular thickness is its
+// vertical height × cos θ, and on a one-month dip at a phone's width it is
+// nearly nothing. And it is measured at the NARROWEST plot each of the chart's
+// three heights is drawn at, where the edges are steepest. A gap is drawn along
+// a segment only where both bands keep at least MIN_KEEP_PX there after every
+// gap that touches them. The gaps work on bands, not sites: Other is one band,
+// the top one, so the gap under it parts it from the kept site beneath (the
+// highest with a height that month), and its own upper edge, where nothing
+// sits, has none.
+
+export const GAP_PX = 2;
+export const MIN_KEEP_PX = 1;
+// A band that gives half a gap on each of its edges gives GAP_PX in all.
+const MIN_BAND_PX = GAP_PX + MIN_KEEP_PX;
+// A run of gap shorter than this many months is dropped: a speck of gap reads
+// as noise, not as a parting.
+export const MIN_GAP_RUN = 3;
+
+// The plot's three heights (the box is `h-[200px] sm:h-[260px] lg:h-[300px]`),
+// the narrowest plot width each is drawn at (the narrowest viewport of its
+// breakpoint less the container's padding: 280 − 2 × 20, 640 − 2 × 24,
+// 1024 − 2 × 24), and the class that shows its set of gaps.
+export const PLOT_SIZES = [
+ { px: 200, minWidth: 240, className: "sm:hidden" },
+ { px: 260, minWidth: 592, className: "hidden sm:inline lg:hidden" },
+ { px: 300, minWidth: 976, className: "hidden lg:inline" },
+] as const;
+export type PlotSize = (typeof PLOT_SIZES)[number];
+
+// The stack, bottom-up: each band's lower and upper value per month.
+export type Band = { lo: readonly number[]; hi: readonly number[] };
+
+type Month = { bySite: Record<string, number | undefined> };
+
+// The layers stack in the summary's order (the first five as they come, any
+// more after them).
+const STACK_ORDER = [0, 1, 2, 3, 4];
+function stackOrder(n: number): number[] {
+ const head = STACK_ORDER.filter((i) => i < n);
+ const tail = Array.from({ length: Math.max(0, n - 5) }, (_, k) => k + 5);
+ return [...head, ...tail];
+}
+
+// ── The fold ──────────────────────────────────────────────────────────────────
+
+export const FOLD_PERCENT = 5;
+export const OTHER_LABEL = "Other";
+// Other's own chart token (tokens.css `--chart-other`), a near-neutral grey:
+// 7.36:1 on the Light ground and 3.22:1 on the Dark one (7.99 / 3.06 on the
+// chart surface). tokens.css has its separation from each chart slot.
+export const OTHER_COLOR = "var(--chart-other)";
+
+// One band of the stack: a site's own (`sites` is its index in the summary's
+// list), or Other (the indexes of every folded site, in the list's order).
+// `key` is the site's id, or OTHER_KEY, which no site id can be (an id is
+// `[a-z0-9][a-z0-9-]*`).
+export type GrowthLayer = { key: string; sites: readonly number[]; other: boolean };
+export const OTHER_KEY = "(other)";
+
+// Each site's transcripts over the whole plotted range.
+function siteSums(months: readonly Month[], sites: readonly { siteId: string }[]): number[] {
+ return sites.map((s) => months.reduce((a, m) => a + (m.bySite[s.siteId] ?? 0), 0));
+}
+
+// The sites folded into Other, as indexes into `sites` in its order: every
+// site under FOLD_PERCENT % of the placed total, when there are two or more of
+// them; none otherwise.
+export function foldedSites(months: readonly Month[], sites: readonly { siteId: string }[]): number[] {
+ const sums = siteSums(months, sites);
+ const all = sums.reduce((a, b) => a + b, 0);
+ if (all <= 0) return [];
+ const small = sums.flatMap((v, i) => (100 * v < FOLD_PERCENT * all ? [i] : []));
+ return small.length >= 2 ? small : [];
+}
+
+// Every site its own band, in stack order: the chart before the fold.
+export function ownLayers(sites: readonly { siteId: string }[]): GrowthLayer[] {
+ return stackOrder(sites.length).map((i) => ({ key: sites[i].siteId, sites: [i], other: false }));
+}
+
+// The chart's bands, bottom-up: the kept sites in stack order, then Other on
+// top when anything folds.
+export function growthLayers(
+ months: readonly Month[],
+ sites: readonly { siteId: string }[],
+): GrowthLayer[] {
+ const folded = foldedSites(months, sites);
+ if (folded.length === 0) return ownLayers(sites);
+ const out = new Set(folded);
+ return [
+ ...ownLayers(sites).filter((l) => !out.has(l.sites[0])),
+ { key: OTHER_KEY, sites: folded, other: true },
+ ];
+}
+
+// Each layer's colour: a kept site its siteChartColors slot over EVERY site
+// (so folding never repaints it), Other the neutral grey.
+export function layerColors(
+ layers: readonly GrowthLayer[],
+ sites: readonly { accentId?: string }[],
+): string[] {
+ const chart = siteChartColors(sites);
+ return layers.map((l) => (l.other ? OTHER_COLOR : chart[l.sites[0]]));
+}
+
+// A clean tick step giving three or four gridlines under `max`.
+function niceStep(max: number): number {
+ const raw = max / 3.5;
+ const pow = 10 ** Math.floor(Math.log10(raw));
+ for (const m of [1, 2, 2.5, 5, 10]) {
+ if (m * pow >= raw) return m * pow;
+ }
+ return 10 * pow;
+}
+
+// ── The stack ─────────────────────────────────────────────────────────────────
+
+// The chart's numbers: each month's total (every site, folded or not), the
+// peak, the value scale (yMax and its gridlines), and one band per layer,
+// bottom-up (`stack[k]` is `layers[k]`'s; a layer's value in a month is the sum
+// of its sites'). Without `layers`, every site is its own band.
+export function growthStack(
+ months: readonly Month[],
+ sites: readonly { siteId: string }[],
+ layers: readonly GrowthLayer[] = ownLayers(sites),
+) {
+ const n = months.length;
+ const totals = months.map((m) => sites.reduce((a, s) => a + (m.bySite[s.siteId] ?? 0), 0));
+ const peak = totals.length ? Math.max(...totals) : 0;
+ const step = peak > 0 ? niceStep(peak) : 1;
+ const yMax = Math.ceil(peak / step) * step;
+ const ticks: number[] = [];
+ if (peak > 0) for (let t = step; t <= yMax; t += step) ticks.push(t);
+ const base = new Array<number>(n).fill(0);
+ const stack: Band[] = layers.map((layer) => {
+ const lo = base.slice();
+ const hi = months.map(
+ (m, i) => (base[i] += layer.sites.reduce((a, si) => a + (m.bySite[sites[si].siteId] ?? 0), 0)),
+ );
+ return { lo, hi };
+ });
+ return { totals, peak, yMax, ticks, layers, stack };
+}
+
+// ── The gaps ──────────────────────────────────────────────────────────────────
+
+const height = (b: Band, i: number) => b.hi[i] - b.lo[i];
+
+// The band a gap along band k's upper edge would share at month i: the next
+// band up with a height there (a band of height 0 lies on the edge), or none —
+// the stack's top meets the surface itself.
+export function bandAbove(stack: readonly Band[], k: number, i: number): number | null {
+ for (let m = k + 1; m < stack.length; m++) if (height(stack[m], i) > 0) return m;
+ return null;
+}
+
+// A band's thickness at right angles to band k's upper edge over the segment
+// i → i + 1, in px at `size`: the smaller of its two vertical heights there
+// × cos θ of the edge.
+function perpendicular(
+ stack: readonly Band[],
+ k: number,
+ band: number,
+ i: number,
+ yMax: number,
+ size: PlotSize,
+ n: number,
+): number {
+ const sy = size.px / yMax;
+ const dx = size.minWidth / Math.max(1, n - 1);
+ const dy = (stack[k].hi[i + 1] - stack[k].hi[i]) * sy;
+ const cos = dx / Math.hypot(dx, dy);
+ return Math.min(height(stack[band], i), height(stack[band], i + 1)) * sy * cos;
+}
+
+// For each band, the month segments (their start index i, for i → i + 1)
+// along its upper edge where a gap is drawn at `size`.
+export function gapSegments(stack: readonly Band[], yMax: number, size: PlotSize): number[][] {
+ const n = stack[0]?.hi.length ?? 0;
+ return stack.map((band, k) => {
+ const parted = (i: number) => {
+ if (height(band, i) <= 0 || height(band, i + 1) <= 0) return false;
+ const up = bandAbove(stack, k, i);
+ if (up === null || up !== bandAbove(stack, k, i + 1)) return false;
+ return (
+ perpendicular(stack, k, k, i, yMax, size, n) >= MIN_BAND_PX &&
+ perpendicular(stack, k, up, i, yMax, size, n) >= MIN_BAND_PX
+ );
+ };
+ const out: number[] = [];
+ let run: number[] = [];
+ const close = () => {
+ if (run.length >= MIN_GAP_RUN) out.push(...run);
+ run = [];
+ };
+ for (let i = 0; i < n - 1; i++) {
+ if (parted(i)) run.push(i);
+ else close();
+ }
+ close();
+ return out;
+ });
+}
+
+// The segments grouped into runs of consecutive months, for drawing each run
+// as one polyline.
+export function runsOf(segments: readonly number[]): number[][] {
+ const runs: number[][] = [];
+ for (const i of segments) {
+ const last = runs.at(-1);
+ if (last && last.at(-1) === i - 1) last.push(i);
+ else runs.push([i]);
+ }
+ return runs;
+}
diff --git a/homepage/app/lib/headers.test.ts b/homepage/app/lib/headers.test.ts
@@ -0,0 +1,119 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { readFileSync } from "node:fs";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+// Run with:
+// pnpm --filter homepage test
+//
+// public/_headers is what keeps the raw source tree from being served as
+// code on this origin, and nothing but the Pages edge applies it. This pins
+// its security property with a small replica of wrangler's `_headers`
+// handling (pages-shared parseHeaders + generateRulesMatcher + attachHeaders,
+// read in wrangler 4.88 and re-checked by the review in 4.142): every
+// matching rule applies in file order, `! Name` detaches a header, and a
+// header a LATER rule sets again is APPENDED ("a, b"), not replaced.
+
+const FILE = path.join(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "public", "_headers");
+const TEXT = readFileSync(FILE, "utf8");
+
+type Rule = { path: string; set: Record<string, string>; unset: string[]; re: RegExp; lines: string[] };
+
+function parse(text: string): Rule[] {
+ const rules: Rule[] = [];
+ let rule: Rule | undefined;
+ for (const raw of text.split("\n")) {
+ const line = raw.trim();
+ if (!line || line.startsWith("#")) continue;
+ if (/^([^\s]+:\/\/|^\/)/.test(line)) {
+ if (rule) rules.push(rule);
+ const esc = (s: string) => s.replace(/[|\\{}()[\]^$+*?.]/g, "\\$&");
+ rule = { path: line, set: {}, unset: [], re: new RegExp(`^${line.split("*").map(esc).join("(?<splat>.*)")}$`), lines: [] };
+ continue;
+ }
+ rule!.lines.push(line);
+ if (line.startsWith("! ")) {
+ rule!.unset.push(line.slice(2).toLowerCase());
+ continue;
+ }
+ const i = line.indexOf(":");
+ const name = line.slice(0, i).trim().toLowerCase();
+ const value = line.slice(i + 1).trim();
+ rule!.set[name] = rule!.set[name] ? `${rule!.set[name]}, ${value}` : value;
+ }
+ if (rule) rules.push(rule);
+ return rules;
+}
+
+function served(rules: Rule[], pathname: string, contentType: string): Headers {
+ const h = new Headers({ "content-type": contentType });
+ const setOnce = new Set<string>();
+ for (const r of rules.filter((x) => x.re.test(pathname))) {
+ for (const k of r.unset) h.delete(k);
+ for (const [k, v] of Object.entries(r.set)) {
+ if (setOnce.has(k)) h.append(k, v);
+ else {
+ h.set(k, v);
+ setOnce.add(k);
+ }
+ }
+ }
+ return h;
+}
+
+const RULES = parse(TEXT);
+
+test("wrangler's limits: at most 100 rules, no line over 2,000 characters, one splat a rule", () => {
+ assert.ok(RULES.length <= 100, `${RULES.length} rules`);
+ for (const line of TEXT.split("\n")) assert.ok(line.length <= 2000, line.slice(0, 60));
+ for (const r of RULES) assert.ok((r.path.match(/\*/g) ?? []).length <= 1, r.path);
+});
+
+test("every /source/tree override detaches Content-Type before it sets its own", () => {
+ const overrides = RULES.filter((r) => r.path.startsWith("/source/tree/") && r.path !== "/source/tree/*" && "content-type" in r.set);
+ assert.ok(overrides.length >= 7, overrides.map((r) => r.path).join(" "));
+ for (const r of overrides) {
+ const unset = r.lines.findIndex((l) => l.toLowerCase() === "! content-type");
+ const set = r.lines.findIndex((l) => /^content-type:/i.test(l));
+ assert.ok(unset !== -1 && unset < set, `${r.path} must say "! Content-Type" before it sets one`);
+ }
+});
+
+test("the raw tree is text, its pages HTML, its binaries their own type — each ONE value", () => {
+ const cases: Array<[string, string, string]> = [
+ ["/source/tree/common/lib/paths.ts", "video/mp2t", "text/plain; charset=utf-8"],
+ ["/source/tree/README.md", "text/markdown", "text/plain; charset=utf-8"],
+ ["/source/tree/", "text/html", "text/html; charset=utf-8"],
+ ["/source/tree/common/", "text/html", "text/html; charset=utf-8"],
+ ["/source/tree/umtool/report-to-video/fonts/Archivo%5Bwdth%2Cwght%5D.ttf", "font/ttf", "font/ttf"],
+ ["/source/tree/homepage/app/docs/%5Bslug%5D/page.tsx", "application/octet-stream", "text/plain; charset=utf-8"],
+ ["/source/tree/umtool/models/yunet.onnx", "application/octet-stream", "application/octet-stream"],
+ ["/source/tree/homepage/public/icon.svg", "image/svg+xml", "image/svg+xml"],
+ // A missing directory: Pages answers with a 404 page, on this rule.
+ ["/source/tree/no/such/dir/", "text/html", "text/html; charset=utf-8"],
+ ];
+ for (const [p, served0, want] of cases) {
+ const h = served(RULES, p, served0);
+ assert.equal(h.get("content-type"), want, p);
+ assert.equal(h.get("x-content-type-options"), "nosniff", p);
+ }
+ assert.match(served(RULES, "/source/tree/a.svg", "image/svg+xml").get("content-security-policy")!, /default-src 'none'/);
+ assert.equal(served(RULES, "/source/manifest.json", "application/json").get("content-type"), "application/json");
+});
+
+test("the history pages (release 15 slice SG) are not indexed, like the raw tree, and keep their own types", () => {
+ const cases: Array<[string, string]> = [
+ ["/source/git/log.html", "text/html"],
+ ["/source/git/commit/0123456789abcdef0123456789abcdef01234567.html", "text/html"],
+ ["/source/git/atom.xml", "application/xml"],
+ ["/source/git/style.css", "text/css"],
+ ];
+ for (const [p, type] of cases) {
+ const h = served(RULES, p, type);
+ assert.equal(h.get("x-robots-tag"), "noindex", p);
+ assert.equal(h.get("content-type"), type, `${p} keeps its type: no /source/tree rule applies`);
+ }
+ assert.equal(served(RULES, "/source/tree/README.md", "text/markdown").get("x-robots-tag"), "noindex");
+ assert.equal(served(RULES, "/source/", "text/html").get("x-robots-tag"), null, "the /source/ page itself is indexed");
+});
diff --git a/homepage/app/lib/nav.ts b/homepage/app/lib/nav.ts
@@ -1,14 +1,24 @@
-// The site's navigation, declared once and rendered by both the header and the
-// footer. Order is editorial: what the software is, how to get it, what this
-// deployment has done, what changed.
+// The site's navigation, declared once. Order is editorial: what the software
+// is, where its code lives, how to get it, what this deployment has done, what
+// changed.
+//
+// HEADER_NAV the header's links;
+// FOOTER_NAV the footer's "Sections" and the 404 page: the header's links,
+// then Changelog, which is in the footer only.
//
// Every entry must resolve on a `build:nodata` tree — /stats/ renders an honest
-// no-data panel rather than 404ing — so the nav never has to be conditional.
+// no-data panel and /source/ an honest "nothing published" rather than 404ing —
+// so the nav never has to be conditional.
export type NavItem = { href: string; label: string };
-export const NAV: NavItem[] = [
+export const HEADER_NAV: readonly NavItem[] = [
{ href: "/docs/", label: "Docs" },
+ { href: "/source/", label: "Source" },
{ href: "/downloads/", label: "Downloads" },
{ href: "/stats/", label: "Stats" },
+];
+
+export const FOOTER_NAV: readonly NavItem[] = [
+ ...HEADER_NAV,
{ href: "/changelog/", label: "Changelog" },
];
diff --git a/homepage/app/lib/snapshot.ts b/homepage/app/lib/snapshot.ts
@@ -1,8 +1,11 @@
import fs from "node:fs";
import path from "node:path";
-// Facts about the published source snapshot, written by create-archives.sh
-// beside the tarball it describes.
+// Facts about the published source snapshot, written by `archilyzer source
+// publish` (common/publish/source.ts) beside the tarball it describes. Since
+// release 12 the tarball is `git archive` of the scrubbed MIRROR's main, so
+// `commit` is a mirror id — the /source/ page names the private main it
+// reflects.
export type Snapshot = {
generatedAt: string;
commit: string;
@@ -20,30 +23,30 @@ export const SNAPSHOT_HREF = "/downloads/archilyzer-source.tar.gz";
//
// The null case is not hypothetical: the tarball and its sidecar are gitignored
// build artefacts, so a fresh unpack of the source has neither until
-// ./create-archives.sh runs. The download page MUST render no link at all in
+// `archilyzer source publish` runs (`archilyzer build homepage` runs it). The download page MUST render no link at all in
// that case — a link to a file that isn't there is worse than an explanation of
// why there isn't one.
export function loadSnapshot(): Snapshot | null {
try {
const file = path.join(
- process.cwd(),
+ /* turbopackIgnore: true */ process.cwd(),
"public",
"downloads",
"snapshot.json",
);
- const parsed = JSON.parse(fs.readFileSync(file, "utf8")) as Snapshot;
+ const parsed = JSON.parse(fs.readFileSync(/* turbopackIgnore: true */ file, "utf8")) as Snapshot;
if (!parsed || typeof parsed.bytes !== "number" || !parsed.sha256) {
return null;
}
// Believe the sidecar only if the file it describes is actually present —
// they are written together but deployed as separate assets.
const tarball = path.join(
- process.cwd(),
+ /* turbopackIgnore: true */ process.cwd(),
"public",
"downloads",
path.basename(SNAPSHOT_HREF),
);
- if (!fs.statSync(tarball).isFile()) return null;
+ if (!fs.statSync(/* turbopackIgnore: true */ tarball).isFile()) return null;
return parsed;
} catch {
return null;
diff --git a/homepage/app/lib/source.test.ts b/homepage/app/lib/source.test.ts
@@ -0,0 +1,108 @@
+import { test, after } from "node:test";
+import assert from "node:assert/strict";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { loadSourceManifest, sourcePublicDir } from "./source";
+
+// Run with:
+// pnpm --filter homepage test
+//
+// The /source/ page's loader: the manifest, believed only when it parses AND
+// the mirror's info/refs and the tarball are beside it. Anything else is the
+// empty state — a bad file must never crash `next build` (review L10).
+
+const TMP = mkdtempSync(path.join(os.tmpdir(), "homepage-source-"));
+after(() => rmSync(TMP, { recursive: true, force: true }));
+
+const manifest = {
+ version: 1,
+ generatedAt: "2026-09-28T12:00:00.000Z",
+ branch: "main",
+ sourceCommit: "1".repeat(40),
+ mirrorHead: "2".repeat(40),
+ subject: "s",
+ files: 10,
+ bytes: 100,
+ mirror: { files: 9, bytes: 80, packs: 2 },
+ tree: { files: 5, dirs: 2, bytes: 20 },
+ tarball: { href: "/downloads/archilyzer-source.tar.gz", bytes: 7, sha256: "3".repeat(64) },
+ cloneUrl: "https://archilyzer.pages.dev/source/archilyzer.git",
+ treeHref: "/source/tree/",
+ audit: { objects: 3, commits: 1, gitleaks: "clean" },
+ tools: { git: "2.55.0", filterRepo: "x" },
+};
+
+let n = 0;
+function pub(opts: { manifest?: unknown; refs?: boolean; tarball?: boolean; log?: boolean }): string {
+ const p = path.join(TMP, `p${n++}`);
+ mkdirSync(path.join(p, "source", "archilyzer.git", "info"), { recursive: true });
+ mkdirSync(path.join(p, "downloads"), { recursive: true });
+ if (opts.manifest !== undefined) {
+ writeFileSync(
+ path.join(p, "source", "manifest.json"),
+ typeof opts.manifest === "string" ? opts.manifest : JSON.stringify(opts.manifest),
+ );
+ }
+ if (opts.refs !== false) writeFileSync(path.join(p, "source", "archilyzer.git", "info", "refs"), "x\trefs/heads/main\n");
+ if (opts.tarball !== false) writeFileSync(path.join(p, "downloads", "archilyzer-source.tar.gz"), "t");
+ if (opts.log) {
+ mkdirSync(path.join(p, "source", "git"), { recursive: true });
+ writeFileSync(path.join(p, "source", "git", "log.html"), "<html></html>");
+ }
+ return p;
+}
+
+test("a good manifest with its files beside it is believed", () => {
+ assert.equal(loadSourceManifest(pub({ manifest }))?.mirrorHead, "2".repeat(40));
+});
+
+test("malformed, wrong-version or orphaned manifests are the empty state, never a throw", () => {
+ const cases: Array<[string, string]> = [
+ ["no manifest", pub({})],
+ ["not JSON", pub({ manifest: "{ half" })],
+ ["version 2", pub({ manifest: { ...manifest, version: 2 } })],
+ ["tree.files missing (the page reads it)", pub({ manifest: { ...manifest, tree: { dirs: 1, bytes: 1 } } })],
+ ["mirror.packs a string", pub({ manifest: { ...manifest, mirror: { ...manifest.mirror, packs: "2" } } })],
+ ["no info/refs", pub({ manifest, refs: false })],
+ ["no tarball", pub({ manifest, tarball: false })],
+ ["no public dir at all", path.join(TMP, "missing")],
+ ];
+ for (const [what, dir] of cases) {
+ assert.doesNotThrow(() => loadSourceManifest(dir), what);
+ assert.equal(loadSourceManifest(dir), null, what);
+ }
+});
+
+// Release 15 slice SG: the history block is believed only beside its log page.
+const history = {
+ href: "/source/git/log.html",
+ commits: 3,
+ total: 3,
+ head: "2".repeat(40),
+ files: 9,
+ bytes: 1234,
+ sha256: "4".repeat(64),
+ tool: "stagit (sha256 0123456789ab)",
+};
+
+test("a history block beside its log page is believed; without the page the manifest comes back without it", () => {
+ assert.deepEqual(loadSourceManifest(pub({ manifest: { ...manifest, history }, log: true }))?.history, history);
+ const orphan = loadSourceManifest(pub({ manifest: { ...manifest, history } }));
+ assert.equal(orphan?.mirrorHead, "2".repeat(40), "the rest of the manifest stands");
+ assert.ok(orphan && !("history" in orphan), "no History links for pages that are not here");
+ assert.equal(loadSourceManifest(pub({ manifest, log: true }))?.history, undefined, "no block, no History");
+ // A malformed block is a manifest nobody believes (parseSourceManifest).
+ assert.equal(loadSourceManifest(pub({ manifest: { ...manifest, history: { ...history, commits: "3" } }, log: true })), null);
+});
+
+test("the e2e's fixture publish is read only outside a production build, and only when it holds a manifest", () => {
+ const fixture = pub({ manifest, log: true });
+ const empty = path.join(TMP, "empty-fixture");
+ mkdirSync(empty, { recursive: true });
+ const PUBLIC = "/checkout/homepage/public";
+ assert.equal(sourcePublicDir({ NODE_ENV: "development", E2E_SOURCE_PUBLIC_DIR: fixture }, PUBLIC), fixture);
+ assert.equal(sourcePublicDir({ NODE_ENV: "production", E2E_SOURCE_PUBLIC_DIR: fixture }, PUBLIC), PUBLIC);
+ assert.equal(sourcePublicDir({ NODE_ENV: "development", E2E_SOURCE_PUBLIC_DIR: empty }, PUBLIC), PUBLIC, "no manifest: public/");
+ assert.equal(sourcePublicDir({ NODE_ENV: "development" }, PUBLIC), PUBLIC);
+});
diff --git a/homepage/app/lib/source.ts b/homepage/app/lib/source.ts
@@ -0,0 +1,76 @@
+import fs from "node:fs";
+import path from "node:path";
+import {
+ HISTORY_DIR,
+ MIRROR_DIR,
+ TARBALL_HREF,
+ parseSourceManifest,
+ type SourceManifest,
+} from "yt-dlp-transcript-common/lib/sourceManifest";
+
+// The published source's manifest, written LAST by `archilyzer source publish`
+// (common/publish/source.ts) into public/source/, or null when this build has
+// none. The /source/ page renders an honest empty state for null — the mirror,
+// the tree and the tarball are gitignored build artefacts, so a fresh clone
+// (or `archilyzer build homepage --no-source`) has none of them.
+//
+// Believed only when the files it describes are here too (the snapshot.ts
+// rule): a manifest beside a missing mirror or tarball would advertise a
+// clone that fails and a download that 404s. The same for the history block
+// (release 15 slice SG): without its log page beside it, the manifest is
+// returned WITHOUT it, and the page shows no History links. A malformed or
+// wrong-version manifest is null too (parseSourceManifest checks every number
+// the page reads), so a bad file is the empty state, never a crash in `next
+// build`. `pub` is the test's seam.
+//
+// The default is a DIRECTORY join on `process.cwd()`, which Turbopack would
+// trace as every file under `public/` (the source mirror among them, and it
+// grows with every publish): the opt-out keeps it a plain run-time path
+// (plans/FACTS.md, "A path joined from `process.cwd()` …").
+export function loadSourceManifest(
+ pub: string = sourcePublicDir(
+ process.env,
+ path.join(/* turbopackIgnore: true */ process.cwd(), "public"),
+ ),
+): SourceManifest | null {
+ try {
+ const manifest = parseSourceManifest(
+ JSON.parse(fs.readFileSync(path.join(/* turbopackIgnore: true */ pub, "source", "manifest.json"), "utf8")),
+ );
+ if (!manifest) return null;
+ if (!fs.statSync(path.join(/* turbopackIgnore: true */ pub, "source", MIRROR_DIR, "info", "refs")).isFile()) return null;
+ if (!fs.statSync(path.join(/* turbopackIgnore: true */ pub, TARBALL_HREF.replace(/^\//, ""))).isFile()) return null;
+ if (manifest.history && !isFile(path.join(/* turbopackIgnore: true */ pub, "source", HISTORY_DIR, "log.html"))) {
+ const without: SourceManifest = { ...manifest };
+ delete without.history;
+ return without;
+ }
+ return manifest;
+ } catch {
+ return null;
+ }
+}
+
+function isFile(p: string): boolean {
+ try {
+ return fs.statSync(p).isFile();
+ } catch {
+ return false;
+ }
+}
+
+// The directory loadSourceManifest reads: `public/`, or the one
+// E2E_SOURCE_PUBLIC_DIR names when it holds a manifest — but ONLY outside a
+// production build. The homepage e2e's `next dev` points it at a fixture
+// publish its spec writes and removes (e2e/fixture-source.ts), so the page's
+// states with and without a history are exercised whatever this checkout has
+// published; with the fixture gone, the page reads `public/` again. `next
+// build` runs with NODE_ENV=production and ignores it (the summary.ts rule).
+export function sourcePublicDir(
+ env: Readonly<Record<string, string | undefined>>,
+ publicDir: string,
+): string {
+ const fixture = env.NODE_ENV !== "production" ? env.E2E_SOURCE_PUBLIC_DIR : undefined;
+ if (fixture && isFile(path.join(/* turbopackIgnore: true */ fixture, "source", "manifest.json"))) return fixture;
+ return publicDir;
+}
diff --git a/homepage/app/lib/summary.test.ts b/homepage/app/lib/summary.test.ts
@@ -1,6 +1,9 @@
import { test } from "node:test";
import assert from "node:assert/strict";
-import { summaryFile } from "./summary";
+import fs from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { loadSummary, summaryFile } from "./summary";
// Run with:
// pnpm --filter homepage test
@@ -22,3 +25,36 @@ test("outside production (the e2e's next dev) it names the file to read", () =>
assert.equal(summaryFile({ NODE_ENV: "development" }, PUBLIC), PUBLIC);
assert.equal(summaryFile({ NODE_ENV: "development", E2E_HOMEPAGE_SUMMARY_FILE: "" }, PUBLIC), PUBLIC);
});
+
+// A summary written before release 14 carries no `wordmarkLead` on its sites;
+// it must still load, and its cards show the plain title (ArchiveCards). A lead
+// that is not a proper prefix of the title (hand-edited, or not a string) is
+// dropped on read, so the card never splits a title it does not start.
+test("the loader keeps a wordmarkLead only when it is a proper prefix of the title", () => {
+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), "hp-summary-"));
+ const file = path.join(dir, "homepage-summary.json");
+ const site = { siteId: "a", siteTitle: "Jeralyzer", siteDescription: "", siteUrl: "https://a.example" };
+ // Next types NODE_ENV as read-only; the test writes through a plain view.
+ const env = process.env as Record<string, string | undefined>;
+ const prev = { NODE_ENV: env.NODE_ENV, E2E: env.E2E_HOMEPAGE_SUMMARY_FILE };
+ const write = (sites: unknown[]) =>
+ fs.writeFileSync(file, JSON.stringify({ version: 5, totals: { transcripts: 1 }, sites }));
+ try {
+ delete env.NODE_ENV;
+ env.E2E_HOMEPAGE_SUMMARY_FILE = file;
+ write([site]);
+ assert.ok(!("wordmarkLead" in loadSummary()!.sites[0]), "an older summary: no lead");
+ write([{ ...site, wordmarkLead: "Jer" }]);
+ assert.equal(loadSummary()?.sites[0].wordmarkLead, "Jer");
+ for (const bad of ["Jeralyzer", "jer", "Xyz", "", 7, null]) {
+ write([{ ...site, wordmarkLead: bad }]);
+ assert.ok(!("wordmarkLead" in loadSummary()!.sites[0]), JSON.stringify(bad));
+ }
+ } finally {
+ if (prev.NODE_ENV === undefined) delete env.NODE_ENV;
+ else env.NODE_ENV = prev.NODE_ENV;
+ if (prev.E2E === undefined) delete env.E2E_HOMEPAGE_SUMMARY_FILE;
+ else env.E2E_HOMEPAGE_SUMMARY_FILE = prev.E2E;
+ fs.rmSync(dir, { recursive: true, force: true });
+ }
+});
diff --git a/homepage/app/lib/summary.ts b/homepage/app/lib/summary.ts
@@ -1,6 +1,10 @@
import fs from "node:fs";
import path from "node:path";
-import type { HomepageSummary } from "yt-dlp-transcript-common/lib/homepageSummary";
+import type {
+ HomepageSummary,
+ HomepageSummarySite,
+} from "yt-dlp-transcript-common/lib/homepageSummary";
+import { wordmarkLeadFor } from "yt-dlp-transcript-common/lib/brand";
// The build-time cross-site summary, or null when this build shipped without
// corpus data (`build:nodata`, or a source-only checkout that has never run
@@ -20,15 +24,15 @@ export function loadSummary(): HomepageSummary | null {
try {
const file = summaryFile(
process.env,
- path.join(process.cwd(), "public", "homepage-summary.json"),
+ path.join(/* turbopackIgnore: true */ process.cwd(), "public", "homepage-summary.json"),
);
const parsed = JSON.parse(
- fs.readFileSync(file, "utf8"),
+ fs.readFileSync(/* turbopackIgnore: true */ file, "utf8"),
) as HomepageSummary;
// A file that exists but carries no sites/totals is as uninformative as no
// file at all; treat it the same rather than rendering an empty dashboard.
if (!parsed || typeof parsed.totals?.transcripts !== "number") return null;
- return parsed;
+ return { ...parsed, sites: (parsed.sites ?? []).map(withCheckedLead) };
} catch {
return null;
}
@@ -47,3 +51,13 @@ export function summaryFile(
const override = env.NODE_ENV !== "production" ? env.E2E_HOMEPAGE_SUMMARY_FILE : undefined;
return override || publicFile;
}
+
+// A site's `wordmarkLead` (release 14) is used only when it is what compose
+// would have written: a proper prefix of its title (lib/brand.ts
+// wordmarkLeadFor). A summary from before release 14 has none, and one edited
+// by hand may hold anything; either way the card then shows the plain title.
+export function withCheckedLead(site: HomepageSummarySite): HomepageSummarySite {
+ const { wordmarkLead, ...rest } = site;
+ const lead = wordmarkLeadFor(site.siteTitle, wordmarkLead);
+ return lead ? { ...rest, wordmarkLead: lead } : rest;
+}
diff --git a/homepage/app/not-found.tsx b/homepage/app/not-found.tsx
@@ -1,6 +1,6 @@
import Link from "next/link";
import { PageShell, PageHeading } from "./components/PageShell";
-import { NAV } from "./lib/nav";
+import { FOOTER_NAV } from "./lib/nav";
// Branded 404. Static export renders this to out/404.html, which Cloudflare
// Pages serves for unmatched paths.
@@ -15,7 +15,7 @@ export default function NotFound() {
<nav aria-label="Site sections" className="flex flex-col gap-3">
<span className="label-machine">Try one of these</span>
<ul className="flex flex-wrap gap-x-6 gap-y-2 list-none">
- {[{ href: "/", label: "Home" }, ...NAV].map((item) => (
+ {[{ href: "/", label: "Home" }, ...FOOTER_NAV].map((item) => (
<li key={item.href}>
<Link
href={item.href}
diff --git a/homepage/app/page.tsx b/homepage/app/page.tsx
@@ -110,8 +110,13 @@ export default function Home() {
)}
{/* ── Official Instances ───────────────────────────────────────────── */}
+ {/* `#instances` is where every archive's header links (common/lib/
+ project.ts INSTANCES_URL); `scroll-mt` is the sticky header's height
+ (the bar, and below md the nav's rule under it), so the section's top
+ edge lands just under it. With no sites the section is absent and the
+ link lands on the top of the page. */}
{sites.length > 0 && (
- <section className="border-t border-[var(--border)]">
+ <section id="instances" className="scroll-mt-24 md:scroll-mt-14 border-t border-[var(--border)]">
<div className={`${CONTAINER} py-14 sm:py-20`}>
<h2 className="mb-8 font-display text-2xl font-semibold text-[var(--foreground)]">
Official Instances
diff --git a/homepage/app/source/page.tsx b/homepage/app/source/page.tsx
@@ -0,0 +1,235 @@
+import Link from "next/link";
+import type { Metadata } from "next";
+import {
+ CLONE_URL,
+ HISTORY_ATOM_HREF,
+ HISTORY_DIR,
+ HISTORY_LOG_HREF,
+ HISTORY_REFS_HREF,
+ TREE_HREF,
+} from "yt-dlp-transcript-common/lib/sourceManifest";
+import { PageShell, PageHeading } from "../components/PageShell";
+import { Fact } from "../components/Fact";
+import { loadSourceManifest } from "../lib/source";
+import { formatBytes } from "../lib/snapshot";
+
+export const metadata: Metadata = {
+ title: "Source",
+ description:
+ "The canonical public copy of Archilyzer's source: a read-only git mirror, its raw tree, and a tarball. MIT licensed.",
+};
+
+const linkClass =
+ "underline decoration-[var(--border-strong)] underline-offset-2 hover:text-[var(--foreground)]";
+
+// The published source (common/publish/source.ts): the mirror, the raw tree
+// and the tarball are build artefacts, so this page has two states and the
+// manifest decides which. A manifest is believed only when the files it
+// describes are here too (lib/source.ts): no clone command or link for a file
+// that isn't there. The History block likewise: only when the manifest has a
+// history (stagit rendered it) and its log page is here.
+export default function SourcePage() {
+ const manifest = loadSourceManifest();
+ const date = manifest
+ ? new Date(manifest.generatedAt).toLocaleDateString(undefined, {
+ year: "numeric",
+ month: "long",
+ day: "numeric",
+ })
+ : null;
+
+ return (
+ <PageShell className="flex flex-col gap-10">
+ <PageHeading
+ eyebrow="Code"
+ title="Source"
+ standfirst="The canonical public copy: a read-only git mirror, its raw tree, and a tarball."
+ />
+
+ <div className="doc-measure flex flex-col gap-4">
+ <p className="leading-[1.7] text-[var(--muted-foreground)]">
+ There is no GitHub, by choice. This site is where the code lives in
+ public: a read-only mirror of the private main branch, regenerated
+ with every deploy of this site. Commit ids differ from the private
+ repository because paths were scrubbed on the way out — same history,
+ different hashes. Nothing here takes a push or a pull request.
+ </p>
+ </div>
+
+ {manifest ? (
+ <div className="flex flex-col gap-6">
+ <div className="doc-measure">
+ <p className="mb-2 label-machine">Clone it</p>
+ <pre
+ data-testid="source-clone"
+ className="p-4 rounded-[var(--radius)] bg-[var(--surface)] border border-[var(--border)] overflow-x-auto text-[0.8125rem] font-mono text-[var(--foreground)]"
+ >
+ {`git clone ${CLONE_URL}`}
+ </pre>
+ </div>
+
+ <div className="doc-measure flex flex-col">
+ <Fact label="Mirror head" testId="source-mirror-head">
+ <span className="tabular">{manifest.mirrorHead}</span>
+ </Fact>
+ <Fact label="Reflects">
+ private <code className="font-mono">main</code> at{" "}
+ <span className="tabular">{manifest.sourceCommit}</span>
+ </Fact>
+ <Fact label="Subject">{manifest.subject}</Fact>
+ <Fact label="Generated">{date}</Fact>
+ <Fact label="Size">
+ <span className="tabular">
+ {manifest.tree.files.toLocaleString()} files in the tree;{" "}
+ {formatBytes(manifest.mirror.bytes)} of history in{" "}
+ {manifest.mirror.packs} pack{manifest.mirror.packs === 1 ? "" : "s"}
+ </span>
+ </Fact>
+ </div>
+
+ {manifest.history ? (
+ <div data-testid="source-history" className="doc-measure flex flex-col gap-2">
+ <p className="label-machine">History</p>
+ <p className="text-sm leading-[1.7] text-[var(--muted-foreground)]">
+ {manifest.history.total > manifest.history.commits
+ ? "The newest commits of main with their diffs, as static pages: the latest "
+ : "Every commit of main with its diff, as static pages: "}
+ <span data-testid="source-history-commits" className="tabular">
+ {manifest.history.commits.toLocaleString("en-US")}
+ </span>
+ {manifest.history.total > manifest.history.commits ? (
+ <>
+ {" "}of{" "}
+ <span data-testid="source-history-total" className="tabular">
+ {manifest.history.total.toLocaleString("en-US")}
+ </span>
+ </>
+ ) : null}{" "}
+ commit{manifest.history.total === 1 ? "" : "s"}, the newest{" "}
+ <a
+ data-testid="source-history-head"
+ href={`/source/${HISTORY_DIR}/commit/${manifest.history.head}.html`}
+ title={manifest.history.head}
+ className={`tabular ${linkClass}`}
+ >
+ {manifest.history.head.slice(0, 12)}
+ </a>
+ .
+ </p>
+ <p className="text-sm text-[var(--muted-foreground)]">
+ <a data-testid="source-history-log" href={HISTORY_LOG_HREF} className={linkClass}>
+ Log
+ </a>
+ {" · "}
+ <a data-testid="source-history-refs" href={HISTORY_REFS_HREF} className={linkClass}>
+ Refs
+ </a>
+ {" · "}
+ <a data-testid="source-history-atom" href={HISTORY_ATOM_HREF} className={linkClass}>
+ Atom feed
+ </a>
+ </p>
+ </div>
+ ) : null}
+
+ <div className="doc-measure flex flex-col gap-3">
+ <p>
+ <a data-testid="source-tree-link" href={TREE_HREF} className={`text-sm ${linkClass}`}>
+ Browse the tree
+ </a>
+ <span className="text-sm text-[var(--muted-foreground)]">
+ {" "}— every tracked file of main, raw, as plain text.
+ </span>
+ </p>
+ <p className="text-sm text-[var(--muted-foreground)]">
+ <a data-testid="source-tarball-link" href={manifest.tarball.href} className={linkClass}>
+ archilyzer-source.tar.gz
+ </a>{" "}
+ <span className="tabular">
+ ({formatBytes(manifest.tarball.bytes)}, sha256{" "}
+ <span data-testid="source-tarball-sha" className="break-all">
+ {manifest.tarball.sha256}
+ </span>
+ )
+ </span>{" "}
+ — the same tree with no history; the{" "}
+ <Link href="/downloads/" className={linkClass}>
+ Downloads
+ </Link>{" "}
+ page says what is in it.
+ </p>
+ </div>
+ </div>
+ ) : (
+ // No manifest, or not every file it describes: say so and render NO
+ // clone command and no link, exactly like the Downloads page.
+ <div data-testid="source-empty" className="doc-measure faceplate p-5 flex flex-col gap-3">
+ <p className="text-[var(--muted-foreground)] leading-relaxed">
+ No source published in this build.
+ </p>
+ <p className="text-sm text-[var(--faint)] leading-relaxed">
+ The mirror, the raw tree and the tarball are build artefacts,
+ generated by{" "}
+ <code className="font-mono">archilyzer source publish</code> (which{" "}
+ <code className="font-mono">archilyzer build homepage</code> runs)
+ and not committed. A site built without it says so here rather than
+ offering a clone that would fail.
+ </p>
+ </div>
+ )}
+
+ <div className="doc-measure flex flex-col gap-8 border-t border-[var(--border)] pt-8">
+ <div className="flex flex-col gap-3">
+ <h2 className="font-display text-base font-semibold text-[var(--foreground)]">
+ What is mirrored
+ </h2>
+ <ul className="list-disc pl-5 space-y-2 text-sm leading-[1.7] text-[var(--muted-foreground)] marker:text-[var(--faint)]">
+ <li>
+ <strong className="text-[var(--foreground)]">The main branch, whole.</strong>{" "}
+ Every commit since the first, with its message and author, and
+ every tracked file. No other branches and no tags.
+ </li>
+ <li>
+ <strong className="text-[var(--foreground)]">Scrubbed on the way out.</strong>{" "}
+ Machine paths such as the home directory are rewritten to generic
+ ones, in files and in commit messages, which is why every id
+ differs from the private repository’s. The build refuses to
+ publish if anything it was told to remove survives, anywhere in
+ the history.
+ </li>
+ <li>
+ <strong className="text-[var(--foreground)]">Regenerated, not pushed to.</strong>{" "}
+ Each deploy rebuilds the mirror from the private main. The ids
+ stay put from one deploy to the next, but a change to the scrub
+ rules rewrites them all — if a{" "}
+ <code className="font-mono">git pull</code> ever refuses
+ unrelated history, clone again.
+ </li>
+ <li>
+ <strong className="text-[var(--foreground)]">Static files.</strong>{" "}
+ The clone uses git’s plain-HTTP protocol, straight from this
+ site’s files: slower than a forge, and a fetch downloads
+ whole packs, but there is no server to go down.
+ </li>
+ <li>
+ <strong className="text-[var(--foreground)]">Not the archive.</strong>{" "}
+ No transcripts, media, settings or dependencies — the same list
+ as the tarball’s.
+ </li>
+ </ul>
+ </div>
+
+ <div className="flex flex-col gap-3">
+ <h2 className="font-display text-base font-semibold text-[var(--foreground)]">
+ License
+ </h2>
+ <p className="text-sm leading-[1.7] text-[var(--muted-foreground)]">
+ MIT. Use it, change it, run it, publish with it. The full text is in{" "}
+ <code className="font-mono text-[var(--foreground)]">LICENSE</code>{" "}
+ at the root of the tree.
+ </p>
+ </div>
+ </div>
+ </PageShell>
+ );
+}
diff --git a/homepage/content/README.md b/homepage/content/README.md
@@ -9,9 +9,12 @@ exists for whoever edits the docs next.
The obvious move is to render `SETUP.md`, `PUBLISH.md` and friends
directly, and keep one copy. That was rejected for a decisive reason:
-> `SETUP.md` says `git clone <this-repo-url>`. **There is no public repository.**
-> Rendering that on a public site publishes an instruction that cannot be
-> followed — the acquisition path is a dated snapshot tarball from `/downloads/`.
+> `SETUP.md` was written for a contributor with the repository, and its
+> acquisition path is not the public one. The public copy is the read-only
+> mirror at `/source/archilyzer.git` (regenerated per deploy, with different
+> commit ids) and the tarball at `/downloads/`; `docs/install.md` says so in
+> an operator's terms. (Until release 12 there was no public repository at all,
+> which is when this rule was made.)
The root docs also lean on contributor furniture that actively confuses an
operator: `pnpm wt` worktrees, the machine-global e2e queue, `plans/`, shard
@@ -33,7 +36,7 @@ its public counterpart needs the same change.
| `docs/operate.md` | `README.md`, `SCHEDULED_SYNC.md` | editor routes, scheduler settings |
| `docs/deploy-cloudflare.md` | `PUBLISH.md` (Cloudflare, R2, cost-abuse) | the 25 MB Pages limit, R2 options |
| `docs/deploy-docker.md` | `PUBLISH.md` (building every site in containers) | phase structure, settings names |
-| `docs/ai-and-mcp.md` | `mcp/README.md` | tool names, `corpus.json` shape |
+| `docs/ai-and-mcp.md` | `mcp/README.md`, `README.md` §1 and §4 | tool names, `corpus.json` shape, the Ten-minute setup's commands (README §1/§4 and mcp/README's "Add to Claude Code" are copies of them: change all three together) |
| `docs/faq.md` | — (written for this site) | claims about cost and hardware |
## House rules for these files
diff --git a/homepage/content/docs/ai-and-mcp.md b/homepage/content/docs/ai-and-mcp.md
@@ -29,8 +29,44 @@ client.
It is **a local tool you run yourself**. It changes nothing: it only reads
already-published static JSON, either from a directory on disk or over HTTP.
-
-What it can do:
+The one exception is `fetch_clip`, which asks a local Archilyzer editor for a
+clip's media: the editor writes it, and the server itself never writes.
+
+### Ten-minute setup
+
+To run Claude Code against a published archive, with nothing of your own
+hosted — the Jeralyzer here, and any archive's URL works the same:
+
+```sh
+git clone https://archilyzer.pages.dev/source/archilyzer.git archilyzer # or the tarball on /downloads/
+cd archilyzer && pnpm install
+claude mcp add archilyzer \
+ --env TRANSCRIPT_SITE_URL=https://jeralyzer.pages.dev \
+ -- pnpm -C "$PWD" --filter yt-dlp-transcript-mcp exec tsx src/index.ts
+claude # then: /ask what has he said about …
+```
+
+- **What you need:** Node.js 20.9 or newer, pnpm 9 or newer, and Claude Code.
+ No corpus, no yt-dlp, no GPU, nothing hosted.
+- **The source** is the project's read-only [git mirror](/source/); without
+ git, the same tree is a [tarball](/downloads/): unpack it and carry on from
+ `cd archilyzer`.
+- **Register it as `archilyzer`.** The shipped `/ask` and `/sweep` commands
+ call `mcp__archilyzer__ask_plan` / `mcp__archilyzer__sweep_plan`, and that
+ tool name embeds the server name as you registered it.
+- **Clips:** two optional `--env` lines, `ARCHILYZER_EDITOR_URL` and
+ `WORKER_TOKEN` (the editor's own), let `fetch_clip` ask a local editor for
+ clip media; leave them out for research alone.
+- **One archive or several:** `TRANSCRIPT_SITE_URL` reads one archive;
+ `TRANSCRIPT_HUB_URL`, given a hub's URL, federates every archive on the hub.
+- **Another client** (Claude Desktop, Cursor): the same server as an
+ `mcp.json` entry is in
+ [mcp/README.md](https://archilyzer.pages.dev/source/tree/mcp/README.md).
+- **On Windows,** run all of this inside WSL2, Claude Code included: see
+ “Claude Code on Windows” in the
+ [README](https://archilyzer.pages.dev/source/tree/README.md).
+
+### What it can do
- **Search transcripts** for a term, a phrase or a regular expression, with
timestamped snippets. Every timestamp is a link to that exact second of the
diff --git a/homepage/content/docs/faq.md b/homepage/content/docs/faq.md
@@ -78,9 +78,12 @@ gone* — never "confirmed still there".
## Where is the git repository?
-There isn't one. The source is published as a
-[dated snapshot tarball](/downloads/): a working tree with no history, no
-branches and no remote. Updating means downloading a newer snapshot.
+On this site, read-only: `git clone https://archilyzer.pages.dev/source/archilyzer.git`.
+It is the main branch with its whole history, regenerated from the private
+repository with every deploy — machine paths scrubbed on the way out, which is
+why its commit ids differ. There is no GitHub, and nothing takes a push or a
+pull request. [Source](/source/) also has every file to browse and the history
+with every commit's diff, and [Downloads](/downloads/) the same tree as a tarball.
## Can I use it for one video?
@@ -92,7 +95,8 @@ exists because of scale.
## What this is not
- **Not a hosted service.** There is no account, no subscription, no upload.
-- **Not a public repository.** Dated snapshots, no `git pull`.
+- **Not a forge.** A read-only mirror: clone and pull, but no pushes, issues or
+ pull requests.
- **Not a downloader.** It supplies no downloader — you install `yt-dlp`,
`ffmpeg` and a transcription backend yourself, and it drives them.
- **Not cloud transcription.** Your hardware, your electricity, your queue.
diff --git a/homepage/content/docs/install.md b/homepage/content/docs/install.md
@@ -1,8 +1,8 @@
# Install
-From an unpacked snapshot to a running editor. The web apps need very little; the
-download-and-transcribe pipeline needs the media tools, and you can add those
-later.
+From a clone (or an unpacked snapshot) to a running editor. The web apps need
+very little; the download-and-transcribe pipeline needs the media tools, and you
+can add those later.
## What you need
@@ -29,8 +29,20 @@ can't fetch anything yet.
## Get the code
-There is no public git repository. The source is published here as a dated
-snapshot tarball:
+The source lives on this site as a read-only git mirror of the main branch —
+there is no GitHub. Clone it:
+
+```sh
+git clone https://archilyzer.pages.dev/source/archilyzer.git archilyzer
+cd archilyzer
+```
+
+`git pull` brings you up to date; nothing takes a push. The commit ids differ
+from the private repository's, because machine paths are scrubbed on the way
+out. [Source](/source/) has the details, a browsable copy of every file, and the
+history with every commit's diff.
+
+No git? The same tree, without history, is a tarball:
```sh
curl -LO https://archilyzer.pages.dev/downloads/archilyzer-source.tar.gz
@@ -38,9 +50,8 @@ tar xzf archilyzer-source.tar.gz
cd archilyzer
```
-That is a working tree, not a clone — no history, no branches, no remote,
-nothing to `git pull`. Updating means downloading a newer snapshot. See
-[Downloads](/downloads/) for what is and isn't inside.
+That is a working tree, not a clone. Updating means downloading a newer
+snapshot. See [Downloads](/downloads/) for what is and isn't inside.
## Install and start
@@ -122,8 +133,8 @@ up rather than looking healthy and quietly transcribing nothing.
`127.0.0.1` — the published archive included — so opening one up is a deliberate
one-line edit in `.env`. The editor has no login of its own, so the containers
refuse to start if you expose *it* without putting a password or an identity
-provider in front; the error says how. `RUNNING_IN_DOCKER.md`, in the snapshot
-you just unpacked, covers that, GPU transcription, and backups.
+provider in front; the error says how. `RUNNING_IN_DOCKER.md`, in the tree you
+just unpacked, covers that, GPU transcription, and backups.
The same stack runs on Linux and macOS. It is described here because it is the
one place where it is clearly the *better* option, not because it is
diff --git a/homepage/content/docs/operate.md b/homepage/content/docs/operate.md
@@ -82,9 +82,9 @@ editor. See [Deploy to Cloudflare](/docs/deploy-cloudflare/) for hosting, and
Channels live in a single shared pool. A **site** is a selection of them with its
own title, description, accent colour and domain. The accent is one of seven
-named colours or a hex of your own, and each site opens in it; a reader can
-still pick another accent, and a light, sepia or dark ground, from the site's
-theme menu. One corpus can therefore publish several public archives without any
+named colours or a hex of your own, and every page of the site wears it; a
+reader picks a light or dark ground (or the system's) with the header's theme
+toggle. One corpus can therefore publish several public archives without any
data being duplicated — and a channel can appear on more than one.
Within a site, channels can be arranged into named groups, which is what drives
diff --git a/homepage/content/docs/what-is-archilyzer.md b/homepage/content/docs/what-is-archilyzer.md
@@ -60,8 +60,8 @@ Steps 1–3 can run unattended on a schedule. See
## What it is not
- Not a hosted service. You install it, you run it, you pay for your own hosting.
-- Not a public repository. The source ships as a
- [dated snapshot](/downloads/) — there is no `git clone` and nothing to pull.
+- Not a forge. The source is a [read-only git mirror](/source/) on this site —
+ clone and pull, but nothing takes a push or a pull request.
- Not a downloader you point at one video. It is built around back catalogues:
thousands of recordings, kept current.
- Not automatic transcription in the cloud. The transcribing happens on your
diff --git a/homepage/e2e/brand.spec.ts b/homepage/e2e/brand.spec.ts
@@ -40,10 +40,10 @@ test("the icon set is rendered from the parent mark", async ({ request }) => {
// The ring on dark (release 10, slice MR): on the dark base the tile has no
// edge, so every mark's tile gets a 1px ring outside it in its palette's dim,
// following the ground's corner (rx 112 of 512). A box-shadow, so the mark's
-// box is the same on every base; light and sepia draw none.
+// box is the same on every base; light draws none.
const BASE_KEY = "ytdlp-tb:base";
-async function onBase(page: Page, base: "light" | "sepia" | "dark") {
+async function onBase(page: Page, base: "light" | "dark") {
await page.evaluate(([k, b]) => localStorage.setItem(k, b), [BASE_KEY, base]);
await page.reload({ waitUntil: "commit" });
await page.waitForFunction(() => document.documentElement?.dataset.themeReady === "1");
@@ -69,7 +69,7 @@ async function markRing(mark: Locator) {
});
}
-test("on dark the header mark has a 1px ring in the slate's dim; on light and sepia none", async ({
+test("on dark the header mark has a 1px ring in the slate's dim; on light none", async ({
page,
}) => {
await page.goto("/");
@@ -85,7 +85,7 @@ test("on dark the header mark has a 1px ring in the slate's dim; on light and se
size: [28, 28],
});
- for (const base of ["light", "sepia"] as const) {
+ for (const base of ["light"] as const) {
await onBase(page, base);
expect(await markRing(mark), base).toEqual({
ring: [],
@@ -110,7 +110,7 @@ test.describe("under forced colours", () => {
const mark = page
.getByRole("link", { name: "Archilyzer home", exact: true })
.locator("svg[data-brand-mark]");
- for (const base of ["dark", "light", "sepia"] as const) {
+ for (const base of ["dark", "light"] as const) {
await onBase(page, base);
expect(await page.evaluate(() => matchMedia("(forced-colors: active)").matches)).toBe(true);
const got = await mark.evaluate((svg) => {
diff --git a/homepage/e2e/docs.spec.ts b/homepage/e2e/docs.spec.ts
@@ -61,6 +61,40 @@ test("an internal cross-link stays in the tab; an external one doesn't", async (
}
});
+// Release 16 slice DX: the research-only setup lives here alone, and every
+// archive's "Use with AI" link lands on this page.
+test("the AI and MCP doc's Ten-minute setup registers archilyzer and links the source", async ({
+ page,
+}) => {
+ await page.goto("/docs/ai-and-mcp/");
+ await expect(page.locator(".doc-measure h3#ten-minute-setup")).toBeVisible();
+
+ const block = page.locator(".doc-measure pre", { hasText: "claude mcp add archilyzer" });
+ await expect(block).toHaveCount(1);
+ const lines = (await block.innerText()).trim().split("\n");
+ expect(lines[0]).toMatch(/^git clone https:\/\/archilyzer\.pages\.dev\/source\/archilyzer\.git archilyzer\b/);
+ expect(lines[1]).toBe("cd archilyzer && pnpm install");
+ expect(lines[2]).toBe("claude mcp add archilyzer \\");
+ expect(lines[3]).toBe(" --env TRANSCRIPT_SITE_URL=https://jeralyzer.pages.dev \\");
+ expect(lines[4]).toBe(
+ ' -- pnpm -C "$PWD" --filter yt-dlp-transcript-mcp exec tsx src/index.ts',
+ );
+ expect(lines[5]).toMatch(/^claude\s+# then:\s+\/ask /);
+
+ // The source and the tarball, linked as the other docs link them: in place.
+ const source = page.locator('.doc-measure a[href="/source/"]', { hasText: "git mirror" });
+ await expect(source).toBeVisible();
+ await expect(source).not.toHaveAttribute("target", "_blank");
+ await expect(
+ page.locator('.doc-measure a[href="/downloads/"]', { hasText: "tarball" }),
+ ).toBeVisible();
+
+ // The name's reason sits beside it.
+ await expect(
+ page.locator(".doc-measure li", { hasText: "Register it as archilyzer" }),
+ ).toContainText("mcp__archilyzer__ask_plan / mcp__archilyzer__sweep_plan");
+});
+
test("an unknown doc slug is a 404, not a crash", async ({ page }) => {
const res = await page.request.get("/docs/not-a-real-page/");
expect(res.status()).toBe(404);
diff --git a/homepage/e2e/fixture-social.ts b/homepage/e2e/fixture-social.ts
@@ -0,0 +1,120 @@
+import fs from "node:fs";
+import {
+ normalizeSocialSvg,
+ type SocialLink,
+} from "../../common/lib/settingsSchema";
+import { REAL_SHAPES } from "../../common/lib/socialSvg.vectors";
+
+// THE HOMEPAGE E2E'S SOCIAL LINKS, never the operator's.
+//
+// The header and the footer render `socialLinks` from settings.json. A checkout's
+// settings.json is the operator's own (a worktree seeds it from the primary), so
+// the e2e dev server reads THIS instead: `SETTINGS_FILE` (common/lib/paths.ts)
+// points it at e2e/.e2e-settings.json (gitignored), written by
+// playwright.config.ts before the server starts. Only the social links are set;
+// every other key reads as its default, and nothing else on the homepage reads
+// settings. (`SITES_DIR` points at an empty directory too, so no homepage.json
+// can override these links.)
+//
+// Each icon is given here in its SOURCE form — as an operator would paste it —
+// and goes through normalizeSocialSvg, the function every save path runs
+// (Settings, a site's form, the homepage config), so the page renders what a
+// save would have stored. The default trio is shaped like the icons operators
+// paste, all synthetic:
+// • Leaf — a gradient (an id, a url(#…) reference) and one solid colour,
+// which the normalizer themes to currentColor;
+// • Bubble — one colour, drawn white, as vendors ship them for dark grounds;
+// themed to currentColor;
+// • Disc — a two-colour disc with a letter, pasted with only width/height and
+// NO viewBox, so the page proves the normalizer made its viewBox.
+//
+// A spec that needs another list (six links, none, a hostile icon) rewrites the
+// file with writeFixtureSettings (or writeRawFixtureSettings, which skips the
+// normalizer the way a hand-edited file would) and restores FIXTURE_SOCIAL_TRIO
+// after: `next dev` reads settings on every request, and the suite runs one
+// worker.
+
+export const FIXTURE_SETTINGS_NAME = ".e2e-settings.json";
+
+// The gradient icon is the checker's regression shape (socialSvg.vectors.ts
+// REAL_SHAPES): a drawing program's root, a gradient with `style` stops, a dark
+// outline under the body (`paint-order`), a negative-origin viewBox.
+const LEAF = REAL_SHAPES.gradient_outlined;
+
+const BUBBLE =
+ `<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none">` +
+ `<path fill="white" d="M4 4h16a2 2 0 0 1 2 2v9a2 2 0 0 1-2 2h-7l-5 4v-4H4a2 2 0 0 1-2-2V6a2 2 0 0 1 2-2z"/></svg>`;
+
+// A two-colour disc with a letter, sized and with no viewBox (the normalizer
+// makes `viewBox="0 0 81 81"`): a dark offset disc, a light disc with a dark
+// outline, a dark "F".
+export const FIXTURE_DISC_SVG =
+ `<svg xmlns="http://www.w3.org/2000/svg" width="81" height="81" fill="none">` +
+ `<circle cx="44" cy="43" r="37" fill="#1a1a1a"/>` +
+ `<circle cx="39" cy="39" r="38" fill="#f4c542" stroke="#1a1a1a" stroke-width="1.5"/>` +
+ `<path fill="#1a1a1a" d="M30 22h20v8H38v6h10v8H38v14h-8z"/></svg>`;
+
+// Icons that must never run: each sets window.__hpHostile if it does. Written
+// raw (writeRawFixtureSettings), as a hand-edited settings.json could hold them.
+export const FIXTURE_HOSTILE_SVGS = [
+ `<svg/onload="window.__hpHostile=1" viewBox="0 0 8 8"><path d="M0 0"/></svg>`,
+ `<svg viewBox="0 0 8 8"><img src="x:" onerror="window.__hpHostile=1"></svg>`,
+ `<svg viewBox="0 0 8 8"><image href="x:"/onerror="window.__hpHostile=1"/></svg>`,
+];
+
+const SQUARE = `<svg viewBox="0 0 24 24"><rect x="4" y="4" width="16" height="16" rx="3"/></svg>`;
+const RING = `<svg viewBox="0 0 24 24"><path fill-rule="evenodd" d="M12 3a9 9 0 1 1 0 18 9 9 0 0 1 0-18zm0 4a5 5 0 1 0 0 10 5 5 0 0 0 0-10z"/></svg>`;
+const TRIANGLE = `<svg viewBox="0 0 24 24"><path d="M12 3l10 18H2z"/></svg>`;
+
+const raw = (label: string, svg: string, featured = false) => ({
+ label,
+ url: `https://${label.toLowerCase()}.example/fixture`,
+ svg,
+ ...(featured ? { featured: true } : {}),
+});
+
+// The LAST link is marked for the header (`featured`): a narrow header shows it
+// alone, a wide one shows all three.
+export const FIXTURE_SOCIAL_TRIO = [
+ raw("Leaf", LEAF),
+ raw("Bubble", BUBBLE),
+ raw("Disc", FIXTURE_DISC_SVG, true),
+];
+
+// Six, the last (Disc) marked: a wide header takes the last four, a narrow one
+// Disc alone.
+export const FIXTURE_SOCIAL_SIX = [
+ raw("Square", SQUARE),
+ raw("Ring", RING),
+ ...FIXTURE_SOCIAL_TRIO.slice(0, 1),
+ raw("Triangle", TRIANGLE),
+ ...FIXTURE_SOCIAL_TRIO.slice(1),
+];
+
+// Six, two marked (Square, Bubble; Disc not): a narrow header takes those two,
+// a wide one those two and then the last two of the rest.
+export const FIXTURE_SOCIAL_SIX_FEATURED = FIXTURE_SOCIAL_SIX.map(({ featured: _, ...l }) =>
+ l.label === "Square" || l.label === "Bubble" ? { ...l, featured: true } : l,
+);
+
+// The same links with no mark at all: a narrow header shows none of them.
+export const unmarked = <T extends { featured?: boolean }>(links: readonly T[]) =>
+ links.map(({ featured: _, ...l }) => l);
+
+type RawLink = { label: string; url: string; svg: string; featured?: boolean };
+
+export function writeFixtureSettings(dest: string, links: RawLink[]): void {
+ const socialLinks: SocialLink[] = links.map((link) => {
+ const svg = normalizeSocialSvg(link.svg);
+ if (svg === null) throw new Error(`fixture icon "${link.label}" was refused`);
+ return { ...link, svg };
+ });
+ writeRawFixtureSettings(dest, socialLinks);
+}
+
+// The links as given, NOT normalized: what a hand-edited settings.json holds.
+export function writeRawFixtureSettings(dest: string, links: RawLink[]): void {
+ const tmp = `${dest}.${process.pid}.tmp`;
+ fs.writeFileSync(tmp, JSON.stringify({ socialLinks: links }, null, 2));
+ fs.renameSync(tmp, dest);
+}
diff --git a/homepage/e2e/fixture-source.ts b/homepage/e2e/fixture-source.ts
@@ -0,0 +1,70 @@
+import fs from "node:fs";
+import path from "node:path";
+
+// A FIXTURE PUBLISH for the /source/ page's History block (release 15 slice
+// SG). The page reads the directory E2E_SOURCE_PUBLIC_DIR names instead of
+// public/ while it holds a manifest (app/lib/source.ts sourcePublicDir,
+// honoured only outside a production build), so a spec can show the page with
+// a history and without one, whatever this checkout has published. The spec
+// writes it, and removes it after each test: with it gone, the page — and
+// every other spec — reads public/ again. playwright.config.ts clears it
+// before the server starts, in case a killed run left one.
+//
+// Only the files the loader requires: the manifest, the mirror's info/refs,
+// the tarball, and — with a history — the log page. Nothing here is served:
+// the dev server serves public/, so the History links are checked by their
+// hrefs, and the real pages by source.spec.ts.
+
+export const FIXTURE_SOURCE_NAME = ".e2e-source";
+
+export const FIXTURE_HISTORY = {
+ href: "/source/git/log.html",
+ commits: 1234,
+ total: 1234,
+ head: "c".repeat(40),
+ files: 1240,
+ bytes: 1_000_000,
+ sha256: "d".repeat(64),
+ tool: "stagit (sha256 0123456789ab)",
+};
+
+export const FIXTURE_MIRROR_HEAD = "c".repeat(40);
+
+// `total` over the history's `commits` is a history past the cap: the page
+// says "the latest N of M".
+export function writeFixtureSource(dir: string, o: { history: boolean; total?: number }): void {
+ clearFixtureSource(dir);
+ fs.mkdirSync(path.join(dir, "source", "archilyzer.git", "info"), { recursive: true });
+ fs.mkdirSync(path.join(dir, "downloads"), { recursive: true });
+ fs.writeFileSync(path.join(dir, "source", "archilyzer.git", "info", "refs"), `${FIXTURE_MIRROR_HEAD}\trefs/heads/main\n`);
+ fs.writeFileSync(path.join(dir, "downloads", "archilyzer-source.tar.gz"), "fixture");
+ if (o.history) {
+ fs.mkdirSync(path.join(dir, "source", "git"), { recursive: true });
+ fs.writeFileSync(path.join(dir, "source", "git", "log.html"), "<html></html>\n");
+ }
+ const manifest = {
+ version: 1,
+ generatedAt: "2026-09-30T12:00:00.000Z",
+ branch: "main",
+ sourceCommit: "b".repeat(40),
+ mirrorHead: FIXTURE_MIRROR_HEAD,
+ subject: "a fixture publish",
+ files: 10,
+ bytes: 100,
+ mirror: { files: 9, bytes: 80, packs: 1 },
+ tree: { files: 5, dirs: 2, bytes: 20 },
+ tarball: { href: "/downloads/archilyzer-source.tar.gz", bytes: 7, sha256: "e".repeat(64) },
+ cloneUrl: "https://archilyzer.pages.dev/source/archilyzer.git",
+ treeHref: "/source/tree/",
+ audit: { objects: 3, commits: 1, gitleaks: "skipped" },
+ tools: { git: "2.55.0", filterRepo: "fixture" },
+ ...(o.history ? { history: { ...FIXTURE_HISTORY, total: o.total ?? FIXTURE_HISTORY.total } } : {}),
+ };
+ // Written last, as the publish writes it: the page reads the fixture only
+ // once a manifest is there.
+ fs.writeFileSync(path.join(dir, "source", "manifest.json"), JSON.stringify(manifest, null, 2));
+}
+
+export function clearFixtureSource(dir: string): void {
+ fs.rmSync(dir, { recursive: true, force: true });
+}
diff --git a/homepage/e2e/fixture-summary.ts b/homepage/e2e/fixture-summary.ts
@@ -26,7 +26,17 @@ import type { VideoStat } from "../../common/lib/stats";
// symlog axis reaches 10K while the Recent weeks stay linear and small;
// • every site transcribes every day of the last 200, so every line is full
// and every series has a non-zero start for Indexed;
-// • uploads spread over 2019–2026, for the growth chart's months.
+// • uploads spread over 2019–2026, for the growth chart's months;
+// • the last two sites each under 5 % of the growth chart's total (4.35 and
+// 3.26 %), so the chart folds them into its Other band (release 14 slice
+// CF; fixture-five transcribes 4 a day, not 5, for it);
+// • a seventh site, UNLISTED (site.json `listed: false`, release 14 slice
+// HS): its own two channels transcribe every day like the rest, and it
+// also exposes the first site's first channel. The summary names it nowhere
+// and counts its own channels in no total, so `buildFixtureSummary()` is
+// exactly `buildFixtureSummary(FIXTURE_SITES, [])` — every number the specs
+// read is the six listed sites'. Every site in the second list is unlisted:
+// `buildFixtureInputs` sets `listed: false` on it, whatever it carries.
export const FIXTURE_SUMMARY_NAME = ".e2e-summary.json";
@@ -34,20 +44,43 @@ export const FIXTURE_SUMMARY_NAME = ".e2e-summary.json";
// not charted; the week and day buckets end here.
export const FIXTURE_NOW = new Date("2026-09-15T12:00:00Z");
-// A pale custom hex: published as is it would be a ghost on light and sepia.
+// A pale custom hex: published as is it would be a ghost on light.
export const FIXTURE_PALE_HEX = "#f4c2d7";
// The six sites, in summary order. `accent` is the site.json setting (an
-// accent id or a custom hex), as compose reads it; `daily` how many
-// recordings each channel transcribes a day.
-export const FIXTURE_SITES = [
- { siteId: "fixture-one", siteTitle: "Fixture One", accent: "brass", channels: 4, daily: 15 },
- { siteId: "fixture-two", siteTitle: "Fixture Two", accent: FIXTURE_PALE_HEX, channels: 3, daily: 9 },
- { siteId: "fixture-three", siteTitle: "Fixture Three", accent: undefined, channels: 3, daily: 7 },
- { siteId: "fixture-four", siteTitle: "Fixture Four", accent: undefined, channels: 2, daily: 8 },
- { siteId: "fixture-five", siteTitle: "Fixture Five", accent: undefined, channels: 2, daily: 5 },
+// accent id or a custom hex), as compose reads it; `wordmarkLead` the
+// site.json one (a named accent's site, the custom hex's and a site with no
+// accent have one — the last ends MID-WORD, "Fix" + "ture Three", as
+// "Jer" + "alyzer" does; the rest show their title plain); `daily` how many
+// recordings each channel transcribes a day. Whether a site is listed is the
+// list it is passed in (buildFixtureInputs), not a field.
+export type FixtureSite = {
+ siteId: string;
+ siteTitle: string;
+ accent?: string;
+ wordmarkLead?: string;
+ channels: number;
+ daily: number;
+};
+
+export const FIXTURE_SITES: readonly FixtureSite[] = [
+ { siteId: "fixture-one", siteTitle: "Fixture One", accent: "brass", wordmarkLead: "Fixture", channels: 4, daily: 15 },
+ { siteId: "fixture-two", siteTitle: "Fixture Two", accent: FIXTURE_PALE_HEX, wordmarkLead: "Fixture", channels: 3, daily: 9 },
+ { siteId: "fixture-three", siteTitle: "Fixture Three", wordmarkLead: "Fix", channels: 3, daily: 7 },
+ { siteId: "fixture-four", siteTitle: "Fixture Four", channels: 2, daily: 8 },
+ { siteId: "fixture-five", siteTitle: "Fixture Five", channels: 2, daily: 4 },
{ siteId: "fixture-six", siteTitle: "Fixture Six", accent: "vermilion", channels: 2, daily: 3 },
-] as const;
+];
+
+// The unlisted site (an invented id, as every fixture's). Not in FIXTURE_SITES,
+// which is the six the pages show; buildFixtureInputs' second list, which
+// unlists it.
+export const FIXTURE_UNLISTED_SITE: FixtureSite = {
+ siteId: "fixture-unlisted",
+ siteTitle: "Fixture Unlisted",
+ channels: 2,
+ daily: 6,
+};
const DAY = 86_400_000;
const SPIKE = { site: 0, start: Date.UTC(2026, 4, 11), days: 7, count: 12_000 };
@@ -64,7 +97,13 @@ function uploadDate(n: number): string {
return ymd(d.getTime());
}
-export function buildFixtureSummary() {
+// The builder's inputs: the recordings, the channel → sites map and the sites.
+// `fixtureSites` are listed; every site in `unlistedSites` is written with
+// `listed: false`, and each also exposes the first listed site's first channel.
+export function buildFixtureInputs(
+ fixtureSites: readonly FixtureSite[] = FIXTURE_SITES,
+ unlistedSites: readonly FixtureSite[] = [FIXTURE_UNLISTED_SITE],
+): { stats: VideoStat[]; channelSites: Record<string, string[]>; sites: Site[] } {
const stats: VideoStat[] = [];
const channelSites: Record<string, string[]> = {};
const sites: Site[] = [];
@@ -100,16 +139,24 @@ export function buildFixtureSummary() {
});
};
const today = Date.UTC(2026, 8, 15);
- FIXTURE_SITES.forEach((s, i) => {
+ // A site's own channels, `<siteId>-ch<N>`, each transcribing every day of the
+ // last HISTORY_DAYS; `shared` are other sites' channels it also exposes.
+ const addSite = (
+ s: FixtureSite,
+ i: number,
+ { listed = true, shared = [] }: { listed?: boolean; shared?: string[] } = {},
+ ) => {
const slugs = Array.from({ length: s.channels }, (_, c) => `${s.siteId}-ch${c + 1}`);
- for (const slug of slugs) channelSites[slug] = [s.siteId];
+ for (const slug of [...slugs, ...shared]) (channelSites[slug] ??= []).push(s.siteId);
sites.push({
siteId: s.siteId,
siteTitle: s.siteTitle,
siteDescription: `${s.siteTitle}, an e2e fixture archive.`,
siteUrl: `https://${s.siteId}.example`,
- channels: slugs.map((slug) => ({ slug })),
+ channels: [...slugs, ...shared].map((slug) => ({ slug })),
...(s.accent ? { accent: s.accent } : {}),
+ ...(s.wordmarkLead ? { wordmarkLead: s.wordmarkLead } : {}),
+ ...(listed ? {} : { listed: false }),
} as unknown as Site);
slugs.forEach((slug, c) => {
const name = `${s.siteTitle} Channel ${c + 1}`;
@@ -120,13 +167,31 @@ export function buildFixtureSummary() {
for (let k = 0; k < count; k++) record(i, slug, name, day);
}
});
- });
+ };
+ fixtureSites.forEach((s, i) => addSite(s, i));
// The megaspike: one week of bulk transcription on the first site's first
// channel.
- const spikeSite = FIXTURE_SITES[SPIKE.site];
+ const spikeSite = fixtureSites[SPIKE.site];
for (let k = 0; k < SPIKE.count; k++) {
record(SPIKE.site, `${spikeSite.siteId}-ch1`, `${spikeSite.siteTitle} Channel 1`, SPIKE.start + (k % SPIKE.days) * DAY);
}
+ // The unlisted sites LAST, so every listed site's recordings (their numbers,
+ // dates and states) are the same with them or without.
+ unlistedSites.forEach((s, j) =>
+ addSite(s, fixtureSites.length + j, {
+ listed: false,
+ shared: [`${spikeSite.siteId}-ch1`],
+ }),
+ );
+ return { stats, channelSites, sites };
+}
+
+// The summary the e2e dev server reads: the real builder over those inputs.
+export function buildFixtureSummary(
+ fixtureSites: readonly FixtureSite[] = FIXTURE_SITES,
+ unlistedSites: readonly FixtureSite[] = [FIXTURE_UNLISTED_SITE],
+) {
+ const { stats, channelSites, sites } = buildFixtureInputs(fixtureSites, unlistedSites);
return buildHomepageSummary(stats, channelSites, sites, FIXTURE_NOW);
}
diff --git a/homepage/e2e/growth-chart.spec.ts b/homepage/e2e/growth-chart.spec.ts
@@ -0,0 +1,249 @@
+import { test, expect, type Page } from "@playwright/test";
+import { buildFixtureSummary } from "./fixture-summary";
+import { painted, rgbOf } from "../../common/testing/chartPixels";
+import { siteChartColors } from "../../common/lib/siteColor";
+import { OTHER_COLOR, growthLayers } from "../app/lib/growthGaps";
+
+// The growth chart's bands are parted by the marks spec's SURFACE GAP: 2 px
+// along each band's upper edge, in the colour behind the plot — the page
+// ground, which the chart sits on — never a line in the text colour
+// (ArchiveGrowthChart.tsx, globals.css `.growth-gap`), and the reader's Canvas
+// in forced colours. The fixture summary (fixture-summary.ts) has six sites
+// with monthly data, so the chart and its gaps render; its last two are each
+// under 5 % of the chart's total, so they are drawn as one Other band on top
+// (lib/growthGaps.ts, release 14 slice CF).
+
+const BASE_KEY = "ytdlp-tb:base";
+
+const resolveColor = (page: Page, css: string) =>
+ page.evaluate((c) => {
+ const el = document.createElement("span");
+ el.style.color = c;
+ document.body.append(el);
+ const out = getComputedStyle(el).color;
+ el.remove();
+ return out;
+ }, css);
+
+const gaps = (page: Page) =>
+ page.locator("figure svg path.growth-gap").evaluateAll((els) =>
+ els.map((el) => {
+ const s = getComputedStyle(el);
+ return { stroke: s.stroke, width: s.strokeWidth, scaling: s.vectorEffect };
+ }),
+ );
+
+test("the bands are parted by a 2 px gap in the ground's colour, never the text colour", async ({
+ page,
+}) => {
+ await page.goto("/");
+ for (const base of ["light", "dark"] as const) {
+ await page.evaluate(([k, v]) => localStorage.setItem(k, v), [BASE_KEY, base]);
+ await page.reload();
+ await expect(page.locator("html")).toHaveAttribute("data-base", base);
+ // The chart sits on the page ground: nothing between them paints.
+ const holder = await page.locator("figure").evaluate((el) => {
+ for (let n: Element | null = el; n && n !== document.body; n = n.parentElement) {
+ const bg = getComputedStyle(n).backgroundColor;
+ if (bg !== "rgba(0, 0, 0, 0)" && bg !== "transparent") return bg;
+ }
+ return getComputedStyle(document.body).backgroundColor;
+ });
+ const ground = await resolveColor(page, "var(--background)");
+ const text = await resolveColor(page, "var(--foreground)");
+ expect(holder, `${base}: what the chart sits on`).toBe(ground);
+ expect(ground).not.toBe(text);
+ const seen = await gaps(page);
+ expect(seen.length, base).toBeGreaterThan(0);
+ for (const g of seen) {
+ expect(g, base).toEqual({ stroke: ground, width: "2px", scaling: "non-scaling-stroke" });
+ }
+ }
+});
+
+test("in forced colours the gap is the reader's Canvas", async ({ page }) => {
+ await page.emulateMedia({ forcedColors: "active" });
+ await page.goto("/");
+ const canvas = await resolveColor(page, "Canvas");
+ const seen = await gaps(page);
+ expect(seen.length).toBeGreaterThan(0);
+ for (const g of seen) expect(g.stroke).toBe(canvas);
+});
+
+// One set of gaps per plot height (lib/growthGaps.ts), each shown only at its
+// own breakpoint: the set worked out for the height the plot is drawn at.
+for (const [width, shown] of [
+ [390, "200"],
+ [768, "260"],
+ [1280, "300"],
+] as const) {
+ test(`${width} px: only the ${shown} px plot's gaps are shown`, async ({ page }) => {
+ await page.setViewportSize({ width, height: 900 });
+ await page.goto("/");
+ const plot = page.locator('figure [role="img"]');
+ expect(Math.round((await plot.boundingBox())!.height)).toBe(Number(shown));
+ const sets = await page.locator("figure svg g[data-plot-height]").evaluateAll((els) =>
+ els.map((g) => [g.getAttribute("data-plot-height"), getComputedStyle(g).display !== "none"]),
+ );
+ expect(sets.filter(([, on]) => on).map(([h]) => h)).toEqual([shown]);
+ });
+}
+
+// THE DATA IS WHAT IS PAINTED. Read back from a screenshot of the plot: at the
+// busiest month the stack's topmost painted row is within 1 px of where the
+// month's true total sits on the value scale, and every band with data — each
+// kept site's and Other's — shows pixels of its own colour: the gaps take no
+// band away.
+for (const width of [390, 1280]) {
+ test(`${width} px: the chart paints its peak at its true height and every band`, async ({ page }) => {
+ await page.setViewportSize({ width, height: 900 });
+ await page.goto("/");
+ const summary = buildFixtureSummary();
+ const months = summary.monthly ?? [];
+ const sites = summary.sites;
+ const layers = growthLayers(months, sites);
+ const totals = months.map((m) => sites.reduce((a, x) => a + (m.bySite[x.siteId] ?? 0), 0));
+ const peak = Math.max(...totals);
+ const peakAt = totals.indexOf(peak);
+ const plot = page.locator('figure [role="img"]');
+ const box = (await plot.boundingBox())!;
+ // The value scale, from the labels on its gridlines (a label's `top` is
+ // its gridline's place, 1 − t / yMax of the plot's height).
+ const scale = await plot.evaluate((el) => {
+ const s = el.querySelector("span.tabular") as HTMLElement;
+ return { t: Number(s.textContent!.replace(/[^\d.]/g, "")), top: parseFloat(s.style.top) / 100 };
+ });
+ const yMax = scale.t / (1 - scale.top);
+ const expected = (1 - peak / yMax) * box.height;
+ const x = (peakAt / (months.length - 1)) * box.width;
+ const ground = rgbOf(await page.evaluate(() => getComputedStyle(document.body).backgroundColor));
+ // The legend's swatches, one per band, bottom-up (kept sites, then Other).
+ const swatches = await page
+ .locator("figure ul li span")
+ .evaluateAll((els) => els.map((e) => getComputedStyle(e).backgroundColor));
+ expect(swatches).toHaveLength(layers.length);
+ const withData = layers.map((l) =>
+ months.some((m) => l.sites.some((si) => (m.bySite[sites[si].siteId] ?? 0) > 0)),
+ );
+ const shot = await painted(page, plot, {
+ columns: [x - 1, x, x + 1],
+ ground,
+ colours: swatches.map(rgbOf),
+ tol: 12,
+ run: 2,
+ });
+ const top = Math.min(...shot.tops.filter((t): t is number => t !== null));
+ expect(Math.abs(top - expected), `top ${top} vs ${expected.toFixed(1)}`).toBeLessThanOrEqual(1);
+ layers.forEach((l, k) => {
+ if (withData[k]) expect(shot.counts[k].columns, `${l.key}'s colour`).toBeGreaterThan(0);
+ });
+ });
+}
+
+const resolveFill = (page: Page, css: string) =>
+ page.evaluate((c) => {
+ const el = document.createElement("span");
+ el.style.backgroundColor = c;
+ document.body.append(el);
+ const out = getComputedStyle(el).backgroundColor;
+ el.remove();
+ return out;
+ }, css);
+
+// WCAG contrast of two "rgb(…)" colours.
+function contrast(a: string, b: string): number {
+ const lum = (css: string) => {
+ const [r, g, bl] = rgbOf(css).map((v) => {
+ const c = v / 255;
+ return c <= 0.04045 ? c / 12.92 : ((c + 0.055) / 1.055) ** 2.4;
+ });
+ return 0.2126 * r + 0.7152 * g + 0.0722 * bl;
+ };
+ const [hi, lo] = [lum(a), lum(b)].sort((p, q) => q - p);
+ return (hi + 0.05) / (lo + 0.05);
+}
+
+// THE FOLD (lib/growthGaps.ts). The fixture's last two sites are each under
+// 5 % of the chart's total: ONE Other band, drawn last (on top), in the chart's
+// neutral grey. The legend shows the kept sites and Other; the table and
+// every month's title name every site.
+test("two sites under 5 % are one Other band on top: the legend shows the kept sites and Other, the table and the titles name all six", async ({
+ page,
+}) => {
+ const summary = buildFixtureSummary();
+ const months = summary.monthly ?? [];
+ const sites = summary.sites;
+ const layers = growthLayers(months, sites);
+ expect(layers.map((l) => l.key)).toEqual([
+ "fixture-one",
+ "fixture-two",
+ "fixture-three",
+ "fixture-four",
+ "(other)",
+ ]);
+ expect(layers.at(-1)!.sites).toEqual([4, 5]);
+ const titles = sites.map((s) => s.siteTitle);
+ await page.goto("/");
+ const figure = page.locator("figure").filter({ has: page.locator('[role="img"]') });
+
+ // The legend: the four kept sites, then Other.
+ await expect(figure.locator("ul[aria-hidden='true'] li")).toHaveText([...titles.slice(0, 4), "Other"]);
+
+ // The bands, in paint order: Other's is the last, in the neutral grey.
+ const fills = await figure
+ .locator("svg > path:not(.growth-gap)")
+ .evaluateAll((els) => els.map((e) => e.getAttribute("fill")));
+ expect(fills).toEqual([...siteChartColors(sites).slice(0, 4), OTHER_COLOR]);
+
+ // The image's label names the folded sites; the caption says what Other is.
+ await expect(figure.locator('[role="img"]')).toHaveAttribute(
+ "aria-label",
+ /Fixture Five and Fixture Six, each under 5% of the total, are drawn together as Other\.$/,
+ );
+ await expect(figure.locator("figcaption")).toContainText(
+ "Instances under 5% of the total are drawn together as Other.",
+ );
+
+ // Every month's title names every site with data that month, folded or not.
+ const hits = await figure
+ .locator("svg rect.growth-hit title")
+ .evaluateAll((els) => els.map((e) => e.textContent ?? ""));
+ expect(hits).toHaveLength(months.length);
+ let foldedNamed = 0;
+ months.forEach((m, i) => {
+ for (const s of sites) {
+ const v = m.bySite[s.siteId] ?? 0;
+ if (v > 0) expect(hits[i], `${m.month}`).toContain(`\n${s.siteTitle} ${v.toLocaleString("en-US")}`);
+ }
+ if (hits[i].includes("\nFixture Five ") || hits[i].includes("\nFixture Six ")) foldedNamed++;
+ });
+ expect(foldedNamed).toBeGreaterThan(0);
+ expect(hits.join("\n")).not.toContain("\nOther");
+
+ // The table: a column per site, all six, then Total.
+ await figure.locator("summary", { hasText: "Numbers by year" }).click();
+ await expect(figure.locator("table thead th")).toHaveText(["Year", ...titles, "Total"]);
+});
+
+test("Other is its own grey (--chart-other), at least 3:1 on both grounds, and no kept site's colour", async ({
+ page,
+}) => {
+ await page.goto("/");
+ for (const base of ["light", "dark"] as const) {
+ await page.evaluate(([k, v]) => localStorage.setItem(k, v), [BASE_KEY, base]);
+ await page.reload();
+ await expect(page.locator("html")).toHaveAttribute("data-base", base);
+ const swatches = await page
+ .locator("figure ul[aria-hidden='true'] li > span")
+ .evaluateAll((els) => els.map((e) => getComputedStyle(e).backgroundColor));
+ const other = swatches.at(-1)!;
+ expect(other, base).toBe(await resolveFill(page, OTHER_COLOR));
+ // Its own token, set on each base (a Dark block without it would fall
+ // back to the Light value from :root).
+ expect(other, base).toBe(base === "light" ? "rgb(62, 84, 92)" : "rgb(98, 98, 92)");
+ expect(other, base).not.toBe(await resolveFill(page, "var(--chart-axis)"));
+ const ground = await page.evaluate(() => getComputedStyle(document.body).backgroundColor);
+ expect(contrast(other, ground), `${base}: Other on the ground`).toBeGreaterThanOrEqual(3);
+ expect(swatches.slice(0, -1), base).not.toContain(other);
+ }
+});
diff --git a/homepage/e2e/helpers.ts b/homepage/e2e/helpers.ts
@@ -0,0 +1,41 @@
+import { expect, type Page } from "@playwright/test";
+import { THEME_BASES, nextBase, type ThemeBase } from "../../common/components/themeConfig";
+
+// THE ONE WAY a homepage spec changes the theme through the UI: the header's
+// toggle (common/components/ThemeToggle.tsx, `variant="bare"`), which cycles
+// the base in nextBase's order, shows the base in force as `data-theme-base`,
+// and is named "Switch to {next}". Specs that only need a ground set
+// `localStorage` instead. No app has an accent control (each shows its own),
+// so there is no accent to choose here.
+
+export const themeToggle = (page: Page) =>
+ page.locator("header").getByRole("button", { name: /^switch to /i });
+
+// The cycle as the toggle walks it, from `start`: every base once. Derived
+// from THEME_BASES and nextBase, so a base added or removed there needs no
+// change here.
+export function themeCycle(start: ThemeBase): ThemeBase[] {
+ const out: ThemeBase[] = [start];
+ for (let b = nextBase(start); b !== start && out.length <= THEME_BASES.length; b = nextBase(b)) {
+ out.push(b);
+ }
+ return out;
+}
+
+export const baseLabel = (b: ThemeBase) =>
+ THEME_BASES.find((x) => x.id === b)!.label.toLowerCase();
+
+// Click the toggle until the base in force is `base`. A click before
+// hydration is lost, so each step waits for the attribute to move.
+export async function chooseTheme(page: Page, { base }: { base: ThemeBase }) {
+ const toggle = themeToggle(page);
+ for (let i = 0; i <= THEME_BASES.length; i++) {
+ const current = (await toggle.getAttribute("data-theme-base")) as ThemeBase;
+ if (current === base) return;
+ await expect(async () => {
+ if ((await toggle.getAttribute("data-theme-base")) === current) await toggle.click();
+ expect(await toggle.getAttribute("data-theme-base")).not.toBe(current);
+ }).toPass({ timeout: 10_000 });
+ }
+ expect(await toggle.getAttribute("data-theme-base")).toBe(base);
+}
diff --git a/homepage/e2e/instance-colours.spec.ts b/homepage/e2e/instance-colours.spec.ts
@@ -4,6 +4,7 @@ import { test, expect, type Page } from "@playwright/test";
import { resolveAccent } from "../../common/lib/accent";
import { ACCENTS } from "../../common/lib/brand";
import { ACCENT_CHART_SLOT, siteChartColors } from "../../common/lib/siteColor";
+import { OTHER_COLOR } from "../app/lib/growthGaps";
import { FIXTURE_PALE_HEX, FIXTURE_SITES, FIXTURE_SUMMARY_NAME } from "./fixture-summary";
// The Official Instances cards wear their site's accent (release 10): a named
@@ -16,7 +17,9 @@ import { FIXTURE_PALE_HEX, FIXTURE_SITES, FIXTURE_SUMMARY_NAME } from "./fixture
// The dev server reads the synthetic summary (fixture-summary.ts): site 0
// Brass, site 1 a pale custom hex, 2–4 none, 5 Vermilion — so site 3's own
// slot (chart-4) is Brass's and it takes the lowest free one, and six sites
-// wear all six validated slots.
+// wear all six validated slots. Sites 4 and 5 are each under 5 % of the
+// chart's total, so the chart draws them as one Other band (release 14 slice
+// CF): the legend has the four kept sites, in their own slots, and Other.
const BASE_KEY = "ytdlp-tb:base";
@@ -67,7 +70,7 @@ async function colours(page: Page) {
return { stripes, legend };
}
-async function useBase(page: Page, base: "dark" | "light" | "sepia") {
+async function useBase(page: Page, base: "dark" | "light") {
await page.evaluate(([k, v]) => localStorage.setItem(k, v), [BASE_KEY, base]);
await page.reload();
await expect(page.locator("html")).toHaveAttribute("data-base", base);
@@ -89,23 +92,28 @@ test("each instance card wears its site's accent on every base, in its chart lay
expect(chart[0]).toBe(`var(--chart-${ACCENT_CHART_SLOT.brass! + 1})`);
expect(chart[5]).toBe("var(--chart-6)");
expect([...chart].sort()).toEqual([1, 2, 3, 4, 5, 6].map((k) => `var(--chart-${k})`));
+ // The chart's bands: the four kept sites, then Other (sites 4 and 5).
+ const kept = 4;
await page.goto("/");
- const on = { dark: "onDark", light: "onLight", sepia: "onSepia" } as const;
+ const on = { dark: "onDark", light: "onLight" } as const;
const pale = resolveAccent(FIXTURE_PALE_HEX);
- for (const base of ["dark", "light", "sepia"] as const) {
+ for (const base of ["dark", "light"] as const) {
await useBase(page, base);
const { stripes, legend } = await colours(page);
expect(stripes).toHaveLength(n);
// The summary's monthly series draws the chart, legend and all: the
- // legend (and so each layer) wears each site's chart colour, six apart.
+ // legend (and so each layer) wears each kept site's chart colour — its
+ // slot among all six, unchanged by the fold — then Other's grey, five
+ // apart.
await expect(page.getByRole("img", { name: /transcripts by the month/i })).toHaveCount(1);
- expect(legend, "legend swatches").toHaveLength(n);
- for (let i = 0; i < n; i++) {
+ expect(legend, "legend swatches").toHaveLength(kept + 1);
+ for (let i = 0; i < kept; i++) {
expect(legend[i], `legend ${i}`).toBe(await resolve(page, chart[i]));
}
- expect(new Set(legend).size, "six layers, six colours").toBe(n);
+ expect(legend[kept], "Other").toBe(await resolve(page, OTHER_COLOR));
+ expect(new Set(legend).size, "five layers, five colours").toBe(kept + 1);
// Site 0: the NAMED accent at this base's value — not the published
// on-dark hex on every base — and its legend swatch is the amber slot,
@@ -114,7 +122,7 @@ test("each instance card wears its site's accent on every base, in its chart lay
expect(await resolve(page, chart[0])).toBe(await resolve(page, "var(--chart-4)"));
expect(hueGap(stripes[0], legend[0]), "brass vs its layer").toBeLessThan(25);
- // Site 1: the pale custom hex, FITTED to this base — on light and sepia
+ // Site 1: the pale custom hex, FITTED to this base — on light
// darkened to 4.5:1 (as published it is ~1.4:1 there), on dark as is.
expect(stripes[1], "the pale hex, fitted").toBe(await resolve(page, pale[base]));
if (base !== "dark") {
@@ -124,16 +132,42 @@ test("each instance card wears its site's accent on every base, in its chart lay
}
// Sites 2–4 have no accent: their chart colour, the same as their legend
- // swatch.
+ // swatch where they keep a band (2 and 3; 4 is in Other).
for (let i = 2; i <= 4; i++) {
expect(stripes[i], `site ${i}`).toBe(await resolve(page, chart[i]));
- expect(stripes[i], `site ${i} vs legend`).toBe(legend[i]);
+ if (i < kept) expect(stripes[i], `site ${i} vs legend`).toBe(legend[i]);
}
- // Site 5: Vermilion, the sixth slot's family — the rust layer, not the
- // golden angle's hue 328 beside the magenta slot.
+ // Site 5: Vermilion, the sixth slot's family — its chart colour is the
+ // rust (chart[5], asserted above), not the golden angle's hue 328 beside
+ // the magenta slot. It is in Other on the growth chart; the test below
+ // reads the rust off its line on /stats/.
expect(stripes[5]).toBe(await resolve(page, ACCENTS.vermilion[on[base]]));
- expect(legend[5]).toBe(await resolve(page, "var(--chart-6)"));
- expect(hueGap(stripes[5], legend[5]), "vermilion vs its layer").toBeLessThan(25);
+ expect(hueGap(stripes[5], await resolve(page, chart[5])), "vermilion vs its chart colour").toBeLessThan(25);
+ }
+});
+
+// The growth chart folds site 5 into Other, so the rust is checked where it is
+// still drawn: /stats/' site lines (homepageChartData, one line per site in the
+// summary's order, each in its siteChartColors colour).
+test("on /stats/ each site's line wears its chart colour: the Vermilion site's is the rust", async ({
+ page,
+}) => {
+ const sites = fixtureSites();
+ const chart = siteChartColors(sites);
+ await page.goto("/stats/");
+ for (const base of ["dark", "light"] as const) {
+ await useBase(page, base);
+ const lines = page.locator(".recharts-line-curve");
+ await expect(lines).toHaveCount(sites.length);
+ const strokes = await lines.evaluateAll((els) => els.map((el) => getComputedStyle(el).stroke));
+ const want = [];
+ for (const c of chart) want.push(await resolve(page, c));
+ expect(strokes, base).toEqual(want);
+ expect(strokes[5], `${base}: the Vermilion site's line`).toBe(await resolve(page, "var(--chart-6)"));
+ expect(
+ hueGap(strokes[5], await resolve(page, ACCENTS.vermilion[base === "dark" ? "onDark" : "onLight"])),
+ "vermilion vs its line",
+ ).toBeLessThan(25);
}
});
diff --git a/homepage/e2e/instance-wordmark.spec.ts b/homepage/e2e/instance-wordmark.spec.ts
@@ -0,0 +1,114 @@
+import { test, expect, type Page } from "@playwright/test";
+import { resolveAccent } from "../../common/lib/accent";
+import { FIXTURE_PALE_HEX, FIXTURE_SITES } from "./fixture-summary";
+
+// Each Official Instances card names its site with the site's WORDMARK when
+// the summary carries its lead: the lead heavy and TINTED in the site's own
+// accent (a named accent's swatch on the base in force, a custom hex fitted to
+// it), the rest light. A site with a lead and no accent keeps the foreground;
+// a site with no lead shows its title plain, as before. The link's accessible
+// name is the plain title throughout.
+//
+// The fixture (fixture-summary.ts): Fixture One (Brass, lead "Fixture"),
+// Fixture Two (a pale custom hex, lead "Fixture"), Fixture Three (no accent,
+// lead "Fix" — mid-word, so a split name would read "Fix ture Three"), Four and
+// Five (neither), Six (Vermilion, no lead).
+
+const BASE_KEY = "ytdlp-tb:base";
+
+const section = (page: Page) =>
+ page.locator("section").filter({
+ has: page.getByRole("heading", { level: 2, name: "Official Instances", exact: true }),
+ });
+
+const card = (page: Page, title: string) =>
+ section(page)
+ .locator("li")
+ .filter({ has: page.getByRole("link", { name: title, exact: true }) });
+
+// The colour a CSS colour resolves to on this page.
+const resolve = (page: Page, css: string) =>
+ page.evaluate((c) => {
+ const el = document.createElement("span");
+ el.style.color = c;
+ document.body.append(el);
+ const out = getComputedStyle(el).color;
+ el.remove();
+ return out;
+ }, css);
+
+function luminance(rgb: string): number {
+ const [r, g, b] = (rgb.match(/[\d.]+/g) ?? []).slice(0, 3).map((n) => {
+ const c = Number(n) / 255;
+ return c <= 0.04045 ? c / 12.92 : ((c + 0.055) / 1.055) ** 2.4;
+ });
+ return 0.2126 * r + 0.7152 * g + 0.0722 * b;
+}
+const contrast = (a: string, b: string) => {
+ const [hi, lo] = [luminance(a), luminance(b)].sort((x, y) => y - x);
+ return (hi + 0.05) / (lo + 0.05);
+};
+
+async function wordmarkOf(page: Page, title: string) {
+ return card(page, title).evaluate((li) => {
+ const lead = li.querySelector("[data-wordmark-lead]");
+ const suffix = li.querySelector("[data-wordmark-suffix]");
+ const cs = (el: Element | null) => (el ? getComputedStyle(el) : null);
+ return {
+ lead: lead?.textContent ?? null,
+ suffix: suffix?.textContent ?? null,
+ leadWeight: Number(cs(lead)?.fontWeight ?? 0),
+ suffixWeight: Number(cs(suffix)?.fontWeight ?? 0),
+ leadColor: cs(lead)?.color ?? null,
+ foreground: getComputedStyle(document.documentElement).getPropertyValue("--foreground").trim(),
+ cardBg: getComputedStyle(li).backgroundColor,
+ };
+ });
+}
+
+test("each card's name: the wordmark where there is a lead, tinted in the site's accent at ≥ 3:1 on every base, and the plain title as its name", async ({
+ page,
+}) => {
+ await page.goto("/");
+ for (const s of FIXTURE_SITES) {
+ await expect(section(page).getByRole("link", { name: s.siteTitle, exact: true })).toHaveCount(1);
+ }
+ const pale = resolveAccent(FIXTURE_PALE_HEX);
+ for (const base of ["light", "dark"] as const) {
+ await page.evaluate(([k, v]) => localStorage.setItem(k, v), [BASE_KEY, base]);
+ await page.reload();
+ await expect(page.locator("html")).toHaveAttribute("data-base", base);
+
+ const want: Record<string, { lead: string; colour: string }> = {
+ "Fixture One": { lead: "Fixture", colour: await resolve(page, "var(--swatch-brass)") },
+ "Fixture Two": { lead: "Fixture", colour: await resolve(page, pale[base]) },
+ "Fixture Three": { lead: "Fix", colour: await resolve(page, "var(--foreground)") },
+ };
+ for (const [title, { lead, colour }] of Object.entries(want)) {
+ const w = await wordmarkOf(page, title);
+ expect(w.lead, `${title} lead`).toBe(lead);
+ expect(w.suffix, `${title} suffix`).toBe(title.slice(lead.length));
+ expect(w.leadWeight, `${title}: the lead is heavier`).toBeGreaterThan(w.suffixWeight);
+ expect(w.leadColor, `${title} on ${base}`).toBe(colour);
+ expect(
+ contrast(w.leadColor!, w.cardBg),
+ `${title}'s lead against its card on ${base}`,
+ ).toBeGreaterThanOrEqual(3);
+ }
+ // The named accent and the custom hex are tinted, not the foreground.
+ const fg = await resolve(page, "var(--foreground)");
+ expect(want["Fixture One"].colour).not.toBe(fg);
+ expect(want["Fixture Two"].colour).not.toBe(fg);
+ // A lead that ends mid-word leaves the name whole: one accessible name,
+ // and the text as a reader sees it, with no space inserted.
+ const three = card(page, "Fixture Three").getByRole("link");
+ await expect(three).toHaveAccessibleName("Fixture Three");
+ expect((await three.innerText()).replace("↗", "").trim()).toBe("Fixture Three");
+
+ // No lead: the plain title, no wordmark — accent or not.
+ for (const title of ["Fixture Four", "Fixture Five", "Fixture Six"]) {
+ await expect(card(page, title).locator("[data-wordmark]"), title).toHaveCount(0);
+ await expect(card(page, title).getByRole("link")).toHaveText(new RegExp(`^${title}`));
+ }
+ }
+});
diff --git a/homepage/e2e/marketing.spec.ts b/homepage/e2e/marketing.spec.ts
@@ -2,6 +2,7 @@ import fs from "node:fs";
import path from "node:path";
import { test, expect } from "@playwright/test";
import { FIXTURE_SUMMARY_NAME } from "./fixture-summary";
+import { INSTANCES_URL, PROJECT_URL } from "../../common/lib/project";
// The home page's job is to say what Archilyzer is and offer the download.
// Nothing asserted here names a count, a site or a headline number. The numeric
@@ -37,11 +38,44 @@ test("the hub link, when there is one, leaves for the hub", async ({ page }) =>
await expect(hub).toHaveAttribute("href", /^https?:\/\/[^/]/);
});
+// The header carries four destinations; the footer's "Sections" carries the
+// same four and Changelog, which is in the footer only (app/lib/nav.ts).
+const HEADER_LINKS = ["Docs", "Source", "Downloads", "Stats"];
+const FOOTER_LINKS = [...HEADER_LINKS, "Changelog"];
+
+const linkNames = (nav: import("@playwright/test").Locator) =>
+ nav.getByRole("link").evaluateAll((els) => els.map((e) => e.textContent?.trim()));
+
+test("the header nav has four links and no Changelog; the footer's has five", async ({
+ page,
+}) => {
+ for (const width of [360, 1280]) {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ const main = page.getByRole("navigation", { name: width < 768 ? "Main, compact" : "Main", exact: true });
+ expect(await linkNames(main)).toEqual(HEADER_LINKS);
+ await expect(page.locator("header").getByRole("link", { name: "Changelog" })).toHaveCount(0);
+ expect(await linkNames(page.getByRole("navigation", { name: "Footer" }))).toEqual(FOOTER_LINKS);
+ }
+});
+
+test("Changelog is reachable from the footer on every page", async ({ page }) => {
+ for (const path of ["/", "/docs/", "/source/", "/downloads/", "/stats/", "/changelog/", "/no-such-page/"]) {
+ await page.goto(path);
+ await expect(
+ page.getByRole("navigation", { name: "Footer" }).getByRole("link", { name: "Changelog" }),
+ path,
+ ).toHaveAttribute("href", "/changelog/");
+ await expect(page.locator("header").getByRole("link", { name: "Changelog" }), path).toHaveCount(0);
+ }
+});
+
test("every nav destination resolves", async ({ page }) => {
// The nav must never point at a 404, including on a build with no corpus
// data — which is why /stats/ always exists and degrades in place.
for (const [label, path] of [
["Docs", "/docs/"],
+ ["Source", "/source/"],
["Downloads", "/downloads/"],
["Stats", "/stats/"],
["Changelog", "/changelog/"],
@@ -79,8 +113,10 @@ test("the growth chart renders from the summary, or not at all", async ({
return;
}
await expect(chart).toBeVisible();
+ // The caption; with sites folded into Other (lib/growthGaps.ts) a second
+ // sentence says what Other is.
await expect(
- page.getByText(/all official instances\.$/i),
+ page.getByText(/all official instances\.( |$)/i),
).toBeVisible();
});
@@ -115,6 +151,25 @@ test("official instances link out, and the numbers carry their build date", asyn
}
});
+// Every archive's header links here (common/lib/project.ts INSTANCES_URL):
+// `/#instances` brings the section's heading into view, clear of the sticky
+// header.
+test("/#instances brings Official Instances into view below the sticky header", async ({
+ page,
+}) => {
+ expect(INSTANCES_URL).toBe(`${PROJECT_URL}/#instances`);
+ await page.setViewportSize({ width: 390, height: 700 });
+ await page.goto("/#instances");
+ const heading = page.getByRole("heading", { level: 2, name: "Official Instances", exact: true });
+ await expect(heading).toBeInViewport();
+ const [h, bar] = await Promise.all([
+ heading.boundingBox(),
+ page.locator("header").first().boundingBox(),
+ ]);
+ expect(h!.y).toBeGreaterThanOrEqual(bar!.y + bar!.height);
+ expect(await page.evaluate(() => window.scrollY)).toBeGreaterThan(0);
+});
+
test("the headings after the H1 are in Title Case", async ({ page }) => {
await expect(
page.getByRole("heading", { level: 2, name: "What It Does", exact: true }),
diff --git a/homepage/e2e/social.spec.ts b/homepage/e2e/social.spec.ts
@@ -0,0 +1,620 @@
+import path from "node:path";
+import { test, expect, type Locator, type Page } from "@playwright/test";
+import {
+ FIXTURE_SETTINGS_NAME,
+ FIXTURE_SOCIAL_SIX,
+ FIXTURE_SOCIAL_SIX_FEATURED,
+ FIXTURE_HOSTILE_SVGS,
+ FIXTURE_SOCIAL_TRIO,
+ unmarked,
+ writeFixtureSettings,
+ writeRawFixtureSettings,
+} from "./fixture-social";
+import { themeToggle } from "./helpers";
+import { ADVERSARIAL, LOADS_ELSEWHERE } from "../../common/lib/socialSvg.vectors";
+import { headerSocialLinks } from "../../common/lib/socialLinks";
+
+// THE SOCIAL ROW: in the footer (every link) and in the header, which keeps the
+// NAME on a narrow screen: below 520 px only the links marked `featured` (none
+// marked → none), from 520 px every link up to four, the marked ones kept first. The links are the e2e's own (fixture-social.ts): Leaf (a
+// gradient and one themed colour), Bubble (one colour) and Disc (two colours,
+// pasted with only a size — no viewBox until the normalizer made one). No copy:
+// every link is an icon whose accessible name is its label.
+//
+// Specs that need another list rewrite the dev server's settings file and the
+// afterEach restores the trio; `next dev` reads it on every request.
+
+const settingsFile = () =>
+ path.resolve(test.info().project.testDir, FIXTURE_SETTINGS_NAME);
+
+test.afterEach(() => {
+ writeFixtureSettings(settingsFile(), FIXTURE_SOCIAL_TRIO);
+});
+
+const TRIO = FIXTURE_SOCIAL_TRIO.map((l) => l.label);
+
+// The header's row at the current width (a narrow and a wide copy are both
+// rendered; CSS shows one), followed by the theme toggle. Only as last resorts
+// does the wordmark's TEXT hide (the mark stays) or the row scroll inside its
+// box (Header.tsx).
+const headerRow = (page: Page) =>
+ page.locator('header [data-social-links="header"]:visible');
+const scrollBox = (page: Page) => page.locator("header [data-social-scroll]:visible");
+const footerRow = (page: Page) => page.locator('footer [data-social-links="footer"]');
+
+const labelsOf = (row: Locator) =>
+ row.getByRole("link").evaluateAll((els) => els.map((e) => e.getAttribute("aria-label")));
+
+async function noHorizontalOverflow(page: Page) {
+ const { scroll, client } = await page.evaluate(() => ({
+ scroll: document.documentElement.scrollWidth,
+ client: document.documentElement.clientWidth,
+ }));
+ expect(scroll, "the page scrolls sideways").toBeLessThanOrEqual(client);
+}
+
+async function tabTo(page: Page, target: Locator) {
+ for (let i = 0; i < 25; i++) {
+ await page.keyboard.press("Tab");
+ if (await target.evaluate((el) => el === document.activeElement)) return;
+ }
+ throw new Error("Tab never reached the link");
+}
+
+// The bar's content width (in px, default text) below which the wordmark's
+// text is hidden, by how many links the header shows and the pointer: the
+// wordmark link (152 px), a 12 px gap, and the keys and the toggle (Header.tsx
+// WORDMARK_FITS).
+const WORDMARK_NEEDS = {
+ mouse: { 0: 200, 1: 236, 3: 308, 4: 344 },
+ touch: { 0: 208, 1: 252, 3: 340, 4: 384 },
+} as const;
+
+const allMarked = <T extends object>(links: readonly T[]) => links.map((l) => ({ ...l, featured: true }));
+
+// The link sets by how many a NARROW header shows (its marked ones).
+const LINK_SETS = {
+ 0: unmarked(FIXTURE_SOCIAL_TRIO),
+ 1: FIXTURE_SOCIAL_TRIO, // Disc, the last, is marked
+ 3: allMarked(FIXTURE_SOCIAL_TRIO),
+ 4: allMarked(FIXTURE_SOCIAL_SIX), // the narrow header shows the last four
+} as const;
+
+async function narrowHeader(page: Page, width: number, count: 0 | 1 | 3 | 4, pointer: "mouse" | "touch") {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ const labels = LINK_SETS[count]
+ .filter((l) => (l as { featured?: boolean }).featured === true)
+ .map((l) => l.label)
+ .slice(-4);
+ expect(await labelsOf(headerRow(page)), `${width} px, ${count} marked`).toEqual(labels);
+ // The copy for the other width is out of the accessibility tree.
+ for (const other of LINK_SETS[count].map((l) => l.label).filter((l) => !labels.includes(l))) {
+ await expect(page.getByRole("banner").getByRole("link", { name: other, exact: true })).toHaveCount(0);
+ }
+
+ // Nothing is hidden but (maybe) the wordmark's text, and nothing scrolls.
+ const home = page.getByRole("link", { name: "Archilyzer home", exact: true });
+ const textShown = await home.locator("[data-wordmark]").isVisible();
+ expect(textShown, `${pointer} ${width} px, ${count}: wordmark text`).toBe(
+ width - 40 >= WORDMARK_NEEDS[pointer][count],
+ );
+ if (textShown) {
+ expect(await home.evaluate((el) => el.scrollWidth <= el.clientWidth + 1), "the wordmark is clipped").toBe(true);
+ expect((await home.boundingBox())!.height).toBeLessThanOrEqual(32);
+ }
+ await expect(home.locator("svg").first()).toBeVisible(); // the mark stays
+
+ const toggle = themeToggle(page);
+ await expect(toggle).toBeInViewport({ ratio: 1 });
+ if (count > 0) {
+ const box = scrollBox(page);
+ const scrolls = await box.evaluate((el) => el.scrollWidth > el.clientWidth + 1);
+ expect(scrolls, `${pointer} ${width} px, ${count}: the row scrolls`).toBe(false);
+ const b = (await box.boundingBox())!;
+ const wm = (await home.boundingBox())!;
+ expect(b.x + 4 - (wm.x + wm.width), "the gap after the wordmark").toBeGreaterThanOrEqual(11.5);
+ for (const key of await headerRow(page).getByRole("link").all()) {
+ const k = (await key.boundingBox())!;
+ expect(k.x).toBeGreaterThanOrEqual(b.x);
+ expect(k.x + k.width).toBeLessThanOrEqual(b.x + b.width);
+ }
+ expect((await toggle.boundingBox())!.x, "the toggle is after the box").toBeGreaterThanOrEqual(b.x + b.width - 4.5);
+ } else {
+ await expect(page.locator("header [data-social-scroll]:visible")).toHaveCount(0);
+ }
+ await noHorizontalOverflow(page);
+}
+
+for (const width of [320, 340, 360, 390]) {
+ test(`${width} px, a mouse: 0, 1, 3 and 4 marked links, only those shown, nothing scrolls, the wordmark's text where it fits`, async ({
+ page,
+ }) => {
+ for (const count of [0, 1, 3, 4] as const) {
+ writeFixtureSettings(settingsFile(), [...LINK_SETS[count]]);
+ await narrowHeader(page, width, count, "mouse");
+ }
+ });
+}
+
+test("768 and 1280 px: the full wordmark, the nav, the row and the toggle in one bar", async ({ page }) => {
+ for (const width of [768, 1280]) {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ const home = page.getByRole("link", { name: "Archilyzer home", exact: true });
+ await expect(home.locator("[data-wordmark]")).toBeVisible();
+ await expect(page.getByRole("navigation", { name: "Main", exact: true })).toBeVisible();
+ expect(await labelsOf(headerRow(page))).toEqual(TRIO);
+ expect(await scrollBox(page).evaluate((el) => el.scrollWidth > el.clientWidth + 1)).toBe(false);
+ await expect(themeToggle(page)).toBeInViewport();
+ await noHorizontalOverflow(page);
+ }
+});
+
+test("every link: its href, its label as its name and title, a new tab with no opener, and no text", async ({
+ page,
+}) => {
+ await page.goto("/");
+ for (const row of [headerRow(page), footerRow(page)]) {
+ for (const want of FIXTURE_SOCIAL_TRIO) {
+ const link = row.getByRole("link", { name: want.label, exact: true });
+ await expect(link).toHaveAttribute("href", want.url);
+ await expect(link).toHaveAttribute("title", want.label);
+ await expect(link).toHaveAttribute("target", "_blank");
+ await expect(link).toHaveAttribute("rel", "noopener noreferrer");
+ // No copy: the link's only content is its icon.
+ await expect(link).toHaveText("");
+ await expect(link.locator("svg")).toHaveAttribute("aria-hidden", "true");
+ }
+ }
+});
+
+test("header icons are the muted foreground; a two-colour icon keeps its colours", async ({
+ page,
+}) => {
+ await page.goto("/");
+ // The glyph itself: Bubble was drawn white and themed to currentColor, so
+ // its path paints the link's colour, the muted foreground.
+ const { glyph, muted } = await page.evaluate(() => {
+ const probe = document.createElement("span");
+ probe.style.color = "var(--muted-foreground)";
+ document.body.append(probe);
+ const muted = getComputedStyle(probe).color;
+ probe.remove();
+ const path = document.querySelector(
+ 'header [data-social-links="header"] a[aria-label="Bubble"] path',
+ )!;
+ return { glyph: getComputedStyle(path).fill, muted };
+ });
+ expect(glyph).toBe(muted);
+ const disc = headerRow(page).getByRole("link", { name: "Disc" });
+ expect(await disc.innerHTML()).toContain('fill="#f4c542"');
+});
+
+// End to end for the normalizer's size rule: the fixture's Disc is pasted with
+// only width="81" height="81"; the page must carry the viewBox the normalizer
+// made, and the render size instead of 81.
+test("an icon pasted with a size and no viewBox renders with the viewBox made from its size", async ({
+ page,
+}) => {
+ await page.goto("/");
+ for (const row of [headerRow(page), footerRow(page)]) {
+ const svg = row.getByRole("link", { name: "Disc" }).locator("svg");
+ await expect(svg).toHaveAttribute("viewBox", "0 0 81 81");
+ await expect(svg).toHaveAttribute("width", "20");
+ await expect(svg).toHaveAttribute("height", "20");
+ const box = (await svg.boundingBox())!;
+ expect(Math.round(box.width)).toBe(20);
+ }
+});
+
+// The same icon is inlined up to three times (the header's narrow and wide
+// rows, the footer). Every copy's ids are its own, and the visible header
+// copy paints its gradient from its own defs.
+test("each copy of an icon owns its ids; the visible copy's gradient resolves inside it", async ({
+ page,
+}) => {
+ await page.setViewportSize({ width: 1280, height: 800 });
+ await page.goto("/");
+ const dupes = await page.evaluate(() => {
+ const seen = new Map<string, number>();
+ for (const el of document.querySelectorAll("[data-social-links] [id]")) {
+ seen.set(el.id, (seen.get(el.id) ?? 0) + 1);
+ }
+ return [...seen].filter(([, n]) => n > 1).map(([id]) => id);
+ });
+ expect(dupes).toEqual([]);
+ const own = await headerRow(page)
+ .getByRole("link", { name: "Leaf" })
+ .evaluate((a) => {
+ const svg = a.querySelector("svg")!;
+ const ref = /url\(#([^)]+)\)/.exec(
+ svg.querySelector("path")!.getAttribute("fill") ?? "",
+ )?.[1];
+ const target = ref ? document.getElementById(ref) : null;
+ return { ref, inside: !!target && svg.contains(target) };
+ });
+ expect(own.ref).toBeTruthy();
+ expect(own.inside).toBe(true);
+});
+
+test.describe("under a coarse pointer", () => {
+ test.use({ hasTouch: true });
+
+ for (const width of [320, 340, 360, 390]) {
+ test(`${width} px, touch: 0, 1, 3 and 4 marked links, only those shown, nothing scrolls, the wordmark's text where it fits`, async ({
+ page,
+ }) => {
+ for (const count of [0, 1, 3, 4] as const) {
+ writeFixtureSettings(settingsFile(), [...LINK_SETS[count]]);
+ await narrowHeader(page, width, count, "touch");
+ }
+ });
+ }
+
+ // THE LAST RESORT: at 280 px, four 44 px keys, the toggle and the mark do not
+ // fit, so the row scrolls inside its box — its END in view first, the toggle
+ // outside it, the page never sideways, and each link scrolled fully into view
+ // (its focus ring too) when it takes focus.
+ test("280 px, four marked links: the row scrolls in its box, its end first; focus brings each link into view", async ({
+ page,
+ }) => {
+ writeFixtureSettings(settingsFile(), LINK_SETS[4]);
+ await page.setViewportSize({ width: 280, height: 800 });
+ await page.goto("/");
+ const box = scrollBox(page);
+ expect(await box.evaluate((el) => el.scrollWidth > el.clientWidth + 1)).toBe(true);
+ // Not tab stops of their own (Firefox makes an overflowing scroll box one):
+ // the row's box, and the compact nav's rule, which overflows here too.
+ await expect(box).toHaveAttribute("tabindex", "-1");
+ await expect(page.getByRole("navigation", { name: "Main, compact" })).toHaveAttribute("tabindex", "-1");
+ await noHorizontalOverflow(page);
+ await expect(themeToggle(page)).toBeInViewport({ ratio: 1 });
+ const inBox = (key: import("@playwright/test").Locator) =>
+ Promise.all([box.boundingBox(), key.boundingBox()]).then(([b, k]) =>
+ k!.x - 2 >= b!.x - 0.5 && k!.x + k!.width + 2 <= b!.x + b!.width + 0.5,
+ );
+ const keys = await headerRow(page).getByRole("link").all();
+ expect(await inBox(keys[keys.length - 1]), "the last link, on load").toBe(true);
+ expect(await inBox(keys[0]), "the first link, on load").toBe(false);
+ for (const key of keys) {
+ await tabTo(page, key);
+ await page.waitForTimeout(100);
+ expect(await inBox(key), (await key.getAttribute("aria-label")) ?? "a link").toBe(true);
+ }
+ });
+
+ test("320 px at a 200 % text size: the header does not scroll sideways; the last link's end and the toggle are in view", async ({
+ page,
+ }) => {
+ writeFixtureSettings(settingsFile(), LINK_SETS[4]);
+ await page.addInitScript(() => {
+ document.addEventListener("DOMContentLoaded", () => {
+ document.documentElement.style.fontSize = "200%";
+ });
+ });
+ await page.setViewportSize({ width: 320, height: 800 });
+ await page.goto("/");
+ await expect(page.locator("html")).toHaveCSS("font-size", "32px");
+ // The header only: at twice the text size the page's own content is not
+ // this slice's to reflow.
+ expect(await page.locator("header").evaluate((el) => el.scrollWidth <= el.clientWidth)).toBe(true);
+ await expect(themeToggle(page)).toBeInViewport({ ratio: 1 });
+ // The box may be narrower than one 88 px key here: the last link is the
+ // one at its end, in view as far as the box allows.
+ const [b, k] = await Promise.all([
+ scrollBox(page).boundingBox(),
+ headerRow(page).getByRole("link").last().boundingBox(),
+ ]);
+ expect(k!.x + k!.width).toBeLessThanOrEqual(b!.x + b!.width + 0.5);
+ expect(k!.x + k!.width).toBeGreaterThan(b!.x + b!.width / 2);
+ });
+
+ for (const width of [360, 1280]) {
+ test(`${width} px: every key is at least 44 px, and the header row still fits`, async ({
+ page,
+ }) => {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ expect(await page.evaluate(() => matchMedia("(pointer: coarse)").matches)).toBe(true);
+ for (const row of [headerRow(page), footerRow(page)]) {
+ for (const key of await row.getByRole("link").all()) {
+ const k = (await key.boundingBox())!;
+ expect(k.width).toBeGreaterThanOrEqual(44);
+ expect(k.height).toBeGreaterThanOrEqual(44);
+ }
+ }
+ const box = (await headerRow(page).boundingBox())!;
+ expect(box.x + box.width).toBeLessThanOrEqual(width);
+ await noHorizontalOverflow(page);
+ });
+ }
+});
+
+test("keyboard focus shows the ring; in forced colours the browser's own outline, and none at rest", async ({
+ page,
+}) => {
+ await page.goto("/");
+ const link = headerRow(page).getByRole("link").first();
+ await tabTo(page, link);
+ const normal = await link.evaluate((el) => {
+ const s = getComputedStyle(el);
+ return { outline: s.outlineStyle, shadow: s.boxShadow };
+ });
+ expect(normal.outline).toBe("none");
+ expect(normal.shadow).toMatch(/0px 0px 0px 2px/);
+
+ await page.emulateMedia({ forcedColors: "active" });
+ await page.goto("/");
+ const forced = headerRow(page).getByRole("link").first();
+ const rest = await forced.evaluate((el) => getComputedStyle(el).outlineStyle);
+ expect(rest, "a permanent outline in forced colours").toBe("none");
+ await tabTo(page, forced);
+ const focused = await forced.evaluate((el) => {
+ const s = getComputedStyle(el);
+ return { style: s.outlineStyle, width: parseFloat(s.outlineWidth) };
+ });
+ expect(focused.style).not.toBe("none");
+ expect(focused.width).toBeGreaterThan(0);
+});
+
+test("six links, the last marked: a narrow header shows it alone, a wide one the last four, the footer all six", async ({
+ page,
+}) => {
+ writeFixtureSettings(settingsFile(), FIXTURE_SOCIAL_SIX);
+ const six = FIXTURE_SOCIAL_SIX.map((l) => l.label);
+ for (const width of [320, 360, 390, 519, 520, 767, 768, 1280]) {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ expect(await labelsOf(headerRow(page)), `${width} px`).toEqual(width < 520 ? ["Disc"] : six.slice(-4));
+ expect(await labelsOf(footerRow(page))).toEqual(six);
+ await noHorizontalOverflow(page);
+ }
+});
+
+// The footer shows every link: with eight under touch at 320 px its row wraps
+// rather than widen the page.
+test.describe("under a coarse pointer, eight links", () => {
+ test.use({ hasTouch: true });
+ test("320 px: the footer's row wraps, and nothing scrolls the page", async ({ page }) => {
+ const extra = (label: string, d: string) => ({
+ label,
+ url: `https://${label.toLowerCase()}.example/fixture`,
+ svg: `<svg viewBox="0 0 24 24"><path d="${d}"/></svg>`,
+ });
+ writeFixtureSettings(settingsFile(), [
+ ...FIXTURE_SOCIAL_SIX,
+ extra("Bar", "M3 10h18v4H3z"),
+ extra("Cross", "M10 3h4v7h7v4h-7v7h-4v-7H3v-4h7z"),
+ ]);
+ await page.setViewportSize({ width: 320, height: 800 });
+ await page.goto("/");
+ await expect(footerRow(page).getByRole("link")).toHaveCount(8);
+ const boxes = await footerRow(page)
+ .getByRole("link")
+ .evaluateAll((els) => els.map((e) => e.getBoundingClientRect()).map((r) => ({ right: r.right, top: r.top })));
+ for (const b of boxes) expect(b.right).toBeLessThanOrEqual(320);
+ expect(new Set(boxes.map((b) => Math.round(b.top))).size, "one row of eight").toBeGreaterThan(1);
+ await noHorizontalOverflow(page);
+ });
+});
+
+// The toggle sits in the row's rhythm: its box directly after the last link's,
+// as the links' boxes sit after each other, so glyph to glyph is the same all
+// along; the nav keeps a larger gap before the group.
+for (const width of [1024, 1280]) {
+ test(`${width} px: the toggle is spaced like a fourth key, and the nav stands apart`, async ({
+ page,
+ }) => {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ const boxes: { x: number; y: number; width: number; height: number }[] = [];
+ for (const key of await headerRow(page).getByRole("link").all()) boxes.push((await key.boundingBox())!);
+ boxes.push((await themeToggle(page).boundingBox())!);
+ expect(boxes).toHaveLength(TRIO.length + 1);
+ const gaps = boxes.slice(1).map((b, i) => b.x - (boxes[i].x + boxes[i].width));
+ for (const g of gaps) {
+ expect(g, "boxes overlap").toBeGreaterThanOrEqual(-0.5);
+ expect(Math.abs(g - gaps[0]), `gaps ${gaps.join(", ")}`).toBeLessThanOrEqual(1);
+ }
+ const lastNav = page.getByRole("navigation", { name: "Main", exact: true }).getByRole("link").last();
+ const nav = (await lastNav.boundingBox())!;
+ expect(boxes[0].x - (nav.x + nav.width), "the nav's gap before the group").toBeGreaterThan(gaps[0] + 16);
+ });
+}
+
+test("six links, two marked: a narrow header shows those two; a wide one those two and the last two of the rest, in order", async ({
+ page,
+}) => {
+ writeFixtureSettings(settingsFile(), FIXTURE_SOCIAL_SIX_FEATURED);
+ await page.setViewportSize({ width: 390, height: 800 });
+ await page.goto("/");
+ expect(await labelsOf(headerRow(page))).toEqual(["Square", "Bubble"]);
+ await page.setViewportSize({ width: 1280, height: 800 });
+ await page.goto("/");
+ expect(await labelsOf(headerRow(page))).toEqual(["Square", "Triangle", "Bubble", "Disc"]);
+ expect(await labelsOf(footerRow(page))).toEqual(FIXTURE_SOCIAL_SIX.map((l) => l.label));
+});
+
+// THE RULING: on a narrow screen the header keeps the NAME and shows only the
+// marked link(s), in the accessibility tree too; focus goes mark → the marked
+// link → the toggle. The footer keeps every link at every width.
+for (const touch of [false, true]) {
+ test.describe(touch ? "the narrow header, touch" : "the narrow header, a mouse", () => {
+ test.use({ hasTouch: touch });
+ for (const width of [360, 390]) {
+ test(`${width} px: the full name, the marked link alone, then the toggle, in focus order`, async ({ page }) => {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ const home = page.getByRole("link", { name: "Archilyzer home", exact: true });
+ await expect(home.locator("[data-wordmark]")).toBeVisible();
+ const banner = page.getByRole("banner");
+ await expect(banner.getByRole("link", { name: "Disc", exact: true })).toHaveCount(1);
+ for (const other of ["Leaf", "Bubble"]) {
+ await expect(banner.getByRole("link", { name: other, exact: true })).toHaveCount(0);
+ }
+ expect(await labelsOf(footerRow(page))).toEqual(TRIO);
+ const order: string[] = [];
+ for (let i = 0; i < 3; i++) {
+ await page.keyboard.press("Tab");
+ order.push(
+ await page.evaluate(() => document.activeElement?.getAttribute("aria-label") ?? document.activeElement?.textContent ?? ""),
+ );
+ }
+ expect(order.slice(0, 2)).toEqual(["Archilyzer home", "Disc"]);
+ expect(order[2]).toMatch(/^Switch to /);
+ });
+ }
+ test("none marked: the narrow header has no social link, and the name shows", async ({ page }) => {
+ writeFixtureSettings(settingsFile(), unmarked(FIXTURE_SOCIAL_TRIO));
+ for (const width of [320, 360, 390]) {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ await expect(page.getByRole("link", { name: "Archilyzer home", exact: true }).locator("[data-wordmark]")).toBeVisible();
+ for (const l of TRIO) {
+ await expect(page.getByRole("banner").getByRole("link", { name: l, exact: true })).toHaveCount(0);
+ }
+ expect(await labelsOf(footerRow(page))).toEqual(TRIO);
+ }
+ });
+ });
+}
+
+test("nothing scrolls the page sideways from 280 to 1400 px, and the footer keeps every link", async ({ page }) => {
+ for (const width of [280, 300, 320, 360, 390, 430, 519, 520, 640, 767, 768, 1024, 1280, 1400]) {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ await noHorizontalOverflow(page);
+ expect(await labelsOf(footerRow(page)), `${width} px`).toEqual(TRIO);
+ }
+});
+
+// AT TWICE THE TEXT SIZE. The switch between the narrow and the wide row is
+// 32.5rem, so it moves with the reader's own text size (the browser's default
+// font size, which rem in a media query follows): at 32 px it is 1040 px, and
+// the bar is the default-size bar at half the width. A page that sets its own
+// root size moves the page's rem but not a media query's; there, from the
+// switch, the wordmark's fit is keyed by the WIDE row's count, so the text
+// drops before that row scrolls.
+async function browserTextSize(page: Page, px: number) {
+ const cdp = await page.context().newCDPSession(page);
+ await cdp.send("Page.enable");
+ await cdp.send("Page.setFontSizes", { fontSizes: { standard: px } });
+}
+
+const barState = (page: Page) =>
+ page.evaluate(() => {
+ const wm = document.querySelector("header [data-wordmark]") as HTMLElement;
+ const box = ([...document.querySelectorAll("header [data-social-scroll]")] as HTMLElement[]).find(
+ (e) => e.offsetParent !== null,
+ );
+ const header = document.querySelector("header") as HTMLElement;
+ return {
+ text: getComputedStyle(wm).display !== "none",
+ scrolls: box ? box.scrollWidth > box.clientWidth + 1 : false,
+ over: header.scrollWidth - header.clientWidth,
+ };
+ });
+
+for (const touch of [false, true]) {
+ test.describe(touch ? "at 200 % text, touch" : "at 200 % text, a mouse", () => {
+ test.use({ hasTouch: touch });
+
+ test("the browser's text size at 32 px: the narrow row until 1040 px, the wide one from there; the row never scrolls while the name shows", async ({
+ page,
+ }) => {
+ test.setTimeout(90_000);
+ await browserTextSize(page, 32);
+ for (const count of [1, 4] as const) {
+ writeFixtureSettings(settingsFile(), [...LINK_SETS[count]]);
+ const narrow = headerSocialLinks(LINK_SETS[count], "narrow").map((l) => l.label);
+ const wide = headerSocialLinks(LINK_SETS[count], "wide").map((l) => l.label);
+ for (const width of [600, 720, 767, 900, 1039, 1040, 1100, 1300]) {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ await expect(page.locator("html")).toHaveCSS("font-size", "32px");
+ const at = `${count} marked, ${width} px`;
+ expect(await labelsOf(headerRow(page)), at).toEqual(width < 1040 ? narrow : wide);
+ const s = await barState(page);
+ if (s.scrolls) expect(s.text, `${at}: the row scrolls while the name shows`).toBe(false);
+ expect(s.over, `${at}: the header overflows`).toBeLessThanOrEqual(0);
+ if (count === 1 && width >= 720) expect(s.text, `${at}: the name`).toBe(true);
+ }
+ }
+ });
+
+ test("the page's root at 200 %: from 520 to 767 px the wide row never scrolls while the name shows", async ({
+ page,
+ }) => {
+ test.setTimeout(90_000);
+ await page.addInitScript(() => {
+ document.addEventListener("DOMContentLoaded", () => {
+ document.documentElement.style.fontSize = "200%";
+ });
+ });
+ for (const count of [1, 4] as const) {
+ writeFixtureSettings(settingsFile(), [...LINK_SETS[count]]);
+ const wide = headerSocialLinks(LINK_SETS[count], "wide").map((l) => l.label);
+ for (const width of [520, 560, 600, 640, 680, 720, 767]) {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ await expect(page.locator("html")).toHaveCSS("font-size", "32px");
+ const at = `${count} marked, ${width} px`;
+ expect(await labelsOf(headerRow(page)), at).toEqual(wide);
+ const s = await barState(page);
+ if (s.scrolls) expect(s.text, `${at}: the row scrolls while the name shows`).toBe(false);
+ expect(s.over, `${at}: the header overflows`).toBeLessThanOrEqual(0);
+ }
+ }
+ });
+ });
+}
+
+test("no links: no row in the header, and no Elsewhere column in the footer", async ({
+ page,
+}) => {
+ writeFixtureSettings(settingsFile(), []);
+ for (const width of [360, 1280]) {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ await expect(page.locator("header [data-social-links]")).toHaveCount(0);
+ await expect(page.locator("footer [data-social-links]")).toHaveCount(0);
+ await expect(page.locator("footer").getByText("Elsewhere")).toHaveCount(0);
+ await expect(page.getByRole("link", { name: "Archilyzer home" })).toBeVisible();
+ }
+});
+
+// A settings.json edited by hand (never through a save) holding icons that
+// would run script: none is inlined, none runs, and each link still renders —
+// as its label — while the good icons beside them render as icons.
+test("a stored icon that would run script or load from elsewhere is not inlined: nothing runs, nothing is fetched, the link shows its label", async ({
+ page,
+}) => {
+ const hostile = [...FIXTURE_HOSTILE_SVGS, ...LOADS_ELSEWHERE.map((n) => ADVERSARIAL[n])].map((svg, i) => ({
+ label: `Hostile ${i + 1}`,
+ url: `https://hostile-${i + 1}.example/`,
+ svg,
+ }));
+ writeRawFixtureSettings(settingsFile(), [...FIXTURE_SOCIAL_TRIO.slice(0, 1), ...hostile]);
+ const elsewhere: string[] = [];
+ page.on("request", (r) => {
+ if (new URL(r.url()).origin !== new URL(page.url() || "http://localhost").origin && !r.url().startsWith("data:")) {
+ elsewhere.push(r.url());
+ }
+ });
+ await page.goto("/");
+ await page.waitForLoadState("load");
+ await page.waitForTimeout(300);
+ expect(await page.evaluate(() => (window as unknown as { __hpHostile?: unknown }).__hpHostile)).toBeUndefined();
+ const footer = footerRow(page);
+ for (const h of hostile) {
+ const link = footer.getByRole("link", { name: h.label, exact: true });
+ await expect(link).toHaveAttribute("href", h.url);
+ await expect(link).toHaveText(h.label);
+ await expect(link.locator("svg, img, image")).toHaveCount(0);
+ }
+ // The good icon beside them is still an icon.
+ await expect(footer.getByRole("link", { name: "Leaf", exact: true }).locator("svg")).toHaveCount(1);
+ expect(await page.evaluate(() => document.querySelectorAll("[data-social-links] img, [data-social-links] image").length)).toBe(0);
+ expect(elsewhere.filter((u) => !u.startsWith(new URL(page.url()).origin)), "requests to another origin").toEqual([]);
+});
diff --git a/homepage/e2e/source-history.spec.ts b/homepage/e2e/source-history.spec.ts
@@ -0,0 +1,58 @@
+import path from "node:path";
+import { test, expect } from "@playwright/test";
+import {
+ FIXTURE_HISTORY,
+ FIXTURE_SOURCE_NAME,
+ clearFixtureSource,
+ writeFixtureSource,
+} from "./fixture-source";
+
+// The /source/ page's History block (release 15 slice SG), in both states,
+// from a FIXTURE publish (e2e/fixture-source.ts): the dev server reads the
+// page's manifest from E2E_SOURCE_PUBLIC_DIR while it holds one
+// (playwright.config.ts, app/lib/source.ts). Each test writes it and removes
+// it, so every other spec reads public/. The history PAGES themselves — what
+// stagit rendered and the publish rewrote — are source.spec.ts's, over the
+// checkout's real publish.
+const FIXTURE = path.resolve(process.cwd(), "e2e", FIXTURE_SOURCE_NAME);
+
+test.describe.configure({ mode: "serial" });
+test.afterEach(() => clearFixtureSource(FIXTURE));
+
+test("with a history in the manifest: the History block names the commits and the head, and links the log, the refs and the feed", async ({ page }) => {
+ writeFixtureSource(FIXTURE, { history: true });
+ await page.goto("/source/");
+ await expect(page.getByTestId("source-mirror-head")).toHaveText(FIXTURE_HISTORY.head);
+ const block = page.getByTestId("source-history");
+ await expect(block).toBeVisible();
+ await expect(block).toContainText("Every commit of main with its diff, as static pages: 1,234 commits, the newest");
+ await expect(page.getByTestId("source-history-commits")).toHaveText("1,234");
+ await expect(page.getByTestId("source-history-total")).toHaveCount(0);
+ // The head, short, linking to its own commit page.
+ await expect(page.getByTestId("source-history-head")).toHaveText(FIXTURE_HISTORY.head.slice(0, 12));
+ await expect(page.getByTestId("source-history-head")).toHaveAttribute("href", `/source/git/commit/${FIXTURE_HISTORY.head}.html`);
+ await expect(page.getByTestId("source-history-log")).toHaveAttribute("href", "/source/git/log.html");
+ await expect(page.getByTestId("source-history-refs")).toHaveAttribute("href", "/source/git/refs.html");
+ await expect(page.getByTestId("source-history-atom")).toHaveAttribute("href", "/source/git/atom.xml");
+ await expect(page.getByTestId("source-history-log")).toHaveText("Log");
+ await expect(page.getByTestId("source-history-atom")).toHaveText("Atom feed");
+});
+
+test("past the cap: the History block says the latest N of M commits", async ({ page }) => {
+ writeFixtureSource(FIXTURE, { history: true, total: 12_345 });
+ await page.goto("/source/");
+ const block = page.getByTestId("source-history");
+ await expect(block).toContainText("The newest commits of main with their diffs, as static pages: the latest 1,234 of 12,345 commits, the newest");
+ await expect(page.getByTestId("source-history-commits")).toHaveText("1,234");
+ await expect(page.getByTestId("source-history-total")).toHaveText("12,345");
+ await expect(page.getByTestId("source-history-log")).toHaveAttribute("href", "/source/git/log.html");
+});
+
+test("without a history in the manifest: no History block, and the rest of the page as before", async ({ page }) => {
+ writeFixtureSource(FIXTURE, { history: false });
+ await page.goto("/source/");
+ await expect(page.getByTestId("source-clone")).toBeVisible();
+ await expect(page.getByTestId("source-tree-link")).toBeVisible();
+ await expect(page.getByTestId("source-history")).toHaveCount(0);
+ await expect(page.getByRole("link", { name: "Atom feed" })).toHaveCount(0);
+});
diff --git a/homepage/e2e/source.spec.ts b/homepage/e2e/source.spec.ts
@@ -0,0 +1,185 @@
+import { test, expect, type Page } from "@playwright/test";
+
+// The /source/ page and the files it points at (common/publish/source.ts).
+// Like downloads.spec.ts, correct in BOTH states: the mirror, the raw tree and
+// the tarball are gitignored build artefacts, so a fresh checkout has none and
+// the page must say so rather than offer a clone that fails. The release gate
+// runs `archilyzer source publish` in the checkout first, so the published
+// branch is the one exercised there.
+//
+// This suite runs against `next dev`, which serves public/ from disk but does
+// NOT serve a directory's index.html at `/<dir>/` (Pages does), and ignores
+// _headers — so indexes are requested by name and content types are left to
+// the preview deploy's checks (and to app/lib/headers.test.ts).
+//
+// E2E_EXPECT_SOURCE=1 (declared in playwright.config.ts) makes the empty state
+// a FAILURE of the three data tests, so a gate that published first cannot
+// pass with a publish that silently produced nothing, or a loader that always
+// says "empty".
+const EXPECT_SOURCE = process.env.E2E_EXPECT_SOURCE === "1";
+
+async function published(page: Page): Promise<boolean> {
+ await page.goto("/source/");
+ const has = (await page.getByTestId("source-clone").count()) > 0;
+ if (EXPECT_SOURCE) {
+ expect(has, "E2E_EXPECT_SOURCE=1, and /source/ shows its empty state: run `archilyzer source publish` first").toBe(true);
+ }
+ return has;
+}
+
+test("says what the source is, and that nothing here takes a push", async ({ page }) => {
+ await page.goto("/source/");
+ await expect(page.getByRole("heading", { level: 1 })).toContainText("Source");
+ await expect(page.getByText(/There is no GitHub, by choice\./)).toBeVisible();
+ await expect(page.getByText(/Nothing here takes a push or a pull request\./)).toBeVisible();
+ await expect(page.getByRole("heading", { name: "What is mirrored" })).toBeVisible();
+ await expect(page.getByRole("heading", { name: "License" })).toBeVisible();
+});
+
+test("with a manifest: the clone command, and one tarball the page, snapshot.json and manifest.json agree on — else the empty state", async ({ page }) => {
+ if (!(await published(page))) {
+ await expect(page.getByTestId("source-empty")).toContainText("No source published in this build.");
+ await expect(page.getByTestId("source-tree-link")).toHaveCount(0);
+ await expect(page.getByTestId("source-tarball-link")).toHaveCount(0);
+ return;
+ }
+ await expect(page.getByTestId("source-clone")).toHaveText(
+ "git clone https://archilyzer.pages.dev/source/archilyzer.git",
+ );
+ const manifest = await (await page.request.get("/source/manifest.json")).json();
+ const snapshot = await (await page.request.get("/downloads/snapshot.json")).json();
+ await expect(page.getByTestId("source-mirror-head")).toHaveText(manifest.mirrorHead);
+ const href = await page.getByTestId("source-tarball-link").getAttribute("href");
+ expect(href).toBe("/downloads/archilyzer-source.tar.gz");
+ const tarball = await page.request.get(href!);
+ expect(tarball.status()).toBe(200);
+ expect((await tarball.body()).length).toBe(manifest.tarball.bytes);
+ const onPage = (await page.getByTestId("source-tarball-sha").textContent())?.trim();
+ expect(onPage).toMatch(/^[0-9a-f]{64}$/);
+ expect(onPage).toBe(manifest.tarball.sha256);
+ expect(onPage).toBe(snapshot.sha256);
+ expect(snapshot.commit).toBe(manifest.mirrorHead);
+});
+
+test("the mirror is a dumb-HTTP git repository: HEAD, info/refs, objects/info/packs", async ({ page }) => {
+ if (!(await published(page))) {
+ await expect(page.getByTestId("source-empty")).toBeVisible();
+ return;
+ }
+ const manifest = await (await page.request.get("/source/manifest.json")).json();
+ const head = await page.request.get("/source/archilyzer.git/HEAD");
+ expect(head.status()).toBe(200);
+ expect(await head.text()).toBe("ref: refs/heads/main\n");
+ const refs = await page.request.get("/source/archilyzer.git/info/refs");
+ expect(await refs.text()).toContain(`${manifest.mirrorHead}\trefs/heads/main`);
+ const packs = await (await page.request.get("/source/archilyzer.git/objects/info/packs")).text();
+ const first = /^P (pack-[0-9a-f]+\.pack)$/m.exec(packs);
+ expect(first, packs).not.toBeNull();
+ const pack = await page.request.head(`/source/archilyzer.git/objects/pack/${first![1]}`);
+ expect(pack.status()).toBe(200);
+});
+
+test("the raw tree: an index per directory, encoded hrefs that resolve, brackets included", async ({ page }) => {
+ if (!(await published(page))) {
+ await expect(page.getByTestId("source-empty")).toBeVisible();
+ return;
+ }
+ const hrefs = (html: string) => [...html.matchAll(/<td class="name"><a href="([^"]+)">/g)].map((m) => m[1]);
+ const root = await page.request.get("/source/tree/index.html");
+ expect(root.status()).toBe(200);
+ const rootHrefs = hrefs(await root.text());
+ const dir = rootHrefs.find((h) => h === "common/");
+ expect(dir, rootHrefs.join(" ")).toBeTruthy();
+ expect((await page.request.get(`/source/tree/${dir}index.html`)).status()).toBe(200);
+ const file = rootHrefs.find((h) => h === "README.md");
+ expect(file).toBeTruthy();
+ const readme = await page.request.get(`/source/tree/${file}`);
+ expect(readme.status()).toBe(200);
+ expect(await readme.text()).toContain("Archilyzer");
+
+ // A variable font's name carries brackets: encoded in the href, and served.
+ const fonts = await page.request.get("/source/tree/umtool/report-to-video/fonts/index.html");
+ expect(fonts.status()).toBe(200);
+ const bracketed = hrefs(await fonts.text()).find((h) => h.includes("%5B"));
+ expect(bracketed).toBeTruthy();
+ const font = await page.request.head(`/source/tree/umtool/report-to-video/fonts/${bracketed}`);
+ expect(font.status()).toBe(200);
+});
+
+// The history pages (release 15 slice SG): stagit's rendering, rewritten by
+// the publish. Run when the checkout's publish has one (a machine with
+// stagit); without, the page must show no History block.
+test("the history pages: the log, a commit, the Files index into the raw tree; the site's line, its theme by stored choice and by the OS; the feed untouched", async ({ page, browser, baseURL }) => {
+ if (!(await published(page))) {
+ await expect(page.getByTestId("source-history")).toHaveCount(0);
+ return;
+ }
+ const manifest = await (await page.request.get("/source/manifest.json")).json();
+ if (!manifest.history) {
+ await expect(page.getByTestId("source-history")).toHaveCount(0);
+ return;
+ }
+ await expect(page.getByTestId("source-history-head")).toHaveText(manifest.history.head.slice(0, 12));
+ const headHref = await page.getByTestId("source-history-head").getAttribute("href");
+ expect(headHref).toBe(`/source/git/commit/${manifest.history.head}.html`);
+ expect((await page.request.get(headHref!)).status()).toBe(200);
+ const shown = (await page.getByTestId("source-history-commits").textContent())!.replace(/\D/g, "");
+ expect(shown).toBe(String(manifest.history.commits));
+ // The log lists exactly the commits with a page (all of them under the cap).
+ const listed = await (await page.request.get("/source/git/log.html")).text();
+ expect(new Set([...listed.matchAll(/<a href="commit\/([0-9a-f]{40})\.html">/g)].map((m) => m[1])).size).toBe(manifest.history.commits);
+
+ // The log, with the one line the publish adds, linking back here, and the
+ // homepage's base (dark) for a reader who has stored none.
+ await page.getByTestId("source-history-log").click();
+ await expect(page).toHaveURL(/\/source\/git\/log\.html$/);
+ const back = page.getByRole("link", { name: "Archilyzer · Source" });
+ await expect(back).toHaveAttribute("href", "/source/");
+ await expect(page.locator("html")).toHaveAttribute("data-base", "dark");
+ await expect(page.locator("html")).toHaveAttribute("data-theme-ready", "1");
+ const bg = () => page.evaluate(() => getComputedStyle(document.body).backgroundColor);
+ expect(await bg()).toBe("rgb(12, 10, 8)");
+ // The stored choice is the homepage's (one origin, one key).
+ await page.evaluate(() => localStorage.setItem("ytdlp-tb:base", "light"));
+ await page.reload();
+ await expect(page.locator("html")).toHaveAttribute("data-base", "light");
+ expect(await bg()).toBe("rgb(243, 246, 247)");
+
+ // The newest commit's page resolves from the log, and its diff's file links
+ // go to the raw tree (no per-file pages are published).
+ const first = page.locator("#log a[href^='commit/']").first();
+ await expect(first).toHaveAttribute("href", `commit/${manifest.history.head}.html`);
+ const commit = await page.request.get(`/source/git/commit/${manifest.history.head}.html`);
+ expect(commit.status()).toBe(200);
+ const commitHtml = await commit.text();
+ expect(commitHtml).toContain('<a href="/source/">Archilyzer · Source</a>');
+ expect(commitHtml).not.toMatch(/href="(?:\.\.\/)*file\//);
+ // Files is an index into the tree: its first link is a raw file, served.
+ await page.goto("/source/git/files.html");
+ const href = await page.locator("#files a").first().getAttribute("href");
+ expect(href).toMatch(/^\.\.\/tree\//);
+ const target = new URL(href!, `${baseURL}/source/git/files.html`).pathname;
+ expect((await page.request.get(target)).status()).toBe(200);
+ expect((await page.request.get("/source/git/refs.html")).status()).toBe(200);
+ // The feeds are not pages: nothing is injected into them.
+ const atom = await (await page.request.get("/source/git/atom.xml")).text();
+ expect(atom).toContain("<feed");
+ expect(atom).toContain(`https://archilyzer.pages.dev/source/git/commit/${manifest.history.head}.html`);
+ expect(atom).not.toContain("archilyzer-source");
+ expect(atom).not.toContain("<script");
+
+ // Without the script (no JS), the OS decides: light, and dark.
+ for (const [scheme, rgb] of [["light", "rgb(243, 246, 247)"], ["dark", "rgb(12, 10, 8)"]] as const) {
+ const ctx = await browser.newContext({ javaScriptEnabled: false, colorScheme: scheme });
+ const p = await ctx.newPage();
+ await p.goto(`${baseURL}/source/git/log.html`);
+ expect(await p.locator("html").getAttribute("data-base"), "the script did not run").toBeNull();
+ expect(await p.evaluate(() => getComputedStyle(document.body).backgroundColor), `no JS, a ${scheme} OS`).toBe(rgb);
+ await ctx.close();
+ }
+});
+
+test("the Downloads page points at the mirror for the history", async ({ page }) => {
+ await page.goto("/downloads/");
+ await expect(page.getByRole("link", { name: "read-only git mirror" })).toHaveAttribute("href", "/source/");
+});
diff --git a/homepage/e2e/svg-vectors.spec.ts b/homepage/e2e/svg-vectors.spec.ts
@@ -0,0 +1,42 @@
+import { test, expect } from "@playwright/test";
+import { ADVERSARIAL, REAL_SHAPES } from "../../common/lib/socialSvg.vectors";
+import { safeSocialSvg, scopeSvgIds, sizeSocialSvg } from "../../common/lib/socialLinks";
+
+// THE INVARIANT THE READ PATH DEPENDS ON, checked with Chromium's own HTML
+// parser: for every icon the checker accepts (the review's adversarial battery
+// and the shapes real icons take), rendered as a page renders it and parsed as
+// a whole document the way the static HTML carries it, the element after the
+// link stays OUTSIDE the icon, no script runs, and nothing is requested from
+// anywhere.
+
+const accepted = Object.entries({ ...ADVERSARIAL, ...REAL_SHAPES })
+ .map(([name, raw]) => {
+ const safe = safeSocialSvg(raw);
+ return safe ? { name, html: sizeSocialSvg(scopeSvgIds(safe, "sl_S_1_-0")) } : null;
+ })
+ .filter((x): x is { name: string; html: string } => x !== null);
+
+test("every accepted icon leaves the page after it outside the icon, runs nothing and fetches nothing", async ({
+ page,
+}) => {
+ expect(accepted.length).toBeGreaterThan(10);
+ const requests: string[] = [];
+ await page.route("**/*", (route) => {
+ requests.push(route.request().url());
+ return route.abort();
+ });
+ for (const { name, html } of accepted) {
+ await page.setContent(
+ `<!doctype html><html><body><header><a id="l" href="#x">${html}</a><b id="after">after</b></header><main id="main">m</main></body></html>`,
+ );
+ await page.waitForTimeout(50);
+ const r = await page.evaluate(() => ({
+ afterInSvg: !!document.getElementById("after")?.closest("svg"),
+ afterHolder: document.getElementById("after")?.parentElement?.tagName ?? null,
+ mainInSvg: !!document.getElementById("main")?.closest("svg"),
+ x: (window as unknown as { __x?: unknown }).__x ?? null,
+ }));
+ expect(r, name).toEqual({ afterInSvg: false, afterHolder: "HEADER", mainInSvg: false, x: null });
+ }
+ expect(requests, "requests").toEqual([]);
+});
diff --git a/homepage/e2e/theme.spec.ts b/homepage/e2e/theme.spec.ts
@@ -1,9 +1,10 @@
import { test, expect, type Page } from "@playwright/test";
-import { REQUIRED_TOKENS } from "../../common/components/themeConfig";
+import { REQUIRED_TOKENS, RETIRED_BASE } from "../../common/components/themeConfig";
import { ACCENTS, BASE_GROUNDS } from "../../common/lib/brand";
+import { chooseTheme } from "./helpers";
// The shared theme system (common/styles/tokens.css + ThemeScript +
-// ThemeProvider + ThemeToggle/ThemeMenu) as the project's own site uses it: it
+// ThemeProvider + the header's toggle) as the project's own site uses it: it
// opens on the DARK base in Signal, the family's accent. A reader's base
// persists across reloads and is applied before hydration (no flash of the
// wrong theme).
@@ -23,7 +24,7 @@ async function htmlState(page: Page) {
}, BASE_KEY);
}
-test("dark + Signal by default; the base toggle persists with no FOUC", async ({
+test("dark + Signal by default; a base chosen with the toggle persists with no FOUC", async ({
page,
}) => {
// The OS says light: the default is still dark (it is the site's, not the
@@ -39,16 +40,10 @@ test("dark + Signal by default; the base toggle persists with no FOUC", async ({
brand: ACCENTS.signal.onDark,
});
- const toggle = page.getByRole("button", { name: /switch to/i });
- await expect(toggle).toBeVisible();
-
- // Cycle to an explicit light base (dark → system → light). A click before
- // hydration is lost, so click until it is stored; a click React took commits
- // synchronously, so the check never races it into a second one.
- await expect(async () => {
- if ((await htmlState(page)).stored !== "light") await toggle.click();
- expect((await htmlState(page)).stored).toBe("light");
- }).toPass({ timeout: 10_000 });
+ // An explicit Light base, through the header's toggle (chooseTheme retries
+ // a click until hydration has made it live).
+ await chooseTheme(page, { base: "light" });
+ await expect.poll(async () => (await htmlState(page)).stored).toBe("light");
let s = await htmlState(page);
expect(s.base).toBe("light");
expect(s.dark).toBe(false);
@@ -104,7 +99,7 @@ test("every base declares a complete palette, and the grounds differ", async ({
await page.goto("/");
- const readOn = (base: "light" | "sepia" | "dark") =>
+ const readOn = (base: "light" | "dark") =>
page.evaluate(
({ base, tokens }) => {
const d = document.documentElement;
@@ -118,7 +113,7 @@ test("every base declares a complete palette, and the grounds differ", async ({
{ base, tokens: TOKENS },
);
- const bases = ["light", "sepia", "dark"] as const;
+ const bases = ["light", "dark"] as const;
const palettes = {} as Record<(typeof bases)[number], Record<string, string>>;
for (const b of bases) palettes[b] = await readOn(b);
@@ -128,9 +123,49 @@ test("every base declares a complete palette, and the grounds differ", async ({
}
expect(palettes[b]["--background"]).toBe(BASE_GROUNDS[b]);
}
- // …and the three must actually differ, or a base is a label on another's
+ // …and the two must actually differ, or a base is a label on the other's
// palette.
- expect(new Set(bases.map((b) => palettes[b]["--background"])).size).toBe(3);
- expect(new Set(bases.map((b) => palettes[b]["--foreground"])).size).toBe(3);
- expect(new Set(bases.map((b) => palettes[b]["--chart-1"])).size).toBe(3);
+ expect(new Set(bases.map((b) => palettes[b]["--background"])).size).toBe(2);
+ expect(new Set(bases.map((b) => palettes[b]["--foreground"])).size).toBe(2);
+ expect(new Set(bases.map((b) => palettes[b]["--chart-1"])).size).toBe(2);
+});
+
+// The retired third ground: a reader who chose it gets Light, before first
+// paint, with no other ground on the way, and the stored value becomes
+// "light" once. Every value `data-base` holds is recorded by a
+// MutationObserver installed before the page's first script.
+test("a stored retired base renders Light, with no other ground on the way, and is rewritten", async ({
+ page,
+}) => {
+ await page.addInitScript(
+ ([key, retired]) => {
+ try {
+ localStorage.setItem(key, retired);
+ } catch {}
+ const held: (string | null)[] = [];
+ (window as unknown as { __bases: (string | null)[] }).__bases = held;
+ new MutationObserver((records) => {
+ for (const r of records) if (r.target === document.documentElement) held.push(r.oldValue);
+ }).observe(document, {
+ subtree: true,
+ attributes: true,
+ attributeFilter: ["data-base"],
+ attributeOldValue: true,
+ });
+ },
+ [BASE_KEY, RETIRED_BASE],
+ );
+ await page.goto("/", { waitUntil: "commit" });
+ await page.waitForFunction(() => document.documentElement?.dataset.themeReady === "1");
+ expect(await page.evaluate(() => document.documentElement.getAttribute("data-base"))).toBe("light");
+ await page.waitForLoadState("load");
+ await page.waitForTimeout(300);
+ const held = await page.evaluate(() => [
+ ...(window as unknown as { __bases: (string | null)[] }).__bases,
+ document.documentElement.getAttribute("data-base"),
+ ]);
+ // The server's dark (the markup, before the pre-paint script), then light.
+ expect(held.filter((v) => v !== "dark" && v !== "light"), JSON.stringify(held)).toEqual([]);
+ expect(held.slice(1).every((v) => v === "light"), JSON.stringify(held)).toBe(true);
+ expect(await htmlState(page)).toMatchObject({ base: "light", dark: false, stored: "light" });
});
diff --git a/homepage/e2e/toggle.spec.ts b/homepage/e2e/toggle.spec.ts
@@ -0,0 +1,142 @@
+import { test, expect, type Page } from "@playwright/test";
+import { nextBase, type ThemeBase } from "../../common/components/themeConfig";
+import { baseLabel, themeCycle, themeToggle } from "./helpers";
+
+// THE HOMEPAGE'S THEME CONTROL is one toggle that cycles the base
+// (ThemeToggle, `variant="bare"`), the last key of the header's group; there is
+// no options dialog and no accent control: the accent is the homepage's own.
+// The expected cycle is derived from THEME_BASES / nextBase, so
+// removing a base there needs no change here.
+
+const htmlState = (page: Page) =>
+ page.evaluate(() => ({
+ base: document.documentElement.getAttribute("data-base"),
+ accent: document.documentElement.getAttribute("data-accent"),
+ ready: document.documentElement.getAttribute("data-theme-ready"),
+ }));
+
+for (const width of [320, 360, 390, 768, 1280]) {
+ test(`${width} px: the toggle is in the header, a 36 px key, and there is no other theme control`, async ({
+ page,
+ }) => {
+ await page.setViewportSize({ width, height: 800 });
+ await page.goto("/");
+ const toggle = themeToggle(page);
+ await expect(toggle).toBeInViewport({ ratio: 1 });
+ const box = (await toggle.boundingBox())!;
+ expect([box.width, box.height]).toEqual([36, 36]);
+ await expect(toggle).toHaveText("");
+ await expect(page.getByRole("button", { name: "Options" })).toHaveCount(0);
+ await expect(page.getByRole("button", { name: "Choose theme" })).toHaveCount(0);
+ await expect(page.getByRole("dialog")).toHaveCount(0);
+ });
+}
+
+test.describe("under a coarse pointer", () => {
+ test.use({ hasTouch: true });
+ test("the toggle is a 44 px key", async ({ page }) => {
+ await page.setViewportSize({ width: 360, height: 800 });
+ await page.goto("/");
+ const box = (await themeToggle(page).boundingBox())!;
+ expect([box.width, box.height]).toEqual([44, 44]);
+ });
+});
+
+test("each click moves to the next base: the page, the icon and the name follow, round the whole cycle", async ({
+ page,
+}) => {
+ // The OS says light, so "system" resolves to light.
+ await page.emulateMedia({ colorScheme: "light" });
+ await page.goto("/");
+ const toggle = themeToggle(page);
+ const start = (await toggle.getAttribute("data-theme-base")) as ThemeBase;
+ expect(start).toBe("dark"); // the homepage's own default
+ const cycle = themeCycle(start);
+ let current = start;
+ for (let i = 0; i < cycle.length; i++) {
+ const next = nextBase(current);
+ await expect(toggle).toHaveAccessibleName(`Switch to ${baseLabel(next)}`);
+ const icon = await toggle.locator("svg").getAttribute("class");
+ await expect(async () => {
+ if ((await toggle.getAttribute("data-theme-base")) === current) await toggle.click();
+ expect(await toggle.getAttribute("data-theme-base")).toBe(next);
+ }).toPass({ timeout: 10_000 });
+ expect(await toggle.locator("svg").getAttribute("class"), `${current} → ${next}: the icon`).not.toBe(icon);
+ const resolved = next === "system" ? "light" : next;
+ await expect.poll(async () => (await htmlState(page)).base).toBe(resolved);
+ current = next;
+ }
+ expect(current, "the cycle comes back to its start").toBe(start);
+});
+
+test("the toggle's focus: the ring, and in forced colours the browser's own outline, none at rest", async ({
+ page,
+}) => {
+ await page.goto("/");
+ const toggle = themeToggle(page);
+ const tabTo = async () => {
+ for (let i = 0; i < 25; i++) {
+ await page.keyboard.press("Tab");
+ if (await toggle.evaluate((el) => el === document.activeElement)) return;
+ }
+ throw new Error("Tab never reached the toggle");
+ };
+ await tabTo();
+ const normal = await toggle.evaluate((el) => {
+ const s = getComputedStyle(el);
+ return { outline: s.outlineStyle, shadow: s.boxShadow };
+ });
+ expect(normal.outline).toBe("none");
+ expect(normal.shadow).toMatch(/0px 0px 0px 2px/);
+
+ await page.emulateMedia({ forcedColors: "active" });
+ await page.goto("/");
+ expect(await toggle.evaluate((el) => getComputedStyle(el).outlineStyle)).toBe("none");
+ await tabTo();
+ const forced = await toggle.evaluate((el) => {
+ const s = getComputedStyle(el);
+ return { style: s.outlineStyle, width: parseFloat(s.outlineWidth) };
+ });
+ expect(forced.style).not.toBe("none");
+ expect(forced.width).toBeGreaterThan(0);
+});
+
+// With no accent control, a stored accent (set on this origin by an earlier
+// build that had one) must not tint the page: the pre-paint script
+// and the provider both ignore it, and it stays in storage untouched. Every
+// value `data-accent` ever holds is recorded by a MutationObserver installed
+// before the page's first script, so a flash between two samples cannot pass.
+test("a stored accent is ignored: the homepage keeps its own, with no flash, and the stored value stays", async ({
+ page,
+}) => {
+ await page.addInitScript(() => {
+ try {
+ localStorage.setItem("ytdlp-tb:accent", "violet");
+ } catch {}
+ const held: (string | null)[] = [];
+ (window as unknown as { __accents: (string | null)[] }).__accents = held;
+ new MutationObserver((records) => {
+ for (const r of records) {
+ if (r.target === document.documentElement) held.push(r.oldValue);
+ }
+ }).observe(document, {
+ subtree: true,
+ attributes: true,
+ attributeFilter: ["data-accent"],
+ attributeOldValue: true,
+ });
+ });
+ await page.goto("/", { waitUntil: "commit" });
+ await page.waitForFunction(() => document.documentElement?.dataset.themeReady === "1");
+ await page.waitForLoadState("load");
+ await expect.poll(async () => (await htmlState(page)).accent).toBe("signal");
+ await page.waitForTimeout(300);
+ const held = await page.evaluate(() => [
+ ...(window as unknown as { __accents: (string | null)[] }).__accents,
+ document.documentElement.getAttribute("data-accent"),
+ ]);
+ // Every value it held before each change, and the one it holds now.
+ expect(held.filter((v) => v !== null && v !== "signal"), JSON.stringify(held)).toEqual([]);
+ expect(held.at(-1)).toBe("signal");
+ expect(await page.evaluate(() => localStorage.getItem("ytdlp-tb:accent"))).toBe("violet");
+});
diff --git a/homepage/e2e/unlisted-site.spec.ts b/homepage/e2e/unlisted-site.spec.ts
@@ -0,0 +1,61 @@
+import fs from "node:fs";
+import path from "node:path";
+import { test, expect } from "@playwright/test";
+import {
+ FIXTURE_SITES,
+ FIXTURE_SUMMARY_NAME,
+ FIXTURE_UNLISTED_SITE,
+ buildFixtureInputs,
+ buildFixtureSummary,
+} from "./fixture-summary";
+
+// Release 14 slice HS: a site with site.json `listed: false` builds and
+// deploys, and the homepage lists it nowhere — not a card, not a chart series,
+// not a recent item — and counts the channels only it exposes in no total.
+//
+// The fixture summary (fixture-summary.ts) is built over six listed sites and
+// one unlisted one, which has two channels of its own and shares the first
+// site's first channel.
+
+const NEEDLES = [FIXTURE_UNLISTED_SITE.siteId, FIXTURE_UNLISTED_SITE.siteTitle];
+
+test("the summary the server reads is the one built without the unlisted site", () => {
+ // The site is in the builder's input: unlisted, with recordings of its own
+ // and a channel shared with the first listed site.
+ const { stats, channelSites, sites } = buildFixtureInputs();
+ const unlisted = sites.find((s) => s.siteId === FIXTURE_UNLISTED_SITE.siteId);
+ expect(unlisted?.listed).toBe(false);
+ const own = stats.filter((s) => s.channelSlug.startsWith(`${FIXTURE_UNLISTED_SITE.siteId}-ch`));
+ expect(own.length).toBeGreaterThan(0);
+ expect(channelSites[`${FIXTURE_SITES[0].siteId}-ch1`]).toContain(FIXTURE_UNLISTED_SITE.siteId);
+ // Listed, the same site would change the summary: a card, and its recordings.
+ const withoutIt = buildFixtureSummary(FIXTURE_SITES, []);
+ const asListed = buildFixtureSummary([...FIXTURE_SITES, FIXTURE_UNLISTED_SITE], []);
+ expect(asListed.sites.map((s) => s.siteId)).toContain(FIXTURE_UNLISTED_SITE.siteId);
+ expect(asListed.totals.transcripts).toBe(withoutIt.totals.transcripts + own.length);
+
+ const onDisk = JSON.parse(
+ fs.readFileSync(path.resolve(process.cwd(), "e2e", FIXTURE_SUMMARY_NAME), "utf8"),
+ );
+ // Every array and every total: what the six listed sites alone produce.
+ expect(onDisk).toEqual(JSON.parse(JSON.stringify(withoutIt)));
+ const text = JSON.stringify(onDisk);
+ for (const needle of NEEDLES) expect(text).not.toContain(needle);
+ expect(onDisk.sites.map((s: { siteId: string }) => s.siteId)).toEqual(
+ FIXTURE_SITES.map((s) => s.siteId),
+ );
+});
+
+for (const route of ["/", "/stats/"]) {
+ test(`${route} names no unlisted site, in its HTML or on the page`, async ({ page }) => {
+ const res = await page.goto(route);
+ expect(res?.ok()).toBe(true);
+ const served = (await res?.text()) ?? "";
+ // The listed sites are there, so the check below is not vacuous.
+ for (const s of FIXTURE_SITES) expect(served).toContain(s.siteId);
+ for (const needle of NEEDLES) {
+ expect(served).not.toContain(needle);
+ expect(await page.content()).not.toContain(needle);
+ }
+ });
+}
diff --git a/homepage/eslint.config.mjs b/homepage/eslint.config.mjs
@@ -12,6 +12,10 @@ const eslintConfig = defineConfig([
"out/**",
"build/**",
"next-env.d.ts",
+ // The published source (`archilyzer source publish`): the whole repo's
+ // raw tree, copied into public/ and from there into out/. Not this app's
+ // code — tsconfig.json excludes both for the same reason.
+ "public/source/**",
]),
]);
diff --git a/homepage/playwright.config.ts b/homepage/playwright.config.ts
@@ -1,3 +1,4 @@
+import fs from "node:fs";
import path from "node:path";
import { defineConfig, devices } from "@playwright/test";
import { portFor } from "yt-dlp-transcript-common/lib/ports.mjs";
@@ -5,6 +6,12 @@ import {
FIXTURE_SUMMARY_NAME,
writeFixtureSummary,
} from "./e2e/fixture-summary";
+import {
+ FIXTURE_SETTINGS_NAME,
+ FIXTURE_SOCIAL_TRIO,
+ writeFixtureSettings,
+} from "./e2e/fixture-social";
+import { FIXTURE_SOURCE_NAME, clearFixtureSource } from "./e2e/fixture-source";
// Homepage e2e. Runs against `next dev` (default mode) so it reflects uncommitted
// source. Kill any stale dev server on the port between runs.
@@ -20,6 +27,24 @@ import {
// summary it reads instead of
// public/homepage-summary.json (app/lib/summary.ts,
// ignored by a production build)
+// SETTINGS_FILE set below on the dev server: a settings.json
+// holding only the fixture's social links
+// (e2e/fixture-social.ts), never the checkout's
+// own — the header and footer render them
+// SITES_DIR set below on the dev server: an EMPTY
+// directory, so no homepage.json (whose
+// socialLinks would win over settings.json's)
+// is read from the checkout
+// E2E_EXPECT_SOURCE `1` (set by whoever runs the suite, after
+// `archilyzer source publish`): e2e/source.spec.ts
+// FAILS on the /source/ page's empty state instead
+// of accepting it
+// E2E_SOURCE_PUBLIC_DIR set below on the dev server: the directory the
+// /source/ page reads the publish from WHILE it
+// holds a manifest (app/lib/source.ts, ignored by
+// a production build). e2e/source-history.spec.ts
+// writes a fixture publish there and removes it;
+// empty, the page reads public/
const PORT = portFor("HOMEPAGE_E2E_PORT");
const baseURL = `http://localhost:${PORT}`;
@@ -31,6 +56,18 @@ const baseURL = `http://localhost:${PORT}`;
const FIXTURE_SUMMARY = path.resolve(process.cwd(), "e2e", FIXTURE_SUMMARY_NAME);
writeFixtureSummary(FIXTURE_SUMMARY);
+// The same for settings.json: the operator's social links move, and a worktree's
+// settings.json is a copy of theirs. The e2e reads the fixture's (the trio).
+const FIXTURE_SETTINGS = path.resolve(process.cwd(), "e2e", FIXTURE_SETTINGS_NAME);
+writeFixtureSettings(FIXTURE_SETTINGS, FIXTURE_SOCIAL_TRIO);
+const FIXTURE_SITES_DIR = path.resolve(process.cwd(), "e2e", ".e2e-sites");
+fs.mkdirSync(FIXTURE_SITES_DIR, { recursive: true });
+
+// The fixture publish (e2e/fixture-source.ts): none until a spec writes one,
+// so a killed run's leftover never stands in for public/.
+const FIXTURE_SOURCE_DIR = path.resolve(process.cwd(), "e2e", FIXTURE_SOURCE_NAME);
+clearFixtureSource(FIXTURE_SOURCE_DIR);
+
export default defineConfig({
testDir: "./e2e",
timeout: 30_000,
@@ -44,7 +81,12 @@ export default defineConfig({
url: baseURL,
timeout: 120_000,
reuseExistingServer: !process.env.CI,
- env: { E2E_HOMEPAGE_SUMMARY_FILE: FIXTURE_SUMMARY },
+ env: {
+ E2E_HOMEPAGE_SUMMARY_FILE: FIXTURE_SUMMARY,
+ SETTINGS_FILE: FIXTURE_SETTINGS,
+ SITES_DIR: FIXTURE_SITES_DIR,
+ E2E_SOURCE_PUBLIC_DIR: FIXTURE_SOURCE_DIR,
+ },
},
use: {
baseURL,
diff --git a/homepage/public/_headers b/homepage/public/_headers
@@ -6,3 +6,48 @@
# for hours while still absorbing a burst of downloads.
/downloads/*
Cache-Control: public, max-age=300, must-revalidate
+
+# The published source (`archilyzer source publish`, common/publish/source.ts).
+/source/manifest.json
+ Cache-Control: public, max-age=300, must-revalidate
+/source/archilyzer.git/*
+ Cache-Control: public, max-age=300, must-revalidate
+# The raw tree is PLAIN TEXT whatever the extension: wrangler's mime map would
+# send .ts as video/mp2t, .mjs as application/javascript, .md as text/markdown
+# and .html as a page on this origin.
+/source/tree/*
+ Content-Type: text/plain; charset=utf-8
+ X-Content-Type-Options: nosniff
+ X-Robots-Tag: noindex
+ Cache-Control: public, max-age=300, must-revalidate
+# ...except the generated directory indexes and the binary files. Every rule
+# that matches applies, in file order, and a header a LATER rule sets again is
+# APPENDED to the earlier value ("text/plain; …, text/html; …" — wrangler's
+# attachHeaders, the Pages asset server's code). So each override first
+# detaches the tree's Content-Type with `! Content-Type`, then sets its own.
+/source/tree/
+ ! Content-Type
+ Content-Type: text/html; charset=utf-8
+/source/tree/*/
+ ! Content-Type
+ Content-Type: text/html; charset=utf-8
+/source/tree/*.ttf
+ ! Content-Type
+ Content-Type: font/ttf
+/source/tree/*.m4a
+ ! Content-Type
+ Content-Type: audio/mp4
+/source/tree/*.zip
+ ! Content-Type
+ Content-Type: application/zip
+/source/tree/*.onnx
+ ! Content-Type
+ Content-Type: application/octet-stream
+/source/tree/*.svg
+ ! Content-Type
+ Content-Type: image/svg+xml
+ Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'
+# The history pages (stagit, sourceHistory.ts) are HTML by extension, and kept
+# out of search indexes like the raw tree (ruled, release 15 slice SG).
+/source/git/*
+ X-Robots-Tag: noindex
diff --git a/homepage/tsconfig.json b/homepage/tsconfig.json
@@ -16,5 +16,5 @@
".next/dev/types/**/*.ts",
"**/*.mts"
],
- "exclude": ["node_modules"]
+ "exclude": ["node_modules", "public", "out"]
}
diff --git a/mcp/README.md b/mcp/README.md
@@ -520,11 +520,11 @@ Logs go to stderr; stdout is the MCP JSON-RPC channel.
## Add to Claude Code
```sh
-claude mcp add rekietalyzer \
+claude mcp add archilyzer \
--env TRANSCRIPT_SITE_URL=https://rekietalyzer.pages.dev \
--env ARCHILYZER_EDITOR_URL=http://localhost:3001 \
--env WORKER_TOKEN=… \
- -- pnpm -C /ABS/PATH/TO/yt-dlp-transcript-browser --filter yt-dlp-transcript-mcp exec tsx src/index.ts
+ -- pnpm -C /ABS/PATH/TO/archilyzer --filter yt-dlp-transcript-mcp exec tsx src/index.ts
```
The two editor lines are optional: they let `fetch_clip` ask a local editor for
@@ -534,9 +534,10 @@ the same; `fetch_clip` then says it has no editor and fetches nothing.
**The `/sweep` and `/ask` commands.** `.claude/commands/{sweep,ask}.md` in this
repo call `mcp__archilyzer__sweep_plan` / `mcp__archilyzer__ask_plan` — the tool
-name embeds the MCP server name **as you registered it**, so if you used another
-name (`rekietalyzer` above), change the `mcp__<name>__` prefix in those two
-files to match. `foo:bar` namespacing is plugin-only, so what you type stays
+name embeds the MCP server name **as you registered it**, which is why every
+example registers it as `archilyzer`. If you use another name (a site's own,
+say `rekietalyzer`), change the `mcp__<name>__` prefix in those two files to
+match. `foo:bar` namespacing is plugin-only, so what you type stays
`/sweep`, not `/archilyzer:sweep`.
## Add to any MCP client (mcp.json)
@@ -544,10 +545,10 @@ files to match. `foo:bar` namespacing is plugin-only, so what you type stays
```json
{
"mcpServers": {
- "rekietalyzer": {
+ "archilyzer": {
"command": "pnpm",
"args": [
- "-C", "/ABS/PATH/TO/yt-dlp-transcript-browser",
+ "-C", "/ABS/PATH/TO/archilyzer",
"--filter", "yt-dlp-transcript-mcp",
"exec", "tsx", "src/index.ts"
],
diff --git a/mcp/src/search.test.ts b/mcp/src/search.test.ts
@@ -15,6 +15,7 @@ import type {
} from "yt-dlp-transcript-common/lib/posts";
import type { SearchAlias } from "yt-dlp-transcript-common/lib/searchAliases";
import type { Cue } from "yt-dlp-transcript-common/lib/vtt";
+import type { VideoStat } from "yt-dlp-transcript-common/lib/stats";
import type { ChannelGroup } from "yt-dlp-transcript-common/lib/channelGroups";
import { newGroup, newLeaf } from "yt-dlp-transcript-common/lib/searchQuery";
import type {
@@ -1296,6 +1297,81 @@ test("server: get_video_metadata reports a missing id as an error", async () =>
await client.close();
});
+// The "## Stats" block is read straight off the archive's stats pages, so its
+// "covers only N% — truncated" warning is exactly as good as the stat. Before
+// stats schema 6 the stat of a video transcribed after it was first indexed
+// stayed at "no transcript, coverage 0" for good, and this warned that a
+// complete transcript was truncated. buildStats.test.ts (a) pins the stat now
+// being recomputed; this pins what the tool then says about it.
+function statFor(
+ record: TranscriptDetail,
+ s: Pick<VideoStat, "hasTranscript" | "cueCount" | "coverage" | "transcribedDate">,
+): VideoStat {
+ return {
+ slug: record.slug,
+ id: record.id,
+ channelSlug: record.channelSlug,
+ channel: "Channel A",
+ title: record.title,
+ platform: "youtube",
+ uploadDate: record.uploadDate,
+ downloadedDate: "20260711",
+ timestamp: null,
+ duration: 3600,
+ viewCount: null,
+ likeCount: null,
+ commentCount: null,
+ channelFollowerCount: null,
+ categories: [],
+ tags: [],
+ language: null,
+ isLivestream: false,
+ mediaType: "video",
+ status: "available",
+ ...s,
+ };
+}
+
+class StatsStubSource extends StubSource {
+ constructor(private stat: VideoStat) {
+ super();
+ }
+ async statsIndex(): Promise<ReadonlyMap<string, VideoStat>> {
+ return new Map([[this.stat.slug, this.stat]]);
+ }
+}
+
+test("server: get_video_metadata on a recomputed stat reports the cues and no truncation", async () => {
+ const a1 = CHAN_A[0];
+ const client = await connectClient(
+ new StatsStubSource(
+ statFor(a1, { hasTranscript: true, cueCount: 7461, coverage: 0.998, transcribedDate: "20260918" }),
+ ),
+ );
+ const out = firstText(
+ await client.callTool({ name: "get_video_metadata", arguments: { video_id: "a1" } }),
+ );
+ assert.match(out, /## Stats/);
+ assert.match(out, /- transcript cues: 7461/);
+ assert.match(out, /- transcribed: /);
+ assert.doesNotMatch(out, /COVERS ONLY/);
+ await client.close();
+});
+
+test("server: get_video_metadata still warns for a transcript that really stops early", async () => {
+ const a1 = CHAN_A[0];
+ const client = await connectClient(
+ new StatsStubSource(
+ statFor(a1, { hasTranscript: true, cueCount: 900, coverage: 0.41, transcribedDate: "20260918" }),
+ ),
+ );
+ const out = firstText(
+ await client.callTool({ name: "get_video_metadata", arguments: { video_id: "a1" } }),
+ );
+ assert.match(out, /TRANSCRIPT COVERS ONLY 41% OF THE RUNTIME/);
+ await client.close();
+});
+
test("server: open_link decodes AND searches in one call, returning the handle", async () => {
const client = await connectRegistry();
const out = firstText(
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -153,6 +153,21 @@ THROWS at declaration — module load — on a name `SUB_FILE_RE` matches;
Never name the curated field `tags`. Never assume a `tags.json` is the keyword list:
`transcripts/tags.json` and `sites/<id>/tags.json` are curated tags.
+**`listed` / `unlisted` mean three unrelated things** (release 14, slice HS):
+
+| Which | Where | What it is |
+| --- | --- | --- |
+| A video's visibility | `common/lib/availability.ts` (the `"unlisted"` state, `isUnlisted`), `common/lib/transcripts{,-server}.ts`, `common/components/shareUrl.ts`, `common/controller/buildIndex.ts` | The platform's own "unlisted" (reachable by link, not listed on the channel). |
+| The hub's list has loaded | `export/app/components/hub/useHubSites.ts` — `listed` | `/hub-sites.json` has been answered and `/hub-summary.json` has settled. |
+| **A site the family lists** | `site.json` `listed` (`common/lib/siteSchema.ts` — `isListedSite`, `channelsOnlyOnUnlistedSites`) | Absent = listed; `false` keeps the site off the homepage, the hub and the other sites' footers, and out of the public totals. |
+
+A grep for either word finds all three; read the file before assuming which.
+
+**Never publish a path segment named `.git`.** wrangler's Pages upload drops `**/.git` (and
+`**/node_modules`) SILENTLY, and Cloudflare's managed rules block `/.git/` requests. The source
+mirror is `/source/archilyzer.git/` for that reason (release 12, below). A mirror named `.git` would
+deploy "successfully" and 404.
+
---
## Reusable helpers (do not rewrite these)
@@ -650,7 +665,10 @@ chunk they agree to within ~2%.
#### The chunk census
Computed free from `statsByPath.cueCount` — present on **all 73,367** transcribed
-videos, so it needed no GPU time and no transcript reads. At the shipped config
+videos, so it needed no GPU time and no transcript reads. (Corrected 2026-09-28: "all" was
+all the CACHE knew of. Until stats schema 6 a video transcribed after its stat was first
+cached kept `cueCount: null` — about 2,500 of them on 2026-09-28; see "The stats cache key"
+at the end of this file.) At the shipped config
(`maxCues` 600, overlap 40, step 560):
```
@@ -3194,6 +3212,53 @@ IS the path segment, the reconcile effect writes it to localStorage (so Dashboar
follow), `onChange` does `router.push("/sites/<other>/<segment>")` and "All sites" pushes
`/sites`.
+**Superseded by release 15 slice SS (2026-09-29): the active site is a cookie.**
+`seedsSiteParam`, the reconcile effect and every localStorage read on the render path are gone;
+`siteIdFromPathname` and the path rule stand.
+- **Store:** the cookie `archilyzer-active-site-<port>` (`activeSiteCookieName(host)` in
+ `app/lib/activeSite.ts`; base name `ACTIVE_SITE_COOKIE`, a host with no port uses it bare):
+ `path=/`, `SameSite=Lax`, one year, `httpOnly`. The port is in the name because a cookie is
+ shared by every port on a host and localStorage was per origin.
+- **One writer:** `setActiveSiteAction` (`app/lib/activeSiteActions.ts`, `"use server"`), which
+ only shape-checks (`isStorableActiveSite`: a `SITE_ID_RE`-shaped id of ≤ 128 chars, or
+ `__all__`). Setting the cookie makes Next re-render the current page and its layouts, which
+ is how Dashboard and Channels re-scope; nothing calls `router.refresh()`.
+- **One read:** `readActiveSite(param?, siteIds?)` (`app/lib/activeSiteServer.ts`,
+ `server-only`). The root layout calls it with no param, and Dashboard (`app/page.tsx`) and
+ Channels (`app/channels/page.tsx`) with `searchParams.site`. The precedence is
+ `resolveActiveSiteFrom(candidates, siteIds)`: the first candidate naming a configured site or
+ `__all__` wins, else `resolveActiveSite`'s default (the lone site, else all). The server passes
+ `[?site=, cookie]`; the picker passes `[its held choice, the path's site, ?site=, stored]`.
+- **The root layout now reads a cookie, so every page renders per request** (the build lists
+ every page `ƒ`; route handlers were `ƒ` already, and `/icon.svg` stays static). The one page
+ whose mode changed is `/_not-found`. The picker's Suspense stays for the build's
+ missing-Suspense check.
+- **`SiteScopeProvider`** (`app/components/SiteScopeProvider.tsx`, context + `useSiteScope()`)
+ holds `stored`, following the layout's value when it changes and not while a picker write is
+ in flight. `choose(value)` (the picker's changes) updates `stored` optimistically; the two
+ effects, visiting `/sites/<id>/…` and the one-time localStorage migration, only WRITE
+ (`record`), because a state change during hydration re-renders the picker before React
+ replays a pre-hydration change, and the replayed event then reads the reset value.
+- **Other tabs.** A client-side navigation does not re-render the root layout, so a tab's
+ `stored` would stay what its layout last read while another tab changed the shared cookie.
+ Every successful write (`choose` and `record`) is therefore posted on
+ `new BroadcastChannel(ACTIVE_SITE_CHANNEL)` (`"archilyzer-active-site"`, per origin and so per
+ port), and the other tabs call `router.refresh()`. A channel instance does not receive its own
+ posts, so a tab does not refresh for its own write. Without `BroadcastChannel` a tab keeps its
+ value until its next full load. Passive refresh (`AutoRefresh`) also re-reads the layout, but
+ only when the pulse token moves.
+- **A `?site=` link** governs its own request and is not stored. Choosing on such a page writes
+ the cookie, THEN `router.replace`s the URL without `site` (`withoutSiteParam`). On
+ `/sites/<id>/…` the picker writes the cookie, THEN pushes, so a page opened once the URL has
+ moved reads the new value. In both, the choice is held for the URL it was made on, so the
+ controlled select does not snap back while the navigation is in flight, and dropped on the first
+ render at another URL (else Back to that URL showed the old choice over the path).
+- **A new channel starts checked on the active site** (review L5): `ChannelFormClient` resolves
+ `resolveActiveSiteFrom([?site=, useSiteScope().stored], siteIds)` on its first render, and
+ `SiteMembershipsSection` takes it into its INITIAL state (there was an effect), so the box is
+ checked in the server's HTML and a later refresh cannot re-check a box the user cleared. With
+ one site configured and nothing stored, that site starts checked.
+
**`BUILD_KINDS` is wrong in both directions** and was moved verbatim, with a comment saying so:
six kinds (`build-index`, `build-stats`, `build-export`, `normalize-transcripts`,
`archive-transcripts`, `archive-combined-transcripts`) — MISSING the three live-chat kinds the
@@ -5703,7 +5768,7 @@ Every `file:line` below was grepped at `7dfd7508`, whose code is byte-identical
- Export pages (buildIndex, buildStats, compose-homepage): compact, no newline.
- Chart, alias and tag stores: indented, no newline.
- Everything else: indented, with a newline.
- The six local `function writeJsonAtomic` left in `buildIndex.ts:404`, `buildStats.ts:238`,
+ The six local `function writeJsonAtomic` left in `buildIndex.ts:404`, `buildStats.ts:376`,
`bin/compose-homepage.ts:40`, `aliasesStore.ts:23`, `chartsStore.ts:50` and
`curatedTagsStore.ts:61` are one-line wrappers that pin those bytes over the shared writer.
They are not copies.
@@ -5808,7 +5873,7 @@ Every `file:line` below was grepped at `7dfd7508`, whose code is byte-identical
- **`readChannelConfigFile(file)`** `:165-170`. It never throws, and answers null for absent,
unreadable, not JSON, or not a channel. `readChannelConfig(paths, slug)` `:172` wraps it,
- and `buildIndex.ts:296` and `buildStats.ts:158` call it directly. The header `:152-157` names
+ and `buildIndex.ts:296` and `buildStats.ts:285` call it directly. The header `:152-157` names
the raw readers that bypass it: `channelMedia.ts:179` (the `dataDir` guard) and the legacy
migrations (`migrateToSites.ts:101` for `group`, `bin/migrate-channel-priority.ts` for
`excludeFromSync`, which also WRITES raw at `:206`).
@@ -5892,7 +5957,7 @@ which is the same race class 4b fixed for `config.json`.
Not in the 14:
-- The export build's streamed page writers `buildIndex.ts:971` and `buildStats.ts:220`. These
+- The export build's streamed page writers `buildIndex.ts:971` and `buildStats.ts:358`. These
are JSON writers still on the per-pid name, not in the list above only because there is one
writer per build. They are owed with the rest (16 + 2).
- `metadataScanStore.ts:188-206` and `autoQueueState.ts:141-156`. They carry a module-level
@@ -6140,7 +6205,7 @@ complete with this release.
- **The two write counters are deleted.** `git grep -n 'writeSeq\|nextWriteSeq' -- common
editor` is empty.
- **What `git grep -n 'tmp-${process.pid}' -- common editor` still finds:**
- - `buildIndex.ts:971` and `buildStats.ts:220`, the export page writers, out of scope;
+ - `buildIndex.ts:971` and `buildStats.ts:358`, the export page writers, out of scope;
- `transcode.ts:31` (ffmpeg's output, renamed at `:55`);
- `transcribeOne.ts:142` (the transcription app's `outputBase`);
- the shared writer's own comment `:9`, code `:136`, and test `:68`.
@@ -6795,8 +6860,12 @@ S4, as shipped"; `release-10.md` "Slice L2 / L1, as shipped". Every anchor below
release 11), else `seriesColor(i)` when free (the first SIX are `var(--chart-1..6)`), else the
lowest free slot — never two sites in one colour, and any six wear the six validated slots. The growth
chart, its legend and `/stats` (`SiteGrid`, `homepageChartData`, the By-site leaderboard) wear
- it; the accents themselves fail the dataviz validator as a chart palette (no five with Brass
- and Blue pass). `useHubSites` lists the official
+ it (release 14 slice CF: the growth chart draws two or more sites each under 5 % of its total as
+ ONE Other band on top, in its own `--chart-other` (tokens.css, Light `#3e545c`, Dark `#62625c`;
+ not in `REQUIRED_TOKENS`); `homepage/app/lib/growthGaps.ts` `growthLayers` /
+ `layerColors`, which run `siteChartColors` over EVERY site, so a kept site keeps its slot); the
+ accents themselves fail the dataviz validator as a chart palette (no five with Brass and Blue
+ pass). `useHubSites` lists the official
instances only once `/hub-summary.json` has SETTLED (found, missing or unreadable;
`useHubSummary.ts:45`, `networkMode: "always"`), so they never reorder a moment later. The order
is applied in the browser: `hub-sites.json` on disk is unchanged.
@@ -7181,3 +7250,738 @@ out. O3's facts are the section just above ("O3 — runner lows"). Anchors are a
byte-identical after every run): the export suite replaces the `sw.js` link with its own file;
`e2e:2origin` (`compose hub`, twice) replaces `sw.js`, `hub-sites.json`, `corpus.json`, `llms.txt`,
`robots.txt` and `_headers`. `e2e:hub` writes none. Re-seed before the next export run.
+
+## Release 12 — the source mirror (verified 2026-09-28, `main` @ `ffdeb2cd`)
+
+Slices Q (`4855f70b`) and R (`ffdeb2cd`): [`release-12.md`](release-12.md), the plan
+[`source-mirror.md`](source-mirror.md), the operator-facing doc `PUBLISH.md` "The source mirror
+(homepage)". Anchors are at `ffdeb2cd`.
+
+### The step (`common/publish/source.ts`)
+
+- **`publishSource`** (`:611`) → `publish` (`:647`) runs these steps in order:
+ 1. The git COMMON dir's `refs/heads/main`: a worktree build mirrors the primary's main. No
+ repository at all (`commonDir` `:543`) says `NO_REPOSITORY` (`:510`).
+ 2. The operator files (`loadSourceRules` `:216`).
+ 3. The tools: `resolveFilterRepo` `:321` and `gitleaksIdentity` `:483`.
+ 4. The skip.
+ 5. A `--no-local --bare --single-branch --no-tags` clone into
+ `mkdtemp(ARCHILYZER_SOURCE_SCRATCH/archilyzer-source-)`. The scratch root is refused inside the
+ checkout or the public dir (`scratchRootProblem` `:582`).
+ 6. filter-repo, then `repack -a -d --max-pack-size=20m`, `prune-packed`, `pack-refs`,
+ `update-server-info`.
+ 7. The object audit.
+ 8. The tree and the tarball.
+ 9. The allowlisted stage.
+ 10. The file audit.
+ 11. The limits: 15,000 files, 24 MiB a file.
+ 12. The link-safe install: the manifest removed first and written LAST.
+
+ `buildHomepage` (`build.ts:991`) runs it between compose and `next build`, with `noRepository:
+ "empty"`. So `archilyzer build homepage`, `/sites` Build homepage and `pnpm ops build-homepage`
+ all do.
+- **A refusal withdraws the source.**
+ - Once the rules are loaded, any non-zero outcome runs `removePublishedSource` (`:520`). `--check`
+ writes nothing, this included.
+ - `buildHomepage` then removes `out/source` and the two download files (`withdrawBuiltSource`,
+ `build.ts:1026`). `out/source/index.html`, the `/source` PAGE, goes with them.
+- **The deploy key.** `homepage/.source-publish.json` sits beside `public/`, NEVER inside it (a
+ published rules hash would confirm a guess at the denylist). It holds `sourceCommit`,
+ `mirrorHead`, `rulesHash`, `filterRepo`, `gitleaks` and `contentDigest`:
+ - `rulesHash` is `rulesHashOf(scrub lines, literals, SOURCE_STEP_VERSION)` (`:243`).
+ **`SOURCE_STEP_VERSION` (`:86`, now 3) must be bumped whenever the scrub or the audit
+ changes**, so an unchanged main is re-published and an old `out/` refuses to deploy.
+ - `gitleaks` is the version line plus the sha256 of the binary. This machine's gitleaks prints
+ `version is set by build process` for every release.
+ - `contentDigest` is `sourceDigest` (`:447`): the sorted path, size and streamed sha256 of every
+ published file — `source/archilyzer.git/**`, `source/tree/**`, `source/manifest.json`, the
+ tarball, `snapshot.json`. It takes about 0.5 s over 2,459 files.
+- **`publishedSourceProblem`** (`:985`), asked by `deployHomepage` (`build.ts:1079`) before every
+ deploy, preview included:
+ - no `/source` page in `out/` refuses: a refused build, or one from before release 12;
+ - the page with no artefacts beside it is a `--no-source` build, and deploys;
+ - otherwise it binds: a valid manifest, a complete state, `rulesHash` = today's, `mirrorHead` =
+ `out`'s, main = the state's and the manifest's, the tarball's sha256, the gitleaks identity, and
+ the content digest.
+
+ A hand-written state file is the only way past it, and the file is the operator's own record.
+- **The report never prints a literal or an object's bytes.**
+ - A literal is `denylist line N (len L)`, `scrub line N lhs (len L)` or `built-in home rule (len
+ L)`.
+ - A hit is its kind, object id, the blob's path in history, the byte offset and a commit/tag
+ field (`objectField`, `sourceAudit.ts:229`).
+ - Every refusal message, and every path it prints, goes through `maskLiterals`
+ (`sourceAudit.ts:144`).
+ - A literal spanning path components (`a/b`) is invisible to the object walk, which reads tree
+ entry names one at a time. The staged-path sweep catches it.
+- **The operator files** are `${ARCHILYZER_CONFIG_DIR:-~/.config/archilyzer}/source-{scrub,denylist}.txt`,
+ read through `getPaths()` (`sourceScrubFile`, `sourceDenylistFile`). The two readers are
+ `operatorLines` (`sourceAudit.ts:72`) and `parseScrubRules` (`source.ts:161`):
+ - they drop a BOM and CRLF;
+ - the home dir loses a trailing `/`;
+ - an empty left side refuses;
+ - every literal left side is denied too — its exact bytes; a new spelling is caught only by the
+ denylist.
+
+### git-filter-repo (2.47.0, pipx `~/.local/bin`)
+
+- **`--replace-text` has no comments.** `get_replace_text` treats a `#` line as a literal to replace
+ with `***REMOVED***`. The step writes `replace.txt` without comments or blank lines.
+ `--replace-message` reads the same syntax. Lines split at the LAST `==>`, then
+ `regex:`/`glob:`/`literal:`.
+- **`git filter-repo --version` prints a hash (`a40bce548d2c`), not `2.47.0`.**
+ `commit-map`/`ref-map` under `<repo>/filter-repo/` hold the PRIVATE ids, and are deleted before
+ staging. The default `--replace-refs` is `update-no-add`; the step passes `delete-no-add`.
+- **A blob with a NUL in its first 8 KiB is never scrubbed** (`_tweak_blob`): a binary is audited,
+ never scrubbed. Compressed content (a zip member, PNG text chunks, a PDF stream) is opaque to the
+ byte search. At release 12 every such blob in history was decompressed by the reviewer: 0 hits.
+- **Deterministic:** the same main, rules and filter-repo gave the same `mirrorHead` on every run
+ (`20c367613f75` for main `e56101fdee5d`). Pack BYTES differ run to run, which is why the digest is
+ taken per publish.
+
+### git's dumb HTTP, and Cloudflare Pages
+
+- **The published file set** is `HEAD`, `packed-refs`, a LOOSE `refs/heads/main`, `info/refs`,
+ `objects/info/packs` and `objects/pack/pack-*.{pack,idx}`, from an allowlist: never `config`
+ (the clone's origin path), `hooks/`, `description`, `logs/`, `filter-repo/`, `*.rev` (git 2.55
+ writes them by default) or `*.bitmap`.
+- **The loose ref is load-bearing.** git treats a directory as a repository only if it has a
+ `refs/` directory, so a `file://` clone or `source audit` of the published dir fails without it.
+ An empty `refs/` would not survive a deploy.
+- **The `?service=git-upload-pack` probe falls back to dumb** when the server returns plain
+ `info/refs`. Verified against `python3 -m http.server`. The live Pages proof is the rollout's
+ preview.
+- **`_headers`** (wrangler 4.88 `attachHeaders`, re-read in 4.142 by the review):
+ - EVERY matching rule applies, in file order. A header a LATER rule sets again is **APPENDED**
+ (`text/plain; charset=utf-8, text/html; charset=utf-8`), so an override must detach it first
+ with `! Content-Type`.
+ - A splat is `(?<splat>.*)` matched against the ENCODED pathname. A rule with two `*` compiles to
+ duplicate group names and is dropped silently; wrangler's parser also refuses it.
+ - The limits are 100 rules and 2,000 characters a line. Only the ROOT `_headers` is read.
+ - `homepage/app/lib/headers.test.ts` holds a replica and pins the source-tree rules.
+- **Pages answers a missing path with the NEAREST `404.html`**, walking up, with the REQUEST path's
+ headers. So a tracked `404.html` in the raw tree would run as HTML on the origin, and the tree
+ step refuses one (`sourceTree.ts:142`), as it refuses a tracked `index.html` or a symlink.
+- **A Pages PREVIEW is a public publication, and every deployment stays reachable at
+ `<hash>.<project>.pages.dev` until that deployment is DELETED.** A newer deploy does not remove
+ an older one. A preview branch name is guessable (`source`, named in the mirrored plans).
+- **`next dev` serves `public/` but not a directory's `index.html` at `/<dir>/`,** and ignores
+ `_headers`. A static export always writes the `/source` page into `out/source/index.html`, so
+ `out/source` exists in every build, `--no-source` included.
+
+### Tailwind, tsc, eslint and docker must not read the mirror
+
+- `/homepage/public/source` and `homepage/.source-publish.json` are gitignored, because Tailwind v4
+ scans every file `.gitignore` does not exclude, and a binary pack yields "class names" that break
+ the stylesheet.
+- `homepage/tsconfig.json` excludes **`public` AND `out`**. `next build` copies `public/` into
+ `out/`, and the SECOND build with a mirror failed its type check on
+ `out/source/tree/common/jobs/registry.ts`: a duplicate `declare global var __yttJobRegistry__`.
+- `homepage/eslint.config.mjs` ignores `public/source/**`, and `.dockerignore` excludes both paths.
+
+### Measured (2026-09-28, the real repo)
+
+- **`main` `e56101fdee5d` publishes:**
+ - 1,703 commits and 21,447 objects;
+ - 2,459 staged files, 69–71 MB;
+ - 2 packs (about 21 and 16 MB);
+ - 2,035 tree files in 412 directories;
+ - a tarball of about 7.2 MB.
+- **`homepage/out`:** 2,640 files, about 78 MB. The largest file is 19.99 MiB, against the step's
+ 24 MiB limit and Pages' 25 MiB.
+- **Time:** the step adds about 19 s to `build homepage` (filter-repo about 5 s, gitleaks about 7 s).
+ An unchanged main, rules, tools and files skip it. The deploy check takes 0.6–0.8 s.
+- **The live :3001 editor runs its BUILT bundle.** Until it is rebuilt on a tree with release 12,
+ its `/sites` Homepage jobs have no source step, no withdrawal and no deploy check.
+
+### A path joined from `process.cwd()` is a directory of assets to Turbopack (slice Q, after rollout)
+
+**The hazard.** In a module a Next app imports, Turbopack traces a path joined from
+`process.cwd()` as a directory of assets. The same goes for a module's own `import.meta.url` or
+`__dirname`.
+- **How:** Turbopack evaluates these statically as paths in the project. A `path.join` /
+ `path.resolve` / fs call on the result becomes an asset reference: to a file, or, when the joined
+ path is a directory, to every file under it (`DirAssetReference`).
+- **A worktree build will not show it.** A worktree has no `transcripts/`, so the reference is
+ empty. **Test with the corpus visible.** Nor does it have umtool's e2e fixture (`.e2e-song`,
+ `.next-e2e`), which is what slice UT's pattern reached (below).
+- **What happened:** slice Q wrote `path.join(REPO_ROOT, "transcripts", "channels")` with
+ `REPO_ROOT = findRepoRoot(process.cwd())` in `umtool/lib/paths.mjs`. The walk's fallback,
+ `path.resolve(start, "..")`, evaluates to the project root.
+ - In `<primary>`, `pnpm --filter umtool exec next build` walked the corpus (hundreds of GB, with
+ channel `data/` symlinked to another drive). It grew until the kernel OOM-killed it: twice, at
+ about 3.7 GB RSS. The live umtool was down until the fix.
+ - Under a memory cap it dies at once instead: `<DirAssetReference as
+ ModuleReference>::resolve_reference failed … Symlink [project]/transcripts/channels/<slug>/archive
+ is invalid, it points out of the filesystem root`.
+ - A worktree built it in 30 s at 0.8 GB, which is how the gate passed.
+- **The opt-out is per expression, and it is not documented for path or fs calls** (corrected by
+ release 15, slice UT). `path.join(/* turbopackIgnore: true */ process.cwd(), bar)` is Turbopack's
+ own advice, in the text of its "Encountered unexpected file in NFT list" issue (the "whole project
+ was traced" warning; the 16.2.3 native binary carries it), and Next's own server uses it, on
+ the join and on the fs call around it: `next/dist/server/next-server.js:620` (16.2.3) reads
+ `existsSync(/* turbopackIgnore: true */ (0, _path.join)(/* turbopackIgnore: true */ this.dir,
+ 'static'))`. The Next docs (`08-turbopack.md` and
+ `02-guides/lazy-loading.md`, "Magic Comments") list the comment only for `import()`, `require()`,
+ `require.resolve()` and `new Worker()`. It goes before the FIRST argument of each call, and it
+ changes nothing at run time. Measured in slice UT, it works on a `path.join` and on an fs call.
+ Per call, not per value:
+ - a nested call needs its own marker (`path.dirname(/* turbopackIgnore: true */
+ fileURLToPath(import.meta.url))`);
+ - **an fs call on an opted-out `path.join` is NOT covered**, nested or through a variable: it
+ traces the join's value. Slice UT, on umtool's clip-audio route. With 4 probe files in its dot
+ directories: the join opted out and the fs calls on its value kept, 4 traced; those calls
+ stubbed, 0. With the primary's fixture: one `existsSync(path.join(/* opt-out */ CACHE_DIR, …))`,
+ 1,704; the same with the `existsSync` opted out as well, 0; the join held in a variable and
+ only `existsSync(/* opt-out */ cached)` reading it, 0. On a cwd-derived value the join is a
+ known path, so the outer call traces the one file it names; the guard lets that through (its
+ comment says so; the release 15 review ruled its four sites safe). Next's own
+ `next-server.js:620` (above) opts out both calls.
+- **A value the tracer cannot know is a dynamic part, not ignored** (corrected by release 15, slice
+ UT). `process.env.*`, `os.homedir()`, a parameter and an imported binding all make patterns over
+ the app's own directory (`umtool/`):
+ - 66 of umtool's 68 routes trace its whole tree outside dot-directories (361 files,
+ `next.config.ts` among them). Opting out every path op in `song/paths.mjs`, `lib/paths.mjs`
+ and `lib/paths.ts` (`path.resolve(process.env.X ?? path.join(os.homedir(), …))` and the like)
+ left 31 of the 68 routes clean; every path op in all 53 modules that have one (319 calls),
+ 49 (`_global-error` and `_not-found` were clean before). The rest come through fs calls
+ (`lib/report/snapshots.mjs` is the first the warning names).
+ - The clip-audio route's `path.join(CACHE_DIR, `${stamp}.${asMp3 ? "mp3" : "wav"}`)` (CACHE_DIR
+ imported, built on the env or home directory) also took in the dot-directories: 1,704 of its
+ 2,167 entries were the e2e fixture (`.e2e-song`), the e2e server's build directory
+ (`.next-e2e`) and `.env.local`. Hoisting the ternary into a `const` changed nothing. Without
+ the ternary (`${stamp}.wav`), as a ternary of two joins, or with the name handed to a function
+ from another module (the fix, `cacheFile` in `umtool/lib/paths.mjs`), it did not. The video
+ route's `.mp4` join did not reach the `.mp4` inside `.e2e-song`.
+ - **A pattern walk does not enter a symlinked directory**, in the root or out of it: a link at
+ `umtool/.e2e-song/data/planted` to `<primary>/transcripts/channels` gave 0 entries before and
+ after the fix, while the old route's walk listed the real files beside it (`data/mkvocals`);
+ one to the worktree's `common/` (675 files) gave 0 on the old route. That is what the
+ synthetic-HOME build showed, not that the env and home directory are unfollowed. A KNOWN
+ directory (slice Q's `<root>/transcripts/channels`) is different: it is walked through its
+ symlinks.
+- **The guard is `scripts/next-build-trace.test.mjs`** (in `test:scripts`; it was
+ `umtool-build-trace.test.mjs` until release 14 widened it). It scans umtool's app, components,
+ lib, `report-to-video/*.mjs` and `song/paths.mjs`, and, since release 14 (F8), `homepage/app`,
+ `export/app`, `editor/app`, `editor/lib`, `editor/instrumentation.ts` and every `common/` module
+ but `bin/`, per module. A path or fs call carrying a value derived in that file from
+ `process.cwd()`, `import.meta.url|dirname|filename`, `__dirname`, or a call to a function
+ declared in the file whose body carries one, must open with the opt-out.
+ - It is static and per module, as Turbopack's value analysis is: an imported binding is opaque to
+ it (to Turbopack it is an unknown, which is a dynamic part: see above).
+ - With the slice Q `paths.mjs` it fails on the defect's line.
+ - **Since release 15 (slice UT):**
+ - The scan set also follows every relative import out of those folders, which adds
+ `common/bin/_publicFile.ts`, `homepage/content/docs.ts` and seven `umtool/song` modules
+ (umtool 211 → 218 modules, the rest 895 → 897); a test pins them.
+ - The checked calls include `open`, `writeFile`, `appendFile`, `createWriteStream` (and their
+ Sync forms) and the `fs.promises.` / `fsPromises.` prefixes. No new finding.
+ - **It reads umtool's last build back.** Every `.nft.json` under `umtool/.next` but the
+ build's own `cache/` and `dev/` fails on an entry outside the repo, under `transcripts/`, or
+ through any name starting with a dot other than the build's own directory and
+ `node_modules/.pnpm`.
+ - **It skips, saying so,** when there is no build (`umtool/.next/server` or `BUILD_ID`
+ missing), and when `BUILD_ID` is older than `umtool/next.config.ts` or any umtool module it
+ scans (review M1). A merge or checkout gives the changed files new mtimes, and umtool runs
+ under `next dev`, which does not refresh `.next`, so a stale build skips until umtool is
+ rebuilt instead of failing on a call already fixed.
+ - **What it can see** (review L2). Without the excludes, the old clip-audio route failed it with
+ 1,704 entries (the primary's pre-fix build: 1,705, `test-results/.last-run.json` too). With
+ the excludes umtool's config now carries, such a pattern shows only through a name they
+ miss: `test-results/.last-run.json` after an e2e run, `.next-shots`, the corpus, a path
+ outside the repo. A checkout with no e2e run behind it is blind to it; the fix at the call
+ is what keeps the route clean.
+- **The build gate** is run with the corpus visible and under a memory cap (the command is in
+ `plans/tools/implementer-rules.md`). Linking `<primary>/transcripts` into a worktree is for a
+ BUILD only. Remove the link afterwards: never run an app, an index or a fixture builder through it.
+- **The other apps were safe by accident; since release 14 (F8) they are by rule:**
+ - `common/lib/paths.ts` builds every path on the repo root through one opted-out `under()`, and
+ `findMonorepoRoot()`'s walk and its `process.cwd()` fallback are opted out. Since release 15
+ `under(first, ...rest)` puts the marker before a named first argument, not a spread (the
+ release 14 review's L3); `getPaths()` is unchanged.
+ - `homepage/app/lib/source.ts`' directory join on `public` (the source mirror) and the homepage's
+ and export's other cwd joins carry the opt-out.
+ - The guard covers them. A homepage build with the published source measured the same with and
+ without it (`release-14.md`, "The final review's Lows").
+
+## The stats cache key (verified 2026-09-28, branch `fix/stats-cache-key`)
+
+The record is [`stats-cache-key.md`](stats-cache-key.md). Anchors are at the branch tip. Two notes
+on anchors elsewhere in this file:
+- The branch added lines to `buildIndex.ts`: anchors above this section that point past `:66`
+ moved by +1, and past `:559` by +6. They are not rewritten in place.
+- The three `buildStats.ts` anchors in the slice-W sections were refreshed here.
+
+- **The stats cache is keyed on the metadata AND the index's own record for the video.**
+ - `statsByPath` (`common/controller/buildStats.ts`) holds `{metaMs, idx, stat}` per
+ `[channelSlug, videoDir]` (`:92`). A stat is recomputed when `metaMs` or `idx` moved (`:482`).
+ - `idx` is `indexSignature` (`:114`) of buildIndex's whole `mtimes` record: `metaMs`,
+ `transcriptMs`, `subsMs`, `availabilityMs`, `digestMs` (every input that makes buildIndex
+ re-process the video, `buildIndex.ts:585`) and `indexKey`. It is `NOT_INDEXED` (`"-"`,
+ `:107`) when the index has no record.
+ - The cues are read under the record's own `indexKey` (`:543`). The key buildIndex used when
+ `transcript.cues.json` is fresh comes from that file's `uploadDate`, which a later metadata
+ rewrite can differ from; the computed key is only the fallback for a video the index lacks.
+ - Cost on the unchanged path: one LMDB get per video, and no file I/O.
+ - **Until schema 6 the key was `metaMs` alone.** A transcript that arrived after a video was
+ first seen never reached its stat: Whisper days later, a Normalize run, or a stats run made
+ before `build:index` had the video. The pool composers make exactly that last kind of run:
+ `poolSummary.ts:76` runs `buildStats` with no index build. A later
+ `build:index && build:stats` then reported "added 0, changed 0" over those videos.
+ - **Why the whole record, not just the transcript mtime:** buildIndex does not re-index when only
+ `transcript.cues.json` changes, but a re-index for another reason (subs, availability, digest)
+ reads the fresher cues.json and changes the cue count. Keying on the whole record redoes the
+ stat then. The cost is a recompute on those rarer changes.
+ - **Race, benign:** buildStats reads the record before the cues, and buildIndex writes the cues
+ (`buildIndex.ts:720`) before `mtimes` (`:841`). A concurrent index build can only pair an
+ older record with newer cues, which the next run recomputes.
+- **Two kinds of video have no index record, and they are counted apart.**
+ - buildIndex writes `meta.scannedAt` (`INDEX_SCANNED_AT_KEY`, `common/lib/stats.ts:19`) at
+ `buildIndex.ts:2006`, when a build completes. The value is the time its scan began (`:564`).
+ - **`notIndexedYet`:** metadata newer than `scannedAt`, or no build has completed yet. The video
+ was downloaded since, and it heals on the first stats run after the next index build.
+ - **`notIndexable`:** older than `scannedAt`, and the index has no record. The build did not
+ index it:
+ - no `upload_date` (`buildIndex.ts:701`);
+ - a processing failure;
+ - or its channel was held during that build (its media unreachable). Since release 15 (slice
+ IG) an incremental build keeps a held channel's records, so this is a video that reached the
+ drive after the last index build that could read it. Once the drive is back, a stats-only run
+ (Build stats dataset, or a pool composer) before the next index build counts it here, and
+ that index build heals it. Under `ARCHILYZER_INDEX_ALLOW_HELD`, a full rebuild drops all of
+ the held channel's records the same way (see "The index build's hold").
+
+ It stays until fixed, and is logged as such rather than as pending (`:500`).
+- **A transcript always has a date, and a caption video takes its captions' arrival.**
+ `resolveAcquisitionDates` (`:227`) tries, in order:
+ 1. transcribe-outcome's `transcribedAt`;
+ 2. the mtime of the picked index transcript, which is `transcript.json`, else the caption VTT (`:241`);
+ 3. `transcript.cues.json`;
+ 4. `downloadedDate` (`:245`).
+
+ Whisper videos resolve as before. The one difference is an outcome sidecar whose date will not
+ parse: it now falls through instead of giving null.
+- **The downgrade guard.** A build never clears a cache that a NEWER schema wrote (`:424`). It
+ throws, naming both versions and `ARCHILYZER_STATS_ALLOW_DOWNGRADE` (`:129`). That variable is
+ declared in `envVars.ts` for a deliberate rollback.
+ - The guard helps FUTURE bumps only. Schema-5 code has no guard, and would clear a schema-6 cache.
+ - `STATS_SCHEMA_VERSION` is 6 (`stats.ts:11`). It versions the cache. The pages' version,
+ `STATS_MANIFEST_VERSION`, stays 1.
+ - `digestPlan.ts:395,447` and `duplicateShorts.ts:228` only warn on a mismatch, and read
+ `value.stat` alone.
+- **An unmounted media drive is not an empty channel, for stats either.**
+ - `scanSource` asks `inspectChannelMedia` per channel (`:294`). A channel that is not `ok` or
+ `in-place` is "held": not rescanned, its cached stats kept and published (`:509`), and logged.
+ - A schema clear with any channel held REFUSES (`:443`), because the clear would drop that
+ channel's stats for good.
+ - The message names each held channel, with its storage location's label and no path.
+ - It lists the ways out, mounting first: mount its media; repair or re-point its location on
+ /storage; finish or clear its move; or, for a channel gone for good, delete it or set
+ `excludeFromBuild`. An excluded channel is skipped before the check.
+ - This is the build's own guard. The job registry's `needsMedia` check
+ (`streamCommand.ts refuseForUnreachableMedia`) is per channel and needs a `channelSlug`, so it
+ never covered this pool-wide build, from the editor or from the CLI.
+ - **buildIndex had no such guard until release 15.** An index build with a drive unmounted
+ dropped those channels' index records, and the site pages built from it lost them. It holds
+ them now: see "The index build's hold" below.
+- **One stats build at a time: an operator rule, not a lock.**
+ - Two concurrent runs are harmless unless one clears the cache (a schema change) after the other
+ has scanned. The other then collects a partly refilled `statsByPath` and publishes truncated
+ pages.
+ - There is no cross-process lock primitive in `common/`: `scripts/queue-lock.mjs` is the e2e
+ queue's flock wrapper.
+ - The editor's build jobs share the queue `"build"` by default. The per-button queue fields can
+ split them.
+ - **The CLI is outside every queue.**
+- **The homepage fold counts a transcript that has no date** (`common/lib/homepageSummary.ts:407-408`).
+ It counts toward totals, channels, hours, the card's `transcribed.total` (`withUndated`) and
+ upload-month placement. Only the transcribed series, "this month" and the recent rail need the
+ date.
+- **MCP `get_video_metadata`** prints its "## Stats" block from the stats pages
+ (`mcp/src/server.ts:2273`). **Still owed:** a video with no transcript has coverage 0 and gets the
+ "covers only 0% — truncated" note (`coverageNote`, `:2309`).
+- **A caption test fixture must carry YouTube's inline timing tags.** `parseVtt` (`vtt.ts:30`) keeps
+ only cue lines containing `<hh:mm:ss.mmm>` (`TIMING_TAG_RE`, `:3`). A plain `WEBVTT` cue parses to
+ zero cues. `maybeMissingBuild.test.ts`'s VTT is such a file, which is harmless there.
+ - The same rule makes a video whose English track is a *manual* caption (no inline tags) index
+ as 0 cues. That is rare: 0 in 3,000 sampled of the-quartering, 6 of chibi-reviews.
+- **Measured before the fix,** on the whole-pool stats of 2026-09-28T20:40Z:
+ - 49,798 transcripts shown, of about 77,000 on disk;
+ - 24,710 records with `hasTranscript` but no `transcribedDate`;
+ - 2,484 with a stale `hasTranscript: false`;
+ - Jasolyzer 0 of 1,889.
+- **The schema bump's cost:** one full re-extraction. That is 79,500 videos and 39.3 GB of
+ metadata, measured at 149 MB/s on NVMe. About a quarter of the video dirs are on a USB drive, at
+ 4–5 random reads per recompute. Expect **about 10–30 minutes, longer with a cold cache**.
+ - The pass commits per batch of 200.
+ - It is interruptible, and it resumes: the schema is written at the clear, so the next run
+ finishes the rest.
+ - Run in the editor, it stalls the editor's event loop for the length of the pass. Prefer the CLI
+ with the editor idle.
+
+## The index build's hold (verified 2026-09-29, branch `r15/index-hold`)
+
+The record is [`release-15.md`](release-15.md), "Slice IG, as shipped". Anchors are at the branch
+tip. The branch added lines to `buildIndex.ts` from `:10` on, so every `buildIndex.ts` anchor above
+this section is stale, by +16 near the top and +203 at the end; they are not rewritten in place.
+
+- **An unmounted drive is not an empty channel, for the index either.**
+ - `scanSource` (`common/controller/buildIndex.ts:309`) calls `inspectChannelMedia({ channelsDir },
+ slug, cfg)` for every channel that is not excluded and not social (`:348`), before it reads
+ `data/`. A status other than `ok` or `in-place` (`isMediaHeld`,
+ `common/lib/channelMediaHold.ts:26`) puts the channel in `held` with a reason and no path, and
+ it is not scanned.
+ - It calls it again after the walk (`:465`), so a drive that goes away mid-walk holds the
+ channel instead of dropping the videos after that point (without it, `buildIndex.test.ts`
+ case (i) removes 3 of 4).
+ - A failed `readdir(data/)` holds too (`its data directory could not be read (<code>)`), except
+ ENOENT on an `in-place` channel, which is a channel with no downloads and is logged as such
+ (`:362`). A per-video metadata `stat` failing with anything but ENOENT or ENOTDIR holds the
+ channel too, logged as `a video in its data directory could not be read (<code>)` (`:387`).
+- **What a held channel keeps, on an incremental build:**
+ - its `mtimes` records: the removal pass skips its keys and counts them (`:715`), so `sums`,
+ `cues`, `subs`, `digests` and `byChannel` keep them too;
+ - its shared transcript, subs and digest trees: not rewritten, not pruned, not removed
+ (`:1204`, `:1393`, `:1772`, and the top-level cleanups `:1513`, `:1879`);
+ - its subs and digest stats, carried from `channelStatsDb` / `channelDigestStatsDb`, so the
+ per-site subs and digest manifests still list it;
+ - its availability states: the maybe-missing overlay skips it (`:1324`), and the last build's
+ `videoState` entries for it are carried over (`:1356`). Its `availability.json` files are on
+ the missing drive, and a missing one reads as `maybe_missing`.
+ - The per-site summaries come from LMDB, so the site built next still lists its videos.
+- **A curated-tag change while a channel is held:** the re-apply pass re-derives its records in
+ LMDB (no disk read), but its pages are not written, so `curatedPagesPending` is NOT cleared while
+ a channel is held and the pass had pages pending (`:1551`). The first build with the drive back
+ rewrites them.
+- **A full rebuild with a channel held refuses** (`:649`). A full rebuild is a schema change or a
+ first build (no `meta.schema`); it clears every sub-DB, and a held channel cannot be re-read.
+ - The scan now runs BEFORE the clear (`:639`), so the refusal leaves the index untouched.
+ `scanStartedAt` is still taken at the scan's start.
+ - The message names each channel with its location's label, the ways out (`HELD_WAYS_OUT`,
+ shared with the stats build's refusal), and `ARCHILYZER_INDEX_ALLOW_HELD` (`:512`, declared in
+ `envVars.ts`) with where it is set: the command's own environment for a CLI run; the editor's
+ own environment, and so a restart, for the editor's Build index job or a site build started
+ from the editor (their children inherit `process.env`). The CLI exits 1 on it, so a site
+ build's data phase fails with it.
+ - With the variable set, the build proceeds: the held channel's records go with the clear, its
+ shared trees are left on disk, and it is out of the index (the site lists it with 0 videos)
+ until its media is back and an index build runs.
+- **Who sees it:** `BuildIndexResult.heldChannels` (`:502`); the log's per-channel line, the
+ `Diff:` line's ` Held: N channel(s), K video(s) kept.` suffix (`:780`), and the `Done in` line's
+ ` Held, their media not readable: <slugs>.` suffix. The CLI (`archilyzer index`, the export's
+ and homepage's `build:index`, so every site build's data phase) prints the log; the editor's
+ **Build index** job (`buildIndexAction`) streams it into the job log, which `pnpm ops build-index
+ --wait` follows. Nothing reads the result's field outside the tests.
+- **The words are shared with the stats build** (`lib/channelMediaHold.ts`): `HELD_REASON` is a
+ `Record<ChannelMediaStatus, string>`, so a new status added to `inspectChannelMedia` fails tsc
+ until it has a reason; `isMediaHeld` treats any status but `ok` and `in-place` as held.
+- **Not covered:** a drive that drops during the PROCESSING phase (after the scan), for a changed
+ video whose metadata was read before the drop. A transcript read that fails after it is caught
+ as "no cues" (`cueList = undefined`, `:838`); a sub-track read that fails is skipped, and with
+ none left the video's subs are removed (`subs.remove`, `:895`); a digest load that fails leaves
+ no digest, which is removed (`digests.remove`, `:962`/`:965`). `mtimes` is then written with the
+ video's current mtimes, so the loss lasts until any of its tracked mtimes (metadata, transcript,
+ subs, availability, digest) moves.
+
+## The storage health gate (verified 2026-09-29, branch `r15/drive-stall`)
+
+The record is [`release-15.md`](release-15.md), "Slice DS, as shipped", with the parent's rulings
+(Q1–Q5) and the review's fixes (M1–M3, L1–L10); the timings became settings in "Slice DT, as
+shipped". Anchors are at slice DT's tip (`r15/drive-timings`).
+
+- **A drive can be mounted and not answering.** Every in-process fs call on it waits on one of
+ libuv's threads (4 by default, 16 in the editor's `start`) until it answers (~30 s for the observed
+ USB reset loop); only a child process isolates a call. A child `stat` of a location's ROOT does not
+ detect it reliably: the root's inode is in the kernel's cache whenever the drive was used lately.
+- **The timings are settings: `settings.storage.health`** (slice DT). `budgetMs` (default 3000,
+ 500–60000), `passIntervalMs` (15000, 5000–300000), `probeTimeoutMs` (3000, 500–30000),
+ `clearAfterCleanPasses` (2, 1–10), `inFlightPerLocation` (4, 1–8: at most half the editor's 16
+ file-access threads, review M1); the defaults, ranges, sanitizer
+ and words are `common/lib/storageHealthTimings.ts` (pure; the /storage form imports it). A read
+ clamps into the range and keeps only a value that differs from its default (an untuned file has no
+ `health` key); the /storage form refuses out of range with a sentence. Every number is read through
+ ONE accessor, `healthTimings()` (`storageHealth.ts:182`), from the timings as last applied on
+ `globalThis` — no file read per call. `applyHealthTimings(stored)` (`:193`) sets them: the health
+ pass on every pass (from the settings it reads), /storage's save at once
+ (`saveHealthTimingsAction`), and the `index` and `build stats` bins once at start. Any other process
+ with no pass (a CLI, `archilyzer doctor` until slice SG adds its line) runs on the defaults. A changed interval is told to
+ `onPassIntervalChange` (`:214`) subscribers, which re-arms the armed pass's timer; a raised cap
+ admits waiting calls, a lowered one is reached as calls return. `resetStorageHealth` keeps the
+ timings (configuration, not health); `setDriveCallBudget` is a test seam below the 500 ms floor and
+ wins over them.
+- **The state is `common/lib/storageHealth.ts`**, one map on `globalThis.__yttStorageHealth__` (the
+ pass writes it from instrumentation's module copy; pages read it from theirs), with each entry's
+ `detector` and the counters' `device`. `recordLocationHealth` (`:251`): one `stalled` answer stalls
+ at once, and every transition to stalled refuses `onDrive`'s waiting calls; `clearAfterCleanPasses`
+ (2 by default) clean answers in a row clear it; `absent` is clean; a new root starts over.
+ `registerLocationHealth` (`:336`) creates entries with no answer. `stalledLocationForPath`
+ (`:388`) matches like `locationOfDataDir`; `stalledLocation` (`:404`) is by id AND root.
+- **Detector 1, every `passIntervalMs` (15 s): the block device's counters** (`detectLocationHealth`,
+ `lib/storageVolumes.ts:607`). The root's device from the last pass (findmnt `-J -T <root> -o
+ SOURCE,UUID`, raced against `probeTimeoutMs` (3 s), only when there is none or its `/sys` entry stops reading;
+ another volume's UUID names none; `[…]` stripped, `/dev/mapper` resolved, basename); then
+ `/sys/class/block/<dev>/stat` (`parseBlockStat`, `storageHealth.ts:831`): completed = fields 1 + 5
+ + 12 + 16 (reads, writes, discards, flushes), in flight = field 9. Stalled ⇔ in flight at both
+ samples AND nothing completed between; samples at least `minCounterIntervalMs()` apart
+ (`storageVolumes.ts:472`: min(10 s, interval − 5 s), floored at half the interval — 10 s at the
+ default 15 s); the first gives no verdict. in_flight counts only requests dispatched to the driver: one requeued
+ during a host reset is not counted, so a sample in that window can read clean (the watchdog covers
+ it). No device → the child `stat -L -c %F` probe (`probeLocationHealth`, `:388`, raced against
+ `probeTimeoutMs`). The samples are
+ on `globalThis.__yttHealthDetector__` (the pass and /storage's Refresh share them).
+- **Detector 2, on every gated call: `onDrive(where, call)`** (`storageHealth.ts:722`). Refused with
+ no call on a stalled location; otherwise raced against `budgetMs` (3 s by default, read as the call
+ starts; test seam `setDriveCallBudget`); the budget covers the whole unit passed in. A timeout marks the location
+ stalled (since now) and throws `DriveNotAnsweringError`, leaving the call to settle — unless the
+ location's device counters (read synchronously from `/sys` through the reader `storageVolumes.ts`
+ registers with `setCounterReader`, `:515`) moved since the call began: then the call is refused
+ as slow and nothing is marked. At most `inFlightPerLocation` (4 by default) calls per slot key in
+ flight (`acquireSlot`, `storageHealth.ts:573`): the rest queue in JS. A waiting call's deadline follows progress: every call that
+ returns on the key (in time or late) restarts it (`releaseSlot`, `:673`, re-arms every waiter), and a
+ waiting call is refused unmarked only when nothing on the key has returned for the budget plus a
+ quarter of it (at most 250 ms) — never for the queue's depth alone. Waiting calls are refused at
+ once by any transition to stalled; and when every slot is held by a call already past its budget
+ (`overdue`, each kept with the counters reading from when it began), a new call is refused at once,
+ and the location is marked stalled again only if the disk has completed nothing since the oldest of
+ them began (otherwise "drive slow", unmarked). A slot is freed when its call really returns. The
+ slot key: a configured location's id; a probe of another root under its id, that root (marks
+ nothing); a path on no configured location, the root it is under (`rootOfUnknownPath`, `:529`;
+ marks nothing). Do not nest it for one key. The timer is not unref'd.
+- **The cadence** is `runStorageHealthPass` (`controller/storageWatch.ts:442`): apply the timings it
+ read, prune, register, then every location concurrently; every `passIntervalMs` (15 s) from
+ `startStorageHealthWatch` (`:566`, re-armed when the interval changes), plus one at arm time,
+ armed by `editor/instrumentation.ts` ABOVE the idle gate (it writes nothing). The five-minute pass
+ (`startStorageWatch`, `:535`) stays below it. A CLI process has no pass (its inspects are still
+ raced). `refreshLocationHealth` (`:499`) is /storage's Refresh.
+- **The gate order in `inspectChannelMedia`** (`lib/channelMedia.ts:317`): config → memo (a
+ remembered `in-transition` is returned as is, anything else is gated first) → the relocation
+ marker (`:367`, corpus disk) → the gate (`:386`) → the link (corpus disk) → the target's `stat`
+ through `onDrive` (`:447`). A stall is never memoised. Other gated calls: `probeLocation`
+ (`storageVolumes.ts:239`, its stat and statfs through `onDrive`) and its memo; `volumeFreeBytes`
+ (`controller/storageLocations.ts:289`, `:326`, `:335`); `readChannelStat` (`controller/channels.ts:205`,
+ the walk through `onDrive`, `null` on a stall); the snapshot walk (`controller/channelSnapshot.ts:753`,
+ its listing, keep-latest keys and per-video unit); the recency tail reads; the move-root check; the
+ saved-video store; `listSavedVideos` with `notAnswering`; the videos list, the video page, the
+ Cleanup stage, the Storage stage's statfs and the media file route. `channelMediaStall(config)`
+ (`channelMedia.ts:232`) is the no-I/O question for a holder of a config.
+- **`stalled` is a sixth `ChannelMediaStatus` and a sixth `StorageLocationStatus`** ("Not
+ answering"). `HELD_REASON`, `MediaLocationBadge`'s two tables and `STORAGE_STATUS_LABEL` are the
+ `Record`s that make tsc name every table a seventh would need. `isMediaHeld` holds it, so both
+ pool-wide builds hold a stalled channel; the storage watch counts it as down (two passes pause),
+ on a location or not, and the pause record carries `cause: "not-answering"` (`ChannelAutoPause`,
+ `lib/channelPriority.ts`; absent = not there).
+- **`inspectChannelMedia` is memoised for 5 s** (`CHANNEL_MEDIA_MEMO_MS`, `:266`), keyed by channels
+ dir, slug and configured `dataDir`, on `globalThis.__yttChannelMediaMemo__`. `{ fresh: true }`
+ skips it and does not store; the deciders that pass it are listed in the record (the guard and its
+ six callers, both movers, both builds, the watch, eviction, the re-point preflight, doctor). The
+ runners' tick shares the status poll's `buildChannelWork` and so reads the memo.
+ `forgetChannelMedia` (`:292`) is called by the channel mover's marker writes and clear,
+ `clearRelocationMarker`, a re-point, /storage's Refresh and the e2e `invalidate-cache` route.
+- **`UV_THREADPOOL_SIZE`** defaults to 16 in `editor/package.json`'s `start` and in
+ `docker/entrypoint.sh`. `ports.test.ts` reads every `${NAME:-N}` in a script as a port and names it
+ as the one exception (`NUMERIC_NOT_PORTS`, `common/lib/ports.test.ts:29`).
+- **Not covered:** a call already in flight when the drive stalls (at most `inFlightPerLocation` per
+ drive for the calls through `onDrive` — every page and poll path and the snapshot walk; a job's own reads that do
+ not go through it, `measureTree`, the index build's processing phase, the snapshot's sequential
+ reconcile pass, are not capped); a hand-typed root is capped but never marked; a drive already
+ stalled at boot before the second counter sample, unless a page reaches it; per-click server
+ actions; the file route's stream; the corpus disk itself.
+
+## The source's history (stagit) (verified 2026-09-30, branch `r15/stagit`)
+
+Release 15 slice SG ([`release-15.md`](release-15.md)); the operator-facing doc is `PUBLISH.md`, "The
+source mirror (homepage)". Anchors are at the branch.
+
+- **stagit** (codemadness.org, C over libgit2; built from `519cae2f`, "bump version to 1.3",
+ 2026-09-14, against the system libgit2 1.9.7) renders the scrubbed mirror into `/source/git/`.
+ It is operator-installed like git-filter-repo: never vendored. `resolveStagit`
+ (`publish/sourceHistory.ts:122`): `paths.stagitBin` (`STAGIT_BIN`, else `"stagit"`, `lib/paths.ts:314`)
+ — a name with a slash is that file or nothing; a bare name is PATH, then `~/.local/bin/<name>`.
+ It has no version flag: `stagitIdentity` (`:136`) is the first 12 hex of its binary's sha256,
+ never the path (the manifest is published, and a home-dir path is a denied literal).
+- **What stagit does** (read in `stagit.c`): it writes into its CWD; the repository's NAME is its
+ directory's basename minus `.git` — so the step's scratch clone is `<scratch>/archilyzer.git`
+ (`source.ts:834`), no longer `bare`; `description` (read with `fgets`, the newline KEPT — write it
+ without one) and `url` (newline stripped) come from the repository directory; the header links
+ `<relpath>file/README.md.html` and `LICENSE` when HEAD has them, and the logo `<a href="../<relpath>">`
+ lands on `/source/` from every depth; `file/` pages are written for HEAD's whole tree on EVERY run;
+ every text from the repository is xmlencoded (`"` → `"`), so no page text can fake an
+ `href="…"`; `-l N` keeps the NEWEST N log lines (the revwalk is newest first) and ends "M more
+ commits remaining, fetch the repository", but still writes a page for EVERY commit (it only skips
+ the diffstat of a page that exists); `-c` with `-l` is a usage error; a diff's paths are percent-encoded (`/` and `,-.` kept); a commit over 1,000 files or
+ 100,000 added or deleted lines prints "Diff is too large, output suppressed"; `-c <file>` walks from
+ HEAD to the commit the cache file names — in COMMIT-DATE order, so a `--no-ff` merge of commits
+ older than that commit leaves them out (the step's count check then renders every page again,
+ about 8 s: common with parallel slices, and correct) — KEEPS every `commit/<sha>.html` already in
+ the CWD, and appends the cached log lines — so the cache is the whole output directory, and a cache
+ whose commit is not an ancestor duplicates the log. Its temp cache file (`cache.XXXXXXXXXXXX`) is made in the CWD.
+- **The cap** (`SOURCE_HISTORY_MAX_COMMITS`, 10,000; ruled 2026-09-30): up to it, `-c`; past it,
+ `-l 10000`. Either way the pages published are exactly the commits the log links
+ (`loggedCommits`: every `href="commit/<sha>.html"` in log.html), each of which must exist, and the
+ first must be the head. The manifest's `history.total` is every commit, `commits` those with a
+ page; `/source/` says "the latest N of M" when they differ. Past the cap each publish computes N
+ diffstats (no `-c`): 8.75 s at 1,834 commits with every page present, against 0.58 s with `-c`.
+ The oldest published page's parent link is then a 404.
+- **The step** (`source.ts`, step 12b) runs after the object audit and the mirror's staging,
+ before the manifest and the file audit: `stageHistoryPages` (`:1114`) → `renderHistory`
+ (`sourceHistory.ts:573`). **An allowlist is staged** (`TOP_FILES`, `COMMIT_PAGE`, `:94`): `log.html`,
+ `files.html`, `refs.html`, `atom.xml`, `tags.xml`, `commit/<40 hex>.html`, plus `style.css`. Never
+ `file/`, never a leftover. Pages are rewritten byte for byte (latin1 in and out; every injection is
+ ASCII: the back link says `·`).
+- **The post-pass** `rewriteHistoryPage` (`:327`), pages only: `<script>` + the homepage's pre-paint
+ script (`homepageThemeScript`, `:88` = `buildThemeScript({ defaultBase: HOMEPAGE_DEFAULT_BASE })`)
+ before `</head>`; `HISTORY_BACK_LINK` (`:85`) after `<body>`; `href="(../)*file/<p>.html"` →
+ `href="$1../tree/<p>"`; `logo.png`/`favicon.png` → `/icons/icon-32.png` (a route the homepage renders
+ at build). With the published set (`PublishedSet`: the staged raw tree's files, decoded; the log's
+ commits), a link to a file main no longer has (a diff of a deleted or renamed file: 3,342 anchors at
+ 1,882 commits) or to a commit with no page loses its ` href="…"` and keeps its text and its `id`
+ (a diff header is the diffstat's `#h<n>` target); `a:not([href])` is styled as text. At `4dfe21e5`
+ every one of the 32,609 remaining tree links and every commit link resolves. The theme config moved to `lib/themeConfig.ts` for this (publish may not import
+ components/; `components/themeConfig.ts` re-exports it); `HOMEPAGE_DEFAULT_BASE` (`:62`) is what the
+ homepage layout passes. The attribute the script sets is `data-base`, not `data-theme`.
+- **`style.css`** is `historyStylesheet` (`:215`) over `common/styles/tokens.css`: the light block
+ (`:root, html[data-base="light"]`), the dark block for `html[data-base="dark"]` AND for
+ `:root:not([data-base])` under `prefers-color-scheme: dark` (no JS); `HISTORY_TOKENS` and whatever
+ they reach through `var()`. Diff `+` lines are `--success`, `−` lines `--destructive`; there is no
+ chart-safe red in the tokens (the chart palette is blue, green, violet, amber, magenta, rust).
+- **Never a failed build over the history.** No stagit: one line with the install (`source.ts:973`).
+ A render that fails (`HistoryProblem`), a stagit timeout (600 s), unreadable tokens, more files
+ than the step's 15,000 in all, or a page over 24 MiB: a `[source] WARNING: …` and no history. A
+ cancel still cancels; any other error is a crash as before.
+- **The manifest's `history`** (`lib/sourceManifest.ts:88`, parsed at `:140`, all-or-nothing):
+ `href`, `commits` (`git rev-list --count`), `head`, `files`, `bytes`, `sha256` (`historyDigest`,
+ `source.ts:508`: `sourceDigest`'s walk over `source/git/**` alone), `tool`. `sourceDigest` (`:492`)
+ now walks `source/git` too. The state (`.source-publish.json`) gains `stagit` and `history:
+ {files, digest} | null`; the skip needs both to match, and a stagit present with no history never
+ skips. `SOURCE_STEP_VERSION` is 4 (`:109`).
+- **The deploy check** (`publishedSourceProblem`, `:1254`) names the history on its own (`:1336`):
+ `out/source/git`'s digest must be the state's (null when none) and the manifest's.
+- **The render cache** is `${XDG_CACHE_HOME:-~/.cache}/archilyzer/source-history/`
+ (`getPaths().sourceHistoryCacheDir`; an empty `XDG_CACHE_HOME` is unset; ruled 2026-09-30):
+ `key.json`, `stagit.cache`, `out/`, and `lock` while held. `historyCacheFor` (`source.ts`) gives
+ none for `--check` and none, with a line, inside the checkout or the public dir. `holdHistoryCache`
+ (`sourceHistory.ts:508`): the lock names the holder's pid; a live holder with a lock under an hour
+ old (`HISTORY_LOCK_STALE_MS`, `:498`) → render without the cache; a holder not running, or a lock
+ over an hour old → emptied and taken, with one line. An I/O error on the dir, the lock or the key
+ (`isIoError`, `:653`: an errno code — EACCES, EROFS, ENOSPC) → one line and a render without the
+ cache; a key that cannot be written is no key. Every line `renderHistory` logs goes through
+ `maskLiterals` (`source.ts`' `onLog` for it).
+ Its pages are kept when the key (`historyCacheKey`: rules hash, filter-repo, stagit, description,
+ clone URL, base URL) matches and `out/log.html` exists; its `-c` file only when the commit it names
+ is an ancestor of the head (`git merge-base --is-ancestor`), else that file alone goes. A page of a
+ commit no longer in history is never published (only the log's commits are). The key is removed
+ before a render and written after it. A cached render whose log names a page that is not there
+ is rendered again from nothing, once. `--force` renders
+ afresh; `--check` renders in its own scratch; a refusal removes the cache (`source.ts:716`).
+- **Measured (2026-09-30, `main` `6b8aa450` → mirror `8c924ca31701`):** 1,872 commits; 1,878 files,
+ 141,299,532 bytes; the largest `commit/7ddfc955….html`, 5,017,646 bytes; stagit 8 s of a 32 s
+ publish (every page); the whole publish 4,396 files, `homepage/out` 4,576. On the published mirror
+ at 1,834 commits: 8.0 s every page, 2.0 s from the cache with nothing new. The cache is 138 MB.
+ At `main` `4dfe21e5`: 1,882 commits, 1,888 files, 4,409 staged in all. One file per commit: the cap
+ keeps the history at 10,006 files at most; the 15,000-file drop is the last resort.
+- **`_headers`**: `/source/git/*` is `X-Robots-Tag: noindex` (ruled at review, M1), like the raw
+ tree; no other rule matches it, so the pages keep their own types (`headers.test.ts` pins both).
+- **Doctor** (`bin/doctor.ts`): the stagit line ends `; cache: <path>, <size>`. Since the merge of
+ slice DT it reads the settings before the corpus and applies `settings.storage.health`
+ (`applyHealthTimings`) before it inspects any drive, and prints the timings in force on a
+ `drive health` line.
+- **The homepage** believes `history` only beside `source/git/log.html` (`homepage/app/lib/source.ts:43`;
+ without it the manifest comes back without the block). Its e2e reads a fixture publish from
+ `E2E_SOURCE_PUBLIC_DIR` while it holds a manifest (`sourcePublicDir`, `:69`; never in a production
+ build).
+
+## Search in (verified 2026-09-30, branch `r16/search-in`, after its review)
+
+- **The row governs "transcripts"-scope leaves only.** The Filters panel's "Search in" row
+ (`common/components/FiltersPanel.tsx:421`, `data-testid="search-in-row"`) says what a leaf of
+ scope `"transcripts"` reads — the default leaf, `newLeaf({scope: "transcripts"})` in
+ `emptyRoot()` (`common/lib/searchQuery.ts:86`), and every plain query. A leaf asked for by
+ name ("Live chat", "Posts", "Title / channel", "Description", "Keywords") is not the row's
+ business — including "Posts" with Posts unticked: `nop` no longer gates the global scope
+ (`SearchSessionContext.tsx:898` checks only the tag filter). The browse listing (an empty
+ query) is the Type row's.
+- **The mechanism is a rewrite of the committed tree just before it runs**, never of the tree
+ itself: `applySearchIn(root, {transcripts, posts, chat})` (`searchQuery.ts:544`) turns an active
+ transcripts leaf into itself (transcripts only), itself with another scope and the same id (one
+ other kind), or an OR group `<id>~in` over copies `<id>~transcripts` / `~posts` / `~chat` (two or
+ three kinds; a negated leaf is a negated one-child AND `<id>~not` around the OR). The session
+ (`SearchSessionContext.tsx:589`) runs the rewrite and folds the progress back with
+ `foldSearchIn` (`common/lib/search/searchIn.ts:18`; called at `SearchSessionContext.tsx:960`):
+ copies' hits are re-filed under the visitor's leaf id, their states folded into one (the count
+ is the OR group's union once every copy has started). `committedRoot`, its canonical hash and
+ `qt=` stay the visitor's tree. `lib/search/` has a seventh module for this, `searchIn.ts`.
+ Each copy has its own hit cap, so a plain leaf reading two or three kinds can show up to 2–3×
+ "Max hits" before "Load more", as an explicit OR of leaves always could.
+- **Posts, as read by a plain query, is new in release 16.** On `main` before it a transcripts
+ leaf never read posts: posts joined the global scope only when a posts leaf was in the tree
+ (`needsPostsManifests`), and a non-posts leaf subtracts the posts slugs
+ (`searchEval.ts:337`). The row reads them through a posts copy of the leaf.
+- **Under a curated-tag filter no posts copy is made** (`searchInUnderTags`, `searchQuery.ts:507`):
+ a post carries no tags. Unless posts are all the row reads — then the copy stays and reads an
+ empty scope, so the leaf does not fall back to its transcripts.
+- **An empty scope settles.** Each streaming driver in `lib/search/leafPipeline.ts` finalizes when
+ no worker is running after its first `ensureWorkers` and after a raised cap (`settleIfIdle`,
+ `:199`, `:342`, `:528`); before, zero slugs never reported done and the query read "searching"
+ for ever. `searchEval.runLeaf` also answers an empty effective scope at once, with no cache
+ lookup (`searchEval.ts:353`).
+- **What counts as ticked is what the site can honour** (`SearchSessionContext.tsx:552`, `:562`):
+ Posts only where `postsManifest.channels` is non-empty, Live chat only where
+ `subsManifest.liveChatTotalCount > 0` — the same conditions the panel offers the boxes on. **It
+ waits for the manifests:** `SearchDataValue.manifestsSettled` (`SearchDataContext.tsx:67`; the
+ single site's settles at each manifest's first answer or first failure, `:165`; the hub's is
+ `summariesReady`, since an archive is ready only once both its manifests have settled). Until
+ then Search is not refused and the query does not run (`SearchSessionContext.tsx:944`).
+ Nothing ticked (`draftSearchInEmpty`, `:583`) disables Search, `Apply filters`, Save and Save
+ as…, and `commitSearch` and the two profile saves refuse it. The row's line (`FiltersPanel.tsx:462`,
+ `aria-live="polite"`, not `role="status"`: from xl the panel is always mounted, and the video
+ modal's own status is found by role) is always mounted, empty unless refused; the Search button is described
+ by an always-mounted copy in the bar (`SearchBar.tsx:182`, `id="search-in-refusal"`), since the
+ panel below xl is a sheet that is not mounted while closed. The words are
+ `SEARCH_IN_REFUSAL` (`SearchSessionContext.tsx:190`).
+- **The keys:** `notr` (Transcripts unticked) and `lc` (Live chat ticked) beside `nop` in
+ `FilterSnapshot` (`common/components/exportFilterStorage.ts:38-40`), in the working snapshot and
+ the profiles of `ytdlp-tb:export-filters`, each written only off its default. **Share links do
+ not carry them** (nor `nop`): no share-v1 key, no `ShareSelection` field. The row is read from
+ the stored snapshot on hydration whatever the URL carries, so a `qt=` or share link reads with
+ the visitor's own row.
+- **The display:** every track hit wears its `TrackBadge`, live chat included
+ (`SearchResults.tsx:933`); a transcripts leaf's section is named for what it holds
+ (`sectionScope`, `:949`): "Posts" on a post's card, "Live chat" when every hit is a chat hit.
+- **Not governed by the row:** the charts' own search series (`components/charts/useSearchSeries.ts`)
+ and `/ask`'s retrieval (`export/app/lib/askRetrieval.ts`) build and run their own trees; the MCP
+ has no row.
+- **The export e2e fixture's posts have words of their own** (`kappa`, `sigma`, `omega` —
+ `export/e2e/fixtures/data.ts`, `postsPage`), so a plain "alpha" or "gamma" in a spec reads no post.
+
+## Use with AI is the homepage's AI and MCP doc (verified 2026-09-30, branch `r16/research-setup`)
+
+- **The research-only setup lives on the homepage doc; `README.md` §1/§4 and `mcp/README.md` carry
+ copies to change with it.** `homepage/content/docs/ai-and-mcp.md` "Ten-minute setup": clone (or
+ the tarball) → `cd archilyzer && pnpm install` → `claude mcp add archilyzer --env
+ TRANSCRIPT_SITE_URL=…` → `claude`, `/ask`. **README §4's quickstart carries that whole sequence,
+ in the same order** (its command adds the two optional editor lines). **README §1 and
+ mcp/README's "Add to Claude Code" and `mcp.json` examples share only the registration**, `claude
+ mcp add archilyzer … -- pnpm -C <checkout> --filter yt-dlp-transcript-mcp exec tsx src/index.ts`:
+ README §1 with `"$PWD"` from the repo's root, mcp/README with the placeholder
+ `/ABS/PATH/TO/archilyzer`. Nothing shares text between `homepage/content` and the READMEs
+ (`homepage/content/README.md`'s drift table names them).
+- **A site has no `/use-with-ai` page** (export and hub). `AI_DOC_URL` (`common/lib/project.ts`,
+ `${PROJECT_URL}/docs/ai-and-mcp/`) is the target of the header's nav entry (a plain `<a>`: the
+ header's and `MobileMenu`'s links take `external: true`), the footer's link and Ask AI's link,
+ label "Use with AI", same tab. `corpus.json`'s `useWithAi` keeps its key and names the doc (site
+ and hub); `llms.txt`'s "## Ask AI" is two lines, `[Ask AI](<base>/ask/)` and `[Use with AI](<doc>)`;
+ the sitemap's routes are `/`, `/changelog` (+ `/downloads`, `/duplicates` when built).
+- **A route outside the workspace, in the same page life, is reached in a spec through
+ `window.next.router.push(…)`.** Next 16.2.3 sets `window.next.router` "for debugging"
+ (`next/dist/client/components/app-router-instance.js:388`), in development and production. It is
+ undocumented (not in `node_modules/next/dist/docs/`): check it first on a Next upgrade; if it is
+ gone, the specs fail with a TypeError rather than pass. Since
+ the header's Use with AI left the site, no site link makes a client navigation to a page outside
+ the workspace (the footer's are plain `<a>`, a new page life; Downloads and Duplicates show only
+ when built), so `first-search` and `restore-no-refire` push `/changelog/` that way.
+- **A multi-line JSX text that holds an HTML entity loses its leading space** under Next 16.2.3's
+ SWC: `<b>X</b> is the⏎project's` compiles to `"is the project's"` (without the entity, or on
+ one line, `" is the …"`). Put `{" "}` after the element. Measured 2026-09-30 by compiling every
+ `.tsx` with and without its entities (`next/dist/build/swc` `transform`): 22 texts in 19 files differ,
+ 1 in `homepage/app/downloads/page.tsx` and 21 in `editor/app/**`; none in `export/app` or
+ `common/components`.
diff --git a/plans/STATE.md b/plans/STATE.md
@@ -3,6 +3,119 @@
The working memory for the local-AI derived-corpus work. Rewritten at the end of every
session, before context is cleared. See [`README.md`](README.md) for the protocol.
+**Now (2026-09-30, evening): release 16 is merged to `main` `1064d04c` — CK the "Search in" row
+(Transcripts, Posts, Live chat; the first two on by default) and DX the research-only setup told
+in one place (the homepage's AI and MCP doc; the sites' Use-with-AI page removed) — on top of
+releases 14 and 15 (`e9e3e4ec`) and the operator's 0.11.0 cut.** Records: [`release-16.md`](release-16.md),
+[`release-14.md`](release-14.md), [`release-15.md`](release-15.md); each slice reviewed SHIP. The
+operator's runbook is `~/reports/release-15/RUNBOOK.html` (its banner carries the current lines).
+- **Live (2026-09-30):** the editor on `:3001` serves the release 14+15 build (`WDRioDSFbYzP4Xmu5kVpO`,
+ 16 fs threads); the index and stats are rebuilt (`Held: 0`); the homepage is in production from
+ `9a259787` (mirror `971e9c46`, the stagit history live) and Jeralyzer from the same tree, both
+ with the renamed `@Archilyzer` X link; the hub and the other five sites are at their earlier
+ deploys.
+- **Owed (the operator's, refused to the session by its permission layer):** umtool's restart
+ (rebuilt and tested); the release 16 production deploys — homepage, hub, six sites — each built,
+ previewed and checked by the session on 2026-09-30 evening.
+- **Ruled and built (2026-09-30):** the source's history under `/source/git/` by stagit (SG;
+ Sendforge was out: JS for history, no sub-path); the drive-health timings as
+ `settings.storage.health` (DT).
+- **Follow-ups the reviews named:** the editor's and export's build traces list dot-directories
+ (cosmetic while `standalone` is off); `REQUIRED_TOKENS` lacks `--chart-other` (the e2e pins it);
+ `export`'s `WorkspaceView` `splitOn` has a one-paint flash from a localStorage restore;
+ "Load more results" never resumes a leaf that settled at its cap (`runQueryTree`'s `setHitLimit`
+ reaches running leaves only; on `main` too — release 16 CK re-review R-I1; wants a spec).
+ - `/changelog/` overflows a 390 px phone: long inline code in a released entry
+ (`export/CHANGELOG.md:103`, the 133-character `export/app/ask/{useAskChat,…}` list) cannot wrap. Wrap `<code>` in the changelog renderer, then drop
+ the `test.fail` in `export/e2e/responsive.spec.ts` (release 16 DX).
+ - The JSX entity/whitespace hazard sweep: 22 texts in 19 files run a word into the element before
+ them (FACTS, "Use with AI is the homepage's AI and MCP doc"; release 16 DX).
+
+**Now (2026-09-28, night): the stats cache key fix — built, reviewed (SHIP AFTER FIXES, then SHIP
+on re-review; every touch-up done), not merged.** The branch is `fix/stats-cache-key`, and [`stats-cache-key.md`](stats-cache-key.md)
+holds the record, the review and the rollout. FACTS has "The stats cache key".
+- **What it fixes:** the homepage showed Jasolyzer as 0 transcripts, 0 channels, 0 hours while it
+ served 1,889 videos. Instance-wide it showed 49,798 transcripts of about 77,000.
+- **The cause:** `statsByPath` was keyed on the metadata mtime alone. It is now keyed on the index's
+ own record as well, and a transcript always has a date.
+- **Added by the review:**
+ - a guard against clearing a newer cache (`ARCHILYZER_STATS_ALLOW_DOWNGRADE`);
+ - an unmounted drive's stats are kept, and a cache clear with one refuses;
+ - "not indexed yet" and "not indexable" are counted apart.
+- **Owed after the merge:** the rollout in the record, in its order. First, rebuild and restart
+ :3001 (until then, never press "Build stats dataset"). Then index, then one full stats pass of
+ 10–30 min, then the homepage, the hub and the sites. The homepage deploy waits on release 12's
+ step 0: it runs the source publish.
+- **Merge note:** `homepage/social-visible` merged `main` (`10cefd15`) at `4d11542c`; the one
+ conflict, `homepage/CHANGELOG.md`'s `[Unreleased]`, kept both sides.
+- **CLOSED by release 15 slice IG (branch `r15/index-hold`, [`release-15.md`](release-15.md)):
+ the index build no longer treats an unmounted drive as an empty channel.** It holds the channel:
+ not rescanned, its index records and shared pages kept, and the `Diff:` line says
+ ` Held: N channel(s), K video(s) kept.` A full rebuild with a channel held refuses unless
+ `ARCHILYZER_INDEX_ALLOW_HELD=1`. FACTS has "The index build's hold". Until that branch is merged
+ and rolled out, the rollout's step 3 `Diff:` check stays the safeguard: thousands removed means
+ a drive was missing.
+
+**Now (2026-09-28, evening): release 12 — the source mirror — is merged to `main` and NOT rolled
+out.** [`release-12.md`](release-12.md) holds Q's and R's records, their reviews, "Merged" and
+"Rollout". The operator's runbook is `~/reports/release-12/RUNBOOK.html`, with its scripts in
+`~/reports/release-12/scripts/`. The plan is [`source-mirror.md`](source-mirror.md).
+- **What merged:**
+ - **Q** (`4855f70b`, and the changelog fix `e6c5d2e3`) fixes the hardcoded umtool paths.
+ - **R** (`ffdeb2cd`) adds `archilyzer source publish`:
+ - a fresh bare clone of `main` is rewritten by git-filter-repo with the operator's scrub rules,
+ then repacked for git's dumb HTTP;
+ - an audit gate refuses a denied literal anywhere;
+ - the raw tree at `/source/tree/` and the tarball are published beside the mirror, and the
+ `/source/` page shows them;
+ - a refusal withdraws the source from `public/` and `out/`;
+ - `deployHomepage` refuses any `out/` it cannot vouch for.
+- **The operator's side is prepared:**
+ - `~/.config/archilyzer/` holds a scrub file (3 rules) and a denylist (3 literals: the user name,
+ the host name, an email address). NEVER print them;
+ - `git-filter-repo` 2.47.0 is installed with pipx;
+ - `source publish --check` exits 0. With `main` at `ffdeb2cd` it would publish `8188e02a7d04`:
+ 2,473 files, 69.8 MB. The parent's first check, on `e56101fdee5d`, gave `20c367613f75`.
+- **Owed, in order** (release-12.md "Rollout"):
+ 0. The operator completes the denylist (real name, handles), then runs `source publish --check`.
+ **No deploy of any kind before this: a Pages preview is public and permanent until it is
+ deleted.**
+ 1. The song link, before any umtool restart.
+ 2. Rebuild and restart :3001 with `~/.local/bin` on PATH. It runs 0.10.0 (`BUILD_ID`
+ `vWCJb87ktCy5akih_pM9X`), which has no source step, no withdrawal and no deploy check.
+ 3. A FRESH `archilyzer build homepage` in the primary. The current `out/` predates `/source`, so
+ the deploy check refuses it.
+ 4. The preview `--preview source` and its live checks.
+ 5. Production.
+ 6–7. What to do if the edge refuses the clone, and if anything private ever ships: delete that
+ deployment.
+ 8. The cut (`release cut editor …`: 2 bullets; export has none) and the worktrees.
+- **Baselines now:** common **2,149**, homepage unit **7**, homepage e2e **36**; editor unit 85,
+ `test:scripts` 185 + 1 and mcp 269 are unchanged.
+
+**Now (2026-09-28, afternoon): release 11 is LIVE as 0.10.0, and Jasolyzer is launched.** The
+rollout ran 12:11–14:05 ([`release-11.md`](release-11.md), "Rollout, as done", every deploy and job
+id): the cut (editor `24c8352e`, export `bf6904e8`), ONE :3001 restart (`BUILD_ID`
+`vWCJb87ktCy5akih_pM9X`, on `main` `e6c5d2e3`), a Settings save (two ran; the second byte-identical — `buildPipeline.mode` dropped, the
+x.com icon `currentColor`), and the six sites by name + the hub + the homepage (the homepage through
+`pnpm ops build-homepage {"deploy":true}` — O4's live proof). **Jasolyzer**
+(https://jasolyzer.pages.dev; `piratesoftware`, 1,889 videos; archives and transcript downloads
+off; vermilion; a draft description the operator may reword) is the sixth site in every footer, the
+hub and the homepage. **The LM chat-only tier is closed as moot** (Legal Mindset's filter has
+`includeLivestreams: true`).
+- **The external drive stalled the rollout** (11:35–12:57): an SMR disk (`sdb`, JMicron JMS578 UAS)
+ under a 10 GB remux timed out, reset and froze :3001 — all four libuv workers blocked in its
+ ext4 reads. No data errors. **Open, for a later slice:** one stalled drive freezes the whole
+ editor. **Operator options:** `echo 180 | sudo tee /sys/block/sdb/device/timeout` (not persistent)
+ and the durable `options usb-storage quirks=152d:a578:u` (a replug; do it with the lanes paused).
+- **Release 12 belongs to a parallel session** (the source mirror, [`source-mirror.md`](source-mirror.md);
+ slice Q merged at `4855f70b`, slice R next). **This session's bundles are release 13**
+ ([`release-13.md`](release-13.md)): W1 editor lows, W2 export/docs lows, W3 the build image (with
+ W3b: container Build all was broken on `main` — `.next` symlink + the image's baked `public/` —
+ and a wrong-site guard now sits before every site deploy), all reviewed SHIP; then one-core
+ Phase 5 (P0, the plan doc `one-core-phase-5.md`, awaits the operator's OK and five rulings).
+- **Worktrees:** the seven `r11-*` stay until release 12's are gone (index-based ports).
+
**Now (2026-09-28, morning): release 11 is merged to `main` in full and NOT rolled out.** The
overnight of 2026-09-28 ([`release-11.md`](release-11.md): every slice's record, then "Integration,
as merged", then "Rollout"). `main` = `eb28a341` + three integration `plans:` commits
@@ -86,6 +199,48 @@ changed at integration. Nothing was deployed, cut, pushed or restarted; :3001 st
8. Optional: rebuild + restart umtool on :3050. Its app changed only in `lib/tools.mjs` (the
shared tool probe) and two test-only variable names; O5's render changes already reach it from
disk (the live :3050 spawns `report-to-video/*`), and an unbranded render is byte-identical.
+- **Release 14 (2026-09-28): HP merged (`bfa1ff3c`, final review SHIP); T1 and H1/H2 built on
+ `r14/two-grounds-headers`, not merged** — `plans/export-header-first-search.md`, record in
+ `release-14.md`. S1 (a clear screen until the first Search) is next, and S2 is a candidate.
+ - **HP** (merged): the homepage's header carries the social row and one theme toggle at every
+ width, through the shared `SocialLinks`; its wordmark's text drops first on a very small
+ screen; Changelog is in the footer; the cards wear the site's wordmark; a social icon is
+ checked by an allowlist on save and at render; `featured` on a social link.
+ - **`r14/two-grounds-headers`** (built, `0f358ee7` + the records):
+ - the final review's Lows (F1, F3, F4, F5, F8);
+ - F8 opts every cwd-derived path op in the three Next apps out of Turbopack's tracing, with
+ the guard widened to `scripts/next-build-trace.test.mjs`;
+ - the charts' foreground separators are withdrawn for a 2 px gap in the surface's colour,
+ the shared charts included;
+ - **T1:** two grounds, Light and Dark, in every app; a stored Sepia reads as Light before
+ paint and is rewritten; each site wears its own accent, and the accent picker is gone;
+ - **H1/H2:** every site's header and the hub's carry the social row and the toggle as one
+ group; an Archilyzer link to the homepage's new `#instances` replaces the sites dropdown
+ and the hub link; Changelog is in the footer; the wordmark's text drops exactly when it
+ does not fit, for any title; the inline nav starts at `lg`.
+ - **Reviewed SHIP AFTER FIXES; the fixes are in:**
+ - stacked areas in the shared charts keep their coloured edge, with the surface gap on
+ stacked bars only (H1);
+ - the growth chart measures thickness at right angles before it takes a gap, so no band is
+ covered; the gaps now read as dashes, and folding the small sites into "Other" or small
+ multiples is the operator's call (M1);
+ - the hub URL texts say what the setting does now (M2);
+ - the Lows (L1, L2, L5, L6, L9, L10); L3 and L7 are left, and L4 needed nothing.
+ - **Owed:** the parent's merge, then the rollout in `release-14.md` ("## Rollout"): the editor,
+ then the homepage (with `#instances`) before any site, then the hub, then the six sites.
+ - **The narrow header, as ruled after the review, is built** (`447ded9a`). Below 520 px every
+ header shows the name and only the links marked **Keep in header on small screens** (none
+ marked → none); from 520 px it shows every link, up to four.
+ - **Re-reviewed SHIP; its findings are in** (`1a2e3342`, `4ea1c495`, `11a33f7a`):
+ - below the switch the export bar's two gaps are 8 px (ruled 2026-09-29): every real title
+ shows in full at 360 px under touch with one marked link (Rekietalyzer at exactly 360);
+ - the wordmark's reservation is a minimum width, so wider-rendering text is never clipped;
+ - the switch is `32.5rem`, so it moves with the reader's text size, and the homepage's
+ wordmark fit is keyed by the wide row's count from it;
+ - a marked link keeps its configured place (ruled as built).
+ - **The operator's settings:** the link to keep in the header is marked in Settings BEFORE any
+ build; the parent does this when the branch lands (`release-14.md`, "## Rollout",
+ precondition 5).
- **Next candidates:** one-core Phase 5 (projects join the core, `plans/one-core.md`); the Diagnostics
cards keeping their retry log (O3's found-and-left); `ChartView.tsx`'s five-slot cycle reaching
`--chart-6` (O2); O5's two wording lows in `svg-faces.mjs` / the README (kerning is not
diff --git a/plans/export-header-first-search.md b/plans/export-header-first-search.md
@@ -0,0 +1,416 @@
+# Release 14 — the social icons within reach, and a clear screen until the first Search
+
+Written 2026-09-28 against `main` `6f03805b`. Self-contained for a fresh session. Rules:
+`plans/tools/implementer-rules.md` (one Opus implementer per slice in a worktree, one read-only
+Opus review, the parent merges; no cut, no deploy and no restart inside a slice). Record file:
+`plans/release-14.md` (NEW, created by the first slice to start, the shape of `release-12.md`).
+Facts below were gathered by two read-only passes on 2026-09-28 and carry `file:line` anchors;
+anything marked UNVERIFIED is a first step for the implementer, not a fact.
+
+**Why 14:** release 12 (the source mirror) is merged and release 13 (`release-13.md`) is a parallel
+session's plan in flight, with its own slice table, merge order and integration gate. These slices
+are not added to it. No `r13/*` branch touches any file this plan owns (checked: `git diff --stat
+main...<branch>` is empty for the search and header components on all five).
+
+## The operator's words
+
+- 2026-09-25: "What if on the export we didn't load all search items until the user hits search,
+ preventing the possibility of an unexpected system overload on re-visit" — shipped in part as
+ release 8 slice E (`00fb567b`): a query restored from localStorage is held, not run.
+- 2026-09-28: "the export feature … where the search doesn't fire / show results until the first
+ time hitting search after load", then: "I was expecting a clear screen (easy to see the footer)
+ when it's actually the default state of viewing an index of all videos like when you hit search
+ with no items."
+- 2026-09-28: "I want the social icons (particularly gumroad) to be more accessible, I'm thinking
+ we move changelog link to footer, theme and dark/light behind a single options modal button, and
+ instead of sites dropdown just link to archilyzer home which lists official instances."
+- 2026-09-28: a link the operator adds is an entry in `settings.json` `socialLinks`, not code; no
+ vendor file goes into the repository, not even as a test fixture.
+- 2026-09-28: the homepage's social links more visible; a change to the social-link schema is
+ allowed where it has a reason (built as slice HP, below).
+- 2026-09-28, on the homepage (slice HP): the Changelog link moves to the footer; Base and Accent
+ move behind a single options button with a gear icon that opens a modal; the gear sits in the
+ social icons' rhythm, and the narrow header carries the icons and the gear in its bar; on
+ Official Instances each site's name uses the bold-lead effect of the sites' own headings, with a
+ slight tint or underline in the site's accent.
+- 2026-09-28, on the homepage chart: light mode has dark lines and dark mode has light lines.
+- 2026-09-28, on the homepage: on very small screens the header keeps its mark and drops the word;
+ the social icons scroll only when they still cannot fit.
+- 2026-09-28: the options menu is dropped for now in favour of a three-way toggle (slice HP on the
+ homepage; slice H2 in the export). A later slice drops the Sepia base in every app and removes
+ the accent picker from the export and editor headers (slice T1, below).
+- 2026-09-28, on the charts: the chart follows standard practice for light and dark grounds; the
+ foreground-coloured separator lines are withdrawn (a 2 px gap in the surface's colour).
+- 2026-09-28 (T1): Sepia is dropped in every app, the editor included; a reader who chose Sepia
+ gets Light; readers no longer pick an accent — each site shows its own, and stored accent choices
+ are ignored.
+- 2026-09-28 (H1/H2): every site's header and the hub's carry the social row with the cycling
+ toggle as the last item of the same group, the homepage's; the Sites dropdown becomes a text
+ link to the Archilyzer home's Official Instances (not on the hub); the header's Hub link goes;
+ Changelog joins the footer after Use with AI; narrow widths behave as the homepage's; the
+ slide-out menu keeps the nav; the editor's header keeps the toggle, with no accent picker.
+- 2026-09-28 (after the branch's review): on narrow screens a site's header keeps the site's NAME
+ and shows only the link(s) marked for the header; the other social links are in the footer.
+ `featured` now means "keep in the header on small screens" (below 520 px only those; from
+ 520 px every link up to four, those kept first). Built on `r14/two-grounds-headers`.
+- 2026-09-29 (after the re-review): below the switch the header bar's two gaps go from 12 to 8 px;
+ type, the mark–name gap and the reserved width's margin stay as they are. The switch is written
+ in rem (32.5rem), so it moves with the reader's text size, and the wordmark's reserved width is a
+ minimum. A marked link keeps its configured place in the wide row (as built).
+- Standing: **no copy** beside any social link: icons with accessible names only.
+
+## Decisions and assumptions
+
+Decided by the operator's words: the four header moves; the clear screen on load.
+
+ASSUMED by the planner (2026-09-28) — each is one line to reverse, and the operator has been told:
+
+| # | Question | Assumed |
+|---|---|---|
+| A1 | A link that carries a query (`qt=`, legacy `q=`) | Still runs on load and shows its results — "a shared link is the visitor asking" (release 8's rule, kept). |
+| A2 | Leaving the search page and coming back within one visit | The listing is still there: the gate is once per page life, not once per mount. |
+| A3 | Which pages get the clear screen | The single-site search page. The hub only if its home renders the same results component (S1 step 0). |
+| A4 | A per-site switch | None. Every export site behaves the same. |
+| A5 | The header's **Hub** link | Dropped with the Sites dropdown; the link to the Archilyzer home covers it. `hubUrl` still parses. |
+| A6 | The footer's social row | Stays, as well as the header's — nothing disappears for a reader who looks there. |
+| A7 | Phones | The icons are visible in the header at every width, not inside the slide-out menu. |
+| A8 | The options modal | None: ruled 2026-09-28, a cycling toggle instead (H2); the accent picker is removed in T1. |
+| A9 | The homepage app's own header | Carries the social row and the theme toggle since slice HP (built), as well as the footer's Elsewhere row; H3 adds only the anchor. |
+| A10 | Deferring the summaries fetch until the first Search | NOT in S1. Measured and written up as S2, a candidate, because `/ask` reads the same data. |
+
+## Dependency graph
+
+```
+HP (homepage social row, toggle, cards + schema, BUILT, merged)
+ ──► T1 ──► H1 (with H3 folded in) ──► H2 BUILT on r14/two-grounds-headers
+S1 (clear screen) ── independent of H*; touches no header file
+S2 (defer summaries) ── after S1, only on the operator's word
+```
+
+As built (2026-09-28): T1, H1 and H2 are one branch, `r14/two-grounds-headers`, off `main`
+`c6b8fc70`, in that order, after the final review's Lows and the chart's gap; H3 is folded into
+H1 (the homepage's anchor ships with the link to it). The record is `release-14.md`, "Branch
+`r14/two-grounds-headers`".
+
+As planned: H1 then H2 stacked on one branch (both rewrite `Header.tsx` and `MobileMenu.tsx`),
+branched after HP merges, with H3 and S1 on their own branches, T1 after HP and before the
+production deploy. As built: the one branch above; S1 is next, on its own.
+
+## Verified facts the implementer must not re-derive
+
+**The header** (`export/app/components/Header.tsx`):
+- Sticky, `min-h-14` (56 px), one breakpoint at `md` (768 px) (`:68-72`;
+ `plans/export-responsive-redesign.md:55-62`).
+- Wide: brand (`:73-80`), nav `hidden md:flex` from `navLinks` (`:50-56`, `:82-92`), then the right
+ cluster `hidden md:flex` (`:94-114`): `SiblingSwitcher` (`:96`), Hub backlink (`:97-106`, shown
+ when `isSite && resolvedHub !== site.siteUrl`, `:35-40`), Changelog (`:107-112`), `ThemeMenu`
+ (`:113`); then, at EVERY width, `ThemeToggle` (`:117`) and the `MobileMenu` trigger (`:118`).
+- `MobileMenu.tsx` is a Radix `Sheet` from the right carrying nav + Offline + Changelog
+ (`menuLinks`, `Header.tsx:61-65`), the Hub backlink (`MobileMenu.tsx:83-92`), the Sites groups,
+ and Base/Accent as native radiogroups (`:124-147`) — it deliberately does not nest `ThemeMenu`'s
+ dropdown ("a focus-trap fight", `:30-35`).
+- The Sites dropdown is **related sites, not official instances**: `SiblingSwitcher.tsx:21-62` over
+ `resolveRelatedSites(site, listSites())` (`common/lib/site.ts:146,203`). The footer renders the
+ same groups in `<nav aria-label="Related sites">` (`Footer.tsx:116-148`), so removing the
+ dropdown loses nothing and orphans no config.
+- Theme: `ThemeMenu.tsx:28-86` — trigger `aria-label="Choose theme"` (`:34`), `menuitemradio`
+ items, two groups: Base (System/Light/Sepia/Dark, `themeConfig.ts:49-54`) and Accent (seven named
+ + the site's own, `siteSchema.ts:116`). `ThemeToggle.tsx:20-39` cycles Base only,
+ `aria-label="Switch to {next}"`. Keys `ytdlp-tb:base`, `ytdlp-tb:accent`; DOM `data-base`,
+ `.dark`, `data-accent` (`ThemeProvider.tsx:100-108`).
+- The no-flash script (`ThemeScript.tsx`, `themeConfig.ts:197-228`) and `ThemeProvider` are
+ untouched by where the controls render: every control reads `useTheme()`.
+- A full Radix dialog wrapper exists and has **no importer**: `common/components/ui/dialog.tsx`.
+ `radix-ui ^1.6.0` is already a dependency of `common`.
+- Changelog route: `export/app/changelog/page.tsx:15-20`. Linked from the header twice (wide
+ cluster and `menuLinks`), from nowhere else.
+
+**The footer** (`export/app/components/Footer.tsx`): row 1 (`:32-75`) eyebrow links Downloads /
+Offline / Use with AI (`:66-74`) — where Changelog goes; row 2 (`:76-114`) credit + social `<ul>`
+(`:97-113`). Social links come from `resolveSocialLinks(site, getSettings())` (`:22`), inlined SVG
+strings sized by `sizeSocialSvg` (`:108`).
+
+**Accessibility, measured in code:** social icons are `w-5 h-5` (20 px) with no padding on the
+link (`Footer.tsx:107`) — under WCAG 2.5.8's 24 px minimum; the links have no focus ring of their
+own; header icon buttons are 32 px (`ThemeMenu.tsx:37`, `ThemeToggle.tsx:31`) and 36 px (the menu
+trigger, `ui/button.tsx:28`).
+
+**The homepage:** Official Instances is `homepage/app/page.tsx:112-122`, a `<section>` with no
+`id` (`:114`), rendered only when `sites.length > 0`. `PROJECT_URL` is `common/lib/project.ts:31`.
+
+**The results area:**
+- `export/app/(workspace)/SiteWorkspace.tsx:22-57` mounts the providers, the search bar and
+ `WorkspaceView` once, above both `/` and `/ask`. `WorkspaceView.tsx:132` renders
+ `<SearchResults/>`, latched once mounted (`:113-116`).
+- `common/components/SearchResults.tsx`: header text branches on `hasActiveQuery` (`:124-137`,
+ "Matching videos (N)" / "All videos (N)"); view toggle (`:155-177`), Copy for AI (`:178-189`),
+ selection toolbar (`:195-247`), chart view (`:249-273`), `browse-hint` (`:276-280`), zero states
+ (`:281-292`), the virtualized list (`:293-305`) whenever `resultGroups.length > 0`.
+- `resultGroups` in browse mode lists every video passing the committed filters whenever
+ `!hasActiveQuery` (`SearchSessionContext.tsx:890-902`) — nothing asks whether the visitor searched.
+- `ranThisPageLife` is a MODULE-level `let` (`SearchSessionContext.tsx:206`): set on a URL query at
+ hydrate (`:1277`), in `commitSearch` unconditionally — an EMPTY commit included (`:1393`) — and
+ in `applySnapshot` (`:1457`). Read once (`:1266`). It survives client-side navigation and resets
+ on reload. It is not reactive.
+- `searchExecuted` is dead: the getter is discarded (`:463-471`), nothing reads it.
+- `commitSearch` (`:1390-1426`) is the one choke point for Enter and the Search button
+ (`SearchBar.tsx:82-96,118-122`).
+- The footer is a sibling of `<main>` in a column (`export/app/layout.tsx:123-138`): the list's
+ height is what pushes it off screen.
+- The listing is client-rendered only (`SearchResults.tsx:1` `"use client"`; 0 rows in
+ `out/index.html`); the sitemap has four URLs and no per-video pages. The clear screen costs
+ nothing in static HTML or crawl.
+- The editor imports none of these components.
+- Specs that load `/` with no query and expect rows at once: `export/e2e/browse-all.spec.ts:32-44`,
+ `workspace-shell.spec.ts:37-39,57,67`, `charts.spec.ts:64-67,96-99,112-115,140-143`. There is no
+ shared "go home and wait for the list" helper. Specs that load with `qt=` or press Search first
+ are unaffected.
+
+**Specs on the header controls:** `theme-accent.spec.ts` (`:32` the trigger by name, `:45-72`
+`menuitemradio`), `theme.spec.ts` (`:64-73` the toggle by `/switch to/i`), `responsive.spec.ts:85-97`
+(the Sheet's `radio` "Sepia"), `brand.spec.ts:81` (`onBase`). No spec opens the Sites dropdown; no
+spec names the Changelog link.
+
+UNVERIFIED: whether `HubHome` renders `SearchResults` (`export/app/(workspace)/page.tsx:12-14`
+branches to it and bypasses `SiteWorkspace`); Back-button scroll restoration into the virtualized
+list; the summaries payload per site (FACTS and the 2026-09-25 session give different figures).
+
+## Slice HP — the homepage shows the social links where they are seen (branch `homepage/social-visible`, BUILT)
+
+Added 2026-09-28 on the operator's ruling above; the record is `release-14.md`, "Slice HP, as
+shipped". What it built, so H1 does not re-derive it:
+
+1. **`common/components/SocialLinks.tsx`** — one row for any header or footer. Props: `links`,
+ `placement: "header" | "footer"`, `className` (on the `<ul>`); nothing site-specific inside.
+ The `<ul>` carries `data-social-links="<placement>"`. Each link is a 36 px key (`size-9`) around
+ the 20 px glyph, 44 px under a coarse pointer (`pointer-coarse:size-11`, a built-in Tailwind v4
+ variant), `text-muted-foreground` → `hover:text-foreground`, a `focus-visible:ring-2
+ focus-visible:ring-ring` ring, and `not-forced-colors:focus-visible:outline-none`, so in forced
+ colours the browser's own outline stays (a box-shadow is not drawn there). `aria-label` and
+ `title` are the link's label; there is no text. Every icon passes the save-time check again
+ before it is inlined (`safeSocialSvg`); one that fails shows the link's label as text.
+2. **The header bound, `headerSocialLinks`** (`common/lib/socialLinks.ts`, pure): at most FOUR —
+ the links marked `featured` when any is marked, else all of them; of those, the last four. The
+ `"header"` placement applies it; the footer shows every link.
+3. **Ids are scoped per copy** (`scopeSvgIds`): an id resolves to the first element carrying it,
+ and a gradient defined inside a `display: none` copy of an icon does not paint in the visible
+ copy. Any header that renders the row twice (one copy per breakpoint) needs this; the
+ component does it, so H1 gets it for free.
+4. **Schema:** the icon's SVG is checked by an allowlist, tag by tag, on save AND at render
+ (`common/lib/socialSvg.ts`; `safeSocialSvg` in `lib/socialLinks.ts` is the render half, and the
+ export footer already goes through it). A refused save names the reason class. A root with a
+ numeric `width`/`height` (unitless or px) and no `viewBox` gets `viewBox="0 0 W H"`.
+ `SocialLink.featured?: boolean` (stored only when true), with a "Show in header" checkbox in the
+ editor's `SocialLinksField`. `sizeSocialSvg` moved to `lib/socialLinks.ts` (re-exported from
+ `settingsSchema`).
+5. **The theme control is one toggle**: `common/components/ThemeToggle.tsx` with `variant="bare"`
+ (dressed as a social key: 36 px, 44 px under a coarse pointer, no border, the same hover square
+ and ring, a 20 px glyph), cycling the base, named "Switch to {next}"; its default rendering is
+ unchanged. The homepage offers no accent control, so `ThemeScript` and `ThemeProvider` take
+ `pinAccent` there (the stored accent is neither read nor removed). An Options dialog was built
+ first and deleted on the operator's ruling; `common/components/ThemeRadios.tsx` stays (the
+ export's `MobileMenu` renders it).
+6. **The homepage's header:** one group — the social row, then the toggle, boxes touching (glyph
+ to glyph 16 px) — in the bar at every width. ≥ `md`: wordmark · nav · group, the nav 32 px
+ before the group's first box. < `md`: wordmark · group, and the four nav links alone on the
+ rule below (they fit 320 px). No link is hidden by width: when the bar (`@container/bar`) is
+ narrower than the full wordmark, a 12 px gap and the group need, the wordmark's text is hidden
+ and the mark stays — a rem threshold per link count and pointer (full wordmark from 348 px with
+ three links and a mouse, 380 px with touch; with four, 384 and 424 px). The last resort is
+ `common/components/SocialScroll.tsx`: the row scrolls in a box, its end first (rtl box, ltr
+ row), the toggle outside it, focus brings a link into view. Changelog is in the footer only
+ (`homepage/app/lib/nav.ts`: `HEADER_NAV`, `FOOTER_NAV`).
+7. **The Official Instances cards** set each site's title with the shared `Wordmark` when the
+ summary carries its `wordmarkLead` (new, optional, still v5), the lead tinted in the site's own
+ accent (`siteAccentColor`). The hub's cards are not touched.
+8. **The homepage's growth chart** parts its strata with 1 px lines in the ground's foreground
+ (`.growth-sep`; `CanvasText` in forced colours), no longer the page background. Homepage-only:
+ the shared charts have no such separator.
+
+## Slice H3 — the homepage's instances anchor (FOLDED INTO H1, built)
+
+Owns `homepage/app/page.tsx`, `homepage/e2e/marketing.spec.ts`, `common/lib/project.ts` (one
+constant), `homepage/CHANGELOG.md`.
+
+1. `id="instances"` on the Official Instances `<section>`, with `scroll-mt` for the sticky header.
+2. `INSTANCES_URL = ${PROJECT_URL}/#instances` beside `PROJECT_URL`.
+3. e2e: `/#instances` scrolls the heading into view. When the section is absent (no sites) the
+ link lands on the top of the page; that is accepted and said in the record.
+4. Gates: tsc, homepage unit, homepage e2e, `archilyzer build homepage --no-source`.
+
+## Slice H1 — the header carries the social row (BUILT on `r14/two-grounds-headers`)
+
+As built, where it differs from the steps below: the inline nav and the Archilyzer link start at
+`lg`, not `md` (at 768 px they, a long title and four icons did not fit); the Archilyzer link is in
+the slide-out menu below `lg`; the wordmark's text drops by a pure-CSS wrap in a one-line clipped
+box, exact for any title, with no measured threshold, its width reserved from Archivo's own metrics
+so the decision is the same before and after the web font loads (review L2); from `lg` the brand
+shrinks first (L1); the footer's row wraps (L6); the footer's `-mx` inset is the shared row's. `header.spec.ts` covers 280–430 px with a short and a long title, 1–4 links, both pointers.
+
+Owns `export/app/components/{Header,MobileMenu,Footer,SiblingSwitcher}.tsx`,
+`export/e2e/{helpers.ts,site-branding.spec.ts,brand.spec.ts,responsive.spec.ts}`, a NEW
+`export/e2e/header.spec.ts`, `export/e2e-hub/official-instances.spec.ts`, `export/CHANGELOG.md`,
+`plans/export-responsive-redesign.md` (the header contract, `:258-260`).
+
+1. **Adopt HP's `SocialLinks`** (`common/components/SocialLinks.tsx`) in the export header
+ (`placement="header"`) and footer (`placement="footer"`), replacing the footer's hand-written
+ `<ul>`. The component is used as it is; a change it needs goes into HP's component, with its
+ homepage spec re-run.
+2. **Tap area and focus** come with it: 36 px keys, 44 px under a coarse pointer, the ring in
+ `--ring`, the browser's outline in forced colours. The export footer's icons change from
+ `hover:text-brand` to the component's `hover:text-foreground`; say so in the changelog.
+3. **The header, wide:** brand · nav · `SocialLinks` · the Archilyzer link · the theme controls
+ (H2). **Narrow:** adopt HP's approach — no link hidden by width; the header first makes room
+ (for a site's header, whatever of the brand can give way while the mark stays), and HP's
+ `SocialScroll` box is the last resort, the row's end shown first. Measure this header (a long
+ `headerTitle`, the toggle, the menu trigger) and record the widths at which the brand collapses
+ and at which anything scrolls. The bound is HP's: at most four links, the `featured` ones when
+ any is marked, else the last four; the footer shows them all.
+4. **Sites dropdown → one link.** `SiblingSwitcher` is deleted. In its place a text link
+ "Archilyzer" to `INSTANCES_URL`, opening in the same tab, with an accessible name that says
+ where it goes ("Archilyzer — official instances"). Also in the slide-out menu, replacing the
+ Sites groups there. On the hub build the link is omitted (the hub lists the instances itself).
+5. **Hub backlink removed** from the header and the slide-out menu (A5). `hubUrl` and
+ `resolveHubUrl` stay; an old `site.json` with the key still loads.
+6. **Changelog** leaves both header places and joins the footer's row-1 links after Use with AI.
+7. **Tests.** New `header.spec.ts`: the header row is visible without scrolling at 360, 390, 768
+ and 1280 px; each link's box is ≥ 24 px (and ≥ 44 px under a coarse pointer emulation,
+ `hasTouch: true`); the focus ring shows; the Archilyzer link's `href`; no `Sites` button;
+ Changelog is in the footer and not in the header; six configured links → four in the header,
+ six in the footer; two marked `featured` → those two. The export e2e's social links come from
+ its own fixture, never the operator's settings. `responsive.spec.ts`: the slide-out menu's
+ contents as changed.
+8. **No copy.** No text beside any icon; the accessible names are the links' labels.
+9. Gates: tsc; common tests; `pnpm --filter export exec next build` (site, fixture site, hub);
+ export e2e — the specs above plus `theme.spec.ts`, `theme-accent.spec.ts`,
+ `related-sites.spec.ts`; hub e2e `official-instances.spec.ts`; screenshots of the header at
+ 360 / 390 / 768 / 1280 px on Light, Sepia and Dark to `~/reports/release-14/shots/`.
+
+## Slice H2 — one theme toggle (BUILT on `r14/two-grounds-headers`)
+
+As built: T1 went first, so the slide-out menu lost its Base radios here with `ThemeRadios`
+(deleted), rather than keeping them until T1.
+
+Owns `export/app/components/{Header,MobileMenu}.tsx`, `export/e2e/{theme,theme-accent,responsive,
+brand}.spec.ts`, `export/CHANGELOG.md`. The editor keeps its own header controls until T1.
+
+1. **The export header's theme control is the cycling toggle** (`ThemeToggle`, `variant="bare"`,
+ built by slice HP) in place of `ThemeMenu` + `ThemeToggle`: one key at the end of the header's
+ group, visible at every width. No options dialog. The accent picker is removed from the export
+ header (the export's `ThemeProvider`/`ThemeScript` take `pinAccent`, as the homepage's do).
+2. The slide-out menu keeps its Base radiogroup (`ThemeRadios`) until T1 drops Sepia and the accent
+ picker everywhere.
+3. **Labels are contracts.** `theme.spec.ts` and `brand.spec.ts` drive the toggle by
+ `/switch to/i`; `theme-accent.spec.ts` drives `Choose theme` and `menuitemradio` and changes
+ with the accent picker's removal. Keep one helper, `chooseTheme(page, { base })`, in
+ `export/e2e/helpers.ts` (the homepage's is `homepage/e2e/helpers.ts`: click the toggle until the
+ base is reached), deriving the cycle from `THEME_BASES` / `nextBase`. Specs that set
+ `localStorage` directly are untouched. The no-flash assertions (`data-theme-ready`) are
+ untouched.
+4. Gates: as H1, plus the homepage e2e `toggle.spec.ts` and `theme.spec.ts`.
+
+## Slice T1 — two grounds (BUILT on `r14/two-grounds-headers`, before H1)
+
+As built: the retired value is spelled once, `RETIRED_BASE` in `themeConfig.ts`; `ThemeMenu` and
+the `pinAccent` option are deleted, and `ThemeRadios` kept the base alone until H2 deleted it.
+
+Owns `common/components/theme*` (`themeConfig.ts`, `ThemeProvider.tsx`, `ThemeScript.tsx`,
+`ThemeToggle.tsx`, `ThemeMenu.tsx`, `ThemeRadios.tsx`), `common/styles/tokens.css`, the three apps'
+`globals.css`, the three headers and the export's `MobileMenu`, and the specs that mention Sepia.
+
+1. **Drop the Sepia base in every app:** the tokens, `THEME_BASES`, `nextBase`, the no-flash
+ script; the legacy `archive` → sepia migration now maps to light, and a stored `sepia` migrates
+ to `light`.
+2. **Remove the accent picker** from the export and editor headers and the slide-out menu; a stored
+ accent is ignored (as the homepage's `pinAccent` does).
+3. **Specs updated.** The parent's count of what mentions Sepia: 40 files, 154 lines, 16
+ specs/tests. The homepage's `toggle.spec.ts` derives its cycle from `THEME_BASES`, so it needs
+ no change beyond the base's removal.
+4. Gates: tsc; common tests; the three apps' builds; the three e2e suites.
+
+## Slice S1 — a clear screen until the first Search (branch `r14/first-search`)
+
+As built (2026-09-29, on the rulings of that day): S1 only, the summaries download not deferred
+(S2 stays a candidate). A link carrying only a filter (`tg=`, a share link, the legacy filter keys),
+like one carrying `qt=`/`q=`, runs on load and shows its results. It runs no query, so a stored
+one stays held (release 8), on that load and on later mounts in the visit: the gate and the hold
+are two page-life flags (review M1). Step 0: the hub renders
+`SearchResults` (`HubHome` → `TranscriptSearch`), so it has the clear screen and `e2e:hub` joined
+the gate; Back restores the listing (measured in `release-14.md`, "Slice S1, as shipped"). Beyond
+the four specs this section owns, the ones that read the listing at load were `responsive` (two
+tests), `tag-chips` (three) and `e2e-hub/federated-search` (eight).
+
+Owns `common/components/{SearchSessionContext,SearchResults,SearchBar}.tsx`,
+`export/app/(workspace)/WorkspaceView.tsx`, `export/e2e/{browse-all,workspace-shell,charts,
+restore-no-refire}.spec.ts`, a NEW `export/e2e/first-search.spec.ts`, `export/e2e/helpers.ts`
+(one helper), `export/CHANGELOG.md`.
+
+0. **First, settle the two unknowns** and write them in the record: does `HubHome` render
+ `SearchResults`; what does the Back button restore into the list. If the hub shares the
+ component, the behaviour applies there too and `e2e-hub` joins the gate; if not, the hub is
+ left and said so.
+1. **One reactive flag.** `searchedThisPageLife` in the session context, initialised from the
+ module-level `ranThisPageLife` and set wherever it is set (`:1277`, `:1393`, `:1457`). Delete
+ the dead `searchExecuted` state. The module variable stays the source of truth across remounts
+ (A2).
+2. **The gate is in the view, not the data.** `SearchResults` renders NOTHING until the flag is
+ true: no header count, no view toggle, no Copy for AI, no selection toolbar, no chart, no list,
+ no `browse-hint`. `resultGroups` is still computed (S2 is where the work is saved). The page's
+ own intro (`export/app/(workspace)/page.tsx:7-37`: the transcript count and New since last
+ visit) stays.
+3. **What the visitor sees:** the search bar, the intro, the footer. The bar's existing line
+ "Press Enter or click Search to apply" is shown in this state too, so the screen says what to
+ do in words that already exist. No new copy.
+4. **An empty Search** shows "All videos (N)" and the listing, exactly as today. A `qt=` URL runs
+ and shows on load (A1). A restored query stays held (release 8) — and now shows a clear screen
+ instead of the browse listing under it.
+5. **Filters before the first Search:** changing a filter does not reveal the listing; the hint
+ line already covers it (`filtersDirty`).
+6. **Tests.** New `first-search.spec.ts`: on load no `results-summary`, no `[data-card-header]`,
+ the footer in the viewport at 1280×800 and 390×844; Enter on an empty box shows "All videos";
+ the Search button does too; `/` → `/ask` → `/` keeps the listing; a reload clears it; `qt=`
+ shows results on load; a restored query shows the clear screen and the filled form. The three
+ affected specs gain `showAll(page)` (one helper: press the Search button, wait for
+ `results-summary`) where they relied on the listing at load; `browse-all.spec.ts`'s first test
+ is rewritten to assert the clear screen and then the listing.
+7. Gates: tsc; common tests; `pnpm --filter export exec next build`; the FULL export e2e suite
+ (the blast radius is a shared component — three specs are known, the suite finds the rest);
+ hub e2e if step 0 says so; `e2e:2origin` from the worktree only.
+
+## Slice S2 — CANDIDATE: fetch the summaries on the first Search
+
+Not scheduled. It is the other half of the 2026-09-25 request ("didn't load all search items
+until the user hits search") and needs the operator's word, because it changes `/ask`.
+
+- `SingleSiteDataProvider` calls `useSummaries("")` on mount (`SearchDataContext.tsx:148-155`), and
+ `useSummaries` fans out EVERY page eagerly (`summariesCache.ts:19-71`, 1,000 records a page).
+- The cheap manifest feeds the Filters channel list (`SearchDataContext.tsx:165-183`) and must stay
+ eager; the fallback at `:176-183` derives channel names from the pages when the manifest has none.
+- `/ask` grounds on the same summaries (`useAskChat.ts:149-151`) under the same shell, so an
+ `/ask`-first visitor needs them without ever pressing Search.
+- Step one of S2 is a measurement, per site, of the bytes and requests a plain visit costs today
+ and would cost after — the figures on record disagree.
+
+## Rollout (parent)
+
+Nothing here deploys inside a slice. After the merges: one export release cut, a rebuild and
+deploy of every site and the hub (the header is in every build), and the homepage deployed FIRST
+so `#instances` exists before any site links to it. HP adds an editor cut and a :3001 rebuild and
+restart, for the "Show in header" checkbox and the normalizer's viewBox rule, which are in the
+editor's save path. The homepage deploy runs the source publish step of release 12 — its
+denylist must be complete (`release-12.md`, "Rollout", step 0). Live checks: the header row visible at 390 px on one site; each icon's `href`; the Archilyzer link lands
+on Official Instances; a plain visit shows the footer without scrolling; a `qt=` link still shows
+results. An HTML runbook at `~/reports/release-14/RUNBOOK.html`.
+
+## Risks
+
+- **Two taps for the ground** (H2.4). The fallback is named.
+- **A first-time visitor sees no videos.** The site's front page becomes a search box over a
+ count. That is the operator's choice; the hint line is the only instruction.
+- **Icons in a 360 px header** compete with the brand's title: a long `headerTitle` wraps
+ (`min-h-14` allows it). The screenshots at 360 px are the check; the bound of four keeps the row
+ bounded.
+- **An operator's icons in the header** are more prominent than in the footer. Each stays as the
+ operator pasted it (normalized, never redrawn), unlabelled and only a link.
+- **Specs outside the known three** may lean on the listing at load through a hydration wait; the
+ full-suite run in S1's gate is how they are found.
diff --git a/plans/export-responsive-redesign.md b/plans/export-responsive-redesign.md
@@ -258,6 +258,10 @@ in `globals.css:3` means `@utility`s defined there are usable from `common/`.
- `export/app/components/Header.tsx:43`: `h-14 … flex-wrap` → `min-h-14 flex-nowrap`. Nav `:51-77` → `hidden md:flex` + add "Ask AI" → `/ask/`. Right cluster `:79-99`: SiblingSwitcher, Hub `<a>`, Changelog, ThemeMenu → `hidden md:…`; `ThemeToggle` stays visible at every width (theme.spec `/switch to/i`; theme-family.spec "Choose theme" is served by the inline ThemeMenu at 1440).
- New client `export/app/components/MobileMenu.tsx`: props `{ links:{href,label}[]; sites: SwitcherGroup[] (type from SiblingSwitcher.tsx:18-19); hubUrl?: string }`. `<Sheet>` + `<SheetTrigger asChild><Button variant="ghost" size="icon-sm" aria-label="Open menu" className="md:hidden">`, `<SheetContent side="right">` with a `<SheetTitle>` (Radix warns without one); every `<Link>` wrapped in `<SheetClose asChild>` (Header lives in the root layout and never remounts on client nav). Theme lists via `useTheme()` (`ThemeProvider.tsx:35-38`) + `THEME_FAMILIES`/`THEME_MODES` from `themeConfig` as native radio groups — do not nest `ThemeMenu`'s DropdownMenu in the dialog. Offline link gated like `Footer.tsx:56` (`site.pwa`).
- `export/app/components/Footer.tsx:29` → `flex flex-col gap-3 sm:flex-row sm:items-center sm:justify-between pb-safe`; `:82` drop `whitespace-nowrap`. Leave the anchor naming (`"Built with"` outside the `<a>`) and `nav aria-label="Related sites"` untouched.
+- **Superseded, release 14 (H1/H2, `r14/two-grounds-headers`):** the header's right cluster is now
+ the social row and the theme toggle as one group at every width; the SiblingSwitcher, the Hub
+ link, Changelog and ThemeMenu are gone from it (Changelog is in the footer); the inline nav and
+ an Archilyzer link start at `lg`; MobileMenu holds only the nav and that link.
### Slice 4 — Results (M, 1 spec edit)
- `common/components/SearchResults.tsx` card header `:470-538`: `flex items-stretch` → `flex flex-col sm:flex-row`; title `:504` `truncate` → `line-clamp-2 sm:line-clamp-none sm:truncate`; meta `:521-526` → `basis-full sm:basis-auto`; Ask `:528-537` `border-l` → `max-sm:border-t`. Preserve `data-result-slug`/`data-card-header` (`:460-461`), `data-card-open` (`:487`), checkbox aria-label (`:480`).
diff --git a/plans/release-11.md b/plans/release-11.md
@@ -1863,3 +1863,68 @@ integration tip (above); the live :3001 editor still runs `BUILD_ID` `S07zTu3MKT
already reach the live :3050 from disk, and an unbranded render is byte-identical.
9. **Afterwards:** remove the seven merged `r11-*` worktrees when nothing runs (it re-sorts the port
blocks; `diet-series` stays).
+
+## Rollout, as done (2026-09-28, 12:11–14:05 local)
+
+Run by the parent from the primary checkout with the scripts in
+`~/reports/overnight-2026-09-28/scripts/`, on the operator's plan of 2026-09-28 (roll out release 11
+as 0.10.0 and launch Jasolyzer, exports off). Logs: `~/reports/overnight-2026-09-28/tmp/r11-*`.
+
+1. **Cut** (`archilyzer release cut all next-minor --commit`): editor **0.10.0** `24c8352e` (8
+ bullets), export **0.10.0** `bf6904e8` (9). The homepage changelog keeps its own convention.
+2. **The external drive stalled first.** From 11:35 the auto-download lane's remux of a 10.3 GB
+ `nuxanor-kick` VOD, reading and writing the same SMR disk (Seagate ST2000DM008 in a JMicron
+ JMS578 UAS enclosure, `sdb`), drove it into 30 s command timeouts, UAS aborts and USB resets
+ (`hostbyte=DID_ERROR cmd_age=30s`; the `I/O error` lines are failfast READAHEAD requests; no
+ medium errors, no ext4 errors). All four of the live editor's libuv threadpool workers blocked in
+ ext4 reads on it, so **:3001 answered nothing** (not even `/api/view/pulse`) from ~12:10 until
+ the remux finished at 12:56:39. Every one of the six sites has a channel on that drive (11
+ channels), so the rollout waited. The operator checked the cable and ran `smartctl` (the identify
+ command failed: the drive was busy). Nothing was killed. Found and left, for a later slice: one
+ stalled drive freezes the whole editor.
+3. **:3001 restart** on `main` `e6c5d2e3` (the cut + release 12's slice Q, whose editor-side changes
+ are comments and an `[Unreleased]` bullet): `r11-build.sh` BUILD_OK 97 s, `BUILD_ID`
+ `S07zTu3MKTjHJa9eDM1GF` → `vWCJb87ktCy5akih_pM9X`; `r11-restart.sh` `/` 200 after 3 s, `/sites`
+ has the Homepage section, 0 ZodError, boot `re-queued 0, cancelled 29 (sync 26, stale 3)`. The
+ md5 sweep (81 files: settings, every `site.json` and `config.json`, `homepage.json`) was identical
+ before, before the restart and after boot. `smoke.sh`: 20 OK lines, `PULSE_OK` ×2,
+ `DEFERRED_OK`, umtool OK; `SMOKE_FAIL=1` from one `PAIR_DIFF`
+ (`/api/auto-queue/status` vs `/api/view/autoQueueStatus`, 10 bytes of 517 KB, the two calls 98 s
+ apart) that is live drift — a `hasanabi` pick and a `HasanAbiVODs3` queue between the calls.
+ The OOM killer fired at 13:01:11 on the OLD editor (already exiting under TERM, anon-rss 0) and a
+ worktree e2e server; the new editor was unaffected.
+4. **One Settings save** (headless, no change): `buildPipeline.mode` is gone and the x.com icon's
+ `fill="white"` is `currentColor`. Two saves ran — the first script's wait resolved on Next's
+ empty route-announcer alert and closed early, but its save still landed (13:06:01); the second
+ rewrote byte-identical JSON. md5: only `settings.json` changed.
+5. **Jasolyzer:** Pages project `jasolyzer` created (`wrangler pages project create jasolyzer
+ --production-branch main`); `/sites/jasolyzer` saved headless with `cloudflareProject`
+ `jasolyzer`, `siteUrl` `https://jasolyzer.pages.dev`, archives off, transcript downloads off and
+ the draft `siteDescription` "Search through the streams of our favorite game dev, Jason 'Pirate
+ Software' Hall" (the operator may reword it). md5: only `transcripts/sites/jasolyzer/site.json`.
+6. **Deploys** (`pnpm ops build-deploy --json '{"siteIds":[six],"skipArchives":true}'`, the host
+ path, by name — `r11-sites6.sh`): SITES_OK in 2,872 s, no site skipped.
+
+ | Site | Job | Deploy |
+ |---|---|---|
+ | jasolyzer (first deploy) | `01M3MFRJ9RVDYPERPY22C3MD49` | https://fa021d5b.jasolyzer.pages.dev |
+ | jeralyzer | `01M3MFRJDSJV341S08JG0PV21E` | https://9bbd2c88.jeralyzer.pages.dev |
+ | anilyzer | `01M3MFRJDZRQWW56APC4NQGVBC` | https://1a6a795d.anilyzer.pages.dev |
+ | bonnellyzer | `01M3MFRJEF4K2GJSE0D06FBPYH` | https://bb5cb426.bonnellyzer.pages.dev |
+ | hasanalyzer | `01M3MFRJEHZ53S9V6DDN9QVNX1` | https://2cd0d19f.hasanalyzer.pages.dev |
+ | rekietalyzer | `01M3MFRJEVNHMNHKQ3KH8BEEZY` | https://1ca2acf8.rekietalyzer.pages.dev |
+ | hub (`build-hub {"deploy":true}`) | `01M3MJG5DG0ZGH16VYWWKCT7QR` | https://c215ae89.archilyzer-hub.pages.dev (84 s) |
+ | homepage (`build-homepage {"deploy":true}` — **O4's live proof**) | `01M3MJK8NV8N1M3PPGQC6F3V0X` | https://41bef909.archilyzer.pages.dev (73 s) |
+
+7. **Live checks, all pass:** every `/changelog` of the eight surfaces reads 0.10.0; the
+ forced-colours rule is in every site and the hub; the x.com footer icon is `currentColor` (it
+ follows the footer colour); Jasolyzer serves `data-accent="vermilion"`, a footer listing the
+ five siblings and a `corpus.json` of 1 channel / 1,889 videos; `hub-summary.json` has 6
+ instances; the homepage's summary has 6 sites and draws Jasolyzer on `var(--chart-6)` (legend and
+ area); the five 410'd Rumble videos publish `isDeleted: true` (Jeralyzer `v4taj2f`, `v4uooni`,
+ `v4we0zh`, `v4xpb46` — URL slugs `v4vriou`, `v4x5o1l`, `v4yqask`, `v501kfc`; Rekietalyzer
+ `v7btopk` — slug `v7e07us`).
+8. **Closed as moot:** the Legal Mindset chat-only tier (its filter has `includeLivestreams: true`,
+ so the tier never changes what is kept).
+9. **Still owed:** re-upload the YouTube picture, watermark and banner (since release 10); the
+ seven `r11-*` worktrees stay until release 12's worktrees are gone (index-based ports).
diff --git a/plans/release-12.md b/plans/release-12.md
@@ -0,0 +1,1005 @@
+# Release 12 — the source mirror: a read-only git clone on the project site
+
+`main` at `90c7f776` (release 11 merged in full and NOT rolled out: the cut, the :3001 restart, a
+settings save and the deploys are still owed by the operator, `release-11.md` "Rollout"). Release
+12 is the operator's ask of 2026-09-27: the repo mirrored on the project site
+(https://archilyzer.pages.dev) as static, read-only files — `git clone
+https://archilyzer.pages.dev/source/archilyzer.git` over git's dumb-HTTP protocol, a raw tree at
+`/source/tree/`, the tarball regenerated per deploy, and a gate that refuses to publish a denied
+literal. Plan: [`source-mirror.md`](source-mirror.md) (written 2026-09-27 in plan mode, re-checked
+2026-09-28 against `90c7f776`; it entered the repo as slice Q's first commit, never as a direct
+commit to `main`). Rules: `plans/tools/implementer-rules.md`, with the commit trailer this release's
+prompts give.
+
+**The operator's standing choices** (`source-mirror.md`, "Decisions"; not re-opened):
+- **Mirror = `main` only.** `/home/user → /home/user` is scrubbed in file contents AND commit
+ messages; the `Co-Authored-By` and `
+- **The private repo's history is never rewritten.** The mirror is generated by `git-filter-repo`
+ on a fresh bare clone at every homepage build; its commit ids differ from the private repo's.
+- **Operator-private inputs live outside the repo**, in `${ARCHILYZER_CONFIG_DIR ??
+ ~/.config/archilyzer}/`. Nothing carrying the username is in code.
+- **Working alongside the parallel session:** nothing is edited in the primary checkout; each slice
+ has its own worktree, and the parent merges with `git merge --no-ff` only on a clean tree.
+
+## The slices
+
+| Slice | Branch | What | Owns |
+|---|---|---|---|
+| Q | `r12/paths-fix` | Fix-forward the hardcoded paths: umtool's `CHANNELS_DIR` from the repo root, `VIDEO_ROOT` and `SONG_DATA` from the home dir (XDG for the song data), the 20 one-off `umtool/song/*.sh` run logs deleted, the machine paths stripped from three tracked umtool manifests, `/run/media/user` out of `common/` | `umtool/**`, `WORKTREES.md`, `common/controller/storageLocations{,.test}.ts`, `common/lib/storageVolumes{,.test}.ts`, `common/views/storage.test.ts`, `common/lib/envVars.ts` (umtool defaults only) + `ENVIRONMENT.md` |
+| R | `r12/source-mirror` | `archilyzer source publish` (filter-repo scrub, repack, dumb-HTTP files, the audit gate, the raw tree, the tarball) run by `buildHomepage`; the `/source/` page; doctor's source block; the docs; `create-archives.sh` deleted | `common/publish/{source,sourceAudit,sourceTree}.ts` + tests, `common/lib/sourceManifest.ts`, `buildHomepage`, `common/bin/{archilyzer,doctor}.ts` + tests, `common/lib/{paths,envVars}.ts` (new variables), `homepage/**`, `.gitignore`, the docs |
+
+**Order:** Q lands first; R branches from `main` after Q merges. The shared files are
+`editor/CHANGELOG.md` (`[Unreleased]`) and this record.
+
+## Record
+
+### Slice Q, as shipped — fix-forward the hardcoded paths (2026-09-28)
+
+Branch `r12/paths-fix` off `main` `90c7f776`, worktree `/home/user/Projects/r12-paths-fix` (ports
+3901/3911, umtool e2e 3951/3952), one Opus implementer. Plan: [`source-mirror.md`](source-mirror.md),
+"Slice Q", steps 1–9. Why: slice R publishes the repo, and its gate refuses any denied literal. The
+Unix username was in code as machine paths (umtool's defaults, 20 run-log scripts, three tracked
+manifests) and in path examples (`/run/media/user`). Scrubbing the published copy is R's job. Q
+fixes the source, so the code no longer depends on one machine's home directory. Scratch files
+`q-*` in the job's `tmp`.
+
+**What shipped.**
+- **Step 1: `CHANNELS_DIR` comes from the checkout.** `umtool/lib/paths.mjs` gains
+ `findRepoRoot(start)` and `REPO_ROOT = findRepoRoot(process.cwd())`. The walk goes up to
+ `pnpm-workspace.yaml` and falls back to the cwd's parent. It starts from the cwd, not
+ `import.meta.url`, which is the `SONG_CODE` rule; the comment says why. `CHANNELS_DIR` is
+ `CHANNELS_DIR ?? $TRANSCRIPTS_DIR/channels ?? <REPO_ROOT>/transcripts/channels`, resolved.
+ `lib/projects/report.mjs`' `GLOBAL_CHANNELS_DIR` is `() => CHANNELS_DIR`, imported from
+ `../paths.mjs`, so there is one definition. `cues.mjs` is untouched.
+- **Step 2: `VIDEO_ROOT`.** `song/spec.mjs` and `song/video-dir.mjs` use `process.env.VIDEO_ROOT ??
+ path.join(os.homedir(), "reports", "quartering-uh-song", "videos")`, with `import os` added.
+- **Step 3: `SONG_DATA`.** `song/paths.mjs` resolves `SONG_DIR ?? ~/.local/share/archilyzer/song`
+ through `realpathSync`. A path that does not exist has no realpath, so it is **used as given**:
+ a missing default still gives a `SONG_DATA`, which every reader finds empty. The header comment
+ says the symlink is the supported way to keep the data where it is. It also says why the
+ realpath: `SONG_SCRATCH = dirname(SONG_DATA)` keeps pointing at the real job dir.
+- **Step 4: the 20 run logs are deleted** with `git rm`:
+ - `backfill-build`, `mk-fatal-finish{,2,3}`, `mk-rebuild`;
+ - `ms2-{bg3,drums,full,jerbg,play-rebuild,triangle}`;
+ - `pk2`, `pk3`, `pk-v12`, `pkmn-{rebuild,video}`, `rpg-remake-v5`, `rpg-short`, `vshort`,
+ `yoshi-rebuild`.
+
+ That was every `.sh` in `umtool/song/`. Nothing enumerated or ran them. `lib/jobs.ts`' header
+ and timeout doc, and `BuildChain.tsx`'s comment and on-screen note, now say it in the past tense:
+ "one-off shell run logs that hardcoded their paths … not in the tree, and not runnable from
+ here". No spec asserts that note's text.
+- **Step 5: `um-manifest.json` loses `vid`.** The one-off below stripped all 1,896 `vid` values
+ from 3,609 items. `build-um.mjs` now writes `[...merged.values()].map(({ vid: _v, ...rest }) =>
+ rest)`, with a comment. The page keeps `vid` in memory: `META` is `JSON.stringify(items)`. The
+ item push at `:224` is unchanged.
+- **Step 6: relative paths in the thumb manifests.** The one-off below made `out` relative to
+ `~/reports/quartering-uh-song` and `bg` relative to the song data dir: 18 paths in
+ `thumb-manifest.json` and 8 in `thumb-accepted.json`. `make-thumb.mjs` writes
+ `out: relTo(SONG_REPORTS, path.resolve(OUT))` and `bg: relTo(SONG_DATA, path.resolve(BG))`, with a
+ comment naming the two roots. `path.resolve` is there because both are CLI arguments and could
+ be cwd-relative.
+- **Step 7: path examples name no user.**
+ - `WORKTREES.md:119` → `cwd /path/to/checkout/editor`.
+ - `fit-hooks.mjs:93` → `home/<user>`.
+ - `/run/media/<user>/` in the comments of `storageLocations.ts`, `storageLocations.test.ts` and
+ `storageVolumes.ts`.
+ - `/run/media/operator/` in the fixtures of `storageVolumes.test.ts` and `views/storage.test.ts`,
+ on both sides of every assertion.
+ - The one `editor/CHANGELOG.md` line (see "Changelog").
+- **Step 8 had nothing to do.** `common/lib/envVars.ts` declares none of `SONG_DIR`, `VIDEO_ROOT`
+ or `CHANNELS_DIR`. Its header says "umtool's own knobs are NOT here", and `envVars.test.ts:17`
+ keeps umtool out of the scan. `ENVIRONMENT.md` is unchanged, and `--check` exits 0. The new
+ defaults are documented where umtool's knobs live: a table under "Environment" in
+ `umtool/docs/cli.md`.
+
+**Beyond the plan (all in `umtool/**`):**
+- **`SONG_REPORTS` moved into `song/paths.mjs`, beside `relTo`, and `lib/paths.mjs` re-exports
+ it.** `make-thumb.mjs` needs it, and the song scripts import only siblings. The e2e fixture copies
+ `song/*.mjs` into `.e2e-song/code/` and runs them from there, where `../lib` does not exist. There
+ is still one definition, and the default is unchanged.
+- **`relTo(root, p)`.** An absolute `p` inside `root` becomes relative to it. Anything else comes
+ back unchanged. Both sides are compared as given and through the realpath of their deepest
+ existing part, so a path spelled through the `~/.local/share` symlink counts as inside the
+ realpath'd `SONG_DATA`, even for a file not written yet.
+- **`accept-thumb.mjs` records `out` through `relTo(SONG_REPORTS, …)`.** It is the other writer of
+ `thumb-accepted.json`, and `/api/browse/thumbs` hands it an ABSOLUTE `file`. Without this, the
+ first accept from the page would have written a machine path back into the tracked file. The
+ route's comment is updated to match. A relative CLI argument is recorded as typed, as before.
+- **The e2e fixture and a new assertion.** The fixture's accepted `alpha-c` carries the relative
+ `out`, while its run log keeps the absolute form, so the suite reads both. deck.spec's
+ "serves the ACCEPTED cover" now resolves a relative `out`. The disjoint-accept test asserts the
+ CLI recorded `alpha-b` as `thumbs/alpha-b.jpg` (see "They bite").
+- **Comments that described the old default:**
+ - `song-capabilities.mjs` and `make-fixture.mjs` (twice) said "the job temp dir … which on most
+ machines no longer exists";
+ - `browse.ts` said "`out` is an ABSOLUTE path".
+
+**Corrections to the plan:**
+- **The common baseline is 2,114, not 2,112.** O6c added 2 after the plan's re-check
+ (`release-11.md`, O6c gates).
+- **Step 8 was a no-op**, as above.
+- **Step 4's `video-dir.mjs:137` is now `:139`**, because step 2 added two lines above it.
+
+**The one-off commands** (the scripts are in the job's `tmp`; what each does is stated here so it
+can be redone):
+- **Step 5:** `node $T/q-strip-vid.mjs umtool/song/um-manifest.json`. It runs `doc.items =
+ doc.items.map(({ vid, ...rest }) => rest)` and writes `JSON.stringify(doc, null, 1)` with no
+ trailing newline, which is the writer's format; a parse and re-stringify of the old file was
+ byte-identical. It printed `stripped vid from 1896 of 3609 items`.
+ - `git diff --numstat`: **0 added, 1,896 deleted**.
+ - Every deleted line is `"vid": "file://…"`.
+- **Step 6:** `node $T/q-rel-thumbs.mjs "$HOME/reports/quartering-uh-song"
+ "$HOME/.claude/jobs/efbe67a7/tmp/song" umtool/song/thumb-manifest.json
+ umtool/song/thumb-accepted.json`. For every entry, an absolute `out` becomes relative to the
+ first root and an absolute `bg` relative to the second. A path outside its root is refused, not
+ guessed. The output format is the same as step 5. It printed `18 paths made relative` and
+ `8 paths made relative`.
+ - `git diff --stat`: **26 insertions, 26 deletions**, the `out`/`bg` lines only.
+
+**Verified: the same files and the same labels.** Run from `umtool/` with
+`SONG_DIR=/home/user/.claude/jobs/efbe67a7/tmp/song`, before and after the rewrite:
+- **`q-thumbs-check.mjs`** prints `resolveInRoots(out)`, `labelFor` and `browse.thumbFor`'s
+ `SONG_REPORTS`-relative path for all 13 entries, plus the absolute `bg`. The two outputs are
+ identical except the `CHANNELS_DIR` line, which is step 1: the worktree's own
+ `transcripts/channels` instead of the primary's. Every `out` still resolves to
+ `/home/user/reports/quartering-uh-song/thumbs/<x>.jpg`, and all 13 exist.
+- **`q-thumbview.mts` runs the app's own readers** (`lib/thumbs.ts` `thumbView` and `lib/browse.ts`
+ `thumbFor`) for yoshi, mario-rpg, metal-slug, mortal-kombat and pokemon. For "before", it pointed
+ `SONG_CODE_DIR` at `90c7f776`'s two manifests. The output is **identical**: accepted names and
+ labels, every candidate's label, the clash strings, `check` and the poster path.
+- **The one visible difference:** a candidate's `file` is now `thumbs/<x>.jpg` instead of the
+ absolute path, which changes the bench's grey text. The accept route resolves it to the same
+ absolute file.
+- **The `SONG_DATA` default:** no env gives `/home/user/.local/share/archilyzer/song`, used as
+ given because the path is absent. `SONG_DIR=<a symlink in scratch to the job dir>` gives the job
+ dir, and `SONG_SCRATCH` = `…/efbe67a7/tmp`.
+- **Nothing was created** under `~/.local/share` or `~/.config`.
+
+| sha | what |
+|---|---|
+| `c3177eeb` | `plans:` the source mirror plan (verbatim) and this record file |
+| `de1ead5d` | `umtool:` `REPO_ROOT` + `CHANNELS_DIR` from the checkout; `GLOBAL_CHANNELS_DIR` returns it |
+| `d4d549c7` | `umtool:` `SONG_DATA` (XDG default, realpath, missing → as given) and `VIDEO_ROOT` defaults; fixture comments; `docs/cli.md` defaults table |
+| `292f7a96` | `umtool:` the 20 run-log scripts deleted; `jobs.ts` / `BuildChain.tsx` in the past tense |
+| `4d12652e` | `umtool:` `um-manifest.json` without `vid` (one-off) and `build-um.mjs` strips it on write |
+| `73c39376` | `umtool:` thumb manifests relative (one-off); `relTo`; `make-thumb` / `accept-thumb` write relative; `SONG_REPORTS` into `song/paths.mjs`; the fixture's relative accepted cover |
+| `0b28a7a4` | `common, docs:` `/run/media/<user>` / `operator`, `WORKTREES.md`, `fit-hooks.mjs` |
+| `29dc507c` | `changelog:` one `[Unreleased]` bullet; the released line's path example |
+| `1059befb` | `e2e:` deck.spec pins the relative `out` an accept records |
+| `57ce980a` | `plans:` this record; `source-mirror.md`'s "As shipped" note under Slice Q |
+| `75327c6f` | `changelog:` review L1 — the bullet leads with the full `mkdir -p … && ln -s …` command |
+| `0416881f` | `umtool:` review L3 — `ThumbEntry` declares `bg` (relative to the song data dir; `dataFile()`, never `resolveInRoots`) |
+| _this_ | `plans:` the review, and the operator step corrected (review I1) |
+
+**Gates** (from the worktree root; logs `$T/q-*.log`):
+- **The grep gate** `git grep -c -i 'user' HEAD -- . ':!plans/'` is **empty** (exit 1) at
+ `29dc507c` and at `1059befb`.
+- **tsc** was clean before every commit (34–48 s; `q-tsc-{0..7}.log`, all seven packages).
+- **`node --check`** passed on all 11 changed `.mjs`.
+- **Unit:**
+ - common **2,114/2,114**, the same count, with tests edited and none added (the three edited
+ files 39/39);
+ - `test:scripts` **185 + 1 skipped**;
+ - editor unit **85/85**;
+ - mcp **269/269**.
+- `pnpm archilyzer docs env --check` exits **0**.
+- `pnpm --filter umtool exec next build` is **ok** (20 s).
+- **The `lib/paths.mjs` print** from `umtool/` gives `/home/user/Projects/r12-paths-fix/transcripts/channels
+ /home/user/.local/share/archilyzer/song`: this checkout's corpus and the XDG path.
+- **umtool e2e** (`faces.spec.ts deck.spec.ts clip-bench.spec.ts`, with
+ `SONG_DIR=~/reports/quartering-uh-song/data`; the queue was free each time):
+ - On `29dc507c`: **67 passed, 8 skipped, 0 failed** (2.0 min).
+ - deck.spec alone with the new assertion: **14 passed, 2 skipped** (20.5 s).
+ - On `1059befb`: **67 passed, 8 skipped, 0 failed** (1.6 min).
+ - The skips are the song-data specs. This machine's `data/` has `cand2/` but no `wav48/`, `asr/`
+ or `media/`, and `make-fixture` says so.
+ - The fixture sets `CHANNELS_DIR` and `SONG_DIR` on both servers, so the green run is also the
+ proof that env still wins.
+- **Numbers tool:** none.
+
+**They bite:**
+- deck.spec's disjoint-accept test fails with `90c7f776`'s `accept-thumb.mjs` swapped in: **1
+ failed**. It expected `thumbs/alpha-b.jpg` and received
+ `…/umtool/.e2e-song/reports/thumbs/alpha-b.jpg`. A trap restored the file, and the tree was clean
+ afterwards (`q-e2e-deck.log`).
+- The default changes have no unit tests; umtool has none for `lib/`. They are verified by the
+ prints above.
+
+**Found and left:**
+- **Historical mentions of the deleted scripts stay, as the plan says:**
+ - `debox-bg.mjs:24` (`pk3.sh`);
+ - `hush-head.mjs:47` and `pick-take.mjs:26` (`mk-fatal-finish3.sh`);
+ - `video-dir.mjs:5,139` (`pkmn-video.sh`).
+- **`cues.mjs`' `DEFAULT_CHANNELS_DIR` is evaluated inside the app bundle too.** `report.mjs`
+ imports `build-video` and `resolve-windows`, which import `cues.mjs`, and there its
+ `import.meta.url` walk points into `.next`. Only their CLI entry points read it, so nothing is
+ wrong today. It is left alone by the plan.
+- **umtool now reads `TRANSCRIPTS_DIR`,** a declared core variable. `envVars.ts`' `readBy` for it
+ does not name umtool, which is out of the registry's scope. I did not change it; it is not this
+ slice's file.
+- **`REPO_ROOT` outside any checkout** falls back to the cwd's parent, as the plan says. Started
+ from `/`, that is `/`.
+- **`bg` is now relative to the song data dir,** and nothing reads it yet. `ThumbEntry` declares
+ it and says how to resolve it (review L3, below).
+
+**Changelog.**
+- **One `[Unreleased]` bullet** in `editor/CHANGELOG.md`. It leads with the change and the command
+ a reader runs, `mkdir -p ~/.local/share/archilyzer && ln -s <where the data is>
+ ~/.local/share/archilyzer/song`, before restarting umtool (review L1). Then: the corpus from
+ `TRANSCRIPTS_DIR` or the checkout, the videos' default, relative manifests, and the run logs gone.
+- **The released `[0.9.0]` storage-locations entry now says `/run/media/<user>/<uuid>`.** A
+ released entry is normally left as written. This one was changed because it is a path example,
+ not a name a reader would search for.
+
+**For the operator** (the plan's Rollout 0, the slice Q step). Do this **before any umtool
+restart**:
+```
+mkdir -p ~/.local/share/archilyzer && ln -s /home/user/.claude/jobs/efbe67a7/tmp/song ~/.local/share/archilyzer/song
+```
+- **What the link keeps (review I1).** The link target is the live :3050's current default
+ `SONG_DATA`, and it holds **only the rebuildable `.cache/umtool`**: 7.9 MB of project index and
+ caches, "safe to delete at any time" (`lib/paths.mjs`). The song project's bulk data is under
+ `~/reports/quartering-uh-song/data`, not there. The link keeps :3050's `SONG_DATA`,
+ `SONG_SCRATCH`, `CACHE_DIR` and `INDEX_DIR` byte-identical to today's (the job dir's realpath is
+ its path).
+- **What a missed link costs:** an index rebuild, and the nesting trap. It loses no data. A
+ restarted :3050 with no link finds nothing at the new default and may create
+ `~/.local/share/archilyzer/song/.cache/umtool` as a real directory. A later `ln -s` would then put
+ the link *inside* it (`song/song`). If that happened, remove the directory first.
+- **Linking to `~/reports/quartering-uh-song/data` instead** is a separate choice for the operator.
+ It moves `CACHE_DIR` there and makes `SONG_SCRATCH` equal `SONG_REPORTS`.
+- **Before the restart,** the live :3050 is a `next start` build and keeps its baked paths, but it
+ runs song scripts from disk:
+ - the song build chain passes `SONG_DIR` explicitly (`lib/trim.ts:226`);
+ - `accept-thumb` needs only `SONG_REPORTS`, whose default is unchanged;
+ - a script run by hand with no `SONG_DIR` looks at the new default.
+
+**Review** (verdict **SHIP**; a read-only Opus review of `90c7f776..57ce980a`, `$T/q-review.md`).
+No High or Medium findings. The coordinator asked for two of the Lows to be fixed:
+- **L1 — fixed, `75327c6f`.** The changelog's bare `ln -s` failed on a machine with no
+ `~/.local/share/archilyzer`. The bullet now leads with the full command,
+ `mkdir -p ~/.local/share/archilyzer && ln -s <where the data is> ~/.local/share/archilyzer/song`.
+ It names no machine path.
+- **L3 — fixed, `0416881f`.** A future reader could resolve the relative `bg` through
+ `resolveInRoots`, which binds it to `SONG_REPORTS`. `ThumbEntry` now declares `bg?` with the
+ rule: relative to the song data dir, resolve with `dataFile()`, never `resolveInRoots`. This is
+ a type and comment only; no behaviour changed.
+- **Left, on the coordinator's word:**
+ - **L2.** Run from a cwd outside any checkout, umtool's `REPO_ROOT` falls back to the cwd's parent
+ and reads a corpus that is not there. This is the plan's design, and every documented
+ invocation runs inside the checkout. A walk from `import.meta.url` as a second try is the
+ follow-up.
+ - **L4.** `cues.mjs` (`build-video`, `resolve-windows`, `check-availability`) still ignores
+ `TRANSCRIPTS_DIR`. The plan leaves `cues.mjs` alone, and this is not a regression.
+ - **L5.** With no song data, `SONG_SCRATCH` widens to `~/.local/share/archilyzer` in the read and
+ write roots. Nothing else lives there today, and it was noted only.
+ - **L6.** Cosmetic: a stale "job temp dir" comment in `lib/paths.mjs`, ragged comment wraps, and
+ `docs/folders.md`'s older roots list. Not worth a commit here.
+- **I2, for slice R.** `29dc507c`'s commit message and older blobs carry `/run/media/user`, which
+ the built-in `${os.homedir()}` rule does not match. R's `source-scrub.txt` needs its own rule, or
+ its denylist refuses the publish (it fails closed).
+- **After the fixes:** tsc is clean (41 s, all seven packages), and the grep gate is empty at the
+ new tip. Per the coordinator, e2e was not re-run for a type comment and a changelog line.
+
+### Slice Q, found after rollout — the umtool build walked the corpus (2026-09-28)
+
+**What.** Slice Q's `umtool/lib/paths.mjs` set `CHANNELS_DIR` to
+`path.join(REPO_ROOT, "transcripts", "channels")`, with
+`REPO_ROOT = findRepoRoot(process.cwd())`.
+- **Turbopack evaluated that statically** as `[project]/transcripts/channels` and made it a
+ directory asset reference. The walk's fallback, `path.resolve(start, "..")`, is the project
+ root.
+- **In `<primary>`,** `pnpm --filter umtool exec next build` walked the real corpus (hundreds of GB,
+ channel `data/` symlinked to another drive). It was OOM-killed twice, at about 3.7 GB RSS, so the
+ live umtool on :3050 stayed down until this fix.
+- **Slice Q's gate built in a worktree,** which has no `transcripts/`, so the reference was empty
+ and the build took 30 s. The hazard and the rule are in FACTS: "A path joined from
+ `process.cwd()` is a directory of assets to Turbopack".
+
+**The repro.** Link the corpus into a worktree for the BUILD only, and cap memory:
+```
+ln -s <primary>/transcripts <worktree>/transcripts
+timeout -s KILL 240 systemd-run --user --scope -q -p MemoryMax=5G -p MemorySwapMax=0 pnpm --filter umtool exec next build
+rm <worktree>/transcripts
+```
+On `main` `10cefd15` it fails in 6.4 s (0.97 GB): `TurbopackInternalError: Failed to write app
+endpoint /page … [project]/umtool/lib/paths.mjs … <DirAssetReference as
+ModuleReference>::resolve_reference failed … Symlink [project]/transcripts/channels/<slug>/archive
+is invalid, it points out of the filesystem root`.
+
+**The fix** (branch `fix/umtool-build-trace` off `main` `10cefd15`). Every path or fs call on a
+value derived from `process.cwd()` or `import.meta.url`, in a umtool module the app imports, now
+opens with `/* turbopackIgnore: true */`. That is the per-expression opt-out Turbopack's own
+message documents. The calls are in:
+- `lib/paths.mjs` (`findRepoRoot`, `CHANNELS_DIR`);
+- `lib/paths.ts` (`SONG_CODE`, `stateFile`);
+- `lib/tools.mjs`, `lib/trim.ts`, `lib/report/driver.mjs`;
+- `report-to-video/brand.mjs` and `cues.mjs`.
+
+The values at run time are unchanged. From `umtool/`, `REPO_ROOT`, `CHANNELS_DIR` and `SONG_DATA`
+print the same as on `main`, and `CHANNELS_DIR` / `TRANSCRIPTS_DIR` / `SONG_DIR` still win.
+`os.homedir()` joins are left as they are. A build with `HOME` pointed at a synthetic home inside
+the project, holding out-of-root symlinks at every home-derived root, succeeded, so the tracer does
+not follow `os.homedir()`.
+
+| build (5 GB cap, `/usr/bin/time -v`) | result | wall | max RSS | `.next` |
+|---|---|---|---|---|
+| `main` `10cefd15`, corpus linked | **fails** (the error above) | 6.4 s | 0.97 GB | — |
+| fix, no corpus | ok | 42.2 s (a busy machine; 22.2 s on an earlier run) | 0.80 GB | 1,009 files, 25,417,403 B |
+| fix, corpus linked | ok | 24.2 s | 0.80 GB | 1,009 files, 25,417,285 B |
+
+- **The two outputs are the same file set,** differing only in the build-id directory.
+- **Nothing from the corpus is traced.** 0 `.nft.json` entries are under `transcripts/`. The 14
+ files that contain the string `transcripts/channels` are all `.js.map` source maps of the code;
+ none is a chunk or an asset.
+
+**The guard: `scripts/umtool-build-trace.test.mjs`** (in `test:scripts`, 3 tests). It is a static,
+per-module check of umtool's app, components, lib, `report-to-video/*.mjs` and `song/paths.mjs`. A
+path or fs call carrying a value derived in that file from `process.cwd()`,
+`import.meta.url|dirname|filename` or `__dirname` must open with the opt-out. It costs about 0.1 s.
+With `main`'s `lib/paths.mjs` swapped in, it fails and names the defect:
+`umtool/lib/paths.mjs:118 : path.join(REPO_ROOT, "transcripts", "channels")`.
+
+| sha | what |
+|---|---|
+| `8346f824` | `umtool:` cwd-derived paths opt out of Turbopack's asset tracing |
+| `55699a20` | `scripts:` the guard |
+| _this_ | `plans:` this note, FACTS, the implementer rules' umtool build gate; the `[Unreleased]` bullet |
+
+**Gates:**
+- tsc is clean (all seven packages).
+- `node --check` passes on the five changed `.mjs`.
+- `test:scripts` **188 + 1 skipped** (`main` 185 + 1, plus the guard's 3).
+- The capped builds are in the table above.
+- umtool e2e (`faces`, `deck`, `clip-bench`, with the corpus link removed): **67 passed, 8
+ skipped, 0 failed** (1.9 min). The skips are the song-data specs, as in slice Q.
+
+**The other apps are safe by accident, not by rule.**
+- `common/lib/paths.ts`' `findMonorepoRoot()` falls back to `process.cwd()` (the app's own
+ directory).
+- `homepage/app/lib/source.ts` joins `process.cwd()` + `public`.
+- Both build today, so there is no finding to fix, only a note in FACTS.
+
+**For the parent:** merge, then rebuild and restart :3050. That is the parent's step.
+
+### Slice R, as shipped — `archilyzer source publish` + `/source` (2026-09-28)
+
+Branch `r12/source-mirror` off `main` `e6c5d2e3` (slice Q merged), worktree
+`~/Projects/r12-source-mirror` (worktree #10: editor 4001, homepage e2e 4040), one Opus
+implementer. Plan: [`source-mirror.md`](source-mirror.md), "Slice R", R1–R7. Why: the operator
+asked for the repo on the project site, read-only, clonable from static files, with a gate that
+refuses to publish a denied literal. `main` did not move during the slice (`git merge main`:
+already up to date). Scratch files `r-m-*` in the job's `tmp`.
+
+**What shipped.**
+- **R1, `common/publish/source.ts`: `publishSource(opts)`** — 0 published / skipped / checked,
+ 1 refused or cancelled; a `SourceRefusal` is logged as `[source] REFUSED: …`. The steps are the
+ plan's 1–17:
+ - `git rev-parse` of the git COMMON dir's `refs/heads/main`;
+ - the two operator files, parsed; a missing one is a refusal naming it (`~/`-relative);
+ - the skip;
+ - `resolveFilterRepo()`: `git filter-repo`, else `pipx run --spec git-filter-repo==2.47.0`,
+ else the install line;
+ - `git clone --no-local --bare --single-branch --no-tags --branch main` into
+ `mkdtemp(<ARCHILYZER_SOURCE_SCRATCH>/archilyzer-source-)`, origin removed;
+ - filter-repo `--force --quiet --replace-refs delete-no-add --replace-text R --replace-message R`,
+ then `filter-repo/` deleted;
+ - `repack -a -d -q --max-pack-size=20m`, `prune-packed`, `pack-refs --all`, `update-server-info`.
+ It refuses on loose objects, no `P` line, any ref but `refs/heads/main`, or a HEAD that is not
+ main.
+ - the object gate; the tree (`git archive` → `tar -x` → pages); the tarball (`--format=tar.gz
+ -9 --prefix=archilyzer/`) and `snapshot.json` in the `Snapshot` shape;
+ - the mirror staged from an allowlist; the manifest staged; the file gate; the limits;
+ `--check` stops; the link-safe install (manifest removed first, written last); one summary line.
+
+ Every child goes through `runChildIntoLog` with `AbortSignal.any([signal, timeout])`, in an
+ environment with the `GIT_DIR`-family variables cleared. The one exception is the audit's
+ binary `cat-file` stream (a `spawn`, with the signal). `clearPublishedSource` is what
+ `--no-source` runs. `common/lib/sourceManifest.ts` is the leaf contract: `MIRROR_DIR`,
+ `CLONE_URL`, `TREE_HREF`, `TARBALL_HREF`, `SourceManifest` and `parseSourceManifest`.
+- **R2, `sourceAudit.ts`.**
+ - `parseDenylist` (`i:`, `#` comments, trimmed, deduped).
+ - `scanBuffer`, and `redactHit`: ±24 bytes, every byte any hit covers masked, a half-shown
+ occurrence included.
+ - `maskLiterals` for every path or line quoted from the repo.
+ - `auditObjects`: ONE `cat-file --batch-all-objects --unordered --batch` stream with a framing
+ parser — blobs and commits/tags whole, trees by entry NAME (not their binary ids).
+ - `auditFiles`: contents, paths, a `.gz` decompressed; a symlink is a refusal.
+ - `runGitleaks`: cwd = scratch, so no checkout's ignore file applies. 0 is clean, 3 is parsed
+ findings, anything else refuses; not on PATH is "skipped" with a WARNING.
+ - `auditBare`: blob hits get their path in history, and gitleaks runs only when the literal
+ audit is clean.
+ - `formatAuditReport`: `#n (x…, len L)`, counts per literal and kind, the first 20 contexts,
+ "… and N more", then the plan's closing line.
+- **R3, `sourceTree.ts`** (`hrefFor`, `escapeHtml`, `renderTreeIndex`, `writeTreeIndexes`, which
+ refuses a tracked `index.html` or a symlink), and the `_headers` block — with the correction
+ below.
+- **R4, the homepage.**
+ - `app/lib/source.ts` `loadSourceManifest()` (also needs `info/refs` and the tarball).
+ - Nav: Source after Docs.
+ - `app/source/page.tsx`: the plan's copy verbatim; `source-clone`, `source-mirror-head`,
+ `source-tree-link`, `source-tarball-link`, and `source-tarball-sha` for the spec; the empty
+ state `source-empty`.
+ - A shared `components/Fact.tsx`.
+ - Downloads: one sentence linking "read-only git mirror", and the empty state names
+ `archilyzer source publish`; every phrase `downloads.spec.ts` asserts is kept.
+ - `marketing.spec.ts`' nav list gains Source.
+- **R5, the docs.** `create-archives.sh` is deleted. README and SETUP lead with the clone and keep
+ the tarball as the no-git path. PUBLISH.md gains "The source mirror (homepage)". The homepage's
+ *Install*, FAQ, *What is Archilyzer* and `content/README.md` stop saying "there is no public
+ repository". `grep -rn 'create-archives' README.md SETUP.md homepage/` is empty.
+- **R6 and the rest.**
+ - `getPaths()` gains `configDir`, `sourceScrubFile`, `sourceDenylistFile` and
+ `sourceScratchDir`, from `ARCHILYZER_CONFIG_DIR`, `SOURCE_SCRUB_FILE`, `SOURCE_DENYLIST_FILE`
+ and `ARCHILYZER_SOURCE_SCRATCH`, declared as `paths` in `envVars.ts`.
+ - `HOMEPAGE_PUBLIC_DIR`'s `readBy` names `source.ts`, and `TRANSCRIPTS_DIR`'s doc says umtool
+ reads `<it>/channels` too (review Q(c)). `ENVIRONMENT.md` is regenerated.
+ - The CLI rows `build homepage [--no-source]`, `source publish [--force] [--check]
+ [--keep-scratch]` and `source audit [<git dir>]`.
+ - `buildHomepage`: compose, then the source step (a lazy import; `publishSource` and
+ `clearSource` are seams), then `next build`.
+ - doctor's "source publish" block. It is never a failure; it WARNs when the operator files exist
+ but no filter-repo is installed, or a file is readable by others. It prints rule counts and
+ modes, never contents.
+
+**Corrections to the plan** (each found by running it):
+- **`_headers`: later rules APPEND, they do not win** (`f218ed86`).
+ - wrangler 4.88's `attachHeaders` (the Pages asset server's code, read in its `cli.js`) `set`s
+ a header on the first matching rule and `append`s it on every later one. The plan's exact
+ text served a directory page as `text/plain; charset=utf-8, text/html; charset=utf-8` and a
+ font as `text/plain; charset=utf-8, font/ttf`.
+ - Each override now starts with `! Content-Type`. A replica of wrangler's parse and attach
+ (`$T/r-m-headers-sim.mjs`) gives single values for `/source/tree/`, `…/common/`, a `.ts`, the
+ bracketed `.ttf` and `page.tsx`, the `.onnx` and `README.md`.
+ - The preview deploy is still the live proof.
+- **The tsconfig must exclude `out` too** (`3fa5ff18`).
+ - `next build` copies `public/` into `out/`. The SECOND build with a mirror failed its type check
+ on `out/source/tree/common/jobs/registry.ts`, a second `declare global var __yttJobRegistry__`.
+ - With `out` excluded, the homepage program is 871 files, none from the mirror; homepage tsc
+ takes 11 s with a mirror in both dirs.
+ - The homepage's `eslint.config.mjs` ignores `public/source/**` (it already ignored `out/**`).
+ Lint itself was not run.
+- **The published manifest carries no `rulesHash`.**
+ - `plans/` ships in the mirror, and so does the plan's description of the two files. What is
+ left unknown is the hostname and the address, so a published hash of the rules would confirm
+ a guess at them.
+ - The skip key (`{sourceCommit, rulesHash}`) is `homepage/.source-publish.json`, beside
+ `public/` and gitignored. A missing key rebuilds; the round trip asserts it is absent from the
+ manifest.
+- **The mirror has a loose `refs/heads/main` beside `packed-refs`.** git treats a directory as a
+ repository only with a `refs/` dir, so without it the `file://` clone and `source audit` of the
+ published dir fail. An empty dir would not survive a deploy.
+- **filter-repo's `--replace-text` does not skip `#` lines** (it would replace a comment as a
+ literal; `get_replace_text` in 2.47). The step writes `replace.txt` without comments or blank
+ lines.
+- **The step adds three safety flags:** `--no-tags`, `--replace-refs delete-no-add` (2.47's
+ default is `update-no-add`, stated for determinism), and a check that the mirror holds exactly
+ `refs/heads/main`.
+- **`--no-source` removes the published source** (manifest first; a linked `downloads/` goes as a
+ link, its target untouched). It does not leave the source there: a copy audited against older
+ rules would ship ungated. That removal gives the empty state R7 expects.
+- **The header nav moves from `sm` to `md`:** five labels do not fit beside the wordmark at 640 px.
+- **The tree has 4 `.ttf`**, not 3 (IBM Plex Mono Bold, O5); `*.ttf` covers them.
+- **No `branch` option:** `SOURCE_BRANCH = "main"`, an operator decision. `now` is `() => Date`.
+- **The packs are 37.5 MB in two** (20,897,277 + 16,633,904 bytes), not the plan's 21.5 MB in one.
+ The history grew by about 150 commits since the plan, and the 20 MB split costs deltas across
+ the two packs. That is well inside the limits.
+
+**The operator's files, as created, refuse the publish** (an action for the rollout, not the
+slice):
+- **The run:** `pnpm archilyzer source publish --check` with `~/.config/archilyzer/*` as the
+ parent created them, at `main` `e6c5d2e3`. It ended `AUDIT REFUSED: 8 hits in 21,441 objects
+ (1,702 commits) against 6 denied literals`, with every hit `#1 (r…, len 5)` in a blob:
+ - `plans/source-mirror.md`, two versions, 3 hits each: the plan's own grep gates (`git grep -c
+ -i '<it>' …`, `git -C <clone> grep -c -i <it> …`) and the "a future transcript quote "<it>
+ that"" risk line;
+ - `plans/release-12.md`, two versions, 1 hit each: slice Q's grep-gate line.
+- **Why the scrub rules miss them.** They cover the backticked spelling and the `/run/media/` path
+ (review I2: no hit came from there), not the bare, single-quoted or double-quoted name.
+- **The operator's step:** add a rule for the bare name (`<name>==>user` covers every spelling,
+ the backticked one included) or one per spelling. Then `archilyzer source publish --check`
+ (about 25 s) must end `check passed`.
+- **How R7 ran anyway,** with the operator files untouched and the gate not weakened:
+ - `SOURCE_SCRUB_FILE` = a scratch file of ONE rule, `<name>==>user`, written with `$(id -un)`,
+ with the REAL denylist. The check was clean: 4 literals, gitleaks clean.
+ - `source audit` of both clones and of the published dir with the REAL files (6 literals) is
+ clean, below.
+
+**Gates** (from the worktree root; logs `$T/r-m-*.log`). Every log of a run over the real files was
+first scanned by `$T/r-m-leakcheck.mts` (counts per literal label, any ASCII case) and read through
+`$T/r-m-maskview.mts`. The step's own lines carried 0 literals; the only hits were pnpm's and
+doctor's home-directory paths.
+- **tsc** was clean before every commit (`r-m-tsc-{0..3}.log`):
+ - 215 s at the first run, 168 s, 59 s, and 73 s with a mirror in `public/` AND `out/`. Another
+ session's tsc was running during the first and the last.
+ - The homepage alone: 11 s, 871 files, none from the mirror.
+- **Unit:**
+ - common **2,138/2,138** (2,114 + 24: sourceAudit 7, sourceTree 5, source 7, build +1, `_cli`
+ +3, doctor +1). The two filter-repo tests RAN (`ok 1832` round trip, `ok 1833` planted-literal
+ refusal); 0 skipped.
+ - `test:scripts` **185 + 1 skipped**; editor unit **85/85**; mcp **269/269**; homepage unit
+ **2/2**.
+- `pnpm archilyzer docs env --check` exits **0**. The editor's `next build` is **ok** (43 s); it
+ bundles `buildHomepage`'s lazy import.
+- **`archilyzer build homepage`** (from `common/`, `SOURCE_SCRUB_FILE` as above):
+ - **ok in 38 s**, against 19 s for `--no-source`. The source step took **19 s**: filter-repo
+ 5.3 s, gitleaks 7.3 s, `[source] published main e6c5d2e322b3 as 78dc0126382b: 2459 files,
+ 69.5 MB (mirror 2 packs, tree 412 dirs), tarball 6.9 MB sha256 c4ceb6d110ee`.
+ - A rebuild with nothing changed logged `[source] up to date at e6c5d2e322b3; skipping`.
+ - **Deterministic:** two runs gave the same `mirrorHead` and tarball sha; the real files' two
+ runs gave `9b880270c49d` both times.
+- **What it published** (`homepage/out`):
+ - **2,640 files, 78.1 MB** in all; `source/` 2,465 files, 65.8 MB;
+ - the mirror: 9 files, 38.1 MB, 2 packs;
+ - the tree: 2,035 files + 412 pages, 27.5 MB;
+ - the tarball: 7,246,992 bytes.
+ - **Against the limits:** 2,459 staged of the step's 15,000 (2,640 of Pages' 20,000); the
+ largest file is 19.93 MiB, against 24 MiB (25 MiB).
+- **Two clones, both HEAD = `manifest.mirrorHead` `78dc0126382b…`:**
+ - `git clone file://$WT/homepage/out/source/archilyzer.git`: 2 s.
+ - `git clone http://127.0.0.1:8765/source/archilyzer.git` from `python3 -m http.server 8765
+ --bind 127.0.0.1` in `homepage/out`: 1 s. The protocol was dumb: `info/refs?service=…` 200,
+ then `HEAD`, `objects/info/packs`, both `.idx` and both `.pack`; the loose-object probes 404.
+ - `:8765` was free before, the server was killed, and it was free after.
+- **The user name:** `git grep -c -i -F "$(id -un)" $(git rev-list --all) | wc -l` is **0** in both
+ clones (1,702 revisions). It is 0 in the identities and messages and 0 in the paths too; the
+ hostname is 0.
+- **`archilyzer source audit`** with the real files (6 literals) is **clean** on the file clone
+ (13 s), the http clone (11 s) and the published dir (10 s): 21,441 objects, 1,702 commits,
+ gitleaks "1530 commits scanned … no leaks found".
+- **The gate, exercised:** `SOURCE_DENYLIST_FILE` = a scratch file of `Co-Authored-By`, then
+ `source publish --check`:
+ - exit **1** in 14 s, `AUDIT REFUSED: 1476 hits … #1 (C…, len 14): 8 in blobs, 1468 in
+ commits`, 20 context lines and "… and 1456 more";
+ - the log holds the literal **0** times;
+ - `public/source/manifest.json`'s mtime and size are unchanged (`1790616152 1057`).
+- **`build homepage --no-source`:** `out/source/index.html` holds `data-testid="source-empty"`
+ and "No source published in this build."; `/downloads/` says "no source snapshot attached".
+- **No filter-repo:** with PATH = a scratch dir of two symlinks (`git`, `node`; nothing
+ uninstalled), `git filter-repo` is "not a git command", and `source publish --check` exits 1 with
+ `REFUSED: git-filter-repo is not installed and pipx is not on PATH — install it once: \`pipx
+ install git-filter-repo\` …`.
+- **The resolved filter-repo** is `git filter-repo` (`~/.local/bin/git-filter-repo`, pipx-installed
+ 2.47.0), whose `--version` prints `a40bce548d2c`; git is 2.55.0. The `pipx run` fallback was not
+ run (it needs the network).
+- **`archilyzer doctor`** prints the block:
+ ```
+ source publish
+ ok filter-repo git filter-repo a40bce548d2c
+ ok gitleaks gitleaks version is set by build process
+ ok scrub rules ~/.config/archilyzer/source-scrub.txt (2 rules, mode 600)
+ ok denylist ~/.config/archilyzer/source-denylist.txt (3 literals, mode 600)
+ -- published main e6c5d2e322b3 as 78dc0126382b, 2026-09-28T17:22:29.301Z (2458 files)
+ ```
+ The real output prints the absolute paths. `2458` is the manifest's `files`, which does not
+ count the manifest itself.
+- **homepage e2e** (`node scripts/worktree.mjs run -- pnpm --filter homepage run e2e`; the queue
+ was free):
+ - **36 passed, 0 skipped, 0 failed, 1.1 min**, WITH the manifest present:
+ `loadSourceManifest()` from `homepage/` gave mirror head `78dc0126382b`, so the published
+ branch of every `source.spec` test ran.
+ - The empty state too: with `public/source` and `public/downloads` moved aside and restored
+ after, `source.spec` + `downloads.spec` + `marketing.spec` gave **16 passed** (30 s).
+ - `downloads.spec.ts` is unchanged.
+- **Numbers tool:** none.
+
+**They bite:**
+- The planted-literal test fails if the object walk misses the blob.
+- The redaction test pins a half-shown second occurrence.
+- `build.test.ts`' fake refusal proves `next build` never runs.
+- The restricted-PATH run proves the install line.
+- The `Co-Authored-By` run proves the gate refuses a literal the scrub does not touch, in commits
+ AND blobs.
+
+| sha | what |
+|---|---|
+| `25f5e241` | `homepage:` `/homepage/public/source` gitignored (the Tailwind note), tsconfig excludes `public` — first, before any mirror |
+| `7fdbe2dc` | `common:` `sourceAudit.ts`, `sourceTree.ts`, `sourceManifest.ts` + tests |
+| `fc31aab5` | `common:` `publishSource` (`source.ts`) + tests; `getPaths()` / `envVars.ts` / `ENVIRONMENT.md` |
+| `2f08cb47` | `common:` `buildHomepage` runs the step; the CLI rows; doctor's block + tests |
+| `1a11a2bb` | `homepage:` `/source/`, nav, `Fact`, Downloads, `_headers`, docs content, `source.spec.ts`, marketing nav |
+| `2cd189f3` | `docs:` README, SETUP, PUBLISH "The source mirror (homepage)"; `create-archives.sh` deleted |
+| `c448b392` | `changelog:` the editor and homepage `[Unreleased]` bullets |
+| `f218ed86` | `homepage:` `_headers` overrides detach `Content-Type` first (correction) |
+| `3fa5ff18` | `homepage:` tsconfig excludes `out`; eslint ignores `public/source/**` (correction) |
+| _this_ | `plans:` this record |
+
+**Found and left** (not this slice's files):
+- **`.dockerignore:20-22`** still names `create-archives.sh`, and it does not exclude
+ `homepage/public/source/`. A `docker build` from a checkout that has published would send about
+ 66 MB of mirror and tree in the build context.
+- **`common/lib/project.ts:37-38`** (`PROJECT_DOWNLOADS_URL`'s comment) says "there is no public git repository".
+- **The editor's /sites Homepage section** (`HomepageBuildButtons.tsx:170`) does not say that the
+ build publishes the source, or that it can refuse.
+- **`export/CHANGELOG.md`'s released entry** names `create-archives.sh`. It is history, and is left.
+- **The label `#n (x…, len L)`** shows a literal's first character and length in the logs. This is
+ the plan's format; nothing of it is published.
+- **gitleaks** scans 1,530 of 1,702 commits: its `git log -p` skips merges. The literal audit reads
+ every object.
+
+**Changelog.**
+- **`editor/CHANGELOG.md`:** one `[Unreleased]` bullet. The build step and the gate; the operator
+ files, and that the build refuses without them; `pipx install git-filter-repo`; the skip, the
+ CLI rows and doctor; `create-archives.sh` gone.
+- **`homepage/CHANGELOG.md`:** one `[Unreleased]` bullet. `/source/` and its nav entry, the empty
+ state, the tarball regenerated per build, the docs, the nav at `md`, and `_headers`.
+
+**For the parent and the operator:**
+1. Add the scrub rule above, then run `archilyzer source publish --check`.
+2. The editor's /sites homepage job finds `git filter-repo` only if the editor's PATH includes
+ `~/.local/bin`. Otherwise it falls back to `pipx run`, which needs the network on first use.
+3. The worktree's `homepage/public/source`, `homepage/out` and `homepage/.source-publish.json` were
+ made with the scratch rule. They are disposable, and nothing here deployed them.
+
+**Review** (verdict **SHIP AFTER FIXES**; a read-only Opus review of `e6c5d2e3..6ddfd498`,
+`$T/r-review.md`). The parent then added the missing scrub rule to the operator file:
+`source publish --check` with the real files exits 0. The coordinator ruled that M1 and the
+listed Lows be fixed on the branch. `main` had moved by one `plans:` commit (the release 11
+rollout record), merged at `439eb106` with no conflict.
+- **M1 — fixed** (`7d64054c`, and `78b32636` below). **A refusal left the previous publish in
+ place**, in `homepage/public` and in the last build's `homepage/out`, for a deploy-only or a
+ raw `next build` to ship under rules it was never audited against. Now it is withdrawn at three
+ points:
+ 1. **`publishSource`:** once the rules are loaded, any outcome but success — an audit hit, a
+ limit, a missing tool, a cancel, a crash — runs `removePublishedSource`. That removes the
+ manifest first, then the mirror, the tree, the tarball, `snapshot.json` and the skip key.
+ `--check` still writes nothing, this included: it is a dry run, and the deploy check below
+ covers what it cannot.
+ 2. **`buildHomepage`:** a non-zero source result also removes `out/source` and the two download
+ files from `homepage/out`.
+ 3. **`deployHomepage` asks `publishedSourceProblem` before every deploy**, preview included.
+ - An `out/` with source artefacts ships only when the skip key says the publish was made
+ under TODAY's rules and step (`rulesHash`), of TODAY's `main`, and is the publish in `out/`:
+ the same mirror head, and a tarball whose sha256 matches.
+ - It keys on the artefacts and the `/source` page, not on `out/source` existing (`78b32636`).
+ A `--no-source` build still renders the page into `out/source/index.html`, and the first
+ version refused exactly that build; I found it by running one.
+ - The page with no artefacts beside it is a `--no-source` build, and deploys as before.
+ - No page at all is a refused build (or one from before the page), and it refuses.
+ - **Tests:** the round trip denies a literal the mirror holds, and the publish then exits 1 with
+ `public/source`, the tarball, `snapshot.json` and the key gone. The skip test covers a changed
+ rule. `build.test` covers the `out/` removal and the deploy refusals. `publishedSourceProblem`
+ is covered on a real publish: ok, tampered tarball, newer `main`, other rules, no record.
+- **Q3/L5 — fixed** (`6936f6e2`). A label says where the literal was written: `denylist line 3
+ (len 5)`, `scrub line 2 lhs (len 11)`, `built-in home rule (len 11)`. No character of the
+ literal appears.
+- **L4 — fixed** (`6936f6e2`).
+ - A hit is listed by kind, object id (with the blob's path in history), byte offset, and for a
+ commit or tag the header field (`author`, `committer`, `tagger`, …) or `message`; a tree hit
+ by entry number. `redactHit` is deleted: no byte of an object is printed.
+ - The test asserts the report carries none of the planted text's neighbours (`/srv/`,
+ `example.invalid`, `Planted <`, the message).
+ - PUBLISH.md and the editor changelog show the new format.
+- **L1 — fixed** (`6936f6e2`): a tracked `404.html` anywhere in the tree is refused, with a test.
+- **L2 — fixed** (`7d64054c`).
+ - A UTF-8 BOM and CRLF are dropped from both operator files (`operatorLines`), and a trailing
+ `/` from the home dir.
+ - An empty left side (`==>x`, `literal:==>x`, `regex:==>x`, `glob:`) is a refusal naming its
+ line.
+ - The reviewer's reproduction is a test: a BOM + CRLF scrub file still scrubs, and still
+ denies its left side. The round trip now runs with a BOM + CRLF scrub file and denylist.
+- **L3 — fixed** (`7d64054c`): the published manifest (and `SourceManifest`) no longer carries
+ `audit.literals`. The log still gives the count.
+- **L6 — left, as ruled:** compressed content is opaque to the byte search. The reviewer
+ decompressed every compressed or binary blob in the history (a zip, PNGs, icons) and found 0
+ hits. PUBLISH.md records it as a known limit, and says a binary is audited, never scrubbed.
+- **L7 — fixed** (`d3c680ea`). `.dockerignore` excludes `homepage/public/source/` and
+ `homepage/.source-publish.json`, and no longer names `create-archives.sh`.
+- **L8 — fixed** (`7d64054c`), for a checkout with no git repository.
+ - `buildHomepage` passes `noRepository: "empty"`. The step logs `[source] no git repository
+ here; nothing to mirror — the /source page will show its empty state`, removes an old publish,
+ and returns 0.
+ - `archilyzer source publish` exits 1 with the same sentence, and no raw git error.
+ - A test runs both over a temp dir with no operator files.
+- **L9 — fixed** (`7d64054c`, `91728a21`).
+ - The round trip reads EVERY object in the published packs from a plain file copy (`cat-file
+ --batch-all-objects`), and asserts `objects/pack` holds only `pack-*.{pack,idx}`.
+ - `homepage/app/lib/headers.test.ts` pins `_headers` with a replica of wrangler's parse and
+ attach:
+ - each `/source/tree` override says `! Content-Type` first;
+ - each path gets ONE value;
+ - the file stays inside wrangler's limits.
+ - `E2E_EXPECT_SOURCE=1` makes the empty state FAIL `source.spec.ts`' three data tests. It is
+ declared in the homepage playwright config and the env registry. It bites: in the empty state
+ with the flag set, **3 failed, 2 passed**.
+- **L10 — fixed** (`7d64054c`, `91728a21`). `parseSourceManifest` checks every number the page
+ reads, and `loadSourceManifest` takes the public dir as a seam. Unit tests cover common (2) and
+ homepage (2): a malformed, wrong-version or orphaned manifest is null, never a throw.
+- **L11 — fixed** (`7d64054c`).
+ - `SOURCE_STEP_VERSION` (now 2, with a comment to bump it whenever the scrub or the audit
+ changes) is in the rules hash.
+ - The skip key also holds the filter-repo label and version, and the mirror head.
+ - A test shows another filter-repo version does not skip.
+- **L12 — fixed** (`7d64054c`).
+ - A scratch root that lands (through symlinks) inside the checkout or the public dir is refused.
+ - `--keep-scratch` deletes `replace.txt` (written mode 600) and says so.
+ - Tests cover both.
+- **L13 — fixed** (`d3c680ea`).
+ - The `project.ts` comment names the mirror.
+ - PUBLISH.md's tsconfig line says `public` and `out`. The "new spelling" line now says the
+ implied denial is the exact bytes, and a new spelling is caught only by the denylist.
+ - The editor's /sites Homepage section gains one `<p>`: Build also publishes the source and can
+ refuse; a refusal removes it from `homepage/out` too; Deploy refuses an unaudited source.
+ I did not run the editor e2e for it:
+ - `sites-homepage.spec.ts`' assertions on that group are `toContainText` substrings of the
+ existing paragraph, role-and-name lookups for buttons, and `getByText(/^Queued/)` and
+ `"Cancelled"` (exact). A new sibling paragraph cannot break any of them.
+ - Its build job is held on the queue and cancelled, so the source step never runs there.
+- **The runbook point (PUBLISH.md, `d3c680ea`):**
+ - a Pages preview is public, and every deployment stays reachable at its hash URL until it is
+ deleted;
+ - the private literals go in the denylist before ANY deploy;
+ - if something private ships, delete that deployment, because a newer deploy does not remove it;
+ - the editor's process needs `~/.local/bin` on its PATH.
+- **The five questions, as ruled:**
+ 1. The rules hash in the gitignored `.source-publish.json` is **acceptable**. The step version
+ has been added (L11).
+ 2. `--no-source` removing the previous publish is **right**.
+ 3. The first character is **dropped** (above).
+ 4. The loose `refs/heads/main` is **acceptable**.
+ 5. `.dockerignore` is **fixed here** (L7).
+
+| sha | what |
+|---|---|
+| `439eb106` | merge `main` (the release 11 rollout record) |
+| `6936f6e2` | `common:` labels by provenance, no object bytes in the report, `404.html` refused (Q3, L4, L1) |
+| `7d64054c` | `common:` a refusal withdraws; the build's `out/` copy; the deploy check; L2, L3, L8, L10, L11, L12 |
+| `91728a21` | `homepage:` `E2E_EXPECT_SOURCE`; the loader and `_headers` unit tests (L9, L10) |
+| `d3c680ea` | `docs:` PUBLISH.md (withdrawal, previews, no-repo, report format, L6); `.dockerignore`; `project.ts`; the /sites copy; the changelog bullet |
+| `78b32636` | `common:` the deploy check keys on the artefacts and the `/source` page (found by running `--no-source` against it) |
+| _this_ | `plans:` this review record and `source-mirror.md`'s "As shipped" note for R |
+
+**Gates after the fixes** (logs `$T/r-m-*.log`):
+- **tsc:** clean before every commit (36–55 s).
+- **Unit:**
+ - common **2,146/2,146** (+8: sourceAudit +2, source +3, build +1, sourceManifest +2). The
+ filter-repo tests ran, 0 skipped.
+ - homepage unit **7/7** (+5); editor unit **85/85**; `test:scripts` **185 + 1 skip**.
+ - `docs env --check` exits **0**.
+- **`source publish --check` with the REAL files** exits **0** in 19 s:
+ - `audit clean: 21,447 objects (1,703 commits), 2,459 staged files against 6 denied literals;
+ gitleaks clean`;
+ - it would publish main `e56101fdee5d` as `20c367613f75`;
+ - a count-only grep of the log for `$(id -un)` gives **0**.
+- **A real `build homepage` in the worktree, with the real files:**
+ - **ok in 38 s** (the source step 19 s); `homepage/out` holds **2,640 files, 77.9 MB**;
+ - the mirror is 37.9 MB in 2 packs, the largest 20,966,235 bytes (19.99 MiB, against the 24 MiB
+ limit); the tree is 2,035 files and 412 dirs; the tarball 7,249,703 bytes.
+- **Dumb-HTTP clone** from `python3 -m http.server 8765 --bind 127.0.0.1` in `homepage/out`:
+ - HEAD `20c367613f75…` = `manifest.mirrorHead`;
+ - the user-name grep over its 1,703 revisions gives **0**;
+ - `:8765` was free before, the server was killed, and it was free after.
+- **The refusal, exercised through `build homepage`** (`SOURCE_DENYLIST_FILE` = a scratch file of
+ `Co-Authored-By`):
+ - exit **1** in 13 s: `1477 hits … denylist line 1 (len 14): 8 in blobs, 1469 in commits`, with
+ each listed as `commit <id> (message, byte N)`;
+ - the log holds the literal **0** times, and the source step's lines hold 0 of the 6 real
+ literals;
+ - afterwards `public/source`, the public tarball, the skip key, `out/source` and the out
+ tarball are **all gone**, and `out/downloads/index.html` stays;
+ - a good rebuild restored them all, and the deploy check then says "ok".
+ - Before the refusal, the deploy check over the real `out/` said ok under the real rules, and
+ "audited under other rules" with the scratch denylist.
+- **`build homepage --no-source`:** `out/source` holds only the page. The tarball is gone, and the
+ deploy check says ok.
+- **homepage e2e with `E2E_EXPECT_SOURCE=1`:** **36 passed**, 0 skipped, 51 s, with the manifest
+ present. The bite run is above: 3 failed and 2 passed in the empty state.
+
+**Re-review** (verdict **SHIP**; `$T/r-review.md`, "Re-review"). M1 is confirmed closed, and every
+deploy path goes through the check. The coordinator ruled that `--check` not withdrawing is
+accepted as built, and that the four new Lows be closed before the merge:
+- **R2-L1 — fixed** (`831763da`). A refusal that names a tree path (the symlink, `index.html` and
+ `404.html` refusals) printed a literal that spans path components (`a/b`) in full. The object
+ walk reads entry names one at a time, so it cannot see such a literal.
+ - `[source] REFUSED: …` now goes through `maskLiterals` with the loaded literals, in both
+ `publishSource` and `auditSource`.
+ - The kept-scratch path, the report's scrub-file path and the audit's "auditing <dir>" line are
+ masked too.
+ - Child stderr was already masked wherever it is quoted or echoed: git's refusal tails,
+ filter-repo's echo, gitleaks' lines. `subject` comes from the scrubbed mirror.
+ - The test plants `plant/secret` as a denied literal above a tracked symlink, then above a
+ tracked `index.html`. Both refusals print `[REDACTED]/…`.
+- **R2-L2 — fixed** (`831763da`). The state carries `contentDigest`, from `sourceDigest()`:
+ - it covers every published file: `source/archilyzer.git/**`, `source/tree/**`,
+ `source/manifest.json`, the tarball and `snapshot.json`;
+ - it hashes the sorted path, the size and a streamed sha256 of each; a symlink or a missing
+ piece only makes it differ;
+ - the skip recomputes it over `public/`, and the deploy check over `out/`.
+
+ A swapped tree file and a flipped pack byte are refused. On the real `out/` (2,640 files) the
+ whole check takes **0.58–0.76 s**, of which the digest is **0.48–0.54 s** (three runs).
+- **R2-L3 — fixed** (`831763da`).
+ - **The skip:** each part of the key (filter-repo, gitleaks, content digest, rules) and an edited
+ `public/` file is changed alone, through a publish that REACHES the skip. The filter-repo is
+ `false`, so a skip returns 0 and a miss returns 1.
+ - **The step version:** `rulesHashOf` is shown to move with it, and `loadSourceRules` uses
+ `SOURCE_STEP_VERSION`.
+ - **The deploy check's mirror-head and gitleaks comparisons** each refuse with their own
+ sentence.
+ - **Proved by reverting each fix** (`$T/r-m-mutate.py` against `source.ts`, restored after):
+ every revert fails its test. The eight reverts are:
+ - the deploy check's mirror-head comparison;
+ - the step version in the hash;
+ - filter-repo, gitleaks and the digest in the skip key (three reverts);
+ - gitleaks and the digest in the deploy check (two reverts);
+ - the refusal masking.
+- **R2-L4 — fixed** (`831763da`).
+ - The state carries `gitleaksIdentity()`: "skipped", "absent", or the version line plus the
+ binary's sha256. The sha is there because this machine's gitleaks prints "version is set by
+ build process" for every release.
+ - The skip and the deploy check compare it.
+ - `SOURCE_STEP_VERSION` is 3.
+- **The rollout note** (PUBLISH.md, `7940cb0c`, and here): **after the merge, rebuild and restart
+ the live editor (:3001) before any /sites Homepage job.** It runs its built bundle, so until then
+ its Build homepage job has no source step and its Deploy homepage job no source check.
+- **The reviewer's "minor" is left:** a real checkout that unexpectedly reads as "not a git
+ repository" (a worktree whose primary moved) builds with the empty state, withdrawing the
+ source. It is safe for privacy, and it is logged.
+
+| sha | what |
+|---|---|
+| `831763da` | `common:` refusals masked; `contentDigest` and `gitleaksIdentity` in the key; step version 3; the key's tests bite (R2-L1…L4) |
+| `7940cb0c` | `docs:` PUBLISH.md — the deploy key, masking, and rebuilding the editor after the merge |
+| _this_ | `plans:` this re-review record |
+
+**Gates after the re-review fixes:**
+- **tsc:** clean, 35 s.
+- **Unit:** common **2,149/2,149** (+3); the three filter-repo tests ran, 0 skipped. Homepage unit
+ **7/7**. `docs env --check` exits 0.
+- **`source publish --check` with the real files** exits **0** (19 s, would publish mirror head
+ `20c367613f75`). The count-only user-name grep of its log gives **0**.
+- **A real `build homepage`** is ok in 38 s (18 s for the source step). The source step's lines
+ hold 0 user-name occurrences.
+- **The deploy check, called read-only** on that `out/`, says **ok (would deploy)**.
+- **A rebuild with nothing changed** logs `up to date … skipping` (20 s), and the check still says
+ ok.
+- Nothing was deployed. `main` had not moved.
+
+## Rollout
+
+**Merged** (by the parent, in the primary, `git merge --no-ff` on a clean tree):
+- **Slice Q** merged as `4855f70b`, then the changelog fix `e6c5d2e3`. The 0.10.0 cut landed between
+ Q's branch point and its merge. A clean textual merge filed Q's bullet inside the released
+ `[0.10.0]` section, and `e6c5d2e3` moved it back under `[Unreleased]`.
+- **Slice R** merged as `ffdeb2cd`. The tree is identical to R's tip `484952ed`.
+- **The parent then prepared the operator's side:**
+ - it created `~/.config/archilyzer/` (mode 700) with the two files (mode 600): the scrub file
+ holds **3 rules**, the denylist **3 literals** — the Unix user name, the host name and one email
+ address. The contents are never printed;
+ - it installed `git-filter-repo` 2.47.0 with pipx (`~/.local/bin`);
+ - it ran `pnpm archilyzer source publish --check`: **exit 0**, it would publish main
+ `e56101fdee5d` as `20c367613f75`, with 2,459 files, 69.3 MB, 2 packs, 412 tree dirs and a
+ 6.9 MB tarball.
+ - The runbook's `r12-check.sh`, run in the primary while this record was written with `main` at
+ `ffdeb2cd`, also exits **0**. It would publish `ffdeb2cd770e` as `8188e02a7d04`, with 2,473
+ files, 69.8 MB, 2 packs, 413 tree dirs and a 7.0 MB tarball; its log holds 0 user-name and 0
+ host-name occurrences.
+- **Nothing is rolled out.** Nothing was deployed. The live :3001 editor runs 0.10.0 (`BUILD_ID`
+ `vWCJb87ktCy5akih_pM9X`, built on `e6c5d2e3`): it has no source step, no withdrawal and no deploy
+ check. The primary's `homepage/out` predates release 12 (it has no `/source` page), so the deploy
+ check refuses it until it is rebuilt.
+- **A side effect of the scrub, cosmetic and in the mirror only:** the bare user name is scrubbed to
+ `user`. The `plans/` text in the mirror (the grep gates, one risk sentence) therefore reads
+ differently from the private repository.
+
+**What is OWED, in order.** The operator's runbook is `~/reports/release-12/RUNBOOK.html`, rendered
+by `~/reports/release-12/make-runbook.py`. Its scripts are in `~/reports/release-12/scripts/` and
+log to `~/reports/release-12/tmp/`. Every script refuses unless the primary's `HEAD` contains
+`ffdeb2cd`.
+
+**Before ANY deploy, a preview included, the denylist must hold everything private.** A Pages
+preview is public, and every deployment stays reachable at its own `<hash>.archilyzer.pages.dev`
+until it is deleted.
+
+0. **The operator completes the denylist**, then runs the check. Add your real name, other handles
+ and anything else that must never appear to `~/.config/archilyzer/source-denylist.txt`: one per
+ line, `i:` for any case. Then:
+ ```
+ cd ~/Projects/yt-dlp-transcript-browser && pnpm archilyzer source publish --check
+ ```
+ (or `sh ~/reports/release-12/scripts/r12-check.sh`, which also counts user-name occurrences in its
+ log: expect 0).
+ - **Expect:** `[source] audit clean: …` and `[source] check passed — would publish main <12 hex>
+ as <12 hex>: 2459 files …; nothing written`.
+ - **If it refuses:** the report names the source only by position (`denylist line N (len L)`) and
+ each hit only by object, field and byte offset. Add a scrub rule to
+ `~/.config/archilyzer/source-scrub.txt` (`<text>==>user`), or drop the file from history. Then
+ re-run.
+1. **The song link, before any umtool restart** (slice Q's operator step, with the review's I1
+ correction):
+ ```
+ mkdir -p ~/.local/share/archilyzer && ln -s ~/.claude/jobs/efbe67a7/tmp/song ~/.local/share/archilyzer/song
+ ```
+ - The target holds only umtool's rebuildable `.cache/umtool`. The song project's bulk data is in
+ `~/reports/quartering-uh-song/data`; pointing the link there instead is a separate choice.
+ - `~/.local/share/archilyzer` did not exist on 2026-09-28. If a restarted :3050 created
+ `…/song/.cache` as a real directory first, remove that directory before linking, or the link
+ lands inside it.
+2. **Rebuild and restart the live editor, with `~/.local/bin` on its PATH.** The precedent is
+ release 11's `r11-build.sh` + `r11-restart.sh` in `~/reports/overnight-2026-09-28/scripts/`.
+ Release 12's copies are `r12-build.sh` and `r12-restart.sh`:
+ - `r12-build.sh` builds into the live `.next`. Run the restart right after it.
+ - `r12-restart.sh` refuses while `BUILD_ID` is still `vWCJb87ktCy5akih_pM9X`. It starts the
+ editor with `PATH=$HOME/.local/bin:$PATH` and checks that the new server's environment has it.
+ Without it, `/sites` Build homepage falls back to `pipx run`, which needs the network.
+ - Run the md5 sweep around the restart, and the smoke after it. `r12-md5.sh` and `r12-smoke.sh`
+ wrap `plans/tools/rollout/md5.sh` and `smoke.sh` with `CLAUDE_JOB_DIR=~/reports/release-12
+ REL=r12`:
+ ```
+ sh ~/reports/release-12/scripts/r12-md5.sh before
+ sh ~/reports/release-12/scripts/r12-build.sh
+ sh ~/reports/release-12/scripts/r12-md5.sh pre-restart
+ sh ~/reports/release-12/scripts/r12-restart.sh
+ sh ~/reports/release-12/scripts/r12-md5.sh after-boot
+ sh ~/reports/release-12/scripts/r12-smoke.sh
+ cd ~/Projects/yt-dlp-transcript-browser && pnpm archilyzer doctor
+ ```
+ - **Expect:** `BUILD_OK`, then `RESTART_DONE editor / 200`, `/sites` carries release 12's
+ sentence ("also publishes the source mirror") and the editor's PATH has `.local/bin`. `r12-md5.sh after-boot` ends `MD5_SAME`, and
+ the smoke ends `SMOKE_FAIL=0`. A `PAIR_DIFF` on a pair that moves live (auto-queue status) is
+ drift, as it was at release 11.
+ - **Doctor** has a **source publish** block:
+ - `filter-repo git filter-repo a40bce548d2c`;
+ - `gitleaks` ok;
+ - `scrub rules … (3 rules, mode 600)` and `denylist … (N literals, mode 600)`;
+ - `published` says nothing is published in the primary yet.
+3. **A FRESH homepage build in the primary.** The deploy check refuses any `out/` built before
+ release 12, because it has no `/source` page.
+ ```
+ sh ~/reports/release-12/scripts/r12-home-build.sh
+ ```
+ - It runs `pnpm archilyzer build homepage`, then checks
+ `homepage/out/source/archilyzer.git/info/refs`, the file count and the deploy check, read-only.
+ - **Expect:** `[source] published main … as …: 2459 files` (about 19 s for the step), `out: ~2,640
+ files`, `info/refs: <40 hex>\trefs/heads/main`, and `deploy check: ok (would deploy)`.
+ - **If the build refuses:** it has already WITHDRAWN the source from `public/` and `out/`. Fix the
+ rule (step 0) and rebuild.
+4. **The preview** (`source.archilyzer.pages.dev`; public):
+ ```
+ sh ~/reports/release-12/scripts/r12-preview.sh
+ sh ~/reports/release-12/scripts/r12-live-check.sh https://source.archilyzer.pages.dev
+ ```
+ - The first script runs `pnpm archilyzer deploy homepage --preview source`, which asks the deploy
+ check first. Its log ends `[preview] https://source.archilyzer.pages.dev (this deployment:
+ https://<hash>.archilyzer.pages.dev)`.
+ - The live check is `source-mirror.md` Rollout step 2, as commands. Each line prints OK or FAIL:
+ - `git clone https://source.archilyzer.pages.dev/source/archilyzer.git` works;
+ - the clone's HEAD = `manifest.json`'s `mirrorHead`;
+ - `…/source/tree/common/lib/paths.ts` is `content-type: text/plain; charset=utf-8`, ONE value,
+ with `x-content-type-options: nosniff`;
+ - `…/source/tree/common/` is `text/html; charset=utf-8`, ONE value. The appended form
+ `text/plain…, text/html…` is the bug `f218ed86` fixed;
+ - `…/source/tree/umtool/report-to-video/fonts/Archivo%5Bwdth%2Cwght%5D.ttf` is 200 `font/ttf`;
+ - `…/source/tree/homepage/app/docs/%5Bslug%5D/page.tsx` is 200 `text/plain; charset=utf-8`;
+ - the tarball's sha256 = `snapshot.json` = `manifest.json` = the one on `/source/`;
+ - `pnpm archilyzer source audit <clone>/.git` is clean;
+ - user-name and host-name counts in the clone's history are 0.
+5. **Production:**
+ ```
+ sh ~/reports/release-12/scripts/r12-prod.sh
+ sh ~/reports/release-12/scripts/r12-live-check.sh https://archilyzer.pages.dev
+ ```
+ - The first script runs `pnpm archilyzer deploy homepage`: branch `main`, the log ends
+ `[deployed] https://<hash>.archilyzer.pages.dev`.
+ - Or use the editor's path, which also proves the step inside its job: `pnpm ops build-homepage
+ --json '{"deploy":true}' --wait` with `WORKER_TOKEN` from `editor/.env`.
+ - The same checks must pass on `archilyzer.pages.dev`.
+6. **If the edge refuses the dumb clone** (`source-mirror.md` Rollout step 4): the tree and the
+ tarball still stand. The mirror would need another host (R2 behind a custom domain, out of scope).
+ Record it, do not improvise.
+7. **If something private ever ships:**
+ - DELETE THAT DEPLOYMENT in the Cloudflare dashboard (Workers & Pages → `archilyzer` →
+ Deployments). A newer deploy does not remove it; for a preview, delete every deployment on
+ that branch.
+ - Then add the literal (and a scrub rule) and run `pnpm archilyzer build homepage`, which refuses
+ and withdraws. Rebuild clean, and deploy again.
+8. **Housekeeping:**
+ - `pnpm archilyzer release show` has 2 editor bullets pending; export has none, so `all` would
+ refuse. When the operator chooses: `pnpm archilyzer release cut editor next --commit` (0.10.1)
+ or `next-minor` (0.11.0).
+ - Remove the worktrees when done: `pnpm wt rm r12-paths-fix` and `pnpm wt rm r12-source-mirror`,
+ from the primary. Removing them re-sorts the index-based port blocks, so do it only when no
+ dev server or e2e runs in any worktree; the parallel session's `r13-*` worktrees are active.
+ The seven `r11-*` wait on these.
+ - Optional: `git branch -d r12/paths-fix r12/source-mirror`.
+
diff --git a/plans/release-14.md b/plans/release-14.md
@@ -0,0 +1,1724 @@
+# Release 14 — the social icons within reach, and a clear screen until the first Search
+
+`main` at `ac438bbc` (release 12 merged and not rolled out; release 13 is a parallel session's).
+Plan: [`export-header-first-search.md`](export-header-first-search.md), written 2026-09-28, with
+slice HP added to it on the operator's ruling of the same day. Rules:
+`plans/tools/implementer-rules.md`, with the commit trailer this release's prompts give.
+
+**The operator's standing choices** (the plan, "The operator's words"; not re-opened):
+- **No copy** beside any social link: icons with accessible names only.
+- **A link the operator adds is an entry in `settings.json` `socialLinks`, never code.** The earlier
+ tip-link branch is parked and not merged; nothing is taken from it.
+- **No vendor file in the repository**, not even as a test fixture: the tracked tree is published
+ by the source mirror.
+- **Nothing is edited in the primary checkout**; each slice has its own worktree, and the parent
+ merges with `git merge --no-ff` only on a clean tree.
+
+## The slices
+
+| Slice | Branch | What | Owns |
+|---|---|---|---|
+| HP | `homepage/social-visible` | The homepage's social row and one theme toggle in the header at every width, the wordmark's text dropped first on a very small screen and the row scrolling only as the last resort, one shared `SocialLinks` component, larger keys with a focus ring; Changelog in the footer only; the instance cards' names as the site's wordmark; the social icon checked by an allowlist on save and at render; a sized SVG with no viewBox gets one; `featured` ("Show in header") on a social link | `common/components/{SocialLinks,SocialScroll,ThemeRadios,ThemeToggle,ThemeScript,ThemeProvider,Wordmark}.tsx` + `themeConfig.ts`, `common/bin/doctor.ts`, the growth chart, `common/lib/{socialSvg,socialLinks}.ts` + tests, `common/lib/settingsSchema.ts` (the social-link type, parser, docs; the normalizer moved to `socialSvg.ts`), `common/lib/normalizeSocialSvg.test.ts`, `common/lib/{settings,site,homepage}.ts` (the save errors), `common/lib/{homepageSummary,siteColor}.ts`, `homepage/app/components/{Header,Footer,ArchiveCards}.tsx`, `homepage/app/lib/{nav,summary}.ts`, `homepage/app/not-found.tsx`, `homepage/e2e/**` (the fixtures, `helpers.ts`, the new and the rewritten specs), `homepage/playwright.config.ts`, `export/app/components/{MobileMenu,Footer}.tsx` (the `ThemeRadios` swap; the footer's read path), `editor/app/components/SocialLinksField.tsx` + `socialLinksJson{,.test}.ts`, `editor/app/{settings,sites}/actions.ts` (the save errors), `editor/e2e/settings.spec.ts`, `SETTINGS.md`, `SITE.md` |
+| Lows, chart gap, T1, H1 (H3 folded in), H2 | `r14/two-grounds-headers` | The final review's Lows; the charts' surface gap; two grounds and each site in its own accent; the export and hub headers carry the social row and the toggle as one group, with an Archilyzer link to the homepage's `#instances` in place of the sites dropdown and the hub link; Changelog to the footer; after its review, the narrow header keeps the name and shows only the marked links (every header) | `common/components/{ThemeProvider,ThemeScript,ThemeToggle,SocialScroll}.tsx` + `themeConfig.ts` (and the deleted `ThemeMenu`, `ThemeRadios`), `common/components/charts/{ChartView,CrossSiteChart,surfaceGap}`, `common/styles/tokens.css`, `common/lib/{brand,accent,siteColor,paths,project,socialSvg,siteSchema,settingsSchema}.ts` + tests, `scripts/next-build-trace.test.mjs`, `export/app/components/{Header,MobileMenu,Footer}.tsx` (and the deleted `SiblingSwitcher`), `export/app/{layout.tsx,globals.css,changelog/page.tsx,lib/brand.ts}`, `export/e2e{,-hub}/**` (the theme, header and branding specs), `export/playwright.config.ts`, `editor/app/{layout.tsx,globals.css,sites/components/SiteForm.tsx}`, `editor/e2e/theme.spec.ts`, `homepage/app/{page.tsx,layout.tsx,globals.css,lib/*,changelog/page.tsx,components/{Header,ArchiveGrowthChart,ArchiveCards}.tsx}`, `homepage/e2e/**`, `homepage/content/docs/operate.md`, `SETTINGS.md`, `SITE.md` |
+| HS | `r14/hidden-sites` | Hidden sites: `site.json` `listed` (absent = listed). An unlisted site builds and deploys as before, and is left off the homepage (cards, chart, `/stats`), the hub (members, federated search, `corpus.json`, `llms.txt`), every other site's footer and the published id lists (`channel-sites.json`, the pooled `stats/`); the channels only it exposes count in no public total. One checkbox in the site form | `common/lib/{siteSchema,site,homepageSummary}.ts` + tests, `common/controller/{poolSummary,buildStats}.ts` + tests, `common/bin/{compose-homepage,compose-hub}.ts` + `compose-hub.test.ts`, `editor/app/sites/{actions.ts,components/SiteForm.tsx}`, `editor/e2e/{helpers.ts,sites-crud.spec.ts}`, `homepage/e2e/{fixture-summary.ts,unlisted-site.spec.ts}`, `SITE.md`, `plans/FACTS.md` (Naming hazards) |
+| S1 | `r14/first-search` | A clear screen until the first Search, on every site's search page and the hub's: no results area until the visitor asks (a Search, a profile load, or a link that carries a query or a filter), with the bar's "Press Enter or click Search to apply" line meanwhile; two page-life flags, the hold of release 8 unchanged and the gate new | `common/components/{SearchSessionContext,SearchResults,SearchBar}.tsx`, `export/e2e/first-search.spec.ts` (new), `export/e2e/{browse-all,workspace-shell,charts,restore-no-refire,responsive,tag-chips}.spec.ts`, `export/e2e/helpers.ts` (`showAll`), `export/e2e-hub/federated-search.spec.ts`, `export/CHANGELOG.md`, `plans/export-header-first-search.md` |
+| CF | `r14/chart-fold` | The homepage's growth chart folds two or more sites each under 5 % of its total into one Other band on top, in its own near-neutral grey (`--chart-other`); the kept sites keep their colours; the legend shows them and Other, the hover titles and the table every site | `homepage/app/lib/growthGaps.ts` + test, `homepage/app/components/ArchiveGrowthChart.tsx`, `common/styles/tokens.css` (`--chart-other`), `homepage/e2e/{fixture-summary.ts,growth-chart.spec.ts,instance-colours.spec.ts,marketing.spec.ts}`, `homepage/CHANGELOG.md`, `plans/FACTS.md` |
+
+**Order:** HP → `r14/two-grounds-headers` → HS (`r14/hidden-sites`) and S1 (`r14/first-search`),
+siblings → CF (`r14/chart-fold`), off `main` after HS, with `main` merged in after S1. The shared
+files are the three changelogs' `[Unreleased]` sections and this record.
+
+## Record
+
+### Slice HP, as shipped — the homepage shows the social links where they are seen (2026-09-28)
+
+Branch `homepage/social-visible` off `main` `ac438bbc`, worktree `~/Projects/homepage-social-visible`
+(block #3: editor 3301, test 3311, homepage e2e 3340, homepage static 3331), one Opus implementer.
+Scratch files `hp-*` in the job's `tmp`. The rulings, all 2026-09-28, built in this order on one
+branch:
+1. The homepage's social links more visible; a change to the social-link schema where it has a
+ reason.
+2. The Changelog link moves to the footer; Base and Accent move behind a single options button with
+ a gear icon that opens a modal.
+3. The gear sits in the social icons' rhythm, with no gap of its own; the narrow header carries the
+ icons and the gear in its bar.
+4. On Official Instances, each site's name uses the bold-lead effect the sites' own headings use,
+ with a slight tint or underline in the site's accent colour.
+5. (The review, ruled by the parent.) No vendor file in the repository; the icon check hardened on
+ save and at render; the shared dialog wrapper unchanged.
+6. On the homepage chart, light mode has dark lines and dark mode has light lines.
+7. On very small screens the header keeps its mark and drops the word; the social icons scroll only
+ when they still cannot fit (rather than hiding icons on small screens).
+8. The options menu is dropped for now in favour of a three-way toggle.
+9. (The re-review, ruled by the parent.) A title or desc can no longer reach a page; an icon loads
+ nothing from elsewhere; a stored icon the checker refuses does not block an unrelated save.
+10. (After the merge, 2026-09-28.) The chart follows standard practice for light and dark grounds;
+ the foreground-coloured separator lines are withdrawn.
+
+**The branch's history was rewritten once** (after the review, ruling 5): its first nine commits,
+one of which added a vendor file as a test fixture, were replaced by the five commits below,
+re-committed from the same tree in the same logical steps, each tsc-clean. No vendor file is in
+any commit of `main..HEAD`. Everything after `afc642fd` is new commits on top, and `main`
+is merged in twice (`10cefd15`, then `918e5f85`).
+
+**The Options dialog was built, then replaced by the toggle** (ruling 8): `OptionsDialog` (a gear
+opening a modal with Base and Accent) shipped in `4d93dc84` and was deleted in `5da60518`, with
+`options.spec.ts` and its screenshots. `ThemeRadios` stays: the export's slide-out menu renders it.
+
+**What shipped.**
+- **One row, `common/components/SocialLinks.tsx`.**
+ - Props `links`, `placement: "header" | "footer"`, `className` (on the `<ul>`, which carries
+ `data-social-links="<placement>"`); nothing site-specific.
+ - Each link is a 36 px key (`size-9`) around the 20 px glyph, 44 px under a coarse pointer
+ (`pointer-coarse:size-11`, Tailwind v4's own variant). The glyph is `text-muted-foreground`,
+ `hover:text-foreground`, with a `hover:bg-muted` key, so a multi-colour icon has a hover state
+ too. Focus is `focus-visible:ring-2 focus-visible:ring-ring` with
+ `not-forced-colors:focus-visible:outline-none`: in forced colours a box-shadow is not drawn,
+ and the browser's own outline is left in place. Nothing is drawn at rest.
+ - `aria-label` and `title` are the label, `target="_blank" rel="noopener noreferrer"`, and the
+ link has no text.
+ - The header placement shows at most four (`headerSocialLinks`, below); the footer every link.
+ - Every icon passes the save-time check AGAIN before it is inlined (`safeSocialSvg`, below). One
+ that fails is not injected: the link shows its label as text.
+ - The ids inside each inlined icon are scoped per row and per link (`scopeSvgIds`, below).
+ - It uses only `useId`, so it works in a server or a client tree.
+- **The header bound, `headerSocialLinks`** (`common/lib/socialLinks.ts`, pure). At most four: the
+ links marked `featured` when any is marked, else all of them; of those, the last four. Marking is
+ choosing: one marked of three shows one.
+- **The homepage header** (`homepage/app/components/Header.tsx`): one group — the social row, then
+ the theme toggle — in the bar at every width. The toggle is dressed as a key and its box sits
+ directly after the last link's, so every glyph is 16 px from the next. From `md` (768 px) the bar
+ is wordmark · nav · group, the nav's last link 32 px before the group's first box (40 px before
+ its first glyph; the nav's own links are 24 px apart). Below `md` it is wordmark · group, and the
+ four nav links have the rule below to themselves, where they fit at 320 px. No link is hidden by
+ width (ruling 7): on a very small screen the wordmark's text goes first, and the row scrolls only
+ as the last resort (see "The header layout").
+- **Changelog is in the footer only** (ruling 2). `homepage/app/lib/nav.ts` declares `HEADER_NAV`
+ (Docs, Source, Downloads, Stats) and `FOOTER_NAV` (the same, then Changelog). The footer's
+ Sections and the 404 page read `FOOTER_NAV`.
+- **One theme toggle** (ruling 8, replacing ruling 2's dialog): `common/components/ThemeToggle.tsx`
+ with a new `variant="bare"` — dressed as a social key (36 px, 44 px under a coarse pointer, no
+ border, the hover square, the ring, a 20 px glyph), cycling the base in `nextBase`'s order and
+ named "Switch to {next}". Its default rendering, which the export and the editor use, is
+ unchanged.
+ - **The accent is pinned on the homepage**: with no accent control there, `ThemeScript` and
+ `ThemeProvider` take `pinAccent`, so the stored `ytdlp-tb:accent` is neither read nor removed
+ and the homepage keeps Signal, with no flash (the pre-paint script skips the read). Without the
+ prop both are unchanged (unit-tested: the script string is identical).
+- **The instance cards' names** (ruling 4).
+ - `homepage-summary.json` `sites[]` gains an optional `wordmarkLead`: site.json's, resolved
+ against `siteTitle` by `lib/brand.ts` `wordmarkLeadFor`, the resolver the sites' header and
+ `siteSchema` use. It is still version 5: nothing reads the version to accept a file.
+ - The homepage's loader keeps a lead only when it is a proper prefix of the title
+ (`withCheckedLead`), so an older or hand-edited summary shows the plain title.
+ - `ArchiveCards` sets the title with the shared `Wordmark` (lead 720, suffix 380) at the card
+ title's size, and tints the lead in the site's own accent: `siteAccentColor`, `siteColor`'s
+ accent half. A site with a lead and no accent keeps the foreground. `Wordmark` gains
+ `leadStyle`.
+ - `headerTitle` is not carried: the card's accessible name is `siteTitle`, and every live site
+ has the two equal.
+- **The social icon's SVG, checked by an allowlist** (`common/lib/socialSvg.ts`, pure;
+ `settingsSchema.ts` re-exports `normalizeSocialSvg` and `socialSvgProblem`). The allowlist was
+ chosen over the denylist: the element and attribute lists stay small, and every shape the
+ existing tests and icons use passes.
+ - The input is read tag by tag. It must be ONE well-formed `<svg>`: tags closed in order,
+ attributes separated by HTML whitespace and quoted, no `<!…>`, CDATA or processing
+ instruction. Comments, a leading XML declaration and a leading DOCTYPE with no internal subset
+ are removed first.
+ - Elements: shapes, groups, `defs`, `symbol`, `use`, gradients, `stop`, `pattern`, `clipPath`,
+ `mask`, filters, `text`/`tspan`, and `animate`/`animateTransform`/`set`. No `script`, `style`,
+ `foreignObject`, `a`, `image`, `title`, `desc` or HTML element (R1: a title or desc is an HTML
+ integration point, where a child element left the icon unclosed around the page; one holding
+ text only is removed first).
+ - Attributes: the SVG presentation, geometry, filter and animation set, plus `aria-*`, `data-*`
+ and `xmlns:*`. No event handler, whatever separates it.
+ - Values: character references (numeric and named, with or without `;`) are decoded and the
+ whitespace a browser ignores in a URL dropped before the checks. No `javascript:` or
+ `vbscript:`; no backslash (a CSS escape) or CSS comment; no function that loads anything
+ (`image-set(`, `-webkit-image-set(`, `image(`, `cross-fade(`, `element(`, `src(`, `paint(`,
+ `@import`, `expression(`) — an animation's `to`/`from`/`values`/`by` included (R2).
+ `href`/`xlink:href` only a plain `#id` as written; every `url(…)` only to `#id`, spelled so the
+ render's scoping rewrites it (R6). A `style` holds presentation properties only (fill, stroke,
+ stop-color, the opacities, stroke width/caps/joins, fill-rule, clip-rule, display, visibility,
+ paint-order). An animation never targets anything named `href` or a handler (R5). Ids are
+ plain names (`^[A-Za-z_][\w.:-]*$`).
+ - The normalized OUTPUT is checked again, so no transform can assemble what the input check
+ refused.
+ - `socialSvgProblem` names the reason class. The editor's save error and the writers' errors
+ append it ("… has an invalid SVG: it has an event handler attribute."); none echoes the markup.
+ - `SETTINGS.md` and `SITE.md` (generated) say what an icon may contain.
+- **The read path.** `safeSocialSvg` (`common/lib/socialLinks.ts`) runs the check again at render.
+ `SocialLinks` and the export footer inline only what passes; a link whose icon fails shows its
+ label as text, capped at 10rem with an ellipsis and the full label in `title` (R7). Every
+ `dangerouslySetInnerHTML` of a social SVG goes through it (there are two). Each key clips its
+ icon's paint (`overflow-hidden`, `contain: paint`; R4); the focus ring is the key's own shadow,
+ outside that clip.
+- **The write path** (R3): `socialLinksForSave` keeps a link whose `svg` is byte-identical to one
+ already stored exactly as it is, and checks only a new or edited icon — in `writeSettings`,
+ `writeSite`, `writeHomepageConfig` and the editor's settings and site actions. So a lane pause, a
+ priority or a title never fails on an icon an older build stored; the render shows such an icon
+ as its label, and `archilyzer doctor` names it ("social icons": file, label, reason class, never
+ the markup). In this worktree the doctor line reads "3 icons in 1 file pass the check".
+- **The size rule** (ruling 1): a root with no viewBox but a numeric `width` and `height`
+ (unitless or px, decimals, either quote) gets `viewBox="0 0 W H"`, read from the root's own
+ attributes (a size spelled inside another attribute's value does not count). A percentage,
+ `em`, `auto`, a negative, or a missing or zero side is still refused, and a viewBox already
+ there is never replaced.
+- **`featured`**: optional on `SocialLink`, parsed only when exactly `true` and stored only when
+ true, through `parseSocialLinks`, so `settings.json`, `site.json` and `homepage.json` all read
+ it. The editor's `SocialLinksField` has a **Show in header** checkbox per row (Settings and a
+ site's form), and beside it "With none checked, the header shows the last four." An old file
+ with neither change parses byte-identically. `featured` also reaches each site's public
+ `/site.json` (`buildSiteDescriptor` passes the link through): accepted, additive, and the hub
+ does not render federated sites' links.
+
+**The header layout** (built and screenshotted before each choice). Measured: the wordmark link is
+148 px (152 px at 1× device scale), the four-link nav 276 px, a key 36 px (44 px under a coarse
+pointer), the toggle the same.
+- **No link is hidden by width** (ruling 7): the header shows its (at most four) links at every
+ width. The bar is a size container (`@container/bar`), and when it is narrower than the full
+ wordmark, a 12 px gap and the group need, the wordmark's TEXT is hidden and the mark stays (the
+ link is still "Archilyzer home"). The threshold is a class per link count and pointer, in rem so
+ it follows the text size (`WORDMARK_FITS`, measured, not guessed):
+
+ | Links | Full wordmark with a mouse | Full wordmark with touch |
+ |---|---|---|
+ | 1 | from 276 px | from 292 px |
+ | 2 | from 312 px | from 336 px |
+ | 3 | from 348 px | from 380 px |
+ | 4 | from 384 px | from 424 px |
+
+ Measured at the threshold the gap is exactly 12 px; 1 px below, the text is hidden.
+- **With the mark alone** (the fixture; the gap between the mark and the group): with a mouse, 3
+ links — 320: 108 px, 340: 128; 4 links — 320: 72, 340: 92, 360: 112. With touch, 3 links — 320:
+ 76, 340: 96, 360: 116; 4 links — 320: 32, 340: 52, 360: 72. Four 44 px links, the toggle and the
+ mark fit a 320 px screen.
+- **The last resort** (`common/components/SocialScroll.tsx`): the row
+ scrolls inside a box, its END shown first (the box is `direction: rtl`, the row `ltr` and
+ `w-max`, so the first scroll position is the right edge — no script, no change to the DOM or tab
+ order), no scrollbar, the toggle outside it, the header never wider than the screen. A link that
+ takes focus is scrolled into view with its ring (the one bit of script: Chromium does not bring
+ a partly hidden link into view inside an rtl box). From 300 px up, with default text, nothing
+ scrolls; it takes 280 px with four 44 px links, or 320 px at a 200 % text size. Checked in
+ Chromium and in an installed Firefox build (end first, nothing sideways); WebKit UNVERIFIED (the
+ installed build does not match this Playwright).
+- **What a visitor sees, by width** (default text size): the full wordmark from the table's width
+ up; the mark alone below it, down to 300 px; the scroll fallback only below that, or at a much
+ larger text size.
+- **Two rows at 360 px** (a mouse, three links): the wordmark (20–168), 28 px, the group (196–340:
+ three keys and the toggle), the 20 px gutter; below, the four nav links (20–291), 49 px to spare.
+- **How it got here:**
+ - First build, with two 32 px theme buttons and five nav links: three keys in the bar made a
+ 360 and a 390 px page scroll sideways by 36 px, so the keys were pinned at the end of the nav's
+ rule, the nav in that rule below `lg`, and the row rendered twice.
+ - Rulings 2 and 3 freed the bar: one gear, four links, and the group in the bar at every width.
+ - The review found 320 px with a mouse scrolled sideways by 8 px; a width × pointer rule then hid
+ links step by step (`6c500818`).
+ - Ruling 7 replaced that rule with the collapsing wordmark and the scroll fallback (`0b3189fb`),
+ and ruling 8 the gear with the toggle, the same size (`5da60518`).
+
+**The cards' accent treatment** (ruling 4). Both treatments were built and screenshotted on the
+three grounds at 390 and 1280 px, from a family-like summary: the six live titles, their site.json
+leads (Jer, Hasan, Ani, Bonnell, Rekieta, Jaso) and their named accents; the numbers are synthetic.
+- **Shipped: the tint.** The lead is in the site's accent, the suffix in the muted foreground.
+ - The accent is one signal on the name and matches the card's stripe.
+ - The underline sat under the card link's own hover underline, and on hover the lead drew two
+ rules. An accent rule also reads as a link state.
+- **Contrast of the lead against the card** (`--surface`): every named accent is at least 4.24:1 on
+ Light, 4.21:1 on Sepia and 6.75:1 on Dark. A custom hex fitted to 4.5:1 on the ground is about
+ 4.1:1 on the card. The e2e checks ≥ 3:1 from the rendered colours.
+- **The swap**, if asked: replace `{ color: accent }` in `ArchiveCards`' `SiteName` with a 2 px
+ accent `text-decoration` on the lead. It is one line.
+
+**The chart's separators** (ruling 6; after the review, a new commit on top). The line along each
+stratum's upper edge in `homepage/app/components/ArchiveGrowthChart.tsx` was the page background
+(`var(--background)`, 1.25 px): light on Light, dark on Dark. It is now `.growth-sep` in
+`homepage/app/globals.css`: `var(--foreground)`, full strength, 1 px, and `CanvasText` in forced
+colours. The gridlines are unchanged.
+- **Three strengths built and screenshotted** (the foreground at 100 %, 60 % and 35 %, 1 px; the
+ chart at 1280 and 390 px and a 4× close-up of the thin strata, on the three grounds; the
+ published summary): `~/reports/release-14/shots/chart/sep-{100,60,35}-…`.
+- **Shipped: 100 %.** Only full strength reaches 3:1 against any band; 60 % reaches it against
+ none (at best 2.69:1) and 35 % against none (1.81:1). At 1 px, a 2–3 px stratum keeps a coloured
+ stripe between its two lines in the close-ups; the old gap was 1.25 px.
+- **Against the six bands, measured on the page** (the separator against each band's rendered
+ colour):
+ - Light: blue 4.25, amber 4.24, green 4.00, magenta 3.80, violet 2.17, rust 2.13 — minimum
+ **2.13**;
+ - Sepia: amber 3.55, blue 3.17, magenta 2.95, green 2.84, violet 1.81, rust 1.78 — minimum
+ **1.78**;
+ - Dark: rust 4.77, blue 4.63, magenta 3.64, violet 2.63, amber 2.54, green 2.49 — minimum
+ **2.49**.
+ With the ground's foreground, 3:1 is out of reach against those bands: Light violet and rust;
+ Sepia green, magenta, violet and rust; Dark green, violet and amber. The series colours are
+ unchanged, as ruled for this commit.
+- **Homepage-only.** The separator is in the homepage's own chart. The shared charts
+ (`common/components/charts/`, the homepage's `/stats`, the export sites and the hub) draw each
+ series' edge in its own colour and have no page-coloured separator. Whether they take a
+ foreground separator is left for the operator to rule on.
+- **Withdrawn (ruling 10).** The separators were first drawn in the foreground colour, then
+ withdrawn for a 2 px surface gap on the operator's ruling; that change is on
+ `r14/two-grounds-headers` (its record, below).
+
+**Beyond the prompt.**
+- **Icon ids are scoped per copy (`scopeSvgIds`).** An id resolves to the first element carrying
+ it, and a gradient defined inside a `display: none` copy does not paint. The first build rendered
+ the header's row twice; with the ids unscoped, the gradient icon at 360 px lost its body and kept
+ only its themed dot. The header's row is one element now, and the scoping stays: the footer
+ inlines the same icons, and the export's header (H1) may render a row per breakpoint. Only plain
+ ids are prefixed; references in `url(#…)`, `url('#…')`, `url("#…")`, `href` and
+ `xlink:href` are rewritten, case-insensitively for the attribute names. A scoped, sized icon
+ passes the checker again (unit-tested).
+- **`sizeSocialSvg` moved** to `lib/socialLinks.ts` and is re-exported from `settingsSchema`.
+- **The homepage e2e read the checkout's own `settings.json` and `homepage.json`.** The dev server
+ now reads `e2e/.e2e-settings.json` through `SETTINGS_FILE` and an empty `e2e/.e2e-sites/` through
+ `SITES_DIR` (both declared variables). The icons are synthetic: a gradient with one solid colour,
+ one colour drawn white, and a two-colour disc with a letter pasted with only its size. Specs that
+ need six links, none, or hostile icons rewrite the file (hostile ones raw, as a hand-edited file
+ holds them), and an `afterEach` restores the trio. `marketing.spec.ts`'s "no link to ko-fi.com"
+ runs against the fixture's links and still catches a link in code.
+- **The footer's icons are `--muted-foreground`, no longer `--faint`**, the same as the header's.
+ On the page ground that is 5.63 / 5.51 / 7.08:1 (Light / Sepia / Dark) against `--faint`'s
+ 3.45 / 3.59 / 3.53:1.
+
+**Corrections to the prompt.**
+- `NAV` had five entries at `ac438bbc`, not six.
+- The homepage e2e baseline at `ac438bbc` was 36 passed, 0 skipped (1.0 min) in the empty-source
+ state. The parent later copied a published source into the worktree, so the later runs are in
+ the published-source state.
+
+**What each icon looks like, per ground** (the fixture's synthetic icons, which behave as an
+operator's pasted icons do):
+- **The gradient icon:** the gradient keeps its colours on every ground. Its one solid part follows
+ the link colour: slate on Light, brown on Sepia, warm grey on Dark, and the foreground on hover.
+- **The one-colour icon:** the muted foreground on every ground (`#55646e` / `#6b5c43` /
+ `#a39a86`), and the foreground on hover.
+- **A two-colour disc with a dark outline and a dark offset disc:** on Light and Sepia the outline
+ and the offset crescent show; on Dark both fall into the `#0c0a08` ground, and the disc reads flat.
+ This is accepted, as ruled: no per-icon ring.
+- **In forced colours:** single-colour parts take the link colour; the disc and the gradient keep
+ theirs.
+
+| sha | what |
+|---|---|
+| `518dd272` | `common:` `lib/socialSvg.ts`, the allowlist checker on save and at render with reasons; the size rule; `featured`; `lib/socialLinks.ts` (header rule, `safeSocialSvg`, id scoping, `sizeSocialSvg`); the writers' errors; the tests; `SETTINGS.md` / `SITE.md` |
+| `a9dfa3de` | `editor:` "Show in header" with its hint; `socialLinksJson` + unit test; the save errors name the reason; the settings spec reads `featured` back |
+| `4d93dc84` | `common:` `SocialLinks`, `OptionsDialog` (later deleted), `ThemeRadios`; `export:` `MobileMenu` renders `ThemeRadios`, the footer inlines only through `safeSocialSvg` |
+| `6c500818` | `homepage:` the social row and the gear in the header at every width, the width × pointer rule (later replaced); Changelog in the footer only; the e2e's own settings and sites; the specs |
+| `88d61908` | `homepage:` the cards' names as the site's wordmark, the lead tinted; `wordmarkLead` in the summary and its check on read; `siteAccentColor`; `Wordmark` `leadStyle`; `instance-wordmark.spec.ts` |
+| `afc642fd` | `plans:` this record; the plan's slice HP, H1 and H2; STATE; the changelogs |
+| `031c2115` | `homepage:` the growth chart's separators in the ground's foreground, 1 px, full strength (ruling 6); `growth-chart.spec.ts`; the record, the plan, the changelog |
+| `0b3189fb` | `homepage:` the wordmark's text drops first on a very small screen (a container query per link count and pointer); `SocialScroll`, the last-resort scroll box, end first; no link hidden by width (ruling 7) |
+| `5da60518` | `homepage:` one toggle (`ThemeToggle variant="bare"`) in place of the Options dialog; the homepage's accent pinned (`pinAccent`); `OptionsDialog` and `options.spec.ts` deleted; `toggle.spec.ts` (ruling 8) |
+| `68ad1464` | `common:` the checker drops title/desc, refuses loading functions, escapes and comments, allows presentation styles only, plain references, reasons that say how to export; the review's vectors (re-review R1, R2, R5, R6, R8, R9) |
+| `0be068cf` | `common, editor:` `socialLinksForSave` — an unchanged stored icon never blocks a save; `archilyzer doctor` names stored icons that fail (R3) |
+| `4f7bc097` | `common, export, homepage:` each key clips its icon's paint; a refused icon's label bounded; `svg-vectors.spec.ts` (the parser invariant, nothing fetched); the hostile e2e checks no other origin (R1, R2, R4, R7) |
+| `4d11542c` | merge `main` (`10cefd15`, the stats fix); `homepage/CHANGELOG.md` both sides kept |
+| `fe9fb4b2` | merge `main` (`918e5f85`, slice Q's umtool build trace); `editor/CHANGELOG.md` both sides kept, `main`'s bullet under the one it refers to |
+| _this_ | `plans:` this record; the plan's slices HP, H1, H2 and the new T1; STATE; the changelogs |
+
+**Gates** (at `4d11542c`, after the first merge of `main`; logs `$T/hp11-*.log`. The second
+merge, `fe9fb4b2`, brings only umtool, `scripts/`, plans and an editor changelog bullet; tsc and
+`test:scripts` were run again after it: see the last line):
+- **tsc** was clean before each commit since `afc642fd`, and before this one.
+- **Unit:** common **2,202/2,202**; editor unit **87/87**; homepage unit **8/8**; `test:scripts`
+ **185 + 1 skipped**; mcp **271/271**.
+- **Docs:** `settings example --check`, `docs files --check` and `docs env --check` all exit **0**.
+- **Doctor:** the new section prints `social icons` / `ok stored icons 3 icons in 1 file pass the
+ check` (the worktree's `settings.json`, the parent's copy; no other file there holds icons).
+- **Builds:** homepage **ok** (22 s, `pnpm --filter homepage exec next build` with the fixture
+ icons), export **ok** (34 s), editor **ok** (59 s).
+- **Homepage e2e, full suite:** **75 passed, 0 failed, 0 skipped (1.7 min)**: `social` 24,
+ `marketing` 10, `toggle` 9, `stats` 7, `source` 5, `brand` 4, `docs` 4, `downloads` 3, `theme` 3,
+ `growth-chart` 2, and one each in `svg-vectors`, `instance-wordmark`, `instance-colours` and
+ `no-data`. `options.spec.ts` went with the dialog.
+- **Export e2e** (`site-branding brand related-sites theme theme-accent responsive archives-off`):
+ **35 passed, 0 failed (1.6 min)**. The theme specs are unchanged: `ThemeToggle`'s default
+ rendering is the export's as before.
+- **Editor e2e** (`settings sites-crud`): **26 passed, 0 failed (1.8 min)**.
+- **Browsers:** the scroll box's end-first start and focus scrolling are covered in Chromium by the
+ suite and were checked by hand in Firefox at 240 px. **WebKit is unverified**: the installed
+ revision does not match this Playwright's.
+- **Earlier rounds:** at `88d61908`, common 2,179, editor unit 86, mcp 269, homepage e2e 68, export
+ e2e 35, editor e2e 45 (with `export-search`); before the review, homepage e2e 36 → 50 → 59 → 63.
+- **Screenshots** (`~/reports/release-14/shots/`, at 2×):
+ - `hp4/` (64), the narrow header, from a dev server on 3330 reading synthetic settings
+ (`SETTINGS_FILE`: the fixture's three icons, or six so the header shows four) and an empty
+ `SITES_DIR`:
+ - `header-{3,4}icons-{320,340,360,390}-{fine,coarse}-{light,sepia,dark}.png`;
+ - `header-{3,4}icons-{768,1280}-{light,sepia,dark}.png`;
+ - the wordmark is collapsed in every 320 and 340 px shot, at 360 px in all but three icons with
+ a mouse, and at 390 px with four icons under touch;
+ - `fallback-280-coarse-rest-dark.png`: four icons under touch at 280 px, the box's end in view;
+ - `fallback-280-coarse-start-dark.png`: the same after Tab reaches the first icon, the box at
+ its start;
+ - `fallback-320-200pct-text-{rest,start}-dark.png`: the same pair at 320 px with the text at
+ 200 %.
+ - `hp5/` (19), the toggle: `header-{320,360,390,768,1280}-{light,sepia,dark}.png` (three icons,
+ the toggle last) and `forced-toggle-{unfocused,focused}-{light,dark}.png`.
+ - `chart/`: the separators at 100, 60 and 35 %, per ground (the chart's section above).
+ - `hp3/`: the cards' tint and underline; the header work does not reach them.
+ - `hp2/` (the gear header) and `hp/` (the first round) are superseded; `hp2`'s dialog shots are
+ deleted.
+ - **The worktree's `homepage/out` is a fixture build: never deploy it; rebuild first.**
+- **Numbers tool:** none.
+- **After the second merge** (`fe9fb4b2`; `$T/hp12-post.log`): tsc clean in every package (42 s);
+ `test:scripts` **188 + 1 skipped**, `main`'s new guard on umtool's build trace among them.
+
+**They bite** (each change made by hand in the worktree, the specs run, the change reverted):
+- The separators in `var(--background)`: `growth-chart.spec.ts`'s ground test fails (Light: the
+ stroke is the ground's own colour).
+- `WORDMARK_FITS` taken off the wordmark: 9 of `social.spec.ts` fail (320, 340 and 360 px with a
+ mouse; 320–390 px under touch; the 280 px fallback; the 200 % text).
+- `pinAccent` taken out of `layout.tsx`: `toggle.spec.ts`'s stored-accent test fails.
+- `title` and `desc` back in the allowlist: `svg-vectors.spec.ts` fails on `title_child_el` (the
+ page's `<main>` parsed inside the icon).
+- The loading-function check skipped: the hostile-icon test fails (an `image-set(` icon is inlined
+ instead of its label).
+- From the earlier rounds:
+ - `social.spec.ts` with the id scoping, the header bound and `pointer-coarse:size-11` removed:
+ 5 failed.
+ - The hostile-icon spec fails without the read path.
+ - The loader test fails without `withCheckedLead`.
+ - `instance-wordmark.spec.ts`'s "Fix" + "ture Three" fails if the wordmark's spans stop being
+ adjacent inline text.
+
+#### Review (verdict SHIP AFTER FIXES; `$T/hp-review.md`)
+
+| Finding | Fix |
+|---|---|
+| M1: 320 px with a mouse scrolled sideways by 8 px | `6c500818`: the width × pointer rule, since replaced by ruling 7 (`0b3189fb`) |
+| M2: the `ui/dialog.tsx` change restyled the editor's command palette | `4d93dc84`: the wrapper is `main`'s, byte for byte (the dialog it served is deleted since) |
+| M3: a vendor file as a test fixture would be published by the source mirror | ruled: no vendor file. The branch was rewritten so the file and its notice never entered it; D1 is tested with synthetic sized SVGs; the e2e disc is synthetic |
+| M4: five inputs passed the normalizer and ran script | `518dd272` (the allowlist checker, the output re-check, the reasons), `4d93dc84` (the read path in both renderers); the review's five inputs and each class tested at the normalizer, the read path and in the browser |
+| L1, L2: id scoping | `518dd272`: plain ids only (the checker refuses others; comments are removed), entity-quoted and case-varied references |
+| L3: tests that could pass broken | the loader checks the lead; a mid-word lead; keys 36/44 px exactly; the glyph's own colour; six links under touch at 360; 320 px |
+| L4: "below 390" vs `max-[389px]` | superseded with the width rule (ruling 7) |
+| L5: `homepage.json` could win in the e2e | `SITES_DIR` → an empty directory. The reused dev server is pre-existing and left |
+| L6: the dialog opened on the first radio | fixed in `4d93dc84`; the dialog is deleted since (ruling 8) |
+| L7: `hp3` at 390 showed an old header | retaken |
+| L8: `featured` in `/site.json` | accepted, recorded above |
+| L9: the fixture build in `homepage/out` | recorded above: never deploy it |
+| L10: the service's name in tracked files | gone from every file this branch adds or changes |
+| L11: the size read from another attribute's value | the size comes from the root's own attributes |
+| L12: the fixture echoed an operator icon's colours | the fixture's colours changed; the pre-existing test file is outside this branch |
+| L13: `label` said "Visible name" | reworded; `SETTINGS.md` / `SITE.md` regenerated |
+
+The answers, as ruled: the tint is kept; the footer colour `--muted-foreground` is kept; marking is
+choosing, kept, and said beside the checkbox. The touch rule was kept, then superseded by ruling 7.
+
+#### Re-review (verdict SHIP AFTER FIXES; `$T/hp-review.md`, "New findings")
+
+| Finding | Fix |
+|---|---|
+| R1: a `title` or `desc` with an element child left the page's parser inside the icon | `68ad1464`: both are out of the allowlist; one holding text only is removed before the check, any other is refused as an element. `4f7bc097`: `svg-vectors.spec.ts` parses every accepted input as the static page carries it and asserts the page after it stays outside the icon |
+| R2: an icon could load from another origin | `68ad1464`: `\` and `/*` refused in every value after decoding; `image-set(`, `image(`, `cross-fade(`, `element(`, `src(`, `paint(`, `@import` and `expression(` refused in every value; `style` holds presentation properties only (an allowlist); a `url(…)` count that differs raw and decoded is refused; the reviewer's eight inputs in the unit tests (`socialSvg.vectors.ts`). `4f7bc097`: the hostile e2e and `svg-vectors.spec.ts` fail on any request to another origin |
+| R3: a stored icon the allowlist refuses blocked every save | `0be068cf`: `socialLinksForSave` checks only a new or edited icon, and keeps an unchanged stored one byte for byte, in the settings, site and homepage writers; `archilyzer doctor`'s "social icons" line names each failing icon by file and label, with the reason; the editor changelog's upgrade text |
+| R4: an icon could paint and catch clicks outside its key | `4f7bc097`: each key, in the header and both footers, is `overflow-hidden` with `contain: paint`, so a `class` or `overflow` stays inside it; `68ad1464`'s style allowlist refuses `position`, `inset` and `z-index` |
+| R5: an `attributeName` naming an href | `68ad1464`: an `attributeName` containing `href`, or starting `on` after any prefix, is refused |
+| R6: an encoded or padded fragment passed the check but was not scoped | `68ad1464`: the raw value must be a plain fragment |
+| R7: a refused icon's label had no width bound | `4f7bc097`: at most 10rem (`max-w-40`) with an ellipsis, the whole label as its title, in the header and both footers |
+| R8: the comment on the echoed name | `68ad1464`: the echoed name is stripped to letters, digits and `_.:-`, at most 40 characters |
+| R9: "remove the style block" drops the colours; a DOCTYPE's reason | `68ad1464`: the reason says to export with presentation attributes rather than a style block (in Inkscape, save as Plain SVG); a DOCTYPE with an internal subset has its own reason; the editor changelog says the same |
+| R10: the branch moved during the review | nothing to fix: every commit after `afc642fd` is for the next review |
+
+**Found and left:**
+- **The export's header and the rest of its footer** are slices H1 and H2; **T1** ("two grounds")
+ is planned, not built.
+- **The shared charts' separators** (`/stats`, the export's charts) are still drawn in the page's
+ background colour; ruling 6 names the homepage chart. A follow-up for the operator to rule on.
+- **WebKit** is unverified for the scroll box (see Gates).
+- **The Settings form's hint** still says the default links show "in every site's footer"; true of
+ the sites until H1. Operator-facing; left alone.
+- **`export-search.spec.ts`'s "a site's own social links win over the global default" depends on
+ the order of the specs.** It reads `editor/test-settings.json`, which only a settings save
+ creates, and fails when it runs first in a fresh worktree. The fix is a `.catch(() => ({}))` on
+ the read, as `auto-queue.spec.ts` does.
+- **`next dev` (16.2.3) refuses a second dev server in the same app directory.** A hand-started
+ homepage dev server must be stopped before the suite runs; `reuseExistingServer` would otherwise
+ reuse one started without the fixture's environment (pre-existing).
+- **A pasted Illustrator or Inkscape file** with a `<style>` block, `<metadata>` or `inkscape:*`
+ attributes is refused, with the element or attribute named and the export setting to use.
+- **The hub's official cards** still show plain titles; ruling 4 names the homepage only.
+- **At a 200 % text size the homepage's own content** is wider than a 320 px screen; the header is
+ not (the e2e checks the header only).
+
+**Changelog.**
+- `homepage/CHANGELOG.md` `[Unreleased]`, worded as the end state, after `main`'s stats bullet: the
+ icons in the header at every width beside the toggle (the wordmark's text drops first, the scroll
+ box is the last resort); one theme toggle and the homepage's own accent; Changelog in the footer
+ only; the cards' wordmark names; the chart's separators; the larger keys with a focus ring; the
+ e2e's fixtures and specs.
+- `editor/CHANGELOG.md` `[Unreleased]`, at the end of the list: the size rule and "Show in header";
+ the icon check with reasons, on a new or edited icon and at render, and the upgrade note (an
+ unchanged stored icon is kept; `archilyzer doctor` names those that fail).
+- `export/CHANGELOG.md` `[Unreleased]` (created by `fix/stats-cache-key`): an icon that fails the
+ check is shown as its label, bounded, and every icon paints inside its box.
+
+### Branch `r14/two-grounds-headers` — the final review's Lows, the chart's gap, T1 and H1/H2 (2026-09-28)
+
+Branch `r14/two-grounds-headers` off `main` `c6b8fc70` (slice HP merged), worktree
+`~/Projects/homepage-social-visible` (block #3: export dev 3300, export e2e 3320, hub e2e 3341,
+homepage e2e 3340, editor test 3311), one Opus implementer. Scratch files `t-*` in the job's
+`tmp`. The rulings, all 2026-09-28:
+1. (The chart, operator.) The chart follows standard practice for light and dark grounds; the
+ foreground-coloured separator lines are withdrawn.
+2. (T1, operator.) Sepia is dropped in every app, the editor included; a reader who chose Sepia
+ gets Light; readers no longer pick an accent — each site shows its own, and stored accent
+ choices are ignored.
+3. (H1/H2, operator.) Every site's header and the hub's carry the social row with the cycling
+ toggle as the last item of the same group, the homepage's; the Sites dropdown is replaced by a
+ text link to the Archilyzer home's Official Instances (omitted on the hub); the header's Hub
+ link is removed; Changelog joins the footer after Use with AI; on narrow widths the wordmark's
+ text drops first and the scroll box is the last resort; the slide-out menu keeps the nav; the
+ editor's header keeps the toggle, with no accent picker and nothing else restyled.
+4. (The final review's Lows, parent.) F1, F3, F4, F5 and F8 fixed; F2, F6 and F7 left.
+5. (After the review, operator.) On narrow screens a site's header keeps the site's NAME and shows
+ only the link(s) marked for the header; the other social links are in the footer.
+
+| sha | what |
+|---|---|
+| `d2b040fc` | `homepage:` the social row's scroll box and the compact nav are not tab stops of their own (F1); the toggle's no-flash test records every accent change (F5) |
+| `ae5d535c` | `common:` the how-to-export hint only on a drawing program's leftovers (F3); stale text: `MobileMenu`'s comment, the older homepage changelog bullet, `featured`'s doc (F4) |
+| `ec64011f` | `common, homepage, export, scripts:` every cwd-derived path op the three Next apps bundle opts out of Turbopack's tracing; the guard, renamed `scripts/next-build-trace.test.mjs`, covers them (F8) |
+| `80228398` | `homepage, common:` chart bands are parted by a 2 px gap in the surface's colour; the foreground separators are withdrawn (ruling 1) |
+| `3d0d1a46` | `common, export, editor, homepage:` two grounds, Light and Dark; each site wears its own accent (T1, ruling 2) |
+| `46bd173c` | `homepage, common:` `id="instances"` on Official Instances and `INSTANCES_URL` (H1; the planned H3 folded in) |
+| `04a1cfac` | `export:` the header's group, the Archilyzer link, no sites menu or hub link, Changelog in the footer, the narrow header; `header.spec.ts` (H1, H2) |
+| `0f358ee7` | `export:` the footer's dot shows only after a downloads link; the social row keeps the shared row's inset |
+| _this_ | `plans:` this record; the plan (H3 folded into H1, T1 and H1/H2 as built); STATE; FACTS' guard entry; the changelogs |
+
+#### The final review's Lows
+
+- **F1:** `SocialScroll`'s box has `tabIndex={-1}`, and so has the homepage's compact nav, which
+ overflows below 320 px. Firefox 146 (the installed build, by its path) made both a tab stop of
+ their own when they overflow. At 220 px with four links, without the fix Tab went home → the box
+ → the icons; with it, home → the icons → the toggle → the nav's links. The suite is Chromium
+ (Playwright's own Firefox revision is not installed), so it asserts the attribute.
+- **F3:** the hint ("export it with presentation attributes rather than a style block (in
+ Inkscape, save as Plain SVG)") follows a `style`, `<metadata>` and an element or attribute in an
+ editor's namespace. An `<a>`, `<image>`, `<title>` with markup, `<foreignObject>` or `src` gets
+ the reason alone. There is a unit test both ways.
+- **F4:** `MobileMenu`'s comment, the older homepage changelog bullet that let readers pick an
+ accent (reworded to the end state), and `featured`'s doc, which said "a narrow header shows
+ fewer" (`SETTINGS.md` and `SITE.md` regenerated).
+- **F5:** the stored-accent test installs a MutationObserver from an init script and records every
+ value `data-accent` holds; a flash between two samples fails it.
+- **F8:**
+ - `/* turbopackIgnore: true */` is on the homepage's `source.ts` directory join and on the
+ `docs.ts`, `snapshot.ts`, `summary.ts` and both changelog pages' cwd joins.
+ - In `common/lib/paths.ts`, every join on a path built from the repo root goes through one
+ opted-out `under()`, and `findMonorepoRoot()`'s walk and fallback are opted out.
+ `getPaths()` returns the same 58 values as before (diffed).
+ - The guard is `scripts/next-build-trace.test.mjs`, in `test:scripts`, 6 tests.
+ - It scans umtool as before, and adds `homepage/app`, `export/app`, `editor/app`,
+ `editor/lib`, `editor/instrumentation.ts` and every `common/` module but `bin/` (over 500
+ modules).
+ - A function declared in the file whose body carries a source is a source.
+ - Brackets inside a regex literal no longer end a call.
+ - On `main`'s files it lists 57 findings. With `main`'s `source.ts` put back, it fails on
+ `source.ts:23`.
+ - **The homepage build, capped, with and without the published source** (`homepage/public/source`,
+ 2,893 files, 70 MB), a clean `.next` each time:
+
+ | Build | Time | Max RSS |
+ |---|---|---|
+ | with the source | 14.55 s, 15.15 s | 807,332 KB, 801,948 KB |
+ | without it | 15.11 s | 797,744 KB |
+ | `main`'s join, with the source | 13.99 s | 837,144 KB |
+
+ No measurable difference: the opt-out is by rule, not a measured fix today.
+- **Left:** F2 (an animation of `style` or `class` is not held to the style allowlist; the key's
+ clip contains it). F6 (the form round-trip trims a stored SVG, so a hand-edited icon with
+ surrounding whitespace is re-checked on a form save). F7 (at 390 px the chart's thinnest upper
+ strata; see the gap below, which leaves a thin band its colour).
+
+#### The chart's surface gap (ruling 1)
+
+- **The growth chart** (`ArchiveGrowthChart.tsx`, `.growth-gap` in `homepage/app/globals.css`):
+ - a 2 px gap along each band's upper edge, in `var(--background)` (the chart sits on the page
+ ground, as the e2e checks); non-scaling, round joins; `Canvas` in forced colours;
+ - drawn centred after every fill, so each neighbour gives 1 px;
+ - no gap at the stack's top (the surface is already there);
+ - one set of gaps per plot height (200, 260, 300 px), each in a `<g>` shown by its class: a gap
+ is drawn only where both bands are at least 3 px tall at that height, so a band that gives
+ one keeps at least 1 px of its colour and a thinner band touches its neighbour instead
+ (after the review, measured at right angles to the edge: M1 in "Review" below);
+ - a run under three months is dropped.
+- **Measured on the published summary** (the first, vertical rule; after M1 see "Review"):
+
+ | Plot height | Smallest band that gives a gap keeps | Smallest band with no gap | Month boundaries parted / touching |
+ |---|---|---|---|
+ | 200 px (a phone, 390 px) | 1.00 px | 0.10 px | 182 / 277 |
+ | 300 px (1280 px) | 1.00 px | 0.15 px | 302 / 157 |
+
+ At 390 px the 2016–2020 bands are 0.1–5.5 px tall.
+- **The shared charts** (`common/components/charts/surfaceGap.ts`, `--chart-gap` in
+ `tokens.css`: the chart surface, `Canvas` in forced colours):
+ - stacked bars get a 2 px stroke per segment;
+ - stacked areas got a 4 px stroke along the band's top edge, half under the band above, so
+ 2 px showed — withdrawn after the review (H1): they keep their series-coloured edge;
+ - single-series charts, lines and side-by-side bars are unchanged.
+ - This covers `ChartView` (the export's charts) and `CrossSiteChart` (the homepage's `/stats`
+ customize view).
+- **Screenshots:** `~/reports/release-14/shots/chart/gap2-{390,1280}-{light,sepia,dark}{,-thin}.png`
+ (taken before T1 removed Sepia).
+- **Departures for the operator to rule on, not changed:**
+ - `ChartView` and `CrossSiteChart` fill stacked areas translucent (0.2 and 0.25); the standard
+ is opaque fills parted by the gap.
+ - `CrossSiteChart`, `LeaderboardChart` and `MomentumChart` draw dashed gridlines
+ (`strokeDasharray="3 3"`); the standard is solid hairlines.
+ - `ChartCard`'s surface is `--card`, equal to `--chart-surface` on both bases today.
+
+### Slice T1, as shipped — two grounds (2026-09-28)
+
+- **Two grounds.**
+ - `THEME_BASES` is System, Light, Dark.
+ - `nextBase` walks `THEME_BASES` (system → light → dark → system).
+ - `ResolvedBase` is light | dark; `isThemeBase` takes the three.
+- **Deleted:**
+ - the Sepia block in `common/styles/tokens.css` and its `--base-sepia` flag;
+ - `onSepia` in every accent, `BASE_GROUNDS.sepia`, `ACCENT_INK.sepia`, the fitted
+ `--accent-custom-sepia`;
+ - the BookOpen icon;
+ - Sepia from every spec, fixture and comment;
+ - `ThemeMenu.tsx` (nothing renders it);
+ - `accentOptions` and `isThemeAccent`;
+ - the `pinAccent` option (it is the only behaviour now).
+- **Kept:**
+ - `ACCENT_KEY`, documented as a pick nothing reads or deletes.
+ - `ThemeRadios`, base-only in this commit; H2 deleted it with its caller.
+- **Migration, no flash:**
+ - A stored `ytdlp-tb:base` of the retired value (`RETIRED_BASE` in `themeConfig.ts`, the one
+ place its name is spelled) is `light` in the pre-paint script and in `ThemeProvider`, and is
+ rewritten to `light` once.
+ - The legacy `archive` + `light` pair maps to `light`.
+ - Any other unknown value falls back to the app's default, as before.
+ - Unit tests cover the whole matrix (the retired value, `archive` + `light`, garbage, a storage
+ that will not take the write).
+ - Each app has an e2e for a stored retired base: an init-script MutationObserver sees no ground
+ but the server's and Light, `data-theme-ready` is set, and storage reads `light`.
+- **The accent is the site's:**
+ - `ThemeScript` and `ThemeProvider` never read `ytdlp-tb:accent` and never remove it.
+ - The export's layout and header, and the editor's header, have no accent picker; the
+ slide-out menu's accent list is gone.
+ - The site form's accent control is config and is unchanged; its hint no longer says a reader
+ can pick another.
+ - `theme-accent.spec.ts` keeps the site's own accent (fitted per base, no picker), a stored
+ accent ignored with no flash and left in place, and the hint in the site's accent on each
+ base. The four-bases menu, "pick Violet" and "pick the site's colour again" went with the
+ picker.
+- **Docs:** `operate.md` (no theme menu; a light or dark ground with the header's toggle),
+ `siteSchema`'s accent text (`SITE.md` regenerated), the site form's hint.
+- **Counts** (`git grep -i sepia -- . ':!plans/' ':!*CHANGELOG.md'`): 47 files, 177 lines at the
+ branch's start (with HP merged). After T1: one line, `RETIRED_BASE = "sepia"`, which the
+ migration needs in order to recognise a stored value. The changelogs' `[Unreleased]` sections
+ name it nowhere; released entries keep 9 lines.
+
+### Slice H1/H2, as shipped — the export and editor headers (2026-09-28)
+
+- **The export header** (every site and the hub):
+ - From `lg` (1024 px): brand · nav · **Archilyzer** · the group.
+ - Below `lg`: brand · the group · the menu trigger.
+ - The group is the homepage's: `SocialLinks` (header, at most four: `featured`, else the last
+ four) in `SocialScroll`, then `ThemeToggle variant="bare"`, 36 px keys (44 px under a coarse
+ pointer) touching.
+ - The footer's row is `SocialLinks` (footer) too: all the links, the same keys, the text colour
+ on hover where it was the accent.
+- **The Archilyzer link** replaces `SiblingSwitcher` (deleted):
+ - visible text "Archilyzer", accessible name "Archilyzer — official instances";
+ - `INSTANCES_URL`, same tab;
+ - not on the hub.
+ - The homepage's Official Instances section has `id="instances"`, with `scroll-mt` equal to the
+ sticky header's height. The planned H3 is folded in here, with a homepage e2e for
+ `/#instances`.
+- **The header's Hub link** is gone from the bar and the menu; `hubUrl` and `resolveHubUrl` still
+ parse. **Changelog** is in the footer after Use with AI, and not in the bar or the menu.
+- **The slide-out menu** holds only the nav, plus the Archilyzer link on a site. It is kept:
+ five-plus nav links do not fit a phone's bar. `ThemeRadios` is deleted.
+- **The nav's breakpoint moved from `md` to `lg`:**
+ - At 768 px the nav, the Archilyzer link and a long title with four icons did not fit, so the
+ wordmark hid and an icon scrolled while the nav was still inline.
+ - With the nav in the menu below `lg`, the icons never scroll at 768.
+- **Narrow widths, for any title** — the approach, and why (superseded in part by ruling 5:
+ below 520 px the header now shows only the marked links, and the wrap and the scroll box are
+ last resorts; see "The narrow header, as ruled"):
+ - The wordmark's text sits in a one-line box (`h-7`, `overflow-hidden`, `flex-wrap`) behind a
+ zero-width strut. The text is one flex item: when it does not fit beside the strut, it wraps
+ to the second line, which is clipped.
+ - So the text drops exactly when it does not fit, measured by the browser in the site's own
+ font at the reader's text size. There is no per-site threshold and no build-time estimate.
+ Titles differ in length, and a threshold written down would be wrong for some title or some
+ text size.
+ - The text stays in the DOM, so the link's name stays the title.
+ - Below `lg` the brand link takes the bar's free space (basis 0, grow 1, minimum the mark), so
+ the group gives way only once the link is the mark alone. The row's box then scrolls, end
+ first, as on the homepage.
+- **Measured** on a dev server (synthetic titles, the viewport width):
+
+ | Title | Links | Full wordmark from (mouse / touch) | Row scrolls below (mouse / touch) |
+ |---|---|---|---|
+ | "Shortlyzer" | 1 | 321 / 337 | never ≥ 240 |
+ | | 2 | 357 / 381 | never / 251 |
+ | | 3 | 393 / 425 | 263 / 295 |
+ | | 4 | 429 / 469 | 299 / 339 |
+ | "Longestfixturealyzer" | 1 | 449 / 465 | never ≥ 240 |
+ | | 2 | 485 / 509 | never / 251 |
+ | | 3 | 521 / 553 | 263 / 295 |
+ | | 4 | 557 / 597 | 299 / 339 |
+
+ The page never scrolls sideways from 240 to 1023 px. The row scrolls at 320 px only with four
+ links under touch (by 19 px): the export bar also carries the menu trigger, which the
+ homepage's does not.
+- **The editor's header:**
+ - `ThemeToggle` (its default variant, the editor's existing chrome) cycles System, Light and
+ Dark.
+ - `ThemeMenu` is gone.
+ - Nothing else is restyled.
+ - There is no social row.
+- **Tests:**
+ - `header.spec.ts`, new:
+ - a short and a long title × 1–4 links × 280–430 px × mouse and touch, checking that the page
+ and the header never overflow, every link shows, the text drops before the row scrolls, and
+ the mark, the toggle and the link's name hold;
+ - a short title in full at 390 px, a long one giving way;
+ - key sizes (36 and 44 px) and touching boxes;
+ - the focus ring;
+ - the Archilyzer link's href, name and text; no sites button, hub link or "Choose theme";
+ - Changelog in the footer after Use with AI and not in the header;
+ - six links and two featured;
+ - a hostile icon in `socialLinks` as its label, running nothing and requesting no other
+ origin.
+ - The fixture site is rewritten per test and restored. It carries its original in
+ `_e2ePristine`, which `playwright.config.ts` restores if a run dies.
+ - `responsive.spec` covers the menu's contents; `site-branding` scopes its footer link;
+ `official-instances` (hub) checks the hub's header has no Archilyzer link, sites menu or theme
+ menu.
+- **Screenshots** (`~/reports/release-14/shots/h1/`, synthetic icons, six configured so the header
+ shows four):
+ - `{short,long,hub}-header-{320,360,390,768,1280}-{light,dark}.png`;
+ - `{short,long,hub}-footer-{…}.png`;
+ - `editor-header-{1280,390}-{light,dark}.png`.
+
+#### Gates (at `0f358ee7`; logs `$T/t-g-*.log`)
+
+- **tsc** was clean before every commit, and at the tip (49 s).
+- **Unit:**
+
+ | Suite | Result |
+ |---|---|
+ | common | 2,204/2,204 |
+ | editor unit | 87/87 |
+ | homepage unit | 8/8 |
+ | `test:scripts` | 191 passed, 1 skipped (192) |
+ | mcp | 271/271 |
+
+- **Docs:** `settings example --check`, `docs files --check` and `docs env --check` all exit **0**.
+- **Builds**, each capped at 5 GB with no swap, one at a time, from a clean `.next` (time, max RSS):
+
+ | Build | Time | Max RSS |
+ |---|---|---|
+ | homepage (fixture icons) | 17 s | 796 MB |
+ | export, site (the worktree's default) | 25 s | 982 MB |
+ | export, the fixture site | 29 s | 990 MB |
+ | export, hub | 26 s | 1,004 MB |
+ | editor, with the corpus visible | 44 s | 1,645 MB |
+ | umtool, with the corpus visible | 19 s | 784 MB |
+
+ For the editor and umtool builds the primary's `transcripts/` was linked in, and the worktree's
+ own set aside. Both were put back after the builds; nothing ran through the link.
+- **e2e**, each detached and queued:
+
+ | Suite | Passed | Failed | Time |
+ |---|---|---|---|
+ | homepage, full | 77 | 0 | 1.8 min |
+ | export, full | 220 | 0 | 8.6 min |
+ | hub, full | 34 | 0 | 1.1 min |
+ | editor: `theme`, `sites-crud`, `settings`, `export-search` (the specs on the theme controls, the site form's accent hint, the social-links field, and the export's header and footer as the editor's export server shows them) | 51 | 0 | 1.7 min |
+
+- **Along the way:**
+ - the chart commit: tsc; homepage unit 8/8; a capped homepage build 16 s; homepage e2e 75/75
+ (2.2 min); export `charts.spec` 9/9;
+ - T1: homepage theme/toggle/brand/instance specs 21, export theme/branding specs 30, hub 8,
+ editor 20;
+ - H1/H2: export header/responsive/branding specs 43 (one assertion order fixed), hub 9,
+ homepage `marketing` 11.
+- **Numbers tool:** none.
+
+#### Found and left
+
+- **The export footer's `-mx` inset** keeps the shared row's value; on a wide screen the last
+ key's box reaches 8 px into the container's padding.
+- **At 320 px with four links under touch** the export header's row scrolls by 19 px. This is the
+ ruled last resort; the menu trigger is what the homepage does not have.
+- **`resolveHubUrl`** has no caller in the export app now. It still parses, as ruled.
+- **Playwright's own Firefox and WebKit are not installed** at this Playwright's revisions; F1 was
+ checked by hand in the installed Firefox 146.
+- **The chart departures** above (translucent stacked areas, dashed gridlines) are for the
+ operator.
+
+#### Decisions the operator could overturn
+
+| What I assumed | The alternative |
+|---|---|
+| The export's inline nav and the Archilyzer link start at `lg` (1024 px), with the menu below | keep `md` and let the wordmark hide and an icon scroll at 768–1023 px |
+| The Archilyzer link is in the slide-out menu below `lg` | leave it out of the menu, so a phone reaches it only by the footer's "Built with Archilyzer" |
+| The link is styled as the old Changelog link (small, muted), with the text "Archilyzer" | mono uppercase like the old Hub link, or the nav's style |
+| The wordmark's text hides by a pure-CSS wrap, exact for any title | a per-site threshold computed at build time from the title's estimated width |
+| The export footer's icons take the shared keys (36 px, hover to the text colour) | keep the footer's 20 px accent-hover icons |
+| The chart's gap is skipped where either band is under 3 px at the drawn height, and runs under three months are dropped | a gap on every boundary, erasing the thinnest bands |
+| Stacked areas in the shared charts get the surface gap over their translucent fills | leave their series-coloured top lines until the fill opacity is ruled on |
+| `RETIRED_BASE = "sepia"` stays in `themeConfig.ts` for the migration | spell it indirectly so the grep reads zero |
+| The export footer's dot shows only after a downloads link | leave the dot unconditional, as before |
+
+#### Review (verdict SHIP AFTER FIXES; `$T/t-review.md`)
+
+| Finding | Fix |
+|---|---|
+| H1: the stacked-area surface gap erased small values and cut peaks (every site's multi-series area charts, `/stats`) | `92dc4084`: stacked areas keep `main`'s series-coloured edge (ChartView 1 px, CrossSiteChart 1.5 px); `BAR_GAP` stays on stacked bars. `charts.spec` reads the chart back from a screenshot (`common/testing/chartPixels.ts`): at 390 and 1280 px the stack's topmost painted row at each month is within 1 px of the value scale's y for the true total, and each series shows pixels of its own fill; with the old gap the tops land 2–4 px low |
+| M1: the growth chart's thin-band rule was vertical, so on steep edges the gap covered whole bands | `f455d7da`: `homepage/app/lib/growthGaps.ts` (pure): thickness at right angles to the edge (min vertical height × cos θ) at the narrowest plot of each height (240, 592, 976 px); a gap only where both bands keep ≥ 1 px after every gap on them. `growthGaps.test.ts` checks every drawn segment independently: 0 covered band-segments on the fixture, a slivers-and-spikes stand-in and a one-month spike (two of the four fail under the old rule). `growth-chart.spec`: only the drawn height's set shows; the peak's painted top is within 1 px of the true total; every site with data shows its colour |
+| M2: four texts still promised the Hub link | `706f62bf`: Settings' Family hub URL and a site's Hub URL hints, `homepageUrl`'s and `hubUrl`'s docs: the value is published as `hubUrl` in the site's `/site.json` and `/corpus.json` so the hub can tell member sites; no page links to it. `SETTINGS.md`, `SITE.md` regenerated; labels and field names unchanged |
+| L1: from 1024 to 1046 px a very long title dropped its text and scrolled the icons at once | `2a5a81c0`: `lg:shrink-[999]` on the brand link; `header.spec` sweeps 1024–1050 px under touch with a 27-character title (scrolls without the fix) |
+| L2: a title could show in the fallback face and vanish when Archivo loaded | `2a5a81c0`: the wordmark's width is reserved from Archivo's metrics at its two instances (`common/lib/wordmarkMetrics.ts`, generated by `common/bin/gen-wordmark-metrics.py` from the vendored font; `lib/wordmarkWidth.ts`, +3 %: the served font renders 0.2–1.7 % wider than the file's advances). `header.spec` blocks the font file and compares the title's visibility at 360, 375, 390, 412 and 430 px: the same before and after Archivo loads (the short title differs without the reservation) |
+| L3: the provider and the script disagree when storage reads but refuses writes | left: reachable only with a stored retired value that can no longer be rewritten |
+| L4: `common/components/ui/` was not untouched | nothing to change: `alert.tsx`, `badge.tsx` and `sonner.tsx` have comment-only edits naming the bases; the palette, `dialog.tsx` and `command.tsx` are untouched |
+| L5: stale text | `689a835f`: `export/globals.css`, `export/manifest.ts`, `brand.test.ts`; the export changelog's T1 bullet no longer names a Base list; the homepage changelog's e2e bullet |
+| L6: the footer's social row did not wrap | `2a5a81c0`: `flex-wrap` on the footer placement; eight links under touch at 320 px, export and homepage specs |
+| L7: the guard's blind spots (`fs.promises.*`, `accessSync`/`open`/`readlink`, a relative literal to fs, a cwd default parameter, a class method as the source; false positives on a string naming `process.cwd()`) | left: none of these shapes is in the covered code today |
+| L8: the rollout order was implied | this commit: "## Rollout" below |
+| L9: 200 % text | `2a5a81c0`: the header itself never overflows at 200 % text (tested at 320 and 360 px, both pointers). The 43 px the review measured at 320 px are the search page's own controls (a label and the view toggle), not the header. At 320 px under touch the mark, the 88 px toggle and the 72 px menu button leave the social row's box no room: the phone header is the operator's to rule on |
+| L10: named-accent coverage | `689a835f`: `theme-accent.spec` rewrites the fixture to a named accent (violet): its value on Light and on Dark, a stored pick not applied and left in place |
+
+**The growth chart's gaps after M1**, on the published summary:
+
+| Plot height | Gap segments | Runs | Covered band-segments |
+|---|---|---|---|
+| 200 px | 25 | 5 | 0 |
+| 260 px | 64 | 11 | 0 |
+| 300 px | 96 | 14 | 0 |
+
+**The gaps now read as dashes.** On a phone the chart has two short dashes, both along
+Jeralyzer's top. At 1280 px they are stretches along Jeralyzer's and Anilyzer's tops, with
+short pieces elsewhere. The rest of the bands touch. This is the underlying issue the operator
+has been told about: four sliver bands beside two large ones. Folding the small sites into
+"Other", or small multiples, is the operator's call and is not made here.
+Screenshots: `~/reports/release-14/shots/chart2/growth-{390,1280}-{light,dark}{,-2019-2022}.png`.
+
+**Screenshots:**
+- `chart2/site-stacked-{area,bar}-{390,1280}-{light,dark}.png`: a site's charts. The fixture has
+ one video a month, so its stacked bars have no touching segments.
+- `chart2/stats-stacked-{area,bar}-{390,1280}-{light,dark}.png`: `/stats` on the published
+ summary. The area has its coloured edges back, and the bars show the 2 px gaps between
+ segments.
+
+**The nine decisions, as ruled:**
+1. The nav and the Archilyzer link are inline from `lg` → keep, with L1.
+2. The Archilyzer link is in the slide-out menu below `lg` → keep.
+3. The link is small and muted, with the text "Archilyzer" → keep.
+4. The pure-CSS wrap → keep, with L1 and L2.
+5. The export footer takes the shared 36 px keys → keep, with L6.
+6. The growth-chart gap is skipped under 3 px, and runs under three months are dropped → keep
+ the rule, measured at right angles (M1).
+7. The surface gap over translucent stacked areas → do not keep (H1).
+8. `RETIRED_BASE = "sepia"` stays → keep.
+9. The footer's dot only after a downloads link → keep.
+
+| sha | what |
+|---|---|
+| `92dc4084` | H1 |
+| `f455d7da` | M1 |
+| `706f62bf` | M2 |
+| `2a5a81c0` | L1, L2, L6, L9 |
+| `689a835f` | L5, L10 |
+| `447ded9a` | the narrow-header ruling (below) |
+| _this_ | `plans:` the Review, the ruling's record, the Rollout, the plan, STATE |
+
+**Gates**, run once after the review fixes and the ruling (at `447ded9a`; logs `$T/t-g3-*.log`):
+
+- **tsc** was clean before every commit, and at the tip (37 s).
+- **Unit:**
+
+ | Suite | Result |
+ |---|---|
+ | common | 2,210/2,210 |
+ | editor unit | 87/87 |
+ | homepage unit | 12/12 |
+ | `test:scripts` | 191 passed, 1 skipped |
+
+- **Docs:** the three `--check` commands exit **0**.
+- **Builds**, each capped at 5 GB with no swap, one at a time:
+
+ | Build | Time | Max RSS |
+ |---|---|---|
+ | homepage | 16 s | 800 MB |
+ | export site | 25 s | 995 MB |
+ | export hub | 23 s | 1,021 MB |
+ | editor | 43 s | 1,650 MB |
+
+- **e2e**, each detached and queued:
+
+ | Suite | Passed | Failed | Time |
+ |---|---|---|---|
+ | homepage, full | 90 | 0 | 2.1 min |
+ | export, full | 235 | 0 | 10.9 min |
+ | hub, full | 35 | 0 | 1.3 min |
+ | editor (`settings`, `sites-crud`, `theme`) | 32 | 0 | 1.3 min |
+
+- **An earlier gate run** at `689a835f` (the review fixes alone) was stopped when the ruling
+ arrived, to run the gates once for both. Before it stopped, it had passed: tsc; common
+ 2,208/2,208; editor unit 87/87; homepage unit 12/12; `test:scripts` 191 + 1 skipped; the docs
+ checks; all four builds; and the homepage e2e, 83/83.
+
+#### The narrow header, as ruled (2026-09-28)
+
+**The ruling** (operator): on narrow screens a site's header keeps the site's NAME and shows only
+the link(s) marked for the header; the other social links are in the footer.
+
+**`featured`, new meaning** (`common/lib/socialLinks.ts` `headerSocialLinks(links, width)`):
+- **Wide:** every link, up to four. With more configured, the `featured` ones are kept first, then
+ the last of the rest, shown in configured order.
+- **Narrow:** only the `featured` ones, up to four (the last four when more are marked). With none
+ marked, the narrow header shows no social link: the name wins, and the footer has them all.
+- **The footer** always shows every link.
+- The key is still `featured`, stored only when true. The editor's checkbox is **Keep in header
+ on small screens**, with the hint "On small screens the header shows only these; the rest stay
+ in the footer." `SETTINGS.md` and `SITE.md` are regenerated.
+
+**Layout** (the homepage, every site and the hub, one behaviour):
+- Both rows are rendered. CSS shows the narrow one below **32.5rem** and the wide one from
+ 32.5rem: 520 px at the default text size, 1040 px at a 32 px browser text size (`11a33f7a`, N5).
+ The other is `display: none`, so exactly one is focusable and in the accessibility tree.
+- Each copy scopes its icons' ids.
+- The narrow bar is mark + name · the marked link(s) · toggle, then the menu button on the export.
+- The wordmark's text drop (the export's reserved-width wrap, the homepage's container
+ thresholds, now keyed by the narrow row's count) and the scroll box stay as last resorts.
+
+**Why 520 px:** every link (up to four, 44 px touch keys), the toggle, the menu button and the
+longest real title (Rekietalyzer, Hasanalyzer) fit the export's bar from about 500 px wide under
+touch. At 519 px the narrow row shows and at 520 px the wide one, and the name shows at both
+(measured on Rekietalyzer and Hasanalyzer, both pointers, 519–1280 px). The homepage's bar needs
+about 424 px.
+
+**The gap ruling (2026-09-29):** below the switch the export bar's two gaps (name → row, toggle →
+menu) are **8 px**, not 12 (`1a2e3342`, N1). Type, the mark–name gap and the reserved width's
+3 % margin are unchanged. It is one header, so every site and the hub. The homepage's bar has one
+gap and keeps 12 px.
+
+**The smallest viewport width at which the full name shows** (a dev server at `11a33f7a`, four
+links configured, re-measured after the gap ruling; before it, each site's and the hub's widths
+were 8 px more):
+
+| Title | One marked link: mouse / touch | None marked: mouse / touch |
+|---|---|---|
+| Jeralyzer | 302 / 318 | 266 / 274 |
+| Anilyzer | 288 / 304 | 252 / 260 |
+| Bonnellyzer | 336 / 352 | 300 / 308 |
+| Hasanalyzer | 343 / 359 | 307 / 315 |
+| Rekietalyzer | 344 / **360** | 308 / 316 |
+| Jasolyzer | 308 / 324 | 272 / 280 |
+| Archilyzer (homepage) | 276 / 292 | ≤ 240 / 248 |
+| Archilyzer (hub) | 313 / 329 | 277 / 285 |
+
+**The target** (every title in full at 360 px, one marked link, touch) is **met by every title**.
+Rekietalyzer meets it at exactly 360 px.
+
+**The wordmark's reservation is a minimum width** (`4ea1c495`, N2). The header sets
+`wordmarkWidthEm` as `min-width`. Where the text renders wider than the reservation, its box grows
+with it and the text drops sooner; its last letter is never clipped. The re-review measured 0.6 to
+0.8 % spare in Chromium. The table above is unchanged by it.
+
+**`featured` keeps its configured place** (N4, ruled as built). With more than four links, a
+marked link is guaranteed a place in the wide row, and the row keeps configured order: `[A★, B,
+C, D★, E, F]` shows `A D E F`.
+
+**Tests:**
+- `socialLinks.test.ts` covers both widths: none marked, one, several, more than four with and
+ without marks, and order kept.
+- The homepage (`social.spec`), a site (`header.spec`: a short title, Bonnellyzer, Hasanalyzer and
+ Rekietalyzer) and the hub (`official-instances.spec`) are checked at 360 and 390 px, both
+ pointers:
+ - the full name shows;
+ - only the marked link is in the header, in the accessibility tree too;
+ - with none marked, the header has no row and the name shows;
+ - from 520 px every link shows;
+ - the footer has every link at every width;
+ - nothing scrolls the page sideways from 280 to 1400 px;
+ - focus goes brand → marked link → toggle, then menu on a site.
+- `header.spec` checks **all six real titles at 360 px under touch with one marked link**: the
+ full name, the link, no scrolling row, no overflow. Hasanalyzer and Rekietalyzer fail it with
+ 12 px gaps.
+- `header.spec` forces the wordmark 8 % wider (letter-spacing) and sweeps 519 to 280 px:
+ wherever the name shows, it does not overflow its box or the clip box. With a fixed width it
+ is clipped from 519 to 344 px.
+- **At 200 % text**, both pointers:
+ - **The browser's text size** (32 px, set through CDP `Page.setFontSizes`; Chromium):
+ - the narrow row shows until 1040 px, and the wide one from there;
+ - the row never scrolls while the name shows, and the header never overflows;
+ - the name shows with one marked link from 720 px on the homepage and from 767 px on
+ Rekietalyzer.
+ - **The page's root at 200 %** (which moves the page's rem but not a media query's):
+ - on the homepage from 520 to 767 px, the wide row never scrolls while the name shows;
+ - on a site the switch stays at 520 px and the row never scrolls while the name shows.
+- The e2e fixtures mark their last link.
+
+**Screenshots:** `~/reports/release-14/shots/h2/`, four synthetic icons, the last marked:
+- `{jeralyzer,anilyzer,bonnellyzer,hasanalyzer,rekietalyzer,jasolyzer,homepage}-{320,360,390}-{light,dark}.png`
+ (a mouse) and `…-{320,360,390}-touch-{light,dark}.png` (touch), re-taken at `11a33f7a`;
+- `…-{768,1280}-{light,dark}.png` from `447ded9a` (the header is unchanged at those widths).
+
+At 360 px every title shows in full under touch. At 320 px Bonnellyzer, Hasanalyzer and
+Rekietalyzer drop their text under both pointers, and Jasolyzer does under touch.
+
+#### Re-review (verdict SHIP; `$T/h-review2.md`)
+
+| Finding | Fix |
+|---|---|
+| N1: at 360 px under touch, with one marked link, Hasanalyzer and Rekietalyzer lost the name | `1a2e3342`: the gap ruling (above). All six titles, the homepage and the hub re-measured; the e2e checks all six |
+| N2: the reserved wordmark width had 0.6–0.8 % spare in Chromium | `4ea1c495`: the reservation is a `min-width` (above) |
+| N3: the Rollout had no step for marking the link, and a live check expected every icon at 390 px | this commit: a precondition before any build, and the 390 px check reads "the marked link" (## Rollout) |
+| N4: "kept first" is priority, not position | ruled as built: a marked link keeps its configured place (above) |
+| N5: on the homepage at 150–200 % text, from 520 to 767 px, the wide row scrolled while the name showed | `11a33f7a`: the switch is `32.5rem` in every header; the homepage's wordmark fit is keyed by the narrow row's count below the switch and by the wide row's from it |
+
+| sha | what |
+|---|---|
+| `1a2e3342` | `export:` the bar's two gaps are 8 px below the switch; `header.spec` all six real titles; the hub's name test under both pointers (N1) |
+| `4ea1c495` | `export, common:` the wordmark's reservation is a `min-width`; the wider-text e2e (N2) |
+| `11a33f7a` | `homepage, export:` the switch in rem; the homepage's fit by the wide row's count from the switch; the 200 % text e2e (N5) |
+| _this_ | `plans:` the ruling's record re-measured, the re-review, the Rollout's marked-link precondition (N3), STATE, the plan; the changelogs |
+
+**Gates** at `11a33f7a` (logs `$T/t-g4*.log`):
+- **tsc** was clean before every commit, and at the tip (32 s).
+- **Unit:** common 2,210/2,210; homepage 12/12.
+- **e2e**, the header specs only, detached and queued:
+
+ | Suite | Specs | Passed | Failed |
+ |---|---|---|---|
+ | homepage | `social`, `toggle`, `svg-vectors`, `brand` | 50 | 0 |
+ | export | `header`, `responsive`, `site-branding` | 50 | 0 |
+ | hub | `official-instances` | 8 | 0 |
+
+- **Builds**, capped at 5 GB with no swap, one at a time: the homepage (13 s, 824 MB) and the
+ export site (21 s, 1,029 MB). Both builds' CSS carry the `32.5rem` switch, and the homepage's
+ carries both fit sets.
+
+**Found and left:** with the page's root at 150–200 % (not a browser setting), from 768 px the
+homepage's nav joins the bar at `md`, whose media query does not follow a page-set root size.
+There the bar overflows and the wide row scrolls while the name shows: measured from 779 to
+1100 px at 150 % and from 779 to 1300 px at 200 %. With the browser's own text size `md` moves too
+and nothing overflows. The `md` layout is slice HP's and unchanged here.
+
+### Slice HS, as shipped — hidden sites: a site that builds and deploys, and is listed nowhere (2026-09-29)
+
+Branch `r14/hidden-sites` off `main` `99d4d76a`, worktree `~/Projects/homepage-social-visible`
+(block #3: editor test 3311, export 3310, homepage e2e 3340, hub e2e 3341), one Opus implementer.
+Scratch files `hs-*` in the job's `tmp`. The rulings (2026-09-29, not re-opened):
+1. A hidden site is absent from the homepage (cards, chart, `/stats`), the hub (members, federated
+ search, `corpus.json`, `llms.txt`), every other site's related-sites footer, and the published
+ id lists (`channel-sites.json`, the pooled `stats/`).
+2. Its hours and transcripts are in no public total.
+3. The editor has one checkbox in each site's settings, and no homepage settings page.
+4. The hidden site itself builds and deploys exactly as before.
+
+| sha | what |
+|---|---|
+| `122b8879` | `common:` `site.json` `listed` (schema, docs, parser, writer; `SITE.md`); `isListedSite` and `channelsOnlyOnUnlistedSites`; the footer drops an unlisted sibling |
+| `77aa1456` | `common:` the homepage summary (v6), `channel-sites.json` and the whole-pool stats leave out an unlisted site and the channels only it exposes |
+| `cbd12a8b` | `common:` `hub-sites.json` (and through it the hub's `corpus.json` and `llms.txt`) lists listed sites only |
+| `5836d6ec` | `sites:` the checkbox, and `saveSiteAction` carries the key; `sites-crud` e2e |
+| `64ae9b47` | `homepage:` the e2e fixture's seventh, unlisted site; `unlisted-site.spec.ts` |
+| `30b3f486` | `plans:` this record; the three changelogs |
+| `e3ee2eb1` | `homepage:` the fixture's second list unlists every site in it; the spec proves the site is in the input (review L1, L2) |
+| `65675373` | `common:` a shared channel stays the listed site's when the unlisted id sorts first (review L3) |
+| `20ff9ecf` | `plans:` FACTS' naming hazards (L4); this release's slices table, Order and Rollout carry HS (L6) |
+| `7ea35dfb` | merge of `main` `ccf90892` (release 15 IG); `buildStats.ts` and `editor/CHANGELOG.md` merged clean, the HS bullet under `[Unreleased]` |
+| _this_ | `plans:` the commit table, the review and the post-merge gates |
+
+The six commits up to `30b3f486` were rewritten after the review for the release's commit trailer
+(`git filter-branch --msg-filter`, trees unchanged); their first shas were `5918ba27`, `10378f7f`,
+`1491a7ab`, `ca22f0e6`, `e12515c2`, `4eeb80ee`.
+
+- **The key:** `listed?: boolean` in `site.json`, after `siteUrl`, in the type, `SITE_FIELD_DOCS`,
+ `siteFieldsSchema` and `siteToDisk`. Absent or anything but `false` reads `true`; only `false` is
+ written (the `archives` idiom). `SITE.md` regenerated.
+- **One predicate.** `isListedSite(site)` (`site.listed !== false`) and
+ `channelsOnlyOnUnlistedSites(sites)` — the channels at least one site exposes and no listed site
+ does. Both live beside the key in `common/lib/siteSchema.ts` and are exported from `lib/site`
+ (its `export *`), like `isValidSiteId`: the summary builder is pure and imports them without
+ `lib/site`'s file I/O. Every filter below calls them.
+- **What each public output does now:**
+
+ | Output | Code | An unlisted site | A channel only unlisted sites expose |
+ |---|---|---|---|
+ | Homepage summary: `sites`, `official`, `monthly`, `series`, `recent`, `channels` | `buildHomepageSummary`, the public-site filter | absent | absent |
+ | Homepage summary: `totals`, `availability` | the same, over the in-scope stats | — | not counted |
+ | `channel-sites.json` | `channelSitesOf` (`controller/poolSummary.ts`) | absent from every list | absent |
+ | The homepage's `stats/` (whole-pool bundle) | `buildStats`, the whole-pool block | — | its records and its manifest entry absent |
+ | `hub-summary.json` | `toHubSummary` of the same summary | absent | not counted |
+ | `hub-sites.json`, the hub's `corpus.json`, `llms.txt` | `compose-hub.ts`, the built-in pool | absent | — |
+ | Every other site's footer | `resolveRelatedSites` | not linked, even from a featured group | — |
+
+ - A channel a listed site also exposes is credited to the listed site: `primarySiteOf` runs over
+ listed public sites only.
+ - The summary's version is 6. No field was added or removed; the number marks the scope change.
+- **What stays, and why:**
+ - The unlisted site's own build and deploy: `buildAll` and the deploy paths
+ (`common/publish/build.ts`), its per-site stats and summaries (`buildStats`, `buildIndex`), its
+ archives and chart templates, its own `/site.json`, `corpus.json`, `llms.txt`, sitemap. Its own
+ footer still lists its listed siblings.
+ - The editor's pages (`/sites`, the channel page's memberships, the nav's site switcher, the
+ priority focus), which list every site.
+ - `siteChannelIndex`, `renameChannel`, `migrateToSites`, `listSiteIds` for the CLI's
+ site argument, the export's default site: none is a public list.
+ - The per-site staging in `buildIndex.ts` (another slice's file, and per site).
+- **Editor.** The site form has **List on the Archilyzer homepage and hub** after Public URL, on
+ by default, with a hint. `saveSiteAction` rebuilds the `Site` from the form; it now carries
+ `listed: false`, so a save of any other field keeps it. The other writers (`writeSite` from the
+ channel page, `renameChannel`, `migrateToSites`) spread the stored site or create a new one.
+- **Tests:**
+ - `siteSchema.test.ts`: absent and `true` read listed, only `false` unlists and only `false` is
+ written, through `writeSite`/`getSite`; the "everything" round-trip fixture carries
+ `listed: false`; `channelsOnlyOnUnlistedSites` keeps a shared channel with the listed site; the
+ footer drops an unlisted sibling named in a featured group, and an unlisted site's footer
+ still lists the rest.
+ - `homepageSummary.test.ts`: an unlisted site with its own channel and one shared with a listed
+ site gives a summary deep-equal to the one without it (every array, every total), and its id,
+ title and channel appear nowhere in the JSON; an unlisted site with no `siteUrl`, and one whose
+ channels are all shared, change nothing either; `listed: true` equals no key.
+ - `buildStats.test.ts` (k): the whole-pool bundle leaves out the unlisted-only channel (records,
+ manifest, count, log line) and keeps a pool-only one; the unlisted site's own bundle keeps all
+ three of its videos.
+ - `poolSummary.test.ts` (new): `channelSitesOf` names listed sites only.
+ - `compose-hub.test.ts`: `hub-sites.json`, `corpus.json` and `llms.txt` name the listed site and
+ not the unlisted one.
+ - Editor `sites-crud`: a `listed: false` file opens unticked; a save that changes only the title
+ keeps `false`; ticking removes the key; unticking writes it again.
+ - Homepage `unlisted-site.spec.ts`: the builder's input holds the unlisted site (`listed:
+ false`, its own records, the shared channel), and listed it would add a card and its records;
+ the summary the dev server reads equals the one built without it; `/` and `/stats/` (served
+ HTML and DOM) name all six listed sites and not the unlisted one.
+ - `homepageSummary.test.ts`, after the review: an unlisted id that sorts BEFORE the listed one
+ (`aaa-hidden` < `beta`) still leaves the shared channel with the listed site.
+- **The homepage e2e fixture** (`homepage/e2e/fixture-summary.ts`), for the slices that build on
+ it:
+ - `FIXTURE_SITES`: the six listed sites, unchanged.
+ - `FIXTURE_UNLISTED_SITE`: `fixture-unlisted`, "Fixture Unlisted", two channels of its own
+ (`fixture-unlisted-ch1/2`, six a day, like the rest); it also exposes `fixture-one-ch1`.
+ - `buildFixtureInputs(fixtureSites = FIXTURE_SITES, unlistedSites = [FIXTURE_UNLISTED_SITE])`
+ returns the builder's inputs (`stats`, `channelSites`, `sites`). The first list is listed;
+ every site in the second is written with `listed: false`, whatever it carries (`FixtureSite`
+ has no `listed` field), and shares the first listed site's first channel.
+ - `buildFixtureSummary(…)` is the real builder over those inputs. The unlisted sites' records are
+ generated last, after the megaspike, so every listed record is the same with them or without,
+ and `buildFixtureSummary()` equals `buildFixtureSummary(FIXTURE_SITES, [])`. Passed in the
+ FIRST list, the same site is listed: 7 sites and 39,599 transcripts instead of 6 and 37,199.
+
+#### Proof: a hidden fixture site through the real builds
+
+A throwaway corpus in the job's scratch dir (`$T/hs-proof/`, `make-corpus.mjs`): three channels
+of three captioned videos each, and two sites with public URLs — `fixture-listed` (the listed
+channel and the shared one) and `fixture-unlisted` (`listed: false`; the shared channel and one of
+its own). Every path the builds write was pinned there (`TRANSCRIPTS_DIR`, `SETTINGS_FILE`,
+`EXPORT_INDEX_DIR`, …) except the two apps' own `public/` and `out/`. The worktree's own gitignored
+`homepage/public` data and `export/out` were set aside first and put back after; the
+`export/public` links were dropped (never their targets) and re-seeded from the primary after. The
+primary's `export/public` was untouched (no entry newer than the slice's start). Each build was
+capped at 5 GB with no swap.
+
+| Build | Result | Time | Max RSS |
+|---|---|---|---|
+| `archilyzer index` | 0 | 4 s | — |
+| `archilyzer build homepage --no-source` | 0 | 20 s | 776 MB |
+| `archilyzer build hub` | 0 | 34 s | 1,005 MB |
+| `archilyzer build site fixture-unlisted --skip-archives` | 0 | 41 s | 962 MB |
+| `archilyzer build site fixture-listed --skip-archives` | 0 | 84 s | 970 MB |
+
+Counts (files holding the string / occurrences, `grep -rF`):
+
+| Tree | `fixture-unlisted` | its title | its own channel's slug | its own channel's name | `fixture-listed` |
+|---|---|---|---|---|---|
+| `homepage/public` (summary, `channel-sites.json`, `stats/`) | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 2 / 25 |
+| `homepage/out` | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 10 / 145 |
+| `export/public` (hub compose) | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 4 / 10 |
+| `export/out` (hub) | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 4 / 10 |
+| `export/out` (the listed site) | 0 / 0 | 0 / 0 | 0 / 0 | — | 9 / 30 (its URL) |
+
+- The compose lines: `compose-homepage: 2 channel(s) mapped across 1 listed site(s) (1 unlisted
+ left out); summary covers 6 transcription(s) / 6 download(s) across 1 public site(s)` and
+ `compose-hub: 1 built-in pool site(s) …; hub-summary.json covers 1 official instance(s)`.
+- The summary: version 6; `totals` 6 transcripts, 6 downloads, 1 site, 2 channels, 6 hours;
+ `official` the same; `availability.counted` 6. With the unlisted site counted they would have
+ been 9 transcripts and 9 hours.
+- The pooled `stats/` manifest: 6 records, channels `proof-listed-channel` and
+ `proof-shared-channel`. `channel-sites.json` maps both to `fixture-listed` alone.
+- **The unlisted site still builds:** its own stats bundle holds all six of its videos (both its
+ channels); its build ships its own pages (its id in 17 files), and its footer links
+ `https://fixture-listed.example`. The listed site's footer links nothing: its only sibling is
+ unlisted.
+
+#### Gates (at `64ae9b47`, the tree of the first `e12515c2`; logs `$T/hs-*.log`)
+
+- **tsc** was clean before every commit (69 s, 33 s, 44 s — the last over the tip's code).
+- **Unit:**
+
+ | Suite | Result |
+ |---|---|
+ | common | 2,218/2,218 (8 new) |
+ | editor unit | 87/87 |
+ | homepage unit | 12/12 |
+ | `test:scripts` | 191 passed, 1 skipped (192) |
+ | mcp | 271/271 |
+
+- **Docs:** `docs files --check`, `settings example --check` and `docs env --check` all exit **0**.
+- **e2e**, each detached and queued:
+
+ | Suite | Passed | Failed | Time |
+ |---|---|---|---|
+ | homepage, full (the new `unlisted-site.spec.ts` 3) | 97 | 0 | 3.6 min |
+ | hub, full | 36 | 0 | 1.4 min (after 3.5 min in the queue) |
+ | editor: `sites-crud` (the new listed round-trip 1) | 15 | 0 | 0.9 min (after 11 min in the queue) |
+
+- **Builds:** the five above, in the proof. The editor's and umtool's `next build` were not run (no
+ route or bundled path changed; tsc covers the form and the action).
+- **Numbers tool:** none.
+- **After the review and the merge of `main` (at `7ea35dfb`; `$T/hs-gates3.log`):** tsc clean
+ (183 s, the machine under load); common 2,229/2,229 (release 15 IG's 2,220, this slice's 8 and
+ the review's 1); homepage unit 12/12; homepage e2e `unlisted-site.spec.ts` 3 passed, 0 failed
+ (19 s). The fixture's second list unlisting a site that carries no key, and the listed variant's
+ 7 sites / 39,599 transcripts, were checked by a script (`$T/hs-fixture-check2.ts`).
+
+#### Review (verdict SHIP; `$T/hs-review.md`)
+
+| Finding | Fix |
+|---|---|
+| L1: the fixture's second list relied on each site's own `listed: false` | `e3ee2eb1`: `buildFixtureInputs` writes `listed: false` on every site in it; `FixtureSite` has no `listed` |
+| L2: the spec's first case would pass if the builder ignored the unlisted site | `e3ee2eb1`: it asserts the site is in the input (unlisted, its records, the shared channel) and that, listed, it adds a card and exactly its own records |
+| L3: every unlisted id sorted after the listed one | `65675373`: `aaa-hidden` sorts before `beta`; the summary is still the one without it |
+| L4: `unlisted` / `listed` already mean other things | `20ff9ecf`: a FACTS "Naming hazards" table of the three |
+| L5: the listed site's out not searched for the own channel's name; the export site suite and `e2e:2origin` not run | left: both changes are no-ops for a site without the key, and the unit test and the real build's footer cover them |
+| L6: this record's slices table, Order and Rollout did not name HS; the hub check's N | `20ff9ecf`: HS row and Order; the Rollout names HS, counts public LISTED sites and checks the new box |
+
+#### Found and left
+
+- **The unlisted site's own `/site.json` and `/corpus.json` still carry its `hubUrl`** (ruling 4:
+ it deploys exactly as before). A visitor who adds its origin to the hub by hand gets it as any
+ added origin, and the hub can read that `hubUrl` as a family member's.
+- **`listed` / `unlisted` mean three things:** a video's visibility, `useHubSites`' `listed`
+ flag (the hub's list has loaded), and a site the family lists. FACTS' "Naming hazards" has the
+ table since the review.
+- **The Rollout's hub check** ("`hub-summary.json covers N official instance(s)`") counts public
+ LISTED sites; the Rollout says so since the review.
+- **Unlisting takes effect at the next builds.** The homepage, the hub and every other site are
+ static: until each is rebuilt and deployed, it still lists the site.
+- **The editor's `/sites` list** shows no marker for an unlisted site; the form's checkbox is the
+ one place (ruling 3).
+
+#### Decisions the operator could overturn
+
+| What I assumed | The alternative |
+|---|---|
+| `totals` and `availability` keep counting pool-only channels and channels of sites with no public URL, as before; only a channel exposed by unlisted sites alone leaves them | `totals` count the listed public sites only, the same scope as `official` |
+| A channel an unlisted site shares with a listed site with NO public URL is not "only on unlisted sites", so `totals` count it | count a channel only when a listed PUBLIC site exposes it |
+| The pooled `stats/` keep pool-only channels, as before | the pooled `stats/` hold only channels a listed public site exposes |
+| `isListedSite` lives beside the key in `siteSchema.ts`, exported from `lib/site` | define it in `lib/site.ts` itself, and have the summary builder import `lib/site`'s file I/O |
+| The summary's version is 6 | stay at 5 (no field changed) |
+| An unlisted site's own footer still links its listed siblings | an unlisted site shows no related-sites footer |
+| The checkbox sits after Public URL, with a hint naming what it removes | at the end of the form, or without a hint |
+
+### Slice S1, as shipped — a clear screen until the first Search (2026-09-29)
+
+Branch `r14/first-search` off `main` `99d4d76a` (`r14/two-grounds-headers` merged), worktree
+`~/Projects/plans-export-header-first-search` (block #4: export e2e 3420, hub e2e 3441, editor test
+3411), one Opus implementer. Scratch files `s1-*` in the job's `tmp`.
+
+The rulings, 2026-09-29, not re-opened:
+1. S1 only. The summaries download is not deferred; S2 stays a candidate.
+2. A link carrying only a filter (`tg=`, `fv=`, `ch=`), like one carrying `qt=`/`q=`, runs on
+ load and shows its results.
+
+The parent's directions for the build (the plan's S1 steps):
+- The gate is in `SearchResults`, which the hub mounts too, so the hub gets the same behaviour and
+ `e2e:hub` joins the gate.
+- The module-level `ranThisPageLife` feeds a reactive `searchedThisPageLife`; the dead
+ `searchExecuted` goes. As built after the review, the flag the view reads is a second module
+ flag, `askedThisPageLife`, and `ranThisPageLife` keeps `main`'s rule (review M1, below).
+- A URL with `qt`, `q` or any filter key counts as asked, at hydration.
+- `SearchResults` renders nothing until the flag is set; `resultGroups` is still computed. The
+ page's intro (the transcript count, the Welcome card) stays.
+- The bar's "Press Enter or click Search to apply" line also shows before the first Search.
+
+My own choices are under "Decisions the operator could overturn".
+
+| sha | what |
+|---|---|
+| `93d72a80` | `common:` the results area renders nothing until the visitor asks in this page life; the bar's line shows meanwhile; `searchedThisPageLife`; `searchExecuted` deleted |
+| `90043b78` | `export:` `first-search.spec.ts`; one `showAll` helper, pressed where the export and hub specs read the listing at load |
+| `dbdaa31b` | `plans:` this record; the plan's S1 as built; the export changelog |
+| `9b01068c` | `common:` two page-life flags, one job each; a filter-only link no longer releases a stored query (review M1); the held-query comment (L1) |
+| `41dc1b4b` | `export:` `restore-no-refire` covers a second mount after a filter-only link (M1) |
+| _this_ | `plans:` the review's fixes recorded (L3, M1, L2 left); the S1 row of the slices table and the Rollout's checks (L4) |
+
+- **What a plain visit shows**, on a site's `/` and on the hub's:
+ - the search bar with its line, "Press Enter or click Search to apply";
+ - the page's intro (on a site the transcript count and the Welcome card; on the hub the shelf,
+ the figures and the archive chips);
+ - the footer, whole on the first screen at 1280×800 and 390×844 (the spec asserts it).
+ - No count, no Results/Chart toggle, no Copy for AI, no selection toolbar, no chart, no listing,
+ no `browse-hint`, and on the hub no "N of M archives answered" line.
+- **What counts as the first Search:** Enter, the Search button, a Filters Apply (all through
+ `commitSearch`), a profile load or Revert (`applySnapshot`), and a URL that asks.
+ - An empty Search shows "All videos (N)" and the listing, as before.
+ - A filter changed before it shows nothing; the line was already there for an unapplied edit.
+- **A URL that asks** (`urlAsks`, `SearchSessionContext.tsx`): any of `qt`, `q`, `tg`, the legacy
+ filter keys (`ch nov nol naa nar nav nd nu m tk`) or the share-link keys
+ (`fv fc ft fa fav fk fdf fdt`).
+ - Presence counts, not a valid value: `?tg=Not%20An%20Id` shows every video, which is what
+ `tag-chips.spec` already asserts.
+ - A video (`v`, `t`), a chart's shape (`view`, `cs`), `re` and `vm` do not ask. The Share button
+ always writes `fv=`, so a shared chart does run.
+- **Two flags, one job each** (module state: they survive client-side navigation and reset on a
+ reload):
+ - `ranThisPageLife` is the hold, exactly as on `main` (release 8). It is set by a commit, a
+ profile load, or an active URL query. While it is unset, a stored query is held, on this
+ mount and on every later one.
+ - `askedThisPageLife` is the gate: `ranThisPageLife`, or a URL that asks. `searchedThisPageLife`
+ starts from it and is set wherever it is set: at hydration, in `commitSearch` and in
+ `applySnapshot`.
+ - So a filter-only link shows its results and runs no query. A stored query stays held on that
+ load, and on a later mount in the same visit, which shows the listing under the held query.
+ - On the server neither module variable is set, so the first client render matches the static
+ HTML. The exported `index.html` now has the line and no results section.
+- **Step 0, settled:**
+ - **The hub renders `SearchResults`:** `HubHome.tsx` → `TranscriptSearch` (common) →
+ `SearchResults`. So the hub has the clear screen, and `e2e:hub` ran in the gate.
+ - **Back restores the listing.** Measured with a probe (not committed) on the 120-video fixture
+ at 1440×1200, scrolled to 3,000 px before leaving:
+
+ | Path | After Back | Scroll before → after |
+ |---|---|---|
+ | empty Search → **Use with AI** (header) → Back | the listing, "All videos (120)" | 2,838 → 2,676 px (the first card drawn: `0024` → `0025`) |
+ | `qt=` link → **Use with AI** → Back | the results | 2,952 → 2,904 px |
+ | empty Search or `qt=` → **Chat** (workspace nav) → Back | the listing | → 0 (the top) |
+ | a card's video (the modal) → Back | leaves the page | — |
+
+ - A route outside the workspace unmounts the session; Back remounts it with the flag already
+ set, and the scroll comes back to within a card.
+ - `/ask` hides the search pane in place, and the window is at the top when Back shows it again.
+ - The modal opens with `replaceState`, so there is no entry of the page's own to go Back to.
+ - The last two are unchanged by S1.
+- **Tests:**
+ - `first-search.spec.ts`, new (11):
+ - on load, no results area, the line, the intro and the whole footer, at 1280×800 and 390×844;
+ - Enter on an empty box shows every video, and the line goes;
+ - the Search button shows every video;
+ - a filter changed before the first Search shows nothing;
+ - `/` → `/ask` → `/` keeps the listing;
+ - Back from Use with AI keeps it;
+ - a reload clears it;
+ - a `qt=` link shows its results on load;
+ - `tg=` and `ch=` links show theirs;
+ - a restored query shows the clear screen and the filled form, and runs on Enter.
+ - `showAll(page)` in `export/e2e/helpers.ts`: wait for the query builder, press Search, wait for
+ `results-summary`.
+ - Where specs read the listing at load:
+ - `browse-all`: the first test rewritten to assert the clear screen, then the listing;
+ - `charts` (7 tests), `workspace-shell` (1), `responsive` (the filters sheet and the selection
+ toolbar), `tag-chips` (3; its `?tg=` tests unchanged);
+ - `restore-no-refire`: the held query now shows no results area. Two new tests cover a second
+ mount after a filter-only link, and each holds with no shard fetched: `?tg=` → Use with AI →
+ Back, and `?ch=` → Use with AI → the header's Search link;
+ - the hub's `federated-search` (8): its first test asserts the clear screen with both archives
+ in, then the listing.
+
+#### Gates (at `90043b78`; logs `$T/s1-*.log`)
+
+- **tsc** was clean before each commit and at the tip (36–38 s). The specs commit's own run
+ timed out at 100 s under the machine's load; it was run again on that commit, clean.
+- **Unit:** common **2,210/2,210** (75 s). Editor unit, `test:scripts` and mcp were not run: S1
+ touches no editor, mcp or scripts file, and the editor imports none of these components.
+- **Builds**, each capped at 5 GB with no swap, from a clean `.next`:
+
+ | Build | Time | Max RSS |
+ |---|---|---|
+ | export, site (the worktree's default) | 36 s | 957 MB |
+ | export, hub | 28 s | 1,034 MB |
+
+- **e2e**, each detached and queued:
+
+ | Suite | Passed | Failed | Time |
+ |---|---|---|---|
+ | export, the specs above first | 58 | 1 | 3.4 min |
+ | hub, full | 36 | 0 | 1.7 min |
+ | export, full | 254 | 0 | 11.4 min |
+ | `e2e:2origin` (`E2E_TWO_ORIGIN_REBUILD=1`, the `export/public` links in place) | 3 | 0 | 49 s |
+
+ The one failure in the first run was the new `/` → `/ask` test: that run's first visit to `/ask`,
+ compiled by the dev server, took longer than 5 s. The test now waits 20 s for the URL. After
+ `e2e:2origin`, the primary's `export/public` files kept their 2026-09-28 mtimes.
+- **Numbers tool:** none.
+
+#### Found and left
+
+- **Back from `/ask`** shows the listing at the top, not where the reader left it (the table
+ above). This is as before S1.
+- **A card's video opens without a history entry**, so Back from the modal leaves the site. This is
+ as before S1.
+- **Enter before the session hydrates (review L2).** Until the manifest is in, a `qt=` link shows the
+ line and not yet its results. Enter in that window commits the empty draft, and `commitSearch`
+ strips `qt` before hydration reads it. The race is on `main` too; the line now shows during it.
+ Gating the line's first-Search half on hydration would close it, at the cost of the line in the
+ static HTML. Left for the operator.
+
+#### Decisions the operator could overturn
+
+| What I assumed | The alternative |
+|---|---|
+| The line also shows under the bar on a fresh `/ask` (the bar is shared, and Search there applies the grounding) | show it only on the search view |
+| A link with only a chart's shape (`view=chart&cs=…`, copied from the address bar after an empty Search) opens on the clear screen | count `view`/`cs` as asking |
+| A video link (`?v=`) opens the video over the clear screen | count `v` as asking |
+| On the hub, the "N of M archives answered" line waits for the first Search with the rest of the results; each failed archive's chip says so, with its Retry, before that | show the line above the gate |
+| After a filter-only link, a later mount in the same visit (a page outside the workspace and back, the header's Search link) shows the listing, with the stored query held in the box: the gate is once per page life | the clear screen again on that mount |
+| A key's presence asks, not a valid value (`?tg=Not%20An%20Id`) | count only the keys the page applied |
+| The Search button keeps its outline look before the first Search; only the line says what to do | fill it as for an unapplied edit |
+
+#### Review (verdict SHIP AFTER FIXES; `$T/s1-review.md`)
+
+The reviewer also ran the editor suite's specs that drive the export's bar, results and `?v=`
+modal (`export-search`, `export-player-platform-cache`): 21 passed, 0 failed.
+
+| Finding | Fix |
+|---|---|
+| M1: a filter-only link set `ranThisPageLife`, so on a later mount in the same visit (a page outside the workspace and Back; the hub's `/` → `/ask`) a query stored by an earlier visit ran and fetched shards, where `main` held it. The reviewer's probe: `?tg=` → Use with AI → Back fetched 1 shard and showed "Matching videos"; `?ch=` → Use with AI → home did the same | `9b01068c`: two module flags. `ranThisPageLife` is set at hydration only by an active URL query, as on `main`, and `askedThisPageLife` (a run, or `urlAsks`) drives the gate. `41dc1b4b`: `restore-no-refire` seeds a stored query and checks both paths: 0 shards, the box filled, Search marked, and "All videos" |
+| L1: the held-query comment still said "the browse listing" | `9b01068c` |
+| L2: Enter before hydration on a `qt=` link commits the empty draft (a race on `main`; the line now shows during it) | left: "Found and left" |
+| L3: the rulings list mixed the rulings with the plan's steps and my choices | this commit: the rulings, the parent's directions, and my decisions, apart |
+| L4: the slices table's S1 row, and the Rollout's live checks | this commit: the row; three S1 checks under "Live checks" |
+
+#### Gates after the fixes (at `41dc1b4b`; logs `$T/s1-*2.log`, `$T/s1-fix-e2e.log`)
+
+The fix changes only which flag each place sets, so the full suites were not re-run.
+- **tsc** clean before each fix commit (182 s under the machine's load).
+- **common** 2,210/2,210.
+- **e2e**, detached and queued (spec lists in `$T/s1-fix-specs.txt`, `$T/s1-fix-hub-specs.txt`):
+
+ | Suite | Specs | Passed | Failed | Time |
+ |---|---|---|---|---|
+ | export | `first-search`, `restore-no-refire`, `browse-all`, `workspace-shell` | 20 | 0 | 5.1 min |
+ | hub | `federated-search` | 12 | 1 | 2.5 min |
+ | hub | `federated-search`, again | — | — | the dev server did not answer in 120 s |
+ | hub | `federated-search`, again | 13 | 0 | 1.7 min |
+
+ The hub's first run failed on its first test's first line, the archive chip still "loading"
+ after 5 s. That line is unchanged from `main` and comes before anything S1 changed; the machine's
+ load average was 33. On the second attempt the dev server did not start within 120 s. The third
+ run passed 13/13.
+
+### Slice CF, as shipped — the growth chart folds the small sites into Other (2026-09-29)
+
+Branch `r14/chart-fold` off `main` `69e058d6` (slice HS merged), worktree
+`~/Projects/homepage-social-visible` (block #3: homepage e2e 3340, homepage static 3331), one Opus
+implementer. Scratch files `cf-*` in the job's `tmp`. The rulings (2026-09-29, not re-opened):
+1. A site under **5 %** of the placed total — the chart's total over the whole plotted range, the
+ sum the bands are placed from — folds into ONE "Other" band, drawn on top of the stack, in a
+ neutral grey from the chart tokens that keeps ≥ 3:1 on the Light and the Dark ground.
+2. A fold of one is no fold: with a single site under the threshold, nothing folds.
+3. The legend shows the kept sites plus "Other"; the hover title and the "Numbers by year" table
+ still name every site.
+4. Colour follows the site, never its rank: a kept site's colour does not change because another
+ folded.
+5. The instance cards and `/stats` are unchanged; the surface gap draws correctly with Other on
+ top.
+6. (The review, ruled by the parent.) Other wears a dedicated `--chart-other` token in both chart
+ blocks of `tokens.css`: Light `#3e545c`, Dark `#62625c` (the near-neutral).
+
+| sha | what |
+|---|---|
+| `8351d24f` | `homepage:` the fold in `lib/growthGaps.ts` (`foldedSites`, `growthLayers`, `ownLayers`, `layerColors`; `growthStack` stacks layers); the chart draws the layers, its legend, label and caption; the unit tests; the e2e fixture's fifth site at 4 a day |
+| `6e0b652d` | `homepage:` e2e: `growth-chart.spec.ts` (the fold, the grey), `instance-colours.spec.ts`, `marketing.spec.ts` adjusted |
+| `e95a8058` | `plans:` this record, the slices table, Order and Rollout; FACTS; the homepage changelog |
+| `17832633` | `common, homepage:` `--chart-other` in both chart blocks with its numbers; `OTHER_COLOR` points at it; the chart's header comment; `growth-chart.spec.ts` resolves `OTHER_COLOR` and pins each base's value (review M1) |
+| `c42d2bbc` | `homepage:` `instance-colours.spec.ts` reads each site line's stroke on `/stats/`, the Vermilion site's the rust (review L2) |
+| `afddd66e` | `plans:` the review's fixes in this record (the grey, the Review table, the screenshots, Found and left, Decisions); the CF row's files; FACTS; the homepage changelog |
+| `9267ec08` | merge of `main` `721ed0eb` (slice S1); `plans/release-14.md` resolved by hand (below), the changelogs clean |
+| _this_ | `plans:` the post-merge gates; the commit table |
+
+**What shipped.**
+- **The rule** (`homepage/app/lib/growthGaps.ts`, pure, beside `growthStack`):
+ - `foldedSites(months, sites)`: each site's sum over the plotted months against the sum of all of
+ them. A site is under when `100 × its sum < FOLD_PERCENT × the total` (`FOLD_PERCENT = 5`), on
+ integers, so exactly 5 % keeps its band. The folded sites are returned only when two or more
+ are under; none when the total is 0.
+ - `growthLayers(months, sites)`: the kept sites in stack order (the summary's, `STACK_ORDER`
+ unchanged), then `{ key: "(other)", sites: [the folded indexes], other: true }` — a key no site
+ id can be (`[a-z0-9][a-z0-9-]*`). `ownLayers(sites)` is every site its own band.
+ - `growthStack(months, sites, layers = ownLayers(sites))`: one band per layer, a layer's value
+ in a month the sum of its sites'. It returns `layers` in place of `order` (the chart was
+ `order`'s only reader). Totals, peak, value scale and gridlines are over every site, so the
+ stack's top and the scale are the same folded or not.
+ - `layerColors(layers, sites)`: `siteChartColors` over EVERY site, and a kept site takes its own
+ entry, so a fold never repaints one (with no accents, the kept list alone would move a site
+ after a folded one to a lower slot; the unit test proves it); Other is `OTHER_COLOR`.
+- **The grey is its own token, `--chart-other`** (`common/styles/tokens.css`, beside `--chart-axis`
+ in both chart blocks, each with its numbers in a comment as `--chart-6` has; review M1): Light
+ `#3e545c` (OKLCH L 0.43, C 0.03), Dark `#62625c` (L 0.49, C 0.01). Contrast: 7.36:1 on the Light
+ ground and 7.99:1 on its chart surface; 3.22:1 on the Dark ground and 3.06:1 on its chart surface
+ (the Dark slots are 3.37–6.46:1 on the ground: Other is the dimmest mark there). Against each
+ chart slot it can sit on (the dataviz skill's validator's measures, OKLab ΔE × 100, normal / the
+ worse of protan and deutan):
+
+ | Slot | Light | Dark |
+ |---|---|---|
+ | blue (`--chart-1`) | 21.9 / 21.4 | 17.8 / 17.9 |
+ | green (`--chart-2`) | 17.4 / 15.2 | 19.2 / 15.7 |
+ | violet (`--chart-3`) | 16.2 / 13.4 | 23.4 / 21.5 |
+ | amber (`--chart-4`) | 22.1 / 18.2 | 20.3 / 18.1 |
+ | magenta (`--chart-5`) | 24.5 / 9.1 | 19.5 / 7.3 |
+ | rust (`--chart-6`) | 14.1 / 10.5 | 13.5 / 9.3 |
+
+ - Every CVD pair clears the floor (6) on both bases, and the target (8) on Light. Dark's magenta
+ (7.3) is in the 6–8 floor band, legal with the legend, the table and Other's place on top.
+ Rust is the one pair under the normal-vision 15, on both bases.
+ - A grey is under the validator's chroma floor by definition (it is the de-emphasis role, not a
+ categorical slot).
+ - Other touches only the kept site beneath it (the highest with a height that month). In
+ today's data that is Bonnellyzer, blue, in every month Other has data.
+ - The first build used `--chart-axis` (Light `#55646e`, Dark `#8a8170`: magenta at CVD 4.5 /
+ 4.2, and on Dark green at 4.1, under the floor). The review's sweep found in-band greys that
+ clear CVD 8 against every slot; the parent ruled the token above.
+- **The chart** (`ArchiveGrowthChart.tsx`):
+ - the legend is the layers: the kept sites' titles, then "Other";
+ - the areas and the gaps are keyed by layer;
+ - the hover title is unchanged: every site with data that month, by name, folded or not;
+ - the table is unchanged: a column per site;
+ - the image's `aria-label` gains, when something folds, "Hasanalyzer, Rekietalyzer and
+ Jasolyzer, each under 5% of the total, are drawn together as Other." (the legend is hidden
+ from assistive tech);
+ - the caption gains, when something folds, "Instances under 5% of the total are drawn together as
+ Other.";
+ - the header comment carries the grey's numbers.
+- **The gaps are unchanged code.** They work on bands, so Other is one band, the top one:
+ `bandAbove` of the top kept site is Other wherever Other has a height, and Other's own upper edge
+ has no gap (nothing sits on it). On today's summary the gap segments are the same folded and
+ unfolded, and none is drawn under Other: Bonnellyzer is under 3 px wherever Other sits on it, so
+ they touch, as the three small bands did before.
+
+ | Plot height | Segments | Runs | Under Other |
+ |---|---|---|---|
+ | 200 px | 73 | 7 | 0 |
+ | 260 px | 112 | 6 | 0 |
+ | 300 px | 119 | 7 | 0 |
+
+ On the fixture, gaps are drawn under Other, and no band is covered at any height (unit test).
+- **Unchanged:** the instance cards, `/stats`, `siteChartColors`, the hub; in `tokens.css` only
+ `--chart-other` is added.
+- **With today's data** (the primary's `homepage-summary.json` of 2026-09-28, 75,821 transcripts
+ on the chart): Jeralyzer 42.03 %, Anilyzer 38.40 %, Bonnellyzer 9.18 % keep their bands;
+ Hasanalyzer 4.33 %, Rekietalyzer 3.79 % and Jasolyzer 2.26 % are Other.
+
+**The e2e fixture** (`homepage/e2e/fixture-summary.ts`). Its six sites were 60.21, 12.90, 9.68,
+8.60, 5.38 and 3.23 % of the chart: one under 5 %, no fold. The smallest change that gives two:
+`fixture-five` transcribes 4 a day, not 5 (1,600 recordings, not 2,000). The shares are now 60.87,
+13.04, 9.78, 8.70, **4.35** and **3.26 %**, and `fixture-five` and `fixture-six` fold. The
+unlisted site stays out of the chart (it is out of the summary). Every changed expectation:
+- The fixture's listed totals: 36,799 transcripts, not 37,199; with the unlisted site listed,
+ 39,199, not 39,599 (HS's record has the old pair). No spec reads either: `unlisted-site.spec.ts`
+ computes both sides.
+- `growth-chart.spec.ts`, the pixel test: one colour per band (four kept sites and Other), not per
+ site.
+- `instance-colours.spec.ts`: the legend has five swatches, not six (the four kept sites in their
+ slots among all six, then Other's grey); `fixture-five`'s card is no longer compared with a
+ legend swatch (it has none); `fixture-six`'s hue check reads its chart colour, the rust, not a
+ legend swatch.
+- `marketing.spec.ts`: the caption's check is `/all official instances\.( |$)/i`, not `/…\.$/i`,
+ since the Other sentence can follow.
+
+**Tests.**
+- `growthGaps.test.ts`: the fixture test now runs the chart's layers (the fold is `[4, 5]`; a gap is
+ drawn between the top kept site and Other; none along Other's top; no band covered at any
+ height, folded or not). New:
+ - today's proportions (42.03 / 38.40 / 9.18 / 4.33 / 3.79 / 2.26 %, the family's accents): the
+ three largest keep their bands, Other holds the other three, its top is every month's total,
+ the colours are amber, magenta, blue and the grey;
+ - the edge: 999 of 20,000 (4.995 %) folds, 1,000 (exactly 5 %) does not; two sites at exactly
+ 5 % fold nothing;
+ - a single site under 5 %: no fold;
+ - no site under: `growthLayers` is `ownLayers`, and the stack is `growthStack`'s default;
+ - all but one under: one kept band and Other;
+ - a site with nothing in the range folds with another; nothing plotted folds nothing;
+ - 25 equal sites: every site folds (see "Decisions");
+ - colour stability: the kept sites' colours equal the unfolded run's, and differ from what the
+ kept list alone would give.
+- `growth-chart.spec.ts`, new:
+ - two sites under 5 % are one Other band on top: the layers are the four kept sites and
+ `(other)`; the legend reads the four titles and "Other"; the areas' fills, in paint order, are
+ the four slots and `OTHER_COLOR` (`var(--chart-other)`) last; the label names "Fixture Five and Fixture Six";
+ the caption says what Other is; every month's title names every site with data that month and
+ no title says "Other"; the table's header is Year, the six titles, Total;
+ - Other is `OTHER_COLOR` on both grounds, rendered `rgb(62, 84, 92)` on Light and `rgb(98, 98, 92)`
+ on Dark (so each base declares its own), not the axis colour, at least 3:1 on the ground, and no
+ kept site's colour (review M1).
+- `instance-colours.spec.ts`, new (review L2): on `/stats/`, on both bases, the six site lines'
+ rendered strokes are the six sites' chart colours in the summary's order; the Vermilion site's is
+ the rust (`--chart-6`) and within 25° of Vermilion's hue. The card test's own site-5 check is the
+ card's hue against its chart colour; the tautology it replaced (`chart[5]` against
+ `var(--chart-6)` twice) is gone.
+
+**They bite** (each change made by hand, the unit tests run, the change reverted):
+- `<=` for `<` in the rule: the edge test fails.
+- A fold of one allowed: the single-site test fails.
+- Colours from `siteChartColors` over the kept sites alone: the colour-stability test fails (the
+ today's-proportions test does not: its sites' slots come from their accents).
+- Other at the bottom of the stack: four tests fail (the fixture's, today's, all-but-one, colour).
+
+#### Gates (at `6e0b652d`; logs `$T/cf-*.log`)
+
+- **tsc** clean in every package on the tree of `6e0b652d`, run before the first commit (79 s);
+ `8351d24f`'s tree has `main`'s versions of the three spec files, which import nothing it changed.
+- **Unit:** common **2,229/2,229**; homepage **20/20** (12 before, 8 new).
+- **Build**, capped at 5 GB with no swap, from a clean `.next`, with the primary's summary copied
+ into the worktree's `homepage/public` for the screenshots (the worktree's own put back after):
+ `pnpm --filter homepage exec next build` **ok**, 39 s, max RSS 779,504 KB. `main`'s chart,
+ built the same way for the "before" shots: 80 s, 754,160 KB. After the review (`17832633`, for
+ the retaken shots): 20 s, 809,752 KB.
+- **Homepage e2e, full** (97 at `main` after HS; 2 new):
+
+ | Run | Passed | Failed | Time |
+ |---|---|---|---|
+ | first (`$T/cf-e2e-homepage.log`, after 9.5 min in the queue) | 98 | 1 | 4.2 min |
+ | again (`$T/cf-e2e-homepage2.log`, after 5.5 min in the queue) | **99** | **0** | 3.9 min |
+
+ The first run's failure was `marketing.spec.ts`' "Changelog is reachable from the footer on
+ every page": the 30 s test timeout, reached on the seventh of its seven pages (the dev server
+ compiling each on first visit, the machine's load average 11–17 with other suites running). It
+ passed in 5.3 s in the second run and in the three-spec run below; nothing in it reads the
+ chart.
+- **Along the way:** `growth-chart`, `instance-colours` and `marketing` specs, 21 passed, 0 failed
+ (1.8 min).
+- **Numbers tool:** none.
+- Not run: the export, hub and editor suites and their builds (no file of theirs changed).
+
+**Screenshots** (`~/reports/release-14/shots/chart3/`, 2×, the static server on 3331, today's
+summary):
+- `after-{390,1280}-{light,dark}.png`: the chart with `--chart-other` (retaken after the review;
+ the operator judges the Dark one); `…-2019-2026.png`: the plot from 2019 on, where Other lies;
+- `after-axis-…`: the same from the first build, Other in `--chart-axis`, for comparison;
+- `before-…`: the same from `main`'s chart and the same summary (six bands);
+- `after-1280-{light,dark}-table.png`: "Numbers by year" open, a column per site.
+
+**Found and left:**
+- **The pairs short of the validator's targets** are Dark's magenta (CVD 7.3, the 6–8 floor band)
+ and rust on both bases (normal 14.1 / 13.5). They matter only when that site is the top kept one under Other; today it
+ is blue.
+- **`/stats`' channel breakdown has an "Other (N)" of its own**, in `--muted-foreground`
+ (`common/lib/homepageChart.ts` `OTHER_COLOR`), not `--chart-other`. `/stats` is unchanged, as
+ ruled.
+- **`--chart-other` is not in `REQUIRED_TOKENS`** (`common/components/themeConfig.ts`, the list
+ `themeTokens.test.ts` checks every base declares): that file is outside this slice.
+ `growth-chart.spec.ts` pins each base's rendered value instead (a Dark block without it would
+ paint the Light value from `:root`).
+- **The fold is the homepage chart's only.** `/stats`, the hub and the sites' charts draw every
+ site, as ruled.
+
+#### Decisions the operator could overturn
+
+| What I assumed | The alternative |
+|---|---|
+| The caption gains one sentence saying what Other is, only when something folds | the caption unchanged |
+| The image's label names the folded sites | name only how many |
+| The legend reads "Other", with no count or names | "Other (3)" |
+| The hover title lists every site with data in the summary's order, with no Other subtotal | an "Other N" line, its sites under it |
+| When every site is under 5 % (21 or more sites), every site folds: one Other band | keep the largest, or fold nothing |
+| A site with nothing in the plotted range is under 5 % and folds with another | leave it out of the chart |
+
+#### Review (verdict SHIP AFTER FIXES; `$T/cf-review.md`)
+
+| Finding | Fix |
+|---|---|
+| M1: the axis grey was under the CVD floor against magenta on both bases (4.5 / 4.2) and against green on Dark (4.1); the record's sweep sentence overstated the case against a better grey | `17832633`: `--chart-other` in both chart blocks (Light `#3e545c`, Dark `#62625c`, as ruled), `OTHER_COLOR` points at it, the chart's comment and the spec follow; the sweep sentence is withdrawn and the token's measured numbers stated ("What shipped"); the labels' shared colour is gone from Found and left; the four `after-*` shots retaken, the first build's kept as `after-axis-*` |
+| L1: the CF row omitted `plans/FACTS.md` | `afddd66e`: the row lists it, and `common/styles/tokens.css` |
+| L2: `instance-colours.spec.ts` checked the sixth site's colour against itself | `c42d2bbc`: a rendered check of every site line's stroke on `/stats/`, the sixth the rust |
+| L3: "Found and left" left rust out of the weak pairs | moot with M1; the line names Dark's magenta and rust |
+| L4: the merge of `main` (`721ed0eb`, S1 merged) conflicts in this record only | `9267ec08`: S1's row, then CF's; S1's section, then CF's, before "## Rollout"; the Order line and the Rollout's intro name S1 and CF; both sets of live checks. The changelogs merged clean, every bullet under `[Unreleased]` (checked by eye) |
+| L5: the changelog's "at this release's numbers" will drift | left, as a release note |
+
+The caption and the image-label sentences are kept (the review: acceptable additions).
+
+**Gates after the review and the merge of `main`** (at `9267ec08`; logs `$T/cf-tsc2.log`,
+`$T/cf-tsc3.log`, `$T/cf-gates2.log`, `$T/cf-e2e3.log`):
+- **tsc** clean in every package before the fix commits (44 s) and on the merged tree before the
+ merge commit (164 s).
+- **Unit:** common **2,229/2,229**; homepage **20/20**.
+- **e2e**, detached and queued: `growth-chart`, `instance-colours` and `marketing`, **22 passed, 0
+ failed** (1.1 min; the new `/stats/` line test among them). The full suite was not re-run, as
+ directed.
+- **Build** (for the retaken shots, at `17832633`): 20 s, 809,752 KB, capped.
+
+## Rollout
+
+Release 14 is slice HP (merged, `bfa1ff3c`), `r14/two-grounds-headers`, slice HS
+(`r14/hidden-sites`), slice S1 (`r14/first-search`) and slice CF (`r14/chart-fold`), each after the
+parent's merge. Every command below is typed **from the primary checkout's root**. There is no `archilyzer` on PATH, so it is `pnpm archilyzer …`. The
+command forms are the ones verified in `plans/stats-cache-key.md`'s rollout.
+
+**Preconditions.**
+1. `main` carries `r14/two-grounds-headers`.
+2. **The homepage build runs release 12's source publish.** So the homepage waits on release 12's
+ rollout step 0: the denylist is complete and `pnpm archilyzer source publish --check` is clean.
+3. **No other index, stats or site build is running.** `/jobs` shows none running or queued.
+4. **If the stats-cache-key rollout has not run yet, run it first.** Its steps 5–7 build the
+ homepage, the hub and the sites in the same order as below, and then cover this release's too.
+5. **The link to keep in the header on small screens is marked, before any build.** The parent
+ marks it in the editor's Settings (**Keep in header on small screens**) and saves. The builds
+ are static: a link marked after them shows only after another build. With none marked, no
+ header shows a social link below 520 px.
+
+**The order, and why.** The homepage goes first, because every site's header links to
+`https://archilyzer.pages.dev/#instances` and that anchor exists only in the new homepage. The
+hub goes before the sites, because in basic mode they share `export/out`. The six sites go last.
+
+1. **The editor**, for its own header (no theme menu; the toggle cycles System, Light and Dark):
+ `pnpm --filter editor build`, then restart :3001 the way it is normally run.
+2. **The homepage.** Use exactly ONE of:
+ - `pnpm archilyzer build homepage && pnpm archilyzer deploy homepage`;
+ - `pnpm ops build-homepage --json '{"deploy":true}' --wait`.
+3. **The hub, before the sites.** Build it, check its log, and only then deploy it. Use exactly
+ ONE pair:
+ - `pnpm archilyzer build hub`, then `pnpm archilyzer deploy hub`;
+ - `pnpm ops build-hub --wait`, then `pnpm ops deploy-hub --wait`.
+
+ **Between the two,** the build's `compose-hub: …` line must end with `hub-summary.json covers
+ N official instance(s)`, where N is the number of public LISTED sites (a site whose
+ **List on the Archilyzer homepage and hub** is unticked is not counted). If it says
+ `hub-summary.json skipped: …`, stop and fix what it names.
+4. **The six sites:** `pnpm ops build-deploy --json '{"all":true}' --wait`.
+
+**Live checks.**
+- `https://archilyzer.pages.dev/#instances` lands on Official Instances, below the header.
+- One site at 390 px wide:
+ - the header shows the mark and the site's name, then the marked link and the theme toggle as
+ one group, then the menu button;
+ - the menu holds the nav and "Archilyzer".
+- The same site at 520 px wide and more: every social link, up to four.
+- At 1280 px wide, the header's **Archilyzer** link goes to the homepage's Official Instances, in
+ the same tab. There is no Sites menu and no Hub link.
+- **Changelog** is in the footer, after Use with AI, and not in the header.
+- **A stored Sepia renders Light.** In a site's console, run
+ `localStorage.setItem("ytdlp-tb:base","sepia")` and reload. `<html data-base>` reads `light`,
+ and `localStorage.getItem("ytdlp-tb:base")` now reads `"light"`.
+- The homepage's growth chart has no slash in the page colour through any band. `/stats` in
+ Area mode has coloured top lines.
+- The growth chart's legend lists the sites with 5 % or more of the chart's total, then
+ **Other** (a grey band on top); the caption ends "Instances under 5% of the total are drawn
+ together as Other."; **Numbers by year** has a column for every site. `/stats` still has a line
+ per site.
+- Every site's settings in the editor show **List on the Archilyzer homepage and hub**, ticked. With
+ none unticked, the homepage, the hub and every footer list the same sites as before, and
+ `https://archilyzer.pages.dev/homepage-summary.json` reads `"version":6`.
+- **S1, a clear screen until the first Search**, is in every site's build and the hub's, so steps 3
+ and 4 above (every site and the hub rebuilt and deployed) carry it. On one site:
+ - a plain visit to `/` shows the search bar, the transcript count and the whole footer, with no
+ scrolling and no listing;
+ - a `qt=` link shows its results on load;
+ - on a site that publishes tags (Anilyzer): Search for a word, then open a `tg=` link in the
+ same tab. Its listing shows with the word held in the box. After another page and Back the
+ word is still held: the box is filled, the Search button is marked, and the results say "All
+ videos", not "Matching videos".
diff --git a/plans/release-15.md b/plans/release-15.md
@@ -0,0 +1,1624 @@
+# Release 15 — storage and build hardening
+
+`main` at `99d4d76a` (release 14's T1, H1 and H2 merged). No plan file of its own: each slice's
+prompt carries its ruling, and this record carries what was built. Rules:
+`plans/tools/implementer-rules.md`, with the commit trailer this release's prompts give.
+
+**The standing choices** (not re-opened):
+- **An unmounted drive is not an empty channel,** for any build that walks the pool. The stats
+ build already holds such a channel ([`stats-cache-key.md`](stats-cache-key.md)); the index
+ build gets the same hold here.
+- **A slice that needs another slice's file stops and says so**; it does not edit it.
+- **Nothing is edited in the primary checkout**; each slice has its own worktree, and the parent
+ merges with `git merge --no-ff` only on a clean tree.
+
+## The slices
+
+| Slice | Branch | What | Owns |
+|---|---|---|---|
+| IG | `r15/index-hold` | The index build holds an unreachable channel instead of emptying it | `common/controller/buildIndex.ts` + new `buildIndex.test.ts`, `common/controller/buildStats.ts` (the hold's words move to a shared module), new `common/lib/channelMediaHold.ts`, `common/lib/envVars.ts`, `ENVIRONMENT.md`; records: `plans/{STATE,FACTS,stats-cache-key}.md` |
+| DS | `r15/drive-stall` | A stalled drive does not stop the editor answering | new `common/lib/storageHealth.ts`; `lib/{storageVolumes,channelMedia,channelMediaHold}.ts`, `controller/storageWatch.ts` and the gated callers; `/storage`, `/channels`, the videos pages; `UV_THREADPOOL_SIZE` (`editor/package.json`, `docker/entrypoint.sh`, `envVars.ts`) |
+| UT | `r15/umtool-trace` | umtool's build stops tracing the whole `umtool/` folder | per its prompt |
+| SS | `r15/site-scope` | The editor's site picker paints the stored site at once: the selection is a cookie | `editor/app/lib/activeSite{,Server,Actions}.ts` + `activeSite.test.ts`, `editor/app/components/SiteScope{Provider,Select}.tsx`, `editor/app/layout.tsx`, the scope lines of `editor/app/page.tsx` and `editor/app/channels/page.tsx`, a comment in `editor/next.config.ts`, `editor/e2e/site-scope.spec.ts`; records: `plans/FACTS.md` |
+| DT | `r15/drive-timings` | The drive-health timings are settings (`settings.storage.health`), edited on `/storage` | new `common/lib/storageHealthTimings.ts` + test; `lib/{storageHealth,storageVolumes,channelMedia,storageLocations,settingsSchema,settingsDocs}.ts`, `controller/storageWatch.ts`, the `index` and `build stats` bins, `SETTINGS.md`; `/storage` (a form, its action and parse), the stall wording on `/channels` and the videos pages; `storage-locations.spec.ts`; records: `plans/FACTS.md` |
+| SG | `r15/stagit` | The source's history and diffs on the homepage, rendered by stagit at `/source/git/` | new `common/publish/sourceHistory.ts` + test, `common/publish/__fixtures__/fakeStagit.ts`; `common/publish/source.ts` (step 12b, the key, the digest, the deploy check) + test; `common/lib/{sourceManifest,paths,envVars}.ts`, `common/lib/themeConfig.ts` (moved from `components/`, which re-exports it); `common/bin/doctor.ts`; `homepage/app/{source/page.tsx,lib/source.ts,layout.tsx}`, `homepage/e2e/{source,source-history}.spec.ts`, `homepage/e2e/fixture-source.ts`, `homepage/playwright.config.ts`; `ENVIRONMENT.md`, `PUBLISH.md`; records: `plans/FACTS.md` |
+
+**Order:** IG → DS. DS adds a health gate inside `inspectChannelMedia`, which IG's hold calls
+through its public signature. UT is independent. The shared files are `editor/CHANGELOG.md`'s
+`[Unreleased]` and this record.
+
+## Record
+
+### Slice IG, as shipped — the index build holds an unreachable channel (2026-09-29)
+
+Branch `r15/index-hold` off `main` `99d4d76a`, worktree `~/Projects/r12-paths-fix` (block #12:
+editor 4201, test 4211, export 4210), one Opus implementer. Scratch files `ig-*` in the job's
+`tmp`. The ruling: an index build meets a channel it cannot read the way the stats build already
+does. It holds the channel instead of reading it as empty, and a full rebuild with one held refuses
+unless `ARCHILYZER_INDEX_ALLOW_HELD=1`.
+
+**What was wrong.** `scanSource` read `data/` with a bare `catch { continue }`. A relocated
+channel whose drive was unmounted (a dangling `data/` link) therefore contributed no videos. The
+removal pass then dropped every record the channel had, the page writers rewrote its shared
+transcript tree empty and removed its subs tree, and the next site build published the channel as
+gone. Only the `Diff: … -R removed` line showed it.
+
+- **The hold** (`common/controller/buildIndex.ts`):
+ - `scanSource` asks `inspectChannelMedia({ channelsDir }, slug, cfg)` for every channel that is
+ not excluded and not social, before it reads `data/`.
+ - Any status but `ok` or `in-place` holds the channel, with a reason and its storage
+ location's label, and no path.
+ - It asks again after the walk, so a drive that goes away mid-walk holds the channel instead of
+ dropping the videos after that point.
+ - A `readdir(data/)` that fails holds the channel too: `its data directory could not be read
+ (<code>)`.
+ - The exception is ENOENT on an `in-place` channel: a channel with nothing downloaded (or its
+ media deleted), which is emptied as before. The log now says it: `Channel <slug>: no data/
+ directory; indexed as a channel with no videos.`
+ - A per-video metadata `stat` failing with anything but ENOENT or ENOTDIR holds the channel,
+ logged as `a video in its data directory could not be read (<code>)`.
+ - **A held channel keeps everything:**
+ - its `mtimes` records, since the removal pass skips its keys, and so its `sums`, `cues`,
+ `subs`, `digests` and `byChannel` entries;
+ - its shared transcript, subs and digest trees: not rewritten, not pruned, and not removed by
+ the top-level cleanups;
+ - its subs and digest stats, carried from their sub-DBs, so the per-site manifests still list
+ it;
+ - its availability states. The maybe-missing overlay skips it, since its `availability.json`
+ files are on the missing drive and a missing one reads as "maybe missing". The last build's
+ `videoState` entries for it are carried over.
+ - The per-site summaries are built from LMDB, so the sites built next still list its videos.
+ - **A curated-tag change while held:** the re-apply pass re-derives the held channel's records
+ in LMDB as usual, but its pages are not written. The pages-pending flag is therefore kept (and
+ logged) while any channel is held, and the first build with the drive back writes them.
+- **A full rebuild refuses** (a schema change, or a first build with no index).
+ - The scan now runs BEFORE the clear, so a refusal leaves the index untouched. `scanStartedAt`
+ is still taken at the scan's start.
+ - The message names each channel with its location's label and says why a clear would publish
+ it as gone. It gives the ways out, mounting first (the same words as the stats build's
+ refusal), then the override by name and where it is set: the command's own environment for a
+ CLI run, and the editor's own environment (which takes a restart) for its Build index job or a
+ site build started from it. Their children inherit the editor's `process.env`.
+ - With `ARCHILYZER_INDEX_ALLOW_HELD=1` (1/true/yes/on, declared in `envVars.ts`, `ENVIRONMENT.md`
+ regenerated), the build proceeds. The held channel's records go with the clear; they cannot be
+ carried across a format change. Its shared trees are left on disk, and it is out of the index
+ until its media is back and an index build runs.
+- **The words are shared:** new `common/lib/channelMediaHold.ts` (`isMediaHeld`, `HELD_REASON`,
+ `heldReason`, `describeHeld`, `HELD_WAYS_OUT`). `buildStats.ts` uses it, and its messages are
+ byte-identical (its case (i) passes unchanged).
+- **The result and the log:** `BuildIndexResult.heldChannels`. The log carries one line per held
+ channel (`Channel <slug>: <why>; its N indexed video(s) are kept as they are, not rescanned, and
+ its transcript, subtitle and digest pages are left as they are.`). The `Diff:` line ends in
+ ` Held: N channel(s), K video(s) kept.` and the `Done in` line in ` Held, their media not
+ readable: <slugs>.` Both prefixes are unchanged: the e2e helper waits for `Done`.
+
+**Who reports it** (every caller of `buildIndex`):
+
+| Caller | What it shows |
+|---|---|
+| `pnpm archilyzer index` (also the root, export and homepage `build:index` scripts) | The log on stdout, with the three lines above. It exits 0 on a hold. On the refusal it exits 1 with the message on stderr (`runIfEntryPoint`). `common/bin/build-index.ts` is unchanged: it prints the log and discards the result. |
+| A site build's data phase (`build:data`, from `buildSite`'s steps and `buildAll`'s Phase A in `common/publish/build.ts`) | The same lines, in the site build's job log. A refusal stops the steps at the data phase (`Build failed (exit 1)` / `Data phase failed (exit 1)`), so nothing is composed or deployed from an emptied index. The hub build does not run the index. |
+| The editor's **Build index** job (`buildIndexAction`, `editor/app/sites/lib/buildAction.ts`) | The lines in the job's log on `/sites` and `/jobs`. A refusal fails the job with the message. The action discards the result, and it was left unchanged: the log already carries every held channel, and `editor/app/sites/**` belongs to another slice this release. |
+| `pnpm ops build-index --wait` | Follows the job's log, so it prints the same lines. |
+| The tests | `heldChannels`. |
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `c6ae51b0` | `plans:` this record: the header, the slices, and empty Record and Rollout sections. |
+| `51328098` | `common:` the hold's words move to `lib/channelMediaHold.ts`; the stats build uses them, with its messages unchanged. |
+| `ffb01d8e` | `common:` the index build's hold, the refusal and its override, `heldChannels`, the log lines; `envVars.ts` + `ENVIRONMENT.md`; new `buildIndex.test.ts` (9 cases). |
+| `702cd0da` | `plans:` this section; FACTS "The index build's hold" (and the two stale statements in "The stats cache key" corrected); the STATE follow-up closed; `stats-cache-key.md` "Left" marked closed; the editor changelog. |
+| `9ec48351` | `common:` review L3 + L5: one unreadable video directory is logged apart from an unreadable `data/`; the refusal says where the override is set. |
+| `5af516b7` | `common(test):` review M1: case (i), the drive lost mid-walk. |
+| this commit | `plans:` the review's findings to their commits; FACTS L1, L2 and the anchors; the changelog's override sentence (L5). |
+
+**Tests** (`common/controller/buildIndex.test.ts`, the real `buildIndex` over a temp corpus). The
+drive channel is seeded with `buildStats.test.ts`'s `seedDriveChannel` shape and unmounted by
+renaming its media root away. Every path is pinned under a temp root, and case (z) spies on
+node:fs writes.
+
+| Case | What it pins | On the pre-change `buildIndex.ts` |
+|---|---|---|
+| (a) | Unmounted, incremental build with another change: records kept, the drive's transcript and subs trees byte-identical (manifest included), the site still lists both videos and its subs count; the log lines, with the label and no path | removed 2, not 0 |
+| (b) | Availability carried: `maybe_missing` stays and a post-scan confirmation stays `available`; `videoState` unchanged | d1 no longer published |
+| (c) | A full rebuild refuses (message, ways out, the variable, no path); the index and pages untouched; the CLI exits non-zero; with the override it holds (records cleared, pages kept); the drive back re-adds both | no refusal |
+| (d) | A first build (no index) with a channel held refuses too | no refusal |
+| (e) | The drive back: held set empty, a video added to the drive meanwhile indexed, 0 changed | removed 2 while away |
+| (f) | A really empty in-place channel (an empty `data/`, and no `data/`) is emptied, not held; the missing `data/` is logged | the new log line only (the emptying matched, as it should) |
+| (g) | An unreadable `data/`, and an unreadable video dir mid-walk (mode 000), hold their channels with `EACCES` | removed 3 |
+| (h) | A tag rule added while held: the held pages untouched, the flag kept; the drive back writes the tag to them and clears the flag | the drive's pages emptied |
+| (i) | The drive lost MID-WALK: `node:fs/promises` `stat` unmounts it right after the walk's first drive metadata stat, so every later stat in the channel is ENOENT. The second look holds the channel: 0 removed, records and pages unchanged, the log line | removed 3 of 4; the same with only the second look deleted from the new code |
+| (z) | No write outside the temp root | passes on both |
+
+The pre-change column was run with the old `buildIndex.ts` swapped in once, with the
+`heldChannels` assertions removed so each case reached its first substantive assertion.
+
+#### Gates (at `ffb01d8e`, and after the review at `5af516b7`; logs `$T/ig-*.log`)
+
+- **tsc** was clean before every commit: 78 s at the branch point, 48 s at `ffb01d8e`, 50 s at
+ `5af516b7`.
+- **Unit:**
+
+ | Suite | Result |
+ |---|---|
+ | common | 2,219/2,219 at `ffb01d8e` (the branch point's 2,210 plus the 9 new cases), 77 s; **2,220/2,220** at `5af516b7` (case (i) added), 78 s |
+ | editor unit | 87/87 |
+ | `test:scripts` | 191 passed, 1 skipped (192) |
+ | mcp | 271/271 |
+
+- **Docs:** `docs env --check`, `docs files --check` and `settings example --check` all exit **0**,
+ at both points.
+- **Build:** the editor's `next build`, with the primary's `transcripts/` linked in and capped at
+ 5 GB with no swap: 64 s, max RSS 1,642 MB. The link was removed after the build, and nothing
+ ran through it.
+- **e2e** (editor, detached and queued; the spec list is every spec that runs Build index:
+ `availability`, `build`, `channel-build-toggle`, `chat-only`, `duplicate-shorts`, `jobs`,
+ `regional-vtt-fallback`, `tags`, passed as `e2e/<name>.spec.ts` so `availability` does not also
+ match `pre-clean-availability`): **35 passed, 0 failed, 3.6 min**, after 1 min 45 s in the
+ queue. No e2e fixture has an unreachable channel, so these confirm the hold changes nothing for a
+ readable corpus. Not rerun after the review: its two log-wording changes touch no spec (`git grep
+ "could not be read" editor/e2e` finds only the curated-tag preview and the title filter).
+- **Numbers tool:** none.
+
+#### Found and left
+
+- **A drive that drops during the processing phase** (after the scan) is not covered, for a changed
+ video whose metadata was read before the drop. Its transcript read is caught as "no cues", so its
+ cues are removed. A failed sub-track read is skipped, and with none left its subs are removed. A
+ failed digest load leaves no digest, which is removed. `mtimes` is then written with the video's
+ current mtimes, so the loss lasts until any of its tracked mtimes (metadata, transcript, subs,
+ availability, digest) moves. These are per-video reads inside the worker, left to the slice that
+ handles drive stalls.
+- **A drive that drops and comes back inside one walk** is not covered either: the second look
+ finds it, and the videos skipped in between are removed.
+- **`inspectChannelMedia` is asked twice per channel** (before and after the walk), three syscalls
+ each today. Slice DS puts a health gate inside it, and that gate runs twice per channel per
+ index build.
+- **Under the override,** a held channel is listed on its sites with 0 videos, and its subs and
+ digest counts are left out of the site manifests. Its old shared page trees stay on disk, and a
+ compose copies them.
+- **While a channel is held after a tag change,** the pages-pending flag stays set. Every build
+ until the drive is back is then a full page walk; it skips unchanged pages by hash, so it rewrites
+ none.
+- **The digest tree's retention has no test.** The code path mirrors the subs tree's, and no
+ fixture carries a digest sidecar.
+- **The editor shows a hold only in the job log.** A `/sites` badge would be in `editor/app/sites/**`.
+
+#### Decisions the operator could overturn
+
+| What I assumed | The alternative |
+|---|---|
+| A held channel's shared pages are not written at all. **Ruled at review: keep the skip.** | Rewrite them from the kept records: byte-identical on an incremental build, and a tag change would reach them at once. After an override rebuild the records are gone, and a rewrite would publish the channel empty. |
+| Under the override, the held channel's records go with the clear. | Carry them across the clear. A schema change means the stored format moved, so the old records cannot be trusted. |
+| An unreadable `data/`, or a per-video `stat` failing with anything but ENOENT, holds the channel. | Hold only for the statuses `inspectChannelMedia` reports, and keep treating other read errors as "no videos", as the old code did. |
+| A channel with no `data/` is logged, one line per build. | Stay silent, as before; the ruling asked for a log. |
+| The CLI exits 0 on a hold, since the hold is the safe outcome; the refusal exits 1. **Ruled at review: both stay.** | Exit non-zero so scripts notice; a site build's data phase would then fail whenever a drive is out. |
+| The words live in a new `lib/channelMediaHold.ts` shared with the stats build. | Duplicate them in `buildIndex.ts` and leave `buildStats.ts` untouched. |
+
+#### Review
+
+**Verdict: SHIP AFTER FIXES** (`ig-review.md` in the job's scratch). No High. Every write path a
+held channel could reach was traced, and the refusal fires before anything is deleted.
+
+| Finding | Where |
+|---|---|
+| M1: the second look after the walk had no test | `5af516b7`: case (i). It fails with 3 of 4 removed when that look is deleted. |
+| L1: FACTS said an unreachable drive reaches `notIndexable` only through the override | this commit: a video that reached the drive after the last index build that could read it is counted there by a stats-only run between the drive's return and the next index build, which heals it. |
+| L2: the processing-phase gap also loses subs and digests, until any tracked mtime moves | this commit, in "Found and left" and FACTS. |
+| L3: one unreadable video dir was logged as the whole data directory | `9ec48351`: `a video in its data directory could not be read (<code>)`; case (g) expects it. |
+| L4 (optional): a narrower `HELD_REASON` type | **Left**, as the review allowed. The current contract already fails tsc on a new status. |
+| L5: the refusal did not say where the override is set | `9ec48351` (message, case (c)) and this commit (changelog). |
+| L6: check the held set before routine builds resume | A rollout note; the parent records it. |
+| L7: stale trees under the override | Already in "Found and left". |
+| Q2: the commit trailer | Ruled correct. |
+
+**What runs which code, for the rollout.** Every CLI command and every spawned data phase runs the
+checkout's code, so they hold from the moment `main` has this branch. The editor's in-process
+**Build index** button runs its built bundle, so it holds only after the editor is rebuilt and
+restarted.
+
+### Slice UT, as shipped — umtool's build stops tracing its dot-directories (2026-09-29)
+
+Branch `r15/umtool-trace` off `main` `ccf90892`, worktree `~/Projects/r12-source-mirror` (block
+#13: editor 4301, test 4311, export 4310), one Opus implementer. Scratch files `ut-*` in the job's
+`tmp`. The ruling: find the one expression that widens the clip-audio route's trace and fix it
+there; exclude the fixture, the e2e build and env files as a second line; narrow or drop the
+`ignoreIssue`; make the trace guard read a build back and close the release 14 review's L1 and L3.
+
+**What was wrong.** With the primary's e2e fixture in place, umtool's
+`app/api/clip/[key]/audio/route.js.nft.json` listed 2,167 files: the 463 its sibling routes list,
+178 under `.e2e-song/` (the fixture, where `make-fixture.mjs` links the song data), 1,525 under
+`.next-e2e/` (the e2e dev server's build directory, 1.1 GB) and `.env.local`. Turbopack's warning
+for it ("Encountered unexpected file in NFT list", the "whole project was traced" text) was silenced
+by the config's `ignoreIssue`. The traces are not consumed while `output: "standalone"` stays off,
+so nothing broke; a fixture with more in it, or a standalone build, would have carried it.
+
+**The bisect.** One change per build, in this worktree with four probe files planted in
+`.e2e-song/probe/` and `.next-e2e/probe/`; the audio route's trace, total / under dot-directories.
+The route as on `main`: **467 / 4**.
+
+| Change (line on `main`) | Entries |
+|---|---|
+| `existsSync(file)` :51 stubbed | 467 / 4 |
+| **the join `path.join(CACHE_DIR, `${stamp}.${asMp3 ? "mp3" : "wav"}`)` :58 written as a string concatenation** | **463 / 0** |
+| `existsSync(cached)` :61, `readFile(cached)` :62, `mkdir(CACHE_DIR)` :78, `readFile(tmpMp3)` :89, `rename(tmpMp3, cached)` :90, `writeFile(tmp)` :93 or `rename(tmp, cached)` :94 stubbed, each alone | 467 / 4 each |
+| the `tmpWav` / `tmpMp3` joins :82-83 as concatenations; `writeFile(tmpWav)` :84 stubbed; both `writeFile`s stubbed; either `writeFile` opted out | 467 / 4 each |
+| the opt-out on the :58 join | 467 / 4 |
+| the :58 ternary hoisted into a `const ext` | 467 / 4 |
+| **:58 without the ternary** (`${stamp}.wav`) | **463 / 0** |
+| :58 as `path.join(CACHE_DIR, asMp3 ? `${stamp}.mp3` : `${stamp}.wav`)` | 463 / 0 |
+| :58 as a ternary of two joins | 463 / 0 |
+| opt-outs on `existsSync(cached)` and `readFile(cached)`, or either alone; both stubbed | 467 / 4 each |
+| all five readers and writers of `cached` stubbed | 467 / 4 |
+| **all five stubbed, and the opt-out on the :58 join** | **463 / 0** |
+| all five stubbed, and the join as a concatenation | 463 / 0 |
+
+With the primary's fixture (the table's 4 are 1,704 there), on top of the last-but-one row:
+
+| Change | Entries |
+|---|---|
+| one `existsSync(path.join(/* opt-out */ CACHE_DIR, …))` | 2,167 / 1,704 |
+| the same `existsSync` opted out as well | 463 / 0 |
+| `existsSync(/* opt-out */ cached)` as the only reader of the opted-out join | 463 / 0 |
+
+**The expression** is the :58 join, and in it the ternary inside the template literal. The join and
+the fs calls on its value each trace the pattern (the join alone with every reader stubbed; the
+readers alone with the join opted out; `existsSync` is one such reader), so no single opt-out or
+stub cleared it. Without the ternary, the pattern stays out of the dot-directories.
+
+**The fix** (`a305b956`). `umtool/lib/paths.mjs` gains `cacheFile(name)`, `path.join(/* opt-out */
+CACHE_DIR, name)`, re-exported by `lib/paths.ts`. The audio route names all four of its cache files
+through it (`cached`, `tmpWav`, `tmpMp3`, and `tmp`, now `cacheFile(`${stamp}.wav.tmp`)`, the same
+path as `${cached}.tmp` in that branch). A value returned by a function from another module is
+opaque to the tracer, so the call site traces nothing: the route lists 463, as its siblings do. The
+video and face-frame routes join `CACHE_DIR` with a fixed extension; they were measured clean (the
+video route's `.mp4` join did not reach the `.mp4` in `.e2e-song`) and name their cache files the
+same way, so no route joins `CACHE_DIR` itself. The run-time paths are unchanged.
+
+**The second line** (`f18fa034`). `outputFileTracingExcludes: { "/*": ["./.e2e-song/**/*",
+"./.next-e2e/**/*", "./.env*"] }` (Next 16.2.3's `05-config/01-next-config-js/output.md`: route
+globs to globs from the project root; Turbopack reads the key natively, `collect-build-traces.js`
+is the webpack path). Measured alone, with `main`'s route: 2,167 → 463.
+
+**The warning stays silenced, narrow as it was (path + title), with its measured reason.** Dropping
+it shows one warning on every build, and after the fix it is still true:
+- 66 of the 68 routes trace umtool's whole tree outside dot-directories: its 361 files,
+ `next.config.ts` (the file the warning names) among them. Only `_global-error` and `_not-found`
+ do not.
+- Path and fs calls on env, home-directory and parameter values do it. Opting out every path op in
+ `song/paths.mjs`, `lib/paths.mjs` and `lib/paths.ts` left 31 of the 68 routes clean (the audio
+ route 463 → 102). Opting out all 319 path ops in the 53 modules that have one left 49 clean. The
+ two clean before are among them. The rest come through fs calls; the next import trace the
+ warning names is `lib/report/snapshots.mjs`.
+- That walk skips dot-directories and does not enter symlinks. The warning names the same file for
+ it as for the audio route's, so it cannot tell the two apart; the second line and the guard below
+ cover the dot-directories instead. The config's comment says all of this.
+
+**A symlinked directory is not entered by these patterns.** The planted link
+`umtool/.e2e-song/data/planted` → `<primary>/transcripts/channels` gave 0 entries before and after
+the fix, and an in-root link to `common/` (675 files) gave 0 on `main`'s route. What the pattern
+reached was the fixture's real files.
+
+**The guard** (`scripts/next-build-trace.test.mjs`, `9a375ecd`, and after the review `a4d100b4`,
+`c3e8a2c7`), 6 → 10 tests:
+- **(a) It reads umtool's last build back.** Every `.nft.json` under `umtool/.next` but the
+ build's own `cache/` and `dev/` fails on an entry outside the repo, under `transcripts/`, or
+ through any name starting with a dot but the build's own directory and `node_modules/.pnpm`.
+ Unit tests pin `forbiddenTrace` and which directories are read.
+ - **It skips, saying so,** with no build, and (review M1) when `umtool/.next/BUILD_ID` is older
+ than `umtool/next.config.ts` or any umtool module the guard scans. The message gives the
+ build's time, the first newer file and how to rebuild. A merge or checkout gives the changed
+ files new mtimes, and umtool runs under `next dev`, which does not refresh `.next`, so a stale
+ build skips instead of failing on a call already fixed.
+ - **What it can see** (review L2). Without the excludes, `main`'s route failed it with 1,704
+ entries (the first 20 listed); the primary's pre-fix build fails it with 1,705 (the review:
+ `test-results/.last-run.json` too). With the excludes in place, a pattern like that one shows
+ only through a name they miss: `test-results/.last-run.json` after an e2e run, `.next-shots`,
+ the corpus, a path outside the repo. A checkout with no e2e run behind it is blind to it; the
+ fix at the call is what keeps the route clean.
+- **(b) The scan set follows relative imports** out of the listed folders, to any depth. It adds
+ the review's L1 modules and no others: `common/bin/_publicFile.ts`, `homepage/content/docs.ts`
+ (895 → 897) and seven `umtool/song` modules, `reasons`, `archive-url`, `pitch`, `flatness`,
+ `clipwindow`, `deplosive`, `orderfeat` (211 → 218; `song/paths.mjs` was listed by hand before and
+ is now reached). A test pins them, and that a song CLI and a common CLI stay out. The comments
+ that called them CLI-only are gone.
+- **(c) The checked calls** add `open`, `writeFile`, `appendFile`, `createWriteStream` and their
+ Sync forms, and the `fs.promises.` / `fsPromises.` prefixes. No new finding.
+- **(d)** `common/lib/paths.ts` `under(first, ...rest)` (`8b3409c2`): the opt-out sits before a
+ named first argument. `getPaths()` hashed identical, old module against new, from the repo root,
+ `editor/` and a directory outside the repo (56 keys). The guard passes; every caller type-checks.
+- The header no longer says the env and home directory are unfollowed or that the opt-out is
+ documented. The nested-call exemption's comment says it is a simplification (below).
+
+**FACTS**, "A path joined from `process.cwd()` …", corrected:
+- "documented": the Next docs list `turbopackIgnore` only for `import()`, `require()`,
+ `require.resolve()` and `new Worker()`. The path form is Turbopack's own advice, in the warning's
+ text in the 16.2.3 binary, and Next's own server uses it on the join and on the fs call around
+ it (`next/dist/server/next-server.js:620`; review L4).
+- "an outer fs call on an opted-out `path.join` is covered": it is not (the second bisect table).
+- "Not followed by the tracer: `os.homedir()` and `process.env.*`": such values are dynamic parts,
+ and make patterns over the app directory. The synthetic-HOME build showed that a pattern walk
+ does not enter symlinks.
+- The guard's entry, the worktree caveat (no fixture either) and `under()`'s shape.
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `a305b956` | `umtool:` `cacheFile`; the audio, video and face-frame routes name their cache files through it |
+| `f18fa034` | `umtool:` `outputFileTracingExcludes`; the `ignoreIssue` kept, its comment the measured reason |
+| `8b3409c2` | `common:` `under(first, ...rest)` |
+| `9a375ecd` | `scripts:` the post-build check, the relative-import scan set, the opens and writes, the header |
+| `750fc850` | `plans:` this section; FACTS; the editor changelog |
+| `a4d100b4` | `scripts:` review M1: the post-build check skips a build older than the code it judges |
+| `c3e8a2c7` | `scripts:` review L1 (only the build's own `cache/` and `dev/` unread, pinned), L2 (the check's comment says what it can see), L3 |
+| `b4d6607d` | `umtool:` review L3 in the `ignoreIssue` comment |
+| `31bb0f8c` | `plans:` the review's findings to their commits; FACTS (M1, L2, L3, L4); the changelog (M1); the rollout note |
+| `7c6d4969` | merge of `main` (`721ed0eb`, release 14 HS and S1); clean, the changelog bullet still under `[Unreleased]` |
+| this commit | `plans:` the gates after the review and the merge |
+
+#### Gates (logs `$T/ut-*.log`)
+
+- **tsc** clean over the combined tree before the first commit, 241 s (load average ~26). The
+ commits are independent pieces of that tree; since then only a comment changed in a
+ type-checked file (`cacheFile`'s, in `umtool/lib/paths.mjs`).
+- **`test:scripts`:** 194 passed, 1 skipped (195): `main`'s 191 + 1 and the guard's three new
+ tests. Before the final build it failed exactly the post-build test, on a build of `main`'s route.
+- **common:** 2,220/2,220, 107 s.
+- **umtool's build, capped at 5 GB with no swap, with the primary's `transcripts/` linked in, the
+ primary's `.e2e-song` and `.next-e2e` hard-linked in, an empty `.env.local`, and the planted
+ link** (all removed afterwards; none committed):
+
+ | Tree | Wall | User | Max RSS | Audio route | `.e2e-song` | `.next-e2e` | `.env*` | `transcripts` / planted |
+ |---|---|---|---|---|---|---|---|---|
+ | `main`'s route and config | 23 s | 57 s | 809 MB | 2,167 | 178 | 1,525 | 1 | 0 / 0 |
+ | the branch | 34 s | 64 s | 765 MB | 463 | 0 | 0 | 0 | 0 / 0 |
+
+ The wall times swing with the machine's load (another slice's e2e and builds ran alongside);
+ compile was 7.5 s and 10.7 s, TypeScript 12 s and 19 s.
+- **The editor's build**, capped, with the corpus linked in (`paths.ts` changed): 97 s wall, 159 s
+ user, max RSS 1,576 MB (IG's: 64 s / 1,642 MB, under less load); 0 of its 81 traces' entries
+ under `transcripts/`, and none that `forbiddenTrace` refuses.
+- **e2e** (umtool's own filter, `SONG_DIR=~/reports/quartering-uh-song/data`, queued):
+ `find.spec.ts` and `triage.spec.ts` fetch the audio route, `faces.spec.ts` the face-frame route,
+ and `find.spec.ts` names the video route: **6 passed, 29 skipped, 0 failed, 27 s** (6.3 min with
+ the queue). The skips are the fixture's: this machine has no `wav48/`, `asr/` or `media/`, so
+ every spec that fetches one of the three routes skipped, and they are not exercised at run time
+ here. What stands for that: calling `cacheFile` gives the same path as the old expression for
+ all six names (the four audio names, and the video and frame temporaries).
+- **After the e2e run** (which built this worktree its own fixture and `.next-e2e`), a last capped
+ build with the corpus linked: the audio route 463, none under a dot-directory; `test:scripts`
+ 194 passed, 1 skipped. The first run of that `test:scripts` failed `queue-lock.test.mjs`'s FIFO
+ case once (`S1E1S3E3S2E2`) under a load average of about 26; it passed on the rerun, and this
+ slice does not touch the queue lock.
+- **Numbers tool:** none.
+- **After the review and the merge of `main`** (at `7c6d4969`):
+ - tsc clean, 98 s;
+ - common **2,229/2,229** (`main`'s 2,229), 108 s;
+ - `test:scripts` **195 passed, 1 skipped (196)**: `main`'s 191 + 1 and the guard's four new
+ tests. The two runs before it each failed `queue-lock.test.mjs`'s "prints a banner naming the
+ holder while waiting" under a load average of about 26, the known flake (alone, 11/11 twice);
+ this slice does not touch the queue lock;
+ - umtool's build, capped, with the corpus linked and this worktree's own e2e fixture present:
+ 48 s wall, max RSS 796 MB, the audio route 463, none under a dot-directory; the post-build
+ check passes on it;
+ - the same build with its `BUILD_ID` set back to 2026-09-01 (in place, then put back; nothing
+ committed): the check skips, `umtool/.next was built 2026-09-01T04:00:00.000Z, before
+ umtool/next.config.ts (219 changed since); rebuild umtool (…) to check its traces`. With the
+ mtime put back it passes again.
+
+#### Found and left
+
+- **The whole-folder trace in 66 routes** (above). Bounded to umtool's own files; cleaning it means
+ opt-outs on hundreds of path and fs calls on unknown values, which no static check can find.
+- **The guard's nested-call exemption.** Without it, four calls would need an outer opt-out:
+ `common/lib/paths.ts:311` (`existsSync` of one file), `export/app/changelog/page.tsx:9` and
+ `homepage/app/changelog/page.tsx:24` (one file each; other slices own them), and
+ `umtool/report-to-video/brand.mjs:82` (the brand kits, which a standalone build needs). Each
+ traces the file or files it reads. Left, and the comment says so.
+- **Which fs calls Turbopack traces is not established per call.** `existsSync` does (the second
+ table); the writes were added to the guard without a measurement, since an extra opt-out costs
+ nothing.
+- **The post-build check reads umtool only.** In this worktree the editor's 81 traces pass the same
+ rule. The review's Info saw `editor/.env` in the primary's; a worktree has none, so it was not
+ re-measured.
+- **The gate command in `implementer-rules.md`, in a worktree that has a `transcripts/` directory,**
+ makes `transcripts/transcripts` and builds without the corpus where the paths point. This worktree
+ had one (an `index.mdb` from 2026-09-28), and my first two corpus-linked builds ran like that. The
+ numbers above are from builds that set it aside and put it back. `ln -sT` would refuse instead.
+- **A checkout whose umtool build predates its code skips the post-build check** until umtool is
+ rebuilt (review M1). The primary's `umtool/.next` is from before this slice, so the check skips
+ there until the rollout rebuilds it.
+
+#### Decisions the operator could overturn
+
+| What I did | The alternative |
+|---|---|
+| A `cacheFile` helper in `lib/paths.mjs`, used by all three routes that join `CACHE_DIR`. **Ruled at review: keep; one way to name a cache file.** | Opt-outs on the audio route's join and on every fs call on its value, in that route only |
+| The `ignoreIssue` stays, narrow, with the measured reason in its comment | Drop it: one warning on every build, naming one route's import trace |
+| The post-build check refuses any dot-named path but the build's own and `node_modules/.pnpm` | Refuse only `.e2e-song`, `.next-e2e`, `.env*` and `.git` |
+| The post-build check covers umtool only. **Ruled at review: umtool only.** In the primary the editor's traces would fail it (75 of 81, `editor/.env` among them) and the export's `.export-index` entries would be misjudged. | Also read the editor's, the export's and the homepage's builds |
+| The static check keeps its nested-call exemption, documented as a simplification. **Ruled at review: it stays; the four sites are safe.** | Require the outer opt-out: four new findings, two in files other slices own |
+| A stale build skips the post-build check (review M1) | Fail on it, as first shipped |
+
+#### Review
+
+**Verdict: SHIP AFTER FIXES** (`ut-review.md` in the job's scratch). No High. The reviewer found
+the fix's nine call-site paths byte-identical, the bisect logs in agreement with the tables, the
+excludes' key and shape right, and 0 dot entries across all 70 traces of a fresh build.
+
+| Finding | Where |
+|---|---|
+| M1: a stale umtool build turns `test:scripts` red, pointing at a call already fixed | `a4d100b4`: the check skips a build older than `next.config.ts` or any module it scans; the changelog, FACTS and "Found and left" say so; the rollout note below |
+| L1: `cache`/`dev` skipped at any depth | `c3e8a2c7`: only directly under `.next`; a test with a route directory named each |
+| L2: with the excludes in place the check cannot see the original defect | `c3e8a2c7` (the test's comment), guard (a) above and FACTS: it sees only a name the excludes miss |
+| L3: "cleaned 31 / 49" | `c3e8a2c7`, `b4d6607d`, this commit: "left 31 / 49 of the 68 clean" |
+| L4: FACTS cited the native binary's string as Next's runtime | this commit: `next-server.js:620` |
+| L5: the build-gate command's `ln -s` | The parent's (`implementer-rules.md`) |
+| Info: the editor's and export's primary builds carry the same class of widening | Recorded in the decisions table; a later slice's |
+
+**For the rollout.** Rebuild umtool in the primary first, under the cap
+(`timeout -s KILL 240 systemd-run --user --scope -q -p MemoryMax=5G -p MemorySwapMax=0 pnpm
+--filter umtool exec next build`), then run `pnpm run test:scripts` there. That is the only proof
+on the real fixture, which carries `.env.local`, `.next-shots` and `test-results/.last-run.json`,
+two of them names the excludes do not cover. Until that rebuild the post-build check skips in the
+primary, saying why. umtool's code changes nothing at run time.
+
+### Slice DS, as shipped — a stalled drive does not stop the editor answering (2026-09-29)
+
+Branch `r15/drive-stall` off `main` `ccf90892` (slice IG merged), worktree `~/Projects/r12-paths-fix`
+(block #12: editor 4201, test 4211, export 4210), one Opus implementer. Scratch files `ds-*` in the
+job's `tmp`. The ruling: a drive that is mounted and not answering must not stop the editor
+answering. Two detectors find the stall without the editor waiting on the drive, and pages and polls
+do not touch it in-process while it is not answering. The parent's five rulings on the first pass
+(Q1–Q5) and the review's fixes (M1–M3, L1–L10) are applied; the tables at the end map each to its
+commit.
+
+**What was wrong.** Node runs every filesystem call on libuv's thread pool, four threads by default.
+On a drive that has stalled (an SMR disk in a USB enclosure resetting under a long write) each call
+blocks its thread for about 30 s. The home page, `/channels` and the three-second auto-queue status
+poll each `stat`ted every relocated channel's target in-process and uncached; the one-second job-list
+poll walked the whole `data/` of every channel with a job listed, 64 calls wide; and the recency layer
+read metadata tails 32 at a time. Four blocked calls were enough for no page, poll or job log to
+answer. The storage watch saw nothing of it: its five-minute pass asks "is the disk here", and a
+stalled disk is here.
+
+- **The health state** (new `common/lib/storageHealth.ts`): one map per process on `globalThis`
+ (`__yttStorageHealth__`, the house pattern: the watch writes it from instrumentation's module copy
+ and pages read it from theirs), by location id: state `ok | stalled | absent`, `since`, the last
+ check, a clean streak, the cause, and the `detector` that gave the last verdict. In memory only.
+ - One `stalled` answer marks the location stalled at once. Two clean answers in a row clear it (an
+ `absent` answer is clean). A miss in between starts the count again. A re-pointed root starts the
+ location over. Locations no longer configured are pruned; every configured one is registered
+ (`registerLocationHealth`) before a pass asks anything, so the watchdog can find a channel's
+ location before the first verdict.
+ - Pure of I/O and without execa, so `lib/channelMedia.ts` can ask it.
+- **Detector 1, every 15 s: the block device's own counters** (`detectLocationHealth`,
+ `common/lib/storageVolumes.ts`; ruling Q1(a)). A child `stat` of the root is answered from the
+ kernel's inode cache whenever the drive was used lately, so it can say "ok" while the reads that
+ reach the device wait out a reset loop. Instead, per location per pass:
+ - the root's device: `findmnt -J -T <root> -o SOURCE,UUID` as a child raced against 3 s, a
+ `[subvolume]` suffix taken off, `/dev/mapper/*` resolved to its `dm-N` (a read of `/dev`), the
+ basename. Asked only when the root has no device yet or its device's `/sys` entry stops reading
+ (a replug under another name): a findmnt per pass would leave one child stuck per pass during a
+ long stall (L10). A UUID other than the location's recorded one names no device (the root is
+ then a directory on another filesystem, not the drive).
+ - `/sys/class/block/<dev>/stat`, which never touches the drive: completed = reads (field 1) +
+ writes (5) + discards (12) + flushes (16) where the kernel counts them (in_flight counts those
+ too, so a long SMR media-cache flush alone moves completions; L2), and requests in flight (9).
+ Against the previous sample for the location (same device, at least `MIN_COUNTER_INTERVAL_MS` =
+ 10 s earlier): **stalled ⇔ in flight at both AND nothing completed between**; anything else is
+ clean. The first sample gives no verdict. The samples are on `globalThis`
+ (`__yttHealthDetector__`), so the pass and `/storage`'s Refresh, in different module copies,
+ compare against one previous sample (L5).
+ - **No device** (a container, no findmnt, a tmpfs or network source, no `/sys` entry) falls back to
+ the child `stat` probe (`probeLocationHealth`): `stat -L -c %F -- <root>` raced against 3 s, the
+ child SIGKILLed and not waited for; `directory` is `ok`, anything else `absent`, no binary `ok`.
+ - The verdict names its detector (`"counters" | "stat"`), the health state records it (and the
+ counters' device, which the watchdog reads), and `/storage`'s line says which watched the drive.
+- **Detector 2, on every gated call: a 3 s watchdog** (`onDrive(where, call)`,
+ `lib/storageHealth.ts`; ruling Q1(b)). The detector that cannot be fooled by a cache: a page or poll
+ that actually reaches the drive finds out.
+ - Refused at once, with no call, when the location is stalled.
+ - Otherwise raced against `DRIVE_CALL_BUDGET_MS` (3 s). A call that has not answered marks its
+ location stalled (since now, cause "a read in the editor did not answer within 3 s") and throws
+ `DriveNotAnsweringError`; the caller answers `stalled`. The call is left to settle on its own:
+ its thread is the stated limit.
+ - **Slow is not stalled** (L6): on a timeout, when the counters detector has named the location's
+ device, its counters are read (synchronously, from `/sys`, so the check does not wait behind the
+ pool it is judging) and compared with a reading taken when the call began. Requests completed
+ meanwhile: the drive is slow; the call is refused and nothing is marked.
+ - **The budget covers a whole unit of work** (L8): a video directory's reads, a page's reads of
+ one video, go through as one call, so a slow drive still answering can be marked by one long
+ unit (unless the counters show it completing, above).
+ - **At most `DRIVE_CALLS_IN_FLIGHT` (4) calls per slot key are in flight.** The rest wait in a
+ queue of its own (not libuv's), so a 64-wide walk that meets a stall puts four calls on the
+ drive, not 64. A slot is released when its call really returns. The queue (M1):
+ - every transition to `stalled` refuses the waiting calls at once, whoever decided it (the
+ pass, the watchdog, a Refresh);
+ - a wait's deadline follows progress (re-review M4): every call that returns on the key, in
+ time or late, restarts the deadline of every call waiting on it, and a waiting call is
+ refused, without marking, only when nothing on the key has returned for the budget plus a
+ grace of a quarter of it (at most 250 ms; the grace lets the calls it waits behind, whose
+ timers start a moment later, time out and mark first). A deep queue on a drive that is busy
+ but answering therefore waits as long as it takes;
+ - calls past their budget are counted per key (`overdue`), each with the counters reading
+ taken when it began; when every slot is held by one, a new call is refused at once, and the
+ location is marked stalled again (even if the pass has since cleared it) unless the disk has
+ completed requests since the oldest of them began: then it is slow, not stalled, and nothing
+ is marked (the same test as a timeout's).
+ - **The slot key** (M2, L7): a configured location's id; a probe of another root under a
+ location's id is keyed by that root and marks nothing; a path on no configured location (a root
+ typed by hand) is keyed by the root it is under (`<root>/<slug>/data` → `<root>`), so it holds
+ at most four threads too, and nothing can mark it. Calls are not nested for one key. The timer
+ is not `unref`'d: it is cleared the moment the call answers.
+- **The cadence** (`common/controller/storageWatch.ts`): `startStorageHealthWatch` arms the 15 s
+ health pass and runs one at once; **`editor/instrumentation.ts` arms it above the idle gate**,
+ beside the storage boot probe (M2): it is in memory and writes nothing, and without it an idle
+ boot has no registered locations and a stall the watchdog marks is never cleared. The five-minute
+ pass (`startStorageWatch`), which may write an auto-pause, stays below the gate. All locations are
+ asked concurrently, each bounded by its own timers, and an overrunning pass is not stacked. A
+ transition is logged (`[storage] "<id>": drive not answering — <cause>; …` / `answering again
+ (ok)`). A CLI process has no pass, but its gated calls still go through the watchdog.
+- **The gate and the watchdog, by caller:**
+
+ | Caller | On a stalled location | Through `onDrive` |
+ |---|---|---|
+ | `inspectChannelMedia` (`lib/channelMedia.ts`) | **The relocation marker is read first** (ruling Q2: it is in the channel dir, on the corpus disk), so a channel mid-move on a stalled drive reads `in-transition`. Then the gate: status `stalled`, detail `drive not answering (location "<label>", since HH:MM)`, before the link and the target. With a config in hand a stalled channel costs one call, the marker read. | The target's `stat` (`:445`). |
+ | `assertChannelMediaReachable` | Refuses it (`ChannelMediaUnreachableError`, status `stalled`), so `runManagedFunction`'s `needsMedia` guard, `generateChannelSnapshot` (also the channel page's **Refresh report**), the operation batch, normalise, keep-videos and the shard action refuse it. | Through inspect. |
+ | The index and stats builds | Hold it: `HELD_REASON.stalled` is "its drive is not answering (a stalled disk)", and IG's `isMediaHeld` holds every status but `ok` and `in-place`. | Through inspect (both of the index build's looks). |
+ | The storage watch's five-minute pass | Counts a stalled location, or a channel whose inspect says `stalled` on no location (L9), as down: two passes auto-pause its channels, and the pass after the drive answers restores them, as for an unmount (ruling Q3). The pause record carries `cause: "not-answering"`, and `autoPauseReasonOf` says "on a drive that is not answering … when the drive answers again" (L4; a record without a cause, all written before, reads as not there). | Through inspect and `probeLocation`. |
+ | `probeLocation` and `probeLocationMemo` (`lib/storageVolumes.ts`) | Probe status `stalled` (`STORAGE_STATUS_LABEL`: "Not answering"), identity unknown, no free space; no `stat`, `statfs` or `findmnt`. The memo is asked after the gate. | The root's `stat` and `statfs` (`:262`, `:279`). |
+ | `volumeFreeBytes` (`controller/storageLocations.ts`) | Unknown ("—"), with no call. | The root's and the mountpoint's `stat`, and the `statfs` (`:288`, `:325`, `:334`). |
+ | `generateChannelSnapshot` (`controller/channelSnapshot.ts`), after every download or sync, sixteen video directories wide (M3) | Refused by its start guard. | Its `data/` listing (a refusal is rethrown, never read as an empty channel), the keep-latest keys' metadata reads (`keyedVideosNewestFirst`, now `mapConcurrent` 16 wide instead of an unbounded `Promise.all`) and each video directory's unit (`:751`, `:844`). A throw keeps the last `snapshot.json`, as on any failed refresh. The sequential reconcile pass before it is not raced. |
+ | `readChannelStat` (`controller/channels.ts`) | `null` (no counts, so no progress bar), with no walk. The one-second job-list poll and the home page ask it for every channel with a job listed. | The `data/` readdir, then each video directory as one call (`:107`); `null` when the drive stops answering mid-walk. The batch jobs' `listChannelStatsFromDisk` passes no drive and is unchanged. |
+ | The recency tail reads (`controller/recencyIndex.ts`) | Skipped, and NOT remembered as misses: layers 3 and 4 until a refresh with the drive answering reads them. | Each tail read of a relocated channel (`:257`). |
+ | `relocationRootPresenceProblem` (`controller/relocateChannelMedia.ts`) | A move onto a stalled location is refused before the root's `stat`. | The root's `stat` (`:307`). |
+ | `inspectSavedVideosStore` (`controller/relocateSavedVideos.ts`) | `unreachable` with the stall's detail; `/storage` skips the store's size walk. | The target's `stat` (`:181`); `/storage`'s store walk too. |
+ | `listSavedVideos` (`controller/savedVideoInventory.ts`), when a page passes `notAnswering` | The channel is skipped and named (its `data/` link is read, not followed). The backup job passes nothing and is unchanged. | Each read of a relocated channel (`:81`). |
+ | The videos list, the video page (and its title), the channel page's Cleanup stage | A notice (`aria-label="media not answering"`, `MediaNotAnswering.tsx`) with links to the channel and `/storage`. | The list's `data/` listing and titles, then the selected video's files; the video page's whole directory read, as one unit; its title; the Cleanup stage's saved-video totals. |
+ | The channel page's Storage stage free space (L1) | "—". | The `statfs` of the relocated target. |
+ | The media file route (`/api/channels/<slug>/videos/<id>/files/<name>`) | 503 with `Retry-After: 15`. | The file's `stat`; the stream after it is not raced. |
+
+- **The memo** (`inspectChannelMedia`): five seconds per channel, keyed by channels dir, slug and
+ the configured `dataDir`, on `globalThis` (`__yttChannelMediaMemo__`). On by default (ruling Q4).
+ A remembered `in-transition` is given as it is; any other remembered answer is gated first; a
+ stall is not remembered. `{ fresh: true }` bypasses it and does not store. **The deciders, every
+ one passing `fresh: true`:**
+
+ | Caller | Where |
+ |---|---|
+ | `assertChannelMediaReachable` (so every guard below) | `lib/channelMedia.ts:514` |
+ | ↳ `runManagedFunction`'s `needsMedia` guard | `jobs/streamCommand.ts:283` |
+ | ↳ the operation batch | `controller/operationBatch.ts:1593` |
+ | ↳ `generateChannelSnapshot` | `controller/channelSnapshot.ts:739` |
+ | ↳ `normalizeAllTranscripts` | `controller/normalizeAll.ts:63` |
+ | ↳ keep-videos | `controller/keepVideosMatching.ts:165` |
+ | ↳ the shard action | `editor/app/channels/[slug]/shardActions.ts:88` |
+ | The channel mover, out and back | `controller/relocateChannelMedia.ts:638`, `:877` |
+ | The index build, before and after the walk | `controller/buildIndex.ts:348`, `:467` |
+ | The stats build | `controller/buildStats.ts:291` |
+ | The storage watch's five-minute pass | `controller/storageWatch.ts:235` |
+ | Clip-window eviction | `controller/evictClipWindows.ts:99` |
+ | The re-point preflight | `controller/storageLocations.ts:564` |
+ | `archilyzer doctor` | `bin/doctor.ts:124` |
+
+ **The memo's readers**, pages and polls: the home page (`editor/app/page.tsx:80`), `/channels`
+ (`editor/app/channels/page.tsx:219`), the channel page (`[slug]/page.tsx:239`), the ops channel
+ route (`api/ops/channel/[slug]/route.ts:70`), `channelsOnLocation`'s rollup for `/storage`
+ (`controller/storageLocations.ts:225`), the runners' `buildChannelWork` (`controller/autoRunner.ts:622`,
+ the tick and the three-second status poll share it), and the bulk actions' skips
+ (`editor/app/channels/actions.ts:562`, `bulkStorageActions.ts:105`; the jobs they queue re-check
+ fresh). `forgetChannelMedia(slug?)` clears it: the channel mover's marker writes and clear (each
+ phase writes its marker after the link and config it changes), `clearRelocationMarker`, a
+ completed re-point, `/storage`'s Refresh and the e2e `invalidate-cache` route.
+- **Headroom:** `UV_THREADPOOL_SIZE=${UV_THREADPOOL_SIZE:-16}` in `editor/package.json`'s `start`
+ (which the rollout restart script runs) and in `docker/entrypoint.sh` before the editor's `exec`.
+ Declared in `envVars.ts` (runtime) and `ENVIRONMENT.md` regenerated. It buys time for calls
+ already in flight and isolates nothing; the doc line says so. `ports.test.ts` reads every
+ `${NAME:-N}` in a script as a port, so the same commit names `UV_THREADPOOL_SIZE` as the one
+ numeric default that is not (ruling Q5).
+- **Where it shows:**
+
+ | Surface | What it says |
+ |---|---|
+ | `/storage` | The row's status badge reads **Not answering**; a line under the details reads `not answering since HH:MM — <cause>. Pages and polls skip this drive until it answers twice in a row. Watched through its disk's request counters.` (or `… with a stat of its root (no disk could be named here).`), `aria-label="location not answering"`. Re-point and Mount are withheld with the status. **Refresh** asks the location's detector first (one answer, counted like the pass's; the counters give none within 10 s of the last sample) and forgets the channels' remembered answers; on a stalled drive its note says so instead of probing. |
+ | `/channels` | The volume chip reads `… · not answering since HH:MM` in place of its free space, with a title saying what it means. Each row's badge reads `on <label> — not answering` (accessible name `media location: Media not answering · on <label>`). |
+ | The channel page | The Storage stage's card is red with the inspector's sentence; its destination list names the location "Not answering"; Move back is withheld for a stalled channel, as for an unreachable one. |
+ | `/saved-videos` | `Not read, because the drive their media is on is not answering: <slugs>.` (`aria-label="saved videos not read"`). |
+ | `/review`, the rack, the channel page (an auto-paused channel) | `Auto-paused — its media is on a drive that is not answering since <date>. It returns to <tier> on its own when the drive answers again.` (L4) |
+ | The runners | `[auto] skipping <slug>: media stalled — drive not answering (…)`, once per state change. |
+ | The logs | The health pass's transition lines, with the cause; the index and stats builds' hold lines. |
+
+ Labels are contracts: no existing accessible name or test id changed.
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `c0ecc55c` | `common:` `lib/storageHealth.ts`, `probeLocationHealth`, the 15 s health pass, the `stalled` statuses and the gate in inspect, the probe and its memo, `volumeFreeBytes`, `readChannelStat`, the recency reads, the move-root check and the saved-video store; `HELD_REASON.stalled`; the 5 s memo with `fresh` for every decider and `forgetChannelMedia` in the movers; the badge's and the stage card's words. Tests. |
+| `cfe551de` | `editor:` `/storage` (the row's line, Refresh asks the health first), the `/channels` volume chip, the Storage panel's Move back, the videos list and video page notice, the media file route's 503, the e2e `invalidate-cache` route; `refreshLocationHealth`; `views/storage.ts` `notAnswering`. |
+| `480f2556` | `editor:` `UV_THREADPOOL_SIZE=16` in the editor's `start` and `docker/entrypoint.sh`; `envVars.ts` + `ENVIRONMENT.md`; `ports.test.ts` names it as a numeric default that is not a port (folded in from the next commit, ruling Q5; no change to the tree at the tip). |
+| `4e1f7a90` | `editor:` `listSavedVideos({ notAnswering })` for `/saved-videos` and the Cleanup stage's notice. |
+| `63f42ef5` | `plans:` the first version of this section, FACTS, the changelog. |
+| `c69ad41a` | `common:` rulings Q2 and Q1(b): the marker before the gate; `onDrive` (the 3 s watchdog and the four-call cap per location) on every common gated call; `registerLocationHealth`. Tests. |
+| `f6a25cf5` | `editor:` the pages' reads of a relocated drive through `onDrive` (the videos list, the video page and its title, the file route, the Cleanup totals, `/storage`'s store walk). |
+| `b1a30902` | `common:` ruling Q1(a): `detectLocationHealth`, the block-device counters with the child-stat fallback; `detector` in the health state and on `/storage`. Tests. |
+| `981e05a0` | `plans:` this section rewritten for the rulings; FACTS; the changelog. |
+| `33094c29` | `common:` review M1, M2 (slot keys), L2, L5, L6, L7, L8, L10: the queue's deadline, the overdue count, waiters refused on every transition to stalled; the hand-typed-root and candidate-root keys; slow is not stalled; discards and flushes counted; the detector's samples on `globalThis`; findmnt only when needed. Tests. |
+| `bd39579e` | `common, editor:` review M2, L4, L9: the health pass armed above the idle gate; the pause record's `cause`; a `stalled` channel on no location is down. SETTINGS.md. Tests. |
+| `ff235c2f` | `common:` review M3: the snapshot walk through `onDrive`; keep-latest keys bounded. Test. |
+| `19c5842d` | `editor:` review L1, L3: the Storage stage's `statfs` through `onDrive`; the stale comments. |
+| `30df4193` | Merge `main` (`bab894db`: release 14 HS, S1, CF; release 15 UT). Two conflicts, both kept: the changelog's `[Unreleased]` carries UT's bullet then DS's; this file is `main`'s with DS's table row and this section after UT's. |
+| `b34a7613` | `plans:` the review's findings to their commits; FACTS; the changelog. |
+| `c367a3d7` | `editor:` re-review L11: the health pass's block above the boot probe's comment. |
+| `d61bdd93` | `common:` re-review M4: a slot wait's deadline follows progress. Tests. |
+| `ee68d225` | `plans:` the re-review's findings to their commits; FACTS. |
+| `9a12308e` | `common:` the overdue refusal tells slow from stalled by the counters, as a timeout does. Tests. |
+| this commit | `plans:` that ruling as a decision row; FACTS; the report. |
+
+**Tests** (unit; no test stalls a real drive: a stalled call is a promise that never settles or a
+fake `stat` that never answers, and a stalled device is a temp `/sys` whose counters stand still)
+
+| File | What it pins |
+|---|---|
+| `lib/storageHealth.test.ts` (28) | The rules: one miss stalls at once, one clean answer after a stall does not clear it and two in a row do, a miss in between starts over, `absent` is clean, a re-pointed root starts over, the path match is `locationOfDataDir`'s, pruning, `globalThis`, the "since" wording, registering without an answer. `onDrive`: an answer passes through (value or error) and frees its slot; a never-settling call is `stalled` on the timer, marks the location (since now) and keeps its slot until it settles; a stalled location is refused with no call; seven calls at once put four in flight and the three that waited are refused without a call when the location stalls; a freed slot runs a waiter; the 3 s default (answered after 3–4.5 s). After the review: a hand-typed root's four slots are shared by its channels and a fifth call is refused at once, with no entry made; a candidate root has its own slots and leaves the location's calls alone; **M1's case** (four hung calls, two clean answers, a fifth refused at once and the location marked again); a transition to stalled from the pass refuses the waiting call; **L6** (counters completing: four calls refused as slow, the location not marked, a waiter's wait runs out unmarked when nothing returns; counters standing still: marked). After the re-review (**M4**): a healthy 64-wide walk of units at half the budget (the last call waits about fifteen units) has no refusals and marks nothing, on a location and on a hand-typed root; a queue behind four hung calls is still refused within the budget (marked on a location, unmarked on a hand-typed root); a call that returns late hands its slot to the calls waiting behind it. Then: four units slower than the budget on a device whose completions move — the fifth call is refused at once and nothing is marked; with completions unchanged, after the pass cleared the location — refused and marked again. |
+| `lib/storageHealthCounters.test.ts` (7) | The stat line parser (17 and 11 fields, garbage); the verdict over sample pairs (stuck → stalled; moving, idle or drained → ok); device names (partition, `[subvolume]` suffix, non-`/dev` sources); the detector end to end with a fake findmnt and a temp `/sys`: the first sample gives no verdict, stuck → stalled with its cause, drained → ok, busy and moving → ok; a sample sooner than 10 s gives none and keeps the first; every no-device fallback goes to the child stat and says so (tmpfs, findmnt failing, no `/sys` entry, another volume's UUID, no binary). After the review: discards and flushes are counted (a flush alone is not a stall); a known device is read without running findmnt, and findmnt runs again only when its `/sys` entry stops reading; the samples are on `globalThis`. |
+| `lib/storageHealthProbe.test.ts` (5) | The child `stat`: a directory is `ok`, a missing path and a file are `absent`; a fake `stat` asleep for 20 s is `stalled` on the timer without being waited for; the 3 s default; an answer inside the budget is taken; no binary is `ok`. |
+| `controller/storageStall.test.ts` (22) | A spy on every `node:fs` and `node:fs/promises` call, with a hang mode that makes a matching promise-API call never settle. The gate: with the location stalled, inspect reads only the marker (with a config) or config.json and the marker (without); a channel mid-move on a stalled drive reads `in-transition` (and is remembered so); the guard, `probeLocation` and its memo (no findmnt run), `volumeFreeBytes`, `readChannelStat`, the recency layer, the move-root check, a snapshot refresh, the saved-video store and the inventory make no call on the drive. The watchdog: with the location answering and one drive call hung, inspect, `readChannelStat` (at most four video directories asked), `probeLocation`, `volumeFreeBytes`, the recency layer (not remembered as a miss) the move-root check and (M3) the snapshot walk each answer `stalled` within the race and mark the location (the snapshot writes no `snapshot.json`, and at most four video directories reach the drive); after it, inspect, the guard and the walk make no call on the drive, and once cleared the drive is asked again. The memo: two inspects inside 5 s stat the target once, `fresh` and a 5 s age ask again, a fresh answer is not stored, another target is another key, `forgetChannelMedia` and `clearRelocationMarker` clear it, and a stall is seen with an `ok` remembered. |
+| `controller/storageWatch.test.ts` (+9) | The pass stalls a location on one miss and a page then gets `stalled`; the five-minute pass suspects its channel; two clean passes clear it; a location no longer configured is forgotten; a probe that throws is `ok`; the health pass arms on its own, runs once at once and stops, and the five-minute watch arms no health pass; a Refresh counts as one answer; the pass registers every location, records a verdict's detector, and a verdict with no answer changes nothing; a counters verdict records its device and a stat verdict forgets it; a stall auto-pauses after two passes with `cause: "not-answering"` and its wording, the sanitizer keeps the cause, and a record without one reads as not there. |
+| `views/storage.test.ts` (+1) | A stalled row reads "Not answering", carries its line, has no free space, and withholds Re-point and Mount. |
+
+#### Gates (logs `$T/ds-*.log`)
+
+- **tsc** was clean before every commit and on the merged tree (65 s). After the re-review: 53 s,
+ once the worktree's gitignored `export/.next/dev` (a generated `validator.ts` an export dev server
+ had left truncated) was deleted; it is not a source file.
+- **Unit, on the merged tree (`30df4193`):**
+
+ | Suite | Result |
+ |---|---|
+ | common | **2,295/2,295**, 58 s: `main`'s 2,229 plus DS's 66 (60 at the rulings, 6 from the review). After the re-review: **2,299/2,299**, 53 s (4 for M4); after the overdue ruling **2,301/2,301**, 75 s (2 more). |
+ | editor unit | 87/87, and 87/87 after the re-review |
+ | `test:scripts` | 194 passed, 2 skipped (196). The second skip is UT's post-build trace check, which skips a umtool build older than its config (this worktree's `umtool/.next` predates UT); the first is the `LIVE=1` archive check. |
+ | mcp | 271/271 at the first pass; no mcp file has changed since. |
+
+- **Docs:** `docs env --check`, `docs files --check` and `settings example --check` all exit **0**
+ (SETTINGS.md regenerated in `bd39579e` for the pause record's `cause`).
+- **Build:** the editor's `next build` on the merged tree, with the primary's `transcripts/` linked
+ in and capped at 5 GB with no swap: 63 s, max RSS 1,641,444 KB, exit 0. The link was removed after the build, and
+ nothing ran through it. (First pass: 104 s, max RSS 1,643,860 KB.)
+- **e2e** (editor, detached and queued):
+
+ | Run | Specs | Result |
+ |---|---|---|
+ | 1, first pass | the `storage` and `channels` specs, `video-page`, `saved-videos`, `dashboard`, `auto-queue`, and IG's eight (`$T/ds-specs.txt`) | **124 passed, 3 failed, 12 skipped, 16.6 min**: three 30 s timeouts while this slice's own tsc and common suite ran beside the suite. |
+ | 2, first pass | `auto-queue`, `tags`, `saved-videos`, four cleanup specs, `video-titles`, `video-page` (`$T/ds-specs2.txt`) | **70 passed, 1 failed, 5.9 min**: `auto-queue.spec.ts:352`, the Start-button race its own comment describes. |
+ | 3, first pass | `auto-queue` alone | **24 passed, 1.3 min**. |
+ | 4, after the rulings (`b1a30902`) | the 9 `storage` + `channels` specs (`$T/ds-specs4.txt`) | **38 passed, 0 failed, 12 skipped, 2.5 min**. |
+ | 7, after the overdue ruling (`9a12308e`) | the 9 `storage` + `channels` specs (`$T/ds-specs4.txt`) | **38 passed, 0 failed, 12 skipped, 2.5 min**. |
+ | 6, after the re-review (`d61bdd93`) | the 9 `storage` + `channels` specs (`$T/ds-specs4.txt`) | **38 passed, 0 failed, 12 skipped, 2.6 min**. |
+ | 5, after the review, on the merged tree (`30df4193`) | the 9 `storage` + `channels` specs; the snapshot scheduler's (`auto-report-refresh`, `channel-work`, `jobs-batch-tasks-drain`, `incomplete-transcript`); the keep-latest users of the bounded key fan-out (`cleanup-holds`, `saved-videos`); `reconcile`; `review` (the auto-pause wording) — `$T/ds-specs5.txt` | **73 passed, 0 failed, 12 skipped, 5.2 min** (37 min 54 s in the queue behind another session's suite). |
+
+ The 12 skips in each are `channels-rack-audit`, which needs `E2E_RACK_SHOTS`. No e2e fixture has
+ a stalled drive: these confirm nothing changed for drives that answer. The stall paths are the
+ unit tests above.
+- **Numbers tool:** none.
+
+#### Found and left
+
+- **The stated limit.** A call already in flight when the drive stalls holds its thread until the
+ kernel gives up (about 30 s in the observed reset loop). On one drive, at most four threads wait
+ that way for the calls that go through `onDrive`: every page and poll path, and the snapshot
+ walk. A job's own reads that do not go through it are not capped: `measureTree` in the movers,
+ the index build's processing phase (IG recorded it), the snapshot's sequential reconcile pass
+ (one read at a time), and the keep-latest reads of the other callers of `computeKeptVideoIds`
+ (cleanup, persist, prune; now 16 at a time). With 16 threads the editor keeps answering while
+ those four wait.
+- **A root typed by hand** (a channel moved to a root no storage location names): its calls are
+ capped by the root they are under, four at a time, and raced, but nothing can mark it, so every
+ page and poll keeps asking it, each answering after 3 s or at once while its four slots are held
+ by calls the watchdog gave up on. Its channels' `stalled` comes from the watchdog alone, and the
+ five-minute pass counts it as down (L9). Naming the root as a location on `/storage` gives it the
+ full treatment.
+- **The drive's spin-up** (L6, a question for the operator): the budget is 3 s, and a USB disk that
+ spins down when idle can take 3–10 s to answer its first read. The watchdog then refuses that
+ read, and marks the location unless the counters show requests completing meanwhile (spinning
+ up completes none), for 15–30 s, during which the start-of-work guard refuses the channel's jobs.
+ **Does the drive spin down when idle?** If it does, a longer budget for the first call after an
+ idle spell, or a spin-down timer on the drive, would avoid it.
+- **A drive slower than the budget per unit is treated as not answering.** A queue's depth no
+ longer matters (M4): a waiting call is refused only when nothing on the drive has returned for
+ 3 s. But a single unit that takes longer than 3 s is refused: without a mark when the counters
+ show the disk completing other requests, with one otherwise. A snapshot refresh with such a unit
+ throws, so the scheduler keeps the last `snapshot.json` and tries again on its next trigger. Once
+ four such units hold every slot past the budget, the next call is refused at once (a decision
+ below says when that also marks).
+- **The counters need two samples.** A drive already stalled when the editor starts is seen by the
+ counters at the second pass (15–30 s), or at once by the watchdog when a page reaches it.
+ in_flight counts only requests dispatched to the driver: a request requeued during a host reset
+ is not counted, so a sample in that window can read clean; the watchdog covers it.
+- **Ungated request paths**, each a click rather than a page or poll: the channel and video server
+ actions that read a video's directory in the request (`bulkVideoActions`, `digestActions`,
+ `videoActions`, `fixIncompleteTranscript`, `pipelineActions`); most of what they do is enqueue
+ jobs, whose guard is fresh and refuses a stalled channel. `/api/media/fetch-window/<jobId>` (one
+ `stat` of the fetched file, once the job is done). The media file route's stream after its `stat`.
+- **The runners' tick reads the memo.** `buildChannelWork` serves both the tick and the status poll,
+ so a drive unmounted in the last 5 s can have one unit dispatched, which fails at the dangling
+ link. The snapshot regeneration and `runManagedFunction`'s guard are fresh.
+- **umtool's twin of the reachability check** (`checkChannelReachable` in
+ `umtool/report-to-video/cues.mjs`) has no stall gate: umtool is its own process with no health
+ state, and `umtool/**` belongs to another slice.
+- **A CLI process** (`archilyzer index`) has no health pass, so nothing is marked in it; its
+ inspects still go through the watchdog, so a target `stat` that takes over 3 s holds the channel.
+ The corpus disk itself is not watched.
+
+#### Rulings (parent, 2026-09-29) and decisions the operator could overturn
+
+| Question | Ruling | Where |
+|---|---|---|
+| Q1: the child stat of the root can be answered from the cache | Two detectors: the block device's counters every 15 s (child stat only with no device, and the state says which), and a 3 s watchdog on every gated call | `b1a30902`, `c69ad41a`, `f6a25cf5` |
+| Q2: the gate before or after the marker | The marker first; the gate covers everything after it | `c69ad41a` |
+| Q3: a stall auto-pauses | Yes, after two five-minute passes; it clears when the drive answers again, as for an unmount | as built |
+| Q4: the memo on by default | As built; every decider listed above | as built |
+| Q5: a commit that failed `ports.test.ts` alone | The test line folded into the thread-pool commit | `480f2556` |
+
+| What I assumed | The alternative |
+|---|---|
+| At most four gated calls per location in flight; the rest queue in JavaScript and are refused on a stall. **Kept at review.** | No cap: the watchdog alone, and a stall mid-walk fills the pool until the kernel gives up. |
+| A wait for a slot runs out at the budget plus a quarter of it (at most 250 ms), so simultaneous timeouts of the calls it waits behind mark first. | Exactly the budget: a waiter queued in the same tick then gives up a moment before those calls and is refused unmarked, and the mark lands a few ms later. |
+| The keep-latest key reads run 16 at a time for every caller (they were an unbounded `Promise.all`). | Bound them only for the snapshot. |
+| **Ruled after the re-review:** when every slot is held by a call past its budget, the next call is refused at once, and marks the location stalled only when the disk has completed nothing since the oldest of those calls began; four slow units on a disk still completing requests are "drive slow", refused and unmarked. With no device named (the stat detector), it marks. | Mark whenever every slot is overdue (the M1 fix as first built): four slow units on a busy disk then marked it stalled for 15–30 s. |
+| An auto-pause record with no `cause` reads as not there. | Word both cases for it ("not there or not answering"). |
+| Two counter samples closer than 10 s give no verdict (a Refresh just after a pass among them). **Kept at review.** | Compare any two samples (a busy healthy drive can read "in flight, nothing completed" over a few milliseconds). |
+| A findmnt that does not answer reuses the last device named for that root; one naming another volume's UUID names none. | Treat a findmnt that does not answer as a stall. |
+| An `absent` answer counts as clean toward clearing a stall. | Only `ok` clears it. |
+| `/storage`'s Refresh is one answer like the pass's. | Refresh clears a stall outright on one clean answer. |
+| `UV_THREADPOOL_SIZE` defaults to 16 in `start` and the entrypoint; a value already set wins. | A fixed 16, or only in the rollout restart script. |
+| The videos list, the video page and the Cleanup stage show a notice instead of their content. | Render what the snapshot knows and leave out only the drive's files. |
+| `readChannelStat` returns `null` on a stall (the job row draws no progress bar). | Counts marked unknown. |
+
+**What runs which code, for the rollout.** The health pass lives in the editor's process, armed with
+the storage watch, so detection takes effect only when the editor is rebuilt and restarted (and not
+on an idle boot). The restart must go through `pnpm run start` in `editor/` (the rollout restart
+script does) for `UV_THREADPOOL_SIZE` to apply; a process started another way keeps Node's 4 threads
+unless the variable is set. CLI builds have no health state; their inspects are raced.
+
+#### Review
+
+**Verdict: SHIP AFTER FIXES** (`ds-review.md` in the job's scratch). No High. The parent's rulings
+on each finding were applied as below.
+
+| Finding | Ruling | Where |
+|---|---|---|
+| M1: a queued `onDrive` call had no deadline | The review's fix: the `overdue` count, refuse at once and re-mark when every slot is overdue, the slot wait raced against the budget and refused without marking, `refuseWaiters` on every transition to stalled; the named test | `33094c29` |
+| M2: nothing protected an idle boot or a hand-typed root | The 15 s health pass armed above the idle gate, the five-minute pass below it; a path on no entry capped by the root it is under; the limit stated above | `bd39579e`, `33094c29`, this commit |
+| M3: the snapshot walk was uncapped and 16 wide | Its per-video unit (and its listing and keep-latest reads) through `onDrive(config.dataDir, …)`; the three "at most four threads" sentences (this record's Found and left, FACTS, the `onDrive` header) say what is true after it | `ff235c2f`, `33094c29`, this commit |
+| L1: the Storage stage's `statfs` | Through `onDrive`, "—" on a refusal | `19c5842d` |
+| L2: discards and flushes; requeued requests | Fields 12 and 16 added to completed; one FACTS line on requeued requests | `33094c29`, this commit |
+| L3: stale comments calling the child stat the detector | Fixed (the five named, and the health pass's header) | `33094c29`, `bd39579e`, `19c5842d` |
+| L4: "a drive that is not there" for a stalled drive | The pause record carries the cause; both cases worded | `bd39579e` |
+| L5: the detector's samples per module copy; "Refresh asks again at once" | Samples on `globalThis`; the changelog's wording | `33094c29`, this commit |
+| L6: a spin-up longer than the budget | Slow, not stalled, when the counters moved since the call began; the spin-down question above | `33094c29`, this commit |
+| L7: a candidate-root probe shared the location's slots | Keyed by its root | `33094c29` |
+| L8: the budget covers a whole unit | Said in the doc and above | `33094c29` |
+| L9: `stalled` on no location was not down | Counted as down | `bd39579e` |
+| L10: a findmnt every pass | Only on a root change or a failed `/sys` read | `33094c29` |
+| The two questions | Keep the four-call cap; keep the 10 s spacing | as built |
+
+**Re-review: SHIP AFTER FIXES.** Every finding above was confirmed closed (L6 for a busy drive; the
+spin-up case stays the operator's question). Two new ones:
+
+| Finding | Ruling | Where |
+|---|---|---|
+| M4: a slot wait was timed from when the call queued, so a deep queue on a busy but answering drive was refused as "not answering" (units of about 1.1 s at 16 wide, 0.46 s at 32, 0.2 s at 64) | The deadline follows progress: every return on the key re-arms its waiters, and a wait is refused only when nothing on the key has returned for the budget; the overdue and transition refusals unchanged. The limit statement and the keep-latest comment corrected | `d61bdd93`, this commit |
+| L11: the health pass's block sat inside the boot probe's comment, which still called itself the only thing an idle boot runs | Moved above it; the phrase dropped | `c367a3d7` |
+| (the implementer's note) four slow units past the budget on a disk whose counters show completions made the overdue refusal mark the location stalled | The same treatment as L6: the overdue refusal compares the counters with those taken when the oldest overdue call began; moved → refused, not marked; unchanged → marked | `9a12308e`, this commit |
+
+### Slice SS, as shipped — the editor's site scope is a cookie (2026-09-29)
+
+Branch `r15/site-scope` off `main` `721ed0eb`, worktree
+`~/Projects/plans-export-header-first-search` (block #4: editor 3401, test 3411, export 3410), one
+Opus implementer. Scratch files `ss-*` in the job's `tmp`. The ruling: the active site is a
+cookie, read once per request by the root layout and supplied through a provider, so the picker,
+Dashboard and Channels render the stored site on the first paint.
+
+**What was wrong.** The sidebar's "Active site" picker (`SiteScopeSelect.tsx`) derived its value
+from the URL on every render: a `?site=` param, else the `/sites/<id>` path, else "All sites". The
+stored choice was `localStorage["activeSite"]`, read only in an effect after paint. That effect
+`router.replace`d `?site=<id>` onto the URL, which re-rendered the picker with the right value. So
+every navigation painted "All sites" first, and Dashboard and Channels (server components that
+cannot read localStorage) rendered unscoped until the param arrived.
+
+- **The store** is a cookie, `archilyzer-active-site-<port>`: `path=/`, `SameSite=Lax`, one year,
+ `httpOnly`. The value is a site id or `__all__`.
+ - The name and its rules are in `app/lib/activeSite.ts`: `ACTIVE_SITE_COOKIE`,
+ `activeSiteCookieName(host)`, `isStorableActiveSite`, `ACTIVE_SITE_COOKIE_MAX_AGE`.
+ - The port is in the name because a cookie is shared by every port on a host, and localStorage
+ was per origin. The live editor and a worktree's editor on one machine keep one selection each,
+ as before. A host with no port (the default port, behind a proxy) uses the base name.
+- **One writer:** `setActiveSiteAction` (`app/lib/activeSiteActions.ts`, `"use server"`).
+ - It checks shape only: a `SITE_ID_RE`-shaped id of at most 128 characters, or `__all__`.
+ Anything else writes nothing and returns false. Whether the id names a site is decided at every
+ read, so a site deleted later resolves as no selection.
+ - Setting a cookie in a server action makes Next re-render the current page and its layouts.
+ That re-render is how Dashboard and Channels re-scope, and how the layout hands the provider
+ the new value; nothing calls `router.refresh()`.
+- **One read:** `readActiveSite(param?, siteIds?)` (`app/lib/activeSiteServer.ts`, `server-only`).
+ - The root layout calls it with no param (a layout has no `searchParams`). Dashboard
+ (`app/page.tsx`) and Channels (`app/channels/page.tsx`, the scope line only) call it with
+ `searchParams.site`.
+ - The precedence is one pure function, `resolveActiveSiteFrom(candidates, siteIds)`: the first
+ candidate naming a configured site or `__all__` wins, else `resolveActiveSite`'s default (the
+ lone site, else all sites). The server passes `[?site=, cookie]`.
+ - The layout reading a cookie makes every page render per request: the capped build lists every
+ page `ƒ`. The editor's pages were all `force-dynamic` already, apart from the not-found page;
+ route handlers were `ƒ` already, and `/icon.svg` stays static.
+- **The provider** (`app/components/SiteScopeProvider.tsx`, new): a React context, mounted by the
+ root layout around `AppFrame` with `{ activeSite, fromCookie, siteIds }`, read with
+ `useSiteScope()`.
+ - It holds `stored`, which starts from the layout's value and follows it when that value changes.
+ It does not follow while one of the picker's writes is in flight, so the first of two quick
+ choices cannot paint over the second.
+ - `choose(value)` is for the picker's own changes. It updates `stored` optimistically, writes
+ through the action, and puts the previous value back if the write fails.
+ - **Two effects, and neither rewrites a URL:**
+ - Visiting `/sites/<id>/…` records that site, so Dashboard and Channels follow. It writes once
+ per arrival at that site's pages (a StrictMode or Fast Refresh re-run does not write again),
+ and not when the store already holds the site.
+ - The one-time **migration**: a visitor with `localStorage["activeSite"]` and no cookie has the
+ key copied into the cookie through the action, then removed once the write succeeds. With a
+ cookie, or on a site's page, a leftover key is removed without being read. This is the only
+ localStorage read. On that one visit the server had no cookie to render from, so the picker
+ paints the default until the write's re-render.
+ - **Both effects only write** (`record`); the value comes back through the server's re-render.
+ They run as the page hydrates. With `choose`'s optimistic state change there, the picker
+ re-rendered before React replayed a change made on the server-rendered select before
+ hydration. The re-render reset the select to its old value, the replayed change read that
+ value, and the two `/sites/<id>/…` cases of `site-scope.spec.ts` navigated nowhere
+ (`d394d0b6`; traced with a throwaway instrumented spec, not committed).
+ - **Other tabs** (`f645572c`, review M1). The cookie is shared by every tab of the origin, but a
+ tab's `stored` comes from its root layout, which a client-side navigation does not re-render.
+ A pick in another tab therefore left this tab's picker on the old site over pages that read
+ the new one, and a client-side visit to a site's page was skipped as already stored.
+ - Every successful write (`choose` and `record`) is posted on the BroadcastChannel
+ `ACTIVE_SITE_CHANNEL` (`"archilyzer-active-site"`). A channel is per origin, so each port
+ has its own, like the cookie's name.
+ - The other tabs answer with `router.refresh()`: the layout and the page re-render with the
+ cookie as it is now, and the tab's router cache is dropped. A channel instance does not
+ receive its own posts, so a tab does not refresh for its own write.
+ - A browser without `BroadcastChannel` keeps each tab's value until its next full load.
+- **The picker** (`SiteScopeSelect.tsx`) renders from the context, the path and the URL. It has no
+ effect and reads no storage, and every accessible name is unchanged.
+ - What it shows, first match wins: the choice just made on this URL, the site the path names, a
+ valid `?site=`, then `stored`.
+ - On a site's pages it writes the cookie, **then** pushes the same tab of the other site (or
+ `/sites` for "All sites"). A page opened once the URL has moved therefore reads the new value.
+ - On a `?site=` link's page it writes, then `router.replace`s the URL without `site`
+ (`withoutSiteParam`), so the page and the picker agree. Elsewhere it only writes.
+ - In the first two cases the choice is held for the URL it was made on, so the controlled select
+ does not snap back while the navigation is in flight. The old picker snapped back on a site's
+ pages until the push landed. The hold is dropped on the first render at another URL
+ (`352ea8f7`); before that, Back to the page it was made on showed the old choice over the
+ path.
+- **A new channel starts checked on the active site** (`5006c28f`, review L5, after DS merged).
+ - `ChannelFormClient` resolves it as the picker does off a site's pages:
+ `resolveActiveSiteFrom([?site=, useSiteScope().stored], siteIds)`, on its first render. It
+ used to read only `window.location.search`, in a mount effect, so `/channels/new` reached from
+ the editor's own links never pre-checked a site.
+ - `SiteMembershipsSection` takes that site into its initial state instead of an effect. The
+ server renders the box checked, so there is no unchecked first paint. A later refresh (from
+ the pulse, or a pick in another tab) cannot re-check a box the user has cleared; the effect
+ re-ran whenever `sites` came back as a new array.
+ - With one site configured and nothing stored, the active site is that site, so it starts
+ checked (see the decisions table).
+
+**`?site=` after this slice.** The editor's own navigation no longer appends it; nothing in
+`editor/app` builds a `?site=` link. What still reads or carries one:
+
+| Where | What it does with `?site=` |
+|---|---|
+| Dashboard, Channels (`readActiveSite(site)`) | A valid one governs that request; it is not stored |
+| The picker | Shows a valid one on that page; a choice there drops it |
+| `next.config.ts` redirects | A retired `/charts`, `/aliases` or `/deploy` bookmark carrying `?site=<id>` lands on that site's tab (the comment now says the picker ignores the query there, `e684668c`) |
+| `ChannelFormClient` (`/channels/new`) | A valid one is the site a new channel starts checked on, instead of the stored one (`5006c28f`) |
+| `ChannelVolumeBar`'s chips | Keep whatever `?site=` the URL has when they add `?location=`; the comment now says the stored scope is a cookie (`5006c28f`) |
+| e2e (`channel-groups`, `channel-priority`, `channels-rack-*`, `channel-site-membership`, `site-scope`, `navigation`) | Deep links; all still work |
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `9dc33d59` | `editor:` the cookie, its one writer and one read; `SiteScopeProvider`; the picker rewritten; the layout mounts the provider; Dashboard and Channels read through `readActiveSite`; `activeSite.test.ts` +8 (87 → 95). |
+| `d394d0b6` | `editor:` the provider's two effects write the cookie without an optimistic state change (the hydration replay above); the visit is recorded once per arrival. |
+| `0465d37b` | `editor(e2e):` `site-scope.spec.ts`: three `?site=` URL assertions inverted, two comments; three new cases. |
+| `e684668c` | `editor:` `next.config.ts`'s retired-satellite comment. |
+| `352ea8f7` | `editor:` the picker drops a held choice once the URL moves; the site-scope case for Back. |
+| `49802455` | `plans:` this section, the slices row, FACTS ("Superseded by release 15 slice SS"), the editor changelog. |
+| `f645572c` | `editor:` review M1: a successful write is broadcast, and the other tabs refresh. |
+| `09559337` | `editor(e2e):` review M1 and L2: the two-tab case; `expectNoSiteParam()` after the settle waits. |
+| `8104732b` | `editor:` review L1: the layout's comment says "every page". |
+| `001c23d0` | `plans:` the review, its findings to their commits, L3, L4 and L6 under "Found and left", the rulings; FACTS (L1, other tabs); the changelog's other-tabs sentence. |
+| `4b61be87` | `plans:` two lines of this section reflowed. |
+| `07d9126e` | Merge `main` `ef4f1d7c` (DS and the rest). The one conflict, this file, kept `main`'s sections whole (IG, UT, DS) with SS's after them and its row after UT's; FACTS and the changelog merged cleanly. |
+| `5006c28f` | `editor:` review L5: a new channel starts checked on the active site, from the provider, as initial state; the stale `?site=` comments in `ChannelFormClient`, `ChannelForm`, `SiteMembershipsSection` and `ChannelVolumeBar`. |
+| `677d1668` | `editor(e2e):` the new-channel case. |
+| this commit | `plans:` L5 and the merge in this section; FACTS; the report. |
+
+**Tests**
+
+- **Unit** (`app/lib/activeSite.test.ts`, +8): the cookie name per port (IPv4, IPv6, no port, no
+ host); what may be stored (a header-injection string, `_homepage`, 129 characters and non-strings
+ refused; 128 accepted); the precedence (a param beats the cookie, a param naming no site falls
+ through to it, `__all__` is a choice, the default); `withoutSiteParam` keeps every other param.
+- **e2e** (`editor/e2e/site-scope.spec.ts`). The contract cases keep their names and labels. The
+ seeding removal changed three assertions and two comments, and nothing else:
+ - in "selecting a site scopes the channels list and persists", two `toHaveURL(/site=alpha/)`
+ became `toHaveURL(/\/channels$/)`;
+ - in "charts is a site's tab", `/site=beta/` likewise;
+ - the comments now say "cookie" where they said "localStorage";
+ - after review L2 (`09559337`), each of the three is followed by `expectNoSiteParam()`. It waits
+ for hydration and the 750 ms settle, then checks the URL has no `site` param, with no retry.
+ `toHaveURL` alone passes on its first poll, before a `replace` from an effect could land. The
+ first-paint case makes the same check after each of its settles.
+- **New cases:**
+
+| Case | What it pins | On the pre-change code |
+|---|---|---|
+| a stored site is the picker's first paint on every page, never All sites | alpha stored through the picker. On `/`, `/channels`, `/jobs`, `/settings`, `/sites/alpha` and `/`, the value is read ONCE, with no retry, right after `domcontentloaded`. After each page hydrates and settles for 750 ms, an init script's record of every value the select ever had (at every DOM mutation and every animation frame) is exactly `["alpha"]`. The same holds across client-side navigations through the sidebar. `/sites/beta` records beta, and the next `/` paints beta. No console message or page error matching `/hydrat/i` | fails at the first read: `/: first paint` expected `"alpha"`, received `"__all__"`. With that read removed, it fails at the record: `["__all__", "alpha"]` |
+| a `?site=` link scopes its own page and is not stored; a choice there drops it | `/channels?site=beta` paints beta and lists beta's channel; the cookie stays alpha, and the next `/channels` is alpha; choosing "All sites" on `/channels?site=beta` leaves `/channels` with both channels and the cookie `__all__` | not run (the cookie assertions cannot hold) |
+| a choice the old picker kept in localStorage moves to the cookie once | the key seeded from a route that mounts no app; `/channels` then shows beta scoped, the cookie is beta and the key is gone; the next `/` paints beta; a leftover key with a cookie is removed and never read | not run |
+| Back to a site's page shows that site, not the choice made there | on `/sites/alpha/charts`, choosing beta goes to `/sites/beta/charts`; Back shows alpha | not run on the pre-change code; with `0465d37b`'s picker (the fix line removed) it received `"beta"` |
+| a new channel starts checked on the stored site, or on a `?site=` link's (`677d1668`) | alpha stored: `/channels/new` has Alpha checked and Beta not, read with no retry after `domcontentloaded`; `/channels/new?site=beta` has Beta checked and not Alpha as well; with Alpha cleared by hand, a pick of beta in another tab refreshes the page and checks neither | on the pre-L5 form (`07d9126e`'s `ChannelFormClient` and `SiteMembershipsSection`): fails at the first read, `stored site, first paint` expected `true`, received `false` |
+| a site picked in another tab reaches this one (`09559337`) | two pages in one context, passive refresh off (`autoRefreshIntervalSeconds: 0`), so only the broadcast can move tab A. Tab A stores alpha and sits on `/settings`; tab B picks beta; tab A's picker becomes beta with no reload, and its next Channels page (a sidebar click) is beta's. A client-side visit from tab A to `/sites/alpha` records alpha, and tab B's picker follows | with the broadcast's `router.refresh()` removed, tab A stays `"alpha"`. With passive refresh left on, tab A had moved anyway, through a pulse-driven tree refresh, and only tab B's check caught the missing broadcast; hence the setting |
+
+The spec's comment says why: Playwright's auto-retrying `toHaveValue` cannot see a one-paint flash;
+it polls until the value is right and passes.
+
+#### Gates (logs `$T/ss-*.log`)
+
+- **tsc** (all workspaces) was clean before every commit: 69 s, 38 s, 102 s and 28 s at the four
+ full runs, and 60 s for the review fixes. The editor-only runs for `0465d37b` and `e684668c`
+ took about 13 s. A killed dev server left a truncated `.next/dev/types/*.ts` once; that
+ directory is generated, and it was removed after each stopped run.
+- **Unit:**
+
+ | Suite | Result |
+ |---|---|
+ | common | 2,229/2,229, 44 s |
+ | editor unit | **95/95** (87 + 8), and again after the review fixes |
+ | `test:scripts` | 191 passed, 1 skipped (192) |
+ | mcp | 271/271 |
+
+- **Docs:** `docs env --check`, `docs files --check` and `settings example --check` all exit **0**.
+- **Build:** the editor's `next build`, with the primary's `transcripts/` linked in and capped at
+ 5 GB with no swap: **33 s, max RSS 1,628 MB**, exit 0. Every page is listed `ƒ`. The link was
+ removed after the build, and nothing ran through it. After the merge of `main` and L5, at
+ `677d1668`: **36 s, max RSS 1,617 MB**, exit 0; only `/icon.svg` is `○`.
+- **After the merge of `main` and L5** (at `677d1668`): tsc (all workspaces) clean, 35 s; common
+ **2,301/2,301** (`main`'s count), 51 s; editor unit **95/95**.
+- **e2e** (editor, detached and queued):
+ - `site-scope.spec.ts` alone:
+ - at `9dc33d59`: 8 passed, 2 failed (the two `/sites/<id>/…` cases, fixed by `d394d0b6`);
+ - at `d394d0b6`: 9 passed, 1 failed, 3.4 min. The failure was "creating a channel under a
+ site": the create never navigated within 10 s, with the Next dev indicator on "Rendering…"
+ and a load average of 26. It passed in the run before, and in every run after;
+ - at `352ea8f7` (with the Back case): **11 passed, 0 failed, 53 s**;
+ - after the review fixes (the code of `8104732b`, whose layout change is a comment): **12
+ passed, 0 failed, 1.6 min**, after about 2 min in the queue.
+ - **After the merge of `main` and L5** (at `677d1668`; `$T/ss-specs-l5.txt`: `site-scope` plus
+ every spec that visits `/channels/new`: `channel-site-membership`, `channels`,
+ `new-channel-onboarding`, `social-channel`, `pipeline`, `queues`, `dashboard`): **56 passed,
+ 0 failed, 6.0 min**. The full suite was not rerun, as the parent directed.
+ - The pre-change checks above: the old code swapped in once, then restored. The Back case was
+ run once with the fix line removed, then restored. So was the two-tab case, with the
+ broadcast's `router.refresh()` replaced by a no-op, twice: with passive refresh on, then off.
+ - **The spec list** (at `e684668c`; `$T/ss-specs.txt`: `site-scope` plus every spec that visits
+ `/` or `/channels` or uses `?site=`, 28 specs, 169 tests):
+ - a first run was spoiled by this implementer. `next.config.ts` was edited mid-run, and the dev
+ server restarted and served 404s. It was stopped, and its orphaned servers were killed.
+ Before the edit, one case had failed: `backfill.spec.ts:462`, whose `uncheck` of "Enable
+ auto-backfill" did not change the box. That spec passed in the clean run;
+ - the clean run: **157 passed, 0 failed, 12 skipped** (the rack screenshot audit, which needs
+ `E2E_RACK_SHOTS=1`), **8.2 min**.
+ - **The full editor suite** at `352ea8f7`: **658 passed, 0 failed, 12 skipped** (the same rack
+ audit), **39.2 min**, after less than a minute in the queue. An earlier full run was stopped
+ two minutes in, to land `352ea8f7` first. It was not rerun for the review fixes, as the parent
+ directed: they touch the provider and `site-scope.spec.ts` only.
+
+#### Found and left
+
+- **`export/app/(workspace)/WorkspaceView.tsx:60-73` (`splitOn`)** has the same class of one-paint
+ flash: a localStorage value restored in an effect after the first paint. `export/**` is not this
+ slice's.
+- **A change on a site's pages waits one server round trip before it navigates** (the cookie write
+ comes first), and a change there right after landing also waits for the visit's own write,
+ because Next runs server actions one at a time. The select shows the choice at once.
+- **A change made before hydration** reaches the picker through React's replay, as before. It
+ would be lost again if anything set state in the provider or picker during hydration; the
+ comment in `SiteScopeProvider.tsx` says so.
+- **`app/sites/[siteId]/layout.tsx:11`** still describes the retired satellites as reading
+ `?site=`. That is history, and it is accurate.
+- **Two server renders per change on a site's page, or on a `?site=` page** (review L3). The
+ write's re-render draws the page being left, then the push or replace draws the next one. This
+ is the cost of the accepted round trip.
+- **A narrow migration race** (review L4). On the first visit after the update, a visitor may
+ have the old key and no cookie, and change the server-rendered select before hydration. If
+ React replays that change before the migration's write is queued, the old key's value is
+ written second and wins. The picker then shows that value, and it agrees with the cookie.
+- **A stored `__all__` with one site left** (review L6, not a regression). The server resolves
+ "all sites", but the picker has no "All sites" option with one site, so the select shows the
+ lone site while Dashboard and Channels show the whole pool. `main` did the same through the
+ seeded `?site=__all__`.
+- **A tab restored from the back-forward cache** is not refreshed; the review offered a
+ `pageshow` handler as optional, and it was not added.
+
+#### Decisions the operator could overturn
+
+| What I assumed | The alternative |
+|---|---|
+| The cookie's name carries the port, so each editor on one host keeps its own selection, as localStorage did. **Ruled at review: it stays.** | One host-wide name: selecting in a worktree's editor would change the live editor's selection, and a site id the other has not got resolves as "All sites" there |
+| `httpOnly`: the page gets the value through the layout, never from `document.cookie` | Readable from script; nothing needs it |
+| The action checks shape only, and existence is decided at read | Refuse ids that name no site at write time; a site deleted later still needs the read-time check |
+| A `?site=` naming no site falls through to the cookie. The old picker ended there too, by replacing the param with the stored value after the first paint. **Ruled at review: it stays.** | Fall to the default (the lone site, else "All sites") for that page |
+| Choosing on a `?site=` link's page drops the param and stores the choice. **Ruled at review: it stays.** | Keep the param and store nothing, as a link's page is "just that page"; the picker and the page would then disagree |
+| On a site's pages, the cookie is written before the push. **Ruled at review: the round trip is accepted.** | Push first and write after: faster, but a page opened right after the URL moves could read the old value, and a navigation started while an action is pending discards the action's re-render |
+| The migration writes without an optimistic update, so its one flash lasts until the write's re-render | Update at once: a shorter flash, but a state change during hydration (see above) |
+| A new channel starts checked on the active site as the picker resolves it, so an editor with one site and nothing stored starts it checked on that site. **Ruled at re-review: it stays.** | Only a stored choice or a `?site=` link; the lone site would then need a click |
+| The new-channel pre-check is the form's initial state, so a pick made in another tab while the form is open does not change its boxes. **Ruled at re-review: it stays.** | Follow the active site while the form is untouched |
+
+#### Review
+
+**Verdict: SHIP AFTER FIXES** (`ss-review.md` in the job's scratch). There was no High. The review
+held that single-tab first paint, hydration, the cookie, the action and the migration are right.
+It reproduced M1 with a two-tab probe under the queue lock.
+
+| Finding | Where |
+|---|---|
+| M1: a pick in another tab left this tab's picker stale, and a client-side visit to a site's page unrecorded | `f645572c` (the broadcast and refresh), `09559337` (the two-tab case) |
+| L1: FACTS said every route renders per request; it is every page | this commit (FACTS), `8104732b` (the layout's comment); this section said it already |
+| L2: the inverted URL assertions pass on their first poll | `09559337`: `expectNoSiteParam()` |
+| L3: two server renders per change on a site's page or a `?site=` page | "Found and left" |
+| L4: a narrow migration race | "Found and left" |
+| L5: stale `?site=` comments, and the `/channels/new` pre-check | After DS reached `main` (merged at `07d9126e`): `5006c28f`, `677d1668` |
+| L6: a stored `__all__` with one site left | "Found and left" (pre-existing) |
+| Questions: the port in the name, `?site=` naming no site, a choice on a `?site=` page, the round trip | Ruled: all four stay (see the decisions table) |
+
+**What runs which code, for the rollout.** The picker, the provider and the pages are in the
+editor's built bundle, so all of it takes effect only after the editor is rebuilt and restarted.
+After that, each browser's first visit migrates its localStorage selection once.
+
+### Slice DT, as shipped — the drive-health timings are settings (2026-09-30)
+
+Branch `r15/drive-timings` off `main` `6b8aa450` (slice DS merged), worktree
+`~/Projects/r12-paths-fix` (block #12: editor 4201, test 4211, export 4210), one Opus implementer.
+Scratch files `dt-*` in the job's `tmp`. The ruling (operator, 2026-09-30): the drive-health timings
+are configurable in `settings.json`, editable on `/storage`, with today's constants as the defaults;
+if a stall is misjudged under heavy external-disk churn, the operator tunes the numbers rather than
+the code.
+
+**What was fixed in code.** Slice DS judged "mounted and not answering" by five constants in
+`lib/storageHealth.ts`: the watchdog's budget (`DRIVE_CALL_BUDGET_MS`, 3 s), the health pass's cadence
+(`HEALTH_PROBE_INTERVAL_MS`, 15 s), the child-stat and findmnt race (`HEALTH_PROBE_TIMEOUT_MS`, 3 s),
+the clean answers that clear a stall (`HEALTH_CLEAN_TO_CLEAR`, 2) and the calls in flight per
+location (`DRIVE_CALLS_IN_FLIGHT`, 4). The constants are gone; tsc named every reader.
+
+- **The setting** is `settings.storage.health`, a nested block of five optional keys:
+
+ | Key | Default | Range | Takes effect |
+ |---|---|---|---|
+ | `budgetMs` | 3000 | 500–60000 | the next `onDrive` call (read as the call starts) |
+ | `passIntervalMs` | 15000 | 5000–300000 | at once on a save from `/storage` (the armed pass re-arms its timer); a hand edit, at the next pass |
+ | `probeTimeoutMs` | 3000 | 500–30000 | the next pass or Refresh (child `stat` and findmnt) |
+ | `clearAfterCleanPasses` | 2 | 1–10 | the next answer |
+ | `inFlightPerLocation` | 4 | 1–8 (review M1: at most half the editor's 16 file-access threads) | the next slot taken; a raised cap admits waiting calls at once, a lowered one is reached as calls return |
+
+ - The type, defaults, ranges, sanitizer, words and `SETTINGS.md` docs are one pure module,
+ `common/lib/storageHealthTimings.ts` (the `/storage` form, a client file, imports it).
+ `StorageSettings.health` is optional; `sanitizeStorage` keeps a block only when something is in it.
+ - **A read clamps; only what differs from a default is kept.** `sanitizeStorageHealth` rounds each
+ number and clamps it into its range (the schema's convention: a read never throws), drops a
+ non-number, and drops a value equal to its default. An untuned `settings.json` therefore has no
+ `health` key, and a save of every default removes it.
+ - The mediaRoot migration keeps a `health` block that spells no `locations` (a hand edit).
+- **The read path: one accessor.** `healthTimings()` (`storageHealth.ts:182`) returns the timings as
+ last applied, on the health state's `globalThis` object, every absent key its default. There is no
+ settings memo in lib (`getSettings()` reads the file on every call), so the accessor reads no file:
+ the numbers are applied into memory by `applyHealthTimings(stored)` (`:193`):
+ - the health pass, at the start of every pass, from the settings it already reads for its
+ locations (so at boot, and within one pass of a hand edit);
+ - the `/storage` save, at once (`saveHealthTimingsAction`);
+ - the `index` and `build stats` bins, once, before the build (a CLI process has no pass).
+ `lib/storageHealth.ts` stays free of I/O. `resetStorageHealth` keeps the timings (configuration, not
+ health). `setDriveCallBudget` stays the test seam below the 500 ms floor and wins over them.
+- **The derived numbers stay derived.** The slot wait's grace is still a quarter of the budget, at
+ most 250 ms. The counters' sample spacing is `counterSampleMinimumMs(passIntervalMs)` =
+ min(10 s, interval − 5 s), floored at half the interval (`minCounterIntervalMs()` in
+ `storageVolumes.ts:472`): 10 s at the default 15 s, as before, and at every interval from 10 s up the
+ ruling's formula exactly (see the decisions table for the floor).
+- **The re-arm.** `startStorageHealthWatch` arms at `healthTimings().passIntervalMs` and subscribes
+ with `onPassIntervalChange` (`storageHealth.ts:214`); `applyHealthTimings` tells the subscribers when
+ the interval changed, and the watch clears and re-arms its interval (log line `[storage] health
+ pass re-armed: every N s`). The subscription is on `globalThis`, so the save (a page's module copy)
+ reaches the pass (instrumentation's). An explicit `intervalMs` (the tests') follows nothing.
+- **The cap, changed live.** `acquireSlot` reads the cap each time. `releaseSlot` hands a slot on only
+ while the key is at or under the cap, and otherwise gives it back; `admitWaiters` lets calls already
+ waiting take the slots a raised cap adds. The overdue refusal compares with the cap as it is, and
+ its words count the overdue calls ("N reads on it have not answered") instead of naming the constant.
+- **`/storage`: Drive health timing.** A collapsed `<details>` at the foot of the page
+ (`components/HealthTimingForm.tsx`, `aria-label="drive health timing"`; its summary says
+ "defaults" or "N changed from the default"). Five text inputs (`inputMode="numeric"`), each with the
+ default as its placeholder, a hint line in the operator's words, and the default and range
+ (`HEALTH_TIMING_HINTS`); the interval's hint says a save re-arms the check at once. An empty field is
+ the default. Accessible names (new): `read budget`, `health check interval`, `health check timeout`,
+ `clean checks to clear`, `reads at once per drive`, `save timing`, `timing saved`, `timing error`;
+ none contains another. No existing name changed.
+ - The server action (`saveHealthTimingsAction`, `app/storage/actions.ts`) parses with
+ `lib/healthTimingsForm.ts`: a value out of range, or not a whole number, is **refused** with a
+ sentence naming the field, the range and the value ("Read budget must be between 500 and 60000 ms
+ (got 200 ms)."), and nothing is written. Otherwise it saves a patch of the storage block through
+ `saveSettings`, applies the timings, revalidates `/storage`, and answers with the timings now in
+ force.
+- **The surfaces say the setting, not the constant.** `/storage`'s "not answering since …" line stays,
+ and its "until it answers twice in a row" follows the clear count (`clearRuleText`: once, twice in a
+ row, N times in a row). So do the Refresh note (and its "checked every N s"), the `/channels`
+ volume chip's title (a new `clears` field on `ChannelVolume`), the health pass's stall log line,
+ and the media-not-answering notice ("within N s", "checked every N s"). The `onDrive` header, the
+ module header, and the comments that said "3 s" or "every 15 s" in the gated callers now name the
+ setting and its default. FACTS' "The storage health gate" has a bullet for the setting and names the
+ keys where it named the constants.
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `13411edc` | `common:` `lib/storageHealthTimings.ts`; `settings.storage.health` in the schema, the docs table and the migration; `healthTimings()`, `applyHealthTimings()`, `onPassIntervalChange()`; the pass applies what it reads and re-arms; the live cap; `minCounterIntervalMs()`; the two bins; `SETTINGS.md`; the comments. Tests. |
+| `41ad2f65` | `editor:` the Drive health timing form, its action and parse (+ unit test), the page; the Refresh note, the stalled line, the media notice, the volume chip's title; the comments. |
+| `b92df1fe` | `editor(e2e):` `storage-locations.spec.ts`: the drive health timing case. |
+| `4372abaf` | `common:` the cap's hint says to keep it well under the editor's 16 file-access threads. |
+| `fd2b8d63` | `plans:` the first version of this section and the slices row; FACTS; the changelog. |
+| `85c2407b` | `common:` review M1: `inFlightPerLocation` at most 8; the hint, the docs string, `SETTINGS.md`, the test values (the form's too). |
+| `6a24ac4e` | `common:` review L4: a timeout on no known location names the budget the call ran against. Test. |
+| `e7a525f3` | `editor:` review L2: a storage patch of the locations keeps `storage.health` (a `saveSettings` merge case). |
+| this commit | `plans:` the rulings and the review in this section; FACTS; the report. |
+
+**Tests** (unit; no test stalls a real drive)
+
+| File | What it pins |
+|---|---|
+| `lib/storageHealthTimings.test.ts` (7, new) | The defaults are the constants they replace, each in its range. Absent, empty, an array, a string, a number: every default, and no block (`getSettings()` with no file, `defaultSiteSettings()`). A read clamps (200 → 500, 900000 → 300000), rounds (7.6 → 8), drops a string and an unknown key, and drops a value equal to its default (also one that rounds onto it). **The settings.json round trip** through `writeSettings`/`getSettings`: `{budgetMs: 4000, clearAfterCleanPasses: 2}` is written as `{budgetMs: 4000}` and read back; a save of the default removes the key; a hand-edited 99 reads as 8 (16 before review M1). The block survives the mediaRoot migration. The spacing (15 s → 10 s, 300 s → 10 s, 12 s → 7 s, 10 s → 5 s, 8 s → 4 s, 5 s → 2.5 s). The words. |
+| `lib/storageHealth.test.ts` (+7) | **The accessor feeds the watchdog:** a stored `budgetMs: 200` applies as 500 ms (the floor), and a 700 ms unit is refused and marks the location ("within 0.5 s"); on the defaults the same unit answers. **The cap:** with `inFlightPerLocation: 2` the third call waits, and runs when a slot frees. A cap raised from 1 to 3 admits the two waiting calls at once; lowered to 1 with three in flight, two returns bring it to one and the fourth call still waits, and runs on the third return. **The clear count:** with 3, two clean answers do not clear and the third does; with 1, one does. A changed interval is told to the subscribers once, an unchanged one and other keys are not, and an unsubscribed one hears nothing. The test seam's budget wins, and a reset keeps the timings. The existing constants' assertions read `HEALTH_TIMING_DEFAULTS`. After review L4: a call on a hand-typed root whose budget changes while it is out (80 ms, then 5 s) is refused naming 0.08 s (the pre-fix code named 5 s). |
+| `lib/storageHealthCounters.test.ts` (+1) | The spacing follows the applied interval: at 8 s, samples 3,999 ms apart give no verdict and 4,000 ms apart compare (stalled); at 300 s it is 10 s. The existing case reads `minCounterIntervalMs()` (10 s). |
+| `lib/storageHealthProbe.test.ts` (+1) | With `probeTimeoutMs: 500` applied and no `timeoutMs` passed, a child that sleeps 20 s is `stalled` after 0.5–2.5 s. |
+| `controller/storageWatch.test.ts` (+2) | **Every pass applies what it reads:** `clearAfterCleanPasses: 3` and `budgetMs: 5000` in the settings are in force after the first pass; its stall line says "until it answers 3 times in a row"; two clean passes do not clear and the third does; a pass handed its locations reads no settings and leaves the timings. **The re-arm:** the armed watch, told `passIntervalMs: 5000` (the save's apply), logs the re-arm and runs its second pass 4.5–7 s later (15 s at the default); a stopped watch re-arms nothing; one armed with an explicit interval does not follow. |
+| `editor/app/storage/lib/healthTimingsForm.test.ts` (5, new) | One field per key, in order, with accessible names none of which contains another. Empty and blank fields write nothing. A value in range is kept and trimmed; one equal to its default is not written. Out of range is refused with the field, range and value, for a millisecond field and both counts (the cap: 8 kept, 9 refused, after review M1). "3.5", "-1", "3e3", "abc", "0x10" and "two" are refused as not whole numbers. |
+| `editor/app/settings/saveSettings.test.ts` (+1, review L2) | A storage patch of `{ locations, defaultLocationId }` (what the /storage location actions write) keeps `storage.health` and `savedVideosLocationId`. |
+
+**e2e** (`storage-locations.spec.ts`, new case "the drive health timing saves to settings.json and
+reads back"): after hydration, the block is collapsed and says "defaults"; opened, `read budget` is
+empty with placeholder 3000 (`reads at once per drive`: 4); 4000 saved → "A read may take 4 s" and
+`test-settings.json` holds `storage.health` = `{budgetMs: 4000}` and nothing else; after a reload the
+summary says "1 changed from the default", the field reads 4000 and the interval is empty; 200 is
+refused with the sentence and the file is unchanged; emptied and saved, "A read may take 3 s" and the
+`health` key is gone. The fixture's settings are its own `test-settings.json`.
+
+#### Gates (logs `$T/dt-*.log`)
+
+- **tsc** (all workspaces): clean before every commit — 58 s on the tree of the first three code
+ commits, 51 s after the hint's; after the review, 44 s (M1) and 58 s (L4, L2).
+- **Unit:**
+
+ | Suite | Result |
+ |---|---|
+ | common | **2,318/2,318**, 50 s (`main`'s 2,301 + 17); after the review **2,319/2,319**, 50 s (+1, L4) |
+ | editor unit | **100/100** (95 + 5); after the review **101/101** (+1, L2) |
+ | `test:scripts` | 194 passed, 2 skipped (196), as at DS |
+ | mcp | not run: no mcp file and nothing it imports changed |
+
+- **Docs:** `settings example --check`, `docs env --check` and `docs files --check` all exit **0**,
+ before and after the review (`SETTINGS.md` regenerated in `13411edc`: the `health` row and the
+ `storage.health` table; and in `85c2407b` for the cap's range; `settings.json.example` unchanged,
+ the default block has no `health`).
+- **Build:** the editor's `next build`, with the primary's `transcripts/` linked in (`ln -sT`) and
+ capped at 5 GB with no swap: **34 s, max RSS 1,648,128 KB**, exit 0. The link was removed after the
+ build, and nothing ran through it.
+- **e2e** (editor, detached and queued; `$T/dt-specs.txt`, the nine `storage` + `channels` specs DS
+ ran), with the new case in `storage-locations`:
+
+ | Run | At | Result |
+ |---|---|---|
+ | 1 | `b92df1fe` (the hint's edit landed on disk while it ran; nothing it asserts) | **39 passed, 0 failed, 12 skipped, 2.7 min** |
+ | 2 | `4372abaf` (the code as shipped) | **39 passed, 0 failed, 12 skipped, 2.5 min** |
+
+ DS's 38 plus the new case. The 12 skips are `channels-rack-audit`, which needs `E2E_RACK_SHOTS`.
+ Neither run waited in the queue. Not re-run after the review, as the parent directed: the fixes
+ change a range, a message and a unit test, and the review's own run of `storage-locations` at
+ `fd2b8d63` passed 9/9.
+- **Numbers tool:** none.
+
+#### Found and left
+
+- **Other CLI processes run on the defaults.** The `index` and `build stats` bins apply the settings;
+ `archilyzer doctor` and any other command whose inspects go through `onDrive` race them against the
+ default 3 s. **Doctor's line belongs to slice SG**, which owns `common/bin/doctor.ts` (ruled below):
+ `applyHealthTimings(settingsFromFile(paths.settingsFile).storage.health)` at its start.
+- **Two drives at the cap's maximum can still hold every thread.** Review M1 lowered the maximum to
+ 8, half of `UV_THREADPOOL_SIZE` (16), so one drive that stops answering cannot hold them all; two
+ such drives at 8 each can, as two at 4 hold half. The cap is per location, not per process.
+- **A hand edit of `passIntervalMs`** re-arms at the next pass, so it can wait up to the old interval
+ (at most 5 minutes). A save from `/storage` re-arms at once.
+- **A refused save clears the typed value.** React resets a form after its action returns, so the
+ field shows the stored value again; the error sentence names the refused value.
+- **umtool's reachability twin** (`checkChannelReachable`) still has no health state (DS left it;
+ `umtool/**` is another slice's).
+
+#### Decisions the operator could overturn
+
+| What I assumed | The alternative |
+|---|---|
+| The counters' spacing is the ruling's min(10 s, interval − 5 s), **floored at half the interval**. Without the floor it is 0 at the 5 s minimum interval (1 s at 6 s), so a Refresh just after a pass would compare two samples milliseconds apart, the case DS's review kept the spacing for. From 10 s up the two agree. **Ruled at review: the floor stays.** | The formula as ruled, 0 at 5 s. Or a higher minimum interval (10 s) |
+| A read clamps an out-of-range value (the schema's convention); the `/storage` form refuses it with a sentence and writes nothing. | The form clamps too, and says what it stored |
+| Only a value that differs from its default is written, so a save of 3000 for the budget writes nothing, and a later release's new default reaches it. | Write what the operator saved, pinning the default of the day |
+| The slice's test as specified (`budgetMs: 200` → a 300 ms unit refused) is below the ruled 500 ms floor, so the test stores 200, shows it clamped to 500, and refuses a 700 ms unit; the default passes the same unit. | Lower the floor so 200 applies |
+| The timings are applied into the health state (the pass on every pass, the save at once, the two bins), not read from `settings.json` on each call: lib has no settings memo, and `onDrive` is on every page's hot path. | A time-limited memo of `getSettings()` inside the accessor (a file read at most every few seconds, and lib/storageHealth.ts no longer free of I/O) |
+| A save re-arms the pass's timer at once, through a subscription on `globalThis`. | Leave the running timer; the new interval at the next restart |
+| A lowered cap is reached as calls return; calls already in flight are not refused. A raised cap admits waiting calls at once. **Ruled at review: it stays.** | Refuse the calls over the new cap |
+| **After review M1:** the cap's maximum is 8, half the editor's 16 file-access threads (the ruling said 16). | 16, with a warning on the form and in `SETTINGS.md` (as first shipped) |
+| The form's inputs are text with a numeric keypad, so the action's sentence is the only validation. | `type="number"` with `min`/`max`: the browser's own bubble, and "3.5" blocked before the action |
+| `resetStorageHealth` (a test seam and the e2e `invalidate-cache` route) keeps the applied timings. | Reset them to the defaults until the next pass |
+
+**What runs which code, for the rollout.** The accessor, the pass's apply and re-arm, and the form
+are in the editor's built bundle and its instrumentation, so all of it takes effect after the editor
+is rebuilt and restarted. Nothing is written until the operator saves the form; with no `health` key
+the editor runs on today's numbers. A CLI `archilyzer index` or `build stats` reads the settings
+itself.
+
+#### Rulings (parent, 2026-09-30)
+
+| Question | Ruling | Where |
+|---|---|---|
+| The counters' spacing: the ruled min(10 s, interval − 5 s) is 0 at a 5 s interval | The half-interval floor stays | as built (`13411edc`) |
+| A lowered cap: refuse the calls in flight over it, or reach it as they return | Reached as calls return; nothing in flight is refused | as built (`13411edc`) |
+| `archilyzer doctor` runs on the default timings | Its one `applyHealthTimings` line goes to slice SG, which owns `common/bin/doctor.ts` | "Found and left" |
+
+#### Review
+
+**Verdict: SHIP AFTER FIXES** (`dt-review.md` in the job's scratch). No High. The review re-ran every
+gate (tsc, common 2,318, editor unit 100, the three docs checks, `storage-locations` 9/9, a clean
+`merge-tree` against `main`), checked the writer paths, the migration, the accessor's `globalThis`
+home, the re-arm, the live cap's arithmetic and 20 FACTS anchors, and found no accessible-name
+collision in the three specs that open `/storage`.
+
+| Finding | Ruling | Where |
+|---|---|---|
+| M1: `inFlightPerLocation` could be 16, the editor's whole thread pool, so one drive that stops answering could hold every thread; the hint only warned | The maximum is 8; the hint, the docs string, `SETTINGS.md`, the test values and these records | `85c2407b`, this commit |
+| L1: the parent's rulings were not in the record | The block above; the doctor bullet names SG | this commit |
+| L2: no test pinned that a location write keeps `storage.health` | One `mergeSettingsPatch` case | `e7a525f3` |
+| L3: a refused save clears the typed value (React's form reset) | As recorded in "Found and left" | as built |
+| L4: the no-location timeout's words read the budget at throw time, not the one the call ran against | The detail carries `secondsText(budget)`; a test that changes the budget mid-call | `6a24ac4e` |
+
+### Slice SG, as shipped — the source's history and diffs, rendered by stagit (2026-09-30)
+
+Branch `r15/stagit` off `main` `6b8aa450`, worktree `~/Projects/r12-source-mirror` (block #13:
+editor 4301, test 4311, export 4310, homepage e2e 4340), one Opus implementer. Scratch files
+`sg-*` in the job's `tmp`. `main` did not move during the slice. The ruling (2026-09-30): visitors
+read the commit history and each commit's diff on the homepage; the tool is **stagit**; its
+per-file pages are dropped, and browsing stays the raw tree at `/source/tree/`. No vendor file:
+stagit is operator-installed, like git-filter-repo.
+
+**What shipped.**
+- **Where it runs** — `source.ts` step 12b, after the object audit and the mirror's staging, before
+ the manifest and the file audit. `stagit -c <cache> -u https://archilyzer.pages.dev/source/git/
+ <the scrubbed clone>`, in the render's directory. The scratch clone is now named
+ `archilyzer.git` (stagit names the repository after its directory; it was `bare`). Its
+ `description` is "Archilyzer" (no newline: stagit keeps it) and its `url` is the clone URL. Neither
+ file is published: the mirror is staged from its allowlist.
+- **What is published** — an allowlist of stagit's output: `log.html`, `files.html`, `refs.html`,
+ `atom.xml`, `tags.xml` and `commit/<40 hex>.html`, plus `style.css`. stagit writes `file/`
+ (HEAD's whole tree) on every run; it is deleted where it was rendered and never staged.
+- **The post-pass** (`rewriteHistoryPage`, pure, unit-tested; pages only, never the two feeds):
+ - the homepage's pre-paint theme script before `</head>`. It is the string `ThemeScript` emits:
+ `buildThemeScript({ defaultBase: HOMEPAGE_DEFAULT_BASE })`;
+ - one line after `<body>`: `Archilyzer · Source`, linking to `/source/`;
+ - `href="(../)*file/<path>.html"` → `href="$1../tree/<path>"`. This covers the Files index, the
+ header's README and LICENSE, and both sides of every diff header;
+ - stagit's `logo.png` and `favicon.png` → the site's `/icons/icon-32.png`. Without it, every page
+ has a broken image in its header.
+
+ The pages are rewritten byte for byte: latin1 in and out, and every injection is ASCII.
+- **The theme config moved to `lib/`.** The publish layer may not import `components/`
+ (`architecture.test.ts`), and the script lived in `components/themeConfig.ts`. That file now
+ re-exports `lib/themeConfig.ts`, so its importers are unchanged. `HOMEPAGE_DEFAULT_BASE` ("dark")
+ is new; the homepage layout passes it to `ThemeScript` and `ThemeProvider`.
+- **`style.css`** — `historyStylesheet(tokens.css)`, stagit's own rules recoloured:
+ - the light block, and the dark one for `html[data-base="dark"]`;
+ - the dark one also for `:root:not([data-base])` under `prefers-color-scheme: dark`, the no-JS
+ case;
+ - links in `--brand` (Signal), diff insertions in `--success`, deletions in `--destructive`;
+ - the clone line wraps at phone width.
+
+ Only the tokens the rules read are copied, with what they reach through `var()`.
+- **The gate covers it.** The pages are in the stage the file sweep reads. A denied literal planted in
+ a fake stagit's output refuses the publish, named as `file
+ source/git/commit/<sha>.html (contents, byte N)` and never by its bytes. The refusal withdraws the
+ publish and removes the render cache.
+- **The manifest and the key.**
+ - The manifest gains `history: {href, commits, total, head, files, bytes, sha256, tool}`
+ (`total` since ruling 1, below: `commits` is how many have a page).
+ `sha256` is `historyDigest`, the content digest's walk over `source/git/**`; `tool` is `stagit
+ (sha256 <12>)`, since stagit has no version flag.
+ - `sourceDigest` walks `source/git` too.
+ - The state gains `stagit` and `history: {files, digest} | null`. The skip needs both, and a stagit
+ present with no history never skips.
+ - `publishedSourceProblem` names history pages that are not the audited ones: "homepage/out's
+ history pages (/source/git/) are not the ones that were audited".
+ - `SOURCE_STEP_VERSION` is 4.
+- **The render cache** — ~~`<ARCHILYZER_SOURCE_SCRATCH>/archilyzer-source-history/`~~, since ruling 2
+ `${XDG_CACHE_HOME:-~/.cache}/archilyzer/source-history/` (below): the `-c` file, stagit's output, a
+ key.
+ - It is used only when the key matches (the rules hash, filter-repo, stagit, the header text),
+ the cached commit is an ancestor of the head, and the last run finished (the key is removed
+ before a render).
+ - One holder at a time, by a pid lock. A live holder means render without the cache; a dead one
+ means empty it and take it.
+ - A cached render whose page count is not the commit count renders every page again, once.
+ - `--force` renders afresh, `--check` renders in its own scratch, and a refusal removes the cache.
+- **Without stagit** there is one line: `[source] stagit not found (not on PATH, not in ~/.local/bin)
+ — publishing without the history pages (/source/git/); install it once: git clone
+ git://git.codemadness.org/stagit && make -C stagit && cp stagit/stagit ~/.local/bin/`. The manifest
+ then has no `history`, and `/source/` shows no History links.
+ - A render that fails, a timeout (600 s), unreadable tokens, or pages over the step's limits (15,000
+ files in all, 24 MiB a file) are a `[source] WARNING` and no history. The build never fails over
+ it.
+ - `STAGIT_BIN` (a `paths` variable, `getPaths().stagitBin`) overrides the lookup: stagit on PATH,
+ then `~/.local/bin/stagit`.
+ - `archilyzer doctor` prints `stagit <path>` right after filter-repo; a `STAGIT_BIN` that names
+ nothing is a warn.
+- **`/source/`'s History block**: the commit count, the newest commit (short, linking to its page),
+ and Log · Refs · Atom feed. It shows only when the manifest has a history and `source/git/log.html`
+ is beside it: the loader drops an orphaned block, and a malformed one untrusts the manifest.
+- **Docs.** PUBLISH.md's source-mirror section gains the history. There is no `operate.md` mention
+ of `/source/`; Install and the FAQ each gain half a sentence.
+
+**stagit, installed.** Cloned `git://git.codemadness.org/stagit` into `$T/sg-stagit/src` at
+**`519cae2fcab6f2427dc8f2805d6a5d1eee4bdb05`** ("bump version to 1.3", 2026-09-14). `make` built it
+against libgit2 1.9.7. The binary was copied to `~/.local/bin/stagit` (sha256 `898752011b07…`).
+
+| Commit | What |
+|---|---|
+| `b482f5fc` | `common:` the theme config moves to `lib/`; `components/themeConfig.ts` re-exports it; `HOMEPAGE_DEFAULT_BASE`, used by the homepage layout. |
+| `3cdf4773` | `common:` `sourceHistory.ts` + 11 tests; `source.ts` step 12b, the key, the digest, the deploy check, the withdrawal + 4 tests (and the old ones: `stagit: null`, the `archilyzer.git` scratch name, step ≥ 4); the manifest's `history` + 2 tests; `STAGIT_BIN`; `ENVIRONMENT.md`. |
+| `8394f4ca` | `common:` doctor's stagit line (its test extended). |
+| `17965d58` | `homepage:` the History block; the loader; `E2E_SOURCE_PUBLIC_DIR` (declared, `ENVIRONMENT.md`); `fixture-source.ts`, `source-history.spec.ts` (2), `source.spec.ts` (+1); +2 unit tests. |
+| `cbe0ac3a` | `homepage:` the head as a short link to its page; the clone line wraps. |
+| `3dd5238d` | `docs:` PUBLISH.md, Install, FAQ; the editor and homepage `[Unreleased]` bullets. |
+| this commit | `plans:` this record, the SG row, FACTS "The source's history (stagit)". |
+
+#### Gates (at `3dd5238d`; logs `$T/sg-*.log`)
+
+- **tsc** clean before every commit (`sg-tsc-{1..4}.log`, each `exit=0`).
+- **common 2,318/2,318** (2,301 at `main`, +17: `sourceHistory` 11, `source` +4, `sourceManifest`
+ +2; doctor's test extended), 0 skipped, 77 s. The filter-repo round trips and the real-stagit
+ test RAN.
+- **homepage unit 22/22** (+2). **editor unit 95/95.** **test:scripts 195 + 1 skipped.**
+- `docs env --check` and `docs files --check` both exit 0.
+- **Builds:**
+ - editor `next build` ok, 34 s. It bundles `buildHomepage`'s lazy import; no `.nft.json` names
+ `homepage/public`, `tokens.css` or the cache.
+ - export `next build` ok, 19 s (`export/public` linked from the primary, no dangling link).
+ - homepage `next build` under `systemd-run … MemoryMax=5G` ok, 12 s.
+- **homepage e2e**, `E2E_EXPECT_SOURCE=1`, over this worktree's publish (the queue was free):
+ - `source.spec` + `source-history.spec` + `downloads.spec`: **11 passed, 0 failed, 56 s**. The
+ history test RAN its published branch (10.1 s): the no-JS light and dark, the stored light
+ choice, a commit, the Files link into the tree, the feed.
+ - The full suite: **103 passed, 0 failed, 2.3 min**.
+- **The proof** — `pnpm -s archilyzer build homepage` in the worktree, with the operator's files
+ (read by the step; never printed), `~/.local/bin` on PATH:
+ - exit 0 in 50 s: `[source] history: stagit (sha256 898752011b07) — 1872 commits, 1872 pages
+ rendered; 1878 files, 134.8 MB, the largest git/commit/7ddfc955….html 4.8 MB (8 s)`;
+ - `[source] audit clean: 23,028 objects (1,872 commits), 4,396 staged files against 10 denied
+ literals; gitleaks clean`;
+ - `[source] published main 6b8aa450ad78 as 8c924ca31701: 4396 files, 211.5 MB (mirror 3 packs,
+ tree 414 dirs, history 1872 commits in 1878 files), tarball 7.3 MB sha256 a072630c6c3a (32 s)`.
+ - **`homepage/out/source/git/`:** 1,878 files, 141,299,532 bytes; the largest
+ `commit/7ddfc955….html` at 5,017,646 bytes. `homepage/out` holds 4,576 files, against Pages'
+ 20,000 and 25 MiB.
+ - `grep -rci "$(id -un)"` gives 0 files and 0 hits; the hostname gives 0. No `file/` directory, no
+ `.git` path segment in `out/`.
+ - `publishedSourceProblem` over the worktree's `out/` is null. With one byte added to
+ `out/source/git/log.html`, it gives the history sentence; restored, it is null again.
+ - A second build: `[source] up to date at 6b8aa450ad78; skipping` (17 s).
+ - `source publish --force` gave the same mirror head and tarball sha.
+ - `STAGIT_BIN=/nonexistent/stagit source publish --check`: the one line ("STAGIT_BIN names no
+ executable"), then `check passed … 2518 files, 76.8 MB (…, no history)`.
+ - doctor: `ok stagit ~/.local/bin/stagit`, right after filter-repo.
+- **The cache, measured** on a copy of the published mirror (1,834 commits): 8.0 s for every page,
+ 2.0 s with nothing new. The real cache is 138 MB in `/tmp`, which is tmpfs here.
+- **Numbers tool:** none.
+
+#### Found and left
+
+- ~~**The file limit is months away, not years.**~~ **Ruled and applied below: the history is capped
+ at the newest 10,000 commits**, so it is at most 10,006 files; the 15,000-file drop stays as the
+ last resort.
+- **`/source/git/` has no index page**, so the directory itself is a 404; `/source/` links
+ `log.html`. A copy of `log.html` as `index.html` would double 0.6 MB.
+- ~~**No `_headers` rule for `/source/git/*`.**~~ Since the review (M1, ruled): `X-Robots-Tag:
+ noindex`, like the raw tree. The pages are HTML by extension, and stagit encodes every repository
+ string. No CSP was added, because the inline theme script would need its hash in `_headers` on
+ every change.
+- **The skip does not see a code change to the step** (the post-pass, the stylesheet). The next
+ `main` does, and every merge moves `main`. `--force` does it at once.
+- **The mirror has 1,872 commits against 1,874 on the private main.** filter-repo prunes commits
+ that become empty. This predates the slice.
+
+#### Decisions the operator could overturn
+
+| What I assumed | The alternative |
+|---|---|
+| **Ruled (2026-09-30): stands.** Diff lines use `--success` and `--destructive`, with no new tokens: the chart palette has no red (it is categorical: blue, green, violet, amber, magenta, rust); both are text-grade on both grounds; every changed line keeps its `+`/`−`, so hue is never the only signal. | Two new tokens (`--diff-add`, `--diff-del`), validated as a pair. A green/red pair cannot be told apart by hue for every colour-blind reader, so the sign column carries it either way. |
+| A failed render, a timeout, unreadable tokens or pages over the limits are a WARNING, with no history. **Ruled: the history is capped by commit count instead (below); the file-limit drop stays only as the last resort.** | Refuse the publish: louder, but it takes the mirror down with the history. |
+| **Ruled (2026-09-30): the `components/themeConfig.ts` re-export stands**; its importers move to `lib/themeConfig` when a slice next touches them. | Move the 11 importers now. |
+| stagit's logo and favicon point at `/icons/icon-32.png`. | Leave them broken, or drop the `<img>`. The ruling said "nothing else is injected"; this is a link rewrite, like `file/`. |
+| The History block's head is 12 characters, linking to its commit page. The full sha is the Mirror head fact just above. | The full 40-character sha, as text. |
+| ~~The cache lives under the scratch root, a tmpfs `/tmp` by default here (138 MB of memory).~~ **Ruled (2026-09-30): `${XDG_CACHE_HOME:-~/.cache}/archilyzer/source-history/`** — applied, below. | — |
+| `--force` empties the cache. | Keep it, and force only the publish. |
+| `--check` renders without the cache (8 s rather than 2 s): it writes nothing outside its own scratch. **Ruled: stays cache-less.** | Use the cache. |
+| The e2e shows the page with and without a history from a fixture publish, through a non-production-only `E2E_SOURCE_PUBLIC_DIR`. The real pages are walked by `source.spec` when the checkout has a history. | Only the checkout's real state, as the other source specs do. |
+
+**For the rollout** (the parent's):
+- **Install stagit on the publishing machine**, the same way (`~/.local/bin/stagit`).
+- **Rebuild and restart the editor** before any `/sites` Homepage job; its built bundle runs the
+ old step.
+- The first build publishes the source again (step 4), with the history, and makes the render
+ cache at `~/.cache/archilyzer/source-history` (about 140 MB on disk).
+- A review should read one live page after the deploy; the preview's `_headers` do not touch
+ `/source/git/`.
+
+#### The parent's rulings, applied (2026-09-30), and the merge of `main` with slice DT
+
+| Ruling | What was done |
+|---|---|
+| 1. The history is capped by commit count: stagit `-l <SOURCE_HISTORY_MAX_COMMITS>`, 10,000, the newest first; `/source/` says "the latest N of M"; the 15,000-file drop only as the last resort | `d8c9e5fe`, `4409ab9d`. **Read in `stagit.c` and measured:** `-l N` keeps the NEWEST N log lines (the revwalk is newest first) and ends "M more commits remaining, fetch the repository". But it still writes a page for EVERY commit (only the diffstat of a page that exists is skipped), and stagit refuses `-c` with `-l` (usage error). So: up to the cap, `-c` as before; past it, `-l 10000`. The pages published are, in both, exactly the commits the log links (`loggedCommits`), and every one must be there. The manifest's `history` gains `total` (`commits` is how many have a page). Measured on the published mirror (1,834 commits, pages present): `-l 10000` 8.75 s, `-c` 0.58 s. Past the cap every publish computes 10,000 diffstats (~5 ms each, ~48 s). |
+| 2. The render cache is `~/.cache/archilyzer/source-history/` (`XDG_CACHE_HOME` honoured), made with a recursive mkdir, never inside the repo; `--check` stays cache-less; PUBLISH.md and doctor ("cache: <path>, <size>") | `d8c9e5fe`, `68670703`, `2acbd080`. `getPaths().sourceHistoryCacheDir`, from `XDG_CACHE_HOME` (empty = unset) else `~/.cache` (`XDG_CACHE_HOME` declared, `ENVIRONMENT.md`). `historyCacheFor` gives none for `--check` and none — with a line — inside the checkout or the public dir. Its pages are kept by key; its `-c` log lines only when they end at an ancestor (the pages stay otherwise). Doctor's stagit line ends `; cache: <path>, <size>` ("none yet"). The old `/tmp` cache was removed. |
+| 3. `--success` / `--destructive` stand | Recorded (the decisions table). |
+| 4. The `components/themeConfig.ts` re-export stands; importers move when a slice next touches them | Recorded (the decisions table). |
+| After DT merged: merge `main`; doctor's `applyHealthTimings(...)` line (DT's Found and left) | `2b2b5305` merges `main` `4dfe21e5`; the conflicts were `release-15.md` (DT's row, then SG's; DT's section, then SG's, before the Rollout) and the editor changelog (DT's bullet, then SG's). `53c1849d`: doctor reads the settings before the corpus and applies `settings.storage.health` before it inspects any drive, as the index and stats bins do; a "drive health" line in the corpus block names the timings in force and whether they are the defaults. |
+
+| Commit | What |
+|---|---|
+| `d8c9e5fe` | `common:` the cap (`SOURCE_HISTORY_MAX_COMMITS`, `-l`, `loggedCommits`, `history.total`); the cache in the XDG cache dir (`sourceHistoryCacheDir`, `historyCacheFor`, `XDG_CACHE_HOME`); one fake stagit (`__fixtures__/fakeStagit.ts`, `-c` and `-l` as `stagit.c` has them); tests +4 (the cap twice, `loggedCommits`, the cache's place) and the cache test rewritten. |
+| `68670703` | `common:` doctor's stagit line ends with the cache's path and size. |
+| `4409ab9d` | `homepage:` "the latest N of M commits"; the fixture can be capped (+1 e2e); `source.spec` checks the log lists exactly the commits with a page. |
+| `2acbd080` | `docs:` PUBLISH.md — the cap, the cache's place, doctor's line. |
+| `44ddb6b2` | `changelog:` the cap and the cache's place in both `[Unreleased]` bullets. |
+| `2b2b5305` | Merge `main` (DT). |
+| `53c1849d` | `common:` doctor applies the drive-health timings, and prints them (+1 test). |
+| this commit | `plans:` this subsection, the ruled rows, FACTS. |
+
+**Re-gates** (after the merge, at `53c1849d`; logs `$T/sg-*-2.log`, `sg-tsc-{5,6,7}.log`, `sg-pub2.log`):
+- **tsc** clean before each commit and on the merge.
+- **common 2,341/2,341**, 0 skipped: SG's 22 over `main` `4dfe21e5`'s (2,319 by difference; `sourceHistory` 13,
+ `source` +6, `sourceManifest` +2, doctor +1).
+- **homepage unit 22/22**; **test:scripts 195 + 1 skipped**; `docs env --check` 0,
+ `docs files --check` 0.
+- **The three source specs** (`source`, `source-history`, `downloads`; `E2E_EXPECT_SOURCE=1`, over
+ the worktree's fresh publish): **12 passed, 0 failed, 17.5 s** (+1: the capped block).
+- **`source publish`** (main `4dfe21e5`) in the worktree: `[source] history: stagit (sha256
+ 898752011b07) — 1882 commits, 1882 pages rendered; 1888 files, 135.3 MB, the largest
+ git/commit/7ddfc955….html 4.8 MB`; `audit clean: 23,128 objects (1,882 commits), 4,409 staged
+ files against 10 denied literals; gitleaks clean`; published as `f81edb46f7e3`, 4,409 files. The
+ cache was made at `~/.cache/archilyzer/source-history` (138 MB, mode 700).
+- **`source publish --check`**: `check passed — would publish main 4dfe21e5714a as f81edb46f7e3: 4409
+ files, 210.3 MB (…, history 1882 commits in 1888 files) …; nothing written (32 s)`. The cache's
+ key was untouched.
+- **doctor**: `ok stagit ~/.local/bin/stagit; cache: ~/.cache/archilyzer/source-history, 134.1 MB`
+ and `-- drive health a read may take 3 s, a check every 15 s (3 s each), a stall clears on a
+ clean check twice in a row, 4 reads in flight per drive — the defaults`.
+- Not re-run after the rulings: the full homepage suite (103 before), the builds. The rulings change
+ `source.ts`, `sourceHistory.ts`, doctor and one page component; the source specs cover the page.
+
+**Found and left, from the rulings:**
+- ~~**Past the cap, the oldest published commit page links its parent's page, which is not
+ published** (a 404). Every other link resolves.~~ Corrected at review (L4): 3,327 links to files
+ main no longer has were 404s too. Since the fix, a link to what is not published is text (below).
+- **Past the cap the render is not incremental**: stagit cannot combine `-l` with `-c`, so each
+ publish computes 10,000 diffstats (about 48 s by the per-commit rate measured here). The pages
+ themselves are still kept.
+
+#### Review: SHIP AFTER FIXES, and the fixes (2026-09-30)
+
+The review (`$T/sg-review.md`, at `10ae765b`) found no High, one Medium and eight Lows; the gate needs
+no change. The parent ruled on each; L7 (the live check greps the live history pages) is the
+parent's, for the runbook.
+
+| Finding | Ruling | Where |
+|---|---|---|
+| M1: the history pages are indexable; the raw tree is `noindex` | **Ruled: `noindex`** (the alternative, indexed, was not taken) | `dd5d03ee`: `_headers` `/source/git/*` `X-Robots-Tag: noindex`; `headers.test.ts` pins it and the pages' own types; PUBLISH.md `2f02cc7e` |
+| L1: the retry line quoted stagit's words unmasked (the home dir in its argv) | Mask | `8a960a0b`: `renderHistory`'s `onLog` masks every line; the cache-location line too; a test plants a literal in the cache's path |
+| L2: a cache I/O error failed the build | One line, render without the cache | `8a960a0b`: EACCES/EROFS/ENOSPC on the dir, the lock or the key → `the render cache <dir> is unusable (<code>); rendering without it`; a key that cannot be written is no key; tests with a read-only cache dir and an unmakeable one |
+| L3: stagit's `-c` misses a `--no-ff` merge of older commits | Say so | `8a960a0b`: the retry line says it ("stagit's -c stops at the last head it rendered, and a merge of older commits falls behind it"); a test merges two commits dated 2001; PUBLISH.md and FACTS |
+| L4: 3,327 tree links were 404s; the record said every link resolves | Unlink | `8a960a0b`: a link to a file not in the staged raw tree, or to a commit with no page, keeps its text and loses its `href` (a diff header keeps its `id`, the diffstat's target); the real render has 3,342 such anchors and 32,609 tree links, **every one resolving**, and every commit link; tests with a deleted file and past the cap (the parent) |
+| L5: stale records | Fix | this commit (the cache path, `total`, the link sentence); PUBLISH.md's timings say which is stagit's and which the step's (`2f02cc7e`); `source publish`'s usage names the history (`8a960a0b`) |
+| L6: the file-count drop was untested | Test | `8a960a0b`: a `historyFileLimit` seam; the test drops nine history files with the WARNING, and publishes the rest |
+| L8: a reused pid could hold the lock forever | Stale by pid or age | `8a960a0b`: a lock whose pid is not running, or over an hour old (`HISTORY_LOCK_STALE_MS`), is replaced with one line |
+| Re-review (SHIP): a withdrawal could throw over the render cache (`dropHistoryCache` on a read-only cache) and lose the refusal's exit | Catch it | `2aa0d7c3`: one masked line ("the history's render cache <dir> could not be removed (<code>); remove it by hand"); the exit 1 and the withdrawal stand; a test refuses with a read-only cache dir (`source` 22 tests, tsc clean) |
+
+| Commit | What |
+|---|---|
+| `dd5d03ee` | `homepage:` `/source/git/*` noindex; `headers.test.ts` +1. |
+| `8a960a0b` | `common:` L1, L2, L3, L4, L5 (usage), L6, L8; `sourceHistory` +4 tests, `source` +2. |
+| `2f02cc7e` | `docs:` PUBLISH.md. |
+| this commit | `plans:` this subsection, the corrections, FACTS. |
+
+**Re-gates** (at `2f02cc7e`; logs `$T/sg-*-3.log`, `sg-tsc-8.log`, `sg-pub3.log`, `sg-capped.log`):
+- **tsc** clean before each commit.
+- **common 2,347/2,347**, 0 skipped (+6: `sourceHistory` 17, `source` 21).
+- **homepage unit 23/23** (+1); **test:scripts 195 + 1 skipped**; `docs env --check` 0, `docs files
+ --check` 0.
+- **The three source specs** (`E2E_EXPECT_SOURCE=1`, over a fresh `--force` publish): **12 passed,
+ 0 failed, 17.2 s**.
+- **`source publish --force`**: `history: … 1882 commits, 1882 pages rendered; 1888 files, 135.1 MB`;
+ `audit clean: 23,128 objects (1,882 commits), 4,411 staged files …; gitleaks clean`; published as
+ `f81edb46f7e3`. Over the rendered pages: 32,609 tree links, 0 that do not resolve; 0 commit links to
+ an unpublished page; `grep -rli` of the user name and of the hostname: 0 files each.
+- **`source publish --check`**: `check passed — would publish main 4dfe21e5714a as f81edb46f7e3: 4411
+ files, 212.5 MB (…, history 1882 commits in 1888 files) …; nothing written (32 s)`.
+- **Capped builds with the corpus linked** (`transcripts/` moved aside, `ln -sT` to the primary's,
+ restored after; `systemd-run … MemoryMax=5G`): editor `next build` **ok, 34 s**; homepage `next
+ build` **ok, 13 s**. No editor `.nft.json` names `transcripts/channels`, `homepage/public/source` or
+ the cache.
+
+
+## Rollout
+
+Release 15 is slices IG (`r15/index-hold`, merged `ccf90892`), UT (`r15/umtool-trace`, `07c991be`),
+DS (`r15/drive-stall`, `ef4f1d7c`), SS (`r15/site-scope`, `b154a9d9`), DT (`r15/drive-timings`,
+`4dfe21e5`) and SG (`r15/stagit`, `e9e3e4ec`). It rolls out together
+with release 14 (HS `69e058d6`, S1 `721ed0eb`, CF `bab894db`) and the stats recount. The operator's
+runbook is `~/reports/release-15/RUNBOOK.html`, generated by `make-runbook.py` beside it, with the
+scripts in `~/reports/release-15/scripts/` (`r15-*.sh`, derived from release 12's). Every command is
+typed from the primary checkout's root.
+
+**Done before this record (2026-09-30).** The homepage is in production: `main` `bab894db`,
+source `880936b5` (2,509 files, audit clean against the denylist), deployment `2bbb46d0`; the live
+checks all passed (clone HEAD = manifest, tarball hashes equal, 0 user/host matches over 1,834
+revisions; Gumroad link, the chart's Other band, `homepage-summary.json` version 6 with 77,547
+transcripts). A live check run within five minutes of a deploy can fail its clone-HEAD line:
+`info/refs` is served with `max-age=300`; wait and run it again.
+
+**The order.**
+1. Preconditions: the primary clean at or after `b154a9d9`; `pnpm -s archilyzer source publish
+ --check` passes; `/jobs` shows nothing running; `/storage` shows every location mounted and no
+ channel `inconsistent` or `in-transition` (the index build now HOLDS those: IG).
+2. The editor: capped build, restart (the start script sets `UV_THREADPOOL_SIZE=16`), smoke. Then
+ umtool: capped rebuild in the primary FIRST, then `pnpm run test:scripts` — its post-build trace
+ check skips a build older than the config and that run is the only proof on the real fixture (UT)
+ — then restart umtool the way it was last started.
+3. Index, then stats, with every drive mounted. The `Diff:` line must end `Held: 0`; thousands
+ removed means a drive was missing. `ARCHILYZER_INDEX_ALLOW_HELD=1` is the override, on the
+ command line for a CLI run or in the editor's environment (and a restart) for the job.
+4. The homepage again: it is owed, for the source's history (SG) — `stagit` is installed at
+ `~/.local/bin/stagit`; the publish renders `/source/git/` and the live check counts the two names
+ over the fetched history pages and requires `noindex` on them.
+5. The hub, then the six sites, each build → preview → production, with the checks the release 14
+ record lists (the name and the marked link at 360 px; S1's three; HS's member count = public
+ LISTED sites; `homepage-summary.json` version 6).
+6. Editor live checks after the restart: pick a site and walk Dashboard → Channels → Jobs →
+ Settings — the select never reads "All sites" (the first visit may flash once while the old
+ localStorage choice migrates to the cookie: SS); `/storage` shows each location's detector
+ (`counters` for a mounted drive; `stat` is the fallback) (DS).
+
+**Left for the operator, from the reviews.**
+- The drive-health timings are settings (DT): `/storage` → Drive health timing; raise `budgetMs`
+ first if the platter is read as not answering under a heavy job.
+- Does the external drive spin down when idle? A spin-up longer than 3 s completes nothing, so the
+ watchdog marks the drive "not answering" for 15–30 s after idle and job starts on it are refused
+ in that window (DS, L6).
+- A root typed by hand on a channel's Storage panel is capped at four calls in flight but is never
+ marked; naming it as a location on `/storage` gives it the full treatment (DS).
+- The editor's and the export's own build traces list dot-directories (`.env`, `.claude`,
+ `test-transcripts/.jobs`; `.export-index`) — cosmetic while `standalone` output is off; a later
+ slice (UT review).
diff --git a/plans/release-16.md b/plans/release-16.md
@@ -0,0 +1,559 @@
+# Release 16 — what a search reads
+
+`main` at `9a259787` (releases 14 and 15 merged and rolled out). No plan file of its own: each
+slice's prompt carries its ruling, and this record carries what was built. Rules:
+`plans/tools/implementer-rules.md`, with the commit trailer this release's prompts give.
+
+**The standing choices** (not re-opened):
+- **A search reads transcripts and posts unless the visitor says otherwise; live chat is read only
+ when asked.** On `main` before this release a plain query read the transcript cues only: posts
+ only through a query-builder leaf whose scope is "Posts", live chat only through one whose scope
+ is "Live chat". A plain query now reads transcripts and posts by default; the three become one
+ row of toggles. (Corrected after slice CK found the first wording, "a plain query already reads
+ … the posts corpus", untrue of the code.)
+- **The toggles say what a query reads, not which records list.** "Type: Videos, Livestreams"
+ keeps saying which records are shown; an empty query with Transcripts off still lists videos.
+- **A slice that needs another slice's file stops and says so**; it does not edit it.
+- **Nothing is edited in the primary checkout**; each slice has its own worktree, and the parent
+ merges with `git merge --no-ff` only on a clean tree.
+
+## The slices
+
+| Slice | Branch | What | Owns |
+|---|---|---|---|
+| CK | `r16/search-in` | A "Search in" row — Transcripts, Posts, Live chat — on the export and hub search, Transcripts and Posts on by default | `common/components/{FiltersPanel,SearchSessionContext,SearchResults,SearchBar,exportFilterStorage}.tsx/.ts`, `common/lib/searchQuery.ts` and `common/lib/search/*` as its prompt names, `export/e2e/search-in.spec.ts` (new) and the specs its prompt names, `export/e2e/helpers.ts`; records: `plans/FACTS.md` |
+| DX | `r16/research-setup` | The research-only setup (source → `pnpm install` → `claude mcp add archilyzer` → `/ask`) in one place, the homepage's AI and MCP doc; the sites' and the hub's Use-with-AI page removed and its links pointed at the doc; `README.md` §1/§4 and `mcp/README.md` their own copies (as amended) | `homepage/content/docs/ai-and-mcp.md`, `export/app/use-with-ai/` (removed), the Use with AI links (`export/app/components/{Header,MobileMenu,Footer}.tsx`, `export/app/(workspace)/ask/page.tsx`), `common/lib/{project,corpus}.ts` + `common/bin/compose-site.ts` (what named the page), `mcp/README.md`, `README.md` §1/§4 (wording only), `homepage/e2e/docs.spec.ts`, `export/e2e{,-hub}/use-with-ai-link.spec.ts` and the specs that visited the page |
+
+## Slice CK — the ruling (2026-09-30)
+
+- One row, **"Search in"**, beside the Type row of the Filters panel: **Transcripts** (on),
+ **Posts** (on; the Posts box moves here from the Type row, same storage key `nop`, same
+ accessible name), **Live chat** (off; offered only when the site's subs manifest reports live
+ chat, as Posts is offered only when the site ships posts).
+- The row governs every query leaf whose scope is "transcripts" (the default leaf): it reads the
+ cues of the kinds ticked. A leaf whose scope is "Live chat" or "Posts" was asked for by name and
+ is not changed by the row.
+- With all three off the Search button does nothing and the panel says which row to fix; the
+ visitor cannot commit a query that reads nothing.
+- A live-chat hit inside a video's row wears the "live chat" track badge whenever the visitor could
+ also be seeing transcript hits — the suppression at `SearchResults.tsx` for that one track ends.
+- Stored like the other toggles (the working snapshot and the profiles, `ytdlp-tb:export-filters`),
+ only when off the default. Share links do not carry the row this release (they do not carry
+ `nop` either); a `qt=` link a visitor opens reads with the visitor's own row.
+- The hub's search gets the row through the shared components; nothing hub-specific.
+
+## Slice DX — the ruling (2026-09-30, amended by the operator the same evening)
+
+- **The setup lives in ONE place: the homepage's AI and MCP doc** (`/docs/ai-and-mcp/`). It tells a
+ visitor who only wants to run Claude Code against a public instance, in one block: get the source
+ (the mirror clone or the tarball on `/downloads/`), `pnpm install`, register the MCP server **as
+ `archilyzer`** against the instance of their choice (`TRANSCRIPT_SITE_URL`; `TRANSCRIPT_HUB_URL` to
+ federate), start `claude`, try `/ask`. Windows → WSL2, one line. The optional editor lines for
+ `fetch_clip`, one sentence. The reason for the name (the shipped `/ask` and `/sweep` call
+ `mcp__archilyzer__…`) sits beside it. What that page already says about the published contract
+ (`/corpus.json`, `/llms.txt`), the MCP server and the honesty features stays.
+- **The sites' `/use-with-ai` page is removed** (export and hub). Its footer entry, its header entry
+ and the link on Ask AI point at the homepage doc instead (`PROJECT_URL` + `/docs/ai-and-mcp/`, same
+ tab), keeping the label **Use with AI**. Nothing a site said there is lost: the contract lines are on
+ the homepage doc, and a site's own `/llms.txt` still describes the site. Specs that used the page as
+ "another page" (`first-search`, `restore-no-refire`, `responsive`) use `/changelog/` instead.
+- **The READMEs are their own copies**: `README.md` §1/§4 and `mcp/README.md` say the same steps in
+ the same order and register `archilyzer`; the record notes that the homepage doc and the READMEs are
+ two places to change together (there is no shared source between markdown in `homepage/content` and
+ the repo's READMEs this release).
+- No code path other than the removed page and its links changes.
+
+## Record
+
+### Slice CK, as shipped — a search reads what the visitor ticked (2026-09-30)
+
+Branch `r16/search-in` off `main` `6c6dcd04`, worktree `~/Projects/plans-export-header-first-search`
+(editor 3401, test 3411, export 3410; the export suite's server on 3420), one Opus implementer.
+Scratch files `ck-*` in the job's `tmp`. The ruling is above ("Slice CK — the ruling").
+
+**The ruling's "today" was not the code; the slice built the ruling.** On `main` a plain query — a
+leaf of scope "transcripts" — read the transcript cues and nothing else. The Posts box in the Type
+row put the posts corpus into the global scope only when a leaf of scope "Posts" was in the tree
+(`needsPostsManifests`), and every other leaf subtracts the posts slugs (`searchEval.ts:337`), so a
+plain query never read a post. The row as ruled (Posts ticked by default, the row governs the
+transcripts leaf, the toggles say what a query reads) therefore makes a plain query read posts on a
+site that ships them: what a plain search returns there changes. Taking it back is one flag — `posts`
+false in the session's `committedSearchIn` and `draftSearchIn` — and the Posts box would then govern
+only a "Posts" leaf, as before.
+
+**What was built.**
+
+- **The row.** "Search in" beside Type in the Filters panel (`FiltersPanel.tsx`,
+ `data-testid="search-in-row"`): **Transcripts** (ticked), **Posts** (ticked; the box moved out of
+ the Type row with its key `nop`, its accessible name and its gate, `postsManifest.channels`
+ non-empty), **Live chat** (unticked; offered only when `subsManifest.liveChatTotalCount > 0`). The
+ Type row keeps Videos and Livestreams. The hub gets it through the shared components.
+- **Mechanism (a), the smaller: a rewrite of the committed tree just before it runs.**
+ `applySearchIn(root, {transcripts, posts, chat})` (`common/lib/searchQuery.ts`) turns each active
+ "transcripts" leaf into:
+ - the leaf itself, the same object, with Transcripts alone ticked (so the default row on a site
+ with neither posts nor live chat runs the very tree it was given);
+ - the leaf with the one other kind's scope, same id, same `negate`, with one other kind alone;
+ - an OR group `<id>~in` over copies `<id>~transcripts`, `~posts`, `~chat`, each with the leaf's
+ `contributeHits`, with two or three kinds. A negated leaf is NOT of the union: the OR sits inside
+ a negated one-child AND `<id>~not`, so the OR's own group state is the union whatever `negate`;
+ - with nothing ticked, the leaf unchanged (see the refusal below).
+
+ Posts are not left on the global-scope path, as the prompt proposed: that path reads posts only
+ for a "Posts" leaf, so the rewrite makes one (the posts copy) and the path then feeds it. `nop`
+ acts only here, by leaving the posts copy out; it no longer gates the global scope (review M2),
+ so a leaf whose scope is "Posts" reads posts with the box unticked, as ruled. Under a curated-tag
+ filter no posts copy is made either (`searchInUnderTags`; a post carries no tags), unless posts
+ are all the row reads — then the copy stays and reads an empty scope, and the leaf does not fall
+ back to its transcripts. A leaf of any other scope, and an empty leaf, are untouched.
+ (b) would have run up to three pipelines per leaf inside `runLeaf`, each with its own cache key,
+ controller and streaming merge; (a) touches the evaluator only in `runLeaf`'s empty-scope
+ short-circuit (review M1) and in what a cached or empty leaf reports as its progress (re-review
+ R-L1).
+- **The fold.** `foldSearchIn(progress, tree)` (`common/lib/search/searchIn.ts`, new) files the
+ copies' hits back under the visitor's leaf id (a chat hit keeps `scope: "chat"` and
+ `track: "live_chat"`, so it lands in the same section of the same video row, in time order), and
+ folds the copies' states into one: the count is the OR group's union once every copy has started
+ (before that, the evaluator reads a copy with no result as the whole scope, so the largest copy
+ stands in and the leaf shows as active), hits, processed and to-process summed, capped if any,
+ cached if all. Nothing else reads the copies' ids.
+- **The session** (`SearchSessionContext.tsx`). What counts as ticked is what the site can honour:
+ Posts only with a posts corpus, Live chat only with live chat — the panel's own conditions — so a
+ stored `lc` on a site without chat asks for no subs manifest (see "Found and left" for what a
+ "Live chat" leaf does there). The session waits for the two manifests before it decides anything
+ from the row (review L4): `SearchDataValue.manifestsSettled` — the single site's settles at each
+ manifest's first answer or first failure (a 404 subs manifest is not held for the retry), the
+ hub's is `summariesReady`, since an archive is ready only once both its manifests have settled.
+ Until then Search is not refused and the query does not run, so a stored row with Transcripts
+ unticked never reads transcripts first and then re-runs — on a single site; for the progressive
+ hub see "Found and left". The committed tree runs as `applySearchIn(committedRoot, …)` and its progress
+ is folded before `setTreeProgress`; `needsChatManifests` and `needsPostsManifests` read the
+ rewritten trees (draft and committed), so Live chat ticked loads the subs manifests and passes
+ `chatScopeSlugs` exactly as a "Live chat" leaf does (the prompt's `|| committedLiveChat`, but only
+ when there is a leaf to read it). The run and the hit-cap reset also depend on the rewritten tree's
+ hash, so a change of the row alone re-runs. `committedRoot`, its hash, `qt=`, the builder's leaves
+ and the result cards' sections stay the visitor's tree.
+- **State.** `notr` (Transcripts unticked) and `lc` (Live chat ticked) in `FilterSnapshot`, threaded
+ like `nop`: `parseSnapshot`, `snapshotsEqual`, the draft/committed pairs, `buildDraftSnapshot`,
+ `committedSnapshot`, `filtersDirty`, `promoteDraftsToCommitted`, `filterKey`, hydration,
+ `applyDraftSnapshot`, `applySnapshot`, each written only off its default. Not in `UrlParams`, not in
+ `ShareSelection`: **share links do not carry the row this release** (they do not carry `nop`
+ either). Hydration reads the row from the stored snapshot (active profile, else working) whatever
+ the URL carries, so a `qt=` link — and a share-v1 link, whose filters otherwise replace the
+ stored ones — reads with the visitor's own row.
+- **`nop` was not threaded, three ways, and is now.** Hydration set the draft `nop` to false and
+ never set the committed one, so Posts unticked came back ticked on every reload;
+ `committedSnapshot` did not write it, so a profile with Posts unticked always showed as changed
+ against what it had just loaded; `applySnapshot` did not commit it, so a loaded profile with Posts
+ unticked searched posts until the next Search.
+- **Nothing ticked.** When nothing the site offers is ticked (`draftSearchInEmpty`), Search is
+ disabled, `Apply filters` in the sheet is disabled (`FiltersContainer.tsx`, a new `applyDisabled`
+ prop), Save and Save as… are disabled (`ProfilesRow`'s `saveDisabled`, review L2), and
+ `commitSearch` (where Enter and Apply arrive) and both profile saves (which commit the draft)
+ refuse. The row's line, "Search in: pick at least one" (`data-testid="search-in-empty"`,
+ `aria-live="polite"`), is always mounted and empty unless Search is refused, so the change is
+ announced (review L1; a live region, not `role="status"`, which the first cut used and which gave
+ the page a second "status" beside the modal's — `modal-digest.spec` finds that one by role); the Search button is described (`aria-describedby`) by an always-mounted `sr-only`
+ copy of the words in the bar (`id="search-in-refusal"`), because below xl the panel is a sheet
+ that is not mounted while closed; its `title` stays for the mouse. The bar's "Press Enter or click
+ Search to apply" is withheld while Search is refused. A tree committed with nothing ticked
+ some other way (a hand-edited profile) reads its transcripts rather than matching nothing.
+- **Display.** The `hit.track !== "live_chat"` suppression in `SearchResults.tsx` is gone: every track
+ hit wears its `TrackBadge`, so a "Live chat" leaf's hits now carry the badge too (no spec asserted
+ its absence; `live-chat.spec.ts` searches nothing and was not changed). A transcripts leaf's
+ section bar is named for what it holds (`sectionScope`): "Posts" on a post's card, "Live chat" when
+ every hit in it is a chat hit, else "Transcripts".
+- **The bar.** The hint reads "Live chat available on N videos — tick Live chat under Search in." and
+ goes once Live chat is ticked. The Filters chip's count gains one for the row off its default (Live
+ chat ticked widens rather than narrows, but the panel that says so may be behind the chip); Videos
+ and Livestreams keep their own one.
+
+**The fixture.** The export e2e fixture's posts said "alpha" and "gamma", like every video's cues. With
+posts read by default, every spec searching those words for its own reasons got two post cards: a first
+full run at `2d68d9d9` (stopped at 174 of 265) had 17 failures — 7 in `ask-chat`, 5 in `ask-workspace`,
+2 in `posts-search`, 2 in `query-tree`, all post cards, and 1 in `charts` (a browse-mode stacked-bar
+case the row cannot reach; it passed in both later runs). The posts now say "kappa" and "sigma"
+(`omega` was already theirs); `posts-search.spec.ts` and `tag-chips.spec.ts` name them in their posts
+leaves, and `search-in.spec.ts` reads both corpora with a regex `alpha|kappa` in one plain leaf.
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `0edc628f` | `common:` `applySearchIn`, `SearchIn`, `SEARCH_IN_DEFAULT`, `searchInReadsNothing` in `lib/searchQuery.ts`; `foldSearchIn` in `lib/search/searchIn.ts`; tests. |
+| `41e75543` | `common:` `notr` and `lc` in the filter snapshot; `parseSnapshot` exported; the round-trip test. |
+| `2d68d9d9` | `common:` the row; the session (rewrite, fold, manifests, state, refusal, the three `nop` fixes); the bar; the badge and the section names; `applyDisabled` on the sheet's Apply. |
+| `5699a1e7` | `export(e2e):` `search-in.spec.ts`; the fixture's posts get their own words; `posts-search` and `tag-chips` follow them. |
+| `861baaa0` | `plans:` this section; FACTS "Search in"; the export changelog. |
+| `1eb089da` | Merge `main` (`0fe719b1`, the 0.11.0 cut, which renamed `[Unreleased]` while this branch added to it): the bullet goes under a fresh `[Unreleased]` above `[0.11.0]`. Changelogs only; no code moved on `main`, so no gate was re-run. |
+| `26c63ccb` | `plans:` the two rows above. |
+| `6980ac75` | `common:` review M1 — the three streaming drivers settle when no worker starts (and after a raised cap); `runLeaf` answers an empty scope at once; tests through the real drivers. |
+| `b22198d5` | `common:` review M2, L1, L2, L4 and M1's tag rule — `nop` no longer gates the global scope; `searchInUnderTags`; `manifestsSettled` on both SearchData providers and the held decision; the always-mounted refusal line and the button's description; Save and Save as… disabled. |
+| `aaf3accd` | `export(e2e):` `search-in` cases 9 (the "Posts" leaf), 10 and 11, the refusal's description and Save as…; `posts-search` for M2. |
+| `76deaac7` | `common:` the refusal line is a polite live region, not a second `role="status"` (it broke `modal-digest.spec`'s `getByRole("status")`). |
+| `1d7da806` | `plans:` the review, its rulings and gates in this section; the standing choice amended; FACTS; the changelog. |
+| `500417f3` | `common:` re-review R-L1 — a cached or empty leaf reports its progress over the scope it read (0 of 0 when empty); a real-driver test asserts the folded "searched N/M" only climbs. |
+| this commit | `plans:` the re-review in this section; "Found and left" gains R-L2, R-I2 and R-I1; the mechanism sentence; the follow-up in `STATE.md`. |
+
+**Tests** (unit)
+
+| File | What it pins |
+|---|---|
+| `lib/searchQuery.test.ts` (11, new) | Transcripts only returns the very same root; the default is Transcripts and Posts; Transcripts + Live chat is `OR(a~transcripts, a~chat)` with both mapped to `a` and `a` → `a~in`; Live chat only and Posts only swap the scope and keep the id; all three in a fixed order; nothing ticked is left alone, and `searchInReadsNothing`; leaves of every named scope and an empty leaf are untouched (same root); a negated leaf becomes a negated AND over an un-negated OR whose copies keep `contributeHits: false`; nested: an untouched subtree keeps its identity, group ids are kept, the input's `stringifyRoot` and `canonicalHash` do not change; two leaves get two sets of copies. |
+| `lib/search/searchIn.test.ts` (8, new) | Through the real `runQueryTree` with a fake leaf runner: Transcripts alone reads the cues only; Transcripts + Posts returns the video and the post, the post hit filed under the visitor's leaf with scope `posts`; Live chat ticked adds the chat-only video, its hit under the leaf with `track: "live_chat"`, the leaf's state folded (only `a` left, count 2 = the union, not active); Transcripts off + Live chat on: a cue-only word finds nothing; a "Live chat" leaf ignores the row; NOT reads NOT of the union and still counts what it matched. The fold alone: before every copy has started, the largest copy and active (not the 30,000 the evaluator reports); nothing rewritten returns the same progress. |
+| `components/exportFilterStorage.test.ts` (5, new) | `notr`, `nop`, `lc` survive a JSON round trip; absent reads as the default and a pre-row profile equals one spelling the defaults; a non-boolean is dropped; `snapshotsEqual` tells each apart; the working snapshot and a profile keep the row through `saveStoredState`/`loadStoredState`. |
+
+**e2e** (`export/e2e/search-in.spec.ts`, new, 11 cases after the review): (1) the defaults — Transcripts and Posts
+ticked, Live chat not, exactly one checkbox named "Posts" on the page, Videos and Livestreams still
+in Type, and the bar's hint, gone once Live chat is ticked; (2) a plain "kappa" returns the two posts
+under a section named "Posts", and none with Posts unticked; (3) one plain regex leaf `alpha|kappa`
+returns 3 videos and 2 posts, "5 videos, 5 hits"; (4) "message" (only in one video's chat) finds
+nothing by default, and with Live chat ticked finds that video, "1 video, 30 hits", every shown hit
+badged "live chat", the section named "Live chat", and `qt=` still the visitor's one transcripts
+leaf; (5) Transcripts off + Live chat on: "line" (only in cues) finds nothing, "message" finds the
+chat video, Transcripts back on finds all three; (6) nothing ticked: Search disabled and described
+by the words, Save as… disabled, the line shown (always mounted, empty otherwise), the "Press Enter"
+line withheld, Enter commits nothing (no `qt=`, no results), Live chat or Posts re-enables it; (7) the row survives a reload, is saved with "Save as…", a diverging Search
+leaves the profile, and loading the profile brings the row back with no unsaved-changes dot and its
+results; storage holds `notr` and `lc` and no `nop`; (8) an empty query with Transcripts off lists
+"All videos (3)"; (9) a "Live chat" leaf from `qt=` reads the chat with the row at its default, and
+a "Posts" leaf reads its two posts with Transcripts and Posts unticked; (10) a plain query under a
+tag chip finishes with its two tagged videos, and with posts alone ticked says "No matching
+videos." and finishes; (11) with Videos and Livestreams unticked a plain "kappa" reads the two posts
+and finishes. `posts-search.spec.ts`: unticking Posts leaves a "Posts" leaf's two posts.
+
+#### Gates (logs `$T/ck-*.log`)
+
+- **tsc** (all workspaces): clean before each code commit — 71 s at `2d68d9d9` (after removing a
+ truncated, stale `editor/.next/dev/types/validator.ts` left in this worktree by an earlier dev run,
+ which failed the first attempt with TS1002); export's alone after the e2e edits.
+- **Unit:**
+
+ | Suite | Result |
+ |---|---|
+ | common | **2,372/2,372**, 84 s (24 new: 11 + 8 + 5) |
+ | mcp | **271/271**, 27 s (it imports `lib/searchQuery.ts`, which only gained exports) |
+ | editor unit, `test:scripts` | not run: no editor file, script or anything they import changed |
+
+- **Build:** `pnpm --filter export exec next build` — exit 0, 35 s, `export/public` links refreshed from
+ the primary and none dangling (three that were — `hub-sites.json`, `hub-summary.json`,
+ `duplicates.json`, gone from the primary — removed first). The editor and umtool builds: not run,
+ neither imports a changed module.
+- **e2e** (detached and queued; none waited in the queue):
+
+ | Run | At | Specs | Result |
+ |---|---|---|---|
+ | 1 | `2d68d9d9` | the prompt's eight (`search-in`, `posts-search`, `live-chat`, `filter-profile-persistence`, `first-search`, `browse-all`, `query-tree`, `share-current-search`) | 50 passed, **6 failed**, 4.4 min — `search-in` 8/8; the six were post cards (above) |
+ | 2 | `2d68d9d9`, old fixture | the full export suite | stopped at 174 of 265: 157 passed, **17 failed** (above) |
+ | 3 | `2d68d9d9` + the fixture and spec edits | the full export suite | 264 passed, **1 failed**, 14.0 min — `tag-chips` named "alpha" in a posts leaf |
+ | 4 | `5699a1e7` | the eight + `tag-chips` | **72 passed**, 0 failed, 3.5 min |
+ | 5 | `5699a1e7` | the hub suite (`e2e:hub`) | **36 passed**, 0 failed, 1.6 min — its fixtures ship no posts and no live chat, so the row is a lone Transcripts box and no tree is rewritten |
+ | 6 | `5699a1e7` | the editor suite's `export-search.spec.ts` (it drives the export's filter rows) | **19 passed**, 0 failed, 1.0 min |
+ | 7 | `5699a1e7` | the full export suite | **265 passed**, 0 failed, 15.1 min |
+
+- **Numbers tool:** none.
+
+#### Found and left
+
+- **Not governed by the row:** a chart's own search series (`components/charts/useSearchSeries.ts`
+ runs its own tree), `/ask`'s retrieval (`export/app/lib/askRetrieval.ts`, its own OR of keywords —
+ though the grounding it is handed from the search page is the session's results, which the row
+ governs), and the MCP (no row).
+- **A late manifest after a failure.** The single site's manifests count as settled at their first
+ failure (so a site with no live chat, whose subs manifest 404s, is not held for the retry). A
+ manifest whose retry does answer arrives later and the row re-runs, as a late manifest always
+ did.
+- **A draft tick restarts the committed run** (review I2, left): the manifests load on the draft's
+ need or the committed one's, and the run waits on both, so ticking Posts or Live chat in the
+ draft cancels a committed run in flight, which re-runs once the manifests are in (cheaply, from
+ the layer cache). The same was already so for a draft "Live chat" leaf.
+- **Hub mode is covered by reasoning** (review I3): no hub or two-origin fixture ships posts or
+ live chat, so their suites see a lone Transcripts box and no rewrite. The rewrite never touches
+ slugs, and the posts and chat scopes are built from the merged, origin-qualified manifests.
+- **On the progressive hub, a stored row of posts only or live chat only can read as empty at
+ first** (re-review R-L2). The hub's `manifestsSettled` is `summariesReady`, true once the first
+ archive is ready, and the merged posts and subs manifests grow one ready archive at a time. If
+ that archive has neither, a stored row with Transcripts unticked reads as nothing: the refusal
+ shows, and a `qt=` link runs through the empty-row fallback, reading transcripts, then re-runs
+ when an archive with posts or chat lands. On a single site the manifests settle together, and the
+ claim holds.
+- **The global scope takes the posts when the draft needs them too** (re-review R-I2):
+ `needsPostsManifests` is the draft's need or the committed tree's. With Posts unticked and a
+ committed negated plain query (`NOT x`), a draft that grows a "Posts" leaf re-runs the committed
+ query, and its result gains every post (the plain leaf read none). Consistent with NOT; the
+ result depends on the draft. On `main` the same held with Posts ticked.
+- **"Load more results" never resumes a leaf that settled at its cap** (re-review R-I1; on `main`
+ too): a driver resolves `done` when its workers stop at the cap, `runLeaf` drops its controller,
+ and the tree's `setHitLimit` reaches running leaves only. More plain queries reach the cap now
+ that each copy has its own. A follow-up in `STATE.md`.
+- **A lone Transcripts box.** On a site with neither posts nor live chat (the hub's e2e fixture) the
+ row is one box, whose only effect unticked is to refuse Search. Shown as ruled (Transcripts is not
+ conditional).
+- **Each post twice under an explicit `(transcripts OR posts)` tree.** With Posts ticked the
+ transcripts leaf reads posts too, so a post matched by both leaves shows one section per leaf. The
+ tree says to read posts twice; the row does not merge leaves.
+- **"Matching videos (N videos, …)" counts posts as videos** — as before, for a "Posts" leaf.
+- **The builder's scope select still says "Transcripts"** for the default leaf, which now reads what
+ the row ticks; its placeholder still says "Search transcripts...". Unchanged (the labels are
+ contracts; `filter-profile-persistence` finds the input by that placeholder).
+- **A "Live chat" leaf on a site with no subs manifest at all waits for ever** (as on `main`): the
+ run waits for `subsManifestReady`, which needs a manifest that 404s. The row does not reach it,
+ since it counts Live chat only where `liveChatTotalCount > 0`.
+- **A stored `lc` on a site without live chat** is kept (the panel does not show the box, and the
+ session counts it as unticked); it applies again on a site that has chat, since profiles are per
+ browser, not per site.
+
+#### Decisions the operator could overturn
+
+| What I assumed | The alternative |
+|---|---|
+| **A plain query reads posts by default** (the ruling as written; the ruling's "today" said it already did, and it did not) | Posts in the row governs only a "Posts" leaf, and a plain query reads no posts, as on `main` (one flag) |
+| Mechanism (a), a rewrite before the run plus a fold of the progress | (b), branching inside `runLeaf` |
+| Posts are read through a posts copy of the leaf, not left on the global-scope path, which reads posts only for a "Posts" leaf | — |
+| Posts and Live chat count as ticked only where the site has them, for the rewrite and for "nothing ticked" | The stored booleans as they are, whatever the site ships |
+| The row always renders, even as a lone Transcripts box | Hide it where neither Posts nor Live chat is offered |
+| With nothing ticked, `Apply filters` and the two profile saves refuse too, and the bar's "Press Enter" line is withheld | Only the Search button |
+| A committed tree with nothing ticked (a hand-edited profile) reads its transcripts | Matches nothing |
+| The row is read from the stored snapshot whatever the URL carries, share-v1 links included | A share-v1 link resets the row to its defaults, as it resets the other filters to the link's |
+| A transcripts leaf's section is named for what it holds ("Posts", "Live chat") | Always "Transcripts" |
+| The Filters chip counts the row off its default as one, Live chat ticked included | Only unticks count, as narrowing |
+| The fixture's posts get words of their own, rather than every spec that searches "alpha" or "gamma" growing two post cards (16 failures in four specs by 174 of 265 tests) | Keep the posts' "alpha" and add the posts to each expectation |
+| The hint goes once Live chat is ticked | Always shown where live chat exists |
+| Each copy of a plain leaf has its own hit cap, so one reading two or three kinds can show up to 2–3× "Max hits" before "Load more" (as an explicit OR of leaves can); the header counts each video once | One cap shared by the copies |
+| After the review: no posts copy under a tag filter, unless posts are all the row reads | A posts copy that reads an empty scope |
+| After the review: the single site's manifests settle at their first failure | At the end of the retry (about a second more on a site with no live chat) |
+| After the review: the Search button is described by a copy of the words in the bar | Described by the panel's line (no description while the sheet is closed) |
+
+#### Review
+
+**Verdict: SHIP AFTER FIXES** (`ck-review.md` in the job's scratch). Rulings (parent, 2026-09-30):
+
+| Finding | Ruling | Where |
+|---|---|---|
+| M1: a copy with nothing to read never finished — none of the three streaming drivers finalized for zero slugs, and the rewrite makes such scopes from a plain query (a posts copy under a tag chip or a channel selection with no posts, a chat copy where no video in scope has chat, a transcripts copy when Type keeps no video), so the query read "searched N/M…" for ever | The drivers settle when idle after their first `ensureWorkers` and after a raised cap; `searchEval.runLeaf` answers an empty scope at once. No posts copy under a tag filter (unless posts are all the row reads). Tests through the real drivers; two spec cases | `6980ac75`, `b22198d5`, `aaf3accd` |
+| M2: `nop` still emptied a leaf of scope "Posts", against the ruling | `nop` no longer gates the global scope; it only leaves out the plain query's posts copy. `posts-search`, FACTS, the storage comment, this record and the changelog follow | `b22198d5`, `aaf3accd`, this commit |
+| L1: the refusal line appeared with its text, and the Search button's reason was only a `title` | The line is always mounted; the button is described by an always-mounted copy in the bar | `b22198d5`, `aaf3accd` |
+| L2: Save and Save as… stayed enabled and silently did nothing | Disabled like Search and Apply | `b22198d5`, `aaf3accd` |
+| L3: case (9) claimed a "Posts" leaf it did not check | Checked, with Transcripts and Posts unticked | `aaf3accd` |
+| L4: a stored row with Transcripts unticked read as empty until the manifests answered, and a `qt=` link could read transcripts first | The decision waits for `manifestsSettled`: no refusal, no run | `b22198d5` |
+| I1: the hit cap applies per copy | Recorded (decisions table) | this commit |
+| I2: a draft tick restarts the committed run | Left, recorded ("Found and left") | this commit |
+| I3: hub mode covered by reasoning only | Recorded ("Found and left") | this commit |
+| I4: the standing-choices paragraph said a plain query already read posts | Amended to what `main` did | this commit |
+
+The empty-scope settling also ends a hang that `main` had for a leaf asked for by name: a "Posts"
+leaf under a tag chip, or a "Live chat" leaf where no video in scope has chat, never finished
+there either (its own changelog bullet).
+
+**Tests after the review:** `lib/searchQuery.test.ts` 12 (+1: `searchInUnderTags`);
+`lib/search/searchIn.test.ts` 15 (+7: each of the transcripts, posts, chat and description drivers
+settles an empty scope and settles again after a raised cap; a plain query finishes through
+`runQueryTree` with the real drivers, with the posts scope null, an empty posts set, and an empty
+chat set, and with posts alone and none to read). All seven new cases fail on the code before the
+fix; the drivers' fix alone passes all 15, the `runLeaf` short-circuit alone passes the three
+`runQueryTree` cases.
+
+#### Gates after the review (logs `$T/ck-*.log`)
+
+- **tsc** (all workspaces): clean, 70 s before the fix commits and 42 s at `76deaac7` (common and
+ export alone before that commit).
+- **common:** **2,380/2,380**, 79 s (+8).
+- **Build:** `pnpm --filter export exec next build` — exit 0, 33 s; no dangling `export/public` link.
+- **e2e:**
+
+ | Run | At | Specs | Result |
+ |---|---|---|---|
+ | 8 | `aaf3accd` | `search-in`, `posts-search`, `query-tree`, `tag-chips`, `live-chat` | **56 passed**, 0 failed, 2.5 min |
+ | 9 | `aaf3accd` | the full export suite | stopped at 214 of 267: 213 passed, **1 failed** — `modal-digest`'s `getByRole("status")` found the always-mounted refusal line beside the modal's status; fixed in `76deaac7` |
+ | 10 | `76deaac7` | the full export suite | **267 passed**, 0 failed, 13.5 min |
+ | 11 | `76deaac7` | the hub suite | **36 passed**, 0 failed, 1.5 min |
+
+#### Re-review
+
+**Verdict: SHIP**, with three small things landed before the merge (parent's rulings, 2026-09-30):
+
+| Finding | Ruling | Where |
+|---|---|---|
+| R-L1: an empty-scope copy reported its parent scope as processed, so the folded "searched N/M" started full and then fell back (a cached copy did the same) | `applyCached` takes the size of the scope the leaf read: 0 of 0 when empty, a cached leaf's effective scope otherwise. The real-driver test with the posts scope null asserts the folded leaf's processed and fraction never fall; it fails on the previous code | `500417f3` |
+| R-L3: "(a) touches the evaluator not at all" was stale | The sentence names `runLeaf`'s empty-scope short-circuit and the progress `applyCached` reports | this commit |
+| R-L2: on the progressive hub a stored posts-only or chat-only row can read as empty until an archive with posts or chat is ready | Recorded ("Found and left"); the single-site claim is marked single-site | this commit |
+| R-I2: the global scope takes the posts when the draft needs them too | Recorded ("Found and left") | this commit |
+| R-I1: "Load more results" never resumes a leaf that settled at its cap (on `main` too) | Recorded ("Found and left") and a one-line follow-up in `STATE.md` | this commit |
+
+**Gates after the re-review:** tsc (all workspaces) clean, 48 s; common **2,380/2,380**, 84 s (the
+count is unchanged: one test extended); `pnpm --filter export exec next build` exit 0, 29 s;
+`search-in`, `query-tree` and `posts-search` at `500417f3`: **38 passed**, 0 failed, 1.9 min. The
+full export suite and the hub suite were not re-run, as the parent directed.
+
+### Slice DX, as shipped — the research-only setup, told where a visitor reads (2026-09-30)
+
+Branch `r16/research-setup` off `main` `2f09b065` (first cut at `2b767bc9`, reset on the amendment), worktree `~/Projects/homepage-social-visible`
+(block #3: editor 3301, test 3311, export 3310), one Opus implementer. Scratch files `dx-*` in the
+job's `tmp`. Built to the ruling as amended the same evening ("Slice DX — the ruling" above): the
+branch was first built to the ruling as first written (a shared `common/lib/researchSetup.ts` and the
+block on every site's page); on the amendment it was reset to `main` `2f09b065` (slice CK merged)
+and rebuilt, and of that first pass only the READMEs commit was kept (cherry-picked as `7921293f`).
+
+**What it does.**
+- **The homepage's AI and MCP doc has a "Ten-minute setup"** (`homepage/content/docs/ai-and-mcp.md`,
+ an `###` under "## The MCP server", right after its two opening paragraphs): one `sh` block —
+ `git clone https://archilyzer.pages.dev/source/archilyzer.git archilyzer # or the tarball on
+ /downloads/`, `cd archilyzer && pnpm install`, `claude mcp add archilyzer --env
+ TRANSCRIPT_SITE_URL=https://jeralyzer.pages.dev -- pnpm -C "$PWD" --filter yt-dlp-transcript-mcp
+ exec tsx src/index.ts`, `claude # then: /ask what has he said about …` — and seven notes: what you
+ need (Node.js 20.9 or newer, pnpm 9 or newer, Claude Code; no corpus, no yt-dlp, no GPU, nothing
+ hosted; the repo has no `engines` field, so the numbers are README's Requirements table), the
+ source ([git mirror](/source/), [tarball](/downloads/)), **register it as `archilyzer`** with the
+ one-sentence reason, the two optional editor lines for `fetch_clip`, `TRANSCRIPT_SITE_URL` /
+ `TRANSCRIPT_HUB_URL`, the `mcp.json` form for another client (in `mcp/README.md` on the raw
+ tree), and WSL2 on Windows (the README's "Claude Code on Windows", on the raw tree). "What it can
+ do:" became `### What it can do`, same words, so the list is not under the setup. Everything else
+ on the page is unchanged.
+- **The sites' `/use-with-ai` page is removed** (`export/app/use-with-ai/`, export and hub). Its
+ links keep the label **Use with AI** and go to `AI_DOC_URL` (`common/lib/project.ts`,
+ `${PROJECT_URL}/docs/ai-and-mcp/`), same tab, as plain anchors: the header's nav (its entry, and
+ `MobileMenu`'s, carry `external: true`), the footer's, and the one on Ask AI.
+- **What named the page names the doc:** `corpus.json`'s `useWithAi` (site and hub; the key stays),
+ `llms.txt`'s "## Ask AI" (two lines now: `[Ask AI](<base>/ask/)` for the chat and `[Use with
+ AI](<doc>)` for the MCP setup, site and hub), and the sitemap (`/use-with-ai` dropped).
+- **The READMEs are their own copies:** `mcp/README.md`'s "Add to Claude Code" and `mcp.json`
+ examples register `archilyzer` (were `rekietalyzer`) from `/ABS/PATH/TO/archilyzer`, the
+ directory the clone makes; its renaming paragraph stays and says why every example uses the name. `README.md` §1 registers with `pnpm -C "$PWD"`, "from the repo's
+ root" (was `/ABS/PATH/TO/this/repo`, against §4 and the Windows notes); §4's quickstart starts
+ from the source (`git clone …` or the tarball, then `cd archilyzer && pnpm install`). The
+ homepage's drift table (`homepage/content/README.md`) names README §1/§4 and mcp/README as copies
+ of the setup's commands, to change together; FACTS says the same.
+- **Specs.** `first-search` and `restore-no-refire` left the workspace through the header's Use with
+ AI, a client navigation in the same page life. No site link makes one to a page outside the
+ workspace now (the footer's Changelog is a plain `<a>`, a new page life), so they push
+ `/changelog/` through `window.next.router` (FACTS) and assert the Changelog heading and no query
+ builder before Back or the header's Search; their assertions are unchanged. `responsive` checks
+ `/changelog/` in place of `/use-with-ai/`, expected to fail (below).
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `7921293f` | `docs:` mcp/README registers `archilyzer` in both examples; README §1's `"$PWD"` |
+| `8bfecf14` | `common:` `AI_DOC_URL`; `corpus.json` `useWithAi`, `llms.txt`'s Ask AI, the sitemap |
+| `e5160555` | `export:` the page removed; the header's, the menu's, the footer's and Ask AI's links; `first-search`, `restore-no-refire`, `responsive`; `use-with-ai-link.spec.ts` (site and hub) |
+| `ab8d46a5` | `homepage:` the Ten-minute setup; the drift table; `docs.spec.ts` |
+| `217d398f` | `docs:` README §4's quickstart starts from the source |
+| `ae206b01` | `export(e2e)`, `homepage(e2e)`: `/changelog/`'s overflow marked `test.fail`; `docs.spec`'s locator |
+| `6027b0dc` | `docs:` mcp/README's two examples name the checkout `/ABS/PATH/TO/archilyzer` |
+| `20605b5f` | `plans:` this section; FACTS; the export and homepage changelogs |
+| `1fcb224d` | `common:` review L3 — `corpus.test` pins `useWithAi` (site and hub) and llms.txt's two Ask AI lines |
+| `cadc595e` | `homepage:` review L2 — the doc names `fetch_clip` as the one exception to "it only reads" |
+| this commit | `plans:` review L1 (FACTS), L4 and the slices table's DX row; the review's findings; STATE's two follow-ups |
+
+#### Gates (logs `$T/dx-*.log`)
+
+- **tsc** (all workspaces) clean, 70 s — after deleting the worktree's `export/.next/dev/types`,
+ left by the first pass's e2e dev server: its `validator.ts` imported the removed page (the build
+ regenerates `.next/types`; the dev copy waits for a dev server). Export and homepage alone again
+ after the spec fixes: clean.
+- **common:** 2,380/2,380, 129 s (the count unchanged: `corpus.test.ts`'s sitemap case takes
+ `/changelog` as its sample route). **`test:scripts`:** 194 passed, 2 skipped (the umtool post-build
+ check, whose worktree build predates its code; `LIVE`). **homepage unit:** 23/23.
+- **Builds:** `pnpm --filter export exec next build` exit 0, 33 s; the hub (`INSTANCE_MODE=hub`, the
+ same command) exit 0, 31 s. In each `out/`: no `use-with-ai/`; every "Use with AI" in the HTML
+ (the header's, the footer's, Ask AI's) has `href="https://archilyzer.pages.dev/docs/ai-and-mcp/"`;
+ no `href="/use-with-ai`. The homepage (`next build`, capped at 5 GB) exit 0, 22 s; the doc carries
+ the block. `eslint` on the touched export files: clean.
+- **e2e:** see the table below.
+- **Numbers tool** (`plans/tools/compose-fixture-one-youtube-channel`, composed into scratch): this
+ slice's differences are exactly `corpus.json`'s `useWithAi`, `llms.txt`'s two Ask AI lines and
+ `sitemap.xml`'s `/use-with-ai` entry. The committed fixture had already drifted from `main` (`spec`
+ 3 → 4, four `_headers` entries, `index/sites/testsite/tag-counts.json`), so it was not updated.
+- **Privacy:** 0 matches of the operator's user name and 0 of the host name in every changed file,
+ but `plans/FACTS.md`'s user-name count of 3, which is `main`'s (0 in the lines this slice adds).
+
+ | Run | At | Specs | Result |
+ |---|---|---|---|
+ | 1 | `217d398f` | homepage `docs.spec.ts` | 4 passed, **1 failed** (`.doc-measure` matched three elements; fixed in `ae206b01`), 19 s |
+ | 2 | `217d398f` | export `use-with-ai-link`, `first-search`, `restore-no-refire`, `responsive`, `header` | 61 passed, **1 failed** (`/changelog/` overflows; marked in `ae206b01`), 4.9 min |
+ | 3 | `ae206b01` | homepage `docs.spec.ts` | **5 passed**, 0 failed, 19 s |
+ | 4 | `ae206b01` | the full export suite | **271 passed**, 0 failed, 15.1 min (`main`'s 267 + 4; `/changelog/`'s overflow case fails as marked, which the list prints as ✘ and counts as passed) |
+ | 5 | `ae206b01` | the hub suite | **39 passed**, 0 failed, 1.6 min (`main`'s 36 + 3) |
+
+#### Found and left
+
+- **`/changelog/` overflows a 390 px phone by about 600 px** (the new `responsive` case; it was not
+ checked before). Long inline `code` in released entries (`export/app/ask/{MessageBubble,…}.tsx/ts`,
+ a JSON literal) has no break opportunity. The case is `test.fail`, so a fix turns it red. Fixing it
+ is a change to the changelog's rendering, outside this ruling.
+- **A multi-line JSX text holding an HTML entity loses its leading space** under Next's SWC (FACTS):
+ 22 texts in 19 files run a word into the element before it — 1 in `homepage/app/downloads/page.tsx`,
+ 21 across `editor/app/**`; none in `export/app` or `common/components`. Found on the first pass's
+ page, which is gone. Not fixed.
+- **The compose fixture under `plans/tools/` is behind `main`** (above). Refreshing it is its own
+ commit.
+- `PUBLISH.md`'s registration (`-- pnpm -C "$PWD" archilyzer mcp`) and `AGENTS.md`'s (with the
+ editor lines) register `archilyzer` and were left as they are; `export/CHANGELOG.md`'s released
+ entry for the page is history.
+- A site deployed before its rebuild still serves `/use-with-ai/` and its old `corpus.json`; the
+ rebuild removes both together.
+
+#### Decisions the operator could overturn
+
+| What I did | The alternative |
+|---|---|
+| `AI_DOC_URL` in `common/lib/project.ts`, beside `INSTANCES_URL` | `${PROJECT_URL}/docs/ai-and-mcp/` written at each of the six places |
+| The header's and the menu's entries render a plain `<a>` (`external: true`) | `next/link` with the absolute URL, which also renders an `<a>` and does not client-navigate |
+| `corpus.json` keeps `useWithAi`, now the doc's URL | Drop the key (a contract change) |
+| `llms.txt`'s Ask AI section: the site's `/ask/` chat and the doc, two lines | One line, to the doc |
+| The sitemap drops `/use-with-ai` and adds nothing | Add `/ask/` |
+| `first-search` and `restore-no-refire` reach `/changelog/` through `window.next.router.push` | Click the footer's Changelog: a hard navigation, a new page life, which is not what those tests prove |
+| `responsive` checks `/changelog/` with `test.fail` | Drop the route; or fix the changelog's wrapping here |
+| "What it can do:" is `### What it can do`, same words | The setup at the end of "## The MCP server", after the list |
+| The doc's notes include where the `mcp.json` form is (the removed page carried it) | Leave it out |
+| README §4's quickstart gains the clone step, with the tarball's host in its comment | Leave §4 starting at `pnpm install` |
+| The doc's `/ask what has he said about …` (README §4's words) | "they", for any archive |
+
+#### Review
+
+**Verdict: SHIP AFTER FIXES** (`dx-review.md` in the job's scratch): four lows, no High or Medium.
+
+| Finding | Where |
+|---|---|
+| L1: FACTS said README §1, §4 and mcp/README "say the same steps" | This commit: README §4 carries the doc's whole sequence in the same order; README §1 and mcp/README share only the registration command (§1 with `"$PWD"`, mcp/README with `/ABS/PATH/TO/archilyzer`) |
+| L2: the doc brought in `fetch_clip` beside "it only reads already-published static JSON" | `cadc595e`: "The one exception is `fetch_clip`, which asks a local Archilyzer editor for a clip's media: the editor writes it, and the server itself never writes." |
+| L3: nothing pinned the composer changes | `1fcb224d`: `useWithAi === AI_DOC_URL` for site and hub; the two "## Ask AI" lines of each llms.txt; no `use-with-ai` in either |
+| L4: the record's base | This commit: off `main` `2f09b065`, first cut at `2b767bc9` |
+| I1: `window.next.router` is undocumented | This commit: FACTS says so, and to check it first on a Next upgrade |
+| I2: `test.fail` is satisfied by any failure, not only the overflow | Left as ruled; STATE's follow-up drops the line once the changelog wraps |
+| I3: the slices table's DX row described the first ruling | This commit: the row states the amended scope |
+| I4: the primary's `export/.next/{,dev/}types` still name the removed page | The parent's, before the post-merge tsc |
+| I5–I8 | No action here (deploy order: the homepage with or before the sites and hub; README's Windows blocks' `<your-remote>` a follow-up) |
+
+**Follow-ups in `STATE.md`:** `/changelog/`'s overflow (wrap `<code>` in the changelog renderer, then
+drop the `test.fail`); the JSX entity/whitespace sweep (22 texts in 19 files, FACTS).
+
+**Gates after the review** (logs `$T/dx-regate.log`, `$T/dx-e2e4.log`): tsc (all workspaces) clean,
+125 s; common **2,381/2,381** (+1), 166 s; mcp **271/271**, 59 s; homepage `docs.spec.ts` **5
+passed**, 19 s; export `use-with-ai-link.spec.ts` **4 passed**, 12 s; the hub's **3 passed**, 10 s.
+The full suites were not re-run, as the parent directed.
+
+## Rollout
+
+Both slices are export- and homepage-side; the editor and umtool are not rebuilt for this release.
+
+1. **Preconditions:** the primary clean on `main` at or after `1064d04c` (DX's merge); the source
+ gate passes (`archilyzer source publish --check`).
+2. **Homepage:** `archilyzer build homepage` (the source step re-audits the 1,937 commits and
+ renders the 32 new history pages), preview, the live check (which also asserts the Ten-minute
+ setup on `/docs/ai-and-mcp/` and the `@Archilyzer` link), production, the live check again after
+ the CDN's five minutes. First, so the doc every site links to answers.
+3. **Hub**, then the **six sites**, each built, previewed and checked: the site check asserts the
+ "Search in" row's markup is absent from the static shell (S1 still holds), the "Use with AI"
+ link carries `https://archilyzer.pages.dev/docs/ai-and-mcp/` and no `/use-with-ai` link
+ remains, and the X link is `x.com/Archilyzer`.
+4. **By eye on one site with posts (Jeralyzer):** a plain search returns post cards beside videos
+ (ruled: Posts on by default); untick Posts → none; tick Live chat on a video with chat → the
+ hits wear "live chat"; untick all three → Search disabled with the row named.
+5. Records: this section, `STATE.md`, memory.
+
+Done 2026-09-30 evening by the session up to production: homepage built (`main 1064d04c` → mirror
+`19f7ea3b`, audit clean), preview checked 20/20; the hub and six sites built, previewed and checked.
+The production deploys are the operator's (`~/reports/release-15/RUNBOOK.html`, banner).
+
diff --git a/plans/source-mirror.md b/plans/source-mirror.md
@@ -0,0 +1,516 @@
+# Release 12 — the source mirror: a read-only git clone on the project site
+
+Written 2026-09-27 in plan mode, RE-CHECKED 2026-09-28 against `main` `90c7f776` (128 commits later;
+see "What changed since the plan was written"). Self-contained for a fresh session. The design was
+verified by three Explore passes and one Plan pass; the dumb-HTTP clone, the filter-repo scrub and
+wrangler's upload filter were PROVED in scratch. Rules: `plans/tools/implementer-rules.md` (one Opus
+implementer per slice in a worktree, one read-only Opus review, the parent merges). Two slices, Q
+then R. Record file: `plans/release-12.md` (NEW, the shape of `plans/release-11.md`). Memory to read
+first: `release-11-overnight`, `fetch-clip-shipped`, and the audit at
+`~/reports/repo-mirror-audit/AUDIT.html`.
+
+## What changed since the plan was written (re-check 2026-09-28)
+
+- **"Release 11" is taken**: the overnight of 2026-09-28 merged slices O1–O6 (+ O2b, O1c, O6c) to
+ `main`, NOT rolled out (owed by the operator: cut, :3001 restart, a settings save, the deploys;
+ runbook `~/reports/overnight-2026-09-28/MORNING.html`). This work is **release 12**, slices Q and R.
+- `DEPLOY_CLOUDFLARE.md` is GONE, absorbed into **`PUBLISH.md`** (`## Cloudflare Pages` :70,
+ `## Download archives and R2` :137). Every doc edit this plan aimed at it goes to `PUBLISH.md`.
+- **Env vars are a registry**: `common/lib/envVars.ts` declares every variable the code reads, a
+ test fails on an undeclared read, `ENVIRONMENT.md` is GENERATED (`pnpm archilyzer docs env`,
+ `--check` in the gates), and path overrides are read by `getPaths()` (`common/lib/paths.ts`) and
+ nowhere else. Test-only variables carry an `E2E_` prefix and are declared in the playwright config.
+- **`archilyzer doctor`** exists (`common/bin/doctor.ts` + test) — slice R adds its checks there.
+- **The homepage builds from `/sites` now** (release 11 O4): `editor/app/sites/lib/
+ homepageDeployActions.ts:51,95` call `buildHomepage` IN-PROCESS, and `pnpm ops build-homepage`
+ reaches them. So the source step also runs inside the editor's process and its PATH.
+- Anchors that moved: `buildHomepage` is `common/publish/build.ts:973`; `WORKTREES.md:119`;
+ `umtool/components/BuildChain.tsx:147-149`. The umtool path literals are where they were.
+- Baselines now: common **2,112**, editor unit 85, `test:scripts` **185 + 1 skip**, mcp 269,
+ homepage unit 2, homepage e2e **31** (specs: brand, docs, downloads, instance-colours, marketing,
+ no-data, stats, theme).
+- Still true: 1,985 lines with `/home/user` outside `plans/`, 20 `umtool/song/*.sh`, `/run/media/user`
+ in five `common/` files AND one line of `editor/CHANGELOG.md`; `create-archives.sh` exists and
+ nothing calls it; pipx and gitleaks installed, git-filter-repo not; `~/.config/archilyzer/` absent.
+
+## Working alongside the parallel session (operator's instruction, 2026-09-28)
+
+- **Nothing is edited in the primary checkout.** Each slice gets its own worktree from `pnpm wt
+ add`: `r12/paths-fix` → `/home/user/Projects/r12-paths-fix`, then `r12/source-mirror` →
+ `/home/user/Projects/r12-source-mirror` (branched after Q merges). Seven `r11-*` worktrees exist
+ and are the operator's to remove; do not touch them.
+- **This plan enters the repo on the slice branch**, as `plans/source-mirror.md` in Q's first
+ commit — never as a direct commit to `main`.
+- **Merging into `main`** happens in the primary with one command, only when `git status --short`
+ is empty and no merge is in progress: `git merge --no-ff r12/<slug>`. If the tree is dirty (the
+ parallel session is mid-change), wait and re-check; never stash or touch its files. Expected
+ conflicts: `editor/CHANGELOG.md` `[Unreleased]` and nothing else (the record file is new).
+- Scratch: `$CLAUDE_JOB_DIR/tmp`, files prefixed `q-` / `r-`. e2e is queued machine-wide.
+
+## Context
+
+The operator wants the repo mirrored on the project site (https://archilyzer.pages.dev, the
+`homepage/` Next.js 16.2.3 static-export app on Cloudflare Pages) as static, read-only files. The
+2026-09-26 pre-publication audit found no secret and no personal identifier in the 1,648-commit
+history except the Unix username in `/home/user` paths (a first name) — plus, found during design,
+`/run/media/user` in a few `common/` tests and two backticked `user` mentions in `plans/`. Today the
+site ships only a hand-run, stale (2026-08-12) `git archive` tarball (`create-archives.sh`, whose
+header says "no public git remote, by choice" — the policy this release reverses on the operator's
+word). Outcome: `git clone https://archilyzer.pages.dev/source/archilyzer.git` works with nothing but
+static files (git's dumb-HTTP protocol), a raw browsable tree sits at `/source/tree/`, the tarball is
+regenerated per deploy, and a gate refuses to publish anything that still carries a denied literal.
+
+## Decisions (made with the operator 2026-09-27; do not re-open)
+
+- **Mirror = `main` only**, no other refs (no tags exist). **Scrub** `/home/user → /home/user` in
+ file contents AND commit messages; **keep** `Co-Authored-By` and `
+ **`plans/` ships as-is.** The 20 one-off `umtool/song/*.sh` run-log scripts are **deleted**.
+- The private repo's history is never rewritten. The public mirror is generated by
+ `git-filter-repo` (deterministic: two runs gave identical ids) on a fresh bare clone at every
+ homepage build; its commit ids differ from the private repo's and the page says so.
+- The mirror directory is `/source/archilyzer.git/` — NEVER a path segment literally named `.git`:
+ wrangler's upload `IGNORE_LIST` (`**/.git`, `**/node_modules`, …) drops it silently, while
+ `archilyzer.git` passes (verified in wrangler 4.107 source). Extensionless files (`HEAD`,
+ `info/refs`) upload fine.
+- Operator-private inputs live OUTSIDE the repo in `${ARCHILYZER_CONFIG_DIR ?? ~/.config/archilyzer}/`:
+ `source-scrub.txt` (filter-repo `lhs==>rhs` lines; the step always prepends `${os.homedir()}==>/home/user`)
+ and `source-denylist.txt` (one literal per line, `i:` prefix = case-insensitive; every scrub lhs is
+ implicitly denied). Missing file → refusal naming the path. Nothing with the username is in code.
+- The step is one CLI verb, `archilyzer source publish`, and `buildHomepage` runs it between
+ `composeHomepage` and `next build` (`common/publish/build.ts:973-985`), so `archilyzer build
+ homepage`, the runbooks' home scripts and the `/sites` homepage jobs all produce it. It
+ writes only under gitignored `homepage/public/{source,downloads}` with the link-safe writers in
+ `common/bin/_publicFile.ts` (`ownDir`, `writePublicFile`, `copyPublicFile`).
+
+## Verified facts the implementer must not re-derive
+
+- `git clone --no-local --bare --single-branch --branch main` is load-bearing: a same-host
+ `--mirror` copies ~52 MB of loose garbage (`.git/objects/ac/tmp_obj_*`). After filter-repo the
+ pack grows 13.1 → 21.5 MB; `git repack -a -d --max-pack-size=20m` splits it (a 21-pack layout at
+ 2 MB each cloned fine over `python3 -m http.server`). Pages caps: 25 MiB per file, 20,000 files
+ per deployment (`src/pages/constants.ts`); the step refuses at 24 MiB / 15,000.
+- `--replace-text` rewrites blobs only; one commit MESSAGE carries the path → also
+ `--replace-message` with the same file. filter-repo is not installed; `pipx run
+ --spec git-filter-repo==2.47.0 git-filter-repo` works without sudo (pipx 1.15.0). `git 2.55.0`.
+ gitleaks is installed and clean on the primary (1,541 commits, "no leaks found").
+- wrangler's mime map serves `.ts` as `video/mp2t`, `.mjs` as `application/javascript`, `.md` as
+ `text/markdown` → `_headers` must override to `text/plain` under `/source/tree/*` (a NEW pattern
+ here; today `homepage/public/_headers` only sets Cache-Control on `/downloads/*`). Only the ROOT
+ `_headers` is parsed; copies inside the tree are served as text.
+- `main` today: 1,556 commits, 2,017 tracked files (25.9 MB), 333 directories, 123 paths with
+ `[`/`]` (Next dynamic-route dirs + two variable fonts) → hrefs must be URL-encoded; 0 spaces,
+ 0 non-ASCII, 0 symlinks, 0 tracked `.html`; largest blob 1.57 MB; largest dir 204 entries.
+ Expected public file count ≈ 2,360.
+- Next static export copies `public/` into `out/` verbatim; `next dev` does NOT serve
+ `public/<dir>/index.html` at `/<dir>/` (Pages does) — e2e requests `index.html` explicitly, and
+ NO `public/source/index.html` may exist (it would collide with the `/source/` app route).
+- **Tailwind v4 scans every file `.gitignore` does not exclude** and pack files are binary (the
+ umtool `.next-*` incident, `.gitignore:151-157`); `homepage/tsconfig.json` includes `**/*.ts`.
+ So `/homepage/public/source` goes into `.gitignore` and `"public"` into the tsconfig `exclude`
+ in R's FIRST commit, before any build with a mirror under `public/`.
+- `umtool/lib/paths.mjs` and `umtool/lib/projects/report.mjs` are imported by the umtool Next app
+ (11 API routes), where Turbopack rewrites `import.meta.url` into `.next/server/chunks`
+ (`umtool/lib/paths.ts:36-43`, the `SONG_CODE` rule): use a cwd walk to `pnpm-workspace.yaml`, not
+ the `cues.mjs:55-64` `import.meta` pattern (CLI-only).
+- `umtool/song/um-manifest.json`'s 1,896 `vid` values are used only IN MEMORY by
+ `umtool/song/build-um.mjs:375,624,643` for an untracked review HTML; nothing reads them back from
+ the JSON (`umtool/lib/clips.ts:16` has no `vid`). `thumb-manifest.json`/`thumb-accepted.json` `out`
+ IS read (`umtool/lib/thumbs.ts:165-203` via `resolveInRoots`, `umtool/lib/paths.mjs:120-128`, which
+ binds a relative path to the FIRST root, `SONG_REPORTS`); `bg` is write-only and lives under
+ `SONG_DATA`. The live umtool on :3050 is started with only `UMTOOL_PORT`
+ (`~/reports/release-10/scripts/r10-umtool.sh:45`) and its data is at
+ `/home/user/.claude/jobs/efbe67a7/tmp/song` — a changed `SONG_DATA` default needs an operator
+ symlink before the next umtool restart.
+- Worktrees do NOT symlink `homepage/public` entries today (only `_headers` is tracked there);
+ `ownDir` covers both cases anyway. filter-repo leaves `<bare>/filter-repo/{commit-map,ref-map}`
+ with the PRIVATE shas and `config` holds `remote.origin.url` = the local path → publish from an
+ ALLOWLIST of files, never the bare dir whole.
+
+## Slice Q — fix-forward the hardcoded paths (branch `r12/paths-fix`, lands first)
+
+Owns `umtool/**`, `WORKTREES.md`, `common/controller/storageLocations{,.test}.ts`,
+`common/lib/storageVolumes{,.test}.ts`, `common/views/storage.test.ts`, `common/lib/envVars.ts`
+(only the defaults of the umtool variables it changes) + the regenerated `ENVIRONMENT.md`; shared:
+`editor/CHANGELOG.md`, `plans/release-12.md` (created by Q with the release-11 shape),
+`plans/source-mirror.md` (this plan, Q's first commit).
+
+1. `umtool/lib/paths.mjs:86`: add `export const REPO_ROOT = findRepoRoot(process.cwd())` (walk up
+ until `pnpm-workspace.yaml` exists; fall back to `path.resolve(cwd, "..")`; comment why cwd, not
+ `import.meta.url`), and `CHANNELS_DIR = path.resolve(process.env.CHANNELS_DIR ??
+ (process.env.TRANSCRIPTS_DIR ? join(TRANSCRIPTS_DIR, "channels") : join(REPO_ROOT, "transcripts",
+ "channels")))`. `umtool/lib/projects/report.mjs:39-41`: `GLOBAL_CHANNELS_DIR = () => CHANNELS_DIR`
+ imported from `../paths.mjs` (one definition). Leave `cues.mjs` alone.
+2. `umtool/song/spec.mjs:31`, `umtool/song/video-dir.mjs:53`: `process.env.VIDEO_ROOT ??
+ path.join(os.homedir(), "reports", "quartering-uh-song", "videos")` (add `import os`).
+3. `umtool/song/paths.mjs:19`: `SONG_DATA` = `fs.realpathSync` of `process.env.SONG_DIR ??
+ path.join(os.homedir(), ".local", "share", "archilyzer", "song")` (realpath so `SONG_SCRATCH =
+ dirname(SONG_DATA)` keeps pointing at the real job dir through a symlink); update the header
+ comment (the symlink is the supported way to keep the data where it is).
+4. `git rm` the 20 run-log scripts (`backfill-build, mk-fatal-finish{,2,3}, mk-rebuild, ms2-{bg3,
+ drums,full,jerbg,play-rebuild,triangle}, pk2, pk3, pk-v12, pkmn-{rebuild,video}, rpg-remake-v5,
+ rpg-short, vshort, yoshi-rebuild}.sh`). Reword `umtool/lib/jobs.ts:8-17,71-72` and the static prose
+ in `umtool/components/BuildChain.tsx:144-150` to past tense ("were one-off shell run logs that
+ hardcoded their paths; not in the tree, not runnable from here"). Historical mentions in
+ `debox-bg.mjs:24`, `hush-head.mjs:47`, `pick-take.mjs:26`, `video-dir.mjs:5,137` stay (record as left).
+5. `um-manifest.json`: one-off strip of `vid` (record the command; indent 1 as the writer uses;
+ `git diff --stat` shows only removed `"vid"` lines); `build-um.mjs:250` writes
+ `items.map(({ vid: _v, ...rest }) => rest)` with a comment (the page needs the `file://` URL, the
+ tracked manifest must not carry a machine path); `:224` unchanged.
+6. `thumb-manifest.json` + `thumb-accepted.json`: one-off rewrite `out` relative to
+ `~/reports/quartering-uh-song` and `bg` relative to the song data dir; writer
+ `umtool/song/make-thumb.mjs:258` → `out: relTo(SONG_REPORTS, OUT), bg: relTo(SONG_DATA, BG)`
+ with a comment naming the two roots. Verify and record: from `umtool/`, `resolveInRoots
+ ("thumbs/<x>.jpg")` prints the same absolute file as before, and `thumbs.ts:165,199` yield the
+ same labels.
+7. Text: `WORKTREES.md:119` → `cwd /path/to/checkout/editor`; `umtool/song/fit-hooks.mjs:93` →
+ `<user>`; the `/run/media/user/` literals in the five `common/` files → `/run/media/<user>/` in
+ comments and `/run/media/operator/` in test fixtures (change both sides of each assertion); the
+ one `editor/CHANGELOG.md` line → `/run/media/<user>/` (a released entry is normally left as
+ written; this is a path example, not a name a reader searches for — say so in the record).
+8. If `common/lib/envVars.ts` states a default for `SONG_DIR`, `VIDEO_ROOT` or `CHANNELS_DIR`,
+ update it to the new default and regenerate `ENVIRONMENT.md` (`pnpm archilyzer docs env`).
+9. Gate: `git grep -c -i 'user' HEAD -- . ':!plans/'` → empty; tsc; `node --check` on every
+ changed `.mjs`; common 2,112 (edited, not added); `test:scripts` 185 + 1; editor unit 85; mcp 269;
+ `pnpm archilyzer docs env --check` clean; `pnpm --filter umtool exec next build`; `cd umtool &&
+ node -e 'import("./lib/paths.mjs").then(p=>console.log(p.CHANNELS_DIR,p.SONG_DATA))'` prints the
+ repo's `transcripts/channels` and the XDG path; umtool e2e `faces.spec.ts deck.spec.ts
+ clip-bench.spec.ts` via `node scripts/worktree.mjs run -- pnpm --filter umtool run e2e …` (the
+ fixture sets `CHANNELS_DIR`/`SONG_DIR`, proving env still wins). One `[Unreleased]` bullet in
+ `editor/CHANGELOG.md`.
+
+**As shipped (2026-09-28, `r12/paths-fix`; record: `release-12.md`, "Slice Q, as shipped"):**
+- **Steps 1–7 as written, with these additions:**
+ - `SONG_REPORTS` moved into `song/paths.mjs` beside a new `relTo(root, p)`, and `lib/paths.mjs`
+ re-exports it. The song scripts import only siblings, because the e2e fixture runs copies of
+ them.
+ - `accept-thumb.mjs`, the other writer of `thumb-accepted.json`, records `out` relative too.
+ - `make-thumb` passes `path.resolve(OUT)` / `path.resolve(BG)` to `relTo`.
+ - deck.spec pins the relative `out`.
+- **Step 3:** a missing `SONG_DIR` path is used as given, with no realpath.
+- **Step 8 was a no-op:** `envVars.ts` declares no umtool variable. The defaults are in
+ `umtool/docs/cli.md` instead.
+- **Step 9's common baseline is 2,114** (O6c's +2), unchanged by Q.
+
+## Slice R — `archilyzer source publish` + `/source` (branch `r12/source-mirror`, after Q)
+
+Owns `common/publish/{source,sourceAudit,sourceTree}.ts` + tests, `common/lib/sourceManifest.ts`,
+`common/publish/build.ts` (`buildHomepage` only), `common/bin/archilyzer.ts` + `_cli.test.ts`,
+`common/bin/doctor.ts` + test, `common/lib/paths.ts` + `common/lib/envVars.ts` (the new variables)
++ the regenerated `ENVIRONMENT.md`, `homepage/**`, `.gitignore`, `create-archives.sh` (deleted),
+`README.md`, `SETUP.md`, `PUBLISH.md`, `homepage/content/docs/install.md`; shared: both changelogs,
+`plans/release-12.md`.
+
+**New variables, declared in `common/lib/envVars.ts` and read through `getPaths()`**:
+`ARCHILYZER_CONFIG_DIR` (default `~/.config/archilyzer`), `SOURCE_SCRUB_FILE`,
+`SOURCE_DENYLIST_FILE` (defaults inside the config dir), `ARCHILYZER_SOURCE_SCRATCH` (default
+`os.tmpdir()`). None is test-only, so none carries `E2E_`.
+
+**`archilyzer doctor`** gains a "source publish" block: which filter-repo command resolves (or the
+install line), whether gitleaks is on PATH, whether the two operator files exist, and the last
+published `mirrorHead` if there is a manifest. Read-only, like the rest of doctor.
+
+**The `/sites` homepage jobs run the step too** (they call `buildHomepage` in-process). A refusal
+fails the job with the redacted audit report in its log. `--no-source` stays CLI-only.
+
+### R1. `common/publish/source.ts` — `publishSource(opts): Promise<number>` (0 ok, 1 refused)
+
+`SourcePublishOpts = PublishOpts & { force?, check?, keepScratch?, branch = "main", sourceRepo?
+(default: git -C monorepoRoot rev-parse --path-format=absolute --git-common-dir — a worktree build
+mirrors the PRIMARY's main), publicDir? (default homepage/public, HOMEPAGE_PUBLIC_DIR honoured),
+scrubFile?, denylistFile?, filterRepo?: string[]|null, gitleaks?: string|null (null = skip, tests),
+scratchRoot? (ARCHILYZER_SOURCE_SCRATCH ?? os.tmpdir()), now? }`. `class SourceRefusal extends Error`.
+Every child through the existing `runChildIntoLog` with `AbortSignal.any([signal,
+AbortSignal.timeout(ms)])`; scratch = `mkdtemp(<scratchRoot>/archilyzer-source-)`, removed in
+`finally` unless `keepScratch`. Order:
+
+1. `sourceCommit = git rev-parse refs/heads/main` in the common dir.
+2. `loadScrubRules` / `loadDenylist` (R1 files); `rulesHash = sha256(rules + denylist)`.
+3. **Skip** unless `force`/`check`: existing `public/source/manifest.json` with equal `sourceCommit`
+ + `rulesHash` AND `public/source/archilyzer.git/info/refs` AND the tarball present → log
+ `[source] up to date at <sha>; skipping (--force to rebuild)`, return 0.
+4. `resolveFilterRepo()`: `git filter-repo --version` → `["git","filter-repo"]`; else pipx →
+ `["pipx","run","--spec","git-filter-repo==2.47.0","git-filter-repo"]`; else `SourceRefusal` with
+ the install line (`pipx install git-filter-repo`). Log the choice and version.
+5. `git clone --no-local --bare --single-branch --branch main --quiet <common-dir> <scratch>/bare`
+ (120 s); `git -C bare remote remove origin`.
+6. Write `replace.txt`; run `<filterRepo> --force --replace-text replace.txt --replace-message
+ replace.txt` in `bare` (900 s; `--force` because origin was removed); `rm -rf bare/filter-repo`.
+7. `git repack -a -d -q --max-pack-size=20m` (300 s), `git prune-packed`, `git pack-refs --all`,
+ `git update-server-info`; refuse if `git count-objects -v` `count` ≠ 0 or `objects/info/packs`
+ is empty.
+8. `mirrorHead = git rev-parse refs/heads/main`, `subject = git log -1 --format=%s` (in bare).
+9. **Gate**: `auditBare` (R2). Any hit → print the redacted report, return 1, write nothing.
+10. Tree: `git archive --format=tar -o scratch/tree.tar refs/heads/main` → `tar -xf … -C
+ scratch/stage/source/tree`; `writeTreeIndexes` (R3).
+11. Tarball: `git archive --format=tar.gz -9 --prefix=archilyzer/ -o
+ scratch/stage/downloads/archilyzer-source.tar.gz refs/heads/main`; sha256 by stream;
+ `snapshot.json` in today's shape `{generatedAt, commit: mirrorHead, subject, bytes, sha256}`
+ (the `Snapshot` type in `homepage/app/lib/snapshot.ts` stays).
+12. Stage the mirror from an ALLOWLIST only: `HEAD`, `packed-refs`, `info/refs`,
+ `objects/info/packs`, `objects/pack/*.pack`, `objects/pack/*.idx` → `stage/source/archilyzer.git/`
+ (no `config`, `hooks/`, `description`, `logs/`, `*.rev`, `*.bitmap`).
+13. `auditFiles(stage)` — a byte sweep of every staged file except `*.pack`/`*.idx` (the object
+ walk covered those). Hit → return 1.
+14. Limits: total files > 15,000 or any file > 24 MiB → refusal.
+15. `--check` stops here (prints the report + counts, returns 0, writes nothing).
+16. Install, link-safe: `ownDir(public/source)`; `rm public/source/manifest.json` FIRST (a crash
+ mid-copy leaves the page in its empty state, never a stale manifest over a half tree);
+ `rm -rf public/source/{archilyzer.git,tree}`; `cp -r stage/source/*`; `ownDir(public/downloads)`;
+ `copyPublicFile(tarball)`; `writePublicFile(snapshot.json)`; LAST `writePublicFile(manifest.json)`.
+17. One summary line: `[source] published main <src12> as <mirror12>: N files, M MB (mirror P
+ packs, tree D dirs), tarball K MB sha256 <sha12>`.
+
+`common/lib/sourceManifest.ts` (leaf): `SOURCE_MANIFEST_VERSION = 1`, `MIRROR_DIR = "archilyzer.git"`,
+`CLONE_URL = ${PROJECT_URL}/source/archilyzer.git`, `TREE_HREF = "/source/tree/"`, and
+`SourceManifest = { version: 1; generatedAt; branch: "main"; sourceCommit; mirrorHead; subject;
+files; bytes; mirror: {files, bytes, packs}; tree: {files, dirs, bytes}; tarball: {href, bytes,
+sha256}; cloneUrl; treeHref; audit: {literals, objects, commits, gitleaks: "clean"|"skipped"};
+rulesHash; tools: {git, filterRepo} }`.
+
+`buildHomepage` becomes compose → `if (!opts.skipSource) { const c = await (opts.publishSource ??
+publishSource)({paths,onLog,signal}); if (c !== 0) return c; }` → `next build`. `r10-home.sh` is
+unchanged and fails closed (a refusal = `HOME_BUILD_FAILED`, no deploy). CLI table rows: `build
+homepage [--no-source]`; `source publish [--force] [--check] [--keep-scratch]`; `source audit
+[<git dir>]` (default: the published `homepage/public/source/archilyzer.git`; the rollout runs it on
+the LIVE clone). `SOURCE_DENYLIST_FILE` / `SOURCE_SCRUB_FILE` env override the file paths (the gate
+uses it to plant a literal).
+
+### R2. `common/publish/sourceAudit.ts`
+
+`parseDenylist(text)` → `{bytes, ci}[]`; `scanBuffer(buf, literals)`; `redactHit` (±24 bytes, the
+literal → `[REDACTED]`, non-printables `.`); `auditObjects(bare, literals)` = ONE stream `git -C bare
+cat-file --batch-all-objects --unordered --batch` with a framing parser scanning EVERY object —
+blobs, commits (message + identity lines), trees (entry NAMES), tags; `auditFiles(dir, literals,
+skip=/\.(pack|idx)$/)`; `runGitleaks(bare)` = `gitleaks git --no-banner --redact --exit-code 3
+--report-format json --report-path <scratch>/gitleaks.json <bare>` (300 s; 0 → "clean", 3 → parse
+`RuleID`/`File`/`Commit` into hits, other → `SourceRefusal`; ENOENT → "skipped" + a WARNING line);
+`auditBare` combines; `formatAuditReport` never prints a literal (shows `#n (x…, len L)`). Refusal
+output ends with `[source] add a rule to ~/.config/archilyzer/source-scrub.txt or drop the file from
+history, then re-run.`
+
+### R3. `common/publish/sourceTree.ts` and `_headers`
+
+`hrefFor(entry)` = `encodeURIComponent(name)` + `/` for dirs (`[slug]` → `%5Bslug%5D/`);
+`escapeHtml`; `renderTreeIndex({relDir, entries, mirrorHead, generatedAt})` — one dependency-free
+template (~40 lines inline CSS honouring `prefers-color-scheme`, `<meta name=robots content=noindex>`,
+breadcrumb with relative `../` hops, `..` row except at root, dirs before files sorted by name,
+sizes via `formatBytes` with raw bytes in `title`, footer `main @ <mirrorHead12> · generated <at> ·
+clone`); `writeTreeIndexes(treeDir, meta)` walks with `readdir({withFileTypes:true})`, REFUSES if a
+directory already has an `index.html`, returns `{files, dirs, bytes}`.
+
+Append to the tracked `homepage/public/_headers` (exact text; later rules win for the same header):
+
+```
+# The published source (`archilyzer source publish`, common/publish/source.ts).
+/source/manifest.json
+ Cache-Control: public, max-age=300, must-revalidate
+/source/archilyzer.git/*
+ Cache-Control: public, max-age=300, must-revalidate
+# The raw tree is PLAIN TEXT whatever the extension: wrangler's mime map would
+# send .ts as video/mp2t, .mjs as application/javascript, .md as text/markdown
+# and .html as a page on this origin.
+/source/tree/*
+ Content-Type: text/plain; charset=utf-8
+ X-Content-Type-Options: nosniff
+ X-Robots-Tag: noindex
+ Cache-Control: public, max-age=300, must-revalidate
+# ...except the generated directory indexes and the binary files.
+/source/tree/
+ Content-Type: text/html; charset=utf-8
+/source/tree/*/
+ Content-Type: text/html; charset=utf-8
+/source/tree/*.ttf
+ Content-Type: font/ttf
+/source/tree/*.m4a
+ Content-Type: audio/mp4
+/source/tree/*.zip
+ Content-Type: application/zip
+/source/tree/*.onnx
+ Content-Type: application/octet-stream
+/source/tree/*.svg
+ Content-Type: image/svg+xml
+ Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'
+```
+
+(`.md` deliberately text/plain so it renders raw. The binary list is what the tree holds today:
+3 `.ttf`, 3 `.m4a`, 1 `.zip`, 1 `.onnx`, 6 `.svg`.)
+
+### R4. Homepage
+
+- `homepage/app/lib/source.ts`: `loadSourceManifest()` — `public/source/manifest.json` with
+ `version === 1`, 40-hex `mirrorHead`, 64-hex sha, AND `public/source/archilyzer.git/info/refs` and
+ the tarball present (the `snapshot.ts:25-46` rule), else null.
+- `homepage/app/lib/nav.ts`: `{ href: "/source/", label: "Source" }` after Docs.
+- `homepage/app/source/page.tsx` (`PageShell`/`PageHeading`/`Fact` rows as `downloads/page.tsx`):
+ title "Source"; standfirst "The canonical public copy: a read-only git mirror, its raw tree, and a
+ tarball."; one paragraph in the operator's voice: "There is no GitHub, by choice. This site is where
+ the code lives in public: a read-only mirror of the private main branch, regenerated with every
+ deploy of this site. Commit ids differ from the private repository because paths were scrubbed on
+ the way out — same history, different hashes. Nothing here takes a push or a pull request."; the
+ clone command in `<pre data-testid="source-clone">`; Facts (`source-mirror-head`, reflects private
+ main, subject, generated, files/size); links "Browse the tree" (`source-tree-link`) and the tarball
+ with bytes + sha256 (`source-tarball-link`); sections "What is mirrored" and "License" (MIT). Empty
+ state `data-testid="source-empty"`: "No source published in this build." + how it is produced.
+- `homepage/app/downloads/page.tsx`: keep every phrase `downloads.spec.ts` asserts; add one sentence
+ pointing at `/source/` for history; the empty-state sentence names `archilyzer source publish`
+ instead of `./create-archives.sh`.
+- `homepage/tsconfig.json` `exclude: ["node_modules", "public"]`; `.gitignore` adds
+ `/homepage/public/source` with the Tailwind note — BOTH in R's first commit.
+- `homepage/e2e/marketing.spec.ts:42-47` nav list gains `["Source", "/source/"]`.
+
+### R5. Docs
+
+Delete `create-archives.sh` and every reference: `README.md:485`, `SETUP.md:162,182`,
+`homepage/app/downloads/page.tsx:102`, `homepage/app/lib/snapshot.ts:4,23`, `.gitignore:49,93`
+(grep again; lines move). `README.md`: the clone command as the canonical public copy (`main` only;
+ids differ because paths are scrubbed), the tree at `/source/tree/`, the tarball at `/downloads/`,
+regenerated by `archilyzer source publish` inside `archilyzer build homepage`. `SETUP.md` and
+`homepage/content/docs/install.md` "Get the code": clone first, tarball as the no-git alternative.
+**`PUBLISH.md`**: a new section `## The source mirror (homepage)` after `## Cloudflare Pages`, with
+the 20,000-file / 25 MiB caps (the step refuses at 15,000 / 24 MiB), the `.git` segment trap, the
+dumb-HTTP file set, the `_headers` block, the two operator files, and that only `wrangler pages dev`
+honours `_headers` locally. `ENVIRONMENT.md` regenerated. `[Unreleased]` bullets in
+`editor/CHANGELOG.md` and `homepage/CHANGELOG.md` (the homepage's keeps its own convention).
+
+### R6. Tests
+
+- `common/publish/source.test.ts` (~9): scrub-rule loading (built-in homedir rule first, comments,
+ missing file → refusal naming the path); denylist (`i:`, implied lhs); skip logic; limits; the
+ allowlist (a planted `config`/`hooks/`/`filter-repo/` are never copied); manifest shape; a **round
+ trip** in a temp repo (`git init -b main`, identity set, the `GIT_*` env vars deleted as
+ `common/controller/cutRelease.test.ts` does): three commits with a planted literal in one blob and
+ one message, rule `/srv/plantedhome==>/home/user`, `gitleaks: null`, `publicDir` temp → `git clone
+ file://<public>/source/archilyzer.git` succeeds, HEAD = `manifest.mirrorHead` ≠ `sourceCommit`,
+ the message reads `/home/user`, `git grep -F plantedhome $(git rev-list --all)` empty, no `config`
+ in the published dir, `info/refs` names `refs/heads/main`, `objects/info/packs` ≥ 1, snapshot sha
+ = fresh sha256 of the tarball, `tree/index.html` and one subdir index exist, no
+ `source/index.html`; a **planted-literal refusal** (denylist without a rule → 1, no manifest,
+ downloads untouched, report free of the literal); `--check` writes nothing. The two filter-repo
+ tests `t.skip` when `resolveFilterRepo()` throws — the gate requires they RAN.
+- `common/publish/sourceAudit.test.ts` (~6): blob / commit / tree-name hits, clean repo, redaction,
+ gitleaks skipped. `common/publish/sourceTree.test.ts` (~5): encoding, escaping, ordering,
+ breadcrumbs, the existing-index refusal. `_cli.test.ts` (+3 usage rows). `build.test.ts` (+1:
+ `skipSource` never calls the injected fake; a fake returning 1 fails the build).
+- `homepage/e2e/source.spec.ts` (5; the suite runs against `next dev` + `homepage/public` on disk,
+ two-state like `downloads.spec.ts`; the R gate runs `source publish` in the worktree BEFORE the
+ suite so the manifest branch is exercised — record which): the page's fixed copy; with a manifest
+ the clone command, tarball href (200 via `page.request`), and the sha on the page = `snapshot.json`
+ = `manifest.json`, else the empty state; the mirror files (`/source/archilyzer.git/HEAD` = `ref:
+ refs/heads/main`, `info/refs` contains `<mirrorHead>\trefs/heads/main`, `objects/info/packs` has a
+ `P pack-` line); tree indexes (`/source/tree/index.html` 200, one dir href + `index.html` 200, one
+ file href 200, the fonts dir index contains `%5B` and that href resolves). Content-Type cannot be
+ asserted under `next dev` (`_headers` is Pages-only) — the preview deploy checks it.
+
+### R7. Gates (worktree root, after merging `main` with Q)
+
+tsc (must stay fast — proves the tsconfig exclude); common 2,112 + ~26 (doctor included);
+`test:scripts` 185 + 1; editor unit 85; mcp 269; homepage unit 2; `pnpm archilyzer docs env --check`
+clean; `pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts build
+homepage` in the worktree (the parent creates `~/.config/archilyzer/{source-scrub,source-denylist}.txt`
+BEFORE the slice starts; the implementer never reads them aloud and never commits them) → `homepage/out/source/archilyzer.git/
+info/refs`, `manifest.json`, the tarball; record `find homepage/out -type f | wc -l` and the build's
+extra wall time; `git clone file://$WT/homepage/out/source/archilyzer.git` AND, with `python3 -m
+http.server 8765` in `homepage/out`, `git clone http://127.0.0.1:8765/source/archilyzer.git` — both
+HEADs = `manifest.mirrorHead`; `git -C <clone> grep -c -i user $(git rev-list --all)` empty; `source
+audit <clone>/.git` clean; the gate EXERCISED: `SOURCE_DENYLIST_FILE` pointing at a copy with a literal
+that IS in history (e.g. `Co-Authored-By`) → `source publish --check` exits 1 with redacted hits and
+`public/source/manifest.json`'s mtime unchanged; `build homepage --no-source` → the `/source/` page's
+empty state; `PATH` without pipx → the install-line refusal; `archilyzer doctor` prints the new
+block; homepage e2e 31 → 36 with the manifest present, `downloads.spec.ts` unchanged. Record the
+resolved filter-repo command and version.
+
+**As shipped (2026-09-28, `r12/source-mirror`; record: `release-12.md`, "Slice R, as shipped" and
+its "Review"):**
+- **R1–R6 as written, with these corrections found by running them:**
+ - **`_headers`:** Pages APPENDS a header a later matching rule sets again. Each override now
+ detaches with `! Content-Type` first, and `homepage/app/lib/headers.test.ts` pins it.
+ - **The homepage tsconfig excludes `out` as well as `public`,** and its eslint config ignores
+ `public/source/**`.
+ - **The mirror carries a loose `refs/heads/main`** beside `packed-refs`, because git needs a
+ `refs/` directory before a `file://` clone or `source audit` will read it.
+ - **filter-repo reads `#` lines as literals,** so `replace.txt` is written without them.
+ - **The clone takes `--no-tags`,** and filter-repo `--replace-refs delete-no-add --quiet`.
+ - **`--no-source` REMOVES the previous publish.**
+ - **The header nav moves to `md`.**
+- **No `rulesHash` and no literal count in the published manifest.**
+ - The skip key is `homepage/.source-publish.json` (gitignored and dockerignored):
+ `{sourceCommit, mirrorHead, rulesHash, filterRepo}`.
+ - `rulesHash` covers the rules, the literals and `SOURCE_STEP_VERSION`. Bump it whenever the
+ scrub or the audit changes.
+- **After the review (M1): a refusal withdraws the source everywhere it could ship from.**
+ - The step removes the last publish from `homepage/public` (not under `--check`).
+ - `buildHomepage` removes it from `homepage/out`.
+ - `deployHomepage` refuses an `out/` whose source was not audited under today's rules, of
+ today's `main`, or that has no `/source` page at all.
+- **The audit report names a literal by where it was written** (`denylist line 3 (len 5)`) and a
+ hit by object, field and byte offset. It prints no byte of any object.
+- **The review's other fixes:**
+ - a tracked `404.html` is refused;
+ - a BOM, CRLF and a trailing `/` on the home dir are handled, and an empty left side refuses;
+ - a checkout with no git repository builds with the `/source` empty state (and the CLI refuses
+ with the same sentence);
+ - a scratch root inside the checkout or the public dir is refused, and `--keep-scratch` deletes
+ `replace.txt`;
+ - `E2E_EXPECT_SOURCE=1` makes the empty state fail the spec;
+ - a malformed manifest is the empty state, never a crash.
+- **Left:** compressed content is opaque to the byte search (review L6; a known limit in
+ PUBLISH.md).
+- **The baselines moved:** common 2,114 → **2,146**; homepage unit 2 → **7**; homepage e2e 31 →
+ **36**.
+
+## Rollout (parent; nothing here edits the primary's tracked files)
+
+Release 11 is merged and NOT rolled out. A production homepage deploy of release 12 also ships
+release 11's homepage changes, so production waits for the operator's release 11 rollout (or is
+done as part of it, on the operator's word). The PREVIEW deploy below is safe at any time.
+
+0. Before slice R's gate, the parent creates `~/.config/archilyzer/` (mode 700) with
+ `source-scrub.txt` (`/run/media/user==>/run/media/user`, `` `user`==>`user` ``) and
+ `source-denylist.txt` (mode 600: `user`, the hostname from `hostname`, the Gmail address the
+ session already knows). **The operator adds what only they know** — real name, other handles,
+ anything else that must never appear — before the first production publish. `pipx install
+ git-filter-repo` (so a build needs no network and the editor's jobs find it on PATH). Q's
+ operator step, BEFORE any umtool restart: `mkdir -p ~/.local/share/archilyzer && ln -s
+ /home/user/.claude/jobs/efbe67a7/tmp/song ~/.local/share/archilyzer/song`.
+1. `archilyzer source publish --check`, then `archilyzer build homepage` (r10-home.sh's first line);
+ check `homepage/out/source/archilyzer.git/info/refs` and the file count.
+2. **Preview first**: `archilyzer deploy homepage --preview source` → `https://source.archilyzer.pages.dev`.
+ Live: `git clone https://source.archilyzer.pages.dev/source/archilyzer.git` (the edge-side unknown:
+ extensionless `HEAD`/`info/refs`, `.git` inside a segment past Cloudflare's managed rules, the dumb
+ fallback after the `?service=` probe); clone HEAD = `manifest.mirrorHead`; `curl -sI
+ …/source/tree/common/lib/paths.ts` → `text/plain; charset=utf-8` + `nosniff`; `…/source/tree/common/`
+ → text/html; `…/source/tree/umtool/report-to-video/fonts/Archivo%5Bwdth,wght%5D.ttf` → 200 font/ttf;
+ `…/source/tree/homepage/app/docs/%5Bslug%5D/page.tsx` → 200; the tarball's sha256 = `snapshot.json`
+ = the page; `archilyzer source audit <clone>/.git` clean.
+3. Production, after release 11's rollout: `archilyzer deploy homepage` (or `pnpm ops
+ build-homepage --json '{"deploy":true}'`, which also proves the step inside the editor's job),
+ the same checks on `archilyzer.pages.dev`; the record's `## Rollout`, STATE ("Now"), FACTS (the
+ `.git` segment trap, Tailwind/tsconfig scanning of `public/`, dumb HTTP on Pages), memory, and an
+ HTML runbook at `~/reports/release-12/RUNBOOK.html` (handoffs are HTML).
+4. If the edge refuses the dumb clone: the tree and tarball still stand; the mirror needs another host
+ (R2 behind a custom domain, out of scope) — record, do not improvise.
+
+## Verification summary
+
+Unit: the round trip (clone → scrub → repack → `update-server-info` → clone back), the planted-literal
+refusal, the audit's three hit kinds, the index encoding. Build: `archilyzer build homepage` in the
+worktree, then a dumb-HTTP clone from `homepage/out` over `python3 -m http.server`. Live: the preview
+deploy's clone, content types, bracket paths, tarball sha, and `source audit` on the live clone. Docs:
+`grep -rn 'create-archives' README.md SETUP.md homepage/` shows nothing.
+
+## Risks (flagged, not solved here)
+
+- Edge-side serving of `HEAD`/`info/refs`, the `.git`-inside-a-segment path and bracket paths are
+ proved upload-side only; the preview deploy is the test. Cloudflare's managed "git exposure" rules
+ match `/.git/`, ours is `/archilyzer.git/`.
+- `pipx run` needs network on first use and every 14 days; the editor's `/sites` homepage jobs run in
+ the editor's environment where `pipx`/`gitleaks` may be off PATH → the build refuses (fail closed)
+ or skips gitleaks with a warning. `pipx install git-filter-repo` once removes both.
+- filter-repo determinism holds for a fixed version; an upgrade that changes rewriting defaults
+ changes `mirrorHead` (readers see a non-fast-forward; the page says ids may change); the manifest
+ records the version. Object ids are stable across git versions; pack bytes are not (irrelevant).
+- The denylist `user` is exact-case; a future transcript quote "user that" trips the gate —
+ intended (add a rule or reword).
+- About a minute more per homepage build (filter-repo ~20-60 s); an unchanged `main` skips it.
+- The first publish replaces the 2026-08-12 tarball and `Snapshot.commit` becomes a mirror sha; only
+ the `/source` page's "reflects private main" row links the two — deliberate.
diff --git a/plans/stats-cache-key.md b/plans/stats-cache-key.md
@@ -0,0 +1,280 @@
+# The stats cache key (fix, 2026-09-28)
+
+Branch `fix/stats-cache-key` from `main` `ac438bbc`. Merged: not yet. Rolled out: not yet.
+
+## What was wrong
+
+- **The stats cache was keyed on the wrong file.** The cache (`statsByPath`, in the index LMDB)
+ was keyed on `metadata.info.json`'s mtime alone. But `hasTranscript`, `cueCount`, `coverage` and
+ `transcribedDate` come from the index and the transcript files.
+- **So a late transcript never reached its stat.** That covers a Whisper run days after the
+ download, a Normalize run weeks later, and a stats run made before `build:index` had the video.
+ The hub and homepage composers make that last kind of run.
+- **Caption videos could not be dated at all.** A caption-only video (a YouTube VTT, no
+ `transcript.json`, no outcome sidecar) got no `transcribedDate` even when computed fresh.
+- **The homepage then dropped every undated transcript.** Its fold required `hasTranscript &&
+ transcribedDate`. Jasolyzer served 1,889 videos and showed 0 transcripts, 0 channels, 0 hours.
+ Every site's counts and "Transcribed over time" charts were low.
+
+## Numbers, before and after (ESTIMATES)
+
+**How they were measured.** The whole-pool stats pages of 2026-09-28T20:40Z (`homepage/public/stats`)
+were classified record by record against the files on disk now. This was read-only: listings,
+`stat` and small JSON reads. No LMDB was opened and nothing was rebuilt.
+
+- **Shown** is the published homepage summary.
+- **After** counts the records a rebuild with this fix would count. These are:
+ - `hasTranscript` with a date source on disk;
+ - stale `hasTranscript: false` records whose `transcript.cues.json` has cues;
+ - caption-only records, which the new date fallback reaches.
+
+The real numbers come from the first rebuild.
+
+| Site | Records | Transcripts shown | Transcripts after (est.) |
+| --- | ---: | ---: | ---: |
+| Anilyzer | 29,836 | 8,370 | ~29,665 |
+| Jeralyzer | 32,994 | 29,983 | ~31,906 (+ up to ~290, see below) |
+| Bonnellyzer | 8,085 | 5,540 | ~7,244 |
+| Hasanalyzer | 3,432 | 3,048 | ~3,335 |
+| Rekietalyzer | 2,931 | 2,855 | ~2,923 |
+| Jasolyzer | 1,889 | 0 | ~1,751 (~3,808 h) |
+| Whole pool | 79,385 | 49,798 | ~76,990 |
+
+- **Why so many were missing:**
+ - 24,710 records had `hasTranscript` with no date. Of those, 23,198 have a date source on disk
+ now, and 1,512 are caption-only.
+ - 2,484 carried a stale `hasTranscript: false`.
+- **About 290 more are probably stale but are not counted above.** These are records with large
+ caption files and `cueCount: null` (Jeralyzer's paramount-tactical, and the pool-only
+ candace-owens). The live Jeralyzer shards serve cues for the three that were checked.
+- **The whole-pool total includes pool-only channels,** so it is more than the sum of the sites.
+
+## The fix
+
+| Commit | What |
+| --- | --- |
+| `ad152529` | `common:` the key becomes the metadata mtime AND buildIndex's `mtimes.transcriptMs`. `transcribedDate` falls back through the outcome sidecar, then the transcript the index read (`transcript.json`, else the caption VTT), then `transcript.cues.json`, then `downloadedDate`, so a transcript always has one. `STATS_SCHEMA_VERSION` 5 → 6. New `buildStats.test.ts`. |
+| `b7a733ad` | `common:` test (b) asserts the heal before the new count. |
+| `9a8bded7` | `common:` the homepage fold counts a transcript with no date (totals, channels, hours, the card's total, upload-month placement). Only the transcribed series, "this month" and the recent rail need the date. |
+| `e765b168` | `mcp:` `get_video_metadata`'s stats block on a recomputed stat: the real cue count and no false "truncated". |
+| `6c7ff664` | `plans:` the first record, FACTS, changelogs. |
+| `9bc3c635` | `common:` buildIndex records `meta.scannedAt`, the time its last completed scan began. |
+| `e4c61c77` | `common:` the review's fixes (see "Review" below): the downgrade guard, the whole index record as the key with the cues read under its `indexKey`, "not indexed yet" and "not indexable" counted apart, an unmounted drive's stats kept (and a clear refused), and test hygiene. |
+| `9e61b119` | `common(test):` `source.test.ts` expands the `~` the kept-scratch log prints (release 12's test). |
+| `000273c0` | `common:` test (i) asserts the kept stats before the new result field. |
+| `22ec383e` | `plans:` the record's review, gates and rollout; FACTS; STATE; the three changelogs. |
+| `b30b52c1` | `common:` re-review R3 and R4: the "not indexable" line names an unreachable drive, and the held-channel refusal names every way out, without paths. Test (z) asserts the LMDB file is under the temp root. |
+| this commit | `plans:` re-review R1, R2, R4 (record), R5 and the nits: the rollout's commands as typed, the index's `Diff:` check, the hub's summary check, numbered preconditions, and the index follow-up in the record and STATE. |
+
+- **Caption videos are dated by when their captions arrived** (the VTT's mtime), not by a later
+ Normalize run. Whisper videos resolve as before. Their "Transcribed over time" curves move: on
+ Jasolyzer, 1,683 videos would otherwise all have landed on the Normalize day.
+- **The published stats page format did not change.** `STATS_MANIFEST_VERSION` stays 1, and
+ `HOMEPAGE_SUMMARY_VERSION` stays 5.
+- **The compose paths warn and do not refuse.** `compose-hub` and `compose-homepage` still run
+ `buildStats` against the index as it stands, and they still do not build the index.
+ - A video they meet before the index has it is keyed `NOT_INDEXED`. It is recomputed on the first
+ run after the next index build.
+ - The log counts separately the videos downloaded since the last index build and the ones older
+ than it that it did not index: no `upload_date`, a failure, or their channel's media unreachable
+ during that build.
+ - A refusal would stop these builds whenever the editor had downloaded since the last index build,
+ which is almost always.
+- **Concurrent stats builds: an operator rule, not a lock** (review L3, brief item 5).
+ - There is no cross-process lock primitive in `common/`. `scripts/queue-lock.mjs` is the e2e
+ queue's flock wrapper, run as a separate holder process. Building one here would be a new
+ lock, with its own stale-lock story.
+ - The rule is: **one stats build at a time.** The editor's build jobs share the queue `"build"`
+ by default. **The CLI is outside every queue.**
+ - Two concurrent runs are harmless unless one clears the cache, which only a schema change does.
+
+## Review
+
+**Verdict: SHIP AFTER FIXES** (the review is `j-review.md` in the job's scratch). The reviewer found
+no code defect; every fix was made on this branch.
+
+| Finding | Where |
+| --- | --- |
+| L1: the record overstated which paths run old code | `22ec383e`: the rollout says only the in-process **Build stats dataset** runs old code, and step 1 is "rebuild the editor bundle, then restart"; the editor changelog says the same. |
+| L2: wrong rollout commands | `22ec383e`: every command checked against `pnpm archilyzer --help` and `pnpm ops --help`. The homepage and the hub are each built and deployed ONE way, the hub before the sites, and `build-deploy` takes `{"all":true}`. |
+| L3: concurrent runs; the removable drive | `e4c61c77`: an unmounted drive's channel is held and its stats kept, and a cache clear with one refuses. Rollout step 2 is the precondition. The lock was **left**: see "Concurrent stats builds" above. |
+| L4: test hygiene | `e4c61c77`: every `getPaths()` path is pinned under the temp root, the root is removed in `after`, and case (z) spies on node:fs and node:fs/promises (async and sync) and fails on any write outside it. |
+| L5: the "not in the index yet" line was wrong for skipped videos | `9bc3c635` + `e4c61c77`: `notIndexedYet` and `notIndexable`, logged apart, with case (h). |
+| L6: the time estimate left out the USB drive | `22ec383e`: 10–30 minutes, with the reason; interruptible and resumes. |
+| L7: no homepage changelog bullet | `22ec383e`: `homepage/CHANGELOG.md`, and the export bullet says "once the site is rebuilt". |
+| The guard (ruled: add it) | `e4c61c77`: a build refuses to clear a cache a newer schema wrote, and names both versions and `ARCHILYZER_STATS_ALLOW_DOWNGRADE`. The variable is declared in `envVars.ts`; `ENVIRONMENT.md` is regenerated and `--check` is clean. Case (j) covers older → cleared, newer → refused (also the CLI, exit non-zero, cache untouched), and the override → cleared. |
+| O1 (ruled: do it) | `e4c61c77`: the cues are read under the `mtimes` record's `indexKey` (case (f)), and the key is the whole record (case (g)). About 40 lines with comments, and no new I/O on the unchanged path: the same one LMDB get per video. |
+| O2: date from the index's mtime | **Left.** The fresh readdir happens only on the recompute path, and it keeps the Whisper rule byte-identical to before. |
+| O3: the spy saw only async fs | `e4c61c77`: the spy now covers node:fs sync and callback APIs too. |
+| O4: the reviewer's extra cases | **Partly.** Case (e) now includes an index rebuild with no churn. The removed-transcript and two-channel cases stay in the reviewer's scratch, where they pass. |
+| O5: manual captions parse to 0 cues | **Left**, noted in FACTS. |
+| `source.test.ts` under a home `TMPDIR` | `9e61b119`: 13 of 13 with `TMPDIR` unset and with it under `~`. |
+
+**The new cases fail on the pre-review code** (`6c7ff664`'s `buildStats.ts`, `buildIndex.ts` and
+`stats.ts` swapped in once):
+
+| Case | Result |
+| --- | --- |
+| (b) | `undefined` for `notIndexedYet` (the field did not exist) |
+| (e) | `NaN` counts, for the same reason |
+| (f) | `hasTranscript` false: the cues missed under the metadata's upload date |
+| (g) | changed 0, not 1: the cue-count drift |
+| (h) | the fields did not exist |
+| (i) | removed 2, not 0: the drive's stats were dropped |
+| (j) | "Missing expected rejection": the newer cache was cleared |
+
+(a), (c), (d) and (z) pass there, as they should: (a) to (d) were fixed before the review.
+
+### Re-review
+
+**Verdict: SHIP.** No code defect; five Low touch-ups and two nits, all made:
+
+| Finding | Where |
+| --- | --- |
+| R1: `archilyzer` is not on PATH | this commit: every rollout command is written as typed from the primary checkout's root, `pnpm archilyzer …`, as release 12's records spell it. Nothing else on the branch spells a bare command: the code's messages name none, and FACTS and the changelogs name scripts or pages. |
+| R2: step 3 had no post-check | this commit: the index step now reads its `Diff:` line. Thousands removed means a drive was missing; mount it and re-run the index before step 4. |
+| R3: "not indexable" blamed the video for a missing drive | `b30b52c1`: the line adds "or its channel's media was unreachable during that build (run an index build with every drive mounted)", and case (h) expects it. |
+| R4: the refusal gave one way out | `b30b52c1` (message) and this commit (record): mount its media first; else repair or re-point its location on /storage, finish or clear its move, or delete the channel or set `excludeFromBuild`. The message carries no path: a held channel is named with its location's label. Case (i) asserts both. |
+| R5: a hub build with no figures still deploys | this commit: step 6 builds the hub, checks the log for `hub-summary.json covers N official instance(s)`, and deploys only then. `hub-summary.json skipped: …` means stop. |
+| Nits | this commit numbers the preconditions; `b30b52c1` asserts `paths.lmdbPath` is under the temp root in case (z). |
+| Follow-up (recommended) | recorded under "Left" and in STATE: **the index build still treats an unmounted drive as an empty channel.** It needs its own slice before routine builds resume. Until then, R2's check is the safeguard. |
+
+## Gates (worktree, 2026-09-28, at the review fixes)
+
+- **tsc:** clean (43 s).
+- **Common tests, with `TMPDIR` unset:** **2,161, all pass**. That is 2,149 at the branch point,
+ plus 11 buildStats cases and 1 homepage fold case.
+- **Other unit suites:**
+ - editor unit: 85 of 85;
+ - `test:scripts`: 185 pass, 1 skipped;
+ - mcp: 271 of 271;
+ - homepage unit: 7 of 7.
+- **Docs:** `pnpm archilyzer docs env --check` is clean.
+- **Builds:** `next build` succeeded for export (36 s), editor (55 s) and homepage (23 s).
+- **e2e:**
+ - Before the review (at `6c7ff664`):
+ - editor `duplicate-shorts`, `build`, `site-scope`, `sites-homepage` and `deploy-page`: 21
+ passed, 0 failed, 2.3 min;
+ - export `charts.spec.ts`: 8 passed, 28 s;
+ - homepage, full suite: 36 passed, 1.0 min.
+ - After the review fixes: the same editor list again, because `duplicate-shorts` drives Build
+ index and Build stats dataset: 21 passed, 0 failed, 1.2 min.
+ - Export and homepage were not rerun. The review fixes change no export or homepage code; the
+ homepage fold is unchanged since `9a8bded7`.
+
+- **At the re-review touch-ups (`b30b52c1`):**
+ - tsc is clean;
+ - `buildStats.test.ts` and `envVars.test.ts` pass 20 of 20;
+ - `pnpm archilyzer docs env --check` is clean;
+ - no e2e was run, since only wording and one assertion changed.
+
+## Rollout (operator) — follow it literally, in this order
+
+Every command below is typed **from the primary checkout's root**. There is no `archilyzer` on
+PATH, so it is `pnpm archilyzer …`.
+
+**What runs which code.**
+- **The live :3001 editor runs its BUILT bundle** until it is rebuilt and restarted. The only
+ stats path that runs inside that bundle is the **Build stats dataset** button (`buildStatsAction`,
+ in-process). On the old code it has no guard: against the new cache it would clear it and refill
+ it the old way.
+- **Everything else spawns the checkout's code from disk**, so it runs the new code the moment
+ `main` has this merge:
+ - a site build's data phase (`pnpm run build:data`);
+ - the hub (`compose:hub`) and the homepage (`compose`);
+ - every CLI command.
+- So **the first of those after the merge is the first schema-6 stats run.** It clears the cache
+ and does the whole pass inside that job.
+
+**Preconditions for steps 3 and 4.**
+1. **The removable media drive is mounted.** `/storage` shows every location **Available**.
+ - The stats build now refuses a cache clear while any channel's media is unreachable.
+ - **The index build has no such guard.** Run with the drive absent, it drops those channels
+ from the index, and the next site build publishes them as gone. Step 3's `Diff:` check is what
+ catches it.
+2. **No other index, stats or site build is running.**
+ - `/jobs` shows no `build-index`, `build-stats`, `build-site`, `build-deploy`, `build-hub` or
+ `build-homepage` job running or queued, on any queue.
+ - No CLI or spawned build is running:
+ `pgrep -af 'archilyzer\.ts (index|build|compose)'` prints nothing. Every CLI build and every
+ spawned data phase or compose goes through `archilyzer.ts`; the in-process editor jobs do not
+ show here, and `/jobs` covers them.
+
+**The steps.**
+
+1. **Rebuild the editor bundle, then restart :3001 onto it:** `pnpm --filter editor build` in the
+ primary checkout, then restart the editor the way it is normally run. Between the merge and this
+ restart:
+ - **never press Build stats dataset**;
+ - **start no site, hub or homepage build**. It would do step 4's full pass itself, inside that
+ job, unannounced.
+2. **Check preconditions 1 and 2 above.**
+3. **Index.** Use `pnpm archilyzer index`, or `/sites` → **Build index**
+ (`pnpm ops build-index --wait`). It writes the same LMDB as the editor, so run it only with no
+ build job running (precondition 2).
+
+ **Then check its `Diff:` line**, `Diff: +A added, ~C changed, -R removed, N total.`:
+ - with `pnpm archilyzer index` or `pnpm ops build-index --wait`, it is in the terminal output;
+ - from the button, it is in the `build-index` job's log on `/jobs`.
+
+ **R should be 0, or a handful.** Thousands removed means a drive was missing during the build,
+ and those channels just left the index. Stop, mount the drive (precondition 1), and run step 3
+ again before step 4.
+4. **Stats.** Use `pnpm archilyzer build stats`: the CLI, with the editor idle. The in-process
+ button stalls the editor for the length of the pass.
+ - The first run logs `Stats schema change (5 -> 6); clearing stats cache.` and re-extracts every
+ video: **about 10–30 minutes, longer with a cold cache.** About a quarter of the video dirs
+ are on the USB drive, at 4–5 random reads each.
+ - It can be interrupted (Ctrl-C, or cancelling the job) and **resumes**: the schema is written at
+ the clear, so the next run only finishes the rest.
+ - **If it refuses because a channel cannot be read,** the message names the channel and its
+ location. The ways out, in order:
+ 1. mount its media and run step 4 again;
+ 2. repair or re-point its location on `/storage`;
+ 3. finish or clear its move, from the channel's Storage panel;
+ 4. if the channel is gone for good, delete it or set `excludeFromBuild` in its config.
+ - **Let it finish before step 5.**
+5. **Homepage.** Use exactly ONE of:
+ - `pnpm archilyzer build homepage && pnpm archilyzer deploy homepage`;
+ - `pnpm ops build-homepage --json '{"deploy":true}' --wait`.
+
+ Building the homepage **also runs release 12's source publish**. So the homepage waits on
+ release 12's rollout step 0: the denylist is complete and `source publish --check` is clean.
+ The homepage and source in `main` at that moment must be the ones the operator has judged.
+6. **Hub, before the sites** (in basic mode the hub and the sites share `export/out`). Build it,
+ check its log, and only then deploy it. Use exactly ONE pair:
+ - `pnpm archilyzer build hub`, then `pnpm archilyzer deploy hub`;
+ - `pnpm ops build-hub --wait`, then `pnpm ops deploy-hub --wait`.
+
+ **Between the two,** the build's output (the compose line, `compose-hub: …`) must end with
+ `hub-summary.json covers N official instance(s)`, where N is the number of public sites (6 today).
+ **`hub-summary.json skipped: …` means the hub would deploy with no figures on its cards.** Stop
+ and fix the cause it names before deploying.
+7. **The six sites:** `pnpm ops build-deploy --json '{"all":true}' --wait`.
+
+**Live check.**
+- The homepage's Jasolyzer card shows about 1,751 transcripts, 1 channel and about 3,808 hours.
+- `https://jasolyzer.pages.dev/stats/page-0000.json` has no record with `hasTranscript: true`
+ and `transcribedDate: null`.
+- Step 4's log has no `Channel …: … cached stat(s) are kept` line, which would mean a held
+ channel.
+
+## Left
+
+- **MCP "truncated" for no transcript at all.** A video with no transcript has coverage 0
+ (`transcriptCoverage(undefined, d > 0)`), so MCP `get_video_metadata` tells it "covers only
+ 0% — truncated". This is older than the cache bug and out of scope. The coverage should be null
+ when there are no cues.
+- **FOLLOW-UP, its own slice: the index build still treats an unmounted drive as an empty
+ channel.**
+ - What it does now: it drops that channel's index records, and the next site build publishes the
+ channel as gone.
+ - What it needs: `buildIndex` gets the same hold the stats build now has (keep the channel's
+ `mtimes`, cues and pages), or at least a refusal with an override.
+ - When: schedule it before routine builds resume after this rollout.
+ - Until then, the rollout's step 3 `Diff:` check is the safeguard.
+ - **Closed by release 15 slice IG** ([`release-15.md`](release-15.md), "Slice IG, as shipped"):
+ the hold, and a refusal with an override for a full rebuild.
+- **A cross-process lock for builds** (see "Concurrent stats builds" above). The rule stands in
+ for it.
+- **O5:** a manual English caption (no inline timing tags) indexes as 0 cues (FACTS).
diff --git a/plans/tools/implementer-rules.md b/plans/tools/implementer-rules.md
@@ -77,6 +77,18 @@ the Next.js reference for this version.
test:scripts` (156 + 1 skip); mcp `pnpm --filter yt-dlp-transcript-mcp test` (219) — check
`package.json` for the exact script names before running.
- `pnpm --filter editor exec next build` and `pnpm --filter export exec next build`.
+- **umtool's build runs with the corpus visible, under a memory cap.** A worktree has no
+ `transcripts/`, so a path Turbopack traces as a directory is empty there. It was hundreds of GB in
+ the primary, and the build was OOM-killed (FACTS, "A path joined from `process.cwd()` …"). From
+ the worktree root:
+ ```
+ ln -sT <primary>/transcripts transcripts && timeout -s KILL 240 systemd-run --user --scope -q -p MemoryMax=5G -p MemorySwapMax=0 pnpm --filter umtool exec next build; rm transcripts
+ ```
+ It is for the BUILD only. Always remove the link, never commit it, and never run an app, an index
+ or a fixture builder through it. `-T` matters: a worktree that already has a `transcripts/`
+ directory (an old `index.mdb`) would otherwise get `transcripts/transcripts`, and the build
+ would run without the corpus and pass for the wrong reason (slice UT, release 15). It should take about 25 s at under 1 GB, the same as without
+ the corpus.
- The slice's e2e spec list (in the prompt), detached and waited on as above.
- The numbers tool the prompt names, diff-empty (or "none", stated).
diff --git a/plans/tools/rollout/smoke.sh b/plans/tools/rollout/smoke.sh
@@ -25,7 +25,7 @@ done
echo "BUILD_ID: $(cat $R/editor/.next/BUILD_ID)"
echo "ZodError in start log: $(grep -c ZodError $T/editor-start-r8.log 2>/dev/null)"
# release 7 additions
-d=$(curl -s --max-time 300 "$B/api/view/autoQueueStatus" | jq -c '.download.deferred | if type=="array" and all(.[]; type=="object" and has("videoId") and has("until")) then "ok \(length)" else "bad" end' 2>/dev/null); case "$d" in '"ok '*) echo "DEFERRED_OK download.deferred well-formed ($d)";; *) echo "DEFERRED_UNEXPECTED $d"; fail=1;; esac
+d=$(curl -s --max-time 300 "$B/api/view/autoQueueStatus" | jq -c '.download.deferred | if type=="array" and all(.[]; type=="object" and has("videoId") and has("untilMs")) then "ok \(length)" else "bad" end' 2>/dev/null); case "$d" in '"ok '*) echo "DEFERRED_OK download.deferred well-formed ($d)";; *) echo "DEFERRED_UNEXPECTED $d"; fail=1;; esac
f="$T/$REL-page_operations_download.html"; c=$(curl -s -o "$f" --max-time 300 -w %{http_code} "$B/operations/download"); n=$(grep -c 'aria-label="Rate-limit cooldown"' "$f")
[ "$c" = 200 ] && echo "PAGE_OK /operations/download rate-limit-regions=$n" || { echo "PAGE_FAIL /operations/download $c"; fail=1; }
f="$T/$REL-page_sites.html"; c=$(curl -s -o "$f" --max-time 300 -w %{http_code} "$B/sites"); echo "PAGE $c /sites Build hub=$(grep -c 'Build hub' "$f") Deploy hub=$(grep -c 'Deploy hub' "$f")"
diff --git a/scripts/next-build-trace.test.mjs b/scripts/next-build-trace.test.mjs
@@ -0,0 +1,510 @@
+// No module a Next app bundles may hand Turbopack a directory to trace: umtool,
+// the editor, the export and the homepage, and the common/ modules they import.
+//
+// Turbopack evaluates `process.cwd()` (and a module's own `import.meta.url` /
+// `__dirname`) statically, as a path in the project, and a `path.join` /
+// `path.resolve` / fs call on such a value becomes an ASSET REFERENCE: to a
+// file, or, when the joined path is a directory, to EVERY file under it. Release
+// 12 slice Q wrote `path.join(REPO_ROOT, "transcripts", "channels")` with
+// `REPO_ROOT = findRepoRoot(process.cwd())`, and `next build` in the primary
+// checkout walked the whole corpus (hundreds of GB, `data/` symlinked to another
+// drive) until the kernel killed it -- while a worktree, which has no
+// `transcripts/`, built in 30 s. So no build-in-a-worktree gate can see this.
+//
+// The rule, checked statically and per module (Turbopack's value analysis is
+// per module; an imported binding is opaque to it): every path or fs call whose
+// arguments carry a value derived IN THAT FILE from `process.cwd()`,
+// `import.meta.url`, `import.meta.dirname|filename` or `__dirname` must open its
+// argument list with Turbopack's opt-out, `/* turbopackIgnore: true */` (the
+// form its own "whole project was traced" warning advises; the Next docs list
+// the comment only for import(), require(), require.resolve() and new Worker()).
+// The comment changes nothing at run time. A function declared in the file whose
+// body carries a source is a source too (`const ROOT = findMonorepoRoot()`).
+//
+// A value Turbopack cannot know -- `process.env.*`, `os.homedir()`, a
+// parameter, an imported binding -- is not one of these sources, and it is not
+// ignored either: it is a dynamic part, and a path or fs call on it becomes a
+// PATTERN over the app's own directory. Measured in umtool (release 15, slice
+// UT): the path ops on env and home-directory values in its path modules took
+// in the app's whole tree outside dot-directories (opting them out left 31 of
+// the 68 routes clean), and the clip-audio route's join with a dynamic extension took
+// in the dot-directories too, the e2e fixture and `.env.local` among them.
+// Neither walk entered a symlinked directory. No static check here can tell
+// such a pattern from a harmless one, so the last test reads a build's traces
+// back instead.
+//
+// Run with: pnpm test:scripts
+import assert from "node:assert/strict";
+import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import test from "node:test";
+import { fileURLToPath } from "node:url";
+
+const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
+const UMTOOL = path.join(REPO, "umtool");
+
+const SOURCE = /process\.cwd\(\)|import\.meta\.(?:url|dirname|filename)|\b__dirname\b/;
+const MARK = "__TURBOPACK_IGNORE__";
+const IGNORE_COMMENT = /\/\*\s*turbopackIgnore\s*:\s*true\s*\*\//g;
+
+// path ops, and fs calls either bare (`existsSync(`) or on a namespace
+// (`fs.readdir(`, `fsp.stat(`, `fs.promises.readFile(`). A method on anything
+// else (`obj.stat(`) is not one. The opens and writes are checked too
+// (`open`, `writeFile`, `appendFile`, `createWriteStream`): which fs calls
+// Turbopack traces is not documented, and an opt-out on one it does not trace
+// costs nothing.
+const SINK =
+ /(?:\bpath\.(?:join|resolve|dirname|relative)|(?<![\w$.])(?:fs\.promises\.|fsPromises\.|fs\.|fsp\.|promises\.)?(?:existsSync|readFileSync|readdirSync|statSync|lstatSync|realpathSync|opendirSync|openSync|writeFileSync|appendFileSync|readFile|readdir|stat|lstat|opendir|open|writeFile|appendFile|createReadStream|createWriteStream))\s*\(/g;
+
+/** Comments out, except the opt-out, which becomes a marker. Strings stay. */
+function prepare(text) {
+ let s = text.replace(IGNORE_COMMENT, ` ${MARK} `);
+ s = s.replace(/\/\*[\s\S]*?\*\//g, (m) => m.replace(/[^\n]/g, " "));
+ // A line comment: `//` not preceded by `:` (URLs, `file://` templates).
+ s = s.replace(/(^|[^:\\])\/\/[^\n]*/g, (m, pre) => pre + " ".repeat(m.length - pre.length));
+ return s;
+}
+
+/** Index just past the regex literal whose opening `/` is at `i`, or -1 when
+ * the `/` there is a division. A `/` opens a regex after an operator, an
+ * opening bracket, a separator or `return` — good enough for this repo. */
+function regexEnd(s, i) {
+ let j = i - 1;
+ while (j >= 0 && /\s/.test(s[j])) j -= 1;
+ const before = j < 0 ? "" : s[j];
+ const word = s.slice(0, j + 1).match(/[A-Za-z_$][\w$]*$/)?.[0];
+ if (!(before === "" || "(,=:[!&|?{};+-*%<>~^".includes(before) || word === "return" || word === "typeof")) return -1;
+ if (s[i + 1] === "/" || s[i + 1] === "*") return -1;
+ let inClass = false;
+ for (let k = i + 1; k < s.length; k += 1) {
+ const c = s[k];
+ if (c === "\n") return -1;
+ if (c === "\\") k += 1;
+ else if (c === "[") inClass = true;
+ else if (c === "]") inClass = false;
+ else if (c === "/" && !inClass) return k + 1;
+ }
+ return -1;
+}
+
+/** Index just past the bracket that closes the one at `open`. Strings and
+ * regex literals are skipped whole. */
+function closeOf(s, open) {
+ let depth = 0;
+ let quote = null;
+ for (let i = open; i < s.length; i += 1) {
+ const c = s[i];
+ if (quote) {
+ if (c === "\\") i += 1;
+ else if (c === quote) quote = null;
+ continue;
+ }
+ if (c === "/") {
+ const end = regexEnd(s, i);
+ if (end !== -1) {
+ i = end - 1;
+ continue;
+ }
+ }
+ if (c === '"' || c === "'" || c === "`") quote = c;
+ else if (c === "(" || c === "[" || c === "{") depth += 1;
+ else if (c === ")" || c === "]" || c === "}") {
+ depth -= 1;
+ if (depth === 0) return i + 1;
+ }
+ }
+ return s.length;
+}
+
+/** Names assigned, in this file, from a source or from another such name. */
+function taintedNames(s) {
+ const decls = [];
+ const re = /\b(?:const|let|var)\s+([A-Za-z_$][\w$]*)\s*=/g;
+ for (let m; (m = re.exec(s)); ) {
+ // The right-hand side runs to the first `;` or newline at depth 0 -- good
+ // enough for this repo's formatter, which ends statements with `;`.
+ let depth = 0;
+ let end = s.length;
+ for (let i = re.lastIndex; i < s.length; i += 1) {
+ const c = s[i];
+ if (c === "(" || c === "[" || c === "{") depth += 1;
+ else if (c === ")" || c === "]" || c === "}") depth -= 1;
+ else if (c === ";" && depth <= 0) {
+ end = i;
+ break;
+ }
+ }
+ decls.push({ name: m[1], rhs: s.slice(re.lastIndex, end) });
+ }
+ // A function declared in this file whose body carries a source: a call to it
+ // is a source too (`const ROOT = findMonorepoRoot()`, where the function
+ // walks up from `process.cwd()` and falls back to it).
+ const fnRe = /\bfunction\s*\*?\s*([A-Za-z_$][\w$]*)\s*\(/g;
+ for (let m; (m = fnRe.exec(s)); ) {
+ const params = fnRe.lastIndex - 1;
+ const bodyOpen = s.indexOf("{", closeOf(s, params));
+ if (bodyOpen === -1) continue;
+ decls.push({ name: m[1], rhs: s.slice(params, closeOf(s, bodyOpen)) });
+ }
+ const names = new Set();
+ for (let grew = true; grew; ) {
+ grew = false;
+ for (const { name, rhs } of decls) {
+ if (names.has(name)) continue;
+ if (SOURCE.test(rhs) || [...names].some((n) => new RegExp(`(?<![\\w$.])${n.replace(/\$/g, "\\$")}\\b`).test(rhs))) {
+ names.add(name);
+ grew = true;
+ }
+ }
+ }
+ return names;
+}
+
+/** Every path/fs call on a cwd-derived value that does not opt out. */
+export function untracedCalls(text) {
+ const s = prepare(text);
+ const names = taintedNames(s);
+ const carries = (args) =>
+ SOURCE.test(args) ||
+ [...names].some((n) => new RegExp(`(?<![\\w$.])${n.replace(/\$/g, "\\$")}\\b(?!\\s*:)`).test(args));
+ // A call nested in these arguments that opts out is let through:
+ // `readFileSync(path.join(/* turbopackIgnore: true */ HERE, "a.json"))`. Its
+ // own arguments are cut out before asking whether this call carries a source.
+ // That is a simplification, not how Turbopack reads it: the outer call traces
+ // the join's value all the same (release 15, slice UT, measured). On a
+ // cwd-derived value that value is a known path, so the outer call traces the
+ // one file it names (a changelog), or the files of one known directory (the
+ // brand kits). Where the join names a directory, or a dynamic part could
+ // reach past the files the call reads, give the outer call its own opt-out.
+ const withoutOptedOut = (args) => {
+ let out = args;
+ for (let i = out.search(new RegExp(`\\(\\s*${MARK}`)); i !== -1; i = out.search(new RegExp(`\\(\\s*${MARK}`))) {
+ out = out.slice(0, i) + out.slice(closeOf(out, i));
+ }
+ return out;
+ };
+ const out = [];
+ for (let m; (m = SINK.exec(s)); ) {
+ const open = SINK.lastIndex - 1;
+ const args = s.slice(open + 1, closeOf(s, open) - 1);
+ if (args.trimStart().startsWith(MARK)) continue;
+ if (!carries(withoutOptedOut(args))) continue;
+ const line = s.slice(0, m.index).split("\n").length;
+ out.push({ line, call: text.split("\n")[line - 1].trim() });
+ }
+ return out;
+}
+
+/** Every module under `dir`, tests and declarations aside. */
+function modulesUnder(dir, out = []) {
+ for (const e of readdirSync(dir, { withFileTypes: true })) {
+ if (e.name === "node_modules" || e.name.startsWith(".")) continue;
+ const p = path.join(dir, e.name);
+ if (e.isDirectory()) modulesUnder(p, out);
+ else if (/\.(?:mjs|cjs|js|ts|tsx)$/.test(e.name) && !/\.test\./.test(e.name) && !e.name.endsWith(".d.ts")) out.push(p);
+ }
+ return out;
+}
+
+const MODULE_EXT = [".ts", ".tsx", ".mjs", ".js", ".cjs"];
+const isModule = (p) => MODULE_EXT.some((x) => p.endsWith(x)) && !/\.test\./.test(p) && !p.endsWith(".d.ts");
+const isFile = (p) => {
+ try {
+ return statSync(p).isFile();
+ } catch {
+ return false;
+ }
+};
+
+/** The module a relative specifier names, the way the bundler resolves it, or null. */
+function resolveRelative(from, spec) {
+ const base = path.resolve(path.dirname(from), spec);
+ const candidates = [base, ...MODULE_EXT.map((x) => base + x), ...MODULE_EXT.map((x) => path.join(base, "index" + x))];
+ return candidates.find((c) => isModule(c) && isFile(c)) ?? null;
+}
+
+// `import … from "./x"`, `export … from "../x"`, `import "./x"`, `import("./x")`,
+// `require("./x")`: the relative specifiers only. A package import (`next`,
+// `yt-dlp-transcript-common/…`) is covered by the package's own directory in
+// the set, or is not this repo's code.
+const RELATIVE_IMPORT = /(?:\bfrom\s*|\bimport\s*\(?\s*|\brequire\s*\(\s*)(["'])(\.{1,2}\/[^"'\n]+)\1/g;
+
+/**
+ * `files` plus every module of this repo they reach by relative imports, to
+ * any depth. A directory list alone misses a module one app imports from a
+ * folder the list treats as CLI-only (`common/bin/_publicFile.ts`, imported by
+ * `common/publish/source.ts`; umtool's `song/pitch.mjs`, imported by
+ * `lib/verdict.ts`) or keeps outside `app/` (`homepage/content/docs.ts`).
+ */
+export function withRelativeImports(files) {
+ const seen = new Set(files);
+ const queue = [...files];
+ while (queue.length) {
+ const file = queue.pop();
+ for (const m of readFileSync(file, "utf8").matchAll(RELATIVE_IMPORT)) {
+ const target = resolveRelative(file, m[2]);
+ if (!target || seen.has(target) || target.includes(`${path.sep}node_modules${path.sep}`)) continue;
+ if (path.relative(REPO, target).startsWith("..")) continue;
+ seen.add(target);
+ queue.push(target);
+ }
+ }
+ return [...seen];
+}
+
+/**
+ * umtool's modules that a Next build can reach: the app, its components and
+ * libs, the report pipeline, and whatever of `song/` they import (`paths.mjs`
+ * through `lib/paths.mjs`, `pitch.mjs`, `reasons.mjs` and the rest through the
+ * libs; the other song scripts are CLIs nothing in the app imports).
+ */
+function umtoolModules() {
+ const out = [];
+ for (const d of ["app", "components", "lib"]) modulesUnder(path.join(UMTOOL, d), out);
+ for (const e of readdirSync(path.join(UMTOOL, "report-to-video"))) {
+ if (e.endsWith(".mjs") && !e.includes(".test.")) out.push(path.join(UMTOOL, "report-to-video", e));
+ }
+ return withRelativeImports(out);
+}
+
+/**
+ * The editor's, the export's and the homepage's modules a Next build can
+ * reach: each app's `app/` (the editor's `lib/` and `instrumentation.ts` too),
+ * every module of common/ but its CLIs in `bin/`, and every module those reach
+ * by a relative import — which brings in the few `bin/` modules an app does
+ * import (`_publicFile.ts`) and the homepage's `content/docs.ts`. The common set
+ * is wider than what the apps import today, on purpose: a module that starts
+ * being imported is already covered.
+ */
+function nextAppModules() {
+ const out = [];
+ for (const d of ["homepage/app", "export/app", "editor/app", "editor/lib"]) modulesUnder(path.join(REPO, d), out);
+ out.push(path.join(REPO, "editor", "instrumentation.ts"));
+ for (const e of readdirSync(path.join(REPO, "common"), { withFileTypes: true })) {
+ if (!e.isDirectory() || e.name === "bin" || e.name === "node_modules" || e.name.startsWith(".")) continue;
+ modulesUnder(path.join(REPO, "common", e.name), out);
+ }
+ return withRelativeImports(out);
+}
+
+function untracedIn(files) {
+ const bad = [];
+ for (const file of files) {
+ for (const f of untracedCalls(readFileSync(file, "utf8"))) {
+ bad.push(`${path.relative(REPO, file)}:${f.line} ${f.call}`);
+ }
+ }
+ return bad;
+}
+
+test("the check flags slice Q's join and passes the opted-out form", () => {
+ const bad = [
+ 'const REPO_ROOT = findRepoRoot(process.cwd());',
+ 'export const CHANNELS_DIR = path.resolve(',
+ ' process.env.CHANNELS_DIR ?? path.join(REPO_ROOT, "transcripts", "channels"),',
+ ');',
+ ].join("\n");
+ const found = untracedCalls(bad);
+ assert.ok(found.some((f) => f.call.includes('path.join(REPO_ROOT, "transcripts"')), JSON.stringify(found));
+
+ const good = bad
+ .replace("path.resolve(", "path.resolve(/* turbopackIgnore: true */")
+ .replace("path.join(REPO_ROOT", "path.join(/* turbopackIgnore: true */ REPO_ROOT");
+ assert.deepEqual(untracedCalls(good), []);
+});
+
+test("sources: cwd, import.meta, __dirname; not homedir, env, or a comment", () => {
+ assert.equal(untracedCalls('const X = path.join(process.cwd(), "song");').length, 1);
+ assert.equal(untracedCalls("const H = path.dirname(fileURLToPath(import.meta.url));").length, 1);
+ assert.equal(untracedCalls('readFileSync(path.join(__dirname, "a.json"));').length, 2);
+ assert.equal(untracedCalls('const R = path.join(os.homedir(), "reports");').length, 0);
+ assert.equal(untracedCalls('const R = path.join(process.env.X, "channels");').length, 0);
+ assert.equal(untracedCalls('// path.join(process.cwd(), "x")\nconst y = 1;').length, 0);
+ // An object key named like a tainted value is not a use of it.
+ assert.equal(untracedCalls('const cwd = path.join(/* turbopackIgnore: true */ process.cwd(), "s");\nf(path.join(a, { cwd: 1 }));').length, 0);
+});
+
+test("the check flags the homepage's source.ts and paths.ts' walk as main had them", () => {
+ // homepage/app/lib/source.ts before the opt-out: a DIRECTORY join on the cwd.
+ const source = [
+ "export function loadSourceManifest(",
+ ' pub: string = path.join(process.cwd(), "public"),',
+ "): SourceManifest | null {",
+ ' return fs.readFileSync(path.join(pub, "source", "manifest.json"), "utf8");',
+ "}",
+ ].join("\n");
+ assert.ok(untracedCalls(source).some((f) => f.call.includes('path.join(process.cwd(), "public")')));
+ // common/lib/paths.ts before: the walk, and a join on the root it returns.
+ const walk = [
+ "function findMonorepoRoot(): string {",
+ " let dir = process.cwd();",
+ ' if (fs.existsSync(path.join(dir, "pnpm-workspace.yaml"))) return dir;',
+ " return process.cwd();",
+ "}",
+ "const monorepoRoot = findMonorepoRoot();",
+ 'const transcriptsDir = process.env.TRANSCRIPTS_DIR ?? path.join(monorepoRoot, "transcripts");',
+ ].join("\n");
+ const found = untracedCalls(walk).map((f) => f.call);
+ assert.ok(found.some((c) => c.includes('path.join(dir, "pnpm-workspace.yaml")')), JSON.stringify(found));
+ assert.ok(found.some((c) => c.includes('path.join(monorepoRoot, "transcripts")')), JSON.stringify(found));
+});
+
+test("brackets inside a regex literal or a string do not end a call early", () => {
+ // A regex holding a quote used to swallow the rest of the file as one body.
+ const text = [
+ "function esc(s) { return s.replace(/['\"&]/g, \"\"); }",
+ "const out = path.join(dir, esc(x));",
+ "if (import.meta.url === `file://${process.argv[1]}`) main();",
+ ].join("\n");
+ assert.deepEqual(untracedCalls(text), []);
+});
+
+test("no umtool module the app can import joins a cwd-derived path without opting out", () => {
+ const bad = untracedIn(umtoolModules());
+ assert.deepEqual(
+ bad,
+ [],
+ "add /* turbopackIgnore: true */ as the first argument (see this file's header):\n" + bad.join("\n"),
+ );
+});
+
+test("the scan set follows relative imports out of the listed folders", () => {
+ const rel = (files) => new Set(files.map((f) => path.relative(REPO, f)));
+ const um = rel(umtoolModules());
+ for (const m of ["paths", "reasons", "archive-url", "pitch", "flatness", "clipwindow", "deplosive", "orderfeat"]) {
+ assert.ok(um.has(`umtool/song/${m}.mjs`), `umtool/song/${m}.mjs is not scanned`);
+ }
+ // A song CLI nothing in the app imports stays out.
+ assert.ok(!um.has("umtool/song/build-um.mjs"));
+ const apps = rel(nextAppModules());
+ assert.ok(apps.has("common/bin/_publicFile.ts"), "common/bin/_publicFile.ts is not scanned");
+ assert.ok(apps.has("homepage/content/docs.ts"), "homepage/content/docs.ts is not scanned");
+ assert.ok(!apps.has("common/bin/compose-site.ts"), "a common CLI no app imports is scanned");
+});
+
+/**
+ * Every `.nft.json` a build wrote under its directory `dist`, its cache and
+ * dev-server output (`dist/cache`, `dist/dev`) aside. Only those two: a route
+ * directory named `cache` or `dev` deeper down is read like any other.
+ */
+function traceFilesUnder(dist, dir = dist, out = []) {
+ for (const e of readdirSync(dir, { withFileTypes: true })) {
+ const p = path.join(dir, e.name);
+ if (e.isDirectory()) {
+ if (dir !== dist || (e.name !== "cache" && e.name !== "dev")) traceFilesUnder(dist, p, out);
+ } else if (e.name.endsWith(".nft.json")) out.push(p);
+ }
+ return out;
+}
+
+/**
+ * Why `abs`, a file a build traced, must not be in the trace, or null.
+ *
+ * The build's own directory (`dist`) holds the chunks every trace lists, and
+ * `node_modules/.pnpm` is where pnpm keeps the packages; any other path through
+ * a directory or file whose name starts with a dot is something no server
+ * needs at run time — a fixture (`.e2e-song`, where the e2e fixture links the
+ * song data), another build (`.next-e2e`), a secret (`.env.local`), `.git` —
+ * and is the mark of a pattern Turbopack could not bound. So are the corpus
+ * and anything outside the repo.
+ */
+export function forbiddenTrace(abs, dist) {
+ const rel = path.relative(REPO, abs);
+ if (rel === "" || rel.startsWith("..") || path.isAbsolute(rel)) return "outside the repo";
+ if (abs.startsWith(dist + path.sep)) return null;
+ const parts = rel.split(path.sep);
+ if (parts[0] === "transcripts") return "the corpus";
+ for (let i = 0; i < parts.length; i += 1) {
+ if (!parts[i].startsWith(".")) continue;
+ if (parts[i] === ".pnpm" && parts[i - 1] === "node_modules") continue;
+ return `under ${parts.slice(0, i + 1).join("/")}`;
+ }
+ return null;
+}
+
+test("traceFilesUnder: only the build's own cache/ and dev/ are left out", () => {
+ const dist = mkdtempSync(path.join(tmpdir(), "next-build-trace-"));
+ try {
+ for (const f of ["cache/a.nft.json", "dev/b.nft.json", "server/app/api/cache/route.js.nft.json", "server/app/dev/page.js.nft.json"]) {
+ mkdirSync(path.dirname(path.join(dist, f)), { recursive: true });
+ writeFileSync(path.join(dist, f), '{"files":[]}');
+ }
+ const found = traceFilesUnder(dist).map((f) => path.relative(dist, f)).sort();
+ assert.deepEqual(found, ["server/app/api/cache/route.js.nft.json", "server/app/dev/page.js.nft.json"]);
+ } finally {
+ rmSync(dist, { recursive: true, force: true });
+ }
+});
+
+test("forbiddenTrace: a fixture, another build, a secret, the corpus, outside the repo", () => {
+ const dist = path.join(UMTOOL, ".next");
+ const at = (p) => forbiddenTrace(path.join(REPO, p), dist);
+ assert.equal(at("umtool/.next/server/chunks/ssr/a.js"), null);
+ assert.equal(at("node_modules/.pnpm/next@16.2.3/node_modules/next/dist/server/next.js"), null);
+ assert.equal(at("umtool/lib/paths.mjs"), null);
+ assert.equal(at("umtool/.e2e-song/data/planted/x/config.json"), "under umtool/.e2e-song");
+ assert.equal(at("umtool/.next-e2e/dev/server/a.js"), "under umtool/.next-e2e");
+ assert.equal(at("umtool/.env.local"), "under umtool/.env.local");
+ assert.equal(at(".git/config"), "under .git");
+ assert.equal(at("transcripts/channels/x/config.json"), "the corpus");
+ assert.equal(forbiddenTrace(path.resolve(REPO, "..", "elsewhere", "a.json"), dist), "outside the repo");
+});
+
+// The static checks above cannot see a value Turbopack reads through an
+// import: `path.join(CACHE_DIR, `${stamp}.${asMp3 ? "mp3" : "wav"}`)` in
+// umtool's clip-audio route took umtool's dot-directories into that route's
+// trace, the fixture and the e2e build's directory included (plans/release-15.md,
+// slice UT). So the last build's traces are read back, when there is one.
+//
+// What this can see: umtool/next.config.ts now excludes `.e2e-song`,
+// `.next-e2e` and `.env*` from every trace, so a pattern like that one shows
+// here only through a name the excludes miss -- `test-results/.last-run.json`
+// after an e2e run, `.next-shots`, the corpus, a path outside the repo. A
+// checkout with no e2e run behind it is blind to it; the fix at the call is
+// what keeps the route clean.
+test("umtool's last build traced no dot-directory, no corpus file and nothing outside the repo", (t) => {
+ const dist = path.join(UMTOOL, ".next");
+ const id = path.join(dist, "BUILD_ID");
+ if (!existsSync(path.join(dist, "server")) || !existsSync(id)) {
+ t.skip("no umtool build to read (umtool/.next/server); `pnpm --filter umtool exec next build` makes one");
+ return;
+ }
+ // A build older than the code that decides its traces judges code that is
+ // gone: after a merge or a checkout it would fail on a call already fixed.
+ // umtool runs under `next dev` day to day, so nothing else refreshes it.
+ const built = statSync(id).mtimeMs;
+ const newer = [path.join(UMTOOL, "next.config.ts"), ...umtoolModules()].filter((f) => statSync(f).mtimeMs > built);
+ if (newer.length) {
+ t.skip(
+ `umtool/.next was built ${new Date(built).toISOString()}, before ${path.relative(REPO, newer[0])}` +
+ ` (${newer.length} changed since); rebuild umtool (\`pnpm --filter umtool exec next build\`) to check its traces`,
+ );
+ return;
+ }
+ const bad = [];
+ for (const nft of traceFilesUnder(dist)) {
+ const { files } = JSON.parse(readFileSync(nft, "utf8"));
+ for (const f of files) {
+ const why = forbiddenTrace(path.resolve(path.dirname(nft), f), dist);
+ if (why) bad.push(`${path.relative(dist, nft)}: ${f} (${why})`);
+ }
+ }
+ assert.deepEqual(
+ bad.slice(0, 20),
+ [],
+ `${bad.length} traced file(s) no server needs; find the fs or path call whose value Turbopack could not bound (this file's header):\n` +
+ bad.slice(0, 20).join("\n"),
+ );
+});
+
+test("no module the editor, the export or the homepage can bundle joins a cwd-derived path without opting out", () => {
+ const files = nextAppModules();
+ assert.ok(files.length > 500, `only ${files.length} modules found`);
+ const bad = untracedIn(files);
+ assert.deepEqual(
+ bad,
+ [],
+ "add /* turbopackIgnore: true */ as the first argument (see this file's header):\n" + bad.join("\n"),
+ );
+});
diff --git a/umtool/app/api/browse/thumbs/route.ts b/umtool/app/api/browse/thumbs/route.ts
@@ -66,8 +66,9 @@ export async function POST(request: Request) {
}
// `file` names WHICH RENDERED VARIANT is being accepted, and accept-thumb
- // takes it verbatim as the new `out`. It must resolve inside the roots, and
- // it is passed absolute so the script cannot resolve it against its own cwd.
+ // records it as the new `out` (relative to SONG_REPORTS when inside it). It
+ // must resolve inside the roots, and it is passed absolute so the script
+ // cannot resolve it against its own cwd.
const argv = [process.execPath, path.join(SONG_CODE, "accept-thumb.mjs"), name];
if (file) {
const abs = resolveInRoots(file);
diff --git a/umtool/app/api/clip/[key]/audio/route.ts b/umtool/app/api/clip/[key]/audio/route.ts
@@ -1,10 +1,9 @@
import { createHash } from "node:crypto";
import { existsSync } from "node:fs";
import { mkdir, readFile, writeFile, rename } from "node:fs/promises";
-import path from "node:path";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
-import { CACHE_DIR } from "@/lib/paths";
+import { CACHE_DIR, cacheFile } from "@/lib/paths";
import { sourceWav } from "@/lib/clips";
import { readWavWindow, encodeWav, peakOver } from "@/lib/wav";
@@ -55,7 +54,8 @@ export async function GET(request: Request, ctx: { params: Promise<{ key: string
.update(`${video}|${from.toFixed(3)}|${to.toFixed(3)}|${asMp3 ? "mp3" : "wav"}`)
.digest("hex")
.slice(0, 16);
- const cached = path.join(CACHE_DIR, `${stamp}.${asMp3 ? "mp3" : "wav"}`);
+ // Through cacheFile, never a join here: see its comment in lib/paths.mjs.
+ const cached = cacheFile(`${stamp}.${asMp3 ? "mp3" : "wav"}`);
const type = asMp3 ? "audio/mpeg" : "audio/wav";
if (existsSync(cached)) {
@@ -79,8 +79,8 @@ export async function GET(request: Request, ctx: { params: Promise<{ key: string
const wav = encodeWav(x, region.sampleRate);
let body: Buffer = wav;
if (asMp3) {
- const tmpWav = path.join(CACHE_DIR, `${stamp}.in.wav`);
- const tmpMp3 = path.join(CACHE_DIR, `${stamp}.out.mp3`);
+ const tmpWav = cacheFile(`${stamp}.in.wav`);
+ const tmpMp3 = cacheFile(`${stamp}.out.mp3`);
await writeFile(tmpWav, wav);
await run("ffmpeg", [
"-nostdin", "-v", "error", "-y", "-i", tmpWav,
@@ -89,7 +89,7 @@ export async function GET(request: Request, ctx: { params: Promise<{ key: string
body = await readFile(tmpMp3);
await rename(tmpMp3, cached).catch(() => {});
} else {
- const tmp = `${cached}.tmp`;
+ const tmp = cacheFile(`${stamp}.wav.tmp`);
await writeFile(tmp, wav);
await rename(tmp, cached).catch(() => {});
}
diff --git a/umtool/app/api/clip/[key]/video/route.ts b/umtool/app/api/clip/[key]/video/route.ts
@@ -1,10 +1,9 @@
import { createHash } from "node:crypto";
import { existsSync } from "node:fs";
import { mkdir, readFile, rename } from "node:fs/promises";
-import path from "node:path";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
-import { CACHE_DIR } from "@/lib/paths";
+import { CACHE_DIR, cacheFile } from "@/lib/paths";
import { sourceVideo } from "@/lib/clips";
const run = promisify(execFile);
@@ -44,7 +43,8 @@ export async function GET(request: Request, ctx: { params: Promise<{ key: string
.update(`v1|${video}|${from.toFixed(3)}|${to.toFixed(3)}`)
.digest("hex")
.slice(0, 16);
- const cached = path.join(CACHE_DIR, `${stamp}.mp4`);
+ // Through cacheFile, never a join here: see its comment in lib/paths.mjs.
+ const cached = cacheFile(`${stamp}.mp4`);
if (existsSync(cached)) {
return new Response(new Uint8Array(await readFile(cached)), {
headers: { "content-type": "video/mp4", "cache-control": "no-store" },
@@ -52,7 +52,7 @@ export async function GET(request: Request, ctx: { params: Promise<{ key: string
}
await mkdir(CACHE_DIR, { recursive: true });
- const tmp = `${cached}.tmp.mp4`;
+ const tmp = cacheFile(`${stamp}.mp4.tmp.mp4`);
// -ss BEFORE -i for the fast seek, then -t for the length. Re-encoded rather
// than copied because a stream copy starts at the previous keyframe, which
// would slide the picture against the audio by up to several seconds.
diff --git a/umtool/app/api/face/frame/route.ts b/umtool/app/api/face/frame/route.ts
@@ -1,11 +1,10 @@
import { createHash } from "node:crypto";
import { existsSync } from "node:fs";
import { mkdir, readFile, rename } from "node:fs/promises";
-import path from "node:path";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { sourceVideo } from "@/lib/clips";
-import { CACHE_DIR } from "@/lib/paths";
+import { CACHE_DIR, cacheFile } from "@/lib/paths";
const run = promisify(execFile);
@@ -55,11 +54,12 @@ export async function GET(request: Request) {
.update(`face1|${video}|${at.toFixed(3)}|${w ?? "native"}`)
.digest("hex")
.slice(0, 16);
- const cached = path.join(CACHE_DIR, `${stamp}.jpg`);
+ // Through cacheFile, never a join here: see its comment in lib/paths.mjs.
+ const cached = cacheFile(`${stamp}.jpg`);
if (!existsSync(cached)) {
await mkdir(CACHE_DIR, { recursive: true });
- const tmp = `${cached}.tmp.jpg`;
+ const tmp = cacheFile(`${stamp}.jpg.tmp.jpg`);
// -ss BEFORE -i for the fast seek. When a width is asked for it is scaled to
// an even one with the aspect preserved; when it is not, the frame comes out
// at the source's own size and no mapping is needed at all.
diff --git a/umtool/components/BuildChain.tsx b/umtool/components/BuildChain.tsx
@@ -141,12 +141,13 @@ export default function BuildChain({ setId }: { setId: string }) {
</button>
</div>
- {/* Nothing here can be re-pointed at the whole-song .sh builds: they
- hardcode their own paths, ignore SONG_DIR, and take 20+ minutes. */}
+ {/* The whole-song .sh builds were never runnable here: they hardcoded
+ their own paths, ignored SONG_DIR, and took 20+ minutes. They were
+ deleted in release 12 (lib/jobs.ts). */}
<p className="mt-2 text-[11px] text-[var(--color-dim)]">
- The whole-song rebuild scripts are deliberately not runnable from here — they hardcode their
- paths, ignore <span className="font-mono">SONG_DIR</span>, and re-arrange
- non-deterministically. Run those from a shell.
+ The whole-song rebuild scripts were one-off shell run logs that hardcoded their paths
+ and ignored <span className="font-mono">SONG_DIR</span>; they are not in the tree, and
+ not runnable from here.
</p>
{error && <p className="mt-2 text-[11px] text-[var(--color-bad)]">{error}</p>}
diff --git a/umtool/docs/cli.md b/umtool/docs/cli.md
@@ -37,7 +37,16 @@ projects answering to one name is reported, never resolved by picking one.
## Environment
`REPORTS_DIR`, `SONG_REPORTS_DIR`, `SONG_DIR`, `CHANNELS_DIR`, `UMTOOL_INDEX_DIR`
-— which is how it is tested against the e2e fixture.
+— which is how it is tested against the e2e fixture. The path defaults
+(`lib/paths.mjs`, `song/paths.mjs`):
+
+| Variable | Default |
+|---|---|
+| `SONG_DIR` | `~/.local/share/archilyzer/song`, through its realpath — a symlink there is the supported way to keep the data where it is |
+| `SONG_REPORTS_DIR` | `~/reports/quartering-uh-song` |
+| `REPORTS_DIR` | `~/reports` (the parent of `SONG_REPORTS_DIR` when that is set) |
+| `CHANNELS_DIR` | `$TRANSCRIPTS_DIR/channels`, else the checkout's `transcripts/channels` (found by walking up from the cwd to `pnpm-workspace.yaml`) |
+| `VIDEO_ROOT` (`song/spec.mjs`, `song/video-dir.mjs`) | `~/reports/quartering-uh-song/videos` |
## `check` is the one to run before every build
diff --git a/umtool/e2e/deck.spec.ts b/umtool/e2e/deck.spec.ts
@@ -281,6 +281,14 @@ test("accepting a disjoint cover runs the CLI and holds the rule", async ({ requ
const view = await (await request.get("/api/browse/thumbs?song=alpha")).json();
expect(view.check.ok).toBe(true);
expect(view.check.videos).toBe(8);
+
+ // The CLI records `out` RELATIVE to the reports tree. The fixture's run log
+ // holds the absolute form, and the real thumb-accepted.json is tracked: an
+ // accept from this page must not write a machine's path back into it.
+ const onDisk = JSON.parse(
+ readFileSync(path.join(process.cwd(), ".e2e-song", "code", "thumb-accepted.json"), "utf8"),
+ ) as { thumbs: Record<string, { out: string }> };
+ expect(onDisk.thumbs["alpha-b"].out).toBe(path.join("thumbs", "alpha-b.jpg"));
});
// ---------------------------------------------------------------------------
diff --git a/umtool/e2e/fixtures/make-fixture.mjs b/umtool/e2e/fixtures/make-fixture.mjs
@@ -26,9 +26,9 @@ import { songCapabilities } from "./song-capabilities.mjs";
const CODE = path.resolve(path.dirname(new URL(import.meta.url).pathname), "..", "..", "song");
const dest = path.resolve(process.argv[2] ?? path.join(process.cwd(), ".e2e-song"));
-// A readdir that answers "nothing" instead of throwing. SONG_DATA's default is
-// the job temp dir the corpus was mined into, which on most machines no longer
-// exists — see the capabilities block below.
+// A readdir that answers "nothing" instead of throwing. SONG_DATA's default
+// (`~/.local/share/archilyzer/song`) does not exist on most machines — see the
+// capabilities block below.
const listDir = (p) => {
try {
return readdirSync(p);
@@ -99,8 +99,8 @@ writeFileSync(path.join(dest, "code", "um-manifest.json"), JSON.stringify({ vers
// -- a few candidates, from videos whose audio is actually present ------------
//
// TOLERANT OF A MISSING SONG_DATA, and that is the whole point of the
-// capabilities file below. `SONG_DIR`'s default is the job temp dir the corpus
-// was mined into, which on most machines no longer exists — so this readdir
+// capabilities file below. `SONG_DIR`'s default does not exist on most
+// machines — so this readdir
// used to throw and take the entire suite down before the test server started,
// including the two dozen specs that have nothing to do with the song project.
const cands = listDir(path.join(SONG_DATA, "cand2")).filter((f) => f.endsWith(".json"));
@@ -543,12 +543,16 @@ const thumbManifest = {
used: [],
};
writeFileSync(path.join(dest, "code", "thumb-manifest.json"), JSON.stringify(thumbManifest, null, 1));
+// The accepted alpha-c records its `out` RELATIVE to the reports tree, as the
+// tracked manifests do since release 12 (make-thumb and accept-thumb write it
+// that way), while the run log above keeps the older absolute form -- so the
+// suite reads both. deck.spec's "serves the ACCEPTED cover" is the relative one.
writeFileSync(
path.join(dest, "code", "thumb-accepted.json"),
JSON.stringify(
{
version: 1,
- thumbs: { "alpha-c": thumbManifest.thumbs["alpha-c"] },
+ thumbs: { "alpha-c": { ...thumbManifest.thumbs["alpha-c"], out: path.join("thumbs", "alpha-c.jpg") } },
used: [1, 2, 3, 4].map(slot),
},
null,
diff --git a/umtool/e2e/fixtures/song-capabilities.mjs b/umtool/e2e/fixtures/song-capabilities.mjs
@@ -5,8 +5,9 @@ import path from "node:path";
// readers.
//
// The 39 GB (`wav48/`, `asr/`, `media/`) is re-derivable from the archive and
-// deliberately not in the repo. `SONG_DIR`'s default is the job temp dir the
-// corpus was mined into, which on most machines no longer exists, so
+// deliberately not in the repo. `SONG_DIR`'s default
+// (`~/.local/share/archilyzer/song`, song/paths.mjs) is empty or absent on
+// most machines, so
// `make-fixture.mjs` builds an empty fixture and every spec that judges a clip
// used to fail — LOUDLY, as a red suite, over a machine that never had the data
// rather than over anything a change broke. Red that means "you are on a
diff --git a/umtool/lib/browse.ts b/umtool/lib/browse.ts
@@ -250,9 +250,10 @@ async function acceptedDoc(): Promise<ThumbDoc> {
export async function thumbFor(id: string): Promise<string | null> {
const accepted = acceptedFor(await acceptedDoc(), id, thumbAliasesFor(id));
if (accepted?.entry?.out) {
- // `out` is an ABSOLUTE path recorded on the machine that rendered it, and
- // the poster route resolves a thumb against SONG_REPORTS -- so anything
- // outside that root (a moved tree, a stale path) falls through rather than
+ // `out` is recorded relative to SONG_REPORTS (since release 12; older
+ // entries hold an ABSOLUTE path from the machine that rendered it), and the
+ // poster route resolves a thumb against SONG_REPORTS -- so anything outside
+ // that root (a moved tree, a stale path) falls through rather than
// producing a join that silently points at the wrong file.
const abs = resolveInRoots(accepted.entry.out);
if (abs && (abs === SONG_REPORTS || abs.startsWith(SONG_REPORTS + path.sep))) {
diff --git a/umtool/lib/jobs.ts b/umtool/lib/jobs.ts
@@ -5,16 +5,17 @@ import { ENV_DENY, type Step } from "./trim";
// Running a chain of steps, with the guardrails that make it safe to put behind
// a button.
//
-// WHAT IS DELIBERATELY NOT HERE: the whole-song build scripts.
+// WHAT WAS DELIBERATELY NEVER HERE: the whole-song build scripts.
//
// mk-rebuild.sh, pkmn-rebuild.sh, yoshi-rebuild.sh, rpg-remake-v5.sh and the
-// ms2-*.sh family all begin with a literal `cd /home/user/.claude/jobs/...`
-// and write to a literal reports path. They ignore SONG_DIR and
-// SONG_REPORTS_DIR entirely, which means an e2e fixture cannot contain them --
-// a spec that clicked the button would render into the real deliverables tree.
-// They also run 20+ minutes, and arrange-poly is non-deterministic across a
-// changed palette, so one click can invalidate a set of judgements already made
-// by ear. The UI prints their command instead.
+// ms2-*.sh family were one-off shell run logs that hardcoded their paths: each
+// began with a literal `cd` into the job temp dir the song was mined in and
+// wrote to a literal reports path, ignoring SONG_DIR and SONG_REPORTS_DIR, so
+// no e2e fixture could contain them -- a spec that clicked a button would have
+// rendered into the real deliverables tree. They also ran 20+ minutes, and
+// arrange-poly is non-deterministic across a changed palette, so one click
+// could invalidate a set of judgements already made by ear. They are not in
+// the tree any more (release 12), and nothing here can run them.
//
// What runs here is the .mjs chain, which honours SONG_DATA through
// song/paths.mjs and finishes in seconds to minutes.
@@ -68,8 +69,8 @@ export const getJob = (id: string) => jobs.get(id) ?? null;
export const recentJobs = (n = 10) =>
[...jobs.values()].sort((a, b) => b.startedAt - a.startedAt).slice(0, n);
-/** The accidental-hour-long-job guard. None of the .sh builds fits under any
- * cap worth setting, which is the other reason they stay out.
+/** The accidental-hour-long-job guard. None of the old .sh builds fitted under
+ * any cap worth setting, which was the other reason they were never here.
*
* A step may ask for more (Step.timeoutMs). A 19-clip crossfaded report build
* runs 20 to 40 minutes and would otherwise be SIGKILLed at 15 -- but raising
diff --git a/umtool/lib/paths.mjs b/umtool/lib/paths.mjs
@@ -4,26 +4,40 @@
// directory is read and which is written must not be able to differ between
// `umtool ls` and the page it is supposed to describe. lib/paths.ts re-exports
// everything here with types; nothing computes a root twice.
+import { existsSync } from "node:fs";
import os from "node:os";
import path from "node:path";
-import { SONG_DATA } from "../song/paths.mjs";
+import { SONG_DATA, SONG_REPORTS } from "../song/paths.mjs";
-export { SONG_DATA };
+// SONG_REPORTS -- the um-song deliverables tree, and BROWSE_ROOT's parent -- is
+// defined in song/paths.mjs beside SONG_DATA, because the song scripts that
+// record paths relative to it (make-thumb, accept-thumb) import only siblings.
+export { SONG_DATA, SONG_REPORTS };
// Derived output (sliced mp3s, waveform peaks, the project index). Lives with
// the data, not in the repo, and is safe to delete at any time.
export const CACHE_DIR = path.join(SONG_DATA, ".cache", "umtool");
+/**
+ * A file in CACHE_DIR, by name. A route names its cache files through this
+ * rather than joining CACHE_DIR itself. Turbopack reads a path it can see as a
+ * pattern of files to trace, in the join and in every fs call its value
+ * reaches, and CACHE_DIR is unknown to it (an env var or the home directory).
+ * So `path.join(CACHE_DIR, `${stamp}.${asMp3 ? "mp3" : "wav"}`)` in the
+ * clip-audio route was a pattern that reached into umtool's dot-directories:
+ * that route's trace listed the e2e fixture, the e2e server's build directory
+ * and `.env.local` (plans/release-15.md, slice UT). A value returned by a
+ * function from another module is opaque to it, so a call site traces nothing.
+ */
+export function cacheFile(name) {
+ return path.join(/* turbopackIgnore: true */ CACHE_DIR, name);
+}
+
// Render scratch: the body render and the cut background sit in the job temp
// dir ABOVE SONG_DATA, not inside it, because render-poly.mjs writes them next
// to its logs.
export const SONG_SCRATCH = path.dirname(SONG_DATA);
-/** The um-song deliverables tree. Also BROWSE_ROOT's parent. */
-export const SONG_REPORTS = path.resolve(
- process.env.SONG_REPORTS_DIR ?? path.join(os.homedir(), "reports", "quartering-uh-song"),
-);
-
// ---------------------------------------------------------------------------
// REPORTS_ROOT -- the tree every PROJECT hangs off.
//
@@ -79,11 +93,56 @@ const dedupe = (list) => [...new Set(list.map((p) => path.resolve(p)))];
// render over it.
//
// Same env var the cue reader already uses (lib/projects/report.mjs
-// GLOBAL_CHANNELS_DIR), so a fixture that confines one confines both.
+// GLOBAL_CHANNELS_DIR, which now returns this one), so a fixture that confines
+// one confines both.
+//
+// With neither CHANNELS_DIR nor TRANSCRIPTS_DIR set, it is the checkout's own
+// transcripts/channels -- the corpus a plain checkout would have. It used to be
+// an absolute path in one machine's home directory, which every other clone
+// silently looked for and never found.
// ---------------------------------------------------------------------------
+
+/**
+ * The checkout this runs in: the nearest directory at or above `start` holding
+ * pnpm-workspace.yaml, or `start`'s parent when there is none.
+ *
+ * Walked from the CWD, not from import.meta.url: the Next app imports this
+ * module (eleven API routes), and Turbopack rewrites module URLs into
+ * .next/server/chunks, so a path derived from one points at the build output
+ * -- the SONG_CODE rule in lib/paths.ts. `next dev`, `next start` and the CLI
+ * all run with cwd inside the checkout (the app at umtool/, which is what the
+ * fallback assumes). report-to-video/cues.mjs keeps its import.meta.url walk:
+ * only its CLI entry points (build-video, resolve-windows) read that default.
+ *
+ * EVERY PATH OP ON A cwd-DERIVED VALUE CARRIES `turbopackIgnore`. Turbopack
+ * evaluates `process.cwd()` statically as the project, so an un-annotated
+ * `path.join(REPO_ROOT, "transcripts", "channels")` became a DIRECTORY ASSET
+ * REFERENCE: `next build` walked the whole corpus (hundreds of GB, `data/`
+ * symlinked to another drive) and was OOM-killed, or died on the first symlink
+ * out of the root. A worktree with no transcripts/ builds fine, which is how it
+ * shipped. The comment is Turbopack's per-expression opt-out (the form its own
+ * "whole project was traced" warning advises; the Next docs list the comment
+ * for import(), require(), require.resolve() and new Worker() only); the values
+ * at run time are unchanged. scripts/next-build-trace.test.mjs holds the line.
+ */
+export function findRepoRoot(start) {
+ let dir = path.resolve(/* turbopackIgnore: true */ start);
+ for (;;) {
+ if (existsSync(path.join(/* turbopackIgnore: true */ dir, "pnpm-workspace.yaml"))) return dir;
+ const up = path.dirname(/* turbopackIgnore: true */ dir);
+ if (up === dir) return path.resolve(/* turbopackIgnore: true */ start, "..");
+ dir = up;
+ }
+}
+
+export const REPO_ROOT = findRepoRoot(process.cwd());
+
export const CHANNELS_DIR = path.resolve(
+ /* turbopackIgnore: true */
process.env.CHANNELS_DIR ??
- "/home/user/Projects/yt-dlp-transcript-browser/transcripts/channels",
+ (process.env.TRANSCRIPTS_DIR
+ ? path.join(/* turbopackIgnore: true */ process.env.TRANSCRIPTS_DIR, "channels")
+ : path.join(/* turbopackIgnore: true */ REPO_ROOT, "transcripts", "channels")),
);
export const READ_ROOTS = dedupe(
diff --git a/umtool/lib/paths.ts b/umtool/lib/paths.ts
@@ -19,6 +19,7 @@ import path from "node:path";
// ---------------------------------------------------------------------------
export {
CACHE_DIR,
+ cacheFile,
INDEX_DIR,
MEDIA_ROOTS,
MIX_CACHE,
@@ -39,9 +40,9 @@ export {
// cwd at the package root.
export const SONG_CODE = process.env.SONG_CODE_DIR
? path.resolve(process.env.SONG_CODE_DIR)
- : path.join(process.cwd(), "song");
+ : path.join(/* turbopackIgnore: true */ process.cwd(), "song");
-export const stateFile = (name: string) => path.join(SONG_CODE, name);
+export const stateFile = (name: string) => path.join(/* turbopackIgnore: true */ SONG_CODE, name);
// dataFile needs SONG_DATA at module scope, which the re-export above does not
// bind locally -- so it is imported again rather than duplicated.
diff --git a/umtool/lib/projects/report.mjs b/umtool/lib/projects/report.mjs
@@ -9,6 +9,7 @@ import { readdir, readFile, stat } from "node:fs/promises";
import path from "node:path";
import { DEFAULT_VARIANT, cachedWindowsFor } from "umtool-report-to-video/build-video";
import { rawCacheOf } from "../report/raw-cache.mjs";
+import { CHANNELS_DIR } from "../paths.mjs";
import { channelName, cleanTitle } from "umtool-report-to-video/attribution";
/**
@@ -35,9 +36,10 @@ export { widen };
export const MANIFEST_NAME = "video.manifest.json";
-export const GLOBAL_CHANNELS_DIR = () =>
- process.env.CHANNELS_DIR ??
- "/home/user/Projects/yt-dlp-transcript-browser/transcripts/channels";
+// ONE definition: lib/paths.mjs's CHANNELS_DIR (the env var, else
+// TRANSCRIPTS_DIR/channels, else the checkout's own transcripts/channels). The
+// bench's read root and this reader cannot disagree about which corpus it is.
+export const GLOBAL_CHANNELS_DIR = () => CHANNELS_DIR;
/** The conventional name make-shadow-channels.sh builds. */
export const SHADOW_CHANNELS = ".shadow-channels";
diff --git a/umtool/lib/report/driver.mjs b/umtool/lib/report/driver.mjs
@@ -11,9 +11,9 @@
import path from "node:path";
/** Where the pipeline lives. One place, so a move is one edit. */
-export const PIPELINE_DIR = path.resolve(process.cwd(), "report-to-video");
+export const PIPELINE_DIR = path.resolve(/* turbopackIgnore: true */ process.cwd(), "report-to-video");
-const script = (name) => path.join(PIPELINE_DIR, name);
+const script = (name) => path.join(/* turbopackIgnore: true */ PIPELINE_DIR, name);
/**
* Presets, in the order somebody actually works.
@@ -322,9 +322,9 @@ export function checkSourcesSteps(projects, env = {}) {
// ---------------------------------------------------------------------------
/** This package's own root. bin/ lives here, and so does report-to-video/. */
-export const UMTOOL_DIR = path.resolve(process.cwd());
+export const UMTOOL_DIR = path.resolve(/* turbopackIgnore: true */ process.cwd());
-const tool = (name) => path.join(UMTOOL_DIR, "bin", name);
+const tool = (name) => path.join(/* turbopackIgnore: true */ UMTOOL_DIR, "bin", name);
/** The interpreter a project's own scripts are run with. */
export const PYTHON = process.env.PYTHON_BIN ?? "python3";
diff --git a/umtool/lib/thumbs.ts b/umtool/lib/thumbs.ts
@@ -33,7 +33,22 @@ export type ThumbCorner = {
frameAt?: number;
crop?: { x: number; y: number; w: number; h: number };
};
-export type ThumbEntry = { out: string; bgAt?: number; corners?: ThumbCorner[] };
+export type ThumbEntry = {
+ /** The rendered cover. Relative to SONG_REPORTS (or absolute, in older entries):
+ * resolveInRoots binds a relative path to SONG_REPORTS, its first root. */
+ out: string;
+ /**
+ * The background video the frame was cut from. RELATIVE TO THE SONG DATA DIR
+ * (SONG_DATA), not to SONG_REPORTS: resolve a relative one with `dataFile(bg)`,
+ * NEVER with `resolveInRoots`, which binds a relative path to SONG_REPORTS and
+ * does not stat -- it would silently name a file that is not there. Absolute
+ * (use as is) when it lay outside SONG_DATA (song/make-thumb.mjs `relTo`).
+ * Nothing reads it yet.
+ */
+ bg?: string;
+ bgAt?: number;
+ corners?: ThumbCorner[];
+};
export type ThumbDoc = { version: number; thumbs: Record<string, ThumbEntry>; used?: string[] };
const EMPTY: ThumbDoc = { version: 1, thumbs: {} };
diff --git a/umtool/lib/tools.mjs b/umtool/lib/tools.mjs
@@ -19,8 +19,8 @@ import { SONG_SCRATCH } from "./paths.mjs";
export const facedetPython = () =>
process.env.FACEDET_PYTHON ?? path.join(SONG_SCRATCH, "facedet", "bin", "python");
export const songCode = () =>
- process.env.SONG_CODE_DIR ? path.resolve(process.env.SONG_CODE_DIR) : path.join(process.cwd(), "song");
-export const facecropPy = () => path.join(songCode(), "facecrop.py");
+ process.env.SONG_CODE_DIR ? path.resolve(process.env.SONG_CODE_DIR) : path.join(/* turbopackIgnore: true */ process.cwd(), "song");
+export const facecropPy = () => path.join(/* turbopackIgnore: true */ songCode(), "facecrop.py");
/**
* The tools, with the env override each pipeline script honours. `required`
diff --git a/umtool/lib/trim.ts b/umtool/lib/trim.ts
@@ -222,7 +222,7 @@ export function hookRecipe(
const trimDir = `mkvocals/hooks-trim-${stamp}`;
const takeDir = `mkvocals/hooks-best-${stamp}`;
const outOverlays = dataFile(pair.to.replace(/\.json$/, `-${stamp}.json`));
- const cwd = path.join(process.cwd(), "song");
+ const cwd = path.join(/* turbopackIgnore: true */ process.cwd(), "song");
const base = { SONG_DIR: SONG_DATA };
return [
diff --git a/umtool/next.config.ts b/umtool/next.config.ts
@@ -18,19 +18,37 @@ const nextConfig: NextConfig = {
// fail the whole module graph. Every page importing lib/projects then 500s
// with "Can't resolve 'cbor-x'", which names a package nothing here uses.
serverExternalPackages: ["lmdb"],
+ // No route's trace may list the e2e fixture (.e2e-song, where
+ // e2e/fixtures/make-fixture.mjs links the song data), the e2e server's own
+ // build directory (.next-e2e) or an env file: none is a run-time input. The
+ // clip-audio route's trace listed 1,704 such files (plans/release-15.md, slice
+ // UT). That was fixed at the call (lib/paths.mjs `cacheFile`); this is the
+ // second line, measured on its own: with the old route it takes the trace
+ // back to what the sibling routes list. scripts/next-build-trace.test.mjs
+ // reads the last build's traces back.
+ outputFileTracingExcludes: {
+ "/*": ["./.e2e-song/**/*", "./.next-e2e/**/*", "./.env*"],
+ },
turbopack: {
// Same reasoning as editor/next.config.ts: Turbopack infers the workspace
// root by walking up for the outermost lockfile, and a stray pnpm-lock.yaml
// above the checkout silently relocates it. Nothing here lives above the
// monorepo root, so pinning it costs nothing.
root: path.join(__dirname, ".."),
- // Suppress the harmless "whole project was traced unintentionally" NFT
- // warning, exactly as editor/next.config.ts does. It fires because the
- // server genuinely does runtime-dynamic fs reads it cannot statically bound
- // -- lib/paths.ts resolves SONG_DATA from an env var and the routes read
- // wav48/<video>.wav by name. We don't use `output: 'standalone'`, so the
- // .nft.json traces are never consumed and the over-tracing is cosmetic.
- // Scoped to this exact issue (path + title) so other warnings still surface.
+ // Silences the "whole project was traced unintentionally" warning, and only
+ // it (path + title), for one measured reason: 66 of the 68 routes trace
+ // umtool's own tree, its 361 files outside dot-directories, next.config.ts
+ // (the file the warning names) among them. A path or fs call on a value
+ // Turbopack cannot know (an env var, the home directory, a parameter) is a
+ // pattern over the project, and umtool has hundreds. Opting out every path
+ // op in the three path modules left 31 of the 68 routes clean; opting out
+ // all 319 in the 53 modules that have one left 49 clean, and the rest come
+ // through fs calls (lib/report/snapshots.mjs, among others). That walk skips
+ // dot-directories and does not enter symlinks, and the traces are not
+ // consumed while `output: "standalone"` stays off. The warning cannot tell
+ // that walk from one that does reach a dot-directory (both name
+ // next.config.ts), so that case is excluded above and checked after the
+ // build by scripts/next-build-trace.test.mjs instead.
ignoreIssue: [
{
path: "**/next.config.ts",
diff --git a/umtool/report-to-video/brand.mjs b/umtool/report-to-video/brand.mjs
@@ -37,17 +37,17 @@ import { readFileSync } from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
-const HERE = path.dirname(fileURLToPath(import.meta.url));
+const HERE = path.dirname(/* turbopackIgnore: true */ fileURLToPath(import.meta.url));
import { BRAND_CHOICES, BRAND_IDS } from "./brand-ids.mjs";
import { IBM_PLEX_SANS } from "./svg-faces.mjs";
export { BRAND_CHOICES, BRAND_IDS };
-export const FONTS_DIR = path.join(HERE, "fonts");
-export const FONTCONFIG_FILE = path.join(FONTS_DIR, "fonts.conf");
-export const MONO_FONT_FILE = path.join(FONTS_DIR, "IBMPlexMono-Regular.ttf");
+export const FONTS_DIR = path.join(/* turbopackIgnore: true */ HERE, "fonts");
+export const FONTCONFIG_FILE = path.join(/* turbopackIgnore: true */ FONTS_DIR, "fonts.conf");
+export const MONO_FONT_FILE = path.join(/* turbopackIgnore: true */ FONTS_DIR, "IBMPlexMono-Regular.ttf");
/** Plex Mono's static Bold: `render.fontBold`, which compose-chrome's HyperFrames band sets its bold in. */
-export const MONO_BOLD_FONT_FILE = path.join(FONTS_DIR, "IBMPlexMono-Bold.ttf");
+export const MONO_BOLD_FONT_FILE = path.join(/* turbopackIgnore: true */ FONTS_DIR, "IBMPlexMono-Bold.ttf");
/** The end card's default length: YouTube's end screen runs in the last 5–20 s. */
export const END_CARD_DEFAULT_SECONDS = 20;
@@ -79,7 +79,7 @@ export function brandKit(id) {
throw new Error(`render.brand "${id}" is not a preset — one of ${BRAND_IDS.join(", ")}`);
}
if (!kits.has(id)) {
- kits.set(id, JSON.parse(readFileSync(path.join(HERE, "brands", `${id}.json`), "utf8")));
+ kits.set(id, JSON.parse(readFileSync(path.join(/* turbopackIgnore: true */ HERE, "brands", `${id}.json`), "utf8")));
}
return kits.get(id);
}
diff --git a/umtool/report-to-video/cues.mjs b/umtool/report-to-video/cues.mjs
@@ -58,10 +58,10 @@ import { fileURLToPath } from "node:url";
// checkout would have is two levels up. Previously this defaulted to an absolute
// path inside the original author's home directory, which meant every other
// clone silently looked in a directory that does not exist.
-const REPO_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
+const REPO_ROOT = path.resolve(/* turbopackIgnore: true */ path.dirname(/* turbopackIgnore: true */ fileURLToPath(import.meta.url)), "..", "..");
export const DEFAULT_CHANNELS_DIR =
- process.env.CHANNELS_DIR ?? path.join(REPO_ROOT, "transcripts", "channels");
+ process.env.CHANNELS_DIR ?? path.join(/* turbopackIgnore: true */ REPO_ROOT, "transcripts", "channels");
const DEFAULT_CACHE_DIR =
process.env.REPORT_CACHE_DIR ??
diff --git a/umtool/song/accept-thumb.mjs b/umtool/song/accept-thumb.mjs
@@ -17,6 +17,7 @@
// means candidates are free and the guarantee still holds where it matters.
import { readFileSync, writeFileSync, existsSync } from "node:fs";
import path from "node:path";
+import { SONG_REPORTS, relTo } from "./paths.mjs";
const DIR = path.resolve(path.dirname(new URL(import.meta.url).pathname));
const MF = path.join(DIR, "thumb-manifest.json");
@@ -87,7 +88,12 @@ if (clash.length) {
process.exit(1);
}
-const out = arg ?? entry.out;
+// Recorded relative to SONG_REPORTS when it lies inside it, as make-thumb.mjs
+// records it: this file is tracked, and /api/browse/thumbs hands over an
+// ABSOLUTE `file` (resolved inside the roots), which would otherwise put one
+// machine's home directory back into it. The readers resolve a relative `out`
+// against SONG_REPORTS first (lib/paths.mjs resolveInRoots).
+const out = relTo(SONG_REPORTS, arg ?? entry.out);
if (arg && !existsSync(arg)) console.log(`note: ${arg} does not exist yet`);
accepted.thumbs[cmd] = { ...entry, out };
accepted.used = [...new Set(Object.values(accepted.thumbs).flatMap(cornersOf))];
diff --git a/umtool/song/backfill-build.sh b/umtool/song/backfill-build.sh
@@ -1,89 +0,0 @@
-#!/usr/bin/env bash
-# Backfill build.json for the songs whose recipe can actually be READ OFF a builder.
-#
-# WHAT IS HERE IS VERIFIED, AND THE GAPS ARE DELIBERATE.
-#
-# Only two builders write into videos/<song>/ themselves, which is what makes their
-# output attributable without guessing:
-#
-# pkmn-video.sh line 116 OUT=$V/<wide|vertical>[-short].mp4 all 4 cuts
-# ms2-jerbg.sh line 102 mux -> "$V/$NAME.mp4" wide, wide-short
-# line 104 mux -> "$V/variants/$NAME-umdrums.mp4"
-# line 84 ffmpeg -> "$V/variants/$NAME-nokit.mp4"
-#
-# The other three are NOT recorded, and must not be guessed at:
-#
-# mortal-kombat mk-fatal-finish3.sh takes OUT as a required env var and names no
-# plan at all -- it assembles audio. Which of the five files in
-# plan/ produced which cut is not in the script.
-# mario-rpg rpg-short.sh defaults PLANF=rpg-plan-short.json, and there is NO
-# such file in videos/mario-rpg/plan/ -- it holds rpg-plan-s2/s3
-# (the 2-voice vs 3-voice pair still awaiting a decision) and v4.
-# So the shipped cuts came from an invocation this script does not
-# describe.
-# yoshi yoshi-rebuild.sh defaults PLANF=yoshi-plan-v7.json, and plan/
-# holds the SHORT plans. Same problem.
-#
-# A build.json that says "probably this plan" is worse than one that says nothing:
-# the whole point of the file is that it can be trusted. The browse UI renders an
-# unrecorded file as unrecorded and offers the plan picker instead.
-#
-# Idempotent -- re-running just rewrites the same entries with a fresh `written`.
-set -euo pipefail
-cd "$(dirname "$0")"
-
-D=/home/user/reports/quartering-uh-song
-V=${VIDEO_ROOT:-$D/videos}
-export VIDEO_ROOT=$V
-
-rec () { node video-dir.mjs "$@"; }
-
-echo "--- pokemon (pkmn-video.sh: FORMAT x CUT, writes each cut directly)"
-JINGLE=pkc-jingle-merged.json
-for fmt in wide vertical; do
- for cut in long short; do
- rel=$fmt$( [ "$cut" = short ] && echo -short || echo "" ).mp4
- [ -f "$V/pokemon/$rel" ] || { echo " skip $rel (not built)"; continue; }
- if [ "$cut" = long ]; then BODY=pkmn-v12-body.json; RUN=pkmn-v12-run.json
- else BODY=pkmn-short-body.json; RUN=pkmn-short-run.json; fi
- rec --song pokemon --rel "$rel" \
- --script pkmn-video.sh \
- --env "FORMAT=$fmt" --env "CUT=$cut" \
- --plan "$BODY" --plan "$RUN" --plan "$JINGLE" \
- --note "body + run + the catch jingle; the shape is the video, see the script header"
- done
-done
-
-echo "--- metal-slug (ms2-jerbg.sh: jer's edit, handover on MISSION 1 START)"
-BG=$D/jer-metalslug-bg.mp4
-ANN=$D/MS_Mission_1_start.ogg
-for name in wide wide-short; do
- case $name in
- wide) PLAN=ms2-plan-11-v3.json ;;
- wide-short) PLAN=ms2-plan-short2.json ;;
- esac
- # The shipped cut carries the game's own kit; -umdrums swaps it for the um kit;
- # -nokit is the same mix with no drums at all. One recipe, three outputs.
- for rel in "$name.mp4" "variants/$name-nokit.mp4" "variants/$name-umdrums.mp4"; do
- [ -f "$V/metal-slug/$rel" ] || { echo " skip $rel (not built)"; continue; }
- case $rel in
- *-nokit.mp4) NOTE="the handover mix with no drum kit" ;;
- *-umdrums.mp4) NOTE="the um kit in place of the game's" ;;
- *) NOTE="shipped: the game's own kit, arrangement shifted +0.669s onto MISSION 1 START" ;;
- esac
- rec --song metal-slug --rel "$rel" \
- --script ms2-jerbg.sh \
- --env "BG=$BG" --env "ANNOUNCER=$ANN" \
- --plan "$PLAN" \
- --asset "$BG" --asset "$ANN" \
- --note "$NOTE"
- done
-done
-
-echo
-echo "NOT recorded, and deliberately so -- the builder does not say which plan:"
-echo " mortal-kombat/* mk-fatal-finish3.sh names no plan (OUT is a required env var)"
-echo " mario-rpg/* rpg-short.sh defaults to rpg-plan-short.json, absent from plan/"
-echo " yoshi/* yoshi-rebuild.sh defaults to yoshi-plan-v7.json, absent from plan/"
-echo
-node video-dir.mjs --list
diff --git a/umtool/song/build-um.mjs b/umtool/song/build-um.mjs
@@ -248,7 +248,14 @@ const MF = path.join(DIR, "um-manifest.json");
let merged = new Map();
try { for (const it of JSON.parse(readFileSync(MF, "utf8")).items) merged.set(it.k, it); } catch {}
for (const it of items) merged.set(it.k, it);
-writeFileSync(MF, JSON.stringify({ version: 1, items: [...merged.values()] }, null, 1));
+// `vid` stays OUT of the manifest. The page needs it -- a file:// URL into this
+// machine's SONG_DATA/media, which `items` keeps in memory for META below --
+// but this file is tracked, and a tracked file must not carry a machine path.
+// Nothing reads `vid` back from here (lib/clips.ts has no such field).
+writeFileSync(
+ MF,
+ JSON.stringify({ version: 1, items: [...merged.values()].map(({ vid: _v, ...rest }) => rest) }, null, 1),
+);
console.log(`manifest holds ${merged.size} clips (${items.length} on this page)`);
// HUD target: Mortal Kombat is the current goal. 1,659 notes for three voices is
diff --git a/umtool/song/fit-hooks.mjs b/umtool/song/fit-hooks.mjs
@@ -90,7 +90,7 @@ const rms = (x, a, b) => {
};
// mk-mix-hooks resolves EVERY path against SONG_DATA, so it must be handed the
// relative names it was given -- an absolute path comes back doubled
-// (".../song/home/user/.../song/mk-overlays.json") and it dies on ENOENT.
+// (".../song/home/<user>/.../song/mk-overlays.json") and it dies on ENOENT.
const mix = (rel, env) => {
execFileSync("node", [path.join(DIR, "mk-mix-hooks.mjs"), SONG, OVL, rel],
{ env: { ...process.env, DRIVE, ...env }, stdio: ["ignore", "ignore", "inherit"] });
diff --git a/umtool/song/make-thumb.mjs b/umtool/song/make-thumb.mjs
@@ -13,7 +13,7 @@ import { readFileSync, writeFileSync, existsSync, mkdirSync, rmSync } from "node
import { execFileSync } from "node:child_process";
import path from "node:path";
-import { SONG_DATA } from "./paths.mjs";
+import { SONG_DATA, SONG_REPORTS, relTo } from "./paths.mjs";
const DIR = path.resolve(path.dirname(new URL(import.meta.url).pathname));
const [NAME, BG, CSV, OUT] = process.argv.slice(2);
const BG_TIME = process.argv[6] !== undefined ? Number(process.argv[6]) : null;
@@ -255,7 +255,13 @@ manifest.thumbs[NAME] = {
// left with a frame nobody could re-cut -- the obvious guess, that a cover's
// background is its own song video, is false for all four. Optional in exactly
// the way `corners[].crop` is: absent means "not recorded", not "no background".
- out: OUT, bg: BG, bgAt: +bgAt.toFixed(2),
+ //
+ // Both paths are recorded RELATIVE TO THEIR ROOT, because this file is tracked
+ // and an absolute path is one machine's home directory: `out` to SONG_REPORTS
+ // (the um-song deliverables tree, which lib/paths.mjs resolveInRoots binds a
+ // relative path to first), `bg` to SONG_DATA (the song's bulk data; nothing
+ // reads it back yet). A path outside its root stays absolute.
+ out: relTo(SONG_REPORTS, path.resolve(OUT)), bg: relTo(SONG_DATA, path.resolve(BG)), bgAt: +bgAt.toFixed(2),
// THE BOX IS RECORDED NOW. It used to be computed and thrown away -- the
// script kept only p[3] and p[4] -- so a corner could be cut and never cut
// again the same way, which is the state two accepted corners are in today.
diff --git a/umtool/song/mk-fatal-finish.sh b/umtool/song/mk-fatal-finish.sh
@@ -1,57 +0,0 @@
-#!/usr/bin/env bash
-# Finish the Mortal Kombat fatality build: put the GAME audio back under the
-# rendered song, then hand over to the fatality at full brightness.
-#
-# The renderer stops its audio at the last note (~144.2s) but leaves ~2.9s of
-# SILENT video after it, so the container duration is not the song's end. Cutting
-# on the container would put the freeze and the shatter inside the dimmed body.
-# The body is therefore cut at the last audible sample, measured here.
-#
-# Video: body (dimmed by the renderer) ++ background tail (full brightness).
-# Audio: the song, plus the game under it -- full through the gong, silent under
-# the song, back up as the song ends so the shatter is heard.
-set -euo pipefail
-cd "$(dirname "$0")"
-
-BODY=${BODY:-/home/user/.claude/jobs/efbe67a7/tmp/mkfatal.mp4}
-BG=${BG:-intro/mk-bg-fatal.mp4}
-OUT=${OUT:-/home/user/reports/quartering-uh-song/quartering-mk-fatality.mp4}
-
-# Where the FATALITY screen gives way to the tournament ladder. Ending here means
-# the video finishes ON the payoff instead of drifting into the next screen.
-END=${END:-153.10}
-# The game's gong is the select-screen sting at 18.086s, after a 0.9s silence --
-# NOT 17s, and an 18s gate would mute it.
-GONG_HOLD=${GONG_HOLD:-19.0} # game full until here
-GAME_OUT=${GAME_OUT:-20.8} # ...faded out by here, before the 21s cut
-FADE_UP=${FADE_UP:-1.4} # game rises over this long, landing on the join
-GAME_GAIN=${GAME_GAIN:-0.90}
-
-# Last audible sample of the body = the song's real end.
-T=$(ffmpeg -v error -i "$BODY" -ac 1 -ar 1000 -f s16le - | node -e '
-const c=[];process.stdin.on("data",d=>c.push(d));process.stdin.on("end",()=>{
-const b=Buffer.concat(c);const a=new Int16Array(b.buffer,b.byteOffset,b.length/2);
-let last=0; for(let i=0;i<a.length;i++) if(Math.abs(a[i])>8) last=i;
-console.log((last/1000+0.04).toFixed(3));});')
-
-FU=$(node -e "console.log(($T - $FADE_UP).toFixed(3))")
-TAILDUR=$(node -e "console.log(($END - $T).toFixed(3))")
-echo "song ends ${T}s · game rises ${FU}->${T}s · fatality tail ${T}->${END}s (${TAILDUR}s)"
-
-ffmpeg -y -v warning \
- -i "$BODY" \
- -i "$BG" \
- -ss "$T" -t "$TAILDUR" -i "$BG" \
- -filter_complex "
-[1:a]atrim=0:${END},asetpts=PTS-STARTPTS,volume='${GAME_GAIN}*(if(lt(t,${GONG_HOLD}),1.0, if(lt(t,${GAME_OUT}),1-(t-${GONG_HOLD})/(${GAME_OUT}-${GONG_HOLD}), if(lt(t,${FU}),0.0, min(1.0,(t-${FU})/${FADE_UP})))))':eval=frame[game];
-[0:a]atrim=0:${T},asetpts=PTS-STARTPTS,apad=whole_dur=${END}[song];
-[song][game]amix=inputs=2:duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a];
-[0:v]trim=0:${T},setpts=PTS-STARTPTS[bodyv];
-[2:v]scale=1280:720,fps=30,setsar=1,format=yuv420p,setpts=PTS-STARTPTS[tailv];
-[bodyv][tailv]concat=n=2:v=1[v]" \
- -map "[v]" -map "[a]" -shortest \
- -c:v libx264 -preset medium -crf 18 -pix_fmt yuv420p \
- -c:a aac -b:a 192k -movflags +faststart "$OUT"
-
-echo "wrote $OUT"
-ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 "$OUT"
diff --git a/umtool/song/mk-fatal-finish2.sh b/umtool/song/mk-fatal-finish2.sh
@@ -1,64 +0,0 @@
-#!/usr/bin/env bash
-# Assemble the Mortal Kombat fatality build from an already-mixed audio track.
-#
-# Ending, as asked for:
-# song ends -> a beat of DEAD AIR -> the game returns QUIETLY (you can see the
-# fatality land, you can only just hear it) -> the game rises to full during
-# its OWN natural quiet gap (source 260.3-261.2s) -> the "dun dun dun" fanfare
-# at source 261.3s is the first sound back at FULL volume, with FATALITY on
-# screen.
-#
-# Background is mk-bg-fatal2.mp4: source 0->21.0 ++ 128.5->266.0, so beyond the
-# cut output = source - 107.5.
-set -euo pipefail
-cd "$(dirname "$0")"
-
-BODY=${BODY:-/home/user/.claude/jobs/efbe67a7/tmp/mkfatal2.mp4} # video (+ the un-hooked song, unused here)
-AUDIO=${AUDIO:?need the mixed audio wav}
-BG=${BG:-intro/mk-bg-fatal2.mp4}
-OUT=${OUT:?need an output path}
-
-DEAD=${DEAD:-2.0} # seconds of silence after the song
-LOW=${LOW:-0.25} # how loud the game is during the fatality itself
-LOW_IN=${LOW_IN:-0.8} # how long it takes to reach LOW
-FULL_AT=${FULL_AT:-152.80} # start of the game's own quiet gap (source 260.30)
-FULL_BY=${FULL_BY:-153.70} # ...end of it (source 261.20); fanfare is at 153.80
-END=${END:-156.10} # FATALITY screen gives way to the ladder (source 263.60)
-GONG_HOLD=${GONG_HOLD:-19.0}
-GAME_OUT=${GAME_OUT:-20.8}
-GAME_GAIN=${GAME_GAIN:-0.90}
-
-# The song's real end = last audible sample of the MIXED audio.
-T=$(ffmpeg -v error -i "$AUDIO" -ac 1 -ar 1000 -f s16le - | node -e '
-const c=[];process.stdin.on("data",d=>c.push(d));process.stdin.on("end",()=>{
-const b=Buffer.concat(c);const a=new Int16Array(b.buffer,b.byteOffset,b.length/2);
-let last=0; for(let i=0;i<a.length;i++) if(Math.abs(a[i])>8) last=i;
-console.log((last/1000+0.04).toFixed(3));});')
-
-Q1=$(node -e "console.log(($T + $DEAD).toFixed(3))")
-Q2=$(node -e "console.log(($T + $DEAD + $LOW_IN).toFixed(3))")
-TAILDUR=$(node -e "console.log(($END - $T).toFixed(3))")
-echo "song ends ${T}s · dead air to ${Q1}s · game at ${LOW} by ${Q2}s · full ${FULL_AT}->${FULL_BY}s · end ${END}s"
-
-# volume envelope, in order: gong held full, out before the cut, silent under the
-# song, dead air, up to LOW for the fatality, up to FULL for the fanfare.
-VOL="${GAME_GAIN}*(if(lt(t,${GONG_HOLD}),1.0, if(lt(t,${GAME_OUT}),1-(t-${GONG_HOLD})/(${GAME_OUT}-${GONG_HOLD}), if(lt(t,${Q1}),0.0, if(lt(t,${Q2}),${LOW}*(t-${Q1})/(${LOW_IN}), if(lt(t,${FULL_AT}),${LOW}, if(lt(t,${FULL_BY}),${LOW}+(1-${LOW})*(t-${FULL_AT})/(${FULL_BY}-${FULL_AT}),1.0)))))))"
-
-ffmpeg -y -v warning \
- -i "$BODY" \
- -i "$AUDIO" \
- -i "$BG" \
- -ss "$T" -t "$TAILDUR" -i "$BG" \
- -filter_complex "
-[2:a]atrim=0:${END},asetpts=PTS-STARTPTS,volume='${VOL}':eval=frame[game];
-[1:a]apad=whole_dur=${END}[song];
-[song][game]amix=inputs=2:duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a];
-[0:v]trim=0:${T},setpts=PTS-STARTPTS[bodyv];
-[3:v]scale=1280:720,fps=30,setsar=1,format=yuv420p,setpts=PTS-STARTPTS[tailv];
-[bodyv][tailv]concat=n=2:v=1[v]" \
- -map "[v]" -map "[a]" -shortest \
- -c:v libx264 -preset medium -crf 18 -pix_fmt yuv420p \
- -c:a aac -b:a 192k -movflags +faststart "$OUT"
-
-echo "wrote $OUT"
-ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 "$OUT"
diff --git a/umtool/song/mk-fatal-finish3.sh b/umtool/song/mk-fatal-finish3.sh
@@ -1,164 +0,0 @@
-#!/usr/bin/env bash
-# Assemble a Mortal Kombat fatality finale from an already-mixed audio track.
-#
-# Supersedes mk-fatal-finish2.sh, whose DEFAULTS DID NOT DESCRIBE THE SHIPPED
-# BUILD: several rounds of feedback ("start the finale earlier", "fade the game
-# in earlier", "fatality a touch quieter") were applied as one-off env overrides
-# and never written back, so rebuilding from that script silently changed the
-# gong and the ending as well as the vocals.
-#
-# The defaults below are MEASURED off the shipped quartering-mk-fatality-full.mp4
-# against the raw background, region by region:
-#
-# region shipped/raw set by
-# attract music 0.410 GAME_GAIN
-# the gong (18.09s) 0.410 GONG_HOLD (still full)
-# after gong (18.6s) 0.266 fading, GONG_HOLD -> GAME_OUT
-# silence (18.95s) 0.000 GAME_OUT
-# dead air (145.0s) 0.048 DEAD
-# fatality plateau 0.242 LOW
-# ramp (149.5s) 1.015 FULL_BY
-# fanfare / end 1.00 (full)
-#
-# New here: FINISH_HOOK lays "Finish him!" over the tail so the finale is led
-# into rather than opened cold on the game's own "dun dun dun" fanfare, and the
-# game's drum kit is mixed on at the end instead of in a separate script.
-set -euo pipefail
-# The DATA dir, not the script's own -- intro/, mkvocals/ and the plan files live
-# there, and the repo copy of this script would otherwise cd into the repo. See
-# pipeline.md: the data directory holds independent, sometimes stale copies, so the
-# repo is authoritative and every builder must name the data dir absolutely.
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-
-BODY=${BODY:?need the rendered body video}
-AUDIO=${AUDIO:?need the mixed audio wav}
-BG=${BG:-intro/mk-bg-fatal2.mp4}
-OUT=${OUT:?need an output path}
-
-DEAD=${DEAD:-1.06} # seconds of silence after the song
-# 0.17, not the 0.24 the shipped file MEASURES against the raw background.
-#
-# Match the envelope in the regions that are NOT at the limiter ceiling. The
-# fatality and the fanfare read ~1.0 of raw in every version regardless, because
-# they ARE at the ceiling — so they cannot be used to set anything. The attract
-# music and the fatality plateau are the two windows that can, and both came out
-# 1.41x hot at the "measured" values, which is the 0.709 whole-envelope factor
-# recorded in mortal-kombat.md. 0.24 x 0.709 = 0.17, 0.41 x 0.709 = 0.29.
-LOW=${LOW:-0.17} # how loud the game is during the fatality itself
-LOW_IN=${LOW_IN:-0.8} # how long it takes to reach LOW
-FULL_AT=${FULL_AT:-148.60} # start of the ramp back to full...
-FULL_BY=${FULL_BY:-149.50} # ...measured as where the shipped file reaches full
-END=${END:-156.10} # FATALITY gives way to the ladder screen
-GONG_HOLD=${GONG_HOLD:-18.5}
-GAME_OUT=${GAME_OUT:-19.0}
-# ATTRACT is the level of the attract music and the gong ONLY. It is not a gain
-# on the whole envelope: the shipped file measures 0.410 against the raw
-# background at the attract music but 1.009 at the fanfare, and one multiplier
-# cannot produce both. Applying it globally (as -finish2.sh's GAME_GAIN does)
-# plays the FATALITY -- the payoff the whole cut is built around -- at 41%.
-ATTRACT=${ATTRACT:-0.29}
-FULL_GAIN=${FULL_GAIN:-1.0}
-
-# "Finish him!" into the fanfare. The game's fanfare is at 153.80s, so the shout
-# is placed to LAND on it rather than to sit anywhere in particular.
-FINISH_HOOK=${FINISH_HOOK:-mkvocals/hooks-verb/hook-17.wav}
-FANFARE_AT=${FANFARE_AT:-153.80}
-# FINISH_GAIN is a PLAIN MULTIPLIER, not a peak target -- and hook-17 peaks at only
-# 0.39 of full scale, so the old 0.85 ATTENUATED it to ~0.33 and dropped it under a game
-# running at ~0.9. Measured on the shipped build: the shout is genuinely there
-# (cross-correlates r=0.27 at exactly its placed time, 152.560s) and simply inaudible.
-# 2.2 puts its peak at ~0.86, level with everything else in the finale.
-FINISH_GAIN=${FINISH_GAIN:-2.2}
-# ...and the game gets out of its way. A shout at the same level as a full-scale fanfare
-# is still a shout nobody can pick out, so the game ducks to FIN_DUCK across the hook,
-# ramped either side so the drop is not itself an event.
-FIN_DUCK=${FIN_DUCK:-0.32}
-FIN_DUCK_IN=${FIN_DUCK_IN:-0.15}
-FIN_DUCK_OUT=${FIN_DUCK_OUT:-0.25}
-FINISH_LEAD=${FINISH_LEAD:-0.05} # how far its END sits before the fanfare
-
-DRUMS=${DRUMS:-1}
-DRUM_GAIN=${DRUM_GAIN:-1.6}
-
-# The song's real end = last audible sample of the MIXED audio. The renderer
-# leaves silent video past the last note, so a container duration would put the
-# freeze and the shatter inside the dimmed body instead of the bright tail.
-T=$(ffmpeg -v error -i "$AUDIO" -ac 1 -ar 1000 -f s16le - | node -e '
-const c=[];process.stdin.on("data",d=>c.push(d));process.stdin.on("end",()=>{
-const b=Buffer.concat(c);const a=new Int16Array(b.buffer,b.byteOffset,b.length/2);
-let last=0; for(let i=0;i<a.length;i++) if(Math.abs(a[i])>8) last=i;
-console.log((last/1000+0.04).toFixed(3));});')
-
-Q1=$(node -e "console.log(($T + $DEAD).toFixed(3))")
-Q2=$(node -e "console.log(($T + $DEAD + $LOW_IN).toFixed(3))")
-TAILDUR=$(node -e "console.log(($END - $T).toFixed(3))")
-echo "song ends ${T}s · dead air to ${Q1}s · game at ${LOW} by ${Q2}s · full ${FULL_AT}->${FULL_BY}s · end ${END}s"
-
-VOL="if(lt(t,${GONG_HOLD}),${ATTRACT}, if(lt(t,${GAME_OUT}),${ATTRACT}*(1-(t-${GONG_HOLD})/(${GAME_OUT}-${GONG_HOLD})), if(lt(t,${Q1}),0.0, if(lt(t,${Q2}),${LOW}*(t-${Q1})/(${LOW_IN}), if(lt(t,${FULL_AT}),${LOW}, if(lt(t,${FULL_BY}),${LOW}+(${FULL_GAIN}-${LOW})*(t-${FULL_AT})/(${FULL_BY}-${FULL_AT}),${FULL_GAIN}))))))"
-
-# Place the shout so its SOUNDED part ends just before the fanfare.
-FIN_AT=$(node -e "
-const {execFileSync}=require('child_process');
-const d=Number(execFileSync('ffprobe',['-v','error','-show_entries','format=duration','-of','csv=p=0','$FINISH_HOOK']).toString().trim());
-console.log(Math.max(0, ${FANFARE_AT} - ${FINISH_LEAD} - d).toFixed(3));")
-# String(), not the bare number: node colourises numeric console.log output on a
-# TTY, and those ANSI escapes end up INSIDE the filtergraph, where ffmpeg reads
-# them as part of a pad name and the graph fails to bind.
-FIN_MS=$(node -e "console.log(String(Math.round($FIN_AT*1000)))")
-echo "\"Finish him!\" at ${FIN_AT}s, landing on the ${FANFARE_AT}s fanfare"
-
-# The duck is applied AFTER VOL is built, because it needs FIN_AT.
-FIN_END=$(node -e "
-const {execFileSync}=require('child_process');
-const d=Number(execFileSync('ffprobe',['-v','error','-show_entries','format=duration','-of','csv=p=0','$FINISH_HOOK']).toString().trim());
-console.log(String(($FIN_AT + d).toFixed(3)));")
-D0=$(node -e "console.log(String(($FIN_AT - $FIN_DUCK_IN).toFixed(3)))")
-D3=$(node -e "console.log(String(($FIN_END + $FIN_DUCK_OUT).toFixed(3)))")
-DUCK="if(lt(t,${D0}),1, if(lt(t,${FIN_AT}), 1-(1-${FIN_DUCK})*(t-${D0})/${FIN_DUCK_IN}, if(lt(t,${FIN_END}), ${FIN_DUCK}, if(lt(t,${D3}), ${FIN_DUCK}+(1-${FIN_DUCK})*(t-${FIN_END})/${FIN_DUCK_OUT}, 1))))"
-VOL="(${VOL})*(${DUCK})"
-echo " game ducks to ${FIN_DUCK} across ${FIN_AT}-${FIN_END}s so the shout is heard"
-
-ffmpeg -y -v warning \
- -i "$BODY" \
- -i "$AUDIO" \
- -i "$BG" \
- -ss "$T" -t "$TAILDUR" -i "$BG" \
- -i "$FINISH_HOOK" \
- -filter_complex "
-[2:a]atrim=0:${END},asetpts=PTS-STARTPTS,volume='${VOL}':eval=frame[game];
-[1:a]apad=whole_dur=${END}[song];
-[4:a]aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo,volume=${FINISH_GAIN},adelay=delays=${FIN_MS}:all=1,apad=whole_dur=${END}[fin];
-[song][game][fin]amix=inputs=3:duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a];
-[0:v]trim=0:${T},setpts=PTS-STARTPTS[bodyv];
-[3:v]scale=1280:720,fps=30,setsar=1,format=yuv420p,setpts=PTS-STARTPTS[tailv];
-[bodyv][tailv]concat=n=2:v=1[v]" \
- -map "[v]" -map "[a]" -shortest \
- -c:v libx264 -preset medium -crf 18 -pix_fmt yuv420p \
- -c:a aac -b:a 192k -movflags +faststart "$OUT"
-
-# ---- the game's own drum kit ------------------------------------------------
-# Track 10 picked by INDEX, not by name: this file's track names spell the
-# author's credit, so a /drum/i match finds nothing. It is 100% GM percussion.
-# The kit stops at the song's end, leaving the fatality tail untouched.
-if [ "$DRUMS" = "1" ]; then
- T2=/home/user/.claude/jobs/efbe67a7/tmp
- # DRUM_WAV plays a kit that was rendered elsewhere -- um-drums.mjs builds one
- # out of chopped ums. Everything downstream is identical, so the two kits are
- # directly comparable: same hits, same gain, same mix.
- if [ -n "${DRUM_WAV:-}" ]; then
- cp "$DRUM_WAV" "$T2/mk-drums4.wav"
- else
- TEMPO_SCALE=1.05 TRIM=0 SHIFT=17.45 START=19.0 END="$T" \
- node midi-drums.mjs ~/Downloads/mkmtheme.mid "$T2/mk-drums4.mid" 10 >/dev/null
- fluidsynth -ni -F "$T2/mk-drums4.wav" -r 48000 -g 0.8 \
- /usr/share/soundfonts/FatBoy.sf2 "$T2/mk-drums4.mid" >/dev/null 2>&1
- fi
- ffmpeg -y -v error -i "$OUT" -i "$T2/mk-drums4.wav" \
- -filter_complex "[1:a]volume=${DRUM_GAIN},aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo[d];[0:a][d]amix=inputs=2:duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a]" \
- -map 0:v -map "[a]" -c:v copy -c:a aac -b:a 192k -movflags +faststart "${OUT%.mp4}-tmp.mp4"
- mv "${OUT%.mp4}-tmp.mp4" "$OUT"
- echo " drums mixed at ${DRUM_GAIN}"
-fi
-
-echo "wrote $OUT"
-ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 "$OUT"
diff --git a/umtool/song/mk-rebuild.sh b/umtool/song/mk-rebuild.sh
@@ -1,55 +0,0 @@
-#!/usr/bin/env bash
-# Mortal Kombat, rebuilt on the 3,959-clip palette.
-#
-# Renders the BODY ONLY -- no hooks. That is what makes the vocal balance
-# iterable: the video takes ~20 minutes and does not depend on the hooks at all,
-# so a new balance is a ~10-second re-mux by mk-mix-hooks.mjs instead of another
-# render. Round 1 (mk-variants/A-E) was built this way.
-#
-# The two finales need two BODIES, not just two tails: the background is spliced
-# at 21s and runs continuously from there, so where segment B starts changes what
-# is on screen DURING the song as well as after it.
-# FULL intro/mk-bg-fatal2.mp4 (segment B at source 128.5, fatality 153.6s)
-# SHORT intro/mk-bg-fatal3.mp4 (segment B at source 136.5, fatality 145.6s)
-set -euo pipefail
-# The DATA dir, not the script's own. `cd $(dirname $0)` worked only while this
-# script was being run from the data directory's own copy; run from the repo it
-# lands where the lead files and poly-plan.json do not exist. rpg-remake-v5.sh
-# carries the same note for the same reason.
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-T=/home/user/.claude/jobs/efbe67a7/tmp
-WHICH="${1:-full}"
-
-# Voice settings recovered from the SHIPPED plan (mk-plan-fatal2.json), not
-# guessed: melody gain 1.0 / bass 0.68, scale 1.05, 488 + 390 notes to bar 70.
-# poolFactor 2.6 is measured -- 1.8 gives melody 0.93 / bass 1.43, 3.0 gives
-# 1.25 / 0.83, and 2.6 is the knee at 1.01 / 0.94.
-export TEMPO_SCALE=1.05 TIME_LIMIT=${TL:-120.88} MAX_IOI=0.70 WINDOW=700 LOCAL=0
-export VOICES='[{"name":"melody","lead":"mkm-lead.json","gain":1.0,"role":"full","unique":true,"reuseGap":20,"contourMax":12,"fill":true},{"name":"bass","lead":"mkm-bass.json","gain":0.68,"role":"pip1","unique":true,"reuseGap":12,"contourMax":4,"poolFactor":2.6,"fill":false}]'
-
-if [ "${SKIP_ARRANGE:-0}" != "1" ]; then
- node arrange-poly.mjs
- cp poly-plan.json "${RAWF:-mk-plan-v4.json}"
-
- # PREROLL 17.45 puts the first um at 19.29s -- after the game's own gong at
- # 18.086s (which is the SELECT-SCREEN sting, not an in-game gong) and after the
- # 0.9s of digital silence that follows it. NO_GONG because the background
- # supplies one; the record's would double it.
- PREROLL=17.45 NO_GONG=1 node place-vocals-full.mjs \
- "${RAWF:-mk-plan-v4.json}" mkvocals/hooks-verb/mk-vocals.json mkvocals/hooks-verb \
- "${FPLAN:-mk-plan-fatal4.json}" "${FOVER:-mk-overlays-fatal4.json}"
-fi
-[ "${ARRANGE_ONLY:-0}" = "1" ] && exit 0
-
-# BG and OUT can be overridden, because a cut of a different LENGTH needs its own
-# splice: where segment B starts is the only thing that decides how long after the
-# last note the fatality lands, so a new TL means a new background.
-if [ -n "${BG:-}" ]; then OUT="${OUT:-$T/mkfatal4-custom.mp4}"
-elif [ "$WHICH" = "short" ]; then BG=intro/mk-bg-fatal3.mp4; OUT="$T/mkfatal4-short.mp4"
-else BG=intro/mk-bg-fatal2.mp4; OUT="$T/mkfatal4.mp4"; fi
-
-# No OVERLAYS= on purpose: the hooks are mixed on afterwards.
-PLAN=${FPLAN:-mk-plan-fatal4.json} BG_VIDEO=$PWD/$BG LAYOUT=bg \
- node render-poly.mjs "$OUT"
-
-echo "MK_BODY_${WHICH}_DONE -> $OUT"
diff --git a/umtool/song/ms2-bg3.sh b/umtool/song/ms2-bg3.sh
@@ -1,31 +0,0 @@
-#!/bin/sh
-# Metal Slug, cut after the first statement. The tune states its material twice
-# (the second pass begins at bar 55 = 79.02s and differs only slightly), so this
-# keeps one full statement, lets ~6 bars of the repeat begin, and fades across
-# them -- the tune comes round again and dissolves rather than stopping.
-cd /home/user/.claude/jobs/efbe67a7/tmp/song || exit 1
-D=/home/user/reports/quartering-uh-song
-
-export MAX_IOI=0.70 WINDOW=700 LOCAL=0 TEMPO_SCALE=1.05 WOBBLE_W=0.08 TIME_LIMIT=${TL:-87.8}
-# At 198 melody notes the melody is in SURPLUS, so the bass pool is free -- the
-# poolFactor cliff that constrains the full-length build does not exist here.
-export VOICES='[{"name":"melody","lead":"ms2-lead-t.json","gain":1.0,"role":"full","unique":true,"reuseGap":20,"contourMax":12,"fill":true},{"name":"bass","lead":"ms2-bass-t.json","gain":0.68,"role":"pip1","unique":true,"reuseGap":12,"contourMax":4,"poolFactor":5.0,"fill":false}]'
-node arrange-poly.mjs | grep -E "melody|bass|total"
-cp poly-plan.json ms2-short-plan.json
-
-TAIL_FADE=6.2 BG_VIDEO=$PWD/intro/ms2-bg.mp4 BG_FIT=pad LAYOUT=bg \
- node render-poly.mjs /tmp/ms2short.mp4
-
-LEAD=$(node -e "
-const {voices}=require('$PWD/poly-plan.json');
-const first=Math.min(...voices.filter(v=>v.plan.length).map(v=>v.plan[0].slotStart));
-console.log(Math.max(0,first).toFixed(3));")
-echo "trimming ${LEAD}s of lead-in off the body"
-ffmpeg -nostdin -v error -y -ss "$LEAD" -i /tmp/ms2short.mp4 -c:v libx264 -preset medium -crf 21 \
- -pix_fmt yuv420p -video_track_timescale 30000 -c:a aac -b:a 192k -ar 48000 -ac 2 /tmp/ms2shortbd.mp4
-
-ffmpeg -nostdin -v error -y -i intro/ms2-intro.mp4 -i /tmp/ms2shortbd.mp4 \
- -filter_complex "[0:v][0:a][1:v][1:a]concat=n=2:v=1:a=1[v][a]" -map "[v]" -map "[a]" \
- -c:v libx264 -preset medium -crf 21 -pix_fmt yuv420p -c:a aac -b:a 192k -ar 48000 -ac 2 \
- -movflags +faststart "$D/quartering-metalslug2-short.mp4"
-echo MS2SHORT_DONE
diff --git a/umtool/song/ms2-drums.sh b/umtool/song/ms2-drums.sh
@@ -1,36 +0,0 @@
-#!/usr/bin/env sh
-# Metal Slug, gameplay build, WITH the game's own drum kit under it.
-#
-# The ums play the pitched voices. The kit is not a pitched voice -- a hi-hat is
-# noise, not a note -- so it comes straight out of the MIDI and is played as
-# drums. Nothing in the build ever used the three drum tracks before this.
-#
-# The drums are mixed onto the FINISHED video (video stream copied, audio
-# re-encoded), not re-rendered: the ums and the game bed are already balanced in
-# that file, and the kit simply goes on top. That is why this runs in seconds.
-set -e
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-D=/home/user/reports/quartering-uh-song
-T=/home/user/.claude/jobs/efbe67a7/tmp
-
-# The arrangement is the MIDI trimmed by 5.4497s, stretched 1.05, shifted to 11s.
-# Verified against ms2-plan-11.json at BOTH ends: first melody slot 11.232s and
-# last 101.698s are both predicted exactly, so there is no drift over the 90s.
-TEMPO_SCALE=1.05 TRIM=5.4497 SHIFT=11.0 START=11.0 END=102.85 \
- node midi-drums.mjs ~/Downloads/ms2_stage1.mid "$T/ms2-drums.mid"
-
-# Tracks 1 "Drum(Bass & Snare)", 2 "Drum(Hi-hat)", 3 "Drum(Cymbal)" are already
-# GM percussion numbers (36 kick, 38 snare, 42 closed hat, 49 crash), so they
-# render as a kit on channel 10 with any GM soundfont.
-fluidsynth -ni -F "$T/ms2-drums.wav" -r 48000 -g 0.8 \
- /usr/share/soundfonts/FatBoy.sf2 "$T/ms2-drums.mid"
-
-# g=1.8 chosen by ear over 1.1 (too subtle) and 2.6 (too prominent): "clearly
-# part of the song but not overpowering". Peaks 27.7k of 32768 -- the limiter is
-# a backstop, it is not working at this level.
-ffmpeg -y -v error -i "$D/quartering-metalslug2-play.mp4" -i "$T/ms2-drums.wav" \
- -filter_complex "[1:a]volume=1.8,aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo[d];[0:a][d]amix=inputs=2:duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a]" \
- -map 0:v -map "[a]" -c:v copy -c:a aac -b:a 192k -movflags +faststart \
- "$D/quartering-metalslug2-drums.mp4"
-
-echo "MS2DRUMS_DONE -> $D/quartering-metalslug2-drums.mp4"
diff --git a/umtool/song/ms2-full.sh b/umtool/song/ms2-full.sh
@@ -1,18 +0,0 @@
-#!/bin/sh
-# Full-length Metal Slug 2, in two flavours. Run once the mine has grown the palette.
-# ./ms2-full.sh 2 -> lead + bass (730 notes)
-# ./ms2-full.sh 3 -> lead + bass + rhythm (1287 notes)
-cd /home/user/.claude/jobs/efbe67a7/tmp/song || exit 1
-D=/home/user/reports/quartering-uh-song
-N="${1:-2}"
-export MAX_IOI=0.70 WINDOW=700 LOCAL=0 TEMPO_SCALE=1.05 WOBBLE_W=0.08 TIME_LIMIT=999
-LEADV='{"name":"melody","lead":"ms2-lead.json","gain":1.0,"role":"full","unique":true,"reuseGap":20,"contourMax":12,"fill":true}'
-BASSV='{"name":"bass","lead":"ms2-bass.json","gain":0.68,"role":"pip1","unique":true,"reuseGap":12,"contourMax":4,"poolFactor":1.05,"fill":false}'
-GTRV='{"name":"rhythm","lead":"ms2-gtr.json","gain":0.55,"role":"pip2","unique":true,"reuseGap":10,"contourMax":6,"poolFactor":1.0,"fill":false}'
-if [ "$N" = "3" ]; then export VOICES="[$LEADV,$BASSV,$GTRV]"; OUT="$D/quartering-metalslug2-full3.mp4"
-else export VOICES="[$LEADV,$BASSV]"; OUT="$D/quartering-metalslug2-full.mp4"; fi
-node arrange-poly.mjs | grep -E "melody|bass|rhythm|total"
-LAYOUT=pip node render-poly.mjs "$OUT" | tail -1
-ffmpeg -nostdin -v error -y -i "$OUT" -c:v libx264 -preset medium -crf 28 -pix_fmt yuv420p \
- -c:a aac -b:a 160k -movflags +faststart "${OUT%.mp4}-small.mp4"
-echo "MS2_FULL_${N}_DONE"
diff --git a/umtool/song/ms2-jerbg.sh b/umtool/song/ms2-jerbg.sh
@@ -1,110 +0,0 @@
-#!/usr/bin/env bash
-# Metal Slug 2 over the USER'S OWN EDIT — the background WITH music, handing over
-# on the game's own "MISSION 1 START".
-#
-# SUPERSEDES the no-music background. `ms2-play-rebuild.sh` is built around
-# TheSoldier's music-free gameplay capture, with the game's sounds blended under
-# the ums at quarter volume. That capture is RETIRED (user, 2026-08-17): every
-# Metal Slug cut now uses jer's edit, which has the game's music in it, and the
-# song takes over rather than sitting under it.
-#
-# The handover, recovered from the lost ms2-triangle.sh and re-verified here:
-# * "MISSION 1 START" is at 11.901s in that audio,
-# * the background's own sound is CUT DEAD there — not ducked, cut,
-# * the CLEAN announcer clip plays in its place, so it is not doubled with the
-# background's own copy of it or muddied by the music under it,
-# * and the ums fade in over 1.6s underneath, because in the game the music
-# starts alongside that voice.
-#
-# THE ARRANGEMENT IS SHIFTED, NOT RE-ARRANGED. Both shipped plans put their first
-# note at 11.232s; landing it on 11.901 is a flat +0.669 on every slot, which
-# keeps every clip the user has already judged. The KIT MUST MOVE WITH IT: the
-# drum map is songTime = (midiTime - 5.4497) x 1.05 + SHIFT, so SHIFT goes
-# 11.0 -> 11.669. Leaving it at 11.0 slides every drum hit 0.669s against the ums,
-# which is the same trap ms2-play-rebuild.sh documents at 0.232s.
-#
-# The background is 640x480, so BG_FIT=pad — cropping a 4:3 source to 16:9 cuts
-# the top and bottom of the gameplay. The pips then overhang the pillarbox bars,
-# which is what the triangle build looked like and is intended.
-set -uo pipefail
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-S=/home/user/Projects/yt-dlp-transcript-browser/umtool/song
-D=/home/user/reports/quartering-uh-song
-T=/home/user/.claude/jobs/efbe67a7/tmp
-V=$D/videos/metal-slug
-
-BG=${BG:-$D/jer-metalslug-bg.mp4}
-ANNOUNCER=${ANNOUNCER:-$D/MS_Mission_1_start.ogg}
-START_AT=${START_AT:-11.901}
-UM_FADE=${UM_FADE:-1.6}
-ANN_GAIN=${ANN_GAIN:-1.0}
-DRUM_GAIN=${DRUM_GAIN:-1.8}
-mkdir -p "$V/variants"
-
-# $1 name $2 source plan $3 body path
-cut () {
- local NAME=$1 SRCPLAN=$2 BODY=$3
- local PLANF=ms2-plan-jer-$NAME.json
- echo "######## $NAME"
-
- local SH END DRUMSHIFT
- SH=$(node -e "
-const {voices}=require('./$SRCPLAN');
-const f=Math.min(...voices.filter(v=>v.plan.length).map(v=>v.plan[0].slotStart));
-console.log(String((${START_AT}-f).toFixed(4)));")
- node -e "
-const fs=require('fs');
-const p=JSON.parse(fs.readFileSync('$SRCPLAN','utf8'));
-for(const v of p.voices) for(const n of v.plan) n.slotStart=+(n.slotStart+$SH).toFixed(4);
-fs.writeFileSync('$PLANF',JSON.stringify(p,null,1));
-console.log(' shifted +$SH s -> first note '+
- Math.min(...p.voices.filter(v=>v.plan.length).map(v=>v.plan[0].slotStart)).toFixed(3)+'s');"
- DRUMSHIFT=$(node -e "console.log(String((11.0+$SH).toFixed(4)))")
-
- if [ "${REUSE_BODY:-1}" = "1" ] && [ -f "$BODY" ]; then
- echo " reusing $BODY"
- else
- PLAN=$PLANF TAIL_FADE=6.2 BG_VIDEO=$BG BG_FIT=pad LAYOUT=bg BG_DIM=0 \
- node "$S/render-poly.mjs" "$BODY" 2>&1 | tail -2
- fi
-
- END=$(node -e "
-const {voices}=require('./$PLANF');
-console.log((Math.max(...voices.flatMap(v=>v.plan.map(n=>n.slotStart+n.slotDur)))+0.6).toFixed(2));")
- echo " game sound full to ${START_AT}s then CUT; announcer + ums from there; ends ${END}s"
-
- # ---- the handover, onto the drum-less deliverable -------------------------
- ffmpeg -nostdin -v error -y -i "$BODY" -i "$BG" -i "$ANNOUNCER" \
- -filter_complex "
-[1:a]atrim=0:${END},asetpts=PTS-STARTPTS,volume='if(lt(t,${START_AT}),1.0,0.0)':eval=frame[game];
-[2:a]aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo,volume=${ANN_GAIN},adelay=delays=$(node -e "console.log(String(Math.round(${START_AT}*1000)))"):all=1,apad=whole_dur=${END}[ann];
-[0:a]atrim=0:${END},asetpts=PTS-STARTPTS,afade=t=in:st=${START_AT}:d=${UM_FADE}[song];
-[song][game][ann]amix=inputs=3:duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a]" \
- -map 0:v -map "[a]" -t "$END" -c:v copy -c:a aac -b:a 192k -ar 48000 -ac 2 \
- -movflags +faststart "$V/variants/$NAME-nokit.mp4"
-
- # ---- the kits ------------------------------------------------------------
- TEMPO_SCALE=1.05 TRIM=5.4497 SHIFT="$DRUMSHIFT" START="$DRUMSHIFT" END="$END" \
- node "$S/midi-drums.mjs" ~/Downloads/ms2_stage1.mid "$T/jer-$NAME.mid" 2>&1 | tail -1
- fluidsynth -ni -F "$T/jer-$NAME-kit.wav" -r 48000 -g 0.8 \
- /usr/share/soundfonts/FatBoy.sf2 "$T/jer-$NAME.mid" >/dev/null 2>&1
-
- TEMPO_SCALE=1.05 TRIM=5.4497 SHIFT="$DRUMSHIFT" START="$DRUMSHIFT" END="$END" \
- node "$S/midi-drums.mjs" ~/Downloads/ms2_stage1.mid "$T/jer-$NAME-hits.json" 2>&1 | tail -1
- MATCH="$T/jer-$NAME-kit.wav" PLAN_OUT="$T/jer-$NAME-drumplan.json" \
- node "$S/um-drums.mjs" "$T/jer-$NAME-hits.json" "$T/jer-$NAME-umkit.wav" 2>&1 | tail -4
-
- mux () {
- ffmpeg -nostdin -v error -y -i "$V/variants/$NAME-nokit.mp4" -i "$1" \
- -filter_complex "[1:a]volume=${DRUM_GAIN},aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo[d];[0:a][d]amix=inputs=2:duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a]" \
- -map 0:v -map "[a]" -c:v copy -c:a aac -b:a 192k -movflags +faststart "$2"
- echo " -> $2 $(ffprobe -v error -show_entries format=duration -of csv=p=0 "$2")s"
- }
- mux "$T/jer-$NAME-kit.wav" "$V/$NAME.mp4"
- mux "$T/jer-$NAME-umkit.wav" "$V/variants/$NAME-umdrums.mp4"
-}
-
-cut wide ms2-plan-11-v3.json "$T/ms2jer-wide.mp4"
-cut wide-short ms2-plan-short2.json "$T/ms2jer-wide-short.mp4"
-
-echo MS2_JERBG_DONE
diff --git a/umtool/song/ms2-play-rebuild.sh b/umtool/song/ms2-play-rebuild.sh
@@ -1,113 +0,0 @@
-#!/bin/sh
-# Metal Slug 2 -- THE SHIPPED BUILD, rebuilt on the 3,959-clip palette:
-# the short cut over TheSoldier's NO-MUSIC gameplay, with the game's own sounds
-# blended in, and then the game's own drum kit on top.
-#
-# The no-music capture is the whole point of this build: it is what lets the
-# game's sounds sit under the ums without the game's MUSIC fighting them. Full
-# volume until the song starts at 11s, quarter volume under the song.
-#
-# Three things this rebuild must not lose, all of which are the build:
-# * background intro/ms2-play-bg.mp4 (the no-music capture), NOT ms2-bg.mp4
-# * the game audio bed, muxed under the song
-# * the drum kit from the MIDI, mixed on at 1.8 by ms2-drums.sh
-#
-# 11s of run-up is deliberate: the game's own "mission 1 start" voice sounds at
-# 19s in the SOURCE and the capture is cut from 8.753s, so 11s of output = 19.99s
-# of source -- a second clear of that voice. 10s was tried and is too close.
-#
-# Note the SLOT TIMES do not depend on the palette at all: they come from the
-# MIDI, TEMPO_SCALE and TIME_LIMIT. So the 11s shift and the drum map (verified
-# at both ends against the old plan) carry over exactly.
-set -e
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-D=/home/user/reports/quartering-uh-song
-T=/home/user/.claude/jobs/efbe67a7/tmp
-
-PLANF=${PLANF:-ms2-plan-11-v3.json}
-# The UNSHIFTED arrangement, kept beside the shifted one. It is named after the
-# build rather than fixed, so a short cut cannot overwrite the record of the long.
-RAWF=${RAWF:-ms2-short-plan-v3.json}
-# The body render and the kit, likewise named after the build.
-BODY=${BODY:-$T/ms2play-v3.mp4}
-KIT=${KIT:-$T/ms2-drums-v3}
-PLAY=${PLAY:-$D/quartering-metalslug2-play-v3.mp4}
-DRUMS=${DRUMS:-$D/quartering-metalslug2-drums-v3.mp4}
-
-# At TIME_LIMIT 87.8 the melody is only ~198 notes and is in SURPLUS, so the bass
-# pool is free and poolFactor 5.0 costs nothing. The 1.3 cliff that constrains
-# the FULL-LENGTH cut does not exist at this length.
-export MAX_IOI=0.70 WINDOW=700 LOCAL=0 TEMPO_SCALE=1.05 WOBBLE_W=0.08 TIME_LIMIT=${TL:-87.8}
-export VOICES='[{"name":"melody","lead":"ms2-lead-t.json","gain":1.0,"role":"full","unique":true,"reuseGap":20,"contourMax":12,"fill":true},{"name":"bass","lead":"ms2-bass-t.json","gain":0.68,"role":"pip1","unique":true,"reuseGap":12,"contourMax":4,"poolFactor":5.0,"fill":false}]'
-
-if [ "${REUSE_PLAN:-0}" != "1" ]; then
- node arrange-poly.mjs
- cp poly-plan.json "$RAWF"
- # Shift the whole arrangement by a FLAT +11.0s.
- #
- # NOT "move the first note to 11.0". The drum map is
- # songTime = (midiTime - 5.4497) x 1.05 + 11.0
- # and the -t voice files already carry that 5.4497 trim, so the shift term IS
- # 11.0 and the first melody slot lands at 11.232 (its trimmed time 0.221 x
- # 1.05). Normalising the first note to 11.000 instead would slide every drum
- # hit 0.232s against the ums -- and the kit is the one thing in the build that
- # is locked to the beat by arithmetic rather than by choice.
- node -e "
- const fs=require('fs');
- const p=JSON.parse(fs.readFileSync('$RAWF','utf8'));
- const SH=11.0;
- for(const v of p.voices) for(const n of v.plan) n.slotStart=+(n.slotStart+SH).toFixed(4);
- fs.writeFileSync('$PLANF',JSON.stringify(p,null,1));
- for(const v of p.voices) console.log(' '+v.name+' first slot '+v.plan[0].slotStart.toFixed(3)+
- 's, last '+(v.plan.at(-1).slotStart+v.plan.at(-1).slotDur).toFixed(3)+'s');"
-fi
-[ "${ARRANGE_ONLY:-0}" = "1" ] && exit 0
-
-# TAIL_FADE 6.2 lets the tune come round again and dissolve rather than stopping
-# dead on the loop point. VIDEO_TAIL_FADE now follows it, so the picture goes
-# with the sound.
-if [ "${REUSE_BODY:-0}" = "1" ] && [ -f "$BODY" ]; then
- echo "reusing $BODY"
-else
- PLAN=$PLANF TAIL_FADE=6.2 BG_VIDEO=$PWD/intro/ms2-play-bg.mp4 BG_FIT=pad LAYOUT=bg \
- node render-poly.mjs "$BODY"
-fi
-
-# End of the song, measured -- not the container duration.
-END=$(node -e "
-const {voices}=require('$PWD/$PLANF');
-console.log((Math.max(...voices.flatMap(v=>v.plan.map(n=>n.slotStart+n.slotDur)))+0.6).toFixed(2));")
-# The game bed fades with the song and the picture. The song's own tail fade is
-# inside the render, and the picture now follows it -- so without this the last
-# 6.2s would be a black screen with the game still chattering under it.
-GFADE_ST=$(node -e "console.log(($END - 6.2).toFixed(2))")
-echo "muxing the game audio under the song, to ${END}s (game bed fades from ${GFADE_ST}s)"
-
-# The game's own sounds: FULL until the song starts at 11s, then a QUARTER under
-# it. normalize=0 so amix does not halve both inputs.
-ffmpeg -nostdin -v error -y -i "$BODY" -i intro/ms2-play-bg.mp4 \
- -filter_complex "
-[1:a]volume='if(lt(t,11),1.0,0.25)':eval=frame,atrim=0:${END},asetpts=PTS-STARTPTS,afade=t=out:st=${GFADE_ST}:d=6.2[game];
-[0:a]atrim=0:${END},asetpts=PTS-STARTPTS[song];
-[song][game]amix=inputs=2:duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a]" \
- -map 0:v -map "[a]" -t "$END" -c:v copy -c:a aac -b:a 192k -ar 48000 -ac 2 \
- -movflags +faststart "$PLAY"
-echo "MS2PLAY_DONE -> $PLAY"
-
-# ---- drums ------------------------------------------------------------------
-# songTime = (midiTime - 5.4497) x 1.05 + 11.0, verified at BOTH ends against the
-# plan. The three drum tracks already use GM percussion numbers (36 kick,
-# 38 snare, 42 closed hat, 49 crash), so they play as a kit on channel 10.
-TEMPO_SCALE=1.05 TRIM=5.4497 SHIFT=11.0 START=11.0 END="$END" \
- node midi-drums.mjs ~/Downloads/ms2_stage1.mid "$KIT.mid"
-fluidsynth -ni -F "$KIT.wav" -r 48000 -g 0.8 \
- /usr/share/soundfonts/FatBoy.sf2 "$KIT.mid" >/dev/null 2>&1
-
-# 1.8 was chosen by ear over 1.1 (too subtle) and 2.6 (too prominent). Mixed onto
-# the finished video with -c:v copy, so re-auditioning a level costs seconds.
-ffmpeg -nostdin -v error -y -i "$PLAY" -i "$KIT.wav" \
- -filter_complex "[1:a]volume=1.8,aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo[d];[0:a][d]amix=inputs=2:duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a]" \
- -map 0:v -map "[a]" -c:v copy -c:a aac -b:a 192k -movflags +faststart "$DRUMS"
-
-echo "MS2DRUMS_DONE -> $DRUMS"
-ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 "$DRUMS"
diff --git a/umtool/song/ms2-triangle.sh b/umtool/song/ms2-triangle.sh
@@ -1,94 +0,0 @@
-#!/bin/sh
-# Metal Slug 2, 3 voices, over the USER'S OWN EDIT as one continuous background,
-# handing over on the game's own "MISSION 1 START".
-#
-# RECOVERED 2026-08-17 from a session transcript: this file had been lost from the
-# repo, and it is the only place the announcer handover was worked out. Everything
-# that now uses that handover (ms2-jerbg.sh, which is the builder for the shipped
-# cuts) descends from it, so it is restored rather than left as a dead reference
-# in metal-slug.md.
-#
-# Two changes from the earlier bg builds.
-#
-# 1. ONE background, not intro-plus-body. `intro/ms2-intro.mp4` is jer's edit
-# 0->14.47s and `intro/ms2-bg.mp4` is the same edit from 14.4706s on, and
-# ms2-bg.mp4 carries NO AUDIO STREAM AT ALL -- which is why that build had no
-# game sound under it. Taking jer-metalslug-bg.mp4 whole keeps the footage
-# continuous AND keeps its audio, and lets the song start anywhere.
-#
-# 2. The handover is the announcer. "MISSION 1 START" sits at 11.901s in that
-# audio -- located by cross-correlating the clean announcer clip against it,
-# and corroborated by the energy dropping to rms 1115 at 11.5s before jumping
-# to 5433 at 12.0s. The background's own sound is CUT there and the CLEAN clip
-# plays instead, with the ums fading in underneath it, because in the game the
-# music starts alongside that voice clip.
-#
-# The picture is a TRIANGLE: the melody panel is only a little larger than the
-# two accompaniment panels and sits top-centre, with those two in the bottom
-# corners, so the character at the bottom centre is framed rather than covered.
-set -e
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-D=/home/user/reports/quartering-uh-song
-T=/home/user/.claude/jobs/efbe67a7/tmp
-
-BG=${BG:-$D/jer-metalslug-bg.mp4}
-ANNOUNCER=${ANNOUNCER:-$D/MS_Mission_1_start.ogg}
-START_AT=${START_AT:-11.901} # "MISSION 1 START" in the background's audio
-UM_FADE=${UM_FADE:-1.6} # the ums swell in under the announcer
-ANN_GAIN=${ANN_GAIN:-1.0}
-PLANF=${PLANF:-ms2-plan-tri.json}
-OUT=${OUT:-$D/quartering-metalslug2-triangle.mp4}
-# PAD, not crop. The transcript's copy of this script said crop; the build that
-# actually SHIPPED is pillarboxed, and metal-slug.md gives the reason -- the
-# source is 640x480 and cropping 4:3 to 16:9 cuts the top and bottom off the
-# gameplay (the mistake documented on Mario RPG). The pips then overhang the
-# pillarbox bars, which is intended.
-BG_FIT=${BG_FIT:-pad}
-mkdir -p "$(dirname "$OUT")"
-
-export MAX_IOI=0.70 WINDOW=700 LOCAL=0 TEMPO_SCALE=1.05 WOBBLE_W=0.08 TIME_LIMIT=999
-export VOICES='[{"name":"melody","lead":"ms2-lead-t.json","gain":1.0,"role":"full","unique":true,"reuseGap":20,"contourMax":12,"fill":true},{"name":"bass","lead":"ms2-bass-t.json","gain":0.68,"role":"pip1","unique":true,"reuseGap":12,"contourMax":4,"poolFactor":1.3,"fill":false},{"name":"rhythm","lead":"ms2-gtr-t.json","gain":0.55,"role":"pip2","unique":true,"reuseGap":10,"contourMax":6,"poolFactor":1.0,"fill":false}]'
-
-if [ "${REUSE_PLAN:-0}" != "1" ]; then
- node arrange-poly.mjs
- node -e "
- const fs=require('fs');
- const p=JSON.parse(fs.readFileSync('poly-plan.json','utf8'));
- const first=Math.min(...p.voices.filter(v=>v.plan.length).map(v=>v.plan[0].slotStart));
- const SH=$START_AT-first;
- for(const v of p.voices) for(const n of v.plan) n.slotStart=+(n.slotStart+SH).toFixed(4);
- fs.writeFileSync('$PLANF',JSON.stringify(p,null,1));
- console.log('shifted '+SH.toFixed(3)+'s -> first note '+
- Math.min(...p.voices.filter(v=>v.plan.length).map(v=>v.plan[0].slotStart)).toFixed(3)+'s, ends '+
- Math.max(...p.voices.flatMap(v=>v.plan.map(n=>n.slotStart+n.slotDur))).toFixed(3)+'s');"
-fi
-[ "${ARRANGE_ONLY:-0}" = "1" ] && exit 0
-
-# BG_MAIN_SCALE 0.36 -> a 460x260 melody panel, 1.20x the 384x216 accompaniment
-# panels, and BG_MAIN_TOP 108 keeps the top edge where it already was.
-# TAIL_FADE 4.5 covers the loop-back restatement; the picture now fades with it.
-if [ "${REUSE_BODY:-0}" = "1" ] && [ -f "$T/ms2tri-body.mp4" ]; then
- echo "reusing $T/ms2tri-body.mp4"
-else
- PLAN=$PLANF TAIL_FADE=4.5 BG_VIDEO=$BG BG_FIT=$BG_FIT LAYOUT=bg \
- BG_MAIN_SCALE=0.36 BG_MAIN_TOP=108 \
- node render-poly.mjs "$T/ms2tri-body.mp4"
-fi
-
-END=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$T/ms2tri-body.mp4")
-echo "background sound full to ${START_AT}s then CUT; announcer + ums from there; ends ${END}s"
-
-# [game] the edit's own sound, cut dead at the announcer
-# [ann] the CLEAN announcer clip, in its place
-# [song] the ums, swelling in underneath it
-ffmpeg -nostdin -v error -y -i "$T/ms2tri-body.mp4" -i "$BG" -i "$ANNOUNCER" \
- -filter_complex "
-[1:a]atrim=0:${END},asetpts=PTS-STARTPTS,volume='if(lt(t,${START_AT}),1.0,0.0)':eval=frame[game];
-[2:a]aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo,volume=${ANN_GAIN},adelay=delays=$(node -e "console.log(String(Math.round($START_AT*1000)))"):all=1,apad=whole_dur=${END}[ann];
-[0:a]afade=t=in:st=${START_AT}:d=${UM_FADE}[song];
-[song][game][ann]amix=inputs=3:duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a]" \
- -map 0:v -map "[a]" -c:v copy -c:a aac -b:a 192k -ar 48000 -ac 2 \
- -movflags +faststart "$OUT"
-
-echo "MS2TRI_DONE -> $OUT"
-ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 "$OUT"
diff --git a/umtool/song/paths.mjs b/umtool/song/paths.mjs
@@ -12,9 +12,78 @@
// SONG_DATA the bulk data, wherever it currently sits
//
// Point SONG_DIR at a copy to run the toolchain somewhere else. The default is
-// the job temp dir the corpus was mined into, which is where it still is.
+// ~/.local/share/archilyzer/song, and the data does NOT have to move there: a
+// symlink is the supported way to keep it where it is,
+//
+// mkdir -p ~/.local/share/archilyzer
+// ln -s <where the song data is> ~/.local/share/archilyzer/song
+//
+// SONG_DATA is that path's REALPATH, so lib/paths.mjs's SONG_SCRATCH
+// (dirname(SONG_DATA), where render-poly.mjs writes its scratch and logs) keeps
+// pointing at the real directory the data sits in, not at ~/.local/share. A
+// path that does not exist has no realpath; it is used as given, so a machine
+// with no song data still gets a SONG_DATA -- one every reader finds empty
+// (e2e/fixtures/song-capabilities.mjs says which specs that skips).
+import { realpathSync } from "node:fs";
+import os from "node:os";
import path from "node:path";
-export const SONG_DATA = path.resolve(
- process.env.SONG_DIR ?? "/home/user/.claude/jobs/efbe67a7/tmp/song",
+const requested = path.resolve(
+ process.env.SONG_DIR ?? path.join(os.homedir(), ".local", "share", "archilyzer", "song"),
);
+
+const realOrAsGiven = (p) => {
+ try {
+ return realpathSync(p);
+ } catch {
+ return p;
+ }
+};
+
+export const SONG_DATA = realOrAsGiven(requested);
+
+/**
+ * The um-song deliverables tree (thumbs/, videos/, the .plan.json files).
+ * Defined HERE rather than in lib/paths.mjs, which re-exports it, because the
+ * song scripts import only their siblings: the e2e fixture copies this
+ * directory's .mjs files into its own code/ and runs them from there, where
+ * ../lib does not exist.
+ */
+export const SONG_REPORTS = path.resolve(
+ process.env.SONG_REPORTS_DIR ?? path.join(os.homedir(), "reports", "quartering-uh-song"),
+);
+
+/**
+ * `p` relative to `root` when it lies inside it, else `p` unchanged.
+ *
+ * For the paths a song script RECORDS in a tracked JSON file
+ * (thumb-manifest.json, thumb-accepted.json): an absolute path there is one
+ * machine's home directory. The readers take either form -- lib/paths.mjs
+ * resolveInRoots binds a relative path to the FIRST root, SONG_REPORTS. Both
+ * sides are compared as given AND through realpath (of the deepest part that
+ * exists, so a file not written yet still counts), so a path spelled via the
+ * ~/.local/share symlink is inside the realpath'd SONG_DATA. A relative `p` is
+ * returned as it is.
+ */
+export function relTo(root, p) {
+ if (typeof p !== "string" || !path.isAbsolute(p)) return p;
+ const real = (x) => {
+ const rest = [];
+ for (let cur = path.resolve(x); ; cur = path.dirname(cur)) {
+ try {
+ return path.join(realpathSync(cur), ...rest);
+ } catch {
+ if (path.dirname(cur) === cur) return path.resolve(x);
+ rest.unshift(path.basename(cur));
+ }
+ }
+ };
+ const roots = [...new Set([path.resolve(root), real(root)])];
+ const paths = [...new Set([path.resolve(p), real(p)])];
+ for (const r of roots) {
+ for (const abs of paths) {
+ if (abs.startsWith(r + path.sep)) return path.relative(r, abs);
+ }
+ }
+ return p;
+}
diff --git a/umtool/song/pk-v12.sh b/umtool/song/pk-v12.sh
@@ -1,69 +0,0 @@
-#!/usr/bin/env bash
-# Pokemon v12 -- everything decided after v11, which needs a different build shape.
-#
-# v11 mixed all three voices inside one body render. Two decisions since make that
-# impossible: the run voice's audio gets its own effects (climb-fx), and its picture is
-# the X strokes rather than a fixed panel. Splitting the run out solves both at once --
-# a body rendered from melody+bass has no run panel to suppress.
-#
-# 1 run rendered alone -> its audio only, picture discarded
-# 2 climb-fx on that audio -> loudness/brightness/pan track the pitch
-# 3 body from melody+bass, and pk3.sh's spare CHOP input carries the run audio back in
-# 4 climb-panel over the result -> the X strokes
-#
-# Steps 1 and 3 are the expensive ones; everything else is a single encode.
-set -euo pipefail
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-T=/home/user/.claude/jobs/efbe67a7/tmp
-D=/home/user/reports/quartering-uh-song
-S=/home/user/Projects/yt-dlp-transcript-browser/umtool/song
-
-RUN_GAIN=${RUN_GAIN:-0.55}
-AMP=${AMP:-0.5} BRI=${BRI:-0.8} PAN=${PAN:-0.7}
-TILE_W=${TILE_W:-420} MIN_RUN=${MIN_RUN:-4} TRAIL=${TRAIL:-0.30}
-EXTENT=${EXTENT:-0.82} MARGIN=${MARGIN:-8}
-OUT=${OUT:-$D/quartering-pokemon-v12.mp4}
-
-# ---- 1. the run voice, alone ------------------------------------------------
-if [ ! -f "$T/v12-run.mp4" ]; then
- echo "=== run voice (516 notes) ==="
- PLAN=pkmn-v12-run.json BG_VIDEO=$PWD/intro/pkmn-late.mp4 BG_FIT=pad LAYOUT=bg OUT_END=0 \
- node render-poly.mjs "$T/v12-run.mp4"
-fi
-
-# ---- 2. its effects ---------------------------------------------------------
-# Applied to the run ALONE, which is the whole reason it is rendered separately: the
-# cues are meant to carry this line's contour, not to tilt the melody and bass with it.
-if [ ! -f "$T/v12-run-fx.wav" ]; then
- echo "=== climb-fx on the run ==="
- AMP=$AMP BRI=$BRI PAN=$PAN node "$S/climb-fx.mjs" pkmn-v12-run.json run "$T/v12-run.mp4" "$T/v12-run-fx.mp4"
- ffmpeg -nostdin -v error -y -i "$T/v12-run-fx.mp4" -vn -ar 48000 -ac 2 "$T/v12-run-fx.wav"
-fi
-
-# ---- 3. body + the whole finale ---------------------------------------------
-# CHOP is pk3.sh's spare audio input; the run rides back in through it at RUN_GAIN.
-echo "=== body (melody + bass) and final assembly ==="
-REUSE_BG=1 PLANF=pkmn-v12-body.json \
- JINGLE_VIDEO=1 JINGLE=$T/pk9/jingle-finale.mp4 \
- CHOP="$T/v12-run-fx.wav" CHOP_GAIN=$RUN_GAIN \
- OUT="$T/v12-base.mp4" \
- FILLER='1745-1782:3.0:17.5 2189-2223:3.0:25.0 3710-3740:1.0:24.0' \
- bash "$S/pk3.sh"
-
-# ---- 4. the run's picture, in two halves ------------------------------------
-# The run is on screen for the whole song, but as two different things, and the split
-# is the melody: while the melody sings, the run is an inner voice and sits in the
-# ordinary bottom-right pip; while the melody rests, it takes the frame as the X
-# strokes. One gate drives both, from opposite sides, so the line is drawn exactly
-# once at every instant -- never doubled, never absent.
-echo "=== run pip (while the melody sounds) ==="
-GATE_PLAN=pkmn-v12-body.json GATE_VOICE=melody \
- node "$S/run-pip.mjs" pkmn-v12-run.json run "$T/v12-run.mp4" "$T/v12-base.mp4" "$T/v12-pip.mp4"
-
-echo "=== climb-panel X strokes (while the melody rests) ==="
-TILE_W=$TILE_W MIN_RUN=$MIN_RUN TRAIL=$TRAIL EXTENT=$EXTENT MARGIN=$MARGIN \
- GATE_PLAN=pkmn-v12-body.json GATE_VOICE=melody \
- node "$S/climb-panel.mjs" pkmn-v12-run.json run "$T/v12-pip.mp4" "$OUT"
-
-echo "PK12_DONE -> $OUT"
-ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 "$OUT"
diff --git a/umtool/song/pk2.sh b/umtool/song/pk2.sh
@@ -1,166 +0,0 @@
-#!/bin/sh
-# Pokemon FR/LG wild battle, rebuilt on the 3,959-clip palette over real
-# gameplay -- the most out-of-date deliverable in the set (its only build was
-# made at ~1,050 clips, before the window and renderer fixes).
-#
-# The background is FCPlaythroughs' Pokemon Red longplay. Timings were derived,
-# not guessed, and all three sources were tied to ONE clock first:
-#
-# * pkmn-ref.mp4 (a combined A+V section) starts at source 620.000s exactly --
-# established by cross-correlating its audio against the complete audio
-# track, which matched to the sample.
-# * pkmn-long-v.webm (the high-quality video) starts at source 644.134s --
-# established by matching the 1100ms black run, which sits at 36.267s in the
-# reference and 12.133s here.
-# * the audio is cut from the complete track at 644.134s to match.
-#
-# That mattered: a first attempt cut the audio at the REQUESTED start rather
-# than the actual one, and the two were ~14s apart, so no audio event lined up
-# with any picture event.
-#
-# The splice is on the AUDIO, not the black frames. The encounter's black run
-# ends at 13.233s, but the game's own rapid-notes alarm keeps playing to ~13.2s
-# and the battle theme's downbeat lands at 13.50s -- which is where the ums come
-# in, so the game plays its own intro and hands over on the beat.
-#
-# FLOURISH=drop the MIDI's own rapid intro is removed; the background's
-# alarm is the intro (default)
-# FLOURISH=keep the ums play the rapid intro themselves, and the game audio
-# ducks BEFORE the alarm so it is not doubled
-set -e
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-D=/home/user/reports/quartering-uh-song
-T=/home/user/.claude/jobs/efbe67a7/tmp
-
-BG_START=${BG_START:-5.866} # 10:50 in the source, in pkmn-long-v time
-SPLICE=${SPLICE:-13.500} # the battle theme's downbeat, same clock
-# Where the game's own ALARM starts, same clock. Read off the pitch contour: the
-# route music before it is B4/C#5/D5, and the alarm is the G5-F#5-F5-G6 figure --
-# audibly higher and quite different, which is how it is picked out. It runs
-# 2.6s, and the MIDI's own alarm is 33 notes over 2.5s, so the two agree.
-#
-# 10.966 was the ORIGINAL reading and it is 1.14s LATE -- the cut landed after the
-# game's alarm instead of on it, so v3 played the game's alarm and then the ums'.
-# The corrected onset is 9.826, measured by the high/low band-energy ratio rather
-# than eyeballed off a pitch contour. Every shipped build passes 9.826 by env; this
-# default now matches them instead of contradicting them.
-ALARM_AT=${ALARM_AT:-9.826}
-FLOURISH=${FLOURISH:-keep}
-# 0 = CUT the game's own sound where the ums take over, rather than leaving a
-# bed under them. The battle music would otherwise play its version of the same
-# tune underneath ours.
-DUCK=${DUCK:-0.0}
-PLANF=${PLANF:-pkmn-plan-v2.json}
-OUT=${OUT:-$D/quartering-pokemon-v2.mp4}
-mkdir -p "$(dirname "$OUT")"
-
-# The MIDI's own alarm is notes 0..32 of the lead (t=0 -> 2.500s, 78ms apart);
-# the tune proper starts at 2.969s. Those are the notes FLOURISH=drop removes.
-CUTAT=${CUTAT:-2.95}
-SCALE=${SCALE:-1.15}
-if [ "$FLOURISH" = "keep" ]; then CUTAT=-1; HANDOVER=$ALARM_AT; else HANDOVER=$SPLICE; fi
-
-# ---- arrange ----------------------------------------------------------------
-# poolFactor 6.0 on the accompaniment, IN LINE WITH THE OTHER TUNES. The old build ran the
-# accompaniment at ~1.0, the starved setting every other tune has since moved off
-# -- Metal Slug went 1.05 -> 1.3, Mario RPG to 6.0, Mortal Kombat to 2.6.
-if [ "${REUSE_PLAN:-0}" != "1" ]; then
- # Voices from PkmRB-Battle1.mid -- Pokemon RED/BLUE, which is what the
- # background actually is. Its track names spell the credit (sequenced by
- # Joao Buaes), like the Mortal Kombat file, so tracks are picked by INDEX:
- # 2 -> melody 144 notes, IOI p50 0.469s, and it carries the alarm
- # 3 -> bass 471 notes, IOI p50 0.156s
- # 1 -> harm 516 notes, IOI p50 0.078s -- LEFT OUT. At 78ms every note is
- # shorter than the palette's 0.14s minimum clip, so it can only
- # come out as fragments; Metal Slug's rhythm voice at 0.183s
- # already reads as percussion rather than melody.
- # CMAX is how far a melody note may be STRETCHED before the arranger folds it
- # an octave instead, and it is the only thing measured that moves the
- # over-pitching at all.
- #
- # At 14 a note may be dragged more than an octave rather than folded, and the
- # notes that need it are exactly the ones at the edges of the corpus's pitch
- # range (the corpus is median 103Hz / p95 127Hz; this melody asks for 73Hz and
- # 196Hz). Measured on the Pokemon melody, 14 -> 9:
- #
- # under 0.5 semitones 59 -> 122 2-3 semitones 51 -> 2
- # 3+ semitones 10 -> 0 down by 2+ 53 -> 2
- #
- # 9, 7 and 5 are identical, so it is a stable setting rather than a tuned one.
- # It is not free: 25 notes fold an octave instead of being dragged, so the
- # melody's contour changes at those notes. That is the trade -- an octave
- # displacement in place of an audibly dragged note.
- #
- # Raising SHIFT_W (what a semitone costs in the score) does NOT work: 12 -> 80
- # leaves the 2-3 cluster at 51/49/50/52. Nor does disabling the 115Hz
- # other-speaker gate, which moves exactly one note.
- # 9, not the 14 this defaulted to originally: 14 is the setting the measurements
- # below argue AGAINST, and every shipped build passes 9 by env.
- CMAX=${CMAX:-9}
- export TEMPO_SCALE=$SCALE MAX_IOI=0.70 WINDOW=700 LOCAL=0 WOBBLE_W=0.08
- export VOICES='[{"name":"melody","lead":"pkrb-lead.json","gain":1.0,"role":"full","unique":true,"reuseGap":20,"contourMax":'"$CMAX"',"fill":true},{"name":"bass","lead":"pkrb-bass.json","gain":0.62,"role":"pip1","unique":true,"reuseGap":12,"contourMax":5,"poolFactor":4.0,"fill":false}]'
- node arrange-poly.mjs
- cp poly-plan.json "$PLANF.raw"
-fi
-
-# Drop the flourish if asked, then shift so the first surviving note lands on
-# the handover.
-#
-# SKIP_SHIFT=1 leaves $PLANF exactly as it is.
-#
-# This block REGENERATES $PLANF from $PLANF.raw, and it runs on every pass --
-# including a REUSE_PLAN=1 render. So anything edited into the plan between the
-# arrange and the render is silently thrown away: a run that arranged, applied
-# nine swap-clip fixes, verified them with a scan, and then rendered produced a
-# video with none of the fixes in it, and the scan output looked perfectly
-# correct because it ran before the clobber. Swap first, then render with
-# SKIP_SHIFT=1.
-if [ "${SKIP_SHIFT:-0}" = "1" ]; then
- echo "SKIP_SHIFT=1: using $PLANF as it stands (already shifted)"
-else
-node -e "
-const fs=require('fs');
-const p=JSON.parse(fs.readFileSync('$PLANF.raw','utf8'));
-const CUT=$CUTAT;
-let dropped=0;
-if(CUT>0) for(const v of p.voices){const n0=v.plan.length; v.plan=v.plan.filter(n=>n.slotStart>=CUT*$SCALE-1e-6); dropped+=n0-v.plan.length;}
-const first=Math.min(...p.voices.filter(v=>v.plan.length).map(v=>v.plan[0].slotStart));
-const SH=$HANDOVER-$BG_START-first;
-for(const v of p.voices) for(const n of v.plan) n.slotStart=+(n.slotStart+SH).toFixed(4);
-fs.writeFileSync('$PLANF',JSON.stringify(p,null,1));
-const f2=Math.min(...p.voices.filter(v=>v.plan.length).map(v=>v.plan[0].slotStart));
-const l2=Math.max(...p.voices.flatMap(v=>v.plan.map(n=>n.slotStart+n.slotDur)));
-console.log('flourish '+('$FLOURISH')+': dropped '+dropped+' notes; shifted '+SH.toFixed(3)+
- 's -> first note '+f2.toFixed(3)+'s, ends '+l2.toFixed(3)+'s');"
-fi
-[ "${ARRANGE_ONLY:-0}" = "1" ] && exit 0
-
-# ---- background -------------------------------------------------------------
-# 10:50 onward, as one file, so the renderer's clock and this script's agree.
-if [ ! -f "$T/pkmn-bg.mp4" ]; then
- ffmpeg -nostdin -v error -y -ss "$BG_START" -i intro/pkmn-bg-full.mkv \
- -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p -video_track_timescale 30000 \
- -c:a aac -b:a 192k -ar 48000 -ac 2 "$T/pkmn-bg.mp4"
-fi
-
-PLAN=$PLANF BG_VIDEO=$T/pkmn-bg.mp4 BG_FIT=pad LAYOUT=bg \
- node render-poly.mjs "$T/pkmn-body.mp4"
-
-# ---- the game's own sound under it ------------------------------------------
-# Full until the handover -- the transition and the alarm ARE the intro -- then
-# down under the ums.
-END=$(node -e "
-const {voices}=require('$PWD/$PLANF');
-console.log((Math.max(...voices.flatMap(v=>v.plan.map(n=>n.slotStart+n.slotDur)))+0.7).toFixed(2));")
-H=$(node -e "console.log(($HANDOVER-$BG_START).toFixed(3))")
-echo "game audio full to ${H}s, then ${DUCK}; output ends ${END}s"
-ffmpeg -nostdin -v error -y -i "$T/pkmn-body.mp4" -i "$T/pkmn-bg.mp4" \
- -filter_complex "
-[1:a]volume='if(lt(t,${H}),1.0,${DUCK})':eval=frame,atrim=0:${END},asetpts=PTS-STARTPTS[game];
-[0:a]atrim=0:${END},asetpts=PTS-STARTPTS[song];
-[song][game]amix=inputs=2:duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a]" \
- -map 0:v -map "[a]" -t "$END" -c:v copy -c:a aac -b:a 192k -ar 48000 -ac 2 \
- -movflags +faststart "$OUT"
-
-echo "PKMN_DONE -> $OUT"
-ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 "$OUT"
diff --git a/umtool/song/pk3.sh b/umtool/song/pk3.sh
@@ -1,375 +0,0 @@
-#!/usr/bin/env bash
-# Pokemon v9 -- the 35:10 opening and the caught-Pikachu finale.
-#
-# v8 opened at source 10:50 and simply stopped when the notes ran out. This one
-# opens in a GAP between notes of the route music at ~35:10, lets the encounter
-# alarm fire into the ums as before, and ends MK-style: right after the song's
-# last note the player throws a Poke Ball, catches a Pikachu, and an um-ified
-# "caught" jingle plays once the frame stills on "PIKACHU was caught!".
-#
-# READ specs/pokemon.md FIRST. In particular:
-# * every timing here was MEASURED, and the measurements are recorded there;
-# * `pk2.sh` clobbers its plan on every pass -- this script never regenerates
-# one, it REUSES v8's, which already lands its first note on the handover.
-#
-# ---------------------------------------------------------------------------
-# The one clock
-#
-# intro/pkmn-late.mp4 is a 302+251 MERGED fetch, and t=0 in it is source
-# 2090.036s -- established by cross-correlating its audio against the complete
-# local track at two independent windows (t=10 and t=45), which agreed to 0.000s.
-# Fetching 302 alone gives no audio to pin with, and the same requested range came
-# back a different length (83.415s vs 80.014s), so both streams must come from the
-# one file. Every SECTION time below is in that file's clock.
-#
-# ---------------------------------------------------------------------------
-# The shape
-#
-# [A] section BG_START -> SEG_A_END the entry gap, the alarm, the battle
-# [F] filler more battle footage, see FILLER
-# [C] section SEG_C_START -> SEG_C_END the ball, the catch, the jingle
-#
-# A is only ~20.7s and the song is 90.2s, so F exists purely to carry the song
-# to its last note with the throw waiting on the other side. F is TRIMMED to
-# whatever length makes the throw land GAP seconds after the final note, so the
-# ranges given in FILLER only have to be long ENOUGH, not exact.
-set -euo pipefail
-# The bulk data -- intro/, the plan files, cand2/, the rendered scratch -- lives in
-# the job temp dir, not beside this script. Same as pk2.sh.
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-
-T=/home/user/.claude/jobs/efbe67a7/tmp
-D=/home/user/reports/quartering-uh-song
-SRC=intro/pkmn-late.mp4
-URL='https://www.youtube.com/watch?v=D1SrSFZrV7A'
-SFX=$D/sfx/pkmn-catch.wav
-
-# ---- the measured section times ---------------------------------------------
-# Inside the 20.188->20.468 inter-note trough (source 2110.224->2110.504), not on
-# its leading onset, so the entry does not clip a note attack. It also puts the
-# alarm at bg 3.960 -- the SAME handover v8 shipped, which is why v8's plan can be
-# reused without reshifting a single note.
-BG_START=${BG_START:-20.324}
-ALARM_AT=${ALARM_AT:-24.284} # source 2114.320, found by pulse REGULARITY (82.4ms)
-# 41.000 is where the FIGHT menu returns after the two moves. The user's
-# constraint: the loop/filler must not contain any of the catching animation, and
-# the catch begins at 42.800 when the ITEM menu opens. Cutting at 41.000 keeps
-# every frame of the ball out of the body.
-SEG_A_END=${SEG_A_END:-41.000}
-SEG_C_START=${SEG_C_START:-41.000}
-SEG_C_END=${SEG_C_END:-54.350}
-GAP=${GAP:-0.80} # dead air between the last note and the throw
-
-# Filler: space-separated SOURCE ranges "start-end" in seconds. These are only
-# ever seen, never heard -- the game's own audio is cut from the handover on --
-# so they need no pinning, unlike the section above.
-FILLER=${FILLER:-}
-
-PLANF=${PLANF:-pkmn-plan-v8.json}
-OUT=${OUT:-$D/quartering-pokemon-v9.mp4}
-
-# ---- catch SFX event times, from the sidecar --------------------------------
-# The donor recording is a DIFFERENT catch attempt: its wiggle SPACING matches the
-# footage exactly (0.933s) but the group sits ~0.27s off and it supplies one wiggle
-# fewer than the picture shows. So the asset is NOT slid as a block -- each event is
-# cut out and delayed to its own measured picture time, and a wiggle tick is reused
-# for the third. That is what pkmn-catch.json's event offsets exist for.
-#
-# asset event picture (section) what it lands on
-A_THROW=0.036; P_THROW=44.860 # the ball leaves the trainer's hand
-A_LAND=1.0405; P_LAND=45.683 # Pikachu is absorbed
-A_SETTLE=1.6632; P_SETTLE=46.417 # the ball drops and settles
-A_W2=2.5937; P_W1=47.150 # wiggle 1
-A_W3=3.5287; P_W2=48.083 # wiggle 2
- P_W3=49.017 # wiggle 3 -- reuses the A_W3 tick
-P_JINGLE=50.344 # "PIKACHU was caught!" finishes; frame stills
-
-JINGLE=${JINGLE:-$T/pkc-jingle.mp4}
-SFX_GAIN=${SFX_GAIN:-0.85}
-JINGLE_GAIN=${JINGLE_GAIN:-1.0}
-
-n() { node -e "console.log(String((($1)).toFixed(4)))"; } # String(): node ANSI-colourises bare numbers
-ms() { node -e "console.log(String(Math.round(($1)*1000)))"; }
-
-LEN_A=$(n "$SEG_A_END - $BG_START")
-HANDOVER=$(n "$ALARM_AT - $BG_START")
-
-# The song, straight off v8's plan -- not rearranged, not reshifted.
-LAST=$(node -e "
-const {voices}=require('$PWD/$PLANF');
-console.log(String(Math.max(...voices.flatMap(v=>v.plan.map(x=>x.slotStart+x.slotDur))).toFixed(4)));")
-
-# Where segment C must begin, in bg time, for the throw to land GAP after the last note
-OFF_C=$(n "$LAST + $GAP - $P_THROW + $SEG_C_START")
-NEED=$(n "$OFF_C - $LEN_A")
-END=$(n "$SEG_C_END - $SEG_C_START + $OFF_C")
-
-echo "song: first note 3.960s, last note ends ${LAST}s"
-echo "segment A ${BG_START}->${SEG_A_END} (${LEN_A}s), handover at bg ${HANDOVER}s"
-echo "filler needed: ${NEED}s segment C starts at bg ${OFF_C}s output ends ${END}s"
-
-if [ -z "$FILLER" ]; then
- echo
- echo "FILLER is empty and ${NEED}s of it is needed."
- echo "Pass source ranges, e.g. FILLER='3600-3640 7200-7235' (seconds, start-end)."
- exit 2
-fi
-
-# WORK is the scratch directory for the spliced background and its pieces. It is
-# configurable so a cut of a different LENGTH -- which needs a different background --
-# does not overwrite the one an existing build was made from.
-WORK=${WORK:-$T/pk9}
-mkdir -p "$WORK" "$(dirname "$OUT")"
-
-# ---- normalise every background piece to ONE format --------------------------
-# concat demuxer needs identical streams. 800x720@60 is the source's own geometry,
-# so segment A and C are never resampled; only fetched filler is.
-norm() { # <in> <ss> <t> <out>
- ffmpeg -nostdin -v error -y -ss "$2" -t "$3" -i "$1" \
- -vf "scale=800:720:force_original_aspect_ratio=decrease,pad=800:720:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=60,format=yuv420p" \
- -c:v libx264 -preset medium -crf 18 -video_track_timescale 60000 \
- -c:a aac -b:a 192k -ar 48000 -ac 2 "$4"
-}
-
-# REUSE_BG=1 keeps an already-spliced background, REUSE_BODY=1 an already-rendered
-# body. The body is ~15 minutes of clip decoding, so re-running it to fix something
-# downstream of it is pure waste.
-if [ "${REUSE_BG:-0}" = "1" ] && [ -f "$WORK/bg.mp4" ]; then
- echo "REUSE_BG=1: keeping $WORK/bg.mp4 ($(ffprobe -v error -show_entries format=duration -of csv=p=0 "$WORK/bg.mp4")s)"
-else
-
-echo "--- segment A ---"
-norm "$SRC" "$BG_START" "$LEN_A" "$WORK/a.mp4"
-
-echo "--- filler ---"
-i=0; ACC=0; LIST="$WORK/concat.txt"
-: > "$LIST"
-echo "file '$WORK/a.mp4'" >> "$LIST"
-#
-# Each entry is FS-FE[:OFF[:LEN]] in SOURCE seconds.
-#
-# OFF skips into the FETCHED file and is not optional bookkeeping:
-# --download-sections does not start where it is asked (630->620, 645->644.134 are
-# the recorded cases), so the first seconds of a window are routinely the walk INTO
-# the battle rather than the battle. OFF is read off a frame strip of the fetch.
-#
-# LEN caps how much of the window is used, and it MATTERS here: most wild
-# encounters in this longplay are Pokedex-filling CATCHES, so a window run to its
-# end shows another Pokemon being caught -- which would undercut the one the whole
-# finale is built around. LEN stops each battle before the ball.
-for R in $FILLER; do
- OIFS=$IFS; IFS=:; set -- $R; IFS=$OIFS
- SPEC=$1; OFF=${2:-0}; LEN=${3:-}
- FS=${SPEC%%-*}; FE=${SPEC##*-}
- i=$((i+1))
- # A window fetched earlier by hand is reused rather than pulled again.
- RAW="$WORK/fill-${FS}.mp4"
- [ -f "$RAW" ] || RAW="$WORK/f${i}-raw.mp4"
- if [ ! -f "$RAW" ]; then
- # Fetch merged, as ever: a 302-only fetch has no audio and cuts differently.
- yt-dlp -f 302+251 --download-sections "*${FS}-${FE}" -P "$WORK" -o "f${i}-raw.%(ext)s" "$URL" >/dev/null 2>&1
- for E in webm mkv mp4; do
- [ -f "$WORK/f${i}-raw.$E" ] && [ "$WORK/f${i}-raw.$E" != "$RAW" ] && mv "$WORK/f${i}-raw.$E" "$RAW"
- done
- fi
- DUR=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$RAW")
- HAVE=$(n "$DUR - $OFF")
- [ -n "$LEN" ] && HAVE=$(node -e "console.log(String(Math.min($HAVE,$LEN).toFixed(4)))")
- WANT=$(n "$NEED - $ACC")
- USE=$(node -e "console.log(String(Math.min($HAVE,$WANT).toFixed(4)))")
- if node -e "process.exit(($USE)>0.04?0:1)"; then
- norm "$RAW" "$OFF" "$USE" "$WORK/f${i}.mp4"
- echo "file '$WORK/f${i}.mp4'" >> "$LIST"
- ACC=$(n "$ACC + $USE")
- echo " filler $i: source ${FS}-${FE} +${OFF}s in, have ${HAVE}s, used ${USE}s (total ${ACC}/${NEED})"
- fi
-done
-if node -e "process.exit(($NEED-$ACC)>0.05?0:1)"; then
- echo "SHORT by $(n "$NEED - $ACC")s of filler -- add another range to FILLER."
- exit 3
-fi
-
-echo "--- segment C ---"
-norm "$SRC" "$SEG_C_START" "$(n "$SEG_C_END - $SEG_C_START")" "$WORK/c.mp4"
-echo "file '$WORK/c.mp4'" >> "$LIST"
-
-echo "--- splice ---"
-ffmpeg -nostdin -v error -y -f concat -safe 0 -i "$LIST" -c copy "$WORK/bg.mp4"
-ffprobe -v error -show_entries format=duration -of default=nw=1 "$WORK/bg.mp4"
-
-fi
-
-# ---- re-derive the segment C offset from what was ACTUALLY built -------------
-# OFF_C above is where segment C was PLANNED to start, and it sizes the filler.
-# It is not where segment C actually starts: every piece gets quantised to a frame
-# when it is encoded, so a.mp4 comes out 20.699s against a requested 20.676s and
-# the pieces together run ~70ms long. Placing the catch SFX on the planned figure
-# put every tick 67ms AHEAD of the wiggle it belongs to -- measured in the finished
-# file, picture wiggles at 93.367/94.300/95.233 against ticks at 93.300/94.233/95.167.
-#
-# So the events are placed on the REAL offset, taken as (whole background - segment
-# C), which needs no bookkeeping of the individual pieces. The song does not move;
-# the gap after its last note just absorbs the difference.
-OFF_C=$(node -e "
-const {execFileSync}=require('child_process');
-const d=f=>Number(execFileSync('ffprobe',['-v','error','-show_entries','format=duration','-of','csv=p=0',f]).toString().trim());
-console.log(String((d('$WORK/bg.mp4')-d('$WORK/c.mp4')).toFixed(4)));")
-END=$(n "$SEG_C_END - $SEG_C_START + $OFF_C")
-echo "segment C really starts at bg ${OFF_C}s; throw lands $(n "$P_THROW - $SEG_C_START + $OFF_C - $LAST")s after the last note; output ends ${END}s"
-
-# ---- body: the ums over the spliced background ------------------------------
-# OUT_END=0 stops the renderer TRIMMING at the last note, but it does not make the
-# picture run any longer than the song -- see the video assembly below.
-if [ "${REUSE_BODY:-0}" = "1" ] && [ -f "$WORK/body.mp4" ]; then
- echo "REUSE_BODY=1: keeping $WORK/body.mp4 ($(ffprobe -v error -show_entries format=duration -of csv=p=0 "$WORK/body.mp4")s)"
-else
-PLAN=$PLANF BG_VIDEO=$WORK/bg.mp4 BG_FIT=pad LAYOUT=bg OUT_END=0 \
- node render-poly.mjs "$WORK/body.mp4"
-fi
-
-# ---- the mix ----------------------------------------------------------------
-# Game audio full until the handover (the route gap and the alarm ARE the intro),
-# then CUT -- not ducked: the battle music would otherwise play its own version of
-# this very tune underneath ours, and the catch would fight our jingle.
-J_MS=$(ms "$P_JINGLE - $SEG_C_START + $OFF_C")
-echo "jingle at bg $(n "$P_JINGLE - $SEG_C_START + $OFF_C")s"
-
-# Each SFX event is a SEPARATE -ss/-t input off the same wav, rather than one input
-# split six ways: the six taps need six different source ranges AND six different
-# delays, so demuxer-level seeking is both simpler and cheaper than splitting and
-# atrimming inside the graph.
-# The handover, and why a HARD CUT is what makes the intro and the song feel like two
-# separate pieces.
-#
-# The game runs at full until HANDOVER and is then CUT to zero, because from the
-# downbeat onwards it plays its own version of our tune and would double it. But
-# between the handover and the downbeat the game is not playing the theme -- it is
-# playing the ENCOUNTER ALARM, the same figure the ums take over. Cutting it there
-# throws away the one passage where the two could overlap, and the join lands as a
-# step: the game stops dead and the ums start alone.
-#
-# GAME_BED holds the game at a low level across that window and fades it to nothing by
-# the downbeat, so the ums' alarm rises out of the game's own rather than replacing it
-# between one sample and the next. It is 0 by default -- the previous hard cut -- and
-# the fade ends AT the downbeat, so nothing of the game's battle theme is ever heard
-# under ours, which is the reason the cut existed in the first place.
-GAME_BED=${GAME_BED:-0}
-DOWNBEAT=${DOWNBEAT:-7.374}
-if [ "$(node -e "console.log(String($GAME_BED>0?1:0))")" = "1" ]; then
- GVOL="if(lt(t,${HANDOVER}),1.0, if(lt(t,${DOWNBEAT}), ${GAME_BED}*(1-(t-${HANDOVER})/(${DOWNBEAT}-${HANDOVER})), 0.0))"
- echo "game audio: full to ${HANDOVER}s, then a ${GAME_BED} bed fading to 0 by the downbeat at ${DOWNBEAT}s"
-else
- GVOL="if(lt(t,${HANDOVER}),1.0,0.0)"
-fi
-FC="[1:a]aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo,volume='${GVOL}':eval=frame,atrim=0:${END},asetpts=PTS-STARTPTS,apad=whole_dur=${END}[game];"
-FC="$FC[0:a]aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo,apad=whole_dur=${END}[song];"
-FC="$FC[2:a]aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo,volume=${JINGLE_GAIN},adelay=delays=${J_MS}:all=1,apad=whole_dur=${END}[jing];"
-
-IN_SFX=""
-k=0
-MIXIN="[song][game][jing]"
-for EV in "$A_THROW 0.492 $P_THROW" "$A_LAND 0.500 $P_LAND" "$A_SETTLE 0.238 $P_SETTLE" \
- "$A_W2 0.243 $P_W1" "$A_W3 0.244 $P_W2" "$A_W3 0.244 $P_W3"; do
- set -- $EV
- PAD=0.030
- SS=$(n "$1 - $PAD"); DU=$(n "$2 + 2*$PAD"); DLY=$(ms "$3 - $SEG_C_START + $OFF_C - $PAD")
- IDX=$((3+k))
- IN_SFX="$IN_SFX -ss $SS -t $DU -i $SFX"
- FC="$FC[${IDX}:a]aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo,volume=${SFX_GAIN},adelay=delays=${DLY}:all=1,apad=whole_dur=${END}[sfx${k}];"
- MIXIN="$MIXIN[sfx${k}]"
- echo " sfx $k: asset ${SS}+${DU}s -> bg $(n "$3 - $SEG_C_START + $OFF_C")s"
- k=$((k+1))
-done
-
-# ---- the chopped third voice -------------------------------------------------
-# PkmRB-Battle1's track 1 runs at 78ms a note, under the palette's 0.14s clip floor,
-# so every build has left it out -- and it is exactly what covers the melody's rests
-# in the game. Those rests are 20.3% of the song: six ~2.1s stretches where only the
-# quiet bass sounds, which is what "some stretches are kind of empty" meant.
-#
-# chop-voice.mjs makes it from slices of a handful of sustained ums instead of a
-# clip per note. It is AUDIO ONLY and deliberately has no video panel: there is no
-# single clip behind it to show, and it is a texture under the tune rather than a
-# voice with a face. That also means it mixes in HERE, costing one encode instead of
-# a 15-minute body re-render.
-CHOP_IN=""
-if [ -n "${CHOP:-}" ] && [ -f "$CHOP" ]; then
- CHOP_IN="-i $CHOP"
- FC="$FC[$((3+k)):a]aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo,volume=${CHOP_GAIN:-1.0},apad=whole_dur=${END}[chop];"
- MIXIN="$MIXIN[chop]"
- k=$((k+1))
- echo " chopped third voice: $CHOP at gain ${CHOP_GAIN:-1.0}"
-fi
-
-FC="$FC${MIXIN}amix=inputs=$((3+k)):duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a]"
-
-# shellcheck disable=SC2086
-# ---- video: the body, then the background carrying on past it -----------------
-# OUT_END=0 is NOT enough. render-poly stops its VIDEO where the song's audio
-# stops, whatever OUT_END says, so the body came out 93.066s against a 100.587s
-# background -- and the whole finale (the ball landing, the wiggles, the caught
-# screen) fell off the end. The container still measured 100.5s because the AUDIO
-# ran that long, which is exactly the trap mk-fatal-finish3.sh warns about: never
-# trust a container duration for where the picture actually ends.
-#
-# So the picture is assembled the way MK's is: the body up to where it really ends,
-# then the background itself from that point on. The tail is padded to 1280x720 the
-# same way BG_FIT=pad does inside the renderer, or it would jump at the join.
-TBODY=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$WORK/body.mp4")
-TBODY=$(node -e "console.log(String((Math.floor($TBODY*30)/30).toFixed(4)))") # to a 30fps frame
-echo "body ends ${TBODY}s; background carries the last $(n "$END - $TBODY")s"
-
-PADV="scale=1280:720:force_original_aspect_ratio=decrease,pad=1280:720:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=30,format=yuv420p"
-# render-poly DIMS the background (BG_DIM, default -0.10) so the um panels read
-# against it. Anything taken straight from bg.mp4 has not been through that, so the
-# picture visibly BRIGHTENS the moment the body ends -- reported as "you're dimming
-# the game during ums and it lightens during the finale". render-poly's own header
-# warns about exactly this join.
-#
-# The fix is to keep it dim the whole way rather than to ramp back: a constant grade
-# reads as deliberate, a switch reads as a mistake. The jingle segment needs nothing,
-# because render-poly composited it and so it is already dimmed.
-# 0, matching render-poly's new default: the background is not dimmed anywhere, so the
-# bg-derived tails need no dim to match it. This was -0.10 to hide a brightness step
-# that no longer exists.
-DIM=${DIM:-0}
-PADV_DIM="$PADV,eq=brightness=${DIM}"
-FC="$FC;[0:v]trim=0:${TBODY},setpts=PTS-STARTPTS[bodyv];"
-
-# The finale's um panels. Without this the jingle plays over bare game footage while
-# every other note in the video has a face on it, which reads as the ums stopping.
-#
-# $JINGLE supplies BOTH streams here -- it is rendered by render-poly over the
-# finale's own frames (cut straight out of the spliced bg.mp4, so the frames are
-# identical rather than merely similar) and its audio is the ums. One input, so the
-# SFX input indices below do not have to shift.
-JAT=$(n "$P_JINGLE - $SEG_C_START + $OFF_C")
-JDUR=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$JINGLE")
-JEND=$(n "$JAT + $JDUR")
-if [ "${JINGLE_VIDEO:-0}" = "1" ]; then
- echo "jingle panels on screen ${JAT}s -> ${JEND}s"
- # split first: a filter graph may consume each input pad only once, and the
- # background is needed both before the jingle and after it.
- FC="$FC[1:v]split=2[bgA][bgB];"
- FC="$FC[bgA]trim=${TBODY}:${JAT},setpts=PTS-STARTPTS,${PADV_DIM}[tail1];"
- FC="$FC[2:v]${PADV},setpts=PTS-STARTPTS[jv];"
- FC="$FC[bgB]trim=${JEND}:${END},setpts=PTS-STARTPTS,${PADV_DIM}[tail2];"
- FC="$FC[bodyv][tail1][jv][tail2]concat=n=4:v=1[v]"
-else
- FC="$FC[1:v]trim=${TBODY}:${END},setpts=PTS-STARTPTS,${PADV_DIM}[tailv];"
- FC="$FC[bodyv][tailv]concat=n=2:v=1[v]"
-fi
-
-ffmpeg -nostdin -v warning -y \
- -i "$WORK/body.mp4" \
- -i "$WORK/bg.mp4" \
- -i "$JINGLE" \
- $IN_SFX \
- $CHOP_IN \
- -filter_complex "$FC" \
- -map "[v]" -map "[a]" -t "$END" \
- -c:v libx264 -preset medium -crf 18 -pix_fmt yuv420p -video_track_timescale 30000 \
- -c:a aac -b:a 192k -ar 48000 -ac 2 -movflags +faststart "$OUT"
-
-echo "PK9_DONE -> $OUT"
-ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 "$OUT"
diff --git a/umtool/song/pkmn-rebuild.sh b/umtool/song/pkmn-rebuild.sh
@@ -1,123 +0,0 @@
-#!/bin/sh
-# Pokemon FR/LG wild battle, rebuilt on the 3,959-clip palette over real
-# gameplay -- the most out-of-date deliverable in the set (its only build was
-# made at ~1,050 clips, before the window and renderer fixes).
-#
-# The background is FCPlaythroughs' Pokemon Red longplay. Timings were derived,
-# not guessed, and all three sources were tied to ONE clock first:
-#
-# * pkmn-ref.mp4 (a combined A+V section) starts at source 620.000s exactly --
-# established by cross-correlating its audio against the complete audio
-# track, which matched to the sample.
-# * pkmn-long-v.webm (the high-quality video) starts at source 644.134s --
-# established by matching the 1100ms black run, which sits at 36.267s in the
-# reference and 12.133s here.
-# * the audio is cut from the complete track at 644.134s to match.
-#
-# That mattered: a first attempt cut the audio at the REQUESTED start rather
-# than the actual one, and the two were ~14s apart, so no audio event lined up
-# with any picture event.
-#
-# The splice is on the AUDIO, not the black frames. The encounter's black run
-# ends at 13.233s, but the game's own rapid-notes alarm keeps playing to ~13.2s
-# and the battle theme's downbeat lands at 13.50s -- which is where the ums come
-# in, so the game plays its own intro and hands over on the beat.
-#
-# FLOURISH=drop the MIDI's own rapid intro is removed; the background's
-# alarm is the intro (default)
-# FLOURISH=keep the ums play the rapid intro themselves, and the game audio
-# ducks BEFORE the alarm so it is not doubled
-set -e
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-D=/home/user/reports/quartering-uh-song
-T=/home/user/.claude/jobs/efbe67a7/tmp
-
-BG_START=${BG_START:-5.866} # 10:50 in the source, in pkmn-long-v time
-SPLICE=${SPLICE:-13.500} # the battle theme's downbeat, same clock
-# Where the game's own ALARM starts, same clock. Read off the pitch contour: the
-# route music before it is B4/C#5/D5, and the alarm is the G5-F#5-F5-G6 figure --
-# audibly higher and quite different, which is how it is picked out. It runs
-# 2.6s, and the MIDI's own alarm is 33 notes over 2.5s, so the two agree.
-ALARM_AT=${ALARM_AT:-10.966}
-FLOURISH=${FLOURISH:-keep}
-# 0 = CUT the game's own sound where the ums take over, rather than leaving a
-# bed under them. The battle music would otherwise play its version of the same
-# tune underneath ours.
-DUCK=${DUCK:-0.0}
-PLANF=${PLANF:-pkmn-plan-v2.json}
-OUT=${OUT:-$D/quartering-pokemon-v2.mp4}
-mkdir -p "$(dirname "$OUT")"
-
-# The MIDI's own alarm is notes 0..32 of the lead (t=0 -> 2.500s, 78ms apart);
-# the tune proper starts at 2.969s. Those are the notes FLOURISH=drop removes.
-CUTAT=${CUTAT:-2.95}
-SCALE=${SCALE:-1.15}
-if [ "$FLOURISH" = "keep" ]; then CUTAT=-1; HANDOVER=$ALARM_AT; else HANDOVER=$SPLICE; fi
-
-# ---- arrange ----------------------------------------------------------------
-# poolFactor 3.0 on the bass. NOT 6.0: that was calibrated when the bass had 143
-# notes, and this MIDI's bass has 471 -- at 6.0 it claims 2,840 clips and STARVES
-# the melody (413 clips, p90 4.64). Measured knee: 3.0 gives melody 2.39 / bass 2.87.
-if [ "${REUSE_PLAN:-0}" != "1" ]; then
- # Voices from PkmRB-Battle1.mid -- Pokemon RED/BLUE, which is what the
- # background actually is. Its track names spell the credit (sequenced by
- # Joao Buaes), like the Mortal Kombat file, so tracks are picked by INDEX:
- # 2 -> melody 144 notes, IOI p50 0.469s, and it carries the alarm
- # 3 -> bass 471 notes, IOI p50 0.156s
- # 1 -> harm 516 notes, IOI p50 0.078s -- LEFT OUT. At 78ms every note is
- # shorter than the palette's 0.14s minimum clip, so it can only
- # come out as fragments; Metal Slug's rhythm voice at 0.183s
- # already reads as percussion rather than melody.
- export TEMPO_SCALE=$SCALE MAX_IOI=0.70 WINDOW=700 LOCAL=0 WOBBLE_W=0.08
- export VOICES='[{"name":"melody","lead":"pkrb-lead.json","gain":1.0,"role":"full","unique":true,"reuseGap":20,"contourMax":14,"fill":true},{"name":"bass","lead":"pkrb-bass.json","gain":0.62,"role":"pip1","unique":true,"reuseGap":12,"contourMax":5,"poolFactor":3.0,"fill":false}]'
- node arrange-poly.mjs
- cp poly-plan.json "$PLANF.raw"
-fi
-
-# Drop the flourish if asked, then shift so the first surviving note lands on
-# the handover.
-node -e "
-const fs=require('fs');
-const p=JSON.parse(fs.readFileSync('$PLANF.raw','utf8'));
-const CUT=$CUTAT;
-let dropped=0;
-if(CUT>0) for(const v of p.voices){const n0=v.plan.length; v.plan=v.plan.filter(n=>n.slotStart>=CUT*$SCALE-1e-6); dropped+=n0-v.plan.length;}
-const first=Math.min(...p.voices.filter(v=>v.plan.length).map(v=>v.plan[0].slotStart));
-const SH=$HANDOVER-$BG_START-first;
-for(const v of p.voices) for(const n of v.plan) n.slotStart=+(n.slotStart+SH).toFixed(4);
-fs.writeFileSync('$PLANF',JSON.stringify(p,null,1));
-const f2=Math.min(...p.voices.filter(v=>v.plan.length).map(v=>v.plan[0].slotStart));
-const l2=Math.max(...p.voices.flatMap(v=>v.plan.map(n=>n.slotStart+n.slotDur)));
-console.log('flourish '+('$FLOURISH')+': dropped '+dropped+' notes; shifted '+SH.toFixed(3)+
- 's -> first note '+f2.toFixed(3)+'s, ends '+l2.toFixed(3)+'s');"
-[ "${ARRANGE_ONLY:-0}" = "1" ] && exit 0
-
-# ---- background -------------------------------------------------------------
-# 10:50 onward, as one file, so the renderer's clock and this script's agree.
-if [ ! -f "$T/pkmn-bg.mp4" ]; then
- ffmpeg -nostdin -v error -y -ss "$BG_START" -i intro/pkmn-bg-full.mkv \
- -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p -video_track_timescale 30000 \
- -c:a aac -b:a 192k -ar 48000 -ac 2 "$T/pkmn-bg.mp4"
-fi
-
-PLAN=$PLANF BG_VIDEO=$T/pkmn-bg.mp4 BG_FIT=pad LAYOUT=bg \
- node render-poly.mjs "$T/pkmn-body.mp4"
-
-# ---- the game's own sound under it ------------------------------------------
-# Full until the handover -- the transition and the alarm ARE the intro -- then
-# down under the ums.
-END=$(node -e "
-const {voices}=require('$PWD/$PLANF');
-console.log((Math.max(...voices.flatMap(v=>v.plan.map(n=>n.slotStart+n.slotDur)))+0.7).toFixed(2));")
-H=$(node -e "console.log(($HANDOVER-$BG_START).toFixed(3))")
-echo "game audio full to ${H}s, then ${DUCK}; output ends ${END}s"
-ffmpeg -nostdin -v error -y -i "$T/pkmn-body.mp4" -i "$T/pkmn-bg.mp4" \
- -filter_complex "
-[1:a]volume='if(lt(t,${H}),1.0,${DUCK})':eval=frame,atrim=0:${END},asetpts=PTS-STARTPTS[game];
-[0:a]atrim=0:${END},asetpts=PTS-STARTPTS[song];
-[song][game]amix=inputs=2:duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a]" \
- -map 0:v -map "[a]" -t "$END" -c:v copy -c:a aac -b:a 192k -ar 48000 -ac 2 \
- -movflags +faststart "$OUT"
-
-echo "PKMN_DONE -> $OUT"
-ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 "$OUT"
diff --git a/umtool/song/pkmn-video.sh b/umtool/song/pkmn-video.sh
@@ -1,122 +0,0 @@
-#!/usr/bin/env bash
-# THE Pokemon video. One script, one shape, both aspect ratios, both lengths.
-#
-# This exists because the shape kept drifting: a build would get the X strokes but not
-# the 2x2, or the 2x2 but not the alarm, or the vertical would get none of them because
-# it was composed before the treatment existed. The shape is not a set of optional
-# effects -- it is the video -- so it lives in one place and every cut runs the same
-# steps in the same order.
-#
-# ---------------------------------------------------------------------------
-# THE SHAPE
-#
-# melody the main panel (wide) / the primary box (vertical)
-# bass the first pip (wide) / secondary box 1 (vertical)
-# run the rapid third voice, which is NEVER a static panel:
-# melody OUT -> X strokes sweeping the screen
-# (vertical: inside the gameplay screen, not the whole frame)
-# melody IN -> its own container, and inside that container
-# notes <0.15s apart -> a 2x2 grid, one corner per note
-# slower notes -> one clip filling the container
-#
-# the intro alarm (bg 3.960 -> 7.374, which the ums play themselves)
-# -> one um per note snaking through whatever space is free:
-# the whole screen in wide, the two bands in vertical,
-# on a grid sized to travel the path about twice
-#
-# the catch jingle
-# -> plays in the RUN'S container with the run's own visuals, so
-# "PIKACHU was caught!" stays readable underneath
-#
-# ---------------------------------------------------------------------------
-# FORMAT=wide|vertical CUT=long|short
-#
-# Everything else is derived. Set REUSE=1 to keep an existing base and only redo the
-# visual layers, which is the cheap loop when the shape is being judged.
-set -euo pipefail
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-S=/home/user/Projects/yt-dlp-transcript-browser/umtool/song
-T=/home/user/.claude/jobs/efbe67a7/tmp
-V=/home/user/reports/quartering-uh-song/videos/pokemon
-
-FORMAT=${FORMAT:-wide}
-CUT=${CUT:-long}
-
-if [ "$CUT" = "short" ]; then
- BODY_PLAN=pkmn-short-body.json; RUN_PLAN=pkmn-short-run.json
- WORK=$T/pkshort; FILLER='1745-1782:3.0:17.5'
- RUNFX=$T/pkshort-run-fx.wav; TAG=short
- LAST=39.16; ENDS=49.53
- SOLO=pkshort
-else
- BODY_PLAN=pkmn-v12-body.json; RUN_PLAN=pkmn-v12-run.json
- WORK=$T/pk9; FILLER='1745-1782:3.0:17.5 2189-2223:3.0:25.0 3710-3740:1.0:24.0'
- RUNFX=$T/v12-run-fx.wav; TAG=long
- LAST=90.21; ENDS=100.60
- SOLO=pkmn
-fi
-ALARM_A=3.96; ALARM_B=7.374
-JINGLE_AT=$(node -e "console.log(String(($ENDS - 4.03).toFixed(4)))") # jingle onset in output time
-
-mkdir -p "$V/plan" "$V/variants"
-PFX=$T/pkv-$TAG-$FORMAT # working prefix (B is taken: bars-snake reads B as its window end)
-
-# ---- 1. the audio: lift the run where the melody rests ----------------------
-# The run exists to carry those rests, and it was mixed at one level throughout.
-LIFT=${LIFT:-5} RAMP=0.25 GATE_PLAN=$BODY_PLAN GATE_VOICE=melody \
- node "$S/rest-lift.mjs" "$RUNFX" "$PFX-runfx.wav" | tail -2
-
-# ---- 2. the base ------------------------------------------------------------
-# GAME_BED=0.30: the game holds a bed across the alarm and fades to nothing BY the
-# downbeat, so the ums' alarm rises out of the game's own instead of replacing it
-# between one sample and the next. JINGLE_VIDEO is off -- the jingle gets the run's
-# visuals in step 4 instead of panels over the payoff text.
-if [ "${REUSE:-0}" = "1" ] && [ -f "$PFX-base.mp4" ]; then
- echo "REUSE=1: keeping $PFX-base.mp4"
-elif [ "$FORMAT" = "wide" ]; then
- WORK=$WORK REUSE_BG=1 REUSE_BODY=1 GAME_BED=0.30 PLANF=$BODY_PLAN \
- JINGLE=$T/pk9/jingle-finale.mp4 CHOP="$PFX-runfx.wav" CHOP_GAIN=0.55 \
- OUT="$PFX-base.mp4" FILLER="$FILLER" bash "$S/pk3.sh" | tail -2
-else
- # the vertical composes from the WIDE cut: it supplies the audio and the gimmick
- # picture, and the voice boxes come from the solo renders.
- WIDE=$V/$( [ "$CUT" = short ] && echo wide-short || echo wide ).mp4
- [ -f "$WIDE" ] || { echo "need $WIDE first -- build FORMAT=wide CUT=$CUT"; exit 2; }
- COMPOSED=$WIDE GAMEPLAY=$WORK/bg.mp4 \
- PRIMARY=$T/solo-$SOLO-melody.mp4 SEC1=$T/solo-$SOLO-bass.mp4 SEC2= CENTRE_LONE=0 \
- OFFSET=0 GIMMICKS="0-$ALARM_B,$LAST-$ENDS" \
- node "$S/shorts-compose.mjs" "$PFX-base.mp4" | tail -1
-fi
-
-# ---- 3/4/5. the shape -------------------------------------------------------
-if [ "$FORMAT" = "wide" ]; then
- BOXARG=384:216:870:478 ; FW=1280 ; FH=720
- STROKE_BOX="" ; STROKE_TILE=420
- SNAKE_BANDS="0:720" ; SNAKE_W=1280
-else
- BOXARG=480:270:552:1537 ; FW=1080 ; FH=1920
- STROKE_BOX=1080:972:0:541 ; STROKE_TILE=300
- SNAKE_BANDS="0:541,1513:1920" ; SNAKE_W=1080
-fi
-
-echo "--- the run, in its container"
-MODE=auto BOX=$BOXARG W=$FW H=$FH GATE_PLAN=$BODY_PLAN GATE_VOICE=melody \
- node "$S/run-line.mjs" "$RUN_PLAN" run "$PFX-base.mp4" "$PFX-run.mp4" 2>&1 | grep -vE "deprecated|Last message" | tail -2
-
-echo "--- the jingle, same container, same visuals"
-MODE=auto BOX=$BOXARG W=$FW H=$FH OFFSET=-$JINGLE_AT \
- node "$S/run-line.mjs" pkc-jingle-merged.json jingle "$PFX-run.mp4" "$PFX-jing.mp4" 2>&1 | grep -vE "deprecated|Last message" | tail -2
-
-echo "--- the X strokes, where the melody is out"
-BOX=$STROKE_BOX W=$FW H=$FH TILE_W=$STROKE_TILE MIN_RUN=4 TRAIL=0.30 EXTENT=0.82 MARGIN=8 \
- GATE_PLAN=$BODY_PLAN GATE_VOICE=melody \
- node "$S/climb-panel.mjs" "$RUN_PLAN" run "$PFX-jing.mp4" "$PFX-str.mp4" 2>&1 | grep -vE "deprecated|Last message" | tail -2
-
-echo "--- the intro alarm, snaking, about two loops"
-OUT=$V/$( [ "$FORMAT" = wide ] && echo wide || echo vertical )$( [ "$CUT" = short ] && echo -short || echo "" ).mp4
-A=$ALARM_A B=$ALARM_B W=$SNAKE_W PASSES=2 BANDS="$SNAKE_BANDS" \
- node "$S/bars-snake.mjs" "$BODY_PLAN" melody "$PFX-str.mp4" "$OUT" 2>&1 | grep -vE "deprecated|Last message" | tail -3
-
-cp "$BODY_PLAN" "$RUN_PLAN" "$V/plan/" 2>/dev/null || true
-echo "PKMN_${FORMAT}_${CUT}_DONE -> $OUT"
-ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 "$OUT"
diff --git a/umtool/song/rpg-remake-v5.sh b/umtool/song/rpg-remake-v5.sh
@@ -1,277 +0,0 @@
-#!/usr/bin/env bash
-# Super Mario RPG, rebuilt on the 3,959-clip palette, with the two changes asked
-# for after v4:
-#
-# DRUM_GAIN the snare roll and the drum hit are lifted harder. They are very
-# quiet in the source (peaks ~500-1600 against the transition's
-# ~6000), and at 1.7 the hit still sits under the ums that follow.
-# A BUMP, not a doubling -- 2.1 and 2.4 are the values to compare.
-#
-# UM_DELAY hold the ums back by a fraction of a second. The body is cut at
-# -ss LEAD (1.9556s, the second slot) so the first um lands at
-# final t=0; cutting at LEAD-UM_DELAY instead gives UM_DELAY
-# seconds of background before the first um WITH THE TUNE ITSELF
-# UNTOUCHED. The ending is pinned by BG_END, which is not moved, so
-# the background's wipe to black still lands on the last note --
-# the video simply becomes UM_DELAY longer at the front.
-#
-# Everything else is v4's, including the two values that took several passes to
-# get right: INTRO_END 10.400 (keeps the drum hit AND its decay; 10.100 cut one
-# frame in front of it and kept the snare's tail instead) and BG_FIT=pad (the
-# background is 960x720, and cropping it to 16:9 reads as the picture ZOOMING IN
-# the moment the song starts).
-set -euo pipefail
-# The DATA dir, not the script's own -- arrange-poly writes poly-plan.json into
-# SONG_DATA, and this script copies it from the working directory. Running the repo
-# copy with `cd $(dirname $0)` landed in the repo, where that file does not exist, and
-# the rebuild died on `cp: cannot stat 'poly-plan.json'` AFTER arranging successfully.
-# yoshi-rebuild.sh and pk3.sh both use the absolute path; this one was the exception.
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-D=/home/user/reports/quartering-uh-song
-T=/home/user/.claude/jobs/efbe67a7/tmp
-
-INTRO_END=${INTRO_END:-10.400}
-DRUM_FROM=${DRUM_FROM:-8.60}
-DRUM_GAIN=${DRUM_GAIN:-2.1}
-# UM_DELAY=auto DERIVES the spacing instead of guessing it.
-#
-# The arrangement's opening note is a lone bass B1 at 1.4666s standing in for the
-# drum hit the theme starts on, and the real downbeat is at 1.9556s -- so the
-# composer's own spacing between the kick and the first melody note is 0.4890s.
-# The intro carries the game's REAL drum hit, whose low-band onset is at 10.115s
-# (measured: the band is silent at 10.110 and rises 1214 -> 2183 -> 3599 over the
-# next three 5ms frames, peaking 4337 at 10.150). Lining the real hit up with the
-# note that stood in for it puts the first um at 10.115 + 0.489 = 10.604s, and
-# the body starts at INTRO_END, so the delay is 10.604 - INTRO_END = 0.204s.
-KICK_AT=${KICK_AT:-10.115}
-UM_DELAY=${UM_DELAY:-auto}
-# UM_FADE fades the first um IN, rather than giving it more room.
-#
-# The body is cut at EXACTLY the slot start with no padding, so the first um's
-# attack lands on the body's very first sample -- a hard splice out of the
-# intro's decay straight onto a note onset. That can read as jarring rather than
-# as early, which is a different complaint with a different fix, so it is a
-# separate knob: UM_DELAY buys space, UM_FADE softens the entry. The body's audio
-# is the ums ALONE (the background supplies picture only), so this touches
-# nothing else.
-UM_FADE=${UM_FADE:-0}
-XFADE=${XFADE:-0}
-# INTRO_GAIN lifts the WHOLE intro, in dB, to meet the body.
-#
-# The seam was described as "stark", and it is a LEVEL STEP rather than a
-# transition problem. Measured on rpg-A-cut with ebur128: the intro runs
-# -22.0 LUFS into the seam and the body arrives at -17.0 LUFS. Five LU is a
-# plainly audible jump, and no crossfade can fix it -- a fade only smears the
-# step over its duration. rpg-D-xfade030 is actually WORSE by this measure: the
-# fade pulls the intro's last two seconds to -30.2 LUFS, so the song arrives
-# against a 13 LU contrast instead of 5.
-#
-# Headroom for it: true peak into the seam is -8.1 dBFS, so +5 dB lands at
-# -3.1 dBFS with the lifted drum hit still intact.
-INTRO_GAIN=${INTRO_GAIN:-0}
-# Separate the um kick from the game's kick IN TIME.
-#
-# At offset 0 the two land on the same instant in the same low band, and the
-# louder simply masks the other -- which is why a variant built to have two
-# kicks was heard as having one. Around 0.1s reads as two distinct hits; around
-# 0.03s reads as a flam (one thickened hit). Note the intro is hard-cut at
-# INTRO_END, so a large offset leaves the um kick very little room.
-KICK_UM_OFFSET=${KICK_UM_OFFSET:-0}
-BG_END=${BG_END:-65.583}
-# The body render, named after the build so a short cannot clobber the long.
-BODY=${BODY:-$T/r5-body.mp4}
-PLANF=${PLANF:-rpg-plan-v4.json}
-OUT=${OUT:-$D/quartering-mario-rpg-v5.mp4}
-FADE_ST=$(node -e "console.log(($INTRO_END - 0.06).toFixed(3))")
-mkdir -p "$(dirname "$OUT")"
-
-# ---- arrange ----------------------------------------------------------------
-# poolFactor 6.0 is free here and nowhere else: the melody needs only 210 of
-# ~2,800 clips, so tripling and doubling the bass pool costs it nothing
-# (p90 0.39 -> 0.41) while the bass median more than halves (0.52 -> 0.21).
-if [ "${REUSE_PLAN:-0}" != "1" ]; then
- export TEMPO_SCALE=1.10 MAX_IOI=0.70 WINDOW=700 LOCAL=0 TIME_LIMIT=${TL:-999}
- # VOICES is overridable so a variant can add the third voice (Brass B) or point at
- # held-last-note lead files, without a second copy of this script.
- export VOICES=${VOICES:-'[{"name":"melody","lead":"rpg-lead.json","gain":1.0,"role":"full","unique":true,"reuseGap":20,"contourMax":12,"fill":true},{"name":"bass","lead":"rpg-bass.json","gain":0.68,"role":"pip1","unique":true,"reuseGap":12,"contourMax":4,"poolFactor":6.0,"fill":false}]'}
- node arrange-poly.mjs
- cp poly-plan.json "$PLANF"
-fi
-[ "${ARRANGE_ONLY:-0}" = "1" ] && exit 0
-
-# ---- drop the stand-in kick note --------------------------------------------
-# v4 got rid of it by CUTTING at the second slot, which worked only because the
-# cut was exactly there. UM_DELAY moves the cut EARLIER, straight back over that
-# note -- so it has to leave the plan, or a truncated low blip returns in front
-# of the tune and the intro's real drum hit is answered by a fake one.
-LEAD=$(node -e "
-const {voices}=require('$PWD/$PLANF');
-const t=[...new Set(voices.flatMap(v=>v.plan.map(n=>n.slotStart)))].sort((a,b)=>a-b);
-console.log((t[1] ?? t[0]).toFixed(4));")
-TRIMF="${PLANF%.json}-trim.json"
-node -e "
-const fs=require('fs');
-const p=JSON.parse(fs.readFileSync('$PLANF','utf8'));
-let dropped=0;
-for(const v of p.voices){const n0=v.plan.length; v.plan=v.plan.filter(n=>n.slotStart>=$LEAD-1e-6); dropped+=n0-v.plan.length;}
-fs.writeFileSync('$TRIMF',JSON.stringify(p,null,1));
-console.log('dropped '+dropped+' note(s) before the downbeat at ${LEAD}s -> $TRIMF');"
-
-# ---- how far to hold the ums back -------------------------------------------
-if [ "$UM_DELAY" = "auto" ]; then
- UM_DELAY=$(node -e "
- const {voices}=require('$PWD/$PLANF');
- const t=[...new Set(voices.flatMap(v=>v.plan.map(n=>n.slotStart)))].sort((a,b)=>a-b);
- const gap=(t[1]??t[0])-t[0]; // the MIDI's own kick -> downbeat spacing
- console.log(Math.max(0, $KICK_AT + gap - $INTRO_END).toFixed(4));")
- echo "UM_DELAY=auto -> ${UM_DELAY}s (real kick at ${KICK_AT}s + the MIDI's own kick-to-downbeat spacing)"
-fi
-
-# ---- body -------------------------------------------------------------------
-# REUSE_BODY=1 skips the ~20-minute render when only the trim or the intro has
-# changed. The body does not depend on either -- they are concatenated, not
-# mixed -- so every UM_DELAY past the first is nearly free.
-if [ "${REUSE_BODY:-0}" = "1" ] && [ -f "$BODY" ]; then
- echo "reusing $BODY (REUSE_BODY=1)"
-else
- # OUT_END=0: do NOT trim the render to the audio's end. The ending here is the
- # background's own wipe to black at 65.583s, and the last note is at 64.53s --
- # trimming to the audio throws the ending away (7 trailing black frames where
- # v4 had 32). BG_END does the cutting instead, below.
- OUT_END=0 PLAN=$TRIMF BG_VIDEO=$PWD/${BG:-intro/smrpg-bg.mp4} BG_FIT=pad LAYOUT=bg \
- node render-poly.mjs "$BODY" | tail -3
-fi
-
-CUT=$(node -e "console.log(Math.max(0, $LEAD - $UM_DELAY).toFixed(4))")
-echo "lead-in ${LEAD}s, holding the ums back ${UM_DELAY}s -> cutting the body at ${CUT}s, ending at ${BG_END}s"
-# String(), not a bare number -- node colourises numeric console.log on a TTY,
-# so this test would compare against an ANSI-wrapped "1" and never fire.
-if [ "$(node -e "console.log(String($UM_FADE>0?1:0))")" = "1" ]; then
- # Start the fade AT the um, not at the body's first sample. UM_DELAY puts
- # that many seconds of silence in front, so a fade shorter than the delay
- # fades nothing at all -- measured: at UM_DELAY 0.204 a 0.18s fade produced a
- # byte-identical file.
- AF="-af afade=t=in:st=${UM_DELAY}:d=${UM_FADE}"
- echo " fading the first um in over ${UM_FADE}s, from ${UM_DELAY}s"
-else
- AF=""
-fi
-# shellcheck disable=SC2086
-ffmpeg -nostdin -v error -y -ss "$CUT" -to "$BG_END" -i "$BODY" $AF \
- -c:v libx264 -preset medium -crf 21 -pix_fmt yuv420p -video_track_timescale 30000 \
- -c:a aac -b:a 192k -ar 48000 -ac 2 "$T/r5-bd.mp4"
-
-# ---- the stand-in kick, as an um, ON the real one ---------------------------
-# KICK_UM=1 plays the dropped note after all -- but placed on the BACKGROUND's
-# own kick rather than after it, so the two land together.
-#
-# It cannot simply be left in the body: the game's kick is at 10.115s and the
-# body does not start until INTRO_END (10.400s), so a note carried in the body
-# can only ever arrive 0.285s LATE. Mixing it onto the intro at the kick's own
-# time is the only way to make them coincide. Its audio comes from a one-note
-# render of the notes the trim dropped, so it is the same clip the arranger
-# chose, at the same gain.
-KICK_MIX=""
-if [ "${KICK_UM:-0}" = "1" ]; then
- KICKF="${PLANF%.json}-kick.json"
- if [ ! -f "$T/r5-kick.wav" ]; then
- # Drop the EMPTY voices and make what is left the `full` one. The stand-in
- # kick is a BASS note, so a straight filter leaves the melody with no notes
- # at all -- and the renderer looks up role:"full" to build the main panel,
- # which then has nothing to build from.
- node -e "
- const fs=require('fs');
- const p=JSON.parse(fs.readFileSync('$PLANF','utf8'));
- for(const v of p.voices) v.plan=v.plan.filter(n=>n.slotStart<$LEAD-1e-6);
- p.voices=p.voices.filter(v=>v.plan.length);
- if(!p.voices.length) throw new Error('nothing before the downbeat to extract');
- p.voices[0].role='full';
- p.voices=[p.voices[0]];
- fs.writeFileSync('$KICKF',JSON.stringify(p,null,1));
- console.log('kick-only plan: '+p.voices.map(v=>v.name+' '+v.plan.length).join(', '));"
- PLAN=$KICKF BG_VIDEO=$PWD/intro/smrpg-bg.mp4 BG_FIT=pad LAYOUT=bg \
- node render-poly.mjs "$T/r5-kickonly.mp4" >/dev/null 2>&1
- KSTART=$(node -e "
- const {voices}=require('$PWD/$KICKF');
- console.log(Math.min(...voices.filter(v=>v.plan.length).map(v=>v.plan[0].slotStart)).toFixed(4));")
- ffmpeg -nostdin -v error -y -ss "$KSTART" -t 0.60 -i "$T/r5-kickonly.mp4" -ac 1 -ar 48000 "$T/r5-kick.wav"
- echo " extracted the stand-in kick um from ${KSTART}s"
- fi
- KDELAY=$(node -e "console.log(String(Math.round(($KICK_AT+$KICK_UM_OFFSET)*1000)))")
- # The um kick is a 0.60s clip laid at 10.115s, but the intro is cut at
- # INTRO_END (10.400s) -- so it was being CHOPPED 285ms in, with no fade, at
- # exactly the seam. That truncation is its own contribution to the stark
- # transition. Fade it out on the same schedule as the intro instead.
- KICK_MIX="[1:a]aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo,volume=${KICK_UM_GAIN:-1.0},adelay=delays=${KDELAY}:all=1,afade=t=out:st=${FADE_ST}:d=0.06[k];[main][k]amix=inputs=2:duration=first:normalize=0[aout]"
- echo " KICK_UM=1: the um kick is laid on the real kick at ${KICK_AT}s"
-fi
-
-# ---- intro: gameplay, transition, black, silence, SNARE ROLL, DRUM HIT -------
-AFILT="volume='if(lt(t,${DRUM_FROM}),1.0,${DRUM_GAIN})':eval=frame,volume=${INTRO_GAIN}dB,afade=t=out:st=${FADE_ST}:d=0.06"
-if [ -n "$KICK_MIX" ]; then
- ffmpeg -nostdin -v error -y -t "$INTRO_END" -i intro/smrpg-intro.mp4 -i "$T/r5-kick.wav" \
- -filter_complex "[0:v]scale=1280:720:force_original_aspect_ratio=decrease,pad=1280:720:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=30[vout];[0:a]${AFILT}[main];${KICK_MIX}" \
- -map "[vout]" -map "[aout]" -t "$INTRO_END" \
- -c:v libx264 -preset medium -crf 21 -pix_fmt yuv420p -video_track_timescale 30000 \
- -c:a aac -b:a 192k -ar 48000 -ac 2 "$T/r5-intro.mp4"
-else
- ffmpeg -nostdin -v error -y -t "$INTRO_END" -i intro/smrpg-intro.mp4 \
- -vf "scale=1280:720:force_original_aspect_ratio=decrease,pad=1280:720:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=30" \
- -af "$AFILT" \
- -c:v libx264 -preset medium -crf 21 -pix_fmt yuv420p -video_track_timescale 30000 \
- -c:a aac -b:a 192k -ar 48000 -ac 2 "$T/r5-intro.mp4"
-fi
-
-# The peak of the lifted section, BEFORE the concat -- a gain that clips the
-# drum hit is worse than one that leaves it quiet.
-node -e "
-const {execFileSync}=require('child_process');
-const b=execFileSync('ffmpeg',['-v','error','-ss','${DRUM_FROM}','-i','$T/r5-intro.mp4','-ac','1','-ar','48000','-f','s16le','-'],{maxBuffer:1<<28});
-const a=new Int16Array(b.buffer,b.byteOffset,b.length/2);
-let pk=0,s=0; for(let i=0;i<a.length;i++){const v=Math.abs(a[i]); if(v>pk)pk=v; s+=a[i]*a[i];}
-console.log(' intro after ${DRUM_FROM}s at gain ${DRUM_GAIN}: peak '+pk+'/32768 ('+(pk/32768*100).toFixed(1)+'%), rms '+Math.sqrt(s/a.length).toFixed(0)+(pk>=32700?' ** CLIPPING **':''));"
-
-# ---- assemble ---------------------------------------------------------------
-# XFADE>0 dissolves the intro into the body instead of cutting, in picture AND
-# sound, so the change from the game's own gameplay+audio to the background+ums
-# is a transition rather than a splice. It costs XFADE seconds of overlap, so the
-# ums arrive that much sooner -- which is why it is a variant and not a default.
-# XFADE_AUDIO crossfades the SOUND only and leaves the picture a hard cut.
-#
-# Judged best: "xfade30 audio but with f-level distance from the kick and static
-# visuals". The picture dissolve was never what the seam needed -- the seam was a LEVEL
-# step, which INTRO_GAIN fixes -- but a short audio crossfade still softens the arrival
-# of the first um without smearing the join visually.
-#
-# An audio crossfade SHORTENS the result by XFADE while a video concat does not, so the
-# picture has to lose the same amount or everything after the join drifts. It comes off
-# the INTRO'S TAIL, which is the static battle scene after the drum hit's decay -- the
-# one stretch where 0.3s of picture is invisible -- and it puts the cut exactly where
-# the body's sound starts fading in.
-if [ "$(node -e "console.log(String(${XFADE_AUDIO:-0}>0?1:0))")" = "1" ]; then
- ITRIM=$(node -e "console.log(($INTRO_END - $XFADE_AUDIO).toFixed(3))")
- echo " audio-only crossfade ${XFADE_AUDIO}s; picture cuts hard at ${ITRIM}s (intro tail trimmed to match)"
- ffmpeg -nostdin -v error -y -i "$T/r5-intro.mp4" -i "$T/r5-bd.mp4" \
- -filter_complex "[0:v]trim=0:${ITRIM},setpts=PTS-STARTPTS[iv];[1:v]setpts=PTS-STARTPTS[bv];\
-[iv][bv]concat=n=2:v=1[v];[0:a][1:a]acrossfade=d=${XFADE_AUDIO}:c1=tri:c2=tri[a]" \
- -map "[v]" -map "[a]" \
- -c:v libx264 -preset medium -crf 21 -pix_fmt yuv420p -c:a aac -b:a 192k -ar 48000 -ac 2 \
- -movflags +faststart "$OUT"
-elif [ "$(node -e "console.log(String($XFADE>0?1:0))")" = "1" ]; then
- IDUR=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$T/r5-intro.mp4")
- OFF=$(node -e "console.log(Math.max(0,$IDUR-$XFADE).toFixed(3))")
- echo " crossfading intro -> body over ${XFADE}s (offset ${OFF}s)"
- ffmpeg -nostdin -v error -y -i "$T/r5-intro.mp4" -i "$T/r5-bd.mp4" \
- -filter_complex "[0:v][1:v]xfade=transition=fade:duration=${XFADE}:offset=${OFF}[v];[0:a][1:a]acrossfade=d=${XFADE}:c1=tri:c2=tri[a]" \
- -map "[v]" -map "[a]" \
- -c:v libx264 -preset medium -crf 21 -pix_fmt yuv420p -c:a aac -b:a 192k -ar 48000 -ac 2 \
- -movflags +faststart "$OUT"
-else
- ffmpeg -nostdin -v error -y -i "$T/r5-intro.mp4" -i "$T/r5-bd.mp4" \
- -filter_complex "[0:v][0:a][1:v][1:a]concat=n=2:v=1:a=1[v][a]" -map "[v]" -map "[a]" \
- -c:v libx264 -preset medium -crf 21 -pix_fmt yuv420p -c:a aac -b:a 192k -ar 48000 -ac 2 \
- -movflags +faststart "$OUT"
-fi
-
-echo "RPG5_DONE -> $OUT"
-ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 "$OUT"
diff --git a/umtool/song/rpg-short.sh b/umtool/song/rpg-short.sh
@@ -1,152 +0,0 @@
-#!/usr/bin/env bash
-# Super Mario RPG, cut at the first repeat, ending on the transition INTO the next
-# battle.
-#
-# The tune is a literal exact double -- find-repeats scores 100% at 28.48s -- so half
-# of the v6 build is the same 28.5 seconds played twice. TL=28.48 is therefore not a
-# trim but the whole tune, once.
-#
-# That leaves the ending to be rebuilt, because v6's ending is the background's own
-# wipe at bg 64.267 and the short song is over at 31.77. The background's black runs
-# are what make the splice free:
-#
-# 0.000 - 0.850 opening black
-# 0.850 - 27.617 THE BATTLE <- the whole short song plays over this
-# 27.617 - 29.000 black, the battle ending
-# ...
-# 57.033 - 58.367 black
-# 58.367 - 64.267 overworld: Mario walks into an enemy
-# 64.267 - 65.617 THE WIPE INTO THE NEXT BATTLE, and its sting
-#
-# Cut segment A inside the first black run and start segment B inside the second, and
-# the seam is black-to-black -- invisible, rather than a jump cut between two scenes.
-# What the viewer sees is one continuous piece: the battle ends, Mario walks, he hits
-# an enemy, the screen wipes.
-#
-# The sting is bg 63.90 -> 64.78 (measured: RMS 1153 -> 2095 -> 2673 -> 2961 across
-# 63.85-64.00), followed by TRUE DIGITAL SILENCE 64.80 - 65.45 over the black. So the
-# tail ends at 65.48 -- inside that silence, before the next battle's audio starts at
-# 65.50 -- and the video simply stops in the black.
-set -euo pipefail
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-D=/home/user/reports/quartering-uh-song
-T=/home/user/.claude/jobs/efbe67a7/tmp
-S=/home/user/Projects/yt-dlp-transcript-browser/umtool/song
-
-A_END=${A_END:-27.90} # inside the black run 27.617-29.000
-B_START=${B_START:-58.10} # inside the black run 57.033-58.367
-B_END=${B_END:-65.48} # inside the digital silence after the sting
-SFX_AT=${SFX_AT:-63.88} # the sting's onset in the SOURCE
-SFX_GAIN=${SFX_GAIN:-0.9}
-OUT=${OUT:-$D/quartering-mario-rpg-short.mp4}
-PLANF=${PLANF:-rpg-plan-short.json}
-BODY=${BODY:-$T/rpg-short-body.mp4}
-BASE=${BASE:-$T/rpg-short-base.mp4}
-# BLEND lays the slightest bit of the GAME'S OWN next note under our first um.
-#
-# Measured on intro/smrpg-intro.mp4: the drum kick is at 10.115, the theme's next beat
-# at 10.570 (a low hit, energy 19 -> 32 dB) and its melodic content at 10.630 (the high
-# band takes 60-65%, peak ~1.2kHz). Our first um lands at 10.604 -- between the two --
-# because UM_DELAY is derived from the MIDI's own kick-to-downbeat spacing, so the two
-# were already in agreement and nothing of the game was audible only because the intro
-# is hard-cut at INTRO_END.
-#
-# So the blend is not a retime: it is letting a little of what was already there through.
-# BLEND_GAIN is deliberately low and BLEND_FADE short -- "mostly the um with just the
-# slightest bit of the beginning of the note".
-BLEND=${BLEND:-0}
-BLEND_FROM=${BLEND_FROM:-10.400}
-BLEND_LEN=${BLEND_LEN:-0.42}
-BLEND_GAIN=${BLEND_GAIN:-0.30}
-BLEND_FADE=${BLEND_FADE:-0.26}
-
-BG=intro/smrpg-bg-short.mp4
-SPLICED=$(node -e "console.log(($A_END + $B_END - $B_START).toFixed(3))")
-STING_IN_BG=$(node -e "console.log(($A_END + $SFX_AT - $B_START).toFixed(3))")
-SFX_LEN=$(node -e "console.log(($B_END - $SFX_AT).toFixed(3))")
-echo "spliced background ${SPLICED}s; the sting lands at ${STING_IN_BG}s, ${SFX_LEN}s long"
-
-# ---- 1. the spliced background ----------------------------------------------
-# Both halves re-encoded with identical parameters so the concat demuxer can join
-# them without a re-wrap surprise. 960x720 and 60fps are the source's own; the
-# renderer pads to 1280x720 itself (BG_FIT=pad -- cropping this 4:3 capture reads
-# as the picture zooming in, see mario-rpg.md).
-if [ ! -f "$BG" ]; then
- for seg in "a:0:$A_END" "b:$B_START:$(node -e "console.log(($B_END-$B_START).toFixed(3))")"; do
- IFS=: read -r n ss dur <<EOF
-$seg
-EOF
- ffmpeg -nostdin -v error -y -ss "$ss" -i intro/smrpg-bg.mp4 -t "$dur" -an \
- -c:v libx264 -preset medium -crf 18 -pix_fmt yuv420p -video_track_timescale 60000 \
- "$T/rpgshort-$n.mp4"
- done
- printf "file '%s'\nfile '%s'\n" "$T/rpgshort-a.mp4" "$T/rpgshort-b.mp4" > "$T/rpgshort-concat.txt"
- ffmpeg -nostdin -v error -y -f concat -safe 0 -i "$T/rpgshort-concat.txt" -c copy "$BG"
- echo " built $BG ($(ffprobe -v error -show_entries format=duration -of csv=p=0 "$BG")s)"
-fi
-
-# ---- 2. the sting, and a silent spacer --------------------------------------
-# The spacer is what makes the picture reach the end of the background. render-poly
-# sizes its master buffer from max(last slot, last overlay) + 1s, and the picture
-# stops where that buffer does -- which is how the Pokemon v9 body came out 7.4s
-# short with a container duration that still read 100.5s. A silent overlay ending at
-# the background's end is the smallest thing that holds the picture open.
-#
-# The sting itself is NOT an overlay: mixed into the master it could only be
-# re-levelled by re-rendering, and levels here are judged by ear. Muxed onto the
-# finished video with -c:v copy instead, a new balance costs seconds.
-mkdir -p sfx
-[ -f sfx/rpg-transition.wav ] || ffmpeg -nostdin -v error -y -ss "$SFX_AT" -i intro/smrpg-bg.mp4 \
- -t "$SFX_LEN" -vn -ar 48000 -ac 2 sfx/rpg-transition.wav
-[ -f sfx/rpg-spacer.wav ] || ffmpeg -nostdin -v error -y -f lavfi -i anullsrc=r=48000:cl=stereo \
- -t "$SFX_LEN" sfx/rpg-spacer.wav
-node -e "
-const fs=require('fs');
-fs.writeFileSync('rpg-short-overlays.json', JSON.stringify(
- [{file:'sfx/rpg-spacer.wav', at:$STING_IN_BG, gain:0.0, label:'spacer: holds the picture open to the wipe'}], null, 1));
-console.log(' overlay spacer at ${STING_IN_BG}s');"
-
-# ---- 3. the body, over the spliced background -------------------------------
-# INTRO_GAIN=5 is the MEASURED fix for the seam, not a taste call: the intro runs
-# -22.0 LUFS into it and the body arrives at -17.0, and no crossfade can close a
-# level step. v6 (the long) was built without it, so short and long differ here.
-REUSE_PLAN=1 TL=28.48 PLANF="$PLANF" \
- BG="$BG" BG_END="$SPLICED" BODY="$BODY" \
- OVERLAYS=rpg-short-overlays.json INTRO_GAIN=${INTRO_GAIN:-5} \
- OUT="$BASE" bash "$S/rpg-remake-v5.sh"
-
-# ---- 4. the sting, onto the finished video ----------------------------------
-# Its time in the OUTPUT: the intro is concatenated in front and the body was cut at
-# CUT, so bg time t becomes 10.400 + t - CUT.
-CUT=$(node -e "
-const {voices}=require('$PWD/$PLANF');
-const t=[...new Set(voices.flatMap(v=>v.plan.map(n=>n.slotStart)))].sort((a,b)=>a-b);
-console.log(Math.max(0,(t[1]??t[0])-0.2040).toFixed(4));")
-AT=$(node -e "console.log((10.400 + $STING_IN_BG - $CUT).toFixed(3))")
-MS=$(node -e "console.log(String(Math.round((10.400 + $STING_IN_BG - $CUT)*1000)))")
-echo "muxing the sting at ${AT}s of the output (body cut at ${CUT}s)"
-
-# The blend rides in on the same mux. The intro occupies 0 -> INTRO_END of the output,
-# so a snippet cut from the intro asset at time X sits at output time X -- no offset
-# arithmetic, which is the one thing that could quietly put it in the wrong place.
-BLEND_IN=""
-BLEND_FC=""
-BLEND_MIX="[0:a][s]"
-NMIX=2
-if [ "$BLEND" = "1" ]; then
- [ -f sfx/rpg-firstnote.wav ] || ffmpeg -nostdin -v error -y -ss "$BLEND_FROM" -i intro/smrpg-intro.mp4 \
- -t "$BLEND_LEN" -vn -ar 48000 -ac 2 sfx/rpg-firstnote.wav
- BMS=$(node -e "console.log(String(Math.round($BLEND_FROM*1000)))")
- BLEND_IN="-i sfx/rpg-firstnote.wav"
- BLEND_FC="[2:a]volume=${BLEND_GAIN},aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo,afade=t=out:st=0:d=${BLEND_FADE},adelay=delays=${BMS}:all=1[b];"
- BLEND_MIX="[0:a][s][b]"
- NMIX=3
- echo " blending ${BLEND_LEN}s of the game's own next note at ${BLEND_FROM}s, gain ${BLEND_GAIN}, fading over ${BLEND_FADE}s"
-fi
-# shellcheck disable=SC2086
-ffmpeg -nostdin -v error -y -i "$BASE" -i sfx/rpg-transition.wav $BLEND_IN \
- -filter_complex "[1:a]volume=${SFX_GAIN},aformat=sample_fmts=fltp:sample_rates=48000:channel_layouts=stereo,adelay=delays=${MS}:all=1[s];${BLEND_FC}${BLEND_MIX}amix=inputs=${NMIX}:duration=first:normalize=0,alimiter=limit=0.98:level=disabled[a]" \
- -map 0:v -map "[a]" -c:v copy -c:a aac -b:a 192k -movflags +faststart "$OUT"
-
-echo "RPG_SHORT_DONE -> $OUT"
-ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 "$OUT"
diff --git a/umtool/song/spec.mjs b/umtool/song/spec.mjs
@@ -24,11 +24,13 @@
// keys are PRESERVED: this file is hand-edited, and silently dropping something
// somebody typed is worse than ignoring it.
//
-// VIDEO_ROOT overrides where the song directories live.
+// VIDEO_ROOT overrides where the song directories live (default
+// ~/reports/quartering-uh-song/videos).
import { readFileSync, writeFileSync, existsSync, readdirSync, statSync, renameSync, mkdirSync } from "node:fs";
+import os from "node:os";
import path from "node:path";
-const ROOT = process.env.VIDEO_ROOT ?? "/home/user/reports/quartering-uh-song/videos";
+const ROOT = process.env.VIDEO_ROOT ?? path.join(os.homedir(), "reports", "quartering-uh-song", "videos");
const argv = process.argv.slice(2);
const file = (song) => path.join(ROOT, song, "spec.json");
diff --git a/umtool/song/thumb-accepted.json b/umtool/song/thumb-accepted.json
@@ -2,8 +2,8 @@
"version": 1,
"thumbs": {
"yoshi": {
- "out": "/home/user/reports/quartering-uh-song/thumbs/yoshi-c.jpg",
- "bg": "/home/user/.claude/jobs/efbe67a7/tmp/song/intro/yoshi-bg.mp4",
+ "out": "thumbs/yoshi-c.jpg",
+ "bg": "intro/yoshi-bg.mp4",
"bgAt": 16.79,
"corners": [
{
@@ -53,8 +53,8 @@
]
},
"mario-rpg": {
- "out": "/home/user/reports/quartering-uh-song/thumbs/rpg-c.jpg",
- "bg": "/home/user/.claude/jobs/efbe67a7/tmp/song/intro/smrpg-bg.mp4",
+ "out": "thumbs/rpg-c.jpg",
+ "bg": "intro/smrpg-bg.mp4",
"bgAt": 63.36,
"corners": [
{
@@ -104,8 +104,8 @@
]
},
"metal-slug": {
- "out": "/home/user/reports/quartering-uh-song/thumbs/ms2-c.jpg",
- "bg": "/home/user/.claude/jobs/efbe67a7/tmp/song/intro/ms2-play-bg.mp4",
+ "out": "thumbs/ms2-c.jpg",
+ "bg": "intro/ms2-play-bg.mp4",
"bgAt": 82.3,
"corners": [
{
@@ -155,8 +155,8 @@
]
},
"mortal-kombat-fatality": {
- "out": "/home/user/reports/quartering-uh-song/thumbs/mk-c-fight.jpg",
- "bg": "/home/user/.claude/jobs/efbe67a7/tmp/song/intro/mk-bg-fatal3.mp4",
+ "out": "thumbs/mk-c-fight.jpg",
+ "bg": "intro/mk-bg-fatal3.mp4",
"bgAt": 53.66,
"corners": [
{
diff --git a/umtool/song/thumb-manifest.json b/umtool/song/thumb-manifest.json
@@ -2,8 +2,8 @@
"version": 1,
"thumbs": {
"mortal-kombat": {
- "out": "/home/user/reports/quartering-uh-song/thumbs/mortal-kombat.jpg",
- "bg": "/home/user/.claude/jobs/efbe67a7/tmp/song/intro/mk-bg.mp4",
+ "out": "thumbs/mortal-kombat.jpg",
+ "bg": "intro/mk-bg.mp4",
"bgAt": 134.4,
"corners": [
{
@@ -53,8 +53,8 @@
]
},
"mortal-kombat-fatality": {
- "out": "/home/user/reports/quartering-uh-song/thumbs/mk-c-fight.jpg",
- "bg": "/home/user/.claude/jobs/efbe67a7/tmp/song/intro/mk-bg-fatal3.mp4",
+ "out": "thumbs/mk-c-fight.jpg",
+ "bg": "intro/mk-bg-fatal3.mp4",
"bgAt": 53.66,
"corners": [
{
@@ -104,8 +104,8 @@
]
},
"yoshi": {
- "out": "/home/user/reports/quartering-uh-song/thumbs/yoshi-c.jpg",
- "bg": "/home/user/.claude/jobs/efbe67a7/tmp/song/intro/yoshi-bg.mp4",
+ "out": "thumbs/yoshi-c.jpg",
+ "bg": "intro/yoshi-bg.mp4",
"bgAt": 16.79,
"corners": [
{
@@ -155,8 +155,8 @@
]
},
"mario-rpg": {
- "out": "/home/user/reports/quartering-uh-song/thumbs/rpg-c.jpg",
- "bg": "/home/user/.claude/jobs/efbe67a7/tmp/song/intro/smrpg-bg.mp4",
+ "out": "thumbs/rpg-c.jpg",
+ "bg": "intro/smrpg-bg.mp4",
"bgAt": 63.36,
"corners": [
{
@@ -206,8 +206,8 @@
]
},
"metal-slug": {
- "out": "/home/user/reports/quartering-uh-song/thumbs/ms2-c.jpg",
- "bg": "/home/user/.claude/jobs/efbe67a7/tmp/song/intro/ms2-play-bg.mp4",
+ "out": "thumbs/ms2-c.jpg",
+ "bg": "intro/ms2-play-bg.mp4",
"bgAt": 82.3,
"corners": [
{
@@ -257,8 +257,8 @@
]
},
"pokemon-catch": {
- "out": "/home/user/reports/quartering-uh-song/thumbs/pokemon-a-catch.jpg",
- "bg": "/home/user/.claude/jobs/efbe67a7/tmp/song/intro/pkmn-bg-full.mkv",
+ "out": "thumbs/pokemon-a-catch.jpg",
+ "bg": "intro/pkmn-bg-full.mkv",
"bgAt": 10.22,
"corners": [
{
@@ -308,8 +308,8 @@
]
},
"pokemon-battle": {
- "out": "/home/user/reports/quartering-uh-song/thumbs/pokemon-b-battle.jpg",
- "bg": "/home/user/.claude/jobs/efbe67a7/tmp/song/intro/pkmn-bg-full.mkv",
+ "out": "thumbs/pokemon-b-battle.jpg",
+ "bg": "intro/pkmn-bg-full.mkv",
"bgAt": 43.43,
"corners": [
{
@@ -359,8 +359,8 @@
]
},
"pokemon-run": {
- "out": "/home/user/reports/quartering-uh-song/thumbs/pokemon-d-run.jpg",
- "bg": "/home/user/.claude/jobs/efbe67a7/tmp/song/intro/pkmn-bg-full.mkv",
+ "out": "thumbs/pokemon-d-run.jpg",
+ "bg": "intro/pkmn-bg-full.mkv",
"bgAt": 150.71,
"corners": [
{
@@ -410,8 +410,8 @@
]
},
"pokemon-battle2": {
- "out": "/home/user/reports/quartering-uh-song/thumbs/pokemon-c-battle2.jpg",
- "bg": "/home/user/.claude/jobs/efbe67a7/tmp/song/intro/pkmn-bg-full.mkv",
+ "out": "thumbs/pokemon-c-battle2.jpg",
+ "bg": "intro/pkmn-bg-full.mkv",
"bgAt": 63.86,
"corners": [
{
diff --git a/umtool/song/um-manifest.json b/umtool/song/um-manifest.json
@@ -40,7 +40,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kT1gUTlBfNc.mp4",
"adj": {
"b": "told.",
"a": "The"
@@ -114,7 +113,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "",
"a": "members"
@@ -138,7 +136,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 224ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/78A048gOuq8.mp4",
"adj": {
"b": "",
"a": ""
@@ -190,7 +187,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 344ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZeqS0DvnorA.mp4",
"adj": {
"b": "was",
"a": ""
@@ -264,7 +260,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 274ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FWv906SubiM.mp4",
"adj": {
"b": "food",
"a": ""
@@ -288,7 +283,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 254ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "",
"a": ""
@@ -426,7 +420,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cwMkQSs6OeM.mp4",
"adj": {
"b": "",
"a": ""
@@ -486,7 +479,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 289ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1Y0204hKq2M.mp4",
"adj": {
"b": "",
"a": ""
@@ -552,7 +544,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "",
"a": "you"
@@ -576,7 +567,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "",
"a": ""
@@ -600,7 +590,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 304ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "it.",
"a": ""
@@ -638,7 +627,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +380 cents from the corpus (128Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Api44x4HVGs.mp4",
"adj": {
"b": "she",
"a": ""
@@ -668,7 +656,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3gORoyt2mC4.mp4",
"adj": {
"b": "know",
"a": ""
@@ -800,7 +787,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "",
"a": ""
@@ -866,7 +852,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 214ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/z5IYf8iDnjA.mp4",
"adj": {
"b": "",
"a": "tied"
@@ -968,7 +953,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 354ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mwtbQffMRic.mp4",
"adj": {
"b": "",
"a": ""
@@ -1014,7 +998,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yjHHiRhtMvQ.mp4",
"adj": {
"b": "is",
"a": "she"
@@ -1102,7 +1085,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/EN9H7RVxZtw.mp4",
"adj": {
"b": "",
"a": "that"
@@ -1288,7 +1270,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "",
"a": ""
@@ -1390,7 +1371,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 359ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/iYbUqMyCOLU.mp4",
"adj": {
"b": "anyway",
"a": ""
@@ -1414,7 +1394,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 434ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "",
"a": ""
@@ -1488,7 +1467,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TTN8zF4Vjz4.mp4",
"adj": {
"b": "of",
"a": "political"
@@ -1610,7 +1588,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/DDRw-m4lyus.mp4",
"adj": {
"b": "that,",
"a": "but"
@@ -1844,7 +1821,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cPKtKx3evdM.mp4",
"adj": {
"b": "",
"a": ""
@@ -1946,7 +1922,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "window moved 249ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "vehicle",
"a": ""
@@ -2062,7 +2037,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 364ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "",
"a": ""
@@ -2162,7 +2136,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 314ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dWMRXD0RJRY.mp4",
"adj": {
"b": "",
"a": "If"
@@ -2250,7 +2223,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 319ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "date",
"a": "all"
@@ -2586,7 +2558,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/GJw2TXJu6kY.mp4",
"adj": {
"b": "great",
"a": "thing"
@@ -2783,7 +2754,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 304ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KssiwNCTEKQ.mp4",
"adj": {
"b": "hard",
"a": "purchasing"
@@ -2821,7 +2791,6 @@
"why": "next word \"was\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OlY74Y-NmMw.mp4",
"adj": {
"b": "",
"a": "was"
@@ -2892,7 +2861,6 @@
"why": "next word \"Paul\" starts here",
"wordIn": "",
"risk": "word \"Paul\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/SMBeGH0Zimw.mp4",
"adj": {
"b": "",
"a": "Paul"
@@ -2936,7 +2904,6 @@
"why": "next word \"requirement\" starts here",
"wordIn": "the",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/VNdJ8BvruCQ.mp4",
"adj": {
"b": "",
"a": "requirement"
@@ -3097,7 +3064,6 @@
"why": "next word \"And\" starts here",
"wordIn": "",
"risk": "word \"And\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "",
"a": "And"
@@ -3233,7 +3199,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 454ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mwtbQffMRic.mp4",
"adj": {
"b": "using",
"a": ""
@@ -3257,7 +3222,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 314ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nrKQ-WoQ_-Y.mp4",
"adj": {
"b": "",
"a": "It"
@@ -3287,7 +3251,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tC_WyEZzLV0.mp4",
"adj": {
"b": "people,",
"a": "so"
@@ -3325,7 +3288,6 @@
"why": "next word \"this\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8PNFWjrpXrE.mp4",
"adj": {
"b": "tweeted,",
"a": "this"
@@ -3439,7 +3401,6 @@
"why": "",
"wordIn": "and",
"risk": "word \"and\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UwD3NXKMsoQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -3595,7 +3556,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/78A048gOuq8.mp4",
"adj": {
"b": "",
"a": "The"
@@ -3723,7 +3683,6 @@
"why": "",
"wordIn": "and",
"risk": "word \"and\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "said",
"a": ""
@@ -3767,7 +3726,6 @@
"why": "next word \"It's\" starts here",
"wordIn": "",
"risk": "word \"It's\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nM5qgOKOEsQ.mp4",
"adj": {
"b": "knife.",
"a": "It's"
@@ -3895,7 +3853,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "window moved 324ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HPIHN8nxCM8.mp4",
"adj": {
"b": "",
"a": ""
@@ -4031,7 +3988,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 329ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "",
"a": ""
@@ -4197,7 +4153,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "a clip here was judged \"other speaker, not Jer\"",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HPIHN8nxCM8.mp4",
"adj": {
"b": "",
"a": "here"
@@ -4422,7 +4377,6 @@
"why": "next word \"to\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Key66cOeiEo.mp4",
"adj": {
"b": "polls",
"a": "to"
@@ -4466,7 +4420,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "trade.",
"a": "But"
@@ -4510,7 +4463,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cQL0Rja_Qik.mp4",
"adj": {
"b": "it's",
"a": "thinking"
@@ -4604,7 +4556,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tC_WyEZzLV0.mp4",
"adj": {
"b": "but",
"a": "there's"
@@ -4684,7 +4635,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/DDRw-m4lyus.mp4",
"adj": {
"b": "in",
"a": "certain"
@@ -4789,7 +4739,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 464ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Nb5iswtRmVM.mp4",
"adj": {
"b": "",
"a": ""
@@ -4872,7 +4821,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 454ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZeqS0DvnorA.mp4",
"adj": {
"b": "but",
"a": ""
@@ -5047,7 +4995,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 384ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-lSWX5qachE.mp4",
"adj": {
"b": "",
"a": "Apparently"
@@ -5099,7 +5046,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7zRNtT2ygCc.mp4",
"adj": {
"b": "to",
"a": ""
@@ -5160,7 +5106,6 @@
"why": "next word \"rotund\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FWv906SubiM.mp4",
"adj": {
"b": "haired",
"a": "rotund"
@@ -5207,7 +5152,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Me0fryGe2X8.mp4",
"adj": {
"b": "revolution",
"a": "and"
@@ -5423,7 +5367,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 289ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "",
"a": ""
@@ -5579,7 +5522,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BKUP9tal4mM.mp4",
"adj": {
"b": "name",
"a": "years"
@@ -5617,7 +5559,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 244ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/EN9H7RVxZtw.mp4",
"adj": {
"b": "perfect",
"a": "last"
@@ -5675,7 +5616,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "about,",
"a": "you"
@@ -5801,7 +5741,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 309ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TYPV_Ej1ND4.mp4",
"adj": {
"b": "",
"a": ""
@@ -5951,7 +5890,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 334ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-CcZe5Bffzs.mp4",
"adj": {
"b": "its",
"a": "eventual"
@@ -6055,7 +5993,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/GdMSdVvQvtc.mp4",
"adj": {
"b": "their",
"a": "siege"
@@ -6167,7 +6104,6 @@
"why": "next word \"Hoodie\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ts76T7tJv-U.mp4",
"adj": {
"b": "Loren",
"a": "Hoodie"
@@ -6214,7 +6150,6 @@
"why": "next word \"Black\" starts here",
"wordIn": "",
"risk": "word \"Black\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "",
"a": "Black"
@@ -6258,7 +6193,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "window moved 289ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "criticism",
"a": "it's"
@@ -6372,7 +6306,6 @@
"why": "next word \"he\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kfzmLf09O0w.mp4",
"adj": {
"b": "",
"a": "he"
@@ -6509,7 +6442,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 214ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3gORoyt2mC4.mp4",
"adj": {
"b": "",
"a": "and"
@@ -6649,7 +6581,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Oo6dXSEh65Y.mp4",
"adj": {
"b": "like",
"a": "Candice,"
@@ -6687,7 +6618,6 @@
"why": "next word \"but\" starts here",
"wordIn": "",
"risk": "word \"but\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "Northwest",
"a": "but"
@@ -6854,7 +6784,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 324ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/azX_WsMqFqg.mp4",
"adj": {
"b": "that",
"a": "this"
@@ -6914,7 +6843,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 314ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dVSd_fotB2U.mp4",
"adj": {
"b": "",
"a": ""
@@ -6980,7 +6908,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kT1gUTlBfNc.mp4",
"adj": {
"b": "person",
"a": "lives"
@@ -7004,7 +6931,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 299ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m_aKA-Khmck.mp4",
"adj": {
"b": "",
"a": ""
@@ -7046,7 +6972,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 284ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oYsPh2E-02E.mp4",
"adj": {
"b": "to",
"a": "garner"
@@ -7098,7 +7023,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 214ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "asking",
"a": ""
@@ -7184,7 +7108,6 @@
"why": "next word \"So\" starts here",
"wordIn": "",
"risk": "word \"So\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-lSWX5qachE.mp4",
"adj": {
"b": "anymore.",
"a": "So"
@@ -7214,7 +7137,6 @@
"why": "",
"wordIn": "didn't",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0A4dIN1dKaY.mp4",
"adj": {
"b": "you",
"a": "I"
@@ -7304,7 +7226,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Hh71Xe7XK6k.mp4",
"adj": {
"b": "Floyd",
"a": "overdosed"
@@ -7334,7 +7255,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/JKozJKyP4Iw.mp4",
"adj": {
"b": "And",
"a": "I'll"
@@ -7369,7 +7289,6 @@
"why": "unvoiced tail",
"wordIn": "And",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/JKozJKyP4Iw.mp4",
"adj": {
"b": "And",
"a": "God"
@@ -7417,7 +7336,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LoaYx6-_arE.mp4",
"adj": {
"b": "say",
"a": "because"
@@ -7709,7 +7627,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +251 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-CcZe5Bffzs.mp4",
"adj": {
"b": "wait",
"a": "Joe"
@@ -7824,7 +7741,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/6dgXutAKziM.mp4",
"adj": {
"b": "",
"a": "Schwab"
@@ -7918,7 +7834,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 234ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9zEisQcriiM.mp4",
"adj": {
"b": "Walgreens",
"a": "their"
@@ -8072,7 +7987,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 299ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Nb5iswtRmVM.mp4",
"adj": {
"b": "",
"a": "fine"
@@ -8096,7 +8010,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "",
"a": "I"
@@ -8126,7 +8039,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "should",
"a": "suck"
@@ -8188,7 +8100,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "",
"a": ""
@@ -8282,7 +8193,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 274ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZEwkRomA9pU.mp4",
"adj": {
"b": "the",
"a": "lazy"
@@ -8352,7 +8262,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 224ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cPKtKx3evdM.mp4",
"adj": {
"b": "",
"a": "because"
@@ -8482,7 +8391,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 274ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6CNFodmbWw.mp4",
"adj": {
"b": "",
"a": ""
@@ -8520,7 +8428,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 249ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "our",
"a": "in"
@@ -8564,7 +8471,6 @@
"why": "",
"wordIn": "I've",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/pNP4B0PJk-Q.mp4",
"adj": {
"b": "but",
"a": "got"
@@ -8622,7 +8528,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 244ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rJ-V9bd9La0.mp4",
"adj": {
"b": "",
"a": ""
@@ -8646,7 +8551,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 384ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "down.",
"a": "You"
@@ -8670,7 +8574,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 449ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "identity",
"a": ""
@@ -8694,7 +8597,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 234ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "",
"a": "here"
@@ -8754,7 +8656,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 374ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "",
"a": ""
@@ -8837,7 +8738,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yVm_CeJYFCc.mp4",
"adj": {
"b": "a",
"a": "hydroponic"
@@ -8861,7 +8761,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 424ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/z5IYf8iDnjA.mp4",
"adj": {
"b": "of",
"a": "bump."
@@ -8905,7 +8804,6 @@
"why": "next word \"this\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-lSWX5qachE.mp4",
"adj": {
"b": "",
"a": "this"
@@ -8968,7 +8866,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 234ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "",
"a": "you"
@@ -9006,7 +8903,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 224ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3gORoyt2mC4.mp4",
"adj": {
"b": "guy",
"a": "in"
@@ -9262,7 +9158,6 @@
"why": "next word \"it's\" starts here",
"wordIn": "",
"risk": "word \"it's\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Nb5iswtRmVM.mp4",
"adj": {
"b": "",
"a": "it's"
@@ -9370,7 +9265,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Tvpplznw9Nw.mp4",
"adj": {
"b": "Villanueva",
"a": "he"
@@ -9394,7 +9288,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "",
"a": "If"
@@ -9443,7 +9336,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "a",
"a": "lunatic"
@@ -9522,7 +9414,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 234ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "do",
"a": "to"
@@ -9546,7 +9437,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/azX_WsMqFqg.mp4",
"adj": {
"b": "you",
"a": ""
@@ -9612,7 +9502,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fp_GOiz5MV0.mp4",
"adj": {
"b": "",
"a": "for"
@@ -9636,7 +9525,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 334ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fp_GOiz5MV0.mp4",
"adj": {
"b": "",
"a": ""
@@ -9660,7 +9548,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/jprDE87Ec58.mp4",
"adj": {
"b": "",
"a": "I'm"
@@ -9726,7 +9613,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 244ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nM5qgOKOEsQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -9932,7 +9818,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 314ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "",
"a": "you"
@@ -9956,7 +9841,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 509ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "",
"a": "and"
@@ -10002,7 +9886,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "",
"a": "He"
@@ -10088,7 +9971,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1Y0204hKq2M.mp4",
"adj": {
"b": "have",
"a": ""
@@ -10130,7 +10012,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7ctY0v_LEV0.mp4",
"adj": {
"b": "",
"a": "and"
@@ -10188,7 +10069,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 289ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BKUP9tal4mM.mp4",
"adj": {
"b": "say",
"a": ""
@@ -10218,7 +10098,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 244ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/DDRw-m4lyus.mp4",
"adj": {
"b": "",
"a": "or"
@@ -10318,7 +10197,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 234ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/JKozJKyP4Iw.mp4",
"adj": {
"b": "this.",
"a": "She's"
@@ -10342,7 +10220,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "window moved 329ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KssiwNCTEKQ.mp4",
"adj": {
"b": "who",
"a": ""
@@ -10414,7 +10291,6 @@
"why": "next word \"thinking\" starts here",
"wordIn": "",
"risk": "word \"thinking\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/N78crpC9o1g.mp4",
"adj": {
"b": "was",
"a": "thinking"
@@ -10516,7 +10392,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 339ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "today.",
"a": ""
@@ -10540,7 +10415,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 244ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QqIWnAhGwfw.mp4",
"adj": {
"b": "know",
"a": "considered"
@@ -10598,7 +10472,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "",
"a": ""
@@ -10644,7 +10517,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 239ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "it",
"a": "send"
@@ -10752,7 +10624,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/YKKC7QzLYss.mp4",
"adj": {
"b": "",
"a": ""
@@ -10826,7 +10697,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dVSd_fotB2U.mp4",
"adj": {
"b": "",
"a": "so"
@@ -10856,7 +10726,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dhOXEakOpaM.mp4",
"adj": {
"b": "event",
"a": "by"
@@ -10886,7 +10755,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/h170V_0AbQ4.mp4",
"adj": {
"b": "",
"a": "sign"
@@ -10910,7 +10778,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 249ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hU43s57gcnI.mp4",
"adj": {
"b": "",
"a": "but"
@@ -10948,7 +10815,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/jprDE87Ec58.mp4",
"adj": {
"b": "obviously",
"a": "you"
@@ -10972,7 +10838,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kT1gUTlBfNc.mp4",
"adj": {
"b": "her",
"a": "as"
@@ -11024,7 +10889,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 234ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/knmVtqTW0MQ.mp4",
"adj": {
"b": "",
"a": "You"
@@ -11160,7 +11024,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 324ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oYsPh2E-02E.mp4",
"adj": {
"b": "film",
"a": "and"
@@ -11184,7 +11047,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oYsPh2E-02E.mp4",
"adj": {
"b": "",
"a": "I"
@@ -11261,7 +11123,6 @@
"why": "next word \"it\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "",
"a": "it"
@@ -11296,7 +11157,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "",
"a": "I"
@@ -11320,7 +11180,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "me",
"a": ""
@@ -11372,7 +11231,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 224ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "that",
"a": "I"
@@ -11396,7 +11254,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 239ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "",
"a": "people"
@@ -11452,7 +11309,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 329ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ybLrSiTQxtQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -11579,7 +11435,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 419ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/78A048gOuq8.mp4",
"adj": {
"b": "and",
"a": ""
@@ -11617,7 +11472,6 @@
"why": "",
"wordIn": "in",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7zRNtT2ygCc.mp4",
"adj": {
"b": "strongly",
"a": "self"
@@ -11696,7 +11550,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 249ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/AMdAOwrYmAI.mp4",
"adj": {
"b": "witness",
"a": ""
@@ -11734,7 +11587,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +380 cents from the corpus (128Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Api44x4HVGs.mp4",
"adj": {
"b": "",
"a": "just"
@@ -11828,7 +11680,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FWv906SubiM.mp4",
"adj": {
"b": "but",
"a": "I"
@@ -11869,7 +11720,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/GdMSdVvQvtc.mp4",
"adj": {
"b": "",
"a": "here"
@@ -11963,7 +11813,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 249ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HPIHN8nxCM8.mp4",
"adj": {
"b": "violate",
"a": "the"
@@ -12023,7 +11872,6 @@
"why": "",
"wordIn": "And",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/JKozJKyP4Iw.mp4",
"adj": {
"b": "area.",
"a": "I"
@@ -12134,7 +11982,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 379ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LoaYx6-_arE.mp4",
"adj": {
"b": "",
"a": ""
@@ -12200,7 +12047,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 274ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/N78crpC9o1g.mp4",
"adj": {
"b": "",
"a": "you"
@@ -12238,7 +12084,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Nb5iswtRmVM.mp4",
"adj": {
"b": "",
"a": "to"
@@ -12322,7 +12167,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 209ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "them",
"a": ""
@@ -12382,7 +12226,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 369ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Qyjo41I1FcI.mp4",
"adj": {
"b": "",
"a": "anything"
@@ -12437,7 +12280,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "bystander",
"a": "DC"
@@ -12467,7 +12309,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TTN8zF4Vjz4.mp4",
"adj": {
"b": "",
"a": "unless"
@@ -12561,7 +12402,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 354ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Zfcmx4_Qxxg.mp4",
"adj": {
"b": "it",
"a": ""
@@ -12591,7 +12431,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/azX_WsMqFqg.mp4",
"adj": {
"b": "",
"a": "I'm"
@@ -12621,7 +12460,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bbVoMOzCSs0.mp4",
"adj": {
"b": "that",
"a": ""
@@ -12651,7 +12489,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bbVoMOzCSs0.mp4",
"adj": {
"b": "garbage.",
"a": "It's"
@@ -12681,7 +12518,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 254ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -12705,7 +12541,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 309ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/acPLSEVKyqI.mp4",
"adj": {
"b": "",
"a": ""
@@ -12767,7 +12602,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 334ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/DMAa1fld4Rc.mp4",
"adj": {
"b": "these",
"a": ""
@@ -12791,7 +12625,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "",
"a": ""
@@ -12815,7 +12648,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "window moved 374ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lraibG7Iu5w.mp4",
"adj": {
"b": "else",
"a": ""
@@ -12839,7 +12671,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "",
"a": "for"
@@ -12863,7 +12694,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 209ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -12909,7 +12739,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 274ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "Cox",
"a": ""
@@ -12933,7 +12762,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KyzBZ19iRaU.mp4",
"adj": {
"b": "Charlie.",
"a": ""
@@ -12979,7 +12807,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 399ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "",
"a": ""
@@ -13047,7 +12874,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "masterclass",
"a": "but"
@@ -13071,7 +12897,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 249ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/4JiT3UO-yCQ.mp4",
"adj": {
"b": "know",
"a": ""
@@ -13095,7 +12920,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lraibG7Iu5w.mp4",
"adj": {
"b": "down",
"a": "look"
@@ -13119,7 +12943,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "it",
"a": ""
@@ -13165,7 +12988,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 449ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "guy",
"a": ""
@@ -13203,7 +13025,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lraibG7Iu5w.mp4",
"adj": {
"b": "odd",
"a": ""
@@ -13335,7 +13156,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KyzBZ19iRaU.mp4",
"adj": {
"b": "Benny.",
"a": ""
@@ -13398,7 +13218,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8hsc6aSGdk.mp4",
"adj": {
"b": "stuff",
"a": "I"
@@ -13422,7 +13241,6 @@
"why": "unvoiced tail",
"wordIn": "crazy",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUoDtOdIza4.mp4",
"adj": {
"b": "these",
"a": ""
@@ -13452,7 +13270,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 344ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mGcQRQ3wqsk.mp4",
"adj": {
"b": "",
"a": "unhinged"
@@ -13476,7 +13293,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 239ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lraibG7Iu5w.mp4",
"adj": {
"b": "reaction",
"a": ""
@@ -13500,7 +13316,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "say",
"a": ""
@@ -13538,7 +13353,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 234ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Yq2EjM3ZXN4.mp4",
"adj": {
"b": "and",
"a": "Tim"
@@ -13621,7 +13435,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 274ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "",
"a": "and"
@@ -13645,7 +13458,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 314ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "at",
"a": "the"
@@ -13669,7 +13481,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 554ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "",
"a": ""
@@ -13728,7 +13539,6 @@
"why": "re-attack after a dip",
"wordIn": "arrested",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/67Qd5J47Y84.mp4",
"adj": {
"b": "",
"a": ""
@@ -13780,7 +13590,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "recently",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/i29P5pQc50A.mp4",
"adj": {
"b": "have",
"a": ""
@@ -13896,7 +13705,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WqAu8gLaW-4.mp4",
"adj": {
"b": "was",
"a": "pretty"
@@ -13966,7 +13774,6 @@
"why": "trailing noise burst",
"wordIn": "these",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FiLs5ovIfts.mp4",
"adj": {
"b": "into",
"a": ""
@@ -14059,7 +13866,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "driving",
"a": "to"
@@ -14083,7 +13889,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/zg5LTpYUrlg.mp4",
"adj": {
"b": "",
"a": "before"
@@ -14107,7 +13912,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 319ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "",
"a": ""
@@ -14145,7 +13949,6 @@
"why": "next word \"Snow\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "",
"a": "Snow"
@@ -14189,7 +13992,6 @@
"why": "re-attack after a dip",
"wordIn": "Foyd",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rATxtI8FhxE.mp4",
"adj": {
"b": "George",
"a": ""
@@ -14241,7 +14043,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "botted",
"a": "he"
@@ -14283,7 +14084,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/acPLSEVKyqI.mp4",
"adj": {
"b": "platformed",
"a": "that's"
@@ -14397,7 +14197,6 @@
"why": "",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WqAu8gLaW-4.mp4",
"adj": {
"b": "there",
"a": ""
@@ -14496,7 +14295,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "",
"a": "We"
@@ -14520,7 +14318,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "so",
"a": "much"
@@ -14566,7 +14363,6 @@
"why": "unvoiced tail",
"wordIn": "have",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tcevefZ9G18.mp4",
"adj": {
"b": "to",
"a": ""
@@ -14624,7 +14420,6 @@
"why": "trailing noise burst",
"wordIn": "say",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/22DisumpVqo.mp4",
"adj": {
"b": "they",
"a": "Somali"
@@ -14697,7 +14492,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "",
"a": "I"
@@ -14763,7 +14557,6 @@
"why": "trailing noise burst",
"wordIn": "injuries",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8hsc6aSGdk.mp4",
"adj": {
"b": "serious",
"a": ""
@@ -14869,7 +14662,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/axL33DS5KeA.mp4",
"adj": {
"b": "",
"a": "I"
@@ -14899,7 +14691,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "threats",
"a": "according"
@@ -14923,7 +14714,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 214ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "",
"a": "hit"
@@ -14967,7 +14757,6 @@
"why": "unvoiced tail",
"wordIn": "like",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m_B_3i9yKX0.mp4",
"adj": {
"b": "he",
"a": "a"
@@ -15097,7 +14886,6 @@
"why": "",
"wordIn": "because",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "too",
"a": "he"
@@ -15218,7 +15006,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "them",
"a": "on"
@@ -15270,7 +15057,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 379ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "",
"a": ""
@@ -15358,7 +15144,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 209ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/l9RG39RojHE.mp4",
"adj": {
"b": "",
"a": "the"
@@ -15382,7 +15167,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "was",
"a": "essentially"
@@ -15420,7 +15204,6 @@
"why": "unvoiced tail",
"wordIn": "Minneapolis,",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/22DisumpVqo.mp4",
"adj": {
"b": "Paul",
"a": ""
@@ -15482,7 +15265,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8hsc6aSGdk.mp4",
"adj": {
"b": "",
"a": ""
@@ -15520,7 +15302,6 @@
"why": "unvoiced tail",
"wordIn": "story",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WqAu8gLaW-4.mp4",
"adj": {
"b": "this",
"a": ""
@@ -15592,7 +15373,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lXmLzzuxVCo.mp4",
"adj": {
"b": "able",
"a": "to"
@@ -15632,7 +15412,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mOF790Bh6I0.mp4",
"adj": {
"b": "and",
"a": ""
@@ -15702,7 +15481,6 @@
"why": "unvoiced tail",
"wordIn": "jeans",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/v-MG7ofJ6X4.mp4",
"adj": {
"b": "his",
"a": ""
@@ -15737,7 +15515,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "",
"a": "Allocating"
@@ -15789,7 +15566,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/MB0fIIPySYs.mp4",
"adj": {
"b": "saying",
"a": "well"
@@ -15833,7 +15609,6 @@
"why": "next word \"I'll\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WqAu8gLaW-4.mp4",
"adj": {
"b": "here",
"a": "I'll"
@@ -15863,7 +15638,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 354ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/acPLSEVKyqI.mp4",
"adj": {
"b": "",
"a": "they"
@@ -15887,7 +15661,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 369ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "and",
"a": ""
@@ -15978,7 +15751,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rPkFK9TgI3w.mp4",
"adj": {
"b": "",
"a": ""
@@ -16002,7 +15774,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "",
"a": "deputies"
@@ -16044,7 +15815,6 @@
"why": "",
"wordIn": "phenomenon,",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0ynoYuPD-B8.mp4",
"adj": {
"b": "aerial",
"a": "which"
@@ -16110,7 +15880,6 @@
"why": "unvoiced tail",
"wordIn": "responses",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/4RFFlS0oMEM.mp4",
"adj": {
"b": "",
"a": ""
@@ -16159,7 +15928,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ep_qtW2K3BY.mp4",
"adj": {
"b": "was",
"a": "we"
@@ -16189,7 +15957,6 @@
"why": "unvoiced tail",
"wordIn": "robbed",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FiLs5ovIfts.mp4",
"adj": {
"b": "get",
"a": "people"
@@ -16224,7 +15991,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "fine",
"a": ""
@@ -16304,7 +16070,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUoDtOdIza4.mp4",
"adj": {
"b": "some",
"a": "plugin"
@@ -16328,7 +16093,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 409ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "or",
"a": ""
@@ -16352,7 +16116,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mkyBXrJhDrQ.mp4",
"adj": {
"b": "go",
"a": "Schumer"
@@ -16431,7 +16194,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 274ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CE3O2H43_tc.mp4",
"adj": {
"b": "be",
"a": "tough"
@@ -16472,7 +16234,6 @@
"why": "next word \"more\" starts here",
"wordIn": "paying",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FiLs5ovIfts.mp4",
"adj": {
"b": "are",
"a": "more"
@@ -16517,7 +16278,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Mf1hEHazJaE.mp4",
"adj": {
"b": "status",
"a": "please"
@@ -16558,7 +16318,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 274ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/V4k0ziHTGog.mp4",
"adj": {
"b": "",
"a": "but"
@@ -16599,7 +16358,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WqAu8gLaW-4.mp4",
"adj": {
"b": "ER",
"a": "one"
@@ -16639,7 +16397,6 @@
"why": "",
"wordIn": "be",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Yq2EjM3ZXN4.mp4",
"adj": {
"b": "would",
"a": ""
@@ -16705,7 +16462,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "and",
"a": ""
@@ -16747,7 +16503,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 214ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/l9RG39RojHE.mp4",
"adj": {
"b": "that.",
"a": "That's"
@@ -16785,7 +16540,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mGcQRQ3wqsk.mp4",
"adj": {
"b": "work.",
"a": ""
@@ -16857,7 +16611,6 @@
"why": "next word \"whatever\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "",
"a": "whatever"
@@ -16887,7 +16640,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/A9Zg-xqDIcM.mp4",
"adj": {
"b": "Trump",
"a": ""
@@ -16917,7 +16669,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CE3O2H43_tc.mp4",
"adj": {
"b": "",
"a": "I"
@@ -16983,7 +16734,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "",
"a": "yeah"
@@ -17007,7 +16757,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WqAu8gLaW-4.mp4",
"adj": {
"b": "with",
"a": ""
@@ -17031,7 +16780,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "and",
"a": ""
@@ -17111,7 +16859,6 @@
"why": "next word \"He\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m_B_3i9yKX0.mp4",
"adj": {
"b": "",
"a": "He"
@@ -17146,7 +16893,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mkyBXrJhDrQ.mp4",
"adj": {
"b": "Here's",
"a": "MSNBC's"
@@ -17246,7 +16992,6 @@
"why": "next word \"As\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xpp98iH3zOs.mp4",
"adj": {
"b": "",
"a": "As"
@@ -17401,7 +17146,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8hsc6aSGdk.mp4",
"adj": {
"b": "wants",
"a": "and"
@@ -17501,7 +17245,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "",
"a": "the"
@@ -17536,7 +17279,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 319ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "",
"a": ""
@@ -17578,7 +17320,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "",
"a": "I"
@@ -17680,7 +17421,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 379ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "",
"a": ""
@@ -17718,7 +17458,6 @@
"why": "trailing noise burst",
"wordIn": "others",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "real",
"a": ""
@@ -17798,7 +17537,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 244ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ylmhShFD48Q.mp4",
"adj": {
"b": "damning.",
"a": "Here's"
@@ -17828,7 +17566,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 309ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ylmhShFD48Q.mp4",
"adj": {
"b": "",
"a": ""
@@ -17851,7 +17588,6 @@
"why": "next word \"Now\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/4JiT3UO-yCQ.mp4",
"adj": {
"b": "",
"a": "Now"
@@ -17937,7 +17673,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 319ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/67Qd5J47Y84.mp4",
"adj": {
"b": "Jordan",
"a": ""
@@ -17961,7 +17696,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7AtUt83XGQU.mp4",
"adj": {
"b": "native",
"a": "American"
@@ -18005,7 +17739,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 324ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7AtUt83XGQU.mp4",
"adj": {
"b": "",
"a": ""
@@ -18029,7 +17762,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 254ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/E_K0pCUIiCI.mp4",
"adj": {
"b": "",
"a": "asylum"
@@ -18119,7 +17851,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/JcEkqCQP_FE.mp4",
"adj": {
"b": "and",
"a": "force"
@@ -18149,7 +17880,6 @@
"why": "next word \"in\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "",
"a": "in"
@@ -18196,7 +17926,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Mf1hEHazJaE.mp4",
"adj": {
"b": "by",
"a": "retired"
@@ -18220,7 +17949,6 @@
"why": "trailing noise burst",
"wordIn": "these",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "these",
"a": ""
@@ -18291,7 +18019,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/V4k0ziHTGog.mp4",
"adj": {
"b": "it",
"a": "or"
@@ -18315,7 +18042,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WJYyCfbjv8s.mp4",
"adj": {
"b": "",
"a": ""
@@ -18378,7 +18104,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "would",
"a": ""
@@ -18419,7 +18144,6 @@
"why": "",
"wordIn": "by",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "posted",
"a": ""
@@ -18476,7 +18200,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/etu7Cy2WmC4.mp4",
"adj": {
"b": "this",
"a": "in"
@@ -18524,7 +18247,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/f5sLSteRRTQ.mp4",
"adj": {
"b": "",
"a": "shout"
@@ -18548,7 +18270,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "know",
"a": "it"
@@ -18595,7 +18316,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 339ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "",
"a": "you"
@@ -18619,7 +18339,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/iAwVWRp5Ayg.mp4",
"adj": {
"b": "",
"a": "K"
@@ -18696,7 +18415,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 344ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mkyBXrJhDrQ.mp4",
"adj": {
"b": "",
"a": "now"
@@ -18734,7 +18452,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "",
"a": "that"
@@ -18775,7 +18492,6 @@
"why": "",
"wordIn": "permissible",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/q3Z3pcAfNss.mp4",
"adj": {
"b": "it's",
"a": ""
@@ -18805,7 +18521,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/q3Z3pcAfNss.mp4",
"adj": {
"b": "and",
"a": ""
@@ -18829,7 +18544,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "",
"a": "cuts"
@@ -18870,7 +18584,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "",
"a": "here"
@@ -18894,7 +18607,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "",
"a": "trust"
@@ -18946,7 +18658,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 299ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "to",
"a": "parent"
@@ -18990,7 +18701,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 304ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "classic",
"a": "and"
@@ -19020,7 +18730,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "get",
"a": "remember"
@@ -19044,7 +18753,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yGvQfTultRw.mp4",
"adj": {
"b": "concerns",
"a": "we"
@@ -19135,7 +18843,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "",
"a": ""
@@ -19159,7 +18866,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/4JiT3UO-yCQ.mp4",
"adj": {
"b": "",
"a": "basically"
@@ -19217,7 +18923,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8RiHnmAGy8U.mp4",
"adj": {
"b": "",
"a": "I"
@@ -19255,7 +18960,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CE3O2H43_tc.mp4",
"adj": {
"b": "dregs",
"a": "or"
@@ -19279,7 +18983,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "footstool",
"a": ""
@@ -19303,7 +19006,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/E_K0pCUIiCI.mp4",
"adj": {
"b": "",
"a": ""
@@ -19341,7 +19043,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ep_qtW2K3BY.mp4",
"adj": {
"b": "gonna",
"a": ""
@@ -19393,7 +19094,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FiLs5ovIfts.mp4",
"adj": {
"b": "",
"a": "for"
@@ -19417,7 +19117,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "",
"a": "but"
@@ -19459,7 +19158,6 @@
"why": "unvoiced tail",
"wordIn": "don't",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/MB0fIIPySYs.mp4",
"adj": {
"b": "please",
"a": ""
@@ -19489,7 +19187,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 314ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Mf1hEHazJaE.mp4",
"adj": {
"b": "",
"a": ""
@@ -19513,7 +19210,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Mf1hEHazJaE.mp4",
"adj": {
"b": "",
"a": "just"
@@ -19543,7 +19239,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Mf1hEHazJaE.mp4",
"adj": {
"b": "only",
"a": ""
@@ -19573,7 +19268,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 349ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -19597,7 +19291,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 304ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Yq2EjM3ZXN4.mp4",
"adj": {
"b": "",
"a": "flagrant,"
@@ -19674,7 +19367,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bdBt-aXn-nA.mp4",
"adj": {
"b": "that",
"a": "seems"
@@ -19704,7 +19396,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 419ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "with",
"a": ""
@@ -19728,7 +19419,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 244ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "",
"a": ""
@@ -19797,7 +19487,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 234ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dVSd_fotB2U.mp4",
"adj": {
"b": "",
"a": ""
@@ -19821,7 +19510,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 284ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dVSd_fotB2U.mp4",
"adj": {
"b": "but",
"a": "holy"
@@ -19845,7 +19533,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dVSd_fotB2U.mp4",
"adj": {
"b": "it",
"a": "but"
@@ -19869,7 +19556,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 249ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dVSd_fotB2U.mp4",
"adj": {
"b": "",
"a": "decided"
@@ -19893,7 +19579,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/etu7Cy2WmC4.mp4",
"adj": {
"b": "minimum",
"a": "and"
@@ -19923,7 +19608,6 @@
"why": "next word \"wherever\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/f5sLSteRRTQ.mp4",
"adj": {
"b": "",
"a": "wherever"
@@ -19984,7 +19668,6 @@
"why": "next word \"not\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hU43s57gcnI.mp4",
"adj": {
"b": "",
"a": "not"
@@ -20019,7 +19702,6 @@
"why": "trailing noise burst",
"wordIn": "of",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "",
"a": ""
@@ -20049,7 +19731,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 359ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "",
"a": ""
@@ -20090,7 +19771,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 299ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hykdhazn8ds.mp4",
"adj": {
"b": "received",
"a": "someone"
@@ -20136,7 +19816,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/iAwVWRp5Ayg.mp4",
"adj": {
"b": "Dallas",
"a": ""
@@ -20174,7 +19853,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kfzmLf09O0w.mp4",
"adj": {
"b": "that",
"a": "kids"
@@ -20209,7 +19887,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/knmVtqTW0MQ.mp4",
"adj": {
"b": "",
"a": "you"
@@ -20281,7 +19958,6 @@
"why": "next word \"and\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lraibG7Iu5w.mp4",
"adj": {
"b": "",
"a": "and"
@@ -20311,7 +19987,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lraibG7Iu5w.mp4",
"adj": {
"b": "work",
"a": "what"
@@ -20349,7 +20024,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 434ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6CNFodmbWw.mp4",
"adj": {
"b": "",
"a": ""
@@ -20373,7 +20047,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mkyBXrJhDrQ.mp4",
"adj": {
"b": "correct",
"a": "with"
@@ -20403,7 +20076,6 @@
"why": "",
"wordIn": "standing",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mkyBXrJhDrQ.mp4",
"adj": {
"b": "",
"a": "right"
@@ -20491,7 +20163,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nPpJc33VuO4.mp4",
"adj": {
"b": "expensive",
"a": "the"
@@ -20515,7 +20186,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "activism.",
"a": ""
@@ -20587,7 +20257,6 @@
"why": "next word \"I\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "",
"a": "I"
@@ -20622,7 +20291,6 @@
"why": "",
"wordIn": "entirely",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oYsPh2E-02E.mp4",
"adj": {
"b": "I",
"a": ""
@@ -20652,7 +20320,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 334ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/pNP4B0PJk-Q.mp4",
"adj": {
"b": "",
"a": ""
@@ -20676,7 +20343,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/r3cTXMxBPbc.mp4",
"adj": {
"b": "",
"a": ""
@@ -20717,7 +20383,6 @@
"why": "next word \"been\" starts here",
"wordIn": "have",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "would",
"a": "been"
@@ -20752,7 +20417,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "",
"a": "this"
@@ -20822,7 +20486,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 404ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "",
"a": ""
@@ -20877,7 +20540,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tdYCm-SSyFw.mp4",
"adj": {
"b": "",
"a": "the"
@@ -20901,7 +20563,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "out",
"a": "it's"
@@ -20925,7 +20586,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "window moved 239ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "Trump.",
"a": "He"
@@ -20969,7 +20629,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 214ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "",
"a": "I'm"
@@ -20999,7 +20658,6 @@
"why": "",
"wordIn": "the",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vfMRAdhSSrc.mp4",
"adj": {
"b": "Yeah",
"a": "I"
@@ -21029,7 +20687,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +252 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vimidN7iVN0.mp4",
"adj": {
"b": "or",
"a": ""
@@ -21053,7 +20710,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xaWnBmX4AjY.mp4",
"adj": {
"b": "Two",
"a": "you"
@@ -21077,7 +20733,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 254ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xulBQwriW5E.mp4",
"adj": {
"b": "be,",
"a": "but"
@@ -21123,7 +20778,6 @@
"why": "next word \"Google\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ylmhShFD48Q.mp4",
"adj": {
"b": "",
"a": "Google"
@@ -21198,7 +20852,6 @@
"why": "next word \"while\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0A4dIN1dKaY.mp4",
"adj": {
"b": "",
"a": "while"
@@ -21291,7 +20944,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0pHYgMHhi60.mp4",
"adj": {
"b": "",
"a": "says"
@@ -21335,7 +20987,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "won't",
"a": "flag"
@@ -21365,7 +21016,6 @@
"why": "next word \"segment\" starts here",
"wordIn": "Chloe's",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "",
"a": "segment"
@@ -21414,7 +21064,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/4JiT3UO-yCQ.mp4",
"adj": {
"b": "",
"a": "with"
@@ -21438,7 +21087,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/5M-1yCq23hY.mp4",
"adj": {
"b": "And",
"a": "I"
@@ -21473,7 +21121,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/67Qd5J47Y84.mp4",
"adj": {
"b": "Mahmoud",
"a": "operated"
@@ -21534,7 +21181,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7ctY0v_LEV0.mp4",
"adj": {
"b": "",
"a": "not"
@@ -21578,7 +21224,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9YQfjW1uasM.mp4",
"adj": {
"b": "fact,",
"a": "as"
@@ -21622,7 +21267,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9YQfjW1uasM.mp4",
"adj": {
"b": "apparently.",
"a": "And"
@@ -21652,7 +21296,6 @@
"why": "re-attack after a dip",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9zEisQcriiM.mp4",
"adj": {
"b": "there",
"a": "it's"
@@ -21687,7 +21330,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9zEisQcriiM.mp4",
"adj": {
"b": "",
"a": "while"
@@ -21717,7 +21359,6 @@
"why": "",
"wordIn": "a",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Api44x4HVGs.mp4",
"adj": {
"b": "or",
"a": "exit"
@@ -21747,7 +21388,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BKUP9tal4mM.mp4",
"adj": {
"b": "",
"a": ""
@@ -21771,7 +21411,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/C-SzFQZbN2M.mp4",
"adj": {
"b": "",
"a": "over"
@@ -21819,7 +21458,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CTbCFWZ7Gdk.mp4",
"adj": {
"b": "and",
"a": "had"
@@ -21871,7 +21509,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 424ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "guy",
"a": ""
@@ -21909,7 +21546,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "",
"a": ""
@@ -21933,7 +21569,6 @@
"why": "next word \"need,\" starts here",
"wordIn": "don't",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FTvrBie8KPI.mp4",
"adj": {
"b": "We",
"a": "need,"
@@ -21999,7 +21634,6 @@
"why": "trailing noise burst",
"wordIn": "story",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/G02gq5ClixA.mp4",
"adj": {
"b": "this",
"a": "is"
@@ -22034,7 +21668,6 @@
"why": "next word \"and\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/G02gq5ClixA.mp4",
"adj": {
"b": "",
"a": "and"
@@ -22069,7 +21702,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/GJw2TXJu6kY.mp4",
"adj": {
"b": "actively",
"a": "living"
@@ -22107,7 +21739,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/GdMSdVvQvtc.mp4",
"adj": {
"b": "have",
"a": "a"
@@ -22213,7 +21844,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KssiwNCTEKQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -22254,7 +21884,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LLnW_XgyHg0.mp4",
"adj": {
"b": "know",
"a": ""
@@ -22309,7 +21938,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "",
"a": ""
@@ -22435,7 +22063,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "",
"a": "but"
@@ -22477,7 +22104,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "they",
"a": ""
@@ -22515,7 +22141,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 394ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "",
"a": "They"
@@ -22585,7 +22210,6 @@
"why": "unvoiced tail",
"wordIn": "funded",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Qyjo41I1FcI.mp4",
"adj": {
"b": "taxpayer",
"a": ""
@@ -22620,7 +22244,6 @@
"why": "next word \"of\" starts here",
"wordIn": "one",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/RT0mKx_hTv4.mp4",
"adj": {
"b": "",
"a": "of"
@@ -22660,7 +22283,6 @@
"why": "next word \"Sesame\" starts here",
"wordIn": "funding",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/RT0mKx_hTv4.mp4",
"adj": {
"b": "were",
"a": "Sesame"
@@ -22730,7 +22352,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 239ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "",
"a": "for"
@@ -22760,7 +22381,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UjzBIdbk2zY.mp4",
"adj": {
"b": "",
"a": "It's"
@@ -22784,7 +22404,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UjzBIdbk2zY.mp4",
"adj": {
"b": "",
"a": "This"
@@ -22905,7 +22524,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZEwkRomA9pU.mp4",
"adj": {
"b": "ideals",
"a": ""
@@ -22929,7 +22547,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZeqS0DvnorA.mp4",
"adj": {
"b": "have",
"a": "horrible"
@@ -22967,7 +22584,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/azX_WsMqFqg.mp4",
"adj": {
"b": "as",
"a": "an"
@@ -22991,7 +22607,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/azX_WsMqFqg.mp4",
"adj": {
"b": "terrifying",
"a": "for"
@@ -23015,7 +22630,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "window moved 209ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "",
"a": "like"
@@ -23057,7 +22671,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cN5k-3sfBdA.mp4",
"adj": {
"b": "",
"a": ""
@@ -23126,7 +22739,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 284ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dVSd_fotB2U.mp4",
"adj": {
"b": "",
"a": ""
@@ -23172,7 +22784,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/etu7Cy2WmC4.mp4",
"adj": {
"b": "to",
"a": "make"
@@ -23219,7 +22830,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fp_GOiz5MV0.mp4",
"adj": {
"b": "",
"a": "Hopefully"
@@ -23249,7 +22859,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "mean",
"a": ""
@@ -23343,7 +22952,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 309ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/jprDE87Ec58.mp4",
"adj": {
"b": "",
"a": ""
@@ -23394,7 +23002,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/knmVtqTW0MQ.mp4",
"adj": {
"b": "this",
"a": ""
@@ -23424,7 +23031,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 329ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/knmVtqTW0MQ.mp4",
"adj": {
"b": "",
"a": "like"
@@ -23448,7 +23054,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/l55Ad1gO5ww.mp4",
"adj": {
"b": "",
"a": ""
@@ -23486,7 +23091,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lC6PCsQT1Qo.mp4",
"adj": {
"b": "",
"a": "I"
@@ -23527,7 +23131,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 444ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mGcQRQ3wqsk.mp4",
"adj": {
"b": "and",
"a": ""
@@ -23551,7 +23154,6 @@
"why": "next word \"let's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mGcQRQ3wqsk.mp4",
"adj": {
"b": "",
"a": "let's"
@@ -23623,7 +23225,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mkyBXrJhDrQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -23661,7 +23262,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "But",
"a": "a"
@@ -23713,7 +23313,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nrKQ-WoQ_-Y.mp4",
"adj": {
"b": "at",
"a": "some"
@@ -23737,7 +23336,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nrKQ-WoQ_-Y.mp4",
"adj": {
"b": "fight.",
"a": "It"
@@ -23761,7 +23359,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "",
"a": "Somebody"
@@ -23825,7 +23422,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nzOE5f6VPqY.mp4",
"adj": {
"b": "",
"a": "We've"
@@ -23849,7 +23445,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "ordered",
"a": "the"
@@ -23893,7 +23488,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 224ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rATxtI8FhxE.mp4",
"adj": {
"b": "follow",
"a": "thirty"
@@ -23974,7 +23568,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/slD9-aQc04g.mp4",
"adj": {
"b": "",
"a": "a"
@@ -24016,7 +23609,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sogHQwGj7KA.mp4",
"adj": {
"b": "",
"a": "as"
@@ -24085,7 +23677,6 @@
"why": "next word \"Rick\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "of",
"a": "Rick"
@@ -24129,7 +23720,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tsQBo2SvOao.mp4",
"adj": {
"b": "sees",
"a": "that"
@@ -24191,7 +23781,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 394ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "these",
"a": ""
@@ -24233,7 +23822,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "",
"a": "I"
@@ -24257,7 +23845,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "opinion",
"a": ""
@@ -24295,7 +23882,6 @@
"why": "",
"wordIn": "supported",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xbT-lWAzaiA.mp4",
"adj": {
"b": "who",
"a": "him"
@@ -24335,7 +23921,6 @@
"why": "next word \"very\" starts here",
"wordIn": "far",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xbT-lWAzaiA.mp4",
"adj": {
"b": "thus",
"a": "very"
@@ -24389,7 +23974,6 @@
"why": "next word \"and\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yGvQfTultRw.mp4",
"adj": {
"b": "",
"a": "and"
@@ -24419,7 +24003,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 389ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yGvQfTultRw.mp4",
"adj": {
"b": "cirigs",
"a": "I'm"
@@ -24449,7 +24032,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yjHHiRhtMvQ.mp4",
"adj": {
"b": "but",
"a": ""
@@ -24473,7 +24055,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/z6kKxW8YBBY.mp4",
"adj": {
"b": "but",
"a": "you"
@@ -24571,7 +24152,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 284ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-pKXHyeyRhQ.mp4",
"adj": {
"b": "composay",
"a": "stores"
@@ -24601,7 +24181,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-pKXHyeyRhQ.mp4",
"adj": {
"b": "",
"a": "But"
@@ -24625,7 +24204,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-qfcXnIXn8Y.mp4",
"adj": {
"b": "",
"a": "I'm"
@@ -24655,7 +24233,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-qfcXnIXn8Y.mp4",
"adj": {
"b": "front.",
"a": "Those"
@@ -24697,7 +24274,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0A4dIN1dKaY.mp4",
"adj": {
"b": "",
"a": "previously"
@@ -24762,7 +24338,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0ynoYuPD-B8.mp4",
"adj": {
"b": "cool.",
"a": "The"
@@ -24803,7 +24378,6 @@
"why": "next word \"notice\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "",
"a": "notice"
@@ -24833,7 +24407,6 @@
"why": "unvoiced tail",
"wordIn": "uh",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "did",
"a": "yeah"
@@ -24891,7 +24464,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3gORoyt2mC4.mp4",
"adj": {
"b": "Montana",
"a": "received"
@@ -24929,7 +24501,6 @@
"why": "next word \"definitely\" starts here",
"wordIn": "We",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/5M-1yCq23hY.mp4",
"adj": {
"b": "",
"a": "definitely"
@@ -24964,7 +24535,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/67Qd5J47Y84.mp4",
"adj": {
"b": "",
"a": ""
@@ -25010,7 +24580,6 @@
"why": "next word \"they're\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7AtUt83XGQU.mp4",
"adj": {
"b": "but",
"a": "they're"
@@ -25045,7 +24614,6 @@
"why": "next word \"let's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7AtUt83XGQU.mp4",
"adj": {
"b": "",
"a": "let's"
@@ -25089,7 +24657,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7ctY0v_LEV0.mp4",
"adj": {
"b": "Don't",
"a": ""
@@ -25119,7 +24686,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7zRNtT2ygCc.mp4",
"adj": {
"b": "",
"a": "but"
@@ -25143,7 +24709,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9YQfjW1uasM.mp4",
"adj": {
"b": "",
"a": "Disney"
@@ -25167,7 +24732,6 @@
"why": "",
"wordIn": "know",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9YQfjW1uasM.mp4",
"adj": {
"b": "I",
"a": "a"
@@ -25197,7 +24761,6 @@
"why": "next word \"been\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9nKl6HY2y4Y.mp4",
"adj": {
"b": "has",
"a": "been"
@@ -25232,7 +24795,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9zEisQcriiM.mp4",
"adj": {
"b": "bad.",
"a": "I"
@@ -25276,7 +24838,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/A9Zg-xqDIcM.mp4",
"adj": {
"b": "jail.",
"a": "This"
@@ -25318,7 +24879,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 244ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/A9Zg-xqDIcM.mp4",
"adj": {
"b": "it",
"a": ""
@@ -25364,7 +24924,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BKUP9tal4mM.mp4",
"adj": {
"b": "anyway",
"a": "biologically."
@@ -25394,7 +24953,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BnPz37ecO58.mp4",
"adj": {
"b": "institute",
"a": "the"
@@ -25455,7 +25013,6 @@
"why": "",
"wordIn": "not",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/DMAa1fld4Rc.mp4",
"adj": {
"b": "we're",
"a": "you"
@@ -25495,7 +25052,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/EN9H7RVxZtw.mp4",
"adj": {
"b": "",
"a": "at"
@@ -25519,7 +25075,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "can't",
"a": "show"
@@ -25635,7 +25190,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FjjL-sGykMM.mp4",
"adj": {
"b": "was",
"a": "above"
@@ -25684,7 +25238,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/GdMSdVvQvtc.mp4",
"adj": {
"b": "",
"a": "you"
@@ -25719,7 +25272,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 224ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HPI5vs1-9yk.mp4",
"adj": {
"b": "",
"a": ""
@@ -25743,7 +25295,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HTRY0dNjZIo.mp4",
"adj": {
"b": "tweet",
"a": "he's"
@@ -25773,7 +25324,6 @@
"why": "next word \"We\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Hh71Xe7XK6k.mp4",
"adj": {
"b": "gift.",
"a": "We"
@@ -25835,7 +25385,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/JKozJKyP4Iw.mp4",
"adj": {
"b": "",
"a": ""
@@ -25904,7 +25453,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Key66cOeiEo.mp4",
"adj": {
"b": "",
"a": "to"
@@ -25928,7 +25476,6 @@
"why": "",
"wordIn": "apparently",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KtZaCYPSunM.mp4",
"adj": {
"b": "where",
"a": ""
@@ -25958,7 +25505,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KyzBZ19iRaU.mp4",
"adj": {
"b": "Johnson.",
"a": "It"
@@ -25982,7 +25528,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 299ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KyzBZ19iRaU.mp4",
"adj": {
"b": "somebody",
"a": "has"
@@ -26055,7 +25600,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LLnW_XgyHg0.mp4",
"adj": {
"b": "Tim",
"a": ""
@@ -26079,7 +25623,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "do.",
"a": "And"
@@ -26189,7 +25732,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Mf1hEHazJaE.mp4",
"adj": {
"b": "",
"a": ""
@@ -26231,7 +25773,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 499ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OBidX_oE7zM.mp4",
"adj": {
"b": "",
"a": "there's"
@@ -26305,7 +25846,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "buying.",
"a": ""
@@ -26329,7 +25869,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "",
"a": "the"
@@ -26364,7 +25903,6 @@
"why": "next word \"on\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "",
"a": "on"
@@ -26399,7 +25937,6 @@
"why": "next word \"having\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "than",
"a": "having"
@@ -26477,7 +26014,6 @@
"why": "",
"wordIn": "And",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "",
"a": "while"
@@ -26512,7 +26048,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QFUX3tJJpiI.mp4",
"adj": {
"b": "the",
"a": ""
@@ -26600,7 +26135,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/RT0mKx_hTv4.mp4",
"adj": {
"b": "",
"a": "incredible"
@@ -26641,7 +26175,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "",
"a": "This"
@@ -26665,7 +26198,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "",
"a": "and"
@@ -26709,7 +26241,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TiCFVgAYxBw.mp4",
"adj": {
"b": "Republican.",
"a": "And"
@@ -26810,7 +26341,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 274ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/V4k0ziHTGog.mp4",
"adj": {
"b": "that.",
"a": "I"
@@ -26834,7 +26364,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WJYyCfbjv8s.mp4",
"adj": {
"b": "okay",
"a": "and"
@@ -26915,7 +26444,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Xa9jjt2XF24.mp4",
"adj": {
"b": "Walmart.",
"a": ""
@@ -26939,7 +26467,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Y7-19Q9gmfg.mp4",
"adj": {
"b": "",
"a": "a"
@@ -26999,7 +26526,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "",
"a": "advocates."
@@ -27023,7 +26549,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 274ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "",
"a": ""
@@ -27047,7 +26572,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZeqS0DvnorA.mp4",
"adj": {
"b": "",
"a": "demonetization"
@@ -27071,7 +26595,6 @@
"why": "",
"wordIn": "Nukem",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Zfcmx4_Qxxg.mp4",
"adj": {
"b": "Duke",
"a": ""
@@ -27132,7 +26655,6 @@
"why": "next word \"there\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bbVoMOzCSs0.mp4",
"adj": {
"b": "air",
"a": "there"
@@ -27211,7 +26733,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "my",
"a": ""
@@ -27235,7 +26756,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 329ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "",
"a": ""
@@ -27259,7 +26779,6 @@
"why": "next word \"yeah\" starts here",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cAvjCjQqOac.mp4",
"adj": {
"b": "so",
"a": "yeah"
@@ -27471,7 +26990,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "the",
"a": ""
@@ -27512,7 +27030,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/iYbUqMyCOLU.mp4",
"adj": {
"b": "",
"a": "you"
@@ -27667,7 +27184,6 @@
"why": "",
"wordIn": "as",
"risk": "word \"as\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lraibG7Iu5w.mp4",
"adj": {
"b": "",
"a": "stores"
@@ -27719,7 +27235,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6CNFodmbWw.mp4",
"adj": {
"b": "",
"a": "look"
@@ -27807,7 +27322,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m_aKA-Khmck.mp4",
"adj": {
"b": "",
"a": "if"
@@ -27856,7 +27370,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mkyBXrJhDrQ.mp4",
"adj": {
"b": "apparently",
"a": "this"
@@ -27919,7 +27432,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "this video's clips sit -413 cents from the corpus (81Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7X-TBkGi_5s.mp4",
"adj": {
"b": "",
"a": ""
@@ -28207,7 +27719,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 399ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3gORoyt2mC4.mp4",
"adj": {
"b": "",
"a": ""
@@ -28231,7 +27742,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 289ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "",
"a": "but"
@@ -28349,7 +27859,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Y7-19Q9gmfg.mp4",
"adj": {
"b": "",
"a": ""
@@ -28373,7 +27882,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 274ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8hsc6aSGdk.mp4",
"adj": {
"b": "",
"a": ""
@@ -28397,7 +27905,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 474ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TYPV_Ej1ND4.mp4",
"adj": {
"b": "",
"a": ""
@@ -28421,7 +27928,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 389ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "And",
"a": "yeah,"
@@ -28539,7 +28045,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 359ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "persona",
"a": "you"
@@ -28819,7 +28324,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lC6PCsQT1Qo.mp4",
"adj": {
"b": "",
"a": ""
@@ -28865,7 +28369,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "window moved 304ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-lSWX5qachE.mp4",
"adj": {
"b": "hire",
"a": ""
@@ -28911,7 +28414,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 374ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "like",
"a": "the"
@@ -28935,7 +28437,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 324ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "on",
"a": "I"
@@ -29045,7 +28546,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/N78crpC9o1g.mp4",
"adj": {
"b": "",
"a": ""
@@ -29135,7 +28635,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "",
"a": ""
@@ -29159,7 +28658,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "window moved 339ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/pNP4B0PJk-Q.mp4",
"adj": {
"b": "me",
"a": "because"
@@ -29249,7 +28747,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 354ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "",
"a": ""
@@ -29383,7 +28880,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/A9Zg-xqDIcM.mp4",
"adj": {
"b": "",
"a": ""
@@ -29517,7 +29013,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 419ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rJ-V9bd9La0.mp4",
"adj": {
"b": "damaging",
"a": "and"
@@ -29541,7 +29036,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ts76T7tJv-U.mp4",
"adj": {
"b": "",
"a": ""
@@ -29609,7 +29103,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fp_GOiz5MV0.mp4",
"adj": {
"b": "",
"a": ""
@@ -29721,7 +29214,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 289ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KyzBZ19iRaU.mp4",
"adj": {
"b": "to",
"a": ""
@@ -29811,7 +29303,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 249ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "",
"a": "and"
@@ -29841,7 +29332,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "",
"a": "find"
@@ -29865,7 +29355,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 344ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8hsc6aSGdk.mp4",
"adj": {
"b": "leftoids",
"a": ""
@@ -29889,7 +29378,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "window moved 379ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "",
"a": ""
@@ -29935,7 +29423,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TYPV_Ej1ND4.mp4",
"adj": {
"b": "and",
"a": "it"
@@ -29987,7 +29474,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8RiHnmAGy8U.mp4",
"adj": {
"b": "",
"a": "you"
@@ -30077,7 +29563,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 424ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rJ-V9bd9La0.mp4",
"adj": {
"b": "",
"a": ""
@@ -30167,7 +29652,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 364ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "",
"a": ""
@@ -30213,7 +29697,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 239ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mGcQRQ3wqsk.mp4",
"adj": {
"b": "charged",
"a": "with"
@@ -30237,7 +29720,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 424ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "",
"a": "you"
@@ -30327,7 +29809,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 389ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "",
"a": ""
@@ -30351,7 +29832,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/iAwVWRp5Ayg.mp4",
"adj": {
"b": "",
"a": "sorry"
@@ -30485,7 +29965,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 379ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fjLMZwGB4Ds.mp4",
"adj": {
"b": "",
"a": "they're"
@@ -30581,7 +30060,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 329ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/78A048gOuq8.mp4",
"adj": {
"b": "on",
"a": "you"
@@ -30619,7 +30097,6 @@
"why": "next word \"let\" starts here",
"wordIn": "",
"risk": "word \"let\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "",
"a": "let"
@@ -30698,7 +30175,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "",
"a": "around"
@@ -30736,7 +30212,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "images",
"a": "between"
@@ -30782,7 +30257,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 239ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/78A048gOuq8.mp4",
"adj": {
"b": "memberships",
"a": "after"
@@ -30870,7 +30344,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 549ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "and",
"a": ""
@@ -30916,7 +30389,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 324ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nzOE5f6VPqY.mp4",
"adj": {
"b": "well",
"a": ""
@@ -30984,7 +30456,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "window moved 319ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "know",
"a": ""
@@ -31044,7 +30515,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FWv906SubiM.mp4",
"adj": {
"b": "",
"a": ""
@@ -31126,7 +30596,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 329ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "",
"a": "they"
@@ -31394,7 +30863,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cPKtKx3evdM.mp4",
"adj": {
"b": "below",
"a": "to"
@@ -31432,7 +30900,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 354ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fp_GOiz5MV0.mp4",
"adj": {
"b": "fine.",
"a": ""
@@ -31500,7 +30967,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hU43s57gcnI.mp4",
"adj": {
"b": "",
"a": ""
@@ -31524,7 +30990,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "them",
"a": "now"
@@ -31570,7 +31035,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 314ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mOF790Bh6I0.mp4",
"adj": {
"b": "joke,",
"a": ""
@@ -31696,7 +31160,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 329ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mwtbQffMRic.mp4",
"adj": {
"b": "Antifa",
"a": "check"
@@ -31738,7 +31201,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 304ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6CNFodmbWw.mp4",
"adj": {
"b": "that",
"a": ""
@@ -31768,7 +31230,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 344ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "",
"a": ""
@@ -31836,7 +31297,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 444ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-qfcXnIXn8Y.mp4",
"adj": {
"b": "",
"a": "my"
@@ -31866,7 +31326,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 314ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -31890,7 +31349,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1Y0204hKq2M.mp4",
"adj": {
"b": "",
"a": "and"
@@ -31914,7 +31372,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 324ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "",
"a": "So"
@@ -31938,7 +31395,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 314ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/78A048gOuq8.mp4",
"adj": {
"b": "",
"a": "And"
@@ -31968,7 +31424,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 299ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "",
"a": ""
@@ -31992,7 +31447,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 449ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -32104,7 +31558,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 354ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xulBQwriW5E.mp4",
"adj": {
"b": "Pentagon,",
"a": "you"
@@ -32128,7 +31581,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xEYsJorlvkw.mp4",
"adj": {
"b": "output",
"a": "I"
@@ -32152,7 +31604,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 329ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Hh71Xe7XK6k.mp4",
"adj": {
"b": "too.",
"a": ""
@@ -32216,7 +31667,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/67Qd5J47Y84.mp4",
"adj": {
"b": "",
"a": "people"
@@ -32298,7 +31748,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 309ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "this",
"a": "is"
@@ -32432,7 +31881,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "",
"a": ""
@@ -32456,7 +31904,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 344ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "",
"a": ""
@@ -32480,7 +31927,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "window moved 398ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/pNP4B0PJk-Q.mp4",
"adj": {
"b": "",
"a": ""
@@ -32504,7 +31950,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QeSmSLqLR6w.mp4",
"adj": {
"b": "",
"a": "that"
@@ -32570,7 +32015,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 314ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3gORoyt2mC4.mp4",
"adj": {
"b": "So",
"a": "at"
@@ -32608,7 +32052,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -32654,7 +32097,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 244ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "",
"a": "you"
@@ -32689,7 +32131,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 319ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/axL33DS5KeA.mp4",
"adj": {
"b": "know",
"a": "you"
@@ -32735,7 +32176,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "window moved 289ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "",
"a": ""
@@ -32803,7 +32243,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 309ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "again.",
"a": "Let's"
@@ -32892,7 +32331,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 309ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bbVoMOzCSs0.mp4",
"adj": {
"b": "dude",
"a": "and"
@@ -32944,7 +32382,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "",
"a": ""
@@ -32968,7 +32405,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 444ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/acPLSEVKyqI.mp4",
"adj": {
"b": "",
"a": ""
@@ -32992,7 +32428,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "",
"a": "the"
@@ -33080,7 +32515,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 384ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "",
"a": ""
@@ -33144,7 +32578,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/l55Ad1gO5ww.mp4",
"adj": {
"b": "",
"a": "and"
@@ -33222,7 +32655,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 389ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8Sid61t1Zs.mp4",
"adj": {
"b": "comedian",
"a": ""
@@ -33252,7 +32684,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 469ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Nb5iswtRmVM.mp4",
"adj": {
"b": "and",
"a": ""
@@ -33320,7 +32751,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vimidN7iVN0.mp4",
"adj": {
"b": "around",
"a": "he"
@@ -33344,7 +32774,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 369ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mGcQRQ3wqsk.mp4",
"adj": {
"b": "and",
"a": ""
@@ -33468,7 +32897,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 319ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "say",
"a": ""
@@ -33492,7 +32920,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 289ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "",
"a": ""
@@ -33534,7 +32961,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CE3O2H43_tc.mp4",
"adj": {
"b": "Mexico",
"a": "yeah"
@@ -33580,7 +33006,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 344ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Hh71Xe7XK6k.mp4",
"adj": {
"b": "",
"a": ""
@@ -33626,7 +33051,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -33694,7 +33118,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/g-OStFPhrlk.mp4",
"adj": {
"b": "",
"a": "It"
@@ -33740,7 +33163,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 284ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m_B_3i9yKX0.mp4",
"adj": {
"b": "headphones.",
"a": "This"
@@ -33796,7 +33218,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KyzBZ19iRaU.mp4",
"adj": {
"b": "",
"a": "He"
@@ -33820,7 +33241,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "",
"a": ""
@@ -33866,7 +33286,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 304ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KyzBZ19iRaU.mp4",
"adj": {
"b": "",
"a": ""
@@ -33911,7 +33330,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "window moved 334ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LLnW_XgyHg0.mp4",
"adj": {
"b": "check",
"a": ""
@@ -33967,7 +33385,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/MXHk10OCoow.mp4",
"adj": {
"b": "and",
"a": "America"
@@ -33991,7 +33408,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "window moved 289ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "",
"a": ""
@@ -34015,7 +33431,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 289ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fp_GOiz5MV0.mp4",
"adj": {
"b": "",
"a": ""
@@ -34039,7 +33454,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 244ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KtZaCYPSunM.mp4",
"adj": {
"b": "that",
"a": "I'm"
@@ -34069,7 +33483,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 369ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/A9Zg-xqDIcM.mp4",
"adj": {
"b": "",
"a": "you"
@@ -34181,7 +33594,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "was",
"a": "this"
@@ -34239,7 +33651,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/GdMSdVvQvtc.mp4",
"adj": {
"b": "know",
"a": ""
@@ -34269,7 +33680,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 324ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "briefed",
"a": ""
@@ -34315,7 +33725,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 434ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rATxtI8FhxE.mp4",
"adj": {
"b": "time",
"a": ""
@@ -34383,7 +33792,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cPKtKx3evdM.mp4",
"adj": {
"b": "",
"a": "They"
@@ -34429,7 +33837,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yjHHiRhtMvQ.mp4",
"adj": {
"b": "",
"a": "you"
@@ -34459,7 +33866,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 334ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "",
"a": ""
@@ -34505,7 +33911,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 329ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/AMdAOwrYmAI.mp4",
"adj": {
"b": "",
"a": ""
@@ -34529,7 +33934,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 349ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "",
"a": ""
@@ -34553,7 +33957,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 299ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ylmhShFD48Q.mp4",
"adj": {
"b": "",
"a": "If"
@@ -34689,7 +34092,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 254ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/knmVtqTW0MQ.mp4",
"adj": {
"b": "strikes",
"a": ""
@@ -34713,7 +34115,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 254ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZEwkRomA9pU.mp4",
"adj": {
"b": "",
"a": ""
@@ -34737,7 +34138,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 299ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "left",
"a": "terror."
@@ -34761,7 +34161,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8RiHnmAGy8U.mp4",
"adj": {
"b": "",
"a": "Here"
@@ -34785,7 +34184,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 299ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/acPLSEVKyqI.mp4",
"adj": {
"b": "",
"a": "So"
@@ -34831,7 +34229,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 319ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Nb5iswtRmVM.mp4",
"adj": {
"b": "",
"a": "accused"
@@ -34891,7 +34288,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8ez7u2StoWI.mp4",
"adj": {
"b": "and",
"a": ""
@@ -34987,7 +34383,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/f5sLSteRRTQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -35055,7 +34450,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8hsc6aSGdk.mp4",
"adj": {
"b": "",
"a": ""
@@ -35259,7 +34653,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/axL33DS5KeA.mp4",
"adj": {
"b": "",
"a": "they're"
@@ -35325,7 +34718,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 384ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "Providence",
"a": "Rhode"
@@ -35405,7 +34797,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/K9rXg1newxo.mp4",
"adj": {
"b": "",
"a": "and"
@@ -35519,7 +34910,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 574ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rATxtI8FhxE.mp4",
"adj": {
"b": "",
"a": ""
@@ -35543,7 +34933,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/78A048gOuq8.mp4",
"adj": {
"b": "YouTube.",
"a": "So"
@@ -35567,7 +34956,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 349ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CE3O2H43_tc.mp4",
"adj": {
"b": "",
"a": ""
@@ -35591,7 +34979,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 309ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "subpoena",
"a": "to"
@@ -35614,7 +35001,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUoDtOdIza4.mp4",
"adj": {
"b": "",
"a": "from"
@@ -35660,7 +35046,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OBidX_oE7zM.mp4",
"adj": {
"b": "",
"a": ""
@@ -35684,7 +35069,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 314ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/6A1JPaiqJNY.mp4",
"adj": {
"b": "",
"a": "the"
@@ -35708,7 +35092,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 224ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "enforcement",
"a": "again"
@@ -35804,7 +35187,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "work.",
"a": "Even"
@@ -35864,7 +35246,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6CNFodmbWw.mp4",
"adj": {
"b": "",
"a": "but"
@@ -35910,7 +35291,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "long.",
"a": "But"
@@ -35940,7 +35320,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 289ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rJ-V9bd9La0.mp4",
"adj": {
"b": "people,",
"a": "you"
@@ -36036,7 +35415,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 274ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kfzmLf09O0w.mp4",
"adj": {
"b": "",
"a": "you"
@@ -36066,7 +35444,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-qfcXnIXn8Y.mp4",
"adj": {
"b": "",
"a": ""
@@ -36090,7 +35467,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mwtbQffMRic.mp4",
"adj": {
"b": "me",
"a": "and"
@@ -36208,7 +35584,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 254ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cN5k-3sfBdA.mp4",
"adj": {
"b": "",
"a": "they"
@@ -36232,7 +35607,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FeUW-Vw02ZQ.mp4",
"adj": {
"b": "by",
"a": ""
@@ -36284,7 +35658,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 304ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/N78crpC9o1g.mp4",
"adj": {
"b": "And",
"a": "we've"
@@ -36336,7 +35709,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 254ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QFUX3tJJpiI.mp4",
"adj": {
"b": "",
"a": "You"
@@ -36380,7 +35752,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 359ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-pKXHyeyRhQ.mp4",
"adj": {
"b": "and",
"a": "actually"
@@ -36432,7 +35803,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0A4dIN1dKaY.mp4",
"adj": {
"b": "",
"a": "you"
@@ -36484,7 +35854,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 309ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "",
"a": "Okay,"
@@ -36562,7 +35931,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 234ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UqCmkStyKow.mp4",
"adj": {
"b": "",
"a": "you"
@@ -36664,7 +36032,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 284ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -36702,7 +36069,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 244ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Zfcmx4_Qxxg.mp4",
"adj": {
"b": "Nukem",
"a": "you"
@@ -36732,7 +36098,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "",
"a": "Two"
@@ -36762,7 +36127,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 284ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "computer",
"a": ""
@@ -36786,7 +36150,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 214ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/N78crpC9o1g.mp4",
"adj": {
"b": "",
"a": ""
@@ -36838,7 +36201,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 304ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "",
"a": "with"
@@ -36905,7 +36267,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BKUP9tal4mM.mp4",
"adj": {
"b": "day",
"a": "I"
@@ -36989,7 +36350,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 214ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Nb5iswtRmVM.mp4",
"adj": {
"b": "know",
"a": "participate"
@@ -37027,7 +36387,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "arrested",
"a": ""
@@ -37065,7 +36424,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 344ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-lSWX5qachE.mp4",
"adj": {
"b": "",
"a": ""
@@ -37089,7 +36447,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 264ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cQL0Rja_Qik.mp4",
"adj": {
"b": "",
"a": "you"
@@ -37127,7 +36484,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 319ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/22DisumpVqo.mp4",
"adj": {
"b": "",
"a": "if"
@@ -37173,7 +36529,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 214ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Xa9jjt2XF24.mp4",
"adj": {
"b": "and",
"a": ""
@@ -37197,7 +36552,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7zRNtT2ygCc.mp4",
"adj": {
"b": "here",
"a": "I'm"
@@ -37221,7 +36575,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Jzw4zVQKEA4.mp4",
"adj": {
"b": "",
"a": "calling"
@@ -37269,7 +36622,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUSzqEViwBw.mp4",
"adj": {
"b": "",
"a": "something"
@@ -37293,7 +36645,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 249ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ep_qtW2K3BY.mp4",
"adj": {
"b": "driver",
"a": "and"
@@ -37342,7 +36693,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/e0d6EzTEt7Y.mp4",
"adj": {
"b": "Doyle",
"a": "count"
@@ -37406,7 +36756,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 329ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8RiHnmAGy8U.mp4",
"adj": {
"b": "court",
"a": "other"
@@ -37430,7 +36779,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 309ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Zfcmx4_Qxxg.mp4",
"adj": {
"b": "",
"a": ""
@@ -37553,7 +36901,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hykdhazn8ds.mp4",
"adj": {
"b": "",
"a": ""
@@ -37577,7 +36924,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 254ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "and",
"a": "I'll"
@@ -37601,7 +36947,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 379ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/DMAa1fld4Rc.mp4",
"adj": {
"b": "",
"a": "we"
@@ -37625,7 +36970,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nzOE5f6VPqY.mp4",
"adj": {
"b": "got,",
"a": "you"
@@ -37663,7 +37007,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 289ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yGvQfTultRw.mp4",
"adj": {
"b": "clickbait",
"a": "I"
@@ -37687,7 +37030,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 239ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8ez7u2StoWI.mp4",
"adj": {
"b": "",
"a": "And"
@@ -37781,7 +37123,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 209ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "",
"a": "I"
@@ -37869,7 +37210,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xulBQwriW5E.mp4",
"adj": {
"b": "",
"a": "By"
@@ -37929,7 +37269,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 314ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "",
"a": ""
@@ -37975,7 +37314,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 239ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FiLs5ovIfts.mp4",
"adj": {
"b": "lot",
"a": "and"
@@ -37999,7 +37337,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 304ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/v-MG7ofJ6X4.mp4",
"adj": {
"b": "theirs",
"a": "Don"
@@ -38045,7 +37382,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 284ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/i29P5pQc50A.mp4",
"adj": {
"b": "",
"a": "And"
@@ -38069,7 +37405,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WJYyCfbjv8s.mp4",
"adj": {
"b": "so",
"a": "it"
@@ -38181,7 +37516,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 229ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/axL33DS5KeA.mp4",
"adj": {
"b": "that",
"a": "saying"
@@ -38211,7 +37545,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 224ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QeSmSLqLR6w.mp4",
"adj": {
"b": "",
"a": "he"
@@ -38235,7 +37568,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 209ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUoDtOdIza4.mp4",
"adj": {
"b": "",
"a": "bobs"
@@ -38259,7 +37591,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 324ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/A9Zg-xqDIcM.mp4",
"adj": {
"b": "",
"a": "that"
@@ -38297,7 +37628,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Jg2PKHgDvyw.mp4",
"adj": {
"b": "know",
"a": "and"
@@ -38399,7 +37729,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rATxtI8FhxE.mp4",
"adj": {
"b": "there",
"a": "you"
@@ -38585,7 +37914,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/GdMSdVvQvtc.mp4",
"adj": {
"b": "",
"a": "police"
@@ -38681,7 +38009,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9zEisQcriiM.mp4",
"adj": {
"b": "",
"a": "I"
@@ -38725,7 +38052,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 244ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CE3O2H43_tc.mp4",
"adj": {
"b": "",
"a": "Now"
@@ -38755,7 +38081,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 309ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "student",
"a": "you"
@@ -38778,7 +38103,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 224ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Hh71Xe7XK6k.mp4",
"adj": {
"b": "overdosed",
"a": "and"
@@ -38896,7 +38220,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "admonish",
"a": "a"
@@ -38934,7 +38257,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "",
"a": ""
@@ -38958,7 +38280,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lraibG7Iu5w.mp4",
"adj": {
"b": "said",
"a": ""
@@ -38988,7 +38309,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/N78crpC9o1g.mp4",
"adj": {
"b": "half",
"a": "but"
@@ -39018,7 +38338,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 214ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/JKozJKyP4Iw.mp4",
"adj": {
"b": "and",
"a": "I'll"
@@ -39062,7 +38381,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 314ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xEYsJorlvkw.mp4",
"adj": {
"b": "to",
"a": ""
@@ -39303,7 +38621,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 254ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "",
"a": ""
@@ -39327,7 +38644,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "",
"a": ""
@@ -39463,7 +38779,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 249ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6CNFodmbWw.mp4",
"adj": {
"b": "this",
"a": ""
@@ -39973,7 +39288,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Oo6dXSEh65Y.mp4",
"adj": {
"b": "night",
"a": ""
@@ -40014,7 +39328,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "a clip here was judged \"other speaker, not Jer\"",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HPIHN8nxCM8.mp4",
"adj": {
"b": "church",
"a": "you"
@@ -40078,7 +39391,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "know",
"a": ""
@@ -40188,7 +39500,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +414 cents from the corpus (130Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/EN9H7RVxZtw.mp4",
"adj": {
"b": "absent",
"a": "we"
@@ -40476,7 +39787,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 239ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/X70q2QJ_VTc.mp4",
"adj": {
"b": "",
"a": ""
@@ -40587,7 +39897,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "word \"and\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/K9rXg1newxo.mp4",
"adj": {
"b": "",
"a": ""
@@ -40738,7 +40047,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 224ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "know,",
"a": ""
@@ -40957,7 +40265,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "a clip here was judged \"other speaker, not Jer\"",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HPIHN8nxCM8.mp4",
"adj": {
"b": "people's",
"a": "constitutionally"
@@ -40998,7 +40305,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 319ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "",
"a": ""
@@ -41261,7 +40567,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 269ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xulBQwriW5E.mp4",
"adj": {
"b": "and",
"a": "he"
@@ -41375,7 +40680,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 294ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/V4k0ziHTGog.mp4",
"adj": {
"b": "beautiful,",
"a": ""
@@ -41850,7 +41154,6 @@
"why": "unvoiced tail",
"wordIn": "And",
"risk": "word \"And\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "it.",
"a": ""
@@ -41914,7 +41217,6 @@
"why": "trailing noise burst",
"wordIn": "and",
"risk": "word \"and\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/f5sLSteRRTQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -42137,7 +41439,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 254ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "",
"a": ""
@@ -42161,7 +41462,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 214ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/pNP4B0PJk-Q.mp4",
"adj": {
"b": "",
"a": "you"
@@ -42409,7 +41709,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 234ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/l8y24jC9JUM.mp4",
"adj": {
"b": "",
"a": "that's"
@@ -42646,7 +41945,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 239ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yjHHiRhtMvQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -42879,7 +42177,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZeqS0DvnorA.mp4",
"adj": {
"b": "",
"a": ""
@@ -42920,7 +42217,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 219ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/P4JErOtgVuA.mp4",
"adj": {
"b": "story",
"a": "with"
@@ -42978,7 +42274,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 284ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mOF790Bh6I0.mp4",
"adj": {
"b": "",
"a": ""
@@ -43019,7 +42314,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 239ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/axL33DS5KeA.mp4",
"adj": {
"b": "common",
"a": ""
@@ -43249,7 +42543,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 284ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kfzmLf09O0w.mp4",
"adj": {
"b": "up,",
"a": "you"
@@ -43279,7 +42572,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 249ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Xa9jjt2XF24.mp4",
"adj": {
"b": "",
"a": ""
@@ -43407,7 +42699,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 259ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/F_atEiYGdSk.mp4",
"adj": {
"b": "channel",
"a": "and"
@@ -43437,7 +42728,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +414 cents from the corpus (130Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/EN9H7RVxZtw.mp4",
"adj": {
"b": "",
"a": "I"
@@ -43495,7 +42785,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 234ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yjHHiRhtMvQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -43726,7 +43015,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 214ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bbVoMOzCSs0.mp4",
"adj": {
"b": "undeniable",
"a": "and"
@@ -44129,7 +43417,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 249ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "baby",
"a": "he"
@@ -44397,7 +43684,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 209ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "",
"a": ""
@@ -44507,7 +43793,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "",
"a": ""
@@ -45140,7 +44425,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 389ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mwtbQffMRic.mp4",
"adj": {
"b": "Chandler's",
"a": ""
@@ -45164,7 +44448,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3gORoyt2mC4.mp4",
"adj": {
"b": "School",
"a": ""
@@ -45206,7 +44489,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +240 cents from the corpus (118Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cwMkQSs6OeM.mp4",
"adj": {
"b": "",
"a": ""
@@ -45265,7 +44547,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 204ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "says",
"a": ""
@@ -45360,7 +44641,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 234ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8hsc6aSGdk.mp4",
"adj": {
"b": "life",
"a": ""
@@ -45418,7 +44698,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 304ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ylmhShFD48Q.mp4",
"adj": {
"b": "",
"a": ""
@@ -45567,7 +44846,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 279ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/22DisumpVqo.mp4",
"adj": {
"b": "",
"a": "and"
@@ -45591,7 +44869,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 239ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "",
"a": ""
@@ -45735,7 +45012,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 249ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/knmVtqTW0MQ.mp4",
"adj": {
"b": "that",
"a": "this"
@@ -45850,7 +45126,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 289ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUoDtOdIza4.mp4",
"adj": {
"b": "good.",
"a": ""
@@ -46188,7 +45463,6 @@
"why": "next word \"And\" starts here",
"wordIn": "",
"risk": "word \"And\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "",
"a": "And"
@@ -46321,7 +45595,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 209ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lraibG7Iu5w.mp4",
"adj": {
"b": "know",
"a": ""
@@ -46644,7 +45917,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +240 cents from the corpus (118Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cwMkQSs6OeM.mp4",
"adj": {
"b": "",
"a": "all"
@@ -46668,7 +45940,6 @@
"why": "next word \"the\" starts here",
"wordIn": "read",
"risk": "word \"read\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "can",
"a": "the"
@@ -46726,7 +45997,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 284ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "Trump",
"a": ""
@@ -46750,7 +46020,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 344ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "that",
"a": ""
@@ -46825,7 +46094,6 @@
"why": "next word \"which\" starts here",
"wordIn": "",
"risk": "word \"which\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Key66cOeiEo.mp4",
"adj": {
"b": "",
"a": "which"
@@ -47053,7 +46321,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +380 cents from the corpus (128Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Api44x4HVGs.mp4",
"adj": {
"b": "",
"a": "along"
@@ -47441,7 +46708,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 234ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ylmhShFD48Q.mp4",
"adj": {
"b": "",
"a": ""
@@ -47517,7 +46783,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 209ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/5M-1yCq23hY.mp4",
"adj": {
"b": "way.",
"a": "And"
@@ -47593,7 +46858,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "have",
"a": "but"
@@ -47804,7 +47068,6 @@
"why": "",
"wordIn": "",
"risk": "window moved 309ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "",
"a": ""
@@ -48285,7 +47548,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "window moved 324ms since you judged it",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "too",
"a": ""
@@ -48471,7 +47733,6 @@
"why": "next word \"there\" starts here",
"wordIn": "",
"risk": "word \"there\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HTRY0dNjZIo.mp4",
"adj": {
"b": "here",
"a": "there"
@@ -48506,7 +47767,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HPI5vs1-9yk.mp4",
"adj": {
"b": "most",
"a": ""
@@ -48530,7 +47790,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3gORoyt2mC4.mp4",
"adj": {
"b": "",
"a": "one"
@@ -48554,7 +47813,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QFUX3tJJpiI.mp4",
"adj": {
"b": "",
"a": "Also"
@@ -48584,7 +47842,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/V4k0ziHTGog.mp4",
"adj": {
"b": "",
"a": "have"
@@ -48608,7 +47865,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fjLMZwGB4Ds.mp4",
"adj": {
"b": "and",
"a": "you"
@@ -48638,7 +47894,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hykdhazn8ds.mp4",
"adj": {
"b": "man",
"a": "was"
@@ -48662,7 +47917,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ylmhShFD48Q.mp4",
"adj": {
"b": "",
"a": ""
@@ -48686,7 +47940,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "",
"a": ""
@@ -48710,7 +47963,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/jprDE87Ec58.mp4",
"adj": {
"b": "teachers",
"a": "by"
@@ -48734,7 +47986,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dhOXEakOpaM.mp4",
"adj": {
"b": "for",
"a": "you"
@@ -48764,7 +48015,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ibJe2PlNOS8.mp4",
"adj": {
"b": "",
"a": "now"
@@ -48794,7 +48044,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1UgCLhoVIfs.mp4",
"adj": {
"b": "think",
"a": "I"
@@ -48824,7 +48073,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kfzmLf09O0w.mp4",
"adj": {
"b": "up",
"a": "on"
@@ -48854,7 +48102,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "given",
"a": "their"
@@ -48889,7 +48136,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/P4JErOtgVuA.mp4",
"adj": {
"b": "",
"a": "as"
@@ -48913,7 +48159,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "does",
"a": "look"
@@ -48937,7 +48182,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FiLs5ovIfts.mp4",
"adj": {
"b": "",
"a": ""
@@ -48961,7 +48205,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cQL0Rja_Qik.mp4",
"adj": {
"b": "",
"a": "these"
@@ -48985,7 +48228,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/AMdAOwrYmAI.mp4",
"adj": {
"b": "",
"a": "you"
@@ -49009,7 +48251,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/V4k0ziHTGog.mp4",
"adj": {
"b": "",
"a": ""
@@ -49033,7 +48274,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WqAu8gLaW-4.mp4",
"adj": {
"b": "",
"a": ""
@@ -49057,7 +48297,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m_aKA-Khmck.mp4",
"adj": {
"b": "",
"a": ""
@@ -49081,7 +48320,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/axL33DS5KeA.mp4",
"adj": {
"b": "",
"a": "people"
@@ -49105,7 +48343,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yjHHiRhtMvQ.mp4",
"adj": {
"b": "cinnabon",
"a": "and"
@@ -49129,7 +48366,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WqAu8gLaW-4.mp4",
"adj": {
"b": "",
"a": "the"
@@ -49159,7 +48395,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/zoWmEz419Kw.mp4",
"adj": {
"b": "Scrum",
"a": "put"
@@ -49183,7 +48418,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/pNP4B0PJk-Q.mp4",
"adj": {
"b": "",
"a": "You"
@@ -49207,7 +48441,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3gORoyt2mC4.mp4",
"adj": {
"b": "",
"a": "Facebook"
@@ -49237,7 +48470,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9SgVb3vhRVo.mp4",
"adj": {
"b": "",
"a": "mostly"
@@ -49267,7 +48499,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Me0fryGe2X8.mp4",
"adj": {
"b": "like",
"a": ""
@@ -49291,7 +48522,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "",
"a": "live"
@@ -49315,7 +48545,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/MXHk10OCoow.mp4",
"adj": {
"b": "is",
"a": "an"
@@ -49339,7 +48568,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "like",
"a": ""
@@ -49363,7 +48591,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LP94fw5JtYk.mp4",
"adj": {
"b": "know",
"a": "scientist"
@@ -49387,7 +48614,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "",
"a": ""
@@ -49411,7 +48637,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kMDV9dG0nVY.mp4",
"adj": {
"b": "",
"a": "covering"
@@ -49441,7 +48666,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QqIWnAhGwfw.mp4",
"adj": {
"b": "Google",
"a": ""
@@ -49471,7 +48695,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FqLw57yn4fs.mp4",
"adj": {
"b": "",
"a": "but"
@@ -49501,7 +48724,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9YQfjW1uasM.mp4",
"adj": {
"b": "know,",
"a": "I"
@@ -49531,7 +48753,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/acPLSEVKyqI.mp4",
"adj": {
"b": "and",
"a": "you"
@@ -49571,7 +48792,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/4RFFlS0oMEM.mp4",
"adj": {
"b": "arrested",
"a": "this"
@@ -49595,7 +48815,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nrKQ-WoQ_-Y.mp4",
"adj": {
"b": "",
"a": "I"
@@ -49625,7 +48844,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cN5k-3sfBdA.mp4",
"adj": {
"b": "",
"a": "I"
@@ -49649,7 +48867,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FeUW-Vw02ZQ.mp4",
"adj": {
"b": "been",
"a": "but"
@@ -49679,7 +48896,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "",
"a": "There's"
@@ -49709,7 +48925,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8RiHnmAGy8U.mp4",
"adj": {
"b": "got",
"a": "the"
@@ -49744,7 +48959,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "",
"a": ""
@@ -50376,7 +49590,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Tvpplznw9Nw.mp4",
"adj": {
"b": "",
"a": "the"
@@ -50406,7 +49619,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kfzmLf09O0w.mp4",
"adj": {
"b": "",
"a": "dealing"
@@ -50436,7 +49648,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Jg2PKHgDvyw.mp4",
"adj": {
"b": "",
"a": "I"
@@ -50466,7 +49677,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xaWnBmX4AjY.mp4",
"adj": {
"b": "and",
"a": "AI"
@@ -50490,7 +49700,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "and",
"a": "you"
@@ -50520,7 +49729,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Mf1hEHazJaE.mp4",
"adj": {
"b": "channel",
"a": "that"
@@ -50544,7 +49752,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/aeC-_9DQWM4.mp4",
"adj": {
"b": "",
"a": "She"
@@ -50568,7 +49775,6 @@
"why": "next word \"I\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8ez7u2StoWI.mp4",
"adj": {
"b": "",
"a": "I"
@@ -50598,7 +49804,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/iYbUqMyCOLU.mp4",
"adj": {
"b": "",
"a": "that"
@@ -50628,7 +49833,6 @@
"why": "",
"wordIn": "who",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hGqSbmfHO7o.mp4",
"adj": {
"b": "",
"a": ""
@@ -50658,7 +49862,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QFUX3tJJpiI.mp4",
"adj": {
"b": "was",
"a": ""
@@ -50688,7 +49891,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/93DFKk1eSoM.mp4",
"adj": {
"b": "saw",
"a": "that"
@@ -50718,7 +49920,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "",
"a": "in"
@@ -50742,7 +49943,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fjLMZwGB4Ds.mp4",
"adj": {
"b": "weekend",
"a": "once"
@@ -50766,7 +49966,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUoDtOdIza4.mp4",
"adj": {
"b": "whatever",
"a": "for"
@@ -50790,7 +49989,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LLnW_XgyHg0.mp4",
"adj": {
"b": "",
"a": ""
@@ -50814,7 +50012,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/N78crpC9o1g.mp4",
"adj": {
"b": "",
"a": "But"
@@ -50838,7 +50035,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/G02gq5ClixA.mp4",
"adj": {
"b": "know",
"a": "training"
@@ -50862,7 +50058,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/y6xjC3znn3g.mp4",
"adj": {
"b": "",
"a": "and"
@@ -50892,7 +50087,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/RJMe6PuMo3A.mp4",
"adj": {
"b": "it",
"a": "and"
@@ -50922,7 +50116,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/z5IYf8iDnjA.mp4",
"adj": {
"b": "",
"a": "You"
@@ -50952,7 +50145,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OBidX_oE7zM.mp4",
"adj": {
"b": "it",
"a": "Israel"
@@ -50976,7 +50168,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/V4k0ziHTGog.mp4",
"adj": {
"b": "",
"a": "through"
@@ -51000,7 +50191,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xulBQwriW5E.mp4",
"adj": {
"b": "and",
"a": "we're"
@@ -51035,7 +50225,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "man",
"a": "are"
@@ -51070,7 +50259,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KtZaCYPSunM.mp4",
"adj": {
"b": "",
"a": "it"
@@ -51100,7 +50288,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/C-SzFQZbN2M.mp4",
"adj": {
"b": "you're",
"a": ""
@@ -51130,7 +50317,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "",
"a": ""
@@ -51154,7 +50340,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/iYbUqMyCOLU.mp4",
"adj": {
"b": "women",
"a": "the"
@@ -51189,7 +50374,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mwtbQffMRic.mp4",
"adj": {
"b": "house",
"a": ""
@@ -51213,7 +50397,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "out",
"a": "but"
@@ -51243,7 +50426,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Key66cOeiEo.mp4",
"adj": {
"b": "it.",
"a": "Oh"
@@ -51267,7 +50449,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sfVzr3Jcro8.mp4",
"adj": {
"b": "",
"a": "got"
@@ -51297,7 +50478,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cN5k-3sfBdA.mp4",
"adj": {
"b": "",
"a": "The"
@@ -51321,7 +50501,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZEwkRomA9pU.mp4",
"adj": {
"b": "",
"a": "and"
@@ -51345,7 +50524,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/MXHk10OCoow.mp4",
"adj": {
"b": "",
"a": ""
@@ -51369,7 +50547,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0pHYgMHhi60.mp4",
"adj": {
"b": "and",
"a": "maybe"
@@ -51399,7 +50576,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mkyBXrJhDrQ.mp4",
"adj": {
"b": "",
"a": "the"
@@ -51423,7 +50599,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/jprDE87Ec58.mp4",
"adj": {
"b": "Trump",
"a": ""
@@ -51453,7 +50628,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mOF790Bh6I0.mp4",
"adj": {
"b": "",
"a": "Gillis"
@@ -51483,7 +50657,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "death.",
"a": "I"
@@ -51507,7 +50680,6 @@
"why": "next word \"I\" starts here",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LiTjFiYApxg.mp4",
"adj": {
"b": "it",
"a": "I"
@@ -51547,7 +50719,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rATxtI8FhxE.mp4",
"adj": {
"b": "know",
"a": "a"
@@ -51577,7 +50748,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "Here's",
"a": "here's"
@@ -51607,7 +50777,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/VYD--C8hJ4I.mp4",
"adj": {
"b": "make",
"a": "stew,"
@@ -51631,7 +50800,6 @@
"why": "unvoiced tail",
"wordIn": "in",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/h170V_0AbQ4.mp4",
"adj": {
"b": "",
"a": ""
@@ -51661,7 +50829,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/RJMe6PuMo3A.mp4",
"adj": {
"b": "that",
"a": "I"
@@ -51691,7 +50858,6 @@
"why": "next word \"a\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/GdMSdVvQvtc.mp4",
"adj": {
"b": "had",
"a": "a"
@@ -51726,7 +50892,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/waF76PwdlS8.mp4",
"adj": {
"b": "",
"a": "tariffs"
@@ -51756,7 +50921,6 @@
"why": "next word \"but\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7uNqvETYmVI.mp4",
"adj": {
"b": "",
"a": "but"
@@ -51786,7 +50950,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-N5_5CpyZ_8.mp4",
"adj": {
"b": "",
"a": "in"
@@ -51816,7 +50979,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OlY74Y-NmMw.mp4",
"adj": {
"b": "channel",
"a": "I"
@@ -51846,7 +51008,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OBidX_oE7zM.mp4",
"adj": {
"b": "Massy",
"a": "was"
@@ -51870,7 +51031,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fjLMZwGB4Ds.mp4",
"adj": {
"b": "and",
"a": "there"
@@ -51894,7 +51054,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KyzBZ19iRaU.mp4",
"adj": {
"b": "",
"a": "according"
@@ -51924,7 +51083,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "",
"a": "at"
@@ -51954,7 +51112,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "that",
"a": "the"
@@ -51978,7 +51135,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yGvQfTultRw.mp4",
"adj": {
"b": "maker",
"a": ""
@@ -52002,7 +51158,6 @@
"why": "unvoiced tail",
"wordIn": "can't",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "",
"a": ""
@@ -52032,7 +51187,6 @@
"why": "unvoiced tail",
"wordIn": "literal",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/axL33DS5KeA.mp4",
"adj": {
"b": "the",
"a": ""
@@ -52067,7 +51221,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rJ-V9bd9La0.mp4",
"adj": {
"b": "and",
"a": "if"
@@ -52091,7 +51244,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8RiHnmAGy8U.mp4",
"adj": {
"b": "disgusting.",
"a": "I'm"
@@ -52115,7 +51267,6 @@
"why": "next word \"Savannah\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CE3O2H43_tc.mp4",
"adj": {
"b": "think",
"a": "Savannah"
@@ -52150,7 +51301,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/VNdJ8BvruCQ.mp4",
"adj": {
"b": "him",
"a": "maybe"
@@ -52185,7 +51335,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "people",
"a": ""
@@ -52220,7 +51369,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Jzw4zVQKEA4.mp4",
"adj": {
"b": "",
"a": ""
@@ -52244,7 +51392,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dhOXEakOpaM.mp4",
"adj": {
"b": "and",
"a": "try"
@@ -52274,7 +51421,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/jprDE87Ec58.mp4",
"adj": {
"b": "any",
"a": "reality"
@@ -52298,7 +51444,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Me0fryGe2X8.mp4",
"adj": {
"b": "",
"a": "an"
@@ -52322,7 +51467,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/DMAa1fld4Rc.mp4",
"adj": {
"b": "it",
"a": "but"
@@ -52352,7 +51496,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yjHHiRhtMvQ.mp4",
"adj": {
"b": "",
"a": "Somali"
@@ -52376,7 +51519,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BKUP9tal4mM.mp4",
"adj": {
"b": "",
"a": "but"
@@ -52406,7 +51548,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7AtUt83XGQU.mp4",
"adj": {
"b": "",
"a": "way"
@@ -52430,7 +51571,6 @@
"why": "next word \"even\" starts here",
"wordIn": "doesn't",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dhOXEakOpaM.mp4",
"adj": {
"b": "typically",
"a": "even"
@@ -52470,7 +51610,6 @@
"why": "next word \"as\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ybLrSiTQxtQ.mp4",
"adj": {
"b": "that",
"a": "as"
@@ -52500,7 +51639,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/e0d6EzTEt7Y.mp4",
"adj": {
"b": "wife",
"a": "to"
@@ -52524,7 +51662,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7lOQD2k3erE.mp4",
"adj": {
"b": "that.",
"a": "I"
@@ -52554,7 +51691,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "probably",
"a": "it's"
@@ -52584,7 +51720,6 @@
"why": "next word \"and\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "late",
"a": "and"
@@ -52619,7 +51754,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/etu7Cy2WmC4.mp4",
"adj": {
"b": "",
"a": "and"
@@ -52649,7 +51783,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "terms",
"a": "that"
@@ -52679,7 +51812,6 @@
"why": "next word \"They're\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mOF790Bh6I0.mp4",
"adj": {
"b": "",
"a": "They're"
@@ -52714,7 +51846,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/r3cTXMxBPbc.mp4",
"adj": {
"b": "",
"a": ""
@@ -52738,7 +51869,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/6dgXutAKziM.mp4",
"adj": {
"b": "",
"a": "of"
@@ -52768,7 +51898,6 @@
"why": "next word \"my\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "like",
"a": "my"
@@ -52808,7 +51937,6 @@
"why": "next word \"a\" starts here",
"wordIn": "be",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TYPV_Ej1ND4.mp4",
"adj": {
"b": "might",
"a": "a"
@@ -52843,7 +51971,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/etu7Cy2WmC4.mp4",
"adj": {
"b": "but",
"a": ""
@@ -52867,7 +51994,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LNNlgkqMj0s.mp4",
"adj": {
"b": "they",
"a": ""
@@ -52891,7 +52017,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7uNqvETYmVI.mp4",
"adj": {
"b": "reason.",
"a": "And"
@@ -52915,7 +52040,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KssiwNCTEKQ.mp4",
"adj": {
"b": "facts?",
"a": "I"
@@ -52939,7 +52063,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "did",
"a": "whopping"
@@ -52974,7 +52097,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/C-SzFQZbN2M.mp4",
"adj": {
"b": "this",
"a": "before"
@@ -53004,7 +52126,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/aeC-_9DQWM4.mp4",
"adj": {
"b": "can",
"a": "obfuscate"
@@ -53028,7 +52149,6 @@
"why": "next word \"need\" starts here",
"wordIn": "you",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/knmVtqTW0MQ.mp4",
"adj": {
"b": "here",
"a": "need"
@@ -53068,7 +52188,6 @@
"why": "next word \"No\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fp_GOiz5MV0.mp4",
"adj": {
"b": "",
"a": "No"
@@ -53098,7 +52217,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vfMRAdhSSrc.mp4",
"adj": {
"b": "with",
"a": "new"
@@ -53133,7 +52251,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/i29P5pQc50A.mp4",
"adj": {
"b": "",
"a": "And"
@@ -53163,7 +52280,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8RiHnmAGy8U.mp4",
"adj": {
"b": "music",
"a": "listen"
@@ -53193,7 +52309,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "",
"a": ""
@@ -53217,7 +52332,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HTRY0dNjZIo.mp4",
"adj": {
"b": "good.",
"a": "I"
@@ -53247,7 +52361,6 @@
"why": "",
"wordIn": "And",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UqCmkStyKow.mp4",
"adj": {
"b": "me.",
"a": ""
@@ -53277,7 +52390,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "it",
"a": ""
@@ -53301,7 +52413,6 @@
"why": "next word \"an\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mwtbQffMRic.mp4",
"adj": {
"b": "",
"a": "an"
@@ -53331,7 +52442,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rATxtI8FhxE.mp4",
"adj": {
"b": "country",
"a": "leftists"
@@ -53361,7 +52471,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "",
"a": "and"
@@ -53391,7 +52500,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/slD9-aQc04g.mp4",
"adj": {
"b": "but",
"a": "would"
@@ -53421,7 +52529,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/X70q2QJ_VTc.mp4",
"adj": {
"b": "",
"a": ""
@@ -53445,7 +52552,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xbT-lWAzaiA.mp4",
"adj": {
"b": "at",
"a": ""
@@ -53475,7 +52581,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/EeVtP-LxWaI.mp4",
"adj": {
"b": "",
"a": "and"
@@ -53505,7 +52610,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KtZaCYPSunM.mp4",
"adj": {
"b": "",
"a": "to"
@@ -53529,7 +52633,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0TcCBOaiZO8.mp4",
"adj": {
"b": "",
"a": ""
@@ -53553,7 +52656,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/4RFFlS0oMEM.mp4",
"adj": {
"b": "",
"a": "pirate"
@@ -53577,7 +52679,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cAvjCjQqOac.mp4",
"adj": {
"b": "",
"a": "needling"
@@ -53601,7 +52702,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "month",
"a": "at"
@@ -53625,7 +52725,6 @@
"why": "next word \"here's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/f5sLSteRRTQ.mp4",
"adj": {
"b": "",
"a": "here's"
@@ -53655,7 +52754,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "code.",
"a": "twenty"
@@ -53685,7 +52783,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LLnW_XgyHg0.mp4",
"adj": {
"b": "",
"a": "probably"
@@ -53715,7 +52812,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "",
"a": "you"
@@ -53755,7 +52851,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "",
"a": ""
@@ -53779,7 +52874,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUSzqEViwBw.mp4",
"adj": {
"b": "shelf",
"a": ""
@@ -53803,7 +52897,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "",
"a": "Now,"
@@ -53833,7 +52926,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nPpJc33VuO4.mp4",
"adj": {
"b": "absolutely",
"a": "love"
@@ -53868,7 +52960,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/93DFKk1eSoM.mp4",
"adj": {
"b": "",
"a": "certainly"
@@ -53898,7 +52989,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/B3uW6WKvWlk.mp4",
"adj": {
"b": "",
"a": "Reputation"
@@ -53922,7 +53012,6 @@
"why": "next word \"there's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "but",
"a": "there's"
@@ -53952,7 +53041,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mxq6w_cMG6c.mp4",
"adj": {
"b": "and",
"a": ""
@@ -53982,7 +53070,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Tvpplznw9Nw.mp4",
"adj": {
"b": "",
"a": "if"
@@ -54012,7 +53099,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KtZaCYPSunM.mp4",
"adj": {
"b": "help",
"a": "to"
@@ -54047,7 +53133,6 @@
"why": "unvoiced tail",
"wordIn": "on",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "this",
"a": "Jeremy"
@@ -54087,7 +53172,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Zfcmx4_Qxxg.mp4",
"adj": {
"b": "",
"a": "and"
@@ -54111,7 +53195,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CE3O2H43_tc.mp4",
"adj": {
"b": "",
"a": ""
@@ -54135,7 +53218,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WlC3m-9S-0E.mp4",
"adj": {
"b": "",
"a": "if"
@@ -54159,7 +53241,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8hsc6aSGdk.mp4",
"adj": {
"b": "speech",
"a": "you"
@@ -54189,7 +53270,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "that",
"a": "only"
@@ -54213,7 +53293,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUoDtOdIza4.mp4",
"adj": {
"b": "",
"a": "Bangladesh"
@@ -54243,7 +53322,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ts76T7tJv-U.mp4",
"adj": {
"b": "",
"a": ""
@@ -54273,7 +53351,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/YvJBXgkKwv8.mp4",
"adj": {
"b": "and",
"a": ""
@@ -54303,7 +53380,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CTbCFWZ7Gdk.mp4",
"adj": {
"b": "took",
"a": "blood"
@@ -54338,7 +53414,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9zEisQcriiM.mp4",
"adj": {
"b": "",
"a": "like"
@@ -54362,7 +53437,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nzOE5f6VPqY.mp4",
"adj": {
"b": "that",
"a": "because"
@@ -54386,7 +53460,6 @@
"why": "next word \"to\" starts here",
"wordIn": "they're",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/DMAa1fld4Rc.mp4",
"adj": {
"b": "how",
"a": "to"
@@ -54426,7 +53499,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/im0yxtppGf4.mp4",
"adj": {
"b": "Quinn",
"a": "are"
@@ -54450,7 +53522,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nPpJc33VuO4.mp4",
"adj": {
"b": "that",
"a": "you"
@@ -54480,7 +53551,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/l8y24jC9JUM.mp4",
"adj": {
"b": "case.",
"a": "Apparently"
@@ -54510,7 +53580,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lXmLzzuxVCo.mp4",
"adj": {
"b": "with",
"a": "poles"
@@ -54534,7 +53603,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "with",
"a": "and"
@@ -54564,7 +53632,6 @@
"why": "next word \"video\" starts here",
"wordIn": "in",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "confessed",
"a": "video"
@@ -54604,7 +53671,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7ctY0v_LEV0.mp4",
"adj": {
"b": "",
"a": "and"
@@ -54634,7 +53700,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-pKXHyeyRhQ.mp4",
"adj": {
"b": "at",
"a": "during"
@@ -54664,7 +53729,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "maybe",
"a": "the"
@@ -54688,7 +53752,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "",
"a": "passing."
@@ -54712,7 +53775,6 @@
"why": "next word \"new\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hGqSbmfHO7o.mp4",
"adj": {
"b": "that",
"a": "new"
@@ -54752,7 +53814,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8hsc6aSGdk.mp4",
"adj": {
"b": "the",
"a": ""
@@ -54782,7 +53843,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/azX_WsMqFqg.mp4",
"adj": {
"b": "know",
"a": ""
@@ -54806,7 +53866,6 @@
"why": "next word \"using\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "deep",
"a": "using"
@@ -54836,7 +53895,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "hour",
"a": "before"
@@ -54860,7 +53918,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rH1VkOpGce4.mp4",
"adj": {
"b": "it",
"a": "or"
@@ -54890,7 +53947,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Oo6dXSEh65Y.mp4",
"adj": {
"b": "care",
"a": "she's"
@@ -54920,7 +53976,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "like",
"a": "I"
@@ -54950,7 +54005,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7zRNtT2ygCc.mp4",
"adj": {
"b": "who",
"a": ""
@@ -54980,7 +54034,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cPKtKx3evdM.mp4",
"adj": {
"b": "like",
"a": "eye"
@@ -55010,7 +54063,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FTS15w2m2Ac.mp4",
"adj": {
"b": "",
"a": "the"
@@ -55040,7 +54092,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/78A048gOuq8.mp4",
"adj": {
"b": "",
"a": "you"
@@ -55085,7 +54136,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cN5k-3sfBdA.mp4",
"adj": {
"b": "just",
"a": ""
@@ -55109,7 +54159,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/H-xC6-gSvEc.mp4",
"adj": {
"b": "Rivers",
"a": ""
@@ -55133,7 +54182,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cN5k-3sfBdA.mp4",
"adj": {
"b": "",
"a": ""
@@ -55157,7 +54205,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xEYsJorlvkw.mp4",
"adj": {
"b": "tracks",
"a": "or"
@@ -55181,7 +54228,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/pNP4B0PJk-Q.mp4",
"adj": {
"b": "breakfast.",
"a": ""
@@ -55205,7 +54251,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/6A1JPaiqJNY.mp4",
"adj": {
"b": "",
"a": ""
@@ -55229,7 +54274,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "And",
"a": "here"
@@ -55259,7 +54303,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oAj5BoNKTYQ.mp4",
"adj": {
"b": "",
"a": "you"
@@ -55283,7 +54326,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ibJe2PlNOS8.mp4",
"adj": {
"b": "bigger",
"a": "and"
@@ -55313,7 +54355,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mGcQRQ3wqsk.mp4",
"adj": {
"b": "go",
"a": ""
@@ -55337,7 +54378,6 @@
"why": "unvoiced tail",
"wordIn": "not",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/z6kKxW8YBBY.mp4",
"adj": {
"b": "is",
"a": ""
@@ -55372,7 +54412,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Qyjo41I1FcI.mp4",
"adj": {
"b": "or",
"a": "as"
@@ -55396,7 +54435,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UwD3NXKMsoQ.mp4",
"adj": {
"b": "",
"a": "I'm"
@@ -55420,7 +54458,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-EG5kNxDKR4.mp4",
"adj": {
"b": "government",
"a": "the"
@@ -55450,7 +54487,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lraibG7Iu5w.mp4",
"adj": {
"b": "worker",
"a": "giving"
@@ -55480,7 +54516,6 @@
"why": "next word \"We\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/4JiT3UO-yCQ.mp4",
"adj": {
"b": "",
"a": "We"
@@ -55515,7 +54550,6 @@
"why": "next word \"what\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/i29P5pQc50A.mp4",
"adj": {
"b": "",
"a": "what"
@@ -55550,7 +54584,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/SMBeGH0Zimw.mp4",
"adj": {
"b": "reads",
"a": "a"
@@ -55580,7 +54613,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/78A048gOuq8.mp4",
"adj": {
"b": "",
"a": "you"
@@ -55615,7 +54647,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/knmVtqTW0MQ.mp4",
"adj": {
"b": "America",
"a": "being"
@@ -55645,7 +54676,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "and",
"a": ""
@@ -55669,7 +54699,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "",
"a": "I"
@@ -55699,7 +54728,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "custody",
"a": "but"
@@ -55723,7 +54751,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fjLMZwGB4Ds.mp4",
"adj": {
"b": "",
"a": "be"
@@ -55747,7 +54774,6 @@
"why": "next word \"wage\" starts here",
"wordIn": "minimum",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9zEisQcriiM.mp4",
"adj": {
"b": "dollars",
"a": "wage"
@@ -55782,7 +54808,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/q2BYaavZUHo.mp4",
"adj": {
"b": "lunatic.",
"a": "I"
@@ -55812,7 +54837,6 @@
"why": "next word \"owned\" starts here",
"wordIn": "independently",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/pNP4B0PJk-Q.mp4",
"adj": {
"b": "than",
"a": "owned"
@@ -55847,7 +54871,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "seven",
"a": "if"
@@ -55871,7 +54894,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/eFU1WRc9lYM.mp4",
"adj": {
"b": "",
"a": "Until"
@@ -55895,7 +54917,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tC_WyEZzLV0.mp4",
"adj": {
"b": "bootlick",
"a": "companies"
@@ -55925,7 +54946,6 @@
"why": "next word \"Hannah\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xulBQwriW5E.mp4",
"adj": {
"b": "",
"a": "Hannah"
@@ -55955,7 +54975,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUoDtOdIza4.mp4",
"adj": {
"b": "bet",
"a": "I"
@@ -55985,7 +55004,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oYsPh2E-02E.mp4",
"adj": {
"b": "",
"a": "to"
@@ -56015,7 +55033,6 @@
"why": "next word \"yes\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/v-MG7ofJ6X4.mp4",
"adj": {
"b": "warrant",
"a": "yes"
@@ -56045,7 +55062,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WzlVDuwup0U.mp4",
"adj": {
"b": "",
"a": "and"
@@ -56069,7 +55085,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xulBQwriW5E.mp4",
"adj": {
"b": "",
"a": "search"
@@ -56093,7 +55108,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/6dgXutAKziM.mp4",
"adj": {
"b": "",
"a": "twenty"
@@ -56123,7 +55137,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/axL33DS5KeA.mp4",
"adj": {
"b": "us",
"a": "already"
@@ -56147,7 +55160,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KtZaCYPSunM.mp4",
"adj": {
"b": "there",
"a": "some"
@@ -56177,7 +55189,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WzlVDuwup0U.mp4",
"adj": {
"b": "just",
"a": ""
@@ -56201,7 +55212,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/H-xC6-gSvEc.mp4",
"adj": {
"b": "Obama",
"a": "was"
@@ -56231,7 +55241,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kT1gUTlBfNc.mp4",
"adj": {
"b": "it",
"a": "and"
@@ -56261,7 +55270,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "",
"a": ""
@@ -56285,7 +55293,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "but",
"a": "I"
@@ -56309,7 +55316,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "she",
"a": "took"
@@ -56333,7 +55339,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/67Qd5J47Y84.mp4",
"adj": {
"b": "center",
"a": "the"
@@ -56357,7 +55362,6 @@
"why": "next word \"militant\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mwtbQffMRic.mp4",
"adj": {
"b": "",
"a": "militant"
@@ -56387,7 +55391,6 @@
"why": "next word \"there's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "and",
"a": "there's"
@@ -56422,7 +55425,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/MXHk10OCoow.mp4",
"adj": {
"b": "",
"a": "but"
@@ -56452,7 +55454,6 @@
"why": "next word \"he\" starts here",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "",
"a": "he"
@@ -56492,7 +55493,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/X70q2QJ_VTc.mp4",
"adj": {
"b": "",
"a": "when"
@@ -56516,7 +55516,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rATxtI8FhxE.mp4",
"adj": {
"b": "dollars",
"a": "on"
@@ -56546,7 +55545,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bbVoMOzCSs0.mp4",
"adj": {
"b": "can",
"a": "the"
@@ -56581,7 +55579,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/y6xjC3znn3g.mp4",
"adj": {
"b": "either",
"a": "there"
@@ -56611,7 +55608,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUSzqEViwBw.mp4",
"adj": {
"b": "space",
"a": ""
@@ -56635,7 +55631,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/X70q2QJ_VTc.mp4",
"adj": {
"b": "again",
"a": ""
@@ -56659,7 +55654,6 @@
"why": "next word \"they\" starts here",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "leaked",
"a": "they"
@@ -56704,7 +55698,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "",
"a": "you"
@@ -56728,7 +55721,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "And",
"a": "I"
@@ -56758,7 +55750,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FWv906SubiM.mp4",
"adj": {
"b": "is",
"a": "the"
@@ -56788,7 +55779,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Tvpplznw9Nw.mp4",
"adj": {
"b": "been",
"a": "with"
@@ -56818,7 +55808,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "",
"a": ""
@@ -56842,7 +55831,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/N78crpC9o1g.mp4",
"adj": {
"b": "conservatives,",
"a": "are"
@@ -56872,7 +55860,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8RiHnmAGy8U.mp4",
"adj": {
"b": "but",
"a": "millions"
@@ -56902,7 +55889,6 @@
"why": "",
"wordIn": "or",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0ynoYuPD-B8.mp4",
"adj": {
"b": "lens",
"a": ""
@@ -56932,7 +55918,6 @@
"why": "next word \"reviews,\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "driven",
"a": "reviews,"
@@ -56962,7 +55947,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6CNFodmbWw.mp4",
"adj": {
"b": "their",
"a": ""
@@ -56986,7 +55970,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "a",
"a": ""
@@ -57010,7 +55993,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0ynoYuPD-B8.mp4",
"adj": {
"b": "one",
"a": "pretty"
@@ -57040,7 +56022,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "world",
"a": "at"
@@ -57070,7 +56051,6 @@
"why": "next word \"it's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KyzBZ19iRaU.mp4",
"adj": {
"b": "and",
"a": "it's"
@@ -57100,7 +56080,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cPKtKx3evdM.mp4",
"adj": {
"b": "and",
"a": "I"
@@ -57135,7 +56114,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oYsPh2E-02E.mp4",
"adj": {
"b": "Grills",
"a": "that"
@@ -57165,7 +56143,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sfVzr3Jcro8.mp4",
"adj": {
"b": "know",
"a": "this"
@@ -57189,7 +56166,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/G02gq5ClixA.mp4",
"adj": {
"b": "dog",
"a": "was"
@@ -57213,7 +56189,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CTbCFWZ7Gdk.mp4",
"adj": {
"b": "",
"a": "is"
@@ -57243,7 +56218,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cQL0Rja_Qik.mp4",
"adj": {
"b": "Piker",
"a": "has"
@@ -57267,7 +56241,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8ez7u2StoWI.mp4",
"adj": {
"b": "",
"a": "I"
@@ -57297,7 +56270,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sfVzr3Jcro8.mp4",
"adj": {
"b": "speech",
"a": "where"
@@ -57327,7 +56299,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cAvjCjQqOac.mp4",
"adj": {
"b": "fucking",
"a": ""
@@ -57351,7 +56322,6 @@
"why": "next word \"when\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-N5_5CpyZ_8.mp4",
"adj": {
"b": "outraged",
"a": "when"
@@ -57391,7 +56361,6 @@
"why": "unvoiced tail",
"wordIn": "who",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dhOXEakOpaM.mp4",
"adj": {
"b": "",
"a": ""
@@ -57421,7 +56390,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ep_qtW2K3BY.mp4",
"adj": {
"b": "and",
"a": ""
@@ -57445,7 +56413,6 @@
"why": "next word \"order\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tRMofQwAjgw.mp4",
"adj": {
"b": "our",
"a": "order"
@@ -57480,7 +56447,6 @@
"why": "next word \"to\" starts here",
"wordIn": "is",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZeqS0DvnorA.mp4",
"adj": {
"b": "YouTube",
"a": "to"
@@ -57525,7 +56491,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UqCmkStyKow.mp4",
"adj": {
"b": "it.",
"a": "There's"
@@ -57555,7 +56520,6 @@
"why": "next word \"even\" starts here",
"wordIn": "And",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/acPLSEVKyqI.mp4",
"adj": {
"b": "And",
"a": "even"
@@ -57595,7 +56559,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/azX_WsMqFqg.mp4",
"adj": {
"b": "",
"a": ""
@@ -57619,7 +56582,6 @@
"why": "re-attack after a dip",
"wordIn": "mearing",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ts76T7tJv-U.mp4",
"adj": {
"b": "",
"a": ""
@@ -57649,7 +56611,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Zfcmx4_Qxxg.mp4",
"adj": {
"b": "there's",
"a": "self"
@@ -57679,7 +56640,6 @@
"why": "next word \"this\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yGvQfTultRw.mp4",
"adj": {
"b": "",
"a": "this"
@@ -57709,7 +56669,6 @@
"why": "next word \"There\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/4RFFlS0oMEM.mp4",
"adj": {
"b": "leftist.",
"a": "There"
@@ -57749,7 +56708,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8Sid61t1Zs.mp4",
"adj": {
"b": "",
"a": "and"
@@ -57773,7 +56731,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Zfcmx4_Qxxg.mp4",
"adj": {
"b": "",
"a": ""
@@ -57797,7 +56754,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7uNqvETYmVI.mp4",
"adj": {
"b": "me",
"a": "as"
@@ -57842,7 +56798,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "like",
"a": "B"
@@ -57866,7 +56821,6 @@
"why": "trailing noise burst",
"wordIn": "hilarious",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/waF76PwdlS8.mp4",
"adj": {
"b": "all",
"a": "and"
@@ -57901,7 +56855,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HPIHN8nxCM8.mp4",
"adj": {
"b": "see",
"a": ""
@@ -57931,7 +56884,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HPIHN8nxCM8.mp4",
"adj": {
"b": "what",
"a": ""
@@ -57961,7 +56913,6 @@
"why": "next word \"by\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/N78crpC9o1g.mp4",
"adj": {
"b": "",
"a": "by"
@@ -57996,7 +56947,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "about",
"a": ""
@@ -58026,7 +56976,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/aeC-_9DQWM4.mp4",
"adj": {
"b": "",
"a": "coordinated"
@@ -58050,7 +56999,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oAj5BoNKTYQ.mp4",
"adj": {
"b": "",
"a": "so"
@@ -58080,7 +57028,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Qyjo41I1FcI.mp4",
"adj": {
"b": "",
"a": "yes."
@@ -58104,7 +57051,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KW-dUugk9_E.mp4",
"adj": {
"b": "",
"a": "and"
@@ -58128,7 +57074,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ybLrSiTQxtQ.mp4",
"adj": {
"b": "",
"a": "Indian"
@@ -58152,7 +57097,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0TcCBOaiZO8.mp4",
"adj": {
"b": "",
"a": "I"
@@ -58182,7 +57126,6 @@
"why": "next word \"my\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KyzBZ19iRaU.mp4",
"adj": {
"b": "And",
"a": "my"
@@ -58217,7 +57160,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8RiHnmAGy8U.mp4",
"adj": {
"b": "",
"a": "and"
@@ -58241,7 +57183,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rATxtI8FhxE.mp4",
"adj": {
"b": "",
"a": "almost"
@@ -58271,7 +57212,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/P4JErOtgVuA.mp4",
"adj": {
"b": "officer",
"a": ""
@@ -58295,7 +57235,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xaWnBmX4AjY.mp4",
"adj": {
"b": "nature",
"a": "we"
@@ -58319,7 +57258,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0A4dIN1dKaY.mp4",
"adj": {
"b": "from",
"a": "shameless."
@@ -58349,7 +57287,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/SMBeGH0Zimw.mp4",
"adj": {
"b": "suit",
"a": "it's"
@@ -58373,7 +57310,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UCanpMuiH2U.mp4",
"adj": {
"b": "are",
"a": "basically"
@@ -58403,7 +57339,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "",
"a": "talking"
@@ -58427,7 +57362,6 @@
"why": "next word \"blah\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sfVzr3Jcro8.mp4",
"adj": {
"b": "",
"a": "blah"
@@ -58457,7 +57391,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OBidX_oE7zM.mp4",
"adj": {
"b": "immigration",
"a": "inside"
@@ -58487,7 +57420,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Hh71Xe7XK6k.mp4",
"adj": {
"b": "off,",
"a": "now"
@@ -58511,7 +57443,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dhOXEakOpaM.mp4",
"adj": {
"b": "officers",
"a": "other"
@@ -58546,7 +57477,6 @@
"why": "next word \"But\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/l9RG39RojHE.mp4",
"adj": {
"b": "sugar.",
"a": "But"
@@ -58576,7 +57506,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/AMdAOwrYmAI.mp4",
"adj": {
"b": "the",
"a": ""
@@ -58606,7 +57535,6 @@
"why": "next word \"Tim\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LLnW_XgyHg0.mp4",
"adj": {
"b": "to",
"a": "Tim"
@@ -58636,7 +57564,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-N5_5CpyZ_8.mp4",
"adj": {
"b": "apart.",
"a": "You"
@@ -58660,7 +57587,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CTbCFWZ7Gdk.mp4",
"adj": {
"b": "that",
"a": "I"
@@ -58695,7 +57621,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/SMBeGH0Zimw.mp4",
"adj": {
"b": "",
"a": "thing"
@@ -58725,7 +57650,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/M-jXumuv28E.mp4",
"adj": {
"b": "",
"a": "and"
@@ -58755,7 +57679,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "",
"a": "That's"
@@ -58779,7 +57702,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "wild.",
"a": "That"
@@ -58809,7 +57731,6 @@
"why": "unvoiced tail",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/e0d6EzTEt7Y.mp4",
"adj": {
"b": "",
"a": ""
@@ -58839,7 +57760,6 @@
"why": "next word \"or\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kfzmLf09O0w.mp4",
"adj": {
"b": "Show",
"a": "or"
@@ -58869,7 +57789,6 @@
"why": "unvoiced tail",
"wordIn": "of",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-lSWX5qachE.mp4",
"adj": {
"b": "",
"a": "you"
@@ -58899,7 +57818,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9YQfjW1uasM.mp4",
"adj": {
"b": "",
"a": "I'm"
@@ -58929,7 +57847,6 @@
"why": "next word \"but\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "",
"a": "but"
@@ -58959,7 +57876,6 @@
"why": "next word \"I'm\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "that,",
"a": "I'm"
@@ -58989,7 +57905,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8ez7u2StoWI.mp4",
"adj": {
"b": "woke",
"a": ""
@@ -59019,7 +57934,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TTN8zF4Vjz4.mp4",
"adj": {
"b": "who",
"a": "went"
@@ -59049,7 +57963,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/34mSBDoWxKc.mp4",
"adj": {
"b": "",
"a": ""
@@ -59073,7 +57986,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/l8y24jC9JUM.mp4",
"adj": {
"b": "",
"a": "and"
@@ -59103,7 +58015,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oYsPh2E-02E.mp4",
"adj": {
"b": "made.",
"a": "We"
@@ -59127,7 +58038,6 @@
"why": "",
"wordIn": "movie",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-EG5kNxDKR4.mp4",
"adj": {
"b": "the",
"a": "Tombstone."
@@ -59172,7 +58082,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ibJe2PlNOS8.mp4",
"adj": {
"b": "platformed",
"a": "because"
@@ -59202,7 +58111,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/pNP4B0PJk-Q.mp4",
"adj": {
"b": "",
"a": "So"
@@ -59226,7 +58134,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dWMRXD0RJRY.mp4",
"adj": {
"b": "intended.",
"a": ""
@@ -59250,7 +58157,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/igmjuXKDgvU.mp4",
"adj": {
"b": "video.",
"a": ""
@@ -59274,7 +58180,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "",
"a": ""
@@ -59298,7 +58203,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZeqS0DvnorA.mp4",
"adj": {
"b": "already",
"a": "you"
@@ -59328,7 +58232,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Tvpplznw9Nw.mp4",
"adj": {
"b": "",
"a": "I"
@@ -59352,7 +58255,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/igmjuXKDgvU.mp4",
"adj": {
"b": "",
"a": "as"
@@ -59376,7 +58278,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Tvpplznw9Nw.mp4",
"adj": {
"b": "to",
"a": "some"
@@ -59400,7 +58301,6 @@
"why": "next word \"his\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Jzw4zVQKEA4.mp4",
"adj": {
"b": "him",
"a": "his"
@@ -59435,7 +58335,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/P4JErOtgVuA.mp4",
"adj": {
"b": "struck",
"a": "get"
@@ -59459,7 +58358,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/EeVtP-LxWaI.mp4",
"adj": {
"b": "",
"a": "A"
@@ -59483,7 +58381,6 @@
"why": "next word \"think\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1UgCLhoVIfs.mp4",
"adj": {
"b": "",
"a": "think"
@@ -59513,7 +58410,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "take.",
"a": "I"
@@ -59537,7 +58433,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QFUX3tJJpiI.mp4",
"adj": {
"b": "the",
"a": ""
@@ -59567,7 +58462,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FeUW-Vw02ZQ.mp4",
"adj": {
"b": "",
"a": "they're"
@@ -59597,7 +58491,6 @@
"why": "next word \"squatting\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7ctY0v_LEV0.mp4",
"adj": {
"b": "",
"a": "squatting"
@@ -59627,7 +58520,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oYsPh2E-02E.mp4",
"adj": {
"b": "that",
"a": "Corano"
@@ -59662,7 +58554,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tcevefZ9G18.mp4",
"adj": {
"b": "",
"a": "it"
@@ -59692,7 +58583,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dhOXEakOpaM.mp4",
"adj": {
"b": "both",
"a": "in"
@@ -59722,7 +58612,6 @@
"why": "next word \"he\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kfzmLf09O0w.mp4",
"adj": {
"b": "up",
"a": "he"
@@ -59757,7 +58646,6 @@
"why": "next word \"she\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/MB0fIIPySYs.mp4",
"adj": {
"b": "music",
"a": "she"
@@ -59787,7 +58675,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/JcEkqCQP_FE.mp4",
"adj": {
"b": "",
"a": "I'm"
@@ -59822,7 +58709,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/N78crpC9o1g.mp4",
"adj": {
"b": "gravely",
"a": ""
@@ -59852,7 +58738,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "that.",
"a": "I"
@@ -59882,7 +58767,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/z5IYf8iDnjA.mp4",
"adj": {
"b": "but",
"a": "I"
@@ -59906,7 +58790,6 @@
"why": "next word \"let's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "",
"a": "let's"
@@ -59936,7 +58819,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/GdMSdVvQvtc.mp4",
"adj": {
"b": "",
"a": "probably"
@@ -59960,7 +58842,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mGcQRQ3wqsk.mp4",
"adj": {
"b": "okay",
"a": "why"
@@ -59984,7 +58865,6 @@
"why": "next word \"watermark\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LoaYx6-_arE.mp4",
"adj": {
"b": "",
"a": "watermark"
@@ -60014,7 +58894,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/GdMSdVvQvtc.mp4",
"adj": {
"b": "",
"a": "They're"
@@ -60038,7 +58917,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mGcQRQ3wqsk.mp4",
"adj": {
"b": "",
"a": "Absolute."
@@ -60062,7 +58940,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tsQBo2SvOao.mp4",
"adj": {
"b": "",
"a": "limited"
@@ -60086,7 +58963,6 @@
"why": "",
"wordIn": "up",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HPIHN8nxCM8.mp4",
"adj": {
"b": "gearing",
"a": "for"
@@ -60126,7 +59002,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yGvQfTultRw.mp4",
"adj": {
"b": "but",
"a": ""
@@ -60156,7 +59031,6 @@
"why": "",
"wordIn": "there",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/RT0mKx_hTv4.mp4",
"adj": {
"b": "there",
"a": "I"
@@ -60196,7 +59070,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "that",
"a": "often"
@@ -60220,7 +59093,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/e0d6EzTEt7Y.mp4",
"adj": {
"b": "as",
"a": "local"
@@ -60250,7 +59122,6 @@
"why": "next word \"maybe\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/AwwrH7NyCLA.mp4",
"adj": {
"b": "",
"a": "maybe"
@@ -60285,7 +59156,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dMs5wvpy1Qk.mp4",
"adj": {
"b": "",
"a": "you"
@@ -60315,7 +59185,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/eFU1WRc9lYM.mp4",
"adj": {
"b": "do.",
"a": "The"
@@ -60339,7 +59208,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kMDV9dG0nVY.mp4",
"adj": {
"b": "committee",
"a": ""
@@ -60369,7 +59237,6 @@
"why": "",
"wordIn": "can't",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KyzBZ19iRaU.mp4",
"adj": {
"b": "I",
"a": ""
@@ -60399,7 +59266,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7zRNtT2ygCc.mp4",
"adj": {
"b": "",
"a": "but"
@@ -60423,7 +59289,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UwD3NXKMsoQ.mp4",
"adj": {
"b": "",
"a": "for"
@@ -60453,7 +59318,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TTN8zF4Vjz4.mp4",
"adj": {
"b": "know",
"a": "propaganda"
@@ -60483,7 +59347,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sfVzr3Jcro8.mp4",
"adj": {
"b": "",
"a": "very"
@@ -60513,7 +59376,6 @@
"why": "",
"wordIn": "it's",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fp_GOiz5MV0.mp4",
"adj": {
"b": "",
"a": ""
@@ -60543,7 +59405,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "being",
"a": ""
@@ -60573,7 +59434,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ibJe2PlNOS8.mp4",
"adj": {
"b": "it",
"a": "from"
@@ -60597,7 +59457,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "can.",
"a": "And"
@@ -60627,7 +59486,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Key66cOeiEo.mp4",
"adj": {
"b": "over.",
"a": ""
@@ -60651,7 +59509,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oAj5BoNKTYQ.mp4",
"adj": {
"b": "as",
"a": "he"
@@ -60681,7 +59538,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/_0hmLgl6OQc.mp4",
"adj": {
"b": "",
"a": ""
@@ -60705,7 +59561,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/im0yxtppGf4.mp4",
"adj": {
"b": "have",
"a": ""
@@ -60735,7 +59590,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fjLMZwGB4Ds.mp4",
"adj": {
"b": "coordinated",
"a": ""
@@ -60765,7 +59619,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/azX_WsMqFqg.mp4",
"adj": {
"b": "anyway",
"a": ""
@@ -60789,7 +59642,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "but",
"a": ""
@@ -60813,7 +59665,6 @@
"why": "next word \"the\" starts here",
"wordIn": "like",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rPkFK9TgI3w.mp4",
"adj": {
"b": "like",
"a": "the"
@@ -60858,7 +59709,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/EExLqtGVifk.mp4",
"adj": {
"b": "",
"a": ""
@@ -60882,7 +59732,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/GdMSdVvQvtc.mp4",
"adj": {
"b": "criminal",
"a": "but"
@@ -60912,7 +59761,6 @@
"why": "unvoiced tail",
"wordIn": "be",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tdYCm-SSyFw.mp4",
"adj": {
"b": "",
"a": ""
@@ -60942,7 +59790,6 @@
"why": "next word \"now\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "",
"a": "now"
@@ -60977,7 +59824,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "",
"a": "want"
@@ -61001,7 +59847,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "",
"a": "it"
@@ -61025,7 +59870,6 @@
"why": "next word \"with\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nzOE5f6VPqY.mp4",
"adj": {
"b": "think",
"a": "with"
@@ -61060,7 +59904,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7ctY0v_LEV0.mp4",
"adj": {
"b": "",
"a": "in"
@@ -61095,7 +59938,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nzOE5f6VPqY.mp4",
"adj": {
"b": "",
"a": "formally"
@@ -61119,7 +59961,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kfzmLf09O0w.mp4",
"adj": {
"b": "the",
"a": ""
@@ -61149,7 +59990,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hykdhazn8ds.mp4",
"adj": {
"b": "",
"a": ""
@@ -61173,7 +60013,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/aU5TsjpTD8Y.mp4",
"adj": {
"b": "know",
"a": ""
@@ -61203,7 +60042,6 @@
"why": "next word \"The\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/_0hmLgl6OQc.mp4",
"adj": {
"b": "",
"a": "The"
@@ -61238,7 +60076,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUoDtOdIza4.mp4",
"adj": {
"b": "Turkey,",
"a": "the"
@@ -61273,7 +60110,6 @@
"why": "unvoiced tail",
"wordIn": "didn't",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Nb5iswtRmVM.mp4",
"adj": {
"b": "he",
"a": ""
@@ -61303,7 +60139,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/q2BYaavZUHo.mp4",
"adj": {
"b": "",
"a": "You"
@@ -61333,7 +60168,6 @@
"why": "next word \"there's\" starts here",
"wordIn": "However",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FiLs5ovIfts.mp4",
"adj": {
"b": "store.",
"a": "there's"
@@ -61373,7 +60207,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Oo6dXSEh65Y.mp4",
"adj": {
"b": "know",
"a": ""
@@ -61397,7 +60230,6 @@
"why": "next word \"AM\" starts here",
"wordIn": "at",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tLQnOAlh6O8.mp4",
"adj": {
"b": "five",
"a": "AM"
@@ -61437,7 +60269,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Tvpplznw9Nw.mp4",
"adj": {
"b": "",
"a": "I"
@@ -61472,7 +60303,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/78A048gOuq8.mp4",
"adj": {
"b": "",
"a": "Charlie"
@@ -61502,7 +60332,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tcevefZ9G18.mp4",
"adj": {
"b": "",
"a": ""
@@ -61526,7 +60355,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/i29P5pQc50A.mp4",
"adj": {
"b": "",
"a": ""
@@ -61550,7 +60378,6 @@
"why": "next word \"after\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dyKWsw1pmlY.mp4",
"adj": {
"b": "",
"a": "after"
@@ -61580,7 +60407,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WJYyCfbjv8s.mp4",
"adj": {
"b": "the",
"a": "Kimmel"
@@ -61610,7 +60436,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WJYyCfbjv8s.mp4",
"adj": {
"b": "",
"a": "that"
@@ -61640,7 +60465,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "do",
"a": "you"
@@ -61675,7 +60499,6 @@
"why": "unvoiced tail",
"wordIn": "or",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "meetings",
"a": ""
@@ -61705,7 +60528,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oYsPh2E-02E.mp4",
"adj": {
"b": "redemption.",
"a": "I"
@@ -61729,7 +60551,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KtZaCYPSunM.mp4",
"adj": {
"b": "that",
"a": "and"
@@ -61759,7 +60580,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3gORoyt2mC4.mp4",
"adj": {
"b": "keep",
"a": ""
@@ -61783,7 +60603,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "she's",
"a": "a"
@@ -61813,7 +60632,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/l9RG39RojHE.mp4",
"adj": {
"b": "know,",
"a": "I"
@@ -61843,7 +60661,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vfMRAdhSSrc.mp4",
"adj": {
"b": "you",
"a": "coming"
@@ -61873,7 +60690,6 @@
"why": "next word \"little\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "Zelensky",
"a": "little"
@@ -61903,7 +60719,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rATxtI8FhxE.mp4",
"adj": {
"b": "",
"a": ""
@@ -61933,7 +60748,6 @@
"why": "unvoiced tail",
"wordIn": "in",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/eEg1SRZj1rA.mp4",
"adj": {
"b": "financial",
"a": ""
@@ -61968,7 +60782,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/i29P5pQc50A.mp4",
"adj": {
"b": "",
"a": "it"
@@ -61992,7 +60805,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/y6xjC3znn3g.mp4",
"adj": {
"b": "but",
"a": "I'm"
@@ -62022,7 +60834,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/X70q2QJ_VTc.mp4",
"adj": {
"b": "it",
"a": ""
@@ -62052,7 +60863,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "",
"a": "but"
@@ -62082,7 +60892,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/pNP4B0PJk-Q.mp4",
"adj": {
"b": "",
"a": ""
@@ -62106,7 +60915,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "",
"a": "you"
@@ -62130,7 +60938,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/EExLqtGVifk.mp4",
"adj": {
"b": "and",
"a": "smash"
@@ -62160,7 +60967,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nPpJc33VuO4.mp4",
"adj": {
"b": "industry",
"a": "you"
@@ -62184,7 +60990,6 @@
"why": "unvoiced tail",
"wordIn": "for",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bbVoMOzCSs0.mp4",
"adj": {
"b": "and",
"a": ""
@@ -62219,7 +61024,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kT1gUTlBfNc.mp4",
"adj": {
"b": "party",
"a": "if"
@@ -62243,7 +61047,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "",
"a": "if"
@@ -62267,7 +61070,6 @@
"why": "next word \"we\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nPpJc33VuO4.mp4",
"adj": {
"b": "and",
"a": "we"
@@ -62302,7 +61104,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUoDtOdIza4.mp4",
"adj": {
"b": "road",
"a": "that"
@@ -62332,7 +61133,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LP94fw5JtYk.mp4",
"adj": {
"b": "gosh",
"a": "it"
@@ -62356,7 +61156,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZEwkRomA9pU.mp4",
"adj": {
"b": "",
"a": ""
@@ -62380,7 +61179,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "more",
"a": ""
@@ -62415,7 +61213,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/RJMe6PuMo3A.mp4",
"adj": {
"b": "like",
"a": ""
@@ -62439,7 +61236,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WlC3m-9S-0E.mp4",
"adj": {
"b": "",
"a": "isn't"
@@ -62469,7 +61265,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-CcZe5Bffzs.mp4",
"adj": {
"b": "pizza",
"a": "pizza"
@@ -62493,7 +61288,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "",
"a": ""
@@ -62517,7 +61311,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LLnW_XgyHg0.mp4",
"adj": {
"b": "and",
"a": "he"
@@ -62547,7 +61340,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/G02gq5ClixA.mp4",
"adj": {
"b": "home",
"a": "that"
@@ -62577,7 +61369,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8Sid61t1Zs.mp4",
"adj": {
"b": "it",
"a": ""
@@ -62607,7 +61398,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vsydUYyjSMU.mp4",
"adj": {
"b": "",
"a": "this"
@@ -62631,7 +61421,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-lSWX5qachE.mp4",
"adj": {
"b": "thousand",
"a": ""
@@ -62655,7 +61444,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "here.",
"a": "They're"
@@ -62685,7 +61473,6 @@
"why": "next word \"making\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ibJe2PlNOS8.mp4",
"adj": {
"b": "that",
"a": "making"
@@ -62715,7 +61502,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/z5IYf8iDnjA.mp4",
"adj": {
"b": "",
"a": ""
@@ -62739,7 +61525,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "",
"a": "here's"
@@ -62763,7 +61548,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FqLw57yn4fs.mp4",
"adj": {
"b": "",
"a": "to"
@@ -62787,7 +61571,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WJYyCfbjv8s.mp4",
"adj": {
"b": "",
"a": "the"
@@ -62817,7 +61600,6 @@
"why": "next word \"with\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "stuff",
"a": "with"
@@ -62857,7 +61639,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "",
"a": ""
@@ -62881,7 +61662,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "any",
"a": ""
@@ -62905,7 +61685,6 @@
"why": "next word \"he\" starts here",
"wordIn": "including",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/X70q2QJ_VTc.mp4",
"adj": {
"b": "including",
"a": "he"
@@ -62940,7 +61719,6 @@
"why": "next word \"let's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/h170V_0AbQ4.mp4",
"adj": {
"b": "food",
"a": "let's"
@@ -62975,7 +61753,6 @@
"why": "next word \"then\" starts here",
"wordIn": "CNN",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6CNFodmbWw.mp4",
"adj": {
"b": "much",
"a": "then"
@@ -63010,7 +61787,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/y6xjC3znn3g.mp4",
"adj": {
"b": "you",
"a": "probably"
@@ -63045,7 +61821,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QFUX3tJJpiI.mp4",
"adj": {
"b": "",
"a": ""
@@ -63069,7 +61844,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "got",
"a": "hurt"
@@ -63104,7 +61878,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cPKtKx3evdM.mp4",
"adj": {
"b": "well",
"a": ""
@@ -63128,7 +61901,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/azX_WsMqFqg.mp4",
"adj": {
"b": "is",
"a": ""
@@ -63152,7 +61924,6 @@
"why": "next word \"rounded\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "Quantumia",
"a": "rounded"
@@ -63182,7 +61953,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "with",
"a": "in"
@@ -63212,7 +61982,6 @@
"why": "next word \"the\" starts here",
"wordIn": "brave",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "would",
"a": "the"
@@ -63247,7 +62016,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dje5peO2Kjs.mp4",
"adj": {
"b": "",
"a": "it"
@@ -63271,7 +62039,6 @@
"why": "unvoiced tail",
"wordIn": "writing",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/igmjuXKDgvU.mp4",
"adj": {
"b": "",
"a": ""
@@ -63301,7 +62068,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "sirens",
"a": "because"
@@ -63336,7 +62102,6 @@
"why": "next word \"the\" starts here",
"wordIn": "from",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oYsPh2E-02E.mp4",
"adj": {
"b": "missing",
"a": "the"
@@ -63371,7 +62136,6 @@
"why": "next word \"them\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sfVzr3Jcro8.mp4",
"adj": {
"b": "protect",
"a": "them"
@@ -63406,7 +62170,6 @@
"why": "next word \"maybe\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "or",
"a": "maybe"
@@ -63441,7 +62204,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mkyBXrJhDrQ.mp4",
"adj": {
"b": "with",
"a": "SNAP"
@@ -63476,7 +62238,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9YQfjW1uasM.mp4",
"adj": {
"b": "equals",
"a": ""
@@ -63500,7 +62261,6 @@
"why": "unvoiced tail",
"wordIn": "media,",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TYPV_Ej1ND4.mp4",
"adj": {
"b": "social",
"a": ""
@@ -63535,7 +62295,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "but",
"a": "that's"
@@ -63565,7 +62324,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U3I_vH0h6WU.mp4",
"adj": {
"b": "casket.",
"a": ""
@@ -63589,7 +62347,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tC_WyEZzLV0.mp4",
"adj": {
"b": "like",
"a": ""
@@ -63619,7 +62376,6 @@
"why": "next word \"this\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "that",
"a": "this"
@@ -63659,7 +62415,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7uNqvETYmVI.mp4",
"adj": {
"b": "",
"a": ""
@@ -63689,7 +62444,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Key66cOeiEo.mp4",
"adj": {
"b": "know,",
"a": "for"
@@ -63713,7 +62467,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "",
"a": ""
@@ -63737,7 +62490,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-N5_5CpyZ_8.mp4",
"adj": {
"b": "",
"a": "you"
@@ -63767,7 +62519,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ts76T7tJv-U.mp4",
"adj": {
"b": "But",
"a": "you"
@@ -63797,7 +62548,6 @@
"why": "",
"wordIn": "doing",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UwD3NXKMsoQ.mp4",
"adj": {
"b": "are",
"a": ""
@@ -63832,7 +62582,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/l8y24jC9JUM.mp4",
"adj": {
"b": "trafficking",
"a": ""
@@ -63856,7 +62605,6 @@
"why": "unvoiced tail",
"wordIn": "particularly",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/G02gq5ClixA.mp4",
"adj": {
"b": "is",
"a": "frustrating"
@@ -63891,7 +62639,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rPkFK9TgI3w.mp4",
"adj": {
"b": "",
"a": "let's"
@@ -63915,7 +62662,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/V4k0ziHTGog.mp4",
"adj": {
"b": "opportunity.",
"a": "If"
@@ -63939,7 +62685,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "film",
"a": ""
@@ -63963,7 +62708,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "Slotkin",
"a": "one"
@@ -63993,7 +62737,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/q2BYaavZUHo.mp4",
"adj": {
"b": "",
"a": "I've"
@@ -64023,7 +62766,6 @@
"why": "unvoiced tail",
"wordIn": "antibacterial,",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -64053,7 +62795,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6CNFodmbWw.mp4",
"adj": {
"b": "",
"a": "and"
@@ -64083,7 +62824,6 @@
"why": "unvoiced tail",
"wordIn": "think",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HTRY0dNjZIo.mp4",
"adj": {
"b": "I",
"a": ""
@@ -64123,7 +62863,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WlC3m-9S-0E.mp4",
"adj": {
"b": "but",
"a": ""
@@ -64153,7 +62892,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7ctY0v_LEV0.mp4",
"adj": {
"b": "",
"a": "so"
@@ -64183,7 +62921,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/EeVtP-LxWaI.mp4",
"adj": {
"b": "which",
"a": "could"
@@ -64213,7 +62950,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/f5sLSteRRTQ.mp4",
"adj": {
"b": "watching",
"a": ""
@@ -64243,7 +62979,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "",
"a": "which"
@@ -64267,7 +63002,6 @@
"why": "",
"wordIn": "the,",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kfzmLf09O0w.mp4",
"adj": {
"b": "with",
"a": "you"
@@ -64297,7 +63031,6 @@
"why": "next word \"one\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/N78crpC9o1g.mp4",
"adj": {
"b": "of",
"a": "one"
@@ -64332,7 +63065,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9nKl6HY2y4Y.mp4",
"adj": {
"b": "",
"a": "transpride"
@@ -64356,7 +63088,6 @@
"why": "next word \"They're\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-N5_5CpyZ_8.mp4",
"adj": {
"b": "",
"a": "They're"
@@ -64386,7 +63117,6 @@
"why": "next word \"who\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-lSWX5qachE.mp4",
"adj": {
"b": "anyone",
"a": "who"
@@ -64426,7 +63156,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/DDRw-m4lyus.mp4",
"adj": {
"b": "",
"a": "And"
@@ -64456,7 +63185,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "",
"a": ""
@@ -64480,7 +63208,6 @@
"why": "next word \"Let's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fp_GOiz5MV0.mp4",
"adj": {
"b": "",
"a": "Let's"
@@ -64510,7 +63237,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "And",
"a": ""
@@ -64540,7 +63266,6 @@
"why": "next word \"The\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "",
"a": "The"
@@ -64575,7 +63300,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/z5IYf8iDnjA.mp4",
"adj": {
"b": "money.",
"a": "They've"
@@ -64605,7 +63329,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HPIHN8nxCM8.mp4",
"adj": {
"b": "quartering",
"a": "members"
@@ -64629,7 +63352,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dWMRXD0RJRY.mp4",
"adj": {
"b": "",
"a": "Disney"
@@ -64653,7 +63375,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mwtbQffMRic.mp4",
"adj": {
"b": "is",
"a": "again"
@@ -64683,7 +63404,6 @@
"why": "unvoiced tail",
"wordIn": "barrel",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "burn",
"a": "burn"
@@ -64718,7 +63438,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QeSmSLqLR6w.mp4",
"adj": {
"b": "",
"a": "Tim,"
@@ -64742,7 +63461,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7uNqvETYmVI.mp4",
"adj": {
"b": "additional",
"a": "features"
@@ -64766,7 +63484,6 @@
"why": "re-attack after a dip",
"wordIn": "ever",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7X-TBkGi_5s.mp4",
"adj": {
"b": "ever",
"a": ""
@@ -64801,7 +63518,6 @@
"why": "",
"wordIn": "",
"risk": "word \"Pedro\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ai46ABTRHOw.mp4",
"adj": {
"b": "",
"a": "Pedro"
@@ -64831,7 +63547,6 @@
"why": "",
"wordIn": "",
"risk": "word \"you\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bdBt-aXn-nA.mp4",
"adj": {
"b": "",
"a": "you"
@@ -64861,7 +63576,6 @@
"why": "",
"wordIn": "",
"risk": "word \"to\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dyKWsw1pmlY.mp4",
"adj": {
"b": "do",
"a": "to"
@@ -64891,7 +63605,6 @@
"why": "",
"wordIn": "",
"risk": "word \"You\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/eFU1WRc9lYM.mp4",
"adj": {
"b": "China.",
"a": "You"
@@ -64921,7 +63634,6 @@
"why": "next word \"And\" starts here",
"wordIn": "",
"risk": "word \"And\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HTRY0dNjZIo.mp4",
"adj": {
"b": "",
"a": "And"
@@ -64956,7 +63668,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "word \"the\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hUU-ueBBnDA.mp4",
"adj": {
"b": "",
"a": "the"
@@ -64986,7 +63697,6 @@
"why": "",
"wordIn": "",
"risk": "word \"you\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/jQYHkWx9XBc.mp4",
"adj": {
"b": "that",
"a": "you"
@@ -65016,7 +63726,6 @@
"why": "next word \"to\" starts here",
"wordIn": "",
"risk": "word \"to\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/L4MiLuFzGD4.mp4",
"adj": {
"b": "",
"a": "to"
@@ -65046,7 +63755,6 @@
"why": "",
"wordIn": "",
"risk": "word \"Father's\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p3_f3j9hZi4.mp4",
"adj": {
"b": "",
"a": "Father's"
@@ -65076,7 +63784,6 @@
"why": "",
"wordIn": "",
"risk": "word \"I\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/qnwEG10Tdr0.mp4",
"adj": {
"b": "way",
"a": "I"
@@ -65111,7 +63818,6 @@
"why": "next word \"Matt\" starts here",
"wordIn": "",
"risk": "word \"Matt\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/RJMe6PuMo3A.mp4",
"adj": {
"b": "from",
"a": "Matt"
@@ -65141,7 +63847,6 @@
"why": "",
"wordIn": "",
"risk": "word \"received\" plays inside the note",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sbyBeQyyrd0.mp4",
"adj": {
"b": "questions,",
"a": "received"
@@ -65171,7 +63876,6 @@
"why": "unvoiced tail",
"wordIn": "hopefully",
"risk": "a clip here was judged \"other speaker, not Jer\"",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/X70q2QJ_VTc.mp4",
"adj": {
"b": "",
"a": ""
@@ -65201,7 +63905,6 @@
"why": "",
"wordIn": "",
"risk": "74% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TiCFVgAYxBw.mp4",
"adj": {
"b": "thinking",
"a": ""
@@ -65225,7 +63928,6 @@
"why": "",
"wordIn": "",
"risk": "74% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TiCFVgAYxBw.mp4",
"adj": {
"b": "",
"a": "direct"
@@ -65249,7 +63951,6 @@
"why": "",
"wordIn": "",
"risk": "74% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TiCFVgAYxBw.mp4",
"adj": {
"b": "interesting",
"a": "case."
@@ -65273,7 +63974,6 @@
"why": "",
"wordIn": "",
"risk": "74% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TiCFVgAYxBw.mp4",
"adj": {
"b": "the",
"a": "comic"
@@ -65297,7 +63997,6 @@
"why": "",
"wordIn": "",
"risk": "74% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TiCFVgAYxBw.mp4",
"adj": {
"b": "wingers.",
"a": "And"
@@ -65321,7 +64020,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "74% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TiCFVgAYxBw.mp4",
"adj": {
"b": "",
"a": ""
@@ -65345,7 +64043,6 @@
"why": "",
"wordIn": "",
"risk": "74% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TiCFVgAYxBw.mp4",
"adj": {
"b": "speak,",
"a": "Ethan"
@@ -65375,7 +64072,6 @@
"why": "",
"wordIn": "",
"risk": "74% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TiCFVgAYxBw.mp4",
"adj": {
"b": "the",
"a": "Brett"
@@ -65405,7 +64101,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "74% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TiCFVgAYxBw.mp4",
"adj": {
"b": "",
"a": "And"
@@ -65429,7 +64124,6 @@
"why": "",
"wordIn": "",
"risk": "84% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6tHdH63lFE.mp4",
"adj": {
"b": "Like",
"a": ""
@@ -65459,7 +64153,6 @@
"why": "",
"wordIn": "it's",
"risk": "84% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6tHdH63lFE.mp4",
"adj": {
"b": "",
"a": ""
@@ -65489,7 +64182,6 @@
"why": "",
"wordIn": "",
"risk": "84% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6tHdH63lFE.mp4",
"adj": {
"b": "The",
"a": ""
@@ -65519,7 +64211,6 @@
"why": "",
"wordIn": "",
"risk": "84% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6tHdH63lFE.mp4",
"adj": {
"b": "this",
"a": "somebody's"
@@ -65543,7 +64234,6 @@
"why": "",
"wordIn": "",
"risk": "84% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6tHdH63lFE.mp4",
"adj": {
"b": "",
"a": ""
@@ -65567,7 +64257,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "84% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6tHdH63lFE.mp4",
"adj": {
"b": "",
"a": ""
@@ -65591,7 +64280,6 @@
"why": "",
"wordIn": "",
"risk": "84% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6tHdH63lFE.mp4",
"adj": {
"b": "",
"a": ""
@@ -65615,7 +64303,6 @@
"why": "",
"wordIn": "",
"risk": "84% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6tHdH63lFE.mp4",
"adj": {
"b": "",
"a": ""
@@ -65639,7 +64326,6 @@
"why": "",
"wordIn": "",
"risk": "84% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6tHdH63lFE.mp4",
"adj": {
"b": "",
"a": ""
@@ -65663,7 +64349,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "84% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6tHdH63lFE.mp4",
"adj": {
"b": "",
"a": "Did"
@@ -65687,7 +64372,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "84% of this video's clips were rejected (average is 29%)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6tHdH63lFE.mp4",
"adj": {
"b": "",
"a": ""
@@ -65711,7 +64395,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +251 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-CcZe5Bffzs.mp4",
"adj": {
"b": "",
"a": ""
@@ -65735,7 +64418,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +310 cents from the corpus (123Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1Sj0UBEXX0c.mp4",
"adj": {
"b": "",
"a": ""
@@ -65759,7 +64441,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +310 cents from the corpus (123Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1Sj0UBEXX0c.mp4",
"adj": {
"b": "",
"a": ""
@@ -65783,7 +64464,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +310 cents from the corpus (123Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1Sj0UBEXX0c.mp4",
"adj": {
"b": "but",
"a": ""
@@ -65813,7 +64493,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +380 cents from the corpus (128Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Api44x4HVGs.mp4",
"adj": {
"b": "",
"a": "Her"
@@ -65837,7 +64516,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +380 cents from the corpus (128Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Api44x4HVGs.mp4",
"adj": {
"b": "comedy,",
"a": "Lily"
@@ -65861,7 +64539,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +380 cents from the corpus (128Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Api44x4HVGs.mp4",
"adj": {
"b": "",
"a": "I"
@@ -65885,7 +64562,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +380 cents from the corpus (128Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Api44x4HVGs.mp4",
"adj": {
"b": "been",
"a": ""
@@ -65909,7 +64585,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +380 cents from the corpus (128Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Api44x4HVGs.mp4",
"adj": {
"b": "",
"a": ""
@@ -65933,7 +64608,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +380 cents from the corpus (128Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Api44x4HVGs.mp4",
"adj": {
"b": "bait",
"a": ""
@@ -65957,7 +64631,6 @@
"why": "",
"wordIn": "and",
"risk": "this video's clips sit +259 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BKUP9tal4mM.mp4",
"adj": {
"b": "player",
"a": "she's"
@@ -65987,7 +64660,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +259 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BKUP9tal4mM.mp4",
"adj": {
"b": "apparently",
"a": "or"
@@ -66017,7 +64689,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +392 cents from the corpus (129Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OlY74Y-NmMw.mp4",
"adj": {
"b": "",
"a": "taking"
@@ -66041,7 +64712,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +392 cents from the corpus (129Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OlY74Y-NmMw.mp4",
"adj": {
"b": "",
"a": "Some"
@@ -66071,7 +64741,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +392 cents from the corpus (129Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OlY74Y-NmMw.mp4",
"adj": {
"b": "",
"a": "She's"
@@ -66095,7 +64764,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +392 cents from the corpus (129Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OlY74Y-NmMw.mp4",
"adj": {
"b": "who",
"a": "are"
@@ -66130,7 +64798,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit -246 cents from the corpus (89Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Oo6dXSEh65Y.mp4",
"adj": {
"b": "",
"a": ""
@@ -66154,7 +64821,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "that",
"a": ""
@@ -66178,7 +64844,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "there",
"a": "so"
@@ -66202,7 +64867,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "",
"a": ""
@@ -66226,7 +64890,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "",
"a": "a"
@@ -66250,7 +64913,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "protection?",
"a": "Can"
@@ -66274,7 +64936,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "use.",
"a": "And"
@@ -66298,7 +64959,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "",
"a": ""
@@ -66322,7 +64982,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "a",
"a": ""
@@ -66346,7 +65005,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "people",
"a": ""
@@ -66376,7 +65034,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "need",
"a": "tradespeople"
@@ -66400,7 +65057,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "trade.",
"a": ""
@@ -66424,7 +65080,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "",
"a": "you"
@@ -66448,7 +65103,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "faith",
"a": ""
@@ -66472,7 +65126,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "",
"a": ""
@@ -66496,7 +65149,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "adequate",
"a": ""
@@ -66520,7 +65172,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "",
"a": ""
@@ -66544,7 +65195,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "",
"a": ""
@@ -66568,7 +65218,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "this video's clips sit -368 cents from the corpus (83Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "history",
"a": ""
@@ -66592,7 +65241,6 @@
"why": "",
"wordIn": "Uh,",
"risk": "this video's clips sit +252 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "",
"a": "this"
@@ -66622,7 +65270,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +252 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "",
"a": "somewhere"
@@ -66646,7 +65293,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +252 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "",
"a": "I"
@@ -66670,7 +65316,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +252 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "that",
"a": "I"
@@ -66694,7 +65339,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +252 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "",
"a": "might"
@@ -66718,7 +65362,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +252 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "",
"a": ""
@@ -66742,7 +65385,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +252 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "",
"a": ""
@@ -66766,7 +65408,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +252 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "",
"a": ""
@@ -66790,7 +65431,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +252 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "one",
"a": "to"
@@ -66814,7 +65454,6 @@
"why": "",
"wordIn": "",
"risk": "this video's clips sit +252 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "",
"a": "anime"
@@ -66838,7 +65477,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +252 cents from the corpus (119Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vimidN7iVN0.mp4",
"adj": {
"b": "",
"a": ""
@@ -66862,7 +65500,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "this video's clips sit +240 cents from the corpus (118Hz vs 103Hz) -- may be another speaker",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cwMkQSs6OeM.mp4",
"adj": {
"b": "",
"a": ""
@@ -66886,7 +65523,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (752ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-fo3pHhPviA.mp4",
"adj": {
"b": "that",
"a": ""
@@ -66910,7 +65546,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (702ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0A4dIN1dKaY.mp4",
"adj": {
"b": "",
"a": "embracing"
@@ -66934,7 +65569,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (627ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0A4dIN1dKaY.mp4",
"adj": {
"b": "",
"a": ""
@@ -66958,7 +65592,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (607ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1UgCLhoVIfs.mp4",
"adj": {
"b": "cool",
"a": ""
@@ -66982,7 +65615,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (602ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1UgCLhoVIfs.mp4",
"adj": {
"b": "",
"a": "then"
@@ -67012,7 +65644,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "never shown to you, and long (622ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/2BNd7dG7Fqc.mp4",
"adj": {
"b": "",
"a": ""
@@ -67036,7 +65667,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (722ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/2BNd7dG7Fqc.mp4",
"adj": {
"b": "",
"a": ""
@@ -67060,7 +65690,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (712ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/2BNd7dG7Fqc.mp4",
"adj": {
"b": "it.",
"a": "You"
@@ -67084,7 +65713,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (742ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/2TXVYOkTJQc.mp4",
"adj": {
"b": "",
"a": ""
@@ -67108,7 +65736,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (617ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/2TXVYOkTJQc.mp4",
"adj": {
"b": "",
"a": ""
@@ -67132,7 +65759,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (702ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/2TXVYOkTJQc.mp4",
"adj": {
"b": "",
"a": ""
@@ -67156,7 +65782,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (657ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "",
"a": ""
@@ -67180,7 +65805,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (717ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "what",
"a": ""
@@ -67204,7 +65828,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (717ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "available.",
"a": ""
@@ -67228,7 +65851,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (632ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "",
"a": ""
@@ -67252,7 +65874,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (817ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "",
"a": ""
@@ -67276,7 +65897,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (622ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "",
"a": ""
@@ -67300,7 +65920,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (662ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "",
"a": ""
@@ -67324,7 +65943,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (677ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "but",
"a": ""
@@ -67348,7 +65966,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (632ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7lOQD2k3erE.mp4",
"adj": {
"b": "Veebbs,",
"a": "promo"
@@ -67372,7 +65989,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (612ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7uNqvETYmVI.mp4",
"adj": {
"b": "Like",
"a": ""
@@ -67396,7 +66012,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (622ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7zRNtT2ygCc.mp4",
"adj": {
"b": "too",
"a": "we"
@@ -67420,7 +66035,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (662ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8PNFWjrpXrE.mp4",
"adj": {
"b": "",
"a": ""
@@ -67444,7 +66058,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (622ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9nKl6HY2y4Y.mp4",
"adj": {
"b": "",
"a": ""
@@ -67468,7 +66081,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (617ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9SgVb3vhRVo.mp4",
"adj": {
"b": "",
"a": ""
@@ -67492,7 +66104,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (637ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9SgVb3vhRVo.mp4",
"adj": {
"b": "dumb",
"a": ""
@@ -67522,7 +66133,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (657ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9SgVb3vhRVo.mp4",
"adj": {
"b": "",
"a": ""
@@ -67546,7 +66156,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (647ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9SgVb3vhRVo.mp4",
"adj": {
"b": "is",
"a": ""
@@ -67576,7 +66185,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (617ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9SgVb3vhRVo.mp4",
"adj": {
"b": "and",
"a": "you"
@@ -67600,7 +66208,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (747ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ai46ABTRHOw.mp4",
"adj": {
"b": "probably",
"a": ""
@@ -67624,7 +66231,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (757ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ai46ABTRHOw.mp4",
"adj": {
"b": "about",
"a": ""
@@ -67648,7 +66254,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (717ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ai46ABTRHOw.mp4",
"adj": {
"b": "",
"a": ""
@@ -67672,7 +66277,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (662ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ai46ABTRHOw.mp4",
"adj": {
"b": "know",
"a": ""
@@ -67696,7 +66300,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (787ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/aU5TsjpTD8Y.mp4",
"adj": {
"b": "",
"a": ""
@@ -67720,7 +66323,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (657ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/aU5TsjpTD8Y.mp4",
"adj": {
"b": "",
"a": ""
@@ -67744,7 +66346,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (642ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/aU5TsjpTD8Y.mp4",
"adj": {
"b": "",
"a": "not"
@@ -67768,7 +66369,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (717ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bbVoMOzCSs0.mp4",
"adj": {
"b": "no",
"a": ""
@@ -67792,7 +66392,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (652ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bL3D0K-kGb8.mp4",
"adj": {
"b": "",
"a": "stories"
@@ -67816,7 +66415,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "never shown to you, and long (617ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BnPz37ecO58.mp4",
"adj": {
"b": "",
"a": ""
@@ -67840,7 +66438,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (697ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/DDRw-m4lyus.mp4",
"adj": {
"b": "put",
"a": ""
@@ -67864,7 +66461,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (612ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/DDRw-m4lyus.mp4",
"adj": {
"b": "had",
"a": ""
@@ -67888,7 +66484,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (637ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/DDRw-m4lyus.mp4",
"adj": {
"b": "how",
"a": ""
@@ -67912,7 +66507,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (622ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dje5peO2Kjs.mp4",
"adj": {
"b": "to",
"a": ""
@@ -67936,7 +66530,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (697ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dMs5wvpy1Qk.mp4",
"adj": {
"b": "and",
"a": "you're"
@@ -67960,7 +66553,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "never shown to you, and long (797ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dMs5wvpy1Qk.mp4",
"adj": {
"b": "faced",
"a": ""
@@ -67984,7 +66576,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (712ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "after",
"a": "Russia"
@@ -68008,7 +66599,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (667ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "",
"a": ""
@@ -68032,7 +66622,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (752ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "",
"a": ""
@@ -68056,7 +66645,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (642ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "",
"a": "Trump"
@@ -68080,7 +66668,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (787ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dyKWsw1pmlY.mp4",
"adj": {
"b": "eighty",
"a": ""
@@ -68104,7 +66691,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (747ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dyKWsw1pmlY.mp4",
"adj": {
"b": "",
"a": ""
@@ -68128,7 +66714,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (772ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/eEg1SRZj1rA.mp4",
"adj": {
"b": "",
"a": "you"
@@ -68152,7 +66737,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (687ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/eEg1SRZj1rA.mp4",
"adj": {
"b": "",
"a": ""
@@ -68176,7 +66760,6 @@
"why": "",
"wordIn": "",
"risk": "never shown to you, and long (622ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/eEg1SRZj1rA.mp4",
"adj": {
"b": "",
"a": ""
@@ -68200,7 +66783,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (922ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/eFU1WRc9lYM.mp4",
"adj": {
"b": "",
"a": ""
@@ -68224,7 +66806,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "never shown to you, and long (642ms)",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/F280ds-NS9A.mp4",
"adj": {
"b": "",
"a": ""
@@ -68248,7 +66829,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "and",
"a": ""
@@ -68278,7 +66858,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OBidX_oE7zM.mp4",
"adj": {
"b": "",
"a": ""
@@ -68302,7 +66881,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cwMkQSs6OeM.mp4",
"adj": {
"b": "",
"a": ""
@@ -68326,7 +66904,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Nb5iswtRmVM.mp4",
"adj": {
"b": "and",
"a": ""
@@ -68356,7 +66933,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mwtbQffMRic.mp4",
"adj": {
"b": "Antifa",
"a": "check"
@@ -68380,7 +66956,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-pKXHyeyRhQ.mp4",
"adj": {
"b": "and",
"a": "actually"
@@ -68410,7 +66985,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "and",
"a": ""
@@ -68440,7 +67014,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/f5sLSteRRTQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -68470,7 +67043,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/X70q2QJ_VTc.mp4",
"adj": {
"b": "hopefully",
"a": ""
@@ -68494,7 +67066,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ep_qtW2K3BY.mp4",
"adj": {
"b": "driver",
"a": "and"
@@ -68529,7 +67100,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "use.",
"a": "And"
@@ -68553,7 +67123,6 @@
"why": "next word \"the\" starts here",
"wordIn": "read",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "can",
"a": "the"
@@ -68598,7 +67167,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1Sj0UBEXX0c.mp4",
"adj": {
"b": "",
"a": ""
@@ -68622,7 +67190,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "that",
"a": "I"
@@ -68646,7 +67213,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TiCFVgAYxBw.mp4",
"adj": {
"b": "the",
"a": "Brett"
@@ -68676,7 +67242,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Api44x4HVGs.mp4",
"adj": {
"b": "bait",
"a": ""
@@ -68706,7 +67271,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Api44x4HVGs.mp4",
"adj": {
"b": "",
"a": "along"
@@ -68730,7 +67294,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "history",
"a": ""
@@ -68754,7 +67317,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OlY74Y-NmMw.mp4",
"adj": {
"b": "who",
"a": "are"
@@ -68789,7 +67351,6 @@
"why": "next word \"It's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nM5qgOKOEsQ.mp4",
"adj": {
"b": "knife.",
"a": "It's"
@@ -68824,7 +67385,6 @@
"why": "unvoiced tail",
"wordIn": "down",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/F280ds-NS9A.mp4",
"adj": {
"b": "down",
"a": ""
@@ -68859,7 +67419,6 @@
"why": "next word \"using\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QFUX3tJJpiI.mp4",
"adj": {
"b": "they're",
"a": "using"
@@ -68894,7 +67453,6 @@
"why": "re-attack after a dip",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UwD3NXKMsoQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -68924,7 +67482,6 @@
"why": "next word \"we\" starts here",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "there",
"a": "we"
@@ -68974,7 +67531,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "",
"a": "or"
@@ -68998,7 +67554,6 @@
"why": "next word \"though\" starts here",
"wordIn": "seeing",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mwJ7GcRGbyY.mp4",
"adj": {
"b": "",
"a": "though"
@@ -69033,7 +67588,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/A9Zg-xqDIcM.mp4",
"adj": {
"b": "",
"a": "but"
@@ -69057,7 +67611,6 @@
"why": "next word \"remember\" starts here",
"wordIn": "y'all",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "do",
"a": "remember"
@@ -69092,7 +67645,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/y6xjC3znn3g.mp4",
"adj": {
"b": "and",
"a": "if"
@@ -69127,7 +67679,6 @@
"why": "unvoiced tail",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QFUX3tJJpiI.mp4",
"adj": {
"b": "president",
"a": "want"
@@ -69157,7 +67708,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/93DFKk1eSoM.mp4",
"adj": {
"b": "",
"a": "the"
@@ -69187,7 +67737,6 @@
"why": "next word \"a\" starts here",
"wordIn": "from",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/6xZktYfRWMQ.mp4",
"adj": {
"b": "prank",
"a": "a"
@@ -69227,7 +67776,6 @@
"why": "next word \"gonna\" starts here",
"wordIn": "I'm",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KtZaCYPSunM.mp4",
"adj": {
"b": "Hey,",
"a": "gonna"
@@ -69267,7 +67815,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/zoWmEz419Kw.mp4",
"adj": {
"b": "which",
"a": ""
@@ -69297,7 +67844,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UqCmkStyKow.mp4",
"adj": {
"b": "people",
"a": ""
@@ -69332,7 +67878,6 @@
"why": "next word \"This\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-lSWX5qachE.mp4",
"adj": {
"b": "",
"a": "This"
@@ -69367,7 +67912,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6CNFodmbWw.mp4",
"adj": {
"b": "",
"a": ""
@@ -69391,7 +67935,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/W82jEF0kNqs.mp4",
"adj": {
"b": "that",
"a": ""
@@ -69421,7 +67964,6 @@
"why": "re-attack after a dip",
"wordIn": "here's",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/E_K0pCUIiCI.mp4",
"adj": {
"b": "",
"a": "Sarah"
@@ -69451,7 +67993,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/f5sLSteRRTQ.mp4",
"adj": {
"b": "at",
"a": "as"
@@ -69475,7 +68016,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "",
"a": ""
@@ -69499,7 +68039,6 @@
"why": "unvoiced tail",
"wordIn": "in",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/XnGbMXzvlGY.mp4",
"adj": {
"b": "but",
"a": ""
@@ -69529,7 +68068,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TTN8zF4Vjz4.mp4",
"adj": {
"b": "",
"a": "So"
@@ -69553,7 +68091,6 @@
"why": "next word \"that\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "was",
"a": "that"
@@ -69593,7 +68130,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/q2BYaavZUHo.mp4",
"adj": {
"b": "",
"a": "they're"
@@ -69623,7 +68159,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/G02gq5ClixA.mp4",
"adj": {
"b": "",
"a": "and"
@@ -69647,7 +68182,6 @@
"why": "unvoiced tail",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/6A1JPaiqJNY.mp4",
"adj": {
"b": "",
"a": ""
@@ -69677,7 +68211,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZEwkRomA9pU.mp4",
"adj": {
"b": "more",
"a": "gave"
@@ -69707,7 +68240,6 @@
"why": "next word \"on\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m_B_3i9yKX0.mp4",
"adj": {
"b": "indirectly",
"a": "on"
@@ -69742,7 +68274,6 @@
"why": "next word \"hundred\" starts here",
"wordIn": "five",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3TjJISVbWVg.mp4",
"adj": {
"b": "thousand",
"a": "hundred"
@@ -69777,7 +68308,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7zRNtT2ygCc.mp4",
"adj": {
"b": "air",
"a": "is"
@@ -69801,7 +68331,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7lOQD2k3erE.mp4",
"adj": {
"b": "solving",
"a": "real"
@@ -69841,7 +68370,6 @@
"why": "re-attack after a dip",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/zg5LTpYUrlg.mp4",
"adj": {
"b": "",
"a": ""
@@ -69865,7 +68393,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "",
"a": ""
@@ -69889,7 +68416,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KyzBZ19iRaU.mp4",
"adj": {
"b": "",
"a": "the"
@@ -69919,7 +68445,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZEwkRomA9pU.mp4",
"adj": {
"b": "",
"a": "she's"
@@ -69949,7 +68474,6 @@
"why": "next word \"in\" starts here",
"wordIn": "team",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rPkFK9TgI3w.mp4",
"adj": {
"b": "best",
"a": "in"
@@ -69999,7 +68523,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/IMsq0B-xJ7M.mp4",
"adj": {
"b": "",
"a": "people's"
@@ -70023,7 +68546,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Tvpplznw9Nw.mp4",
"adj": {
"b": "",
"a": ""
@@ -70047,7 +68569,6 @@
"why": "next word \"I\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WqAu8gLaW-4.mp4",
"adj": {
"b": "",
"a": "I"
@@ -70077,7 +68598,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/aeC-_9DQWM4.mp4",
"adj": {
"b": "is",
"a": ""
@@ -70107,7 +68627,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/aeC-_9DQWM4.mp4",
"adj": {
"b": "and",
"a": "these"
@@ -70142,7 +68661,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/NGEPcloQ_0Q.mp4",
"adj": {
"b": "with",
"a": "Marjorie"
@@ -70166,7 +68684,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "",
"a": "ABZ"
@@ -70196,7 +68713,6 @@
"why": "next word \"of\" starts here",
"wordIn": "one",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xbT-lWAzaiA.mp4",
"adj": {
"b": "on",
"a": "of"
@@ -70241,7 +68757,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "",
"a": ""
@@ -70271,7 +68786,6 @@
"why": "unvoiced tail",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/HPIHN8nxCM8.mp4",
"adj": {
"b": "people",
"a": "go"
@@ -70306,7 +68820,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "stuff",
"a": "So"
@@ -70336,7 +68849,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/zoWmEz419Kw.mp4",
"adj": {
"b": "",
"a": ""
@@ -70366,7 +68878,6 @@
"why": "next word \"He\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "",
"a": "He"
@@ -70396,7 +68907,6 @@
"why": "next word \"He's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "Kayton.",
"a": "He's"
@@ -70426,7 +68936,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7zRNtT2ygCc.mp4",
"adj": {
"b": "",
"a": "getting"
@@ -70450,7 +68959,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9zEisQcriiM.mp4",
"adj": {
"b": "the",
"a": ""
@@ -70480,7 +68988,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LiTjFiYApxg.mp4",
"adj": {
"b": "",
"a": "apparently"
@@ -70504,7 +69011,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dWMRXD0RJRY.mp4",
"adj": {
"b": "that",
"a": "in"
@@ -70528,7 +69034,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/So4TRDhr3M0.mp4",
"adj": {
"b": "",
"a": ""
@@ -70552,7 +69057,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dhOXEakOpaM.mp4",
"adj": {
"b": "mostly",
"a": "maim,"
@@ -70587,7 +69091,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "competitor",
"a": "the"
@@ -70622,7 +69125,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/67Qd5J47Y84.mp4",
"adj": {
"b": "center",
"a": "unasssoated"
@@ -70646,7 +69148,6 @@
"why": "next word \"as\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/P4JErOtgVuA.mp4",
"adj": {
"b": "and",
"a": "as"
@@ -70681,7 +69182,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Hh71Xe7XK6k.mp4",
"adj": {
"b": "",
"a": ""
@@ -70705,7 +69205,6 @@
"why": "unvoiced tail",
"wordIn": "have",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/VNdJ8BvruCQ.mp4",
"adj": {
"b": "have",
"a": "oh"
@@ -70745,7 +69244,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "marketing",
"a": ""
@@ -70775,7 +69273,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fp_GOiz5MV0.mp4",
"adj": {
"b": "",
"a": ""
@@ -70799,7 +69296,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fp_GOiz5MV0.mp4",
"adj": {
"b": "administration",
"a": "you"
@@ -70829,7 +69325,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/W82jEF0kNqs.mp4",
"adj": {
"b": "have",
"a": "you've"
@@ -70859,7 +69354,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/GdMSdVvQvtc.mp4",
"adj": {
"b": "way",
"a": "you"
@@ -70899,7 +69393,6 @@
"why": "unvoiced tail",
"wordIn": "he's",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dVSd_fotB2U.mp4",
"adj": {
"b": "like",
"a": ""
@@ -70934,7 +69427,6 @@
"why": "unvoiced tail",
"wordIn": "absolutely",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/JKozJKyP4Iw.mp4",
"adj": {
"b": "was",
"a": ""
@@ -70969,7 +69461,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/MXHk10OCoow.mp4",
"adj": {
"b": "that",
"a": "former"
@@ -70999,7 +69490,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oYsPh2E-02E.mp4",
"adj": {
"b": "",
"a": "you"
@@ -71029,7 +69519,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OlY74Y-NmMw.mp4",
"adj": {
"b": "",
"a": "she"
@@ -71059,7 +69548,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nrKQ-WoQ_-Y.mp4",
"adj": {
"b": "",
"a": "he"
@@ -71089,7 +69577,6 @@
"why": "unvoiced tail",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TTN8zF4Vjz4.mp4",
"adj": {
"b": "listen",
"a": ""
@@ -71124,7 +69611,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TiCFVgAYxBw.mp4",
"adj": {
"b": "",
"a": ""
@@ -71148,7 +69634,6 @@
"why": "next word \"You\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8RiHnmAGy8U.mp4",
"adj": {
"b": "illegal.",
"a": "You"
@@ -71183,7 +69668,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "bag",
"a": "the"
@@ -71218,7 +69702,6 @@
"why": "next word \"or\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/r3cTXMxBPbc.mp4",
"adj": {
"b": "out",
"a": "or"
@@ -71248,7 +69731,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xaWnBmX4AjY.mp4",
"adj": {
"b": "names",
"a": "and"
@@ -71272,7 +69754,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "up",
"a": "you"
@@ -71307,7 +69788,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KssiwNCTEKQ.mp4",
"adj": {
"b": "",
"a": "so"
@@ -71331,7 +69811,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/aU5TsjpTD8Y.mp4",
"adj": {
"b": "all",
"a": "produced"
@@ -71361,7 +69840,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xulBQwriW5E.mp4",
"adj": {
"b": "",
"a": "reporter,"
@@ -71385,7 +69863,6 @@
"why": "unvoiced tail",
"wordIn": "meal",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xaWnBmX4AjY.mp4",
"adj": {
"b": "",
"a": ""
@@ -71415,7 +69892,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ibJe2PlNOS8.mp4",
"adj": {
"b": "that",
"a": "I'm"
@@ -71450,7 +69926,6 @@
"why": "next word \"what\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ybLrSiTQxtQ.mp4",
"adj": {
"b": "like",
"a": "what"
@@ -71485,7 +69960,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/IMsq0B-xJ7M.mp4",
"adj": {
"b": "",
"a": "politics"
@@ -71515,7 +69989,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xaWnBmX4AjY.mp4",
"adj": {
"b": "more",
"a": "important"
@@ -71545,7 +70018,6 @@
"why": "next word \"Vanity\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "",
"a": "Vanity"
@@ -71575,7 +70047,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ai46ABTRHOw.mp4",
"adj": {
"b": "like",
"a": "you"
@@ -71605,7 +70076,6 @@
"why": "next word \"to\" starts here",
"wordIn": "going",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "I'm",
"a": "to"
@@ -71655,7 +70125,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BKUP9tal4mM.mp4",
"adj": {
"b": "",
"a": "OF"
@@ -71685,7 +70154,6 @@
"why": "next word \"photo\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UjzBIdbk2zY.mp4",
"adj": {
"b": "a",
"a": "photo"
@@ -71720,7 +70188,6 @@
"why": "next word \"He\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "him.",
"a": "He"
@@ -71750,7 +70217,6 @@
"why": "next word \"LA\" starts here",
"wordIn": "the",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UCanpMuiH2U.mp4",
"adj": {
"b": "for",
"a": "LA"
@@ -71790,7 +70256,6 @@
"why": "next word \"Many\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/q2BYaavZUHo.mp4",
"adj": {
"b": "",
"a": "Many"
@@ -71830,7 +70295,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ibJe2PlNOS8.mp4",
"adj": {
"b": "",
"a": "he"
@@ -71860,7 +70324,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9nKl6HY2y4Y.mp4",
"adj": {
"b": "",
"a": ""
@@ -71884,7 +70347,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/aeC-_9DQWM4.mp4",
"adj": {
"b": "Minnesota",
"a": ""
@@ -71908,7 +70370,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/K9rXg1newxo.mp4",
"adj": {
"b": "and",
"a": "made"
@@ -71943,7 +70404,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BKUP9tal4mM.mp4",
"adj": {
"b": "biologically.",
"a": "And"
@@ -71967,7 +70427,6 @@
"why": "unvoiced tail",
"wordIn": "And",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8hsc6aSGdk.mp4",
"adj": {
"b": "",
"a": "I"
@@ -71997,7 +70456,6 @@
"why": "next word \"Our\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8ez7u2StoWI.mp4",
"adj": {
"b": "it.",
"a": "Our"
@@ -72032,7 +70490,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "",
"a": "can't"
@@ -72056,7 +70513,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sfVzr3Jcro8.mp4",
"adj": {
"b": "",
"a": ""
@@ -72086,7 +70542,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fjLMZwGB4Ds.mp4",
"adj": {
"b": "",
"a": "it's"
@@ -72110,7 +70565,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/waF76PwdlS8.mp4",
"adj": {
"b": "",
"a": "here"
@@ -72140,7 +70594,6 @@
"why": "next word \"in\" starts here",
"wordIn": "down",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3TjJISVbWVg.mp4",
"adj": {
"b": "doubling",
"a": "in"
@@ -72185,7 +70638,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vfMRAdhSSrc.mp4",
"adj": {
"b": "News",
"a": "Some"
@@ -72209,7 +70661,6 @@
"why": "next word \"dude\" starts here",
"wordIn": "this",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oAj5BoNKTYQ.mp4",
"adj": {
"b": "",
"a": "dude"
@@ -72244,7 +70695,6 @@
"why": "unvoiced tail",
"wordIn": "Hmm",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BKUP9tal4mM.mp4",
"adj": {
"b": "",
"a": ""
@@ -72274,7 +70724,6 @@
"why": "",
"wordIn": "take",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/V4k0ziHTGog.mp4",
"adj": {
"b": "to",
"a": ""
@@ -72309,7 +70758,6 @@
"why": "next word \"was\" starts here",
"wordIn": "claiming",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ep_qtW2K3BY.mp4",
"adj": {
"b": "for",
"a": "was"
@@ -72354,7 +70802,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WzlVDuwup0U.mp4",
"adj": {
"b": "nine",
"a": ""
@@ -72378,7 +70825,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dWMRXD0RJRY.mp4",
"adj": {
"b": "positions",
"a": "while"
@@ -72402,7 +70848,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/X70q2QJ_VTc.mp4",
"adj": {
"b": "",
"a": "you"
@@ -72432,7 +70877,6 @@
"why": "next word \"Joshua\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "",
"a": "Joshua"
@@ -72462,7 +70906,6 @@
"why": "next word \"If\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yjHHiRhtMvQ.mp4",
"adj": {
"b": "there.",
"a": "If"
@@ -72492,7 +70935,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CTbCFWZ7Gdk.mp4",
"adj": {
"b": "me",
"a": "IV"
@@ -72522,7 +70964,6 @@
"why": "next word \"made\" starts here",
"wordIn": "it",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ep_qtW2K3BY.mp4",
"adj": {
"b": "that",
"a": "made"
@@ -72557,7 +70998,6 @@
"why": "unvoiced tail",
"wordIn": "him",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZeqS0DvnorA.mp4",
"adj": {
"b": "calling",
"a": ""
@@ -72592,7 +71032,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/l55Ad1gO5ww.mp4",
"adj": {
"b": "",
"a": "last"
@@ -72622,7 +71061,6 @@
"why": "next word \"video\" starts here",
"wordIn": "the",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dhOXEakOpaM.mp4",
"adj": {
"b": "the",
"a": "video"
@@ -72667,7 +71105,6 @@
"why": "next word \"the\" starts here",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/jQYHkWx9XBc.mp4",
"adj": {
"b": "",
"a": "the"
@@ -72702,7 +71139,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mxq6w_cMG6c.mp4",
"adj": {
"b": "know",
"a": ""
@@ -72737,7 +71173,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "and",
"a": "you"
@@ -72772,7 +71207,6 @@
"why": "next word \"video\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "another",
"a": "video"
@@ -72807,7 +71241,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hykdhazn8ds.mp4",
"adj": {
"b": "",
"a": "well"
@@ -72837,7 +71270,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WlC3m-9S-0E.mp4",
"adj": {
"b": "",
"a": "baffles"
@@ -72867,7 +71299,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/34mSBDoWxKc.mp4",
"adj": {
"b": "",
"a": "where"
@@ -72897,7 +71328,6 @@
"why": "unvoiced tail",
"wordIn": "when",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/e0d6EzTEt7Y.mp4",
"adj": {
"b": "or",
"a": "RC"
@@ -72932,7 +71362,6 @@
"why": "next word \"Here's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "",
"a": "Here's"
@@ -72962,7 +71391,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "like",
"a": ""
@@ -72997,7 +71425,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TiCFVgAYxBw.mp4",
"adj": {
"b": "",
"a": "More"
@@ -73027,7 +71454,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/RT0mKx_hTv4.mp4",
"adj": {
"b": "out",
"a": "you"
@@ -73062,7 +71488,6 @@
"why": "unvoiced tail",
"wordIn": "but",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cN5k-3sfBdA.mp4",
"adj": {
"b": "video",
"a": ""
@@ -73092,7 +71517,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/67Qd5J47Y84.mp4",
"adj": {
"b": "totally",
"a": "unreasonable"
@@ -73127,7 +71551,6 @@
"why": "next word \"tractor\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fjLMZwGB4Ds.mp4",
"adj": {
"b": "",
"a": "tractor"
@@ -73157,7 +71580,6 @@
"why": "unvoiced tail",
"wordIn": "the",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oAj5BoNKTYQ.mp4",
"adj": {
"b": "",
"a": "that's"
@@ -73192,7 +71614,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cN5k-3sfBdA.mp4",
"adj": {
"b": "like",
"a": "you"
@@ -73232,7 +71653,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/azX_WsMqFqg.mp4",
"adj": {
"b": "",
"a": ""
@@ -73256,7 +71676,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "",
"a": "a"
@@ -73286,7 +71705,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "Now",
"a": "what"
@@ -73316,7 +71734,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xulBQwriW5E.mp4",
"adj": {
"b": "",
"a": ""
@@ -73340,7 +71757,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cN5k-3sfBdA.mp4",
"adj": {
"b": "",
"a": ""
@@ -73364,7 +71780,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/MyXmQNqa2sY.mp4",
"adj": {
"b": "",
"a": "the"
@@ -73399,7 +71814,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pdneh4I4KcQ.mp4",
"adj": {
"b": "freeze,",
"a": "you"
@@ -73434,7 +71848,6 @@
"why": "next word \"these\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7lOQD2k3erE.mp4",
"adj": {
"b": "",
"a": "these"
@@ -73464,7 +71877,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ai46ABTRHOw.mp4",
"adj": {
"b": "to",
"a": ""
@@ -73488,7 +71900,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/aU5TsjpTD8Y.mp4",
"adj": {
"b": "the",
"a": "Amber"
@@ -73518,7 +71929,6 @@
"why": "",
"wordIn": "or",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "description",
"a": "pinn"
@@ -73553,7 +71963,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "urging",
"a": "while"
@@ -73588,7 +71997,6 @@
"why": "next word \"they're\" starts here",
"wordIn": "where",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UqCmkStyKow.mp4",
"adj": {
"b": "money,",
"a": "they're"
@@ -73623,7 +72031,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QqIWnAhGwfw.mp4",
"adj": {
"b": "",
"a": "and"
@@ -73647,7 +72054,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7X-TBkGi_5s.mp4",
"adj": {
"b": "",
"a": ""
@@ -73671,7 +72077,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "However,",
"a": "there's"
@@ -73701,7 +72106,6 @@
"why": "",
"wordIn": "individual",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3gORoyt2mC4.mp4",
"adj": {
"b": "trans",
"a": ""
@@ -73731,7 +72135,6 @@
"why": "next word \"a\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UqCmkStyKow.mp4",
"adj": {
"b": "then",
"a": "a"
@@ -73761,7 +72164,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7X-TBkGi_5s.mp4",
"adj": {
"b": "",
"a": ""
@@ -73785,7 +72187,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/q2BYaavZUHo.mp4",
"adj": {
"b": "But",
"a": "that's"
@@ -73815,7 +72216,6 @@
"why": "unvoiced tail",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "about",
"a": ""
@@ -73845,7 +72245,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nrKQ-WoQ_-Y.mp4",
"adj": {
"b": "",
"a": ""
@@ -73869,7 +72268,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/F_atEiYGdSk.mp4",
"adj": {
"b": "sparring",
"a": "this"
@@ -73899,7 +72297,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lXmLzzuxVCo.mp4",
"adj": {
"b": "but",
"a": ""
@@ -73929,7 +72326,6 @@
"why": "next word \"who\" starts here",
"wordIn": "the",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/aeC-_9DQWM4.mp4",
"adj": {
"b": "said",
"a": "who"
@@ -73974,7 +72370,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FiLs5ovIfts.mp4",
"adj": {
"b": "Walmart's",
"a": "sorry,"
@@ -74004,7 +72399,6 @@
"why": "next word \"video\" starts here",
"wordIn": "in",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/U8hsc6aSGdk.mp4",
"adj": {
"b": "chats",
"a": "video"
@@ -74049,7 +72443,6 @@
"why": "next word \"my\" starts here",
"wordIn": "but",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KssiwNCTEKQ.mp4",
"adj": {
"b": "today",
"a": "my"
@@ -74084,7 +72477,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "try",
"a": ""
@@ -74119,7 +72511,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WJYyCfbjv8s.mp4",
"adj": {
"b": "video",
"a": "commenting"
@@ -74154,7 +72545,6 @@
"why": "next word \"Donald\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/i29P5pQc50A.mp4",
"adj": {
"b": "",
"a": "Donald"
@@ -74184,7 +72574,6 @@
"why": "next word \"get\" starts here",
"wordIn": "let's",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nzOE5f6VPqY.mp4",
"adj": {
"b": "chart",
"a": "get"
@@ -74224,7 +72613,6 @@
"why": "",
"wordIn": "several",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QFUX3tJJpiI.mp4",
"adj": {
"b": "had",
"a": ""
@@ -74259,7 +72647,6 @@
"why": "next word \"Can\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "",
"a": "Can"
@@ -74289,7 +72676,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "here",
"a": "pursuant"
@@ -74319,7 +72705,6 @@
"why": "next word \"literally\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/knmVtqTW0MQ.mp4",
"adj": {
"b": "",
"a": "literally"
@@ -74354,7 +72739,6 @@
"why": "next word \"enjoy\" starts here",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Mf1hEHazJaE.mp4",
"adj": {
"b": "us",
"a": "enjoy"
@@ -74394,7 +72778,6 @@
"why": "",
"wordIn": "you're",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/67Qd5J47Y84.mp4",
"adj": {
"b": "and",
"a": "gonna"
@@ -74429,7 +72812,6 @@
"why": "unvoiced tail",
"wordIn": "ridiculous",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ybLrSiTQxtQ.mp4",
"adj": {
"b": "completely",
"a": ""
@@ -74459,7 +72841,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sfVzr3Jcro8.mp4",
"adj": {
"b": "it",
"a": ""
@@ -74489,7 +72870,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/IMsq0B-xJ7M.mp4",
"adj": {
"b": "there's",
"a": "the"
@@ -74529,7 +72909,6 @@
"why": "trailing noise burst",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/eEg1SRZj1rA.mp4",
"adj": {
"b": "",
"a": "He"
@@ -74559,7 +72938,6 @@
"why": "",
"wordIn": "Man,",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1Sj0UBEXX0c.mp4",
"adj": {
"b": "Spider",
"a": "Hulk,"
@@ -74594,7 +72972,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "morning",
"a": ""
@@ -74618,7 +72995,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FeUW-Vw02ZQ.mp4",
"adj": {
"b": "it",
"a": "of"
@@ -74648,7 +73024,6 @@
"why": "next word \"I\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LoaYx6-_arE.mp4",
"adj": {
"b": "and",
"a": "I"
@@ -74683,7 +73058,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yVm_CeJYFCc.mp4",
"adj": {
"b": "at",
"a": ""
@@ -74707,7 +73081,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yGvQfTultRw.mp4",
"adj": {
"b": "",
"a": ""
@@ -74731,7 +73104,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Tvpplznw9Nw.mp4",
"adj": {
"b": "",
"a": "I'm"
@@ -74755,7 +73127,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OBidX_oE7zM.mp4",
"adj": {
"b": "",
"a": "foreign"
@@ -74779,7 +73150,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FiLs5ovIfts.mp4",
"adj": {
"b": "",
"a": "is"
@@ -74803,7 +73173,6 @@
"why": "next word \"they're\" starts here",
"wordIn": "didn't",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rATxtI8FhxE.mp4",
"adj": {
"b": "they",
"a": "they're"
@@ -74838,7 +73207,6 @@
"why": "next word \"and\" starts here",
"wordIn": "that",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/93DFKk1eSoM.mp4",
"adj": {
"b": "offering",
"a": "and"
@@ -74873,7 +73241,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9SgVb3vhRVo.mp4",
"adj": {
"b": "",
"a": "seven"
@@ -74897,7 +73264,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QFUX3tJJpiI.mp4",
"adj": {
"b": "",
"a": "you"
@@ -74932,7 +73298,6 @@
"why": "next word \"mean\" starts here",
"wordIn": "I",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "",
"a": "mean"
@@ -74967,7 +73332,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hGqSbmfHO7o.mp4",
"adj": {
"b": "",
"a": ""
@@ -74991,7 +73355,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/etu7Cy2WmC4.mp4",
"adj": {
"b": "life",
"a": "away"
@@ -75026,7 +73389,6 @@
"why": "next word \"that\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/K9rXg1newxo.mp4",
"adj": {
"b": "today",
"a": "that"
@@ -75061,7 +73423,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/h170V_0AbQ4.mp4",
"adj": {
"b": "Maga",
"a": "maybe"
@@ -75091,7 +73452,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rJ-V9bd9La0.mp4",
"adj": {
"b": "people",
"a": ""
@@ -75121,7 +73481,6 @@
"why": "unvoiced tail",
"wordIn": "the",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/CTbCFWZ7Gdk.mp4",
"adj": {
"b": "to",
"a": ""
@@ -75156,7 +73515,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "clearly",
"a": ""
@@ -75186,7 +73544,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/93DFKk1eSoM.mp4",
"adj": {
"b": "",
"a": ""
@@ -75210,7 +73567,6 @@
"why": "",
"wordIn": "for",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "",
"a": "I"
@@ -75240,7 +73596,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "night",
"a": "so"
@@ -75270,7 +73625,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oVBhIQFKH38.mp4",
"adj": {
"b": "",
"a": ""
@@ -75300,7 +73654,6 @@
"why": "",
"wordIn": "who",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/H-xC6-gSvEc.mp4",
"adj": {
"b": "old",
"a": "did"
@@ -75330,7 +73683,6 @@
"why": "trailing noise burst",
"wordIn": "hopefully",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/MyXmQNqa2sY.mp4",
"adj": {
"b": "and",
"a": "you'll"
@@ -75360,7 +73712,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/22DisumpVqo.mp4",
"adj": {
"b": "Obama",
"a": "who"
@@ -75384,7 +73735,6 @@
"why": "next word \"I'm\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rATxtI8FhxE.mp4",
"adj": {
"b": "and",
"a": "I'm"
@@ -75414,7 +73764,6 @@
"why": "unvoiced tail",
"wordIn": "her",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nzOE5f6VPqY.mp4",
"adj": {
"b": "on",
"a": "financial"
@@ -75454,7 +73803,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "",
"a": ""
@@ -75484,7 +73832,6 @@
"why": "next word \"Gretchen\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/z5IYf8iDnjA.mp4",
"adj": {
"b": "not",
"a": "Gretchen"
@@ -75514,7 +73861,6 @@
"why": "next word \"Sank's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vTng6A4Irt8.mp4",
"adj": {
"b": "writing",
"a": "Sank's"
@@ -75544,7 +73890,6 @@
"why": "trailing noise burst",
"wordIn": "remove",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FTvrBie8KPI.mp4",
"adj": {
"b": "to",
"a": ""
@@ -75579,7 +73924,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "",
"a": "on"
@@ -75609,7 +73953,6 @@
"why": "unvoiced tail",
"wordIn": "in",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cQL0Rja_Qik.mp4",
"adj": {
"b": "",
"a": "fellow"
@@ -75639,7 +73982,6 @@
"why": "next word \"no\" starts here",
"wordIn": "there's",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tRMofQwAjgw.mp4",
"adj": {
"b": "",
"a": "no"
@@ -75674,7 +74016,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xbT-lWAzaiA.mp4",
"adj": {
"b": "",
"a": "Bannon"
@@ -75704,7 +74045,6 @@
"why": "unvoiced tail",
"wordIn": "by",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/5M-1yCq23hY.mp4",
"adj": {
"b": "live",
"a": "them"
@@ -75744,7 +74084,6 @@
"why": "next word \"but\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "sleeping,",
"a": "but"
@@ -75779,7 +74118,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "thirty",
"a": ""
@@ -75803,7 +74141,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/P4JErOtgVuA.mp4",
"adj": {
"b": "and",
"a": ""
@@ -75833,7 +74170,6 @@
"why": "",
"wordIn": "follow",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/acPLSEVKyqI.mp4",
"adj": {
"b": "don't",
"a": ""
@@ -75868,7 +74204,6 @@
"why": "re-attack after a dip",
"wordIn": "teenager",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/g-OStFPhrlk.mp4",
"adj": {
"b": "a",
"a": ""
@@ -75898,7 +74233,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m_B_3i9yKX0.mp4",
"adj": {
"b": "",
"a": ""
@@ -75922,7 +74256,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FTvrBie8KPI.mp4",
"adj": {
"b": "of",
"a": "basically"
@@ -75952,7 +74285,6 @@
"why": "next word \"was\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "in",
"a": "was"
@@ -75987,7 +74319,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OlY74Y-NmMw.mp4",
"adj": {
"b": "",
"a": "industry"
@@ -76011,7 +74342,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/K9rXg1newxo.mp4",
"adj": {
"b": "",
"a": "Elon"
@@ -76041,7 +74371,6 @@
"why": "next word \"new\" starts here",
"wordIn": "as",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/P4JErOtgVuA.mp4",
"adj": {
"b": "",
"a": "new"
@@ -76076,7 +74405,6 @@
"why": "next word \"media\" starts here",
"wordIn": "social",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rPkFK9TgI3w.mp4",
"adj": {
"b": "official",
"a": "media"
@@ -76111,7 +74439,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Zfcmx4_Qxxg.mp4",
"adj": {
"b": "dude",
"a": "and"
@@ -76135,7 +74462,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cQL0Rja_Qik.mp4",
"adj": {
"b": "",
"a": "owning"
@@ -76159,7 +74485,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rPkFK9TgI3w.mp4",
"adj": {
"b": "for",
"a": "for"
@@ -76183,7 +74508,6 @@
"why": "",
"wordIn": "obviously",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "",
"a": ""
@@ -76213,7 +74537,6 @@
"why": "next word \"double\" starts here",
"wordIn": "nearly",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FWv906SubiM.mp4",
"adj": {
"b": "pay",
"a": "double"
@@ -76248,7 +74571,6 @@
"why": "",
"wordIn": "that",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZeqS0DvnorA.mp4",
"adj": {
"b": "that",
"a": "might"
@@ -76278,7 +74600,6 @@
"why": "unvoiced tail",
"wordIn": "Pete",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8ez7u2StoWI.mp4",
"adj": {
"b": "at",
"a": "Hegseth"
@@ -76318,7 +74639,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Oo6dXSEh65Y.mp4",
"adj": {
"b": "",
"a": "a"
@@ -76353,7 +74673,6 @@
"why": "next word \"from\" starts here",
"wordIn": "short",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xpp98iH3zOs.mp4",
"adj": {
"b": "a",
"a": "from"
@@ -76393,7 +74712,6 @@
"why": "unvoiced tail",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "admitted",
"a": ""
@@ -76428,7 +74746,6 @@
"why": "",
"wordIn": "online",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/jQYHkWx9XBc.mp4",
"adj": {
"b": "an",
"a": ""
@@ -76463,7 +74780,6 @@
"why": "trailing noise burst",
"wordIn": "plasma",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LiTjFiYApxg.mp4",
"adj": {
"b": "",
"a": ""
@@ -76493,7 +74809,6 @@
"why": "next word \"if\" starts here",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "me",
"a": "if"
@@ -76533,7 +74848,6 @@
"why": "next word \"new\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8PNFWjrpXrE.mp4",
"adj": {
"b": "my",
"a": "new"
@@ -76568,7 +74882,6 @@
"why": "next word \"level\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6tHdH63lFE.mp4",
"adj": {
"b": "",
"a": "level"
@@ -76598,7 +74911,6 @@
"why": "trailing noise burst",
"wordIn": "minor",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "a",
"a": "when"
@@ -76633,7 +74945,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6CNFodmbWw.mp4",
"adj": {
"b": "with",
"a": "the"
@@ -76673,7 +74984,6 @@
"why": "",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hGqSbmfHO7o.mp4",
"adj": {
"b": "platform",
"a": "for"
@@ -76703,7 +75013,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tBHm09OttbQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -76733,7 +75042,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cQL0Rja_Qik.mp4",
"adj": {
"b": "",
"a": "than"
@@ -76757,7 +75065,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3gORoyt2mC4.mp4",
"adj": {
"b": "around",
"a": "their"
@@ -76792,7 +75099,6 @@
"why": "next word \"army\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/f5sLSteRRTQ.mp4",
"adj": {
"b": "have",
"a": "army"
@@ -76822,7 +75128,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ESeyg7BVkBY.mp4",
"adj": {
"b": "there",
"a": "I"
@@ -76852,7 +75157,6 @@
"why": "next word \"they're\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/G02gq5ClixA.mp4",
"adj": {
"b": "know",
"a": "they're"
@@ -76892,7 +75196,6 @@
"why": "unvoiced tail",
"wordIn": "they're",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9zEisQcriiM.mp4",
"adj": {
"b": "",
"a": "like"
@@ -76922,7 +75225,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/4RFFlS0oMEM.mp4",
"adj": {
"b": "that",
"a": "you"
@@ -76957,7 +75259,6 @@
"why": "",
"wordIn": "the",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OBidX_oE7zM.mp4",
"adj": {
"b": "",
"a": ""
@@ -76987,7 +75288,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vfMRAdhSSrc.mp4",
"adj": {
"b": "about",
"a": ""
@@ -77017,7 +75317,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/gGs4O8jAUWg.mp4",
"adj": {
"b": "second",
"a": "degree"
@@ -77052,7 +75351,6 @@
"why": "next word \"have\" starts here",
"wordIn": "I",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/waF76PwdlS8.mp4",
"adj": {
"b": "gift",
"a": "have"
@@ -77092,7 +75390,6 @@
"why": "next word \"that,\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sagZQzRk_BE.mp4",
"adj": {
"b": "",
"a": "that,"
@@ -77122,7 +75419,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kT1gUTlBfNc.mp4",
"adj": {
"b": "",
"a": "the"
@@ -77157,7 +75453,6 @@
"why": "next word \"who,\" starts here",
"wordIn": "someone",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sogHQwGj7KA.mp4",
"adj": {
"b": "",
"a": "who,"
@@ -77192,7 +75487,6 @@
"why": "next word \"be\" starts here",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Oo6dXSEh65Y.mp4",
"adj": {
"b": "used",
"a": "be"
@@ -77232,7 +75526,6 @@
"why": "next word \"that\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/axL33DS5KeA.mp4",
"adj": {
"b": "prove",
"a": "that"
@@ -77267,7 +75560,6 @@
"why": "next word \"the\" starts here",
"wordIn": "on",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/z5IYf8iDnjA.mp4",
"adj": {
"b": "name",
"a": "the"
@@ -77302,7 +75594,6 @@
"why": "next word \"wrong\" starts here",
"wordIn": "the",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/6A1JPaiqJNY.mp4",
"adj": {
"b": "whoops",
"a": "wrong"
@@ -77337,7 +75628,6 @@
"why": "",
"wordIn": "There's",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/4RFFlS0oMEM.mp4",
"adj": {
"b": "stuff.",
"a": "a"
@@ -77367,7 +75657,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yVm_CeJYFCc.mp4",
"adj": {
"b": "for",
"a": "American"
@@ -77397,7 +75686,6 @@
"why": "next word \"This\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/m6CNFodmbWw.mp4",
"adj": {
"b": "do.",
"a": "This"
@@ -77432,7 +75720,6 @@
"why": "",
"wordIn": "his",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0A4dIN1dKaY.mp4",
"adj": {
"b": "hide",
"a": "I'm"
@@ -77467,7 +75754,6 @@
"why": "next word \"army\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/f5sLSteRRTQ.mp4",
"adj": {
"b": "",
"a": "army"
@@ -77497,7 +75783,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-N5_5CpyZ_8.mp4",
"adj": {
"b": "need",
"a": ""
@@ -77537,7 +75822,6 @@
"why": "next word \"let's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/93DFKk1eSoM.mp4",
"adj": {
"b": "",
"a": "let's"
@@ -77567,7 +75851,6 @@
"why": "next word \"editions\" starts here",
"wordIn": "Later",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/r3cTXMxBPbc.mp4",
"adj": {
"b": "",
"a": "editions"
@@ -77602,7 +75885,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mkyBXrJhDrQ.mp4",
"adj": {
"b": "",
"a": "the"
@@ -77637,7 +75919,6 @@
"why": "unvoiced tail",
"wordIn": "Stan",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1Sj0UBEXX0c.mp4",
"adj": {
"b": "where",
"a": "Lee"
@@ -77677,7 +75958,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/OlY74Y-NmMw.mp4",
"adj": {
"b": "",
"a": "a"
@@ -77701,7 +75981,6 @@
"why": "",
"wordIn": "interesting",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/knmVtqTW0MQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -77731,7 +76010,6 @@
"why": "",
"wordIn": "remember",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/M-jXumuv28E.mp4",
"adj": {
"b": "may",
"a": ""
@@ -77766,7 +76044,6 @@
"why": "re-attack after a dip",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tdYCm-SSyFw.mp4",
"adj": {
"b": "",
"a": ""
@@ -77796,7 +76073,6 @@
"why": "next word \"know,\" starts here",
"wordIn": "You",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/z5IYf8iDnjA.mp4",
"adj": {
"b": "",
"a": "know,"
@@ -77836,7 +76112,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yjHHiRhtMvQ.mp4",
"adj": {
"b": "is",
"a": ""
@@ -77866,7 +76141,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0TcCBOaiZO8.mp4",
"adj": {
"b": "",
"a": ""
@@ -77890,7 +76164,6 @@
"why": "next word \"Father's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p3_f3j9hZi4.mp4",
"adj": {
"b": "",
"a": "Father's"
@@ -77920,7 +76193,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "",
"a": "I"
@@ -77950,7 +76222,6 @@
"why": "next word \"I\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fp_GOiz5MV0.mp4",
"adj": {
"b": "",
"a": "I"
@@ -77985,7 +76256,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "submit",
"a": "to"
@@ -78015,7 +76285,6 @@
"why": "next word \"going\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-qfcXnIXn8Y.mp4",
"adj": {
"b": "not",
"a": "going"
@@ -78055,7 +76324,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Mf1hEHazJaE.mp4",
"adj": {
"b": "here",
"a": "I"
@@ -78090,7 +76358,6 @@
"why": "unvoiced tail",
"wordIn": "but",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kfzmLf09O0w.mp4",
"adj": {
"b": "",
"a": "she's"
@@ -78120,7 +76387,6 @@
"why": "next word \"Mark\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/C-SzFQZbN2M.mp4",
"adj": {
"b": "",
"a": "Mark"
@@ -78150,7 +76416,6 @@
"why": "",
"wordIn": "one's",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/TYPV_Ej1ND4.mp4",
"adj": {
"b": "this",
"a": ""
@@ -78185,7 +76450,6 @@
"why": "next word \"Oh,\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7AtUt83XGQU.mp4",
"adj": {
"b": "",
"a": "Oh,"
@@ -78215,7 +76479,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Yq2EjM3ZXN4.mp4",
"adj": {
"b": "",
"a": "or"
@@ -78245,7 +76508,6 @@
"why": "re-attack after a dip",
"wordIn": "attention",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sagZQzRk_BE.mp4",
"adj": {
"b": "Nympho",
"a": ""
@@ -78275,7 +76537,6 @@
"why": "next word \"what\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/igmjuXKDgvU.mp4",
"adj": {
"b": "",
"a": "what"
@@ -78310,7 +76571,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/igmjuXKDgvU.mp4",
"adj": {
"b": "that",
"a": ""
@@ -78340,7 +76600,6 @@
"why": "",
"wordIn": "sending",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3TjJISVbWVg.mp4",
"adj": {
"b": "anyway",
"a": "more"
@@ -78375,7 +76634,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/6xZktYfRWMQ.mp4",
"adj": {
"b": "lunch",
"a": "but"
@@ -78405,7 +76663,6 @@
"why": "next word \"out\" starts here",
"wordIn": "found",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KssiwNCTEKQ.mp4",
"adj": {
"b": "",
"a": "out"
@@ -78440,7 +76697,6 @@
"why": "next word \"but\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "",
"a": "but"
@@ -78470,7 +76726,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Vz5mf9lqDYg.mp4",
"adj": {
"b": "stuff",
"a": "we're"
@@ -78505,7 +76760,6 @@
"why": "unvoiced tail",
"wordIn": "timing",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/YvJBXgkKwv8.mp4",
"adj": {
"b": "incredible",
"a": ""
@@ -78535,7 +76789,6 @@
"why": "unvoiced tail",
"wordIn": "people",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/G02gq5ClixA.mp4",
"adj": {
"b": "",
"a": ""
@@ -78565,7 +76818,6 @@
"why": "timbre changes (new sound starts)",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/QeSmSLqLR6w.mp4",
"adj": {
"b": "so",
"a": "Riley"
@@ -78595,7 +76847,6 @@
"why": "unvoiced tail",
"wordIn": "one",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/YvJBXgkKwv8.mp4",
"adj": {
"b": "is",
"a": ""
@@ -78630,7 +76881,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tsQBo2SvOao.mp4",
"adj": {
"b": "not",
"a": "games"
@@ -78670,7 +76920,6 @@
"why": "next word \"and\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FTvrBie8KPI.mp4",
"adj": {
"b": "Ripkin",
"a": "and"
@@ -78700,7 +76949,6 @@
"why": "next word \"we'll\" starts here",
"wordIn": "we'll",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LoaYx6-_arE.mp4",
"adj": {
"b": "and",
"a": "we'll"
@@ -78745,7 +76993,6 @@
"why": "",
"wordIn": "will",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "their",
"a": ""
@@ -78780,7 +77027,6 @@
"why": "next word \"the\" starts here",
"wordIn": "of",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Me0fryGe2X8.mp4",
"adj": {
"b": "of",
"a": "the"
@@ -78820,7 +77066,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1Y0204hKq2M.mp4",
"adj": {
"b": "",
"a": "Almost"
@@ -78844,7 +77089,6 @@
"why": "next word \"Maybe\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "",
"a": "Maybe"
@@ -78874,7 +77118,6 @@
"why": "unvoiced tail",
"wordIn": "were",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bbVoMOzCSs0.mp4",
"adj": {
"b": "them",
"a": ""
@@ -78909,7 +77152,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/2BNd7dG7Fqc.mp4",
"adj": {
"b": "",
"a": ""
@@ -78933,7 +77175,6 @@
"why": "",
"wordIn": "paying",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FiLs5ovIfts.mp4",
"adj": {
"b": "are",
"a": "to"
@@ -78973,7 +77214,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ts76T7tJv-U.mp4",
"adj": {
"b": "do",
"a": "more"
@@ -79008,7 +77248,6 @@
"why": "unvoiced tail",
"wordIn": "been",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "has",
"a": ""
@@ -79038,7 +77277,6 @@
"why": "unvoiced tail",
"wordIn": "you",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dVSd_fotB2U.mp4",
"adj": {
"b": "",
"a": "enjoy"
@@ -79068,7 +77306,6 @@
"why": "next word \"be\" starts here",
"wordIn": "going",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/JKozJKyP4Iw.mp4",
"adj": {
"b": "going",
"a": "be"
@@ -79118,7 +77355,6 @@
"why": "unvoiced tail",
"wordIn": "military",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/V4k0ziHTGog.mp4",
"adj": {
"b": "",
"a": ""
@@ -79148,7 +77384,6 @@
"why": "",
"wordIn": "that",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rH1VkOpGce4.mp4",
"adj": {
"b": "",
"a": ""
@@ -79178,7 +77413,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yjHHiRhtMvQ.mp4",
"adj": {
"b": "and",
"a": "they"
@@ -79208,7 +77442,6 @@
"why": "next word \"now\" starts here",
"wordIn": "for",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "but",
"a": "now"
@@ -79248,7 +77481,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7zRNtT2ygCc.mp4",
"adj": {
"b": "the",
"a": "whole"
@@ -79288,7 +77520,6 @@
"why": "next word \"Tony\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mOF790Bh6I0.mp4",
"adj": {
"b": "",
"a": "Tony"
@@ -79318,7 +77549,6 @@
"why": "next word \"babies\" starts here",
"wordIn": "Palestinian",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/mOF790Bh6I0.mp4",
"adj": {
"b": "Palestinian",
"a": "babies"
@@ -79353,7 +77583,6 @@
"why": "next word \"but\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/acPLSEVKyqI.mp4",
"adj": {
"b": "",
"a": "but"
@@ -79388,7 +77617,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8PNFWjrpXrE.mp4",
"adj": {
"b": "",
"a": "it's"
@@ -79423,7 +77651,6 @@
"why": "",
"wordIn": "hope",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "I",
"a": "doing"
@@ -79458,7 +77685,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "safe",
"a": "thanks"
@@ -79488,7 +77714,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Zfcmx4_Qxxg.mp4",
"adj": {
"b": "social",
"a": "media"
@@ -79523,7 +77748,6 @@
"why": "next word \"you\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UqCmkStyKow.mp4",
"adj": {
"b": "that",
"a": "you"
@@ -79553,7 +77777,6 @@
"why": "next word \"said\" starts here",
"wordIn": "fan",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "another",
"a": "said"
@@ -79588,7 +77811,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nrKQ-WoQ_-Y.mp4",
"adj": {
"b": "not",
"a": "he"
@@ -79638,7 +77860,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rATxtI8FhxE.mp4",
"adj": {
"b": "",
"a": "the"
@@ -79673,7 +77894,6 @@
"why": "unvoiced tail",
"wordIn": "here's",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/H-xC6-gSvEc.mp4",
"adj": {
"b": "so",
"a": ""
@@ -79703,7 +77923,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/oYsPh2E-02E.mp4",
"adj": {
"b": "",
"a": "the"
@@ -79738,7 +77957,6 @@
"why": "next word \"Let's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/tdYCm-SSyFw.mp4",
"adj": {
"b": "",
"a": "Let's"
@@ -79768,7 +77986,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "governor",
"a": ""
@@ -79798,7 +78015,6 @@
"why": "",
"wordIn": "Ohio",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/jprDE87Ec58.mp4",
"adj": {
"b": "",
"a": ""
@@ -79828,7 +78044,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/4RFFlS0oMEM.mp4",
"adj": {
"b": "house",
"a": "it's"
@@ -79858,7 +78073,6 @@
"why": "",
"wordIn": "on",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "out",
"a": "the"
@@ -79888,7 +78102,6 @@
"why": "next word \"of\" starts here",
"wordIn": "people",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/M-jXumuv28E.mp4",
"adj": {
"b": "two",
"a": "of"
@@ -79928,7 +78141,6 @@
"why": "next word \"There's\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KtZaCYPSunM.mp4",
"adj": {
"b": "money.",
"a": "There's"
@@ -79958,7 +78170,6 @@
"why": "next word \"and\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lXmLzzuxVCo.mp4",
"adj": {
"b": "",
"a": "and"
@@ -79988,7 +78199,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/WJYyCfbjv8s.mp4",
"adj": {
"b": "",
"a": ""
@@ -80012,7 +78222,6 @@
"why": "",
"wordIn": "white",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9YQfjW1uasM.mp4",
"adj": {
"b": "a",
"a": ""
@@ -80047,7 +78256,6 @@
"why": "",
"wordIn": "by",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/M-jXumuv28E.mp4",
"adj": {
"b": "",
"a": ""
@@ -80082,7 +78290,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-N5_5CpyZ_8.mp4",
"adj": {
"b": "",
"a": ""
@@ -80106,7 +78313,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nPpJc33VuO4.mp4",
"adj": {
"b": "cobbler",
"a": "wildly"
@@ -80141,7 +78347,6 @@
"why": "unvoiced tail",
"wordIn": "you",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/etu7Cy2WmC4.mp4",
"adj": {
"b": "what",
"a": "This"
@@ -80186,7 +78391,6 @@
"why": "",
"wordIn": "community",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Zfcmx4_Qxxg.mp4",
"adj": {
"b": "our",
"a": ""
@@ -80221,7 +78425,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rFuudT4Erls.mp4",
"adj": {
"b": "motion",
"a": "the"
@@ -80251,7 +78454,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/waF76PwdlS8.mp4",
"adj": {
"b": "later",
"a": "but"
@@ -80281,7 +78483,6 @@
"why": "",
"wordIn": "you",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/heZ3z-SOcoA.mp4",
"adj": {
"b": "vehicle",
"a": ""
@@ -80316,7 +78517,6 @@
"why": "next word \"was\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "",
"a": "was"
@@ -80346,7 +78546,6 @@
"why": "",
"wordIn": "or",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/im0yxtppGf4.mp4",
"adj": {
"b": "wrong",
"a": "non"
@@ -80376,7 +78575,6 @@
"why": "next word \"Joe\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/AwwrH7NyCLA.mp4",
"adj": {
"b": "got",
"a": "Joe"
@@ -80406,7 +78604,6 @@
"why": "",
"wordIn": "eighty",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/AMdAOwrYmAI.mp4",
"adj": {
"b": "or",
"a": "advents"
@@ -80446,7 +78643,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/X70q2QJ_VTc.mp4",
"adj": {
"b": "tenth",
"a": ""
@@ -80470,7 +78666,6 @@
"why": "trailing noise burst",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xEYsJorlvkw.mp4",
"adj": {
"b": "",
"a": ""
@@ -80500,7 +78695,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/BKUP9tal4mM.mp4",
"adj": {
"b": "that",
"a": "you"
@@ -80530,7 +78724,6 @@
"why": "",
"wordIn": "He's",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/jzhK7_qn7Gg.mp4",
"adj": {
"b": "",
"a": "somebody"
@@ -80565,7 +78758,6 @@
"why": "next word \"Disney\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dWMRXD0RJRY.mp4",
"adj": {
"b": "",
"a": "Disney"
@@ -80595,7 +78787,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fp_GOiz5MV0.mp4",
"adj": {
"b": "but",
"a": "you"
@@ -80619,7 +78810,6 @@
"why": "",
"wordIn": "affidavit",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LP94fw5JtYk.mp4",
"adj": {
"b": "cause",
"a": ""
@@ -80649,7 +78839,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/axL33DS5KeA.mp4",
"adj": {
"b": "",
"a": "LGBT,"
@@ -80673,7 +78862,6 @@
"why": "next word \"need\" starts here",
"wordIn": "don't",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FTvrBie8KPI.mp4",
"adj": {
"b": "We",
"a": "need"
@@ -80713,7 +78901,6 @@
"why": "next word \"this\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0A4dIN1dKaY.mp4",
"adj": {
"b": "",
"a": "this"
@@ -80748,7 +78935,6 @@
"why": "",
"wordIn": "terminated",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "self",
"a": ""
@@ -80783,7 +78969,6 @@
"why": "",
"wordIn": "these",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3c_VHrAQU8M.mp4",
"adj": {
"b": "of",
"a": ""
@@ -80823,7 +79008,6 @@
"why": "next word \"hired\" starts here",
"wordIn": "We've",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/eFU1WRc9lYM.mp4",
"adj": {
"b": "",
"a": "hired"
@@ -80858,7 +79042,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/rslLnyzMlxM.mp4",
"adj": {
"b": "certainly",
"a": ""
@@ -80888,7 +79071,6 @@
"why": "next word \"Foxford\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "horrifying",
"a": "Foxford"
@@ -80918,7 +79100,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KtZaCYPSunM.mp4",
"adj": {
"b": "this",
"a": "interview"
@@ -80953,7 +79134,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/azX_WsMqFqg.mp4",
"adj": {
"b": "going",
"a": "I"
@@ -80983,7 +79163,6 @@
"why": "",
"wordIn": "describing",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "while",
"a": ""
@@ -81018,7 +79197,6 @@
"why": "trailing noise burst",
"wordIn": "foremost",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/3FQpBHdU3Gs.mp4",
"adj": {
"b": "and",
"a": ""
@@ -81053,7 +79231,6 @@
"why": "",
"wordIn": "in",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/kfzmLf09O0w.mp4",
"adj": {
"b": "legendary",
"a": "Port"
@@ -81088,7 +79265,6 @@
"why": "",
"wordIn": "server",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/X70q2QJ_VTc.mp4",
"adj": {
"b": "",
"a": ""
@@ -81118,7 +79294,6 @@
"why": "",
"wordIn": "or",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LoaYx6-_arE.mp4",
"adj": {
"b": "today",
"a": "over"
@@ -81148,7 +79323,6 @@
"why": "",
"wordIn": "song",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8RiHnmAGy8U.mp4",
"adj": {
"b": "disgusting",
"a": ""
@@ -81183,7 +79357,6 @@
"why": "unvoiced tail",
"wordIn": "they",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nPpJc33VuO4.mp4",
"adj": {
"b": "",
"a": ""
@@ -81218,7 +79391,6 @@
"why": "next word \"no\" starts here",
"wordIn": "There's",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/C-SzFQZbN2M.mp4",
"adj": {
"b": "",
"a": "no"
@@ -81253,7 +79425,6 @@
"why": "",
"wordIn": "federally",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sfVzr3Jcro8.mp4",
"adj": {
"b": "be",
"a": ""
@@ -81283,7 +79454,6 @@
"why": "next word \"Obviously\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cPKtKx3evdM.mp4",
"adj": {
"b": "properties.",
"a": "Obviously"
@@ -81313,7 +79483,6 @@
"why": "",
"wordIn": "do",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/4RFFlS0oMEM.mp4",
"adj": {
"b": "to",
"a": ""
@@ -81348,7 +79517,6 @@
"why": "next word \"and\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Dr1sWSSUspY.mp4",
"adj": {
"b": "",
"a": "and"
@@ -81378,7 +79546,6 @@
"why": "unvoiced tail",
"wordIn": "from",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/RJMe6PuMo3A.mp4",
"adj": {
"b": "name",
"a": ""
@@ -81408,7 +79575,6 @@
"why": "next word \"of\" starts here",
"wordIn": "buddy",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/cPKtKx3evdM.mp4",
"adj": {
"b": "a",
"a": "of"
@@ -81448,7 +79614,6 @@
"why": "",
"wordIn": "down",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/MB0fIIPySYs.mp4",
"adj": {
"b": "run",
"a": ""
@@ -81483,7 +79648,6 @@
"why": "next word \"just\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "at",
"a": "just"
@@ -81513,7 +79677,6 @@
"why": "",
"wordIn": "last",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/lraibG7Iu5w.mp4",
"adj": {
"b": "thing",
"a": "bit"
@@ -81548,7 +79711,6 @@
"why": "re-attack after a dip",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7zRNtT2ygCc.mp4",
"adj": {
"b": "",
"a": ""
@@ -81578,7 +79740,6 @@
"why": "next word \"has\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Z2Q87gaouqs.mp4",
"adj": {
"b": "",
"a": "has"
@@ -81613,7 +79774,6 @@
"why": "next word \"recording\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/jprDE87Ec58.mp4",
"adj": {
"b": "recorded",
"a": "recording"
@@ -81643,7 +79803,6 @@
"why": "unvoiced tail",
"wordIn": "babies",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/hGqSbmfHO7o.mp4",
"adj": {
"b": "",
"a": ""
@@ -81673,7 +79832,6 @@
"why": "next word \"but\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UGcJln5uj_Y.mp4",
"adj": {
"b": "",
"a": "but"
@@ -81708,7 +79866,6 @@
"why": "next word \"like\" starts here",
"wordIn": "down",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/AMdAOwrYmAI.mp4",
"adj": {
"b": "we're",
"a": "like"
@@ -81753,7 +79910,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nvIKzQM_hBo.mp4",
"adj": {
"b": "laws",
"a": "they"
@@ -81783,7 +79939,6 @@
"why": "",
"wordIn": "who's",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nr9bGYAjVpY.mp4",
"adj": {
"b": "",
"a": ""
@@ -81818,7 +79973,6 @@
"why": "",
"wordIn": "when",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Mf1hEHazJaE.mp4",
"adj": {
"b": "",
"a": "they"
@@ -81848,7 +80002,6 @@
"why": "unvoiced tail",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7zRNtT2ygCc.mp4",
"adj": {
"b": "going",
"a": ""
@@ -81883,7 +80036,6 @@
"why": "next word \"I've\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/0A4dIN1dKaY.mp4",
"adj": {
"b": "say",
"a": "I've"
@@ -81913,7 +80065,6 @@
"why": "",
"wordIn": "more",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/uy3ber32v1s.mp4",
"adj": {
"b": "",
"a": "extortion"
@@ -81948,7 +80099,6 @@
"why": "unvoiced tail",
"wordIn": "people",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/KW-dUugk9_E.mp4",
"adj": {
"b": "that",
"a": ""
@@ -81978,7 +80128,6 @@
"why": "",
"wordIn": "tomorrow",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/bpQk0X2Ela4.mp4",
"adj": {
"b": "hour",
"a": ""
@@ -82008,7 +80157,6 @@
"why": "next word \"but\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9zEisQcriiM.mp4",
"adj": {
"b": "them",
"a": "but"
@@ -82038,7 +80186,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/vfMRAdhSSrc.mp4",
"adj": {
"b": "of",
"a": "a"
@@ -82062,7 +80209,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ZEwkRomA9pU.mp4",
"adj": {
"b": "for",
"a": "the"
@@ -82092,7 +80238,6 @@
"why": "trailing noise burst",
"wordIn": "That's",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/waF76PwdlS8.mp4",
"adj": {
"b": "",
"a": "part"
@@ -82122,7 +80267,6 @@
"why": "next word \"they're\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Yq2EjM3ZXN4.mp4",
"adj": {
"b": "goofballs",
"a": "they're"
@@ -82157,7 +80301,6 @@
"why": "trailing noise burst",
"wordIn": "the",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-lSWX5qachE.mp4",
"adj": {
"b": "know",
"a": ""
@@ -82187,7 +80330,6 @@
"why": "",
"wordIn": "star",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1Sj0UBEXX0c.mp4",
"adj": {
"b": "a",
"a": "attraction."
@@ -82227,7 +80369,6 @@
"why": "",
"wordIn": "this?",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/V4k0ziHTGog.mp4",
"adj": {
"b": "all",
"a": ""
@@ -82262,7 +80403,6 @@
"why": "trailing noise burst",
"wordIn": "garbage",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/fUoDtOdIza4.mp4",
"adj": {
"b": "your",
"a": ""
@@ -82297,7 +80437,6 @@
"why": "unvoiced tail",
"wordIn": "hmm",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/xCGveTqqars.mp4",
"adj": {
"b": "say",
"a": ""
@@ -82327,7 +80466,6 @@
"why": "next word \"But\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Me0fryGe2X8.mp4",
"adj": {
"b": "",
"a": "But"
@@ -82357,7 +80495,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "OMG",
"a": "the"
@@ -82397,7 +80534,6 @@
"why": "re-attack after a dip",
"wordIn": "hospitality",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/UCanpMuiH2U.mp4",
"adj": {
"b": "the",
"a": ""
@@ -82427,7 +80563,6 @@
"why": "trailing noise burst",
"wordIn": "And",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Pk37xFzjXF0.mp4",
"adj": {
"b": "world.",
"a": ""
@@ -82457,7 +80592,6 @@
"why": "next word \"want\" starts here",
"wordIn": "I",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/28q67bgixx8.mp4",
"adj": {
"b": "here",
"a": "want"
@@ -82492,7 +80626,6 @@
"why": "",
"wordIn": "cases",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/azX_WsMqFqg.mp4",
"adj": {
"b": "of",
"a": ""
@@ -82527,7 +80660,6 @@
"why": "next word \"mister\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/p-eArRK_1s4.mp4",
"adj": {
"b": "on",
"a": "mister"
@@ -82557,7 +80689,6 @@
"why": "next word \"what\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/9zEisQcriiM.mp4",
"adj": {
"b": "at",
"a": "what"
@@ -82592,7 +80723,6 @@
"why": "unvoiced tail",
"wordIn": "composay",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-pKXHyeyRhQ.mp4",
"adj": {
"b": "",
"a": ""
@@ -82622,7 +80752,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Ep_qtW2K3BY.mp4",
"adj": {
"b": "it",
"a": "she's"
@@ -82652,7 +80781,6 @@
"why": "trailing noise burst",
"wordIn": "husband",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/93DFKk1eSoM.mp4",
"adj": {
"b": "her",
"a": "she"
@@ -82687,7 +80815,6 @@
"why": "next word \"want\" starts here",
"wordIn": "didn't",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/Hh71Xe7XK6k.mp4",
"adj": {
"b": "they",
"a": "want"
@@ -82727,7 +80854,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/ibJe2PlNOS8.mp4",
"adj": {
"b": "Quartering",
"a": "you"
@@ -82762,7 +80888,6 @@
"why": "",
"wordIn": "the",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/b1u84ZgMcvo.mp4",
"adj": {
"b": "like",
"a": "you"
@@ -82792,7 +80917,6 @@
"why": "next word \"of\" starts here",
"wordIn": "kind",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/B3uW6WKvWlk.mp4",
"adj": {
"b": "any",
"a": "of"
@@ -82832,7 +80956,6 @@
"why": "next word \"just\" starts here",
"wordIn": "channel",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nrKQ-WoQ_-Y.mp4",
"adj": {
"b": "main",
"a": "just"
@@ -82872,7 +80995,6 @@
"why": "next word \"then\" starts here",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/LgPpZp_Tork.mp4",
"adj": {
"b": "footage",
"a": "then"
@@ -82907,7 +81029,6 @@
"why": "unvoiced tail",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/nrKQ-WoQ_-Y.mp4",
"adj": {
"b": "",
"a": "that"
@@ -82931,7 +81052,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/K9rXg1newxo.mp4",
"adj": {
"b": "",
"a": ""
@@ -82961,7 +81081,6 @@
"why": "next word \"and\" starts here",
"wordIn": "Quartering",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/IMsq0B-xJ7M.mp4",
"adj": {
"b": "the",
"a": "and"
@@ -82996,7 +81115,6 @@
"why": "next word \"is\" starts here",
"wordIn": "what",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/us3ktDhlC7w.mp4",
"adj": {
"b": "",
"a": "is"
@@ -83036,7 +81154,6 @@
"why": "unvoiced tail",
"wordIn": "individual",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/M-jXumuv28E.mp4",
"adj": {
"b": "the",
"a": ""
@@ -83071,7 +81188,6 @@
"why": "unvoiced tail",
"wordIn": "to",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/azX_WsMqFqg.mp4",
"adj": {
"b": "and",
"a": ""
@@ -83101,7 +81217,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/-CcZe5Bffzs.mp4",
"adj": {
"b": "and",
"a": ""
@@ -83131,7 +81246,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/1Y0204hKq2M.mp4",
"adj": {
"b": "screen",
"a": ""
@@ -83161,7 +81275,6 @@
"why": "next word \"loser\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/v-MG7ofJ6X4.mp4",
"adj": {
"b": "this",
"a": "loser"
@@ -83191,7 +81304,6 @@
"why": "next word \"the\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/yGvQfTultRw.mp4",
"adj": {
"b": "",
"a": "the"
@@ -83221,7 +81333,6 @@
"why": "next word \"we've\" starts here",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/7zRNtT2ygCc.mp4",
"adj": {
"b": "back",
"a": "we've"
@@ -83251,7 +81362,6 @@
"why": "next word \"we\" starts here",
"wordIn": "do",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/22DisumpVqo.mp4",
"adj": {
"b": "So",
"a": "we"
@@ -83296,7 +81406,6 @@
"why": "next word \"idea\" starts here",
"wordIn": "the",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/jQYHkWx9XBc.mp4",
"adj": {
"b": "",
"a": "idea"
@@ -83336,7 +81445,6 @@
"why": "next word \"men\" starts here",
"wordIn": "the",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/FTvrBie8KPI.mp4",
"adj": {
"b": "all",
"a": "men"
@@ -83376,7 +81484,6 @@
"why": "",
"wordIn": "",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/dhOXEakOpaM.mp4",
"adj": {
"b": "just",
"a": "it"
@@ -83400,7 +81507,6 @@
"why": "",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/sfVzr3Jcro8.mp4",
"adj": {
"b": "resurfacing",
"a": ""
@@ -83430,7 +81536,6 @@
"why": "unvoiced tail",
"wordIn": "and",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/YvJBXgkKwv8.mp4",
"adj": {
"b": "Council",
"a": "Investment"
@@ -83470,7 +81575,6 @@
"why": "",
"wordIn": "booty",
"risk": "",
- "vid": "file:///home/user/.claude/jobs/efbe67a7/tmp/song/media/8RiHnmAGy8U.mp4",
"adj": {
"b": "eating",
"a": ""
diff --git a/umtool/song/video-dir.mjs b/umtool/song/video-dir.mjs
@@ -45,12 +45,14 @@
// Recording is ADDITIVE: writing `wide.mp4` leaves every other entry alone, so a
// builder that makes one cut at a time does not erase the other three.
//
-// VIDEO_ROOT overrides where the song directories live.
+// VIDEO_ROOT overrides where the song directories live (default
+// ~/reports/quartering-uh-song/videos).
import { readFileSync, writeFileSync, mkdirSync, existsSync, statSync, readdirSync, renameSync } from "node:fs";
import { execFileSync } from "node:child_process";
+import os from "node:os";
import path from "node:path";
-const ROOT = process.env.VIDEO_ROOT ?? "/home/user/reports/quartering-uh-song/videos";
+const ROOT = process.env.VIDEO_ROOT ?? path.join(os.homedir(), "reports", "quartering-uh-song", "videos");
const argv = process.argv.slice(2);
const take = (flag) => {
diff --git a/umtool/song/vshort.sh b/umtool/song/vshort.sh
@@ -1,52 +0,0 @@
-#!/usr/bin/env bash
-# Build one tune's vertical (1080x1920) cut: render every voice alone over black, then
-# compose the four boxes.
-#
-# VOICES="melody bass arp" from PLAN
-# VOICES="melody bass run@other.json" a voice whose plan is elsewhere (Pokemon's run
-# is rendered separately, so it has its own)
-#
-# The first voice is the PRIMARY box, the next two the secondaries. A tune with two
-# voices simply leaves the second secondary box empty -- which is what an idle box
-# looks like anyway, so nothing special is needed for it.
-set -euo pipefail
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-T=/home/user/.claude/jobs/efbe67a7/tmp
-S=/home/user/Projects/yt-dlp-transcript-browser/umtool/song
-
-TUNE=${TUNE:?need TUNE}
-PLAN=${PLAN:?need PLAN}
-VOICES=${VOICES:?need VOICES}
-COMPOSED=${COMPOSED:?need COMPOSED}
-GAMEPLAY=${GAMEPLAY:?need GAMEPLAY}
-OUT=${OUT:?need OUT}
-OFFSET=${OFFSET:-0}
-GIMMICKS=${GIMMICKS:-}
-
-SOLOS=()
-for spec in $VOICES; do
- v="${spec%%@*}"
- p="${spec#*@}"; [ "$p" = "$spec" ] && p="$PLAN"
- solo="solo-$TUNE-$v.json"
- vid="$T/solo-$TUNE-$v.mp4"
- node "$S/solo-voice.mjs" "$p" "$v" "$solo"
- if [ -f "$vid" ]; then
- echo " reusing $vid"
- else
- # BG_MAIN_SCALE=1.0 fills the frame -- the default 0.70 insets the panel to
- # 896x504 and the box would carry a black border of its own. BG_DIM=0 because
- # there is no background to dim; dimming black just makes the render slower.
- PLAN="$solo" BG_VIDEO=$PWD/intro/black1280.mp4 LAYOUT=bg \
- BG_MAIN_SCALE=1.0 BG_DIM=0 OUT_END=0 \
- node render-poly.mjs "$vid" | tail -2
- fi
- SOLOS+=("$vid")
-done
-
-COMPOSED="$COMPOSED" GAMEPLAY="$GAMEPLAY" \
- PRIMARY="${SOLOS[0]}" SEC1="${SOLOS[1]:-}" SEC2="${SOLOS[2]:-}" \
- OFFSET="$OFFSET" GIMMICKS="$GIMMICKS" \
- node "$S/shorts-compose.mjs" "$OUT"
-
-echo "VSHORT_${TUNE}_DONE -> $OUT"
-ffprobe -v error -show_entries format=duration:stream=width,height -of csv=p=0 "$OUT" | head -2
diff --git a/umtool/song/yoshi-rebuild.sh b/umtool/song/yoshi-rebuild.sh
@@ -1,81 +0,0 @@
-#!/bin/sh
-# Yoshi's Island athletic theme, rebuilt on the 3,959-clip palette.
-#
-# TIME_LIMIT 94.5 reproduces v6 exactly (melody 411 / bass 204 / arp 108). The
-# length is the lever on this tune more than any other: at full length the melody
-# degrades to p90 3.8-4.5 semitones, and raising poolFactor does NOTHING because
-# the melody is claimed LAST and takes whatever the bass and arp leave.
-#
-# The vowel is a SCORED preference (tokenPenalty), not a filter. A hard filter
-# took the melody from p90 1.59 to 4.47 at full length; scoring lets a much
-# better pitch match still buy an "uh" while "um" wins every tie. v6 landed at
-# 87% "um" that way.
-set -e
-cd /home/user/.claude/jobs/efbe67a7/tmp/song
-D=/home/user/reports/quartering-uh-song
-T=/home/user/.claude/jobs/efbe67a7/tmp
-
-TP=${TP:-6}
-PLANF=${PLANF:-yoshi-plan-v7.json}
-OUT=${OUT:-$D/quartering-yoshi-v7.mp4}
-mkdir -p "$(dirname "$OUT")"
-
-export TEMPO_SCALE=1.05 TIME_LIMIT=${TL:-94.5} MAX_IOI=0.70 WINDOW=700 LOCAL=0 WOBBLE_W=0.08
-export VOICES='[{"name":"melody","lead":"lead2.json","gain":1.0,"role":"full","unique":true,"reuseGap":20,"contourMax":12,"fill":true,"tokens":["um"],"tokenPenalty":'"$TP"',"noKeeps":true},{"name":"bass","lead":"yoshi-bass.json","gain":0.68,"role":"pip1","unique":true,"reuseGap":12,"contourMax":4,"poolFactor":2.5,"fill":false,"noKeeps":true},{"name":"arp","lead":"yoshi-harm.json","gain":0.46,"role":"pip2","unique":true,"reuseGap":12,"contourMax":5,"poolFactor":2.5,"fill":false,"noKeeps":true}]'
-
-if [ "${REUSE_PLAN:-0}" != "1" ]; then
- node arrange-poly.mjs
- cp poly-plan.json "$PLANF"
-fi
-
-# What the vowel preference actually bought, measured on the PLAN rather than
-# assumed from the setting.
-node -e "
-const {readFileSync,readdirSync}=require('fs');const path=require('path');
-const S='$PWD';const cand=new Map();
-for(const f of readdirSync(path.join(S,'cand2')).filter(f=>f.endsWith('.json')))
- for(const c of JSON.parse(readFileSync(path.join(S,'cand2',f),'utf8')).candidates){
- const k=c.video+'@'+(+c.start).toFixed(2); if(!cand.has(k))cand.set(k,c);}
-const W=JSON.parse(readFileSync(path.join(S,'words.json'),'utf8'));
-const p=JSON.parse(readFileSync('$PLANF','utf8'));
-for(const v of p.voices){const t={};
- for(const n of v.plan){const k=n.video+'@'+(+n.srcStart).toFixed(2);
- const w=((W[k]&&W[k].word)||(cand.get(k)||{}).token||'?').toLowerCase().trim(); t[w]=(t[w]||0)+1;}
- const tot=v.plan.length;
- console.log(' vowel '+v.name.padEnd(7)+' '+Object.entries(t).sort((a,b)=>b[1]-a[1])
- .map(([k,n])=>k+' '+(100*n/tot).toFixed(0)+'%').join(' '));}
-"
-[ "${ARRANGE_ONLY:-0}" = "1" ] && exit 0
-
-# TAIL_FADE is NOT set: this cut ends on the reprise, not on a loop point, so it
-# has an ending of its own. VIDEO_TAIL_FADE follows TAIL_FADE, so the picture is
-# left alone too.
-# BG_VIDEO overridable like TP/PLANF/OUT/TL above, so a treated bed -- say one
-# debox-bg.mjs has taken the game's own text-box freezes out of -- can be
-# rendered against this exact arrangement without editing the recipe.
-PLAN=$PLANF BG_VIDEO=${BG_VIDEO:-$PWD/intro/yoshi-bg.mp4} BG_FIT=pad LAYOUT=bg \
- node render-poly.mjs "$T/yoshi-${PLANF%.json}-body.mp4" | tail -3
-
-LEAD=$(node -e "
-const {voices}=require('$PWD/$PLANF');
-console.log(Math.max(0,Math.min(...voices.filter(v=>v.plan.length).map(v=>v.plan[0].slotStart))).toFixed(3));")
-echo "trimming ${LEAD}s of lead-in off the body"
-ffmpeg -nostdin -v error -y -ss "$LEAD" -i "$T/yoshi-${PLANF%.json}-body.mp4" \
- -c:v libx264 -preset medium -crf 21 -pix_fmt yuv420p -video_track_timescale 30000 \
- -c:a aac -b:a 192k -ar 48000 -ac 2 "$T/yoshi-${PLANF%.json}-bd.mp4"
-
-# Intro: yoshi-src.mp4 6.24 -> 10.05s. 6.24 is inside the FIRST FULLY BLACK
-# frame -- the source is 60fps, and detecting black by MEAN luma finds the wrong
-# frames because averaging hides a small lit region.
-ffmpeg -nostdin -v error -y -ss 6.24 -to 10.05 -i intro/yoshi-src.mp4 \
- -vf "scale=1280:720:force_original_aspect_ratio=decrease,pad=1280:720:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=30" \
- -c:v libx264 -preset medium -crf 21 -pix_fmt yuv420p -video_track_timescale 30000 \
- -c:a aac -b:a 192k -ar 48000 -ac 2 "$T/yoshi-${PLANF%.json}-intro.mp4"
-
-ffmpeg -nostdin -v error -y -i "$T/yoshi-${PLANF%.json}-intro.mp4" -i "$T/yoshi-${PLANF%.json}-bd.mp4" \
- -filter_complex "[0:v][0:a][1:v][1:a]concat=n=2:v=1:a=1[v][a]" -map "[v]" -map "[a]" \
- -c:v libx264 -preset medium -crf 21 -pix_fmt yuv420p -c:a aac -b:a 192k -ar 48000 -ac 2 \
- -movflags +faststart "$OUT"
-
-echo "YOSHI7_DONE -> $OUT"
-ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1 "$OUT"
