commit a878b96815e53417c2a5e1646216132a35a7f3d8
parent f560271492ed555cc6a83b10423d83972e06babe
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Mon, 28 Sep 2026 20:04:49 -0400
plans: slice Q, found after rollout — the umtool build walked the corpus; the hazard in FACTS, the capped corpus-visible build gate
release-12.md gains "Slice Q, found after rollout": the defect, the
7-second repro, the fix (f991e35b) and the guard (f5602714), both capped
builds' wall time, RSS and output, and the gates. FACTS records the
hazard: a path joined from process.cwd() in a module a Next app imports
is a directory of assets to Turbopack; a worktree build will not show
it; test with the corpus visible. The implementer rules' gates gain the
umtool build with the corpus linked under a 5 GB cap. One [Unreleased]
bullet: the umtool build no longer reads the corpus folder.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
4 files changed, 143 insertions(+), 0 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -5,6 +5,7 @@
- **A stats build keeps the stats of a channel whose drive is not mounted, and will not undo a newer version's stats.** A channel whose media is on a drive that is not mounted (or is being moved) is left as it was instead of being read as a channel with no videos; a stats rebuild that has to start over refuses until the drive is back. A stats build refuses to clear stats written by a newer version of the editor; set `ARCHILYZER_STATS_ALLOW_DOWNGRADE=1` to roll back on purpose. Its log also says apart how many videos were downloaded since the last index build (they catch up after the next one) and how many the index skipped (no upload date, or it failed on them).
- **Building the homepage now publishes the source: a read-only git mirror, its raw tree and a fresh tarball, behind a gate.** `archilyzer build homepage`, the `/sites` Homepage jobs and `pnpm ops build-homepage` run `archilyzer source publish` between compose and `next build`. It makes a fresh clone of the private `main` (the repository itself is never rewritten), rewrites that copy with git-filter-repo using your scrub rules (file contents and commit messages; your home directory becomes `/home/user` without a rule), and publishes it under `homepage/public` for `git clone https://archilyzer.pages.dev/source/archilyzer.git`, beside `/source/tree/` and the Downloads tarball. Before anything is written, every object of the rewritten history and every file about to be published is searched for every string you have denied; **one hit refuses the build**, and its log names the string only by where you wrote it (`denylist line 3 (len 5)`) and each hit by its object, field and byte offset — never a byte of the object. **A refusal withdraws the source**: the last publish is removed from `homepage/public` and the last build's copy from `homepage/out`, and **Deploy homepage refuses** a build whose source was not audited under today's rules and today's `main` ("run `archilyzer build homepage`, then deploy"). The rules live outside the repo, in `~/.config/archilyzer/source-scrub.txt` and `source-denylist.txt` (`ARCHILYZER_CONFIG_DIR`, `SOURCE_SCRUB_FILE`, `SOURCE_DENYLIST_FILE`); **without them the build refuses**, naming the missing file. **Put everything private in the denylist before any deploy, a preview included**: previews are public, and every deployment stays reachable at its own address until you delete it. Install git-filter-repo once (`pipx install git-filter-repo`; the editor's process needs `~/.local/bin` on its `PATH` to find it) — without it the build fetches it through `pipx run`, which needs the network — and gitleaks if you want its secret scan too. An unchanged `main` with unchanged rules is skipped, so a rebuild costs about 20 seconds only when something moved. A checkout with no git repository (the docker image, a tarball install) builds with the /source page's empty state. `archilyzer source publish --check` audits without writing, `archilyzer source audit <clone>/.git` checks any clone, `archilyzer build homepage --no-source` removes the published source instead, and `archilyzer doctor` reports the tools, the two files (rule counts and permissions, never their contents) and the last publish. `create-archives.sh` is gone. See PUBLISH.md, "The source mirror (homepage)".
- **umtool reads the corpus from its checkout (or `TRANSCRIPTS_DIR`), and the song project's data defaults to `~/.local/share/archilyzer/song`.** If yours is elsewhere, link it there before restarting umtool: `mkdir -p ~/.local/share/archilyzer && ln -s <where the data is> ~/.local/share/archilyzer/song` (the data stays where it is). With no `CHANNELS_DIR`, umtool reads the corpus at `$TRANSCRIPTS_DIR/channels`, else the checkout's own `transcripts/channels`; it used to fall back to an absolute path that existed on one machine only. The song project's videos default to `~/reports/quartering-uh-song/videos`; `SONG_DIR` and `VIDEO_ROOT` still win. The song project's tracked manifests record their paths relative to the song folders, and the twenty one-off `umtool/song/*.sh` run logs, which only ever ran on the machine that wrote them, are gone.
+- **umtool's production build no longer reads the corpus folder.** Since umtool began finding the corpus from its checkout (the bullet above), `next build` treated the checkout's whole `transcripts/channels` as files to bundle. On a real archive it ran out of memory and was killed, so umtool could not be rebuilt. The build now ignores that folder and finishes in about 25 s at under 1 GB, the same as a checkout with no corpus. Nothing changes when umtool runs.
## [0.10.0] - 2026-09-28
- **The homepage can be built and deployed from `/sites`.** Under a new **Homepage** section, after Hub, there is **Build homepage** (tick **Deploy after build** to ship it in the same job, only if the build succeeds) and **Deploy homepage**, which ships the build already in `homepage/out`. A **Preview branch** box beside them sends either deploy to a Cloudflare Pages preview of the `archilyzer` project instead of production, and shows the preview's address as you type; a name Cloudflare would refuse or rewrite, or `main`, greys the deploy buttons out and says why. A line under the buttons says what a deploy would ship: when `homepage/out` was built (or that it holds no build yet), and where it goes, with the live URL. Deploy homepage with nothing built is refused before any job starts. The homepage reads the search index as it stands, so run **Build index** first when its numbers should move. The jobs run the same code as `archilyzer build homepage` / `deploy homepage`, and show on `/jobs` as `build-homepage`, `deploy-homepage` and `build-deploy-homepage`. The Hub section no longer describes the homepage.
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -7320,6 +7320,57 @@ Slices Q (`4855f70b`) and R (`ffdeb2cd`): [`release-12.md`](release-12.md), the
- **The live :3001 editor runs its BUILT bundle.** Until it is rebuilt on a tree with release 12,
its `/sites` Homepage jobs have no source step, no withdrawal and no deploy check.
+### A path joined from `process.cwd()` is a directory of assets to Turbopack (slice Q, after rollout)
+
+**The hazard.** In a module a Next app imports, Turbopack traces a path joined from
+`process.cwd()` as a directory of assets. The same goes for a module's own `import.meta.url` or
+`__dirname`.
+- **How:** Turbopack evaluates these statically as paths in the project. A `path.join` /
+ `path.resolve` / fs call on the result becomes an asset reference: to a file, or, when the joined
+ path is a directory, to every file under it (`DirAssetReference`).
+- **A worktree build will not show it.** A worktree has no `transcripts/`, so the reference is
+ empty. **Test with the corpus visible.**
+- **What happened:** slice Q wrote `path.join(REPO_ROOT, "transcripts", "channels")` with
+ `REPO_ROOT = findRepoRoot(process.cwd())` in `umtool/lib/paths.mjs`. The walk's fallback,
+ `path.resolve(start, "..")`, evaluates to the project root.
+ - In `<primary>`, `pnpm --filter umtool exec next build` walked the corpus (hundreds of GB, with
+ channel `data/` symlinked to another drive). It grew until the kernel OOM-killed it: twice, at
+ about 3.7 GB RSS. The live umtool was down until the fix.
+ - Under a memory cap it dies at once instead: `<DirAssetReference as
+ ModuleReference>::resolve_reference failed … Symlink [project]/transcripts/channels/<slug>/archive
+ is invalid, it points out of the filesystem root`.
+ - A worktree built it in 30 s at 0.8 GB, which is how the gate passed.
+- **The opt-out is per expression and documented:** `path.join(/* turbopackIgnore: true */
+ process.cwd(), bar)`. That exact text is Turbopack's own advice in its "whole project was traced"
+ message; the table is in the Next docs, `03-api-reference/08-turbopack.md`, "Magic comments". It
+ goes before the FIRST argument of each path or fs call on such a value, and it changes nothing
+ at run time. Per call, not per value:
+ - a nested call needs its own marker (`path.dirname(/* turbopackIgnore: true */
+ fileURLToPath(import.meta.url))`);
+ - an outer fs call on an opted-out `path.join` is covered.
+- **Not followed by the tracer:** `os.homedir()` and `process.env.*`. A build with `HOME` pointed at
+ a synthetic home inside the project, full of out-of-root symlinks under `reports/`,
+ `.local/share/archilyzer/song` and `.cache/`, succeeded. So `~/reports` and the XDG song path are
+ safe as `path.join(os.homedir(), …)`.
+- **The guard is `scripts/umtool-build-trace.test.mjs`** (in `test:scripts`). It scans umtool's
+ app, components, lib, `report-to-video/*.mjs` and `song/paths.mjs`, per module. A path or fs call
+ carrying a value derived in that file from `process.cwd()`, `import.meta.url|dirname|filename` or
+ `__dirname` must open with the opt-out.
+ - It is static and per module, as Turbopack's value analysis is: an imported binding is opaque to
+ it.
+ - With the slice Q `paths.mjs` it fails on the defect's line.
+- **The build gate** is run with the corpus visible and under a memory cap (the command is in
+ `plans/tools/implementer-rules.md`). Linking `<primary>/transcripts` into a worktree is for a
+ BUILD only. Remove the link afterwards: never run an app, an index or a fixture builder through it.
+- **The other apps are safe by accident, not by rule:**
+ - `common/lib/paths.ts`' `findMonorepoRoot()` falls back to `process.cwd()` (the app's own
+ directory, which has no `transcripts/`). An editor build with the corpus present is 39 s today.
+ A fallback that evaluated to the repo root would make `path.join(monorepoRoot, "transcripts")`
+ this same bug.
+ - `homepage/app/lib/source.ts` joins `process.cwd()` + `public`, which holds the source mirror.
+ The homepage builds in about 20 s today.
+ - Neither has a guard.
+
## The stats cache key (verified 2026-09-28, branch `fix/stats-cache-key`)
The record is [`stats-cache-key.md`](stats-cache-key.md). Anchors are at the branch tip. Two notes
diff --git a/plans/release-12.md b/plans/release-12.md
@@ -281,6 +281,87 @@ No High or Medium findings. The coordinator asked for two of the Lows to be fixe
- **After the fixes:** tsc is clean (41 s, all seven packages), and the grep gate is empty at the
new tip. Per the coordinator, e2e was not re-run for a type comment and a changelog line.
+### Slice Q, found after rollout — the umtool build walked the corpus (2026-09-28)
+
+**What.** Slice Q's `umtool/lib/paths.mjs` set `CHANNELS_DIR` to
+`path.join(REPO_ROOT, "transcripts", "channels")`, with
+`REPO_ROOT = findRepoRoot(process.cwd())`.
+- **Turbopack evaluated that statically** as `[project]/transcripts/channels` and made it a
+ directory asset reference. The walk's fallback, `path.resolve(start, "..")`, is the project
+ root.
+- **In `<primary>`,** `pnpm --filter umtool exec next build` walked the real corpus (hundreds of GB,
+ channel `data/` symlinked to another drive). It was OOM-killed twice, at about 3.7 GB RSS, so the
+ live umtool on :3050 stayed down until this fix.
+- **Slice Q's gate built in a worktree,** which has no `transcripts/`, so the reference was empty
+ and the build took 30 s. The hazard and the rule are in FACTS: "A path joined from
+ `process.cwd()` is a directory of assets to Turbopack".
+
+**The repro.** Link the corpus into a worktree for the BUILD only, and cap memory:
+```
+ln -s <primary>/transcripts <worktree>/transcripts
+timeout -s KILL 240 systemd-run --user --scope -q -p MemoryMax=5G -p MemorySwapMax=0 pnpm --filter umtool exec next build
+rm <worktree>/transcripts
+```
+On `main` `10cefd15` it fails in 6.4 s (0.97 GB): `TurbopackInternalError: Failed to write app
+endpoint /page … [project]/umtool/lib/paths.mjs … <DirAssetReference as
+ModuleReference>::resolve_reference failed … Symlink [project]/transcripts/channels/<slug>/archive
+is invalid, it points out of the filesystem root`.
+
+**The fix** (branch `fix/umtool-build-trace` off `main` `10cefd15`). Every path or fs call on a
+value derived from `process.cwd()` or `import.meta.url`, in a umtool module the app imports, now
+opens with `/* turbopackIgnore: true */`. That is the per-expression opt-out Turbopack's own
+message documents. The calls are in:
+- `lib/paths.mjs` (`findRepoRoot`, `CHANNELS_DIR`);
+- `lib/paths.ts` (`SONG_CODE`, `stateFile`);
+- `lib/tools.mjs`, `lib/trim.ts`, `lib/report/driver.mjs`;
+- `report-to-video/brand.mjs` and `cues.mjs`.
+
+The values at run time are unchanged. From `umtool/`, `REPO_ROOT`, `CHANNELS_DIR` and `SONG_DATA`
+print the same as on `main`, and `CHANNELS_DIR` / `TRANSCRIPTS_DIR` / `SONG_DIR` still win.
+`os.homedir()` joins are left as they are. A build with `HOME` pointed at a synthetic home inside
+the project, holding out-of-root symlinks at every home-derived root, succeeded, so the tracer does
+not follow `os.homedir()`.
+
+| build (5 GB cap, `/usr/bin/time -v`) | result | wall | max RSS | `.next` |
+|---|---|---|---|---|
+| `main` `10cefd15`, corpus linked | **fails** (the error above) | 6.4 s | 0.97 GB | — |
+| fix, no corpus | ok | 42.2 s (a busy machine; 22.2 s on an earlier run) | 0.80 GB | 1,009 files, 25,417,403 B |
+| fix, corpus linked | ok | 24.2 s | 0.80 GB | 1,009 files, 25,417,285 B |
+
+- **The two outputs are the same file set,** differing only in the build-id directory.
+- **Nothing from the corpus is traced.** 0 `.nft.json` entries are under `transcripts/`. The 14
+ files that contain the string `transcripts/channels` are all `.js.map` source maps of the code;
+ none is a chunk or an asset.
+
+**The guard: `scripts/umtool-build-trace.test.mjs`** (in `test:scripts`, 3 tests). It is a static,
+per-module check of umtool's app, components, lib, `report-to-video/*.mjs` and `song/paths.mjs`. A
+path or fs call carrying a value derived in that file from `process.cwd()`,
+`import.meta.url|dirname|filename` or `__dirname` must open with the opt-out. It costs about 0.1 s.
+With `main`'s `lib/paths.mjs` swapped in, it fails and names the defect:
+`umtool/lib/paths.mjs:118 : path.join(REPO_ROOT, "transcripts", "channels")`.
+
+| sha | what |
+|---|---|
+| `8346f824` | `umtool:` cwd-derived paths opt out of Turbopack's asset tracing |
+| `55699a20` | `scripts:` the guard |
+| _this_ | `plans:` this note, FACTS, the implementer rules' umtool build gate; the `[Unreleased]` bullet |
+
+**Gates:**
+- tsc is clean (all seven packages).
+- `node --check` passes on the five changed `.mjs`.
+- `test:scripts` **188 + 1 skipped** (`main` 185 + 1, plus the guard's 3).
+- The capped builds are in the table above.
+- umtool e2e (`faces`, `deck`, `clip-bench`, with the corpus link removed): **67 passed, 8
+ skipped, 0 failed** (1.9 min). The skips are the song-data specs, as in slice Q.
+
+**The other apps are safe by accident, not by rule.**
+- `common/lib/paths.ts`' `findMonorepoRoot()` falls back to `process.cwd()` (the app's own
+ directory).
+- `homepage/app/lib/source.ts` joins `process.cwd()` + `public`.
+- Both build today, so there is no finding to fix, only a note in FACTS.
+
+**For the parent:** merge, then rebuild and restart :3050. That is the parent's step.
+
### Slice R, as shipped — `archilyzer source publish` + `/source` (2026-09-28)
Branch `r12/source-mirror` off `main` `e6c5d2e3` (slice Q merged), worktree
diff --git a/plans/tools/implementer-rules.md b/plans/tools/implementer-rules.md
@@ -77,6 +77,16 @@ the Next.js reference for this version.
test:scripts` (156 + 1 skip); mcp `pnpm --filter yt-dlp-transcript-mcp test` (219) — check
`package.json` for the exact script names before running.
- `pnpm --filter editor exec next build` and `pnpm --filter export exec next build`.
+- **umtool's build runs with the corpus visible, under a memory cap.** A worktree has no
+ `transcripts/`, so a path Turbopack traces as a directory is empty there. It was hundreds of GB in
+ the primary, and the build was OOM-killed (FACTS, "A path joined from `process.cwd()` …"). From
+ the worktree root:
+ ```
+ ln -s <primary>/transcripts transcripts && timeout -s KILL 240 systemd-run --user --scope -q -p MemoryMax=5G -p MemorySwapMax=0 pnpm --filter umtool exec next build; rm transcripts
+ ```
+ It is for the BUILD only. Always remove the link, never commit it, and never run an app, an index
+ or a fixture builder through it. It should take about 25 s at under 1 GB, the same as without
+ the corpus.
- The slice's e2e spec list (in the prompt), detached and waited on as above.
- The numbers tool the prompt names, diff-empty (or "none", stated).
