commit 14d9a5d0d03f96869330fb51b85643747bbd9bdf
parent 90c65c254c1338d8eb7f41d02f857719e30e4bf2
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 29 Sep 2026 22:53:55 -0400
plans: UT's review (SHIP AFTER FIXES) to its commits — M1 (a stale umtool build skips the post-build check; the changelog, FACTS and "Found and left" say so; the rollout rebuilds umtool in the primary, then runs test:scripts), L1, L2 (with the excludes the check sees only a name they miss), L3 ("left 31 / 49 of the 68 clean"), L4 (next-server.js:620); the rulings on the three questions
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
3 files changed, 89 insertions(+), 33 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -4,7 +4,7 @@
- **Transcripts that arrived after a video was first seen are counted.** The stats behind the homepage, the hub and every site's charts were cached per video and refreshed only when the video's metadata changed, so a transcript that came later — a Whisper run days after the download, or a video downloaded after the last index build — never reached them, and a video with YouTube captions alone had no transcription date. Counts and charts were low; the homepage could show a site with 0 transcripts, 0 channels and 0 hours while it served its videos. A stat is now also redone whenever the index re-reads the video, every transcript has a date, and a captioned video is dated by when its captions arrived rather than by a later Normalize run, so its place on "Transcribed over time" can move. **After updating, rebuild and restart the editor before anything else:** until then, **Build stats dataset** runs the old code and would undo the new stats, while a site, hub or homepage build already runs the new code — and the first stats build of any kind re-reads every video once (about 10–30 minutes on a large archive; it can be stopped and picks up where it stopped). Then build the index, the stats, the homepage, the hub, and the sites.
- **A stats build keeps the stats of a channel whose drive is not mounted, and will not undo a newer version's stats.** A channel whose media is on a drive that is not mounted (or is being moved) is left as it was instead of being read as a channel with no videos; a stats rebuild that has to start over refuses until the drive is back. A stats build refuses to clear stats written by a newer version of the editor; set `ARCHILYZER_STATS_ALLOW_DOWNGRADE=1` to roll back on purpose. Its log also says apart how many videos were downloaded since the last index build (they catch up after the next one) and how many the index skipped (no upload date, or it failed on them).
- **An index build keeps a channel whose drive is not mounted, instead of dropping it from the sites.** **Build index**, a site build's data phase and `archilyzer index` read a channel whose media is on a drive that is not mounted (or is being moved, or whose link and config disagree) as a channel with no videos: they removed its videos from the index, and the next site build published the channel as gone. Such a channel is now left as the last build had it — its videos stay in the index, its pages stay as they were, and the sites built next still list it — and the log names it, with its storage location: one line per channel, ` Held: N channel(s), K video(s) kept.` at the end of the `Diff:` line, and the channels again on the last line. A data folder that fails to read is held the same way, and a channel with no data folder at all is said in the log instead of passed over. An index rebuild that has to start over (after an update that changes the index's format, or with no index yet) refuses while any channel is held and says which; mount the drive first, or set `ARCHILYZER_INDEX_ALLOW_HELD=1` to rebuild without that channel until its drive is back and the index is built again — on the command for a command-line build (`ARCHILYZER_INDEX_ALLOW_HELD=1 pnpm archilyzer index`), or in the editor's own environment, with a restart, for **Build index** and the site builds started from the editor.
-- **umtool's build no longer lists its e2e test data, the e2e server's build folder or `.env.local` among a route's files.** The clip-audio route named its cache files in a way the bundler read as a pattern reaching into umtool's hidden folders, so its list of files took in the e2e fixture (where the tests link the song data), the e2e dev server's build folder and the env file: 1,704 of its 2,167 entries. It now lists what the other routes list (463). Those folders and env files are also excluded from every route's list, and `pnpm test:scripts` reads the last umtool build's lists back and fails on any such entry, so a checkout whose umtool build predates this fails that test until umtool is rebuilt. Nothing changes when umtool runs.
+- **umtool's build no longer lists its e2e test data, the e2e server's build folder or `.env.local` among a route's files.** The clip-audio route named its cache files in a way the bundler read as a pattern reaching into umtool's hidden folders, so its list of files took in the e2e fixture (where the tests link the song data), the e2e dev server's build folder and the env file: 1,704 of its 2,167 entries. It now lists what the other routes list (463). Those folders and env files are also excluded from every route's list, and `pnpm test:scripts` reads the last umtool build's lists back and fails on any such entry. A checkout whose umtool build predates its code (this change included) skips that check, saying so, until umtool is rebuilt (`pnpm --filter umtool exec next build`). Nothing changes when umtool runs.
- **Building the homepage now publishes the source: a read-only git mirror, its raw tree and a fresh tarball, behind a gate.** `archilyzer build homepage`, the `/sites` Homepage jobs and `pnpm ops build-homepage` run `archilyzer source publish` between compose and `next build`. It makes a fresh clone of the private `main` (the repository itself is never rewritten), rewrites that copy with git-filter-repo using your scrub rules (file contents and commit messages; your home directory becomes `/home/user` without a rule), and publishes it under `homepage/public` for `git clone https://archilyzer.pages.dev/source/archilyzer.git`, beside `/source/tree/` and the Downloads tarball. Before anything is written, every object of the rewritten history and every file about to be published is searched for every string you have denied; **one hit refuses the build**, and its log names the string only by where you wrote it (`denylist line 3 (len 5)`) and each hit by its object, field and byte offset — never a byte of the object. **A refusal withdraws the source**: the last publish is removed from `homepage/public` and the last build's copy from `homepage/out`, and **Deploy homepage refuses** a build whose source was not audited under today's rules and today's `main` ("run `archilyzer build homepage`, then deploy"). The rules live outside the repo, in `~/.config/archilyzer/source-scrub.txt` and `source-denylist.txt` (`ARCHILYZER_CONFIG_DIR`, `SOURCE_SCRUB_FILE`, `SOURCE_DENYLIST_FILE`); **without them the build refuses**, naming the missing file. **Put everything private in the denylist before any deploy, a preview included**: previews are public, and every deployment stays reachable at its own address until you delete it. Install git-filter-repo once (`pipx install git-filter-repo`; the editor's process needs `~/.local/bin` on its `PATH` to find it) — without it the build fetches it through `pipx run`, which needs the network — and gitleaks if you want its secret scan too. An unchanged `main` with unchanged rules is skipped, so a rebuild costs about 20 seconds only when something moved. A checkout with no git repository (the docker image, a tarball install) builds with the /source page's empty state. `archilyzer source publish --check` audits without writing, `archilyzer source audit <clone>/.git` checks any clone, `archilyzer build homepage --no-source` removes the published source instead, and `archilyzer doctor` reports the tools, the two files (rule counts and permissions, never their contents) and the last publish. `create-archives.sh` is gone. See PUBLISH.md, "The source mirror (homepage)".
- **umtool reads the corpus from its checkout (or `TRANSCRIPTS_DIR`), and the song project's data defaults to `~/.local/share/archilyzer/song`.** If yours is elsewhere, link it there before restarting umtool: `mkdir -p ~/.local/share/archilyzer && ln -s <where the data is> ~/.local/share/archilyzer/song` (the data stays where it is). With no `CHANNELS_DIR`, umtool reads the corpus at `$TRANSCRIPTS_DIR/channels`, else the checkout's own `transcripts/channels`; it used to fall back to an absolute path that existed on one machine only. The song project's videos default to `~/reports/quartering-uh-song/videos`; `SONG_DIR` and `VIDEO_ROOT` still win. The song project's tracked manifests record their paths relative to the song folders, and the twenty one-off `umtool/song/*.sh` run logs, which only ever ran on the machine that wrote them, are gone.
- **umtool's production build no longer reads the corpus folder.** Since umtool began finding the corpus from its checkout (the bullet above), `next build` treated the checkout's whole `transcripts/channels` as files to bundle. On a real archive it ran out of memory and was killed, so umtool could not be rebuilt. The build now ignores that folder and finishes in about 25 s at under 1 GB, the same as a checkout with no corpus. Nothing changes when umtool runs.
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -7344,8 +7344,10 @@ Slices Q (`4855f70b`) and R (`ffdeb2cd`): [`release-12.md`](release-12.md), the
- **The opt-out is per expression, and it is not documented for path or fs calls** (corrected by
release 15, slice UT). `path.join(/* turbopackIgnore: true */ process.cwd(), bar)` is Turbopack's
own advice, in the text of its "Encountered unexpected file in NFT list" issue (the "whole project
- was traced" warning; the 16.2.3 native binary carries it), and Next's own runtime uses the form
- (`join(/* turbopackIgnore: true */ contextDir, …)`). The Next docs (`08-turbopack.md` and
+ was traced" warning; the 16.2.3 native binary carries it), and Next's own server uses it, on
+ the join and on the fs call around it: `next/dist/server/next-server.js:620` (16.2.3) reads
+ `existsSync(/* turbopackIgnore: true */ (0, _path.join)(/* turbopackIgnore: true */ this.dir,
+ 'static'))`. The Next docs (`08-turbopack.md` and
`02-guides/lazy-loading.md`, "Magic Comments") list the comment only for `import()`, `require()`,
`require.resolve()` and `new Worker()`. It goes before the FIRST argument of each call, and it
changes nothing at run time. Measured in slice UT, it works on a `path.join` and on an fs call.
@@ -7359,15 +7361,17 @@ Slices Q (`4855f70b`) and R (`ffdeb2cd`): [`release-12.md`](release-12.md), the
1,704; the same with the `existsSync` opted out as well, 0; the join held in a variable and
only `existsSync(/* opt-out */ cached)` reading it, 0. On a cwd-derived value the join is a
known path, so the outer call traces the one file it names; the guard lets that through (its
- comment says so).
+ comment says so; the release 15 review ruled its four sites safe). Next's own
+ `next-server.js:620` (above) opts out both calls.
- **A value the tracer cannot know is a dynamic part, not ignored** (corrected by release 15, slice
UT). `process.env.*`, `os.homedir()`, a parameter and an imported binding all make patterns over
the app's own directory (`umtool/`):
- 66 of umtool's 68 routes trace its whole tree outside dot-directories (361 files,
`next.config.ts` among them). Opting out every path op in `song/paths.mjs`, `lib/paths.mjs`
and `lib/paths.ts` (`path.resolve(process.env.X ?? path.join(os.homedir(), …))` and the like)
- cleaned 31 routes; every path op in all 53 modules that have one (319 calls), 49. The rest
- come through fs calls (`lib/report/snapshots.mjs` is the first the warning names).
+ left 31 of the 68 routes clean; every path op in all 53 modules that have one (319 calls),
+ 49 (`_global-error` and `_not-found` were clean before). The rest come through fs calls
+ (`lib/report/snapshots.mjs` is the first the warning names).
- The clip-audio route's `path.join(CACHE_DIR, `${stamp}.${asMp3 ? "mp3" : "wav"}`)` (CACHE_DIR
imported, built on the env or home directory) also took in the dot-directories: 1,704 of its
2,167 entries were the e2e fixture (`.e2e-song`), the e2e server's build directory
@@ -7376,10 +7380,12 @@ Slices Q (`4855f70b`) and R (`ffdeb2cd`): [`release-12.md`](release-12.md), the
from another module (the fix, `cacheFile` in `umtool/lib/paths.mjs`), it did not. The video
route's `.mp4` join did not reach the `.mp4` inside `.e2e-song`.
- **A pattern walk does not enter a symlinked directory**, in the root or out of it: a link at
- `umtool/.e2e-song/data/planted` to `<primary>/transcripts/channels`, and one to the worktree's
- `common/` (675 files), gave 0 entries, before and after the fix. That is what the synthetic-HOME
- build showed, not that the env and home directory are unfollowed. A KNOWN directory (slice Q's
- `<root>/transcripts/channels`) is different: it is walked through its symlinks.
+ `umtool/.e2e-song/data/planted` to `<primary>/transcripts/channels` gave 0 entries before and
+ after the fix, while the old route's walk listed the real files beside it (`data/mkvocals`);
+ one to the worktree's `common/` (675 files) gave 0 on the old route. That is what the
+ synthetic-HOME build showed, not that the env and home directory are unfollowed. A KNOWN
+ directory (slice Q's `<root>/transcripts/channels`) is different: it is walked through its
+ symlinks.
- **The guard is `scripts/next-build-trace.test.mjs`** (in `test:scripts`; it was
`umtool-build-trace.test.mjs` until release 14 widened it). It scans umtool's app, components,
lib, `report-to-video/*.mjs` and `song/paths.mjs`, and, since release 14 (F8), `homepage/app`,
@@ -7396,12 +7402,21 @@ Slices Q (`4855f70b`) and R (`ffdeb2cd`): [`release-12.md`](release-12.md), the
(umtool 211 → 218 modules, the rest 895 → 897); a test pins them.
- The checked calls include `open`, `writeFile`, `appendFile`, `createWriteStream` (and their
Sync forms) and the `fs.promises.` / `fsPromises.` prefixes. No new finding.
- - **It reads umtool's last build back** when `umtool/.next/server` exists (it skips, saying
- so, when not). Every `.nft.json` under `umtool/.next` but `cache/` and `dev/` fails on an
- entry outside the repo, under `transcripts/`, or through any name starting with a dot other
- than the build's own directory and `node_modules/.pnpm`. On the old clip-audio route it
- fails with 1,704 entries; on the fix it passes. It reads what is on disk: in a checkout whose
- umtool build predates the fix, it fails until umtool is rebuilt.
+ - **It reads umtool's last build back.** Every `.nft.json` under `umtool/.next` but the
+ build's own `cache/` and `dev/` fails on an entry outside the repo, under `transcripts/`, or
+ through any name starting with a dot other than the build's own directory and
+ `node_modules/.pnpm`.
+ - **It skips, saying so,** when there is no build (`umtool/.next/server` or `BUILD_ID`
+ missing), and when `BUILD_ID` is older than `umtool/next.config.ts` or any umtool module it
+ scans (review M1). A merge or checkout gives the changed files new mtimes, and umtool runs
+ under `next dev`, which does not refresh `.next`, so a stale build skips until umtool is
+ rebuilt instead of failing on a call already fixed.
+ - **What it can see** (review L2). Without the excludes, the old clip-audio route failed it with
+ 1,704 entries (the primary's pre-fix build: 1,705, `test-results/.last-run.json` too). With
+ the excludes umtool's config now carries, such a pattern shows only through a name they
+ miss: `test-results/.last-run.json` after an e2e run, `.next-shots`, the corpus, a path
+ outside the repo. A checkout with no e2e run behind it is blind to it; the fix at the call
+ is what keeps the route clean.
- **The build gate** is run with the corpus visible and under a memory cap (the command is in
`plans/tools/implementer-rules.md`). Linking `<primary>/transcripts` into a worktree is for a
BUILD only. Remove the link afterwards: never run an app, an index or a fixture builder through it.
diff --git a/plans/release-15.md b/plans/release-15.md
@@ -285,9 +285,10 @@ it shows one warning on every build, and after the fix it is still true:
`next.config.ts` (the file the warning names) among them. Only `_global-error` and `_not-found`
do not.
- Path and fs calls on env, home-directory and parameter values do it. Opting out every path op in
- `song/paths.mjs`, `lib/paths.mjs` and `lib/paths.ts` cleaned 31 routes (the audio route 463 →
- 102). Opting out all 319 path ops in the 53 modules that have one cleaned 49. The rest come
- through fs calls; the next import trace the warning names is `lib/report/snapshots.mjs`.
+ `song/paths.mjs`, `lib/paths.mjs` and `lib/paths.ts` left 31 of the 68 routes clean (the audio
+ route 463 → 102). Opting out all 319 path ops in the 53 modules that have one left 49 clean. The
+ two clean before are among them. The rest come through fs calls; the next import trace the
+ warning names is `lib/report/snapshots.mjs`.
- That walk skips dot-directories and does not enter symlinks. The warning names the same file for
it as for the audio route's, so it cannot tell the two apart; the second line and the guard below
cover the dot-directories instead. The config's comment says all of this.
@@ -297,13 +298,23 @@ it shows one warning on every build, and after the fix it is still true:
the fix, and an in-root link to `common/` (675 files) gave 0 on `main`'s route. What the pattern
reached was the fixture's real files.
-**The guard** (`scripts/next-build-trace.test.mjs`, `9a375ecd`), 6 → 9 tests:
-- **(a) It reads umtool's last build back.** When `umtool/.next/server` exists, every `.nft.json`
- under `umtool/.next` (`cache/` and `dev/` aside) fails on an entry outside the repo, under
- `transcripts/`, or through any name starting with a dot but the build's own directory and
- `node_modules/.pnpm`. With no build it skips and says how to make one. On `main`'s route it fails
- with 1,704 entries (the first 20 listed); on the fix it passes. A unit test pins
- `forbiddenTrace`.
+**The guard** (`scripts/next-build-trace.test.mjs`, `9a375ecd`, and after the review `a4d100b4`,
+`c3e8a2c7`), 6 → 10 tests:
+- **(a) It reads umtool's last build back.** Every `.nft.json` under `umtool/.next` but the
+ build's own `cache/` and `dev/` fails on an entry outside the repo, under `transcripts/`, or
+ through any name starting with a dot but the build's own directory and `node_modules/.pnpm`.
+ Unit tests pin `forbiddenTrace` and which directories are read.
+ - **It skips, saying so,** with no build, and (review M1) when `umtool/.next/BUILD_ID` is older
+ than `umtool/next.config.ts` or any umtool module the guard scans. The message gives the
+ build's time, the first newer file and how to rebuild. A merge or checkout gives the changed
+ files new mtimes, and umtool runs under `next dev`, which does not refresh `.next`, so a stale
+ build skips instead of failing on a call already fixed.
+ - **What it can see** (review L2). Without the excludes, `main`'s route failed it with 1,704
+ entries (the first 20 listed); the primary's pre-fix build fails it with 1,705 (the review:
+ `test-results/.last-run.json` too). With the excludes in place, a pattern like that one shows
+ only through a name they miss: `test-results/.last-run.json` after an e2e run, `.next-shots`,
+ the corpus, a path outside the repo. A checkout with no e2e run behind it is blind to it; the
+ fix at the call is what keeps the route clean.
- **(b) The scan set follows relative imports** out of the listed folders, to any depth. It adds
the review's L1 modules and no others: `common/bin/_publicFile.ts`, `homepage/content/docs.ts`
(895 → 897) and seven `umtool/song` modules, `reasons`, `archive-url`, `pitch`, `flatness`,
@@ -321,7 +332,8 @@ reached was the fixture's real files.
**FACTS**, "A path joined from `process.cwd()` …", corrected:
- "documented": the Next docs list `turbopackIgnore` only for `import()`, `require()`,
`require.resolve()` and `new Worker()`. The path form is Turbopack's own advice, in the warning's
- text in the 16.2.3 binary, and Next's own runtime uses it.
+ text in the 16.2.3 binary, and Next's own server uses it on the join and on the fs call around
+ it (`next/dist/server/next-server.js:620`; review L4).
- "an outer fs call on an opted-out `path.join` is covered": it is not (the second bisect table).
- "Not followed by the tracer: `os.homedir()` and `process.env.*`": such values are dynamic parts,
and make patterns over the app directory. The synthetic-HOME build showed that a pattern walk
@@ -336,7 +348,11 @@ reached was the fixture's real files.
| `f18fa034` | `umtool:` `outputFileTracingExcludes`; the `ignoreIssue` kept, its comment the measured reason |
| `8b3409c2` | `common:` `under(first, ...rest)` |
| `9a375ecd` | `scripts:` the post-build check, the relative-import scan set, the opens and writes, the header |
-| this commit | `plans:` this section; FACTS; the editor changelog |
+| `750fc850` | `plans:` this section; FACTS; the editor changelog |
+| `a4d100b4` | `scripts:` review M1: the post-build check skips a build older than the code it judges |
+| `c3e8a2c7` | `scripts:` review L1 (only the build's own `cache/` and `dev/` unread, pinned), L2 (the check's comment says what it can see), L3 |
+| `b4d6607d` | `umtool:` review L3 in the `ignoreIssue` comment |
+| this commit | `plans:` the review's findings to their commits; FACTS (M1, L2, L3, L4); the changelog (M1); the rollout note |
#### Gates (logs `$T/ut-*.log`)
@@ -393,17 +409,42 @@ reached was the fixture's real files.
makes `transcripts/transcripts` and builds without the corpus where the paths point. This worktree
had one (an `index.mdb` from 2026-09-28), and my first two corpus-linked builds ran like that. The
numbers above are from builds that set it aside and put it back. `ln -sT` would refuse instead.
-- **A checkout whose umtool build predates this fails `test:scripts`** until umtool is rebuilt:
- the post-build check reads what is on disk. The primary's `umtool/.next` is from before it.
+- **A checkout whose umtool build predates its code skips the post-build check** until umtool is
+ rebuilt (review M1). The primary's `umtool/.next` is from before this slice, so the check skips
+ there until the rollout rebuilds it.
#### Decisions the operator could overturn
| What I did | The alternative |
|---|---|
-| A `cacheFile` helper in `lib/paths.mjs`, used by all three routes that join `CACHE_DIR` | Opt-outs on the audio route's join and on every fs call on its value, in that route only |
+| A `cacheFile` helper in `lib/paths.mjs`, used by all three routes that join `CACHE_DIR`. **Ruled at review: keep; one way to name a cache file.** | Opt-outs on the audio route's join and on every fs call on its value, in that route only |
| The `ignoreIssue` stays, narrow, with the measured reason in its comment | Drop it: one warning on every build, naming one route's import trace |
| The post-build check refuses any dot-named path but the build's own and `node_modules/.pnpm` | Refuse only `.e2e-song`, `.next-e2e`, `.env*` and `.git` |
-| The post-build check covers umtool only | Also read the editor's, the export's and the homepage's builds |
-| The static check keeps its nested-call exemption, documented as a simplification | Require the outer opt-out: four new findings, two in files other slices own |
+| The post-build check covers umtool only. **Ruled at review: umtool only.** In the primary the editor's traces would fail it (75 of 81, `editor/.env` among them) and the export's `.export-index` entries would be misjudged. | Also read the editor's, the export's and the homepage's builds |
+| The static check keeps its nested-call exemption, documented as a simplification. **Ruled at review: it stays; the four sites are safe.** | Require the outer opt-out: four new findings, two in files other slices own |
+| A stale build skips the post-build check (review M1) | Fail on it, as first shipped |
+
+#### Review
+
+**Verdict: SHIP AFTER FIXES** (`ut-review.md` in the job's scratch). No High. The reviewer found
+the fix's nine call-site paths byte-identical, the bisect logs in agreement with the tables, the
+excludes' key and shape right, and 0 dot entries across all 70 traces of a fresh build.
+
+| Finding | Where |
+|---|---|
+| M1: a stale umtool build turns `test:scripts` red, pointing at a call already fixed | `a4d100b4`: the check skips a build older than `next.config.ts` or any module it scans; the changelog, FACTS and "Found and left" say so; the rollout note below |
+| L1: `cache`/`dev` skipped at any depth | `c3e8a2c7`: only directly under `.next`; a test with a route directory named each |
+| L2: with the excludes in place the check cannot see the original defect | `c3e8a2c7` (the test's comment), guard (a) above and FACTS: it sees only a name the excludes miss |
+| L3: "cleaned 31 / 49" | `c3e8a2c7`, `b4d6607d`, this commit: "left 31 / 49 of the 68 clean" |
+| L4: FACTS cited the native binary's string as Next's runtime | this commit: `next-server.js:620` |
+| L5: the build-gate command's `ln -s` | The parent's (`implementer-rules.md`) |
+| Info: the editor's and export's primary builds carry the same class of widening | Recorded in the decisions table; a later slice's |
+
+**For the rollout.** Rebuild umtool in the primary first, under the cap
+(`timeout -s KILL 240 systemd-run --user --scope -q -p MemoryMax=5G -p MemorySwapMax=0 pnpm
+--filter umtool exec next build`), then run `pnpm run test:scripts` there. That is the only proof
+on the real fixture, which carries `.env.local`, `.next-shots` and `test-results/.last-run.json`,
+two of them names the excludes do not cover. Until that rebuild the post-build check skips in the
+primary, saying why. umtool's code changes nothing at run time.
## Rollout
