commit 046047adaaa165d9771ffe2db48e9bc75abf4ea6
parent 429e3e639b0976e3ad71dff7a67222eeaeec7f99
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Mon, 28 Sep 2026 13:20:45 -0400
plans: release 13 slice W3, follow-up W3b recorded — the container Build all works; one [Unreleased] bullet
The rulings (Q1 kept; Q2 fixed here), the merge of main 6a769e92, bugs A and B
fixed in build-site.sh, the bundle check in the container and in the deploy
phase, Dockerfile.build.dockerignore's allow-list (807 files, 7.4 MB from any
checkout; the image 1.65 GB), the gates (tsc, common 2,125, the bites), and the
two-site fixture proof including the poisoned-image refusal. The W3 table row
owns build-site.sh and the ignore file.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
2 files changed, 168 insertions(+), 2 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -4,6 +4,7 @@
- **umtool reads the corpus from its checkout (or `TRANSCRIPTS_DIR`), and the song project's data defaults to `~/.local/share/archilyzer/song`.** If yours is elsewhere, link it there before restarting umtool: `mkdir -p ~/.local/share/archilyzer && ln -s <where the data is> ~/.local/share/archilyzer/song` (the data stays where it is). With no `CHANNELS_DIR`, umtool reads the corpus at `$TRANSCRIPTS_DIR/channels`, else the checkout's own `transcripts/channels`; it used to fall back to an absolute path that existed on one machine only. The song project's videos default to `~/reports/quartering-uh-song/videos`; `SONG_DIR` and `VIDEO_ROOT` still win. The song project's tracked manifests record their paths relative to the song folders, and the twenty one-off `umtool/song/*.sh` run logs, which only ever ran on the machine that wrote them, are gone.
- **`archilyzer doctor` checks the image Build all builds sites in.** When a container engine answers, a new **build image** section says whether the image named under **Settings → Build pipeline** is there, when it was built and how big it is. It warns when the image is missing, or older than the last change to its Dockerfile, and prints the one command that rebuilds it. Build all still builds or refreshes the image itself before it builds any site; the warning tells you ahead of time that the next Build all will spend that time. With no container engine the check is skipped in one line, and with no corpus it is only a note. It never fails the doctor.
- **The site build image runs Node 22 and pnpm 11**, the versions the rest of the workspace runs on, instead of Node 20 and pnpm 9, which did not read the workspace's install rules. The next Build all rebuilds the image from its first step, reinstalling every dependency, before it builds any site.
+- **Build all sites works in containers again.** Every site's container build had been failing while it prerendered `/favicon.ico`. Each site now builds from the data composed for it, never from files baked into the build image. A bundle whose `site.json` and `corpus.json` do not both name its site is refused before it is handed back or deployed. The image carries no corpus data, and its build context is about 7 MB from any checkout.
## [0.10.0] - 2026-09-28
- **The homepage can be built and deployed from `/sites`.** Under a new **Homepage** section, after Hub, there is **Build homepage** (tick **Deploy after build** to ship it in the same job, only if the build succeeds) and **Deploy homepage**, which ships the build already in `homepage/out`. A **Preview branch** box beside them sends either deploy to a Cloudflare Pages preview of the `archilyzer` project instead of production, and shows the preview's address as you type; a name Cloudflare would refuse or rewrite, or `main`, greys the deploy buttons out and says why. A line under the buttons says what a deploy would ship: when `homepage/out` was built (or that it holds no build yet), and where it goes, with the live URL. Deploy homepage with nothing built is refused before any job starts. The homepage reads the search index as it stands, so run **Build index** first when its numbers should move. The jobs run the same code as `archilyzer build homepage` / `deploy homepage`, and show on `/jobs` as `build-homepage`, `deploy-homepage` and `build-deploy-homepage`. The Hub section no longer describes the homepage.
diff --git a/plans/release-13.md b/plans/release-13.md
@@ -30,7 +30,7 @@ join, and a code conflict goes back to its slice.
|---|---|---|---|
| W1 | `r13/lows-editor` | Diagnostics cards keep their retry log when a retry empties the bucket; the `transcript-source.spec.ts:76` flake; `/sites` "built <when>" refreshes when a homepage build ends; `/jobs` labels for the four hub/homepage kinds; a queued-cancel writes its final sidecar status; doctor's worker engine binary from `paths` (L7); an `editor/package.json` `test` script | `editor/app/channels/[slug]/components/stages/**`, `editor/app/sites/components/{JobLane,HomepageBuildButtons}.tsx`, `common/jobs/{jobKinds,registry}.ts`, `common/lib/streamCommand.ts`, `common/bin/doctor.ts` (L7 only), the two specs |
| W2 | `r13/lows-export` | `ChartView` reaches `--chart-6` through `seriesColor(i)`; an export unit `test` script (named in the rules and CONTRIBUTING); O5's wording lows; the 2origin stage guard; `no-data.spec` skips on `workers > 1`; the mcp docs move to `archilyzer mcp` | `common/components/charts/ChartView.tsx`, `export/package.json`, `umtool/report-to-video/{svg-faces.mjs,README.md}` (comments), `export/playwright.2origin.config.ts`, `homepage/e2e/no-data.spec.ts`, `mcp/README.md`, `AGENTS.md`, `README.md`, `CONTRIBUTING.md`, `plans/tools/implementer-rules.md` |
-| W3 | `r13/build-image` | `Dockerfile.build` on Node 22 and pnpm 11; the stale `--network=none` comments; `archilyzer doctor` checks the build image's presence and age | `Dockerfile.build`, `common/publish/build.ts` (comments), `common/bin/doctor.ts` (the image check) |
+| W3 | `r13/build-image` | `Dockerfile.build` on Node 22 and pnpm 11; the stale `--network=none` comments; `archilyzer doctor` checks the build image's presence and age | `Dockerfile.build`, `common/publish/build.ts` (comments), `common/bin/doctor.ts` (the image check); from W3b `docker/build-site.sh`, `Dockerfile.build.dockerignore` |
| P0–P3 | `r13/phase-5` | one-core Phase 5: the corrected plan doc (P0, to the operator first), then slices 1–3 stacked | per `plans/one-core-phase-5.md` |
**Order:** W1 ‖ W2 ‖ W3, merged W1 → W2 → W3; then the parent rebuilds the build image and runs
@@ -47,7 +47,8 @@ Opus implementer. `git merge main` first fast-forwarded to `441bdbb2` (this file
`Dockerfile.build` on the workspace's Node and pnpm, its stale `--network=none` comments, and an
`archilyzer doctor` check of the image. The proof is a scratch-tag image and a fixture Build all,
not e2e. **It found that a container Build all fails every site on `main` as well** — see "Found
-and left" 1, before the parent's build-only Build all.
+and left" 1, before the parent's build-only Build all. Fixed in this slice by "Follow-up W3b"
+below.
**`Dockerfile.build`.**
- `ARG NODE_IMAGE=node:22.23.2-bookworm-slim` and `ARG PNPM_VERSION=11.26.0`.
@@ -239,4 +240,168 @@ built or replaced.
them before the parent's build-only Build all? As things stand, that Build all fails every site
in containers.
+#### Follow-up W3b — the container Build all works (2026-09-28)
+
+**The parent's rulings.**
+- **Q1: keep it.** The doctor warns about an absent or stale image only beside a corpus.
+- **Q2: fix bugs A and B in this slice**, before the parent's build-only Build all.
+ - **Owns, added:** `docker/build-site.sh` and a new `Dockerfile.build.dockerignore`.
+ - **Also touched:** `common/lib/builtExport.ts` + its test. It holds the one reader of a
+ bundle's identity, and the container check imports only `node:fs` from it. `PUBLISH.md`'s
+ container section, three sentences that this change made false.
+ - **Not touched:** the root `Dockerfile`, `Dockerfile.test`, the shared `.dockerignore` and the
+ rest of `docker/**`.
+ - **The runtime image does not use `build-site.sh`.** It is `Dockerfile.build`'s entrypoint and
+ what `runDockerBuildOne` mounts. The runtime image's `publish-site.sh` only names it in a
+ comment, and inside that container there is no engine, so Build all falls back to the host.
+
+**`git merge main` (`e6c5d2e3`, release 12 slice Q):** `644bd2d5`. One conflict, in
+`editor/CHANGELOG.md`'s `[Unreleased]`: both sides are kept, main's umtool bullet first, then W3's
+block. There was no code conflict, and tsc on the merge was clean (`w3b-tsc-1.log`, 60 s).
+
+**(A) `.next` stays inside the container** (`docker/build-site.sh`).
+- The symlink to `/site/.next` is gone. That symlink put turbopack's chunks at a real path under
+ `/site`, from which `next` cannot be resolved.
+- **Incremental cache: tried, measured, dropped.**
+ - The try: copy `.next/cache` into the container before the build and back out after it. It
+ holds the TypeScript check's `.tsbuildinfo` and the fetch cache, 492 KB.
+ - Two consecutive two-site runs:
+ - TypeScript: **30.3 s → 32.6 s**;
+ - compile: 18.3 s → 21.4 s;
+ - the whole run: **90 s → 88 s** (`w3b-fixture-build-{1,2}.log`).
+ - `export/next.config.ts` does not set `experimental.turbopackFileSystemCacheForBuild`, so a
+ Turbopack production build keeps no filesystem cache. The old symlinked `.next` never made a
+ container build incremental either.
+ - **The cost, plainly:** none measured. Each container's `next build` starts from an empty
+ `.next`.
+
+**(B) `export/public` is the composed `/site/public`.**
+- `build-site.sh` copies the image's tracked assets (`export/public/*.svg` only) into
+ `/site/public`, never over a file that is already there. It then replaces `export/public` with a
+ link to `/site/public` before `archilyzer build site <id> --nodata`.
+- Only `.svg` is copied, so an image that did bake data (see the proof) cannot leak it into a site
+ through the copy.
+
+**A wrong-site bundle cannot ship: two checks, both over `builtBundleProblem`.**
+- **The rule** (`common/lib/builtExport.ts`, new): `site.json`'s `siteId` and `corpus.json`'s
+ `site.id` must both be there and both name the site. The one sentence it returns names the
+ directory and the file that disagreed.
+- **In the container.** `build-site.sh` runs the check through `tsx -e` after the build and before
+ publishing. A refusal prints `[build-site] REFUSED: …`, exits 1 and leaves `/site/out` as it
+ was. A pass prints `out/ is the bundle of <id> (site.json, corpus.json)`.
+- **On the host.** `runDockerDeployAllPhase` now checks every per-site `out/` FIRST, before the
+ "no Cloudflare project" skip, the R2 upload and the Pages deploy.
+ - Before this, it shipped whatever `export/.export-builds/<id>/out` held. The host deploy checks
+ `builtSiteProblem`; this path checked nothing.
+ - A refusal is `failed` with the sentence as its reason, and the log line is
+ `[<id>] deploy REFUSED — …`.
+
+**`Dockerfile.build.dockerignore`: an allow-list.** BuildKit reads `<Dockerfile>.dockerignore` in
+place of the shared file.
+- **It admits:**
+ - `package.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml` and `tsconfig.base.json`;
+ - `common/` and `export/`;
+ - `docker/build-site.sh`.
+- **Inside those it excludes:**
+ - dot-entries, `node_modules`, `.next` and `out`;
+ - `*.tsbuildinfo`, `next-env.d.ts`, `.env*` and `*.pem`;
+ - `export/test-*`, and the playwright report dirs;
+ - `export/public/*` except `*.svg`.
+ No tracked file matches an exclusion except the five svgs, which are let back in.
+- **The export build reads nothing outside these.** The reads relative to `monorepoRoot` on the
+ build path are `export/service-worker/*` and `export/CHANGELOG.md`, and the rest is mounted.
+- **Measured:**
+ - The context is **807 files, 7.4 MB** from the primary checkout, and the same from the
+ worktree. `w3b-context.py` emulates the rules as a read-only walk, and its worktree list
+ matched the built image's `/repo` file-for-file (807 = 807, `w3b-image-files.txt`). No docker
+ build was run from the primary.
+ - The image's `/repo` is `common docker export node_modules` and the four manifests.
+ `/repo/export/public` holds only `file.svg globe.svg next.svg vercel.svg window.svg`.
+ - Under the shared `.dockerignore`, the primary sent `export/public`'s generated data (`subs/`
+ alone 1.8 GB), `.diarize/` (1.3 GB), umtool's data and the rest. The live image's
+ `COPY . .` layer is 3.23 GB, and its chmod layer 3.18 GB more.
+ - Now `COPY . .` is **7.14 MB** and the chmod layer **1.04 MB**. `COPY --chmod` was not worth a
+ portability question to podman, so the chmod stays.
+ - The image is **1.65 GB** (1,654,559,800 B; the live one is 7.72 GB). The final
+ `--no-cache --pull` build took **48 s**. Of the 1.65 GB, 1.39 GB is the pnpm install
+ (`w3b-image-final.log`).
+- **pnpm 11's check now passes on its own.** The image holds exactly the installed projects, so
+ `pnpm exec` works as the host uid even with `verify_deps_before_run=install` or `=error`. The
+ ENV stays because a builder that reads only the shared `.dockerignore` bakes all seven packages,
+ and then the first exec dies (W3's proof). `Dockerfile.build`'s comment now says so.
+
+| sha | what |
+|---|---|
+| `644bd2d5` | merge `main` `e6c5d2e3` (release 12 slice Q); `[Unreleased]` joined |
+| `698ecc16` | `common:` `builtBundleProblem` + 3 tests |
+| `d22faf21` | `docker:` `build-site.sh` (A) `.next` in the container, (B) `export/public` → `/site/public` (the `.svg` assets only), the bundle check before publishing |
+| `84988249` | `common:` `runDockerDeployAllPhase` refuses a bundle that is not the site's, first; 1 test |
+| `2c7b57be` | `docker:` `Dockerfile.build.dockerignore` (the allow-list); `Dockerfile.build`'s comments |
+| `ab75a5b8` | `docs:` `PUBLISH.md`'s container section (the per-site mount, the image's context, the refusals) |
+| `8b7c3864` | `common:` the doctor's git test turns off git's background maintenance (a W3 flake, below) |
+| _this_ | `plans:` this follow-up and the W3 table row's Owns; one `[Unreleased]` bullet |
+
+**Gates.**
+- **tsc:** clean before the code commits (`w3b-tsc-2.log`, 121 s) and before `8b7c3864`
+ (`w3b-gates-1.log`, 121 s).
+- **common: 2,125/2,125** (`w3b-gates-1.log`). That is 2,121 + 4: `builtExport` 3,
+ `build.test` 1.
+ - The first full run (`w3b-common-test.log`) was 2,124/2,125. The failure was W3's own git test:
+ git 2.55's detached `maintenance run --auto` after the test's commit took
+ `.git/objects/maintenance.lock` while the doctor ran.
+ - The fix turns maintenance off in the temp repo. After it the test passed 8/8 alone, and it
+ still fails without `GIT_OPTIONAL_LOCKS=0` (`.git/index` moves).
+- **`test:scripts`:** none of `scripts/*.test.mjs` covers `build-site.sh` or `Dockerfile.build`,
+ so it was not run.
+- **They bite** (`w3b-bite.log`), against `46d9b0cd`, before W3b:
+ - `builtExport.test.ts`, with a permissive `builtBundleProblem` shim so it loads: **9 passed,
+ 2 failed**. The two are the refusal tests; the third new test asserts a pass.
+ - `build.test.ts`: **10 passed, 1 failed**, the new deploy test. The old phase `skipped` both
+ sites for having no project. It could not deploy, because the test's sites have no Cloudflare
+ project on purpose: a real `wrangler` is never reachable from this test, even with the check
+ removed.
+
+**The proof: Build all's own code over a two-site fixture.** `$T/w3-fixture`: `w3site` over
+`test-youtube` (editor e2e's `one-youtube-channel-with-data`), and `w3other` over `tagchan`
+(`curated-tags-channel`). Settings name the scratch tag, with `maxParallelBuilds: 2`.
+1. **`pnpm archilyzer build all` passed** (`w3b-fixture-build-3.log`, 110 s, the final script):
+ - Phase A ran on the host, then `ensureBuildImage` on the scratch tag, then Phase B with both
+ sites in parallel. **`2/2 built`, exit 0.**
+ - Each `out/` has 167 files, and 0 files in either per-site dir are not host-owned.
+ - Each has `corpus.json`, `site.json`, `_headers`, `llms.txt`, `robots.txt`,
+ `summaries/manifest.json`, `archives/manifest.json`, `favicon.ico`, `index.html` and the
+ svgs.
+ - `w3site`'s `site.json` and `corpus.json` say `w3site`, and its `transcripts/` is
+ `test-youtube`. `w3other`'s say `w3other`, over `tagchan`.
+ - `builtBundleProblem` on the host is null for both (`w3b-fixture-outputs.log`).
+2. **The dangerous case, reproduced and refused** (`w3b-poisoned-runs.log`).
+ - The setup: a scratch image `FROM` the test tag, with `w3site`'s composed `public/` copied
+ into `/repo/export/public`. That is what a checkout that last composed `w3site` baked under
+ the shared ignore file. Then `w3other` was built, through `runDockerBuildOne`'s own
+ `docker run` argv.
+ - **The old `public/` handling** (the committed script minus the link block, bundle check
+ kept) produced `w3site`'s bundle for `w3other`:
+ - `[build-site] REFUSED: /repo/export/out holds a build of "w3site", not "w3other"
+ (site.json) — /site/out is left as it was`;
+ - exit 1;
+ - `/site/out` still held only its earlier `MARKER`.
+ - **The committed script** over the same image produced exit 0, with `site.json` and
+ `corpus.json` both `w3other`, `transcripts/` `tagchan` only, and 0 paths or files anywhere
+ in `out/` or `public/` mentioning `test-youtube`.
+3. **Every scratch tag has been removed:** `r13-build-image-test` and `r13-build-image-poisoned`.
+ `yt-dlp-transcript-browser-build` is still `ddb3fbce9f60`, from 2026-07-07.
+
+**What only the parent's post-merge Build all can prove:** the real sites at real parallelism,
+the real fonts and corpus sizes, and memory under `maxParallelBuilds`. The image itself is the
+same 807 files from the primary.
+
+**Found and left.**
+- **Legacy per-site dirs.** A host that ran the old container path has
+ `export/.export-builds/<id>/.next` from those runs, now unused. This machine has no
+ `.export-builds/` at all.
+- **The in-container bundle check adds a few seconds per site**: `pnpm exec tsx -e` is about 5 s
+ on the host, with `builtExport.ts` importing only `node:fs`.
+- **W3's "Found and left" 1 and 2 are fixed here.** Item 3 (the root `Dockerfile` and
+ `Dockerfile.test` on node 20 and pnpm 9.15.4) stands, and so does item 4.
+
## Rollout
