commit 404d87b9c7e19f0d587186da4c78953c2f678919
parent 3bd33837f5e37f17bc661e0e9a7f92ead2e5ef4f
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Mon, 28 Sep 2026 12:53:00 -0400
plans: release 13 slice W3, as shipped — the build image refreshed; two [Unreleased] bullets
The record: Dockerfile.build on node:22.23.2-bookworm-slim + pnpm 11.26.0 (and
the pnpm 11 verify-deps env the proof found it needs), the network comments,
the doctor's build-image block, the gates (tsc, common 2,121, the bite), and the
scratch-tag proof. The proof also found that a container Build all fails every
site on main as well: docker/build-site.sh's .next symlink breaks turbopack's
external imports, and next build publishes the baked export/public rather than
the composed /site/public. Both were probed to a working fix, left uncommitted
(docker/** is not W3's).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
2 files changed, 203 insertions(+), 0 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,5 +1,9 @@
# Changelog
+## [Unreleased]
+- **`archilyzer doctor` checks the image Build all builds sites in.** When a container engine answers, a new **build image** section says whether the image named under **Settings → Build pipeline** is there, when it was built and how big it is. It warns when the image is missing, or older than the last change to its Dockerfile, and prints the one command that rebuilds it. Build all still builds or refreshes the image itself before it builds any site; the warning tells you ahead of time that the next Build all will spend that time. With no container engine the check is skipped in one line, and with no corpus it is only a note. It never fails the doctor.
+- **The site build image runs Node 22 and pnpm 11**, the versions the rest of the workspace runs on, instead of Node 20 and pnpm 9, which did not read the workspace's install rules. The next Build all rebuilds the image from its first step, reinstalling every dependency, before it builds any site.
+
## [0.10.0] - 2026-09-28
- **The homepage can be built and deployed from `/sites`.** Under a new **Homepage** section, after Hub, there is **Build homepage** (tick **Deploy after build** to ship it in the same job, only if the build succeeds) and **Deploy homepage**, which ships the build already in `homepage/out`. A **Preview branch** box beside them sends either deploy to a Cloudflare Pages preview of the `archilyzer` project instead of production, and shows the preview's address as you type; a name Cloudflare would refuse or rewrite, or `main`, greys the deploy buttons out and says why. A line under the buttons says what a deploy would ship: when `homepage/out` was built (or that it holds no build yet), and where it goes, with the live URL. Deploy homepage with nothing built is refused before any job starts. The homepage reads the search index as it stands, so run **Build index** first when its numbers should move. The jobs run the same code as `archilyzer build homepage` / `deploy homepage`, and show on `/jobs` as `build-homepage`, `deploy-homepage` and `build-deploy-homepage`. The Hub section no longer describes the homepage.
- **`pnpm ops build-homepage` and `pnpm ops deploy-homepage`.** The same two jobs over HTTP: `build-homepage` takes `{"deploy": true}` to deploy after a successful build, and both take `{"preview": "<branch>"}` for a preview (`build-homepage` only with `deploy`). `deploy-homepage` answers with the preview's address, and refuses a bad preview name or a missing build before any job starts.
diff --git a/plans/release-13.md b/plans/release-13.md
@@ -40,4 +40,203 @@ Then one integration gate on `main` (`r13/integration`) and the runbook
## Record
+### Slice W3, as shipped — the build image refreshed (2026-09-28)
+
+Branch `r13/build-image` off `main` `bf6904e8`, worktree `/home/user/Projects/r13-build-image`, one
+Opus implementer. `git merge main` first fast-forwarded to `441bdbb2` (this file). Three items:
+`Dockerfile.build` on the workspace's Node and pnpm, its stale `--network=none` comments, and an
+`archilyzer doctor` check of the image. The proof is a scratch-tag image and a fixture Build all,
+not e2e. **It found that a container Build all fails every site on `main` as well** — see "Found
+and left" 1, before the parent's build-only Build all.
+
+**`Dockerfile.build`.**
+- `ARG NODE_IMAGE=node:22.23.2-bookworm-slim` and `ARG PNPM_VERSION=11.26.0`.
+ - Node is the host's `node --version`; nothing else pins it (no `.nvmrc`, no `engines`, no
+ `packageManager`). It is bookworm like the root Dockerfile's build stage.
+ - pnpm is the host's `pnpm --version`, the one the worktree installed with. pnpm 11 needs
+ Node >= 22.13.
+ - `ensureBuildImage` passes no build args, so these defaults are what Build all builds.
+- **pnpm 11 needed one more line, and the proof found it:**
+ `ENV pnpm_config_verify_deps_before_run=false pnpm_config_update_notifier=false`.
+ - Before every `pnpm exec` / `pnpm run`, pnpm 11 checks that the whole workspace is installed,
+ and runs `pnpm install` when it is not.
+ - The image installs root, common and export. After `COPY . .` the workspace has 8 projects, so
+ the first `pnpm --filter … exec` in `build-site.sh` ran `pnpm install`. As the host uid that
+ died with `EACCES: permission denied, open '/repo/_tmp_…'` (`w3-image-versions.log`).
+ - pnpm 11 reads `pnpm_config_*` from the environment, not `npm_config_*` (both were tried). The
+ update notice is off because every site's log would otherwise print it.
+- **Comments:**
+ - The header no longer calls the containers "hermetic". It says the network is on
+ (`runDockerBuildOne`, for `next/font/google`) and names the entry command as
+ `archilyzer build site <id> --nodata`.
+ - The corepack reason is now "no `packageManager` pin, so corepack would download a pnpm of its
+ own choosing in every container". It used to say "offline".
+ - The chmod comment rests on `--rm` and "writes nothing back but /site". It used to rest on
+ `--network=none`.
+- The contract with `build.ts` is unchanged: `WORKDIR /repo`, the entrypoint, the mounts, `-u`,
+ and the chmod of `/repo/export`.
+
+**`common/publish/build.ts`** (no behaviour change).
+- `dockerBin(env = process.env)` is now exported. It was private and read only `process.env`.
+- `buildImageArgs(pipeline)` is new, and `ensureBuildImage` now runs it. It is the one spelling of
+ the image's `docker build` argv, which the doctor prints.
+- The comments at `:437-456`: `ensureBuildImage`'s comment now says it runs before every Phase B,
+ what a Dockerfile or lockfile change costs there, and that the doctor warns about it ahead of
+ time. `runDockerBuildOne`'s network comment was already right and is unchanged.
+
+**`archilyzer doctor`: a "build image" section** between umtool's and the ports. It is a block of
+its own: W1 changes the worker-engine line in "tools", and release 12 adds a source block; neither
+is adjacent.
+- **The probe.** `<DOCKER_BIN or docker> version` must exit 0, which is what `dockerAvailable`
+ asks. Then `<bin> image inspect --format '{{.Created}}|{{.Size}}' <tag>`.
+- **The tag and the Dockerfile** come from the effective settings (`settingsFromFile`, or the
+ defaults when the file is absent). That is the same `getSettings().buildPipeline` that `build.ts`
+ reads, so `yt-dlp-transcript-browser-build` is not spelled a second time.
+- **The Dockerfile's last change.**
+ - In a checkout it is the file's last commit (`git log -1 --format=%ct`), unless
+ `git status --porcelain` shows uncommitted edits; then it is the file's mtime.
+ - With no checkout, or no git, it is the mtime.
+ - `GIT_OPTIONAL_LOCKS=0` stops `git status` from refreshing `.git/index`, so the doctor stays
+ read-only. Without it, the test's tree comparison fails because `.git/index`'s mtime moves.
+- **Grading:**
+ - With no engine there is one `--` line, "skipped: `docker version` did not answer …".
+ - With an engine, an absent image is a WARN, and so is an image created before the Dockerfile's
+ last change. Both print the command. Anything else is `ok`, with the date, age and size.
+ - A Dockerfile the settings name that is not there gets its own `dockerfile` WARN.
+ - **Never a FAIL.** The WARNs apply only beside a corpus: with no channels the same lines are
+ notes, as for a tool nothing needs yet (Question 1).
+- **The command** is `rebuild it now: cd <root> && docker build -f Dockerfile.build -t <tag> .`,
+ built from `buildImageArgs` and shell-quoted. `DOCKER_BIN` changes both the engine asked and the
+ command printed.
+- `parseEngineTime` reads docker's RFC 3339 with nanoseconds and podman's Go default format.
+- On this machine, run from the worktree (no corpus there, so a note):
+ ```
+ build image
+ -- build-image "yt-dlp-transcript-browser-build" built 2026-07-07 16:04 (83 days ago), 7.72 GB — before Dockerfile.build's last change (2026-09-28 16:47, its last commit), so the next Build all rebuilds it from the changed step on
+ rebuild it now: cd /home/user/Projects/r13-build-image && docker build -f Dockerfile.build -t yt-dlp-transcript-browser-build .
+ ```
+ With the fixture corpus and the scratch tag, the same line is a `WARN`.
+
+| sha | what |
+|---|---|
+| `bb3d0f6a` | `docker:` `Dockerfile.build` on node:22.23.2-bookworm-slim + pnpm 11.26.0 (build args), the pnpm 11 `verify_deps_before_run` / `update_notifier` env, the network and corepack comments |
+| `40dfc008` | `common:` the doctor's build-image block + 7 tests; `build.ts` exports `dockerBin(env)` and `buildImageArgs`, and `ensureBuildImage`'s comment |
+| _this_ | `plans:` this record; two `[Unreleased]` bullets in `editor/CHANGELOG.md` |
+
+**Gates**, all from the worktree root:
+- **tsc** (`pnpm -r --no-bail --workspace-concurrency=1 exec tsc --noEmit`) was clean before both
+ commits: `w3-tsc-1.log` (168 s), then `w3-tsc-2.log` (50 s) on the final tree. The only change
+ between the two runs was one comment.
+- **common: 2,121/2,121** (`w3-common-test.log`, 132 s). The 7 new tests are all in
+ `bin/doctor.test.ts`, so `main` has 2,114; the prompt said 2,112. `test:scripts` does not cover
+ `common/bin`, so it was not run.
+- No generated doc, no editor or export build, and no e2e: nothing under `editor/app` or `export/`
+ changed, and the proof below replaces e2e.
+- **The new tests bite.** `w3-bite.log` ran the new `doctor.test.ts` against `main`'s `doctor.ts`,
+ with a one-line `parseEngineTime` shim so the file loads: **8 passed, 7 failed**, and the 7
+ failures are exactly the new tests. Without the shim the whole file fails to import.
+ Separately, with `GIT_OPTIONAL_LOCKS: "0"` removed, the git test fails on `.git/index`'s mtime.
+
+**The proof, instead of e2e.** Every log is under `$T`. The scratch tags were `r13-build-image-test`,
+`-old` and `-probe`, all removed afterwards. `yt-dlp-transcript-browser-build` was never tagged,
+built or replaced.
+1. **The image.** `docker build -f Dockerfile.build -t r13-build-image-test .` from the worktree
+ root took **53 s** on the first build (base pulled, no cached layers). The final
+ `--no-cache --pull` build took **52 s**. Size **1.67 GB** (1,673,627,497 B), from a 26.7 MB
+ context (`w3-docker-build*.log`).
+2. **Inside it** (`w3-image-versions*.log`):
+ - node **v22.23.2**, pnpm **11.26.0**.
+ - As the host uid with `HOME=/tmp`, which is how `runDockerBuildOne` runs it:
+ `pnpm --filter export exec next --version` gives Next.js v16.2.3, and
+ `pnpm --filter yt-dlp-transcript-common exec tsx --version` gives tsx v4.21.0. `lmdb`'s
+ native module opens, writes and reads.
+ - `pnpm ls --depth 0` over export and common lists 53 packages in 2 projects. This was run as
+ root: as another uid, pnpm 11's `ls` opens the store index under `/root` and fails EACCES.
+ `ls` is not on the build path.
+3. **Build all's own code over a fixture** (`w3-fixture-build.log`, 40 s).
+ - The run: `pnpm archilyzer build all` from the worktree, with `TRANSCRIPTS_DIR`,
+ `SETTINGS_FILE` and `EXPORT_{PUBLIC,INDEX,BUILDS}_DIR` pointed at `$T/w3-fixture`.
+ - The fixture: editor e2e's `one-youtube-channel-with-data` channel and one site, `w3site`.
+ Settings: `buildPipeline.dockerImage: "r13-build-image-test"`, `maxParallelBuilds: 1`.
+ - Phase A on the host passed: index, stats, templates and the archive warm.
+ - `ensureBuildImage` rebuilt the scratch tag from cached layers.
+ - In the Phase B container, `build-site.sh` composed into `/site/public` (with the read-only
+ archive cache materialized), and `next build` compiled and type-checked.
+ - It then **failed prerendering `/favicon.ico`**:
+ `Failed to load external module next/dist/compiled/@vercel/og/index.node.js: … Cannot find
+ package 'next' imported from /site/.next/server/chunks/[turbopack]_runtime.js`.
+4. **The failure predates W3.** `main`'s `Dockerfile.build` (node:20, pnpm 9.15.4) was built as
+ `r13-build-image-old` through the same Build all and failed identically
+ (`w3-fixture-build-old.log`, 136 s).
+5. **Two probes of the proposed fix** replaced the entrypoint with a copy from `$T` over the
+ `docker run` argv `runDockerBuildOne` builds. Nothing was committed under `docker/`.
+ - **Probe 1** kept `.next` in the container instead of linking it to `/site/.next`. The run
+ used a derived image with `export/public`'s dangling links deleted, which is what a clean
+ clone has.
+ - It exits 0 with 149 files, but `/site/out` has no `corpus.json`, `_headers`, `llms.txt`,
+ `robots.txt`, `site.json`, `summaries/`, `transcripts/`, `archives/` or `stats/`. All of
+ these are only in `/site/public` (`w3-probe1-run.log`).
+ - **Probe 2** made the same `.next` change and, in addition, ran
+ `cp -an export/public/. /site/public/ && rm -rf export/public && ln -s /site/public export/public`.
+ - It exits 0 with **167 files**, in 51 s. `out/` holds `w3site`'s `corpus.json`,
+ `transcripts/test-youtube/{manifest,page-0000}.json`, `summaries/` and `archives/`, all
+ host-owned (`w3-probe-run.log`).
+6. **What only the parent's post-merge Build all can prove:**
+ - the image built from the primary's context: its size, and the time to transfer that context;
+ - every real site at `maxParallelBuilds`, with the real fonts;
+ - the deploy phase.
+ As `main` stands, that Build all fails every site at `/favicon.ico` on the container path
+ (bug A below), with or without W3.
+
+**Found and left.**
+1. **`docker/build-site.sh` has two bugs, and a container Build all cannot ship a site until they
+ are fixed.** `docker/**` is not W3's.
+ - **(A) `export/.next` is a symlink to `/site/.next`.**
+ - Turbopack's runtime imports next's externals from its own real path, and under `/site`
+ there is no `node_modules`. `next build` dies at `/favicon.ico`.
+ - **(B) `next build` copies `export/public` into `out/`.**
+ - `export/public` is the copy BAKED into the image, not the composed `EXPORT_PUBLIC_DIR=/site/public`.
+ - Built from a clean clone, `out/` would have none of the site's data (probe 1).
+ - Built from the primary, `out/` would carry the primary's stale export/public. That is
+ whatever site the host composed last, without `summaries/` and `transcripts/`, which
+ `.dockerignore` excludes.
+ - **Why no one saw it.** The live image dates from 2026-07-07, and `ensureBuildImage` runs
+ before every fan-out, so no container Build all has run since then.
+ - **The host path is unaffected:** single-site builds, `archilyzer build site`, and Build all
+ when no engine answers.
+ - **The proposed fix** is proven by probe 2 and is uncommitted: `$T/w3-build-site-probe.sh`.
+ Its cost is that `.next` stops persisting between runs, so incremental `next build` is lost.
+ Persisting only `.next/cache` could win that back, but it is untested.
+2. **The build context bakes the checkout's generated data, twice.**
+ - From the primary it includes:
+ - the gitignored entries of `export/public` that `.dockerignore` does not name (`subs/`
+ 1.8 GB, `posts/`, `stats/`, `digests/`, `corpus.json`, …);
+ - `.diarize/` (1.3 GB);
+ - umtool's data (1.1 GB).
+ - `/repo/export` is in the image twice, once in the `COPY` layer and once in the `chmod` layer.
+ The live image is 3.23 GB + 3.18 GB of its 7.72 GB.
+ - From a worktree, `export/public`'s entries are absolute links into the primary and are baked
+ dangling. The first probe died on `stat '/repo/export/public/_headers'`.
+ - Fix B keeps this data out of `out/`. It still sizes the image, and it invalidates the
+ `COPY` layer after every host build.
+ - `.dockerignore` is shared with the root `Dockerfile` and `Dockerfile.test`, so W3 left it.
+ The follow-up is a BuildKit per-Dockerfile ignore (`Dockerfile.build.dockerignore`) or an
+ allow-list.
+3. **The root `Dockerfile` and `Dockerfile.test` still pin node 20 and pnpm 9.15.4.** They are not
+ W3's, and they have the same pnpm 9 gap with `allowBuilds`. The root Dockerfile's `deps` stage
+ installs all seven packages, so pnpm 11's check would pass there. Its runtime stages'
+ `npm install -g pnpm@9.15.4` needs the same review.
+4. **Built from uncommitted edits, then committed, an image reads as stale** under the doctor's
+ commit-time rule until it is rebuilt. That happened here: the scratch image was built at
+ 16:39Z and the commit is 16:47Z.
+
+**Questions for the reviewer.**
+1. The prompt says "WARN when it is absent". Here the WARN applies only beside a corpus, and with
+ no channels it is a note. The doctor's own rule is that a clone with no corpus must not be told
+ it is broken, and the tools block grades a binary nothing needs yet the same way. Keep it, or
+ WARN everywhere?
+2. Bugs A and B above belong to `docker/build-site.sh`, which is outside W3. Should a slice fix
+ them before the parent's build-only Build all? As things stand, that Build all fails every site
+ in containers.
+
## Rollout
