commit ac54353bd75f27beda782728eaa89fb9090b6783
parent d6cf8734ba8702dbfdef81847d99ff8fb88f0525
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Wed, 30 Sep 2026 20:45:02 -0400
Merge main (b36ccebb, release 13 slice W3) into r13/lows-editor — the second join
Two conflicts, both kept whole: editor/CHANGELOG.md's [Unreleased] holds
W3's four bullets, then W1's four; plans/release-13.md's Record keeps W3's
sections as main has them and puts W1's after them, before the Rollout.
doctor.ts and doctor.test.ts merged cleanly: W3's build-image section and
its tests beside W1's L7 engine line and its test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
13 files changed, 1489 insertions(+), 45 deletions(-)
diff --git a/Dockerfile.build b/Dockerfile.build
@@ -1,13 +1,24 @@
# Build image for the docker export pipeline (Build all, whenever a container engine answers).
#
-# Bakes the repo source + installed deps so each per-site build container is
-# hermetic and reproducible. The corpus, the shared LMDB index, the .export-index
+# Bakes the repo source + installed deps so every per-site build container runs
+# the same toolchain and code. The corpus, the shared LMDB index, the .export-index
# staging, and the archive cache are bind-mounted READ-ONLY at run time — never
# baked (they're hundreds of GB and change constantly). Each container writes only
# its per-site output mount (/site). Deploy never runs here; it stays on the host.
+# The network is left ON at run time (common/publish/build.ts, runDockerBuildOne):
+# `next build` fetches each site's fonts through next/font/google.
#
-# Entry: docker/build-site.sh runs `compose:site + next build` for one SITE_ID.
-FROM node:20-bookworm-slim
+# Entry: docker/build-site.sh runs `archilyzer build site <SITE_ID> --nodata`
+# (compose + next build) for one SITE_ID.
+#
+# Node and pnpm are pinned to what the workspace runs on. pnpm 11 is not optional:
+# pnpm-workspace.yaml's `allowBuilds` and `minimumReleaseAgeExclude` are keys
+# pnpm 9 does not know, and pnpm 11 itself needs Node >= 22.13. bookworm, like
+# the root Dockerfile's build stage.
+ARG NODE_IMAGE=node:22.23.2-bookworm-slim
+FROM ${NODE_IMAGE}
+
+ARG PNPM_VERSION=11.26.0
# Archive compressors the export build shells out to. `zip` is the default format;
# `tar`/`xz`/`gzip` cover the other configurable archive formats.
@@ -15,30 +26,47 @@ RUN apt-get update \
&& apt-get install -y --no-install-recommends zip tar xz-utils gzip \
&& rm -rf /var/lib/apt/lists/*
-# Install pnpm as a plain global binary (NOT corepack): the fan-out containers run
-# fully offline (--network=none) as an arbitrary host uid with HOME=/tmp, where
-# corepack — lacking a packageManager pin — would try to fetch pnpm from the
-# registry and fail. A global install needs no network at run time.
-RUN npm install -g pnpm@9.15.4
+# Install pnpm as a plain global binary (NOT corepack). There is no packageManager
+# pin in package.json, so corepack would download a pnpm of its own choosing at
+# run time — in every container, since each runs as an arbitrary host uid with
+# HOME=/tmp and keeps nothing between runs. A global install is baked once and
+# needs nothing at run time.
+RUN npm install -g "pnpm@${PNPM_VERSION}"
WORKDIR /repo
# Install deps first for layer caching — rebuilds only when a manifest or the
-# lockfile moves. `onlyBuiltDependencies` in pnpm-workspace.yaml rebuilds the
-# native modules (lmdb, msgpackr-extract, esbuild).
+# lockfile moves. `allowBuilds` in pnpm-workspace.yaml rebuilds the native
+# modules (lmdb, msgpackr-extract, esbuild) for this image.
COPY pnpm-lock.yaml pnpm-workspace.yaml package.json ./
COPY common/package.json common/package.json
COPY export/package.json export/package.json
RUN pnpm install --frozen-lockfile
-# Bake source last so a code change only re-runs from here.
+# Only common and export (and the root) are installed — all the export build
+# needs. Before every `pnpm exec` / `pnpm run`, pnpm 11 checks that the WHOLE
+# workspace is installed and runs `pnpm install` when it is not, which fails as
+# the non-root runtime uid (EACCES on /repo). Dockerfile.build.dockerignore
+# admits no other workspace package, so today the check passes; it is off anyway,
+# because a builder that reads only the shared .dockerignore bakes all seven and
+# the first `pnpm exec` dies. The deps are frozen here. The update notice is off
+# too: every site's log would print it.
+ENV pnpm_config_verify_deps_before_run=false \
+ pnpm_config_update_notifier=false
+
+# Bake source last so a code change only re-runs from here. The context is
+# Dockerfile.build.dockerignore's allow-list — root manifests, common/, export/
+# (its public/ only the tracked .svg assets) and docker/build-site.sh: about
+# 7 MB from any checkout, and never a corpus byte.
COPY . .
# Containers run with `-u <host-uid>` (so /site outputs are host-owned, not root).
# Next writes a couple of fixed-location files into the export package dir
-# (next-env.d.ts, tsconfig.tsbuildinfo) and the entrypoint symlinks .next/out from
-# there — so that one dir must be writable by an arbitrary runtime uid. The image
-# is ephemeral and isolated (--network=none), so widening it here is harmless.
+# (next-env.d.ts, .next/) and the entrypoint replaces export/public with a link
+# to the composed /site/public — so export/ must be writable by an arbitrary
+# runtime uid. Every container is ephemeral (`docker run --rm`) and writes nothing
+# back but its /site mount, so widening it here is harmless. The layer repeats
+# export/'s few MB of source, not the image.
RUN chmod -R a+rwX /repo/export
ENTRYPOINT ["bash", "docker/build-site.sh"]
diff --git a/Dockerfile.build.dockerignore b/Dockerfile.build.dockerignore
@@ -0,0 +1,41 @@
+# The build context of Dockerfile.build ONLY. BuildKit reads <Dockerfile>.dockerignore
+# beside the Dockerfile in place of the shared .dockerignore, which the root Dockerfile
+# and Dockerfile.test keep using unchanged.
+#
+# An allow-list: everything is out, then only what one site's export build reads
+# (docker/build-site.sh → `archilyzer build site <id> --nodata` → compose + next
+# build) is let back in. The corpus, the index, the staging and the settings are
+# MOUNTED at run time (common/publish/build.ts, runDockerBuildOne) and never baked.
+#
+# What this keeps out, measured on the primary checkout: export/public's generated
+# data (subs/ alone is 1.8 GB — and a site's `next build` would publish whatever sat
+# there), .diarize/ (1.3 GB), umtool's data, editor/, homepage/, mcp/, plans/.
+*
+!package.json
+!pnpm-lock.yaml
+!pnpm-workspace.yaml
+!tsconfig.base.json
+!common
+!export
+!docker/build-site.sh
+
+# Inside common/ and export/: nothing generated, cached, local or secret. No
+# tracked file in either starts with a dot.
+common/.*
+export/.*
+**/node_modules
+**/.next
+**/out
+**/*.tsbuildinfo
+**/next-env.d.ts
+**/.env*
+**/*.pem
+export/test-*
+export/playwright-report
+export/blob-report
+
+# export/public holds generated data beside the repo's tracked assets (all .svg).
+# Only the assets are baked: build-site.sh copies them into the site's composed
+# public/, which is what `next build` publishes.
+export/public/*
+!export/public/*.svg
diff --git a/PUBLISH.md b/PUBLISH.md
@@ -677,11 +677,17 @@ serial host build+deploy (one site at a time).
|---|---|---|
| `transcripts/` (corpus + `index.mdb` + archive cache) | `/data/transcripts` | ro |
| `export/.export-index` (shared + per-site staging) | `/data/export/.export-index` | ro |
-| `export/.export-builds/<siteId>` (public/out/.next/caches) | `/site` | rw |
+| `export/.export-builds/<siteId>` (public/, out/, .compose-cache) | `/site` | rw |
| `settings.json` (build config, mounted fresh — not baked) | `/data/settings.json` | ro |
-The per-site `/site` mount is persistent, so incremental `next build` (`.next`) and
-incremental compose (`.compose-cache`) stay warm across builds.
+The per-site `/site` mount is persistent, so incremental compose (`.compose-cache`)
+stays warm across builds. `next build` runs in the container's own `.next` and starts
+fresh each time. A Turbopack production build keeps no cache between runs anyway.
+The container's `export/public` is a link to the composed `/site/public`, so `out/`
+carries this site's data and nothing baked into the image. Before handing `out/`
+back, the container checks that its `site.json` and `corpus.json` both name the site,
+and the deploy phase checks again. A bundle that names another site, or no site, is
+refused.
**Tuning.**
@@ -694,11 +700,20 @@ incremental compose (`.compose-cache`) stay warm across builds.
per-site archive materialize for a faster build with no download bundles.
**Notes.** Containers run as your host uid/gid (`-u`), so files under
-`.export-builds/` are host-owned, not root-owned. The image bakes the repo source and
-deps; a code change rebuilds it, but layer caching keeps that cheap (deps re-install
-only when the lockfile moves). `.export-builds/` is gitignored and excluded from the
-image build context. The editor mounts the host's `docker/build-site.sh` over the
-baked one, so an image older than the checkout still runs today's script.
+`.export-builds/` are host-owned, not root-owned. The image bakes the export build's
+source (common/, export/) and its deps; a code change rebuilds it, but layer caching
+keeps that cheap (deps re-install only when the lockfile moves). Its build context is
+the allow-list in `Dockerfile.build.dockerignore`, about 7 MB from any checkout. It
+never includes the corpus, generated `export/public` data or `.export-builds/`. That
+file is read by BuildKit (Docker's default builder). A builder that reads only the
+shared `.dockerignore` sends several GB from a working checkout, and podman is
+unverified here. That fallback is slower but still safe: each site's `out/` is built
+from its own composed data, and only the tracked `.svg` assets are copied in from the
+image. `archilyzer doctor` says whether the image is there and older than its
+Dockerfile.
+
+The editor mounts the host's `docker/build-site.sh` over the baked one, so an image
+older than the checkout still runs today's script.
---
diff --git a/common/bin/doctor.test.ts b/common/bin/doctor.test.ts
@@ -8,6 +8,7 @@
// wants. The last assertion of each is the one that matters most: the tree is
// byte-for-byte and mtime-for-mtime what it was — the doctor wrote nothing.
+import { execFileSync } from "node:child_process";
import { test, after } from "node:test";
import assert from "node:assert/strict";
import {
@@ -23,7 +24,14 @@ import {
import os from "node:os";
import path from "node:path";
import type { Paths } from "../lib/paths";
-import { collectDoctorReport, renderDoctorReport, type DoctorReport } from "./doctor";
+import {
+ collectDoctorReport,
+ parseEngineTime,
+ renderDoctorReport,
+ type BuildImageProbe,
+ type DoctorDeps,
+ type DoctorReport,
+} from "./doctor";
const TMP = mkdtempSync(path.join(os.tmpdir(), "doctor-"));
after(() => rmSync(TMP, { recursive: true, force: true }));
@@ -89,7 +97,14 @@ function tree(root: string): string[] {
return out.sort();
}
-async function run(c: ReturnType<typeof checkout>, env: NodeJS.ProcessEnv = {}): Promise<DoctorReport> {
+// No container engine unless a scenario hands one in: a unit test never asks docker.
+const noEngine = async (): Promise<BuildImageProbe> => ({ engine: false });
+
+async function run(
+ c: ReturnType<typeof checkout>,
+ env: NodeJS.ProcessEnv = {},
+ more: Partial<DoctorDeps> = {},
+): Promise<DoctorReport> {
return collectDoctorReport({
env: { PATH: c.bin, ...env },
paths: c.paths,
@@ -97,6 +112,8 @@ async function run(c: ReturnType<typeof checkout>, env: NodeJS.ProcessEnv = {}):
portBlock: async () => null,
portInUse: async () => false,
umtoolTools: async () => null,
+ buildImage: noEngine,
+ ...more,
});
}
@@ -246,6 +263,7 @@ test("node older than next's floor fails", async () => {
portBlock: async () => null,
portInUse: async () => false,
umtoolTools: async () => null,
+ buildImage: noEngine,
});
assert.equal(status(r, "node"), "fail");
});
@@ -259,12 +277,180 @@ test("umtool's table and the port block are reported, never failed", async () =>
portBlock: async () => ({ label: "worktree #3 (offset 300)", ports: { EDITOR_PORT: "3301" } }),
portInUse: async (p) => p === 3301,
umtoolTools: async () => [{ id: "qrencode", bin: "qrencode", neededBy: ["report-video QR"], required: true }],
+ buildImage: noEngine,
});
assert.equal(status(r, "qrencode"), "warn");
assert.match(r.checks.find((x) => x.id === "EDITOR_PORT")!.detail, /^3301 in use/);
assert.equal(r.ok, true);
});
+// ── the build image ─────────────────────────────────────────────────────────
+// Build all's per-site containers run the image the settings name. The engine
+// is always handed in (`buildImage`); only the Dockerfile's date is ever read
+// for real, and then from a temp checkout.
+
+// A corpus whose tools all pass, so every warning below is the image's.
+function corpusCheckout(settings: Record<string, unknown> = {}) {
+ const c = checkout();
+ mkdirSync(path.join(c.paths.channelsDir, "chan"), { recursive: true });
+ writeFileSync(path.join(c.paths.channelsDir, "chan", "config.json"), "{}");
+ writeFileSync(c.paths.lmdbPath, "");
+ for (const b of ["yt-dlp", "ffmpeg", "ffprobe"]) fake(c.bin, b);
+ writeFileSync(c.paths.settingsFile, JSON.stringify({ workers: [], ...settings }));
+ writeFileSync(path.join(c.root, "Dockerfile.build"), "FROM scratch\n");
+ return c;
+}
+
+const imageLine = (r: DoctorReport) => r.checks.find((x) => x.id === "build-image")!;
+const JUL = new Date("2026-07-07T16:04:55Z");
+const SEP = new Date("2026-09-28T17:00:00Z");
+const NOW = new Date("2026-09-29T12:00:00Z");
+
+test("the build image: no container engine is one line saying the check was skipped", async () => {
+ const c = corpusCheckout();
+ const r = await run(c);
+ const lines = r.checks.filter((x) => x.section === "build image");
+ assert.equal(lines.length, 1, renderDoctorReport(r));
+ assert.equal(lines[0].id, "build-image");
+ assert.equal(lines[0].status, "info");
+ assert.match(lines[0].detail, /^skipped: `docker version` did not answer/);
+ assert.equal(r.ok, true, renderDoctorReport(r));
+});
+
+test("the build image: absent warns beside a corpus, with the configured tag and the one command — never a failure", async () => {
+ const c = corpusCheckout({ buildPipeline: { dockerImage: "my-build" } });
+ const asked: { bin: string; image: string }[] = [];
+ const before = tree(c.root);
+ const r = await run(c, {}, {
+ buildImage: async (q) => {
+ asked.push(q);
+ return { engine: true, image: null };
+ },
+ });
+ assert.deepEqual(asked, [{ bin: "docker", image: "my-build" }]);
+ const line = imageLine(r);
+ assert.equal(line.status, "warn", renderDoctorReport(r));
+ assert.match(line.detail, /^no image "my-build" — the next Build all builds it first/);
+ assert.ok(
+ line.detail.includes(`\nrebuild it now: cd ${c.root} && docker build -f Dockerfile.build -t my-build .`),
+ line.detail,
+ );
+ assert.equal(r.ok, true, renderDoctorReport(r));
+ assert.deepEqual(tree(c.root), before);
+
+ // No corpus, no site to build: the same words, as a note.
+ const bare = checkout();
+ const r2 = await run(bare, {}, { buildImage: async () => ({ engine: true, image: null }) });
+ assert.equal(status(r2, "build-image"), "info");
+ assert.equal(r2.ok, true);
+});
+
+test("the build image: older than its Dockerfile's last change warns; newer is ok, with its age", async () => {
+ const c = corpusCheckout();
+ const changed = async () => ({ at: SEP, source: "commit" as const });
+ let r = await run(c, {}, {
+ buildImage: async () => ({ engine: true, image: { created: JUL, sizeBytes: 7_720_000_000 } }),
+ dockerfileChanged: changed,
+ now: NOW,
+ });
+ let line = imageLine(r);
+ assert.equal(line.status, "warn", renderDoctorReport(r));
+ assert.match(
+ line.detail,
+ /^"yt-dlp-transcript-browser-build" built 2026-07-07 16:04 \(84 days ago\), 7\.72 GB — before Dockerfile\.build's last change \(2026-09-28 17:00, its last commit\)/,
+ );
+ assert.match(line.detail, /\nrebuild it now: cd \S+ && docker build -f Dockerfile\.build -t yt-dlp-transcript-browser-build \.$/);
+ assert.equal(r.ok, true);
+
+ r = await run(c, {}, {
+ buildImage: async () => ({
+ engine: true,
+ image: { created: new Date("2026-09-28T18:00:00Z"), sizeBytes: 1_673_611_166 },
+ }),
+ dockerfileChanged: changed,
+ now: NOW,
+ });
+ line = imageLine(r);
+ assert.equal(line.status, "ok", renderDoctorReport(r));
+ assert.match(line.detail, /built 2026-09-28 18:00 \(18 hours ago\), 1\.67 GB, after Dockerfile\.build's last change/);
+ assert.doesNotMatch(line.detail, /rebuild/);
+});
+
+test("the build image: DOCKER_BIN names the engine it asks and the command it prints", async () => {
+ const c = corpusCheckout();
+ const asked: string[] = [];
+ const r = await run(c, { DOCKER_BIN: "podman" }, {
+ buildImage: async (q) => {
+ asked.push(q.bin);
+ return { engine: true, image: null };
+ },
+ });
+ assert.deepEqual(asked, ["podman"]);
+ assert.match(imageLine(r).detail, /&& podman build -f Dockerfile\.build -t yt-dlp-transcript-browser-build \.$/);
+});
+
+test("the build image: a Dockerfile the settings name that is not there warns", async () => {
+ const c = corpusCheckout({ buildPipeline: { dockerfile: "docker/nope.Dockerfile" } });
+ const r = await run(c, {}, {
+ buildImage: async () => ({ engine: true, image: { created: JUL, sizeBytes: null } }),
+ now: NOW,
+ });
+ assert.equal(status(r, "dockerfile"), "warn", renderDoctorReport(r));
+ assert.match(r.checks.find((x) => x.id === "dockerfile")!.detail, /^docker\/nope\.Dockerfile is not at /);
+ assert.equal(status(r, "build-image"), "ok"); // here, and nothing to date it against
+});
+
+test("the build image's Dockerfile dates from its last commit, or its mtime once edited — and git writes nothing", async () => {
+ const git = execFileSync("sh", ["-c", "command -v git"], { encoding: "utf8" }).trim();
+ const c = corpusCheckout();
+ symlinkSync(git, path.join(c.bin, "git"));
+ const at = "2026-08-01T00:00:00Z";
+ const g = (...args: string[]) =>
+ execFileSync(git, args, {
+ cwd: c.root,
+ env: {
+ ...process.env,
+ GIT_AUTHOR_DATE: at,
+ GIT_COMMITTER_DATE: at,
+ GIT_AUTHOR_NAME: "t",
+ GIT_AUTHOR_EMAIL: "t@example.invalid",
+ GIT_COMMITTER_NAME: "t",
+ GIT_COMMITTER_EMAIL: "t@example.invalid",
+ },
+ });
+ g("init", "-q");
+ // No background maintenance after the commit: a detached `git maintenance
+ // run --auto` takes .git/objects/maintenance.lock while the doctor runs, and
+ // the tree comparison below would blame the doctor for git's own write.
+ g("config", "maintenance.auto", "false");
+ g("config", "gc.auto", "0");
+ g("add", "Dockerfile.build");
+ g("commit", "-q", "-m", "the build image");
+ const image = async (): Promise<BuildImageProbe> => ({
+ engine: true,
+ image: { created: new Date("2026-09-01T00:00:00Z"), sizeBytes: null },
+ });
+ const before = tree(c.root);
+ let r = await run(c, {}, { buildImage: image, now: NOW });
+ assert.equal(imageLine(r).status, "ok", renderDoctorReport(r));
+ assert.match(imageLine(r).detail, /Dockerfile\.build's last change \(2026-08-01 00:00, its last commit\)/);
+ // .git included: `git status` refreshed no index (GIT_OPTIONAL_LOCKS=0).
+ assert.deepEqual(tree(c.root), before);
+
+ // An uncommitted edit is newer than any commit: its mtime (now) decides.
+ writeFileSync(path.join(c.root, "Dockerfile.build"), "FROM scratch\nRUN true\n");
+ r = await run(c, {}, { buildImage: image, now: NOW });
+ assert.equal(imageLine(r).status, "warn", renderDoctorReport(r));
+ assert.match(imageLine(r).detail, /its mtime: uncommitted or no checkout\)/);
+});
+
+test("an engine's image creation time parses in docker's and podman's spelling", () => {
+ assert.equal(parseEngineTime("2026-07-07T12:04:55.302907312-04:00")?.toISOString(), "2026-07-07T16:04:55.302Z");
+ assert.equal(parseEngineTime("2026-07-07 12:04:55.302907312 -0400 EDT")?.toISOString(), "2026-07-07T16:04:55.302Z");
+ assert.equal(parseEngineTime("2026-09-28T16:35:39Z")?.toISOString(), "2026-09-28T16:35:39.000Z");
+ assert.equal(parseEngineTime("<no value>"), null);
+});
+
test("the source publish block: the tools, the operator files by count and mode (never their contents), the last publish — never a failure", async () => {
const c = checkout();
const deps = (tools: Awaited<ReturnType<NonNullable<Parameters<typeof collectDoctorReport>[0]["sourceTools"]>>>) => ({
@@ -274,6 +460,7 @@ test("the source publish block: the tools, the operator files by count and mode
portBlock: async () => null,
portInUse: async () => false,
umtoolTools: async () => null,
+ buildImage: noEngine,
sourceTools: async () => tools,
});
// A clone that never publishes: notes, not warnings.
@@ -361,6 +548,7 @@ test("social icons: a refused stored icon is named by file and label, not its ma
portInUse: async () => false,
portBlock: async () => null,
umtoolTools: async () => null,
+ buildImage: noEngine,
sourceTools: async () => ({ filterRepo: null, gitleaks: null, stagit: null }),
});
const line = report.checks.find((x) => x.section === "social icons")!;
diff --git a/common/bin/doctor.ts b/common/bin/doctor.ts
@@ -5,7 +5,11 @@
// this reads and whose probe it shares — lib/toolProbe.mjs) and the port block
// (scripts/worktree.mjs over lib/ports.mjs).
//
-// STRICTLY READ-ONLY. It stats, reads and runs version flags. It never opens
+// It also asks the container engine whether Build all's site build image is
+// there and older than its Dockerfile (common/publish/build.ts).
+//
+// STRICTLY READ-ONLY. It stats, reads and runs version flags, plus the engine's
+// `image inspect` and a lock-free `git status` / `git log`. It never opens
// LMDB (the index is stat'd, not opened), never mkdirs, never writes settings,
// and never binds a port (a port is "in use" when a TCP connect succeeds). The
// one process-state change is a chdir around umtool's table, which resolves a
@@ -70,6 +74,14 @@ export type DoctorDeps = {
portBlock?: () => Promise<{ label: string; ports: Record<string, string> } | null>;
// umtool's report-pipeline table, or null when there is no umtool here.
umtoolTools?: () => Promise<ToolSpec[] | null>;
+ // The container engine and the site build image. Default: `<bin> version`,
+ // then `<bin> image inspect <image>` — both read-only.
+ buildImage?: (q: { bin: string; image: string }) => Promise<BuildImageProbe>;
+ // When the build image's Dockerfile last changed, or null when it is not
+ // there. Default: its last commit in a checkout, its mtime when it has
+ // uncommitted changes or there is no checkout.
+ dockerfileChanged?: (file: string) => Promise<DockerfileChange | null>;
+ now?: Date;
// The source publish's tools. Default: probeSourceTools(env).
sourceTools?: () => Promise<SourceTools>;
};
@@ -82,6 +94,12 @@ export type SourceTools = {
stagit: string | null;
};
+export type BuildImageProbe =
+ | { engine: false }
+ | { engine: true; image: { created: Date | null; sizeBytes: number | null } | null };
+
+export type DockerfileChange = { at: Date; source: "commit" | "mtime" };
+
const MIN_NODE = [20, 9, 0] as const; // next 16's engines field
export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorReport> {
@@ -304,6 +322,49 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
}
}
+ // ── build image ──────────────────────────────────────────────────────────
+ // Build all builds every site in a container whenever the engine answers
+ // `version`, from the image the settings name (common/publish/build.ts). Never
+ // a failure — Build all (re)builds the image itself before its fan-out — and,
+ // like a tool nothing needs yet, only a note while there is no corpus to build.
+ const B = "build image";
+ {
+ const { buildImageArgs, dockerBin } = await import("../publish/build");
+ const { defaultBuildPipeline } = await import("../lib/settingsSchema");
+ const pipeline = settings?.buildPipeline ?? defaultBuildPipeline();
+ const bin = dockerBin(env);
+ const probed = await (deps.buildImage ?? ((q) => probeBuildImage(q, env)))({ bin, image: pipeline.dockerImage });
+ if (!probed.engine) {
+ add(B, "build-image", "info",
+ `skipped: \`${bin} version\` did not answer (DOCKER_BIN names the engine) — Build all builds sites one at a time on the host`);
+ } else {
+ const grade: CheckStatus = hasCorpus ? "warn" : "info";
+ const now = deps.now ?? new Date();
+ const file = path.resolve(root, pipeline.dockerfile);
+ const changed = await (deps.dockerfileChanged ?? ((f) => dockerfileChangedAt(f, root, env)))(file);
+ const rebuild = `rebuild it now: cd ${shellQuote(root)} && ${[bin, ...buildImageArgs(pipeline)].map(shellQuote).join(" ")}`;
+ const img = probed.image;
+ if (!changed) {
+ add(B, "dockerfile", grade, `${pipeline.dockerfile} is not at ${file} — Build all's image build will fail (Settings → Build pipeline names it)`);
+ }
+ if (!img) {
+ add(B, "build-image", grade,
+ `no image "${pipeline.dockerImage}" — the next Build all builds it first, installing every dependency\n${rebuild}`);
+ } else if (!img.created) {
+ add(B, "build-image", grade, `"${pipeline.dockerImage}" is here but its creation time did not parse\n${rebuild}`);
+ } else {
+ const built = `"${pipeline.dockerImage}" built ${stamp(img.created)} (${ago(img.created, now)})${img.sizeBytes ? `, ${gigabytes(img.sizeBytes)}` : ""}`;
+ const last = changed ? `${pipeline.dockerfile}'s last change (${stamp(changed.at)}, ${changed.source === "commit" ? "its last commit" : "its mtime: uncommitted or no checkout"})` : "";
+ if (changed && img.created < changed.at) {
+ add(B, "build-image", grade,
+ `${built} — before ${last}, so the next Build all rebuilds it from the changed step on\n${rebuild}`);
+ } else {
+ add(B, "build-image", "ok", changed ? `${built}, after ${last}` : built);
+ }
+ }
+ }
+ }
+
// ── source publish ───────────────────────────────────────────────────────
// `archilyzer build homepage` runs it (common/publish/source.ts). Never a
// failure: a checkout that never publishes the homepage is not broken. A
@@ -552,6 +613,91 @@ async function worktreePortBlock(
}
}
+// The engine answers `version` (what Build all asks before choosing containers),
+// then the image is inspected. Both only read the daemon's state.
+async function probeBuildImage(
+ q: { bin: string; image: string },
+ env: NodeJS.ProcessEnv,
+): Promise<BuildImageProbe> {
+ try {
+ await execFileP(q.bin, ["version"], { env, timeout: 15_000 });
+ } catch {
+ return { engine: false };
+ }
+ try {
+ const { stdout } = await execFileP(q.bin, ["image", "inspect", "--format", "{{.Created}}|{{.Size}}", q.image], {
+ env,
+ timeout: 15_000,
+ });
+ const [created = "", size = ""] = stdout.trim().split("|");
+ const bytes = Number.parseInt(size, 10);
+ return { engine: true, image: { created: parseEngineTime(created), sizeBytes: Number.isFinite(bytes) ? bytes : null } };
+ } catch {
+ return { engine: true, image: null };
+ }
+}
+
+// An image's creation time as an engine prints it: docker's RFC 3339 with
+// nanoseconds (2026-07-07T12:04:55.302907312-04:00), or podman's Go default
+// (2026-07-07 12:04:55.302907312 -0400 EDT). V8 happens to read both, through
+// its implementation-defined fallback; normalized here to the ISO form the
+// spec defines, so the answer does not rest on that.
+export function parseEngineTime(s: string): Date | null {
+ const m = /^(\d{4}-\d{2}-\d{2})[T ](\d{2}:\d{2}:\d{2})(?:\.(\d+))?\s*(Z|[+-]\d{2}:?\d{2})?/.exec(s.trim());
+ if (!m) return null;
+ const frac = (m[3] ?? "").slice(0, 3).padEnd(3, "0");
+ const tz = (m[4] ?? "Z").replace(/^([+-]\d{2})(\d{2})$/, "$1:$2");
+ const d = new Date(`${m[1]}T${m[2]}.${frac}${tz}`);
+ return Number.isNaN(d.getTime()) ? null : d;
+}
+
+// When the Dockerfile last changed. In a checkout that is its last commit — a
+// clone's mtimes are the clone's — unless the file has uncommitted changes, when
+// the file on disk is newer than any commit and its mtime is the answer. No
+// checkout (or no git): the mtime. GIT_OPTIONAL_LOCKS=0 keeps `git status` from
+// refreshing the index, which would be a write.
+async function dockerfileChangedAt(
+ file: string,
+ root: string,
+ env: NodeJS.ProcessEnv,
+): Promise<DockerfileChange | null> {
+ const st = statOrNull(file);
+ if (!st) return null;
+ const byMtime: DockerfileChange = { at: st.mtime, source: "mtime" };
+ const opts = { cwd: root, env: { ...env, GIT_OPTIONAL_LOCKS: "0" }, timeout: 10_000 };
+ const rel = path.relative(root, file);
+ try {
+ const { stdout: dirty } = await execFileP("git", ["status", "--porcelain", "--", rel], opts);
+ if (dirty.trim() !== "") return byMtime;
+ const { stdout } = await execFileP("git", ["log", "-1", "--format=%ct", "--", rel], opts);
+ const secs = Number.parseInt(stdout.trim(), 10);
+ return Number.isFinite(secs) ? { at: new Date(secs * 1000), source: "commit" } : byMtime;
+ } catch {
+ return byMtime;
+ }
+}
+
+function shellQuote(s: string): string {
+ return /^[\w@%+=:,./-]+$/.test(s) ? s : `'${s.replace(/'/g, `'\\''`)}'`;
+}
+
+// The same minute-precision UTC stamp the index line uses.
+function stamp(d: Date): string {
+ return d.toISOString().slice(0, 16).replace("T", " ");
+}
+
+function ago(then: Date, now: Date): string {
+ const mins = Math.max(0, Math.round((now.getTime() - then.getTime()) / 60_000));
+ if (mins < 60) return `${mins} minute${mins === 1 ? "" : "s"} ago`;
+ const hours = Math.round(mins / 60);
+ if (hours < 48) return `${hours} hour${hours === 1 ? "" : "s"} ago`;
+ return `${Math.round(hours / 24)} days ago`;
+}
+
+function gigabytes(bytes: number): string {
+ return bytes >= 1e9 ? `${(bytes / 1e9).toFixed(2)} GB` : `${Math.round(bytes / 1e6)} MB`;
+}
+
// What `source publish` would run, by version flags only: `git filter-repo
// --version` answering 0 is an installed filter-repo; otherwise pipx's own
// version (never `pipx run`, which would download). gitleaks likewise. stagit
diff --git a/common/lib/builtExport.test.ts b/common/lib/builtExport.test.ts
@@ -4,6 +4,7 @@ import { mkdtempSync, mkdirSync, rmSync, utimesSync, writeFileSync } from "node:
import { tmpdir } from "node:os";
import path from "node:path";
import {
+ builtBundleProblem,
builtHomepageAt,
builtHomepageProblem,
builtHubProblem,
@@ -57,6 +58,8 @@ test("builtSiteIdIn answers null for every flavour of 'nothing was built here'",
test("builtSiteProblem passes a build of the site being deployed", () => {
const { dir, cleanup } = tempOut(JSON.stringify({ siteId: "anilyzer" }));
+ // compose writes corpus.json beside site.json, from the same descriptor.
+ writeFileSync(path.join(dir, "corpus.json"), JSON.stringify({ site: { id: "anilyzer" } }));
try {
assert.equal(builtSiteProblem(dir, "anilyzer"), null);
// Trimmed, so a padded id from a form is not itself the complaint.
@@ -160,3 +163,88 @@ test("builtHomepageAt is index.html's mtime, and null with no build", () => {
empty.cleanup();
}
});
+
+// A per-site container build publishes its own out/ (docker/build-site.sh). It
+// once published the public/ baked into the image instead of the one compose
+// had just written — so out/ carried no data, or another site's. Both identity
+// files must be there and both must name the site.
+function bundle(files: { site?: unknown; corpus?: unknown }): { dir: string; cleanup: () => void } {
+ const t = tempOut(files.site === undefined ? undefined : JSON.stringify(files.site));
+ if (files.corpus !== undefined) {
+ writeFileSync(path.join(t.dir, "corpus.json"), JSON.stringify(files.corpus));
+ }
+ return t;
+}
+
+test("builtBundleProblem passes a bundle whose site.json and corpus.json both name the site", () => {
+ const t = bundle({ site: { siteId: "anilyzer" }, corpus: { spec: 4, kind: "site", site: { id: "anilyzer" } } });
+ try {
+ assert.equal(builtBundleProblem(t.dir, "anilyzer"), null);
+ assert.equal(builtBundleProblem(t.dir, " anilyzer "), null);
+ } finally {
+ t.cleanup();
+ }
+});
+
+test("builtBundleProblem refuses another site's bundle, by either file", () => {
+ const other = bundle({ site: { siteId: "jeralyzer" }, corpus: { site: { id: "jeralyzer" } } });
+ const torn = bundle({ site: { siteId: "anilyzer" }, corpus: { site: { id: "jeralyzer" } } });
+ try {
+ assert.match(builtBundleProblem(other.dir, "anilyzer")!, /holds a build of "jeralyzer", not "anilyzer" \(site\.json\)$/);
+ assert.match(builtBundleProblem(torn.dir, "anilyzer")!, /describes "jeralyzer", not "anilyzer" \(corpus\.json\)$/);
+ } finally {
+ other.cleanup();
+ torn.cleanup();
+ }
+});
+
+test("builtBundleProblem refuses a bundle missing either identity file, naming the directory", () => {
+ const none = bundle({});
+ const noCorpus = bundle({ site: { siteId: "anilyzer" } });
+ const unnamed = bundle({ site: { siteId: "anilyzer" }, corpus: { site: {} } });
+ try {
+ assert.equal(builtBundleProblem(none.dir, "anilyzer"), `${none.dir} has no site.json naming a site — it is not a build of "anilyzer"`);
+ assert.match(builtBundleProblem(noCorpus.dir, "anilyzer")!, /has no corpus\.json naming a site/);
+ assert.match(builtBundleProblem(unnamed.dir, "anilyzer")!, /has no corpus\.json naming a site/);
+ } finally {
+ none.cleanup();
+ noCorpus.cleanup();
+ unnamed.cleanup();
+ }
+});
+
+// deploySite and the editor's deploy action answer with builtSiteProblem before
+// any job; the deploy itself refuses with builtBundleProblem just before
+// wrangler. The two must never disagree about a bundle: a legitimately built
+// site refused only at the last step, or a bad one let through to it.
+test("builtSiteProblem refuses exactly what builtBundleProblem refuses", () => {
+ const cases: { name: string; site?: unknown; corpus?: unknown }[] = [
+ { name: "a good build", site: { siteId: "anilyzer" }, corpus: { site: { id: "anilyzer" } } },
+ { name: "nothing built" },
+ { name: "another site", site: { siteId: "jeralyzer" }, corpus: { site: { id: "jeralyzer" } } },
+ { name: "no corpus.json", site: { siteId: "anilyzer" } },
+ { name: "a torn build", site: { siteId: "anilyzer" }, corpus: { site: { id: "jeralyzer" } } },
+ { name: "a corpus.json that names no site", site: { siteId: "anilyzer" }, corpus: { spec: 4 } },
+ { name: "a hub build", corpus: { kind: "hub" } },
+ ];
+ for (const c of cases) {
+ const t = bundle(c);
+ try {
+ const site = builtSiteProblem(t.dir, "anilyzer");
+ const deploy = builtBundleProblem(t.dir, "anilyzer");
+ assert.equal(site === null, deploy === null, `${c.name}: builtSiteProblem ${site} / builtBundleProblem ${deploy}`);
+ if (site) assert.match(site, /^export\/out holds .* — build anilyzer first$/, c.name);
+ } finally {
+ t.cleanup();
+ }
+ }
+ const torn = bundle({ site: { siteId: "anilyzer" }, corpus: { site: { id: "jeralyzer" } } });
+ try {
+ assert.equal(
+ builtSiteProblem(torn.dir, "anilyzer"),
+ 'export/out holds an incomplete build of "anilyzer" (its corpus.json does not name it) — build anilyzer first',
+ );
+ } finally {
+ torn.cleanup();
+ }
+});
diff --git a/common/lib/builtExport.ts b/common/lib/builtExport.ts
@@ -46,6 +46,11 @@ export function builtSiteIdIn(outDir: string): string | null {
* Refuses rather than rebuilding, because a deploy-only action is the operator
* saying "ship what is there"; quietly building something else would be a much
* larger surprise than a refusal naming the fix.
+ *
+ * It refuses exactly what builtBundleProblem refuses — the check the deploy
+ * itself makes before wrangler (publish/build.ts, runDeployIntoLog) — in the
+ * operator's words, so an action's fast answer and the deploy's last word
+ * never disagree about a bundle.
*/
export function builtSiteProblem(outDir: string, siteId: string): string | null {
const built = builtSiteIdIn(outDir);
@@ -56,9 +61,54 @@ export function builtSiteProblem(outDir: string, siteId: string): string | null
if (built !== asked) {
return `export/out holds a build of "${built}", not "${asked}" — build ${asked} first`;
}
+ if (corpusSiteIdIn(outDir) !== asked) {
+ return `export/out holds an incomplete build of "${asked}" (its corpus.json does not name it) — build ${asked} first`;
+ }
+ return null;
+}
+
+/**
+ * Why the bundle in a per-site container's `outDir` is not `siteId`'s own, as
+ * one sentence naming the directory — or null when it is.
+ *
+ * Stricter than builtSiteProblem: both identity files compose writes must be
+ * there and both must name the site — `site.json`'s `siteId` and
+ * `corpus.json`'s `site.id`. A container build once published the public/
+ * baked into its image instead of the one compose had just written, so its
+ * out/ carried whatever the image's build context held: no data at all, or a
+ * DIFFERENT site's. docker/build-site.sh refuses to hand back such an out/,
+ * and the container deploy phase refuses to ship one (publish/build.ts).
+ */
+export function builtBundleProblem(outDir: string, siteId: string): string | null {
+ const asked = siteId.trim();
+ const built = builtSiteIdIn(outDir);
+ if (built === null) {
+ return `${outDir} has no site.json naming a site — it is not a build of "${asked}"`;
+ }
+ if (built !== asked) {
+ return `${outDir} holds a build of "${built}", not "${asked}" (site.json)`;
+ }
+ const described = corpusSiteIdIn(outDir);
+ if (described === null) {
+ return `${outDir} has no corpus.json naming a site — it is not a complete build of "${asked}"`;
+ }
+ if (described !== asked) {
+ return `${outDir} describes "${described}", not "${asked}" (corpus.json)`;
+ }
return null;
}
+// corpus.json's `site.id`, or null when there is no readable one.
+function corpusSiteIdIn(outDir: string): string | null {
+ try {
+ const parsed: unknown = JSON.parse(readFileSync(path.join(outDir, "corpus.json"), "utf8"));
+ const site = (parsed as { site?: { id?: unknown } } | null)?.site;
+ return typeof site?.id === "string" && site.id.trim() ? site.id.trim() : null;
+ } catch {
+ return null;
+ }
+}
+
/**
* Why `outDir` may not be deployed as the HUB, as one sentence — or null when
* it holds a hub build.
diff --git a/common/publish/build.test.ts b/common/publish/build.test.ts
@@ -1,9 +1,19 @@
import { test } from "node:test";
import assert from "node:assert/strict";
-import { chmodSync, existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import {
+ chmodSync,
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ readFileSync,
+ rmSync,
+ writeFileSync,
+} from "node:fs";
import os from "node:os";
import path from "node:path";
+import { runChildIntoLog } from "../jobs/runChild";
import type { Paths } from "../lib/paths";
+import type { Site } from "../lib/site";
import {
HOMEPAGE_PAGES_PROJECT,
buildHomepage,
@@ -16,6 +26,8 @@ import {
dockerSiteOutDir,
dockerSiteStagingDir,
resolveOutDir,
+ runDeployIntoLog,
+ runDockerDeployAllPhase,
} from "./build";
// Run with:
@@ -275,3 +287,109 @@ test("deployHomepage refuses a bad preview branch before it looks for a build",
/homepage\/out holds no build/,
);
});
+
+// Phase C ships each per-site out/ a container wrote. It must be THAT site's
+// bundle: a container once published the public/ baked into its image, so an
+// out/ could carry another site's data. The check comes first. These sites have
+// NO Cloudflare project, so were it ever removed they would be "skipped", and
+// this test could never reach a real upload or deploy.
+test("runDockerDeployAllPhase refuses a per-site out/ that is not the site's own bundle", async () => {
+ const root = mkdtempSync(path.join(os.tmpdir(), "deploy-all-"));
+ try {
+ const outFor = (id: string) => path.join(root, id, "out");
+ mkdirSync(outFor("anilyzer"), { recursive: true });
+ writeFileSync(path.join(outFor("anilyzer"), "site.json"), JSON.stringify({ siteId: "jeralyzer" }));
+ writeFileSync(path.join(outFor("anilyzer"), "corpus.json"), JSON.stringify({ site: { id: "jeralyzer" } }));
+ mkdirSync(outFor("bonnellyzer"), { recursive: true }); // built, but no bundle in it
+ const log: string[] = [];
+ const sites = [
+ { siteId: "anilyzer" },
+ { siteId: "bonnellyzer" },
+ ] as Site[];
+ const outcomes = await runDockerDeployAllPhase(
+ (l) => log.push(l),
+ new AbortController().signal,
+ sites,
+ new Set(["anilyzer", "bonnellyzer"]),
+ { ...paths, exportBuildsDir: root } as Paths,
+ outFor,
+ );
+ assert.deepEqual(outcomes.map((o) => [o.siteId, o.status]), [["anilyzer", "failed"], ["bonnellyzer", "failed"]]);
+ assert.match(outcomes[0].reason!, /holds a build of "jeralyzer", not "anilyzer"/);
+ assert.match(outcomes[1].reason!, /has no site\.json naming a site/);
+ assert.ok(log.some((l) => l.startsWith("[anilyzer] deploy REFUSED — ")), log.join("\n"));
+ assert.ok(!log.some((l) => l.startsWith("=== Deploy")), "nothing reached the deploy");
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+function writeBundle(dir: string, siteId: string | null, corpusId: string | null): void {
+ mkdirSync(dir, { recursive: true });
+ if (siteId !== null) writeFileSync(path.join(dir, "site.json"), JSON.stringify({ siteId }));
+ if (corpusId !== null) writeFileSync(path.join(dir, "corpus.json"), JSON.stringify({ site: { id: corpusId } }));
+ writeFileSync(path.join(dir, "index.html"), "<!doctype html>");
+}
+
+// runDeployIntoLog is the one door every SITE deploy goes through — the
+// container Phase C, deploySite, the Publish tab's Build & deploy, the host
+// Build & deploy all — and the build queue can rewrite export/out between a
+// site's build and its deploy. So it refuses a bundle that is not the site's
+// own right before wrangler. PATH here holds only a fake `pnpm` that records
+// its argv: with the check or without it, no run of this test can reach a real
+// wrangler, and the fake is proved to answer before anything is deployed.
+test("runDeployIntoLog refuses a bundle that is not the site's own before wrangler, and ships the site's own", async () => {
+ const root = mkdtempSync(path.join(os.tmpdir(), "deploy-guard-"));
+ const bin = path.join(root, "bin");
+ const argvFile = path.join(root, "pnpm-argv");
+ mkdirSync(bin);
+ writeFileSync(
+ path.join(bin, "pnpm"),
+ `#!/bin/sh\nprintf '%s\\n' "$@" >> '${argvFile}'\necho "Take a peek over at https://abc123.w3c-never-real.pages.dev"\n`,
+ );
+ chmodSync(path.join(bin, "pnpm"), 0o755);
+ const savedPath = process.env.PATH;
+ process.env.PATH = bin;
+ const signal = new AbortController().signal;
+ const site = { siteId: "anilyzer", cloudflareProject: "w3c-never-real" } as Site;
+ const testPaths = { ...paths, exportDir: root } as Paths;
+ try {
+ // The fake answers the same spawn the deploy makes, or nothing below runs.
+ await runChildIntoLog(() => {}, signal, { command: "pnpm", args: ["--fake?"], cwd: root, env: { ...process.env } });
+ assert.equal(readFileSync(argvFile, "utf8"), "--fake?\n");
+ rmSync(argvFile);
+
+ const refusals: [string, string | null, string | null, RegExp][] = [
+ ["another site's bundle", "jeralyzer", "jeralyzer", /holds a build of "jeralyzer", not "anilyzer" \(site\.json\)/],
+ ["the hub's bundle (no site.json)", null, null, /has no site\.json naming a site/],
+ ["a torn bundle", "anilyzer", "jeralyzer", /describes "jeralyzer", not "anilyzer" \(corpus\.json\)/],
+ ["a bundle with no corpus.json", "anilyzer", null, /has no corpus\.json naming a site/],
+ ];
+ for (const [name, siteId, corpusId, why] of refusals) {
+ const out = path.join(root, name.replace(/\W+/g, "-"), "out");
+ writeBundle(out, siteId, corpusId);
+ const log: string[] = [];
+ const code = await runDeployIntoLog((l) => log.push(l), signal, site, out, testPaths);
+ assert.equal(code, 1, name);
+ assert.equal(log.length, 1, `${name}: ${log.join("")}`);
+ assert.match(log[0], /^\[deploy\] REFUSED — /, name);
+ assert.match(log[0], why, name);
+ assert.match(log[0], /Nothing was sent to Cloudflare Pages; build anilyzer again, then deploy\.\n$/, name);
+ assert.equal(existsSync(argvFile), false, `${name}: pnpm was spawned`);
+ }
+
+ // A legitimately built site is never refused: the same argv as ever.
+ const good = path.join(root, "good", "out");
+ writeBundle(good, "anilyzer", "anilyzer");
+ const log: string[] = [];
+ assert.equal(await runDeployIntoLog((l) => log.push(l), signal, site, good, testPaths), 0, log.join("\n"));
+ assert.equal(
+ readFileSync(argvFile, "utf8"),
+ ["dlx", "wrangler", "pages", "deploy", good, "--project-name", "w3c-never-real", ""].join("\n"),
+ );
+ assert.ok(log.includes("[deployed] https://abc123.w3c-never-real.pages.dev\n"), log.join("\n"));
+ } finally {
+ process.env.PATH = savedPath;
+ rmSync(root, { recursive: true, force: true });
+ }
+});
diff --git a/common/publish/build.ts b/common/publish/build.ts
@@ -14,7 +14,7 @@ import { createReadStream, existsSync } from "node:fs";
import { S3Client, HeadObjectCommand } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import { runChildIntoLog } from "../jobs/runChild";
-import { builtHubProblem, builtSiteProblem } from "../lib/builtExport";
+import { builtBundleProblem, builtHubProblem, builtSiteProblem } from "../lib/builtExport";
import { getHomepageConfig } from "../lib/homepage";
import {
deploymentUrlIn,
@@ -332,6 +332,22 @@ export async function runDeployIntoLog(
paths: Paths,
opts?: { previewBranch?: string },
): Promise<number> {
+ // The last word before wrangler: the bundle must be this site's own
+ // (site.json AND corpus.json name it — builtBundleProblem). Every SITE deploy
+ // comes through here — the container Phase C, deploySite, the Publish tab's
+ // Build & deploy and the host Build & deploy all — and the deploy queue runs
+ // beside the build queue, so between a site's build and this line another
+ // job (a build of another site, the hub) can rewrite export/out. Nothing runs
+ // between this check and the spawn. The hub and the homepage deploy through
+ // runPagesDeployIntoLog and never come here.
+ const bundleProblem = builtBundleProblem(outDir, site.siteId);
+ if (bundleProblem) {
+ onLog(
+ `[deploy] REFUSED — ${bundleProblem}. Nothing was sent to Cloudflare Pages; ` +
+ `build ${site.siteId} again, then deploy.\n`,
+ );
+ return 1;
+ }
const project = site.cloudflareProject as string;
const previewBranch = opts?.previewBranch?.trim() || undefined;
@@ -388,8 +404,19 @@ export async function runDeployIntoLog(
// The container engine binary. Defaults to `docker`; podman is a CLI drop-in
// (and rootless podman yields host-owned outputs without needing `-u`).
-function dockerBin(): string {
- return process.env.DOCKER_BIN?.trim() || "docker";
+// `archilyzer doctor` asks the same engine, from the environment it was given.
+export function dockerBin(env: NodeJS.ProcessEnv = process.env): string {
+ return env.DOCKER_BIN?.trim() || "docker";
+}
+
+// The build image's `docker build` argv, run from the monorepo root: the one
+// spelling of it. ensureBuildImage runs it; `archilyzer doctor` prints it as the
+// command that rebuilds an absent or stale image.
+export function buildImageArgs(pipeline: {
+ dockerImage: string;
+ dockerfile: string;
+}): string[] {
+ return ["build", "-f", pipeline.dockerfile, "-t", pipeline.dockerImage, "."];
}
export type SiteBuildOutcome = { siteId: string; code: number };
@@ -434,17 +461,23 @@ async function runHostScript(
});
}
-// Build (or reuse cached layers of) the per-site build image.
+// Build (or reuse cached layers of) the per-site build image. Runs before every
+// Phase B, so a fan-out never meets an image older than the checkout: a code
+// change re-runs only `COPY . .` onward, but a Dockerfile or lockfile change
+// re-installs every dependency first, inside this job. `archilyzer doctor`
+// warns ahead of that when the image is absent or older than the Dockerfile.
async function ensureBuildImage(
onLog: (line: string) => void,
signal: AbortSignal,
paths: Paths,
): Promise<number> {
- const { dockerImage, dockerfile } = getSettings().buildPipeline;
- onLog(`[docker] building image "${dockerImage}" from ${dockerfile} (cached layers reused)`);
+ const pipeline = getSettings().buildPipeline;
+ onLog(
+ `[docker] building image "${pipeline.dockerImage}" from ${pipeline.dockerfile} (cached layers reused)`,
+ );
return runChildIntoLog(onLog, signal, {
command: dockerBin(),
- args: ["build", "-f", dockerfile, "-t", dockerImage, "."],
+ args: buildImageArgs(pipeline),
cwd: paths.monorepoRoot,
});
}
@@ -552,7 +585,8 @@ export async function runDockerBuildAllPhase(
// Phase C: deploy each built site SERIALLY on the host, after the build barrier.
// Partial-failure tolerant — a site that fails to upload/deploy is recorded and
// the loop continues. Sites that failed to build, or have no Cloudflare project,
-// are skipped. `outDirFor` resolves each site's built bundle (docker: per-site;
+// are skipped; a bundle that is not the site's own is refused (builtBundleProblem).
+// `outDirFor` resolves each site's built bundle (docker: per-site;
// basic fallback: export/out).
export async function runDockerDeployAllPhase(
onLog: (line: string) => void,
@@ -570,6 +604,16 @@ export async function runDockerDeployAllPhase(
outcomes.push({ siteId: site.siteId, status: "skipped", reason: "build failed" });
continue;
}
+ // The bundle must be this site's own before anything else is asked of it —
+ // the check build-site.sh makes before it hands the bundle back, made again
+ // over whatever the per-site dir holds now. First, so that nothing past it
+ // (the R2 upload, the Pages deploy) is ever reached with another site's data.
+ const bundleProblem = builtBundleProblem(outDirFor(site.siteId), site.siteId);
+ if (bundleProblem) {
+ onLog(`[${site.siteId}] deploy REFUSED — ${bundleProblem}`);
+ outcomes.push({ siteId: site.siteId, status: "failed", reason: bundleProblem });
+ continue;
+ }
if (!site.cloudflareProject) {
onLog(`[${site.siteId}] deploy skipped — no Cloudflare project configured`);
outcomes.push({
diff --git a/common/publish/buildImage.test.ts b/common/publish/buildImage.test.ts
@@ -0,0 +1,90 @@
+// The container build's contract with the repo: what Dockerfile.build's
+// image bakes into export/public and how docker/build-site.sh puts it in front
+// of each site's `next build`. No docker here — the invariant is read from git,
+// and the shell function is read out of build-site.sh and run with bash.
+//
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { execFileSync } from "node:child_process";
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
+
+// The site build image bakes only export/public/*.svg (Dockerfile.build.
+// dockerignore) and docker/build-site.sh copies only those into each site's
+// public/. A tracked asset of any other kind, or in a subdirectory, would be in
+// every host build and silently missing from every container build.
+test("every tracked file in export/public is a top-level .svg, as the container build assumes", (t) => {
+ let listed: string;
+ try {
+ listed = execFileSync("git", ["-C", REPO, "ls-files", "export/public"], { encoding: "utf8" });
+ } catch {
+ t.skip("not a git checkout");
+ return;
+ }
+ const odd = listed.split("\n").filter((f) => f && !/^export\/public\/[^/]+\.svg$/.test(f));
+ assert.deepEqual(
+ odd,
+ [],
+ `export/public tracks ${odd.join(", ")} — not a top-level .svg. The container build ships only ` +
+ `those: admit the new asset in Dockerfile.build.dockerignore and copy it in docker/build-site.sh ` +
+ `(sync_public_assets), or every container-built site goes without it.`,
+ );
+});
+
+// /site/public persists between container builds, so the tracked assets are
+// synced into it every run: a changed one must ship, a removed one must stop
+// shipping (an icon dropped from the repo — the Ko-fi mark was one — must not
+// live on in a site), and nothing compose wrote may be touched. The function is
+// read out of build-site.sh itself and run with bash over temp dirs.
+test("build-site.sh's asset sync ships a changed svg, drops a removed one, and touches nothing compose wrote", () => {
+ const script = readFileSync(path.join(REPO, "docker", "build-site.sh"), "utf8");
+ const fn = /^sync_public_assets\(\) \{[\s\S]*?^\}$/m.exec(script)?.[0];
+ assert.ok(fn, "docker/build-site.sh defines sync_public_assets()");
+ const root = mkdtempSync(path.join(tmpdir(), "asset-sync-"));
+ const src = path.join(root, "image-public");
+ const dest = path.join(root, "site-public");
+ const list = path.join(root, ".tracked-public-assets");
+ const sync = () =>
+ execFileSync("bash", ["-euo", "pipefail", "-c", `${fn}\nsync_public_assets "$1" "$2" "$3"`, "sync", src, dest, list]);
+ const read = (p: string) => readFileSync(path.join(dest, p), "utf8");
+ try {
+ mkdirSync(src);
+ mkdirSync(path.join(dest, "transcripts"), { recursive: true });
+ writeFileSync(path.join(src, "globe.svg"), "<svg>v1</svg>");
+ writeFileSync(path.join(src, "kofi-symbol.svg"), "<svg>kofi</svg>");
+ writeFileSync(path.join(src, "stale.json"), "{}"); // not an asset: never copied
+ // What compose wrote (or any file not copied from the image): never touched.
+ writeFileSync(path.join(dest, "corpus.json"), '{"site":{"id":"anilyzer"}}');
+ writeFileSync(path.join(dest, "transcripts", "page-0000.json"), "[]");
+ writeFileSync(path.join(dest, "composed.svg"), "<svg>compose</svg>");
+
+ sync();
+ assert.equal(read("globe.svg"), "<svg>v1</svg>");
+ assert.equal(read("kofi-symbol.svg"), "<svg>kofi</svg>");
+ assert.equal(existsSync(path.join(dest, "stale.json")), false);
+ assert.equal(readFileSync(list, "utf8"), "globe.svg\nkofi-symbol.svg\n");
+
+ // The repo changes one asset and drops another. And a list naming a file
+ // compose writes (a hand-edited or damaged list) must not cost the site it:
+ // only an .svg is ever removed.
+ writeFileSync(path.join(src, "globe.svg"), "<svg>v2</svg>");
+ rmSync(path.join(src, "kofi-symbol.svg"));
+ writeFileSync(list, "globe.svg\nkofi-symbol.svg\ncorpus.json\n");
+ sync();
+ assert.equal(read("globe.svg"), "<svg>v2</svg>", "a changed asset is shipped");
+ assert.equal(existsSync(path.join(dest, "kofi-symbol.svg")), false, "a removed asset stops shipping");
+ assert.equal(readFileSync(list, "utf8"), "globe.svg\n");
+ assert.equal(read("corpus.json"), '{"site":{"id":"anilyzer"}}');
+ assert.equal(read("transcripts/page-0000.json"), "[]");
+ assert.equal(read("composed.svg"), "<svg>compose</svg>", "only a name the last run copied is removed");
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
diff --git a/docker/build-site.sh b/docker/build-site.sh
@@ -7,7 +7,7 @@
# mounted READ-ONLY. This container only composes ITS site's public/ and runs
# `next build`, writing everything into the per-site mount at /site.
#
-# Expected run-time mounts (see runDockerBuildAllPhase):
+# Expected run-time mounts (see runDockerBuildOne in common/publish/build.ts):
# <transcriptsDir> -> /data/transcripts (ro)
# <export/.export-index> -> /data/export/.export-index (ro)
# <exportBuildsDir>/<siteId> -> /site (rw)
@@ -24,14 +24,56 @@ export EXPORT_PUBLIC_DIR=/site/public
# never re-generate — see compose-site.ts / archiveTranscripts.ts.
export ARCHIVES_READONLY=1
-# Persist Next's build cache (.next) into the per-site mount so incremental
-# next builds stay warm across runs. A symlink is safe for .next because Next
-# reuses — never wholesale-replaces — its cache dir. The export dir was made
-# writable in the image so this symlink (and Next's next-env.d.ts) can be created
-# as an arbitrary runtime uid.
-mkdir -p /site/.next /site/out /site/public
+mkdir -p /site/out /site/public
+
+# .next stays INSIDE the container, and starts empty. It used to be a symlink to
+# /site/.next, but turbopack's server runtime imports next's own externals (the
+# icon routes' next/dist/compiled/@vercel/og) from each chunk's REAL path —
+# under /site, where there is no node_modules — so `next build` died
+# prerendering /favicon.ico. Nothing measurable is lost: a Turbopack production
+# build keeps no filesystem cache unless experimental.
+# turbopackFileSystemCacheForBuild is set (export/next.config.ts does not), and
+# carrying .next/cache (the TypeScript .tsbuildinfo, the fetch cache) between
+# runs was measured to save nothing (plans/release-13.md, W3b).
rm -rf export/.next
-ln -s /site/.next export/.next
+
+# `next build` publishes export/public into out/, so export/public must BE the
+# public/ compose writes for this site (EXPORT_PUBLIC_DIR=/site/public). It used
+# to be the copy baked into the image, which shipped as the site's data: none at
+# all, or whatever site the image's build context had composed last. The export
+# dir was made writable in the image so this link (and Next's next-env.d.ts) can
+# be created as an arbitrary runtime uid.
+#
+# The repo's tracked assets in public/ — all .svg (Dockerfile.build.dockerignore;
+# common/publish/buildImage.test.ts fails if one is not) — are synced into the
+# site's public/ first, on EVERY run, because /site/public persists between runs:
+# - each is copied over whatever copy is there, so a changed asset is shipped;
+# - one the repo no longer has is removed, so a dropped asset stops shipping.
+# Only a name the previous run copied is ever removed — they are listed in
+# <list>, outside public/ so the list never ships — so nothing compose wrote
+# is touched.
+# Nothing but the .svg assets is copied: anything else in an image's public/
+# would be data, and not this site's.
+sync_public_assets() { # <image public dir> <site public dir> <list file>
+ local src="$1" dest="$2" list="$3" f name
+ if [ -f "$list" ]; then
+ while IFS= read -r name; do
+ case "$name" in "" | */* | . | ..) continue ;; esac
+ case "$name" in *.svg) ;; *) continue ;; esac # only an asset is ever removed
+ if [ ! -e "$src/$name" ] && [ -f "$dest/$name" ]; then rm -f "$dest/$name"; fi
+ done < "$list"
+ fi
+ : > "$list.new"
+ for f in "$src"/*.svg; do
+ [ -f "$f" ] || continue
+ cp -a "$f" "$dest/"
+ printf '%s\n' "${f##*/}" >> "$list.new"
+ done
+ mv "$list.new" "$list"
+}
+sync_public_assets export/public /site/public /site/.tracked-public-assets
+rm -rf export/public
+ln -s /site/public export/public
echo "[build-site] building site '${SITE_ID}'"
# `build site --nodata` = compose:site + next build, WITHOUT the data phase
@@ -40,9 +82,25 @@ echo "[build-site] building site '${SITE_ID}'"
# compose through the environment.
pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts build site "${SITE_ID}" --nodata
+# Refuse to hand back a bundle that is not this site's own: its site.json and
+# corpus.json must both name SITE_ID (common/lib/builtExport.ts,
+# builtBundleProblem — the check the deploy phase makes again on the host). On a
+# refusal /site/out keeps whatever it held, and the build fails.
+BUILT_OUT_DIR="$PWD/export/out" pnpm --filter yt-dlp-transcript-common exec tsx -e '
+ import("./lib/builtExport.ts").then(({ builtBundleProblem }) => {
+ const problem = builtBundleProblem(process.env.BUILT_OUT_DIR, process.env.SITE_ID);
+ if (problem) {
+ console.error(`[build-site] REFUSED: ${problem} — /site/out is left as it was`);
+ process.exit(1);
+ }
+ console.log(`[build-site] out/ is the bundle of ${process.env.SITE_ID} (site.json, corpus.json)`);
+ });
+'
+
# next build writes export/out as a FRESH real dir (it removes+recreates out, so
# a symlink there wouldn't survive) — publish it into the per-site mount. Reached
-# only when the build succeeded (set -e aborts otherwise).
+# only when the build succeeded and the bundle is the site's (set -e aborts
+# otherwise).
echo "[build-site] publishing out/ -> /site/out"
rm -rf /site/out
mkdir -p /site/out
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,6 +1,10 @@
# Changelog
## [Unreleased]
+- **`archilyzer doctor` checks the image Build all builds sites in.** When a container engine answers, a new **build image** section says whether the image named under **Settings → Build pipeline** is there, when it was built and how big it is. It warns when the image is missing, or older than the last change to its Dockerfile, and prints the one command that rebuilds it. Build all still builds or refreshes the image itself before it builds any site; the warning tells you ahead of time that the next Build all will spend that time. With no container engine the check is skipped in one line, and with no corpus it is only a note. It never fails the doctor.
+- **The site build image runs Node 22 and pnpm 11**, the versions the rest of the workspace runs on, instead of Node 20 and pnpm 9, which did not read the workspace's install rules. The next Build all rebuilds the image from its first step, reinstalling every dependency, before it builds any site.
+- **Build all sites works in containers again.** Every site's container build had been failing while it prerendered `/favicon.ico`. Each site now builds from the data composed for it, never from files baked into the build image. A bundle whose `site.json` and `corpus.json` do not both name its site is refused before it is handed back or deployed. The image carries no corpus data, and its build context is about 7 MB from any checkout.
+- **A site deploy refuses a bundle that is not the site's own, and says why.** A site's **Build & deploy**, `pnpm ops build-deploy`, and Build & deploy all on a machine without containers used to ship whatever `export/out` held when the deploy began. If another site's build or the hub's had replaced it meanwhile, that is what shipped. Every site deploy now checks, just before handing the bundle to Cloudflare Pages, that its `site.json` and `corpus.json` both name the site. If they do not, it stops with `[deploy] REFUSED —` and what it found, and nothing is sent. Deploying a build that has a `site.json` but no matching `corpus.json` is refused before the job starts, as an incomplete build.
- **A Retry on the Diagnostics stage keeps its log when it empties its bucket.** Retrying the **Missing metadata.info.json**, **Archived ID with no directory** or **Skipped: live or upcoming** card, or the **Needs auth** or **Error** availability card, made the card vanish as soon as its bucket emptied, and the retry's log went with it. The card now stays until you leave the page, with its button greyed out at **Retry (0)**. A reload drops it, as before. The Download stage's cards have worked this way since 0.10.0.
- **The "built <when>" line in the Homepage section of `/sites` updates after a build.** When a **Build homepage** or **Build & deploy homepage** lane ends, the page re-renders, so the line says what **Deploy homepage** would ship now. It used to keep saying what `homepage/out` held when the page loaded. This happens whether the build succeeds, fails or is cancelled, because a failed build may already have changed `homepage/out`.
- **`/jobs` names the hub's and the homepage's jobs.** They show as **Build hub**, **Deploy hub**, **Build & deploy hub**, **Build homepage**, **Deploy homepage** and **Build & deploy homepage**, not as `build-hub`, `build-homepage` and so on.
diff --git a/plans/release-13.md b/plans/release-13.md
@@ -30,7 +30,7 @@ join, and a code conflict goes back to its slice.
|---|---|---|---|
| W1 | `r13/lows-editor` | Diagnostics cards keep their retry log when a retry empties the bucket; the `transcript-source.spec.ts:76` flake; `/sites` "built <when>" refreshes when a homepage build ends; `/jobs` labels for the four hub/homepage kinds; a queued-cancel writes its final sidecar status; doctor's worker engine binary from `paths` (L7); an `editor/package.json` `test` script | `editor/app/channels/[slug]/components/stages/**`, `editor/app/sites/components/{JobLane,HomepageBuildButtons}.tsx`, `common/jobs/{jobKinds,registry}.ts`, `common/lib/streamCommand.ts`, `common/bin/doctor.ts` (L7 only), the two specs |
| W2 | `r13/lows-export` | `ChartView` reaches `--chart-6` through `seriesColor(i)`; an export unit `test` script (named in the rules and CONTRIBUTING); O5's wording lows; the 2origin stage guard; `no-data.spec` skips on `workers > 1`; the mcp docs move to `archilyzer mcp` | `common/components/charts/ChartView.tsx`, `export/package.json`, `umtool/report-to-video/{svg-faces.mjs,README.md}` (comments), `export/playwright.2origin.config.ts`, `homepage/e2e/no-data.spec.ts`, `mcp/README.md`, `AGENTS.md`, `README.md`, `CONTRIBUTING.md`, `plans/tools/implementer-rules.md` |
-| W3 | `r13/build-image` | `Dockerfile.build` on Node 22 and pnpm 11; the stale `--network=none` comments; `archilyzer doctor` checks the build image's presence and age | `Dockerfile.build`, `common/publish/build.ts` (comments), `common/bin/doctor.ts` (the image check) |
+| W3 | `r13/build-image` | `Dockerfile.build` on Node 22 and pnpm 11; the stale `--network=none` comments; `archilyzer doctor` checks the build image's presence and age | `Dockerfile.build`, `common/publish/build.ts` (comments), `common/bin/doctor.ts` (the image check); from W3b `docker/build-site.sh`, `Dockerfile.build.dockerignore` |
| P0–P3 | `r13/phase-5` | one-core Phase 5: the corrected plan doc (P0, to the operator first), then slices 1–3 stacked | per `plans/one-core-phase-5.md` |
**Order:** W1 ‖ W2 ‖ W3, merged W1 → W2 → W3; then the parent rebuilds the build image and runs
@@ -40,6 +40,580 @@ Then one integration gate on `main` (`r13/integration`) and the runbook
## Record
+### Slice W3, as shipped — the build image refreshed (2026-09-28)
+
+Branch `r13/build-image` off `main` `bf6904e8`, worktree `/home/user/Projects/r13-build-image`, one
+Opus implementer. `git merge main` first fast-forwarded to `441bdbb2` (this file). Three items:
+`Dockerfile.build` on the workspace's Node and pnpm, its stale `--network=none` comments, and an
+`archilyzer doctor` check of the image. The proof is a scratch-tag image and a fixture Build all,
+not e2e. **It found that a container Build all fails every site on `main` as well** — see "Found
+and left" 1, before the parent's build-only Build all. Fixed in this slice by "Follow-up W3b"
+below.
+
+**`Dockerfile.build`.**
+- `ARG NODE_IMAGE=node:22.23.2-bookworm-slim` and `ARG PNPM_VERSION=11.26.0`.
+ - Node is the host's `node --version`; nothing else pins it (no `.nvmrc`, no `engines`, no
+ `packageManager`). It is bookworm like the root Dockerfile's build stage.
+ - pnpm is the host's `pnpm --version`, the one the worktree installed with. pnpm 11 needs
+ Node >= 22.13.
+ - `ensureBuildImage` passes no build args, so these defaults are what Build all builds.
+- **pnpm 11 needed one more line, and the proof found it:**
+ `ENV pnpm_config_verify_deps_before_run=false pnpm_config_update_notifier=false`.
+ - Before every `pnpm exec` / `pnpm run`, pnpm 11 checks that the whole workspace is installed,
+ and runs `pnpm install` when it is not.
+ - The image installs root, common and export. After `COPY . .` the workspace has 8 projects, so
+ the first `pnpm --filter … exec` in `build-site.sh` ran `pnpm install`. As the host uid that
+ died with `EACCES: permission denied, open '/repo/_tmp_…'` (`w3-image-versions.log`).
+ - pnpm 11 reads `pnpm_config_*` from the environment, not `npm_config_*` (both were tried). The
+ update notice is off because every site's log would otherwise print it.
+- **Comments:**
+ - The header no longer calls the containers "hermetic". It says the network is on
+ (`runDockerBuildOne`, for `next/font/google`) and names the entry command as
+ `archilyzer build site <id> --nodata`.
+ - The corepack reason is now "no `packageManager` pin, so corepack would download a pnpm of its
+ own choosing in every container". It used to say "offline".
+ - The chmod comment rests on `--rm` and "writes nothing back but /site". It used to rest on
+ `--network=none`.
+- The contract with `build.ts` is unchanged: `WORKDIR /repo`, the entrypoint, the mounts, `-u`,
+ and the chmod of `/repo/export`.
+
+**`common/publish/build.ts`** (no behaviour change).
+- `dockerBin(env = process.env)` is now exported. It was private and read only `process.env`.
+- `buildImageArgs(pipeline)` is new, and `ensureBuildImage` now runs it. It is the one spelling of
+ the image's `docker build` argv, which the doctor prints.
+- The comments at `:437-456`: `ensureBuildImage`'s comment now says it runs before every Phase B,
+ what a Dockerfile or lockfile change costs there, and that the doctor warns about it ahead of
+ time. `runDockerBuildOne`'s network comment was already right and is unchanged.
+
+**`archilyzer doctor`: a "build image" section** between umtool's and the ports. It is a block of
+its own: W1 changes the worker-engine line in "tools", and release 12 adds a source block; neither
+is adjacent.
+- **The probe.** `<DOCKER_BIN or docker> version` must exit 0, which is what `dockerAvailable`
+ asks. Then `<bin> image inspect --format '{{.Created}}|{{.Size}}' <tag>`.
+- **The tag and the Dockerfile** come from the effective settings (`settingsFromFile`, or the
+ defaults when the file is absent). That is the same `getSettings().buildPipeline` that `build.ts`
+ reads, so `yt-dlp-transcript-browser-build` is not spelled a second time.
+- **The Dockerfile's last change.**
+ - In a checkout it is the file's last commit (`git log -1 --format=%ct`), unless
+ `git status --porcelain` shows uncommitted edits; then it is the file's mtime.
+ - With no checkout, or no git, it is the mtime.
+ - `GIT_OPTIONAL_LOCKS=0` stops `git status` from refreshing `.git/index`, so the doctor stays
+ read-only. Without it, the test's tree comparison fails because `.git/index`'s mtime moves.
+- **Grading:**
+ - With no engine there is one `--` line, "skipped: `docker version` did not answer …".
+ - With an engine, an absent image is a WARN, and so is an image created before the Dockerfile's
+ last change. Both print the command. Anything else is `ok`, with the date, age and size.
+ - A Dockerfile the settings name that is not there gets its own `dockerfile` WARN.
+ - **Never a FAIL.** The WARNs apply only beside a corpus: with no channels the same lines are
+ notes, as for a tool nothing needs yet (Question 1).
+- **The command** is `rebuild it now: cd <root> && docker build -f Dockerfile.build -t <tag> .`,
+ built from `buildImageArgs` and shell-quoted. `DOCKER_BIN` changes both the engine asked and the
+ command printed.
+- `parseEngineTime` reads docker's RFC 3339 with nanoseconds and podman's Go default format.
+- On this machine, run from the worktree (no corpus there, so a note):
+ ```
+ build image
+ -- build-image "yt-dlp-transcript-browser-build" built 2026-07-07 16:04 (83 days ago), 7.72 GB — before Dockerfile.build's last change (2026-09-28 16:47, its last commit), so the next Build all rebuilds it from the changed step on
+ rebuild it now: cd /home/user/Projects/r13-build-image && docker build -f Dockerfile.build -t yt-dlp-transcript-browser-build .
+ ```
+ With the fixture corpus and the scratch tag, the same line is a `WARN`.
+
+| sha | what |
+|---|---|
+| `bb3d0f6a` | `docker:` `Dockerfile.build` on node:22.23.2-bookworm-slim + pnpm 11.26.0 (build args), the pnpm 11 `verify_deps_before_run` / `update_notifier` env, the network and corepack comments |
+| `40dfc008` | `common:` the doctor's build-image block + 7 tests; `build.ts` exports `dockerBin(env)` and `buildImageArgs`, and `ensureBuildImage`'s comment |
+| _this_ | `plans:` this record; two `[Unreleased]` bullets in `editor/CHANGELOG.md` |
+
+**Gates**, all from the worktree root:
+- **tsc** (`pnpm -r --no-bail --workspace-concurrency=1 exec tsc --noEmit`) was clean before both
+ commits: `w3-tsc-1.log` (168 s), then `w3-tsc-2.log` (50 s) on the final tree. The only change
+ between the two runs was one comment.
+- **common: 2,121/2,121** (`w3-common-test.log`, 132 s). The 7 new tests are all in
+ `bin/doctor.test.ts`, so `main` has 2,114; the prompt said 2,112. `test:scripts` does not cover
+ `common/bin`, so it was not run.
+- No generated doc, no editor or export build, and no e2e: nothing under `editor/app` or `export/`
+ changed, and the proof below replaces e2e.
+- **The new tests bite.** `w3-bite.log` ran the new `doctor.test.ts` against `main`'s `doctor.ts`,
+ with a one-line `parseEngineTime` shim so the file loads: **8 passed, 7 failed**, and the 7
+ failures are exactly the new tests. Without the shim the whole file fails to import.
+ Separately, with `GIT_OPTIONAL_LOCKS: "0"` removed, the git test fails on `.git/index`'s mtime.
+
+**The proof, instead of e2e.** Every log is under `$T`. The scratch tags were `r13-build-image-test`,
+`-old` and `-probe`, all removed afterwards. `yt-dlp-transcript-browser-build` was never tagged,
+built or replaced.
+1. **The image.** `docker build -f Dockerfile.build -t r13-build-image-test .` from the worktree
+ root took **53 s** on the first build (base pulled, no cached layers). The final
+ `--no-cache --pull` build took **52 s**. Size **1.67 GB** (1,673,627,497 B), from a 26.7 MB
+ context (`w3-docker-build*.log`).
+2. **Inside it** (`w3-image-versions*.log`):
+ - node **v22.23.2**, pnpm **11.26.0**.
+ - As the host uid with `HOME=/tmp`, which is how `runDockerBuildOne` runs it:
+ `pnpm --filter export exec next --version` gives Next.js v16.2.3, and
+ `pnpm --filter yt-dlp-transcript-common exec tsx --version` gives tsx v4.21.0. `lmdb`'s
+ native module opens, writes and reads.
+ - `pnpm ls --depth 0` over export and common lists 53 packages in 2 projects. This was run as
+ root: as another uid, pnpm 11's `ls` opens the store index under `/root` and fails EACCES.
+ `ls` is not on the build path.
+3. **Build all's own code over a fixture** (`w3-fixture-build.log`, 40 s).
+ - The run: `pnpm archilyzer build all` from the worktree, with `TRANSCRIPTS_DIR`,
+ `SETTINGS_FILE` and `EXPORT_{PUBLIC,INDEX,BUILDS}_DIR` pointed at `$T/w3-fixture`.
+ - The fixture: editor e2e's `one-youtube-channel-with-data` channel and one site, `w3site`.
+ Settings: `buildPipeline.dockerImage: "r13-build-image-test"`, `maxParallelBuilds: 1`.
+ - Phase A on the host passed: index, stats, templates and the archive warm.
+ - `ensureBuildImage` rebuilt the scratch tag from cached layers.
+ - In the Phase B container, `build-site.sh` composed into `/site/public` (with the read-only
+ archive cache materialized), and `next build` compiled and type-checked.
+ - It then **failed prerendering `/favicon.ico`**:
+ `Failed to load external module next/dist/compiled/@vercel/og/index.node.js: … Cannot find
+ package 'next' imported from /site/.next/server/chunks/[turbopack]_runtime.js`.
+4. **The failure predates W3.** `main`'s `Dockerfile.build` (node:20, pnpm 9.15.4) was built as
+ `r13-build-image-old` through the same Build all and failed identically
+ (`w3-fixture-build-old.log`, 136 s).
+5. **Two probes of the proposed fix** replaced the entrypoint with a copy from `$T` over the
+ `docker run` argv `runDockerBuildOne` builds. Nothing was committed under `docker/`.
+ - **Probe 1** kept `.next` in the container instead of linking it to `/site/.next`. The run
+ used a derived image with `export/public`'s dangling links deleted, which is what a clean
+ clone has.
+ - It exits 0 with 149 files, but `/site/out` has no `corpus.json`, `_headers`, `llms.txt`,
+ `robots.txt`, `site.json`, `summaries/`, `transcripts/`, `archives/` or `stats/`. All of
+ these are only in `/site/public` (`w3-probe1-run.log`).
+ - **Probe 2** made the same `.next` change and, in addition, ran
+ `cp -an export/public/. /site/public/ && rm -rf export/public && ln -s /site/public export/public`.
+ - It exits 0 with **167 files**, in 51 s. `out/` holds `w3site`'s `corpus.json`,
+ `transcripts/test-youtube/{manifest,page-0000}.json`, `summaries/` and `archives/`, all
+ host-owned (`w3-probe-run.log`).
+6. **What only the parent's post-merge Build all can prove:**
+ - the image built from the primary's context: its size, and the time to transfer that context;
+ - every real site at `maxParallelBuilds`, with the real fonts;
+ - the deploy phase.
+ As `main` stands, that Build all fails every site at `/favicon.ico` on the container path
+ (bug A below), with or without W3.
+
+**Found and left.**
+1. **`docker/build-site.sh` has two bugs, and a container Build all cannot ship a site until they
+ are fixed.** `docker/**` is not W3's.
+ - **(A) `export/.next` is a symlink to `/site/.next`.**
+ - Turbopack's runtime imports next's externals from its own real path, and under `/site`
+ there is no `node_modules`. `next build` dies at `/favicon.ico`.
+ - **(B) `next build` copies `export/public` into `out/`.**
+ - `export/public` is the copy BAKED into the image, not the composed `EXPORT_PUBLIC_DIR=/site/public`.
+ - Built from a clean clone, `out/` would have none of the site's data (probe 1).
+ - Built from the primary, `out/` would carry the primary's stale export/public. That is
+ whatever site the host composed last, without `summaries/` and `transcripts/`, which
+ `.dockerignore` excludes.
+ - **Why no one saw it.** The live image dates from 2026-07-07, and `ensureBuildImage` runs
+ before every fan-out, so no container Build all has run since then.
+ - **The host path is unaffected:** single-site builds, `archilyzer build site`, and Build all
+ when no engine answers.
+ - **The proposed fix** is proven by probe 2 and is uncommitted: `$T/w3-build-site-probe.sh`.
+ Its cost is that `.next` stops persisting between runs, so incremental `next build` is lost.
+ Persisting only `.next/cache` could win that back, but it is untested.
+2. **The build context bakes the checkout's generated data, twice.**
+ - From the primary it includes:
+ - the gitignored entries of `export/public` that `.dockerignore` does not name (`subs/`
+ 1.8 GB, `posts/`, `stats/`, `digests/`, `corpus.json`, …);
+ - `.diarize/` (1.3 GB);
+ - umtool's data (1.1 GB).
+ - `/repo/export` is in the image twice, once in the `COPY` layer and once in the `chmod` layer.
+ The live image is 3.23 GB + 3.18 GB of its 7.72 GB.
+ - From a worktree, `export/public`'s entries are absolute links into the primary and are baked
+ dangling. The first probe died on `stat '/repo/export/public/_headers'`.
+ - Fix B keeps this data out of `out/`. It still sizes the image, and it invalidates the
+ `COPY` layer after every host build.
+ - `.dockerignore` is shared with the root `Dockerfile` and `Dockerfile.test`, so W3 left it.
+ The follow-up is a BuildKit per-Dockerfile ignore (`Dockerfile.build.dockerignore`) or an
+ allow-list.
+3. **The root `Dockerfile` and `Dockerfile.test` still pin node 20 and pnpm 9.15.4.** They are not
+ W3's, and they have the same pnpm 9 gap with `allowBuilds`. The root Dockerfile's `deps` stage
+ installs all seven packages, so pnpm 11's check would pass there. Its runtime stages'
+ `npm install -g pnpm@9.15.4` needs the same review.
+4. **Built from uncommitted edits, then committed, an image reads as stale** under the doctor's
+ commit-time rule until it is rebuilt. That happened here: the scratch image was built at
+ 16:39Z and the commit is 16:47Z.
+
+**Questions for the reviewer.**
+1. The prompt says "WARN when it is absent". Here the WARN applies only beside a corpus, and with
+ no channels it is a note. The doctor's own rule is that a clone with no corpus must not be told
+ it is broken, and the tools block grades a binary nothing needs yet the same way. Keep it, or
+ WARN everywhere?
+2. Bugs A and B above belong to `docker/build-site.sh`, which is outside W3. Should a slice fix
+ them before the parent's build-only Build all? As things stand, that Build all fails every site
+ in containers.
+
+#### Follow-up W3b — the container Build all works (2026-09-28)
+
+**The parent's rulings.**
+- **Q1: keep it.** The doctor warns about an absent or stale image only beside a corpus.
+- **Q2: fix bugs A and B in this slice**, before the parent's build-only Build all.
+ - **Owns, added:** `docker/build-site.sh` and a new `Dockerfile.build.dockerignore`.
+ - **Also touched:** `common/lib/builtExport.ts` + its test. It holds the one reader of a
+ bundle's identity, and the container check imports only `node:fs` from it. `PUBLISH.md`'s
+ container section, three sentences that this change made false.
+ - **Not touched:** the root `Dockerfile`, `Dockerfile.test`, the shared `.dockerignore` and the
+ rest of `docker/**`.
+ - **The runtime image does not use `build-site.sh`.** It is `Dockerfile.build`'s entrypoint and
+ what `runDockerBuildOne` mounts. The runtime image's `publish-site.sh` only names it in a
+ comment, and inside that container there is no engine, so Build all falls back to the host.
+
+**`git merge main` (`e6c5d2e3`, release 12 slice Q):** `644bd2d5`. One conflict, in
+`editor/CHANGELOG.md`'s `[Unreleased]`: both sides are kept, main's umtool bullet first, then W3's
+block. There was no code conflict, and tsc on the merge was clean (`w3b-tsc-1.log`, 60 s).
+
+**(A) `.next` stays inside the container** (`docker/build-site.sh`).
+- The symlink to `/site/.next` is gone. That symlink put turbopack's chunks at a real path under
+ `/site`, from which `next` cannot be resolved.
+- **Incremental cache: tried, measured, dropped.**
+ - The try: copy `.next/cache` into the container before the build and back out after it. It
+ holds the TypeScript check's `.tsbuildinfo` and the fetch cache, 492 KB.
+ - Two consecutive two-site runs:
+ - TypeScript: **30.3 s → 32.6 s**;
+ - compile: 18.3 s → 21.4 s;
+ - the whole run: **90 s → 88 s** (`w3b-fixture-build-{1,2}.log`).
+ - `export/next.config.ts` does not set `experimental.turbopackFileSystemCacheForBuild`, so a
+ Turbopack production build keeps no filesystem cache. The old symlinked `.next` never made a
+ container build incremental either.
+ - **The cost, plainly:** none measured. Each container's `next build` starts from an empty
+ `.next`.
+
+**(B) `export/public` is the composed `/site/public`.**
+- `build-site.sh` copies the image's tracked assets (`export/public/*.svg` only) into
+ `/site/public`, never over a file that is already there. It then replaces `export/public` with a
+ link to `/site/public` before `archilyzer build site <id> --nodata`.
+- Only `.svg` is copied, so an image that did bake data (see the proof) cannot leak it into a site
+ through the copy.
+
+**A wrong-site bundle cannot ship: two checks, both over `builtBundleProblem`.**
+- **The rule** (`common/lib/builtExport.ts`, new): `site.json`'s `siteId` and `corpus.json`'s
+ `site.id` must both be there and both name the site. The one sentence it returns names the
+ directory and the file that disagreed.
+- **In the container.** `build-site.sh` runs the check through `tsx -e` after the build and before
+ publishing. A refusal prints `[build-site] REFUSED: …`, exits 1 and leaves `/site/out` as it
+ was. A pass prints `out/ is the bundle of <id> (site.json, corpus.json)`.
+- **On the host.** `runDockerDeployAllPhase` now checks every per-site `out/` FIRST, before the
+ "no Cloudflare project" skip, the R2 upload and the Pages deploy.
+ - Before this, it shipped whatever `export/.export-builds/<id>/out` held. The host deploy checks
+ `builtSiteProblem`; this path checked nothing.
+ - A refusal is `failed` with the sentence as its reason, and the log line is
+ `[<id>] deploy REFUSED — …`.
+
+**`Dockerfile.build.dockerignore`: an allow-list.** BuildKit reads `<Dockerfile>.dockerignore` in
+place of the shared file.
+- **It admits:**
+ - `package.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml` and `tsconfig.base.json`;
+ - `common/` and `export/`;
+ - `docker/build-site.sh`.
+- **Inside those it excludes:**
+ - dot-entries, `node_modules`, `.next` and `out`;
+ - `*.tsbuildinfo`, `next-env.d.ts`, `.env*` and `*.pem`;
+ - `export/test-*`, and the playwright report dirs;
+ - `export/public/*` except `*.svg`.
+ No tracked file matches an exclusion except the five svgs, which are let back in.
+- **The export build reads nothing outside these.** The reads relative to `monorepoRoot` on the
+ build path are `export/service-worker/*` and `export/CHANGELOG.md`, and the rest is mounted.
+- **Measured:**
+ - The context is **807 files, 7.4 MB** from the primary checkout, and the same from the
+ worktree. `w3b-context.py` emulates the rules as a read-only walk, and its worktree list
+ matched the built image's `/repo` file-for-file (807 = 807, `w3b-image-files.txt`). No docker
+ build was run from the primary.
+ - The image's `/repo` is `common docker export node_modules` and the four manifests.
+ `/repo/export/public` holds only `file.svg globe.svg next.svg vercel.svg window.svg`.
+ - Under the shared `.dockerignore`, the primary sent `export/public`'s generated data (`subs/`
+ alone 1.8 GB), `.diarize/` (1.3 GB), umtool's data and the rest. The live image's
+ `COPY . .` layer is 3.23 GB, and its chmod layer 3.18 GB more.
+ - Now `COPY . .` is **7.14 MB** and the chmod layer **1.04 MB**. `COPY --chmod` was not worth a
+ portability question to podman, so the chmod stays.
+ - The image is **1.65 GB** (1,654,559,800 B; the live one is 7.72 GB). The final
+ `--no-cache --pull` build took **48 s**. Of the 1.65 GB, 1.39 GB is the pnpm install
+ (`w3b-image-final.log`).
+- **pnpm 11's check now passes on its own.** The image holds exactly the installed projects, so
+ `pnpm exec` works as the host uid even with `verify_deps_before_run=install` or `=error`. The
+ ENV stays because a builder that reads only the shared `.dockerignore` bakes all seven packages,
+ and then the first exec dies (W3's proof). `Dockerfile.build`'s comment now says so.
+
+| sha | what |
+|---|---|
+| `644bd2d5` | merge `main` `e6c5d2e3` (release 12 slice Q); `[Unreleased]` joined |
+| `698ecc16` | `common:` `builtBundleProblem` + 3 tests |
+| `d22faf21` | `docker:` `build-site.sh` (A) `.next` in the container, (B) `export/public` → `/site/public` (the `.svg` assets only), the bundle check before publishing |
+| `84988249` | `common:` `runDockerDeployAllPhase` refuses a bundle that is not the site's, first; 1 test |
+| `2c7b57be` | `docker:` `Dockerfile.build.dockerignore` (the allow-list); `Dockerfile.build`'s comments |
+| `ab75a5b8` | `docs:` `PUBLISH.md`'s container section (the per-site mount, the image's context, the refusals) |
+| `8b7c3864` | `common:` the doctor's git test turns off git's background maintenance (a W3 flake, below) |
+| _this_ | `plans:` this follow-up and the W3 table row's Owns; one `[Unreleased]` bullet |
+
+**Gates.**
+- **tsc:** clean before the code commits (`w3b-tsc-2.log`, 121 s) and before `8b7c3864`
+ (`w3b-gates-1.log`, 121 s).
+- **common: 2,125/2,125** (`w3b-gates-1.log`). That is 2,121 + 4: `builtExport` 3,
+ `build.test` 1.
+ - The first full run (`w3b-common-test.log`) was 2,124/2,125. The failure was W3's own git test:
+ git 2.55's detached `maintenance run --auto` after the test's commit took
+ `.git/objects/maintenance.lock` while the doctor ran.
+ - The fix turns maintenance off in the temp repo. After it the test passed 8/8 alone, and it
+ still fails without `GIT_OPTIONAL_LOCKS=0` (`.git/index` moves).
+- **`test:scripts`:** none of `scripts/*.test.mjs` covers `build-site.sh` or `Dockerfile.build`,
+ so it was not run.
+- **They bite** (`w3b-bite.log`), against `46d9b0cd`, before W3b:
+ - `builtExport.test.ts`, with a permissive `builtBundleProblem` shim so it loads: **9 passed,
+ 2 failed**. The two are the refusal tests; the third new test asserts a pass.
+ - `build.test.ts`: **10 passed, 1 failed**, the new deploy test. The old phase `skipped` both
+ sites for having no project. It could not deploy, because the test's sites have no Cloudflare
+ project on purpose: a real `wrangler` is never reachable from this test, even with the check
+ removed.
+
+**The proof: Build all's own code over a two-site fixture.** `$T/w3-fixture`: `w3site` over
+`test-youtube` (editor e2e's `one-youtube-channel-with-data`), and `w3other` over `tagchan`
+(`curated-tags-channel`). Settings name the scratch tag, with `maxParallelBuilds: 2`.
+1. **`pnpm archilyzer build all` passed** (`w3b-fixture-build-3.log`, 110 s, the final script):
+ - Phase A ran on the host, then `ensureBuildImage` on the scratch tag, then Phase B with both
+ sites in parallel. **`2/2 built`, exit 0.**
+ - Each `out/` has 167 files, and 0 files in either per-site dir are not host-owned.
+ - Each has `corpus.json`, `site.json`, `_headers`, `llms.txt`, `robots.txt`,
+ `summaries/manifest.json`, `archives/manifest.json`, `favicon.ico`, `index.html` and the
+ svgs.
+ - `w3site`'s `site.json` and `corpus.json` say `w3site`, and its `transcripts/` is
+ `test-youtube`. `w3other`'s say `w3other`, over `tagchan`.
+ - `builtBundleProblem` on the host is null for both (`w3b-fixture-outputs.log`).
+2. **The dangerous case, reproduced and refused** (`w3b-poisoned-runs.log`).
+ - The setup: a scratch image `FROM` the test tag, with `w3site`'s composed `public/` copied
+ into `/repo/export/public`. That is what a checkout that last composed `w3site` baked under
+ the shared ignore file. Then `w3other` was built, through `runDockerBuildOne`'s own
+ `docker run` argv.
+ - **The old `public/` handling** (the committed script minus the link block, bundle check
+ kept) produced `w3site`'s bundle for `w3other`:
+ - `[build-site] REFUSED: /repo/export/out holds a build of "w3site", not "w3other"
+ (site.json) — /site/out is left as it was`;
+ - exit 1;
+ - `/site/out` still held only its earlier `MARKER`.
+ - **The committed script** over the same image produced exit 0, with `site.json` and
+ `corpus.json` both `w3other`, `transcripts/` `tagchan` only, and 0 paths or files anywhere
+ in `out/` or `public/` mentioning `test-youtube`.
+3. **Every scratch tag has been removed:** `r13-build-image-test` and `r13-build-image-poisoned`.
+ `yt-dlp-transcript-browser-build` is still `ddb3fbce9f60`, from 2026-07-07.
+
+**What only the parent's post-merge Build all can prove:** the real sites at real parallelism,
+the real fonts and corpus sizes, and memory under `maxParallelBuilds`. The image itself is the
+same 807 files from the primary.
+
+**Found and left.**
+- **Legacy per-site dirs.** A host that ran the old container path has
+ `export/.export-builds/<id>/.next` from those runs, now unused. This machine has no
+ `.export-builds/` at all.
+- **The in-container bundle check adds a few seconds per site**: `pnpm exec tsx -e` is about 5 s
+ on the host, with `builtExport.ts` importing only `node:fs`.
+- **W3's "Found and left" 1 and 2 are fixed here.** Item 3 (the root `Dockerfile` and
+ `Dockerfile.test` on node 20 and pnpm 9.15.4) stands, and so does item 4.
+
+#### Follow-up W3c — every site deploy checks its bundle; the assets stay in sync (2026-09-28)
+
+**The review** (`w3-review.md`) is **SHIP**, with five findings:
+- **S1:** a should-fix, release-level and pre-existing. It is now this slice's.
+- **L1** and **L2:** fixed here.
+- **L3** and **N1:** recorded below as found and left.
+- **N2:** a nit, fixed in the doc.
+
+`main` was not merged again, as instructed. There was no docker build, e2e run or `next build`:
+the primary was running a live six-site build and deploy.
+
+**S1: every SITE deploy checks its bundle right before wrangler** (`common/publish/build.ts`,
+`runDeployIntoLog`).
+- **The gap.** Two host paths shipped `export/out` with no identity check between their build and
+ wrangler:
+ - `buildAndDeployAction`: the Publish tab's Build & deploy, and `pnpm ops build-deploy`;
+ - `basicBuildAndDeployAll`: Build & deploy all with no engine.
+ The deploy queue runs beside the build queue. A build of another site, or of the hub, that
+ rewrote `export/out` during the R2 upload would therefore have shipped to this site's Pages
+ project.
+- **The fix.**
+ - `runDeployIntoLog` now calls `builtBundleProblem(outDir, site.siteId)` first. Nothing runs
+ between the check and the spawn.
+ - A refusal logs `[deploy] REFUSED — <why>. Nothing was sent to Cloudflare Pages; build <id>
+ again, then deploy.` and returns 1.
+ - Every caller already turns 1 into a failed deploy:
+ - Phase C: `failed`;
+ - `deploySite` and `buildAndDeployAction`: they throw `Deploy failed (exit 1).`;
+ - `basicBuildAndDeployAll`: `deploy FAILED — exit 1`.
+ - The hub and the homepage deploy through `runPagesDeployIntoLog`, so they are unaffected.
+- **Every caller still passes a real site bundle.**
+ - Phase C passes the per-site `out/`, already checked first by W3b.
+ - `deploySite`, `buildAndDeployAction` and `basicBuildAndDeployAll` pass `export/out`, which
+ after a site build always holds both files: compose writes `corpus.json` whenever it writes
+ `site.json`, from the same descriptor.
+- **The two checks now agree.** `builtSiteProblem` is the fast answer that `deploySite` and the
+ editor's `deployAction.ts` give before any job. It now also refuses a `site.json` naming the site
+ whose `corpus.json` does not, as `export/out holds an incomplete build of "<id>" (its corpus.json
+ does not name it) — build <id> first`.
+ - So it refuses exactly what `builtBundleProblem` refuses. A new test walks seven bundle shapes
+ through both: good, nothing, another site, no corpus, torn, unnamed corpus, hub.
+ - The existing e2e regex (`ops-api.spec.ts:1022`) still holds: the fixture site is never in
+ `export/out`, so the new sentence cannot arise there.
+- **The R2 upload still runs before the check** in the two editor paths (`editor/app` is not
+ mine). The upload sends this site's own staged archives, from the per-site `.r2-staging/<id>`
+ and not from `export/out`, so it cannot carry another site's data. A refusal after it leaves
+ what a wrangler failure leaves today: R2 has the newer archives, and Pages is unchanged.
+- **The test** (`build.test.ts`) cannot reach a real wrangler.
+ - Its `PATH` holds only a fake `pnpm` that records its argv, and the test proves the fake
+ answers the same `runChildIntoLog` spawn before anything deploys.
+ - Four refusals are checked: another site, the hub's shape, torn, no corpus. Each returns 1 with
+ one log line, and the fake is never spawned.
+ - A good bundle returns 0 with the argv unchanged (`dlx wrangler pages deploy <out>
+ --project-name …`) and the `[deployed]` line.
+
+**L1: the tracked assets are synced every run** (`docker/build-site.sh`, `sync_public_assets`).
+- Each tracked `.svg` is copied over whatever copy is there, so a changed asset ships.
+- A name the previous run copied that the repo no longer has is removed, so a dropped icon stops
+ shipping. The Ko-fi mark was such an icon.
+- The copied names are listed in `/site/.tracked-public-assets`, outside `public/`, so the list
+ never ships. Only a listed name, and only as a regular file, is ever removed. So nothing compose
+ wrote, and no svg the image did not put there, is touched.
+- The sync runs before compose, so compose would win over any shared name anyway.
+- **N3 (the re-read):** the removal loop now skips any listed name that is not an `.svg`, so a
+ hand-edited or damaged list naming `corpus.json` cannot delete what compose wrote. The sync test
+ lists `corpus.json`, and without the guard it fails with ENOENT on `site-public/corpus.json`
+ (`w3d-bite.log`).
+- **Review question 1:** no case exists where a file already in `/site/public` must win over the
+ image's svg. compose writes no top-level `.svg`: its files are `_headers`, `site.json`,
+ `corpus.json`, `llms.txt`, `robots.txt`, `sitemap.xml`, `sw.js`, the JSON data files and the
+ data dirs.
+
+**L2:** a test that every file `git ls-files export/public` lists is a top-level `.svg`
+(`common/publish/buildImage.test.ts`, the image contract's own file).
+- Its failure names both `Dockerfile.build.dockerignore` and `build-site.sh`
+ (`sync_public_assets`).
+- It skips when the tree is not a git checkout (`Dockerfile.test`'s context has no `.git`).
+- L1's test lives beside it: the function is read out of `build-site.sh` and run with bash over
+ temp dirs.
+
+**N2** (`PUBLISH.md`):
+- The doctor sentence gets its own paragraph break.
+- BuildKit reads the per-Dockerfile ignore file. A builder that reads only the shared one sends
+ several GB and stays safe: `out/` is built from the composed data, and only the svgs are copied.
+- **Review question 2:** podman was not checked. There is no podman or buildah on this machine, and
+ the doc now says it is unverified.
+
+| sha | what |
+|---|---|
+| `526e9af0` | `docker:` `sync_public_assets` in `build-site.sh`; `common/publish/buildImage.test.ts` (L1 sync + L2 svg-only, 2 tests) |
+| `85929901` | `common:` `runDeployIntoLog` refuses first; `builtSiteProblem` agrees with `builtBundleProblem`; 1 + 1 tests, and the good-bundle test gains its `corpus.json` |
+| `c52e116c` | `docs:` `PUBLISH.md` (N2) |
+| _this_ | `plans:` this follow-up; one `[Unreleased]` bullet (a refused deploy names why) |
+
+**Gates.**
+- **tsc:** clean before the code commits (`w3c-tsc-2.log`, 39 s, the final tree).
+- **common:** **2,129/2,129** (`w3c-common.log`, 66 s): 2,125 + 4 (`buildImage` 2, `build.test` 1, `builtExport` 1).
+- **`test:scripts`:** not run. No script was touched.
+- **They bite** (`w3c-bite.log`), against `31a74988`, before W3c:
+ - `build.test.ts` + `builtExport.test.ts`: **22 passed, 2 failed**.
+ - The two are `runDeployIntoLog refuses…`: on "another site's bundle" the old code returned 0,
+ having spawned the fake `pnpm`.
+ - And `builtSiteProblem refuses exactly what builtBundleProblem refuses`.
+ - `buildImage.test.ts`'s sync test:
+ - against the old script it fails at "defines sync_public_assets()";
+ - with the old no-clobber copy wrapped as the function, it fails at "a changed asset is
+ shipped" (`v1` where `v2` was expected).
+ - The svg-only test: with an intent-to-add `export/public/brand.png`, it fails with its message
+ naming both files. The file was then un-staged and deleted.
+
+**Found and left.**
+- **L3: the doctor's stale rule can stick.** A fully cached `docker build` keeps the image's old
+ `Created`, so after a comment-only commit to `Dockerfile.build`, with nothing in common/export
+ changed, the WARN persists. Its "rebuild it now" command cannot clear it; only `--no-cache` or a
+ context change can. This is rare, since nearly every commit touches common/export. If it ever
+ matters, bake a `--label` with the Dockerfile's hash in `buildImageArgs` and compare that.
+- **N1: the doctor's image block loads the AWS SDK.** It does
+ `await import("../publish/build")` for `dockerBin` and `buildImageArgs`. A small
+ `publish/buildImage.ts`, re-exported by `build.ts`, would keep the doctor light.
+- **The two editor host paths upload to R2 before the check** (above). Moving a check ahead of the
+ upload there is an `editor/app` change.
+
+#### Brought to main (2026-09-30)
+
+Release 13's W slices were reviewed SHIP on 2026-09-28 and never merged. `main` then moved through
+releases 14, 15 and 16 and the 0.11.0 cut. This brings the branch to today's `main` and proves it
+again. It is a merge, not a rebase, so the reviewed commits keep their shas.
+
+**The merge:** `abefa892` merges `main` `7a77536b` (238 commits) into the branch tip `83ee15ef`.
+It was first committed as `f7a047ee`, then reworded to this round's uniform commit trailer; the
+tree is the same (`git diff f7a047ee abefa892` is empty), so the gates below, run at `f7a047ee`,
+hold for it.
+Git marked four content conflicts and no modify/delete conflict. Each was resolved by keeping both
+sides.
+
+| File | Conflict | Resolution |
+|---|---|---|
+| `common/bin/doctor.ts` | Three hunks against SG's source-publish block: the `DoctorDeps` fields, the section, and the helpers | Both kept. `DoctorDeps` has W3's `buildImage`, `dockerfileChanged` and `now`, then SG's `sourceTools`. The sections run umtool, **build image**, **source publish**, then ports: W3's block stays right after umtool's and SG's stays right before the ports. The helpers are W3's (`probeBuildImage`, `parseEngineTime`, `dockerfileChangedAt`, `shellQuote`, `stamp`, `ago`, `gigabytes`) and then SG's (`probeSourceTools`, `dirSizeText`); no name is defined twice. DT's drive-health line, in the corpus block, merged with no conflict. |
+| `common/bin/doctor.test.ts` | W3's seven tests against SG's source-publish test, the social-icons test and DT's drive-health test | All kept, in that order: 8 + 7 + 3 = **18**. main's two tests that call `collectDoctorReport` directly (source publish, social icons) now also pass `buildImage: noEngine`. Without it they would ask the machine's real container engine, and W3's rule is that a unit test never does. |
+| `common/publish/build.test.ts` | The import lines: W3 added `readFileSync`, `tmpdir` and `runChildIntoLog`; main used `os` | One import block. W3's two `tmpdir()` calls now read `os.tmpdir()`, as main's do. 10 + 2 + 2 = **14** tests. |
+| `editor/CHANGELOG.md` | The cut renamed the old `[Unreleased]` to `[0.11.0]`, and W3's four bullets sat in it | W3's four bullets moved to a fresh `## [Unreleased]` above `## [0.11.0]`, which is now exactly main's (`git diff main` on the file adds only those six lines). |
+
+**Also touched by both sides, merged by git:**
+- `common/publish/build.ts`: W3's import, `runDeployIntoLog`'s check, `dockerBin(env)`,
+ `buildImageArgs`, `ensureBuildImage` and Phase C's check. `git diff main` on the file shows only
+ W3's hunks. On the merged tree, the call sites of `runDeployIntoLog` (four),
+ `builtSiteProblem` (two) and `builtBundleProblem` (two) are the same as on the branch, so main
+ added no path that deploys a site bundle.
+- `PUBLISH.md`: W3b's container paragraphs, beside SG's source-mirror history. `git diff main`
+ shows only W3's two hunks.
+
+**What moved under W3, and why nothing yields:**
+- main's `.dockerignore` gained the homepage's published source (`homepage/public/source/`,
+ `homepage/.source-publish.json`). `Dockerfile.build.dockerignore` admits nothing under
+ `homepage/`, so it needs no change.
+- No file tracked under `common/` or `export/` matches one of the allow-list's exclusions, by
+ `git ls-files`. `export/public` still tracks only the five svgs.
+- The export build reads nothing new from outside `common/` and `export/`. Since the fork, no
+ package manifest changed except `editor/package.json` (DS's `UV_THREADPOOL_SIZE`), and neither
+ did the lockfile or `export/next.config.ts`. DX's compose changes are URL strings. The image's
+ `pnpm install` layer is therefore the same.
+- Nothing of main's was removed or rewritten. The only line of W3's that changed is the
+ `os.tmpdir()` spelling.
+
+**Gates** (logs `$T/w3-*.log`, all from the worktree root):
+- **tsc** (all workspaces): clean on the merged tree before the merge commit, 137 s
+ (`w3-tsc-1.log`).
+- **Unit:**
+
+ | Suite | Result |
+ |---|---|
+ | `buildImage`, `builtExport`, `build`, `doctor` tests | **46/46**, 12 s (2 + 12 + 14 + 18) |
+ | common | **2,396/2,396**, 146 s. That is main's 2,381 (release 16's last count) + W3's 15. |
+ | editor unit (`app/**/*.test.ts`) | **101/101**, 9 s |
+ | `test:scripts` | **194 passed, 2 skipped** of 196, 12 s. The skips are umtool's trace check (no umtool build in this worktree) and the `LIVE=1` archive check. |
+ | mcp | **271/271**, 45 s |
+
+- **The build image, built for real** (`w3-docker-build.log`, `w3-docker-build-cold.log`).
+ `docker build -f Dockerfile.build -t archilyzer-build:w3-check .` from the worktree at `f7a047ee`.
+ - **With the build cache: 15 s.** The base, apt, pnpm and `pnpm install` layers were cached from
+ W3b, since the lockfile has not moved; `COPY . .` onward ran.
+ - **`--no-cache --pull`: 69 s.** apt 6.8 s, pnpm 6.0 s, `pnpm install --frozen-lockfile` 23.6 s,
+ and the layer export 28.4 s.
+ - **Size: 1,655,428,299 B (1.66 GB)** from `docker image inspect --format '{{.Size}}'`. The
+ cached build was 1,655,410,190 B. W3b's was 1,654,559,800 B, and the live image is 7.72 GB.
+ - `COPY . .` is **7.91 MB** (W3b: 7.14 MB) and the chmod layer 1.12 MB. `/repo` holds 849 files
+ outside `node_modules`: releases 14–16 added code to `common/` and `export/`.
+ - Inside it: node **v22.23.2** and pnpm **11.26.0**. `/repo` is `common docker export
+ node_modules` and the four root manifests, and `/repo/export/public` is the five svgs.
+ - **No site build ran through it.** The tag was removed with `docker rmi`, along with the first
+ build's image, which the cold build's re-tag had left untagged.
+ `yt-dlp-transcript-browser-build` is untouched: still `ddb3fbce9f60`, from 2026-07-07.
+- **The doctor on the merged tree** (`w3-doctor.log`, run from the worktree, so no corpus). It
+ exits 0, and its sections run workspace, corpus, settings, social icons, tools, report pipeline
+ (umtool), build image, source publish, then ports. The build-image line is a note: the live
+ image predates `Dockerfile.build`'s last commit (W3b's).
+- **The editor's `next build`**, because the editor bundles `publish/build.ts` and
+ `lib/builtExport.ts` (it does not import `doctor.ts`). It ran with the primary's `transcripts/`
+ linked in (`ln -sT`) and was capped at 5 GB with no swap: **119 s, max RSS 1,648,884 KB**,
+ exit 0. None of its 81 traces names the corpus. The link was removed after the build, and
+ nothing ran through it.
+- **e2e:** W3's record names no spec, since its proof was the fixture Build all. This run covers
+ the editor's paths into W3's changed code: the deploy refusals through `builtSiteProblem`
+ (`ops-api`), the Build all controls (`deploy-page`) and the publish tab's cannot-deploy line
+ (`site-scope`). It ran detached and queued.
+
+ | Run | At | Specs | Result |
+ |---|---|---|---|
+ | 1 | `f7a047ee` | `ops-api`, `deploy-page`, `site-scope` | **40 passed**, 0 failed, 3.2 min; 7.6 min wall, of which 4.3 min waiting in the queue behind W1 |
+
### Slice W1, as shipped — the editor lows (2026-09-28)
Branch `r13/lows-editor` off `main` `bf6904e8` (`441bdbb2` merged first, fast-forward), worktree
