commit bdd81e51122910799013d6e56a026c6c7fc59ed6
parent fadc46eb57697aaf645a2f213b81ab68d910cd33
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 6 Oct 2026 09:57:56 -0400
Merge r18/integration (S1 stage core, S5 image half) into r18/deploy-hardening
Conflicts: editor/CHANGELOG.md and plans/release-18.md, both sides kept.
envVars.ts: S5's CLOUDFLARE_API_TOKEN, XDG_CONFIG_HOME and
ARCHILYZER_HOMEPAGE_OUT rows kept, S2's marked copies deleted, their readBy
(and ARCHILYZER_SITE_OUT's) name pagesDeploy.ts / deployStage.ts;
ENVIRONMENT.md regenerated. stamps.ts's local imageBuildFacts is now
lib/envVars.ts's, re-exported.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
42 files changed, 6222 insertions(+), 216 deletions(-)
diff --git a/.env.example b/.env.example
@@ -86,9 +86,72 @@
# Run `yt-dlp -U` on every boot. An image pins yt-dlp at build time and a stale
# yt-dlp is the most common reason downloads start failing. Off by default
-# because it is a network call at startup.
+# because it is a network call at startup. It updates the IMAGE's yt-dlp only:
+# with YTDLP_BIN pointing elsewhere (below) it warns and leaves that one alone.
#YTDLP_AUTO_UPDATE=1
+# ---------------------------------------------------------------------------
+# Substituting yt-dlp (your own build, no image rebuild)
+# ---------------------------------------------------------------------------
+# The image ships the release yt-dlp at /usr/local/bin/yt-dlp
+# (ARCHILYZER_IMAGE_YTDLP — baked, do not set it). To run another one, point
+# YTDLP_BIN at it. Two ways:
+#
+# a zipapp built on the host (`make yt-dlp` in a yt-dlp checkout; it runs on
+# the image's python3), copied into the config volume:
+# docker compose exec editor mkdir -p /data/config/bin
+# docker compose cp ./yt-dlp editor:/data/config/bin/yt-dlp
+# docker compose exec editor chmod 755 /data/config/bin/yt-dlp
+#YTDLP_BIN=/data/config/bin/yt-dlp
+#
+# a SOURCE checkout, run with the image's python — set the checkout's host
+# path and start with the overlay, which sets YTDLP_BIN for you:
+# docker compose -f docker-compose.yml -f docker-compose.ytdlp.yml up -d
+#YTDLP_SOURCE_HOST_DIR=/home/you/yt-dlp-patched
+#
+# Every editor boot logs `yt-dlp: <path> <version> (image|override)`.
+
+# ---------------------------------------------------------------------------
+# Publishing to Cloudflare (deploys run IN the container)
+# ---------------------------------------------------------------------------
+# The editor reads these and every publish stage it runs inherits them. Values
+# are never printed — `docker compose exec editor pnpm archilyzer doctor` says
+# only whether each is set. A container has no browser for `wrangler login`, so
+# deploying from one needs the token.
+#
+# An API token with "Cloudflare Pages: Edit" (dash.cloudflare.com → My Profile →
+# API Tokens), and your account id.
+#CLOUDFLARE_API_TOKEN=
+#CLOUDFLARE_ACCOUNT_ID=
+#
+# R2 S3 credentials — only when Settings → archive storage names a bucket for
+# oversize archive zips.
+#R2_ACCESS_KEY_ID=
+#R2_SECRET_ACCESS_KEY=
+#
+# The wrangler deploys run. Default: the one pinned in the workspace
+# (common/node_modules/.bin/wrangler); nothing is fetched at deploy time.
+#WRANGLER_BIN=
+#
+# Driving the editor from a shell or an agent (`pnpm ops`, the MCP's
+# fetch_clip): the bearer token /api/ops/* and /api/worker/* answer to. Unset,
+# both surfaces are off.
+#WORKER_TOKEN=
+
+# ---------------------------------------------------------------------------
+# The homepage's /source mirror (optional)
+# ---------------------------------------------------------------------------
+# Publishing the homepage mirrors this repo's `main`, and the image has no .git.
+# docker-compose.source.yml mounts the host's git dir read-only and sets
+# ARCHILYZER_SOURCE_REPO=/data/source.git for you. Default `./.git`; in a git
+# worktree, the primary checkout's `.git`.
+#ARCHILYZER_SOURCE_HOST_DIR=./.git
+#
+# The scrub rules and denylist it needs live in ARCHILYZER_CONFIG_DIR, which
+# docker-compose.yml sets to /data/config/archilyzer (the config volume):
+# docker compose cp source-scrub.txt editor:/data/config/archilyzer/
+# docker compose cp source-denylist.txt editor:/data/config/archilyzer/
+
# Boot the editor WITHOUT resuming the schedulers, the auto-queue runners or the
# digest/backfill sweeps. Set this the first time you point a container at a
# corpus somebody else configured: its stored policies may say "sweep", and you
@@ -107,3 +170,9 @@
# Image
# ---------------------------------------------------------------------------
#ARCHILYZER_TAG=local
+#
+# Which commit and branch the image is built from, baked in for the publish
+# stamps (the image has no .git). Usually passed from the shell instead:
+# ARCHILYZER_COMMIT=$(git rev-parse HEAD) ARCHILYZER_BRANCH=$(git branch --show-current) docker compose build
+#ARCHILYZER_COMMIT=
+#ARCHILYZER_BRANCH=
diff --git a/.gitignore b/.gitignore
@@ -57,6 +57,9 @@ yarn-error.log*
# them, leaving generated data showing up as untracked in every worktree.
/export/public/subs
/export/public/posts
+# export/out is a SYMLINK to the bundle built last (release 18,
+# publish/build.ts pointExportOutAt) — `**/out/` matches only a directory.
+/export/out
/export/public/digests
/export/public/summaries
/export/public/transcripts
diff --git a/Dockerfile b/Dockerfile
@@ -39,10 +39,18 @@
# bookworm is 2.36, trixie 2.41, ubuntu 24.04 2.39 — so building on bookworm and
# running on any of them is safe, and moving NODE_IMAGE to trixie would silently
# break the CUDA target. If you bump one of these, check that direction first.
-ARG NODE_IMAGE=node:20-bookworm-slim
+#
+# NODE'S MAJOR IS THE SECOND RULE, and it must be the same everywhere: the
+# native modules are compiled against the build stage's Node ABI, so every
+# runtime (NODE_IMAGE, RUNTIME_IMAGE here and in docker-compose.vulkan.yml,
+# NODE_MAJOR in runtime-cuda) carries the same major. 22, not 20: the wrangler
+# pinned in common/package.json refuses to start on anything older (its
+# engines floor), and every deploy runs it from this image.
+# common/publish/buildImage.test.ts holds all four to one major and that floor.
+ARG NODE_IMAGE=node:22-bookworm-slim
# The runtime base, overridable per target: docker-compose.vulkan.yml builds with
# trixie because the Vulkan stack needs it (see the parakeet stage below).
-ARG RUNTIME_IMAGE=node:20-bookworm-slim
+ARG RUNTIME_IMAGE=node:22-bookworm-slim
# ubuntu24.04, not 22.04: 22.04 is glibc 2.35, OLDER than the bookworm the
# workspace is built on, and the native modules would not load.
ARG CUDA_DEVEL_IMAGE=nvidia/cuda:12.6.3-devel-ubuntu24.04
@@ -303,15 +311,34 @@ FROM ${RUNTIME_IMAGE} AS runtime-base
# from a `docker compose exec` shell. debian:slim ships without it.
# aria2: archive.org files over BitTorrent (archive.org as the web seed), then
# seeded for a while; without it they are downloaded directly.
+# python3 + yt-dlp's optional modules (certifi, brotli, websockets, mutagen,
+# pycryptodome, requests): NOT for the yt-dlp below, which is a standalone
+# binary with its own python — for /usr/local/bin/yt-dlp-from-source, which
+# runs a yt-dlp SOURCE tree mounted at run time (docker-compose.ytdlp.yml).
+# pipx (+ python3-venv, which it needs to make a venv): installs git-filter-repo
+# below, which the homepage's source mirror runs.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ffmpeg aria2 zip unzip tar xz-utils gzip rsync curl ca-certificates git procps \
+ python3 python3-venv pipx python3-certifi python3-brotli python3-websockets \
+ python3-mutagen python3-pycryptodome python3-requests \
&& rm -rf /var/lib/apt/lists/*
+# git-filter-repo, PINNED, for `archilyzer source publish` (the homepage's
+# /source mirror). Installed, not left to `pipx run`, so a container publishes
+# with no network fetch at build time. The version is the one
+# common/publish/source.ts names in FILTER_REPO_PIPX_SPEC — a drift test
+# (common/publish/buildImage.test.ts) holds every `pipx install` here to it.
+# Into /opt + /usr/local/bin, not root's home: on PATH for every process.
+RUN PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install git-filter-repo==2.47.0 \
+ && git filter-repo --version
+
# yt-dlp as the standalone release binary (it bundles its own python), installed
# writable so `yt-dlp -U` works — see YTDLP_AUTO_UPDATE in docker/entrypoint.sh.
# A pinned yt-dlp goes stale fast, and a stale yt-dlp is the single most common
-# reason downloads start failing.
+# reason downloads start failing. It is THE IMAGE'S yt-dlp
+# (ARCHILYZER_IMAGE_YTDLP below); YTDLP_BIN may name another one at run time —
+# RUNNING_IN_DOCKER.md, "Substituting yt-dlp".
ARG TARGETARCH=amd64
RUN set -eux; \
case "${TARGETARCH}" in \
@@ -353,12 +380,21 @@ RUN npm install -g pnpm@9.15.4
WORKDIR /repo
COPY --from=build /repo /repo
+# The yt-dlp substitution hook: a wrapper that runs a yt-dlp SOURCE tree
+# mounted at YTDLP_SOURCE_DIR (default /opt/yt-dlp-src) with the python above.
+# Selected by YTDLP_BIN=/usr/local/bin/yt-dlp-from-source; see
+# docker-compose.ytdlp.yml.
+RUN ln -s /repo/docker/yt-dlp-from-source.sh /usr/local/bin/yt-dlp-from-source
+
# Data lives in volumes, never in the image. See docker-compose.yml.
+# YTDLP_BIN is the one the app runs; ARCHILYZER_IMAGE_YTDLP is the one this
+# image ships, so the entrypoint and `archilyzer doctor` can say "override".
ENV NODE_ENV=production \
NEXT_TELEMETRY_DISABLED=1 \
TRANSCRIPTS_DIR=/data/transcripts \
SETTINGS_FILE=/data/config/settings.json \
YTDLP_BIN=/usr/local/bin/yt-dlp \
+ ARCHILYZER_IMAGE_YTDLP=/usr/local/bin/yt-dlp \
EXPORT_INDEX_DIR=/data/builds/.export-index \
EXPORT_BUILDS_DIR=/data/builds/.export-builds \
ARCHILYZER_SITE_OUT=/data/builds/site
@@ -376,6 +412,15 @@ ENV ARCHILYZER_TRANSCRIBER=whisper-cpp \
WHISPER_BIN=/usr/local/bin/whisper-cli \
WHISPER_MODEL=/data/models/ggml-base.en.bin
+# Which commit and branch this image was built from — the publish stamps'
+# `commit`/`branch` where there is no .git (.dockerignore keeps it out). Last,
+# so a new commit re-runs one ENV layer and nothing else. Passed by compose from
+# the shell: ARCHILYZER_COMMIT=$(git rev-parse HEAD) docker compose build.
+ARG ARCHILYZER_COMMIT=
+ARG ARCHILYZER_BRANCH=
+ENV ARCHILYZER_COMMIT=${ARCHILYZER_COMMIT} \
+ ARCHILYZER_BRANCH=${ARCHILYZER_BRANCH}
+
# ---------------------------------------------------------------------------
# runtime-vulkan — parakeet.cpp on any Vulkan GPU. docker-compose.vulkan.yml.
#
@@ -405,18 +450,32 @@ ENV ARCHILYZER_TRANSCRIBER=parakeet \
WHISPER_BIN=/usr/local/bin/whisper-cli \
WHISPER_MODEL=/data/models/ggml-base.en.bin
+# Which commit and branch this image was built from — the publish stamps'
+# `commit`/`branch` where there is no .git (.dockerignore keeps it out). Last,
+# so a new commit re-runs one ENV layer and nothing else. Passed by compose from
+# the shell: ARCHILYZER_COMMIT=$(git rev-parse HEAD) docker compose build.
+ARG ARCHILYZER_COMMIT=
+ARG ARCHILYZER_BRANCH=
+ENV ARCHILYZER_COMMIT=${ARCHILYZER_COMMIT} \
+ ARCHILYZER_BRANCH=${ARCHILYZER_BRANCH}
+
# ---------------------------------------------------------------------------
# runtime-cuda — whisper.cpp on CUDA. NVIDIA only. docker-compose.gpu.yml.
# ---------------------------------------------------------------------------
FROM ${CUDA_RUNTIME_IMAGE} AS runtime-cuda
-ARG NODE_MAJOR=20
+ARG NODE_MAJOR=22
+# The same python + pipx + git-filter-repo as runtime-base (read its comments).
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ffmpeg aria2 zip unzip tar xz-utils gzip rsync curl ca-certificates git gnupg procps \
+ python3 python3-venv pipx python3-certifi python3-brotli python3-websockets \
+ python3-mutagen python3-pycryptodome python3-requests \
&& curl -fsSL "https://deb.nodesource.com/setup_${NODE_MAJOR}.x" | bash - \
&& apt-get install -y --no-install-recommends nodejs \
&& rm -rf /var/lib/apt/lists/*
+RUN PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install git-filter-repo==2.47.0 \
+ && git filter-repo --version
ARG TARGETARCH=amd64
RUN set -eux; \
@@ -461,6 +520,7 @@ RUN ldconfig
WORKDIR /repo
COPY --from=build /repo /repo
+RUN ln -s /repo/docker/yt-dlp-from-source.sh /usr/local/bin/yt-dlp-from-source
ENV NODE_ENV=production \
NEXT_TELEMETRY_DISABLED=1 \
@@ -470,6 +530,7 @@ ENV NODE_ENV=production \
WHISPER_BIN=/usr/local/bin/whisper-cli \
WHISPER_MODEL=/data/models/ggml-base.en.bin \
YTDLP_BIN=/usr/local/bin/yt-dlp \
+ ARCHILYZER_IMAGE_YTDLP=/usr/local/bin/yt-dlp \
EXPORT_INDEX_DIR=/data/builds/.export-index \
EXPORT_BUILDS_DIR=/data/builds/.export-builds \
ARCHILYZER_SITE_OUT=/data/builds/site
@@ -477,3 +538,12 @@ RUN mkdir -p /data/transcripts /data/config /data/models /data/builds
ENTRYPOINT ["/repo/docker/entrypoint.sh"]
CMD ["editor"]
+
+# Which commit and branch this image was built from — the publish stamps'
+# `commit`/`branch` where there is no .git (.dockerignore keeps it out). Last,
+# so a new commit re-runs one ENV layer and nothing else. Passed by compose from
+# the shell: ARCHILYZER_COMMIT=$(git rev-parse HEAD) docker compose build.
+ARG ARCHILYZER_COMMIT=
+ARG ARCHILYZER_BRANCH=
+ENV ARCHILYZER_COMMIT=${ARCHILYZER_COMMIT} \
+ ARCHILYZER_BRANCH=${ARCHILYZER_BRANCH}
diff --git a/ENVIRONMENT.md b/ENVIRONMENT.md
@@ -40,7 +40,7 @@ The one override surface for where things live and which binary runs. Every one
| `ARIA2C_BIN` | `aria2c` on PATH | Fetches an archive.org file over BitTorrent (the item's torrent, archive.org as web seed), then seeds it for a while. Optional: without it archive.org files are downloaded directly. See `archiveOrg` in [SETTINGS.md](SETTINGS.md). | common/lib/paths.ts (getPaths) |
| `OLLAMA_URL` | `http://127.0.0.1:11434` | The local ollama server, the local digest and attribution engine. | common/lib/paths.ts (getPaths) |
| `CLAUDE_BIN` | `claude` on PATH | The `claude` CLI, driving the opt-in metered digest lane. | common/lib/paths.ts (getPaths) |
-| `ARCHILYZER_CONFIG_DIR` | `~/.config/archilyzer` | The operator's private config dir, outside the repo: the two inputs of `archilyzer source publish` below. Never committed. | common/lib/paths.ts (getPaths) |
+| `ARCHILYZER_CONFIG_DIR` | `~/.config/archilyzer` | The operator's private config dir, outside the repo: the two inputs of `archilyzer source publish` below. Never committed. In Docker it is `/data/config/archilyzer`, in the config volume (docker-compose.yml). | common/lib/paths.ts (getPaths) |
| `SOURCE_SCRUB_FILE` | `<ARCHILYZER_CONFIG_DIR>/source-scrub.txt` | git-filter-repo `lhs==>rhs` rules applied to file contents AND commit messages when the source mirror is generated (`<home dir>==>/home/user` is built in and runs first). Every rule's left side is also denied. See [PUBLISH.md](PUBLISH.md). | common/lib/paths.ts (getPaths) |
| `SOURCE_DENYLIST_FILE` | `<ARCHILYZER_CONFIG_DIR>/source-denylist.txt` | Literals the published source must never contain, one per line (`i:` = any case). One hit anywhere in the mirror, the tree or the tarball refuses the publish. | common/lib/paths.ts (getPaths) |
| `ARCHILYZER_SOURCE_SCRATCH` | the OS temp dir | Where `source publish` makes its scratch clone and stage (removed afterwards unless `--keep-scratch`). | common/lib/paths.ts (getPaths) |
@@ -57,9 +57,16 @@ Tokens, credentials and knobs a running process reads. Most configuration is not
| `SYNC_HEARTBEAT_SECONDS` | `settings.syncScheduler.heartbeatSeconds` | Overrides the editor's in-process sync heartbeat. `0` = no internal timer (tick from cron instead). | editor/app/scheduler/heartbeat.ts |
| `SYNC_TICK_URL` | `http://127.0.0.1:3001/api/scheduler/tick` | Where `archilyzer sync tick` (cron's heartbeat) posts. | common/bin/sync-tick.ts |
| `SYNC_TICK_TOKEN` | unset (no auth) | Bearer token for the tick endpoint; set on both the editor and the cron job. | common/bin/sync-tick.ts, editor/app/scheduler/auth.ts |
-| `R2_ACCESS_KEY_ID` | — | R2 S3 credentials for uploading oversize archives at deploy time (with `R2_SECRET_ACCESS_KEY` and `CLOUDFLARE_ACCOUNT_ID`). See [PUBLISH.md](PUBLISH.md). | common/publish/build.ts |
-| `R2_SECRET_ACCESS_KEY` | — | See `R2_ACCESS_KEY_ID`. | common/publish/build.ts |
-| `CLOUDFLARE_ACCOUNT_ID` | — | The account the R2 endpoint belongs to. wrangler reads its own credentials. | common/publish/build.ts |
+| `R2_ACCESS_KEY_ID` | — | R2 S3 credentials for uploading oversize archives at deploy time (with `R2_SECRET_ACCESS_KEY` and `CLOUDFLARE_ACCOUNT_ID`), needed only when `archiveStorage.bucket` is set. In Docker they come from `.env`. See [PUBLISH.md](PUBLISH.md). | common/publish/build.ts, common/bin/doctor.ts (set or not) |
+| `R2_SECRET_ACCESS_KEY` | — | See `R2_ACCESS_KEY_ID`. | common/publish/build.ts, common/bin/doctor.ts (set or not) |
+| `CLOUDFLARE_ACCOUNT_ID` | — | The Cloudflare account: the R2 endpoint's, and the one wrangler deploys to when the token can see more than one. In Docker it comes from `.env`. | common/publish/build.ts, wrangler, common/bin/doctor.ts (set or not) |
+| `CLOUDFLARE_API_TOKEN` | unset (wrangler's own `wrangler login` config, on a host) | The API token every deploy's wrangler authenticates with (Cloudflare Pages: Edit). The way a container deploys — there is no browser for `wrangler login` in one; set it in `.env`. | wrangler (every deploy), common/bin/doctor.ts, common/lib/pagesDeploy.ts (set or not, never the value) |
+| `ARCHILYZER_HOST_ID` | the hostname | Which host the publish lock (`<EXPORT_BUILDS_DIR>/.publish.lock`) names as its holder's: a lock from this host whose pid is dead is stale and taken over; another host's is waited on. docker-compose.yml fixes it for the editor (`archilyzer-editor`), whose hostname is a container id that changes on every recreate. | common/publish/stageLock.ts (the publish lock) |
+| `ARCHILYZER_SOURCE_REPO` | this checkout's git common dir | The git DIR `archilyzer source publish` mirrors `main` from, when the checkout has none: in Docker, `/data/source.git`, the host's git common dir mounted read-only by docker-compose.source.yml. A value that names nothing refuses the publish. | common/publish/source.ts, common/bin/doctor.ts, docker/entrypoint.sh |
+| `YTDLP_SOURCE_HOST_DIR` | — (required by the overlay) | Docker: the HOST path of a yt-dlp source checkout (the directory holding `yt_dlp/`), mounted read-only at `/opt/yt-dlp-src` by docker-compose.ytdlp.yml. See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md), "Substituting yt-dlp". | docker-compose.ytdlp.yml |
+| `YTDLP_AUTO_UPDATE` | off | Docker: `1` runs `yt-dlp -U` on every editor boot — on the image's yt-dlp only; an override (`YTDLP_BIN` naming another) is left alone, with a warning. | docker/entrypoint.sh, common/bin/doctor.ts |
+| `XDG_CONFIG_HOME` | `~/.config` | Where `wrangler login` keeps its config (`<it>/.wrangler/config/default.toml`); the doctor looks for it there, by path, never reading it. | common/bin/doctor.ts, common/lib/pagesDeploy.ts |
+| `YTDLP_SOURCE_DIR` | `/opt/yt-dlp-src` | Docker: where `/usr/local/bin/yt-dlp-from-source` finds the yt-dlp source tree it runs with the image's python. | docker/yt-dlp-from-source.sh |
| `DOCKER_BIN` | `docker` | The container engine for docker-mode builds (e.g. `podman`). | common/publish/build.ts |
| `DOCKER_BUILD_MEMORY` | no cap | Per-container memory cap for a docker-mode build (`--memory`). | common/publish/build.ts |
| `DOCKER_BUILD_CPUS` | no cap | Per-container CPU cap for a docker-mode build (`--cpus`). | common/publish/build.ts |
@@ -67,8 +74,6 @@ Tokens, credentials and knobs a running process reads. Most configuration is not
| `HOST` | every interface | The address `pnpm start:export` (serve-out) listens on; `127.0.0.1` keeps a private site on this machine. | export/scripts/serve-out.mjs |
| `MAX_ARCHIVE_BYTES` | the Cloudflare-safe cap | The served-file size cap for archives, in bytes; `0` = no cap. A site's own `archiveMaxBytes` wins. | common/bin/compose-site.ts |
| `WRANGLER_BIN` | `common/node_modules/.bin/wrangler` (the pinned devDependency) | The wrangler a deploy spawns. The editor's e2e suite points it at its fake. | common/lib/pagesDeploy.ts (wranglerBin), common/publish/deployStage.ts |
-| `CLOUDFLARE_API_TOKEN` | unset (a `wrangler login` on a host) | The API token every deploy runs wrangler with. With none and no `wrangler login`, a deploy is refused before wrangler runs; one Cloudflare rejects ends the deploy on "REFUSED by Cloudflare". Never printed. | wrangler (every deploy), common/lib/pagesDeploy.ts (set or not, never the value) |
-| `XDG_CONFIG_HOME` | `~/.config` | Where `wrangler login` keeps its config (`<it>/.wrangler/config/default.toml`); a deploy's credential check looks for it there, by path. | common/lib/pagesDeploy.ts |
| `CHOUGH_BIN` | `chough` on PATH | The chough transcription engine, when a worker names no binary. | common/lib/transcriptionApps.ts |
| `CHOUGH_MODEL` | chough's own | Passed to chough from a worker's model field; chough auto-downloads one when unset. | chough (set by common/lib/transcriptionApps.ts) |
| `CHOUGH_URL` | local | Passed to chough from a worker's remote-server field. | chough (set by common/lib/transcriptionApps.ts) |
@@ -144,7 +149,7 @@ The publish pipeline sets these for a process it spawns. Listed so a reader know
## Docker
-The container's own set, read by `docker/*.sh`, the compose files and Caddy — not by the apps' code (except `ARCHILYZER_IDLE_BOOT`). See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md).
+The container's own set, read by `docker/*.sh`, the compose files and Caddy — not by the apps' code (except `ARCHILYZER_IDLE_BOOT`, and the image's facts `archilyzer doctor` and the publish stamps read). See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md).
| Variable | Default | What it does | Read by |
|---|---|---|---|
@@ -152,7 +157,12 @@ The container's own set, read by `docker/*.sh`, the compose files and Caddy —
| `ARCHILYZER_FETCH_MODEL` | per transcriber | Which model the first boot downloads; `none` skips it. | docker/entrypoint.sh |
| `ARCHILYZER_MODELS_DIR` | `/data/models` | Where models live in the container. | docker/entrypoint.sh |
| `ARCHILYZER_BUILDS_DIR` | `/data/builds` | Where the container keeps built sites. | docker/entrypoint.sh |
-| `ARCHILYZER_SITE_OUT` | `/data/builds/site` | The built export site the `site` service serves. | docker/entrypoint.sh, docker/publish-site.sh |
+| `ARCHILYZER_SITE_OUT` | `/data/builds/site` | The built export site the `site` service serves. | docker/entrypoint.sh, docker/publish-site.sh, common/publish/deployStage.ts |
+| `ARCHILYZER_HOMEPAGE_OUT` | `/data/builds/homepage` | The locally deployed homepage (`publish homepage --deploy --to local`). The `homepage` service serves it when it is non-empty, else the image's baked build. | docker/entrypoint.sh, common/publish/deployStage.ts |
+| `ARCHILYZER_IMAGE_YTDLP` | baked: `/usr/local/bin/yt-dlp` | The yt-dlp the image ships. `YTDLP_BIN` naming anything else is an OVERRIDE: the boot's `yt-dlp:` line and `archilyzer doctor` say so, and `YTDLP_AUTO_UPDATE` leaves it alone. | docker/entrypoint.sh, common/bin/doctor.ts |
+| `ARCHILYZER_COMMIT` | baked: empty unless the build passed it | The commit the image was built from — the publish stamps' `commit` where there is no .git. `ARCHILYZER_COMMIT=$(git rev-parse HEAD) docker compose build`. | the Dockerfile (a build arg), the publish stamps (`IMAGE_COMMIT_ENV`) |
+| `ARCHILYZER_BRANCH` | baked: empty unless the build passed it | The branch the image was built from — the stamps' `branch`, which a production deploy checks. | the Dockerfile (a build arg), the publish stamps (`IMAGE_BRANCH_ENV`) |
+| `ARCHILYZER_SOURCE_HOST_DIR` | `./.git` | The HOST's git common dir docker-compose.source.yml mounts at `/data/source.git`. In a git worktree, the primary checkout's `.git`. | docker-compose.source.yml |
| `ARCHILYZER_IDLE_BOOT` | off | `1` boots the editor without arming the heartbeat or any auto-queue runner. | common/lib/idleBoot.ts (the editor) |
| `ARCHILYZER_AUTH_MODE` | `basic` | `basic`, `forward` or `none` — the only escape hatch from the exposure guard. | docker/guard-exposure.sh, docker/caddy-start.sh |
| `ARCHILYZER_AUTH_USER` | `archilyzer` | Basic-auth user. | docker/Caddyfile |
@@ -161,7 +171,6 @@ The container's own set, read by `docker/*.sh`, the compose files and Caddy —
| `ARCHILYZER_FORWARD_AUTH_UPSTREAM` | — | Forward-auth server (Authelia, tinyauth, …), `host:port`. | docker/Caddyfile |
| `ARCHILYZER_FORWARD_AUTH_URI` | `/api/auth/caddy` | The forward-auth server's verify path. | docker/Caddyfile |
| `ARCHILYZER_TAG` | `local` | The image tag the compose files build and run. | docker-compose*.yml |
-| `ARCHILYZER_HOMEPAGE_OUT` | `/data/builds/homepage` | The locally deployed homepage: `deploy homepage --to local` copies its bundle here. | common/publish/deployStage.ts |
## Tests only
diff --git a/RUNNING_IN_DOCKER.md b/RUNNING_IN_DOCKER.md
@@ -5,10 +5,12 @@ One command stands up a working archive server: the editor, the tools it drives
Vulkan image), and a reverse proxy that is the only thing on the box with an open
port.
-> This document is about **running the apps**. Fanning per-site *export builds* out
-> across containers is a different thing entirely — it uses `Dockerfile.build`, and
-> it is in [PUBLISH.md](PUBLISH.md#building-every-site-in-containers). Neither
-> affects the other.
+> This document is about **running the apps** — publishing included: every publish
+> stage (index, build, deploy) runs inside the editor's container. Fanning per-site
+> *export builds* out across containers is a different thing — the opt-in docker
+> runner of a HOST install, from `Dockerfile.build`, in
+> [PUBLISH.md](PUBLISH.md#building-every-site-in-containers). Neither affects the
+> other.
---
@@ -194,7 +196,10 @@ it accurately.
### Publish the archive
The site server serves a *static build of your corpus*, which does not exist
-until you make one:
+until you make one. Publishing is a set of **stages** — update the index, build a
+site, deploy it — and every one of them runs **inside the editor's container**:
+the queued jobs the editor starts (Sites → Publish, the publish lane) and the
+commands below alike. The quick way, for the local `site` service:
```sh
docker compose exec editor /repo/docker/publish-site.sh <site-id>
@@ -202,9 +207,105 @@ docker compose --profile site up -d
```
Create the site itself in the editor first (**Sites → New**); run the script with
-no argument to list the ones you have. It runs the same `pnpm --filter export run
-build` a host install runs, then copies the output into the volume the `site`
-service serves. No restart needed afterwards.
+no argument to list the ones you have. The script is three stages in a row, and
+you can run them yourself:
+
+```sh
+docker compose exec editor pnpm archilyzer publish index
+docker compose exec editor pnpm archilyzer publish build <site-id>
+docker compose exec editor pnpm archilyzer publish deploy <site-id> --to local
+docker compose exec editor pnpm archilyzer publish status
+```
+
+A stage that is already fresh does nothing, so running one twice is cheap. A
+local deploy copies the built bundle into the volume the `site` service serves;
+no restart needed afterwards. A **private** site (`audience: "private"`) is
+built but never deployed, locally or anywhere else.
+
+**`exec`, never `run --rm`.** `docker compose run --rm editor …` starts a SECOND
+container with its own copy of the image's `export/public` and a second writer on
+the index, and the publish lock (which keeps a stage you start from colliding
+with one the editor is running) cannot see across containers — worse, the second
+container carries the editor's host identity below with its own pids, so it would
+judge the editor's live lock dead and take it. `exec` runs in the
+editor's own container, beside its jobs, under the same lock. `pnpm ops publish`
+from the host goes through the editor too.
+
+The lock names its holder by host and pid. A container's hostname changes every
+time it is recreated, so compose gives the editor a fixed identity
+(`ARCHILYZER_HOST_ID=archilyzer-editor`): a lock left by a stage that died with the
+container is then recognised as this editor's own, and taken over once its pid is
+gone. A lock naming any OTHER host is waited on, never taken. If one is left
+behind — say, from before this setting, or by a host install sharing the volume
+that is gone for good — and you are sure nothing is publishing, delete it:
+`docker compose exec editor rm /data/builds/.export-builds/.publish.lock`.
+
+#### Deploying to Cloudflare from the container
+
+The same stages deploy to Cloudflare Pages. The container has no browser for
+`wrangler login`, so it authenticates with a token from `.env`, which the editor
+reads and every stage it runs inherits:
+
+```sh
+# .env
+CLOUDFLARE_API_TOKEN=... # an API token with "Cloudflare Pages: Edit"
+CLOUDFLARE_ACCOUNT_ID=...
+# only when Settings → archive storage names an R2 bucket for oversize archives:
+R2_ACCESS_KEY_ID=...
+R2_SECRET_ACCESS_KEY=...
+```
+
+```sh
+docker compose up -d # picks up .env
+docker compose exec editor pnpm archilyzer publish deploy <site-id> --preview <branch>
+docker compose exec editor pnpm archilyzer publish deploy <site-id> # production
+```
+
+Nothing is fetched at deploy time: wrangler is pinned in the workspace the image
+ships. With no token a deploy is refused before wrangler runs, with a sentence
+saying so; `archilyzer doctor` reports whether each credential is **set** (never
+its value). Every deploy is checked live afterwards. PUBLISH.md has the whole
+publish flow; this is only what is different in a container.
+
+#### The homepage, and its /source mirror
+
+The image carries a build of the project's own homepage, served by the
+`homepage` service. A real one — the corpus's numbers, the `/source` mirror of
+this repo — is a publish like any other:
+
+```sh
+docker compose exec editor pnpm archilyzer publish homepage --deploy --to local
+docker compose --profile homepage restart homepage # once, after the first
+```
+
+The `homepage` service serves the local deploy (`/data/builds/homepage`) once
+there is one, else the baked build — chosen at boot, hence the one restart.
+
+The `/source` mirror needs two things the image deliberately lacks:
+
+- **The repository.** The image has no `.git`. `docker-compose.source.yml` mounts
+ the host's git directory read-only at `/data/source.git` and points
+ `ARCHILYZER_SOURCE_REPO` at it (default `./.git`; from a git worktree set
+ `ARCHILYZER_SOURCE_HOST_DIR` to the primary checkout's `.git`). Without it the
+ homepage builds with an empty `/source` page.
+- **The operator's scrub rules and denylist.** They live in the config volume —
+ `ARCHILYZER_CONFIG_DIR` is `/data/config/archilyzer` in the container, made
+ (mode 700, empty) on the first boot — and are never printed by anything:
+
+```sh
+docker compose -f docker-compose.yml -f docker-compose.source.yml up -d
+docker compose cp ~/.config/archilyzer/source-scrub.txt editor:/data/config/archilyzer/
+docker compose cp ~/.config/archilyzer/source-denylist.txt editor:/data/config/archilyzer/
+docker compose exec editor chmod 600 /data/config/archilyzer/source-scrub.txt /data/config/archilyzer/source-denylist.txt
+```
+
+`git-filter-repo` (pinned) is in the image. **gitleaks and stagit are not**, so a
+source publish from the container is weaker than a host's in two ways it tells you
+about: it skips the secret scan, with a `WARNING` in its log (the literal audit
+against your denylist still runs, and still refuses), and it publishes the source
+without its history pages (`/source/git/`). `archilyzer doctor` lists both as
+absent. Publish the mirror from a host checkout that has them if you want either;
+a pinned gitleaks in the image is a planned follow-up.
### Model choice
@@ -224,12 +325,57 @@ For the Vulkan image it names a parakeet GGUF instead — see
An image pins yt-dlp at build time, and a stale yt-dlp is the most common reason
downloads suddenly start failing. Set `YTDLP_AUTO_UPDATE=1` in `.env` and each
-boot self-updates it. Or, once:
+boot self-updates it. Exactly `1`, `true`, `yes` or `on` turns it on, as written
+— `TRUE` or `Yes` does not (the entrypoint and `archilyzer doctor` read it the
+same way). Or, once:
```sh
docker compose exec editor yt-dlp -U
```
+Every editor boot logs which yt-dlp it will run, and whether it runs:
+
+```
+[entrypoint] yt-dlp: /usr/local/bin/yt-dlp 2026.09.30 (image)
+```
+
+### Substituting yt-dlp
+
+To run your own yt-dlp — a patched build, say — set `YTDLP_BIN`. It is a swap at
+run time: nothing is rebuilt, and the image's release binary stays where it is
+(`ARCHILYZER_IMAGE_YTDLP`, `/usr/local/bin/yt-dlp`). Two ways:
+
+**A source checkout**, run with the image's python (yt-dlp's optional modules
+are installed beside it):
+
+```sh
+# .env
+YTDLP_SOURCE_HOST_DIR=/home/you/yt-dlp-patched # the directory holding yt_dlp/
+```
+
+```sh
+docker compose -f docker-compose.yml -f docker-compose.ytdlp.yml up -d
+```
+
+The overlay mounts the checkout read-only at `/opt/yt-dlp-src` and sets
+`YTDLP_BIN=/usr/local/bin/yt-dlp-from-source`, a wrapper baked into the image.
+
+**A zipapp** built on the host (`make yt-dlp` in the checkout), copied into the
+config volume:
+
+```sh
+docker compose exec editor mkdir -p /data/config/bin
+docker compose cp ./yt-dlp editor:/data/config/bin/yt-dlp
+docker compose exec editor chmod 755 /data/config/bin/yt-dlp
+# .env
+YTDLP_BIN=/data/config/bin/yt-dlp
+```
+
+Either way the boot line says `(override)`, and `YTDLP_AUTO_UPDATE` leaves an
+override alone with a warning — update it where you build it. A substitute that
+does not run is reported as `yt-dlp: MISSING — … does not run: …` (the editor
+still starts), and by `archilyzer doctor`.
+
### Driving the editor without a browser
Every editor gesture is a server action, which is fine for a person and hostile
@@ -511,6 +657,8 @@ The `site` profile does not need the mount: an export build never reads `data/`.
docker compose logs -f editor
docker compose exec editor bash # a shell in the image
docker compose exec editor yt-dlp --version
+docker compose exec editor pnpm archilyzer doctor # what this container can do
+docker compose exec editor pnpm archilyzer publish status
docker compose restart editor
docker compose down # stop; volumes survive
docker compose down -v # stop AND DELETE the corpus
@@ -520,10 +668,13 @@ docker compose down -v # stop AND DELETE the corp
## What is in the image, and what is not
-**Baked in:** node + the installed workspace, the built editor / umtool / homepage,
+**Baked in:** Node 22 (the pinned wrangler needs it) + the installed workspace, the built editor / umtool / homepage,
`yt-dlp`, a JavaScript runtime for it (`deno` — without one yt-dlp warns and
silently loses formats), `ffmpeg`/`ffprobe`, `whisper-cli` (statically linked),
-`zip`/`tar`/`xz`/`gzip`, `rsync`, `git`, `curl`, `ps`. The Vulkan image adds
+`zip`/`tar`/`xz`/`gzip`, `rsync`, `git`, `curl`, `ps`; `python3` with yt-dlp's
+optional modules (for a substituted yt-dlp), `pipx` and a pinned `git-filter-repo`
+(the homepage's source mirror), and — in the workspace's `node_modules` — a
+pinned `wrangler`, so a deploy fetches nothing. The Vulkan image adds
`parakeet-cli`, the Mesa Vulkan drivers and `vulkaninfo`.
**Not baked, on purpose:**
@@ -532,7 +683,14 @@ silently loses formats), `ffmpeg`/`ffprobe`, `whisper-cli` (statically linked),
in the `corpus` volume.
- **whisper models.** 142 MB to 3 GB, and the choice is yours. Fetched on boot.
- **The export site build.** It is a static render *of a corpus*, and there is no
- corpus at image-build time. `docker/publish-site.sh` makes it at run time.
+ corpus at image-build time. The publish stages make it at run time.
+- **Credentials.** `CLOUDFLARE_API_TOKEN` and friends come from `.env` at run
+ time, never from the image.
+- **The repository.** No `.git`; `docker-compose.source.yml` mounts the host's
+ for the `/source` mirror. The build's commit and branch are baked as
+ `ARCHILYZER_COMMIT`/`ARCHILYZER_BRANCH` when the build passes them
+ (`ARCHILYZER_COMMIT=$(git rev-parse HEAD) ARCHILYZER_BRANCH=$(git branch
+ --show-current) docker compose build`) — the publish stamps record them.
- **ImageMagick with Pango, and `qrencode`.** Only `umtool/report-to-video/`
needs them, and rendering a report to video is a workstation task, not
something a server does. Run that part on a host checkout.
@@ -542,20 +700,21 @@ silently loses formats), `ffmpeg`/`ffprobe`, `whisper-cli` (statically linked),
Point `OLLAMA_URL` at a host ollama (`http://host.docker.internal:11434`) if you
want digests.
-### The multi-site build pipeline falls back inside a container
+### Two build runners, and the container has one
-The editor's **Build all sites** fans per-site export builds out across containers
-whenever a container engine answers (see [PUBLISH.md](PUBLISH.md#building-every-site-in-containers)). Inside a container there
-is no `docker` binary, so that path is unavailable. It already handles this — the
-build logs
+Every site build goes through one stage contract with two runners:
-```
-[notice] No container engine available — building sites serially on the host.
-```
+- **local** — the default everywhere, in a container or not: each stage is a child
+ process of the editor (or of the CLI you ran), one at a time.
+- **docker** — a HOST install's opt-in fan-out (`settings.publish.runner:
+ "docker"`, or `publish build all --runner docker`): one `Dockerfile.build`
+ container per site (see [PUBLISH.md](PUBLISH.md#building-every-site-in-containers)).
+ Both write the same bundles and stamps, so a deploy does not care which built it.
-and does the builds one after another in-process. Correct, just not parallel. **Do
-not** try to make docker-in-docker work for this; run the parallel pipeline from a
-host checkout if you need it.
+Inside the container there is no engine, and the docker runner **refuses** ("the
+docker runner needs an engine on this host") rather than pretend. The editor never
+gets the docker socket — **do not** mount it, and do not try to make
+docker-in-docker work; run the fan-out from a host checkout if you want it.
### It runs as root
@@ -582,6 +741,23 @@ Open http://localhost:8081. Keep the clone on the Windows filesystem — the bui
context is small, and the corpus lives in a Docker volume inside the WSL2 VM,
which is where the I/O actually happens.
+Publishing needs nothing else on Windows — Docker Desktop is the whole
+requirement. A checklist, in PowerShell from the clone:
+
+1. `copy .env.example .env`, then set in `.env`: `WORKER_TOKEN` (for `pnpm ops`
+ and agents), `CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_ACCOUNT_ID` (to deploy).
+2. `docker compose up -d --build`
+3. Open http://localhost:8081 → **Sites** → **Publish now**, or:
+ `docker compose exec editor pnpm archilyzer publish now`
+4. A local preview: `docker compose --profile site up -d site`, then
+ `docker compose exec editor pnpm archilyzer publish deploy <site-id> --to local`,
+ and open http://localhost:8080.
+5. `docker compose exec editor pnpm archilyzer doctor` — every check, including
+ whether the Cloudflare token is set (never its value).
+6. Optional, your own yt-dlp: set `YTDLP_SOURCE_HOST_DIR` (a Windows path such as
+ `C:\src\yt-dlp` works) and run
+ `docker compose -f docker-compose.yml -f docker-compose.ytdlp.yml up -d`.
+
You still want WSL2 directly if you intend to *develop* the project or run Claude
Code against it — see [README.md](README.md#claude-code-on-windows). For running
an archive, this is enough.
@@ -609,7 +785,15 @@ comparison above. The usual cause is a missing `/dev/dri` — a container withou
the render node still transcribes, just slowly.
**Downloads started failing on every channel.** Almost always a stale yt-dlp:
-`docker compose exec editor yt-dlp -U`.
+`docker compose exec editor yt-dlp -U`. Check `docker compose logs editor | grep
+yt-dlp:` first — `(override)` means `YTDLP_BIN` names your own build, which `-U`
+does not touch, and `MISSING` means it does not run at all (a from-source
+override with no checkout mounted, say).
+
+**A deploy is refused before wrangler runs.** `CLOUDFLARE_API_TOKEN` is not set in
+`.env`, or the editor was started before it was: `docker compose up -d` again
+(a restart does not re-read `.env`; `up` recreates the container).
+`archilyzer doctor`'s `cloudflare-auth` line says which.
**Port 8080 is already taken.** `SITE_HTTP_PORT=9080` in `.env`. Those variables
move the *outside* port; nothing inside the containers changes.
diff --git a/SETUP.md b/SETUP.md
@@ -30,7 +30,7 @@ homepage):
| Tool | Version | Notes |
| --- | --- | --- |
-| **Node.js** | **≥ 20.9** (LTS 20 or 22) | Required by Next.js 16. The code is typed against Node 20. |
+| **Node.js** | **22** (≥ 20.9 runs the apps) | Next.js 16 needs ≥ 20.9, but **deploying** runs the wrangler pinned in `common/package.json`, which refuses anything below Node 22 — so use 22. The Docker image ships 22. |
| **pnpm** | **9+** | Lockfile is v9. Easiest via Corepack (bundled with Node) — see below. |
| **git** | any recent | To clone the repo. |
| **C/C++ toolchain** | platform default | Only if pnpm can't find a prebuilt binary for a native module (`lmdb`, `sharp`, …). Usually not needed on mainstream platforms. |
diff --git a/common/bin/_cli.test.ts b/common/bin/_cli.test.ts
@@ -404,3 +404,83 @@ test("every bin in common/bin is reachable as a subcommand", () => {
const paths = COMMANDS.map((c) => c.path.join(" "));
assert.equal(new Set(paths).size, paths.length);
});
+
+// ── publishing as stages (release 18 slice S1) ──────────────────────────────
+
+test("the stage row: kind and target, the flags the child's argv carries, nothing else", async () => {
+ const { STAGE_FLAGS, stageArgv } = await import("../publish/stages");
+ const hit = resolveCommand(COMMANDS, ["stage", "deploy-site", "jer"]);
+ assert.deepEqual(hit?.command.path, ["stage"]);
+ assert.deepEqual(hit?.rest, ["deploy-site", "jer"]);
+ assert.deepEqual(hit!.command.flags, { ...STAGE_FLAGS });
+ // The argv the editor spawns parses through runCli's own parser, cleanly.
+ const argv = stageArgv({
+ kind: "build-site",
+ target: "_all",
+ runId: "run-1",
+ runner: "docker",
+ force: true,
+ skipArchives: true,
+ indexAfter: 17,
+ });
+ assert.deepEqual(argv, [
+ "stage", "build-site", "_all", "--run-id", "run-1", "--runner", "docker", "--force", "--skip-archives", "--index-after", "17",
+ ]);
+ const parsed = parseArgv(argv, booleanFlags(COMMANDS));
+ assert.deepEqual(parsed.positionals, ["stage", "build-site", "_all"]);
+ assert.equal(argumentProblem(hit!.command, parsed.flags, ["build-site", "_all"]), null);
+ assert.match(argumentProblem(hit!.command, {}, ["a", "b", "c"])!, /unexpected argument "c"/);
+ assert.match(argumentProblem(hit!.command, { nodata: true }, [])!, /unknown flag --nodata/);
+});
+
+test("the publish rows and their flags (status and now are S3's)", () => {
+ const row = (...p: string[]) => resolveCommand(COMMANDS, p)?.command;
+ assert.deepEqual(row("publish", "index")?.flags ?? {}, {});
+ assert.deepEqual(row("publish", "build", "jer")?.flags, { runner: "string", force: "boolean", "skip-archives": "boolean" });
+ assert.equal(row("publish", "build", "all")?.maxPositionals, 1);
+ assert.deepEqual(row("publish", "deploy", "all")?.flags, { preview: "string", to: "string", force: "boolean" });
+ assert.deepEqual(row("publish", "hub")?.flags, { deploy: "boolean", preview: "string", force: "boolean" });
+ assert.deepEqual(row("publish", "homepage")?.flags, {
+ deploy: "boolean",
+ preview: "string",
+ to: "string",
+ force: "boolean",
+ });
+ const parsed = parseArgv(["publish", "hub", "--deploy", "--preview", "r18"], booleanFlags(COMMANDS));
+ assert.deepEqual(parsed, { positionals: ["publish", "hub"], flags: { deploy: true, preview: "r18" } });
+ const deploy = row("publish", "deploy")!;
+ assert.match(argumentProblem(deploy, { preveiw: "x" }, ["jer"])!, /unknown flag --preveiw \(accepts --preview, --to, --force\)/);
+ assert.match(argumentProblem(deploy, { to: true }, ["jer"])!, /--to needs a value/);
+ const u = usage(COMMANDS);
+ assert.match(u, /archilyzer publish index\s+update the index/);
+ assert.match(u, /archilyzer publish build\s+<id\|all> \[--runner local\|docker\|auto\] \[--force\] \[--skip-archives\]/);
+ assert.match(u, /archilyzer publish deploy\s+<id\|all> \[--preview <branch>\] \[--to local\] \[--force\]/);
+ assert.match(u, /archilyzer stage\s+<kind> <target> --run-id <id>/);
+});
+
+test("build site, build all and deploy site are printed aliases of the publish rows", async () => {
+ const { ALIASES } = await import("./publish");
+ assert.equal(ALIASES.buildSite("jer", false), "archilyzer publish index && archilyzer publish build jer --force");
+ assert.equal(ALIASES.buildSite("jer", true), "archilyzer publish build jer --force", "--nodata skips the index");
+ assert.equal(ALIASES.buildAll, "archilyzer publish index && archilyzer publish build all --runner auto");
+ assert.equal(ALIASES.deploySite("jer"), "archilyzer publish deploy jer");
+ assert.equal(ALIASES.deploySite("jer", "r18"), "archilyzer publish deploy jer --preview r18");
+ const row = (...p: string[]) => resolveCommand(COMMANDS, p)!.command;
+ // Same flags as ever: a script that called them still parses.
+ assert.deepEqual(row("build", "site").flags, { nodata: "boolean", "skip-archives": "boolean", "allow-missing-media": "boolean" });
+ assert.deepEqual(row("build", "all").flags, { "skip-archives": "boolean" });
+ assert.deepEqual(row("deploy", "site").flags, { preview: "string" });
+ for (const r of [row("build", "site"), row("build", "all"), row("deploy", "site")]) {
+ assert.match(r.usage, /alias: /);
+ }
+});
+
+test("publish hub / homepage refuse a deploy's flag without --deploy (usage, nothing run)", async () => {
+ const { publishHub, publishHomepage } = await import("./publish");
+ const errors: string[] = [];
+ const out = { log: () => {}, error: (s: string) => errors.push(s) };
+ assert.equal(await publishHub({ preview: "r18" }, out), 2);
+ assert.equal(await publishHomepage({ to: "local" }, out), 2);
+ assert.match(errors.join("\n"), /--preview is a deploy — add --deploy/);
+ assert.match(errors.join("\n"), /--preview and --to are a deploy's — add --deploy/);
+});
diff --git a/common/bin/archilyzer.ts b/common/bin/archilyzer.ts
@@ -76,60 +76,133 @@ export const COMMANDS: Command[] = [
return 0;
},
},
+ // --- publishing as stages (release 18; bin/publish.ts) ----------------------
+ {
+ path: ["publish", "index"],
+ usage:
+ "update the index: the LMDB index, the stats datasets and the chart templates in one child (8 GB heap), then the index stamp every build reads",
+ run: async () => (await import("./publish")).publishIndex(),
+ },
+ {
+ path: ["publish", "build"],
+ usage:
+ "<id|all> [--runner local|docker|auto] [--force] [--skip-archives] build a site (or every stale one) into its bundle <exportBuildsDir>/<id>/out from the current index; --runner docker builds every site in containers (host only); a fresh site is a no-op without --force",
+ flags: { runner: "string", force: "boolean", "skip-archives": "boolean" },
+ maxPositionals: 1,
+ run: async ({ positionals, flags }) => {
+ const [target] = positionals;
+ const runner = flags.runner;
+ if (!target || (runner !== undefined && runner !== "local" && runner !== "docker" && runner !== "auto")) {
+ console.error("publish build: give <id|all> [--runner local|docker|auto]");
+ return 2;
+ }
+ return (await import("./publish")).publishBuild({
+ target,
+ runner: runner as "local" | "docker" | "auto" | undefined,
+ force: flags.force === true,
+ skipArchives: flags["skip-archives"] === true,
+ });
+ },
+ },
+ {
+ path: ["publish", "deploy"],
+ usage:
+ "<id|all> [--preview <branch>] [--to local] [--force] ship a site's bundle to its Pages project (a preview with --preview), or with --to local into ARCHILYZER_SITE_OUT; a bundle already deployed there is a no-op without --force",
+ flags: { preview: "string", to: "string", force: "boolean" },
+ maxPositionals: 1,
+ run: async ({ positionals, flags }) => {
+ const [target] = positionals;
+ const to = flags.to;
+ if (!target || (to !== undefined && to !== "pages" && to !== "local")) {
+ console.error("publish deploy: give <id|all> [--preview <branch>] [--to local]");
+ return 2;
+ }
+ return (await import("./publish")).publishDeploy({
+ target,
+ preview: typeof flags.preview === "string" ? flags.preview : undefined,
+ to: to as "pages" | "local" | undefined,
+ force: flags.force === true,
+ });
+ },
+ },
+ {
+ path: ["publish", "hub"],
+ usage:
+ "[--deploy] [--preview <branch>] [--force] build the hub into its bundle <exportBuildsDir>/_hub/out, then (--deploy) ship it",
+ flags: { deploy: "boolean", preview: "string", force: "boolean" },
+ run: async ({ flags }) =>
+ (await import("./publish")).publishHub({
+ deploy: flags.deploy === true,
+ preview: typeof flags.preview === "string" ? flags.preview : undefined,
+ force: flags.force === true,
+ }),
+ },
+ {
+ path: ["publish", "homepage"],
+ usage:
+ "[--deploy] [--preview <branch>] [--to local] [--force] build homepage/out (source mirror included), then (--deploy) ship it",
+ flags: { deploy: "boolean", preview: "string", to: "string", force: "boolean" },
+ run: async ({ flags }) => {
+ const to = flags.to;
+ if (to !== undefined && to !== "pages" && to !== "local") {
+ console.error("publish homepage: --to is pages or local");
+ return 2;
+ }
+ return (await import("./publish")).publishHomepage({
+ deploy: flags.deploy === true,
+ preview: typeof flags.preview === "string" ? flags.preview : undefined,
+ to: to as "pages" | "local" | undefined,
+ force: flags.force === true,
+ });
+ },
+ },
+ // `publish status` and `publish now` read the publish state view — release 18
+ // slice S3 adds their rows here.
+ {
+ path: ["stage"],
+ usage:
+ "<kind> <target> --run-id <id> [--preview <b>] [--to local] [--runner docker] [--force] [--skip-archives] [--allow-missing-media] [--index-after <ms>] [--built-after <ms>] INTERNAL: one publish stage, as the editor's job runs it (exit 0 ran/no-op, 1 failed, 2 usage, 3 precondition not met, 130 cancelled)",
+ // publish/stages.ts STAGE_FLAGS, spelled out so this table stays free of
+ // imports (_cli.test.ts holds the two equal).
+ flags: {
+ "run-id": "string",
+ preview: "string",
+ to: "string",
+ runner: "string",
+ force: "boolean",
+ "skip-archives": "boolean",
+ "allow-missing-media": "boolean",
+ "index-after": "string",
+ "built-after": "string",
+ },
+ maxPositionals: 2,
+ run: async (ctx) => (await import("./publish")).stageRow(ctx),
+ },
+ // The rows the stages replaced, kept as printed aliases.
{
path: ["build", "site"],
usage:
- "<id> [--nodata] [--skip-archives] [--allow-missing-media] data phase + compose + next build into export/out (default id: SITE_ID)",
+ "<id> [--nodata] [--skip-archives] [--allow-missing-media] alias: publish index (not with --nodata) + publish build <id> --force (default id: SITE_ID)",
flags: { nodata: "boolean", "skip-archives": "boolean", "allow-missing-media": "boolean" },
maxPositionals: 1,
run: async ({ positionals, flags, env }) => {
const siteId = siteIdFrom(positionals, env, "build site");
if (!siteId) return 2;
- // A missing site.json reads as a site of defaults, so a typo would build
- // the whole data phase before compose noticed. Refuse it up front.
- const { listSiteIds } = await import("../lib/site");
- const known = listSiteIds();
- if (!known.includes(siteId)) {
- console.error(
- `build site: no site "${siteId}" (configured: ${known.join(", ") || "none"})`,
- );
- return 2;
- }
- const { buildSite } = await import("../publish/build");
- const code = await buildSite(siteId, {
- signal: interrupted(),
- skipData: flags.nodata === true,
+ return (await import("./publish")).buildSiteAlias({
+ siteId,
+ nodata: flags.nodata === true,
skipArchives: flags["skip-archives"] === true,
allowMissingMedia: flags["allow-missing-media"] === true,
});
- if (code !== 0) console.error(`build site ${siteId}: failed (exit ${code})`);
- return code;
},
},
{
path: ["build", "all"],
usage:
- "[--skip-archives] build every site: docker fan-out when an engine answers, else serially on the host",
+ "[--skip-archives] alias: publish index + publish build all --runner auto (containers when an engine answers, else serially on the host)",
flags: { "skip-archives": "boolean" },
- run: async ({ flags }) => {
- const { buildAll, dockerAvailable } = await import("../publish/build");
- const signal = interrupted();
- const useDocker = await dockerAvailable(signal);
- if (!useDocker) {
- console.log(
- "[notice] No container engine available — building sites serially on the host.",
- );
- }
- const outcomes = await buildAll({
- signal,
- mode: useDocker ? "docker" : "basic",
- skipArchives: flags["skip-archives"] === true,
- });
- const failed = outcomes.filter((o) => o.code !== 0);
- console.log(`\n=== Summary: ${outcomes.length - failed.length}/${outcomes.length} built ===`);
- for (const f of failed) console.error(` ${f.siteId}: exit ${f.code}`);
- return failed.length || signal.aborted ? 1 : 0;
- },
+ run: async ({ flags }) =>
+ (await import("./publish")).buildAllAlias({ skipArchives: flags["skip-archives"] === true }),
},
{
path: ["build", "hub"],
@@ -263,19 +336,16 @@ export const COMMANDS: Command[] = [
{
path: ["deploy", "site"],
usage:
- "<id> [--preview <branch>] ship the site built in export/out to its Pages project (default id: SITE_ID)",
+ "<id> [--preview <branch>] alias: publish deploy <id> [--preview <branch>] — ship the site's bundle to its Pages project (default id: SITE_ID)",
flags: { preview: "string" },
maxPositionals: 1,
run: async ({ positionals, flags, env }) => {
const siteId = siteIdFrom(positionals, env, "deploy site");
if (!siteId) return 2;
- const { deploySite } = await import("../publish/build");
- return refusalsExit(() =>
- deploySite(siteId, {
- signal: interrupted(),
- previewBranch: typeof flags.preview === "string" ? flags.preview : undefined,
- }),
- );
+ return (await import("./publish")).deploySiteAlias({
+ siteId,
+ preview: typeof flags.preview === "string" ? flags.preview : undefined,
+ });
},
},
{
diff --git a/common/bin/compose-site.ts b/common/bin/compose-site.ts
@@ -23,9 +23,9 @@
import path from "node:path";
import { cp, link, mkdir, rm, readdir, access, readFile, writeFile, stat, rename } from "node:fs/promises";
-import { createHash } from "node:crypto";
import type { Dirent } from "node:fs";
import { getPaths, type Paths } from "../lib/paths";
+import { dirSignature } from "../lib/dirSignature";
import { getSite, resolveSocialLinks, resolveHubUrl, type Site } from "../lib/site";
import { getSettings } from "../lib/settings";
import {
@@ -697,51 +697,6 @@ async function writeComposeCache(p: string, cache: ComposeCache): Promise<void>
await writeFile(p, JSON.stringify(cache));
}
-// A cheap content signature for a directory: relative path + size + mtime of
-// every file within, hashed. It reflects exactly what a recursive copy would
-// move, so an unchanged source (build:index skipped it incrementally) yields the
-// same signature and the copy is skipped. The shared transcript/subs trees hold
-// only a bounded handful of paginated page files per channel, so this is far
-// cheaper than the copy it guards. Returns "" when the dir is absent/empty.
-//
-// `ignoreBasename` excludes files by name from the signature. The shared
-// per-channel trees carry a manifest.json that build:index rewrites with a fresh
-// `generatedAt` for EVERY channel whenever ANY channel mutates (the shared
-// page-writer loop runs over all channels). Its pageCount/slugToPage only change
-// when the channel's pages change — which the page files already capture — so
-// excluding it lets an unchanged channel stay skipped on a partial-change build
-// instead of re-copying all 50-odd channels. The tiny stale manifest left behind
-// on a skip is structurally identical (only its timestamp differs).
-async function dirSignature(
- dir: string,
- ignoreBasename?: string,
-): Promise<string> {
- const h = createHash("sha1");
- let any = false;
- const walk = async (rel: string): Promise<void> => {
- let ents: Dirent[];
- try {
- ents = await readdir(path.join(dir, rel), { withFileTypes: true });
- } catch {
- return;
- }
- ents.sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0));
- for (const e of ents) {
- if (!e.isDirectory() && e.name === ignoreBasename) continue;
- const childRel = rel ? `${rel}/${e.name}` : e.name;
- if (e.isDirectory()) {
- await walk(childRel);
- } else {
- any = true;
- const s = await stat(path.join(dir, childRel));
- h.update(`${childRel}\t${s.size}\t${s.mtimeMs}\n`);
- }
- }
- };
- await walk("");
- return any ? h.digest("hex") : "";
-}
-
// Materialize the member subset of a shared per-channel tree into public/,
// skipping channels whose source is unchanged since the last compose. Replaces
// the previous "rm -rf the whole tree then cp every member" with an in-place
diff --git a/common/bin/doctor.test.ts b/common/bin/doctor.test.ts
@@ -67,6 +67,8 @@ function checkout(): { root: string; bin: string; paths: Paths } {
parakeetBin: path.join(root, "scripts", "parakeet-stitch.mjs"),
parakeetModel: "",
parakeetCliBin: "parakeet-cli",
+ configDir: path.join(root, ".config"),
+ exportBuildsDir: path.join(root, "export", ".export-builds"),
sourceScrubFile: path.join(root, ".config", "source-scrub.txt"),
sourceDenylistFile: path.join(root, ".config", "source-denylist.txt"),
// Outside the checkout, as the XDG cache is: the tests that compare the
@@ -628,3 +630,182 @@ test("download pacing: a held platform and a raised pace warn, and nothing is wr
assert.match(dp[2].detail, /^4s between requests \(base 1s\)/);
assert.deepEqual(tree(c.root), before);
});
+
+// ── release 18: the downloader and publish checks ─────────────────────────
+
+const find = (r: DoctorReport, section: string, id: string) =>
+ r.checks.find((x) => x.section === section && x.id === id);
+
+// A HOME of the scenario's own, so wrangler's login config is never this
+// machine's.
+function home(c: ReturnType<typeof checkout>): string {
+ const h = path.join(TMP, `${path.basename(c.root)}-home`);
+ mkdirSync(h, { recursive: true });
+ return h;
+}
+
+test("downloader: a host's yt-dlp is a note; the image's is ok; an override says so, and warns beside YTDLP_AUTO_UPDATE or when it does not run", async () => {
+ const c = checkout();
+ fake(c.bin, "yt-dlp", "2026.09.30");
+ fake(c.bin, "yt-dlp-patched", "2026.10.01.patched");
+ // The from-source wrapper with no checkout mounted: a sentence, exit 127.
+ const broken = path.join(c.bin, "yt-dlp-from-source");
+ writeFileSync(broken, "#!/bin/sh\necho 'no yt_dlp package' >&2\nexit 127\n");
+ chmodSync(broken, 0o755);
+ const before = tree(c.root);
+ // A host install: no ARCHILYZER_IMAGE_YTDLP to compare with.
+ let r = await run(c);
+ assert.equal(find(r, "downloader", "yt-dlp")?.status, "info");
+ assert.match(find(r, "downloader", "yt-dlp")!.detail, /yt-dlp 2026\.09\.30 \(host: not in the runtime image\)$/);
+ // The runtime image, running its own.
+ const image = path.join(c.bin, "yt-dlp");
+ r = await run(c, { ARCHILYZER_IMAGE_YTDLP: image });
+ assert.equal(find(r, "downloader", "yt-dlp")?.status, "ok");
+ assert.match(find(r, "downloader", "yt-dlp")!.detail, /2026\.09\.30 \(image\)$/);
+ // An override (the paths' binary is another file).
+ (c.paths as { ytdlpBin: string }).ytdlpBin = path.join(c.bin, "yt-dlp-patched");
+ r = await run(c, { ARCHILYZER_IMAGE_YTDLP: image });
+ assert.equal(find(r, "downloader", "yt-dlp")?.status, "ok");
+ assert.match(find(r, "downloader", "yt-dlp")!.detail, new RegExp(`2026\\.10\\.01\\.patched \\(override; the image's is ${image.replace(/[.]/g, "\\.")}\\)$`));
+ r = await run(c, { ARCHILYZER_IMAGE_YTDLP: image, YTDLP_AUTO_UPDATE: "1" });
+ assert.equal(find(r, "downloader", "yt-dlp")?.status, "warn");
+ assert.match(find(r, "downloader", "yt-dlp")!.detail, /YTDLP_AUTO_UPDATE is set, and the entrypoint never self-updates an override/);
+ assert.equal(r.ok, true, "a warning, never a failure");
+ // The entrypoint's exact-match rule: `TRUE` (or ` 1`) is off there, so here.
+ for (const off of ["TRUE", " 1", "0", ""]) {
+ r = await run(c, { ARCHILYZER_IMAGE_YTDLP: image, YTDLP_AUTO_UPDATE: off });
+ assert.equal(find(r, "downloader", "yt-dlp")?.status, "ok", JSON.stringify(off));
+ }
+ // An override that does not run.
+ (c.paths as { ytdlpBin: string }).ytdlpBin = broken;
+ r = await run(c, { ARCHILYZER_IMAGE_YTDLP: image });
+ assert.equal(find(r, "downloader", "yt-dlp")?.status, "warn");
+ assert.match(find(r, "downloader", "yt-dlp")!.detail, /\(override; .*\) — does not run .*docker-compose\.ytdlp\.yml/);
+ assert.deepEqual(tree(c.root), before);
+});
+
+test("publish: cloudflare-auth reports a token SET (never its value), a wrangler login by path, and warns only when a site names a project", async () => {
+ const c = checkout();
+ const sitesDir = path.join(c.paths.transcriptsDir, "sites");
+ (c.paths as { sitesDir: string }).sitesDir = sitesDir;
+ const h = home(c);
+ // Nothing configured to deploy, no credential: a note.
+ let r = await run(c, { HOME: h });
+ assert.equal(find(r, "publish", "cloudflare-auth")?.status, "info");
+ // A site that deploys, no credential: a warning naming the site.
+ mkdirSync(path.join(sitesDir, "alpha"), { recursive: true });
+ writeFileSync(path.join(sitesDir, "alpha", "site.json"), JSON.stringify({ title: "A", cloudflareProject: "alpha-pages" }));
+ mkdirSync(path.join(sitesDir, "beta"), { recursive: true });
+ writeFileSync(path.join(sitesDir, "beta", "site.json"), JSON.stringify({ title: "B" }));
+ const before = tree(c.root);
+ r = await run(c, { HOME: h });
+ assert.equal(find(r, "publish", "cloudflare-auth")?.status, "warn");
+ assert.match(find(r, "publish", "cloudflare-auth")!.detail, /1 site names a Cloudflare project \(alpha\) — every deploy refuses; set CLOUDFLARE_API_TOKEN/);
+ assert.equal(r.ok, true);
+ // A token: ok, and the value appears nowhere in the report.
+ const secret = "PLANTED-TOKEN-0123456789";
+ r = await run(c, { HOME: h, CLOUDFLARE_API_TOKEN: secret, CLOUDFLARE_ACCOUNT_ID: "PLANTED-ACCOUNT" });
+ assert.equal(find(r, "publish", "cloudflare-auth")?.status, "ok");
+ assert.match(find(r, "publish", "cloudflare-auth")!.detail, /^CLOUDFLARE_API_TOKEN is set \(never printed\); CLOUDFLARE_ACCOUNT_ID set$/);
+ assert.ok(!/PLANTED/.test(renderDoctorReport(r)) && !/PLANTED/.test(JSON.stringify(r)));
+ // A host's `wrangler login`: ok, by path.
+ const cfg = path.join(h, ".config", ".wrangler", "config");
+ mkdirSync(cfg, { recursive: true });
+ writeFileSync(path.join(cfg, "default.toml"), 'oauth_token = "PLANTED-OAUTH"\n');
+ r = await run(c, { HOME: h });
+ assert.equal(find(r, "publish", "cloudflare-auth")?.status, "ok");
+ assert.match(find(r, "publish", "cloudflare-auth")!.detail, /wrangler's login config is at .*default\.toml/);
+ assert.ok(!/PLANTED/.test(renderDoctorReport(r)), "the login config is located, never read");
+ assert.deepEqual(tree(c.root), before);
+});
+
+test("publish: r2-keys appears only with a bucket, and names what is missing — never a value", async () => {
+ const c = checkout();
+ const h = home(c);
+ let r = await run(c, { HOME: h });
+ assert.equal(find(r, "publish", "r2-keys"), undefined, "no bucket, no check");
+ writeFileSync(c.paths.settingsFile, JSON.stringify({ archiveStorage: { bucket: "overflow", publicBaseUrl: "https://r2.example" } }));
+ const before = tree(c.root);
+ r = await run(c, { HOME: h, R2_ACCESS_KEY_ID: "PLANTED-KEY" });
+ assert.equal(find(r, "publish", "r2-keys")?.status, "warn");
+ assert.match(find(r, "publish", "r2-keys")!.detail, /"overflow" is set but R2_SECRET_ACCESS_KEY, CLOUDFLARE_ACCOUNT_ID are not/);
+ r = await run(c, { HOME: h, R2_ACCESS_KEY_ID: "PLANTED-KEY", R2_SECRET_ACCESS_KEY: "PLANTED-SECRET", CLOUDFLARE_ACCOUNT_ID: "PLANTED-ACCOUNT" });
+ assert.equal(find(r, "publish", "r2-keys")?.status, "ok");
+ assert.ok(!/PLANTED/.test(renderDoctorReport(r)));
+ assert.deepEqual(tree(c.root), before);
+});
+
+test("publish: export-builds — not there yet is a note; bundles with under 1.5x their size free warn; not writable warns", async () => {
+ const c = checkout();
+ const h = home(c);
+ let r = await run(c, { HOME: h }, { freeBytes: () => 5e9 });
+ assert.equal(find(r, "publish", "export-builds")?.status, "info");
+ assert.match(find(r, "publish", "export-builds")!.detail, /does not exist yet — the first site build makes it \(5\.00 GB free\)/);
+ const builds = c.paths.exportBuildsDir;
+ for (const [id, n] of [["alpha", 3_000_000], ["_hub", 1_000_000]] as const) {
+ mkdirSync(path.join(builds, id, "out", "sub"), { recursive: true });
+ writeFileSync(path.join(builds, id, "out", "sub", "f.bin"), Buffer.alloc(n));
+ }
+ // Staging beside a bundle is not a bundle.
+ mkdirSync(path.join(builds, "alpha", ".r2-staging"), { recursive: true });
+ writeFileSync(path.join(builds, "alpha", ".r2-staging", "big.zip"), Buffer.alloc(9_000_000));
+ const before = tree(c.root);
+ r = await run(c, { HOME: h }, { freeBytes: () => 5e9 });
+ assert.equal(find(r, "publish", "export-builds")?.status, "ok");
+ assert.match(find(r, "publish", "export-builds")!.detail, /: 2 bundles, 4 MB; 5\.00 GB free$/);
+ r = await run(c, { HOME: h }, { freeBytes: () => 5_000_000 });
+ assert.equal(find(r, "publish", "export-builds")?.status, "warn");
+ assert.match(find(r, "publish", "export-builds")!.detail, /under 1\.5× the bundles \(6 MB\)/);
+ chmodSync(builds, 0o555);
+ try {
+ if (process.getuid?.() !== 0) {
+ r = await run(c, { HOME: h }, { freeBytes: () => 5e9 });
+ assert.equal(find(r, "publish", "export-builds")?.status, "warn");
+ assert.match(find(r, "publish", "export-builds")!.detail, /is not writable/);
+ }
+ } finally {
+ chmodSync(builds, 0o755);
+ }
+ assert.equal(r.ok, true);
+ assert.deepEqual(tree(c.root), before);
+});
+
+test("source publish: source-repo — the variable naming nothing fails, a readable main is ok, none is a note (a warning once the operator's files exist); config-dir is counted", async () => {
+ const c = checkout();
+ const h = home(c);
+ // No repository, nothing meant: notes.
+ let r = await run(c, { HOME: h });
+ assert.equal(find(r, "source publish", "source-repo")?.status, "info");
+ assert.equal(find(r, "source publish", "config-dir")?.status, "info");
+ assert.match(find(r, "source publish", "config-dir")!.detail, /does not exist — ARCHILYZER_CONFIG_DIR moves it/);
+ // The operator's files exist: no repository is now a warning.
+ mkdirSync(path.dirname(c.paths.sourceScrubFile), { recursive: true });
+ writeFileSync(c.paths.sourceScrubFile, "PLANTED==>x\n");
+ writeFileSync(c.paths.sourceDenylistFile, "PLANTED\n");
+ const before = tree(c.root);
+ r = await run(c, { HOME: h });
+ assert.equal(find(r, "source publish", "source-repo")?.status, "warn");
+ assert.match(find(r, "source publish", "source-repo")!.detail, /docker-compose\.source\.yml/);
+ assert.equal(find(r, "source publish", "config-dir")?.status, "ok");
+ assert.match(find(r, "source publish", "config-dir")!.detail, /\(2 entries, writable\)$/);
+ assert.ok(!/PLANTED/.test(renderDoctorReport(r)));
+ // The variable naming nothing: the publish refuses, so the doctor fails.
+ r = await run(c, { HOME: h, ARCHILYZER_SOURCE_REPO: path.join(TMP, "not-mounted.git") });
+ assert.equal(find(r, "source publish", "source-repo")?.status, "fail");
+ assert.equal(r.ok, false);
+ // A real repository's git dir, through the default probe (git on PATH).
+ const repo = path.join(TMP, `${path.basename(c.root)}-repo`);
+ mkdirSync(repo);
+ const git = (...a: string[]) => execFileSync("git", a, { cwd: repo, stdio: "pipe", env: { ...process.env, GIT_CONFIG_NOSYSTEM: "1", HOME: h } });
+ git("init", "-q", "-b", "main");
+ git("-c", "user.name=t", "-c", "user.email=t@example.invalid", "-c", "commit.gpgsign=false", "commit", "-q", "--allow-empty", "-m", "one");
+ const head = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repo }).toString().trim();
+ r = await run(c, { HOME: h, PATH: `${c.bin}${path.delimiter}${process.env.PATH}`, ARCHILYZER_SOURCE_REPO: path.join(repo, ".git") });
+ assert.equal(find(r, "source publish", "source-repo")?.status, "ok");
+ assert.equal(find(r, "source publish", "source-repo")!.detail, `${path.join(repo, ".git")} (ARCHILYZER_SOURCE_REPO): main ${head.slice(0, 12)}`);
+ // An injected probe: main that does not read is a warning naming why.
+ r = await run(c, { HOME: h }, { sourceRepo: async () => ({ via: "env", repo: "/data/source.git", main: null, error: "fatal: detected dubious ownership" }) });
+ assert.equal(find(r, "source publish", "source-repo")?.status, "warn");
+ assert.match(find(r, "source publish", "source-repo")!.detail, /^\/data\/source\.git \(ARCHILYZER_SOURCE_REPO\): main does not read — fatal: detected dubious ownership$/);
+ assert.deepEqual(tree(c.root), before);
+});
diff --git a/common/bin/doctor.ts b/common/bin/doctor.ts
@@ -6,10 +6,14 @@
// (scripts/worktree.mjs over lib/ports.mjs).
//
// It also asks the container engine whether Build all's site build image is
-// there and older than its Dockerfile (common/publish/build.ts).
+// there and older than its Dockerfile (common/publish/build.ts), and what a
+// publish needs from this machine (release 18): which yt-dlp (the image's or
+// an override), whether deploy credentials are SET (never their values), room
+// for the bundles, and the repository the source mirror reads.
//
// STRICTLY READ-ONLY. It stats, reads and runs version flags, plus the engine's
-// `image inspect` and a lock-free `git status` / `git log`. It never opens
+// `image inspect`, a lock-free `git status` / `git log` and a `git rev-parse`
+// of the source repository's main. It never opens
// LMDB (the index is stat'd, not opened), never mkdirs, never writes settings,
// and never binds a port (a port is "in use" when a TCP connect succeeds). The
// one process-state change is a chdir around umtool's table, which resolves a
@@ -23,9 +27,10 @@
// and must not be told it is broken.
import { execFile } from "node:child_process";
-import { accessSync, constants, existsSync, readFileSync, statSync } from "node:fs";
+import { accessSync, constants, existsSync, readFileSync, realpathSync, statfsSync, statSync } from "node:fs";
import { readdir } from "node:fs/promises";
import net from "node:net";
+import os from "node:os";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { promisify } from "node:util";
@@ -84,6 +89,23 @@ export type DoctorDeps = {
now?: Date;
// The source publish's tools. Default: probeSourceTools(env).
sourceTools?: () => Promise<SourceTools>;
+ // Which repository `source publish` would mirror, and whether its main reads.
+ // Default: probeSourceRepo — ARCHILYZER_SOURCE_REPO, else the checkout's git
+ // common dir, then `rev-parse` of main (read-only).
+ sourceRepo?: () => Promise<SourceRepoProbe>;
+ // Free bytes on the filesystem holding `dir`, or null when it cannot be
+ // asked. Default: statfs.
+ freeBytes?: (dir: string) => number | null;
+};
+
+// Where `source publish` would read main from: the variable's path or the
+// checkout's common dir (null: neither), and main's commit or why it did not
+// read.
+export type SourceRepoProbe = {
+ via: "env" | "checkout";
+ repo: string | null;
+ main: string | null;
+ error?: string;
};
// Which git-filter-repo `archilyzer source publish` would run, gitleaks, and
@@ -356,6 +378,42 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
else add(T, r.id, "info", `${r.error ?? "absent"} — needed only for ${r.neededBy.join(", ")}`);
}
+ // ── downloader ───────────────────────────────────────────────────────────
+ // WHICH yt-dlp, beside the tools row's "is it there". In the runtime image
+ // ARCHILYZER_IMAGE_YTDLP names the one it ships; YTDLP_BIN landing anywhere
+ // else is the operator's substitute (RUNNING_IN_DOCKER.md, "Substituting
+ // yt-dlp") — an override, which the entrypoint's YTDLP_AUTO_UPDATE leaves
+ // alone. Graded here only for that conflict and for a substitute that does
+ // not run; presence is the tools row's to grade.
+ const DL = "downloader";
+ {
+ const r = reports.find((x) => x.id === "yt-dlp");
+ const imageYtdlp = env.ARCHILYZER_IMAGE_YTDLP?.trim() || null;
+ const resolved = onPath(paths.ytdlpBin, env.PATH) ?? paths.ytdlpBin;
+ const origin = imageYtdlp === null ? "host" : sameFile(resolved, imageYtdlp) ? "image" : "override";
+ // The entrypoint's rule exactly (a shell `case`: 1, true, yes or on, as
+ // written — `TRUE` is off there, so it is off here).
+ const autoUpdate = ["1", "true", "yes", "on"].includes(env.YTDLP_AUTO_UPDATE ?? "");
+ // The shared probe counts any output as presence (an odd version flag is
+ // still a binary); "does it RUN" is asked here by exit status — the
+ // from-source wrapper with no checkout mounted prints a sentence and exits
+ // 127.
+ const ran = r?.present ? await versionRuns(resolved, env) : { ok: false as const, error: r?.error ?? "absent" };
+ const what =
+ `${resolved}${ran.ok && ran.version ? ` ${ran.version}` : ""} (${origin}` +
+ `${origin === "override" ? `; the image's is ${imageYtdlp}` : origin === "host" ? ": not in the runtime image" : ""})`;
+ if (!ran.ok) {
+ add(DL, "yt-dlp", origin === "override" ? "warn" : "info",
+ `${what} — does not run (${ran.error})${origin === "override" ? "; a from-source override needs its checkout mounted (docker-compose.ytdlp.yml)" : ""}`);
+ } else if (origin === "override" && autoUpdate) {
+ add(DL, "yt-dlp", "warn",
+ `${what} — YTDLP_AUTO_UPDATE is set, and the entrypoint never self-updates an override: update it where it is built, or unset one of the two`);
+ } else {
+ add(DL, "yt-dlp", origin === "host" ? "info" : "ok",
+ `${what}${autoUpdate && origin === "image" ? " — self-updated on every boot (YTDLP_AUTO_UPDATE)" : ""}`);
+ }
+ }
+
// ── report pipeline (umtool) ─────────────────────────────────────────────
const U = "report pipeline (umtool)";
const umtoolSpecs = await (deps.umtoolTools ?? (() => loadUmtoolTools(root)))();
@@ -411,6 +469,59 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
}
}
+ // ── publish ──────────────────────────────────────────────────────────────
+ // What a deploy needs from this machine (release 18): credentials — whether
+ // they are SET, never a value — and room for the bundles. A warning at
+ // worst: a checkout that never deploys is not broken, and one whose sites
+ // name a Cloudflare project but holds no credential is told so.
+ const PB = "publish";
+ {
+ const deployable = await sitesWithCloudflareProject(paths);
+ const set = (k: string) => Boolean(env[k]?.trim());
+ const oauth = wranglerLoginConfig(env);
+ const account = set("CLOUDFLARE_ACCOUNT_ID") ? "CLOUDFLARE_ACCOUNT_ID set" : "CLOUDFLARE_ACCOUNT_ID unset (fine with one account)";
+ if (set("CLOUDFLARE_API_TOKEN")) {
+ add(PB, "cloudflare-auth", "ok", `CLOUDFLARE_API_TOKEN is set (never printed); ${account}`);
+ } else if (oauth) {
+ add(PB, "cloudflare-auth", "ok",
+ `no CLOUDFLARE_API_TOKEN; wrangler's login config is at ${oauth} — a host login, which a container cannot use (set the token in .env there)`);
+ } else {
+ add(PB, "cloudflare-auth", deployable.length > 0 ? "warn" : "info",
+ deployable.length > 0
+ ? `neither CLOUDFLARE_API_TOKEN nor a \`wrangler login\` config, and ${deployable.length} site${deployable.length === 1 ? " names" : "s name"} a Cloudflare project (${deployable.join(", ")}) — every deploy refuses; set CLOUDFLARE_API_TOKEN (in Docker: .env)`
+ : "neither CLOUDFLARE_API_TOKEN nor a `wrangler login` config — needed only to deploy to Cloudflare Pages");
+ }
+ const bucket = settings?.archiveStorage?.bucket?.trim();
+ if (bucket) {
+ const missing = ["R2_ACCESS_KEY_ID", "R2_SECRET_ACCESS_KEY", "CLOUDFLARE_ACCOUNT_ID"].filter((k) => !set(k));
+ add(PB, "r2-keys", missing.length === 0 ? "ok" : "warn",
+ missing.length === 0
+ ? `archiveStorage.bucket "${bucket}": R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY and CLOUDFLARE_ACCOUNT_ID are set (never printed)`
+ : `archiveStorage.bucket "${bucket}" is set but ${missing.join(", ")} ${missing.length === 1 ? "is" : "are"} not — a deploy with an oversize archive refuses before it uploads`);
+ }
+ const builds = paths.exportBuildsDir;
+ if (builds) {
+ const free = (deps.freeBytes ?? statfsFree)(nearestExisting(builds));
+ const st = statOrNull(builds);
+ const freeText = free === null ? "free space unknown" : `${gigabytes(free)} free`;
+ if (!st) {
+ add(PB, "export-builds", "info", `${builds} does not exist yet — the first site build makes it (${freeText})`);
+ } else if (!writable(builds)) {
+ add(PB, "export-builds", "warn", `${builds} is not writable — every site build fails (${freeText})`);
+ } else {
+ const bundles = await bundleBytes(builds);
+ const need = Math.ceil(bundles.bytes * 1.5);
+ const what = `${builds}: ${bundles.count} bundle${bundles.count === 1 ? "" : "s"}, ${gigabytes(bundles.bytes)}; ${freeText}`;
+ if (free !== null && bundles.count > 0 && free < need) {
+ add(PB, "export-builds", "warn",
+ `${what} — under 1.5× the bundles (${gigabytes(need)}): a build writes its new bundle beside the old one before it swaps`);
+ } else {
+ add(PB, "export-builds", "ok", what);
+ }
+ }
+ }
+ }
+
// ── source publish ───────────────────────────────────────────────────────
// `archilyzer build homepage` runs it (common/publish/source.ts). Never a
// failure: a checkout that never publishes the homepage is not broken. A
@@ -449,6 +560,41 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
}
add(SP, "gitleaks", tools.gitleaks ? "ok" : "info",
tools.gitleaks ? `gitleaks ${tools.gitleaks.version}` : "absent — the gate skips the secret scan with a WARNING (the literal audit still runs)");
+ // The repository whose main is mirrored: ARCHILYZER_SOURCE_REPO (a
+ // container's read-only mount of the host's git dir), else the checkout's.
+ // A variable naming nothing fails — the publish refuses by name.
+ {
+ const sr = await (deps.sourceRepo ?? (() => probeSourceRepo(env, root)))();
+ const via = sr.via === "env" ? "ARCHILYZER_SOURCE_REPO" : "this checkout";
+ if (sr.repo === null && sr.via === "env") {
+ add(SP, "source-repo", "fail",
+ `ARCHILYZER_SOURCE_REPO names ${env.ARCHILYZER_SOURCE_REPO?.trim()}, which is not there — \`source publish\` refuses; mount it (docker-compose.source.yml) or unset it`);
+ } else if (sr.repo === null) {
+ add(SP, "source-repo", intends ? "warn" : "info",
+ "no git repository here and ARCHILYZER_SOURCE_REPO is unset — the homepage builds with an empty /source page; in Docker, add docker-compose.source.yml");
+ } else if (sr.main === null) {
+ add(SP, "source-repo", "warn", `${sr.repo} (${via}): main does not read${sr.error ? ` — ${sr.error}` : ""}`);
+ } else {
+ add(SP, "source-repo", "ok", `${sr.repo} (${via}): main ${sr.main.slice(0, 12)}`);
+ }
+ }
+ // The operator's private config dir, which holds the two files below.
+ // Counted, never listed.
+ {
+ const dir = paths.configDir;
+ const st = dir ? statOrNull(dir) : null;
+ if (!dir || !st) {
+ add(SP, "config-dir", "info",
+ `${dir ?? "(unset)"} does not exist — ARCHILYZER_CONFIG_DIR moves it (in Docker: /data/config/archilyzer, the config volume)`);
+ } else if (!st.isDirectory()) {
+ add(SP, "config-dir", "warn", `${dir} is not a directory`);
+ } else {
+ const count = (await readdir(dir).catch(() => [] as string[])).length;
+ const w = writable(dir);
+ add(SP, "config-dir", w ? "ok" : "info",
+ `${dir} (${count} entr${count === 1 ? "y" : "ies"}${w ? ", writable" : ", read-only: fine for reading the rules"})`);
+ }
+ }
for (const [id, file, unit] of [
["scrub rules", scrubFile, "rule"],
["denylist", denylistFile, "literal"],
@@ -783,6 +929,147 @@ async function dirSizeText(dir: string): Promise<string> {
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
}
+// The same file, through symlinks (the image's yt-dlp-from-source is a link
+// into /repo/docker; `YTDLP_BIN=yt-dlp` resolves on PATH first).
+function sameFile(a: string, b: string): boolean {
+ try {
+ return realpathSync(a) === realpathSync(b);
+ } catch {
+ return path.resolve(a) === path.resolve(b);
+ }
+}
+
+function writable(p: string): boolean {
+ try {
+ accessSync(p, constants.W_OK);
+ return true;
+ } catch {
+ return false;
+ }
+}
+
+// `<bin> --version` exiting 0, and its first line; else why not.
+async function versionRuns(
+ bin: string,
+ env: NodeJS.ProcessEnv,
+): Promise<{ ok: true; version: string | null } | { ok: false; error: string }> {
+ try {
+ const { stdout } = await execFileP(bin, ["--version"], { env, timeout: 15_000 });
+ return { ok: true, version: stdout.trim().split("\n")[0] || null };
+ } catch (err) {
+ const e = err as { code?: unknown; stderr?: unknown };
+ const said = String(e.stderr ?? "").trim().split("\n")[0];
+ return { ok: false, error: said || `exited ${String(e.code ?? "?")}` };
+ }
+}
+
+// `p`, or its deepest ancestor that exists — where its free space is asked.
+function nearestExisting(p: string): string {
+ let cur = path.resolve(p);
+ while (!existsSync(cur)) {
+ const up = path.dirname(cur);
+ if (up === cur) break;
+ cur = up;
+ }
+ return cur;
+}
+
+function statfsFree(dir: string): number | null {
+ try {
+ const s = statfsSync(dir);
+ return s.bavail * s.bsize;
+ } catch {
+ return null;
+ }
+}
+
+// Every `<builds>/<target>/out` bundle and their bytes, by stat (links not
+// followed). A bundle's own size is what a rebuild writes again beside it.
+async function bundleBytes(buildsDir: string): Promise<{ count: number; bytes: number }> {
+ let count = 0;
+ let bytes = 0;
+ const walk = async (d: string): Promise<void> => {
+ for (const ent of await readdir(d, { withFileTypes: true }).catch(() => [])) {
+ const p = path.join(d, ent.name);
+ if (ent.isDirectory()) await walk(p);
+ else if (ent.isFile()) bytes += statOrNull(p)?.size ?? 0;
+ }
+ };
+ for (const ent of await readdir(buildsDir, { withFileTypes: true }).catch(() => [])) {
+ if (!ent.isDirectory()) continue;
+ const out = path.join(buildsDir, ent.name, "out");
+ if (!statOrNull(out)?.isDirectory()) continue;
+ count += 1;
+ await walk(out);
+ }
+ return { count, bytes };
+}
+
+// The sites whose site.json names a Cloudflare Pages project — what "this
+// machine is configured to deploy" means. Read-only; an unreadable file is
+// skipped.
+async function sitesWithCloudflareProject(paths: Paths): Promise<string[]> {
+ if (!paths.sitesDir) return [];
+ const out: string[] = [];
+ for (const e of await readdir(paths.sitesDir, { withFileTypes: true }).catch(() => [])) {
+ if (!e.isDirectory() || e.name.startsWith("_")) continue;
+ const text = readOrNull(path.join(paths.sitesDir, e.name, "site.json"));
+ if (text === null) continue;
+ try {
+ const raw = JSON.parse(text) as { cloudflareProject?: unknown };
+ if (typeof raw.cloudflareProject === "string" && raw.cloudflareProject.trim()) out.push(e.name);
+ } catch {
+ /* not JSON: the site's own problem */
+ }
+ }
+ return out.sort();
+}
+
+// wrangler's `wrangler login` (OAuth) config, where wrangler keeps it: under
+// XDG_CONFIG_HOME (~/.config) since v3, ~/.wrangler before, ~/Library/
+// Preferences on macOS. Its PATH is reported; it is never read.
+function wranglerLoginConfig(env: NodeJS.ProcessEnv): string | null {
+ const home = env.HOME || os.homedir();
+ const xdg = env.XDG_CONFIG_HOME || path.join(home, ".config");
+ for (const p of [
+ path.join(xdg, ".wrangler", "config", "default.toml"),
+ path.join(home, ".wrangler", "config", "default.toml"),
+ path.join(home, "Library", "Preferences", ".wrangler", "config", "default.toml"),
+ ]) {
+ if (existsSync(p)) return p;
+ }
+ return null;
+}
+
+// Which repository `source publish` would mirror (the same order as
+// source.ts's sourceRepoFor) and main's commit there. Read-only: rev-parse
+// with GIT_OPTIONAL_LOCKS=0.
+async function probeSourceRepo(env: NodeJS.ProcessEnv, root: string): Promise<SourceRepoProbe> {
+ const opts = { cwd: root, env: { ...env, GIT_OPTIONAL_LOCKS: "0" }, timeout: 10_000 };
+ const named = env.ARCHILYZER_SOURCE_REPO?.trim();
+ let via: SourceRepoProbe["via"] = "checkout";
+ let repo: string | null = null;
+ if (named) {
+ via = "env";
+ repo = existsSync(named) ? named : null;
+ } else {
+ try {
+ const { stdout } = await execFileP("git", ["rev-parse", "--path-format=absolute", "--git-common-dir"], opts);
+ repo = stdout.trim().split("\n").pop() || null;
+ } catch {
+ repo = null;
+ }
+ }
+ if (repo === null) return { via, repo, main: null };
+ try {
+ const { stdout } = await execFileP("git", [`--git-dir=${repo}`, "rev-parse", "--verify", "--quiet", "refs/heads/main^{commit}"], opts);
+ return { via, repo, main: stdout.trim() || null, error: stdout.trim() ? undefined : "no main branch" };
+ } catch (err) {
+ const stderr = String((err as { stderr?: unknown }).stderr ?? "").trim().split("\n")[0];
+ return { via, repo, main: null, error: stderr || "no main branch" };
+ }
+}
+
// In use = something accepts a TCP connection on 127.0.0.1. Never binds.
function tcpPortInUse(port: number): Promise<boolean> {
return new Promise((resolve) => {
diff --git a/common/bin/publish.ts b/common/bin/publish.ts
@@ -0,0 +1,299 @@
+// `archilyzer publish …` and `archilyzer stage …` (release 18): the publish
+// stages from the command line. The table rows are in archilyzer.ts; this is
+// what they run.
+//
+// publish index update-index (as a child: its heap cap)
+// publish build <id|all> [--runner local|docker|auto] [--force] [--skip-archives]
+// publish deploy <id|all> [--preview <b>] [--to local] [--force]
+// publish hub [--deploy] [--preview <b>] [--force]
+// publish homepage [--deploy] [--preview <b>] [--to local] [--force]
+// stage <kind> <target> [flags] the child the editor spawns
+//
+// Every stage runs under the publish lock (publish/stageLock.ts), so a CLI run
+// beside the editor waits for the editor's stage — and the editor's for it.
+// `publish index` runs the SAME child the editor spawns (stageCommand: the
+// index and stats builds want its 8 GB heap); the rest run in this process.
+// `publish status` and `publish now` are release 18 S3's (publishState.ts).
+
+import { killChildTreesNow, runChildIntoLog, setKillChildTrees } from "../jobs/runChild";
+import { getPaths, type Paths } from "../lib/paths";
+import { siteDeployProblem } from "../lib/builtExport";
+import { listSites, listSiteIds } from "../lib/site";
+import { newStampId } from "../publish/stamps";
+import { STAGE_EXIT, runStage, stageCommand, stageMain } from "../publish/stageRun";
+import { parseStageArgs, type StageRequest } from "../publish/stages";
+import type { CommandContext } from "./_cli";
+
+type Out = { log: (s: string) => void; error: (s: string) => void };
+
+function terminalLog(line: string): void {
+ process.stdout.write(line.endsWith("\n") ? line : `${line}\n`);
+}
+
+/** A run id for the stages one CLI command runs (the editor's are its own). */
+export function cliRunId(): string {
+ return `cli-${newStampId()}`;
+}
+
+// Ctrl-C / SIGTERM cancel the stage in flight, as the editor's Cancel does. A
+// SECOND one does not wait for the unwind: it kills the detached process
+// groups this process started (`next build`'s, wrangler's) and exits 130 —
+// never leaving an orphan builder writing export/out. Idempotent: one handler.
+function interrupted(): AbortSignal {
+ const ac = new AbortController();
+ const onSignal = () => {
+ if (!ac.signal.aborted) {
+ ac.abort();
+ return;
+ }
+ killChildTreesNow("SIGKILL");
+ process.exit(STAGE_EXIT.cancelled);
+ };
+ process.on("SIGINT", onSignal);
+ process.on("SIGTERM", onSignal);
+ return ac.signal;
+}
+
+// --- stage (the child) --------------------------------------------------------
+
+export async function stageRow(ctx: CommandContext, out: Out = console): Promise<number> {
+ const req = parseStageArgs(ctx.positionals, ctx.flags);
+ if ("error" in req) {
+ out.error(req.error);
+ return STAGE_EXIT.usage;
+ }
+ return stageMain(req);
+}
+
+// --- publish index ------------------------------------------------------------
+
+/**
+ * Run update-index as the editor does: a child with the index heap cap. A
+ * Ctrl-C reaches the child from the terminal (it unwinds and exits 130); a
+ * SIGTERM to this process is passed on to it.
+ */
+export async function publishIndex(opts: { paths?: Paths; runId?: string } = {}): Promise<number> {
+ const paths = opts.paths ?? getPaths();
+ const req: StageRequest = { kind: "update-index", target: "_index", runId: opts.runId ?? cliRunId() };
+ const cmd = stageCommand(paths, req);
+ const ac = new AbortController();
+ const onInt = () => {};
+ const onTerm = () => ac.abort();
+ process.on("SIGINT", onInt);
+ process.on("SIGTERM", onTerm);
+ try {
+ return await runChildIntoLog(terminalLog, ac.signal, cmd);
+ } finally {
+ process.off("SIGINT", onInt);
+ process.off("SIGTERM", onTerm);
+ }
+}
+
+// --- publish build / deploy / hub / homepage ----------------------------------
+
+function knownSite(id: string, name: string, out: Out, paths?: Paths): boolean {
+ const known = listSiteIds(paths);
+ if (known.includes(id)) return true;
+ out.error(`${name}: no site "${id}" (configured: ${known.join(", ") || "none"})`);
+ return false;
+}
+
+async function run(req: StageRequest, signal: AbortSignal, paths?: Paths): Promise<number> {
+ setKillChildTrees(true);
+ return (await runStage(req, { paths, signal })).code;
+}
+
+export type BuildArgs = {
+ target: string; // a site id or "all"
+ runner?: "local" | "docker" | "auto";
+ force?: boolean;
+ skipArchives?: boolean;
+ allowMissingMedia?: boolean;
+ runId?: string;
+ paths?: Paths;
+ signal?: AbortSignal;
+};
+
+export async function publishBuild(a: BuildArgs, out: Out = console): Promise<number> {
+ const all = a.target === "all";
+ if (!all && !knownSite(a.target, "publish build", out, a.paths)) return STAGE_EXIT.usage;
+ const signal = a.signal ?? interrupted();
+ let runner: "local" | "docker" = "local";
+ if (a.runner === "docker" || a.runner === "auto") {
+ if (!all) {
+ out.error("publish build: --runner docker builds every site in containers — publish build all --runner docker");
+ return STAGE_EXIT.usage;
+ }
+ if (a.runner === "docker") runner = "docker";
+ else {
+ const { dockerAvailable } = await import("../publish/build");
+ runner = (await dockerAvailable(signal)) ? "docker" : "local";
+ if (runner === "local") out.log("[notice] No container engine available — building sites serially on the host.");
+ }
+ }
+ return run(
+ {
+ kind: "build-site",
+ target: all ? "_all" : a.target,
+ runId: a.runId ?? cliRunId(),
+ ...(all ? { runner } : {}),
+ ...(a.force ? { force: true } : {}),
+ ...(a.skipArchives ? { skipArchives: true } : {}),
+ ...(a.allowMissingMedia ? { allowMissingMedia: true } : {}),
+ },
+ signal,
+ a.paths,
+ );
+}
+
+export type DeployArgs = {
+ target: string; // a site id or "all"
+ preview?: string;
+ to?: "pages" | "local";
+ force?: boolean;
+ runId?: string;
+ paths?: Paths;
+ signal?: AbortSignal;
+};
+
+export async function publishDeploy(a: DeployArgs, out: Out = console): Promise<number> {
+ const all = a.target === "all";
+ if (!all && !knownSite(a.target, "publish deploy", out, a.paths)) return STAGE_EXIT.usage;
+ const signal = a.signal ?? interrupted();
+ const runId = a.runId ?? cliRunId();
+ // `all` passes over — quietly, one line — only the sites that are never
+ // deployable by their configuration: a private site, and (to Pages) a site
+ // with no Pages project. Every other refusal (never built, a bundle
+ // problem, a build not of main, a build this run has not made) is a
+ // FAILURE: the rest are still tried, and the run exits 1.
+ const ids: string[] = [];
+ for (const site of all ? listSites(a.paths) : []) {
+ const never = siteDeployProblem(site) ?? (a.to === "local" || site.cloudflareProject?.trim() ? null : "no Cloudflare Pages project");
+ if (never) out.log(`[publish] ${site.siteId}: skipped — ${never}`);
+ else ids.push(site.siteId);
+ }
+ if (!all) ids.push(a.target);
+ let worst = 0;
+ for (const id of ids) {
+ if (signal.aborted) return STAGE_EXIT.cancelled;
+ const code = await run(
+ {
+ kind: "deploy-site",
+ target: id,
+ runId,
+ ...(a.preview ? { preview: a.preview } : {}),
+ ...(a.to ? { to: a.to } : {}),
+ ...(a.force ? { force: true } : {}),
+ },
+ signal,
+ a.paths,
+ );
+ if (code === STAGE_EXIT.cancelled) return code;
+ if (code !== 0) worst = all ? STAGE_EXIT.failed : worst || code;
+ }
+ return worst;
+}
+
+export async function publishHub(
+ a: { deploy?: boolean; preview?: string; force?: boolean; runId?: string; paths?: Paths; signal?: AbortSignal },
+ out: Out = console,
+): Promise<number> {
+ if (a.preview && !a.deploy) {
+ out.error("publish hub: --preview is a deploy — add --deploy");
+ return STAGE_EXIT.usage;
+ }
+ const signal = a.signal ?? interrupted();
+ const runId = a.runId ?? cliRunId();
+ const built = await run({ kind: "build-hub", target: "_hub", runId, ...(a.force ? { force: true } : {}) }, signal, a.paths);
+ if (built !== 0 || !a.deploy) return built;
+ return run({ kind: "deploy-hub", target: "_hub", runId, ...(a.preview ? { preview: a.preview } : {}) }, signal, a.paths);
+}
+
+export async function publishHomepage(
+ a: {
+ deploy?: boolean;
+ preview?: string;
+ to?: "pages" | "local";
+ force?: boolean;
+ runId?: string;
+ paths?: Paths;
+ signal?: AbortSignal;
+ },
+ out: Out = console,
+): Promise<number> {
+ if ((a.preview || a.to) && !a.deploy) {
+ out.error("publish homepage: --preview and --to are a deploy's — add --deploy");
+ return STAGE_EXIT.usage;
+ }
+ const signal = a.signal ?? interrupted();
+ const runId = a.runId ?? cliRunId();
+ const built = await run(
+ { kind: "build-homepage", target: "_homepage", runId, ...(a.force ? { force: true } : {}) },
+ signal,
+ a.paths,
+ );
+ if (built !== 0 || !a.deploy) return built;
+ return run(
+ {
+ kind: "deploy-homepage",
+ target: "_homepage",
+ runId,
+ ...(a.preview ? { preview: a.preview } : {}),
+ ...(a.to ? { to: a.to } : {}),
+ },
+ signal,
+ a.paths,
+ );
+}
+
+// --- the old rows, as printed aliases -------------------------------------------
+
+export const ALIASES = {
+ buildSite: (id: string, nodata: boolean) =>
+ `${nodata ? "" : "archilyzer publish index && "}archilyzer publish build ${id} --force`,
+ buildAll: "archilyzer publish index && archilyzer publish build all --runner auto",
+ deploySite: (id: string, preview?: string) =>
+ `archilyzer publish deploy ${id}${preview ? ` --preview ${preview}` : ""}`,
+} as const;
+
+/** `build site <id> [--nodata]` = `publish index` (skipped by --nodata) + `publish build <id> --force`. */
+export async function buildSiteAlias(
+ a: { siteId: string; nodata: boolean; skipArchives: boolean; allowMissingMedia: boolean },
+ out: Out = console,
+): Promise<number> {
+ if (!knownSite(a.siteId, "build site", out)) return STAGE_EXIT.usage;
+ out.log(`[alias] build site is now: ${ALIASES.buildSite(a.siteId, a.nodata)}`);
+ const runId = cliRunId();
+ if (!a.nodata) {
+ const code = await publishIndex({ runId });
+ if (code !== 0) return code;
+ }
+ return publishBuild(
+ {
+ target: a.siteId,
+ force: true,
+ skipArchives: a.skipArchives,
+ allowMissingMedia: a.allowMissingMedia,
+ runId,
+ },
+ out,
+ );
+}
+
+/** `build all` = `publish index` + `publish build all --runner auto`. */
+export async function buildAllAlias(a: { skipArchives: boolean }, out: Out = console): Promise<number> {
+ out.log(`[alias] build all is now: ${ALIASES.buildAll}`);
+ const runId = cliRunId();
+ const code = await publishIndex({ runId });
+ if (code !== 0) return code;
+ return publishBuild({ target: "all", runner: "auto", skipArchives: a.skipArchives, runId }, out);
+}
+
+/** `deploy site <id>` = `publish deploy <id>`. */
+export async function deploySiteAlias(
+ a: { siteId: string; preview?: string },
+ out: Out = console,
+): Promise<number> {
+ out.log(`[alias] deploy site is now: ${ALIASES.deploySite(a.siteId, a.preview)}`);
+ return publishDeploy({ target: a.siteId, preview: a.preview }, out);
+}
diff --git a/common/jobs/runChild.test.ts b/common/jobs/runChild.test.ts
@@ -0,0 +1,98 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { mkdtempSync, readFileSync, rmSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { killChildTreesNow, killsChildTrees, runChildIntoLog, setKillChildTrees } from "./runChild";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// Release 18: a publish stage child turns on tree-kill, so a cancel takes the
+// child's whole process group — `next build`'s workers, wrangler — not just
+// the direct child. A shell that starts a background grandchild stands in.
+
+function alive(pid: number): boolean {
+ try {
+ process.kill(pid, 0);
+ return true;
+ } catch {
+ return false;
+ }
+}
+
+async function waitFor(cond: () => boolean, ms = 5_000): Promise<boolean> {
+ const end = Date.now() + ms;
+ while (Date.now() < end) {
+ if (cond()) return true;
+ await new Promise((r) => setTimeout(r, 25));
+ }
+ return cond();
+}
+
+async function runWithGrandchild(tree: boolean): Promise<{ grandchild: number; code: number }> {
+ const dir = mkdtempSync(path.join(os.tmpdir(), "run-child-"));
+ const pidFile = path.join(dir, "gc.pid");
+ setKillChildTrees(tree);
+ try {
+ const ac = new AbortController();
+ const running = runChildIntoLog(() => {}, ac.signal, {
+ command: "sh",
+ args: ["-c", `sleep 30 & echo $! > '${pidFile}'; wait`],
+ cwd: dir,
+ });
+ let grandchild = 0;
+ await waitFor(() => {
+ try {
+ grandchild = Number(readFileSync(pidFile, "utf8").trim());
+ return grandchild > 0;
+ } catch {
+ return false;
+ }
+ });
+ ac.abort();
+ const code = await running;
+ return { grandchild, code };
+ } finally {
+ setKillChildTrees(false);
+ rmSync(dir, { recursive: true, force: true });
+ }
+}
+
+test("tree-kill is off by default", () => {
+ assert.equal(killsChildTrees(), false);
+});
+
+test("with tree-kill on, a cancel takes the grandchildren too", { skip: process.platform === "win32" }, async () => {
+ const { grandchild, code } = await runWithGrandchild(true);
+ assert.notEqual(code, 0);
+ assert.ok(await waitFor(() => !alive(grandchild)), `grandchild ${grandchild} survived`);
+});
+
+test("killChildTreesNow (a second Ctrl-C) takes every live group down at once, no cancel needed", { skip: process.platform === "win32" }, async () => {
+ const dir = mkdtempSync(path.join(os.tmpdir(), "run-child-now-"));
+ const pidFile = path.join(dir, "gc.pid");
+ setKillChildTrees(true);
+ try {
+ const running = runChildIntoLog(() => {}, new AbortController().signal, {
+ command: "sh",
+ args: ["-c", `sleep 30 & echo $! > '${pidFile}'; wait`],
+ cwd: dir,
+ });
+ let grandchild = 0;
+ await waitFor(() => {
+ try {
+ grandchild = Number(readFileSync(pidFile, "utf8").trim());
+ return grandchild > 0;
+ } catch {
+ return false;
+ }
+ });
+ killChildTreesNow("SIGKILL");
+ assert.notEqual(await running, 0);
+ assert.ok(await waitFor(() => !alive(grandchild)), `grandchild ${grandchild} survived`);
+ } finally {
+ setKillChildTrees(false);
+ rmSync(dir, { recursive: true, force: true });
+ }
+});
diff --git a/common/jobs/runChild.ts b/common/jobs/runChild.ts
@@ -10,6 +10,44 @@ export type RunChildOpts = {
label?: string;
};
+// Release 18: a publish STAGE CHILD (and `archilyzer publish …`) is a process of
+// its own whose children run whole trees — `pnpm exec next build` and its
+// workers, wrangler, docker. Killing only the direct child leaves the rest
+// running after a Cancel. With tree-kill on, each child is spawned as the
+// leader of its own process group, and a cancel signals the whole GROUP. Off
+// by default, so the editor's own in-process jobs are unchanged; a stage child
+// turns it on once at start. Per process, on globalThis like the registry.
+const TREE_KILL_KEY = Symbol.for("archilyzer.runChild.treeKill");
+type TreeKillGlobal = { [TREE_KILL_KEY]?: boolean };
+
+export function setKillChildTrees(on: boolean): void {
+ (globalThis as TreeKillGlobal)[TREE_KILL_KEY] = on;
+}
+
+export function killsChildTrees(): boolean {
+ return (globalThis as TreeKillGlobal)[TREE_KILL_KEY] === true;
+}
+
+// The process groups this process leads right now (tree-kill mode), so a
+// second Ctrl-C / SIGTERM — which exits at once, without waiting for the
+// cancel to unwind — can still take them down first (`killChildTreesNow`).
+const LIVE_GROUPS_KEY = Symbol.for("archilyzer.runChild.liveGroups");
+function liveGroups(): Set<number> {
+ const g = globalThis as { [LIVE_GROUPS_KEY]?: Set<number> };
+ return (g[LIVE_GROUPS_KEY] ??= new Set());
+}
+
+/** Signal every live child process group this process started (tree-kill mode). */
+export function killChildTreesNow(sig: NodeJS.Signals = "SIGKILL"): void {
+ for (const pgid of liveGroups()) {
+ try {
+ process.kill(-pgid, sig);
+ } catch {
+ // Already gone.
+ }
+ }
+}
+
// Spawn a child process and stream its combined stdout/stderr into `onLog`,
// resolving with the exit code. Used inside a managed function's `fn` to run a
// sub-command as part of a larger job (build-then-deploy, multi-phase builds)
@@ -25,13 +63,28 @@ export async function runChildIntoLog(
opts: RunChildOpts,
): Promise<number> {
const prefix = opts.label ?? "";
+ const tree = killsChildTrees();
const child: ResultPromise = execa(opts.command, opts.args, {
cwd: opts.cwd,
env: opts.env,
all: true,
buffer: false,
reject: false,
+ ...(tree ? { detached: true } : {}),
});
+ if (tree && child.pid) liveGroups().add(child.pid);
+ // Signal the child — or, with tree-kill, its whole process group.
+ const signalChild = (sig: NodeJS.Signals) => {
+ if (tree && child.pid) {
+ try {
+ process.kill(-child.pid, sig);
+ return;
+ } catch {
+ // The group is gone (or was never made): the child alone.
+ }
+ }
+ child.kill(sig);
+ };
// Line-buffer: execa chunks aren't line-aligned, and the managed-function
// onLog appends a newline per call, so emitting raw chunks would inject
@@ -55,9 +108,10 @@ export async function runChildIntoLog(
let killTimer: ReturnType<typeof setTimeout> | null = null;
const onAbort = () => {
- child.kill("SIGTERM");
+ signalChild("SIGTERM");
killTimer = setTimeout(() => {
- if (child.killed === false) child.kill("SIGKILL");
+ if (tree) signalChild("SIGKILL");
+ else if (child.killed === false) child.kill("SIGKILL");
}, 5_000);
killTimer.unref?.();
};
@@ -73,6 +127,9 @@ export async function runChildIntoLog(
buf = "";
}
if (killTimer) clearTimeout(killTimer);
+ // The leader is gone; whatever of its group a cancel left is not wanted.
+ if (tree && signal.aborted) signalChild("SIGKILL");
+ if (tree && child.pid) liveGroups().delete(child.pid);
signal.removeEventListener("abort", onAbort);
}
}
diff --git a/common/lib/dirSignature.ts b/common/lib/dirSignature.ts
@@ -0,0 +1,54 @@
+// A directory's cheap content signature — moved here, unchanged, from
+// compose-site.ts (release 18): the publish stages' `inputSig` is computed
+// with the SAME function compose uses to decide what to skip, so "a site is
+// fresh" and "compose would skip everything" can never disagree.
+
+import path from "node:path";
+import { readdir, stat } from "node:fs/promises";
+import { createHash } from "node:crypto";
+import type { Dirent } from "node:fs";
+
+// A cheap content signature for a directory: relative path + size + mtime of
+// every file within, hashed. It reflects exactly what a recursive copy would
+// move, so an unchanged source (build:index skipped it incrementally) yields the
+// same signature and the copy is skipped. The shared transcript/subs trees hold
+// only a bounded handful of paginated page files per channel, so this is far
+// cheaper than the copy it guards. Returns "" when the dir is absent/empty.
+//
+// `ignoreBasename` excludes files by name from the signature. The shared
+// per-channel trees carry a manifest.json that build:index rewrites with a fresh
+// `generatedAt` for EVERY channel whenever ANY channel mutates (the shared
+// page-writer loop runs over all channels). Its pageCount/slugToPage only change
+// when the channel's pages change — which the page files already capture — so
+// excluding it lets an unchanged channel stay skipped on a partial-change build
+// instead of re-copying all 50-odd channels. The tiny stale manifest left behind
+// on a skip is structurally identical (only its timestamp differs).
+export async function dirSignature(
+ dir: string,
+ ignoreBasename?: string,
+): Promise<string> {
+ const h = createHash("sha1");
+ let any = false;
+ const walk = async (rel: string): Promise<void> => {
+ let ents: Dirent[];
+ try {
+ ents = await readdir(path.join(dir, rel), { withFileTypes: true });
+ } catch {
+ return;
+ }
+ ents.sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0));
+ for (const e of ents) {
+ if (!e.isDirectory() && e.name === ignoreBasename) continue;
+ const childRel = rel ? `${rel}/${e.name}` : e.name;
+ if (e.isDirectory()) {
+ await walk(childRel);
+ } else {
+ any = true;
+ const s = await stat(path.join(dir, childRel));
+ h.update(`${childRel}\t${s.size}\t${s.mtimeMs}\n`);
+ }
+ }
+ };
+ await walk("");
+ return any ? h.digest("hex") : "";
+}
diff --git a/common/lib/envVars.ts b/common/lib/envVars.ts
@@ -81,7 +81,7 @@ const DECLARED: EnvVarDecl[] = [
paths("ARIA2C_BIN", "`aria2c` on PATH", "Fetches an archive.org file over BitTorrent (the item's torrent, archive.org as web seed), then seeds it for a while. Optional: without it archive.org files are downloaded directly. See `archiveOrg` in [SETTINGS.md](SETTINGS.md)."),
paths("OLLAMA_URL", "`http://127.0.0.1:11434`", "The local ollama server, the local digest and attribution engine."),
paths("CLAUDE_BIN", "`claude` on PATH", "The `claude` CLI, driving the opt-in metered digest lane."),
- paths("ARCHILYZER_CONFIG_DIR", "`~/.config/archilyzer`", "The operator's private config dir, outside the repo: the two inputs of `archilyzer source publish` below. Never committed."),
+ paths("ARCHILYZER_CONFIG_DIR", "`~/.config/archilyzer`", "The operator's private config dir, outside the repo: the two inputs of `archilyzer source publish` below. Never committed. In Docker it is `/data/config/archilyzer`, in the config volume (docker-compose.yml)."),
paths("SOURCE_SCRUB_FILE", "`<ARCHILYZER_CONFIG_DIR>/source-scrub.txt`", "git-filter-repo `lhs==>rhs` rules applied to file contents AND commit messages when the source mirror is generated (`<home dir>==>/home/user` is built in and runs first). Every rule's left side is also denied. See [PUBLISH.md](PUBLISH.md)."),
paths("SOURCE_DENYLIST_FILE", "`<ARCHILYZER_CONFIG_DIR>/source-denylist.txt`", "Literals the published source must never contain, one per line (`i:` = any case). One hit anywhere in the mirror, the tree or the tarball refuses the publish."),
paths("ARCHILYZER_SOURCE_SCRATCH", "the OS temp dir", "Where `source publish` makes its scratch clone and stage (removed afterwards unless `--keep-scratch`)."),
@@ -93,9 +93,16 @@ const DECLARED: EnvVarDecl[] = [
{ name: "SYNC_HEARTBEAT_SECONDS", audience: "runtime", default: "`settings.syncScheduler.heartbeatSeconds`", readBy: "editor/app/scheduler/heartbeat.ts", doc: "Overrides the editor's in-process sync heartbeat. `0` = no internal timer (tick from cron instead)." },
{ name: "SYNC_TICK_URL", audience: "runtime", default: "`http://127.0.0.1:3001/api/scheduler/tick`", readBy: "common/bin/sync-tick.ts", doc: "Where `archilyzer sync tick` (cron's heartbeat) posts." },
{ name: "SYNC_TICK_TOKEN", audience: "runtime", default: "unset (no auth)", readBy: "common/bin/sync-tick.ts, editor/app/scheduler/auth.ts", doc: "Bearer token for the tick endpoint; set on both the editor and the cron job." },
- { name: "R2_ACCESS_KEY_ID", audience: "runtime", default: "—", readBy: "common/publish/build.ts", doc: "R2 S3 credentials for uploading oversize archives at deploy time (with `R2_SECRET_ACCESS_KEY` and `CLOUDFLARE_ACCOUNT_ID`). See [PUBLISH.md](PUBLISH.md)." },
- { name: "R2_SECRET_ACCESS_KEY", audience: "runtime", default: "—", readBy: "common/publish/build.ts", doc: "See `R2_ACCESS_KEY_ID`." },
- { name: "CLOUDFLARE_ACCOUNT_ID", audience: "runtime", default: "—", readBy: "common/publish/build.ts", doc: "The account the R2 endpoint belongs to. wrangler reads its own credentials." },
+ { name: "R2_ACCESS_KEY_ID", audience: "runtime", default: "—", readBy: "common/publish/build.ts, common/bin/doctor.ts (set or not)", doc: "R2 S3 credentials for uploading oversize archives at deploy time (with `R2_SECRET_ACCESS_KEY` and `CLOUDFLARE_ACCOUNT_ID`), needed only when `archiveStorage.bucket` is set. In Docker they come from `.env`. See [PUBLISH.md](PUBLISH.md)." },
+ { name: "R2_SECRET_ACCESS_KEY", audience: "runtime", default: "—", readBy: "common/publish/build.ts, common/bin/doctor.ts (set or not)", doc: "See `R2_ACCESS_KEY_ID`." },
+ { name: "CLOUDFLARE_ACCOUNT_ID", audience: "runtime", default: "—", readBy: "common/publish/build.ts, wrangler, common/bin/doctor.ts (set or not)", doc: "The Cloudflare account: the R2 endpoint's, and the one wrangler deploys to when the token can see more than one. In Docker it comes from `.env`." },
+ { name: "CLOUDFLARE_API_TOKEN", audience: "runtime", default: "unset (wrangler's own `wrangler login` config, on a host)", readBy: "wrangler (every deploy), common/bin/doctor.ts, common/lib/pagesDeploy.ts (set or not, never the value)", doc: "The API token every deploy's wrangler authenticates with (Cloudflare Pages: Edit). The way a container deploys — there is no browser for `wrangler login` in one; set it in `.env`." },
+ { name: "ARCHILYZER_HOST_ID", audience: "runtime", default: "the hostname", readBy: "common/publish/stageLock.ts (the publish lock)", doc: "Which host the publish lock (`<EXPORT_BUILDS_DIR>/.publish.lock`) names as its holder's: a lock from this host whose pid is dead is stale and taken over; another host's is waited on. docker-compose.yml fixes it for the editor (`archilyzer-editor`), whose hostname is a container id that changes on every recreate." },
+ { name: "ARCHILYZER_SOURCE_REPO", audience: "runtime", default: "this checkout's git common dir", readBy: "common/publish/source.ts, common/bin/doctor.ts, docker/entrypoint.sh", doc: "The git DIR `archilyzer source publish` mirrors `main` from, when the checkout has none: in Docker, `/data/source.git`, the host's git common dir mounted read-only by docker-compose.source.yml. A value that names nothing refuses the publish." },
+ { name: "YTDLP_SOURCE_HOST_DIR", audience: "runtime", default: "— (required by the overlay)", readBy: "docker-compose.ytdlp.yml", doc: "Docker: the HOST path of a yt-dlp source checkout (the directory holding `yt_dlp/`), mounted read-only at `/opt/yt-dlp-src` by docker-compose.ytdlp.yml. See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md), \"Substituting yt-dlp\"." },
+ { name: "YTDLP_AUTO_UPDATE", audience: "runtime", default: "off", readBy: "docker/entrypoint.sh, common/bin/doctor.ts", doc: "Docker: `1` runs `yt-dlp -U` on every editor boot — on the image's yt-dlp only; an override (`YTDLP_BIN` naming another) is left alone, with a warning." },
+ { name: "XDG_CONFIG_HOME", audience: "runtime", default: "`~/.config`", readBy: "common/bin/doctor.ts, common/lib/pagesDeploy.ts", doc: "Where `wrangler login` keeps its config (`<it>/.wrangler/config/default.toml`); the doctor looks for it there, by path, never reading it." },
+ { name: "YTDLP_SOURCE_DIR", audience: "runtime", default: "`/opt/yt-dlp-src`", readBy: "docker/yt-dlp-from-source.sh", doc: "Docker: where `/usr/local/bin/yt-dlp-from-source` finds the yt-dlp source tree it runs with the image's python." },
{ name: "DOCKER_BIN", audience: "runtime", default: "`docker`", readBy: "common/publish/build.ts", doc: "The container engine for docker-mode builds (e.g. `podman`)." },
{ name: "DOCKER_BUILD_MEMORY", audience: "runtime", default: "no cap", readBy: "common/publish/build.ts", doc: "Per-container memory cap for a docker-mode build (`--memory`)." },
{ name: "DOCKER_BUILD_CPUS", audience: "runtime", default: "no cap", readBy: "common/publish/build.ts", doc: "Per-container CPU cap for a docker-mode build (`--cpus`)." },
@@ -103,11 +110,6 @@ const DECLARED: EnvVarDecl[] = [
{ name: "HOST", audience: "runtime", default: "every interface", readBy: "export/scripts/serve-out.mjs", doc: "The address `pnpm start:export` (serve-out) listens on; `127.0.0.1` keeps a private site on this machine." },
{ name: "MAX_ARCHIVE_BYTES", audience: "runtime", default: "the Cloudflare-safe cap", readBy: "common/bin/compose-site.ts", doc: "The served-file size cap for archives, in bytes; `0` = no cap. A site's own `archiveMaxBytes` wins." },
{ name: "WRANGLER_BIN", audience: "runtime", default: "`common/node_modules/.bin/wrangler` (the pinned devDependency)", readBy: "common/lib/pagesDeploy.ts (wranglerBin), common/publish/deployStage.ts", doc: "The wrangler a deploy spawns. The editor's e2e suite points it at its fake." },
- // Release 18 S2, until S5 is merged: S5 declares these two itself (the
- // credential and wrangler-login rows); on that merge keep S5's rows, delete
- // these, and add common/lib/pagesDeploy.ts to their readBy.
- { name: "CLOUDFLARE_API_TOKEN", audience: "runtime", default: "unset (a `wrangler login` on a host)", readBy: "wrangler (every deploy), common/lib/pagesDeploy.ts (set or not, never the value)", doc: "The API token every deploy runs wrangler with. With none and no `wrangler login`, a deploy is refused before wrangler runs; one Cloudflare rejects ends the deploy on \"REFUSED by Cloudflare\". Never printed." },
- { name: "XDG_CONFIG_HOME", audience: "runtime", default: "`~/.config`", readBy: "common/lib/pagesDeploy.ts", doc: "Where `wrangler login` keeps its config (`<it>/.wrangler/config/default.toml`); a deploy's credential check looks for it there, by path." },
{ name: "CHOUGH_BIN", audience: "runtime", default: "`chough` on PATH", readBy: "common/lib/transcriptionApps.ts", doc: "The chough transcription engine, when a worker names no binary." },
{ name: "CHOUGH_MODEL", audience: "runtime", default: "chough's own", readBy: "chough (set by common/lib/transcriptionApps.ts)", doc: "Passed to chough from a worker's model field; chough auto-downloads one when unset." },
{ name: "CHOUGH_URL", audience: "runtime", default: "local", readBy: "chough (set by common/lib/transcriptionApps.ts)", doc: "Passed to chough from a worker's remote-server field." },
@@ -158,7 +160,12 @@ const DECLARED: EnvVarDecl[] = [
{ name: "ARCHILYZER_FETCH_MODEL", audience: "docker", default: "per transcriber", readBy: "docker/entrypoint.sh", doc: "Which model the first boot downloads; `none` skips it." },
{ name: "ARCHILYZER_MODELS_DIR", audience: "docker", default: "`/data/models`", readBy: "docker/entrypoint.sh", doc: "Where models live in the container." },
{ name: "ARCHILYZER_BUILDS_DIR", audience: "docker", default: "`/data/builds`", readBy: "docker/entrypoint.sh", doc: "Where the container keeps built sites." },
- { name: "ARCHILYZER_SITE_OUT", audience: "docker", default: "`/data/builds/site`", readBy: "docker/entrypoint.sh, docker/publish-site.sh", doc: "The built export site the `site` service serves." },
+ { name: "ARCHILYZER_SITE_OUT", audience: "docker", default: "`/data/builds/site`", readBy: "docker/entrypoint.sh, docker/publish-site.sh, common/publish/deployStage.ts", doc: "The built export site the `site` service serves." },
+ { name: "ARCHILYZER_HOMEPAGE_OUT", audience: "docker", default: "`/data/builds/homepage`", readBy: "docker/entrypoint.sh, common/publish/deployStage.ts", doc: "The locally deployed homepage (`publish homepage --deploy --to local`). The `homepage` service serves it when it is non-empty, else the image's baked build." },
+ { name: "ARCHILYZER_IMAGE_YTDLP", audience: "docker", default: "baked: `/usr/local/bin/yt-dlp`", readBy: "docker/entrypoint.sh, common/bin/doctor.ts", doc: "The yt-dlp the image ships. `YTDLP_BIN` naming anything else is an OVERRIDE: the boot's `yt-dlp:` line and `archilyzer doctor` say so, and `YTDLP_AUTO_UPDATE` leaves it alone." },
+ { name: "ARCHILYZER_COMMIT", audience: "docker", default: "baked: empty unless the build passed it", readBy: "the Dockerfile (a build arg), the publish stamps (`IMAGE_COMMIT_ENV`)", doc: "The commit the image was built from — the publish stamps' `commit` where there is no .git. `ARCHILYZER_COMMIT=$(git rev-parse HEAD) docker compose build`." },
+ { name: "ARCHILYZER_BRANCH", audience: "docker", default: "baked: empty unless the build passed it", readBy: "the Dockerfile (a build arg), the publish stamps (`IMAGE_BRANCH_ENV`)", doc: "The branch the image was built from — the stamps' `branch`, which a production deploy checks." },
+ { name: "ARCHILYZER_SOURCE_HOST_DIR", audience: "docker", default: "`./.git`", readBy: "docker-compose.source.yml", doc: "The HOST's git common dir docker-compose.source.yml mounts at `/data/source.git`. In a git worktree, the primary checkout's `.git`." },
{ name: "ARCHILYZER_IDLE_BOOT", audience: "docker", default: "off", readBy: "common/lib/idleBoot.ts (the editor)", doc: "`1` boots the editor without arming the heartbeat or any auto-queue runner." },
{ name: "ARCHILYZER_AUTH_MODE", audience: "docker", default: "`basic`", readBy: "docker/guard-exposure.sh, docker/caddy-start.sh", doc: "`basic`, `forward` or `none` — the only escape hatch from the exposure guard." },
{ name: "ARCHILYZER_AUTH_USER", audience: "docker", default: "`archilyzer`", readBy: "docker/Caddyfile", doc: "Basic-auth user." },
@@ -167,9 +174,6 @@ const DECLARED: EnvVarDecl[] = [
{ name: "ARCHILYZER_FORWARD_AUTH_UPSTREAM", audience: "docker", default: "—", readBy: "docker/Caddyfile", doc: "Forward-auth server (Authelia, tinyauth, …), `host:port`." },
{ name: "ARCHILYZER_FORWARD_AUTH_URI", audience: "docker", default: "`/api/auth/caddy`", readBy: "docker/Caddyfile", doc: "The forward-auth server's verify path." },
{ name: "ARCHILYZER_TAG", audience: "docker", default: "`local`", readBy: "docker-compose*.yml", doc: "The image tag the compose files build and run." },
- // Release 18 S2, until S5 is merged: S5 declares it (with docker/entrypoint.sh);
- // keep S5's row, delete this one, add common/publish/deployStage.ts to its readBy.
- { name: "ARCHILYZER_HOMEPAGE_OUT", audience: "docker", default: "`/data/builds/homepage`", readBy: "common/publish/deployStage.ts", doc: "The locally deployed homepage: `deploy homepage --to local` copies its bundle here." },
// ── test: harnesses, fakes and test-mode branches ──────────────────────
{ name: "E2E_TEST_ROUTES", audience: "test", default: "off", readBy: "editor/app/api/test/_guard.ts, editor/instrumentation.ts", doc: "`1` opens the editor's `/api/test/*` routes and marks a test server at boot. Set by `editor/playwright.config.ts` on its test server, and by nothing else." },
@@ -227,10 +231,21 @@ export const ENV_AUDIENCES: ReadonlyArray<{ id: EnvAudience; title: string; intr
{ id: "runtime", title: "Runtime", intro: "Tokens, credentials and knobs a running process reads. Most configuration is not here but in `settings.json` ([SETTINGS.md](SETTINGS.md))." },
{ id: "port", title: "Ports", intro: "Every local server's default port, from `common/lib/ports.mjs`. The primary checkout uses these; worktree N adds N × 100 (`pnpm wt list`)." },
{ id: "internal", title: "Set by the pipeline", intro: "The publish pipeline sets these for a process it spawns. Listed so a reader knows what they are; nobody sets them by hand." },
- { id: "docker", title: "Docker", intro: "The container's own set, read by `docker/*.sh`, the compose files and Caddy — not by the apps' code (except `ARCHILYZER_IDLE_BOOT`). See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md)." },
+ { id: "docker", title: "Docker", intro: "The container's own set, read by `docker/*.sh`, the compose files and Caddy — not by the apps' code (except `ARCHILYZER_IDLE_BOOT`, and the image's facts `archilyzer doctor` and the publish stamps read). See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md)." },
{ id: "test", title: "Tests only", intro: "Read only by a test harness, a fake binary or a test-mode branch. Never set one on a real instance." },
];
+// The two build facts the runtime image bakes (Dockerfile build args → ENV), by
+// name: the publish stamps' `commit`/`branch` where there is no .git to ask.
+export const IMAGE_COMMIT_ENV = "ARCHILYZER_COMMIT";
+export const IMAGE_BRANCH_ENV = "ARCHILYZER_BRANCH";
+
+// Those facts, or null each — an image built without the args bakes them empty.
+export function imageBuildFacts(env: NodeJS.ProcessEnv = process.env): { commit: string | null; branch: string | null } {
+ const v = (k: string) => env[k]?.trim() || null;
+ return { commit: v(IMAGE_COMMIT_ENV), branch: v(IMAGE_BRANCH_ENV) };
+}
+
export function envVar(name: string): EnvVarDecl | undefined {
return ENV_VARS.find((v) => v.name === name);
}
diff --git a/common/publish/__fixtures__/stamps.ts b/common/publish/__fixtures__/stamps.ts
@@ -0,0 +1,40 @@
+// Stamp builders shared by the publish tests (stamps, stages).
+import type { BuiltStamp, IndexStamp } from "../stamps";
+
+export function indexStamp(over: Partial<IndexStamp> = {}): IndexStamp {
+ return {
+ v: 1,
+ stampId: "s1",
+ generation: 7,
+ scannedAt: 1_000,
+ builtAt: 2_000,
+ templatesAt: 1_900,
+ commit: "abc",
+ index: { shortCircuited: false, added: 1, changed: 2, removed: 0, heldChannels: [] },
+ stats: { shortCircuited: true, notIndexedYet: 0, notIndexable: 3 },
+ sites: { jer: { siteFp: "f1", statsFp: null, inputSig: "sig-jer" } },
+ hubSig: "hub-1",
+ ...over,
+ };
+}
+
+export function builtStamp(over: Partial<BuiltStamp> = {}): BuiltStamp {
+ return {
+ v: 1,
+ stampId: "b1",
+ target: "jer",
+ kind: "site",
+ indexStampId: "s1",
+ inputSig: "sig-jer",
+ builtAt: 3_000,
+ commit: "abc",
+ branch: "main",
+ runner: "local",
+ audience: "public",
+ corpusGeneratedAt: "2026-10-06T00:00:00.000Z",
+ files: 10,
+ bytes: 1234,
+ archivesStaged: 0,
+ ...over,
+ };
+}
diff --git a/common/publish/build.ts b/common/publish/build.ts
@@ -9,7 +9,7 @@
// forbids non-serializable args). Keep them as plain helpers.
import path from "node:path";
-import { mkdir, readdir, rm, stat } from "node:fs/promises";
+import { cp, lstat, mkdir, readdir, readFile, rename, rm, stat, symlink } from "node:fs/promises";
import { createReadStream, existsSync } from "node:fs";
import { S3Client, HeadObjectCommand } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
@@ -35,6 +35,7 @@ import {
import { getPaths, type Paths } from "../lib/paths";
import { getSettings } from "../lib/settings";
import { getSite, listSites, type Site } from "../lib/site";
+import { builtStampPath } from "./stamps";
// Where the basic (host) build writes the static bundle to deploy: the fixed
// export/out, composed one site at a time. The docker fan-out writes per-site
@@ -482,7 +483,7 @@ export async function dockerAvailable(signal: AbortSignal): Promise<boolean> {
// These are pool-wide: no SITE_ID, and no EXPORT_PUBLIC_DIR override so the
// shared index/staging land at their canonical export/.export-index location
// (exactly what the fan-out containers mount read-only).
-async function runHostScript(
+export async function runHostScript(
onLog: (line: string) => void,
signal: AbortSignal,
paths: Paths,
@@ -507,7 +508,7 @@ async function runHostScript(
// change re-runs only `COPY . .` onward, but a Dockerfile or lockfile change
// re-installs every dependency first, inside this job. `archilyzer doctor`
// warns ahead of that when the image is absent or older than the Dockerfile.
-async function ensureBuildImage(
+export async function ensureBuildImage(
onLog: (line: string) => void,
signal: AbortSignal,
paths: Paths,
@@ -529,7 +530,7 @@ async function ensureBuildImage(
// next/font/google, which fetches the site's fonts from Google at build time;
// `--network=none` would fail the build. Isolation still comes from the per-site
// output dir, the read-only shared mounts, and the non-root `-u` user.
-async function runDockerBuildOne(
+export async function runDockerBuildOne(
onLog: (line: string) => void,
signal: AbortSignal,
siteId: string,
@@ -557,6 +558,10 @@ async function runDockerBuildOne(
"-v", `${paths.exportIndexDir}:/data/export/.export-index:ro`,
"-v", `${siteDir}:/site`,
"-e", `SITE_ID=${siteId}`,
+ // The container's `build site` is `publish build --force` (release 18):
+ // its publish lock lives inside the container, never on the host's
+ // mount — the host's lock is held by the fan-out stage itself.
+ "-e", `EXPORT_BUILDS_DIR=${CONTAINER_BUILDS_DIR}`,
);
// Mount the host settings.json fresh (build config: archive storage, size caps)
// rather than relying on a possibly-stale copy — it is NOT baked into the image.
@@ -718,7 +723,7 @@ export async function runDockerDeployAllPhase(
// Bounded-concurrency map over a fixed work set, preserving input order in the
// results. No external dep; a fresh worker pulls the next index until exhausted.
-async function runWithConcurrency<T, R>(
+export async function runWithConcurrency<T, R>(
items: T[],
limit: number,
worker: (item: T) => Promise<R>,
@@ -797,7 +802,14 @@ export async function buildSite(
*/
export async function deploySite(
siteId: string,
- opts: PublishOpts & { previewBranch?: string } = {},
+ opts: PublishOpts & {
+ previewBranch?: string;
+ // The bundle to ship and where its oversize archives were staged. Default:
+ // export/out and the host staging dir (the editor's deploy action); the
+ // deploy-site stage passes the site's own bundle under exportBuildsDir.
+ outDir?: string;
+ stagingDir?: string;
+ } = {},
): Promise<void> {
const { paths, onLog, signal } = resolved(opts);
if (opts.previewBranch !== undefined) {
@@ -814,7 +826,7 @@ export async function deploySite(
`Site "${site.siteId}" has no Cloudflare Pages project configured.`,
);
}
- const outDir = resolveOutDir(site.siteId, paths);
+ const outDir = opts.outDir ?? resolveOutDir(site.siteId, paths);
const builtProblem = builtSiteProblem(outDir, site.siteId);
if (builtProblem) throw new Error(builtProblem);
// Before the R2 upload below: a bundle built private is never deployed.
@@ -831,7 +843,7 @@ export async function deploySite(
}
// Push oversize archives to R2 first, so the manifest URLs the Pages deploy
// publishes resolve immediately. No-op when R2 isn't configured.
- const uploadCode = await runArchiveUploadIntoLog(onLog, signal, site, paths);
+ const uploadCode = await runArchiveUploadIntoLog(onLog, signal, site, paths, opts.stagingDir);
if (signal.aborted) return;
if (uploadCode !== 0) {
throw new Error(`Archive R2 upload failed (exit ${uploadCode}).`);
@@ -1015,7 +1027,7 @@ async function runPagesDeployIntoLog(
* quietly on a cancel. No R2 step: the hub holds no archives.
*/
export async function deployHub(
- opts: PublishOpts & { previewBranch?: string } = {},
+ opts: PublishOpts & { previewBranch?: string; outDir?: string } = {},
): Promise<void> {
const { paths, onLog, signal } = resolved(opts);
if (opts.previewBranch !== undefined) {
@@ -1026,7 +1038,7 @@ export async function deployHub(
const project = getHomepageConfig(paths).cloudflareProject;
const projectProblem = hubProjectProblem(project);
if (projectProblem) throw new Error(projectProblem);
- const outDir = resolveOutDir("", paths);
+ const outDir = opts.outDir ?? resolveOutDir("", paths);
const builtProblem = builtHubProblem(outDir);
if (builtProblem) throw new Error(builtProblem);
if (branch) onLog(`=== Deploy hub (preview "${branch}") ===\n`);
@@ -1187,3 +1199,254 @@ export async function deployHomepage(
if (signal.aborted) return;
if (code !== 0) throw new Error(`Homepage deploy failed (exit ${code}).`);
}
+
+// ---------------------------------------------------------------------------
+// Per-target bundles (release 18). `<exportBuildsDir>/<target>/out` is THE
+// bundle every deploy of `target` ships, whichever runner built it: a site's
+// id, `_hub`, or (stamps only) `_homepage` — the homepage's bundle stays
+// homepage/out, where the source gate withdraws from.
+//
+// The local runner builds into the shared export/out (Next's `distDir` may not
+// leave the project, and `public/` is copied into `out/` — the plan's
+// "Decided"), then INSTALLS it: moved to `<target>/out.next`, swapped in
+// (`out → out.prev`, `out.next → out`, `out.prev` removed). A leftover
+// `out.next` is deleted first. On one filesystem the move is a rename; across
+// two (the container: export/out is an image layer, the builds dir a volume)
+// rename fails EXDEV and the bundle is copied, then the source removed.
+// export/out is then a SYMLINK to the bundle built last, so older readers and
+// `serve out` keep working; `next build` removes the link itself (an `rm` of
+// the path, recursive — never its target) before it writes a real out/.
+// ---------------------------------------------------------------------------
+
+// Inside the docker per-site build container: where `publish build` keeps its
+// lock (runDockerBuildOne passes it as EXPORT_BUILDS_DIR).
+export const CONTAINER_BUILDS_DIR = "/tmp/archilyzer-builds";
+
+/** `<exportBuildsDir>/<target>/out` — the target's bundle (= dockerSiteOutDir for a site). */
+export function bundleDir(paths: Pick<Paths, "exportBuildsDir">, target: string): string {
+ return path.join(paths.exportBuildsDir, target, "out");
+}
+
+/** export/out: where `next build` writes, and afterwards a link to the last bundle. */
+export function exportOutPath(paths: Pick<Paths, "exportDir">): string {
+ return path.join(paths.exportDir, "out");
+}
+
+// The filesystem calls a bundle move makes, injectable so the EXDEV path is
+// tested without two filesystems.
+export type BundleFs = {
+ rename: typeof rename;
+ cp: typeof cp;
+ rm: typeof rm;
+ mkdir: typeof mkdir;
+};
+const realBundleFs: BundleFs = { rename, cp, rm, mkdir };
+
+async function lexists(p: string): Promise<boolean> {
+ return lstat(p).then(
+ () => true,
+ () => false,
+ );
+}
+
+// Move a directory: rename, or across filesystems a copy then a remove.
+async function moveDir(src: string, dest: string, fsOps: BundleFs): Promise<"rename" | "copy"> {
+ try {
+ await fsOps.rename(src, dest);
+ return "rename";
+ } catch (err) {
+ if ((err as NodeJS.ErrnoException).code !== "EXDEV") throw err;
+ await fsOps.cp(src, dest, { recursive: true, verbatimSymlinks: true, preserveTimestamps: true });
+ await fsOps.rm(src, { recursive: true, force: true });
+ return "copy";
+ }
+}
+
+/**
+ * A crash between installBundle's two renames leaves `out.prev` and no `out`:
+ * the last bundle is whole, under the wrong name. Put it back. Asked first by
+ * every build and every install, before anything else touches the target.
+ * Answers whether it restored one.
+ */
+export async function recoverInterruptedInstall(
+ destOut: string,
+ fsOps: BundleFs = realBundleFs,
+): Promise<boolean> {
+ const prev = `${destOut}.prev`;
+ if ((await lexists(destOut)) || !(await lexists(prev))) return false;
+ await fsOps.rename(prev, destOut);
+ return true;
+}
+
+/**
+ * Install the build at `src` as the bundle `destOut` (see the section header):
+ * `src` is gone afterwards and `destOut` holds it. Answers how it moved.
+ */
+export async function installBundle(
+ src: string,
+ destOut: string,
+ fsOps: BundleFs = realBundleFs,
+): Promise<"rename" | "copy"> {
+ const next = `${destOut}.next`;
+ const prev = `${destOut}.prev`;
+ await recoverInterruptedInstall(destOut, fsOps);
+ await fsOps.mkdir(path.dirname(destOut), { recursive: true });
+ await fsOps.rm(next, { recursive: true, force: true });
+ const how = await moveDir(src, next, fsOps);
+ await fsOps.rm(prev, { recursive: true, force: true });
+ // Siblings: these two renames never cross a filesystem.
+ if (await lexists(destOut)) await fsOps.rename(destOut, prev);
+ await fsOps.rename(next, destOut);
+ await fsOps.rm(prev, { recursive: true, force: true });
+ return how;
+}
+
+/**
+ * Before a build: export/out must not still point at an older bundle, or a
+ * build that failed before `next build` reached it would leave that bundle
+ * readable at export/out. A link is removed; a real directory is left for
+ * `next build` to replace.
+ */
+export async function unlinkExportOut(paths: Pick<Paths, "exportDir">): Promise<void> {
+ const link = exportOutPath(paths);
+ const st = await lstat(link).catch(() => null);
+ if (st?.isSymbolicLink()) await rm(link, { force: true });
+}
+
+/** After a build: export/out becomes a (relative) link to `bundle`, replaced atomically. */
+export async function pointExportOutAt(paths: Pick<Paths, "exportDir">, bundle: string): Promise<void> {
+ const link = exportOutPath(paths);
+ const st = await lstat(link).catch(() => null);
+ if (st && !st.isSymbolicLink()) await rm(link, { recursive: true, force: true });
+ const tmp = `${link}.link-${process.pid}`;
+ await rm(tmp, { force: true });
+ await symlink(path.relative(path.dirname(link), bundle), tmp, "dir");
+ await rename(tmp, link);
+}
+
+/**
+ * The host compose stages a site's oversize archives beside export/public
+ * (`.r2-staging/<id>/archives`); the bundle's are `dockerSiteStagingDir` for
+ * both runners. Moves this build's (replacing the last build's) and answers
+ * how many were staged. A no-op where the two are one place.
+ */
+export async function stageSiteArchives(
+ paths: Pick<Paths, "exportPublicDir" | "exportBuildsDir">,
+ siteId: string,
+ fsOps: BundleFs = realBundleFs,
+): Promise<number> {
+ const src = path.join(path.dirname(paths.exportPublicDir), ".r2-staging", siteId);
+ const dest = path.join(paths.exportBuildsDir, siteId, ".r2-staging", siteId);
+ if (path.resolve(src) !== path.resolve(dest)) {
+ await fsOps.rm(dest, { recursive: true, force: true });
+ if (await lexists(src)) {
+ await fsOps.mkdir(path.dirname(dest), { recursive: true });
+ await moveDir(src, dest, fsOps);
+ }
+ }
+ const staged = await readdir(dockerSiteStagingDir(paths as Paths, siteId)).catch(() => [] as string[]);
+ return staged.filter((f) => !f.startsWith(".")).length;
+}
+
+/** Files and bytes under `dir` (links not followed). */
+export async function bundleCounts(dir: string): Promise<{ files: number; bytes: number }> {
+ let files = 0;
+ let bytes = 0;
+ const walk = async (d: string): Promise<void> => {
+ const ents = await readdir(d, { withFileTypes: true }).catch(() => []);
+ for (const e of ents) {
+ const p = path.join(d, e.name);
+ if (e.isDirectory()) await walk(p);
+ else if (e.isFile()) {
+ files++;
+ bytes += (await stat(p)).size;
+ }
+ }
+ };
+ await walk(dir);
+ return { files, bytes };
+}
+
+/** The bundle's corpus.json `generatedAt`, or null. */
+export async function corpusGeneratedAtIn(outDir: string): Promise<string | null> {
+ try {
+ const v = JSON.parse(await readFile(path.join(outDir, "corpus.json"), "utf8")) as { generatedAt?: unknown };
+ return typeof v.generatedAt === "string" ? v.generatedAt : null;
+ } catch {
+ return null;
+ }
+}
+
+/**
+ * Build one site and install it as its bundle (the build-site stage's local
+ * body): export/out unlinked, buildSiteSteps with `skipData` (the index stage
+ * ran the data phase), the cited/scope checks of runBuildPhase,
+ * builtBundleProblem, then the install, the archive staging and the link.
+ * Returns the exit code; on 0 the bundle is `bundleDir(paths, siteId)`.
+ *
+ * `inPlace` (the docker per-site container) stops after the checks: the
+ * container hands export/out back itself, and the host stamps it.
+ */
+export async function buildSiteBundle(
+ siteId: string,
+ opts: PublishOpts & { skipArchives?: boolean; allowMissingMedia?: boolean; inPlace?: boolean } = {},
+): Promise<{ code: number; archivesStaged: number }> {
+ const { paths, onLog, signal } = resolved(opts);
+ if (!opts.inPlace) {
+ if (await recoverInterruptedInstall(bundleDir(paths, siteId))) {
+ onLog(`[build] ${siteId}: restored the bundle an interrupted install left as out.prev\n`);
+ }
+ await unlinkExportOut(paths);
+ }
+ const code = await runBuildPhase(onLog, signal, siteId, paths, {
+ skipData: true,
+ skipArchives: opts.skipArchives,
+ allowMissingMedia: opts.allowMissingMedia,
+ });
+ if (code !== 0 || signal.aborted) return { code: code || 1, archivesStaged: 0 };
+ const out = exportOutPath(paths);
+ const problem = builtBundleProblem(out, siteId);
+ if (problem) {
+ onLog(`[build] REFUSED — ${problem}.\n`);
+ return { code: 1, archivesStaged: 0 };
+ }
+ if (opts.inPlace) return { code: 0, archivesStaged: 0 };
+ // The old stamp must never describe the new bundle: it goes first, and the
+ // stage writes the new one as its LAST step, once the bundle is in place.
+ await rm(builtStampPath(paths, siteId), { force: true });
+ const how = await installBundle(out, bundleDir(paths, siteId));
+ const archivesStaged = await stageSiteArchives(paths, siteId);
+ await pointExportOutAt(paths, bundleDir(paths, siteId));
+ onLog(
+ `[build] ${siteId}: bundle installed at ${bundleDir(paths, siteId)} (${how === "rename" ? "moved" : "copied across filesystems"})` +
+ (archivesStaged ? `, ${archivesStaged} archive(s) staged for R2` : "") +
+ "\n",
+ );
+ return { code: 0, archivesStaged };
+}
+
+/**
+ * Build the hub and install it as `_hub/out` (the build-hub stage's body).
+ * Returns the exit code.
+ */
+export async function buildHubBundle(opts: PublishOpts = {}): Promise<number> {
+ const { paths, onLog, signal } = resolved(opts);
+ if (await recoverInterruptedInstall(bundleDir(paths, "_hub"))) {
+ onLog("[build] hub: restored the bundle an interrupted install left as out.prev\n");
+ }
+ await unlinkExportOut(paths);
+ const code = await buildHub({ paths, onLog, signal });
+ if (code !== 0 || signal.aborted) return code || 1;
+ const out = exportOutPath(paths);
+ const problem = builtHubProblem(out);
+ if (problem) {
+ onLog(`[build] REFUSED — ${problem}.\n`);
+ return 1;
+ }
+ const dest = bundleDir(paths, "_hub");
+ await rm(builtStampPath(paths, "_hub"), { force: true });
+ const how = await installBundle(out, dest);
+ await pointExportOutAt(paths, dest);
+ onLog(`[build] hub: bundle installed at ${dest} (${how === "rename" ? "moved" : "copied across filesystems"})\n`);
+ return 0;
+}
diff --git a/common/publish/buildImage.test.ts b/common/publish/buildImage.test.ts
@@ -1,7 +1,9 @@
// The container build's contract with the repo: what Dockerfile.build's
// image bakes into export/public and how docker/build-site.sh puts it in front
-// of each site's `next build`. No docker here — the invariant is read from git,
-// and the shell function is read out of build-site.sh and run with bash.
+// of each site's `next build`; and what the runtime image (the root Dockerfile)
+// must agree with the code about. No docker here — the invariant is read from
+// git or the Dockerfile's text, and the shell function is read out of
+// build-site.sh and run with bash.
//
// Run with:
// pnpm --filter yt-dlp-transcript-common test
@@ -13,6 +15,7 @@ import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync
import { tmpdir } from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
+import { FILTER_REPO_PIPX_SPEC } from "./source";
const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
@@ -88,3 +91,85 @@ test("build-site.sh's asset sync ships a changed svg, drops a removed one, and t
rmSync(root, { recursive: true, force: true });
}
});
+
+// The runtime image (the root Dockerfile) installs git-filter-repo with pipx so
+// a container can publish the homepage's /source mirror without a network fetch.
+// The version it bakes must be the one `source publish` would fetch with
+// `pipx run --spec` on a host (FILTER_REPO_PIPX_SPEC): the filter-repo version
+// is part of the publish's skip key, so a drift between the two is a rebuild
+// nobody asked for, or a mirror rewritten by a version nobody tested.
+test("the runtime image installs the git-filter-repo that source publish names, in every target", () => {
+ const dockerfile = readFileSync(path.join(REPO, "Dockerfile"), "utf8");
+ const installs = [...dockerfile.matchAll(/pipx install (\S+)/g)].map((m) => m[1]);
+ // runtime-base (runtime + runtime-vulkan) and runtime-cuda, which repeats it.
+ assert.equal(installs.length, 2, `expected two \`pipx install\` lines, found ${installs.length}`);
+ for (const spec of installs) {
+ assert.equal(
+ spec,
+ FILTER_REPO_PIPX_SPEC,
+ `the Dockerfile installs ${spec}, source.ts names ${FILTER_REPO_PIPX_SPEC} — move both together`,
+ );
+ }
+});
+
+// The yt-dlp substitution hook: the image ships its own yt-dlp as
+// ARCHILYZER_IMAGE_YTDLP beside YTDLP_BIN (the entrypoint and the doctor tell an
+// override by the difference), and links the from-source wrapper — in every
+// target.
+test("every runtime target names its own yt-dlp and links the from-source wrapper", () => {
+ const dockerfile = readFileSync(path.join(REPO, "Dockerfile"), "utf8");
+ const image = [...dockerfile.matchAll(/^\s+ARCHILYZER_IMAGE_YTDLP=(\S+) \\$/gm)].map((m) => m[1]);
+ const bin = [...dockerfile.matchAll(/^\s+YTDLP_BIN=(\S+) \\$/gm)].map((m) => m[1]);
+ assert.equal(image.length, 2);
+ assert.deepEqual(image, bin, "ARCHILYZER_IMAGE_YTDLP and YTDLP_BIN start equal in each target");
+ const links = dockerfile.match(/ln -s \/repo\/docker\/yt-dlp-from-source\.sh \/usr\/local\/bin\/yt-dlp-from-source/g) ?? [];
+ assert.equal(links.length, 2);
+ assert.ok(existsSync(path.join(REPO, "docker", "yt-dlp-from-source.sh")));
+});
+
+// Node's major, everywhere the image is built or run: the build stage
+// (NODE_IMAGE), the default runtime (RUNTIME_IMAGE), the Vulkan overlay's
+// runtime and runtime-cuda's nodesource major. The native modules are compiled
+// once against the build stage's ABI, so all of them must agree.
+function imageNodeMajors(): Record<string, number> {
+ const dockerfile = readFileSync(path.join(REPO, "Dockerfile"), "utf8");
+ const vulkan = readFileSync(path.join(REPO, "docker-compose.vulkan.yml"), "utf8");
+ const one = (re: RegExp, text: string, what: string) => {
+ const m = re.exec(text);
+ assert.ok(m, `${what} not found`);
+ return Number(m[1]);
+ };
+ const out: Record<string, number> = {
+ NODE_IMAGE: one(/^ARG NODE_IMAGE=node:(\d+)-/m, dockerfile, "ARG NODE_IMAGE=node:<major>-…"),
+ RUNTIME_IMAGE: one(/^ARG RUNTIME_IMAGE=node:(\d+)-/m, dockerfile, "ARG RUNTIME_IMAGE=node:<major>-…"),
+ NODE_MAJOR: one(/^ARG NODE_MAJOR=(\d+)$/m, dockerfile, "ARG NODE_MAJOR=<major> (runtime-cuda)"),
+ };
+ const vk = [...vulkan.matchAll(/RUNTIME_IMAGE: node:(\d+)-/g)].map((m) => Number(m[1]));
+ assert.ok(vk.length > 0, "docker-compose.vulkan.yml names a RUNTIME_IMAGE");
+ vk.forEach((v, i) => (out[`vulkan RUNTIME_IMAGE #${i + 1}`] = v));
+ return out;
+}
+
+test("every image target runs the Node major the build stage compiled the native modules for", () => {
+ const majors = imageNodeMajors();
+ assert.equal(new Set(Object.values(majors)).size, 1, `one Node major everywhere, found ${JSON.stringify(majors)}`);
+});
+
+// Every deploy runs the wrangler pinned in common's devDependencies from this
+// image, and wrangler refuses to start below its engines floor (4.x: >=22).
+// Skipped where wrangler is not installed.
+test("the image's Node major meets the pinned wrangler's engines floor", (t) => {
+ const pkg = path.join(REPO, "common", "node_modules", "wrangler", "package.json");
+ if (!existsSync(pkg)) {
+ t.skip("wrangler is not installed in common/node_modules");
+ return;
+ }
+ const engines = (JSON.parse(readFileSync(pkg, "utf8")) as { engines?: { node?: string } }).engines?.node ?? "";
+ const floor = /(\d+)/.exec(engines);
+ assert.ok(floor, `wrangler's engines.node (${JSON.stringify(engines)}) names a major`);
+ const major = imageNodeMajors().NODE_IMAGE;
+ assert.ok(
+ major >= Number(floor[1]),
+ `the image runs Node ${major}, and wrangler needs ${engines} — every deploy from the container would exit 1`,
+ );
+});
diff --git a/common/publish/bundle.test.ts b/common/publish/bundle.test.ts
@@ -0,0 +1,257 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import {
+ existsSync,
+ lstatSync,
+ mkdirSync,
+ mkdtempSync,
+ readFileSync,
+ readlinkSync,
+ rmSync,
+ symlinkSync,
+ writeFileSync,
+} from "node:fs";
+import { cp, mkdir, rename, rm } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import type { Paths } from "../lib/paths";
+import {
+ bundleCounts,
+ bundleDir,
+ corpusGeneratedAtIn,
+ dockerSiteOutDir,
+ exportOutPath,
+ installBundle,
+ pointExportOutAt,
+ recoverInterruptedInstall,
+ stageSiteArchives,
+ unlinkExportOut,
+ type BundleFs,
+} from "./build";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// The per-target bundle layout (release 18): a build in export/out is
+// installed as <exportBuildsDir>/<target>/out by a swap through out.next, the
+// move falls back to a copy across filesystems (an injected EXDEV here), and
+// export/out becomes a link to the bundle built last.
+
+function tmp(): { root: string; paths: Paths } {
+ const root = mkdtempSync(path.join(os.tmpdir(), "bundle-"));
+ const exportDir = path.join(root, "export");
+ mkdirSync(exportDir, { recursive: true });
+ return {
+ root,
+ paths: {
+ exportDir,
+ exportPublicDir: path.join(exportDir, "public"),
+ exportBuildsDir: path.join(exportDir, ".export-builds"),
+ } as Paths,
+ };
+}
+
+function writeBuild(dir: string, marker: string): void {
+ mkdirSync(path.join(dir, "_next"), { recursive: true });
+ writeFileSync(path.join(dir, "index.html"), marker);
+ writeFileSync(path.join(dir, "_next", "app.js"), "x".repeat(10));
+ writeFileSync(path.join(dir, "corpus.json"), JSON.stringify({ generatedAt: `at-${marker}` }));
+}
+
+// A filesystem whose renames out of `crossFrom` fail EXDEV, as export/out's do
+// in the container (an image layer) when the builds dir is a volume.
+function exdevFs(crossFrom: string, calls: string[]): BundleFs {
+ return {
+ rename: (async (a: string, b: string) => {
+ calls.push(`rename ${path.basename(String(a))} -> ${path.basename(String(b))}`);
+ if (String(a).startsWith(crossFrom)) {
+ throw Object.assign(new Error("EXDEV: cross-device link not permitted"), { code: "EXDEV" });
+ }
+ return rename(a, b);
+ }) as typeof rename,
+ cp: (async (a: string, b: string, o?: object) => {
+ calls.push(`cp ${path.basename(String(a))} -> ${path.basename(String(b))}`);
+ return cp(a, b, o);
+ }) as typeof cp,
+ rm,
+ mkdir,
+ };
+}
+
+test("bundleDir is <exportBuildsDir>/<target>/out — a site's is dockerSiteOutDir", () => {
+ const p = { exportBuildsDir: "/e/.export-builds", exportDir: "/e" } as Paths;
+ assert.equal(bundleDir(p, "jer"), "/e/.export-builds/jer/out");
+ assert.equal(bundleDir(p, "jer"), dockerSiteOutDir(p, "jer"));
+ assert.equal(bundleDir(p, "_hub"), "/e/.export-builds/_hub/out");
+ assert.equal(exportOutPath(p), "/e/out");
+});
+
+test("installBundle moves the build in through out.next, replacing the last bundle; a leftover out.next goes first", async () => {
+ const { root, paths } = tmp();
+ try {
+ const dest = bundleDir(paths, "jer");
+ writeBuild(dest, "old");
+ writeBuild(`${dest}.next`, "leftover");
+ const out = exportOutPath(paths);
+ writeBuild(out, "new");
+ assert.equal(await installBundle(out, dest), "rename");
+ assert.equal(readFileSync(path.join(dest, "index.html"), "utf8"), "new");
+ assert.ok(!existsSync(out), "export/out moved away");
+ assert.ok(!existsSync(`${dest}.next`));
+ assert.ok(!existsSync(`${dest}.prev`));
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+test("installBundle copies across filesystems (EXDEV), then removes the source", async () => {
+ const { root, paths } = tmp();
+ try {
+ const dest = bundleDir(paths, "jer");
+ writeBuild(dest, "old");
+ const out = exportOutPath(paths);
+ writeBuild(out, "new");
+ const calls: string[] = [];
+ assert.equal(await installBundle(out, dest, exdevFs(out, calls)), "copy");
+ assert.equal(readFileSync(path.join(dest, "index.html"), "utf8"), "new");
+ assert.equal(readFileSync(path.join(dest, "_next", "app.js"), "utf8"), "x".repeat(10));
+ assert.ok(!existsSync(out), "the source is removed after the copy");
+ assert.deepEqual(calls, [
+ "rename out -> out.next", // EXDEV
+ "cp out -> out.next",
+ "rename out -> out.prev", // siblings: same filesystem
+ "rename out.next -> out",
+ ]);
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+test("export/out becomes a relative link to the bundle; unlinkExportOut removes a link, never a real build", async () => {
+ const { root, paths } = tmp();
+ try {
+ const out = exportOutPath(paths);
+ writeBuild(out, "real");
+ await unlinkExportOut(paths);
+ assert.ok(lstatSync(out).isDirectory(), "a real out/ is left for next build");
+ const dest = bundleDir(paths, "jer");
+ await installBundle(out, dest);
+ await pointExportOutAt(paths, dest);
+ assert.ok(lstatSync(out).isSymbolicLink());
+ assert.equal(readlinkSync(out), path.join(".export-builds", "jer", "out"));
+ assert.equal(readFileSync(path.join(out, "index.html"), "utf8"), "real", "reads through the link");
+ // Pointing it at another bundle replaces the link, not the bundle.
+ const hub = bundleDir(paths, "_hub");
+ writeBuild(hub, "hub");
+ await pointExportOutAt(paths, hub);
+ assert.equal(readFileSync(path.join(out, "index.html"), "utf8"), "hub");
+ assert.ok(existsSync(path.join(dest, "index.html")), "the other bundle is untouched");
+ await unlinkExportOut(paths);
+ assert.ok(!existsSync(out));
+ assert.ok(existsSync(path.join(hub, "index.html")), "the link's target survives");
+ // A real out/ left behind is replaced by the link.
+ writeBuild(out, "stray");
+ await pointExportOutAt(paths, dest);
+ assert.ok(lstatSync(out).isSymbolicLink());
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+test("stageSiteArchives moves the host staging into the bundle's (dockerSiteStagingDir) and counts it", async () => {
+ const { root, paths } = tmp();
+ try {
+ const host = path.join(paths.exportDir, ".r2-staging", "jer", "archives");
+ mkdirSync(host, { recursive: true });
+ writeFileSync(path.join(host, "a.zip"), "a");
+ writeFileSync(path.join(host, "b.zip"), "b");
+ writeFileSync(path.join(host, ".hidden"), "");
+ const bundleStaging = path.join(paths.exportBuildsDir, "jer", ".r2-staging", "jer", "archives");
+ mkdirSync(bundleStaging, { recursive: true });
+ writeFileSync(path.join(bundleStaging, "last-build.zip"), "old");
+ assert.equal(await stageSiteArchives(paths, "jer"), 2);
+ assert.ok(existsSync(path.join(bundleStaging, "a.zip")));
+ assert.ok(!existsSync(path.join(bundleStaging, "last-build.zip")), "the last build's staging is replaced");
+ assert.ok(!existsSync(host));
+ // Nothing staged this build: the bundle has none either.
+ assert.equal(await stageSiteArchives(paths, "jer"), 0);
+ assert.ok(!existsSync(bundleStaging));
+ // Across filesystems.
+ mkdirSync(host, { recursive: true });
+ writeFileSync(path.join(host, "c.zip"), "c");
+ const calls: string[] = [];
+ assert.equal(await stageSiteArchives(paths, "jer", exdevFs(path.join(paths.exportDir, ".r2-staging"), calls)), 1);
+ assert.ok(calls.some((c) => c.startsWith("cp ")));
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+test("stageSiteArchives is a no-op where the two are one place (the build container's compose)", async () => {
+ const root = mkdtempSync(path.join(os.tmpdir(), "bundle-same-"));
+ try {
+ // EXPORT_PUBLIC_DIR=/site/public and a builds dir whose <id> IS /site.
+ const site = path.join(root, "jer");
+ const paths = { exportPublicDir: path.join(site, "public"), exportBuildsDir: root } as Paths;
+ const staged = path.join(site, ".r2-staging", "jer", "archives");
+ mkdirSync(staged, { recursive: true });
+ writeFileSync(path.join(staged, "a.zip"), "a");
+ assert.equal(await stageSiteArchives(paths, "jer"), 1);
+ assert.ok(existsSync(path.join(staged, "a.zip")));
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+test("bundleCounts and corpusGeneratedAtIn read the bundle; links are not followed", async () => {
+ const { root } = tmp();
+ try {
+ const dir = path.join(root, "b");
+ writeBuild(dir, "m");
+ symlinkSync("/etc", path.join(dir, "link"));
+ const c = await bundleCounts(dir);
+ assert.equal(c.files, 3);
+ assert.equal(c.bytes, 1 + 10 + JSON.stringify({ generatedAt: "at-m" }).length);
+ assert.equal(await corpusGeneratedAtIn(dir), "at-m");
+ assert.equal(await corpusGeneratedAtIn(path.join(root, "none")), null);
+ assert.deepEqual(await bundleCounts(path.join(root, "none")), { files: 0, bytes: 0 });
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+test("a crash between the two renames leaves out.prev and no out; the next build or install restores it first", async () => {
+ const { root, paths } = tmp();
+ try {
+ const dest = bundleDir(paths, "jer");
+ writeBuild(dest, "old");
+ const out = exportOutPath(paths);
+ writeBuild(out, "new");
+ // The process dies on the second rename (out.next -> out).
+ const dying: BundleFs = {
+ rename: (async (a: string, b: string) => {
+ if (String(a).endsWith("out.next") && String(b).endsWith(path.join("jer", "out"))) throw new Error("killed");
+ return rename(a, b);
+ }) as typeof rename,
+ cp,
+ rm,
+ mkdir,
+ };
+ await assert.rejects(installBundle(out, dest, dying), /killed/);
+ assert.ok(!existsSync(dest), "no out");
+ assert.ok(existsSync(`${dest}.prev`), "the last bundle, under the wrong name");
+ // Nothing to do when out is there; the old bundle back when it is not.
+ assert.equal(await recoverInterruptedInstall(dest), true);
+ assert.equal(readFileSync(path.join(dest, "index.html"), "utf8"), "old");
+ assert.equal(await recoverInterruptedInstall(dest), false);
+ // And an install over the same crash recovers before it swaps.
+ rmSync(`${dest}.next`, { recursive: true, force: true });
+ await rename(dest, `${dest}.prev`);
+ writeBuild(out, "newer");
+ await installBundle(out, dest);
+ assert.equal(readFileSync(path.join(dest, "index.html"), "utf8"), "newer");
+ assert.ok(!existsSync(`${dest}.prev`));
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
diff --git a/common/publish/inputSig.test.ts b/common/publish/inputSig.test.ts
@@ -0,0 +1,215 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { mkdirSync, mkdtempSync, rmSync, utimesSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import type { ChannelConfig } from "../lib/channelConfig";
+import type { Paths } from "../lib/paths";
+import type { Site } from "../lib/site";
+import { hubInputSig, siteInputSig } from "./inputSig";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// A site's inputSig moves exactly with what compose reads (release 18): its
+// staged index tree, its PUBLISHED members' shared trees (manifest.json aside,
+// a manifest-only tree signed as one), site.json, the corpus-wide files and the
+// two settings — and not with another site's channel, a churned manifest, or
+// a rewritten-but-identical chart-templates.json.
+
+function setup() {
+ const root = mkdtempSync(path.join(os.tmpdir(), "input-sig-"));
+ const idx = path.join(root, ".export-index");
+ const paths = {
+ transcriptsDir: path.join(root, "transcripts"),
+ sitesDir: path.join(root, "transcripts", "sites"),
+ exportSitesIndexDir: path.join(idx, "sites"),
+ exportSharedTranscriptsDir: path.join(idx, "shared", "transcripts"),
+ exportSharedSubsDir: path.join(idx, "shared", "subs"),
+ exportSharedPostsDir: path.join(idx, "shared", "posts"),
+ exportSharedDigestsDir: path.join(idx, "shared", "digests"),
+ globalAliasesFile: path.join(root, "transcripts", "search-aliases.json"),
+ globalTagsFile: path.join(root, "transcripts", "tags.json"),
+ homepageConfigFile: path.join(root, "transcripts", "sites", "_homepage", "homepage.json"),
+ } as Paths;
+ const site = {
+ siteId: "jer",
+ siteTitle: "Jer",
+ channels: [{ slug: "a" }, { slug: "x" }],
+ } as unknown as Site;
+ const write = (file: string, body: string) => {
+ mkdirSync(path.dirname(file), { recursive: true });
+ writeFileSync(file, body);
+ };
+ write(path.join(paths.sitesDir, "jer", "site.json"), JSON.stringify({ siteId: "jer" }));
+ write(path.join(paths.exportSitesIndexDir, "jer", "summaries", "page-0000.json"), "[]");
+ write(path.join(paths.exportSitesIndexDir, "jer", "chart-templates.json"), '{"v":1}');
+ write(path.join(paths.exportSharedTranscriptsDir, "a", "page-0000.json"), "[1]");
+ write(path.join(paths.exportSharedTranscriptsDir, "a", "manifest.json"), '{"generatedAt":"1"}');
+ write(path.join(paths.exportSharedPostsDir, "x", "page-0000.json"), "[2]");
+ write(path.join(paths.exportSharedTranscriptsDir, "other", "page-0000.json"), "[3]");
+ const configs: Record<string, Partial<ChannelConfig>> = {
+ a: {},
+ x: { sourceKind: "social", platform: "twitter" } as Partial<ChannelConfig>,
+ };
+ let settings: Record<string, unknown> = { socialLinks: [], buildArchives: true };
+ let sites: Site[] = [
+ site,
+ { siteId: "ani", siteTitle: "Ani", siteUrl: "https://ani.pages.dev", channels: [] } as unknown as Site,
+ ];
+ const sig = () =>
+ siteInputSig({
+ paths,
+ site,
+ settings: settings as never,
+ sites,
+ configOf: (slug) => configs[slug] as ChannelConfig,
+ });
+ const bump = (file: string) => {
+ const t = new Date(Date.now() + 5_000);
+ utimesSync(file, t, t);
+ };
+ return {
+ root,
+ paths,
+ write,
+ sig,
+ bump,
+ setXPrivate: () => {
+ settings = { ...settings, social: { x: { visibility: "private" } } };
+ },
+ setSettings: (over: Record<string, unknown>) => {
+ settings = { ...settings, ...over };
+ },
+ setSites: (next: Site[]) => {
+ sites = next;
+ },
+ site,
+ };
+}
+
+test("inputSig is stable, and ignores what compose ignores", async () => {
+ const s = setup();
+ try {
+ const base = await s.sig();
+ assert.match(base, /^[0-9a-f]{40}$/);
+ assert.equal(await s.sig(), base, "nothing changed");
+ // A churned shared manifest (generatedAt) does not count…
+ s.write(path.join(s.paths.exportSharedTranscriptsDir, "a", "manifest.json"), '{"generatedAt":"2"}');
+ assert.equal(await s.sig(), base);
+ // …nor another site's channel…
+ s.write(path.join(s.paths.exportSharedTranscriptsDir, "other", "page-0001.json"), "[4]");
+ assert.equal(await s.sig(), base);
+ // …nor chart-templates.json rewritten with the same bytes.
+ const tpl = path.join(s.paths.exportSitesIndexDir, "jer", "chart-templates.json");
+ s.bump(tpl);
+ assert.equal(await s.sig(), base);
+ } finally {
+ rmSync(s.root, { recursive: true, force: true });
+ }
+});
+
+test("inputSig moves with each input compose reads", async () => {
+ const s = setup();
+ try {
+ let last = await s.sig();
+ const moved = async (what: string) => {
+ const now = await s.sig();
+ assert.notEqual(now, last, what);
+ last = now;
+ };
+ s.write(path.join(s.paths.exportSharedTranscriptsDir, "a", "page-0001.json"), "[5]");
+ await moved("a member's shared transcripts");
+ s.write(path.join(s.paths.exportSharedPostsDir, "x", "page-0001.json"), "[6]");
+ await moved("a member's shared posts");
+ s.write(path.join(s.paths.exportSitesIndexDir, "jer", "summaries", "page-0001.json"), "[]");
+ await moved("the site's staged summaries");
+ s.write(path.join(s.paths.exportSitesIndexDir, "jer", "chart-templates.json"), '{"v":2}');
+ await moved("the chart templates' bytes");
+ s.write(path.join(s.paths.sitesDir, "jer", "site.json"), JSON.stringify({ siteId: "jer", t: 1 }));
+ await moved("site.json");
+ s.write(path.join(s.paths.sitesDir, "jer", "tags.json"), "{}");
+ await moved("the site's config dir");
+ s.write(s.paths.globalAliasesFile, "{}");
+ await moved("the global search aliases");
+ s.write(s.paths.globalTagsFile, "{}");
+ await moved("the curated tags");
+ s.write(path.join(s.paths.transcriptsDir, "duplicates.json"), "{}");
+ await moved("the duplicates report");
+ // X posts private: the X member is withheld from a public site.
+ s.setXPrivate();
+ await moved("social.x.visibility");
+ s.write(path.join(s.paths.exportSharedPostsDir, "x", "page-0002.json"), "[7]");
+ assert.equal(await s.sig(), last, "a withheld member's tree is not this site's input");
+ } finally {
+ rmSync(s.root, { recursive: true, force: true });
+ }
+});
+
+test("inputSig moves with what the export BUILD renders: social links, the hub url, buildArchives, the sibling sites", async () => {
+ const s = setup();
+ try {
+ let last = await s.sig();
+ const moved = async (what: string) => {
+ const now = await s.sig();
+ assert.notEqual(now, last, what);
+ last = now;
+ };
+ s.setSettings({ socialLinks: [{ label: "X", url: "https://x.com/a" }] });
+ await moved("settings.socialLinks (the header/footer links)");
+ s.setSettings({ homepageUrl: "https://hub.pages.dev" });
+ await moved("settings.homepageUrl (the site's hubUrl)");
+ s.setSettings({ buildArchives: false });
+ await moved("settings.buildArchives");
+ const ani = { siteId: "ani", siteTitle: "Ani", siteUrl: "https://ani.pages.dev", channels: [] };
+ s.setSites([s.site, { ...ani, siteTitle: "Ani 2" } as unknown as Site]);
+ await moved("a sibling's title");
+ s.setSites([s.site, { ...ani, siteTitle: "Ani 2", siteUrl: "https://ani2.pages.dev" } as unknown as Site]);
+ await moved("a sibling's url");
+ s.setSites([s.site, { ...ani, siteTitle: "Ani 2", siteUrl: "https://ani2.pages.dev" } as unknown as Site, {
+ siteId: "bon", siteTitle: "Bon", siteUrl: "https://bon.pages.dev", channels: [],
+ } as unknown as Site]);
+ await moved("a sibling launched");
+ // A sibling the footer does not list (no url) changes nothing.
+ s.setSites([s.site, { ...ani, siteTitle: "Ani 2", siteUrl: "https://ani2.pages.dev" } as unknown as Site, {
+ siteId: "bon", siteTitle: "Bon", siteUrl: "https://bon.pages.dev", channels: [],
+ } as unknown as Site, { siteId: "dark", siteTitle: "Dark", channels: [] } as unknown as Site]);
+ assert.equal(await s.sig(), last, "an unlinkable sibling is not in the footer");
+ } finally {
+ rmSync(s.root, { recursive: true, force: true });
+ }
+});
+
+test("a manifest-only member tree is signed as one, not as absent", async () => {
+ const s = setup();
+ try {
+ const before = await s.sig();
+ s.write(path.join(s.paths.exportSharedSubsDir, "a", "manifest.json"), "{}");
+ assert.notEqual(await s.sig(), before);
+ } finally {
+ rmSync(s.root, { recursive: true, force: true });
+ }
+});
+
+test("hubSig: the stamp, homepage.json, and each listed site's id + url + title", async () => {
+ const s = setup();
+ try {
+ const sites = [
+ { siteId: "jer", siteTitle: "Jer", siteUrl: "https://jer.pages.dev", channels: [] },
+ { siteId: "mine", siteTitle: "Mine", siteUrl: "https://m.pages.dev", audience: "private", channels: [] },
+ { siteId: "nourl", siteTitle: "No", channels: [] },
+ ] as unknown as Site[];
+ const a = await hubInputSig(s.paths, "s1", sites);
+ assert.equal(await hubInputSig(s.paths, "s1", sites), a);
+ assert.notEqual(await hubInputSig(s.paths, "s2", sites), a, "a new index stamp");
+ const retitled = sites.map((x) => (x.siteId === "jer" ? { ...x, siteTitle: "Jer 2" } : x));
+ assert.notEqual(await hubInputSig(s.paths, "s1", retitled), a);
+ // A private site or one with no url is not listed: changing it moves nothing.
+ const privRetitled = sites.map((x) => (x.siteId === "mine" ? { ...x, siteTitle: "M2" } : x));
+ assert.equal(await hubInputSig(s.paths, "s1", privRetitled), a);
+ s.write(s.paths.homepageConfigFile, '{"title":"hub"}');
+ assert.notEqual(await hubInputSig(s.paths, "s1", sites), a);
+ } finally {
+ rmSync(s.root, { recursive: true, force: true });
+ }
+});
diff --git a/common/publish/inputSig.ts b/common/publish/inputSig.ts
@@ -0,0 +1,222 @@
+// What a site's (and the hub's) build reads, as one sha1 — computed by the
+// update-index stage and stored in the IndexStamp (release 18).
+//
+// THE RULE: a site is fresh exactly when compose AND the export build would
+// produce the same bundle. So the signature is made of what they read, signed
+// with compose's own `dirSignature` (lib/dirSignature.ts) and compose's own
+// manifest-only rule:
+//
+// - the site's whole `.export-index/sites/<id>/` tree (summaries, stats,
+// chart templates, tag counts, the subs/posts/digests manifests, the
+// report media and exports);
+// - each PUBLISHED member channel's shared transcripts / subs / posts /
+// digests tree, `manifest.json` ignored (its `generatedAt` churns);
+// - the bytes of `site.json`, and the signature of the site's config dir
+// (`sites/<id>/`: tags, aliases, reports);
+// - the corpus-wide files compose reads at compose time (search aliases,
+// curated tags, the duplicates report and its overrides), by size + mtime;
+// - the `archiveStorage`, `social.x.visibility` and `buildArchives` settings;
+// - what `next build` renders beyond compose (release 18 S1 review): the
+// resolved social links, the hub url, and the footer's sibling sites.
+//
+// A superset of those inputs is CONSERVATIVE: a needless rebuild, never a
+// wrong skip. `hubSig` = sha1(stampId, homepage.json, each listed
+// site's id + siteUrl + title).
+
+import { createHash } from "node:crypto";
+import { existsSync } from "node:fs";
+import { readFile, stat } from "node:fs/promises";
+import path from "node:path";
+import { open } from "lmdb";
+import { dirSignature } from "../lib/dirSignature";
+import { DUPLICATES_FILENAME, DUPLICATE_OVERRIDES_FILENAME } from "../lib/duplicates";
+import type { Paths } from "../lib/paths";
+import { publishedMemberSlugs } from "../lib/postsVisibility";
+import type { SiteSettings } from "../lib/settings";
+import {
+ resolveHubUrl,
+ resolveRelatedSites,
+ resolveSocialLinks,
+ siteConfigFile,
+ siteDir,
+ siteIndexDir,
+ type Site,
+} from "../lib/site";
+import { isListedSite } from "../lib/siteSchema";
+import { INDEX_SCANNED_AT_KEY } from "../lib/stats";
+import { readChannelConfig } from "../controller/channels";
+import type { ChannelConfig } from "../lib/channelConfig";
+
+// compose-site.ts MANIFEST_ONLY_SIGNATURE — a tree holding only its manifest is
+// a channel with nothing in it, signed by a constant (compose's rule).
+const MANIFEST_ONLY = "manifest-only";
+// lib/chartsStore.ts siteTemplatesStagingPath's basename.
+const CHART_TEMPLATES = "chart-templates.json";
+
+const sha1 = (s: string | Buffer) => createHash("sha1").update(s).digest("hex");
+
+async function fileBytesSig(file: string): Promise<string> {
+ try {
+ return sha1(await readFile(file));
+ } catch {
+ return "";
+ }
+}
+
+async function fileStatSig(file: string): Promise<string> {
+ try {
+ const s = await stat(file);
+ return `${s.size}\t${s.mtimeMs}`;
+ } catch {
+ return "";
+ }
+}
+
+/** A memo over the shared per-channel trees: each is walked once per stamp. */
+export type TreeSigCache = Map<string, string>;
+
+async function channelTreeSig(root: string, slug: string, cache: TreeSigCache): Promise<string> {
+ const dir = path.join(root, slug);
+ const hit = cache.get(dir);
+ if (hit !== undefined) return hit;
+ let sig = await dirSignature(dir, "manifest.json");
+ if (sig === "" && existsSync(path.join(dir, "manifest.json"))) sig = MANIFEST_ONLY;
+ cache.set(dir, sig);
+ return sig;
+}
+
+export type SiteSigInputs = {
+ paths: Paths;
+ site: Site;
+ settings: Pick<SiteSettings, "archiveStorage" | "social" | "socialLinks" | "homepageUrl" | "buildArchives">;
+ // Every configured site: the footer's sibling list is part of the bundle.
+ sites: Site[];
+ // The site's members' configs (the visibility rule reads their platform).
+ configOf: (slug: string) => ChannelConfig | null | undefined;
+ cache?: TreeSigCache;
+};
+
+/** The site's inputSig (see the header). */
+export async function siteInputSig(i: SiteSigInputs): Promise<string> {
+ const { paths, site, settings } = i;
+ const cache = i.cache ?? new Map();
+ const members = [...publishedMemberSlugs(site, i.configOf, settings)].sort();
+ const lines: string[] = ["inputSig v1"];
+ lines.push(`members\t${members.join(",")}`);
+ // `build templates` rewrites chart-templates.json on every run (and compose
+ // copies it on every run): signed by its bytes, not its mtime.
+ const indexDir = siteIndexDir(paths, site.siteId);
+ lines.push(`site-index\t${await dirSignature(indexDir, CHART_TEMPLATES)}`);
+ lines.push(`${CHART_TEMPLATES}\t${await fileBytesSig(path.join(indexDir, CHART_TEMPLATES))}`);
+ for (const slug of members) {
+ for (const [tree, root] of [
+ ["transcripts", paths.exportSharedTranscriptsDir],
+ ["subs", paths.exportSharedSubsDir],
+ ["posts", paths.exportSharedPostsDir],
+ ["digests", paths.exportSharedDigestsDir],
+ ] as const) {
+ lines.push(`${tree}/${slug}\t${await channelTreeSig(root, slug, cache)}`);
+ }
+ }
+ lines.push(`site.json\t${await fileBytesSig(siteConfigFile(paths, site.siteId))}`);
+ lines.push(`site-dir\t${await dirSignature(siteDir(paths, site.siteId))}`);
+ for (const file of [
+ paths.globalAliasesFile,
+ paths.globalTagsFile,
+ path.join(paths.transcriptsDir, DUPLICATES_FILENAME),
+ path.join(paths.transcriptsDir, DUPLICATE_OVERRIDES_FILENAME),
+ ]) {
+ lines.push(`${path.basename(file)}\t${await fileStatSig(file)}`);
+ }
+ lines.push(
+ `settings\t${JSON.stringify({
+ archiveStorage: settings.archiveStorage ?? null,
+ xVisibility: settings.social?.x?.visibility ?? null,
+ buildArchives: settings.buildArchives ?? null,
+ })}`,
+ );
+ // What the export BUILD renders from beyond compose's inputs, resolved as it
+ // resolves them: the header/footer social links, the hub link the descriptor
+ // carries, and the footer's sibling sites (their urls, titles, listing).
+ const full = settings as SiteSettings;
+ lines.push(`social-links\t${JSON.stringify(resolveSocialLinks(site, full) ?? null)}`);
+ lines.push(`hub-url\t${resolveHubUrl(site, full) ?? ""}`);
+ lines.push(`related-sites\t${JSON.stringify(resolveRelatedSites(site, i.sites))}`);
+ return sha1(lines.join("\n"));
+}
+
+/** Every member's channel config, read once (unreadable = null). */
+export async function readMemberConfigs(
+ paths: Paths,
+ sites: Site[],
+): Promise<Map<string, ChannelConfig | null>> {
+ const slugs = new Set(sites.flatMap((s) => s.channels.map((c) => c.slug)));
+ const out = new Map<string, ChannelConfig | null>();
+ await Promise.all(
+ [...slugs].map(async (slug) => {
+ out.set(slug, await readChannelConfig(paths, slug).catch(() => null));
+ }),
+ );
+ return out;
+}
+
+/** The hub's signature: the index it was built from, its config, the pool it lists. */
+export async function hubInputSig(
+ paths: Paths,
+ stampId: string,
+ sites: Site[],
+): Promise<string> {
+ const lines = [`hubSig v1`, `stamp\t${stampId}`];
+ lines.push(`homepage.json\t${await fileBytesSig(paths.homepageConfigFile)}`);
+ for (const s of [...sites].sort((a, b) => a.siteId.localeCompare(b.siteId))) {
+ if (!isListedSite(s) || !s.siteUrl) continue;
+ lines.push(`site\t${s.siteId}\t${s.siteUrl}\t${s.siteTitle}`);
+ }
+ return sha1(lines.join("\n"));
+}
+
+export type IndexMeta = {
+ generation: number;
+ scannedAt: number | null;
+ // sha1 of each site's stored fingerprints, null when absent.
+ siteFp: Record<string, string | null>;
+ statsFp: Record<string, string | null>;
+};
+
+/**
+ * The LMDB index's own bookkeeping, read-only, after the index and stats
+ * builds have closed it: `generation`, INDEX_SCANNED_AT_KEY and each site's
+ * `siteFp:<id>` / `statsFp:<id>`. An absent index reads as zeros.
+ */
+export function readIndexMeta(paths: Paths, siteIds: string[]): IndexMeta {
+ const out: IndexMeta = { generation: 0, scannedAt: null, siteFp: {}, statsFp: {} };
+ for (const id of siteIds) {
+ out.siteFp[id] = null;
+ out.statsFp[id] = null;
+ }
+ if (!existsSync(paths.lmdbPath)) return out;
+ const root = open({ path: paths.lmdbPath, readOnly: true, maxDbs: 18, compression: true });
+ try {
+ const meta = root.openDB<unknown, string>({ name: "meta", encoding: "msgpack" });
+ // An index that stats never ran over has no statsMeta sub-DB.
+ let statsMeta: typeof meta | null = null;
+ try {
+ statsMeta = root.openDB<unknown, string>({ name: "statsMeta", encoding: "msgpack" });
+ } catch {
+ statsMeta = null;
+ }
+ const gen = meta.get("generation");
+ out.generation = typeof gen === "number" ? gen : 0;
+ const scanned = meta.get(INDEX_SCANNED_AT_KEY);
+ out.scannedAt = typeof scanned === "number" ? scanned : null;
+ for (const id of siteIds) {
+ const fp = meta.get(`siteFp:${id}`);
+ const sfp = statsMeta?.get(`statsFp:${id}`);
+ out.siteFp[id] = typeof fp === "string" ? sha1(fp) : null;
+ out.statsFp[id] = typeof sfp === "string" ? sha1(sfp) : null;
+ }
+ } finally {
+ root.close();
+ }
+ return out;
+}
diff --git a/common/publish/source.test.ts b/common/publish/source.test.ts
@@ -375,6 +375,35 @@ test("no git repository here (a docker runtime, a tarball install): the build ge
assert.ok(!existsSync(path.join(pub, "source")));
});
+test("ARCHILYZER_SOURCE_REPO names the repository where the checkout has none (the container's mount); one that names nothing refuses by name (release 18)", async () => {
+ const bare = dir("no-git-env");
+ const pub = path.join(dir("site"), "public");
+ const logs: string[] = [];
+ const base: SourcePublishOpts = {
+ paths: { monorepoRoot: bare } as Paths,
+ publicDir: pub,
+ onLog: (l) => logs.push(l),
+ scrubFile: path.join(bare, "none.txt"),
+ denylistFile: path.join(bare, "none.txt"),
+ check: true,
+ };
+ // A mounted repository: found, so the publish goes on to the next step —
+ // here, the operator's files, which this scenario leaves out on purpose.
+ // It is a git DIR — what the overlay mounts (the host's common dir), and what
+ // the checkout lookup answers.
+ const repo = path.join(sourceRepo(), ".git");
+ assert.equal(await publishSource({ ...base, env: { ...process.env, ARCHILYZER_SOURCE_REPO: repo } }), 1);
+ assert.ok(!logs.includes(`[source] ${NO_REPOSITORY}`), logs.join("\n"));
+ assert.match(logs.join("\n"), /none\.txt/, "it reached the operator's files");
+ // A variable naming a path that is not there: a refusal that names it, never
+ // the "no repository" sentence.
+ logs.length = 0;
+ const gone = path.join(bare, "not-mounted.git");
+ assert.equal(await publishSource({ ...base, env: { ...process.env, ARCHILYZER_SOURCE_REPO: gone } }), 1);
+ assert.match(logs.join("\n"), /REFUSED: ARCHILYZER_SOURCE_REPO names .*not-mounted\.git, which is not there/);
+ assert.ok(!logs.includes(`[source] ${NO_REPOSITORY}`));
+});
+
test("round trip: --check writes nothing; publish; a dumb clone of the mirror is main, scrubbed, from static files", async (t) => {
if (filterRepoProblem) return t.skip(`git-filter-repo unavailable: ${filterRepoProblem}`);
const repo = sourceRepo();
diff --git a/common/publish/source.ts b/common/publish/source.ts
@@ -133,7 +133,8 @@ export type SourcePublishOpts = PublishOpts & {
check?: boolean;
// Leave the scratch dir (the rewritten bare clone, the stage) for a look.
keepScratch?: boolean;
- // The repository to mirror. Default: this checkout's git COMMON dir, so a
+ // The repository to mirror. Default: ARCHILYZER_SOURCE_REPO (a container's
+ // mount of the host's repository), else this checkout's git COMMON dir, so a
// worktree build mirrors the primary's main.
sourceRepo?: string;
// Default: HOMEPAGE_PUBLIC_DIR, else <repo>/homepage/public.
@@ -611,6 +612,25 @@ async function commonDir(ctx: Ctx, cwd: string): Promise<string | null> {
throw new SourceRefusal(`git rev-parse exited ${r.code}${tail ? `: ${tail}` : ""}`);
}
+// The repository to mirror when the caller names none: ARCHILYZER_SOURCE_REPO
+// when it is set — in a container, the host's git common dir mounted read-only
+// by docker-compose.source.yml, since the image has no .git — else this
+// checkout's common dir. A variable that names nothing is a refusal, not a
+// silent fall-through to "no repository" (which would withdraw the publish
+// with a sentence about tarball installs).
+async function sourceRepoFor(ctx: Ctx, cwd: string): Promise<string | null> {
+ const named = ctx.env.ARCHILYZER_SOURCE_REPO?.trim();
+ if (named) {
+ if (!existsSync(named)) {
+ throw new SourceRefusal(
+ `ARCHILYZER_SOURCE_REPO names ${named}, which is not there — mount the host's git common dir there (docker-compose.source.yml), or unset it`,
+ );
+ }
+ return named;
+ }
+ return commonDir(ctx, cwd);
+}
+
// The real path of `p`, or of its deepest existing ancestor with the rest
// appended: a scratch root may not exist yet, and a symlink must not hide
// where it lands.
@@ -764,7 +784,7 @@ async function publish(
// 1. The private main — or no repository at all (L8: the docker runtime,
// a tarball install), where nothing can be mirrored and nothing can leak.
- const sourceRepo = opts.sourceRepo ?? (await commonDir(ctx, paths.monorepoRoot));
+ const sourceRepo = opts.sourceRepo ?? (await sourceRepoFor(ctx, paths.monorepoRoot));
if (sourceRepo === null) {
onLog(`[source] ${NO_REPOSITORY}`);
if (opts.check) return 1;
@@ -1328,7 +1348,7 @@ export async function publishedSourceProblem(
};
let main: string | null = null;
try {
- const repo = opts.sourceRepo ?? (await commonDir(ctx, paths.monorepoRoot));
+ const repo = opts.sourceRepo ?? (await sourceRepoFor(ctx, paths.monorepoRoot));
main = repo ? await revParse(ctx, repo, `refs/heads/${SOURCE_BRANCH}^{commit}`) : null;
} catch (err) {
if (!(err instanceof SourceRefusal)) throw err;
diff --git a/common/publish/stageBodies.ts b/common/publish/stageBodies.ts
@@ -0,0 +1,736 @@
+// The publish stages' bodies (release 18): what `run()` does, in the stage
+// child the editor spawns (`archilyzer stage <kind> <target> …`) or in the
+// CLI's own process (`archilyzer publish …`), always under the publish lock
+// (stageRun.ts takes it).
+//
+// Every body but update-index first asks its stage's `needs()` over the state
+// ON DISK (`readNeedsInput`): blocked → StageFailure exit 3 (the precondition
+// is not met — "update the index first", "no build of X"), fresh and not
+// forced → a no-op. Ordering between the stages of one run is enforced here,
+// not in anybody's memory.
+//
+// The three deploy bodies call today's deploy functions in build.ts with the
+// target's own bundle; release 18 S2 rewires them to its deploy stage
+// (pinned wrangler, credential preflight, the live check).
+
+import { execFile } from "node:child_process";
+import { cp, mkdir, readdir, rm } from "node:fs/promises";
+import path from "node:path";
+import { promisify } from "node:util";
+import {
+ builtAudienceProblem,
+ builtBundleProblem,
+ builtHomepageProblem,
+ builtHubProblem,
+ siteDeployProblem,
+} from "../lib/builtExport";
+import { getHomepageConfig } from "../lib/homepage";
+import { previewAliasUrl, previewBranchProblem } from "../lib/pagesDeploy";
+import type { Paths } from "../lib/paths";
+import { getSettings } from "../lib/settings";
+import { getSite, listSites, type Site } from "../lib/site";
+import {
+ ALL_TARGET,
+ HOMEPAGE_TARGET,
+ HUB_TARGET,
+ imageBuildFacts,
+ newStampId,
+ readBuiltStamp,
+ readDeployedFile,
+ readIndexStamp,
+ recordDeploy,
+ writeBuiltStamp,
+ type BuiltKind,
+ type BuiltStamp,
+ type DeployRecord,
+ type IndexStamp,
+ type Runner,
+} from "./stamps";
+import {
+ STAGES,
+ deployKindOf,
+ type NeedsInput,
+ type StageContext,
+ type StageOutcome,
+ type StageRequest,
+ type TargetState,
+} from "./stages";
+
+/** A stage that did not run to the end: its exit code says why (stageRun.ts). */
+export class StageFailure extends Error {
+ constructor(
+ message: string,
+ readonly exitCode: 1 | 2 | 3,
+ ) {
+ super(message);
+ this.name = "StageFailure";
+ }
+}
+
+export class StageCancelled extends Error {
+ constructor() {
+ super("cancelled");
+ this.name = "StageCancelled";
+ }
+}
+
+function checkCancel(signal: AbortSignal): void {
+ if (signal.aborted) throw new StageCancelled();
+}
+
+// ---------------------------------------------------------------------------
+// git facts for the stamps
+// ---------------------------------------------------------------------------
+
+const exec = promisify(execFile);
+
+async function git(cwd: string, args: string[]): Promise<string | null> {
+ try {
+ const { stdout } = await exec("git", args, { cwd, timeout: 10_000 });
+ const out = stdout.trim();
+ return out || null;
+ } catch {
+ return null;
+ }
+}
+
+/**
+ * The commit and branch a stamp records. `ARCHILYZER_COMMIT` /
+ * `ARCHILYZER_BRANCH`, when set, WIN over git, each on its own: the runtime
+ * image bakes them (it has no .git), and a test server sets
+ * `ARCHILYZER_BRANCH=main` because a worktree's branch is never `main`. Else
+ * the checkout's HEAD and branch; a detached HEAD records `branch: null`,
+ * which a production deploy refuses like any branch but `main`.
+ */
+export async function checkoutInfo(
+ paths: Pick<Paths, "monorepoRoot">,
+ env: NodeJS.ProcessEnv = process.env,
+): Promise<{ commit: string | null; branch: string | null }> {
+ const facts = imageBuildFacts(env);
+ const commit = facts.commit ?? (await git(paths.monorepoRoot, ["rev-parse", "HEAD"]));
+ if (facts.branch !== null) return { commit, branch: facts.branch };
+ const branch = await git(paths.monorepoRoot, ["rev-parse", "--abbrev-ref", "HEAD"]);
+ return { commit, branch: branch === null || branch === "HEAD" ? null : branch };
+}
+
+/** `main`'s HEAD where a repository is reachable, else null. */
+export async function mainHeadOf(paths: Pick<Paths, "monorepoRoot">): Promise<string | null> {
+ return git(paths.monorepoRoot, ["rev-parse", "--verify", "--quiet", "refs/heads/main^{commit}"]);
+}
+
+// ---------------------------------------------------------------------------
+// The state needs() reads, from disk alone
+// ---------------------------------------------------------------------------
+
+function sitePagesProblem(site: Site): string | null {
+ return site.cloudflareProject?.trim()
+ ? null
+ : `Site "${site.siteId}" has no Cloudflare Pages project configured`;
+}
+
+/**
+ * The NeedsInput a stage child can build from disk alone: the stamps and the
+ * bundles. It does not read job metas or config mtimes (`changedChannels` is
+ * empty, `configChangedAt` null): a child judges by signatures, and the
+ * signature in the index stamp is the authority once the index has run.
+ */
+export async function readNeedsInput(
+ paths: Paths,
+ opts: { mainHead?: boolean } = {},
+): Promise<NeedsInput> {
+ const { bundleDir, homepageOutDir, hubProjectProblem } = await import("./build");
+ const stamp = await readIndexStamp(paths);
+ const sites: Record<string, TargetState> = {};
+ for (const site of listSites(paths)) {
+ const built = await readBuiltStamp(paths, site.siteId);
+ sites[site.siteId] = {
+ built,
+ deployed: await readDeployedFile(paths, site.siteId),
+ changedChannels: [],
+ configChangedAt: null,
+ bundleProblem: built ? builtBundleProblem(bundleDir(paths, site.siteId), site.siteId) : null,
+ deployProblem: siteDeployProblem(site),
+ pagesProblem: sitePagesProblem(site),
+ };
+ }
+ const hubBuilt = await readBuiltStamp(paths, HUB_TARGET);
+ const homeBuilt = await readBuiltStamp(paths, HOMEPAGE_TARGET);
+ return {
+ index: { stamp, lastIngestDoneAt: null, configChangedAt: null },
+ sites,
+ hub: {
+ built: hubBuilt,
+ deployed: await readDeployedFile(paths, HUB_TARGET),
+ changedChannels: [],
+ configChangedAt: null,
+ bundleProblem: hubBuilt ? builtHubProblem(bundleDir(paths, HUB_TARGET)) : null,
+ pagesProblem: hubProjectProblem(getHomepageConfig(paths).cloudflareProject),
+ },
+ homepage: {
+ built: homeBuilt,
+ deployed: await readDeployedFile(paths, HOMEPAGE_TARGET),
+ changedChannels: [],
+ configChangedAt: null,
+ bundleProblem: homeBuilt ? builtHomepageProblem(homepageOutDir(paths)) : null,
+ mainHead: opts.mainHead ? await mainHeadOf(paths) : null,
+ },
+ };
+}
+
+// ---------------------------------------------------------------------------
+// Stamping a build
+// ---------------------------------------------------------------------------
+
+async function stampBuilt(
+ paths: Paths,
+ outDir: string,
+ s: {
+ target: string;
+ kind: BuiltKind;
+ indexStampId: string | null;
+ inputSig: string;
+ runner: Runner;
+ archivesStaged: number;
+ sourceCommit?: string | null;
+ },
+): Promise<BuiltStamp> {
+ const { bundleCounts, corpusGeneratedAtIn } = await import("./build");
+ const { commit, branch } = await checkoutInfo(paths);
+ const counts = await bundleCounts(outDir);
+ const built: BuiltStamp = {
+ v: 1,
+ stampId: newStampId(),
+ target: s.target,
+ kind: s.kind,
+ indexStampId: s.indexStampId,
+ inputSig: s.inputSig,
+ builtAt: Date.now(),
+ commit,
+ branch,
+ runner: s.runner,
+ audience: builtAudienceProblem(outDir) ? "private" : "public",
+ corpusGeneratedAt: await corpusGeneratedAtIn(outDir),
+ files: counts.files,
+ bytes: counts.bytes,
+ archivesStaged: s.archivesStaged,
+ ...(s.sourceCommit !== undefined ? { sourceCommit: s.sourceCommit } : {}),
+ };
+ await writeBuiltStamp(paths, built);
+ return built;
+}
+
+function plural(n: number, word: string): string {
+ return `${n} ${word}${n === 1 ? "" : "s"}`;
+}
+
+// ---------------------------------------------------------------------------
+// update-index
+// ---------------------------------------------------------------------------
+
+async function runUpdateIndex(ctx: StageContext): Promise<StageOutcome> {
+ const { paths, onLog, signal } = ctx;
+ const { settingsFromFile } = await import("../lib/settings");
+ const { applyHealthTimings } = await import("../lib/storageHealth");
+ const { buildIndex } = await import("../controller/buildIndex");
+ const { buildStats } = await import("../controller/buildStats");
+ const { syncTemplatesToExport } = await import("../lib/chartsStore");
+ const { hubInputSig, readIndexMeta, readMemberConfigs, siteInputSig } = await import("./inputSig");
+
+ // A CLI process has no health pass: the drive-health timings the builds'
+ // watchdog runs on are applied here, once (bin/build-index.ts does the same).
+ applyHealthTimings(settingsFromFile(paths.settingsFile).storage.health);
+ const log = (m: string) => onLog(m.endsWith("\n") ? m : `${m}\n`);
+
+ onLog("=== update-index: the LMDB index ===\n");
+ const idx = await buildIndex({ paths, onLog: log });
+ checkCancel(signal);
+ onLog("=== update-index: the stats datasets ===\n");
+ const st = await buildStats({ paths, onLog: log, signal });
+ checkCancel(signal);
+ onLog("=== update-index: chart templates ===\n");
+ const sites = listSites(paths);
+ for (const site of sites) syncTemplatesToExport(paths, site.siteId);
+ const templatesAt = Date.now();
+ checkCancel(signal);
+
+ onLog("=== update-index: signatures ===\n");
+ const meta = readIndexMeta(
+ paths,
+ sites.map((s) => s.siteId),
+ );
+ const settings = getSettings();
+ const configs = await readMemberConfigs(paths, sites);
+ const cache = new Map<string, string>();
+ const siteEntries: IndexStamp["sites"] = {};
+ for (const site of sites) {
+ siteEntries[site.siteId] = {
+ siteFp: meta.siteFp[site.siteId] ?? null,
+ statsFp: meta.statsFp[site.siteId] ?? null,
+ inputSig: await siteInputSig({
+ paths,
+ site,
+ settings,
+ sites,
+ configOf: (slug) => configs.get(slug),
+ cache,
+ }),
+ };
+ }
+
+ // The same index as the last stamp (nothing rebuilt, every signature the
+ // same) keeps its stamp id, so the builds made from it stay current.
+ const prev = await readIndexStamp(paths);
+ let stampId = newStampId();
+ let reused = false;
+ if (
+ prev &&
+ idx.shortCircuited &&
+ st.shortCircuited &&
+ prev.generation === meta.generation &&
+ sameSites(prev.sites, siteEntries) &&
+ (await hubInputSig(paths, prev.stampId, sites)) === prev.hubSig
+ ) {
+ stampId = prev.stampId;
+ reused = true;
+ }
+ const { commit } = await checkoutInfo(paths);
+ const stamp: IndexStamp = {
+ v: 1,
+ stampId,
+ generation: meta.generation,
+ scannedAt: meta.scannedAt ?? Date.now(),
+ builtAt: Date.now(),
+ templatesAt,
+ commit,
+ index: {
+ shortCircuited: idx.shortCircuited,
+ added: idx.added,
+ changed: idx.changed,
+ removed: idx.removed,
+ heldChannels: idx.heldChannels,
+ },
+ stats: {
+ shortCircuited: st.shortCircuited,
+ notIndexedYet: st.notIndexedYet,
+ notIndexable: st.notIndexable,
+ },
+ sites: siteEntries,
+ hubSig: await hubInputSig(paths, stampId, sites),
+ };
+ const { writeIndexStamp } = await import("./stamps");
+ await writeIndexStamp(paths, stamp);
+ const summary =
+ `index +${idx.added} ~${idx.changed} -${idx.removed}` +
+ (idx.heldChannels.length ? ` (held: ${idx.heldChannels.join(", ")})` : "") +
+ `; ${plural(sites.length, "site")} signed` +
+ (reused ? "; nothing changed — the stamp stands" : "");
+ return { status: reused ? "noop" : "ran", stamp: stampId, summary };
+}
+
+function sameSites(a: IndexStamp["sites"], b: IndexStamp["sites"]): boolean {
+ const ka = Object.keys(a).sort();
+ const kb = Object.keys(b).sort();
+ if (ka.join("\n") !== kb.join("\n")) return false;
+ return ka.every((k) => a[k].inputSig === b[k].inputSig);
+}
+
+// ---------------------------------------------------------------------------
+// build-site (one site, every site locally, every site in containers)
+// ---------------------------------------------------------------------------
+
+// Inside the docker per-site build container (docker/build-site.sh sets
+// ARCHIVES_READONLY=1): the build stays in export/out for the container to hand
+// back, and the HOST stamps it — nothing is installed or stamped here.
+function inBuildContainer(): boolean {
+ return process.env.ARCHIVES_READONLY === "1";
+}
+
+// What a container build reads where the host wrote no stamp (the editor's
+// Build all before release 18's surfaces): nothing is stamped in a container.
+const CONTAINER_NO_STAMP: IndexStamp = {
+ v: 1,
+ stampId: "",
+ generation: 0,
+ scannedAt: 0,
+ builtAt: 0,
+ templatesAt: 0,
+ commit: null,
+ index: { shortCircuited: true, added: 0, changed: 0, removed: 0, heldChannels: [] },
+ stats: { shortCircuited: true, notIndexedYet: 0, notIndexable: 0 },
+ sites: {},
+ hubSig: "",
+};
+
+async function buildOneSite(ctx: StageContext, r: StageRequest, stamp: IndexStamp): Promise<StageOutcome> {
+ const { paths, onLog, signal } = ctx;
+ const { buildSiteBundle, bundleDir } = await import("./build");
+ const siteId = r.target;
+ const inPlace = inBuildContainer();
+ const res = await buildSiteBundle(siteId, {
+ paths,
+ onLog,
+ signal,
+ skipArchives: r.skipArchives,
+ allowMissingMedia: r.allowMissingMedia,
+ inPlace,
+ });
+ checkCancel(signal);
+ if (res.code !== 0) throw new StageFailure(`build of ${siteId} failed (exit ${res.code})`, 1);
+ if (inPlace) return { status: "ran", stamp: "", summary: `${siteId} built in place (build container)` };
+ const built = await stampBuilt(paths, bundleDir(paths, siteId), {
+ target: siteId,
+ kind: "site",
+ indexStampId: stamp.stampId,
+ inputSig: stamp.sites[siteId].inputSig,
+ runner: "local",
+ archivesStaged: res.archivesStaged,
+ });
+ return {
+ status: "ran",
+ stamp: built.stampId,
+ summary: `${siteId} built: ${plural(built.files, "file")}, ${(built.bytes / 1e6).toFixed(1)} MB`,
+ };
+}
+
+// The sites `_all` builds: each stale one (every one with --force).
+// (A fresh one is a no-op build: its `checkedAt` is moved on.)
+async function sitesToBuild(
+ paths: Paths,
+ input: NeedsInput,
+ r: StageRequest,
+ onLog: (l: string) => void,
+): Promise<string[]> {
+ const ids: string[] = [];
+ for (const id of Object.keys(input.sites)) {
+ const f = STAGES["build-site"].needs(input, { ...r, target: id });
+ if (f.state === "fresh") {
+ onLog(`[publish] ${id}: fresh — skipped\n`);
+ await markChecked(paths, input.sites[id].built);
+ } else if (f.state === "blocked") onLog(`[publish] ${id}: blocked — ${f.reason}\n`);
+ else ids.push(id);
+ }
+ return ids;
+}
+
+/**
+ * A no-op build: the bundle still matches its inputs, as of now. Recorded as
+ * `checkedAt`, so the "channels changed" signal (measured against
+ * max(builtAt, checkedAt)) clears, without pretending a build happened.
+ */
+async function markChecked(paths: Paths, built: BuiltStamp | null | undefined): Promise<void> {
+ if (built) await writeBuiltStamp(paths, { ...built, checkedAt: Date.now() });
+}
+
+async function buildAllLocal(ctx: StageContext, r: StageRequest, input: NeedsInput): Promise<StageOutcome> {
+ const stamp = input.index.stamp!;
+ const ids = await sitesToBuild(ctx.paths, input, r, ctx.onLog);
+ const failed: string[] = [];
+ let built = 0;
+ for (const [i, id] of ids.entries()) {
+ checkCancel(ctx.signal);
+ ctx.onLog(`\n=== Build ${id} (${i + 1}/${ids.length}) ===\n`);
+ try {
+ await buildOneSite(ctx, { ...r, target: id }, stamp);
+ built++;
+ } catch (err) {
+ if (err instanceof StageCancelled) throw err;
+ failed.push(id);
+ ctx.onLog(`[publish] ${id}: ${(err as Error).message}\n`);
+ }
+ }
+ const summary = `${built}/${ids.length} built` + (failed.length ? `; failed: ${failed.join(", ")}` : "");
+ if (failed.length) throw new StageFailure(summary, 1);
+ return { status: built ? "ran" : "noop", stamp: stamp.stampId, summary };
+}
+
+export const NO_ENGINE = "the docker runner needs an engine on this host";
+
+async function buildAllDocker(ctx: StageContext, r: StageRequest, input: NeedsInput): Promise<StageOutcome> {
+ const { paths, onLog, signal } = ctx;
+ const b = await import("./build");
+ if (!(await b.dockerAvailable(signal))) throw new StageFailure(NO_ENGINE, 3);
+ const stamp = input.index.stamp!;
+ const ids = await sitesToBuild(paths, input, r, onLog);
+ if (ids.length === 0) return { status: "noop", stamp: stamp.stampId, summary: "every site is fresh" };
+ if (!r.skipArchives) {
+ onLog("=== the archive cache (host) ===\n");
+ const code = await b.runHostScript(onLog, signal, paths, "build:archives");
+ checkCancel(signal);
+ if (code !== 0) throw new StageFailure(`warming the archive cache failed (exit ${code})`, 1);
+ }
+ const img = await b.ensureBuildImage(onLog, signal, paths);
+ checkCancel(signal);
+ if (img !== 0) throw new StageFailure(`the build image failed (exit ${img})`, 1);
+ const { maxParallelBuilds } = getSettings().buildPipeline;
+ onLog(`=== building ${plural(ids.length, "site")} in containers, up to ${maxParallelBuilds} at once ===\n`);
+ const outcomes = await b.runWithConcurrency(ids, maxParallelBuilds, async (id) => {
+ if (signal.aborted) return { id, ok: false };
+ const code = await b.runDockerBuildOne(onLog, signal, id, paths, { skipArchives: r.skipArchives });
+ const out = b.bundleDir(paths, id);
+ const problem = code === 0 ? builtBundleProblem(out, id) : null;
+ if (code !== 0 || problem) {
+ onLog(`[${id}] build FAILED — ${problem ?? `exit ${code}`}\n`);
+ return { id, ok: false };
+ }
+ const staged = await readdir(b.dockerSiteStagingDir(paths, id)).catch(() => [] as string[]);
+ await stampBuilt(paths, out, {
+ target: id,
+ kind: "site",
+ indexStampId: stamp.stampId,
+ inputSig: stamp.sites[id].inputSig,
+ runner: "docker",
+ archivesStaged: staged.filter((f) => !f.startsWith(".")).length,
+ });
+ onLog(`[${id}] build ok\n`);
+ return { id, ok: true };
+ });
+ checkCancel(signal);
+ const failed = outcomes.filter((o) => !o.ok).map((o) => o.id);
+ const summary = `${ids.length - failed.length}/${ids.length} built in containers` + (failed.length ? `; failed: ${failed.join(", ")}` : "");
+ if (failed.length) throw new StageFailure(summary, 1);
+ return { status: "ran", stamp: stamp.stampId, summary };
+}
+
+// ---------------------------------------------------------------------------
+// hub and homepage builds
+// ---------------------------------------------------------------------------
+
+async function buildHubStage(ctx: StageContext, stamp: IndexStamp): Promise<StageOutcome> {
+ const { buildHubBundle, bundleDir } = await import("./build");
+ const code = await buildHubBundle(ctx);
+ checkCancel(ctx.signal);
+ if (code !== 0) throw new StageFailure(`the hub build failed (exit ${code})`, 1);
+ const built = await stampBuilt(ctx.paths, bundleDir(ctx.paths, HUB_TARGET), {
+ target: HUB_TARGET,
+ kind: "hub",
+ indexStampId: stamp.stampId,
+ inputSig: stamp.hubSig,
+ runner: "local",
+ archivesStaged: 0,
+ });
+ return { status: "ran", stamp: built.stampId, summary: `hub built: ${plural(built.files, "file")}` };
+}
+
+async function buildHomepageStage(ctx: StageContext, stamp: IndexStamp): Promise<StageOutcome> {
+ const { buildHomepage, homepageOutDir } = await import("./build");
+ const { readPublishedManifest } = await import("./source");
+ const code = await buildHomepage(ctx);
+ checkCancel(ctx.signal);
+ if (code !== 0) throw new StageFailure(`the homepage build failed (exit ${code})`, 1);
+ const out = homepageOutDir(ctx.paths);
+ const manifest = await readPublishedManifest(out);
+ const built = await stampBuilt(ctx.paths, out, {
+ target: HOMEPAGE_TARGET,
+ kind: "homepage",
+ indexStampId: stamp.stampId,
+ inputSig: stamp.stampId,
+ runner: "local",
+ archivesStaged: 0,
+ sourceCommit: manifest?.sourceCommit ?? null,
+ });
+ return { status: "ran", stamp: built.stampId, summary: `homepage built: ${plural(built.files, "file")}` };
+}
+
+// ---------------------------------------------------------------------------
+// deploys
+// ---------------------------------------------------------------------------
+
+/** Where `--to local` publishes a site: the `site` service's volume. */
+export function localSiteOut(env: NodeJS.ProcessEnv = process.env): string | null {
+ return env.ARCHILYZER_SITE_OUT?.trim() || null;
+}
+
+// …and the homepage: what the `homepage` service serves (release 18 S5
+// declares the name; read by name here until both slices are merged).
+const HOMEPAGE_OUT_NAME = "ARCHILYZER_HOMEPAGE_OUT";
+
+export function localHomepageOut(env: NodeJS.ProcessEnv = process.env): string | null {
+ return env[HOMEPAGE_OUT_NAME]?.trim() || null;
+}
+
+// Replace the CONTENTS of `dest` (a volume mount: never the directory itself).
+async function publishLocal(src: string, dest: string): Promise<void> {
+ await mkdir(dest, { recursive: true });
+ for (const name of await readdir(dest)) await rm(path.join(dest, name), { recursive: true, force: true });
+ await cp(src, dest, { recursive: true });
+}
+
+// Spot the deployment URL build.ts logs on success.
+function urlWatcher(onLog: (l: string) => void): { onLog: (l: string) => void; url: () => string | null } {
+ let url: string | null = null;
+ return {
+ onLog: (line) => {
+ const m = /^\[deployed\] (\S+)/.exec(line) ?? /\(this deployment: (\S+)\)/.exec(line);
+ if (m) url = m[1];
+ onLog(line);
+ },
+ url: () => url,
+ };
+}
+
+async function deployStage(
+ ctx: StageContext,
+ r: StageRequest,
+ target: string,
+ ship: (o: { onLog: (l: string) => void; previewBranch?: string }) => Promise<void>,
+ local: { src: string; dest: string | null; needs: string } | null,
+ project: string | null,
+): Promise<StageOutcome> {
+ const { paths, onLog, signal } = ctx;
+ const built = (await readBuiltStamp(paths, target))!;
+ const kind = deployKindOf(r);
+ let url: string | null = null;
+ let alias: string | undefined;
+ if (kind === "local") {
+ if (!local) throw new StageFailure(`${target} has no local target`, 2);
+ if (!local.dest) {
+ throw new StageFailure(`--to local needs ${local.needs}`, 3);
+ }
+ // A bundle built private is never published, here either.
+ const priv = builtAudienceProblem(local.src);
+ if (priv) throw new StageFailure(`${priv}. Build ${target} again, then deploy.`, 3);
+ onLog(`[publish] copying ${local.src} -> ${local.dest}\n`);
+ await publishLocal(local.src, local.dest);
+ } else {
+ if (r.preview !== undefined) {
+ const problem = previewBranchProblem(r.preview);
+ if (problem) throw new StageFailure(problem, 2);
+ }
+ const w = urlWatcher(onLog);
+ try {
+ await ship({ onLog: w.onLog, previewBranch: r.preview });
+ } catch (err) {
+ checkCancel(signal);
+ throw new StageFailure((err as Error).message, 1);
+ }
+ checkCancel(signal);
+ url = w.url();
+ if (r.preview && project) alias = previewAliasUrl(project, r.preview);
+ }
+ const record: DeployRecord = {
+ builtStampId: built.stampId,
+ builtAt: built.builtAt,
+ kind,
+ ...(r.preview ? { branch: r.preview } : {}),
+ url,
+ ...(alias ? { alias } : {}),
+ at: Date.now(),
+ liveCheck: null,
+ };
+ await recordDeploy(paths, target, record);
+ const where = kind === "local" ? "local" : kind === "preview" ? `preview "${r.preview}"` : "production";
+ return { status: "ran", stamp: built.stampId, summary: `${target} deployed (${where})${url ? ` ${url}` : ""}` };
+}
+
+// ---------------------------------------------------------------------------
+// The dispatcher
+// ---------------------------------------------------------------------------
+
+/** Run the stage `r` names (the body of every Stage's `run`). */
+export async function runStageBody(ctx: StageContext, r: StageRequest): Promise<StageOutcome> {
+ const { paths } = ctx;
+ if (r.kind === "update-index") return runUpdateIndex(ctx);
+ // Usage before state: a bad preview name is said as such, whatever is built.
+ if (r.kind.startsWith("deploy-")) {
+ if (r.preview !== undefined) {
+ const problem = previewBranchProblem(r.preview);
+ if (problem) throw new StageFailure(problem, 2);
+ }
+ if (r.preview && r.to === "local") {
+ throw new StageFailure("--preview and --to local are two different deploys", 2);
+ }
+ }
+ // The docker per-site container builds what the host's fan-out decided to
+ // build: no stamps of its own to judge by (its builds dir is scratch).
+ if (r.kind === "build-site" && inBuildContainer()) {
+ return buildOneSite(ctx, r, (await readIndexStamp(paths)) ?? CONTAINER_NO_STAMP);
+ }
+
+ const input = await readNeedsInput(paths, { mainHead: r.kind === "build-homepage" });
+ const f = STAGES[r.kind].needs(input, r);
+ if (f.state === "blocked") throw new StageFailure(f.reason, 3);
+ if (f.state === "fresh") {
+ const stampOf =
+ r.kind.startsWith("deploy-") || r.target === ALL_TARGET
+ ? (input.index.stamp?.stampId ?? "")
+ : "";
+ const built =
+ r.kind === "build-site" || r.kind === "deploy-site"
+ ? input.sites[r.target]?.built
+ : r.kind.endsWith("-hub")
+ ? input.hub.built
+ : input.homepage.built;
+ if (r.kind.startsWith("build-")) {
+ if (r.target === ALL_TARGET) {
+ for (const t of Object.values(input.sites)) await markChecked(paths, t.built);
+ } else {
+ await markChecked(paths, built);
+ }
+ }
+ return {
+ status: "noop",
+ stamp: built?.stampId ?? stampOf,
+ summary: `${r.target}: fresh — nothing to do (--force runs it anyway)`,
+ };
+ }
+ ctx.onLog(`[publish] ${STAGES[r.kind].label} ${r.target}: ${f.reason}\n`);
+ const stamp = input.index.stamp;
+
+ const b = await import("./build");
+ switch (r.kind) {
+ case "build-site":
+ if (r.target === ALL_TARGET) {
+ return r.runner === "docker" ? buildAllDocker(ctx, r, input) : buildAllLocal(ctx, r, input);
+ }
+ return buildOneSite(ctx, r, stamp!);
+ case "build-hub":
+ return buildHubStage(ctx, stamp!);
+ case "build-homepage":
+ return buildHomepageStage(ctx, stamp!);
+ case "deploy-site": {
+ const site = getSite(r.target, paths);
+ const out = b.bundleDir(paths, site.siteId);
+ return deployStage(
+ ctx,
+ r,
+ site.siteId,
+ (o) =>
+ b.deploySite(site.siteId, {
+ paths,
+ signal: ctx.signal,
+ onLog: o.onLog,
+ previewBranch: o.previewBranch,
+ outDir: out,
+ stagingDir: b.dockerSiteStagingDir(paths, site.siteId),
+ }),
+ { src: out, dest: localSiteOut(), needs: "ARCHILYZER_SITE_OUT (the directory the docker `site` service serves)" },
+ site.cloudflareProject?.trim() || null,
+ );
+ }
+ case "deploy-hub":
+ return deployStage(
+ ctx,
+ r,
+ HUB_TARGET,
+ (o) =>
+ b.deployHub({
+ paths,
+ signal: ctx.signal,
+ onLog: o.onLog,
+ previewBranch: o.previewBranch,
+ outDir: b.bundleDir(paths, HUB_TARGET),
+ }),
+ null,
+ getHomepageConfig(paths).cloudflareProject?.trim() || null,
+ );
+ case "deploy-homepage":
+ return deployStage(
+ ctx,
+ r,
+ HOMEPAGE_TARGET,
+ (o) => b.deployHomepage({ paths, signal: ctx.signal, onLog: o.onLog, previewBranch: o.previewBranch }),
+ { src: b.homepageOutDir(paths), dest: localHomepageOut(), needs: "ARCHILYZER_HOMEPAGE_OUT (the directory the docker `homepage` service serves)" },
+ b.HOMEPAGE_PAGES_PROJECT,
+ );
+ }
+}
diff --git a/common/publish/stageLock.test.ts b/common/publish/stageLock.test.ts
@@ -0,0 +1,205 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, utimesSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import type { Paths } from "../lib/paths";
+import {
+ LockWaitCancelled,
+ acquirePublishLock,
+ START_SLACK_MS,
+ holderIsGone,
+ lockHostId,
+ pidStartOf,
+ processStartedAtMs,
+ publishLockPath,
+ readLockHolder,
+ withPublishLock,
+ type LockHolder,
+} from "./stageLock";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// The publish lock (release 18): a stale holder on this host is taken over, a
+// live one is waited for (cancellably), and a holder on another host is never
+// stolen. The process probes are injected; nothing here signals a real pid.
+
+function tmp(): { root: string; paths: Paths } {
+ const root = mkdtempSync(path.join(os.tmpdir(), "stage-lock-"));
+ return { root, paths: { exportBuildsDir: path.join(root, ".export-builds") } as Paths };
+}
+
+function plant(paths: Paths, holder: Partial<LockHolder>): void {
+ mkdirSync(paths.exportBuildsDir, { recursive: true });
+ writeFileSync(
+ publishLockPath(paths),
+ JSON.stringify({ pid: 4242, host: "here", kind: "build-site", target: "jer", since: 1, ...holder }),
+ );
+}
+
+const here = { host: "here", pid: 100, startOf: () => null, startedAtMs: () => null };
+
+test("holderIsGone: same host and a dead pid, or a pid reused by a later process", () => {
+ const h: LockHolder = { pid: 7, host: "here", kind: "k", target: "t", since: 1, pidStart: "500" };
+ assert.equal(holderIsGone(h, { host: "here", isAlive: () => false }), true);
+ assert.equal(holderIsGone(h, { host: "here", isAlive: () => true, startOf: () => "500", startedAtMs: () => null }), false);
+ assert.equal(holderIsGone(h, { host: "here", isAlive: () => true, startOf: () => "900", startedAtMs: () => null }), true, "pid reused");
+ assert.equal(holderIsGone(h, { host: "here", isAlive: () => true, startOf: () => null, startedAtMs: () => null }), false);
+ assert.equal(holderIsGone(h, { host: "there", isAlive: () => false }), false, "another host: never");
+ assert.equal(holderIsGone({ ...h, pidStart: null }, { host: "here", isAlive: () => true, startOf: () => "1", startedAtMs: () => null }), false);
+});
+
+test("holderIsGone: a live pid whose process started AFTER the lock was taken is not the holder (a recreated container's pid 1)", () => {
+ const since = 1_000_000;
+ const h: LockHolder = { pid: 1, host: "archilyzer-editor", kind: "k", target: "t", since };
+ const probe = (startedAt: number | null) =>
+ holderIsGone(h, { host: "archilyzer-editor", isAlive: () => true, startOf: () => null, startedAtMs: () => startedAt });
+ assert.equal(probe(since + 60_000), true, "started a minute after the lock: stale");
+ assert.equal(probe(since - 60_000), false, "started before the lock: may be the holder");
+ assert.equal(probe(since + 1_000), false, "within btime's slack: not judged");
+ assert.equal(probe(null), false, "no /proc: pid-alive alone decides");
+});
+
+test("the host identity is ARCHILYZER_HOST_ID when set, else the hostname", () => {
+ assert.equal(lockHostId({ ARCHILYZER_HOST_ID: " archilyzer-editor " }), "archilyzer-editor");
+ assert.equal(lockHostId({ ARCHILYZER_HOST_ID: "" }), os.hostname());
+ assert.equal(lockHostId({}), os.hostname());
+});
+
+test("processStartedAtMs: this process started before now, and no such pid is null", () => {
+ if (process.platform === "linux") {
+ const t = processStartedAtMs(process.pid);
+ assert.ok(t !== null && t <= Date.now() + START_SLACK_MS && t > Date.now() - 86_400_000 * 365, String(t));
+ }
+ assert.equal(processStartedAtMs(2 ** 30), null);
+});
+
+test("pidStartOf reads this process's start time on Linux and is null for no such pid", () => {
+ if (process.platform === "linux") {
+ assert.match(pidStartOf(process.pid) ?? "", /^\d+$/);
+ assert.equal(pidStartOf(process.pid), pidStartOf(process.pid));
+ }
+ assert.equal(pidStartOf(2 ** 30), null);
+});
+
+test("the lock is taken, names its holder, and is released", async () => {
+ const { root, paths } = tmp();
+ try {
+ const lock = await acquirePublishLock(paths, { kind: "update-index", target: "_index" }, here);
+ assert.deepEqual(await readLockHolder(paths), lock.holder);
+ assert.equal(lock.holder.pid, 100);
+ assert.equal(lock.holder.kind, "update-index");
+ await lock.release();
+ assert.equal(existsSync(publishLockPath(paths)), false);
+ // withPublishLock releases on a throw too.
+ await assert.rejects(
+ withPublishLock(paths, { kind: "k", target: "t" }, async () => {
+ throw new Error("boom");
+ }, here),
+ /boom/,
+ );
+ assert.equal(existsSync(publishLockPath(paths)), false);
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+test("a stale holder on this host is taken over, and said so", async () => {
+ const { root, paths } = tmp();
+ try {
+ plant(paths, { pid: 4242, host: "here" });
+ const logs: string[] = [];
+ const lock = await acquirePublishLock(paths, { kind: "build-site", target: "ani" }, {
+ ...here,
+ isAlive: (pid) => pid !== 4242,
+ onLog: (l) => logs.push(l),
+ });
+ assert.equal(lock.holder.target, "ani");
+ assert.match(logs.join(""), /taking over a stale publish lock: build-site jer \(pid 4242 on here/);
+ await lock.release();
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+test("a live holder is waited for, with one log line, until it releases", async () => {
+ const { root, paths } = tmp();
+ try {
+ plant(paths, { pid: 4242, host: "here" });
+ const logs: string[] = [];
+ let polls = 0;
+ const taking = acquirePublishLock(paths, { kind: "build-site", target: "ani" }, {
+ ...here,
+ isAlive: () => true,
+ pollMs: 5,
+ onLog: (l) => {
+ logs.push(l);
+ },
+ now: () => {
+ polls++;
+ // The holder releases after a few polls.
+ if (polls === 6) rmSync(publishLockPath(paths));
+ return polls;
+ },
+ });
+ const lock = await taking;
+ assert.equal(lock.holder.target, "ani");
+ assert.equal(logs.length, 1, logs.join(""));
+ assert.match(logs[0], /waiting for the publish lock — held by build-site jer \(pid 4242 on here/);
+ await lock.release();
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+test("a holder on another host is never stolen; the wait is cancellable", async () => {
+ const { root, paths } = tmp();
+ try {
+ plant(paths, { pid: 4242, host: "elsewhere" });
+ const ac = new AbortController();
+ const logs: string[] = [];
+ const taking = acquirePublishLock(paths, { kind: "k", target: "t" }, {
+ ...here,
+ isAlive: () => false, // dead HERE means nothing for a pid over there
+ pollMs: 5,
+ signal: ac.signal,
+ onLog: (l) => logs.push(l),
+ });
+ setTimeout(() => ac.abort(), 40);
+ await assert.rejects(taking, LockWaitCancelled);
+ assert.equal(logs.length, 1);
+ assert.match(logs[0], /on ANOTHER host \("elsewhere"; this one is "here"\)/);
+ assert.match(logs[0], /If no publish stage is running there, remove .*\.publish\.lock/);
+ assert.equal(JSON.parse(readFileSync(publishLockPath(paths), "utf8")).host, "elsewhere", "untouched");
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+test("a torn lock file is waited for, then taken over after the grace", async () => {
+ const { root, paths } = tmp();
+ try {
+ mkdirSync(paths.exportBuildsDir, { recursive: true });
+ writeFileSync(publishLockPath(paths), "");
+ const old = new Date(Date.now() - 120_000);
+ utimesSync(publishLockPath(paths), old, old);
+ const lock = await acquirePublishLock(paths, { kind: "k", target: "t" }, { ...here, pollMs: 5 });
+ assert.equal(lock.holder.kind, "k");
+ await lock.release();
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+test("release never removes a lock somebody else holds now", async () => {
+ const { root, paths } = tmp();
+ try {
+ const lock = await acquirePublishLock(paths, { kind: "k", target: "t" }, here);
+ plant(paths, { pid: 9, host: "here", since: 99 });
+ await lock.release();
+ assert.equal(JSON.parse(readFileSync(publishLockPath(paths), "utf8")).pid, 9);
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
diff --git a/common/publish/stageLock.ts b/common/publish/stageLock.ts
@@ -0,0 +1,291 @@
+// The publish lock (release 18): ONE publish stage at a time on this machine,
+// whoever started it — a stage child the editor spawned, or `archilyzer
+// publish …` run by hand (`docker compose exec editor …` included). The
+// editor's `publish` queue already runs its stages one by one; this file is
+// what keeps a CLI started beside it from building into the same shared
+// export/public at once.
+//
+// <exportBuildsDir>/.publish.lock {pid, host, kind, target, since, pidStart}
+//
+// Taken with O_EXCL (`open(…, "wx")`). A holder is STALE — and its lock taken
+// over — only when it is on THIS host and its process is gone: the pid does
+// not answer (`processIsAlive`, jobs/bootQueuedJobs.ts), or it answers with a
+// different start time (`/proc/<pid>/stat`, where there is one: a container's
+// editor comes back with the same small pids on every restart), or the
+// process that pid names now STARTED AFTER the lock was taken (`since`) — so
+// it cannot be the holder, whatever `pidStart` the lock carries. A lock naming
+// another host is never stolen — a pid means nothing across a namespace. A
+// live holder makes the taker WAIT: a poll every 5 s, one log line, and a
+// cancel (the AbortSignal) gives up the wait.
+//
+// THE HOST is `ARCHILYZER_HOST_ID` when set, else `os.hostname()`. In the
+// container the hostname is the container id, new on every recreate — which
+// would make the last container's lock "another host's" for ever — so the
+// compose file sets a fixed ARCHILYZER_HOST_ID (release 18 S5). With a fixed
+// id a recreated container's editor is pid 1 again, alive: the start-time
+// rules above are what tell it from the holder. Where there is no /proc (not
+// Linux) neither start-time rule can be asked, and staleness is pid-alive alone.
+
+import { readFileSync } from "node:fs";
+import { mkdir, open, readFile, rm, stat } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { processIsAlive } from "../jobs/bootQueuedJobs";
+import type { Paths } from "../lib/paths";
+
+export type LockHolder = {
+ pid: number;
+ host: string;
+ kind: string;
+ target: string;
+ since: number;
+ // The holder process's start time (/proc/<pid>/stat field 22), when known.
+ pidStart?: string | null;
+};
+
+export const LOCK_POLL_MS = 5_000;
+// A lock file that stays unreadable this long was left by a taker that died
+// between creating it and writing it.
+export const LOCK_TORN_GRACE_MS = 60_000;
+
+export class LockWaitCancelled extends Error {
+ constructor() {
+ super("cancelled while waiting for the publish lock");
+ this.name = "LockWaitCancelled";
+ }
+}
+
+export function publishLockPath(paths: Pick<Paths, "exportBuildsDir">): string {
+ return path.join(paths.exportBuildsDir, ".publish.lock");
+}
+
+// /proc reports starttime in USER_HZ ticks, which Linux fixes at 100 for
+// every userspace interface whatever the kernel's HZ.
+const USER_HZ = 100;
+
+/**
+ * When `pid`'s process started, in ms since the epoch — /proc/<pid>/stat's
+ * starttime (ticks since boot) plus /proc/stat's `btime` — or null where
+ * there is no /proc. `btime` is whole seconds, so this is good to about 1 s.
+ */
+export function processStartedAtMs(pid: number): number | null {
+ const raw = pidStartOf(pid);
+ const ticks = raw === null ? NaN : Number(raw);
+ if (!Number.isFinite(ticks)) return null;
+ try {
+ const btime = /^btime (\d+)$/m.exec(readFileSync("/proc/stat", "utf8"));
+ if (!btime) return null;
+ return Number(btime[1]) * 1000 + (ticks * 1000) / USER_HZ;
+ } catch {
+ return null;
+ }
+}
+
+// The slack on "started after the lock was taken": btime's whole seconds.
+export const START_SLACK_MS = 2_000;
+
+// The host identity's override (release 18 S5 declares it in lib/envVars.ts;
+// read by name here until both slices are merged).
+const HOST_ID_NAME = "ARCHILYZER_HOST_ID";
+
+/** This machine's identity for the lock: ARCHILYZER_HOST_ID, else the hostname. */
+export function lockHostId(envVars: NodeJS.ProcessEnv = process.env): string {
+ return envVars[HOST_ID_NAME]?.trim() || os.hostname();
+}
+
+/** A process's start time from /proc (Linux), or null where there is none. */
+export function pidStartOf(pid: number): string | null {
+ try {
+ const raw = readFileSync(`/proc/${pid}/stat`, "utf8");
+ // `pid (comm) state ppid …` — comm may hold spaces and parens, so split
+ // after the LAST ')'. Field 22 (starttime) is index 19 of the rest.
+ const rest = raw.slice(raw.lastIndexOf(")") + 2).split(" ");
+ return rest[19] ?? null;
+ } catch {
+ return null;
+ }
+}
+
+export type LockEnv = {
+ host?: string;
+ pid?: number;
+ isAlive?: (pid: number) => boolean;
+ startOf?: (pid: number) => string | null;
+ // When the process `pid` names now started (ms), or null when unknown.
+ startedAtMs?: (pid: number) => number | null;
+ now?: () => number;
+};
+
+function env(e: LockEnv = {}) {
+ return {
+ host: e.host ?? lockHostId(),
+ pid: e.pid ?? process.pid,
+ isAlive: e.isAlive ?? processIsAlive,
+ startOf: e.startOf ?? pidStartOf,
+ startedAtMs: e.startedAtMs ?? processStartedAtMs,
+ now: e.now ?? Date.now,
+ };
+}
+
+/**
+ * True when `holder`'s process is certainly gone: same host, and its pid is
+ * dead or now belongs to a process that started at another time. Pure over
+ * the injected probes.
+ */
+export function holderIsGone(holder: LockHolder, e: LockEnv = {}): boolean {
+ const { host, isAlive, startOf, startedAtMs } = env(e);
+ if (holder.host !== host) return false;
+ if (!isAlive(holder.pid)) return true;
+ if (holder.pidStart) {
+ const now = startOf(holder.pid);
+ if (now !== null && now !== holder.pidStart) return true;
+ }
+ // The pid's process started after the lock was taken: not the holder.
+ const started = startedAtMs(holder.pid);
+ if (started !== null && started > holder.since + START_SLACK_MS) return true;
+ return false;
+}
+
+export function parseLockHolder(text: string): LockHolder | null {
+ try {
+ const v = JSON.parse(text) as Partial<LockHolder>;
+ if (
+ typeof v?.pid !== "number" ||
+ typeof v.host !== "string" ||
+ typeof v.kind !== "string" ||
+ typeof v.target !== "string" ||
+ typeof v.since !== "number"
+ ) {
+ return null;
+ }
+ return v as LockHolder;
+ } catch {
+ return null;
+ }
+}
+
+/** Who holds the publish lock now, or null (none, or unreadable). */
+export async function readLockHolder(paths: Pick<Paths, "exportBuildsDir">): Promise<LockHolder | null> {
+ try {
+ return parseLockHolder(await readFile(publishLockPath(paths), "utf8"));
+ } catch {
+ return null;
+ }
+}
+
+export function describeHolder(h: LockHolder): string {
+ return `${h.kind} ${h.target} (pid ${h.pid} on ${h.host}, since ${new Date(h.since).toISOString()})`;
+}
+
+export type PublishLock = { holder: LockHolder; release: () => Promise<void> };
+
+function sleep(ms: number, signal?: AbortSignal): Promise<void> {
+ return new Promise((resolve) => {
+ const t = setTimeout(done, ms);
+ function done() {
+ clearTimeout(t);
+ signal?.removeEventListener("abort", done);
+ resolve();
+ }
+ signal?.addEventListener("abort", done, { once: true });
+ });
+}
+
+/**
+ * Take the publish lock for `who`, waiting while a live holder has it.
+ * Throws LockWaitCancelled when `signal` aborts first.
+ */
+export async function acquirePublishLock(
+ paths: Pick<Paths, "exportBuildsDir">,
+ who: { kind: string; target: string },
+ opts: LockEnv & { signal?: AbortSignal; onLog?: (line: string) => void; pollMs?: number } = {},
+): Promise<PublishLock> {
+ const e = env(opts);
+ const file = publishLockPath(paths);
+ await mkdir(path.dirname(file), { recursive: true });
+ let said = false;
+ for (;;) {
+ if (opts.signal?.aborted) throw new LockWaitCancelled();
+ const holder: LockHolder = {
+ pid: e.pid,
+ host: e.host,
+ kind: who.kind,
+ target: who.target,
+ since: e.now(),
+ pidStart: e.startOf(e.pid),
+ };
+ try {
+ const fh = await open(file, "wx");
+ try {
+ await fh.writeFile(JSON.stringify(holder) + "\n");
+ await fh.sync().catch(() => {});
+ } finally {
+ await fh.close();
+ }
+ return { holder, release: () => releaseLock(file, holder) };
+ } catch (err) {
+ if ((err as NodeJS.ErrnoException).code !== "EEXIST") throw err;
+ }
+ // Someone holds it (or held it).
+ let text = "";
+ try {
+ text = await readFile(file, "utf8");
+ } catch (err) {
+ if ((err as NodeJS.ErrnoException).code === "ENOENT") continue; // released meanwhile
+ throw err;
+ }
+ const current = parseLockHolder(text);
+ if (current === null) {
+ // Torn: a taker between its create and its write — or one that died
+ // there. Give it the grace, then take the lock over.
+ const age = await stat(file).then((s) => e.now() - s.mtimeMs, () => 0);
+ if (age > LOCK_TORN_GRACE_MS) {
+ await removeIfUnchanged(file, text);
+ continue;
+ }
+ } else if (holderIsGone(current, e)) {
+ opts.onLog?.(`[publish] taking over a stale publish lock: ${describeHolder(current)} is gone\n`);
+ await removeIfUnchanged(file, text);
+ continue;
+ } else if (!said) {
+ said = true;
+ opts.onLog?.(
+ current.host === e.host
+ ? `[publish] waiting for the publish lock — held by ${describeHolder(current)}\n`
+ : `[publish] waiting for the publish lock — held by ${describeHolder(current)}, on ANOTHER host ` +
+ `("${current.host}"; this one is "${e.host}"), which this one can never judge stale. If no ` +
+ `publish stage is running there, remove ${file} (or give both the same ARCHILYZER_HOST_ID ` +
+ `when they are one machine)\n`,
+ );
+ }
+ await sleep(opts.pollMs ?? LOCK_POLL_MS, opts.signal);
+ }
+}
+
+// Remove the lock only if it still holds exactly what was judged stale.
+async function removeIfUnchanged(file: string, judged: string): Promise<void> {
+ const now = await readFile(file, "utf8").catch(() => null);
+ if (now === judged) await rm(file, { force: true });
+}
+
+async function releaseLock(file: string, holder: LockHolder): Promise<void> {
+ const current = parseLockHolder(await readFile(file, "utf8").catch(() => ""));
+ if (current && current.pid === holder.pid && current.host === holder.host && current.since === holder.since) {
+ await rm(file, { force: true });
+ }
+}
+
+/** Run `fn` holding the publish lock; always released after. */
+export async function withPublishLock<T>(
+ paths: Pick<Paths, "exportBuildsDir">,
+ who: { kind: string; target: string },
+ fn: () => Promise<T>,
+ opts: LockEnv & { signal?: AbortSignal; onLog?: (line: string) => void; pollMs?: number } = {},
+): Promise<T> {
+ const lock = await acquirePublishLock(paths, who, opts);
+ try {
+ return await fn();
+ } finally {
+ await lock.release();
+ }
+}
diff --git a/common/publish/stageRun.test.ts b/common/publish/stageRun.test.ts
@@ -0,0 +1,306 @@
+import { after, test } from "node:test";
+import assert from "node:assert/strict";
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, utimesSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import os from "node:os";
+import path from "node:path";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// The publish stages run for real over a scratch corpus (release 18): the
+// update-index body writes the stamp and keeps its id when nothing changed;
+// every other stage asks its precondition ON DISK first (exit 3), a fresh
+// target is a no-op, `--to local` copies the bundle and records it; the lock
+// is waited for and a wait is cancellable (130). No `next build` and no
+// wrangler run here: the build and Pages paths are the bundle and build tests'.
+//
+// Every path getPaths() can resolve to a place this file may write is pinned
+// under ROOT before anything calls it (the buildStats.test.ts pattern).
+const ROOT = mkdtempSync(path.join(tmpdir(), "stage-run-"));
+const PINNED: Record<string, string> = {
+ TRANSCRIPTS_DIR: path.join(ROOT, "transcripts"),
+ SAVED_VIDEOS_DIR: path.join(ROOT, "saved-videos"),
+ SITES_DIR: path.join(ROOT, "transcripts", "sites"),
+ SETTINGS_FILE: path.join(ROOT, "settings.json"),
+ EXPORT_PUBLIC_DIR: path.join(ROOT, "export", "public"),
+ EXPORT_INDEX_DIR: path.join(ROOT, "export", ".export-index"),
+ EXPORT_BUILDS_DIR: path.join(ROOT, "export", ".export-builds"),
+ EDITOR_CHANGELOG_FILE: path.join(ROOT, "editor-CHANGELOG.md"),
+ EXPORT_CHANGELOG_FILE: path.join(ROOT, "export-CHANGELOG.md"),
+ CHARTS_CONFIG_FILE: path.join(ROOT, "chart-templates.json"),
+ SEARCH_ALIASES_FILE: path.join(ROOT, "transcripts", "search-aliases.json"),
+ CURATED_TAGS_FILE: path.join(ROOT, "transcripts", "tags.json"),
+ ARCHILYZER_CONFIG_DIR: path.join(ROOT, "config"),
+ ARCHILYZER_SOURCE_SCRATCH: path.join(ROOT, "source-scratch"),
+};
+Object.assign(process.env, PINNED);
+delete process.env.ARCHILYZER_SITE_OUT;
+delete process.env.ARCHIVES_READONLY;
+after(() => rmSync(ROOT, { recursive: true, force: true }));
+
+mkdirSync(path.join(ROOT, "transcripts", "channels"), { recursive: true });
+writeFileSync(PINNED.SETTINGS_FILE, "{}");
+function writeSite(id: string, extra: Record<string, unknown> = {}): void {
+ mkdirSync(path.join(PINNED.SITES_DIR, id), { recursive: true });
+ writeFileSync(
+ path.join(PINNED.SITES_DIR, id, "site.json"),
+ JSON.stringify({ siteId: id, siteTitle: id, channels: [], ...extra }),
+ );
+}
+writeSite("jer", { cloudflareProject: "w3c-never-real" });
+
+const { getPaths } = await import("../lib/paths");
+const { runStage } = await import("./stageRun");
+const stamps = await import("./stamps");
+const { bundleDir } = await import("./build");
+const { publishLockPath } = await import("./stageLock");
+const paths = getPaths();
+
+const quiet = () => {};
+async function stage(kind: string, target: string, extra: Record<string, unknown> = {}, logs?: string[]) {
+ return runStage(
+ { kind: kind as never, target, runId: "t", ...extra },
+ { paths, onLog: (l) => logs?.push(l) ?? quiet() },
+ );
+}
+
+function plantBundle(siteId: string): void {
+ const out = bundleDir(paths, siteId);
+ mkdirSync(out, { recursive: true });
+ writeFileSync(path.join(out, "index.html"), "<!doctype html>");
+ writeFileSync(path.join(out, "site.json"), JSON.stringify({ siteId }));
+ writeFileSync(path.join(out, "corpus.json"), JSON.stringify({ site: { id: siteId }, generatedAt: "g1" }));
+}
+
+test("before any index: a build is refused 'update the index first' (exit 3), and the lock is released", async () => {
+ const logs: string[] = [];
+ const r = await stage("build-site", "jer", {}, logs);
+ assert.equal(r.code, 3);
+ assert.equal(r.message, "update the index first");
+ assert.match(logs.join(""), /\[stage\] build-site jer: REFUSED — update the index first/);
+ assert.ok(!existsSync(publishLockPath(paths)));
+ assert.equal((await stage("build-hub", "_hub")).code, 3);
+ assert.equal((await stage("build-homepage", "_homepage")).code, 3);
+});
+
+test("update-index writes the stamp; a rerun with nothing changed keeps its id; a site.json edit makes a new one", async () => {
+ const first = await stage("update-index", "_index");
+ assert.equal(first.code, 0, first.message ?? "");
+ assert.equal(first.outcome?.status, "ran");
+ const stamp = await stamps.readIndexStamp(paths);
+ assert.ok(stamp);
+ assert.equal(stamp.stampId, first.outcome?.stamp);
+ assert.deepEqual(Object.keys(stamp.sites), ["jer"]);
+ assert.match(stamp.sites.jer.inputSig, /^[0-9a-f]{40}$/);
+ assert.equal(typeof stamp.scannedAt, "number");
+
+ const again = await stage("update-index", "_index");
+ assert.equal(again.code, 0);
+ assert.equal(again.outcome?.status, "noop");
+ const same = await stamps.readIndexStamp(paths);
+ assert.equal(same?.stampId, stamp.stampId);
+ assert.equal(same?.sites.jer.inputSig, stamp.sites.jer.inputSig);
+ assert.ok(same!.builtAt >= stamp.builtAt);
+
+ writeSite("jer", { cloudflareProject: "w3c-never-real", siteDescription: "edited" });
+ const edited = await stage("update-index", "_index");
+ assert.equal(edited.outcome?.status, "ran");
+ const next = await stamps.readIndexStamp(paths);
+ assert.notEqual(next?.stampId, stamp.stampId);
+ assert.notEqual(next?.sites.jer.inputSig, stamp.sites.jer.inputSig, "site.json is in the signature");
+ assert.notEqual(next?.hubSig, stamp.hubSig, "the hub follows the stamp");
+});
+
+test("a site the index has not seen is blocked; a fresh site's build is a no-op; deploys judge the built stamp", async () => {
+ writeSite("ani");
+ assert.match((await stage("build-site", "ani")).message!, /the index has not seen site "ani"/);
+ rmSync(path.join(PINNED.SITES_DIR, "ani"), { recursive: true });
+
+ // Never built: a deploy is refused with the command that fixes it.
+ const none = await stage("deploy-site", "jer");
+ assert.equal(none.code, 3);
+ assert.equal(none.message, "no build of jer — archilyzer publish build jer");
+
+ // A bundle stamped from the current index: fresh, so building is a no-op.
+ const stamp = (await stamps.readIndexStamp(paths))!;
+ plantBundle("jer");
+ await stamps.writeBuiltStamp(paths, {
+ v: 1,
+ stampId: "b-jer",
+ target: "jer",
+ kind: "site",
+ indexStampId: stamp.stampId,
+ inputSig: stamp.sites.jer.inputSig,
+ builtAt: Date.now(),
+ commit: null,
+ branch: "main",
+ runner: "local",
+ audience: "public",
+ corpusGeneratedAt: "g1",
+ files: 3,
+ bytes: 1,
+ archivesStaged: 0,
+ });
+ const fresh = await stage("build-site", "jer");
+ assert.equal(fresh.code, 0);
+ assert.equal(fresh.outcome?.status, "noop");
+ assert.equal(fresh.outcome?.stamp, "b-jer");
+ // A no-op build records that it checked (checkedAt), and nothing else.
+ const checked = await stamps.readBuiltStamp(paths, "jer");
+ assert.equal(checked?.stampId, "b-jer");
+ assert.equal(typeof checked?.checkedAt, "number");
+ assert.equal(checked?.inputSig, stamp.sites.jer.inputSig);
+
+ // A bad preview name is usage (2), before anything else is asked.
+ assert.equal((await stage("deploy-site", "jer", { preview: "main" })).code, 2);
+ assert.equal((await stage("deploy-site", "jer", { preview: "x", to: "local" })).code, 2);
+
+ // --to local needs the site service's directory.
+ const noOut = await stage("deploy-site", "jer", { to: "local" });
+ assert.equal(noOut.code, 3);
+ assert.match(noOut.message!, /--to local needs ARCHILYZER_SITE_OUT/);
+
+ // …and with it, copies the bundle into it (its CONTENTS replaced) and records the deploy.
+ const siteOut = path.join(ROOT, "builds", "site");
+ mkdirSync(siteOut, { recursive: true });
+ writeFileSync(path.join(siteOut, "stale.html"), "old");
+ process.env.ARCHILYZER_SITE_OUT = siteOut;
+ try {
+ const local = await stage("deploy-site", "jer", { to: "local" });
+ assert.equal(local.code, 0, local.message ?? "");
+ assert.equal(local.outcome?.status, "ran");
+ assert.ok(existsSync(path.join(siteOut, "index.html")));
+ assert.ok(!existsSync(path.join(siteOut, "stale.html")));
+ const rec = stamps.deployRecordFor(await stamps.readDeployedFile(paths, "jer"), "local");
+ assert.equal(rec?.builtStampId, "b-jer");
+ assert.equal(rec?.liveCheck, null);
+ // Deployed already: a no-op, until --force.
+ assert.equal((await stage("deploy-site", "jer", { to: "local" })).outcome?.status, "noop");
+ assert.equal((await stage("deploy-site", "jer", { to: "local", force: true })).outcome?.status, "ran");
+ // builtAfter: the build this run waits on has not happened.
+ const waiting = await stage("deploy-site", "jer", { to: "local", builtAfter: Date.now() + 60_000 });
+ assert.equal(waiting.code, 3);
+ assert.match(waiting.message!, /waiting for the build of jer/);
+ } finally {
+ delete process.env.ARCHILYZER_SITE_OUT;
+ }
+
+ // Production refuses a build made on another branch.
+ const built = (await stamps.readBuiltStamp(paths, "jer"))!;
+ await stamps.writeBuiltStamp(paths, { ...built, branch: "r18/stage-core" });
+ const off = await stage("deploy-site", "jer");
+ assert.equal(off.code, 3);
+ assert.match(off.message!, /production ships only a build of main/);
+});
+
+test("a private site is never deployed, here either (exit 3)", async () => {
+ writeSite("mine", { audience: "private", cloudflareProject: "w3c-never-real" });
+ try {
+ const r = await stage("deploy-site", "mine", { to: "local" });
+ assert.equal(r.code, 3);
+ assert.match(r.message!, /Site "mine" is private/);
+ } finally {
+ rmSync(path.join(PINNED.SITES_DIR, "mine"), { recursive: true });
+ }
+});
+
+test("a live holder of the publish lock is waited for; a cancel during the wait exits 130", async () => {
+ mkdirSync(path.dirname(publishLockPath(paths)), { recursive: true });
+ writeFileSync(
+ publishLockPath(paths),
+ JSON.stringify({ pid: process.pid, host: os.hostname(), kind: "build-site", target: "x", since: Date.now() }),
+ );
+ try {
+ const ac = new AbortController();
+ const logs: string[] = [];
+ const running = runStage(
+ { kind: "build-site", target: "jer", runId: "t" },
+ { paths, signal: ac.signal, onLog: (l) => logs.push(l), lockEnv: { pollMs: 5 } },
+ );
+ setTimeout(() => ac.abort(), 50);
+ const r = await running;
+ assert.equal(r.code, 130);
+ assert.match(logs.join(""), /waiting for the publish lock — held by build-site x/);
+ assert.match(logs.join(""), /cancelled while waiting/);
+ } finally {
+ rmSync(publishLockPath(paths), { force: true });
+ }
+});
+
+test("a lock left by a dead process on this host is taken over", async () => {
+ writeFileSync(
+ publishLockPath(paths),
+ JSON.stringify({ pid: 2 ** 30, host: os.hostname(), kind: "update-index", target: "_index", since: 1 }),
+ );
+ const old = new Date(Date.now() - 1000);
+ utimesSync(publishLockPath(paths), old, old);
+ const logs: string[] = [];
+ const r = await stage("build-site", "jer", {}, logs);
+ assert.equal(r.code, 0, r.message ?? "");
+ assert.match(logs.join(""), /taking over a stale publish lock/);
+ assert.ok(!existsSync(publishLockPath(paths)));
+ assert.equal(readFileSync(path.join(bundleDir(paths, "jer"), "index.html"), "utf8"), "<!doctype html>");
+});
+
+test("stamps' commit/branch: ARCHILYZER_COMMIT / ARCHILYZER_BRANCH win over git, each on its own; no repository is null", async () => {
+ const { checkoutInfo } = await import("./stageBodies");
+ const repo = { monorepoRoot: path.resolve(path.dirname(new URL(import.meta.url).pathname), "..", "..") };
+ const fromGit = await checkoutInfo(repo, {});
+ assert.match(fromGit.commit ?? "", /^[0-9a-f]{40}$/);
+ assert.deepEqual(await checkoutInfo(repo, { ARCHILYZER_BRANCH: "main", ARCHILYZER_COMMIT: "abc" }), {
+ commit: "abc",
+ branch: "main",
+ });
+ const branchOnly = await checkoutInfo(repo, { ARCHILYZER_BRANCH: "main" });
+ assert.equal(branchOnly.branch, "main");
+ assert.equal(branchOnly.commit, fromGit.commit);
+ const noRepo = { monorepoRoot: ROOT };
+ assert.deepEqual(await checkoutInfo(noRepo, {}), { commit: null, branch: null });
+ assert.deepEqual(await checkoutInfo(noRepo, { ARCHILYZER_COMMIT: "c1", ARCHILYZER_BRANCH: "main" }), {
+ commit: "c1",
+ branch: "main",
+ });
+});
+
+test("publish deploy all: only a private site and a site with no Pages project are skipped; any other refusal fails the run (exit 1) after the rest are tried", async () => {
+ const { publishDeploy } = await import("../bin/publish");
+ writeSite("mine", { audience: "private", cloudflareProject: "w3c-never-real" });
+ writeSite("noproj");
+ writeSite("unbuilt", { cloudflareProject: "w3c-never-real" });
+ const siteOut = path.join(ROOT, "builds", "site-all");
+ process.env.ARCHILYZER_SITE_OUT = siteOut;
+ const built = (await stamps.readBuiltStamp(paths, "jer"))!;
+ await stamps.writeBuiltStamp(paths, { ...built, branch: "main" });
+ try {
+ const lines: string[] = [];
+ const out = { log: (l: string) => lines.push(l), error: (l: string) => lines.push(l) };
+ // Pages: private and no-project are skipped quietly; jer and unbuilt are
+ // TRIED (jer: never deployed there, w3c-never-real is no real project —
+ // its deploy is refused at wrangler or before; unbuilt: exit 3).
+ const local = await publishDeploy({ target: "all", to: "local", force: true, paths, signal: new AbortController().signal }, out);
+ assert.equal(local, 1, lines.join("\n"));
+ assert.ok(lines.includes("[publish] mine: skipped — " + 'Site "mine" is private (audience: private): it is built for reading on this machine and is never deployed. Build it without deploying, or set its audience to public on its Settings tab'));
+ assert.ok(!lines.some((l) => l.startsWith("[publish] noproj: skipped")), "--to local needs no project");
+ assert.ok(existsSync(path.join(siteOut, "index.html")), "jer was still deployed");
+ const local2 = stamps.deployRecordFor(await stamps.readDeployedFile(paths, "unbuilt"), "local");
+ assert.equal(local2, null, "unbuilt was refused");
+ lines.length = 0;
+ // Only never-deployable sites skipped: an all of nothing but those is exit 0.
+ rmSync(path.join(PINNED.SITES_DIR, "unbuilt"), { recursive: true });
+ rmSync(path.join(PINNED.SITES_DIR, "noproj"), { recursive: true });
+ const onlyOk = await publishDeploy({ target: "all", to: "local", paths, signal: new AbortController().signal }, out);
+ assert.equal(onlyOk, 0, lines.join("\n"));
+ // To Pages: a site with no project is skipped too. (jer's build is taken
+ // away first, so its refusal comes before any wrangler.)
+ writeSite("noproj");
+ rmSync(stamps.builtStampPath(paths, "jer"));
+ lines.length = 0;
+ const pages = await publishDeploy({ target: "all", preview: "r18", paths, signal: new AbortController().signal }, out);
+ assert.equal(pages, 1, "jer, never built, is a failure");
+ assert.ok(lines.includes("[publish] noproj: skipped — no Cloudflare Pages project"), lines.join("\n"));
+ } finally {
+ delete process.env.ARCHILYZER_SITE_OUT;
+ for (const id of ["mine", "noproj", "unbuilt"]) rmSync(path.join(PINNED.SITES_DIR, id), { recursive: true, force: true });
+ }
+});
diff --git a/common/publish/stageRun.ts b/common/publish/stageRun.ts
@@ -0,0 +1,164 @@
+// Running one publish stage (release 18): under the publish lock, with the exit
+// codes every caller reads.
+//
+// 0 ran, or a no-op (the target was fresh)
+// 1 failed
+// 2 usage (a bad flag, an unknown site, a bad preview name)
+// 3 precondition not met ("update the index first", "no build of X", …)
+// 130 cancelled (SIGTERM / SIGINT, before or during the stage)
+//
+// Two ways in. The editor spawns `stageCommand(paths, req)` — the CLI's
+// internal `stage` row — as a `runManagedCommand` job on the `publish` queue:
+// that child (`stageMain`) traps SIGTERM so a Cancel unwinds the stage, and
+// turns on runChildIntoLog's tree-kill so `next build`, wrangler and docker
+// go with it. `archilyzer publish …` calls `runStage` in its own process.
+// Either way the stage takes `<exportBuildsDir>/.publish.lock` first.
+
+import path from "node:path";
+import { killChildTreesNow, setKillChildTrees } from "../jobs/runChild";
+import { getPaths, type Paths } from "../lib/paths";
+import { StageCancelled, StageFailure } from "./stageBodies";
+import { LockWaitCancelled, acquirePublishLock, type LockEnv } from "./stageLock";
+import { STAGES, type StageOutcome, type StageRequest } from "./stages";
+
+export const STAGE_EXIT = {
+ ok: 0,
+ failed: 1,
+ usage: 2,
+ precondition: 3,
+ cancelled: 130,
+} as const;
+
+// The update-index child's heap: the index and stats builds of a large corpus
+// (what export's `build:index` / `build:stats` scripts have always set).
+export const INDEX_HEAP_MB = 8192;
+
+export type StageCommand = {
+ command: string;
+ args: string[];
+ cwd: string;
+ env: Record<string, string | undefined>;
+};
+
+/**
+ * The child the editor spawns for `req`: `<common>/node_modules/.bin/tsx
+ * bin/archilyzer.ts stage <kind> <target> [flags]`, cwd `common/`, the
+ * editor's environment (+ the heap cap for update-index). S3's
+ * `enqueueStage` hands this to `runManagedCommand` as it is.
+ */
+export function stageCommand(
+ paths: Pick<Paths, "monorepoRoot">,
+ req: StageRequest,
+ baseEnv: NodeJS.ProcessEnv = process.env,
+): StageCommand {
+ const commonDir = path.join(paths.monorepoRoot, "common");
+ const env: Record<string, string | undefined> = { ...baseEnv };
+ if (req.kind === "update-index") {
+ env.NODE_OPTIONS = [baseEnv.NODE_OPTIONS?.trim(), `--max-old-space-size=${INDEX_HEAP_MB}`]
+ .filter(Boolean)
+ .join(" ");
+ }
+ return {
+ command: path.join(commonDir, "node_modules", ".bin", "tsx"),
+ args: ["bin/archilyzer.ts", ...STAGES[req.kind].argv(req)],
+ cwd: commonDir,
+ env,
+ };
+}
+
+export type StageRunResult = { code: number; outcome: StageOutcome | null; message: string | null };
+
+function terminal(line: string): void {
+ process.stdout.write(line.endsWith("\n") ? line : `${line}\n`);
+}
+
+/**
+ * Run `req` in this process under the publish lock (waiting for a live
+ * holder). Never throws: the result carries the exit code, the outcome on
+ * 0, and the one sentence a failure or refusal ended on.
+ */
+export async function runStage(
+ req: StageRequest,
+ opts: {
+ paths?: Paths;
+ onLog?: (line: string) => void;
+ signal?: AbortSignal;
+ lockEnv?: LockEnv & { pollMs?: number };
+ } = {},
+): Promise<StageRunResult> {
+ const paths = opts.paths ?? getPaths();
+ const onLog = opts.onLog ?? terminal;
+ const signal = opts.signal ?? new AbortController().signal;
+ const name = `${req.kind} ${req.target}`;
+ let lock;
+ try {
+ lock = await acquirePublishLock(paths, { kind: req.kind, target: req.target }, {
+ ...opts.lockEnv,
+ signal,
+ onLog,
+ });
+ } catch (err) {
+ if (err instanceof LockWaitCancelled) {
+ onLog(`[stage] ${name}: cancelled while waiting for the publish lock\n`);
+ return { code: STAGE_EXIT.cancelled, outcome: null, message: err.message };
+ }
+ const message = (err as Error).message;
+ onLog(`[stage] ${name}: FAILED — ${message}\n`);
+ return { code: STAGE_EXIT.failed, outcome: null, message };
+ }
+ try {
+ const outcome = await STAGES[req.kind].run({ paths, onLog, signal }, req);
+ if (signal.aborted) throw new StageCancelled();
+ onLog(`[stage] ${name}: ${outcome.status === "noop" ? "no-op" : "done"} — ${outcome.summary}\n`);
+ return { code: STAGE_EXIT.ok, outcome, message: null };
+ } catch (err) {
+ if (err instanceof StageCancelled || signal.aborted) {
+ onLog(`[stage] ${name}: cancelled\n`);
+ return { code: STAGE_EXIT.cancelled, outcome: null, message: "cancelled" };
+ }
+ const message = (err as Error).message;
+ if (err instanceof StageFailure) {
+ const word = err.exitCode === STAGE_EXIT.failed ? "FAILED" : "REFUSED";
+ onLog(`[stage] ${name}: ${word} — ${message}\n`);
+ return { code: err.exitCode, outcome: null, message };
+ }
+ onLog(`[stage] ${name}: FAILED — ${(err as Error).stack ?? message}\n`);
+ return { code: STAGE_EXIT.failed, outcome: null, message };
+ } finally {
+ await lock.release();
+ }
+}
+
+// After a first SIGTERM the stage unwinds (its children are signalled); a
+// stage still running this long after is left to the next taker's stale-lock
+// check, and the process exits.
+const CANCEL_GRACE_MS = 15_000;
+
+/**
+ * The stage child's entry (the `stage` row): SIGTERM / SIGINT cancel the stage
+ * — a second one exits at once — and every child it runs is killed as a tree.
+ * Returns the exit code.
+ */
+export async function stageMain(req: StageRequest, opts: { paths?: Paths } = {}): Promise<number> {
+ const ac = new AbortController();
+ // Exiting without the unwind (a second signal, or the grace run out) takes
+ // the detached process groups down first: no orphan `next build`.
+ const exitNow = () => {
+ killChildTreesNow("SIGKILL");
+ process.exit(STAGE_EXIT.cancelled);
+ };
+ const onSignal = () => {
+ if (ac.signal.aborted) exitNow();
+ ac.abort();
+ setTimeout(exitNow, CANCEL_GRACE_MS).unref();
+ };
+ process.on("SIGTERM", onSignal);
+ process.on("SIGINT", onSignal);
+ setKillChildTrees(true);
+ try {
+ return (await runStage(req, { paths: opts.paths, signal: ac.signal })).code;
+ } finally {
+ process.off("SIGTERM", onSignal);
+ process.off("SIGINT", onSignal);
+ }
+}
diff --git a/common/publish/stages.test.ts b/common/publish/stages.test.ts
@@ -0,0 +1,354 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { parseArgv } from "../bin/_parseFlags";
+import type { Paths } from "../lib/paths";
+import { builtStamp, indexStamp } from "./__fixtures__/stamps";
+import type { BuiltStamp, DeployedFile, DeployRecord } from "./stamps";
+import { STAGE_EXIT, stageCommand } from "./stageRun";
+import {
+ STAGES,
+ STAGE_FLAGS,
+ STAGE_KINDS,
+ builtCheckedAt,
+ deployKindOf,
+ parseStageArgs,
+ stageArgv,
+ type Freshness,
+ type NeedsInput,
+ type StageRequest,
+ type TargetState,
+} from "./stages";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// The publish stages' needs() — one case (at least) per row of the plan's
+// "Stages" table (plans/release-18.md), plus `changedChannels`, `--force` and
+// "code newer is not stale" — and the child's argv, both ways.
+
+function target(over: Partial<TargetState> = {}): TargetState {
+ return {
+ built: null,
+ deployed: null,
+ changedChannels: [],
+ configChangedAt: null,
+ bundleProblem: null,
+ ...over,
+ };
+}
+
+function input(over: Partial<NeedsInput> = {}): NeedsInput {
+ return {
+ index: { stamp: indexStamp(), lastIngestDoneAt: null, configChangedAt: null },
+ sites: { jer: target({ built: builtStamp() }) },
+ hub: target({ built: builtStamp({ target: "_hub", kind: "hub", inputSig: "hub-1" }) }),
+ homepage: {
+ ...target({
+ built: builtStamp({ target: "_homepage", kind: "homepage", inputSig: "s1", sourceCommit: "main-1" }),
+ }),
+ mainHead: "main-1",
+ },
+ ...over,
+ };
+}
+
+const req = (kind: StageRequest["kind"], tgt: string, over: Partial<StageRequest> = {}): StageRequest => ({
+ kind,
+ target: tgt,
+ runId: "run-1",
+ ...over,
+});
+
+const needs = (s: NeedsInput, r: StageRequest): Freshness => STAGES[r.kind].needs(s, r);
+const state = (f: Freshness) => f.state;
+const reason = (f: Freshness) => (f.state === "fresh" ? "" : f.reason);
+
+function deployed(over: Partial<DeployedFile> = {}, rec: Partial<DeployRecord> = {}): DeployedFile {
+ const r: DeployRecord = {
+ builtStampId: "b1",
+ builtAt: 3_000,
+ kind: "production",
+ url: null,
+ at: 4_000,
+ liveCheck: null,
+ ...rec,
+ };
+ return { v: 1, target: "jer", production: r, previews: {}, ...over };
+}
+
+test("the table: seven stages, each a publish-<kind> job on the publish queue", () => {
+ assert.deepEqual(Object.keys(STAGES), [...STAGE_KINDS]);
+ for (const k of STAGE_KINDS) {
+ assert.equal(STAGES[k].kind, k);
+ assert.equal(STAGES[k].jobKind, `publish-${k}`);
+ assert.equal(STAGES[k].queueKey, "publish");
+ }
+});
+
+// --- update-index -------------------------------------------------------------
+
+test("update-index: fresh when a stamp exists and nothing is newer than its scan", () => {
+ assert.equal(state(needs(input(), req("update-index", "_index"))), "fresh");
+ const none = input({ index: { stamp: null, lastIngestDoneAt: null, configChangedAt: null } });
+ assert.equal(reason(needs(none, req("update-index", "_index"))), "no index stamp yet");
+ const data = input({ index: { stamp: indexStamp(), lastIngestDoneAt: 1_001, configChangedAt: null } });
+ assert.equal(reason(needs(data, req("update-index", "_index"))), "new data since the last index");
+ const older = input({ index: { stamp: indexStamp(), lastIngestDoneAt: 999, configChangedAt: 1_000 } });
+ assert.equal(state(needs(older, req("update-index", "_index"))), "fresh", "not NEWER than scannedAt");
+ const cfg = input({ index: { stamp: indexStamp(), lastIngestDoneAt: null, configChangedAt: 5_000 } });
+ assert.match(reason(needs(cfg, req("update-index", "_index"))), /a config file changed/);
+ assert.equal(reason(needs(input(), req("update-index", "_index", { force: true }))), "forced");
+});
+
+// --- build-site <id> ----------------------------------------------------------
+
+test("build-site: blocked 'update the index first' with no stamp — even forced", () => {
+ const none = input({ index: { stamp: null, lastIngestDoneAt: null, configChangedAt: null } });
+ assert.deepEqual(needs(none, req("build-site", "jer")), { state: "blocked", reason: "update the index first" });
+ assert.equal(state(needs(none, req("build-site", "jer", { force: true }))), "blocked");
+});
+
+test("build-site: fresh when the built inputSig is the stamp's, nothing changed and the bundle is sound", () => {
+ assert.equal(state(needs(input(), req("build-site", "jer"))), "fresh");
+});
+
+test("build-site: changed channels make it stale before any index runs, named", () => {
+ const s = input({ sites: { jer: target({ built: builtStamp(), changedChannels: ["a", "b", "c", "d", "e"] }) } });
+ assert.equal(reason(needs(s, req("build-site", "jer"))), "5 channels changed (a, b, c, d, …)");
+ const one = input({ sites: { jer: target({ built: builtStamp(), changedChannels: ["a"] }) } });
+ assert.equal(reason(needs(one, req("build-site", "jer"))), "1 channel changed (a)");
+});
+
+test("build-site: an inputSig mismatch is 'data changed'; a config change, a bad bundle, no build are stale", () => {
+ const sig = input({ sites: { jer: target({ built: builtStamp({ inputSig: "older" }) }) } });
+ assert.equal(reason(needs(sig, req("build-site", "jer"))), "data changed");
+ const cfg = input({ sites: { jer: target({ built: builtStamp(), configChangedAt: 3_001 }) } });
+ assert.equal(reason(needs(cfg, req("build-site", "jer"))), "config changed");
+ const bad = input({ sites: { jer: target({ built: builtStamp(), bundleProblem: "holds a build of x" }) } });
+ assert.equal(reason(needs(bad, req("build-site", "jer"))), "holds a build of x");
+ const never = input({ sites: { jer: target() } });
+ assert.equal(reason(needs(never, req("build-site", "jer"))), "never built");
+});
+
+test("build-site: --force builds a fresh site; a code change alone is NOT stale", () => {
+ assert.equal(reason(needs(input(), req("build-site", "jer", { force: true }))), "forced");
+ const code = input({ sites: { jer: target({ built: builtStamp({ commit: "an-older-commit", branch: "x" }) }) } });
+ assert.equal(state(needs(code, req("build-site", "jer"))), "fresh");
+});
+
+test("build-site: indexAfter waits for this run's index; an unknown site or one the index never saw is blocked", () => {
+ assert.equal(state(needs(input(), req("build-site", "jer", { indexAfter: 2_000 }))), "fresh");
+ assert.match(reason(needs(input(), req("build-site", "jer", { indexAfter: 2_001 }))), /waiting for the index update/);
+ assert.equal(reason(needs(input(), req("build-site", "nope"))), 'no site "nope"');
+ const unseen = input({ sites: { jer: target({ built: builtStamp() }), ani: target() } });
+ assert.match(reason(needs(unseen, req("build-site", "ani"))), /the index has not seen site "ani" — update the index first/);
+});
+
+// --- build-site _all --runner docker ------------------------------------------
+
+test("build-site _all: per site, as above", () => {
+ const s = input({
+ index: {
+ stamp: indexStamp({ sites: { jer: indexStamp().sites.jer, ani: { siteFp: null, statsFp: null, inputSig: "x" } } }),
+ lastIngestDoneAt: null,
+ configChangedAt: null,
+ },
+ sites: { jer: target({ built: builtStamp() }), ani: target() },
+ });
+ assert.equal(reason(needs(s, req("build-site", "_all", { runner: "docker" }))), "1 of 2 sites to build (ani)");
+ const allFresh = input();
+ assert.equal(state(needs(allFresh, req("build-site", "_all", { runner: "docker" }))), "fresh");
+ const none = input({ index: { stamp: null, lastIngestDoneAt: null, configChangedAt: null } });
+ assert.equal(state(needs(none, req("build-site", "_all"))), "blocked");
+});
+
+// --- deploy-site --------------------------------------------------------------
+
+test("deploy-site: fresh exactly when the record for that kind/branch names the built stamp", () => {
+ const prod = input({ sites: { jer: target({ built: builtStamp(), deployed: deployed() }) } });
+ assert.equal(state(needs(prod, req("deploy-site", "jer"))), "fresh");
+ const newer = input({ sites: { jer: target({ built: builtStamp({ stampId: "b2" }), deployed: deployed() }) } });
+ assert.equal(reason(needs(newer, req("deploy-site", "jer"))), "a newer build is not deployed");
+ // The production record says nothing about a preview, or the local copy.
+ assert.equal(reason(needs(prod, req("deploy-site", "jer", { preview: "r18" }))), 'never deployed to preview "r18"');
+ assert.equal(reason(needs(prod, req("deploy-site", "jer", { to: "local" }))), "never deployed (local)");
+ const pv = input({
+ sites: { jer: target({ built: builtStamp(), deployed: deployed({ previews: { r18: deployed().production! } }) }) },
+ });
+ assert.equal(state(needs(pv, req("deploy-site", "jer", { preview: "r18" }))), "fresh");
+ assert.equal(reason(needs(prod, req("deploy-site", "jer", { force: true }))), "forced");
+});
+
+test("deploy-site: refused with no build, a private site, no project (pages only), a bad bundle, builtAfter", () => {
+ const never = input({ sites: { jer: target() } });
+ assert.equal(reason(needs(never, req("deploy-site", "jer"))), "no build of jer — archilyzer publish build jer");
+ const priv = input({ sites: { jer: target({ built: builtStamp(), deployProblem: "Site jer is private" }) } });
+ assert.equal(reason(needs(priv, req("deploy-site", "jer", { to: "local" }))), "Site jer is private");
+ const noProject = input({ sites: { jer: target({ built: builtStamp(), pagesProblem: "no project" }) } });
+ assert.equal(reason(needs(noProject, req("deploy-site", "jer"))), "no project");
+ assert.equal(state(needs(noProject, req("deploy-site", "jer", { to: "local" }))), "stale", "local needs no project");
+ const bad = input({ sites: { jer: target({ built: builtStamp(), bundleProblem: "torn" }) } });
+ assert.deepEqual(needs(bad, req("deploy-site", "jer")), { state: "blocked", reason: "torn" });
+ assert.match(reason(needs(input(), req("deploy-site", "jer", { builtAfter: 3_001 }))), /waiting for the build of jer/);
+ assert.equal(state(needs(never, req("deploy-site", "jer", { force: true }))), "blocked", "force never deploys nothing");
+});
+
+test("deploy-site: production refuses a build made on another branch; a preview of it is fine", () => {
+ const off = input({ sites: { jer: target({ built: builtStamp({ branch: "r18/stage-core" }) }) } });
+ assert.match(reason(needs(off, req("deploy-site", "jer"))), /built from branch "r18\/stage-core"; production ships only a build of main/);
+ assert.equal(state(needs(off, req("deploy-site", "jer", { preview: "r18" }))), "stale");
+ const detached = input({ sites: { jer: target({ built: builtStamp({ branch: null }) }) } });
+ assert.match(
+ reason(needs(detached, req("deploy-site", "jer"))),
+ /built with no branch recorded \(a detached HEAD, or an image built without ARCHILYZER_BRANCH\); production ships only a build of main/,
+ "no branch is refused like another branch",
+ );
+ assert.equal(state(needs(detached, req("deploy-site", "jer", { preview: "r18" }))), "stale");
+});
+
+test("deploy-site under builtAfter: a run's NO-OP build (the bundle still matches the index this run updated) does not hold the deploy", () => {
+ // The run started at 1_500; its index ran at 2_000 (stamp.builtAt); the
+ // site's build was a no-op, so built.builtAt (500) predates the run.
+ const old = builtStamp({ builtAt: 500 });
+ const s = input({ sites: { jer: target({ built: old }) } });
+ assert.equal(state(needs(s, req("deploy-site", "jer", { builtAfter: 1_500 }))), "stale", "same inputs, same bundle");
+ // …but not when the bundle does NOT match the current index (the build failed, say).
+ const drifted = input({ sites: { jer: target({ built: builtStamp({ builtAt: 500, inputSig: "older" }) }) } });
+ assert.match(reason(needs(drifted, req("deploy-site", "jer", { builtAfter: 1_500 }))), /waiting for the build of jer/);
+ // …nor when the index itself has not run since the run began.
+ assert.match(reason(needs(s, req("deploy-site", "jer", { builtAfter: 2_500 }))), /waiting for the build of jer/);
+ // A no-op build's checkedAt counts as well.
+ const checked = input({ sites: { jer: target({ built: builtStamp({ builtAt: 500, checkedAt: 3_000, inputSig: "older" }) }) } });
+ assert.equal(state(needs(checked, req("deploy-site", "jer", { builtAfter: 2_500 }))), "stale");
+ // The hub and the homepage judge "current" by their own signatures.
+ const hub = input({ hub: target({ built: builtStamp({ target: "_hub", kind: "hub", inputSig: "hub-1", builtAt: 500 }) }) });
+ assert.equal(state(needs(hub, req("deploy-hub", "_hub", { builtAfter: 1_500 }))), "stale");
+ const home = input();
+ home.homepage.built = { ...(home.homepage.built as BuiltStamp), builtAt: 500 };
+ assert.equal(state(needs(home, req("deploy-homepage", "_homepage", { builtAfter: 1_500 }))), "stale");
+});
+
+test("changedChannels and config changes are measured against max(builtAt, checkedAt)", () => {
+ assert.equal(builtCheckedAt(builtStamp({ builtAt: 3_000 })), 3_000);
+ assert.equal(builtCheckedAt(builtStamp({ builtAt: 3_000, checkedAt: 9_000 })), 9_000);
+ const cfg = input({ sites: { jer: target({ built: builtStamp({ checkedAt: 9_000 }), configChangedAt: 5_000 }) } });
+ assert.equal(state(needs(cfg, req("build-site", "jer"))), "fresh", "a config change older than the last check");
+});
+
+// --- the hub ------------------------------------------------------------------
+
+test("build-hub: fresh when built.inputSig is the stamp's hubSig", () => {
+ assert.equal(state(needs(input(), req("build-hub", "_hub"))), "fresh");
+ const moved = input({ index: { stamp: indexStamp({ hubSig: "hub-2" }), lastIngestDoneAt: null, configChangedAt: null } });
+ assert.equal(reason(needs(moved, req("build-hub", "_hub"))), "the index or the pool it lists changed");
+ const ch = input({ hub: target({ built: builtStamp({ target: "_hub", kind: "hub", inputSig: "hub-1" }), changedChannels: ["x"] }) });
+ assert.equal(reason(needs(ch, req("build-hub", "_hub"))), "1 channel changed (x)");
+ const none = input({ index: { stamp: null, lastIngestDoneAt: null, configChangedAt: null } });
+ assert.equal(reason(needs(none, req("build-hub", "_hub"))), "update the index first");
+});
+
+test("deploy-hub: as deploy-site", () => {
+ const s = input({ hub: target({ built: builtStamp({ target: "_hub", kind: "hub" }), deployed: deployed({ target: "_hub" }) }) });
+ assert.equal(state(needs(s, req("deploy-hub", "_hub"))), "fresh");
+ assert.equal(reason(needs(input({ hub: target() }), req("deploy-hub", "_hub"))), "no build of the hub — archilyzer publish hub");
+ const noProject = input({ hub: target({ built: builtStamp(), pagesProblem: "The hub has no Cloudflare Pages project" }) });
+ assert.match(reason(needs(noProject, req("deploy-hub", "_hub"))), /no Cloudflare Pages project/);
+});
+
+// --- the homepage -------------------------------------------------------------
+
+test("build-homepage: fresh when built from the current index stamp and main has not moved", () => {
+ assert.equal(state(needs(input(), req("build-homepage", "_homepage"))), "fresh");
+ const idx = input({ index: { stamp: indexStamp({ stampId: "s2" }), lastIngestDoneAt: null, configChangedAt: null } });
+ assert.equal(reason(needs(idx, req("build-homepage", "_homepage"))), "the index was updated");
+ const moved = input();
+ moved.homepage.mainHead = "main-2";
+ assert.equal(reason(needs(moved, req("build-homepage", "_homepage"))), "main has moved since the source was published");
+ const noRepo = input();
+ noRepo.homepage.mainHead = null;
+ noRepo.homepage.built = { ...(noRepo.homepage.built as BuiltStamp), sourceCommit: null };
+ assert.equal(state(needs(noRepo, req("build-homepage", "_homepage"))), "fresh", "no repository: the index decides");
+});
+
+test("deploy-homepage: as deploy-site", () => {
+ const s = input();
+ s.homepage.deployed = deployed({ target: "_homepage" });
+ assert.equal(state(needs(s, req("deploy-homepage", "_homepage"))), "fresh");
+ s.homepage.built = null;
+ assert.equal(reason(needs(s, req("deploy-homepage", "_homepage"))), "no build of the homepage — archilyzer publish homepage");
+});
+
+test("deployKindOf: --to local, --preview, else production", () => {
+ assert.equal(deployKindOf({}), "production");
+ assert.equal(deployKindOf({ preview: "x" }), "preview");
+ assert.equal(deployKindOf({ to: "local" }), "local");
+ assert.equal(deployKindOf({ to: "pages" }), "production");
+});
+
+// --- argv -------------------------------------------------------------------
+
+test("argv: the child's command line, pinned, and parsed back to the same request", () => {
+ const r: StageRequest = {
+ kind: "deploy-site",
+ target: "jer",
+ runId: "run-9",
+ preview: "r18",
+ force: true,
+ builtAfter: 1_700_000_000_000,
+ };
+ assert.deepEqual(STAGES["deploy-site"].argv(r), [
+ "stage", "deploy-site", "jer", "--run-id", "run-9", "--preview", "r18", "--force", "--built-after", "1700000000000",
+ ]);
+ const booleans = Object.entries(STAGE_FLAGS).filter(([, k]) => k === "boolean").map(([n]) => n);
+ for (const sample of [
+ r,
+ { kind: "update-index", target: "_index", runId: "a" },
+ { kind: "build-site", target: "_all", runId: "b", runner: "docker", skipArchives: true, indexAfter: 5 },
+ { kind: "build-site", target: "ani", runId: "c", allowMissingMedia: true },
+ { kind: "deploy-homepage", target: "_homepage", runId: "d", to: "local" },
+ ] as StageRequest[]) {
+ const { positionals, flags } = parseArgv(stageArgv(sample), booleans);
+ assert.equal(positionals[0], "stage");
+ assert.deepEqual(parseStageArgs(positionals.slice(1), flags), sample);
+ }
+});
+
+test("parseStageArgs refuses what the child must never guess", () => {
+ const err = (pos: string[], flags: Record<string, string | boolean> = { "run-id": "r" }) => {
+ const out = parseStageArgs(pos, flags);
+ return "error" in out ? out.error : "";
+ };
+ assert.match(err(["build"]), /which stage\?/);
+ assert.match(err(["build-site"]), /which target\?/);
+ assert.match(err(["build-site", "jer"], {}), /--run-id is required/);
+ assert.match(err(["update-index", "jer"]), /the target is _index/);
+ assert.match(err(["build-hub", "_index"]), /the target is _hub/);
+ assert.match(err(["deploy-site", "_all"]), /the target is a site id/);
+ assert.match(err(["deploy-site", "jer"], { "run-id": "r", to: "ftp" }), /--to is pages or local/);
+ assert.match(err(["build-site", "_all"], { "run-id": "r", runner: "podman" }), /--runner is local or docker/);
+ assert.match(err(["build-site", "jer"], { "run-id": "r", "index-after": "soon" }), /--index-after is a time/);
+ assert.match(err(["deploy-site", "jer"], { "run-id": "r", preview: "x", to: "local" }), /two different deploys/);
+ assert.match(err(["build-site", "../etc"]), /which target\?/);
+});
+
+// --- the spawned command ------------------------------------------------------
+
+test("stageCommand: common's tsx running bin/archilyzer.ts stage …, cwd common/, the heap cap for update-index only", () => {
+ const paths = { monorepoRoot: "/repo" } as Paths;
+ const idx = stageCommand(paths, { kind: "update-index", target: "_index", runId: "r1" }, { PATH: "/bin" });
+ assert.deepEqual(idx, {
+ command: "/repo/common/node_modules/.bin/tsx",
+ args: ["bin/archilyzer.ts", "stage", "update-index", "_index", "--run-id", "r1"],
+ cwd: "/repo/common",
+ env: { PATH: "/bin", NODE_OPTIONS: "--max-old-space-size=8192" },
+ });
+ const kept = stageCommand(paths, { kind: "update-index", target: "_index", runId: "r1" }, { NODE_OPTIONS: "--trace-warnings" });
+ assert.equal(kept.env.NODE_OPTIONS, "--trace-warnings --max-old-space-size=8192");
+ const build = stageCommand(paths, { kind: "build-site", target: "jer", runId: "r1" }, { PATH: "/bin" });
+ assert.deepEqual(build.env, { PATH: "/bin" });
+ assert.deepEqual(build.args, ["bin/archilyzer.ts", "stage", "build-site", "jer", "--run-id", "r1"]);
+});
+
+test("the exit codes", () => {
+ assert.deepEqual(STAGE_EXIT, { ok: 0, failed: 1, usage: 2, precondition: 3, cancelled: 130 });
+});
diff --git a/common/publish/stages.ts b/common/publish/stages.ts
@@ -0,0 +1,419 @@
+// The publish stages (release 18) — the ONLY module that knows all of them.
+//
+// Publishing is seven independent, queueable stages driven by on-disk state
+// (publish/stamps.ts), like the ingest lanes: one index build shared by every
+// site build, then builds and deploys one at a time. Each stage is:
+//
+// needs(input, req) PURE: is the target fresh, stale (and why), or blocked?
+// argv(req) the child argv the editor spawns: ["stage", kind, target, …]
+// run(ctx, req) the body, in that child or in the CLI's own process
+//
+// `needs()` reads a `NeedsInput` — the minimal PublishStatus-shaped input
+// defined here. S3's status view (common/views/publishStatus.ts) satisfies it
+// from the stamps plus the job metas (`changedChannels`) and config mtimes; the
+// stage child builds one from disk alone (`readNeedsInput`, stageBodies.ts),
+// because ordering is enforced ON DISK: a stage whose precondition is not met
+// when it starts exits 3, whatever the queue believed when it enqueued it.
+//
+// Nothing here loads LMDB, next or the AWS SDK: the bodies are imported lazily.
+
+import type { Paths } from "../lib/paths";
+import {
+ ALL_TARGET,
+ HOMEPAGE_TARGET,
+ HUB_TARGET,
+ INDEX_TARGET,
+ deployRecordFor,
+ type BuiltStamp,
+ type DeployKind,
+ type DeployedFile,
+ type IndexStamp,
+} from "./stamps";
+
+export type StageKind =
+ | "update-index"
+ | "build-site"
+ | "deploy-site"
+ | "build-hub"
+ | "deploy-hub"
+ | "build-homepage"
+ | "deploy-homepage";
+
+export const STAGE_KINDS: readonly StageKind[] = [
+ "update-index",
+ "build-site",
+ "deploy-site",
+ "build-hub",
+ "deploy-hub",
+ "build-homepage",
+ "deploy-homepage",
+];
+
+export function isStageKind(v: unknown): v is StageKind {
+ return typeof v === "string" && (STAGE_KINDS as readonly string[]).includes(v);
+}
+
+export type StageRequest = {
+ kind: StageKind;
+ // "_index" | siteId | "_all" | "_hub" | "_homepage"
+ target: string;
+ runId: string;
+ preview?: string;
+ to?: "pages" | "local";
+ runner?: "local" | "docker";
+ force?: boolean;
+ skipArchives?: boolean;
+ // On-disk preconditions of a run: the index stamp (for a build) or the
+ // target's built stamp (for a deploy) must be at least this new (ms).
+ indexAfter?: number;
+ builtAfter?: number;
+ // `build site --allow-missing-media` (a report citation with no prepared
+ // media is let through compose). Not part of the plan's shape; optional.
+ allowMissingMedia?: boolean;
+};
+
+export type Freshness =
+ | { state: "fresh" }
+ | { state: "stale"; reason: string }
+ | { state: "blocked"; reason: string };
+
+export type StageOutcome = { status: "ran" | "noop"; stamp: string; summary: string };
+
+export type StageContext = {
+ paths: Paths;
+ onLog: (line: string) => void;
+ signal: AbortSignal;
+};
+
+export type Stage = {
+ kind: StageKind;
+ label: string;
+ jobKind: `publish-${StageKind}`;
+ queueKey: "publish";
+ needs(s: NeedsInput, r: StageRequest): Freshness;
+ argv(r: StageRequest): string[];
+ run(ctx: StageContext, r: StageRequest): Promise<StageOutcome>;
+};
+
+// ---------------------------------------------------------------------------
+// The input needs() reads (S3's PublishStatus satisfies it)
+// ---------------------------------------------------------------------------
+
+export type TargetState = {
+ built: BuiltStamp | null;
+ deployed: DeployedFile | null;
+ // Member channels (a site's, or every listed site's for the hub) with an
+ // ingest job ended `done` after `builtCheckedAt(built)` — the later of
+ // `builtAt` and `checkedAt`, so a no-op build clears the chip.
+ changedChannels: string[];
+ // The newest mtime (ms) of a config file this target's build reads (its
+ // site.json, tags.json, search-aliases.json, duplicates*.json), or null.
+ configChangedAt: number | null;
+ // What is wrong with the bundle on disk (builtBundleProblem / builtHubProblem
+ // / builtHomepageProblem), or null. Only asked when `built` is set.
+ bundleProblem: string | null;
+ // Why it is never deployed anywhere (a private site), or null.
+ deployProblem?: string | null;
+ // Why it cannot go to Cloudflare Pages (no project), or null.
+ pagesProblem?: string | null;
+};
+
+export type NeedsInput = {
+ index: {
+ stamp: IndexStamp | null;
+ // When the newest drainable ingest job ended `done` (ms), or null.
+ lastIngestDoneAt: number | null;
+ // The newest mtime (ms) of an index input config file (tags.json,
+ // search-aliases.json, duplicates*.json, sites/*/site.json,
+ // homepage.json, the settings file, the charts config), or null.
+ configChangedAt: number | null;
+ };
+ sites: Record<string, TargetState>;
+ hub: TargetState;
+ homepage: TargetState & {
+ // `main`'s HEAD where a repository is reachable, else null.
+ mainHead: string | null;
+ };
+};
+
+// ---------------------------------------------------------------------------
+// needs()
+// ---------------------------------------------------------------------------
+
+const FRESH: Freshness = { state: "fresh" };
+const stale = (reason: string): Freshness => ({ state: "stale", reason });
+const blocked = (reason: string): Freshness => ({ state: "blocked", reason });
+
+export const UPDATE_INDEX_FIRST = "update the index first";
+
+/** The deploy kind a request names: --to local, --preview <b>, else production. */
+export function deployKindOf(r: Pick<StageRequest, "to" | "preview">): DeployKind {
+ if (r.to === "local") return "local";
+ return r.preview ? "preview" : "production";
+}
+
+function namesList(slugs: string[], max = 4): string {
+ const shown = slugs.slice(0, max).join(", ");
+ return slugs.length > max ? `${shown}, …` : shown;
+}
+
+function needsIndex(s: NeedsInput, r: StageRequest): Freshness {
+ const stamp = s.index.stamp;
+ if (!stamp) return stale("no index stamp yet");
+ if (r.force) return stale("forced");
+ if (s.index.lastIngestDoneAt !== null && s.index.lastIngestDoneAt > stamp.scannedAt) {
+ return stale("new data since the last index");
+ }
+ if (s.index.configChangedAt !== null && s.index.configChangedAt > stamp.scannedAt) {
+ return stale("a config file changed since the last index");
+ }
+ return FRESH;
+}
+
+// The stamp a build needs, or why it is blocked.
+function indexGate(s: NeedsInput, r: StageRequest): Freshness | IndexStamp {
+ const stamp = s.index.stamp;
+ if (!stamp) return blocked(UPDATE_INDEX_FIRST);
+ if (r.indexAfter !== undefined && stamp.builtAt < r.indexAfter) {
+ return blocked("waiting for the index update this run started");
+ }
+ return stamp;
+}
+
+// Stale reasons shared by the three builds, after the target's own signature.
+/**
+ * When the bundle was last known to match its inputs: built, or found fresh
+ * by a later no-op build (`checkedAt`). What `changedChannels` and a config
+ * change are measured against.
+ */
+export function builtCheckedAt(built: BuiltStamp): number {
+ return Math.max(built.builtAt, built.checkedAt ?? 0);
+}
+
+function builtStale(t: TargetState, sigMatches: boolean, sigReason: string): Freshness {
+ const built = t.built!;
+ if (t.changedChannels.length > 0) {
+ const n = t.changedChannels.length;
+ return stale(`${n} channel${n === 1 ? "" : "s"} changed (${namesList(t.changedChannels)})`);
+ }
+ if (t.configChangedAt !== null && t.configChangedAt > builtCheckedAt(built)) return stale("config changed");
+ if (!sigMatches) return stale(sigReason);
+ if (t.bundleProblem) return stale(t.bundleProblem);
+ return FRESH;
+}
+
+function needsBuildSite(s: NeedsInput, r: StageRequest): Freshness {
+ const gate = indexGate(s, r);
+ if ("state" in gate) return gate;
+ if (r.target === ALL_TARGET) {
+ const ids = Object.keys(s.sites).sort();
+ const staleIds = ids.filter((id) => needsBuildSite(s, { ...r, target: id }).state !== "fresh");
+ if (staleIds.length === 0) return FRESH;
+ return stale(`${staleIds.length} of ${ids.length} sites to build (${namesList(staleIds)})`);
+ }
+ const t = s.sites[r.target];
+ if (!t) return blocked(`no site "${r.target}"`);
+ const entry = gate.sites[r.target];
+ if (!entry) return blocked(`the index has not seen site "${r.target}" — ${UPDATE_INDEX_FIRST}`);
+ if (r.force) return stale("forced");
+ if (!t.built) return stale("never built");
+ return builtStale(t, t.built.inputSig === entry.inputSig, "data changed");
+}
+
+function needsBuildHub(s: NeedsInput, r: StageRequest): Freshness {
+ const gate = indexGate(s, r);
+ if ("state" in gate) return gate;
+ if (r.force) return stale("forced");
+ if (!s.hub.built) return stale("never built");
+ return builtStale(s.hub, s.hub.built.inputSig === gate.hubSig, "the index or the pool it lists changed");
+}
+
+function needsBuildHomepage(s: NeedsInput, r: StageRequest): Freshness {
+ const gate = indexGate(s, r);
+ if ("state" in gate) return gate;
+ const h = s.homepage;
+ if (r.force) return stale("forced");
+ if (!h.built) return stale("never built");
+ if (h.built.indexStampId !== gate.stampId) return stale("the index was updated");
+ if (h.mainHead !== null && (h.built.sourceCommit ?? null) !== h.mainHead) {
+ return stale("main has moved since the source was published");
+ }
+ if (h.bundleProblem) return stale(h.bundleProblem);
+ return FRESH;
+}
+
+// Is `built` what the CURRENT index would build? (Same inputs, same bundle.)
+function builtFromCurrentIndex(s: NeedsInput, kind: "site" | "hub" | "homepage", id: string): boolean {
+ const stamp = s.index.stamp;
+ const built = kind === "site" ? s.sites[id]?.built : kind === "hub" ? s.hub.built : s.homepage.built;
+ if (!stamp || !built) return false;
+ if (kind === "site") return stamp.sites[id] !== undefined && built.inputSig === stamp.sites[id].inputSig;
+ if (kind === "hub") return built.inputSig === stamp.hubSig;
+ return built.indexStampId === stamp.stampId;
+}
+
+function needsDeploy(
+ t: TargetState | undefined,
+ name: string,
+ buildCmd: string,
+ r: StageRequest,
+ // The bundle matches the current index, and that index ran at or after
+ // `builtAfter` (a run's no-op build: nothing was rebuilt because nothing
+ // needed to be).
+ currentSince: (after: number) => boolean,
+): Freshness {
+ if (!t) return blocked(`no site "${name}"`);
+ const kind = deployKindOf(r);
+ const never = t.deployProblem ?? (kind === "local" ? null : (t.pagesProblem ?? null));
+ if (never) return blocked(never);
+ const built = t.built;
+ if (!built) return blocked(`no build of ${name} — ${buildCmd}`);
+ if (
+ r.builtAfter !== undefined &&
+ builtCheckedAt(built) < r.builtAfter &&
+ !currentSince(r.builtAfter)
+ ) {
+ return blocked(`waiting for the build of ${name} this run started`);
+ }
+ if (t.bundleProblem) return blocked(t.bundleProblem);
+ // Production ships only a build of main. A null branch (a detached HEAD, or
+ // an image built without ARCHILYZER_BRANCH) is refused the same way.
+ if (kind === "production" && built.branch !== "main") {
+ return blocked(
+ built.branch === null
+ ? `${name} was built with no branch recorded (a detached HEAD, or an image built without ARCHILYZER_BRANCH); production ships only a build of main (deploy it as a preview)`
+ : `${name} was built from branch "${built.branch}"; production ships only a build of main (deploy it as a preview)`,
+ );
+ }
+ if (r.force) return stale("forced");
+ const rec = deployRecordFor(t.deployed, kind, r.preview);
+ if (!rec) return stale(kind === "preview" ? `never deployed to preview "${r.preview}"` : `never deployed (${kind})`);
+ if (rec.builtStampId !== built.stampId) return stale("a newer build is not deployed");
+ return FRESH;
+}
+
+// ---------------------------------------------------------------------------
+// argv (the child's command line) and its parser
+// ---------------------------------------------------------------------------
+
+export function stageArgv(r: StageRequest): string[] {
+ const out = ["stage", r.kind, r.target, "--run-id", r.runId];
+ if (r.preview) out.push("--preview", r.preview);
+ if (r.to) out.push("--to", r.to);
+ if (r.runner) out.push("--runner", r.runner);
+ if (r.force) out.push("--force");
+ if (r.skipArchives) out.push("--skip-archives");
+ if (r.allowMissingMedia) out.push("--allow-missing-media");
+ if (r.indexAfter !== undefined) out.push("--index-after", String(r.indexAfter));
+ if (r.builtAfter !== undefined) out.push("--built-after", String(r.builtAfter));
+ return out;
+}
+
+/** The flags the `stage` row accepts (archilyzer.ts), by kind. */
+export const STAGE_FLAGS = {
+ "run-id": "string",
+ preview: "string",
+ to: "string",
+ runner: "string",
+ force: "boolean",
+ "skip-archives": "boolean",
+ "allow-missing-media": "boolean",
+ "index-after": "string",
+ "built-after": "string",
+} as const;
+
+const TARGET_RE = /^(?:_index|_all|_hub|_homepage|[a-z0-9][a-z0-9-]*)$/;
+
+/** The default target of a kind that has only one. */
+export function fixedTarget(kind: StageKind): string | null {
+ if (kind === "update-index") return INDEX_TARGET;
+ if (kind === "build-hub" || kind === "deploy-hub") return HUB_TARGET;
+ if (kind === "build-homepage" || kind === "deploy-homepage") return HOMEPAGE_TARGET;
+ return null;
+}
+
+/**
+ * The StageRequest a `stage <kind> <target> [flags]` command line names, or
+ * the usage problem. The inverse of `stageArgv`.
+ */
+export function parseStageArgs(
+ positionals: string[],
+ flags: Record<string, string | boolean | undefined>,
+): StageRequest | { error: string } {
+ const [kind, target] = positionals;
+ if (!isStageKind(kind)) return { error: `stage: which stage? one of ${STAGE_KINDS.join(", ")}` };
+ if (!target || !TARGET_RE.test(target)) return { error: `stage ${kind}: which target?` };
+ const fixed = fixedTarget(kind);
+ if (fixed !== null && target !== fixed) return { error: `stage ${kind}: the target is ${fixed}` };
+ if (fixed === null && (target.startsWith("_") && !(kind === "build-site" && target === ALL_TARGET))) {
+ return { error: `stage ${kind}: the target is a site id` };
+ }
+ const runId = typeof flags["run-id"] === "string" ? flags["run-id"] : "";
+ if (!runId) return { error: `stage ${kind}: --run-id is required` };
+ const r: StageRequest = { kind, target, runId };
+ if (typeof flags.preview === "string") r.preview = flags.preview;
+ if (flags.to !== undefined) {
+ if (flags.to !== "pages" && flags.to !== "local") return { error: `stage ${kind}: --to is pages or local` };
+ r.to = flags.to;
+ }
+ if (flags.runner !== undefined) {
+ if (flags.runner !== "local" && flags.runner !== "docker") return { error: `stage ${kind}: --runner is local or docker` };
+ r.runner = flags.runner;
+ }
+ if (flags.force === true) r.force = true;
+ if (flags["skip-archives"] === true) r.skipArchives = true;
+ if (flags["allow-missing-media"] === true) r.allowMissingMedia = true;
+ for (const [flag, key] of [
+ ["index-after", "indexAfter"],
+ ["built-after", "builtAfter"],
+ ] as const) {
+ const v = flags[flag];
+ if (v === undefined) continue;
+ const n = typeof v === "string" ? Number(v) : NaN;
+ if (!Number.isFinite(n)) return { error: `stage ${kind}: --${flag} is a time in ms` };
+ r[key] = n;
+ }
+ if (r.preview && r.to === "local") return { error: `stage ${kind}: --preview and --to local are two different deploys` };
+ return r;
+}
+
+// ---------------------------------------------------------------------------
+// The table
+// ---------------------------------------------------------------------------
+
+function stage(
+ kind: StageKind,
+ label: string,
+ needs: (s: NeedsInput, r: StageRequest) => Freshness,
+): Stage {
+ return {
+ kind,
+ label,
+ jobKind: `publish-${kind}`,
+ queueKey: "publish",
+ needs,
+ argv: stageArgv,
+ run: async (ctx, r) => (await import("./stageBodies")).runStageBody(ctx, r),
+ };
+}
+
+export const STAGES: Record<StageKind, Stage> = {
+ "update-index": stage("update-index", "Update the index", needsIndex),
+ "build-site": stage("build-site", "Build site", needsBuildSite),
+ "deploy-site": stage("deploy-site", "Deploy site", (s, r) =>
+ needsDeploy(s.sites[r.target], r.target, `archilyzer publish build ${r.target}`, r, (after) =>
+ builtFromCurrentIndex(s, "site", r.target) && (s.index.stamp?.builtAt ?? 0) >= after)),
+ "build-hub": stage("build-hub", "Build hub", needsBuildHub),
+ "deploy-hub": stage("deploy-hub", "Deploy hub", (s, r) =>
+ needsDeploy(s.hub, "the hub", "archilyzer publish hub", r, (after) =>
+ builtFromCurrentIndex(s, "hub", "_hub") && (s.index.stamp?.builtAt ?? 0) >= after)),
+ "build-homepage": stage("build-homepage", "Build homepage", needsBuildHomepage),
+ "deploy-homepage": stage("deploy-homepage", "Deploy homepage", (s, r) =>
+ needsDeploy(s.homepage, "the homepage", "archilyzer publish homepage", r, (after) =>
+ builtFromCurrentIndex(s, "homepage", "_homepage") && (s.index.stamp?.builtAt ?? 0) >= after)),
+};
+
+/** The job kind a stage runs as on the editor's `publish` queue. */
+export function stageJobKind(kind: StageKind): `publish-${StageKind}` {
+ return STAGES[kind].jobKind;
+}
diff --git a/common/publish/stamps.test.ts b/common/publish/stamps.test.ts
@@ -0,0 +1,138 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import type { Paths } from "../lib/paths";
+import {
+ asBuiltStamp,
+ asIndexStamp,
+ builtStampPath,
+ deployRecordFor,
+ deployedPath,
+ imageBuildFacts,
+ indexStampPath,
+ newStampId,
+ readBuiltStamp,
+ readDeployedFile,
+ readIndexStamp,
+ recordDeploy,
+ writeBuiltStamp,
+ writeIndexStamp,
+ type DeployRecord,
+} from "./stamps";
+import { builtStamp, indexStamp } from "./__fixtures__/stamps";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// The publish stamps (release 18): a round trip through disk, and the
+// tolerance every `needs()` relies on — a missing, unparseable or
+// wrongly-shaped stamp reads as null (stale), never as half a stamp.
+
+function tmpPaths(): { root: string; paths: Paths } {
+ const root = mkdtempSync(path.join(os.tmpdir(), "stamps-"));
+ return {
+ root,
+ paths: {
+ exportIndexDir: path.join(root, ".export-index"),
+ exportBuildsDir: path.join(root, ".export-builds"),
+ } as Paths,
+ };
+}
+
+test("the three stamps live where the plan says", () => {
+ const p = { exportIndexDir: "/e/.export-index", exportBuildsDir: "/e/.export-builds" } as Paths;
+ assert.equal(indexStampPath(p), "/e/.export-index/stamp.json");
+ assert.equal(builtStampPath(p, "jer"), "/e/.export-builds/jer/built.json");
+ assert.equal(builtStampPath(p, "_hub"), "/e/.export-builds/_hub/built.json");
+ assert.equal(deployedPath(p, "_homepage"), "/e/.export-builds/_homepage/deployed.json");
+});
+
+test("stamps round-trip through disk", async () => {
+ const { root, paths } = tmpPaths();
+ try {
+ assert.equal(await readIndexStamp(paths), null, "no stamp yet");
+ await writeIndexStamp(paths, indexStamp());
+ assert.deepEqual(await readIndexStamp(paths), indexStamp());
+ await writeBuiltStamp(paths, builtStamp({ sourceCommit: null }));
+ assert.deepEqual(await readBuiltStamp(paths, "jer"), builtStamp({ sourceCommit: null }));
+ assert.equal(await readBuiltStamp(paths, "other"), null);
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+test("a malformed or wrongly-shaped stamp reads as null", async () => {
+ const { root, paths } = tmpPaths();
+ try {
+ mkdirSync(paths.exportIndexDir, { recursive: true });
+ writeFileSync(indexStampPath(paths), "{not json");
+ assert.equal(await readIndexStamp(paths), null);
+ writeFileSync(indexStampPath(paths), JSON.stringify({ ...indexStamp(), v: 2 }));
+ assert.equal(await readIndexStamp(paths), null, "another version");
+ writeFileSync(indexStampPath(paths), JSON.stringify({ ...indexStamp(), hubSig: 3 }));
+ assert.equal(await readIndexStamp(paths), null, "a field of the wrong type");
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+ assert.equal(asIndexStamp(null), null);
+ assert.equal(asIndexStamp([]), null);
+ assert.equal(asIndexStamp({ ...indexStamp(), sites: { x: { inputSig: 1 } } }), null);
+ assert.equal(asIndexStamp({ ...indexStamp(), index: { ...indexStamp().index, heldChannels: [1] } }), null);
+ assert.equal(asBuiltStamp({ ...builtStamp(), kind: "export" }), null);
+ assert.equal(asBuiltStamp({ ...builtStamp(), runner: "podman" }), null);
+ assert.equal(asBuiltStamp({ ...builtStamp(), files: "10" }), null);
+ assert.equal(asBuiltStamp({ ...builtStamp(), sourceCommit: 5 }), null);
+ const { stampId: _drop, ...noId } = builtStamp();
+ assert.equal(asBuiltStamp(noId), null);
+});
+
+test("recordDeploy keeps every other record, and deployRecordFor finds each kind", async () => {
+ const { root, paths } = tmpPaths();
+ const rec = (over: Partial<DeployRecord>): DeployRecord => ({
+ builtStampId: "b1",
+ builtAt: 3_000,
+ kind: "production",
+ url: "https://x.pages.dev",
+ at: 4_000,
+ liveCheck: null,
+ ...over,
+ });
+ try {
+ await recordDeploy(paths, "jer", rec({}));
+ await recordDeploy(paths, "jer", rec({ kind: "preview", branch: "r18", url: "https://r18.x.pages.dev" }));
+ await recordDeploy(paths, "jer", rec({ kind: "local", url: null }));
+ await recordDeploy(paths, "jer", rec({ kind: "preview", branch: "r19", builtStampId: "b2" }));
+ const file = await readDeployedFile(paths, "jer");
+ assert.equal(deployRecordFor(file, "production")?.url, "https://x.pages.dev");
+ assert.equal(deployRecordFor(file, "preview", "r18")?.url, "https://r18.x.pages.dev");
+ assert.equal(deployRecordFor(file, "preview", "r19")?.builtStampId, "b2");
+ assert.equal(deployRecordFor(file, "preview", "nope"), null);
+ assert.equal(deployRecordFor(file, "preview"), null);
+ assert.equal(deployRecordFor(file, "local")?.url, null);
+ assert.equal(deployRecordFor(null, "production"), null);
+ // A malformed file is replaced, not merged.
+ writeFileSync(deployedPath(paths, "jer"), "[]");
+ assert.equal(await readDeployedFile(paths, "jer"), null);
+ await recordDeploy(paths, "jer", rec({ builtStampId: "b3" }));
+ const again = JSON.parse(readFileSync(deployedPath(paths, "jer"), "utf8"));
+ assert.deepEqual(Object.keys(again.previews), []);
+ assert.equal(again.production.builtStampId, "b3");
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+test("imageBuildFacts: the image's commit and branch, empty = null", () => {
+ assert.deepEqual(imageBuildFacts({ ARCHILYZER_COMMIT: " abc ", ARCHILYZER_BRANCH: "" }), { commit: "abc", branch: null });
+ assert.deepEqual(imageBuildFacts({}), { commit: null, branch: null });
+ assert.deepEqual(imageBuildFacts({ ARCHILYZER_COMMIT: "c", ARCHILYZER_BRANCH: "main" }), { commit: "c", branch: "main" });
+});
+
+test("newStampId is unique and sorts by time", () => {
+ const a = newStampId(1_000);
+ const b = newStampId(2_000);
+ assert.notEqual(newStampId(1_000), a);
+ assert.ok(a < b);
+});
diff --git a/common/publish/stamps.ts b/common/publish/stamps.ts
@@ -0,0 +1,271 @@
+// The publish stages' on-disk state (release 18): three stamp files, read by
+// every stage's `needs()` and written only by the stage that owns them.
+//
+// <exportIndexDir>/stamp.json IndexStamp (update-index)
+// <exportBuildsDir>/<target>/built.json BuiltStamp (build-site/hub/homepage)
+// <exportBuildsDir>/<target>/deployed.json DeployedFile (deploy-*)
+//
+// `target` is a site id, "_hub" or "_homepage". Every write is atomic (temp +
+// rename, lib/jsonFile-server.ts); every read is TOLERANT: a missing,
+// unparseable or wrongly-shaped file is null, and null means "stale" to every
+// `needs()` — a stage never trusts half a stamp.
+//
+// The shapes are the plan's ("Model", plans/release-18.md) and other slices
+// code against them: S2's live check fills `DeployRecord.liveCheck` with the
+// `LiveCheck` / `Probe` types declared here, S3's status view reads all three.
+
+import { randomBytes } from "node:crypto";
+import path from "node:path";
+import { readJsonFile, writeJsonAtomic } from "../lib/jsonFile-server";
+import type { Paths } from "../lib/paths";
+
+export const HUB_TARGET = "_hub";
+export const HOMEPAGE_TARGET = "_homepage";
+export const INDEX_TARGET = "_index";
+export const ALL_TARGET = "_all";
+
+export type IndexStamp = {
+ v: 1;
+ stampId: string;
+ // LMDB meta `generation` after the build (buildIndex.ts).
+ generation: number;
+ // LMDB meta INDEX_SCANNED_AT_KEY: when the completed scan began (ms).
+ scannedAt: number;
+ // When the stage finished (ms).
+ builtAt: number;
+ // When `build templates` finished (ms).
+ templatesAt: number;
+ commit: string | null;
+ index: {
+ shortCircuited: boolean;
+ added: number;
+ changed: number;
+ removed: number;
+ heldChannels: string[];
+ };
+ stats: { shortCircuited: boolean; notIndexedYet: number; notIndexable: number };
+ // Per site: sha1 of the LMDB `siteFp:<id>` / `statsFp:<id>` fingerprints
+ // (null when absent), and the inputSig compose's skip rule is computed from.
+ sites: Record<string, { siteFp: string | null; statsFp: string | null; inputSig: string }>;
+ hubSig: string;
+};
+
+export type BuiltKind = "site" | "hub" | "homepage";
+export type Runner = "local" | "docker";
+
+export type BuiltStamp = {
+ v: 1;
+ stampId: string;
+ target: string;
+ kind: BuiltKind;
+ // The IndexStamp the bundle was built from (null: none on disk then).
+ indexStampId: string | null;
+ inputSig: string;
+ builtAt: number;
+ // When a later no-op build last found this bundle still matching its inputs
+ // (absent: never). `changedChannels` is measured against max(builtAt, it).
+ checkedAt?: number;
+ commit: string | null;
+ branch: string | null;
+ runner: Runner;
+ audience: "public" | "private";
+ // The bundle's corpus.json `generatedAt` (what the live check compares).
+ corpusGeneratedAt: string | null;
+ files: number;
+ bytes: number;
+ // Oversize archives staged for R2 beside the bundle.
+ archivesStaged: number;
+ // The homepage only: the commit its published source was cut from.
+ sourceCommit?: string | null;
+};
+
+export type Probe = {
+ status: number | null;
+ generatedAt?: string;
+ cfCacheStatus?: string;
+ age?: number;
+ cacheControl?: string;
+ error?: string;
+};
+
+export type LiveCheck = {
+ at: number;
+ url: string;
+ plain: Probe;
+ busted: Probe;
+ expected: string | null;
+ verdict: "ok" | "stale-edge" | "mismatch" | "unreachable" | "skipped";
+ tombstones?: { path: string; plain: Probe; busted: Probe; ok: boolean }[];
+};
+
+export type DeployKind = "production" | "preview" | "local";
+
+export type DeployRecord = {
+ builtStampId: string;
+ builtAt: number;
+ kind: DeployKind;
+ branch?: string;
+ url: string | null;
+ alias?: string;
+ at: number;
+ wrangler?: string;
+ liveCheck: LiveCheck | null;
+};
+
+export type DeployedFile = {
+ v: 1;
+ target: string;
+ production?: DeployRecord;
+ local?: DeployRecord;
+ previews: Record<string, DeployRecord>;
+};
+
+// --- paths --------------------------------------------------------------------
+
+export function indexStampPath(paths: Pick<Paths, "exportIndexDir">): string {
+ return path.join(paths.exportIndexDir, "stamp.json");
+}
+
+export function targetDir(paths: Pick<Paths, "exportBuildsDir">, target: string): string {
+ return path.join(paths.exportBuildsDir, target);
+}
+
+export function builtStampPath(paths: Pick<Paths, "exportBuildsDir">, target: string): string {
+ return path.join(targetDir(paths, target), "built.json");
+}
+
+export function deployedPath(paths: Pick<Paths, "exportBuildsDir">, target: string): string {
+ return path.join(targetDir(paths, target), "deployed.json");
+}
+
+// The runtime image's build facts (Dockerfile build args → ENV): the stamps'
+// `commit` / `branch` where there is no .git to ask. Declared, with the two
+// names, in lib/envVars.ts; re-exported here for the stage bodies.
+export { imageBuildFacts } from "../lib/envVars";
+
+/** A fresh, sortable, unique stamp id. */
+export function newStampId(now = Date.now()): string {
+ return `${now.toString(36).padStart(9, "0")}-${randomBytes(4).toString("hex")}`;
+}
+
+// --- shape checks (pure; exported for the tests) -------------------------------
+
+type Obj = Record<string, unknown>;
+const isObj = (v: unknown): v is Obj => typeof v === "object" && v !== null && !Array.isArray(v);
+const isStr = (v: unknown): v is string => typeof v === "string";
+const isNum = (v: unknown): v is number => typeof v === "number" && Number.isFinite(v);
+const isStrOrNull = (v: unknown) => v === null || isStr(v);
+const isBool = (v: unknown): v is boolean => typeof v === "boolean";
+
+export function asIndexStamp(v: unknown): IndexStamp | null {
+ if (!isObj(v) || v.v !== 1) return null;
+ if (!isStr(v.stampId) || !isNum(v.generation) || !isNum(v.scannedAt)) return null;
+ if (!isNum(v.builtAt) || !isNum(v.templatesAt) || !isStrOrNull(v.commit)) return null;
+ const ix = v.index;
+ if (!isObj(ix) || !isBool(ix.shortCircuited) || !isNum(ix.added) || !isNum(ix.changed)) return null;
+ if (!isNum(ix.removed) || !Array.isArray(ix.heldChannels) || !ix.heldChannels.every(isStr)) return null;
+ const st = v.stats;
+ if (!isObj(st) || !isBool(st.shortCircuited) || !isNum(st.notIndexedYet) || !isNum(st.notIndexable)) {
+ return null;
+ }
+ if (!isObj(v.sites) || !isStr(v.hubSig)) return null;
+ for (const s of Object.values(v.sites)) {
+ if (!isObj(s) || !isStr(s.inputSig) || !isStrOrNull(s.siteFp) || !isStrOrNull(s.statsFp)) return null;
+ }
+ return v as unknown as IndexStamp;
+}
+
+export function asBuiltStamp(v: unknown): BuiltStamp | null {
+ if (!isObj(v) || v.v !== 1) return null;
+ if (!isStr(v.stampId) || !isStr(v.target) || !isStr(v.inputSig) || !isNum(v.builtAt)) return null;
+ if (v.kind !== "site" && v.kind !== "hub" && v.kind !== "homepage") return null;
+ if (v.runner !== "local" && v.runner !== "docker") return null;
+ if (v.audience !== "public" && v.audience !== "private") return null;
+ if (!isStrOrNull(v.indexStampId) || !isStrOrNull(v.commit) || !isStrOrNull(v.branch)) return null;
+ if (!isStrOrNull(v.corpusGeneratedAt)) return null;
+ if (!isNum(v.files) || !isNum(v.bytes) || !isNum(v.archivesStaged)) return null;
+ if (v.sourceCommit !== undefined && !isStrOrNull(v.sourceCommit)) return null;
+ if (v.checkedAt !== undefined && !isNum(v.checkedAt)) return null;
+ return v as unknown as BuiltStamp;
+}
+
+function asDeployRecord(v: unknown): DeployRecord | null {
+ if (!isObj(v)) return null;
+ if (!isStr(v.builtStampId) || !isNum(v.builtAt) || !isNum(v.at)) return null;
+ if (v.kind !== "production" && v.kind !== "preview" && v.kind !== "local") return null;
+ if (!isStrOrNull(v.url)) return null;
+ if (v.liveCheck !== null && !isObj(v.liveCheck)) return null;
+ return v as unknown as DeployRecord;
+}
+
+export function asDeployedFile(v: unknown): DeployedFile | null {
+ if (!isObj(v) || v.v !== 1 || !isStr(v.target) || !isObj(v.previews)) return null;
+ for (const key of ["production", "local"] as const) {
+ if (v[key] !== undefined && asDeployRecord(v[key]) === null) return null;
+ }
+ for (const r of Object.values(v.previews)) if (asDeployRecord(r) === null) return null;
+ return v as unknown as DeployedFile;
+}
+
+// --- read / write -------------------------------------------------------------
+
+async function readAs<T>(file: string, as: (v: unknown) => T | null): Promise<T | null> {
+ const r = await readJsonFile(file);
+ return r.ok ? as(r.value) : null;
+}
+
+export function readIndexStamp(paths: Pick<Paths, "exportIndexDir">): Promise<IndexStamp | null> {
+ return readAs(indexStampPath(paths), asIndexStamp);
+}
+
+export function writeIndexStamp(paths: Pick<Paths, "exportIndexDir">, stamp: IndexStamp): Promise<void> {
+ return writeJsonAtomic(indexStampPath(paths), stamp, { mkdir: true });
+}
+
+export function readBuiltStamp(
+ paths: Pick<Paths, "exportBuildsDir">,
+ target: string,
+): Promise<BuiltStamp | null> {
+ return readAs(builtStampPath(paths, target), asBuiltStamp);
+}
+
+export function writeBuiltStamp(paths: Pick<Paths, "exportBuildsDir">, stamp: BuiltStamp): Promise<void> {
+ return writeJsonAtomic(builtStampPath(paths, stamp.target), stamp, { mkdir: true });
+}
+
+export function readDeployedFile(
+ paths: Pick<Paths, "exportBuildsDir">,
+ target: string,
+): Promise<DeployedFile | null> {
+ return readAs(deployedPath(paths, target), asDeployedFile);
+}
+
+/** The record a deploy of `kind` (and `branch`, for a preview) left, or null. */
+export function deployRecordFor(
+ file: DeployedFile | null,
+ kind: DeployKind,
+ branch?: string,
+): DeployRecord | null {
+ if (!file) return null;
+ if (kind === "production") return file.production ?? null;
+ if (kind === "local") return file.local ?? null;
+ return branch ? (file.previews[branch] ?? null) : null;
+}
+
+/**
+ * Record one deploy into `<target>/deployed.json`, keeping every other record
+ * (a malformed file is replaced). Returns the file written.
+ */
+export async function recordDeploy(
+ paths: Pick<Paths, "exportBuildsDir">,
+ target: string,
+ record: DeployRecord,
+): Promise<DeployedFile> {
+ const prev = (await readDeployedFile(paths, target)) ?? { v: 1 as const, target, previews: {} };
+ const next: DeployedFile = { ...prev, v: 1, target, previews: { ...prev.previews } };
+ if (record.kind === "production") next.production = record;
+ else if (record.kind === "local") next.local = record;
+ else if (record.branch) next.previews[record.branch] = record;
+ await writeJsonAtomic(deployedPath(paths, target), next, { mkdir: true });
+ return next;
+}
diff --git a/docker-compose.source.yml b/docker-compose.source.yml
@@ -0,0 +1,36 @@
+# Overlay: let the container publish the homepage's /source mirror.
+#
+# docker compose -f docker-compose.yml -f docker-compose.source.yml up -d
+#
+# `archilyzer source publish` (run by the homepage build) mirrors this repo's
+# `main` — but the image has no .git (.dockerignore keeps it out, and it would be
+# stale the day after the build anyway). This mounts the HOST's git common dir,
+# read-only, at /data/source.git and points ARCHILYZER_SOURCE_REPO at it;
+# common/publish/source.ts reads that before it looks for a checkout. Nothing in
+# the container writes to it: the publish clones it into scratch first.
+#
+# ARCHILYZER_SOURCE_HOST_DIR is the host path, default `./.git` — right for a
+# plain clone. In a git WORKTREE `.git` is a file; use the primary checkout's
+# `.git` (`git rev-parse --git-common-dir` prints it).
+#
+# The entrypoint marks /data/source.git as a git safe.directory (the files
+# belong to your host user, the container runs as root). The scrub rules and
+# the denylist the publish needs live in the config volume
+# (ARCHILYZER_CONFIG_DIR, /data/config/archilyzer): `docker compose cp` them in.
+# RUNNING_IN_DOCKER.md, "Publish the archive".
+#
+# The long mount syntax with create_host_path: false, on purpose: the short form
+# makes a missing host path (a tarball install, a typo) into an empty root-owned
+# directory on the host. This way `up` fails and names the path instead.
+
+services:
+ editor:
+ environment:
+ ARCHILYZER_SOURCE_REPO: /data/source.git
+ volumes:
+ - type: bind
+ source: ${ARCHILYZER_SOURCE_HOST_DIR:-./.git}
+ target: /data/source.git
+ read_only: true
+ bind:
+ create_host_path: false
diff --git a/docker-compose.vulkan.yml b/docker-compose.vulkan.yml
@@ -38,7 +38,7 @@ services:
# The Vulkan stack needs a newer Debian than the default image runs on;
# the Dockerfile explains why, and why the workspace is still BUILT on
# the older one.
- RUNTIME_IMAGE: node:20-trixie-slim
+ RUNTIME_IMAGE: node:22-trixie-slim
image: archilyzer:${ARCHILYZER_TAG:-local}-vulkan
devices:
# The render node. This is the whole GPU passthrough — no toolkit, no
@@ -59,17 +59,17 @@ services:
build:
target: runtime-vulkan
args:
- RUNTIME_IMAGE: node:20-trixie-slim
+ RUNTIME_IMAGE: node:22-trixie-slim
image: archilyzer:${ARCHILYZER_TAG:-local}-vulkan
homepage:
build:
target: runtime-vulkan
args:
- RUNTIME_IMAGE: node:20-trixie-slim
+ RUNTIME_IMAGE: node:22-trixie-slim
image: archilyzer:${ARCHILYZER_TAG:-local}-vulkan
umtool:
build:
target: runtime-vulkan
args:
- RUNTIME_IMAGE: node:20-trixie-slim
+ RUNTIME_IMAGE: node:22-trixie-slim
image: archilyzer:${ARCHILYZER_TAG:-local}-vulkan
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -31,6 +31,13 @@ x-app: &app
build:
context: .
target: runtime
+ # Which commit the image is built from, for the publish stamps (the image
+ # has no .git). Empty unless the shell sets them:
+ # ARCHILYZER_COMMIT=$(git rev-parse HEAD) \
+ # ARCHILYZER_BRANCH=$(git branch --show-current) docker compose build
+ args:
+ ARCHILYZER_COMMIT: ${ARCHILYZER_COMMIT:-}
+ ARCHILYZER_BRANCH: ${ARCHILYZER_BRANCH:-}
image: archilyzer:${ARCHILYZER_TAG:-local}
restart: unless-stopped
networks: [archilyzer]
@@ -38,6 +45,11 @@ x-app: &app
# cosmetic: a bcrypt hash is full of `$`, and compose interpolates `${...}` in
# the YAML but passes env_file values through verbatim. Put the hash in .env
# and it arrives intact, with nothing to escape.
+ #
+ # The same goes for the publish credentials: CLOUDFLARE_API_TOKEN,
+ # CLOUDFLARE_ACCOUNT_ID and the R2_* keys are read from .env by the editor,
+ # and every publish stage it runs (a child process) inherits them. They are
+ # deliberately NOT in x-app-env below — a value there would override .env.
env_file:
- path: .env
required: false
@@ -59,6 +71,13 @@ x-app-env: &app-env
EXPORT_INDEX_DIR: /data/builds/.export-index
EXPORT_BUILDS_DIR: /data/builds/.export-builds
ARCHILYZER_SITE_OUT: /data/builds/site
+ # `publish homepage --deploy --to local` writes here; the homepage service
+ # serves it once it is non-empty (else the image's baked build).
+ ARCHILYZER_HOMEPAGE_OUT: /data/builds/homepage
+ # The operator's private config dir (common/lib/paths.ts): the source
+ # mirror's scrub rules and denylist live here, in the config volume —
+ # `docker compose cp` them in; they are never printed.
+ ARCHILYZER_CONFIG_DIR: /data/config/archilyzer
# Set explicitly so a stray EDITOR_PORT/UMTOOL_PORT in .env cannot move an app
# off the port Caddy proxies to. These are INTERNAL ports; the published ones
# are on the caddy service below.
@@ -99,6 +118,13 @@ services:
command: ["editor"]
environment:
<<: *app-env
+ # The publish lock's host identity (common/publish/stageLock.ts). Its
+ # default, the hostname, is the container id here and changes on every
+ # recreate, so a lock left by a crashed stage would look like another
+ # host's forever. Fixed, so this editor recognises its own stale lock.
+ # The EDITOR only, deliberately: another container with the same id but
+ # its own pid namespace would judge the editor's live lock dead.
+ ARCHILYZER_HOST_ID: archilyzer-editor
volumes:
- corpus:/data/transcripts
- config:/data/config
@@ -133,7 +159,10 @@ services:
- builds:/data/builds
# -------------------------------------------------------------------------
- # The project's own site (marketing + docs). Baked into the image.
+ # The project's own site (marketing + docs). A build is baked into the
+ # image; a local deploy (`publish homepage --deploy --to local`) into the
+ # builds volume replaces it once there is one — restart this service after
+ # the first.
# -------------------------------------------------------------------------
homepage:
<<: *app
@@ -141,6 +170,10 @@ services:
command: ["homepage"]
environment:
<<: *app-env
+ volumes:
+ # Writable for the same reason as the site's: the entrypoint makes the
+ # (empty) directory it looks in.
+ - builds:/data/builds
# -------------------------------------------------------------------------
# umtool: the clip/report bench. Private, like the editor.
@@ -167,12 +200,15 @@ volumes:
# The corpus. This is the one that matters: real media, real transcripts,
# hundreds of GB when it grows up. Back it up.
corpus:
- # settings.json, and anything else operational that must outlive the container.
+ # settings.json, the operator's private config dir (archilyzer/: the source
+ # mirror's rules), and anything else operational that must outlive the
+ # container.
config:
# whisper .bin models — 142 MB to 3 GB, fetched once.
models:
- # Export build staging + the published static site. Reproducible; losing it
- # costs a rebuild, not data.
+ # Export build staging, the shared index, every site's bundle and stamps, and
+ # the locally deployed site and homepage. Reproducible; losing it costs a
+ # rebuild, not data.
builds:
caddy-data:
caddy-config:
diff --git a/docker-compose.ytdlp.yml b/docker-compose.ytdlp.yml
@@ -0,0 +1,25 @@
+# Overlay: run YOUR yt-dlp — a source checkout on the host — instead of the
+# image's release binary. No rebuild: the swap happens at run time.
+#
+# # .env
+# YTDLP_SOURCE_HOST_DIR=/home/you/yt-dlp-patched # holds the yt_dlp/ package
+#
+# docker compose -f docker-compose.yml -f docker-compose.ytdlp.yml up -d
+#
+# The checkout is mounted read-only at /opt/yt-dlp-src and YTDLP_BIN points at
+# /usr/local/bin/yt-dlp-from-source, a wrapper baked into the image
+# (docker/yt-dlp-from-source.sh) that runs `python3 -m yt_dlp` from it with the
+# image's python and yt-dlp's optional modules.
+#
+# Every editor boot prints `yt-dlp: <path> <version> (override)`, and
+# YTDLP_AUTO_UPDATE leaves an override alone — update the checkout instead.
+# A zipapp built on the host (`make yt-dlp` in the checkout) works too, with no
+# overlay: copy it into the config volume and set YTDLP_BIN to its path there.
+# RUNNING_IN_DOCKER.md, "Substituting yt-dlp".
+
+services:
+ editor:
+ environment:
+ YTDLP_BIN: /usr/local/bin/yt-dlp-from-source
+ volumes:
+ - ${YTDLP_SOURCE_HOST_DIR:?set YTDLP_SOURCE_HOST_DIR in .env to your yt-dlp checkout}:/opt/yt-dlp-src:ro
diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
@@ -4,7 +4,8 @@
#
# editor the admin app (next start, :3001)
# site the published archive (serve, :3000)
-# homepage the project's own site (serve, :3031)
+# homepage the project's own site (serve, :3031) — the local deploy in the
+# builds volume when there is one, else the image's baked build
# umtool the clip/report bench (next start, :3050)
# shell drop into bash — for `docker compose run --rm editor shell`
#
@@ -22,6 +23,7 @@ SETTINGS_FILE="${SETTINGS_FILE:-/data/config/settings.json}"
MODELS_DIR="${ARCHILYZER_MODELS_DIR:-/data/models}"
BUILDS_DIR="${ARCHILYZER_BUILDS_DIR:-/data/builds}"
SITE_OUT="${ARCHILYZER_SITE_OUT:-${BUILDS_DIR}/site}"
+HOMEPAGE_OUT="${ARCHILYZER_HOMEPAGE_OUT:-${BUILDS_DIR}/homepage}"
WHISPER_MODEL="${WHISPER_MODEL:-${MODELS_DIR}/ggml-base.en.bin}"
# Which transcription backend THIS IMAGE was built with. Set by the Dockerfile
# per target, not by the operator: `runtime`/`runtime-cuda` carry whisper.cpp,
@@ -42,7 +44,26 @@ mkdir -p \
"$(dirname "${SETTINGS_FILE}")" \
"${MODELS_DIR}" \
"${BUILDS_DIR}" \
- "${SITE_OUT}"
+ "${SITE_OUT}" \
+ "${HOMEPAGE_OUT}"
+# The operator's private config dir (compose sets it inside the config volume),
+# made so `docker compose cp` has somewhere to put the source mirror's rules.
+# Only made, never filled: what goes in it is the operator's.
+if [ -n "${ARCHILYZER_CONFIG_DIR:-}" ] && [ ! -d "${ARCHILYZER_CONFIG_DIR}" ]; then
+ mkdir -p "${ARCHILYZER_CONFIG_DIR}"
+ chmod 700 "${ARCHILYZER_CONFIG_DIR}" || true
+fi
+
+# The host's git common dir, mounted read-only by docker-compose.source.yml for
+# the homepage's source mirror (`archilyzer source publish`). It belongs to the
+# host's user and the container runs as root, so git would refuse it as
+# "dubious ownership" without this. Added once — `--add` on every boot would
+# pile up duplicates in a container that is restarted rather than recreated.
+SOURCE_REPO_DIR="${ARCHILYZER_SOURCE_REPO:-/data/source.git}"
+if ! git config --global --get-all safe.directory 2>/dev/null | grep -qxF "${SOURCE_REPO_DIR}"; then
+ git config --global --add safe.directory "${SOURCE_REPO_DIR}" ||
+ printf '[entrypoint] WARNING: could not mark %s safe for git\n' "${SOURCE_REPO_DIR}"
+fi
# ---------------------------------------------------------------------------
# 2. Seed settings.json — with a worker.
@@ -163,20 +184,67 @@ fetch_model() {
}
# ---------------------------------------------------------------------------
-# 4. yt-dlp self-update.
+# 4. yt-dlp: which one, and its self-update.
+#
+# YTDLP_BIN is the yt-dlp every fetch runs; ARCHILYZER_IMAGE_YTDLP is the one
+# this image ships. They differ when the operator substituted their own (a
+# zipapp, or /usr/local/bin/yt-dlp-from-source over a mounted source tree —
+# RUNNING_IN_DOCKER.md, "Substituting yt-dlp"): an OVERRIDE.
#
# An image pins yt-dlp at build time, and a stale yt-dlp is the most common
# reason downloads suddenly start failing — YouTube changes, yt-dlp ships a fix
-# within days, and an image from last month has none of them. Opt-in because it
-# is a network call on every boot.
+# within days, and an image from last month has none of them. The self-update is
+# opt-in because it is a network call on every boot, and it never touches an
+# override: a patched build is updated where it is built, and `-U` on one either
+# fails or replaces the patch with a release.
# ---------------------------------------------------------------------------
+YTDLP="${YTDLP_BIN:-yt-dlp}"
+IMAGE_YTDLP="${ARCHILYZER_IMAGE_YTDLP:-}"
+
+# "image" or "override", by where each one actually lands.
+ytdlp_origin() {
+ local mine theirs
+ if [ -z "${IMAGE_YTDLP}" ]; then
+ echo image
+ return 0
+ fi
+ mine="$(readlink -f "$(command -v "${YTDLP}" 2>/dev/null || echo "${YTDLP}")" 2>/dev/null || echo "${YTDLP}")"
+ theirs="$(readlink -f "${IMAGE_YTDLP}" 2>/dev/null || echo "${IMAGE_YTDLP}")"
+ if [ "${mine}" = "${theirs}" ]; then echo image; else echo override; fi
+}
+
update_ytdlp() {
case "${YTDLP_AUTO_UPDATE:-0}" in
1 | true | yes | on) ;;
*) return 0 ;;
esac
+ if [ "$(ytdlp_origin)" = "override" ]; then
+ log "WARNING: YTDLP_AUTO_UPDATE is set, but YTDLP_BIN (${YTDLP}) is not the image's yt-dlp"
+ log " (${IMAGE_YTDLP}) — an override is not self-updated; update it where it is built"
+ return 0
+ fi
log "yt-dlp -U (YTDLP_AUTO_UPDATE is set)"
- "${YTDLP_BIN:-yt-dlp}" -U || log "WARNING: yt-dlp self-update failed — continuing with $("${YTDLP_BIN:-yt-dlp}" --version 2>/dev/null || echo unknown)"
+ "${YTDLP}" -U || log "WARNING: yt-dlp self-update failed — continuing with $("${YTDLP}" --version 2>/dev/null || echo unknown)"
+}
+
+# One line on every boot, like the vulkan: line — "which yt-dlp is this, and
+# does it run?" answered before the first download has to. A yt-dlp that is
+# missing or does not run is a WARNING, never a refusal: the editor is still
+# useful (reading, building, publishing) without one.
+ytdlp_line() {
+ local origin version err
+ origin="$(ytdlp_origin)"
+ if ! command -v "${YTDLP}" >/dev/null 2>&1; then
+ log "yt-dlp: MISSING ${YTDLP} (${origin}) — every download will fail until it exists"
+ return 0
+ fi
+ if version="$(timeout 30 "${YTDLP}" --version 2>/dev/null | head -n 1)" && [ -n "${version}" ]; then
+ log "yt-dlp: ${YTDLP} ${version} (${origin})"
+ else
+ err="$(timeout 30 "${YTDLP}" --version 2>&1 >/dev/null | head -n 1 || true)"
+ log "yt-dlp: MISSING — ${YTDLP} (${origin}) does not run: ${err:-no version printed}"
+ log " every download will fail until it does; see RUNNING_IN_DOCKER.md"
+ fi
}
# ---------------------------------------------------------------------------
@@ -203,6 +271,7 @@ case "${APP}" in
editor)
fetch_model
update_ytdlp
+ ytdlp_line
log "corpus: ${TRANSCRIPTS_DIR}"
log "settings: ${SETTINGS_FILE}"
if [ "${TRANSCRIBER}" = "parakeet" ]; then
@@ -271,9 +340,21 @@ HTML
;;
homepage)
- # Corpus-independent, so this one IS baked into the image at build time.
- log "serving /repo/homepage/out on :3031"
- exec "$(serve_bin /repo/homepage)" /repo/homepage/out -l 3031 \
+ # Corpus-independent, so a build IS baked into the image — but that build
+ # has no /source mirror and none of the corpus's numbers (there is no corpus
+ # and no .git at image-build time). `archilyzer publish homepage --deploy
+ # --to local` writes a real one into the builds volume; once that directory
+ # is non-empty it is what is served. Chosen at boot: restart this service
+ # after the FIRST local deploy (later ones are served as they land — serve
+ # reads from disk per request).
+ HOMEPAGE_DIR=/repo/homepage/out
+ if [ -n "$(ls -A "${HOMEPAGE_OUT}" 2>/dev/null)" ]; then
+ HOMEPAGE_DIR="${HOMEPAGE_OUT}"
+ log "serving the locally deployed homepage ${HOMEPAGE_DIR} on :3031"
+ else
+ log "nothing deployed at ${HOMEPAGE_OUT} — serving the image's baked homepage ${HOMEPAGE_DIR} on :3031"
+ fi
+ exec "$(serve_bin /repo/homepage)" "${HOMEPAGE_DIR}" -l 3031 \
-c /repo/homepage/serve.json --no-clipboard --no-port-switching
;;
diff --git a/docker/publish-site.sh b/docker/publish-site.sh
@@ -4,15 +4,19 @@
#
# docker compose exec editor /repo/docker/publish-site.sh <site-id>
#
-# Why this exists rather than a baked build: the export site is a static render
-# OF A CORPUS, and there is no corpus at image-build time — so the image ships
-# the code and this publishes the output. It is the same `archilyzer build site`
-# a host install runs; the only container-specific part is the copy at the end.
+# A wrapper over the three publish stages, nothing more (release 18):
#
-# The copy is a copy and not a symlink on purpose. `next build` REMOVES and
-# recreates export/out (docker/build-site.sh has the same note), so a symlink
-# there survives exactly one build, and a bind mount there breaks the build
-# outright — the rm fails on a busy mount point.
+# archilyzer publish index the shared index, stats, templates
+# archilyzer publish build <id> the site's bundle in the builds volume
+# archilyzer publish deploy <id> --to local copied into ARCHILYZER_SITE_OUT
+#
+# Each stage is a no-op when it is already fresh, so running this twice costs
+# seconds; each takes the publish lock, so it waits for (never races) a stage
+# the editor is running. Run them one at a time for more control — the same
+# commands, through `docker compose exec editor pnpm archilyzer publish …`.
+#
+# Why the export site is built at run time rather than baked: it is a static
+# render OF A CORPUS, and there is no corpus at image-build time.
set -euo pipefail
SITE_ID="${1:-${SITE_ID:-}}"
@@ -28,39 +32,23 @@ if [ -z "${SITE_ID}" ]; then
exit 2
fi
-export SITE_ID
-
# A PRIVATE site (site.json `audience: "private"`) is never deployed, and the
-# volume this script fills is what the `site` service serves: refused before
-# the build, and again over the built corpus.json below (lib/builtExport.ts).
-# Building it without publishing is `archilyzer build site <id>`.
+# volume this fills is what the `site` service serves. The deploy stage refuses
+# it too (over the built corpus.json); refusing here first saves the build.
SITE_JSON="${SITES_DIR:-${TRANSCRIPTS_DIR:-/data/transcripts}/sites}/${SITE_ID}/site.json"
-refuse_private() {
- echo "[publish-site] REFUSED — site '${SITE_ID}' is private (audience: private): it is built for reading on this machine and is never deployed. Nothing was published to ${SITE_OUT}. Build it without publishing: pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts build site ${SITE_ID}" >&2
- exit 1
-}
if grep -Eq '"audience"[[:space:]]*:[[:space:]]*"private"' "${SITE_JSON}" 2>/dev/null; then
- refuse_private
+ echo "[publish-site] REFUSED — site '${SITE_ID}' is private (audience: private): it is built for reading on this machine and is never deployed. Nothing was published to ${SITE_OUT}. Build it without publishing: pnpm archilyzer publish build ${SITE_ID}" >&2
+ exit 1
fi
cd /repo
-echo "[publish-site] building '${SITE_ID}'"
-# The full build: the data phase (shared LMDB index + stats + chart
-# templates), then compose:site for THIS site, then next build.
-pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts build site "${SITE_ID}"
-
-[ -d /repo/export/out ] || { echo "[publish-site] no export/out after build" >&2; exit 1; }
-if grep -Eq '"audience"[[:space:]]*:[[:space:]]*"private"' /repo/export/out/corpus.json 2>/dev/null; then
- refuse_private
-fi
+archilyzer() {
+ pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts "$@"
+}
-echo "[publish-site] publishing -> ${SITE_OUT}"
-mkdir -p "${SITE_OUT}"
-# Empty the directory's CONTENTS, never the directory: it is a volume mount
-# point, and removing it fails.
-find "${SITE_OUT}" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
-cp -a /repo/export/out/. "${SITE_OUT}/"
+archilyzer publish index
+archilyzer publish build "${SITE_ID}"
+archilyzer publish deploy "${SITE_ID}" --to local
-echo "[publish-site] done — $(find "${SITE_OUT}" -type f | wc -l) files"
-echo "[publish-site] the site service serves it immediately; no restart needed."
+echo "[publish-site] the site service serves ${SITE_OUT} immediately; no restart needed."
diff --git a/docker/yt-dlp-from-source.sh b/docker/yt-dlp-from-source.sh
@@ -0,0 +1,20 @@
+#!/bin/sh
+# yt-dlp run from a SOURCE tree — the image's hook for substituting your own
+# yt-dlp (a patched checkout) without rebuilding the image.
+#
+# YTDLP_BIN=/usr/local/bin/yt-dlp-from-source (.env)
+# docker compose -f docker-compose.yml -f docker-compose.ytdlp.yml up -d
+#
+# docker-compose.ytdlp.yml mounts YTDLP_SOURCE_HOST_DIR (the checkout: the
+# directory that holds the `yt_dlp/` package) read-only at /opt/yt-dlp-src.
+# YTDLP_SOURCE_DIR moves that mount point. The python is the image's, with
+# yt-dlp's optional modules installed beside it (the Dockerfile's apt line).
+#
+# Linked into /usr/local/bin by the Dockerfile; RUNNING_IN_DOCKER.md,
+# "Substituting yt-dlp".
+src="${YTDLP_SOURCE_DIR:-/opt/yt-dlp-src}"
+if [ ! -f "${src}/yt_dlp/__init__.py" ]; then
+ echo "yt-dlp-from-source: no yt_dlp package in ${src} — mount a yt-dlp checkout there (docker-compose.ytdlp.yml, YTDLP_SOURCE_HOST_DIR)" >&2
+ exit 127
+fi
+PYTHONPATH="${src}${PYTHONPATH:+:${PYTHONPATH}}" exec python3 -m yt_dlp "$@"
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -3,6 +3,13 @@
## [Unreleased]
- **Deploys are pinned and checked live.** wrangler is an exact dependency of the workspace (4.147.0), so a deploy runs the version installed with the code instead of whatever `pnpm dlx` fetched that day, and every deploy names its branch: production is `--branch main`, never taken from the checkout it ran in (where a "production" deploy from a feature branch used to land as a preview). The publish stages' deploy (release 18) refuses before wrangler runs when there is no Cloudflare credential at all — "set CLOUDFLARE_API_TOKEN in .env" — and says "REFUSED by Cloudflare — the API token was not accepted" when Cloudflare rejects one; it refuses a production deploy of a build made from a branch other than `main`. After each deploy it reads `corpus.json` at the site's address twice, as a visitor would and cache-busted, and records the verdict: ok, stale-edge (the deployment is right, Cloudflare's edge still serves an older copy), mismatch, or unreachable. A verdict short of ok is a warning in the log; the deploy itself succeeded. What each target last shipped, where, and how it read is kept in `deployed.json` beside its build.
- **Withdrawn X posts ship tombstones.** While X posts are private, a public site's build no longer just leaves an X channel's posts out: at every path they were served from it ships an empty stand-in — the channel's posts manifest with no pages, and an empty page for each page the channel has — served uncached. The hub, which carries no posts, ships the same for every X channel a public site carries, with an empty posts manifest; a channel only private sites carry is never named on the hub. Leaving a path out of a deploy does not take it off Cloudflare's edge, which kept serving a withdrawn copy for up to a week; a changed object at the same path replaces it. The hub's deploy reads each of those paths back.
+- **Publishing is stages, from the command line: `archilyzer publish`.** `publish index` updates the index — the LMDB index, the stats datasets and the chart templates, in one child process with an 8 GB heap — and writes an index stamp (`export/.export-index/stamp.json`) naming, for each site, a signature of everything that site's build reads. `publish build <id|all>` builds a site from that index (no data phase of its own) into its own bundle, `export/.export-builds/<id>/out`, and stamps it (`built.json`); a site whose bundle already matches the index is a no-op unless `--force`. `publish deploy <id|all> [--preview <branch>] [--to local]` ships that bundle — to Cloudflare Pages, or with `--to local` into the directory the docker `site` service serves — and records the deploy (`deployed.json`); deploying the same build again is a no-op unless `--force`. `all` passes over private sites and, to Pages, sites with no Pages project; any other site it cannot deploy is a failure, said after the rest are tried. `publish hub [--deploy]` and `publish homepage [--deploy]` do the same for the hub (`_hub/out`) and the homepage. A stage whose input is not there says so and exits 3: "update the index first", "no build of jeralyzer — archilyzer publish build jeralyzer". Production refuses a bundle built on a branch other than `main`, or with no branch recorded (a detached checkout; an image sets `ARCHILYZER_BRANCH`) — a preview of it is fine. Exit codes: 0 done or nothing to do, 1 failed, 2 usage, 3 precondition not met, 130 cancelled.
+- **One publish at a time on a machine.** Every stage takes `export/.export-builds/.publish.lock`; a second one — an `archilyzer publish` beside the editor, say — waits for it, saying once whom it waits for, and Ctrl-C ends the wait. A lock left by a process that is gone is taken over. A cancelled stage takes the whole process tree it started with it (`next build`'s workers, wrangler, docker).
+- **`export/out` is now a link to the bundle built last.** Each site, and the hub, keeps its own bundle, so building one site no longer replaces another's; `export/out` points at whichever was built most recently, so `serve out` and anything else that read it keeps working.
+- **`build site`, `build all` and `deploy site` are aliases of the publish commands** and print what they run: `build site <id>` is `publish index` (skipped with `--nodata`) then `publish build <id> --force`; `build all` is `publish index` then `publish build all --runner auto` (containers when an engine answers, else one site at a time on the host); `deploy site <id>` is `publish deploy <id>`, which now ships the site's own bundle and refuses a site never built that way. `publish build all --runner docker` builds every stale site in containers on a Linux host and refuses with "the docker runner needs an engine on this host" where there is none.
+- **Substitute your own yt-dlp in Docker.** Point `YTDLP_BIN` at a zipapp you built, or set `YTDLP_SOURCE_HOST_DIR` to a yt-dlp checkout and start with `docker-compose.ytdlp.yml`: the image runs it with its own python, and nothing is rebuilt. Every editor boot logs `yt-dlp: <path> <version> (image|override)` (`MISSING` when it does not run; the editor still starts), and `YTDLP_AUTO_UPDATE` updates the image's yt-dlp only, warning instead of touching yours.
+- **The Docker image can publish.** It carries python, `pipx` and a pinned `git-filter-repo`, so the homepage's `/source` mirror builds in the container; `docker-compose.source.yml` mounts your repository read-only for it, and the scrub rules and denylist live in the config volume (`/data/config/archilyzer`). Cloudflare and R2 credentials come from `.env`. Run publish commands with `docker compose exec editor pnpm archilyzer …`, not `run --rm`. The `homepage` service serves a local deploy from the builds volume once there is one. RUNNING_IN_DOCKER.md has a Windows checklist.
+- **`archilyzer doctor` checks what a publish needs.** Which yt-dlp runs (the image's, the host's or an override, and whether it runs), whether the Cloudflare token and the R2 keys are set (never their values; R2 only when a bucket is configured), free space for the site bundles, the repository the source mirror reads, and the private config dir.
- **A cited moment at the very end of a recording prepares.** Prepare evidence media cuts a clip whose padding runs past the recording's end at the end (the recording's duration from its metadata), where it found no media for the padded span; a span that starts past the end is still refused. report-to-video keeps its strict rule.
- **Exporting a changed report records a new revision of it.** `reports export` (and **Export reports** on a site's Reports tab, and the end of a prepare) commits a revision to the report's own git history, `sites/<site>/reports/<id>/history-git/`, whenever its `report.json` changed since the last one: the `report.json`, its Markdown export and the checksums of every export file, with a message of `Revision N` and a summary of the change. A re-export of an unchanged report records nothing. The commits carry the site's name and a `noreply@<site>.invalid` address with dates in UTC, never your git name, email or time zone. The Reports tab shows each report's revision, its commit and the last change under **Exports**, and the site's next build publishes the history. Add `history-git/` to the corpus repository's `.gitignore`.
- **archive.org files come over BitTorrent when possible, else straight from archive.org — never through yt-dlp.** The chosen file of an archive.org import is fetched from the item's own torrent (`<identifier>_archive.torrent`, which lists archive.org as a web seed, so other peers take load off archive.org) with aria2c, only that file of the item, and seeded afterwards for 10 minutes or to a ratio of 1, whichever comes first; the log shows "torrent: <file> (n of m pieces, peers p, web seed yes)" and "seeding 10 min…". With no aria2c, a torrent that does not carry the file, or no progress for 5 minutes, it is downloaded directly from `archive.org/download/…` instead (resumable, backing off on 429/503), and the log says "fell back to direct download: <reason>". Every file is checked against archive.org's sha1/md5: a mismatch is downloaded once more directly, a second one fails the record. The record is written from the item's metadata: `metadata.info.json` with the file's page, the canonical id, the duration ffprobe measures and archive.org's playable copies of the file, the `archiveorg.json` provenance (a mirror's original title, date and uploader), and `audio.<fmt>` — an audio file already in the channel's format is used as is, anything else goes through the app's audio extraction, a video kept in the saved-video store when the channel keeps sources. An .avi/.mpeg/.flac/.wav original is fetched as archive.org's mp4 or mp3 of it. aria2c runs in its own process group: cancelling the job stops it and everything it started, and it stops itself if the editor exits. New settings block `archiveOrg` (`torrent`, `seedMinutes`, `seedRatio`, `stallMinutes`, `maxPeers`, `maxDownloadKiBps`, `maxUploadKiBps`), `ARIA2C_BIN`, an aria2c row in `archilyzer doctor`, and `aria2` in the runtime Docker images.
diff --git a/plans/release-18.md b/plans/release-18.md
@@ -609,6 +609,373 @@ Gates after the fixes: tsc (all workspaces) clean; **common 3,199/3,199**, 135 s
| Run | At | Specs | Result |
|---|---|---|---|
| 4 (editor) | `801124c3` | the five specs of run 1 | **29 passed**, 0 failed, 2.4 min (no queue wait) |
+### Slice S1, as shipped — the stage contract, the stamps, the lock, per-target bundles and the CLI (2026-10-06)
+
+Branch `r18/stage-core` off `ce66f2d3` (the plan commit on `r18/integration`), worktree `~/Projects/r18-stage-core`
+(editor 7201, test 7211, export 7210 — `pnpm wt list`'s #42), one Opus implementer, beside S2 and S5's image half.
+Scratch files `s1-*` in the job's `tmp`. The plan is "Model", "Stages" and "CLI" above; the deploy bodies are S2's to
+rewire, the status view, the lane and `publish status|now` S3's.
+
+**What it does.**
+- **`common/publish/stages.ts`** — the only module that knows every stage: `StageKind`, `StageRequest`, `Freshness`,
+ `Stage`, `StageOutcome` as "Model" lists them (+ an optional `allowMissingMedia` on the request, for `build site
+ --allow-missing-media`); `STAGES` (seven, `jobKind: publish-<kind>`, `queueKey: "publish"`); a PURE `needs()` per
+ stage over **`NeedsInput`** — the minimal PublishStatus-shaped input S3's view satisfies: `index: {stamp,
+ lastIngestDoneAt, configChangedAt}`, per site / `hub` / `homepage` a `TargetState` `{built, deployed,
+ changedChannels, configChangedAt, bundleProblem, deployProblem?, pagesProblem?}`, and the homepage's `mainHead`.
+ `stageArgv` / `parseStageArgs` (inverse, round-trip tested) and `STAGE_FLAGS`.
+- **`needs()`, row by row.** update-index: stale with no stamp, an ingest ended `done` after `stamp.scannedAt`, or a
+ config newer than it. build-site: BLOCKED "update the index first" with no stamp (forced too), "waiting for the
+ index update this run started" under `indexAfter`, "the index has not seen site X" when the stamp has no entry;
+ then stale by `changedChannels` ("N channels changed (a, b, …)"), a config change after `builtCheckedAt(built)`, an
+ `inputSig` mismatch ("data changed"), a bundle problem, never built; `--force` stale; a `built.commit` that differs
+ is NOT stale. `_all`: per site. deploy-*: blocked with no build ("no build of X — archilyzer publish build X"), a
+ private target (every kind), no Pages project (pages kinds only), a bundle problem, `builtAfter` (unless the
+ bundle matches the index this run updated — see the review fixes), and a PRODUCTION deploy of a bundle whose
+ `branch` is not `main` — a null branch (a detached HEAD, an image built without `ARCHILYZER_BRANCH`) is refused the
+ same way; fresh exactly when `deployed[kind/branch].builtStampId === built.stampId`. build-hub: `built.inputSig ===
+ stamp.hubSig` (+ `changedChannels`). build-homepage: `indexStampId` current and `sourceCommit === mainHead` when a
+ repository answers.
+- **`common/publish/stamps.ts`** — `IndexStamp`, `BuiltStamp`, `DeployRecord`, `DeployedFile`, `LiveCheck`, `Probe`
+ exactly as listed; paths (`<exportIndexDir>/stamp.json`, `<exportBuildsDir>/<target>/{built,deployed}.json`); atomic
+ writes (`writeJsonAtomic`); tolerant reads (missing, unparseable or wrongly shaped = null); `recordDeploy` keeps
+ every other record; `newStampId` sorts by time; `imageBuildFacts` (the image's `ARCHILYZER_COMMIT` /
+ `ARCHILYZER_BRANCH`, empty = null — S5's names, read by name here until S5's helper of the same name replaces it).
+ `BuiltStamp.checkedAt?` (review fix): when a later no-op build last found the bundle still matching its inputs.
+- **Commit and branch in a stamp (the S4 seam):** `ARCHILYZER_COMMIT` / `ARCHILYZER_BRANCH`, when set, WIN over git,
+ each on its own — the runtime image bakes them (no `.git`), and S4's e2e webServer sets `ARCHILYZER_BRANCH=main`,
+ because a worktree's branch is never `main` and production refuses any other. Else git's HEAD and branch; a
+ detached HEAD records `branch: null`.
+- **`common/publish/stageLock.ts`** — `<exportBuildsDir>/.publish.lock` `{pid, host, kind, target, since, pidStart}`,
+ `open(…, "wx")`; the host is `ARCHILYZER_HOST_ID` when set (S5 fixes one in compose), else `os.hostname()`; stale
+ when same host and the pid is dead (`processIsAlive`), answers with another `/proc/<pid>/stat` start time, or names
+ a process that STARTED AFTER the lock's `since` (starttime/100 + `/proc/stat` btime, 2 s slack; a recreated
+ container's pid 1) — with no `/proc`, pid-alive alone; another host's is never stolen, and its wait line names
+ both hosts and how to clear it; a torn file is taken over after 60 s; a live holder is waited for (5 s poll, ONE log
+ line, the signal cancels the wait); release removes only its own.
+- **`common/publish/stageRun.ts`** — `runStage(req)` (in-process, under the lock, never throws: `{code, outcome,
+ message}`), exit codes 0 / 1 / 2 / 3 / 130, `stageMain` (the child: SIGTERM/SIGINT abort the stage, a second one
+ SIGKILLs the child process groups and exits, tree-kill on), and **`stageCommand(paths, req)`** — `<common>/node_modules/.bin/tsx bin/archilyzer.ts stage
+ <kind> <target> [flags]`, cwd `common/`, the caller's env + `NODE_OPTIONS=… --max-old-space-size=8192` for
+ update-index only — what S3's `enqueueStage` hands `runManagedCommand`.
+- **`common/publish/stageBodies.ts`** — every body but update-index first asks its `needs()` over the state ON DISK
+ (`readNeedsInput`: the stamps, the bundles, `siteDeployProblem`, the Pages project, `main`'s head; no job metas):
+ blocked → exit 3, fresh and not forced → a no-op (a build's no-op writes `checkedAt`). **update-index**: `buildIndex` → `buildStats` → the chart
+ templates in ONE process, then the stamp — `generation`, `scannedAt` and each site's `siteFp`/`statsFp` (sha1 of
+ the LMDB keys, read-only), each site's `inputSig` and the `hubSig` (`common/publish/inputSig.ts`). An index that
+ rebuilt nothing and whose every signature is unchanged KEEPS its stamp id (status `noop`), so the builds made from
+ it stay current. **build-site**: `buildSiteBundle`, then `built.json`; `_all` local = each stale site in turn
+ (failures collected); `_all --runner docker` = no engine → exit 3 "the docker runner needs an engine on this host",
+ else `build archives` on the host → `ensureBuildImage` → `runDockerBuildOne` per stale site at
+ `maxParallelBuilds` → `builtBundleProblem` → `built.json` (`runner: "docker"`). **build-hub** / **build-homepage**:
+ `buildHubBundle` / `buildHomepage` (source mirror included; `sourceCommit` from `homepage/out/source/manifest.json`).
+ **deploy-site / hub / homepage**: today's `deploySite` / `deployHub` / `deployHomepage` over the TARGET'S bundle
+ (`outDir` / `stagingDir` options added), then `deployed.json` (`url` = the deployment URL the log line names,
+ `alias` = the preview alias, `liveCheck: null` — S2's); `--to local` copies the bundle's contents into
+ `ARCHILYZER_SITE_OUT` (a site) or `ARCHILYZER_HOMEPAGE_OUT` (the homepage — S5's name), private refused.
+- **`inputSig`** (`common/publish/inputSig.ts`) signs, with compose's `dirSignature`: the site's whole
+ `.export-index/sites/<id>/` tree (chart-templates.json by its BYTES — `build templates` rewrites it every run), each
+ PUBLISHED member's shared transcripts/subs/posts/digests tree (`manifest.json` ignored, a manifest-only tree as
+ compose's constant), `site.json`'s bytes, the `sites/<id>/` dir, the global aliases, curated tags and duplicates
+ files (size + mtime), `archiveStorage` + `social.x.visibility` + `buildArchives`, and — what the export BUILD
+ renders beyond compose (review fix) — the resolved social links, the resolved hub url and the footer's sibling
+ sites (`resolveRelatedSites(site, listSites())`: each sibling's url, title and listing). The rule: a site is fresh
+ exactly when compose AND the export build would produce the same bundle. A superset of their inputs:
+ conservative. `hubSig` = sha1(stampId, homepage.json, each listed site's id + siteUrl + title).
+- **`common/lib/dirSignature.ts`** — compose's `dirSignature`, moved unchanged; `compose-site.ts` imports it (that
+ line, and its now-unused `createHash` import removed, are the only compose-site edits).
+- **`common/publish/build.ts`** — per-target bundles: `bundleDir(paths, target)` (= `dockerSiteOutDir` for a site),
+ `installBundle(src, <target>/out)` (rename into `out.next`, `out → out.prev`, `out.next → out`, `out.prev` removed, a
+ leftover `out.next` deleted first; EXDEV → `fs.cp` + remove, injectable `BundleFs`), `recoverInterruptedInstall` (an
+ `out.prev` with no `out` is renamed back before a build or an install touches the target; the old `built.json` is
+ removed before the swap and the new one written last), `unlinkExportOut` (before a
+ build: a link at export/out is removed so a failed build cannot leave an older bundle there), `pointExportOutAt`
+ (export/out → a RELATIVE symlink, replaced atomically), `stageSiteArchives` (the host compose's `.r2-staging/<id>`
+ moved to `dockerSiteStagingDir`, a no-op where they are one place), `bundleCounts`, `corpusGeneratedAtIn`,
+ `buildSiteBundle`, `buildHubBundle`. `ensureBuildImage`, `runDockerBuildOne`, `runHostScript`,
+ `runWithConcurrency` are exported. A build container gets `EXPORT_BUILDS_DIR=/tmp/archilyzer-builds`
+ (`CONTAINER_BUILDS_DIR`) so its own `publish build` lock never lands on the host mount. Every existing export is
+ unchanged; the editor's actions still build into `export/out` (S4 rewires them).
+- **`common/bin/archilyzer.ts` + `common/bin/publish.ts`** — rows `publish index` (the SAME child the editor spawns,
+ for its heap), `publish build <id|all> [--runner local|docker|auto] [--force] [--skip-archives]`, `publish deploy
+ <id|all> [--preview b] [--to local] [--force]` (`all` passes over, one line each, only a private site and — to
+ Pages — a site with no project; every other refusal is a failure, the rest are still tried and the run exits 1), `publish hub [--deploy] [--preview b] [--force]`, `publish homepage [--deploy]
+ [--preview b] [--to local] [--force]`, and the internal `stage <kind> <target> --run-id …`. A comment marks where
+ S3's `publish status` / `publish now` rows go. **Aliases, printed first**: `build site <id>` = `publish index` (not
+ with `--nodata`) + `publish build <id> --force`; `build all` = `publish index` + `publish build all --runner auto`;
+ `deploy site <id>` = `publish deploy <id>`.
+
+**Deviations from the plan** (one sentence each):
+1. `publish index` runs the stage child (`stageCommand`, the 8 GB heap) rather than the body in the CLI's process: the
+ index and stats builds of the real corpus have always run with that cap (export's `build:index`), and the CLI's own
+ node has the default heap.
+2. `build all`'s alias runs `publish index` first — the old row always ran the data phase, and building every site
+ from a stale index would not be what it said.
+3. Tree-kill lives in `common/jobs/runChild.ts` (`setKillChildTrees`, off by default; a stage child and the publish
+ CLI turn it on): each child leads its own process group and a cancel signals the group, then SIGKILLs what is left
+ once the leader exits. The editor's in-process jobs are unchanged.
+4. `.gitignore` gains `/export/out` (no trailing slash): `**/out/` matches only a directory, and the link showed as
+ untracked — which `release cut --commit` refuses.
+5. Inside the docker per-site build container (`ARCHIVES_READONLY=1`, set by `docker/build-site.sh`) `publish build`
+ builds IN PLACE and stamps nothing — the container hands export/out back and the host stamps it — and asks no
+ stamp, so the editor's existing Build all (host `build:data`, no stamp) keeps working until S4 rewires it. The
+ container still writes `<id>/out` with build-site.sh's `rm` + `cp`, not through `out.next` (S5's file).
+6. `StageRequest.allowMissingMedia` (optional) carries `build site --allow-missing-media` through the stage.
+7. The bundle-layout tests are a new `common/publish/bundle.test.ts`, not `build.test.ts`, which S2 also edits.
+8. `ARCHILYZER_COMMIT` / `ARCHILYZER_BRANCH` / `ARCHILYZER_HOMEPAGE_OUT` are read by name through an `env[name]`
+ helper: they are declared in `lib/envVars.ts` on S5's branch, not on this one (`envVars.test` stays green here;
+ after the merge the helper can be S5's `imageBuildFacts`).
+
+**Open question 3, settled: yes — a `generation` bump rewrites EVERY site's aggregates.** `generation` is bumped
+whenever any record anywhere is added, changed or removed (`buildIndex.ts` ~:2036), and every site's fingerprint
+carries `gen` (~:2083), so every site is rebuilt: its summary pages (`writeJsonAtomic`, no sha1 skip for site pages),
+its four manifests (fresh `generatedAt`) and its `tag-counts.json`. Their mtimes move, compose would re-copy the
+summaries, and every site's `inputSig` changes. So `inputSig` is conservative as the plan expected: any data change
+anywhere makes every site stale (more rebuilds, never a wrong skip); `changedChannels` is the precise per-site signal.
+A follow-up could sign the site's summaries by content less `generatedAt`.
+
+**Found, not fixed (not this slice's files).**
+- **Every hub bundle is refused since `5c09cd7b` (2026-10-05).** `builtHubProblem` (`common/lib/builtExport.ts`)
+ refuses a hub `out/` that holds `reports/` or `m/` ("still carries a site's data (reports, m)"), but the export app's
+ own `/reports/` and `/m/[...moment]` routes render `out/reports/index.html` (+ `__next.*.txt`) and `out/m/…` in
+ EVERY build, the hub's included — `export/public` had neither when measured. So `deployHub` (old path and new) and
+ `publish hub` refuse every hub; rollout step 5 is blocked until the check looks for report DATA (e.g.
+ `reports/index.json`, a `reports/<id>/page.json`) or the hub stops rendering those routes. Measured in the S1 smoke
+ (a scratch corpus; `publish hub` exit 1 after a 119 s build).
+- `pnpm --filter … exec` (and so `pnpm archilyzer`) reports a stage's exit 2 / 3 / 130 as 1; the editor spawns tsx
+ directly and sees the real code.
+- The worktree export build gate needs a FULL site's compose in `export/public`; the primary's held a cited site's
+ (no `summaries/`) at the time, and `/` failed to prerender against it. The gate was run over a scratch site
+ composed into the worktree's own `public/` (then cleaned and re-linked).
+
+**Left for the other slices.** S2: rewire the three deploy bodies (`stageBodies.ts` `deployStage`) to its deploy
+stage, import `LiveCheck`/`Probe` from `stamps.ts`, fill `DeployRecord.liveCheck`/`wrangler`. S3: build
+`PublishStatus` to satisfy `NeedsInput` (job metas → `lastIngestDoneAt` / `changedChannels`, config mtimes), spawn
+`stageCommand`, the `publish status|now` rows, the `publish-*` job kinds. S4: the editor's actions still call
+`buildSite` / `deploySite` on export/out. S5: `docker/publish-site.sh` and `build-site.sh` (the swap), and the
+envVars names above.
+
+| commit | what |
+|---|---|
+| `ffa95b73` | `dirSignature` moves to `lib/dirSignature.ts`, unchanged; compose imports it |
+| `0122cd1b` | the stamp files and the publish lock (+ tests) |
+| `75403df3` | per-target bundles: install, link, archive staging; `buildSiteBundle`/`buildHubBundle`; deploy `outDir`; tree-kill |
+| `0a7d88ac` | the seven stages, the runner, `inputSig`, the CLI rows and the aliases |
+| `e040bb3f` | tests: `needs()` per row, argv, bundle install (EXDEV), inputSig, stages over a scratch corpus, tree-kill, CLI |
+| `3033d9f2` | stamps fall back to the image's commit/branch; `deploy-homepage --to local` → `ARCHILYZER_HOMEPAGE_OUT` |
+| `89b16716` | `.gitignore`: `/export/out` |
+
+**Review fixes** (review `s1-review.md`: SHIP AFTER FIXES; the coordinator's list, plus S5's host-id note):
+
+| sev | fix | commit |
+|---|---|---|
+| HIGH | `inputSig` also signs the resolved social links, the hub url, `buildArchives` and the footer's sibling sites; one test each | `b88cbe8d` |
+| MEDIUM | the lock's host is `ARCHILYZER_HOST_ID` (else the hostname); a pid whose process started after the lock's `since` is not its holder (`/proc` start time, injectable; no `/proc` = pid-alive alone); a foreign host's wait line names both hosts and how to clear it | `99a8a939` |
+| MEDIUM | a run's no-op build no longer holds its deploy: under `builtAfter` the bundle counts as current when it matches the current index (`inputSig` / `hubSig` / `indexStampId`) and that index ran at or after `builtAfter`; a no-op build writes `built.checkedAt`, and `changedChannels` / config changes are measured against `builtCheckedAt` = max(builtAt, checkedAt) — S3's chip uses the same | `d1d19add` |
+| LOW | a detached HEAD records `branch: null`, and production refuses null like any branch but `main` | `d1d19add` |
+| SEAM | `ARCHILYZER_BRANCH` / `ARCHILYZER_COMMIT` win over git in the stamps (S4's e2e sets `ARCHILYZER_BRANCH=main`) | `d1d19add` |
+| MEDIUM | `publish deploy all` skips only private and (to Pages) project-less sites; every other refusal fails the run (exit 1) after the rest | `32f8a37d` |
+| LOW | an interrupted install (`out.prev`, no `out`) is restored before anything else; `built.json` is removed before the swap, written last | `32f8a37d` |
+| LOW | a second SIGTERM / Ctrl-C (CLI and stage child) SIGKILLs the detached process groups before exiting (`killChildTreesNow`) | `32f8a37d` |
+| LOW | the `stage` usage names `--allow-missing-media` | `32f8a37d` |
+
+Not taken (the review's other lows, left for S6's list): the read-then-`rm` race in `removeIfUnchanged`; the lock's
+place beside the checkout's builds dir while a worktree's `export/public` links into the primary's.
+
+**Gates** (all from the worktree root): tsc clean at every commit; common **3219 passed** (58 new: stamps 6,
+stageLock 8, stages 21, bundle 7, stageRun 6, inputSig 4, runChild 2, `_cli` +4); editor unit **142**;
+`test:scripts` **596 + 3 skipped**; mcp **289**; export unit **116**; homepage unit **23**; `pnpm --filter editor exec
+next build` ok (402 s, the machine busy); `pnpm --filter export exec next build` ok (59 s, over a scratch full site —
+see "Found"); `pnpm --filter homepage run build:nodata` ok (44 s); umtool's capped build ok (38 s). e2e (editor
+suite, `s1-specs.txt`: build, deploy-page, site-publish-preview, sites-homepage, duplicate-shorts, cut-release,
+ops-api): **52 passed, 0 failed, 2.5 min** — after a first launch died on "Timed out waiting 120000ms from
+config.webServer" (the linked primary `export/public` held a cited site's compose, so the export dev server 500ed on
+`summaries/manifest.json`); re-run with a scratch FULL site composed into the worktree's own `public/` (FACTS :3485's
+"Copy a composed fixture site into it"), cleaned after. **After the review fixes:** tsc clean; common
+**3229 passed** (10 new: inputSig +1, stageLock +3, stages +2, stageRun +2, bundle +1, runChild +1); e2e (same seven, same seeding) **52 passed, 0 failed, 2.1 min**. Numbers tool: none. Live smoke over a scratch corpus (`s1-smoke-build.sh`): `publish index`
+(9 s) → `publish build smoke` (85 s, bundle installed by rename, export/out a relative link) → again: no-op →
+`stage deploy-site … --to local` without the env: refused → `publish deploy smoke --to local`: copied + recorded →
+`publish hub`: refused by `builtHubProblem` (above) → `build site smoke --nodata`: alias printed, forced rebuild.
+
+### Slice S5, as shipped — the image half: the container can publish, yt-dlp can be substituted, the doctor checks it (2026-10-06)
+
+Branch `r18/docker-publish` off `r18/integration` `ce66f2d3`, worktree `~/Projects/r18-docker-publish`
+(editor 7001, test 7011, export 7010), one Opus implementer. Scratch files `s5-*` in the job's `tmp`. The
+plan is "Docker (a)–(g)" and "`archilyzer doctor` adds" above. This is the IMAGE HALF: S1 and S2 had not
+merged, so the doctor's `wrangler`, `publish-lock` and `index-stamp` checks, the `WRANGLER_BIN` /
+`E2E_LIVE_CHECK` declarations and the publish-stage smoke are the second half (below, "Left").
+
+**What was found before building.**
+- `ARCHILYZER_CONFIG_DIR` is already honoured by `getPaths()` (`paths.ts:218`), and `FILTER_REPO_PIPX_SPEC`
+ (`git-filter-repo==2.47.0`) already existed in `source.ts:113` — the drift test pins the Dockerfile to it.
+- `source.ts` reads the repository with `git --git-dir=<repo>`, so `ARCHILYZER_SOURCE_REPO` must name a git
+ DIR (the host's common dir), not a work tree — which is what the overlay mounts.
+- `git filter-repo --version` prints the script's hash (`a40bce548d2c`), never `2.47.0`; the version is
+ read from `pipx list` (`git-filter-repo 2.47.0`).
+- `env_file: .env` (x-app) already passes every key of `.env` to every app; `x-app-env` sets none of the
+ credentials, so nothing overrides them. Verified by reading the merged `docker compose config`.
+- Debian's `python3-pycryptodome` installs as `Cryptodome` (yt-dlp tries it first) and bookworm's
+ `python3-websockets` is 10.4 — below what yt-dlp's websockets handler wants, so a from-source yt-dlp
+ runs without that handler (optional; only some live-stream extractors use it).
+
+**What it does.**
+- **Dockerfile.** runtime-base and runtime-cuda (which repeats it) add `python3 python3-venv pipx` and
+ yt-dlp's optional modules; `PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install
+ git-filter-repo==2.47.0`; `ARCHILYZER_IMAGE_YTDLP=/usr/local/bin/yt-dlp` beside `YTDLP_BIN`;
+ `/usr/local/bin/yt-dlp-from-source` → `docker/yt-dlp-from-source.sh` (refuses with a sentence and exit
+ 127 when no `yt_dlp/` package is mounted). `ARCHILYZER_COMMIT` / `ARCHILYZER_BRANCH` build args become
+ ENV as the LAST layer of each of the three targets, so a new commit re-runs one ENV layer. The three
+ targets and the glibc ordering are untouched; wrangler is not installed globally.
+- **Entrypoint.** Every editor boot prints `yt-dlp: <path> <version> (image|override)` after the
+ optional self-update — `MISSING` (with the first stderr line) when it is not there or does not run; the
+ editor still starts. image vs override compares the two paths after `readlink -f`. `update_ytdlp` skips
+ an override with a two-line warning. `/data/source.git` (or `ARCHILYZER_SOURCE_REPO`) is added to git's
+ `safe.directory` once — only when absent, so a restarted container does not pile up entries.
+ `ARCHILYZER_CONFIG_DIR` is made (700, empty) when absent. The `homepage` service serves
+ `ARCHILYZER_HOMEPAGE_OUT` (`/data/builds/homepage`) when it is non-empty, else the baked
+ `homepage/out` — chosen at boot. `exec "$@"` stays.
+- **Compose.** `x-app-env` gains `ARCHILYZER_CONFIG_DIR=/data/config/archilyzer` and
+ `ARCHILYZER_HOMEPAGE_OUT`; `x-app.build.args` passes the two build facts from the shell; the
+ `homepage` service mounts `builds`. New overlays: `docker-compose.source.yml` (`ARCHILYZER_SOURCE_HOST_DIR`,
+ default `./.git`, read-only at `/data/source.git`; sets `ARCHILYZER_SOURCE_REPO`) and
+ `docker-compose.ytdlp.yml` (`${YTDLP_SOURCE_HOST_DIR:?…}` read-only at `/opt/yt-dlp-src`; sets
+ `YTDLP_BIN=/usr/local/bin/yt-dlp-from-source`).
+- **`source.ts`.** `sourceRepoFor(ctx, cwd)` — `ARCHILYZER_SOURCE_REPO` first, then the checkout's common
+ dir — replaces both `commonDir` call sites (the publish and `publishedSourceProblem`). A variable naming a
+ path that is not there is a `SourceRefusal` naming it (never the "no repository" sentence).
+- **`docker/publish-site.sh`** is a wrapper: usage, the early private refusal, then `publish index`,
+ `publish build <id>`, `publish deploy <id> --to local` (S1's CLI rows, by name).
+- **`envVars.ts`** (ENVIRONMENT.md regenerated): `CLOUDFLARE_API_TOKEN`, `ARCHILYZER_SOURCE_REPO`,
+ `YTDLP_SOURCE_HOST_DIR`, `YTDLP_SOURCE_DIR`, `YTDLP_AUTO_UPDATE`, `XDG_CONFIG_HOME` (runtime);
+ `ARCHILYZER_HOMEPAGE_OUT`, `ARCHILYZER_IMAGE_YTDLP`, `ARCHILYZER_COMMIT`, `ARCHILYZER_BRANCH`,
+ `ARCHILYZER_SOURCE_HOST_DIR` (docker); the R2/account rows and `ARCHILYZER_CONFIG_DIR` say where they
+ come from in Docker. Exported for S1's stamps: `IMAGE_COMMIT_ENV`, `IMAGE_BRANCH_ENV` and
+ `imageBuildFacts(env)` → `{commit, branch}` (an empty baked value is null).
+- **Doctor.** New section `downloader` (`yt-dlp`: path, version by exit status, `image` | `host` |
+ `override`; an override that does not run, or one beside `YTDLP_AUTO_UPDATE`, warns); new section
+ `publish` (`cloudflare-auth`: the token SET, or wrangler's login config by PATH, never read; warns only
+ when a site names a `cloudflareProject`; `r2-keys`: only with `archiveStorage.bucket`, names the unset
+ keys; `export-builds`: writable, free ≥ 1.5× the `<id>/out` bundles, a missing dir is a note);
+ `source publish` gains `source-repo` (`ARCHILYZER_SOURCE_REPO` naming nothing FAILS — the publish
+ refuses; no repository is a note, a warning once the operator's files exist; main's commit when it
+ reads) and `config-dir` (exists, entries counted, writable). No value is printed; still read-only.
+- **RUNNING_IN_DOCKER.md**: "Publish the archive" (stages in the container; `exec`, never `run --rm`;
+ Cloudflare from `.env`; the homepage and its `/source` mount, the config volume), "Substituting yt-dlp",
+ "Two build runners, and the container has one" (replaces the fallback section), the Windows checklist,
+ what is in the image, troubleshooting. `.env.example` names every new variable (not `E2E_LIVE_CHECK`:
+ a test knob does not belong in a real instance's `.env`).
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `060927fb` | `publish:` `sourceRepoFor` — `ARCHILYZER_SOURCE_REPO` first; a missing path refuses by name; 1 test |
+| `f0dfa39a` | `docker:` the image (python, pipx, filter-repo, the yt-dlp hook, build facts); entrypoint; compose + two overlays; `publish-site.sh` wrapper; envVars + ENVIRONMENT.md; 2 drift tests in `buildImage.test.ts` |
+| `e1a3b49b` | `doctor:` `downloader/yt-dlp`, `publish/{cloudflare-auth,r2-keys,export-builds}`, `source publish/{source-repo,config-dir}`; 5 tests |
+| `90722346` | `docker:` `.env.example` |
+| `32c16698` | `docker:` the entrypoint makes the config dir |
+| `ed5b5db8` | `docs:` RUNNING_IN_DOCKER.md |
+| this one | `plans:` this section; the editor changelog |
+
+#### Gates (logs `$T/s5-*.log`)
+
+- **tsc** (all workspaces) clean at every commit (91 s at `ed5b5db8`).
+- **common:** **3,169/3,169** (new: `doctor.test.ts` +5, `buildImage.test.ts` +2, `source.test.ts` +1).
+ **Editor unit:** 142/142. **mcp:** 289/289. **test:scripts:** 595 passed, 1 failed, 3 skipped (599) —
+ `queue-lock.test.mjs` "prints a banner naming the holder while waiting" at a load average of 24, with
+ the editor build running; the file alone afterwards: 11/11. The slice touches nothing under `scripts/`.
+- **Build:** `pnpm --filter editor exec next build` exit 0, 253 s.
+- **Image:** `docker buildx build --target runtime` in a builder capped at 8 GB (`--driver-opt memory=8g
+ memory-swap=8g`), `WHISPER_BUILD_JOBS=4`: exit 0 in 280 s from an empty builder cache, 241 s for the
+ rebuild after the doctor commit. `archilyzer:r18smoke` is **1.76 GB** (`docker image inspect .Size`).
+ **Only `--target runtime` was built.** `runtime-vulkan` (trixie apt, the same pipx install) and
+ `runtime-cuda` (ubuntu 24.04: `pipx`, `python3-pycryptodome`, `python3-brotli` from universe) carry the
+ same package list unverified by a build — each to be built once, capped, before the rollout.
+- **Compose smoke** (`-p r18smoke`, the `channel-with-counts` e2e fixture + `sites/testsite` copied into
+ `$T/s5-corpus` and bind-mounted over the corpus volume, `ARCHILYZER_FETCH_MODEL=none`,
+ `ARCHILYZER_IDLE_BOOT=1`, the editor alone): the boot log shows `yt-dlp: /usr/local/bin/yt-dlp
+ 2026.08.19 (image)`; `exec editor pnpm archilyzer doctor` lists `downloader/yt-dlp` ok (image),
+ `publish/cloudflare-auth` and `export-builds` (notes), `filter-repo` ok (`git filter-repo a40bce548d2c`),
+ `source-repo` and `config-dir` (exit 1 only for the model the smoke skipped); `python3 -c "import
+ yt_dlp"` exit 1 and `yt-dlp-from-source --version` exit 127 with its sentence; `pipx list` →
+ `git-filter-repo 2.47.0`; `ARCHILYZER_COMMIT`/`BRANCH` baked. With `docker-compose.ytdlp.yml` over a
+ two-file `yt_dlp/` package and `YTDLP_AUTO_UPDATE=1`: the entrypoint's skip warning, `yt-dlp:
+ /usr/local/bin/yt-dlp-from-source 2099.01.01.s5-smoke (override)`, the wrapper exit 0, the doctor's
+ `yt-dlp` WARN. `safe.directory` has one entry after a `docker restart`. With
+ `docker-compose.source.yml` over a throwaway repo in `$T`: `source-repo` ok with its main, `rev-parse`
+ as root works, the mount is read-only. The `homepage` service serves the baked build, then the builds
+ volume's after a file lands there and it restarts. `down -v` after.
+- **e2e:** none for this slice. **Numbers tool:** none. **Publish-stage smoke** (`publish
+ index/build/deploy` in the container): not run — S1's CLI rows are not on this branch; it is the
+ parent's after S1/S2 merge.
+
+**Deviations from the plan, one sentence each.**
+- `ARCHILYZER_HOMEPAGE_OUT` and `ARCHILYZER_SOURCE_HOST_DIR` are new names the plan did not list: the
+ first is where `deploy-homepage --to local` writes (S1 must read it), the second lets a worktree mount
+ the primary's `.git`.
+- `python3-venv` is installed beside `pipx` (pipx makes a venv); the plan's package list omitted it.
+- The build facts are ENV in each final target rather than once in runtime-base, so a commit does not
+ invalidate the Vulkan apt layer.
+- `config-dir` not writable is a note, not a warning: the publish only reads the rules.
+- The source-repo "homepage policy on" grade waits for S3's `settings.publish.homepage`; today the
+ warning keys off the operator's files existing (the doctor's existing "intends" signal).
+- `E2E_LIVE_CHECK` and `WRANGLER_BIN` are not declared yet: nothing on this branch reads them, and the
+ envVars test refuses a declaration nothing names (S2 adds the reads).
+
+**Left for the second half (after S1/S2 merge).**
+- Doctor `wrangler` (the binary from `wranglerBin(paths)`, its major against `WRANGLER_MAJOR`),
+ `publish-lock` (a dead pid in `.publish.lock`, via `stageLock.ts`), `index-stamp` (age; sites built from
+ an older stamp, via `stamps.ts`); `export-builds` to read `built.json` `bytes` instead of walking.
+- `WRANGLER_BIN` / `E2E_LIVE_CHECK` in `envVars.ts` (whichever of S2/S5 lands second).
+- The publish-stage smoke in the container (rollout's shape: `publish index && publish build <fixture> &&
+ publish deploy <fixture> --preview smoke`, no token → refused before wrangler, a bogus token → refused by
+ Cloudflare), plus `exec editor pnpm archilyzer source publish --check` over a throwaway repo with
+ throwaway rules copied into `/data/config/archilyzer` (the container's mirror over the read-only,
+ foreign-owned mount, which the doctor's `rev-parse` alone does not prove).
+- The envVars dedupe at the merges: S2 also declares `CLOUDFLARE_API_TOKEN` and `ARCHILYZER_HOMEPAGE_OUT`
+ (one row per name, `readBy` unioned); S1's `stamps.ts` imports `imageBuildFacts` from `lib/envVars`
+ instead of its own. `cloudflare-auth` grades through S2's `cloudflareCredentialProblem` (which accepts
+ `CLOUDFLARE_API_KEY` + `CLOUDFLARE_EMAIL`).
+- Building `runtime-vulkan` and `runtime-cuda` once (above).
+
+**Found and left.**
+- The image has no gitleaks and no stagit: a source publish from the container skips the secret scan
+ (with its WARNING; the literal audit still runs) and has no history pages. RUNNING_IN_DOCKER.md says so;
+ a pinned gitleaks in the image is a follow-up.
+- `.env` (now carrying the Cloudflare token and the R2 keys) reaches `site`, `homepage` and `umtool`
+ through the shared `env_file`, as it always did; nothing serves or prints it. An editor-only credentials
+ file is an option, not done.
+- The shared tool probe (`toolProbe.mjs`) counts any output from a failing `--version` as presence, so the
+ `tools/yt-dlp` row reads `ok` with the wrapper's sentence as its "version" when no checkout is mounted;
+ the new `downloader/yt-dlp` row asks by exit status and is the honest one. Not changed: the probe is
+ shared with `umtool doctor`.
+- `docker images` reported the previous `archilyzer:local` (6 weeks old) at 7.3 GB; this build is
+ 1.76 GB. Not investigated.
+
+**Review fixes** (review `SHIP AFTER FIXES`, no highs; the four asked for now)
+
+| Commit | Fix |
+|---|---|
+| `ba83cbff` | `docker-compose.source.yml`: long bind syntax, `create_host_path: false` — a missing host `.git` now fails `up` ("bind source path does not exist: …", verified) instead of becoming an empty root-owned dir |
+| `fb1a1f24` | `YTDLP_AUTO_UPDATE`: the doctor reads it with the entrypoint's exact-match rule (`1`, `true`, `yes`, `on` as written; `TRUE` is off in both), +1 test loop; RUNNING_IN_DOCKER.md states the rule |
+| `07bc2395` | RUNNING_IN_DOCKER.md: no gitleaks or stagit in the image — the container's source publish skips the secret scan (with its warning) and the history pages; pinned gitleaks a follow-up |
+| `e9fe6bdd` | the record: only `--target runtime` was built (vulkan and cuda unverified, left for the rollout); the second half's added items; found and left |
+| `908c7c4a` | **Node 22** (S2's review: the pinned wrangler 4.147.0 has `engines.node >=22.0.0`, so every deploy from a Node 20 image would exit 1): `NODE_IMAGE` and `RUNTIME_IMAGE` `node:22-bookworm-slim` (glibc 2.36, unchanged — the glibc rule holds), the Vulkan overlay `node:22-trixie-slim`, runtime-cuda `NODE_MAJOR=22` on ubuntu 24.04. Two drift tests in `buildImage.test.ts`: one Node major across all of them (the native modules are built once, against the build stage's ABI; proven red with `NODE_MAJOR=20`), and that major ≥ wrangler's `engines.node` floor (skipped here — wrangler is S2's devDependency; S2's worktree has 4.147.0, `>=22.0.0`) |
+| `fc793037` | RUNNING_IN_DOCKER.md names Node 22 in the image's contents; this table |
+| `55d779a1` | **The publish lock's host identity** (S1's review: `os.hostname()` in a container is its id, new on every recreate, so a crashed holder's lock would look foreign forever; S1's `stageLock.ts` reads `ARCHILYZER_HOST_ID ?? os.hostname()`): `ARCHILYZER_HOST_ID: archilyzer-editor` on the **editor service's** `environment`, not `x-app-env` — site, homepage and umtool share the builds volume, and a container carrying the same id with its own pid namespace would judge the editor's live lock dead and take it (visible in `docker compose config` either way; checked: only the editor has it). envVars row, no TODO needed: the compose file names it, which the test accepts (`readBy` names `stageLock.ts`, S1's). RUNNING_IN_DOCKER.md: why the id is fixed, that `run --rm` would now carry it with other pids (one more reason for `exec`), and how to clear a foreign-host lock (`rm /data/builds/.export-builds/.publish.lock`, only when nothing is publishing) |
+| `c238970a` | SETUP.md: Node 22 — Next needs ≥ 20.9, deploying runs the pinned wrangler (≥ 22) |
+| this one | this table |
+
+Re-run after the fixes at `07bc2395`: tsc (all workspaces) clean; `doctor`, `buildImage`, `source` and
+`envVars` tests **61/61** (`$T/s5-fix-tests.log`).
+
+Node 22, at `908c7c4a`: `buildImage.test.ts` 5 passed, 1 skipped (the wrangler floor); common tsc clean.
+`--target runtime` rebuilt in the capped builder, exit 0, 349 s; the image is 1.79 GB. Smoke (`-p r18smoke`,
+the same fixture, `down -v` after): `node --version` in the container is **v22.23.3**; the boot log's
+`yt-dlp: /usr/local/bin/yt-dlp 2026.08.19 (image)`; `/api/pulse` 200; the build stage's native modules
+load in the runtime — `lmdb` opens, writes and reads (`process.versions.modules` 127), `msgpackr-extract`'s
+binding loads; `archilyzer doctor` reports `node v22.23.3` ok and every S5 check, exit 1 only for the
+model the smoke skips. vulkan and cuda still unbuilt (above).
## Rollout
