commit 938b4241bf3319976bf2fa43adfb936c161ed5f5
parent 4153f62ed4e01c0b3bd52b09c8a9e9ccacc58262
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 6 Oct 2026 09:34:16 -0400
Merge r18/docker-publish (slice S5: a fixed host id for the publish lock; SETUP.md names Node 22 for deploying)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
6 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/ENVIRONMENT.md b/ENVIRONMENT.md
@@ -61,6 +61,7 @@ Tokens, credentials and knobs a running process reads. Most configuration is not
| `R2_SECRET_ACCESS_KEY` | — | See `R2_ACCESS_KEY_ID`. | common/publish/build.ts, common/bin/doctor.ts (set or not) |
| `CLOUDFLARE_ACCOUNT_ID` | — | The Cloudflare account: the R2 endpoint's, and the one wrangler deploys to when the token can see more than one. In Docker it comes from `.env`. | common/publish/build.ts, wrangler, common/bin/doctor.ts (set or not) |
| `CLOUDFLARE_API_TOKEN` | unset (wrangler's own `wrangler login` config, on a host) | The API token every deploy's wrangler authenticates with (Cloudflare Pages: Edit). The way a container deploys — there is no browser for `wrangler login` in one; set it in `.env`. | wrangler (every deploy), common/bin/doctor.ts (set or not, never the value) |
+| `ARCHILYZER_HOST_ID` | the hostname | Which host the publish lock (`<EXPORT_BUILDS_DIR>/.publish.lock`) names as its holder's: a lock from this host whose pid is dead is stale and taken over; another host's is waited on. docker-compose.yml fixes it for the editor (`archilyzer-editor`), whose hostname is a container id that changes on every recreate. | common/publish/stageLock.ts (the publish lock) |
| `ARCHILYZER_SOURCE_REPO` | this checkout's git common dir | The git DIR `archilyzer source publish` mirrors `main` from, when the checkout has none: in Docker, `/data/source.git`, the host's git common dir mounted read-only by docker-compose.source.yml. A value that names nothing refuses the publish. | common/publish/source.ts, common/bin/doctor.ts, docker/entrypoint.sh |
| `YTDLP_SOURCE_HOST_DIR` | — (required by the overlay) | Docker: the HOST path of a yt-dlp source checkout (the directory holding `yt_dlp/`), mounted read-only at `/opt/yt-dlp-src` by docker-compose.ytdlp.yml. See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md), "Substituting yt-dlp". | docker-compose.ytdlp.yml |
| `YTDLP_AUTO_UPDATE` | off | Docker: `1` runs `yt-dlp -U` on every editor boot — on the image's yt-dlp only; an override (`YTDLP_BIN` naming another) is left alone, with a warning. | docker/entrypoint.sh, common/bin/doctor.ts |
diff --git a/RUNNING_IN_DOCKER.md b/RUNNING_IN_DOCKER.md
@@ -225,10 +225,21 @@ built but never deployed, locally or anywhere else.
**`exec`, never `run --rm`.** `docker compose run --rm editor …` starts a SECOND
container with its own copy of the image's `export/public` and a second writer on
the index, and the publish lock (which keeps a stage you start from colliding
-with one the editor is running) cannot see across containers. `exec` runs in the
+with one the editor is running) cannot see across containers — worse, the second
+container carries the editor's host identity below with its own pids, so it would
+judge the editor's live lock dead and take it. `exec` runs in the
editor's own container, beside its jobs, under the same lock. `pnpm ops publish`
from the host goes through the editor too.
+The lock names its holder by host and pid. A container's hostname changes every
+time it is recreated, so compose gives the editor a fixed identity
+(`ARCHILYZER_HOST_ID=archilyzer-editor`): a lock left by a stage that died with the
+container is then recognised as this editor's own, and taken over once its pid is
+gone. A lock naming any OTHER host is waited on, never taken. If one is left
+behind — say, from before this setting, or by a host install sharing the volume
+that is gone for good — and you are sure nothing is publishing, delete it:
+`docker compose exec editor rm /data/builds/.export-builds/.publish.lock`.
+
#### Deploying to Cloudflare from the container
The same stages deploy to Cloudflare Pages. The container has no browser for
diff --git a/SETUP.md b/SETUP.md
@@ -30,7 +30,7 @@ homepage):
| Tool | Version | Notes |
| --- | --- | --- |
-| **Node.js** | **≥ 20.9** (LTS 20 or 22) | Required by Next.js 16. The code is typed against Node 20. |
+| **Node.js** | **22** (≥ 20.9 runs the apps) | Next.js 16 needs ≥ 20.9, but **deploying** runs the wrangler pinned in `common/package.json`, which refuses anything below Node 22 — so use 22. The Docker image ships 22. |
| **pnpm** | **9+** | Lockfile is v9. Easiest via Corepack (bundled with Node) — see below. |
| **git** | any recent | To clone the repo. |
| **C/C++ toolchain** | platform default | Only if pnpm can't find a prebuilt binary for a native module (`lmdb`, `sharp`, …). Usually not needed on mainstream platforms. |
diff --git a/common/lib/envVars.ts b/common/lib/envVars.ts
@@ -97,6 +97,7 @@ const DECLARED: EnvVarDecl[] = [
{ name: "R2_SECRET_ACCESS_KEY", audience: "runtime", default: "—", readBy: "common/publish/build.ts, common/bin/doctor.ts (set or not)", doc: "See `R2_ACCESS_KEY_ID`." },
{ name: "CLOUDFLARE_ACCOUNT_ID", audience: "runtime", default: "—", readBy: "common/publish/build.ts, wrangler, common/bin/doctor.ts (set or not)", doc: "The Cloudflare account: the R2 endpoint's, and the one wrangler deploys to when the token can see more than one. In Docker it comes from `.env`." },
{ name: "CLOUDFLARE_API_TOKEN", audience: "runtime", default: "unset (wrangler's own `wrangler login` config, on a host)", readBy: "wrangler (every deploy), common/bin/doctor.ts (set or not, never the value)", doc: "The API token every deploy's wrangler authenticates with (Cloudflare Pages: Edit). The way a container deploys — there is no browser for `wrangler login` in one; set it in `.env`." },
+ { name: "ARCHILYZER_HOST_ID", audience: "runtime", default: "the hostname", readBy: "common/publish/stageLock.ts (the publish lock)", doc: "Which host the publish lock (`<EXPORT_BUILDS_DIR>/.publish.lock`) names as its holder's: a lock from this host whose pid is dead is stale and taken over; another host's is waited on. docker-compose.yml fixes it for the editor (`archilyzer-editor`), whose hostname is a container id that changes on every recreate." },
{ name: "ARCHILYZER_SOURCE_REPO", audience: "runtime", default: "this checkout's git common dir", readBy: "common/publish/source.ts, common/bin/doctor.ts, docker/entrypoint.sh", doc: "The git DIR `archilyzer source publish` mirrors `main` from, when the checkout has none: in Docker, `/data/source.git`, the host's git common dir mounted read-only by docker-compose.source.yml. A value that names nothing refuses the publish." },
{ name: "YTDLP_SOURCE_HOST_DIR", audience: "runtime", default: "— (required by the overlay)", readBy: "docker-compose.ytdlp.yml", doc: "Docker: the HOST path of a yt-dlp source checkout (the directory holding `yt_dlp/`), mounted read-only at `/opt/yt-dlp-src` by docker-compose.ytdlp.yml. See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md), \"Substituting yt-dlp\"." },
{ name: "YTDLP_AUTO_UPDATE", audience: "runtime", default: "off", readBy: "docker/entrypoint.sh, common/bin/doctor.ts", doc: "Docker: `1` runs `yt-dlp -U` on every editor boot — on the image's yt-dlp only; an override (`YTDLP_BIN` naming another) is left alone, with a warning." },
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -118,6 +118,13 @@ services:
command: ["editor"]
environment:
<<: *app-env
+ # The publish lock's host identity (common/publish/stageLock.ts). Its
+ # default, the hostname, is the container id here and changes on every
+ # recreate, so a lock left by a crashed stage would look like another
+ # host's forever. Fixed, so this editor recognises its own stale lock.
+ # The EDITOR only, deliberately: another container with the same id but
+ # its own pid namespace would judge the editor's live lock dead.
+ ARCHILYZER_HOST_ID: archilyzer-editor
volumes:
- corpus:/data/transcripts
- config:/data/config
diff --git a/plans/release-18.md b/plans/release-18.md
@@ -517,7 +517,10 @@ merged, so the doctor's `wrangler`, `publish-lock` and `index-stamp` checks, the
| `07bc2395` | RUNNING_IN_DOCKER.md: no gitleaks or stagit in the image — the container's source publish skips the secret scan (with its warning) and the history pages; pinned gitleaks a follow-up |
| `e9fe6bdd` | the record: only `--target runtime` was built (vulkan and cuda unverified, left for the rollout); the second half's added items; found and left |
| `908c7c4a` | **Node 22** (S2's review: the pinned wrangler 4.147.0 has `engines.node >=22.0.0`, so every deploy from a Node 20 image would exit 1): `NODE_IMAGE` and `RUNTIME_IMAGE` `node:22-bookworm-slim` (glibc 2.36, unchanged — the glibc rule holds), the Vulkan overlay `node:22-trixie-slim`, runtime-cuda `NODE_MAJOR=22` on ubuntu 24.04. Two drift tests in `buildImage.test.ts`: one Node major across all of them (the native modules are built once, against the build stage's ABI; proven red with `NODE_MAJOR=20`), and that major ≥ wrangler's `engines.node` floor (skipped here — wrangler is S2's devDependency; S2's worktree has 4.147.0, `>=22.0.0`) |
-| this one | RUNNING_IN_DOCKER.md names Node 22 in the image's contents; this table |
+| `fc793037` | RUNNING_IN_DOCKER.md names Node 22 in the image's contents; this table |
+| `55d779a1` | **The publish lock's host identity** (S1's review: `os.hostname()` in a container is its id, new on every recreate, so a crashed holder's lock would look foreign forever; S1's `stageLock.ts` reads `ARCHILYZER_HOST_ID ?? os.hostname()`): `ARCHILYZER_HOST_ID: archilyzer-editor` on the **editor service's** `environment`, not `x-app-env` — site, homepage and umtool share the builds volume, and a container carrying the same id with its own pid namespace would judge the editor's live lock dead and take it (visible in `docker compose config` either way; checked: only the editor has it). envVars row, no TODO needed: the compose file names it, which the test accepts (`readBy` names `stageLock.ts`, S1's). RUNNING_IN_DOCKER.md: why the id is fixed, that `run --rm` would now carry it with other pids (one more reason for `exec`), and how to clear a foreign-host lock (`rm /data/builds/.export-builds/.publish.lock`, only when nothing is publishing) |
+| `c238970a` | SETUP.md: Node 22 — Next needs ≥ 20.9, deploying runs the pinned wrangler (≥ 22) |
+| this one | this table |
Re-run after the fixes at `07bc2395`: tsc (all workspaces) clean; `doctor`, `buildImage`, `source` and
`envVars` tests **61/61** (`$T/s5-fix-tests.log`).
