commit 6b7a36a1050a92daf10552158971bb357f7dc2e6
parent 0278e0af2c2dfefde5332afce1cd9ec53772875b
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 6 Oct 2026 09:06:44 -0400
Merge r18/docker-publish (slice S5, image half: the runtime image publishes — python/pipx/git-filter-repo, yt-dlp substitution, build stamps, doctor checks, compose overlays)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
18 files changed, 1332 insertions(+), 91 deletions(-)
diff --git a/.env.example b/.env.example
@@ -86,9 +86,72 @@
# Run `yt-dlp -U` on every boot. An image pins yt-dlp at build time and a stale
# yt-dlp is the most common reason downloads start failing. Off by default
-# because it is a network call at startup.
+# because it is a network call at startup. It updates the IMAGE's yt-dlp only:
+# with YTDLP_BIN pointing elsewhere (below) it warns and leaves that one alone.
#YTDLP_AUTO_UPDATE=1
+# ---------------------------------------------------------------------------
+# Substituting yt-dlp (your own build, no image rebuild)
+# ---------------------------------------------------------------------------
+# The image ships the release yt-dlp at /usr/local/bin/yt-dlp
+# (ARCHILYZER_IMAGE_YTDLP — baked, do not set it). To run another one, point
+# YTDLP_BIN at it. Two ways:
+#
+# a zipapp built on the host (`make yt-dlp` in a yt-dlp checkout; it runs on
+# the image's python3), copied into the config volume:
+# docker compose exec editor mkdir -p /data/config/bin
+# docker compose cp ./yt-dlp editor:/data/config/bin/yt-dlp
+# docker compose exec editor chmod 755 /data/config/bin/yt-dlp
+#YTDLP_BIN=/data/config/bin/yt-dlp
+#
+# a SOURCE checkout, run with the image's python — set the checkout's host
+# path and start with the overlay, which sets YTDLP_BIN for you:
+# docker compose -f docker-compose.yml -f docker-compose.ytdlp.yml up -d
+#YTDLP_SOURCE_HOST_DIR=/home/you/yt-dlp-patched
+#
+# Every editor boot logs `yt-dlp: <path> <version> (image|override)`.
+
+# ---------------------------------------------------------------------------
+# Publishing to Cloudflare (deploys run IN the container)
+# ---------------------------------------------------------------------------
+# The editor reads these and every publish stage it runs inherits them. Values
+# are never printed — `docker compose exec editor pnpm archilyzer doctor` says
+# only whether each is set. A container has no browser for `wrangler login`, so
+# deploying from one needs the token.
+#
+# An API token with "Cloudflare Pages: Edit" (dash.cloudflare.com → My Profile →
+# API Tokens), and your account id.
+#CLOUDFLARE_API_TOKEN=
+#CLOUDFLARE_ACCOUNT_ID=
+#
+# R2 S3 credentials — only when Settings → archive storage names a bucket for
+# oversize archive zips.
+#R2_ACCESS_KEY_ID=
+#R2_SECRET_ACCESS_KEY=
+#
+# The wrangler deploys run. Default: the one pinned in the workspace
+# (common/node_modules/.bin/wrangler); nothing is fetched at deploy time.
+#WRANGLER_BIN=
+#
+# Driving the editor from a shell or an agent (`pnpm ops`, the MCP's
+# fetch_clip): the bearer token /api/ops/* and /api/worker/* answer to. Unset,
+# both surfaces are off.
+#WORKER_TOKEN=
+
+# ---------------------------------------------------------------------------
+# The homepage's /source mirror (optional)
+# ---------------------------------------------------------------------------
+# Publishing the homepage mirrors this repo's `main`, and the image has no .git.
+# docker-compose.source.yml mounts the host's git dir read-only and sets
+# ARCHILYZER_SOURCE_REPO=/data/source.git for you. Default `./.git`; in a git
+# worktree, the primary checkout's `.git`.
+#ARCHILYZER_SOURCE_HOST_DIR=./.git
+#
+# The scrub rules and denylist it needs live in ARCHILYZER_CONFIG_DIR, which
+# docker-compose.yml sets to /data/config/archilyzer (the config volume):
+# docker compose cp source-scrub.txt editor:/data/config/archilyzer/
+# docker compose cp source-denylist.txt editor:/data/config/archilyzer/
+
# Boot the editor WITHOUT resuming the schedulers, the auto-queue runners or the
# digest/backfill sweeps. Set this the first time you point a container at a
# corpus somebody else configured: its stored policies may say "sweep", and you
@@ -107,3 +170,9 @@
# Image
# ---------------------------------------------------------------------------
#ARCHILYZER_TAG=local
+#
+# Which commit and branch the image is built from, baked in for the publish
+# stamps (the image has no .git). Usually passed from the shell instead:
+# ARCHILYZER_COMMIT=$(git rev-parse HEAD) ARCHILYZER_BRANCH=$(git branch --show-current) docker compose build
+#ARCHILYZER_COMMIT=
+#ARCHILYZER_BRANCH=
diff --git a/Dockerfile b/Dockerfile
@@ -303,15 +303,34 @@ FROM ${RUNTIME_IMAGE} AS runtime-base
# from a `docker compose exec` shell. debian:slim ships without it.
# aria2: archive.org files over BitTorrent (archive.org as the web seed), then
# seeded for a while; without it they are downloaded directly.
+# python3 + yt-dlp's optional modules (certifi, brotli, websockets, mutagen,
+# pycryptodome, requests): NOT for the yt-dlp below, which is a standalone
+# binary with its own python — for /usr/local/bin/yt-dlp-from-source, which
+# runs a yt-dlp SOURCE tree mounted at run time (docker-compose.ytdlp.yml).
+# pipx (+ python3-venv, which it needs to make a venv): installs git-filter-repo
+# below, which the homepage's source mirror runs.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ffmpeg aria2 zip unzip tar xz-utils gzip rsync curl ca-certificates git procps \
+ python3 python3-venv pipx python3-certifi python3-brotli python3-websockets \
+ python3-mutagen python3-pycryptodome python3-requests \
&& rm -rf /var/lib/apt/lists/*
+# git-filter-repo, PINNED, for `archilyzer source publish` (the homepage's
+# /source mirror). Installed, not left to `pipx run`, so a container publishes
+# with no network fetch at build time. The version is the one
+# common/publish/source.ts names in FILTER_REPO_PIPX_SPEC — a drift test
+# (common/publish/buildImage.test.ts) holds every `pipx install` here to it.
+# Into /opt + /usr/local/bin, not root's home: on PATH for every process.
+RUN PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install git-filter-repo==2.47.0 \
+ && git filter-repo --version
+
# yt-dlp as the standalone release binary (it bundles its own python), installed
# writable so `yt-dlp -U` works — see YTDLP_AUTO_UPDATE in docker/entrypoint.sh.
# A pinned yt-dlp goes stale fast, and a stale yt-dlp is the single most common
-# reason downloads start failing.
+# reason downloads start failing. It is THE IMAGE'S yt-dlp
+# (ARCHILYZER_IMAGE_YTDLP below); YTDLP_BIN may name another one at run time —
+# RUNNING_IN_DOCKER.md, "Substituting yt-dlp".
ARG TARGETARCH=amd64
RUN set -eux; \
case "${TARGETARCH}" in \
@@ -353,12 +372,21 @@ RUN npm install -g pnpm@9.15.4
WORKDIR /repo
COPY --from=build /repo /repo
+# The yt-dlp substitution hook: a wrapper that runs a yt-dlp SOURCE tree
+# mounted at YTDLP_SOURCE_DIR (default /opt/yt-dlp-src) with the python above.
+# Selected by YTDLP_BIN=/usr/local/bin/yt-dlp-from-source; see
+# docker-compose.ytdlp.yml.
+RUN ln -s /repo/docker/yt-dlp-from-source.sh /usr/local/bin/yt-dlp-from-source
+
# Data lives in volumes, never in the image. See docker-compose.yml.
+# YTDLP_BIN is the one the app runs; ARCHILYZER_IMAGE_YTDLP is the one this
+# image ships, so the entrypoint and `archilyzer doctor` can say "override".
ENV NODE_ENV=production \
NEXT_TELEMETRY_DISABLED=1 \
TRANSCRIPTS_DIR=/data/transcripts \
SETTINGS_FILE=/data/config/settings.json \
YTDLP_BIN=/usr/local/bin/yt-dlp \
+ ARCHILYZER_IMAGE_YTDLP=/usr/local/bin/yt-dlp \
EXPORT_INDEX_DIR=/data/builds/.export-index \
EXPORT_BUILDS_DIR=/data/builds/.export-builds \
ARCHILYZER_SITE_OUT=/data/builds/site
@@ -376,6 +404,15 @@ ENV ARCHILYZER_TRANSCRIBER=whisper-cpp \
WHISPER_BIN=/usr/local/bin/whisper-cli \
WHISPER_MODEL=/data/models/ggml-base.en.bin
+# Which commit and branch this image was built from — the publish stamps'
+# `commit`/`branch` where there is no .git (.dockerignore keeps it out). Last,
+# so a new commit re-runs one ENV layer and nothing else. Passed by compose from
+# the shell: ARCHILYZER_COMMIT=$(git rev-parse HEAD) docker compose build.
+ARG ARCHILYZER_COMMIT=
+ARG ARCHILYZER_BRANCH=
+ENV ARCHILYZER_COMMIT=${ARCHILYZER_COMMIT} \
+ ARCHILYZER_BRANCH=${ARCHILYZER_BRANCH}
+
# ---------------------------------------------------------------------------
# runtime-vulkan — parakeet.cpp on any Vulkan GPU. docker-compose.vulkan.yml.
#
@@ -405,18 +442,32 @@ ENV ARCHILYZER_TRANSCRIBER=parakeet \
WHISPER_BIN=/usr/local/bin/whisper-cli \
WHISPER_MODEL=/data/models/ggml-base.en.bin
+# Which commit and branch this image was built from — the publish stamps'
+# `commit`/`branch` where there is no .git (.dockerignore keeps it out). Last,
+# so a new commit re-runs one ENV layer and nothing else. Passed by compose from
+# the shell: ARCHILYZER_COMMIT=$(git rev-parse HEAD) docker compose build.
+ARG ARCHILYZER_COMMIT=
+ARG ARCHILYZER_BRANCH=
+ENV ARCHILYZER_COMMIT=${ARCHILYZER_COMMIT} \
+ ARCHILYZER_BRANCH=${ARCHILYZER_BRANCH}
+
# ---------------------------------------------------------------------------
# runtime-cuda — whisper.cpp on CUDA. NVIDIA only. docker-compose.gpu.yml.
# ---------------------------------------------------------------------------
FROM ${CUDA_RUNTIME_IMAGE} AS runtime-cuda
ARG NODE_MAJOR=20
+# The same python + pipx + git-filter-repo as runtime-base (read its comments).
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ffmpeg aria2 zip unzip tar xz-utils gzip rsync curl ca-certificates git gnupg procps \
+ python3 python3-venv pipx python3-certifi python3-brotli python3-websockets \
+ python3-mutagen python3-pycryptodome python3-requests \
&& curl -fsSL "https://deb.nodesource.com/setup_${NODE_MAJOR}.x" | bash - \
&& apt-get install -y --no-install-recommends nodejs \
&& rm -rf /var/lib/apt/lists/*
+RUN PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install git-filter-repo==2.47.0 \
+ && git filter-repo --version
ARG TARGETARCH=amd64
RUN set -eux; \
@@ -461,6 +512,7 @@ RUN ldconfig
WORKDIR /repo
COPY --from=build /repo /repo
+RUN ln -s /repo/docker/yt-dlp-from-source.sh /usr/local/bin/yt-dlp-from-source
ENV NODE_ENV=production \
NEXT_TELEMETRY_DISABLED=1 \
@@ -470,6 +522,7 @@ ENV NODE_ENV=production \
WHISPER_BIN=/usr/local/bin/whisper-cli \
WHISPER_MODEL=/data/models/ggml-base.en.bin \
YTDLP_BIN=/usr/local/bin/yt-dlp \
+ ARCHILYZER_IMAGE_YTDLP=/usr/local/bin/yt-dlp \
EXPORT_INDEX_DIR=/data/builds/.export-index \
EXPORT_BUILDS_DIR=/data/builds/.export-builds \
ARCHILYZER_SITE_OUT=/data/builds/site
@@ -477,3 +530,12 @@ RUN mkdir -p /data/transcripts /data/config /data/models /data/builds
ENTRYPOINT ["/repo/docker/entrypoint.sh"]
CMD ["editor"]
+
+# Which commit and branch this image was built from — the publish stamps'
+# `commit`/`branch` where there is no .git (.dockerignore keeps it out). Last,
+# so a new commit re-runs one ENV layer and nothing else. Passed by compose from
+# the shell: ARCHILYZER_COMMIT=$(git rev-parse HEAD) docker compose build.
+ARG ARCHILYZER_COMMIT=
+ARG ARCHILYZER_BRANCH=
+ENV ARCHILYZER_COMMIT=${ARCHILYZER_COMMIT} \
+ ARCHILYZER_BRANCH=${ARCHILYZER_BRANCH}
diff --git a/ENVIRONMENT.md b/ENVIRONMENT.md
@@ -40,7 +40,7 @@ The one override surface for where things live and which binary runs. Every one
| `ARIA2C_BIN` | `aria2c` on PATH | Fetches an archive.org file over BitTorrent (the item's torrent, archive.org as web seed), then seeds it for a while. Optional: without it archive.org files are downloaded directly. See `archiveOrg` in [SETTINGS.md](SETTINGS.md). | common/lib/paths.ts (getPaths) |
| `OLLAMA_URL` | `http://127.0.0.1:11434` | The local ollama server, the local digest and attribution engine. | common/lib/paths.ts (getPaths) |
| `CLAUDE_BIN` | `claude` on PATH | The `claude` CLI, driving the opt-in metered digest lane. | common/lib/paths.ts (getPaths) |
-| `ARCHILYZER_CONFIG_DIR` | `~/.config/archilyzer` | The operator's private config dir, outside the repo: the two inputs of `archilyzer source publish` below. Never committed. | common/lib/paths.ts (getPaths) |
+| `ARCHILYZER_CONFIG_DIR` | `~/.config/archilyzer` | The operator's private config dir, outside the repo: the two inputs of `archilyzer source publish` below. Never committed. In Docker it is `/data/config/archilyzer`, in the config volume (docker-compose.yml). | common/lib/paths.ts (getPaths) |
| `SOURCE_SCRUB_FILE` | `<ARCHILYZER_CONFIG_DIR>/source-scrub.txt` | git-filter-repo `lhs==>rhs` rules applied to file contents AND commit messages when the source mirror is generated (`<home dir>==>/home/user` is built in and runs first). Every rule's left side is also denied. See [PUBLISH.md](PUBLISH.md). | common/lib/paths.ts (getPaths) |
| `SOURCE_DENYLIST_FILE` | `<ARCHILYZER_CONFIG_DIR>/source-denylist.txt` | Literals the published source must never contain, one per line (`i:` = any case). One hit anywhere in the mirror, the tree or the tarball refuses the publish. | common/lib/paths.ts (getPaths) |
| `ARCHILYZER_SOURCE_SCRATCH` | the OS temp dir | Where `source publish` makes its scratch clone and stage (removed afterwards unless `--keep-scratch`). | common/lib/paths.ts (getPaths) |
@@ -57,9 +57,15 @@ Tokens, credentials and knobs a running process reads. Most configuration is not
| `SYNC_HEARTBEAT_SECONDS` | `settings.syncScheduler.heartbeatSeconds` | Overrides the editor's in-process sync heartbeat. `0` = no internal timer (tick from cron instead). | editor/app/scheduler/heartbeat.ts |
| `SYNC_TICK_URL` | `http://127.0.0.1:3001/api/scheduler/tick` | Where `archilyzer sync tick` (cron's heartbeat) posts. | common/bin/sync-tick.ts |
| `SYNC_TICK_TOKEN` | unset (no auth) | Bearer token for the tick endpoint; set on both the editor and the cron job. | common/bin/sync-tick.ts, editor/app/scheduler/auth.ts |
-| `R2_ACCESS_KEY_ID` | — | R2 S3 credentials for uploading oversize archives at deploy time (with `R2_SECRET_ACCESS_KEY` and `CLOUDFLARE_ACCOUNT_ID`). See [PUBLISH.md](PUBLISH.md). | common/publish/build.ts |
-| `R2_SECRET_ACCESS_KEY` | — | See `R2_ACCESS_KEY_ID`. | common/publish/build.ts |
-| `CLOUDFLARE_ACCOUNT_ID` | — | The account the R2 endpoint belongs to. wrangler reads its own credentials. | common/publish/build.ts |
+| `R2_ACCESS_KEY_ID` | — | R2 S3 credentials for uploading oversize archives at deploy time (with `R2_SECRET_ACCESS_KEY` and `CLOUDFLARE_ACCOUNT_ID`), needed only when `archiveStorage.bucket` is set. In Docker they come from `.env`. See [PUBLISH.md](PUBLISH.md). | common/publish/build.ts, common/bin/doctor.ts (set or not) |
+| `R2_SECRET_ACCESS_KEY` | — | See `R2_ACCESS_KEY_ID`. | common/publish/build.ts, common/bin/doctor.ts (set or not) |
+| `CLOUDFLARE_ACCOUNT_ID` | — | The Cloudflare account: the R2 endpoint's, and the one wrangler deploys to when the token can see more than one. In Docker it comes from `.env`. | common/publish/build.ts, wrangler, common/bin/doctor.ts (set or not) |
+| `CLOUDFLARE_API_TOKEN` | unset (wrangler's own `wrangler login` config, on a host) | The API token every deploy's wrangler authenticates with (Cloudflare Pages: Edit). The way a container deploys — there is no browser for `wrangler login` in one; set it in `.env`. | wrangler (every deploy), common/bin/doctor.ts (set or not, never the value) |
+| `ARCHILYZER_SOURCE_REPO` | this checkout's git common dir | The git DIR `archilyzer source publish` mirrors `main` from, when the checkout has none: in Docker, `/data/source.git`, the host's git common dir mounted read-only by docker-compose.source.yml. A value that names nothing refuses the publish. | common/publish/source.ts, common/bin/doctor.ts, docker/entrypoint.sh |
+| `YTDLP_SOURCE_HOST_DIR` | — (required by the overlay) | Docker: the HOST path of a yt-dlp source checkout (the directory holding `yt_dlp/`), mounted read-only at `/opt/yt-dlp-src` by docker-compose.ytdlp.yml. See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md), "Substituting yt-dlp". | docker-compose.ytdlp.yml |
+| `YTDLP_AUTO_UPDATE` | off | Docker: `1` runs `yt-dlp -U` on every editor boot — on the image's yt-dlp only; an override (`YTDLP_BIN` naming another) is left alone, with a warning. | docker/entrypoint.sh, common/bin/doctor.ts |
+| `XDG_CONFIG_HOME` | `~/.config` | Where `wrangler login` keeps its config (`<it>/.wrangler/config/default.toml`); the doctor looks for it there, by path, never reading it. | common/bin/doctor.ts |
+| `YTDLP_SOURCE_DIR` | `/opt/yt-dlp-src` | Docker: where `/usr/local/bin/yt-dlp-from-source` finds the yt-dlp source tree it runs with the image's python. | docker/yt-dlp-from-source.sh |
| `DOCKER_BIN` | `docker` | The container engine for docker-mode builds (e.g. `podman`). | common/publish/build.ts |
| `DOCKER_BUILD_MEMORY` | no cap | Per-container memory cap for a docker-mode build (`--memory`). | common/publish/build.ts |
| `DOCKER_BUILD_CPUS` | no cap | Per-container CPU cap for a docker-mode build (`--cpus`). | common/publish/build.ts |
@@ -141,7 +147,7 @@ The publish pipeline sets these for a process it spawns. Listed so a reader know
## Docker
-The container's own set, read by `docker/*.sh`, the compose files and Caddy — not by the apps' code (except `ARCHILYZER_IDLE_BOOT`). See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md).
+The container's own set, read by `docker/*.sh`, the compose files and Caddy — not by the apps' code (except `ARCHILYZER_IDLE_BOOT`, and the image's facts `archilyzer doctor` and the publish stamps read). See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md).
| Variable | Default | What it does | Read by |
|---|---|---|---|
@@ -150,6 +156,11 @@ The container's own set, read by `docker/*.sh`, the compose files and Caddy —
| `ARCHILYZER_MODELS_DIR` | `/data/models` | Where models live in the container. | docker/entrypoint.sh |
| `ARCHILYZER_BUILDS_DIR` | `/data/builds` | Where the container keeps built sites. | docker/entrypoint.sh |
| `ARCHILYZER_SITE_OUT` | `/data/builds/site` | The built export site the `site` service serves. | docker/entrypoint.sh, docker/publish-site.sh |
+| `ARCHILYZER_HOMEPAGE_OUT` | `/data/builds/homepage` | The locally deployed homepage (`publish homepage --deploy --to local`). The `homepage` service serves it when it is non-empty, else the image's baked build. | docker/entrypoint.sh |
+| `ARCHILYZER_IMAGE_YTDLP` | baked: `/usr/local/bin/yt-dlp` | The yt-dlp the image ships. `YTDLP_BIN` naming anything else is an OVERRIDE: the boot's `yt-dlp:` line and `archilyzer doctor` say so, and `YTDLP_AUTO_UPDATE` leaves it alone. | docker/entrypoint.sh, common/bin/doctor.ts |
+| `ARCHILYZER_COMMIT` | baked: empty unless the build passed it | The commit the image was built from — the publish stamps' `commit` where there is no .git. `ARCHILYZER_COMMIT=$(git rev-parse HEAD) docker compose build`. | the Dockerfile (a build arg), the publish stamps (`IMAGE_COMMIT_ENV`) |
+| `ARCHILYZER_BRANCH` | baked: empty unless the build passed it | The branch the image was built from — the stamps' `branch`, which a production deploy checks. | the Dockerfile (a build arg), the publish stamps (`IMAGE_BRANCH_ENV`) |
+| `ARCHILYZER_SOURCE_HOST_DIR` | `./.git` | The HOST's git common dir docker-compose.source.yml mounts at `/data/source.git`. In a git worktree, the primary checkout's `.git`. | docker-compose.source.yml |
| `ARCHILYZER_IDLE_BOOT` | off | `1` boots the editor without arming the heartbeat or any auto-queue runner. | common/lib/idleBoot.ts (the editor) |
| `ARCHILYZER_AUTH_MODE` | `basic` | `basic`, `forward` or `none` — the only escape hatch from the exposure guard. | docker/guard-exposure.sh, docker/caddy-start.sh |
| `ARCHILYZER_AUTH_USER` | `archilyzer` | Basic-auth user. | docker/Caddyfile |
diff --git a/RUNNING_IN_DOCKER.md b/RUNNING_IN_DOCKER.md
@@ -5,10 +5,12 @@ One command stands up a working archive server: the editor, the tools it drives
Vulkan image), and a reverse proxy that is the only thing on the box with an open
port.
-> This document is about **running the apps**. Fanning per-site *export builds* out
-> across containers is a different thing entirely — it uses `Dockerfile.build`, and
-> it is in [PUBLISH.md](PUBLISH.md#building-every-site-in-containers). Neither
-> affects the other.
+> This document is about **running the apps** — publishing included: every publish
+> stage (index, build, deploy) runs inside the editor's container. Fanning per-site
+> *export builds* out across containers is a different thing — the opt-in docker
+> runner of a HOST install, from `Dockerfile.build`, in
+> [PUBLISH.md](PUBLISH.md#building-every-site-in-containers). Neither affects the
+> other.
---
@@ -194,7 +196,10 @@ it accurately.
### Publish the archive
The site server serves a *static build of your corpus*, which does not exist
-until you make one:
+until you make one. Publishing is a set of **stages** — update the index, build a
+site, deploy it — and every one of them runs **inside the editor's container**:
+the queued jobs the editor starts (Sites → Publish, the publish lane) and the
+commands below alike. The quick way, for the local `site` service:
```sh
docker compose exec editor /repo/docker/publish-site.sh <site-id>
@@ -202,9 +207,94 @@ docker compose --profile site up -d
```
Create the site itself in the editor first (**Sites → New**); run the script with
-no argument to list the ones you have. It runs the same `pnpm --filter export run
-build` a host install runs, then copies the output into the volume the `site`
-service serves. No restart needed afterwards.
+no argument to list the ones you have. The script is three stages in a row, and
+you can run them yourself:
+
+```sh
+docker compose exec editor pnpm archilyzer publish index
+docker compose exec editor pnpm archilyzer publish build <site-id>
+docker compose exec editor pnpm archilyzer publish deploy <site-id> --to local
+docker compose exec editor pnpm archilyzer publish status
+```
+
+A stage that is already fresh does nothing, so running one twice is cheap. A
+local deploy copies the built bundle into the volume the `site` service serves;
+no restart needed afterwards. A **private** site (`audience: "private"`) is
+built but never deployed, locally or anywhere else.
+
+**`exec`, never `run --rm`.** `docker compose run --rm editor …` starts a SECOND
+container with its own copy of the image's `export/public` and a second writer on
+the index, and the publish lock (which keeps a stage you start from colliding
+with one the editor is running) cannot see across containers. `exec` runs in the
+editor's own container, beside its jobs, under the same lock. `pnpm ops publish`
+from the host goes through the editor too.
+
+#### Deploying to Cloudflare from the container
+
+The same stages deploy to Cloudflare Pages. The container has no browser for
+`wrangler login`, so it authenticates with a token from `.env`, which the editor
+reads and every stage it runs inherits:
+
+```sh
+# .env
+CLOUDFLARE_API_TOKEN=... # an API token with "Cloudflare Pages: Edit"
+CLOUDFLARE_ACCOUNT_ID=...
+# only when Settings → archive storage names an R2 bucket for oversize archives:
+R2_ACCESS_KEY_ID=...
+R2_SECRET_ACCESS_KEY=...
+```
+
+```sh
+docker compose up -d # picks up .env
+docker compose exec editor pnpm archilyzer publish deploy <site-id> --preview <branch>
+docker compose exec editor pnpm archilyzer publish deploy <site-id> # production
+```
+
+Nothing is fetched at deploy time: wrangler is pinned in the workspace the image
+ships. With no token a deploy is refused before wrangler runs, with a sentence
+saying so; `archilyzer doctor` reports whether each credential is **set** (never
+its value). Every deploy is checked live afterwards. PUBLISH.md has the whole
+publish flow; this is only what is different in a container.
+
+#### The homepage, and its /source mirror
+
+The image carries a build of the project's own homepage, served by the
+`homepage` service. A real one — the corpus's numbers, the `/source` mirror of
+this repo — is a publish like any other:
+
+```sh
+docker compose exec editor pnpm archilyzer publish homepage --deploy --to local
+docker compose --profile homepage restart homepage # once, after the first
+```
+
+The `homepage` service serves the local deploy (`/data/builds/homepage`) once
+there is one, else the baked build — chosen at boot, hence the one restart.
+
+The `/source` mirror needs two things the image deliberately lacks:
+
+- **The repository.** The image has no `.git`. `docker-compose.source.yml` mounts
+ the host's git directory read-only at `/data/source.git` and points
+ `ARCHILYZER_SOURCE_REPO` at it (default `./.git`; from a git worktree set
+ `ARCHILYZER_SOURCE_HOST_DIR` to the primary checkout's `.git`). Without it the
+ homepage builds with an empty `/source` page.
+- **The operator's scrub rules and denylist.** They live in the config volume —
+ `ARCHILYZER_CONFIG_DIR` is `/data/config/archilyzer` in the container, made
+ (mode 700, empty) on the first boot — and are never printed by anything:
+
+```sh
+docker compose -f docker-compose.yml -f docker-compose.source.yml up -d
+docker compose cp ~/.config/archilyzer/source-scrub.txt editor:/data/config/archilyzer/
+docker compose cp ~/.config/archilyzer/source-denylist.txt editor:/data/config/archilyzer/
+docker compose exec editor chmod 600 /data/config/archilyzer/source-scrub.txt /data/config/archilyzer/source-denylist.txt
+```
+
+`git-filter-repo` (pinned) is in the image. **gitleaks and stagit are not**, so a
+source publish from the container is weaker than a host's in two ways it tells you
+about: it skips the secret scan, with a `WARNING` in its log (the literal audit
+against your denylist still runs, and still refuses), and it publishes the source
+without its history pages (`/source/git/`). `archilyzer doctor` lists both as
+absent. Publish the mirror from a host checkout that has them if you want either;
+a pinned gitleaks in the image is a planned follow-up.
### Model choice
@@ -224,12 +314,57 @@ For the Vulkan image it names a parakeet GGUF instead — see
An image pins yt-dlp at build time, and a stale yt-dlp is the most common reason
downloads suddenly start failing. Set `YTDLP_AUTO_UPDATE=1` in `.env` and each
-boot self-updates it. Or, once:
+boot self-updates it. Exactly `1`, `true`, `yes` or `on` turns it on, as written
+— `TRUE` or `Yes` does not (the entrypoint and `archilyzer doctor` read it the
+same way). Or, once:
```sh
docker compose exec editor yt-dlp -U
```
+Every editor boot logs which yt-dlp it will run, and whether it runs:
+
+```
+[entrypoint] yt-dlp: /usr/local/bin/yt-dlp 2026.09.30 (image)
+```
+
+### Substituting yt-dlp
+
+To run your own yt-dlp — a patched build, say — set `YTDLP_BIN`. It is a swap at
+run time: nothing is rebuilt, and the image's release binary stays where it is
+(`ARCHILYZER_IMAGE_YTDLP`, `/usr/local/bin/yt-dlp`). Two ways:
+
+**A source checkout**, run with the image's python (yt-dlp's optional modules
+are installed beside it):
+
+```sh
+# .env
+YTDLP_SOURCE_HOST_DIR=/home/you/yt-dlp-patched # the directory holding yt_dlp/
+```
+
+```sh
+docker compose -f docker-compose.yml -f docker-compose.ytdlp.yml up -d
+```
+
+The overlay mounts the checkout read-only at `/opt/yt-dlp-src` and sets
+`YTDLP_BIN=/usr/local/bin/yt-dlp-from-source`, a wrapper baked into the image.
+
+**A zipapp** built on the host (`make yt-dlp` in the checkout), copied into the
+config volume:
+
+```sh
+docker compose exec editor mkdir -p /data/config/bin
+docker compose cp ./yt-dlp editor:/data/config/bin/yt-dlp
+docker compose exec editor chmod 755 /data/config/bin/yt-dlp
+# .env
+YTDLP_BIN=/data/config/bin/yt-dlp
+```
+
+Either way the boot line says `(override)`, and `YTDLP_AUTO_UPDATE` leaves an
+override alone with a warning — update it where you build it. A substitute that
+does not run is reported as `yt-dlp: MISSING — … does not run: …` (the editor
+still starts), and by `archilyzer doctor`.
+
### Driving the editor without a browser
Every editor gesture is a server action, which is fine for a person and hostile
@@ -511,6 +646,8 @@ The `site` profile does not need the mount: an export build never reads `data/`.
docker compose logs -f editor
docker compose exec editor bash # a shell in the image
docker compose exec editor yt-dlp --version
+docker compose exec editor pnpm archilyzer doctor # what this container can do
+docker compose exec editor pnpm archilyzer publish status
docker compose restart editor
docker compose down # stop; volumes survive
docker compose down -v # stop AND DELETE the corpus
@@ -523,7 +660,10 @@ docker compose down -v # stop AND DELETE the corp
**Baked in:** node + the installed workspace, the built editor / umtool / homepage,
`yt-dlp`, a JavaScript runtime for it (`deno` — without one yt-dlp warns and
silently loses formats), `ffmpeg`/`ffprobe`, `whisper-cli` (statically linked),
-`zip`/`tar`/`xz`/`gzip`, `rsync`, `git`, `curl`, `ps`. The Vulkan image adds
+`zip`/`tar`/`xz`/`gzip`, `rsync`, `git`, `curl`, `ps`; `python3` with yt-dlp's
+optional modules (for a substituted yt-dlp), `pipx` and a pinned `git-filter-repo`
+(the homepage's source mirror), and — in the workspace's `node_modules` — a
+pinned `wrangler`, so a deploy fetches nothing. The Vulkan image adds
`parakeet-cli`, the Mesa Vulkan drivers and `vulkaninfo`.
**Not baked, on purpose:**
@@ -532,7 +672,14 @@ silently loses formats), `ffmpeg`/`ffprobe`, `whisper-cli` (statically linked),
in the `corpus` volume.
- **whisper models.** 142 MB to 3 GB, and the choice is yours. Fetched on boot.
- **The export site build.** It is a static render *of a corpus*, and there is no
- corpus at image-build time. `docker/publish-site.sh` makes it at run time.
+ corpus at image-build time. The publish stages make it at run time.
+- **Credentials.** `CLOUDFLARE_API_TOKEN` and friends come from `.env` at run
+ time, never from the image.
+- **The repository.** No `.git`; `docker-compose.source.yml` mounts the host's
+ for the `/source` mirror. The build's commit and branch are baked as
+ `ARCHILYZER_COMMIT`/`ARCHILYZER_BRANCH` when the build passes them
+ (`ARCHILYZER_COMMIT=$(git rev-parse HEAD) ARCHILYZER_BRANCH=$(git branch
+ --show-current) docker compose build`) — the publish stamps record them.
- **ImageMagick with Pango, and `qrencode`.** Only `umtool/report-to-video/`
needs them, and rendering a report to video is a workstation task, not
something a server does. Run that part on a host checkout.
@@ -542,20 +689,21 @@ silently loses formats), `ffmpeg`/`ffprobe`, `whisper-cli` (statically linked),
Point `OLLAMA_URL` at a host ollama (`http://host.docker.internal:11434`) if you
want digests.
-### The multi-site build pipeline falls back inside a container
+### Two build runners, and the container has one
-The editor's **Build all sites** fans per-site export builds out across containers
-whenever a container engine answers (see [PUBLISH.md](PUBLISH.md#building-every-site-in-containers)). Inside a container there
-is no `docker` binary, so that path is unavailable. It already handles this — the
-build logs
+Every site build goes through one stage contract with two runners:
-```
-[notice] No container engine available — building sites serially on the host.
-```
+- **local** — the default everywhere, in a container or not: each stage is a child
+ process of the editor (or of the CLI you ran), one at a time.
+- **docker** — a HOST install's opt-in fan-out (`settings.publish.runner:
+ "docker"`, or `publish build all --runner docker`): one `Dockerfile.build`
+ container per site (see [PUBLISH.md](PUBLISH.md#building-every-site-in-containers)).
+ Both write the same bundles and stamps, so a deploy does not care which built it.
-and does the builds one after another in-process. Correct, just not parallel. **Do
-not** try to make docker-in-docker work for this; run the parallel pipeline from a
-host checkout if you need it.
+Inside the container there is no engine, and the docker runner **refuses** ("the
+docker runner needs an engine on this host") rather than pretend. The editor never
+gets the docker socket — **do not** mount it, and do not try to make
+docker-in-docker work; run the fan-out from a host checkout if you want it.
### It runs as root
@@ -582,6 +730,23 @@ Open http://localhost:8081. Keep the clone on the Windows filesystem — the bui
context is small, and the corpus lives in a Docker volume inside the WSL2 VM,
which is where the I/O actually happens.
+Publishing needs nothing else on Windows — Docker Desktop is the whole
+requirement. A checklist, in PowerShell from the clone:
+
+1. `copy .env.example .env`, then set in `.env`: `WORKER_TOKEN` (for `pnpm ops`
+ and agents), `CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_ACCOUNT_ID` (to deploy).
+2. `docker compose up -d --build`
+3. Open http://localhost:8081 → **Sites** → **Publish now**, or:
+ `docker compose exec editor pnpm archilyzer publish now`
+4. A local preview: `docker compose --profile site up -d site`, then
+ `docker compose exec editor pnpm archilyzer publish deploy <site-id> --to local`,
+ and open http://localhost:8080.
+5. `docker compose exec editor pnpm archilyzer doctor` — every check, including
+ whether the Cloudflare token is set (never its value).
+6. Optional, your own yt-dlp: set `YTDLP_SOURCE_HOST_DIR` (a Windows path such as
+ `C:\src\yt-dlp` works) and run
+ `docker compose -f docker-compose.yml -f docker-compose.ytdlp.yml up -d`.
+
You still want WSL2 directly if you intend to *develop* the project or run Claude
Code against it — see [README.md](README.md#claude-code-on-windows). For running
an archive, this is enough.
@@ -609,7 +774,15 @@ comparison above. The usual cause is a missing `/dev/dri` — a container withou
the render node still transcribes, just slowly.
**Downloads started failing on every channel.** Almost always a stale yt-dlp:
-`docker compose exec editor yt-dlp -U`.
+`docker compose exec editor yt-dlp -U`. Check `docker compose logs editor | grep
+yt-dlp:` first — `(override)` means `YTDLP_BIN` names your own build, which `-U`
+does not touch, and `MISSING` means it does not run at all (a from-source
+override with no checkout mounted, say).
+
+**A deploy is refused before wrangler runs.** `CLOUDFLARE_API_TOKEN` is not set in
+`.env`, or the editor was started before it was: `docker compose up -d` again
+(a restart does not re-read `.env`; `up` recreates the container).
+`archilyzer doctor`'s `cloudflare-auth` line says which.
**Port 8080 is already taken.** `SITE_HTTP_PORT=9080` in `.env`. Those variables
move the *outside* port; nothing inside the containers changes.
diff --git a/common/bin/doctor.test.ts b/common/bin/doctor.test.ts
@@ -67,6 +67,8 @@ function checkout(): { root: string; bin: string; paths: Paths } {
parakeetBin: path.join(root, "scripts", "parakeet-stitch.mjs"),
parakeetModel: "",
parakeetCliBin: "parakeet-cli",
+ configDir: path.join(root, ".config"),
+ exportBuildsDir: path.join(root, "export", ".export-builds"),
sourceScrubFile: path.join(root, ".config", "source-scrub.txt"),
sourceDenylistFile: path.join(root, ".config", "source-denylist.txt"),
// Outside the checkout, as the XDG cache is: the tests that compare the
@@ -628,3 +630,182 @@ test("download pacing: a held platform and a raised pace warn, and nothing is wr
assert.match(dp[2].detail, /^4s between requests \(base 1s\)/);
assert.deepEqual(tree(c.root), before);
});
+
+// ── release 18: the downloader and publish checks ─────────────────────────
+
+const find = (r: DoctorReport, section: string, id: string) =>
+ r.checks.find((x) => x.section === section && x.id === id);
+
+// A HOME of the scenario's own, so wrangler's login config is never this
+// machine's.
+function home(c: ReturnType<typeof checkout>): string {
+ const h = path.join(TMP, `${path.basename(c.root)}-home`);
+ mkdirSync(h, { recursive: true });
+ return h;
+}
+
+test("downloader: a host's yt-dlp is a note; the image's is ok; an override says so, and warns beside YTDLP_AUTO_UPDATE or when it does not run", async () => {
+ const c = checkout();
+ fake(c.bin, "yt-dlp", "2026.09.30");
+ fake(c.bin, "yt-dlp-patched", "2026.10.01.patched");
+ // The from-source wrapper with no checkout mounted: a sentence, exit 127.
+ const broken = path.join(c.bin, "yt-dlp-from-source");
+ writeFileSync(broken, "#!/bin/sh\necho 'no yt_dlp package' >&2\nexit 127\n");
+ chmodSync(broken, 0o755);
+ const before = tree(c.root);
+ // A host install: no ARCHILYZER_IMAGE_YTDLP to compare with.
+ let r = await run(c);
+ assert.equal(find(r, "downloader", "yt-dlp")?.status, "info");
+ assert.match(find(r, "downloader", "yt-dlp")!.detail, /yt-dlp 2026\.09\.30 \(host: not in the runtime image\)$/);
+ // The runtime image, running its own.
+ const image = path.join(c.bin, "yt-dlp");
+ r = await run(c, { ARCHILYZER_IMAGE_YTDLP: image });
+ assert.equal(find(r, "downloader", "yt-dlp")?.status, "ok");
+ assert.match(find(r, "downloader", "yt-dlp")!.detail, /2026\.09\.30 \(image\)$/);
+ // An override (the paths' binary is another file).
+ (c.paths as { ytdlpBin: string }).ytdlpBin = path.join(c.bin, "yt-dlp-patched");
+ r = await run(c, { ARCHILYZER_IMAGE_YTDLP: image });
+ assert.equal(find(r, "downloader", "yt-dlp")?.status, "ok");
+ assert.match(find(r, "downloader", "yt-dlp")!.detail, new RegExp(`2026\\.10\\.01\\.patched \\(override; the image's is ${image.replace(/[.]/g, "\\.")}\\)$`));
+ r = await run(c, { ARCHILYZER_IMAGE_YTDLP: image, YTDLP_AUTO_UPDATE: "1" });
+ assert.equal(find(r, "downloader", "yt-dlp")?.status, "warn");
+ assert.match(find(r, "downloader", "yt-dlp")!.detail, /YTDLP_AUTO_UPDATE is set, and the entrypoint never self-updates an override/);
+ assert.equal(r.ok, true, "a warning, never a failure");
+ // The entrypoint's exact-match rule: `TRUE` (or ` 1`) is off there, so here.
+ for (const off of ["TRUE", " 1", "0", ""]) {
+ r = await run(c, { ARCHILYZER_IMAGE_YTDLP: image, YTDLP_AUTO_UPDATE: off });
+ assert.equal(find(r, "downloader", "yt-dlp")?.status, "ok", JSON.stringify(off));
+ }
+ // An override that does not run.
+ (c.paths as { ytdlpBin: string }).ytdlpBin = broken;
+ r = await run(c, { ARCHILYZER_IMAGE_YTDLP: image });
+ assert.equal(find(r, "downloader", "yt-dlp")?.status, "warn");
+ assert.match(find(r, "downloader", "yt-dlp")!.detail, /\(override; .*\) — does not run .*docker-compose\.ytdlp\.yml/);
+ assert.deepEqual(tree(c.root), before);
+});
+
+test("publish: cloudflare-auth reports a token SET (never its value), a wrangler login by path, and warns only when a site names a project", async () => {
+ const c = checkout();
+ const sitesDir = path.join(c.paths.transcriptsDir, "sites");
+ (c.paths as { sitesDir: string }).sitesDir = sitesDir;
+ const h = home(c);
+ // Nothing configured to deploy, no credential: a note.
+ let r = await run(c, { HOME: h });
+ assert.equal(find(r, "publish", "cloudflare-auth")?.status, "info");
+ // A site that deploys, no credential: a warning naming the site.
+ mkdirSync(path.join(sitesDir, "alpha"), { recursive: true });
+ writeFileSync(path.join(sitesDir, "alpha", "site.json"), JSON.stringify({ title: "A", cloudflareProject: "alpha-pages" }));
+ mkdirSync(path.join(sitesDir, "beta"), { recursive: true });
+ writeFileSync(path.join(sitesDir, "beta", "site.json"), JSON.stringify({ title: "B" }));
+ const before = tree(c.root);
+ r = await run(c, { HOME: h });
+ assert.equal(find(r, "publish", "cloudflare-auth")?.status, "warn");
+ assert.match(find(r, "publish", "cloudflare-auth")!.detail, /1 site names a Cloudflare project \(alpha\) — every deploy refuses; set CLOUDFLARE_API_TOKEN/);
+ assert.equal(r.ok, true);
+ // A token: ok, and the value appears nowhere in the report.
+ const secret = "PLANTED-TOKEN-0123456789";
+ r = await run(c, { HOME: h, CLOUDFLARE_API_TOKEN: secret, CLOUDFLARE_ACCOUNT_ID: "PLANTED-ACCOUNT" });
+ assert.equal(find(r, "publish", "cloudflare-auth")?.status, "ok");
+ assert.match(find(r, "publish", "cloudflare-auth")!.detail, /^CLOUDFLARE_API_TOKEN is set \(never printed\); CLOUDFLARE_ACCOUNT_ID set$/);
+ assert.ok(!/PLANTED/.test(renderDoctorReport(r)) && !/PLANTED/.test(JSON.stringify(r)));
+ // A host's `wrangler login`: ok, by path.
+ const cfg = path.join(h, ".config", ".wrangler", "config");
+ mkdirSync(cfg, { recursive: true });
+ writeFileSync(path.join(cfg, "default.toml"), 'oauth_token = "PLANTED-OAUTH"\n');
+ r = await run(c, { HOME: h });
+ assert.equal(find(r, "publish", "cloudflare-auth")?.status, "ok");
+ assert.match(find(r, "publish", "cloudflare-auth")!.detail, /wrangler's login config is at .*default\.toml/);
+ assert.ok(!/PLANTED/.test(renderDoctorReport(r)), "the login config is located, never read");
+ assert.deepEqual(tree(c.root), before);
+});
+
+test("publish: r2-keys appears only with a bucket, and names what is missing — never a value", async () => {
+ const c = checkout();
+ const h = home(c);
+ let r = await run(c, { HOME: h });
+ assert.equal(find(r, "publish", "r2-keys"), undefined, "no bucket, no check");
+ writeFileSync(c.paths.settingsFile, JSON.stringify({ archiveStorage: { bucket: "overflow", publicBaseUrl: "https://r2.example" } }));
+ const before = tree(c.root);
+ r = await run(c, { HOME: h, R2_ACCESS_KEY_ID: "PLANTED-KEY" });
+ assert.equal(find(r, "publish", "r2-keys")?.status, "warn");
+ assert.match(find(r, "publish", "r2-keys")!.detail, /"overflow" is set but R2_SECRET_ACCESS_KEY, CLOUDFLARE_ACCOUNT_ID are not/);
+ r = await run(c, { HOME: h, R2_ACCESS_KEY_ID: "PLANTED-KEY", R2_SECRET_ACCESS_KEY: "PLANTED-SECRET", CLOUDFLARE_ACCOUNT_ID: "PLANTED-ACCOUNT" });
+ assert.equal(find(r, "publish", "r2-keys")?.status, "ok");
+ assert.ok(!/PLANTED/.test(renderDoctorReport(r)));
+ assert.deepEqual(tree(c.root), before);
+});
+
+test("publish: export-builds — not there yet is a note; bundles with under 1.5x their size free warn; not writable warns", async () => {
+ const c = checkout();
+ const h = home(c);
+ let r = await run(c, { HOME: h }, { freeBytes: () => 5e9 });
+ assert.equal(find(r, "publish", "export-builds")?.status, "info");
+ assert.match(find(r, "publish", "export-builds")!.detail, /does not exist yet — the first site build makes it \(5\.00 GB free\)/);
+ const builds = c.paths.exportBuildsDir;
+ for (const [id, n] of [["alpha", 3_000_000], ["_hub", 1_000_000]] as const) {
+ mkdirSync(path.join(builds, id, "out", "sub"), { recursive: true });
+ writeFileSync(path.join(builds, id, "out", "sub", "f.bin"), Buffer.alloc(n));
+ }
+ // Staging beside a bundle is not a bundle.
+ mkdirSync(path.join(builds, "alpha", ".r2-staging"), { recursive: true });
+ writeFileSync(path.join(builds, "alpha", ".r2-staging", "big.zip"), Buffer.alloc(9_000_000));
+ const before = tree(c.root);
+ r = await run(c, { HOME: h }, { freeBytes: () => 5e9 });
+ assert.equal(find(r, "publish", "export-builds")?.status, "ok");
+ assert.match(find(r, "publish", "export-builds")!.detail, /: 2 bundles, 4 MB; 5\.00 GB free$/);
+ r = await run(c, { HOME: h }, { freeBytes: () => 5_000_000 });
+ assert.equal(find(r, "publish", "export-builds")?.status, "warn");
+ assert.match(find(r, "publish", "export-builds")!.detail, /under 1\.5× the bundles \(6 MB\)/);
+ chmodSync(builds, 0o555);
+ try {
+ if (process.getuid?.() !== 0) {
+ r = await run(c, { HOME: h }, { freeBytes: () => 5e9 });
+ assert.equal(find(r, "publish", "export-builds")?.status, "warn");
+ assert.match(find(r, "publish", "export-builds")!.detail, /is not writable/);
+ }
+ } finally {
+ chmodSync(builds, 0o755);
+ }
+ assert.equal(r.ok, true);
+ assert.deepEqual(tree(c.root), before);
+});
+
+test("source publish: source-repo — the variable naming nothing fails, a readable main is ok, none is a note (a warning once the operator's files exist); config-dir is counted", async () => {
+ const c = checkout();
+ const h = home(c);
+ // No repository, nothing meant: notes.
+ let r = await run(c, { HOME: h });
+ assert.equal(find(r, "source publish", "source-repo")?.status, "info");
+ assert.equal(find(r, "source publish", "config-dir")?.status, "info");
+ assert.match(find(r, "source publish", "config-dir")!.detail, /does not exist — ARCHILYZER_CONFIG_DIR moves it/);
+ // The operator's files exist: no repository is now a warning.
+ mkdirSync(path.dirname(c.paths.sourceScrubFile), { recursive: true });
+ writeFileSync(c.paths.sourceScrubFile, "PLANTED==>x\n");
+ writeFileSync(c.paths.sourceDenylistFile, "PLANTED\n");
+ const before = tree(c.root);
+ r = await run(c, { HOME: h });
+ assert.equal(find(r, "source publish", "source-repo")?.status, "warn");
+ assert.match(find(r, "source publish", "source-repo")!.detail, /docker-compose\.source\.yml/);
+ assert.equal(find(r, "source publish", "config-dir")?.status, "ok");
+ assert.match(find(r, "source publish", "config-dir")!.detail, /\(2 entries, writable\)$/);
+ assert.ok(!/PLANTED/.test(renderDoctorReport(r)));
+ // The variable naming nothing: the publish refuses, so the doctor fails.
+ r = await run(c, { HOME: h, ARCHILYZER_SOURCE_REPO: path.join(TMP, "not-mounted.git") });
+ assert.equal(find(r, "source publish", "source-repo")?.status, "fail");
+ assert.equal(r.ok, false);
+ // A real repository's git dir, through the default probe (git on PATH).
+ const repo = path.join(TMP, `${path.basename(c.root)}-repo`);
+ mkdirSync(repo);
+ const git = (...a: string[]) => execFileSync("git", a, { cwd: repo, stdio: "pipe", env: { ...process.env, GIT_CONFIG_NOSYSTEM: "1", HOME: h } });
+ git("init", "-q", "-b", "main");
+ git("-c", "user.name=t", "-c", "user.email=t@example.invalid", "-c", "commit.gpgsign=false", "commit", "-q", "--allow-empty", "-m", "one");
+ const head = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repo }).toString().trim();
+ r = await run(c, { HOME: h, PATH: `${c.bin}${path.delimiter}${process.env.PATH}`, ARCHILYZER_SOURCE_REPO: path.join(repo, ".git") });
+ assert.equal(find(r, "source publish", "source-repo")?.status, "ok");
+ assert.equal(find(r, "source publish", "source-repo")!.detail, `${path.join(repo, ".git")} (ARCHILYZER_SOURCE_REPO): main ${head.slice(0, 12)}`);
+ // An injected probe: main that does not read is a warning naming why.
+ r = await run(c, { HOME: h }, { sourceRepo: async () => ({ via: "env", repo: "/data/source.git", main: null, error: "fatal: detected dubious ownership" }) });
+ assert.equal(find(r, "source publish", "source-repo")?.status, "warn");
+ assert.match(find(r, "source publish", "source-repo")!.detail, /^\/data\/source\.git \(ARCHILYZER_SOURCE_REPO\): main does not read — fatal: detected dubious ownership$/);
+ assert.deepEqual(tree(c.root), before);
+});
diff --git a/common/bin/doctor.ts b/common/bin/doctor.ts
@@ -6,10 +6,14 @@
// (scripts/worktree.mjs over lib/ports.mjs).
//
// It also asks the container engine whether Build all's site build image is
-// there and older than its Dockerfile (common/publish/build.ts).
+// there and older than its Dockerfile (common/publish/build.ts), and what a
+// publish needs from this machine (release 18): which yt-dlp (the image's or
+// an override), whether deploy credentials are SET (never their values), room
+// for the bundles, and the repository the source mirror reads.
//
// STRICTLY READ-ONLY. It stats, reads and runs version flags, plus the engine's
-// `image inspect` and a lock-free `git status` / `git log`. It never opens
+// `image inspect`, a lock-free `git status` / `git log` and a `git rev-parse`
+// of the source repository's main. It never opens
// LMDB (the index is stat'd, not opened), never mkdirs, never writes settings,
// and never binds a port (a port is "in use" when a TCP connect succeeds). The
// one process-state change is a chdir around umtool's table, which resolves a
@@ -23,9 +27,10 @@
// and must not be told it is broken.
import { execFile } from "node:child_process";
-import { accessSync, constants, existsSync, readFileSync, statSync } from "node:fs";
+import { accessSync, constants, existsSync, readFileSync, realpathSync, statfsSync, statSync } from "node:fs";
import { readdir } from "node:fs/promises";
import net from "node:net";
+import os from "node:os";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { promisify } from "node:util";
@@ -84,6 +89,23 @@ export type DoctorDeps = {
now?: Date;
// The source publish's tools. Default: probeSourceTools(env).
sourceTools?: () => Promise<SourceTools>;
+ // Which repository `source publish` would mirror, and whether its main reads.
+ // Default: probeSourceRepo — ARCHILYZER_SOURCE_REPO, else the checkout's git
+ // common dir, then `rev-parse` of main (read-only).
+ sourceRepo?: () => Promise<SourceRepoProbe>;
+ // Free bytes on the filesystem holding `dir`, or null when it cannot be
+ // asked. Default: statfs.
+ freeBytes?: (dir: string) => number | null;
+};
+
+// Where `source publish` would read main from: the variable's path or the
+// checkout's common dir (null: neither), and main's commit or why it did not
+// read.
+export type SourceRepoProbe = {
+ via: "env" | "checkout";
+ repo: string | null;
+ main: string | null;
+ error?: string;
};
// Which git-filter-repo `archilyzer source publish` would run, gitleaks, and
@@ -356,6 +378,42 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
else add(T, r.id, "info", `${r.error ?? "absent"} — needed only for ${r.neededBy.join(", ")}`);
}
+ // ── downloader ───────────────────────────────────────────────────────────
+ // WHICH yt-dlp, beside the tools row's "is it there". In the runtime image
+ // ARCHILYZER_IMAGE_YTDLP names the one it ships; YTDLP_BIN landing anywhere
+ // else is the operator's substitute (RUNNING_IN_DOCKER.md, "Substituting
+ // yt-dlp") — an override, which the entrypoint's YTDLP_AUTO_UPDATE leaves
+ // alone. Graded here only for that conflict and for a substitute that does
+ // not run; presence is the tools row's to grade.
+ const DL = "downloader";
+ {
+ const r = reports.find((x) => x.id === "yt-dlp");
+ const imageYtdlp = env.ARCHILYZER_IMAGE_YTDLP?.trim() || null;
+ const resolved = onPath(paths.ytdlpBin, env.PATH) ?? paths.ytdlpBin;
+ const origin = imageYtdlp === null ? "host" : sameFile(resolved, imageYtdlp) ? "image" : "override";
+ // The entrypoint's rule exactly (a shell `case`: 1, true, yes or on, as
+ // written — `TRUE` is off there, so it is off here).
+ const autoUpdate = ["1", "true", "yes", "on"].includes(env.YTDLP_AUTO_UPDATE ?? "");
+ // The shared probe counts any output as presence (an odd version flag is
+ // still a binary); "does it RUN" is asked here by exit status — the
+ // from-source wrapper with no checkout mounted prints a sentence and exits
+ // 127.
+ const ran = r?.present ? await versionRuns(resolved, env) : { ok: false as const, error: r?.error ?? "absent" };
+ const what =
+ `${resolved}${ran.ok && ran.version ? ` ${ran.version}` : ""} (${origin}` +
+ `${origin === "override" ? `; the image's is ${imageYtdlp}` : origin === "host" ? ": not in the runtime image" : ""})`;
+ if (!ran.ok) {
+ add(DL, "yt-dlp", origin === "override" ? "warn" : "info",
+ `${what} — does not run (${ran.error})${origin === "override" ? "; a from-source override needs its checkout mounted (docker-compose.ytdlp.yml)" : ""}`);
+ } else if (origin === "override" && autoUpdate) {
+ add(DL, "yt-dlp", "warn",
+ `${what} — YTDLP_AUTO_UPDATE is set, and the entrypoint never self-updates an override: update it where it is built, or unset one of the two`);
+ } else {
+ add(DL, "yt-dlp", origin === "host" ? "info" : "ok",
+ `${what}${autoUpdate && origin === "image" ? " — self-updated on every boot (YTDLP_AUTO_UPDATE)" : ""}`);
+ }
+ }
+
// ── report pipeline (umtool) ─────────────────────────────────────────────
const U = "report pipeline (umtool)";
const umtoolSpecs = await (deps.umtoolTools ?? (() => loadUmtoolTools(root)))();
@@ -411,6 +469,59 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
}
}
+ // ── publish ──────────────────────────────────────────────────────────────
+ // What a deploy needs from this machine (release 18): credentials — whether
+ // they are SET, never a value — and room for the bundles. A warning at
+ // worst: a checkout that never deploys is not broken, and one whose sites
+ // name a Cloudflare project but holds no credential is told so.
+ const PB = "publish";
+ {
+ const deployable = await sitesWithCloudflareProject(paths);
+ const set = (k: string) => Boolean(env[k]?.trim());
+ const oauth = wranglerLoginConfig(env);
+ const account = set("CLOUDFLARE_ACCOUNT_ID") ? "CLOUDFLARE_ACCOUNT_ID set" : "CLOUDFLARE_ACCOUNT_ID unset (fine with one account)";
+ if (set("CLOUDFLARE_API_TOKEN")) {
+ add(PB, "cloudflare-auth", "ok", `CLOUDFLARE_API_TOKEN is set (never printed); ${account}`);
+ } else if (oauth) {
+ add(PB, "cloudflare-auth", "ok",
+ `no CLOUDFLARE_API_TOKEN; wrangler's login config is at ${oauth} — a host login, which a container cannot use (set the token in .env there)`);
+ } else {
+ add(PB, "cloudflare-auth", deployable.length > 0 ? "warn" : "info",
+ deployable.length > 0
+ ? `neither CLOUDFLARE_API_TOKEN nor a \`wrangler login\` config, and ${deployable.length} site${deployable.length === 1 ? " names" : "s name"} a Cloudflare project (${deployable.join(", ")}) — every deploy refuses; set CLOUDFLARE_API_TOKEN (in Docker: .env)`
+ : "neither CLOUDFLARE_API_TOKEN nor a `wrangler login` config — needed only to deploy to Cloudflare Pages");
+ }
+ const bucket = settings?.archiveStorage?.bucket?.trim();
+ if (bucket) {
+ const missing = ["R2_ACCESS_KEY_ID", "R2_SECRET_ACCESS_KEY", "CLOUDFLARE_ACCOUNT_ID"].filter((k) => !set(k));
+ add(PB, "r2-keys", missing.length === 0 ? "ok" : "warn",
+ missing.length === 0
+ ? `archiveStorage.bucket "${bucket}": R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY and CLOUDFLARE_ACCOUNT_ID are set (never printed)`
+ : `archiveStorage.bucket "${bucket}" is set but ${missing.join(", ")} ${missing.length === 1 ? "is" : "are"} not — a deploy with an oversize archive refuses before it uploads`);
+ }
+ const builds = paths.exportBuildsDir;
+ if (builds) {
+ const free = (deps.freeBytes ?? statfsFree)(nearestExisting(builds));
+ const st = statOrNull(builds);
+ const freeText = free === null ? "free space unknown" : `${gigabytes(free)} free`;
+ if (!st) {
+ add(PB, "export-builds", "info", `${builds} does not exist yet — the first site build makes it (${freeText})`);
+ } else if (!writable(builds)) {
+ add(PB, "export-builds", "warn", `${builds} is not writable — every site build fails (${freeText})`);
+ } else {
+ const bundles = await bundleBytes(builds);
+ const need = Math.ceil(bundles.bytes * 1.5);
+ const what = `${builds}: ${bundles.count} bundle${bundles.count === 1 ? "" : "s"}, ${gigabytes(bundles.bytes)}; ${freeText}`;
+ if (free !== null && bundles.count > 0 && free < need) {
+ add(PB, "export-builds", "warn",
+ `${what} — under 1.5× the bundles (${gigabytes(need)}): a build writes its new bundle beside the old one before it swaps`);
+ } else {
+ add(PB, "export-builds", "ok", what);
+ }
+ }
+ }
+ }
+
// ── source publish ───────────────────────────────────────────────────────
// `archilyzer build homepage` runs it (common/publish/source.ts). Never a
// failure: a checkout that never publishes the homepage is not broken. A
@@ -449,6 +560,41 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
}
add(SP, "gitleaks", tools.gitleaks ? "ok" : "info",
tools.gitleaks ? `gitleaks ${tools.gitleaks.version}` : "absent — the gate skips the secret scan with a WARNING (the literal audit still runs)");
+ // The repository whose main is mirrored: ARCHILYZER_SOURCE_REPO (a
+ // container's read-only mount of the host's git dir), else the checkout's.
+ // A variable naming nothing fails — the publish refuses by name.
+ {
+ const sr = await (deps.sourceRepo ?? (() => probeSourceRepo(env, root)))();
+ const via = sr.via === "env" ? "ARCHILYZER_SOURCE_REPO" : "this checkout";
+ if (sr.repo === null && sr.via === "env") {
+ add(SP, "source-repo", "fail",
+ `ARCHILYZER_SOURCE_REPO names ${env.ARCHILYZER_SOURCE_REPO?.trim()}, which is not there — \`source publish\` refuses; mount it (docker-compose.source.yml) or unset it`);
+ } else if (sr.repo === null) {
+ add(SP, "source-repo", intends ? "warn" : "info",
+ "no git repository here and ARCHILYZER_SOURCE_REPO is unset — the homepage builds with an empty /source page; in Docker, add docker-compose.source.yml");
+ } else if (sr.main === null) {
+ add(SP, "source-repo", "warn", `${sr.repo} (${via}): main does not read${sr.error ? ` — ${sr.error}` : ""}`);
+ } else {
+ add(SP, "source-repo", "ok", `${sr.repo} (${via}): main ${sr.main.slice(0, 12)}`);
+ }
+ }
+ // The operator's private config dir, which holds the two files below.
+ // Counted, never listed.
+ {
+ const dir = paths.configDir;
+ const st = dir ? statOrNull(dir) : null;
+ if (!dir || !st) {
+ add(SP, "config-dir", "info",
+ `${dir ?? "(unset)"} does not exist — ARCHILYZER_CONFIG_DIR moves it (in Docker: /data/config/archilyzer, the config volume)`);
+ } else if (!st.isDirectory()) {
+ add(SP, "config-dir", "warn", `${dir} is not a directory`);
+ } else {
+ const count = (await readdir(dir).catch(() => [] as string[])).length;
+ const w = writable(dir);
+ add(SP, "config-dir", w ? "ok" : "info",
+ `${dir} (${count} entr${count === 1 ? "y" : "ies"}${w ? ", writable" : ", read-only: fine for reading the rules"})`);
+ }
+ }
for (const [id, file, unit] of [
["scrub rules", scrubFile, "rule"],
["denylist", denylistFile, "literal"],
@@ -783,6 +929,147 @@ async function dirSizeText(dir: string): Promise<string> {
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
}
+// The same file, through symlinks (the image's yt-dlp-from-source is a link
+// into /repo/docker; `YTDLP_BIN=yt-dlp` resolves on PATH first).
+function sameFile(a: string, b: string): boolean {
+ try {
+ return realpathSync(a) === realpathSync(b);
+ } catch {
+ return path.resolve(a) === path.resolve(b);
+ }
+}
+
+function writable(p: string): boolean {
+ try {
+ accessSync(p, constants.W_OK);
+ return true;
+ } catch {
+ return false;
+ }
+}
+
+// `<bin> --version` exiting 0, and its first line; else why not.
+async function versionRuns(
+ bin: string,
+ env: NodeJS.ProcessEnv,
+): Promise<{ ok: true; version: string | null } | { ok: false; error: string }> {
+ try {
+ const { stdout } = await execFileP(bin, ["--version"], { env, timeout: 15_000 });
+ return { ok: true, version: stdout.trim().split("\n")[0] || null };
+ } catch (err) {
+ const e = err as { code?: unknown; stderr?: unknown };
+ const said = String(e.stderr ?? "").trim().split("\n")[0];
+ return { ok: false, error: said || `exited ${String(e.code ?? "?")}` };
+ }
+}
+
+// `p`, or its deepest ancestor that exists — where its free space is asked.
+function nearestExisting(p: string): string {
+ let cur = path.resolve(p);
+ while (!existsSync(cur)) {
+ const up = path.dirname(cur);
+ if (up === cur) break;
+ cur = up;
+ }
+ return cur;
+}
+
+function statfsFree(dir: string): number | null {
+ try {
+ const s = statfsSync(dir);
+ return s.bavail * s.bsize;
+ } catch {
+ return null;
+ }
+}
+
+// Every `<builds>/<target>/out` bundle and their bytes, by stat (links not
+// followed). A bundle's own size is what a rebuild writes again beside it.
+async function bundleBytes(buildsDir: string): Promise<{ count: number; bytes: number }> {
+ let count = 0;
+ let bytes = 0;
+ const walk = async (d: string): Promise<void> => {
+ for (const ent of await readdir(d, { withFileTypes: true }).catch(() => [])) {
+ const p = path.join(d, ent.name);
+ if (ent.isDirectory()) await walk(p);
+ else if (ent.isFile()) bytes += statOrNull(p)?.size ?? 0;
+ }
+ };
+ for (const ent of await readdir(buildsDir, { withFileTypes: true }).catch(() => [])) {
+ if (!ent.isDirectory()) continue;
+ const out = path.join(buildsDir, ent.name, "out");
+ if (!statOrNull(out)?.isDirectory()) continue;
+ count += 1;
+ await walk(out);
+ }
+ return { count, bytes };
+}
+
+// The sites whose site.json names a Cloudflare Pages project — what "this
+// machine is configured to deploy" means. Read-only; an unreadable file is
+// skipped.
+async function sitesWithCloudflareProject(paths: Paths): Promise<string[]> {
+ if (!paths.sitesDir) return [];
+ const out: string[] = [];
+ for (const e of await readdir(paths.sitesDir, { withFileTypes: true }).catch(() => [])) {
+ if (!e.isDirectory() || e.name.startsWith("_")) continue;
+ const text = readOrNull(path.join(paths.sitesDir, e.name, "site.json"));
+ if (text === null) continue;
+ try {
+ const raw = JSON.parse(text) as { cloudflareProject?: unknown };
+ if (typeof raw.cloudflareProject === "string" && raw.cloudflareProject.trim()) out.push(e.name);
+ } catch {
+ /* not JSON: the site's own problem */
+ }
+ }
+ return out.sort();
+}
+
+// wrangler's `wrangler login` (OAuth) config, where wrangler keeps it: under
+// XDG_CONFIG_HOME (~/.config) since v3, ~/.wrangler before, ~/Library/
+// Preferences on macOS. Its PATH is reported; it is never read.
+function wranglerLoginConfig(env: NodeJS.ProcessEnv): string | null {
+ const home = env.HOME || os.homedir();
+ const xdg = env.XDG_CONFIG_HOME || path.join(home, ".config");
+ for (const p of [
+ path.join(xdg, ".wrangler", "config", "default.toml"),
+ path.join(home, ".wrangler", "config", "default.toml"),
+ path.join(home, "Library", "Preferences", ".wrangler", "config", "default.toml"),
+ ]) {
+ if (existsSync(p)) return p;
+ }
+ return null;
+}
+
+// Which repository `source publish` would mirror (the same order as
+// source.ts's sourceRepoFor) and main's commit there. Read-only: rev-parse
+// with GIT_OPTIONAL_LOCKS=0.
+async function probeSourceRepo(env: NodeJS.ProcessEnv, root: string): Promise<SourceRepoProbe> {
+ const opts = { cwd: root, env: { ...env, GIT_OPTIONAL_LOCKS: "0" }, timeout: 10_000 };
+ const named = env.ARCHILYZER_SOURCE_REPO?.trim();
+ let via: SourceRepoProbe["via"] = "checkout";
+ let repo: string | null = null;
+ if (named) {
+ via = "env";
+ repo = existsSync(named) ? named : null;
+ } else {
+ try {
+ const { stdout } = await execFileP("git", ["rev-parse", "--path-format=absolute", "--git-common-dir"], opts);
+ repo = stdout.trim().split("\n").pop() || null;
+ } catch {
+ repo = null;
+ }
+ }
+ if (repo === null) return { via, repo, main: null };
+ try {
+ const { stdout } = await execFileP("git", [`--git-dir=${repo}`, "rev-parse", "--verify", "--quiet", "refs/heads/main^{commit}"], opts);
+ return { via, repo, main: stdout.trim() || null, error: stdout.trim() ? undefined : "no main branch" };
+ } catch (err) {
+ const stderr = String((err as { stderr?: unknown }).stderr ?? "").trim().split("\n")[0];
+ return { via, repo, main: null, error: stderr || "no main branch" };
+ }
+}
+
// In use = something accepts a TCP connection on 127.0.0.1. Never binds.
function tcpPortInUse(port: number): Promise<boolean> {
return new Promise((resolve) => {
diff --git a/common/lib/envVars.ts b/common/lib/envVars.ts
@@ -81,7 +81,7 @@ const DECLARED: EnvVarDecl[] = [
paths("ARIA2C_BIN", "`aria2c` on PATH", "Fetches an archive.org file over BitTorrent (the item's torrent, archive.org as web seed), then seeds it for a while. Optional: without it archive.org files are downloaded directly. See `archiveOrg` in [SETTINGS.md](SETTINGS.md)."),
paths("OLLAMA_URL", "`http://127.0.0.1:11434`", "The local ollama server, the local digest and attribution engine."),
paths("CLAUDE_BIN", "`claude` on PATH", "The `claude` CLI, driving the opt-in metered digest lane."),
- paths("ARCHILYZER_CONFIG_DIR", "`~/.config/archilyzer`", "The operator's private config dir, outside the repo: the two inputs of `archilyzer source publish` below. Never committed."),
+ paths("ARCHILYZER_CONFIG_DIR", "`~/.config/archilyzer`", "The operator's private config dir, outside the repo: the two inputs of `archilyzer source publish` below. Never committed. In Docker it is `/data/config/archilyzer`, in the config volume (docker-compose.yml)."),
paths("SOURCE_SCRUB_FILE", "`<ARCHILYZER_CONFIG_DIR>/source-scrub.txt`", "git-filter-repo `lhs==>rhs` rules applied to file contents AND commit messages when the source mirror is generated (`<home dir>==>/home/user` is built in and runs first). Every rule's left side is also denied. See [PUBLISH.md](PUBLISH.md)."),
paths("SOURCE_DENYLIST_FILE", "`<ARCHILYZER_CONFIG_DIR>/source-denylist.txt`", "Literals the published source must never contain, one per line (`i:` = any case). One hit anywhere in the mirror, the tree or the tarball refuses the publish."),
paths("ARCHILYZER_SOURCE_SCRATCH", "the OS temp dir", "Where `source publish` makes its scratch clone and stage (removed afterwards unless `--keep-scratch`)."),
@@ -93,9 +93,15 @@ const DECLARED: EnvVarDecl[] = [
{ name: "SYNC_HEARTBEAT_SECONDS", audience: "runtime", default: "`settings.syncScheduler.heartbeatSeconds`", readBy: "editor/app/scheduler/heartbeat.ts", doc: "Overrides the editor's in-process sync heartbeat. `0` = no internal timer (tick from cron instead)." },
{ name: "SYNC_TICK_URL", audience: "runtime", default: "`http://127.0.0.1:3001/api/scheduler/tick`", readBy: "common/bin/sync-tick.ts", doc: "Where `archilyzer sync tick` (cron's heartbeat) posts." },
{ name: "SYNC_TICK_TOKEN", audience: "runtime", default: "unset (no auth)", readBy: "common/bin/sync-tick.ts, editor/app/scheduler/auth.ts", doc: "Bearer token for the tick endpoint; set on both the editor and the cron job." },
- { name: "R2_ACCESS_KEY_ID", audience: "runtime", default: "—", readBy: "common/publish/build.ts", doc: "R2 S3 credentials for uploading oversize archives at deploy time (with `R2_SECRET_ACCESS_KEY` and `CLOUDFLARE_ACCOUNT_ID`). See [PUBLISH.md](PUBLISH.md)." },
- { name: "R2_SECRET_ACCESS_KEY", audience: "runtime", default: "—", readBy: "common/publish/build.ts", doc: "See `R2_ACCESS_KEY_ID`." },
- { name: "CLOUDFLARE_ACCOUNT_ID", audience: "runtime", default: "—", readBy: "common/publish/build.ts", doc: "The account the R2 endpoint belongs to. wrangler reads its own credentials." },
+ { name: "R2_ACCESS_KEY_ID", audience: "runtime", default: "—", readBy: "common/publish/build.ts, common/bin/doctor.ts (set or not)", doc: "R2 S3 credentials for uploading oversize archives at deploy time (with `R2_SECRET_ACCESS_KEY` and `CLOUDFLARE_ACCOUNT_ID`), needed only when `archiveStorage.bucket` is set. In Docker they come from `.env`. See [PUBLISH.md](PUBLISH.md)." },
+ { name: "R2_SECRET_ACCESS_KEY", audience: "runtime", default: "—", readBy: "common/publish/build.ts, common/bin/doctor.ts (set or not)", doc: "See `R2_ACCESS_KEY_ID`." },
+ { name: "CLOUDFLARE_ACCOUNT_ID", audience: "runtime", default: "—", readBy: "common/publish/build.ts, wrangler, common/bin/doctor.ts (set or not)", doc: "The Cloudflare account: the R2 endpoint's, and the one wrangler deploys to when the token can see more than one. In Docker it comes from `.env`." },
+ { name: "CLOUDFLARE_API_TOKEN", audience: "runtime", default: "unset (wrangler's own `wrangler login` config, on a host)", readBy: "wrangler (every deploy), common/bin/doctor.ts (set or not, never the value)", doc: "The API token every deploy's wrangler authenticates with (Cloudflare Pages: Edit). The way a container deploys — there is no browser for `wrangler login` in one; set it in `.env`." },
+ { name: "ARCHILYZER_SOURCE_REPO", audience: "runtime", default: "this checkout's git common dir", readBy: "common/publish/source.ts, common/bin/doctor.ts, docker/entrypoint.sh", doc: "The git DIR `archilyzer source publish` mirrors `main` from, when the checkout has none: in Docker, `/data/source.git`, the host's git common dir mounted read-only by docker-compose.source.yml. A value that names nothing refuses the publish." },
+ { name: "YTDLP_SOURCE_HOST_DIR", audience: "runtime", default: "— (required by the overlay)", readBy: "docker-compose.ytdlp.yml", doc: "Docker: the HOST path of a yt-dlp source checkout (the directory holding `yt_dlp/`), mounted read-only at `/opt/yt-dlp-src` by docker-compose.ytdlp.yml. See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md), \"Substituting yt-dlp\"." },
+ { name: "YTDLP_AUTO_UPDATE", audience: "runtime", default: "off", readBy: "docker/entrypoint.sh, common/bin/doctor.ts", doc: "Docker: `1` runs `yt-dlp -U` on every editor boot — on the image's yt-dlp only; an override (`YTDLP_BIN` naming another) is left alone, with a warning." },
+ { name: "XDG_CONFIG_HOME", audience: "runtime", default: "`~/.config`", readBy: "common/bin/doctor.ts", doc: "Where `wrangler login` keeps its config (`<it>/.wrangler/config/default.toml`); the doctor looks for it there, by path, never reading it." },
+ { name: "YTDLP_SOURCE_DIR", audience: "runtime", default: "`/opt/yt-dlp-src`", readBy: "docker/yt-dlp-from-source.sh", doc: "Docker: where `/usr/local/bin/yt-dlp-from-source` finds the yt-dlp source tree it runs with the image's python." },
{ name: "DOCKER_BIN", audience: "runtime", default: "`docker`", readBy: "common/publish/build.ts", doc: "The container engine for docker-mode builds (e.g. `podman`)." },
{ name: "DOCKER_BUILD_MEMORY", audience: "runtime", default: "no cap", readBy: "common/publish/build.ts", doc: "Per-container memory cap for a docker-mode build (`--memory`)." },
{ name: "DOCKER_BUILD_CPUS", audience: "runtime", default: "no cap", readBy: "common/publish/build.ts", doc: "Per-container CPU cap for a docker-mode build (`--cpus`)." },
@@ -153,6 +159,11 @@ const DECLARED: EnvVarDecl[] = [
{ name: "ARCHILYZER_MODELS_DIR", audience: "docker", default: "`/data/models`", readBy: "docker/entrypoint.sh", doc: "Where models live in the container." },
{ name: "ARCHILYZER_BUILDS_DIR", audience: "docker", default: "`/data/builds`", readBy: "docker/entrypoint.sh", doc: "Where the container keeps built sites." },
{ name: "ARCHILYZER_SITE_OUT", audience: "docker", default: "`/data/builds/site`", readBy: "docker/entrypoint.sh, docker/publish-site.sh", doc: "The built export site the `site` service serves." },
+ { name: "ARCHILYZER_HOMEPAGE_OUT", audience: "docker", default: "`/data/builds/homepage`", readBy: "docker/entrypoint.sh", doc: "The locally deployed homepage (`publish homepage --deploy --to local`). The `homepage` service serves it when it is non-empty, else the image's baked build." },
+ { name: "ARCHILYZER_IMAGE_YTDLP", audience: "docker", default: "baked: `/usr/local/bin/yt-dlp`", readBy: "docker/entrypoint.sh, common/bin/doctor.ts", doc: "The yt-dlp the image ships. `YTDLP_BIN` naming anything else is an OVERRIDE: the boot's `yt-dlp:` line and `archilyzer doctor` say so, and `YTDLP_AUTO_UPDATE` leaves it alone." },
+ { name: "ARCHILYZER_COMMIT", audience: "docker", default: "baked: empty unless the build passed it", readBy: "the Dockerfile (a build arg), the publish stamps (`IMAGE_COMMIT_ENV`)", doc: "The commit the image was built from — the publish stamps' `commit` where there is no .git. `ARCHILYZER_COMMIT=$(git rev-parse HEAD) docker compose build`." },
+ { name: "ARCHILYZER_BRANCH", audience: "docker", default: "baked: empty unless the build passed it", readBy: "the Dockerfile (a build arg), the publish stamps (`IMAGE_BRANCH_ENV`)", doc: "The branch the image was built from — the stamps' `branch`, which a production deploy checks." },
+ { name: "ARCHILYZER_SOURCE_HOST_DIR", audience: "docker", default: "`./.git`", readBy: "docker-compose.source.yml", doc: "The HOST's git common dir docker-compose.source.yml mounts at `/data/source.git`. In a git worktree, the primary checkout's `.git`." },
{ name: "ARCHILYZER_IDLE_BOOT", audience: "docker", default: "off", readBy: "common/lib/idleBoot.ts (the editor)", doc: "`1` boots the editor without arming the heartbeat or any auto-queue runner." },
{ name: "ARCHILYZER_AUTH_MODE", audience: "docker", default: "`basic`", readBy: "docker/guard-exposure.sh, docker/caddy-start.sh", doc: "`basic`, `forward` or `none` — the only escape hatch from the exposure guard." },
{ name: "ARCHILYZER_AUTH_USER", audience: "docker", default: "`archilyzer`", readBy: "docker/Caddyfile", doc: "Basic-auth user." },
@@ -216,10 +227,21 @@ export const ENV_AUDIENCES: ReadonlyArray<{ id: EnvAudience; title: string; intr
{ id: "runtime", title: "Runtime", intro: "Tokens, credentials and knobs a running process reads. Most configuration is not here but in `settings.json` ([SETTINGS.md](SETTINGS.md))." },
{ id: "port", title: "Ports", intro: "Every local server's default port, from `common/lib/ports.mjs`. The primary checkout uses these; worktree N adds N × 100 (`pnpm wt list`)." },
{ id: "internal", title: "Set by the pipeline", intro: "The publish pipeline sets these for a process it spawns. Listed so a reader knows what they are; nobody sets them by hand." },
- { id: "docker", title: "Docker", intro: "The container's own set, read by `docker/*.sh`, the compose files and Caddy — not by the apps' code (except `ARCHILYZER_IDLE_BOOT`). See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md)." },
+ { id: "docker", title: "Docker", intro: "The container's own set, read by `docker/*.sh`, the compose files and Caddy — not by the apps' code (except `ARCHILYZER_IDLE_BOOT`, and the image's facts `archilyzer doctor` and the publish stamps read). See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md)." },
{ id: "test", title: "Tests only", intro: "Read only by a test harness, a fake binary or a test-mode branch. Never set one on a real instance." },
];
+// The two build facts the runtime image bakes (Dockerfile build args → ENV), by
+// name: the publish stamps' `commit`/`branch` where there is no .git to ask.
+export const IMAGE_COMMIT_ENV = "ARCHILYZER_COMMIT";
+export const IMAGE_BRANCH_ENV = "ARCHILYZER_BRANCH";
+
+// Those facts, or null each — an image built without the args bakes them empty.
+export function imageBuildFacts(env: NodeJS.ProcessEnv = process.env): { commit: string | null; branch: string | null } {
+ const v = (k: string) => env[k]?.trim() || null;
+ return { commit: v(IMAGE_COMMIT_ENV), branch: v(IMAGE_BRANCH_ENV) };
+}
+
export function envVar(name: string): EnvVarDecl | undefined {
return ENV_VARS.find((v) => v.name === name);
}
diff --git a/common/publish/buildImage.test.ts b/common/publish/buildImage.test.ts
@@ -1,7 +1,9 @@
// The container build's contract with the repo: what Dockerfile.build's
// image bakes into export/public and how docker/build-site.sh puts it in front
-// of each site's `next build`. No docker here — the invariant is read from git,
-// and the shell function is read out of build-site.sh and run with bash.
+// of each site's `next build`; and what the runtime image (the root Dockerfile)
+// must agree with the code about. No docker here — the invariant is read from
+// git or the Dockerfile's text, and the shell function is read out of
+// build-site.sh and run with bash.
//
// Run with:
// pnpm --filter yt-dlp-transcript-common test
@@ -13,6 +15,7 @@ import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync
import { tmpdir } from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
+import { FILTER_REPO_PIPX_SPEC } from "./source";
const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
@@ -88,3 +91,38 @@ test("build-site.sh's asset sync ships a changed svg, drops a removed one, and t
rmSync(root, { recursive: true, force: true });
}
});
+
+// The runtime image (the root Dockerfile) installs git-filter-repo with pipx so
+// a container can publish the homepage's /source mirror without a network fetch.
+// The version it bakes must be the one `source publish` would fetch with
+// `pipx run --spec` on a host (FILTER_REPO_PIPX_SPEC): the filter-repo version
+// is part of the publish's skip key, so a drift between the two is a rebuild
+// nobody asked for, or a mirror rewritten by a version nobody tested.
+test("the runtime image installs the git-filter-repo that source publish names, in every target", () => {
+ const dockerfile = readFileSync(path.join(REPO, "Dockerfile"), "utf8");
+ const installs = [...dockerfile.matchAll(/pipx install (\S+)/g)].map((m) => m[1]);
+ // runtime-base (runtime + runtime-vulkan) and runtime-cuda, which repeats it.
+ assert.equal(installs.length, 2, `expected two \`pipx install\` lines, found ${installs.length}`);
+ for (const spec of installs) {
+ assert.equal(
+ spec,
+ FILTER_REPO_PIPX_SPEC,
+ `the Dockerfile installs ${spec}, source.ts names ${FILTER_REPO_PIPX_SPEC} — move both together`,
+ );
+ }
+});
+
+// The yt-dlp substitution hook: the image ships its own yt-dlp as
+// ARCHILYZER_IMAGE_YTDLP beside YTDLP_BIN (the entrypoint and the doctor tell an
+// override by the difference), and links the from-source wrapper — in every
+// target.
+test("every runtime target names its own yt-dlp and links the from-source wrapper", () => {
+ const dockerfile = readFileSync(path.join(REPO, "Dockerfile"), "utf8");
+ const image = [...dockerfile.matchAll(/^\s+ARCHILYZER_IMAGE_YTDLP=(\S+) \\$/gm)].map((m) => m[1]);
+ const bin = [...dockerfile.matchAll(/^\s+YTDLP_BIN=(\S+) \\$/gm)].map((m) => m[1]);
+ assert.equal(image.length, 2);
+ assert.deepEqual(image, bin, "ARCHILYZER_IMAGE_YTDLP and YTDLP_BIN start equal in each target");
+ const links = dockerfile.match(/ln -s \/repo\/docker\/yt-dlp-from-source\.sh \/usr\/local\/bin\/yt-dlp-from-source/g) ?? [];
+ assert.equal(links.length, 2);
+ assert.ok(existsSync(path.join(REPO, "docker", "yt-dlp-from-source.sh")));
+});
diff --git a/common/publish/source.test.ts b/common/publish/source.test.ts
@@ -375,6 +375,35 @@ test("no git repository here (a docker runtime, a tarball install): the build ge
assert.ok(!existsSync(path.join(pub, "source")));
});
+test("ARCHILYZER_SOURCE_REPO names the repository where the checkout has none (the container's mount); one that names nothing refuses by name (release 18)", async () => {
+ const bare = dir("no-git-env");
+ const pub = path.join(dir("site"), "public");
+ const logs: string[] = [];
+ const base: SourcePublishOpts = {
+ paths: { monorepoRoot: bare } as Paths,
+ publicDir: pub,
+ onLog: (l) => logs.push(l),
+ scrubFile: path.join(bare, "none.txt"),
+ denylistFile: path.join(bare, "none.txt"),
+ check: true,
+ };
+ // A mounted repository: found, so the publish goes on to the next step —
+ // here, the operator's files, which this scenario leaves out on purpose.
+ // It is a git DIR — what the overlay mounts (the host's common dir), and what
+ // the checkout lookup answers.
+ const repo = path.join(sourceRepo(), ".git");
+ assert.equal(await publishSource({ ...base, env: { ...process.env, ARCHILYZER_SOURCE_REPO: repo } }), 1);
+ assert.ok(!logs.includes(`[source] ${NO_REPOSITORY}`), logs.join("\n"));
+ assert.match(logs.join("\n"), /none\.txt/, "it reached the operator's files");
+ // A variable naming a path that is not there: a refusal that names it, never
+ // the "no repository" sentence.
+ logs.length = 0;
+ const gone = path.join(bare, "not-mounted.git");
+ assert.equal(await publishSource({ ...base, env: { ...process.env, ARCHILYZER_SOURCE_REPO: gone } }), 1);
+ assert.match(logs.join("\n"), /REFUSED: ARCHILYZER_SOURCE_REPO names .*not-mounted\.git, which is not there/);
+ assert.ok(!logs.includes(`[source] ${NO_REPOSITORY}`));
+});
+
test("round trip: --check writes nothing; publish; a dumb clone of the mirror is main, scrubbed, from static files", async (t) => {
if (filterRepoProblem) return t.skip(`git-filter-repo unavailable: ${filterRepoProblem}`);
const repo = sourceRepo();
diff --git a/common/publish/source.ts b/common/publish/source.ts
@@ -133,7 +133,8 @@ export type SourcePublishOpts = PublishOpts & {
check?: boolean;
// Leave the scratch dir (the rewritten bare clone, the stage) for a look.
keepScratch?: boolean;
- // The repository to mirror. Default: this checkout's git COMMON dir, so a
+ // The repository to mirror. Default: ARCHILYZER_SOURCE_REPO (a container's
+ // mount of the host's repository), else this checkout's git COMMON dir, so a
// worktree build mirrors the primary's main.
sourceRepo?: string;
// Default: HOMEPAGE_PUBLIC_DIR, else <repo>/homepage/public.
@@ -611,6 +612,25 @@ async function commonDir(ctx: Ctx, cwd: string): Promise<string | null> {
throw new SourceRefusal(`git rev-parse exited ${r.code}${tail ? `: ${tail}` : ""}`);
}
+// The repository to mirror when the caller names none: ARCHILYZER_SOURCE_REPO
+// when it is set — in a container, the host's git common dir mounted read-only
+// by docker-compose.source.yml, since the image has no .git — else this
+// checkout's common dir. A variable that names nothing is a refusal, not a
+// silent fall-through to "no repository" (which would withdraw the publish
+// with a sentence about tarball installs).
+async function sourceRepoFor(ctx: Ctx, cwd: string): Promise<string | null> {
+ const named = ctx.env.ARCHILYZER_SOURCE_REPO?.trim();
+ if (named) {
+ if (!existsSync(named)) {
+ throw new SourceRefusal(
+ `ARCHILYZER_SOURCE_REPO names ${named}, which is not there — mount the host's git common dir there (docker-compose.source.yml), or unset it`,
+ );
+ }
+ return named;
+ }
+ return commonDir(ctx, cwd);
+}
+
// The real path of `p`, or of its deepest existing ancestor with the rest
// appended: a scratch root may not exist yet, and a symlink must not hide
// where it lands.
@@ -764,7 +784,7 @@ async function publish(
// 1. The private main — or no repository at all (L8: the docker runtime,
// a tarball install), where nothing can be mirrored and nothing can leak.
- const sourceRepo = opts.sourceRepo ?? (await commonDir(ctx, paths.monorepoRoot));
+ const sourceRepo = opts.sourceRepo ?? (await sourceRepoFor(ctx, paths.monorepoRoot));
if (sourceRepo === null) {
onLog(`[source] ${NO_REPOSITORY}`);
if (opts.check) return 1;
@@ -1328,7 +1348,7 @@ export async function publishedSourceProblem(
};
let main: string | null = null;
try {
- const repo = opts.sourceRepo ?? (await commonDir(ctx, paths.monorepoRoot));
+ const repo = opts.sourceRepo ?? (await sourceRepoFor(ctx, paths.monorepoRoot));
main = repo ? await revParse(ctx, repo, `refs/heads/${SOURCE_BRANCH}^{commit}`) : null;
} catch (err) {
if (!(err instanceof SourceRefusal)) throw err;
diff --git a/docker-compose.source.yml b/docker-compose.source.yml
@@ -0,0 +1,36 @@
+# Overlay: let the container publish the homepage's /source mirror.
+#
+# docker compose -f docker-compose.yml -f docker-compose.source.yml up -d
+#
+# `archilyzer source publish` (run by the homepage build) mirrors this repo's
+# `main` — but the image has no .git (.dockerignore keeps it out, and it would be
+# stale the day after the build anyway). This mounts the HOST's git common dir,
+# read-only, at /data/source.git and points ARCHILYZER_SOURCE_REPO at it;
+# common/publish/source.ts reads that before it looks for a checkout. Nothing in
+# the container writes to it: the publish clones it into scratch first.
+#
+# ARCHILYZER_SOURCE_HOST_DIR is the host path, default `./.git` — right for a
+# plain clone. In a git WORKTREE `.git` is a file; use the primary checkout's
+# `.git` (`git rev-parse --git-common-dir` prints it).
+#
+# The entrypoint marks /data/source.git as a git safe.directory (the files
+# belong to your host user, the container runs as root). The scrub rules and
+# the denylist the publish needs live in the config volume
+# (ARCHILYZER_CONFIG_DIR, /data/config/archilyzer): `docker compose cp` them in.
+# RUNNING_IN_DOCKER.md, "Publish the archive".
+#
+# The long mount syntax with create_host_path: false, on purpose: the short form
+# makes a missing host path (a tarball install, a typo) into an empty root-owned
+# directory on the host. This way `up` fails and names the path instead.
+
+services:
+ editor:
+ environment:
+ ARCHILYZER_SOURCE_REPO: /data/source.git
+ volumes:
+ - type: bind
+ source: ${ARCHILYZER_SOURCE_HOST_DIR:-./.git}
+ target: /data/source.git
+ read_only: true
+ bind:
+ create_host_path: false
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -31,6 +31,13 @@ x-app: &app
build:
context: .
target: runtime
+ # Which commit the image is built from, for the publish stamps (the image
+ # has no .git). Empty unless the shell sets them:
+ # ARCHILYZER_COMMIT=$(git rev-parse HEAD) \
+ # ARCHILYZER_BRANCH=$(git branch --show-current) docker compose build
+ args:
+ ARCHILYZER_COMMIT: ${ARCHILYZER_COMMIT:-}
+ ARCHILYZER_BRANCH: ${ARCHILYZER_BRANCH:-}
image: archilyzer:${ARCHILYZER_TAG:-local}
restart: unless-stopped
networks: [archilyzer]
@@ -38,6 +45,11 @@ x-app: &app
# cosmetic: a bcrypt hash is full of `$`, and compose interpolates `${...}` in
# the YAML but passes env_file values through verbatim. Put the hash in .env
# and it arrives intact, with nothing to escape.
+ #
+ # The same goes for the publish credentials: CLOUDFLARE_API_TOKEN,
+ # CLOUDFLARE_ACCOUNT_ID and the R2_* keys are read from .env by the editor,
+ # and every publish stage it runs (a child process) inherits them. They are
+ # deliberately NOT in x-app-env below — a value there would override .env.
env_file:
- path: .env
required: false
@@ -59,6 +71,13 @@ x-app-env: &app-env
EXPORT_INDEX_DIR: /data/builds/.export-index
EXPORT_BUILDS_DIR: /data/builds/.export-builds
ARCHILYZER_SITE_OUT: /data/builds/site
+ # `publish homepage --deploy --to local` writes here; the homepage service
+ # serves it once it is non-empty (else the image's baked build).
+ ARCHILYZER_HOMEPAGE_OUT: /data/builds/homepage
+ # The operator's private config dir (common/lib/paths.ts): the source
+ # mirror's scrub rules and denylist live here, in the config volume —
+ # `docker compose cp` them in; they are never printed.
+ ARCHILYZER_CONFIG_DIR: /data/config/archilyzer
# Set explicitly so a stray EDITOR_PORT/UMTOOL_PORT in .env cannot move an app
# off the port Caddy proxies to. These are INTERNAL ports; the published ones
# are on the caddy service below.
@@ -133,7 +152,10 @@ services:
- builds:/data/builds
# -------------------------------------------------------------------------
- # The project's own site (marketing + docs). Baked into the image.
+ # The project's own site (marketing + docs). A build is baked into the
+ # image; a local deploy (`publish homepage --deploy --to local`) into the
+ # builds volume replaces it once there is one — restart this service after
+ # the first.
# -------------------------------------------------------------------------
homepage:
<<: *app
@@ -141,6 +163,10 @@ services:
command: ["homepage"]
environment:
<<: *app-env
+ volumes:
+ # Writable for the same reason as the site's: the entrypoint makes the
+ # (empty) directory it looks in.
+ - builds:/data/builds
# -------------------------------------------------------------------------
# umtool: the clip/report bench. Private, like the editor.
@@ -167,12 +193,15 @@ volumes:
# The corpus. This is the one that matters: real media, real transcripts,
# hundreds of GB when it grows up. Back it up.
corpus:
- # settings.json, and anything else operational that must outlive the container.
+ # settings.json, the operator's private config dir (archilyzer/: the source
+ # mirror's rules), and anything else operational that must outlive the
+ # container.
config:
# whisper .bin models — 142 MB to 3 GB, fetched once.
models:
- # Export build staging + the published static site. Reproducible; losing it
- # costs a rebuild, not data.
+ # Export build staging, the shared index, every site's bundle and stamps, and
+ # the locally deployed site and homepage. Reproducible; losing it costs a
+ # rebuild, not data.
builds:
caddy-data:
caddy-config:
diff --git a/docker-compose.ytdlp.yml b/docker-compose.ytdlp.yml
@@ -0,0 +1,25 @@
+# Overlay: run YOUR yt-dlp — a source checkout on the host — instead of the
+# image's release binary. No rebuild: the swap happens at run time.
+#
+# # .env
+# YTDLP_SOURCE_HOST_DIR=/home/you/yt-dlp-patched # holds the yt_dlp/ package
+#
+# docker compose -f docker-compose.yml -f docker-compose.ytdlp.yml up -d
+#
+# The checkout is mounted read-only at /opt/yt-dlp-src and YTDLP_BIN points at
+# /usr/local/bin/yt-dlp-from-source, a wrapper baked into the image
+# (docker/yt-dlp-from-source.sh) that runs `python3 -m yt_dlp` from it with the
+# image's python and yt-dlp's optional modules.
+#
+# Every editor boot prints `yt-dlp: <path> <version> (override)`, and
+# YTDLP_AUTO_UPDATE leaves an override alone — update the checkout instead.
+# A zipapp built on the host (`make yt-dlp` in the checkout) works too, with no
+# overlay: copy it into the config volume and set YTDLP_BIN to its path there.
+# RUNNING_IN_DOCKER.md, "Substituting yt-dlp".
+
+services:
+ editor:
+ environment:
+ YTDLP_BIN: /usr/local/bin/yt-dlp-from-source
+ volumes:
+ - ${YTDLP_SOURCE_HOST_DIR:?set YTDLP_SOURCE_HOST_DIR in .env to your yt-dlp checkout}:/opt/yt-dlp-src:ro
diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
@@ -4,7 +4,8 @@
#
# editor the admin app (next start, :3001)
# site the published archive (serve, :3000)
-# homepage the project's own site (serve, :3031)
+# homepage the project's own site (serve, :3031) — the local deploy in the
+# builds volume when there is one, else the image's baked build
# umtool the clip/report bench (next start, :3050)
# shell drop into bash — for `docker compose run --rm editor shell`
#
@@ -22,6 +23,7 @@ SETTINGS_FILE="${SETTINGS_FILE:-/data/config/settings.json}"
MODELS_DIR="${ARCHILYZER_MODELS_DIR:-/data/models}"
BUILDS_DIR="${ARCHILYZER_BUILDS_DIR:-/data/builds}"
SITE_OUT="${ARCHILYZER_SITE_OUT:-${BUILDS_DIR}/site}"
+HOMEPAGE_OUT="${ARCHILYZER_HOMEPAGE_OUT:-${BUILDS_DIR}/homepage}"
WHISPER_MODEL="${WHISPER_MODEL:-${MODELS_DIR}/ggml-base.en.bin}"
# Which transcription backend THIS IMAGE was built with. Set by the Dockerfile
# per target, not by the operator: `runtime`/`runtime-cuda` carry whisper.cpp,
@@ -42,7 +44,26 @@ mkdir -p \
"$(dirname "${SETTINGS_FILE}")" \
"${MODELS_DIR}" \
"${BUILDS_DIR}" \
- "${SITE_OUT}"
+ "${SITE_OUT}" \
+ "${HOMEPAGE_OUT}"
+# The operator's private config dir (compose sets it inside the config volume),
+# made so `docker compose cp` has somewhere to put the source mirror's rules.
+# Only made, never filled: what goes in it is the operator's.
+if [ -n "${ARCHILYZER_CONFIG_DIR:-}" ] && [ ! -d "${ARCHILYZER_CONFIG_DIR}" ]; then
+ mkdir -p "${ARCHILYZER_CONFIG_DIR}"
+ chmod 700 "${ARCHILYZER_CONFIG_DIR}" || true
+fi
+
+# The host's git common dir, mounted read-only by docker-compose.source.yml for
+# the homepage's source mirror (`archilyzer source publish`). It belongs to the
+# host's user and the container runs as root, so git would refuse it as
+# "dubious ownership" without this. Added once — `--add` on every boot would
+# pile up duplicates in a container that is restarted rather than recreated.
+SOURCE_REPO_DIR="${ARCHILYZER_SOURCE_REPO:-/data/source.git}"
+if ! git config --global --get-all safe.directory 2>/dev/null | grep -qxF "${SOURCE_REPO_DIR}"; then
+ git config --global --add safe.directory "${SOURCE_REPO_DIR}" ||
+ printf '[entrypoint] WARNING: could not mark %s safe for git\n' "${SOURCE_REPO_DIR}"
+fi
# ---------------------------------------------------------------------------
# 2. Seed settings.json — with a worker.
@@ -163,20 +184,67 @@ fetch_model() {
}
# ---------------------------------------------------------------------------
-# 4. yt-dlp self-update.
+# 4. yt-dlp: which one, and its self-update.
+#
+# YTDLP_BIN is the yt-dlp every fetch runs; ARCHILYZER_IMAGE_YTDLP is the one
+# this image ships. They differ when the operator substituted their own (a
+# zipapp, or /usr/local/bin/yt-dlp-from-source over a mounted source tree —
+# RUNNING_IN_DOCKER.md, "Substituting yt-dlp"): an OVERRIDE.
#
# An image pins yt-dlp at build time, and a stale yt-dlp is the most common
# reason downloads suddenly start failing — YouTube changes, yt-dlp ships a fix
-# within days, and an image from last month has none of them. Opt-in because it
-# is a network call on every boot.
+# within days, and an image from last month has none of them. The self-update is
+# opt-in because it is a network call on every boot, and it never touches an
+# override: a patched build is updated where it is built, and `-U` on one either
+# fails or replaces the patch with a release.
# ---------------------------------------------------------------------------
+YTDLP="${YTDLP_BIN:-yt-dlp}"
+IMAGE_YTDLP="${ARCHILYZER_IMAGE_YTDLP:-}"
+
+# "image" or "override", by where each one actually lands.
+ytdlp_origin() {
+ local mine theirs
+ if [ -z "${IMAGE_YTDLP}" ]; then
+ echo image
+ return 0
+ fi
+ mine="$(readlink -f "$(command -v "${YTDLP}" 2>/dev/null || echo "${YTDLP}")" 2>/dev/null || echo "${YTDLP}")"
+ theirs="$(readlink -f "${IMAGE_YTDLP}" 2>/dev/null || echo "${IMAGE_YTDLP}")"
+ if [ "${mine}" = "${theirs}" ]; then echo image; else echo override; fi
+}
+
update_ytdlp() {
case "${YTDLP_AUTO_UPDATE:-0}" in
1 | true | yes | on) ;;
*) return 0 ;;
esac
+ if [ "$(ytdlp_origin)" = "override" ]; then
+ log "WARNING: YTDLP_AUTO_UPDATE is set, but YTDLP_BIN (${YTDLP}) is not the image's yt-dlp"
+ log " (${IMAGE_YTDLP}) — an override is not self-updated; update it where it is built"
+ return 0
+ fi
log "yt-dlp -U (YTDLP_AUTO_UPDATE is set)"
- "${YTDLP_BIN:-yt-dlp}" -U || log "WARNING: yt-dlp self-update failed — continuing with $("${YTDLP_BIN:-yt-dlp}" --version 2>/dev/null || echo unknown)"
+ "${YTDLP}" -U || log "WARNING: yt-dlp self-update failed — continuing with $("${YTDLP}" --version 2>/dev/null || echo unknown)"
+}
+
+# One line on every boot, like the vulkan: line — "which yt-dlp is this, and
+# does it run?" answered before the first download has to. A yt-dlp that is
+# missing or does not run is a WARNING, never a refusal: the editor is still
+# useful (reading, building, publishing) without one.
+ytdlp_line() {
+ local origin version err
+ origin="$(ytdlp_origin)"
+ if ! command -v "${YTDLP}" >/dev/null 2>&1; then
+ log "yt-dlp: MISSING ${YTDLP} (${origin}) — every download will fail until it exists"
+ return 0
+ fi
+ if version="$(timeout 30 "${YTDLP}" --version 2>/dev/null | head -n 1)" && [ -n "${version}" ]; then
+ log "yt-dlp: ${YTDLP} ${version} (${origin})"
+ else
+ err="$(timeout 30 "${YTDLP}" --version 2>&1 >/dev/null | head -n 1 || true)"
+ log "yt-dlp: MISSING — ${YTDLP} (${origin}) does not run: ${err:-no version printed}"
+ log " every download will fail until it does; see RUNNING_IN_DOCKER.md"
+ fi
}
# ---------------------------------------------------------------------------
@@ -203,6 +271,7 @@ case "${APP}" in
editor)
fetch_model
update_ytdlp
+ ytdlp_line
log "corpus: ${TRANSCRIPTS_DIR}"
log "settings: ${SETTINGS_FILE}"
if [ "${TRANSCRIBER}" = "parakeet" ]; then
@@ -271,9 +340,21 @@ HTML
;;
homepage)
- # Corpus-independent, so this one IS baked into the image at build time.
- log "serving /repo/homepage/out on :3031"
- exec "$(serve_bin /repo/homepage)" /repo/homepage/out -l 3031 \
+ # Corpus-independent, so a build IS baked into the image — but that build
+ # has no /source mirror and none of the corpus's numbers (there is no corpus
+ # and no .git at image-build time). `archilyzer publish homepage --deploy
+ # --to local` writes a real one into the builds volume; once that directory
+ # is non-empty it is what is served. Chosen at boot: restart this service
+ # after the FIRST local deploy (later ones are served as they land — serve
+ # reads from disk per request).
+ HOMEPAGE_DIR=/repo/homepage/out
+ if [ -n "$(ls -A "${HOMEPAGE_OUT}" 2>/dev/null)" ]; then
+ HOMEPAGE_DIR="${HOMEPAGE_OUT}"
+ log "serving the locally deployed homepage ${HOMEPAGE_DIR} on :3031"
+ else
+ log "nothing deployed at ${HOMEPAGE_OUT} — serving the image's baked homepage ${HOMEPAGE_DIR} on :3031"
+ fi
+ exec "$(serve_bin /repo/homepage)" "${HOMEPAGE_DIR}" -l 3031 \
-c /repo/homepage/serve.json --no-clipboard --no-port-switching
;;
diff --git a/docker/publish-site.sh b/docker/publish-site.sh
@@ -4,15 +4,19 @@
#
# docker compose exec editor /repo/docker/publish-site.sh <site-id>
#
-# Why this exists rather than a baked build: the export site is a static render
-# OF A CORPUS, and there is no corpus at image-build time — so the image ships
-# the code and this publishes the output. It is the same `archilyzer build site`
-# a host install runs; the only container-specific part is the copy at the end.
+# A wrapper over the three publish stages, nothing more (release 18):
#
-# The copy is a copy and not a symlink on purpose. `next build` REMOVES and
-# recreates export/out (docker/build-site.sh has the same note), so a symlink
-# there survives exactly one build, and a bind mount there breaks the build
-# outright — the rm fails on a busy mount point.
+# archilyzer publish index the shared index, stats, templates
+# archilyzer publish build <id> the site's bundle in the builds volume
+# archilyzer publish deploy <id> --to local copied into ARCHILYZER_SITE_OUT
+#
+# Each stage is a no-op when it is already fresh, so running this twice costs
+# seconds; each takes the publish lock, so it waits for (never races) a stage
+# the editor is running. Run them one at a time for more control — the same
+# commands, through `docker compose exec editor pnpm archilyzer publish …`.
+#
+# Why the export site is built at run time rather than baked: it is a static
+# render OF A CORPUS, and there is no corpus at image-build time.
set -euo pipefail
SITE_ID="${1:-${SITE_ID:-}}"
@@ -28,39 +32,23 @@ if [ -z "${SITE_ID}" ]; then
exit 2
fi
-export SITE_ID
-
# A PRIVATE site (site.json `audience: "private"`) is never deployed, and the
-# volume this script fills is what the `site` service serves: refused before
-# the build, and again over the built corpus.json below (lib/builtExport.ts).
-# Building it without publishing is `archilyzer build site <id>`.
+# volume this fills is what the `site` service serves. The deploy stage refuses
+# it too (over the built corpus.json); refusing here first saves the build.
SITE_JSON="${SITES_DIR:-${TRANSCRIPTS_DIR:-/data/transcripts}/sites}/${SITE_ID}/site.json"
-refuse_private() {
- echo "[publish-site] REFUSED — site '${SITE_ID}' is private (audience: private): it is built for reading on this machine and is never deployed. Nothing was published to ${SITE_OUT}. Build it without publishing: pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts build site ${SITE_ID}" >&2
- exit 1
-}
if grep -Eq '"audience"[[:space:]]*:[[:space:]]*"private"' "${SITE_JSON}" 2>/dev/null; then
- refuse_private
+ echo "[publish-site] REFUSED — site '${SITE_ID}' is private (audience: private): it is built for reading on this machine and is never deployed. Nothing was published to ${SITE_OUT}. Build it without publishing: pnpm archilyzer publish build ${SITE_ID}" >&2
+ exit 1
fi
cd /repo
-echo "[publish-site] building '${SITE_ID}'"
-# The full build: the data phase (shared LMDB index + stats + chart
-# templates), then compose:site for THIS site, then next build.
-pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts build site "${SITE_ID}"
-
-[ -d /repo/export/out ] || { echo "[publish-site] no export/out after build" >&2; exit 1; }
-if grep -Eq '"audience"[[:space:]]*:[[:space:]]*"private"' /repo/export/out/corpus.json 2>/dev/null; then
- refuse_private
-fi
+archilyzer() {
+ pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts "$@"
+}
-echo "[publish-site] publishing -> ${SITE_OUT}"
-mkdir -p "${SITE_OUT}"
-# Empty the directory's CONTENTS, never the directory: it is a volume mount
-# point, and removing it fails.
-find "${SITE_OUT}" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
-cp -a /repo/export/out/. "${SITE_OUT}/"
+archilyzer publish index
+archilyzer publish build "${SITE_ID}"
+archilyzer publish deploy "${SITE_ID}" --to local
-echo "[publish-site] done — $(find "${SITE_OUT}" -type f | wc -l) files"
-echo "[publish-site] the site service serves it immediately; no restart needed."
+echo "[publish-site] the site service serves ${SITE_OUT} immediately; no restart needed."
diff --git a/docker/yt-dlp-from-source.sh b/docker/yt-dlp-from-source.sh
@@ -0,0 +1,20 @@
+#!/bin/sh
+# yt-dlp run from a SOURCE tree — the image's hook for substituting your own
+# yt-dlp (a patched checkout) without rebuilding the image.
+#
+# YTDLP_BIN=/usr/local/bin/yt-dlp-from-source (.env)
+# docker compose -f docker-compose.yml -f docker-compose.ytdlp.yml up -d
+#
+# docker-compose.ytdlp.yml mounts YTDLP_SOURCE_HOST_DIR (the checkout: the
+# directory that holds the `yt_dlp/` package) read-only at /opt/yt-dlp-src.
+# YTDLP_SOURCE_DIR moves that mount point. The python is the image's, with
+# yt-dlp's optional modules installed beside it (the Dockerfile's apt line).
+#
+# Linked into /usr/local/bin by the Dockerfile; RUNNING_IN_DOCKER.md,
+# "Substituting yt-dlp".
+src="${YTDLP_SOURCE_DIR:-/opt/yt-dlp-src}"
+if [ ! -f "${src}/yt_dlp/__init__.py" ]; then
+ echo "yt-dlp-from-source: no yt_dlp package in ${src} — mount a yt-dlp checkout there (docker-compose.ytdlp.yml, YTDLP_SOURCE_HOST_DIR)" >&2
+ exit 127
+fi
+PYTHONPATH="${src}${PYTHONPATH:+:${PYTHONPATH}}" exec python3 -m yt_dlp "$@"
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,6 +1,9 @@
# Changelog
## [Unreleased]
+- **Substitute your own yt-dlp in Docker.** Point `YTDLP_BIN` at a zipapp you built, or set `YTDLP_SOURCE_HOST_DIR` to a yt-dlp checkout and start with `docker-compose.ytdlp.yml`: the image runs it with its own python, and nothing is rebuilt. Every editor boot logs `yt-dlp: <path> <version> (image|override)` (`MISSING` when it does not run; the editor still starts), and `YTDLP_AUTO_UPDATE` updates the image's yt-dlp only, warning instead of touching yours.
+- **The Docker image can publish.** It carries python, `pipx` and a pinned `git-filter-repo`, so the homepage's `/source` mirror builds in the container; `docker-compose.source.yml` mounts your repository read-only for it, and the scrub rules and denylist live in the config volume (`/data/config/archilyzer`). Cloudflare and R2 credentials come from `.env`. Run publish commands with `docker compose exec editor pnpm archilyzer …`, not `run --rm`. The `homepage` service serves a local deploy from the builds volume once there is one. RUNNING_IN_DOCKER.md has a Windows checklist.
+- **`archilyzer doctor` checks what a publish needs.** Which yt-dlp runs (the image's, the host's or an override, and whether it runs), whether the Cloudflare token and the R2 keys are set (never their values; R2 only when a bucket is configured), free space for the site bundles, the repository the source mirror reads, and the private config dir.
- **A cited moment at the very end of a recording prepares.** Prepare evidence media cuts a clip whose padding runs past the recording's end at the end (the recording's duration from its metadata), where it found no media for the padded span; a span that starts past the end is still refused. report-to-video keeps its strict rule.
- **Exporting a changed report records a new revision of it.** `reports export` (and **Export reports** on a site's Reports tab, and the end of a prepare) commits a revision to the report's own git history, `sites/<site>/reports/<id>/history-git/`, whenever its `report.json` changed since the last one: the `report.json`, its Markdown export and the checksums of every export file, with a message of `Revision N` and a summary of the change. A re-export of an unchanged report records nothing. The commits carry the site's name and a `noreply@<site>.invalid` address with dates in UTC, never your git name, email or time zone. The Reports tab shows each report's revision, its commit and the last change under **Exports**, and the site's next build publishes the history. Add `history-git/` to the corpus repository's `.gitignore`.
- **archive.org files come over BitTorrent when possible, else straight from archive.org — never through yt-dlp.** The chosen file of an archive.org import is fetched from the item's own torrent (`<identifier>_archive.torrent`, which lists archive.org as a web seed, so other peers take load off archive.org) with aria2c, only that file of the item, and seeded afterwards for 10 minutes or to a ratio of 1, whichever comes first; the log shows "torrent: <file> (n of m pieces, peers p, web seed yes)" and "seeding 10 min…". With no aria2c, a torrent that does not carry the file, or no progress for 5 minutes, it is downloaded directly from `archive.org/download/…` instead (resumable, backing off on 429/503), and the log says "fell back to direct download: <reason>". Every file is checked against archive.org's sha1/md5: a mismatch is downloaded once more directly, a second one fails the record. The record is written from the item's metadata: `metadata.info.json` with the file's page, the canonical id, the duration ffprobe measures and archive.org's playable copies of the file, the `archiveorg.json` provenance (a mirror's original title, date and uploader), and `audio.<fmt>` — an audio file already in the channel's format is used as is, anything else goes through the app's audio extraction, a video kept in the saved-video store when the channel keeps sources. An .avi/.mpeg/.flac/.wav original is fetched as archive.org's mp4 or mp3 of it. aria2c runs in its own process group: cancelling the job stops it and everything it started, and it stops itself if the editor exits. New settings block `archiveOrg` (`torrent`, `seedMinutes`, `seedRatio`, `stallMinutes`, `maxPeers`, `maxDownloadKiBps`, `maxUploadKiBps`), `ARIA2C_BIN`, an aria2c row in `archilyzer doctor`, and `aria2` in the runtime Docker images.
diff --git a/plans/release-18.md b/plans/release-18.md
@@ -353,6 +353,173 @@ Probe = { status|null; generatedAt?; cfCacheStatus?; age?; cacheControl?; error?
(Each slice adds a "### Slice <X>, as shipped" section here, before "## Rollout".)
+### Slice S5, as shipped — the image half: the container can publish, yt-dlp can be substituted, the doctor checks it (2026-10-06)
+
+Branch `r18/docker-publish` off `r18/integration` `ce66f2d3`, worktree `~/Projects/r18-docker-publish`
+(editor 7001, test 7011, export 7010), one Opus implementer. Scratch files `s5-*` in the job's `tmp`. The
+plan is "Docker (a)–(g)" and "`archilyzer doctor` adds" above. This is the IMAGE HALF: S1 and S2 had not
+merged, so the doctor's `wrangler`, `publish-lock` and `index-stamp` checks, the `WRANGLER_BIN` /
+`E2E_LIVE_CHECK` declarations and the publish-stage smoke are the second half (below, "Left").
+
+**What was found before building.**
+- `ARCHILYZER_CONFIG_DIR` is already honoured by `getPaths()` (`paths.ts:218`), and `FILTER_REPO_PIPX_SPEC`
+ (`git-filter-repo==2.47.0`) already existed in `source.ts:113` — the drift test pins the Dockerfile to it.
+- `source.ts` reads the repository with `git --git-dir=<repo>`, so `ARCHILYZER_SOURCE_REPO` must name a git
+ DIR (the host's common dir), not a work tree — which is what the overlay mounts.
+- `git filter-repo --version` prints the script's hash (`a40bce548d2c`), never `2.47.0`; the version is
+ read from `pipx list` (`git-filter-repo 2.47.0`).
+- `env_file: .env` (x-app) already passes every key of `.env` to every app; `x-app-env` sets none of the
+ credentials, so nothing overrides them. Verified by reading the merged `docker compose config`.
+- Debian's `python3-pycryptodome` installs as `Cryptodome` (yt-dlp tries it first) and bookworm's
+ `python3-websockets` is 10.4 — below what yt-dlp's websockets handler wants, so a from-source yt-dlp
+ runs without that handler (optional; only some live-stream extractors use it).
+
+**What it does.**
+- **Dockerfile.** runtime-base and runtime-cuda (which repeats it) add `python3 python3-venv pipx` and
+ yt-dlp's optional modules; `PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install
+ git-filter-repo==2.47.0`; `ARCHILYZER_IMAGE_YTDLP=/usr/local/bin/yt-dlp` beside `YTDLP_BIN`;
+ `/usr/local/bin/yt-dlp-from-source` → `docker/yt-dlp-from-source.sh` (refuses with a sentence and exit
+ 127 when no `yt_dlp/` package is mounted). `ARCHILYZER_COMMIT` / `ARCHILYZER_BRANCH` build args become
+ ENV as the LAST layer of each of the three targets, so a new commit re-runs one ENV layer. The three
+ targets and the glibc ordering are untouched; wrangler is not installed globally.
+- **Entrypoint.** Every editor boot prints `yt-dlp: <path> <version> (image|override)` after the
+ optional self-update — `MISSING` (with the first stderr line) when it is not there or does not run; the
+ editor still starts. image vs override compares the two paths after `readlink -f`. `update_ytdlp` skips
+ an override with a two-line warning. `/data/source.git` (or `ARCHILYZER_SOURCE_REPO`) is added to git's
+ `safe.directory` once — only when absent, so a restarted container does not pile up entries.
+ `ARCHILYZER_CONFIG_DIR` is made (700, empty) when absent. The `homepage` service serves
+ `ARCHILYZER_HOMEPAGE_OUT` (`/data/builds/homepage`) when it is non-empty, else the baked
+ `homepage/out` — chosen at boot. `exec "$@"` stays.
+- **Compose.** `x-app-env` gains `ARCHILYZER_CONFIG_DIR=/data/config/archilyzer` and
+ `ARCHILYZER_HOMEPAGE_OUT`; `x-app.build.args` passes the two build facts from the shell; the
+ `homepage` service mounts `builds`. New overlays: `docker-compose.source.yml` (`ARCHILYZER_SOURCE_HOST_DIR`,
+ default `./.git`, read-only at `/data/source.git`; sets `ARCHILYZER_SOURCE_REPO`) and
+ `docker-compose.ytdlp.yml` (`${YTDLP_SOURCE_HOST_DIR:?…}` read-only at `/opt/yt-dlp-src`; sets
+ `YTDLP_BIN=/usr/local/bin/yt-dlp-from-source`).
+- **`source.ts`.** `sourceRepoFor(ctx, cwd)` — `ARCHILYZER_SOURCE_REPO` first, then the checkout's common
+ dir — replaces both `commonDir` call sites (the publish and `publishedSourceProblem`). A variable naming a
+ path that is not there is a `SourceRefusal` naming it (never the "no repository" sentence).
+- **`docker/publish-site.sh`** is a wrapper: usage, the early private refusal, then `publish index`,
+ `publish build <id>`, `publish deploy <id> --to local` (S1's CLI rows, by name).
+- **`envVars.ts`** (ENVIRONMENT.md regenerated): `CLOUDFLARE_API_TOKEN`, `ARCHILYZER_SOURCE_REPO`,
+ `YTDLP_SOURCE_HOST_DIR`, `YTDLP_SOURCE_DIR`, `YTDLP_AUTO_UPDATE`, `XDG_CONFIG_HOME` (runtime);
+ `ARCHILYZER_HOMEPAGE_OUT`, `ARCHILYZER_IMAGE_YTDLP`, `ARCHILYZER_COMMIT`, `ARCHILYZER_BRANCH`,
+ `ARCHILYZER_SOURCE_HOST_DIR` (docker); the R2/account rows and `ARCHILYZER_CONFIG_DIR` say where they
+ come from in Docker. Exported for S1's stamps: `IMAGE_COMMIT_ENV`, `IMAGE_BRANCH_ENV` and
+ `imageBuildFacts(env)` → `{commit, branch}` (an empty baked value is null).
+- **Doctor.** New section `downloader` (`yt-dlp`: path, version by exit status, `image` | `host` |
+ `override`; an override that does not run, or one beside `YTDLP_AUTO_UPDATE`, warns); new section
+ `publish` (`cloudflare-auth`: the token SET, or wrangler's login config by PATH, never read; warns only
+ when a site names a `cloudflareProject`; `r2-keys`: only with `archiveStorage.bucket`, names the unset
+ keys; `export-builds`: writable, free ≥ 1.5× the `<id>/out` bundles, a missing dir is a note);
+ `source publish` gains `source-repo` (`ARCHILYZER_SOURCE_REPO` naming nothing FAILS — the publish
+ refuses; no repository is a note, a warning once the operator's files exist; main's commit when it
+ reads) and `config-dir` (exists, entries counted, writable). No value is printed; still read-only.
+- **RUNNING_IN_DOCKER.md**: "Publish the archive" (stages in the container; `exec`, never `run --rm`;
+ Cloudflare from `.env`; the homepage and its `/source` mount, the config volume), "Substituting yt-dlp",
+ "Two build runners, and the container has one" (replaces the fallback section), the Windows checklist,
+ what is in the image, troubleshooting. `.env.example` names every new variable (not `E2E_LIVE_CHECK`:
+ a test knob does not belong in a real instance's `.env`).
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `060927fb` | `publish:` `sourceRepoFor` — `ARCHILYZER_SOURCE_REPO` first; a missing path refuses by name; 1 test |
+| `f0dfa39a` | `docker:` the image (python, pipx, filter-repo, the yt-dlp hook, build facts); entrypoint; compose + two overlays; `publish-site.sh` wrapper; envVars + ENVIRONMENT.md; 2 drift tests in `buildImage.test.ts` |
+| `e1a3b49b` | `doctor:` `downloader/yt-dlp`, `publish/{cloudflare-auth,r2-keys,export-builds}`, `source publish/{source-repo,config-dir}`; 5 tests |
+| `90722346` | `docker:` `.env.example` |
+| `32c16698` | `docker:` the entrypoint makes the config dir |
+| `ed5b5db8` | `docs:` RUNNING_IN_DOCKER.md |
+| this one | `plans:` this section; the editor changelog |
+
+#### Gates (logs `$T/s5-*.log`)
+
+- **tsc** (all workspaces) clean at every commit (91 s at `ed5b5db8`).
+- **common:** **3,169/3,169** (new: `doctor.test.ts` +5, `buildImage.test.ts` +2, `source.test.ts` +1).
+ **Editor unit:** 142/142. **mcp:** 289/289. **test:scripts:** 595 passed, 1 failed, 3 skipped (599) —
+ `queue-lock.test.mjs` "prints a banner naming the holder while waiting" at a load average of 24, with
+ the editor build running; the file alone afterwards: 11/11. The slice touches nothing under `scripts/`.
+- **Build:** `pnpm --filter editor exec next build` exit 0, 253 s.
+- **Image:** `docker buildx build --target runtime` in a builder capped at 8 GB (`--driver-opt memory=8g
+ memory-swap=8g`), `WHISPER_BUILD_JOBS=4`: exit 0 in 280 s from an empty builder cache, 241 s for the
+ rebuild after the doctor commit. `archilyzer:r18smoke` is **1.76 GB** (`docker image inspect .Size`).
+ **Only `--target runtime` was built.** `runtime-vulkan` (trixie apt, the same pipx install) and
+ `runtime-cuda` (ubuntu 24.04: `pipx`, `python3-pycryptodome`, `python3-brotli` from universe) carry the
+ same package list unverified by a build — each to be built once, capped, before the rollout.
+- **Compose smoke** (`-p r18smoke`, the `channel-with-counts` e2e fixture + `sites/testsite` copied into
+ `$T/s5-corpus` and bind-mounted over the corpus volume, `ARCHILYZER_FETCH_MODEL=none`,
+ `ARCHILYZER_IDLE_BOOT=1`, the editor alone): the boot log shows `yt-dlp: /usr/local/bin/yt-dlp
+ 2026.08.19 (image)`; `exec editor pnpm archilyzer doctor` lists `downloader/yt-dlp` ok (image),
+ `publish/cloudflare-auth` and `export-builds` (notes), `filter-repo` ok (`git filter-repo a40bce548d2c`),
+ `source-repo` and `config-dir` (exit 1 only for the model the smoke skipped); `python3 -c "import
+ yt_dlp"` exit 1 and `yt-dlp-from-source --version` exit 127 with its sentence; `pipx list` →
+ `git-filter-repo 2.47.0`; `ARCHILYZER_COMMIT`/`BRANCH` baked. With `docker-compose.ytdlp.yml` over a
+ two-file `yt_dlp/` package and `YTDLP_AUTO_UPDATE=1`: the entrypoint's skip warning, `yt-dlp:
+ /usr/local/bin/yt-dlp-from-source 2099.01.01.s5-smoke (override)`, the wrapper exit 0, the doctor's
+ `yt-dlp` WARN. `safe.directory` has one entry after a `docker restart`. With
+ `docker-compose.source.yml` over a throwaway repo in `$T`: `source-repo` ok with its main, `rev-parse`
+ as root works, the mount is read-only. The `homepage` service serves the baked build, then the builds
+ volume's after a file lands there and it restarts. `down -v` after.
+- **e2e:** none for this slice. **Numbers tool:** none. **Publish-stage smoke** (`publish
+ index/build/deploy` in the container): not run — S1's CLI rows are not on this branch; it is the
+ parent's after S1/S2 merge.
+
+**Deviations from the plan, one sentence each.**
+- `ARCHILYZER_HOMEPAGE_OUT` and `ARCHILYZER_SOURCE_HOST_DIR` are new names the plan did not list: the
+ first is where `deploy-homepage --to local` writes (S1 must read it), the second lets a worktree mount
+ the primary's `.git`.
+- `python3-venv` is installed beside `pipx` (pipx makes a venv); the plan's package list omitted it.
+- The build facts are ENV in each final target rather than once in runtime-base, so a commit does not
+ invalidate the Vulkan apt layer.
+- `config-dir` not writable is a note, not a warning: the publish only reads the rules.
+- The source-repo "homepage policy on" grade waits for S3's `settings.publish.homepage`; today the
+ warning keys off the operator's files existing (the doctor's existing "intends" signal).
+- `E2E_LIVE_CHECK` and `WRANGLER_BIN` are not declared yet: nothing on this branch reads them, and the
+ envVars test refuses a declaration nothing names (S2 adds the reads).
+
+**Left for the second half (after S1/S2 merge).**
+- Doctor `wrangler` (the binary from `wranglerBin(paths)`, its major against `WRANGLER_MAJOR`),
+ `publish-lock` (a dead pid in `.publish.lock`, via `stageLock.ts`), `index-stamp` (age; sites built from
+ an older stamp, via `stamps.ts`); `export-builds` to read `built.json` `bytes` instead of walking.
+- `WRANGLER_BIN` / `E2E_LIVE_CHECK` in `envVars.ts` (whichever of S2/S5 lands second).
+- The publish-stage smoke in the container (rollout's shape: `publish index && publish build <fixture> &&
+ publish deploy <fixture> --preview smoke`, no token → refused before wrangler, a bogus token → refused by
+ Cloudflare), plus `exec editor pnpm archilyzer source publish --check` over a throwaway repo with
+ throwaway rules copied into `/data/config/archilyzer` (the container's mirror over the read-only,
+ foreign-owned mount, which the doctor's `rev-parse` alone does not prove).
+- The envVars dedupe at the merges: S2 also declares `CLOUDFLARE_API_TOKEN` and `ARCHILYZER_HOMEPAGE_OUT`
+ (one row per name, `readBy` unioned); S1's `stamps.ts` imports `imageBuildFacts` from `lib/envVars`
+ instead of its own. `cloudflare-auth` grades through S2's `cloudflareCredentialProblem` (which accepts
+ `CLOUDFLARE_API_KEY` + `CLOUDFLARE_EMAIL`).
+- Building `runtime-vulkan` and `runtime-cuda` once (above).
+
+**Found and left.**
+- The image has no gitleaks and no stagit: a source publish from the container skips the secret scan
+ (with its WARNING; the literal audit still runs) and has no history pages. RUNNING_IN_DOCKER.md says so;
+ a pinned gitleaks in the image is a follow-up.
+- `.env` (now carrying the Cloudflare token and the R2 keys) reaches `site`, `homepage` and `umtool`
+ through the shared `env_file`, as it always did; nothing serves or prints it. An editor-only credentials
+ file is an option, not done.
+- The shared tool probe (`toolProbe.mjs`) counts any output from a failing `--version` as presence, so the
+ `tools/yt-dlp` row reads `ok` with the wrapper's sentence as its "version" when no checkout is mounted;
+ the new `downloader/yt-dlp` row asks by exit status and is the honest one. Not changed: the probe is
+ shared with `umtool doctor`.
+- `docker images` reported the previous `archilyzer:local` (6 weeks old) at 7.3 GB; this build is
+ 1.76 GB. Not investigated.
+
+**Review fixes** (review `SHIP AFTER FIXES`, no highs; the four asked for now)
+
+| Commit | Fix |
+|---|---|
+| `ba83cbff` | `docker-compose.source.yml`: long bind syntax, `create_host_path: false` — a missing host `.git` now fails `up` ("bind source path does not exist: …", verified) instead of becoming an empty root-owned dir |
+| `fb1a1f24` | `YTDLP_AUTO_UPDATE`: the doctor reads it with the entrypoint's exact-match rule (`1`, `true`, `yes`, `on` as written; `TRUE` is off in both), +1 test loop; RUNNING_IN_DOCKER.md states the rule |
+| `07bc2395` | RUNNING_IN_DOCKER.md: no gitleaks or stagit in the image — the container's source publish skips the secret scan (with its warning) and the history pages; pinned gitleaks a follow-up |
+| this one | the record: only `--target runtime` was built (vulkan and cuda unverified, left for the rollout); the second half's added items; found and left |
+
+Re-run after the fixes at `07bc2395`: tsc (all workspaces) clean; `doctor`, `buildImage`, `source` and
+`envVars` tests **61/61** (`$T/s5-fix-tests.log`).
+
## Rollout
(Steps 1–7 above; "### As it went" is written as the rollout runs.)
