commit cee23eea84b8a1fbec291fa4038574dcf50f8982
parent b2c76161e1120c771fbea7cb8cf37e905aba7171
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 6 Oct 2026 08:33:09 -0400
docs: RUNNING_IN_DOCKER — publishing runs in the container (exec, not run --rm), Cloudflare from .env, the homepage and its /source mount, substituting yt-dlp, the two runners, a Windows checklist
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
| M | RUNNING_IN_DOCKER.md | | | 209 | ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------- |
1 file changed, 187 insertions(+), 22 deletions(-)
diff --git a/RUNNING_IN_DOCKER.md b/RUNNING_IN_DOCKER.md
@@ -5,10 +5,12 @@ One command stands up a working archive server: the editor, the tools it drives
Vulkan image), and a reverse proxy that is the only thing on the box with an open
port.
-> This document is about **running the apps**. Fanning per-site *export builds* out
-> across containers is a different thing entirely — it uses `Dockerfile.build`, and
-> it is in [PUBLISH.md](PUBLISH.md#building-every-site-in-containers). Neither
-> affects the other.
+> This document is about **running the apps** — publishing included: every publish
+> stage (index, build, deploy) runs inside the editor's container. Fanning per-site
+> *export builds* out across containers is a different thing — the opt-in docker
+> runner of a HOST install, from `Dockerfile.build`, in
+> [PUBLISH.md](PUBLISH.md#building-every-site-in-containers). Neither affects the
+> other.
---
@@ -194,7 +196,10 @@ it accurately.
### Publish the archive
The site server serves a *static build of your corpus*, which does not exist
-until you make one:
+until you make one. Publishing is a set of **stages** — update the index, build a
+site, deploy it — and every one of them runs **inside the editor's container**:
+the queued jobs the editor starts (Sites → Publish, the publish lane) and the
+commands below alike. The quick way, for the local `site` service:
```sh
docker compose exec editor /repo/docker/publish-site.sh <site-id>
@@ -202,9 +207,88 @@ docker compose --profile site up -d
```
Create the site itself in the editor first (**Sites → New**); run the script with
-no argument to list the ones you have. It runs the same `pnpm --filter export run
-build` a host install runs, then copies the output into the volume the `site`
-service serves. No restart needed afterwards.
+no argument to list the ones you have. The script is three stages in a row, and
+you can run them yourself:
+
+```sh
+docker compose exec editor pnpm archilyzer publish index
+docker compose exec editor pnpm archilyzer publish build <site-id>
+docker compose exec editor pnpm archilyzer publish deploy <site-id> --to local
+docker compose exec editor pnpm archilyzer publish status
+```
+
+A stage that is already fresh does nothing, so running one twice is cheap. A
+local deploy copies the built bundle into the volume the `site` service serves;
+no restart needed afterwards. A **private** site (`audience: "private"`) is
+built but never deployed, locally or anywhere else.
+
+**`exec`, never `run --rm`.** `docker compose run --rm editor …` starts a SECOND
+container with its own copy of the image's `export/public` and a second writer on
+the index, and the publish lock (which keeps a stage you start from colliding
+with one the editor is running) cannot see across containers. `exec` runs in the
+editor's own container, beside its jobs, under the same lock. `pnpm ops publish`
+from the host goes through the editor too.
+
+#### Deploying to Cloudflare from the container
+
+The same stages deploy to Cloudflare Pages. The container has no browser for
+`wrangler login`, so it authenticates with a token from `.env`, which the editor
+reads and every stage it runs inherits:
+
+```sh
+# .env
+CLOUDFLARE_API_TOKEN=... # an API token with "Cloudflare Pages: Edit"
+CLOUDFLARE_ACCOUNT_ID=...
+# only when Settings → archive storage names an R2 bucket for oversize archives:
+R2_ACCESS_KEY_ID=...
+R2_SECRET_ACCESS_KEY=...
+```
+
+```sh
+docker compose up -d # picks up .env
+docker compose exec editor pnpm archilyzer publish deploy <site-id> --preview <branch>
+docker compose exec editor pnpm archilyzer publish deploy <site-id> # production
+```
+
+Nothing is fetched at deploy time: wrangler is pinned in the workspace the image
+ships. With no token a deploy is refused before wrangler runs, with a sentence
+saying so; `archilyzer doctor` reports whether each credential is **set** (never
+its value). Every deploy is checked live afterwards. PUBLISH.md has the whole
+publish flow; this is only what is different in a container.
+
+#### The homepage, and its /source mirror
+
+The image carries a build of the project's own homepage, served by the
+`homepage` service. A real one — the corpus's numbers, the `/source` mirror of
+this repo — is a publish like any other:
+
+```sh
+docker compose exec editor pnpm archilyzer publish homepage --deploy --to local
+docker compose --profile homepage restart homepage # once, after the first
+```
+
+The `homepage` service serves the local deploy (`/data/builds/homepage`) once
+there is one, else the baked build — chosen at boot, hence the one restart.
+
+The `/source` mirror needs two things the image deliberately lacks:
+
+- **The repository.** The image has no `.git`. `docker-compose.source.yml` mounts
+ the host's git directory read-only at `/data/source.git` and points
+ `ARCHILYZER_SOURCE_REPO` at it (default `./.git`; from a git worktree set
+ `ARCHILYZER_SOURCE_HOST_DIR` to the primary checkout's `.git`). Without it the
+ homepage builds with an empty `/source` page.
+- **The operator's scrub rules and denylist.** They live in the config volume —
+ `ARCHILYZER_CONFIG_DIR` is `/data/config/archilyzer` in the container, made
+ (mode 700, empty) on the first boot — and are never printed by anything:
+
+```sh
+docker compose -f docker-compose.yml -f docker-compose.source.yml up -d
+docker compose cp ~/.config/archilyzer/source-scrub.txt editor:/data/config/archilyzer/
+docker compose cp ~/.config/archilyzer/source-denylist.txt editor:/data/config/archilyzer/
+docker compose exec editor chmod 600 /data/config/archilyzer/source-scrub.txt /data/config/archilyzer/source-denylist.txt
+```
+
+`git-filter-repo` (pinned) is in the image.
### Model choice
@@ -230,6 +314,49 @@ boot self-updates it. Or, once:
docker compose exec editor yt-dlp -U
```
+Every editor boot logs which yt-dlp it will run, and whether it runs:
+
+```
+[entrypoint] yt-dlp: /usr/local/bin/yt-dlp 2026.09.30 (image)
+```
+
+### Substituting yt-dlp
+
+To run your own yt-dlp — a patched build, say — set `YTDLP_BIN`. It is a swap at
+run time: nothing is rebuilt, and the image's release binary stays where it is
+(`ARCHILYZER_IMAGE_YTDLP`, `/usr/local/bin/yt-dlp`). Two ways:
+
+**A source checkout**, run with the image's python (yt-dlp's optional modules
+are installed beside it):
+
+```sh
+# .env
+YTDLP_SOURCE_HOST_DIR=/home/you/yt-dlp-patched # the directory holding yt_dlp/
+```
+
+```sh
+docker compose -f docker-compose.yml -f docker-compose.ytdlp.yml up -d
+```
+
+The overlay mounts the checkout read-only at `/opt/yt-dlp-src` and sets
+`YTDLP_BIN=/usr/local/bin/yt-dlp-from-source`, a wrapper baked into the image.
+
+**A zipapp** built on the host (`make yt-dlp` in the checkout), copied into the
+config volume:
+
+```sh
+docker compose exec editor mkdir -p /data/config/bin
+docker compose cp ./yt-dlp editor:/data/config/bin/yt-dlp
+docker compose exec editor chmod 755 /data/config/bin/yt-dlp
+# .env
+YTDLP_BIN=/data/config/bin/yt-dlp
+```
+
+Either way the boot line says `(override)`, and `YTDLP_AUTO_UPDATE` leaves an
+override alone with a warning — update it where you build it. A substitute that
+does not run is reported as `yt-dlp: MISSING — … does not run: …` (the editor
+still starts), and by `archilyzer doctor`.
+
### Driving the editor without a browser
Every editor gesture is a server action, which is fine for a person and hostile
@@ -511,6 +638,8 @@ The `site` profile does not need the mount: an export build never reads `data/`.
docker compose logs -f editor
docker compose exec editor bash # a shell in the image
docker compose exec editor yt-dlp --version
+docker compose exec editor pnpm archilyzer doctor # what this container can do
+docker compose exec editor pnpm archilyzer publish status
docker compose restart editor
docker compose down # stop; volumes survive
docker compose down -v # stop AND DELETE the corpus
@@ -523,7 +652,10 @@ docker compose down -v # stop AND DELETE the corp
**Baked in:** node + the installed workspace, the built editor / umtool / homepage,
`yt-dlp`, a JavaScript runtime for it (`deno` — without one yt-dlp warns and
silently loses formats), `ffmpeg`/`ffprobe`, `whisper-cli` (statically linked),
-`zip`/`tar`/`xz`/`gzip`, `rsync`, `git`, `curl`, `ps`. The Vulkan image adds
+`zip`/`tar`/`xz`/`gzip`, `rsync`, `git`, `curl`, `ps`; `python3` with yt-dlp's
+optional modules (for a substituted yt-dlp), `pipx` and a pinned `git-filter-repo`
+(the homepage's source mirror), and — in the workspace's `node_modules` — a
+pinned `wrangler`, so a deploy fetches nothing. The Vulkan image adds
`parakeet-cli`, the Mesa Vulkan drivers and `vulkaninfo`.
**Not baked, on purpose:**
@@ -532,7 +664,14 @@ silently loses formats), `ffmpeg`/`ffprobe`, `whisper-cli` (statically linked),
in the `corpus` volume.
- **whisper models.** 142 MB to 3 GB, and the choice is yours. Fetched on boot.
- **The export site build.** It is a static render *of a corpus*, and there is no
- corpus at image-build time. `docker/publish-site.sh` makes it at run time.
+ corpus at image-build time. The publish stages make it at run time.
+- **Credentials.** `CLOUDFLARE_API_TOKEN` and friends come from `.env` at run
+ time, never from the image.
+- **The repository.** No `.git`; `docker-compose.source.yml` mounts the host's
+ for the `/source` mirror. The build's commit and branch are baked as
+ `ARCHILYZER_COMMIT`/`ARCHILYZER_BRANCH` when the build passes them
+ (`ARCHILYZER_COMMIT=$(git rev-parse HEAD) ARCHILYZER_BRANCH=$(git branch
+ --show-current) docker compose build`) — the publish stamps record them.
- **ImageMagick with Pango, and `qrencode`.** Only `umtool/report-to-video/`
needs them, and rendering a report to video is a workstation task, not
something a server does. Run that part on a host checkout.
@@ -542,20 +681,21 @@ silently loses formats), `ffmpeg`/`ffprobe`, `whisper-cli` (statically linked),
Point `OLLAMA_URL` at a host ollama (`http://host.docker.internal:11434`) if you
want digests.
-### The multi-site build pipeline falls back inside a container
+### Two build runners, and the container has one
-The editor's **Build all sites** fans per-site export builds out across containers
-whenever a container engine answers (see [PUBLISH.md](PUBLISH.md#building-every-site-in-containers)). Inside a container there
-is no `docker` binary, so that path is unavailable. It already handles this — the
-build logs
+Every site build goes through one stage contract with two runners:
-```
-[notice] No container engine available — building sites serially on the host.
-```
+- **local** — the default everywhere, in a container or not: each stage is a child
+ process of the editor (or of the CLI you ran), one at a time.
+- **docker** — a HOST install's opt-in fan-out (`settings.publish.runner:
+ "docker"`, or `publish build all --runner docker`): one `Dockerfile.build`
+ container per site (see [PUBLISH.md](PUBLISH.md#building-every-site-in-containers)).
+ Both write the same bundles and stamps, so a deploy does not care which built it.
-and does the builds one after another in-process. Correct, just not parallel. **Do
-not** try to make docker-in-docker work for this; run the parallel pipeline from a
-host checkout if you need it.
+Inside the container there is no engine, and the docker runner **refuses** ("the
+docker runner needs an engine on this host") rather than pretend. The editor never
+gets the docker socket — **do not** mount it, and do not try to make
+docker-in-docker work; run the fan-out from a host checkout if you want it.
### It runs as root
@@ -582,6 +722,23 @@ Open http://localhost:8081. Keep the clone on the Windows filesystem — the bui
context is small, and the corpus lives in a Docker volume inside the WSL2 VM,
which is where the I/O actually happens.
+Publishing needs nothing else on Windows — Docker Desktop is the whole
+requirement. A checklist, in PowerShell from the clone:
+
+1. `copy .env.example .env`, then set in `.env`: `WORKER_TOKEN` (for `pnpm ops`
+ and agents), `CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_ACCOUNT_ID` (to deploy).
+2. `docker compose up -d --build`
+3. Open http://localhost:8081 → **Sites** → **Publish now**, or:
+ `docker compose exec editor pnpm archilyzer publish now`
+4. A local preview: `docker compose --profile site up -d site`, then
+ `docker compose exec editor pnpm archilyzer publish deploy <site-id> --to local`,
+ and open http://localhost:8080.
+5. `docker compose exec editor pnpm archilyzer doctor` — every check, including
+ whether the Cloudflare token is set (never its value).
+6. Optional, your own yt-dlp: set `YTDLP_SOURCE_HOST_DIR` (a Windows path such as
+ `C:\src\yt-dlp` works) and run
+ `docker compose -f docker-compose.yml -f docker-compose.ytdlp.yml up -d`.
+
You still want WSL2 directly if you intend to *develop* the project or run Claude
Code against it — see [README.md](README.md#claude-code-on-windows). For running
an archive, this is enough.
@@ -609,7 +766,15 @@ comparison above. The usual cause is a missing `/dev/dri` — a container withou
the render node still transcribes, just slowly.
**Downloads started failing on every channel.** Almost always a stale yt-dlp:
-`docker compose exec editor yt-dlp -U`.
+`docker compose exec editor yt-dlp -U`. Check `docker compose logs editor | grep
+yt-dlp:` first — `(override)` means `YTDLP_BIN` names your own build, which `-U`
+does not touch, and `MISSING` means it does not run at all (a from-source
+override with no checkout mounted, say).
+
+**A deploy is refused before wrangler runs.** `CLOUDFLARE_API_TOKEN` is not set in
+`.env`, or the editor was started before it was: `docker compose up -d` again
+(a restart does not re-read `.env`; `up` recreates the container).
+`archilyzer doctor`'s `cloudflare-auth` line says which.
**Port 8080 is already taken.** `SITE_HTTP_PORT=9080` in `.env`. Those variables
move the *outside* port; nothing inside the containers changes.
