commit 09bf26cdbb415b596079c50be392c8c4a9227e3d
parent 52419c40f4355a7346226648ade0417a4c662255
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 6 Oct 2026 13:34:09 -0400
docs: release 18's records — PUBLISH.md around the stages, AGENTS.md's runtime container, FACTS' The publish stages (+17 superseded markers), STATE's Now, the generated docs from their sources, the changelogs folded; plans: S6 as shipped, open questions settled, rulings, carried-over follow-ups
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
16 files changed, 929 insertions(+), 228 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
@@ -144,8 +144,9 @@ This tooling is on `main` as of 2026-08-20.
`docker compose up -d` stands up a working archive: the editor plus Caddy, with
`site`, `homepage` and `umtool` behind compose **profiles**. The root `Dockerfile`
-is new and is a THIRD Dockerfile — `Dockerfile.build` (per-site export build
-fan-out) and `Dockerfile.test` (sharded e2e) are untouched and unrelated.
+is new and is a THIRD Dockerfile — `Dockerfile.build` (the opt-in docker build
+runner's per-site containers, host only) and `Dockerfile.test` (sharded e2e) are
+untouched and unrelated.
Three runtime targets share one build: `runtime` (CPU whisper.cpp, the default),
`runtime-vulkan` (parakeet.cpp on Vulkan — AMD/Intel/NVIDIA, `/dev/dri` passed
@@ -199,7 +200,12 @@ Two things the image cannot bake, and the reasons matter:
- **The export site.** It is a static render OF a corpus, and there is no corpus at
image-build time. `docker/publish-site.sh` builds it at run time into the volume
- the `site` service serves.
+ the `site` service serves — a wrapper over three publish stages (`publish index`,
+ `publish build <id>`, `publish deploy <id> --to local`). Stages run IN the
+ container, through `docker compose exec editor …` (never `run --rm`: a second
+ container would not share the editor's lock or job registry), and every stage
+ takes the publish lock (`export/.export-builds/.publish.lock`, its host named by
+ `ARCHILYZER_HOST_ID`), so a CLI stage and the editor's never build at once.
- **whisper models.** 142 MB to 3 GB, and the choice is the operator's.
`docker/entrypoint.sh` fetches one on first boot — and seeds a `settings.json`
carrying one enabled worker for the image's engine: with no `workers` key (or no
@@ -214,9 +220,11 @@ the four auto-queue lane runners (`common/lib/idleBoot.ts`) — for pointing a f
container at a corpus whose stored policies would otherwise resume GPU-weeks of
work. The shutdown reaper stays armed regardless.
-Inside a container the multi-site build pipeline has no `docker` binary and falls
-back to the serial host build it already handles. Do not try to make
-docker-in-docker work.
+Inside a container the publish stages build locally, one at a time
+(`settings.publish.runner: "local"`, the default everywhere); the docker runner
+(`runner: "docker"` / `publish build all --runner docker`, `Dockerfile.build`'s
+per-site containers) is for a Linux host with an engine and REFUSES in a
+container. Do not try to make docker-in-docker work.
See [RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md) — which is about *running the
apps*, not [PUBLISH.md](PUBLISH.md), which is about *building and publishing sites*.
diff --git a/ENVIRONMENT.md b/ENVIRONMENT.md
@@ -141,7 +141,7 @@ The publish pipeline sets these for a process it spawns. Listed so a reader know
| Variable | Default | What it does | Read by |
|---|---|---|---|
-| `SITE_ID` | — | Which site a compose or an export build is for. `archilyzer build site <id>` sets it; `compose site` and `build site` fall back to it when no id is given. | common/bin/compose-site.ts, export/app/lib/site.ts |
+| `SITE_ID` | — | Which site a compose or an export build is for. The build stage (`archilyzer publish build <id>`) sets it for its children; `compose site`, `build site` and `deploy site` fall back to it when no id is given. | common/bin/compose-site.ts, export/app/lib/site.ts |
| `INSTANCE_MODE` | a site | `hub` makes the export build the hub. Set by `archilyzer build hub`. | export/app/lib/mode.ts, common/lib/archive/contract.ts |
| `BUILD_ARCHIVES` | on | `0` skips archive-zip generation for one build (`--skip-archives`). | common/bin/compose-site.ts, common/bin/build-archives.ts |
| `REPORTS_ALLOW_MISSING_MEDIA` | off | `1` lets a report citation whose evidence media was not prepared through compose (`--allow-missing-media`): its moment page renders without a clip. Off, compose fails with the list. | common/bin/compose-site.ts |
diff --git a/PUBLISH.md b/PUBLISH.md
@@ -1,12 +1,15 @@
# Publishing
-What gets published, how to build and deploy it, and how to keep a public archive
-cheap and safe. For *running* the apps in containers see
+What gets published, how the publish stages build and deploy it, and how to keep a
+public archive cheap and safe. For *running* the apps in containers see
[RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md); every environment variable named here
-is in [ENVIRONMENT.md](ENVIRONMENT.md).
+is in [ENVIRONMENT.md](ENVIRONMENT.md), every key of `settings.json` and `site.json`
+in [SETTINGS.md](SETTINGS.md) and [SITE.md](SITE.md).
- [What gets published](#what-gets-published)
+- [Publishing is stages](#publishing-is-stages)
- [The three ways to drive it](#the-three-ways-to-drive-it)
+- [The publish lane](#the-publish-lane)
- [Cloudflare Pages](#cloudflare-pages)
- [The source mirror (homepage)](#the-source-mirror-homepage)
- [Preview deployments](#preview-deployments)
@@ -22,21 +25,25 @@ is in [ENVIRONMENT.md](ENVIRONMENT.md).
Three static artefacts, each pre-rendered to plain HTML and JSON — no database, no
server-side code, servable by anything:
-| Artefact | What it is | Built into | Config |
+| Artefact | What it is | Its bundle | Config |
|---|---|---|---|
-| A **site** | The export app over the channels one site selects: search, transcripts, charts, downloads. One corpus can publish several. | `export/out` (docker mode: `export/.export-builds/<siteId>/out`) | `transcripts/sites/<id>/site.json` ([SITE.md](SITE.md)) |
-| The **hub** | The export app in hub mode: federated search across the family of sites. | `export/out` | `transcripts/sites/_homepage/homepage.json` |
-| The **homepage** | The project's own site (`homepage/`), with the source mirror, its raw tree, its history pages and the source tarball. | `homepage/out` | `~/.config/archilyzer/` (the source mirror's two operator files) |
+| A **site** | The export app over the channels one site selects: search, transcripts, charts, downloads. One corpus can publish several. | `export/.export-builds/<id>/out` | `transcripts/sites/<id>/site.json` ([SITE.md](SITE.md)) |
+| The **hub** | The export app in hub mode: federated search across the family of sites. | `export/.export-builds/_hub/out` | `transcripts/sites/_homepage/homepage.json` |
+| The **homepage** | The project's own site (`homepage/`), with the source mirror, its raw tree, its history pages and the source tarball. | `homepage/out` (its stamps in `export/.export-builds/_homepage/`) | `~/.config/archilyzer/` (the source mirror's two operator files) |
+
+**Each target keeps its own bundle, and a deploy ships that bundle and nothing
+else.** `export/out` is a link to the bundle built last, so `serve out` and anything
+else that read it keep working.
The hub and the homepage are two different Pages projects: the hub deploys to the
project `homepage.json` names (e.g. `archilyzer-hub`) and is refused `archilyzer`,
which is the homepage's.
-A site build has three steps, run in `export/`: the **data phase** (the LMDB index,
-the stats datasets and the chart templates — `archilyzer index`, `build stats`,
-`build templates`), **compose** (the site's slice of the shared index into
-`export/public`, plus its download archives — `archilyzer compose site <id>`) and
-`next build`. `--nodata` skips the data phase and reuses the last one's staging.
+**Every target is built from ONE index.** The update-index stage (`archilyzer
+publish index`) runs the data phase once — the LMDB index, the stats datasets and the
+chart templates — and writes the index stamp. A site's build is then **compose** (the
+site's slice of the shared index into `export/public`, plus its download archives —
+`archilyzer compose site <id>`) and `next build`; it has no data phase of its own.
A transcript record in the shared pages carries its other English caption tracks
(`altTracks`, with the transcript's own `track`) only where one's words differ from
@@ -48,42 +55,19 @@ reader switches to them. English VTTs are not published as subtitle tracks. The
index build after this reads them once (`Alternate tracks v1: N record(s) re-read.`),
for exactly the records that can hold one.
-## The three ways to drive it
-
-The editor's **/sites** page, `pnpm ops` (HTTP to a running editor, with its
-`WORKER_TOKEN`) and the `archilyzer` CLI (local, no editor needed) call the same
-entry points in `common/publish/build.ts`.
+### Reports and cited sites
-| To | Editor | `pnpm ops` | `pnpm archilyzer …` |
-|---|---|---|---|
-| Build one site | a site's **Publish** tab → *Build static export* | `build-site` | `build site <id> [--nodata] [--skip-archives] [--allow-missing-media]` |
-| Deploy the built site | *Deploy to production* / *Deploy preview* | `deploy-site` | `deploy site <id> [--preview <branch>]` |
-| Build, then deploy | *Build & deploy* | `build-deploy` | `build site <id>` then `deploy site <id>` |
-| Build every site | /sites → **Build all sites** | — | `build all [--skip-archives]` |
-| The hub | /sites → Hub → **Build hub** / **Deploy hub** | `build-hub`, `deploy-hub` | `build hub`, `deploy hub [--preview <branch>]` |
-| The homepage | /sites → Homepage → **Build homepage** (tick *Deploy after build*) / **Deploy homepage**, with an optional preview branch | `build-homepage` (`{"deploy":true}` to deploy after), `deploy-homepage` (`{"preview":"<branch>"}`) | `build homepage [--no-source]`, `deploy homepage [--preview <branch>]` |
-| The source mirror alone | — (every homepage build runs it) | — | `source publish [--force] [--check] [--keep-scratch]`, `source audit [<git dir>]` |
-| A site's report evidence media (then its exports) | a site's **Reports** tab → *Prepare evidence media* | `reports-prepare` (`{"siteId"}`) | `reports prepare <id>` |
-| A site's report exports (HTML, PDF, Markdown, evidence pack) | a site's **Reports** tab → *Export reports* | `reports-export` (`{"siteId", "reportId"?, "formats"?}`) | `reports export <id> [--report <rid>] [--formats html,pdf,md,zip]` |
-| A report from a /sweep report, an /ask answer or a report-to-video manifest, and a starter manifest from a report | — | — | `reports convert <sweep\|ask\|manifest> <in> --out <report.json> [--channels-dir <dir>]`, `reports to-manifest <report.json> --out <manifest.json>` |
-
-`pnpm archilyzer <command>` is the short form of
-`pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts <command>`;
-`pnpm archilyzer --help` lists every command, and `pnpm archilyzer doctor` checks that
-this machine has what a build needs. From `export/`, `pnpm run build` is
-`archilyzer build site` and `pnpm run deploy` is `archilyzer deploy site`; both take
-the site from `SITE_ID` when no id is given.
-
-A site with `reports` (site.json) has one more step, before its build and on the
-host: **reports prepare** cuts the evidence clip of every span its published reports
-cite — from the editor's clip windows, the saved-video store or a record's audio,
-fitted inside 1280×720 (H.264 crf 23, AAC; an audio span is an `.m4a`) — and copies
-the screenshot and media of every cited post, only those, into
-`.export-index/sites/<id>/report-media/`, with a manifest (`index.json`) of each
-moment's file, size, hash and duration. Nothing is fetched: a citation whose media
-is not on disk, a clip over 24 MiB or an invalid report is listed and fails the run
-(exit 1, or a failed job) — fetch the window or persist the video, capture the
-post, and run it again; what is already cut is reused.
+A site with `reports` (site.json) has one more step before its build. It is not a
+stage: run it yourself (or from the site's **Reports** tab), or that site's build
+fails on the first citation with no prepared media. **reports prepare** cuts the
+evidence clip of every span its published reports cite — from the editor's clip
+windows, the saved-video store or a record's audio, fitted inside 1280×720 (H.264
+crf 23, AAC; an audio span is an `.m4a`) — and copies the screenshot and media of
+every cited post, only those, into `.export-index/sites/<id>/report-media/`, with a
+manifest (`index.json`) of each moment's file, size, hash and duration. Nothing is
+fetched: a citation whose media is not on disk, a clip over 24 MiB or an invalid
+report is listed and fails the run (exit 1, or a failed job) — fetch the window or
+persist the video, capture the post, and run it again; what is already cut is reused.
When nothing is missing, prepare ends by **exporting** the reports (`reports export`
runs the same step alone): each published report is written, as compose would
@@ -134,8 +118,8 @@ citation, replacing any typed by hand. Compose fails, before it writes any of it
the list of everything wrong: an invalid report, a citation of a channel outside the
site or a post the site may not carry, a missing record, still or post, a quote below
60 %, and a citation whose media was not prepared, or was cut for a span the report no
-longer cites. `--allow-missing-media` (on `compose site` and `build site`) lets the
-last two through; their pages render without a clip.
+longer cites. `--allow-missing-media` (on `compose site`, `publish build` and `build
+site`) lets the last two through; their pages render without a clip.
A **cited** (report-only) site is one with search off (`site.json` `search: false`;
a legacy `publish: "cited"` reads the same): it publishes its reports and nothing else. Its
@@ -146,7 +130,7 @@ with `site.scope: "cited"` (spec 5), and an `llms.txt` and sitemap listing the r
and moment pages. After `next build`, the built `out/` is audited: a cited build that
holds anything but `_next/`, the reports, the moment pages, the cited media, the
shell's own pages and files, or a file over 25 MiB, or more than 20,000 files, fails
-the build, and every deploy path refuses it. A site switched to search off is refused at
+the build, and every deploy refuses it. A site switched to search off is refused at
deploy until it is built again. With no published reports it is an empty report index.
A cited site is not a searchable archive, so the family never lists it, whatever
@@ -168,11 +152,220 @@ service) takes a video or audio moment's directory name, `<start>-<end>` in seco
with decimals, for a file with an extension, and lists the directory instead of
serving its page.
-Deploy-only ships whatever is in `export/out`, which the basic build composes one
-site at a time into a single shared directory — so it **refuses, before starting a
-job, if `export/out` holds a build of another site** (or no build at all), naming the
-site to build first. A deploy queued behind another site's build re-checks when it
-starts. `build-deploy` cannot hit this: it builds.
+---
+
+## Publishing is stages
+
+Publishing is seven queueable **stages** driven by what is on disk
+(`common/publish/stages.ts`). Each asks a pure `needs()` — is its target **fresh**,
+**stale** (and why) or **blocked** — runs only when stale or forced, and writes the
+stamp the next stage reads. One index update serves every build; builds and deploys
+run one at a time. Paths below are under `export/.export-builds/` unless named.
+
+| Stage (job `publish-<stage>`) | Target | Fresh when | Writes |
+|---|---|---|---|
+| `update-index` | `_index` | a stamp exists and nothing it reads is newer (below) | `export/.export-index/stamp.json` |
+| `build-site` | `<id>`, or `_all` (each stale site) | the bundle's `inputSig` is the stamp's, no member channel and no config of the site changed since it was built or last checked, and the bundle passes `builtBundleProblem` | `<id>/built.json` |
+| `deploy-site` | `<id>` | the slot it deploys to — `production`, `local` or `previews.<branch>` — already holds this build | `<id>/deployed.json` |
+| `build-hub` | `_hub` | the bundle's `inputSig` is the stamp's `hubSig` and nothing of a listed site changed | `_hub/built.json` |
+| `deploy-hub` | `_hub` | as `deploy-site` (the hub has no local target) | `_hub/deployed.json` |
+| `build-homepage` | `_homepage` | built from the current stamp, and its source is of `main`'s head (when a repository answers) | `_homepage/built.json` |
+| `deploy-homepage` | `_homepage` | as `deploy-site` | `_homepage/deployed.json` |
+
+- **update-index** runs `buildIndex`, `buildStats` and the chart templates in ONE
+ child (8 GB heap), then signs every site and the hub. It runs whenever asked; a
+ rerun that finds nothing short-circuits and **keeps its stamp id** ("nothing changed
+ — the stamp stands"), so the bundles built from it stay current.
+- **A build** composes into the one shared `export/public`, runs `next build` into
+ `export/out`, audits it and installs it as `<target>/out` (a swap through `out.next`;
+ across filesystems, as in the container, a copy), moves the archive staging beside
+ it and points `export/out` at it. The homepage's runs the source mirror into
+ `homepage/out`.
+- **A deploy** ships the target's bundle ([Cloudflare Pages](#cloudflare-pages)), or
+ with `--to local` copies it into `ARCHILYZER_SITE_OUT` / `ARCHILYZER_HOMEPAGE_OUT`.
+
+A stage whose input is not there is **blocked** and exits 3 with a sentence that says
+so: "update the index first"; "no build of jeralyzer in export/.export-builds/jeralyzer
+— archilyzer publish build jeralyzer"; a private site, no Pages project, a bundle that
+fails its guards, a production deploy of a build not made on `main`. `--force` runs a
+fresh stage; Publish now and the lane never force. A build by older code shows a
+"code newer" chip and is not stale.
+
+### The stamps
+
+Written atomically, each only by its stage. A missing or malformed file is no stamp,
+and no stamp is stale.
+
+- **`stamp.json`** (the index): its id, the LMDB `generation`, `scannedAt`, `builtAt`
+ and the commit; per site an **`inputSig`** over everything the site's compose and
+ export build read (its `.export-index` tree, each published member's shared trees,
+ its `site.json`, the curated tags, aliases and duplicates files, the archive and X
+ visibility settings, the social links, the hub URL, its sibling sites) — fresh means
+ the build would produce the same bundle; the **`hubSig`**; the settings signature.
+ A record added anywhere bumps `generation` and moves every site's signature: more
+ rebuilds, never a wrong skip.
+- **`<target>/built.json`**: its id, the index stamp it came from, `inputSig`,
+ `builtAt`, `checkedAt` (a later no-op build found it current), the **commit and
+ branch** (`ARCHILYZER_COMMIT` / `ARCHILYZER_BRANCH` when set — the image bakes them —
+ else git; a detached HEAD records none), the runner, the bundle's `generatedAt`, its
+ size and the archives staged for R2.
+- **`<target>/deployed.json`**: one record per slot — the build's stamp id, when,
+ the deployment URL, the preview alias, the wrangler version and the live check.
+
+**What makes the index stale.** The signals decide *when* to run; the index child
+decides *what* changed. (1) An ingest job — a download, transcription, digest,
+normalize, import, post fetch, tag write — ended `done` after the stamp's
+`scannedAt`, or a channel's report regenerated since; (2) a config file newer than the
+stamp: `tags.json`, `search-aliases.json`, `duplicates*.json`, any `site.json`,
+`homepage.json`, the charts config; (3) **the settings the index reads, by a
+signature of those keys, not `settings.json`'s mtime** (which every pause click
+moves): `socialLinks`, `homepageUrl`, `buildArchives`, `archiveStorage`,
+`social.x.visibility`, `maxTranscriptPageBytes`, the storage locations' roots. The
+same channel read marks a **site** stale before any index runs — "stale: 3 channels
+changed (a, b, c)" — and the signature confirms it after ("stale: data changed").
+
+### The publish lock
+
+**One stage at a time on a machine, whoever started it.** Every stage — a child the
+editor spawned, or `archilyzer publish …` in a terminal — takes
+`export/.export-builds/.publish.lock` (`{pid, host, kind, target, since}`, created
+exclusively). A second one **waits**, saying once whom for ("[publish] waiting for the
+publish lock — held by build-site jeralyzer (pid 4242 on <host>, since …)"); Cancel or
+Ctrl-C gives up the wait. The host is `ARCHILYZER_HOST_ID`, else the hostname.
+
+A lock left by a process that is gone is **taken over** by the next stage: on this
+host, a pid that does not answer, answers with another start time, or started after
+the lock was taken (where there is no `/proc`, pid-alive alone); a file that does not
+parse for 60 s. **A lock naming another host is never taken over**; its wait line
+names both hosts. `archilyzer doctor`'s `publish-lock` row says free, held, stale,
+torn or another host's, and prints the `rm` that clears it — it never removes one.
+Remove it by hand only when nothing is publishing.
+
+### Exit codes, Cancel
+
+`0` ran, or nothing to do · `1` failed (a build, R2, wrangler, a credential refusal)
+· `2` usage (a bad flag, an unknown site, a bad preview name) · `3` precondition not
+met (blocked, above; the docker runner with no engine) · `130` cancelled.
+**`pnpm archilyzer` reports 2, 3 and 130 as 1**: it is `pnpm --filter … exec`. The
+editor spawns `tsx bin/archilyzer.ts` from `common/` directly and sees the real code.
+`publish build all` and `publish deploy all` try every site and exit 1 when any
+failed; `publish now` exits with the worst code (1 over 3). **Cancel takes the stage's whole tree** — `next build`'s workers,
+wrangler, docker; in a terminal a second Ctrl-C kills it at once.
+
+---
+
+## The three ways to drive it
+
+The editor's **/sites** page, `pnpm ops` (HTTP to a running editor, with its
+`WORKER_TOKEN`) and the `archilyzer` CLI (no editor needed) run the same stages: the
+first two as jobs on the editor's `publish` queue, the CLI in its own process, under
+the same lock.
+
+| To | Editor | `pnpm ops publish`, `{"verb": …}` | `pnpm archilyzer …` |
+|---|---|---|---|
+| Update the index | /sites → Pool → **Build index** | `"index"` | `publish index` |
+| Build a site | its /sites row → **Build**; its **Publish** tab → *Build static export* | `"build"` + `"siteId"`/`"siteIds"` | `publish build <id> [--force] [--skip-archives]` |
+| Build every stale site | /sites → **Build all stale** | `"stale"` | `publish build all [--runner docker]` |
+| Deploy a built site | its row → **Deploy preview** / **Deploy production** / **Deploy local**; its Publish tab | `"deploy"` + `"preview"`, `"to": "local"`, `"force"` | `publish deploy <id\|all> [--preview <b>] [--to local] [--force]` |
+| Build, then deploy | its Publish tab → **Build & deploy** | the `build-deploy` alias | `publish build <id>`, then `publish deploy <id>` |
+| The hub | the Hub row → **Build hub** (tick *Deploy after build*), **Deploy hub** | `"hub"` + `"deploy": true`, `"preview"` | `publish hub [--deploy \| --deploy-only] [--preview <b>]` |
+| The homepage | the Homepage row → **Build homepage**, **Deploy homepage**, **Deploy local** | `"homepage"` + `"deploy"`, `"preview"`, `"to"` | `publish homepage [--deploy \| --deploy-only] [--preview <b>] [--to local]` |
+| What the lane would run | /sites → **Publish now** | `"now"` | `publish now` |
+| The status | the /sites Publish panel; /operations/publish | `pnpm ops get publish` | `publish status [--json]` |
+| The source mirror alone | — (every homepage build runs it) | — | `source publish [--force] [--check] [--keep-scratch]`, `source audit [<git dir>]` |
+| A site's report evidence media (then its exports) | its **Reports** tab → *Prepare evidence media* | `reports-prepare` (`{"siteId"}`) | `reports prepare <id>` |
+| A site's report exports (HTML, PDF, Markdown, evidence pack) | its **Reports** tab → *Export reports* | `reports-export` (`{"siteId", "reportId"?, "formats"?}`) | `reports export <id> [--report <rid>] [--formats html,pdf,md,zip]` |
+| A report from a /sweep report, an /ask answer or a report-to-video manifest, and a starter manifest from a report | — | — | `reports convert <sweep\|ask\|manifest> <in> --out <report.json> [--channels-dir <dir>]`, `reports to-manifest <report.json> --out <manifest.json>` |
+
+**The editor.** The /sites **Publish** panel has a row per site, then the hub and the
+homepage, each with four chips — `index`, `built`, `deployed`, `live` — its policy and
+what is next; Deploy local shows where `ARCHILYZER_SITE_OUT` /
+`ARCHILYZER_HOMEPAGE_OUT` is set. Above the rows, beside **Publish now** and **Build
+all stale**, is the plan Publish now would enqueue and what it leaves out. A site's
+Publish tab shows when its bundle was built and from which branch, what it last
+deployed and how the live check read. **A button always runs**: a Build is forced, and
+the index update goes first when the index is not fresh; a Deploy is forced too. The
+CLI runs only the stage you name — `publish build` builds from the index as it stands.
+
+**A run is ordered on disk.** A run's stages are enqueued together under one run id;
+a build carries `indexAfter` and a deploy `builtAfter`, and each is blocked ("waiting
+for the index update this run started") until that step's stamp is new enough, so a
+failed or cancelled step leaves the next one refused rather than shipping old data.
+Cancel on a run's console cancels every stage of that run still queued or running.
+/jobs reads each stage as `run <run id> · <target>`. A stage still queued when the
+editor restarts is cancelled, never re-queued: the lane works out again what is stale.
+
+**`pnpm ops publish`** answers `{runId, jobs: [{target, kind, jobId, previewUrl?}],
+skipped, refused}`; `--wait` follows every job, and a request refused before any job
+exists is named in `refused` (all refused: a 400). The routes it replaced are
+**aliases** with their old bodies and answers (`"skipData"` is ignored): `build-index`,
+`build-site`, `build-deploy`, `deploy-site`, `build-hub`, `deploy-hub`,
+`build-homepage`, `deploy-homepage`. `build-deploy {"all": true}` builds and deploys
+every deployable site to production, whatever its policy.
+
+**The CLI.** `pnpm archilyzer <command>` is the short form of
+`pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts <command>`;
+`--help` lists every command, and `doctor` checks what a publish needs (`wrangler`,
+`cloudflare-auth`, `r2-keys`, `export-builds`, `publish-lock`, `index-stamp`). The old
+rows are **aliases** that print what they run: `build site <id> [--nodata]` = `publish
+index` (not with `--nodata`) + `publish build <id> --force`; `build all` = `publish
+index` + `publish build all --runner auto`; `deploy site <id>` = `publish deploy <id>`;
+`deploy hub` and `deploy homepage` = `publish hub --deploy-only` and `publish homepage
+--deploy-only`. From `export/`, `pnpm run build` and `pnpm run deploy` are `build site`
+and `deploy site` (the site from `SITE_ID`). `build hub` and `build homepage
+[--no-source]` are still bare builds that stamp nothing, so no deploy stage ships them:
+build with `publish hub` / `publish homepage`.
+
+### In the runtime container
+
+Every stage runs inside the editor's container too ([RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md#publish-the-archive)).
+Run commands with **`docker compose exec editor pnpm archilyzer publish …`, never
+`docker compose run --rm`**: a second container has its own `export/public`, is a
+second writer on the index, and carries the editor's fixed host id
+(`ARCHILYZER_HOST_ID=archilyzer-editor`) with pids of its own, so it would judge the
+editor's live lock dead and take it. Cloudflare and R2 credentials come from `.env`.
+`--to local` fills the volumes the `site` and `homepage` services serve. The homepage's
+source mirror builds there too, with `docker-compose.source.yml` (the host's git
+directory, read-only) and the operator's files in the config volume; the image has the
+pinned git-filter-repo but not gitleaks or stagit, so the secret scan is skipped with
+its WARNING and no history pages ship. The docker runner is refused there.
+
+---
+
+## The publish lane
+
+The lane keeps the published targets current by itself: when the index is stale it
+updates it, then builds what changed and deploys where each target's **policy** says,
+one stage at a time. It is **off by default** (`settings.publish.enabled`); the
+buttons and the CLI work either way.
+
+**Policies**: `off` (default; the lane leaves it alone), `build`, `preview` (built,
+then deployed to `settings.publish.previewBranch`, default `preview`) or `production`.
+A site's is `site.json` `publish.auto` (the **Publish policy** on its settings form).
+**A private site is only ever built** — its policy is clamped to `build` — and
+`preview`/`production` need a `cloudflareProject` (a save without one is refused; a
+file that says so reads as `build`). The hub's is `settings.publish.hub` (production
+needs its Pages project in `homepage.json`), the homepage's `settings.publish.homepage`
+(which also publishes the source mirror).
+
+**A pass.** The runner (`auto-publish` on /jobs) wakes every `checkEveryMinutes` (10)
+and starts no pass while the lane is **held**, in its `quietHours`, or while any
+publish stage is queued or running. A pass is due with no index stamp; when the index
+is stale and its last update is at least `refreshEveryMinutes` (360; 0 = whenever
+stale) old; or when a policy target is left stale and the last pass is that old. It
+dispatches ONE stage, waits for it, then re-plans; it never forces. A failed index
+update ends the pass and a failed build drops its deploy. A hold, quiet hours or the
+lane switched off stop the dispatching between stages and never kill one. **Drain**
+finishes the stage in flight and ends the runner; **Stop** ends the runner and leaves
+a running stage to finish (Cancel it on /jobs). An idle boot leaves the runner off.
+
+**Publish now** is the same plan enqueued at once: the index update when stale, each
+site's build and its policy's deploy, then the hub, then the homepage — with every
+policy off, the index update alone. **Build all stale** builds every stale site
+whatever its policy and deploys nothing. **/operations/publish** is the lane's page:
+Start, Drain, Stop, **Hold the lane** (`settings.publish.held`), when a pass is due and
+why, the last pass, what a pass would run now, and the lane's settings — including
+`runner` ([docker](#building-every-site-in-containers), host only).
---
@@ -185,15 +378,61 @@ fits inside Cloudflare's **free tier**.
- **A Pages project must exist before its first deploy.** wrangler offers to create a
missing project only on an interactive terminal, and the deploy's stdin is a pipe,
so a missing project fails at once with wrangler's own "does not exist" sentence.
- Create it first: `pnpm dlx wrangler pages project create <name> --production-branch
- main`. A site's project is `site.json`'s `cloudflareProject`.
-- **wrangler's own auth.** The deploy reuses whatever auth wrangler already has —
- `wrangler login`, or `CLOUDFLARE_API_TOKEN` in the environment.
-- **A production deploy inherits the checkout's git branch.** `wrangler pages deploy`
- with no `--branch` infers one from the repository it runs in, so a *production*
- deploy from a feature-branch checkout silently produces a preview instead. Only the
- preview path passes `--branch`; `deploy homepage` passes `--branch main`. If a
- "production" deploy did not go live, check what branch the checkout is on.
+ Create it first: `pnpm --filter yt-dlp-transcript-common exec wrangler pages project
+ create <name> --production-branch main`. A site's project is `site.json`'s
+ `cloudflareProject`.
+- **wrangler is pinned**: an exact devDependency of `common` (4.147.0), run as
+ `common/node_modules/.bin/wrangler` — nothing is fetched at deploy time.
+ `WRANGLER_BIN` overrides it (the e2e suite's fake). It needs Node 22 or later.
+- **The credential preflight**: `CLOUDFLARE_API_TOKEN`, or a `wrangler login` on this
+ machine. With neither, the deploy is refused before wrangler runs: "[deploy] REFUSED
+ — no Cloudflare credentials: set CLOUDFLARE_API_TOKEN in .env (or run `wrangler
+ login` on this machine). Nothing was sent to Cloudflare." A token Cloudflare rejects
+ — wrong, expired or malformed — ends the log on **"[deploy] REFUSED by Cloudflare —
+ the API token was not accepted"**. Both exit 1.
+- **Every deploy names its branch**: production `--branch main`, a preview `--branch
+ <b>`, never inferred from the checkout. **Production ships only a build of `main`**:
+ a bundle whose `built.json` records another branch, or none (a detached HEAD, an
+ image built without `ARCHILYZER_BRANCH`), is refused — build it from `main`, or
+ deploy it as a preview. A project whose production branch is not `main` would take
+ `--branch main` as a preview.
+
+**A deploy, in order**: the target's refusals; its `built.json` (already in this
+slot: a no-op unless forced); the bundle guards (its `site.json` and `corpus.json`
+must name the site); the preflight; a site's oversize archives to R2; wrangler; the
+live check; then `deployed.json`, written only when everything before it succeeded.
+
+**The live check.** A wrangler exit 0 says a bundle was uploaded, not that a visitor
+gets it. The stage reads `<url>/corpus.json` **plain** and **cache-busted**
+(`?cb=<build stamp id>`), up to three tries 10 s apart, and compares each
+`generatedAt` with the build's. The URL is the preview alias, or for production the
+target's public URL (a site's `siteUrl`, the hub's, the project's), else the
+deployment URL wrangler printed; the homepage's check reads `/` for a 2xx.
+
+| Verdict | Means |
+|---|---|
+| `ok` | both reads serve this build (or the plain one does and the busted one failed) |
+| `stale-edge` | the busted read serves this build, the plain one an older copy: Cloudflare's edge still holds the old object |
+| `mismatch` | the busted read serves another build |
+| `unreachable` | neither read answered 2xx, or the plain read failed |
+| `skipped` | `E2E_LIVE_CHECK=skip` |
+
+**A verdict short of `ok` is a warning, never a failure**: `[live] WARNING <verdict>
+— …` with each read's status, `generatedAt`, `cf-cache-status`, `age` and
+`cache-control`; the job ends `done` and the verdict is recorded. A rebuild with no
+data change carries the same `generatedAt`, so the check tells a stale or wrong
+deployment from the build's data, not two builds of the same data apart.
+
+**Tombstones for withdrawn X posts.** Leaving a path out of a deploy does not take it
+off Cloudflare's edge (the hub served a withdrawn X shard from a week-long cache), so
+while `social.x.visibility` is `private` withdrawn posts are **replaced, never
+deleted** (`common/publish/tombstones.ts`). A public site writes, per withheld X
+channel, `posts/<slug>/manifest.json` with no pages and `[]` for every page the shared
+tree has (and an empty `posts/manifest.json` when it lists no posts at all); the hub
+writes the same for every X channel a non-private site carries, plus an empty posts
+manifest — a channel only private sites carry is never named on the hub. `_headers`
+serves them `Cache-Control: no-store`. **The hub's deploy reads each tombstone back**,
+plain and busted, into its live check.
## The source mirror (homepage)
@@ -215,11 +454,13 @@ The homepage carries the project's own source, read-only, as static files:
| `/downloads/archilyzer-source.tar.gz` + `snapshot.json` | The same tree without history |
| `/source/manifest.json` | What was published, from which private commit, audited how |
-**`archilyzer source publish`** makes them, and **`archilyzer build homepage` runs it**
-between compose and `next build` — so do the editor's /sites homepage jobs and
-`pnpm ops build-homepage`, in the editor's own process and `PATH` (that process needs
-`~/.local/bin` on its `PATH` to find a pipx-installed `git filter-repo`; without it the
-step falls back to `pipx run`, which needs the network).
+**`archilyzer source publish`** makes them, and **the build-homepage stage runs it**
+between compose and `next build` — `archilyzer publish homepage`, the Homepage row's
+Build homepage on /sites, `pnpm ops publish {"verb": "homepage"}` and the lane. A stage
+is a child of the editor (or of the CLI) and runs in its environment and `PATH`: the
+editor's process needs `~/.local/bin` on its `PATH` to find a pipx-installed `git
+filter-repo`; without it the step falls back to `pipx run`, which needs the network.
+The bare `archilyzer build homepage` runs it too.
**A refusal withdraws the source, everywhere it could ship from.** It fails the build
before `next build`, and:
@@ -232,36 +473,40 @@ before `next build`, and:
a limit, a missing tool, a cancel, a crash); `--check` writes nothing, this included;
- the build removes the last BUILD's copy from `homepage/out` (`out/source`, the
tarball, `snapshot.json`);
-- **`deploy homepage` (and /sites → Deploy homepage) refuses** an `out/` holding a
- source unless the skip key says that publish was made under today's rules and step
- version, of today's `main`, by today's gitleaks, and is exactly the one in `out/`:
- its mirror head, and a digest over every published file (the mirror, the tree, the
- history pages, the manifest, the tarball, `snapshot.json` — sorted path, size and
- sha256, recomputed over `out/`), so a mixed or edited `out/` refuses too: "run
- `archilyzer build homepage` (it re-audits), then deploy". History pages that are
- not the audited ones (edited, missing, or present when none were published) are
- named on their own: "homepage/out's history pages (/source/git/) are not the ones
- that were audited". An `out/` whose `/source` page shows the empty
- state (`--no-source`) deploys as before; one with no `/source` page (a refused build)
- does not.
-
-**After merging a change to this step, rebuild and restart the editor before any
-/sites Homepage job.** The editor runs its BUILT bundle: until it is rebuilt, its
-Build homepage job runs the old `buildHomepage` (without this step, or without the
-withdrawal) and its Deploy homepage job has no source check.
-
-`build homepage --no-source` (CLI only) removes the previously published source
-instead, because it was audited against the rules of its own day. A checkout with no
-git repository — the docker runtime, a tarball install — has nothing to mirror: the
-build says `no git repository here; nothing to mirror — the /source page will show its
-empty state`, removes any old publish and goes on; `source publish` run directly there
-exits 1 with the same sentence.
+- **the deploy-homepage stage refuses** an `out/` holding a source unless the skip key
+ says that publish was made under today's rules and step version, of today's `main`,
+ by today's gitleaks, and is exactly the one in `out/`: its mirror head, and a digest
+ over every published file (the mirror, the tree, the history pages, the manifest, the
+ tarball, `snapshot.json` — sorted path, size and sha256, recomputed over `out/`), so
+ a mixed or edited `out/` refuses too: "run `archilyzer build homepage` (it
+ re-audits), then deploy" — rebuild with `archilyzer publish homepage`, the stage,
+ which stamps the bundle the deploy reads. History pages that are not the audited ones
+ (edited, missing, or present when none were published) are named on their own:
+ "homepage/out's history pages (/source/git/) are not the ones that were audited". An
+ `out/` whose `/source` page shows the empty state (`--no-source`) deploys as before;
+ one with no `/source` page (a refused build) does not.
+
+**A stage runs the checkout's source, not the editor's bundle.** Each is `tsx
+bin/archilyzer.ts stage …` spawned from `common/`, so a merged change to this step
+applies to the next build-homepage stage without an editor rebuild. What the editor
+decides itself — the panel, the plan, the refusals before a job exists — is its built
+bundle, rebuilt and restarted as usual.
+
+`build homepage --no-source` (CLI only, the bare build) removes the previously
+published source instead, because it was audited against the rules of its own day. A
+checkout with no git repository and no `ARCHILYZER_SOURCE_REPO` — a tarball install,
+the runtime container without `docker-compose.source.yml` — has nothing to mirror:
+the build says `no git repository here; nothing to mirror — the /source page will
+show its empty state`, removes any old publish and goes on; `source publish` run
+directly there exits 1 with the same sentence.
What one publish does:
1. A **fresh** `git clone --no-local --bare --single-branch --no-tags --branch main` of
- the checkout's git *common* dir — so a worktree build mirrors the primary's `main`.
- The private repository's history is never rewritten.
+ `ARCHILYZER_SOURCE_REPO` when it is set (a git DIR — in a container, the read-only
+ mount of the host's), else the checkout's git *common* dir — so a worktree build
+ mirrors the primary's `main`. A variable naming a path that is not there is a
+ refusal naming it. The private repository's history is never rewritten.
2. **git-filter-repo** rewrites that copy: file contents (`--replace-text`) AND commit
messages (`--replace-message`) with the operator's scrub rules. Author and committer
identities are not rewritten — the gate still reads them.
@@ -339,12 +584,13 @@ must be removed from history by hand.
**Tools.** `git filter-repo` (`pipx install git-filter-repo`; without it the step runs
`pipx run --spec git-filter-repo==2.47.0`, which needs the network on first use, and
-without pipx it refuses with the install line). gitleaks is optional: without it the
-secret scan is skipped with a WARNING and the literal audit still runs. stagit is
-optional too (next section). `archilyzer doctor` has a "source publish" block: which
-filter-repo would run, stagit (its path, or not found, and the render cache's path and
-size), gitleaks, the two files (rule
-counts and modes, never contents) and the last publish.
+without pipx it refuses with the install line; the runtime image has that version
+installed). gitleaks is optional: without it the secret scan is skipped with a
+WARNING and the literal audit still runs. stagit is optional too (next section).
+Neither is in the runtime image. `archilyzer doctor` has a "source publish" block:
+which filter-repo would run, stagit (its path, or not found, and the render cache's
+path and size), gitleaks, the repository it would mirror (`source-repo`), the config
+dir, the two files (rule counts and modes, never contents) and the last publish.
The mirror's ids are deterministic for a given filter-repo version; an upgrade that
changes its rewriting changes every id (readers re-clone). The manifest records both
tools.
@@ -467,8 +713,9 @@ What the step does with it:
## Preview deployments
-A **preview** is the same built bundle deployed to a branch that is not the Pages
-project's production branch. Cloudflare publishes it at a **branch alias** —
+A **preview** is the same bundle — the target's own, as built — deployed to a branch
+that is not the Pages project's production branch. Cloudflare publishes it at a
+**branch alias** —
```
https://<branch>.<project>.pages.dev
@@ -476,8 +723,9 @@ https://<branch>.<project>.pages.dev
— and leaves the live site alone. Each deploy also gets an immutable
per-deployment URL (`https://<hash>.<project>.pages.dev`), which wrangler prints as
-"Deployment complete! Take a peek over at …"; the editor repeats both on a
-`[preview]` line at the end of the job log, because the streamed log scrolls.
+"Deployment complete! Take a peek over at …"; the deploy repeats both on a
+`[preview]` line at the end of its log, because the streamed log scrolls, and records
+them in the preview's slot of `deployed.json`. Its live check reads the alias.
The alias is a function of the project and the branch and nothing else, so it is
known *before* the deploy runs — which is why the editor can link it while you are
@@ -491,20 +739,24 @@ is the live site.
| Surface | How |
|---|---|
-| Editor | A site's **Publish** tab → *Individual steps* → **Deploy a preview**: type a branch, press **Deploy preview**. The production button beside it says **Deploy to production**. |
-| Ops API | `POST /api/ops/deploy-site` `{ "siteId": "...", "preview": "<branch>" }` — deploy-only, of the already-built `export/out`. `POST /api/ops/build-deploy` takes `preview` too (build *then* preview-deploy). Both answer with `previewUrl`. `pnpm ops deploy-site --json '{"siteId":"anilyzer","preview":"tags-exclude"}' --wait` prints the alias on its own line after the log. |
-| CLI | `pnpm archilyzer deploy site anilyzer --preview tags-exclude` |
+| Editor | A site's row on /sites: type a branch (it starts on `settings.publish.previewBranch`), press **Deploy preview**; **Deploy production** beside it. A site's **Publish** tab → *Individual steps* → **Deploy a preview**. The hub's and the homepage's rows take a branch in their preview box (empty = production). |
+| Ops API | `pnpm ops publish --json '{"verb":"deploy","siteId":"anilyzer","preview":"tags-exclude"}' --wait` deploys the built bundle and prints the alias on its own line after the log; the answer carries `previewUrl`. `"hub"` / `"homepage"` take `"deploy": true, "preview"`. The `deploy-site` and `build-deploy` aliases take `"preview"` as before. |
+| CLI | `pnpm archilyzer publish deploy anilyzer --preview tags-exclude` |
+| The lane | a target whose policy is `preview` deploys to `settings.publish.previewBranch` |
The high-value loop is **build once, preview, then promote**: build the site, deploy
it with a preview branch, look at it, then deploy again with no preview — the same
-`export/out`, unrebuilt.
+bundle, unrebuilt. Promotion is a production deploy, so the bundle must have been built
+on `main`; a build from a feature branch previews but never promotes. Each branch is
+its own slot: deploying the same build to the same branch again does nothing unless
+forced.
**A preview shares the production R2 archive bucket.** R2 has no per-branch
namespace, and the keys are `<siteId>/archives/<file>.zip` either way. In practice
this is cheap and harmless — the upload skips any object R2 already holds at the same
size, and an unchanged channel re-zips byte-stable — but a *changed* archive replaces
-the one production's manifest links to. The editor says so once at the top of every
-preview deploy.
+the one production's manifest links to. Every preview deploy of a site says so once,
+at the top of its log.
**Who can open a preview is a Cloudflare setting, not ours.** Pages projects have a
*preview deployment access* setting (Settings → General): **public** by default, or
@@ -520,7 +772,7 @@ live-chat zip **per channel** into `export/public/archives/`, and records them i
`public/archives/manifest.json`. The `/downloads` page and the header **Downloads**
link read that manifest. A site can turn its archives off (`site.json` `archives: false`), and a build
can skip them (`--skip-archives`, `BUILD_ARCHIVES=0`, or the **Skip archive zips**
-checkbox).
+checkbox on a site's Publish tab).
Cloudflare Pages rejects any single asset larger than **25 MB**, and real channels
blow past that easily (a channel's live-chat zip can be hundreds of MB). So the
@@ -528,20 +780,21 @@ pipeline splits archives by size:
| Archive size | Where it's served from | Manifest entry |
|---|---|---|
-| ≤ 25 MB | Cloudflare **Pages** (shipped in `out/`, free) | `filename`, no `url` |
+| ≤ 25 MB | Cloudflare **Pages** (shipped in the bundle, free) | `filename`, no `url` |
| > 25 MB, R2 configured | Cloudflare **R2**, uploaded on deploy | `url` → R2 |
| > 25 MB, R2 **not** configured | not served | `oversize: true`, shown as "Too large to host" |
(The cap is `MAX_ARCHIVE_BYTES`, or a site's own `archiveMaxBytes`; `0` = no cap.)
-Oversize archives are staged during compose into `export/.r2-staging/<siteId>/archives/`
-(gitignored, kept out of `public/`), then uploaded to
-`<bucket>/<siteId>/archives/<file>.zip` **before** the Pages deploy runs, so the
-manifest URLs resolve immediately. **Every deploy path uploads them** — the editor's
-**Deploy** / **Build & deploy**, `pnpm ops deploy-site`, and `archilyzer deploy site`
-(which `pnpm run deploy` in `export/` runs). A build alone only *stages* them. The
-bucket is read from `settings.json`; the R2 credentials come from the environment
-(step 3 below).
+Oversize archives are staged by compose into `export/.r2-staging/<siteId>/archives/`
+(gitignored, kept out of `public/`), and **the build moves them beside the site's
+bundle**, `export/.export-builds/<siteId>/.r2-staging/<siteId>/archives/`, replacing
+the last build's (the docker runner's containers write them there directly);
+`built.json` counts them. They are uploaded to `<bucket>/<siteId>/archives/<file>.zip`
+**before** the Pages deploy runs, so the manifest URLs resolve immediately: **every
+Pages deploy of a site uploads them**, however the deploy stage was started. A build
+alone only stages them, and a `--to local` deploy uploads nothing. The bucket is read
+from `settings.json`; the R2 credentials come from the environment (step 3 below).
Uploads go through R2's **S3 API** (the AWS SDK's multipart uploader), not
`wrangler r2 object put` — wrangler caps a single upload at **300 MiB**, and real
@@ -582,7 +835,8 @@ one:
**3. The R2 S3 credentials.** Archive objects are uploaded over R2's S3-compatible
API, which needs an Access Key ID + Secret. In the dashboard: **R2 → Manage R2 API
Tokens → Create API token**, permission **Object Read & Write**, scoped to your
-bucket. Then set three environment variables where the editor (or the CLI) runs:
+bucket. Then set three environment variables where the editor (or the CLI) runs — in
+Docker, in `.env`:
```sh
export R2_ACCESS_KEY_ID=<access key id>
@@ -595,7 +849,8 @@ The account id is on the R2 overview page; the S3 endpoint is derived as
environment (a shell profile, a systemd unit, a `.env` the editor loads) — **not** in
the settings JSON, which is not a place for secrets. If a deploy has oversize
archives to upload but these are unset, it fails *before* the Pages deploy (so the
-site never links to a missing file) with a message pointing back here.
+site never links to a missing file) with a message pointing back here. `archilyzer
+doctor`'s `r2-keys` names any that are unset, once a bucket is configured.
**4. Point the editor at the bucket.** In **Settings**:
@@ -735,50 +990,46 @@ any limit.
> is about *building sites*: fanning per-site export builds out across containers,
> using `Dockerfile.build`. The two share nothing but the word "docker".
-**Build all sites** builds **every site in parallel** in isolated containers, then
-deploys them serially — a large speedup when you host several sites, and stronger
-isolation than building one site at a time in `export/`. It does so **whenever a
-container engine answers** (`docker version`), and builds serially on the host when
-none does. There is no mode to set. A single site's build always runs in `export/`,
-one at a time.
-
-**Prerequisites.**
-
-- A container engine: **Docker**, or **podman** (set `DOCKER_BIN=podman`). Rootless
- podman is a good fit — it maps container files to your host user automatically.
-- The editor host still needs Node + pnpm (Phase A and the deploys run on the host)
- and the Cloudflare/R2 credentials in the environment. Credentials are **never**
- passed into a container — deploy runs on the host.
-- Inside the runtime container (`docker compose up`) there is no `docker` binary, and
- the pipeline falls back to the serial host build.
-
-The build image is built (and cached) automatically from `Dockerfile.build` the first
-time you run; **Build image** / **Dockerfile** in Settings override the tag and path.
-
-**How it works.** Trigger it with **Build all sites** on /sites, or `pnpm archilyzer
-build all`; both use containers when `docker version` answers. One job runs three
-ordered phases:
-
-1. **Phase A — shared, on the host, once.** The data phase (the search index and the
- `.export-index` staging), then `archilyzer build archives` (the shared archive-zip
- cache for the union of all sites' channels). Only the host writes this shared
- state, so containers never race it. This phase is serial and is the long pole on a
- cold build; on a warm rebuild it is near-instant (unchanged channels are skipped).
-2. **Phase B — per-site, in parallel containers.** Each site's compose + `next build`
- runs in its own container (`docker/build-site.sh`: `archilyzer build site <id>
- --nodata`), capped by **Max parallel builds**. Each writes an isolated `out/` under
- `export/.export-builds/<siteId>/`. Containers mount the corpus, index, staging and
- archive cache **read-only** (`ARCHIVES_READONLY=1`). Network is left on: `next
- build` fetches the site's fonts through `next/font/google`; isolation comes from the
- read-only mounts, the per-site output dir and the non-root user.
-3. **Phase C — deploy, on the host, serially.** After every build finishes, each built
- site is deployed in turn (the R2 upload, then `wrangler pages deploy`). A single
- site failing to build or deploy is reported and skipped; the rest still ship.
-
-If no container engine is available, the action logs a notice and falls back to a
-serial host build+deploy (one site at a time).
-
-**Mounts, per Phase-B container.**
+Every build goes through one stage contract with two **runners**. **local** is the
+default everywhere, in a container or not: each site in turn, as a child of the editor
+or the CLI. **docker** is an opt-in for a **Linux host with a container engine**: every
+stale site at once, each in its own isolated container. Both write the same bundles
+and `built.json` (`runner: "docker"`), so a deploy does not care which built a site;
+the docker runner only builds.
+
+**Choosing it**: `settings.publish.runner: "docker"` (/operations/publish → Build
+runner) for the lane and Publish now, or `archilyzer publish build all --runner
+docker` (`pnpm ops publish {"verb": "build", "runner": "docker"}`); `--runner auto`
+(the `build all` alias) takes docker when an engine answers. Where none answers
+`docker version` — **the runtime container included**, which has no engine and never
+gets the socket — the stage is **refused**, exit 3: "the docker runner needs an engine
+on this host". No docker-in-docker.
+
+**Prerequisites.** Docker, or **podman** (`DOCKER_BIN=podman`; rootless podman maps
+container files to your host user). A current index stamp — the runner builds from the
+update-index stage's index and is blocked without one. Node + pnpm on the host (the
+archive warm and the deploys run there) and the Cloudflare/R2 credentials in its
+environment; credentials **never** enter a container. The image is built (and cached)
+from `Dockerfile.build` at the start of every fan-out; Settings → Build pipeline →
+**Docker image tag** and **Dockerfile path** override the tag and path.
+
+**How it works.** One stage, `build-site _all --runner docker`, holding the publish
+lock throughout:
+
+1. **Which sites**: each site's freshness, as the build stage judges it; fresh sites
+ are skipped (`--force` builds every one).
+2. **The archive cache, on the host, once** (`build:archives`, skipped by
+ `--skip-archives`): the shared archive-zip cache for the union of the sites'
+ channels. Only the host writes it, so containers never race it.
+3. **Per site, in parallel containers**, up to **Max parallel builds** (Settings →
+ Build pipeline): `docker/build-site.sh` runs `archilyzer build site <id> --nodata`,
+ which in the container builds in place and stamps nothing (`ARCHIVES_READONLY=1`;
+ its lock stays inside the container), and copies `out/` to
+ `export/.export-builds/<id>/out`. Network is left on: `next build` fetches fonts
+ through `next/font/google`; isolation is the read-only mounts, the per-site output
+ dir and the non-root user.
+4. **On the host, per site**: `builtBundleProblem`, then `built.json`. A failed site is
+ named and the rest are still stamped; the stage then exits 1.
| Host | Container | Mode |
|---|---|---|
@@ -788,39 +1039,23 @@ serial host build+deploy (one site at a time).
| `settings.json` (build config, mounted fresh — not baked) | `/data/settings.json` | ro |
The per-site `/site` mount is persistent, so incremental compose (`.compose-cache`)
-stays warm across builds. `next build` runs in the container's own `.next` and starts
-fresh each time. A Turbopack production build keeps no cache between runs anyway.
-The container's `export/public` is a link to the composed `/site/public`, so `out/`
-carries this site's data and nothing baked into the image. Before handing `out/`
-back, the container checks that its `site.json` and `corpus.json` both name the site,
-and the deploy phase checks again. A bundle that names another site, or no site, is
-refused.
-
-**Tuning.**
-
-- **Max parallel builds** (setting) — how many site containers run at once. Each `next
- build` can use up to ~8 GB; a safe starting point is `floor(RAM_GB / 9)`.
-- `DOCKER_BIN` — the container binary (default `docker`; e.g. `podman`).
-- `DOCKER_BUILD_MEMORY`, `DOCKER_BUILD_CPUS` — optional per-container `--memory` /
- `--cpus` caps so a fan-out cannot OOM or peg the host.
-- `BUILD_ARCHIVES=0` (or **Skip archive zips**) — skip the archive warm and the
- per-site archive materialize for a faster build with no download bundles.
-
-**Notes.** Containers run as your host uid/gid (`-u`), so files under
-`.export-builds/` are host-owned, not root-owned. The image bakes the export build's
-source (common/, export/) and its deps; a code change rebuilds it, but layer caching
-keeps that cheap (deps re-install only when the lockfile moves). Its build context is
-the allow-list in `Dockerfile.build.dockerignore`, about 7 MB from any checkout. It
-never includes the corpus, generated `export/public` data or `.export-builds/`. That
-file is read by BuildKit (Docker's default builder). A builder that reads only the
-shared `.dockerignore` sends several GB from a working checkout, and podman is
-unverified here. That fallback is slower but still safe: each site's `out/` is built
-from its own composed data, and only the tracked `.svg` assets are copied in from the
-image. `archilyzer doctor` says whether the image is there and older than its
-Dockerfile.
-
-The editor mounts the host's `docker/build-site.sh` over the baked one, so an image
-older than the checkout still runs today's script.
+stays warm; `next build` starts fresh each time in the container's own `.next`. The
+container's `export/public` is a link to `/site/public`, so `out/` carries this site's
+data and nothing baked into the image. The container checks that `out/`'s `site.json`
+and `corpus.json` both name the site before handing it back, the host before it
+stamps, and the deploy stage again.
+
+**Tuning.** **Max parallel builds** — each `next build` can use ~8 GB; start at
+`floor(RAM_GB / 9)`. `DOCKER_BUILD_MEMORY`, `DOCKER_BUILD_CPUS` — per-container
+`--memory` / `--cpus` caps. `--skip-archives` (or `BUILD_ARCHIVES=0`) — no archive
+warm and no download bundles.
+
+**Notes.** Containers run as your host uid/gid (`-u`), so `.export-builds/` stays
+host-owned. The image bakes common/ and export/ with their deps; layer caching keeps a
+code change cheap. Its build context is `Dockerfile.build.dockerignore`'s allow-list
+(about 7 MB, read by BuildKit; podman is unverified). The host's
+`docker/build-site.sh` is mounted over the baked one. `archilyzer doctor` says whether
+the image is there and older than its Dockerfile.
---
diff --git a/README.md b/README.md
@@ -124,7 +124,7 @@ expected. Add your first channel from the Channels page.
Then, to build and serve the public site:
```bash
-pnpm build # static site under export/out/
+pnpm build # publish index + publish build $SITE_ID; export/out links to the bundle
pnpm start:export # serve it at http://localhost:3000
```
diff --git a/SETTINGS.md b/SETTINGS.md
@@ -629,13 +629,13 @@ Default:
## `buildPipeline`
-The build pipeline's settings. A single site builds in the shared export/ tree, serialized on one queue. Build all sites (and Build & deploy all) builds every site at once, each in its own container (Dockerfile.build, the image and maxParallelBuilds below), then deploys them serially, whenever a container engine answers, and serially on the host when none does — see PUBLISH.md. There is no mode switch; a `mode` key left in an older file is dropped on the next save.
+The docker build runner's settings. A site's build is a publish stage (release 18): by default (`publish.runner: "local"`) every site builds in turn as a child of the editor, one stage at a time on the `publish` queue. With `publish.runner: "docker"` (or `archilyzer publish build all --runner docker`) every stale site builds in its own container — Dockerfile.build, the image and maxParallelBuilds below — on a Linux host whose container engine answers; it is refused inside a container. Deploys are their own stages. See PUBLISH.md. There is no mode switch; a `mode` key left in an older file is dropped on the next save.
#### `buildPipeline`
| Key | Default | Description |
|---|---|---|
-| `maxParallelBuilds` | `2` | Cap on concurrent per-site container builds when Build all sites runs in containers (whenever a container engine answers). Clamped to [1, BUILD_MAX_PARALLEL_MAX]. |
+| `maxParallelBuilds` | `2` | Cap on concurrent per-site container builds when the docker build runner builds every site (`publish.runner: "docker"`, or `archilyzer publish build all --runner docker`). Clamped to [1, BUILD_MAX_PARALLEL_MAX]. |
| `dockerImage` | `"yt-dlp-transcript-browser-build"` | Tag of the reusable build image (built once, reused for every site). |
| `dockerfile` | `"Dockerfile.build"` | Dockerfile path relative to the monorepo root, used to (re)build the image. |
diff --git a/SETUP.md b/SETUP.md
@@ -206,7 +206,7 @@ pnpm dev:editor # editor (admin UI) at http://localhost:3001
To build and serve the read-only public site:
```sh
-pnpm build # static site under export/out/
+pnpm build # publish index + publish build $SITE_ID (export/out links to the bundle)
pnpm start:export # serve export/out/ at http://localhost:3000
```
diff --git a/SITE.md b/SITE.md
@@ -170,7 +170,7 @@ Default: `true`
## `audience`
-Who this site is built for. `"public"` (the default; absent) or `"private"`: the operator's own reading copy, built on this machine and never deployed — every deploy path (Build & deploy, Deploy, `archilyzer deploy site`, Build & deploy all, docker/publish-site.sh) refuses it before any upload, while a build without a deploy still works. A private site is never listed (as `listed: false`, whatever `listed` says), publishes no `hubUrl`, and its `/corpus.json` says `"audience": "private"`. Content kept from the public — X posts while `social.x.visibility` is `"private"` — is built only into private sites. Only `"private"` is written.
+Who this site is built for. `"public"` (the default; absent) or `"private"`: the operator's own reading copy, built on this machine and never deployed — the deploy stage refuses it before any upload — Deploy production, Deploy preview, Deploy local, Build & deploy, Publish now, the publish lane, `archilyzer publish deploy`, docker/publish-site.sh — while a build without a deploy still works (its publish policy reads as `build`). A private site is never listed (as `listed: false`, whatever `listed` says), publishes no `hubUrl`, and its `/corpus.json` says `"audience": "private"`. Content kept from the public — X posts while `social.x.visibility` is `"private"` — is built only into private sites. Only `"private"` is written.
Default: absent
diff --git a/common/lib/envVars.ts b/common/lib/envVars.ts
@@ -149,7 +149,7 @@ const DECLARED: EnvVarDecl[] = [
{ name: "PARAKEET_DEVICE", audience: "runtime", default: "parakeet-cli's", readBy: "scripts/parakeet-stitch.mjs", doc: "Compute device (`cpu`, `CUDA0`, `Vulkan1`, …), exported to parakeet-cli." },
// ── internal: the pipeline sets these for a process it spawns ──────────
- { name: "SITE_ID", audience: "internal", default: "—", readBy: "common/bin/compose-site.ts, export/app/lib/site.ts", doc: "Which site a compose or an export build is for. `archilyzer build site <id>` sets it; `compose site` and `build site` fall back to it when no id is given." },
+ { name: "SITE_ID", audience: "internal", default: "—", readBy: "common/bin/compose-site.ts, export/app/lib/site.ts", doc: "Which site a compose or an export build is for. The build stage (`archilyzer publish build <id>`) sets it for its children; `compose site`, `build site` and `deploy site` fall back to it when no id is given." },
{ name: "INSTANCE_MODE", audience: "internal", default: "a site", readBy: "export/app/lib/mode.ts, common/lib/archive/contract.ts", doc: "`hub` makes the export build the hub. Set by `archilyzer build hub`." },
{ name: "BUILD_ARCHIVES", audience: "internal", default: "on", readBy: "common/bin/compose-site.ts, common/bin/build-archives.ts", doc: "`0` skips archive-zip generation for one build (`--skip-archives`)." },
{ name: "REPORTS_ALLOW_MISSING_MEDIA", audience: "internal", default: "off", readBy: "common/bin/compose-site.ts", doc: "`1` lets a report citation whose evidence media was not prepared through compose (`--allow-missing-media`): its moment page renders without a clip. Off, compose fails with the list." },
diff --git a/common/lib/settingsSchema.ts b/common/lib/settingsSchema.ts
@@ -458,11 +458,14 @@ export const DIGEST_SETTINGS_FIELD_DOCS: FieldDocs<DigestSettings> = {
"fresh. Empty = default.",
};
-// Build all / Build & deploy all (editor buildAction.ts, `archilyzer build all`)
-// fan out in containers (publish/stageBodies.ts, buildAllDocker) whenever
-// `docker version` answers, and build serially on the host otherwise. There is
-// no mode switch: the `mode` key ("basic" | "docker") was a label nothing read,
-// and it was dropped on 2026-09-28 (release 11, follow-up O6c). A settings.json
+// The docker build runner (release 18: `settings.publish.runner: "docker"`,
+// `archilyzer publish build all --runner docker`) fans every stale site out in
+// containers (publish/stageBodies.ts, buildAllDocker) on a host whose engine
+// answers; `--runner auto` (`archilyzer build all`) does so when one answers and
+// builds serially otherwise. The default runner is "local": one site at a time,
+// a child of the editor. There is no mode switch: the `mode` key ("basic" |
+// "docker") was a label nothing read, and it was dropped on 2026-09-28
+// (release 11, follow-up O6c). A settings.json
// that still carries it loads, and loses it on the next save — the sanitizer
// below builds its output from the three real fields only.
//
@@ -475,8 +478,9 @@ export type BuildPipelineSettings = {
export const BUILD_PIPELINE_SETTINGS_FIELD_DOCS: FieldDocs<BuildPipelineSettings> = {
maxParallelBuilds:
- "Cap on concurrent per-site container builds when Build all sites runs in" +
- " containers (whenever a container engine answers)." +
+ "Cap on concurrent per-site container builds when the docker build runner" +
+ " builds every site (`publish.runner: \"docker\"`, or" +
+ " `archilyzer publish build all --runner docker`)." +
" Clamped to [1, " +
"BUILD_MAX_PARALLEL_MAX].",
dockerImage:
@@ -1798,7 +1802,7 @@ export const siteSettingsSchema = z.object({
"Where a channel's downloaded media goes when it is relocated off the corpus disk. A DEFAULT ONLY: the relocate controller never reads it and always takes an explicit root, so this is the value the per-channel Storage panel prefills and the /channels bulk move falls back to. Blank = no default. See StorageSettings.",
),
buildPipeline: settingsField((v): BuildPipelineSettings => sanitizeBuildPipeline(v)).describe(
- "The build pipeline's settings. A single site builds in the shared export/ tree, serialized on one queue. Build all sites (and Build & deploy all) builds every site at once, each in its own container (Dockerfile.build, the image and maxParallelBuilds below), then deploys them serially, whenever a container engine answers, and serially on the host when none does — see PUBLISH.md. There is no mode switch; a `mode` key left in an older file is dropped on the next save.",
+ "The docker build runner's settings. A site's build is a publish stage (release 18): by default (`publish.runner: \"local\"`) every site builds in turn as a child of the editor, one stage at a time on the `publish` queue. With `publish.runner: \"docker\"` (or `archilyzer publish build all --runner docker`) every stale site builds in its own container — Dockerfile.build, the image and maxParallelBuilds below — on a Linux host whose container engine answers; it is refused inside a container. Deploys are their own stages. See PUBLISH.md. There is no mode switch; a `mode` key left in an older file is dropped on the next save.",
),
digest: settingsField((v): DigestSettings => sanitizeDigest(v)).describe(
"AI digest generation (chapters + topic tags over the existing transcripts). Local-first: the metered lane is off by default. See DigestSettings.",
diff --git a/common/lib/siteSchema.ts b/common/lib/siteSchema.ts
@@ -191,7 +191,7 @@ export const SITE_FIELD_DOCS: FieldDocs<Site> = {
listed:
"Whether the family lists this site. Opt-OUT: absent/true = listed, only an explicit `false` is written. An unlisted site still builds and deploys as before, and its own pages are unchanged; it is left out of the homepage (cards, chart, `/stats`), the hub (members, federated search, `/corpus.json`, `/llms.txt`), every other site's footer, and the published `channel-sites.json` and pooled `stats/`. A channel only unlisted sites expose is in none of the family's public totals; a channel a listed site also exposes is credited to the listed one. A private site and a report-only site (`search: false`) are never listed, whatever `listed` says.",
audience:
- 'Who this site is built for. `"public"` (the default; absent) or `"private"`: the operator\'s own reading copy, built on this machine and never deployed — every deploy path (Build & deploy, Deploy, `archilyzer deploy site`, Build & deploy all, docker/publish-site.sh) refuses it before any upload, while a build without a deploy still works. A private site is never listed (as `listed: false`, whatever `listed` says), publishes no `hubUrl`, and its `/corpus.json` says `"audience": "private"`. Content kept from the public — X posts while `social.x.visibility` is `"private"` — is built only into private sites. Only `"private"` is written.',
+ 'Who this site is built for. `"public"` (the default; absent) or `"private"`: the operator\'s own reading copy, built on this machine and never deployed — the deploy stage refuses it before any upload — Deploy production, Deploy preview, Deploy local, Build & deploy, Publish now, the publish lane, `archilyzer publish deploy`, docker/publish-site.sh — while a build without a deploy still works (its publish policy reads as `build`). A private site is never listed (as `listed: false`, whatever `listed` says), publishes no `hubUrl`, and its `/corpus.json` says `"audience": "private"`. Content kept from the public — X posts while `social.x.visibility` is `"private"` — is built only into private sites. Only `"private"` is written.',
reports:
"The site's published reports, in display order: report ids (lowercase slugs, `[a-z0-9][a-z0-9-]*`), each a directory under `sites/<siteId>/reports/`. A report directory not named here is a draft and is not published. Invalid and repeated ids are dropped. Absent/empty = no reports.",
relatedSites:
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -5,10 +5,7 @@
- **One index for every site.** The index is updated once and every site, the hub and the homepage are built from it; `archilyzer publish status` says, per site, whether its build is current — "stale: 3 channels changed (a, b, c)" as soon as a download, transcription or digest on one of its channels finishes, before any index runs; "stale: data changed" once the index has run and the site's data moved; "stale: config changed" after its site.json, tags or aliases changed — and whether what is deployed is that build, with a build made by older code marked "code newer" but not stale.
- **Deploys are pinned and checked live.** wrangler is an exact dependency of the workspace (4.147.0), so a deploy runs the version installed with the code instead of whatever `pnpm dlx` fetched that day, and every deploy names its branch: production is `--branch main`, never taken from the checkout it ran in (where a "production" deploy from a feature branch used to land as a preview). The publish stages' deploy (release 18) refuses before wrangler runs when there is no Cloudflare credential at all — "set CLOUDFLARE_API_TOKEN in .env" — and says "REFUSED by Cloudflare — the API token was not accepted" when Cloudflare rejects one; it refuses a production deploy of a build made from a branch other than `main`. After each deploy it reads `corpus.json` at the site's address twice, as a visitor would and cache-busted, and records the verdict: ok, stale-edge (the deployment is right, Cloudflare's edge still serves an older copy), mismatch, or unreachable. A verdict short of ok is a warning in the log; the deploy itself succeeded. What each target last shipped, where, and how it read is kept in `deployed.json` beside its build.
- **Withdrawn X posts ship tombstones.** While X posts are private, a public site's build no longer just leaves an X channel's posts out: at every path they were served from it ships an empty stand-in — the channel's posts manifest with no pages, and an empty page for each page the channel has — served uncached. The hub, which carries no posts, ships the same for every X channel a public site carries, with an empty posts manifest; a channel only on a private site, or on no site, is never named on the hub. Leaving a path out of a deploy does not take it off Cloudflare's edge, which kept serving a withdrawn copy for up to a week; a changed object at the same path replaces it. The hub's deploy reads each of those paths back.
-- **Publishing is stages, from the command line: `archilyzer publish`.** `publish index` updates the index — the LMDB index, the stats datasets and the chart templates, in one child process with an 8 GB heap — and writes an index stamp (`export/.export-index/stamp.json`) naming, for each site, a signature of everything that site's build reads. `publish build <id|all>` builds a site from that index (no data phase of its own) into its own bundle, `export/.export-builds/<id>/out`, and stamps it (`built.json`); a site whose bundle already matches the index is a no-op unless `--force`. `publish deploy <id|all> [--preview <branch>] [--to local]` ships that bundle — to Cloudflare Pages, or with `--to local` into the directory the docker `site` service serves — and records the deploy (`deployed.json`); deploying the same build again is a no-op unless `--force`. `all` passes over private sites and, to Pages, sites with no Pages project; any other site it cannot deploy is a failure, said after the rest are tried. `publish hub [--deploy]` and `publish homepage [--deploy]` do the same for the hub (`_hub/out`) and the homepage. A stage whose input is not there says so and exits 3: "update the index first", "no build of jeralyzer — archilyzer publish build jeralyzer". Production refuses a bundle built on a branch other than `main`, or with no branch recorded (a detached checkout; an image sets `ARCHILYZER_BRANCH`) — a preview of it is fine. Exit codes: 0 done or nothing to do, 1 failed, 2 usage, 3 precondition not met, 130 cancelled. In the editor, **/sites has a Publish panel** in place of "Build all sites" and the hub's and the homepage's build sections: a row per site, the hub and the homepage, each with four chips — index, built, deployed, live — and Build, Deploy preview, Deploy production (and Deploy local where the container serves one); **Publish now** runs what the lane would, **Build all stale** builds every stale site whatever its policy, and the plan Publish now would run is listed above the rows. Every button is stages on the `publish` queue, followed in one log, and a manual Build or Deploy always runs (the index is updated first when it is stale). The Pool's **Build index** is the index stage, and **Build stats dataset is gone**: the stats are part of it. A site's Publish tab is stages too, and its "Last deployed" is what that site last shipped and how its live check read. Over HTTP, `pnpm ops publish` takes `{"verb": "index" | "build" | "deploy" | "hub" | "homepage" | "now" | "stale"}` and `pnpm ops get publish` is the status; `build-index`, `build-site`, `build-deploy`, `deploy-site`, `build-hub`, `deploy-hub`, `build-homepage` and `deploy-homepage` still answer as before, as stages — except that `build-site` and `build-deploy` with `"all": true` answer a job per site (their `jobId` is the run's last), and a deploy-only of a build already deployed there is a no-op. A console's **Cancel** cancels its whole run.
-- **One publish at a time on a machine.** Every stage takes `export/.export-builds/.publish.lock`; a second one — an `archilyzer publish` beside the editor, say — waits for it, saying once whom it waits for, and Ctrl-C ends the wait. A lock left by a process that is gone is taken over. A cancelled stage takes the whole process tree it started with it (`next build`'s workers, wrangler, docker).
-- **`export/out` is now a link to the bundle built last.** Each site, and the hub, keeps its own bundle, so building one site no longer replaces another's; `export/out` points at whichever was built most recently, so `serve out` and anything else that read it keeps working.
-- **`build site`, `build all` and `deploy site` are aliases of the publish commands** and print what they run: `build site <id>` is `publish index` (skipped with `--nodata`) then `publish build <id> --force`; `build all` is `publish index` then `publish build all --runner auto` (containers when an engine answers, else one site at a time on the host); `deploy site <id>` is `publish deploy <id>`, which now ships the site's own bundle and refuses a site never built that way; `deploy hub` and `deploy homepage` are `publish hub --deploy-only` and `publish homepage --deploy-only` (a no-op when that build is already deployed there, unless `--force`; a refusal exits 3). `publish build all --runner docker` builds every stale site in containers on a Linux host and refuses with "the docker runner needs an engine on this host" where there is none.
+- **Publishing is stages, from the command line: `archilyzer publish`.** `publish index` updates the index — the LMDB index, the stats datasets and the chart templates, in one child process with an 8 GB heap — and writes an index stamp (`export/.export-index/stamp.json`) naming, for each site, a signature of everything that site's build reads. `publish build <id|all>` builds a site from that index (no data phase of its own) into its own bundle, `export/.export-builds/<id>/out`, and stamps it (`built.json`); a site whose bundle already matches the index is a no-op unless `--force`. `publish deploy <id|all> [--preview <branch>] [--to local]` ships that bundle — to Cloudflare Pages, or with `--to local` into the directory the docker `site` service serves — and records the deploy (`deployed.json`); deploying the same build again is a no-op unless `--force`. `all` passes over private sites and, to Pages, sites with no Pages project; any other site it cannot deploy is a failure, said after the rest are tried. `publish hub [--deploy]` and `publish homepage [--deploy]` do the same for the hub (`_hub/out`) and the homepage. A stage whose input is not there says so and exits 3: "update the index first", "no build of jeralyzer — archilyzer publish build jeralyzer". Production refuses a bundle built on a branch other than `main`, or with no branch recorded (a detached checkout; an image sets `ARCHILYZER_BRANCH`) — a preview of it is fine. Exit codes: 0 done or nothing to do, 1 failed, 2 usage, 3 precondition not met, 130 cancelled. In the editor, **/sites has a Publish panel** in place of "Build all sites" and the hub's and the homepage's build sections: a row per site, the hub and the homepage, each with four chips — index, built, deployed, live — and Build, Deploy preview, Deploy production (and Deploy local where the container serves one); **Publish now** runs what the lane would, **Build all stale** builds every stale site whatever its policy, and the plan Publish now would run is listed above the rows. Every button is stages on the `publish` queue, followed in one log, and a manual Build or Deploy always runs (the index is updated first when it is stale). The Pool's **Build index** is the index stage, and **Build stats dataset is gone**: the stats are part of it. A site's Publish tab is stages too, and its "Last deployed" is what that site last shipped and how its live check read. Over HTTP, `pnpm ops publish` takes `{"verb": "index" | "build" | "deploy" | "hub" | "homepage" | "now" | "stale"}` and `pnpm ops get publish` is the status; `build-index`, `build-site`, `build-deploy`, `deploy-site`, `build-hub`, `deploy-hub`, `build-homepage` and `deploy-homepage` still answer as before, as stages — except that `build-site` and `build-deploy` with `"all": true` answer a job per site (their `jobId` is the run's last), and a deploy-only of a build already deployed there is a no-op. A console's **Cancel** cancels its whole run. **One publish at a time on a machine.** Every stage takes `export/.export-builds/.publish.lock`; a second one — an `archilyzer publish` beside the editor, say — waits for it, saying once whom it waits for, and Ctrl-C ends the wait. A lock left by a process that is gone is taken over. A cancelled stage takes the whole process tree it started with it (`next build`'s workers, wrangler, docker). **`export/out` is now a link to the bundle built last.** Each site, and the hub, keeps its own bundle, so building one site no longer replaces another's; `export/out` points at whichever was built most recently, so `serve out` and anything else that read it keeps working. **`build site`, `build all` and `deploy site` are aliases of the publish commands** and print what they run: `build site <id>` is `publish index` (skipped with `--nodata`) then `publish build <id> --force`; `build all` is `publish index` then `publish build all --runner auto` (containers when an engine answers, else one site at a time on the host); `deploy site <id>` is `publish deploy <id>`, which now ships the site's own bundle and refuses a site never built that way; `deploy hub` and `deploy homepage` are `publish hub --deploy-only` and `publish homepage --deploy-only` (a no-op when that build is already deployed there, unless `--force`; a refusal exits 3). `publish build all --runner docker` builds every stale site in containers on a Linux host and refuses with "the docker runner needs an engine on this host" where there is none.
- **Substitute your own yt-dlp in Docker.** Point `YTDLP_BIN` at a zipapp you built, or set `YTDLP_SOURCE_HOST_DIR` to a yt-dlp checkout and start with `docker-compose.ytdlp.yml`: the image runs it with its own python, and nothing is rebuilt. Every editor boot logs `yt-dlp: <path> <version> (image|override)` (`MISSING` when it does not run; the editor still starts), and `YTDLP_AUTO_UPDATE` updates the image's yt-dlp only, warning instead of touching yours.
- **The Docker image can publish.** It carries python, `pipx` and a pinned `git-filter-repo`, so the homepage's `/source` mirror builds in the container; `docker-compose.source.yml` mounts your repository read-only for it, and the scrub rules and denylist live in the config volume (`/data/config/archilyzer`). Cloudflare and R2 credentials come from `.env`. Run publish commands with `docker compose exec editor pnpm archilyzer …`, not `run --rm`. The `homepage` service serves a local deploy from the builds volume once there is one. RUNNING_IN_DOCKER.md has a Windows checklist.
- **`archilyzer doctor` checks what a publish needs.** Which yt-dlp runs (the image's, the host's or an override, and whether it runs), whether the Cloudflare token and the R2 keys are set (never their values; R2 only when a bucket is configured) — judged exactly as a deploy judges them —, the wrangler a deploy runs (the pinned one or your `WRANGLER_BIN`, and that it starts and is the expected major), free space for the site bundles, the publish lock (free, held by a running stage, or left by one that is gone — with the command to clear it; never cleared for you), the index stamp's age and which sites were built from an older one, the repository the source mirror reads, the private config dir, and whether this Node is new enough for the pinned wrangler (deploys need 22).
diff --git a/export/CHANGELOG.md b/export/CHANGELOG.md
@@ -1,6 +1,7 @@
# Changelog
## [Unreleased]
+- **Posts withdrawn from a site leave a stand-in, not a stale copy.** When an X channel's posts stop being published on a site — while X posts are private, say — the site's build ships an empty posts manifest and an empty page at every address they were served from, sent with `Cache-Control: no-store`, so a reader (or the hub) asking for them gets "no posts" instead of the copy Cloudflare's edge kept for up to a week. The hub does the same for every X channel a public site carries.
- **Search reads every English track of a video, and the transcript switches tracks.** Where a video has another English caption track whose words differ from its transcript — the uploaded captions beside the original audio's, a regional or auto-translated track — a query matches it too: a hit only that track holds says so ("in uploaded captions") and opens the transcript on that track at that moment, and a word both say is found once, in the transcript. The transcript reader shows a small "Track:" switcher beside the mode buttons on such a video; the transcript stays the default, and the choice rides on the share link (`vt`). Downloads and Copy MD take the track on show. Needs an index build and a rebuild and deploy of each site.
- **A citation of a Wayback Machine copy links its original and the copy.** A cited record downloaded from a Wayback capture shows "Original (may be gone)", the original at the cited second where its platform takes one, and "Wayback Machine copy, <capture date>", the capture page, which plays. Its moment link is the capture: a capture URL never takes a time param.
- **Transcripts read the original-audio captions.** Where a video has both, its transcript is YouTube's `en-orig` track (the captions of what was said) rather than the served `en`, which can reword it; a track with no text falls through to the next. Videos whose only captions are in cue blocks (some livestream recordings) have their text.
diff --git a/homepage/CHANGELOG.md b/homepage/CHANGELOG.md
@@ -1,6 +1,7 @@
# Homepage Changelog
## [Unreleased]
+- **The homepage builds and deploys from the Docker image too.** It is a publish stage like a site's: `archilyzer publish homepage [--deploy]` (or its row on /sites → Publish) builds it — the `/source` mirror included, from the repository `docker-compose.source.yml` mounts read-only — into `homepage/out`, stamps it, and deploys that build to its Pages project, or with `--to local` into the volume the container's `homepage` service serves. The image has the pinned git-filter-repo; it has no gitleaks or stagit, so a container build skips the secret scan with a warning and ships no history pages.
- **A site that publishes only its reports is not on the homepage.** A site with `publish: "cited"` has no Official Instances card, chart series, `/stats` entry or recent item, is not in `channel-sites.json`, and the channels only it carries count in no total — as an unlisted site, whatever its listing setting says. A site with reports that publishes its full corpus is listed as before.
- **The AI and MCP doc has a Ten-minute setup.** Right after the MCP server's introduction, one block runs Claude Code against a published archive, the Jeralyzer as the example: clone the source (or unpack the tarball on Downloads), `pnpm install`, `claude mcp add archilyzer`, start `claude` and try `/ask`; then what it needs, why the server must be registered as `archilyzer` (the shipped `/ask` and `/sweep` call `mcp__archilyzer__…`), the two optional editor lines for `fetch_clip`, `TRANSCRIPT_HUB_URL`, where the `mcp.json` form for other clients is, and WSL2 on Windows. "What it can do" is a heading of its own after it. Every archive's **Use with AI** link now lands on this page.
- **The Ten-minute setup starts the server with the source's own command, `pnpm --silent -C "$PWD" archilyzer mcp`**, as the README and the MCP server's README do. `--silent` keeps pnpm's own lines off the output the client reads the server's replies on, and a note says so. The note on other clients gives the `mcp.json` entry's arguments in the same form.
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -3160,6 +3160,7 @@ only their routes say otherwise", is wrong for two of them):
| `/deploy` | **per-site target + global chrome** | `listSites`; `export/CHANGELOG.md`; `export/out/index.html` mtime (`resolveOutDir` ignores siteId); `settings.buildPipeline.mode` *(gone 2026-09-28, O6c)* | `cutReleaseAction`, `setBuildModeAction`, `deployExportAction(siteId)` + four from `buildAction` | optional |
| `/build` | **corpus-wide, no site notion at all** | `activeQueueNames()`, `liveJobRows(BUILD_KINDS)` — no `searchParams`, no `listSites` | eleven actions in `buildAction.ts` | **never** |
| `/homepage` | **global — the hub's identity** | `getHomepageConfig` → `sites/_homepage/homepage.json` | `saveHomepageConfigAction` | **never**; page was sync |
+> **Superseded by release 18 (2026-10-06):** the `/deploy` row's `export/out` is now a link to the bundle built last; a deploy ships `<exportBuildsDir>/<target>/out` — see "The publish stages" at the end of this file.
**The seven import edges, and where each went.** `build/` and `deploy/` were already one module
in two directories: `build/buildAction.ts:34` imported `deploy/buildDeployCore`, and four
@@ -3205,6 +3206,7 @@ visited page. It is `revalidatePath("/sites/[siteId]", "layout")` now (the h1 ab
reads `siteTitle`, which that form edits). `buildStatsAction`'s `/charts` became
`revalidatePath("/sites/[siteId]/charts", "page")` (precedent: `channels/actions.ts:312`).
`revalidatePath("/sites")` now has **eight** callers (was five).
+> **Superseded by release 18 (2026-10-06):** `buildStatsAction` is deleted; the stats datasets are built inside the `update-index` stage — see "The publish stages" at the end of this file.
**`seedsSiteParam` and the path rule beside it.** `SiteScopeSelect.tsx`'s
`seedsSiteParam = !pathname.startsWith("/sites")` is UNCHANGED and its reason still holds: the
@@ -3309,6 +3311,7 @@ unique: none of the ~15 buttons added to `/sites` contains "delete".
`<details>` hides**, so anything inside the Pool disclosure or "Individual steps" needs the
click first — which is why `e2e/helpers.ts` grew `buildIndex(page)` (goto `/sites`, click
"Pool jobs", press, assert "Done") and now imports `expect` as a value.
+> **Superseded by release 18 (2026-10-06):** `BuildSitesPanel`, "Build all sites", "Build selected" and "Build stats dataset" are gone from `/sites`; "Build index" enqueues the `update-index` stage and `buildIndex(page)` waits for `[stage] update-index _index: Done` — see "The publish stages" at the end of this file.
## Verified 2026-08-30 — transcode: what it was, what stayed
@@ -5355,6 +5358,7 @@ Three branches off `4ac8ceda`, merged in order: `1a011d96` alone, then `tags/rul
`opsFail`, whose default status is 400 (`:35-41`).
- `build-deploy` returns `{ok: true, jobs, skipped}` plus `jobId` only when `jobs.length === 1`
— a caller that reads `jobId` unconditionally breaks on a multi-site body.
+ > **Superseded by release 18 (2026-10-06):** with `all`, `build-deploy` and `build-site` also answer a top-level `jobId`, the run's last job — see "The publish stages" at the end of this file.
### `wt rm` finds the directory `wt add` made
@@ -5570,6 +5574,7 @@ checkout it runs in. Cloudflare Pages treats a deploy to the project's productio
branch as production and any other branch as a **preview**. So the pre-existing
hazard is real and is unchanged: a "production" deploy run from a feature-branch
checkout silently lands as a preview. Only the preview path passes `--branch`.
+> **Superseded by release 18 (2026-10-06):** every deploy names its branch (`--branch main` for production) and runs the pinned wrangler, never `pnpm dlx` — see "The publish stages" at the end of this file.
**The rules live in `common/lib/pagesDeploy.ts`, which is node-free on purpose.**
Four pure functions — `previewBranchProblem`, `pagesDeployArgs`, `previewAliasUrl`,
@@ -5606,6 +5611,7 @@ logged once at the top of every preview deploy, by both actions.
`previewBranchProblem` and return `{ ok: false, error }` — build-and-deploy checks
*before* the build, so a typo does not cost one. **Job kinds are unchanged**
(`deploy-export`, `build-deploy`): a preview is not a different operation.
+> **Superseded by release 18 (2026-10-06):** `deployExportAction` and `buildAndDeployAction` are gone and those two kinds are no longer created; a deploy is a `publish-deploy-site` stage — see "The publish stages" at the end of this file.
**`/api/ops/deploy-site` is new — deploy-only, of the already-built `export/out`.**
`POST { siteId | siteIds, preview? }`, answering build-deploy's shape plus
@@ -5614,6 +5620,7 @@ previewing is only useful if the SAME bundle can then go to production unrebuilt
and `build-deploy` always builds. `build-deploy` gained `preview` too, but **refuses
it alongside `all`**: the all-sites runner has nowhere to thread a branch, and
building every site to ship it to production is the worst reading of the request.
+> **Superseded by release 18 (2026-10-06):** `/api/ops/deploy-site` is an alias over the deploy stage, which ships the site's own stamped bundle `<exportBuildsDir>/<id>/out`, never `export/out` — see "The publish stages" at the end of this file.
**The fan-out loop is `fanOutSiteJobs` in `editor/app/api/ops/_lib.ts`**, shared by
both deploy routes — extracted, not copied, because the careful parts (one site's
@@ -5635,6 +5642,7 @@ decided before a job exists and proves it by watching the `.jobs/*.meta.json` si
not appear; a valid preview is shown to reach the action by the refusal it gets there
("no Cloudflare Pages project"), which is distinguishable from both "unknown key" and
a branch complaint. `site-publish-preview.spec.ts` covers the control itself.
+> **Superseded by release 18 (2026-10-06):** wrangler has a fake now (`editor/e2e/fixtures/bin/fake-wrangler.mjs`, via `WRANGLER_BIN`), and `publish.spec.ts` deploys through it — see "The publish stages" at the end of this file.
**Who can open a preview is a Cloudflare project setting, not ours** — *preview
deployment access*, **public by default**. Documented in `DEPLOY_CLOUDFLARE.md` →
@@ -5649,6 +5657,7 @@ one site at a time into the shared `export/` tree. So a deploy-ONLY action ships
whatever was built last, to whichever project was asked for: building jeralyzer and
then deploying anilyzer put jeralyzer's bundle on anilyzer's Pages project, in
production, with a green log.
+> **Superseded by release 18 (2026-10-06):** each target has its own bundle, `<exportBuildsDir>/<target>/out`, and a deploy ships only that; `export/out` is a link to the bundle built last — see "The publish stages" at the end of this file.
**The bundle names itself.** `compose-site.ts` writes the federation contract
`site.json` (carrying `siteId`) into the public dir, and `next build` copies
@@ -5663,6 +5672,7 @@ that). It refuses rather than rebuilding: "deploy the export" is the operator
saying *ship what is there*. `buildAndDeployAction` needs no check — it builds.
The ops route surfaces it as the 400/`skipped` reason it already returns for any
action error.
+> **Superseded by release 18 (2026-10-06):** `deployExportAction` is gone; the deploy stage runs `builtBundleProblem` over the target's own bundle — see "The publish stages" at the end of this file.
## One-core Phase 3 slices 2 + 4a (verified 2026-09-23, `main` @ `ae2fa5a9`)
@@ -6626,6 +6636,7 @@ Line numbers are `plans/FACTS.md` lines at `e172749b`, before this record's in-p
homepage`; `sync tick`; `settings example [--check]`. Exit 0 ok, 1 failed/refused, 2 usage.
Not yet: `doctor`, `run`, `mcp` (Phase 4 slice 3). The ten bins export `main(opts)` and
auto-run through `runIfEntryPoint`, so `tsx bin/x.ts` still works.
+ > **Superseded by release 18 (2026-10-06):** `build site|all` and `deploy site|hub|homepage` are printed aliases of `archilyzer publish …`, and a stage exits 0, 1, 2, 3 or 130 — see "The publish stages" at the end of this file.
- **`export/package.json` `build` IS the CLI** (`tsx ../common/bin/archilyzer.ts build site`,
`SITE_ID` from env); `build:hub` → `… build hub`, `deploy` → `… deploy site` (refuses without a
project). **`prebuild` and `build:nodata` are gone** — the data phase is an explicit step of
@@ -6633,6 +6644,7 @@ Line numbers are `plans/FACTS.md` lines at `e172749b`, before this record's in-p
build`). `docker/build-site.sh` / `publish-site.sh` call the CLI. `common/publish/build.ts` has
the named entry points `buildSite`, `deploySite`, `buildAll`, `composeHub`, `buildHub`,
`deployHub`, `composeHomepage`, `buildHomepage`, `deployHomepage`; the editor's jobs call them.
+ > **Superseded by release 18 (2026-10-06):** `buildSite`, `deploySite`, `buildAll`, `deployHub` and `deployHomepage` are deleted from `build.ts` (`514e9674`); the editor's jobs are publish stages — see "The publish stages" at the end of this file.
- **The hub builds and deploys.** `buildHub` = compose-hub + `INSTANCE_MODE=hub next build` into
**the shared `export/out`** (it removes `public/site.json` first; compose-site removes
`public/hub-sites.json`). `deployHub` reads `homepage.json` `cloudflareProject` — live value
@@ -6645,6 +6657,7 @@ Line numbers are `plans/FACTS.md` lines at `e172749b`, before this record's in-p
first deploy:** wrangler offers to create one only on a TTY, and a job or the CLI spawns it with
piped stdin, so it fails fast. A production `deploy hub` takes its branch from the git checkout;
`deploy homepage` passes `--branch main` to `archilyzer`.
+ > **Superseded by release 18 (2026-10-06):** the hub's build is installed as `<exportBuildsDir>/_hub/out`, and every production deploy, the hub's included, passes `--branch main` — see "The publish stages" at the end of this file.
- **Posts-only channels publish their manifest.** `common/bin/compose-site.ts`
`reconcileChannelTree` copies a manifest-only tree under `MANIFEST_ONLY_SIGNATURE` instead of
removing it; `corpus.json` still advertises `manifests.transcripts` unconditionally, spec 4.
@@ -6660,6 +6673,7 @@ Line numbers are `plans/FACTS.md` lines at `e172749b`, before this record's in-p
runs only when `../export/out/hub-sites.json` is absent or a `site.json` is present, else it
records `subcase-not-exercisable` (`211d4666`) — before that guard it started a real deploy job.
Any spec that can reach a deploy must check the build dir before calling with a valid project.
+ > **Superseded by release 18 (2026-10-06):** a deploy reads the target's stamped bundle, never `export/out`; the guard is gone and the sub-case runs in any checkout — see "The publish stages" at the end of this file.
- **tsc in the primary checkout reports errors in `editor/.next/dev/types/validator.ts`** — a
stale generated file left by an old `next dev`, naming routes that moved. 0 source errors;
worktrees without that file are clean. Clear it with `rm -rf editor/.next/dev` when no dev
@@ -7352,6 +7366,7 @@ out. O3's facts are the section just above ("O3 — runner lows"). Anchors are a
`build-homepage` on the `build` queue (`:47-48`), `deploy-homepage` (`:69-70`) and
`build-deploy-homepage` (`:90-91`) on `deploy`; the bodies are `buildHomepage` / `deployHomepage`,
unchanged, and a build-deploy deploys only on exit 0. No `jobKinds.ts` entries (the hub's have none).
+ > **Superseded by release 18 (2026-10-06):** `homepageDeployActions.ts` is gone; the homepage is built and deployed by the `publish-build-homepage` / `publish-deploy-homepage` stages, and the three old kinds stay in `jobKinds.ts` for archived metas only — see "The publish stages" at the end of this file.
- **Refusals before any job:** a bad preview name (`previewBranchProblem`), and for a deploy-only
`builtHomepageProblem(homepageOutDir(paths))` (`common/lib/builtExport.ts:93`; `builtHomepageAt`
`:106` is index.html's mtime; `homepageOutDir` `common/publish/build.ts:948`, exported so the editor
@@ -7820,6 +7835,7 @@ on anchors elsewhere in this file:
(Build stats dataset, or a pool composer) before the next index build counts it here, and
that index build heals it. Under `ARCHILYZER_INDEX_ALLOW_HELD`, a full rebuild drops all of
the held channel's records the same way (see "The index build's hold").
+ > **Superseded by release 18 (2026-10-06):** the "Build stats dataset" button is gone; the stats build runs inside the `update-index` stage, after the index build — see "The publish stages" at the end of this file.
It stays until fixed, and is logged as such rather than as pending (`:500`).
- **A transcript always has a date, and a caption video takes its captions' arrival.**
@@ -7942,6 +7958,7 @@ this section is stale, by +16 near the top and +203 at the end; they are not rew
and homepage's `build:index`, so every site build's data phase) prints the log; the editor's
**Build index** job (`buildIndexAction`) streams it into the job log, which `pnpm ops build-index
--wait` follows. Nothing reads the result's field outside the tests.
+ > **Superseded by release 18 (2026-10-06):** `buildIndexAction` is deleted and a site build runs no data phase; Build index is the `update-index` stage child, and its stamp records `index.heldChannels` — see "The publish stages" at the end of this file.
- **The words are shared with the stats build** (`lib/channelMediaHold.ts`): `HELD_REASON` is a
`Record<ChannelMediaStatus, string>`, so a new status added to `inspectChannelMedia` fails tsc
until it has a reason; `isMediaHeld` treats any status but `ok` and `in-place` as held.
@@ -8606,3 +8623,332 @@ phase deletes from the destination.
`"original"`. `downloadOneManaged`'s `persistFormatPreset` takes the result; unset is the historical
`bestvideo*+bestaudio/best`. The download lane's keep-latest persists pass nothing, so they stay
`"original"`.
+
+## The publish stages (verified 2026-10-06, branch `r18/surfaces` @ `5a77682e`)
+
+The record is [`release-18.md`](release-18.md) ("Slice S1/S2/S3/S5, as shipped"). Every anchor below
+was read at that tip. Release 18 replaced the one-shot build and deploy jobs with seven queueable
+stages driven by stamps on disk; the facts above that describe `export/out` as the shared deploy
+source, a site build's data phase, Build all, `pnpm dlx wrangler` or the in-process Build index /
+Build stats jobs carry a "Superseded by release 18" line pointing here.
+
+### The seven stages and their targets
+
+- **`common/publish/stages.ts` is the only module that knows every stage.** `StageKind`
+ (`:33-40`): `update-index`, `build-site`, `deploy-site`, `build-hub`, `deploy-hub`,
+ `build-homepage`, `deploy-homepage`. `STAGES` (`:406-420`) gives each a label, `jobKind:
+ publish-<kind>` and `queueKey: "publish"` (`stage()`, `:390-404`), a pure `needs()`, `argv`
+ (`stageArgv`, `:305-316`) and a `run` that lazily imports `stageBodies.ts`.
+- **Targets** (`stamps.ts:22-25`): `_index` (update-index), a site id (build-site, deploy-site),
+ `_all` (build-site only), `_hub`, `_homepage`. `fixedTarget` (`stages.ts:334-339`) pins the hub's
+ and the homepage's kinds to their one target, and `parseStageArgs` (`:345-384`) refuses any
+ other target for them ("stage deploy-homepage: the target is _homepage") and a `_`-target for a
+ site kind except `build-site _all`. `--run-id` is required; `--preview` with `--to local` is
+ refused.
+- **`StageRequest`** (`:56-73`): `{kind, target, runId, preview?, to?: pages|local, runner?:
+ local|docker, force?, skipArchives?, indexAfter?, builtAfter?, allowMissingMedia?}`. `indexAfter`
+ / `builtAfter` are ms times: the on-disk preconditions of a run.
+
+### The stamps — `common/publish/stamps.ts`
+
+- **Three files** (`:128-142`): `<exportIndexDir>/stamp.json` (`IndexStamp`, written by
+ update-index), `<exportBuildsDir>/<target>/built.json` (`BuiltStamp`, the build stages) and
+ `<exportBuildsDir>/<target>/deployed.json` (`DeployedFile`, the deploy stages). `target` is a
+ site id, `_hub` or `_homepage`. By default `exportIndexDir` is `export/.export-index` and
+ `exportBuildsDir` is `export/.export-builds` — siblings of `export/public` (`paths.ts:212-214`,
+ `:253-255`); `EXPORT_INDEX_DIR` / `EXPORT_BUILDS_DIR` override them (the compose file sets
+ `/data/builds/.export-index` and `/data/builds/.export-builds`, `docker-compose.yml:71-72`).
+- **`IndexStamp`** (`:27-54`): `stampId`, LMDB `generation`, `scannedAt` (`INDEX_SCANNED_AT_KEY`:
+ when the completed scan began), `builtAt`, `templatesAt`, `commit`, `index {shortCircuited,
+ added, changed, removed, heldChannels}`, `stats {shortCircuited, notIndexedYet, notIndexable}`,
+ `sites[id] {siteFp, statsFp, inputSig}`, `hubSig`, and `settingsSig?` (`indexSettingsSig` over
+ the settings the index read; a stamp without one reads as changed, once).
+- **`BuiltStamp`** (`:59-83`): `stampId`, `target`, `kind: site|hub|homepage`, `indexStampId`,
+ `inputSig`, `builtAt`, `checkedAt?` (a later no-op build found the bundle still current),
+ `commit`, `branch`, `runner: local|docker`, `audience`, `corpusGeneratedAt`, `files`, `bytes`,
+ `archivesStaged`, `sourceCommit?` (homepage). **`DeployedFile`** (`:118-124`) holds one
+ `production`, one `local` and `previews[branch]`; `recordDeploy` (`:263-275`) replaces one slot
+ and keeps the others.
+- **Malformed = null = stale.** Writes are atomic (`writeJsonAtomic`); every read goes through
+ `readAs` (`:216-219`) and a shape check (`asIndexStamp` `:163`, `asBuiltStamp` `:182`,
+ `asDeployedFile` `:205`): a missing, unparseable or wrongly-shaped file is `null`, and `null` is
+ "no stamp" to every `needs()` (header `:8-11`). A build removes its old `built.json` before the
+ swap and writes the new one last (`build.ts:935`, `:966`; `stageBodies.ts` `stampBuilt`).
+- **`commit` / `branch`** come from `checkoutInfo` (`stageBodies.ts:100-109`):
+ `ARCHILYZER_COMMIT` / `ARCHILYZER_BRANCH` WIN over git, each on its own (the image bakes them;
+ the e2e server sets the branch); else git's HEAD, and a detached HEAD records `branch: null`.
+- **The update-index stage keeps its stamp id when nothing changed** (`stageBodies.ts:277-290`):
+ both builds short-circuited, the same `generation`, every site's `inputSig` and the `hubSig`
+ unchanged. The outcome is `noop`, and the builds made from that index stay current. `builtAt` is
+ still rewritten (`:297`), so a run's `indexAfter` is met by a no-op index update.
+
+### The publish lock — `common/publish/stageLock.ts`
+
+- **`<exportBuildsDir>/.publish.lock`** (`:58-60`), JSON `{pid, host, kind, target, since,
+ pidStart}`, taken with `open(…, "wx")` (`:218`) by every stage: the child the editor spawns and
+ the CLI (`stageRun.ts:96`). One publish stage at a time per builds dir, whoever started it.
+- **The host is `ARCHILYZER_HOST_ID` when set, else `os.hostname()`** (`lockHostId`, `:92-94`).
+ The compose file sets `ARCHILYZER_HOST_ID: archilyzer-editor` on the editor service only
+ (`docker-compose.yml:127`).
+- **Stale only on the same host with the holder gone** (`holderIsGone`, `:135-147`): the pid is
+ dead (`processIsAlive`), or answers with another `/proc/<pid>/stat` start time, or names a
+ process that started more than 2 s (`START_SLACK_MS`) after the lock's `since`. Without `/proc`
+ only pid-alive is asked. **Another host's lock is never judged stale**; the wait line names both
+ hosts and the file to remove (`:252-259`). A torn (unparseable) lock is taken over after 60 s
+ (`LOCK_TORN_GRACE_MS`, `:49`, `:238-245`). A live holder is waited for: a 5 s poll
+ (`LOCK_POLL_MS`), one log line, and the stage's signal cancels the wait (exit 130,
+ `stageRun.ts:102-104`). Release removes the file only if it is still this holder's (`:271-276`).
+
+### Bundles — `<exportBuildsDir>/<target>/out`
+
+- **Every deploy ships `<exportBuildsDir>/<target>/out`** (`bundleDir`, `build.ts:745-747`), the
+ homepage excepted: its bundle stays `homepage/out` (`homepageOutDir`, `:630-632`) and only its
+ stamps live in `<exportBuildsDir>/_homepage/`. The hub's bundle is `_hub/out`.
+- **The local runner still builds into the checkout's `export/out`** (Next's `distDir` may not
+ leave the project), then installs it (`installBundle`, `:804-823`): moved to `out.next`, then
+ `out → out.prev`, `out.next → out`, `out.prev` removed; a leftover `out.next` is deleted first; a
+ rename that fails EXDEV is a copy + remove. `recoverInterruptedInstall` (`:790-798`) renames an
+ `out.prev` with no `out` back before any build or install.
+- **`export/out` is a relative SYMLINK to the bundle built last** (`pointExportOutAt`,
+ `:836-844`); `unlinkExportOut` (`:829-833`) removes the link before each build, so a failed build
+ cannot leave an older bundle readable there. `.gitignore:62` is `/export/out` (no trailing slash:
+ the link is not a directory to `**/out/`). Seen 2026-10-06 in a worktree after an e2e run:
+ `export/out -> ../editor/test-transcripts/.export-builds/pubsite/out` — the test server's builds
+ link the CHECKOUT's `export/out` (`exportDir` is always `<repo>/export`, `paths.ts:206`).
+- **A site's oversize archives** are staged at `<exportBuildsDir>/<id>/.r2-staging/<id>/archives`
+ (`dockerSiteStagingDir`, `:43-45`) for both runners (`stageSiteArchives`, `:852-869`).
+- **`buildSiteBundle`** (`:909-949`) runs `buildSiteSteps` with `skipData: true` (compose:site,
+ then `next build`; the data phase is update-index's), `builtBundleProblem` over `export/out`, then
+ install, staging, link. **`buildHubBundle`** (`:951-971`) runs `buildHub` (removes
+ `public/site.json`, compose:hub, `INSTANCE_MODE=hub` `next build`), `builtHubProblem`, install,
+ link.
+
+### What makes each stage stale — `needs()`
+
+- **update-index** (`needsIndex`, `stages.ts:163-177`): no stamp; forced; an ingest job ended
+ `done` after `stamp.scannedAt` ("new data since the last index"); a config file newer than
+ `scannedAt`; or `settingsSig` differs from the stamp's ("the settings the index reads changed").
+ The ingest signal is every job meta whose kind is in `INGEST_KINDS` (`jobs/jobKinds.ts:943`,
+ `isIngestKind` `:980`) ended `done`, plus each `channels/<slug>/snapshot.json` mtime (the lane
+ units that make no job record; `publishState.ts:194-209`, `:312-315`). The index's config files:
+ `tags.json`, `search-aliases.json`, `duplicates*.json`, every `sites/*/site.json`,
+ `homepage.json`, the charts config (`publishState.ts:317-331`). **The settings are judged by
+ signature, never the file's mtime:** `indexSettingsSig` (`inputSig.ts:171-184`) over
+ `socialLinks`, `homepageUrl`, `buildArchives`, `archiveStorage`, `social.x.visibility`,
+ `maxTranscriptPageBytes` and the storage locations' `[id, root]`.
+- **build-site** (`needsBuildSite`, `:211-227`): BLOCKED "update the index first" with no stamp
+ (forced too), "waiting for the index update this run started" while `stamp.builtAt <
+ indexAfter` (`indexGate`, `:180-187`), and when the stamp has no entry for the site. Then stale
+ (`builtStale`, `:199-209`), in order: forced; never built; `changedChannels` ("N channels changed
+ (a, b, …)"); a config file newer than `builtCheckedAt(built)` = max(`builtAt`, `checkedAt`)
+ (`:195-197`); `built.inputSig !== stamp.sites[id].inputSig` ("data changed"); a bundle problem.
+ `_all` is stale when any site is. A different `built.commit` is NOT stale (the status's
+ `codeNewer` chip, `publishPlan.ts:444`).
+- **build-hub** (`:229-235`): as a site, against `stamp.hubSig`. `hubInputSig` (`inputSig.ts:202-214`)
+ signs the stamp id, `homepage.json`'s bytes and each listed site's id + `siteUrl` + title — so
+ EVERY new index stamp id stales the hub.
+- **build-homepage** (`:237-249`): `built.indexStampId !== stamp.stampId`; `built.sourceCommit` ≠
+ `main`'s head when a repository answers; a bundle problem. Its `built.inputSig` is the stamp id
+ (`stageBodies.ts:522`).
+- **deploy-*** (`needsDeploy`, `:261-299`): BLOCKED by a private target (`deployProblem`), no Pages
+ project (`pagesProblem`, not for `--to local`), no build ("no build of X — archilyzer publish
+ build X"), `builtAfter` not met (unless the bundle matches the current index and that index ran
+ at or after `builtAfter` — a run's no-op build), a bundle problem, and **a production deploy of a
+ build whose `branch` is not `main` — `null` included** (`:287-293`). Then stale when forced or
+ when `deployed[kind/branch].builtStampId !== built.stampId`.
+- **`inputSig`** (`siteInputSig`, `inputSig.ts:100-146`): the site's whole `.export-index/sites/<id>/`
+ tree (`chart-templates.json` by its bytes), each PUBLISHED member's shared transcripts / subs /
+ posts / digests tree (`manifest.json` ignored), `site.json`'s bytes, the `sites/<id>/` dir, the
+ global aliases, curated tags and duplicates files (size + mtime), `archiveStorage` + X visibility
+ + `buildArchives`, the resolved social links, hub url and footer siblings. A superset: a needless
+ rebuild, never a wrong skip.
+
+### Who asks `needs()`, with what
+
+- **The status asks with every signal; a stage child asks with signatures only.** `needsInputOf`
+ (`publishPlan.ts:296-321`) fills `changedChannels` (a site's members with an ingest after
+ `builtCheckedAt`; the hub's: the listed sites' members; the homepage's: every channel with a
+ signal), the config mtimes and `settingsSig`. The child's `readNeedsInput`
+ (`stageBodies.ts:132-173`) leaves `changedChannels` empty, `configChangedAt` null and
+ `settingsSig` absent: the child rebuilds only when forced, never built, its signature moved or
+ its bundle is wrong. A build the plan chose for changed channels that the index update then
+ finds unchanged is a no-op that writes `checkedAt` (`markChecked`, `:416-418`), which clears the
+ chip.
+- **update-index's body never asks `needs()`** (`runStageBody`, `stageBodies.ts:549`): run, it
+ always runs `buildIndex` → `buildStats` → the chart templates in one child, then the stamp.
+- **Every other body asks first** (`:566-592`): blocked → `StageFailure` exit 3; fresh and not
+ forced → `noop` (a build's no-op writes `checkedAt`).
+
+### Exit codes
+
+- `STAGE_EXIT` (`stageRun.ts:25-31`): **0** ran or no-op, **1** failed, **2** usage, **3**
+ precondition not met, **130** cancelled. A `DeployStageError` carries its own code and its
+ sentence is not printed twice (`:125-133`).
+- **`pnpm --filter … exec` collapses 2, 3 and 130 to 1.** Measured 2026-10-06 on pnpm 11.26.0 in a
+ scratch workspace: `pnpm --filter <pkg> exec sh -c 'exit 3'` exits 1 with
+ `ERR_PNPM_RECURSIVE_EXEC_FIRST_FAIL … Command failed with exit code 3` (130 the same); a plain
+ `pnpm exec` in a single package passed 2, 3 and 130 through. The root `archilyzer` script is
+ `pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts` (`package.json:8`), so
+ `pnpm archilyzer publish …` reports 1 for any non-zero code. The editor spawns `tsx` directly
+ (`stageCommand`, `stageRun.ts:50-68`) and sees the real code.
+
+### The queue, the runs, the lane
+
+- **One stage = one `runManagedCommand` job on queue `publish`** (`enqueueStage`,
+ `publishStages.ts:113-139`), the child `<common>/node_modules/.bin/tsx bin/archilyzer.ts stage
+ <kind> <target> …`, cwd `common/`, update-index with `--max-old-space-size=8192`
+ (`stageRun.ts:50-68`). Cancel kills the child; the child traps SIGTERM and tree-kills its
+ children (`stageMain`, `:156-178`). The spec is `{kind: publish-<stage>, slug: target, params:
+ the request}` (`stageSpec`, `:82-86`) — Retry replays it. A duplicate (same kind, target and
+ destination queued or running) is refused `{ok: false, info: true, jobId}` (`:118-126`).
+- **Order is on disk, not in memory.** `enqueuePublishRun` (`:162-196`) enqueues a whole plan
+ under one run id; builds carry `indexAfter`, deploys `builtAfter`, and a child whose
+ precondition is unmet exits 3. /jobs shows `run <last six of the run id> · <target>`
+ (`jobs/jobDetail.ts:24-29`). At boot a queued publish stage is cancelled — "server restarted; the
+ publish lane re-derives stages from on-disk state" — never re-queued (`bootQueuedJobs.ts:63-64`,
+ `:292-293`).
+- **`publish` is a PIPELINE lane, not in `LANES`** (`PIPELINE_LANES`, `lib/autoQueueTypes.ts:219`;
+ `LANES` `:203`). `PauseLane = AutoQueueKind | PipelineLane` (`lib/pauseGates.ts:53`);
+ `isGateHeld` / `withGateHeld` read and write `settings.publish.held` for it (`:104`, `:125-127`).
+- **`settings.publish`** (`defaultPublish`, `lib/settingsSchema.ts:1624`; `sanitizePublish`
+ `:1651`): `{enabled: false, held: false, checkEveryMinutes: 10 [1–1440], refreshEveryMinutes: 360
+ [0–43200], quietHours: null, runner: "local", previewBranch: "preview", hub: "off", homepage:
+ "off"}`. A site's policy is `site.json` `publish.auto` (`sitePublishPolicy`,
+ `lib/siteSchema.ts:239`: absent = off); `preview` / `production` clamp to `build` for a private
+ site or one with no `cloudflareProject` (`clampSitePublishPolicy`, `:228-236`).
+- **The runner** (`publish/publishRunner.ts`): kind `auto-publish`, queueKey `""`
+ (`startPublishRunner`, `:286-314`; `jobKinds.ts` `auto-publish` drainable). It wakes every
+ `checkEveryMinutes`; a pass is due (`publishPassDecision`, `publishPlan.ts:555`) with no stamp,
+ when the index is stale and the stamp is `refreshEveryMinutes` old, or when a policy target is
+ left stale and the last pass is that old; never while off, held, in quiet hours or with any
+ publish stage queued or running (`laneBlockedReason`, `:534`). **A pass re-reads the status
+ and re-plans before EACH stage** (`runPublishPass`, `publishRunner.ts:110-219`), dispatches one
+ stage (`background: true`) and awaits its end; a failed index update ends the pass; a failed build
+ drops its deploy. **A hold, quiet hours or the lane switched off stop the dispatching, never a
+ stage** (`gateShut`, `:89-94`). A drain lets the stage in flight finish; **Stop ends the runner
+ and does not cancel the stage in flight** — it is its own job (`:178-191`, `stopPublishRunner`
+ `:322-327`). Started from `editor/instrumentation.ts:226-237`, below the idle-boot return
+ (`:191`).
+- **The plan** (`planPublishRun`, `publishPlan.ts:758`): update-index when the index is stale;
+ per site (by id) its build when `needs()` says stale and its policy is not off (else skipped
+ "stale, and its policy is off", unless `builds: "stale"`), and its deploy where `publish.auto`
+ says; `settings.publish.runner: "docker"` plans one `build-site _all --runner docker`; then the
+ hub, then the homepage, by `settings.publish.hub|homepage`. Never forced.
+
+### The surfaces
+
+(Anchored by symbol; verified at the S4 merge, `dddbc183`.)
+
+- **Every editor surface, and the ops routes through `api/ops/_publish.ts`, enqueue through
+ `editor/app/sites/lib/publishCore.ts`**: `wantedPlan` puts the index update first when the index
+ is not fresh and a build is asked, builds `--force`, and gives a deploy `builtAfter` when its
+ build is in the run; `refusal` answers with the deploy stage's own `resolveDeployRequest`
+ sentence before any job. `publishActions.ts`: `publishNowAction` = `planPublishRun(status,
+ {deploys: "policy"})`, `buildAllStaleAction` = `{builds: "stale", deploys: "none"}`,
+ `updateIndexAction` (the Pool's Build index) = the update-index step always (`indexPlan`; the
+ child runs, and ends `noop` when nothing changed), `deployTargetAction` deploys with `force:
+ true`. Nothing builds in the editor's process. `publishRunStream.ts` `followRun` joins a run's
+ jobs into one console.
+- **`editor/app/api/ops/publish/route.ts`**: `POST {verb: index | build | deploy | hub | homepage
+ | now | stale, …}` (`VERBS`); `GET` returns `readPublishStatus`. The old
+ routes (`build-site`, `build-deploy`, `deploy-site`, `build-hub`, `deploy-hub`, `build-homepage`,
+ `deploy-homepage`, `build-index`) are aliases over the same stages; `build-site` accepts and
+ ignores `skipData`.
+- **One status:** `readPublishStatus` (`publish/publishState.ts:400-406`) is what the /sites panel,
+ the ops route, `archilyzer publish status` and the lane read.
+- **A console's Cancel is the run's**: every publish console (the /sites rows, Publish now, Build all stale,
+ the Pool's Build index, a site's Publish tab) cancels through `cancelPublishRunAction`
+ (`sites/lib/publishActions.ts`), which cancels every queued or running `publish` job whose
+ `spec.params.runId` is the console job's, newest first. `followRun` also cancels a run's later jobs when one
+ ends `cancelled`, but never a part another run queued (`RunPart.existing` — a duplicate stage answered by
+ `enqueuePublishRun` with the existing job's id), and the console's job is the first this run enqueued.
+- **Refused before any job** (`publishCore.ts` `refusal`): an unknown site (`no site "<id>"`), a preview with
+ no or a bad branch name (`previewBranchProblem` — never let fall to production), and every
+ `resolveDeployRequest` sentence, except "no build of X" when the run builds first. The ops route also refuses
+ a local deploy with a preview, and `{build, runner: "docker"}` puts the index update first when the index is
+ not fresh (`indexAfter` on the `_all` build). One target's throw is that target's refusal (`_publish.ts`).
+- **CLI:** `publish hub|homepage --deploy-only [--force]` ships the bundle as built (`--force` re-ships);
+ `--deploy` with `--deploy-only` is a usage error (exit 2); `deploy hub|homepage` are its printed aliases.
+ `build hub` and `build homepage` stay RAW builds that stamp nothing (export's `build:hub`, which e2e:2origin
+ runs) — a deploy stage never ships them.
+
+### The deploy stage
+
+- **`runDeployStage`** (`publish/deployStage.ts`, order in its header `:16-40`) asks the request
+ (exit 2), the target's refusals (exit 3), `built.json` (exit 3 "no build of X in <dir>"), the slot
+ already holding this `stampId` (a no-op unless forced), production-only-from-`main` (exit 3), the
+ bundle guards (exit 3), then `--to local` (into `ARCHILYZER_SITE_OUT` / `ARCHILYZER_HOMEPAGE_OUT`,
+ unset = exit 3) or the credential preflight (exit 1), R2, the pinned wrangler, the live check, and
+ LAST `recordDeploy`. Nothing is written to `deployed.json` on a refusal.
+- **wrangler is pinned**: `common/package.json:94` `"wrangler": "4.147.0"`; `wranglerBin`
+ (`lib/pagesDeploy.ts:101-106`) = `WRANGLER_BIN`, else `<repo>/common/node_modules/.bin/wrangler`;
+ `pagesDeployArgs` always passes `--branch` (`main` for production, `PRODUCTION_BRANCH`, `:68`). Nothing
+ spawns `pnpm dlx` any more: `git grep "pnpm dlx"` over `common/`, `editor/`, `export/`,
+ `homepage/`, `scripts/` and `docker/` finds one comment (`pagesDeploy.ts:21`) and the changelog.
+- **The live check** (`publish/liveCheck.ts`): `<url>/corpus.json` plain and `?cb=<builtStampId>`
+ (`&try=N` on a retry, `:67-70`), 3 tries 10 s apart (`:57-58`); verdicts `ok`, `stale-edge`,
+ `mismatch`, `unreachable`, `skipped` (`E2E_LIVE_CHECK=skip`, `:163`) — a warning, never a failure.
+ The hub's deploy also probes its tombstones (`deployStage.ts:537`).
+
+### The hub's bundle check — `builtHubProblem`
+
+- `common/lib/builtExport.ts:422-454` refuses a hub bundle that holds a `site.json` (any site's), or
+ no `hub-sites.json`, or any of `HUB_FORBIDDEN_TREES` (`:510-519`: `summaries`, `transcripts`,
+ `subs`, `posts`, `digests`, `stats`, `archives`, `media`) — except a `posts/` that
+ `isTombstonePostsTree` (`:460-501`) finds holds tombstones only (an empty posts manifest; per
+ channel a `pageCount: 0` manifest with no `slugToPage` and only `[]` `page-N.json` files).
+- **`reports/` and `m/` are not refused as trees** — the export app renders `reports/index.html`
+ and the `_none` placeholder pages into every build. `hubReportDataIn` (`:541-571`) looks for report
+ DATA by name instead: `reports/index.json`, `m/index.json` (`REPORTS_INDEX_PATH` /
+ `MOMENTS_INDEX_PATH`, `lib/report/views.ts:71-72`), under `reports/` any `page.json`,
+ `citations.json`, `citations.csv`, `report.html`, `report.pdf`, `report.md`, `evidence-pack.zip`
+ or `history.json` below the top level, any `reports/**/history/repo/`, and any
+ `m/**/moment.json`.
+- Its sentences still say "export/out"; the deploy stage rewrites that prefix to the bundle's
+ directory (`deployStage.ts:436`).
+
+### Tombstones — `common/publish/tombstones.ts`
+
+- **The hub tombstones only X channels that are a member of at least one non-private site**
+ (`withdrawnXChannels`, `:157-183`): only while `social.x.visibility` is private, only channels
+ with a shared posts tree, and the site list is `listSites(paths)` — every configured site
+ (`bin/compose-hub.ts:142`), filtered by `isPrivateSite` alone (`audience === "private"`,
+ `lib/siteSchema.ts:94-96`). So **a member of an unlisted site or a cited (report-only) site is
+ still tombstoned**; a channel only private sites carry, or no site, is never named on the hub.
+- A tombstone (`writePostsTombstones`, `:89-116`) is the channel's posts manifest at `pageCount: 0`,
+ `maxPageBytes: 0`, `slugToPage: {}`, plus `[]` for every page below the SHARED tree's
+ `pageCount`; the hub adds an empty `posts/manifest.json` (`compose-hub.ts:137-155`). `_headers`
+ serves them `Cache-Control: no-store` (`renderHeadersFile` `noStore`, `lib/archive/headers.ts:103`;
+ the hub `/posts/*`, a site `/posts/manifest.json` + `/posts/<slug>/*`, `tombstones.ts:137-145`).
+
+### The `publish/` layering
+
+- `common/architecture.test.ts` `FORBIDDEN` (`:31-52`): `lib` may not import `publish` (`:38`),
+ `jobs` and `controller` may not import `publish` (`:39-40`), and `publish` may not import `views`
+ or `components` (`:51`). `views` may import `publish`: `views/publishStatus.ts` re-exports
+ `publish/publishPlan.ts`. That is why the status builder, the enqueuer and the runner live in
+ `publish/`, not `controller/`, and why the runner is started from `instrumentation.ts`, not from
+ `controller/autoRunner.ts`.
+- **Only static imports are checked** (`IMPORT_RE`, `:106-107`): a dynamic `import()` is not seen.
+ `stages.ts` and `stageBodies.ts` load `build.ts` and the bodies lazily for weight, not to dodge it.
+
+### The e2e seams — `editor/playwright.config.ts`
+
+- `E2E_SERVER_ENV` (`:68-80`) gives the editor's test server: `E2E_LIVE_CHECK=skip`;
+ `WRANGLER_BIN` → `e2e/fixtures/bin/fake-wrangler.mjs`; **`ARCHILYZER_BRANCH=main`** (a worktree's
+ branch is never `main`, and production refuses any other); **`CLOUDFLARE_API_TOKEN` a dummy**
+ (`e2e-fake-token-never-sent`: the preflight asks for one, the fake never sends it, and the suite
+ must not lean on the host's `wrangler login`); `EXPORT_NEXT_BIN` →
+ `e2e/fixtures/bin/fake-next.mjs`.
+- **`fake-next.mjs`**: `build` copies `EXPORT_PUBLIC_DIR` to `./out` (cwd `export/`) and writes an
+ `index.html` — compose runs for real, nothing is rendered. `nextBuildStep` (`build.ts:112-117`)
+ runs `<EXPORT_NEXT_BIN> build` when set, else `pnpm exec next build`.
+- **`fake-wrangler.mjs`**: `pages deploy <outDir> --project-name <p> --branch <b>` appends
+ `{argv, cwd, project, branch, outDir, files}` to **`.fake-wrangler.json` beside the bundle**
+ (`<exportBuildsDir>/<target>/.fake-wrangler.json`, `:106`) and prints wrangler 4's success lines;
+ **auth-fail mode** comes from the MODE sidecar `.fake-wrangler-mode.json` `{"authFail": true}`
+ (looked for in `EXPORT_BUILDS_DIR`, then `<dirname(EXPORT_PUBLIC_DIR)>/.export-builds`, then two
+ levels above the bundle, `:58-74`) or `E2E_FAKE_WRANGLER_AUTH_FAIL=1`: it prints
+ `Authentication error [code: 10000]` and exits 1 with no argv sidecar. `--version` answers
+ `4.147.0`. `publish.spec.ts` reads and writes both under `test-transcripts/.export-builds`
+ (`:32`, `:65`, `:174`).
diff --git a/plans/STATE.md b/plans/STATE.md
@@ -3,7 +3,26 @@
The working memory for the local-AI derived-corpus work. Rewritten at the end of every
session, before context is cleared. See [`README.md`](README.md) for the protocol.
-**Now (2026-10-02, 03:20): `main` is `a6155bf2` (+ plans commits) — release 17, the media tier, is complete
+**Now (2026-10-06): release 18 — publishing as queueable stages — is complete on `r18/integration`** (record:
+[`release-18.md`](release-18.md): slices S1 the stage contract, stamps, lock, bundles and CLI; S2 deploy hardening; S3
+the status, the queue and the publish lane; S4 the surfaces; S5 the image half; S6 the records — each reviewed, merged
+`--no-ff`; `main` `edadc712` merged in first). **Not on `main` yet, and nothing of it is live.**
+- **Owed, in order** (the plan's steps 5–6): fast-forward `main` to `r18/integration` from a session that is not
+ worktree-isolated (an empty `status --short` in the primary first; if `main` moved, merge it into the integration
+ branch and re-gate instead); remove the idle `r18-*` worktrees (keep `r18-integration` until the rollout is done);
+ then the rollout: ONE editor rebuild + restart (adapt `~/reports/release-17/scripts/r17-*.sh`), Publish now with
+ every policy off (= the index update alone; `/` answers under 5 s throughout), a jeralyzer build → preview deploy →
+ live-check verdict, the hub with its tombstone probes (open question 1 for real), production deploys site by site,
+ then policies (`build` for all six; `preview` where wanted; `production` for none until a week of lane runs reads
+ clean) and the lane on (`checkEveryMinutes` 10, `refreshEveryMinutes` 360); `archilyzer doctor`; build
+ `runtime-vulkan` and `runtime-cuda` once.
+- **What changes for the operator:** /sites → **Publish** replaces "Build all sites" and the hub/homepage sections
+ (Publish now, Build all stale, a row per target with index | built | deployed | live chips); **Build stats dataset is
+ gone** (the index update builds the stats); `/operations/publish` is the lane; a site's **Publish policy** is on its
+ form; `pnpm ops publish {"verb": …}`; `archilyzer publish …` (PUBLISH.md). Every stage is a child process under one
+ publish lock, so a build no longer starves the editor.
+
+**Previously (2026-10-02, 03:20): `main` is `a6155bf2` (+ plans commits) — release 17, the media tier, is complete
and LIVE: the editor on `:3001` (`Dt4zuo5uJcPKZ9fKk0L-f`, restarted 02:51 after the migration) and umtool on
`:3050` (02:40, `UMTOOL_MEDIA_DIR` set to a folder on the Platter).** Record: [`release-17.md`](release-17.md)
(slices D0, T1, T2, T3, U1, U2, XP, RL, each with its review; the Rollout section). The operator's runbook is
diff --git a/plans/release-18.md b/plans/release-18.md
@@ -329,6 +329,32 @@ Probe = { status|null; generatedAt?; cfCacheStatus?; age?; cacheControl?; error?
4. Should the lane also rebuild on a code change (`built.commit` differs)? Default no.
5. Six per-site bundles cost disk (`export/out` is ~491 MB for the largest site); hardlink or accept.
+### Open questions, settled
+
+1. **The hub's `s-maxage=604800`** is not the repository's: no rule it renders or ships sets it, so it is Cloudflare's
+ side of a `*.pages.dev` hostname (S2 record, "Open question 1"). Whether `no-store` wins over it is read off the
+ hub's next deploy — its live check records `cache-control`, `cf-cache-status` and `age` for `corpus.json` and each
+ tombstone, plain and busted (rollout step 4).
+2. **`main` is the production branch of every Pages project** as far as the records say without asking Cloudflare
+ (S2 record, "Open question 2"): `--branch main` changes nothing for the eight projects.
+3. **A `generation` bump rewrites every site's aggregates** (S1 record, "Open question 3"): `inputSig` is conservative —
+ any data change anywhere stales every site's signature; `changedChannels` is the precise signal. A follow-up below.
+4. **No** — a code change (`built.commit` differs) shows "code newer" and is not stale (S1 `needs()`; ruled 2026-10-06).
+5. **Accept** the per-site bundles: ~500 MB × 6 against 225 GB free; no hardlinks (ruled 2026-10-06).
+
+### Rulings settled during the release (2026-10-06; do not re-open)
+
+- **Private data is never named on the hub** (S2 H1): the hub tombstones only X channels that are a member of at
+ least one non-private site. Members of unlisted or cited public sites ARE still tombstoned — consistent, and said to
+ the operator as I1.
+- **S3's five reviewer questions:** keep the `snapshot.json` ingest signal (its cost is no-op stages);
+ `common/publish/` is the home of `publishState` / `publishStages` / `publishRunner` (the layering test forbids
+ controller → publish and publish → views); Stop does NOT cancel the stage in flight (a hold stops dispatching, never
+ kills a stage; Cancel on /jobs does); `archilyzer publish now` goes on after a failure and exits with the worst code;
+ keep the third pass trigger (policy work left, once per `refreshEveryMinutes`).
+- **The hub-bundle regression (`5c09cd7b`, 2026-10-05) never shipped in a release**: it is stated in the S2 record,
+ not in the changelog.
+
## Assumptions the operator can overturn
| assumed | alternative |
@@ -345,6 +371,24 @@ Probe = { status|null; generatedAt?; cfCacheStatus?; age?; cacheControl?; error?
| the fan-out runner is host-only, opt-in | drop the fan-out entirely (single runner) |
## Follow-ups carried over (not scheduled; keep)
+- **From release 18's reviews:** `stageLock.ts` `removeIfUnchanged` read-then-rm race (two waiters on one stale lock);
+ the lock does not cover a worktree whose `export/public` links into the primary's; `pnpm --filter … exec` collapses
+ exit codes 2/3/130 to 1 (documented, PUBLISH.md); a `generation` bump stales every site's `inputSig` (sign summaries
+ by content minus `generatedAt`); a site's live check does not probe its tombstones; `generatedAt` cannot tell two
+ no-data deploys apart; the homepage's live check reads only `/` (compare `/source/manifest.json`'s commit); a cited
+ site that withholds an X channel writes no tombstones; `builtExport.ts` `REPORT_DATA_FILES` hard-codes names instead
+ of `revisions.ts` helpers, and report stills are unnamed; a Cancel arriving as wrangler exits 0 records nothing
+ (harmless); no gitleaks / stagit in the image (a pinned gitleaks later); `.env` credentials reach site / homepage /
+ umtool through the shared `env_file` (an editor-only credentials file); `toolProbe.mjs` counts a failing `--version`
+ that printed output as present (umtool's doctor shares it); the stale 7.3 GB `archilyzer:local` image; the
+ `snapshot.json` signal is loose (accepted).
+- **From S4:** a dangling `export/out` link (its bundle deleted by hand) fails a bare `next build` of the export app on
+ `stat export/out` — the stages unlink it first, and the editor e2e drops its own; `pnpm ops lane` does not take the
+ publish lane (its hold is the UI's or `settings.publish.held`); a click's run can reuse the lane's still-QUEUED
+ background index update and start its foreground build first (noted in `publishStages.ts`; Retry); `build hub` and
+ `build homepage` stay raw, unstamped builds (e2e:2origin runs `build:hub`); `jobs.spec` "tails its log" and
+ `site-scope.spec` "charts is a site's tab" each failed once in a long run and pass alone (dev-mode first compiles
+ under the stage children's load).
- normalize/archive pool jobs still run in-process (move them to child processes).
- `migrate-tier --reclaim` on the platter; the en/en-orig duplicate-track ruling + the `en` → 0 cues bug (212 degraded);
"text on platter" per-channel option (low priority, /home has 225 GB free); Syncthing folder paused.
@@ -355,7 +399,9 @@ Probe = { status|null; generatedAt?; cfCacheStatus?; age?; cacheControl?; error?
## Record
-(Each slice adds a "### Slice <X>, as shipped" section here, before "## Rollout".)
+(Each slice adds a "### Slice <X>, as shipped" section here, before "## Rollout".) The sections are in merge order,
+not slice order: S2 (deploy hardening), S1 (the stage contract, the stamps, the lock, the bundles, the CLI), S5 (the
+image half), S3 (the status, the queue, the lane), S4 (the surfaces), S6 (the records).
### Slice S2, as shipped — deploy hardening (2026-10-06)
@@ -1435,6 +1481,50 @@ noted in `publishStages.ts`. From the S6 drafts' readings, in the same commit: t
"Build stats dataset"; the source gate's refusal names `archilyzer publish homepage` (only a stage stamps what a
deploy ships); `build hub` stays a raw, unstamped build and says so (e2e:2origin's `build:hub`); three stale comments.
+### Slice S6, as shipped — the records (2026-10-06)
+
+Branch `r18/records` off `r18/integration` `dddbc183` (S4 merged), in the integration worktree, by the orchestrating
+session with two Opus drafting agents (PUBLISH.md; FACTS.md) writing to scratch only — their drafts were read, checked
+against the code at the S4 merge and installed here. Docs only, except the describe strings the generated docs come
+from (`settingsSchema.ts` `buildPipeline` + `maxParallelBuilds` + the runner comment, `siteSchema.ts` `audience`,
+`envVars.ts` `SITE_ID`).
+
+**What it does.**
+- **PUBLISH.md** rewritten around the stages (1078 lines, from 843; the source-mirror, R2, cost-abuse, previews,
+ reports and MCP sections — ~715 lines — kept): "Publishing is stages" (the stage table, the stamps, what makes the
+ index stale incl. the settings signature, the publish lock with `ARCHILYZER_HOST_ID` and clearing a dead holder's
+ lock, exit codes and the `pnpm --filter … exec` collapse of 2/3/130 to 1, Cancel); "The three ways to drive it" as
+ one table (editor / `pnpm ops publish` / `archilyzer`) with the aliases; "In the runtime container" (`exec`, never
+ `run --rm`); "The publish lane" and the policies; deploy hardening (the pinned wrangler, `--branch`, the main-only
+ production rule, the credential preflight and the Cloudflare refusal sentence, the live check's verdicts,
+ `deployed.json`); tombstones and `no-store`; previews from the bundle; R2 staging under the bundle; "Building every
+ site in containers" as the opt-in docker runner (host only, refused in a container). Every anchor another doc
+ links to is kept.
+- **AGENTS.md** "The runtime container": `Dockerfile.build` is the opt-in docker runner's; `publish-site.sh` is three
+ stages; stages run in the container through `exec` under the publish lock (`ARCHILYZER_HOST_ID`); the docker runner
+ refuses in a container; no docker-in-docker.
+- **plans/FACTS.md**: a new `## The publish stages` (the stages, stamps, lock, bundles, `needs()` per stage and who
+ asks it, exit codes — the `pnpm` collapse measured on pnpm 11.26.0 —, the queue, runs and lane, the surfaces incl.
+ the run-wide Cancel, the deploy stage, the hub's bundle check, tombstones, the `publish/` layering, the e2e seams),
+ and 17 one-line "Superseded by release 18" markers after the facts it made stale (the plan's eleven ranges — one,
+ 5352–5357, turned out still true and got its own marker only for `all`'s new `jobId` — and five more found stale:
+ `buildStatsAction`, "wrangler has no fake", `deployExportAction`, the old CLI list and exit codes).
+- **plans/STATE.md** "Now": release 18 complete on `r18/integration`, not on `main`, nothing live; what is owed, in
+ order; what changes for the operator.
+- **Generated docs from their sources**: SETTINGS.md and `settings.json.example` (`buildPipeline` is the docker
+ runner's), SITE.md (`audience` names the deploy stage's paths), ENVIRONMENT.md (`SITE_ID`); README.md and SETUP.md
+ say what `pnpm build` runs now. RUNNING_IN_DOCKER.md (S5) and SETUP.md's Node 22 (S5) were already done.
+- **Changelogs**: editor `[Unreleased]` — "One publish at a time", "`export/out` is now a link" and the aliases folded
+ into "Publishing is stages" (eight r18 leads: the six planned, the Docker yt-dlp, the doctor); export one bullet
+ (withdrawn posts leave an uncached stand-in); homepage one bullet (built and deployed from the image, as a stage).
+- **This file**: "Open questions, settled" (Q1–Q5), "Rulings settled during the release", the reviews' leftovers under
+ "Follow-ups carried over", and an index of the record's sections (they are in merge order).
+
+**Gates** (logs `$T/s6-*.log`): tsc clean; `docs env --check`, `docs files --check`, `settings example --check`
+clean; the schema doc tests pass; `pnpm --filter editor exec next build` ok; `pnpm --filter homepage run
+build:nodata` ok; counts-only privacy greps over the slice's diff: the refused identifier suffix 0, home paths 0 in added lines, no
+`transcripts/` content, no operator reasons. The whole editor suite runs once more in step 4 (S6 touched `common/`).
+
## Rollout
(Steps 1–7 above; "### As it went" is written as the rollout runs.)
