commit 053fcf94a70b6ac8157a7aaca366266db26e585b
parent ba6e09d699a082505048de70293f49d2ed5bfdd2
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 6 Oct 2026 11:40:02 -0400
plans: release 18 — slice S3, as shipped (the publish status, the stage queue, the publish lane); the changelog's Publish lane and One index for every site
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
2 files changed, 154 insertions(+), 0 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,6 +1,8 @@
# Changelog
## [Unreleased]
+- **The publish lane.** Publishing can run itself: turn on `publish.enabled` in settings and the lane checks every `checkEveryMinutes` (10) whether the index is stale; when it is — and its last update is at least `refreshEveryMinutes` (360) old — it updates it, then builds every site whose channels changed or whose data the new index moved, one stage at a time on the `publish` queue. What it may do with a site is the site's own: `site.json` `publish.auto` is `off` (the default: left alone), `build`, `preview` (built and deployed to the preview branch `publish.previewBranch`) or `production`; the hub and the homepage have `publish.hub` and `publish.homepage`. A private site is only ever built, and a site needs its Cloudflare Pages project before it may deploy. Hold the lane and the stage running finishes and no next one starts; quiet hours (`publish.quietHours`) do the same; Drain finishes the stage and ends the runner. The lane never forces a stage: a stage that finds its target current does nothing. On /jobs every stage of one run reads `run <id> · <target>`, and a stage still queued when the editor restarts is cancelled, never re-queued — the lane works out again what is stale from what is on disk. `archilyzer publish now` runs the same plan from the command line, one stage after another in its own process.
+- **One index for every site.** The index is updated once and every site, the hub and the homepage are built from it; `archilyzer publish status` says, per site, whether its build is current — "stale: 3 channels changed (a, b, c)" as soon as a download, transcription or digest on one of its channels finishes, before any index runs; "stale: data changed" once the index has run and the site's data moved; "stale: config changed" after its site.json, tags or aliases changed — and whether what is deployed is that build, with a build made by older code marked "code newer" but not stale.
- **Deploys are pinned and checked live.** wrangler is an exact dependency of the workspace (4.147.0), so a deploy runs the version installed with the code instead of whatever `pnpm dlx` fetched that day, and every deploy names its branch: production is `--branch main`, never taken from the checkout it ran in (where a "production" deploy from a feature branch used to land as a preview). The publish stages' deploy (release 18) refuses before wrangler runs when there is no Cloudflare credential at all — "set CLOUDFLARE_API_TOKEN in .env" — and says "REFUSED by Cloudflare — the API token was not accepted" when Cloudflare rejects one; it refuses a production deploy of a build made from a branch other than `main`. After each deploy it reads `corpus.json` at the site's address twice, as a visitor would and cache-busted, and records the verdict: ok, stale-edge (the deployment is right, Cloudflare's edge still serves an older copy), mismatch, or unreachable. A verdict short of ok is a warning in the log; the deploy itself succeeded. What each target last shipped, where, and how it read is kept in `deployed.json` beside its build.
- **Withdrawn X posts ship tombstones.** While X posts are private, a public site's build no longer just leaves an X channel's posts out: at every path they were served from it ships an empty stand-in — the channel's posts manifest with no pages, and an empty page for each page the channel has — served uncached. The hub, which carries no posts, ships the same for every X channel a public site carries, with an empty posts manifest; a channel only on a private site, or on no site, is never named on the hub. Leaving a path out of a deploy does not take it off Cloudflare's edge, which kept serving a withdrawn copy for up to a week; a changed object at the same path replaces it. The hub's deploy reads each of those paths back.
- **Publishing is stages, from the command line: `archilyzer publish`.** `publish index` updates the index — the LMDB index, the stats datasets and the chart templates, in one child process with an 8 GB heap — and writes an index stamp (`export/.export-index/stamp.json`) naming, for each site, a signature of everything that site's build reads. `publish build <id|all>` builds a site from that index (no data phase of its own) into its own bundle, `export/.export-builds/<id>/out`, and stamps it (`built.json`); a site whose bundle already matches the index is a no-op unless `--force`. `publish deploy <id|all> [--preview <branch>] [--to local]` ships that bundle — to Cloudflare Pages, or with `--to local` into the directory the docker `site` service serves — and records the deploy (`deployed.json`); deploying the same build again is a no-op unless `--force`. `all` passes over private sites and, to Pages, sites with no Pages project; any other site it cannot deploy is a failure, said after the rest are tried. `publish hub [--deploy]` and `publish homepage [--deploy]` do the same for the hub (`_hub/out`) and the homepage. A stage whose input is not there says so and exits 3: "update the index first", "no build of jeralyzer — archilyzer publish build jeralyzer". Production refuses a bundle built on a branch other than `main`, or with no branch recorded (a detached checkout; an image sets `ARCHILYZER_BRANCH`) — a preview of it is fine. Exit codes: 0 done or nothing to do, 1 failed, 2 usage, 3 precondition not met, 130 cancelled.
diff --git a/plans/release-18.md b/plans/release-18.md
@@ -1127,6 +1127,158 @@ the image built beside it; the file alone afterwards: 8/8 (S5 touches nothing it
S5 is complete with this round. Left for the rollout, as recorded above: building `runtime-vulkan` and
`runtime-cuda` once.
+### Slice S3, as shipped — the publish status, the stage queue and the publish lane (2026-10-06)
+
+Branch `r18/publish-lane` off `r18/integration` `0ce00f76` (S1 and S5's image half merged; `1f07ca2f` — S2 and
+S5 complete — merged in before the gates), worktree `~/Projects/r18-publish-lane` (editor 7201, test 7211, export
+7210), one Opus implementer. Scratch files `s3-*` in the job's `tmp`. The plan is "Stages" (the two staleness
+bullets), "Lane", "Surfaces: One state builder", the CLI's `publish status|now` and "Migration"; S1's "Seams for
+the other slices" are what it codes against. No S1 file changed.
+
+**What it does.**
+- **One status, one plan — `common/publish/publishPlan.ts` (pure).** `buildPublishStatus(inputs, now)` folds the
+ inputs into `PublishStatus`: per target (each site, `_hub`, `_homepage`) `indexFresh`, `built`,
+ `builtFromCurrentIndex`, the build stage's own `needs()` answer, `buildStale {reason: channels | data | config,
+ changedChannels}`, `codeNewer` (never stale), `deployed`, the policy's `deployKind` and record,
+ `deployedIsBuilt`, `previewUrl`, `liveCheck`, `policy`, `deployProblem`, `next`, `running`, `queued[]`, the
+ newest ended build/deploy job, and four chips (`index | built | deployed | live`, each `{tone, text}`); plus the
+ index (chip, freshness, its jobs), the lane (`due` + `reason`, `blockedReason`, chip) and `plan` — what Publish
+ now would run. Every freshness question is the stage's `needs()` over a `NeedsInput` built here; the status adds
+ only what a stage child cannot see: `changedChannels` (a member channel with an ingest ended after
+ `builtCheckedAt(built)` — so a no-op build clears it; the hub's are the listed sites' members, the homepage's
+ every channel) and the config mtimes. Chip words: "no index yet", "stale: new data since the last index",
+ "never built", "stale: N channels changed (a, b, …)", "stale: data changed", "stale: config changed", "built
+ 12 min ago · code newer", "production · 3 min ago", "production: a newer build is not deployed", "never
+ deployed (private)", and — from the newest ended stage, exit 3 — "waiting for its build" (a deploy that carried
+ `builtAfter`), "waiting for the index" (a build that carried `indexAfter`), else "last deploy refused:
+ precondition not met"; live: "live ok", "live: the edge serves an older build", ….
+- **`planPublishRun(status, {deploys: policy | none, builds: policy | stale, runStart, exclude, skipIndex})`**:
+ update-index when the index is stale; per site (by id) its build when the stage's `needs()` says stale — changed
+ channels, a signature that no longer matches the index, never built, its config, its bundle — and its deploy
+ where `publish.auto` says (`preview` → `settings.publish.previewBranch`, with the alias URL); then the hub, then
+ the homepage, by `settings.publish.hub|homepage` (the index update in the run moves them when their channels
+ changed). Builds carry `indexAfter = runStart` when the index is in the run, deploys `builtAfter = runStart`
+ when their build is. A policy-off site is never built (`skipped`: "stale, and its policy is off") unless
+ `builds: "stale"` ("Build all stale"); a private or project-less target is never planned as a deploy (skipped
+ with the reason). Never forced. `settings.publish.runner: "docker"` plans one `build-site _all --runner docker`.
+ `views/publishStatus.ts` re-exports it: the view layer's name for it.
+- **`common/publish/publishState.ts` `readPublishStatus(paths)`** — the one function the /sites panel, `GET
+ /api/ops/publish`, `archilyzer publish status` and the lane call. `readPublishInputs` reads the stamps and bundle
+ problems (S1's `readNeedsInput`), the sites (membership = `site.json` `channels`) and their policies, the config
+ mtimes of the plan's list (for the index tags.json, search-aliases.json, duplicates*.json, every site.json,
+ homepage.json, the settings file, the charts config; for a site its site.json, its tags and aliases and the
+ corpus-wide three; for the hub homepage.json and every site.json; for the homepage homepage.json), the ingest
+ signal per channel, the live publish jobs, the newest ended stage per kind + target, the lane's memory, and the
+ running commit and `main`'s head (git, memoized 30 s). Job metas are read through the registry and the `.jobs`
+ sidecars, each terminal meta once per process.
+- **`common/publish/publishStages.ts`.** `enqueueStage(paths, req, {background})`: one `runManagedCommand` over
+ S1's `stageCommand` on queue `publish`, spec `{kind: "publish-<stage>", slug: target, params: the request}`; a
+ duplicate — the same kind, target and destination queued or running — is refused `{ok: false, info: true,
+ jobId}`. `enqueuePublishRun(paths, plan, {runId})`: the plan in its order under one run id, each request
+ carrying its step's preconditions; returns `{runId, jobs: [{kind, target, jobId, previewUrl?, existing?}],
+ skipped, refused}`. `stageRequestFromSpec` reads a spec back through the stage row's own parser.
+- **The lane — `common/publish/publishRunner.ts`.** `startPublishRunner` (kind `auto-publish`, queueKey `""`)
+ wakes every `checkEveryMinutes`; a pass is due when there is no stamp, when the index is stale and
+ `now − stamp.builtAt ≥ refreshEveryMinutes`, or when a policy target is left stale and the last pass is at least
+ that old; never while held, in quiet hours or while any publish stage is queued or running. A pass dispatches
+ ONE stage at a time (`background: true`) and awaits its job's end, re-reading the status and re-planning before
+ each next one (so the builds after its index update see the new stamp); a stage it ran is not run again in the
+ pass; a failed index update ends the pass; a failed build drops its deploy; the gate is re-checked between
+ stages — a hold, quiet hours or the lane switched off stop the dispatching, never a stage; a drain finishes the
+ stage in flight and ends the runner `done`; Stop ends the runner at once and leaves a running stage to finish as
+ its own job. `stopPublishRunner` / `drainPublishRunner` / `startPublishRunnerBlockedReason`;
+ `publishLaneState.ts` holds the lane's memory (last check, next check, last pass and its summary). The editor's
+ `instrumentation.ts` starts it below the idle-boot gate.
+- **Settings and site.json.** `settings.publish {enabled: false, held: false, checkEveryMinutes: 10 [1–1440],
+ refreshEveryMinutes: 360 [0–43200; 0 = whenever stale], quietHours: {start, end} | null, runner: local,
+ previewBranch: "preview" (an invalid name reads as it), hub: off, homepage: off}`; `site.json` `publish: {auto:
+ off | build | preview | production}` — absent = off, only another policy written; a private site reads and is
+ written as `build`; a site with no `cloudflareProject` reads as `build`, and a save of `preview`/`production`
+ without one is refused ("publish.auto "production" deploys the site, and it has no cloudflareProject — …").
+ SETTINGS.md, settings.json.example and SITE.md regenerated.
+- **The pipeline lane.** `PIPELINE_LANES = ["publish"]` (`autoQueueTypes.ts`), not in `LANES`; `PauseLane =
+ AutoQueueKind | "publish"`; `isGateHeld` / `withGateHeld` read and write `settings.publish.held`. The editor's
+ lane pause action saves that block for `publish`, and the pause control has its words ("Hold the lane",
+ `pause publishing` / `resume publishing`).
+- **Jobs.** The seven `publish-*` kinds (labels = the stages', replayable, not drainable) and `auto-publish`
+ (drainable); `build-index`, `build-stats`, `build-export`, `build-deploy`, `build-all`, `build-deploy-all`,
+ `deploy-export` known with NO label (/jobs shows their raw kind, as it always has; `jobs.spec` reads it) beside
+ the six hub/homepage kinds that keep theirs. `isIngestKind`. /jobs reads a stage as `run <last six of the run
+ id> · <target>`. A queued publish stage at boot is cancelled as `publish` — "server restarted; the publish lane
+ re-derives stages from on-disk state" — never re-queued. Retry re-enqueues a stage from its spec (same run id).
+- **CLI.** `archilyzer publish status [--json]` (the index, the lane, a row per target with its policy and chips
+ and what is next, then the plan) and `archilyzer publish now` — the plan's stages one at a time IN THE CLI's
+ PROCESS under the publish lock, the index update as `publish index`'s child, as the editor's queue would run
+ them (the CLI has no queue); the rest are tried after a failure; exit 0, or the worst (1 over 3).
+
+**Deviations from the plan** (one sentence each):
+1. The three modules are `common/publish/{publishState,publishStages,publishRunner}.ts`, not `controller/`, and
+ the pure builder is `publish/publishPlan.ts` re-exported by `views/publishStatus.ts`: `architecture.test.ts`
+ forbids controller → publish and publish → views, and the runner must plan.
+2. The runner is started from `instrumentation.ts` beside `startAutoRunnersIfEnabled`, not inside it (dispatch may
+ not import publish); the idle boot leaves it off the same way.
+3. "The drainable kinds" became an explicit `isIngestKind` set: every drainable per-channel kind (tested) plus the
+ non-drainable writers of the same text and posts (the auto-download unit, single imports, transcriptions and
+ downloads, the availability checks, the forum import, the feed backfill, the cues sweep); the lane runners'
+ metas stay `running` for days and are not in it.
+4. A channel's `snapshot.json` mtime is also an ingest signal: the transcription, digest and backfill lanes' units
+ make no job record at all, and each requests the channel's report regeneration when it settles.
+5. A duplicate stage is the same kind, target AND destination: a production deploy behind a preview of the same
+ site is not refused.
+6. A pass is also due when a policy target is left stale (a failed stage, a policy just turned on) and the last
+ pass is `refreshEveryMinutes` old — else a fresh index would never retry it.
+7. The lane re-plans before each stage instead of enqueueing a plan computed once.
+8. Stop does not cancel the stage in flight (it is its own job, with its own Cancel); the plan did not say.
+9. Files beyond the slice's list, each a line or a table entry: `common/lib/site.ts` (writeSite's refusal),
+ `common/lib/{settingsDocs,fileSchemaDocs}.ts` (the doc tables), `editor/app/operations/actions.ts` and
+ `editor/app/components/lanes/pauseControl.tsx` (the widened `PauseLane` needs its save and its words).
+
+**Exports added to S1's files:** none.
+
+**Found, not fixed.**
+- The settings file is an index input (the plan's list), and the settings file is written by every pause click,
+ priority change and drive auto-pause: each makes the index "stale" — one short-circuited index update per
+ refresh interval, no rebuild (the signatures are unchanged).
+- The snapshot signal is loose: a report regenerates after any channel job, failed ones included; the cost is a
+ short-circuited index and no-op builds.
+
+| commit | what |
+|---|---|
+| `8b55064a` | jobs: the seven `publish-*` kinds, `auto-publish`, the label-only kinds; `isIngestKind`; `run <id> · <target>`; boot category `publish` |
+| `1df0e276` | settings: `settings.publish`, `site.json` `publish.auto`, `PIPELINE_LANES`, the publish gate; SETTINGS.md, SITE.md, example |
+| `105c2663` | publish: `publishPlan.ts` — the status builder and the planner; `views/publishStatus.ts` |
+| `3adafcb7` | merge `r18/integration` `1f07ca2f` (S2, S5) |
+| `0081d1dd` | publish: `readPublishStatus`, `enqueueStage` / `enqueuePublishRun`, the runner, the lane's memory |
+| `344ffee0` | editor: the runner at boot; Retry for publish stages |
+| `0527be27` | cli: `publish status [--json]`, `publish now` |
+
+**Gates** (all from the worktree root): tsc clean at every commit; common **3339 passed** (57 new: publishPlan 23,
+publishRunner 10, publishStages 6, publishState 2, publishNow 2, jobKinds 3, jobDetail 3, bootQueuedJobs 1,
+settingsSchema 2, siteSchema 4, pauseGates 1); editor unit **142**; `test:scripts` **596 + 3 skipped**; mcp
+**289**; export unit **116**; homepage unit **23**; `pnpm --filter editor exec next build` ok (51 s); `pnpm
+--filter export exec next build` ok (27 s, over the committed fixture compose linked into the worktree's
+`export/public` — the primary's holds a reports-only compose); `pnpm --filter homepage run build:nodata` ok
+(15 s); umtool's capped build ok (19 s, link removed). e2e (editor suite, `s3-specs.txt`: lane-runner, auto-queue, jobs, jobs-filters, settings, sites-crud, site-scope, build, scheduler, operation-settings; the worktree's `export/public` seeded with the fixture compose, cleaned after): **88 passed, 0 failed, 6.2 min**. Numbers tool: none. CLI smoke over a scratch
+corpus (`s3-smoke.sh`, one site on `build`): `publish status` → "no index yet", plan `update-index _index`,
+`build-site smoke`; `publish now` → both ran (40 s, the build under `indexAfter`); `publish status` → "fresh",
+"built just now", nothing to run; `publish now` again → "nothing to do"; the policy switched off → "stale: config
+changed", "stale, and its policy is off". The scratch compose was removed from `export/public` after.
+
+**Left for the other slices.** S4: the /sites Publish panel, `/operations/publish`, `GET|POST /api/ops/publish`
+read and call the names in "Seams" below; S4's e2e sets policies through `site.json` / `settings.publish`.
+S5's doctor: the `source-repo` grade can key on `settings.publish.homepage` now. S6: PUBLISH.md (the lane, the
+policies, `publish status|now`), FACTS.
+
+**Seams (the names S4 calls).** `readPublishStatus(paths?, {now?, laneKnown?})` →
+`PublishStatus` (`publish/publishState.ts`; types from `views/publishStatus.ts`); `planPublishRun(status,
+{deploys, builds, runStart?})` (Publish now = `{deploys: "policy"}`; Build all stale = `{builds: "stale",
+deploys: "none"}`); `enqueuePublishRun(paths, plan, {runId?})` and `enqueueStage(paths, req, {background?})`
+(`publish/publishStages.ts`, with `newPublishRunId`, `PUBLISH_QUEUE`, `stageRequestFromSpec`); the lane:
+`startPublishRunner(paths?)`, `stopPublishRunner()`, `drainPublishRunner()`,
+`startPublishRunnerBlockedReason(settings?)` (`publish/publishRunner.ts`), its hold `pauseLaneAction("publish")` /
+`resumeLaneAction("publish")` (`editor/app/operations/actions.ts`) and `isGateHeld(settings, "publish")`; the
+policies `sitePublishPolicy(site)`, `sitePublishProblem(site)`, `settings.publish`.
+
## Rollout
(Steps 1–7 above; "### As it went" is written as the rollout runs.)
