commit 0d0c9082a4e42a07384dd27ce387f08cc91755a9
parent a532d8004ac204f508beafb7a8d4defde323512a
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 6 Oct 2026 09:17:30 -0400
plans: release 18 — slice S1 (stage core) as shipped; the editor changelog
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
2 files changed, 161 insertions(+), 0 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,6 +1,10 @@
# Changelog
## [Unreleased]
+- **Publishing is stages, from the command line: `archilyzer publish`.** `publish index` updates the index — the LMDB index, the stats datasets and the chart templates, in one child process with an 8 GB heap — and writes an index stamp (`export/.export-index/stamp.json`) naming, for each site, a signature of everything that site's build reads. `publish build <id|all>` builds a site from that index (no data phase of its own) into its own bundle, `export/.export-builds/<id>/out`, and stamps it (`built.json`); a site whose bundle already matches the index is a no-op unless `--force`. `publish deploy <id|all> [--preview <branch>] [--to local]` ships that bundle — to Cloudflare Pages, or with `--to local` into the directory the docker `site` service serves — and records the deploy (`deployed.json`); deploying the same build again is a no-op unless `--force`. `publish hub [--deploy]` and `publish homepage [--deploy]` do the same for the hub (`_hub/out`) and the homepage. A stage whose input is not there says so and exits 3: "update the index first", "no build of jeralyzer — archilyzer publish build jeralyzer". Production refuses a bundle built on a branch other than `main` (a preview of it is fine). Exit codes: 0 done or nothing to do, 1 failed, 2 usage, 3 precondition not met, 130 cancelled.
+- **One publish at a time on a machine.** Every stage takes `export/.export-builds/.publish.lock`; a second one — an `archilyzer publish` beside the editor, say — waits for it, saying once whom it waits for, and Ctrl-C ends the wait. A lock left by a process that is gone is taken over. A cancelled stage takes the whole process tree it started with it (`next build`'s workers, wrangler, docker).
+- **`export/out` is now a link to the bundle built last.** Each site, and the hub, keeps its own bundle, so building one site no longer replaces another's; `export/out` points at whichever was built most recently, so `serve out` and anything else that read it keeps working.
+- **`build site`, `build all` and `deploy site` are aliases of the publish commands** and print what they run: `build site <id>` is `publish index` (skipped with `--nodata`) then `publish build <id> --force`; `build all` is `publish index` then `publish build all --runner auto` (containers when an engine answers, else one site at a time on the host); `deploy site <id>` is `publish deploy <id>`, which now ships the site's own bundle and refuses a site never built that way. `publish build all --runner docker` builds every stale site in containers on a Linux host and refuses with "the docker runner needs an engine on this host" where there is none.
- **A cited moment at the very end of a recording prepares.** Prepare evidence media cuts a clip whose padding runs past the recording's end at the end (the recording's duration from its metadata), where it found no media for the padded span; a span that starts past the end is still refused. report-to-video keeps its strict rule.
- **Exporting a changed report records a new revision of it.** `reports export` (and **Export reports** on a site's Reports tab, and the end of a prepare) commits a revision to the report's own git history, `sites/<site>/reports/<id>/history-git/`, whenever its `report.json` changed since the last one: the `report.json`, its Markdown export and the checksums of every export file, with a message of `Revision N` and a summary of the change. A re-export of an unchanged report records nothing. The commits carry the site's name and a `noreply@<site>.invalid` address with dates in UTC, never your git name, email or time zone. The Reports tab shows each report's revision, its commit and the last change under **Exports**, and the site's next build publishes the history. Add `history-git/` to the corpus repository's `.gitignore`.
- **archive.org files come over BitTorrent when possible, else straight from archive.org — never through yt-dlp.** The chosen file of an archive.org import is fetched from the item's own torrent (`<identifier>_archive.torrent`, which lists archive.org as a web seed, so other peers take load off archive.org) with aria2c, only that file of the item, and seeded afterwards for 10 minutes or to a ratio of 1, whichever comes first; the log shows "torrent: <file> (n of m pieces, peers p, web seed yes)" and "seeding 10 min…". With no aria2c, a torrent that does not carry the file, or no progress for 5 minutes, it is downloaded directly from `archive.org/download/…` instead (resumable, backing off on 429/503), and the log says "fell back to direct download: <reason>". Every file is checked against archive.org's sha1/md5: a mismatch is downloaded once more directly, a second one fails the record. The record is written from the item's metadata: `metadata.info.json` with the file's page, the canonical id, the duration ffprobe measures and archive.org's playable copies of the file, the `archiveorg.json` provenance (a mirror's original title, date and uploader), and `audio.<fmt>` — an audio file already in the channel's format is used as is, anything else goes through the app's audio extraction, a video kept in the saved-video store when the channel keeps sources. An .avi/.mpeg/.flac/.wav original is fetched as archive.org's mp4 or mp3 of it. aria2c runs in its own process group: cancelling the job stops it and everything it started, and it stops itself if the editor exits. New settings block `archiveOrg` (`torrent`, `seedMinutes`, `seedRatio`, `stallMinutes`, `maxPeers`, `maxDownloadKiBps`, `maxUploadKiBps`), `ARIA2C_BIN`, an aria2c row in `archilyzer doctor`, and `aria2` in the runtime Docker images.
diff --git a/plans/release-18.md b/plans/release-18.md
@@ -353,6 +353,163 @@ Probe = { status|null; generatedAt?; cfCacheStatus?; age?; cacheControl?; error?
(Each slice adds a "### Slice <X>, as shipped" section here, before "## Rollout".)
+### Slice S1, as shipped — the stage contract, the stamps, the lock, per-target bundles and the CLI (2026-10-06)
+
+Branch `r18/stage-core` off `ce66f2d3` (the plan commit on `r18/integration`), worktree `~/Projects/r18-stage-core`
+(editor 7201, test 7211, export 7210 — `pnpm wt list`'s #42), one Opus implementer, beside S2 and S5's image half.
+Scratch files `s1-*` in the job's `tmp`. The plan is "Model", "Stages" and "CLI" above; the deploy bodies are S2's to
+rewire, the status view, the lane and `publish status|now` S3's.
+
+**What it does.**
+- **`common/publish/stages.ts`** — the only module that knows every stage: `StageKind`, `StageRequest`, `Freshness`,
+ `Stage`, `StageOutcome` as "Model" lists them (+ an optional `allowMissingMedia` on the request, for `build site
+ --allow-missing-media`); `STAGES` (seven, `jobKind: publish-<kind>`, `queueKey: "publish"`); a PURE `needs()` per
+ stage over **`NeedsInput`** — the minimal PublishStatus-shaped input S3's view satisfies: `index: {stamp,
+ lastIngestDoneAt, configChangedAt}`, per site / `hub` / `homepage` a `TargetState` `{built, deployed,
+ changedChannels, configChangedAt, bundleProblem, deployProblem?, pagesProblem?}`, and the homepage's `mainHead`.
+ `stageArgv` / `parseStageArgs` (inverse, round-trip tested) and `STAGE_FLAGS`.
+- **`needs()`, row by row.** update-index: stale with no stamp, an ingest ended `done` after `stamp.scannedAt`, or a
+ config newer than it. build-site: BLOCKED "update the index first" with no stamp (forced too), "waiting for the
+ index update this run started" under `indexAfter`, "the index has not seen site X" when the stamp has no entry;
+ then stale by `changedChannels` ("N channels changed (a, b, …)"), a config change after `built.builtAt`, an
+ `inputSig` mismatch ("data changed"), a bundle problem, never built; `--force` stale; a `built.commit` that differs
+ is NOT stale. `_all`: per site. deploy-*: blocked with no build ("no build of X — archilyzer publish build X"), a
+ private target (every kind), no Pages project (pages kinds only), a bundle problem, `builtAfter`, and a PRODUCTION
+ deploy of a bundle built on a branch other than `main` (a null branch — an image with no `ARCHILYZER_BRANCH` —
+ passes); fresh exactly when `deployed[kind/branch].builtStampId === built.stampId`. build-hub: `built.inputSig ===
+ stamp.hubSig` (+ `changedChannels`). build-homepage: `indexStampId` current and `sourceCommit === mainHead` when a
+ repository answers.
+- **`common/publish/stamps.ts`** — `IndexStamp`, `BuiltStamp`, `DeployRecord`, `DeployedFile`, `LiveCheck`, `Probe`
+ exactly as listed; paths (`<exportIndexDir>/stamp.json`, `<exportBuildsDir>/<target>/{built,deployed}.json`); atomic
+ writes (`writeJsonAtomic`); tolerant reads (missing, unparseable or wrongly shaped = null); `recordDeploy` keeps
+ every other record; `newStampId` sorts by time; `imageBuildFacts` (the image's `ARCHILYZER_COMMIT` /
+ `ARCHILYZER_BRANCH`, empty = null — S5's names, read by name here until S5's helper of the same name replaces it).
+- **`common/publish/stageLock.ts`** — `<exportBuildsDir>/.publish.lock` `{pid, host, kind, target, since, pidStart}`,
+ `open(…, "wx")`; stale when same host and the pid is dead (`processIsAlive`) or answers with another
+ `/proc/<pid>/stat` start time (a container's small pids come back after a restart); another host's is never
+ stolen; a torn file is taken over after 60 s; a live holder is waited for (5 s poll, ONE log line, the signal
+ cancels the wait); release removes only its own.
+- **`common/publish/stageRun.ts`** — `runStage(req)` (in-process, under the lock, never throws: `{code, outcome,
+ message}`), exit codes 0 / 1 / 2 / 3 / 130, `stageMain` (the child: SIGTERM/SIGINT abort the stage, a second one
+ exits, tree-kill on), and **`stageCommand(paths, req)`** — `<common>/node_modules/.bin/tsx bin/archilyzer.ts stage
+ <kind> <target> [flags]`, cwd `common/`, the caller's env + `NODE_OPTIONS=… --max-old-space-size=8192` for
+ update-index only — what S3's `enqueueStage` hands `runManagedCommand`.
+- **`common/publish/stageBodies.ts`** — every body but update-index first asks its `needs()` over the state ON DISK
+ (`readNeedsInput`: the stamps, the bundles, `siteDeployProblem`, the Pages project, `main`'s head; no job metas):
+ blocked → exit 3, fresh and not forced → a no-op. **update-index**: `buildIndex` → `buildStats` → the chart
+ templates in ONE process, then the stamp — `generation`, `scannedAt` and each site's `siteFp`/`statsFp` (sha1 of
+ the LMDB keys, read-only), each site's `inputSig` and the `hubSig` (`common/publish/inputSig.ts`). An index that
+ rebuilt nothing and whose every signature is unchanged KEEPS its stamp id (status `noop`), so the builds made from
+ it stay current. **build-site**: `buildSiteBundle`, then `built.json`; `_all` local = each stale site in turn
+ (failures collected); `_all --runner docker` = no engine → exit 3 "the docker runner needs an engine on this host",
+ else `build archives` on the host → `ensureBuildImage` → `runDockerBuildOne` per stale site at
+ `maxParallelBuilds` → `builtBundleProblem` → `built.json` (`runner: "docker"`). **build-hub** / **build-homepage**:
+ `buildHubBundle` / `buildHomepage` (source mirror included; `sourceCommit` from `homepage/out/source/manifest.json`).
+ **deploy-site / hub / homepage**: today's `deploySite` / `deployHub` / `deployHomepage` over the TARGET'S bundle
+ (`outDir` / `stagingDir` options added), then `deployed.json` (`url` = the deployment URL the log line names,
+ `alias` = the preview alias, `liveCheck: null` — S2's); `--to local` copies the bundle's contents into
+ `ARCHILYZER_SITE_OUT` (a site) or `ARCHILYZER_HOMEPAGE_OUT` (the homepage — S5's name), private refused.
+- **`inputSig`** (`common/publish/inputSig.ts`) signs, with compose's `dirSignature`: the site's whole
+ `.export-index/sites/<id>/` tree (chart-templates.json by its BYTES — `build templates` rewrites it every run), each
+ PUBLISHED member's shared transcripts/subs/posts/digests tree (`manifest.json` ignored, a manifest-only tree as
+ compose's constant), `site.json`'s bytes, the `sites/<id>/` dir, the global aliases, curated tags and duplicates
+ files (size + mtime), and `archiveStorage` + `social.x.visibility`. A superset of compose's skip inputs:
+ conservative. `hubSig` = sha1(stampId, homepage.json, each listed site's id + siteUrl + title).
+- **`common/lib/dirSignature.ts`** — compose's `dirSignature`, moved unchanged; `compose-site.ts` imports it (that
+ line, and its now-unused `createHash` import removed, are the only compose-site edits).
+- **`common/publish/build.ts`** — per-target bundles: `bundleDir(paths, target)` (= `dockerSiteOutDir` for a site),
+ `installBundle(src, <target>/out)` (rename into `out.next`, `out → out.prev`, `out.next → out`, `out.prev` removed, a
+ leftover `out.next` deleted first; EXDEV → `fs.cp` + remove, injectable `BundleFs`), `unlinkExportOut` (before a
+ build: a link at export/out is removed so a failed build cannot leave an older bundle there), `pointExportOutAt`
+ (export/out → a RELATIVE symlink, replaced atomically), `stageSiteArchives` (the host compose's `.r2-staging/<id>`
+ moved to `dockerSiteStagingDir`, a no-op where they are one place), `bundleCounts`, `corpusGeneratedAtIn`,
+ `buildSiteBundle`, `buildHubBundle`. `ensureBuildImage`, `runDockerBuildOne`, `runHostScript`,
+ `runWithConcurrency` are exported. A build container gets `EXPORT_BUILDS_DIR=/tmp/archilyzer-builds`
+ (`CONTAINER_BUILDS_DIR`) so its own `publish build` lock never lands on the host mount. Every existing export is
+ unchanged; the editor's actions still build into `export/out` (S4 rewires them).
+- **`common/bin/archilyzer.ts` + `common/bin/publish.ts`** — rows `publish index` (the SAME child the editor spawns,
+ for its heap), `publish build <id|all> [--runner local|docker|auto] [--force] [--skip-archives]`, `publish deploy
+ <id|all> [--preview b] [--to local] [--force]` (`all` skips a site refused with 3 — private, no project, never
+ built — and fails on anything else), `publish hub [--deploy] [--preview b] [--force]`, `publish homepage [--deploy]
+ [--preview b] [--to local] [--force]`, and the internal `stage <kind> <target> --run-id …`. A comment marks where
+ S3's `publish status` / `publish now` rows go. **Aliases, printed first**: `build site <id>` = `publish index` (not
+ with `--nodata`) + `publish build <id> --force`; `build all` = `publish index` + `publish build all --runner auto`;
+ `deploy site <id>` = `publish deploy <id>`.
+
+**Deviations from the plan** (one sentence each):
+1. `publish index` runs the stage child (`stageCommand`, the 8 GB heap) rather than the body in the CLI's process: the
+ index and stats builds of the real corpus have always run with that cap (export's `build:index`), and the CLI's own
+ node has the default heap.
+2. `build all`'s alias runs `publish index` first — the old row always ran the data phase, and building every site
+ from a stale index would not be what it said.
+3. Tree-kill lives in `common/jobs/runChild.ts` (`setKillChildTrees`, off by default; a stage child and the publish
+ CLI turn it on): each child leads its own process group and a cancel signals the group, then SIGKILLs what is left
+ once the leader exits. The editor's in-process jobs are unchanged.
+4. `.gitignore` gains `/export/out` (no trailing slash): `**/out/` matches only a directory, and the link showed as
+ untracked — which `release cut --commit` refuses.
+5. Inside the docker per-site build container (`ARCHIVES_READONLY=1`, set by `docker/build-site.sh`) `publish build`
+ builds IN PLACE and stamps nothing — the container hands export/out back and the host stamps it — and asks no
+ stamp, so the editor's existing Build all (host `build:data`, no stamp) keeps working until S4 rewires it. The
+ container still writes `<id>/out` with build-site.sh's `rm` + `cp`, not through `out.next` (S5's file).
+6. `StageRequest.allowMissingMedia` (optional) carries `build site --allow-missing-media` through the stage.
+7. The bundle-layout tests are a new `common/publish/bundle.test.ts`, not `build.test.ts`, which S2 also edits.
+8. `ARCHILYZER_COMMIT` / `ARCHILYZER_BRANCH` / `ARCHILYZER_HOMEPAGE_OUT` are read by name through an `env[name]`
+ helper: they are declared in `lib/envVars.ts` on S5's branch, not on this one (`envVars.test` stays green here;
+ after the merge the helper can be S5's `imageBuildFacts`).
+
+**Open question 3, settled: yes — a `generation` bump rewrites EVERY site's aggregates.** `generation` is bumped
+whenever any record anywhere is added, changed or removed (`buildIndex.ts` ~:2036), and every site's fingerprint
+carries `gen` (~:2083), so every site is rebuilt: its summary pages (`writeJsonAtomic`, no sha1 skip for site pages),
+its four manifests (fresh `generatedAt`) and its `tag-counts.json`. Their mtimes move, compose would re-copy the
+summaries, and every site's `inputSig` changes. So `inputSig` is conservative as the plan expected: any data change
+anywhere makes every site stale (more rebuilds, never a wrong skip); `changedChannels` is the precise per-site signal.
+A follow-up could sign the site's summaries by content less `generatedAt`.
+
+**Found, not fixed (not this slice's files).**
+- **Every hub bundle is refused since `5c09cd7b` (2026-10-05).** `builtHubProblem` (`common/lib/builtExport.ts`)
+ refuses a hub `out/` that holds `reports/` or `m/` ("still carries a site's data (reports, m)"), but the export app's
+ own `/reports/` and `/m/[...moment]` routes render `out/reports/index.html` (+ `__next.*.txt`) and `out/m/…` in
+ EVERY build, the hub's included — `export/public` had neither when measured. So `deployHub` (old path and new) and
+ `publish hub` refuse every hub; rollout step 5 is blocked until the check looks for report DATA (e.g.
+ `reports/index.json`, a `reports/<id>/page.json`) or the hub stops rendering those routes. Measured in the S1 smoke
+ (a scratch corpus; `publish hub` exit 1 after a 119 s build).
+- `pnpm --filter … exec` (and so `pnpm archilyzer`) reports a stage's exit 2 / 3 / 130 as 1; the editor spawns tsx
+ directly and sees the real code.
+- The worktree export build gate needs a FULL site's compose in `export/public`; the primary's held a cited site's
+ (no `summaries/`) at the time, and `/` failed to prerender against it. The gate was run over a scratch site
+ composed into the worktree's own `public/` (then cleaned and re-linked).
+
+**Left for the other slices.** S2: rewire the three deploy bodies (`stageBodies.ts` `deployStage`) to its deploy
+stage, import `LiveCheck`/`Probe` from `stamps.ts`, fill `DeployRecord.liveCheck`/`wrangler`. S3: build
+`PublishStatus` to satisfy `NeedsInput` (job metas → `lastIngestDoneAt` / `changedChannels`, config mtimes), spawn
+`stageCommand`, the `publish status|now` rows, the `publish-*` job kinds. S4: the editor's actions still call
+`buildSite` / `deploySite` on export/out. S5: `docker/publish-site.sh` and `build-site.sh` (the swap), and the
+envVars names above.
+
+| commit | what |
+|---|---|
+| `ffa95b73` | `dirSignature` moves to `lib/dirSignature.ts`, unchanged; compose imports it |
+| `0122cd1b` | the stamp files and the publish lock (+ tests) |
+| `75403df3` | per-target bundles: install, link, archive staging; `buildSiteBundle`/`buildHubBundle`; deploy `outDir`; tree-kill |
+| `0a7d88ac` | the seven stages, the runner, `inputSig`, the CLI rows and the aliases |
+| `e040bb3f` | tests: `needs()` per row, argv, bundle install (EXDEV), inputSig, stages over a scratch corpus, tree-kill, CLI |
+| `3033d9f2` | stamps fall back to the image's commit/branch; `deploy-homepage --to local` → `ARCHILYZER_HOMEPAGE_OUT` |
+| `89b16716` | `.gitignore`: `/export/out` |
+
+**Gates** (all from the worktree root): tsc clean at every commit; common **3219 passed** (58 new: stamps 6,
+stageLock 8, stages 21, bundle 7, stageRun 6, inputSig 4, runChild 2, `_cli` +4); editor unit **142**;
+`test:scripts` **596 + 3 skipped**; mcp **289**; export unit **116**; homepage unit **23**; `pnpm --filter editor exec
+next build` ok (402 s, the machine busy); `pnpm --filter export exec next build` ok (59 s, over a scratch full site —
+see "Found"); `pnpm --filter homepage run build:nodata` ok (44 s); umtool's capped build ok (38 s). e2e (editor
+suite, `s1-specs.txt`: build, deploy-page, site-publish-preview, sites-homepage, duplicate-shorts, cut-release,
+ops-api): **52 passed, 0 failed, 2.5 min** — after a first launch died on "Timed out waiting 120000ms from
+config.webServer" (the linked primary `export/public` held a cited site's compose, so the export dev server 500ed on
+`summaries/manifest.json`); re-run with a scratch FULL site composed into the worktree's own `public/` (FACTS :3485's
+"Copy a composed fixture site into it"), cleaned after. Numbers tool: none. Live smoke over a scratch corpus (`s1-smoke-build.sh`): `publish index`
+(9 s) → `publish build smoke` (85 s, bundle installed by rename, export/out a relative link) → again: no-op →
+`stage deploy-site … --to local` without the env: refused → `publish deploy smoke --to local`: copied + recorded →
+`publish hub`: refused by `builtHubProblem` (above) → `build site smoke --nodata`: alias printed, forced rebuild.
+
## Rollout
(Steps 1–7 above; "### As it went" is written as the rollout runs.)
