commit 337df28c8d85a692df39555fc844ed4f5a0efa6d
parent 8885166c47ff136c00b360cd66a1469aac04ede2
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 6 Oct 2026 10:07:51 -0400
plans: S5 second half — publish-lock, index-stamp, built.json bytes, node vs wrangler; the publish-stage and source --check smoke in the container; the doctor's changelog bullet
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
2 files changed, 40 insertions(+), 2 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -7,7 +7,7 @@
- **`build site`, `build all` and `deploy site` are aliases of the publish commands** and print what they run: `build site <id>` is `publish index` (skipped with `--nodata`) then `publish build <id> --force`; `build all` is `publish index` then `publish build all --runner auto` (containers when an engine answers, else one site at a time on the host); `deploy site <id>` is `publish deploy <id>`, which now ships the site's own bundle and refuses a site never built that way. `publish build all --runner docker` builds every stale site in containers on a Linux host and refuses with "the docker runner needs an engine on this host" where there is none.
- **Substitute your own yt-dlp in Docker.** Point `YTDLP_BIN` at a zipapp you built, or set `YTDLP_SOURCE_HOST_DIR` to a yt-dlp checkout and start with `docker-compose.ytdlp.yml`: the image runs it with its own python, and nothing is rebuilt. Every editor boot logs `yt-dlp: <path> <version> (image|override)` (`MISSING` when it does not run; the editor still starts), and `YTDLP_AUTO_UPDATE` updates the image's yt-dlp only, warning instead of touching yours.
- **The Docker image can publish.** It carries python, `pipx` and a pinned `git-filter-repo`, so the homepage's `/source` mirror builds in the container; `docker-compose.source.yml` mounts your repository read-only for it, and the scrub rules and denylist live in the config volume (`/data/config/archilyzer`). Cloudflare and R2 credentials come from `.env`. Run publish commands with `docker compose exec editor pnpm archilyzer …`, not `run --rm`. The `homepage` service serves a local deploy from the builds volume once there is one. RUNNING_IN_DOCKER.md has a Windows checklist.
-- **`archilyzer doctor` checks what a publish needs.** Which yt-dlp runs (the image's, the host's or an override, and whether it runs), whether the Cloudflare token and the R2 keys are set (never their values; R2 only when a bucket is configured), free space for the site bundles, the repository the source mirror reads, and the private config dir.
+- **`archilyzer doctor` checks what a publish needs.** Which yt-dlp runs (the image's, the host's or an override, and whether it runs), whether the Cloudflare token and the R2 keys are set (never their values; R2 only when a bucket is configured), free space for the site bundles, the publish lock (free, held by a running stage, or left by one that is gone — with the command to clear it; never cleared for you), the index stamp's age and which sites were built from an older one, the repository the source mirror reads, the private config dir, and whether this Node is new enough for the pinned wrangler (deploys need 22).
- **A cited moment at the very end of a recording prepares.** Prepare evidence media cuts a clip whose padding runs past the recording's end at the end (the recording's duration from its metadata), where it found no media for the padded span; a span that starts past the end is still refused. report-to-video keeps its strict rule.
- **Exporting a changed report records a new revision of it.** `reports export` (and **Export reports** on a site's Reports tab, and the end of a prepare) commits a revision to the report's own git history, `sites/<site>/reports/<id>/history-git/`, whenever its `report.json` changed since the last one: the `report.json`, its Markdown export and the checksums of every export file, with a message of `Revision N` and a summary of the change. A re-export of an unchanged report records nothing. The commits carry the site's name and a `noreply@<site>.invalid` address with dates in UTC, never your git name, email or time zone. The Reports tab shows each report's revision, its commit and the last change under **Exports**, and the site's next build publishes the history. Add `history-git/` to the corpus repository's `.gitignore`.
- **archive.org files come over BitTorrent when possible, else straight from archive.org — never through yt-dlp.** The chosen file of an archive.org import is fetched from the item's own torrent (`<identifier>_archive.torrent`, which lists archive.org as a web seed, so other peers take load off archive.org) with aria2c, only that file of the item, and seeded afterwards for 10 minutes or to a ratio of 1, whichever comes first; the log shows "torrent: <file> (n of m pieces, peers p, web seed yes)" and "seeding 10 min…". With no aria2c, a torrent that does not carry the file, or no progress for 5 minutes, it is downloaded directly from `archive.org/download/…` instead (resumable, backing off on 429/503), and the log says "fell back to direct download: <reason>". Every file is checked against archive.org's sha1/md5: a mismatch is downloaded once more directly, a second one fails the record. The record is written from the item's metadata: `metadata.info.json` with the file's page, the canonical id, the duration ffprobe measures and archive.org's playable copies of the file, the `archiveorg.json` provenance (a mirror's original title, date and uploader), and `audio.<fmt>` — an audio file already in the channel's format is used as is, anything else goes through the app's audio extraction, a video kept in the saved-video store when the channel keeps sources. An .avi/.mpeg/.flac/.wav original is fetched as archive.org's mp4 or mp3 of it. aria2c runs in its own process group: cancelling the job stops it and everything it started, and it stops itself if the editor exits. New settings block `archiveOrg` (`torrent`, `seedMinutes`, `seedRatio`, `stallMinutes`, `maxPeers`, `maxDownloadKiBps`, `maxUploadKiBps`), `ARIA2C_BIN`, an aria2c row in `archilyzer doctor`, and `aria2` in the runtime Docker images.
diff --git a/plans/release-18.md b/plans/release-18.md
@@ -708,7 +708,7 @@ merged, so the doctor's `wrangler`, `publish-lock` and `index-stamp` checks, the
| `fc793037` | RUNNING_IN_DOCKER.md names Node 22 in the image's contents; this table |
| `55d779a1` | **The publish lock's host identity** (S1's review: `os.hostname()` in a container is its id, new on every recreate, so a crashed holder's lock would look foreign forever; S1's `stageLock.ts` reads `ARCHILYZER_HOST_ID ?? os.hostname()`): `ARCHILYZER_HOST_ID: archilyzer-editor` on the **editor service's** `environment`, not `x-app-env` — site, homepage and umtool share the builds volume, and a container carrying the same id with its own pid namespace would judge the editor's live lock dead and take it (visible in `docker compose config` either way; checked: only the editor has it). envVars row, no TODO needed: the compose file names it, which the test accepts (`readBy` names `stageLock.ts`, S1's). RUNNING_IN_DOCKER.md: why the id is fixed, that `run --rm` would now carry it with other pids (one more reason for `exec`), and how to clear a foreign-host lock (`rm /data/builds/.export-builds/.publish.lock`, only when nothing is publishing) |
| `c238970a` | SETUP.md: Node 22 — Next needs ≥ 20.9, deploying runs the pinned wrangler (≥ 22) |
-| this one | this table |
+| `65d549e7` | this table |
Re-run after the fixes at `07bc2395`: tsc (all workspaces) clean; `doctor`, `buildImage`, `source` and
`envVars` tests **61/61** (`$T/s5-fix-tests.log`).
@@ -721,6 +721,44 @@ load in the runtime — `lmdb` opens, writes and reads (`process.versions.module
binding loads; `archilyzer doctor` reports `node v22.23.3` ok and every S5 check, exit 1 only for the
model the smoke skips. vulkan and cuda still unbuilt (above).
+**Second half** (S1 merged: `r18/integration` `0ce00f76`, a fast-forward of this branch; S2 not yet)
+
+| Commit | What |
+|---|---|
+| `0df61c8c` | `doctor:` **`publish/publish-lock`** over S1's `stageLock.ts` — free → ok; a live holder on this host → a note; the lock's own `holderIsGone` (dead pid, a different start time, a pid younger than the lock) → stale, with `rm <exportBuildsDir>/.publish.lock`; a lock that has not parsed past `LOCK_TORN_GRACE_MS` → torn, the same; another host's (`lockHostId(env)` differs) → named, never judged. Never cleared by the doctor. **`publish/index-stamp`** over S1's `stamps.ts` — id, age, generation; the built targets (sites, `_hub`, `_homepage`) whose `indexStampId` is older, as a warning with `publish build <id>`; no stamp → "update the index first: archilyzer publish index" (a warning once anything is built or configured). **`export-builds`** sums each bundle's `built.json` `bytes` (a bundle no stamp describes is still walked). **`workspace/node`** grades against wrangler's `engines.node` when `common/node_modules/wrangler/package.json` is there (below it: a warning — every deploy refuses), read the way the image's drift test reads it. `stamps.ts`'s local `imageBuildFacts` became a re-export of `lib/envVars`'s (one definition; S1's `stamps.test.ts` 6/6 unchanged). 3 new doctor tests (lock: 5 states; stamp + bytes; node vs the floor) |
+| this one | `plans:` this table and the smoke; the doctor's changelog bullet names the new checks |
+
+Gates at `0df61c8c`: tsc (all workspaces) clean; `doctor`, `buildImage`, `source`, `envVars`, `stamps`,
+`stageLock` tests **82 passed, 1 skipped** (the wrangler floor — not installed here).
+
+**Compose smoke, second half** (`--target runtime` rebuilt from `0df61c8c`, 363 s, 1.79 GB; `-p r18smoke`;
+the e2e `curated-tags-channel` fixture — 3 videos with VTTs — and one site `s5site` copied into
+`$T/s5-corpus2`; `docker-compose.source.yml` over a throwaway repo in `$T`; editor + `site`; `down -v` after,
+nothing left):
+- `exec editor pnpm archilyzer publish index` → exit 0, 10 s (index +3, stats built, signatures, the stamp).
+- `publish build s5site` → exit 0, 76 s: "bundle installed at /data/builds/.export-builds/s5site/out (copied
+ across filesystems)", 198 files, 5.3 MB. Again → "fresh — nothing to do", exit 0.
+- `publish deploy s5site --to local` → exit 0, 4 s, copied into `/data/builds/site`; the `site` service
+ serves it: `corpus.json` names `s5site` (1 channel, 3 videos), `/` 200.
+- `publish deploy s5site --preview smoke` with no token: **skipped — the credential preflight lands with
+ S2.** S1's deploy body still calls `build.ts`'s `deploySite` (unpinned `pnpm dlx wrangler`), which has no
+ preflight to refuse before wrangler; the bogus-token run is the third round's.
+- `source publish --check` over the throwaway repo (read-only, host-owned, `safe.directory`) with throwaway
+ scrub/denylist files put in the config volume by `docker compose cp` (mode 600): exit 0, 6 s — "check
+ passed — would publish main 60a126e08086 as 60a126e08086: 12 files … nothing written"; gitleaks skipped
+ with its WARNING and no history pages, as RUNNING_IN_DOCKER.md says. This is the review's open medium:
+ the container's mirror over the `:ro`, foreign-owned mount works.
+- `doctor` (exit 1 only for the model the smoke skips): `node v22.23.3` ok, `downloader/yt-dlp` ok (image),
+ `cloudflare-auth` note, `export-builds` "1 bundle, 5 MB" (from built.json), `publish-lock` "free",
+ `index-stamp` "…, generation 1; 1 bundle built from it", `filter-repo` ok, `source-repo` ok (main
+ 60a126e08086), `config-dir` "2 entries", `scrub rules` / `denylist` ok (1 each, mode 600 — counted).
+- `publish status` and `publish now` (RUNNING_IN_DOCKER.md names both) are S3's rows, not on this branch
+ yet: `archilyzer: unknown command "publish status"` here.
+
+**Third round** (after S2): the `wrangler` check (`wranglerBin` / `WRANGLER_MAJOR`), `cloudflare-auth`
+through `cloudflareCredentialProblem` + `wranglerOAuthConfigFiles`, the envVars `readBy` touch-ups and
+dedupe, the no-token and bogus-token deploy smoke.
+
## Rollout
(Steps 1–7 above; "### As it went" is written as the rollout runs.)
