commit 64ecd5b34ea2e40ae6ae94884a87d12ac8a810e9
parent cee23eea84b8a1fbec291fa4038574dcf50f8982
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 6 Oct 2026 08:46:29 -0400
plans: release 18 — slice S5 (image half) as shipped; the editor changelog
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
2 files changed, 142 insertions(+), 0 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,6 +1,9 @@
# Changelog
## [Unreleased]
+- **Substitute your own yt-dlp in Docker.** Point `YTDLP_BIN` at a zipapp you built, or set `YTDLP_SOURCE_HOST_DIR` to a yt-dlp checkout and start with `docker-compose.ytdlp.yml`: the image runs it with its own python, and nothing is rebuilt. Every editor boot logs `yt-dlp: <path> <version> (image|override)` (`MISSING` when it does not run; the editor still starts), and `YTDLP_AUTO_UPDATE` updates the image's yt-dlp only, warning instead of touching yours.
+- **The Docker image can publish.** It carries python, `pipx` and a pinned `git-filter-repo`, so the homepage's `/source` mirror builds in the container; `docker-compose.source.yml` mounts your repository read-only for it, and the scrub rules and denylist live in the config volume (`/data/config/archilyzer`). Cloudflare and R2 credentials come from `.env`. Run publish commands with `docker compose exec editor pnpm archilyzer …`, not `run --rm`. The `homepage` service serves a local deploy from the builds volume once there is one. RUNNING_IN_DOCKER.md has a Windows checklist.
+- **`archilyzer doctor` checks what a publish needs.** Which yt-dlp runs (the image's, the host's or an override, and whether it runs), whether the Cloudflare token and the R2 keys are set (never their values; R2 only when a bucket is configured), free space for the site bundles, the repository the source mirror reads, and the private config dir.
- **A cited moment at the very end of a recording prepares.** Prepare evidence media cuts a clip whose padding runs past the recording's end at the end (the recording's duration from its metadata), where it found no media for the padded span; a span that starts past the end is still refused. report-to-video keeps its strict rule.
- **Exporting a changed report records a new revision of it.** `reports export` (and **Export reports** on a site's Reports tab, and the end of a prepare) commits a revision to the report's own git history, `sites/<site>/reports/<id>/history-git/`, whenever its `report.json` changed since the last one: the `report.json`, its Markdown export and the checksums of every export file, with a message of `Revision N` and a summary of the change. A re-export of an unchanged report records nothing. The commits carry the site's name and a `noreply@<site>.invalid` address with dates in UTC, never your git name, email or time zone. The Reports tab shows each report's revision, its commit and the last change under **Exports**, and the site's next build publishes the history. Add `history-git/` to the corpus repository's `.gitignore`.
- **archive.org files come over BitTorrent when possible, else straight from archive.org — never through yt-dlp.** The chosen file of an archive.org import is fetched from the item's own torrent (`<identifier>_archive.torrent`, which lists archive.org as a web seed, so other peers take load off archive.org) with aria2c, only that file of the item, and seeded afterwards for 10 minutes or to a ratio of 1, whichever comes first; the log shows "torrent: <file> (n of m pieces, peers p, web seed yes)" and "seeding 10 min…". With no aria2c, a torrent that does not carry the file, or no progress for 5 minutes, it is downloaded directly from `archive.org/download/…` instead (resumable, backing off on 429/503), and the log says "fell back to direct download: <reason>". Every file is checked against archive.org's sha1/md5: a mismatch is downloaded once more directly, a second one fails the record. The record is written from the item's metadata: `metadata.info.json` with the file's page, the canonical id, the duration ffprobe measures and archive.org's playable copies of the file, the `archiveorg.json` provenance (a mirror's original title, date and uploader), and `audio.<fmt>` — an audio file already in the channel's format is used as is, anything else goes through the app's audio extraction, a video kept in the saved-video store when the channel keeps sources. An .avi/.mpeg/.flac/.wav original is fetched as archive.org's mp4 or mp3 of it. aria2c runs in its own process group: cancelling the job stops it and everything it started, and it stops itself if the editor exits. New settings block `archiveOrg` (`torrent`, `seedMinutes`, `seedRatio`, `stallMinutes`, `maxPeers`, `maxDownloadKiBps`, `maxUploadKiBps`), `ARIA2C_BIN`, an aria2c row in `archilyzer doctor`, and `aria2` in the runtime Docker images.
diff --git a/plans/release-18.md b/plans/release-18.md
@@ -353,6 +353,145 @@ Probe = { status|null; generatedAt?; cfCacheStatus?; age?; cacheControl?; error?
(Each slice adds a "### Slice <X>, as shipped" section here, before "## Rollout".)
+### Slice S5, as shipped — the image half: the container can publish, yt-dlp can be substituted, the doctor checks it (2026-10-06)
+
+Branch `r18/docker-publish` off `r18/integration` `ce66f2d3`, worktree `~/Projects/r18-docker-publish`
+(editor 7001, test 7011, export 7010), one Opus implementer. Scratch files `s5-*` in the job's `tmp`. The
+plan is "Docker (a)–(g)" and "`archilyzer doctor` adds" above. This is the IMAGE HALF: S1 and S2 had not
+merged, so the doctor's `wrangler`, `publish-lock` and `index-stamp` checks, the `WRANGLER_BIN` /
+`E2E_LIVE_CHECK` declarations and the publish-stage smoke are the second half (below, "Left").
+
+**What was found before building.**
+- `ARCHILYZER_CONFIG_DIR` is already honoured by `getPaths()` (`paths.ts:218`), and `FILTER_REPO_PIPX_SPEC`
+ (`git-filter-repo==2.47.0`) already existed in `source.ts:113` — the drift test pins the Dockerfile to it.
+- `source.ts` reads the repository with `git --git-dir=<repo>`, so `ARCHILYZER_SOURCE_REPO` must name a git
+ DIR (the host's common dir), not a work tree — which is what the overlay mounts.
+- `git filter-repo --version` prints the script's hash (`a40bce548d2c`), never `2.47.0`; the version is
+ read from `pipx list` (`git-filter-repo 2.47.0`).
+- `env_file: .env` (x-app) already passes every key of `.env` to every app; `x-app-env` sets none of the
+ credentials, so nothing overrides them. Verified by reading the merged `docker compose config`.
+- Debian's `python3-pycryptodome` installs as `Cryptodome` (yt-dlp tries it first) and bookworm's
+ `python3-websockets` is 10.4 — below what yt-dlp's websockets handler wants, so a from-source yt-dlp
+ runs without that handler (optional; only some live-stream extractors use it).
+
+**What it does.**
+- **Dockerfile.** runtime-base and runtime-cuda (which repeats it) add `python3 python3-venv pipx` and
+ yt-dlp's optional modules; `PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install
+ git-filter-repo==2.47.0`; `ARCHILYZER_IMAGE_YTDLP=/usr/local/bin/yt-dlp` beside `YTDLP_BIN`;
+ `/usr/local/bin/yt-dlp-from-source` → `docker/yt-dlp-from-source.sh` (refuses with a sentence and exit
+ 127 when no `yt_dlp/` package is mounted). `ARCHILYZER_COMMIT` / `ARCHILYZER_BRANCH` build args become
+ ENV as the LAST layer of each of the three targets, so a new commit re-runs one ENV layer. The three
+ targets and the glibc ordering are untouched; wrangler is not installed globally.
+- **Entrypoint.** Every editor boot prints `yt-dlp: <path> <version> (image|override)` after the
+ optional self-update — `MISSING` (with the first stderr line) when it is not there or does not run; the
+ editor still starts. image vs override compares the two paths after `readlink -f`. `update_ytdlp` skips
+ an override with a two-line warning. `/data/source.git` (or `ARCHILYZER_SOURCE_REPO`) is added to git's
+ `safe.directory` once — only when absent, so a restarted container does not pile up entries.
+ `ARCHILYZER_CONFIG_DIR` is made (700, empty) when absent. The `homepage` service serves
+ `ARCHILYZER_HOMEPAGE_OUT` (`/data/builds/homepage`) when it is non-empty, else the baked
+ `homepage/out` — chosen at boot. `exec "$@"` stays.
+- **Compose.** `x-app-env` gains `ARCHILYZER_CONFIG_DIR=/data/config/archilyzer` and
+ `ARCHILYZER_HOMEPAGE_OUT`; `x-app.build.args` passes the two build facts from the shell; the
+ `homepage` service mounts `builds`. New overlays: `docker-compose.source.yml` (`ARCHILYZER_SOURCE_HOST_DIR`,
+ default `./.git`, read-only at `/data/source.git`; sets `ARCHILYZER_SOURCE_REPO`) and
+ `docker-compose.ytdlp.yml` (`${YTDLP_SOURCE_HOST_DIR:?…}` read-only at `/opt/yt-dlp-src`; sets
+ `YTDLP_BIN=/usr/local/bin/yt-dlp-from-source`).
+- **`source.ts`.** `sourceRepoFor(ctx, cwd)` — `ARCHILYZER_SOURCE_REPO` first, then the checkout's common
+ dir — replaces both `commonDir` call sites (the publish and `publishedSourceProblem`). A variable naming a
+ path that is not there is a `SourceRefusal` naming it (never the "no repository" sentence).
+- **`docker/publish-site.sh`** is a wrapper: usage, the early private refusal, then `publish index`,
+ `publish build <id>`, `publish deploy <id> --to local` (S1's CLI rows, by name).
+- **`envVars.ts`** (ENVIRONMENT.md regenerated): `CLOUDFLARE_API_TOKEN`, `ARCHILYZER_SOURCE_REPO`,
+ `YTDLP_SOURCE_HOST_DIR`, `YTDLP_SOURCE_DIR`, `YTDLP_AUTO_UPDATE`, `XDG_CONFIG_HOME` (runtime);
+ `ARCHILYZER_HOMEPAGE_OUT`, `ARCHILYZER_IMAGE_YTDLP`, `ARCHILYZER_COMMIT`, `ARCHILYZER_BRANCH`,
+ `ARCHILYZER_SOURCE_HOST_DIR` (docker); the R2/account rows and `ARCHILYZER_CONFIG_DIR` say where they
+ come from in Docker. Exported for S1's stamps: `IMAGE_COMMIT_ENV`, `IMAGE_BRANCH_ENV` and
+ `imageBuildFacts(env)` → `{commit, branch}` (an empty baked value is null).
+- **Doctor.** New section `downloader` (`yt-dlp`: path, version by exit status, `image` | `host` |
+ `override`; an override that does not run, or one beside `YTDLP_AUTO_UPDATE`, warns); new section
+ `publish` (`cloudflare-auth`: the token SET, or wrangler's login config by PATH, never read; warns only
+ when a site names a `cloudflareProject`; `r2-keys`: only with `archiveStorage.bucket`, names the unset
+ keys; `export-builds`: writable, free ≥ 1.5× the `<id>/out` bundles, a missing dir is a note);
+ `source publish` gains `source-repo` (`ARCHILYZER_SOURCE_REPO` naming nothing FAILS — the publish
+ refuses; no repository is a note, a warning once the operator's files exist; main's commit when it
+ reads) and `config-dir` (exists, entries counted, writable). No value is printed; still read-only.
+- **RUNNING_IN_DOCKER.md**: "Publish the archive" (stages in the container; `exec`, never `run --rm`;
+ Cloudflare from `.env`; the homepage and its `/source` mount, the config volume), "Substituting yt-dlp",
+ "Two build runners, and the container has one" (replaces the fallback section), the Windows checklist,
+ what is in the image, troubleshooting. `.env.example` names every new variable (not `E2E_LIVE_CHECK`:
+ a test knob does not belong in a real instance's `.env`).
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `060927fb` | `publish:` `sourceRepoFor` — `ARCHILYZER_SOURCE_REPO` first; a missing path refuses by name; 1 test |
+| `f0dfa39a` | `docker:` the image (python, pipx, filter-repo, the yt-dlp hook, build facts); entrypoint; compose + two overlays; `publish-site.sh` wrapper; envVars + ENVIRONMENT.md; 2 drift tests in `buildImage.test.ts` |
+| `e1a3b49b` | `doctor:` `downloader/yt-dlp`, `publish/{cloudflare-auth,r2-keys,export-builds}`, `source publish/{source-repo,config-dir}`; 5 tests |
+| `90722346` | `docker:` `.env.example` |
+| `32c16698` | `docker:` the entrypoint makes the config dir |
+| `ed5b5db8` | `docs:` RUNNING_IN_DOCKER.md |
+| this one | `plans:` this section; the editor changelog |
+
+#### Gates (logs `$T/s5-*.log`)
+
+- **tsc** (all workspaces) clean at every commit (91 s at `ed5b5db8`).
+- **common:** **3,169/3,169** (new: `doctor.test.ts` +5, `buildImage.test.ts` +2, `source.test.ts` +1).
+ **Editor unit:** 142/142. **mcp:** 289/289. **test:scripts:** 595 passed, 1 failed, 3 skipped (599) —
+ `queue-lock.test.mjs` "prints a banner naming the holder while waiting" at a load average of 24, with
+ the editor build running; the file alone afterwards: 11/11. The slice touches nothing under `scripts/`.
+- **Build:** `pnpm --filter editor exec next build` exit 0, 253 s.
+- **Image:** `docker buildx build --target runtime` in a builder capped at 8 GB (`--driver-opt memory=8g
+ memory-swap=8g`), `WHISPER_BUILD_JOBS=4`: exit 0 in 280 s from an empty builder cache, 241 s for the
+ rebuild after the doctor commit. `archilyzer:r18smoke` is **1.76 GB** (`docker image inspect .Size`).
+- **Compose smoke** (`-p r18smoke`, the `channel-with-counts` e2e fixture + `sites/testsite` copied into
+ `$T/s5-corpus` and bind-mounted over the corpus volume, `ARCHILYZER_FETCH_MODEL=none`,
+ `ARCHILYZER_IDLE_BOOT=1`, the editor alone): the boot log shows `yt-dlp: /usr/local/bin/yt-dlp
+ 2026.08.19 (image)`; `exec editor pnpm archilyzer doctor` lists `downloader/yt-dlp` ok (image),
+ `publish/cloudflare-auth` and `export-builds` (notes), `filter-repo` ok (`git filter-repo a40bce548d2c`),
+ `source-repo` and `config-dir` (exit 1 only for the model the smoke skipped); `python3 -c "import
+ yt_dlp"` exit 1 and `yt-dlp-from-source --version` exit 127 with its sentence; `pipx list` →
+ `git-filter-repo 2.47.0`; `ARCHILYZER_COMMIT`/`BRANCH` baked. With `docker-compose.ytdlp.yml` over a
+ two-file `yt_dlp/` package and `YTDLP_AUTO_UPDATE=1`: the entrypoint's skip warning, `yt-dlp:
+ /usr/local/bin/yt-dlp-from-source 2099.01.01.s5-smoke (override)`, the wrapper exit 0, the doctor's
+ `yt-dlp` WARN. `safe.directory` has one entry after a `docker restart`. With
+ `docker-compose.source.yml` over a throwaway repo in `$T`: `source-repo` ok with its main, `rev-parse`
+ as root works, the mount is read-only. The `homepage` service serves the baked build, then the builds
+ volume's after a file lands there and it restarts. `down -v` after.
+- **e2e:** none for this slice. **Numbers tool:** none. **Publish-stage smoke** (`publish
+ index/build/deploy` in the container): not run — S1's CLI rows are not on this branch; it is the
+ parent's after S1/S2 merge.
+
+**Deviations from the plan, one sentence each.**
+- `ARCHILYZER_HOMEPAGE_OUT` and `ARCHILYZER_SOURCE_HOST_DIR` are new names the plan did not list: the
+ first is where `deploy-homepage --to local` writes (S1 must read it), the second lets a worktree mount
+ the primary's `.git`.
+- `python3-venv` is installed beside `pipx` (pipx makes a venv); the plan's package list omitted it.
+- The build facts are ENV in each final target rather than once in runtime-base, so a commit does not
+ invalidate the Vulkan apt layer.
+- `config-dir` not writable is a note, not a warning: the publish only reads the rules.
+- The source-repo "homepage policy on" grade waits for S3's `settings.publish.homepage`; today the
+ warning keys off the operator's files existing (the doctor's existing "intends" signal).
+- `E2E_LIVE_CHECK` and `WRANGLER_BIN` are not declared yet: nothing on this branch reads them, and the
+ envVars test refuses a declaration nothing names (S2 adds the reads).
+
+**Left for the second half (after S1/S2 merge).**
+- Doctor `wrangler` (the binary from `wranglerBin(paths)`, its major against `WRANGLER_MAJOR`),
+ `publish-lock` (a dead pid in `.publish.lock`, via `stageLock.ts`), `index-stamp` (age; sites built from
+ an older stamp, via `stamps.ts`); `export-builds` to read `built.json` `bytes` instead of walking.
+- `WRANGLER_BIN` / `E2E_LIVE_CHECK` in `envVars.ts` (whichever of S2/S5 lands second).
+- The publish-stage smoke in the container (rollout's shape: `publish index && publish build <fixture> &&
+ publish deploy <fixture> --preview smoke`, no token → refused before wrangler, a bogus token → refused by
+ Cloudflare).
+
+**Found and left.**
+- The shared tool probe (`toolProbe.mjs`) counts any output from a failing `--version` as presence, so the
+ `tools/yt-dlp` row reads `ok` with the wrapper's sentence as its "version" when no checkout is mounted;
+ the new `downloader/yt-dlp` row asks by exit status and is the honest one. Not changed: the probe is
+ shared with `umtool doctor`.
+- `docker images` reported the previous `archilyzer:local` (6 weeks old) at 7.3 GB; this build is
+ 1.76 GB. Not investigated.
+
## Rollout
(Steps 1–7 above; "### As it went" is written as the rollout runs.)
