commit c127b4e017d70166652f2c038ca41882a3ffe401
parent 7314e90ff1e950806f1c3a486eb1bb414619adf8
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Fri, 25 Sep 2026 11:34:22 -0400
plans: release 7 slice C record — the archilyzer CLI, hub deploy path, posts-only fix
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
2 files changed, 110 insertions(+), 0 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -5,6 +5,10 @@
- **umtool's report videos can fetch Rumble clips again.** The clip fetch and the source availability check in `umtool/report-to-video` ran yt-dlp without the browser fingerprint Rumble now requires, so every Rumble clip failed with 403 and every Rumble source looked missing. They now pass the same Rumble arguments as the editor, from the same single table.
- **A transcript pulled back from a remote worker is written safely.** It used to be written straight onto `transcript.json`, so a crash part-way through left a truncated transcript; it now goes through the editor's one atomic write (temp file, then rename), like every other file the editor writes.
- **Nothing changes when you build or deploy a site; the code that does it has moved into the shared core.** The site build, the docker per-site fan-out, the R2 archive upload and the Cloudflare Pages deploy used to live inside the editor. They are now `common/publish/build.ts`, with the same log lines, exit codes and output paths, so a later command-line tool can build and deploy without the editor. The editor's Build, Deploy and Build & deploy controls and `pnpm ops build-site` / `build-deploy` / `deploy-site` call them as before. The AWS SDK packages used for the R2 upload moved with the code, from the editor's dependencies to the core's.
+- **`archilyzer`, one command line for building and publishing.** `pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts <command>` (export's scripts call it as `tsx ../common/bin/archilyzer.ts`): `index`, `compose site <id>|hub|homepage`, `build site <id> [--nodata] [--skip-archives]`, `build all [--skip-archives]`, `build hub`, `build homepage`, `deploy site <id> [--preview <branch>]`, `deploy hub [--preview <branch>]`, `deploy homepage`, `sync tick` and `settings example [--check]`. `--help` lists them. An unknown flag is refused before anything runs, so a misspelt `--preview` can never turn into a production deploy. The site commands are the same code the editor's Build and Deploy buttons run. Root `pnpm build:index` and `pnpm sync:tick` now go through it. The old `tsx bin/<name>.ts` scripts still work as before.
+- **The hub can be built and deployed.** On `/sites`, under Hub, there is now **Build hub** (tick **Deploy after build** to ship it in the same job) and **Deploy hub**. Both are also `pnpm ops build-hub` / `deploy-hub` and `archilyzer build hub` / `deploy hub`. The hub deploys to the Cloudflare Pages project set in the Hub form, and it refuses when none is set. It also refuses `archilyzer`, because that project is Archilyzer's own homepage. It builds into the same `export/out` as a site, so a deploy checks what is actually there first: deploying the hub refuses a site's build, and deploying a site refuses the hub's. The homepage deploys with `archilyzer deploy homepage` (project `archilyzer`, as before), and its front page shows a **Search all archives** link to the hub once the Hub form has a public URL. The example hub URL throughout is now `https://archilyzer-hub.pages.dev`.
+- **A site build runs its steps itself, and export's `build:nodata` and `prebuild` scripts are gone.** `build:nodata` had the same body as `build`. Only npm's `prebuild` hook told them apart: it ran the data phase for `build` and not for `build:nodata`. The data phase is now an explicit first step (skipped by **Skip data rebuild**, or `--nodata`), followed by compose and `next build`, with the same log notices. `pnpm run build` in `export/` is `archilyzer build site` (the site comes from `SITE_ID`, and `pnpm run build -- --nodata` works). `pnpm run deploy` now deploys to the site's own Pages project, where it used to pass no project at all. The docker build scripts call the same command.
+- **A posts-only channel's transcripts manifest is published again.** A social channel has no videos, so its transcripts folder in the build holds only a manifest saying "0 transcripts". The compose step read that folder as empty and deleted it, while the site's `corpus.json` still listed the manifest, so readers got a 404 (Jeralyzer's `thequartering-X`). That manifest is now copied like any other. The file format is unchanged.
- **Rumble works again, and a Rumble full sweep that gets rate-limited no longer fails the sync.** Every Rumble request had started coming back 403 from Cloudflare unless yt-dlp presents a browser fingerprint (yt-dlp #17496), so Rumble downloads failed and a Rumble channel could not even be added. Every yt-dlp run for a Rumble channel — sync, download, metadata scan, availability check, the clip-window fetch and the new-channel probe — now passes `--impersonate chrome --sleep-requests 1`, from one table in the code; a channel's own extra yt-dlp arguments still come last and still win. Separately, a full sweep that hits HTTP 429 part-way through the listing used to fail the whole sync and try again on the next one, so a large channel (The Quartering on Rumble, 44 days) never synced at all. What it read is now treated as *incomplete* — not a listing, so nothing is flagged missing and the stored playlist is untouched: the job records the platform's rate-limit cooldown, says "sweep incomplete: 429 at page N of the listing, M entries" in its log, does the ordinary newest-first sync instead, and succeeds. Syncs for that platform are then refused until its cooldown ends, and the full sweep is tried again after that. Any other yt-dlp failure still fails the sync as before.
- **A site can turn off its visitors' per-video transcript downloads.** The transcript viewer on a published site has always offered three ways to take a video's text away: a **Download** menu (txt, srt, json), **Copy MD**, and **Copy download command** (a `yt-dlp` line for a marked clip). A site's settings form now has a checkbox for them, *Per-video transcript downloads*, beside the archive zips one. Unticked, the site's next build shows none of the three; **Share** and the clip marks stay. It is on by default, so a site nobody touches is unchanged, and the file stores `"transcriptDownloads": false` only when it is off (`SITE.md` has the key). The site's machine contract (`/corpus.json`, `llms.txt`, the manifests and shards the MCP server and report-to-video read) is published either way. The hub follows the same switch: the hub form on **Sites** has the same checkbox, stored as `"transcriptDownloads": false` in the hub's `homepage.json`, and it hides the three controls on the hub's Browse and Ask pages. The editor's own video pages are unaffected.
- **Channel rows no longer scroll over a group's controls on `/channels`.** Scrolled down and to the right, the pinned Slug column of every row painted over the pinned group header and its five station buttons (Sync, Download, Transcribe, Digest and the speaker lane), and took the clicks. The pinned Slug cell and the group header sat at the same stacking level, and the later rows won. The rack now has one named layer order, kept in one file: the Advanced panel, then the column header, then the group header, then the pinned checkbox and Slug cells. Nothing ties any more. The screenshot audit found four more problems, fixed as well. A group header's name and buttons now stay on screen however far the columns scroll across (they used to scroll off to the left). An Advanced panel opened near the bottom or the right edge scrolls itself into view instead of being cut off. The rule above a pinned group header moves with it instead of leaving a gap the rows showed through. On a phone, the column header no longer paints over the selection bar pinned to the bottom of the screen.
diff --git a/plans/release-7.md b/plans/release-7.md
@@ -314,4 +314,110 @@ incomplete; the LM chat-only tier (operator config).
## Record
+### Slice C, as shipped — the archilyzer CLI, hub deploy path, posts-only fix (2026-09-25)
+
+Branch `one-core/r7-cli` off `main` `6ee1d336`. There is now one command line, `common/bin/archilyzer.ts`,
+over the publish layer and the bins. `common/publish/build.ts` has named entry points (`buildSite`,
+`deploySite`, `buildAll`, `buildHub`, `deployHub`, `composeHub`, `composeHomepage`, `buildHomepage`,
+`deployHomepage`), and the editor's jobs call the same ones. export's `build` / `build:nodata` twins
+and the `prebuild` hook are gone. The data phase is now an explicit step, so `pnpm run build` in
+`export/` can be the CLI without running itself. The hub now has a build and deploy path: on
+`/sites`, through `pnpm ops`, and through the CLI. The homepage deploys through the CLI. The
+posts-only 404 is fixed in the composer, and the contract is unchanged. Items 1–8 of the spec are
+done. `doctor`, `run` and `mcp` were not started (the cut line).
+
+| sha | what |
+|---|---|
+| `66f138cd` | `_parseFlags.ts` gains `parseArgv(argv, booleans) → {flags, positionals}` beside `parseFlags`. A declared boolean never takes the next word as its value (`build site --nodata jeralyzer`), and a lone `--` is skipped. The skip matters because pnpm 11 hands `pnpm run build -- --nodata` to the script as `-- --nodata`, `--` included (checked in scratch). New `_cli.ts`: `Command = {path, usage, flags?, maxPositionals?, run}`, `resolveCommand` (longest path), `argumentProblem` (unknown flag, a boolean given a value, a string flag given none, extra positionals), `usage`, `runCli` (a refusal exits 2 before `run`). `Command` gained `flags` and `maxPositionals` on top of the spec's `{path, usage, run}`, so a typo such as `--preveiw` is refused before a deploy runs instead of shipping production. The `test` glob gains `bin`. `_cli.test.ts` (9) |
+| `d97b6ad8` | The ten bins (`build-index`, `build-stats`, `build-chart-templates`, `build-archives`, `compose-site`, `compose-hub`, `compose-homepage`, `sync-tick`, `settings-example`, `file-schemas-docs`) `export async function main(opts)` and auto-run through `runIfEntryPoint(import.meta.url, …)`. That is the `worktree.mjs:350` idiom with both sides realpath'd, because `import.meta.url` is always the real file and argv[1] can reach it through a workspace symlink. A returned number becomes `process.exitCode` without cutting the process short, and a throw prints and exits 1, as before. `compose-site` `main({siteId})` falls back to `SITE_ID` and throws when there is neither. It used to `process.exit(1)`. `sync-tick` `main()` returns 1 on an HTTP error, and `tick()` keeps the old `request failed:` line. `archilyzer.ts` is the table, with lazy `import()` per row. Root `build:index` → `archilyzer index`, `sync:tick` → `archilyzer sync tick`. Flag-driven bins (`_parseFlags` users) are untouched |
+| `5dabc262` | `buildSiteSteps({siteId, paths, skipData, skipArchives, baseEnv}) → [{command, args, cwd, env}]` lists the steps: `pnpm run build:data` (unless skipData), `pnpm run compose:site`, then `pnpm exec next build`. All three run in `export/` with the old env block (NODE_ENV, TRANSCRIPTS_DIR, EXPORT_PUBLIC_DIR, SITE_ID, and BUILD_ARCHIVES=0 when archives are skipped). The data phase keeps the env the old `prebuild` inherited, EXPORT_PUBLIC_DIR included, rather than `runHostScript`'s narrower one, which matters for the e2e server's `.export-public`. `runBuildPhase` runs the steps with the same `[notice]` lines. Named entry points: `buildSite`, `deploySite` (the preview, project and built-bundle refusals, then R2, then Pages; it throws the deploy job's exact sentences) and `buildAll({mode: "docker"\|"basic"})` (the serial loop lifted from `buildAllSitesAction`). `build-export`, `build-deploy` (its build half), `deploy-export` and `build-all` call them, and their pre-job refusals stay in the actions. export: `prebuild` and `build:nodata` deleted, `build` → `archilyzer build site`, `deploy` → `archilyzer deploy site` (it names the site's project and refuses without one). CLI gains `build site`, `build all` and `deploy site`. `build site` refuses an id with no `site.json`, because a missing site reads as defaults and would otherwise run the whole data phase first. `build.test.ts` +2 |
+| `e8460a16` | `docker/build-site.sh` → `pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts build site "$SITE_ID" --nodata`, `publish-site.sh` → `… build site "$SITE_ID"` |
+| `077fcba4` | Hub and homepage entry points. `buildHubSteps` runs `compose:hub`, then `next build` with `INSTANCE_MODE=hub`, in `export/`. `buildHub` removes `public/site.json` first. `compose-site` now removes `public/hub-sites.json`, the one other line in that bin, so `out/` says which one it holds. `builtHubProblem(outDir)` (`builtExport.ts`, +1 test): a `site.json` means a site's bundle, even with a stale `hub-sites.json` beside it. `hubProjectProblem` refuses a missing project with the spec's sentence, and refuses `archilyzer` (`HOMEPAGE_PAGES_PROJECT`) by name. `deployHub` reads `getHomepageConfig(paths).cloudflareProject` and runs `pagesDeployArgs` through `runChildIntoLog`, with the `deploymentUrlIn` / `[preview]` line of `runDeployIntoLog`. `buildHomepage` is `pnpm run compose` then `next build` in `homepage/`. `deployHomepage` ships `homepage/out` to `archilyzer` with `--branch main`, as the hardcoded `homepage/package.json` line did (refused when there is no `out/index.html`). CLI: `build hub`, `deploy hub [--preview]`, `build homepage`, `deploy homepage`. export `build:hub` and homepage `deploy` call it. `build.test.ts` +2 |
+| `c82aad09` | `editor/app/sites/lib/hubActions.ts`: `buildHubAction` (kind `build-hub`, queue `build`), `deployHubAction` (`deploy-hub`, `deploy`) and `buildAndDeployHubAction` (`build-deploy-hub`, `deploy`). Before any job, they refuse a bad preview, a missing project or the homepage's project, and (deploy-only) a bundle that is not the hub. `HubBuildButtons.tsx` sits under the Hub form, in a group named `Hub build`: button **Build hub**, checkbox **Deploy after build** (unchecked by default, unlike the all-sites batch, because the first hub deploy should be a choice), button **Deploy hub**, lanes **Build hub** / **Build & deploy hub** / **Deploy hub**. The Hub section's copy now says the hub and the homepage are two projects. `/api/ops/build-hub` `{deploy?, preview?}` (a preview without deploy is a 400) and `/api/ops/deploy-hub` `{preview?}` (→ `previewUrl`). `archilyzer-ops.mjs` ACTIONS + usage + test (+1). `ops-api.spec` +1: no project, `archilyzer`, and not-a-hub-build are each refused, and no job starts. Hub-URL hints → `https://archilyzer-hub.pages.dev`: `HomepageConfigForm` placeholder (plus an `archilyzer-hub` placeholder on the project field), `SiteForm` Hub URL hint, `settingsSchema` `homepageUrl` (+ `SETTINGS.md` regenerated through `archilyzer settings example`), `mcp/README.md` :457, :508 |
+| `c66b9d4a` | Homepage hero: **Search all archives** → `homepage.json` `siteUrl` (the field `hubSite()` reads), rendered only when set. `marketing.spec` +1, conditional like the rail test: absent is legal, and a present link must be absolute |
+| `baa7ef45` | Posts-only 404. When the signature is `""` but `src/manifest.json` exists, `reconcileChannelTree` (now exported) copies the tree under the constant `MANIFEST_ONLY_SIGNATURE`. `corpus.ts` is untouched and spec stays 4. New `bin/compose-site.test.ts` (3): a manifest-only tree is copied (then skipped when unchanged, re-copied once pages arrive), an unchanged tree is skipped, and a member with no manifest is removed and a non-member pruned. Two of the three fail with the fix reverted (checked) |
+| *(this commit)* | this record, four `[Unreleased]` bullets |
+
+**The CLI as shipped** (`pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts …`, or
+`tsx ../common/bin/archilyzer.ts …` from `export/` / `homepage/`; `--help` anywhere):
+
+| command | does |
+|---|---|
+| `index` | `buildIndex` (the LMDB index) |
+| `compose site [<id>]` | compose-site `main({siteId})`, id else `SITE_ID` |
+| `compose hub` / `compose homepage` | compose-hub / compose-homepage `main()` (in-process) |
+| `build site [<id>] [--nodata] [--skip-archives]` | `buildSite`: data phase + compose + `next build` into `export/out`; refuses an unknown id |
+| `build all [--skip-archives]` | `buildAll`: docker fan-out when `docker version` answers, else serial host (as the editor's action decides) |
+| `build hub` | `buildHub` → `export/out` |
+| `build homepage` | `buildHomepage` → `homepage/out` (reads the index as it stands) |
+| `deploy site [<id>] [--preview <b>]` | `deploySite`: refusals, R2, Pages |
+| `deploy hub [--preview <b>]` | `deployHub` → `homepage.json` `cloudflareProject` (never `archilyzer`) |
+| `deploy homepage` | `deployHomepage` → project `archilyzer`, branch `main` |
+| `sync tick` | sync-tick (`SYNC_TICK_URL`, `SYNC_TICK_TOKEN`) |
+| `settings example [--check]` | settings-example `main({check})` |
+
+Exit codes: 0 ok, 1 failed or refused by the entry point, 2 usage (unknown command or flag, no site).
+
+**Gates** (worktree root, final tree before this commit). tsc clean at every commit. common
+**1771/1771**: 1754 + 9 `_cli` + 4 `build` + 1 `builtExport` + 3 `compose-site`. Editor unit
+**72/72**. test:scripts **160 pass + 1 skip** (159 + 1 ops). mcp **219/219**.
+`pnpm --filter editor exec next build` ok (compiled in 29.8 s, `/api/ops/build-hub` and
+`/api/ops/deploy-hub` listed). `pnpm --filter export exec next build` ok (12.5 s, no dangling
+`export/public` links). EDITOR e2e `sites-crud site-publish-preview build deploy-page cut-release
+channel-build-toggle ops-api` (all seven exist; `$T/c-specs.txt`): **49 passed, 0 failed, 2.9 min**,
+after 1m33s in the queue behind slice Y. EXPORT e2e in full (`node scripts/worktree.mjs run -- pnpm --filter export run e2e`, no dangling
+links): **192 passed, 0 failed, 7.4 min**.
+`e2e:hub` (`… pnpm --filter export run e2e:hub`): **8 passed, 16 s**.
+Homepage e2e (`… pnpm --filter homepage run e2e`, five specs): **15 passed, 7 skipped, 0 failed,
+33 s**. The 7 skips are `stats.spec.ts`'s `test.skip(noData, …)`: the worktree has no corpus data.
+The new hub-link test took its "absent" branch here, because the worktree has no `homepage.json`.
+Numbers: `plans/tools/phase3-files-numbers.ts` over one frozen copy (`FREEZE_TO`, 71 configs,
+1,763 sidecars; `TMPDIR=$T`), `6ee1d336` against this branch: **diff empty** (3,859 lines each).
+No build, deploy, compose or data build ran against the real corpus. The CLI's refusal paths were
+exercised in the worktree, which has no corpus: an unknown site, `--preview main`, no id, no
+project, no hub project and no homepage build.
+
+**Found and left.**
+- **No export e2e pins the posts-only fix.** No spec reads `corpus.json`, and the fixture site has
+ only `test-youtube`, with no social channel. The composer is pinned by `compose-site.test.ts`,
+ and the index side (a `pageCount: 0` manifest for a channel with no transcripts) by
+ `editor/e2e/build.spec.ts:10`. The live proof is `curl
+ https://jeralyzer.pages.dev/transcripts/thequartering-X/manifest.json` returning 200 after the
+ next jeralyzer build + deploy. That build needs no `--nodata` caveat, because the staging already
+ has the manifest.
+- **`Dockerfile.build` needs nothing.** It installs root + common + export `package.json` (tsx is
+ in common's devDependencies, and no `NODE_ENV=production` is set at install), then
+ `COPY . .`, so the CLI and the new `export/package.json` are baked from source. Every docker
+ fan-out calls `ensureBuildImage` (`docker build`, with cached layers reused) before Phase B, and
+ mounts the host `build-site.sh` over the baked one, so the editor's docker path never runs the
+ new script on an old image. Only an image built BEFORE this slice and run by hand, outside the
+ editor, would find `build:nodata` missing. The runtime `Dockerfile` copies all seven
+ `package.json` files and installs dev dependencies in its build stage, so `publish-site.sh`
+ has tsx too.
+- **A queued deploy now re-checks the bundle when it starts.** `deploySite` / `deployHub`
+ repeat the pre-job refusals inside the job. A deploy queued behind another site's build (which
+ is on the `build` queue, so they do not serialize) used to ship whatever that build left in
+ `export/out`. It now refuses. This is new behaviour, and deliberate.
+- **Two more hub-URL examples still name `archilyzer.pages.dev`**, outside this slice's files:
+ `editor/app/settings/components/SettingsForm.tsx:48` (the `homepageUrl` hint, which now
+ disagrees with `SETTINGS.md`) and the comment at `common/lib/homepage.ts:31`. Both are one-line
+ changes for slice 3. `DEPLOY_CLOUDFLARE.md:41` still says a raw `pnpm deploy` from `export`
+ only stages archives. `pnpm run deploy` is now `archilyzer deploy site`, which uploads them.
+ That doc is due to be absorbed into `PUBLISH.md` in slice 3. `settingsSchema.ts:417,433`
+ ("basic — `pnpm run build` in export/") is still true, since that script is now the CLI.
+- **`wrangler pages deploy` to a project that does not exist yet** asks whether to create it,
+ and cannot do that without a TTY. Neither `archilyzer-hub` nor (per the context) a deployed
+ `archilyzer` exists. Step 9 of the rollout should run the two first deploys from a terminal
+ (`archilyzer deploy hub` / `deploy homepage`), or create the projects in the dashboard first.
+ `pnpm ops deploy-hub` streams into a job log that has no terminal to answer.
+- `build homepage` does not rebuild the index. The homepage's own `pnpm run build` still does,
+ through its `prebuild` (its twins were left as they were, as specified). Run `archilyzer index`
+ first when the index is stale.
+- The `build-deploy` action's deploy half still calls the `run*` phases directly. It has its own
+ banners (a leading newline, no second built-bundle check straight after its own build), which
+ `deploySite` would change.
+- `common/lib/builtExport.test.ts` gained a test. It is the test of an owned file, but not on
+ the ownership list by name.
+- **Commit trailers** name `Claude Opus 5.5 (1M context)`, as in releases 5 and 6.
+
## Rollout
