commit fadc46eb57697aaf645a2f213b81ab68d910cd33
parent a8de529ce42503854b06a2a1fb6d5283834f3040
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 6 Oct 2026 09:32:21 -0400
plans: release 18 — S2 review fixes recorded; deviations from the plan; the hub wording in the changelog
The ruling clarification (private data is never named on the hub), the
review-fixes table, a "Deviations from the plan" subsection, headings for
the found/left lists, open question 2 softened, and the gates after the
fixes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
2 files changed, 80 insertions(+), 17 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -2,7 +2,7 @@
## [Unreleased]
- **Deploys are pinned and checked live.** wrangler is an exact dependency of the workspace (4.147.0), so a deploy runs the version installed with the code instead of whatever `pnpm dlx` fetched that day, and every deploy names its branch: production is `--branch main`, never taken from the checkout it ran in (where a "production" deploy from a feature branch used to land as a preview). The publish stages' deploy (release 18) refuses before wrangler runs when there is no Cloudflare credential at all — "set CLOUDFLARE_API_TOKEN in .env" — and says "REFUSED by Cloudflare — the API token was not accepted" when Cloudflare rejects one; it refuses a production deploy of a build made from a branch other than `main`. After each deploy it reads `corpus.json` at the site's address twice, as a visitor would and cache-busted, and records the verdict: ok, stale-edge (the deployment is right, Cloudflare's edge still serves an older copy), mismatch, or unreachable. A verdict short of ok is a warning in the log; the deploy itself succeeded. What each target last shipped, where, and how it read is kept in `deployed.json` beside its build.
-- **Withdrawn X posts ship tombstones.** While X posts are private, a public site's build no longer just leaves an X channel's posts out: at every path they were served from it ships an empty stand-in — the channel's posts manifest with no pages, and an empty page for each page the channel has — served uncached. The hub, which carries no posts, ships the same for every X channel, with an empty posts manifest. Leaving a path out of a deploy does not take it off Cloudflare's edge, which kept serving a withdrawn copy for up to a week; a changed object at the same path replaces it. The hub's deploy reads each of those paths back.
+- **Withdrawn X posts ship tombstones.** While X posts are private, a public site's build no longer just leaves an X channel's posts out: at every path they were served from it ships an empty stand-in — the channel's posts manifest with no pages, and an empty page for each page the channel has — served uncached. The hub, which carries no posts, ships the same for every X channel a public site carries, with an empty posts manifest; a channel only private sites carry is never named on the hub. Leaving a path out of a deploy does not take it off Cloudflare's edge, which kept serving a withdrawn copy for up to a week; a changed object at the same path replaces it. The hub's deploy reads each of those paths back.
- **A cited moment at the very end of a recording prepares.** Prepare evidence media cuts a clip whose padding runs past the recording's end at the end (the recording's duration from its metadata), where it found no media for the padded span; a span that starts past the end is still refused. report-to-video keeps its strict rule.
- **Exporting a changed report records a new revision of it.** `reports export` (and **Export reports** on a site's Reports tab, and the end of a prepare) commits a revision to the report's own git history, `sites/<site>/reports/<id>/history-git/`, whenever its `report.json` changed since the last one: the `report.json`, its Markdown export and the checksums of every export file, with a message of `Revision N` and a summary of the change. A re-export of an unchanged report records nothing. The commits carry the site's name and a `noreply@<site>.invalid` address with dates in UTC, never your git name, email or time zone. The Reports tab shows each report's revision, its commit and the last change under **Exports**, and the site's next build publishes the history. Add `history-git/` to the corpus repository's `.gitignore`.
- **archive.org files come over BitTorrent when possible, else straight from archive.org — never through yt-dlp.** The chosen file of an archive.org import is fetched from the item's own torrent (`<identifier>_archive.torrent`, which lists archive.org as a web seed, so other peers take load off archive.org) with aria2c, only that file of the item, and seeded afterwards for 10 minutes or to a ratio of 1, whichever comes first; the log shows "torrent: <file> (n of m pieces, peers p, web seed yes)" and "seeding 10 min…". With no aria2c, a torrent that does not carry the file, or no progress for 5 minutes, it is downloaded directly from `archive.org/download/…` instead (resumable, backing off on 429/503), and the log says "fell back to direct download: <reason>". Every file is checked against archive.org's sha1/md5: a mismatch is downloaded once more directly, a second one fails the record. The record is written from the item's metadata: `metadata.info.json` with the file's page, the canonical id, the duration ffprobe measures and archive.org's playable copies of the file, the `archiveorg.json` provenance (a mirror's original title, date and uploader), and `audio.<fmt>` — an audio file already in the channel's format is used as is, anything else goes through the app's audio extraction, a video kept in the saved-video store when the channel keeps sources. An .avi/.mpeg/.flac/.wav original is fetched as archive.org's mp4 or mp3 of it. aria2c runs in its own process group: cancelling the job stops it and everything it started, and it stops itself if the editor exits. New settings block `archiveOrg` (`torrent`, `seedMinutes`, `seedRatio`, `stallMinutes`, `maxPeers`, `maxDownloadKiBps`, `maxUploadKiBps`), `ARIA2C_BIN`, an aria2c row in `archilyzer doctor`, and `aria2` in the runtime Docker images.
diff --git a/plans/release-18.md b/plans/release-18.md
@@ -395,14 +395,20 @@ export 6910), one Opus implementer, beside S1 (stage core) and S5's image half.
from `dockerSiteStagingDir` (`<id>/.r2-staging`), a preview logging `PREVIEW_SHARES_ARCHIVES_NOTICE`; the pinned
wrangler with the classifier on its stream; the live check; and LAST the record, written atomically (temp +
rename) into `deployed.json` (`production` / `local` / `previews[branch]`, other slots kept). A refusal or failure
- throws `DeployStageError` (`exitCode` 1, 3 or 130) whose message is the line the log already ends on, and
- `deployed.json` is untouched. The stamp types (`BuiltStamp`, `DeployRecord`, `DeployedFile`) are declared there with
+ throws `DeployStageError` (`exitCode` 1 refused/failed, 2 a request it cannot run — a bad branch name, local with a
+ preview, the hub locally, a kind and target that do not match — 3 precondition not met, 130 cancelled) whose message
+ is the line the log already ends on, word for word, and `deployed.json` is untouched. `deploy-hub` and
+ `deploy-homepage` take only `_hub` / `_homepage`, and a site deploy refuses either. Wrangler's own lines reach the log
+ through the same `log()`, each ending in a newline. A local destination that holds the checkout, the corpus, the
+ builds or the bundle, or is not empty and has no `index.html`, is refused before anything is emptied. Cancel reaches
+ the live check; a deploy cancelled during it is recorded without one and ends 130. The stamp types (`BuiltStamp`, `DeployRecord`, `DeployedFile`) are declared there with
the plan's field names until S1's `stamps.ts` lands.
-- **`common/publish/liveCheck.ts` — `runLiveCheck`** (injected `fetch`, `sleep`, `now`, `env`): `<url>/corpus.json`
+- **`common/publish/liveCheck.ts` — `runLiveCheck`** (injected `fetch`, `sleep`, `now`, `env`, `signal`): `<url>/corpus.json`
plain and `?cb=<builtStampId>` (`&try=N` on a retry), 3 tries 10 s apart until ok, records `cf-cache-status`, `age`,
`cache-control` and `generatedAt`, compares with `built.corpusGeneratedAt` (else the bundle's own `corpus.json`).
- Verdicts (`liveCheckVerdict`, pure): `ok`; `stale-edge` (busted serves this build, plain does not); `mismatch`
- (busted serves another); `unreachable` (neither answers 2xx); `skipped` (`E2E_LIVE_CHECK=skip`, nothing fetched).
+ Verdicts (`liveCheckVerdict`, pure): `ok` (both serve this build, or plain does and busted failed); `stale-edge`
+ (busted serves this build, plain answers 2xx with another `generatedAt`); `mismatch` (busted serves another);
+ `unreachable` (neither answers 2xx, or the plain read fails); `skipped` (`E2E_LIVE_CHECK=skip`, nothing fetched).
The URL is the preview alias for a preview, the site's `siteUrl` (the hub's `homepage.json` `siteUrl`, the homepage's
`PROJECT_URL`) for production, else the deployment URL wrangler printed. The homepage has no `corpus.json`: it reads
`/` and asks only for a 2xx. The hub also probes `posts/manifest.json` and each withdrawn channel's
@@ -411,16 +417,21 @@ export 6910), one Opus implementer, beside S1 (stage core) and S5's image half.
`liveCheckLines` is the log: `[live] ok — …`, or `[live] WARNING <verdict> — …` with every probe's status,
`generatedAt`, `cf-cache-status`, `age`, `cache-control`. A warning never fails the stage.
- **Tombstones, `common/publish/tombstones.ts`:** `writePostsTombstones` replaces each slug's served `posts/<slug>/`
- with its `ChannelPostsManifest` at `pageCount: 0` (`slugToPage: {}`, the shared tree's `maxPageBytes`) and `[]` for
+ with its `ChannelPostsManifest` at `pageCount: 0` (`slugToPage: {}`, `maxPageBytes: 0` — no size) and `[]` for
every page below the SHARED tree's `pageCount` — through `bin/_publicFile.ts`, never through a worktree link;
- `withdrawnXChannels` (every X channel with a shared posts tree, only while `social.x.visibility` is private);
+ `withdrawnXChannels` (every X channel with a shared posts tree that at least one non-private site carries, only
+ while `social.x.visibility` is private);
`tombstonePaths`, `tombstoneNoStoreForSite`, `tombstoneNoStoreForHub`.
- **compose-site** writes them for each withheld member (`site.channels` less `publishedMemberSlugs`) after the posts
reconcile and the posts manifest; with no posts manifest from the index it writes an empty one, replacing whatever
an earlier compose left there. `corpus.json` advertises no posts for them (it reads the posts manifest).
- **compose-hub** removes `SITE_ONLY_PUBLIC_ENTRIES` as before, then writes the tombstones and an empty
- `posts/manifest.json`; `main` takes `settings` (default `getSettings()`). With X public, or no X channel, the hub
- ships no `posts/` at all, as before. For the X channel the live hub served, that is, path for path,
+ `posts/manifest.json`; `main` takes `settings` (default `getSettings()`). **Ruling clarification (review H1):
+ private data is never named on the hub.** The plan's "every X channel with a shared posts tree" is narrowed to
+ the X channels that at least one site whose audience is not private carries (`site.channels`): only those could
+ ever have been served by a public bundle, so only their paths can sit at the edge. An X channel only private sites
+ carry, or no site, gets no tombstone — its slug appears nowhere in the hub's `public/` (tested). With X public, or
+ no such channel, the hub ships no `posts/` at all, as before. For the X channel the live hub served, that is, path for path,
`posts/manifest.json` (empty), `posts/thequartering-X/manifest.json` (`pageCount` 0),
`posts/thequartering-X/page-0000.json` (`[]`), and one `[]` page per further page its shared tree holds by then —
`compose-hub.test.ts` pins the three named paths with that slug.
@@ -474,15 +485,18 @@ back at its `*.pages.dev`), the homepage (`archilyzer`, 2026-09-26: wrangler pri
deploy has always passed `--branch main`), and the hub's first deploys (release 7). PUBLISH.md's "create the project
first" line is `wrangler pages project create <name> --production-branch main`. So `--branch main` changes nothing for
these eight projects. A project whose production branch is NOT `main` would now take a "production" deploy as a
-preview: production unchanged, and the stage's live check at the site's public URL reads `mismatch` — the warning
-names it.
+preview: production unchanged, and wrangler's output would show a preview URL. The live check reads `mismatch` only
+where it probes the production URL (a site with a `siteUrl`, the hub, the homepage) and the build's `generatedAt`
+differs from what production serves; a rebuild with unchanged data would still read `ok`.
+
+#### Found on the way, fixed here
-**Found on the way, fixed here.**
- compose-site left a stale `public/posts/manifest.json` from an earlier compose when the index wrote none for the
site; with tombstones to write it is now replaced by an empty one (`compose-site.postsVisibility.test.ts` case). A
site with neither tombstones nor an index manifest keeps the old behaviour.
-**Found and left.**
+#### Found and left
+
- **A site's live check does not probe its tombstones** — only the hub's does (the plan's scope). A site's tombstones
are written and served no-store; its `deployed.json` says nothing about them.
- **The site `generatedAt` is the summaries manifest's**, so a rebuild with no data change carries the same
@@ -496,7 +510,26 @@ names it.
- **`refuse` logs the sentence and throws it**: a runner that prints `err.message` after a failed stage will print it
twice. The wiring step (S1's `stages.ts`) should print only on a non-`DeployStageError`.
-**Commits**
+#### Deviations from the plan
+
+- **The hub's tombstone set** is narrower than the plan's wording: X channels a non-private site carries, not every X
+ channel with a shared tree (review H1; the ruling clarification above).
+- **The cache-busted key on a retry** is `?cb=<builtStampId>&try=N` (the plan: `?cb=<builtStampId>`): a retry needs
+ a key the edge has not seen either.
+- **A busted read that fails while the plain one serves this build is `ok`** — a visitor gets the build. The plan's
+ table did not name that case.
+- **The homepage's live check reads `/` for a 2xx** (the homepage has no `corpus.json`); comparing
+ `/source/manifest.json`'s commit is a follow-up.
+- **Exit 2** is used for request-shape refusals (bad branch name, local with a preview, the hub locally, a kind and
+ target that do not match); the S1 argv parser may catch most of them first.
+- **Files outside the slice's list:** `common/lib/builtExport.ts` (`isTombstonePostsTree` + one clause of
+ `builtHubProblem`), `common/publish/build.test.ts` (argv pins follow the spawn change), `pnpm-workspace.yaml`
+ (`workerd: false`), and `common/lib/envVars.ts` + ENVIRONMENT.md (above).
+- **A site's tombstones are kept per withheld member, as the plan asks** (review M3, an operator ruling to keep or
+ narrow): an X channel added to a public site after X went private is still tombstoned there, so the tombstone
+ names its slug and page count — and, since the review, no page size.
+
+#### Commits
| Commit | What |
|---|---|
@@ -504,7 +537,9 @@ names it.
| `82929307` | `common:` tombstones (`publish/tombstones.ts` + test), compose-site and compose-hub write them, `renderHeadersFile` `noStore`, `isTombstonePostsTree` + `builtHubProblem`; the postsVisibility, compose-hub, headers and builtExport tests |
| `cf920f8c` | `common:` `publish/deployStage.ts` + `publish/liveCheck.ts` (+ tests); `editor(e2e):` `fake-wrangler.mjs`, the playwright env; `envVars.ts` + ENVIRONMENT.md |
| `211b064f` | `common:` the homepage's local deploy is `ARCHILYZER_HOMEPAGE_OUT` (refused without it); the preflight is a token or a `wrangler login`; compose-site replaces a stale posts manifest beside tombstones; the `envVars.ts` rows split into this slice's and S5's (+ tests) |
-| this commit | `plans:` this section; the editor changelog |
+| `0f9621c4` | `plans:` this section; the editor changelog |
+| `801124c3` | `common:` the review fixes (below) |
+| this commit | `plans:` the review fixes in this section; the changelog's hub wording |
#### Gates (logs `$T/s2-*.log`)
@@ -526,7 +561,7 @@ names it.
time, whose `/` a plain export build cannot prerender (ENOENT `summaries/manifest.json`, exit 1, 47 s — the same on
any tree; no export file changed in this slice).
- **Numbers tool:** none. **Privacy gate:** counts only over this branch's added lines and this section — the source
- denylist's 4 entries: 0 hits.
+ denylist's 4 entries: 0 hits; identifiers with the suffix the plan forbids: 0.
| Run | At | Specs | Result |
|---|---|---|---|
@@ -546,6 +581,34 @@ names it.
| A deploy of the same `stampId` to the same slot is a no-op unless `force` | Always deploy (the plan's `needs()` already skips fresh stages; the no-op is the stage's own second word) |
| Only the hub's live check probes tombstones | Probe a site's too (a site's withheld channels are listed in no manifest a reader follows) |
| The homepage's live check reads `/` and asks only for a 2xx (it has no `corpus.json`) | Probe `/source/manifest.json` and compare its commit |
+| A public site tombstones every withheld X member (the plan), with `maxPageBytes: 0` | Tombstone only members a public build ever served (a channel added after X went private is never named) |
+| A deploy cancelled during its live check is recorded (with no check) and ends 130 | Leave `deployed.json` untouched, though the bundle is live |
+
+#### Review fixes (review SHIP AFTER FIXES, `$T/s2-review.md`)
+
+| # | Fix | Commit |
+|---|---|---|
+| H1 | The hub's tombstones only for X channels a non-private site carries; `compose-hub.test.ts` adds an X channel on a private site only and one on no site — neither is named anywhere in the hub's `public/`; `tombstones.test.ts` pins the set. Record and changelog say so (ruling clarification: private data is never named on the hub) | `801124c3`, this commit |
+| H2 | Node 22 in the image (the pinned wrangler needs `>=22`) — S5's files, left to S5 | — |
+| M1 | wrangler's lines through `log()`, each ending in a newline (test) | `801124c3` |
+| M2 | `deploy-hub` / `deploy-homepage` pinned to `_hub` / `_homepage`; a site deploy refuses either; exit 2, nothing written (test: `deploy-homepage` with target `jeralyzer`) | `801124c3` |
+| M3 | Per-withheld-member tombstones kept on a site as planned; `maxPageBytes: 0` | `801124c3` |
+| L1 | The live check takes the stage's `signal`: reads abort (`AbortSignal.any` with the timeout), the interval sleep wakes, no retry after Cancel; a deploy cancelled mid-check is recorded without one, exit 130 (tests) | `801124c3` |
+| L2 | A failed plain read is `unreachable`, never `stale-edge` (verdict table test) | `801124c3` |
+| L3 | Request-shape refusals exit 2 | `801124c3` |
+| L4 | R2 and wrangler failures throw the line they logged (test) | `801124c3` |
+| L5 | `--to local` refuses a destination that holds the checkout, corpus, builds or bundle, or is non-empty with no `index.html` — naming the path, emptying nothing (tests) | `801124c3` |
+| L6 | doctor importing `wranglerOAuthConfigFiles`, `ARCHILYZER_SITE_OUT`'s `readBy` — at the S5 merge | — |
+| L7 | `#### Deviations from the plan`; the bold labels are headings | this commit |
+| L8 | Open question 2's sentence softened | this commit |
+| L9 | PUBLISH.md's `pnpm dlx wrangler pages project create` — S6 | — |
+
+Gates after the fixes: tsc (all workspaces) clean; **common 3,199/3,199**, 135 s (+6: `deployStage.test.ts` +5,
+`liveCheck.test.ts` +1; `fetchPosts.test.ts` passes again at a load average of 16–19); e2e below.
+
+ | Run | At | Specs | Result |
+ |---|---|---|---|
+ | 4 (editor) | `801124c3` | the five specs of run 1 | **29 passed**, 0 failed, 2.4 min (no queue wait) |
## Rollout
