commit 4d36f8e7222a9c4940ee2f1f31cc50624d58c7a8
parent dcf45f12e40614b846e2cc335d15823058fc342c
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Sat, 10 Oct 2026 02:59:26 -0400
plans: release 19 A6–A9, as shipped (Track A of the overnight batch)
The four slices' records under Track A, tonight's gates appended to the Track
A gates, and the editor changelog's [Unreleased] bullets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
2 files changed, 116 insertions(+), 0 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,6 +1,12 @@
# Changelog
## [Unreleased]
+- **archive.org imports take many items, or a search.** `pnpm ops import-archive-org` now takes `"items"` (identifiers, or `{item, files | match}`) or `"query"` (an archive.org search, its first `"limit"` items) as one job, paced between files and between items. Records already held — on disk or in the saved-video store — are skipped; a dry run lists every file as held, RESTRICTED (archive.org will not hand it out) or would-get; three items in a row refused by archive.org stop the job and back the platform off.
+- **What an Odysee or BitChute channel lists that it does not hold.** `pnpm ops get remote-listing <slug>` reads the channel's listing once, on the platform's own queue at its pace, and prints the videos listed but not held and the ones held but no longer listed. Nothing is written.
+- **Cues without an index build.** `pnpm ops build-cues` writes `transcript.cues.json` for a channel (or chosen ids); `pnpm ops get transcript <id>` reads one video's cues off disk; the MCP's `get_transcript` reads a video the archive has not published yet through the editor, marked as such.
+- **Publish a private build elsewhere.** `archilyzer publish build <id> --out <dir>` lays the site's bundle out in a directory another server serves (hard links where it can); `--allow-missing-media` on it, and `"allowMissingMedia": true` on the ops publish `build` verb, let a report citation with no prepared media through.
+- **The MCP can ask the editor for archival work.** With the editor configured, `enqueue` queues a sync, download-missing, retry-bucket (chosen ids), transcribe-bucket, post fetch or video import; `get_job` follows it; `channel_coverage` shows what a channel holds by date and its gaps (a VOD mirror's videos by the day they were recorded); `notes` lists and reads the operator's umtool notes (`UMTOOL_URL`). Settings, storage and deletes stay `pnpm ops`. `pnpm ops get coverage <slug>` is the same coverage from a shell.
+- **Every `pnpm ops` action has its help paragraph** (twelve had none), so COMMANDS.md describes each.
- **A channel that mirrors another's streams can date its videos by the stream, not the upload.** A new channel setting, "Recorded date from the title (regex)" on the Configure form (`recordedDate.titlePattern` in config.json; `pnpm ops channel-config` with `recordedDateTitlePattern`), names where a title carries the recording's date — a regex with the groups `year`, `month` (a number or a month name) and `day`. The index then gives each of that channel's videos a `recordedDate`, which coverage reads before the upload date; a title without a date, or with one after the upload, keeps the upload date. Changing the pattern re-dates the channel's videos at the next index build.
- **A clip window of a video whose source is saved is cut from it, not fetched.** `fetch_clip`, `POST /api/media/fetch-window` and `pnpm ops fetch-windows` now cut a window out of the video's saved container (a persisted source, a full-source fetch, or media attached from a local archive) when it covers the seconds asked for, and answer at once as a cached window — no request to the platform, so a deleted channel's held videos are clippable. The window's sidecar records `source: "saved-video"`, and the video page marks it "cut from the saved video". A batch runs such windows as their own job on `clips:saved-video`, outside every platform's queue, hold and cooldown. A saved video whose file cannot be read (its drive unplugged, the file gone) is refused with the media guard's sentence rather than fetched; one that ends before the window is fetched as before.
- **`pnpm e2e` runs against a production build, and rebuilds it when the code changed.** The editor and umtool suites now run under `next start` by default (release 19's full editor suite: 24 min, against 71 under `next dev`). Before a run starts, the build's stamp — a fingerprint of the files the build reads, uncommitted edits included — is checked against the tree, and a stale build is rebuilt first through the heavy slot under a 5 GB cap, with the reason printed (`e2e build: rebuilding editor — common changed since the build …`). The test build has its own directory (`editor/.next/e2e`, `umtool/.next-e2e-start`), so it never replaces the build a running editor or umtool serves. `E2E_MODE=dev` runs `next dev` for iterating on one spec. The export and homepage suites stay on `next dev` (they are static exports). Every suite also writes `test-results/timings.json`, and `node scripts/e2e-timings.mjs` prints each spec file's time against the branch's last run. Four slow tests no longer wait on real clocks: the first rate-limit cooldown is 20 s and the clip-window gap 2 s on the test server only (`E2E_BACKOFF_BASE_MS`, `E2E_CLIP_WINDOW_GAP_MS`).
diff --git a/plans/release-19.md b/plans/release-19.md
@@ -242,6 +242,103 @@ merged in before this record. Scratch files `a-*` in the job's `tmp`.
loop (a re-run after its timeout gets the job already fetching), and `pnpm ops job wait <id>` waits with no timeout,
printing the queue position.
+#### Slices A6–A9, as shipped (2026-10-10, overnight Track A)
+
+Branch `r20/a6-a9` (worktree `r20-a6-a9`, ports editor 4001, test 4011, export 4010) off `r20/integration`
+`5b690d2e`, one Opus implementer, A6–A9 in order; `r20/integration` (release 20 D1) merged in before A9. Scratch
+files `a-*` in the job's `tmp`.
+
+| commit | slice | one line |
+|---|---|---|
+| `cf0bb7a6` | A6 | `import-archive-org {items \| query}`: one job over many items, held records skipped, RESTRICTED flagged, 401/403 storms stop; `remote-listing` / `get remote-listing <slug>` for Odysee/BitChute |
+| `f83d17b1` | — | a `usage()` paragraph for the twelve actions that had none (C2's found-and-left) |
+| `890b3355` | A7 | `build-cues {slug, ids?, force?}`; `GET /api/ops/transcript` / `get transcript <id>`; the MCP's `get_transcript` falls back to the editor |
+| `1b08ccc0` | A8 | `publish {verb: "build", allowMissingMedia}`; `archilyzer publish build <id> [--allow-missing-media] [--out <dir>]` |
+| `79b62feb` | — | `r20/integration` merged in (release 20 D1's `recordedDate.ts`) |
+| `d9318dbe` | A9 | MCP `get_job`, `enqueue`, `channel_coverage`, `notes`; `GET /api/ops/coverage` / `get coverage <slug>`; instructions, README, ENVIRONMENT |
+
+#### Slice A6, as shipped — imports and remote listings
+
+- **`import-archive-org` takes three shapes** (route, `importArchiveOrgAction`): `{item, files | match}` as before;
+ `{items: ["<id>" | {item, files? | match?}, …]}` (at most 500; a bare id takes the top-level `match`, else every
+ media original, each one record); `{query, limit?}` — archive.org's advanced search (`archiveOrgSearchUrl`: `q`,
+ `fl[]=identifier,title`, `sort[]=identifier asc`, `rows` ≤ 500, one request through the polite client's chain,
+ gap and backoff), its first `limit` (100) items. Every item is validated before any job; a stray shape is refused
+ by name.
+- **One job, the politeness shared** (`runArchiveOrgBatchImport`, `controller/archiveOrgImport.ts`): the download
+ gap (`archiveOrgGapMs`) before every download after the first across the whole batch, the consecutive-failure
+ count, the rate-limit stop (the rest `notReached`; a re-run resumes), plus **an inter-item gap**
+ (`ARCHIVE_ORG_ITEM_GAP_SECONDS` = 3 s, up to half again at random) before each item's metadata request, on top of
+ the client's 2 s. An item archive.org has no record of is `missing` and passed over. `runArchiveOrgImport` (one
+ item) is the batch of one, with its old result shape plus `restricted`.
+- **Held is skipped**: `isHeldArchiveOrgRecord` = `destinationExists` (what every download path asks) **or** a
+ saved-video pointer (`isSavedVideo`) — a record whose media lives in the saved-container tier is never fetched again.
+- **Restricted** (`archiveOrgRestriction`): an item carrying `access-restricted-item`, else each file marked
+ `private` — both answer a download with 401/403. A dry run lists each file as `held`, `RESTRICTED` or
+ `would get`; an import skips restricted files. A download answered 401/403 (`isRefusalError`: the client's
+ "archive.org answered HTTP 40[13]") marks the rest of its item restricted and is not a failure streak; **three
+ such items in a row stop the job** (`refusedStorm`) and back the platform off as a rate limit does. The log ends
+ with `summary: {dryRun, items, planned, imported, held, restricted: [{identifier, files}], failed, missing,
+ notReached, stopped?}`. A dry run is refused, as an import is, while archive.org is held or cooling down (it asks
+ archive.org for metadata and searches).
+- **`remote-listing {slug}`** (`controller/remoteListing.ts`, job kind `remote-listing`: platform queue, `needsText`,
+ not ingest): an Odysee or BitChute channel's own URL, one `fetchFlatPlaylistUrls` read (the platform's paced args)
+ on `platform:<p>` — one stream with every other request there — refused while the platform is held or cooling
+ down, after the platform's import floor (`waitForPlatformGap`), noting the floor after; an
+ `EnumerationIncompleteError` (429) backs the platform off. **Nothing is written** (no roster merge, no
+ maybe-missing). The log ends with `@@remote-listing {slug, platform, url, listedAt, listed, unparsed, held,
+ notHeld: [{id, url}], heldNotListed: [id]}`; `pnpm ops get remote-listing <slug>` posts it, waits and prints that
+ JSON on stdout (the transcribe result-marker path). Any other platform is refused, naming its sync.
+
+#### Slice A7, as shipped — cues without an index
+
+- **`build-cues {slug, ids?, force?, queueKey?}`** → the digest card's `normalizeChannelAction` (now `{ids, force}`;
+ `normalizeAllTranscripts` gains `videoIds` and `force`), on the channel's queue; every id must be held — a stray is
+ refused, named, before any job.
+- **`GET /api/ops/transcript?id[&slug]`** (`controller/videoCues.ts`, `pnpm ops get transcript <id> [--slug]`): one
+ video's cues off disk, nothing written — a fresh `transcript.cues.json` (`isCuesJsonFresh`), else what normalize
+ would write (`buildNormalizedTranscript`, factored out of `normalizeTranscript`, which now calls it), else the
+ English VTT alone when there is no metadata. Without `slug` the channel holding `data/<id>/` is found; two
+ holders are a 409 naming them. Text-guarded (503 when the drive is not answering).
+- **The MCP's `get_transcript`**, for a video its archive does not hold and no `track` asked, asks that route
+ through the configured editor (`mcp/src/editorOps.ts`, the fetch_clip env and timeout) and renders the cues headed
+ `NOT IN THE ARCHIVE — read from the local editor's disk`, with no moment links. No editor, or a 404, keeps
+ "video not found".
+
+#### Slice A8, as shipped — publish edges
+
+- `POST /api/ops/publish {verb: "build", allowMissingMedia: true}` carries the flag through `TargetAsk.build` →
+ the plan step → the stage request (`--allow-missing-media`); the docker runner refuses it (its builds do not take
+ it).
+- **`pnpm ops lane` covers `publish`** — shipped in A4 (`lane` takes `publish`); nothing added.
+- **`archilyzer publish build <id> [--allow-missing-media] [--out <dir>]`** (`common/bin/publish.ts`
+ `layOutBundle`): after the build — or the no-op of a fresh one — the bundle is laid out in `<dir>` with **hard
+ links** where the filesystem allows (a build installs a new bundle and never edits one in place), else copies;
+ symlinks kept. `<dir>` is emptied first (its contents) and refused **before any build** where a local deploy's
+ destination would be (`localDestProblem`); `--out` with `all` is refused. Not a deploy: nothing recorded.
+
+#### Slice A9, as shipped — MCP archival writes
+
+- On the fetch_clip pattern, one existing route each (`mcp/src/archivalTools.ts`): **`get_job`** (`GET
+ /api/ops/job/<id>?tail=N`: status, times, exit code, queue place and head, the log tail); **`enqueue`** (`kind`:
+ `sync` (`full`), `download-missing`, `retry-bucket` (`bucket`, `ids` — the way to download chosen videos),
+ `transcribe-bucket` (`ids`), `fetch-posts` (`full`, `older`), `import-video` (`url`); a key a kind does not take
+ is refused before any request; answers the job id and how to follow it); **`channel_coverage`** (`GET
+ /api/ops/coverage`); **`notes`** (umtool via `UMTOOL_URL`: `list` from `/api/browse/decisions`, `read` from
+ `/api/notes/context`).
+- **`GET /api/ops/coverage?slug[&gapDays][&from][&to][&list]`** (`controller/channelCoverage.ts`, `pnpm ops get
+ coverage <slug>`): every held video dated off disk with `coverageDate()` — the recorded date when the channel has a
+ `recordedDate` title rule and the title yields one, else the upload date — a small `metadata.info.json` parsed
+ whole, a big one read at both ends (16 KB head for `title`, 8 KB tail for `upload_date`); `undated` named, never
+ guessed; per year and month; gaps longer than `gapDays` (30). Text-guarded; nothing written.
+- **Ruling recorded: a notes reply is not written by the MCP.** umtool's `/api/notes` stamps every write as the
+ operator's (its own rule: no way to claim to be an agent there), so `notes` `reply` answers with the `umtool
+ notes reply <id> "<text>" [--resolve]` command instead of posting.
+- Settings, storage and deletes are not reachable from the MCP (ruled). `mcp/src/instructions.ts` gains the
+ archive-work step (both plans; only when asked; report the job, do not wait it out) and both heads say the MCP
+ changes nothing itself; README tool table; `UMTOOL_URL`, `WORKER_TOKEN`, `ARCHILYZER_EDITOR_URL` name their MCP
+ readers (ENVIRONMENT.md regenerated). `channel-config`'s help names `recordedDateTitlePattern`.
+
#### Track A gates (A1–A5, on the merged tip)
- `pnpm -r --no-bail --workspace-concurrency=1 exec tsc --noEmit` — clean before every commit and on the merged tip
@@ -271,6 +368,19 @@ merged in before this record. Scratch files `a-*` in the job's `tmp`.
ending `[stage] update-index _index: Done` — the index stage, which this branch does not touch, finishing late
under load.
+**Tonight, A6–A9 (on `d9318dbe`):**
+
+- `pnpm -r --no-bail --workspace-concurrency=1 exec tsc --noEmit` — clean before every commit (A6 94 s, A7, A8, A9).
+- **common** 3672/3672 (+15 in the files this track added or extended: archiveOrgImport 7 → 12, remoteListing 4,
+ channelCoverage 3, publish-out 3; `_cli`'s publish-row case updated). **editor unit** 237/237 (+9 here:
+ import-archive-org 1, remote-listing 2, build-cues 3, publish +1, coverage 2). **test:scripts** 732: 729 pass, 3 skip
+ (`archilyzer-ops.test.mjs` 59 → 63). **mcp** 303/303 (298 → 303: editorOps 5, archivalTools 5, protocol's tool
+ list). `docs env|files|cli --check` 0.
+- **e2e** `ops-api.spec` alone, start mode (first build here 41 s, after a 1 min 37 s wait for the heavy slot):
+ **13 passed, 27.7 s**.
+- Not run (the brief): the full editor suite (the orchestrator's); an editor build of its own beyond the e2e start
+ build; a live `/ask`-style MCP session against the live editor.
+
### Track B
Branch `worktree-agent-a8b654c51bf472562` off `4cffda3f` (r18/integration with main merged), one Opus implementer,
