commit a21a7592179d5b31871e52b16fbf27c5e41aaa71
parent ab142494a9039c18a3afe6a62830be8f203f5a51
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Fri, 9 Oct 2026 11:42:02 -0400
plans: FACTS for main's October work outside release 18; release 19 moves the ops route tests to A2b and B2 after B5
- FACTS "Main's October work, outside release 18" (verified at 85d56ad9): ops transcribe (body, job kind, the
16 kHz cut, word timings from parakeet only, durationMs including the worker wait, the result marker); the
metadata refresh (job, yt-dlp pass, history writer, refusals); fetch-windows (queue by the window's URL, 30 s /
Rumble 120 s pacing, the cross-run gap is in memory, the cache, dedupe within one request only, Rumble picky
first); channel create/rename/delete over ops; umtool's notes store (where, shape, one writer, who stamps what,
never published) and kinds declaring capabilities.
- release-19.md: the ops-api.spec route cases are A2b (Track A owns the ops routes), out of B6; B2 is Track B's
after B5, reviewed by the Candace session before merge (ruled 2026-10-09).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
2 files changed, 118 insertions(+), 7 deletions(-)
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -8974,3 +8974,113 @@ Build stats jobs carry a "Superseded by release 18" line pointing here.
`Authentication error [code: 10000]` and exits 1 with no argv sidecar. `--version` answers
`4.147.0`. `publish.spec.ts` reads and writes both under `test-transcripts/.export-builds`
(`:32`, `:65`, `:174`).
+
+## Main's October work, outside release 18 (verified 2026-10-09, `4cffda3f`)
+
+The record is [`landed-2026-10.md`](landed-2026-10.md). Every anchor below was read at `4cffda3f`.
+
+### `pnpm ops transcribe` — one local file through the editor's workers
+
+- **The body is `path`, `start`, `end`, `workerId`, `out`, `words`** (`TRANSCRIBE_FILE_BODY_KEYS`,
+ `common/controller/transcribeFile.ts:71-78`); any other key is a 400. `path` is absolute and may be anywhere,
+ the corpus included (`:15-17`); `out` is refused inside any corpus root (`:307`); `"words"` must be a boolean
+ (`:171`).
+- **Job kind `transcribe-file`** (`:64`), `queueKey: ""` (`:547`): parallel at the queue; the worker pool
+ serialises it (`common/jobs/jobKinds.ts:884-896`, not drainable, not replayable). Only `kind: "local"` workers;
+ default = the one auto-transcribe would get.
+- **The audio is always a 16 kHz mono WAV cut by ffmpeg** into scratch (`:21-25`, `:374`), so parakeet's wrapper
+ never writes beside the source.
+- **The result** (`TranscribeFileResult`, `:107`) carries `cues` (seconds, shifted back onto the SOURCE file's
+ clock), `text`, `worker`, `durationMs`, and with `"words": true` a `words` array of
+ `TranscribedWord = {w, start, end, conf?}` (`:97`). Only parakeet emits word timings (`--words`,
+ `common/lib/transcriptionApps.ts:252-256`); any other engine gives `words: []`.
+- **`durationMs` includes the wait for a worker**: `started` is taken at `:444`, before the ffmpeg cut and the
+ "Waiting for a free local worker…" lease (`:472`); it is read at `:511`. (Release 19 B4 changes this.)
+- **The job's last log line is `@@transcribe-result <json>`** (`TRANSCRIBE_RESULT_MARKER`, `:68`, the same
+ string at `scripts/archilyzer-ops.mjs:205`); `pnpm ops transcribe --wait` prints that JSON alone on stdout
+ (`resultMarker`, `:331`).
+
+### Refresh one video's metadata
+
+- **Job kind `refresh-metadata`** (`REFRESH_METADATA_JOB_KIND`, `common/controller/refreshVideoMetadataJob.ts:47`;
+ `common/jobs/jobKinds.ts:648-656`: platform-queued, replayable, `needsText`). It runs on the channel's
+ download queue (`resolveQueueKey(downloadQueueKey(channelConfig), …)`, `refreshVideoMetadataJob.ts:95`); a
+ rate limit records the platform's backoff, a clean pass settles it (`:115-116`).
+- **One yt-dlp spawn**: `--skip-download --write-info-json`, no subtitles, `--sleep-requests 1`, the channel's
+ extra args before the negations (`buildRefreshArgs`, `common/controller/refreshVideoMetadata.ts:181`).
+- **The rewrite goes through `withMetadataHistory`** (`:424`) as writer `"refresh"` (`:418`;
+ `common/lib/metadataHistory.ts:80`), so what moved lands in `metadata.history.json`.
+- **Refused:** a video with no `data/<id>/` (`notFetchedRefusal`, `:92` — the directory is never created); a
+ record a non-yt-dlp writer owns — archive.org, Wayback, feed backfill (`NON_YTDLP_WRITERS`, `:154`); a held
+ or cooling platform. The same server action backs the video page's button and `POST /api/ops/refresh-metadata`
+ `{slug, id, queueKey?}`.
+
+### fetch-windows — a paced batch of clip windows
+
+- **One job per queue key, and the WINDOW'S URL picks it**, not the channel's (`planFetchWindows`,
+ `editor/app/channels/[slug]/videos/fetchWindowsAction.ts:165-169`, `queueKeyForUrl`). Job kind
+ `fetch-windows`: platform-queued, drainable, replayable, `needsText` (`common/jobs/jobKinds.ts:318-326`).
+- **Pacing:** at least `CLIP_WINDOW_MIN_GAP_SECONDS` = 30 between two network fetches
+ (`common/controller/fetchWindows.ts:82`), Rumble 120 (`CLIP_WINDOW_PLATFORM_MIN_GAP_SECONDS`, `:90-91`). The
+ gap ACROSS runs is keyed `clip-window:<platform>` (`:387`) in `common/jobs/platformGap.ts`, which is
+ process-wide and IN MEMORY (`globalThis.__yttPlatformGap__`, `platformGap.ts:13-20`): an editor restart
+ forgets it.
+- **Cache:** a window is `data/<id>/clips/<from>-<to>.mp4`; `findContainingClipWindow`
+ (`common/lib/clipWindow-server.ts:79`) answers with the tightest held window that covers the ask, checked
+ before any pacing (`fetchWindows.ts:345`) and costing no pause.
+- **Dedupe is within ONE request only** (`dedupeFetchWindowsItems`, `fetchWindows.ts:213-218`): nothing looks
+ for an existing job, so two requests for one window make two jobs (release 19 A5).
+- **Rumble gets `-extension_picky 0` on the first try** (`common/ytdlp/fetchWindowManaged.ts:261`): every
+ attempt reloads the Rumble page, and Cloudflare refuses a share of loads. Elsewhere it is a retry on
+ `HLS_EXTENSION_REFUSED`, and a picky run that finds no HLS retries without it (`:65-67`, `:288-296`).
+- **The /jobs row** says who asked, for what, how many: the first log line
+ ``Fetch windows: N window(s) for <requestedBy>[ · <manifest>]`` (`fetchWindows.ts:269`); progress metric
+ `clips`.
+
+### Channel create, rename and delete over ops
+
+- `create-channel`, `rename-channel`, `delete-channel` and `channel-config`'s `sites` are the New channel form
+ and the channel page's Danger zone over HTTP; there is no separate sites route.
+- **A social URL makes a social channel**: `parseChannelForm` infers `sourceKind: "social"` from the platform
+ (X, Bluesky, XenForo; `isSocialPlatform`, `common/lib/platform.ts:42`) and refuses one with no derivable
+ handle (`editor/app/channels/components/parseChannelForm.ts:85-107`).
+- **Confirmation:** delete needs `confirm` = the slug, rename the current slug
+ (`editor/app/channels/actions.ts:635`, `:695`). Both are refused while `channelMediaBusyReason` names a job or a
+ lane unit on the channel (`:644`, `:723`; `editor/app/channels/lib/mediaBusy.ts:48`), and delete while
+ `.relocating.json` exists (`common/controller/channels.ts:581-592`).
+- A social channel's page now carries the same Danger zone (`data-social-danger`,
+ `editor/app/channels/[slug]/page.tsx:214`).
+
+### umtool's operator notes
+
+- **Where:** an article's notes are `SITES_DIR/<site>/reports/<report>/notes.json`, beside `report.json`; a
+ report-video project's are `<project>/notes.json` beside its manifest (`umtool/lib/annotations/targets.mjs:1-12`).
+ The article file is the ONE corpus file umtool writes, only through `isCorpusNotesFile`
+ (`umtool/lib/paths.mjs:252`: exactly `<site>/reports/<report>/notes.json`, realpath-checked).
+ `SITES_DIR` = `$SITES_DIR`, else `$TRANSCRIPTS_DIR/sites`, else `<repo>/transcripts/sites` (`paths.mjs:217-223`).
+- **Shape:** `{format: "umtool-notes", version: 1, subject, source?, notes}` (`umtool/lib/annotations/shape.mjs:13-14`);
+ statuses `open|resolved|wontfix`, authors `operator|agent` (`:16-17`); anchors
+ `text|cite|section|whole|moment|entry|take|edit` (`:18`) — a `text` anchor is a quote with prefix/suffix,
+ re-located on read, never an offset (`umtool/lib/annotations/anchor.mjs`).
+- **One writer, `writeOp`** (`umtool/lib/annotations/store.mjs:139`): a `<file>.lock` taken exclusively (stale
+ after `LOCK_STALE_MS` 30 s, `:28`), an mtime+size token (`notesToken`, `:50`) — a stale token is `StaleNotes`
+ (409, `:32`), an unparseable file is never overwritten (`NotesUnreadable`, `:41`) — then tmp + rename (`:152`);
+ the last note deleted removes the file.
+- **Who:** `/api/notes` stamps every write `operator` (`umtool/app/api/notes/route.ts:46`); `umtool notes` stamps
+ `agent` (`umtool/lib/annotations/cli.mjs:67`, `:80`). The agent digest (`digest`,
+ `umtool/lib/annotations/digest.mjs:160`) is what both `umtool notes <target>` and
+ `GET /api/notes/context` print.
+- **Never published:** compose and the report history read named files only; the guard is the test
+ `common/publish/composeReports.test.ts:794` (a sentinel in `notes.json` appears in no built file and no
+ history revision).
+
+### umtool kinds declare capabilities; nothing branches on a kind id
+
+- `PROJECT_KINDS` (`umtool/lib/projects/kinds.mjs`): `report-video` declares `notes: true` and
+ `linksArticles: true` (`:109`, `:112`); `song` and `sweep-report` neither. Callers ask
+ `kindTakesNotes` / `kindLinksArticles` (`:192`, `:195`) — `umtool/lib/annotations/targets.mjs:80`, `:164`;
+ `umtool/lib/articles/links.mjs:47` — and an e2e spec fails on a kind id written as a literal outside
+ `lib/projects/` and `components/projects/`.
+- **/sites** lists every site's articles (`umtool/lib/articles/sites.ts`): published ids unioned with draft
+ report dirs, each with its open notes, its source and its linked video. An article row's `kind` is the REPORT's
+ kind (`factcheck|sweep`), not a project kind.
diff --git a/plans/release-19.md b/plans/release-19.md
@@ -43,6 +43,7 @@ second editor against the real corpus; the homepage build gate is `build:nodata`
|---|---|---|---|
| **A1** `pnpm ops` finds its token | `[unit]` | read `WORKER_TOKEN` / `ARCHILYZER_EDITOR_URL` from `editor/.env` when unset; `--wait` polls a real authed route instead of the `/api/jobs/active` rewrite | Wave 0 |
| **A2** jobs over ops | `[unit]` + `[spec ops-api]` | `get jobs [--active\|--failed\|--kind\|--slug]`, `get job <id> [--tail]`, `job cancel\|retry\|retry-failed\|drain\|promote\|force-release <id…>`, `--wait` on many ids with queue position; wraps `editor/app/jobs/actions.ts` | A1 |
+| **A2b** ops routes to unit tests | `[unit]` | the `ops-api.spec` route cases move to `route.test.ts` beside each route (moved from B6 on 2026-10-09: Track A owns the ops routes) | A2 |
| **A3** read side | `[unit]` | `get settings\|storage\|sites\|workers\|auto-queue\|scheduler\|cleanup <slug>` over the existing view builders/readers; `/api/auto-queue/control` gated behind `opsAuth` (unauthenticated today) | A2 |
| **A4** archival writes | `[unit]` | `settings` patch (through `saveSettings` + the schema); `lane` start/stop/drain for every lane incl. `publish`; `clear-platform-hold`; `workers enable\|disable`; per-video `transcribe-one\|delete-file\|do-not-clean`; cleanup buckets; `relocate {dryRun}`; `archilyzer storage report` (tierable bytes per channel) | A3 |
| **A5** fetch queue hygiene | `[unit]` | the editor dedupes identical fetch-window requests (same slug/id/window → the existing job); `fetch-via-editor.mjs` waits without a timeout, printing queue position; small fetch-window jobs get their own queue key / priority instead of waiting behind a multi-hour `persist-videos` | A4 |
@@ -57,11 +58,11 @@ second editor against the real corpus; the homepage build gate is `build:nodata`
| slice | class | what | after |
|---|---|---|---|
| **B1** heavy-work gate | `[unit]` | `scripts/queue-lock.mjs` generalized into `pnpm heavy -- <cmd>`: one heavy slot machine-wide + a ≥ 6 GB free-memory floor; used by e2e, `next build` (publish stages, `build-site.sh`), `build-video.mjs` renders; a render may hold the transcription lane for its duration (ops `lane`); documented in WORKTREES.md + AGENTS.md | — |
-| **B2** report-to-video robustness | `[unit]` | prune `*-frames` after the final mux (`--keep-frames` keeps them); `--chrome-only` builds missing segments; a manifest lint before render (teaser > 34 chars, image src relative to the manifest, a QR legible at 720p). `umtool/report-to-video/` is the Candace session's: coordinated with it | B1 |
+| **B2** report-to-video robustness | `[unit]` | prune `*-frames` after the final mux (`--keep-frames` keeps them); `--chrome-only` builds missing segments; a manifest lint before render (teaser > 34 chars, image src relative to the manifest, a QR legible at 720p). Track B's, after B5; the Candace session reviews it before merge (ruled 2026-10-09) | B5 |
| **B3** report CLI | `[unit]` | `archilyzer reports check <site> [--reports …]` (compose without a build); `reports verify-quotes <report.json>` (quote vs cue span, the en-orig check); `reports attach-video` encoding to fit the 24 MiB compose limit | — |
| **B4** ops transcribe | `[unit]` | `durationMs` excludes queue wait; a priority for short one-file jobs over long auto jobs | Wave 0 |
| **B5** umtool small debts | `[unit]` → `[spec umtool]` | per-worktree umtool e2e ports (`portFor`); `mix.spec` order dependence; `umtool window` writes edit notes; a selection spanning two blocks gets a Note button; the CLI finds `SITES_DIR` from the repo, not the cwd; umtool's /sites heading becomes "Articles" (naming hazard vs the editor's /sites). ONE umtool e2e run | B1 |
-| **B6** test economy | `[unit]` | pure/API e2e moves to unit: `audio-check-classifier.spec` → common; `ops-api.spec` routes → `route.test.ts`; `view-route`, `worker-unit`, `media-file-abort`. Root `pnpm test` and `pnpm typecheck`; the editor gets an eslint config or loses its broken `lint` script | — |
+| **B6** test economy | `[unit]` | pure/API e2e moves to unit: `audio-check-classifier.spec` → common; `view-route`, `worker-unit`, `media-file-abort`. Root `pnpm test` and `pnpm typecheck`; the editor gets an eslint config or loses its broken `lint` script | — |
### Track C — docs and plans (owner: the Track C implementer; no e2e)
@@ -77,17 +78,17 @@ second editor against the real corpus; the homepage build gate is `build:nodata`
| track | owns |
|---|---|
-| A | `editor/app/api/ops/**` (except B4's transcribe route), `scripts/archilyzer-ops.mjs` + its test, `mcp/src/**`, `editor/app/jobs/actions.ts`, `common/controller/fetchWindows.ts`, `fetch-via-editor.mjs` |
-| B | `scripts/queue-lock.mjs` and the heavy gate, `umtool/**` (except `umtool/report-to-video/*`, the Candace session's), the e2e specs B6 moves, the report CLI, `common/controller/transcribeFile.ts`, `editor/app/api/ops/transcribe/**` |
+| A | `editor/app/api/ops/**` (except B4's transcribe route), `scripts/archilyzer-ops.mjs` + its test, `mcp/src/**`, `editor/app/jobs/actions.ts`, `common/controller/fetchWindows.ts`, `fetch-via-editor.mjs`, the `ops-api.spec` cases A2b moves |
+| B | `scripts/queue-lock.mjs` and the heavy gate, `umtool/**` (`umtool/report-to-video/*` for B2 only, the Candace session reviewing before merge), the e2e specs B6 moves, the report CLI, `common/controller/transcribeFile.ts`, `editor/app/api/ops/transcribe/**` |
| C | `plans/**` (except the records others append), root `*.md` docs, `homepage/content/docs/**`, `mcp/README.md`, the `archilyzer docs cli` generator |
| shared, append-only | `editor/CHANGELOG.md`, `AGENTS.md`, `common/bin/_cli.ts` (rows only) |
### Graph and merge order
```
-Wave 0 (release 18 lands) ──► A1 ──► A2 ──► … ──► A9 ──► release-end suite (Track A is sequential: A1–A5, then A6–A9)
+Wave 0 (release 18 lands) ──► A1 ──► A2 ──► A2b ──► … ──► A9 ──► release-end suite (Track A is sequential: A1–A5, then A6–A9)
└─► B4
-now: B1 ──► {B2, B5}; B3; B6 (B6 touches no release-18 file)
+now: B1 ──► B5 ──► B2 (Candace session review); B3; B6 (B6 touches no release-18 file)
now: C1, C3 ──► C2 ──► C4, C5; C2 regenerated after A's actions land
```
@@ -95,7 +96,7 @@ now: C1, C3 ──► C2 ──► C4, C5; C2 regenerated after A's actions la
18). C1, C3 and B6 touch no release-18 file and may merge first. Track A merges slice by slice in its own order;
B and C merge whenever a slice is green. The parent regenerates the command reference (C2) after each Track A merge
that adds an ops action or CLI row.
-- **Do not touch:** `umtool/report-to-video/*` without the Candace session; `transcripts/**` except through the
+- **Do not touch:** `umtool/report-to-video/*` except B2, which the Candace session reviews before merge; `transcripts/**` except through the
writers; the working sessions' `~/reports/*` workspaces.
## Verification
