commit 087ddf9e6a7a3bd219f32ebc409be27b97359fc9
parent f851f8f4bbdf25e404470ace01ad6e2d4b20f7c9
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Sat, 26 Sep 2026 17:08:08 -0400
plans: release 10 slice N as shipped — forceMedia (persist-only with a transcript on disk) and metadata.history.json; gates (common 1,984, editor unit 85, test:scripts 173 + 1, mcp 269, editor build, e2e 78/78 in 8.7 min, bite run 3/3 new tests fail on the reverted fix); two [Unreleased] bullets; FACTS: slice M's silent no-op is fixed, and yt-dlp re-writes an existing subtitle file by default
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
3 files changed, 176 insertions(+), 5 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -7,6 +7,8 @@
- **`/jobs` says why a job was cancelled at boot.** A job the boot settled shows its reason under its status on `/jobs` and as *Cancelled because* on its own page: for example "server restarted; the scheduler re-derives syncs" or "superseded by a newer queued job (…)". The reason used to be only in the job's log.
- **The server log says how often a queued job skips its page refresh.** When a queued job finishes outside any request, the editor skips its page refresh and notes it in the log. The note used to appear once and never again. Now the first one after a quiet spell is logged at once, any more in the next 10 minutes are counted, and one line at the end gives the count, with a running total.
- **An agent working through the MCP server asks the editor for a clip instead of running yt-dlp.** The MCP server has a new tool, `fetch_clip`. Given a citation's channel, video id, start and end and a one-line reason, it asks the local editor for that window through `POST /api/media/fetch-window`: the same paced, cookie-aware job umtool uses, which records who asked and why beside the file. It answers with the file's path in the corpus (`channels/<slug>/data/<id>/clips/`). The window is the cited span with 3 seconds either side, at most 15 minutes. `full: true` asks for the whole recording instead, which lands in the saved-video store and needs a video the editor already knows. A Rumble citation's id (the embed id the archive publishes) is mapped to the id the editor names the video's folder by, through the archive record's link. The editor must already archive the channel: pointed at a public site with a fresh editor, every clip gets a 404 `Channel "<slug>" not found`. The tool waits up to 90 seconds by default (at most 300) and otherwise returns the job's id, to wait on with `job`; the fetch carries on in the editor either way. While it waits it sends a progress notification per poll to a client that asks for progress. A client whose requests time out at 60 seconds (the MCP SDK's default) must raise that or pass `wait_seconds` of 50 or less. No request to the editor waits more than 15 seconds. If the editor stops answering mid-fetch, the answer gives the job's id and says not to ask again from scratch. The `/ask` and `/sweep` plans now tell the agent to use it and never to run yt-dlp itself. The MCP needs `ARCHILYZER_EDITOR_URL` and `WORKER_TOKEN` (the editor's own) in its environment, so re-register it with the two `--env` lines in the README; without them the tool says so and fetches nothing. The MCP server itself still writes nothing. The README's `yt-dlp --download-sections` command is now only the fallback for a machine with no editor.
+- **"Persist source video" and a whole-recording fetch download the video even when it already has a transcript.** On a channel that takes YouTube's subtitles, the button — and a whole-recording request from umtool or the MCP server's `fetch_clip` with `full: true`, which run the same job — fetched only the subtitles again when the video already had a transcript or captions, and finished with no file. It now downloads the source and moves it into the saved-video store. A transcript already on disk is kept: no audio file is extracted beside it and nothing is transcribed. **Persist kept now** on a channel's Cleanup stage does the same for every kept video, so on such a channel it now downloads each kept video's source. The video page's Source video card offers the button on these channels too; it used to say persistence was for transcribe-handling channels only.
+- **Each video keeps a history of how its metadata changed at the source.** Every download that rewrites a video's `metadata.info.json` and changes anything in it adds one entry to `metadata.history.json` beside it: the old and new value of each field that changed (title, description, duration, availability, chapters and the rest), the view, like and comment counts that moved, and which of the fields that change on every fetch (format URLs, thumbnails, caption URLs) differed, compared by fingerprint only. A caption language appearing or disappearing counts as a change. The newest 200 entries are kept. The video page shows the history under the description: "Metadata rewritten N× · last … by …: <what changed>", with each entry's old → new values when opened. The history starts with the first rewrite after this update.
## [0.9.0] - 2026-09-26
- **Every page now has a ground and an accent to choose, and the five theme families are gone.** The theme menu (the palette button beside the quick toggle, in the editor's sidebar and in the header of every published site, the hub and the homepage) has two groups. **Base** is System, Light, Sepia or Dark; Sepia is new, a warm paper ground for long reading. **Accent** is Signal, Brass, Vermilion, Violet, Sakura, Blue or Green, with the site's own tagged *default*; a site with a custom hex offers it first as *Site colour*. The quick toggle cycles System → Light → Sepia → Dark. A published site opens on the reader's system setting, in the accent its site form sets. The hub and the homepage open on Dark, in Signal, even with JavaScript off, and the editor follows the system, in Signal. Each accent has a value for each ground that reads at 4.5:1, and a custom hex is darkened or lightened per ground to match. A reader's accent is remembered only while it differs from the site's: picking the site's own again forgets it, so the reader follows the site if its accent changes later. Base, Archive, Selenized, Swiss and Archilyzer are gone. A choice made before this update carries over once: light stays light (Archive light becomes Sepia), dark stays dark and system stays system; the family itself is dropped. Headings are Archivo, text is IBM Plex Sans and figures are IBM Plex Mono everywhere, with one corner radius. Success, warning and other status text reads at 4.5:1 on its own tinted fill on every ground; on Light, success and warning are a shade deeper than before for it. Chart colours are fixed per ground and never follow the accent; the third is a violet, well clear of the red that marks a recording as gone. The phone's browser bar takes the page's ground, not the accent. Needs a rebuild and deploy of every site, the hub and the homepage.
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -6757,11 +6757,22 @@ S4, as shipped"; `release-10.md` "Slice L2 / L1, as shipped". Every anchor below
the editor's `fetchFullSourceAction` uses the saved-video store (`transcripts/saved-videos/<slug>/
<id>/source-media.<ext>`, pointer `data/<id>/saved-video.json` with `keepReason: "override"` and
`origin`), job kind `redownload-archive`, and takes NO URL (404 when the video has neither
- metadata nor a playlist entry). It is a silent no-op on a youtube-handling video that already has
- a transcript (`downloadOneManaged.ts:1233-1235`: the keep-source override reaches only transcribe
- handling or the no-captions fallback) and it rewrites `metadata.info.json`; the window path never
- writes metadata. Neither path dedupes a running job: a repeated request while one runs queues a
- second fetch, which is why every text says "resume with job".
+ metadata nor a playlist entry). It WAS a silent no-op on a youtube-handling video that already had
+ a transcript (the keep-source override reached only transcribe handling or the no-captions
+ fallback) — **fixed by release 10 slice N**: `archiveSourceVideo` and `persistKept` pass
+ `forceMedia: true`, so the no-subs fallback (attempt 3, `downloadOneManaged.ts` `forced` /
+ `keepTranscript`) runs whenever the subtitle pass succeeded; with a transcript on disk it adds
+ `--no-write-subs --no-write-auto-subs` after the channel's args and finalizes persist-only
+ (`finalizeAppExtraction` `extractAudio: false`: no `audio.<fmt>`, no whisper, no short-audio
+ probe, `fellBackToTranscribe` unset). The primary subtitle pass still runs as on any re-download,
+ and yt-dlp re-writes an existing subtitle file by default (`YoutubeDL.existing_file`,
+ `default_overwrite=True`, with `overwrites` popped when unset — verified in the installed
+ `yt-dlp-patched` 2026.08.19), so a `transcript.<lang>.vtt` gets a new mtime; a whisper
+ `transcript.json` is untouched. It rewrites `metadata.info.json` (every managed download's
+ prefetch does); since slice N each rewrite that changes the bytes appends to
+ `metadata.history.json` (`common/lib/metadataHistory.ts`). The window path never writes metadata.
+ Neither path dedupes a running job: a repeated request while one runs queues a second fetch, which
+ is why every text says "resume with job".
- The editor must already HAVE the channel (404 `Channel "<slug>" not found` otherwise); a
public-only setup gets the "no editor configured" error naming both env vars, and the README's
`yt-dlp --download-sections` is the no-editor fallback only.
diff --git a/plans/release-10.md b/plans/release-10.md
@@ -1166,6 +1166,164 @@ nit N1 taken).
**Gates** (`m-gate4.log`): tsc clean (whole workspace); **mcp 269/269**, 16 s.
+### Slice N, as shipped — the whole-recording fetch downloads anyway; metadata history (2026-09-26)
+
+Branch `editor/full-fetch-media` off `main` `555bc454`, worktree
+`/home/user/Projects/editor-full-fetch-media`, one Opus implementer. The plan is
+[`full-fetch-forces-media.md`](full-fetch-forces-media.md) (Decisions 1–6, Implementation 1–8). It
+answers the operator's ask of 2026-09-26: the whole-recording fetch downloads whatever is on disk,
+and every rewrite of the metadata is kept so changes can be detected. No settings, site or channel
+key; no new job kind; no new directory.
+
+**`forceMedia`** (`common/ytdlp/downloadOneManaged.ts`).
+- One option on `ManagedDownloadOpts` (:154): the caller wants the media; a transcript or captions
+ on disk are not a reason to skip. `archiveSourceVideo` (`videoActions.ts:310`, so both "Persist
+ source video" and `fetchFullSourceAction`) and `persistKept` (`persistKept.ts:137`) set it. The
+ download lane, re-acquire and import do not.
+- **Attempt 3** (:1285-): `forced = !noSubsFallback && forceMedia` (:1304) — a video with no
+ transcript and no captions takes today's path whoever asked. The pass then runs when the primary
+ succeeded, and logs one line first: `forceMedia: downloading the source although a transcript is
+ on disk` (or `… although captions exist`).
+- **`keepTranscript = forced && hasTranscript`** (:1314) makes it persist-only:
+ - `--no-write-subs --no-write-auto-subs` after the channel's own args (:1368), so a channel whose
+ `ytdlpExtraArgs` carry `--write-auto-subs` cannot overwrite the transcript (the chat-only
+ pass's last-occurrence rule);
+ - `finalizeAppExtraction` gains `extractAudio?: boolean` (:253); `false` moves the container into
+ the store and extracts nothing, logging `Persist only: a transcript is on disk, so no
+ audio.<fmt> is extracted from <source>.`;
+ - no inline whisper, no short-audio probe (it would probe audio the pass never made);
+ `fellBackToTranscribe` stays unset and the status stays the subtitle pass's (`ok`); the attempt
+ is still recorded as `no-subs-fallback`, n: 3.
+- With `forceMedia` and NO transcript (captions listed, none fetched), the pass is today's fallback
+ in full: download, extract, transcribe inline if configured, persist.
+- A caller that does not set `forceMedia` gets byte-for-byte today's behaviour: the same gate, log,
+ args and finalize. Every existing e2e arg-ordering assertion passed unchanged.
+
+**Metadata history.**
+- `common/lib/metadataHistory.ts` (pure; `node:crypto` only): the key sets exactly as the plan
+ lists them; `normalizeForDiff`, `diffMetadata`, `buildMetadataHistoryEntry` (null when the sha is
+ unchanged), `appendMetadataHistoryEntry` (cap 200, oldest dropped), the 16 KB guard
+ (`guardStoredValue` → `{ sha256, bytes }`, `isValueDigest`), `coerceMetadataHistory` (drops a torn
+ entry instead of failing the file). Volatile values are compared on canonical (key-sorted) JSON.
+- `common/lib/metadataHistory-server.ts`: `sidecar("metadata.history.json", …)` (indented, trailing
+ newline); `snapshotMetadata`; `recordMetadataRewrite`, a read-modify-write under
+ `withJsonFileLock`; and `withMetadataHistory(videoDir, {by, requestedBy, onLog}, run)`. That
+ snapshots before `run()` and records in a `finally`, so a failed run is recorded too. It returns
+ or rethrows `run`'s own result, and logs and swallows a failure to record.
+- Wrapped, with `requestedBy` = `persistOrigin.requestedBy` where there is one:
+ - the prefetch spawn and its auth retry (`by: "prefetch"`, :700);
+ - the audio-check primary when the canonical id is known (`by: "audio-check"`, :1088);
+ - `downloadOneAudio` when its output dir is pinned (`by: "download-one"`,
+ `runYtdlp.ts:1395`).
+ `dataDirIdForUrl` (`runYtdlp.ts:321`) is the id test `outputArgsForUrl` already made, now named.
+- **Not on `discardPrefetchDir`'s allow-list**, said at `PREFETCH_OWN_FILES` (:407) and in the
+ server module's head.
+- `SIDECAR_FILENAMES`' enumeration test: ten names.
+
+**The surface.**
+- `common/views/metadataHistoryView.ts` (pure, the clock as `nowMs`): `metadataHistoryView(history,
+ nowMs)` → `{count, line, entries}` newest first, or null.
+ - The line is `Metadata rewritten N× · last <5m ago> by <by>: <summary>`.
+ - Values are cut at 300 characters, with the full text kept for `title=`.
+ - `atLabel` is fixed UTC.
+- `page.tsx` reads the sidecar once beside `availability.json`. The new server component
+ `components/MetadataHistoryDetails.tsx` draws it in the page header, under the Description:
+ a `<details>` whose summary is the line, with each entry's keys as from → to and the volatile
+ keys named. No client state and no control.
+
+**Where the code departs from, or fills in, the plan** (for the reviewer):
+- **The "Persist source video" button had to be un-gated.** `SourceVideoSection` returned
+ "Source-video persistence applies to transcribe-handling channels only." for any other handling,
+ so the plan's e2e 6(a) — click the button on a youtube-handling video — had no button to click.
+ - The gate is gone.
+ - A youtube channel gets its own description line: "…A transcript already on disk is kept, and
+ no audio is extracted beside it."
+ - Labels, aria-labels and test ids are unchanged.
+ - `videoChoreCards.ts`'s `shown` text is updated.
+- **`keepTranscript` with a plan that persists nothing fetches nothing.** The plan said "do not
+ assume `plan.persist`". With a transcript the pass may extract no audio, so a plan that keeps no
+ container leaves the download with no destination. It logs `forceMedia: a transcript is on disk
+ and this download keeps no source video (<reason>); nothing to fetch.` and skips. Unreachable
+ today: every `forceMedia` caller sets `keepSourceVideoOverride: true`.
+- **The summary names counters when nothing else moved:** `only counters (view_count, …)`.
+ - "nothing meaningful (formats only)" is kept for an all-volatile entry;
+ `unreadable metadata (no diff)` for a torn side.
+ - Real rewrites nearly always move a counter, so without this the line would read "formats only"
+ over a view count that changed.
+- **An unparseable side** records the rewrite (both fingerprints) with an empty diff and
+ `unparseable: ["from"|"to"]`, rather than diffing against `{}` (which would list every key).
+- **A counter present on one side only** is `[null, x]` in `counters`, not in added/removed.
+- **The surface is in the header, not in a card.** There is no metadata card. The header is where
+ the page shows the metadata, and a header `<details>` is visible without opening a collapsed
+ card.
+- **README: nothing.** `grep -n "Persist source video" README.md` is empty. Its `full: true` line
+ ("it needs a video the editor already knows") is still true.
+
+| sha | what |
+|---|---|
+| `92ae844c` | `common:` `metadataHistory.ts` + `metadataHistory-server.ts`; `metadataHistory.test.ts` (13), `metadataHistory-server.test.ts` (5); the sidecar enumeration test names ten |
+| `51166736` | `common:` `forceMedia` (attempt-3 gate, the log line, the subs refusals, persist-only finalize); the three history wraps + `dataDirIdForUrl`; `archiveSourceVideo` and `persistKept` pass it; `forceMedia.test.ts` (6) |
+| `1877d3a5` | `editor:` `views/metadataHistoryView.ts` (+ test, 6), `MetadataHistoryDetails.tsx`, the page loader; the Source video card un-gated |
+| `92c26c73` | `e2e:` `persist-youtube-handling.spec.ts` (2), `fetch-window.spec.ts` +1, the fake's `.fake-ytdlp-metadata.json` knob |
+| `6adbd455` | `docs:` AGENTS.md, "Clips and report-to-video" |
+| `31eeece4` | `common:` the `PREFETCH_OWN_FILES` comment (comment only) |
+| _this_ | `plans:` this record; two `[Unreleased]` bullets; the FACTS amendment (slice M's "silent no-op" is fixed, with the VTT-overwrite fact) |
+
+**Gates**, all from the worktree root:
+- **tsc** (`pnpm -r --no-bail --workspace-concurrency=1 exec tsc --noEmit`) clean before every
+ commit (`n-tsc1.log` … `n-tsc5.log`).
+- **common 1,984/1,984** (on `6adbd455`). That is 1,954 + 30: `metadataHistory` 13, `-server` 5, `forceMedia` 6,
+ the view 6.
+- **editor unit 85/85.**
+- **`test:scripts` 173 + 1 skip of 174.**
+- **mcp 269/269**, unchanged (`n-gates1.log`).
+- **`pnpm --filter editor exec next build`** ok: compiled in 17.5 s, 50 s total (`n-build1.log`).
+- **e2e** (queued, detached, `export/public` links in place, no dangling links): **78 passed,
+ 0 failed, 8.7 min** (`n-e2e1.log`), on `92c26c73` (the two commits after it are AGENTS.md and
+ a comment). The specs:
+ - the plan's: `saved-videos`, `fetch-window` and the new `persist-youtube-handling`;
+ - the grep for `downloadOneManaged|Persist source video|persist-kept|redownload`: `import-video`
+ and `incomplete-transcript` (plus `saved-videos`);
+ - the paths this slice touched: `no-subs-fallback` (attempt 3), `video-page` (the header),
+ `title-filter` and `chat-only` (the prefetch wrap beside `discardPrefetchDir`), `skip-live`
+ (the prefetch), `audio-check-scenarios` (the audio-check wrap).
+- **The new tests bite.**
+ - e2e: with `forceMedia: true` removed from `archiveSourceVideo` and `withMetadataHistory` made a
+ pass-through (uncommitted), `persist-youtube-handling.spec.ts fetch-window.spec.ts` gave
+ **6 passed, 3 failed, 1.3 min** (`n-bite1.log`), and the three failures are exactly the new
+ tests:
+ - the full fetch ends `done` with `file` undefined (slice M's live no-op, reproduced);
+ - no `forceMedia:` line;
+ - no `metadata.history.json`.
+ - unit: with the attempt-3 gate forced off, 3 of the 6 `forceMedia.test.ts` fail; with
+ `extractAudio: true`, 1 fails.
+- **Numbers tool: none**, as the plan says. Nothing was run against the live :3001 editor or the
+ real corpus.
+
+**Found and left.**
+- **The primary subtitle pass rewrites a `transcript.<lang>.vtt`.**
+ - This is not new: the decision keeps that pass "as today", and every re-download already does
+ it. yt-dlp deletes and re-fetches an existing subtitle file unless `--no-overwrites` is passed
+ (`YoutubeDL.existing_file`, `default_overwrite=True`; verified in the installed
+ `yt-dlp-patched` 2026.08.19).
+ - So the rollout's check "the transcript's mtime unchanged" on `teamrcn/dbnS-cBgStY` holds for a
+ whisper `transcript.json`, not for a VTT. A VTT's bytes are what YouTube serves now.
+ - What slice N guarantees is that the MEDIA pass writes no transcript. The new e2e asserts it on a
+ `transcript.json`, with inline whisper on.
+- **A title-filter rejection now keeps a directory that has a history** (the plan's allow-list
+ rule). This takes a dir left by an earlier pass that was not rejected (a failed download) and a
+ filter that rejects it now. Real rewrites always differ (`epoch`), so such a dir gets a history
+ and keeps its metadata stub.
+- **Two history entries per audio-checked download** (the prefetch, then the audio-check primary's
+ own re-extraction), as the plan's wraps imply.
+- **No history for an unidentifiable URL** (the `%(id)s` dir is known only afterwards), nor for a
+ rewrite by a pass the plan did not wrap: the subtitle primary and the fallback load the
+ prefetched info json and write none, unless a channel's `ytdlpExtraArgs` carry
+ `--write-info-json`.
+- **"Persist kept now" on a youtube-handling channel now downloads every kept video's source**
+ (the plan's known limitation). `persistKept` still counts a returned-but-failed download as
+ `persisted` (pre-existing).
+
## Rollout
Nothing is rolled out, except that **Jeralyzer is already on the brand, in Signal** (a build-deploy
