commit 945b20ccd5a1c32e3bd71d4a33dd698b85b6421e
parent 002a07520a27f6bea242f8f1e6e222fa801f5879
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Sat, 26 Sep 2026 16:24:34 -0400
plans: slice N — the whole-recording fetch downloads anyway (forceMedia through the no-captions fallback, transcript untouched, persist-only finalize) and every metadata.info.json rewrite appends a normalized diff to metadata.history.json
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
1 file changed, 196 insertions(+), 0 deletions(-)
diff --git a/plans/full-fetch-forces-media.md b/plans/full-fetch-forces-media.md
@@ -0,0 +1,196 @@
+# The whole-recording fetch downloads anyway, and every metadata rewrite keeps the old version
+
+Written 2026-09-26 (evening) in the fetch_clip session; anchors verified on `main` `151934e9` by one
+Explore pass. Rules: `plans/tools/implementer-rules.md` (one Opus implementer in a worktree, one
+read-only Opus review, the parent merges). Release 10 slice **N**; record in `plans/release-10.md`
+before `## Rollout`, shape of "Slice M, as shipped".
+
+## The operator's ask (verbatim, 2026-09-26)
+
+"Change the editor to download anyway despite having metadata and/or subs, this specific call implies
+that doesn't matter. Save old versions of metadata, maybe as diffs, so we can detect changes."
+
+"This specific call" = the whole-recording fetch (`fetchFullSourceAction`, reached by the MCP's
+`fetch_clip` with `full: true` and umtool's `--full`). Slice M's live proof found it a silent no-op on
+`teamrcn/dbnS-cBgStY` (`handling: "youtube"`, transcript on disk): two yt-dlp passes, both
+`--skip-download`, 7 s, no file — and `metadata.info.json` rewritten.
+
+## Root cause (verified)
+
+`common/ytdlp/downloadOneManaged.ts`:
+- The primary attempt for `handling: "youtube"` is `youtubeHandlingArgs` (:159-169): subs only,
+ **always** `--skip-download -t sleep`. `keepSourceVideoOverride` never reaches it.
+- The only branch that downloads real media for youtube handling is attempt 3, the no-subs fallback
+ (:1225-1270), gated `!hasAnyTranscriptOnDisk(videoDir) && metadataReportsNoCaptions(videoDir)`
+ (:1233-1235). With a transcript on disk it is skipped. When it does run it switches
+ `channelConfig` to `handling: "transcribe"` (:1240-1242), uses `transcribeMediaArgs(url,
+ fallbackConfig, plan, fmt, /*reuseInfoJson*/ true, selector)` (:1256-1264) with `--load-info-json`
+ (:1269-1271), and on success in app mode calls `finalizeAppExtraction` (:1289), which is what
+ calls `persistSourceVideo` (:277) when `plan.persist`.
+- `resolvePersistenceDecision` (`common/ytdlp/persistencePlan.ts:58-91`): `keepSourceVideoOverride:
+ true` → `persist: true, extractionMode: "app"` unconditionally. The intent is right; it is unreachable.
+- Two callers pass that override: `archiveSourceVideo`
+ (`editor/app/channels/[slug]/videos/[id]/videoActions.ts:254-320`, reached from
+ `redownloadToArchiveAction` :241-247 = the **"Persist source video"** button in
+ `components/cards/SourceVideoSection.tsx:73-86`, and from `fetchFullSourceAction` ~:1094) and the
+ bulk pass `common/controller/persistKept.ts:119-135` ("Persist kept now"). Same bug in both.
+- The metadata prefetch (:643-751, `buildPrefetchArgs` :650-659) runs on EVERY attempt when the
+ canonical id is known and rewrites `metadata.info.json` with no existence check
+ (`backfillReacquire.ts:33-36` says so in words). Other writers: the audio-check primary
+ (`transcribeHandlingArgsForAudioCheck` :315, always writes) and the legacy single-video job
+ `downloadOneAudio` (`common/ytdlp/runYtdlp.ts:1365`). The window fetch (`fetchWindowManaged.ts:217`),
+ `downloadSubsForUrl` (:1198) and the live-chat pass (:466) pass `--no-write-info-json`. The
+ metadata scan never writes it (`metadataScanStore.ts:4-21`).
+- No history/diff of anything per video exists; the precedent is `availability.json`'s `history[]`
+ (`common/lib/availability-server.ts:38-66`, `sidecar(AVAILABILITY_FILENAME, sidecarField(...))`).
+- A real info-json (`teamrcn/dbnS-cBgStY`, 52,231 bytes, 61 keys): `formats` = 43,496 bytes (83 %,
+ signed URLs that differ on every fetch), `thumbnails` 4,923; `epoch`, `_version` and the counters
+ (`view_count`, `like_count`, `comment_count`, `channel_follower_count`) drift every fetch.
+- `sidecar()` (`common/lib/sidecar-server.ts:69-96`) declares a bare filename only (no
+ subdirectories; refuses `transcript.<x>.<y>`). A new DIRECTORY under `data/<id>/` would need the
+ `clips/` treatment in `channelSnapshot.ts:843-870`, `storageLocations.ts:100-101,177`,
+ `views/storage.ts:91` and `reconcileVideoDirs.ts:84-96` — so the history is ONE FILE.
+- Tests: `downloadOneManaged.test.ts` covers only the prefetch allow-list; e2e drives the real
+ function through `editor/e2e/fixtures/bin/fake-ytdlp.mjs` (prefetch branch :783-789).
+ `editor/e2e/saved-videos.spec.ts` ("Persist source video", fixture
+ `one-transcribe-channel-with-audio`, slug `test-transcribe`, :106-169) and
+ `editor/e2e/fetch-window.spec.ts` (window + full, :225-265 asserts the info-json args, :386 the
+ cached full 200) — neither uses a `handling: "youtube"` fixture, so the bug has no coverage.
+ Youtube-handling fixtures exist: `editor/e2e/fixtures/test-transcripts/one-youtube-channel*`,
+ `youtube-with-playlist`.
+
+## Decisions (made; do not re-open)
+
+1. **One option, `forceMedia: true`, on `ManagedDownloadOpts`** (`downloadOneManaged.ts:98-151`).
+ Meaning: "the caller wants the media; a transcript or captions on disk are not a reason to skip".
+ Set by `archiveSourceVideo` (both callers) AND by `persistKept.ts:119-135` — "persist kept" means
+ persist, and the operator sees a no-op there too. Not set by the download lane, re-acquire or
+ import (their semantics are unchanged).
+2. **Mechanics**: for `handling: "youtube"` with `forceMedia`, attempt 3 runs whenever the primary
+ succeeded (`lastSucceeded`), regardless of `hasTranscript`/`noCaptions`; it logs ONE line first:
+ `forceMedia: downloading the source although a transcript is on disk` (or `… although captions
+ exist`). The primary subs pass still runs as today (it is what every re-download does). The
+ transcript on disk is NEVER touched: with `forceMedia` and `hasTranscript`, the fallback's
+ transcript-producing steps (inline transcription on fallback, any subs → transcript normalize
+ the fallback triggers) are skipped, and `finalizeAppExtraction` runs in a **persist-only** mode
+ (move the container into the saved-video store; no `audio.<fmt>` extraction — slice M's proof
+ left an `audio.mp3` beside an existing transcript, which is waste). With `forceMedia` and NO
+ transcript, the fallback behaves exactly as today (download, extract, transcribe inline if
+ configured, persist). Transcribe-handling channels: no change (they already download).
+ `plan.persist` is true on every `forceMedia` caller today; do not assume it — read the plan.
+3. **Metadata history is one sidecar per video, `metadata.history.json`**, declared with
+ `sidecar()` in a new `common/lib/metadataHistory-server.ts` (types + pure diff in
+ `common/lib/metadataHistory.ts`, testable without fs). Shape:
+ `{ entries: MetadataHistoryEntry[] }`, newest LAST, capped at 200 (drop oldest). An entry:
+ ```
+ { at: ISO, by: "prefetch" | "audio-check" | "download-one", requestedBy?: string,
+ from: { sha256, bytes, at?: ISO (the old file's mtime) }, to: { sha256, bytes },
+ changed: { [key]: { from, to } }, // content keys whose normalized value differs
+ added: { [key]: value }, removed: { [key]: value },
+ counters: { [key]: [from, to] }, // view_count etc., only those that moved
+ volatile: string[] } // fingerprinted keys that differed (e.g. "formats")
+ ```
+ **Normalization**: VOLATILE keys are compared by fingerprint (`sha256` of their JSON) and never
+ stored: `formats, requested_formats, requested_downloads, thumbnails, thumbnail, heatmap,
+ automatic_captions, subtitles, epoch, _version, _format_sort_fields, url, http_headers,
+ format, format_id, format_note, filesize, filesize_approx, tbr, abr, vbr, asr, acodec, vcodec,
+ fps, width, height, resolution, aspect_ratio, dynamic_range, protocol, ext, audio_channels,
+ container, downloader_options, quality, source_preference, has_drm, language_preference`.
+ For `automatic_captions`/`subtitles` ALSO record the language-key lists as a content key
+ (`subtitles_langs`, `automatic_captions_langs`) so a caption appearing IS detected. COUNTERS:
+ `view_count, like_count, dislike_count, comment_count, channel_follower_count, average_rating,
+ repost_count`. Everything else is CONTENT (title, fulltitle, description, duration,
+ duration_string, upload_date, timestamp, release_timestamp, availability, live_status, is_live,
+ was_live, age_limit, categories, tags, chapters, uploader*, channel*, playable_in_embed,
+ media_type, webpage_url, …) — diffed by deep-equal on the raw value, stored whole (a
+ description's from/to is exactly what the operator wants to read). Any single stored value
+ above 16 KB is replaced by `{ sha256, bytes }`.
+ **Append ALWAYS** on a rewrite (the operator wants to know it was rewritten even when only the
+ formats moved — an all-volatile entry is ~250 bytes), bounded by the cap; the entry is written
+ atomically through the sidecar. When the new file is byte-identical to the old (same sha), write
+ nothing. When there was no old file, write nothing (a first write is not a rewrite).
+4. **Interception**: `withMetadataHistory(videoDir, { by, requestedBy }, run)` in the server module:
+ snapshot the existing file (bytes + sha + mtime) before `run()`, then after `run()` (success OR
+ failure — the prefetch rewrites on failure too) compare and append. Wrap the three writers:
+ the prefetch spawn (`downloadOneManaged.ts:658-662`), the audio-check primary attempt, and
+ `downloadOneAudio` (`runYtdlp.ts:1365`). `requestedBy` = `opts.persistOrigin?.requestedBy` when
+ present. Nothing else changes: the file yt-dlp writes is still the file.
+5. **Surface** (small, read-only): the video page's source/metadata card gets one line when the
+ history has entries — `Metadata rewritten N× · last <relative time> by <by>: <changed keys or
+ "nothing meaningful (formats only)">` — and a `<details>` listing the entries newest first with
+ the changed keys and, per key, from → to (strings truncated at 300 chars, full text in `title=`).
+ Build it as a pure view in `common/views/` (`metadataHistoryView.ts`) fed by the loaded sidecar,
+ in the style of the other views there; no new route, no client state.
+6. No new job kind, no new directory, no eviction: the file is bounded and lives with the video.
+
+## Implementation (one Opus slice, branch `editor/full-fetch-media`, worktree
+`/home/user/Projects/editor-full-fetch-media`, ports editor 3601 / test 3611 / export 3610)
+
+1. `common/lib/metadataHistory.ts` (pure): the key sets, `normalizeForDiff`, `diffMetadata(old,
+ new) → { changed, added, removed, counters, volatile }`, `appendEntry(history, entry, cap=200)`,
+ the 16 KB value guard. Tests: `metadataHistory.test.ts` — formats-only change → all-volatile
+ entry with empty `changed`; a title change → `changed.title {from,to}`; a new caption language →
+ `changed.subtitles_langs`; counters → `counters` only; cap at 200 drops the oldest; the 16 KB
+ guard; byte-identical → `null` (no entry).
+2. `common/lib/metadataHistory-server.ts`: `sidecar("metadata.history.json", …)` (check
+ `SIDECAR_FILENAMES`' enumeration test and `PREFETCH_OWN_FILES` in `discardPrefetchDir`: a dir
+ holding this file is never prefetch-only, so it must NOT be in the allow-list — say so in a
+ comment), `withMetadataHistory`, `loadMetadataHistory`. Test with temp dirs: rewrite → one entry;
+ identical rewrite → none; no prior file → none; failure inside `run` still records the rewrite.
+3. `downloadOneManaged.ts`: `forceMedia` on the opts; the attempt-3 gate and log line; the
+ persist-only finalize when a transcript exists; the three `withMetadataHistory` wraps
+ (prefetch, audio-check primary) + `runYtdlp.ts` `downloadOneAudio`. Keep every existing arg
+ ordering the e2e asserts (`fetch-window.spec.ts:225-265`).
+4. Callers: `videoActions.ts` `archiveSourceVideo` and `persistKept.ts` pass `forceMedia: true`.
+ The "Persist source video" button's label/aria-label are contracts — unchanged.
+5. The view + the card line + `<details>` (5 above). `videoOperations.ts`/the video page loader
+ reads the sidecar once (follow how `availability.json` reaches the page).
+6. e2e: (a) extend `saved-videos.spec.ts` or add `persist-youtube-handling.spec.ts`: a
+ `handling: "youtube"` fixture video WITH a transcript on disk → "Persist source video" → the
+ fake yt-dlp receives a media download (not `--skip-download`) on the fallback pass, the pointer
+ lands, `saved-video.json` exists, the transcript's bytes are unchanged, NO `audio.<fmt>` is
+ created; (b) `fetch-window.spec.ts`: the full path on the same fixture → 202 → done with a file;
+ (c) metadata history: two downloads of one fixture video where the fake yt-dlp's info-json
+ differs the second time (add a knob to `fake-ytdlp.mjs` — e.g. an env or a marker file the
+ spec sets — that changes `title` and bumps `view_count`; keep the default output byte-stable)
+ → `metadata.history.json` has one entry with `changed.title` and `counters.view_count`; the
+ video page shows "Metadata rewritten 1×" and the from → to. Check `fake-ytdlp.mjs` for an
+ existing variation knob before adding one.
+7. Docs: `CHANNEL.md`/`SETTINGS.md` are generated — nothing there. One paragraph in `AGENTS.md`'s
+ "Clips and report-to-video" (after the fetch_clip paragraph): `full: true` / "Persist source
+ video" download the source even when a transcript or captions exist; every rewrite of
+ `metadata.info.json` appends to `metadata.history.json` (the old values of changed content
+ keys; formats/thumbnails only by fingerprint). Also the README's saved-video mention if there
+ is one (grep "Persist source video"). `plans/FACTS.md`: amend the slice M facts paragraph's
+ "silent no-op" sentence to say fixed by slice N.
+8. Record: `### Slice N, as shipped — the whole-recording fetch downloads anyway; metadata history
+ (2026-09-26)` in `plans/release-10.md` before `## Rollout`; `[Unreleased]` bullets in
+ `editor/CHANGELOG.md` (two: the fetch, the history).
+
+## Gates (worktree root)
+
+tsc `pnpm -r --no-bail --workspace-concurrency=1 exec tsc --noEmit`; `pnpm --filter
+yt-dlp-transcript-common test` (1,954 baseline → new count); editor unit `pnpm exec tsx --test
+"app/**/*.test.ts"` in `editor/` (85); `pnpm run test:scripts` (173 + 1 skip); mcp unchanged (269);
+`pnpm --filter editor exec next build`; e2e from the worktree root (queued, detached, `export/public`
+links in place per the rules): `saved-videos.spec.ts fetch-window.spec.ts <the new spec>` plus every
+spec that greps `downloadOneManaged|Persist source video|persist-kept|redownload` (list them in the
+record). The numbers tool: none (say so).
+
+## Rollout (parent)
+
+The live :3001 editor must be RESTARTED to pick this up — that is the operator's runbook step 1
+(release 10 is merged, not rolled out; this slice joins it, FINAL moves again). After the restart:
+`fetch_clip` with `full: true` on `teamrcn/dbnS-cBgStY` (the slice M no-op) → 202 → done with a file
+under `transcripts/saved-videos/teamrcn/dbnS-cBgStY/`, the transcript's mtime unchanged, no
+`audio.mp3`; and `data/dbnS-cBgStY/metadata.history.json` gains one entry (its `changed` may be
+empty — that is a valid "rewritten, nothing meaningful moved" record). Then the same on a video whose
+title has changed upstream, if one is known, to see a `changed.title`.
+
+## Known limitations
+
+- The history starts at the first rewrite AFTER this ships; earlier rewrites are gone.
+- `metadata.history.json` is per video and capped at 200; a corpus-wide "what changed lately" list
+ is a later slice (the data is there).
+- `forceMedia` on the bulk "Persist kept now" pass over a youtube-handling channel now downloads
+ every kept video's source — that is what the button says, but it is new bytes.
