# The whole-recording fetch downloads anyway, and every metadata rewrite keeps the old version

Written 2026-09-26 (evening) in the fetch_clip session; anchors verified on `main` `151934e9` by one
Explore pass. Rules: `plans/tools/implementer-rules.md` (one Opus implementer in a worktree, one
read-only Opus review, the parent merges). Release 10 slice **N**; record in `plans/release-10.md`
before `## Rollout`, shape of "Slice M, as shipped".

## The operator's ask (verbatim, 2026-09-26)

"Change the editor to download anyway despite having metadata and/or subs, this specific call implies
that doesn't matter. Save old versions of metadata, maybe as diffs, so we can detect changes."

"This specific call" = the whole-recording fetch (`fetchFullSourceAction`, reached by the MCP's
`fetch_clip` with `full: true` and umtool's `--full`). Slice M's live proof found it a silent no-op on
`teamrcn/dbnS-cBgStY` (`handling: "youtube"`, transcript on disk): two yt-dlp passes, both
`--skip-download`, 7 s, no file — and `metadata.info.json` rewritten.

## Root cause (verified)

`common/ytdlp/downloadOneManaged.ts`:
- The primary attempt for `handling: "youtube"` is `youtubeHandlingArgs` (:159-169): subs only,
  **always** `--skip-download -t sleep`. `keepSourceVideoOverride` never reaches it.
- The only branch that downloads real media for youtube handling is attempt 3, the no-subs fallback
  (:1225-1270), gated `!hasAnyTranscriptOnDisk(videoDir) && metadataReportsNoCaptions(videoDir)`
  (:1233-1235). With a transcript on disk it is skipped. When it does run it switches
  `channelConfig` to `handling: "transcribe"` (:1240-1242), uses `transcribeMediaArgs(url,
  fallbackConfig, plan, fmt, /*reuseInfoJson*/ true, selector)` (:1256-1264) with `--load-info-json`
  (:1269-1271), and on success in app mode calls `finalizeAppExtraction` (:1289), which is what
  calls `persistSourceVideo` (:277) when `plan.persist`.
- `resolvePersistenceDecision` (`common/ytdlp/persistencePlan.ts:58-91`): `keepSourceVideoOverride:
  true` → `persist: true, extractionMode: "app"` unconditionally. The intent is right; it is unreachable.
- Two callers pass that override: `archiveSourceVideo`
  (`editor/app/channels/[slug]/videos/[id]/videoActions.ts:254-320`, reached from
  `redownloadToArchiveAction` :241-247 = the **"Persist source video"** button in
  `components/cards/SourceVideoSection.tsx:73-86`, and from `fetchFullSourceAction` ~:1094) and the
  bulk pass `common/controller/persistKept.ts:119-135` ("Persist kept now"). Same bug in both.
- The metadata prefetch (:643-751, `buildPrefetchArgs` :650-659) runs on EVERY attempt when the
  canonical id is known and rewrites `metadata.info.json` with no existence check
  (`backfillReacquire.ts:33-36` says so in words). Other writers: the audio-check primary
  (`transcribeHandlingArgsForAudioCheck` :315, always writes) and the legacy single-video job
  `downloadOneAudio` (`common/ytdlp/runYtdlp.ts:1365`). The window fetch (`fetchWindowManaged.ts:217`),
  `downloadSubsForUrl` (:1198) and the live-chat pass (:466) pass `--no-write-info-json`. The
  metadata scan never writes it (`metadataScanStore.ts:4-21`).
- No history/diff of anything per video exists; the precedent is `availability.json`'s `history[]`
  (`common/lib/availability-server.ts:38-66`, `sidecar(AVAILABILITY_FILENAME, sidecarField(...))`).
- A real info-json (`teamrcn/dbnS-cBgStY`, 52,231 bytes, 61 keys): `formats` = 43,496 bytes (83 %,
  signed URLs that differ on every fetch), `thumbnails` 4,923; `epoch`, `_version` and the counters
  (`view_count`, `like_count`, `comment_count`, `channel_follower_count`) drift every fetch.
- `sidecar()` (`common/lib/sidecar-server.ts:69-96`) declares a bare filename only (no
  subdirectories; refuses `transcript.<x>.<y>`). A new DIRECTORY under `data/<id>/` would need the
  `clips/` treatment in `channelSnapshot.ts:843-870`, `storageLocations.ts:100-101,177`,
  `views/storage.ts:91` and `reconcileVideoDirs.ts:84-96` — so the history is ONE FILE.
- Tests: `downloadOneManaged.test.ts` covers only the prefetch allow-list; e2e drives the real
  function through `editor/e2e/fixtures/bin/fake-ytdlp.mjs` (prefetch branch :783-789).
  `editor/e2e/saved-videos.spec.ts` ("Persist source video", fixture
  `one-transcribe-channel-with-audio`, slug `test-transcribe`, :106-169) and
  `editor/e2e/fetch-window.spec.ts` (window + full, :225-265 asserts the info-json args, :386 the
  cached full 200) — neither uses a `handling: "youtube"` fixture, so the bug has no coverage.
  Youtube-handling fixtures exist: `editor/e2e/fixtures/test-transcripts/one-youtube-channel*`,
  `youtube-with-playlist`.

## Decisions (made; do not re-open)

1. **One option, `forceMedia: true`, on `ManagedDownloadOpts`** (`downloadOneManaged.ts:98-151`).
   Meaning: "the caller wants the media; a transcript or captions on disk are not a reason to skip".
   Set by `archiveSourceVideo` (both callers) AND by `persistKept.ts:119-135` — "persist kept" means
   persist, and the operator sees a no-op there too. Not set by the download lane, re-acquire or
   import (their semantics are unchanged).
2. **Mechanics**: for `handling: "youtube"` with `forceMedia`, attempt 3 runs whenever the primary
   succeeded (`lastSucceeded`), regardless of `hasTranscript`/`noCaptions`; it logs ONE line first:
   `forceMedia: downloading the source although a transcript is on disk` (or `… although captions
   exist`). The primary subs pass still runs as today (it is what every re-download does). The
   transcript on disk is NEVER touched: with `forceMedia` and `hasTranscript`, the fallback's
   transcript-producing steps (inline transcription on fallback, any subs → transcript normalize
   the fallback triggers) are skipped, and `finalizeAppExtraction` runs in a **persist-only** mode
   (move the container into the saved-video store; no `audio.<fmt>` extraction — slice M's proof
   left an `audio.mp3` beside an existing transcript, which is waste). With `forceMedia` and NO
   transcript, the fallback behaves exactly as today (download, extract, transcribe inline if
   configured, persist). Transcribe-handling channels: no change (they already download).
   `plan.persist` is true on every `forceMedia` caller today; do not assume it — read the plan.
3. **Metadata history is one sidecar per video, `metadata.history.json`**, declared with
   `sidecar()` in a new `common/lib/metadataHistory-server.ts` (types + pure diff in
   `common/lib/metadataHistory.ts`, testable without fs). Shape:
   `{ entries: MetadataHistoryEntry[] }`, newest LAST, capped at 200 (drop oldest). An entry:
   ```
   { at: ISO, by: "prefetch" | "audio-check" | "download-one", requestedBy?: string,
     from: { sha256, bytes, at?: ISO (the old file's mtime) }, to: { sha256, bytes },
     changed: { [key]: { from, to } },      // content keys whose normalized value differs
     added: { [key]: value }, removed: { [key]: value },
     counters: { [key]: [from, to] },       // view_count etc., only those that moved
     volatile: string[] }                   // fingerprinted keys that differed (e.g. "formats")
   ```
   **Normalization**: VOLATILE keys are compared by fingerprint (`sha256` of their JSON) and never
   stored: `formats, requested_formats, requested_downloads, thumbnails, thumbnail, heatmap,
   automatic_captions, subtitles, epoch, _version, _format_sort_fields, url, http_headers,
   format, format_id, format_note, filesize, filesize_approx, tbr, abr, vbr, asr, acodec, vcodec,
   fps, width, height, resolution, aspect_ratio, dynamic_range, protocol, ext, audio_channels,
   container, downloader_options, quality, source_preference, has_drm, language_preference`.
   For `automatic_captions`/`subtitles` ALSO record the language-key lists as a content key
   (`subtitles_langs`, `automatic_captions_langs`) so a caption appearing IS detected. COUNTERS:
   `view_count, like_count, dislike_count, comment_count, channel_follower_count, average_rating,
   repost_count`. Everything else is CONTENT (title, fulltitle, description, duration,
   duration_string, upload_date, timestamp, release_timestamp, availability, live_status, is_live,
   was_live, age_limit, categories, tags, chapters, uploader*, channel*, playable_in_embed,
   media_type, webpage_url, …) — diffed by deep-equal on the raw value, stored whole (a
   description's from/to is exactly what the operator wants to read). Any single stored value
   above 16 KB is replaced by `{ sha256, bytes }`.
   **Append ALWAYS** on a rewrite (the operator wants to know it was rewritten even when only the
   formats moved — an all-volatile entry is ~250 bytes), bounded by the cap; the entry is written
   atomically through the sidecar. When the new file is byte-identical to the old (same sha), write
   nothing. When there was no old file, write nothing (a first write is not a rewrite).
4. **Interception**: `withMetadataHistory(videoDir, { by, requestedBy }, run)` in the server module:
   snapshot the existing file (bytes + sha + mtime) before `run()`, then after `run()` (success OR
   failure — the prefetch rewrites on failure too) compare and append. Wrap the three writers:
   the prefetch spawn (`downloadOneManaged.ts:658-662`), the audio-check primary attempt, and
   `downloadOneAudio` (`runYtdlp.ts:1365`). `requestedBy` = `opts.persistOrigin?.requestedBy` when
   present. Nothing else changes: the file yt-dlp writes is still the file.
5. **Surface** (small, read-only): the video page's source/metadata card gets one line when the
   history has entries — `Metadata rewritten N× · last <relative time> by <by>: <changed keys or
   "nothing meaningful (formats only)">` — and a `<details>` listing the entries newest first with
   the changed keys and, per key, from → to (strings truncated at 300 chars, full text in `title=`).
   Build it as a pure view in `common/views/` (`metadataHistoryView.ts`) fed by the loaded sidecar,
   in the style of the other views there; no new route, no client state.
6. No new job kind, no new directory, no eviction: the file is bounded and lives with the video.

## Implementation (one Opus slice, branch `editor/full-fetch-media`, worktree
`/home/user/Projects/editor-full-fetch-media`, ports editor 3601 / test 3611 / export 3610)

1. `common/lib/metadataHistory.ts` (pure): the key sets, `normalizeForDiff`, `diffMetadata(old,
   new) → { changed, added, removed, counters, volatile }`, `appendEntry(history, entry, cap=200)`,
   the 16 KB value guard. Tests: `metadataHistory.test.ts` — formats-only change → all-volatile
   entry with empty `changed`; a title change → `changed.title {from,to}`; a new caption language →
   `changed.subtitles_langs`; counters → `counters` only; cap at 200 drops the oldest; the 16 KB
   guard; byte-identical → `null` (no entry).
2. `common/lib/metadataHistory-server.ts`: `sidecar("metadata.history.json", …)` (check
   `SIDECAR_FILENAMES`' enumeration test and `PREFETCH_OWN_FILES` in `discardPrefetchDir`: a dir
   holding this file is never prefetch-only, so it must NOT be in the allow-list — say so in a
   comment), `withMetadataHistory`, `loadMetadataHistory`. Test with temp dirs: rewrite → one entry;
   identical rewrite → none; no prior file → none; failure inside `run` still records the rewrite.
3. `downloadOneManaged.ts`: `forceMedia` on the opts; the attempt-3 gate and log line; the
   persist-only finalize when a transcript exists; the three `withMetadataHistory` wraps
   (prefetch, audio-check primary) + `runYtdlp.ts` `downloadOneAudio`. Keep every existing arg
   ordering the e2e asserts (`fetch-window.spec.ts:225-265`).
4. Callers: `videoActions.ts` `archiveSourceVideo` and `persistKept.ts` pass `forceMedia: true`.
   The "Persist source video" button's label/aria-label are contracts — unchanged.
5. The view + the card line + `<details>` (5 above). `videoOperations.ts`/the video page loader
   reads the sidecar once (follow how `availability.json` reaches the page).
6. e2e: (a) extend `saved-videos.spec.ts` or add `persist-youtube-handling.spec.ts`: a
   `handling: "youtube"` fixture video WITH a transcript on disk → "Persist source video" → the
   fake yt-dlp receives a media download (not `--skip-download`) on the fallback pass, the pointer
   lands, `saved-video.json` exists, the transcript's bytes are unchanged, NO `audio.<fmt>` is
   created; (b) `fetch-window.spec.ts`: the full path on the same fixture → 202 → done with a file;
   (c) metadata history: two downloads of one fixture video where the fake yt-dlp's info-json
   differs the second time (add a knob to `fake-ytdlp.mjs` — e.g. an env or a marker file the
   spec sets — that changes `title` and bumps `view_count`; keep the default output byte-stable)
   → `metadata.history.json` has one entry with `changed.title` and `counters.view_count`; the
   video page shows "Metadata rewritten 1×" and the from → to. Check `fake-ytdlp.mjs` for an
   existing variation knob before adding one.
7. Docs: `CHANNEL.md`/`SETTINGS.md` are generated — nothing there. One paragraph in `AGENTS.md`'s
   "Clips and report-to-video" (after the fetch_clip paragraph): `full: true` / "Persist source
   video" download the source even when a transcript or captions exist; every rewrite of
   `metadata.info.json` appends to `metadata.history.json` (the old values of changed content
   keys; formats/thumbnails only by fingerprint). Also the README's saved-video mention if there
   is one (grep "Persist source video"). `plans/FACTS.md`: amend the slice M facts paragraph's
   "silent no-op" sentence to say fixed by slice N.
8. Record: `### Slice N, as shipped — the whole-recording fetch downloads anyway; metadata history
   (2026-09-26)` in `plans/release-10.md` before `## Rollout`; `[Unreleased]` bullets in
   `editor/CHANGELOG.md` (two: the fetch, the history).

## Gates (worktree root)

tsc `pnpm -r --no-bail --workspace-concurrency=1 exec tsc --noEmit`; `pnpm --filter
yt-dlp-transcript-common test` (1,954 baseline → new count); editor unit `pnpm exec tsx --test
"app/**/*.test.ts"` in `editor/` (85); `pnpm run test:scripts` (173 + 1 skip); mcp unchanged (269);
`pnpm --filter editor exec next build`; e2e from the worktree root (queued, detached, `export/public`
links in place per the rules): `saved-videos.spec.ts fetch-window.spec.ts <the new spec>` plus every
spec that greps `downloadOneManaged|Persist source video|persist-kept|redownload` (list them in the
record). The numbers tool: none (say so).

## Rollout (parent)

The live :3001 editor must be RESTARTED to pick this up — that is the operator's runbook step 1
(release 10 is merged, not rolled out; this slice joins it, FINAL moves again). After the restart:
`fetch_clip` with `full: true` on `teamrcn/dbnS-cBgStY` (the slice M no-op) → 202 → done with a file
under `transcripts/saved-videos/teamrcn/dbnS-cBgStY/`, the transcript's mtime unchanged, no
`audio.mp3`; and `data/dbnS-cBgStY/metadata.history.json` gains one entry (its `changed` may be
empty — that is a valid "rewritten, nothing meaningful moved" record). Then the same on a video whose
title has changed upstream, if one is known, to see a `changed.title`.

## Known limitations

- The history starts at the first rewrite AFTER this ships; earlier rewrites are gone.
- `metadata.history.json` is per video and capped at 200; a corpus-wide "what changed lately" list
  is a later slice (the data is there).
- `forceMedia` on the bulk "Persist kept now" pass over a youtube-handling channel now downloads
  every kept video's source — that is what the button says, but it is new bytes.