commit 6b98abb7391aa04ff7d06032addc5fa1a7abcbd3
parent b7b88a56bb42860a3ab031f40dbbd63f9f109086
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Fri, 9 Oct 2026 12:40:18 -0400
Merge r19/integration into track C (release 21's plan, the jobKinds test fix) before the record commit
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
3 files changed, 186 insertions(+), 4 deletions(-)
diff --git a/PLAN.md b/PLAN.md
@@ -687,7 +687,7 @@ The phases above are the AI track. Later work is planned and recorded one releas
| 18 | publishing as queueable stages: one index, per-site bundles, serial builds, deploys checked live, the publish lane | [`plans/release-18.md`](plans/release-18.md) | complete on `r18/integration` (`main` merged in, `4cffda3f`); fast-forward + rollout owed |
| 19 | agents run the archive: ops/CLI/MCP (Track A), machine safety and tooling (Track B), OPERATING.md and the docs (Track C) | [`plans/release-19.md`](plans/release-19.md) | in flight |
| 20 | the data model and the index: recorded dates, Twitch ids, the caption-track bug closed | [`plans/release-20.md`](plans/release-20.md) | planned, after 19 |
-| 21 | — | — | not yet planned |
+| 21 | playable archives: local media attached to held videos (clips cut locally), per-video torrents played in the page, a home seeder of last resort behind a VPN; pilot TISM on jeralyzer-private | [`plans/release-21.md`](plans/release-21.md) | planned 2026-10-09 |
| 22 | — | — | not yet planned |
Work that landed on `main` without a plan of its own: [`plans/landed-2026-10.md`](plans/landed-2026-10.md).
diff --git a/common/jobs/jobKinds.test.ts b/common/jobs/jobKinds.test.ts
@@ -274,10 +274,12 @@ test("release 18: the kinds no longer created keep their labels (seven of them n
assert.equal(jobKindLabel("build-deploy-homepage"), "Build & deploy homepage");
});
-test("release 18: every drainable kind but the lane runners is an ingest kind", () => {
+test("release 18: every drainable kind but the lane runners and the clip-window batch is an ingest kind", () => {
const runners = new Set(["auto-transcribe", "auto-download", "auto-digest", "auto-backfill", "auto-publish"]);
+ // fetch-windows writes only `data/<id>/clips/`, a cache the index never reads.
+ const notIngest = new Set(["fetch-windows"]);
for (const kind of jobKindIds()) {
- if (!isDrainableKind(kind) || runners.has(kind)) continue;
+ if (!isDrainableKind(kind) || runners.has(kind) || notIngest.has(kind)) continue;
assert.equal(isIngestKind(kind), true, `${kind} is drainable and per channel: an ingest kind`);
}
for (const kind of runners) assert.equal(isIngestKind(kind), false, kind);
@@ -286,7 +288,7 @@ test("release 18: every drainable kind but the lane runners is an ingest kind",
assert.equal(isIngestKind(kind), true, kind);
}
// Nothing that publishes, moves or only reads.
- for (const kind of ["publish-build-site", "build-export", "relocate-channel-media", "scan-media", "fetch-window"]) {
+ for (const kind of ["publish-build-site", "build-export", "relocate-channel-media", "scan-media", "fetch-window", "fetch-windows"]) {
assert.equal(isIngestKind(kind), false, kind);
}
assert.equal(isIngestKind("no-such-kind"), false);
diff --git a/plans/release-21.md b/plans/release-21.md
@@ -0,0 +1,180 @@
+# Release 21 — playable archives: local media for held videos, WebTorrent playback on the sites
+
+Written 2026-10-09. Pilot: the TISM (The Incredible Salt Mine) v1 torrent archive →
+`the-incredible-salt-mine` (a Jeralyzer member channel). Two outcomes:
+
+1. **Agents can clip TISM** — every held TISM video gets its media on disk, through the editor, so
+ `fetch_clip` / `fetch-windows` / `reports prepare` / report-to-video cut it locally.
+2. **Readers can play TISM** — the site plays the video in the transcript page (click-to-seek works) from a
+ torrent. Nothing is hosted on Cloudflare: the bytes come from peers, and a home seeder behind a VPN is the
+ **peer of last resort** — it seeds a torrent only while no other seeder has it.
+
+## Rulings (operator, 2026-10-09)
+
+- **Pilot on `jeralyzer-private` first**; public Jeralyzer is a later, separate ruling.
+- **No web seed, no R2.** A **home seeder only**, behind a VPN, with an app layer that seeds a torrent only when no
+ other seeder exists (peer of last resort). Accepted consequence: a torrent with no other seeder plays only while
+ the home seeder is up.
+- **ASR for the 3 new videos only** (`pnpm ops transcribe-one`); the other 161 keep their YouTube captions.
+- *Open:* the attribution line for the archived copy on the video page (the archive's README names its maker and
+ sources). Records state facts; the wording is the operator's.
+
+## What is there (measured 2026-10-09)
+
+| | |
+|---|---|
+| Archive | `<archive drive>/v1-torrent/`: `Uploads/YouTube.zip` (36.5 GB, **stored**, 183 folders), `SoundCloud.7z` (2 mp3: TISM001/002), `3rd-party-Odysee.7z` (2 mp4, solid LZMA2), `Images.7z`, `Official-page-archives.7z`, `README-v1.html` |
+| YouTube.zip folders | 183: **169 with media** (36.4 GB, **222.7 h**), 10 `[LOST]` (source.txt only), 4 others without media; 33 carry an archive.org `youtube-<id>_archive.torrent`; each folder has yt-dlp `.info.json`/`.description`/thumbnail, or `description.txt` + `source.txt` |
+| Codecs (ffprobe, in place) | 124 mp4 h264/aac with the **moov at the back**, 21 mp4 h264/aac faststart, 9 webm vp9/opus, 11 mkv vp9/aac, 2 mkv h264/opus, 1 mp4 av1/aac, 1 mkv av1/opus. Heights: 109×360p, 35×720p, 21×1080p, 4×144p |
+| Corpus | `the-incredible-salt-mine` (handling youtube, the channel is deleted): 171 videos, each `metadata.info.json` + cues (161 YouTube VTT, 10 `transcript.json`), **no media**. All 171 ids are in the archive. 3 archive videos with media are **not held**: `6ijPdSuZdws`, `VcFA_LOnQoc`, `xLgf_MGQzXk` |
+| Disk | the archive drive has 849 GB free; the saved-video store already lives on it (`archilyzer-media/saved-videos`) |
+
+## Gaps in the code (verified)
+
+- **No local-file ingest.** Every import takes a URL or an archive.org identifier (`import-one`,
+ `import-archive-org`); `sourceKind` is `video | social`. Only the downloader attaches a container to a held
+ record (`persistSourceVideo`, `common/lib/savedVideo-server.ts:102`).
+- **A clip window never reads a saved container.** `fetchWindowAction`
+ (`editor/app/channels/[slug]/videos/[id]/videoActions.ts`, ~1080) asks `findContainingClipWindow`, then the
+ network; only `full: true` returns the saved container (~1259), and only `evidenceClip-server.ts` (reports
+ prepare, report-to-video) cuts from it. For a deleted channel, a window request fails every time today.
+- **The sites already play files and seek**: every player behind `PlayerProvider.tsx` exposes `seekTo`, and cue
+ rows call it (`TranscriptModal.tsx` `CueRow`), so click-to-seek needs nothing new.
+- **No torrent code beyond archive.org ingest** (aria2c, `archiveOrgTorrent*.ts`). No WebTorrent anywhere.
+- **The export site ships a root-scope service worker** (`export/service-worker/site-sw.js`). WebTorrent 2's
+ browser streaming needs a service worker (`client.createServer({ controller })`), so the two must be one worker.
+ The site sends no CSP, so wss trackers and blob media are not blocked.
+
+## Slices
+
+### D1 — attach local media to held videos `[unit]` (clippability; nothing public)
+
+- `common/controller/attachMedia.ts` + job kind `attach-media` (drainable, `needsMedia`), ops action
+ `attach-media {slug, source, items?:[{id, path}], match?, createRecords?, dryRun?}`, CLI `archilyzer media attach`.
+- **Sources:** a directory, or a **zip read in place** (a stored entry is copied straight out, no temp dir; a
+ deflated or 7z entry is extracted to a scratch dir on the target disk first). The id comes from the folder's
+ trailing `(<11-char id>)`, else the media file's `-<id>.` suffix, else an explicit item. `dryRun` lists matched,
+ unmatched, already attached, not held and `[LOST]`.
+- **Writes** through `persistSourceVideo` (the one writer): the container into the saved-video store as
+ `source-media.<ext>`, `keepReason: "pin"`, a new `origin: {kind: "local-archive", archive, entry, sha256,
+ attachedAt}` (which archive, which entry, its hash). Refuses over an existing pointer unless `replace`; never
+ modifies the archive. Guards: `assertSavedVideosStoreWritable`, the media-reachable guard.
+- **Not held** (the 3): `createRecords: true` writes `metadata.info.json` from the folder's yt-dlp `.info.json`
+ (else from `description.txt`/`source.txt`, the title from the folder name), then `pnpm ops transcribe-one` for each.
+- `[LOST]` folders are listed, never attached.
+
+### D2 — clip windows cut from a saved container `[unit]` + `[spec ops-api]`
+
+- In `fetchWindowAction`, after the clip-window cache and BEFORE the network: when the video has a saved container
+ covering `[from − pad, to + pad]`, cut the window locally (evidenceClip's ffmpeg path), write it into
+ `data/<id>/clips/` like a fetched window with provenance `{source: "saved-video"}`, and answer `cached: true`.
+ `fetch-windows` batches then use no network for such videos. Helps every persisted video, not only TISM.
+- `mcp/src/instructions.ts`: a video held locally is cut without a fetch.
+- Starts after release 19's A5 merges (A5 edits the same fetch-window code: dedupe + queue key).
+
+### D3 — streamable copies + torrents `[unit]`
+
+- `common/lib/playableMedia.ts` picks a **lossless** remux (`-c copy`) per container: h264/vp9/av1 + aac/opus → mp4
+ `+faststart`; vp9/av1 + opus in webm stays webm. 169 copies, ~36 GB, on the archive drive under
+ `archilyzer-media/playable/<slug>/<id>/<id>.<ext>`, keyed by the source's sha256 (a re-run is a no-op).
+- One **single-file torrent per video** (`create-torrent`, from the WebTorrent ecosystem): piece length scaled to the
+ size (256 KiB–1 MiB), no `urlList`, `name` = `<id>.<ext>`, no `comment`/`createdBy` naming a person or a private
+ URL. The announce list is not in the info dict, so the infohash depends only on the bytes. The pilot announces
+ only to a **self-hosted tracker**, so nothing public learns the infohashes. Going public later means adding public
+ WSS trackers (browsers) and UDP trackers (desktop clients) to the magnet: same infohash, no re-hash.
+- Job kind `prepare-playable` (heavy: runs in release 19 B1's `pnpm heavy` slot once that lands).
+
+### D4 — publish the torrents `[unit]`
+
+- `site.json` gains `playable: { channels: string[], trackers: string[] }` (schema + SITE.md regenerated): the
+ channels whose records carry a torrent, and the announce URLs that site's magnets name (the pilot: the
+ self-hosted tracker only).
+- The index stage writes `.export-index/sites/<id>/playable.json` (id → infohash, magnet, mime, bytes). The site's
+ `inputSig` includes it, so adding a torrent makes the build stale. No media bytes leave the machine.
+- Compose: records of a playable channel gain `playable: {magnet, infoHash, mime, bytes}` in the published JSON;
+ each `.torrent` ships in the bundle (`/t/<id>.torrent`, ~170 small files, within the 15,000 cap).
+ `corpus.json` / `llms.txt` name the field.
+
+### D4b — the seeder: peer of last resort, behind a VPN `[unit]` + a manual network test
+
+- `archilyzer seed`, a long-running node service: WebTorrent 2 in node, with WebRTC via `node-datachannel` so it
+ serves browsers, and TCP/uTP for desktop clients. It seeds the playable manifests of the sites named in settings
+ (`seeder: { sites, maxUploadKiBps, maxConnections, pollSeconds, standbyAfterSeconds }`, SETTINGS.md regenerated).
+- **Last resort**, per torrent: the seeder polls the trackers' scrape (`complete`, minus itself) and its own peer
+ view.
+ - Other seeders present for `standbyAfterSeconds` → **standby**: stop announcing, close peers, keep the data.
+ - No other seeder (`complete` = 0) while a leecher announces, or a connected leecher with no other source → resume.
+ - Both edges have hysteresis, so it does not flap, and the log names each transition and why.
+ - Known cost: if the last other seeder leaves mid-stream, the viewer stalls for up to `pollSeconds` until the
+ seeder resumes.
+- **Behind a VPN, with no way around it:** the seeder runs in its own network namespace or container whose only
+ route is the VPN tunnel (a docker compose profile `seeder` with a WireGuard sidecar). Kill switch: tunnel down
+ means no network, never the home connection. ICE gathers candidates only from the tunnel interface, so WebRTC
+ never offers the home address (the classic WebRTC leak). `archilyzer doctor` reports the seeder's egress IP
+ against the host's and refuses to call the seeder healthy if they match.
+- **The risk to measure first:** whether the VPN's NAT lets browser peers connect directly (UDP hole punching).
+ If it does not, the options are a VPN with port forwarding or a TURN relay. That is a ruling, not a default.
+- Manual network test (recorded): a browser on another network plays a torrent through the seeder, and the
+ seeder's candidates and the tracker's view of its address show only the VPN exit.
+
+### D5 — the player `[unit]` + `[spec]` (export e2e)
+
+- `common/components/TorrentPlayer.tsx`: WebTorrent 2 (npm, bundled, dynamically imported only on a playable record),
+ `client.add(magnet)` → streamed into a native `<video>` through the service worker; the same
+ `seekTo`/`onReady`/`onProgress` shape as `FilePlayer`.
+- **One service worker:** `site-sw.js` gains WebTorrent's stream handler on its own path prefix (no second
+ registration); sites with `pwa: false` register a minimal worker for it alone.
+- **Choice rule in `PlayerProvider`:**
+ - A playable record whose platform copy is gone (`isDeleted`/unavailable, as for all of TISM) plays the archived
+ copy by default.
+ - A record whose embed still works shows "Play archived copy".
+ - There is no file fallback: with no peer after a timeout, the player says no one is seeding it right now and
+ offers the magnet and `.torrent` ("seed this").
+ - A status line shows peers and download rate.
+- e2e: a fixture torrent (a few seconds of video) seeded by a node WebTorrent peer the export e2e starts, announced
+ to a local tracker the e2e starts. Checks: it plays, a cue click seeks, and the no-peer state shows once the seed
+ is stopped.
+
+### D6 — the rest of the archive `[none]` → `[unit]`
+
+- SoundCloud TISM001/002 (audio) and the two Odysee mirrors: a dryRun maps them to held episodes by title and
+ duration; each is attached as alternate media or skipped, by operator ruling. `Official-page-archives.7z` and
+ `Images.7z` are out of scope.
+
+## Graph and order
+
+```
+D1 (attach) ──► D2 (window cut; after release 19 A5) ← agents can clip TISM from here
+D1 ──► D3 (copies + torrents) ──► D4 (publish torrents) ──► D5 (player) ──┐
+ └──► D4b (seeder + VPN) ─────────────────────┴► pilot on jeralyzer-private ──► public ruling
+```
+
+D1, D3 and D4b touch no file owned by tracks A/B/C and may start now. D2 waits for A5. D4 and D5 wait until release
+19 merges (the ops routes, `_cli.ts`, the `site.json` schema docs).
+
+## Rollout (pilot)
+
+1. `pnpm ops attach-media --json '{"slug":"the-incredible-salt-mine","source":"<archive>/Uploads/YouTube.zip","dryRun":true}'`.
+ Expect about 166 held and matched, 3 with media but not held (`6ijPdSuZdws`, `VcFA_LOnQoc`, `xLgf_MGQzXk`) and
+ 10 `[LOST]`; the dryRun gives the exact counts, including held videos with no media in the archive.
+2. The same without `dryRun`, plus `createRecords`, then `transcribe-one` for the 3.
+3. MCP `fetch_clip` on a TISM cue → `cached: true`, provenance `saved-video`, no network.
+4. `prepare-playable`.
+5. Start the self-hosted tracker and `archilyzer seed` in the VPN namespace, and check its egress IP.
+6. Build `jeralyzer-private` and deploy it locally.
+7. From a browser on another network: play and seek. A second browser that finishes and seeds sends the home
+ seeder to standby; closing it makes the seeder resume.
+8. Public ruling → Jeralyzer (public trackers added to the magnets, same infohashes).
+
+## Verification
+
+- D1: unit tests over a fixture zip (stored and deflated entries, id matching, dryRun, refusing an overwrite,
+ provenance).
+- D2: route/action unit tests plus one `ops-api.spec` case.
+- D3: the remux decision table, and torrent determinism (same input → same infohash).
+- D4: a compose test (the field, the `.torrent` files, `inputSig` moves).
+- D4b: the standby/resume state machine against a fake scrape source (both edges, hysteresis), a kill-switch config
+ test, and the manual network test.
+- D5: the export e2e above.
+- Privacy: no personal identifier of the archive's maker (donation addresses and the like) in any published file;
+ no private site URL in any torrent.
