commit daadb0befa068115ad0682881f926b8cb03c6f5a
parent 2ba8a09e4581e93d26eee593c8faa22d9e936905
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Fri, 9 Oct 2026 14:07:07 -0400
plans: release 21 Track D record — D1 attach-media, D3 prepare-playable, D4b the seeder of last resort, as shipped, with the gates and what is owed to the rollout
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
| M | plans/release-21.md | | | 142 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
1 file changed, 142 insertions(+), 0 deletions(-)
diff --git a/plans/release-21.md b/plans/release-21.md
@@ -152,6 +152,148 @@ D1 ──► D3 (copies + torrents) ──► D4 (publish torrents) ──► D5
D1, D3 and D4b touch no file owned by tracks A/B/C and may start now. D2 waits for A5. D4 and D5 wait until release
19 merges (the ops routes, `_cli.ts`, the `site.json` schema docs).
+## Record
+
+### Track D
+
+One Opus implementer, branch `worktree-agent-a4b4aebbb6c0d28bb` off `5bc20d32` (`r19/integration` with this plan),
+`r19/integration` merged in before this record (`2e3a9395`: tracks B and C — the docs cli, the heavy slot). Slices
+D1, D3, D4b, in that order; D2, D4, D5 are not this track's. Nothing ran against the real corpus or the real
+archive beyond read-only listings of the archive's central directory (`unzip -l`, `zipinfo`, and the new zip
+reader over it); every fixture is synthetic.
+
+#### D1, as shipped — attach local media to held videos
+
+**What was found before building.**
+- The archive is ZIP64 (790 entries, 36.45 GB, every entry stored): its later offsets do not fit 32 bits. No zip
+ library is in the workspace, so `common/lib/zipReader.ts` reads the central directory itself (EOCD, the ZIP64
+ locator and record, the 0x0001 extra, the Info-ZIP Unicode-path extra) and copies a stored entry as a byte range
+ of the archive, a deflated one through `inflateRaw`, checking the size and CRC32 against the central directory.
+ Over the real archive (read-only) it lists the 790 entries and their byte total matches `zipinfo`'s exactly.
+- One media file name in the archive is non-ASCII (a full-width `｜` and `？`); the folder's `(<id>)` names it.
+- The archive's folder classes, from the listing alone (no corpus read): 182 folders with files, 169 with media,
+ 10 `[LOST]`, 2 with an id and no media, and the archive's root (no media, no id).
+- `persistSourceVideo` MOVES `<videoDir>/<sourceFilename>`. Copying a 36 GB archive into the corpus SSD first would
+ be the wrong disk twice, so it gains `sourcePath` (the file to move, a staging file beside the store dir on the
+ store's disk — a same-filesystem rename) and `sha256` (recorded when the caller hashed while copying).
+- Every consumer of `SavedVideoOrigin` reads `requestedBy` as required (the video page's "Requested by", the clip
+ requester line in `videoActions.ts`, which this track may not touch). So the local-archive origin keeps
+ `requestedBy: "attach-media"` and adds `kind: "local-archive"`, `archive`, `entry`, `sha256`, `attachedAt`;
+ `parseOrigin` reads them only under that kind, and a pointer without `kind` parses exactly as before.
+- The non-yt-dlp writer of `metadata.info.json` for a new record is `withMetadataHistory` around
+ `writeJsonAtomic(…, {indent: 0, newline: false})`, as `archiveOrgDownload` writes its records; a new writer
+ `local-archive` joins `METADATA_HISTORY_WRITERS`.
+
+**What it does.** `common/lib/attachMediaPlan.ts` (pure): folders grouped by parent; the id from the folder's
+trailing `(<11-char id>)`, else the file's `-<id>.`, ` [<id>].` or bare `<id>.`; classes `attach`, `create`,
+`already-attached`, `not-held`, `lost`, `no-media`, `unmatched`, `ambiguous` (two video files in a folder, or two
+folders claiming one id), plus the held videos no folder supplied (only when the whole archive was considered).
+`common/controller/attachMedia.ts`: sources are a directory, a `.zip` read in place, or a `.7z` (listed with
+`7z l -slt`, one entry extracted with `7z e -so -spd` into the staging file); the store's root must exist (an
+unmounted drive is never materialised), `assertSavedVideosStoreWritable` before each copy, size plus 2 GiB free
+on the store's disk or the run stops (a re-run resumes); drain and cancel between videos, a cancel aborts the copy
+and removes its staging file; `keepReason: "pin"`; refuses over a pointer unless `replace` (a replaced container of
+another name is removed after the new pointer is written). `createRecords` writes the folder's own yt-dlp
+`.info.json` (refused when its `id` is another video's) or one built from `description.txt` ("Published on …" →
+`upload_date`, last key) and the folder name, the duration from ffprobe, and appends `youtube <id>` to the
+channel's `archive`. The log ends with `summary: {…}`. Job kind `attach-media` (drainable, `needsMedia`, an ingest
+kind — its pointers and records are what the index reads; not replayable; the channel's queue). The action is
+`editor/app/channels/[slug]/attachMediaActions.ts` (a new file); `POST /api/ops/attach-media` on the `_lib.ts`
+precedent; one `ACTIONS` entry and one usage paragraph in `scripts/archilyzer-ops.mjs`; `archilyzer media attach
+<slug> <source>` (offline; asks the channel's media guard itself).
+
+#### D3, as shipped — playable copies and torrents
+
+`common/lib/playableMedia.ts` (pure) is the decision table: h264/vp9/av1 with aac/opus/mp3 (or no audio) → mp4
+`+faststart`; vp9/vp8/av1 with opus/vorbis in a `.webm` → webm with `-cues_to_front 1`; anything else (HEVC,
+MPEG-4 Part 2, AC-3 audio, no video) is listed as not playable, never re-encoded. Only the first real video stream
+(`0:V:0`, not cover art) and the first audio are kept; subtitles, data, global metadata and chapters are dropped;
+`+bitexact`, so the same source remuxes to the same bytes (tested). The measured mix in this plan is every row of
+the table's playable half. `common/controller/preparePlayable.ts`: copies, torrents and `playable.json` under
+`playable/<slug>/<id>/` beside the saved-video store's real directory (or `root`); keyed by the source's sha256 (a
+pointer with none is hashed once and the hash recorded on it), so a re-run with the files on disk at their size is
+a no-op; each copy written to a `.part` and renamed; the manifest rewritten after each video. Each remux takes the
+heavy slot (`scripts/queue-lock.mjs --heavy`, release 19 B1) per video, so a long run interleaves with builds and
+e2e; a checkout without the gate runs ffmpeg directly. **The torrent:** `create-torrent` with `name` `<id>.<ext>`,
+piece length a power of two in 256 KiB–1 MiB scaled to ~2,048 pieces, a fixed creation date, the announce list from
+the `trackers` parameter (an explicit empty list, never create-torrent's default public trackers), and NOT passed:
+`createdBy`, `comment`, `urlList`, `private` — even `private: false` would put `private: 0` inside the info dict and
+move the infohash. Tested: same bytes and piece length → same infohash whatever the path or the trackers; another
+piece length → another; the .torrent byte-identical across runs. Job kind `prepare-playable` (drainable,
+`needsMedia`, ingest — D4's build reads the manifest; one `playable` queue machine-wide), `POST
+/api/ops/prepare-playable`, one ops row, `archilyzer media playable <slug>`.
+
+#### D4b, as shipped — the seeder of last resort, behind a VPN
+
+- **The state machine** (`common/lib/lastResortSeeder.ts`, pure): a torrent starts seeding; other seeders (the
+ scrape's `complete` minus itself, or connected seeds, whichever is more) on every poll for `standbyAfterSeconds`
+ → standby; in standby, no other seeder and a leecher waiting (the scrape's `incomplete` or a wire) → seeding at
+ once, or no other seeder on every poll for the resume window → seeding; trackers silent for the window while in
+ standby → seeding (a last resort assumes it is the only one). A transition restarts both clocks. The resume window
+ is `standbyAfterSeconds` (no separate key). 13 tests against a scripted scrape: both edges, hysteresis on each,
+ a flapping swarm that never flaps it, the plan's rollout step 7 as a script.
+- **The service** (`common/controller/seeder.ts`): WebTorrent 3 in node — TCP, and WebRTC through
+ `webrtc-polyfill` → `node-datachannel` (WebTorrent's own dependency chain). Standby removes the torrent from the
+ client with its data kept (`stopped` announced, peers closed); seeding adds it back with `skipVerify` after a
+ size check of the copy. No DHT, LSD, UPnP/NAT-PMP or web seeds. Scrapes are one request per tracker for every
+ infohash, merged per torrent by max; wires are counted by peer (two peers that dialled each other hold two
+ wires — found by the integration test, which first saw "2 other seeders"). `bindInterface` held: the seeder
+ waits for it and drops every torrent when it disappears. Targets are re-read each poll from the playable
+ manifests of every channel of `settings.seeder.sites`.
+- **The tracker** (`archilyzer tracker`): bittorrent-tracker, HTTP + WebSocket, no UDP, no stats page, bound to
+ `--host` (default 127.0.0.1), refusing every infohash not in the manifests (re-read at most once a minute).
+- **Settings** `seeder: {sites, trackers, maxUploadKiBps, maxConnections, pollSeconds, standbyAfterSeconds,
+ bindInterface}` (`common/lib/seederSettings.ts`, SETTINGS.md and settings.json.example regenerated); not
+ configured = `sites: []`, and `archilyzer seed` refuses to start.
+- **The VPN** (`docker-compose.seeder.yml`, profile `seeder`, an overlay on `docker-compose.yml`): gluetun
+ `v3.41.3` (custom provider, WireGuard; its firewall is always on in v3 — there is no switch, and the file names
+ none), the seeder and the tracker with `network_mode: service:seeder-vpn`, no networks or ports of their own,
+ waiting for a healthy tunnel, the corpus read-only. The WireGuard config is the operator's: `${SEEDER_WG_CONF:?…}`
+ mounted read-only — compose refuses the file without it (checked with `docker compose … config`). Inbound only
+ through `SEEDER_VPN_INPUT_PORTS` (empty by default). `lib/seederCompose.test.ts` pins that construction.
+- **Doctor** `seeder` section: "seeder not configured" (a note, no network) while `sites` is empty; else the host's
+ egress address and the seeder container's (`docker exec archilyzer-seeder-1 node -e fetch(…)`, one echo URL):
+ the same address FAILS, either unanswered warns, different is ok.
+- **Measured (localhost, the integration tests):** a TCP leecher completes from the seeder; it then seeds, the
+ seeder logs `seeding → standby — 1 other seeder(s) on every poll for 1s`; the leecher leaves and it logs
+ `standby → seeding — no other seeder on every poll for 1s`. A WebRTC leecher completes through the WebSocket
+ tracker over a `webrtc` wire (node-datachannel). Two node peers that announce in the same instant each answer the
+ other's offer and each keep the connection the other dropped; the test starts the leecher after the seeder's
+ announce is in, as a browser arriving at a running seeder does.
+- **Not measured, owed to the rollout:** whether the VPN's NAT lets browser peers connect (the plan's risk), and
+ the manual network test. WebTorrent's default `rtcConfig` asks public STUN servers for the seeder's address:
+ inside the profile that address is the VPN exit; on a bare host it would be the home address — run it in the
+ profile.
+
+**New dependencies (common):** `create-torrent` 6.1.3, `parse-torrent` 11.0.24 (D3: the torrent and its
+infohash), `webtorrent` 3.0.21 (node ≥ 22; the repo runs 22.23), `bittorrent-tracker` 11.2.3 (D4b). Types for the
+four are `common/types/webtorrent-modules.d.ts` (they ship none). `pnpm-workspace.yaml` `allowBuilds`:
+`node-datachannel: true` (its prebuilt binary); `bufferutil`, `utf-8-validate`, `utp-native`, `ip-set` false (pure-JS
+fallbacks, optional uTP, a package-manager check).
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `9686fe67` | `common:` attach-media — zip reader, plan, controller, origin kind, `sourcePath`, job kind, ops route + action, ops row, CLI; 33 tests (31 common, 2 editor) |
+| `f1370406` | `common:` prepare-playable — decision table, remux, torrents, manifest, job kind, ops route + action, ops row, CLI; 13 tests (11 common, 2 editor) |
+| `2f704bc6` | `common:` the seeder — state machine, service, tracker, settings, compose profile, doctor; 26 tests |
+| `2e3a9395` | merge `r19/integration` |
+| `0f79dd6e` | `common:` each remux takes the heavy slot (per video; 1 test); COMMANDS.md regenerated |
+| `c066fb37` | `common:` the plan test's titles and ids synthetic throughout |
+
+**Gates.** `pnpm -r --no-bail --workspace-concurrency=1 exec tsc --noEmit` clean at each commit and after the
+merge. Common 3,530 / 3,533 at `2f704bc6` (+68 over the base, 3,465): the three failures are two `autoRunner.test.ts`
+cases (`computeLeafPending …`, `a channel with a move marker …`) that fail identically with the base's
+`jobKinds.ts`, and `fetchPosts.test.ts`'s drain case, which passes alone (load). Editor 163/163 (+4 route tests).
+`test:scripts` 676 + 3 skips at `2f704bc6`, one `queue-lock` banner case failing under load and passing alone (11/11).
+No builds, no e2e (none in this track's budget). After the merge, an operator hold (no heavy runs) left only tsc
+and single test files: preparePlayable 5/5 (the heavy-slot gate included), cli-docs + jobKinds 21/21, and (before the merge) the settings/doctor/CLI/env tests 95/95.
+
+**Left.** `COMMANDS.md` gains the four rows (`media attach`, `media playable`, `seed`, `tracker` — regenerated
+after the merge brought `docs cli`). The rollout's dryRun on the live editor is the parent's. Transcription of
+the three new records is the operator's (`transcribe-one`), as ruled. D2, D4, D5 are other slices.
+
## Rollout (pilot)
1. `pnpm ops attach-media --json '{"slug":"the-incredible-salt-mine","source":"<archive>/Uploads/YouTube.zip","dryRun":true}'`.
