commit 7563c14c9c8456b6ddd100ced0bbad6595f9c1e7
parent d45e1cc4cdfda5d3ba5008811da95e0f69cd1f41
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Thu, 1 Oct 2026 20:56:12 -0400
plans: slice U1, as shipped — umtool's render scratch goes to a media root
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
| M | plans/release-17.md | | | 175 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
1 file changed, 175 insertions(+), 0 deletions(-)
diff --git a/plans/release-17.md b/plans/release-17.md
@@ -323,4 +323,179 @@ hand; a dirent `isFile()` filter over a video dir hides it."** The `.relocating.
## Record
+### Slice U1, as shipped — umtool's render scratch goes to a media root (2026-10-01)
+
+Branch `r17/umtool-media-root` off `main` `7f4901f1`, `main` `90bd8384` (the deck/posts-room merge)
+merged in mid-slice, worktree `~/Projects/homepage-social-visible` (`pnpm wt list` block #11: editor
+4101, test 4111, export 4110), one Opus implementer. Scratch files `U1-*` in the job's `tmp`. The
+ruling is the plan's: render scratch (`out/`) goes to a media root by default; deliverables move per
+project by a switch (slice U2); manifests, `revisions/`, the caches and the cue cache stay put.
+
+**What it does.**
+- **Two roots, one new knob.** `umtool/lib/paths.mjs`: `MEDIA_ROOT = UMTOOL_MEDIA_DIR || REPORTS_ROOT`,
+ `MEDIA_TIERED` (they differ), `mediaMirror(abs, roots?)` (a path under `REPORTS_ROOT` → the same
+ relative path under `MEDIA_ROOT`, null outside; pure). `MEDIA_ROOT` joins `READ_ROOTS`, never
+ `WRITE_ROOTS`. Unset, nothing changes: `out/` is a directory in the project, and `READ_ROOTS` dedupes
+ it away. Every path op carries `turbopackIgnore`.
+- **The cache leaves `SONG_DATA`.** `CACHE_DIR = UMTOOL_CACHE_DIR || $XDG_CACHE_HOME/archilyzer/umtool`
+ (an empty `XDG_CACHE_HOME` is unset, as `common/lib/paths.ts` reads it; default `~/.cache`).
+ `INDEX_DIR`, `MIX_CACHE`, the posters, loudness and clip audio follow it. `OLD_CACHE_DIR`
+ (`<SONG_DATA>/.cache/umtool`) is named only for the doctor. Nothing is migrated: the index is
+ rebuilt by `umtool index` and everything else is remade on demand. The cue cache
+ (`REPORT_CACHE_DIR`, `report-to-video/cues.mjs`) is untouched.
+- **`umtool/lib/report/storage.mjs`** (new; modelled on `common/controller/relocateDir.ts`, not
+ importing it):
+ - `ensureOutDir(projectDir, roots?)`: a real `out/` → kept; a link to a directory → kept; a
+ **dangling link → refused** ("… is a link to …, which is not there — is the media drive mounted?
+ Nothing was written, and nothing was created in its place."); absent and tiered →
+ `mkdir -p <mirror>/out` and an absolute `symlink`; absent and not tiered → `mkdir` as before. The
+ media root itself is **stat'd and never created** (`mediaRootProblem`: missing, not a directory, or
+ inside/around `REPORTS_ROOT`). A project outside `REPORTS_ROOT` is never tiered. EEXIST from a
+ concurrent first writer is accepted when the winner resolves.
+ - `ensureWriteDir(dir)`: a directory a pipeline step writes into; when it is a project's `out` or up
+ to four levels under one, that `out` goes through `ensureOutDir` first, then `mkdir -p`.
+ - `moveDirToMedia(projectDir, name, opts)` / `moveDirToLocal(...)`: by NAME (`out` now; `clips`,
+ `share-*` for U2). Copy (`rsync -a --partial`), mirror toward the copy only (`-a --delete
+ --info=del`; refused when source and copy contain one another), verify (`--dry-run
+ --itemize-changes --delete` empty, one more mirror pass on a difference, a second refuses; equal
+ counts/bytes), then park (`<name>.moved-<ts>`), link, delete the parked copy. Space check on the
+ destination's volume (bytes + 1 GB). Every state is dispatched on the disk, so a cut run is
+ finished by running it again: a link to the mirror → `already` (a leftover parked copy removed);
+ absent with one parked copy → link and delete it; the reverse uses `<name>.incoming`, and after
+ the rename deletes the media copy and every directory above it the move left empty, never the
+ root. `dryRun` measures and changes nothing.
+ - `outDirState`, `pathState`, `measureTree` for readers and the CLI.
+- **Call sites.** `build-video.mjs` (`outRoot`, before any fetch), `check-availability.mjs` (its one
+ write), `render-cards.mjs` (CLI `--out`), `compose-chrome.mjs` (its `out/<variant>` base),
+ `lib/report/onscreen.mjs` (`deckStill`'s scratch) all make `out/` through `ensureWriteDir`.
+ `lib/report/export.mjs` reads only: it now says "out/ is a link to …, which is not there — is the
+ media drive mounted?" instead of "no build" when the link dangles. tmp-then-rename sites (`cut.mjs`,
+ the clip route) are untouched.
+- **The walk.** `kinds.mjs` `SKIP_DIRS` adds `clips` (`out` was already there) and `SKIP_PREFIXES =
+ ["share-"]`, read through `skipsDir(name)` by `walk.mjs`, so the project walk never stats a link
+ into a drive that is not there. The mix picker (`lib/media.ts`) follows a project's `out` link when
+ it points INTO the media root (so a tiered deliverable stays in the picker under its project) and
+ does not walk `MEDIA_ROOT` as a root of its own (it would list every tiered file twice).
+- **CLI.** `umtool doctor` adds `roots` (JSON) / a "roots" block: reports, media (tiered or "=
+ reports"), cache (and whether an index exists), and the old cache with its size while it is there;
+ it exits 1 when the media root is set and missing (the tools' `ok` keeps its meaning). `umtool
+ storage [<project>]` lists every project's `out` (dir, link, DANGLING, none); `umtool storage
+ move-out|move-back <project>|--all [--dry-run] [--json]` runs the movers, one line per project and a
+ total; move-out without `UMTOOL_MEDIA_DIR` refuses once.
+- **e2e env.** The app server and the specs' CLIs get `UMTOOL_CACHE_DIR=<fixture>/cache`
+ (`playwright.config.ts`, `projects.spec.ts`, `report-longform.spec.ts`; the index-deletion spec
+ now removes `cache/index`), so no run writes `~/.cache`. `make-fixture.mjs` adds
+ `storage-fixture` (a cached window, buildable offline), `storage-fresh-fixture` (no `out/`) and the
+ media root `umtool/.e2e-song-media/`, a sibling of the fixture (inside it would be inside
+ `REPORTS_ROOT`, which is refused), reset every run; `.gitignore` and umtool's trace excludes name
+ it. `storage.spec.ts` (new, 5): move-out (dry run first; the index's state and facts unchanged
+ through the link; again → already); **a build the app runs writes through the link and leaves it a
+ link** (the app has no `UMTOOL_MEDIA_DIR` at all); the root renamed away → `storage` says
+ dangling, `check-availability` refuses with the drive sentence on the moved project and with "is
+ not there" on the fresh one, no `out` made, the root not recreated, `doctor` exits 1; the first
+ writer of the fresh project makes the link; move-back → a real `out/`, the project's mirror gone,
+ the root and the other project's mirror kept.
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `65a3d146` | `umtool:` `MEDIA_ROOT`, `MEDIA_TIERED`, `mediaMirror`; `MEDIA_ROOT` in `READ_ROOTS`; `CACHE_DIR` from `UMTOOL_CACHE_DIR` / `XDG_CACHE_HOME`; `OLD_CACHE_DIR` |
+| `47d6d1b4` | `umtool:` `lib/report/storage.mjs`; the five writers through `ensureWriteDir`; export's dangling sentence; `clips` + `share-*` skips; the picker follows `out` links into the media root; `doctor` roots; `umtool storage` |
+| `4c2cd0c7` | merge of `main` `90bd8384` (the deck/posts-room branch: `build-video.mjs`, `make-fixture.mjs` and more) — one conflict, `export.mjs`'s imports, both kept |
+| `cb08e57a` | `umtool:` `storage.test.mjs` (20); the e2e cache in the fixture; the storage fixtures, media root and `storage.spec.ts`; `umtool storage <project>` status |
+| `efef56b3` | `umtool:` `docs/folders.md` (`MEDIA_ROOT`, `CACHE_DIR`), `docs/cli.md`; two `[Unreleased]` bullets in `editor/CHANGELOG.md` |
+| this commit | `plans:` this section |
+
+#### Gates (logs `$T/U1-*`)
+
+- **tsc** (all workspaces) clean at `47d6d1b4`, at the merge `4c2cd0c7` and at `cb08e57a`.
+- **common:** 2,484/2,484 (300 s, under load). **Editor unit:** 109/109. Neither touched; run on the
+ merged tree.
+- **test:scripts:** 390 tests (the merged `main`'s 370 + `storage.test.mjs`'s 20): 386 passed, 1
+ skipped, 3 failed, then 385/2/3 on a rerun — the three are `queue-lock.test.mjs` timing cases, a
+ different three each time, at a load average of 27–47 (other implementers' suites and builds);
+ `node --test scripts/queue-lock.test.mjs` alone: **11/11**. `storage.test.mjs` **20/20**.
+ `next-build-trace.test.mjs` is in it: **10/10** after each capped build below (its second skip on
+ the rerun is the staleness rule: the baseline run's `git checkout` of `main`'s umtool, below, gave
+ the modules new mtimes after the build).
+- **The capped umtool build with the corpus linked** (`ln -sT <primary>/transcripts transcripts`,
+ 76 channels visible through it; `systemd-run --scope -p MemoryMax=5G -p MemorySwapMax=0`, `timeout
+ -s KILL 240`, the link removed after), at `cb08e57a`: **exit 0, 53 s, 0.83 GB peak**; and once more
+ with `UMTOOL_MEDIA_DIR` set to a scratch directory: **exit 0, 53 s, 0.83 GB**. The two builds'
+ `.nft.json` entries (39,658 each, every route) are **identical** (`diff` empty); none names
+ `transcripts`, the scratch media root or `.e2e-song`. (The worktree carries an old `transcripts/`
+ directory — an `index.mdb` — which `ln -sT` refuses to replace: the script sets it aside for the
+ build and puts it back. A first attempt that did not was stopped before it counted.) The capped
+ editor build was not run: no editor code changed (only `editor/CHANGELOG.md`).
+- **Numbers tool:** none.
+- **umtool e2e** (`SONG_DIR=~/reports/quartering-uh-song/data pnpm --filter umtool run e2e …` from
+ the worktree root; the fixture found song data, `cand2`, `wav48` and `media`, no `asr`, no face
+ detector, so `find.spec`'s 14 skip):
+
+ | Run | At | Specs | Result |
+ |---|---|---|---|
+ | 1 | `efef56b3` | `storage`, `projects`, `report-longform`, `dashboard` | 37 passed, 6 failed, 10.3 min — `storage.spec` **5/5**; the six (`dashboard` ×3, `projects` ×3) are `page.goto: net::ERR_ABORTED` and 30 s timeouts at a load average of 47, and all six pass in run 2 |
+ | 2 | `efef56b3` | the full suite (21 files) | **214 passed**, 17 failed, 14 skipped, 13.8 min of tests (48 min with 34 min in the queue) — `faces` ×4 (`/api/face/detect` 503: no detector here), `triage` ×9 (no `asr` here), `browse:241`, `mix:166`, `mix:201`, `usage:112` |
+ | 3 | `main` `90bd8384`'s umtool, checked out into the worktree and restored after | `browse`, `faces`, `mix`, `triage`, `usage` | 48 passed, 15 failed, 7.2 min — the same `faces` ×4 and `triage` ×9, plus `browse:15`/`:34` (30 s timeouts) |
+ | 4 | `efef56b3` | the same five | 49 passed, 14 failed, 3.2 min — `faces` ×4 and `triage` ×9 as on `main`; `browse:241`, `mix:166`, `mix:201` pass; `usage:112` fails again |
+ | 5 | `efef56b3` | `usage` | **7 passed**, 0 failed, 19.6 s |
+
+ So against `main` on this machine: the `faces` and `triage` failures are the machine's (both
+ missing capabilities fail rather than skip — on `main` too); `mix:166`/`:201` and `browse:241`
+ fail only after the whole suite (the corpus window an earlier spec fetched for `vid1` wins the
+ picker's lookup), and pass in isolation on both; `usage:112` ("confirming the drop writes it
+ through") failed twice when it ran right after the failing `triage` specs on this branch, passed
+ once in that position on `main`, and passes alone — the verdict path it drives reads no cache and
+ no `out/`. Left to the reviewer as an order/timing question, not changed.
+
+#### Found and left
+
+- **Open question 2 — the four `*.mp4` near the project roots** (measured in `~/reports`, depth ≤ 2,
+ outside any `out/`): `kirsche-pippa/latest-contact-2026-06-20.mp4` (7.4 MB) is a cited clip fetched
+ through the MCP's `fetch_clip`, with its `.provenance.json` beside it — the sweep report's evidence,
+ a deliverable of a project that has no `out/`; `quartering-uh-song/jer-metalslug-bg.mp4` (51.5 MB)
+ and `quartering-uh-song/pokemon-no-music-recording.mp4` (3.1 MB) are song-project INPUTS (a song
+ spec's `background.path` names such a file relative to a media root, `song/spec.mjs`); `clips/
+ tim-pool-…mp4` (23.6 MB) is a loose cut at the reports root, in no project. None is render scratch:
+ all four are left untouched, and none is under U2's `clips/` or `share-*/`.
+- **What move-out would move today:** `umtool storage move-out --all --dry-run` against `~/reports`
+ (a scratch media root): **10 projects, 10.6 GB** (quartering-diet 5.0 GB, ferret-rescue 1.5 GB,
+ quartering-employee-count 1.2 GB, elfpire-eva 1.1 GB, …) — less than the plan's "≈ 18 of the 20
+ GB": the rest of `~/reports` is song data and loose files, not project `out/`s.
+- **Rollout, once:** set `UMTOOL_MEDIA_DIR` (the live umtool's environment) to a directory that
+ exists on the media drive, outside `~/reports`; restart umtool; run `umtool index` (the index is
+ rebuilt under `~/.cache/archilyzer/umtool`; until then everything works, slower); `umtool doctor`
+ shows the roots and the old cache (8.7 MB here), which can then be deleted; `umtool storage
+ move-out --all` (when nothing is building) moves the existing `out/`s.
+- **A dangling `out` reads as "no build" to the summary readers** (`lib/projects/report.mjs`'s
+ `stat0(out)`, `readAvailability`): only `export` and the writers say "is the media drive mounted?".
+ `umtool storage` and `umtool doctor` name it. `umtool check` learning it is U2's (plan: "`umtool
+ check` learns the two values").
+- **For U2:** `lib/report/deliver.mjs` `listBatches` filters `isDirectory()` on the project's dirents,
+ so a `share-*` that is a link would vanish from it; `sharedIdsIn`'s walk likewise does not follow a
+ link. The movers take any one-segment name and return `{ state, src, dest|from, bytes, files }`.
+- **A CLI move cannot see the app's jobs** (they live in its memory): the verify refuses when the tree
+ keeps changing, but a write in the instant between the verify and the park would be deleted with
+ the parked copy. The CLI says "run when nothing is building"; U2's bench button runs in the app and
+ can check.
+- **The worktree's stray `transcripts/`** (an `index.mdb` from 2026-09-28) is the trap the rules
+ describe; left in place.
+
+#### Deviations from the plan
+
+- `ensureOutDir` is not called in `driver.mjs`: its step builders are synchronous, are unit-tested with
+ a fake project directory, and only build argv; the call is in the scripts those steps run
+ (`build-video.mjs`, `check-availability.mjs`) through `ensureWriteDir`, which also covers a
+ hand-run script and the three other writers the plan did not list (`render-cards.mjs`,
+ `compose-chrome.mjs`, `onscreen.mjs`).
+- `export.mjs` writes nothing under `out/`, so it does not create it; it reports a dangling link instead.
+- `umtool storage move-back` and the plain `umtool storage [<project>]` listing were added beside
+ `move-out`: the e2e needs the way back, and an operator needs to see which projects moved.
+- `lib/media.ts` (not in the plan) follows `out` links into the media root and skips the root as its
+ own: without it every moved deliverable fell out of the mix picker.
+
+`[Unreleased]` (`editor/CHANGELOG.md`): "umtool can keep each report's render folder on a media
+drive." and "umtool's cache moves to `~/.cache/archilyzer/umtool`".
+
## Rollout
