commit d74c28484b7be9efa799443b05d4b904fb6ce874
parent 5c877a7f8351f6d26fdab95e47543fd8ffbc84f1
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Fri, 2 Oct 2026 00:37:05 -0400
Merge r17/umtool-deliverables (release 17 slice U2) — umtool's deliverables move per project by a switch: the manifest's storage.deliverables (local|media, one writer), deliverableDir for clips/ and share-*/ (links into the media mirror under media; a dangling link or a cut move's leftover refuses), umtool storage deliverables and a bench Move deliverables job, umtool check reports an unreachable or mismatched storage, the busy scan sees the app's own cuts and share batches; reviewed SHIP
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
21 files changed, 1847 insertions(+), 92 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -2,6 +2,7 @@
## [Unreleased]
- **umtool can keep each report's render folder on a media drive.** With `UMTOOL_MEDIA_DIR` set, in umtool's environment (restart umtool after setting it), to a directory inside that drive, a report project's `out/` (its fetched windows, segments and finished video) is a link to the same path under that directory: a project's first build makes it there, and `umtool storage move-out <project>` (or `--all`) moves an existing one, copying it, checking the copy and only then leaving the link; `--dry-run` says how much would move, and `umtool storage move-back` brings one home. The manifest, its revisions, notes and sources stay where they are, and nothing in umtool reads a project differently. When the drive is not mounted, a build or source check refuses and says so instead of starting a new folder on the main disk; umtool never creates the media directory itself. `umtool storage` lists where each project's `out/` is. With `UMTOOL_MEDIA_DIR` unset nothing changes.
+- **umtool can move a report's cut clips and share batches to the media drive, one project at a time.** The **Move deliverables** button in a report's deliver panel, or `umtool storage deliverables <project> --to media` (`--to local` to bring them back, `--dry-run` to see what would move), moves the project's `clips/` folder and every `share-…` batch folder to the same path under `UMTOOL_MEDIA_DIR`, checks each copy, leaves a link in its place, and then records the choice in the report's `video.manifest.json` as `"storage": { "deliverables": "media" }`. From then on a cut or a new batch is made on the media drive, and everything that opens `clips/<id>.mp4` keeps working through the link. The panel shows where the deliverables are, and the button is disabled, with the reason beside it, while a cut or a batch is running, while the drive is not mounted, or (to the media drive) when `UMTOOL_MEDIA_DIR` is not set. When the drive is not mounted, a cut or a batch refuses and says so instead of starting again on the main disk, and `umtool check` reports it as blocking, for the render folder too. Nothing moves until you ask: without the switch, a report's deliverables stay in the project.
- **umtool's cache moves to `~/.cache/archilyzer/umtool`** (`$XDG_CACHE_HOME/archilyzer/umtool` when that is set, or `UMTOOL_CACHE_DIR`). It was inside the song project's data folder, so it followed that folder onto whatever drive it was on. Run `umtool index` once after updating to rebuild the project index in its new place; umtool works without it, only slower, and the rest of the cache is remade as it is needed. `umtool doctor` now also shows the reports, media and cache folders, and the old cache folder while it is still there; it can be deleted.
- **umtool's report videos keep every clip's sound on its picture.** In a crossfaded cut each clip's audio was placed by the audio's own length and its picture by the picture's, and an encoded clip's audio is routinely a few to twenty milliseconds shorter or longer than its video, so the sound drifted further ahead clip by clip: by the end of a seventeen-clip cut it was a third of a second early, and two seconds on one with title and sources cards. Each clip's sound is now padded or trimmed to exactly its picture's length before the crossfade. Every crossfaded report video changes when it is rebuilt, and is in sync; a hard-cut video was not affected.
- **umtool's report videos can wear an on-screen deck: one panel under the footage for the whole cut, with a pip timeline, a title per clip, its source and date, and its QR.** A report manifest whose `render` says `"chrome": { "engine": "hyperframes", "layout": "deck" }` scales the footage into a box above a 190 px panel (both sizes are settings) and draws, over the whole cut, one unlabelled pip per clip on a track that fills as the cut plays, the clip's own title from `onscreen.title`, a subtitle naming the recording and its date (the channel too when the cut spans more than one; `onscreen.subtitle` replaces it), and the clip's QR. At each clip change the marker travels to the next pip and the title, subtitle and QR hand over; over a card the panel slides away and comes back after. The citation header, the corner QR and the section footer are not drawn on such a cut, and chapters take the clip's on-screen title. Every setting (sizes, spacing, date format, what the subtitle names, whether cards keep the panel, the motion's timings) is in `render.chrome.deck` and checked when it is saved; an unknown or out-of-range one is refused with a sentence saying why. The panel is rendered once per cut by HyperFrames (pinned to 0.8.24; `HYPERFRAMES_PKG` or `HYPERFRAMES_BIN` override it) and reused until its text or settings change. `build-video.mjs --chrome-only` redraws it over the built segments without rebuilding or fetching anything, `--no-chrome` builds the framed cut without it, and `--chrome-preview <at> <dur>` renders a short window. In umtool, the report page has an **On-screen** section — a switch, the settings, a table of every entry's title and subtitle with the automatic subtitle as its placeholder and a character counter, a live preview with a scrubber, a true still, **Re-render on-screen** and the built video — and the clip bench has on-screen title and subtitle fields with the panel previewed over the clip. The deck changes nothing, byte for byte, in a cut whose manifest has no `render.chrome`.
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -8357,3 +8357,29 @@ phase deletes from the destination.
`test-transcripts/` (`common/social/__fixtures__/firefoxCookieStore.ts`, imported by relative path)
and points `cookiesFromBrowser` at it as `firefox:<abs path>` — a profile PATH in the spec, so no
home directory is searched.
+
+## umtool's deliverables switch (verified 2026-10-01, branch `r17/umtool-deliverables`)
+
+- **`clips/` and `share-*/` of a report project may be links** into `UMTOOL_MEDIA_DIR` — per project,
+ when `video.manifest.json` says `"storage": { "deliverables": "media" }` (absent = local). `out/` is
+ the other directory that may be a link, and it follows `UMTOOL_MEDIA_DIR` on its own (slice U1).
+ Readers open `clips/<id>.mp4` by path through the link; a dirent `isDirectory()` filter over a
+ project hides a moved one (`listBatches` follows links for that reason).
+- **One writer for the value:** `updateStorage` in `umtool/lib/report/manifest.mjs`, called only by
+ `moveDeliverables` (`umtool/lib/report/storage.mjs`) after every directory has moved. A no-op is not
+ rewritten.
+- **A missing deliverable directory is made by `deliverableDir(project, name)`** — never a bare
+ `mkdir` (`cut.mjs`, `buildShareBatch`). Under `"media"` it is a link to a new mirror directory; the
+ media root is never created; a dangling link, a cut move's leftover (`<name>.moved-*`,
+ `<name>.incoming`), a bad value, or `"media"` in a process without `UMTOOL_MEDIA_DIR` refuse.
+- **`umtool storage deliverables <project> --to media|local`** is the move (the bench's "Move
+ deliverables" runs it as a job). It, and `move-out`/`move-back`, refuse a project while
+ `lib/report/busy.mjs` sees a pipeline process in it: a script argument under the project directory
+ (the build steps), a `--project` value equal to the project's id or name or resolving to its
+ directory (how the app runs `cut-from-cache.mjs` and `share-batch.mjs` — matched whole, never by
+ prefix), or `apply-manifest.py`/`build.py` by cwd. A hand-run command that is none of these, and
+ another machine, are not seen. `umtool check` reports `storage-unreachable` (blocking: an `out`,
+ `clips` or `share-*` link whose target is gone) and `storage-mismatch` (open).
+- The e2e app server has no media root (`UMTOOL_MEDIA_DIR=` in `playwright.config.ts`); specs give
+ their CLI one (`<fixture>-media`), so `deliverables.spec.ts` shows an app cutting through a link it
+ did not make and refusing a new batch under `"media"`.
diff --git a/plans/release-17.md b/plans/release-17.md
@@ -564,6 +564,174 @@ media tier; a dangling link still reads as "no build" there (`umtool check` lear
mix render into it lands on the media root through the link (the write check is lexical; that matches
the semantics).
+### Slice U2, as shipped — umtool's deliverables switch (2026-10-01)
+
+Branch `r17/umtool-deliverables` off `main` `bb877f93` (slice U1 merged), `main` `1d5c33bf` (slice
+XP) merged in before the gates, worktree `~/Projects/r12-source-mirror` (editor 5101, test 5111,
+export 5110), one Opus implementer. Scratch files `U2-*` in the job's `tmp`. The ruling is the plan's:
+deliverables (`clips/`, `share-*/`, final mp4s) move per project by a switch "like channels";
+manifests, `revisions/`, the caches and the cue cache stay put; final mp4s travel with `out/` (U1).
+
+**What it does.**
+- **The switch.** `video.manifest.json` gains `"storage": { "deliverables": "local" | "media" }`
+ (absent = local). Its one writer is `updateStorage` in `lib/report/manifest.mjs` (the manifest lock,
+ tmp + rename, the mtime token; any other key under `storage` kept; a no-op is not rewritten, so no
+ open bench page's token goes stale). Its one caller is `moveDeliverables`, after every directory has
+ moved.
+- **`lib/report/storage.mjs`.** `ensureOutDir`'s body is generalised to a name (`ensureProjectDir`;
+ `ensureOutDir` keeps its behaviour and sentences). New: `DELIVERABLES_MODES`, `isDeliverableName`
+ (`clips` or `share-<x>`, never a move's leftover), `deliverablesModeOf` / `deliverablesMode` (a plain
+ JSON read, so the module stays free of the writer's imports), **`deliverableDir(project, name)`** —
+ an existing directory or link is used as it is; absent and local → a directory; absent and media →
+ `mkdir -p <mirror>/<name>` + an absolute link, the media root stat'd and never created; refused,
+ creating nothing, on a dangling link, a cut move's leftover (the sentence names `umtool storage
+ deliverables <project> --to …`), a bad value, a project outside the reports root, or `"media"` in a
+ process with no `UMTOOL_MEDIA_DIR` (never silently local: a batch on the wrong drive is a split
+ nobody chose); `deliverableNames` (clips first, every `share-*` dir or link, and a name present only
+ as a leftover so a move finishes it); `deliverablesState`; `deliverablesProblems(state, name)` (the
+ sentences that stop a write: a dangling link, a leftover, and for a directory not there yet a switch
+ this process cannot honour); **`moveDeliverables(project, to, { writeMode })`** — U1's movers over
+ each name, then the switch only when none failed; idempotent; a cut move resumes; `--dry-run`
+ measures. `finishCommand` names the right command in the leftover sentence (`out` keeps U1's).
+- **The writers.** `cut.mjs` makes `clips/` through `deliverableDir` (a refusal is `reason: "storage"`,
+ before any ffmpeg; tmp and final still share the directory). `buildShareBatch` makes `share-<name>/`
+ through it, and first refuses on `deliverablesProblems` — a batch that cannot read an earlier batch
+ would ship its clips again. A batch name shaped like a leftover (`x.incoming`, `x.moved-…`) is
+ refused by `share-batch.mjs` and the route.
+- **The readers.** `listBatches` lists a `share-*` link like a directory, a dangling one as
+ `dangling` with no ids, and no leftover; `sharedIdsIn`'s walk follows a link to a directory (six
+ levels at most). `deliverStateOf` carries `storage` (the state, `cutBlocked`, `shareBlocked`). The
+ mix picker (`lib/media.ts`) follows `clips` and `share-*` links into the media root as it does `out`.
+ `kinds.mjs` needed nothing: U1's `SKIP_DIRS`/`SKIP_PREFIXES` already keep the walk out of both.
+- **`umtool check`** (through `reportDecisions`): `storage-unreachable`, **blocking** — an `out`,
+ `clips` or `share-*` link whose target is gone ("is the media drive mounted?"); `storage-mismatch`,
+ open — a cut move's leftover (with the command that finishes it), a directory while the switch says
+ media, a link while it says local, or media with no `UMTOOL_MEDIA_DIR` in this process; a value
+ that is neither is `manifest-invalid` on `storage.deliverables`, blocking.
+- **CLI.** `umtool storage deliverables <project> --to media|local [--dry-run] [--json]`: one line per
+ directory and the switch's before → after; exit 1 on any failure (the switch then stays). Refused
+ while a pipeline process works in the project (`lib/report/busy.mjs` after the review, H1): a script
+ argument under the project directory (U1's scan, how the build steps name a project), a `--project`
+ value equal to the project's id or name or resolving to its directory (how the app runs a cut and a
+ share batch; whole values, never a prefix), and the report's own `apply-manifest.py` and `build.py`
+ by working directory. Not seen: a hand-run command that is none of these, another machine. From the
+ bench the move is itself a job, so the app's one-job-at-a-time rule keeps its cuts and batches out
+ as well. `umtool storage [<project>]` lists each report's deliverables and switch.
+- **Bench.** The deliver panel says where the deliverables are (`data-deliverables` = `local` |
+ `media` | `invalid`; one `data-deliverable="<name>"` with `data-deliverable-state` per directory) and
+ gains **"Move deliverables to media|local"** (`data-action="deliver-move"`, `data-move-to`), the
+ `move` action of `/api/report/deliver`, a job (`driver.mjs` `moveDeliverablesSteps` runs the CLI).
+ It is disabled, the reason in `data-move-reason`, while a job of the project runs ("… is running —
+ move when it has finished"), while a deliverable link dangles, and toward media when the app has no
+ `UMTOOL_MEDIA_DIR`. Cut and share are disabled with `deliverablesProblems`' sentences
+ (`data-deliver-blocked`), and the route refuses them with 409.
+- **e2e.** `deliverables-fixture` (d01/d02 cuttable from a cached window; d03 cut and shipped in
+ `share-first`) and `deliverables.spec.ts` (5, serial): the CLI move (dry run first; links; the switch
+ set; again → already, not rewritten; `check` silent); the app (no media root) cuts THROUGH the
+ `clips/` link while the move button is disabled with "is running"; the app refuses a new batch
+ (button disabled with the reason, route 409, nothing made) and the CLI with the media root makes
+ `share-second` as a link, reading `share-first` through its own (d03 not shipped again); the root
+ unplugged → `check` exits 1 with `storage-unreachable` on all three links, the panel and the route
+ refuse, nothing made, the root not recreated; the bench's button brings everything home (the app,
+ untiered, leaves the media side and says "left in place"), the switch reads local, and the button
+ then offers media, disabled, naming `UMTOOL_MEDIA_DIR`.
+
+**Commits**
+
+| Commit | What |
+|---|---|
+| `a61b60e2` | `umtool:` the switch — `updateStorage`, `deliverableDir` and the deliverables helpers in `storage.mjs`, `cut.mjs`/`buildShareBatch` through it, `listBatches`/`sharedIdsIn` follow links, `umtool check`'s storage decisions, `umtool storage deliverables`, the busy scan's `apply-manifest.py`/`build.py`, the picker; `deliverables.test.mjs` (14) |
+| `ec7c84f3` | `umtool:` "Move deliverables" on the bench, the route's `move` action and refusals, `deliverables-fixture` + `deliverables.spec.ts`, `docs/folders.md` + `docs/cli.md`, the `[Unreleased]` bullet |
+| `eb9bb64e` | merge of `main` `1d5c33bf` (slice XP) — clean, nothing under `umtool/` |
+| `875dbff1` | `umtool:` the spec's batch count includes the id its LIST.md says was already shared (run 1's one failure) |
+| `f316fd46` | `plans:` this section; FACTS "umtool's deliverables switch" |
+| `3ff84ad5` | `umtool:` review H1 (the busy scan by `--project`, `lib/report/busy.mjs`) and N1; +4 unit cases |
+| `f52f1eb1` | `umtool:` review N2 (the route's 409 for a move during a cut) |
+| `1d176e2a` | `plans:` the review subsection, the corrected busy-scan claim |
+| `ca30be18` | merge of `main` `fa24a57f` (slice T1, the deck's finale-dip branch) — clean; `driver.mjs` and `make-fixture.mjs` merged without conflict, both sides kept |
+| this commit | `plans:` the gates after the merge |
+
+#### Gates (logs `$T/U2-*`)
+
+- **tsc** (all workspaces) clean at `ec7c84f3` and on the merged tree.
+- **common:** 2,501/2,501 on the merged tree (no common code touched). **Editor unit:** skipped — no
+ editor code touched (only `editor/CHANGELOG.md`).
+- **test:scripts:** 408 tests: 406 passed, 1 skipped (LIVE), 1 failed — `queue-lock.test.mjs`'s
+ "QUEUE_LOCK_HELD passes straight through" (waited 1,065 ms at a load average of 16–22, the timing
+ case U1 met); `node --test scripts/queue-lock.test.mjs` alone right after: **11/11**.
+ `deliverables.test.mjs` **14/14**, `storage.test.mjs` 24/24, `next-build-trace.test.mjs` **10/10**
+ against the fresh build below.
+- **The capped umtool build with the corpus linked** (worktree `transcripts/` set aside, `ln -sT`,
+ 77 channels visible, `systemd-run --scope -p MemoryMax=5G -p MemorySwapMax=0`, `timeout -s KILL
+ 240`, the link removed and the directory put back), on the merged tree: **exit 0, 32 s, 0.82 GB**;
+ with `UMTOOL_MEDIA_DIR` set to a scratch directory: **exit 0, 35 s, 0.82 GB**. `.nft.json` entries
+ 39,809 each, **identical** (`diff` empty), none naming `transcripts`, the scratch media root or
+ `.e2e-song`.
+- **Numbers tool:** none.
+- **umtool e2e** (`SONG_DIR=~/reports/quartering-uh-song/data`, `UMTOOL_MEDIA_DIR=` in the shell,
+ `pnpm --filter umtool run e2e …` from the worktree root, queued; lists in `$T/U2-specs*.txt`):
+
+ | Run | At | Specs | Result |
+ |---|---|---|---|
+ | 1 | `eb9bb64e` | `storage`, `deliverables`, `deliver`, `clip-bench`, `report-fetch-via-editor`, `projects` | 91 passed, 1 failed, 2 did not run (serial), 7.7 min of tests (21 min with the queue and the fixture build) — `deliverables:143` expected the batch count `(2)`; the panel said `(3)`, which is right (fixed in `875dbff1`) |
+ | 2 | `875dbff1` | `deliverables` | **5 passed**, 0 failed, 2.1 min of tests (21 min with the queue) |
+
+ `projects.spec` is the one extra beyond the prompt's list: it reads the decisions inbox, which gained
+ two kinds.
+
+#### Found and left
+
+- **The app's decisions index is signed without the deliverables** (`reportSignature`: the manifest,
+ `out`, `availability.json`, `revisions`): a drive unmounted under an unchanged manifest leaves a
+ cached inbox row set stale until something else changes. `umtool check` computes afresh. Signing
+ `clips`/`share-*` would stat through the links on every index read — on a stalled drive, the
+ hang U1 left for `out`. Left with U1's note.
+- **A dangling `clips/` still reads as "nothing cut" to `deliverStateOf`'s counts** (the panel's
+ `need-cut` and per-section numbers); the panel now says why beside them and every write refuses.
+- **The app moves home without a media root of its own** (as in the e2e): the media copy is left in
+ place and named, as U1's L3 rule says. The real app runs with `UMTOOL_MEDIA_DIR` set, where the
+ project's own mirror is removed.
+- **U1's two items for this slice** are done: `listBatches`/`sharedIdsIn` follow a `share-*` link;
+ `umtool check` reports a dangling `out/` (and `clips/`, `share-*`).
+
+#### Deviations from the plan
+
+- The plan's files `report-to-video/{cut,deliver,check}.mjs` are `lib/report/cut.mjs`,
+ `lib/report/deliver.mjs` and `bin/umtool.mjs`'s `check` (through `lib/projects/report.mjs`'s
+ `reportDecisions`, so `umtool check` and the decisions inbox say the same).
+- The movers' loop and the switch are one function in `storage.mjs` (`moveDeliverables`) taking the
+ writer as `writeMode`, so `storage.mjs` — imported by the app's routes — does not import the
+ manifest writer's dependencies.
+- The bench move is the deliver route's fifth action and a job, not a route of its own: the job
+ registry is what keeps a cut or a batch from running under it.
+- `lib/media.ts` (the mix picker) and `driver.mjs` (the step builder) are touched beyond the owned
+ list, one function each.
+- A FACTS section was added ("umtool's deliverables switch"); no existing fact changed.
+
+`[Unreleased]` (`editor/CHANGELOG.md`): "umtool can move a report's cut clips and share batches to
+the media drive, one project at a time."
+
+#### Review (SHIP AFTER FIXES) and the fixes
+
+| Finding | Fix |
+|---|---|
+| H1 (HIGH) — the busy scan counted a pipeline script only when an argument was the absolute project path, but the app runs `cut-from-cache.mjs` and `share-batch.mjs` as `--project <id>`: a shell-run move did not see the app's cuts and batches, the two writers it exists to wait for; the docs, FACTS, this record and the CLI comment claimed it did | `3ff84ad5`: the scan moves to `lib/report/busy.mjs` (imported by the CLI only) and also counts a script whose `--project` value equals the project's id or name, or resolves against the process's cwd to the project directory — whole values, never a prefix; `move-out`/`move-back` get it too. Tests: `namesProject`'s cases, a dummy process carrying `cut-from-cache.mjs --project <id>` is seen and `<id>-other` is not, the CLI refuses as `busy` with its pid and moves once only `<id>-other` runs. The claim is corrected in the CLI comments, `docs/folders.md`, `docs/cli.md`, FACTS and the "What it does" bullet above: what it sees, and that a hand-run command that is none of these scripts, and another machine, are not seen |
+| N1 — a link under an absent key (absent means local) was not flagged, only under an explicit `"local"` | `3ff84ad5`: a `clips`/`share-*` link into the media root under no key is `storage-mismatch` too (a hand-made link elsewhere under no key is not); a unit case runs `umtool check` on no key, `local`, `media` |
+| N2 — the route's 409 for a move while a job runs was asserted nowhere | `f52f1eb1`: `deliverables.spec`'s cut test posts a `move` during the cut and expects 409 "a job is already running" |
+| L1 — the decisions inbox and the deliver panel stat through every `out`/`clips`/`share-*` link, so a stalled media drive blocks them (U1's class) | left, a follow-up (below) |
+
+**Gates after the fixes:** umtool tsc clean. `deliverables.test.mjs` **18/18** (+4), `storage.test.mjs`
+24/24. umtool e2e at `f52f1eb1`, `deliverables` + `storage`: **10 passed**, 0 failed, 1.2 min of tests (10 min with the queue).
+
+**Gates after merging `main` `fa24a57f`** (`ca30be18`; log `$T/U2-regate.log`): umtool tsc clean;
+`deliverables.test.mjs` + `storage.test.mjs` + `driver.test.mjs` **45/45**; the capped umtool build
+with the corpus linked (77 channels; the worktree's `transcripts/` set aside and restored): **exit 0,
+30 s, 0.80 GB**, no trace entry naming `transcripts/` or `.e2e-song`, `next-build-trace` 10/10; umtool
+e2e `deliverables` + `storage`: **10 passed**, 0 failed, 41 s of tests (6 min with the queue).
+
+**Follow-up (L1):** with a stalled (not absent) media drive, the inbox's decisions pass and the deliver
+panel block on the stat through each link, as U1's project summary does for `out`.
+
### Slice XP, as shipped — X posts are private (2026-10-01)
Branch `r17/x-posts-private` off `main` `90bd8384`, worktree `~/Projects/r13-lows-export` (editor 5501,
diff --git a/umtool/app/api/report/deliver/route.ts b/umtool/app/api/report/deliver/route.ts
@@ -2,10 +2,12 @@ import { cancelJob, getJob, jobView, runningJob, startJob } from "@/lib/jobs";
import {
applyRulingsSteps,
cutSteps,
+ moveDeliverablesSteps,
rebuildReportSteps,
shareBatchSteps,
} from "@/lib/report/driver.mjs";
-import { deliverStateOf } from "@/lib/report/deliver.mjs";
+import { SHARE_PREFIX, deliverStateOf } from "@/lib/report/deliver.mjs";
+import { deliverablesProblems, isDeliverableName } from "@/lib/report/storage.mjs";
import { readClipDetail } from "@/lib/projects/report.mjs";
import { projectRef } from "@/lib/projects";
import type { Step } from "@/lib/trim";
@@ -19,6 +21,9 @@ export const dynamic = "force-dynamic";
// share a batch of those, in three encodes, for whoever is writing
// apply the project's own apply-manifest.py, then `umtool corrections`
// rebuild build.py, once per content variant present
+// move the deliverables (clips/, every share-*) to the media root or
+// back, then the manifest's storage.deliverables (release 17) --
+// a job like the others, so nothing cuts or shares while it moves
//
// Each is a JOB, and jobs.ts allows exactly one at a time. That is not a
// limitation to work around here: every one of these reads or writes the same
@@ -31,7 +36,7 @@ export const dynamic = "force-dynamic";
// directory's own contents, which is what stops "rebuild" from being able to
// run an arbitrary python file.
-const ACTIONS = ["cut", "share", "apply", "rebuild"] as const;
+const ACTIONS = ["cut", "share", "apply", "rebuild", "move"] as const;
type Action = (typeof ACTIONS)[number];
export async function GET(request: Request) {
@@ -96,6 +101,11 @@ export async function POST(request: Request) {
let kind = "";
if (action === "cut") {
+ // A clips/ on a drive that is not there reads as "nothing cut", and the
+ // list below would be every confirmed clip: refuse with the reason instead.
+ if (state.storage.cutBlocked.length) {
+ return Response.json({ error: state.storage.cutBlocked.join("; "), ok: false }, { status: 409 });
+ }
// The list is the SERVER's: confirmed, no mp4, and a window on this disk
// that holds it. A client-supplied list could name a clip nobody judged.
const ids = state.needCut.map((c: { id: string }) => c.id);
@@ -109,12 +119,16 @@ export async function POST(request: Request) {
kind = `cut ${project.id} (${ids.length} clips)`;
} else if (action === "share") {
const name = String(body.name ?? state.nextName);
- if (!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(name)) {
+ if (!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(name) || !isDeliverableName(`${SHARE_PREFIX}${name}`)) {
return Response.json(
- { error: "a batch name is letters, digits, dot, dash and underscore" },
+ { error: "a batch name is letters, digits, dot, dash and underscore, not ending .incoming or .moved-…" },
{ status: 400 },
);
}
+ const blocked = deliverablesProblems(state.storage, `${SHARE_PREFIX}${name}`);
+ if (blocked.length) {
+ return Response.json({ error: blocked.join("; "), ok: false }, { status: 409 });
+ }
if (!state.candidates.length) {
return Response.json(
{ error: "nothing to ship: every confirmed clip is already shared, or not cut yet" },
@@ -150,6 +164,19 @@ export async function POST(request: Request) {
}
steps = applyRulingsSteps(project);
kind = `apply rulings ${project.id}`;
+ } else if (action === "move") {
+ const to = String(body.to ?? "");
+ if (to !== "media" && to !== "local") {
+ return Response.json({ error: "move needs `to`: media or local" }, { status: 400 });
+ }
+ if (to === "media" && !state.storage.tiered) {
+ return Response.json(
+ { error: "UMTOOL_MEDIA_DIR is not set in umtool's environment — there is no media root to move to" },
+ { status: 400 },
+ );
+ }
+ steps = moveDeliverablesSteps(project, to);
+ kind = `move deliverables ${project.id} to ${to}`;
} else {
if (!state.hasBuildScript || !state.variants.length) {
return Response.json(
diff --git a/umtool/bin/share-batch.mjs b/umtool/bin/share-batch.mjs
@@ -13,6 +13,7 @@
import process from "node:process";
import { resolveProject } from "../lib/projects/core.mjs";
import { buildShareBatch } from "../lib/report/deliver.mjs";
+import { isDeliverableName } from "../lib/report/storage.mjs";
const argv = process.argv.slice(2);
const val = (flag) => {
@@ -26,8 +27,10 @@ if (!projectArg || !name) {
console.error("usage: share-batch.mjs --project <id> --name <batch name>");
process.exit(2);
}
-if (!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(name)) {
- console.error(`"${name}" is not a batch name: letters, digits, dot, dash and underscore`);
+if (!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(name) || !isDeliverableName(`share-${name}`)) {
+ // The second test: `x.incoming` and `x.moved-<ts>` are what a cut move of
+ // share-x leaves behind (release 17), so a batch may not be called that.
+ console.error(`"${name}" is not a batch name: letters, digits, dot, dash and underscore, not ending .incoming or .moved-…`);
process.exit(2);
}
diff --git a/umtool/bin/umtool.mjs b/umtool/bin/umtool.mjs
@@ -30,12 +30,13 @@
// or the media root is set and not there
// umtool storage [move-out|move-back <project>|--all] [--dry-run] [--json]
// where each project's out/ lives; move it
+// umtool storage deliverables <project> --to media|local [--dry-run] [--json]
+// move clips/ and every share-*/, then set the switch
// umtool snapshot <project> [--label L] copy the manifest into revisions/
// umtool diff <project> <snapshot> what changed since that snapshot
// umtool export <project> --format toc-bbcode|toc-markdown|description|chapters [--variant V]
// umtool check-sources [<project>…] prints the re-check chain
import process from "node:process";
-import { readdirSync, readFileSync } from "node:fs";
import {
PROJECT_KINDS,
REPORTS_ROOT,
@@ -58,14 +59,24 @@ import { createSnapshot, listSnapshots, readSnapshot } from "../lib/report/snaps
import { diffManifests, formatChange } from "../lib/report/manifest-diff.mjs";
import { EXPORT_FORMATS, exportProject } from "../lib/report/export.mjs";
import path from "node:path";
-import { updateClip } from "../lib/report/manifest.mjs";
+import { updateClip, updateStorage } from "../lib/report/manifest.mjs";
import { hms } from "umtool-report-to-video/attribution";
import { buildSteps, checkSourcesSteps, PRESETS } from "../lib/report/driver.mjs";
import { openIndex, signRecord } from "../lib/projects/index-db.mjs";
+import { pipelineProcessesFor } from "../lib/report/busy.mjs";
import { probeTools } from "../lib/tools.mjs";
import { scaffoldReportVideo } from "../lib/projects/scaffold.mjs";
import { CACHE_DIR, INDEX_DIR, MEDIA_ROOT, MEDIA_TIERED, OLD_CACHE_DIR } from "../lib/paths.mjs";
-import { measureTree, mediaRootProblem, moveDirToLocal, moveDirToMedia, outDirState, pathState } from "../lib/report/storage.mjs";
+import {
+ deliverablesState,
+ measureTree,
+ mediaRootProblem,
+ moveDeliverables,
+ moveDirToLocal,
+ moveDirToMedia,
+ outDirState,
+ pathState,
+} from "../lib/report/storage.mjs";
const argv = process.argv.slice(2);
const cmd = argv.find((a) => !a.startsWith("-")) ?? "help";
@@ -416,58 +427,55 @@ async function cmdDoctor() {
// umtool storage every project's out/: dir | link | DANGLING | none
// umtool storage move-out <p>|--all out/ to the media root, a link left in its place
// umtool storage move-back <p>|--all out/ back to a real directory in the project
+// umtool storage deliverables <p> --to media|local
+// clips/ and every share-*/ to the media root or
+// back, then storage.deliverables in the manifest
// --dry-run measure and say; change nothing
//
// The movers are lib/report/storage.mjs's, which `umtool` and the app share.
-// Run a move when nothing is building: the app's jobs live in its memory, so
-// this cannot see them -- the verify refuses when the tree keeps changing, but
-// a write in the last instant before the swap would be lost with the parked copy.
+// A project is skipped (move-out/back) or refused (deliverables) while a
+// pipeline process works in it -- lib/report/busy.mjs says what that scan sees
+// (the app's build steps, cuts and share batches, which are all processes, and
+// the report's own scripts) and what it cannot (a hand-run command that is none
+// of them, another machine). The verify refuses a tree that keeps changing, but
+// a write it cannot see, in the last instant before the swap, would be lost
+// with the parked copy.
// ---------------------------------------------------------------------------
-/**
- * The pids of report-pipeline processes whose command line names this project
- * (its manifest, its out/, or the directory itself). Linux /proc; elsewhere,
- * none. Cheap and coarse: it sees the pipeline's scripts, not the app's
- * in-process deck previews.
- */
-function pipelineProcessesFor(projectDir) {
- const SCRIPTS = /(build-video|check-availability|render-cards|compose-chrome|verify-build|fetch-via-editor|resolve-windows|cut-from-cache|share-batch)\.mjs/;
- const pids = [];
- let entries = [];
- try {
- entries = readdirSync("/proc").filter((n) => /^\d+$/.test(n));
- } catch {
- return pids;
- }
- for (const pid of entries) {
- if (Number(pid) === process.pid) continue;
- let args;
- try {
- args = readFileSync(`/proc/${pid}/cmdline`, "utf8").split("\0");
- } catch {
- continue;
- }
- if (!args.some((a) => SCRIPTS.test(a))) continue;
- if (args.some((a) => a === projectDir || a.startsWith(projectDir + "/"))) pids.push(Number(pid));
- }
- return pids;
-}
-
async function cmdStorage() {
const sub = positional[0];
const dryRun = has("--dry-run");
const log = (m) => (json ? console.error(m) : console.log(m));
+ if (sub === "deliverables") return cmdStorageDeliverables(dryRun, log);
if (sub !== "move-out" && sub !== "move-back") {
// `umtool storage`, `umtool storage <project>`, `umtool storage status [<project>]`.
const one = sub === "status" ? positional[1] : sub;
const refs = one ? [await pick(one)] : await projectRefs();
const rows = [];
- for (const p of refs) rows.push({ id: p.id, ...(await outDirState(p.dir)) });
+ for (const p of refs) {
+ const row = { id: p.id, ...(await outDirState(p.dir)) };
+ // A report video's deliverables, when it has any or its switch is set.
+ const d = p.kind === "report-video" ? await deliverablesState(p.dir) : null;
+ if (d && (d.dirs.length || d.value !== undefined)) {
+ row.deliverables = { mode: d.mode, value: d.value, dirs: d.dirs };
+ }
+ rows.push(row);
+ }
if (json) return out({ media: { path: MEDIA_ROOT, tiered: MEDIA_TIERED }, projects: rows });
console.log(MEDIA_TIERED ? `media root ${MEDIA_ROOT}` : "media root unset (UMTOOL_MEDIA_DIR): out/ stays in each project");
+ const word = { absent: "none", dir: "dir", link: "link", dangling: "DANGLING", other: "OTHER" };
for (const r of rows) {
- const what = { absent: "none", dir: "dir", link: "link", dangling: "DANGLING", other: "OTHER" }[r.state] ?? r.state;
+ const what = word[r.state] ?? r.state;
console.log(`${what.padEnd(9)} ${r.id}${r.target ? ` -> ${r.target}` : ""}`);
+ if (r.deliverables) {
+ const d = r.deliverables;
+ console.log(
+ ` deliverables: ${d.mode ?? `INVALID ${JSON.stringify(d.value)}`}` +
+ (d.dirs.length
+ ? ` · ${d.dirs.map((x) => `${x.name} ${word[x.state] ?? x.state}${x.leftovers.length ? ` (+${x.leftovers.join(", ")})` : ""}`).join(" · ")}`
+ : ""),
+ );
+ }
}
return;
}
@@ -485,7 +493,7 @@ async function cmdStorage() {
for (const p of refs) {
// The app's jobs live in its memory, but the pipeline runs as processes:
// one whose command line names this project is building it now.
- const busy = pipelineProcessesFor(p.dir);
+ const busy = pipelineProcessesFor(p);
if (busy.length) {
results.push({ id: p.id, state: "busy", pids: busy });
if (!json) console.log(`${"busy".padEnd(12)} ${p.id} — a pipeline process is writing it (pid ${busy.join(", ")}); skipped`);
@@ -518,6 +526,67 @@ async function cmdStorage() {
if (failed) process.exit(1);
}
+/**
+ * `umtool storage deliverables <project> --to media|local [--dry-run]`: move
+ * clips/ and every share-* directory with the movers, then set storage.deliverables
+ * through the manifest's writer. Refused while a pipeline process works in the
+ * project (lib/report/busy.mjs): a build step naming its directory, a cut or a
+ * share batch whose `--project` is this project's id, name or directory, the
+ * report's own apply/build scripts by their working directory.
+ *
+ * What the refusal cannot see: a hand-run command that is none of those
+ * scripts (an ffmpeg writing into clips/), and another machine writing over a
+ * network share. The app's own jobs are processes and are seen; run from the
+ * bench, this is itself a job, so the app's one-job-at-a-time rule keeps its
+ * cuts and batches out as well. The movers' verify refuses a tree that keeps
+ * changing, but a write it cannot see, in the instant between the verify and
+ * the swap, would be lost with the parked copy.
+ */
+async function cmdStorageDeliverables(dryRun, log) {
+ const to = val("--to");
+ if (to !== "media" && to !== "local") die("where to? `umtool storage deliverables <project> --to media|local`");
+ const p = await pick(positional[1]);
+ if (p.kind !== "report-video") die(`${p.id} is a ${p.kind} project — deliverables are a report video's`);
+ const busy = pipelineProcessesFor(p);
+ if (busy.length) {
+ const msg = `${p.id}: a pipeline process is working in it (pid ${busy.join(", ")}) — nothing moved; run this when it has finished`;
+ if (json) out({ ok: false, busy: busy, error: msg });
+ else console.error(msg);
+ process.exit(1);
+ }
+ let r;
+ try {
+ r = await moveDeliverables(p.dir, to, {
+ dryRun,
+ log,
+ writeMode: (dir, mode) => updateStorage(dir, { deliverables: mode }),
+ });
+ } catch (e) {
+ if (json) out({ ok: false, error: e?.message ?? String(e) });
+ else console.error(e?.message ?? String(e));
+ process.exit(1);
+ }
+ if (json) out({ project: p.id, ...r });
+ else {
+ if (!r.results.length) console.log(`${p.id}: no clips/ and no share-*/ yet`);
+ for (const x of r.results) {
+ const size = x.bytes !== undefined ? ` ${x.files} file(s), ${mb(x.bytes)}` : "";
+ console.log(`${String(x.state === "failed" ? "FAILED" : x.state).padEnd(12)} ${x.name}${size}`);
+ if (x.error) console.log(` ${x.error}`);
+ if (x.mediaCopyLeft) console.log(` left in place: ${x.mediaCopyLeft} (not deleted; remove it by hand once checked)`);
+ }
+ const before = r.before ?? "local (unset)";
+ console.log(
+ dryRun
+ ? `\nstorage.deliverables: ${before} — would be ${to}; dry run, nothing changed`
+ : r.ok
+ ? `\nstorage.deliverables: ${r.written ? `${before} -> ${to}` : `${to} (unchanged)`}`
+ : `\nstorage.deliverables left at ${before}: ${r.results.filter((x) => x.state === "failed").length} FAILED — fix that and run it again`,
+ );
+ }
+ if (!r.ok) process.exit(1);
+}
+
async function cmdSnapshot() {
const p = await pick(positional[0]);
try {
@@ -623,6 +692,7 @@ function usage() {
" or the media root is set and not there; the roots, and a leftover old cache",
" umtool storage [<project>] where each project's out/ lives (dir, link, DANGLING)",
" umtool storage move-out|move-back <project>|--all [--dry-run]",
+ " umtool storage deliverables <project> --to media|local [--dry-run] clips/ + share-*/, then the switch",
" out/ to the media root (UMTOOL_MEDIA_DIR) and back; run when nothing is building",
" umtool snapshot <project> [--label L] copy the manifest into revisions/",
" umtool diff <project> <snapshot> what changed since that snapshot",
diff --git a/umtool/components/projects/DeliverActions.tsx b/umtool/components/projects/DeliverActions.tsx
@@ -37,6 +37,17 @@ type JobView = {
type Variant = { module: string; out: string };
+/** Where the deliverables are, and what would refuse a cut, a batch or a move (release 17). */
+type StorageView = {
+ mode: "local" | "media" | null;
+ /** UMTOOL_MEDIA_DIR is set in the app's environment. */
+ tiered: boolean;
+ /** Sentences that stop a move: a link whose drive is not there. */
+ blocked: string[];
+ cutBlocked: string[];
+ shareBlocked: string[];
+};
+
export default function DeliverActions({
project,
needCut,
@@ -47,6 +58,7 @@ export default function DeliverActions({
hasApply,
hasBuild,
variants,
+ storage,
}: {
project: string;
/** Confirmed, no mp4, and a window on this disk that holds it. */
@@ -60,6 +72,7 @@ export default function DeliverActions({
hasApply: boolean;
hasBuild: boolean;
variants: Variant[];
+ storage: StorageView;
}) {
const [job, setJob] = useState<JobView | null>(null);
const [error, setError] = useState<string | null>(null);
@@ -132,13 +145,27 @@ export default function DeliverActions({
const n = job?.steps.length ?? 0;
const k = Math.min((job?.stepIndex ?? 0) + 1, n);
+ // MOVE DELIVERABLES: the switch, as a job (release 17). Its direction is the
+ // other side of where they are; the reason it cannot run is said beside it
+ // rather than left to a 409 -- above all while a cut or a batch is writing
+ // into the very directories it would move.
+ const moveTo = storage.mode === "media" ? "local" : "media";
+ const moveReason = running
+ ? `${job!.kind} is running — move when it has finished`
+ : storage.blocked.length
+ ? storage.blocked.join("; ")
+ : moveTo === "media" && !storage.tiered
+ ? "UMTOOL_MEDIA_DIR is not set in umtool's environment — there is no media root to move to"
+ : null;
+
return (
<div data-deliver-actions="" className="space-y-2">
<div className="flex flex-wrap items-center gap-2">
<button
type="button"
data-action="deliver-cut"
- disabled={busy || running || !needCut}
+ disabled={busy || running || !needCut || storage.cutBlocked.length > 0}
+ title={storage.cutBlocked.join("; ") || undefined}
className={buttonVariants({ size: "sm" })}
onClick={() => void post("cut")}
>
@@ -156,7 +183,8 @@ export default function DeliverActions({
<button
type="button"
data-action="deliver-share"
- disabled={busy || running || !candidates}
+ disabled={busy || running || !candidates || storage.shareBlocked.length > 0}
+ title={storage.shareBlocked.join("; ") || undefined}
className={buttonVariants({ variant: "ghost", size: "sm" })}
onClick={() => void post("share", { name })}
>
@@ -201,6 +229,18 @@ export default function DeliverActions({
</button>
)}
+ <button
+ type="button"
+ data-action="deliver-move"
+ data-move-to={moveTo}
+ disabled={busy || moveReason !== null}
+ title={moveReason ?? undefined}
+ className={buttonVariants({ variant: "ghost", size: "sm" })}
+ onClick={() => void post("move", { to: moveTo })}
+ >
+ Move deliverables to {moveTo}
+ </button>
+
{running && (
<button
type="button"
@@ -215,6 +255,18 @@ export default function DeliverActions({
)}
</div>
+ {moveReason && (
+ <p data-move-reason="" className="text-[11px] text-[var(--color-dim)]">
+ Move deliverables: {moveReason}
+ </p>
+ )}
+
+ {(storage.cutBlocked.length > 0 || storage.shareBlocked.length > 0) && (
+ <p data-deliver-blocked="" className="text-[11px] text-[var(--color-bad)]">
+ {[...new Set([...storage.cutBlocked, ...storage.shareBlocked])].join("; ")}
+ </p>
+ )}
+
{notFetched > 0 && (
<p className="text-[11px] text-[var(--color-dim)]">
{notFetched} confirmed clip{notFetched === 1 ? " has" : "s have"} nothing cached to cut
diff --git a/umtool/components/projects/DeliverSection.tsx b/umtool/components/projects/DeliverSection.tsx
@@ -44,6 +44,19 @@ type State = {
incorrect: { id: string; correction: string; hits: { file: string; line: number; text: string }[] }[];
hasApplyScript: boolean;
hasBuildScript: boolean;
+ storage: Storage;
+};
+
+/** Where the deliverables live (lib/report/storage.mjs deliverablesState, release 17). */
+type Storage = {
+ mode: "local" | "media" | null;
+ value: unknown;
+ error?: string;
+ tiered: boolean;
+ mediaRoot: string | null;
+ dirs: { name: string; state: string; target?: string; leftovers: string[] }[];
+ cutBlocked: string[];
+ shareBlocked: string[];
};
export default async function DeliverSection({
@@ -137,9 +150,68 @@ export default async function DeliverSection({
hasApply={state.hasApplyScript}
hasBuild={state.hasBuildScript}
variants={state.variants.map((v) => ({ module: v.module, out: v.out }))}
+ storage={{
+ mode: state.storage.mode,
+ tiered: state.storage.tiered,
+ // What would stop a move: a link whose drive is not there, or a cut
+ // move's leftovers (a move refuses over both, and says so).
+ blocked: state.storage.dirs.flatMap((d) =>
+ d.state === "dangling"
+ ? [`${d.name}/ is a link to ${d.target}, which is not there — is the media drive mounted?`]
+ : [],
+ ),
+ cutBlocked: state.storage.cutBlocked,
+ shareBlocked: state.storage.shareBlocked,
+ }}
/>
</div>
+ {/* --- where the deliverables live (release 17) --------------------- */}
+ {/*
+ clips/ and every share-* live in the project, or on the media root
+ behind a link, as the manifest's storage.deliverables says. A reader
+ opens them by path either way; this line is the one place that says
+ which, and what each directory is right now.
+ */}
+ <p
+ data-deliverables={state.storage.mode ?? "invalid"}
+ className="mt-2 text-[11px] text-[var(--color-dim)]"
+ >
+ <span className="micro">deliverables — </span>
+ {state.storage.mode === "media"
+ ? "on the media root"
+ : state.storage.mode === "local"
+ ? "in the project"
+ : `storage.deliverables is not "local" or "media"`}
+ {state.storage.mode === "media" && state.storage.mediaRoot && (
+ <>
+ {" "}
+ (<code className="font-mono">{state.storage.mediaRoot}</code>)
+ </>
+ )}
+ {state.storage.dirs.map((d) => (
+ <span key={d.name} data-deliverable={d.name} data-deliverable-state={d.state}>
+ {" · "}
+ <code className="font-mono">{d.name}/</code>{" "}
+ <span
+ className={
+ d.state === "dangling" || d.leftovers.length ? "text-[var(--color-bad)]" : undefined
+ }
+ >
+ {d.state === "dangling"
+ ? "link, NOT THERE"
+ : d.state === "dir"
+ ? "here"
+ : d.state === "absent"
+ ? "moving"
+ : d.state}
+ {d.leftovers.length ? ` (a cut move left ${d.leftovers.join(", ")})` : ""}
+ </span>
+ </span>
+ ))}
+ {state.storage.dirs.length === 0 && " · nothing cut or shared yet"}
+ </p>
+
{/* --- confirmed, but no file yet ---------------------------------- */}
{state.needCut.length > 0 && (
<p data-need-cut="" className="mt-2 text-[11px] text-[var(--color-dim)]">
diff --git a/umtool/docs/cli.md b/umtool/docs/cli.md
@@ -18,7 +18,7 @@ decisions inbox cannot disagree about what is wrong with one.
|---|---|
| `ls [--kind --template --state --open --blocking --q --sort --json]` | the index, as text or JSON |
| `show <project> [--json]` | one project: summary, the cut, per-clip status, decisions |
-| `check [<project>] [--json]` | **exit 1 on anything blocking** |
+| `check [<project>] [--json]` | **exit 1 on anything blocking** — including an `out/`, `clips/` or `share-*/` link whose drive is not there (`storage-unreachable`) |
| `decisions [--json]` | the inbox |
| `folders [--json]` · `kinds [--json]` | the tree, the registry |
| `window <project> <clip> [--start S] [--end E] [--lock] [--lock-end] [--no-lock-end] [--note …]` | edit a window |
@@ -26,8 +26,9 @@ decisions inbox cannot disagree about what is wrong with one.
| `index [--rebuild] [--prune] [--since MS] [--json]` | the cache |
| `new <slug> [--from <report.md>\|<share URL>\|<channel>/<id>] [--site-origin URL] [--seed chapters]` | scaffold |
| `doctor [--json]` | which tools are on this machine and where the roots are; **exit 1** if the report pipeline is missing one, or `UMTOOL_MEDIA_DIR` is set and not there |
-| `storage [<project>] [--json]` | where each project's `out/` lives: dir, link, DANGLING, none |
+| `storage [<project>] [--json]` | where each project's `out/` lives: dir, link, DANGLING, none — and a report's deliverables (`clips/`, `share-*/`) and its switch |
| `storage move-out\|move-back <project>\|--all [--dry-run] [--json]` | `out/` to the media root (a link left behind) and back — copied, mirrored, verified first; run when nothing is building |
+| `storage deliverables <project> --to media\|local [--dry-run] [--json]` | `clips/` and every `share-*/` to the media root and back, then `storage.deliverables` in the manifest; refused while a build step, a cut or a share batch (by `--project` id, name or directory, as the app starts them) or the report's own scripts run in the project — a hand-run command that is none of these is not seen; **exit 1** on any failure (the switch is then left as it was) |
| `snapshot <project> [--label L]` | copy the manifest into `revisions/` |
| `diff <project> <snapshot>` | added / removed / moved / window / retyped, by entry id |
| `export <project> --format toc-bbcode\|toc-markdown\|description\|chapters [--variant V]` | the posting artifacts, from the build's chapter offsets |
diff --git a/umtool/docs/folders.md b/umtool/docs/folders.md
@@ -37,6 +37,50 @@ first writer (`lib/report/storage.mjs` `ensureOutDir`) or by
- `umtool storage` lists every project's `out` (dir, link, DANGLING, none);
`umtool doctor` names the roots and exits 1 when this one is set and missing.
+### Deliverables: `clips/` and every `share-*/` (a switch per project)
+
+A report's deliverables do not follow `UMTOOL_MEDIA_DIR` on their own. They move
+per project, by a switch in `video.manifest.json`:
+
+```json
+"storage": { "deliverables": "media" }
+```
+
+`"local"` (or no `storage` key) keeps them in the project; `"media"` puts them
+under the same project-relative path on the media root, behind a link, as `out`
+is. Only `lib/report/manifest.mjs`'s `updateStorage` writes the value, and only
+`umtool storage deliverables` (or the bench's **Move deliverables** button,
+which runs it as a job) calls that — after every directory has moved:
+
+- `umtool storage deliverables <project> --to media|local [--dry-run]` moves
+ `clips/` and each `share-*/` with the same copy-mirror-verify-swap movers, then
+ sets the switch. Run again, it finds each one `already` there and rewrites
+ nothing; a cut move is finished by running it again. It refuses while a
+ pipeline process works in the project (`lib/report/busy.mjs`, Linux `/proc`):
+ a build step whose arguments name the project's directory; a cut
+ (`cut-from-cache.mjs`) or share batch (`share-batch.mjs`) whose `--project`
+ is the project's id, its name, or a path that resolves to its directory —
+ whole values, never a prefix — which is how the app starts them; and the
+ report's own `apply-manifest.py` or `build.py`, by working directory. It
+ cannot see a hand-run command that is none of those scripts (an `ffmpeg` into
+ `clips/`) or another machine. From the bench the move is itself a job, so the
+ app's one-job-at-a-time rule keeps its cuts and batches out too.
+- A directory that does not exist yet is made where the switch says, by the
+ writer: a cut makes `clips/`, a batch its `share-<name>/`
+ (`storage.mjs` `deliverableDir`). Under `"media"` that is a link to a new
+ directory on the media root; the root itself is never created. An existing
+ directory or link is used as it is — a move is what changes where it lives.
+- Under `"media"`, a process with no `UMTOOL_MEDIA_DIR` refuses to make a new
+ one rather than make it in the project: a batch on the wrong drive is a split
+ nobody chose. Existing links keep working there.
+- A link whose drive is not there refuses every cut and batch (a batch that
+ cannot read an earlier batch would ship its clips again), and `umtool check`
+ reports it as `storage-unreachable`, blocking — for `out` too. A directory
+ that disagrees with the switch, or a cut move, is `storage-mismatch`, open.
+- References stay relative: `clips/<id>.mp4` resolves through the link.
+- Manifests, `revisions/`, the caches and the cue cache never move. Final mp4s
+ are in `out/` and travel with it.
+
## `CACHE_DIR`
`UMTOOL_CACHE_DIR`, else `$XDG_CACHE_HOME/archilyzer/umtool`, else
diff --git a/umtool/e2e/deliverables.spec.ts b/umtool/e2e/deliverables.spec.ts
@@ -0,0 +1,243 @@
+import { test, expect } from "@playwright/test";
+import { execFileSync, spawnSync } from "node:child_process";
+import { existsSync, lstatSync, readdirSync, readFileSync, readlinkSync, renameSync } from "node:fs";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+// ---------------------------------------------------------------------------
+// A report's deliverables behind the per-project switch (release 17, slice U2).
+//
+// `storage.deliverables` in the manifest says where clips/ and every share-*/
+// live: in the project ("local", the default) or on the media root behind a
+// link ("media"). `umtool storage deliverables` moves them and then sets it;
+// a cut or a batch makes a missing directory where it says; a reader opens
+// `clips/<id>.mp4` by path either way.
+//
+// Only THIS spec's CLI is given UMTOOL_MEDIA_DIR (make-fixture's media root,
+// shared with storage.spec.ts and reset every run). The app is not -- which is
+// what this spec leans on: it cuts THROUGH a linked clips/ without knowing a
+// media root exists, refuses a NEW batch it could only make on the wrong
+// drive, and brings everything home with the bench's own button.
+//
+// The tests run in order and hand the project's state on.
+// ---------------------------------------------------------------------------
+
+const HERE = path.dirname(fileURLToPath(import.meta.url));
+const UMTOOL = path.join(HERE, "..");
+const FIXTURE = path.join(UMTOOL, ".e2e-song");
+const MEDIA = `${FIXTURE}-media`;
+const UNPLUGGED = `${MEDIA}.unplugged`;
+const PROJECT = "reports/deliverables-fixture";
+const DIR = path.join(FIXTURE, PROJECT);
+const MIRROR = path.join(MEDIA, PROJECT);
+
+const env = {
+ ...process.env,
+ SONG_REPORTS_DIR: path.join(FIXTURE, "reports"),
+ SONG_DIR: path.join(FIXTURE, "data"),
+ CHANNELS_DIR: path.join(FIXTURE, "channels"),
+ UMTOOL_CACHE_DIR: path.join(FIXTURE, "cache"),
+ YTDLP_BIN: path.join(FIXTURE, "bin", "yt-dlp"),
+ UMTOOL_MEDIA_DIR: MEDIA,
+};
+const umtool = (args: string[]) =>
+ spawnSync("node", ["bin/umtool.mjs", ...args, "--json"], { cwd: UMTOOL, encoding: "utf8", env });
+const umtoolJson = (args: string[]) => {
+ const r = umtool(args);
+ expect(r.status, r.stderr).toBe(0);
+ return JSON.parse(r.stdout);
+};
+
+const isLink = (p: string) => lstatSync(p, { throwIfNoEntry: false })?.isSymbolicLink() === true;
+const isRealDir = (p: string) => lstatSync(p, { throwIfNoEntry: false })?.isDirectory() === true;
+const manifestStorage = () =>
+ JSON.parse(readFileSync(path.join(DIR, "video.manifest.json"), "utf8")).storage as
+ | { deliverables?: string }
+ | undefined;
+const seconds = (file: string): number =>
+ Number(
+ execFileSync("ffprobe", ["-v", "error", "-show_entries", "format=duration", "-of", "default=nw=1:nk=1", file])
+ .toString()
+ .trim(),
+ );
+
+async function waitForJob(page: import("@playwright/test").Page, timeout = 120_000) {
+ await expect(page.locator("[data-deliver-job]")).toHaveAttribute("data-deliver-state", /done|failed/, { timeout });
+ const state = await page.locator("[data-deliver-job]").getAttribute("data-deliver-state");
+ const log = (await page.locator("[data-deliver-log]").textContent()) ?? "";
+ return { state, log };
+}
+
+test.describe.configure({ mode: "serial" });
+
+test.afterAll(() => {
+ if (existsSync(UNPLUGGED) && !existsSync(MEDIA)) renameSync(UNPLUGGED, MEDIA);
+});
+
+test("the switch moves clips/ and every share-* to the media root, then says media", () => {
+ expect(isRealDir(path.join(DIR, "clips"))).toBe(true);
+ expect(manifestStorage()).toBeUndefined();
+
+ // A dry run measures and changes nothing -- not even the switch.
+ const dry = umtoolJson(["storage", "deliverables", PROJECT, "--to", "media", "--dry-run"]);
+ expect(dry.results.map((r: { name: string; state: string }) => [r.name, r.state])).toEqual([
+ ["clips", "would-move"],
+ ["share-first", "would-move"],
+ ]);
+ expect(dry.written).toBe(false);
+ expect(isRealDir(path.join(DIR, "clips"))).toBe(true);
+ expect(manifestStorage()).toBeUndefined();
+
+ const moved = umtoolJson(["storage", "deliverables", PROJECT, "--to", "media"]);
+ expect(moved.ok).toBe(true);
+ expect(moved.written).toBe(true);
+ for (const name of ["clips", "share-first"]) {
+ expect(isLink(path.join(DIR, name)), name).toBe(true);
+ expect(readlinkSync(path.join(DIR, name))).toBe(path.join(MIRROR, name));
+ }
+ expect(existsSync(path.join(MIRROR, "clips", "d03.mp4"))).toBe(true);
+ expect(manifestStorage()).toEqual({ deliverables: "media" });
+
+ // Again: nothing to do, and the manifest is not rewritten.
+ const again = umtoolJson(["storage", "deliverables", PROJECT, "--to", "media"]);
+ expect(again.results.map((r: { state: string }) => r.state)).toEqual(["already", "already"]);
+ expect(again.written).toBe(false);
+
+ // `umtool storage` names them; `umtool check` has nothing to say.
+ const status = umtoolJson(["storage", PROJECT]);
+ expect(status.projects[0].deliverables.mode).toBe("media");
+ expect(status.projects[0].deliverables.dirs.map((d: { state: string }) => d.state)).toEqual(["link", "link"]);
+ const check = JSON.parse(umtool(["check", PROJECT]).stdout);
+ expect(check.decisions.filter((d: { kind: string }) => d.kind.startsWith("storage-"))).toEqual([]);
+});
+
+test("the app cuts through the linked clips/, and the move waits for the cut", async ({ page, request }) => {
+ test.setTimeout(180_000);
+ await page.goto(`/browse/${PROJECT}`);
+ await expect(page.locator("[data-deliverables]")).toHaveAttribute("data-deliverables", "media");
+ await expect(page.locator('[data-deliverable="clips"]')).toHaveAttribute("data-deliverable-state", "link");
+ const move = page.locator('[data-action="deliver-move"]');
+ await expect(move).toHaveAttribute("data-move-to", "local");
+ await expect(move).toBeEnabled();
+
+ await page.locator('[data-action="deliver-cut"]').click();
+ await expect(page.locator("[data-deliver-progress]")).toContainText(/of 2|2 steps/);
+ // While the cut writes into clips/, the move that would carry clips/ away
+ // is disabled, and says why.
+ await expect(move).toBeDisabled();
+ await expect(page.locator("[data-move-reason]")).toContainText("is running — move when it has finished");
+ // And the route refuses a move asked for any other way while the cut runs.
+ const during = await request.post("/api/report/deliver", { data: { project: PROJECT, action: "move", to: "local" } });
+ expect(during.status()).toBe(409);
+ expect(((await during.json()) as { error: string }).error).toContain("a job is already running");
+
+ const { state, log } = await waitForJob(page);
+ expect(state, log).toBe("done");
+ // Written through the link: the files are on the media root, clips/ is still a link.
+ expect(isLink(path.join(DIR, "clips"))).toBe(true);
+ for (const id of ["d01", "d02"]) {
+ expect(seconds(path.join(MIRROR, "clips", `${id}.mp4`))).toBeCloseTo(3.0, 1);
+ }
+ expect(readdirSync(path.join(MIRROR, "clips")).filter((n) => n.startsWith("."))).toEqual([]);
+ await page.reload();
+ await expect(page.locator("[data-deliver]")).toHaveAttribute("data-deliver-need-cut", "0");
+ await expect(move).toBeEnabled();
+});
+
+test("a new batch: refused by an app with no media root, made as a link by a CLI with one", async ({ page, request }) => {
+ test.setTimeout(180_000);
+ await page.goto(`/browse/${PROJECT}`);
+ // The app could only make share-<x>/ in the project, which is not where this
+ // project's deliverables live: it says so rather than splitting them.
+ await expect(page.locator('[data-action="deliver-share"]')).toBeDisabled();
+ await expect(page.locator("[data-deliver-blocked]")).toContainText("UMTOOL_MEDIA_DIR is not set in umtool's environment");
+ const refused = await request.post("/api/report/deliver", { data: { project: PROJECT, action: "share", name: "second" } });
+ expect(refused.status()).toBe(409);
+ expect(existsSync(path.join(DIR, "share-second"))).toBe(false);
+
+ // The batch's CLI, with the media root: share-second is made as a link, and
+ // share-first is read THROUGH its link, so d03 is not shipped again.
+ const r = spawnSync("node", ["bin/share-batch.mjs", "--project", PROJECT, "--name", "second"], {
+ cwd: UMTOOL,
+ encoding: "utf8",
+ env,
+ });
+ expect(r.status, r.stderr).toBe(0);
+ const root = path.join(DIR, "share-second");
+ expect(isLink(root)).toBe(true);
+ expect(readlinkSync(root)).toBe(path.join(MIRROR, "share-second"));
+ const list = readFileSync(path.join(MIRROR, "share-second", "LIST.md"), "utf8");
+ expect(list).toContain("d01_2025-03-01");
+ expect(list).toContain("d02_2025-03-02");
+ expect(list).not.toContain("d03_");
+ expect(list).toContain("1 already shared (d03)");
+
+ // And the panel lists it, through its link. Three ids, not two: a batch's
+ // LIST.md is read for the ids it says were "already shared" as well as the
+ // files it names (deliver.mjs sharedIdsIn), so d03 counts as shipped here too.
+ await page.reload();
+ await expect(page.locator('[data-batch="share-second"]')).toContainText("share-second (3)");
+ await expect(page.locator('[data-deliverable="share-second"]')).toHaveAttribute("data-deliverable-state", "link");
+});
+
+test("an unplugged media root: check blocks, the panel refuses, nothing is made", async ({ page, request }) => {
+ renameSync(MEDIA, UNPLUGGED);
+ try {
+ const check = umtool(["check", PROJECT]);
+ expect(check.status).toBe(1);
+ const unreachable = JSON.parse(check.stdout)
+ .decisions.filter((d: { kind: string }) => d.kind === "storage-unreachable")
+ .map((d: { target: string; severity: string; why: string }) => [d.target, d.severity, d.why.includes("is the media drive mounted?")]);
+ expect(unreachable).toEqual([
+ ["clips", "blocking", true],
+ ["share-first", "blocking", true],
+ ["share-second", "blocking", true],
+ ]);
+
+ await page.goto(`/browse/${PROJECT}`);
+ await expect(page.locator('[data-deliverable="clips"]')).toHaveAttribute("data-deliverable-state", "dangling");
+ await expect(page.locator('[data-action="deliver-move"]')).toBeDisabled();
+ await expect(page.locator("[data-move-reason]")).toContainText("is the media drive mounted?");
+ await expect(page.locator('[data-action="deliver-cut"]')).toBeDisabled();
+ await expect(page.locator("[data-deliver-blocked]")).toContainText("is the media drive mounted?");
+ // The route says the same to a caller that is not the panel.
+ const cut = await request.post("/api/report/deliver", { data: { project: PROJECT, action: "cut" } });
+ expect(cut.status()).toBe(409);
+ expect(((await cut.json()) as { error: string }).error).toContain("is the media drive mounted?");
+
+ // Nothing was made in the links' place, and the root was not recreated.
+ expect(isLink(path.join(DIR, "clips"))).toBe(true);
+ expect(existsSync(MEDIA)).toBe(false);
+ } finally {
+ renameSync(UNPLUGGED, MEDIA);
+ }
+});
+
+test("the bench moves the deliverables home, and offers no move to media without a media root", async ({ page }) => {
+ test.setTimeout(180_000);
+ await page.goto(`/browse/${PROJECT}`);
+ const move = page.locator('[data-action="deliver-move"]');
+ await expect(move).toHaveText("Move deliverables to local");
+ await move.click();
+ const { state, log } = await waitForJob(page);
+ expect(state, log).toBe("done");
+ // The app has no media root, so it brings each copy home and leaves the
+ // media side where it is, saying so -- it cannot tell it is the project's own.
+ expect(log).toContain("left in place");
+ expect(log).toContain("storage.deliverables: media -> local");
+
+ for (const name of ["clips", "share-first", "share-second"]) {
+ expect(isRealDir(path.join(DIR, name)), name).toBe(true);
+ }
+ expect(readdirSync(path.join(DIR, "clips")).sort()).toEqual(["d01.mp4", "d02.mp4", "d03.mp4"]);
+ expect(manifestStorage()).toEqual({ deliverables: "local" });
+
+ await page.reload();
+ await expect(page.locator("[data-deliverables]")).toHaveAttribute("data-deliverables", "local");
+ await expect(move).toHaveText("Move deliverables to media");
+ await expect(move).toBeDisabled();
+ await expect(page.locator("[data-move-reason]")).toContainText("UMTOOL_MEDIA_DIR is not set in umtool's environment");
+ // Local again: a batch is the app's to make.
+ await expect(page.locator('[data-action="deliver-share"]')).toBeDisabled(); // nothing left to ship
+ await expect(page.locator("[data-deliver-blocked]")).toHaveCount(0);
+});
diff --git a/umtool/e2e/fixtures/make-fixture.mjs b/umtool/e2e/fixtures/make-fixture.mjs
@@ -1739,6 +1739,38 @@ for (const slug of ["storage-fixture", "storage-fresh-fixture"]) {
);
}
+// deliverables.spec.ts (release 17, slice U2): a report's deliverables behind
+// the per-project switch. Its CLI moves clips/ and share-first/ to the media
+// root above; the app (no UMTOOL_MEDIA_DIR) then cuts THROUGH the clips/ link,
+// and is refused a new batch; the CLI's share batch makes share-second as a
+// link and reads share-first through its own; the app moves them back.
+// d01 d02 confirmed, cached in the window, no file yet -> the app's cut
+// d03 confirmed, has a file, and shipped in share-first -> excluded
+const DELIVERABLES = writeProject(
+ "deliverables-fixture",
+ manifest("deliverables-fixture", "The Deliverables Fixture", { siteOrigin: "https://archive.example" }, [
+ { type: "clip", id: "d01", video: "vid6", start: 3.0, end: 6.0, cite: 3, section: 0, lock: true, verdict: "confirmed", date: "2025-03-01", title: "A Fixture Stream", quote: "The first clip is confirmed." },
+ { type: "clip", id: "d02", video: "vid6", start: 6.0, end: 9.0, cite: 6, section: 0, lock: true, verdict: "confirmed", date: "2025-03-02", title: "A Fixture Stream", quote: "So is the second." },
+ { type: "clip", id: "d03", video: "vid6", start: 0.0, end: 3.0, cite: 0, section: 0, lock: true, verdict: "confirmed", date: "2025-03-03", title: "A Fixture Stream", quote: "The deliver fixture opens." },
+ ]),
+);
+mkdirSync(path.join(DELIVERABLES, "out", "clips-raw"), { recursive: true });
+copyFileSync(
+ path.join(DELIVER, "out", "clips-raw", "vid6_0.00-24.00.mp4"),
+ path.join(DELIVERABLES, "out", "clips-raw", "vid6_0.00-24.00.mp4"),
+);
+mkdirSync(path.join(DELIVERABLES, "clips"), { recursive: true });
+copyFileSync(path.join(DELIVER, "clips", "b01.mp4"), path.join(DELIVERABLES, "clips", "d03.mp4"));
+mkdirSync(path.join(DELIVERABLES, "share-first", "orig", "D"), { recursive: true });
+copyFileSync(
+ path.join(DELIVER, "clips", "b01.mp4"),
+ path.join(DELIVERABLES, "share-first", "orig", "D", "d03_2025-03-03_A-Fixture-Stream.mp4"),
+);
+writeFileSync(
+ path.join(DELIVERABLES, "share-first", "LIST.md"),
+ "# deliverables-fixture — share batch `first`\n\n- **d03_2025-03-03_A-Fixture-Stream.mp4**\n",
+);
+
console.log(`fixture at ${dest}`);
if (planned) console.log(` planned clip (used in a build): ${planned}`);
console.log(` videos/: alpha (4 cuts, 3 variants), beta (2 cuts), deck (1 cut, 2 variants)`);
@@ -1755,6 +1787,7 @@ console.log(` SONG_DIR=${path.join(dest, "data")}`);
console.log(` SONG_REPORTS_DIR=${reports}`);
console.log(` UMTOOL_CACHE_DIR=${path.join(dest, "cache")} (removed with the fixture; never ~/.cache)`);
console.log(` storage spec media root: ${MEDIA} (UMTOOL_MEDIA_DIR on its CLI only; reset here)`);
+console.log(` deliverables-fixture: clips/ (d03) + share-first/, d01/d02 cuttable (deliverables.spec)`);
console.log(` YTDLP_BIN=${path.join(BIN, "yt-dlp")} QRENCODE_BIN=${path.join(BIN, "qrencode")} HYPERFRAMES_BIN=${path.join(BIN, "hyperframes")}`);
console.log(` CHANNELS_DIR=${CHANNELS} (testchan/vid1 punctuated, vid2 not; vid3/vid4/vid5 for the editor fetch)`);
console.log(` projects: report-fixture (4 clips, 1 mid-sentence), no-origin-fixture,`);
diff --git a/umtool/lib/media.ts b/umtool/lib/media.ts
@@ -85,11 +85,16 @@ const MIN_INTERESTING = 256 * 1024;
* picker under its project. Only a link INTO the media root: every other link
* stays unfollowed, as it always was (SONG_DATA's 39 GB are links).
*/
-/** The project directories that may be links to the media root (U2 adds deliverables). */
-const MEDIA_LINKS = new Set(["out"]);
+/**
+ * The project directories that may be links to the media root: render scratch
+ * (`out`), and a report's deliverables (`clips`, every `share-*`) once the
+ * project's switch has moved them (release 17, slice U2).
+ */
+const MEDIA_LINKS = new Set(["out", "clips"]);
+const isMediaLinkName = (name: string) => MEDIA_LINKS.has(name) || name.startsWith("share-");
async function isMediaLink(e: { name: string; isSymbolicLink(): boolean }, abs: string): Promise<boolean> {
- if (!MEDIA_TIERED || !e.isSymbolicLink() || !MEDIA_LINKS.has(e.name)) return false;
+ if (!MEDIA_TIERED || !e.isSymbolicLink() || !isMediaLinkName(e.name)) return false;
const real = await realpath(abs).catch(() => null);
if (!real || !inside(MEDIA_ROOT, real)) return false;
return stat(real).then((s) => s.isDirectory(), () => false);
diff --git a/umtool/lib/projects/report.mjs b/umtool/lib/projects/report.mjs
@@ -9,7 +9,8 @@ import { readdir, readFile, stat } from "node:fs/promises";
import path from "node:path";
import { DEFAULT_VARIANT, cachedWindowsFor } from "umtool-report-to-video/build-video";
import { rawCacheOf } from "../report/raw-cache.mjs";
-import { CHANNELS_DIR } from "../paths.mjs";
+import { deliverablesState, finishCommand, outDirState, leftoversOf } from "../report/storage.mjs";
+import { CHANNELS_DIR, inside } from "../paths.mjs";
import { channelName, cleanTitle } from "umtool-report-to-video/attribution";
import { teaserTitle } from "umtool-report-to-video/deck";
@@ -762,6 +763,13 @@ export const REPORT_DECISION_KINDS = [
"claim-incoherent",
"stale-build",
"unbuilt",
+ // `out/`, `clips/` or a `share-*/` is a link whose target is not there: the
+ // media drive is not mounted (release 17). BLOCKING -- a build, a cut or a
+ // batch refuses at that directory, and a reader sees "nothing built/cut".
+ "storage-unreachable",
+ // Where the deliverables are disagrees with `storage.deliverables`, or a
+ // move was cut. `open`: nothing is lost, and one command settles it.
+ "storage-mismatch",
];
export async function reportDecisions(ctx, summary) {
@@ -1057,6 +1065,65 @@ export async function reportDecisions(ctx, summary) {
}
}
+ // --- where the bulk lives (release 17) -----------------------------------
+ // `out/` follows UMTOOL_MEDIA_DIR; the deliverables follow
+ // `storage.deliverables`. A link whose drive is not there reads as "never
+ // built" and "nothing cut" everywhere else, which is the sentence this
+ // exists to replace.
+ const outState = await outDirState(dir);
+ if (outState.state === "dangling") {
+ add("storage-unreachable", "out", `a link to ${outState.target}, which is not there — is the media drive mounted? Nothing can be built until it is`, "blocking");
+ }
+ const outLeft = await leftoversOf(dir, "out");
+ if (outLeft.length) {
+ add(
+ "storage-mismatch",
+ "out",
+ `a move of out/ was cut (left: ${outLeft.map((p) => path.basename(p)).join(", ")}) — \`${finishCommand("out", outLeft.some((p) => p.endsWith(".incoming"))).replace("<project>", id)}\` finishes it`,
+ "open",
+ );
+ }
+ const store = await deliverablesState(dir);
+ if (!store.mode) add("manifest-invalid", "storage.deliverables", `${store.error} — a cut or a batch refuses until it is`, "blocking");
+ for (const d of store.dirs) {
+ if (d.state === "dangling") {
+ add("storage-unreachable", d.name, `a link to ${d.target}, which is not there — is the media drive mounted? A cut or a batch refuses until it is`, "blocking");
+ }
+ if (d.leftovers.length) {
+ add(
+ "storage-mismatch",
+ d.name,
+ `a move of ${d.name}/ was cut (left: ${d.leftovers.join(", ")}) — \`${finishCommand(d.name, d.leftovers.some((l) => l.endsWith(".incoming"))).replace("<project>", id)}\` finishes it`,
+ "open",
+ );
+ } else if (store.mode === "media" && d.state === "dir") {
+ add("storage-mismatch", d.name, `a directory in the project, while storage.deliverables is media — \`umtool storage deliverables ${id} --to media\` moves it`, "open");
+ } else if (
+ store.mode === "local" &&
+ d.state === "link" &&
+ // Absent means local, so a link INTO the media root under no key at all
+ // is the same disagreement (review N1). A hand-made link elsewhere under
+ // no key is nobody's decision to second-guess; under an explicit
+ // "local", any link is.
+ (store.value !== undefined || (store.mediaRoot && d.target && inside(store.mediaRoot, d.target)))
+ ) {
+ add(
+ "storage-mismatch",
+ d.name,
+ `a link to ${d.target}, while storage.deliverables is ${store.value === undefined ? "unset (local)" : "local"} — \`umtool storage deliverables ${id} --to local\` brings it back, \`--to media\` records it`,
+ "open",
+ );
+ }
+ }
+ if (store.mode === "media" && !store.tiered) {
+ add(
+ "storage-mismatch",
+ "storage.deliverables",
+ "media, and UMTOOL_MEDIA_DIR is not set here — a new clips/ or batch would be refused. Set it where umtool runs",
+ "open",
+ );
+ }
+
// --- the build -----------------------------------------------------------
if (s.build.stale) {
add(
diff --git a/umtool/lib/report/busy.mjs b/umtool/lib/report/busy.mjs
@@ -0,0 +1,96 @@
+// Which report-pipeline processes are working in a project RIGHT NOW (release
+// 17): what `umtool storage move-out|move-back|deliverables` refuses over.
+//
+// Linux /proc; elsewhere, none. Imported by the CLI only (bin/umtool.mjs) --
+// never by anything the app bundles: a literal /proc path is a directory to
+// Turbopack.
+//
+// WHAT IT SEES, and how:
+// - the pipeline's node scripts (SCRIPTS), when an argument is the project
+// directory or a path under it -- how the build steps name a project
+// (`<project>/video.manifest.json`, `--out <project>/out`);
+// - the same scripts when their `--project` value names THIS project: equal
+// to its id, equal to its name, or resolving against the process's own
+// working directory to the project directory. Whole values, never a
+// prefix (`elfpire-eva-2` is not `elfpire-eva`). This is how the app runs
+// a cut (`cut-from-cache.mjs --project <id>`) and a share batch
+// (`share-batch.mjs --project <id>`). A name matched by a same-named
+// project elsewhere reads busy too, which only ever refuses;
+// - the report's OWN scripts (apply-manifest.py, build.py), which name no
+// path: by script name and working directory.
+// WHAT IT CANNOT SEE: the app's in-process work (none of it writes clips/ or
+// share-*; the app's own jobs are kept apart by its one-job-at-a-time rule),
+// a hand-run command that is none of these scripts (an ffmpeg into clips/),
+// and anything on another machine.
+import { readdirSync, readFileSync, readlinkSync, realpathSync } from "node:fs";
+import path from "node:path";
+
+export const PIPELINE_SCRIPTS =
+ /(build-video|check-availability|render-cards|compose-chrome|verify-build|fetch-via-editor|resolve-windows|cut-from-cache|share-batch)\.mjs/;
+const OWN_SCRIPTS = /(^|\/)(apply-manifest|build)\.py$/;
+
+/**
+ * Does one process's argv (and working directory) name this project?
+ * Pure: the /proc reads are the caller's.
+ * @param {string[]} args
+ * @param {string | null} cwd the process's working directory, when readable
+ * @param {{ dir: string, realDir?: string, id?: string, name?: string }} project
+ */
+export function namesProject(args, cwd, project) {
+ const dirs = [...new Set([project.dir, project.realDir ?? project.dir])];
+ const underDir = (a) => dirs.some((d) => a === d || a.startsWith(d + "/"));
+ if (args.some((a) => PIPELINE_SCRIPTS.test(a))) {
+ if (args.some(underDir)) return true;
+ for (let i = 0; i < args.length - 1; i++) {
+ if (args[i] !== "--project") continue;
+ const v = args[i + 1];
+ if (!v) continue;
+ if (v === project.id || v === project.name) return true;
+ if (cwd && dirs.includes(path.resolve(cwd, v))) return true;
+ }
+ return false;
+ }
+ if (args.some((a) => OWN_SCRIPTS.test(a))) return !!cwd && dirs.includes(cwd);
+ return false;
+}
+
+/**
+ * The pids of pipeline processes working in `project` (see the top of the file).
+ * @param {{ dir: string, id?: string, name?: string }} project
+ * @param {{ procRoot?: string, selfPid?: number }} [opts]
+ * @returns {number[]}
+ */
+export function pipelineProcessesFor(project, { procRoot = "/proc", selfPid = process.pid } = {}) {
+ let realDir = project.dir;
+ try {
+ realDir = realpathSync(project.dir);
+ } catch {
+ /* gone: nothing can be running in it */
+ }
+ const ref = { ...project, realDir };
+ const pids = [];
+ let entries = [];
+ try {
+ entries = readdirSync(procRoot).filter((n) => /^\d+$/.test(n));
+ } catch {
+ return pids;
+ }
+ for (const pid of entries) {
+ if (Number(pid) === selfPid) continue;
+ let args;
+ try {
+ args = readFileSync(`${procRoot}/${pid}/cmdline`, "utf8").split("\0").filter((a) => a !== "");
+ } catch {
+ continue;
+ }
+ if (!args.some((a) => PIPELINE_SCRIPTS.test(a) || OWN_SCRIPTS.test(a))) continue;
+ let cwd = null;
+ try {
+ cwd = readlinkSync(`${procRoot}/${pid}/cwd`);
+ } catch {
+ /* another user's process, or gone */
+ }
+ if (namesProject(args, cwd, ref)) pids.push(Number(pid));
+ }
+ return pids;
+}
diff --git a/umtool/lib/report/cut.mjs b/umtool/lib/report/cut.mjs
@@ -16,13 +16,14 @@
// pass, exported rather than copied, so the seconds this writes and the seconds
// the video renders are the same arithmetic.
import { execFile } from "node:child_process";
-import { mkdir, rename, rm } from "node:fs/promises";
+import { rename, rm } from "node:fs/promises";
import path from "node:path";
import { promisify } from "node:util";
import { FFMPEG_BIN, cutArgs } from "umtool-report-to-video/build-video";
import { clipsOf, readManifest } from "../projects/report.mjs";
import { ACCURATE_CUT_ARGS, probeSeconds } from "./encode.mjs";
import { projectCache } from "./serve.mjs";
+import { deliverableDir } from "./storage.mjs";
const execFileP = promisify(execFile);
@@ -93,8 +94,16 @@ export async function cutClipFromCache(
const want = end - start;
const a = Math.max(0, start - win.from);
const b = a + want;
- const dir = path.join(project.dir, CLIPS_DIR);
- await mkdir(dir, { recursive: true });
+ // Made where the project's deliverables switch says: a directory, or a link
+ // to the media root (release 17). A dangling link refuses here, before any
+ // ffmpeg runs, and nothing is made in its place. The tmp file below sits
+ // beside the final one either way, so the rename never crosses a volume.
+ let dir;
+ try {
+ dir = await deliverableDir(project.dir, CLIPS_DIR);
+ } catch (e) {
+ return { ok: false, id: clipId, reason: "storage", error: e instanceof Error ? e.message : String(e) };
+ }
const out = path.join(dir, `${clipId}.mp4`);
const tmp = path.join(dir, `.${clipId}.cutting.mp4`);
diff --git a/umtool/lib/report/deliver.mjs b/umtool/lib/report/deliver.mjs
@@ -22,6 +22,7 @@ import { FFMPEG_BIN } from "umtool-report-to-video/build-video";
import { citeUrlFor, clipVerdict, clipsOf, readManifest } from "../projects/report.mjs";
import { SHARE_PROFILES } from "./encode.mjs";
import { CLIPS_DIR } from "./cut.mjs";
+import { deliverableDir, deliverablesProblems, deliverablesState, isDeliverableName } from "./storage.mjs";
const execFileP = promisify(execFile);
@@ -123,9 +124,19 @@ export async function sharedIdsIn(dir) {
}
}
// And the files themselves, for a batch assembled before anyone wrote a list.
- const walk = async (d) => {
+ //
+ // A link to a directory is walked like one (release 17: a batch moved to the
+ // media root is a link, and so may be anything a person linked in), but only
+ // a few levels down -- a batch is `<variant>/<section>/<file>`, and a link
+ // that loops must not hang the panel.
+ const walk = async (d, depth = 0) => {
+ if (depth > 6) return;
for (const ent of await readdir(d, { withFileTypes: true }).catch(() => [])) {
- if (ent.isDirectory()) await walk(path.join(d, ent.name));
+ const p = path.join(d, ent.name);
+ const isDir =
+ ent.isDirectory() ||
+ (ent.isSymbolicLink() && (await stat(p).then((st) => st.isDirectory(), () => false)));
+ if (isDir) await walk(p, depth + 1);
else {
const m = /^([A-Za-z]{1,3}\d{1,3})_.*\.mp4$/.exec(ent.name);
if (m) ids.add(m[1]);
@@ -136,22 +147,41 @@ export async function sharedIdsIn(dir) {
return ids;
}
-/** The batches already in this project, newest name last. */
+/**
+ * The batches already in this project, newest name last.
+ *
+ * A batch may be a LINK to the media root (release 17: `umtool storage
+ * deliverables <p> --to media`), and is listed like the directory it replaced.
+ * A link whose drive is not there is listed too, as `dangling` with no ids:
+ * leaving it out would read as "never shipped" and the next batch would ship
+ * its clips again -- so a batch refuses while one dangles (buildShareBatch).
+ * A cut move's leftover (`share-x.moved-<ts>`, `share-x.incoming`) is not a
+ * batch.
+ */
export async function listBatches(projectDir) {
- const names = (await readdir(projectDir, { withFileTypes: true }).catch(() => []))
- .filter((e) => e.isDirectory() && e.name.startsWith(SHARE_PREFIX))
- .map((e) => e.name)
- .sort();
+ const entries = (await readdir(projectDir, { withFileTypes: true }).catch(() => [])).filter(
+ (e) => (e.isDirectory() || e.isSymbolicLink()) && e.name.startsWith(SHARE_PREFIX) && isDeliverableName(e.name),
+ );
+ const found = await Promise.all(
+ entries.map(async (e) => {
+ const dir = path.join(projectDir, e.name);
+ const isDir = e.isDirectory() || (await stat(dir).then((st) => st.isDirectory(), () => false));
+ // A symlink to a FILE named share-x is nothing of ours; a dangling one is.
+ if (!isDir && (await stat(dir).then(() => true, () => false))) return null;
+ return { name: e.name, dir, dangling: !isDir };
+ }),
+ );
+ const batches = found.filter((b) => b !== null).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0));
return Promise.all(
- names.map(async (name) => {
- const dir = path.join(projectDir, name);
- const ids = [...(await sharedIdsIn(dir))].sort();
+ batches.map(async ({ name, dir, dangling }) => {
+ const ids = dangling ? [] : [...(await sharedIdsIn(dir))].sort();
return {
name,
label: name.slice(SHARE_PREFIX.length),
dir,
ids,
- hasList: !!(await readFile(path.join(dir, "LIST.md"), "utf8").catch(() => null)),
+ dangling,
+ hasList: !dangling && !!(await readFile(path.join(dir, "LIST.md"), "utf8").catch(() => null)),
};
}),
);
@@ -233,6 +263,12 @@ export async function deliverStateOf(project, { manifest = null, entries = null
const batches = await listBatches(project.dir);
const shared = new Set(batches.flatMap((b) => b.ids));
const variants = await contentVariants(project.dir);
+ // Where the deliverables live, and what stops a cut or a batch being
+ // written now (release 17). A dangling clips/ reads as "nothing cut" above;
+ // these sentences are what keep that from turning into a re-cut of
+ // everything on the wrong drive.
+ const storage = await deliverablesState(project.dir);
+ const nextName = `batch-${new Date().toISOString().slice(0, 10)}`;
const rows = clips.map((e) => ({
id: e.id,
@@ -295,11 +331,17 @@ export async function deliverStateOf(project, { manifest = null, entries = null
incorrect: rows.filter((r) => r.verdict === "incorrect").map((r) => r.id),
},
candidates: candidates.map((r) => ({ id: r.id, section: r.section, file: r.file })),
- nextName: `batch-${new Date().toISOString().slice(0, 10)}`,
+ nextName,
variants,
incorrect: await incorrectCitations(project.dir, m, variants),
hasApplyScript: await exists(path.join(project.dir, "apply-manifest.py")),
hasBuildScript: await exists(path.join(project.dir, "build.py")),
+ storage: {
+ ...storage,
+ // The reasons a cut (into clips/) or a new batch would be refused.
+ cutBlocked: deliverablesProblems(storage, CLIPS_DIR),
+ shareBlocked: deliverablesProblems(storage, `${SHARE_PREFIX}${nextName}`),
+ },
};
}
@@ -356,10 +398,17 @@ export async function buildShareBatch(project, name, { log = console.log } = {})
const m = await readManifest(project.dir);
const clips = new Map(clipsOf(m).map((e) => [e.id, e]));
const headings = await sectionHeadings(project.dir);
+ // Refused while a deliverable cannot be read or written (release 17): a
+ // batch that cannot see an earlier batch would ship its clips again, and one
+ // that cannot see clips/ would ship nothing.
+ const blocked = deliverablesProblems(state.storage, `${SHARE_PREFIX}${name}`);
+ if (blocked.length) throw new Error(`share-${name} not built: ${blocked.join("; ")}`);
const chosen = state.candidates.map((c) => c.id);
if (!chosen.length) throw new Error("nothing to ship: every confirmed clip is already shared, or not cut yet");
- const root = path.join(project.dir, `${SHARE_PREFIX}${name}`);
+ // Made where the project's deliverables switch says: a directory, or a link
+ // to the media root. Everything below is made under it, through the link.
+ const root = await deliverableDir(project.dir, `${SHARE_PREFIX}${name}`);
const bySection = new Map();
for (const id of chosen) {
const e = clips.get(id);
diff --git a/umtool/lib/report/deliverables.test.mjs b/umtool/lib/report/deliverables.test.mjs
@@ -0,0 +1,474 @@
+// A report's deliverables (clips/, every share-*/) behind the per-project
+// switch (release 17, slice U2): `storage.deliverables` in the manifest,
+// deliverableDir for the writers, moveDeliverables for the move, and the
+// readers that must follow a moved batch.
+//
+// Every call names its own roots (`reportsRoot`, `mediaRoot`), as
+// storage.test.mjs does, so nothing touches the process's REPORTS_ROOT.
+//
+// Run with: pnpm test:scripts
+import assert from "node:assert/strict";
+import { execFileSync, spawn, spawnSync } from "node:child_process";
+import { lstat, mkdir, mkdtemp, readFile, readdir, readlink, rename, rm, stat, symlink, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import test from "node:test";
+import { fileURLToPath } from "node:url";
+
+import { namesProject, pipelineProcessesFor } from "./busy.mjs";
+import { listBatches, sharedIdsIn } from "./deliver.mjs";
+import { updateStorage } from "./manifest.mjs";
+import {
+ deliverableDir,
+ deliverableNames,
+ deliverablesMode,
+ deliverablesProblems,
+ deliverablesState,
+ isDeliverableName,
+ moveDeliverables,
+} from "./storage.mjs";
+
+/** A reports root with one report project, its deliverables, and a media root beside it. */
+async function world({ manifest = { title: "fixture", timeline: [] }, deliverables = true } = {}) {
+ const base = await mkdtemp(path.join(tmpdir(), "umtool-deliverables-"));
+ const reportsRoot = path.join(base, "reports");
+ const mediaRoot = path.join(base, "media");
+ const projectDir = path.join(reportsRoot, "folder", "proj");
+ await mkdir(projectDir, { recursive: true });
+ await mkdir(mediaRoot);
+ await writeFile(path.join(projectDir, "video.manifest.json"), JSON.stringify(manifest, null, 2) + "\n");
+ if (deliverables) {
+ await mkdir(path.join(projectDir, "clips"));
+ await writeFile(path.join(projectDir, "clips", "a01.mp4"), Buffer.alloc(3000, 1));
+ await mkdir(path.join(projectDir, "share-first", "orig", "A"), { recursive: true });
+ await writeFile(path.join(projectDir, "share-first", "orig", "A", "a01_2025-01-01_x.mp4"), Buffer.alloc(1000, 2));
+ await writeFile(path.join(projectDir, "share-first", "LIST.md"), "<!-- shared-ids: a01 -->\n");
+ }
+ const roots = { reportsRoot, mediaRoot };
+ const untiered = { reportsRoot, mediaRoot: reportsRoot };
+ const mirror = path.join(mediaRoot, "folder", "proj");
+ return { base, reportsRoot, mediaRoot, projectDir, roots, untiered, mirror, done: () => rm(base, { recursive: true, force: true }) };
+}
+
+const kind = async (p) => {
+ const st = await lstat(p).catch(() => null);
+ if (!st) return "missing";
+ return st.isSymbolicLink() ? "link" : st.isDirectory() ? "dir" : "file";
+};
+const readManifest = async (w) => JSON.parse(await readFile(path.join(w.projectDir, "video.manifest.json"), "utf8"));
+const writeMode = (dir, mode) => updateStorage(dir, { deliverables: mode });
+
+// ---------------------------------------------------------------------------
+// The switch, and the names
+// ---------------------------------------------------------------------------
+
+test("storage.deliverables: absent is local; the two values; anything else is an error", async () => {
+ const w = await world({ deliverables: false });
+ try {
+ assert.equal((await deliverablesMode(w.projectDir)).mode, "local");
+ await writeFile(path.join(w.projectDir, "video.manifest.json"), JSON.stringify({ storage: { deliverables: "media" } }));
+ assert.equal((await deliverablesMode(w.projectDir)).mode, "media");
+ await writeFile(path.join(w.projectDir, "video.manifest.json"), JSON.stringify({ storage: { deliverables: "platter" } }));
+ const bad = await deliverablesMode(w.projectDir);
+ assert.equal(bad.mode, null);
+ assert.match(bad.error, /"platter" — it is "local" or "media"/);
+ } finally {
+ await w.done();
+ }
+});
+
+test("updateStorage writes the switch, keeps the rest, and does not rewrite a no-op", async () => {
+ const w = await world({ manifest: { title: "kept", storage: { note: "kept too" } }, deliverables: false });
+ try {
+ await assert.rejects(updateStorage(w.projectDir, { deliverables: "platter" }), /"local" or "media"/);
+ const r = await updateStorage(w.projectDir, { deliverables: "media" });
+ assert.equal(r.changed, true);
+ const m = await readManifest(w);
+ assert.deepEqual(m.storage, { note: "kept too", deliverables: "media" });
+ assert.equal(m.title, "kept");
+ // The CLI's formatting: two-space indent and a trailing newline.
+ assert.match(await readFile(path.join(w.projectDir, "video.manifest.json"), "utf8"), /\n "storage": \{\n[\s\S]*\}\n$/);
+ const mtime = (await stat(path.join(w.projectDir, "video.manifest.json"))).mtimeMs;
+ const again = await updateStorage(w.projectDir, { deliverables: "media" });
+ assert.equal(again.changed, false);
+ assert.equal((await stat(path.join(w.projectDir, "video.manifest.json"))).mtimeMs, mtime);
+ } finally {
+ await w.done();
+ }
+});
+
+test("deliverable names: clips and share-*, never a leftover, a file or another directory", async () => {
+ assert.ok(isDeliverableName("clips") && isDeliverableName("share-first") && isDeliverableName("share-batch-2026.10.01"));
+ for (const no of ["out", "share-", "shares", "clips.moved-20261001T000000Z", "share-x.incoming", "../clips", "share-a/b"]) {
+ assert.equal(isDeliverableName(no), false, no);
+ }
+ const w = await world();
+ try {
+ await mkdir(path.join(w.projectDir, "share-b.moved-20261001T000000Z"));
+ await mkdir(path.join(w.projectDir, "share-c.incoming"));
+ await writeFile(path.join(w.projectDir, "share-file"), "not a directory");
+ await mkdir(path.join(w.projectDir, "revisions"));
+ const other = path.join(w.base, "elsewhere");
+ await mkdir(other);
+ await symlink(other, path.join(w.projectDir, "share-linked"));
+ // clips first; a batch present only as a leftover is named, so a move finishes it.
+ assert.deepEqual(await deliverableNames(w.projectDir), ["clips", "share-b", "share-c", "share-first", "share-linked"]);
+ } finally {
+ await w.done();
+ }
+});
+
+// ---------------------------------------------------------------------------
+// deliverableDir: what a cut and a batch call before they write
+// ---------------------------------------------------------------------------
+
+test("deliverableDir, local: a directory in the project, tiered or not", async () => {
+ const w = await world({ deliverables: false });
+ try {
+ const clips = await deliverableDir(w.projectDir, "clips", w.roots);
+ assert.equal(clips, path.join(w.projectDir, "clips"));
+ assert.equal(await kind(clips), "dir");
+ assert.deepEqual(await readdir(w.mediaRoot), []);
+ await assert.rejects(deliverableDir(w.projectDir, "out", w.roots), /not a deliverable directory/);
+ } finally {
+ await w.done();
+ }
+});
+
+test("deliverableDir, media: the first writer makes the mirror and the link; an existing directory is kept", async () => {
+ const w = await world({ manifest: { storage: { deliverables: "media" } }, deliverables: false });
+ try {
+ const share = await deliverableDir(w.projectDir, "share-next", w.roots);
+ assert.equal(await kind(share), "link");
+ assert.equal(await readlink(share), path.join(w.mirror, "share-next"));
+ // Written through the link, the file lands on the media root.
+ await mkdir(path.join(share, "orig"), { recursive: true });
+ assert.equal(await kind(path.join(w.mirror, "share-next", "orig")), "dir");
+ // Again: the link is it.
+ assert.equal(await deliverableDir(w.projectDir, "share-next", w.roots), share);
+ // A real clips/ is never replaced by a writer: moving it is the switch's job.
+ await mkdir(path.join(w.projectDir, "clips"));
+ assert.equal(await deliverableDir(w.projectDir, "clips", w.roots), path.join(w.projectDir, "clips"));
+ assert.equal(await kind(path.join(w.projectDir, "clips")), "dir");
+ } finally {
+ await w.done();
+ }
+});
+
+test("deliverableDir refuses, and creates nothing: no media root here, an unplugged root, a dangling link, a bad switch", async () => {
+ const w = await world({ manifest: { storage: { deliverables: "media" } }, deliverables: false });
+ try {
+ const clips = path.join(w.projectDir, "clips");
+ await assert.rejects(deliverableDir(w.projectDir, "clips", w.untiered), /UMTOOL_MEDIA_DIR is not set here.*--to local/);
+ assert.equal(await kind(clips), "missing");
+
+ await rename(w.mediaRoot, `${w.mediaRoot}.unplugged`);
+ await assert.rejects(deliverableDir(w.projectDir, "clips", w.roots), /is not there — is its drive mounted\? Nothing was created/);
+ assert.equal(await kind(clips), "missing");
+ assert.equal(await kind(w.mediaRoot), "missing");
+
+ await symlink(path.join(w.mirror, "clips"), clips);
+ await assert.rejects(deliverableDir(w.projectDir, "clips", w.roots), /is the media drive mounted\? Nothing was written/);
+ assert.equal(await kind(w.mediaRoot), "missing");
+ await rename(`${w.mediaRoot}.unplugged`, w.mediaRoot);
+ await rm(clips);
+
+ await writeFile(path.join(w.projectDir, "video.manifest.json"), JSON.stringify({ storage: { deliverables: "platter" } }));
+ await assert.rejects(deliverableDir(w.projectDir, "clips", w.roots), /"platter" — it is "local" or "media"/);
+ assert.equal(await kind(clips), "missing");
+ } finally {
+ await w.done();
+ }
+});
+
+test("deliverableDir refuses over a cut move's leftover and names the switch that finishes it", async () => {
+ const w = await world({ deliverables: false });
+ try {
+ await mkdir(path.join(w.projectDir, "clips.moved-20261001T000000Z"));
+ await assert.rejects(
+ deliverableDir(w.projectDir, "clips", w.roots),
+ /a move of clips\/ .* was cut .*umtool storage deliverables <project> --to media/,
+ );
+ assert.equal(await kind(path.join(w.projectDir, "clips")), "missing");
+ } finally {
+ await w.done();
+ }
+});
+
+// ---------------------------------------------------------------------------
+// moveDeliverables: the switch
+// ---------------------------------------------------------------------------
+
+test("moveDeliverables to media and back: every deliverable moves, then the switch is set", async () => {
+ const w = await world();
+ try {
+ const r = await moveDeliverables(w.projectDir, "media", { ...w.roots, writeMode });
+ assert.equal(r.ok, true);
+ assert.equal(r.written, true);
+ assert.equal(r.before, null);
+ assert.deepEqual(r.results.map((x) => [x.name, x.state]), [["clips", "moved"], ["share-first", "moved"]]);
+ for (const name of ["clips", "share-first"]) {
+ assert.equal(await kind(path.join(w.projectDir, name)), "link");
+ assert.equal(await readlink(path.join(w.projectDir, name)), path.join(w.mirror, name));
+ }
+ // A relative reference still resolves through the link.
+ assert.deepEqual(await readFile(path.join(w.projectDir, "clips", "a01.mp4")), Buffer.alloc(3000, 1));
+ assert.equal((await readManifest(w)).storage.deliverables, "media");
+ assert.equal((await readManifest(w)).title, "fixture");
+
+ // Idempotent: all "already", and the manifest is not rewritten.
+ const mtime = (await stat(path.join(w.projectDir, "video.manifest.json"))).mtimeMs;
+ const again = await moveDeliverables(w.projectDir, "media", { ...w.roots, writeMode });
+ assert.deepEqual(again.results.map((x) => x.state), ["already", "already"]);
+ assert.equal(again.written, false);
+ assert.equal((await stat(path.join(w.projectDir, "video.manifest.json"))).mtimeMs, mtime);
+
+ // The state readers see.
+ const st = await deliverablesState(w.projectDir, w.roots);
+ assert.equal(st.mode, "media");
+ assert.deepEqual(st.dirs.map((d) => [d.name, d.state]), [["clips", "link"], ["share-first", "link"]]);
+ assert.deepEqual(deliverablesProblems(st, "share-next"), []);
+
+ const back = await moveDeliverables(w.projectDir, "local", { ...w.roots, writeMode });
+ assert.equal(back.ok, true);
+ for (const name of ["clips", "share-first"]) assert.equal(await kind(path.join(w.projectDir, name)), "dir");
+ assert.equal((await readManifest(w)).storage.deliverables, "local");
+ // The project's mirror is gone, the media root is kept.
+ assert.equal(await kind(w.mirror), "missing");
+ assert.equal(await kind(w.mediaRoot), "dir");
+ } finally {
+ await w.done();
+ }
+});
+
+test("moveDeliverables: a dry run changes nothing; no media root or no manifest refuses", async () => {
+ const w = await world();
+ try {
+ const dry = await moveDeliverables(w.projectDir, "media", { ...w.roots, writeMode, dryRun: true });
+ assert.deepEqual(dry.results.map((x) => x.state), ["would-move", "would-move"]);
+ assert.equal(dry.results[0].bytes, 3000);
+ assert.equal(dry.written, false);
+ assert.equal(await kind(path.join(w.projectDir, "clips")), "dir");
+ assert.equal((await readManifest(w)).storage, undefined);
+ assert.deepEqual(await readdir(w.mediaRoot), []);
+
+ await assert.rejects(moveDeliverables(w.projectDir, "media", { ...w.untiered, writeMode }), /UMTOOL_MEDIA_DIR is not set/);
+ await assert.rejects(moveDeliverables(w.projectDir, "nowhere", { ...w.roots, writeMode }), /--to is "media" or "local"/);
+ await rm(path.join(w.projectDir, "video.manifest.json"));
+ await assert.rejects(moveDeliverables(w.projectDir, "media", { ...w.roots, writeMode }), /no video\.manifest\.json/);
+ } finally {
+ await w.done();
+ }
+});
+
+test("moveDeliverables: a project with nothing cut yet just sets the switch, and the first cut makes the link", async () => {
+ const w = await world({ deliverables: false });
+ try {
+ const r = await moveDeliverables(w.projectDir, "media", { ...w.roots, writeMode });
+ assert.deepEqual(r.results, []);
+ assert.equal(r.written, true);
+ const clips = await deliverableDir(w.projectDir, "clips", w.roots);
+ assert.equal(await kind(clips), "link");
+ } finally {
+ await w.done();
+ }
+});
+
+test("moveDeliverables: one failure leaves the switch where it was", async () => {
+ const w = await world();
+ try {
+ const r = await moveDeliverables(w.projectDir, "media", { ...w.roots, writeMode, rsyncBin: "false" });
+ assert.equal(r.ok, false);
+ assert.equal(r.written, false);
+ assert.ok(r.results.every((x) => x.state === "failed" && /rsync failed/.test(x.error)));
+ assert.equal((await readManifest(w)).storage, undefined);
+ assert.equal(await kind(path.join(w.projectDir, "clips")), "dir");
+ } finally {
+ await w.done();
+ }
+});
+
+test("moveDeliverables finishes a move cut between the park and the link", async () => {
+ const w = await world();
+ try {
+ // What a cut leaves for clips/: its verified copy on the media root, the source parked.
+ await mkdir(w.mirror, { recursive: true });
+ execFileSync("cp", ["-a", path.join(w.projectDir, "clips"), path.join(w.mirror, "clips")]);
+ await rename(path.join(w.projectDir, "clips"), path.join(w.projectDir, "clips.moved-20261001T000000Z"));
+ const st = await deliverablesState(w.projectDir, w.roots);
+ assert.deepEqual(st.dirs[0], { name: "clips", state: "absent", leftovers: ["clips.moved-20261001T000000Z"] });
+ assert.match(deliverablesProblems(st)[0], /a move of clips\/ was cut .*--to media/);
+
+ const r = await moveDeliverables(w.projectDir, "media", { ...w.roots, writeMode });
+ assert.equal(r.ok, true);
+ assert.equal(r.results[0].resumed, true);
+ assert.equal(await kind(path.join(w.projectDir, "clips")), "link");
+ assert.deepEqual((await readdir(w.projectDir)).sort(), ["clips", "share-first", "video.manifest.json", "video.manifest.json.bak"]);
+ } finally {
+ await w.done();
+ }
+});
+
+// ---------------------------------------------------------------------------
+// What a dangling deliverable stops, and the readers that follow a moved batch
+// ---------------------------------------------------------------------------
+
+test("deliverablesProblems: a dangling link stops everything; media without a root stops only a new directory", async () => {
+ const w = await world();
+ try {
+ await moveDeliverables(w.projectDir, "media", { ...w.roots, writeMode });
+ // The same project seen by a process with no media root: existing links
+ // still work, a NEW batch could not be made.
+ const here = await deliverablesState(w.projectDir, w.untiered);
+ assert.deepEqual(deliverablesProblems(here, "clips"), []);
+ assert.match(deliverablesProblems(here, "share-next")[0], /UMTOOL_MEDIA_DIR is not set in umtool's environment/);
+
+ await rename(w.mediaRoot, `${w.mediaRoot}.unplugged`);
+ const gone = await deliverablesState(w.projectDir, w.roots);
+ assert.deepEqual(gone.dirs.map((d) => d.state), ["dangling", "dangling"]);
+ const problems = deliverablesProblems(gone, "clips");
+ assert.equal(problems.length, 2);
+ assert.match(problems[0], /clips\/ is a link to .*, which is not there — is the media drive mounted\?/);
+ } finally {
+ await w.done();
+ }
+});
+
+test("listBatches and sharedIdsIn follow a moved batch; a dangling one is listed with no ids; a leftover is no batch", async () => {
+ const w = await world();
+ try {
+ await moveDeliverables(w.projectDir, "media", { ...w.roots, writeMode });
+ // A nested link inside a batch is followed too.
+ const extra = path.join(w.base, "extra");
+ await mkdir(extra);
+ await writeFile(path.join(extra, "b07_2025-02-02_y.mp4"), "x");
+ await symlink(extra, path.join(w.projectDir, "share-first", "small"));
+ assert.deepEqual([...(await sharedIdsIn(path.join(w.projectDir, "share-first")))].sort(), ["a01", "b07"]);
+
+ await mkdir(path.join(w.projectDir, "share-old.moved-20261001T000000Z"));
+ const batches = await listBatches(w.projectDir);
+ assert.deepEqual(batches.map((b) => [b.name, b.dangling, b.ids, b.hasList]), [["share-first", false, ["a01", "b07"], true]]);
+
+ await rename(w.mediaRoot, `${w.mediaRoot}.unplugged`);
+ const gone = await listBatches(w.projectDir);
+ assert.deepEqual(gone.map((b) => [b.name, b.dangling, b.ids]), [["share-first", true, []]]);
+ } finally {
+ await w.done();
+ }
+});
+
+// ---------------------------------------------------------------------------
+// The busy scan (lib/report/busy.mjs) and the CLI's refusal (review H1), and
+// `umtool check` on a link under no key (review N1)
+// ---------------------------------------------------------------------------
+
+
+const UMTOOL_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
+
+test("namesProject: a script names a project by path, or by --project id, name or cwd-relative dir — whole values only", () => {
+ const p = { dir: "/r/folder/proj", id: "folder/proj", name: "proj" };
+ const cut = (v, cwd = "/umtool") => namesProject(["node", "/umtool/bin/cut-from-cache.mjs", "--project", v, "--clip", "a01"], cwd, p);
+ assert.equal(cut("folder/proj"), true);
+ assert.equal(cut("proj"), true);
+ assert.equal(cut("/r/folder/proj"), true);
+ assert.equal(cut("../proj", "/r/folder/other"), true);
+ for (const no of ["folder/proj-other", "folder/pro", "proj-2", "folder", "/r/folder/proj2"]) assert.equal(cut(no), false, no);
+ assert.equal(namesProject(["node", "share-batch.mjs", "--project", "folder/proj", "--name", "x"], null, p), true);
+ // A build step names the manifest by path.
+ assert.equal(namesProject(["node", "build-video.mjs", "/r/folder/proj/video.manifest.json"], null, p), true);
+ assert.equal(namesProject(["node", "build-video.mjs", "/r/folder/proj-2/video.manifest.json"], null, p), false);
+ // The report's own scripts, by working directory.
+ assert.equal(namesProject(["python3", "apply-manifest.py"], "/r/folder/proj", p), true);
+ assert.equal(namesProject(["python3", "build.py"], "/r/folder/proj-2", p), false);
+ // Anything else is not a pipeline process, whatever it names.
+ assert.equal(namesProject(["ffmpeg", "-i", "/r/folder/proj/clips/a.mp4"], null, p), false);
+});
+
+/** A process that does nothing for a while, with `cut-from-cache.mjs --project <v>` in its argv. */
+function dummyCut(value) {
+ const child = spawn(process.execPath, ["-e", "setTimeout(() => {}, 20000)", "cut-from-cache.mjs", "--project", value], {
+ stdio: "ignore",
+ });
+ return child;
+}
+
+test("pipelineProcessesFor sees a cut the app started with --project <id>, and not <id>-other", async () => {
+ const w = await world();
+ const mine = dummyCut("folder/proj");
+ const other = dummyCut("folder/proj-other");
+ try {
+ await new Promise((r) => setTimeout(r, 300));
+ const pids = pipelineProcessesFor({ dir: w.projectDir, id: "folder/proj", name: "proj" });
+ assert.ok(pids.includes(mine.pid), "the cut is seen");
+ assert.ok(!pids.includes(other.pid), "a project whose id merely starts with this one's is not");
+ } finally {
+ mine.kill();
+ other.kill();
+ await w.done();
+ }
+});
+
+/** The CLI against a temp reports root, with no media root unless given. */
+function cli(w, args, extra = {}) {
+ return spawnSync(process.execPath, [path.join(UMTOOL_DIR, "bin", "umtool.mjs"), ...args, "--json"], {
+ cwd: UMTOOL_DIR,
+ encoding: "utf8",
+ env: {
+ ...process.env,
+ REPORTS_DIR: w.reportsRoot,
+ SONG_DIR: path.join(w.base, "no-song-data"),
+ UMTOOL_CACHE_DIR: path.join(w.base, "cache"),
+ UMTOOL_MEDIA_DIR: "",
+ ...extra,
+ },
+ });
+}
+
+test("umtool storage deliverables refuses, as busy, while a cut runs in the project", async () => {
+ const w = await world();
+ const ls = cli(w, ["ls"]);
+ assert.equal(ls.status, 0, ls.stderr);
+ const id = JSON.parse(ls.stdout)[0].id;
+ const mine = dummyCut(id);
+ try {
+ await new Promise((r) => setTimeout(r, 300));
+ const r = cli(w, ["storage", "deliverables", id, "--to", "local"]);
+ assert.equal(r.status, 1);
+ const j = JSON.parse(r.stdout);
+ assert.ok(j.busy.includes(mine.pid), r.stdout);
+ assert.match(j.error, /a pipeline process is working in it .* nothing moved/);
+ assert.equal((await readManifest(w)).storage, undefined);
+ } finally {
+ mine.kill();
+ }
+ const other = dummyCut(`${id}-other`);
+ try {
+ await new Promise((r) => setTimeout(r, 300));
+ const r = cli(w, ["storage", "deliverables", id, "--to", "local"]);
+ assert.equal(r.status, 0, r.stderr + r.stdout);
+ assert.equal((await readManifest(w)).storage.deliverables, "local");
+ } finally {
+ other.kill();
+ await w.done();
+ }
+});
+
+test("umtool check: a link into the media root under no key is a mismatch, as under an explicit local", async () => {
+ const w = await world({ deliverables: false });
+ try {
+ await mkdir(path.join(w.mirror, "clips"), { recursive: true });
+ await symlink(path.join(w.mirror, "clips"), path.join(w.projectDir, "clips"));
+ const id = JSON.parse(cli(w, ["ls"]).stdout)[0].id;
+ const kinds = (extra) =>
+ JSON.parse(cli(w, ["check", id], extra).stdout)
+ .decisions.filter((d) => d.kind.startsWith("storage-"))
+ .map((d) => [d.kind, d.target, d.severity]);
+ assert.deepEqual(kinds({ UMTOOL_MEDIA_DIR: w.mediaRoot }), [["storage-mismatch", "clips", "open"]]);
+ await updateStorage(w.projectDir, { deliverables: "local" });
+ assert.deepEqual(kinds({ UMTOOL_MEDIA_DIR: w.mediaRoot }), [["storage-mismatch", "clips", "open"]]);
+ await updateStorage(w.projectDir, { deliverables: "media" });
+ assert.deepEqual(kinds({ UMTOOL_MEDIA_DIR: w.mediaRoot }), []);
+ } finally {
+ await w.done();
+ }
+});
diff --git a/umtool/lib/report/driver.mjs b/umtool/lib/report/driver.mjs
@@ -383,6 +383,28 @@ export function shareBatchSteps(project, name) {
}
/**
+ * Move a project's deliverables (`clips/`, every `share-*` directory) to the media root
+ * or back, then set `storage.deliverables` (release 17). One step: the CLI's
+ * own log is per directory. Run as a JOB so the app's one-job-at-a-time rule
+ * keeps every cut and batch out while it moves; the CLI's process scan covers
+ * what runs outside the app.
+ * @param {{ id: string }} project
+ * @param {"media" | "local"} to
+ */
+export function moveDeliverablesSteps(project, to) {
+ return [
+ {
+ cwd: UMTOOL_DIR,
+ env: {},
+ label: `move deliverables to ${to}`,
+ argv: ["node", tool("umtool.mjs"), "storage", "deliverables", project.id, "--to", to],
+ // A copy, a mirror and a verify of what can be gigabytes of mp4.
+ timeoutMs: 60 * 60_000,
+ },
+ ];
+}
+
+/**
* Fold the bench's rulings back into the report's sources.
*
* Step 1 is the project's own apply-manifest.py: it syncs clips.json from the
diff --git a/umtool/lib/report/manifest.mjs b/umtool/lib/report/manifest.mjs
@@ -32,6 +32,7 @@ import {
import { isCalendarDate } from "umtool-report-to-video/attribution";
import { normalizeOnscreen, validateChrome, validatePosts } from "umtool-report-to-video/deck";
import { parseMuteFrom } from "./playback.mjs";
+import { DELIVERABLES_MODES } from "./storage.mjs";
// Its own write queue, not lib/state.ts's.
//
@@ -707,3 +708,43 @@ export class PostsRefused extends Error {
this.errors = errors;
}
}
+
+
+// ---------------------------------------------------------------------------
+// STORAGE: where the project's deliverables live (release 17, slice U2).
+//
+// `"storage": { "deliverables": "local" | "media" }`, absent = local. Only
+// lib/report/storage.mjs's moveDeliverables calls this, and only once every
+// deliverable is where the value says -- a switch set before the move would
+// send the next cut to a drive the earlier ones are not on.
+// ---------------------------------------------------------------------------
+
+/**
+ * Set `storage.deliverables`. Any other key under `storage` is kept. When the
+ * file already says `deliverables`, nothing is written (no new mtime, so no
+ * open bench page's token goes stale over a no-op).
+ *
+ * @param {string} dir
+ * @param {{ deliverables: "local" | "media" }} patch
+ * @param {{ token?: string | null }} [opts]
+ * @returns {Promise<{ storage: Record<string, unknown>, token: string | null, changed: boolean }>}
+ */
+export async function updateStorage(dir, { deliverables } = {}, { token = null } = {}) {
+ if (!DELIVERABLES_MODES.includes(deliverables)) {
+ throw new Error(`storage.deliverables is "local" or "media", not ${JSON.stringify(deliverables)}`);
+ }
+ return withManifestLock(async () => {
+ const current = await manifestToken(dir);
+ if (token !== null && current !== token) throw new StaleToken(token, current);
+
+ const manifest = JSON.parse(await readFile(manifestFile(dir), "utf8"));
+ const storage =
+ manifest.storage && typeof manifest.storage === "object" && !Array.isArray(manifest.storage)
+ ? manifest.storage
+ : {};
+ if (storage.deliverables === deliverables) return { storage, token: current, changed: false };
+ manifest.storage = { ...storage, deliverables };
+ const nextToken = await writeManifestAtomic(dir, manifest);
+ return { storage: manifest.storage, token: nextToken, changed: true };
+ });
+}
diff --git a/umtool/lib/report/storage.mjs b/umtool/lib/report/storage.mjs
@@ -16,7 +16,10 @@
//
// The movers take a NAME, not "out": `umtool storage move-out` moves `out/`
// with them, and a project's deliverables (`clips/`, `share-*/`) are moved by
-// the same two calls behind the deliverables switch.
+// the same two calls behind the deliverables switch (slice U2, at the end of
+// this file): `storage.deliverables` in video.manifest.json says where they
+// live ("local", the default, or "media"), deliverableDir makes one where the
+// switch says, and moveDeliverables moves them all and then sets the switch.
//
// Modelled on the editor's common/controller/relocateDir.ts, not imported from
// it (umtool's scripts are plain .mjs under node, and that is a TypeScript
@@ -35,7 +38,7 @@
// imports this module, and the app's routes import that (plans/FACTS.md, "A
// path joined from `process.cwd()` …").
import { spawn } from "node:child_process";
-import { lstat, mkdir, readdir, readlink, rename, rm, rmdir, stat, statfs, symlink, unlink } from "node:fs/promises";
+import { lstat, mkdir, readFile, readdir, readlink, rename, rm, rmdir, stat, statfs, symlink, unlink } from "node:fs/promises";
import path from "node:path";
import { MEDIA_ROOT, REPORTS_ROOT, inside, mediaMirror } from "../paths.mjs";
@@ -128,51 +131,62 @@ export async function outDirState(projectDir) {
* is never tiered.
*/
export async function ensureOutDir(projectDir, opts = {}) {
- const out = path.join(/* turbopackIgnore: true */ projectDir, "out");
- const s = await pathState(out);
- if (s.kind === "dir") return out;
+ return ensureProjectDir(projectDir, "out", opts, (roots) => roots.tiered);
+}
+
+/**
+ * The one way a project directory that may live on the media root is made:
+ * `out` (tiered whenever UMTOOL_MEDIA_DIR is set) and a deliverable (tiered
+ * when the manifest says so). `tierFor(roots)` is asked only when the directory
+ * is absent, and may throw to refuse.
+ */
+async function ensureProjectDir(projectDir, name, opts, tierFor) {
+ const target = path.join(/* turbopackIgnore: true */ projectDir, name);
+ const s = await pathState(target);
+ if (s.kind === "dir") return target;
if (s.kind === "link") {
- if (s.targetIsDir) return out;
+ if (s.targetIsDir) return target;
throw new Error(
- `${out} is a link to ${s.target}, which is not there — is the media drive mounted? ` +
+ `${target} is a link to ${s.target}, which is not there — is the media drive mounted? ` +
`Nothing was written, and nothing was created in its place.`,
);
}
- if (s.kind === "other") throw new Error(`${out} exists and is not a directory`);
+ if (s.kind === "other") throw new Error(`${target} exists and is not a directory`);
- // A cut move leaves `out.moved-<ts>` or `out.incoming` beside a missing
- // `out`. Making a fresh, empty out/ there would let the next move-out mirror
+ // A cut move leaves `<name>.moved-<ts>` or `<name>.incoming` beside a missing
+ // `<name>`. Making a fresh, empty one there would let the next move mirror
// it over the complete media copy (review L5): refuse, and say how to finish.
- await assertNoLeftovers(projectDir, "out", "nothing was created");
+ await assertNoLeftovers(projectDir, name, "nothing was created");
const roots = rootsOf(opts);
- const mirror = roots.tiered ? mediaMirror(projectDir, roots) : null;
+ const tier = await tierFor(roots);
+ const mirror = tier && roots.tiered ? mediaMirror(projectDir, roots) : null;
if (!mirror) {
- await mkdir(/* turbopackIgnore: true */ out, { recursive: true });
- return out;
+ await mkdir(/* turbopackIgnore: true */ target, { recursive: true });
+ return target;
}
const problem = await mediaRootProblem(roots);
- if (problem) throw new Error(`cannot make ${out}: ${problem}`);
+ if (problem) throw new Error(`cannot make ${target}: ${problem}`);
// The project must exist before its mirror is made: otherwise the symlink
// fails and leaves an empty mirror on the media root (review N5).
// stat, not lstat: a project directory may itself be a link (the walk follows them).
if (!(await stat(/* turbopackIgnore: true */ projectDir).then((st) => st.isDirectory(), () => false))) {
- throw new Error(`cannot make ${out}: ${projectDir} is not a directory`);
+ throw new Error(`cannot make ${target}: ${projectDir} is not a directory`);
}
- const target = path.join(/* turbopackIgnore: true */ mirror, "out");
+ const dest = path.join(/* turbopackIgnore: true */ mirror, name);
// Recursive is safe here: the root itself was just seen to exist.
- await mkdir(/* turbopackIgnore: true */ target, { recursive: true });
+ await mkdir(/* turbopackIgnore: true */ dest, { recursive: true });
try {
- await symlink(/* turbopackIgnore: true */ target, out, "dir");
+ await symlink(/* turbopackIgnore: true */ dest, target, "dir");
} catch (e) {
// Two writers starting at once: whichever linked first won, and a link (or
// directory) that now resolves is as good as ours.
if (e?.code !== "EEXIST") throw e;
- const again = await pathState(out);
- if (again.kind === "dir" || (again.kind === "link" && again.targetIsDir)) return out;
+ const again = await pathState(target);
+ if (again.kind === "dir" || (again.kind === "link" && again.targetIsDir)) return target;
throw e;
}
- return out;
+ return target;
}
/**
@@ -362,11 +376,17 @@ async function assertNoLeftovers(projectDir, name, what, { beside = false } = {}
);
}
throw new Error(
- `a move of ${name}/ in ${projectDir} was cut (left: ${left.map((p) => path.basename(/* turbopackIgnore: true */ p)).join(", ")}) — ` +
- `run \`umtool storage ${left.some((p) => p.endsWith(".incoming")) ? "move-back" : "move-out"} <project>\` to finish it; ${what}`,
+ `a move of ${name}/ in ${projectDir} was cut (left: ${names}) — ` +
+ `run \`${finishCommand(name, left.some((p) => p.endsWith(".incoming")))}\` to finish it; ${what}`,
);
}
+/** The command that finishes a cut move of `<name>`: `out` has its own pair, a deliverable the switch. */
+export const finishCommand = (name, back) =>
+ name === "out"
+ ? `umtool storage ${back ? "move-back" : "move-out"} <project>`
+ : `umtool storage deliverables <project> --to ${back ? "local" : "media"}`;
+
const defaults = (opts) => ({
rsyncBin: opts.rsyncBin ?? process.env.RSYNC_BIN ?? "rsync",
log: opts.log ?? (() => {}),
@@ -567,3 +587,235 @@ async function dropMediaCopy(from, ownCopy, mediaRoot) {
}
return undefined;
}
+
+// ---------------------------------------------------------------------------
+// Deliverables: `clips/` and every `share-*/` (release 17, slice U2)
+//
+// Unlike out/, they do not follow UMTOOL_MEDIA_DIR on their own: they move per
+// project, by a switch -- `"storage": { "deliverables": "local" | "media" }` in
+// video.manifest.json (absent = local), written only through
+// lib/report/manifest.mjs's updateStorage. moveDeliverables moves what exists
+// with the two movers above and then sets the switch; deliverableDir is how a
+// writer (a cut, a share batch) makes a deliverable directory that does not
+// exist yet, where the switch says. A reader keeps opening
+// `<project>/clips/<id>.mp4` by path: through the link when it is one.
+// ---------------------------------------------------------------------------
+
+/** The two values of `storage.deliverables`. Absent is "local". */
+export const DELIVERABLES_MODES = ["local", "media"];
+
+/** The directory a project's cut clips live in (lib/report/cut.mjs's CLIPS_DIR). */
+const CLIPS = "clips";
+/** A share batch's directory prefix (lib/report/deliver.mjs's SHARE_PREFIX). */
+const SHARE = "share-";
+
+/** What a cut move leaves beside a deliverable: `<name>.moved-<stamp>` or `<name>.incoming`. */
+const LEFTOVER = /^(.+)\.(moved-[^/]*|incoming)$/;
+
+/** Is `name` a deliverable directory's name (and not a cut move's leftover)? */
+export const isDeliverableName = (name) =>
+ typeof name === "string" &&
+ !LEFTOVER.test(name) &&
+ (name === CLIPS || (name.startsWith(SHARE) && name.length > SHARE.length && !/[\/\\\0]/.test(name)));
+
+/**
+ * `storage.deliverables` of a parsed manifest.
+ * @returns {{ mode: "local" | "media" | null, value: unknown, error?: string }}
+ * `mode` null when the value is not one of the two (the error says so).
+ */
+export function deliverablesModeOf(manifest) {
+ const v = manifest?.storage?.deliverables;
+ if (v === undefined) return { mode: "local", value: undefined };
+ if (DELIVERABLES_MODES.includes(v)) return { mode: v, value: v };
+ return { mode: null, value: v, error: `storage.deliverables is ${JSON.stringify(v)} — it is "local" or "media"` };
+}
+
+/**
+ * `storage.deliverables`, read off the project's manifest. Read here (a plain
+ * JSON read) rather than through lib/projects/report.mjs, which this module's
+ * importers must not pull in; written only by manifest.mjs.
+ */
+export async function deliverablesMode(projectDir) {
+ const file = path.join(/* turbopackIgnore: true */ projectDir, "video.manifest.json");
+ const text = await readFile(/* turbopackIgnore: true */ file, "utf8").catch(() => null);
+ if (text === null) return { mode: "local", value: undefined, manifest: false };
+ try {
+ return { ...deliverablesModeOf(JSON.parse(text)), manifest: true };
+ } catch {
+ return { mode: null, value: undefined, manifest: true, error: `${file} is not valid JSON` };
+ }
+}
+
+/**
+ * Make sure the deliverable directory `<projectDir>/<name>` exists, and return
+ * its path. What a cut (`clips`) and a share batch (`share-<x>`) call before
+ * they write.
+ *
+ * a directory, or a link to one -> it, whatever the switch says (a move is
+ * what changes where an existing one lives)
+ * a DANGLING link -> refused, as ensureOutDir refuses: nothing
+ * is created in its place
+ * absent, the switch "local" -> a directory in the project
+ * absent, the switch "media" -> `<MEDIA_ROOT>/<project-relative>/<name>`
+ * and a link to it -- or refused when there
+ * is no media root here (UMTOOL_MEDIA_DIR
+ * unset in this process), it is not there,
+ * or the project is outside the reports
+ * root. Never silently local: a batch made
+ * on the wrong drive is a split nobody chose.
+ * a cut move's leftover beside it -> refused, naming the command that finishes it
+ *
+ * @param {string} projectDir
+ * @param {string} name "clips" or "share-<x>"
+ * @param {{ mode?: "local" | "media", reportsRoot?: string, mediaRoot?: string }} [opts]
+ * `mode` overrides the manifest's switch (tests).
+ */
+export async function deliverableDir(projectDir, name, opts = {}) {
+ if (!isDeliverableName(name)) throw new Error(`not a deliverable directory: ${JSON.stringify(name)}`);
+ return ensureProjectDir(projectDir, name, opts, async (roots) => {
+ const m = opts.mode ? { mode: opts.mode } : await deliverablesMode(projectDir);
+ if (!m.mode) throw new Error(`cannot make ${path.join(/* turbopackIgnore: true */ projectDir, name)}: ${m.error}`);
+ if (m.mode === "local") return false;
+ const target = path.join(/* turbopackIgnore: true */ projectDir, name);
+ if (!roots.tiered) {
+ throw new Error(
+ `cannot make ${target}: this project keeps its deliverables on the media root (storage.deliverables: media), ` +
+ `and UMTOOL_MEDIA_DIR is not set here — set it, or bring them back with ` +
+ `\`umtool storage deliverables <project> --to local\`. Nothing was created.`,
+ );
+ }
+ if (!mediaMirror(projectDir, roots)) {
+ throw new Error(`cannot make ${target}: ${projectDir} is not under the reports root ${roots.reportsRoot}, so it has no place on the media root`);
+ }
+ return true;
+ });
+}
+
+/**
+ * Every deliverable a project has, by name: `clips` and each `share-*` that is
+ * a directory or a link -- and the name of any whose cut move left only a
+ * leftover (`clips.moved-<ts>` with no `clips`), so a move finishes it.
+ * `clips` first, then the batches by name.
+ */
+export async function deliverableNames(projectDir) {
+ const entries = await readdir(/* turbopackIgnore: true */ projectDir, { withFileTypes: true }).catch(() => []);
+ const names = new Set();
+ for (const e of entries) {
+ const left = LEFTOVER.exec(e.name);
+ if (left && e.isDirectory() && isDeliverableName(left[1])) names.add(left[1]);
+ else if (isDeliverableName(e.name) && (e.isDirectory() || e.isSymbolicLink())) names.add(e.name);
+ }
+ return [...names].sort((a, b) => (a === CLIPS ? -1 : b === CLIPS ? 1 : a.localeCompare(b)));
+}
+
+/**
+ * Where a project's deliverables are, for the bench, `umtool storage` and
+ * `umtool check`. One lstat per name, and one stat through each link (which is
+ * what a reader does anyway).
+ *
+ * @returns {Promise<{ mode: "local" | "media" | null, value: unknown, error?: string,
+ * tiered: boolean, mediaRoot: string | null,
+ * dirs: Array<{ name: string, state: "absent" | "dir" | "link" | "dangling" | "other", target?: string, leftovers: string[] }> }>}
+ */
+export async function deliverablesState(projectDir, opts = {}) {
+ const roots = rootsOf(opts);
+ const m = await deliverablesMode(projectDir);
+ const dirs = [];
+ for (const name of await deliverableNames(projectDir)) {
+ const s = await pathState(path.join(/* turbopackIgnore: true */ projectDir, name));
+ const state = s.kind === "missing" ? "absent" : s.kind === "link" ? (s.targetIsDir ? "link" : "dangling") : s.kind;
+ const leftovers = (await leftoversOf(projectDir, name)).map((p) => path.basename(/* turbopackIgnore: true */ p));
+ dirs.push({ name, state, ...(s.kind === "link" ? { target: s.target } : {}), leftovers });
+ }
+ return {
+ mode: m.mode,
+ value: m.value,
+ ...(m.error ? { error: m.error } : {}),
+ tiered: roots.tiered,
+ mediaRoot: roots.tiered ? roots.mediaRoot : null,
+ dirs,
+ };
+}
+
+/**
+ * What stops a deliverable being WRITTEN now, as sentences, from a
+ * deliverablesState: a link whose drive is not there, a cut move's leftovers,
+ * and -- for a directory that does not exist yet -- a switch this process
+ * cannot honour. `name` narrows it to one directory (a cut asks for `clips`,
+ * a batch for its own `share-<x>`); every dangling link and leftover counts
+ * whatever the name, because a batch reads `clips/` and every earlier batch.
+ *
+ * @param {{ mode: string | null, error?: string, tiered: boolean,
+ * dirs: Array<{ name: string, state: string, target?: string, leftovers: string[] }> }} state
+ * @param {string | null} [name]
+ * @returns {string[]}
+ */
+export function deliverablesProblems(state, name = null) {
+ const out = [];
+ for (const d of state.dirs) {
+ if (d.state === "dangling") {
+ out.push(`${d.name}/ is a link to ${d.target}, which is not there — is the media drive mounted?`);
+ }
+ if (d.leftovers.length) {
+ out.push(
+ `a move of ${d.name}/ was cut (left: ${d.leftovers.join(", ")}) — ` +
+ `\`${finishCommand(d.name, d.leftovers.some((l) => l.endsWith(".incoming")))}\` finishes it`,
+ );
+ }
+ }
+ const exists = name && state.dirs.some((d) => d.name === name && d.state !== "absent");
+ if (name && !exists) {
+ if (!state.mode) out.push(state.error ?? "storage.deliverables is not \"local\" or \"media\"");
+ else if (state.mode === "media" && !state.tiered) {
+ out.push("this project keeps its deliverables on the media root (storage.deliverables: media), and UMTOOL_MEDIA_DIR is not set in umtool's environment");
+ }
+ }
+ return out;
+}
+
+/**
+ * The switch: move every deliverable to `to` ("media" or "local") with the
+ * movers, then set `storage.deliverables` -- only when every one of them is
+ * where it should be. Idempotent: a second run finds each "already" there and
+ * writes nothing (the manifest is not rewritten when the switch already says
+ * `to`). A run that was cut is finished by running it again, as the movers
+ * finish theirs.
+ *
+ * Nothing may be cutting or sharing into the project while it runs. The
+ * callers check (the app's one-job-at-a-time registry, the CLI's process
+ * scan); the movers' verify refuses a tree that keeps changing.
+ *
+ * @param {string} projectDir
+ * @param {"local" | "media"} to
+ * @param {{ writeMode: (dir: string, mode: "local" | "media") => Promise<unknown>,
+ * dryRun?: boolean, log?: (m: string) => void, rsyncBin?: string, reportsRoot?: string, mediaRoot?: string }} opts
+ * `writeMode` is manifest.mjs's updateStorage -- passed in, so this module
+ * (which the app's routes import) does not pull the manifest writer's imports.
+ */
+export async function moveDeliverables(projectDir, to, opts) {
+ if (!DELIVERABLES_MODES.includes(to)) throw new Error(`--to is "media" or "local", not ${JSON.stringify(to)}`);
+ const { dryRun } = defaults(opts);
+ const roots = rootsOf(opts);
+ const m = await deliverablesMode(projectDir);
+ if (!m.manifest) throw new Error(`${projectDir} has no video.manifest.json — deliverables are a report video's`);
+ if (to === "media" && !roots.tiered) {
+ throw new Error("UMTOOL_MEDIA_DIR is not set: there is no media root to move deliverables to. Nothing moved.");
+ }
+ const move = to === "media" ? moveDirToMedia : moveDirToLocal;
+ const results = [];
+ for (const name of await deliverableNames(projectDir)) {
+ try {
+ results.push({ name, ...(await move(projectDir, name, opts)) });
+ } catch (e) {
+ results.push({ name, state: "failed", error: e instanceof Error ? e.message : String(e) });
+ }
+ }
+ const failed = results.filter((r) => r.state === "failed").length;
+ const before = m.value ?? null;
+ let written = false;
+ if (!failed && !dryRun && m.value !== to) {
+ await opts.writeMode(projectDir, to);
+ written = true;
+ }
+ return { ok: failed === 0, to, before, written, dryRun, results };
+}
