Archilyzer · Source

archilyzer

Archilyzer
git clone https://archilyzer.pages.dev/source/archilyzer.git
Log | Files | Refs | README | LICENSE

commit a5010d57c126f978fc42b9b5f1be88005a316d40
parent c8b7b726db1b18f77c61b03ed495084ca1dc32b5
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date:   Sat, 19 Sep 2026 16:47:26 -0400

Merge storage/locations-s4: a destination is a name, a move can be resumed, the badge names the drive

Full suite on the branch tip: 527/538 first pass under load, the 11 reruns green (65/67 then 13/13
after two spec fixes). Closes the storage-locations plan slices S0-S4.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Diffstat:
MAGENTS.md | 10+++++++++-
MRUNNING_IN_DOCKER.md | 36++++++++++++++++++++++++++++++++++++
Mcommon/lib/storageLocations.ts | 30++++++++++++++++++++++++++----
Mcommon/views/storage.ts | 25+++++++++++++++----------
Meditor/CHANGELOG.md | 3++-
Meditor/app/channels/[slug]/components/stages/StorageStage.tsx | 375++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------------
Meditor/app/channels/[slug]/page.tsx | 100+++++++++++++++++++++++++++++++++++++++++++++++++++++--------------------------
Meditor/app/channels/[slug]/storageActions.ts | 102++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------
Meditor/app/channels/bulkStorageActions.ts | 50+++++++++++++++++++++++++++-----------------------
Meditor/app/channels/components/ChannelSelectionDeck.tsx | 113+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------
Meditor/app/channels/components/ChannelsTable.tsx | 43+++++++++++++++++++++++++++++--------------
Aeditor/app/channels/lib/moveDestination.ts | 42++++++++++++++++++++++++++++++++++++++++++
Meditor/app/channels/page.tsx | 36+++++++++++++++++++++++++-----------
Meditor/app/components/MediaLocationBadge.tsx | 64++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------
Meditor/app/page.tsx | 13++++++++++++-
Meditor/e2e/channel-storage.spec.ts | 252++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------
16 files changed, 1033 insertions(+), 261 deletions(-)

diff --git a/AGENTS.md b/AGENTS.md @@ -192,6 +192,11 @@ apps*, not [DEPLOY_DOCKER.md](DEPLOY_DOCKER.md), which is about *building sites* | `transcripts/saved-videos/` | Persisted source-video store. | | `transcripts/search-aliases.json`, `duplicates*.json` | Corpus-wide curated data. | +**The roots a channel's media may be moved to are named entities**, `settings.storage.locations` +(id, label, root, `autoRepoint`, and the volume UUID learned at the last probe) — managed on +**`/storage`**, which reports whether each one is mounted and can re-point a whole location to a +new path without moving a byte. They migrated from the single `settings.storage.mediaRoot` string. + **The public-URL key in `site.json` is `siteUrl`.** The editor form labels the field "Public URL", so grepping for `publicUrl` finds the UI hint and misses the data. @@ -199,7 +204,10 @@ apps*, not [DEPLOY_DOCKER.md](DEPLOY_DOCKER.md), which is about *building sites* `channels/<slug>/data` can be an **absolute symlink** to `<root>/<slug>/data` on another disk, with `config.dataDir` recording the target. The editor's Storage panel -(channel page → Storage) moves it; nothing else writes that field. The on-disk contract +(channel page → Storage) moves it — to one of the locations configured on `/storage`, +or to a root typed by hand; nothing else writes that field. **A channel is not tagged +with its location**: it is on location L iff its `config.dataDir` is under `L.root`, +which is why re-pointing a location rewrites only links and `dataDir`. The on-disk contract `channelDir/data/<id>/…` is unchanged, so **no reader needs to know** — yt-dlp's cwd-relative writes, the LMDB index (it stores mtimes, and `rsync -a` preserves them) and the export build all keep working with no call-site changes. diff --git a/RUNNING_IN_DOCKER.md b/RUNNING_IN_DOCKER.md @@ -397,6 +397,42 @@ its report is not regenerated, rather than the alternative, which is every count channel reading zero and the download runner treating the whole archive as missing. The badge on `/channels` and the channel's Storage panel name the path they cannot reach. +#### Storage locations in a container: re-point by path, and that is the whole story + +`/storage` names each media root as a **location** and reports whether it is there. On +a host it can do more than that: it learns the volume's filesystem UUID from `findmnt`, +so when a drive comes back at a different mountpoint the page offers **Re-point** and +the operator takes it in one click. + +**Inside a container none of that identity exists.** Block devices are not passed +through, so `findmnt -J -T <root>` describes the bind mount and not the disk behind it: +no UUID, no `/dev/disk/by-uuid` entry, nothing to mount with `udisksctl`. Every identity +probe **fails open to "unknown"** — deliberately, because a probe that turned "I could +not ask" into "your disk is gone" would declare every containerised corpus broken. What +a location reports here is what `stat` says and nothing more: + +| Status | What it means in a container | +|---|---| +| Available | The root is a directory. The bind mount is up. | +| Missing | The root is not there, and there is no identity to look for. | +| Not attached | The root is not there and a **recorded** UUID was found nowhere. | +| Mounted elsewhere / Not mounted | **Never reported.** Both need a UUID the probe can find *now*. | + +`Not attached` does appear here, and only for one reason: a `settings.json` authored on a +host carries the `volume.uuid` learned there, and it survives being bind-mounted into the +container. Inside, `findmnt -S UUID=…` and `/dev/disk/by-uuid/<u>` both miss, so the probe +reports `absent` — which reads as "Not attached" but **means exactly what Missing means +here**: the root is not at that path. There is no disk to go looking for and nothing to +mount; fix the path. + +So the container's remedy is the manual one, and it is not a downgrade: **change the +location's root to the path the media is actually at, and re-point.** Edit the location +on `/storage`, or press Re-point after correcting the root; the job rewrites each +channel's `data/` symlink and its `config.dataDir` and moves no bytes. Equivalently, +fix the compose file so the bind mount lands where the editor recorded — the same +`-v /host/path:/container/path` line above — and nothing needs re-pointing at all. +`autoRepoint` has nothing to act on here and can stay off. + The `site` profile does not need the mount: an export build never reads `data/`. ### Useful commands diff --git a/common/lib/storageLocations.ts b/common/lib/storageLocations.ts @@ -3,11 +3,13 @@ import path from "node:path"; // STORAGE LOCATIONS — the named places a channel's media may live. // // This module is PURE, and deliberately so: it is the only storage module a -// `"use client"` file is allowed to import (for the types, and for -// `locationOfDataDir` when a table projects "on Platter" from a channel's -// `dataDir`). Anything that shells out — every identity and availability probe +// `"use client"` file may name at all — and then for its TYPES ONLY, which are +// erased. It imports `node:path`, so a runtime call from a client component +// would pull that into a browser bundle; the two server pages that need a +// location's NAME for a row call `locationLabelOfDataDir` themselves and ship +// the string. Anything that shells out — every identity and availability probe // — lives in `storageVolumes.ts`, which imports execa and must therefore never -// be reachable from a client component (`next build` enforces that). +// be reachable from a client component at all (`next build` enforces that). // // The entity is stored in `settings.storage`. A channel is NOT tagged with its // location: it is on location L iff its `config.dataDir` is under `L.root`. @@ -96,6 +98,26 @@ export function locationOfDataDir( return best; } +// The NAME of the location a channel's media is on, or undefined for none. +// +// The badge's projection, and the reason it lives here rather than beside the +// badge: `MediaLocationBadge.tsx` is imported by `"use client"` files, so it may +// take TYPES from this module but must never call into it — this file imports +// `node:path`, which has no business in a browser bundle. The two server pages +// that build channel rows call this and ship the resulting string. +// +// `label || id` is the same fallback the sanitizer applies on write and +// `views/storage.ts` applies on render: a location whose label was blanked by a +// hand edit is still named by something. +export function locationLabelOfDataDir( + dataDir: string | undefined, + locations: StorageLocation[], +): string | undefined { + if (!dataDir) return undefined; + const found = locationOfDataDir(dataDir, locations); + return found ? found.label || found.id : undefined; +} + // The root of the default location, or "" when there is none. This is the // one-line replacement for every `settings.storage.mediaRoot` read: the Storage // panel's prefill, the bulk move's fallback, the selection deck's box. diff --git a/common/views/storage.ts b/common/views/storage.ts @@ -30,11 +30,7 @@ import type { RegistryReader } from "./inputs"; // `offered` flag and, when it is false, the sentence saying what to fix. export type StorageActionKind = - | "refresh" - | "repoint" - | "mount" - | "edit" - | "delete"; + "refresh" | "repoint" | "mount" | "edit" | "delete"; export type StorageActionView = { kind: StorageActionKind; @@ -110,7 +106,11 @@ export type StorageRowsInputs = { now: number; }; -const STATUS_LABEL: Record<StorageLocationStatus, string> = { +// THE ONE WORDING of each probe status. Exported because the channel page's +// Storage panel names the same five states beside its destination select, and +// two tables of five strings is how "Not mounted" becomes "Unmounted" on one +// page and not the other. +export const STORAGE_STATUS_LABEL: Record<StorageLocationStatus, string> = { available: "Available", "mounted-elsewhere": "Mounted elsewhere", unmounted: "Not mounted", @@ -192,7 +192,7 @@ export function buildStorageRows(i: StorageRowsInputs): StorageRowsPayload { isDefault: loc.id === i.defaultLocationId, autoRepoint: loc.autoRepoint, status, - statusLabel: STATUS_LABEL[status], + statusLabel: STORAGE_STATUS_LABEL[status], identity: probe ? identityLine(probe.identity) : null, channels: counts, channelsText: `${counts.ok} ok / ${counts.unreachable} unreachable / ${counts.moving} moving`, @@ -219,7 +219,12 @@ function repointAction( busy: string | null, ): StorageActionView { if (busy) { - return { kind: "repoint", label: "Re-point", offered: false, withheld: busy }; + return { + kind: "repoint", + label: "Re-point", + offered: false, + withheld: busy, + }; } if (status !== "mounted-elsewhere") { return { @@ -229,7 +234,7 @@ function repointAction( withheld: status === "available" ? "The root is there — nothing to re-point." - : `Re-point needs the volume mounted somewhere else; this location reads ${STATUS_LABEL[status].toLowerCase()}.`, + : `Re-point needs the volume mounted somewhere else; this location reads ${STORAGE_STATUS_LABEL[status].toLowerCase()}.`, }; } if (!candidateRoot) { @@ -273,7 +278,7 @@ function mountAction( withheld: status === "absent" ? "The volume is not attached to this machine." - : `Mount is only offered for an attached, unmounted volume; this location reads ${STATUS_LABEL[status].toLowerCase()}.`, + : `Mount is only offered for an attached, unmounted volume; this location reads ${STORAGE_STATUS_LABEL[status].toLowerCase()}.`, }; } if (!udisksctlAvailable) { diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md @@ -1,9 +1,10 @@ # Changelog ## [Unreleased] +- **The drives a channel's media lives on are named places now, and one click re-points them.** The cold root used to be a single string typed into Settings, and a relocated channel's `config.dataDir` an absolute path — so when the platter was automounted at `/run/media/user/<uuid>` and came back somewhere else, every channel on it read *unreachable* and the only remedy was SSH and hand edits. **`/storage`** (twelfth entry, under Machine) lists each media root as a **location** with a name, a status and the channels on it: `Available`, `Not mounted`, `Not attached`, or **`Mounted elsewhere`** — which is the one that matters, because it means the disk is here under a different mountpoint, and the row then offers **Re-point**, which rewrites every channel's `data/` symlink and `config.dataDir` and **moves no bytes at all**. There is a **Refresh** per row (a probe is `findmnt`, memoised for ten seconds, and it never writes availability to `settings.json`), a **Mount** for an attached-but-unmounted volume, a per-location **auto re-point** opt-in for operators who would rather it just happened, and a boot pass that checks every location as the editor starts. Identity is the volume's filesystem **UUID**, learned at the last successful probe — in a container there are no block devices to learn it from, every probe **fails open to "unknown", and re-point by path is the whole story** (see RUNNING_IN_DOCKER.md). The old `settings.storage.mediaRoot` **migrates on read** into a one-entry list called *Default*; the Settings field is now a link to the page. Everywhere a move starts, the destination is a **name picked from a list** rather than a path retyped per channel: the channel's Storage panel has a **destination select** showing each drive's current state (with *Another root…* keeping the free-text box), and `/channels`' selection deck has the same select for a whole batch — and what reaches the server is the **id**, never the root, so a page rendered before a re-point cannot aim a batch at a root that has since moved. The badge on every channel row says **`on Platter`** instead of sixty columns of absolute path, or **`on Platter — unreachable`** when the drive is not there. **And an interrupted move can be finished.** The controller has always resumed a half-done copy; nothing in the editor could reach it, so the only offered way out of a killed rsync was *Clear marker* and a full re-copy — which for the incident behind this work meant re-copying 131 GB that was already correctly on the far side. The panel now offers **Resume move** beside it, and the same release closes the three ways that incident happened: the relocate copy raced an auto-queue digest unit that made **no job record**, so "is this channel busy" now asks the lanes as well as the registry, and a lane that finds a relocation marker stops instead of writing into a directory being copied; a sidecar written mid-copy left one directory timestamp differing and `verifyCopy` refused the whole 131 GB, so a drift that is *only* directory mtimes now gets one more `rsync -a` pass and a re-verify (content drift still refuses, and still says the source has not been touched). Also fixed: on `/channels` a dimmed row's **Advanced menu drew underneath the rows below it** — `opacity` on a `<tr>` makes a stacking context, so the row is dimmed cell by cell now, and the cell hosting the popover is left alone. - **The operations poll reads the auto-queue's state file once instead of four times.** Every payload the editor draws — the jobs head, the workers grid, the operations board, the sync schedule, the widget's tiles, the pulse token — used to be computed by a function that did its own reading, so each of the four lanes on `/operations` opened `.auto-queue/state.json` for itself: four parses of the same document every three seconds, on a page whose four lanes were always reading one document. Those builders are pure functions in the shared core now — they are handed the settings, the registry, the scheduler, the pool, the clock and their readings, and they cannot reach disk or construct a singleton, which a layer test enforces rather than a comment asking nicely. The reading happens once, at the edge, and is shared. **Nothing moved that you can see**: same pages, same URLs, same JSON on every endpoint, same numbers — the difference is that each payload now has unit tests of its own (the console's cooldown filter, the pulse token's sensitivity, the workers grid's task grouping), where previously the only way to test one was to render the page that showed it. - **`/channels` is a rack now, with one selection deck and a meter bridge.** The page had two selection bars for one selection — a floating one for Tier and Focus, and a second block below sixty-seven rows for Move media, both saying "N selected" and both offering Clear. There is **one deck**: it docks under the table when you tick a row, carries **Tier**, **Focus** and **Media** side by side, and unmounts when you untick. The destination root lives in its own box beside the button (the button used to carry it in its label, where it truncated to *Move media to…* and you could not read where the files were going). **The table stops spilling off the screen.** It lives in one scroll region: the column headers pin to its top, the checkbox and slug cells pin to its left, and a section's name pins under the headers — so the identity column and the meter bridge header stay on screen while sixteen columns scroll sideways. The six pipeline columns read as **one block** rather than six loose dashes: a shared *Pipeline* eyebrow, a surface behind them, a rule at each end. **Rows are 41 px instead of ~90.** The tier cell is one line, and being held by a focus is a small **held** chip rather than the same orange sentence repeated on sixty-one rows — the sentence is stated once, with a count, on the focus line above the table, and each chip still carries the full reason for a screen reader and on hover. Opening a row's *Advanced* overlays the rows below instead of pushing them down. **The page's caveat is at the top.** The note saying every number here is read from each channel's last report, and how old the oldest one is, used to be the last thing on the page in 11 px type under a floating bar; it is the subtitle beside the title now, with the channel count. The band legend and the Names·A / Names·T explainer moved above the table too, beside *Group by section*. In the header, *Sync every channel*, *Full sweep every channel* and *Update all reports* are outlines under an **Every channel** eyebrow that says what they sweep, and **New channel** is the only filled button. Nothing on disk moved and no control changed its name. -- **A channel's media can live on another drive.** A channel page has a **Storage** panel: where its media actually is, how much audio is on disk, how much room is free on the volume holding it, and **Move media to…** — give it a directory on another disk, press *Preview* to see the bytes and the free space there, and the move copies, **verifies**, and only then swaps `data/` for a link to the new location and records it. **Move back in place** reverses it. The source is never touched until the copy has verified, so a cancelled or crashed move leaves everything where it was and the partial copy resumable; re-running finishes it. Nothing else changes: every page, every job, yt-dlp and the search index read the channel exactly as before, because the path they use is unchanged. **The point is what happens when the drive is not mounted.** `data/` reads as empty then, and an empty `data/` means "nothing has been downloaded" to the download runner — an instruction to re-fetch the entire channel onto the disk that was too full to hold it. So an unreachable channel is **refused rather than guessed at**: its media jobs will not start, the four lane runners skip it (and keep running every other channel — this is not a lane stop), its report will not regenerate over an empty directory, and a red **Media unreachable** badge names the path on `/channels`, on the dashboard and on the channel itself. A relocated-and-reachable channel gets a neutral badge saying where; a channel in place gets none. The low-disk floor now measures **the volume the bytes are actually going to** rather than always the corpus disk, and holds each volume separately — a full SSD no longer pauses downloads landing on the platter. The **Media location** line on a channel's Configure form is read-only on purpose: it is a record of what is on disk, written only by a move that succeeded. The cold drive is typed **once**: **Settings → Default media root** seeds the root box in every channel's Storage panel, and `/channels` rows can now be ticked — select several and **Move media to…** queues one job per channel on that channel's own queue, so they serialize instead of fanning out, each one running its own space check at run time rather than at enqueue time (a root that fills partway through refuses the remainder cleanly, and a channel already on that root is skipped rather than failed). The default is a default and nothing more: it is never read by the move itself, which always takes an explicit root, and a relocated channel is not thereby deprioritized. **Nothing moves on its own, and nothing on disk changes until you move a channel.** +- **A channel's media can live on another drive.** A channel page has a **Storage** panel: where its media actually is, how much audio is on disk, how much room is free on the volume holding it, and **Move media to…** — give it a directory on another disk, press *Preview* to see the bytes and the free space there, and the move copies, **verifies**, and only then swaps `data/` for a link to the new location and records it. **Move back in place** reverses it. The source is never touched until the copy has verified, so a cancelled or crashed move leaves everything where it was and the partial copy resumable; re-running finishes it. Nothing else changes: every page, every job, yt-dlp and the search index read the channel exactly as before, because the path they use is unchanged. **The point is what happens when the drive is not mounted.** `data/` reads as empty then, and an empty `data/` means "nothing has been downloaded" to the download runner — an instruction to re-fetch the entire channel onto the disk that was too full to hold it. So an unreachable channel is **refused rather than guessed at**: its media jobs will not start, the four lane runners skip it (and keep running every other channel — this is not a lane stop), its report will not regenerate over an empty directory, and a red **Media unreachable** badge names the path on `/channels`, on the dashboard and on the channel itself. A relocated-and-reachable channel gets a neutral badge saying where; a channel in place gets none. The low-disk floor now measures **the volume the bytes are actually going to** rather than always the corpus disk, and holds each volume separately — a full SSD no longer pauses downloads landing on the platter. The **Media location** line on a channel's Configure form is read-only on purpose: it is a record of what is on disk, written only by a move that succeeded. The cold drive is typed **once**: **Settings → Default media root** seeds the root box in every channel's Storage panel, and `/channels` rows can now be ticked — select several and **Move media to…** queues one job per channel on that channel's own queue, so they serialize instead of fanning out, each one running its own space check at run time rather than at enqueue time (a root that fills partway through refuses the remainder cleanly, and a channel already on that root is skipped rather than failed). The default is a default and nothing more: it is never read by the move itself, which always takes an explicit root, and a relocated channel is not thereby deprioritized. **Nothing moves on its own, and nothing on disk changes until you move a channel.** *(Superseded above: **Settings → Default media root** is gone — the roots are named locations on `/storage` now, and every destination is picked from that list by name rather than typed.)* - **Channels have priorities now, and the auto-queue's rules are generated from them.** Focusing on one group of channels — "finish Jeralyzer, hold the rest" — used to mean hand-editing four rule trees, and the only per-channel switch on `/channels` was **Sync included / excluded**, which gated sync and nothing else. Every channel row now carries a **tier** — *Normal*, *Low* or *Paused* — plus a corpus-wide **focus**: pick channels and press *Focus these*, or focus a whole site, and every lane runs the focused channels until they have nothing left, then falls through to the rest and retakes the lane the moment new focused work arrives. A focus is one fact, not four: the download, transcription, digest and speaker lanes are all held by it, and each lane's console carries a banner saying what is focused, how much of it is pending there, how many channels are held behind it, and **End focus**. Behind the disclosure on each row, any single operation can be pinned to its own tier — "keep this channel's playlist current but stop downloading it" is a *download* pin, and *Sync only* is a preset for it. A paused channel is dropped from the automatic lanes and from the sync scheduler, and **still runs from every Run button**: a hold is not a stop. Its row dims and its Build toggle is untouched, because publishing is a different question from scheduling. The four rule trees are **generated** from all of this: the policy editor on an operation's page shows them read-only with a link back to `/channels`, keeps editing everything that is not generated (enable, workers, order, the replace-auto-captions lane), and the channel leaves you had are replaced by the compiled ones. **Sync included / excluded is gone**, and it is the same statement said better: the 15 channels that carried it become *paused for sync alone* and keep every lane they were on. **Run the migration before you first start this version.** `Sync included / excluded` is a deleted field, and until the migration has moved those 15 channels to *paused for sync*, the editor reads them as having said nothing about sync — so they are back in the schedule, back in **Sync every channel**, back in each group's **Sync**, and shown as auto-sync eligible. Nothing downloads or transcribes differently, and the automatic tick only fires if your scheduler heartbeat is on, but a *Sync all* click in that window sweeps channels you had excluded. The order is: **stop the editor → `pnpm -C common exec tsx bin/migrate-channel-priority.ts` → start it again.** Run it with `--dry-run` first to see exactly what it would write, per channel, and what each row was derived from; the real run takes its own timestamped backup of `settings.json` beside the file, so there is nothing to copy by hand. After that it is a no-op — run it twice and the second run changes nothing. Your rule trees survive either way: they are what the migration reads the channel order out of, and if you set a tier before running it, the first save seeds itself from those same trees rather than replacing the order you hand-built. - **Every pipeline is dispatched by one thing now: its lane’s runner. The two corpus sweeps and the arbiter are gone.** Digest and Speaker work were driven by a *sweep* — a corpus walk armed by its own switch, with its own scope, its own order and its own console — while Download and Transcription were driven by the auto-queue runner, with rules, a claim ladder, a next-up and a pick log. Two mechanisms, two vocabularies, two sets of bugs. There is one: **each of the four lanes has a runner, a rule list, and Start / Drain / Stop beside its pause**, on the operation’s own page. Arming a corpus pass is switching the lane on; scoping it to particular channels or operations is a *rule*, written the same way auto-transcribe’s have been written since it shipped. The dashboard and the widget keep a one-click switch per lane — **Run every channel** / **Stop the lane** where they said *Sweep every channel* / *Stop sweeping* — and the scope lives on the lane’s page, where you can see what it would do next. **Your armed scope is carried over, and no lane is switched on that was not.** The ten settings fields the sweeps used (`digest.sweepEnabled`, `sweepChannels`, `recencyOrder`, `recencyReach`; `backfill.sweepEnabled`, `sweepKinds`, `sweepChannels`, `order`, `reach`, `weight`) are read once and written into the lane’s rules the first time the editor starts: a sweep armed on three channels becomes three rules, an unscoped one becomes a single *every channel* rule, and a disarmed sweep becomes a switched-off lane. What is retired rather than migrated: **Reach**, because a rule already orders every video it claims across every channel — which rule goes first is the rule list’s job; the digest **order**, whose real meaning was always *newest day first, shortest video within a day* and which the lane spells as **Shortest first** (pick *Newest first* there if you want the date order alone); and the backfill lane’s **Resource share**, which was one number answering two different questions. A lane now stands aside for transcription when it would actually compete for the graphics card, and keeps its slots when it would not — so speaker-naming over an LLM endpoint no longer parks itself behind a transcription it was not competing with. **The arbiter, which never ran a single unit in production, is deleted**; the runner is what dispatches an operation-named rule. **Nothing on disk changes**, and the retired keys are left in `settings.json` — harmless, ignored, and yours to delete. - **The transcode operation is gone — it never fired.** A channel page had a *Transcode* stage, `/operations/transcode` had a "no console here" panel, `/cleanup` offered "Clear failed transcodings", and the video list drew a third status dot — all for a re-encode step built against two failures that never happened in production: in 68 channels, no snapshot has ever listed a video as missing its target format, no `failed-transcodings` file has ever held an id, and only four channels even met the stage's gate. Transcription never needed it — a video whose audio is in another format transcribes from that file. What stayed is everything that was never the operation's: the download path still re-encodes what it extracts itself, the video page still offers **Transcode audio.\<ext\> → \<fmt\>** per file, and both audio-format sweeps on the Cleanup stage and `/cleanup` are unchanged (gated on the channel having an `audioFormat`, which is what they compare against). The snapshot bucket behind the sweep is `wrongFormatAudio` now — its operator-facing name — and old reports keep their stray key until their next refresh. A `?stage=transcode` bookmark opens the channel overview. **Nothing on disk changes.** Also: the Pool's running-jobs list names the eight kinds its buttons enqueue, and the site's Search aliases tab no longer carries a "no site selected" branch that could not run. diff --git a/editor/app/channels/[slug]/components/stages/StorageStage.tsx b/editor/app/channels/[slug]/components/stages/StorageStage.tsx @@ -1,5 +1,6 @@ "use client"; +import Link from "next/link"; import { useState } from "react"; import { StreamActionLog } from "yt-dlp-transcript-common/components/StreamActionLog"; import { formatBytes } from "yt-dlp-transcript-common/lib/format"; @@ -12,33 +13,68 @@ import { moveChannelMediaBackAction, previewRelocationAction, relocateChannelMediaAction, + resumeRelocationAction, } from "../../storageActions"; +import type { MoveDestination } from "../../../lib/moveDestination"; -// WHERE THIS CHANNEL'S MEDIA LIVES, and the two buttons that change it. +// WHERE THIS CHANNEL'S MEDIA LIVES, and the buttons that change it. // // The move itself is common/controller/relocateChannelMedia.ts; what this panel // owns is the operator's decision. Three numbers are enough to make it: how much -// there is to move, how much room is free where it is now, and — once a root is -// named — how much room is free there. The third is what the preview is for, and -// it is the reason the Move button is gated behind one: "is there space" is not -// a question this panel should let anyone skip. +// there is to move, how much room is free where it is now, and — once a +// destination is named — how much room is free there. The third is what the +// preview is for, and it is the reason the Move button is gated behind one: "is +// there space" is not a question this panel should let anyone skip. // -// ⚠️ NEITHER RUN PANEL IS EVER UNMOUNTED BY ITS OWN RESULT. +// THE DESTINATION IS A NAMED LOCATION FIRST and a typed root second. /storage +// owns the list; this panel picks from it by id and never posts a root the +// server could look up for itself. "Another root…" keeps the old free-text box +// for the one-off, and it is the whole control on a corpus with no locations +// configured. +// +// ⚠️ NO RUN PANEL IS EVER UNMOUNTED BY ITS OWN RESULT. // StreamActionLog holds its streamed log in React state and calls // router.refresh() the instant a run ends (plans/FACTS.md, "a run log lives in // the panel's React state"). That refresh re-renders this panel from the server // with `location.relocated` FLIPPED — so a naive `{!relocated && <MoveOut/>}` // would delete the log of the move that just succeeded, at the exact moment the -// operator wants to read it. Both halves therefore follow the documented shape: +// operator wants to read it — and the same is true of Resume move, whose own +// success clears the marker that offered it. All three therefore follow the +// documented shape: // the parent renders them unconditionally and passes the CONDITION down; each // holds a `ranHere` flag set inside its own trigger; each returns null only // while `!condition && !ranHere`; and each puts its log LAST, keyed, in a fixed // slot, with the now-cleared condition fed to StreamActionLog's `disabled` so // the panel that stays for its log is not a second Run button. +// ONE CONFIGURED DESTINATION, projected by the page. `statusLabel` is the +// location's own probe ("Available", "Not mounted", …) taken on the SERVER: +// this file is `"use client"` and may never reach `storageVolumes.ts`, which +// shells out. It is a fact about the drive, refreshed whenever the page +// re-renders (the probe memo makes that cheap), and it is what stops the +// operator picking a destination whose disk is not there and finding out from a +// job log twenty seconds later. +export type StorageDestination = { + id: string; + label: string; + root: string; + statusLabel: string; + // The probe said the root is a directory right now. + available: boolean; +}; + +// The select's escape hatch, and the value of the option that reveals the old +// free-text box. Not a legal location id (`/^[a-z0-9][a-z0-9-]{0,63}$/`), so it +// can never collide with one. +const CUSTOM = "__custom"; + type Props = { slug: string; location: ChannelMediaLocation; + // The name of the storage location this channel's media is on right now, for + // the badge. Undefined when it is on none — in place, or on a root nobody + // named. See common/lib/storageLocations.ts. + locationLabel?: string; // Audio bytes this channel holds, from the loaded snapshot — NOT a walk. Null // when the snapshot predates the field (or there is no snapshot), and rendered // as "—" rather than "0": a zero here would claim a measurement nobody took. @@ -56,28 +92,33 @@ type Props = { // each other, and reading the hatch off `blockedReason === null` would hide // it in exactly the state it exists for. canClearMarker: boolean; - // settings.storage.mediaRoot — the cold root the operator configured once, or - // "" when there is none. READ ONLY: this panel prefills its destination box - // with it and never writes it back. The settings page is the one writer. - defaultRoot: string; + // The configured storage locations, each with its current probe. READ ONLY: + // this panel picks one and never writes the list back — /storage is its one + // writer. Empty when none are configured, which leaves the destination as the + // free-text root it has always been. + destinations: StorageDestination[]; + // `settings.storage.defaultLocationId` — which one the select opens on. + defaultLocationId: string; }; export function StorageStage({ slug, location, + locationLabel, mediaBytes, freeBytes, volumeDir, blockedReason, canClearMarker, - defaultRoot, + destinations, + defaultLocationId, }: Props) { return ( <div className="flex flex-col gap-6"> <section className="flex flex-col gap-2"> <div className="flex items-center gap-3"> <h3 className="text-base font-semibold">Location</h3> - <MediaLocationBadge media={location} /> + <MediaLocationBadge media={location} locationLabel={locationLabel} /> </div> <dl className="grid grid-cols-[max-content_1fr] gap-x-4 gap-y-1 text-sm"> <dt className="text-muted-foreground">Media path</dt> @@ -124,14 +165,24 @@ export function StorageStage({ </p> )} - {canClearMarker && <StaleMarker key="stale-marker" slug={slug} />} + {/* UNCONDITIONAL, with the condition passed down — the same shape the two + moves use, and for the same reason: a successful Resume CLEARS the + marker, so a `{canClearMarker && …}` here would delete the log of the + run that just finished at the instant the operator wants to read it. */} + <StaleMarker + key="stale-marker" + slug={slug} + canAct={canClearMarker} + marker={location.marker ?? null} + /> <MoveOut key="move-out" slug={slug} canMoveOut={!location.relocated} blockedReason={blockedReason} - defaultRoot={defaultRoot} + destinations={destinations} + defaultLocationId={defaultLocationId} /> <MoveBack key="move-back" @@ -164,46 +215,67 @@ function MoveOut({ slug, canMoveOut, blockedReason, - defaultRoot, + destinations, + defaultLocationId, }: { slug: string; canMoveOut: boolean; blockedReason: string | null; - defaultRoot: string; + destinations: StorageDestination[]; + defaultLocationId: string; }) { - // Seeded from settings.storage.mediaRoot, then owned by the operator. It is - // an initial value and NOT a controlled default: a router.refresh() (which - // every finished run triggers) must not throw away a root being typed. It - // also does not enable the move — the preview gate is unchanged, so a - // prefilled root still has to be previewed before Move lights up. - const [root, setRoot] = useState(defaultRoot); + // WHICH CONFIGURED LOCATION, or `__custom` for a one-off root. Opens on the + // default location; with none configured there is nothing to pick, so the + // free-text box is the whole control and this opens on it — which is exactly + // the panel as it was before locations existed. + const [destId, setDestId] = useState(() => + destinations.some((d) => d.id === defaultLocationId) + ? defaultLocationId + : (destinations[0]?.id ?? CUSTOM), + ); + // The one-off root, owned by the operator. An initial value and NOT a + // controlled default: a router.refresh() (which every finished run triggers) + // must not throw away a root being typed. + const [customRoot, setCustomRoot] = useState(""); const [preview, setPreview] = useState<RelocationPreview | null>(null); - // The root the preview above describes. Edit the input and the confirmation - // goes stale — the numbers were measured against a different volume, and - // letting them authorise a move to this one is exactly the mistake the gate - // exists to prevent. - const [previewedRoot, setPreviewedRoot] = useState(""); + // WHICH DESTINATION the preview above describes. Change the select or edit + // the box and the confirmation goes stale — the numbers were measured against + // a different volume, and letting them authorise a move to this one is + // exactly the mistake the gate exists to prevent. Keyed by destination rather + // than by root so switching between two locations that happen to share a root + // is still one confirmed preview, and switching away from one is not. + const [previewedKey, setPreviewedKey] = useState(""); const [previewError, setPreviewError] = useState<string | null>(null); const [previewing, setPreviewing] = useState(false); const [ranHere, setRanHere] = useState(false); if (!canMoveOut && !ranHere) return null; - const trimmed = root.trim(); - const confirmed = trimmed !== "" && trimmed === previewedRoot.trim(); + const chosen = destinations.find((d) => d.id === destId) ?? null; + const custom = destId === CUSTOM || chosen === null; + const trimmed = customRoot.trim(); + // THE ID ALONE GOES TO THE SERVER for a configured location: the root is + // looked up there, from the settings the location list lives in, so the form + // can never send a root a re-point has since moved. + const destination: MoveDestination = custom + ? { kind: "custom", root: trimmed } + : { kind: "location", locationId: chosen.id }; + const key = custom ? `custom:${trimmed}` : `location:${chosen.id}`; + const named = custom ? trimmed !== "" : true; + const confirmed = named && key === previewedKey; const disabled = !canMoveOut || !confirmed || blockedReason !== null; async function runPreview() { setPreviewing(true); setPreviewError(null); - const result = await previewRelocationAction(slug, root); + const result = await previewRelocationAction(slug, destination); setPreviewing(false); if (result.ok) { setPreview(result.preview); - setPreviewedRoot(root); + setPreviewedKey(key); } else { setPreview(null); - setPreviewedRoot(""); + setPreviewedKey(""); setPreviewError(result.error); } } @@ -219,28 +291,86 @@ function MoveOut({ copy verifies. </p> </div> - <label className="flex flex-col gap-1 text-sm"> - <span className="font-medium">Destination root</span> - <input - type="text" - name="mediaRoot" - aria-label="destination root" - value={root} - onChange={(e) => setRoot(e.target.value)} - placeholder="/mnt/platter/archilyzer-media" - disabled={!canMoveOut} - className="rounded border border-border bg-card px-2 py-1 text-sm font-mono" - /> - <span className="text-xs text-muted-foreground"> - An absolute directory that already exists. One root holds many - channels; each gets its own <code>&lt;slug&gt;/data</code> under it. - </span> - </label> + {destinations.length > 0 && ( + <label className="flex flex-col gap-1 text-sm"> + <span className="font-medium">Destination</span> + <select + aria-label="destination location" + // DERIVED, not the raw state. `destId` is seeded once; a location + // deleted on /storage while this page is open would leave the state + // naming an id that is no longer an option, and a <select> whose + // value matches nothing silently shows the first one. `custom` is + // already true in that case — this makes the control agree with it. + value={custom ? CUSTOM : destId} + disabled={!canMoveOut} + onChange={(e) => { + setDestId(e.target.value); + // A new destination is a new question. Dropping the preview with + // the key is belt and braces — `confirmed` already fails — but it + // stops a stale free-space figure sitting under a root it was not + // measured against. + setPreview(null); + setPreviewedKey(""); + setPreviewError(null); + }} + className="rounded border border-border bg-card px-2 py-1 text-sm" + > + {destinations.map((d) => ( + <option key={d.id} value={d.id}> + {d.label} — {d.statusLabel} + </option> + ))} + <option value={CUSTOM}>Another root…</option> + </select> + {chosen && ( + <span className="text-xs text-muted-foreground font-mono break-all"> + {chosen.root} + </span> + )} + {/* The probe's answer, said out loud rather than left in the option's + text alone: a move to a root that is not mounted refuses in the + job's preflight, and the operator should read that here first. */} + {chosen && !chosen.available && ( + <span role="status" className="text-xs text-destructive"> + {chosen.label} reads {chosen.statusLabel.toLowerCase()} — the move + will refuse until the drive is there. See{" "} + <Link href="/storage" className="underline"> + Storage + </Link> + . + </span> + )} + </label> + )} + {custom && ( + <label className="flex flex-col gap-1 text-sm"> + <span className="font-medium">Destination root</span> + <input + type="text" + name="destinationRoot" + aria-label="destination root" + value={customRoot} + onChange={(e) => setCustomRoot(e.target.value)} + placeholder="/mnt/platter/archilyzer-media" + disabled={!canMoveOut} + className="rounded border border-border bg-card px-2 py-1 text-sm font-mono" + /> + <span className="text-xs text-muted-foreground"> + An absolute directory that already exists. One root holds many + channels; each gets its own <code>&lt;slug&gt;/data</code> under it. + A root you expect to use again belongs on{" "} + <Link href="/storage" className="underline"> + Storage + </Link>{" "} + as a named location, where its drive can be checked and re-pointed. + </span> + </label> + )} <div> <button type="button" onClick={runPreview} - disabled={!canMoveOut || trimmed === "" || previewing} + disabled={!canMoveOut || !named || previewing} className="px-3 py-1.5 rounded-md border border-border text-sm font-medium disabled:opacity-50" > {previewing ? "Checking…" : "Preview"} @@ -289,16 +419,16 @@ function MoveOut({ run resumes it rather than starting over. </p> )} - {!confirmed && trimmed !== "" && ( + {!confirmed && named && ( <p className="text-xs text-muted-foreground"> - Preview this root to enable the move. + Preview this destination to enable the move. </p> )} <StreamActionLog key="move-out-log" trigger={() => { setRanHere(true); - return relocateChannelMediaAction(slug, root); + return relocateChannelMediaAction(slug, destination); }} cancelAction={cancelJobAction} buttonLabel="Move media" @@ -362,52 +492,119 @@ function MoveBack({ ); } -// The way out of a marker whose run is gone. +// THE TWO WAYS OUT OF A MARKER WHOSE RUN IS GONE — finish it, or throw it away. // -// It is offered ONLY when the channel is in-transition AND no job is running — +// Both are offered ONLY when the channel is in-transition AND no job is running: // with a live job the marker is not stale, it belongs to that run. Nothing else -// on this panel is available in that state (both moves are disabled while a -// marker stands), so without this a killed copy leaves the channel skipped by -// every lane, refused by every media job and unable to regenerate its snapshot, -// with the only fix being to delete a dotfile over SSH. +// on this panel is available in that state (both moves are hidden or disabled +// while a marker stands), so without this section a killed copy leaves the +// channel skipped by every lane, refused by every media job and unable to +// regenerate its snapshot, with the only fix being to delete a dotfile over SSH. // -// It clears the marker and nothing else, which is why the copy says to look -// first: whatever is on disk stays on disk, and the status underneath may well -// be `inconsistent`. That is the honest answer. -function StaleMarker({ slug }: { slug: string }) { +// RESUME IS THE FIRST ONE, and it is the answer the omnimirror incident wanted: +// `relocateChannelMedia` has always continued a same-direction marker (it logs +// "(resumed an interrupted move)"), and rsync skips what is already correctly on +// the far side — so a copy killed at 131 GB costs a verify pass, not 131 GB +// again. Nothing in the editor could reach it until now. +// +// CLEAR IS THE SECOND, and it stays exactly what it was: it removes the marker +// file and NOTHING else — no files moved, copied or deleted — which is why the +// copy says to look at the drive first. The status underneath may well be +// `inconsistent`, and that is the honest answer rather than a repair nobody +// asked for. +function StaleMarker({ + slug, + canAct, + marker, +}: { + slug: string; + // A marker is present AND nothing is running. Passed rather than inferred so + // this section survives its own success — see the parent's comment. + canAct: boolean; + marker: ChannelMediaLocation["marker"] | null; +}) { const [busy, setBusy] = useState(false); const [error, setError] = useState<string | null>(null); + const [ranHere, setRanHere] = useState(false); + if (!canAct && !ranHere) return null; return ( - <section className="flex flex-col gap-2 rounded border border-destructive/50 bg-destructive/5 px-3 py-2"> - <h3 className="text-base font-semibold">Clear the relocation marker</h3> - <p className="text-sm text-muted-foreground"> - A move is in flight, or one was interrupted. While the marker stands this - channel is skipped by every lane and its media jobs are refused. If no - move is actually running, clear the marker — it removes the marker file - and nothing else: no files are moved, copied or deleted. Check what is on - the drive first. - </p> + <section className="flex flex-col gap-3 rounded border border-destructive/50 bg-destructive/5 px-3 py-2"> + <div className="flex flex-col gap-2"> + <h3 className="text-base font-semibold"> + Finish the interrupted move, or clear its marker + </h3> + {/* THE COPY IS GATED ON `canAct`, not merely the buttons. This section + outlives its own success on purpose — the log of the run that just + finished is the thing the operator wants to read — and a Resume that + worked has CLEARED the marker these three paragraphs describe. Left + ungated they would go on claiming a channel out of transition is + "skipped by every lane", which is exactly the kind of confident + wrongness the rest of this panel exists to avoid. */} + {canAct ? ( + <> + <p className="text-sm text-muted-foreground"> + A move is in flight, or one was interrupted + {marker + ? ` at phase "${marker.phase}" (${marker.direction === "out" ? "to" : "from"} ${marker.target})` + : ""} + . While the marker stands this channel is skipped by every lane + and its media jobs are refused. + </p> + <p className="text-sm text-muted-foreground"> + <strong>Resume move</strong> runs the same move again from where + it stopped — whatever already copied correctly is not copied + twice, and the source is not touched until the copy verifies. That + is the usual answer. + </p> + <p className="text-sm text-muted-foreground"> + <strong>Clear marker</strong> removes the marker file and nothing + else: no files are moved, copied or deleted. Check what is on the + drive first, because whatever this channel&rsquo;s location reads + afterwards is what the disk was already saying underneath. + </p> + </> + ) : ( + <p role="status" className="text-sm text-muted-foreground"> + The marker is gone — this channel is out of transition. + </p> + )} + </div> {error && ( <p role="alert" className="text-sm text-destructive"> {error} </p> )} - <div> - <button - type="button" - disabled={busy} - onClick={async () => { - setBusy(true); - setError(null); - const result = await clearRelocationMarkerAction(slug); - setBusy(false); - if (!result.ok) setError(result.error); - }} - className="px-3 py-1.5 rounded-md border border-destructive text-destructive text-sm font-medium disabled:opacity-50" - > - {busy ? "Clearing…" : "Clear marker"} - </button> - </div> + {/* ONE CONTROL ROW AND THE LOG LAST. Clear rides along as `extraControls` + so the two escapes sit together — and so the log below belongs + unambiguously to Resume, the only one of the two that runs a job. */} + <StreamActionLog + key="resume-move-log" + trigger={() => { + setRanHere(true); + return resumeRelocationAction(slug); + }} + cancelAction={cancelJobAction} + buttonLabel="Resume move" + runningLabel="Resuming…" + label="Resume move" + disabled={!canAct || busy} + extraControls={ + <button + type="button" + disabled={!canAct || busy} + onClick={async () => { + setBusy(true); + setError(null); + const result = await clearRelocationMarkerAction(slug); + setBusy(false); + if (!result.ok) setError(result.error); + }} + className="px-3 py-1.5 rounded-md border border-destructive text-destructive text-sm font-medium disabled:opacity-50" + > + {busy ? "Clearing…" : "Clear marker"} + </button> + } + /> </section> ); } diff --git a/editor/app/channels/[slug]/page.tsx b/editor/app/channels/[slug]/page.tsx @@ -31,7 +31,9 @@ import { countPlaylist } from "yt-dlp-transcript-common/controller/channels"; import { loadFailedTranscriptions } from "yt-dlp-transcript-common/controller/failedTranscriptions"; import { savedVideoTotals } from "yt-dlp-transcript-common/controller/savedVideoInventory"; import { getSettings } from "yt-dlp-transcript-common/lib/settings"; -import { defaultLocationRoot } from "yt-dlp-transcript-common/lib/storageLocations"; +import { locationLabelOfDataDir } from "yt-dlp-transcript-common/lib/storageLocations"; +import { probeAllLocations } from "yt-dlp-transcript-common/controller/storageLocations"; +import { STORAGE_STATUS_LABEL } from "yt-dlp-transcript-common/views/storage"; import { loadShardConfig, type ShardConfig, @@ -131,12 +133,14 @@ export default async function ChannelDetailPage({ // good follow-up, not this change. if (isSocialChannel(config)) { const channelRoot = path.join(paths.channelsDir, slug); - const [postCount, shards, fetchState, postAvailability] = await Promise.all([ - countPosts(channelRoot), - listPostShards(channelRoot), - readPostFetchState(channelRoot), - readPostAvailability(channelRoot), - ]); + const [postCount, shards, fetchState, postAvailability] = await Promise.all( + [ + countPosts(channelRoot), + listPostShards(channelRoot), + readPostFetchState(channelRoot), + readPostAvailability(channelRoot), + ], + ); const availabilityRecords = Object.values(postAvailability); const fetcher = resolveSocialFetcher(config.postFetcher, config.url); // Through the one builder, so this list has the same progress bars /jobs @@ -158,10 +162,12 @@ export default async function ChannelDetailPage({ fetcherLabel: fetcher?.label ?? "no fetcher configured", fetcherId: fetcher?.id ?? "", availableFetchers: await listPostFetchersFor(config.platform ?? ""), - deletedCount: availabilityRecords.filter((r) => isPostGone(r.availability)) - .length, + deletedCount: availabilityRecords.filter((r) => + isPostGone(r.availability), + ).length, checkedCount: availabilityRecords.length, - canCheckAvailability: typeof fetcher?.checkAvailability === "function", + canCheckAvailability: + typeof fetcher?.checkAvailability === "function", handle: config.socialHandle ?? slug, accountUrl: config.url, }} @@ -230,9 +236,8 @@ export default async function ChannelDetailPage({ const actionableNoTranscriptIds = buckets.noTranscript.filter( (id) => !excludedDownloadIds.has(id), ); - const actionableDownloadedNoTranscriptIds = buckets.downloadedNoTranscript.filter( - (id) => !excludedDownloadIds.has(id), - ); + const actionableDownloadedNoTranscriptIds = + buckets.downloadedNoTranscript.filter((id) => !excludedDownloadIds.has(id)); // Enabled lane backfills. A settings read, no I/O. // // backfillLaneOperations, still: this is the BACKFILL lane's list, and the snapshot @@ -284,9 +289,12 @@ export default async function ChannelDetailPage({ // the overview with no hint that the panel it named is still there under a new // id. Resolve the alias first and the old link keeps working. const STAGE_ALIASES: Record<string, StageId> = { backfill: "speakers" }; - const resolvedStage = rawStage ? (STAGE_ALIASES[rawStage] ?? rawStage) : undefined; + const resolvedStage = rawStage + ? (STAGE_ALIASES[rawStage] ?? rawStage) + : undefined; const selectedStage: StageId | null = - (stageOrder.find((id) => id === resolvedStage) as StageId | undefined) ?? null; + (stageOrder.find((id) => id === resolvedStage) as StageId | undefined) ?? + null; const flow = computeChannelFlow({ snapshot, @@ -304,9 +312,7 @@ export default async function ChannelDetailPage({ // site list. Rendering all ten panels on every request is what put four file // reads and a directory walk in front of a page whose default view needs // neither. `panel` is null on the overview, which does no I/O at all. - const panel = selectedStage - ? await buildPanel(selectedStage) - : null; + const panel = selectedStage ? await buildPanel(selectedStage) : null; async function buildPanel(id: StageId): Promise<ReactNode> { const summarize = (c: ShardConfig | null) => @@ -332,15 +338,17 @@ export default async function ChannelDetailPage({ defaultGroupId: s.defaultGroupId, groups: sortGroups(s.groups).map((g) => ({ id: g.id, name: g.name })), })); - const initialMemberships: InitialMembership[] = allSites.flatMap((s) => { - const m = s.channels.find((c) => c.slug === slug); - if (!m) return []; - return [ - m.groupId - ? { siteId: s.siteId, groupId: m.groupId } - : { siteId: s.siteId }, - ]; - }); + const initialMemberships: InitialMembership[] = allSites.flatMap( + (s) => { + const m = s.channels.find((c) => c.slug === slug); + if (!m) return []; + return [ + m.groupId + ? { siteId: s.siteId, groupId: m.groupId } + : { siteId: s.siteId }, + ]; + }, + ); return ( <ChannelFormClient action={ @@ -461,7 +469,10 @@ export default async function ChannelDetailPage({ case "cleanup": { // Saved-video store summary for this channel + whether backups are // configured, for the Retention & persistence section. - const savedTotals = await savedVideoTotals({ paths, channelSlug: slug }); + const savedTotals = await savedVideoTotals({ + paths, + channelSlug: slug, + }); return ( <CleanupStage slug={slug} @@ -517,7 +528,9 @@ export default async function ChannelDetailPage({ // unmounted target and is why the status line above it is the thing to // read first. const volumeDir = - media.status === "ok" && media.target ? media.target : paths.channelsDir; + media.status === "ok" && media.target + ? media.target + : paths.channelsDir; // `media` is inspectChannelMedia's answer and it already CARRIES the // marker — reading the file again here was a second read of the same // bytes that could disagree with the status rendered beside it. @@ -532,12 +545,22 @@ export default async function ChannelDetailPage({ const blockedReason = busy ?? (marker - ? `A relocation (${marker.direction}) to ${marker.target} is in flight, or was interrupted at phase "${marker.phase}". A channel in transition is not moved again from here — the running job finishes it, and an interrupted one is resumed by rerunning the move.` + ? `A relocation (${marker.direction}) to ${marker.target} is in flight, or was interrupted at phase "${marker.phase}". A channel in transition is not moved again from here — the running job finishes it, and an interrupted one is finished by "Resume move" below.` : null); + // THE DESTINATIONS, EACH WITH ITS DRIVE'S CURRENT STATE. One probe per + // configured location, memoised for 10 s inside the controller — so a + // re-render costs nothing and a stage the operator is not looking at + // costs nothing either, because this branch only runs for the Storage + // stage. The panel is `"use client"` and can never ask this itself: + // `storageVolumes.ts` shells out. + const locations = settings.storage.locations; + const probes = await probeAllLocations(locations, paths); return ( <StorageStage slug={slug} location={media} + // Which named location the media is on NOW, for the badge. + locationLabel={locationLabelOfDataDir(media.target, locations)} // From the loaded snapshot, not a walk. Null (rendered "—") when the // snapshot predates the field or does not exist: a 0 would claim a // measurement nobody took. @@ -548,9 +571,20 @@ export default async function ChannelDetailPage({ // A marker with no job behind it is a stale marker: the run that // wrote it is gone, and nothing else will ever clear it. canClearMarker={marker !== null && busy === null} - // The configured cold root, prefilled into the destination box so - // it is typed once in Settings instead of once per channel. - defaultRoot={defaultLocationRoot(settings.storage)} + // The named destinations, so a move is a choice from a list the + // operator maintains once on /storage instead of a path retyped per + // channel — and so the panel can say whether that drive is there. + destinations={locations.map((loc) => { + const probe = probes[loc.id]; + return { + id: loc.id, + label: loc.label || loc.id, + root: loc.root, + statusLabel: STORAGE_STATUS_LABEL[probe?.status ?? "missing"], + available: probe?.status === "available", + }; + })} + defaultLocationId={settings.storage.defaultLocationId} /> ); } diff --git a/editor/app/channels/[slug]/storageActions.ts b/editor/app/channels/[slug]/storageActions.ts @@ -1,6 +1,6 @@ "use server"; -// WHERE A CHANNEL'S MEDIA LIVES — the three actions the Storage panel drives. +// WHERE A CHANNEL'S MEDIA LIVES — the actions the Storage panel drives. // // All of the mechanism is in common/controller/relocateChannelMedia.ts. What is // here is the editor's half: the preview (a plain async call, no job — it is @@ -24,21 +24,30 @@ // also serialized by the queue — but the queue would make it WAIT, and waiting // is the wrong answer for a move the operator can see is already in flight. +import path from "node:path"; import { revalidatePath } from "next/cache"; import { getPaths } from "yt-dlp-transcript-common/lib/paths"; +import { getSettings } from "yt-dlp-transcript-common/lib/settings"; import type { StreamActionResult } from "yt-dlp-transcript-common/jobs/streamCommand"; -import { clearRelocationMarker } from "yt-dlp-transcript-common/lib/channelMedia"; +import { + clearRelocationMarker, + readRelocationMarker, + relocatedDataDir, +} from "yt-dlp-transcript-common/lib/channelMedia"; import { previewRelocation, relocationRootProblem, type RelocationPreview, } from "yt-dlp-transcript-common/controller/relocateChannelMedia"; import { enqueueRelocation } from "../lib/relocationJob"; +import { + resolveMoveDestination, + type MoveDestination, +} from "../lib/moveDestination"; import { channelMediaBusyReason } from "../lib/mediaBusy"; export type PreviewRelocationResult = - | { ok: true; preview: RelocationPreview } - | { ok: false; error: string }; + { ok: true; preview: RelocationPreview } | { ok: false; error: string }; // Read-only: one tree walk of the channel's data dir plus two statfs calls. It // runs INLINE rather than as a job because its whole purpose is to answer a @@ -46,15 +55,18 @@ export type PreviewRelocationResult = // would be a worse way to show two numbers. export async function previewRelocationAction( slug: string, - root: string, + dest: MoveDestination, ): Promise<PreviewRelocationResult> { - const trimmed = root.trim(); - if (!trimmed) return { ok: false, error: "Enter a destination root." }; + const resolved = resolveMoveDestination( + dest, + getSettings().storage.locations, + ); + if ("error" in resolved) return { ok: false, error: resolved.error }; try { const preview = await previewRelocation({ paths: getPaths(), slug, - root: trimmed, + root: resolved.root, }); return { ok: true, preview }; } catch (e) { @@ -64,22 +76,81 @@ export async function previewRelocationAction( export async function relocateChannelMediaAction( slug: string, - root: string, + dest: MoveDestination, ): Promise<StreamActionResult> { - const trimmed = root.trim(); - if (!trimmed) return { ok: false, error: "Enter a destination root." }; + const resolved = resolveMoveDestination( + dest, + getSettings().storage.locations, + ); + if ("error" in resolved) return { ok: false, error: resolved.error }; + const root = resolved.root; // Relative, or inside the corpus. The job refuses both too — it is the guard — // but a root that would copy the channel onto itself should not become a job // record and a log the operator has to open to read the reason. const rootProblem = await relocationRootProblem({ paths: getPaths(), slug, - root: trimmed, + root, }); if (rootProblem) return { ok: false, error: rootProblem }; const refusal = channelMediaBusyReason(slug, "moving its media"); if (refusal) return { ok: false, error: refusal }; - return enqueueRelocation({ slug, direction: "out", root: trimmed }); + return enqueueRelocation({ slug, direction: "out", root }); +} + +// FINISH THE MOVE THAT WAS INTERRUPTED — the other half of "Clear marker", and +// the one the operator wanted first. +// +// `relocateChannelMedia` has ALWAYS resumed a same-direction marker +// (relocateChannelMedia.ts: `resumed` relaxes the "must be in place" +// precondition, and the job logs "(resumed an interrupted move)"). Nothing in +// the editor could reach it: the panel's Move button is hidden once the config +// records a target and disabled while a marker stands, so a killed copy left +// exactly one affordance — throw the marker away and re-copy from scratch, +// which for the incident that motivated this was 131 GB already correctly on +// the far side. +// +// So this enqueues the SAME relocation job with the SAME direction, and the +// controller does the resuming. What it adds is the root, which the marker does +// not carry: a marker records `<root>/<slug>/data`, so the root is its +// grandparent. That inversion is checked rather than assumed — if rebuilding +// the target from the derived root does not give back the marker's own target +// (a hand-edited marker, a slug with a separator in it, a future layout), this +// refuses instead of resuming a move towards a directory nobody named. The move +// BACK needs no root at all: its target is the config's, and the controller +// reads it. +export async function resumeRelocationAction( + slug: string, +): Promise<StreamActionResult> { + const paths = getPaths(); + const marker = await readRelocationMarker(paths, slug); + if (!marker) { + return { + ok: false, + error: + `Channel "${slug}" has no relocation marker — there is no ` + + `interrupted move to resume.`, + }; + } + // The same guard the two moves use, and for the same reason: a marker with a + // LIVE run behind it is not interrupted, it is in progress, and a second job + // would copy into the directory the first one is writing. + const refusal = channelMediaBusyReason(slug, "resuming its move"); + if (refusal) return { ok: false, error: refusal }; + if (marker.direction === "back") { + return enqueueRelocation({ slug, direction: "back" }); + } + const root = path.dirname(path.dirname(marker.target)); + if (marker.target !== relocatedDataDir(root, slug)) { + return { + ok: false, + error: + `The relocation marker points at ${marker.target}, which is not ` + + `<root>/${slug}/data — the destination root cannot be recovered from ` + + `it. Clear the marker and start the move again.`, + }; + } + return enqueueRelocation({ slug, direction: "out", root }); } export async function moveChannelMediaBackAction( @@ -108,7 +179,10 @@ export async function moveChannelMediaBackAction( export async function clearRelocationMarkerAction( slug: string, ): Promise<{ ok: true } | { ok: false; error: string }> { - const refusal = channelMediaBusyReason(slug, "clearing its relocation marker"); + const refusal = channelMediaBusyReason( + slug, + "clearing its relocation marker", + ); if (refusal) return { ok: false, error: refusal }; try { await clearRelocationMarker(getPaths(), slug); diff --git a/editor/app/channels/bulkStorageActions.ts b/editor/app/channels/bulkStorageActions.ts @@ -31,7 +31,6 @@ import path from "node:path"; import { stat } from "node:fs/promises"; import { getPaths } from "yt-dlp-transcript-common/lib/paths"; import { getSettings } from "yt-dlp-transcript-common/lib/settings"; -import { defaultLocationRoot } from "yt-dlp-transcript-common/lib/storageLocations"; import { isSocialChannel } from "yt-dlp-transcript-common/lib/channelConfig"; import { inspectChannelMedia } from "yt-dlp-transcript-common/lib/channelMedia"; import { readChannelConfig } from "yt-dlp-transcript-common/controller/channels"; @@ -39,6 +38,10 @@ import { relocationRootProblem } from "yt-dlp-transcript-common/controller/reloc import { enqueueRelocation } from "./lib/relocationJob"; import { queueForSlugs, type QueueOutcome } from "./lib/queueForSlugs"; import { channelMediaBusyReason } from "./lib/mediaBusy"; +import { + resolveMoveDestination, + type MoveDestination, +} from "./lib/moveDestination"; // The same shape syncAllChannelsAction and the per-group stage buttons return, // so the bar renders it the same way they do. @@ -56,33 +59,35 @@ async function isDirectory(p: string): Promise<boolean> { } } +// THE DESTINATION IS A LOCATION ID, not a root — see lib/moveDestination.ts. +// The deck picks a name from the list /storage maintains and sends the id; the +// root is resolved HERE, from the settings, so a stale page cannot aim a batch +// at a root a re-point has moved. `__custom` and a typed root are still +// accepted, for the one-off. export async function bulkRelocateChannelMediaAction( slugs: string[], - root?: string, + dest: MoveDestination, ): Promise<BulkRelocateResult> { const paths = getPaths(); - // The bar's own box wins; blank falls back to the DEFAULT LOCATION's root. - // /storage is the only writer of the location list — this only reads it. - const chosen = - (root ?? "").trim() || defaultLocationRoot(getSettings().storage).trim(); + const resolved = resolveMoveDestination( + dest, + getSettings().storage.locations, + ); + const refuseAll = (reason: string): BulkRelocateResult => ({ + queued: [], + skipped: slugs.map((slug) => ({ slug, reason })), + }); + if ("error" in resolved) return refuseAll(resolved.error); + const chosen = resolved.root.trim(); if (!chosen) { - return { - queued: [], - skipped: slugs.map((slug) => ({ - slug, - reason: - "no destination root — add a media location on /storage, or type one here", - })), - }; + return refuseAll( + "no destination root — add a media location on /storage, or type one here", + ); } if (!path.isAbsolute(chosen)) { - return { - queued: [], - skipped: slugs.map((slug) => ({ - slug, - reason: `the destination root must be an absolute path (got "${chosen}")`, - })), - }; + return refuseAll( + `the destination root must be an absolute path (got "${chosen}")`, + ); } return queueForSlugs(slugs, { @@ -121,7 +126,6 @@ export async function bulkRelocateChannelMediaAction( // can resolve into one channel's directory and not another's. return relocationRootProblem({ paths, slug, root: chosen }); }, - run: (slug) => - enqueueRelocation({ slug, direction: "out", root: chosen }), + run: (slug) => enqueueRelocation({ slug, direction: "out", root: chosen }), }); } diff --git a/editor/app/channels/components/ChannelSelectionDeck.tsx b/editor/app/channels/components/ChannelSelectionDeck.tsx @@ -42,6 +42,7 @@ import { bulkRelocateChannelMediaAction, type BulkRelocateResult, } from "../bulkStorageActions"; +import type { MoveDestination } from "../lib/moveDestination"; import { useBarAction } from "./ChannelFocusBar"; const TIER_LABEL: Record<StoredChannelTier, string> = { @@ -59,29 +60,58 @@ const EYEBROW = // the eye reads three groups and not eight loose buttons. const GROUP = "flex items-center gap-2 pl-4 border-l border-border"; +// One configured destination, projected by the server that built the page. +// Label and root only — the deck names a place and sends its id; whether that +// drive is mounted is /storage's report, and a probe per location is not +// something a table of 67 rows should pay for on every render. +export type BulkDestination = { id: string; label: string; root: string }; + +// The select's escape hatch: the option that reveals the free-text root box. +// Not a legal location id, so it can never collide with one. +const CUSTOM = "__custom"; + export function ChannelSelectionDeck({ slugs, onClear, - defaultRoot, + destinations, + defaultLocationId, }: { slugs: string[]; onClear: () => void; - // settings.storage.mediaRoot, read on the server. "" when no cold root is - // configured, which leaves the box empty and the action refusing with that as - // the reason. - defaultRoot: string; + // The configured storage locations, read on the server. Empty when none are + // configured, which leaves the free-text box as the whole control and the + // action refusing with "no destination root" as the reason. + destinations: BulkDestination[]; + // Which one the select opens on — `settings.storage.defaultLocationId`. + defaultLocationId: string; }) { // Two runners, because they are two writers: the priority actions clear the // selection on success, the relocate queues jobs and keeps it. - const { pending: priorityPending, error: priorityError, run } = useBarAction(); + const { + pending: priorityPending, + error: priorityError, + run, + } = useBarAction(); const [tier, setTier] = useState<StoredChannelTier>("normal"); const [movePending, startMove] = useTransition(); - const [root, setRoot] = useState(defaultRoot); + const [destId, setDestId] = useState(() => + destinations.some((d) => d.id === defaultLocationId) + ? defaultLocationId + : (destinations[0]?.id ?? CUSTOM), + ); + const [root, setRoot] = useState(""); const [result, setResult] = useState<BulkRelocateResult | null>(null); const [moveError, setMoveError] = useState<string | null>(null); if (slugs.length === 0) return null; + const chosen = destinations.find((d) => d.id === destId) ?? null; + const custom = destId === CUSTOM || chosen === null; const trimmed = root.trim(); + // The id alone for a configured location; the typed root only for `__custom`. + const destination: MoveDestination = custom + ? { kind: "custom", root: trimmed } + : { kind: "location", locationId: chosen.id }; + const named = custom ? trimmed !== "" : true; const busy = priorityPending || movePending; return ( @@ -119,7 +149,9 @@ export function ChannelSelectionDeck({ <button type="button" disabled={busy} - onClick={() => run(() => setChannelTierAction(slugs, tier), onClear)} + onClick={() => + run(() => setChannelTierAction(slugs, tier), onClear) + } className="rounded-md bg-primary px-3 py-1.5 text-xs font-medium text-primary-foreground hover:opacity-90 disabled:opacity-50" > Apply tier @@ -140,24 +172,55 @@ export function ChannelSelectionDeck({ <div className={GROUP}> <span className={EYEBROW}>Media</span> - {/* ONE root for the whole batch: a per-row destination is a per-row - decision, and that is what the channel's own Storage panel is for. - The root lives in the box rather than in the button's label — - `Move media to /mnt/platter/archilyzer-media` is what used to - truncate to "Move media to…" and leave the operator unable to read - where the files were going. */} - <input - type="text" - aria-label="bulk media root" - value={root} - disabled={busy} - onChange={(e) => setRoot(e.target.value)} - placeholder="/mnt/platter/archilyzer-media" - className="w-full max-w-72 md:w-80 md:max-w-none rounded-md border border-border bg-card px-2 py-1 text-xs font-mono disabled:opacity-50" - /> + {/* ONE destination for the whole batch: a per-row destination is a + per-row decision, and that is what the channel's own Storage panel + is for. The destination is on the deck rather than in the button's + label — `Move media to /mnt/platter/archilyzer-media` is what used + to truncate to "Move media to…" and leave the operator unable to + read where the files were going. */} + {destinations.length > 0 && ( + <select + aria-label="bulk media location" + // DERIVED, not the raw state — see StorageStage.tsx. `destId` is + // seeded once; a location deleted on /storage while this page is + // open would leave it naming an id that is no longer an option, + // and a <select> whose value matches nothing silently shows the + // first. `custom` is already true then; this agrees with it. + value={custom ? CUSTOM : destId} + disabled={busy} + onChange={(e) => setDestId(e.target.value)} + className="rounded-md border border-border bg-card px-2 py-1 text-xs disabled:opacity-50" + > + {destinations.map((d) => ( + <option key={d.id} value={d.id}> + {d.label} + </option> + ))} + <option value={CUSTOM}>Another root…</option> + </select> + )} + {chosen && ( + <span + aria-label="bulk media destination" + className="font-mono text-[11px] text-muted-foreground break-all" + > + {chosen.root} + </span> + )} + {custom && ( + <input + type="text" + aria-label="bulk media root" + value={root} + disabled={busy} + onChange={(e) => setRoot(e.target.value)} + placeholder="/mnt/platter/archilyzer-media" + className="w-full max-w-72 md:w-80 md:max-w-none rounded-md border border-border bg-card px-2 py-1 text-xs font-mono disabled:opacity-50" + /> + )} <button type="button" - disabled={busy || trimmed === ""} + disabled={busy || !named} aria-label="move media for selected channels" title="One relocate job per channel, on that channel's own queue. Channels that are already relocated, mid-move, social, or busy are skipped with a reason." onClick={() => @@ -166,7 +229,7 @@ export function ChannelSelectionDeck({ setResult(null); try { setResult( - await bulkRelocateChannelMediaAction(slugs, trimmed), + await bulkRelocateChannelMediaAction(slugs, destination), ); } catch (e) { setMoveError((e as Error).message); diff --git a/editor/app/channels/components/ChannelsTable.tsx b/editor/app/channels/components/ChannelsTable.tsx @@ -7,10 +7,7 @@ import { bandSentence, type OperationBand, } from "yt-dlp-transcript-common/views/pipeline/band"; -import { - BandLegend, - StateBand, -} from "../../components/pipelines/StateBand"; +import { BandLegend, StateBand } from "../../components/pipelines/StateBand"; import type { ChannelGroupSection } from "yt-dlp-transcript-common/views/channelGroupSections"; import { ChannelGroupHeaderRow } from "./ChannelGroupHeaderRow"; import { ChannelAvailabilityButton } from "./ChannelAvailabilityButton"; @@ -23,7 +20,10 @@ import { MediaLocationBadge, type MediaBadgeInput, } from "../../components/MediaLocationBadge"; -import { ChannelSelectionDeck } from "./ChannelSelectionDeck"; +import { + ChannelSelectionDeck, + type BulkDestination, +} from "./ChannelSelectionDeck"; import { tierOrder, type PriorityOperation, @@ -52,7 +52,10 @@ export type ChannelRow = ChannelStat & { // How old this channel's report is. Every count and every band on this row is // projected from that report, so its age is the caveat on all of them — which // is why it belongs beside them rather than on a page of its own. - report: { generatedAt: string | null; state: "current" | "stale" | "missing" }; + report: { + generatedAt: string | null; + state: "current" | "stale" | "missing"; + }; // Where this channel's media physically is, from inspectChannelMedia on the // server. Null for an in-place channel — the overwhelming majority — so the // badge column is empty for them and the two that matter stand out. See @@ -223,7 +226,8 @@ export function ChannelsTable({ columns, sections = null, siteId, - defaultMediaRoot = "", + mediaDestinations = [], + defaultLocationId = "", sites = [], focusLabel = null, }: { @@ -238,9 +242,12 @@ export function ChannelsTable({ // pool. That path is today's flat table, unchanged. sections?: ChannelGroupSection[] | null; siteId?: string; - // settings.storage.mediaRoot, resolved on the server. Seeds the bulk bar's - // root box; "" when no cold root is configured. - defaultMediaRoot?: string; + // The configured storage locations, resolved on the server. The selection + // deck's destination list; empty when none are configured, which leaves the + // deck's free-text root box as the whole control. + mediaDestinations?: BulkDestination[]; + // `settings.storage.defaultLocationId` — which destination the deck opens on. + defaultLocationId?: string; // Every configured site, for the "Focus site" control. Not the same list as // the page's scope selector: a focus is corpus-wide, so it can name a site // whose channels are not the ones on screen. @@ -518,7 +525,8 @@ export function ChannelsTable({ <ChannelSelectionDeck slugs={selectedSlugs} onClear={() => setSelected(new Set())} - defaultRoot={defaultMediaRoot} + destinations={mediaDestinations} + defaultLocationId={defaultLocationId} /> </div> ); @@ -671,7 +679,9 @@ function ChannelTableRow({ <Td ariaLabel={`report age for ${c.slug}`} className={`whitespace-nowrap text-xs tabular-nums ${ - c.report.state === "current" ? "text-muted-foreground" : "text-warning" + c.report.state === "current" + ? "text-muted-foreground" + : "text-warning" }${dim}`} > {c.report.state === "current" @@ -703,7 +713,9 @@ function ChannelTableRow({ <div className="flex items-center gap-2"> <ChannelSyncButton slug={c.slug} disabled={!c.config.url} /> <ChannelAvailabilityButton slug={c.slug} disabled={!c.config.url} /> - <InlineActionButton variant={{ kind: "refreshReport", slug: c.slug }} /> + <InlineActionButton + variant={{ kind: "refreshReport", slug: c.slug }} + /> </div> </Td> </tr> @@ -772,7 +784,10 @@ function SortableTh({ > <span>{label}</span> {indicator && ( - <span aria-hidden="true" className="text-[10px] text-muted-foreground"> + <span + aria-hidden="true" + className="text-[10px] text-muted-foreground" + > {indicator} </span> )} diff --git a/editor/app/channels/lib/moveDestination.ts b/editor/app/channels/lib/moveDestination.ts @@ -0,0 +1,42 @@ +import type { StorageLocation } from "yt-dlp-transcript-common/lib/storageLocations"; + +// WHERE A MOVE IS GOING, as the two panels that start one can say it. +// +// A destination is EITHER a configured location, named by its id, OR a root the +// operator typed. The difference is not cosmetic: for a location the client +// sends the ID AND NOTHING ELSE, and the root is looked up on the server, from +// the same settings.json /storage writes. A form that posted the root alongside +// the id would be a second copy of a fact that already has one home, and the +// copy the server trusted would be the one the browser held when the page was +// rendered — stale the moment a re-point moved the location somewhere else. +// +// NOT IN EITHER ACTIONS FILE, for the reason lib/relocationJob.ts and +// lib/queueForSlugs.ts are not either: both carry "use server", where every +// non-type export is a server action. A shared resolver cannot live in one. +export type MoveDestination = + { kind: "location"; locationId: string } | { kind: "custom"; root: string }; + +export type ResolvedDestination = { root: string } | { error: string }; + +// The root, or the sentence explaining why there is not one. Pure: the caller +// hands it `getSettings().storage.locations`, so this is unit-testable and has +// no opinion about which process is asking. +export function resolveMoveDestination( + dest: MoveDestination, + locations: readonly StorageLocation[], +): ResolvedDestination { + if (dest.kind === "custom") { + const root = dest.root.trim(); + return root ? { root } : { error: "Enter a destination root." }; + } + const id = dest.locationId.trim(); + const found = locations.find((l) => l.id === id); + if (!found) { + return { + error: + `No storage location "${id}" is configured. ` + + `Add it on /storage, or choose a different destination.`, + }; + } + return { root: found.root }; +} diff --git a/editor/app/channels/page.tsx b/editor/app/channels/page.tsx @@ -30,7 +30,7 @@ import { getSettings, type SiteSettings, } from "yt-dlp-transcript-common/lib/settings"; -import { defaultLocationRoot } from "yt-dlp-transcript-common/lib/storageLocations"; +import { locationLabelOfDataDir } from "yt-dlp-transcript-common/lib/storageLocations"; import { buildChannelBands } from "yt-dlp-transcript-common/views/pipeline/buildBands"; import { EXTERNAL_BAND_IDS } from "yt-dlp-transcript-common/views/pipeline/buildBands"; import { @@ -53,9 +53,10 @@ export const metadata: Metadata = { title: "Channels" }; // makes the page load in milliseconds instead of seconds. That trade is only // honest if the page says how old the numbers are — so report the OLDEST // snapshot on screen, and name the channels that have never had one. -function summariseFreshness( - briefs: ReadonlyArray<ChannelBrief>, -): { oldest: string | null; missing: string[] } { +function summariseFreshness(briefs: ReadonlyArray<ChannelBrief>): { + oldest: string | null; + missing: string[]; +} { let oldest: number | null = null; let oldestIso: string | null = null; const missing: string[] = []; @@ -221,6 +222,15 @@ export default async function ChannelsPage({ status: media.status, target: media.target, detail: media.detail, + // WHICH NAMED LOCATION — a pure prefix match of the recorded + // target against the configured roots, done here because the + // settings are here and the table is a client component. NEVER a + // probe: this table draws one badge per row. Undefined for a root + // nobody named, which renders exactly what it rendered before. + locationLabel: locationLabelOfDataDir( + media.target, + settings.storage.locations, + ), } : null, priority: { @@ -256,9 +266,7 @@ export default async function ChannelsPage({ ? buildChannelGroupSections(activeSite, channels, briefs, settings) : null; const shown = new Set(channels.map((c) => c.slug)); - const freshness = summariseFreshness( - briefs.filter((b) => shown.has(b.slug)), - ); + const freshness = summariseFreshness(briefs.filter((b) => shown.has(b.slug))); return ( // ON md+ THE DOCUMENT STOPS SCROLLING. The page is a flex column exactly // the height of the viewport (main carries py-6, hence -3rem) so that the @@ -353,10 +361,16 @@ export default async function ChannelsPage({ columns={columns} sections={sections} siteId={activeSite?.siteId} - // The configured cold root, for the selection deck's root box. Read - // here, not in the client component — the settings page is its one - // writer. - defaultMediaRoot={defaultLocationRoot(getSettings().storage)} + // The configured storage locations, for the selection deck's + // destination select. Read here, not in the client component — + // /storage is their one writer. No probe: the deck names a place and + // sends its id, and whether the drive is mounted is /storage's report. + mediaDestinations={settings.storage.locations.map((loc) => ({ + id: loc.id, + label: loc.label || loc.id, + root: loc.root, + }))} + defaultLocationId={settings.storage.defaultLocationId} sites={sites.map((s) => ({ siteId: s.siteId, title: s.siteTitle || s.siteId, diff --git a/editor/app/components/MediaLocationBadge.tsx b/editor/app/components/MediaLocationBadge.tsx @@ -35,7 +35,12 @@ import type { export type MediaBadgeInput = Pick< ChannelMediaLocation, "status" | "target" | "detail" ->; +> & { + // The name of the storage location this channel's media is on, projected by + // the server that built the row (see `mediaBadgeOf`). A fourth string is + // still cheaper than shipping the location list to every table. + locationLabel?: string; +}; export type MediaBadgeTone = "neutral" | "danger"; @@ -43,6 +48,10 @@ export type MediaBadge = { label: string; title: string; tone: MediaBadgeTone; + // What a table cell draws. Named here rather than recomputed at the call site + // so the accessible name (always the full `label`) and the visible text + // cannot drift apart. + short: string; }; const LABELS: Record<ChannelMediaStatus, string | null> = { @@ -53,17 +62,54 @@ const LABELS: Record<ChannelMediaStatus, string | null> = { inconsistent: "Media inconsistent", }; +// The one-word state, for the compact rendering of a NAMED location: "on +// Platter — unreachable" says more in less room than "Media unreachable" plus a +// hover, because the name is the half the operator already recognises. +const SHORT_STATUS: Record<ChannelMediaStatus, string | null> = { + "in-place": null, + ok: null, + unreachable: "unreachable", + "in-transition": "moving", + inconsistent: "inconsistent", +}; + // Null means "draw nothing" — an in-place channel, or no location at all (a // caller that could not inspect). Both are the same instruction to a renderer. +// +// `locationLabel` IS THE NAME OF THE STORAGE LOCATION the channel's media sits +// on — `locationOfDataDir(config.dataDir, settings.storage.locations)`, resolved +// on the SERVER where the settings are, and passed down as a string. Undefined +// when it sits on none: a root the operator typed by hand, or a corpus with no +// locations configured. +// +// IT ONLY EVER ADDS. The label keeps its existing words and gains " · on +// Platter", so everything that addresses this badge by `media location: Media +// relocated…` still finds it, while the row now names the drive in the +// operator's own vocabulary instead of an absolute path. +// +// NEVER A PROBE. A table draws dozens of these and a probe is up to three +// subprocesses; the name comes from a pure prefix match against the configured +// roots. Whether that drive is reachable *right now* is `media.status`, which +// inspect() already measured per channel — /storage is where a location's own +// availability is reported. export function mediaBadgeOf( media: MediaBadgeInput | null | undefined, + locationLabel?: string, ): MediaBadge | null { if (!media) return null; const label = LABELS[media.status] ?? null; if (!label) return null; const where = media.target ? ` to ${media.target}` : ""; + const named = (locationLabel ?? "").trim(); + const on = named ? ` · on ${named}` : ""; + const shortStatus = SHORT_STATUS[media.status]; return { - label: media.status === "ok" ? `${label}${where}` : label, + label: (media.status === "ok" ? `${label}${where}` : label) + on, + short: named + ? `on ${named}${shortStatus ? ` — ${shortStatus}` : ""}` + : media.status === "ok" + ? (LABELS.ok as string) + : label, // The detail carries the reason; the target alone is the fallback so a // location written by an older inspect() still says where it points. title: media.detail ?? `${label}${where}`, @@ -77,18 +123,24 @@ const TONE_CLASS: Record<MediaBadgeTone, string> = { }; // `compact` drops the target from the label: a table cell wants the four-word -// state, and the full path is one hover away on the title. +// state, and the full path is one hover away on the title. With a named +// location that becomes "on Platter" / "on Platter — unreachable", which is +// shorter AND says more. export function MediaLocationBadge({ media, compact = false, + locationLabel, }: { media: MediaBadgeInput | null | undefined; compact?: boolean; + locationLabel?: string; }) { - const badge = mediaBadgeOf(media); + // The explicit prop wins, for the Storage panel — it holds a whole + // ChannelMediaLocation from inspect(), which has no room for a name. A table + // row carries the name inside its projected `media` instead. + const badge = mediaBadgeOf(media, locationLabel ?? media?.locationLabel); if (!badge) return null; - const label = - compact && media?.status === "ok" ? (LABELS.ok as string) : badge.label; + const label = compact ? badge.short : badge.label; return ( <span title={badge.title} diff --git a/editor/app/page.tsx b/editor/app/page.tsx @@ -3,6 +3,7 @@ import type { Metadata } from "next"; import { getPaths } from "yt-dlp-transcript-common/lib/paths"; import { inspectChannelMedia } from "yt-dlp-transcript-common/lib/channelMedia"; import { getSettings } from "yt-dlp-transcript-common/lib/settings"; +import { locationLabelOfDataDir } from "yt-dlp-transcript-common/lib/storageLocations"; import { getSite, listSiteIds, @@ -84,6 +85,8 @@ export default async function Dashboard({ ), ), ); + // Read once for the whole table, not once per row. + const locations = getSettings().storage.locations; const channels: DashboardChannel[] = rows.map((r) => ({ slug: r.channel.slug, handling: r.channel.config.handling, @@ -98,7 +101,15 @@ export default async function Dashboard({ media: (() => { const m = mediaBySlug.get(r.channel.slug); return m && m.status !== "in-place" - ? { status: m.status, target: m.target, detail: m.detail } + ? { + status: m.status, + target: m.target, + detail: m.detail, + // The named location, matched against the configured roots here + // because the settings are here and the table is a client + // component. A pure prefix match, never a probe. + locationLabel: locationLabelOfDataDir(m.target, locations), + } : null; })(), })); diff --git a/editor/e2e/channel-storage.spec.ts b/editor/e2e/channel-storage.spec.ts @@ -1,4 +1,11 @@ -import { lstat, mkdir, readdir, symlink, writeFile } from "node:fs/promises"; +import { + copyFile, + lstat, + mkdir, + readdir, + symlink, + writeFile, +} from "node:fs/promises"; import { join } from "node:path"; import { test, expect, type Page } from "@playwright/test"; import { baseUrl } from "./baseUrl"; @@ -93,7 +100,7 @@ test("relocate a channel's media to another root, and move it back", async ({ // own would pass for the wrong reason. await page.getByLabel("destination root").fill(root); await expect( - page.getByText("Preview this root to enable the move."), + page.getByText("Preview this destination to enable the move."), ).toBeVisible(); await expect(moveButton).toBeDisabled(); @@ -114,9 +121,11 @@ test("relocate a channel's media to another root, and move it back", async ({ // by the job, on success, after the copy verified. expect((await lstat(dataDir())).isSymbolicLink()).toBe(true); expect( - (await readJson<{ dataDir?: string }>( - `test-transcripts/channels/${SLUG}/config.json`, - )).dataDir, + ( + await readJson<{ dataDir?: string }>( + `test-transcripts/channels/${SLUG}/config.json`, + ) + ).dataDir, ).toBe(target); // The source was reclaimed: no parked copy left holding a second copy of the // channel on the volume the move exists to free, and no marker. @@ -164,9 +173,11 @@ test("relocate a channel's media to another root, and move it back", async ({ // A real directory again, the config field gone, and the target reclaimed. expect((await lstat(dataDir())).isDirectory()).toBe(true); expect( - (await readJson<{ dataDir?: string }>( - `test-transcripts/channels/${SLUG}/config.json`, - )).dataDir, + ( + await readJson<{ dataDir?: string }>( + `test-transcripts/channels/${SLUG}/config.json`, + ) + ).dataDir, ).toBe(undefined); const back = await page.request.get( `${baseUrl}/api/channels/${SLUG}/videos/${VIDEO}/files/transcript.en.vtt`, @@ -212,7 +223,9 @@ test("the /channels bulk move queues one job per channel and skips the rest", as test.setTimeout(120_000); await resetData("one-youtube-channel-with-data"); const root = testInfo.outputPath("bulk-root"); + const warm = testInfo.outputPath("warm-root"); await mkdir(root, { recursive: true }); + await mkdir(warm, { recursive: true }); // THE DEFAULT ROOT, set the way the operator does. The bulk bar's box is // seeded from it, so this is also what asserts the settings value reaches the // client. The other four keys mirror fixtures/test-settings.default.json, @@ -223,13 +236,16 @@ test("the /channels bulk move queues one job per channel and skips the rest", as minFreeDiskGB: 0, verifyAvailabilityBeforeClean: false, syncScheduler: { fullSweepIntervalMinutes: 0 }, - // ONE LOCATION, and it is the default — the shape the old single - // `mediaRoot` string migrates into. `defaultLocationRoot` is what the - // Storage panel and the bulk bar read, so this is still what asserts the - // settings value reaches the client. + // TWO LOCATIONS, and the default is NOT the one this spec moves to. The + // deck opens on `warm` and the spec picks `cold` by name, which is what + // asserts the select's value reaches the action — the old single-location + // fixture could not tell "the id was sent" from "the default was used". storage: { - locations: [{ id: "cold", label: "Cold", root, autoRepoint: false }], - defaultLocationId: "cold", + locations: [ + { id: "warm", label: "Warm", root: warm, autoRepoint: false }, + { id: "cold", label: "Cold", root, autoRepoint: false }, + ], + defaultLocationId: "warm", }, }); @@ -241,21 +257,30 @@ test("the /channels bulk move queues one job per channel and skips the rest", as "WEBVTT\n\n00:00.000 --> 00:01.000\nhello\n", ); await writeChannelConfig(PRE, { dataDir: preTarget }); - await symlink(preTarget, resolvePath(`test-transcripts/channels/${PRE}/data`)); + await symlink( + preTarget, + resolvePath(`test-transcripts/channels/${PRE}/data`), + ); await page.goto("/channels"); // One badge before the move: the channel that is already on the root. const badges = page.getByLabel(/^media location: Media relocated/); await expect(badges).toHaveCount(1); - // The bar only exists with a selection — it is a selection bar, and a root box - // with nothing to apply it to is a control that cannot do anything. - await expect(page.getByLabel("bulk media root")).toHaveCount(0); + // The deck only exists with a selection — it is a selection deck, and a + // destination with nothing to apply it to is a control that cannot do + // anything. + const destination = page.getByLabel("bulk media location"); + await expect(destination).toHaveCount(0); await page.getByLabel("select all channels").check(); await expect(page.getByLabel(`select ${SLUG}`)).toBeChecked(); await expect(page.getByLabel(`select ${PRE}`)).toBeChecked(); - // And it carries the configured root with no typing. - await expect(page.getByLabel("bulk media root")).toHaveValue(root); + // It opens on the DEFAULT location, and names every configured one. No root + // is typed anywhere in this test: the deck sends an id. + await expect(destination).toHaveValue("warm"); + await expect(page.getByLabel("bulk media destination")).toHaveText(warm); + await destination.selectOption("cold"); + await expect(page.getByLabel("bulk media destination")).toHaveText(root); await page.getByLabel("move media for selected channels").click(); const result = page.getByLabel("bulk media move result"); @@ -279,13 +304,21 @@ test("the /channels bulk move queues one job per channel and skips the rest", as { timeout: 90_000 }, ) .toBe(2); + // AND THE BADGE NAMES THE PLACE. Both channels are under the `cold` root, so + // the row says the operator's own word for that drive rather than a 60-column + // absolute path nobody reads. The accessible name still carries the full + // sentence, which is what the locator above matches on. + await expect(badges.first()).toHaveText("on Cold"); + await expect(badges.nth(1)).toHaveText("on Cold"); // The moved channel: config records the target under the tmp root, `data/` is // a link, and the videos list still lists the video through it. expect( - (await readJson<{ dataDir?: string }>( - `test-transcripts/channels/${SLUG}/config.json`, - )).dataDir, + ( + await readJson<{ dataDir?: string }>( + `test-transcripts/channels/${SLUG}/config.json`, + ) + ).dataDir, ).toBe(join(root, SLUG, "data")); expect((await lstat(dataDir())).isSymbolicLink()).toBe(true); await page.goto(channelVideos(SLUG)); @@ -293,13 +326,16 @@ test("the /channels bulk move queues one job per channel and skips the rest", as // The skipped channel was not touched: same target, still a link, no marker. expect( - (await readJson<{ dataDir?: string }>( - `test-transcripts/channels/${PRE}/config.json`, - )).dataDir, + ( + await readJson<{ dataDir?: string }>( + `test-transcripts/channels/${PRE}/config.json`, + ) + ).dataDir, ).toBe(preTarget); expect( - (await lstat(resolvePath(`test-transcripts/channels/${PRE}/data`))) - .isSymbolicLink(), + ( + await lstat(resolvePath(`test-transcripts/channels/${PRE}/data`)) + ).isSymbolicLink(), ).toBe(true); expect( await pathExists(`test-transcripts/channels/${PRE}/.relocating.json`), @@ -346,7 +382,9 @@ test("a bulk move puts every job on one queue and skips a channel with nothing t url: "https://www.youtube.com/@second/videos", }); await mkdir( - resolvePath(`test-transcripts/channels/${SECOND}/data/20240103_second12345`), + resolvePath( + `test-transcripts/channels/${SECOND}/data/20240103_second12345`, + ), { recursive: true }, ); await writeFile( @@ -401,3 +439,159 @@ test("a bulk move puts every job on one queue and skips a channel with nothing t await expect(rows.nth(i)).not.toContainText("channel:"); } }); + +// THE DESTINATION IS A NAME NOW, not a path retyped per channel. +// +// Two locations are configured and the DEFAULT IS NOT THE ONE THIS SPEC MOVES +// TO, which is the whole point: the panel opens on `warm`, the spec picks +// `cold` by name, and what reaches the server is the ID — the root is looked up +// there, from the same settings.json /storage writes. A fixture with one +// location could not tell "the id was sent" from "the default was used". +// +// The free-text box is GONE while a location is picked. It is one option away +// ("Another root…") and it is the whole control on a corpus with no locations +// configured, which is what the first test in this file exercises. +test("the Storage panel moves to a location picked by name", async ({ + page, +}, testInfo) => { + test.setTimeout(90_000); + await resetData("one-youtube-channel-with-data"); + const warm = testInfo.outputPath("warm-root"); + const cold = testInfo.outputPath("cold-root"); + await mkdir(warm, { recursive: true }); + await mkdir(cold, { recursive: true }); + await writeSettings({ + adminTitle: "Test Admin", + minFreeDiskGB: 0, + storage: { + locations: [ + { id: "warm", label: "Warm", root: warm, autoRepoint: false }, + { id: "cold", label: "Cold", root: cold, autoRepoint: false }, + ], + defaultLocationId: "warm", + }, + }); + await generateReport(page, SLUG); + await quiet(page); + + await page.goto(channelStage(SLUG, "storage")); + const destination = page.getByLabel("destination location"); + await expect(destination).toHaveValue("warm"); + // No root box while a location is picked — there is nothing to type. + await expect(page.getByLabel("destination root")).toHaveCount(0); + // Both names are offered, with each drive's state from the server-side probe. + await expect(destination).toContainText("Warm"); + await expect(destination).toContainText("Cold"); + + await destination.selectOption("cold"); + const target = join(cold, SLUG, "data"); + await page.getByRole("button", { name: "Preview" }).click(); + const preview = page.getByLabel("relocation preview"); + await expect(preview).toBeVisible({ timeout: 15_000 }); + // The preview names the target the SERVER resolved from the id. + await expect(preview).toContainText(target); + + const moveButton = page.getByRole("button", { name: "Move media" }); + await expect(moveButton).toBeEnabled(); + await moveButton.click(); + await expect(page.getByLabel("Move media output")).toContainText("Moved", { + timeout: 60_000, + }); + expect( + ( + await readJson<{ dataDir?: string }>( + `test-transcripts/channels/${SLUG}/config.json`, + ) + ).dataDir, + ).toBe(target); + + // AND THE BADGE READS THE NAME. `on Cold`, not sixty columns of absolute + // path — while the accessible name keeps the full sentence, which is what + // every other assertion in this file matches on. + await page.goto("/channels"); + const badge = page.getByLabel(/^media location: Media relocated/); + await expect(badge).toHaveText("on Cold"); + await expect(badge).toHaveAttribute("aria-label", new RegExp(`on Cold$`)); +}); + +// FINISHING AN INTERRUPTED MOVE, rather than throwing away what already copied. +// +// `relocateChannelMedia` has always resumed a same-direction marker; nothing in +// the editor could reach it, so the only offered way out of a killed copy was +// Clear marker and a full re-copy — for the omnimirror incident, 131 GB already +// correctly on the far side. +// +// The state is BUILT ON DISK rather than produced by killing a real run: a +// marker at phase "copy" plus a partial copy at the target is exactly what a +// killed rsync leaves, and it is three filesystem calls instead of a job that +// has to be interrupted at the right instant. What is under test is the resume, +// not the interruption. +test("Resume move finishes an interrupted move and clears its marker", async ({ + page, +}, testInfo) => { + test.setTimeout(90_000); + await resetData("one-youtube-channel-with-data"); + const root = testInfo.outputPath("resume-root"); + const target = join(root, SLUG, "data"); + // The partial copy: the video dir is there with ONE of its two files. + await mkdir(join(target, VIDEO), { recursive: true }); + await copyFile( + resolvePath( + `test-transcripts/channels/${SLUG}/data/${VIDEO}/transcript.en.vtt`, + ), + join(target, VIDEO, "transcript.en.vtt"), + ); + await writeFile( + resolvePath(`test-transcripts/channels/${SLUG}/.relocating.json`), + JSON.stringify( + { + target, + direction: "out", + startedAt: new Date().toISOString(), + phase: "copy", + }, + null, + 2, + ), + ); + + // Same wait the other cases make: the Resume action refuses while the channel + // has a queued or running job, and resetData's cache invalidation can leave + // one behind for a moment. + await quiet(page); + await page.goto(channelStage(SLUG, "storage")); + // In transition, with both escapes offered and neither move available. + await expect(page.getByLabel(/^media location: Media moving/)).toBeVisible(); + await expect( + page.getByRole("button", { name: "Clear marker" }), + ).toBeVisible(); + const resume = page.getByRole("button", { name: "Resume move" }); + await expect(resume).toBeEnabled(); + await resume.click(); + + // THE CONTROLLER SAYS SO ITSELF. The root came from the marker's target, not + // from any form on the page, and the job's summary line is where "this was a + // continuation" is stated. + await expect(page.getByLabel("Resume move output")).toContainText( + "resumed an interrupted move", + { timeout: 60_000 }, + ); + + // Finished like any other move: link, config, no marker. + expect( + await pathExists(`test-transcripts/channels/${SLUG}/.relocating.json`), + ).toBe(false); + expect((await lstat(dataDir())).isSymbolicLink()).toBe(true); + expect( + ( + await readJson<{ dataDir?: string }>( + `test-transcripts/channels/${SLUG}/config.json`, + ) + ).dataDir, + ).toBe(target); + // And the bytes are readable through the same URL as ever. + const file = await page.request.get( + `${baseUrl}/api/channels/${SLUG}/videos/${VIDEO}/files/transcript.en.vtt`, + ); + expect(file.status()).toBe(200); +});