commit 3b54ab45e68025ba50ba7711025284fe451c3290
parent 4e9720332aed1840a8a24c00586c7169b023806e
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Sat, 26 Sep 2026 03:09:12 -0400
plans: slice L2 as shipped — soft block backs off, bounded storage-pass wait, cancelReason on /jobs, counted safeRevalidate skips
The record in release-10.md (before Rollout): what each item changed and
why, the soft-block strings and where they came from, the 60 s bound and its
arithmetic, the commit table, gates (tsc x5; common 1,924, editor unit 85,
scripts 162 + 1 skip, mcp 219; editor build x2; e2e 37 specs 213/2 then the
two failures alone 6/6 — both pre-existing flakes), found and left.
FACTS: the three release 9 bullets this slice makes stale are amended.
editor/CHANGELOG.md: a new [Unreleased] above [0.9.0], four bullets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
3 files changed, 162 insertions(+), 3 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,5 +1,11 @@
# Changelog
+## [Unreleased]
+- **YouTube's "try again later" block now backs off instead of reading as a deleted video.** When YouTube rate-limits a session it answers "This content isn't available, try again later." The editor read that as a removed video: the download moved straight on to the next video (into the same block), no cooldown was recorded, and the video was set aside as deleted, so later download runs skipped it. It is now handled like an HTTP 429. A download stops its batch and records the platform cooldown that the auto-download runner and Sync honour, and the runner defers the video for 6 hours. The availability check records a temporary error instead of "deleted". A metadata scan stops (retrying once with cookies when the channel has them) instead of recording the video as gone. Videos that really are gone ("Video unavailable", removed by the uploader, a terminated account, Rumble's 410) are still recorded as deleted.
+- **Jobs a restart left queued are settled even when a drive hangs.** At boot the editor settles those jobs after it has checked where its storage locations are. A hung network mount could stall that check forever, and the jobs then stayed "queued" on `/jobs`. The settling now waits at most 60 seconds, logs `[boot] storage pass still running after 60 s …` and carries on. The storage check keeps running and logs when it ends.
+- **`/jobs` says why a job was cancelled at boot.** A job the boot settled shows its reason under its status on `/jobs` and as *Cancelled because* on its own page: for example "server restarted; the scheduler re-derives syncs" or "superseded by a newer queued job (…)". The reason used to be only in the job's log.
+- **The server log says how often a queued job skips its page refresh.** When a queued job finishes outside any request, the editor skips its page refresh and notes it in the log. The note used to appear once and never again. Now the first one after a quiet spell is logged at once, any more in the next 10 minutes are counted, and one line at the end gives the count, with a running total.
+
## [0.9.0] - 2026-09-26
- **Every page now has a ground and an accent to choose, and the five theme families are gone.** The theme menu (the palette button beside the quick toggle, in the editor's sidebar and in the header of every published site, the hub and the homepage) has two groups. **Base** is System, Light, Sepia or Dark; Sepia is new, a warm paper ground for long reading. **Accent** is Signal, Brass, Vermilion, Violet, Sakura, Blue or Green, with the site's own tagged *default*; a site with a custom hex offers it first as *Site colour*. The quick toggle cycles System → Light → Sepia → Dark. A published site opens on the reader's system setting, in the accent its site form sets. The hub and the homepage open on Dark, in Signal, even with JavaScript off, and the editor follows the system, in Signal. Each accent has a value for each ground that reads at 4.5:1, and a custom hex is darkened or lightened per ground to match. A reader's accent is remembered only while it differs from the site's: picking the site's own again forgets it, so the reader follows the site if its accent changes later. Base, Archive, Selenized, Swiss and Archilyzer are gone. A choice made before this update carries over once: light stays light (Archive light becomes Sepia), dark stays dark and system stays system; the family itself is dropped. Headings are Archivo, text is IBM Plex Sans and figures are IBM Plex Mono everywhere, with one corner radius. Success, warning and other status text reads at 4.5:1 on its own tinted fill on every ground; on Light, success and warning are a shade deeper than before for it. Chart colours are fixed per ground and never follow the accent; the third is a violet, well clear of the red that marks a recording as gone. The phone's browser bar takes the page's ground, not the accent. Needs a rebuild and deploy of every site, the hub and the homepage.
- **Every site, the hub, the homepage and the editor wear the new Found-line mark, and a site's header splits its wordmark.** The mark is four transcript lines on a rounded square, the second lit and carrying a play head. The favicon, app icons and touch icon are no longer committed files: each build draws them from the mark and writes `/icons/icon.svg`, `maskable.svg`, `icon-32.png`, `icon-192.png`, `icon-512.png`, `maskable-512.png`, `apple-touch-icon.png` and `/favicon.ico` (16, 32 and 48 px). A site's icons are an ink tile lit with its own accent (Signal when it sets none; a custom colour is lightened until it reads on the tile). The hub, the homepage and the editor use the parent mark, bone on slate. The site header shows the mark and the header title split at the site's **Wordmark lead**, the lead heavy and the rest light (Jer|alyzer); with no lead the whole title is heavy. The header mark's lit line follows the reader's accent; the icons keep the site's. The footer's "Built with Archilyzer" has the small parent mark in front of it, outside the link. The homepage header shows the parent mark and "Archi|lyzer", no longer in spaced capitals. An installed app's title bar is the dark ground (`#0c0a08`) and its splash the icon's tile, where a site's used to be the old default blue. The service workers fetch icons fresh whenever the reader is online and keep a copy for offline, so an installed app picks up the new icons, and any later accent change, on its next online visit. Their shell cache is renamed to `shell-v2`: readers' offline channel downloads are kept, but the old cached pages and scripts go, so an installed app opens offline again only after one more online visit. The editor's sidebar shows the parent mark beside the admin title, and the editor now has a favicon. Needs a rebuild and deploy of every site, the hub and the homepage.
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -6417,13 +6417,24 @@ Line numbers are `plans/FACTS.md` lines at `e172749b`, before this record's in-p
request store and Next throws `Invariant: static generation store missing`, which marked the job
`failed` after its work was done. Job bodies and `onDone`/`afterRun` hooks call
`safeRevalidate(paths, tags?)` (`editor/app/lib/safeRevalidate.ts`): it swallows exactly that
- invariant (one warning per bundle), rethrows anything else. Request-context server actions keep
- the plain call.
+ invariant, rethrows anything else. Request-context server actions keep the plain call.
+ **Release 10 (L2):** every skip is counted by one `SkipReporter` per process
+ (`globalThis.__yttSafeRevalidateSkips__`): the first after a quiet spell is logged at once and
+ opens a 10 min window (`SKIP_REPORT_WINDOW_MS`), the rest are reported as one
+ `[safeRevalidate] N more skip(s) …; T since this process started.` line when it ends. It was one
+ warning per module instance.
- **`--sleep-requests 1` is a YouTube platform arg** (`common/ytdlp/platformArgs.mjs`), like
Rumble's; a channel's `ytdlpExtraArgs` comes after and wins.
- **`runManagedDownloads` skips the inter-download sleep only for `skipped-filtered`** (no media
request was made). Every failure class still sleeps — YouTube's soft block "try again later"
classifies as `deleted`/per_video, so skipping there would hammer a soft block.
+ **Release 10 (L2): no longer.** `isSoftBlock` (`common/lib/availability.ts`, "isn't available,
+ try again later", either apostrophe) is checked first: `parseUnavailableFromStderr` → `error`,
+ `classifyDownloadFailure` → `rate_limit` even over a per-video class. The batch records the
+ platform cooldown and aborts; the metadata scan stops as a `soft-block` block. Every failure
+ still sleeps. No live sidecar held the string (2026-09-26 read-only grep of 136,401
+ `availability.json`/`download-outcome.json` and every `metadata-scan.json`), but an availability
+ check stores no text for `deleted`, so an earlier misread there cannot be found after the fact.
- **A job record carries `progressAt`** (live-only, stamped by the one progress writer and by task
completion). `/jobs` marks `possibly-stalled` only when tasks are idle AND progress has not moved
for 10 min (`common/views/jobRows.ts`).
@@ -6432,7 +6443,11 @@ Line numbers are `plans/FACTS.md` lines at `e172749b`, before this record's in-p
metas queued > 24 h before the boot are cancelled as stale; of the rest only the newest per kind +
channel + bucket + params is re-queued via the retry path; the others are cancelled as superseded;
`ARCHILYZER_IDLE_BOOT` cancels everything. A `cancelReason` field on the meta says why. First live
- boot: re-queued 1, cancelled 390.
+ boot: re-queued 1, cancelled 390. **Release 10 (L2):** the wait on the storage pass is bounded
+ (`settleAfterStoragePass`, `STORAGE_PASS_WAIT_MS` 60 s; a `stat`/statfs on a hung mount has no
+ timeout of its own) and logged when it fires; the storage pass is not cancelled. `cancelReason`
+ is drawn on `/jobs` (`JobListEntry` → `JobRowView.cancelReason`, only on a `cancelled` meta;
+ `data-testid="cancel-reason"` on the row and the job page's "Cancelled because" cell).
- **The hub embeds `public/hub-summary.json`** at `compose:hub` (`common/controller/poolSummary.ts`
shared with `compose-homepage`; `common/lib/hubSummary.ts` projects `official` + per-site figures
from the same `buildHomepageSummary`). Optional end to end: missing/404/malformed → cards without
diff --git a/plans/release-10.md b/plans/release-10.md
@@ -222,6 +222,144 @@ safeRevalidate helper.
- **L2:** the editor `next build`, and the editor e2e specs for `/jobs`, boot and downloads (the
full editor suite runs at integration). Unit tests for the classification (6) and the timeout (7).
+### Slice L2, as shipped — runner lows (2026-09-26)
+
+Branch `r10/runner-lows` off `main` `5dfc9c3a`, one Opus implementer, beside L1 (hub lows) and brand
+S4. Items 6–9 of "Planned: the lows". Nothing on disk moves, and no settings, site or channel key
+changes; `JobListEntry.cancelReason` and `JobRowView.cancelReason` are additive and optional.
+
+**6 — YouTube's soft block backs off.** YouTube answers a session it is rate-limiting with the
+playability reason "This content isn't available, try again later." Its "content isn't available"
+matched `parseUnavailableFromStderr`'s `deleted` group, so it was `per_video`: no platform cooldown,
+the batch kept requesting into the block, and the video joined `EXCLUDED_FROM_DOWNLOAD` as gone.
+`isSoftBlock` (`common/lib/availability.ts`, `/isn['’]?t available,? try again later/i`) is now
+checked first in both functions: `parseUnavailableFromStderr` → `error` (transient: not excluded, not
+pinned as gone), `classifyDownloadFailure` → `rate_limit`, even over a per-video class a caller
+already holds. What that drives is the existing path, unchanged: `runManagedDownloads` records the
+platform cooldown and aborts the batch; the auto runner's `applyUnitOutcome` backs the platform off
+and defers the video 6 h; `fetchWindowManaged` records the cooldown; the metadata scan stops the
+pass (its block `kind` is `"soft-block"` for this line, which only changes the log wording; the one
+cookie retry every block gets is unchanged). "Try again later" is what marks it. A bare "Video
+unavailable", "…removed by the uploader", a terminated account, a ToS removal and Rumble's `HTTP Error
+410: Gone` stay `deleted` / `per_video`, and so does "This content isn't available." with no retry
+advice. Every failure still sleeps between downloads; the release 9 comments that gave the soft block
+as the reason now say why the pace stays anyway.
+- **The strings, and where they came from.** No sidecar on the live corpus holds one (read-only grep,
+ 2026-09-26: 136,401 `availability.json` / `download-outcome.json` files and every
+ `metadata-scan.json`, zero hits for "try again later", "content isn't available" or "rate-limited by
+ YouTube"). The tests use:
+ - `ERROR: [youtube] H64QQZuw-aA: This content isn't available, try again later.` — the line
+ reported in yt-dlp issue #11426 (what a yt-dlp before #12958 prints);
+ - `ERROR: [youtube] <id>: This content isn't available, try again later. The current session has
+ been rate-limited by YouTube for up to an hour. It is recommended to use `-t sleep` to add a delay
+ between video requests to avoid exceeding the rate limit. For more information, refer to
+ https://github.com/yt-dlp/yt-dlp/wiki/Extractors#this-content-isnt-available-try-again-later` —
+ built by `yt_dlp/extractor/youtube/_video.py` (yt-dlp #12958, commit `26feac3dd`) in the build the
+ editor runs (`~/Projects/yt-dlp-patched`, 2026.08.19); "Your account" instead of "The current
+ session" when cookies were passed;
+ - the bare line with a right single quote.
+ yt-dlp's wiki puts the limit at ~300 videos/hour for a guest session, ~2,000 signed in.
+
+**7 — the boot pass's wait on the storage pass is bounded.** `settleAfterStoragePass`
+(`common/jobs/bootQueuedJobs.ts`) waits for the storage pass for at most `STORAGE_PASS_WAIT_MS`,
+then settles anyway; `instrumentation.ts` calls it in place of `storagePass.then(settle…)`.
+- **60 s, and why.** A healthy pass takes milliseconds: `findmnt -T` answers in under 10 ms on this
+ machine. Its bounded worst case is two or three `findmnt`s per location at `FINDMNT_TIMEOUT_MS`
+ (3 s) — identity and fstab, or where the uuid is mounted — plus, for a location being re-pointed,
+ the preflight's second probe and a stat per channel on it: about 12 s a location. Nothing bounds
+ the `stat` of a location's root or its statfs, and on a hung network mount they never return. 60 s
+ covers several locations at their worst; past it the pass is stuck on a syscall, and waiting buys
+ nothing.
+- **When it fires:** `[boot] storage pass still running after 60 s; settling queued jobs without it
+ (a hung mount? a job re-queued for a channel it has not re-pointed yet will be refused as
+ unreachable, and /jobs will say so)`, and, when the pass finally ends, `[boot] storage pass
+ finished N s after the queued-job pass began waiting (it stopped waiting at 60 s)`. The race is
+ over a derived promise, so the storage pass is never cancelled and a re-point it enqueued runs to
+ the end. A pass that throws counts as finished.
+
+**8 — `/jobs` shows `cancelReason`.** `JobListEntry.cancelReason` is read from the meta only when its
+status is `cancelled`; `fromEntry` copies it to `JobRowView.cancelReason`. `JobRow.tsx` draws it
+wherever the cancelled pill is: under the pill in the table's Status cell, at the end of a card row's
+heading, and as the pill's `title` on a compact row. The job page (`/jobs/[id]`) adds a full-width
+**Cancelled because** cell. All carry `data-testid="cancel-reason"`. Everything is added; no existing
+label, text or test id changed (`editor/e2e` had no `cancel-reason`, `Cancelled because` or detail-page
+cell assertions to collide with). A live row never has one: only the boot pass writes it, to metas
+from before the boot.
+
+**9 — the safeRevalidate warning is counted.** `SkipReporter` (`editor/app/lib/safeRevalidate.ts`),
+one per process on `globalThis.__yttSafeRevalidateSkips__` (LOW-4: one per module instance was not
+even once). A skip is one `safeRevalidate` call with no request store, however many targets it names.
+The first after a quiet spell is logged at once with its paths, under the old prefix, and opens a
+`SKIP_REPORT_WINDOW_MS` (10 min) window. The rest of the window are counted, and one line at its end
+reports them: `[safeRevalidate] N more skip(s) with no request store in the last 10 min (latest:
+…); T since this process started.` So at most two lines per window. The end-of-window timer is
+unref'd. It is not reset by `/api/test/invalidate-cache`: it decides only how many log lines a skip
+costs, and nothing reads that log.
+
+| sha | what |
+|---|---|
+| `deaff1d9` | 6: `isSoftBlock`; `parseUnavailableFromStderr` → `error`, `classifyDownloadFailure` → `rate_limit`; the scan's `soft-block` kind; stale comments; `availability.test.ts` +3, `managedDownloadsSleep.test.ts` +1 |
+| `d2ff1411` | 7: `STORAGE_PASS_WAIT_MS`, `waitForStoragePass`, `settleAfterStoragePass`, `instrumentation.ts`; `bootQueuedJobs.test.ts` +5 |
+| `904ffc4e` | 8: `cancelReason` through `listJobs` → `fromEntry` → `JobRow` and the job page; `listJobs.test.ts` +1, `jobRows.test.ts` +1, `jobs-filters.spec.ts` +1 |
+| `11446fba` | 9: `SkipReporter`, one per process; `safeRevalidate.test.ts` 3 → 9 |
+| `fc2ce63c` | 8: a card row's reason at the end of its heading, not beside the pill |
+| _this_ | `plans:` this record, FACTS (three release 9 bullets amended), the editor `[Unreleased]` bullets |
+
+**Gates**, all from the worktree root.
+- **tsc** (`pnpm -r --no-bail --workspace-concurrency=1 exec tsc --noEmit`) clean before every code
+ commit (`l2-tsc-1..5.log`).
+- **common 1,924/1,924** (1,913 + 11: availability +3, managedDownloadsSleep +1, bootQueuedJobs +5,
+ listJobs +1, jobRows +1); **editor unit 85/85** (79 + 6, `safeRevalidate.test.ts`);
+ **`test:scripts` 162 + 1 skip of 163**; **mcp 219/219** (`l2-units.log`, on `11446fba`).
+- **Editor build** `pnpm --filter editor exec next build` ok: 89 s on `11446fba` (`l2-build.log`) and
+ 52 s on the code tip `fc2ce63c` (`l2-build-2.log`).
+ `export/public` in the worktree: the generated paths linked from the primary, `sw.js` a plain copy.
+- **EDITOR e2e.** Spec list `l2-specs.txt`, 37 specs: every spec that opens `/jobs` or its jobs
+ (`jobs jobs-filters jobs-retry jobs-channel jobs-active-order jobs-reorder jobs-batch-tasks-drain
+ job-stream-cancel cancel queue queues dashboard widget channel-work`), the boot and runner specs
+ (`auto-queue lane-runner ops-api backfill`), downloads and their classification (`availability
+ availability-backfill metadata-scan-softblock metadata-scan-botcheck pacing retry-bucket
+ rumble-sweep fetch-window undownloaded download-part-files partial-downloads-bucket title-filter
+ pre-clean-availability maybe-missing`), sync (`sync-deep sync-break-on-existing scheduler`), a
+ revalidating job (`truncated-check`) and `storage-locations`, all `.spec.ts`. A content grep for
+ the prompt's five words matches 122 of 123 specs ("sync" is in "async"), so the list is by name and
+ by what each spec drives.
+ - Run 1 (`l2-e2e-1.log`, on `fc2ce63c`, 7 min in the queue behind L1): **213 passed, 2 failed,
+ 13.9 min** (215 tests). `ops-api.spec.ts:447` (B1) printed the new first line live: `[safeRevalidate]
+ no request store …; skipped revalidating /channels/slow-b, /channels, /operations/[id] (page),
+ /cleanup, /. … 1 skip(s) since this process started; more in the next 10 min are counted and
+ reported together.`
+ - `channel-work.spec.ts:208`, 233 ms: `EEXIST: file already exists, mkdir
+ '…/editor/test-transcripts/channels'` in `resetData`'s fixture copy (`helpers.ts:69`), before the
+ test touched the page — the pre-existing fixture-reset race S3 met in `pipeline.spec.ts:164`.
+ - `lane-runner.spec.ts:203`, 7.9 s: `apiRequestContext.get: read ECONNRESET` on a
+ `/api/auto-queue/status` poll; the next four lane-runner tests passed.
+ - Rerun alone, `channel-work.spec.ts:208 lane-runner.spec.ts:203 --repeat-each 3`
+ (`l2-e2e-rerun.log`, 4m55s in the queue behind S4): **6 passed, 0 failed, 1.0 min**. Both are
+ **pre-existing flakes**, not this slice: the EEXIST is the fixture-reset race in the helper, and the
+ reset was a socket dropped mid-poll by the dev server. The slice's diff touches neither
+ (`git diff --stat 5dfc9c3a HEAD -- editor/e2e/helpers.ts editor/app/api/auto-queue
+ common/controller/autoRunner.ts common/jobs/autoQueueState.ts editor/app/channels` is empty).
+ - The primary checkout's `export/public/sw.js` was not written (10,027 B, md5 `55cbf381…`, mtime
+ 2026-09-25 20:47:37, before and after).
+- **Numbers tools: none.**
+
+**Found and left.**
+- **A soft-blocked prefetch still makes the primary attempt.** `downloadOneManaged` goes on to attempt
+ 1 after any failed metadata prefetch, whatever the class (a 429 and a deleted video alike): one more
+ request into the block before the batch stops. Stopping there would back off more, but it changes
+ the per-video flow for every class, so it is not in this slice.
+- **The availability check has no rate-limit stop.** It now records the soft block as `error` (was
+ `deleted`), but it keeps probing the rest of its list. It has no cooldown path to plug into.
+- **Earlier misreads cannot be found.** An availability check stores no stderr for a `deleted`
+ result, so a soft block it read before this release is indistinguishable from a real removal. None
+ shows in the download outcomes or scan stores (above).
+- **No e2e drives the soft block through a download.** The fake yt-dlp has no sentinel for the
+ "try again later" line; the chain is pinned by the classifier tests and by
+ `managedDownloadsSleep.test.ts`, which feeds the real line through the classifier into the batch
+ loop.
+- `plans/STATE.md` still lists items 6–9 as open; it is shared with L1, so the merge updates it.
+
## Rollout
Nothing is rolled out. The live :3001 editor still runs `0213f6c8` (the pre-brand build); the five
