commit 43be5f96bb18b7a7987aa1a3a5d7501c71f9a226
parent cb5a826548507442c42eff6d75fae8883187ec58
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Sat, 26 Sep 2026 03:59:48 -0400
plans: L2 re-read fix — every unprobed pre-clean suspect is unverified; resolveMaybeMissingState's error-as-available is a new low
release-10.md, slice L2: the Review fixes paragraph says the guard now
covers any suspect missing from `probedIds` (`cb5a8265`), with its test and
gates (tsc; common 1,934; cleanup + pre-clean e2e 12/12, 1.1 min); found and
left strikes the no-URL gap and records `resolveMaybeMissingState` reading
an `error` probe as `available` as a new, display-only low needing its own
export check. FACTS and the soft-block [Unreleased] bullet follow.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
3 files changed, 40 insertions(+), 13 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,7 +1,7 @@
# Changelog
## [Unreleased]
-- **YouTube's "try again later" block now backs off instead of reading as a deleted video.** When YouTube rate-limits a session it answers "This content isn't available, try again later." The editor read that as a removed video: the download moved straight on to the next video (into the same block), no cooldown was recorded, and the video was set aside as deleted, so later download runs skipped it. It is now handled like an HTTP 429. A download stops its batch and records the platform cooldown that the auto-download runner and Sync honour, and the runner defers the video for 6 hours. A metadata scan stops (retrying once with cookies when the channel has them) instead of recording the video as gone. The availability check records a temporary error instead of "deleted", and stops probing for the rest of that run. Every rate limit also ends things sooner now: a download whose metadata request is refused no longer goes on to try the download itself, and the availability check stops at the first refused probe instead of asking about every remaining video. When the check that runs before cleaning audio is cut short this way, the cleanup keeps the audio of every video it did not reach, rather than trusting what an older check said about it. Videos that really are gone ("Video unavailable", removed by the uploader, a terminated account, Rumble's 410) are still recorded as deleted.
+- **YouTube's "try again later" block now backs off instead of reading as a deleted video.** When YouTube rate-limits a session it answers "This content isn't available, try again later." The editor read that as a removed video: the download moved straight on to the next video (into the same block), no cooldown was recorded, and the video was set aside as deleted, so later download runs skipped it. It is now handled like an HTTP 429. A download stops its batch and records the platform cooldown that the auto-download runner and Sync honour, and the runner defers the video for 6 hours. A metadata scan stops (retrying once with cookies when the channel has them) instead of recording the video as gone. The availability check records a temporary error instead of "deleted", and stops probing for the rest of that run. Every rate limit also ends things sooner now: a download whose metadata request is refused no longer goes on to try the download itself, and the availability check stops at the first refused probe instead of asking about every remaining video. When the check that runs before cleaning audio is cut short this way, or has no link to check a video by, the cleanup keeps the audio of every video it did not reach, rather than trusting what an older check said about it. Videos that really are gone ("Video unavailable", removed by the uploader, a terminated account, Rumble's 410) are still recorded as deleted.
- **Jobs a restart left queued are settled even when a drive hangs.** At boot the editor settles those jobs after it has checked where its storage locations are. A hung network mount could stall that check forever, and the jobs then stayed "queued" on `/jobs`. The settling now waits at most 60 seconds, logs `[boot] storage pass still running after 60 s …` and carries on. The storage check keeps running and logs when it ends. A job re-queued for a channel on the hung drive itself still waits for that drive, and any re-queued after it wait too.
- **`/jobs` says why a job was cancelled at boot.** A job the boot settled shows its reason under its status on `/jobs` and as *Cancelled because* on its own page: for example "server restarted; the scheduler re-derives syncs" or "superseded by a newer queued job (…)". The reason used to be only in the job's log.
- **The server log says how often a queued job skips its page refresh.** When a queued job finishes outside any request, the editor skips its page refresh and notes it in the log. The note used to appear once and never again. Now the first one after a quiet spell is logged at once, any more in the next 10 minutes are counted, and one line at the end gives the count, with a running total.
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -6435,9 +6435,9 @@ Line numbers are `plans/FACTS.md` lines at `e172749b`, before this record's in-p
still sleeps. **L2 review fixes:** a metadata prefetch that classifies `rate_limit` ends
`downloadOneManaged` there (no attempt 1; `writeOutcome` writes the sidecar for both exits), and
`runAvailabilityCheck` stops at its first `rate_limit` probe (`blocked`, `probedIds`, one
- `recordDownloadBackoff`). **`verifyBeforeClean` leaves every tier C suspect a blocked check did not
- probe `unverified`** — never judged on its old `availability.json`, which is what would clear an
- irreversible delete. No live sidecar held the string (2026-09-26 read-only grep of 136,401
+ `recordDownloadBackoff`). **`verifyBeforeClean` leaves every tier C suspect the check did not probe
+ `unverified`** (not in `probedIds`: the check was blocked before it, or it had no `webpage_url`) —
+ never judged on its old `availability.json`, which is what would clear an irreversible delete. No live sidecar held the string (2026-09-26 read-only grep of 136,401
`availability.json`/`download-outcome.json` and every `metadata-scan.json`), but an availability
check stores no text for `deleted`, so an earlier misread there cannot be found after the fact.
- **A job record carries `progressAt`** (live-only, stamped by the one progress writer and by task
diff --git a/plans/release-10.md b/plans/release-10.md
@@ -373,11 +373,16 @@ costs, and nothing reads that log.
`managedDownloadsSleep.test.ts`, which feeds the real line through the classifier into the batch
loop.
- `plans/STATE.md` still lists items 6–9 as open; it is shared with L1, so the merge updates it.
-- **Seen while fixing, pre-existing, not changed:** an unblocked tier C check skips a suspect with no
- `webpage_url` in its metadata, and `verifyBeforeClean` then judges it on its old record (the
- blocked case is now guarded; this one is as before). And `resolveMaybeMissingState` reads an
- `error` probe newer than the scan as `available` (`stateFromAvailability`'s default), so a
- maybe-missing video whose confirm probe failed stops being flagged.
+- ~~An unblocked tier C check judges a suspect with no `webpage_url` on its old record~~
+ (pre-existing, seen while fixing): fixed in the re-read, below.
+- **New low: `resolveMaybeMissingState` reads an `error` probe newer than the scan as `available`**
+ (`stateFromAvailability`'s default; pre-existing, seen while fixing). A maybe-missing video whose
+ confirm probe failed (a 403, a network error, or the one rate-limited probe of a blocked run)
+ stops being flagged. It is display-only: it feeds the maybe-missing state in `buildIndex.ts`
+ (~:1180), i.e. the published site's presence badge. No deletion path reads it; the clean gate uses
+ `resolveEffectiveAvailability`. The fix is one line (`error` → `maybe_missing`), but it changes
+ published presence semantics and the maybe-missing count, so it needs its own export check: not
+ in this slice. Before this slice the soft block published a false "deleted", which was worse.
**Review fixes (review verdict SHIP AFTER FIXES, `l2-review.md`).**
@@ -401,9 +406,14 @@ costs, and nothing reads that log.
`recordDownloadBackoff(detectPlatform(config?.url ?? url) ?? "unknown", paths)`, the key the batch
and the Sync gate use (`onPlatformBackoff` is the test seam). The result gains `blocked`,
`blockMessage` and `probedIds`.
- - **`verifyBeforeClean`: when the tier C check was blocked, every suspect it did not probe is
- `unverified`.** Judged on its old `availability.json` (a months-old `public`), it would have been
- cleared for an irreversible delete. An unblocked check is judged exactly as before.
+ - **`verifyBeforeClean`: every tier C suspect the check did not probe is `unverified`.** Judged on
+ its old `availability.json` (a months-old `public`), it would have been cleared for an
+ irreversible delete. As first fixed (`b8053655`) this applied only to a BLOCKED check. The re-read
+ (`52bb00ef`) dropped that condition, so a suspect an unblocked check skipped for having no
+ `webpage_url` is unverified too. That closes a pre-existing gap. The review's read-only scan found
+ no such video on the live corpus (79,700 dirs, 1,186 clean candidates), so nothing changes today;
+ such a video is simply never cleaned until it has a `webpage_url`. A suspect every probe reached
+ is judged exactly as before.
- The other callers are unaffected, as the review said. `checkKeptDeleted` only pins, and an unprobed
id keeps its old verdict (pinning is the safe direction). The full-sweep confirm leaves an
unprobed id `maybe_missing`, because its probe time is older than the scan's.
@@ -415,7 +425,9 @@ costs, and nothing reads that log.
| `61c03ac1` | (a) the rate-limited prefetch exit, `writeOutcome`; `prefetchRateLimit.test.ts` (4) |
| `24e0f7d5` | item 7's comment and timeout line say what a hung mount does |
| `b8053655` | (b) the availability check's stop + cooldown + `probedIds`; `verifyBeforeClean` leaves unprobed suspects unverified; `checkAvailability.test.ts` (3), `verifyBeforeClean.test.ts` +2 |
-| _this_ | `plans:` this paragraph, the item 7 record text, found and left, FACTS, the `[Unreleased]` bullets |
+| `89218c38` | `plans:` this paragraph, the item 7 record text, found and left, FACTS, the `[Unreleased]` bullets |
+| `52bb00ef` | (re-read) every unprobed tier C suspect is unverified, blocked or not; `verifyBeforeClean.test.ts` +1 (a no-URL suspect in an unblocked check) |
+| _this_ | `plans:` the re-read fix and the `resolveMaybeMissingState` low in this record, FACTS |
**Tests, and the proof they bite.**
- `prefetchRateLimit.test.ts` drives the real `downloadOneManaged` against a temp yt-dlp script that
@@ -443,6 +455,21 @@ costs, and nothing reads that log.
`maybe-missing` were already in the list). **256 passed, 0 failed, 16.2 min**, no queue wait.
- The primary's `export/public/sw.js` was untouched again.
+**Re-read fix (`l2-review.md`, "Re-read of fixes": no must-fix, one should-fix), on `52bb00ef`.**
+- The fix is `52bb00ef`, above.
+- The new `verifyBeforeClean.test.ts` case puts a no-URL suspect with a stale `public` into an unblocked
+ check:
+ - only the other suspect is probed;
+ - the no-URL one comes back `unverified` and excluded;
+ - the probed `public` suspect and the listed video stay cleanable;
+ - no cooldown is recorded.
+ With the old blocked-only condition it fails; the blocked and unblocked cases still pass.
+- tsc clean (`l2-tsc-9.log`).
+- common **1,934/1,934** (+1, `l2-units-3.log`).
+- EDITOR e2e `cleanup-actionable cleanup-holds cleanup-page do-not-clean pre-clean-availability`
+ (`l2-e2e-3.log`): **12 passed, 0 failed, 1.1 min** (9 s in the queue). `pre-clean-availability`
+ was added to the four named specs because it drives all three tiers of the gate end to end.
+
## Rollout
Nothing is rolled out. The live :3001 editor still runs `0213f6c8` (the pre-brand build); the five
