commit 31d0a414eba9018155c808bcec2e33bf44cc3363
parent add512464a999f38383331c608f2d7b79f0e5e6b
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 4 Aug 2026 15:08:07 -0400
Tell the viewer a video is missing before we know why
The export insisted a video was fine during exactly the window where it most
likely wasn't. runQuickAvailabilityCheck diffs a channel's fresh flat playlist
against the data dirs and writes the difference to maybe-missing.json -- one
yt-dlp spawn per channel -- but that file was read in two editor-side places
only. The build never looked at it. A video that fell out of its channel
listing but hadn't had its individual --dump-json probe yet still carried a
stale `availability: "public"`, and the per-video confirmation pass is slow, so
a large channel could sit for a long time knowing "these N videos are gone"
while the site rendered them as Available.
Rather than bolt on a fourth boolean, isDeleted/isUnlisted/nav/nd/nu are
replaced by one shared enum (VideoState) used by the viewer, the search
filters, saved profiles, share links, the MCP, and the charts:
available
missing --+-- maybe_missing left the listing, not yet individually probed
+-- deleted
+-- private
+-- members_only
+-- unlisted
"Missing" means *left the channel listing*, for any reason. Unlisted sits under
it even though such a video is still watchable by direct link -- that keeps the
rule one sentence long. maybe_missing is named for the vocabulary already on
disk (maybe-missing.json, "Maybe-missing (N)"); its label is "Unconfirmed" and
its badge is MISSING?, amber, with the question mark load-bearing: it is a
suspicion, not a finding.
A video is maybe_missing when all three hold: its id is in maybe-missing.json,
its availability.json doesn't already name a state that EXPLAINS the absence,
and it hasn't been re-probed since the scan. Both extra conditions earn their
keep. An unlisted video legitimately falls out of a listing forever and
checkMaybeMissingAction passes skipExpectedAbsent, so a known-gone video is
never re-probed and would otherwise stay flagged permanently. The converse
matters too: a video re-probed AFTER the scan that came back public anyway (a
truncated or oddly paginated playlist fetch) is resolved and must stop being
flagged.
State is computed at the top of buildIndex, beside the sets it replaces, NOT in
the per-video loop. maybe-missing.json is a CHANNEL-level file but the mtimes
cache keys off per-video mtimes, so a channel-level change would never
invalidate it and the flag would simply never ship. Computing it up top
sidesteps invalidation entirely, and the maybe-missing set is small by
construction, so reading availability.json for just those ids is cheap.
MtimeRecord gains `availability` (the exact string) and KEEPS the two booleans
as the fallback for records written before this. Deliberately no SCHEMA_VERSION
bump: that wipes the whole LMDB cache and re-pages the entire corpus, whereas
availabilityMs already participates in change detection, so any video whose
availability.json is rewritten from here on picks up the exact string -- and the
overlay reads availability.json directly anyway, which covers every video this
is actually about. Residual gap: a video confirmed private/members_only by an
older check, in a channel never quick-checked, reads Available until its
availability file is touched again.
Two departures from the plan, both to avoid shipping a regression:
- buildIndex reads resolveEffectiveAvailability, not loadAvailability. The
status chart was already using the effective value, which also consults
download-outcome.json -- and a download that fails with "deleted" records the
class THERE and deliberately leaves availability.json's top-level fields
alone (downloadOneManaged), so the file alone still reads "public". Reading
only that file would have silently dropped that signal from the chart. It
does append an availability history entry, which bumps the mtime, which is
what brings the build back to notice. Same cost -- already gated on
availabilityMs -- and the chart and the viewer now cannot disagree.
- The videoState map is hashed into the per-site stats fingerprint. Without it
the staleness fix below doesn't land at all: an availability flip touches no
metadata mtime, so added/changed/removed all stay 0, every site skips its
rebuild, and the corrected status never reaches a chart.
That staleness bug is one buildStats already documented at the statsByPath
write: keyed on metadata mtime alone, "a pure availability flip refreshes on
the next metadata touch or schema bump" -- i.e. deleting a video today did not
change the status chart today. Status stops being a cached field and is applied
at collection time from a new sparse videoState sub-DB (only non-available
videos get an entry), which also avoids re-reading ~76k availability.json files
at stats time -- the exact cost buildIndex's own comment says it was optimized
to avoid. No STATS_SCHEMA_VERSION bump needed, since the overlay overrides the
cached value unconditionally and so self-heals stale ones.
Viewer: VideoStateBadge is the FIRST per-card availability indicator -- until
now a deleted or unlisted video carried no visual marker at all, only a filter
you had to already suspect. The Availability row becomes Available + a Missing
parent over its five leaves (indeterminate when partial). The nav/nd/nu trio
becomes one kept-set: six booleans through ten sites would be untenable, and
one Set collapses passesFilter, filterKey, filtersDirty, both snapshot
builders, promoteDrafts, hydration and the share build.
Back-compat is a correctness property, not politeness -- a share link outlives
the tab that made it. `fav` gains one token per state and the sentinel goes to
fv=2. The bump is what makes old links safe: a v1 link carries at most
fav=a&fav=u&fav=d and its author meant everything we now call missing, whereas
a NEW link that deliberately drops missing is otherwise byte-identical. v1 maps
a -> {available, maybe_missing} (unconfirmed videos read as available when the
link was written) and d -> {deleted, private, members_only}. FilterSnapshot.av
is likewise absent-means-keep-all, and legacy nav/nd/nu are still read so saved
profiles migrate rather than reset. DisplaySummary keeps emitting
isDeleted/isUnlisted, derived from state: the MCP's RemoteSource reads
summaries pages from live origins it does not control, so hub/member version
skew is real in both directions.
Also:
- parseSnapshot read nov/nol/naa/nar/nav/nd but NOT nu or nop, both of which
committedSnapshot writes -- so excluding posts, or excluding unlisted videos,
silently reverted on every reload. snapshotsEqual omitted them too.
- The share callback's dep list omitted committedNu; collapsing to one set
fixes it incidentally.
- buildStats.ts had a raw NUL byte inside a template literal where every
sibling writes \x00. Identical at runtime, but it made grep and ripgrep
classify the file as binary and return zero matches -- the whole stats
pipeline was invisible to codebase search.
- turbopack.root is pinned in both next.config.ts. UNRELATED to this work and
droppable: Turbopack infers the workspace root by walking up for a lockfile
and taking the outermost one, and a stray pnpm-lock.yaml in $HOME (created
2026-08-04) relocated it, after which globals.css's
@import "../../common/styles/tokens.css" resolved outside the project and
`next dev` failed to boot -- blocking the entire export AND editor e2e suites
before any of this was touched. Deleting that lockfile fixes it too.
Verified: tsc --noEmit clean in common/, export/, mcp/, editor/ (pnpm lint in
editor/ cannot pass -- no eslint config there). MCP 106/106, including a case
per state and the v1 back-compat path. 29 new unit tests: the predicate over
tmpdir fixtures (never probed / each explaining state / re-probed after vs
before the scan / malformed date) and a share round-trip incl. the empty
selection, which must NOT decay into keep-all. Export e2e 159/159 with 9 new;
editor export-search 19/19, whose availability tests run on legacy-boolean
fixtures and so exercise the fallback. Two existing export specs asserted the
old encodings (fv=1, working.nd) and were updated -- behaviour preserved.
Ran against a real fixture corpus rather than only mocks: a1 in the set with no
availability.json -> maybe_missing, a2 + private -> private, b1 + public
re-probed after the scan -> resolved to available, b2 in no set -> available;
then flipping b2's availability alone rebuilt the stats with added 0, changed
0, removed 0 and moved it to deleted, which is the staleness fix demonstrated
end to end. Not run against the live corpus: booting an editor there arms the
runners and resumes the GPU-weeks digest sweep.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Diffstat:
27 files changed, 1415 insertions(+), 217 deletions(-)
diff --git a/common/components/SearchResults.tsx b/common/components/SearchResults.tsx
@@ -16,6 +16,7 @@ import {
AgeRestrictedBadge,
DuplicateBadge,
LivestreamBadge,
+ VideoStateBadge,
VodExpiredBadge,
} from "./badges";
import {
@@ -502,6 +503,7 @@ const ResultCard = memo(function ResultCard({
</span>
{group.isLivestream && <LivestreamBadge />}
{group.ageRestricted && <AgeRestrictedBadge />}
+ <VideoStateBadge state={group.state} />
{siblings.length > 0 && (
<DuplicateBadge count={siblings.length} />
)}
diff --git a/common/components/SearchSessionContext.tsx b/common/components/SearchSessionContext.tsx
@@ -45,6 +45,11 @@ import {
type GroupNode,
type LayerScope,
} from "../lib/searchQuery";
+import {
+ VIDEO_STATES,
+ summaryState,
+ type VideoState,
+} from "../lib/availability";
import { useUrlParams, writeUrlParams } from "./urlState";
import {
buildShareSearchParams,
@@ -58,7 +63,10 @@ import {
emptyStoredState,
loadStoredState,
saveStoredState,
+ snapshotStates,
snapshotsEqual,
+ statesFromLegacy,
+ writeSnapshotStates,
type FilterSnapshot,
type StoredState,
} from "./exportFilterStorage";
@@ -132,6 +140,9 @@ export type ResultGroup = {
date: string;
isLivestream: boolean;
ageRestricted: boolean;
+ // Presence on the source platform, for the card's state badge. Posts are
+ // always "available" — a post has no channel listing to fall out of.
+ state: VideoState;
// Carried through so the card can flag likely-expired stream VODs (Kick,
// Twitch) via vodExpiry(platform, uploadDate).
platform: Platform;
@@ -274,7 +285,7 @@ function anyLeafHasScope(root: GroupNode, scope: LayerScope): boolean {
return found;
}
-function sameSet<T>(a: Set<T>, b: Set<T>): boolean {
+function sameSet<T>(a: ReadonlySet<T>, b: ReadonlySet<T>): boolean {
if (a.size !== b.size) return false;
for (const v of a) if (!b.has(v)) return false;
return true;
@@ -342,9 +353,14 @@ function useSearchSessionState() {
const [committedNop, setCommittedNop] = useState(false);
const [committedNaa, setCommittedNaa] = useState(false);
const [committedNar, setCommittedNar] = useState(false);
- const [committedNav, setCommittedNav] = useState(false);
- const [committedNd, setCommittedNd] = useState(false);
- const [committedNu, setCommittedNu] = useState(false);
+ // Availability is a six-state enum (VideoState), not a set of "exclude X"
+ // booleans: a video is available, or missing for one of five reasons. Kept as
+ // the set of states to KEEP — absent means filtered out — because six
+ // booleans threaded through commit/hydrate/persist/share would be untenable.
+ // Default keeps everything.
+ const [committedStates, setCommittedStates] = useState<ReadonlySet<VideoState>>(
+ () => new Set(VIDEO_STATES),
+ );
// Inclusive upload-date bounds ("YYYYMMDD"); "" => unbounded on that end.
const [committedDateFrom, setCommittedDateFrom] = useState("");
const [committedDateTo, setCommittedDateTo] = useState("");
@@ -374,9 +390,9 @@ function useSearchSessionState() {
const [draftNol, setDraftNol] = useState(false);
const [draftNaa, setDraftNaa] = useState(false);
const [draftNar, setDraftNar] = useState(false);
- const [draftNav, setDraftNav] = useState(false);
- const [draftNd, setDraftNd] = useState(false);
- const [draftNu, setDraftNu] = useState(false);
+ const [draftStates, setDraftStates] = useState<ReadonlySet<VideoState>>(
+ () => new Set(VIDEO_STATES),
+ );
const [draftDateFrom, setDraftDateFrom] = useState("");
const [draftDateTo, setDraftDateTo] = useState("");
@@ -685,16 +701,7 @@ function useSearchSessionState() {
if (committedDateTo && t.uploadDate > committedDateTo) return false;
if (t.isLivestream ? committedNol : committedNov) return false;
if (t.ageRestricted ? committedNar : committedNaa) return false;
- // Availability is a three-way split: deleted / unlisted / available.
- // Unlisted has its own bucket, so excluding "Available" no longer hides
- // unlisted videos.
- if (t.isDeleted) {
- if (committedNd) return false;
- } else if (t.isUnlisted) {
- if (committedNu) return false;
- } else if (committedNav) {
- return false;
- }
+ if (!committedStates.has(summaryState(t))) return false;
return true;
};
}, [
@@ -706,12 +713,15 @@ function useSearchSessionState() {
committedNol,
committedNaa,
committedNar,
- committedNav,
- committedNd,
- committedNu,
+ committedStates,
]);
- const filterKey = `${committedChannelsKey}|${committedNov ? 1 : 0}|${committedNop ? 1 : 0}|${committedNol ? 1 : 0}|${committedNaa ? 1 : 0}|${committedNar ? 1 : 0}|${committedNav ? 1 : 0}|${committedNd ? 1 : 0}|${committedNu ? 1 : 0}|${committedDateFrom}|${committedDateTo}`;
+ const committedStatesKey = useMemo(
+ () => VIDEO_STATES.filter((s) => committedStates.has(s)).join(","),
+ [committedStates],
+ );
+
+ const filterKey = `${committedChannelsKey}|${committedNov ? 1 : 0}|${committedNop ? 1 : 0}|${committedNol ? 1 : 0}|${committedNaa ? 1 : 0}|${committedNar ? 1 : 0}|${committedStatesKey}|${committedDateFrom}|${committedDateTo}`;
// The scope universe is videos AND posts. The two namespaces are disjoint;
// searchEval partitions them per-leaf (see EvalCtx.postScopeSlugs) so a posts
@@ -806,6 +816,7 @@ function useSearchSessionState() {
date: t.date,
isLivestream: t.isLivestream,
ageRestricted: t.ageRestricted,
+ state: summaryState(t),
platform: t.platform,
uploadDate: t.uploadDate,
hits: [],
@@ -829,6 +840,7 @@ function useSearchSessionState() {
date: t.date,
isLivestream: t.isLivestream,
ageRestricted: t.ageRestricted,
+ state: summaryState(t),
platform: t.platform,
uploadDate: t.uploadDate,
hits: hitsBySlug.get(t.slug) ?? [],
@@ -855,6 +867,8 @@ function useSearchSessionState() {
date: post.createdAt,
isLivestream: false,
ageRestricted: false,
+ // A post has no channel listing to fall out of.
+ state: "available",
platform: post.platform,
uploadDate: post.uploadDate,
hits: hitsBySlug.get(slug) ?? [],
@@ -918,9 +932,7 @@ function useSearchSessionState() {
draftNol !== committedNol ||
draftNaa !== committedNaa ||
draftNar !== committedNar ||
- draftNav !== committedNav ||
- draftNd !== committedNd ||
- draftNu !== committedNu ||
+ !sameSet(draftStates, committedStates) ||
draftDateFrom !== committedDateFrom ||
draftDateTo !== committedDateTo;
@@ -935,9 +947,7 @@ function useSearchSessionState() {
if (committedNol) snap.nol = true;
if (committedNaa) snap.naa = true;
if (committedNar) snap.nar = true;
- if (committedNav) snap.nav = true;
- if (committedNd) snap.nd = true;
- if (committedNu) snap.nu = true;
+ writeSnapshotStates(snap, committedStates);
if (committedDateFrom) snap.dateFrom = committedDateFrom;
if (committedDateTo) snap.dateTo = committedDateTo;
snap.query = stringifyRoot(committedRoot);
@@ -950,9 +960,7 @@ function useSearchSessionState() {
committedNol,
committedNaa,
committedNar,
- committedNav,
- committedNd,
- committedNu,
+ committedStates,
committedDateFrom,
committedDateTo,
committedRoot,
@@ -971,9 +979,7 @@ function useSearchSessionState() {
if (draftNol) snap.nol = true;
if (draftNaa) snap.naa = true;
if (draftNar) snap.nar = true;
- if (draftNav) snap.nav = true;
- if (draftNd) snap.nd = true;
- if (draftNu) snap.nu = true;
+ writeSnapshotStates(snap, draftStates);
if (draftDateFrom) snap.dateFrom = draftDateFrom;
if (draftDateTo) snap.dateTo = draftDateTo;
snap.query = stringifyRoot(draftRoot);
@@ -987,9 +993,7 @@ function useSearchSessionState() {
draftNol,
draftNaa,
draftNar,
- draftNav,
- draftNd,
- draftNu,
+ draftStates,
draftDateFrom,
draftDateTo,
draftRoot,
@@ -1002,9 +1006,7 @@ function useSearchSessionState() {
setCommittedNol(draftNol);
setCommittedNaa(draftNaa);
setCommittedNar(draftNar);
- setCommittedNav(draftNav);
- setCommittedNd(draftNd);
- setCommittedNu(draftNu);
+ setCommittedStates(new Set(draftStates));
setCommittedDateFrom(draftDateFrom);
setCommittedDateTo(draftDateTo);
stripAllFilterParamsFromUrl();
@@ -1015,9 +1017,7 @@ function useSearchSessionState() {
draftNol,
draftNaa,
draftNar,
- draftNav,
- draftNd,
- draftNu,
+ draftStates,
draftDateFrom,
draftDateTo,
]);
@@ -1073,9 +1073,7 @@ function useSearchSessionState() {
let initialNol = false;
let initialNaa = false;
let initialNar = false;
- let initialNav = false;
- let initialNd = false;
- let initialNu = false;
+ let initialStates: ReadonlySet<VideoState> = new Set(VIDEO_STATES);
let initialDateFrom = "";
let initialDateTo = "";
@@ -1090,9 +1088,7 @@ function useSearchSessionState() {
initialNol = !sel.livestreams;
initialNaa = !sel.allAges;
initialNar = !sel.restricted;
- initialNav = !sel.available;
- initialNd = !sel.deleted;
- initialNu = !sel.unlisted;
+ initialStates = sel.states;
initialDateFrom = sel.dateFrom ?? "";
initialDateTo = sel.dateTo ?? "";
} else if (urlHasAnyFilterParam()) {
@@ -1102,9 +1098,15 @@ function useSearchSessionState() {
initialNol = params.get("nol") === "1";
initialNaa = params.get("naa") === "1";
initialNar = params.get("nar") === "1";
- initialNav = params.get("nav") === "1";
- initialNd = params.get("nd") === "1";
- initialNu = params.get("nu") === "1";
+ // Legacy exclusion keys, still parsed so old links keep working. `nav`
+ // meant "exclude available" back when unconfirmed/private/members-only
+ // had no bucket of their own, so it drops maybe_missing with available:
+ // those videos were counted as available when the link was written.
+ initialStates = statesFromLegacy({
+ nav: params.get("nav") === "1",
+ nd: params.get("nd") === "1",
+ nu: params.get("nu") === "1",
+ });
}
let resolvedRoot = rootFromUrl;
@@ -1124,9 +1126,7 @@ function useSearchSessionState() {
initialNol = snapshot?.nol === true;
initialNaa = snapshot?.naa === true;
initialNar = snapshot?.nar === true;
- initialNav = snapshot?.nav === true;
- initialNd = snapshot?.nd === true;
- initialNu = snapshot?.nu === true;
+ initialStates = snapshotStates(snapshot);
initialDateFrom = snapshot?.dateFrom ?? "";
initialDateTo = snapshot?.dateTo ?? "";
if (!resolvedRoot && snapshot?.query) {
@@ -1146,9 +1146,7 @@ function useSearchSessionState() {
setDraftNol(initialNol);
setDraftNaa(initialNaa);
setDraftNar(initialNar);
- setDraftNav(initialNav);
- setDraftNd(initialNd);
- setDraftNu(initialNu);
+ setDraftStates(new Set(initialStates));
setDraftDateFrom(initialDateFrom);
setDraftDateTo(initialDateTo);
setCommittedExcludedChannels(initialExcluded);
@@ -1156,9 +1154,7 @@ function useSearchSessionState() {
setCommittedNol(initialNol);
setCommittedNaa(initialNaa);
setCommittedNar(initialNar);
- setCommittedNav(initialNav);
- setCommittedNd(initialNd);
- setCommittedNu(initialNu);
+ setCommittedStates(new Set(initialStates));
setCommittedDateFrom(initialDateFrom);
setCommittedDateTo(initialDateTo);
@@ -1288,9 +1284,7 @@ function useSearchSessionState() {
setDraftNol(snapshot?.nol === true);
setDraftNaa(snapshot?.naa === true);
setDraftNar(snapshot?.nar === true);
- setDraftNav(snapshot?.nav === true);
- setDraftNd(snapshot?.nd === true);
- setDraftNu(snapshot?.nu === true);
+ setDraftStates(snapshotStates(snapshot));
setDraftDateFrom(snapshot?.dateFrom ?? "");
setDraftDateTo(snapshot?.dateTo ?? "");
if (snapshot?.query) {
@@ -1314,9 +1308,7 @@ function useSearchSessionState() {
setCommittedNol(snapshot?.nol === true);
setCommittedNaa(snapshot?.naa === true);
setCommittedNar(snapshot?.nar === true);
- setCommittedNav(snapshot?.nav === true);
- setCommittedNd(snapshot?.nd === true);
- setCommittedNu(snapshot?.nu === true);
+ setCommittedStates(snapshotStates(snapshot));
setCommittedDateFrom(snapshot?.dateFrom ?? "");
setCommittedDateTo(snapshot?.dateTo ?? "");
if (snapshot?.query) {
@@ -1500,9 +1492,7 @@ function useSearchSessionState() {
livestreams: !committedNol,
allAges: !committedNaa,
restricted: !committedNar,
- available: !committedNav,
- deleted: !committedNd,
- unlisted: !committedNu,
+ states: committedStates,
tracks: new Set(),
dateFrom: committedDateFrom || undefined,
dateTo: committedDateTo || undefined,
@@ -1542,8 +1532,7 @@ function useSearchSessionState() {
committedNol,
committedNaa,
committedNar,
- committedNav,
- committedNd,
+ committedStates,
committedDateFrom,
committedDateTo,
committedRoot,
@@ -1853,12 +1842,8 @@ function useSearchSessionState() {
setDraftNaa,
draftNar,
setDraftNar,
- draftNav,
- setDraftNav,
- draftNd,
- setDraftNd,
- draftNu,
- setDraftNu,
+ draftStates,
+ setDraftStates,
draftDateFrom,
setDraftDateFrom,
draftDateTo,
diff --git a/common/components/WorkspaceSearchBar.tsx b/common/components/WorkspaceSearchBar.tsx
@@ -6,8 +6,13 @@
// (and keeps its committed search) across `/` ⇄ `/ask`. All of its state comes
// from the shared SearchSession.
-import { Fragment } from "react";
+import { Fragment, useCallback } from "react";
import { ChevronRightIcon } from "lucide-react";
+import {
+ MISSING_STATES,
+ VIDEO_STATE_LABELS,
+ type VideoState,
+} from "../lib/availability";
import { Button } from "./ui/button";
import { Checkbox } from "./ui/checkbox";
import QueryBuilder from "./QueryBuilder";
@@ -75,12 +80,8 @@ export default function WorkspaceSearchBar() {
setDraftNaa,
draftNar,
setDraftNar,
- draftNav,
- setDraftNav,
- draftNd,
- setDraftNd,
- draftNu,
- setDraftNu,
+ draftStates,
+ setDraftStates,
draftDateFrom,
setDraftDateFrom,
draftDateTo,
@@ -102,6 +103,33 @@ export default function WorkspaceSearchBar() {
const { postsManifest } = useSearchData();
const hasPostsCorpus = (postsManifest?.channels.length ?? 0) > 0;
+ const toggleState = useCallback(
+ (state: VideoState, keep: boolean) => {
+ setDraftStates((prev) => {
+ const next = new Set(prev);
+ if (keep) next.add(state);
+ else next.delete(state);
+ return next;
+ });
+ },
+ [setDraftStates],
+ );
+ const setAllMissing = useCallback(
+ (keep: boolean) => {
+ setDraftStates((prev) => {
+ const next = new Set(prev);
+ for (const s of MISSING_STATES) {
+ if (keep) next.add(s);
+ else next.delete(s);
+ }
+ return next;
+ });
+ },
+ [setDraftStates],
+ );
+ const allMissingKept = MISSING_STATES.every((s) => draftStates.has(s));
+ const someMissingKept = MISSING_STATES.some((s) => draftStates.has(s));
+
return (
<div className="flex flex-col gap-6">
<ProfilesRow
@@ -473,31 +501,58 @@ export default function WorkspaceSearchBar() {
Age-restricted
</label>
</div>
- <div className="flex flex-wrap items-center gap-x-3 gap-y-1">
- <span className="text-xs uppercase tracking-wide text-muted-foreground">
+ {/* Availability is one enum, not three independent booleans:
+ "Missing" is an umbrella over the five ways a video can leave
+ its channel listing, so it renders as a parent with the leaves
+ nested under it. Toggling the parent sets or clears all five;
+ it goes indeterminate when only some are kept. */}
+ <div className="flex flex-wrap items-start gap-x-3 gap-y-1">
+ <span className="text-xs uppercase tracking-wide text-muted-foreground pt-0.5">
Availability
</span>
<label className="flex items-center gap-1.5 select-none">
<Checkbox
- checked={!draftNav}
- onCheckedChange={(value) => setDraftNav(value !== true)}
+ data-testid="av-available"
+ checked={draftStates.has("available")}
+ onCheckedChange={(value) =>
+ toggleState("available", value === true)
+ }
/>
Available
</label>
- <label className="flex items-center gap-1.5 select-none">
- <Checkbox
- checked={!draftNu}
- onCheckedChange={(value) => setDraftNu(value !== true)}
- />
- Unlisted
- </label>
- <label className="flex items-center gap-1.5 select-none">
- <Checkbox
- checked={!draftNd}
- onCheckedChange={(value) => setDraftNd(value !== true)}
- />
- Deleted
- </label>
+ <div className="flex flex-col gap-1">
+ <label className="flex items-center gap-1.5 select-none">
+ <Checkbox
+ data-testid="av-missing"
+ checked={
+ allMissingKept
+ ? true
+ : someMissingKept
+ ? "indeterminate"
+ : false
+ }
+ onCheckedChange={(value) => setAllMissing(value === true)}
+ />
+ Missing
+ </label>
+ <div className="flex flex-wrap items-center gap-x-3 gap-y-1 pl-5">
+ {MISSING_STATES.map((state) => (
+ <label
+ key={state}
+ className="flex items-center gap-1.5 select-none"
+ >
+ <Checkbox
+ data-testid={`av-${state}`}
+ checked={draftStates.has(state)}
+ onCheckedChange={(value) =>
+ toggleState(state, value === true)
+ }
+ />
+ {VIDEO_STATE_LABELS[state]}
+ </label>
+ ))}
+ </div>
+ </div>
</div>
<div className="flex flex-wrap items-center gap-x-3 gap-y-1">
<span className="text-xs uppercase tracking-wide text-muted-foreground">
diff --git a/common/components/badges.tsx b/common/components/badges.tsx
@@ -1,6 +1,70 @@
+import { VIDEO_STATE_LABELS, type VideoState } from "../lib/availability";
+
const badgeBase =
"shrink-0 text-[10px] font-semibold uppercase tracking-wide px-1.5 py-0.5 rounded";
+// Presence on the source platform. Nothing renders for "available" — the
+// overwhelming majority — so a card only grows a badge when something is
+// actually wrong.
+//
+// The `?` on MISSING? is load-bearing: that state is a suspicion (the video
+// dropped out of its channel's listing) and not yet a finding, and its amber
+// separates it from the confirmed-gone states. Unlisted is slate rather than
+// red because such a video is still watchable by direct link; it sits under
+// "missing" only because it left the listing.
+const STATE_STYLES: Record<
+ Exclude<VideoState, "available">,
+ { label: string; className: string; title: string }
+> = {
+ maybe_missing: {
+ label: "Missing?",
+ className:
+ "bg-amber-100 text-amber-800 dark:bg-amber-900/40 dark:text-amber-200",
+ title:
+ "Gone from its channel's listing at the last check, but not yet confirmed — it may be deleted, private, members-only, or merely unlisted",
+ },
+ deleted: {
+ label: "Deleted",
+ className:
+ "bg-rose-100 text-rose-800 dark:bg-rose-900/40 dark:text-rose-200",
+ title: "Removed from the source platform",
+ },
+ private: {
+ label: "Private",
+ className:
+ "bg-rose-100 text-rose-800 dark:bg-rose-900/40 dark:text-rose-200",
+ title: "Made private on the source platform",
+ },
+ members_only: {
+ label: "Members",
+ className:
+ "bg-rose-100 text-rose-800 dark:bg-rose-900/40 dark:text-rose-200",
+ title: "Members-only on the source platform",
+ },
+ unlisted: {
+ label: "Unlisted",
+ className:
+ "bg-slate-200 text-slate-700 dark:bg-slate-700/50 dark:text-slate-200",
+ title: "Not in its channel's listing, but still watchable by direct link",
+ },
+};
+
+export function VideoStateBadge({ state }: { state: VideoState }) {
+ if (state === "available") return null;
+ const style = STATE_STYLES[state];
+ return (
+ <span
+ title={style.title}
+ data-testid={`state-badge-${state}`}
+ data-state={state}
+ aria-label={VIDEO_STATE_LABELS[state]}
+ className={`${badgeBase} ${style.className}`}
+ >
+ {style.label}
+ </span>
+ );
+}
+
export function LivestreamBadge() {
return (
<span
diff --git a/common/components/exportFilterStorage.ts b/common/components/exportFilterStorage.ts
@@ -3,6 +3,11 @@
// — any malformed value resets storage and returns null so the UI falls
// back to group defaults.
+import {
+ VIDEO_STATES,
+ isVideoState,
+ type VideoState,
+} from "../lib/availability";
import type { SearchMode } from "./urlState";
const KEY = "ytdlp-tb:export-filters";
@@ -27,6 +32,11 @@ export type FilterSnapshot = {
nol?: boolean;
naa?: boolean;
nar?: boolean;
+ // Availability states to KEEP. Absent => keep everything, which is also what
+ // a profile saved before the state enum existed means once `nav`/`nd`/`nu`
+ // below are folded in. Those three are legacy: still read (so saved profiles
+ // migrate instead of resetting) but never written.
+ av?: VideoState[];
nav?: boolean;
nd?: boolean;
nu?: boolean;
@@ -55,6 +65,48 @@ export function emptySnapshot(): FilterSnapshot {
return { channels: { included: [], excluded: [] } };
}
+// Fold the legacy exclusion booleans into a kept-set. `nav` meant "exclude
+// available" when the taxonomy was available/unlisted/deleted and nothing else,
+// so it also drops `maybe_missing`: a video now flagged unconfirmed would have
+// counted as available for whoever saved the profile.
+export function statesFromLegacy(legacy: {
+ nav?: boolean;
+ nd?: boolean;
+ nu?: boolean;
+}): Set<VideoState> {
+ const keep = new Set<VideoState>(VIDEO_STATES);
+ if (legacy.nav) {
+ keep.delete("available");
+ keep.delete("maybe_missing");
+ }
+ if (legacy.nd) keep.delete("deleted");
+ if (legacy.nu) keep.delete("unlisted");
+ return keep;
+}
+
+// The availability states a stored snapshot keeps. `av` wins; otherwise the
+// legacy booleans are migrated; a snapshot with neither keeps everything.
+export function snapshotStates(
+ snapshot: FilterSnapshot | null | undefined,
+): Set<VideoState> {
+ if (snapshot?.av) return new Set(snapshot.av);
+ return statesFromLegacy({
+ nav: snapshot?.nav,
+ nd: snapshot?.nd,
+ nu: snapshot?.nu,
+ });
+}
+
+// Write the kept-set onto a snapshot, omitting it when everything is kept so
+// the default profile stays byte-identical to one saved before this existed.
+export function writeSnapshotStates(
+ snap: FilterSnapshot,
+ states: ReadonlySet<VideoState>,
+): void {
+ if (VIDEO_STATES.every((s) => states.has(s))) return;
+ snap.av = VIDEO_STATES.filter((s) => states.has(s));
+}
+
export function emptyStoredState(): StoredState {
return {
v: VERSION,
@@ -81,11 +133,16 @@ function parseSnapshot(raw: unknown): FilterSnapshot | null {
},
};
if (typeof r.nov === "boolean") snap.nov = r.nov;
+ // `nop` and `nu` were written by committedSnapshot but never read back here,
+ // so both were silently dropped on every reload.
+ if (typeof r.nop === "boolean") snap.nop = r.nop;
if (typeof r.nol === "boolean") snap.nol = r.nol;
if (typeof r.naa === "boolean") snap.naa = r.naa;
if (typeof r.nar === "boolean") snap.nar = r.nar;
+ if (Array.isArray(r.av) && r.av.every(isVideoState)) snap.av = r.av.slice();
if (typeof r.nav === "boolean") snap.nav = r.nav;
if (typeof r.nd === "boolean") snap.nd = r.nd;
+ if (typeof r.nu === "boolean") snap.nu = r.nu;
if (typeof r.dateFrom === "string" && /^\d{8}$/.test(r.dateFrom)) {
snap.dateFrom = r.dateFrom;
}
@@ -194,11 +251,17 @@ export function snapshotsEqual(a: FilterSnapshot, b: FilterSnapshot): boolean {
return false;
}
if ((a.nov ?? false) !== (b.nov ?? false)) return false;
+ if ((a.nop ?? false) !== (b.nop ?? false)) return false;
if ((a.nol ?? false) !== (b.nol ?? false)) return false;
if ((a.naa ?? false) !== (b.naa ?? false)) return false;
if ((a.nar ?? false) !== (b.nar ?? false)) return false;
- if ((a.nav ?? false) !== (b.nav ?? false)) return false;
- if ((a.nd ?? false) !== (b.nd ?? false)) return false;
+ // Compared through the resolved kept-set, so an `av` snapshot and the legacy
+ // booleans it migrates from don't read as different profiles.
+ if (
+ sortedJson([...snapshotStates(a)]) !== sortedJson([...snapshotStates(b)])
+ ) {
+ return false;
+ }
if ((a.dateFrom ?? "") !== (b.dateFrom ?? "")) return false;
if ((a.dateTo ?? "") !== (b.dateTo ?? "")) return false;
if ((a.mode ?? "transcripts") !== (b.mode ?? "transcripts")) return false;
diff --git a/common/components/shareUrl.test.ts b/common/components/shareUrl.test.ts
@@ -0,0 +1,171 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import {
+ MISSING_STATES,
+ VIDEO_STATES,
+ type VideoState,
+} from "../lib/availability";
+import {
+ buildShareSearchParams,
+ hasShareV1,
+ parseShareV1,
+ type ShareSelection,
+} from "./shareUrl";
+import {
+ emptySnapshot,
+ snapshotStates,
+ snapshotsEqual,
+ statesFromLegacy,
+ writeSnapshotStates,
+} from "./exportFilterStorage";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common exec tsx --test common/components/shareUrl.test.ts
+//
+// The share schema is the one part of the filter state that outlives the tab it
+// was created in, so back-compat is a correctness property, not politeness: a
+// link somebody pasted into chat last year still has to resolve to what its
+// author selected.
+
+function selection(states: VideoState[]): ShareSelection {
+ return {
+ selectedChannels: new Set(["Chan A"]),
+ videos: true,
+ livestreams: true,
+ allAges: true,
+ restricted: true,
+ states: new Set(states),
+ tracks: new Set(),
+ };
+}
+
+function roundTrip(states: VideoState[]): Set<VideoState> {
+ const params = buildShareSearchParams(selection(states), {
+ q: "",
+ mode: "transcripts",
+ regex: false,
+ });
+ const parsed = parseShareV1(`?${params.toString()}`, ["Chan A"]);
+ return new Set(parsed.states);
+}
+
+test("build writes the v2 sentinel, and both versions are accepted", () => {
+ const params = buildShareSearchParams(selection([...VIDEO_STATES]), {
+ q: "",
+ mode: "transcripts",
+ regex: false,
+ });
+ assert.equal(params.get("fv"), "2");
+ assert.ok(hasShareV1(`?${params.toString()}`));
+ assert.ok(hasShareV1("?fv=1&fav=a"));
+ assert.ok(!hasShareV1("?fv=3"));
+ assert.ok(!hasShareV1("?q=coffee"));
+});
+
+test("every state round-trips on its own", () => {
+ for (const state of VIDEO_STATES) {
+ assert.deepEqual([...roundTrip([state])], [state], `state ${state}`);
+ }
+});
+
+test("the full set round-trips", () => {
+ const out = roundTrip([...VIDEO_STATES]);
+ assert.equal(out.size, VIDEO_STATES.length);
+ for (const state of VIDEO_STATES) assert.ok(out.has(state));
+});
+
+test("an empty selection round-trips as empty, not as keep-all", () => {
+ // The distinction the fv bump exists to preserve: "the author deselected
+ // everything" must not decay into "the author selected nothing explicit".
+ assert.equal(roundTrip([]).size, 0);
+});
+
+test("a mixed selection round-trips exactly", () => {
+ const picked: VideoState[] = ["available", "maybe_missing", "deleted"];
+ assert.deepEqual([...roundTrip(picked)].sort(), [...picked].sort());
+});
+
+// ─── v1 back-compat ───
+
+test("a legacy fv=1 link that kept 'available' also keeps unconfirmed", () => {
+ const sel = parseShareV1("?fv=1&fav=a", []);
+ assert.deepEqual([...sel.states].sort(), ["available", "maybe_missing"]);
+});
+
+test("a legacy fv=1 all-buckets link yields every state", () => {
+ const sel = parseShareV1("?fv=1&fav=a&fav=u&fav=d", []);
+ assert.equal(sel.states.size, VIDEO_STATES.length);
+ for (const state of VIDEO_STATES) assert.ok(sel.states.has(state));
+});
+
+test("a legacy fv=1 'deleted' link covers the states v1 could not express", () => {
+ const sel = parseShareV1("?fv=1&fav=d", []);
+ assert.deepEqual(
+ [...sel.states].sort(),
+ ["deleted", "members_only", "private"],
+ );
+});
+
+test("a v2 link with the same tokens as a v1 link means something different", () => {
+ // fav=a&fav=u&fav=d is byte-identical across the two schemas, and the two
+ // readings genuinely differ — which is the whole reason for the sentinel.
+ const v1 = parseShareV1("?fv=1&fav=a&fav=u&fav=d", []);
+ const v2 = parseShareV1("?fv=2&fav=a&fav=u&fav=d", []);
+ assert.equal(v1.states.size, 6);
+ assert.deepEqual([...v2.states].sort(), ["available", "deleted", "unlisted"]);
+});
+
+// ─── stored snapshots ───
+
+test("a snapshot with no availability keys keeps everything", () => {
+ assert.equal(snapshotStates(null).size, VIDEO_STATES.length);
+ assert.equal(snapshotStates(emptySnapshot()).size, VIDEO_STATES.length);
+});
+
+test("legacy nav/nd/nu profiles migrate instead of resetting", () => {
+ // nav dropped "available" back when unconfirmed videos counted as available,
+ // so it has to drop maybe_missing with it.
+ assert.deepEqual(
+ [...statesFromLegacy({ nav: true })].sort(),
+ ["deleted", "members_only", "private", "unlisted"],
+ );
+ assert.ok(!statesFromLegacy({ nd: true }).has("deleted"));
+ assert.ok(!statesFromLegacy({ nu: true }).has("unlisted"));
+ assert.deepEqual(
+ [...snapshotStates({ ...emptySnapshot(), nd: true, nu: true })].sort(),
+ ["available", "maybe_missing", "members_only", "private"],
+ );
+});
+
+test("av wins over the legacy booleans on the same snapshot", () => {
+ const snap = { ...emptySnapshot(), av: ["deleted" as VideoState], nd: true };
+ assert.deepEqual([...snapshotStates(snap)], ["deleted"]);
+});
+
+test("writeSnapshotStates omits av when everything is kept", () => {
+ const snap = emptySnapshot();
+ writeSnapshotStates(snap, new Set(VIDEO_STATES));
+ assert.equal(snap.av, undefined);
+ writeSnapshotStates(snap, new Set(MISSING_STATES));
+ assert.deepEqual(snap.av, [...MISSING_STATES].sort((a, b) =>
+ VIDEO_STATES.indexOf(a) - VIDEO_STATES.indexOf(b),
+ ));
+});
+
+test("snapshotsEqual compares through the resolved kept-set", () => {
+ // A profile saved as legacy booleans and the same profile saved as `av` are
+ // the same filter; treating them as different would show a phantom "unsaved
+ // changes" state on load.
+ const legacy = { ...emptySnapshot(), nd: true };
+ const modern = { ...emptySnapshot(), av: [...snapshotStates(legacy)] };
+ assert.ok(snapshotsEqual(legacy, modern));
+ assert.ok(!snapshotsEqual(legacy, emptySnapshot()));
+});
+
+test("nop and nu survive a parse round-trip", () => {
+ // Both were written by committedSnapshot but never read back, so they were
+ // silently dropped on every reload.
+ const snap = { ...emptySnapshot(), nop: true, nu: true };
+ assert.ok(!snapshotsEqual(snap, { ...emptySnapshot(), nu: true }));
+ assert.ok(!snapshotsEqual(snap, { ...emptySnapshot(), nop: true }));
+});
diff --git a/common/components/shareUrl.ts b/common/components/shareUrl.ts
@@ -1,11 +1,12 @@
-// Share-link URL schema v1. Positive (selection-based) encoding of the
+// Share-link URL schema v1/v2. Positive (selection-based) encoding of the
// export filter state so links don't drift when the channel manifest grows.
//
-// Keep this file the *only* home for v1 parsing/building. The legacy schema
-// lives in `urlState.ts` (the `ch/nov/nol/.../tk` keys on `UrlParams`) plus
-// the `snapshotToExcluded` helper in `TranscriptSearch.tsx`. Retiring the
+// Keep this file the *only* home for share-link parsing/building. The legacy
+// schema lives in `urlState.ts` (the `ch/nov/nol/.../tk` keys on `UrlParams`)
+// plus the `snapshotToExcluded` helper in `TranscriptSearch.tsx`. Retiring the
// legacy schema later means deleting those legacy bits — this file stays.
+import type { VideoState } from "../lib/availability";
import type { SearchMode } from "./urlState";
export type ShareSelection = {
@@ -14,9 +15,9 @@ export type ShareSelection = {
livestreams: boolean;
allAges: boolean;
restricted: boolean;
- available: boolean;
- unlisted: boolean;
- deleted: boolean;
+ // Availability states to keep. Empty means the link's author deselected
+ // every one — a legitimate (if empty-result) selection.
+ states: ReadonlySet<VideoState>;
tracks: Set<string>;
// Inclusive upload-date bounds, "YYYYMMDD". Absent => unbounded on that end.
dateFrom?: string;
@@ -50,8 +51,48 @@ export const FILTER_URL_KEYS_V1 = [
"fdt",
] as const;
+// One token per VideoState in `fav`. Single letters because `fav` repeats per
+// kept state and share links get pasted into chat clients that wrap.
+const STATE_TOKENS: ReadonlyArray<[VideoState, string]> = [
+ ["available", "a"],
+ ["maybe_missing", "m"],
+ ["unlisted", "u"],
+ ["private", "p"],
+ ["members_only", "o"],
+ ["deleted", "d"],
+];
+
+export const SHARE_VERSION = "2";
+
+// v1 and v2 links are both accepted. The sentinel bump is what makes old links
+// safe: a v1 link carries at most `fav=a&fav=u&fav=d` and its author meant to
+// include everything we now call missing, whereas a NEW link that deliberately
+// drops the missing states is otherwise byte-identical. Without the version we
+// could not tell "wanted everything, three states existed" from "wanted only
+// these three".
export function hasShareV1(search: string): boolean {
- return new URLSearchParams(search).get("fv") === "1";
+ const v = new URLSearchParams(search).get("fv");
+ return v === "1" || v === "2";
+}
+
+// v1's three tokens mapped onto the six states: `a` also carries maybe_missing
+// (unconfirmed videos read as available when the link was written), `u` →
+// unlisted, `d` → deleted plus the two states v1 could not express, since a
+// private or members-only video showed up as deleted-or-nothing back then and
+// the author of a v1 link asking for removed videos wanted those too.
+function statesFromV1(tokens: ReadonlySet<string>): Set<VideoState> {
+ const keep = new Set<VideoState>();
+ if (tokens.has("a")) {
+ keep.add("available");
+ keep.add("maybe_missing");
+ }
+ if (tokens.has("u")) keep.add("unlisted");
+ if (tokens.has("d")) {
+ keep.add("deleted");
+ keep.add("private");
+ keep.add("members_only");
+ }
+ return keep;
}
// Channels not listed in `fc` are *not selected*. Unknown channel names in
@@ -70,6 +111,14 @@ export function parseShareV1(
const types = new Set(p.getAll("ft"));
const audience = new Set(p.getAll("fa"));
const availability = new Set(p.getAll("fav"));
+ const states =
+ p.get("fv") === "1"
+ ? statesFromV1(availability)
+ : new Set(
+ STATE_TOKENS.filter(([, tok]) => availability.has(tok)).map(
+ ([state]) => state,
+ ),
+ );
const tracks = new Set(p.getAll("fk"));
const dfRaw = p.get("fdf");
const dtRaw = p.get("fdt");
@@ -79,9 +128,7 @@ export function parseShareV1(
livestreams: types.has("l"),
allAges: audience.has("a"),
restricted: audience.has("r"),
- available: availability.has("a"),
- unlisted: availability.has("u"),
- deleted: availability.has("d"),
+ states,
tracks,
dateFrom: dfRaw && /^\d{8}$/.test(dfRaw) ? dfRaw : undefined,
dateTo: dtRaw && /^\d{8}$/.test(dtRaw) ? dtRaw : undefined,
@@ -93,16 +140,16 @@ export function buildShareSearchParams(
opts: { q: string; mode: SearchMode; regex: boolean },
): URLSearchParams {
const p = new URLSearchParams();
- p.set("fv", "1");
+ p.set("fv", SHARE_VERSION);
const channelNames = Array.from(committed.selectedChannels).sort();
for (const name of channelNames) p.append("fc", name);
if (committed.videos) p.append("ft", "v");
if (committed.livestreams) p.append("ft", "l");
if (committed.allAges) p.append("fa", "a");
if (committed.restricted) p.append("fa", "r");
- if (committed.available) p.append("fav", "a");
- if (committed.unlisted) p.append("fav", "u");
- if (committed.deleted) p.append("fav", "d");
+ for (const [state, tok] of STATE_TOKENS) {
+ if (committed.states.has(state)) p.append("fav", tok);
+ }
for (const tk of Array.from(committed.tracks).sort()) p.append("fk", tk);
if (committed.dateFrom) p.set("fdf", committed.dateFrom);
if (committed.dateTo) p.set("fdt", committed.dateTo);
diff --git a/common/controller/buildIndex.ts b/common/controller/buildIndex.ts
@@ -86,8 +86,17 @@ import {
type IndexTranscript,
type SubTrack,
} from "../lib/videoStatus";
-import { loadAvailability } from "../lib/availability-server";
-import { AVAILABILITY_FILENAME } from "../lib/availability";
+import {
+ resolveEffectiveAvailability,
+ resolveMaybeMissingState,
+} from "../lib/availability-server";
+import {
+ AVAILABILITY_FILENAME,
+ stateFromAvailability,
+ type Availability,
+ type VideoState,
+} from "../lib/availability";
+import { loadMaybeMissing } from "./quickAvailabilityCheck";
import {
POSTS_MANIFEST_VERSION,
SITE_POSTS_MANIFEST_VERSION,
@@ -180,11 +189,32 @@ type MtimeRecord = {
// mutation, so the correction would never ship. That is the entire reason
// the overrides file exists. See the subsMs multi-file loop it follows.
digestMs: number | null;
+ // The effective availability at the last index of this video (availability
+ // .json reconciled with download-outcome.json). Absent on records written
+ // before the VideoState work, which is why the two booleans below stay as
+ // the fallback (see stateFromRecord). Not worth a SCHEMA_VERSION bump
+ // — that wipes the whole cache and re-pages the entire corpus, whereas
+ // availabilityMs already participates in change detection, so any video whose
+ // availability.json is rewritten from here on picks up the exact string.
+ availability?: Availability;
isDeleted: boolean;
isUnlisted: boolean;
indexKey: IndexKey;
};
+// The probe result an MtimeRecord carries, preferring the exact availability
+// string and falling back to the pre-VideoState booleans.
+function stateFromRecord(rec: MtimeRecord): VideoState {
+ if (rec.availability) return stateFromAvailability(rec.availability);
+ if (rec.isDeleted) return "deleted";
+ if (rec.isUnlisted) return "unlisted";
+ return "available";
+}
+
+function indexKeyId(k: IndexKey): string {
+ return `${k[0]}\x00${k[1]}\x00${k[2]}`;
+}
+
type PageHashRecord = {
hash: string;
entryCount: number;
@@ -411,7 +441,7 @@ export async function buildIndex({
const root = open({
path: dbPath,
- maxDbs: 17,
+ maxDbs: 18,
compression: true,
});
const sums = root.openDB<TranscriptSummary, IndexKey>({
@@ -479,6 +509,15 @@ export async function buildIndex({
name: "channelDigestStats",
encoding: "msgpack",
});
+ // pathKeyId -> VideoState, sparse (only non-`available` videos). Written for
+ // buildStats, which runs after us against this same LMDB file and whose own
+ // per-video cache is keyed on metadata mtime alone — so it cannot see an
+ // availability flip on its own. Keyed by path rather than indexKey because
+ // that is what buildStats has to hand.
+ const videoState = root.openDB<VideoState, string>({
+ name: "videoState",
+ encoding: "msgpack",
+ });
const meta = root.openDB<unknown, string>({
name: "meta",
encoding: "msgpack",
@@ -504,6 +543,7 @@ export async function buildIndex({
await digests.clearAsync();
await digestPageHashes.clearAsync();
await channelDigestStatsDb.clearAsync();
+ await videoState.clearAsync();
await meta.put("schema", SCHEMA_VERSION);
}
@@ -737,12 +777,18 @@ export async function buildIndex({
digests.remove(indexKey);
}
- let isDeleted = false;
- let isUnlisted = false;
+ // Effective, not just availability.json's top-level field: a download
+ // attempt that fails with "deleted" records the class in
+ // download-outcome.json and deliberately leaves the top-level fields
+ // alone (see downloadOneManaged), so the file alone would still read
+ // "public". It does append an availability *history* entry, which
+ // bumps availability.json's mtime — which is what brings us back here
+ // to notice. Same source buildStats' status chart used to read
+ // directly, so the two can't disagree.
+ let availability: Availability | undefined;
if (s.availabilityMs !== null) {
- const availability = await loadAvailability(videoFullDir);
- isDeleted = availability?.availability === "deleted";
- isUnlisted = availability?.availability === "unlisted";
+ availability =
+ (await resolveEffectiveAvailability(videoFullDir)) ?? undefined;
}
byChannel.put(indexToChannelKey(indexKey), 1);
@@ -752,8 +798,9 @@ export async function buildIndex({
subsMs: s.subsMs,
availabilityMs: s.availabilityMs,
digestMs: s.digestMs,
- isDeleted,
- isUnlisted,
+ ...(availability ? { availability } : {}),
+ isDeleted: availability === "deleted",
+ isUnlisted: availability === "unlisted",
indexKey,
});
} catch (err) {
@@ -1021,22 +1068,68 @@ export async function buildIndex({
log(`Shared transcript pages up to date; skipping rewrite.`);
}
- // Build lookups from indexKey → isDeleted / isUnlisted from MtimeRecord, which
- // caches the availability check done at mutation-processing time. Previously
- // this loop read availability.json for every video on every build (~28k
- // sequential disk reads), which stalled GC right before the heaviest phase.
- const deletedByIndexKey = new Set<string>();
- const unlistedByIndexKey = new Set<string>();
- for (const { value } of mtimes.getRange()) {
+ // Build lookups from indexKey → VideoState out of MtimeRecord, which caches
+ // the availability check done at mutation-processing time. Previously this
+ // loop read availability.json for every video on every build (~28k sequential
+ // disk reads), which stalled GC right before the heaviest phase.
+ //
+ // Both maps are SPARSE: only non-`available` videos get an entry, so the
+ // common case costs nothing. stateByPath is the same data keyed the way
+ // buildStats needs it (see the videoState sub-DB written below).
+ const stateByIndexKey = new Map<string, VideoState>();
+ const stateByPath = new Map<string, VideoState>();
+ for (const { key, value } of mtimes.getRange()) {
const rec = value as MtimeRecord;
- const ik = rec.indexKey;
- if (rec.isDeleted) {
- deletedByIndexKey.add(`${ik[0]}\x00${ik[1]}\x00${ik[2]}`);
- }
- if (rec.isUnlisted) {
- unlistedByIndexKey.add(`${ik[0]}\x00${ik[1]}\x00${ik[2]}`);
+ const st = stateFromRecord(rec);
+ if (st === "available") continue;
+ stateByIndexKey.set(indexKeyId(rec.indexKey), st);
+ stateByPath.set(pathKeyId(key as PathKey), st);
+ }
+
+ // Overlay the channel-level quick availability check. Computed here rather
+ // than in the per-video loop above on purpose: maybe-missing.json is a
+ // *channel*-level file, but the per-video `mtimes` cache keys off per-video
+ // mtimes, so a channel-level change would never invalidate it and the flag
+ // would simply never ship. Doing it here sidesteps invalidation entirely, and
+ // since the maybe-missing set is small by construction, reading
+ // availability.json for just those ids is cheap and exact.
+ let maybeMissingCount = 0;
+ for (const slug of channelConfigs.keys()) {
+ const record = await loadMaybeMissing(paths, slug);
+ if (!record?.ids.length) continue;
+ const scannedAtMs = Date.parse(record.checkedAt);
+ if (!Number.isFinite(scannedAtMs)) continue;
+ for (const id of record.ids) {
+ // The quick check's ids ARE data-dir names, which is exactly the second
+ // half of a PathKey (see quickAvailabilityCheck's own comment).
+ const rec = mtimes.get([slug, id]);
+ if (!rec) continue; // not indexed → nothing to show
+ const ikId = indexKeyId(rec.indexKey);
+ if (stateByIndexKey.has(ikId)) continue; // already a confirmed state
+ const state = await resolveMaybeMissingState(
+ path.join(channelsDir, slug, "data", id),
+ scannedAtMs,
+ );
+ if (state === "available") continue;
+ if (state === "maybe_missing") maybeMissingCount++;
+ stateByIndexKey.set(ikId, state);
+ stateByPath.set(pathKeyId([slug, id]), state);
}
}
+ if (maybeMissingCount > 0) {
+ log(
+ `Availability: ${maybeMissingCount} video(s) unconfirmed-missing from a channel listing.`,
+ );
+ }
+
+ // Publish the sparse map for buildStats, which runs after us against the same
+ // LMDB file and would otherwise have to re-read ~76k availability.json files
+ // to build the status chart. Rewritten wholesale each build: the map is small
+ // and maybe_missing is derived at build time, so reconciling would cost more
+ // than it saves.
+ await videoState.clearAsync();
+ for (const [id, st] of stateByPath) videoState.put(id, st);
+ await videoState.flushed;
let subsPagesWritten = 0;
let subsPagesSkipped = 0;
@@ -1078,10 +1171,9 @@ export async function buildIndex({
if (!stored || stored.length === 0) continue;
const summary = sums.get(indexKey);
if (!summary) continue;
- const subsKey = `${indexKey[0]}\x00${indexKey[1]}\x00${indexKey[2]}`;
- const isDeleted = deletedByIndexKey.has(subsKey);
- const isUnlisted = unlistedByIndexKey.has(subsKey);
- const display = toDisplaySummary(summary, { isDeleted, isUnlisted });
+ const display = toDisplaySummary(summary, {
+ state: stateByIndexKey.get(indexKeyId(indexKey)),
+ });
const tracks: Record<string, Cue[]> = {};
let hasLiveChat = false;
for (const t of stored) {
@@ -1634,12 +1726,8 @@ export async function buildIndex({
const s = value as TranscriptSummary;
const acc = chan.get(ik[1]);
if (acc) acc.count++;
- const summaryKey = `${ik[0]}\x00${ik[1]}\x00${ik[2]}`;
buffer.push(
- toDisplaySummary(s, {
- isDeleted: deletedByIndexKey.has(summaryKey),
- isUnlisted: unlistedByIndexKey.has(summaryKey),
- }),
+ toDisplaySummary(s, { state: stateByIndexKey.get(indexKeyId(ik)) }),
);
total++;
if (buffer.length >= pageSize) await flushPage();
diff --git a/common/controller/buildStats.ts b/common/controller/buildStats.ts
Binary files differ.
diff --git a/common/lib/availability-server.ts b/common/lib/availability-server.ts
@@ -3,7 +3,9 @@ import { readFile, rename, writeFile } from "node:fs/promises";
import {
AVAILABILITY_FILENAME,
AVAILABILITY_VALUES,
+ stateFromAvailability,
type Availability,
+ type VideoState,
type AvailabilityHistoryEntry,
type AvailabilityHistorySource,
type AvailabilityRecord,
@@ -138,6 +140,37 @@ export async function resolveEffectiveAvailability(
return outcome?.availability ?? null;
}
+// Decide the VideoState for one video that the last quick availability check
+// found absent from its channel's fresh playlist listing. `scannedAtMs` is that
+// check's `checkedAt`. Three outcomes, and both of the non-obvious ones earn
+// their keep:
+//
+// - a per-video probe that already *explains* the absence wins outright.
+// An unlisted video legitimately falls out of a listing forever, and
+// checkMaybeMissingAction passes `skipExpectedAbsent: true`, so a
+// known-gone video is never re-probed and would otherwise stay flagged
+// as unconfirmed permanently.
+// - the converse: a video re-probed *after* the scan that came back public
+// anyway (a truncated or oddly paginated playlist fetch) is resolved, and
+// must stop being flagged.
+//
+// Anything else — never probed, or last probed before the scan — is
+// `maybe_missing`: we know it left the listing, not yet why.
+export async function resolveMaybeMissingState(
+ videoDir: string,
+ scannedAtMs: number,
+): Promise<VideoState> {
+ const record = await loadAvailability(videoDir);
+ if (!record) return "maybe_missing";
+ const confirmed = stateFromAvailability(record.availability);
+ if (confirmed !== "available") return confirmed;
+ const checkedAtMs = Date.parse(record.checkedAt);
+ if (Number.isFinite(checkedAtMs) && checkedAtMs >= scannedAtMs) {
+ return "available";
+ }
+ return "maybe_missing";
+}
+
async function loadDownloadOutcomeAvailability(
videoDir: string,
): Promise<{ availability: Availability; finishedAt: string | null } | null> {
diff --git a/common/lib/availability.ts b/common/lib/availability.ts
@@ -55,6 +55,87 @@ export const AUTH_RETRY_CLASSES: ReadonlySet<Availability> = new Set<Availabilit
"private",
]);
+// What the viewer shows for a video's presence on its source platform. Distinct
+// from `Availability`, which is what a per-video probe *reported*: this adds
+// `maybe_missing` (fell out of the channel listing, not yet individually
+// probed) and folds needs_auth/error back into `available`, since neither is
+// evidence the video is gone.
+export type VideoState =
+ | "available"
+ | "maybe_missing"
+ | "unlisted"
+ | "private"
+ | "members_only"
+ | "deleted";
+
+// Ordered for UI: the umbrella's leaves in the order they're offered.
+export const MISSING_STATES: ReadonlyArray<VideoState> = [
+ "maybe_missing",
+ "deleted",
+ "private",
+ "members_only",
+ "unlisted",
+];
+
+export const VIDEO_STATES: ReadonlyArray<VideoState> = [
+ "available",
+ ...MISSING_STATES,
+];
+
+// Every state except `available` means the video left its channel's listing.
+export function isMissingState(s: VideoState): boolean {
+ return s !== "available";
+}
+
+export function isVideoState(v: unknown): v is VideoState {
+ return typeof v === "string" && (VIDEO_STATES as string[]).includes(v);
+}
+
+// What a per-video probe result means for the viewer. `needs_auth` and `error`
+// map to `available`: an age-gated video or a transient fetch failure is not
+// evidence the video left its channel.
+export function stateFromAvailability(
+ a: Availability | null | undefined,
+): VideoState {
+ switch (a) {
+ case "deleted":
+ return "deleted";
+ case "private":
+ return "private";
+ case "members_only":
+ return "members_only";
+ case "unlisted":
+ return "unlisted";
+ default:
+ return "available";
+ }
+}
+
+// Read a summary's state, falling back to the legacy booleans. The fallback is
+// not dead code: in hub mode the MCP and the federated viewer read summaries
+// pages straight off member origins, which may have been built before `state`
+// existed. Structurally typed so DisplaySummary (which pulls in node-only
+// helpers) needn't be imported into client code.
+export function summaryState(t: {
+ state?: VideoState;
+ isDeleted?: boolean;
+ isUnlisted?: boolean;
+}): VideoState {
+ if (t.state) return t.state;
+ if (t.isDeleted) return "deleted";
+ if (t.isUnlisted) return "unlisted";
+ return "available";
+}
+
+export const VIDEO_STATE_LABELS: Record<VideoState, string> = {
+ available: "Available",
+ maybe_missing: "Unconfirmed",
+ deleted: "Deleted",
+ private: "Private",
+ members_only: "Members-only",
+ unlisted: "Unlisted",
+};
+
export type AvailabilityHistorySource = "check" | "backfill" | "download";
export type AvailabilityHistoryEntry = {
diff --git a/common/lib/stats.ts b/common/lib/stats.ts
@@ -1,13 +1,15 @@
import type { Platform } from "./platform";
+import type { VideoState } from "./availability";
// Bumping this invalidates the LMDB `statsByPath` incremental cache and forces
// a full re-extraction (e.g. when a new field is added below).
export const STATS_SCHEMA_VERSION = 5;
export const STATS_MANIFEST_VERSION = 1;
-// Visibility of a video on its source platform, derived from the availability
-// check (available = still public, unlisted, or fully removed/deleted).
-export type VideoStatus = "available" | "unlisted" | "deleted";
+// Visibility of a video on its source platform. One type with the viewer's, so
+// the status chart and the search filter can never drift apart; see VideoState
+// in lib/availability for the taxonomy (available + the five missing leaves).
+export type VideoStatus = VideoState;
// Coarse media kind. YouTube distinguishes shorts; other platforms only carry
// video vs. livestream, so they fall back to one of those.
diff --git a/common/lib/transcripts-server.ts b/common/lib/transcripts-server.ts
@@ -4,6 +4,7 @@ import { formatDate, formatDuration } from "./format";
import { defaultWebpageUrl } from "./platform";
import type { DisplaySummary, Platform, TranscriptSummary } from "./transcripts";
import type { MediaType, VideoStat, VideoStatus } from "./stats";
+import type { VideoState } from "./availability";
export type RawMetadata = {
id?: string;
@@ -193,8 +194,9 @@ export function summarizeStats(
export function toDisplaySummary(
t: TranscriptSummary,
- extras: { isDeleted?: boolean; isUnlisted?: boolean } = {},
+ extras: { state?: VideoState } = {},
): DisplaySummary {
+ const state = extras.state ?? "available";
return {
slug: t.slug,
id: t.id,
@@ -206,8 +208,16 @@ export function toDisplaySummary(
channel: t.channel,
isLivestream: t.isLivestream,
ageRestricted: t.ageRestricted,
- isDeleted: extras.isDeleted === true,
- isUnlisted: extras.isUnlisted === true,
+ // Derived, not passed: `state` is the single source of truth. These stay on
+ // the wire for readers that predate it (see DisplaySummary).
+ isDeleted: state === "deleted",
+ isUnlisted: state === "unlisted",
+ // Omitted rather than written as "available": this key is new, so a page
+ // with no missing video stays byte-identical to the one already on disk.
+ // The summaries pages themselves are rewritten unconditionally
+ // (writeJsonAtomic), so the beneficiaries are the subs page hashes in
+ // createPageWriter and the compose-site dirSignature reconcile.
+ ...(state === "available" ? {} : { state }),
platform: t.platform,
webpageUrl: t.webpageUrl,
};
diff --git a/common/lib/transcripts.ts b/common/lib/transcripts.ts
@@ -5,6 +5,7 @@ import { getPaths } from "./paths";
export type { Platform } from "./platform";
import type { Platform } from "./platform";
+import type { VideoState } from "./availability";
export type TranscriptSummary = {
slug: string;
@@ -38,8 +39,17 @@ export type DisplaySummary = {
channel: string;
isLivestream: boolean;
ageRestricted: boolean;
+ // Legacy booleans, derived from `state`. Still emitted because the MCP's
+ // RemoteSource reads summaries pages from live origins it does not control,
+ // so version skew between a hub and its members is real: an older reader must
+ // keep seeing deleted/unlisted, and an older *site* must keep answering a
+ // newer reader.
isDeleted: boolean;
isUnlisted: boolean;
+ // Presence on the source platform. Omitted when "available" (absent =>
+ // available), so a page with no missing videos stays byte-identical to the
+ // one already on disk and the build's downstream hashing still short-circuits.
+ state?: VideoState;
platform: Platform;
webpageUrl: string;
};
diff --git a/common/lib/videoState.test.ts b/common/lib/videoState.test.ts
@@ -0,0 +1,159 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import {
+ AVAILABILITY_FILENAME,
+ stateFromAvailability,
+ summaryState,
+ type Availability,
+} from "./availability";
+import { resolveMaybeMissingState } from "./availability-server";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common exec tsx --test common/lib/videoState.test.ts
+//
+// Covers the maybe_missing predicate — the one piece of the VideoState work
+// that is a genuine decision rather than a mapping. Its two non-obvious rules
+// (a probe that explains the absence wins; a probe made AFTER the scan that
+// came back public clears the flag) are what keep the "Missing?" badge from
+// either sticking forever on legitimately-unlisted videos or lingering after a
+// truncated playlist fetch is disproved.
+
+const SCAN_AT = "2026-08-01T12:00:00.000Z";
+const SCAN_AT_MS = Date.parse(SCAN_AT);
+const BEFORE_SCAN = "2026-07-20T00:00:00.000Z";
+const AFTER_SCAN = "2026-08-02T00:00:00.000Z";
+
+async function withVideoDir(
+ fn: (videoDir: string) => Promise<void>,
+): Promise<void> {
+ const dir = await mkdtemp(path.join(tmpdir(), "ttb-vstate-"));
+ const videoDir = path.join(dir, "data", "vid1");
+ await mkdir(videoDir, { recursive: true });
+ try {
+ await fn(videoDir);
+ } finally {
+ await rm(dir, { recursive: true, force: true });
+ }
+}
+
+async function seedAvailability(
+ videoDir: string,
+ availability: Availability,
+ checkedAt: string,
+): Promise<void> {
+ await writeFile(
+ path.join(videoDir, AVAILABILITY_FILENAME),
+ JSON.stringify({ checkedAt, availability }),
+ );
+}
+
+test("in the maybe-missing set with no availability.json → maybe_missing", async () => {
+ await withVideoDir(async (videoDir) => {
+ assert.equal(
+ await resolveMaybeMissingState(videoDir, SCAN_AT_MS),
+ "maybe_missing",
+ );
+ });
+});
+
+// A probe that names a reason for the absence wins outright. This is not a
+// nicety: checkMaybeMissingAction passes skipExpectedAbsent, so these videos
+// are never re-probed — without this rule they would carry "Missing?" forever.
+for (const [availability, expected] of [
+ ["deleted", "deleted"],
+ ["private", "private"],
+ ["members_only", "members_only"],
+ ["unlisted", "unlisted"],
+] as const) {
+ test(`in the set + ${availability} → ${expected}, not maybe_missing`, async () => {
+ await withVideoDir(async (videoDir) => {
+ await seedAvailability(videoDir, availability, BEFORE_SCAN);
+ assert.equal(
+ await resolveMaybeMissingState(videoDir, SCAN_AT_MS),
+ expected,
+ );
+ });
+ });
+}
+
+test("in the set + public, re-probed AFTER the scan → available", async () => {
+ await withVideoDir(async (videoDir) => {
+ await seedAvailability(videoDir, "public", AFTER_SCAN);
+ assert.equal(
+ await resolveMaybeMissingState(videoDir, SCAN_AT_MS),
+ "available",
+ );
+ });
+});
+
+test("in the set + public, last probed BEFORE the scan → maybe_missing", async () => {
+ await withVideoDir(async (videoDir) => {
+ await seedAvailability(videoDir, "public", BEFORE_SCAN);
+ assert.equal(
+ await resolveMaybeMissingState(videoDir, SCAN_AT_MS),
+ "maybe_missing",
+ );
+ });
+});
+
+// needs_auth/error are not evidence the video left its channel, so they behave
+// exactly like public: still absent from the listing, still unconfirmed.
+for (const availability of ["needs_auth", "error"] as const) {
+ test(`in the set + ${availability} before the scan → maybe_missing`, async () => {
+ await withVideoDir(async (videoDir) => {
+ await seedAvailability(videoDir, availability, BEFORE_SCAN);
+ assert.equal(
+ await resolveMaybeMissingState(videoDir, SCAN_AT_MS),
+ "maybe_missing",
+ );
+ });
+ });
+}
+
+test("a malformed checkedAt cannot resolve the flag", async () => {
+ await withVideoDir(async (videoDir) => {
+ await seedAvailability(videoDir, "public", "not-a-date");
+ assert.equal(
+ await resolveMaybeMissingState(videoDir, SCAN_AT_MS),
+ "maybe_missing",
+ );
+ });
+});
+
+// ─── the availability → state mapping ───
+
+test("stateFromAvailability folds needs_auth/error/public into available", () => {
+ assert.equal(stateFromAvailability("public"), "available");
+ assert.equal(stateFromAvailability("needs_auth"), "available");
+ assert.equal(stateFromAvailability("error"), "available");
+ assert.equal(stateFromAvailability(null), "available");
+ assert.equal(stateFromAvailability(undefined), "available");
+ assert.equal(stateFromAvailability("deleted"), "deleted");
+ assert.equal(stateFromAvailability("private"), "private");
+ assert.equal(stateFromAvailability("members_only"), "members_only");
+ assert.equal(stateFromAvailability("unlisted"), "unlisted");
+});
+
+// A video not in any channel's maybe-missing set never reaches the predicate;
+// its state comes from the summary, which is `available` when nothing is set.
+test("summaryState: absent state + no legacy booleans → available", () => {
+ assert.equal(summaryState({}), "available");
+ assert.equal(summaryState({ isDeleted: false, isUnlisted: false }), "available");
+});
+
+// The hub reads summaries pages off member origins it does not control, so a
+// page built before `state` existed must still filter correctly.
+test("summaryState: falls back to the legacy booleans", () => {
+ assert.equal(summaryState({ isDeleted: true }), "deleted");
+ assert.equal(summaryState({ isUnlisted: true }), "unlisted");
+});
+
+test("summaryState: an explicit state wins over the legacy booleans", () => {
+ assert.equal(
+ summaryState({ state: "private", isDeleted: true }),
+ "private",
+ );
+});
diff --git a/editor/next.config.ts b/editor/next.config.ts
@@ -1,7 +1,16 @@
+import path from "node:path";
import type { NextConfig } from "next";
const nextConfig: NextConfig = {
turbopack: {
+ // Pin the workspace root. Turbopack infers it by walking up for a lockfile
+ // and taking the outermost one, so an unrelated pnpm-lock.yaml anywhere
+ // above the checkout (a stray one in $HOME is enough) silently relocates
+ // the root — after which `@import "../../common/styles/tokens.css"` in
+ // app/globals.css resolves outside the project and `next dev` fails to
+ // boot at all. Nothing in this repo lives above the monorepo root, so
+ // pinning it costs nothing.
+ root: path.join(__dirname, ".."),
// Suppress the harmless "whole project was traced unintentionally" NFT
// warning. Turbopack's file tracer conservatively globs the cwd because our
// server code does legitimate runtime-dynamic fs reads it can't statically
diff --git a/export/CHANGELOG.md b/export/CHANGELOG.md
@@ -1,5 +1,11 @@
# Changelog
+## [Unreleased]
+- **A video that has gone missing now says so, and says how confidently.** Availability used to be three unlabelled buckets — available, unlisted, deleted — with no marking on the result cards themselves: a deleted video looked exactly like a live one unless you already suspected something and went hunting in the filters. It is now a **state on every card**: `DELETED`, `PRIVATE`, `MEMBERS`, `UNLISTED`, or `MISSING?`. The last one is new and it is the point of the change. Checking a channel's listing is cheap (one request per channel); confirming *why* an individual video vanished is slow, so on a large channel there is a long window where the archive knows a video has dropped out of its channel but not yet what happened to it. The site used to spend that window insisting the video was fine. It now shows **`MISSING?`** — amber, with the question mark, because it is a suspicion and not a finding — and swaps in the confirmed reason once the per-video check catches up. A video re-checked after the scan that turns out to be present simply loses the flag.
+- **The availability filter follows the same shape.** "Available" and "Missing" are now a parent and its five leaves (unconfirmed, deleted, private, members-only, unlisted); ticking the parent takes all five, and it shows a dash when you have only some. Unlisted sits under "missing" because the rule is one sentence — *it left the channel's listing* — even though an unlisted video is still watchable by direct link. **Saved filter profiles and old share links keep working**: an existing link that asked for available/unlisted/deleted still means "all of them" under the new taxonomy rather than silently narrowing.
+- **Two filter bugs fixed on the way past.** Excluding social posts, and excluding unlisted videos, were both written to your saved filters but never read back — so either choice quietly reverted on reload. Both now persist.
+- **The status chart stops lagging behind reality.** Chart data was keyed on a video's metadata timestamp alone, so deleting a video today did not change the chart until something unrelated happened to touch that video's metadata. Availability is now read from the index directly, which also means the chart understands private and members-only rather than folding them into "available".
+
## [0.8.3] - 2026-08-04
- **The archive now recognises far more cross-platform re-uploads as the same video.** Duplicate detection compared two transcripts and called them the same only above a similarity of 0.6 — a threshold tuned for two transcripts of the same *text*, which quietly failed the case detection exists for. The two sides of a YouTube↔Rumble mirror are transcribed by **different speech-recognition engines**, and word-level disagreement between them lands a five-word-window comparison at roughly 0.35–0.60, i.e. just *under* the old cutoff. The threshold is now **0.35**, which takes the archive from 2,846 duplicate clusters over 5,746 videos to **7,434 clusters over 14,997 videos** — so a search result is far more likely to tell you the same recording exists elsewhere, and to offer you the jump. The change was bracketed at four values on the full archive before being made, and every step is a **strict superset**: no video that was previously flagged as a duplicate stopped being one. What it newly admits was inspected rather than counted — 96–97% have byte-identical titles, 99% span two platforms, and the handful of same-channel cases were read individually. Nothing about *how* a duplicate is decided changed: pairs are still confirmed by comparing real transcripts, a pair that merely shares a title and a runtime is still an internal review item that never reaches you, and the "jump to this moment in the other copy" button still only carries your timestamp when the two were *measured* as aligned.
- **AI chapters: jump straight to the part of a video you want.** Videos that have been through the local-AI digest pass now ship their derived **chapters and topic tags** to the site, and the player gains a third panel beside Transcript and Live chat. Open it and you get a titled list of moments — click one and the player **seeks there**; the chapter you're currently inside stays marked as the video plays. The layer is **sparse on purpose and honest about it**: only a small, growing fraction of the archive has been digested (generation is a multi-week GPU pass), so the control simply **isn't shown** on a video that has no digest, rather than offering a button that opens an empty panel — and if a digest can't be loaded, the panel snaps back to the transcript with a notice instead of stranding you. What ships is the **composed** digest: any human correction is applied and any chapter a human rejected is dropped, so you see what a person approved rather than raw model output, with hand-edited chapters marked **edited** and a provenance line naming the model that wrote the rest. **A digest borrowed from a duplicate upload says so, prominently** — when the same recording exists twice in the archive, one copy's chapters can be shared onto the other, and the panel names the source video and the measured timing offset rather than passing them off as native (plausible chapters describing a *different* upload is the failure that looks like success). Digests live at `/digests/<channel>/` under the same paginated-shard scheme as transcripts and posts, are offline-cached by the service worker, and are described in `corpus.json` — which bumps to **spec 3** with a `digestScheme` and per-channel `digests` manifest pointers, so an AI tool reading the corpus can navigate them and knows that an absent video means "not yet generated" rather than "nothing to say". Deep links carry the panel (`?vm=digest`), and share links reopen on it. Operator telemetry (why the model's proposals were rejected, the regeneration history) is deliberately **not** shipped — that stays in the editor. See `common/lib/digests.ts`, `common/components/{digestCache,digestStore}.ts`, `common/components/{PlayerProvider,TranscriptModal,urlState}.tsx/ts`, and `export/e2e/modal-digest.spec.ts`.
diff --git a/export/e2e/availability-state.spec.ts b/export/e2e/availability-state.spec.ts
@@ -0,0 +1,329 @@
+import { expect, test, type Page, type Route } from "@playwright/test";
+
+// Self-contained fixtures with one video per VideoState, so the state badges
+// and the nested Availability filter (common/components/WorkspaceSearchBar.tsx
+// + SearchSessionContext.passesFilter) each have every leaf to exercise.
+//
+// Availability is the first per-card indicator this viewer has ever had:
+// before the state enum, a deleted or unlisted video looked exactly like an
+// available one and the only signal was a filter you had to already suspect.
+//
+// Searches run in the "Title / channel" (metadata) scope — every title
+// contains "report", so presence state is the only variable.
+
+const CHANNEL = "Test Channel";
+const CHANNEL_SLUG = "test-channel";
+
+type State =
+ | "available"
+ | "maybe_missing"
+ | "unlisted"
+ | "private"
+ | "members_only"
+ | "deleted";
+
+type Vid = { id: string; title: string; state: State };
+
+const VIDS: Vid[] = [
+ { id: "vid-available", title: "Report available", state: "available" },
+ { id: "vid-maybe", title: "Report unconfirmed", state: "maybe_missing" },
+ { id: "vid-unlisted", title: "Report unlisted", state: "unlisted" },
+ { id: "vid-private", title: "Report private", state: "private" },
+ { id: "vid-members", title: "Report members", state: "members_only" },
+ { id: "vid-deleted", title: "Report deleted", state: "deleted" },
+];
+
+function summary(v: Vid) {
+ return {
+ slug: `${CHANNEL_SLUG}/${v.id}`,
+ id: v.id,
+ channelSlug: CHANNEL_SLUG,
+ title: v.title,
+ uploadDate: "20260101",
+ date: "2026-01-01",
+ duration: "5:00",
+ channel: CHANNEL,
+ isLivestream: false,
+ ageRestricted: false,
+ // Written the way the build writes them: derived from state, and `state`
+ // itself omitted when available.
+ isDeleted: v.state === "deleted",
+ isUnlisted: v.state === "unlisted",
+ ...(v.state === "available" ? {} : { state: v.state }),
+ platform: "youtube" as const,
+ webpageUrl: `https://example.com/${v.id}`,
+ };
+}
+
+async function fulfillJson(route: Route, body: unknown) {
+ await route.fulfill({
+ status: 200,
+ contentType: "application/json",
+ body: JSON.stringify(body),
+ });
+}
+
+async function installRoutes(page: Page) {
+ const list = VIDS.map(summary);
+ await page.route("**/summaries/manifest.json", async (route) => {
+ await fulfillJson(route, {
+ version: 3,
+ totalCount: list.length,
+ pageSize: 1000,
+ pageCount: 1,
+ generatedAt: new Date().toISOString(),
+ channels: [{ name: CHANNEL, count: list.length, groupId: "default" }],
+ groups: [{ id: "default", name: "All channels", selectedByDefault: true }],
+ defaultGroupId: "default",
+ });
+ });
+ await page.route("**/summaries/page-*.json", async (route) => {
+ await fulfillJson(route, list);
+ });
+ await page.route(/\/transcripts\/[^/]+\/manifest\.json$/, async (route) => {
+ const slugToPage: Record<string, number> = {};
+ for (const v of VIDS) slugToPage[v.id] = 0;
+ await fulfillJson(route, {
+ version: 1,
+ channelSlug: CHANNEL_SLUG,
+ pageCount: 1,
+ maxPageBytes: 8388608,
+ generatedAt: new Date().toISOString(),
+ slugToPage,
+ });
+ });
+ await page.route(/\/transcripts\/[^/]+\/page-\d+\.json$/, async (route) => {
+ await fulfillJson(
+ route,
+ list.map((s) => ({ slug: s.slug, id: s.id, cues: [] })),
+ );
+ });
+ await page.route("**/subs/manifest.json", async (route) => {
+ await fulfillJson(route, {
+ version: 4,
+ channels: [],
+ totalCount: 0,
+ liveChatTotalCount: 0,
+ generatedAt: new Date().toISOString(),
+ });
+ });
+}
+
+function leafInput(page: Page) {
+ return page.locator('[data-testid^="leaf-query-"]').first();
+}
+function scopeSelect(page: Page) {
+ return page.locator('[data-testid^="leaf-scope-"]').first();
+}
+
+async function waitForHydration(page: Page) {
+ await page.getByTestId("query-builder").waitFor();
+}
+
+async function searchReports(page: Page) {
+ await scopeSelect(page).selectOption({ label: "Title / channel" });
+ const input = leafInput(page);
+ await input.click();
+ await input.fill("report");
+ await input.press("Enter");
+ await page.waitForURL(/[?&]qt=/);
+}
+
+// The result-card header button carries "<title> <channel>".
+function card(page: Page, titleRe: RegExp) {
+ return page.getByRole("button", {
+ name: new RegExp(`${titleRe.source}.*Test Channel`),
+ });
+}
+
+// Toggle a filter checkbox and commit. The checkboxes are Radix roots, so
+// click the control itself rather than its label text.
+async function setFilter(page: Page, testId: string, checked: boolean) {
+ const box = page.getByTestId(testId);
+ const state = await box.getAttribute("data-state");
+ const isChecked = state === "checked";
+ if (isChecked !== checked) await box.click();
+ await page.getByTestId("search-submit").click();
+}
+
+test.describe("export search — availability state badges", () => {
+ test.beforeEach(async ({ page }) => {
+ await installRoutes(page);
+ await page.goto("/");
+ await waitForHydration(page);
+ });
+
+ test("each missing state renders its badge; available renders none", async ({
+ page,
+ }) => {
+ await searchReports(page);
+ for (const v of VIDS) await expect(card(page, new RegExp(v.title))).toBeVisible();
+
+ await expect(page.getByTestId("state-badge-maybe_missing")).toHaveText(
+ /Missing\?/i,
+ );
+ await expect(page.getByTestId("state-badge-deleted")).toHaveText(/Deleted/i);
+ await expect(page.getByTestId("state-badge-private")).toHaveText(/Private/i);
+ await expect(page.getByTestId("state-badge-members_only")).toHaveText(
+ /Members/i,
+ );
+ await expect(page.getByTestId("state-badge-unlisted")).toHaveText(
+ /Unlisted/i,
+ );
+ // Six videos, five badges — the available one carries nothing.
+ await expect(page.locator('[data-testid^="state-badge-"]')).toHaveCount(5);
+ });
+
+ test("the unconfirmed badge reads as a suspicion, not a finding", async ({
+ page,
+ }) => {
+ await searchReports(page);
+ const badge = page.getByTestId("state-badge-maybe_missing");
+ // The "?" is the whole point: we know it left the listing, not why.
+ await expect(badge).toHaveText(/\?$/);
+ await expect(badge).toHaveAttribute("title", /not yet confirmed/i);
+ });
+});
+
+test.describe("export search — nested availability filter", () => {
+ test.beforeEach(async ({ page }) => {
+ await installRoutes(page);
+ await page.goto("/");
+ await waitForHydration(page);
+ });
+
+ test("unchecking Missing drops all five leaves, keeping only available", async ({
+ page,
+ }) => {
+ await searchReports(page);
+ await setFilter(page, "av-missing", false);
+
+ await expect(card(page, /Report available/)).toBeVisible();
+ for (const v of VIDS.filter((x) => x.state !== "available")) {
+ await expect(card(page, new RegExp(v.title))).toHaveCount(0);
+ }
+ });
+
+ test("unchecking Available keeps every missing video", async ({ page }) => {
+ await searchReports(page);
+ await setFilter(page, "av-available", false);
+
+ await expect(card(page, /Report available/)).toHaveCount(0);
+ for (const v of VIDS.filter((x) => x.state !== "available")) {
+ await expect(card(page, new RegExp(v.title))).toBeVisible();
+ }
+ });
+
+ test("a single leaf filters to exactly its own state", async ({ page }) => {
+ await searchReports(page);
+ // Drop the parent, then re-check one leaf.
+ await setFilter(page, "av-missing", false);
+ await setFilter(page, "av-available", false);
+ await setFilter(page, "av-private", true);
+
+ await expect(card(page, /Report private/)).toBeVisible();
+ await expect(card(page, /Report available/)).toHaveCount(0);
+ await expect(card(page, /Report deleted/)).toHaveCount(0);
+ await expect(card(page, /Report unconfirmed/)).toHaveCount(0);
+ });
+
+ test("the Missing parent goes indeterminate when only some leaves are kept", async ({
+ page,
+ }) => {
+ await searchReports(page);
+ const parent = page.getByTestId("av-missing");
+ await expect(parent).toHaveAttribute("data-state", "checked");
+
+ // One leaf off → some but not all.
+ await page.getByTestId("av-deleted").click();
+ await expect(parent).toHaveAttribute("data-state", "indeterminate");
+
+ // All leaves off → unchecked.
+ for (const s of ["maybe_missing", "private", "members_only", "unlisted"]) {
+ await page.getByTestId(`av-${s}`).click();
+ }
+ await expect(parent).toHaveAttribute("data-state", "unchecked");
+
+ // Toggling the parent back on restores every leaf.
+ await parent.click();
+ await expect(parent).toHaveAttribute("data-state", "checked");
+ for (const s of [
+ "maybe_missing",
+ "deleted",
+ "private",
+ "members_only",
+ "unlisted",
+ ]) {
+ await expect(page.getByTestId(`av-${s}`)).toHaveAttribute(
+ "data-state",
+ "checked",
+ );
+ }
+ });
+
+ test("the state filter persists across a reload", async ({ page }) => {
+ await searchReports(page);
+ await setFilter(page, "av-missing", false);
+ await expect(card(page, /Report deleted/)).toHaveCount(0);
+
+ await page.reload();
+ await waitForHydration(page);
+
+ await expect(page.getByTestId("av-missing")).toHaveAttribute(
+ "data-state",
+ "unchecked",
+ );
+ await expect(card(page, /Report available/)).toBeVisible();
+ await expect(card(page, /Report deleted/)).toHaveCount(0);
+ });
+});
+
+test.describe("export search — availability in shared links", () => {
+ test.use({ permissions: ["clipboard-read", "clipboard-write"] });
+
+ test.beforeEach(async ({ page, context }) => {
+ await context.grantPermissions(["clipboard-read", "clipboard-write"]);
+ await installRoutes(page);
+ await page.goto("/");
+ await waitForHydration(page);
+ });
+
+ test("a share link carries per-state tokens under fv=2 and round-trips", async ({
+ page,
+ }) => {
+ await searchReports(page);
+ await setFilter(page, "av-missing", false);
+
+ await page
+ .getByTestId("profiles-row")
+ .getByRole("button", { name: "Share current search" })
+ .click();
+ await expect(
+ page.getByRole("button", { name: "Link copied!" }),
+ ).toBeVisible();
+
+ const clip = await page.evaluate(() => navigator.clipboard.readText());
+ const url = new URL(clip);
+ expect(url.searchParams.get("fv")).toBe("2");
+ expect(url.searchParams.getAll("fav")).toEqual(["a"]);
+
+ await page.goto(`${url.pathname}${url.search}`);
+ await waitForHydration(page);
+ await expect(card(page, /Report available/)).toBeVisible();
+ await expect(card(page, /Report deleted/)).toHaveCount(0);
+ });
+
+ test("a legacy fv=1 link still yields the missing videos", async ({
+ page,
+ }) => {
+ // fav=a&fav=u&fav=d is what an old "everything" link looks like; under the
+ // new taxonomy its author meant all six states, so nothing may drop out.
+ await page.goto("/?fv=1&fav=a&fav=u&fav=d&fc=Test+Channel&ft=v&fa=a");
+ await waitForHydration(page);
+ await searchReports(page);
+
+ for (const v of VIDS) {
+ await expect(card(page, new RegExp(v.title))).toBeVisible();
+ }
+ });
+});
diff --git a/export/e2e/filter-profile-persistence.spec.ts b/export/e2e/filter-profile-persistence.spec.ts
@@ -55,7 +55,9 @@ test.describe("filter persistence — diverging commit clears active profile", (
await page.getByPlaceholder("Search transcripts...").press("Enter");
// The commit should clear activeProfileName and persist the new
- // divergent state into `working` (nd=true added).
+ // divergent state into `working`. Availability now persists as the `av`
+ // kept-set rather than the legacy `nd` exclusion boolean, so "Deleted
+ // unchecked" means deleted is absent from av.
await expect
.poll(
async () =>
@@ -65,13 +67,17 @@ test.describe("filter persistence — diverging commit clears active profile", (
const parsed = JSON.parse(raw);
return {
activeProfileName: parsed.activeProfileName,
- workingNd: parsed.working?.nd ?? false,
+ deletedKept: (parsed.working?.av ?? []).includes("deleted"),
workingNol: parsed.working?.nol ?? false,
};
}, STORAGE_KEY),
{ timeout: 5_000 },
)
- .toEqual({ activeProfileName: null, workingNd: true, workingNol: true });
+ .toEqual({
+ activeProfileName: null,
+ deletedKept: false,
+ workingNol: true,
+ });
// Reload. Without the fix, hydration would re-read profile p1 (Deleted
// still on) and silently revert the commit.
diff --git a/export/e2e/share-current-search.spec.ts b/export/e2e/share-current-search.spec.ts
@@ -51,7 +51,10 @@ test.describe("search page — share current search", () => {
).toBeVisible();
const clip = await page.evaluate(() => navigator.clipboard.readText());
const url = new URL(clip);
- expect(url.searchParams.get("fv")).toBe("1");
+ // v2: `fav` now carries one token per VideoState. The sentinel bump is
+ // what lets a reader tell a deliberate new selection from a v1 link whose
+ // author meant "everything that existed back then".
+ expect(url.searchParams.get("fv")).toBe("2");
// Composite query is encoded in `qt=`; legacy `q=` is omitted.
expect(url.searchParams.get("q")).toBeNull();
expect(url.searchParams.get("qt")).not.toBeNull();
@@ -63,8 +66,15 @@ test.describe("search page — share current search", () => {
expect(url.searchParams.getAll("ft").sort()).toEqual(["v"]);
// Audience: both selected by default.
expect(url.searchParams.getAll("fa").sort()).toEqual(["a", "r"]);
- // Availability: Deleted unchecked → Available + Unlisted remain.
- expect(url.searchParams.getAll("fav").sort()).toEqual(["a", "u"]);
+ // Availability: Deleted unchecked → every other state remains
+ // (a=available, m=unconfirmed, u=unlisted, p=private, o=members-only).
+ expect(url.searchParams.getAll("fav").sort()).toEqual([
+ "a",
+ "m",
+ "o",
+ "p",
+ "u",
+ ]);
// Legacy filter params must NOT leak into the new link.
expect(url.searchParams.get("ch")).toBeNull();
expect(url.searchParams.get("nol")).toBeNull();
diff --git a/export/next.config.ts b/export/next.config.ts
@@ -1,6 +1,14 @@
+import path from "node:path";
import type { NextConfig } from "next";
const nextConfig: NextConfig = {
+ // Pin the workspace root. Turbopack infers it by walking up for a lockfile
+ // and taking the outermost one, so an unrelated pnpm-lock.yaml anywhere above
+ // the checkout (a stray one in $HOME is enough) silently relocates the root —
+ // after which `@import "../../common/styles/tokens.css"` in app/globals.css
+ // resolves outside the project and `next dev` fails to boot at all. Nothing
+ // in this repo lives above the monorepo root, so pinning it costs nothing.
+ turbopack: { root: path.join(__dirname, "..") },
output: "export",
trailingSlash: true,
images: { unoptimized: true },
diff --git a/mcp/src/search.test.ts b/mcp/src/search.test.ts
@@ -33,6 +33,12 @@ import {
type SearchFilters,
} from "./search";
import { createServer } from "./server";
+import {
+ MISSING_STATES,
+ VIDEO_STATES,
+ type VideoState,
+} from "yt-dlp-transcript-common/lib/availability";
+import { parseShareV1 } from "yt-dlp-transcript-common/components/shareUrl";
// Every filter facet kept (the "no-op" filter) — override fields per test.
const KEEP_ALL: SearchFilters = {
@@ -40,11 +46,15 @@ const KEEP_ALL: SearchFilters = {
livestreams: true,
allAges: true,
restricted: true,
- available: true,
- unlisted: true,
- deleted: true,
+ states: new Set(VIDEO_STATES),
};
+// Availability kept-set helper: `keep("available", "deleted")` etc.
+const keep = (...states: VideoState[]): SearchFilters => ({
+ ...KEEP_ALL,
+ states: new Set(states),
+});
+
// ─── A tiny in-memory ShardSource for the tests ───
function cues(...pairs: [number, string][]): Cue[] {
@@ -116,11 +126,15 @@ const CHAT: Record<string, Cue[]> = {
"chan-b/b1": cues([12, "@fan: banana bread anyone"], [15, "@mod: stay on topic"]),
};
-// Availability overrides, keyed by video slug (defaults: available). a1 deleted,
-// a2 unlisted — the rest available.
+// Availability overrides, keyed by video slug (default: available). One video
+// per non-available VideoState so the `fav` filter is exercised across the
+// whole enum; b1 is the only video left available.
const AVAILABILITY: Record<string, VideoAvailability> = {
- "chan-a/a1": { isDeleted: true, isUnlisted: false },
- "chan-a/a2": { isDeleted: false, isUnlisted: true },
+ "chan-a/a1": { state: "deleted" },
+ "chan-a/a2": { state: "unlisted" },
+ "chan-a/a3": { state: "private" },
+ "chan-a/a4": { state: "members_only" },
+ "chan-a/a5": { state: "maybe_missing" },
};
// Channel B: one long transcript for windowing (a single match at 100s).
@@ -607,20 +621,59 @@ test("spec filter fa: keep only age-restricted → a4", async () => {
assert.deepEqual(ids, ["a4"]);
});
-test("spec filter fav: available-only drops deleted a1 + unlisted a2", async () => {
+test("spec filter fav: available-only drops every missing state", async () => {
+ const src = new StubSource();
+ const ids = await specIds(src, leafTree("coffee", "transcripts"), {
+ filters: keep("available"),
+ });
+ assert.deepEqual(ids, ["b1"]);
+});
+
+test("spec filter fav: missing-only drops the available b1", async () => {
+ const src = new StubSource();
+ const ids = await specIds(src, leafTree("coffee", "transcripts"), {
+ filters: keep(...MISSING_STATES),
+ });
+ assert.deepEqual(ids, ["a1", "a2", "a3", "a4", "a5"]);
+});
+
+// Each leaf on its own — the point of the enum is that these are five distinct
+// states, not one "gone" bucket.
+for (const [state, id] of [
+ ["deleted", "a1"],
+ ["unlisted", "a2"],
+ ["private", "a3"],
+ ["members_only", "a4"],
+ ["maybe_missing", "a5"],
+] as const) {
+ test(`spec filter fav: ${state}-only keeps just ${id}`, async () => {
+ const src = new StubSource();
+ const ids = await specIds(src, leafTree("coffee", "transcripts"), {
+ filters: keep(state),
+ });
+ assert.deepEqual(ids, [id]);
+ });
+}
+
+// A pre-existing share link (fv=1) knew three buckets. Its `a` token has to
+// keep unconfirmed videos too: they read as available when the link was
+// written, so dropping them would silently narrow somebody's saved search.
+test("spec filter fav: a legacy fv=1 'available' link keeps maybe_missing", async () => {
+ const sel = parseShareV1("?fv=1&fav=a", []);
const src = new StubSource();
const ids = await specIds(src, leafTree("coffee", "transcripts"), {
- filters: { ...KEEP_ALL, unlisted: false, deleted: false },
+ filters: { ...KEEP_ALL, states: sel.states },
});
- assert.deepEqual(ids, ["a3", "a4", "a5", "b1"]);
+ assert.deepEqual(ids, ["a5", "b1"]);
});
-test("spec filter fav: deleted+unlisted-only keeps a1 + a2", async () => {
+test("spec filter fav: a legacy fv=1 'deleted' link also keeps private/members", async () => {
+ const sel = parseShareV1("?fv=1&fav=d", []);
const src = new StubSource();
const ids = await specIds(src, leafTree("coffee", "transcripts"), {
- filters: { ...KEEP_ALL, available: false },
+ filters: { ...KEEP_ALL, states: sel.states },
});
- assert.deepEqual(ids, ["a1", "a2"]);
+ assert.deepEqual(ids, ["a1", "a3", "a4"]);
});
test("spec filter dates: fdf bound keeps only the later upload (a5)", async () => {
diff --git a/mcp/src/search.ts b/mcp/src/search.ts
@@ -26,6 +26,10 @@ import {
resolveChannelGroupId,
type ChannelGroup,
} from "yt-dlp-transcript-common/lib/channelGroups";
+import {
+ VIDEO_STATES,
+ type VideoState,
+} from "yt-dlp-transcript-common/lib/availability";
import type { ChannelRef, ShardSource, VideoAvailability } from "./source";
export type Snippet = { clock: string; seconds: number; text: string };
@@ -658,10 +662,9 @@ export type SearchFilters = {
// fa — all-ages / age-restricted kept.
allAges: boolean;
restricted: boolean;
- // fav — availability three-way (available / unlisted / deleted) kept.
- available: boolean;
- unlisted: boolean;
- deleted: boolean;
+ // fav — the VideoState values kept (see common/lib/availability). Absent
+ // from the set means filtered out.
+ states: ReadonlySet<VideoState>;
// fdf / fdt — inclusive upload-date bounds, "YYYYMMDD".
dateFrom?: string;
dateTo?: string;
@@ -911,14 +914,8 @@ function passesFilters(
if (rec.isLivestream ? !f.livestreams : !f.videos) return false;
// fa — audience
if (rec.ageRestricted ? !f.restricted : !f.allAges) return false;
- // fav — availability three-way
- if (avail?.isDeleted) {
- if (!f.deleted) return false;
- } else if (avail?.isUnlisted) {
- if (!f.unlisted) return false;
- } else if (!f.available) {
- return false;
- }
+ // fav — presence on the source platform
+ if (!f.states.has(avail?.state ?? "available")) return false;
// fdf / fdt — upload-date range (lexicographic on YYYYMMDD)
if (f.dateFrom && rec.uploadDate < f.dateFrom) return false;
if (f.dateTo && rec.uploadDate > f.dateTo) return false;
@@ -928,7 +925,7 @@ function passesFilters(
// True when the fav filter could exclude something (so availability must be
// fetched). If every availability bucket is kept, there's nothing to look up.
function needsAvailability(f: SearchFilters | null | undefined): boolean {
- return !!f && (!f.available || !f.unlisted || !f.deleted);
+ return !!f && !VIDEO_STATES.every((s) => f.states.has(s));
}
// A small lazy live-chat fetcher: per-channel subs manifest + page caches, so a
diff --git a/mcp/src/server.ts b/mcp/src/server.ts
@@ -513,7 +513,10 @@ const TOOLS = [
clear_filters: { type: "boolean", description: "Drop all filters." },
clear_availability: {
type: "boolean",
- description: "Remove the availability (deleted/unlisted) filter.",
+ description:
+ "Remove the availability filter, so videos in every state " +
+ "are kept (available plus the missing ones: unconfirmed, " +
+ "deleted, private, members-only, unlisted).",
},
clear_type: {
type: "boolean",
diff --git a/mcp/src/shareLink.test.ts b/mcp/src/shareLink.test.ts
@@ -1,5 +1,6 @@
import { test } from "node:test";
import assert from "node:assert/strict";
+import { VIDEO_STATES } from "yt-dlp-transcript-common/lib/availability";
import {
newGroup,
newLeaf,
@@ -67,10 +68,13 @@ test("decode: every v1 filter facet is decoded", () => {
// fa=a → all-ages kept, restricted dropped.
assert.equal(f.allAges, true);
assert.equal(f.restricted, false);
- // fav=a → available kept, unlisted/deleted dropped.
- assert.equal(f.available, true);
- assert.equal(f.unlisted, false);
- assert.equal(f.deleted, false);
+ // fav=a on a v1 link → available kept, and unconfirmed with it (those
+ // videos read as available when a v1 link was written); everything
+ // confirmed-gone dropped.
+ assert.deepEqual(
+ [...f.states].sort(),
+ ["available", "maybe_missing"],
+ );
// dates.
assert.equal(f.dateFrom, "20250101");
assert.equal(f.dateTo, "20251231");
@@ -122,9 +126,8 @@ test("override: clearAvailability resets the fav facet to keep-all", () => {
const d = applyLinkOverrides(decodeShareLink(fullLink()), {
clearAvailability: true,
});
- assert.equal(d.filters!.available, true);
- assert.equal(d.filters!.unlisted, true);
- assert.equal(d.filters!.deleted, true);
+ assert.equal(d.filters!.states.size, VIDEO_STATES.length);
+ for (const st of VIDEO_STATES) assert.ok(d.filters!.states.has(st));
assert.ok(d.warnings.some((w) => /availability filter removed/.test(w)));
});
diff --git a/mcp/src/shareLink.ts b/mcp/src/shareLink.ts
@@ -29,6 +29,10 @@ import {
hasShareV1,
parseShareV1,
} from "yt-dlp-transcript-common/components/shareUrl";
+import {
+ VIDEO_STATES,
+ VIDEO_STATE_LABELS,
+} from "yt-dlp-transcript-common/lib/availability";
import type { SearchFilters } from "./search";
export type DecodedLink = {
@@ -104,9 +108,7 @@ export function decodeShareLink(link: string): DecodedLink {
livestreams: sel.livestreams,
allAges: sel.allAges,
restricted: sel.restricted,
- available: sel.available,
- unlisted: sel.unlisted,
- deleted: sel.deleted,
+ states: sel.states,
...(sel.dateFrom ? { dateFrom: sel.dateFrom } : {}),
...(sel.dateTo ? { dateTo: sel.dateTo } : {}),
}
@@ -153,9 +155,7 @@ const KEEP_ALL_FILTERS: SearchFilters = {
livestreams: true,
allAges: true,
restricted: true,
- available: true,
- unlisted: true,
- deleted: true,
+ states: new Set(VIDEO_STATES),
};
// Apply overrides to a decoded link, returning a new DecodedLink. Records what
@@ -209,9 +209,7 @@ export function applyLinkOverrides(
} else if (next.filters) {
const f = next.filters;
if (overrides.clearAvailability) {
- f.available = true;
- f.unlisted = true;
- f.deleted = true;
+ f.states = new Set(VIDEO_STATES);
note("availability filter removed");
}
if (overrides.clearType) {
@@ -303,11 +301,12 @@ export function describeFilters(f: SearchFilters | null): string[] {
out.push("audience: none kept");
}
// Availability (fav).
- const av: string[] = [];
- if (f.available) av.push("available");
- if (f.unlisted) av.push("unlisted");
- if (f.deleted) av.push("deleted");
- if (av.length < 3) out.push(`availability: ${av.length ? av.join(" + ") : "none"} kept`);
+ const av = VIDEO_STATES.filter((s) => f.states.has(s));
+ if (av.length < VIDEO_STATES.length) {
+ out.push(
+ `availability: ${av.length ? av.map((s) => VIDEO_STATE_LABELS[s].toLowerCase()).join(" + ") : "none"} kept`,
+ );
+ }
// Dates (fdf/fdt).
if (f.dateFrom || f.dateTo) {
out.push(`uploaded: ${f.dateFrom ?? "…"} → ${f.dateTo ?? "…"}`);
diff --git a/mcp/src/source.ts b/mcp/src/source.ts
@@ -12,6 +12,10 @@ import type {
TranscriptDetail,
DisplaySummary,
} from "yt-dlp-transcript-common/lib/transcripts";
+import {
+ summaryState,
+ type VideoState,
+} from "yt-dlp-transcript-common/lib/availability";
import type { SubsDetail } from "yt-dlp-transcript-common/lib/subs";
import {
postsPageFileName,
@@ -62,10 +66,10 @@ const EMPTY_GROUPS: ChannelGroups = {
};
// Per-video availability, joined in from the summaries shards (a
-// TranscriptDetail record does not carry deleted/unlisted flags). Keyed by the
+// TranscriptDetail record does not carry presence state). Keyed by the
// member-local video slug (`<channelSlug>/<id>`) — the same slug a transcript
// page record carries — so the search engine can apply the `fav` filter.
-export type VideoAvailability = { isDeleted: boolean; isUnlisted: boolean };
+export type VideoAvailability = { state: VideoState };
// Read a site's global summaries shards (summaries/manifest.json +
// summaries/page-NNNN.json) via `readPage` and fold them into a slug →
@@ -94,10 +98,11 @@ async function buildAvailabilityMap(
if (!records) continue;
for (const r of records) {
if (typeof r.slug !== "string") continue;
- map.set(r.slug, {
- isDeleted: r.isDeleted === true,
- isUnlisted: r.isUnlisted === true,
- });
+ // summaryState falls back to the legacy isDeleted/isUnlisted booleans,
+ // which matters here more than anywhere: a hub reads summaries pages
+ // from member origins it does not control, so some of them will have
+ // been built before `state` existed.
+ map.set(r.slug, { state: summaryState(r) });
}
}
return map;
