import type { SiteSettings } from "../lib/settings"; import { digestCountOf, type ChannelBrief } from "../controller/channels"; import { digestWorkOf } from "../controller/channelSnapshot"; import { isGateHeld } from "../lib/pauseGates"; import { backfillLaneOperations, backfillLaneOperationEntriesOf, operationCostBasis, operationLabel, operationsGroupLabel, presentOperationWork, reachableOperationWork, } from "../lib/operations"; import { buildScheduleView } from "../jobs/syncScheduler"; import type { SchedulerState } from "../jobs/syncSchedulerState"; // Backs the monitor widget's last-sync and scheduler-status strips. Keeps the // payload tiny (a handful of scalars) rather than shipping the full // scheduler-status payload on every poll — the widget only needs freshness // markers and a one-line scheduler summary. Reuses the same helpers the // /scheduler status page does so the two never drift. export type WidgetSyncPayload = { // Epoch ms of the last manual full "Sync all" sweep. null = never. lastSyncAllAt: number | null; // Newest per-channel lastSyncedAt across all channels. null = none synced. lastIndividualSyncAt: number | null; scheduler: { enabled: boolean; lastRunAt: number | null; // last scheduled sweep (state.runs[0].at) nextRunAt: number | null; // soonest eligible channel's nextDueAt overdue: boolean; // some eligible channel is due now }; // Corpus-wide digest coverage. SCALARS ONLY, honouring this payload's stated // constraint — three numbers and two booleans, not a per-channel breakdown. // // It belongs on the poll rather than a page load because the number it // reports moves over WEEKS: an operator watching an 80-day backfill needs to // see it move at all, and "0.13% of 77,207" is not a figure any single // channel page can show. digest: { digested: number; // videos carrying a non-empty ai-digest.json videos: number; // every video dir in the corpus // Videos that CAN be digested: transcribed and not untranscribable, summed // from each channel's digest work list. The honest denominator — `videos` // counts video dirs, including the ~1,700 that have no transcript and never // will, so a coverage percentage against it can never reach 100%. // // NULL UNTIL EVERY CHANNEL CAN REPORT IT, deliberately. This is summed // across channels, and a channel with no `backfill.digest` entry contributes // videos to `digested` but nothing to this — so a partial sum would be a // denominator smaller than its own numerator, which is a worse lie than the // one it replaces. A null says "not yet knowable" and the band falls back to // `videos`. Every channel on the measured corpus reports it (68/68 on // 2026-08-26); the remaining case is a channel added since, whose first // snapshot has not been written yet. eligible: number | null; channelsWithAny: number; // channels the layer has reached at all // The DENOMINATOR for channelsWithAny. Without it "66 channels reached" is a // count with nothing to be a fraction of, and the card cannot tell a corpus // where the lane has touched everything from one where it has barely begun. channels: number; // Videos with no transcript yet, and videos held back for want of a current // normalized one. NEVER summed with `digested` nor with each other — they are // the two reasons a digest cannot happen, and each has a different fix (wait // for transcription; run Normalize transcripts). blocked: number; deferred: number; // held — computed by isGateHeld; nothing downstream inverts anything. held: boolean; // The lane is ARMED — `autoQueue.digest.enabled`, the switch its runner // resumes from at boot. It was `sweeping` (a corpus-wide sweep is armed) // until slice 1.3; the sweeps are gone and a lane's tree is its scope, so // this is the same question asked of the thing that now answers it. armed: boolean; }; // Corpus-wide backfill state, and free for the same reason as `digest`: every // brief already carries its channel's snapshot, so this is a sum rather than a // corpus walk (which cost 4.4 s on an endpoint the widget polls). // // `reachable` and `needsMedia` are separate FIELDS, not a total, because on the // measured corpus they are 835 and ~76,270. A single number here would report // a backfill as barely begun forever, no matter how much of the reachable work // was finished. // // THE "SCALARS ONLY" CONSTRAINT IS REVISED, NOT DROPPED. `kinds` is an array, // and it is bounded by the REGISTRY — three entries of five numbers today, off // exactly the sums beside it, not a row per channel. What the constraint // protects is that this endpoint is POLLED: a per-channel breakdown would grow // with the corpus (67 channels and climbing) and put a corpus walk back on a // 15-second timer. A per-KIND breakdown cannot, because adding a kind means // adding an entry to operations.ts. That distinction still holds, and it is // the line to keep: bounded by the code, never by the data. backfill: { reachable: number; // missing + stale: what the lane can do now needsMedia: number; // missing-input: needs an opt-in re-download first videos: number; // videos in the corpus (the denominator) // held — computed by isGateHeld; nothing downstream inverts anything. The // field on disk is `autoQueue.backfill.held`, in the plain polarity; the // inverted `backfill.enabled` it replaced is deleted (S0-pause). It was // `enabled` here until slice 7, and a pinned widget tab that predates the // rename reads `undefined ?? false` — not held — until it is reloaded. held: boolean; // The lane is ARMED — see `digest.armed` above. armed: boolean; // Whether any backfill FEATURE is on. Distinct from the gate: with no // feature on there is nothing to report at all, and that must not look like // "all caught up". anyKind: boolean; // The lane's name in the operator's terms, derived on the server from the // GROUP its enabled kinds declare — "Speakers" today. The client must not // resolve this itself: operations.ts reaches the filesystem. // // "Backfill" is a queue key. Nobody arms, pauses or runs "a backfill"; the // word survives only on the controls that genuinely act on the shared queue, // and even there the card now lists what is in it. groupLabel: string; // The same numbers, kept apart by kind — which is the only form of them that // means anything. Summed, this lane reads "77,952 reachable · 77,134 need // media": both correct, and together a figure in no unit, since 99.5% of the // first is attribution-text at ~1 model call per transcript CHUNK and all of // the second is diarization at 329 runs. The split also surfaces the largest // single fact about this corpus, which the sums hide completely — 77,463 // videos BLOCKED on diarization's output, which is why attribution can only // run text-only. // // Sorted by `reachable` descending, tie-broken by id, so the order is stable // across polls rather than following whatever order the snapshots were // written in. // // `eligible` and `present` are the COVERAGE half, and they are `number | // null` for the same reason `digest.eligible` above is: they are summed // across channels, a third of the snapshots on disk predate `eligible`, and // a partial sum is a denominator smaller than its own numerator. One // channel that cannot report voids the kind's whole figure — see // presentOperationWork, which returns null rather than 0 "because the honest // answer there is unknown". kinds: { id: string; label: string; // operationLabel(id) — resolved here, not in the client // What ONE UNIT of this operation costs, in words. The fact that hid // behind the lane name: attribution-text is armed on ~194,000 model calls // corpus-wide because its unit is the transcript CHUNK, has completed one // video, and read as a quiet row on every screen. Stated flat, with no // threshold — what is affordable is the operator's call. costBasis: string; reachable: number; needsMedia: number; blocked: number; deferred: number; // How many videos this operation has an opinion about. null = unknown. eligible: number | null; // How many it is DONE with. null = unknown. Never derived by subtracting // the work counts from `eligible` at a call site — that is exactly what // presentOperationWork does, once, with the right guard. present: number | null; }[]; }; }; export type WidgetSyncInputs = { settings: SiteSettings; now: number; state: SchedulerState; // Every number below is summed off the snapshot each brief already carries. // This used to be a full corpus walk — a 4.4 s cost on an endpoint the widget // polls — which is why the briefs arrive as an argument rather than being // read here: the caller already has them, memoized per request. briefs: readonly ChannelBrief[]; }; // Build the widget sync payload. Both the HTTP route and the dashboard cockpit's // SSR seed fold through this same function, so the first paint and the first // poll can never disagree. export function buildWidgetSyncPayload( inputs: WidgetSyncInputs, ): WidgetSyncPayload { const { settings, now, state } = inputs; const channels = inputs.briefs; let lastIndividualSyncAt: number | null = null; for (const c of channels) { const last = c.config.lastSyncedAt; if (!last) continue; const parsed = Date.parse(last); if (Number.isNaN(parsed)) continue; if (lastIndividualSyncAt === null || parsed > lastIndividualSyncAt) { lastIndividualSyncAt = parsed; } } const view = buildScheduleView({ channels: channels.map((c) => ({ slug: c.slug, config: c.config })), scheduler: settings.syncScheduler, state, now, priority: settings.channelPriority, }); const eligible = view.filter((v) => v.autoSyncEligible); let nextRunAt: number | null = null; let overdue = false; for (const v of eligible) { if (v.overdue) overdue = true; if (v.nextDueAt !== null && (nextRunAt === null || v.nextDueAt < nextRunAt)) { nextRunAt = v.nextDueAt; } } // Free: the snapshot each brief already carries has both halves. This used to // come off a full corpus walk — a 4.4 s cost on an endpoint the widget polls. // `digestEngines` postdates 11 of the live snapshots, so those read as 0 until // their next report refresh (see digestCountOf). let digested = 0; let videos = 0; let channelsWithAny = 0; // The eligible denominator, and whether it is complete. One channel that // cannot report voids the whole sum — see the payload type for why a partial // one would be worse than no answer at all. let digestEligible = 0; let digestEligibleKnown = true; // Summed off the same snapshots, in the same pass. Kept apart all the way // through — see the payload type. let backfillReachable = 0; let backfillNeedsMedia = 0; // The per-kind breakdown, accumulated in the SAME pass rather than a second // one. A Map keyed by kind id, so a snapshot naming a kind another snapshot // does not is simply added rather than dropped. const backfillByKind = new Map< string, { reachable: number; needsMedia: number; blocked: number; deferred: number; eligible: number | null; present: number | null; } >(); // Digest's two non-work counters, off the digestWorkOf() call already made // below — no extra read. let digestBlocked = 0; let digestDeferred = 0; for (const c of channels) { videos += c.snapshot?.totals.videos ?? 0; const n = digestCountOf(c.snapshot); digested += n; if (n > 0) channelsWithAny++; // Eligibility, minus what is waiting on a transcript: a blocked video is // eligible in principle and cannot be digested today, and including it would // make the coverage bar sag every time the downloader finds new videos — // which reads as digest progress going backwards. const work = digestWorkOf(c.snapshot); if (work.eligible == null) digestEligibleKnown = false; else digestEligible += Math.max(0, work.eligible - work.blocked); digestBlocked += work.blocked; digestDeferred += work.deferred; // The backfill LANE only. The snapshot's per-kind map now carries every // catalog operation, digest included, and the widget's backfill strip has // only ever meant diarization plus attribution — the digest coverage figure // it shows beside this one is computed separately, from digestCountOf above. for (const [id, entry] of backfillLaneOperationEntriesOf(c.snapshot?.backfill)) { backfillReachable += reachableOperationWork(entry); backfillNeedsMedia += entry.missingInput; const acc = backfillByKind.get(id) ?? { reachable: 0, needsMedia: 0, blocked: 0, deferred: 0, eligible: 0 as number | null, present: 0 as number | null, }; acc.reachable += reachableOperationWork(entry); acc.needsMedia += entry.missingInput; // `?? 0` at every read site: every snapshot written before these fields // existed lacks them, and undefined poisons the sum to NaN. acc.blocked += entry.blocked ?? 0; acc.deferred += entry.deferred ?? 0; // The SAME null-voids-the-sum discipline digestEligibleKnown uses above, // per kind rather than corpus-wide — a kind whose channels can all report // must not be dragged to "unknown" by a kind whose channels cannot. acc.eligible = acc.eligible == null || entry.eligible == null ? null : acc.eligible + entry.eligible; const done = presentOperationWork(entry); acc.present = acc.present == null || done == null ? null : acc.present + done; backfillByKind.set(id, acc); } } return { lastSyncAllAt: state.lastSyncAllAt, lastIndividualSyncAt, scheduler: { enabled: settings.syncScheduler.enabled, lastRunAt: state.runs[0]?.at ?? null, nextRunAt, overdue, }, digest: { digested, eligible: digestEligibleKnown ? digestEligible : null, videos, channelsWithAny, channels: channels.length, blocked: digestBlocked, deferred: digestDeferred, held: isGateHeld(settings, "digest"), armed: settings.autoQueue.digest.enabled, }, backfill: { reachable: backfillReachable, needsMedia: backfillNeedsMedia, videos, held: isGateHeld(settings, "backfill"), armed: settings.autoQueue.backfill.enabled, anyKind: backfillLaneOperations(settings).length > 0, groupLabel: operationsGroupLabel( backfillLaneOperations(settings).map((k) => k.id), ), kinds: [...backfillByKind] .map(([id, counts]) => ({ id, label: operationLabel(id), costBasis: operationCostBasis(id), ...counts, })) // Biggest reachable first, so the kind an operator can act on most leads. // Tie-broken by id so the order does not shuffle between polls. .sort((a, b) => b.reachable - a.reachable || a.id.localeCompare(b.id)), }, } satisfies WidgetSyncPayload; }