commit a2097c5df8fa99b153c9fcd4205f8c2fa276c166
parent c1fee8649d70172ccf29b1aa0a4f51bd5b677072
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Fri, 25 Sep 2026 20:37:30 -0400
plans: release 9 slice C2, as shipped — federated search (scope chips, per-archive state, progressive, partial failure, attribution, per-origin page gate); CHANGELOG
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Diffstat:
2 files changed, 107 insertions(+), 0 deletions(-)
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,6 +1,7 @@
# Changelog
## [Unreleased]
+- **The hub's search shows each archive's state, lets you choose which archives to search, and no longer waits for the slowest.** Under the line "Searching N archives …" is a row of chips, one per archive on the hub (official and added). Each says whether that archive is loading, how many videos it has in the search once it is in, or that it failed, with a Retry beside it. Pressing a chip takes that archive out of the search: nothing more is fetched from it and its results disappear. The choice is kept in this browser, and an archive it has never seen is searched. The search now runs as soon as one archive has loaded and runs again as each further one arrives, where it used to wait for all of them. When an archive does not answer, a line above the results says so ("4 of 5 archives answered. Hasanalyzer did not, so its videos are not in these results.") with a Retry, and the other archives' results show as usual; it used to contribute nothing, silently. Each result names its archive in text before the channel ("Jeralyzer · TheQuartering · 2026-09-25"). The hub now loads at most six summaries pages at a time from each archive, so a large archive does not hold up the small ones. `/ask` on the hub is unchanged: it searches every archive and waits for all of them. Published sites are unchanged. Needs a rebuild and deploy of the hub.
- **The hub and the homepage drop their subtitles, and both carry a Ko-fi link.** The headings are in Title Case on both pages: **Official Instances**, **Archives You Added** (hub) and **What It Does** (homepage). The line "The archives I run. Anyone can run their own." is gone from both, and the official-instance cards no longer show the site's description under its name; the four figures stay. The footer of the homepage and of the hub has a plain link reading "Ko-fi" to `https://ko-fi.com/archilyzer`. A published site's footer does not: an archive someone else hosts never carries it. Needs a rebuild and deploy of the hub and the homepage.
- **A job that waited in a queue no longer ends `failed` after doing its work.** A sync, download or other job queued behind another on the same platform ran its final page refresh outside any request, where Next refuses it, so the job read `failed` and `pnpm ops … --wait` exited 1 even though the work was done (the teamrcn sync on 2026-09-25). The refresh is now skipped there with one warning in the server log; the pages re-read disk on their next load anyway.
- **Downloads no longer sleep after a video the download filter declined.** The "sleep between downloads" (30 s by default) ran after every video, including each one the channel's download filter declined before fetching anything. A filtered channel's download-missing slept 193 times for 14 downloads on 2026-09-25. It still sleeps after every real fetch and after every failure, per-video ones included.
diff --git a/plans/release-9.md b/plans/release-9.md
@@ -463,6 +463,112 @@ editor unit, test:scripts and mcp not run: nothing they build or test changed be
- Needs a rebuild + deploy of the hub AND the homepage; this slice did not run build-hub, deploy-hub
or deploy homepage.
+### Slice C2, as shipped — federated search (2026-09-25)
+
+Branch `one-core/c2-federated-search` off `main` `6820bd21`; merged `main` (`8eb55add`, slice C1b) as
+`9d20d3fc`. **The operator-approved target (2026-09-25):** the hub searches every official instance
+by default, a visitor can take any archive out, each archive's state is visible, the search runs as
+soon as one archive is in, a failed member is named and retryable, a result names its archive in text,
+and pages are fetched per archive behind its manifest. Scoping: `$T/r9-hub-scope.md` §3.
+
+**The provider** (`common/components/SearchDataContext.tsx`, `MultiSiteDataProvider`). `FederatedSite`
+gains `enabled` (default true): false means none of that archive's feeds are fetched (`enabled: false`
+on its manifest, subs, posts and alias queries; no page descriptors) and nothing already cached from it
+is merged — TanStack returns cached `data` for a disabled query, so every merge checks scope itself.
+Per archive, its state is derived from its manifest query and its page queries: `ready` (manifest in
+and every page in), `failed` (manifest or a page errored and nothing of it is fetching), `loading`
+(otherwise, including while a Retry is in flight), `off`. `SearchDataValue` gains two OPTIONAL fields,
+set only by the multi-site provider: `federation: {sites: FederatedSiteState[], retry(origin)}`
+(`{origin, siteTitle, accent?, status, count?, error?}`, `count` = the manifest's `totalCount`) and
+`siteTitleOf(origin)`. `retry` is `refetchQueries` over every query whose key's second element is that
+origin and whose status is `error`. `summariesState.error` is no longer always null: it is set when
+EVERY in-scope archive failed (read by `/ask`'s corpus-error line). `SingleSiteDataProvider` is not
+touched.
+- **Merge.** An archive's records join the merged list only when it is `ready`, so the list grows one
+ whole archive at a time; the merged list's identity is keyed on the ready set alone, so the search
+ session (which re-runs its pipeline on every new `summaries` identity) re-runs once per archive, not
+ per page or per manifest. A failed archive contributes nothing (before: whatever pages had loaded).
+ The synthetic merged manifest (HubStats' transcript figure) covers the ready archives. Channel groups
+ and channels cover the in-scope archives whose manifest has arrived.
+- **Progressive.** A new `progressive` prop: `summariesReady` turns true at the first ready in-scope
+ archive. Without it (the `/ask` hub, `AskHub.tsx`, unchanged), it waits until every in-scope archive
+ has settled, ready or failed. HubHome passes it. The release-8 slice E rule holds unchanged: a
+ restored query is `runHeld` in `SearchSessionContext`, independent of readiness, so it still waits
+ for the visitor; a `qt=` link runs over the first archive and re-runs as each further one arrives.
+- **Speed.** Pages are requested per archive only once its manifest has settled and only while it is in
+ scope (as before for the manifest gate; new for scope), and each page fetch goes through a per-origin
+ gate of **6 in flight** (`PAGE_FETCHES_PER_SITE`), so a big member cannot take the browser's
+ connections from the small ones. No new build-time data.
+
+**The page** (`export/app/components/hub/`). `useHubScope.ts` keeps the set of origins switched OFF in
+`localStorage["ytdlp-tb:hub-scope"]` (`{off: [...]}`, every access in try/catch), so an archive the
+browser has not seen — a new official instance, an added one — is searched. `HubScope.tsx` is the row
+of chips under the live line, above the query builder: one per archive on the page (official and
+added), `data-testid="hub-scope-chip-<origin>"` + `data-status`, a toggle button (`aria-pressed`, the
+archive's title and `loading…` / its record count / `failed` / `off`) and, when failed, a `Retry
+<title>` button. `HubStats` counts the archives in scope. `common/components/SearchResults.tsx`: one line
+above the results when an in-scope archive failed, `data-testid="hub-scope-status"`: "N of M archives
+answered. X did not, so its videos are not in these results. Retry" — nothing while every archive is
+fine or still loading (the chips show loading). A result card's meta line reads
+`<archive> · <channel> · <date>` (`data-testid="result-source"`, from `ResultGroup.source`, set via
+`siteTitleOf` in `SearchSessionContext`); single-site groups carry no `source` key. The accent stripe
+and the `data-result-slug` contract are unchanged.
+
+**Specs.** New `export/e2e-hub/federated-search.spec.ts` (+4), two official members route-mocked with
+CORS: (1) both searched by default, both chips `ready`, each card names its archive, no status line;
+(2) toggling Origin B off removes its card and after a reload it stays off with **zero requests** to its
+origin, and back on it is fetched and returns; (3) Origin B's pages aborted → chip `failed` with `Retry
+Origin B`, "1 of 2 archives answered." / "Origin B did not", Origin A's card renders; fixed + the line's
+Retry → chip `ready`, line gone, both cards; (4) Origin B's page held → Origin A's search result renders
+while B is `loading`, and the same search covers B once it is released. `federation.spec.ts`: "Archives
+You Added" exact (C1b's Title Case). Every existing hub test id / accessible name is unchanged.
+
+| sha | what |
+|---|---|
+| `af8b30ed` | provider: scope (`enabled`), per-archive state + Retry (`federation`), ready-only merge, `progressive`, per-origin page gate of 6, `error` when all failed, `siteTitleOf` |
+| `f7f7482f` | hub: `useHubScope`, `HubScope` chips, HubHome wiring (`progressive`), HubStats counts the archives in scope |
+| `d0225a20` | a result names its archive (`ResultGroup.source`); the "N of M archives answered." line with Retry |
+| `0b891b8f` | `e2e-hub/federated-search.spec.ts` (+4) |
+| `9d20d3fc` | merge `main` (`8eb55add`, C1b) — no conflicts |
+| `6a69c034` | `federation.spec.ts`: "Archives You Added", exact |
+| `a0bd7b02` | a chip's title says what its number counts and what pressing it does |
+| `40818bbb` | the merged list's identity keys on the ready set alone (no re-run when another manifest lands) |
+
+**Gates.** tsc clean (per commit; full workspace on `9d20d3fc`+ and on `40818bbb`). Common tests
+**1,845/1,845** (no new unit tests: the provider is a hook, covered by the e2e). test:scripts **162 passed
++ 1 skipped**. `pnpm --filter export exec next build` (site) ok, 29 s; `compose:hub` (worktree, no
+corpus: "0 built-in pool site(s) …; hub-summary.json skipped") + `INSTANCE_MODE=hub … next build` ok,
+32 s. e2e, queued, all on the merged tree: **`e2e:hub` 15 passed** (pre-merge, 48.9 s), **16 passed**
+(merged, 53.2 s), **16 passed** (`40818bbb`, 47.0 s); **export full suite 195 passed**, 8.5 min (single
+site unchanged); **`e2e:2origin`** (`TWO_ORIGIN_REBUILD=1`) **3 passed**, 46.9 s on `a0bd7b02`, and **3 passed**,
+1.0 min on `40818bbb`. The seven compose-hub outputs + `sw.js` in the worktree's `export/public` were
+swapped for copies before every build/e2e and relinked after; the primary's copies were not written by
+this slice (one change seen, `hub-summary.json` at 20:18:20, is the parent's build-hub, which wrote the
+other six at 20:17:53). Editor build/unit and mcp not run: nothing they build changed. **Numbers tools:
+none.**
+
+**Seen against the live members** (the built hub served locally with the real five-site
+`hub-sites.json`, service workers blocked so routes apply): first result card at 1.0–2.2 s; chips ready
+Hasanalyzer/Rekietalyzer ~1.6–2.0 s, Bonnellyzer ~2–2.9 s, Anilyzer ~2–4.5 s, Jeralyzer ~3.1–5.1 s.
+With Hasanalyzer's pages aborted: "4 of 5 archives answered. Hasanalyzer did not, …", its chip failed
+with Retry, 72,360 records from the other four listed. Checked at 1280 dark and 390 dark.
+
+**Found and left:**
+- The chip's number is the member's summaries `totalCount` — its record (video) count, the number the
+ results header counts ("All videos (N)") — not the card's "transcripts" figure, which is the
+ homepage's build-time `transcribed.total`. The chip's title says "N videos from X are in the search";
+ the live line still says "transcripts" (C1's wording, left).
+- A member whose manifest loaded but whose pages failed keeps its channel group in the filter panel
+ (the channels are known); its records are not searched. Hiding the group on failure would churn the
+ groups on every Retry.
+- The `/ask` hub has no scope chips and searches every archive on the page, as before; the scope is
+ the front page's. It waits for every archive to settle (not progressive), so a chat never grounds in
+ a half-loaded federation silently.
+- Official cards have no `accent` (C1: they wear `seriesColor` on the page), so result stripes and chip
+ dots appear only for an archive that sets one — the text attribution is what names the source.
+- `route()` does not see requests a service worker makes; a manual check of the built hub needs
+ `serviceWorkers: "block"`. The e2e-hub suite runs `next dev`, where no SW is registered.
+
## Rollout
## Rollout 2026-09-25 (evening) — `0213f6c8` live on :3001 (third restart of the day)
