# Plan — universal per-video tags ("all streams where Eva collabs", on any site, any channel)

Approved by the operator 2026-09-21. Implemented as three slices (S1 core, S2 site, S3 editor) after
a frozen S1.1 lands on main.

## Context

The operator wants to open Anilyzer and narrow any video list to "streams where Eva collabs", including
streams on channels that are not hers (Legal Mindset, Destiny, Nux Taku…), with a mechanism that is
generic: any person/topic, any site, applied by the operator, by an agent, or by umtool. Today the only
cross-channel curation on a site is channel **groups** (`sites/<id>/site.json` `groups`; Anilyzer's
"Elfpire Eva" group holds only her own two channels) — whole channels, nothing per video. The corpus has
three unrelated things called "tags" already (yt-dlp keyword `TranscriptSummary.tags`, digest topic tags,
the `"tags"` search scope), which is the naming hazard this plan works around. Research on 2026-09-20
produced the first concrete tag set: 7 metadata-flagged LM collabs, ~19 LM streams discussing her, ~64
with her in chat.

## Operator decisions (2026-09-21)

1. **Assignment — all three**: by hand (video toggle + bulk "Tag selected"), by **rules** on a tag
   (re-evaluated at index build; operator pins/suppresses exceptions), by **import from a umtool project**.
2. **One tag per kind**: `eva-collab` (on mic), `eva-in-chat`, `eva-topic`; filter is OR across chips.
3. **Vocabulary both corpus-wide and per site, like aliases** — corpus file authoritative for
   definitions AND assignments; the site file is presentation ONLY (relabel/hide/order/colour). **Amended
   2026-09-21 during S1**: a site-layer rule can never tag a record — a record is shared by every site carrying
   its channel — so rules bind only from the corpus file and the site tab offers no rule editor.
4. **Surfaces — all four**: export lists (channel + all-videos), export search, MCP filter, editor lists.
5. **Universal + flexible, three writers**: free-form tag ids for any subject, optional `group` so chips
   read "Eva: Collab · In chat · Discussed"; operator UI, `pnpm ops` (agents; MCP stays read-only per
   AGENTS.md), umtool — all through ONE server action, every assignment carrying provenance.
6. **The export lists selectable tags dynamically** (operator, 2026-09-21): a site's chip row is built
   from its published `/tags.json`, and compose writes only tags with a non-zero count ON THAT SITE — a
   global tag with no matches on a site does not appear there, nor does a hidden one. No static list anywhere.

## Naming

| Surface | Name |
|---|---|
| UI word, files, CLI, wire | **tags**: `transcripts/tags.json`, `sites/<id>/tags.json`, published `/tags.json`, `pnpm ops tags`, `pnpm ops tag-videos` |
| Record field on shards | **`curatedTags?: string[]`** (never `tags` — `common/lib/transcripts.ts:21` is yt-dlp keywords) |
| Modules / types | `common/lib/curatedTags.ts`, `common/lib/curatedTagsStore.ts`, `common/controller/curatedTagsIndex.ts`; `CuratedTagDef`, `CuratedTagRule`, `CuratedTagAssignment`, `CuratedTagsConfig`, `PublishedTags` |
| Existing keyword scope | UI label "Tags" → **"Keywords"** at `common/components/QueryLeafView.tsx:46` (`SCOPE_LABELS`) and `:156-159` (placeholder); code token `"tags"`, URL and MCP `scopes` enum unchanged |

Tag ids: `^[a-z0-9][a-z0-9._-]*$`. Nothing in code names Eva; she is seed data (§ Seed).

## Data model

**`transcripts/tags.json`** (authoritative; `writeJsonAtomic`; `sanitizeTagsConfig` on read):
```json
{ "version": 1,
  "tags": [ { "id": "eva-collab", "label": "Collab", "group": "eva", "groupLabel": "Eva",
              "description": "Eva is on mic", "color": "#b48ead", "order": 1, "hidden": false,
              "rules": [ { "id": "meta", "kind": "metadata", "pattern": "elfpire|elf ?pire ?eva",
                           "channels": [], "dateFrom": null, "dateTo": null, "enabled": true } ] } ],
  "assignments": {
    "legal-mindset/XZqL6k9IHGA": {
      "manual": ["eva-collab"], "suppressed": ["eva-topic"],
      "sources": { "eva-collab": { "source": "umtool:elfpire-eva", "setAt": "2026-09-21T18:04:11Z" },
                   "eva-topic":  { "source": "operator",           "setAt": "2026-09-21T18:05:02Z" } } } } }
```
- `source` ∈ `operator | agent:<label> | umtool:<project> | rule:<ruleId>`; recorded for pins AND suppressions.
- `rules` optional (pure-manual tags are legal); `kind` ∈ `metadata | chat-author | caption`; a rule whose
  regex fails to compile is disabled with a reason, never thrown. If a tag is in both `manual` and
  `suppressed` for one key, `manual` wins.
- **Rule hits are never persisted** — only pins/suppressions. Edit a rule → rebuild index → done.

**`sites/<id>/tags.json`**: same shape. `mergeTagDefs(global, site)`: for an existing id the site entry is a
field-wise overlay of `label/groupLabel/color/order/hidden`; it can never delete a
global rule or assignment (an assignment is a fact about the video). A new id is a full site-only tag.
(Deliberately NOT `mergeAliases`' wholesale replacement.)

> **Corrected 2026-09-22 (`dcd065e7`).** The sentence above used to say the site entry "may append
> rules". It may not, and the code now enforces what decision 3-as-amended already said: a site layer
> carries NO rules at all. `coerceTagDef(raw, layer)` (`common/lib/curatedTags.ts:214`) returns
> `rules: []` whenever `layer === "site"` (`:235-238`), and both `readSiteTags` and `writeSiteTags`
> (`common/lib/curatedTagsStore.ts:94-113`) sanitize with `{layer: "site"}` — so a hand-written site
> rule is **dropped on read AND on write**, not merely ignored at merge time. `mergeTagDefs`
> (`common/lib/curatedTags.ts:371`) consequently has no rule branch for it to reach. The reason is
> the one in decision 3: a record is shared by every site carrying its channel, so a site-layer rule
> could only tag records the other sites also publish.

**Pure functions** (`common/lib/curatedTags.ts`): `effectiveTagsFor(ruleHits, assignment) = (hits ∪ manual)
− suppressed`, sorted by def `order` then id; `evaluateTagRules(input, rules)` with
`input = {channelSlug, id, title, description, tags, uploadDate, captionCues?, chatCues?}`:
`metadata` → one regex over `title\ndescription\ntags.join(" ")`; `chat-author` → the author is the
`<author>: ` prefix of each chat cue (`common/lib/liveChat.ts:11,59-60,100` — no `author` field on `Cue`;
the prefix is the contract); `caption` → regex over caption cue text, first hit short-circuits. Channel
scope and date range are applied before compiling; each kind compiles once per build.

## Derivation pipeline

- `curatedTags?: string[]` on `TranscriptSummary` (`common/lib/transcripts.ts:10-29`) and `DisplaySummary`
  (`:31-55`, optional like `state?` at `:52`) and `toDisplaySummary` (`common/lib/transcripts-server.ts:195`).
  **Omitted when empty** so untagged corpora's pages stay byte-identical (`createPageWriter` sha1 skip,
  `buildIndex.ts:869`).
- Computed in `common/controller/buildIndex.ts` in the per-video worker just before `sums.put(indexKey,
  summary)` (`:697`) — caption cues and the `live_chat` track are already in hand there (`:705`, `:720`).
- **Invalidation** (`common/controller/curatedTagsIndex.ts`): store `curatedRulesHash` (sha1 of enabled
  rules) and `curatedAssignHash` (sha1 of assignments) in the existing `meta` sub-DB (no new sub-DB, so no
  new `clearAsync()` at `:528-548`). `reapplyCuratedTags` runs after the mtime diff (`:561-587`): assignments
  changed → re-derive only the symmetric difference of keys (no cue reads); rules changed → re-derive every
  video from the `cues`/`subs` sub-DBs (no disk re-read). Returns the changed `indexKey`s, which are unioned
  into the dirty-channel set so transcript/subs/summaries pages rewrite — `common/lib/channelSignature.ts`
  hashes only mtimes (FACTS ~:203-207), so this explicit dirtying is what keeps compose from skipping them.
  `SCHEMA_VERSION` stays **13** (amended during S1: an absent `curatedRulesHash` can never match, so the first
  build re-derives every record from LMDB in ~80 s; a bump would re-read 76k video dirs for the same state). Build log prints `curated tags: rules <hash8> (changed), re-derived N`.
- **Counts**: while streaming `sums` into per-site summaries pages (`:1712`) accumulate per-tag totals and
  per-channel breakdown → `tag-counts.json` in `siteIndexDir(paths, siteId)` (`common/lib/site.ts:138`).
- **Compose** (`common/bin/compose-site.ts:744-753`, beside aliases): `effectiveSiteTags(paths, siteId)` +
  counts → `/tags.json` `{version, tags:[{id,label,group,groupLabel,color,order,count,channels:{slug:n}}]}`;
  hidden and zero-count tags dropped; nothing left → file not written (404 is legitimate, like `duplicates.json`).
- **Contract**: `CONTRACT.corpusSpec` 3 → **4** and `"tags.json"` in `ROOT_FILES` (`common/lib/archive/
  contract.ts:31-51, :121`) — per the rule at `common/lib/corpus.ts:25-34` a bump announces a new fetchable
  document; the per-record `curatedTags` key is additive and bumps no manifest version. `buildSiteCorpus`
  (`corpus.ts:204`) gains `tags: {url, videoField: "curatedTags"}`; `llms.txt` (`:320`) gains one line.

## Slices (three Opus implementers; S1.1 lands first and alone, then S1 ∥ S2 ∥ S3)

### S1 — common store, rules, index, export data (branch `tags/core`)
1. **S1.1 FREEZE**: `common/lib/curatedTags.ts` (types, `sanitizeTagsConfig`, `mergeTagDefs`,
   `effectiveTagsFor`, `evaluateTagRules`, `PublishedTags`, `TAGS_FILENAME`) + tests; the two record fields;
   `export/e2e/fixtures/data.ts` emits `curatedTags` on two summaries and a `/tags.json` fixture. Merge to
   main immediately — S2 and S3 branch from it.
2. `common/lib/curatedTagsStore.ts` (mirror `common/lib/aliasesStore.ts`): `readGlobalTags/writeGlobalTags/
   readSiteTags/writeSiteTags/effectiveSiteTags` + **`applyTagAssignments(paths, {op: add|remove|suppress|unsuppress, tag,
   videos, source, at})`**; `paths.globalTagsFile` (`common/lib/paths.ts:105,219`), `siteTagsFile`
   (`common/lib/site.ts:132`); tests: sanitize, layering, both-lists conflict, provenance.
3. `curatedTagsIndex.ts` + `buildIndex.ts` hook at `:697`, meta hashes, `reapplyCuratedTags`,
   NO schema bump (see Derivation), per-site counts at `:1712`; `toDisplaySummary`. Measure rule cost on legal-mindset
   (478 chat streams) and record it in FACTS.
4. Contract + compose: `contract.ts` spec 4 + `ROOT_FILES`; `corpus.ts:204,:320`; `compose-site.ts:744-753`
   writes `/tags.json`; `corpus.test.ts`.
5. `plans/FACTS.md` (the three meanings of "tags" in the naming-hazards table; chat-author prefix contract;
   the two meta hashes; SCHEMA_VERSION unchanged) + `AGENTS.md` (tags.json is curated data; never hand-edit).

### S2 — export UI + MCP (branch `tags/site`, needs only S1.1)
6. `common/lib/search/evalTree.ts`: `curatedTags?` on `SearchFilters` (`:311`) and `FilterableRecord`
   (`:328`), OR test in `passesFilters` (`:335`), `filterIsSelective` (`:365`) true for a non-empty set; tests.
7. `common/components/tagsCache.ts` (mirror `aliasesCache.ts`) + `SearchDataContext.tsx`; `urlState.ts`
   param `tg` (the `tk` convention: `p.getAll("tk")` `:94`, delete/append `:185-186`).
8. `SearchSessionContext.tsx`: `draftTags/committedTags` (`:389` pattern), `passesFilter` (`:703`), filter
   key, snapshot/commit (`:986, :1013`), `curatedTags` on `ResultGroup` (`:141-158`).
9. UI: grouped chip row in `common/components/FiltersPanel.tsx` beside the channel-group chips (`:225-250`),
   rendered only when `/tags.json` has a tag with `count > 0`, chips show counts, OR semantics; tag chips on
   the video card (`SearchResults.tsx:607`); "Keywords" relabel (`QueryLeafView.tsx:46,156-159`).
10. MCP: `tags: string[]` in `SEARCH_FILTER_ARGS` (`mcp/src/server.ts:196`), parsed in `parseSearchArgs`
    (`:1382`, filters literal ~`:1435`) for `search_transcripts` (`:309`) and `enumerate_matches` (`:413`);
    `curatedTags` on `IndexedVideo` (`common/lib/archive/reader.ts:100`) + `buildVideoIndex` (`:242`) so the
    scan-plan pruner (`mcp/src/search.ts:383`) prunes by tag; `loadTags()` on `ArchiveReader` (sibling of
    `readAliasConfig` `:761`); new `list_tags` tool after `list_channels` (`:285`); a defs line in
    `resolve_source` (`:682`); footer warning when a source ships no `/tags.json` but a tag filter was given
    (pre-spec-4 sites: `loadTags()` → `[]`, pruner treats "index never heard of it" as "read the page",
    the invariant at `search.ts:319`).
11. Tests: `export/e2e/tag-chips.spec.ts` (model: `export/e2e/channel-group-chips.spec.ts`), URL round-trip,
    `mcp/src/search.test.ts` filter case.

### S3 — editor UI + ops + umtool (branch `tags/editor`, needs S1.1; stubs `applyTagAssignments` until S1.2)
12. `editor/app/tags/page.tsx` + `components/EditorTagsClient.tsx` + `actions.ts` (`saveTagDefsAction`,
    `previewTagRuleAction` = dry-run `evaluateTagRules` over LMDB `sums/cues/subs`, listing title/date/channel
    with Pin all / Pin / Unpin, `applyTagAssignmentsAction`) — pattern `editor/app/sites/[siteId]/aliases/`.
    Nav entry (twelve → thirteen; `editor/scripts/measure-nav.mjs`).
13. Per-site overlay tab `editor/app/sites/[siteId]/tags/page.tsx` (presentation fields only — no rule editor,
    see decision 3 as amended).
14. Video page `videos/[id]/components/TagsPanel.tsx` beside `OperationPanel` (`page.tsx:251`);
    `toggleVideoTagAction` in `videoActions.ts` (shows each tag's provenance).
15. `VideoListPane.tsx`: `tag`/`untag` in `BULK_ACTION_OPTIONS` (`:38-49`) with a tag picker via
    `doSummaryBulk` (`:254`); `bulkApplyTagAction` in `bulkVideoActions.ts` = ONE `applyTagAssignments` call
    (single file write, not a per-id loop); a Tags chip filter on the list.
16. Ops: `editor/app/api/ops/tag-videos/route.ts` (`POST {tag, op, videos:[{slug,id}], source?}`) and
    `editor/app/api/ops/tags/route.ts` (`GET` defs+counts, `?tag=` → assignments with provenance; `POST`
    define/remove defs and rules) — adapters per `_lib.ts:7-27`; register `tag-videos`, `tags` in
    `scripts/archilyzer-ops.mjs:51-65`; add `--file <path>` for large bodies (agents write big id lists via
    script, not model output) and a GET entry beside `channel` (`:45-49`). Default `source` for the CLI:
    `agent:<$ARCHILYZER_AGENT or "cli">`.
17. umtool: `umtool/app/api/report/tag/route.ts` + project-page action "Tag cited videos as …" (whole
    project or per section) → `${ARCHILYZER_EDITOR_URL}/api/ops/tag-videos` with `WORKER_TOKEN`
    (`umtool/report-to-video/fetch-via-editor.mjs:24,70-80`), `source: "umtool:<projectId>"`, honouring
    per-clip `siteChannel`/`siteVideo` (Rumble's two ids, `cues.mjs:206-207`).
18. `editor/e2e/tags.spec.ts` (defs CRUD, rule preview, pin/unpin, bulk tag, list filter) + cases in
    `editor/e2e/ops-api.spec.ts`; umtool spec for the project action against a fixture editor stub
    (`EDITOR_STUB_PORT` pattern from the fetch-window work).

**Three writers, one action**: editor UI, ops route and umtool all reach `applyTagAssignmentsAction` →
`applyTagAssignments` in `curatedTagsStore.ts`. The MCP gains no write tool.

**Graph**: S1.1 → { S1.2 → S1.3 → S1.4 → S1.5 } ‖ { S2.1 → S2.2 → S2.3 → S2.4, S2.5, S2.6 } ‖
{ S3.1 → S3.2, S3.3, S3.4 → S3.5 → S3.6 → S3.7 }. Review: Opus for S1 (index + contract) and S3.5/S3.6
(writers), Sonnet for S2. Merge order S1 → S2 → S3 (S3 re-merges main once S1.2 lands).

## Risks
- Invalidation: a skipped `reapplyCuratedTags` makes a rule edit silently inert until an unrelated mtime
  moves — the two meta hashes are checked unconditionally, and tag-only changes explicitly dirty channels.
- Name collision (three "tags"): `curatedTags` is the only field name; the UI says "Keywords" for yt-dlp tags.
- Shard growth: omitted-when-empty keeps Jeralyzer (30k videos) byte-identical until tagged; fully tagged
  ≈ 25 B/record ≈ 750 KB across its pages.
- Chat regex cost: author-prefix only, compiled once, channel/date scoped first, short-circuit; measured in S1.3.
- Remote sources built before spec 4: `/tags.json` 404 → empty with an explicit warning, never a silent miss.
- Site-layer overreach: the field-wise overlay cannot delete global rules or assignments (tested in S1.2).

## Seed (applied through the new surfaces — never by hand-writing `transcripts/`)
```sh
pnpm ops tags --json '{"op":"define","tag":{"id":"eva-collab","label":"Collab","group":"eva","groupLabel":"Eva","rules":[{"id":"meta","kind":"metadata","pattern":"elfpire|elf ?pire ?eva","enabled":true}]}}'
pnpm ops tags --json '{"op":"define","tag":{"id":"eva-in-chat","label":"In chat","group":"eva","groupLabel":"Eva","rules":[{"id":"chat","kind":"chat-author","pattern":"elfpire","enabled":true}]}}'
pnpm ops tags --json '{"op":"define","tag":{"id":"eva-topic","label":"Discussed","group":"eva","groupLabel":"Eva","rules":[{"id":"cap","kind":"caption","pattern":"\\belf ?pire\\b|elfpyre|legal loli","enabled":true}]}}'
```
Expected from the 2026-09-20 research: 7 metadata hits on `eva-collab`, ~64 streams on `eva-in-chat`, ~19 on
`eva-topic` (Legal Mindset alone). Review in `/tags` → Preview, pin survivors, suppress false positives;
then the umtool project page "Tag cited videos as eva-collab"; then `pnpm ops build-index --wait` and
`pnpm ops build-deploy --json '{"siteId":"anilyzer"}' --wait`.

> **Corrected 2026-09-22 (`52f5c64c`).** The note here used to warn that build-site takes `siteIds`
> and build-deploy takes `siteId`. That asymmetry is gone: **both routes take either key**, through
> `reqSiteIds(body)` (`editor/app/api/ops/_lib.ts:179-199`), which refuses both-at-once and
> neither-at-all, validates every id, and throws `OpsInputError` — a **400 before any job is
> enqueued** — so a typo'd site id can no longer start a partial fan-out. `build-deploy` fans out and
> returns `{ok, jobs, skipped}` plus `jobId` when there is exactly one job.

## Verification
- Unit: `pnpm --filter yt-dlp-transcript-common run test` (curatedTags, store, evalTree, corpus); tsc for
  common/editor/umtool/mcp; `pnpm run test:scripts`; `mcp` tests.
- Index/export on the fixture and then the real corpus: `pnpm ops build-index --wait`;
  `pnpm ops build-site --json '{"siteIds":["anilyzer"]}' --wait` (the hand-poll workaround this note used
  to prescribe is gone as of `d653b504`: `--wait` POLLS `/api/jobs/<id>/log`, a poll failure is not a job
  failure, and `followJob` falls back to `/api/jobs/active` to decide whether the job is still there —
  see the `--wait` entry in [`FACTS.md`](FACTS.md#--wait-polls-a-log-route-and-a-dropped-poll-is-not-a-failure));
  the site composes into `export/public` and exports to `export/out`, so `jq .tags export/out/tags.json`; find a legal-mindset id's page via `slugToPage` and assert `.curatedTags` on the record;
  `curl -H "authorization: Bearer $WORKER_TOKEN" localhost:3001/api/ops/tags | jq '.tags[].count'`.
- Browser: `?tg=eva-collab` narrows the all-videos list and a search; chip count == `/tags.json` count;
  clearing the chip drops the param. e2e: `export/e2e/tag-chips.spec.ts`, `editor/e2e/tags.spec.ts`,
  ops-api cases, umtool tag spec; full editor suite on the final main.
- MCP: `list_tags {source:"remote:https://anilyzer.pages.dev"}` after deploy; `search_transcripts {…,
  tags:["eva-collab"]}` returns only tagged videos; the same against a pre-spec-4 site (Jeralyzer until
  rebuilt) returns the empty-with-warning path.

## Rollout log (2026-09-21, local only, no deploy)

Seeded through `pnpm ops tags --file`, index rebuilt (first derivation: examined 77,224, re-derived 831,
623 s; 103 transcript pages rewritten, 906 unchanged), Anilyzer composed locally. The seed patterns were
then corrected from what the corpus actually contains:

- `eva-collab` metadata: `@elfpire|#elfpire|elf ?pire ?eva|^[^\n]*elf ?pire` — Legal Mindset's 7 hits are
  all `@ElfpireEva` guest credits; MommaOcco's 200 were a friends-list boilerplate (`[EvaElfpire]` + URL),
  which the bare `elfpire` matched. `^[^\n]*` = "in the title" (the haystack's first line; no `m` flag).
- `eva-in-chat` chat-author: `^@?elfpire ?eva$` — the bare `elfpire` also matched `@PapaElfpire`.
- `eva-topic` caption: `\b(alpire|elpire|elfire|alpier|elpyre|elfpire|elfpyre)\b|legal[ -]?lol+i|\blawli\b` —
  ASR never spells "Elfpire"; without `\b` it matched "Jason Angelfire", "channelfireball" and
  "RekietaLawLive" on four other sites.