# Operating an archive

Recipes for running an archive from a shell or an agent. Every command named here is in
[COMMANDS.md](COMMANDS.md), generated from the commands' own help.

## The three surfaces

| surface | what it is | needs |
|---|---|---|
| `pnpm ops <action>` | the running editor's actions over HTTP (`/api/ops/*`): the same code a click runs, as jobs on the editor's queues | a running editor; `ARCHILYZER_EDITOR_URL` (default `http://localhost:3001`) and `WORKER_TOKEN` (the editor's own, from `editor/.env`) |
| `pnpm archilyzer <command>` | the core's CLI: index, compose, publish stages, reports, offline refreshes, doctor | the checkout and its `transcripts/`; in the container, `docker compose exec editor pnpm archilyzer …` |
| the MCP server | reads a published archive (search, transcripts, reports); `fetch_clip` asks the editor for clip media | registered as `archilyzer` ([AGENTS.md](AGENTS.md), [mcp/README.md](mcp/README.md)) |

- A job-starting `pnpm ops` action answers with a `jobId` once the job is queued. `--wait` follows its log to the
  end and exits with its status; a platform queue may hold a job for hours.
- What is there: `pnpm ops get channels`, `pnpm ops get channel <slug> --counts`, `pnpm ops get publish`,
  `pnpm archilyzer doctor`.
- Every fetch goes through the editor (paced per platform, cookie-aware, provenanced). Never run yt-dlp, whisper or
  parakeet by hand, and never edit `transcripts/**` by hand: each file has one writer, named below.

## Add a channel

```sh
pnpm ops create-channel --json '{"fields":{"name":"Example","handling":"youtube","url":"https://www.youtube.com/@example"}}'
pnpm ops channel-config --json '{"slug":"example","patch":{"downloadFilterExclude":"#shorts"}}'
pnpm ops channel-config --json '{"slug":"example","sites":[{"siteId":"jeralyzer"}]}'
pnpm ops get channel example
```

- `handling`: `"youtube"` fetches the platform's captions; `"transcribe"` downloads audio and transcribes it here.
  `fields` takes the New channel form's field names, the same as `channel-config`'s `patch`
  ([RUNNING_IN_DOCKER.md](RUNNING_IN_DOCKER.md), "Driving the editor without a browser"). Every `config.json` key:
  [CHANNEL.md](CHANNEL.md).
- `sites` is the whole membership set; absent or `[]` keeps the channel on no site.
- `"fetchPlaylist": true` on `create-channel` stores the playlist at once (the form's "Fetch playlist now"); a social channel's is `"fetchPostsNow": true`.
- Rename or delete: `pnpm ops rename-channel --json '{"slug":"old","newSlug":"new"}'`,
  `pnpm ops delete-channel --json '{"slug":"x","confirm":"x"}'` — both refused while the channel is busy.

## Sync and download

```sh
pnpm ops sync             --json '{"slug":"example"}' --wait
pnpm ops metadata-scan    --json '{"slug":"example"}' --wait
pnpm ops download-missing --json '{"slug":"example"}' --wait
pnpm ops retry-bucket     --json '{"slug":"example","bucket":"<bucket>"}' --wait
pnpm ops refresh-metadata --json '{"slug":"example","id":"<videoId>"}' --wait
pnpm ops persist-videos   --json '{"items":[{"slug":"example","id":"<videoId>"}],"dryRun":true}'
```

- The lanes do this unattended: `pnpm ops lane --json '{"lane":"download","enabled":true,"action":"start"}'`;
  `"held": true` pauses dispatch and keeps the runner's place. Lanes: `transcription`, `download`, `digest`,
  `backfill`.
- A channel's priority: `pnpm ops channel-priority --json '{"slugs":["example"],"tier":"low"}'`
  (`"operation"` pins one operation's tier).
- Bucket names and sizes: `pnpm ops get channel example`.
- Keep videos out of cleanup by title or description: `pnpm ops keep-videos --json '{"slug":"example","match":"interview","dryRun":true}'`.

## Import from archive.org, Odysee, BitChute and the Wayback Machine

```sh
pnpm ops import-archive-org --json '{"slug":"example-archive","item":"<item>","match":"\\.mp4$","dryRun":true}' --wait
pnpm ops import-archive-org --json '{"slug":"example-archive","item":"<item>","match":"\\.mp4$"}' --wait
pnpm ops import-archive-org --json '{"slug":"example-archive","items":["<item>","<item>"],"dryRun":true}' --wait
pnpm ops import-archive-org --json '{"slug":"example-archive","query":"collection:<collection>","limit":50,"dryRun":true}' --wait
pnpm ops get remote-listing example-odysee
pnpm ops import-video       --json '{"slug":"example","url":"https://www.bitchute.com/video/<id>/"}' --wait
pnpm ops import-video       --json '{"slug":"example","url":"https://odysee.com/@example:0/<video>:0"}' --wait
pnpm ops import-video       --json '{"slug":"example","url":"https://web.archive.org/web/<timestamp>/<original-url>"}' --wait
```

- `import-archive-org` takes one item's files (`files` exact, or `match` a regex), many items (`items`), or the
  items an archive.org search finds (`query`, the first `limit`), one file at a time on archive.org's queue with a
  pause between files and between items; records already held (on disk, or in the saved-video store) are skipped.
  A dry run lists each file as held, RESTRICTED (archive.org will answer 401/403) or would-get. archive.org is
  fetched over BitTorrent when it can be, never by yt-dlp.
- `get remote-listing <slug>` lists an Odysee or BitChute channel upstream (one paced read on the platform's queue)
  and prints what it lists that the channel does not hold, and what it holds that is no longer listed.
- A one-off Odysee or BitChute import is paced like that platform's own downloads. A whole Odysee or BitChute
  channel is a channel with that URL, then `sync`.
- A Wayback capture is named by what it copies; `wayback.json` beside it records the capture.
- Bring records up to the current rules, offline (dry run first):
  `pnpm archilyzer archive-org refresh <slug> --dry-run`, `pnpm archilyzer wayback refresh <slug> --dry-run`,
  `pnpm archilyzer feeds backfill-metadata <slug> --dry-run` (or `pnpm ops feed-metadata` on the editor).

## Fetch and capture posts

```sh
pnpm ops create-channel --json '{"fields":{"name":"Example (X)","handling":"transcribe","url":"https://x.com/example"}}'
pnpm ops fetch-posts    --json '{"slug":"example-x"}' --wait
pnpm ops fetch-posts    --json '{"slug":"example-x","older":true}' --wait
pnpm ops capture-posts  --json '{"slug":"example-x","ids":["<postId>"]}' --wait
```

- An X, Bluesky or XenForo URL makes a social channel (its handle derived from the URL). `"older": true` walks back below the oldest
  archived post and resumes from its saved place; `"full": true` re-walks the timeline.
- `capture-posts` saves a screenshot and the attached media of named posts (and a linked X Article).
- Forum pages saved from a browser: `pnpm archilyzer posts import-html <slug> <file-or-dir> --dry-run`.
- Which archived posts were deleted at the source: `pnpm archilyzer posts check --slug <slug>`.

## Transcribe

```sh
pnpm ops lane              --json '{"lane":"transcription","enabled":true,"action":"start"}'
pnpm ops transcribe-bucket --json '{"slug":"example"}' --wait
pnpm ops transcribe        --json '{"path":"/abs/clip.mp4","start":120,"end":150,"words":true}' --wait
```

- A transcript the index has not caught up with: `pnpm ops get transcript <videoId> --slug <slug>` reads its cues
  off disk (the MCP's `get_transcript` does the same through the editor), and
  `pnpm ops build-cues --json '{"slug":"<slug>","ids":["<videoId>"]}' --wait` writes its `transcript.cues.json`.
- The transcription lane takes every channel's downloaded, untranscribed audio; `transcribe-bucket` runs one
  channel's now.
- `transcribe` runs one local file (or a window of it) through the corpus's own engine and model; the result JSON
  (cues, text, and with `"words": true` the word timings parakeet keeps) is printed on stdout with `--wait`.
  `"out"` writes it to a file outside the corpus.

## Tag

```sh
pnpm ops get tags
pnpm ops tags       --json '{"op":"define","tag":{"id":"example-collab","label":"Collab"}}'
pnpm ops tag-videos --json '{"tag":"example-collab","op":"add","videos":[{"slug":"example","id":"<videoId>"}]}'
pnpm ops tag-videos --file ids.json
```

- `transcripts/tags.json` has one writer, and `tags` and `tag-videos` go through it, recording who asked
  (`ARCHILYZER_AGENT`). `remove` unpins; `suppress` rejects a rule's hit.
- A tag's `sites` limits the sites it exists on.
- Read side: the MCP's `list_tags`, and `tags` on `search_transcripts` / `enumerate_matches`.

## Prepare and export reports

```sh
pnpm archilyzer reports convert sweep <sweep.md> --out <report.json>
pnpm ops fetch-windows   --json '{"siteId":"example-site","dryRun":true}'
pnpm ops fetch-windows   --json '{"siteId":"example-site"}' --wait
pnpm ops reports-prepare --json '{"siteId":"example-site"}' --wait
pnpm ops reports-export  --json '{"siteId":"example-site","formats":["html","md"]}' --wait
```

- A report is `transcripts/sites/<site>/reports/<id>/report.json` ([REPORT.md](REPORT.md),
  [CITATIONS.md](CITATIONS.md)); `reports convert` makes one from a `/sweep` report, an `/ask` answer or a
  report-to-video manifest.
- `fetch-windows` with `siteId` fetches every clip window the site's published reports cite that the disk does not
  hold, one paced job per platform; a re-run resumes.
- `reports-prepare` cuts the cited clips and copies the cited post captures, then exports; it fails naming each
  citation that lacks media. Offline: `pnpm archilyzer reports prepare <siteId>`,
  `pnpm archilyzer reports export <siteId>`.
- One cited moment's media, from an agent: the MCP's `fetch_clip`.

## Publish

```sh
pnpm ops get publish
pnpm ops publish --json '{"verb":"now"}' --wait
pnpm ops publish --json '{"verb":"build","siteId":"example-site"}' --wait
pnpm ops publish --json '{"verb":"deploy","siteId":"example-site","preview":"check"}' --wait
pnpm archilyzer publish status
pnpm archilyzer publish build example-site --out /srv/example-private
```

- Publishing is stages on one queue: update the index, build a site into its bundle, deploy the bundle, each
  checked live. `now` runs what Publish now runs (the stale index, then each site's policy). Offline, under the same
  lock: `pnpm archilyzer publish index`, `publish build <id>`, `publish deploy <id> --preview <branch>`.
- A private build served by something other than Pages: `publish build <id> --out <dir>` lays the bundle out in
  `<dir>` (hard links where it can; the directory is emptied first and guarded like a local deploy's). A report
  citation whose media is not prepared yet passes with `--allow-missing-media` (`"allowMissingMedia": true` on the
  ops `build` verb).
- The publish lane: `pnpm ops lane --json '{"lane":"publish","action":"start"}'` (`"held": true` pauses it).
- Details, policies and the hub and homepage: [PUBLISH.md](PUBLISH.md).

## Research with the MCP

- `/ask` answers a question with citations in the conversation; `/sweep` writes a cited report to a file.
- Search first, then pull only the cited seconds with `fetch_clip`. The editor fetches only for a channel it
  already archives.
- With the editor configured, the MCP also queues archival work (`enqueue`: sync, download-missing, retry-bucket,
  transcribe-bucket, fetch-posts, import-video), follows it (`get_job`), reports what a channel holds by date and
  where the gaps are (`channel_coverage`), and reads a video not yet published (`get_transcript`). The operator's
  notes: `notes` (with `UMTOOL_URL`). Settings, storage and deletes stay `pnpm ops`.
- Tools and their arguments: [mcp/README.md](mcp/README.md).