commit 6eaa21a8482e67681396ea126f07ebde76bf97d7
parent 9a22aec421d7f9b3222ce60f5370a7b9b8140544
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Mon, 28 Sep 2026 13:20:58 -0400
docs: the clone is the canonical public copy; PUBLISH.md "The source mirror (homepage)"; create-archives.sh deleted
README and SETUP lead with `git clone https://archilyzer.pages.dev/source/archilyzer.git`
(main only, ids differ because paths are scrubbed), the raw tree at
/source/tree/ and the tarball as the no-git alternative, regenerated by
`archilyzer source publish` inside `archilyzer build homepage`. PUBLISH.md gains
the section: what a publish does, the gate, the two operator files and their
formats, the tools, the skip key, and the Pages traps — the `.git` segment,
the 20,000-file / 25 MiB caps (refused at 15,000 / 24 MiB), `_headers` (root
only; only `wrangler pages dev` honours it locally), Tailwind and tsc. The
hand-run create-archives.sh, which nothing called, is gone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
| M | PUBLISH.md | | | 108 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- |
| M | README.md | | | 20 | ++++++++++++++++---- |
| M | SETUP.md | | | 25 | ++++++++++++++++++------- |
| D | create-archives.sh | | | 62 | -------------------------------------------------------------- |
4 files changed, 140 insertions(+), 75 deletions(-)
diff --git a/PUBLISH.md b/PUBLISH.md
@@ -8,6 +8,7 @@ is in [ENVIRONMENT.md](ENVIRONMENT.md).
- [What gets published](#what-gets-published)
- [The three ways to drive it](#the-three-ways-to-drive-it)
- [Cloudflare Pages](#cloudflare-pages)
+- [The source mirror (homepage)](#the-source-mirror-homepage)
- [Preview deployments](#preview-deployments)
- [Download archives and R2](#download-archives-and-r2)
- [Securing downloads against cost-abuse](#securing-downloads-against-cost-abuse)
@@ -25,7 +26,7 @@ server-side code, servable by anything:
|---|---|---|---|
| A **site** | The export app over the channels one site selects: search, transcripts, charts, downloads. One corpus can publish several. | `export/out` (docker mode: `export/.export-builds/<siteId>/out`) | `transcripts/sites/<id>/site.json` ([SITE.md](SITE.md)) |
| The **hub** | The export app in hub mode: federated search across the family of sites. | `export/out` | `transcripts/sites/_homepage/homepage.json` |
-| The **homepage** | The project's own site (`homepage/`). | `homepage/out` | — |
+| The **homepage** | The project's own site (`homepage/`), with the source mirror, its raw tree and the source tarball. | `homepage/out` | `~/.config/archilyzer/` (the source mirror's two operator files) |
The hub and the homepage are two different Pages projects: the hub deploys to the
project `homepage.json` names (e.g. `archilyzer-hub`) and is refused `archilyzer`,
@@ -50,7 +51,8 @@ entry points in `common/publish/build.ts`.
| Build, then deploy | *Build & deploy* | `build-deploy` | `build site <id>` then `deploy site <id>` |
| Build every site | /sites → **Build all sites** | — | `build all [--skip-archives]` |
| The hub | /sites → Hub → **Build hub** / **Deploy hub** | `build-hub`, `deploy-hub` | `build hub`, `deploy hub [--preview <branch>]` |
-| The homepage | /sites → Homepage → **Build homepage** (tick *Deploy after build*) / **Deploy homepage**, with an optional preview branch | `build-homepage` (`{"deploy":true}` to deploy after), `deploy-homepage` (`{"preview":"<branch>"}`) | `build homepage`, `deploy homepage [--preview <branch>]` |
+| The homepage | /sites → Homepage → **Build homepage** (tick *Deploy after build*) / **Deploy homepage**, with an optional preview branch | `build-homepage` (`{"deploy":true}` to deploy after), `deploy-homepage` (`{"preview":"<branch>"}`) | `build homepage [--no-source]`, `deploy homepage [--preview <branch>]` |
+| The source mirror alone | — (every homepage build runs it) | — | `source publish [--force] [--check] [--keep-scratch]`, `source audit [<git dir>]` |
`pnpm archilyzer <command>` is the short form of
`pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts <command>`;
@@ -86,6 +88,108 @@ fits inside Cloudflare's **free tier**.
preview path passes `--branch`; `deploy homepage` passes `--branch main`. If a
"production" deploy did not go live, check what branch the checkout is on.
+## The source mirror (homepage)
+
+The homepage carries the project's own source, read-only, as static files:
+
+| Path on the site | What |
+|---|---|
+| `/source/` | The page: the clone command, the mirror head, the private `main` it reflects |
+| `/source/archilyzer.git/` | A git repository for git's **dumb HTTP** protocol: `git clone https://archilyzer.pages.dev/source/archilyzer.git` |
+| `/source/tree/` | Every tracked file of `main`, raw, with a generated `index.html` per directory |
+| `/downloads/archilyzer-source.tar.gz` + `snapshot.json` | The same tree without history |
+| `/source/manifest.json` | What was published, from which private commit, audited how |
+
+**`archilyzer source publish`** makes them, and **`archilyzer build homepage` runs it**
+between compose and `next build` — so do the editor's /sites homepage jobs and
+`pnpm ops build-homepage`, in the editor's own process and `PATH`. A refusal fails
+the build before `next build`; nothing is deployed with a missing or stale source.
+`build homepage --no-source` (CLI only) removes the previously published source
+instead, because it was audited against the rules of its own day.
+
+What one publish does:
+
+1. A **fresh** `git clone --no-local --bare --single-branch --no-tags --branch main` of
+ the checkout's git *common* dir — so a worktree build mirrors the primary's `main`.
+ The private repository's history is never rewritten.
+2. **git-filter-repo** rewrites that copy: file contents (`--replace-text`) AND commit
+ messages (`--replace-message`) with the operator's scrub rules. Author and committer
+ identities are not rewritten — the gate still reads them.
+3. `git repack -a -d --max-pack-size=20m`, `pack-refs`, `update-server-info`: the
+ dumb-HTTP file set is `HEAD`, `refs/heads/main`, `packed-refs`, `info/refs`,
+ `objects/info/packs` and `objects/pack/*.{pack,idx}`, copied from an **allowlist** —
+ never `config` (it names the clone's origin path), `hooks/`, `filter-repo/` (the
+ PRIVATE commit ids) or `logs/`.
+4. **The gate** (`common/publish/sourceAudit.ts`): every object of the rewritten
+ mirror — blobs, commits and tags whole (messages and identity lines), trees by
+ entry name — then every staged file and path (the tarball decompressed), searched
+ for every denied literal; then gitleaks over the history, when it is installed.
+ **One hit and nothing is written.** The report names a literal only as
+ `#n (x…, len L)` and masks every occurrence in its context; it ends with
+ `add a rule to ~/.config/archilyzer/source-scrub.txt or drop the file from history,
+ then re-run.`
+5. The tree (`git archive` → `tar -x`, a page per directory; a tracked `index.html`
+ or a symlink is a refusal) and the tarball (`git archive --format=tar.gz -9`).
+6. Limits, then the install: link-safe (`common/bin/_publicFile.ts`), the manifest
+ removed first and written **last**, so a crash leaves the page's empty state,
+ never a manifest over a half-copied tree.
+
+An unchanged `main` with unchanged rules skips (`[source] up to date at …`;
+`--force` rebuilds). `--check` does everything but the install and writes nothing.
+`--keep-scratch` leaves the scratch clone (`ARCHILYZER_SOURCE_SCRATCH`, default the
+OS temp dir) for a look. `archilyzer source audit [<git dir>]` runs the gate alone —
+over the published mirror by default, or over a clone: `archilyzer source audit
+<clone>/.git`.
+
+**The two operator files** live outside the repo, in
+`${ARCHILYZER_CONFIG_DIR:-~/.config/archilyzer}/` (`SOURCE_SCRUB_FILE` and
+`SOURCE_DENYLIST_FILE` override each path; [ENVIRONMENT.md](ENVIRONMENT.md)). They are
+never committed; keep them mode 600. A missing file is a refusal naming it.
+
+- `source-scrub.txt` — git-filter-repo `--replace-text` lines: `lhs==>rhs` (split at
+ the last `==>`), `literal:…`, `regex:…==>…`, `glob:…==>…`; a line with no `==>` is
+ replaced by `***REMOVED***`. Lines whose first non-blank character is `#` are
+ comments (the step drops them — filter-repo itself would treat one as a literal).
+ `<your home dir>==>/home/user` is built in and runs first. Rules apply in order.
+- `source-denylist.txt` — one literal per line, `i:` in front for any ASCII case;
+ `#` comments. **Every literal scrub rule's left side is denied too**, so a rule that
+ stops matching (a new spelling in history) refuses instead of leaking.
+- The rules' hash is the skip key, kept in `homepage/.source-publish.json` — beside
+ `public/`, never in it: a published hash of the denylist would confirm a guess at it.
+
+**Tools.** `git filter-repo` (`pipx install git-filter-repo`; without it the step runs
+`pipx run --spec git-filter-repo==2.47.0`, which needs the network on first use, and
+without pipx it refuses with the install line). gitleaks is optional: without it the
+secret scan is skipped with a WARNING and the literal audit still runs.
+`archilyzer doctor` has a "source publish" block: which filter-repo would run,
+gitleaks, the two files (rule counts and modes, never contents) and the last publish.
+The mirror's ids are deterministic for a given filter-repo version; an upgrade that
+changes its rewriting changes every id (readers re-clone). The manifest records both
+tools.
+
+**Cloudflare Pages, and the traps it sets.**
+
+- **Never a path segment named `.git`.** wrangler's upload ignore list drops `**/.git`
+ (and `**/node_modules`) *silently*, and Cloudflare's managed rules block `/.git/`
+ requests. The mirror is `archilyzer.git`, which passes both. Extensionless files
+ (`HEAD`, `info/refs`) upload and serve fine.
+- **The caps:** 20,000 files per deployment and 25 MiB per file. The step refuses at
+ **15,000 files** (the rest of the site needs room) and **24 MiB**; packs are split
+ at 20 MB.
+- **`_headers` makes the raw tree plain text.** wrangler's mime map would serve `.ts`
+ as `video/mp2t`, `.mjs` as JavaScript and `.html` as a page on this origin, so
+ `homepage/public/_headers` sets `/source/tree/*` to `text/plain; charset=utf-8` with
+ `nosniff` and `noindex`, then puts back `text/html` for the directory pages and the
+ real types of the few binaries (`.ttf`, `.m4a`, `.zip`, `.onnx`, `.svg` — the SVG
+ under a `default-src 'none'` CSP). Only the ROOT `_headers` is read; the tree's own
+ copies are served as text. Locally, only `wrangler pages dev` honours `_headers` —
+ `next dev` and `serve` do not, and neither serves a directory's `index.html` at
+ `/<dir>/` the way Pages does.
+- **Tailwind and tsc must not see the mirror.** Tailwind v4 scans every file
+ `.gitignore` does not exclude (a binary pack yields "class names" that break the
+ stylesheet), and `homepage/tsconfig.json` includes `**/*.ts`: `/homepage/public/source`
+ is gitignored and `public` is excluded from the tsconfig. Keep both.
+
## Preview deployments
A **preview** is the same built bundle deployed to a branch that is not the Pages
diff --git a/README.md b/README.md
@@ -477,13 +477,25 @@ Channels can sync automatically on a per-channel cadence via a cron heartbeat
Archilyzer is MIT licensed and deliberately **forge-neutral**: there is no `.github/`
directory, no provider-specific CI configuration, and nothing in the build that assumes
-a particular host. However you obtained this tree — a release archive, a mirror, or a
+a particular host. However you obtained this tree — a clone, a release archive, or a
fork on whichever platform — it is a complete, self-contained working copy.
-The project site publishes a dated source snapshot at
+**The canonical public copy is a read-only git mirror on the project site:**
+
+```sh
+git clone https://archilyzer.pages.dev/source/archilyzer.git
+```
+
+It is `main` only, with its whole history, served as static files over git's dumb
+HTTP protocol, and regenerated from the private repository by every homepage build
+(`archilyzer source publish`, which `archilyzer build homepage` runs). Its commit ids
+differ from the private repository's because machine paths are scrubbed on the way
+out; a gate refuses to publish anything that still carries a denied string. Every
+file is browsable raw at [archilyzer.pages.dev/source/tree/](https://archilyzer.pages.dev/source/tree/),
+and the same tree without history is a tarball at
[archilyzer.pages.dev/downloads](https://archilyzer.pages.dev/downloads/), with a
-checksum. `./create-archives.sh` regenerates one, writing a `snapshot.json` sidecar
-(commit, size, SHA-256) beside it.
+`snapshot.json` sidecar (commit, size, SHA-256). How it is built:
+[PUBLISH.md](PUBLISH.md#the-source-mirror-homepage).
See [CONTRIBUTING.md](CONTRIBUTING.md) to work on the code.
diff --git a/SETUP.md b/SETUP.md
@@ -159,8 +159,9 @@ Caveats on native Windows:
- Building the **whisper.cpp / parakeet.cpp** transcription backends is Unix-oriented;
do transcription work under WSL2, or let the container do it (it ships a backend
already built).
-- `create-archives.sh` and the external-cron sync path (`pnpm sync:tick` from cron)
- assume a Unix shell — use WSL2 or Task Scheduler equivalents.
+- The external-cron sync path (`pnpm sync:tick` from cron) assumes a Unix shell — use
+ WSL2 or Task Scheduler equivalents. So does publishing the source mirror
+ (`archilyzer source publish`: git-filter-repo, `tar`).
---
@@ -168,8 +169,20 @@ Caveats on native Windows:
Archilyzer is MIT licensed and **forge-neutral** — there is no provider-specific CI
config or `.github/` directory, and nothing in the build assumes a particular host. Any
-complete copy of the tree works: a clone from wherever the project is mirrored, a fork
-of your own, or the dated source snapshot published by the project site:
+complete copy of the tree works. The canonical public copy is the read-only git mirror
+on the project site — `main` with its whole history:
+
+```sh
+git clone https://archilyzer.pages.dev/source/archilyzer.git archilyzer
+cd archilyzer
+```
+
+`git pull` updates it; nothing takes a push. Its commit ids differ from the private
+repository's (machine paths are scrubbed on the way out), and every homepage build
+regenerates it (`archilyzer source publish`; [PUBLISH.md](PUBLISH.md#the-source-mirror-homepage)).
+
+No git? The same tree without history is a tarball, with a `snapshot.json` sidecar
+(commit, size, SHA-256) beside it:
```sh
curl -LO https://archilyzer.pages.dev/downloads/archilyzer-source.tar.gz
@@ -178,9 +191,7 @@ cd archilyzer
```
The snapshot is a working tree at one commit — no history, no branches, no remote — so
-updating means fetching a newer one. Regenerate a snapshot yourself with
-`./create-archives.sh`, which also writes a `snapshot.json` sidecar (commit, size,
-SHA-256) beside it.
+updating means fetching a newer one.
Once you have a tree, from a clone or a snapshot alike:
diff --git a/create-archives.sh b/create-archives.sh
@@ -1,62 +0,0 @@
-#!/bin/bash
-# Build the downloadable source snapshot served at /downloads/ on the project
-# site, plus a sidecar of facts about it.
-#
-# WHY A SNAPSHOT AND NOT A REPOSITORY. There is no public git remote, by choice.
-# `git archive` of `main` produces the working tree at one commit: no history, no
-# branches, no remote, nothing to `git pull`. The /downloads/ page says exactly
-# that rather than implying a clone.
-#
-# WHY THE SIDECAR. The filename is deliberately STABLE — a dated filename would
-# break every link the moment a new snapshot shipped. So the date, commit, size
-# and checksum live in snapshot.json beside it, and the page states the truth it
-# reads there instead of hardcoding facts that go stale. No sidecar → the page
-# renders an explanation and NO download link, rather than a link that lies.
-#
-# Safe to publish: `git archive` only includes TRACKED files, so the corpus
-# (transcripts/, gitignored), settings.json (untracked; only .example is
-# tracked) and node_modules are all structurally excluded.
-set -euo pipefail
-
-cd "$(dirname "$0")"
-
-REF="${1:-main}"
-OUT_DIR="homepage/public/downloads"
-TARBALL="$OUT_DIR/archilyzer-source.tar.gz"
-SIDECAR="$OUT_DIR/snapshot.json"
-
-mkdir -p "$OUT_DIR"
-
-# --prefix so the tarball unpacks into `archilyzer/` rather than spilling into
-# the current directory. The docs tell people to `cd archilyzer` afterwards.
-git archive -9 --format tar.gz --prefix=archilyzer/ -o "$TARBALL" "$REF"
-
-BYTES=$(stat -c %s "$TARBALL" 2>/dev/null || stat -f %z "$TARBALL")
-SHA=$(sha256sum "$TARBALL" | cut -d' ' -f1)
-COMMIT=$(git rev-parse "$REF")
-SUBJECT=$(git log -1 --format=%s "$REF")
-GENERATED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
-
-# node rather than a heredoc so the commit subject is JSON-escaped properly —
-# it is arbitrary human text and routinely contains quotes.
-GENERATED_AT="$GENERATED_AT" COMMIT="$COMMIT" SUBJECT="$SUBJECT" \
-BYTES="$BYTES" SHA="$SHA" node -e '
- const fs = require("fs");
- fs.writeFileSync(process.argv[1], JSON.stringify({
- generatedAt: process.env.GENERATED_AT,
- commit: process.env.COMMIT,
- subject: process.env.SUBJECT,
- bytes: Number(process.env.BYTES),
- sha256: process.env.SHA,
- }, null, 2) + "\n");
-' "$SIDECAR"
-
-echo "create-archives: $TARBALL ($(( BYTES / 1024 )) KiB) at ${COMMIT:0:12}"
-
-# Cloudflare Pages rejects any single asset over 25 MB. The snapshot is an order
-# of magnitude under that, but say so loudly if that ever stops being true —
-# silently shipping an asset Pages will refuse is a deploy that "succeeds" and
-# then 404s.
-if [ "$BYTES" -gt 26214400 ]; then
- echo "create-archives: WARNING — $TARBALL exceeds Cloudflare Pages' 25 MB per-file limit." >&2
-fi
