commit 0a6ca1ddf93fc94fef4a059d9d7e593b8f428331
parent d56bae92e79738d5a0722f30c617516a154ec1cb
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Mon, 28 Sep 2026 15:01:21 -0400
Merge r12/source-mirror (release 12 slice R) — `archilyzer source publish` and /source: a read-only git mirror of main on the project site (dumb HTTP), its raw tree and a tarball, scrubbed by git-filter-repo and refused by an audit gate when a denied literal remains; a refusal withdraws the previous publish and the deploy refuses a source it cannot vouch for; create-archives.sh deleted; reviewed SHIP
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
48 files changed, 4859 insertions(+), 139 deletions(-)
diff --git a/.dockerignore b/.dockerignore
@@ -17,9 +17,14 @@ export/public/summaries/
export/public/transcripts/
export/public/transcripts.tar.gz
export/public/transcripts.tar.xz
-# The source snapshot served by the project site — regenerated by
-# create-archives.sh on the host, never needed inside a build container.
+# The published source (`archilyzer source publish`, run by `archilyzer build
+# homepage`): the tarball, the git mirror and the raw tree (~66 MB), and the
+# skip key beside them, which holds the rules hash. All are regenerated on the
+# host; none is baked into an image, where a plain `next build` would ship a
+# copy that today's rules were never applied to.
homepage/public/downloads/
+homepage/public/source/
+homepage/.source-publish.json
# Build staging / caches / per-site outputs — bind-mounted at run time, never baked.
export/.export-index/
diff --git a/.gitignore b/.gitignore
@@ -46,7 +46,8 @@ yarn-error.log*
# nested transcripts repo
/transcripts
-# downloadable transcript archives (see the commented block in create-archives.sh)
+# downloadable transcript archives (an old whole-corpus format; ignored so a
+# stray copy is never committed)
/export/public/transcripts.tar.gz
/export/public/transcripts.tar.xz
@@ -90,10 +91,22 @@ yarn-error.log*
/homepage/public/channel-sites.json
/homepage/public/chart-templates.json
/homepage/public/homepage-summary.json
-# source snapshot tarball + its sidecar (regenerate with ./create-archives.sh).
+# source snapshot tarball + its sidecar (regenerated by `archilyzer source
+# publish`, which `archilyzer build homepage` runs).
# Note this does NOT cover /homepage/public/_headers — the `_headers` rule above
# is scoped to /export/public, so the homepage's stays tracked.
/homepage/public/downloads
+# The published source: the git mirror (archilyzer.git: packs, info/refs), the
+# raw tree and manifest.json, all written by `archilyzer source publish`
+# (common/publish/source.ts). Ignored for Tailwind as much as for git: Tailwind
+# v4 scans every file .gitignore does not exclude, and a pack file is binary —
+# the umtool `.next-*` failure below, where a scanned binary yields "class
+# names" that break the stylesheet. homepage/tsconfig.json excludes `public`
+# for the same reason: the raw tree is ~2,000 .ts/.tsx files of the whole repo.
+/homepage/public/source
+# The publish step's skip key (rules hash + source commit). Kept out of
+# public/ on purpose: a hash of the denylist is a guess-confirmation oracle.
+/homepage/.source-publish.json
# editor e2e fixtures and ephemeral state
/editor/test-transcripts/
diff --git a/ENVIRONMENT.md b/ENVIRONMENT.md
@@ -12,7 +12,7 @@ The one override surface for where things live and which binary runs. Every one
| Variable | Default | What it does | Read by |
|---|---|---|---|
-| `TRANSCRIPTS_DIR` | `<repo>/transcripts` | The corpus: channels, sites, the LMDB index, job logs, the saved-video store. | common/lib/paths.ts (getPaths) |
+| `TRANSCRIPTS_DIR` | `<repo>/transcripts` | The corpus: channels, sites, the LMDB index, job logs, the saved-video store. umtool reads `<it>/channels` too, when its own `CHANNELS_DIR` is unset. | common/lib/paths.ts (getPaths) |
| `SAVED_VIDEOS_DIR` | `<TRANSCRIPTS_DIR>/saved-videos` | The persisted source-video store, when it should live on another disk. | common/lib/paths.ts (getPaths) |
| `SITES_DIR` | `<TRANSCRIPTS_DIR>/sites` | Per-site config (`<id>/site.json`, every key in [SITE.md](SITE.md)) and the homepage's `_homepage/`. | common/lib/paths.ts (getPaths) |
| `SETTINGS_FILE` | `<repo>/settings.json` | The settings file (every key in [SETTINGS.md](SETTINGS.md)). | common/lib/paths.ts (getPaths) |
@@ -39,6 +39,10 @@ The one override surface for where things live and which binary runs. Every one
| `GALLERY_DL_BIN` | `gallery-dl` on PATH | The X/Twitter post fetcher, for social channels. | common/lib/paths.ts (getPaths) |
| `OLLAMA_URL` | `http://127.0.0.1:11434` | The local ollama server, the local digest and attribution engine. | common/lib/paths.ts (getPaths) |
| `CLAUDE_BIN` | `claude` on PATH | The `claude` CLI, driving the opt-in metered digest lane. | common/lib/paths.ts (getPaths) |
+| `ARCHILYZER_CONFIG_DIR` | `~/.config/archilyzer` | The operator's private config dir, outside the repo: the two inputs of `archilyzer source publish` below. Never committed. | common/lib/paths.ts (getPaths) |
+| `SOURCE_SCRUB_FILE` | `<ARCHILYZER_CONFIG_DIR>/source-scrub.txt` | git-filter-repo `lhs==>rhs` rules applied to file contents AND commit messages when the source mirror is generated (`<home dir>==>/home/user` is built in and runs first). Every rule's left side is also denied. See [PUBLISH.md](PUBLISH.md). | common/lib/paths.ts (getPaths) |
+| `SOURCE_DENYLIST_FILE` | `<ARCHILYZER_CONFIG_DIR>/source-denylist.txt` | Literals the published source must never contain, one per line (`i:` = any case). One hit anywhere in the mirror, the tree or the tarball refuses the publish. | common/lib/paths.ts (getPaths) |
+| `ARCHILYZER_SOURCE_SCRATCH` | the OS temp dir | Where `source publish` makes its scratch clone and stage (removed afterwards unless `--keep-scratch`). | common/lib/paths.ts (getPaths) |
## Runtime
@@ -123,7 +127,7 @@ The publish pipeline sets these for a process it spawns. Listed so a reader know
| `INSTANCE_MODE` | a site | `hub` makes the export build the hub. Set by `archilyzer build hub`. | export/app/lib/mode.ts, common/lib/archive/contract.ts |
| `BUILD_ARCHIVES` | on | `0` skips archive-zip generation for one build (`--skip-archives`). | common/bin/compose-site.ts, common/bin/build-archives.ts |
| `ARCHIVES_READONLY` | off | `1` inside a docker-mode build container: materialize archives, never write the shared cache. | common/bin/compose-site.ts |
-| `HOMEPAGE_PUBLIC_DIR` | `<repo>/homepage/public` | Where `compose homepage` writes. | common/bin/compose-homepage.ts |
+| `HOMEPAGE_PUBLIC_DIR` | `<repo>/homepage/public` | Where `compose homepage` and `source publish` write. | common/bin/compose-homepage.ts, common/publish/source.ts |
## Docker
@@ -182,4 +186,5 @@ Read only by a test harness, a fake binary or a test-mode branch. Never set one
| `E2E_RETRIES` | `0` | Retries per shard (`--retries N` wins); 0 keeps a sharded run comparable to a serial one. | scripts/run-sharded-e2e.mjs |
| `E2E_IMAGE` | `yt-dlp-transcript-browser-e2e` | The sharded e2e run's image tag. | scripts/run-sharded-e2e.mjs |
| `E2E_SKIP_BUILD` | off | `1` reuses the sharded e2e image instead of rebuilding it (`--no-build`). | scripts/run-sharded-e2e.mjs |
+| `E2E_EXPECT_SOURCE` | off (both states pass) | `1` makes the homepage suite's `/source/` specs fail on the empty state; a gate that ran `archilyzer source publish` first sets it. | homepage/e2e/source.spec.ts |
| `E2E_HOMEPAGE_SUMMARY_FILE` | `homepage/public/homepage-summary.json` | The synthetic summary the homepage's e2e dev server reads; set by `homepage/playwright.config.ts`, ignored by a production build. | homepage/app/lib/summary.ts |
diff --git a/PUBLISH.md b/PUBLISH.md
@@ -8,6 +8,7 @@ is in [ENVIRONMENT.md](ENVIRONMENT.md).
- [What gets published](#what-gets-published)
- [The three ways to drive it](#the-three-ways-to-drive-it)
- [Cloudflare Pages](#cloudflare-pages)
+- [The source mirror (homepage)](#the-source-mirror-homepage)
- [Preview deployments](#preview-deployments)
- [Download archives and R2](#download-archives-and-r2)
- [Securing downloads against cost-abuse](#securing-downloads-against-cost-abuse)
@@ -25,7 +26,7 @@ server-side code, servable by anything:
|---|---|---|---|
| A **site** | The export app over the channels one site selects: search, transcripts, charts, downloads. One corpus can publish several. | `export/out` (docker mode: `export/.export-builds/<siteId>/out`) | `transcripts/sites/<id>/site.json` ([SITE.md](SITE.md)) |
| The **hub** | The export app in hub mode: federated search across the family of sites. | `export/out` | `transcripts/sites/_homepage/homepage.json` |
-| The **homepage** | The project's own site (`homepage/`). | `homepage/out` | — |
+| The **homepage** | The project's own site (`homepage/`), with the source mirror, its raw tree and the source tarball. | `homepage/out` | `~/.config/archilyzer/` (the source mirror's two operator files) |
The hub and the homepage are two different Pages projects: the hub deploys to the
project `homepage.json` names (e.g. `archilyzer-hub`) and is refused `archilyzer`,
@@ -50,7 +51,8 @@ entry points in `common/publish/build.ts`.
| Build, then deploy | *Build & deploy* | `build-deploy` | `build site <id>` then `deploy site <id>` |
| Build every site | /sites → **Build all sites** | — | `build all [--skip-archives]` |
| The hub | /sites → Hub → **Build hub** / **Deploy hub** | `build-hub`, `deploy-hub` | `build hub`, `deploy hub [--preview <branch>]` |
-| The homepage | /sites → Homepage → **Build homepage** (tick *Deploy after build*) / **Deploy homepage**, with an optional preview branch | `build-homepage` (`{"deploy":true}` to deploy after), `deploy-homepage` (`{"preview":"<branch>"}`) | `build homepage`, `deploy homepage [--preview <branch>]` |
+| The homepage | /sites → Homepage → **Build homepage** (tick *Deploy after build*) / **Deploy homepage**, with an optional preview branch | `build-homepage` (`{"deploy":true}` to deploy after), `deploy-homepage` (`{"preview":"<branch>"}`) | `build homepage [--no-source]`, `deploy homepage [--preview <branch>]` |
+| The source mirror alone | — (every homepage build runs it) | — | `source publish [--force] [--check] [--keep-scratch]`, `source audit [<git dir>]` |
`pnpm archilyzer <command>` is the short form of
`pnpm --filter yt-dlp-transcript-common exec tsx bin/archilyzer.ts <command>`;
@@ -86,6 +88,179 @@ fits inside Cloudflare's **free tier**.
preview path passes `--branch`; `deploy homepage` passes `--branch main`. If a
"production" deploy did not go live, check what branch the checkout is on.
+## The source mirror (homepage)
+
+The homepage carries the project's own source, read-only, as static files:
+
+> **Before ANY deploy of the homepage — a preview included — the denylist must hold
+> everything private.** A Pages preview is a public publication, and every Pages
+> deployment stays reachable at its own `<hash>.<project>.pages.dev` URL until that
+> deployment is deleted; a preview's branch alias is guessable (the plans that name
+> it ship in the mirror). If a deploy ever carries something private, DELETE that
+> deployment in the Pages dashboard: a newer deploy does not remove the old one.
+
+| Path on the site | What |
+|---|---|
+| `/source/` | The page: the clone command, the mirror head, the private `main` it reflects |
+| `/source/archilyzer.git/` | A git repository for git's **dumb HTTP** protocol: `git clone https://archilyzer.pages.dev/source/archilyzer.git` |
+| `/source/tree/` | Every tracked file of `main`, raw, with a generated `index.html` per directory |
+| `/downloads/archilyzer-source.tar.gz` + `snapshot.json` | The same tree without history |
+| `/source/manifest.json` | What was published, from which private commit, audited how |
+
+**`archilyzer source publish`** makes them, and **`archilyzer build homepage` runs it**
+between compose and `next build` — so do the editor's /sites homepage jobs and
+`pnpm ops build-homepage`, in the editor's own process and `PATH` (that process needs
+`~/.local/bin` on its `PATH` to find a pipx-installed `git filter-repo`; without it the
+step falls back to `pipx run`, which needs the network).
+
+**A refusal withdraws the source, everywhere it could ship from.** It fails the build
+before `next build`, and:
+
+- the step removes the LAST publish from `homepage/public` (manifest first, then the
+ mirror, the tree, the tarball, `snapshot.json` and the skip key) — it was audited
+ under rules that may not be today's, and the refusal is the best evidence that they
+ are not. Any outcome but success does this once the rules are loaded (an audit hit,
+ a limit, a missing tool, a cancel, a crash); `--check` writes nothing, this included;
+- the build removes the last BUILD's copy from `homepage/out` (`out/source`, the
+ tarball, `snapshot.json`);
+- **`deploy homepage` (and /sites → Deploy homepage) refuses** an `out/` holding a
+ source unless the skip key says that publish was made under today's rules and step
+ version, of today's `main`, by today's gitleaks, and is exactly the one in `out/`:
+ its mirror head, and a digest over every published file (the mirror, the tree, the
+ manifest, the tarball, `snapshot.json` — sorted path, size and sha256, recomputed
+ over `out/`), so a mixed or edited `out/` refuses too: "run `archilyzer build
+ homepage` (it re-audits), then deploy". An `out/` whose `/source` page shows the empty
+ state (`--no-source`) deploys as before; one with no `/source` page (a refused build)
+ does not.
+
+**After merging a change to this step, rebuild and restart the editor before any
+/sites Homepage job.** The editor runs its BUILT bundle: until it is rebuilt, its
+Build homepage job runs the old `buildHomepage` (without this step, or without the
+withdrawal) and its Deploy homepage job has no source check.
+
+`build homepage --no-source` (CLI only) removes the previously published source
+instead, because it was audited against the rules of its own day. A checkout with no
+git repository — the docker runtime, a tarball install — has nothing to mirror: the
+build says `no git repository here; nothing to mirror — the /source page will show its
+empty state`, removes any old publish and goes on; `source publish` run directly there
+exits 1 with the same sentence.
+
+What one publish does:
+
+1. A **fresh** `git clone --no-local --bare --single-branch --no-tags --branch main` of
+ the checkout's git *common* dir — so a worktree build mirrors the primary's `main`.
+ The private repository's history is never rewritten.
+2. **git-filter-repo** rewrites that copy: file contents (`--replace-text`) AND commit
+ messages (`--replace-message`) with the operator's scrub rules. Author and committer
+ identities are not rewritten — the gate still reads them.
+3. `git repack -a -d --max-pack-size=20m`, `pack-refs`, `update-server-info`: the
+ dumb-HTTP file set is `HEAD`, `refs/heads/main`, `packed-refs`, `info/refs`,
+ `objects/info/packs` and `objects/pack/*.{pack,idx}`, copied from an **allowlist** —
+ never `config` (it names the clone's origin path), `hooks/`, `filter-repo/` (the
+ PRIVATE commit ids) or `logs/`.
+4. **The gate** (`common/publish/sourceAudit.ts`): every object of the rewritten
+ mirror — blobs, commits and tags whole (messages and identity lines), trees by
+ entry name — then every staged file and path (the tarball decompressed), searched
+ for every denied literal; then gitleaks over the history, when it is installed.
+ **One hit and nothing is written** (and the last publish is withdrawn, above). The
+ report names a literal only by where it was written — `denylist line 3 (len 5)`,
+ `scrub line 2 lhs (len 11)`, `built-in home rule (len 11)` — never by any of its
+ characters, and a hit only by its object: kind, id, the blob's path in history, the
+ byte offset, and for a commit or tag the field (`author`, `committer`, `tagger`,
+ `message`). It prints no byte from the object, because what sits beside a denied
+ name (a surname, the rest of an address) is as private as the name, and every
+ refusal message and path it prints is masked (`[REDACTED]`), so a literal that spans
+ path components (`a/b`) is not printed by a refusal that names a tree path. It ends with
+ `add a rule to ~/.config/archilyzer/source-scrub.txt or drop the file from history,
+ then re-run.`
+5. The tree (`git archive` → `tar -x`, a page per directory; a tracked `index.html`,
+ a tracked `404.html` — Pages would serve it as HTML for every missing path below it —
+ or a symlink is a refusal) and the tarball (`git archive --format=tar.gz -9`).
+6. Limits, then the install: link-safe (`common/bin/_publicFile.ts`), the manifest
+ removed first and written **last**, so a crash leaves the page's empty state,
+ never a manifest over a half-copied tree.
+
+An unchanged `main` with unchanged rules, step version, filter-repo and gitleaks, and
+published files that still match their digest, skips (`[source] up to date at …`;
+`--force` rebuilds). `--check` does everything but the
+install and writes nothing. `--keep-scratch` leaves the scratch clone
+(`ARCHILYZER_SOURCE_SCRATCH`, default the OS temp dir) for a look, minus
+`replace.txt` (the scrub rules), which is always deleted; a scratch root inside the
+checkout or the public dir is refused, since a kept scratch there could be committed
+or published. `archilyzer source audit [<git dir>]` runs the gate alone —
+over the published mirror by default, or over a clone: `archilyzer source audit
+<clone>/.git`.
+
+**The two operator files** live outside the repo, in
+`${ARCHILYZER_CONFIG_DIR:-~/.config/archilyzer}/` (`SOURCE_SCRUB_FILE` and
+`SOURCE_DENYLIST_FILE` override each path; [ENVIRONMENT.md](ENVIRONMENT.md)). They are
+never committed; keep them mode 600. A missing file is a refusal naming it.
+
+- `source-scrub.txt` — git-filter-repo `--replace-text` lines: `lhs==>rhs` (split at
+ the last `==>`), `literal:…`, `regex:…==>…`, `glob:…==>…`; a line with no `==>` is
+ replaced by `***REMOVED***`. Lines whose first non-blank character is `#` are
+ comments (the step drops them — filter-repo itself would treat one as a literal).
+ `<your home dir>==>/home/user` is built in and runs first (a trailing `/` on the home
+ dir is dropped). Rules apply in order. An empty left side is a refusal.
+- `source-denylist.txt` — one literal per line, `i:` in front for any ASCII case;
+ `#` comments.
+- In both, a UTF-8 byte-order mark and CRLF line endings are dropped first: either
+ would silently turn the first rule into one that matches nothing.
+- **Every literal scrub rule's left side is denied too** — its exact bytes, so a rule
+ that stopped matching because its text was mistyped refuses instead of leaking. A
+ NEW spelling in history (another case, another path) is caught only by the denylist:
+ deny the name itself, not just the paths it appears in.
+- The rules' hash (with the step's version) is the skip key and the deploy check, kept
+ in `homepage/.source-publish.json` — beside `public/`, never in it: a published hash
+ of the denylist would confirm a guess at it. For the same reason the manifest does
+ not say how many literals there are.
+
+**A known limit: compressed content is opaque.** The gate is a byte search, so it
+cannot see inside a ZIP member, a PNG's compressed text chunks, a PDF stream or a
+woff2 (the tarball is the exception: it is decompressed). And git-filter-repo never
+scrubs a blob with a NUL in its first 8 KiB, so a binary is audited, never scrubbed.
+At release 12 the review decompressed every such blob in the published history (a
+zip, PNGs and icons) and found no denied literal. A private string inside a binary
+must be removed from history by hand.
+
+**Tools.** `git filter-repo` (`pipx install git-filter-repo`; without it the step runs
+`pipx run --spec git-filter-repo==2.47.0`, which needs the network on first use, and
+without pipx it refuses with the install line). gitleaks is optional: without it the
+secret scan is skipped with a WARNING and the literal audit still runs.
+`archilyzer doctor` has a "source publish" block: which filter-repo would run,
+gitleaks, the two files (rule counts and modes, never contents) and the last publish.
+The mirror's ids are deterministic for a given filter-repo version; an upgrade that
+changes its rewriting changes every id (readers re-clone). The manifest records both
+tools.
+
+**Cloudflare Pages, and the traps it sets.**
+
+- **Never a path segment named `.git`.** wrangler's upload ignore list drops `**/.git`
+ (and `**/node_modules`) *silently*, and Cloudflare's managed rules block `/.git/`
+ requests. The mirror is `archilyzer.git`, which passes both. Extensionless files
+ (`HEAD`, `info/refs`) upload and serve fine.
+- **The caps:** 20,000 files per deployment and 25 MiB per file. The step refuses at
+ **15,000 files** (the rest of the site needs room) and **24 MiB**; packs are split
+ at 20 MB.
+- **`_headers` makes the raw tree plain text.** wrangler's mime map would serve `.ts`
+ as `video/mp2t`, `.mjs` as JavaScript and `.html` as a page on this origin, so
+ `homepage/public/_headers` sets `/source/tree/*` to `text/plain; charset=utf-8` with
+ `nosniff` and `noindex`, then puts back `text/html` for the directory pages and the
+ real types of the few binaries (`.ttf`, `.m4a`, `.zip`, `.onnx`, `.svg` — the SVG
+ under a `default-src 'none'` CSP). **Every matching rule applies, and a header a
+ later rule sets again is APPENDED** (`text/plain; charset=utf-8, text/html;
+ charset=utf-8`), so each of those overrides first detaches the tree's type with
+ `! Content-Type`. Only the ROOT `_headers` is read; the tree's own
+ copies are served as text. Locally, only `wrangler pages dev` honours `_headers` —
+ `next dev` and `serve` do not, and neither serves a directory's `index.html` at
+ `/<dir>/` the way Pages does.
+- **Tailwind and tsc must not see the mirror.** Tailwind v4 scans every file
+ `.gitignore` does not exclude (a binary pack yields "class names" that break the
+ stylesheet), and `homepage/tsconfig.json` includes `**/*.ts`: `/homepage/public/source`
+ is gitignored, and both `public` and `out` (where `next build` copies it) are excluded
+ from the tsconfig. Keep all three. `.dockerignore` leaves the mirror and the skip key
+ out of every image.
+
## Preview deployments
A **preview** is the same built bundle deployed to a branch that is not the Pages
diff --git a/README.md b/README.md
@@ -477,13 +477,25 @@ Channels can sync automatically on a per-channel cadence via a cron heartbeat
Archilyzer is MIT licensed and deliberately **forge-neutral**: there is no `.github/`
directory, no provider-specific CI configuration, and nothing in the build that assumes
-a particular host. However you obtained this tree — a release archive, a mirror, or a
+a particular host. However you obtained this tree — a clone, a release archive, or a
fork on whichever platform — it is a complete, self-contained working copy.
-The project site publishes a dated source snapshot at
+**The canonical public copy is a read-only git mirror on the project site:**
+
+```sh
+git clone https://archilyzer.pages.dev/source/archilyzer.git
+```
+
+It is `main` only, with its whole history, served as static files over git's dumb
+HTTP protocol, and regenerated from the private repository by every homepage build
+(`archilyzer source publish`, which `archilyzer build homepage` runs). Its commit ids
+differ from the private repository's because machine paths are scrubbed on the way
+out; a gate refuses to publish anything that still carries a denied string. Every
+file is browsable raw at [archilyzer.pages.dev/source/tree/](https://archilyzer.pages.dev/source/tree/),
+and the same tree without history is a tarball at
[archilyzer.pages.dev/downloads](https://archilyzer.pages.dev/downloads/), with a
-checksum. `./create-archives.sh` regenerates one, writing a `snapshot.json` sidecar
-(commit, size, SHA-256) beside it.
+`snapshot.json` sidecar (commit, size, SHA-256). How it is built:
+[PUBLISH.md](PUBLISH.md#the-source-mirror-homepage).
See [CONTRIBUTING.md](CONTRIBUTING.md) to work on the code.
diff --git a/SETUP.md b/SETUP.md
@@ -159,8 +159,9 @@ Caveats on native Windows:
- Building the **whisper.cpp / parakeet.cpp** transcription backends is Unix-oriented;
do transcription work under WSL2, or let the container do it (it ships a backend
already built).
-- `create-archives.sh` and the external-cron sync path (`pnpm sync:tick` from cron)
- assume a Unix shell — use WSL2 or Task Scheduler equivalents.
+- The external-cron sync path (`pnpm sync:tick` from cron) assumes a Unix shell — use
+ WSL2 or Task Scheduler equivalents. So does publishing the source mirror
+ (`archilyzer source publish`: git-filter-repo, `tar`).
---
@@ -168,8 +169,20 @@ Caveats on native Windows:
Archilyzer is MIT licensed and **forge-neutral** — there is no provider-specific CI
config or `.github/` directory, and nothing in the build assumes a particular host. Any
-complete copy of the tree works: a clone from wherever the project is mirrored, a fork
-of your own, or the dated source snapshot published by the project site:
+complete copy of the tree works. The canonical public copy is the read-only git mirror
+on the project site — `main` with its whole history:
+
+```sh
+git clone https://archilyzer.pages.dev/source/archilyzer.git archilyzer
+cd archilyzer
+```
+
+`git pull` updates it; nothing takes a push. Its commit ids differ from the private
+repository's (machine paths are scrubbed on the way out), and every homepage build
+regenerates it (`archilyzer source publish`; [PUBLISH.md](PUBLISH.md#the-source-mirror-homepage)).
+
+No git? The same tree without history is a tarball, with a `snapshot.json` sidecar
+(commit, size, SHA-256) beside it:
```sh
curl -LO https://archilyzer.pages.dev/downloads/archilyzer-source.tar.gz
@@ -178,9 +191,7 @@ cd archilyzer
```
The snapshot is a working tree at one commit — no history, no branches, no remote — so
-updating means fetching a newer one. Regenerate a snapshot yourself with
-`./create-archives.sh`, which also writes a `snapshot.json` sidecar (commit, size,
-SHA-256) beside it.
+updating means fetching a newer one.
Once you have a tree, from a clone or a snapshot alike:
diff --git a/common/bin/_cli.test.ts b/common/bin/_cli.test.ts
@@ -314,6 +314,40 @@ test("release show says the latest release, its date and what is pending, and wh
assert.equal(formatAllLine([editor]), null);
});
+// --- archilyzer source (release 12 slice R) --------------------------------
+
+test("usage lists source publish, source audit and build homepage's --no-source", () => {
+ const u = usage(COMMANDS);
+ assert.match(u, /archilyzer source publish\s+\[--force\] \[--check\] \[--keep-scratch\] the scrubbed git mirror/);
+ assert.match(u, /archilyzer source audit\s+\[<git dir>\] the denied-literal gate/);
+ assert.match(u, /archilyzer build homepage\s+\[--no-source\] compose \+ source publish \+ next build/);
+});
+
+test("source publish takes its three booleans and nothing else; none swallows a word", () => {
+ const hit = resolveCommand(COMMANDS, ["source", "publish"]);
+ assert.deepEqual(hit?.command.path, ["source", "publish"]);
+ const parsed = parseArgv(["source", "publish", "--check", "--force", "--keep-scratch"], booleanFlags(COMMANDS));
+ assert.deepEqual(parsed, {
+ positionals: ["source", "publish"],
+ flags: { check: true, force: true, "keep-scratch": true },
+ });
+ assert.equal(argumentProblem(hit!.command, parsed.flags, []), null);
+ assert.match(argumentProblem(hit!.command, { preview: "x" }, [])!, /unknown flag --preview \(accepts --force, --check, --keep-scratch\)/);
+ assert.match(argumentProblem(hit!.command, {}, ["extra"])!, /unexpected argument "extra"/);
+});
+
+test("source audit takes one git dir; build homepage takes --no-source only", () => {
+ const audit = resolveCommand(COMMANDS, ["source", "audit", "/tmp/clone/.git"]);
+ assert.deepEqual(audit?.rest, ["/tmp/clone/.git"]);
+ assert.equal(argumentProblem(audit!.command, {}, audit!.rest), null);
+ assert.match(argumentProblem(audit!.command, {}, ["a", "b"])!, /unexpected argument "b"/);
+ const home = resolveCommand(COMMANDS, ["build", "homepage"]);
+ const parsed = parseArgv(["build", "homepage", "--no-source"], booleanFlags(COMMANDS));
+ assert.deepEqual(parsed, { positionals: ["build", "homepage"], flags: { "no-source": true } });
+ assert.equal(argumentProblem(home!.command, parsed.flags, []), null);
+ assert.match(argumentProblem(home!.command, { force: true }, [])!, /unknown flag --force \(accepts --no-source\)/);
+});
+
// ── passthrough commands (one-core Phase 4 slice 3) ─────────────────────────
test("a passthrough command gets every word after its path, verbatim and unchecked", async () => {
diff --git a/common/bin/archilyzer.ts b/common/bin/archilyzer.ts
@@ -137,15 +137,45 @@ export const COMMANDS: Command[] = [
},
{
path: ["build", "homepage"],
- usage: "compose + next build in homepage/ (reads the index as it stands)",
- run: async () => {
+ usage:
+ "[--no-source] compose + source publish + next build in homepage/ (reads the index as it stands); --no-source removes the published source instead",
+ flags: { "no-source": "boolean" },
+ run: async ({ flags }) => {
const { buildHomepage } = await import("../publish/build");
- const code = await buildHomepage({ signal: interrupted() });
+ const code = await buildHomepage({
+ signal: interrupted(),
+ skipSource: flags["no-source"] === true,
+ });
if (code !== 0) console.error(`build homepage: failed (exit ${code})`);
return code;
},
},
{
+ path: ["source", "publish"],
+ usage:
+ "[--force] [--check] [--keep-scratch] the scrubbed git mirror, raw tree and tarball into homepage/public, behind the denied-literal gate (--check: audit and count, write nothing)",
+ flags: { force: "boolean", check: "boolean", "keep-scratch": "boolean" },
+ run: async ({ flags }) => {
+ const { publishSource } = await import("../publish/source");
+ return publishSource({
+ signal: interrupted(),
+ force: flags.force === true,
+ check: flags.check === true,
+ keepScratch: flags["keep-scratch"] === true,
+ });
+ },
+ },
+ {
+ path: ["source", "audit"],
+ usage:
+ "[<git dir>] the denied-literal gate (+ gitleaks) over a git dir; default the published homepage/public/source/archilyzer.git",
+ maxPositionals: 1,
+ run: async ({ positionals }) => {
+ const { auditSource } = await import("../publish/source");
+ return auditSource({ signal: interrupted(), gitDir: positionals[0] });
+ },
+ },
+ {
path: ["deploy", "site"],
usage:
"<id> [--preview <branch>] ship the site built in export/out to its Pages project (default id: SITE_ID)",
diff --git a/common/bin/doctor.test.ts b/common/bin/doctor.test.ts
@@ -56,6 +56,8 @@ function checkout(): { root: string; bin: string; paths: Paths } {
whisperModel: path.join(root, "models", "ggml-base.en.bin"),
parakeetModel: "",
parakeetCliBin: "parakeet-cli",
+ sourceScrubFile: path.join(root, ".config", "source-scrub.txt"),
+ sourceDenylistFile: path.join(root, ".config", "source-denylist.txt"),
} as unknown as Paths;
return { root, bin, paths };
}
@@ -229,3 +231,57 @@ test("umtool's table and the port block are reported, never failed", async () =>
assert.match(r.checks.find((x) => x.id === "EDITOR_PORT")!.detail, /^3301 in use/);
assert.equal(r.ok, true);
});
+
+test("the source publish block: the tools, the operator files by count and mode (never their contents), the last publish — never a failure", async () => {
+ const c = checkout();
+ const deps = (tools: Awaited<ReturnType<NonNullable<Parameters<typeof collectDoctorReport>[0]["sourceTools"]>>>) => ({
+ env: { PATH: c.bin },
+ paths: c.paths,
+ nodeVersion: "22.0.0",
+ portBlock: async () => null,
+ portInUse: async () => false,
+ umtoolTools: async () => null,
+ sourceTools: async () => tools,
+ });
+ // A clone that never publishes: notes, not warnings.
+ let r = await collectDoctorReport(deps({ filterRepo: null, gitleaks: null }));
+ assert.equal(r.ok, true, renderDoctorReport(r));
+ for (const id of ["filter-repo", "gitleaks", "scrub rules", "denylist", "published"]) {
+ assert.equal(status(r, id), "info", id);
+ }
+ assert.match(r.checks.find((x) => x.id === "filter-repo")!.detail, /install: `pipx install git-filter-repo`/);
+
+ // The operator's files exist (one readable by others), pipx only, a publish.
+ mkdirSync(path.dirname(c.paths.sourceScrubFile), { recursive: true });
+ writeFileSync(c.paths.sourceScrubFile, "# mine\nPLANTED-A==>x\n\nPLANTED-B==>y\n");
+ chmodSync(c.paths.sourceScrubFile, 0o600);
+ writeFileSync(c.paths.sourceDenylistFile, "PLANTED-SECRET\n");
+ chmodSync(c.paths.sourceDenylistFile, 0o644);
+ const pub = path.join(c.root, "homepage", "public", "source");
+ mkdirSync(pub, { recursive: true });
+ writeFileSync(path.join(pub, "manifest.json"), JSON.stringify({
+ version: 1, generatedAt: "2026-09-28T12:00:00.000Z", branch: "main",
+ sourceCommit: "1".repeat(40), mirrorHead: "2".repeat(40), subject: "s", files: 2400, bytes: 1,
+ mirror: { files: 9, bytes: 1, packs: 2 }, tree: { files: 1, dirs: 1, bytes: 1 },
+ tarball: { href: "/downloads/archilyzer-source.tar.gz", bytes: 1, sha256: "3".repeat(64) },
+ audit: { objects: 1, commits: 1, gitleaks: "clean" }, tools: {},
+ }));
+ const before = tree(c.root);
+ r = await collectDoctorReport(deps({ filterRepo: { via: "pipx", version: "1.15.0" }, gitleaks: { version: "8.28.0" } }));
+ assert.equal(r.ok, true, renderDoctorReport(r));
+ assert.equal(status(r, "filter-repo"), "warn");
+ assert.match(r.checks.find((x) => x.id === "filter-repo")!.detail, /pipx run --spec git-filter-repo==2\.47\.0/);
+ assert.equal(status(r, "gitleaks"), "ok");
+ assert.equal(status(r, "scrub rules"), "ok");
+ assert.match(r.checks.find((x) => x.id === "scrub rules")!.detail, /\(2 rules, mode 600\)$/);
+ assert.equal(status(r, "denylist"), "warn");
+ assert.match(r.checks.find((x) => x.id === "denylist")!.detail, /\(1 literal, mode 644\) — readable by others: chmod 600/);
+ assert.match(r.checks.find((x) => x.id === "published")!.detail, /^main 111111111111 as 222222222222, 2026-09-28T12:00:00\.000Z \(2400 files\)/);
+ const text = renderDoctorReport(r);
+ assert.match(text, /\nsource publish\n/);
+ assert.ok(!/PLANTED/.test(text), "the doctor never prints an operator file's contents");
+ assert.deepEqual(tree(c.root), before);
+
+ r = await collectDoctorReport(deps({ filterRepo: { via: "git", version: "a40bce548d2c" }, gitleaks: null }));
+ assert.equal(status(r, "filter-repo"), "ok");
+});
diff --git a/common/bin/doctor.ts b/common/bin/doctor.ts
@@ -70,6 +70,14 @@ export type DoctorDeps = {
portBlock?: () => Promise<{ label: string; ports: Record<string, string> } | null>;
// umtool's report-pipeline table, or null when there is no umtool here.
umtoolTools?: () => Promise<ToolSpec[] | null>;
+ // The source publish's tools. Default: probeSourceTools(env).
+ sourceTools?: () => Promise<SourceTools>;
+};
+
+// Which git-filter-repo `archilyzer source publish` would run, and gitleaks.
+export type SourceTools = {
+ filterRepo: { via: "git"; version: string } | { via: "pipx"; version: string } | null;
+ gitleaks: { version: string } | null;
};
const MIN_NODE = [20, 9, 0] as const; // next 16's engines field
@@ -254,6 +262,50 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
}
}
+ // ── source publish ───────────────────────────────────────────────────────
+ // `archilyzer build homepage` runs it (common/publish/source.ts). Never a
+ // failure: a checkout that never publishes the homepage is not broken. A
+ // WARN is a machine that means to (its operator files exist) and cannot.
+ // The operator files are stat'd and their RULES COUNTED — their contents
+ // are never printed.
+ const SP = "source publish";
+ const src = await import("../publish/source");
+ const tools = await (deps.sourceTools ?? (() => probeSourceTools(env)))();
+ const scrubFile = paths.sourceScrubFile;
+ const denylistFile = paths.sourceDenylistFile;
+ const intends = [scrubFile, denylistFile].some((f) => f && existsSync(f));
+ if (tools.filterRepo?.via === "git") {
+ add(SP, "filter-repo", "ok", `git filter-repo ${tools.filterRepo.version}`);
+ } else if (tools.filterRepo?.via === "pipx") {
+ add(SP, "filter-repo", intends ? "warn" : "info",
+ `not installed; \`pipx run --spec ${src.FILTER_REPO_PIPX_SPEC}\` fetches it at build time (network on first use; pipx ${tools.filterRepo.version}) — \`${src.FILTER_REPO_INSTALL}\` once removes that`);
+ } else {
+ add(SP, "filter-repo", intends ? "warn" : "info",
+ `neither git-filter-repo nor pipx — \`archilyzer build homepage\` refuses; install: \`${src.FILTER_REPO_INSTALL}\``);
+ }
+ add(SP, "gitleaks", tools.gitleaks ? "ok" : "info",
+ tools.gitleaks ? `gitleaks ${tools.gitleaks.version}` : "absent — the gate skips the secret scan with a WARNING (the literal audit still runs)");
+ for (const [id, file, unit] of [
+ ["scrub rules", scrubFile, "rule"],
+ ["denylist", denylistFile, "literal"],
+ ] as const) {
+ const st = file ? statOrNull(file) : null;
+ if (!file || !st) {
+ add(SP, id, "info", `${file ?? "(unset)"} is missing — \`source publish\` (and so \`build homepage\`) refuses until it exists (PUBLISH.md, "The source mirror")`);
+ continue;
+ }
+ const count = (readOrNull(file) ?? "").split("\n").filter((l) => l.trim() && !l.trim().startsWith("#")).length;
+ const mode = st.mode & 0o777;
+ const open = (mode & 0o077) !== 0;
+ add(SP, id, open ? "warn" : "ok",
+ `${file} (${count} ${unit}${count === 1 ? "" : "s"}, mode ${mode.toString(8)})${open ? " — readable by others: chmod 600" : ""}`);
+ }
+ const publicDir = env.HOMEPAGE_PUBLIC_DIR || path.join(root, "homepage", "public");
+ const published = await src.readPublishedManifest(publicDir);
+ add(SP, "published", "info", published
+ ? `main ${published.sourceCommit.slice(0, 12)} as ${published.mirrorHead.slice(0, 12)}, ${published.generatedAt} (${published.files} files)`
+ : `nothing published in ${path.join(publicDir, "source")}`);
+
// ── ports ────────────────────────────────────────────────────────────────
const P = "ports";
const block = await (deps.portBlock ?? (() => worktreePortBlock(root)))();
@@ -396,6 +448,27 @@ async function worktreePortBlock(
}
}
+// What `source publish` would run, by version flags only: `git filter-repo
+// --version` answering 0 is an installed filter-repo; otherwise pipx's own
+// version (never `pipx run`, which would download). gitleaks likewise.
+async function probeSourceTools(env: NodeJS.ProcessEnv): Promise<SourceTools> {
+ const version = async (bin: string, args: string[]): Promise<string | null> => {
+ try {
+ const { stdout, stderr } = await execFileP(bin, args, { env, timeout: 10_000 });
+ return (stdout || stderr).trim().split("\n")[0] ?? "";
+ } catch {
+ return null;
+ }
+ };
+ const git = await version("git", ["filter-repo", "--version"]);
+ const pipx = git === null ? await version("pipx", ["--version"]) : null;
+ const leaks = await version("gitleaks", ["version"]);
+ return {
+ filterRepo: git !== null ? { via: "git", version: git } : pipx !== null ? { via: "pipx", version: pipx } : null,
+ gitleaks: leaks !== null ? { version: leaks } : null,
+ };
+}
+
// In use = something accepts a TCP connection on 127.0.0.1. Never binds.
function tcpPortInUse(port: number): Promise<boolean> {
return new Promise((resolve) => {
diff --git a/common/lib/envVars.ts b/common/lib/envVars.ts
@@ -53,7 +53,7 @@ const paths = (name: string, def: string, doc: string): EnvVarDecl => ({
const DECLARED: EnvVarDecl[] = [
// ── paths: getPaths() ──────────────────────────────────────────────────
- paths("TRANSCRIPTS_DIR", "`<repo>/transcripts`", "The corpus: channels, sites, the LMDB index, job logs, the saved-video store."),
+ paths("TRANSCRIPTS_DIR", "`<repo>/transcripts`", "The corpus: channels, sites, the LMDB index, job logs, the saved-video store. umtool reads `<it>/channels` too, when its own `CHANNELS_DIR` is unset."),
paths("SAVED_VIDEOS_DIR", "`<TRANSCRIPTS_DIR>/saved-videos`", "The persisted source-video store, when it should live on another disk."),
paths("SITES_DIR", "`<TRANSCRIPTS_DIR>/sites`", "Per-site config (`<id>/site.json`, every key in [SITE.md](SITE.md)) and the homepage's `_homepage/`."),
paths("SETTINGS_FILE", "`<repo>/settings.json`", "The settings file (every key in [SETTINGS.md](SETTINGS.md))."),
@@ -80,6 +80,10 @@ const DECLARED: EnvVarDecl[] = [
paths("GALLERY_DL_BIN", "`gallery-dl` on PATH", "The X/Twitter post fetcher, for social channels."),
paths("OLLAMA_URL", "`http://127.0.0.1:11434`", "The local ollama server, the local digest and attribution engine."),
paths("CLAUDE_BIN", "`claude` on PATH", "The `claude` CLI, driving the opt-in metered digest lane."),
+ paths("ARCHILYZER_CONFIG_DIR", "`~/.config/archilyzer`", "The operator's private config dir, outside the repo: the two inputs of `archilyzer source publish` below. Never committed."),
+ paths("SOURCE_SCRUB_FILE", "`<ARCHILYZER_CONFIG_DIR>/source-scrub.txt`", "git-filter-repo `lhs==>rhs` rules applied to file contents AND commit messages when the source mirror is generated (`<home dir>==>/home/user` is built in and runs first). Every rule's left side is also denied. See [PUBLISH.md](PUBLISH.md)."),
+ paths("SOURCE_DENYLIST_FILE", "`<ARCHILYZER_CONFIG_DIR>/source-denylist.txt`", "Literals the published source must never contain, one per line (`i:` = any case). One hit anywhere in the mirror, the tree or the tarball refuses the publish."),
+ paths("ARCHILYZER_SOURCE_SCRATCH", "the OS temp dir", "Where `source publish` makes its scratch clone and stage (removed afterwards unless `--keep-scratch`)."),
// ── runtime ────────────────────────────────────────────────────────────
{ name: "WORKER_TOKEN", audience: "runtime", default: "unset (both surfaces off)", readBy: "common/lib/workerToken.ts, scripts/archilyzer-ops.mjs, mcp/src/fetchClip.ts", doc: "Bearer token for the remote-worker API and for `/api/ops/*` (`pnpm ops`, the MCP's `fetch_clip`). Set the same value on both ends." },
@@ -132,7 +136,7 @@ const DECLARED: EnvVarDecl[] = [
{ name: "INSTANCE_MODE", audience: "internal", default: "a site", readBy: "export/app/lib/mode.ts, common/lib/archive/contract.ts", doc: "`hub` makes the export build the hub. Set by `archilyzer build hub`." },
{ name: "BUILD_ARCHIVES", audience: "internal", default: "on", readBy: "common/bin/compose-site.ts, common/bin/build-archives.ts", doc: "`0` skips archive-zip generation for one build (`--skip-archives`)." },
{ name: "ARCHIVES_READONLY", audience: "internal", default: "off", readBy: "common/bin/compose-site.ts", doc: "`1` inside a docker-mode build container: materialize archives, never write the shared cache." },
- { name: "HOMEPAGE_PUBLIC_DIR", audience: "internal", default: "`<repo>/homepage/public`", readBy: "common/bin/compose-homepage.ts", doc: "Where `compose homepage` writes." },
+ { name: "HOMEPAGE_PUBLIC_DIR", audience: "internal", default: "`<repo>/homepage/public`", readBy: "common/bin/compose-homepage.ts, common/publish/source.ts", doc: "Where `compose homepage` and `source publish` write." },
// ── docker: the container's set ────────────────────────────────────────
{ name: "ARCHILYZER_TRANSCRIBER", audience: "docker", default: "baked per image target (`whisper-cpp` in `runtime`)", readBy: "docker/entrypoint.sh", doc: "`whisper-cpp` or `parakeet`: which worker the first boot seeds and which model it fetches." },
@@ -181,6 +185,7 @@ const DECLARED: EnvVarDecl[] = [
{ name: "E2E_RETRIES", audience: "test", default: "`0`", readBy: "scripts/run-sharded-e2e.mjs", doc: "Retries per shard (`--retries N` wins); 0 keeps a sharded run comparable to a serial one." },
{ name: "E2E_IMAGE", audience: "test", default: "`yt-dlp-transcript-browser-e2e`", readBy: "scripts/run-sharded-e2e.mjs", doc: "The sharded e2e run's image tag." },
{ name: "E2E_SKIP_BUILD", audience: "test", default: "off", readBy: "scripts/run-sharded-e2e.mjs", doc: "`1` reuses the sharded e2e image instead of rebuilding it (`--no-build`)." },
+ { name: "E2E_EXPECT_SOURCE", audience: "test", default: "off (both states pass)", readBy: "homepage/e2e/source.spec.ts", doc: "`1` makes the homepage suite's `/source/` specs fail on the empty state; a gate that ran `archilyzer source publish` first sets it." },
{ name: "E2E_HOMEPAGE_SUMMARY_FILE", audience: "test", default: "`homepage/public/homepage-summary.json`", readBy: "homepage/app/lib/summary.ts", doc: "The synthetic summary the homepage's e2e dev server reads; set by `homepage/playwright.config.ts`, ignored by a production build." },
];
diff --git a/common/lib/paths.ts b/common/lib/paths.ts
@@ -158,6 +158,16 @@ export type Paths = {
// settings.digest.remoteEnabled). Not bundled; install it separately and point
// CLAUDE_BIN at it if it isn't on PATH.
claudeBin: string;
+ // The operator's PRIVATE config dir, outside the repo (~/.config/archilyzer
+ // by default). Holds the two inputs of `archilyzer source publish`
+ // (common/publish/source.ts), which are never committed:
+ // sourceScrubFile — git-filter-repo `lhs==>rhs` rules for the mirror
+ // sourceDenylistFile — literals the published source must never contain
+ configDir: string;
+ sourceScrubFile: string;
+ sourceDenylistFile: string;
+ // Where `source publish` makes its scratch clone (removed afterwards).
+ sourceScratchDir: string;
};
let cached: Paths | null = null;
@@ -179,6 +189,9 @@ export function getPaths(): Paths {
const exportSharedDir = path.join(exportIndexDir, "shared");
const sitesDir = process.env.SITES_DIR ?? path.join(transcriptsDir, "sites");
const homepageDir = path.join(sitesDir, "_homepage");
+ const configDir =
+ process.env.ARCHILYZER_CONFIG_DIR ??
+ path.join(os.homedir(), ".config", "archilyzer");
cached = {
monorepoRoot,
transcriptsDir,
@@ -264,6 +277,13 @@ export function getPaths(): Paths {
"",
),
claudeBin: process.env.CLAUDE_BIN ?? "claude",
+ configDir,
+ sourceScrubFile:
+ process.env.SOURCE_SCRUB_FILE ?? path.join(configDir, "source-scrub.txt"),
+ sourceDenylistFile:
+ process.env.SOURCE_DENYLIST_FILE ??
+ path.join(configDir, "source-denylist.txt"),
+ sourceScratchDir: process.env.ARCHILYZER_SOURCE_SCRATCH ?? os.tmpdir(),
};
return cached;
}
diff --git a/common/lib/project.ts b/common/lib/project.ts
@@ -34,8 +34,9 @@ export const PROJECT_URL = "https://archilyzer.pages.dev";
export const PROJECT_TAGLINE = "Self-hosted, searchable video-transcript archives.";
-// Where a visitor gets the source. A dated snapshot tarball — there is no
-// public git repository (see homepage/app/downloads).
+// Where a visitor gets the source as a tarball. The canonical public copy is
+// the read-only git mirror on the same site (lib/sourceManifest.ts CLONE_URL,
+// homepage/app/source); this is its no-git alternative.
export const PROJECT_DOWNLOADS_URL = `${PROJECT_URL}/downloads/`;
// The `generator` string stamped into corpus.json and llms.txt, so anything
diff --git a/common/lib/sourceManifest.test.ts b/common/lib/sourceManifest.test.ts
@@ -0,0 +1,50 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { parseSourceManifest, TARBALL_HREF } from "./sourceManifest";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// The homepage believes a manifest only through parseSourceManifest, and the
+// /source/ page reads its numbers during `next build`: a malformed or
+// foreign one must be the empty state (null), never a crash (review L10).
+
+const good = () => ({
+ version: 1,
+ generatedAt: "2026-09-28T12:00:00.000Z",
+ branch: "main",
+ sourceCommit: "1".repeat(40),
+ mirrorHead: "2".repeat(40),
+ subject: "s",
+ files: 10,
+ bytes: 100,
+ mirror: { files: 9, bytes: 80, packs: 2 },
+ tree: { files: 5, dirs: 2, bytes: 20 },
+ tarball: { href: TARBALL_HREF, bytes: 7, sha256: "3".repeat(64) },
+ cloneUrl: "x",
+ treeHref: "/source/tree/",
+ audit: { objects: 3, commits: 1, gitleaks: "clean" },
+ tools: { git: "2", filterRepo: "x" },
+});
+
+test("a well-formed version-1 manifest parses", () => {
+ assert.deepEqual(parseSourceManifest(good()), good());
+});
+
+test("a wrong version, a bad id or sha, or any number the page reads that is not one is null", () => {
+ const broken: Array<[string, (m: ReturnType<typeof good>) => unknown]> = [
+ ["version 2", (m) => ({ ...m, version: 2 })],
+ ["short mirror head", (m) => ({ ...m, mirrorHead: "abc" })],
+ ["upper-case sha", (m) => ({ ...m, tarball: { ...m.tarball, sha256: "A".repeat(64) } })],
+ ["no tree", (m) => ({ ...m, tree: undefined })],
+ ["tree.files a string", (m) => ({ ...m, tree: { ...m.tree, files: "5" } })],
+ ["mirror.packs missing", (m) => ({ ...m, mirror: { files: 1, bytes: 1 } })],
+ ["mirror.bytes NaN", (m) => ({ ...m, mirror: { ...m.mirror, bytes: Number.NaN } })],
+ ["negative files", (m) => ({ ...m, files: -1 })],
+ ["tarball.bytes missing", (m) => ({ ...m, tarball: { href: TARBALL_HREF, sha256: "3".repeat(64) } })],
+ ["unparsable date", (m) => ({ ...m, generatedAt: "yesterday" })],
+ ["no audit", (m) => ({ ...m, audit: null })],
+ ];
+ for (const [what, make] of broken) assert.equal(parseSourceManifest(make(good())), null, what);
+ for (const junk of [null, 1, "x", [], {}]) assert.equal(parseSourceManifest(junk), null, JSON.stringify(junk));
+});
diff --git a/common/lib/sourceManifest.ts b/common/lib/sourceManifest.ts
@@ -0,0 +1,105 @@
+// The published source's contract: where `archilyzer source publish`
+// (common/publish/source.ts) puts things on the project site, and the shape of
+// the manifest it writes LAST, beside them (public/source/manifest.json). The
+// homepage's /source/ page (homepage/app/lib/source.ts) reads the same shape.
+//
+// A leaf: it imports only lib/project.ts (itself import-free), so the homepage
+// can pull it into a server component without dragging node: modules along.
+//
+// The mirror is a DUMB-HTTP git repository: static files only (HEAD,
+// info/refs, objects/info/packs, the packs), which `git clone` reads with no
+// server-side git at all. Its directory is `archilyzer.git`, never a path
+// segment named `.git` — wrangler's upload ignore list drops `**/.git`
+// silently, and Cloudflare's managed rules block `/.git/` paths.
+
+import { PROJECT_URL } from "./project";
+
+export const SOURCE_MANIFEST_VERSION = 1;
+
+/** The mirror's directory under /source/. */
+export const MIRROR_DIR = "archilyzer.git";
+
+/** What a reader types: `git clone <CLONE_URL>`. */
+export const CLONE_URL = `${PROJECT_URL}/source/${MIRROR_DIR}`;
+
+/** The raw tree's generated index. */
+export const TREE_HREF = "/source/tree/";
+
+/** The manifest's public path. */
+export const SOURCE_MANIFEST_HREF = "/source/manifest.json";
+
+/**
+ * The tarball's public path — the one the Downloads page has always linked
+ * (homepage/app/lib/snapshot.ts SNAPSHOT_HREF). Stable by design: the date and
+ * commit live in snapshot.json beside it.
+ */
+export const TARBALL_HREF = "/downloads/archilyzer-source.tar.gz";
+
+export type SourceManifest = {
+ version: typeof SOURCE_MANIFEST_VERSION;
+ generatedAt: string;
+ branch: "main";
+ // The PRIVATE repository's main, which the mirror reflects. Its history is
+ // never rewritten; the mirror is generated from a fresh clone of it.
+ sourceCommit: string;
+ // The mirror's main: a different id for the same history, because paths
+ // were scrubbed on the way out.
+ mirrorHead: string;
+ // The mirror head's subject line (audited like every other commit).
+ subject: string;
+ // Everything else the step published: the mirror, the tree and its
+ // indexes, the tarball and snapshot.json (not this manifest itself).
+ files: number;
+ bytes: number;
+ mirror: { files: number; bytes: number; packs: number };
+ // The tracked files of main, extracted; `files`/`bytes` do not count the
+ // generated index.html pages, `dirs` counts the directories (each has one).
+ tree: { files: number; dirs: number; bytes: number };
+ tarball: { href: string; bytes: number; sha256: string };
+ cloneUrl: string;
+ treeHref: string;
+ // What the gate read: how many git objects (commits among them), and
+ // whether gitleaks ran. NOT how many literals it searched for: `plans/`
+ // publishes which ones the plan put in the files, so the count would say
+ // whether the operator added private ones.
+ audit: {
+ objects: number;
+ commits: number;
+ gitleaks: "clean" | "skipped";
+ };
+ // The tools that made it (a filter-repo upgrade may change mirrorHead).
+ tools: { git: string; filterRepo: string };
+};
+
+const HEX40 = /^[0-9a-f]{40}$/;
+const HEX64 = /^[0-9a-f]{64}$/;
+
+/**
+ * The manifest, or null when `value` is not one this code can trust: version
+ * 1, 40-hex commit ids, a 64-hex tarball sha, and a finite number wherever a
+ * page reads one. A malformed or foreign manifest is the page's empty state,
+ * never a crash in `next build`. The homepage additionally requires the files
+ * it describes to be present (homepage/app/lib/source.ts).
+ */
+export function parseSourceManifest(value: unknown): SourceManifest | null {
+ if (!value || typeof value !== "object") return null;
+ const m = value as Partial<SourceManifest>;
+ const num = (x: unknown) => typeof x === "number" && Number.isFinite(x) && x >= 0;
+ const obj = (x: unknown): x is Record<string, unknown> => !!x && typeof x === "object";
+ if (m.version !== SOURCE_MANIFEST_VERSION) return null;
+ if (typeof m.mirrorHead !== "string" || !HEX40.test(m.mirrorHead)) return null;
+ if (typeof m.sourceCommit !== "string" || !HEX40.test(m.sourceCommit)) return null;
+ if (typeof m.subject !== "string" || typeof m.generatedAt !== "string") return null;
+ if (Number.isNaN(Date.parse(m.generatedAt))) return null;
+ if (!num(m.files) || !num(m.bytes)) return null;
+ if (!obj(m.tarball) || typeof m.tarball.sha256 !== "string" || !HEX64.test(m.tarball.sha256)) {
+ return null;
+ }
+ if (!num(m.tarball.bytes) || typeof m.tarball.href !== "string") return null;
+ if (!obj(m.mirror) || !num(m.mirror.files) || !num(m.mirror.bytes) || !num(m.mirror.packs)) {
+ return null;
+ }
+ if (!obj(m.tree) || !num(m.tree.files) || !num(m.tree.dirs) || !num(m.tree.bytes)) return null;
+ if (!obj(m.audit)) return null;
+ return m as SourceManifest;
+}
diff --git a/common/publish/build.test.ts b/common/publish/build.test.ts
@@ -1,8 +1,12 @@
import { test } from "node:test";
import assert from "node:assert/strict";
+import { chmodSync, existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
import type { Paths } from "../lib/paths";
import {
HOMEPAGE_PAGES_PROJECT,
+ buildHomepage,
buildHubSteps,
buildSiteSteps,
hubProjectProblem,
@@ -143,6 +147,119 @@ test("homepageOutDir is homepage/out of the checkout", () => {
assert.equal(homepageOutDir({ monorepoRoot: "/repo" } as Paths), "/repo/homepage/out");
});
+// buildHomepage's order — compose, the source step, `next build` — over a temp
+// homepage/ whose `compose` is `true` and whose `next` only says it ran.
+test("buildHomepage: the source step sits between compose and next build; a refusal stops the build; --no-source clears instead", async () => {
+ const root = mkdtempSync(path.join(os.tmpdir(), "build-homepage-"));
+ try {
+ const home = path.join(root, "homepage");
+ mkdirSync(path.join(home, "node_modules", ".bin"), { recursive: true });
+ writeFileSync(
+ path.join(home, "package.json"),
+ JSON.stringify({ name: "fake-homepage", private: true, scripts: { compose: "echo COMPOSED" } }),
+ );
+ const next = path.join(home, "node_modules", ".bin", "next");
+ writeFileSync(next, "#!/bin/sh\necho NEXT RAN\n");
+ chmodSync(next, 0o755);
+ const fakePaths = { monorepoRoot: root } as Paths;
+ // The last good build's source, in out/ (M1: a refusal must take it out).
+ const out = path.join(home, "out");
+ const plantOut = () => {
+ mkdirSync(path.join(out, "source", "archilyzer.git"), { recursive: true });
+ mkdirSync(path.join(out, "downloads"), { recursive: true });
+ writeFileSync(path.join(out, "source", "manifest.json"), "{}");
+ writeFileSync(path.join(out, "downloads", "archilyzer-source.tar.gz"), "old");
+ writeFileSync(path.join(out, "downloads", "snapshot.json"), "{}");
+ writeFileSync(path.join(out, "downloads", "index.html"), "<p>the page</p>");
+ };
+
+ const run = async (o: Parameters<typeof buildHomepage>[0]) => {
+ const logs: string[] = [];
+ const code = await buildHomepage({ paths: fakePaths, onLog: (l) => logs.push(l), ...o });
+ return { code, logs: logs.join("\n") };
+ };
+ const calls: string[] = [];
+
+ // A refusal (1) fails the build before next build runs, and takes the
+ // last build's source out of homepage/out, so a deploy-only ships none.
+ plantOut();
+ let r = await run({
+ publishSource: async (p) => {
+ calls.push(`publish:${p.paths === fakePaths}:${p.noRepository}`);
+ return 1;
+ },
+ });
+ assert.equal(r.code, 1);
+ assert.match(r.logs, /COMPOSED/);
+ assert.doesNotMatch(r.logs, /NEXT RAN/);
+ assert.deepEqual(calls, ["publish:true:empty"], "a checkout with no git builds with the empty state");
+ assert.ok(!existsSync(path.join(out, "source")), "out/source is withdrawn");
+ assert.ok(!existsSync(path.join(out, "downloads", "archilyzer-source.tar.gz")));
+ assert.ok(!existsSync(path.join(out, "downloads", "snapshot.json")));
+ assert.ok(existsSync(path.join(out, "downloads", "index.html")), "the page itself stays");
+ assert.match(r.logs, /removed from homepage\/out too/);
+
+ // Published: next build follows.
+ r = await run({ publishSource: async () => (calls.push("publish"), 0) });
+ assert.equal(r.code, 0, r.logs);
+ assert.ok(r.logs.indexOf("COMPOSED") < r.logs.indexOf("NEXT RAN"));
+
+ // --no-source: the publish is never called; the clear is.
+ calls.length = 0;
+ r = await run({
+ skipSource: true,
+ publishSource: async () => {
+ throw new Error("--no-source must not publish");
+ },
+ clearSource: async () => {
+ calls.push("clear");
+ },
+ });
+ assert.equal(r.code, 0, r.logs);
+ assert.deepEqual(calls, ["clear"]);
+ assert.match(r.logs, /NEXT RAN/);
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
+// M1: a deploy-only ships homepage/out as the last build left it. Its source
+// ships only with a record that it was audited under today's rules (the
+// positive case is source.test.ts' round trip, over publishedSourceProblem).
+test("deployHomepage refuses an out/ whose source has no record of the rules it was audited under", async () => {
+ const root = mkdtempSync(path.join(os.tmpdir(), "deploy-homepage-"));
+ try {
+ const out = path.join(root, "homepage", "out");
+ mkdirSync(path.join(out, "source"), { recursive: true });
+ writeFileSync(path.join(out, "index.html"), "<p>home</p>");
+ writeFileSync(path.join(out, "source", "index.html"), "<p>the /source page</p>");
+ writeFileSync(
+ path.join(out, "source", "manifest.json"),
+ JSON.stringify({
+ version: 1, generatedAt: "2026-09-28T12:00:00.000Z", branch: "main", sourceCommit: "1".repeat(40),
+ mirrorHead: "2".repeat(40), subject: "s", files: 1, bytes: 1, mirror: { files: 1, bytes: 1, packs: 1 },
+ tree: { files: 1, dirs: 1, bytes: 1 }, tarball: { href: "/downloads/archilyzer-source.tar.gz", bytes: 1, sha256: "3".repeat(64) },
+ audit: { objects: 1, commits: 1, gitleaks: "clean" }, tools: {},
+ }),
+ );
+ const p = { monorepoRoot: root } as Paths;
+ await assert.rejects(
+ deployHomepage({ paths: p, previewBranch: "r12-source" }),
+ /homepage\/out's source has no record of the rules it was audited under — run `archilyzer build homepage`/,
+ );
+ // A half-removed source (no manifest) refuses too…
+ rmSync(path.join(out, "source", "manifest.json"));
+ mkdirSync(path.join(out, "source", "archilyzer.git"));
+ await assert.rejects(deployHomepage({ paths: p }), /without a valid manifest — run `archilyzer build homepage`/);
+ // …and so does a build whose source step refused (buildHomepage took
+ // out/source away, the page with it).
+ rmSync(path.join(out, "source"), { recursive: true });
+ await assert.rejects(deployHomepage({ paths: p }), /has no \/source page \(its source step refused/);
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
+
test("deployHomepage refuses a bad preview branch before it looks for a build", async () => {
const noBuild = { monorepoRoot: "/nonexistent-repo" } as Paths;
await assert.rejects(
diff --git a/common/publish/build.ts b/common/publish/build.ts
@@ -969,11 +969,48 @@ export async function composeHomepage(opts: PublishOpts = {}): Promise<number> {
]);
}
-/** composeHomepage, then `next build` in homepage/ (→ homepage/out). */
-export async function buildHomepage(opts: PublishOpts = {}): Promise<number> {
+/**
+ * composeHomepage, then `archilyzer source publish` (the git mirror, the raw
+ * tree and the tarball into homepage/public — source.ts), then `next build` in
+ * homepage/ (→ homepage/out).
+ *
+ * A source REFUSAL fails the build before `next build`, and withdraws the
+ * source twice over: the step itself removes the last publish from
+ * homepage/public, and this removes the last BUILD's copy from homepage/out
+ * (`out/source`, the tarball, `snapshot.json`), so a deploy-only cannot ship
+ * a source today's rules were never applied to (deployHomepage checks too).
+ * The audit report is in the log.
+ *
+ * `skipSource` (the CLI's `--no-source`) REMOVES the published source instead:
+ * a copy left from an earlier publish was audited against the rules of ITS
+ * day, so a build that skips the gate ships none — the pages show their empty
+ * states. A checkout with no git repository (the docker runtime, a tarball
+ * install) builds with the empty state too. `publishSource` / `clearSource`
+ * are the test's seams.
+ */
+export async function buildHomepage(
+ opts: PublishOpts & {
+ skipSource?: boolean;
+ publishSource?: (o: PublishOpts & { noRepository?: "refuse" | "empty" }) => Promise<number>;
+ clearSource?: (o: PublishOpts) => Promise<void>;
+ } = {},
+): Promise<number> {
const { paths, onLog, signal } = resolved(opts);
const code = await composeHomepage({ paths, onLog, signal });
if (code !== 0) return code;
+ // Loaded lazily: the source step pulls nothing the other publish entry
+ // points need, and it imports this module's types.
+ if (opts.skipSource) {
+ const clear = opts.clearSource ?? (await import("./source")).clearPublishedSource;
+ await clear({ paths, onLog, signal });
+ } else {
+ const publish = opts.publishSource ?? (await import("./source")).publishSource;
+ const sourceCode = await publish({ paths, onLog, signal, noRepository: "empty" });
+ if (sourceCode !== 0 || signal.aborted) {
+ await withdrawBuiltSource(paths, onLog);
+ return sourceCode || 1;
+ }
+ }
return runSteps(onLog, signal, [
{
command: "pnpm",
@@ -984,6 +1021,22 @@ export async function buildHomepage(opts: PublishOpts = {}): Promise<number> {
]);
}
+// The last build's copy of the source, out of homepage/out: a refused source
+// step must not leave it for a deploy-only to ship.
+async function withdrawBuiltSource(paths: Paths, onLog: (line: string) => void): Promise<void> {
+ const out = homepageOutDir(paths);
+ const had =
+ existsSync(path.join(out, "source", "manifest.json")) ||
+ existsSync(path.join(out, "downloads", "archilyzer-source.tar.gz"));
+ await rm(path.join(out, "source", "manifest.json"), { force: true });
+ await rm(path.join(out, "source"), { recursive: true, force: true });
+ await rm(path.join(out, "downloads", "archilyzer-source.tar.gz"), { force: true });
+ await rm(path.join(out, "downloads", "snapshot.json"), { force: true });
+ if (had) {
+ onLog("[source] the last build's source was removed from homepage/out too, so a deploy-only ships none.\n");
+ }
+}
+
/**
* The wrangler argv (after `pnpm dlx`) for a homepage deploy of `outDir`: the
* production branch `main` — what homepage/package.json's hardcoded `deploy`
@@ -1021,6 +1074,10 @@ export async function deployHomepage(
if (!existsSync(path.join(outDir, "index.html"))) {
throw new Error("homepage/out holds no build — run archilyzer build homepage first");
}
+ // The source in out/ ships only if it was audited under TODAY's rules, of
+ // today's main (source.ts). An out/ with no source deploys as before.
+ const sourceProblem = await (await import("./source")).publishedSourceProblem(paths, outDir);
+ if (sourceProblem) throw new Error(sourceProblem);
if (branch) onLog(`=== Deploy homepage (preview "${branch}") ===\n`);
const code = await runPagesDeployIntoLog(onLog, signal, {
outDir,
diff --git a/common/publish/source.test.ts b/common/publish/source.test.ts
@@ -0,0 +1,603 @@
+import { test, after, before } from "node:test";
+import assert from "node:assert/strict";
+import { createHash } from "node:crypto";
+import { execFileSync } from "node:child_process";
+import {
+ chmodSync,
+ cpSync,
+ existsSync,
+ lstatSync,
+ mkdirSync,
+ mkdtempSync,
+ readdirSync,
+ readFileSync,
+ rmSync,
+ statSync,
+ symlinkSync,
+ writeFileSync,
+} from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import type { Paths } from "../lib/paths";
+import { CLONE_URL, MIRROR_DIR, TARBALL_HREF, TREE_HREF } from "../lib/sourceManifest";
+import {
+ MAX_FILES,
+ MAX_FILE_BYTES,
+ NO_REPOSITORY,
+ SOURCE_STEP_VERSION,
+ SourceRefusal,
+ clearPublishedSource,
+ limitProblem,
+ loadSourceRules,
+ parseScrubRules,
+ publishSource,
+ publishedSourceProblem,
+ gitleaksIdentity,
+ resolveFilterRepo,
+ rulesHashOf,
+ scratchRootProblem,
+ sourceDigest,
+ type SourcePublishOpts,
+} from "./source";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// `archilyzer source publish` (source.ts) over temp repos. The two tests that
+// rewrite history need git-filter-repo; they SKIP when resolveFilterRepo()
+// cannot find one (say so in a gate: the release gate requires they RAN).
+// Every repo, public dir and scratch dir is under the OS temp dir, and the git
+// variables a hook or a wrapper might export are cleared first.
+for (const key of ["GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE", "GIT_PREFIX"]) {
+ delete process.env[key];
+}
+
+const TMP = mkdtempSync(path.join(os.tmpdir(), "source-publish-"));
+after(() => rmSync(TMP, { recursive: true, force: true }));
+
+// Planted, never real: the home dir the built-in rule scrubs, and the paths.
+const HOME = "/home/not-a-real-user";
+const PLANTED = "plantedhome";
+const SECRET = "plantedsecret";
+
+let filterRepoProblem: string | null = null;
+before(async () => {
+ try {
+ await resolveFilterRepo({});
+ } catch (err) {
+ filterRepoProblem = (err as Error).message;
+ }
+});
+
+let n = 0;
+function dir(name: string): string {
+ const d = path.join(TMP, `${name}-${n++}`);
+ mkdirSync(d, { recursive: true });
+ return d;
+}
+
+function gitIn(cwd: string, ...args: string[]): string {
+ return execFileSync("git", args, { cwd, stdio: "pipe" }).toString().trim();
+}
+
+// A repo with `main` of three commits: a planted path in one blob and one
+// message, and the names the tree pages must encode.
+function sourceRepo(extra: Record<string, string> = {}): string {
+ const repo = dir("src");
+ gitIn(repo, "init", "-q", "-b", "main");
+ gitIn(repo, "config", "user.name", "source test");
+ gitIn(repo, "config", "user.email", "source@example.invalid");
+ gitIn(repo, "config", "commit.gpgsign", "false");
+ const put = (files: Record<string, string>, message: string) => {
+ for (const [f, text] of Object.entries(files)) {
+ mkdirSync(path.dirname(path.join(repo, f)), { recursive: true });
+ writeFileSync(path.join(repo, f), text);
+ }
+ gitIn(repo, "add", "-A");
+ gitIn(repo, "commit", "-q", "-m", message);
+ };
+ put(
+ {
+ "README.md": "hello\n",
+ "app/[slug]/page.tsx": "export default 1;\n",
+ "fonts/Archivo[wdth,wght].ttf": "not really a font\n",
+ },
+ "first",
+ );
+ put({ "notes.txt": `data lives at /srv/${PLANTED}/x\n`, ...extra }, "add notes");
+ put({ "README.md": "hello again\n" }, `moved from /srv/${PLANTED}`);
+ return repo;
+}
+
+function operatorFiles(scrub: string, deny: string): { scrubFile: string; denylistFile: string } {
+ const d = dir("config");
+ writeFileSync(path.join(d, "source-scrub.txt"), scrub);
+ writeFileSync(path.join(d, "source-denylist.txt"), deny);
+ return { scrubFile: path.join(d, "source-scrub.txt"), denylistFile: path.join(d, "source-denylist.txt") };
+}
+
+// A checkout of its own (never TMP itself, which holds the scratch root: the
+// step refuses a scratch dir inside the checkout).
+function opts(repo: string, files: { scrubFile: string; denylistFile: string }, logs: string[], extra: Partial<SourcePublishOpts> = {}): SourcePublishOpts {
+ return {
+ paths: { monorepoRoot: dir("checkout") } as Paths,
+ sourceRepo: path.join(repo, ".git"),
+ publicDir: path.join(dir("site"), "public"),
+ scratchRoot: path.join(TMP, "scratch"),
+ gitleaks: null,
+ homeDir: HOME,
+ onLog: (l) => logs.push(l),
+ now: () => new Date("2026-09-28T12:00:00.000Z"),
+ ...files,
+ ...extra,
+ };
+}
+
+const sha256 = (file: string) => createHash("sha256").update(readFileSync(file)).digest("hex");
+
+// A manifest the page would believe, for the tests that fake a publish.
+function fakeManifest(sourceCommit: string, mirrorHead: string, sha = "b".repeat(64)): string {
+ return JSON.stringify({
+ version: 1, generatedAt: "2026-09-28T12:00:00.000Z", branch: "main", sourceCommit, mirrorHead, subject: "s",
+ files: 1, bytes: 1, mirror: { files: 1, bytes: 1, packs: 1 }, tree: { files: 1, dirs: 1, bytes: 1 },
+ tarball: { href: TARBALL_HREF, bytes: 7, sha256: sha }, cloneUrl: CLONE_URL, treeHref: TREE_HREF,
+ audit: { objects: 1, commits: 1, gitleaks: "skipped" }, tools: { git: "x", filterRepo: "x" },
+ });
+}
+
+test("scrub rules: the home-dir rule first, comments and blanks dropped, every literal left side denied", () => {
+ const rules = parseScrubRules(
+ ["# a comment", "", `/srv/${PLANTED}==>/home/user`, " # indented comment", "literal:abc==>x", "regex:a+b==>c", "glob:*.x==>y", "bare-line", "a==>b==>c", "\r"].join("\n"),
+ HOME,
+ );
+ assert.deepEqual(rules.lines, [
+ `${HOME}==>/home/user`,
+ `/srv/${PLANTED}==>/home/user`,
+ "literal:abc==>x",
+ "regex:a+b==>c",
+ "glob:*.x==>y",
+ "bare-line",
+ "a==>b==>c",
+ ]);
+ // filter-repo splits at the LAST ==>; regex and glob rules deny nothing.
+ // Each denial is named by where it was written (the report's label).
+ assert.deepEqual(rules.denied, [
+ { text: HOME, from: "built-in home rule" },
+ { text: `/srv/${PLANTED}`, from: "scrub line 3 lhs" },
+ { text: "abc", from: "scrub line 5 lhs" },
+ { text: "bare-line", from: "scrub line 8 lhs" },
+ { text: "a==>b", from: "scrub line 9 lhs" },
+ ]);
+ // A home dir of `/`, or one that IS the replacement, gets no built-in rule;
+ // a trailing slash is dropped, or the rule would match nothing.
+ assert.deepEqual(parseScrubRules("", "/").lines, []);
+ assert.deepEqual(parseScrubRules("", "/home/user").lines, []);
+ assert.deepEqual(parseScrubRules("", `${HOME}/`).lines, [`${HOME}==>/home/user`]);
+});
+
+test("scrub rules: a byte-order mark and CRLF are dropped; an empty left side refuses (review L2)", () => {
+ // The reviewer's reproduction: a BOM made the first rule — and its implied
+ // denial — `\uFEFF/srv/…`, which matches nothing.
+ const bom = parseScrubRules(`\uFEFF/srv/${PLANTED}==>/home/user\r\nx==>y\r\n`, HOME);
+ assert.deepEqual(bom.lines, [`${HOME}==>/home/user`, `/srv/${PLANTED}==>/home/user`, "x==>y"]);
+ assert.deepEqual(bom.denied.map((d) => d.text), [HOME, `/srv/${PLANTED}`, "x"]);
+ for (const line of ["==>user", "literal:==>user", "regex:==>user", "glob:"]) {
+ assert.throws(
+ () => parseScrubRules(`ok==>fine\n${line}\n`, HOME),
+ (e) => e instanceof SourceRefusal && /scrub line 2 has an empty left side/.test(e.message),
+ line,
+ );
+ }
+});
+
+test("the operator's files: a missing one refuses by name; the denylist's i: and the implied left sides; the hash moves with them", async () => {
+ const d = dir("cfg");
+ await assert.rejects(
+ loadSourceRules({ scrubFile: path.join(d, "nope.txt"), denylistFile: path.join(d, "deny.txt"), homeDir: HOME }),
+ (e) => e instanceof SourceRefusal && /no scrub rules at .*nope\.txt — create it/.test(e.message),
+ );
+ writeFileSync(path.join(d, "scrub.txt"), `/srv/${PLANTED}==>/home/user\n`);
+ await assert.rejects(
+ loadSourceRules({ scrubFile: path.join(d, "scrub.txt"), denylistFile: path.join(d, "deny.txt"), homeDir: HOME }),
+ (e) => e instanceof SourceRefusal && /no denylist at .*deny\.txt/.test(e.message),
+ );
+ writeFileSync(path.join(d, "deny.txt"), `\uFEFF# mine\r\ni:${SECRET.toUpperCase()}\r\n`);
+ const a = await loadSourceRules({ scrubFile: path.join(d, "scrub.txt"), denylistFile: path.join(d, "deny.txt"), homeDir: HOME });
+ assert.deepEqual(
+ a.literals.map((l) => [l.bytes.toString(), l.ci, l.from]),
+ [[SECRET, true, "denylist line 2"], [HOME, false, "built-in home rule"], [`/srv/${PLANTED}`, false, "scrub line 1 lhs"]],
+ );
+ writeFileSync(path.join(d, "deny.txt"), `i:${SECRET}\nanother\n`);
+ const b = await loadSourceRules({ scrubFile: path.join(d, "scrub.txt"), denylistFile: path.join(d, "deny.txt"), homeDir: HOME });
+ assert.notEqual(a.rulesHash, b.rulesHash, "a new literal must defeat the skip");
+});
+
+test("the limits: 15,000 files and 24 MiB a file, inside Pages' 20,000 and 25 MiB", () => {
+ assert.equal(MAX_FILES, 15_000);
+ assert.equal(MAX_FILE_BYTES, 24 * 1024 * 1024);
+ assert.equal(limitProblem([{ rel: "a", bytes: MAX_FILE_BYTES }]), null);
+ assert.match(limitProblem([{ rel: "big.pack", bytes: MAX_FILE_BYTES + 1 }])!, /^big\.pack is 24\.0 MiB, over the step's limit of 24\.0 MiB/);
+ const many = Array.from({ length: MAX_FILES + 1 }, (_, i) => ({ rel: `f${i}`, bytes: 1 }));
+ assert.match(limitProblem(many)!, /^15001 files to publish, over the step's limit of 15000/);
+ assert.equal(limitProblem(many.slice(1)), null);
+});
+
+test("skip: an unchanged main with unchanged rules, tools and files does nothing; each part of the key defeats it; a refusal withdraws", async () => {
+ const repo = sourceRepo();
+ const files = operatorFiles(`/srv/${PLANTED}==>/home/user\n`, "");
+ const logs: string[] = [];
+ // filter-repo is `false`: reaching it would refuse, so a 0 proves the skip
+ // and a 1 proves the key did NOT match (review R2-L3: every case below goes
+ // through the skip, never `check: true`, which skips nothing).
+ const o = opts(repo, files, logs, { filterRepo: ["false"] });
+ const pub = o.publicDir!;
+ const sourceCommit = gitIn(repo, "rev-parse", "main");
+ const rules = await loadSourceRules({ ...files, homeDir: HOME });
+ const mirrorHead = "a".repeat(40);
+ const state = path.join(path.dirname(pub), ".source-publish.json");
+ const plant = async (tweak: Record<string, string> = {}) => {
+ mkdirSync(path.join(pub, "source", MIRROR_DIR, "info"), { recursive: true });
+ mkdirSync(path.join(pub, "downloads"), { recursive: true });
+ writeFileSync(path.join(pub, "source", MIRROR_DIR, "info", "refs"), "x\trefs/heads/main\n");
+ writeFileSync(path.join(pub, "downloads", path.basename(TARBALL_HREF)), "tarball");
+ writeFileSync(path.join(pub, "downloads", "snapshot.json"), "{}");
+ writeFileSync(path.join(pub, "source", "manifest.json"), fakeManifest(sourceCommit, mirrorHead));
+ const key = {
+ sourceCommit,
+ mirrorHead,
+ rulesHash: rules.rulesHash,
+ filterRepo: "false (given)",
+ gitleaks: "skipped",
+ contentDigest: await sourceDigest(pub),
+ };
+ writeFileSync(state, JSON.stringify({ ...key, ...tweak }));
+ };
+ const run = async () => {
+ logs.length = 0;
+ return publishSource(o);
+ };
+
+ await plant();
+ assert.equal(await run(), 0);
+ assert.match(logs.join("\n"), new RegExp(`up to date at ${sourceCommit.slice(0, 12)}; skipping`));
+
+ // Each part of the key, changed alone, reaches the rewrite (and the refusal
+ // withdraws the planted publish, so it is planted again each time).
+ for (const [what, tweak] of [
+ ["another filter-repo", { filterRepo: "git filter-repo 0.1" }],
+ ["another gitleaks", { gitleaks: "gitleaks 8.0 (abc)" }],
+ ["other files", { contentDigest: "0".repeat(64) }],
+ ["other rules", { rulesHash: "0".repeat(64) }],
+ ] as Array<[string, Record<string, string>]>) {
+ await plant(tweak);
+ assert.equal(await run(), 1, `${what} must not skip`);
+ assert.match(logs.join("\n"), /REFUSED: false --force --quiet exited 1/, what);
+ }
+ // A published file edited in place defeats the skip too.
+ await plant();
+ writeFileSync(path.join(pub, "downloads", "snapshot.json"), '{"edited":true}');
+ assert.equal(await run(), 1, "an edited public/ file must not skip");
+
+ // A changed rule refuses AND withdraws the last publish.
+ await plant();
+ writeFileSync(files.denylistFile, "a-new-literal\n");
+ assert.equal(await run(), 1, "the new rule reached the rewrite");
+ assert.match(logs.join("\n"), /REFUSED: false --force --quiet exited 1/);
+ assert.match(logs.join("\n"), /the previous publish was WITHDRAWN/);
+ assert.ok(!existsSync(path.join(pub, "source")), "the last publish is withdrawn");
+ assert.ok(!existsSync(path.join(pub, "downloads", path.basename(TARBALL_HREF))));
+ assert.ok(!existsSync(path.join(pub, "downloads", "snapshot.json")));
+ assert.ok(!existsSync(state));
+ assert.equal(readdirSync(o.scratchRoot!).length, 0, "the scratch dir is removed");
+});
+
+test("the rules hash moves with the step's version (review R2-L3), and loadSourceRules uses the current one", async () => {
+ const lines = ["a==>b"];
+ const literals = [{ bytes: Buffer.from("a"), ci: false }];
+ assert.notEqual(rulesHashOf(lines, literals, 1), rulesHashOf(lines, literals, 2));
+ assert.notEqual(rulesHashOf(lines, literals, 1), rulesHashOf(lines, [{ bytes: Buffer.from("a"), ci: true }], 1));
+ const files = operatorFiles("a==>b\n", "");
+ const rules = await loadSourceRules({ ...files, homeDir: "/" });
+ assert.equal(rules.rulesHash, rulesHashOf(["a==>b"], rules.literals, SOURCE_STEP_VERSION));
+ assert.ok(SOURCE_STEP_VERSION >= 3);
+});
+
+test("gitleaksIdentity: skipped, absent, or the version line with the binary's sha256 (review R2-L4)", async () => {
+ const bin = dir("gl-bin");
+ assert.equal(await gitleaksIdentity(null, { PATH: bin }), "skipped");
+ assert.equal(await gitleaksIdentity("gitleaks", { PATH: bin }), "absent");
+ const fake = path.join(bin, "gitleaks");
+ writeFileSync(fake, "#!/bin/sh\necho 'version is set by build process'\n");
+ chmodSync(fake, 0o755);
+ const a = await gitleaksIdentity("gitleaks", { PATH: bin });
+ assert.match(a, /^version is set by build process \([0-9a-f]{12}\)$/);
+ // Same version line, another binary (an upgrade on a distribution that
+ // prints a constant line): another identity.
+ writeFileSync(fake, "#!/bin/sh\necho 'version is set by build process'\n# 8.30\n");
+ assert.notEqual(await gitleaksIdentity("gitleaks", { PATH: bin }), a);
+});
+
+test("the scratch root: refused inside the checkout or the public dir, through a symlink too (review L12)", async () => {
+ const checkout = dir("chk");
+ const pub = path.join(dir("site"), "public");
+ const places: Array<[string, string]> = [["the checkout", checkout], ["the public dir", pub]];
+ assert.match((await scratchRootProblem(path.join(checkout, "tmp", "x"), places))!, /inside the checkout/);
+ assert.match((await scratchRootProblem(path.join(pub, "s"), places))!, /inside the public dir/);
+ const link = path.join(dir("links"), "into-checkout");
+ symlinkSync(checkout, link);
+ assert.match((await scratchRootProblem(path.join(link, "new"), places))!, /inside the checkout/);
+ assert.equal(await scratchRootProblem(path.join(TMP, "scratch"), places), null);
+ // …and publishSource says so before making anything.
+ const repo = sourceRepo();
+ const logs: string[] = [];
+ const o = opts(repo, operatorFiles("", ""), logs, { filterRepo: ["false"] });
+ assert.equal(await publishSource({ ...o, scratchRoot: path.join(o.publicDir!, "scratch") }), 1);
+ assert.match(logs.join("\n"), /REFUSED: the scratch root .* is inside the public dir/);
+ assert.ok(!existsSync(path.join(o.publicDir!, "scratch")));
+});
+
+test("no git repository here (a docker runtime, a tarball install): the build gets the empty state, the CLI a sentence (review L8)", async () => {
+ const bare = dir("no-git");
+ const pub = path.join(dir("site"), "public");
+ mkdirSync(path.join(pub, "source"), { recursive: true });
+ writeFileSync(path.join(pub, "source", "manifest.json"), "{}");
+ const logs: string[] = [];
+ const o: SourcePublishOpts = {
+ paths: { monorepoRoot: bare } as Paths,
+ publicDir: pub,
+ onLog: (l) => logs.push(l),
+ // No operator files at all: nothing is read before the repository check.
+ scrubFile: path.join(bare, "none.txt"),
+ denylistFile: path.join(bare, "none.txt"),
+ };
+ assert.equal(await publishSource({ ...o, check: true }), 1);
+ assert.ok(existsSync(path.join(pub, "source", "manifest.json")), "--check writes nothing");
+ logs.length = 0;
+ assert.equal(await publishSource(o), 1, "the CLI refuses");
+ assert.equal(logs[0], `[source] ${NO_REPOSITORY}`);
+ assert.ok(!logs.join("\n").includes("fatal"), "no raw git error");
+ assert.ok(!existsSync(path.join(pub, "source")), "an old publish is removed");
+ mkdirSync(path.join(pub, "source"), { recursive: true });
+ writeFileSync(path.join(pub, "source", "manifest.json"), "{}");
+ logs.length = 0;
+ assert.equal(await publishSource({ ...o, noRepository: "empty" }), 0, "buildHomepage builds on");
+ assert.equal(logs[0], `[source] ${NO_REPOSITORY}`);
+ assert.ok(!existsSync(path.join(pub, "source")));
+});
+
+test("round trip: --check writes nothing; publish; a dumb clone of the mirror is main, scrubbed, from static files", async (t) => {
+ if (filterRepoProblem) return t.skip(`git-filter-repo unavailable: ${filterRepoProblem}`);
+ const repo = sourceRepo();
+ // A byte-order mark and CRLF endings, as some editors write them: the rule
+ // must still scrub, and its left side must still be denied (review L2).
+ const files = operatorFiles(`\uFEFF# the planted path\r\n/srv/${PLANTED}==>/home/user\r\n`, `i:${PLANTED}\r\n`);
+ const logs: string[] = [];
+ const o = opts(repo, files, logs);
+ const pub = o.publicDir!;
+ const site = path.dirname(pub);
+
+ assert.equal(await publishSource({ ...o, check: true }), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /check passed — would publish main .* nothing written/);
+ assert.ok(!existsSync(pub), "--check wrote nothing");
+ assert.ok(!existsSync(path.join(site, ".source-publish.json")));
+
+ logs.length = 0;
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /\[source\] audit clean: \d+ objects \(3 commits\), \d+ staged files against 3 denied literals; gitleaks skipped/);
+ assert.match(logs.join("\n"), /\[source\] published main [0-9a-f]{12} as [0-9a-f]{12}: \d+ files/);
+ const manifest = JSON.parse(readFileSync(path.join(pub, "source", "manifest.json"), "utf8"));
+ const sourceCommit = gitIn(repo, "rev-parse", "main");
+ assert.equal(manifest.version, 1);
+ assert.equal(manifest.branch, "main");
+ assert.equal(manifest.sourceCommit, sourceCommit);
+ assert.notEqual(manifest.mirrorHead, sourceCommit, "scrubbed history, different ids");
+ assert.equal(manifest.subject, "moved from /home/user");
+ assert.equal(manifest.cloneUrl, CLONE_URL);
+ assert.equal(manifest.treeHref, TREE_HREF);
+ assert.deepEqual(manifest.tree, { files: 4, dirs: 4, bytes: manifest.tree.bytes });
+ assert.ok(manifest.mirror.packs >= 1);
+ assert.equal(manifest.audit.commits, 3);
+ assert.equal(manifest.audit.gitleaks, "skipped");
+ assert.ok(!("rulesHash" in manifest), "the rules hash is never published");
+ assert.ok(!("literals" in manifest.audit), "nor how many literals there are (review L3)");
+ assert.ok(existsSync(path.join(site, ".source-publish.json")), "…it is kept beside public/");
+
+ // The mirror: the allowlist and the dumb-HTTP files.
+ const mirror = path.join(pub, "source", MIRROR_DIR);
+ for (const f of ["config", "hooks", "description", "filter-repo", "logs", "info/exclude"]) {
+ assert.ok(!existsSync(path.join(mirror, f)), `${f} is never published`);
+ }
+ assert.equal(readFileSync(path.join(mirror, "HEAD"), "utf8"), "ref: refs/heads/main\n");
+ assert.equal(readFileSync(path.join(mirror, "info", "refs"), "utf8"), `${manifest.mirrorHead}\trefs/heads/main\n`);
+ assert.match(readFileSync(path.join(mirror, "objects", "info", "packs"), "utf8"), /^P pack-[0-9a-f]+\.pack$/m);
+ // Only packs and their indexes: git 2.55 also writes .rev files, which the
+ // allowlist leaves behind.
+ const packDir = readdirSync(path.join(mirror, "objects", "pack"));
+ assert.ok(packDir.length >= 2);
+ for (const f of packDir) assert.match(f, /^pack-[0-9a-f]+\.(pack|idx)$/);
+
+ // EVERY object in the published packs, reachable or not — what a dumb-HTTP
+ // reader gets — read from a plain copy of the published files (review L9).
+ const copy = path.join(dir("copy"), "m.git");
+ cpSync(mirror, copy, { recursive: true });
+ const all = execFileSync("git", ["--git-dir", copy, "cat-file", "--batch-all-objects", "--batch"], { maxBuffer: 1 << 26 });
+ assert.equal(all.indexOf(PLANTED), -1, "no object in the packs holds the planted path");
+ assert.ok(all.includes(Buffer.from("/home/user")), "…which was scrubbed, not dropped");
+ assert.ok(!existsSync(path.join(pub, "source", "index.html")), "the /source/ page keeps its route");
+
+ // The clone.
+ const clone = path.join(dir("clone"), "c");
+ execFileSync("git", ["clone", "-q", `file://${mirror}`, clone], { stdio: "pipe" });
+ assert.equal(gitIn(clone, "rev-parse", "HEAD"), manifest.mirrorHead);
+ assert.equal(gitIn(clone, "log", "-1", "--format=%s"), "moved from /home/user");
+ assert.equal(readFileSync(path.join(clone, "notes.txt"), "utf8"), "data lives at /home/user/x\n");
+ const revs = gitIn(clone, "rev-list", "--all").split("\n");
+ assert.equal(revs.length, 3);
+ assert.throws(
+ () => execFileSync("git", ["grep", "-F", PLANTED, ...revs], { cwd: clone, stdio: "pipe" }),
+ (e: { status?: number }) => e.status === 1,
+ "no revision holds the planted path",
+ );
+ assert.ok(!gitIn(clone, "log", "--all", "--format=%B%an%ae").includes(PLANTED));
+
+ // The tarball and its sidecar agree with the manifest and the bytes.
+ const tarball = path.join(pub, "downloads", path.basename(TARBALL_HREF));
+ const snapshot = JSON.parse(readFileSync(path.join(pub, "downloads", "snapshot.json"), "utf8"));
+ assert.equal(snapshot.sha256, sha256(tarball));
+ assert.equal(snapshot.sha256, manifest.tarball.sha256);
+ assert.equal(snapshot.bytes, statSync(tarball).size);
+ assert.equal(snapshot.commit, manifest.mirrorHead);
+ assert.equal(snapshot.subject, "moved from /home/user");
+
+ // The tree pages.
+ const tree = path.join(pub, "source", "tree");
+ for (const d of ["", "app", "app/[slug]", "fonts"]) assert.ok(existsSync(path.join(tree, d, "index.html")), d);
+ assert.match(readFileSync(path.join(tree, "fonts", "index.html"), "utf8"), /href="Archivo%5Bwdth%2Cwght%5D\.ttf"/);
+ assert.equal(readFileSync(path.join(tree, "notes.txt"), "utf8"), "data lives at /home/user/x\n");
+ assert.equal(readdirSync(o.scratchRoot!).length, 0, "the scratch dir is removed");
+
+ // Nothing changed: the next publish skips.
+ logs.length = 0;
+ assert.equal(await publishSource(o), 0);
+ assert.match(logs.join("\n"), /up to date at/);
+
+ // The deploy check (M1): a build's out/ that is this publish deploys…
+ const out = path.join(dir("out"), "out");
+ const check = { ...files, homeDir: HOME, publicDir: pub, sourceRepo: path.join(repo, ".git"), gitleaks: null };
+ const checkPaths = o.paths!;
+ // A build's out/ always has the /source page; a --no-source one, nothing more.
+ mkdirSync(path.join(out, "source"), { recursive: true });
+ writeFileSync(path.join(out, "source", "index.html"), "<p>No source published in this build.</p>");
+ assert.equal(await publishedSourceProblem(checkPaths, out, check), null, "--no-source deploys as before");
+ cpSync(path.join(pub, "source"), path.join(out, "source"), { recursive: true });
+ cpSync(path.join(pub, "downloads"), path.join(out, "downloads"), { recursive: true });
+ assert.equal(await publishedSourceProblem(checkPaths, out, check), null);
+ // Review R2-L3: a state naming another mirror head, or another gitleaks,
+ // refuses — each by its own sentence (the digest below would refuse
+ // neither, since out/ is untouched).
+ const stateFile = path.join(site, ".source-publish.json");
+ const good = readFileSync(stateFile, "utf8");
+ const stateWith = (tweak: object) => writeFileSync(stateFile, JSON.stringify({ ...JSON.parse(good), ...tweak }));
+ stateWith({ mirrorHead: "f".repeat(40) });
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /is not the last publish \(ffffffffffff\)/);
+ stateWith({ gitleaks: "gitleaks 8.0 (abc)" });
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /scanned by another gitleaks/);
+ writeFileSync(stateFile, good);
+ // Review R2-L2: a mixed out/ — one tree file swapped, or one pack byte
+ // flipped — does not match the digest of what was audited.
+ const readme = path.join(out, "source", "tree", "README.md");
+ const readmeWas = readFileSync(readme);
+ writeFileSync(readme, "another publish's README\n");
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /not the ones that were audited/);
+ writeFileSync(readme, readmeWas);
+ const packDirOut = path.join(out, "source", MIRROR_DIR, "objects", "pack");
+ const pack = path.join(packDirOut, readdirSync(packDirOut).find((f) => f.endsWith(".pack"))!);
+ const packWas = readFileSync(pack);
+ chmodSync(pack, 0o644); // git writes packs read-only; the digest reads bytes, not modes
+ const flipped = Buffer.from(packWas);
+ flipped[Math.floor(flipped.length / 2)] ^= 0x01;
+ writeFileSync(pack, flipped);
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /not the ones that were audited/);
+ writeFileSync(pack, packWas);
+ assert.equal(await publishedSourceProblem(checkPaths, out, check), null, "restored, it deploys again");
+ // …a tampered tarball does not…
+ writeFileSync(path.join(out, "downloads", path.basename(TARBALL_HREF)), "other bytes");
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /tarball is not the one its manifest describes/);
+ cpSync(path.join(pub, "downloads"), path.join(out, "downloads"), { recursive: true });
+ // …nor one of an older main.
+ writeFileSync(path.join(repo, "later.txt"), "x\n");
+ gitIn(repo, "add", "-A");
+ gitIn(repo, "commit", "-q", "-m", "later");
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /main is now [0-9a-f]{12} — run `archilyzer build homepage`/);
+
+ // A REFUSAL WITHDRAWS THE PUBLISH (M1): deny a literal the mirror holds.
+ // The deploy check refuses first (other rules), then the publish refuses
+ // and removes public/source and the downloads.
+ writeFileSync(files.denylistFile, `i:${PLANTED}\nhello again\n`);
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /audited under other rules/);
+ logs.length = 0;
+ assert.equal(await publishSource(o), 1);
+ assert.match(logs.join("\n"), /AUDIT REFUSED/);
+ assert.match(logs.join("\n"), /denylist line 2 \(len 11\)/);
+ assert.ok(!logs.join("\n").includes("hello again"), "the report never prints the literal");
+ assert.ok(!existsSync(path.join(pub, "source")), "public/source is withdrawn");
+ assert.ok(!existsSync(tarball), "the tarball is withdrawn");
+ assert.ok(!existsSync(path.join(pub, "downloads", "snapshot.json")));
+ assert.ok(!existsSync(path.join(site, ".source-publish.json")));
+ assert.match((await publishedSourceProblem(checkPaths, out, check))!, /no record of the rules/);
+});
+
+test("a denied literal no rule removes: refused, nothing written, the report never prints it", async (t) => {
+ if (filterRepoProblem) return t.skip(`git-filter-repo unavailable: ${filterRepoProblem}`);
+ const repo = sourceRepo({ "keys.txt": `the ${SECRET} is here\n` });
+ const files = operatorFiles(`/srv/${PLANTED}==>/home/user\n`, `${SECRET}\n`);
+ const logs: string[] = [];
+ const o = opts(repo, files, logs);
+ const downloads = path.join(o.publicDir!, "downloads");
+ mkdirSync(downloads, { recursive: true });
+ writeFileSync(path.join(downloads, "snapshot.json"), "yesterday's");
+ assert.equal(await publishSource({ ...o, keepScratch: true }), 1);
+ const report = logs.join("\n");
+ assert.ok(!report.includes(SECRET), report);
+ assert.ok(!report.includes("is here"), "no byte from beside the hit (review L4)");
+ assert.match(report, /AUDIT REFUSED: 1 hit in \d+ objects/);
+ assert.match(report, /denylist line 1 \(len 13\): 1 in blob/);
+ assert.match(report, /blob [0-9a-f]{12} keys\.txt \(byte 4\): denylist line 1 \(len 13\)/);
+ assert.match(report, /add a rule to .*source-scrub\.txt or drop the file from history, then re-run\./);
+ assert.ok(!existsSync(path.join(o.publicDir!, "source")), "no manifest, no mirror");
+ assert.ok(!existsSync(path.join(downloads, "snapshot.json")), "yesterday's snapshot is withdrawn too");
+ // --keep-scratch keeps the clone for a look, never the scrub rules.
+ const kept = /scratch kept at (\S+) \(replace\.txt, the scrub rules, deleted\)/.exec(report);
+ assert.ok(kept, report);
+ assert.ok(existsSync(path.join(kept[1], "bare")));
+ assert.ok(!existsSync(path.join(kept[1], "replace.txt")));
+ rmSync(kept[1], { recursive: true, force: true });
+});
+
+test("a refusal naming a tree path masks a literal that spans path components (review R2-L1)", async (t) => {
+ if (filterRepoProblem) return t.skip(`git-filter-repo unavailable: ${filterRepoProblem}`);
+ // `plant/secret` is invisible to the object walk (which reads entry names
+ // one at a time), so the first refusal to name it is the tree's own: a
+ // tracked symlink below it, then a tracked index.html.
+ const spanning = "plant/secret";
+ const repo = sourceRepo();
+ mkdirSync(path.join(repo, "plant", "secret"), { recursive: true });
+ symlinkSync("../../README.md", path.join(repo, "plant", "secret", "link"));
+ gitIn(repo, "add", "-A");
+ gitIn(repo, "commit", "-q", "-m", "a link");
+ const files = operatorFiles(`/srv/${PLANTED}==>/home/user\n`, `${spanning}\n`);
+ const logs: string[] = [];
+ assert.equal(await publishSource(opts(repo, files, logs)), 1);
+ let log = logs.join("\n");
+ assert.match(log, /REFUSED: the tree holds a symlink \(\[REDACTED\]\/link\)/);
+ assert.ok(!log.includes(spanning), log);
+
+ gitIn(repo, "rm", "-q", "plant/secret/link");
+ mkdirSync(path.join(repo, "plant", "secret"), { recursive: true });
+ writeFileSync(path.join(repo, "plant", "secret", "index.html"), "<p>x</p>");
+ gitIn(repo, "add", "-A");
+ gitIn(repo, "commit", "-q", "-m", "an index");
+ logs.length = 0;
+ assert.equal(await publishSource(opts(repo, files, logs)), 1);
+ log = logs.join("\n");
+ assert.match(log, /REFUSED: the tree already has \[REDACTED\]\/index\.html/);
+ assert.ok(!log.includes(spanning), log);
+});
+
+test("--no-source's clear: the manifest, mirror, tree, tarball and skip key go; a linked downloads/ goes as a link", async () => {
+ const site = dir("clear");
+ const pub = path.join(site, "public");
+ mkdirSync(path.join(pub, "source", MIRROR_DIR), { recursive: true });
+ writeFileSync(path.join(pub, "source", "manifest.json"), "{}");
+ const elsewhere = dir("elsewhere");
+ writeFileSync(path.join(elsewhere, "snapshot.json"), "the other checkout's");
+ symlinkSync(elsewhere, path.join(pub, "downloads"));
+ writeFileSync(path.join(site, ".source-publish.json"), "{}");
+ const logs: string[] = [];
+ await clearPublishedSource({ paths: {} as Paths, publicDir: pub, onLog: (l) => logs.push(l) });
+ assert.ok(!existsSync(path.join(pub, "source")));
+ assert.ok(!existsSync(path.join(site, ".source-publish.json")));
+ assert.equal(lstatSync(path.join(pub, "downloads"), { throwIfNoEntry: false }), undefined);
+ assert.equal(readFileSync(path.join(elsewhere, "snapshot.json"), "utf8"), "the other checkout's");
+ assert.match(logs.join("\n"), /previously published source .* was removed/);
+});
diff --git a/common/publish/source.ts b/common/publish/source.ts
@@ -0,0 +1,1136 @@
+// `archilyzer source publish` — the repo on the project site, read-only.
+//
+// The private repository is never rewritten. Every publish makes a FRESH bare
+// clone of its `main`, rewrites that copy with git-filter-repo (the operator's
+// scrub rules over file contents AND commit messages), repacks it for git's
+// dumb-HTTP protocol, and publishes three things under homepage/public:
+//
+// source/archilyzer.git/ a clonable mirror: HEAD, refs, info/refs,
+// objects/info/packs, the packs — static files only
+// source/tree/ the tracked files of main, raw, with an
+// index.html per directory (sourceTree.ts)
+// downloads/ archilyzer-source.tar.gz + snapshot.json, the
+// tarball the Downloads page has always offered
+// source/manifest.json written LAST: what was published, from what
+//
+// THE GATE. Before anything is staged for the site, every object of the
+// rewritten mirror, and then every staged file, is searched for every denied
+// literal (sourceAudit.ts): the operator's denylist plus every scrub rule's
+// left side. One hit and nothing is written; the report names the literal by
+// number, never by its bytes.
+//
+// OPERATOR-PRIVATE INPUTS live outside the repo, in
+// `${ARCHILYZER_CONFIG_DIR ?? ~/.config/archilyzer}/`: source-scrub.txt
+// (git-filter-repo `lhs==>rhs` lines; `<home dir>==>/home/user` is always
+// applied first) and source-denylist.txt (one literal per line, `i:` = any
+// case). A missing file is a refusal naming it. Nothing here names a user.
+//
+// `buildHomepage` runs this between compose and `next build`, so `archilyzer
+// build homepage`, the runbooks' home scripts and the editor's /sites homepage
+// jobs all publish it; an unchanged main with unchanged rules skips.
+
+import { execFile } from "node:child_process";
+import { createHash } from "node:crypto";
+import { createReadStream, existsSync } from "node:fs";
+import { cp, lstat, mkdir, mkdtemp, readdir, readFile, realpath, rm, stat, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { runChildIntoLog } from "../jobs/runChild";
+import { copyPublicFile, ownDir, writePublicFile } from "../bin/_publicFile";
+import { getPaths, type Paths } from "../lib/paths";
+import {
+ CLONE_URL,
+ MIRROR_DIR,
+ SOURCE_MANIFEST_VERSION,
+ TARBALL_HREF,
+ TREE_HREF,
+ parseSourceManifest,
+ type SourceManifest,
+} from "../lib/sourceManifest";
+import {
+ SourceRefusal,
+ auditBare,
+ auditFiles,
+ cleanGitEnv,
+ dedupeLiterals,
+ formatAuditReport,
+ operatorLines,
+ maskLiterals,
+ onPath,
+ parseDenylist,
+ tildify,
+ type Literal,
+} from "./sourceAudit";
+import { writeTreeIndexes } from "./sourceTree";
+// Type-only: build.ts imports THIS module lazily, and must not load it (or the
+// AWS SDK this would drag in) at import time.
+import type { PublishOpts } from "./build";
+
+export { SourceRefusal } from "./sourceAudit";
+
+/** The one branch mirrored (an operator decision: no other refs, no tags). */
+export const SOURCE_BRANCH = "main";
+
+/**
+ * The step's own version, hashed into the rules hash (the skip key and the
+ * deploy check). BUMP IT WHENEVER THE SCRUB OR THE AUDIT CHANGES — what the
+ * rules parse to, what the gate reads, what it refuses — so an unchanged
+ * `main` is re-published through the fixed step instead of skipped, and a
+ * `homepage/out` built by the old step refuses to deploy.
+ * 1 — release 12 slice R.
+ * 2 — release 12 slice R review: BOM/CRLF/trailing-slash parsing, empty
+ * left sides refused, `404.html` refused, no context bytes.
+ * 3 — the re-review: masked refusal messages, a digest of every published
+ * file and the gitleaks identity in the key.
+ */
+export const SOURCE_STEP_VERSION = 3;
+
+/** What `pipx run` fetches when `git filter-repo` is not installed. */
+export const FILTER_REPO_PIPX_SPEC = "git-filter-repo==2.47.0";
+export const FILTER_REPO_INSTALL = "pipx install git-filter-repo";
+
+/** The home-directory rule's replacement (always the first scrub rule). */
+export const HOME_REPLACEMENT = "/home/user";
+
+// Cloudflare Pages allows 20,000 files per deployment and 25 MiB per file;
+// the step refuses well inside both, leaving the rest of the site its room.
+export const MAX_FILES = 15_000;
+export const MAX_FILE_BYTES = 24 * 1024 * 1024;
+
+// Packs are split at this size (under the per-file cap, with room to grow).
+const PACK_SIZE = "20m";
+
+const TARBALL_NAME = path.basename(TARBALL_HREF);
+
+export type SourcePublishOpts = PublishOpts & {
+ // Rebuild even when main and the rules are unchanged.
+ force?: boolean;
+ // Build, audit and count everything, then write NOTHING.
+ check?: boolean;
+ // Leave the scratch dir (the rewritten bare clone, the stage) for a look.
+ keepScratch?: boolean;
+ // The repository to mirror. Default: this checkout's git COMMON dir, so a
+ // worktree build mirrors the primary's main.
+ sourceRepo?: string;
+ // Default: HOMEPAGE_PUBLIC_DIR, else <repo>/homepage/public.
+ publicDir?: string;
+ scrubFile?: string;
+ denylistFile?: string;
+ // The filter-repo argv; default: resolveFilterRepo().
+ filterRepo?: string[] | null;
+ // The gitleaks binary; null skips the secret scan (tests). Default "gitleaks".
+ gitleaks?: string | null;
+ // Where the scratch dir is made. Default: paths.sourceScratchDir.
+ scratchRoot?: string;
+ // The environment the children run in (PATH decides which tools). Default:
+ // process.env.
+ env?: NodeJS.ProcessEnv;
+ now?: () => Date;
+ // The home dir the built-in rule scrubs. Default: os.homedir().
+ homeDir?: string;
+ // A checkout with no git repository: "refuse" (the CLI: exit 1) or "empty"
+ // (buildHomepage: the /source page's empty state, exit 0). Either way it
+ // says NO_REPOSITORY and removes an old publish.
+ noRepository?: "refuse" | "empty";
+};
+
+// ── the operator's files ────────────────────────────────────────────────────
+
+export type ScrubRules = {
+ // replace.txt as filter-repo will read it: the built-in rule first, then
+ // the operator's rules in order (comments and blank lines dropped —
+ // filter-repo itself would treat a `#` line as a literal to replace).
+ lines: string[];
+ // Every rule's LITERAL left side: denied, exact case, by implication, with
+ // where it was written (the report's name for it).
+ denied: Array<{ text: string; from: string }>;
+};
+
+export const BUILT_IN_HOME_RULE = "built-in home rule";
+
+/**
+ * The scrub file's text as rules. A line is `lhs==>rhs` (split at the LAST
+ * `==>`, as filter-repo splits it), `literal:lhs==>rhs`, `regex:…==>…` or
+ * `glob:…==>…`; a line with no `==>` is replaced by filter-repo's
+ * `***REMOVED***`. Lines whose first non-blank character is `#` are comments.
+ * A byte-order mark and CRLF endings are dropped first (operatorLines), and
+ * the home dir loses a trailing `/`: either would make a rule — and the
+ * denial it implies — silently match nothing. An empty left side is a
+ * refusal, not a rule filter-repo would skip.
+ */
+export function parseScrubRules(text: string, homeDir: string): ScrubRules {
+ const lines: string[] = [];
+ const denied: ScrubRules["denied"] = [];
+ const home = homeDir.replace(/\/+$/, "");
+ // A home dir of `/` (a container user) would scrub every slash, and one
+ // that IS the replacement would deny the replacement itself.
+ if (home.length > 1 && home !== HOME_REPLACEMENT) {
+ lines.push(`${home}==>${HOME_REPLACEMENT}`);
+ denied.push({ text: home, from: BUILT_IN_HOME_RULE });
+ }
+ operatorLines(text).forEach((line, n) => {
+ const t = line.trim();
+ if (t === "" || t.startsWith("#")) return;
+ const i = line.lastIndexOf("==>");
+ let lhs = i === -1 ? line : line.slice(0, i);
+ const from = `scrub line ${n + 1} lhs`;
+ for (const prefix of ["regex:", "glob:", "literal:"]) {
+ if (lhs.startsWith(prefix)) {
+ if (lhs.length === prefix.length) {
+ throw new SourceRefusal(`scrub line ${n + 1} has an empty left side — it would match nothing and deny nothing`);
+ }
+ if (prefix === "literal:") lhs = lhs.slice(prefix.length);
+ else lhs = "";
+ break;
+ }
+ }
+ if (i === 0) {
+ throw new SourceRefusal(`scrub line ${n + 1} has an empty left side — it would match nothing and deny nothing`);
+ }
+ lines.push(line);
+ if (lhs) denied.push({ text: lhs, from });
+ });
+ return { lines, denied };
+}
+
+export type SourceRules = {
+ scrub: ScrubRules;
+ literals: Literal[];
+ // Changes whenever a rule or a literal does: part of the skip key. Never
+ // published (a hash of the denylist would confirm a guess at it).
+ rulesHash: string;
+};
+
+async function readOperatorFile(file: string, what: string, how: string): Promise<string> {
+ try {
+ return await readFile(file, "utf8");
+ } catch (err) {
+ if ((err as NodeJS.ErrnoException).code === "ENOENT") {
+ throw new SourceRefusal(`no ${what} at ${tildify(file)} — ${how}`);
+ }
+ throw err;
+ }
+}
+
+/** Both operator files, parsed, with the literal list the gate searches for. */
+export async function loadSourceRules(opts: {
+ scrubFile: string;
+ denylistFile: string;
+ homeDir?: string;
+}): Promise<SourceRules> {
+ const scrubText = await readOperatorFile(
+ opts.scrubFile,
+ "scrub rules",
+ "create it (git-filter-repo `lhs==>rhs` lines; the home-directory rule is built in, so it may be empty) or point SOURCE_SCRUB_FILE at one",
+ );
+ const denyText = await readOperatorFile(
+ opts.denylistFile,
+ "denylist",
+ "create it (one literal per line, `i:` for any case; every scrub rule's left side is denied too, so it may be empty) or point SOURCE_DENYLIST_FILE at one",
+ );
+ const scrub = parseScrubRules(scrubText, opts.homeDir ?? os.homedir());
+ const literals = dedupeLiterals([
+ ...parseDenylist(denyText),
+ ...scrub.denied.map((d) => ({ bytes: Buffer.from(d.text, "utf8"), ci: false, from: d.from })),
+ ]);
+ return { scrub, literals, rulesHash: rulesHashOf(scrub.lines, literals, SOURCE_STEP_VERSION) };
+}
+
+/**
+ * The rules hash: the scrub lines, every literal (and whether it is `i:`),
+ * and the step's version — so a fix to the step moves it like a new rule does.
+ */
+export function rulesHashOf(lines: readonly string[], literals: readonly Literal[], step: number): string {
+ return createHash("sha256")
+ .update(
+ JSON.stringify({
+ step,
+ rules: lines,
+ literals: literals.map((l) => `${l.ci ? "i" : "x"}:${l.bytes.toString("hex")}`),
+ }),
+ )
+ .digest("hex");
+}
+
+// ── children ────────────────────────────────────────────────────────────────
+
+type Ctx = {
+ onLog: (line: string) => void;
+ signal: AbortSignal;
+ env: NodeJS.ProcessEnv;
+ // Every echoed or quoted child line is masked with these once they are known.
+ literals: readonly Literal[];
+};
+
+class Cancelled extends Error {}
+
+/**
+ * One child through runChildIntoLog, with a timeout. Returns its combined
+ * output. A non-zero exit or a timeout is a refusal quoting its last lines
+ * (masked); a cancel throws Cancelled.
+ */
+async function run(
+ ctx: Ctx,
+ command: string,
+ args: string[],
+ o: { cwd: string; timeoutMs: number; echo?: boolean; allowFail?: boolean },
+): Promise<{ code: number; out: string[] }> {
+ const out: string[] = [];
+ const timeout = AbortSignal.timeout(o.timeoutMs);
+ const code = await runChildIntoLog(
+ (line) => {
+ out.push(line);
+ if (o.echo) ctx.onLog(`[source] ${maskLiterals(line, ctx.literals)}`);
+ },
+ AbortSignal.any([ctx.signal, timeout]),
+ { command, args, cwd: o.cwd, env: ctx.env },
+ );
+ if (ctx.signal.aborted) throw new Cancelled();
+ // `git --git-dir <path> repack …` is named by its verb, not the path.
+ const what = `${command} ${(args[0] === "--git-dir" ? args.slice(2, 3) : args.slice(0, 2)).join(" ")}`;
+ if (timeout.aborted) {
+ throw new SourceRefusal(`${what} timed out after ${Math.round(o.timeoutMs / 1000)} s`);
+ }
+ if (code !== 0 && !o.allowFail) {
+ const tail = out.slice(-3).map((l) => maskLiterals(l, ctx.literals)).join(" / ");
+ throw new SourceRefusal(`${what} exited ${code}${tail ? `: ${tail}` : ""}`);
+ }
+ return { code, out };
+}
+
+const lastLine = (out: string[]) => (out.filter((l) => l.trim()).at(-1) ?? "").trim();
+const OID = /^[0-9a-f]{40}(?:[0-9a-f]{24})?$/;
+
+async function revParse(ctx: Ctx, gitDir: string, rev: string): Promise<string> {
+ const { out } = await run(ctx, "git", ["--git-dir", gitDir, "rev-parse", "--verify", rev], {
+ cwd: gitDir,
+ timeoutMs: 30_000,
+ });
+ const oid = lastLine(out);
+ if (!OID.test(oid)) throw new SourceRefusal(`git rev-parse ${rev}: not an object id`);
+ return oid;
+}
+
+export type FilterRepoChoice = { argv: string[]; label: string; version: string };
+
+/**
+ * Which git-filter-repo runs: an installed `git filter-repo`, else `pipx run`
+ * of the pinned version (network on first use), else a refusal with the
+ * install line.
+ */
+export async function resolveFilterRepo(opts: {
+ env?: NodeJS.ProcessEnv;
+ signal?: AbortSignal;
+ onLog?: (line: string) => void;
+}): Promise<FilterRepoChoice> {
+ const ctx: Ctx = {
+ onLog: opts.onLog ?? (() => {}),
+ signal: opts.signal ?? new AbortController().signal,
+ env: cleanGitEnv(opts.env ?? process.env),
+ literals: [],
+ };
+ const cwd = os.tmpdir();
+ const installed = await run(ctx, "git", ["filter-repo", "--version"], {
+ cwd,
+ timeoutMs: 30_000,
+ allowFail: true,
+ });
+ if (installed.code === 0) {
+ return { argv: ["git", "filter-repo"], label: "git filter-repo", version: lastLine(installed.out) };
+ }
+ if (!onPath("pipx", ctx.env.PATH)) {
+ throw new SourceRefusal(
+ `git-filter-repo is not installed and pipx is not on PATH — install it once: \`${FILTER_REPO_INSTALL}\` (pipx comes from your OS's packages)`,
+ );
+ }
+ const argv = ["pipx", "run", "--spec", FILTER_REPO_PIPX_SPEC, "git-filter-repo"];
+ const viaPipx = await run(ctx, argv[0], [...argv.slice(1), "--version"], {
+ cwd,
+ timeoutMs: 300_000,
+ allowFail: true,
+ });
+ if (viaPipx.code !== 0) {
+ throw new SourceRefusal(
+ `\`pipx run --spec ${FILTER_REPO_PIPX_SPEC}\` failed (exit ${viaPipx.code}; it needs the network on first use) — install it once: \`${FILTER_REPO_INSTALL}\``,
+ );
+ }
+ return {
+ argv,
+ label: `pipx run --spec ${FILTER_REPO_PIPX_SPEC} git-filter-repo`,
+ version: lastLine(viaPipx.out),
+ };
+}
+
+// ── helpers ─────────────────────────────────────────────────────────────────
+
+function terminalLog(line: string): void {
+ process.stdout.write(line.endsWith("\n") ? line : `${line}\n`);
+}
+
+function sha256File(file: string): Promise<string> {
+ return new Promise((resolve, reject) => {
+ const h = createHash("sha256");
+ createReadStream(file)
+ .on("data", (c) => h.update(c))
+ .on("error", reject)
+ .on("end", () => resolve(h.digest("hex")));
+ });
+}
+
+async function walkFiles(dir: string): Promise<Array<{ rel: string; bytes: number }>> {
+ const out: Array<{ rel: string; bytes: number }> = [];
+ const walk = async (rel: string) => {
+ for (const ent of await readdir(path.join(dir, rel), { withFileTypes: true })) {
+ const r = rel ? `${rel}/${ent.name}` : ent.name;
+ if (ent.isDirectory()) await walk(r);
+ else out.push({ rel: r, bytes: (await stat(path.join(dir, r))).size });
+ }
+ };
+ await walk("");
+ return out;
+}
+
+const mb = (bytes: number) => (bytes / (1024 * 1024)).toFixed(1);
+
+/**
+ * Why `files` may not be published on Pages, as one sentence — or null. The
+ * step's limits sit inside the host's: 15,000 files of its 20,000 per
+ * deployment (the rest of the site needs room), 24 MiB of its 25 MiB per file.
+ */
+export function limitProblem(files: ReadonlyArray<{ rel: string; bytes: number }>): string | null {
+ if (files.length > MAX_FILES) {
+ return `${files.length} files to publish, over the step's limit of ${MAX_FILES} (Pages allows 20,000 per deployment)`;
+ }
+ const big = files.find((f) => f.bytes > MAX_FILE_BYTES);
+ if (big) {
+ return `${big.rel} is ${mb(big.bytes)} MiB, over the step's limit of ${mb(MAX_FILE_BYTES)} MiB (Pages allows 25 MiB per file)`;
+ }
+ return null;
+}
+
+/** Where publishSource writes, for a given checkout. */
+export function sourcePublicDir(paths: Paths, override?: string): string {
+ return override ?? process.env.HOMEPAGE_PUBLIC_DIR ?? path.join(paths.monorepoRoot, "homepage", "public");
+}
+
+// The skip key, kept BESIDE the public dir, never in it: it holds the rules
+// hash, and public/ is deployed. It is also what `deployHomepage` checks
+// homepage/out's source against (publishedSourceProblem).
+function statePath(publicDir: string): string {
+ return path.join(path.dirname(publicDir), ".source-publish.json");
+}
+
+type PublishState = {
+ sourceCommit: string;
+ mirrorHead: string;
+ // The rules, the literals and SOURCE_STEP_VERSION.
+ rulesHash: string;
+ // "<label> <version>": an upgrade may rewrite every id, so it re-publishes.
+ filterRepo: string;
+ // gitleaksIdentity(): a newer gitleaks may find what the old one did not,
+ // so it re-publishes, and an out/ it never scanned does not deploy.
+ gitleaks: string;
+ // sourceDigest() over every published file: an out/ (or public/) holding
+ // another publish's mirror or tree beside this manifest does not match.
+ contentDigest: string;
+};
+
+/**
+ * One digest over every file a publish puts on the site, under `root` laid
+ * out as public/ and out/ both are: `source/archilyzer.git/**`,
+ * `source/tree/**`, `source/manifest.json`, `downloads/archilyzer-source.tar.gz`
+ * and `downloads/snapshot.json`. Each file contributes its relative path, size
+ * and sha256 (streamed), in sorted path order; a symlink or a missing piece
+ * contributes a marker, so it can only differ. The /source PAGE and anything
+ * else Next writes beside them are not part of it.
+ */
+export async function sourceDigest(root: string): Promise<string> {
+ const entries: string[] = [];
+ const walk = async (rel: string): Promise<void> => {
+ const abs = path.join(root, rel);
+ const st = await lstat(abs).catch(() => null);
+ if (!st) {
+ entries.push(`${rel}\0missing`);
+ } else if (st.isSymbolicLink()) {
+ entries.push(`${rel}\0symlink`);
+ } else if (st.isDirectory()) {
+ for (const name of await readdir(abs)) await walk(`${rel}/${name}`);
+ } else {
+ entries.push(`${rel}\0${st.size}\0${await sha256File(abs)}`);
+ }
+ };
+ for (const top of [
+ `source/${MIRROR_DIR}`,
+ "source/tree",
+ "source/manifest.json",
+ `downloads/${TARBALL_NAME}`,
+ "downloads/snapshot.json",
+ ]) {
+ await walk(top);
+ }
+ entries.sort((a, b) => (a < b ? -1 : a > b ? 1 : 0));
+ const h = createHash("sha256");
+ for (const e of entries) h.update(`${e}\n`);
+ return h.digest("hex");
+}
+
+/**
+ * Which gitleaks the audit runs: "skipped" (none asked for), "absent" (not on
+ * PATH), else its version line and the sha256 of its binary — the version
+ * line alone is not enough (a distribution build can print the same line for
+ * every release).
+ */
+export async function gitleaksIdentity(bin: string | null, env: NodeJS.ProcessEnv): Promise<string> {
+ if (bin === null) return "skipped";
+ const found = onPath(bin, env.PATH);
+ if (!found) return "absent";
+ const version = await new Promise<string>((resolve) => {
+ execFile(found, ["version"], { env, timeout: 10_000 }, (_err, stdout, stderr) =>
+ resolve(String(stdout || stderr).trim().split("\n")[0] ?? ""),
+ );
+ });
+ const sha = await sha256File(await realpath(found));
+ return `${version || "unknown version"} (${sha.slice(0, 12)})`;
+}
+
+async function readJson(file: string): Promise<unknown> {
+ try {
+ return JSON.parse(await readFile(file, "utf8"));
+ } catch {
+ return null;
+ }
+}
+
+/** The last published manifest in `publicDir`, or null. */
+export async function readPublishedManifest(publicDir: string): Promise<SourceManifest | null> {
+ return parseSourceManifest(await readJson(path.join(publicDir, "source", "manifest.json")));
+}
+
+/** Said, and nothing mirrored, in a checkout without git history. */
+export const NO_REPOSITORY =
+ "no git repository here; nothing to mirror — the /source page will show its empty state";
+
+/**
+ * Remove a publish from `publicDir`: the manifest FIRST (the page never
+ * describes a half-removed tree), the skip key, the mirror and the tree, the
+ * tarball and snapshot.json. Link-safe like the install: a linked
+ * `source/` or `downloads/` (a worktree's, into the primary) goes as a link,
+ * its target untouched. True when there was something to remove.
+ */
+export async function removePublishedSource(publicDir: string): Promise<boolean> {
+ const pubSource = path.join(publicDir, "source");
+ const pubDownloads = path.join(publicDir, "downloads");
+ const had =
+ existsSync(path.join(pubSource, "manifest.json")) ||
+ existsSync(path.join(pubSource, MIRROR_DIR)) ||
+ existsSync(path.join(pubDownloads, TARBALL_NAME));
+ await rm(path.join(pubSource, "manifest.json"), { force: true });
+ await rm(statePath(publicDir), { force: true });
+ // fs.rm reads the path with lstat: a linked public/source goes as a link.
+ await rm(pubSource, { recursive: true, force: true });
+ const linked = await lstat(pubDownloads).then((s) => s.isSymbolicLink(), () => false);
+ if (linked) {
+ await rm(pubDownloads, { force: true });
+ } else {
+ await rm(path.join(pubDownloads, "snapshot.json"), { force: true });
+ await rm(path.join(pubDownloads, TARBALL_NAME), { force: true });
+ }
+ return had;
+}
+
+// The checkout's git common dir, or null when there is no repository here (or
+// no git at all — a docker runtime, a tarball install).
+async function commonDir(ctx: Ctx, cwd: string): Promise<string | null> {
+ const r = await run(ctx, "git", ["rev-parse", "--path-format=absolute", "--git-common-dir"], {
+ cwd,
+ timeoutMs: 30_000,
+ allowFail: true,
+ });
+ if (r.code === 0) return lastLine(r.out);
+ if (!onPath("git", ctx.env.PATH) || r.out.some((l) => /not a git repository/i.test(l))) return null;
+ const tail = r.out.slice(-2).map((l) => maskLiterals(l, ctx.literals)).join(" / ");
+ throw new SourceRefusal(`git rev-parse exited ${r.code}${tail ? `: ${tail}` : ""}`);
+}
+
+// The real path of `p`, or of its deepest existing ancestor with the rest
+// appended: a scratch root may not exist yet, and a symlink must not hide
+// where it lands.
+async function landsAt(p: string): Promise<string> {
+ const abs = path.resolve(p);
+ let head = abs;
+ const rest: string[] = [];
+ for (;;) {
+ try {
+ return path.join(await realpath(head), ...rest);
+ } catch {
+ const up = path.dirname(head);
+ if (up === head) return abs;
+ rest.unshift(path.basename(head));
+ head = up;
+ }
+ }
+}
+
+const within = (child: string, parent: string) =>
+ child === parent || child.startsWith(parent.endsWith(path.sep) ? parent : parent + path.sep);
+
+/**
+ * Why `scratchRoot` may not hold the scratch dir, or null. Inside the checkout
+ * or the public dir, a kept scratch (`--keep-scratch`) could be committed or
+ * published — and it held replace.txt, the scrub rules in plain text.
+ */
+export async function scratchRootProblem(
+ scratchRoot: string,
+ places: ReadonlyArray<[string, string]>,
+): Promise<string | null> {
+ const at = await landsAt(scratchRoot);
+ for (const [what, dir] of places) {
+ if (within(at, await landsAt(dir))) {
+ return `the scratch root ${tildify(scratchRoot)} (ARCHILYZER_SOURCE_SCRATCH) is inside ${what}, where a kept scratch dir could be committed or published — point it outside`;
+ }
+ }
+ return null;
+}
+
+// ── the step ────────────────────────────────────────────────────────────────
+
+/**
+ * Publish the source (see the header). Returns 0 when published, skipped or
+ * checked, 1 when refused or cancelled; the refusal's reason (and the audit's
+ * report) is in the log. Throws only on a bug or an I/O failure.
+ *
+ * A REFUSAL WITHDRAWS THE PREVIOUS PUBLISH. Once the rules are loaded, any
+ * outcome but success — an audit hit, a limit, a missing tool, a cancel, a
+ * crash — removes what the last publish left (removePublishedSource): it was
+ * audited under rules that may not be today's, and a refusal is the best
+ * evidence that they are not. `--check` writes nothing, this included.
+ *
+ * `noRepository: "empty"` (buildHomepage) turns a checkout with no git
+ * repository into an empty /source page and 0; the CLI's default refuses.
+ */
+export async function publishSource(opts: SourcePublishOpts = {}): Promise<number> {
+ const paths = opts.paths ?? getPaths();
+ const onLog = opts.onLog ?? terminalLog;
+ const signal = opts.signal ?? new AbortController().signal;
+ const ctx: Ctx = { onLog, signal, env: cleanGitEnv(opts.env ?? process.env), literals: [] };
+ const publicDir = sourcePublicDir(paths, opts.publicDir);
+ const progress = { rulesLoaded: false };
+ const withdraw = async () => {
+ if (!progress.rulesLoaded || opts.check) return;
+ if (await removePublishedSource(publicDir)) {
+ onLog(
+ "[source] the previous publish was WITHDRAWN (mirror, tree, tarball): it was audited under rules that may not be today's. /source shows its empty state until a publish passes.",
+ );
+ }
+ };
+ let code: number;
+ try {
+ code = await publish(opts, paths, ctx, publicDir, progress);
+ } catch (err) {
+ if (err instanceof Cancelled || signal.aborted) {
+ onLog("[source] cancelled — nothing published");
+ code = 1;
+ } else if (err instanceof SourceRefusal) {
+ // A refusal may name a tree path or quote a child's stderr: masked, so
+ // a literal spanning path components (`a/b`) is never printed.
+ onLog(`[source] REFUSED: ${maskLiterals(err.message, ctx.literals)}`);
+ code = 1;
+ } else {
+ await withdraw().catch(() => {});
+ throw err;
+ }
+ }
+ if (code !== 0) await withdraw();
+ return code;
+}
+
+async function publish(
+ opts: SourcePublishOpts,
+ paths: Paths,
+ ctx: Ctx,
+ publicDir: string,
+ progress: { rulesLoaded: boolean },
+): Promise<number> {
+ const { onLog } = ctx;
+ const started = Date.now();
+ const pubSource = path.join(publicDir, "source");
+ const pubDownloads = path.join(publicDir, "downloads");
+
+ // 1. The private main — or no repository at all (L8: the docker runtime,
+ // a tarball install), where nothing can be mirrored and nothing can leak.
+ const sourceRepo = opts.sourceRepo ?? (await commonDir(ctx, paths.monorepoRoot));
+ if (sourceRepo === null) {
+ onLog(`[source] ${NO_REPOSITORY}`);
+ if (opts.check) return 1;
+ if (await removePublishedSource(publicDir)) onLog("[source] the previous publish was removed.");
+ return opts.noRepository === "empty" ? 0 : 1;
+ }
+ const sourceCommit = await revParse(ctx, sourceRepo, `refs/heads/${SOURCE_BRANCH}^{commit}`);
+
+ // 2. The operator's rules. From here on, a failure withdraws the last publish.
+ const rules = await loadSourceRules({
+ scrubFile: opts.scrubFile ?? paths.sourceScrubFile,
+ denylistFile: opts.denylistFile ?? paths.sourceDenylistFile,
+ homeDir: opts.homeDir,
+ });
+ ctx.literals = rules.literals;
+ progress.rulesLoaded = true;
+
+ // 3. The tools (the filter-repo version is part of the skip key).
+ const filterRepo: FilterRepoChoice =
+ opts.filterRepo && opts.filterRepo.length > 0
+ ? { argv: opts.filterRepo, label: opts.filterRepo.join(" "), version: "(given)" }
+ : await resolveFilterRepo({ env: ctx.env, signal: ctx.signal });
+ const filterRepoId = `${filterRepo.label} ${filterRepo.version}`;
+ const gitleaksBin = opts.gitleaks === undefined ? "gitleaks" : opts.gitleaks;
+ const gitleaksId = await gitleaksIdentity(gitleaksBin, ctx.env);
+
+ // 4. Nothing changed: skip. The key is main, the rules (with the step's
+ // version), both tools, and the published files themselves.
+ if (!opts.force && !opts.check) {
+ const manifest = await readPublishedManifest(publicDir);
+ const state = (await readJson(statePath(publicDir))) as Partial<PublishState> | null;
+ if (
+ manifest?.sourceCommit === sourceCommit &&
+ state?.sourceCommit === sourceCommit &&
+ state.rulesHash === rules.rulesHash &&
+ state.filterRepo === filterRepoId &&
+ state.gitleaks === gitleaksId &&
+ state.mirrorHead === manifest.mirrorHead &&
+ existsSync(path.join(pubSource, MIRROR_DIR, "info", "refs")) &&
+ existsSync(path.join(pubDownloads, TARBALL_NAME)) &&
+ state.contentDigest === (await sourceDigest(publicDir))
+ ) {
+ onLog(`[source] up to date at ${sourceCommit.slice(0, 12)}; skipping (--force to rebuild)`);
+ return 0;
+ }
+ }
+ if (existsSync(path.join(pubSource, "index.html"))) {
+ throw new SourceRefusal(
+ `${tildify(path.join(pubSource, "index.html"))} exists and would replace the /source/ page — remove it`,
+ );
+ }
+ const gitVersion = lastLine((await run(ctx, "git", ["--version"], { cwd: os.tmpdir(), timeoutMs: 30_000 })).out)
+ .replace(/^git version /, "");
+ onLog(`[source] main ${sourceCommit.slice(0, 12)}; git ${gitVersion}; ${filterRepoId}`);
+
+ const scratchRoot = opts.scratchRoot ?? paths.sourceScratchDir;
+ const badRoot = await scratchRootProblem(scratchRoot, [
+ ["the checkout", paths.monorepoRoot],
+ ["the public dir", publicDir],
+ ]);
+ if (badRoot) throw new SourceRefusal(badRoot);
+ await mkdir(scratchRoot, { recursive: true });
+ const scratch = await mkdtemp(path.join(scratchRoot, "archilyzer-source-"));
+ const replace = path.join(scratch, "replace.txt");
+ try {
+ const bare = path.join(scratch, "bare");
+
+ // 5. A fresh bare clone of main alone. --no-local: through upload-pack, not
+ // a copy of the object dir (which would carry every loose leftover).
+ await run(
+ ctx,
+ "git",
+ ["clone", "--no-local", "--bare", "--single-branch", "--no-tags", "--branch", SOURCE_BRANCH, "--quiet", sourceRepo, bare],
+ { cwd: scratch, timeoutMs: 120_000 },
+ );
+ await run(ctx, "git", ["--git-dir", bare, "remote", "remove", "origin"], { cwd: bare, timeoutMs: 30_000 });
+
+ // 6. The rewrite. --force: filter-repo wants a fresh clone with an origin.
+ await writeFile(replace, rules.scrub.lines.join("\n") + "\n", { mode: 0o600 });
+ onLog(`[source] rewriting history (${rules.scrub.lines.length} scrub rule${rules.scrub.lines.length === 1 ? "" : "s"})…`);
+ await run(
+ ctx,
+ filterRepo.argv[0],
+ [
+ ...filterRepo.argv.slice(1),
+ "--force",
+ "--quiet",
+ "--replace-refs",
+ "delete-no-add",
+ "--replace-text",
+ replace,
+ "--replace-message",
+ replace,
+ ],
+ { cwd: bare, timeoutMs: 900_000, echo: true },
+ );
+ // commit-map / ref-map hold the PRIVATE ids.
+ await rm(path.join(bare, "filter-repo"), { recursive: true, force: true });
+
+ // 7. Packed for dumb HTTP.
+ const g = (args: string[], timeoutMs = 60_000) =>
+ run(ctx, "git", ["--git-dir", bare, ...args], { cwd: bare, timeoutMs });
+ await g(["repack", "-a", "-d", "-q", `--max-pack-size=${PACK_SIZE}`], 300_000);
+ await g(["prune-packed"]);
+ await g(["pack-refs", "--all"]);
+ await g(["update-server-info"]);
+ const counts = (await g(["count-objects", "-v"])).out;
+ const loose = Number(/^count:\s*(\d+)/m.exec(counts.join("\n"))?.[1] ?? NaN);
+ if (loose !== 0) throw new SourceRefusal(`the repacked mirror still has ${loose} loose objects`);
+ const packsList = await readFile(path.join(bare, "objects", "info", "packs"), "utf8").catch(() => "");
+ const packs = packsList.split("\n").filter((l) => l.startsWith("P ")).length;
+ if (packs === 0) throw new SourceRefusal("the repacked mirror lists no packs in objects/info/packs");
+ const refs = (await g(["for-each-ref", "--format=%(refname)"])).out.filter((l) => l.trim());
+ if (refs.length !== 1 || refs[0] !== `refs/heads/${SOURCE_BRANCH}`) {
+ throw new SourceRefusal(`the mirror must hold refs/heads/${SOURCE_BRANCH} alone, and holds ${refs.length} refs`);
+ }
+ const head = (await readFile(path.join(bare, "HEAD"), "utf8")).trim();
+ if (head !== `ref: refs/heads/${SOURCE_BRANCH}`) {
+ throw new SourceRefusal(`the mirror's HEAD is not refs/heads/${SOURCE_BRANCH}`);
+ }
+
+ // 8. What it became.
+ const mirrorHead = await revParse(ctx, bare, `refs/heads/${SOURCE_BRANCH}`);
+ const subject = lastLine((await g(["log", "-1", "--format=%s", `refs/heads/${SOURCE_BRANCH}`])).out);
+
+ // 9. THE GATE, over every object.
+ onLog(`[source] auditing ${mirrorHead.slice(0, 12)} for ${rules.literals.length} denied literals…`);
+ const audit = await auditBare(bare, rules.literals, {
+ scratch,
+ onLog,
+ signal: ctx.signal,
+ gitleaks: gitleaksBin,
+ env: ctx.env,
+ });
+ const scrubFile = opts.scrubFile ?? paths.sourceScrubFile;
+ if (audit.hits.length > 0) {
+ for (const l of formatAuditReport(audit, rules.literals, { scrubFile })) onLog(l);
+ return 1;
+ }
+
+ const now = opts.now?.() ?? new Date();
+ const generatedAt = now.toISOString();
+ const stage = path.join(scratch, "stage");
+ const stageSource = path.join(stage, "source");
+ const stageMirror = path.join(stageSource, MIRROR_DIR);
+ const stageTree = path.join(stageSource, "tree");
+ const stageDownloads = path.join(stage, "downloads");
+ await mkdir(stageTree, { recursive: true });
+ await mkdir(stageDownloads, { recursive: true });
+
+ // 10. The raw tree, with its directory pages.
+ const treeTar = path.join(scratch, "tree.tar");
+ await g(["archive", "--format=tar", "-o", treeTar, `refs/heads/${SOURCE_BRANCH}`], 120_000);
+ await run(ctx, "tar", ["-xf", treeTar, "-C", stageTree], { cwd: scratch, timeoutMs: 120_000 });
+ await rm(treeTar, { force: true });
+ const tree = await writeTreeIndexes(stageTree, { mirrorHead, generatedAt });
+
+ // 11. The tarball and its sidecar (the Snapshot shape the Downloads page reads).
+ const tarball = path.join(stageDownloads, TARBALL_NAME);
+ await g(
+ ["archive", "--format=tar.gz", "-9", "--prefix=archilyzer/", "-o", tarball, `refs/heads/${SOURCE_BRANCH}`],
+ 120_000,
+ );
+ const tarBytes = (await stat(tarball)).size;
+ const tarSha = await sha256File(tarball);
+ const snapshotText =
+ JSON.stringify(
+ { generatedAt, commit: mirrorHead, subject, bytes: tarBytes, sha256: tarSha },
+ null,
+ 2,
+ ) + "\n";
+ await writeFile(path.join(stageDownloads, "snapshot.json"), snapshotText);
+
+ // 12. The mirror, from an ALLOWLIST: never config (it names the clone's
+ // origin path), hooks/, description, logs/, filter-repo/, *.rev, *.bitmap.
+ // refs/heads/<branch> is written beside packed-refs so the directory is a
+ // git dir to git itself too (a file:// clone, `source audit`): git wants a
+ // refs/ directory, and an empty one would not survive the deploy.
+ await mkdir(path.join(stageMirror, "info"), { recursive: true });
+ await mkdir(path.join(stageMirror, "objects", "info"), { recursive: true });
+ await mkdir(path.join(stageMirror, "objects", "pack"), { recursive: true });
+ await mkdir(path.join(stageMirror, "refs", "heads"), { recursive: true });
+ for (const f of ["HEAD", "packed-refs", "info/refs", "objects/info/packs"]) {
+ await cp(path.join(bare, f), path.join(stageMirror, f));
+ }
+ await writeFile(path.join(stageMirror, "refs", "heads", SOURCE_BRANCH), `${mirrorHead}\n`);
+ for (const f of await readdir(path.join(bare, "objects", "pack"))) {
+ if (/^pack-[0-9a-f]+\.(pack|idx)$/.test(f)) {
+ await cp(path.join(bare, "objects", "pack", f), path.join(stageMirror, "objects", "pack", f));
+ }
+ }
+ const mirrorFiles = await walkFiles(stageMirror);
+
+ // The manifest, staged with the rest so the file sweep reads it too. It
+ // carries no rules hash and no literal count: `plans/` publishes which
+ // literals the plan put in the files, so either would say whether the
+ // operator added private ones — and the hash would confirm a guess.
+ const staged = await walkFiles(stage);
+ const manifest: SourceManifest = {
+ version: SOURCE_MANIFEST_VERSION,
+ generatedAt,
+ branch: SOURCE_BRANCH,
+ sourceCommit,
+ mirrorHead,
+ subject,
+ files: staged.length,
+ bytes: staged.reduce((n, f) => n + f.bytes, 0),
+ mirror: {
+ files: mirrorFiles.length,
+ bytes: mirrorFiles.reduce((n, f) => n + f.bytes, 0),
+ packs,
+ },
+ tree,
+ tarball: { href: TARBALL_HREF, bytes: tarBytes, sha256: tarSha },
+ cloneUrl: CLONE_URL,
+ treeHref: TREE_HREF,
+ audit: {
+ objects: audit.objects,
+ commits: audit.commits,
+ gitleaks: audit.gitleaks === "clean" ? "clean" : "skipped",
+ },
+ tools: { git: gitVersion, filterRepo: filterRepoId },
+ };
+ const manifestText = JSON.stringify(manifest, null, 2) + "\n";
+ await writeFile(path.join(stageSource, "manifest.json"), manifestText);
+ // Every published file, as it will land: the state carries it, and the
+ // skip and the deploy check recompute it over public/ and out/.
+ const contentDigest = await sourceDigest(stage);
+
+ // 13. THE GATE, over every staged file and path (packs excepted: step 9
+ // read their objects).
+ await auditFiles(stage, rules.literals, { result: audit });
+ if (audit.hits.length > 0) {
+ for (const l of formatAuditReport(audit, rules.literals, { scrubFile })) onLog(l);
+ return 1;
+ }
+ for (const l of formatAuditReport(audit, rules.literals, { scrubFile })) onLog(l);
+
+ // 14. The host's limits.
+ const all = await walkFiles(stage);
+ const tooMuch = limitProblem(all);
+ if (tooMuch) throw new SourceRefusal(maskLiterals(tooMuch, rules.literals));
+ const totalBytes = all.reduce((n, f) => n + f.bytes, 0);
+ const summary =
+ `main ${sourceCommit.slice(0, 12)} as ${mirrorHead.slice(0, 12)}: ${all.length} files, ${mb(totalBytes)} MB ` +
+ `(mirror ${packs} pack${packs === 1 ? "" : "s"}, tree ${tree.dirs} dirs), tarball ${mb(tarBytes)} MB sha256 ${tarSha.slice(0, 12)}`;
+
+ // 15. --check writes nothing.
+ if (opts.check) {
+ onLog(`[source] check passed — would publish ${summary}; nothing written (${elapsed(started)})`);
+ return 0;
+ }
+
+ // 16. Install, link-safe. The manifest goes first and comes back last: a
+ // crash mid-copy leaves the page's empty state, never a manifest over a
+ // half-written tree.
+ await ownDir(pubSource);
+ await rm(path.join(pubSource, "manifest.json"), { force: true });
+ await rm(path.join(pubSource, MIRROR_DIR), { recursive: true, force: true });
+ await rm(path.join(pubSource, "tree"), { recursive: true, force: true });
+ await cp(stageMirror, path.join(pubSource, MIRROR_DIR), { recursive: true });
+ await cp(stageTree, path.join(pubSource, "tree"), { recursive: true });
+ await ownDir(pubDownloads);
+ await copyPublicFile(tarball, path.join(pubDownloads, TARBALL_NAME));
+ await writePublicFile(path.join(pubDownloads, "snapshot.json"), snapshotText);
+ const state: PublishState = {
+ sourceCommit,
+ mirrorHead,
+ rulesHash: rules.rulesHash,
+ filterRepo: filterRepoId,
+ gitleaks: gitleaksId,
+ contentDigest,
+ };
+ await writeFile(statePath(publicDir), JSON.stringify(state, null, 2) + "\n");
+ await writePublicFile(path.join(pubSource, "manifest.json"), manifestText);
+
+ // 17.
+ onLog(`[source] published ${summary} (${elapsed(started)})`);
+ return 0;
+ } finally {
+ if (opts.keepScratch) {
+ // The scrub rules in plain text never outlive the run.
+ await rm(replace, { force: true });
+ onLog(`[source] scratch kept at ${maskLiterals(tildify(scratch), ctx.literals)} (replace.txt, the scrub rules, deleted)`);
+ } else {
+ await rm(scratch, { recursive: true, force: true });
+ }
+ }
+}
+
+function elapsed(started: number): string {
+ return `${Math.round((Date.now() - started) / 1000)} s`;
+}
+
+/**
+ * `build homepage --no-source`: remove what an earlier publish left, because
+ * it was audited against the rules of ITS day. The pages then show their
+ * empty states.
+ */
+export async function clearPublishedSource(
+ opts: PublishOpts & { publicDir?: string } = {},
+): Promise<void> {
+ const paths = opts.paths ?? getPaths();
+ const onLog = opts.onLog ?? terminalLog;
+ const had = await removePublishedSource(sourcePublicDir(paths, opts.publicDir));
+ onLog(
+ had
+ ? "[notice] --no-source: the previously published source (mirror, tree, tarball) was removed — this build ships none.\n"
+ : "[notice] --no-source: no source published in this build.\n",
+ );
+}
+
+/**
+ * Why homepage/out's source may not be deployed, as one sentence — or null.
+ * `deployHomepage` asks before every deploy, production and preview alike: a
+ * deploy-only ships `out/` as the last build left it, and that build's source
+ * was audited under the rules of its day. It deploys only when the last
+ * publish (the skip key beside public/) was made under TODAY's rules and step
+ * (`rulesHash`), of TODAY's `main`, and is the one in `out/` (its mirror head,
+ * its tarball). An `out/` whose /source page shows the empty state — a
+ * `--no-source` build — deploys as it always has; one with no /source page at
+ * all (a refused build) does not.
+ */
+export async function publishedSourceProblem(
+ paths: Paths,
+ outDir: string,
+ opts: {
+ publicDir?: string;
+ scrubFile?: string;
+ denylistFile?: string;
+ homeDir?: string;
+ sourceRepo?: string;
+ env?: NodeJS.ProcessEnv;
+ // The gitleaks the audit would run (default "gitleaks"; null: none).
+ gitleaks?: string | null;
+ } = {},
+): Promise<string | null> {
+ const outSource = path.join(outDir, "source");
+ const outTarball = path.join(outDir, "downloads", TARBALL_NAME);
+ const rebuild = "run `archilyzer build homepage` (it re-audits), then deploy";
+ // The /source/ PAGE is always in a finished build (it renders its empty
+ // state without a publish). Without it, the build's source step refused —
+ // it took out/source away — or the build predates the page.
+ if (!existsSync(path.join(outSource, "index.html"))) {
+ return `homepage/out has no /source page (its source step refused, or the build predates it) — ${rebuild}`;
+ }
+ const artefacts = [
+ path.join(outSource, "manifest.json"),
+ path.join(outSource, MIRROR_DIR),
+ path.join(outSource, "tree"),
+ outTarball,
+ ];
+ // A `--no-source` build: the page's empty state and nothing else.
+ if (!artefacts.some((a) => existsSync(a))) return null;
+ const manifest = parseSourceManifest(await readJson(path.join(outSource, "manifest.json")));
+ if (!manifest) return `homepage/out holds a source publish without a valid manifest — ${rebuild}`;
+ const state = (await readJson(statePath(sourcePublicDir(paths, opts.publicDir)))) as Partial<PublishState> | null;
+ if (!state?.rulesHash || !state.mirrorHead || !state.sourceCommit || !state.contentDigest || !state.gitleaks) {
+ return `homepage/out's source has no record of the rules it was audited under — ${rebuild}`;
+ }
+ let rules: SourceRules;
+ try {
+ rules = await loadSourceRules({
+ scrubFile: opts.scrubFile ?? paths.sourceScrubFile,
+ denylistFile: opts.denylistFile ?? paths.sourceDenylistFile,
+ homeDir: opts.homeDir,
+ });
+ } catch (err) {
+ if (err instanceof SourceRefusal) return `${err.message}; homepage/out's source cannot be checked`;
+ throw err;
+ }
+ if (state.rulesHash !== rules.rulesHash) {
+ return `homepage/out's source was audited under other rules (or an older version of the step) — ${rebuild}`;
+ }
+ if (state.mirrorHead !== manifest.mirrorHead) {
+ return `homepage/out's source (${manifest.mirrorHead.slice(0, 12)}) is not the last publish (${state.mirrorHead.slice(0, 12)}) — ${rebuild}`;
+ }
+ const ctx: Ctx = {
+ onLog: () => {},
+ signal: new AbortController().signal,
+ env: cleanGitEnv(opts.env ?? process.env),
+ literals: rules.literals,
+ };
+ let main: string | null = null;
+ try {
+ const repo = opts.sourceRepo ?? (await commonDir(ctx, paths.monorepoRoot));
+ main = repo ? await revParse(ctx, repo, `refs/heads/${SOURCE_BRANCH}^{commit}`) : null;
+ } catch (err) {
+ if (!(err instanceof SourceRefusal)) throw err;
+ }
+ if (main === null) return `homepage/out holds a source publish, and main cannot be read here — ${rebuild}`;
+ if (main !== state.sourceCommit || main !== manifest.sourceCommit) {
+ return `homepage/out's source mirrors main at ${manifest.sourceCommit.slice(0, 12)}, and main is now ${main.slice(0, 12)} — ${rebuild}`;
+ }
+ if (existsSync(outTarball) && (await sha256File(outTarball)) !== manifest.tarball.sha256) {
+ return `homepage/out's source tarball is not the one its manifest describes — ${rebuild}`;
+ }
+ const env = cleanGitEnv(opts.env ?? process.env);
+ const gitleaks = await gitleaksIdentity(opts.gitleaks === undefined ? "gitleaks" : opts.gitleaks, env);
+ if (state.gitleaks !== gitleaks) {
+ return `homepage/out's source was scanned by another gitleaks (${state.gitleaks}; this machine has ${gitleaks}) — ${rebuild}`;
+ }
+ // Last, and the one that binds every byte: the mirror, the tree, the
+ // manifest and both downloads, against the digest of what was audited.
+ if ((await sourceDigest(outDir)) !== state.contentDigest) {
+ return `homepage/out's source files are not the ones that were audited (a mixed or edited out/) — ${rebuild}`;
+ }
+ return null;
+}
+
+// ── `archilyzer source audit` ───────────────────────────────────────────────
+
+/**
+ * The gate alone, over any git dir — by default the published mirror; the
+ * rollout runs it on a LIVE clone. 0 clean, 1 hits (the redacted report is
+ * logged) or refused.
+ */
+export async function auditSource(
+ opts: PublishOpts & {
+ gitDir?: string;
+ publicDir?: string;
+ scrubFile?: string;
+ denylistFile?: string;
+ gitleaks?: string | null;
+ scratchRoot?: string;
+ env?: NodeJS.ProcessEnv;
+ homeDir?: string;
+ } = {},
+): Promise<number> {
+ const paths = opts.paths ?? getPaths();
+ const onLog = opts.onLog ?? terminalLog;
+ const signal = opts.signal ?? new AbortController().signal;
+ const env = cleanGitEnv(opts.env ?? process.env);
+ const gitDir = path.resolve(
+ opts.gitDir ?? path.join(sourcePublicDir(paths, opts.publicDir), "source", MIRROR_DIR),
+ );
+ const scrubFile = opts.scrubFile ?? paths.sourceScrubFile;
+ let scratch: string | null = null;
+ let literals: readonly Literal[] = [];
+ try {
+ const rules = await loadSourceRules({
+ scrubFile,
+ denylistFile: opts.denylistFile ?? paths.sourceDenylistFile,
+ homeDir: opts.homeDir,
+ });
+ literals = rules.literals;
+ const ctx: Ctx = { onLog, signal, env, literals: rules.literals };
+ const head = await revParse(ctx, gitDir, "HEAD");
+ const scratchRoot = opts.scratchRoot ?? paths.sourceScratchDir;
+ await mkdir(scratchRoot, { recursive: true });
+ scratch = await mkdtemp(path.join(scratchRoot, "archilyzer-source-audit-"));
+ onLog(`[source] auditing ${maskLiterals(tildify(gitDir), literals)} (HEAD ${head.slice(0, 12)}) for ${rules.literals.length} denied literals…`);
+ const audit = await auditBare(gitDir, rules.literals, {
+ scratch,
+ onLog,
+ signal,
+ gitleaks: opts.gitleaks === undefined ? "gitleaks" : opts.gitleaks,
+ env,
+ });
+ for (const l of formatAuditReport(audit, rules.literals, { scrubFile })) onLog(l);
+ return audit.hits.length > 0 ? 1 : 0;
+ } catch (err) {
+ if (err instanceof Cancelled || signal.aborted) {
+ onLog("[source] cancelled");
+ return 1;
+ }
+ if (err instanceof SourceRefusal) {
+ onLog(`[source] REFUSED: ${maskLiterals(err.message, literals)}`);
+ return 1;
+ }
+ throw err;
+ } finally {
+ if (scratch) await rm(scratch, { recursive: true, force: true });
+ }
+}
diff --git a/common/publish/sourceAudit.test.ts b/common/publish/sourceAudit.test.ts
@@ -0,0 +1,195 @@
+import { test, after } from "node:test";
+import assert from "node:assert/strict";
+import { mkdirSync, mkdtempSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { gzipSync } from "node:zlib";
+import { execFileSync } from "node:child_process";
+import {
+ SourceRefusal,
+ auditBare,
+ auditFiles,
+ auditObjects,
+ formatAuditReport,
+ literalLabel,
+ maskLiterals,
+ objectField,
+ parseDenylist,
+ runGitleaks,
+ scanBuffer,
+ treeEntryNames,
+ type Literal,
+} from "./sourceAudit";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// The source mirror's gate (sourceAudit.ts): the literal list, the byte scan,
+// the redaction, the object walk over a real temp repo, the staged-file sweep
+// and gitleaks' absence. Every repo is built in the OS temp dir; the git
+// variables a hook or a wrapper might export are cleared first so nothing here
+// can reach the real repo.
+for (const key of ["GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE", "GIT_PREFIX"]) {
+ delete process.env[key];
+}
+
+const TMP = mkdtempSync(path.join(os.tmpdir(), "source-audit-"));
+after(() => rmSync(TMP, { recursive: true, force: true }));
+
+// The planted literal every test hunts for. Never a real name.
+const PLANTED = "plantedhome";
+const lits = (...xs: string[]): Literal[] =>
+ xs.map((x, i) => ({ bytes: Buffer.from(x), ci: false, from: `denylist line ${i + 1}` }));
+
+let n = 0;
+function repo(): string {
+ const dir = path.join(TMP, `r${n++}`);
+ mkdirSync(dir);
+ const git = (...args: string[]) => execFileSync("git", args, { cwd: dir, stdio: "pipe" });
+ git("init", "-q", "-b", "main");
+ git("config", "user.name", "audit test");
+ git("config", "user.email", "audit@example.invalid");
+ git("config", "commit.gpgsign", "false");
+ return dir;
+}
+function commit(dir: string, files: Record<string, string>, message: string, author?: string): void {
+ for (const [f, text] of Object.entries(files)) {
+ mkdirSync(path.dirname(path.join(dir, f)), { recursive: true });
+ writeFileSync(path.join(dir, f), text);
+ }
+ const git = (...args: string[]) => execFileSync("git", args, { cwd: dir, stdio: "pipe" });
+ git("add", "-A");
+ git("commit", "-q", "-m", message, ...(author ? ["--author", author] : []));
+}
+
+test("the denylist: one literal a line, i: for any case, comments and blanks skipped, duplicates once", () => {
+ const list = parseDenylist(`# a comment\n\n${PLANTED}\ni:MiXeD\n ${PLANTED} \r\ni:\n`);
+ assert.equal(list.length, 2);
+ assert.deepEqual(list[0], { bytes: Buffer.from(PLANTED), ci: false, from: "denylist line 3" });
+ assert.deepEqual(list[1], { bytes: Buffer.from("mixed"), ci: true, from: "denylist line 4" });
+ // Named by where it was written and its length — none of its characters.
+ assert.equal(literalLabel(list, 0), `denylist line 3 (len ${PLANTED.length})`);
+ assert.equal(literalLabel(list, 1), "denylist line 4 (len 5, any case)");
+});
+
+test("a byte-order mark and CRLF endings never become part of a literal", () => {
+ const list = parseDenylist(`\uFEFF${PLANTED}\r\ni:Other\r\n`);
+ assert.deepEqual(list.map((l) => l.bytes.toString()), [PLANTED, "other"]);
+ assert.equal(list[1].from, "denylist line 2");
+});
+
+test("scanBuffer finds every occurrence; an i: literal matches any ASCII case", () => {
+ const buf = Buffer.from(`a ${PLANTED} b PlantedHome c mixed MIXED`);
+ const hits = scanBuffer(buf, [...lits(PLANTED), ...parseDenylist("i:plantedHOME\ni:mixed")]);
+ assert.deepEqual(
+ hits.map((h) => [h.lit, h.offset]),
+ [[0, 2], [1, 2], [1, 16], [2, 30], [2, 36]],
+ );
+});
+
+test("maskLiterals merges overlapping occurrences into one mark and leaves the rest", () => {
+ const list = lits(PLANTED, "homeplanted");
+ assert.equal(maskLiterals(`/srv/${PLANTED}/x ${PLANTED}`, list), "/srv/[REDACTED]/x [REDACTED]");
+ assert.equal(maskLiterals(`a plantedhomeplanted b`, list), "a [REDACTED] b");
+ assert.equal(maskLiterals("nothing here", list), "nothing here");
+});
+
+test("objectField names a commit's header field or its message, never its bytes", () => {
+ const commit = Buffer.from(
+ "tree 1234\nparent 5678\nauthor A <a@x> 1 +0000\ncommitter C <c@x> 1 +0000\ngpgsig -----BEGIN\n sig line\n -----END\n\nthe message\n",
+ );
+ const at = (s: string) => commit.indexOf(s);
+ assert.equal(objectField(commit, at("a@x")), "author");
+ assert.equal(objectField(commit, at("c@x")), "committer");
+ assert.equal(objectField(commit, at("sig line")), "gpgsig");
+ assert.equal(objectField(commit, at("message")), "message");
+ assert.equal(objectField(commit, at("5678")), "parent");
+});
+
+test("a tree's entry names are what is scanned, not its binary ids", () => {
+ const id = Buffer.alloc(20, 0x70); // 'p' x20: would match "ppp" if ids were read
+ const tree = Buffer.concat([
+ Buffer.from("100644 a.txt\0"), id,
+ Buffer.from(`40000 ${PLANTED}\0`), id,
+ ]);
+ assert.equal(treeEntryNames(tree, 20).toString("latin1"), `a.txt\0${PLANTED}\0`);
+ assert.equal(scanBuffer(treeEntryNames(tree, 20), lits("ppp")).length, 0);
+});
+
+test("the object walk finds a literal in a blob, a commit message, an author line and a tree entry name", async () => {
+ const dir = repo();
+ commit(dir, { "README.md": "clean\n" }, "first");
+ commit(dir, { "notes.txt": `path /srv/${PLANTED}/data\n` }, "a blob carries it");
+ commit(dir, { "README.md": "clean 2\n" }, `the message says /srv/${PLANTED}`);
+ commit(dir, { "README.md": "clean 3\n" }, "an identity", `Planted <${PLANTED}@example.invalid>`);
+ commit(dir, { [`dir-${PLANTED}/x.txt`]: "x\n" }, "a name");
+ const result = await auditObjects(path.join(dir, ".git"), lits(PLANTED));
+ const kinds = result.hits.map((h) => `${h.kind}:${h.field ?? ""}`).sort();
+ // The message and the author line are one commit object each.
+ // The tree's hit is its second entry: README.md, dir-…, notes.txt.
+ assert.deepEqual(kinds, ["blob:", "commit:author", "commit:message", "tree:entry 2"]);
+ assert.equal(result.commits, 5);
+ const objects = execFileSync("git", ["count-objects", "-v"], { cwd: dir }).toString();
+ assert.equal(result.objects, Number(/^count: (\d+)/m.exec(objects)![1]), "every object was read");
+ // The report names the literal by where it was written, and prints no
+ // byte of any object: not the literal, not what sits beside it.
+ const report = formatAuditReport(result, lits(PLANTED), {
+ scrubFile: path.join(os.homedir(), ".config", "archilyzer", "source-scrub.txt"),
+ }).join("\n");
+ assert.ok(!report.includes(PLANTED), report);
+ for (const beside of ["/srv/", "/data", "example.invalid", "Planted <", "the message says"]) {
+ assert.ok(!report.includes(beside), `the report carries "${beside}" from an object`);
+ }
+ assert.match(report, /AUDIT REFUSED: 4 hits/);
+ assert.match(report, /denylist line 1 \(len 11\): 1 in blob, 2 in commits, 1 in tree/);
+ assert.match(report, /commit [0-9a-f]{12} \(author, byte \d+\): denylist line 1 \(len 11\)/);
+ assert.match(report, /commit [0-9a-f]{12} \(message, byte \d+\): denylist line 1 \(len 11\)/);
+ assert.match(report, /blob [0-9a-f]{12} \(byte 10\): denylist line 1 \(len 11\)/);
+ assert.match(report, /tree [0-9a-f]{12} \(entry 2\): denylist line 1 \(len 11\)/);
+ assert.match(report, /add a rule to ~\/\.config\/archilyzer\/source-scrub\.txt or drop the file from history, then re-run\.$/);
+});
+
+test("a clean repo audits clean, with gitleaks skipped when it is not on PATH", async () => {
+ const dir = repo();
+ commit(dir, { "a.txt": "nothing to see\n" }, "clean");
+ const logs: string[] = [];
+ const result = await auditBare(path.join(dir, ".git"), lits(PLANTED), {
+ scratch: TMP,
+ onLog: (l) => logs.push(l),
+ signal: new AbortController().signal,
+ gitleaks: "gitleaks-not-installed-here",
+ });
+ assert.equal(result.hits.length, 0);
+ assert.equal(result.gitleaks, "skipped");
+ assert.match(logs.join("\n"), /WARNING: gitleaks-not-installed-here is not on PATH — the secret scan is skipped/);
+ assert.match(formatAuditReport(result, lits(PLANTED), { scrubFile: "/x" })[0], /^\[source\] audit clean: \d+ objects \(1 commit\)/);
+ const skipped = await runGitleaks(path.join(dir, ".git"), {
+ bin: "gitleaks",
+ scratch: TMP,
+ literals: [],
+ onLog: () => {},
+ signal: new AbortController().signal,
+ env: { PATH: path.join(TMP, "empty-path") },
+ });
+ assert.equal(skipped.status, "skipped");
+});
+
+test("the staged-file sweep reads contents, names and gzip'd bytes decompressed, skips packs, refuses a symlink", async () => {
+ const dir = path.join(TMP, "stage");
+ mkdirSync(path.join(dir, "tree", `d-${PLANTED}`), { recursive: true });
+ writeFileSync(path.join(dir, "tree", "ok.txt"), "fine\n");
+ writeFileSync(path.join(dir, "tree", "bad.txt"), `x ${PLANTED} y\n`);
+ writeFileSync(path.join(dir, "tree", `d-${PLANTED}`, "f.txt"), "fine\n");
+ writeFileSync(path.join(dir, "t.tar.gz"), gzipSync(Buffer.from(`inside ${PLANTED}`)));
+ writeFileSync(path.join(dir, "pack-1.pack"), PLANTED); // the object walk's job
+ const result = await auditFiles(dir, lits(PLANTED));
+ assert.deepEqual(result.hits.map((h) => `${h.where} ${h.field}`).sort(), [
+ "t.tar.gz decompressed",
+ "tree/bad.txt contents",
+ "tree/d-[REDACTED] path",
+ "tree/d-[REDACTED]/f.txt path",
+ ]);
+ assert.equal(result.files, 5);
+ symlinkSync("/etc/hostname", path.join(dir, "tree", "link"));
+ await assert.rejects(auditFiles(dir, lits(PLANTED)), (err) => err instanceof SourceRefusal && /symlink/.test(err.message));
+});
diff --git a/common/publish/sourceAudit.ts b/common/publish/sourceAudit.ts
@@ -0,0 +1,656 @@
+// The source mirror's gate: does anything about to be published carry a
+// literal the operator denied? (`archilyzer source publish`, source.ts.)
+//
+// Three sweeps, all of them byte searches for the SAME literal list (the
+// denylist plus every scrub rule's left side — source.ts builds it):
+// - auditObjects: EVERY object in a git dir, read in one
+// `git cat-file --batch-all-objects --batch` stream — blobs whole, commits
+// and tags whole (message AND the author/committer/tagger lines), trees by
+// entry NAME. Reachable or not: a leftover of the private history in a
+// pack would be published with it, so it is read with it.
+// - auditFiles: every file staged beside the mirror (the raw tree, the index
+// pages, the tarball decompressed, the ref files, the manifest), and every
+// staged PATH.
+// - runGitleaks: the secret scanner over the mirror's history, when it is
+// installed (a WARNING, not a refusal, when it is not).
+//
+// THE REPORT NEVER PRINTS A LITERAL, AND NO BYTES OF THE OBJECT AROUND ONE. A
+// literal is named by where the operator wrote it — `denylist line 3 (len 5)`,
+// `scrub line 2 lhs (len 11)`, `built-in home rule (len 11)` — never by any of
+// its characters. A hit is named by its object (kind, id, the path when one
+// is known), the byte offset, and for a commit or tag the field it sits in
+// (author, committer, tagger, message): a context window would print what sits
+// NEXT to a denied name (a surname, the rest of an address), which is exactly
+// as private. Paths and child lines quoted in the log are masked
+// (`[REDACTED]`).
+
+import { spawn } from "node:child_process";
+import { existsSync, statSync, accessSync, constants } from "node:fs";
+import { lstat, readdir, readFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { gunzipSync } from "node:zlib";
+import { runChildIntoLog } from "../jobs/runChild";
+
+/** A refusal: the step stops, publishes nothing, and says why. */
+export class SourceRefusal extends Error {
+ constructor(message: string) {
+ super(message);
+ this.name = "SourceRefusal";
+ }
+}
+
+// ── literals ────────────────────────────────────────────────────────────────
+
+/**
+ * One denied literal. `ci` literals match ASCII case-insensitively: their
+ * `bytes` are stored folded, and are searched for in a folded copy. `from`
+ * says where the operator wrote it (`denylist line 3`), which is how a report
+ * names it.
+ */
+export type Literal = { bytes: Buffer; ci: boolean; from?: string };
+
+const LOWER_A = 0x61;
+const UPPER_A = 0x41;
+const UPPER_Z = 0x5a;
+
+/** A copy of `buf` with A–Z folded to a–z (every other byte as it is). */
+export function foldAscii(buf: Uint8Array): Buffer {
+ const out = Buffer.from(buf);
+ for (let i = 0; i < out.length; i++) {
+ const b = out[i];
+ if (b >= UPPER_A && b <= UPPER_Z) out[i] = b - UPPER_A + LOWER_A;
+ }
+ return out;
+}
+
+/**
+ * An operator file's text as lines: a UTF-8 byte-order mark at the start is
+ * dropped and CRLF endings become LF. An editor that writes either must not
+ * turn the first rule (and the denial it implies) into one that never matches.
+ */
+export function operatorLines(text: string): string[] {
+ return text.replace(/^\uFEFF/, "").split("\n").map((l) => l.replace(/\r$/, ""));
+}
+
+/**
+ * The denylist file: one literal per line. `i:` in front makes it ASCII
+ * case-insensitive. Blank lines and lines starting with `#` are skipped;
+ * surrounding whitespace is trimmed. Duplicates collapse (the first line
+ * names it).
+ */
+export function parseDenylist(text: string): Literal[] {
+ const out: Literal[] = [];
+ operatorLines(text).forEach((raw, i) => {
+ const line = raw.trim();
+ if (line === "" || line.startsWith("#")) return;
+ const from = `denylist line ${i + 1}`;
+ if (line.startsWith("i:")) {
+ const lit = line.slice(2).trim();
+ if (lit) out.push({ bytes: foldAscii(Buffer.from(lit, "utf8")), ci: true, from });
+ } else {
+ out.push({ bytes: Buffer.from(line, "utf8"), ci: false, from });
+ }
+ });
+ return dedupeLiterals(out);
+}
+
+export function dedupeLiterals(list: Literal[]): Literal[] {
+ const seen = new Set<string>();
+ const out: Literal[] = [];
+ for (const l of list) {
+ const key = `${l.ci ? "i" : "x"}:${l.bytes.toString("hex")}`;
+ if (seen.has(key) || l.bytes.length === 0) continue;
+ seen.add(key);
+ out.push(l);
+ }
+ return out;
+}
+
+/**
+ * How a literal is named in a report: where it was written and its length —
+ * never any of its characters (a first letter and a length all but spell a
+ * short first name).
+ */
+export function literalLabel(literals: readonly Literal[], index: number): string {
+ const l = literals[index];
+ return `${l.from ?? `literal ${index + 1}`} (len ${l.bytes.length}${l.ci ? ", any case" : ""})`;
+}
+
+// ── scanning ────────────────────────────────────────────────────────────────
+
+export type Hit = { lit: number; offset: number; length: number };
+
+/** Every occurrence of every literal in `buf`, by offset. */
+export function scanBuffer(buf: Buffer, literals: readonly Literal[]): Hit[] {
+ const hits: Hit[] = [];
+ let folded: Buffer | null = null;
+ literals.forEach((l, lit) => {
+ let hay = buf;
+ if (l.ci) {
+ folded ??= foldAscii(buf);
+ hay = folded;
+ }
+ let at = hay.indexOf(l.bytes);
+ while (at !== -1) {
+ hits.push({ lit, offset: at, length: l.bytes.length });
+ at = hay.indexOf(l.bytes, at + 1);
+ }
+ });
+ return hits.sort((a, b) => a.offset - b.offset || a.lit - b.lit);
+}
+
+/** `text` with every occurrence of every literal replaced by `[REDACTED]`. */
+export function maskLiterals(text: string, literals: readonly Literal[]): string {
+ const buf = Buffer.from(text, "utf8");
+ const hits = scanBuffer(buf, literals);
+ if (hits.length === 0) return text;
+ // Merge overlapping occurrences into runs, then splice one mark per run.
+ const runs: Array<[number, number]> = [];
+ for (const h of hits) {
+ const last = runs[runs.length - 1];
+ if (last && h.offset <= last[1]) last[1] = Math.max(last[1], h.offset + h.length);
+ else runs.push([h.offset, h.offset + h.length]);
+ }
+ const parts: Buffer[] = [];
+ let pos = 0;
+ for (const [a, b] of runs) {
+ parts.push(buf.subarray(pos, a), Buffer.from("[REDACTED]"));
+ pos = b;
+ }
+ parts.push(buf.subarray(pos));
+ return Buffer.concat(parts).toString("utf8");
+}
+
+// ── the audit's result ──────────────────────────────────────────────────────
+
+export type HitKind = "blob" | "commit" | "tree" | "tag" | "file" | "gitleaks";
+const KIND_ORDER: readonly HitKind[] = ["blob", "commit", "tree", "tag", "file", "gitleaks"];
+
+export type AuditHit = {
+ kind: HitKind;
+ // An object id (blob/commit/tree/tag) — with the blob's path in history
+ // once known — a path relative to the staged dir (file), or the finding
+ // (gitleaks). Paths are masked.
+ where: string;
+ // A literal's index, or -1 for a gitleaks finding.
+ lit: number;
+ // The byte offset of the hit in the object or file (-1 for gitleaks).
+ offset: number;
+ // Where in the object, without its bytes: a commit's or tag's header field
+ // (`author`, `committer`, `tagger`, …) or `message`; a tree's `entry N`.
+ field?: string;
+};
+
+export type AuditResult = {
+ literals: number;
+ objects: number;
+ commits: number;
+ files: number;
+ gitleaks: "clean" | "skipped" | "not run";
+ hits: AuditHit[];
+};
+
+// Only this many hits get a line of their own: a literal every commit carries
+// (the gate's own planted `Co-Authored-By`) would otherwise print thousands.
+const LISTED = 20;
+
+export function emptyAudit(literals: number): AuditResult {
+ return { literals, objects: 0, commits: 0, files: 0, gitleaks: "not run", hits: [] };
+}
+
+function record(
+ result: AuditResult,
+ kind: HitKind,
+ where: string,
+ hits: Hit[],
+ fieldOf?: (offset: number) => string,
+ // A tree's hits are offsets into its joined entry NAMES, not the object:
+ // the entry number says where, and the offset is left out.
+ withOffset = true,
+): void {
+ for (const h of hits) {
+ result.hits.push({
+ kind,
+ where,
+ lit: h.lit,
+ offset: withOffset ? h.offset : -1,
+ ...(fieldOf ? { field: fieldOf(h.offset) } : {}),
+ });
+ }
+}
+
+/**
+ * Which part of a commit or tag object `offset` falls in: `message` past the
+ * blank line that ends the headers, else the header line's keyword (`author`,
+ * `committer`, `tagger`, `tree`, `parent`, …) — a word git wrote, never the
+ * operator's bytes.
+ */
+export function objectField(data: Buffer, offset: number): string {
+ const end = data.indexOf("\n\n");
+ if (end !== -1 && offset > end) return "message";
+ let lineStart = data.lastIndexOf(0x0a, offset - 1) + 1;
+ // A continuation line (a signature, a mergetag) begins with a space: walk
+ // back to the header it continues.
+ while (lineStart > 0 && data[lineStart] === 0x20) {
+ lineStart = data.lastIndexOf(0x0a, lineStart - 2) + 1;
+ }
+ const sp = data.indexOf(0x20, lineStart);
+ const word = data.subarray(lineStart, sp === -1 ? lineStart : sp).toString("latin1");
+ return /^[a-z][a-z-]{0,15}$/.test(word) ? word : "header";
+}
+
+/** A tree hit's entry number (1-based) in the NUL-separated names buffer. */
+function treeEntry(names: Buffer, offset: number): string {
+ let n = 1;
+ for (let i = names.indexOf(0, 0); i !== -1 && i < offset; i = names.indexOf(0, i + 1)) n++;
+ return `entry ${n}`;
+}
+
+// ── the object walk ─────────────────────────────────────────────────────────
+
+/** The git environment a child must not inherit: it would point git elsewhere. */
+export function cleanGitEnv(env: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEnv {
+ const out: NodeJS.ProcessEnv = { ...env };
+ for (const k of Object.keys(out)) {
+ if (/^GIT_(DIR|WORK_TREE|INDEX_FILE|PREFIX|OBJECT_DIRECTORY|ALTERNATE_OBJECT_DIRECTORIES|COMMON_DIR|NAMESPACE|CEILING_DIRECTORIES|CONFIG|CONFIG_PARAMETERS|CONFIG_COUNT)$/.test(k)) {
+ out[k] = undefined;
+ }
+ }
+ return out;
+}
+
+/** A tree object's entry names, NUL-separated (a literal never holds a NUL). */
+export function treeEntryNames(data: Buffer, hashBytes: number): Buffer {
+ const names: Buffer[] = [];
+ let i = 0;
+ while (i < data.length) {
+ const sp = data.indexOf(0x20, i);
+ if (sp === -1) break;
+ const nul = data.indexOf(0x00, sp + 1);
+ if (nul === -1) break;
+ names.push(data.subarray(sp + 1, nul), Buffer.from([0]));
+ i = nul + 1 + hashBytes;
+ }
+ return Buffer.concat(names);
+}
+
+/**
+ * Read every object in `gitDir` once, scan it, and count. One child, one
+ * stream: `cat-file --batch` frames each object as `<oid> <type> <size>\n`,
+ * then the bytes, then `\n`.
+ */
+export async function auditObjects(
+ gitDir: string,
+ literals: readonly Literal[],
+ opts: { signal?: AbortSignal; env?: NodeJS.ProcessEnv; result?: AuditResult } = {},
+): Promise<AuditResult> {
+ const result = opts.result ?? emptyAudit(literals.length);
+ const child = spawn(
+ "git",
+ ["--git-dir", gitDir, "cat-file", "--batch-all-objects", "--unordered", "--batch"],
+ { stdio: ["ignore", "pipe", "pipe"], env: cleanGitEnv(opts.env), signal: opts.signal },
+ );
+ let stderr = "";
+ child.stderr.on("data", (c: Buffer) => {
+ if (stderr.length < 4000) stderr += c.toString("utf8");
+ });
+ const exited = new Promise<number>((resolve, reject) => {
+ child.once("error", reject);
+ child.once("close", (code) => resolve(code ?? 1));
+ });
+ exited.catch(() => {}); // awaited below; a throw in the loop must not orphan it
+
+ let state: "header" | "body" | "lf" = "header";
+ let headerParts: Buffer[] = [];
+ let oid = "";
+ let type = "";
+ let body = Buffer.alloc(0);
+ let filled = 0;
+
+ const onObject = () => {
+ result.objects++;
+ if (literals.length === 0) return;
+ if (type === "tree") {
+ const names = treeEntryNames(body, oid.length / 2);
+ const hits = scanBuffer(names, literals);
+ if (hits.length) record(result, "tree", oid, hits, (o) => treeEntry(names, o), false);
+ return;
+ }
+ const hits = scanBuffer(body, literals);
+ if (hits.length === 0) return;
+ if (type === "commit" || type === "tag") {
+ const data = body;
+ record(result, type, oid, hits, (o) => objectField(data, o));
+ } else {
+ record(result, "blob", oid, hits);
+ }
+ };
+
+ let drained = false;
+ try {
+ for await (const chunk of child.stdout as AsyncIterable<Buffer>) {
+ let i = 0;
+ // One pass per framing step; the three steps run in order inside one
+ // turn, so an empty object goes header -> body -> newline at once.
+ while (i < chunk.length) {
+ if (state === "header") {
+ const nl = chunk.indexOf(0x0a, i);
+ if (nl === -1) {
+ headerParts.push(chunk.subarray(i));
+ break;
+ }
+ headerParts.push(chunk.subarray(i, nl));
+ i = nl + 1;
+ const header = Buffer.concat(headerParts).toString("latin1");
+ headerParts = [];
+ const [o, t, size] = header.split(" ");
+ if (t === "missing" || size === undefined) {
+ throw new SourceRefusal(`git cat-file: unexpected header "${header.slice(0, 80)}"`);
+ }
+ oid = o;
+ type = t;
+ body = Buffer.allocUnsafe(Number(size));
+ filled = 0;
+ state = "body";
+ if (type === "commit") result.commits++;
+ }
+ if (state === "body") {
+ const n = Math.min(body.length - filled, chunk.length - i);
+ chunk.copy(body, filled, i, i + n);
+ filled += n;
+ i += n;
+ if (filled < body.length) break;
+ state = "lf";
+ }
+ if (state === "lf") {
+ if (i >= chunk.length) break;
+ i++; // the framing newline
+ onObject();
+ state = "header";
+ }
+ }
+ }
+ drained = true;
+ } finally {
+ // Only a loop that threw leaves a child to stop; a drained one is exiting.
+ if (!drained) child.kill();
+ }
+ const code = await exited;
+ if (code !== 0) {
+ throw new SourceRefusal(
+ `git cat-file over ${tildify(gitDir)} exited ${code}: ${stderr.trim().split("\n")[0] ?? ""}`,
+ );
+ }
+ if (state !== "header" || headerParts.length > 0) {
+ throw new SourceRefusal(`git cat-file over ${tildify(gitDir)} ended mid-object`);
+ }
+ return result;
+}
+
+/**
+ * Where each blob sits in history (the first path it was seen at), for the
+ * report. Only asked for when a blob hit, so a clean audit never pays for it.
+ */
+async function blobPaths(
+ gitDir: string,
+ wanted: Set<string>,
+ env?: NodeJS.ProcessEnv,
+): Promise<Map<string, string>> {
+ const out = new Map<string, string>();
+ const child = spawn("git", ["--git-dir", gitDir, "rev-list", "--objects", "--all"], {
+ stdio: ["ignore", "pipe", "ignore"],
+ env: cleanGitEnv(env),
+ });
+ let rest = "";
+ for await (const chunk of child.stdout as AsyncIterable<Buffer>) {
+ const lines = (rest + chunk.toString("utf8")).split("\n");
+ rest = lines.pop() ?? "";
+ for (const line of lines) {
+ const sp = line.indexOf(" ");
+ if (sp === -1) continue;
+ const o = line.slice(0, sp);
+ if (wanted.has(o) && !out.has(o)) out.set(o, line.slice(sp + 1));
+ }
+ }
+ await new Promise((r) => child.once("close", r));
+ return out;
+}
+
+// ── the staged files ────────────────────────────────────────────────────────
+
+/**
+ * Every file under `dir` except `skip` (the packs, which the object walk read
+ * decompressed), plus every path. A `.gz` is scanned decompressed — its
+ * compressed bytes would never match. A symlink is a hit of its own: nothing
+ * staged may point outside the stage.
+ */
+export async function auditFiles(
+ dir: string,
+ literals: readonly Literal[],
+ opts: { skip?: RegExp; result?: AuditResult } = {},
+): Promise<AuditResult> {
+ const skip = opts.skip ?? /\.(pack|idx)$/;
+ const result = opts.result ?? emptyAudit(literals.length);
+ const walk = async (rel: string): Promise<void> => {
+ const abs = path.join(dir, rel);
+ for (const ent of await readdir(abs, { withFileTypes: true })) {
+ const r = rel ? `${rel}/${ent.name}` : ent.name;
+ const nameHits = scanBuffer(Buffer.from(r, "utf8"), literals);
+ if (nameHits.length) record(result, "file", maskLiterals(r, literals), nameHits, () => "path");
+ if (ent.isSymbolicLink()) {
+ throw new SourceRefusal(`the stage holds a symlink (${maskLiterals(r, literals)}); nothing published may point outside it`);
+ }
+ if (ent.isDirectory()) {
+ await walk(r);
+ continue;
+ }
+ result.files++;
+ if (skip.test(ent.name)) continue;
+ let data = await readFile(path.join(abs, ent.name));
+ if (ent.name.endsWith(".gz")) data = gunzipSync(data);
+ const hits = scanBuffer(data, literals);
+ if (hits.length) {
+ record(result, "file", maskLiterals(r, literals), hits, () => (ent.name.endsWith(".gz") ? "decompressed" : "contents"));
+ }
+ }
+ };
+ if ((await lstat(dir)).isDirectory()) await walk("");
+ return result;
+}
+
+// ── gitleaks ────────────────────────────────────────────────────────────────
+
+/** A bare name resolved against PATH the way a spawn would, or null. */
+export function onPath(bin: string, envPath: string | undefined): string | null {
+ if (bin.includes("/")) return existsSync(bin) ? bin : null;
+ for (const d of (envPath ?? "").split(path.delimiter)) {
+ if (!d) continue;
+ const p = path.join(d, bin);
+ try {
+ accessSync(p, constants.X_OK);
+ if (statSync(p).isFile()) return p;
+ } catch {
+ /* not here */
+ }
+ }
+ return null;
+}
+
+type GitleaksFinding = { RuleID?: string; File?: string; Commit?: string };
+
+/**
+ * gitleaks over the mirror's history. Exit 0 is clean, 3 is findings (parsed
+ * from its JSON report), anything else is a refusal. Not installed: "skipped",
+ * with a WARNING line — the literal audit still ran.
+ */
+export async function runGitleaks(
+ gitDir: string,
+ opts: {
+ bin: string;
+ scratch: string;
+ literals: readonly Literal[];
+ onLog: (line: string) => void;
+ signal: AbortSignal;
+ env?: NodeJS.ProcessEnv;
+ timeoutMs?: number;
+ },
+): Promise<{ status: "clean" | "skipped"; hits: AuditHit[] }> {
+ const env = cleanGitEnv(opts.env ?? process.env);
+ if (!onPath(opts.bin, env.PATH)) {
+ opts.onLog(
+ `[source] WARNING: ${opts.bin} is not on PATH — the secret scan is skipped (the literal audit still ran). Install gitleaks to add it.`,
+ );
+ return { status: "skipped", hits: [] };
+ }
+ const report = path.join(opts.scratch, "gitleaks.json");
+ const timeout = AbortSignal.timeout(opts.timeoutMs ?? 300_000);
+ const code = await runChildIntoLog(
+ (line) => opts.onLog(`[gitleaks] ${maskLiterals(line, opts.literals)}`),
+ AbortSignal.any([opts.signal, timeout]),
+ {
+ command: opts.bin,
+ args: [
+ "git",
+ "--no-banner",
+ "--no-color",
+ "--redact",
+ "--exit-code",
+ "3",
+ "--report-format",
+ "json",
+ "--report-path",
+ report,
+ gitDir,
+ ],
+ // Its own ignore file is read from the cwd: the scratch dir has none.
+ cwd: opts.scratch,
+ env,
+ },
+ );
+ if (timeout.aborted) throw new SourceRefusal("gitleaks timed out");
+ if (code === 0) return { status: "clean", hits: [] };
+ if (code !== 3) throw new SourceRefusal(`gitleaks exited ${code}`);
+ let findings: GitleaksFinding[] = [];
+ try {
+ findings = JSON.parse(await readFile(report, "utf8")) as GitleaksFinding[];
+ } catch {
+ throw new SourceRefusal("gitleaks reported findings but its report did not parse");
+ }
+ return {
+ status: "clean",
+ hits: findings.map((f) => ({
+ kind: "gitleaks" as const,
+ where: maskLiterals(
+ `${f.RuleID ?? "?"} in ${f.File ?? "?"} @ ${(f.Commit ?? "").slice(0, 12)}`,
+ opts.literals,
+ ),
+ lit: -1,
+ offset: -1,
+ })),
+ };
+}
+
+// ── together ────────────────────────────────────────────────────────────────
+
+/**
+ * The gate over a git dir: the object walk, then (only when it is clean — a
+ * refusal is already certain otherwise) gitleaks. Blob hits are given their
+ * path in history.
+ */
+export async function auditBare(
+ gitDir: string,
+ literals: readonly Literal[],
+ opts: {
+ scratch: string;
+ onLog: (line: string) => void;
+ signal: AbortSignal;
+ gitleaks: string | null;
+ env?: NodeJS.ProcessEnv;
+ },
+): Promise<AuditResult> {
+ const result = await auditObjects(gitDir, literals, { signal: opts.signal, env: opts.env });
+ const blobs = new Set(result.hits.filter((h) => h.kind === "blob").map((h) => h.where));
+ if (blobs.size > 0) {
+ const where = await blobPaths(gitDir, blobs, opts.env);
+ for (const h of result.hits) {
+ const p = h.kind === "blob" ? where.get(h.where) : undefined;
+ if (p) h.where = `${h.where} ${maskLiterals(p, literals)}`;
+ }
+ }
+ if (result.hits.length > 0) return result;
+ if (opts.gitleaks === null) {
+ result.gitleaks = "skipped";
+ return result;
+ }
+ const g = await runGitleaks(gitDir, {
+ bin: opts.gitleaks,
+ scratch: opts.scratch,
+ literals,
+ onLog: opts.onLog,
+ signal: opts.signal,
+ env: opts.env,
+ });
+ result.gitleaks = g.status;
+ result.hits.push(...g.hits);
+ return result;
+}
+
+/** A path under the home directory as `~/…`: the home dir names the user. */
+export function tildify(p: string): string {
+ const home = os.homedir();
+ return p === home || p.startsWith(`${home}/`) ? `~${p.slice(home.length)}` : p;
+}
+
+/**
+ * The report, as lines. Clean: one line of counts. Hits: the counts per
+ * literal and kind, then the first hits — each by object, field and byte
+ * offset, never by its bytes — and what to do. No line carries a literal or
+ * anything read from beside one.
+ */
+export function formatAuditReport(
+ result: AuditResult,
+ literals: readonly Literal[],
+ opts: { scrubFile: string },
+): string[] {
+ const read =
+ `${result.objects.toLocaleString("en-US")} objects (${result.commits.toLocaleString("en-US")} commit${result.commits === 1 ? "" : "s"})` +
+ (result.files ? `, ${result.files.toLocaleString("en-US")} staged files` : "") +
+ ` against ${result.literals} denied literal${result.literals === 1 ? "" : "s"}; gitleaks ${result.gitleaks}`;
+ if (result.hits.length === 0) return [`[source] audit clean: ${read}`];
+ const lines = [`[source] AUDIT REFUSED: ${result.hits.length} hit${result.hits.length === 1 ? "" : "s"} in ${read}`];
+ const byLit = new Map<number, Map<HitKind, number>>();
+ for (const h of result.hits) {
+ const m = byLit.get(h.lit) ?? new Map<HitKind, number>();
+ m.set(h.kind, (m.get(h.kind) ?? 0) + 1);
+ byLit.set(h.lit, m);
+ }
+ for (const [lit, kinds] of [...byLit].sort((a, b) => a[0] - b[0])) {
+ const label = lit === -1 ? "gitleaks findings" : literalLabel(literals, lit);
+ const parts = KIND_ORDER.filter((k) => kinds.has(k)).map((k) => {
+ const n = kinds.get(k)!;
+ return `${n} in ${k}${n === 1 ? "" : "s"}`;
+ });
+ lines.push(`[source] ${label}: ${parts.join(", ")}`);
+ }
+ const shown = result.hits.slice(0, LISTED);
+ for (const h of shown) {
+ const where = h.kind === "file" || h.kind === "gitleaks" ? h.where : shortWhere(h.where);
+ const at = [h.field, h.offset >= 0 ? `byte ${h.offset}` : ""].filter(Boolean).join(", ");
+ const label = h.lit === -1 ? "" : `: ${literalLabel(literals, h.lit)}`;
+ lines.push(`[source] ${h.kind} ${where}${at ? ` (${at})` : ""}${label}`);
+ }
+ if (result.hits.length > shown.length) {
+ lines.push(`[source] … and ${result.hits.length - shown.length} more`);
+ }
+ lines.push(
+ `[source] add a rule to ${maskLiterals(tildify(opts.scrubFile), literals)} or drop the file from history, then re-run.`,
+ );
+ return lines;
+}
+
+// "<oid> <path>" → "<oid12> <path>"; a bare oid → oid12.
+function shortWhere(where: string): string {
+ const sp = where.indexOf(" ");
+ return sp === -1 ? where.slice(0, 12) : `${where.slice(0, 12)}${where.slice(sp)}`;
+}
diff --git a/common/publish/sourceTree.test.ts b/common/publish/sourceTree.test.ts
@@ -0,0 +1,96 @@
+import { test, after } from "node:test";
+import assert from "node:assert/strict";
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { SourceRefusal } from "./sourceAudit";
+import { escapeHtml, hrefFor, renderTreeIndex, sortEntries, writeTreeIndexes } from "./sourceTree";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// The raw tree's directory pages (sourceTree.ts): hrefs a static host can
+// resolve for the repo's real bracketed names, escaped labels, the order,
+// the relative breadcrumbs, and the refusals.
+
+const TMP = mkdtempSync(path.join(os.tmpdir(), "source-tree-"));
+after(() => rmSync(TMP, { recursive: true, force: true }));
+
+const META = { mirrorHead: "0123456789abcdef0123456789abcdef01234567", generatedAt: "2026-09-28T12:00:00.000Z" };
+
+test("hrefs are URL-encoded, a directory's with a trailing slash — the repo's brackets included", () => {
+ assert.equal(hrefFor({ name: "[slug]", dir: true }), "%5Bslug%5D/");
+ assert.equal(hrefFor({ name: "Archivo[wdth,wght].ttf", dir: false }), "Archivo%5Bwdth%2Cwght%5D.ttf");
+ assert.equal(hrefFor({ name: "a b#c?.md", dir: false }), "a%20b%23c%3F.md");
+ assert.equal(hrefFor({ name: "page.tsx", dir: false }), "page.tsx");
+});
+
+test("names are escaped wherever they are printed", () => {
+ assert.equal(escapeHtml(`<a href="x">&'</a>`), "<a href="x">&'</a>");
+ const html = renderTreeIndex({
+ relDir: "x",
+ entries: [{ name: "<script>.ts", dir: false, bytes: 3 }],
+ ...META,
+ });
+ assert.ok(!html.includes("<script>.ts"), "a name never becomes markup");
+ assert.match(html, /href="%3Cscript%3E\.ts"><script>\.ts<\/a>/);
+});
+
+test("directories first, then files, each by name; sizes with raw bytes in the title", () => {
+ const sorted = sortEntries([
+ { name: "b.ts", dir: false, bytes: 1 },
+ { name: "z", dir: true, bytes: 0 },
+ { name: "a.ts", dir: false, bytes: 2048 },
+ { name: "B", dir: true, bytes: 0 },
+ ]);
+ assert.deepEqual(sorted.map((e) => e.name), ["B", "z", "a.ts", "b.ts"]);
+ const html = renderTreeIndex({ relDir: "", entries: sorted, ...META });
+ assert.match(html, /<span title="2048 bytes">2\.0 KB<\/span>/);
+ assert.match(html, /<meta name="robots" content="noindex">/);
+ assert.match(html, /main @ 0123456789ab · generated 2026-09-28T12:00:00\.000Z · <a href="\/source\/">clone<\/a>/);
+});
+
+test("breadcrumbs hop up relatively; the root has no `..` row", () => {
+ const root = renderTreeIndex({ relDir: "", entries: [], ...META });
+ assert.ok(!root.includes(`href="../"`));
+ assert.match(root, /<nav><strong>archilyzer<\/strong><\/nav>/);
+ const deep = renderTreeIndex({ relDir: "homepage/app/[slug]", entries: [], ...META });
+ assert.match(
+ deep,
+ /<nav><a href="\.\.\/\.\.\/\.\.\/">archilyzer<\/a><span class="sep">\/<\/span><a href="\.\.\/\.\.\/">homepage<\/a><span class="sep">\/<\/span><a href="\.\.\/">app<\/a><span class="sep">\/<\/span><strong>\[slug\]<\/strong><\/nav>/,
+ );
+ assert.match(deep, /<tr><td class="name"><a href="\.\.\/">\.\.<\/a>/);
+});
+
+test("writeTreeIndexes pages every directory and counts the tree, refusing a tracked index.html, a 404.html or a symlink", async () => {
+ const tree = path.join(TMP, "t1");
+ mkdirSync(path.join(tree, "app", "[slug]"), { recursive: true });
+ writeFileSync(path.join(tree, "README.md"), "hello\n");
+ writeFileSync(path.join(tree, "app", "[slug]", "page.tsx"), "x");
+ const totals = await writeTreeIndexes(tree, META);
+ assert.deepEqual(totals, { files: 2, dirs: 3, bytes: 7 });
+ for (const d of ["", "app", "app/[slug]"]) assert.ok(existsSync(path.join(tree, d, "index.html")), d);
+ const appPage = readFileSync(path.join(tree, "app", "index.html"), "utf8");
+ assert.match(appPage, /href="%5Bslug%5D\/">\[slug\]\/<\/a>/);
+ assert.ok(!appPage.includes(`>index.html<`), "a page does not list itself");
+
+ const withIndex = path.join(TMP, "t2");
+ mkdirSync(path.join(withIndex, "docs"), { recursive: true });
+ writeFileSync(path.join(withIndex, "docs", "index.html"), "<p>tracked</p>");
+ await assert.rejects(writeTreeIndexes(withIndex, META), (e) => e instanceof SourceRefusal && /docs\/index\.html/.test(e.message));
+
+ // A tracked 404.html anywhere: Pages would serve it, as HTML on this
+ // origin, for every missing path below its directory.
+ const with404 = path.join(TMP, "t4");
+ mkdirSync(path.join(with404, "docs", "deep"), { recursive: true });
+ writeFileSync(path.join(with404, "docs", "deep", "404.html"), "<script>x</script>");
+ await assert.rejects(
+ writeTreeIndexes(with404, META),
+ (e) => e instanceof SourceRefusal && /docs\/deep\/404\.html; Pages would serve it, as HTML/.test(e.message),
+ );
+
+ const withLink = path.join(TMP, "t3");
+ mkdirSync(withLink);
+ symlinkSync("/etc/hostname", path.join(withLink, "leak"));
+ await assert.rejects(writeTreeIndexes(withLink, META), (e) => e instanceof SourceRefusal && /symlink/.test(e.message));
+});
diff --git a/common/publish/sourceTree.ts b/common/publish/sourceTree.ts
@@ -0,0 +1,186 @@
+// The raw tree's directory pages: one dependency-free index.html per directory
+// of the extracted `main`, so /source/tree/ is browsable on a static host.
+//
+// The FILES under the tree are served as text/plain whatever their extension
+// (homepage/public/_headers, `/source/tree/*`); only these generated pages are
+// HTML. Every name is escaped for HTML and every href is URL-encoded — the tree
+// holds Next's dynamic-route directories (`[slug]`) and variable fonts
+// (`Archivo[wdth,wght].ttf`), whose brackets a browser would otherwise send
+// raw.
+
+import { readdir, stat, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { SourceRefusal } from "./sourceAudit";
+
+export type TreeEntry = { name: string; dir: boolean; bytes: number };
+
+export type TreeMeta = { mirrorHead: string; generatedAt: string };
+
+/** The href of one entry, relative to its directory's page. */
+export function hrefFor(entry: Pick<TreeEntry, "name" | "dir">): string {
+ return encodeURIComponent(entry.name) + (entry.dir ? "/" : "");
+}
+
+export function escapeHtml(s: string): string {
+ return s
+ .replace(/&/g, "&")
+ .replace(/</g, "<")
+ .replace(/>/g, ">")
+ .replace(/"/g, """)
+ .replace(/'/g, "'");
+}
+
+export function formatTreeBytes(bytes: number): string {
+ if (bytes < 1024) return `${bytes} B`;
+ if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`;
+ return `${(bytes / (1024 * 1024)).toFixed(2)} MB`;
+}
+
+/** Directories first, then files, each by name (code-unit order: stable everywhere). */
+export function sortEntries(entries: readonly TreeEntry[]): TreeEntry[] {
+ return [...entries].sort((a, b) =>
+ a.dir !== b.dir ? (a.dir ? -1 : 1) : a.name < b.name ? -1 : a.name > b.name ? 1 : 0,
+ );
+}
+
+const CSS = `
+:root { color-scheme: light dark; --fg: #1d1b18; --muted: #6b665e; --bg: #faf8f4;
+ --rule: #e4dfd6; --link: #7a5a1c; --hover: #f1ede5; }
+@media (prefers-color-scheme: dark) {
+ :root { --fg: #ece8e1; --muted: #a39d93; --bg: #161512; --rule: #2e2b26;
+ --link: #d9b46a; --hover: #221f1b; }
+}
+* { box-sizing: border-box; }
+body { margin: 0; background: var(--bg); color: var(--fg);
+ font: 14px/1.5 ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; }
+main { max-width: 64rem; margin: 0 auto; padding: 1.5rem 1rem 3rem; }
+nav { font-size: 1rem; margin-bottom: 1rem; word-break: break-all; }
+nav .sep { color: var(--muted); padding: 0 0.25rem; }
+a { color: var(--link); text-decoration: none; }
+a:hover { text-decoration: underline; }
+table { width: 100%; border-collapse: collapse; }
+td { padding: 0.3rem 0.5rem; border-top: 1px solid var(--rule); }
+tr:hover td { background: var(--hover); }
+td.size { text-align: right; color: var(--muted); white-space: nowrap; width: 1%; }
+td.name { word-break: break-all; }
+footer { margin-top: 1.5rem; color: var(--muted); font-size: 0.8125rem; }
+`.trim();
+
+/**
+ * One directory's page. `relDir` is the directory relative to the tree root
+ * ("" for the root); every link on the page is relative, so it works under
+ * /source/tree/ on any host and from a directory's `index.html` alike.
+ */
+export function renderTreeIndex(opts: {
+ relDir: string;
+ entries: readonly TreeEntry[];
+ mirrorHead: string;
+ generatedAt: string;
+}): string {
+ const segs = opts.relDir ? opts.relDir.split("/") : [];
+ const depth = segs.length;
+ const up = (n: number) => (n === 0 ? "./" : "../".repeat(n));
+ const crumbs: string[] = [
+ depth === 0
+ ? `<strong>archilyzer</strong>`
+ : `<a href="${up(depth)}">archilyzer</a>`,
+ ];
+ segs.forEach((seg, i) => {
+ const last = i === depth - 1;
+ crumbs.push(
+ last
+ ? `<strong>${escapeHtml(seg)}</strong>`
+ : `<a href="${up(depth - 1 - i)}">${escapeHtml(seg)}</a>`,
+ );
+ });
+ const rows: string[] = [];
+ if (depth > 0) {
+ rows.push(`<tr><td class="name"><a href="../">..</a></td><td class="size"></td></tr>`);
+ }
+ for (const e of sortEntries(opts.entries)) {
+ const label = escapeHtml(e.name) + (e.dir ? "/" : "");
+ const size = e.dir
+ ? ""
+ : `<span title="${e.bytes} bytes">${formatTreeBytes(e.bytes)}</span>`;
+ rows.push(
+ `<tr><td class="name"><a href="${escapeHtml(hrefFor(e))}">${label}</a></td><td class="size">${size}</td></tr>`,
+ );
+ }
+ const title = depth === 0 ? "archilyzer" : `archilyzer/${opts.relDir}`;
+ return [
+ "<!doctype html>",
+ `<html lang="en">`,
+ "<head>",
+ `<meta charset="utf-8">`,
+ `<meta name="viewport" content="width=device-width, initial-scale=1">`,
+ `<meta name="robots" content="noindex">`,
+ `<title>${escapeHtml(title)} · source</title>`,
+ `<style>${CSS}</style>`,
+ "</head>",
+ "<body>",
+ "<main>",
+ `<nav>${crumbs.join(`<span class="sep">/</span>`)}</nav>`,
+ `<table>`,
+ ...rows,
+ `</table>`,
+ `<footer>main @ ${escapeHtml(opts.mirrorHead.slice(0, 12))} · generated ${escapeHtml(opts.generatedAt)} · <a href="/source/">clone</a></footer>`,
+ "</main>",
+ "</body>",
+ "</html>",
+ "",
+ ].join("\n");
+}
+
+/**
+ * Write an index.html into every directory under `treeDir`, the root
+ * included. REFUSES when a directory already holds an `index.html` (a tracked
+ * one would be overwritten, or served in the index's place) or a `404.html`
+ * (Pages would serve it as HTML for any missing path below it), or when
+ * anything is a symlink. Returns the tree's own files and bytes (not the pages) and how
+ * many directories got a page.
+ */
+export async function writeTreeIndexes(
+ treeDir: string,
+ meta: TreeMeta,
+): Promise<{ files: number; dirs: number; bytes: number }> {
+ const totals = { files: 0, dirs: 0, bytes: 0 };
+ const walk = async (rel: string): Promise<void> => {
+ const abs = rel ? path.join(treeDir, rel) : treeDir;
+ const ents = await readdir(abs, { withFileTypes: true });
+ const entries: TreeEntry[] = [];
+ for (const ent of ents) {
+ const r = rel ? `${rel}/${ent.name}` : ent.name;
+ if (ent.isSymbolicLink()) {
+ throw new SourceRefusal(`the tree holds a symlink (${r}); the raw tree publishes files only`);
+ }
+ if (ent.isDirectory()) {
+ entries.push({ name: ent.name, dir: true, bytes: 0 });
+ continue;
+ }
+ if (ent.name === "index.html") {
+ throw new SourceRefusal(
+ `the tree already has ${r}; its directory page would replace it — rename the file or publish without the raw tree`,
+ );
+ }
+ // Pages answers a missing path with the NEAREST 404.html, as HTML (the
+ // directory-page rule): a tracked one would run on this origin.
+ if (ent.name === "404.html") {
+ throw new SourceRefusal(
+ `the tree has ${r}; Pages would serve it, as HTML on this origin, for every missing path below it — rename the file or publish without the raw tree`,
+ );
+ }
+ const { size } = await stat(path.join(abs, ent.name));
+ entries.push({ name: ent.name, dir: false, bytes: size });
+ totals.files++;
+ totals.bytes += size;
+ }
+ await writeFile(
+ path.join(abs, "index.html"),
+ renderTreeIndex({ relDir: rel, entries, ...meta }),
+ );
+ totals.dirs++;
+ for (const e of entries) if (e.dir) await walk(rel ? `${rel}/${e.name}` : e.name);
+ };
+ await walk("");
+ return totals;
+}
diff --git a/create-archives.sh b/create-archives.sh
@@ -1,62 +0,0 @@
-#!/bin/bash
-# Build the downloadable source snapshot served at /downloads/ on the project
-# site, plus a sidecar of facts about it.
-#
-# WHY A SNAPSHOT AND NOT A REPOSITORY. There is no public git remote, by choice.
-# `git archive` of `main` produces the working tree at one commit: no history, no
-# branches, no remote, nothing to `git pull`. The /downloads/ page says exactly
-# that rather than implying a clone.
-#
-# WHY THE SIDECAR. The filename is deliberately STABLE — a dated filename would
-# break every link the moment a new snapshot shipped. So the date, commit, size
-# and checksum live in snapshot.json beside it, and the page states the truth it
-# reads there instead of hardcoding facts that go stale. No sidecar → the page
-# renders an explanation and NO download link, rather than a link that lies.
-#
-# Safe to publish: `git archive` only includes TRACKED files, so the corpus
-# (transcripts/, gitignored), settings.json (untracked; only .example is
-# tracked) and node_modules are all structurally excluded.
-set -euo pipefail
-
-cd "$(dirname "$0")"
-
-REF="${1:-main}"
-OUT_DIR="homepage/public/downloads"
-TARBALL="$OUT_DIR/archilyzer-source.tar.gz"
-SIDECAR="$OUT_DIR/snapshot.json"
-
-mkdir -p "$OUT_DIR"
-
-# --prefix so the tarball unpacks into `archilyzer/` rather than spilling into
-# the current directory. The docs tell people to `cd archilyzer` afterwards.
-git archive -9 --format tar.gz --prefix=archilyzer/ -o "$TARBALL" "$REF"
-
-BYTES=$(stat -c %s "$TARBALL" 2>/dev/null || stat -f %z "$TARBALL")
-SHA=$(sha256sum "$TARBALL" | cut -d' ' -f1)
-COMMIT=$(git rev-parse "$REF")
-SUBJECT=$(git log -1 --format=%s "$REF")
-GENERATED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
-
-# node rather than a heredoc so the commit subject is JSON-escaped properly —
-# it is arbitrary human text and routinely contains quotes.
-GENERATED_AT="$GENERATED_AT" COMMIT="$COMMIT" SUBJECT="$SUBJECT" \
-BYTES="$BYTES" SHA="$SHA" node -e '
- const fs = require("fs");
- fs.writeFileSync(process.argv[1], JSON.stringify({
- generatedAt: process.env.GENERATED_AT,
- commit: process.env.COMMIT,
- subject: process.env.SUBJECT,
- bytes: Number(process.env.BYTES),
- sha256: process.env.SHA,
- }, null, 2) + "\n");
-' "$SIDECAR"
-
-echo "create-archives: $TARBALL ($(( BYTES / 1024 )) KiB) at ${COMMIT:0:12}"
-
-# Cloudflare Pages rejects any single asset over 25 MB. The snapshot is an order
-# of magnitude under that, but say so loudly if that ever stops being true —
-# silently shipping an asset Pages will refuse is a deploy that "succeeds" and
-# then 404s.
-if [ "$BYTES" -gt 26214400 ]; then
- echo "create-archives: WARNING — $TARBALL exceeds Cloudflare Pages' 25 MB per-file limit." >&2
-fi
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,6 +1,7 @@
# Changelog
## [Unreleased]
+- **Building the homepage now publishes the source: a read-only git mirror, its raw tree and a fresh tarball, behind a gate.** `archilyzer build homepage`, the `/sites` Homepage jobs and `pnpm ops build-homepage` run `archilyzer source publish` between compose and `next build`. It makes a fresh clone of the private `main` (the repository itself is never rewritten), rewrites that copy with git-filter-repo using your scrub rules (file contents and commit messages; your home directory becomes `/home/user` without a rule), and publishes it under `homepage/public` for `git clone https://archilyzer.pages.dev/source/archilyzer.git`, beside `/source/tree/` and the Downloads tarball. Before anything is written, every object of the rewritten history and every file about to be published is searched for every string you have denied; **one hit refuses the build**, and its log names the string only by where you wrote it (`denylist line 3 (len 5)`) and each hit by its object, field and byte offset — never a byte of the object. **A refusal withdraws the source**: the last publish is removed from `homepage/public` and the last build's copy from `homepage/out`, and **Deploy homepage refuses** a build whose source was not audited under today's rules and today's `main` ("run `archilyzer build homepage`, then deploy"). The rules live outside the repo, in `~/.config/archilyzer/source-scrub.txt` and `source-denylist.txt` (`ARCHILYZER_CONFIG_DIR`, `SOURCE_SCRUB_FILE`, `SOURCE_DENYLIST_FILE`); **without them the build refuses**, naming the missing file. **Put everything private in the denylist before any deploy, a preview included**: previews are public, and every deployment stays reachable at its own address until you delete it. Install git-filter-repo once (`pipx install git-filter-repo`; the editor's process needs `~/.local/bin` on its `PATH` to find it) — without it the build fetches it through `pipx run`, which needs the network — and gitleaks if you want its secret scan too. An unchanged `main` with unchanged rules is skipped, so a rebuild costs about 20 seconds only when something moved. A checkout with no git repository (the docker image, a tarball install) builds with the /source page's empty state. `archilyzer source publish --check` audits without writing, `archilyzer source audit <clone>/.git` checks any clone, `archilyzer build homepage --no-source` removes the published source instead, and `archilyzer doctor` reports the tools, the two files (rule counts and permissions, never their contents) and the last publish. `create-archives.sh` is gone. See PUBLISH.md, "The source mirror (homepage)".
- **umtool reads the corpus from its checkout (or `TRANSCRIPTS_DIR`), and the song project's data defaults to `~/.local/share/archilyzer/song`.** If yours is elsewhere, link it there before restarting umtool: `mkdir -p ~/.local/share/archilyzer && ln -s <where the data is> ~/.local/share/archilyzer/song` (the data stays where it is). With no `CHANNELS_DIR`, umtool reads the corpus at `$TRANSCRIPTS_DIR/channels`, else the checkout's own `transcripts/channels`; it used to fall back to an absolute path that existed on one machine only. The song project's videos default to `~/reports/quartering-uh-song/videos`; `SONG_DIR` and `VIDEO_ROOT` still win. The song project's tracked manifests record their paths relative to the song folders, and the twenty one-off `umtool/song/*.sh` run logs, which only ever ran on the machine that wrote them, are gone.
## [0.10.0] - 2026-09-28
diff --git a/editor/app/sites/components/HomepageBuildButtons.tsx b/editor/app/sites/components/HomepageBuildButtons.tsx
@@ -170,6 +170,13 @@ export function HomepageBuildButtons({ project, builtAt }: Props) {
The homepage reads the search index as it stands: run Build index
(under Pool jobs, below) first when its numbers should move.
</p>
+ <p className="text-xs text-muted-foreground">
+ Build homepage also publishes the source mirror (<code>archilyzer
+ source publish</code>) and refuses if a denied string survives the
+ scrub; a refusal removes the last published source, from
+ homepage/out too. Deploy homepage refuses a build whose source was
+ not audited under today’s rules.
+ </p>
{lane && (
<JobLane
key={lane.key}
diff --git a/homepage/CHANGELOG.md b/homepage/CHANGELOG.md
@@ -2,6 +2,7 @@
## [Unreleased]
+- **The source is on the site, with its history: `/source/`.** A new **Source** page (and nav entry) gives `git clone https://archilyzer.pages.dev/source/archilyzer.git`, a read-only mirror of the main branch regenerated with every deploy, with its head, the private commit it reflects, a link to browse every file raw at `/source/tree/`, and the tarball with its size and sha256. Commit ids differ from the private repository's, because machine paths are scrubbed on the way out, and the page says so. A build without a published source says "No source published in this build." instead of offering a clone. The Downloads tarball is now regenerated by every build (its commit is the mirror's), and the page points at the mirror for history. The docs that said there is no public repository (*Install*, the FAQ, *What is Archilyzer*) now say how to clone. Below `md` the header's nav drops to its own row, as it did below `sm`, because five labels no longer fit beside the wordmark. `_headers` serves the raw tree as plain text.
- **The docs' *Building several sites at once* page says what Build all does.** It called the container pipeline opt-in, turned on in the settings. Build all sites builds every site in parallel in containers whenever a container engine is available, and one after another when none is; there is nothing to switch on.
- **A single-colour social icon shows on every ground.** The footer's social icons are the operator's (`homepage.json`'s, else `settings.socialLinks`), normalized when they are saved (`normalizeSocialSvg`, release 11 slice O1). An icon drawn in one colour now takes the footer's colour throughout; before, a part that carried its own colour kept it, so X's official logo, which is white, was invisible on the Light ground. An icon of two or more colours, such as YouTube's red mark with its white triangle, keeps its colours as pasted. "No fill", gradients, masks, clip paths and animation timing are never changed, and a clip path's own colour does not count, so a one-colour icon exported from Figma follows the footer too. It applies when the settings are next saved, then needs a rebuild and deploy of the homepage.
- **In high-contrast mode the header mark's tile keeps its edge.** In Windows' high-contrast mode (forced colours) the reader's own background replaces the page on every ground and can be as dark as the slate tile, whose ring is only drawn on Dark. In that mode the tile gets a 1-pixel outline in the reader's text colour, on every ground, following its rounded corners. Nothing changes outside that mode.
diff --git a/homepage/app/components/Fact.tsx b/homepage/app/components/Fact.tsx
@@ -0,0 +1,25 @@
+// One labelled fact in a page's fact list — the Downloads and Source pages'
+// rows. `testId` marks the value, for the specs that compare it with a file.
+export function Fact({
+ label,
+ children,
+ testId,
+}: {
+ label: string;
+ children: React.ReactNode;
+ testId?: string;
+}) {
+ return (
+ <div className="flex flex-col gap-1 border-t border-[var(--border)] py-3 sm:flex-row sm:gap-6">
+ <span className="label-machine sm:w-32 sm:shrink-0 sm:pt-0.5">
+ {label}
+ </span>
+ <span
+ data-testid={testId}
+ className="min-w-0 break-all text-sm text-[var(--muted-foreground)]"
+ >
+ {children}
+ </span>
+ </div>
+ );
+}
diff --git a/homepage/app/components/Header.tsx b/homepage/app/components/Header.tsx
@@ -27,7 +27,7 @@ function NavList({ className }: { className?: string }) {
);
}
-// The project site's header: a hairline bar with the wordmark and the four
+// The project site's header: a hairline bar with the wordmark and the five
// destinations. It carried NO links at all before this — the page it sat above
// was the whole site.
//
@@ -56,23 +56,23 @@ export default function Header() {
className="text-[1.3rem] leading-none tracking-[-0.01em]"
/>
</Link>
- <nav aria-label="Main" className="ml-auto hidden sm:block">
+ <nav aria-label="Main" className="ml-auto hidden md:block">
<NavList className="gap-6" />
</nav>
- <div className="ml-auto sm:ml-0 flex items-center gap-2">
+ <div className="ml-auto md:ml-0 flex items-center gap-2">
<ThemeMenu />
<ThemeToggle />
</div>
</div>
- {/* Below `sm` the bar has no room for four labels beside the wordmark and
- the theme controls, so the nav drops to its own scrollable rule rather
- than collapsing behind a menu button — four links do not earn a
- disclosure widget. Only one of the two is ever in the accessibility
+ {/* Below `md` the bar has no room for five labels beside the wordmark and
+ the theme controls (at `sm` four fitted; Source made it five), so the
+ nav drops to its own scrollable rule rather than collapsing behind a
+ menu button — five links do not earn a disclosure widget. Only one of the two is ever in the accessibility
tree (the other is display:none), but they carry distinct labels so a
test or a screen reader can never conflate them. */}
<nav
aria-label="Main, compact"
- className="sm:hidden border-t border-[var(--border)] overflow-x-auto"
+ className="md:hidden border-t border-[var(--border)] overflow-x-auto"
>
<NavList className="gap-5 px-5 h-10" />
</nav>
diff --git a/homepage/app/downloads/page.tsx b/homepage/app/downloads/page.tsx
@@ -1,6 +1,7 @@
import Link from "next/link";
import type { Metadata } from "next";
import { PageShell, PageHeading } from "../components/PageShell";
+import { Fact } from "../components/Fact";
import { loadSnapshot, formatBytes, SNAPSHOT_HREF } from "../lib/snapshot";
export const metadata: Metadata = {
@@ -9,19 +10,6 @@ export const metadata: Metadata = {
"Get the Archilyzer source as a dated snapshot tarball — MIT licensed, with a checksum.",
};
-function Fact({ label, children }: { label: string; children: React.ReactNode }) {
- return (
- <div className="flex flex-col gap-1 border-t border-[var(--border)] py-3 sm:flex-row sm:gap-6">
- <span className="label-machine sm:w-32 sm:shrink-0 sm:pt-0.5">
- {label}
- </span>
- <span className="min-w-0 break-all text-sm text-[var(--muted-foreground)]">
- {children}
- </span>
- </div>
- );
-}
-
export default function DownloadsPage() {
const snapshot = loadSnapshot();
const date = snapshot
@@ -50,6 +38,16 @@ export default function DownloadsPage() {
branches, nothing to <code className="font-mono">git pull</code>.
Updating means downloading a newer snapshot.
</p>
+ <p className="leading-[1.7] text-[var(--muted-foreground)]">
+ The history is in the{" "}
+ <Link
+ href="/source/"
+ className="underline decoration-[var(--border-strong)] underline-offset-2 hover:text-[var(--foreground)]"
+ >
+ read-only git mirror
+ </Link>
+ , published by the same build as this tarball.
+ </p>
</div>
{snapshot ? (
@@ -99,8 +97,9 @@ export default function DownloadsPage() {
</p>
<p className="text-sm text-[var(--faint)] leading-relaxed">
The tarball and its sidecar are build artefacts, generated by{" "}
- <code className="font-mono">./create-archives.sh</code> and not
- committed. A site built without running it ships no download, and
+ <code className="font-mono">archilyzer source publish</code> (which{" "}
+ <code className="font-mono">archilyzer build homepage</code> runs)
+ and not committed. A site built without it ships no download, and
says so here rather than linking to a file that isn’t there.
</p>
</div>
diff --git a/homepage/app/lib/headers.test.ts b/homepage/app/lib/headers.test.ts
@@ -0,0 +1,103 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { readFileSync } from "node:fs";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+// Run with:
+// pnpm --filter homepage test
+//
+// public/_headers is what keeps the raw source tree from being served as
+// code on this origin, and nothing but the Pages edge applies it. This pins
+// its security property with a small replica of wrangler's `_headers`
+// handling (pages-shared parseHeaders + generateRulesMatcher + attachHeaders,
+// read in wrangler 4.88 and re-checked by the review in 4.142): every
+// matching rule applies in file order, `! Name` detaches a header, and a
+// header a LATER rule sets again is APPENDED ("a, b"), not replaced.
+
+const FILE = path.join(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "public", "_headers");
+const TEXT = readFileSync(FILE, "utf8");
+
+type Rule = { path: string; set: Record<string, string>; unset: string[]; re: RegExp; lines: string[] };
+
+function parse(text: string): Rule[] {
+ const rules: Rule[] = [];
+ let rule: Rule | undefined;
+ for (const raw of text.split("\n")) {
+ const line = raw.trim();
+ if (!line || line.startsWith("#")) continue;
+ if (/^([^\s]+:\/\/|^\/)/.test(line)) {
+ if (rule) rules.push(rule);
+ const esc = (s: string) => s.replace(/[|\\{}()[\]^$+*?.]/g, "\\$&");
+ rule = { path: line, set: {}, unset: [], re: new RegExp(`^${line.split("*").map(esc).join("(?<splat>.*)")}$`), lines: [] };
+ continue;
+ }
+ rule!.lines.push(line);
+ if (line.startsWith("! ")) {
+ rule!.unset.push(line.slice(2).toLowerCase());
+ continue;
+ }
+ const i = line.indexOf(":");
+ const name = line.slice(0, i).trim().toLowerCase();
+ const value = line.slice(i + 1).trim();
+ rule!.set[name] = rule!.set[name] ? `${rule!.set[name]}, ${value}` : value;
+ }
+ if (rule) rules.push(rule);
+ return rules;
+}
+
+function served(rules: Rule[], pathname: string, contentType: string): Headers {
+ const h = new Headers({ "content-type": contentType });
+ const setOnce = new Set<string>();
+ for (const r of rules.filter((x) => x.re.test(pathname))) {
+ for (const k of r.unset) h.delete(k);
+ for (const [k, v] of Object.entries(r.set)) {
+ if (setOnce.has(k)) h.append(k, v);
+ else {
+ h.set(k, v);
+ setOnce.add(k);
+ }
+ }
+ }
+ return h;
+}
+
+const RULES = parse(TEXT);
+
+test("wrangler's limits: at most 100 rules, no line over 2,000 characters, one splat a rule", () => {
+ assert.ok(RULES.length <= 100, `${RULES.length} rules`);
+ for (const line of TEXT.split("\n")) assert.ok(line.length <= 2000, line.slice(0, 60));
+ for (const r of RULES) assert.ok((r.path.match(/\*/g) ?? []).length <= 1, r.path);
+});
+
+test("every /source/tree override detaches Content-Type before it sets its own", () => {
+ const overrides = RULES.filter((r) => r.path.startsWith("/source/tree/") && r.path !== "/source/tree/*" && "content-type" in r.set);
+ assert.ok(overrides.length >= 7, overrides.map((r) => r.path).join(" "));
+ for (const r of overrides) {
+ const unset = r.lines.findIndex((l) => l.toLowerCase() === "! content-type");
+ const set = r.lines.findIndex((l) => /^content-type:/i.test(l));
+ assert.ok(unset !== -1 && unset < set, `${r.path} must say "! Content-Type" before it sets one`);
+ }
+});
+
+test("the raw tree is text, its pages HTML, its binaries their own type — each ONE value", () => {
+ const cases: Array<[string, string, string]> = [
+ ["/source/tree/common/lib/paths.ts", "video/mp2t", "text/plain; charset=utf-8"],
+ ["/source/tree/README.md", "text/markdown", "text/plain; charset=utf-8"],
+ ["/source/tree/", "text/html", "text/html; charset=utf-8"],
+ ["/source/tree/common/", "text/html", "text/html; charset=utf-8"],
+ ["/source/tree/umtool/report-to-video/fonts/Archivo%5Bwdth%2Cwght%5D.ttf", "font/ttf", "font/ttf"],
+ ["/source/tree/homepage/app/docs/%5Bslug%5D/page.tsx", "application/octet-stream", "text/plain; charset=utf-8"],
+ ["/source/tree/umtool/models/yunet.onnx", "application/octet-stream", "application/octet-stream"],
+ ["/source/tree/homepage/public/icon.svg", "image/svg+xml", "image/svg+xml"],
+ // A missing directory: Pages answers with a 404 page, on this rule.
+ ["/source/tree/no/such/dir/", "text/html", "text/html; charset=utf-8"],
+ ];
+ for (const [p, served0, want] of cases) {
+ const h = served(RULES, p, served0);
+ assert.equal(h.get("content-type"), want, p);
+ assert.equal(h.get("x-content-type-options"), "nosniff", p);
+ }
+ assert.match(served(RULES, "/source/tree/a.svg", "image/svg+xml").get("content-security-policy")!, /default-src 'none'/);
+ assert.equal(served(RULES, "/source/manifest.json", "application/json").get("content-type"), "application/json");
+});
diff --git a/homepage/app/lib/nav.ts b/homepage/app/lib/nav.ts
@@ -1,13 +1,15 @@
// The site's navigation, declared once and rendered by both the header and the
-// footer. Order is editorial: what the software is, how to get it, what this
-// deployment has done, what changed.
+// footer. Order is editorial: what the software is, where its code lives, how
+// to get it, what this deployment has done, what changed.
//
// Every entry must resolve on a `build:nodata` tree — /stats/ renders an honest
-// no-data panel rather than 404ing — so the nav never has to be conditional.
+// no-data panel and /source/ an honest "nothing published" rather than 404ing —
+// so the nav never has to be conditional.
export type NavItem = { href: string; label: string };
export const NAV: NavItem[] = [
{ href: "/docs/", label: "Docs" },
+ { href: "/source/", label: "Source" },
{ href: "/downloads/", label: "Downloads" },
{ href: "/stats/", label: "Stats" },
{ href: "/changelog/", label: "Changelog" },
diff --git a/homepage/app/lib/snapshot.ts b/homepage/app/lib/snapshot.ts
@@ -1,8 +1,11 @@
import fs from "node:fs";
import path from "node:path";
-// Facts about the published source snapshot, written by create-archives.sh
-// beside the tarball it describes.
+// Facts about the published source snapshot, written by `archilyzer source
+// publish` (common/publish/source.ts) beside the tarball it describes. Since
+// release 12 the tarball is `git archive` of the scrubbed MIRROR's main, so
+// `commit` is a mirror id — the /source/ page names the private main it
+// reflects.
export type Snapshot = {
generatedAt: string;
commit: string;
@@ -20,7 +23,7 @@ export const SNAPSHOT_HREF = "/downloads/archilyzer-source.tar.gz";
//
// The null case is not hypothetical: the tarball and its sidecar are gitignored
// build artefacts, so a fresh unpack of the source has neither until
-// ./create-archives.sh runs. The download page MUST render no link at all in
+// `archilyzer source publish` runs (`archilyzer build homepage` runs it). The download page MUST render no link at all in
// that case — a link to a file that isn't there is worse than an explanation of
// why there isn't one.
export function loadSnapshot(): Snapshot | null {
diff --git a/homepage/app/lib/source.test.ts b/homepage/app/lib/source.test.ts
@@ -0,0 +1,71 @@
+import { test, after } from "node:test";
+import assert from "node:assert/strict";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { loadSourceManifest } from "./source";
+
+// Run with:
+// pnpm --filter homepage test
+//
+// The /source/ page's loader: the manifest, believed only when it parses AND
+// the mirror's info/refs and the tarball are beside it. Anything else is the
+// empty state — a bad file must never crash `next build` (review L10).
+
+const TMP = mkdtempSync(path.join(os.tmpdir(), "homepage-source-"));
+after(() => rmSync(TMP, { recursive: true, force: true }));
+
+const manifest = {
+ version: 1,
+ generatedAt: "2026-09-28T12:00:00.000Z",
+ branch: "main",
+ sourceCommit: "1".repeat(40),
+ mirrorHead: "2".repeat(40),
+ subject: "s",
+ files: 10,
+ bytes: 100,
+ mirror: { files: 9, bytes: 80, packs: 2 },
+ tree: { files: 5, dirs: 2, bytes: 20 },
+ tarball: { href: "/downloads/archilyzer-source.tar.gz", bytes: 7, sha256: "3".repeat(64) },
+ cloneUrl: "https://archilyzer.pages.dev/source/archilyzer.git",
+ treeHref: "/source/tree/",
+ audit: { objects: 3, commits: 1, gitleaks: "clean" },
+ tools: { git: "2.55.0", filterRepo: "x" },
+};
+
+let n = 0;
+function pub(opts: { manifest?: unknown; refs?: boolean; tarball?: boolean }): string {
+ const p = path.join(TMP, `p${n++}`);
+ mkdirSync(path.join(p, "source", "archilyzer.git", "info"), { recursive: true });
+ mkdirSync(path.join(p, "downloads"), { recursive: true });
+ if (opts.manifest !== undefined) {
+ writeFileSync(
+ path.join(p, "source", "manifest.json"),
+ typeof opts.manifest === "string" ? opts.manifest : JSON.stringify(opts.manifest),
+ );
+ }
+ if (opts.refs !== false) writeFileSync(path.join(p, "source", "archilyzer.git", "info", "refs"), "x\trefs/heads/main\n");
+ if (opts.tarball !== false) writeFileSync(path.join(p, "downloads", "archilyzer-source.tar.gz"), "t");
+ return p;
+}
+
+test("a good manifest with its files beside it is believed", () => {
+ assert.equal(loadSourceManifest(pub({ manifest }))?.mirrorHead, "2".repeat(40));
+});
+
+test("malformed, wrong-version or orphaned manifests are the empty state, never a throw", () => {
+ const cases: Array<[string, string]> = [
+ ["no manifest", pub({})],
+ ["not JSON", pub({ manifest: "{ half" })],
+ ["version 2", pub({ manifest: { ...manifest, version: 2 } })],
+ ["tree.files missing (the page reads it)", pub({ manifest: { ...manifest, tree: { dirs: 1, bytes: 1 } } })],
+ ["mirror.packs a string", pub({ manifest: { ...manifest, mirror: { ...manifest.mirror, packs: "2" } } })],
+ ["no info/refs", pub({ manifest, refs: false })],
+ ["no tarball", pub({ manifest, tarball: false })],
+ ["no public dir at all", path.join(TMP, "missing")],
+ ];
+ for (const [what, dir] of cases) {
+ assert.doesNotThrow(() => loadSourceManifest(dir), what);
+ assert.equal(loadSourceManifest(dir), null, what);
+ }
+});
diff --git a/homepage/app/lib/source.ts b/homepage/app/lib/source.ts
@@ -0,0 +1,36 @@
+import fs from "node:fs";
+import path from "node:path";
+import {
+ MIRROR_DIR,
+ TARBALL_HREF,
+ parseSourceManifest,
+ type SourceManifest,
+} from "yt-dlp-transcript-common/lib/sourceManifest";
+
+// The published source's manifest, written LAST by `archilyzer source publish`
+// (common/publish/source.ts) into public/source/, or null when this build has
+// none. The /source/ page renders an honest empty state for null — the mirror,
+// the tree and the tarball are gitignored build artefacts, so a fresh clone
+// (or `archilyzer build homepage --no-source`) has none of them.
+//
+// Believed only when the files it describes are here too (the snapshot.ts
+// rule): a manifest beside a missing mirror or tarball would advertise a
+// clone that fails and a download that 404s. A malformed or wrong-version
+// manifest is null too (parseSourceManifest checks every number the page
+// reads), so a bad file is the empty state, never a crash in `next build`.
+// `pub` is the test's seam.
+export function loadSourceManifest(
+ pub: string = path.join(process.cwd(), "public"),
+): SourceManifest | null {
+ try {
+ const manifest = parseSourceManifest(
+ JSON.parse(fs.readFileSync(path.join(pub, "source", "manifest.json"), "utf8")),
+ );
+ if (!manifest) return null;
+ if (!fs.statSync(path.join(pub, "source", MIRROR_DIR, "info", "refs")).isFile()) return null;
+ if (!fs.statSync(path.join(pub, TARBALL_HREF.replace(/^\//, ""))).isFile()) return null;
+ return manifest;
+ } catch {
+ return null;
+ }
+}
diff --git a/homepage/app/source/page.tsx b/homepage/app/source/page.tsx
@@ -0,0 +1,182 @@
+import Link from "next/link";
+import type { Metadata } from "next";
+import { CLONE_URL, TREE_HREF } from "yt-dlp-transcript-common/lib/sourceManifest";
+import { PageShell, PageHeading } from "../components/PageShell";
+import { Fact } from "../components/Fact";
+import { loadSourceManifest } from "../lib/source";
+import { formatBytes } from "../lib/snapshot";
+
+export const metadata: Metadata = {
+ title: "Source",
+ description:
+ "The canonical public copy of Archilyzer's source: a read-only git mirror, its raw tree, and a tarball. MIT licensed.",
+};
+
+const linkClass =
+ "underline decoration-[var(--border-strong)] underline-offset-2 hover:text-[var(--foreground)]";
+
+// The published source (common/publish/source.ts): the mirror, the raw tree
+// and the tarball are build artefacts, so this page has two states and the
+// manifest decides which. A manifest is believed only when the files it
+// describes are here too (lib/source.ts): no clone command or link for a file
+// that isn't there.
+export default function SourcePage() {
+ const manifest = loadSourceManifest();
+ const date = manifest
+ ? new Date(manifest.generatedAt).toLocaleDateString(undefined, {
+ year: "numeric",
+ month: "long",
+ day: "numeric",
+ })
+ : null;
+
+ return (
+ <PageShell className="flex flex-col gap-10">
+ <PageHeading
+ eyebrow="Code"
+ title="Source"
+ standfirst="The canonical public copy: a read-only git mirror, its raw tree, and a tarball."
+ />
+
+ <div className="doc-measure flex flex-col gap-4">
+ <p className="leading-[1.7] text-[var(--muted-foreground)]">
+ There is no GitHub, by choice. This site is where the code lives in
+ public: a read-only mirror of the private main branch, regenerated
+ with every deploy of this site. Commit ids differ from the private
+ repository because paths were scrubbed on the way out — same history,
+ different hashes. Nothing here takes a push or a pull request.
+ </p>
+ </div>
+
+ {manifest ? (
+ <div className="flex flex-col gap-6">
+ <div className="doc-measure">
+ <p className="mb-2 label-machine">Clone it</p>
+ <pre
+ data-testid="source-clone"
+ className="p-4 rounded-[var(--radius)] bg-[var(--surface)] border border-[var(--border)] overflow-x-auto text-[0.8125rem] font-mono text-[var(--foreground)]"
+ >
+ {`git clone ${CLONE_URL}`}
+ </pre>
+ </div>
+
+ <div className="doc-measure flex flex-col">
+ <Fact label="Mirror head" testId="source-mirror-head">
+ <span className="tabular">{manifest.mirrorHead}</span>
+ </Fact>
+ <Fact label="Reflects">
+ private <code className="font-mono">main</code> at{" "}
+ <span className="tabular">{manifest.sourceCommit}</span>
+ </Fact>
+ <Fact label="Subject">{manifest.subject}</Fact>
+ <Fact label="Generated">{date}</Fact>
+ <Fact label="Size">
+ <span className="tabular">
+ {manifest.tree.files.toLocaleString()} files in the tree;{" "}
+ {formatBytes(manifest.mirror.bytes)} of history in{" "}
+ {manifest.mirror.packs} pack{manifest.mirror.packs === 1 ? "" : "s"}
+ </span>
+ </Fact>
+ </div>
+
+ <div className="doc-measure flex flex-col gap-3">
+ <p>
+ <a data-testid="source-tree-link" href={TREE_HREF} className={`text-sm ${linkClass}`}>
+ Browse the tree
+ </a>
+ <span className="text-sm text-[var(--muted-foreground)]">
+ {" "}— every tracked file of main, raw, as plain text.
+ </span>
+ </p>
+ <p className="text-sm text-[var(--muted-foreground)]">
+ <a data-testid="source-tarball-link" href={manifest.tarball.href} className={linkClass}>
+ archilyzer-source.tar.gz
+ </a>{" "}
+ <span className="tabular">
+ ({formatBytes(manifest.tarball.bytes)}, sha256{" "}
+ <span data-testid="source-tarball-sha" className="break-all">
+ {manifest.tarball.sha256}
+ </span>
+ )
+ </span>{" "}
+ — the same tree with no history; the{" "}
+ <Link href="/downloads/" className={linkClass}>
+ Downloads
+ </Link>{" "}
+ page says what is in it.
+ </p>
+ </div>
+ </div>
+ ) : (
+ // No manifest, or not every file it describes: say so and render NO
+ // clone command and no link, exactly like the Downloads page.
+ <div data-testid="source-empty" className="doc-measure faceplate p-5 flex flex-col gap-3">
+ <p className="text-[var(--muted-foreground)] leading-relaxed">
+ No source published in this build.
+ </p>
+ <p className="text-sm text-[var(--faint)] leading-relaxed">
+ The mirror, the raw tree and the tarball are build artefacts,
+ generated by{" "}
+ <code className="font-mono">archilyzer source publish</code> (which{" "}
+ <code className="font-mono">archilyzer build homepage</code> runs)
+ and not committed. A site built without it says so here rather than
+ offering a clone that would fail.
+ </p>
+ </div>
+ )}
+
+ <div className="doc-measure flex flex-col gap-8 border-t border-[var(--border)] pt-8">
+ <div className="flex flex-col gap-3">
+ <h2 className="font-display text-base font-semibold text-[var(--foreground)]">
+ What is mirrored
+ </h2>
+ <ul className="list-disc pl-5 space-y-2 text-sm leading-[1.7] text-[var(--muted-foreground)] marker:text-[var(--faint)]">
+ <li>
+ <strong className="text-[var(--foreground)]">The main branch, whole.</strong>{" "}
+ Every commit since the first, with its message and author, and
+ every tracked file. No other branches and no tags.
+ </li>
+ <li>
+ <strong className="text-[var(--foreground)]">Scrubbed on the way out.</strong>{" "}
+ Machine paths such as the home directory are rewritten to generic
+ ones, in files and in commit messages, which is why every id
+ differs from the private repository’s. The build refuses to
+ publish if anything it was told to remove survives, anywhere in
+ the history.
+ </li>
+ <li>
+ <strong className="text-[var(--foreground)]">Regenerated, not pushed to.</strong>{" "}
+ Each deploy rebuilds the mirror from the private main. The ids
+ stay put from one deploy to the next, but a change to the scrub
+ rules rewrites them all — if a{" "}
+ <code className="font-mono">git pull</code> ever refuses
+ unrelated history, clone again.
+ </li>
+ <li>
+ <strong className="text-[var(--foreground)]">Static files.</strong>{" "}
+ The clone uses git’s plain-HTTP protocol, straight from this
+ site’s files: slower than a forge, and a fetch downloads
+ whole packs, but there is no server to go down.
+ </li>
+ <li>
+ <strong className="text-[var(--foreground)]">Not the archive.</strong>{" "}
+ No transcripts, media, settings or dependencies — the same list
+ as the tarball’s.
+ </li>
+ </ul>
+ </div>
+
+ <div className="flex flex-col gap-3">
+ <h2 className="font-display text-base font-semibold text-[var(--foreground)]">
+ License
+ </h2>
+ <p className="text-sm leading-[1.7] text-[var(--muted-foreground)]">
+ MIT. Use it, change it, run it, publish with it. The full text is in{" "}
+ <code className="font-mono text-[var(--foreground)]">LICENSE</code>{" "}
+ at the root of the tree.
+ </p>
+ </div>
+ </div>
+ </PageShell>
+ );
+}
diff --git a/homepage/content/README.md b/homepage/content/README.md
@@ -9,9 +9,12 @@ exists for whoever edits the docs next.
The obvious move is to render `SETUP.md`, `PUBLISH.md` and friends
directly, and keep one copy. That was rejected for a decisive reason:
-> `SETUP.md` says `git clone <this-repo-url>`. **There is no public repository.**
-> Rendering that on a public site publishes an instruction that cannot be
-> followed — the acquisition path is a dated snapshot tarball from `/downloads/`.
+> `SETUP.md` was written for a contributor with the repository, and its
+> acquisition path is not the public one. The public copy is the read-only
+> mirror at `/source/archilyzer.git` (regenerated per deploy, with different
+> commit ids) and the tarball at `/downloads/`; `docs/install.md` says so in
+> an operator's terms. (Until release 12 there was no public repository at all,
+> which is when this rule was made.)
The root docs also lean on contributor furniture that actively confuses an
operator: `pnpm wt` worktrees, the machine-global e2e queue, `plans/`, shard
diff --git a/homepage/content/docs/faq.md b/homepage/content/docs/faq.md
@@ -78,9 +78,12 @@ gone* — never "confirmed still there".
## Where is the git repository?
-There isn't one. The source is published as a
-[dated snapshot tarball](/downloads/): a working tree with no history, no
-branches and no remote. Updating means downloading a newer snapshot.
+On this site, read-only: `git clone https://archilyzer.pages.dev/source/archilyzer.git`.
+It is the main branch with its whole history, regenerated from the private
+repository with every deploy — machine paths scrubbed on the way out, which is
+why its commit ids differ. There is no GitHub, and nothing takes a push or a
+pull request. [Source](/source/) also has every file to browse, and
+[Downloads](/downloads/) the same tree as a tarball.
## Can I use it for one video?
@@ -92,7 +95,8 @@ exists because of scale.
## What this is not
- **Not a hosted service.** There is no account, no subscription, no upload.
-- **Not a public repository.** Dated snapshots, no `git pull`.
+- **Not a forge.** A read-only mirror: clone and pull, but no pushes, issues or
+ pull requests.
- **Not a downloader.** It supplies no downloader — you install `yt-dlp`,
`ffmpeg` and a transcription backend yourself, and it drives them.
- **Not cloud transcription.** Your hardware, your electricity, your queue.
diff --git a/homepage/content/docs/install.md b/homepage/content/docs/install.md
@@ -1,8 +1,8 @@
# Install
-From an unpacked snapshot to a running editor. The web apps need very little; the
-download-and-transcribe pipeline needs the media tools, and you can add those
-later.
+From a clone (or an unpacked snapshot) to a running editor. The web apps need
+very little; the download-and-transcribe pipeline needs the media tools, and you
+can add those later.
## What you need
@@ -29,8 +29,19 @@ can't fetch anything yet.
## Get the code
-There is no public git repository. The source is published here as a dated
-snapshot tarball:
+The source lives on this site as a read-only git mirror of the main branch —
+there is no GitHub. Clone it:
+
+```sh
+git clone https://archilyzer.pages.dev/source/archilyzer.git archilyzer
+cd archilyzer
+```
+
+`git pull` brings you up to date; nothing takes a push. The commit ids differ
+from the private repository's, because machine paths are scrubbed on the way
+out. [Source](/source/) has the details and a browsable copy of every file.
+
+No git? The same tree, without history, is a tarball:
```sh
curl -LO https://archilyzer.pages.dev/downloads/archilyzer-source.tar.gz
@@ -38,9 +49,8 @@ tar xzf archilyzer-source.tar.gz
cd archilyzer
```
-That is a working tree, not a clone — no history, no branches, no remote,
-nothing to `git pull`. Updating means downloading a newer snapshot. See
-[Downloads](/downloads/) for what is and isn't inside.
+That is a working tree, not a clone. Updating means downloading a newer
+snapshot. See [Downloads](/downloads/) for what is and isn't inside.
## Install and start
@@ -122,8 +132,8 @@ up rather than looking healthy and quietly transcribing nothing.
`127.0.0.1` — the published archive included — so opening one up is a deliberate
one-line edit in `.env`. The editor has no login of its own, so the containers
refuse to start if you expose *it* without putting a password or an identity
-provider in front; the error says how. `RUNNING_IN_DOCKER.md`, in the snapshot
-you just unpacked, covers that, GPU transcription, and backups.
+provider in front; the error says how. `RUNNING_IN_DOCKER.md`, in the tree you
+just unpacked, covers that, GPU transcription, and backups.
The same stack runs on Linux and macOS. It is described here because it is the
one place where it is clearly the *better* option, not because it is
diff --git a/homepage/content/docs/what-is-archilyzer.md b/homepage/content/docs/what-is-archilyzer.md
@@ -60,8 +60,8 @@ Steps 1–3 can run unattended on a schedule. See
## What it is not
- Not a hosted service. You install it, you run it, you pay for your own hosting.
-- Not a public repository. The source ships as a
- [dated snapshot](/downloads/) — there is no `git clone` and nothing to pull.
+- Not a forge. The source is a [read-only git mirror](/source/) on this site —
+ clone and pull, but nothing takes a push or a pull request.
- Not a downloader you point at one video. It is built around back catalogues:
thousands of recordings, kept current.
- Not automatic transcription in the cloud. The transcribing happens on your
diff --git a/homepage/e2e/marketing.spec.ts b/homepage/e2e/marketing.spec.ts
@@ -42,6 +42,7 @@ test("every nav destination resolves", async ({ page }) => {
// data — which is why /stats/ always exists and degrades in place.
for (const [label, path] of [
["Docs", "/docs/"],
+ ["Source", "/source/"],
["Downloads", "/downloads/"],
["Stats", "/stats/"],
["Changelog", "/changelog/"],
diff --git a/homepage/e2e/source.spec.ts b/homepage/e2e/source.spec.ts
@@ -0,0 +1,112 @@
+import { test, expect, type Page } from "@playwright/test";
+
+// The /source/ page and the files it points at (common/publish/source.ts).
+// Like downloads.spec.ts, correct in BOTH states: the mirror, the raw tree and
+// the tarball are gitignored build artefacts, so a fresh checkout has none and
+// the page must say so rather than offer a clone that fails. The release gate
+// runs `archilyzer source publish` in the checkout first, so the published
+// branch is the one exercised there.
+//
+// This suite runs against `next dev`, which serves public/ from disk but does
+// NOT serve a directory's index.html at `/<dir>/` (Pages does), and ignores
+// _headers — so indexes are requested by name and content types are left to
+// the preview deploy's checks (and to app/lib/headers.test.ts).
+//
+// E2E_EXPECT_SOURCE=1 (declared in playwright.config.ts) makes the empty state
+// a FAILURE of the three data tests, so a gate that published first cannot
+// pass with a publish that silently produced nothing, or a loader that always
+// says "empty".
+const EXPECT_SOURCE = process.env.E2E_EXPECT_SOURCE === "1";
+
+async function published(page: Page): Promise<boolean> {
+ await page.goto("/source/");
+ const has = (await page.getByTestId("source-clone").count()) > 0;
+ if (EXPECT_SOURCE) {
+ expect(has, "E2E_EXPECT_SOURCE=1, and /source/ shows its empty state: run `archilyzer source publish` first").toBe(true);
+ }
+ return has;
+}
+
+test("says what the source is, and that nothing here takes a push", async ({ page }) => {
+ await page.goto("/source/");
+ await expect(page.getByRole("heading", { level: 1 })).toContainText("Source");
+ await expect(page.getByText(/There is no GitHub, by choice\./)).toBeVisible();
+ await expect(page.getByText(/Nothing here takes a push or a pull request\./)).toBeVisible();
+ await expect(page.getByRole("heading", { name: "What is mirrored" })).toBeVisible();
+ await expect(page.getByRole("heading", { name: "License" })).toBeVisible();
+});
+
+test("with a manifest: the clone command, and one tarball the page, snapshot.json and manifest.json agree on — else the empty state", async ({ page }) => {
+ if (!(await published(page))) {
+ await expect(page.getByTestId("source-empty")).toContainText("No source published in this build.");
+ await expect(page.getByTestId("source-tree-link")).toHaveCount(0);
+ await expect(page.getByTestId("source-tarball-link")).toHaveCount(0);
+ return;
+ }
+ await expect(page.getByTestId("source-clone")).toHaveText(
+ "git clone https://archilyzer.pages.dev/source/archilyzer.git",
+ );
+ const manifest = await (await page.request.get("/source/manifest.json")).json();
+ const snapshot = await (await page.request.get("/downloads/snapshot.json")).json();
+ await expect(page.getByTestId("source-mirror-head")).toHaveText(manifest.mirrorHead);
+ const href = await page.getByTestId("source-tarball-link").getAttribute("href");
+ expect(href).toBe("/downloads/archilyzer-source.tar.gz");
+ const tarball = await page.request.get(href!);
+ expect(tarball.status()).toBe(200);
+ expect((await tarball.body()).length).toBe(manifest.tarball.bytes);
+ const onPage = (await page.getByTestId("source-tarball-sha").textContent())?.trim();
+ expect(onPage).toMatch(/^[0-9a-f]{64}$/);
+ expect(onPage).toBe(manifest.tarball.sha256);
+ expect(onPage).toBe(snapshot.sha256);
+ expect(snapshot.commit).toBe(manifest.mirrorHead);
+});
+
+test("the mirror is a dumb-HTTP git repository: HEAD, info/refs, objects/info/packs", async ({ page }) => {
+ if (!(await published(page))) {
+ await expect(page.getByTestId("source-empty")).toBeVisible();
+ return;
+ }
+ const manifest = await (await page.request.get("/source/manifest.json")).json();
+ const head = await page.request.get("/source/archilyzer.git/HEAD");
+ expect(head.status()).toBe(200);
+ expect(await head.text()).toBe("ref: refs/heads/main\n");
+ const refs = await page.request.get("/source/archilyzer.git/info/refs");
+ expect(await refs.text()).toContain(`${manifest.mirrorHead}\trefs/heads/main`);
+ const packs = await (await page.request.get("/source/archilyzer.git/objects/info/packs")).text();
+ const first = /^P (pack-[0-9a-f]+\.pack)$/m.exec(packs);
+ expect(first, packs).not.toBeNull();
+ const pack = await page.request.head(`/source/archilyzer.git/objects/pack/${first![1]}`);
+ expect(pack.status()).toBe(200);
+});
+
+test("the raw tree: an index per directory, encoded hrefs that resolve, brackets included", async ({ page }) => {
+ if (!(await published(page))) {
+ await expect(page.getByTestId("source-empty")).toBeVisible();
+ return;
+ }
+ const hrefs = (html: string) => [...html.matchAll(/<td class="name"><a href="([^"]+)">/g)].map((m) => m[1]);
+ const root = await page.request.get("/source/tree/index.html");
+ expect(root.status()).toBe(200);
+ const rootHrefs = hrefs(await root.text());
+ const dir = rootHrefs.find((h) => h === "common/");
+ expect(dir, rootHrefs.join(" ")).toBeTruthy();
+ expect((await page.request.get(`/source/tree/${dir}index.html`)).status()).toBe(200);
+ const file = rootHrefs.find((h) => h === "README.md");
+ expect(file).toBeTruthy();
+ const readme = await page.request.get(`/source/tree/${file}`);
+ expect(readme.status()).toBe(200);
+ expect(await readme.text()).toContain("Archilyzer");
+
+ // A variable font's name carries brackets: encoded in the href, and served.
+ const fonts = await page.request.get("/source/tree/umtool/report-to-video/fonts/index.html");
+ expect(fonts.status()).toBe(200);
+ const bracketed = hrefs(await fonts.text()).find((h) => h.includes("%5B"));
+ expect(bracketed).toBeTruthy();
+ const font = await page.request.head(`/source/tree/umtool/report-to-video/fonts/${bracketed}`);
+ expect(font.status()).toBe(200);
+});
+
+test("the Downloads page points at the mirror for the history", async ({ page }) => {
+ await page.goto("/downloads/");
+ await expect(page.getByRole("link", { name: "read-only git mirror" })).toHaveAttribute("href", "/source/");
+});
diff --git a/homepage/eslint.config.mjs b/homepage/eslint.config.mjs
@@ -12,6 +12,10 @@ const eslintConfig = defineConfig([
"out/**",
"build/**",
"next-env.d.ts",
+ // The published source (`archilyzer source publish`): the whole repo's
+ // raw tree, copied into public/ and from there into out/. Not this app's
+ // code — tsconfig.json excludes both for the same reason.
+ "public/source/**",
]),
]);
diff --git a/homepage/playwright.config.ts b/homepage/playwright.config.ts
@@ -20,6 +20,10 @@ import {
// summary it reads instead of
// public/homepage-summary.json (app/lib/summary.ts,
// ignored by a production build)
+// E2E_EXPECT_SOURCE `1` (set by whoever runs the suite, after
+// `archilyzer source publish`): e2e/source.spec.ts
+// FAILS on the /source/ page's empty state instead
+// of accepting it
const PORT = portFor("HOMEPAGE_E2E_PORT");
const baseURL = `http://localhost:${PORT}`;
diff --git a/homepage/public/_headers b/homepage/public/_headers
@@ -6,3 +6,44 @@
# for hours while still absorbing a burst of downloads.
/downloads/*
Cache-Control: public, max-age=300, must-revalidate
+
+# The published source (`archilyzer source publish`, common/publish/source.ts).
+/source/manifest.json
+ Cache-Control: public, max-age=300, must-revalidate
+/source/archilyzer.git/*
+ Cache-Control: public, max-age=300, must-revalidate
+# The raw tree is PLAIN TEXT whatever the extension: wrangler's mime map would
+# send .ts as video/mp2t, .mjs as application/javascript, .md as text/markdown
+# and .html as a page on this origin.
+/source/tree/*
+ Content-Type: text/plain; charset=utf-8
+ X-Content-Type-Options: nosniff
+ X-Robots-Tag: noindex
+ Cache-Control: public, max-age=300, must-revalidate
+# ...except the generated directory indexes and the binary files. Every rule
+# that matches applies, in file order, and a header a LATER rule sets again is
+# APPENDED to the earlier value ("text/plain; …, text/html; …" — wrangler's
+# attachHeaders, the Pages asset server's code). So each override first
+# detaches the tree's Content-Type with `! Content-Type`, then sets its own.
+/source/tree/
+ ! Content-Type
+ Content-Type: text/html; charset=utf-8
+/source/tree/*/
+ ! Content-Type
+ Content-Type: text/html; charset=utf-8
+/source/tree/*.ttf
+ ! Content-Type
+ Content-Type: font/ttf
+/source/tree/*.m4a
+ ! Content-Type
+ Content-Type: audio/mp4
+/source/tree/*.zip
+ ! Content-Type
+ Content-Type: application/zip
+/source/tree/*.onnx
+ ! Content-Type
+ Content-Type: application/octet-stream
+/source/tree/*.svg
+ ! Content-Type
+ Content-Type: image/svg+xml
+ Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'
diff --git a/homepage/tsconfig.json b/homepage/tsconfig.json
@@ -16,5 +16,5 @@
".next/dev/types/**/*.ts",
"**/*.mts"
],
- "exclude": ["node_modules"]
+ "exclude": ["node_modules", "public", "out"]
}
diff --git a/plans/release-12.md b/plans/release-12.md
@@ -281,4 +281,497 @@ No High or Medium findings. The coordinator asked for two of the Lows to be fixe
- **After the fixes:** tsc is clean (41 s, all seven packages), and the grep gate is empty at the
new tip. Per the coordinator, e2e was not re-run for a type comment and a changelog line.
+### Slice R, as shipped — `archilyzer source publish` + `/source` (2026-09-28)
+
+Branch `r12/source-mirror` off `main` `e6c5d2e3` (slice Q merged), worktree
+`~/Projects/r12-source-mirror` (worktree #10: editor 4001, homepage e2e 4040), one Opus
+implementer. Plan: [`source-mirror.md`](source-mirror.md), "Slice R", R1–R7. Why: the operator
+asked for the repo on the project site, read-only, clonable from static files, with a gate that
+refuses to publish a denied literal. `main` did not move during the slice (`git merge main`:
+already up to date). Scratch files `r-m-*` in the job's `tmp`.
+
+**What shipped.**
+- **R1, `common/publish/source.ts`: `publishSource(opts)`** — 0 published / skipped / checked,
+ 1 refused or cancelled; a `SourceRefusal` is logged as `[source] REFUSED: …`. The steps are the
+ plan's 1–17:
+ - `git rev-parse` of the git COMMON dir's `refs/heads/main`;
+ - the two operator files, parsed; a missing one is a refusal naming it (`~/`-relative);
+ - the skip;
+ - `resolveFilterRepo()`: `git filter-repo`, else `pipx run --spec git-filter-repo==2.47.0`,
+ else the install line;
+ - `git clone --no-local --bare --single-branch --no-tags --branch main` into
+ `mkdtemp(<ARCHILYZER_SOURCE_SCRATCH>/archilyzer-source-)`, origin removed;
+ - filter-repo `--force --quiet --replace-refs delete-no-add --replace-text R --replace-message R`,
+ then `filter-repo/` deleted;
+ - `repack -a -d -q --max-pack-size=20m`, `prune-packed`, `pack-refs --all`, `update-server-info`.
+ It refuses on loose objects, no `P` line, any ref but `refs/heads/main`, or a HEAD that is not
+ main.
+ - the object gate; the tree (`git archive` → `tar -x` → pages); the tarball (`--format=tar.gz
+ -9 --prefix=archilyzer/`) and `snapshot.json` in the `Snapshot` shape;
+ - the mirror staged from an allowlist; the manifest staged; the file gate; the limits;
+ `--check` stops; the link-safe install (manifest removed first, written last); one summary line.
+
+ Every child goes through `runChildIntoLog` with `AbortSignal.any([signal, timeout])`, in an
+ environment with the `GIT_DIR`-family variables cleared. The one exception is the audit's
+ binary `cat-file` stream (a `spawn`, with the signal). `clearPublishedSource` is what
+ `--no-source` runs. `common/lib/sourceManifest.ts` is the leaf contract: `MIRROR_DIR`,
+ `CLONE_URL`, `TREE_HREF`, `TARBALL_HREF`, `SourceManifest` and `parseSourceManifest`.
+- **R2, `sourceAudit.ts`.**
+ - `parseDenylist` (`i:`, `#` comments, trimmed, deduped).
+ - `scanBuffer`, and `redactHit`: ±24 bytes, every byte any hit covers masked, a half-shown
+ occurrence included.
+ - `maskLiterals` for every path or line quoted from the repo.
+ - `auditObjects`: ONE `cat-file --batch-all-objects --unordered --batch` stream with a framing
+ parser — blobs and commits/tags whole, trees by entry NAME (not their binary ids).
+ - `auditFiles`: contents, paths, a `.gz` decompressed; a symlink is a refusal.
+ - `runGitleaks`: cwd = scratch, so no checkout's ignore file applies. 0 is clean, 3 is parsed
+ findings, anything else refuses; not on PATH is "skipped" with a WARNING.
+ - `auditBare`: blob hits get their path in history, and gitleaks runs only when the literal
+ audit is clean.
+ - `formatAuditReport`: `#n (x…, len L)`, counts per literal and kind, the first 20 contexts,
+ "… and N more", then the plan's closing line.
+- **R3, `sourceTree.ts`** (`hrefFor`, `escapeHtml`, `renderTreeIndex`, `writeTreeIndexes`, which
+ refuses a tracked `index.html` or a symlink), and the `_headers` block — with the correction
+ below.
+- **R4, the homepage.**
+ - `app/lib/source.ts` `loadSourceManifest()` (also needs `info/refs` and the tarball).
+ - Nav: Source after Docs.
+ - `app/source/page.tsx`: the plan's copy verbatim; `source-clone`, `source-mirror-head`,
+ `source-tree-link`, `source-tarball-link`, and `source-tarball-sha` for the spec; the empty
+ state `source-empty`.
+ - A shared `components/Fact.tsx`.
+ - Downloads: one sentence linking "read-only git mirror", and the empty state names
+ `archilyzer source publish`; every phrase `downloads.spec.ts` asserts is kept.
+ - `marketing.spec.ts`' nav list gains Source.
+- **R5, the docs.** `create-archives.sh` is deleted. README and SETUP lead with the clone and keep
+ the tarball as the no-git path. PUBLISH.md gains "The source mirror (homepage)". The homepage's
+ *Install*, FAQ, *What is Archilyzer* and `content/README.md` stop saying "there is no public
+ repository". `grep -rn 'create-archives' README.md SETUP.md homepage/` is empty.
+- **R6 and the rest.**
+ - `getPaths()` gains `configDir`, `sourceScrubFile`, `sourceDenylistFile` and
+ `sourceScratchDir`, from `ARCHILYZER_CONFIG_DIR`, `SOURCE_SCRUB_FILE`, `SOURCE_DENYLIST_FILE`
+ and `ARCHILYZER_SOURCE_SCRATCH`, declared as `paths` in `envVars.ts`.
+ - `HOMEPAGE_PUBLIC_DIR`'s `readBy` names `source.ts`, and `TRANSCRIPTS_DIR`'s doc says umtool
+ reads `<it>/channels` too (review Q(c)). `ENVIRONMENT.md` is regenerated.
+ - The CLI rows `build homepage [--no-source]`, `source publish [--force] [--check]
+ [--keep-scratch]` and `source audit [<git dir>]`.
+ - `buildHomepage`: compose, then the source step (a lazy import; `publishSource` and
+ `clearSource` are seams), then `next build`.
+ - doctor's "source publish" block. It is never a failure; it WARNs when the operator files exist
+ but no filter-repo is installed, or a file is readable by others. It prints rule counts and
+ modes, never contents.
+
+**Corrections to the plan** (each found by running it):
+- **`_headers`: later rules APPEND, they do not win** (`f218ed86`).
+ - wrangler 4.88's `attachHeaders` (the Pages asset server's code, read in its `cli.js`) `set`s
+ a header on the first matching rule and `append`s it on every later one. The plan's exact
+ text served a directory page as `text/plain; charset=utf-8, text/html; charset=utf-8` and a
+ font as `text/plain; charset=utf-8, font/ttf`.
+ - Each override now starts with `! Content-Type`. A replica of wrangler's parse and attach
+ (`$T/r-m-headers-sim.mjs`) gives single values for `/source/tree/`, `…/common/`, a `.ts`, the
+ bracketed `.ttf` and `page.tsx`, the `.onnx` and `README.md`.
+ - The preview deploy is still the live proof.
+- **The tsconfig must exclude `out` too** (`3fa5ff18`).
+ - `next build` copies `public/` into `out/`. The SECOND build with a mirror failed its type check
+ on `out/source/tree/common/jobs/registry.ts`, a second `declare global var __yttJobRegistry__`.
+ - With `out` excluded, the homepage program is 871 files, none from the mirror; homepage tsc
+ takes 11 s with a mirror in both dirs.
+ - The homepage's `eslint.config.mjs` ignores `public/source/**` (it already ignored `out/**`).
+ Lint itself was not run.
+- **The published manifest carries no `rulesHash`.**
+ - `plans/` ships in the mirror, and so does the plan's description of the two files. What is
+ left unknown is the hostname and the address, so a published hash of the rules would confirm
+ a guess at them.
+ - The skip key (`{sourceCommit, rulesHash}`) is `homepage/.source-publish.json`, beside
+ `public/` and gitignored. A missing key rebuilds; the round trip asserts it is absent from the
+ manifest.
+- **The mirror has a loose `refs/heads/main` beside `packed-refs`.** git treats a directory as a
+ repository only with a `refs/` dir, so without it the `file://` clone and `source audit` of the
+ published dir fail. An empty dir would not survive a deploy.
+- **filter-repo's `--replace-text` does not skip `#` lines** (it would replace a comment as a
+ literal; `get_replace_text` in 2.47). The step writes `replace.txt` without comments or blank
+ lines.
+- **The step adds three safety flags:** `--no-tags`, `--replace-refs delete-no-add` (2.47's
+ default is `update-no-add`, stated for determinism), and a check that the mirror holds exactly
+ `refs/heads/main`.
+- **`--no-source` removes the published source** (manifest first; a linked `downloads/` goes as a
+ link, its target untouched). It does not leave the source there: a copy audited against older
+ rules would ship ungated. That removal gives the empty state R7 expects.
+- **The header nav moves from `sm` to `md`:** five labels do not fit beside the wordmark at 640 px.
+- **The tree has 4 `.ttf`**, not 3 (IBM Plex Mono Bold, O5); `*.ttf` covers them.
+- **No `branch` option:** `SOURCE_BRANCH = "main"`, an operator decision. `now` is `() => Date`.
+- **The packs are 37.5 MB in two** (20,897,277 + 16,633,904 bytes), not the plan's 21.5 MB in one.
+ The history grew by about 150 commits since the plan, and the 20 MB split costs deltas across
+ the two packs. That is well inside the limits.
+
+**The operator's files, as created, refuse the publish** (an action for the rollout, not the
+slice):
+- **The run:** `pnpm archilyzer source publish --check` with `~/.config/archilyzer/*` as the
+ parent created them, at `main` `e6c5d2e3`. It ended `AUDIT REFUSED: 8 hits in 21,441 objects
+ (1,702 commits) against 6 denied literals`, with every hit `#1 (r…, len 5)` in a blob:
+ - `plans/source-mirror.md`, two versions, 3 hits each: the plan's own grep gates (`git grep -c
+ -i '<it>' …`, `git -C <clone> grep -c -i <it> …`) and the "a future transcript quote "<it>
+ that"" risk line;
+ - `plans/release-12.md`, two versions, 1 hit each: slice Q's grep-gate line.
+- **Why the scrub rules miss them.** They cover the backticked spelling and the `/run/media/` path
+ (review I2: no hit came from there), not the bare, single-quoted or double-quoted name.
+- **The operator's step:** add a rule for the bare name (`<name>==>user` covers every spelling,
+ the backticked one included) or one per spelling. Then `archilyzer source publish --check`
+ (about 25 s) must end `check passed`.
+- **How R7 ran anyway,** with the operator files untouched and the gate not weakened:
+ - `SOURCE_SCRUB_FILE` = a scratch file of ONE rule, `<name>==>user`, written with `$(id -un)`,
+ with the REAL denylist. The check was clean: 4 literals, gitleaks clean.
+ - `source audit` of both clones and of the published dir with the REAL files (6 literals) is
+ clean, below.
+
+**Gates** (from the worktree root; logs `$T/r-m-*.log`). Every log of a run over the real files was
+first scanned by `$T/r-m-leakcheck.mts` (counts per literal label, any ASCII case) and read through
+`$T/r-m-maskview.mts`. The step's own lines carried 0 literals; the only hits were pnpm's and
+doctor's home-directory paths.
+- **tsc** was clean before every commit (`r-m-tsc-{0..3}.log`):
+ - 215 s at the first run, 168 s, 59 s, and 73 s with a mirror in `public/` AND `out/`. Another
+ session's tsc was running during the first and the last.
+ - The homepage alone: 11 s, 871 files, none from the mirror.
+- **Unit:**
+ - common **2,138/2,138** (2,114 + 24: sourceAudit 7, sourceTree 5, source 7, build +1, `_cli`
+ +3, doctor +1). The two filter-repo tests RAN (`ok 1832` round trip, `ok 1833` planted-literal
+ refusal); 0 skipped.
+ - `test:scripts` **185 + 1 skipped**; editor unit **85/85**; mcp **269/269**; homepage unit
+ **2/2**.
+- `pnpm archilyzer docs env --check` exits **0**. The editor's `next build` is **ok** (43 s); it
+ bundles `buildHomepage`'s lazy import.
+- **`archilyzer build homepage`** (from `common/`, `SOURCE_SCRUB_FILE` as above):
+ - **ok in 38 s**, against 19 s for `--no-source`. The source step took **19 s**: filter-repo
+ 5.3 s, gitleaks 7.3 s, `[source] published main e6c5d2e322b3 as 78dc0126382b: 2459 files,
+ 69.5 MB (mirror 2 packs, tree 412 dirs), tarball 6.9 MB sha256 c4ceb6d110ee`.
+ - A rebuild with nothing changed logged `[source] up to date at e6c5d2e322b3; skipping`.
+ - **Deterministic:** two runs gave the same `mirrorHead` and tarball sha; the real files' two
+ runs gave `9b880270c49d` both times.
+- **What it published** (`homepage/out`):
+ - **2,640 files, 78.1 MB** in all; `source/` 2,465 files, 65.8 MB;
+ - the mirror: 9 files, 38.1 MB, 2 packs;
+ - the tree: 2,035 files + 412 pages, 27.5 MB;
+ - the tarball: 7,246,992 bytes.
+ - **Against the limits:** 2,459 staged of the step's 15,000 (2,640 of Pages' 20,000); the
+ largest file is 19.93 MiB, against 24 MiB (25 MiB).
+- **Two clones, both HEAD = `manifest.mirrorHead` `78dc0126382b…`:**
+ - `git clone file://$WT/homepage/out/source/archilyzer.git`: 2 s.
+ - `git clone http://127.0.0.1:8765/source/archilyzer.git` from `python3 -m http.server 8765
+ --bind 127.0.0.1` in `homepage/out`: 1 s. The protocol was dumb: `info/refs?service=…` 200,
+ then `HEAD`, `objects/info/packs`, both `.idx` and both `.pack`; the loose-object probes 404.
+ - `:8765` was free before, the server was killed, and it was free after.
+- **The user name:** `git grep -c -i -F "$(id -un)" $(git rev-list --all) | wc -l` is **0** in both
+ clones (1,702 revisions). It is 0 in the identities and messages and 0 in the paths too; the
+ hostname is 0.
+- **`archilyzer source audit`** with the real files (6 literals) is **clean** on the file clone
+ (13 s), the http clone (11 s) and the published dir (10 s): 21,441 objects, 1,702 commits,
+ gitleaks "1530 commits scanned … no leaks found".
+- **The gate, exercised:** `SOURCE_DENYLIST_FILE` = a scratch file of `Co-Authored-By`, then
+ `source publish --check`:
+ - exit **1** in 14 s, `AUDIT REFUSED: 1476 hits … #1 (C…, len 14): 8 in blobs, 1468 in
+ commits`, 20 context lines and "… and 1456 more";
+ - the log holds the literal **0** times;
+ - `public/source/manifest.json`'s mtime and size are unchanged (`1790616152 1057`).
+- **`build homepage --no-source`:** `out/source/index.html` holds `data-testid="source-empty"`
+ and "No source published in this build."; `/downloads/` says "no source snapshot attached".
+- **No filter-repo:** with PATH = a scratch dir of two symlinks (`git`, `node`; nothing
+ uninstalled), `git filter-repo` is "not a git command", and `source publish --check` exits 1 with
+ `REFUSED: git-filter-repo is not installed and pipx is not on PATH — install it once: \`pipx
+ install git-filter-repo\` …`.
+- **The resolved filter-repo** is `git filter-repo` (`~/.local/bin/git-filter-repo`, pipx-installed
+ 2.47.0), whose `--version` prints `a40bce548d2c`; git is 2.55.0. The `pipx run` fallback was not
+ run (it needs the network).
+- **`archilyzer doctor`** prints the block:
+ ```
+ source publish
+ ok filter-repo git filter-repo a40bce548d2c
+ ok gitleaks gitleaks version is set by build process
+ ok scrub rules ~/.config/archilyzer/source-scrub.txt (2 rules, mode 600)
+ ok denylist ~/.config/archilyzer/source-denylist.txt (3 literals, mode 600)
+ -- published main e6c5d2e322b3 as 78dc0126382b, 2026-09-28T17:22:29.301Z (2458 files)
+ ```
+ The real output prints the absolute paths. `2458` is the manifest's `files`, which does not
+ count the manifest itself.
+- **homepage e2e** (`node scripts/worktree.mjs run -- pnpm --filter homepage run e2e`; the queue
+ was free):
+ - **36 passed, 0 skipped, 0 failed, 1.1 min**, WITH the manifest present:
+ `loadSourceManifest()` from `homepage/` gave mirror head `78dc0126382b`, so the published
+ branch of every `source.spec` test ran.
+ - The empty state too: with `public/source` and `public/downloads` moved aside and restored
+ after, `source.spec` + `downloads.spec` + `marketing.spec` gave **16 passed** (30 s).
+ - `downloads.spec.ts` is unchanged.
+- **Numbers tool:** none.
+
+**They bite:**
+- The planted-literal test fails if the object walk misses the blob.
+- The redaction test pins a half-shown second occurrence.
+- `build.test.ts`' fake refusal proves `next build` never runs.
+- The restricted-PATH run proves the install line.
+- The `Co-Authored-By` run proves the gate refuses a literal the scrub does not touch, in commits
+ AND blobs.
+
+| sha | what |
+|---|---|
+| `25f5e241` | `homepage:` `/homepage/public/source` gitignored (the Tailwind note), tsconfig excludes `public` — first, before any mirror |
+| `7fdbe2dc` | `common:` `sourceAudit.ts`, `sourceTree.ts`, `sourceManifest.ts` + tests |
+| `fc31aab5` | `common:` `publishSource` (`source.ts`) + tests; `getPaths()` / `envVars.ts` / `ENVIRONMENT.md` |
+| `2f08cb47` | `common:` `buildHomepage` runs the step; the CLI rows; doctor's block + tests |
+| `1a11a2bb` | `homepage:` `/source/`, nav, `Fact`, Downloads, `_headers`, docs content, `source.spec.ts`, marketing nav |
+| `2cd189f3` | `docs:` README, SETUP, PUBLISH "The source mirror (homepage)"; `create-archives.sh` deleted |
+| `c448b392` | `changelog:` the editor and homepage `[Unreleased]` bullets |
+| `f218ed86` | `homepage:` `_headers` overrides detach `Content-Type` first (correction) |
+| `3fa5ff18` | `homepage:` tsconfig excludes `out`; eslint ignores `public/source/**` (correction) |
+| _this_ | `plans:` this record |
+
+**Found and left** (not this slice's files):
+- **`.dockerignore:20-22`** still names `create-archives.sh`, and it does not exclude
+ `homepage/public/source/`. A `docker build` from a checkout that has published would send about
+ 66 MB of mirror and tree in the build context.
+- **`common/lib/project.ts:37-38`** (`PROJECT_DOWNLOADS_URL`'s comment) says "there is no public git repository".
+- **The editor's /sites Homepage section** (`HomepageBuildButtons.tsx:170`) does not say that the
+ build publishes the source, or that it can refuse.
+- **`export/CHANGELOG.md`'s released entry** names `create-archives.sh`. It is history, and is left.
+- **The label `#n (x…, len L)`** shows a literal's first character and length in the logs. This is
+ the plan's format; nothing of it is published.
+- **gitleaks** scans 1,530 of 1,702 commits: its `git log -p` skips merges. The literal audit reads
+ every object.
+
+**Changelog.**
+- **`editor/CHANGELOG.md`:** one `[Unreleased]` bullet. The build step and the gate; the operator
+ files, and that the build refuses without them; `pipx install git-filter-repo`; the skip, the
+ CLI rows and doctor; `create-archives.sh` gone.
+- **`homepage/CHANGELOG.md`:** one `[Unreleased]` bullet. `/source/` and its nav entry, the empty
+ state, the tarball regenerated per build, the docs, the nav at `md`, and `_headers`.
+
+**For the parent and the operator:**
+1. Add the scrub rule above, then run `archilyzer source publish --check`.
+2. The editor's /sites homepage job finds `git filter-repo` only if the editor's PATH includes
+ `~/.local/bin`. Otherwise it falls back to `pipx run`, which needs the network on first use.
+3. The worktree's `homepage/public/source`, `homepage/out` and `homepage/.source-publish.json` were
+ made with the scratch rule. They are disposable, and nothing here deployed them.
+
+**Review** (verdict **SHIP AFTER FIXES**; a read-only Opus review of `e6c5d2e3..6ddfd498`,
+`$T/r-review.md`). The parent then added the missing scrub rule to the operator file:
+`source publish --check` with the real files exits 0. The coordinator ruled that M1 and the
+listed Lows be fixed on the branch. `main` had moved by one `plans:` commit (the release 11
+rollout record), merged at `439eb106` with no conflict.
+- **M1 — fixed** (`7d64054c`, and `78b32636` below). **A refusal left the previous publish in
+ place**, in `homepage/public` and in the last build's `homepage/out`, for a deploy-only or a
+ raw `next build` to ship under rules it was never audited against. Now it is withdrawn at three
+ points:
+ 1. **`publishSource`:** once the rules are loaded, any outcome but success — an audit hit, a
+ limit, a missing tool, a cancel, a crash — runs `removePublishedSource`. That removes the
+ manifest first, then the mirror, the tree, the tarball, `snapshot.json` and the skip key.
+ `--check` still writes nothing, this included: it is a dry run, and the deploy check below
+ covers what it cannot.
+ 2. **`buildHomepage`:** a non-zero source result also removes `out/source` and the two download
+ files from `homepage/out`.
+ 3. **`deployHomepage` asks `publishedSourceProblem` before every deploy**, preview included.
+ - An `out/` with source artefacts ships only when the skip key says the publish was made
+ under TODAY's rules and step (`rulesHash`), of TODAY's `main`, and is the publish in `out/`:
+ the same mirror head, and a tarball whose sha256 matches.
+ - It keys on the artefacts and the `/source` page, not on `out/source` existing (`78b32636`).
+ A `--no-source` build still renders the page into `out/source/index.html`, and the first
+ version refused exactly that build; I found it by running one.
+ - The page with no artefacts beside it is a `--no-source` build, and deploys as before.
+ - No page at all is a refused build (or one from before the page), and it refuses.
+ - **Tests:** the round trip denies a literal the mirror holds, and the publish then exits 1 with
+ `public/source`, the tarball, `snapshot.json` and the key gone. The skip test covers a changed
+ rule. `build.test` covers the `out/` removal and the deploy refusals. `publishedSourceProblem`
+ is covered on a real publish: ok, tampered tarball, newer `main`, other rules, no record.
+- **Q3/L5 — fixed** (`6936f6e2`). A label says where the literal was written: `denylist line 3
+ (len 5)`, `scrub line 2 lhs (len 11)`, `built-in home rule (len 11)`. No character of the
+ literal appears.
+- **L4 — fixed** (`6936f6e2`).
+ - A hit is listed by kind, object id (with the blob's path in history), byte offset, and for a
+ commit or tag the header field (`author`, `committer`, `tagger`, …) or `message`; a tree hit
+ by entry number. `redactHit` is deleted: no byte of an object is printed.
+ - The test asserts the report carries none of the planted text's neighbours (`/srv/`,
+ `example.invalid`, `Planted <`, the message).
+ - PUBLISH.md and the editor changelog show the new format.
+- **L1 — fixed** (`6936f6e2`): a tracked `404.html` anywhere in the tree is refused, with a test.
+- **L2 — fixed** (`7d64054c`).
+ - A UTF-8 BOM and CRLF are dropped from both operator files (`operatorLines`), and a trailing
+ `/` from the home dir.
+ - An empty left side (`==>x`, `literal:==>x`, `regex:==>x`, `glob:`) is a refusal naming its
+ line.
+ - The reviewer's reproduction is a test: a BOM + CRLF scrub file still scrubs, and still
+ denies its left side. The round trip now runs with a BOM + CRLF scrub file and denylist.
+- **L3 — fixed** (`7d64054c`): the published manifest (and `SourceManifest`) no longer carries
+ `audit.literals`. The log still gives the count.
+- **L6 — left, as ruled:** compressed content is opaque to the byte search. The reviewer
+ decompressed every compressed or binary blob in the history (a zip, PNGs, icons) and found 0
+ hits. PUBLISH.md records it as a known limit, and says a binary is audited, never scrubbed.
+- **L7 — fixed** (`d3c680ea`). `.dockerignore` excludes `homepage/public/source/` and
+ `homepage/.source-publish.json`, and no longer names `create-archives.sh`.
+- **L8 — fixed** (`7d64054c`), for a checkout with no git repository.
+ - `buildHomepage` passes `noRepository: "empty"`. The step logs `[source] no git repository
+ here; nothing to mirror — the /source page will show its empty state`, removes an old publish,
+ and returns 0.
+ - `archilyzer source publish` exits 1 with the same sentence, and no raw git error.
+ - A test runs both over a temp dir with no operator files.
+- **L9 — fixed** (`7d64054c`, `91728a21`).
+ - The round trip reads EVERY object in the published packs from a plain file copy (`cat-file
+ --batch-all-objects`), and asserts `objects/pack` holds only `pack-*.{pack,idx}`.
+ - `homepage/app/lib/headers.test.ts` pins `_headers` with a replica of wrangler's parse and
+ attach:
+ - each `/source/tree` override says `! Content-Type` first;
+ - each path gets ONE value;
+ - the file stays inside wrangler's limits.
+ - `E2E_EXPECT_SOURCE=1` makes the empty state FAIL `source.spec.ts`' three data tests. It is
+ declared in the homepage playwright config and the env registry. It bites: in the empty state
+ with the flag set, **3 failed, 2 passed**.
+- **L10 — fixed** (`7d64054c`, `91728a21`). `parseSourceManifest` checks every number the page
+ reads, and `loadSourceManifest` takes the public dir as a seam. Unit tests cover common (2) and
+ homepage (2): a malformed, wrong-version or orphaned manifest is null, never a throw.
+- **L11 — fixed** (`7d64054c`).
+ - `SOURCE_STEP_VERSION` (now 2, with a comment to bump it whenever the scrub or the audit
+ changes) is in the rules hash.
+ - The skip key also holds the filter-repo label and version, and the mirror head.
+ - A test shows another filter-repo version does not skip.
+- **L12 — fixed** (`7d64054c`).
+ - A scratch root that lands (through symlinks) inside the checkout or the public dir is refused.
+ - `--keep-scratch` deletes `replace.txt` (written mode 600) and says so.
+ - Tests cover both.
+- **L13 — fixed** (`d3c680ea`).
+ - The `project.ts` comment names the mirror.
+ - PUBLISH.md's tsconfig line says `public` and `out`. The "new spelling" line now says the
+ implied denial is the exact bytes, and a new spelling is caught only by the denylist.
+ - The editor's /sites Homepage section gains one `<p>`: Build also publishes the source and can
+ refuse; a refusal removes it from `homepage/out` too; Deploy refuses an unaudited source.
+ I did not run the editor e2e for it:
+ - `sites-homepage.spec.ts`' assertions on that group are `toContainText` substrings of the
+ existing paragraph, role-and-name lookups for buttons, and `getByText(/^Queued/)` and
+ `"Cancelled"` (exact). A new sibling paragraph cannot break any of them.
+ - Its build job is held on the queue and cancelled, so the source step never runs there.
+- **The runbook point (PUBLISH.md, `d3c680ea`):**
+ - a Pages preview is public, and every deployment stays reachable at its hash URL until it is
+ deleted;
+ - the private literals go in the denylist before ANY deploy;
+ - if something private ships, delete that deployment, because a newer deploy does not remove it;
+ - the editor's process needs `~/.local/bin` on its PATH.
+- **The five questions, as ruled:**
+ 1. The rules hash in the gitignored `.source-publish.json` is **acceptable**. The step version
+ has been added (L11).
+ 2. `--no-source` removing the previous publish is **right**.
+ 3. The first character is **dropped** (above).
+ 4. The loose `refs/heads/main` is **acceptable**.
+ 5. `.dockerignore` is **fixed here** (L7).
+
+| sha | what |
+|---|---|
+| `439eb106` | merge `main` (the release 11 rollout record) |
+| `6936f6e2` | `common:` labels by provenance, no object bytes in the report, `404.html` refused (Q3, L4, L1) |
+| `7d64054c` | `common:` a refusal withdraws; the build's `out/` copy; the deploy check; L2, L3, L8, L10, L11, L12 |
+| `91728a21` | `homepage:` `E2E_EXPECT_SOURCE`; the loader and `_headers` unit tests (L9, L10) |
+| `d3c680ea` | `docs:` PUBLISH.md (withdrawal, previews, no-repo, report format, L6); `.dockerignore`; `project.ts`; the /sites copy; the changelog bullet |
+| `78b32636` | `common:` the deploy check keys on the artefacts and the `/source` page (found by running `--no-source` against it) |
+| _this_ | `plans:` this review record and `source-mirror.md`'s "As shipped" note for R |
+
+**Gates after the fixes** (logs `$T/r-m-*.log`):
+- **tsc:** clean before every commit (36–55 s).
+- **Unit:**
+ - common **2,146/2,146** (+8: sourceAudit +2, source +3, build +1, sourceManifest +2). The
+ filter-repo tests ran, 0 skipped.
+ - homepage unit **7/7** (+5); editor unit **85/85**; `test:scripts` **185 + 1 skip**.
+ - `docs env --check` exits **0**.
+- **`source publish --check` with the REAL files** exits **0** in 19 s:
+ - `audit clean: 21,447 objects (1,703 commits), 2,459 staged files against 6 denied literals;
+ gitleaks clean`;
+ - it would publish main `e56101fdee5d` as `20c367613f75`;
+ - a count-only grep of the log for `$(id -un)` gives **0**.
+- **A real `build homepage` in the worktree, with the real files:**
+ - **ok in 38 s** (the source step 19 s); `homepage/out` holds **2,640 files, 77.9 MB**;
+ - the mirror is 37.9 MB in 2 packs, the largest 20,966,235 bytes (19.99 MiB, against the 24 MiB
+ limit); the tree is 2,035 files and 412 dirs; the tarball 7,249,703 bytes.
+- **Dumb-HTTP clone** from `python3 -m http.server 8765 --bind 127.0.0.1` in `homepage/out`:
+ - HEAD `20c367613f75…` = `manifest.mirrorHead`;
+ - the user-name grep over its 1,703 revisions gives **0**;
+ - `:8765` was free before, the server was killed, and it was free after.
+- **The refusal, exercised through `build homepage`** (`SOURCE_DENYLIST_FILE` = a scratch file of
+ `Co-Authored-By`):
+ - exit **1** in 13 s: `1477 hits … denylist line 1 (len 14): 8 in blobs, 1469 in commits`, with
+ each listed as `commit <id> (message, byte N)`;
+ - the log holds the literal **0** times, and the source step's lines hold 0 of the 6 real
+ literals;
+ - afterwards `public/source`, the public tarball, the skip key, `out/source` and the out
+ tarball are **all gone**, and `out/downloads/index.html` stays;
+ - a good rebuild restored them all, and the deploy check then says "ok".
+ - Before the refusal, the deploy check over the real `out/` said ok under the real rules, and
+ "audited under other rules" with the scratch denylist.
+- **`build homepage --no-source`:** `out/source` holds only the page. The tarball is gone, and the
+ deploy check says ok.
+- **homepage e2e with `E2E_EXPECT_SOURCE=1`:** **36 passed**, 0 skipped, 51 s, with the manifest
+ present. The bite run is above: 3 failed and 2 passed in the empty state.
+
+**Re-review** (verdict **SHIP**; `$T/r-review.md`, "Re-review"). M1 is confirmed closed, and every
+deploy path goes through the check. The coordinator ruled that `--check` not withdrawing is
+accepted as built, and that the four new Lows be closed before the merge:
+- **R2-L1 — fixed** (`831763da`). A refusal that names a tree path (the symlink, `index.html` and
+ `404.html` refusals) printed a literal that spans path components (`a/b`) in full. The object
+ walk reads entry names one at a time, so it cannot see such a literal.
+ - `[source] REFUSED: …` now goes through `maskLiterals` with the loaded literals, in both
+ `publishSource` and `auditSource`.
+ - The kept-scratch path, the report's scrub-file path and the audit's "auditing <dir>" line are
+ masked too.
+ - Child stderr was already masked wherever it is quoted or echoed: git's refusal tails,
+ filter-repo's echo, gitleaks' lines. `subject` comes from the scrubbed mirror.
+ - The test plants `plant/secret` as a denied literal above a tracked symlink, then above a
+ tracked `index.html`. Both refusals print `[REDACTED]/…`.
+- **R2-L2 — fixed** (`831763da`). The state carries `contentDigest`, from `sourceDigest()`:
+ - it covers every published file: `source/archilyzer.git/**`, `source/tree/**`,
+ `source/manifest.json`, the tarball and `snapshot.json`;
+ - it hashes the sorted path, the size and a streamed sha256 of each; a symlink or a missing
+ piece only makes it differ;
+ - the skip recomputes it over `public/`, and the deploy check over `out/`.
+
+ A swapped tree file and a flipped pack byte are refused. On the real `out/` (2,640 files) the
+ whole check takes **0.58–0.76 s**, of which the digest is **0.48–0.54 s** (three runs).
+- **R2-L3 — fixed** (`831763da`).
+ - **The skip:** each part of the key (filter-repo, gitleaks, content digest, rules) and an edited
+ `public/` file is changed alone, through a publish that REACHES the skip. The filter-repo is
+ `false`, so a skip returns 0 and a miss returns 1.
+ - **The step version:** `rulesHashOf` is shown to move with it, and `loadSourceRules` uses
+ `SOURCE_STEP_VERSION`.
+ - **The deploy check's mirror-head and gitleaks comparisons** each refuse with their own
+ sentence.
+ - **Proved by reverting each fix** (`$T/r-m-mutate.py` against `source.ts`, restored after):
+ every revert fails its test. The eight reverts are:
+ - the deploy check's mirror-head comparison;
+ - the step version in the hash;
+ - filter-repo, gitleaks and the digest in the skip key (three reverts);
+ - gitleaks and the digest in the deploy check (two reverts);
+ - the refusal masking.
+- **R2-L4 — fixed** (`831763da`).
+ - The state carries `gitleaksIdentity()`: "skipped", "absent", or the version line plus the
+ binary's sha256. The sha is there because this machine's gitleaks prints "version is set by
+ build process" for every release.
+ - The skip and the deploy check compare it.
+ - `SOURCE_STEP_VERSION` is 3.
+- **The rollout note** (PUBLISH.md, `7940cb0c`, and here): **after the merge, rebuild and restart
+ the live editor (:3001) before any /sites Homepage job.** It runs its built bundle, so until then
+ its Build homepage job has no source step and its Deploy homepage job no source check.
+- **The reviewer's "minor" is left:** a real checkout that unexpectedly reads as "not a git
+ repository" (a worktree whose primary moved) builds with the empty state, withdrawing the
+ source. It is safe for privacy, and it is logged.
+
+| sha | what |
+|---|---|
+| `831763da` | `common:` refusals masked; `contentDigest` and `gitleaksIdentity` in the key; step version 3; the key's tests bite (R2-L1…L4) |
+| `7940cb0c` | `docs:` PUBLISH.md — the deploy key, masking, and rebuilding the editor after the merge |
+| _this_ | `plans:` this re-review record |
+
+**Gates after the re-review fixes:**
+- **tsc:** clean, 35 s.
+- **Unit:** common **2,149/2,149** (+3); the three filter-repo tests ran, 0 skipped. Homepage unit
+ **7/7**. `docs env --check` exits 0.
+- **`source publish --check` with the real files** exits **0** (19 s, would publish mirror head
+ `20c367613f75`). The count-only user-name grep of its log gives **0**.
+- **A real `build homepage`** is ok in 38 s (18 s for the source step). The source step's lines
+ hold 0 user-name occurrences.
+- **The deploy check, called read-only** on that `out/`, says **ok (would deploy)**.
+- **A rebuild with nothing changed** logs `up to date … skipping` (20 s), and the check still says
+ ok.
+- Nothing was deployed. `main` had not moved.
+
## Rollout
diff --git a/plans/source-mirror.md b/plans/source-mirror.md
@@ -419,6 +419,45 @@ empty state; `PATH` without pipx → the install-line refusal; `archilyzer docto
block; homepage e2e 31 → 36 with the manifest present, `downloads.spec.ts` unchanged. Record the
resolved filter-repo command and version.
+**As shipped (2026-09-28, `r12/source-mirror`; record: `release-12.md`, "Slice R, as shipped" and
+its "Review"):**
+- **R1–R6 as written, with these corrections found by running them:**
+ - **`_headers`:** Pages APPENDS a header a later matching rule sets again. Each override now
+ detaches with `! Content-Type` first, and `homepage/app/lib/headers.test.ts` pins it.
+ - **The homepage tsconfig excludes `out` as well as `public`,** and its eslint config ignores
+ `public/source/**`.
+ - **The mirror carries a loose `refs/heads/main`** beside `packed-refs`, because git needs a
+ `refs/` directory before a `file://` clone or `source audit` will read it.
+ - **filter-repo reads `#` lines as literals,** so `replace.txt` is written without them.
+ - **The clone takes `--no-tags`,** and filter-repo `--replace-refs delete-no-add --quiet`.
+ - **`--no-source` REMOVES the previous publish.**
+ - **The header nav moves to `md`.**
+- **No `rulesHash` and no literal count in the published manifest.**
+ - The skip key is `homepage/.source-publish.json` (gitignored and dockerignored):
+ `{sourceCommit, mirrorHead, rulesHash, filterRepo}`.
+ - `rulesHash` covers the rules, the literals and `SOURCE_STEP_VERSION`. Bump it whenever the
+ scrub or the audit changes.
+- **After the review (M1): a refusal withdraws the source everywhere it could ship from.**
+ - The step removes the last publish from `homepage/public` (not under `--check`).
+ - `buildHomepage` removes it from `homepage/out`.
+ - `deployHomepage` refuses an `out/` whose source was not audited under today's rules, of
+ today's `main`, or that has no `/source` page at all.
+- **The audit report names a literal by where it was written** (`denylist line 3 (len 5)`) and a
+ hit by object, field and byte offset. It prints no byte of any object.
+- **The review's other fixes:**
+ - a tracked `404.html` is refused;
+ - a BOM, CRLF and a trailing `/` on the home dir are handled, and an empty left side refuses;
+ - a checkout with no git repository builds with the `/source` empty state (and the CLI refuses
+ with the same sentence);
+ - a scratch root inside the checkout or the public dir is refused, and `--keep-scratch` deletes
+ `replace.txt`;
+ - `E2E_EXPECT_SOURCE=1` makes the empty state fail the spec;
+ - a malformed manifest is the empty state, never a crash.
+- **Left:** compressed content is opaque to the byte search (review L6; a known limit in
+ PUBLISH.md).
+- **The baselines moved:** common 2,114 → **2,146**; homepage unit 2 → **7**; homepage e2e 31 →
+ **36**.
+
## Rollout (parent; nothing here edits the primary's tracked files)
Release 11 is merged and NOT rolled out. A production homepage deploy of release 12 also ships
