commit 4456b4352441ec37d6cf4468c9fd76f0dbf414ea
parent dd71de60bd3faad0bb2ed2ebfb7e8c1bd4fa894c
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Mon, 28 Sep 2026 14:25:29 -0400
docs: PUBLISH.md says a refusal withdraws, previews are public, compressed content is opaque; .dockerignore, project.ts, /sites copy
Review L7: .dockerignore leaves homepage/public/source/ and
homepage/.source-publish.json out of every image (a plain `next build` there
would ship a stale mirror) and stops naming create-archives.sh. L13: the
project.ts comment names the mirror; PUBLISH.md's tsconfig line says `public`
AND `out`, and the "new spelling" line says what the implied denial catches
(exact bytes) and what only the denylist does. The /sites Homepage section
gains one paragraph: Build publishes the source and can refuse, a refusal
removes it from homepage/out too, Deploy refuses an unaudited source (a new
<p>; the editor e2e's assertions on that group are substrings of the existing
text). PUBLISH.md, "The source mirror": the runbook point (a Pages preview is
public, every deployment stays reachable at its hash URL until deleted, deny
everything private before any deploy, delete a deployment that leaked), the
withdrawal and the deploy refusal, the no-repository case, `~/.local/bin` on
the editor's PATH, the new report format, and L6 as a known limit. The editor
changelog bullet says the same.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
5 files changed, 90 insertions(+), 21 deletions(-)
diff --git a/.dockerignore b/.dockerignore
@@ -17,9 +17,14 @@ export/public/summaries/
export/public/transcripts/
export/public/transcripts.tar.gz
export/public/transcripts.tar.xz
-# The source snapshot served by the project site — regenerated by
-# create-archives.sh on the host, never needed inside a build container.
+# The published source (`archilyzer source publish`, run by `archilyzer build
+# homepage`): the tarball, the git mirror and the raw tree (~66 MB), and the
+# skip key beside them, which holds the rules hash. All are regenerated on the
+# host; none is baked into an image, where a plain `next build` would ship a
+# copy that today's rules were never applied to.
homepage/public/downloads/
+homepage/public/source/
+homepage/.source-publish.json
# Build staging / caches / per-site outputs — bind-mounted at run time, never baked.
export/.export-index/
diff --git a/PUBLISH.md b/PUBLISH.md
@@ -92,6 +92,13 @@ fits inside Cloudflare's **free tier**.
The homepage carries the project's own source, read-only, as static files:
+> **Before ANY deploy of the homepage — a preview included — the denylist must hold
+> everything private.** A Pages preview is a public publication, and every Pages
+> deployment stays reachable at its own `<hash>.<project>.pages.dev` URL until that
+> deployment is deleted; a preview's branch alias is guessable (the plans that name
+> it ship in the mirror). If a deploy ever carries something private, DELETE that
+> deployment in the Pages dashboard: a newer deploy does not remove the old one.
+
| Path on the site | What |
|---|---|
| `/source/` | The page: the clone command, the mirror head, the private `main` it reflects |
@@ -102,10 +109,32 @@ The homepage carries the project's own source, read-only, as static files:
**`archilyzer source publish`** makes them, and **`archilyzer build homepage` runs it**
between compose and `next build` — so do the editor's /sites homepage jobs and
-`pnpm ops build-homepage`, in the editor's own process and `PATH`. A refusal fails
-the build before `next build`; nothing is deployed with a missing or stale source.
+`pnpm ops build-homepage`, in the editor's own process and `PATH` (that process needs
+`~/.local/bin` on its `PATH` to find a pipx-installed `git filter-repo`; without it the
+step falls back to `pipx run`, which needs the network).
+
+**A refusal withdraws the source, everywhere it could ship from.** It fails the build
+before `next build`, and:
+
+- the step removes the LAST publish from `homepage/public` (manifest first, then the
+ mirror, the tree, the tarball, `snapshot.json` and the skip key) — it was audited
+ under rules that may not be today's, and the refusal is the best evidence that they
+ are not. Any outcome but success does this once the rules are loaded (an audit hit,
+ a limit, a missing tool, a cancel, a crash); `--check` writes nothing, this included;
+- the build removes the last BUILD's copy from `homepage/out` (`out/source`, the
+ tarball, `snapshot.json`);
+- **`deploy homepage` (and /sites → Deploy homepage) refuses** an `out/` holding a
+ source unless the skip key says that publish was made under today's rules and step
+ version, of today's `main`, and is the one in `out/` (its mirror head and its
+ tarball's sha256): "run `archilyzer build homepage` (it re-audits), then deploy".
+ An `out/` with no source (`--no-source`) deploys as before.
+
`build homepage --no-source` (CLI only) removes the previously published source
-instead, because it was audited against the rules of its own day.
+instead, because it was audited against the rules of its own day. A checkout with no
+git repository — the docker runtime, a tarball install — has nothing to mirror: the
+build says `no git repository here; nothing to mirror — the /source page will show its
+empty state`, removes any old publish and goes on; `source publish` run directly there
+exits 1 with the same sentence.
What one publish does:
@@ -124,20 +153,29 @@ What one publish does:
mirror — blobs, commits and tags whole (messages and identity lines), trees by
entry name — then every staged file and path (the tarball decompressed), searched
for every denied literal; then gitleaks over the history, when it is installed.
- **One hit and nothing is written.** The report names a literal only as
- `#n (x…, len L)` and masks every occurrence in its context; it ends with
+ **One hit and nothing is written** (and the last publish is withdrawn, above). The
+ report names a literal only by where it was written — `denylist line 3 (len 5)`,
+ `scrub line 2 lhs (len 11)`, `built-in home rule (len 11)` — never by any of its
+ characters, and a hit only by its object: kind, id, the blob's path in history, the
+ byte offset, and for a commit or tag the field (`author`, `committer`, `tagger`,
+ `message`). It prints no byte from the object, because what sits beside a denied
+ name (a surname, the rest of an address) is as private as the name. It ends with
`add a rule to ~/.config/archilyzer/source-scrub.txt or drop the file from history,
then re-run.`
-5. The tree (`git archive` → `tar -x`, a page per directory; a tracked `index.html`
+5. The tree (`git archive` → `tar -x`, a page per directory; a tracked `index.html`,
+ a tracked `404.html` — Pages would serve it as HTML for every missing path below it —
or a symlink is a refusal) and the tarball (`git archive --format=tar.gz -9`).
6. Limits, then the install: link-safe (`common/bin/_publicFile.ts`), the manifest
removed first and written **last**, so a crash leaves the page's empty state,
never a manifest over a half-copied tree.
-An unchanged `main` with unchanged rules skips (`[source] up to date at …`;
-`--force` rebuilds). `--check` does everything but the install and writes nothing.
-`--keep-scratch` leaves the scratch clone (`ARCHILYZER_SOURCE_SCRATCH`, default the
-OS temp dir) for a look. `archilyzer source audit [<git dir>]` runs the gate alone —
+An unchanged `main` with unchanged rules, step version and filter-repo version skips
+(`[source] up to date at …`; `--force` rebuilds). `--check` does everything but the
+install and writes nothing. `--keep-scratch` leaves the scratch clone
+(`ARCHILYZER_SOURCE_SCRATCH`, default the OS temp dir) for a look, minus
+`replace.txt` (the scrub rules), which is always deleted; a scratch root inside the
+checkout or the public dir is refused, since a kept scratch there could be committed
+or published. `archilyzer source audit [<git dir>]` runs the gate alone —
over the published mirror by default, or over a clone: `archilyzer source audit
<clone>/.git`.
@@ -150,12 +188,28 @@ never committed; keep them mode 600. A missing file is a refusal naming it.
the last `==>`), `literal:…`, `regex:…==>…`, `glob:…==>…`; a line with no `==>` is
replaced by `***REMOVED***`. Lines whose first non-blank character is `#` are
comments (the step drops them — filter-repo itself would treat one as a literal).
- `<your home dir>==>/home/user` is built in and runs first. Rules apply in order.
+ `<your home dir>==>/home/user` is built in and runs first (a trailing `/` on the home
+ dir is dropped). Rules apply in order. An empty left side is a refusal.
- `source-denylist.txt` — one literal per line, `i:` in front for any ASCII case;
- `#` comments. **Every literal scrub rule's left side is denied too**, so a rule that
- stops matching (a new spelling in history) refuses instead of leaking.
-- The rules' hash is the skip key, kept in `homepage/.source-publish.json` — beside
- `public/`, never in it: a published hash of the denylist would confirm a guess at it.
+ `#` comments.
+- In both, a UTF-8 byte-order mark and CRLF line endings are dropped first: either
+ would silently turn the first rule into one that matches nothing.
+- **Every literal scrub rule's left side is denied too** — its exact bytes, so a rule
+ that stopped matching because its text was mistyped refuses instead of leaking. A
+ NEW spelling in history (another case, another path) is caught only by the denylist:
+ deny the name itself, not just the paths it appears in.
+- The rules' hash (with the step's version) is the skip key and the deploy check, kept
+ in `homepage/.source-publish.json` — beside `public/`, never in it: a published hash
+ of the denylist would confirm a guess at it. For the same reason the manifest does
+ not say how many literals there are.
+
+**A known limit: compressed content is opaque.** The gate is a byte search, so it
+cannot see inside a ZIP member, a PNG's compressed text chunks, a PDF stream or a
+woff2 (the tarball is the exception: it is decompressed). And git-filter-repo never
+scrubs a blob with a NUL in its first 8 KiB, so a binary is audited, never scrubbed.
+At release 12 the review decompressed every such blob in the published history (a
+zip, PNGs and icons) and found no denied literal. A private string inside a binary
+must be removed from history by hand.
**Tools.** `git filter-repo` (`pipx install git-filter-repo`; without it the step runs
`pipx run --spec git-filter-repo==2.47.0`, which needs the network on first use, and
@@ -191,7 +245,9 @@ tools.
- **Tailwind and tsc must not see the mirror.** Tailwind v4 scans every file
`.gitignore` does not exclude (a binary pack yields "class names" that break the
stylesheet), and `homepage/tsconfig.json` includes `**/*.ts`: `/homepage/public/source`
- is gitignored and `public` is excluded from the tsconfig. Keep both.
+ is gitignored, and both `public` and `out` (where `next build` copies it) are excluded
+ from the tsconfig. Keep all three. `.dockerignore` leaves the mirror and the skip key
+ out of every image.
## Preview deployments
diff --git a/common/lib/project.ts b/common/lib/project.ts
@@ -34,8 +34,9 @@ export const PROJECT_URL = "https://archilyzer.pages.dev";
export const PROJECT_TAGLINE = "Self-hosted, searchable video-transcript archives.";
-// Where a visitor gets the source. A dated snapshot tarball — there is no
-// public git repository (see homepage/app/downloads).
+// Where a visitor gets the source as a tarball. The canonical public copy is
+// the read-only git mirror on the same site (lib/sourceManifest.ts CLONE_URL,
+// homepage/app/source); this is its no-git alternative.
export const PROJECT_DOWNLOADS_URL = `${PROJECT_URL}/downloads/`;
// The `generator` string stamped into corpus.json and llms.txt, so anything
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,7 +1,7 @@
# Changelog
## [Unreleased]
-- **Building the homepage now publishes the source: a read-only git mirror, its raw tree and a fresh tarball, behind a gate.** `archilyzer build homepage`, the `/sites` Homepage jobs and `pnpm ops build-homepage` run `archilyzer source publish` between compose and `next build`. It makes a fresh clone of the private `main` (the repository itself is never rewritten), rewrites that copy with git-filter-repo using your scrub rules (file contents and commit messages; your home directory becomes `/home/user` without a rule), and publishes it under `homepage/public` for `git clone https://archilyzer.pages.dev/source/archilyzer.git`, beside `/source/tree/` and the Downloads tarball. Before anything is written, every object of the rewritten history and every file about to be published is searched for every string you have denied; **one hit refuses the build**, and its log names the string only as `#n (x…, len L)`. The rules live outside the repo, in `~/.config/archilyzer/source-scrub.txt` and `source-denylist.txt` (`ARCHILYZER_CONFIG_DIR`, `SOURCE_SCRUB_FILE`, `SOURCE_DENYLIST_FILE`); **without them the build refuses**, naming the missing file. Install git-filter-repo once (`pipx install git-filter-repo`) — without it the build fetches it through `pipx run`, which needs the network — and gitleaks if you want its secret scan too. An unchanged `main` with unchanged rules is skipped, so a rebuild costs about 20 seconds only when something moved. `archilyzer source publish --check` audits without writing, `archilyzer source audit <clone>/.git` checks any clone, `archilyzer build homepage --no-source` removes the published source instead, and `archilyzer doctor` reports the tools, the two files (rule counts and permissions, never their contents) and the last publish. `create-archives.sh` is gone. See PUBLISH.md, "The source mirror (homepage)".
+- **Building the homepage now publishes the source: a read-only git mirror, its raw tree and a fresh tarball, behind a gate.** `archilyzer build homepage`, the `/sites` Homepage jobs and `pnpm ops build-homepage` run `archilyzer source publish` between compose and `next build`. It makes a fresh clone of the private `main` (the repository itself is never rewritten), rewrites that copy with git-filter-repo using your scrub rules (file contents and commit messages; your home directory becomes `/home/user` without a rule), and publishes it under `homepage/public` for `git clone https://archilyzer.pages.dev/source/archilyzer.git`, beside `/source/tree/` and the Downloads tarball. Before anything is written, every object of the rewritten history and every file about to be published is searched for every string you have denied; **one hit refuses the build**, and its log names the string only by where you wrote it (`denylist line 3 (len 5)`) and each hit by its object, field and byte offset — never a byte of the object. **A refusal withdraws the source**: the last publish is removed from `homepage/public` and the last build's copy from `homepage/out`, and **Deploy homepage refuses** a build whose source was not audited under today's rules and today's `main` ("run `archilyzer build homepage`, then deploy"). The rules live outside the repo, in `~/.config/archilyzer/source-scrub.txt` and `source-denylist.txt` (`ARCHILYZER_CONFIG_DIR`, `SOURCE_SCRUB_FILE`, `SOURCE_DENYLIST_FILE`); **without them the build refuses**, naming the missing file. **Put everything private in the denylist before any deploy, a preview included**: previews are public, and every deployment stays reachable at its own address until you delete it. Install git-filter-repo once (`pipx install git-filter-repo`; the editor's process needs `~/.local/bin` on its `PATH` to find it) — without it the build fetches it through `pipx run`, which needs the network — and gitleaks if you want its secret scan too. An unchanged `main` with unchanged rules is skipped, so a rebuild costs about 20 seconds only when something moved. A checkout with no git repository (the docker image, a tarball install) builds with the /source page's empty state. `archilyzer source publish --check` audits without writing, `archilyzer source audit <clone>/.git` checks any clone, `archilyzer build homepage --no-source` removes the published source instead, and `archilyzer doctor` reports the tools, the two files (rule counts and permissions, never their contents) and the last publish. `create-archives.sh` is gone. See PUBLISH.md, "The source mirror (homepage)".
- **umtool reads the corpus from its checkout (or `TRANSCRIPTS_DIR`), and the song project's data defaults to `~/.local/share/archilyzer/song`.** If yours is elsewhere, link it there before restarting umtool: `mkdir -p ~/.local/share/archilyzer && ln -s <where the data is> ~/.local/share/archilyzer/song` (the data stays where it is). With no `CHANNELS_DIR`, umtool reads the corpus at `$TRANSCRIPTS_DIR/channels`, else the checkout's own `transcripts/channels`; it used to fall back to an absolute path that existed on one machine only. The song project's videos default to `~/reports/quartering-uh-song/videos`; `SONG_DIR` and `VIDEO_ROOT` still win. The song project's tracked manifests record their paths relative to the song folders, and the twenty one-off `umtool/song/*.sh` run logs, which only ever ran on the machine that wrote them, are gone.
## [0.10.0] - 2026-09-28
diff --git a/editor/app/sites/components/HomepageBuildButtons.tsx b/editor/app/sites/components/HomepageBuildButtons.tsx
@@ -170,6 +170,13 @@ export function HomepageBuildButtons({ project, builtAt }: Props) {
The homepage reads the search index as it stands: run Build index
(under Pool jobs, below) first when its numbers should move.
</p>
+ <p className="text-xs text-muted-foreground">
+ Build homepage also publishes the source mirror (<code>archilyzer
+ source publish</code>) and refuses if a denied string survives the
+ scrub; a refusal removes the last published source, from
+ homepage/out too. Deploy homepage refuses a build whose source was
+ not audited under today’s rules.
+ </p>
{lane && (
<JobLane
key={lane.key}
