commit c2473a1588ae7ce0fa9720ce8af3042093243114
parent b1cd49373928ce73d1294e465c1d7fc800234469
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Wed, 30 Sep 2026 10:18:09 -0400
Merge r15/stagit (release 15 slice SG) — the source's history and diffs on the homepage: stagit renders the audited mirror into /source/git/ (log, refs, one page per commit with its diff, Atom feed; the Files page links into the raw tree), inside the source gate and the deploy check, noindex like the tree, capped at 10,000 commits, cached under ~/.cache/archilyzer; without the binary the publish says so and goes on; doctor reports stagit, its cache and the drive-health timings; reviewed SHIP
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Diffstat:
33 files changed, 3453 insertions(+), 286 deletions(-)
diff --git a/.gitignore b/.gitignore
@@ -145,6 +145,8 @@ yarn-error.log*
/homepage/e2e/.e2e-settings.json*
# the e2e's empty sites directory (no homepage.json; homepage/playwright.config.ts)
/homepage/e2e/.e2e-sites/
+# the e2e's fixture publish for /source/ (homepage/e2e/fixture-source.ts)
+/homepage/e2e/.e2e-source/
# site config
/settings.json
diff --git a/ENVIRONMENT.md b/ENVIRONMENT.md
@@ -43,6 +43,8 @@ The one override surface for where things live and which binary runs. Every one
| `SOURCE_SCRUB_FILE` | `<ARCHILYZER_CONFIG_DIR>/source-scrub.txt` | git-filter-repo `lhs==>rhs` rules applied to file contents AND commit messages when the source mirror is generated (`<home dir>==>/home/user` is built in and runs first). Every rule's left side is also denied. See [PUBLISH.md](PUBLISH.md). | common/lib/paths.ts (getPaths) |
| `SOURCE_DENYLIST_FILE` | `<ARCHILYZER_CONFIG_DIR>/source-denylist.txt` | Literals the published source must never contain, one per line (`i:` = any case). One hit anywhere in the mirror, the tree or the tarball refuses the publish. | common/lib/paths.ts (getPaths) |
| `ARCHILYZER_SOURCE_SCRATCH` | the OS temp dir | Where `source publish` makes its scratch clone and stage (removed afterwards unless `--keep-scratch`). | common/lib/paths.ts (getPaths) |
+| `XDG_CACHE_HOME` | `~/.cache` | The cache root: `source publish` keeps the history pages' render cache in `<it>/archilyzer/source-history/` (about 140 MB; never inside the checkout). | common/lib/paths.ts (getPaths) |
+| `STAGIT_BIN` | `stagit` on PATH, then `~/.local/bin/stagit` | stagit, which renders the source's history pages (`/source/git/`: the log and a page per commit with its diff). Optional: without it the source is published without them. See [PUBLISH.md](PUBLISH.md). | common/lib/paths.ts (getPaths) |
## Runtime
@@ -189,5 +191,6 @@ Read only by a test harness, a fake binary or a test-mode branch. Never set one
| `E2E_RETRIES` | `0` | Retries per shard (`--retries N` wins); 0 keeps a sharded run comparable to a serial one. | scripts/run-sharded-e2e.mjs |
| `E2E_IMAGE` | `yt-dlp-transcript-browser-e2e` | The sharded e2e run's image tag. | scripts/run-sharded-e2e.mjs |
| `E2E_SKIP_BUILD` | off | `1` reuses the sharded e2e image instead of rebuilding it (`--no-build`). | scripts/run-sharded-e2e.mjs |
+| `E2E_SOURCE_PUBLIC_DIR` | unset (the page reads `homepage/public`) | The fixture publish the homepage's e2e dev server reads the `/source/` page from while it holds a manifest; set by `homepage/playwright.config.ts`, written and removed by `homepage/e2e/source-history.spec.ts`, ignored by a production build. | homepage/app/lib/source.ts |
| `E2E_EXPECT_SOURCE` | off (both states pass) | `1` makes the homepage suite's `/source/` specs fail on the empty state; a gate that ran `archilyzer source publish` first sets it. | homepage/e2e/source.spec.ts |
| `E2E_HOMEPAGE_SUMMARY_FILE` | `homepage/public/homepage-summary.json` | The synthetic summary the homepage's e2e dev server reads; set by `homepage/playwright.config.ts`, ignored by a production build. | homepage/app/lib/summary.ts |
diff --git a/PUBLISH.md b/PUBLISH.md
@@ -26,7 +26,7 @@ server-side code, servable by anything:
|---|---|---|---|
| A **site** | The export app over the channels one site selects: search, transcripts, charts, downloads. One corpus can publish several. | `export/out` (docker mode: `export/.export-builds/<siteId>/out`) | `transcripts/sites/<id>/site.json` ([SITE.md](SITE.md)) |
| The **hub** | The export app in hub mode: federated search across the family of sites. | `export/out` | `transcripts/sites/_homepage/homepage.json` |
-| The **homepage** | The project's own site (`homepage/`), with the source mirror, its raw tree and the source tarball. | `homepage/out` | `~/.config/archilyzer/` (the source mirror's two operator files) |
+| The **homepage** | The project's own site (`homepage/`), with the source mirror, its raw tree, its history pages and the source tarball. | `homepage/out` | `~/.config/archilyzer/` (the source mirror's two operator files) |
The hub and the homepage are two different Pages projects: the hub deploys to the
project `homepage.json` names (e.g. `archilyzer-hub`) and is refused `archilyzer`,
@@ -104,6 +104,7 @@ The homepage carries the project's own source, read-only, as static files:
| `/source/` | The page: the clone command, the mirror head, the private `main` it reflects |
| `/source/archilyzer.git/` | A git repository for git's **dumb HTTP** protocol: `git clone https://archilyzer.pages.dev/source/archilyzer.git` |
| `/source/tree/` | Every tracked file of `main`, raw, with a generated `index.html` per directory |
+| `/source/git/` | The history: `log.html`, a page per commit with its diff (`commit/<sha>.html`), `refs.html`, `files.html` (an index into the raw tree), `atom.xml` and `tags.xml` — rendered by **stagit** when it is installed (below) |
| `/downloads/archilyzer-source.tar.gz` + `snapshot.json` | The same tree without history |
| `/source/manifest.json` | What was published, from which private commit, audited how |
@@ -117,7 +118,8 @@ step falls back to `pipx run`, which needs the network).
before `next build`, and:
- the step removes the LAST publish from `homepage/public` (manifest first, then the
- mirror, the tree, the tarball, `snapshot.json` and the skip key) — it was audited
+ mirror, the tree, the history pages, the tarball, `snapshot.json` and the skip key,
+ and the history's render cache) — it was audited
under rules that may not be today's, and the refusal is the best evidence that they
are not. Any outcome but success does this once the rules are loaded (an audit hit,
a limit, a missing tool, a cancel, a crash); `--check` writes nothing, this included;
@@ -127,9 +129,12 @@ before `next build`, and:
source unless the skip key says that publish was made under today's rules and step
version, of today's `main`, by today's gitleaks, and is exactly the one in `out/`:
its mirror head, and a digest over every published file (the mirror, the tree, the
- manifest, the tarball, `snapshot.json` — sorted path, size and sha256, recomputed
- over `out/`), so a mixed or edited `out/` refuses too: "run `archilyzer build
- homepage` (it re-audits), then deploy". An `out/` whose `/source` page shows the empty
+ history pages, the manifest, the tarball, `snapshot.json` — sorted path, size and
+ sha256, recomputed over `out/`), so a mixed or edited `out/` refuses too: "run
+ `archilyzer build homepage` (it re-audits), then deploy". History pages that are
+ not the audited ones (edited, missing, or present when none were published) are
+ named on their own: "homepage/out's history pages (/source/git/) are not the ones
+ that were audited". An `out/` whose `/source` page shows the empty
state (`--no-source`) deploys as before; one with no `/source` page (a refused build)
does not.
@@ -175,7 +180,9 @@ What one publish does:
then re-run.`
5. The tree (`git archive` → `tar -x`, a page per directory; a tracked `index.html`,
a tracked `404.html` — Pages would serve it as HTML for every missing path below it —
- or a symlink is a refusal) and the tarball (`git archive --format=tar.gz -9`).
+ or a symlink is a refusal) and the tarball (`git archive --format=tar.gz -9`). Then
+ the history pages (stagit, below), into the same stage — so the file sweep of step 4
+ reads every one of them too.
6. Limits, then the install: link-safe (`common/bin/_publicFile.ts`), the manifest
removed first and written **last**, so a crash leaves the page's empty state,
never a manifest over a half-copied tree.
@@ -226,13 +233,102 @@ must be removed from history by hand.
**Tools.** `git filter-repo` (`pipx install git-filter-repo`; without it the step runs
`pipx run --spec git-filter-repo==2.47.0`, which needs the network on first use, and
without pipx it refuses with the install line). gitleaks is optional: without it the
-secret scan is skipped with a WARNING and the literal audit still runs.
-`archilyzer doctor` has a "source publish" block: which filter-repo would run,
-gitleaks, the two files (rule counts and modes, never contents) and the last publish.
+secret scan is skipped with a WARNING and the literal audit still runs. stagit is
+optional too (next section). `archilyzer doctor` has a "source publish" block: which
+filter-repo would run, stagit (its path, or not found, and the render cache's path and
+size), gitleaks, the two files (rule
+counts and modes, never contents) and the last publish.
The mirror's ids are deterministic for a given filter-repo version; an upgrade that
changes its rewriting changes every id (readers re-clone). The manifest records both
tools.
+**The history pages (`/source/git/`) are stagit's.** [stagit](https://codemadness.org/stagit.html)
+(C over libgit2) renders the scrubbed mirror as static pages: the log, a page per commit
+with its diffstat and diff, the refs, and two Atom feeds. It is an operator-installed
+tool like git-filter-repo — never vendored, never committed. Install it once (libgit2
+and its headers are the one dependency):
+
+```sh
+git clone git://git.codemadness.org/stagit && make -C stagit && cp stagit/stagit ~/.local/bin/
+```
+
+The step finds it as `STAGIT_BIN`, else `stagit` on `PATH`, else `~/.local/bin/stagit`
+(the editor's process needs `~/.local/bin` on its `PATH` for a pipx `git filter-repo`;
+for stagit the fallback covers it). **Without it the source is published without the
+history**: one line says so and how to install it, the manifest has no `history`
+block, and `/source/` shows no History links. A render that fails, or history pages
+that would break the host's limits, are the same, with a WARNING — never a failed
+build, and the next build tries again (a stagit present and no history never skips).
+What the step does with it:
+
+- It runs after the mirror is built and its objects audited: `stagit -c <cache> -u
+ https://archilyzer.pages.dev/source/git/ <the scrubbed bare clone>` (past the cap,
+ `-l 10000` in place of `-c`, below). The clone is
+ named `archilyzer.git` (stagit names the repository after its directory); its
+ `description` is set to "Archilyzer" and its `url` to the clone URL, for stagit's
+ header. Neither file is published.
+- **Only an allowlist is published:** `log.html`, `files.html`, `refs.html`,
+ `atom.xml`, `tags.xml`, the page of each commit the log lists (`commit/<sha>.html`),
+ and a `style.css` the step writes from
+ `common/styles/tokens.css` (the homepage's two grounds, light and dark; diff
+ insertions in `--success`, deletions in `--destructive`). **stagit's per-file
+ pages (`file/…`) are not**: browsing is the raw tree, so every link into `file/` —
+ the Files index, the README and LICENSE links in the header, a diff's file names —
+ is rewritten to `../tree/<path>`.
+- Every `.html` page (never the feeds) gets exactly two additions: the homepage's
+ pre-paint theme script (the one `ThemeScript` emits, `lib/themeConfig.ts`), so a
+ page opens on the visitor's stored base or the homepage's dark default — without
+ JavaScript, `prefers-color-scheme` decides — and one line at the top, "Archilyzer ·
+ Source", linking to `/source/`. stagit's logo and favicon point at the site's
+ `/icons/icon-32.png`. A link to what is not published keeps its text and loses its
+ `href`: a diff of a file main no longer has (deleted or renamed since — stagit links
+ every diff side), and a commit with no page (past the cap, the oldest page's parent);
+ the raw tree's file list decides. Nothing else is changed; the pages are rewritten byte
+ for byte.
+- **The gate reads every page**: they are in the stage the file sweep reads, so a
+ denied literal in one refuses the publish, named by its path (`file
+ source/git/commit/<sha>.html (contents, byte N)`) and never by its bytes. The pages
+ are rendered from objects the object sweep already read, so a hit there means
+ something stagit or the step added.
+- The manifest's `history` block has the log's href, the commits with a page and the
+ commits in all (`commits`, `total`), the head, the file count and bytes, a sha256
+ over the pages, and stagit's identity (the sha256 of its binary; it has no version
+ flag). The skip key has stagit's identity and the pages' digest.
+- **The cap: the newest 10,000 commits** (`SOURCE_HISTORY_MAX_COMMITS`). Past it, stagit
+ runs with `-l 10000`: its log lists the newest 10,000 and ends "N more commits
+ remaining, fetch the repository", and `/source/` says "the latest 10,000 of M
+ commits". `-l` does not cap the pages — stagit still writes one for every commit —
+ so the step publishes the page of each commit the log lists and no other; and stagit
+ refuses `-c` with `-l`, so past the cap every publish computes 10,000 diffstats
+ (about 5 ms each here) where `-c` computes only the new ones. The 15,000-file drop
+ below stays, as the last resort.
+- **The render cache is `${XDG_CACHE_HOME:-~/.cache}/archilyzer/source-history/`**
+ (about 140 MB at 1,872 commits; made on first use; never inside the checkout or the
+ public dir — the step renders without it there, and says so). stagit keeps a commit
+ page once rendered, and `-c` keeps its log lines, so a publish renders only the new
+ commits. At 1,834 commits, stagit alone took 0.6 s with nothing new against 8 s for
+ every page; the whole history step (with the post-pass and the copy) 2 s against 8 s.
+ **But `-c`'s walk is in commit-date order and stops at the head it rendered last**, so
+ a `--no-ff` merge of commits older than that head leaves them out: the step's count
+ check sees the log come up short and renders every page again (about 8 s), with a line
+ that says so. With this repo's merges of parallel slices that is common, and correct.
+ The cache holds stagit's `-c` file, its output and a key. Its pages are kept only
+ under the same rules, step, filter-repo, stagit and header text, and after a run that
+ finished; its log lines only when they end at an ancestor of today's head. One
+ publish holds it at a time; a lock whose pid is not running, or over an hour old, is
+ stale and replaced (one line). A cache that cannot be made or written (EACCES, EROFS,
+ ENOSPC) is one line and a render without it, never a failed build. `--force` renders
+ every page again, `--check` never touches it (it renders in its own scratch), and a
+ refusal removes it. `archilyzer doctor` ends its stagit line with `cache: <path>,
+ <size>`.
+- **Its size, and the limit ahead.** At 1,872 commits (2026-09-30): 1,878 files,
+ 141.3 MB, the largest page 5.0 MB (stagit prints "Diff is too large, output
+ suppressed" past 1,000 files or 100,000 lines in one commit); the whole publish is
+ 4,396 files, and `homepage/out` 4,576. One file per commit counts against the step's
+ 15,000 (of Pages' 20,000); the cap keeps it at 10,006 at most. Should the rest of the
+ publish ever grow past 5,000 files, the history is dropped with a WARNING, never the
+ publish.
+
**Cloudflare Pages, and the traps it sets.**
- **Never a path segment named `.git`.** wrangler's upload ignore list drops `**/.git`
@@ -247,7 +343,8 @@ tools.
`homepage/public/_headers` sets `/source/tree/*` to `text/plain; charset=utf-8` with
`nosniff` and `noindex`, then puts back `text/html` for the directory pages and the
real types of the few binaries (`.ttf`, `.m4a`, `.zip`, `.onnx`, `.svg` — the SVG
- under a `default-src 'none'` CSP). **Every matching rule applies, and a header a
+ under a `default-src 'none'` CSP). `/source/git/*`, the history pages, is `noindex`
+ too (ruled), and keeps its own types. **Every matching rule applies, and a header a
later rule sets again is APPENDED** (`text/plain; charset=utf-8, text/html;
charset=utf-8`), so each of those overrides first detaches the tree's type with
`! Content-Type`. Only the ROOT `_headers` is read; the tree's own
diff --git a/common/bin/archilyzer.ts b/common/bin/archilyzer.ts
@@ -153,7 +153,7 @@ export const COMMANDS: Command[] = [
{
path: ["source", "publish"],
usage:
- "[--force] [--check] [--keep-scratch] the scrubbed git mirror, raw tree and tarball into homepage/public, behind the denied-literal gate (--check: audit and count, write nothing)",
+ "[--force] [--check] [--keep-scratch] the scrubbed git mirror, raw tree, history pages (stagit, when installed) and tarball into homepage/public, behind the denied-literal gate (--check: audit and count, write nothing)",
flags: { force: "boolean", check: "boolean", "keep-scratch": "boolean" },
run: async ({ flags }) => {
const { publishSource } = await import("../publish/source");
diff --git a/common/bin/doctor.test.ts b/common/bin/doctor.test.ts
@@ -58,6 +58,9 @@ function checkout(): { root: string; bin: string; paths: Paths } {
parakeetCliBin: "parakeet-cli",
sourceScrubFile: path.join(root, ".config", "source-scrub.txt"),
sourceDenylistFile: path.join(root, ".config", "source-denylist.txt"),
+ // Outside the checkout, as the XDG cache is: the tests that compare the
+ // checkout's tree before and after never see it.
+ sourceHistoryCacheDir: path.join(TMP, `${path.basename(root)}-cache`, "archilyzer", "source-history"),
} as unknown as Paths;
return { root, bin, paths };
}
@@ -244,12 +247,20 @@ test("the source publish block: the tools, the operator files by count and mode
sourceTools: async () => tools,
});
// A clone that never publishes: notes, not warnings.
- let r = await collectDoctorReport(deps({ filterRepo: null, gitleaks: null }));
+ let r = await collectDoctorReport(deps({ filterRepo: null, gitleaks: null, stagit: null }));
assert.equal(r.ok, true, renderDoctorReport(r));
- for (const id of ["filter-repo", "gitleaks", "scrub rules", "denylist", "published"]) {
+ for (const id of ["filter-repo", "gitleaks", "stagit", "scrub rules", "denylist", "published"]) {
assert.equal(status(r, id), "info", id);
}
assert.match(r.checks.find((x) => x.id === "filter-repo")!.detail, /install: `pipx install git-filter-repo`/);
+ // stagit, beside git-filter-repo: where it is, or not found and how to
+ // install it; then the render cache, where it is and how big.
+ assert.equal(
+ r.checks.find((x) => x.id === "stagit")!.detail,
+ `not found (PATH, ~/.local/bin) — the source is published without its history pages (/source/git/); install it once: git clone git://git.codemadness.org/stagit && make -C stagit && cp stagit/stagit ~/.local/bin/; cache: ${c.paths.sourceHistoryCacheDir}, none yet`,
+ );
+ const order = r.checks.filter((x) => x.section === "source publish").map((x) => x.id);
+ assert.equal(order.indexOf("stagit"), order.indexOf("filter-repo") + 1, "beside git-filter-repo");
// The operator's files exist (one readable by others), pipx only, a publish.
mkdirSync(path.dirname(c.paths.sourceScrubFile), { recursive: true });
@@ -267,11 +278,16 @@ test("the source publish block: the tools, the operator files by count and mode
audit: { objects: 1, commits: 1, gitleaks: "clean" }, tools: {},
}));
const before = tree(c.root);
- r = await collectDoctorReport(deps({ filterRepo: { via: "pipx", version: "1.15.0" }, gitleaks: { version: "8.28.0" } }));
+ r = await collectDoctorReport(deps({ filterRepo: { via: "pipx", version: "1.15.0" }, gitleaks: { version: "8.28.0" }, stagit: "/opt/stagit/stagit" }));
assert.equal(r.ok, true, renderDoctorReport(r));
assert.equal(status(r, "filter-repo"), "warn");
assert.match(r.checks.find((x) => x.id === "filter-repo")!.detail, /pipx run --spec git-filter-repo==2\.47\.0/);
assert.equal(status(r, "gitleaks"), "ok");
+ assert.equal(status(r, "stagit"), "ok");
+ mkdirSync(path.join(c.paths.sourceHistoryCacheDir, "out"), { recursive: true });
+ writeFileSync(path.join(c.paths.sourceHistoryCacheDir, "out", "log.html"), Buffer.alloc(3 * 1024 * 1024));
+ const withCache = await collectDoctorReport(deps({ filterRepo: null, gitleaks: null, stagit: "/opt/stagit/stagit" }));
+ assert.equal(withCache.checks.find((x) => x.id === "stagit")!.detail, `/opt/stagit/stagit; cache: ${c.paths.sourceHistoryCacheDir}, 3.0 MB`);
assert.equal(status(r, "scrub rules"), "ok");
assert.match(r.checks.find((x) => x.id === "scrub rules")!.detail, /\(2 rules, mode 600\)$/);
assert.equal(status(r, "denylist"), "warn");
@@ -282,8 +298,14 @@ test("the source publish block: the tools, the operator files by count and mode
assert.ok(!/PLANTED/.test(text), "the doctor never prints an operator file's contents");
assert.deepEqual(tree(c.root), before);
- r = await collectDoctorReport(deps({ filterRepo: { via: "git", version: "a40bce548d2c" }, gitleaks: null }));
+ r = await collectDoctorReport(deps({ filterRepo: { via: "git", version: "a40bce548d2c" }, gitleaks: null, stagit: null }));
assert.equal(status(r, "filter-repo"), "ok");
+
+ // A STAGIT_BIN that names nothing: a warning, never a failure.
+ r = await collectDoctorReport({ ...deps({ filterRepo: null, gitleaks: null, stagit: null }), env: { PATH: c.bin, STAGIT_BIN: "/nowhere/stagit" } });
+ assert.equal(status(r, "stagit"), "warn");
+ assert.match(r.checks.find((x) => x.id === "stagit")!.detail, /^not found: STAGIT_BIN=\/nowhere\/stagit names no executable/);
+ assert.equal(r.ok, true);
});
// A stored icon the checker refuses is named by file and label with its
@@ -309,7 +331,7 @@ test("social icons: a refused stored icon is named by file and label, not its ma
portInUse: async () => false,
portBlock: async () => null,
umtoolTools: async () => null,
- sourceTools: async () => ({ filterRepo: null, gitleaks: null }),
+ sourceTools: async () => ({ filterRepo: null, gitleaks: null, stagit: null }),
});
const line = report.checks.find((x) => x.section === "social icons")!;
assert.equal(line.status, "warn");
@@ -319,3 +341,25 @@ test("social icons: a refused stored icon is named by file and label, not its ma
assert.ok(!line.detail.includes(secret), "no markup");
assert.ok(report.ok, "a warning, not a failure");
});
+
+// Release 15 slice DT left doctor's line to SG: the drive-health timings are
+// applied before any drive is inspected (as the index and stats bins apply
+// them), and the corpus block says which are in force.
+test("the drive-health timings: applied from settings.storage.health before the corpus is inspected, and printed", async () => {
+ const { healthTimings, applyHealthTimings } = await import("../lib/storageHealth");
+ const c = checkout();
+ mkdirSync(c.paths.channelsDir, { recursive: true });
+ let r = await run(c);
+ assert.equal(
+ r.checks.find((x) => x.id === "drive health")!.detail,
+ "a read may take 3 s, a check every 15 s (3 s each), a stall clears on a clean check twice in a row, 4 reads in flight per drive — the defaults",
+ );
+ writeFileSync(c.paths.settingsFile, JSON.stringify({ storage: { health: { budgetMs: 5000, clearAfterCleanPasses: 3 } } }));
+ r = await run(c);
+ assert.equal(
+ r.checks.find((x) => x.id === "drive health")!.detail,
+ "a read may take 5 s, a check every 15 s (3 s each), a stall clears on a clean check 3 times in a row, 4 reads in flight per drive — settings.storage.health",
+ );
+ assert.equal(healthTimings().budgetMs, 5000, "applied to this process");
+ applyHealthTimings();
+});
diff --git a/common/bin/doctor.ts b/common/bin/doctor.ts
@@ -74,10 +74,12 @@ export type DoctorDeps = {
sourceTools?: () => Promise<SourceTools>;
};
-// Which git-filter-repo `archilyzer source publish` would run, and gitleaks.
+// Which git-filter-repo `archilyzer source publish` would run, gitleaks, and
+// the stagit that renders the history pages (its path, or null).
export type SourceTools = {
filterRepo: { via: "git"; version: string } | { via: "pipx"; version: string } | null;
gitleaks: { version: string } | null;
+ stagit: string | null;
};
const MIN_NODE = [20, 9, 0] as const; // next 16's engines field
@@ -109,6 +111,22 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
? "no path or binary overrides set (ENVIRONMENT.md lists them)"
: overrides.map((v) => `${v.name}=${env[v.name]}`).join(" "));
+ // The effective settings, read the way every process reads them (defaults
+ // when the file is absent). Read-only: the reader never writes. Read before
+ // the corpus, for the drive-health timings (release 15 slice DT): applied
+ // here, as the index and stats bins apply them, a CLI process's drive
+ // inspects run on the machine's timings instead of racing the defaults.
+ const { settingsFromFile } = await import("../lib/settings");
+ let settings: ReturnType<typeof settingsFromFile> | null = null;
+ let settingsError: Error | null = null;
+ try {
+ settings = settingsFromFile(paths.settingsFile);
+ } catch (err) {
+ settingsError = err as Error;
+ }
+ const { applyHealthTimings } = await import("../lib/storageHealth");
+ const timings = applyHealthTimings(settings?.storage.health);
+
// ── corpus ───────────────────────────────────────────────────────────────
const C = "corpus";
let channelSlugs: string[] = [];
@@ -133,6 +151,12 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
} else if (channelSlugs.length > 0) {
add(C, "media", "ok", "every channel's data/ is reachable");
}
+ const { secondsText, clearRuleText } = await import("../lib/storageHealthTimings");
+ const tuned = Object.keys(settings?.storage.health ?? {}).length > 0;
+ add(C, "drive health", "info",
+ `a read may take ${secondsText(timings.budgetMs)}, a check every ${secondsText(timings.passIntervalMs)} ` +
+ `(${secondsText(timings.probeTimeoutMs)} each), a stall clears on a clean check ${clearRuleText(timings.clearAfterCleanPasses)}, ` +
+ `${timings.inFlightPerLocation} reads in flight per drive — ${tuned ? "settings.storage.health" : "the defaults"}`);
const index = statOrNull(paths.lmdbPath);
if (index) {
add(C, "index", "ok", `${paths.lmdbPath} (stat only; built ${index.mtime.toISOString().slice(0, 16).replace("T", " ")})`);
@@ -161,15 +185,7 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
`${paths.settingsFile} is not a JSON object${parsed instanceof Error ? ` (${parsed.message})` : ""} — every process silently reads it as the defaults`);
}
}
- // The effective settings, read the way every process reads them (defaults
- // when the file is absent). Read-only: the reader never writes.
- const { settingsFromFile } = await import("../lib/settings");
- let settings: ReturnType<typeof settingsFromFile> | null = null;
- try {
- settings = settingsFromFile(paths.settingsFile);
- } catch (err) {
- add(S, "schema", "fail", `settings do not load: ${(err as Error).message}`);
- }
+ if (settingsError) add(S, "schema", "fail", `settings do not load: ${settingsError.message}`);
// ── social icons ─────────────────────────────────────────────────────────
// Every stored social link's icon — settings.json, each site.json,
@@ -291,7 +307,7 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
// are never printed.
const SP = "source publish";
const src = await import("../publish/source");
- const tools = await (deps.sourceTools ?? (() => probeSourceTools(env)))();
+ const tools = await (deps.sourceTools ?? (() => probeSourceTools(env, paths)))();
const scrubFile = paths.sourceScrubFile;
const denylistFile = paths.sourceDenylistFile;
const intends = [scrubFile, denylistFile].some((f) => f && existsSync(f));
@@ -304,6 +320,21 @@ export async function collectDoctorReport(deps: DoctorDeps): Promise<DoctorRepor
add(SP, "filter-repo", intends ? "warn" : "info",
`neither git-filter-repo nor pipx — \`archilyzer build homepage\` refuses; install: \`${src.FILTER_REPO_INSTALL}\``);
}
+ // stagit renders the history pages (/source/git/); without it the publish
+ // goes on without them. A STAGIT_BIN that names nothing is a warning. The
+ // line ends with the render cache: where it is and how big (stat'd only).
+ const hist = await import("../publish/sourceHistory");
+ const cacheDir = paths.sourceHistoryCacheDir;
+ const cache = cacheDir ? `; cache: ${cacheDir}, ${await dirSizeText(cacheDir)}` : "";
+ if (tools.stagit) {
+ add(SP, "stagit", "ok", `${tools.stagit}${cache}`);
+ } else if (env.STAGIT_BIN) {
+ add(SP, "stagit", "warn",
+ `not found: STAGIT_BIN=${env.STAGIT_BIN} names no executable — the source is published without its history pages (/source/git/)${cache}`);
+ } else {
+ add(SP, "stagit", "info",
+ `not found (PATH, ~/.local/bin) — the source is published without its history pages (/source/git/); install it once: ${hist.STAGIT_INSTALL}${cache}`);
+ }
add(SP, "gitleaks", tools.gitleaks ? "ok" : "info",
tools.gitleaks ? `gitleaks ${tools.gitleaks.version}` : "absent — the gate skips the secret scan with a WARNING (the literal audit still runs)");
for (const [id, file, unit] of [
@@ -518,8 +549,9 @@ async function worktreePortBlock(
// What `source publish` would run, by version flags only: `git filter-repo
// --version` answering 0 is an installed filter-repo; otherwise pipx's own
-// version (never `pipx run`, which would download). gitleaks likewise.
-async function probeSourceTools(env: NodeJS.ProcessEnv): Promise<SourceTools> {
+// version (never `pipx run`, which would download). gitleaks likewise. stagit
+// has no version flag: it is looked up the way the publish looks it up.
+async function probeSourceTools(env: NodeJS.ProcessEnv, paths: Paths): Promise<SourceTools> {
const version = async (bin: string, args: string[]): Promise<string | null> => {
try {
const { stdout, stderr } = await execFileP(bin, args, { env, timeout: 10_000 });
@@ -531,10 +563,27 @@ async function probeSourceTools(env: NodeJS.ProcessEnv): Promise<SourceTools> {
const git = await version("git", ["filter-repo", "--version"]);
const pipx = git === null ? await version("pipx", ["--version"]) : null;
const leaks = await version("gitleaks", ["version"]);
+ const { resolveStagit } = await import("../publish/sourceHistory");
return {
filterRepo: git !== null ? { via: "git", version: git } : pipx !== null ? { via: "pipx", version: pipx } : null,
gitleaks: leaks !== null ? { version: leaks } : null,
+ stagit: resolveStagit(paths.stagitBin ?? "stagit", env),
+ };
+}
+
+// A directory's size, by stat alone ("none yet" when it is not there).
+async function dirSizeText(dir: string): Promise<string> {
+ if (!existsSync(dir)) return "none yet";
+ let bytes = 0;
+ const walk = async (d: string): Promise<void> => {
+ for (const ent of await readdir(d, { withFileTypes: true }).catch(() => [])) {
+ const p = path.join(d, ent.name);
+ if (ent.isDirectory()) await walk(p);
+ else if (ent.isFile()) bytes += statOrNull(p)?.size ?? 0;
+ }
};
+ await walk(dir);
+ return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
}
// In use = something accepts a TCP connection on 127.0.0.1. Never binds.
diff --git a/common/components/themeConfig.ts b/common/components/themeConfig.ts
@@ -1,207 +1,4 @@
-// Shared, dependency-light theme constants, types and the pre-paint script's
-// source. Used by the pre-paint ThemeScript (server), the runtime ThemeProvider
-// (client), the toggle, the unit tests and the homepage e2e (REQUIRED_TOKENS).
-// Keys are namespaced under the existing `ytdlp-tb:*` localStorage convention.
-//
-// THE THEME (plans/brand-and-themes.md; two grounds since release 14, T1):
-// • the reader's BASE — the light or the dark ground, or "system", which
-// follows the OS between them. `html[data-base]` selects one of the two
-// token blocks in common/styles/tokens.css; `.dark` is on <html> iff the
-// resolved base is dark, so Tailwind's `dark:` utilities keep working. A
-// stored RETIRED_BASE (the third ground, retired in release 14) is read
-// as, and rewritten to, "light".
-// • the ACCENT — one of the seven named accents in lib/brand.ts, or a
-// site's own hex. It is the SITE's (a server-rendered `html[data-accent]`),
-// never the reader's: a reader's stored pick from before is ignored, and
-// left in storage.
-//
-// PURE: no React, no DOM at import time. lib/brand.ts is pure too.
-
-import { BASE_GROUNDS, type AccentId } from "../lib/brand";
-
-// The reader's base ("light" | "dark" | "system"; RETIRED_BASE, from before
-// release 14, is migrated to "light").
-export const BASE_KEY = "ytdlp-tb:base";
-// A reader's accent pick from before release 14. Nothing reads it and nothing
-// deletes it: each app paints its own accent.
-export const ACCENT_KEY = "ytdlp-tb:accent";
-// The retired theme-family and light/dark-mode keys. Read once by the
-// migration (migrateLegacy, and the same table inside the pre-paint script),
-// then deleted.
-export const LEGACY_THEME_KEY = "ytdlp-tb:theme";
-export const LEGACY_MODE_KEY = "ytdlp-tb:mode";
-
-// The two grounds a base resolves to, and the reader's choice (which adds
-// "system").
-export type ResolvedBase = "light" | "dark";
-export type ThemeBase = ResolvedBase | "system";
-
-// The third ground, retired in release 14: a stored base of this value is
-// light. The one place its name is spelled.
-export const RETIRED_BASE = "sepia";
-
-// What `html[data-accent]` can carry: a named accent, or "custom" — a site
-// whose site.json accent is its own hex (the inline `--accent-custom-*` vars).
-export type ThemeAccent = AccentId | "custom";
-
-// The toggle's cycle order, and every base a reader can store.
-export const THEME_BASES: ReadonlyArray<{ id: ThemeBase; label: string }> = [
- { id: "system", label: "System" },
- { id: "light", label: "Light" },
- { id: "dark", label: "Dark" },
-];
-
-export function isThemeBase(v: unknown): v is ThemeBase {
- return v === "system" || v === "light" || v === "dark";
-}
-
-// What a stored base means: a base, RETIRED_BASE → "light", anything else null
-// (the app's default applies).
-export function storedBase(v: unknown): ThemeBase | null {
- if (v === RETIRED_BASE) return "light";
- return isThemeBase(v) ? v : null;
-}
-
-// ThemeToggle's cycle, THEME_BASES in order: system → light → dark → system.
-export function nextBase(b: ThemeBase): ThemeBase {
- const i = THEME_BASES.findIndex((t) => t.id === b);
- return THEME_BASES[(i + 1) % THEME_BASES.length].id;
-}
-
-// A choice resolved against the OS preference.
-export function resolveBase(b: ThemeBase, systemDark: boolean): ResolvedBase {
- return b === "system" ? (systemDark ? "dark" : "light") : b;
-}
-
-// Every colour token a base block in tokens.css must declare. The failure this
-// guards is silent: a base that omits a token inherits the light block's value
-// (`:root` always matches), so a half-declared palette looks "a bit off" rather
-// than broken, and only on the base nobody checked. The unit test
-// (themeTokens.test.ts) parses tokens.css against this list; the homepage e2e
-// reads every one off the computed style of each base.
-export const REQUIRED_TOKENS = [
- "--background",
- "--foreground",
- "--card",
- "--card-foreground",
- "--popover",
- "--popover-foreground",
- "--primary",
- "--primary-foreground",
- "--secondary",
- "--secondary-foreground",
- "--muted",
- "--muted-foreground",
- "--accent",
- "--accent-foreground",
- "--destructive",
- "--destructive-foreground",
- "--destructive-soft",
- "--border",
- "--border-strong",
- "--input",
- "--ring",
- "--surface",
- "--faint",
- "--panel",
- "--panel-2",
- "--success",
- "--success-foreground",
- "--success-soft",
- "--warning",
- "--warning-foreground",
- "--warning-soft",
- "--info",
- "--info-foreground",
- "--info-soft",
- "--brand",
- "--brand-strong",
- "--brand-soft",
- "--brand-ink",
- "--state-gone",
- "--state-gone-soft",
- "--chart-1",
- "--chart-2",
- "--chart-3",
- "--chart-4",
- "--chart-5",
- "--chart-6",
- "--chart-surface",
- "--chart-grid",
- "--chart-axis",
- "--chart-tooltip-bg",
-] as const;
-
-// The retired keys → a base, once. After it runs, both legacy keys are deleted
-// by the caller, whatever it returned.
-//
-// stored mode result
-// light light (the old "archive" paper theme too: it was the
-// third ground until that was retired)
-// dark dark
-// system system
-// absent/other null: nothing is stored and the app's default base applies
-//
-// The old theme FAMILY is otherwise dropped: every family retired, and the
-// accent is the site's.
-export function migrateLegacy({
- mode,
-}: {
- theme: string | null;
- mode: string | null;
-}): ThemeBase | null {
- if (mode === "light") return "light";
- if (mode === "dark") return "dark";
- if (mode === "system") return "system";
- return null;
-}
-
-// The pre-paint script, as the string ThemeScript.tsx inlines. It runs
-// synchronously before first paint, in this order:
-// 1. when no base is stored yet, migrate the legacy keys (migrateLegacy's
-// table); then delete both legacy keys;
-// 2. a stored RETIRED_BASE becomes "light", in storage too (once);
-// 3. validate the stored base, falling back to `defaultBase`;
-// 4. set `data-base` to the RESOLVED ground and toggle `.dark`;
-// 5. point every `meta[name=theme-color]` at the resolved ground;
-// 6. LAST: `data-theme-ready="1"`, the e2e no-flash marker, so it is present
-// only once every attribute above is set. e2e's reloads resolve on
-// navigation commit, possibly before this head script has run, and wait
-// for the marker instead of racing.
-// The accent is not read: `html[data-accent]` is the server's (the site's
-// own), and a stored ACCENT_KEY stays where it is, unread.
-// Storage may throw (privacy modes): each storage touch is guarded, so a
-// failure still paints the default base and still sets the marker.
-// themeConfig.test.ts runs this string in node:vm over the whole matrix.
-export function buildThemeScript({
- defaultBase = "system",
-}: { defaultBase?: ThemeBase } = {}): string {
- const fallback: ThemeBase = isThemeBase(defaultBase) ? defaultBase : "system";
- const q = JSON.stringify;
- return (
- "(function(){try{" +
- "var d=document.documentElement,b=null,s;" +
- "try{s=window.localStorage;" +
- `b=s.getItem(${q(BASE_KEY)});` +
- `var lt=s.getItem(${q(LEGACY_THEME_KEY)}),lm=s.getItem(${q(LEGACY_MODE_KEY)});` +
- "if(lt!==null||lm!==null){" +
- "if(b===null){" +
- "var m=lm==='light'?'light':lm==='dark'?'dark':lm==='system'?'system':null;" +
- `if(m){s.setItem(${q(BASE_KEY)},m);b=m;}` +
- "}" +
- `s.removeItem(${q(LEGACY_THEME_KEY)});s.removeItem(${q(LEGACY_MODE_KEY)});` +
- "}" +
- `if(b===${q(RETIRED_BASE)}){b='light';s.setItem(${q(BASE_KEY)},'light');}` +
- "}catch(e){}" +
- `if(b===${q(RETIRED_BASE)})b='light';` +
- `if(b!=='light'&&b!=='dark'&&b!=='system')b=${q(fallback)};` +
- "var r=b;" +
- "if(b==='system'){r='light';try{if(window.matchMedia('(prefers-color-scheme: dark)').matches)r='dark';}catch(e){}}" +
- "d.setAttribute('data-base',r);" +
- "d.classList.toggle('dark',r==='dark');" +
- `var g=${q(BASE_GROUNDS)}[r];` +
- "try{var ms=document.querySelectorAll('meta[name=\"theme-color\"]');for(var i=0;i<ms.length;i++)ms[i].setAttribute('content',g);}catch(e){}" +
- "d.setAttribute('data-theme-ready','1');" +
- "}catch(e){}})();"
- );
-}
+// The theme's constants, types and pre-paint script live in lib/themeConfig.ts
+// (the source publish reads the script too, and the publish layer may not
+// import components/). Re-exported here for the UI and its tests.
+export * from "../lib/themeConfig";
diff --git a/common/lib/envVars.ts b/common/lib/envVars.ts
@@ -84,6 +84,8 @@ const DECLARED: EnvVarDecl[] = [
paths("SOURCE_SCRUB_FILE", "`<ARCHILYZER_CONFIG_DIR>/source-scrub.txt`", "git-filter-repo `lhs==>rhs` rules applied to file contents AND commit messages when the source mirror is generated (`<home dir>==>/home/user` is built in and runs first). Every rule's left side is also denied. See [PUBLISH.md](PUBLISH.md)."),
paths("SOURCE_DENYLIST_FILE", "`<ARCHILYZER_CONFIG_DIR>/source-denylist.txt`", "Literals the published source must never contain, one per line (`i:` = any case). One hit anywhere in the mirror, the tree or the tarball refuses the publish."),
paths("ARCHILYZER_SOURCE_SCRATCH", "the OS temp dir", "Where `source publish` makes its scratch clone and stage (removed afterwards unless `--keep-scratch`)."),
+ paths("XDG_CACHE_HOME", "`~/.cache`", "The cache root: `source publish` keeps the history pages' render cache in `<it>/archilyzer/source-history/` (about 140 MB; never inside the checkout)."),
+ paths("STAGIT_BIN", "`stagit` on PATH, then `~/.local/bin/stagit`", "stagit, which renders the source's history pages (`/source/git/`: the log and a page per commit with its diff). Optional: without it the source is published without them. See [PUBLISH.md](PUBLISH.md)."),
// ── runtime ────────────────────────────────────────────────────────────
{ name: "WORKER_TOKEN", audience: "runtime", default: "unset (both surfaces off)", readBy: "common/lib/workerToken.ts, scripts/archilyzer-ops.mjs, mcp/src/fetchClip.ts", doc: "Bearer token for the remote-worker API and for `/api/ops/*` (`pnpm ops`, the MCP's `fetch_clip`). Set the same value on both ends." },
@@ -188,6 +190,7 @@ const DECLARED: EnvVarDecl[] = [
{ name: "E2E_RETRIES", audience: "test", default: "`0`", readBy: "scripts/run-sharded-e2e.mjs", doc: "Retries per shard (`--retries N` wins); 0 keeps a sharded run comparable to a serial one." },
{ name: "E2E_IMAGE", audience: "test", default: "`yt-dlp-transcript-browser-e2e`", readBy: "scripts/run-sharded-e2e.mjs", doc: "The sharded e2e run's image tag." },
{ name: "E2E_SKIP_BUILD", audience: "test", default: "off", readBy: "scripts/run-sharded-e2e.mjs", doc: "`1` reuses the sharded e2e image instead of rebuilding it (`--no-build`)." },
+ { name: "E2E_SOURCE_PUBLIC_DIR", audience: "test", default: "unset (the page reads `homepage/public`)", readBy: "homepage/app/lib/source.ts", doc: "The fixture publish the homepage's e2e dev server reads the `/source/` page from while it holds a manifest; set by `homepage/playwright.config.ts`, written and removed by `homepage/e2e/source-history.spec.ts`, ignored by a production build." },
{ name: "E2E_EXPECT_SOURCE", audience: "test", default: "off (both states pass)", readBy: "homepage/e2e/source.spec.ts", doc: "`1` makes the homepage suite's `/source/` specs fail on the empty state; a gate that ran `archilyzer source publish` first sets it." },
{ name: "E2E_HOMEPAGE_SUMMARY_FILE", audience: "test", default: "`homepage/public/homepage-summary.json`", readBy: "homepage/app/lib/summary.ts", doc: "The synthetic summary the homepage's e2e dev server reads; set by `homepage/playwright.config.ts`, ignored by a production build." },
];
diff --git a/common/lib/paths.ts b/common/lib/paths.ts
@@ -168,6 +168,16 @@ export type Paths = {
sourceDenylistFile: string;
// Where `source publish` makes its scratch clone (removed afterwards).
sourceScratchDir: string;
+ // The history pages' render cache (publish/sourceHistory.ts): the XDG
+ // cache dir's archilyzer/source-history — ~/.cache unless XDG_CACHE_HOME
+ // says otherwise (an empty one is unset, as the XDG spec has it). Never
+ // inside the checkout or the public dir: the step renders without it there.
+ sourceHistoryCacheDir: string;
+ // stagit, which renders the source's history pages (/source/git/). An
+ // operator-installed tool, never vendored: `stagit` on PATH, then
+ // ~/.local/bin/stagit (publish/sourceHistory.ts resolveStagit). Without it
+ // the publish goes on without the history pages.
+ stagitBin: string;
};
let cached: Paths | null = null;
@@ -296,6 +306,12 @@ export function getPaths(): Paths {
process.env.SOURCE_DENYLIST_FILE ??
path.join(configDir, "source-denylist.txt"),
sourceScratchDir: process.env.ARCHILYZER_SOURCE_SCRATCH ?? os.tmpdir(),
+ sourceHistoryCacheDir: path.join(
+ process.env.XDG_CACHE_HOME || path.join(os.homedir(), ".cache"),
+ "archilyzer",
+ "source-history",
+ ),
+ stagitBin: process.env.STAGIT_BIN ?? "stagit",
};
return cached;
}
diff --git a/common/lib/sourceManifest.test.ts b/common/lib/sourceManifest.test.ts
@@ -48,3 +48,42 @@ test("a wrong version, a bad id or sha, or any number the page reads that is not
for (const [what, make] of broken) assert.equal(parseSourceManifest(make(good())), null, what);
for (const junk of [null, 1, "x", [], {}]) assert.equal(parseSourceManifest(junk), null, JSON.stringify(junk));
});
+
+// Release 15 slice SG: the history block is optional — absent is a manifest
+// without History links — and all-or-nothing when present.
+const history = () => ({
+ href: "/source/git/log.html",
+ commits: 3,
+ total: 3,
+ head: "2".repeat(40),
+ files: 9,
+ bytes: 1234,
+ sha256: "4".repeat(64),
+ tool: "stagit (sha256 0123456789ab)",
+});
+
+test("a manifest with a history block parses; without one it is the same manifest", () => {
+ const withHistory = { ...good(), history: history() };
+ assert.deepEqual(parseSourceManifest(withHistory), withHistory);
+ const capped = { ...good(), history: { ...history(), total: 12_345 } };
+ assert.deepEqual(parseSourceManifest(capped), capped, "the latest 3 of 12,345");
+ assert.equal(parseSourceManifest(good())?.history, undefined);
+});
+
+test("a history block that is present but malformed makes the manifest untrusted", () => {
+ const broken: Array<[string, (h: ReturnType<typeof history>) => unknown]> = [
+ ["null", () => null],
+ ["commits a string", (h) => ({ ...h, commits: "3" })],
+ ["no total", (h) => ({ ...h, total: undefined })],
+ ["fewer in all than with a page", (h) => ({ ...h, total: 2 })],
+ ["files negative", (h) => ({ ...h, files: -1 })],
+ ["bytes missing", (h) => ({ ...h, bytes: undefined })],
+ ["short head", (h) => ({ ...h, head: "abc" })],
+ ["sha256 not hex", (h) => ({ ...h, sha256: "z".repeat(64) })],
+ ["no href", (h) => ({ ...h, href: undefined })],
+ ["tool a number", (h) => ({ ...h, tool: 1 })],
+ ];
+ for (const [what, make] of broken) {
+ assert.equal(parseSourceManifest({ ...good(), history: make(history()) }), null, what);
+ }
+});
diff --git a/common/lib/sourceManifest.ts b/common/lib/sourceManifest.ts
@@ -25,6 +25,16 @@ export const CLONE_URL = `${PROJECT_URL}/source/${MIRROR_DIR}`;
/** The raw tree's generated index. */
export const TREE_HREF = "/source/tree/";
+/**
+ * The history pages' directory under /source/: stagit's rendering of the
+ * mirror (publish/sourceHistory.ts) — the log, a page per commit with its
+ * diff, the refs, the files index (into the raw tree) and two Atom feeds.
+ */
+export const HISTORY_DIR = "git";
+export const HISTORY_LOG_HREF = `/source/${HISTORY_DIR}/log.html`;
+export const HISTORY_REFS_HREF = `/source/${HISTORY_DIR}/refs.html`;
+export const HISTORY_ATOM_HREF = `/source/${HISTORY_DIR}/atom.xml`;
+
/** The manifest's public path. */
export const SOURCE_MANIFEST_HREF = "/source/manifest.json";
@@ -69,6 +79,30 @@ export type SourceManifest = {
};
// The tools that made it (a filter-repo upgrade may change mirrorHead).
tools: { git: string; filterRepo: string };
+ // The history pages, when stagit rendered them (absent: no stagit on the
+ // publishing machine, or a render that failed — the page then shows no
+ // History links).
+ history?: SourceHistory;
+};
+
+export type SourceHistory = {
+ // The log page (HISTORY_LOG_HREF).
+ href: string;
+ // How many commits have a page (the newest ones: the log lists exactly
+ // these), of `total`, every commit of the mirror's main. Equal until main
+ // passes the cap (publish/sourceHistory.ts SOURCE_HISTORY_MAX_COMMITS).
+ // `head` is the one the log starts at (the manifest's mirrorHead).
+ commits: number;
+ total: number;
+ head: string;
+ // Every file under /source/git/ (the pages, the two feeds, style.css): how
+ // many, their bytes, and one sha256 over them — each file's path, size and
+ // sha256, in sorted path order — which the deploy check recomputes over out/.
+ files: number;
+ bytes: number;
+ sha256: string;
+ // The renderer: "stagit (sha256 <first 12 of its binary's>)".
+ tool: string;
};
const HEX40 = /^[0-9a-f]{40}$/;
@@ -101,5 +135,16 @@ export function parseSourceManifest(value: unknown): SourceManifest | null {
}
if (!obj(m.tree) || !num(m.tree.files) || !num(m.tree.dirs) || !num(m.tree.bytes)) return null;
if (!obj(m.audit)) return null;
+ // Optional, and all-or-nothing: a history block that is present but
+ // malformed makes the whole manifest untrusted, like any other bad field.
+ if (m.history !== undefined) {
+ const h = m.history as Partial<SourceHistory> | null;
+ if (!obj(h)) return null;
+ if (!num(h.commits) || !num(h.total) || !num(h.files) || !num(h.bytes)) return null;
+ if ((h.total as number) < (h.commits as number)) return null;
+ if (typeof h.head !== "string" || !HEX40.test(h.head)) return null;
+ if (typeof h.sha256 !== "string" || !HEX64.test(h.sha256)) return null;
+ if (typeof h.href !== "string" || typeof h.tool !== "string") return null;
+ }
return m as SourceManifest;
}
diff --git a/common/lib/themeConfig.ts b/common/lib/themeConfig.ts
@@ -0,0 +1,217 @@
+// Shared, dependency-light theme constants, types and the pre-paint script's
+// source. Used by the pre-paint ThemeScript (server), the runtime ThemeProvider
+// (client), the toggle, the unit tests and the homepage e2e (REQUIRED_TOKENS) —
+// and by the source publish, which puts the homepage's pre-paint script on
+// every history page (publish/sourceHistory.ts). That last reader is why this
+// module lives in lib/: the publish layer may not import components/, which
+// re-exports it (components/themeConfig.ts) for the UI's importers.
+// Keys are namespaced under the existing `ytdlp-tb:*` localStorage convention.
+//
+// THE THEME (plans/brand-and-themes.md; two grounds since release 14, T1):
+// • the reader's BASE — the light or the dark ground, or "system", which
+// follows the OS between them. `html[data-base]` selects one of the two
+// token blocks in common/styles/tokens.css; `.dark` is on <html> iff the
+// resolved base is dark, so Tailwind's `dark:` utilities keep working. A
+// stored RETIRED_BASE (the third ground, retired in release 14) is read
+// as, and rewritten to, "light".
+// • the ACCENT — one of the seven named accents in lib/brand.ts, or a
+// site's own hex. It is the SITE's (a server-rendered `html[data-accent]`),
+// never the reader's: a reader's stored pick from before is ignored, and
+// left in storage.
+//
+// PURE: no React, no DOM at import time. lib/brand.ts is pure too.
+
+import { BASE_GROUNDS, type AccentId } from "./brand";
+
+// The reader's base ("light" | "dark" | "system"; RETIRED_BASE, from before
+// release 14, is migrated to "light").
+export const BASE_KEY = "ytdlp-tb:base";
+// A reader's accent pick from before release 14. Nothing reads it and nothing
+// deletes it: each app paints its own accent.
+export const ACCENT_KEY = "ytdlp-tb:accent";
+// The retired theme-family and light/dark-mode keys. Read once by the
+// migration (migrateLegacy, and the same table inside the pre-paint script),
+// then deleted.
+export const LEGACY_THEME_KEY = "ytdlp-tb:theme";
+export const LEGACY_MODE_KEY = "ytdlp-tb:mode";
+
+// The two grounds a base resolves to, and the reader's choice (which adds
+// "system").
+export type ResolvedBase = "light" | "dark";
+export type ThemeBase = ResolvedBase | "system";
+
+// The third ground, retired in release 14: a stored base of this value is
+// light. The one place its name is spelled.
+export const RETIRED_BASE = "sepia";
+
+// What `html[data-accent]` can carry: a named accent, or "custom" — a site
+// whose site.json accent is its own hex (the inline `--accent-custom-*` vars).
+export type ThemeAccent = AccentId | "custom";
+
+// The toggle's cycle order, and every base a reader can store.
+export const THEME_BASES: ReadonlyArray<{ id: ThemeBase; label: string }> = [
+ { id: "system", label: "System" },
+ { id: "light", label: "Light" },
+ { id: "dark", label: "Dark" },
+];
+
+// The project site's base for a reader who has stored none: it opens on the
+// dark ground (homepage/app/layout.tsx passes it to ThemeScript and
+// ThemeProvider). The source's history pages run the same pre-paint script
+// with it, so they open where the homepage does.
+export const HOMEPAGE_DEFAULT_BASE: ThemeBase = "dark";
+
+export function isThemeBase(v: unknown): v is ThemeBase {
+ return v === "system" || v === "light" || v === "dark";
+}
+
+// What a stored base means: a base, RETIRED_BASE → "light", anything else null
+// (the app's default applies).
+export function storedBase(v: unknown): ThemeBase | null {
+ if (v === RETIRED_BASE) return "light";
+ return isThemeBase(v) ? v : null;
+}
+
+// ThemeToggle's cycle, THEME_BASES in order: system → light → dark → system.
+export function nextBase(b: ThemeBase): ThemeBase {
+ const i = THEME_BASES.findIndex((t) => t.id === b);
+ return THEME_BASES[(i + 1) % THEME_BASES.length].id;
+}
+
+// A choice resolved against the OS preference.
+export function resolveBase(b: ThemeBase, systemDark: boolean): ResolvedBase {
+ return b === "system" ? (systemDark ? "dark" : "light") : b;
+}
+
+// Every colour token a base block in tokens.css must declare. The failure this
+// guards is silent: a base that omits a token inherits the light block's value
+// (`:root` always matches), so a half-declared palette looks "a bit off" rather
+// than broken, and only on the base nobody checked. The unit test
+// (themeTokens.test.ts) parses tokens.css against this list; the homepage e2e
+// reads every one off the computed style of each base.
+export const REQUIRED_TOKENS = [
+ "--background",
+ "--foreground",
+ "--card",
+ "--card-foreground",
+ "--popover",
+ "--popover-foreground",
+ "--primary",
+ "--primary-foreground",
+ "--secondary",
+ "--secondary-foreground",
+ "--muted",
+ "--muted-foreground",
+ "--accent",
+ "--accent-foreground",
+ "--destructive",
+ "--destructive-foreground",
+ "--destructive-soft",
+ "--border",
+ "--border-strong",
+ "--input",
+ "--ring",
+ "--surface",
+ "--faint",
+ "--panel",
+ "--panel-2",
+ "--success",
+ "--success-foreground",
+ "--success-soft",
+ "--warning",
+ "--warning-foreground",
+ "--warning-soft",
+ "--info",
+ "--info-foreground",
+ "--info-soft",
+ "--brand",
+ "--brand-strong",
+ "--brand-soft",
+ "--brand-ink",
+ "--state-gone",
+ "--state-gone-soft",
+ "--chart-1",
+ "--chart-2",
+ "--chart-3",
+ "--chart-4",
+ "--chart-5",
+ "--chart-6",
+ "--chart-surface",
+ "--chart-grid",
+ "--chart-axis",
+ "--chart-tooltip-bg",
+] as const;
+
+// The retired keys → a base, once. After it runs, both legacy keys are deleted
+// by the caller, whatever it returned.
+//
+// stored mode result
+// light light (the old "archive" paper theme too: it was the
+// third ground until that was retired)
+// dark dark
+// system system
+// absent/other null: nothing is stored and the app's default base applies
+//
+// The old theme FAMILY is otherwise dropped: every family retired, and the
+// accent is the site's.
+export function migrateLegacy({
+ mode,
+}: {
+ theme: string | null;
+ mode: string | null;
+}): ThemeBase | null {
+ if (mode === "light") return "light";
+ if (mode === "dark") return "dark";
+ if (mode === "system") return "system";
+ return null;
+}
+
+// The pre-paint script, as the string ThemeScript.tsx inlines. It runs
+// synchronously before first paint, in this order:
+// 1. when no base is stored yet, migrate the legacy keys (migrateLegacy's
+// table); then delete both legacy keys;
+// 2. a stored RETIRED_BASE becomes "light", in storage too (once);
+// 3. validate the stored base, falling back to `defaultBase`;
+// 4. set `data-base` to the RESOLVED ground and toggle `.dark`;
+// 5. point every `meta[name=theme-color]` at the resolved ground;
+// 6. LAST: `data-theme-ready="1"`, the e2e no-flash marker, so it is present
+// only once every attribute above is set. e2e's reloads resolve on
+// navigation commit, possibly before this head script has run, and wait
+// for the marker instead of racing.
+// The accent is not read: `html[data-accent]` is the server's (the site's
+// own), and a stored ACCENT_KEY stays where it is, unread.
+// Storage may throw (privacy modes): each storage touch is guarded, so a
+// failure still paints the default base and still sets the marker.
+// themeConfig.test.ts runs this string in node:vm over the whole matrix.
+export function buildThemeScript({
+ defaultBase = "system",
+}: { defaultBase?: ThemeBase } = {}): string {
+ const fallback: ThemeBase = isThemeBase(defaultBase) ? defaultBase : "system";
+ const q = JSON.stringify;
+ return (
+ "(function(){try{" +
+ "var d=document.documentElement,b=null,s;" +
+ "try{s=window.localStorage;" +
+ `b=s.getItem(${q(BASE_KEY)});` +
+ `var lt=s.getItem(${q(LEGACY_THEME_KEY)}),lm=s.getItem(${q(LEGACY_MODE_KEY)});` +
+ "if(lt!==null||lm!==null){" +
+ "if(b===null){" +
+ "var m=lm==='light'?'light':lm==='dark'?'dark':lm==='system'?'system':null;" +
+ `if(m){s.setItem(${q(BASE_KEY)},m);b=m;}` +
+ "}" +
+ `s.removeItem(${q(LEGACY_THEME_KEY)});s.removeItem(${q(LEGACY_MODE_KEY)});` +
+ "}" +
+ `if(b===${q(RETIRED_BASE)}){b='light';s.setItem(${q(BASE_KEY)},'light');}` +
+ "}catch(e){}" +
+ `if(b===${q(RETIRED_BASE)})b='light';` +
+ `if(b!=='light'&&b!=='dark'&&b!=='system')b=${q(fallback)};` +
+ "var r=b;" +
+ "if(b==='system'){r='light';try{if(window.matchMedia('(prefers-color-scheme: dark)').matches)r='dark';}catch(e){}}" +
+ "d.setAttribute('data-base',r);" +
+ "d.classList.toggle('dark',r==='dark');" +
+ `var g=${q(BASE_GROUNDS)}[r];` +
+ "try{var ms=document.querySelectorAll('meta[name=\"theme-color\"]');for(var i=0;i<ms.length;i++)ms[i].setAttribute('content',g);}catch(e){}" +
+ "d.setAttribute('data-theme-ready','1');" +
+ "}catch(e){}})();"
+ );
+}
diff --git a/common/publish/__fixtures__/fakeStagit.ts b/common/publish/__fixtures__/fakeStagit.ts
@@ -0,0 +1,82 @@
+import { chmodSync, mkdirSync, writeFileSync } from "node:fs";
+import path from "node:path";
+
+// A stand-in for stagit, for the source step's tests (sourceHistory.test.ts,
+// source.test.ts; the real one is tested where it is installed). It writes
+// what stagit writes, where stagit writes it (its CWD), the way stagit.c does:
+// - `-c <file>`: walks from HEAD down to the commit the file names, writes a
+// log line and a page for each NEW commit, appends the file's old lines,
+// and rewrites the file (the head, then every line);
+// - `-l <n>`: a log line for the newest n and "<m> more commits remaining,
+// fetch the repository"; a page for EVERY commit that has none (-l does
+// not cap the pages);
+// - neither: a line and a page for every commit;
+// - `-c` with `-l`: stagit's usage error;
+// - and every run: files.html, refs.html, the two feeds (atom.xml carries
+// the -u base), a per-file page, a leftover in the CWD. A commit page is
+// never rewritten once it exists.
+// Each call is appended to `log` as `cache=… limit=… base=… repo=<dir name>`.
+// `exit` fails at once (with a stderr line); `extra` goes into every commit
+// page; `pad` bytes are added to the head's page.
+export function writeFakeStagit(
+ file: string,
+ log: string,
+ o: { exit?: number; extra?: string; pad?: number } = {},
+): string {
+ mkdirSync(path.dirname(file), { recursive: true });
+ writeFileSync(
+ file,
+ `#!/bin/sh
+cache=""; base=""; repo=""; limit=""
+while [ $# -gt 0 ]; do
+ case "$1" in
+ -c) cache="$2"; shift 2;;
+ -l) limit="$2"; shift 2;;
+ -u) base="$2"; shift 2;;
+ *) repo="$1"; shift;;
+ esac
+done
+echo "cache=$cache limit=$limit base=$base repo=$(basename "$repo")" >> '${log}'
+${o.exit ? `echo 'stagit: something broke' >&2; exit ${o.exit}` : ""}
+if [ -n "$cache" ] && [ -n "$limit" ]; then echo 'usage: stagit [-c cachefile | -l commits] [-u baseurl] repodir' >&2; exit 1; fi
+mkdir -p commit file
+tip=$(git --git-dir "$repo" rev-parse HEAD)
+last=""
+if [ -n "$cache" ] && [ -f "$cache" ]; then last=$(head -n 1 "$cache"); fi
+: > lines.tmp
+n=0; rem=0
+for c in $(git --git-dir "$repo" rev-list HEAD); do
+ [ "$c" = "$last" ] && break
+ if [ -n "$limit" ] && [ "$n" -ge "$limit" ]; then
+ rem=$((rem + 1))
+ [ -f "commit/$c.html" ] && continue
+ else
+ n=$((n + 1))
+ echo "<tr><td><a href=\\"commit/$c.html\\">$c</a></td></tr>" >> lines.tmp
+ fi
+ [ -f "commit/$c.html" ] || printf '<html>\\n<head>\\n</head>\\n<body>\\n<a href="../file/README.md.html">README</a> ${o.extra ?? ""}\\n</body>\\n</html>\\n' > "commit/$c.html"
+done
+if [ -n "$cache" ]; then
+ if [ -n "$last" ]; then tail -n +2 "$cache" >> lines.tmp; fi
+ { echo "$tip"; cat lines.tmp; } > "$cache"
+fi
+if [ "$rem" -gt 0 ]; then echo "<tr><td></td><td colspan=\\"5\\">$rem more commits remaining, fetch the repository</td></tr>" >> lines.tmp; fi
+{
+ printf '<html>\\n<head>\\n</head>\\n<body>\\n<span class="desc">%s</span> %s <a href="file/README.md.html">README</a> <img src="logo.png" />\\n<table id="log">\\n' "$(cat "$repo/description")" "$(cat "$repo/url")"
+ cat lines.tmp
+ printf '</table>\\n</body>\\n</html>\\n'
+} > log.html
+rm -f lines.tmp
+printf '<html>\\n<head>\\n</head>\\n<body>\\n<a href="file/app/%%5Bslug%%5D/page.tsx.html">x</a>\\n</body>\\n</html>\\n' > files.html
+printf '<html>\\n<head>\\n</head>\\n<body>\\n</body>\\n</html>\\n' > refs.html
+printf '<feed>%s file/README.md.html</feed>\\n' "$base" > atom.xml
+printf '<feed/>\\n' > tags.xml
+printf 'x\\n' > file/README.md.html
+touch cache.XXXXleftover
+${o.pad ? `head -c ${o.pad} /dev/zero >> "commit/$tip.html"` : ""}
+exit 0
+`,
+ );
+ chmodSync(file, 0o755);
+ return file;
+}
diff --git a/common/publish/source.test.ts b/common/publish/source.test.ts
@@ -18,6 +18,7 @@ import {
} from "node:fs";
import os from "node:os";
import path from "node:path";
+import { fileURLToPath } from "node:url";
import type { Paths } from "../lib/paths";
import { CLONE_URL, MIRROR_DIR, TARBALL_HREF, TREE_HREF } from "../lib/sourceManifest";
import {
@@ -33,12 +34,16 @@ import {
publishSource,
publishedSourceProblem,
gitleaksIdentity,
+ historyCacheFor,
+ historyDigest,
resolveFilterRepo,
rulesHashOf,
scratchRootProblem,
sourceDigest,
type SourcePublishOpts,
} from "./source";
+import { writeFakeStagit } from "./__fixtures__/fakeStagit";
+import { HISTORY_BACK_LINK } from "./sourceHistory";
// Run with:
// pnpm --filter yt-dlp-transcript-common test
@@ -125,6 +130,8 @@ function opts(repo: string, files: { scrubFile: string; denylistFile: string },
publicDir: path.join(dir("site"), "public"),
scratchRoot: path.join(TMP, "scratch"),
gitleaks: null,
+ // No history pages unless a test gives a stagit (the machine may have one).
+ stagit: null,
homeDir: HOME,
onLog: (l) => logs.push(l),
now: () => new Date("2026-09-28T12:00:00.000Z"),
@@ -249,6 +256,8 @@ test("skip: an unchanged main with unchanged rules, tools and files does nothing
filterRepo: "false (given)",
gitleaks: "skipped",
contentDigest: await sourceDigest(pub),
+ stagit: "absent",
+ history: null,
};
writeFileSync(state, JSON.stringify({ ...key, ...tweak }));
};
@@ -268,6 +277,7 @@ test("skip: an unchanged main with unchanged rules, tools and files does nothing
["another gitleaks", { gitleaks: "gitleaks 8.0 (abc)" }],
["other files", { contentDigest: "0".repeat(64) }],
["other rules", { rulesHash: "0".repeat(64) }],
+ ["another stagit", { stagit: "stagit (sha256 0123456789ab)" }],
] as Array<[string, Record<string, string>]>) {
await plant(tweak);
assert.equal(await run(), 1, `${what} must not skip`);
@@ -299,7 +309,7 @@ test("the rules hash moves with the step's version (review R2-L3), and loadSourc
const files = operatorFiles("a==>b\n", "");
const rules = await loadSourceRules({ ...files, homeDir: "/" });
assert.equal(rules.rulesHash, rulesHashOf(["a==>b"], rules.literals, SOURCE_STEP_VERSION));
- assert.ok(SOURCE_STEP_VERSION >= 3);
+ assert.ok(SOURCE_STEP_VERSION >= 4, "release 15 slice SG: the history pages");
});
test("gitleaksIdentity: skipped, absent, or the version line with the binary's sha256 (review R2-L4)", async () => {
@@ -551,7 +561,7 @@ test("a denied literal no rule removes: refused, nothing written, the report nev
assert.ok(kept, report);
// The log tildifies with the real home dir, so a TMPDIR under it prints "~/…".
const keptDir = kept[1].replace(/^~(?=\/|$)/, os.homedir());
- assert.ok(existsSync(path.join(keptDir, "bare")));
+ assert.ok(existsSync(path.join(keptDir, MIRROR_DIR)), "the clone is named as the mirror is (stagit names it after its directory)");
assert.ok(!existsSync(path.join(keptDir, "replace.txt")));
rmSync(keptDir, { recursive: true, force: true });
});
@@ -603,3 +613,337 @@ test("--no-source's clear: the manifest, mirror, tree, tarball and skip key go;
assert.equal(readFileSync(path.join(elsewhere, "snapshot.json"), "utf8"), "the other checkout's");
assert.match(logs.join("\n"), /previously published source .* was removed/);
});
+
+// ── the history pages (release 15 slice SG) ─────────────────────────────────
+//
+// Over a fake stagit that writes what stagit writes (sourceHistory.test.ts
+// runs the real one), and with filter-repo replaced by `true` — no rewrite,
+// so these run on any machine; the history does not care what the rewrite did.
+
+const TOKENS_FILE = path.join(path.dirname(fileURLToPath(import.meta.url)), "..", "styles", "tokens.css");
+
+// The fake (__fixtures__/fakeStagit.ts) writes what stagit writes, with -c
+// and -l as stagit.c has them.
+function fakeStagit(log: string, o: { exit?: number; extra?: string; pad?: number } = {}): string {
+ return writeFakeStagit(path.join(dir("fake-stagit"), "stagit"), log, o);
+}
+
+// A render cache of its own, where XDG_CACHE_HOME would put it.
+const cacheDirIn = (root: string) => path.join(root, "archilyzer", "source-history");
+
+const stagitCalls = (log: string) => (existsSync(log) ? readFileSync(log, "utf8").trim().split("\n") : []);
+
+test("history: published with the source — the allowlist at /source/git/, the manifest's block, the skip key, incremental, and the deploy check names it", async () => {
+ const repo = sourceRepo();
+ const files = operatorFiles("", "");
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const cacheDir = cacheDirIn(dir("cache-home"));
+ const o = opts(repo, files, logs, { filterRepo: ["true"], stagit: fakeStagit(log), tokensFile: TOKENS_FILE, historyCacheDir: cacheDir });
+ const pub = o.publicDir!;
+ const site = path.dirname(pub);
+
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ let text = logs.join("\n");
+ assert.match(text, /\[source\] history: stagit \(sha256 [0-9a-f]{12}\) — 3 commits, 3 pages rendered; 9 files, [\d.]+ MB, the largest git\/\S+ [\d.]+ MB/);
+ assert.match(text, /\[source\] published main .* tree \d+ dirs, history 3 commits in 9 files\)/);
+ assert.deepEqual(stagitCalls(log), [
+ `cache=${path.join(cacheDir, "stagit.cache")} limit= base=https://archilyzer.pages.dev/source/git/ repo=${MIRROR_DIR}`,
+ ]);
+
+ const manifest = JSON.parse(readFileSync(path.join(pub, "source", "manifest.json"), "utf8"));
+ assert.deepEqual(Object.keys(manifest.history).sort(), ["bytes", "commits", "files", "head", "href", "sha256", "tool", "total"]);
+ assert.equal(manifest.history.href, "/source/git/log.html");
+ assert.equal(manifest.history.commits, 3);
+ assert.equal(manifest.history.total, 3);
+ assert.equal(manifest.history.head, manifest.mirrorHead);
+ assert.equal(manifest.history.files, 9);
+ assert.equal(manifest.history.sha256, await historyDigest(pub));
+ assert.match(manifest.history.tool, /^stagit \(sha256 [0-9a-f]{12}\)$/);
+ assert.ok(manifest.files >= 9, "the manifest's count includes the history");
+
+ const git = path.join(pub, "source", "git");
+ assert.deepEqual(readdirSync(git).sort(), ["atom.xml", "commit", "files.html", "log.html", "refs.html", "style.css", "tags.xml"]);
+ const logHtml = readFileSync(path.join(git, "log.html"), "utf8");
+ assert.ok(logHtml.includes(`<span class="desc">Archilyzer</span> ${CLONE_URL} `), "stagit's header: the name and the clone URL");
+ assert.ok(logHtml.includes(HISTORY_BACK_LINK));
+ assert.match(logHtml, /<script>\(function\(\)\{try\{var d=document\.documentElement/);
+ assert.match(logHtml, /href="\.\.\/tree\/README\.md"/);
+ // The feed is copied as it is: not even its file/ text is rewritten.
+ assert.equal(readFileSync(path.join(git, "atom.xml"), "utf8"), "<feed>https://archilyzer.pages.dev/source/git/ file/README.md.html</feed>\n");
+ assert.match(readFileSync(path.join(git, "style.css"), "utf8"), /html\[data-base="dark"\] \{/);
+ // The description and url files stagit read are never published.
+ for (const f of ["description", "url"]) assert.ok(!existsSync(path.join(pub, "source", MIRROR_DIR, f)), f);
+
+ const stateFile = path.join(site, ".source-publish.json");
+ const state = JSON.parse(readFileSync(stateFile, "utf8"));
+ assert.equal(state.stagit, manifest.history.tool);
+ assert.deepEqual(state.history, { files: 9, digest: manifest.history.sha256 });
+
+ // Unchanged: skipped, stagit not run.
+ logs.length = 0;
+ assert.equal(await publishSource(o), 0);
+ assert.match(logs.join("\n"), /up to date at/);
+ assert.equal(stagitCalls(log).length, 1);
+
+ // The deploy check: this publish in out/ deploys; a history page edited,
+ // or the history missing, is named.
+ const out = path.join(dir("out"), "out");
+ mkdirSync(path.join(out, "source"), { recursive: true });
+ writeFileSync(path.join(out, "source", "index.html"), "<p>the page</p>");
+ cpSync(path.join(pub, "source"), path.join(out, "source"), { recursive: true });
+ cpSync(path.join(pub, "downloads"), path.join(out, "downloads"), { recursive: true });
+ const check = { ...files, homeDir: HOME, publicDir: pub, sourceRepo: path.join(repo, ".git"), gitleaks: null, stagit: null };
+ assert.equal(await publishedSourceProblem(o.paths!, out, check), null);
+ const outLog = path.join(out, "source", "git", "log.html");
+ const was = readFileSync(outLog);
+ writeFileSync(outLog, Buffer.concat([was, Buffer.from(" ")]));
+ assert.match((await publishedSourceProblem(o.paths!, out, check))!, /history pages \(\/source\/git\/\) are not the ones that were audited — run `archilyzer build homepage`/);
+ rmSync(path.join(out, "source", "git"), { recursive: true });
+ assert.match((await publishedSourceProblem(o.paths!, out, check))!, /history pages \(\/source\/git\/\) are not the ones that were audited/);
+ cpSync(path.join(pub, "source", "git"), path.join(out, "source", "git"), { recursive: true });
+ assert.equal(await publishedSourceProblem(o.paths!, out, check), null, "restored, it deploys");
+
+ // A new commit on main: only its page is rendered, the rest from the cache.
+ writeFileSync(path.join(repo, "later.txt"), "x\n");
+ gitIn(repo, "add", "-A");
+ gitIn(repo, "commit", "-q", "-m", "later");
+ logs.length = 0;
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /— 4 commits, 1 page rendered \(the rest from the cache\); 10 files/);
+ // --force renders every page again.
+ logs.length = 0;
+ assert.equal(await publishSource({ ...o, force: true }), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /— 4 commits, 4 pages rendered; 10 files/);
+ // --check writes nothing — the cache included — and still renders (and
+ // sweeps) the pages in its own scratch.
+ const cacheBefore = readFileSync(path.join(cacheDir, "key.json"), "utf8");
+ logs.length = 0;
+ assert.equal(await publishSource({ ...o, check: true }), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /— 4 commits, 4 pages rendered;/);
+ assert.equal(stagitCalls(log).at(-1), `cache= limit= base=https://archilyzer.pages.dev/source/git/ repo=${MIRROR_DIR}`);
+ assert.equal(readFileSync(path.join(cacheDir, "key.json"), "utf8"), cacheBefore);
+});
+
+test("history: without stagit — one line with the install, nothing at /source/git/, no block in the manifest; installing it re-publishes", async () => {
+ const repo = sourceRepo();
+ const files = operatorFiles("", "");
+ const logs: string[] = [];
+ const o = opts(repo, files, logs, { filterRepo: ["true"], stagit: null, tokensFile: TOKENS_FILE, scratchRoot: dir("scratch-root") });
+ const pub = o.publicDir!;
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ const said = logs.filter((l) => /stagit/.test(l));
+ assert.deepEqual(said, [
+ "[source] stagit not found (not on PATH, not in ~/.local/bin) — publishing without the history pages (/source/git/); install it once: git clone git://git.codemadness.org/stagit && make -C stagit && cp stagit/stagit ~/.local/bin/",
+ ]);
+ assert.match(logs.join("\n"), /\[source\] published main .*, no history\)/);
+ const manifest = JSON.parse(readFileSync(path.join(pub, "source", "manifest.json"), "utf8"));
+ assert.ok(!("history" in manifest));
+ assert.ok(!existsSync(path.join(pub, "source", "git")));
+ const state = JSON.parse(readFileSync(path.join(path.dirname(pub), ".source-publish.json"), "utf8"));
+ assert.equal(state.stagit, "absent");
+ assert.equal(state.history, null);
+
+ logs.length = 0;
+ assert.equal(await publishSource(o), 0);
+ assert.match(logs.join("\n"), /up to date at/, "no stagit, nothing changed: skipped");
+
+ // stagit installed since: not skipped, and the history is published.
+ logs.length = 0;
+ const log = path.join(dir("stagit-log"), "calls");
+ assert.equal(await publishSource({ ...o, stagit: fakeStagit(log) }), 0, logs.join("\n"));
+ assert.ok(!/up to date/.test(logs.join("\n")));
+ assert.equal(JSON.parse(readFileSync(path.join(pub, "source", "manifest.json"), "utf8")).history.commits, 3);
+ assert.ok(existsSync(path.join(pub, "source", "git", "log.html")));
+});
+
+test("history: a denied literal in a history page refuses, names the page and never the literal, and withdraws the publish and the render cache", async () => {
+ const repo = sourceRepo();
+ const planted = "plantedinhistory";
+ const files = operatorFiles("", "");
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const cacheDir = cacheDirIn(dir("cache-home"));
+ // The literal is only ever in what stagit writes, never in the repository:
+ // what this proves is that the file gate reads source/git/**.
+ const o = opts(repo, files, logs, { filterRepo: ["true"], stagit: fakeStagit(log, { extra: `said ${planted} here` }), tokensFile: TOKENS_FILE, historyCacheDir: cacheDir });
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ assert.ok(existsSync(path.join(o.publicDir!, "source", "git", "log.html")));
+ assert.ok(existsSync(cacheDir));
+
+ // Deny it: the rules change, so every page is rendered again — and swept.
+ writeFileSync(files.denylistFile, `${planted}\n`);
+ logs.length = 0;
+ assert.equal(await publishSource(o), 1);
+ const report = logs.join("\n");
+ assert.match(report, /AUDIT REFUSED: 3 hits in \d+ objects \(3 commits\), \d+ staged files/);
+ assert.match(report, /denylist line 1 \(len 16\): 3 in files/);
+ assert.match(report, /\[source\] {3}file source\/git\/commit\/[0-9a-f]{40}\.html \(contents, byte \d+\): denylist line 1 \(len 16\)/);
+ assert.ok(!report.includes(planted), report);
+ assert.ok(!report.includes("said"), "no byte from beside the hit");
+ assert.match(report, /the previous publish was WITHDRAWN \(mirror, tree, history, tarball\)/);
+ assert.ok(!existsSync(path.join(o.publicDir!, "source")), "public/source, the history with it, is withdrawn");
+ assert.ok(!existsSync(cacheDir), "the render cache is removed too");
+});
+
+test("history: a stagit that fails, or a page over the host's limit, is a WARNING; the rest is published, and the next build tries again", async () => {
+ const repo = sourceRepo();
+ const files = operatorFiles("", "");
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const o = opts(repo, files, logs, { filterRepo: ["true"], stagit: fakeStagit(log, { exit: 3 }), tokensFile: TOKENS_FILE, scratchRoot: dir("scratch-root") });
+ const pub = o.publicDir!;
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /\[source\] WARNING: the history pages were not rendered: stagit exited 3: stagit: something broke — publishing without the history pages \(\/source\/git\/\)/);
+ assert.ok(existsSync(path.join(pub, "source", MIRROR_DIR, "info", "refs")), "the mirror is published");
+ assert.ok(!existsSync(path.join(pub, "source", "git")));
+ const state = JSON.parse(readFileSync(path.join(path.dirname(pub), ".source-publish.json"), "utf8"));
+ assert.match(state.stagit, /^stagit \(sha256/);
+ assert.equal(state.history, null);
+ logs.length = 0;
+ assert.equal(await publishSource(o), 0);
+ assert.ok(!/up to date/.test(logs.join("\n")), "a stagit present and no history: the next build tries again");
+ assert.equal(stagitCalls(log).length, 2);
+
+ // One page over the step's 24 MiB: the history is dropped, not the publish.
+ logs.length = 0;
+ assert.equal(await publishSource({ ...o, stagit: fakeStagit(log, { pad: MAX_FILE_BYTES }) }), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /\[source\] WARNING: git\/commit\/[0-9a-f]{40}\.html is 24\.0 MiB, over the step's limit of 24\.0 MiB \(Pages allows 25 MiB per file\) — publishing without the history pages/);
+ assert.ok(!existsSync(path.join(pub, "source", "git")));
+ assert.ok(!("history" in JSON.parse(readFileSync(path.join(pub, "source", "manifest.json"), "utf8"))));
+
+ // Unreadable design tokens: the same.
+ logs.length = 0;
+ assert.equal(await publishSource({ ...o, stagit: fakeStagit(log), tokensFile: path.join(dir("none"), "tokens.css") }), 0);
+ assert.match(logs.join("\n"), /\[source\] WARNING: the design tokens \(.*tokens\.css\) could not be read — publishing without the history pages/);
+});
+
+test("history: past the cap, the latest N of M — the manifest's commits and total, the log line, the summary; the pages of the newest only", async () => {
+ const repo = sourceRepo();
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const o = opts(repo, operatorFiles("", ""), logs, {
+ filterRepo: ["true"],
+ stagit: fakeStagit(log),
+ tokensFile: TOKENS_FILE,
+ historyCacheDir: cacheDirIn(dir("cache-home")),
+ maxHistoryCommits: 2,
+ });
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ const text = logs.join("\n");
+ assert.equal(stagitCalls(log).at(-1), `cache= limit=2 base=https://archilyzer.pages.dev/source/git/ repo=${MIRROR_DIR}`, "-l, and no -c");
+ assert.match(text, /\[source\] history: stagit \(sha256 [0-9a-f]{12}\) — the latest 2 of 3 commits, 3 pages rendered; 8 files/);
+ assert.match(text, /history the latest 2 of 3 commits in 8 files\)/);
+ const manifest = JSON.parse(readFileSync(path.join(o.publicDir!, "source", "manifest.json"), "utf8"));
+ assert.equal(manifest.history.commits, 2);
+ assert.equal(manifest.history.total, 3);
+ const newest = gitIn(repo, "rev-list", "--max-count=2", "main").split("\n");
+ const published = readdirSync(path.join(o.publicDir!, "source", "git", "commit")).sort();
+ assert.equal(published.length, 2);
+ // filter-repo is `true` here, so the mirror's ids are main's.
+ assert.deepEqual(published, newest.map((c) => `${c}.html`).sort());
+});
+
+test("history: the render cache is the XDG cache's archilyzer/source-history — none for --check, none inside the checkout or the public dir", async () => {
+ const home = dir("xdg");
+ const checkout = dir("chk");
+ const pub = path.join(dir("site"), "public");
+ const paths = { monorepoRoot: checkout, sourceHistoryCacheDir: cacheDirIn(home) } as Paths;
+ const logs: string[] = [];
+ const onLog = (l: string) => logs.push(l);
+ assert.equal(await historyCacheFor({}, paths, pub, onLog), cacheDirIn(home));
+ assert.equal(await historyCacheFor({ check: true }, paths, pub, onLog), null);
+ assert.equal(await historyCacheFor({ historyCacheDir: null }, paths, pub, onLog), null);
+ assert.equal(await historyCacheFor({ historyCacheDir: path.join(home, "other") }, paths, pub, onLog), path.join(home, "other"));
+ assert.deepEqual(logs, []);
+ assert.equal(await historyCacheFor({ historyCacheDir: path.join(checkout, ".cache", "h") }, paths, pub, onLog), null);
+ assert.match(logs.at(-1)!, /the render cache .* \(XDG_CACHE_HOME\) is inside the checkout; rendering without it/);
+ assert.equal(await historyCacheFor({ historyCacheDir: path.join(pub, "h") }, paths, pub, onLog), null);
+ assert.match(logs.at(-1)!, /is inside the public dir/);
+ // getPaths: XDG_CACHE_HOME, else ~/.cache (an empty one is unset).
+ const { getPaths } = await import("../lib/paths");
+ const real = getPaths().sourceHistoryCacheDir;
+ assert.ok(real.endsWith(path.join("archilyzer", "source-history")), real);
+ const want = process.env.XDG_CACHE_HOME || path.join(os.homedir(), ".cache");
+ assert.equal(real, path.join(want, "archilyzer", "source-history"));
+});
+
+test("history: an unusable render cache is one masked line and a render without it — the publish and its history go on (review L1, L2)", async () => {
+ const repo = sourceRepo();
+ // The cache's path carries a denied literal: the line naming it is masked,
+ // as every line of the step is.
+ const planted = "plantedcachedir";
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const root = path.join(dir("ro-cache"), planted);
+ mkdirSync(root, { recursive: true });
+ chmodSync(root, 0o555);
+ try {
+ const o = opts(repo, operatorFiles("", `${planted}\n`), logs, {
+ filterRepo: ["true"],
+ stagit: fakeStagit(log),
+ tokensFile: TOKENS_FILE,
+ historyCacheDir: path.join(root, "archilyzer", "source-history"),
+ });
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ const text = logs.join("\n");
+ assert.match(text, /\[source\] history: the render cache .*\[REDACTED\].* is unusable \(EACCES\); rendering without it/);
+ assert.ok(!text.includes(planted), "never the literal");
+ assert.equal(stagitCalls(log).at(-1), `cache= limit= base=https://archilyzer.pages.dev/source/git/ repo=${MIRROR_DIR}`);
+ assert.equal(JSON.parse(readFileSync(path.join(o.publicDir!, "source", "manifest.json"), "utf8")).history.commits, 3);
+ } finally {
+ chmodSync(root, 0o755);
+ }
+});
+
+test("history: pages that would take the publish over the step's file limit are dropped with a WARNING, never the publish (review L6)", async () => {
+ const repo = sourceRepo();
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const o = opts(repo, operatorFiles("", ""), logs, {
+ filterRepo: ["true"],
+ stagit: fakeStagit(log),
+ tokensFile: TOKENS_FILE,
+ historyCacheDir: null,
+ // Any limit the rest already fills: the history is what is dropped.
+ historyFileLimit: 1,
+ });
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ assert.match(logs.join("\n"), /\[source\] WARNING: 9 history files would make \d+ files to publish, over the step's limit of 1 \(Pages allows 20,000 per deployment\) — publishing without the history pages \(\/source\/git\/\)/);
+ assert.ok(!existsSync(path.join(o.publicDir!, "source", "git")));
+ assert.ok(existsSync(path.join(o.publicDir!, "source", MIRROR_DIR, "info", "refs")), "the rest is published");
+ assert.ok(!("history" in JSON.parse(readFileSync(path.join(o.publicDir!, "source", "manifest.json"), "utf8"))));
+});
+
+test("history: a refusal with a render cache it cannot remove still exits 1 and withdraws the source; one masked line says so (re-review)", async () => {
+ const repo = sourceRepo();
+ const planted = "plantedinhistory";
+ const files = operatorFiles("", "");
+ const logs: string[] = [];
+ const log = path.join(dir("stagit-log"), "calls");
+ const cacheDir = cacheDirIn(dir("cache-home"));
+ const o = opts(repo, files, logs, {
+ filterRepo: ["true"],
+ stagit: fakeStagit(log, { extra: `said ${planted} here` }),
+ tokensFile: TOKENS_FILE,
+ historyCacheDir: cacheDir,
+ });
+ assert.equal(await publishSource(o), 0, logs.join("\n"));
+ assert.ok(existsSync(path.join(cacheDir, "key.json")));
+ // The cache becomes read-only; then a rule denies what its pages say.
+ chmodSync(cacheDir, 0o555);
+ try {
+ writeFileSync(files.denylistFile, `${planted}\n`);
+ logs.length = 0;
+ assert.equal(await publishSource(o), 1, "the refusal's exit stands");
+ const text = logs.join("\n");
+ assert.match(text, /AUDIT REFUSED/);
+ assert.match(text, /the previous publish was WITHDRAWN/);
+ assert.match(text, /\[source\] the history's render cache .* could not be removed \(EACCES\); remove it by hand/);
+ assert.ok(!text.includes(planted));
+ assert.ok(!existsSync(path.join(o.publicDir!, "source")), "the source is withdrawn");
+ assert.ok(!existsSync(path.join(path.dirname(o.publicDir!), ".source-publish.json")));
+ } finally {
+ chmodSync(cacheDir, 0o755);
+ }
+});
diff --git a/common/publish/source.ts b/common/publish/source.ts
@@ -3,12 +3,15 @@
// The private repository is never rewritten. Every publish makes a FRESH bare
// clone of its `main`, rewrites that copy with git-filter-repo (the operator's
// scrub rules over file contents AND commit messages), repacks it for git's
-// dumb-HTTP protocol, and publishes three things under homepage/public:
+// dumb-HTTP protocol, and publishes these under homepage/public:
//
// source/archilyzer.git/ a clonable mirror: HEAD, refs, info/refs,
// objects/info/packs, the packs — static files only
// source/tree/ the tracked files of main, raw, with an
// index.html per directory (sourceTree.ts)
+// source/git/ the history: the log, a page per commit with its
+// diff, the refs, two Atom feeds — stagit's, when
+// it is installed (sourceHistory.ts)
// downloads/ archilyzer-source.tar.gz + snapshot.json, the
// tarball the Downloads page has always offered
// source/manifest.json written LAST: what was published, from what
@@ -38,16 +41,33 @@ import path from "node:path";
import { runChildIntoLog } from "../jobs/runChild";
import { copyPublicFile, ownDir, writePublicFile } from "../bin/_publicFile";
import { getPaths, type Paths } from "../lib/paths";
+import { PROJECT_NAME, PROJECT_URL } from "../lib/project";
import {
CLONE_URL,
+ HISTORY_DIR,
+ HISTORY_LOG_HREF,
MIRROR_DIR,
SOURCE_MANIFEST_VERSION,
TARBALL_HREF,
TREE_HREF,
parseSourceManifest,
+ type SourceHistory,
type SourceManifest,
} from "../lib/sourceManifest";
import {
+ HistoryProblem,
+ STAGIT_INSTALL,
+ dropHistoryCache,
+ SOURCE_HISTORY_MAX_COMMITS,
+ historyCacheKey,
+ historyStylesheet,
+ homepageThemeScript,
+ renderHistory,
+ resolveStagit,
+ stagitIdentity,
+ type HistoryRun,
+} from "./sourceHistory";
+import {
SourceRefusal,
auditBare,
auditFiles,
@@ -82,8 +102,11 @@ export const SOURCE_BRANCH = "main";
* left sides refused, `404.html` refused, no context bytes.
* 3 — the re-review: masked refusal messages, a digest of every published
* file and the gitleaks identity in the key.
+ * 4 — release 15 slice SG: the history pages (source/git/, stagit) are
+ * staged, swept by the file gate and bound by the digest; stagit's
+ * identity is in the key.
*/
-export const SOURCE_STEP_VERSION = 3;
+export const SOURCE_STEP_VERSION = 4;
/** What `pipx run` fetches when `git filter-repo` is not installed. */
export const FILTER_REPO_PIPX_SPEC = "git-filter-repo==2.47.0";
@@ -120,6 +143,20 @@ export type SourcePublishOpts = PublishOpts & {
filterRepo?: string[] | null;
// The gitleaks binary; null skips the secret scan (tests). Default "gitleaks".
gitleaks?: string | null;
+ // The stagit binary; null publishes without the history pages, as a
+ // machine without stagit does. Default: resolveStagit(paths.stagitBin).
+ stagit?: string | null;
+ // The history's render cache; null renders without one. Default:
+ // paths.sourceHistoryCacheDir (~/.cache/archilyzer/source-history).
+ historyCacheDir?: string | null;
+ // The history's cap. Default: SOURCE_HISTORY_MAX_COMMITS (the tests' seam).
+ maxHistoryCommits?: number;
+ // The file limit the history's last-resort drop checks. Default: MAX_FILES
+ // (the tests' seam; step 14 always applies MAX_FILES).
+ historyFileLimit?: number;
+ // The design tokens the history pages' style.css is written from. Default:
+ // <repo>/common/styles/tokens.css.
+ tokensFile?: string;
// Where the scratch dir is made. Default: paths.sourceScratchDir.
scratchRoot?: string;
// The environment the children run in (PATH decides which tools). Default:
@@ -433,18 +470,46 @@ type PublishState = {
// sourceDigest() over every published file: an out/ (or public/) holding
// another publish's mirror or tree beside this manifest does not match.
contentDigest: string;
+ // stagitIdentity(): "absent", or which stagit rendered the history. An
+ // install (or an upgrade) re-publishes.
+ stagit: string;
+ // The history pages as audited (historyDigest), or null when none were
+ // published. A null beside a stagit that is present (a render that failed,
+ // pages over the limits) never skips: the next build tries again.
+ history: { files: number; digest: string } | null;
};
/**
* One digest over every file a publish puts on the site, under `root` laid
* out as public/ and out/ both are: `source/archilyzer.git/**`,
- * `source/tree/**`, `source/manifest.json`, `downloads/archilyzer-source.tar.gz`
- * and `downloads/snapshot.json`. Each file contributes its relative path, size
- * and sha256 (streamed), in sorted path order; a symlink or a missing piece
- * contributes a marker, so it can only differ. The /source PAGE and anything
- * else Next writes beside them are not part of it.
+ * `source/tree/**`, `source/git/**`, `source/manifest.json`,
+ * `downloads/archilyzer-source.tar.gz` and `downloads/snapshot.json`. Each
+ * file contributes its relative path, size and sha256 (streamed), in sorted
+ * path order; a symlink or a missing piece contributes a marker, so it can
+ * only differ. The /source PAGE and anything else Next writes beside them are
+ * not part of it.
*/
export async function sourceDigest(root: string): Promise<string> {
+ return digestOf(root, [
+ `source/${MIRROR_DIR}`,
+ "source/tree",
+ `source/${HISTORY_DIR}`,
+ "source/manifest.json",
+ `downloads/${TARBALL_NAME}`,
+ "downloads/snapshot.json",
+ ]);
+}
+
+/**
+ * The same digest over the history pages alone (`source/git/**`): the
+ * manifest's `history.sha256`, and what the deploy check recomputes over
+ * out/ to name a history that is not the audited one.
+ */
+export async function historyDigest(root: string): Promise<string> {
+ return digestOf(root, [`source/${HISTORY_DIR}`]);
+}
+
+async function digestOf(root: string, tops: readonly string[]): Promise<string> {
const entries: string[] = [];
const walk = async (rel: string): Promise<void> => {
const abs = path.join(root, rel);
@@ -459,15 +524,7 @@ export async function sourceDigest(root: string): Promise<string> {
entries.push(`${rel}\0${st.size}\0${await sha256File(abs)}`);
}
};
- for (const top of [
- `source/${MIRROR_DIR}`,
- "source/tree",
- "source/manifest.json",
- `downloads/${TARBALL_NAME}`,
- "downloads/snapshot.json",
- ]) {
- await walk(top);
- }
+ for (const top of tops) await walk(top);
entries.sort((a, b) => (a < b ? -1 : a > b ? 1 : 0));
const h = createHash("sha256");
for (const e of entries) h.update(`${e}\n`);
@@ -512,10 +569,10 @@ export const NO_REPOSITORY =
/**
* Remove a publish from `publicDir`: the manifest FIRST (the page never
- * describes a half-removed tree), the skip key, the mirror and the tree, the
- * tarball and snapshot.json. Link-safe like the install: a linked
- * `source/` or `downloads/` (a worktree's, into the primary) goes as a link,
- * its target untouched. True when there was something to remove.
+ * describes a half-removed tree), the skip key, the mirror, the tree and the
+ * history pages, the tarball and snapshot.json. Link-safe like the install: a
+ * linked `source/` or `downloads/` (a worktree's, into the primary) goes as a
+ * link, its target untouched. True when there was something to remove.
*/
export async function removePublishedSource(publicDir: string): Promise<boolean> {
const pubSource = path.join(publicDir, "source");
@@ -523,6 +580,7 @@ export async function removePublishedSource(publicDir: string): Promise<boolean>
const had =
existsSync(path.join(pubSource, "manifest.json")) ||
existsSync(path.join(pubSource, MIRROR_DIR)) ||
+ existsSync(path.join(pubSource, HISTORY_DIR)) ||
existsSync(path.join(pubDownloads, TARBALL_NAME));
await rm(path.join(pubSource, "manifest.json"), { force: true });
await rm(statePath(publicDir), { force: true });
@@ -583,15 +641,47 @@ export async function scratchRootProblem(
scratchRoot: string,
places: ReadonlyArray<[string, string]>,
): Promise<string | null> {
- const at = await landsAt(scratchRoot);
+ const what = await insideOf(scratchRoot, places);
+ return what
+ ? `the scratch root ${tildify(scratchRoot)} (ARCHILYZER_SOURCE_SCRATCH) is inside ${what}, where a kept scratch dir could be committed or published — point it outside`
+ : null;
+}
+
+// Which of `places` ([name, dir]) `p` lands inside, through symlinks, or null.
+async function insideOf(p: string, places: ReadonlyArray<[string, string]>): Promise<string | null> {
+ const at = await landsAt(p);
for (const [what, dir] of places) {
- if (within(at, await landsAt(dir))) {
- return `the scratch root ${tildify(scratchRoot)} (ARCHILYZER_SOURCE_SCRATCH) is inside ${what}, where a kept scratch dir could be committed or published — point it outside`;
- }
+ if (within(at, await landsAt(dir))) return what;
}
return null;
}
+/**
+ * The history's render cache for this publish, or null: none for `--check`
+ * (it writes nothing outside its scratch), none when asked for none, and none
+ * — with a line saying so — when it would land inside the checkout or the
+ * public dir, where it could be committed or published.
+ */
+export async function historyCacheFor(
+ opts: Pick<SourcePublishOpts, "check" | "historyCacheDir">,
+ paths: Paths,
+ publicDir: string,
+ onLog: (line: string) => void,
+): Promise<string | null> {
+ if (opts.check) return null;
+ const dir = opts.historyCacheDir === undefined ? (paths.sourceHistoryCacheDir ?? null) : opts.historyCacheDir;
+ if (!dir) return null;
+ const what = await insideOf(dir, [
+ ["the checkout", paths.monorepoRoot],
+ ["the public dir", publicDir],
+ ]);
+ if (what) {
+ onLog(`[source] history: the render cache ${tildify(dir)} (XDG_CACHE_HOME) is inside ${what}; rendering without it`);
+ return null;
+ }
+ return dir;
+}
+
// ── the step ────────────────────────────────────────────────────────────────
/**
@@ -614,14 +704,29 @@ export async function publishSource(opts: SourcePublishOpts = {}): Promise<numbe
const signal = opts.signal ?? new AbortController().signal;
const ctx: Ctx = { onLog, signal, env: cleanGitEnv(opts.env ?? process.env), literals: [] };
const publicDir = sourcePublicDir(paths, opts.publicDir);
- const progress = { rulesLoaded: false };
+ const progress = { rulesLoaded: false, historyCache: null as string | null };
const withdraw = async () => {
if (!progress.rulesLoaded || opts.check) return;
if (await removePublishedSource(publicDir)) {
onLog(
- "[source] the previous publish was WITHDRAWN (mirror, tree, tarball): it was audited under rules that may not be today's. /source shows its empty state until a publish passes.",
+ "[source] the previous publish was WITHDRAWN (mirror, tree, history, tarball): it was audited under rules that may not be today's. /source shows its empty state until a publish passes.",
);
}
+ // The history's render cache was rendered under those rules too. It is
+ // never why a withdrawal fails: one line, and the refusal stands.
+ if (progress.historyCache) {
+ try {
+ await dropHistoryCache(progress.historyCache);
+ } catch (err) {
+ const why = (err as NodeJS.ErrnoException)?.code ?? (err as Error)?.message ?? String(err);
+ onLog(
+ maskLiterals(
+ `[source] the history's render cache ${tildify(progress.historyCache)} could not be removed (${why}); remove it by hand`,
+ ctx.literals,
+ ),
+ );
+ }
+ }
};
let code: number;
try {
@@ -649,7 +754,7 @@ async function publish(
paths: Paths,
ctx: Ctx,
publicDir: string,
- progress: { rulesLoaded: boolean },
+ progress: { rulesLoaded: boolean; historyCache: string | null },
): Promise<number> {
const { onLog } = ctx;
const started = Date.now();
@@ -684,9 +789,17 @@ async function publish(
const filterRepoId = `${filterRepo.label} ${filterRepo.version}`;
const gitleaksBin = opts.gitleaks === undefined ? "gitleaks" : opts.gitleaks;
const gitleaksId = await gitleaksIdentity(gitleaksBin, ctx.env);
+ // stagit is optional: absent, the source is published without its history.
+ const stagit =
+ opts.stagit === undefined
+ ? resolveStagit(paths.stagitBin ?? "stagit", ctx.env)
+ : opts.stagit && existsSync(opts.stagit)
+ ? opts.stagit
+ : null;
+ const stagitId = await stagitIdentity(stagit);
// 4. Nothing changed: skip. The key is main, the rules (with the step's
- // version), both tools, and the published files themselves.
+ // version), the tools, and the published files themselves.
if (!opts.force && !opts.check) {
const manifest = await readPublishedManifest(publicDir);
const state = (await readJson(statePath(publicDir))) as Partial<PublishState> | null;
@@ -696,6 +809,8 @@ async function publish(
state.rulesHash === rules.rulesHash &&
state.filterRepo === filterRepoId &&
state.gitleaks === gitleaksId &&
+ state.stagit === stagitId &&
+ (stagitId === "absent" ? state.history === null : !!state.history) &&
state.mirrorHead === manifest.mirrorHead &&
existsSync(path.join(pubSource, MIRROR_DIR, "info", "refs")) &&
existsSync(path.join(pubDownloads, TARBALL_NAME)) &&
@@ -720,11 +835,16 @@ async function publish(
["the public dir", publicDir],
]);
if (badRoot) throw new SourceRefusal(badRoot);
+ // From here a refusal removes the history's render cache too.
+ const historyCache = await historyCacheFor(opts, paths, publicDir, (l) => onLog(maskLiterals(l, ctx.literals)));
+ progress.historyCache = historyCache;
await mkdir(scratchRoot, { recursive: true });
const scratch = await mkdtemp(path.join(scratchRoot, "archilyzer-source-"));
const replace = path.join(scratch, "replace.txt");
try {
- const bare = path.join(scratch, "bare");
+ // Named as the published mirror is: stagit names the repository after its
+ // directory ("archilyzer", the `.git` dropped).
+ const bare = path.join(scratch, MIRROR_DIR);
// 5. A fresh bare clone of main alone. --no-local: through upload-pack, not
// a copy of the object dir (which would carry every loose leftover).
@@ -852,6 +972,43 @@ async function publish(
}
const mirrorFiles = await walkFiles(stageMirror);
+ // 12b. The history pages (stagit), rendered from the audited mirror into
+ // the stage, so the file gate (13) reads every page. Without stagit — or
+ // when the render fails, or would break the host's limits — one line says
+ // so and the rest is published without them: never a failed build.
+ const stageHistoryDir = path.join(stageSource, HISTORY_DIR);
+ let history: SourceHistory | null = null;
+ if (!stagit) {
+ const why = (paths.stagitBin ?? "stagit").includes("/")
+ ? "STAGIT_BIN names no executable"
+ : "not on PATH, not in ~/.local/bin";
+ onLog(
+ `[source] stagit not found (${why}) — publishing without the history pages (/source/${HISTORY_DIR}/); install it once: ${STAGIT_INSTALL}`,
+ );
+ } else {
+ const commits = Number(lastLine((await g(["rev-list", "--count", `refs/heads/${SOURCE_BRANCH}`])).out));
+ history = await stageHistoryPages({
+ ctx,
+ opts,
+ paths,
+ stagit,
+ stagitId,
+ bare,
+ mirrorHead,
+ commits,
+ scratch,
+ historyCache,
+ rulesHash: rules.rulesHash,
+ filterRepoId,
+ stage,
+ dest: stageHistoryDir,
+ // The raw tree's files, as tracked: links to any other become text.
+ treeFiles: new Set((await walkFiles(stageTree)).map((f) => f.rel)),
+ // Everything staged so far, and the manifest still to come.
+ stagedSoFar: (await walkFiles(stage)).length + 1,
+ });
+ }
+
// The manifest, staged with the rest so the file sweep reads it too. It
// carries no rules hash and no literal count: `plans/` publishes which
// literals the plan put in the files, so either would say whether the
@@ -881,6 +1038,7 @@ async function publish(
gitleaks: audit.gitleaks === "clean" ? "clean" : "skipped",
},
tools: { git: gitVersion, filterRepo: filterRepoId },
+ ...(history ? { history } : {}),
};
const manifestText = JSON.stringify(manifest, null, 2) + "\n";
await writeFile(path.join(stageSource, "manifest.json"), manifestText);
@@ -904,7 +1062,9 @@ async function publish(
const totalBytes = all.reduce((n, f) => n + f.bytes, 0);
const summary =
`main ${sourceCommit.slice(0, 12)} as ${mirrorHead.slice(0, 12)}: ${all.length} files, ${mb(totalBytes)} MB ` +
- `(mirror ${packs} pack${packs === 1 ? "" : "s"}, tree ${tree.dirs} dirs), tarball ${mb(tarBytes)} MB sha256 ${tarSha.slice(0, 12)}`;
+ `(mirror ${packs} pack${packs === 1 ? "" : "s"}, tree ${tree.dirs} dirs, ` +
+ `${history ? `history ${history.commits < history.total ? `the latest ${history.commits} of ${history.total}` : history.commits} commits in ${history.files} files` : "no history"}), ` +
+ `tarball ${mb(tarBytes)} MB sha256 ${tarSha.slice(0, 12)}`;
// 15. --check writes nothing.
if (opts.check) {
@@ -919,8 +1079,10 @@ async function publish(
await rm(path.join(pubSource, "manifest.json"), { force: true });
await rm(path.join(pubSource, MIRROR_DIR), { recursive: true, force: true });
await rm(path.join(pubSource, "tree"), { recursive: true, force: true });
+ await rm(path.join(pubSource, HISTORY_DIR), { recursive: true, force: true });
await cp(stageMirror, path.join(pubSource, MIRROR_DIR), { recursive: true });
await cp(stageTree, path.join(pubSource, "tree"), { recursive: true });
+ if (history) await cp(stageHistoryDir, path.join(pubSource, HISTORY_DIR), { recursive: true });
await ownDir(pubDownloads);
await copyPublicFile(tarball, path.join(pubDownloads, TARBALL_NAME));
await writePublicFile(path.join(pubDownloads, "snapshot.json"), snapshotText);
@@ -931,6 +1093,8 @@ async function publish(
filterRepo: filterRepoId,
gitleaks: gitleaksId,
contentDigest,
+ stagit: stagitId,
+ history: history ? { files: history.files, digest: history.sha256 } : null,
};
await writeFile(statePath(publicDir), JSON.stringify(state, null, 2) + "\n");
await writePublicFile(path.join(pubSource, "manifest.json"), manifestText);
@@ -954,6 +1118,124 @@ function elapsed(started: number): string {
}
/**
+ * Step 12b: render the history pages into `a.dest` (stage/source/git) and say
+ * what they are, or say why not and leave no `dest` — a WARNING, never a
+ * refusal: the tokens unreadable, stagit failing, or pages that would break
+ * the host's limits (the step's, which step 14 applies to the whole publish).
+ * A cancel still cancels.
+ */
+async function stageHistoryPages(a: {
+ ctx: Ctx;
+ opts: SourcePublishOpts;
+ paths: Paths;
+ stagit: string;
+ stagitId: string;
+ bare: string;
+ mirrorHead: string;
+ commits: number;
+ scratch: string;
+ historyCache: string | null;
+ rulesHash: string;
+ filterRepoId: string;
+ stage: string;
+ dest: string;
+ treeFiles: ReadonlySet<string>;
+ stagedSoFar: number;
+}): Promise<SourceHistory | null> {
+ const { ctx } = a;
+ const started = Date.now();
+ const where = `/source/${HISTORY_DIR}/`;
+ const without = async (why: string): Promise<null> => {
+ await rm(a.dest, { recursive: true, force: true });
+ ctx.onLog(`[source] WARNING: ${maskLiterals(why, ctx.literals)} — publishing without the history pages (${where})`);
+ return null;
+ };
+ const tokensFile =
+ a.opts.tokensFile ??
+ path.join(/* turbopackIgnore: true */ a.paths.monorepoRoot, "common", "styles", "tokens.css");
+ let stylesheet: string;
+ try {
+ stylesheet = historyStylesheet(await readFile(tokensFile, "utf8"));
+ } catch (err) {
+ if (err instanceof HistoryProblem) return without(err.message);
+ return without(`the design tokens (${tildify(tokensFile)}) could not be read`);
+ }
+ // stagit and git through the step's own runner (the environment, the
+ // cancel, the timeout); a failure or a timeout is the history's problem.
+ const child: HistoryRun = async (command, args, o) => {
+ try {
+ return await run(ctx, command, args, { ...o, allowFail: true });
+ } catch (err) {
+ if (err instanceof SourceRefusal) throw new HistoryProblem(err.message);
+ throw err;
+ }
+ };
+ const baseUrl = `${PROJECT_URL}${where}`;
+ let r;
+ try {
+ r = await renderHistory({
+ stagit: a.stagit,
+ gitDir: a.bare,
+ head: a.mirrorHead,
+ commits: a.commits,
+ maxCommits: a.opts.maxHistoryCommits ?? SOURCE_HISTORY_MAX_COMMITS,
+ dest: a.dest,
+ scratch: a.scratch,
+ cacheDir: a.historyCache,
+ cacheKey: historyCacheKey({
+ rulesHash: a.rulesHash,
+ filterRepo: a.filterRepoId,
+ stagit: a.stagitId,
+ description: PROJECT_NAME,
+ cloneUrl: CLONE_URL,
+ baseUrl,
+ }),
+ fresh: !!a.opts.force,
+ description: PROJECT_NAME,
+ cloneUrl: CLONE_URL,
+ baseUrl,
+ stylesheet,
+ themeScript: homepageThemeScript(),
+ treeFiles: a.treeFiles,
+ run: child,
+ // stagit's words (a failure's tail, a timeout's argv) can name the home
+ // dir: every line is masked, as every other child line of the step is.
+ onLog: (l) => ctx.onLog(maskLiterals(l, ctx.literals)),
+ });
+ } catch (err) {
+ if (err instanceof HistoryProblem) return without(`the history pages were not rendered: ${err.message}`);
+ throw err;
+ }
+ const fileLimit = a.opts.historyFileLimit ?? MAX_FILES;
+ if (a.stagedSoFar + r.files > fileLimit) {
+ return without(
+ `${r.files} history files would make ${a.stagedSoFar + r.files} files to publish, over the step's limit of ${fileLimit} (Pages allows 20,000 per deployment)`,
+ );
+ }
+ if (r.largest.bytes > MAX_FILE_BYTES) {
+ return without(
+ `${HISTORY_DIR}/${r.largest.rel} is ${mb(r.largest.bytes)} MiB, over the step's limit of ${mb(MAX_FILE_BYTES)} MiB (Pages allows 25 MiB per file)`,
+ );
+ }
+ const sha256 = await historyDigest(a.stage);
+ ctx.onLog(
+ `[source] history: ${a.stagitId} — ${r.shown < a.commits ? `the latest ${r.shown} of ${a.commits}` : a.commits} commits, ${r.rendered} page${r.rendered === 1 ? "" : "s"} rendered` +
+ `${r.cached ? " (the rest from the cache)" : ""}; ${r.files} files, ${mb(r.bytes)} MB, ` +
+ `the largest ${HISTORY_DIR}/${r.largest.rel} ${mb(r.largest.bytes)} MB (${elapsed(started)})`,
+ );
+ return {
+ href: HISTORY_LOG_HREF,
+ commits: r.shown,
+ total: a.commits,
+ head: a.mirrorHead,
+ files: r.files,
+ bytes: r.bytes,
+ sha256,
+ tool: a.stagitId,
+ };
+}
+
+/**
* `build homepage --no-source`: remove what an earlier publish left, because
* it was audited against the rules of ITS day. The pages then show their
* empty states.
@@ -966,7 +1248,7 @@ export async function clearPublishedSource(
const had = await removePublishedSource(sourcePublicDir(paths, opts.publicDir));
onLog(
had
- ? "[notice] --no-source: the previously published source (mirror, tree, tarball) was removed — this build ships none.\n"
+ ? "[notice] --no-source: the previously published source (mirror, tree, history, tarball) was removed — this build ships none.\n"
: "[notice] --no-source: no source published in this build.\n",
);
}
@@ -1009,6 +1291,7 @@ export async function publishedSourceProblem(
path.join(outSource, "manifest.json"),
path.join(outSource, MIRROR_DIR),
path.join(outSource, "tree"),
+ path.join(outSource, HISTORY_DIR),
outTarball,
];
// A `--no-source` build: the page's empty state and nothing else.
@@ -1061,6 +1344,13 @@ export async function publishedSourceProblem(
if (state.gitleaks !== gitleaks) {
return `homepage/out's source was scanned by another gitleaks (${state.gitleaks}; this machine has ${gitleaks}) — ${rebuild}`;
}
+ // The history pages: exactly the audited ones, or none when none were
+ // published. The digest below binds them too; this names them.
+ const historyNow = existsSync(path.join(outSource, HISTORY_DIR)) ? await historyDigest(outDir) : null;
+ const historyThen = state.history?.digest ?? null;
+ if (historyNow !== historyThen || (manifest.history?.sha256 ?? null) !== historyThen) {
+ return `homepage/out's history pages (/source/${HISTORY_DIR}/) are not the ones that were audited — ${rebuild}`;
+ }
// Last, and the one that binds every byte: the mirror, the tree, the
// manifest and both downloads, against the digest of what was audited.
if ((await sourceDigest(outDir)) !== state.contentDigest) {
diff --git a/common/publish/sourceHistory.test.ts b/common/publish/sourceHistory.test.ts
@@ -0,0 +1,632 @@
+import { test, after } from "node:test";
+import assert from "node:assert/strict";
+import { execFileSync, spawnSync } from "node:child_process";
+import {
+ chmodSync,
+ utimesSync,
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ readdirSync,
+ readFileSync,
+ rmSync,
+ writeFileSync,
+} from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { buildThemeScript, HOMEPAGE_DEFAULT_BASE } from "../lib/themeConfig";
+import { writeFakeStagit } from "./__fixtures__/fakeStagit";
+import {
+ HISTORY_BACK_LINK,
+ HISTORY_LOCK_STALE_MS,
+ HISTORY_TOKENS,
+ HistoryProblem,
+ SITE_ICON_HREF,
+ SOURCE_HISTORY_MAX_COMMITS,
+ loggedCommits,
+ historyStylesheet,
+ holdHistoryCache,
+ homepageThemeScript,
+ renderHistory,
+ resolveStagit,
+ rewriteHistoryPage,
+ stageHistory,
+ stagitIdentity,
+ tokenBlocks,
+ type HistoryRun,
+ type RenderHistoryOpts,
+} from "./sourceHistory";
+
+// Run with:
+// pnpm --filter yt-dlp-transcript-common test
+//
+// The history pages (sourceHistory.ts): the post-pass, the stylesheet from
+// the tokens, the binary's lookup, and the render with its cache — over a
+// fake stagit that writes what stagit writes, and over the real one when it
+// is installed (that test SKIPS without it; a gate says whether it ran).
+for (const key of ["GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE", "GIT_PREFIX"]) {
+ delete process.env[key];
+}
+
+const HERE = path.dirname(fileURLToPath(import.meta.url));
+const TOKENS = readFileSync(path.join(HERE, "..", "styles", "tokens.css"), "utf8");
+const TMP = mkdtempSync(path.join(os.tmpdir(), "source-history-"));
+after(() => rmSync(TMP, { recursive: true, force: true }));
+
+let n = 0;
+function dir(name: string): string {
+ const d = path.join(TMP, `${name}-${n++}`);
+ mkdirSync(d, { recursive: true });
+ return d;
+}
+
+// ── the post-pass ───────────────────────────────────────────────────────────
+
+// stagit's own markup, as it writes a commit page two levels down (relpath
+// "../"): the header, the diffstat's anchors, a diff header linking both
+// sides into file/, and a message that quotes an href as text.
+const COMMIT_PAGE = `<!DOCTYPE html>
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+<title>a subject - archilyzer - Archilyzer</title>
+<link rel="icon" type="image/png" href="../favicon.png" />
+<link rel="alternate" type="application/atom+xml" title="archilyzer.git Atom Feed" href="../atom.xml" />
+<link rel="stylesheet" type="text/css" href="../style.css" />
+</head>
+<body>
+<table><tr><td><a href="../../"><img src="../logo.png" alt="" width="32" height="32" /></a></td><td><h1>archilyzer</h1></td></tr><tr><td></td><td>
+<a href="../log.html">Log</a> | <a href="../files.html">Files</a> | <a href="../refs.html">Refs</a> | <a href="../file/README.md.html">README</a> | <a href="../file/LICENSE.html">LICENSE</a></td></tr></table>
+<hr/>
+<div id="content">
+<pre><b>commit</b> <a href="../commit/${"a".repeat(40)}.html">${"a".repeat(40)}</a>
+a message that says href="file/x.html" as text
+<b>diff --git a/<a id="h0" href="../file/app/%5Bslug%5D/page.tsx.html">app/[slug]/page.tsx</a> b/<a href="../file/app/%5Bslug%5D/page.tsx.html">app/[slug]/page.tsx</a></b>
+<a href="#h0-0-0" id="h0-0-0" class="i">+added
+</a></pre>
+</div>
+</body>
+</html>
+`;
+
+test("the post-pass: the theme script before </head>, the back link after <body>, file/ links to the raw tree, the logo and favicon to the site's icon — nothing else", () => {
+ const script = "var x=1;";
+ const out = rewriteHistoryPage(COMMIT_PAGE, script);
+ const expected = COMMIT_PAGE
+ .replace("</head>", `<script>${script}</script>\n</head>`)
+ .replace("<body>\n", `<body>\n${HISTORY_BACK_LINK}\n`)
+ .replace('href="../favicon.png"', `href="${SITE_ICON_HREF}"`)
+ .replace('src="../logo.png"', `src="${SITE_ICON_HREF}"`)
+ .replace('href="../file/README.md.html"', 'href="../../tree/README.md"')
+ .replace('href="../file/LICENSE.html"', 'href="../../tree/LICENSE"')
+ .replaceAll('href="../file/app/%5Bslug%5D/page.tsx.html"', 'href="../../tree/app/%5Bslug%5D/page.tsx"');
+ assert.equal(out, expected);
+ // Text that quotes an href is encoded by stagit, and left alone.
+ assert.match(out, /says href="file\/x\.html" as text/);
+ // The back link is the one line added to the body, and it is ASCII.
+ assert.equal(HISTORY_BACK_LINK, '<p class="archilyzer-source"><a href="/source/">Archilyzer · Source</a></p>');
+ assert.ok(!/[^\x00-\x7f]/.test(HISTORY_BACK_LINK));
+});
+
+test("the post-pass at the top level: files.html's links go to ../tree/<path>, as stagit encoded them", () => {
+ const files = `<html>\n<head>\n</head>\n<body>\n<tr><td>-rw-r--r--</td><td><a href="file/fonts/Archivo%5Bwdth%2Cwght%5D.ttf.html">fonts/Archivo[wdth,wght].ttf</a></td></tr>\n<a href="file/a.html.html">a.html</a>\n</body>\n</html>\n`;
+ const out = rewriteHistoryPage(files, "s()");
+ assert.match(out, /<a href="\.\.\/tree\/fonts\/Archivo%5Bwdth%2Cwght%5D\.ttf">/);
+ assert.match(out, /<a href="\.\.\/tree\/a\.html">a\.html<\/a>/, "a file that is itself .html keeps its name");
+ assert.ok(!out.includes('href="file/'));
+});
+
+test("the post-pass with what is published: a link to a file main no longer has, or to a commit with no page, becomes text (its id kept)", () => {
+ const sha = "a".repeat(40);
+ const gone = "b".repeat(40);
+ const page = `<html>\n<head>\n</head>\n<body>\n<a href="../commit/${sha}.html">${sha}</a> <b>parent</b> <a href="../commit/${gone}.html">${gone}</a>\n<b>diff --git a/<a id="h0" href="../file/old/name.txt.html">old/name.txt</a> b/<a href="../file/app/%5Bslug%5D/page.tsx.html">app/[slug]/page.tsx</a></b>\n<a href="../file/bad%zz.html">bad</a>\n</body>\n</html>\n`;
+ const published = {
+ tree: (p: string) => p === "app/[slug]/page.tsx",
+ commit: (c: string) => c === sha,
+ };
+ const out = rewriteHistoryPage(page, "t()", published);
+ assert.ok(out.includes(`<a href="../commit/${sha}.html">${sha}</a>`), "a published commit keeps its link");
+ assert.ok(out.includes(`<b>parent</b> <a>${gone}</a>`), "an unpublished commit is text");
+ assert.ok(out.includes(`a/<a id="h0">old/name.txt</a>`), "a file main no longer has is text, and keeps the diffstat's target");
+ assert.ok(out.includes(`b/<a href="../../tree/app/%5Bslug%5D/page.tsx">`), "a file in the tree keeps its (encoded) link");
+ assert.ok(out.includes(`<a>bad</a>`), "an encoding that does not decode is not linked");
+ // Without the set, nothing is dropped (the unit tests above).
+ assert.ok(rewriteHistoryPage(page, "t()").includes(`href="../../tree/old/name.txt"`));
+});
+
+test("the theme script is the homepage's own: its base, its string", () => {
+ assert.equal(HOMEPAGE_DEFAULT_BASE, "dark");
+ assert.equal(homepageThemeScript(), buildThemeScript({ defaultBase: HOMEPAGE_DEFAULT_BASE }));
+ assert.ok(!/[^\x00-\x7f]/.test(homepageThemeScript()), "ASCII, for the byte-for-byte rewrite");
+ // It must not be able to end its own element.
+ assert.throws(() => rewriteHistoryPage("<head></head>", "a</script><b>"));
+ // The homepage's layout passes the same constant to ThemeScript.
+ const layout = readFileSync(path.join(HERE, "..", "..", "homepage", "app", "layout.tsx"), "utf8");
+ assert.match(layout, /<ThemeScript defaultBase=\{HOMEPAGE_DEFAULT_BASE\} \/>/);
+});
+
+// ── the stylesheet ──────────────────────────────────────────────────────────
+
+test("style.css: the homepage's two grounds from tokens.css, by prefers-color-scheme and by data-base", () => {
+ const css = historyStylesheet(TOKENS);
+ const { light, dark } = tokenBlocks(TOKENS);
+ const block = (selector: RegExp) => {
+ const m = selector.exec(css);
+ assert.ok(m, String(selector));
+ return m[1];
+ };
+ const lightBlock = block(/:root,\nhtml\[data-base="light"\] \{\n([\s\S]*?)\n\}/);
+ const mediaBlock = block(/@media \(prefers-color-scheme: dark\) \{\n {2}:root:not\(\[data-base\]\) \{\n([\s\S]*?)\n {2}\}/);
+ const darkBlock = block(/\nhtml\[data-base="dark"\] \{\n([\s\S]*?)\n\}/);
+ for (const name of [...HISTORY_TOKENS, "color-scheme", "--swatch-signal"]) {
+ assert.ok(lightBlock.includes(` ${name}: ${light.get(name)};`), `light ${name}`);
+ assert.ok(darkBlock.includes(` ${name}: ${dark.get(name)};`), `dark ${name}`);
+ assert.ok(mediaBlock.includes(` ${name}: ${dark.get(name)};`), `no-JS dark ${name}`);
+ }
+ // Only what the rules read (and what that reaches through var()).
+ assert.ok(!lightBlock.includes("--chart-1"));
+ // Diff lines: insertions in --success, deletions in --destructive.
+ assert.match(css, /pre a\.i \{ color: var\(--success\); \}/);
+ assert.match(css, /pre a\.d \{ color: var\(--destructive\); \}/);
+ assert.match(css, /p\.archilyzer-source a \{/);
+});
+
+test("style.css follows the tokens: a changed value is the new value; a missing block or token is a HistoryProblem", () => {
+ const moved = TOKENS.replace(/(html\[data-base="dark"\] \{[\s\S]*?--background: )#[0-9a-f]+;/, "$1#010203;");
+ assert.notEqual(moved, TOKENS);
+ assert.match(historyStylesheet(moved), /html\[data-base="dark"\] \{[\s\S]*?--background: #010203;/);
+ assert.throws(() => historyStylesheet(":root { --background: #fff; }"), HistoryProblem);
+ const noInfo = TOKENS.replace(/--info: #[0-9a-f]+;/g, "");
+ assert.throws(() => historyStylesheet(noInfo), (e) => e instanceof HistoryProblem && /no --info/.test(e.message));
+});
+
+// ── the binary ──────────────────────────────────────────────────────────────
+
+function exe(file: string, body = "#!/bin/sh\nexit 0\n"): string {
+ mkdirSync(path.dirname(file), { recursive: true });
+ writeFileSync(file, body);
+ chmodSync(file, 0o755);
+ return file;
+}
+
+test("resolveStagit: STAGIT_BIN as a path, else stagit on PATH, else ~/.local/bin/stagit, else null", async () => {
+ const home = dir("home");
+ const bin = dir("bin");
+ const empty = dir("empty");
+ assert.equal(resolveStagit("stagit", { PATH: empty }, home), null);
+ const local = exe(path.join(home, ".local", "bin", "stagit"));
+ assert.equal(resolveStagit("stagit", { PATH: empty }, home), local);
+ const onPathBin = exe(path.join(bin, "stagit"));
+ assert.equal(resolveStagit("stagit", { PATH: `${empty}:${bin}` }, home), onPathBin, "PATH first");
+ const given = exe(path.join(dir("given"), "my-stagit"));
+ assert.equal(resolveStagit(given, { PATH: bin }, home), given);
+ assert.equal(resolveStagit(path.join(empty, "nope"), { PATH: bin }, home), null, "a path that is not there is nothing, not a PATH lookup");
+ writeFileSync(path.join(empty, "not-exec"), "x");
+ assert.equal(resolveStagit(path.join(empty, "not-exec"), { PATH: bin }, home), null);
+
+ assert.equal(await stagitIdentity(null), "absent");
+ const a = await stagitIdentity(given);
+ assert.match(a, /^stagit \(sha256 [0-9a-f]{12}\)$/);
+ exe(given, "#!/bin/sh\nexit 1\n");
+ assert.notEqual(await stagitIdentity(given), a, "another binary, another identity");
+ assert.ok(!a.includes(given), "never the path");
+});
+
+// ── the render, over a fake stagit ──────────────────────────────────────────
+
+function gitIn(cwd: string, ...args: string[]): string {
+ return execFileSync("git", args, { cwd, stdio: "pipe" }).toString().trim();
+}
+
+// A bare repository named archilyzer.git with `count` commits on main.
+function bareRepo(count: number): { gitDir: string; work: string } {
+ const work = dir("work");
+ gitIn(work, "init", "-q", "-b", "main");
+ gitIn(work, "config", "user.name", "history test");
+ gitIn(work, "config", "user.email", "history@example.invalid");
+ gitIn(work, "config", "commit.gpgsign", "false");
+ for (let i = 0; i < count; i++) addCommit(work, i);
+ const gitDir = path.join(dir("bare"), "archilyzer.git");
+ gitIn(TMP, "clone", "-q", "--bare", work, gitDir);
+ return { gitDir, work };
+}
+
+function addCommit(work: string, i: number): void {
+ writeFileSync(path.join(work, "README.md"), `hello ${i}\n`);
+ mkdirSync(path.join(work, "app", "[slug]"), { recursive: true });
+ writeFileSync(path.join(work, "app", "[slug]", "page.tsx"), `export default ${i};\n`);
+ gitIn(work, "add", "-A");
+ gitIn(work, "commit", "-q", "-m", `commit ${i}`);
+}
+
+// The fake (__fixtures__/fakeStagit.ts) writes what stagit writes, with -c
+// and -l as stagit.c has them; every call is logged to `log`.
+function fakeStagit(log: string, o: { exit?: number; extra?: string; pad?: number } = {}): string {
+ return writeFakeStagit(path.join(dir("fake"), "stagit"), log, o);
+}
+
+const run: HistoryRun = async (command, args, o) => {
+ const p = spawnSync(command, args, { cwd: o.cwd, encoding: "utf8" });
+ return { code: p.status ?? 1, out: `${p.stdout}${p.stderr}`.split("\n") };
+};
+
+function renderOpts(gitDir: string, stagit: string, extra: Partial<RenderHistoryOpts> = {}): RenderHistoryOpts & { logs: string[] } {
+ const logs: string[] = [];
+ const head = gitIn(TMP, "--git-dir", gitDir, "rev-parse", "main");
+ const commits = Number(gitIn(TMP, "--git-dir", gitDir, "rev-list", "--count", "main"));
+ return {
+ stagit,
+ gitDir,
+ head,
+ commits,
+ maxCommits: SOURCE_HISTORY_MAX_COMMITS,
+ dest: path.join(dir("stage"), "source", "git"),
+ scratch: dir("scratch"),
+ cacheDir: null,
+ cacheKey: "k1",
+ fresh: false,
+ description: "Archilyzer",
+ cloneUrl: "https://archilyzer.pages.dev/source/archilyzer.git",
+ baseUrl: "https://archilyzer.pages.dev/source/git/",
+ stylesheet: "/* css */\n",
+ themeScript: "t()",
+ run,
+ onLog: (l) => logs.push(l),
+ logs,
+ ...extra,
+ };
+}
+
+const calls = (log: string) => (existsSync(log) ? readFileSync(log, "utf8").trim().split("\n") : []);
+
+test("render: the allowlist of stagit's output, each page through the post-pass, the feeds as they are, style.css; never file/", async () => {
+ const { gitDir } = bareRepo(3);
+ const log = path.join(dir("log"), "calls");
+ const o = renderOpts(gitDir, fakeStagit(log));
+ const r = await renderHistory(o);
+ assert.deepEqual(calls(log), [`cache= limit= base=${o.baseUrl} repo=archilyzer.git`]);
+ assert.equal(r.files, 3 + 5 + 1);
+ assert.equal(r.rendered, 3);
+ assert.equal(r.shown, 3);
+ assert.equal(r.cached, false);
+ assert.deepEqual(readdirSync(o.dest).sort(), ["atom.xml", "commit", "files.html", "log.html", "refs.html", "style.css", "tags.xml"]);
+ assert.equal(readdirSync(path.join(o.dest, "commit")).length, 3);
+ const logHtml = readFileSync(path.join(o.dest, "log.html"), "utf8");
+ // stagit's header reads the description and the clone URL from the repo.
+ assert.match(logHtml, /<span class="desc">Archilyzer<\/span> https:\/\/archilyzer\.pages\.dev\/source\/archilyzer\.git /);
+ assert.ok(logHtml.includes(`<body>\n${HISTORY_BACK_LINK}\n`));
+ assert.ok(logHtml.includes("<script>t()</script>\n</head>"));
+ assert.match(logHtml, /href="\.\.\/tree\/README\.md"/);
+ assert.match(logHtml, new RegExp(`src="${SITE_ICON_HREF}"`));
+ assert.match(readFileSync(path.join(o.dest, "files.html"), "utf8"), /href="\.\.\/tree\/app\/%5Bslug%5D\/page\.tsx"/);
+ // The feeds are not pages: not one byte changes.
+ assert.equal(readFileSync(path.join(o.dest, "atom.xml"), "utf8"), `<feed>${o.baseUrl} file/README.md.html</feed>\n`);
+ assert.equal(readFileSync(path.join(o.dest, "style.css"), "utf8"), "/* css */\n");
+ // No cache: the render ran in the publish's own scratch.
+ assert.ok(existsSync(path.join(o.scratch, "history", "log.html")));
+ assert.ok(!existsSync(path.join(o.scratch, "history", "file")), "the per-file pages are dropped where they were rendered");
+});
+
+test("render: the page bytes are kept (a diff of a file that is not UTF-8), only the ASCII injections added", async () => {
+ const work = dir("bytes");
+ mkdirSync(path.join(work, "commit"));
+ const odd = Buffer.concat([Buffer.from("<html>\n<head>\n</head>\n<body>\n"), Buffer.from([0xff, 0xfe, 0x80]), Buffer.from("\n</body>\n</html>\n")]);
+ writeFileSync(path.join(work, "log.html"), odd);
+ for (const f of ["files.html", "refs.html"]) writeFileSync(path.join(work, f), "<html>\n<head>\n</head>\n<body>\n</body>\n</html>\n");
+ for (const f of ["atom.xml", "tags.xml"]) writeFileSync(path.join(work, f), Buffer.from([0x3c, 0xff, 0x3e]));
+ const dest = path.join(dir("dest"), "git");
+ await stageHistory(work, dest, { themeScript: "t()", stylesheet: "", commits: [] });
+ const out = readFileSync(path.join(dest, "log.html"));
+ assert.ok(out.includes(Buffer.from([0xff, 0xfe, 0x80])), "the odd bytes survive");
+ assert.ok(out.includes(Buffer.from(HISTORY_BACK_LINK)));
+ assert.deepEqual(readFileSync(path.join(dest, "atom.xml")), Buffer.from([0x3c, 0xff, 0x3e]));
+ await assert.rejects(stageHistory(work, dest, { themeScript: "é", stylesheet: "", commits: [] }), /ASCII/);
+ // A commit the log lists must have its page.
+ await assert.rejects(
+ stageHistory(work, dest, { themeScript: "t()", stylesheet: "", commits: ["a".repeat(40)] }),
+ (e) => e instanceof HistoryProblem && /the log lists aaaaaaaaaaaa, whose page is not there/.test(e.message),
+ );
+});
+
+test("render: a failing stagit, or one that leaves the wrong pages, is a HistoryProblem", async () => {
+ const { gitDir } = bareRepo(2);
+ const log = path.join(dir("log"), "calls");
+ await assert.rejects(
+ renderHistory(renderOpts(gitDir, fakeStagit(log, { exit: 3 }))),
+ (e) => e instanceof HistoryProblem && /stagit exited 3: stagit: something broke/.test(e.message),
+ );
+ await assert.rejects(
+ renderHistory(renderOpts(gitDir, fakeStagit(log), { commits: 5 })),
+ (e) => e instanceof HistoryProblem && /stagit's log lists 2 commits, not 5 \(5 in all, at most 10000\)/.test(e.message),
+ );
+});
+
+test("loggedCommits: the commits the log links, in order, once each", () => {
+ const a = "a".repeat(40);
+ const b = "b".repeat(40);
+ const log = `<tr><td><a href="commit/${a}.html">x</a></td></tr>\n<tr><td><a href="commit/${b}.html">y</a></td></tr>\n<a href="commit/${a}.html">again</a> says href="commit/${"c".repeat(40)}.html"`;
+ assert.deepEqual(loggedCommits(log), [a, b]);
+});
+
+test("the cap: past it, -l keeps the newest in the log, and only their pages are published", async () => {
+ const { gitDir, work } = bareRepo(5);
+ const newest = gitIn(TMP, "--git-dir", gitDir, "rev-list", "--max-count=3", "main").split("\n");
+ const log = path.join(dir("log"), "calls");
+ const stagit = fakeStagit(log);
+
+ // No cache: `-l 3`, no `-c`.
+ let o = renderOpts(gitDir, stagit, { maxCommits: 3 });
+ let r = await renderHistory(o);
+ assert.equal(calls(log).at(-1), `cache= limit=3 base=${o.baseUrl} repo=archilyzer.git`);
+ assert.equal(r.shown, 3);
+ assert.equal(r.files, 3 + 5 + 1);
+ assert.deepEqual(readdirSync(path.join(o.dest, "commit")).sort(), newest.map((c) => `${c}.html`).sort());
+ assert.equal(readdirSync(path.join(o.scratch, "history", "commit")).length, 5, "stagit wrote all five; three are published");
+ assert.match(readFileSync(path.join(o.dest, "log.html"), "utf8"), /2 more commits remaining, fetch the repository/);
+
+ // With the cache: still `-l 3` (stagit refuses -c with -l). A new commit:
+ // its page is the one rendered, and the oldest of the three drops out.
+ const cacheDir = path.join(dir("cache-home"), "archilyzer", "source-history");
+ o = renderOpts(gitDir, stagit, { maxCommits: 3, cacheDir });
+ r = await renderHistory(o);
+ assert.equal(calls(log).at(-1), `cache= limit=3 base=${o.baseUrl} repo=archilyzer.git`);
+ assert.equal(r.rendered, 5);
+ addCommit(work, 9);
+ gitIn(TMP, "--git-dir", gitDir, "fetch", "-q", work, "main:main");
+ const now = gitIn(TMP, "--git-dir", gitDir, "rev-list", "--max-count=3", "main").split("\n");
+ o = renderOpts(gitDir, stagit, { maxCommits: 3, cacheDir });
+ r = await renderHistory(o);
+ assert.equal(r.cached, true);
+ assert.equal(r.rendered, 1);
+ assert.equal(r.shown, 3);
+ assert.deepEqual(readdirSync(path.join(o.dest, "commit")).sort(), now.map((c) => `${c}.html`).sort());
+ assert.match(readFileSync(path.join(o.dest, "log.html"), "utf8"), /3 more commits remaining/);
+});
+
+test("the cache: -c in the cache dir, incremental the next time, the key and the log's ancestry checked, busy or cut-off locks handled", async () => {
+ const { gitDir, work } = bareRepo(2);
+ const log = path.join(dir("log"), "calls");
+ const stagit = fakeStagit(log);
+ const cacheDir = path.join(dir("cache-home"), "archilyzer", "source-history");
+ const cacheFile = path.join(cacheDir, "stagit.cache");
+ const cacheArgs = (o: RenderHistoryOpts) => `cache=${cacheFile} limit= base=${o.baseUrl} repo=archilyzer.git`;
+
+ // First: everything rendered, the cache made (recursively) and kept (key,
+ // cache file, output), unlocked; no per-file pages, no stagit leftovers.
+ let o = renderOpts(gitDir, stagit, { cacheDir });
+ let r = await renderHistory(o);
+ assert.equal(r.cached, false);
+ assert.equal(r.rendered, 2);
+ assert.equal(calls(log).at(-1), cacheArgs(o));
+ assert.deepEqual(readdirSync(cacheDir).sort(), ["key.json", "out", "stagit.cache"]);
+ assert.ok(!existsSync(path.join(cacheDir, "out", "file")));
+ assert.ok(!existsSync(path.join(o.dest, "cache.XXXXleftover")), "only the allowlist is published");
+
+ // A new commit: the cache is used, and only the new page is rendered.
+ addCommit(work, 2);
+ gitIn(TMP, "--git-dir", gitDir, "fetch", "-q", work, "main:main");
+ o = renderOpts(gitDir, stagit, { cacheDir });
+ r = await renderHistory(o);
+ assert.equal(r.cached, true);
+ assert.equal(r.rendered, 1);
+ assert.equal(r.files, 3 + 5 + 1);
+ assert.equal(loggedCommits(readFileSync(path.join(o.dest, "log.html"), "utf8")).length, 3, "the new line and the cached ones");
+
+ // Another key (other rules, another stagit…): every page rendered again.
+ const junk = path.join(cacheDir, "out", "commit", `${"f".repeat(40)}.html`);
+ writeFileSync(junk, "junk");
+ o = renderOpts(gitDir, stagit, { cacheDir, cacheKey: "k2" });
+ r = await renderHistory(o);
+ assert.equal(r.cached, false);
+ assert.equal(r.rendered, 3);
+ assert.ok(!existsSync(junk));
+
+ // Log lines ending at a commit that is not an ancestor of the head (a
+ // rewritten main): the cache file goes, the pages stay, the log is whole.
+ writeFileSync(cacheFile, `${"e".repeat(40)}\n<tr><td><a href="commit/${"e".repeat(40)}.html">x</a></td></tr>\n`);
+ o = renderOpts(gitDir, stagit, { cacheDir, cacheKey: "k2" });
+ r = await renderHistory(o);
+ assert.equal(r.cached, true);
+ assert.equal(r.rendered, 0);
+ assert.equal(loggedCommits(readFileSync(path.join(o.dest, "log.html"), "utf8")).length, 3);
+
+ // A page of a commit history does not have is never published.
+ writeFileSync(junk, "junk");
+ o = renderOpts(gitDir, stagit, { cacheDir, cacheKey: "k2" });
+ r = await renderHistory(o);
+ assert.equal(r.files, 3 + 5 + 1);
+ assert.ok(!existsSync(path.join(o.dest, "commit", `${"f".repeat(40)}.html`)));
+
+ // A listed commit's page gone from the cache: noticed, and every page is
+ // rendered again.
+ const head = gitIn(TMP, "--git-dir", gitDir, "rev-parse", "main");
+ const oldest = gitIn(TMP, "--git-dir", gitDir, "rev-list", "main").split("\n").at(-1)!;
+ assert.notEqual(oldest, head);
+ rmSync(path.join(cacheDir, "out", "commit", `${oldest}.html`));
+ o = renderOpts(gitDir, stagit, { cacheDir, cacheKey: "k2" });
+ r = await renderHistory(o);
+ assert.equal(r.cached, false);
+ assert.match(o.logs.join("\n"), new RegExp(`the cached render does not cover this head \\(the log lists ${oldest.slice(0, 12)}, whose page is not there\\) — stagit's -c stops at the last head it rendered, and a merge of older commits falls behind it; rendering every page again`));
+ assert.ok(existsSync(path.join(o.dest, "commit", `${oldest}.html`)));
+
+ // `fresh` (--force) renders every page, whatever the cache holds.
+ o = renderOpts(gitDir, stagit, { cacheDir, cacheKey: "k2", fresh: true });
+ r = await renderHistory(o);
+ assert.equal(r.cached, false);
+ assert.equal(r.rendered, 3);
+
+ // Busy: another live publish holds it (this process's pid stands in).
+ writeFileSync(path.join(cacheDir, "lock"), `${process.pid}\n`);
+ o = renderOpts(gitDir, stagit, { cacheDir, cacheKey: "k2" });
+ r = await renderHistory(o);
+ assert.equal(r.cached, false);
+ assert.equal(calls(log).at(-1), `cache= limit= base=${o.baseUrl} repo=archilyzer.git`, "rendered without -c");
+ assert.match(o.logs.join("\n"), /the render cache is in use by another publish; rendering without it/);
+ assert.ok(existsSync(path.join(cacheDir, "lock")), "the other holder's lock is left alone");
+
+ // Cut off: the holder is gone, so nothing in the cache is trusted.
+ const dead = spawnSync("sh", ["-c", "echo $$"], { encoding: "utf8" }).stdout.trim();
+ writeFileSync(path.join(cacheDir, "lock"), `${dead}\n`);
+ writeFileSync(junk, "junk");
+ assert.equal(await holdHistoryCache(cacheDir), true);
+ assert.ok(!existsSync(junk), "emptied");
+ assert.ok(!existsSync(path.join(cacheDir, "key.json")));
+ rmSync(path.join(cacheDir, "lock"));
+
+ // A render that fails empties the cache and releases it.
+ o = renderOpts(gitDir, fakeStagit(log, { exit: 2 }), { cacheDir, cacheKey: "k2" });
+ await assert.rejects(renderHistory(o), HistoryProblem);
+ assert.deepEqual(readdirSync(cacheDir), []);
+});
+
+test("a stale lock: its pid not running, or over an hour old whoever runs its pid now, is replaced with one line", async () => {
+ const cacheDir = path.join(dir("stale"), "archilyzer", "source-history");
+ mkdirSync(path.join(cacheDir, "out"), { recursive: true });
+ const lock = path.join(cacheDir, "lock");
+ const logs: string[] = [];
+ const onLog = (l: string) => {
+ logs.push(l);
+ };
+ // Live and fresh: busy.
+ writeFileSync(lock, `${process.pid}\n`);
+ assert.equal(await holdHistoryCache(cacheDir, onLog), false);
+ assert.deepEqual(logs, []);
+ // Live, but two hours old: stale.
+ const old = new Date(Date.now() - 2 * HISTORY_LOCK_STALE_MS);
+ utimesSync(lock, old, old);
+ assert.equal(await holdHistoryCache(cacheDir, onLog), true);
+ assert.match(logs.join("\n"), new RegExp(`a stale lock on the render cache \\(pid ${process.pid}, 120 minutes old\\) was replaced; the cache is rendered afresh`));
+ assert.ok(!existsSync(path.join(cacheDir, "out")), "emptied");
+ assert.equal(readFileSync(lock, "utf8"), `${process.pid}\n`, "and held");
+ rmSync(lock);
+ // Not running: stale, whatever its age.
+ const dead = spawnSync("sh", ["-c", "echo $$"], { encoding: "utf8" }).stdout.trim();
+ writeFileSync(lock, `${dead}\n`);
+ logs.length = 0;
+ assert.equal(await holdHistoryCache(cacheDir, onLog), true);
+ assert.match(logs.join("\n"), new RegExp(`\\(pid ${dead}, not running\\) was replaced`));
+});
+
+test("a render cache that cannot be written (a read-only dir, or one that cannot be made) is one line and a render without it", async () => {
+ const { gitDir } = bareRepo(2);
+ const log = path.join(dir("log"), "calls");
+ const stagit = fakeStagit(log);
+ const root = dir("ro");
+ const readOnly = path.join(root, "cache");
+ mkdirSync(readOnly);
+ chmodSync(readOnly, 0o555);
+ const parent = path.join(root, "parent");
+ mkdirSync(parent);
+ chmodSync(parent, 0o555);
+ try {
+ for (const cacheDir of [readOnly, path.join(parent, "archilyzer", "source-history")]) {
+ const o = renderOpts(gitDir, stagit, { cacheDir });
+ const r = await renderHistory(o);
+ assert.equal(r.cached, false);
+ assert.equal(r.shown, 2);
+ assert.equal(calls(log).at(-1), `cache= limit= base=${o.baseUrl} repo=archilyzer.git`, "rendered without -c");
+ assert.match(o.logs.join("\n"), /the render cache .* is unusable \(EACCES\); rendering without it/);
+ assert.equal(readdirSync(path.join(o.dest, "commit")).length, 2);
+ }
+ } finally {
+ chmodSync(readOnly, 0o755);
+ chmodSync(parent, 0o755);
+ }
+});
+
+test("a --no-ff merge of commits older than the cached head: stagit's -c misses them, the count notices, and every page is rendered again", async () => {
+ const { gitDir, work } = bareRepo(2);
+ const log = path.join(dir("log"), "calls");
+ const stagit = fakeStagit(log);
+ const cacheDir = path.join(dir("merge-cache"), "archilyzer", "source-history");
+ // A branch whose commits are dated before the head the cache will name.
+ const base = gitIn(work, "rev-parse", "HEAD~1");
+ gitIn(work, "checkout", "-q", "-b", "side", base);
+ for (const i of [7, 8]) {
+ writeFileSync(path.join(work, `side-${i}.txt`), `${i}\n`);
+ gitIn(work, "add", "-A");
+ execFileSync("git", ["commit", "-q", "-m", `side ${i}`], {
+ cwd: work,
+ stdio: "pipe",
+ env: { ...process.env, GIT_COMMITTER_DATE: `2001-01-0${i - 6}T00:00:00Z`, GIT_AUTHOR_DATE: `2001-01-0${i - 6}T00:00:00Z` },
+ });
+ }
+ gitIn(work, "checkout", "-q", "main");
+ let o = renderOpts(gitDir, stagit, { cacheDir });
+ await renderHistory(o);
+ gitIn(work, "merge", "-q", "--no-ff", "-m", "merge side", "side");
+ gitIn(TMP, "--git-dir", gitDir, "fetch", "-q", work, "main:main");
+ o = renderOpts(gitDir, stagit, { cacheDir });
+ const r = await renderHistory(o);
+ assert.equal(r.shown, 5);
+ assert.equal(r.cached, false);
+ assert.match(o.logs.join("\n"), /the cached render does not cover this head \(stagit's log lists 3 commits, not 5 .*\) — stagit's -c stops at the last head it rendered, and a merge of older commits falls behind it; rendering every page again/);
+ assert.equal(readdirSync(path.join(o.dest, "commit")).length, 5);
+});
+
+// ── the real stagit ─────────────────────────────────────────────────────────
+
+test("the real stagit: a page per commit, every link into file/ now into the tree, every relative link resolves", async (t) => {
+ const stagit = resolveStagit(process.env.STAGIT_BIN ?? "stagit", process.env);
+ if (!stagit) return t.skip("stagit is not installed (STAGIT_BIN, PATH, ~/.local/bin)");
+ const { gitDir, work } = bareRepo(1);
+ // A file that main no longer has: its diffs are published, its links not.
+ writeFileSync(path.join(work, "gone.txt"), "soon gone\n");
+ gitIn(work, "add", "-A");
+ gitIn(work, "commit", "-q", "-m", "add gone.txt");
+ gitIn(work, "rm", "-q", "gone.txt");
+ gitIn(work, "commit", "-q", "-m", "remove gone.txt");
+ gitIn(TMP, "--git-dir", gitDir, "fetch", "-q", work, "main:main");
+ const treeFiles = new Set(["README.md", "app/[slug]/page.tsx"]);
+ const o = renderOpts(gitDir, stagit, { cacheDir: path.join(dir("real-root"), "archilyzer", "source-history"), treeFiles });
+ const r = await renderHistory(o);
+ assert.equal(r.rendered, 3);
+ assert.equal(readdirSync(path.join(o.dest, "commit")).length, 3);
+ assert.ok(!existsSync(path.join(o.dest, "file")));
+ for (const page of ["log.html", "files.html", "refs.html", ...readdirSync(path.join(o.dest, "commit")).map((f) => `commit/${f}`)]) {
+ const html = readFileSync(path.join(o.dest, page), "utf8");
+ assert.ok(html.includes(`<body>\n${HISTORY_BACK_LINK}\n`), page);
+ assert.ok(html.includes("<script>t()</script>\n</head>"), page);
+ assert.ok(!/href="(?:\.\.\/)*file\//.test(html), `${page} links into file/`);
+ assert.ok(!/(?:logo|favicon)\.png"/.test(html), `${page} names stagit's images`);
+ for (const m of html.matchAll(/(?:href|src)="([^"#:]+)"/g)) {
+ const target = m[1];
+ if (target.startsWith("/")) continue; // the site's own: /source/, the icon
+ const resolved = path.posix
+ .normalize(path.posix.join(path.posix.dirname(`source/git/${page}`), target))
+ .replace(/\/$/, "");
+ if (resolved.startsWith("source/tree/")) {
+ // Into the raw tree: only a file main has (`gone.txt` is text).
+ assert.ok(treeFiles.has(decodeURIComponent(resolved.slice("source/tree/".length))), `${page}: ${target}`);
+ } else if (resolved.startsWith("source/git/")) {
+ assert.ok(existsSync(path.join(path.dirname(path.dirname(o.dest)), resolved)), `${page}: ${target} → ${resolved}`);
+ } else {
+ assert.equal(resolved, "source", `${page}: ${target}`); // the logo's link, to /source/
+ }
+ }
+ }
+ const pages = readdirSync(path.join(o.dest, "commit")).map((f) => readFileSync(path.join(o.dest, "commit", f), "utf8"));
+ assert.ok(pages.some((h) => /<a id="h0">gone\.txt<\/a>/.test(h)), "gone.txt's diff header is text, its #h0 target kept");
+ const logHtml = readFileSync(path.join(o.dest, "log.html"), "utf8");
+ assert.match(logHtml, /<title>Log - archilyzer - Archilyzer<\/title>/);
+ assert.match(logHtml, /git clone <a href="https:\/\/archilyzer\.pages\.dev\/source\/archilyzer\.git">/);
+ assert.match(readFileSync(path.join(o.dest, "atom.xml"), "utf8"), /href="https:\/\/archilyzer\.pages\.dev\/source\/git\/commit\/[0-9a-f]{40}\.html"/);
+
+ // The cap, by stagit itself: -l keeps the NEWEST in the log (and says how
+ // many more), writes a page for every commit, and only the listed are
+ // published.
+ const capped = renderOpts(gitDir, stagit, { maxCommits: 2, treeFiles });
+ const rc = await renderHistory(capped);
+ const newest = gitIn(TMP, "--git-dir", gitDir, "rev-list", "--max-count=2", "main").split("\n");
+ assert.equal(rc.shown, 2);
+ assert.deepEqual(loggedCommits(readFileSync(path.join(capped.dest, "log.html"), "utf8")), newest);
+ assert.deepEqual(readdirSync(path.join(capped.dest, "commit")).sort(), newest.map((c) => `${c}.html`).sort());
+ assert.equal(readdirSync(path.join(capped.scratch, "history", "commit")).length, 3);
+ assert.match(readFileSync(path.join(capped.dest, "log.html"), "utf8"), /1 more commits remaining, fetch the repository/);
+ // The oldest published page's parent has no page: its link is text.
+ const oldestShown = readFileSync(path.join(capped.dest, "commit", `${newest[1]}.html`), "utf8");
+ const parent = gitIn(TMP, "--git-dir", gitDir, "rev-parse", `${newest[1]}^`);
+ assert.ok(oldestShown.includes(`<b>parent</b> <a>${parent}</a>`), "the parent past the cap is text");
+ assert.ok(!oldestShown.includes(`commit/${parent}.html`));
+});
diff --git a/common/publish/sourceHistory.ts b/common/publish/sourceHistory.ts
@@ -0,0 +1,767 @@
+// The source's history pages: stagit's rendering of the scrubbed mirror,
+// published at /source/git/ by `archilyzer source publish` (source.ts), which
+// runs this after the mirror is built and its objects audited, and before the
+// staged files are audited — so every page goes through the same gate as the
+// mirror, the tree and the tarball.
+//
+// stagit (codemadness.org, C over libgit2) is an OPERATOR-INSTALLED tool, like
+// git-filter-repo: never vendored, never committed. It is found as STAGIT_BIN,
+// else `stagit` on PATH, else ~/.local/bin/stagit (resolveStagit). WITHOUT IT
+// THE PUBLISH GOES ON: one log line says so and how to install it, and the
+// source is published without /source/git/ (the /source/ page then shows no
+// History links). A render that fails, or pages over the host's limits, are
+// the same: a WARNING, and no history — never a failed build.
+//
+// WHAT IS PUBLISHED, from an allowlist of what stagit writes:
+// log.html, files.html, refs.html, atom.xml, tags.xml, commit/<sha>.html
+// and style.css, written here from common/styles/tokens.css. NOT stagit's
+// per-file pages (file/…): browsing is the raw tree at /source/tree/, so
+// every link into file/ is rewritten to it and file/ is never staged.
+//
+// THE POST-PASS (rewriteHistoryPage) touches every .html page and never the
+// two feeds. It adds exactly two things — the homepage's pre-paint theme
+// script (lib/themeConfig.ts buildThemeScript, the string the homepage's
+// ThemeScript emits), and one line at the top linking back to /source/ — and
+// points two kinds of stagit link somewhere that exists: `…file/<path>.html`
+// at the raw tree, and its logo.png / favicon.png at the site's own icon.
+// stagit's header (the name, the description, the clone line, Log | Files |
+// Refs) stays as stagit writes it.
+//
+// THE CAP. At most SOURCE_HISTORY_MAX_COMMITS commits, the newest, get a page:
+// past it stagit runs with `-l`, the log lists that many and says how many
+// more there are, and only the listed commits' pages are published (stagit
+// still writes one for every commit). The /source/ page then says "the latest
+// N of M". The 15,000-file drop in source.ts stays, as the last resort.
+//
+// THE CACHE. stagit's `-c <cachefile>` renders incrementally: it walks from
+// HEAD to the commit the cache names and keeps every commit page already in
+// its output directory (`-l` keeps them too; it cannot be combined with
+// `-c`). So the cache is a DIRECTORY kept between publishes —
+// `${XDG_CACHE_HOME:-~/.cache}/archilyzer/source-history/`
+// (paths.sourceHistoryCacheDir; never inside the checkout or the public dir)
+// holding the cache file, stagit's output and a key. Its pages are kept only
+// when the key matches (the scrub rules and step, filter-repo, stagit, the
+// header text) and the last run finished (the key is removed before a render
+// and written after it); its log lines only when the commit they end at is an
+// ancestor of today's head. One publish holds it at a time (`lock`, the
+// holder's pid); a second renders without it, and a lock whose pid is not
+// running or that is over an hour old is stale and replaced. A cache that
+// cannot be written (EACCES, EROFS, ENOSPC) is one line and a render without
+// it. `--force` renders it afresh, `--check` never touches it, and a refusal
+// removes it.
+//
+// LINKS TO WHAT IS NOT PUBLISHED become text: a diff's file that main no
+// longer has, a commit with no page (past the cap). The raw tree's file list
+// decides (`treeFiles`).
+
+import { createHash } from "node:crypto";
+import { accessSync, constants, existsSync, statSync } from "node:fs";
+import { cp, mkdir, readdir, readFile, realpath, rm, stat, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { PROJECT_NAME } from "../lib/project";
+import { buildThemeScript, HOMEPAGE_DEFAULT_BASE } from "../lib/themeConfig";
+import { onPath, tildify } from "./sourceAudit";
+
+/** How to install stagit, as the log line and the doctor say it. */
+export const STAGIT_INSTALL =
+ "git clone git://git.codemadness.org/stagit && make -C stagit && cp stagit/stagit ~/.local/bin/";
+
+/**
+ * The cap: at most this many commits — the newest — get a page and a log
+ * line (stagit `-l`); the log says how many more there are. One file per
+ * commit counts against the step's 15,000 (Pages' 20,000).
+ */
+export const SOURCE_HISTORY_MAX_COMMITS = 10_000;
+
+/** The site's own 32 px icon, which stagit's logo.png and favicon.png become. */
+export const SITE_ICON_HREF = "/icons/icon-32.png";
+
+/**
+ * The one line put at the top of every page. ASCII only (`·`): the
+ * pages are rewritten byte for byte (latin1 in, latin1 out), so anything
+ * injected must be the same bytes in every encoding.
+ */
+export const HISTORY_BACK_LINK = `<p class="archilyzer-source"><a href="/source/">${PROJECT_NAME} · Source</a></p>`;
+
+/** The pre-paint script the homepage emits (homepage/app/layout.tsx). */
+export function homepageThemeScript(): string {
+ return buildThemeScript({ defaultBase: HOMEPAGE_DEFAULT_BASE });
+}
+
+// What stagit writes that is published: the top-level pages and feeds, and
+// one page per commit.
+const TOP_FILES = ["log.html", "files.html", "refs.html", "atom.xml", "tags.xml"] as const;
+const COMMIT_PAGE = /^[0-9a-f]{40}\.html$/;
+
+/** Something about the history went wrong: a WARNING, and no history. */
+export class HistoryProblem extends Error {
+ constructor(message: string) {
+ super(message);
+ this.name = "HistoryProblem";
+ }
+}
+
+// ── the binary ──────────────────────────────────────────────────────────────
+
+function isExecutableFile(p: string): boolean {
+ try {
+ accessSync(p, constants.X_OK);
+ return statSync(p).isFile();
+ } catch {
+ return false;
+ }
+}
+
+/**
+ * The stagit binary to run, or null. `bin` is paths.stagitBin: STAGIT_BIN,
+ * else "stagit". A name with a slash is that file, or nothing. A bare name is
+ * looked up on PATH, then in ~/.local/bin (the editor's process may not have
+ * it on its PATH, where a hand-built tool is usually put).
+ */
+export function resolveStagit(bin: string, env: NodeJS.ProcessEnv, homeDir: string = os.homedir()): string | null {
+ if (bin.includes("/")) return isExecutableFile(bin) ? bin : null;
+ const found = onPath(bin, env.PATH);
+ if (found) return found;
+ const local = path.join(homeDir, ".local", "bin", bin);
+ return isExecutableFile(local) ? local : null;
+}
+
+/**
+ * Which stagit renders: "absent", else the first 12 hex of its binary's
+ * sha256 (stagit has no version flag). Part of the publish's skip key and of
+ * the cache's, and the manifest's `history.tool`. Never the path: the
+ * manifest is published, and a path under the home dir is a denied literal.
+ */
+export async function stagitIdentity(found: string | null): Promise<string> {
+ if (!found) return "absent";
+ const bytes = await readFile(await realpath(found));
+ return `stagit (sha256 ${createHash("sha256").update(bytes).digest("hex").slice(0, 12)})`;
+}
+
+// ── the stylesheet ──────────────────────────────────────────────────────────
+
+// The tokens the stylesheet reads, on each base. `--brand` is Signal on both
+// (the homepage's accent: the base blocks default to it), and every token a
+// value names through var() comes along.
+export const HISTORY_TOKENS = [
+ "--background",
+ "--surface",
+ "--foreground",
+ "--muted-foreground",
+ "--faint",
+ "--border-strong",
+ "--brand",
+ "--brand-soft",
+ "--info",
+ "--success",
+ "--destructive",
+] as const;
+
+type Decls = Map<string, string>;
+
+function declarations(body: string): Decls {
+ const out: Decls = new Map();
+ for (const part of body.split(";")) {
+ const i = part.indexOf(":");
+ if (i === -1) continue;
+ const name = part.slice(0, i).trim();
+ if (name.startsWith("--") || name === "color-scheme") out.set(name, part.slice(i + 1).trim());
+ }
+ return out;
+}
+
+/**
+ * The light and the dark base blocks of tokens.css, as declarations: the rule
+ * whose selector list holds `html[data-base="light"]`, and the one holding
+ * `html[data-base="dark"]`. A file without both is an error.
+ */
+export function tokenBlocks(tokensCss: string): { light: Decls; dark: Decls } {
+ const text = tokensCss.replace(/\/\*[\s\S]*?\*\//g, "");
+ let light: Decls | null = null;
+ let dark: Decls | null = null;
+ for (const m of text.matchAll(/([^{}]+)\{([^{}]*)\}/g)) {
+ const selectors = m[1].split(",").map((s) => s.trim());
+ if (selectors.includes('html[data-base="light"]')) light = declarations(m[2]);
+ else if (selectors.includes('html[data-base="dark"]')) dark = declarations(m[2]);
+ }
+ if (!light || !dark) throw new HistoryProblem("tokens.css has no light or no dark base block");
+ return { light, dark };
+}
+
+// `names` and every custom property their values reach through var(), in
+// the block's own order, as ` name: value;` lines.
+function closure(block: Decls, names: readonly string[], base: string): string {
+ const want = new Set<string>(["color-scheme"]);
+ const visit = (name: string) => {
+ if (want.has(name)) return;
+ const value = block.get(name);
+ if (value === undefined) throw new HistoryProblem(`tokens.css's ${base} block has no ${name}`);
+ want.add(name);
+ for (const ref of value.matchAll(/var\(\s*(--[\w-]+)/g)) visit(ref[1]);
+ };
+ for (const n of names) visit(n);
+ return [...block].filter(([k]) => want.has(k)).map(([k, v]) => ` ${k}: ${v};`).join("\n");
+}
+
+/**
+ * style.css for the history pages, from common/styles/tokens.css: the
+ * homepage's two grounds. Without the theme script (no JS) the pages follow
+ * `prefers-color-scheme`; with it, `html[data-base]` is the visitor's stored
+ * choice, or the homepage's default. The rules are stagit's own stylesheet,
+ * recoloured: links in the accent, diff insertions in --success and deletions
+ * in --destructive (each line also keeps its + or − sign).
+ */
+export function historyStylesheet(tokensCss: string): string {
+ const { light, dark } = tokenBlocks(tokensCss);
+ const lightVars = closure(light, HISTORY_TOKENS, "light");
+ const darkVars = closure(dark, HISTORY_TOKENS, "dark");
+ const indent = (s: string) => s.split("\n").map((l) => ` ${l}`).join("\n");
+ return `/* The source's history pages (stagit), styled by \`archilyzer source publish\`
+ from common/styles/tokens.css — the homepage's two grounds. Generated. */
+:root,
+html[data-base="light"] {
+${lightVars}
+}
+@media (prefers-color-scheme: dark) {
+ :root:not([data-base]) {
+${indent(darkVars)}
+ }
+}
+html[data-base="dark"] {
+${darkVars}
+}
+
+html { background: var(--background); }
+body {
+ margin: 0;
+ padding: 1rem;
+ background: var(--background);
+ color: var(--foreground);
+ font-family: ui-monospace, "IBM Plex Mono", SFMono-Regular, Menlo, Consolas, "Liberation Mono", monospace;
+ font-size: 0.875rem;
+ line-height: 1.5;
+}
+a { color: var(--brand); }
+a:hover { color: var(--foreground); }
+a:not([href]) { color: inherit; text-decoration: none; }
+p.archilyzer-source {
+ margin: 0 0 1rem;
+ font-family: system-ui, -apple-system, "Segoe UI", sans-serif;
+ font-size: 0.8125rem;
+}
+p.archilyzer-source a { color: var(--muted-foreground); text-decoration: none; }
+p.archilyzer-source a:hover { color: var(--foreground); text-decoration: underline; }
+h1, h2, h3, h4, h5, h6 { font-size: 1em; margin: 0; }
+tr.url a { overflow-wrap: anywhere; }
+img, h1, h2 { vertical-align: middle; }
+img { border: 0; }
+a:target { background-color: var(--brand-soft); }
+a.d, a.h, a.i, a.line { text-decoration: none; }
+#blob a { color: var(--faint); }
+#blob a:hover { color: var(--brand); text-decoration: none; }
+table thead td { font-weight: bold; }
+table td { padding: 0 0.4em; }
+#content { overflow-x: auto; }
+#content table td { vertical-align: top; white-space: nowrap; }
+#branches tr:hover td,
+#tags tr:hover td,
+#index tr:hover td,
+#log tr:hover td,
+#files tr:hover td { background-color: var(--surface); }
+#index tr td:nth-child(2),
+#tags tr td:nth-child(3),
+#branches tr td:nth-child(3),
+#log tr td:nth-child(2) { white-space: normal; }
+td.num { text-align: right; }
+.desc { color: var(--muted-foreground); }
+hr { border: 0; border-top: 1px solid var(--border-strong); height: 1px; }
+pre { font-family: inherit; }
+pre a.h { color: var(--info); }
+.A,
+span.i,
+pre a.i { color: var(--success); }
+.D,
+span.d,
+pre a.d { color: var(--destructive); }
+pre a.h:hover,
+pre a.i:hover,
+pre a.d:hover { text-decoration: none; }
+`;
+}
+
+// ── the post-pass ───────────────────────────────────────────────────────────
+
+/**
+ * What is published beside the pages, for the post-pass to link to only what
+ * is there: a path of the raw tree (decoded, as tracked), a commit with a page.
+ */
+export type PublishedSet = {
+ tree: (path: string) => boolean;
+ commit: (sha: string) => boolean;
+};
+
+// stagit's percent-encoding undone; null when it is not valid.
+function decodedPath(p: string): string | null {
+ try {
+ return decodeURIComponent(p);
+ } catch {
+ return null;
+ }
+}
+
+/**
+ * One stagit page, as published. Adds the theme script before `</head>` and
+ * HISTORY_BACK_LINK after `<body>`; points every `href="…file/<path>.html"`
+ * (the Files index, the header's README and LICENSE, a diff's file names) at
+ * the raw tree — `../tree/<path>` from the same depth, the path as stagit
+ * encoded it — and stagit's logo.png and favicon.png at the site's icon.
+ *
+ * With `published`, a link to what is not published loses its `href` and
+ * stays as text (an `<a>` with no `href`, its `id` kept — a diff header is the
+ * diffstat's `#h<n>` target): a file no longer in main (a diff of a deleted or
+ * renamed file), and a commit with no page (past the cap, the oldest page's
+ * parent). Nothing else changes. Page text cannot fake an `href="…"`: stagit
+ * encodes every `"` it prints from the repository as `"`.
+ */
+export function rewriteHistoryPage(html: string, themeScript: string, published?: PublishedSet): string {
+ if (/<\/script/i.test(themeScript)) throw new Error("the theme script may not close its own element");
+ let out = html;
+ const head = out.indexOf("</head>");
+ if (head !== -1) out = `${out.slice(0, head)}<script>${themeScript}</script>\n${out.slice(head)}`;
+ const body = /<body[^>]*>\n?/.exec(out);
+ if (body) {
+ const at = body.index + body[0].length;
+ out = `${out.slice(0, at)}${HISTORY_BACK_LINK}\n${out.slice(at)}`;
+ }
+ out = out
+ .replace(/ href="((?:\.\.\/)*)file\/([^"]*)\.html"/g, (_m, up: string, p: string) => {
+ if (published) {
+ const tracked = decodedPath(p);
+ if (tracked === null || !published.tree(tracked)) return "";
+ }
+ return ` href="${up}../tree/${p}"`;
+ })
+ .replace(/src="(?:\.\.\/)*logo\.png"/g, `src="${SITE_ICON_HREF}"`)
+ .replace(/href="(?:\.\.\/)*favicon\.png"/g, `href="${SITE_ICON_HREF}"`);
+ if (published) {
+ out = out.replace(/ href="((?:\.\.\/)*)commit\/([0-9a-f]{40})\.html"/g, (m, _up: string, sha: string) =>
+ published.commit(sha) ? m : "",
+ );
+ }
+ return out;
+}
+
+/**
+ * The commits the log lists, newest first: every `href="commit/<sha>.html"`
+ * in log.html (stagit writes one per log line; a commit's own text cannot
+ * fake one, since stagit encodes every `"` it prints as `"`). These, and
+ * only these, have their pages published.
+ */
+export function loggedCommits(logHtml: string): string[] {
+ const seen = new Set<string>();
+ for (const m of logHtml.matchAll(/<a href="commit\/([0-9a-f]{40})\.html">/g)) seen.add(m[1]);
+ return [...seen];
+}
+
+/**
+ * Copy the allowlist of stagit's output from `work` into `dest` — the
+ * top-level pages and feeds, and the page of each commit in `commits` (the
+ * ones the log lists; any other page in `work` stays there) — each page
+ * through the post-pass (byte for byte otherwise: read and written as latin1,
+ * so a diff of a file that is not UTF-8 keeps its bytes), then style.css.
+ * The feeds are copied as they are. A listed page that is not there is a
+ * HistoryProblem.
+ */
+export async function stageHistory(
+ work: string,
+ dest: string,
+ o: {
+ themeScript: string;
+ stylesheet: string;
+ commits: readonly string[];
+ // The raw tree's files (as tracked); absent, no tree link is dropped.
+ treeFiles?: ReadonlySet<string>;
+ },
+): Promise<{ files: number; bytes: number; largest: { rel: string; bytes: number } }> {
+ if (/[^\x00-\x7f]/.test(o.themeScript)) throw new Error("the theme script must be ASCII");
+ const rels: string[] = [];
+ for (const f of TOP_FILES) {
+ if (!existsSync(path.join(work, f))) throw new HistoryProblem(`stagit wrote no ${f}`);
+ rels.push(f);
+ }
+ for (const sha of o.commits) {
+ const rel = `commit/${sha}.html`;
+ if (!COMMIT_PAGE.test(`${sha}.html`)) throw new HistoryProblem(`the log names ${sha.slice(0, 40)}, not a commit id`);
+ if (!existsSync(path.join(work, rel))) throw new HistoryProblem(`the log lists ${sha.slice(0, 12)}, whose page is not there`);
+ rels.push(rel);
+ }
+ const pages = new Set(o.commits);
+ const treeFiles = o.treeFiles;
+ const published: PublishedSet | undefined = treeFiles
+ ? { tree: (p) => treeFiles.has(p), commit: (sha) => pages.has(sha) }
+ : undefined;
+ await mkdir(path.join(dest, "commit"), { recursive: true });
+ let bytes = 0;
+ let largest = { rel: "", bytes: -1 };
+ const note = (rel: string, n: number) => {
+ bytes += n;
+ if (n > largest.bytes) largest = { rel, bytes: n };
+ };
+ for (const rel of rels) {
+ const src = path.join(work, rel);
+ const dst = path.join(dest, rel);
+ if (rel.endsWith(".html")) {
+ const text = rewriteHistoryPage((await readFile(src)).toString("latin1"), o.themeScript, published);
+ const buf = Buffer.from(text, "latin1");
+ await writeFile(dst, buf);
+ note(rel, buf.length);
+ } else {
+ await cp(src, dst);
+ note(rel, (await stat(dst)).size);
+ }
+ }
+ await writeFile(path.join(dest, "style.css"), o.stylesheet);
+ note("style.css", Buffer.byteLength(o.stylesheet));
+ return { files: rels.length + 1, bytes, largest };
+}
+
+// ── the render, with its cache ──────────────────────────────────────────────
+
+export type HistoryRun = (
+ command: string,
+ args: string[],
+ o: { cwd: string; timeoutMs: number },
+) => Promise<{ code: number; out: string[] }>;
+
+export type RenderHistoryOpts = {
+ stagit: string;
+ // The scrubbed bare clone — a directory named archilyzer.git (stagit names
+ // the repository after it) — at `head`, with `commits` commits in all.
+ gitDir: string;
+ head: string;
+ commits: number;
+ // The cap: at most this many commits (the newest) get a page and a log
+ // line. SOURCE_HISTORY_MAX_COMMITS, but for the tests.
+ maxCommits: number;
+ // Where the published copy goes (stage/source/git).
+ dest: string;
+ // The publish's own scratch dir: the render's directory when there is no
+ // cache to use.
+ scratch: string;
+ // The kept cache directory, or null (a `--check`, or a cache dir the step
+ // may not use).
+ cacheDir: string | null;
+ // Changes whenever pages rendered before must not be kept.
+ cacheKey: string;
+ // `--force`: render every page again.
+ fresh: boolean;
+ // stagit's header: the description line and the clone URL.
+ description: string;
+ cloneUrl: string;
+ // The Atom feeds' absolute base (`<site>/source/git/`).
+ baseUrl: string;
+ stylesheet: string;
+ themeScript: string;
+ // The raw tree's files, as tracked: a page's link to a file not among them
+ // (deleted or renamed since) becomes text. Absent: every link is kept.
+ treeFiles?: ReadonlySet<string>;
+ // A child, run with the publish's environment and cancel signal.
+ run: HistoryRun;
+ onLog: (line: string) => void;
+};
+
+export type RenderedHistory = {
+ files: number;
+ bytes: number;
+ largest: { rel: string; bytes: number };
+ // The commits with a page — min(commits, maxCommits), the newest.
+ shown: number;
+ // Pages stagit wrote this run (all of them without a usable cache).
+ rendered: number;
+ cached: boolean;
+};
+
+const pidAlive = (pid: number): boolean => {
+ try {
+ process.kill(pid, 0);
+ return true;
+ } catch (err) {
+ return (err as NodeJS.ErrnoException).code === "EPERM";
+ }
+};
+
+/**
+ * A lock this old is stale whoever holds its pid now: a publish holds the
+ * cache for one render (stagit's timeout is 10 minutes), and a pid is reused.
+ */
+export const HISTORY_LOCK_STALE_MS = 60 * 60 * 1000;
+
+/**
+ * Hold the cache directory, or say it is busy. The directory is made on the
+ * way (recursively). The lock names its holder's pid. A lock is stale when
+ * that pid is not running OR the lock is older than HISTORY_LOCK_STALE_MS: a
+ * render was cut off, so nothing in the directory is trusted — it is emptied
+ * and taken, with one line. An I/O error (an unwritable or full cache dir)
+ * is thrown: renderHistory then renders without the cache.
+ */
+export async function holdHistoryCache(
+ dir: string,
+ onLog: (line: string) => void = () => {},
+ now: () => number = Date.now,
+): Promise<boolean> {
+ await mkdir(dir, { recursive: true, mode: 0o700 });
+ const lock = path.join(dir, "lock");
+ for (let attempt = 0; attempt < 2; attempt++) {
+ try {
+ await writeFile(lock, `${process.pid}\n`, { flag: "wx" });
+ return true;
+ } catch (err) {
+ if ((err as NodeJS.ErrnoException).code !== "EEXIST") throw err;
+ const pid = Number((await readFile(lock, "utf8").catch(() => "")).trim());
+ const mtime = (await stat(lock).catch(() => null))?.mtimeMs ?? now();
+ const age = now() - mtime;
+ const running = Number.isInteger(pid) && pid > 0 && pidAlive(pid);
+ if (running && age < HISTORY_LOCK_STALE_MS) return false;
+ onLog(
+ `[source] history: a stale lock on the render cache (pid ${pid > 0 ? pid : "unknown"}, ` +
+ `${running ? `${Math.round(age / 60_000)} minutes old` : "not running"}) was replaced; the cache is rendered afresh`,
+ );
+ await emptyDir(dir);
+ }
+ }
+ return false;
+}
+
+async function emptyDir(dir: string): Promise<void> {
+ for (const f of await readdir(dir).catch(() => [] as string[])) {
+ await rm(path.join(dir, f), { recursive: true, force: true });
+ }
+}
+
+async function releaseHistoryCache(dir: string): Promise<void> {
+ await rm(path.join(dir, "lock"), { force: true });
+}
+
+/**
+ * Remove the cache (a refusal: it was rendered under rules that may not be
+ * today's). Left alone while another publish holds it.
+ */
+export async function dropHistoryCache(dir: string): Promise<void> {
+ if (!existsSync(dir)) return;
+ if (await holdHistoryCache(dir)) await rm(dir, { recursive: true, force: true });
+}
+
+async function countCommitPages(work: string): Promise<number> {
+ const names = await readdir(path.join(work, "commit")).catch(() => [] as string[]);
+ return names.filter((f) => COMMIT_PAGE.test(f)).length;
+}
+
+/**
+ * Render the history into `o.dest`. Throws HistoryProblem when stagit fails
+ * or its output is not what it should be; the caller publishes without the
+ * history then.
+ *
+ * THE CAP. Up to `maxCommits` commits, stagit runs with `-c` (its log lines
+ * cached: a publish renders only the new commits). Past it, with `-l
+ * <maxCommits>`: the log lists the newest `maxCommits` and says how many more
+ * there are ("N more commits remaining, fetch the repository"). stagit refuses
+ * `-c` with `-l`, and `-l` still writes a page for EVERY commit — so what is
+ * published is the page of each commit the log lists, and no other. The
+ * pages already in the cache's directory are kept by stagit either way.
+ */
+export async function renderHistory(o: RenderHistoryOpts): Promise<RenderedHistory> {
+ // stagit's header reads both from the repository directory. Neither is
+ // published as a file (the mirror is staged from an allowlist). No newline
+ // after the description: stagit keeps it, in the <title> and the header.
+ await writeFile(path.join(o.gitDir, "description"), o.description);
+ await writeFile(path.join(o.gitDir, "url"), `${o.cloneUrl}\n`);
+
+ const withoutCache = async (): Promise<RenderedHistory> => ({
+ ...(await renderOnce(o, path.join(o.scratch, "history"), null)),
+ cached: false,
+ });
+ const dir = o.cacheDir;
+ if (!dir) return withoutCache();
+ // A cache that cannot be used (unwritable, read-only, full) is one line and
+ // a render without it: never a failed build.
+ const unusable = (err: unknown) =>
+ o.onLog(`[source] history: the render cache ${tildify(dir)} is unusable (${ioCode(err)}); rendering without it`);
+ let held: boolean;
+ try {
+ held = await holdHistoryCache(dir, o.onLog);
+ } catch (err) {
+ if (!isIoError(err)) throw err;
+ unusable(err);
+ return withoutCache();
+ }
+ if (!held) {
+ o.onLog("[source] history: the render cache is in use by another publish; rendering without it");
+ return withoutCache();
+ }
+ try {
+ const keyFile = path.join(dir, "key.json");
+ const cacheFile = path.join(dir, "stagit.cache");
+ const work = path.join(dir, "out");
+ let usable: boolean;
+ try {
+ usable = !o.fresh && (await cacheUsable(o, keyFile, work));
+ if (!usable) await emptyCache(dir);
+ else await dropStaleLogCache(o, dir, cacheFile);
+ // The key goes before the render and comes back after it: a render cut
+ // off leaves none, and the next publish starts over.
+ await rm(keyFile, { force: true });
+ } catch (err) {
+ if (!isIoError(err)) throw err;
+ unusable(err);
+ return await withoutCache();
+ }
+ try {
+ const r = await renderOnce(o, work, cacheFile);
+ await writeKey(o, keyFile);
+ return { ...r, cached: usable };
+ } catch (err) {
+ if (!(err instanceof HistoryProblem) || !usable) throw err;
+ // Not a fault: stagit's -c walk is in commit-date order and stops at the
+ // head it rendered last, so the commits of a merge that are older than
+ // that head are left out, and the log comes up short. Once more, every
+ // page.
+ o.onLog(
+ `[source] history: the cached render does not cover this head (${err.message}) — stagit's -c stops at the last head it rendered, and a merge of older commits falls behind it; rendering every page again`,
+ );
+ try {
+ await emptyCache(dir);
+ } catch (ioErr) {
+ if (!isIoError(ioErr)) throw ioErr;
+ unusable(ioErr);
+ return await withoutCache();
+ }
+ const r = await renderOnce(o, work, cacheFile);
+ await writeKey(o, keyFile);
+ return { ...r, cached: false };
+ }
+ } catch (err) {
+ await emptyCache(dir).catch(() => {});
+ throw err;
+ } finally {
+ await releaseHistoryCache(dir).catch(() => {});
+ }
+}
+
+// An I/O error from the file system (it carries an errno code), as opposed to
+// the history's own problems, a cancel, or a bug.
+function isIoError(err: unknown): err is NodeJS.ErrnoException {
+ return err instanceof Error && !(err instanceof HistoryProblem) && typeof (err as NodeJS.ErrnoException).code === "string";
+}
+
+function ioCode(err: unknown): string {
+ return (err as NodeJS.ErrnoException)?.code ?? "an I/O error";
+}
+
+// The key, after a render that finished. A key that cannot be written is no
+// key: the next publish renders every page again.
+async function writeKey(o: RenderHistoryOpts, keyFile: string): Promise<void> {
+ try {
+ await writeFile(keyFile, JSON.stringify({ key: o.cacheKey }) + "\n");
+ } catch (err) {
+ if (!isIoError(err)) throw err;
+ o.onLog(`[source] history: the render cache's key could not be written (${ioCode(err)}); the next publish renders every page again`);
+ }
+}
+
+// Everything in the cache but its lock.
+async function emptyCache(dir: string): Promise<void> {
+ for (const f of await readdir(dir).catch(() => [] as string[])) {
+ if (f !== "lock") await rm(path.join(dir, f), { recursive: true, force: true });
+ }
+}
+
+// The cache's pages can be kept: the same key, and a last run that finished.
+// (A page is its commit's, by id; pages of commits no longer in history are
+// never published, since only the log's commits are.)
+async function cacheUsable(o: RenderHistoryOpts, keyFile: string, work: string): Promise<boolean> {
+ let key: unknown = null;
+ try {
+ key = (JSON.parse(await readFile(keyFile, "utf8")) as { key?: unknown }).key;
+ } catch {
+ return false;
+ }
+ return key === o.cacheKey && existsSync(path.join(work, "log.html"));
+}
+
+// stagit's `-c` file holds the log lines down to the commit it names, and
+// stagit appends them to the new ones: when that commit is not an ancestor
+// of today's head (a main rewritten since), the lines are of another history,
+// and the file goes (the pages stay).
+async function dropStaleLogCache(o: RenderHistoryOpts, dir: string, cacheFile: string): Promise<void> {
+ if (!existsSync(cacheFile)) return;
+ const last = (await readFile(cacheFile, "utf8").catch(() => "")).split("\n")[0].trim();
+ const ancestor =
+ /^[0-9a-f]{40}$/.test(last) &&
+ (await o.run("git", ["--git-dir", o.gitDir, "merge-base", "--is-ancestor", last, o.head], {
+ cwd: dir,
+ timeoutMs: 30_000,
+ })).code === 0;
+ if (!ancestor) await rm(cacheFile, { force: true });
+}
+
+async function renderOnce(
+ o: RenderHistoryOpts,
+ work: string,
+ cacheFile: string | null,
+): Promise<Omit<RenderedHistory, "cached">> {
+ try {
+ await mkdir(work, { recursive: true });
+ } catch (err) {
+ if (!isIoError(err)) throw err;
+ throw new HistoryProblem(`the render directory cannot be made (${ioCode(err)})`);
+ }
+ const before = await countCommitPages(work);
+ const capped = o.commits > o.maxCommits;
+ const shown = Math.min(o.commits, o.maxCommits);
+ const args = [
+ ...(capped ? ["-l", String(o.maxCommits)] : cacheFile ? ["-c", cacheFile] : []),
+ "-u",
+ o.baseUrl,
+ o.gitDir,
+ ];
+ const r = await o.run(o.stagit, args, { cwd: work, timeoutMs: 600_000 });
+ // The per-file pages are never published, and stagit writes them all again
+ // on every run: none is kept.
+ await rm(path.join(work, "file"), { recursive: true, force: true }).catch(() => {});
+ if (r.code !== 0) {
+ const tail = r.out.filter((l) => l.trim()).slice(-2).join(" / ");
+ throw new HistoryProblem(`stagit exited ${r.code}${tail ? `: ${tail}` : ""}`);
+ }
+ const logPath = path.join(work, "log.html");
+ if (!existsSync(logPath)) throw new HistoryProblem("stagit wrote no log.html");
+ const commits = loggedCommits((await readFile(logPath)).toString("latin1"));
+ if (commits.length !== shown) {
+ throw new HistoryProblem(`stagit's log lists ${commits.length} commits, not ${shown} (${o.commits} in all, at most ${o.maxCommits})`);
+ }
+ if (commits[0] !== o.head) throw new HistoryProblem(`stagit's log does not start at the head`);
+ const staged = await stageHistory(work, o.dest, {
+ themeScript: o.themeScript,
+ stylesheet: o.stylesheet,
+ commits,
+ treeFiles: o.treeFiles,
+ });
+ const after = await countCommitPages(work);
+ return { ...staged, shown, rendered: cacheFile ? after - before : after };
+}
+
+/**
+ * The cache's key: whatever, changed, makes a page rendered before wrong — the
+ * rules (with the step's version) and filter-repo that made the ids, the
+ * stagit that wrote the pages, and the header text every page carries.
+ */
+export function historyCacheKey(parts: {
+ rulesHash: string;
+ filterRepo: string;
+ stagit: string;
+ description: string;
+ cloneUrl: string;
+ baseUrl: string;
+}): string {
+ return createHash("sha256").update(JSON.stringify(parts)).digest("hex");
+}
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -16,6 +16,7 @@
- **A site can be left off the homepage and the hub.** A site's settings have a new checkbox, **List on the Archilyzer homepage and hub**, on by default (`listed` in `site.json`; only `false` is written). Turned off, the site still builds and deploys at its own URL as before, but the homepage has no card, chart series, `/stats` entry or recent item for it; the hub does not list it as a member, search it, or name it in its `corpus.json` and `llms.txt`; no other site's footer links it; and `channel-sites.json` and the homepage's `stats/` leave it out. A channel only unlisted sites carry is in none of the published totals, the homepage's headline numbers included; a channel a listed site also carries is counted under the listed site. The editor's own pages still show every site. It takes effect at the next homepage, hub and site builds.
- **The sidebar's site picker shows your site from the first paint.** It used to show "All sites" on every page and then jump to the site you had picked, and Dashboard and Channels came up in your site only after a `?site=` had been added to the address. The picked site is now kept in a cookie that the editor reads before it draws a page, so the picker, Dashboard and Channels open in it at once, and the address is left alone. A link that carries `?site=<id>` still opens that page in that site, without changing the one you picked; picking a site on such a page drops the `?site=` from the address. On a site's own pages (Charts, Publish, …) the picker still follows the page, and opening one still makes that site the picked one. The first time you open the editor after updating, a site picked before is moved into the cookie; the picker may show "All sites" for a moment that once. A site picked in one tab reaches the editor's other open tabs without a reload. **New channel** starts with the picked site ticked under its sites (or the site of a `?site=` link), including when it is opened from the editor's own links. Each editor keeps its own pick, as before, when several run on one machine on different ports.
- **The drive check's timings can be changed on `/storage`.** The numbers the editor decides a drive is "not answering" by were fixed: a read may take 3 seconds, each drive is checked every 15 seconds (a check that asks the drive from a separate process waits up to 3 seconds), two clean checks in a row put a drive back in use, and at most four reads wait on one drive at a time. They are now **Drive health timing**, a collapsed block at the foot of `/storage`, with those numbers as the defaults — for when a drive that is busy but working is marked not answering, or a stalled one is not. An empty field is its default; a number outside a field's range is refused, with the range. A save takes effect at once: the next read, the next check, and a new check interval re-times the checks. `settings.json` keeps only the values that differ from a default, under `storage.health` (see `SETTINGS.md`), so an editor that never changes them follows the defaults; `archilyzer index` and the stats build read them too. The messages that said "3 s", "every 15 s" or "twice in a row" now say the numbers in force.
+- **Building the homepage publishes the source's history too: every commit of main with its diff, at `/source/git/`, when stagit is installed.** `archilyzer build homepage`, the `/sites` Homepage jobs and `pnpm ops build-homepage` render the scrubbed mirror with stagit into a log, a page per commit, the refs and two Atom feeds; each page carries the homepage's grounds and one line back to `/source/`, its Files page links into the raw tree, and every page goes through the same gate as the mirror (a denied literal in one refuses the publish). stagit is installed once, outside the repository: `git clone git://git.codemadness.org/stagit && make -C stagit && cp stagit/stagit ~/.local/bin/` (or point `STAGIT_BIN` at it). Without it the build goes on without the history and says so in one line; a render that fails, or pages that would pass the host's limits, are a warning, never a failed build. `archilyzer doctor` shows where stagit is, beside git-filter-repo. The newest 10,000 commits have pages; past that the log says how many more there are. The first build after this change publishes the source again (the step's version is 4). A render cache of about 140 MB is kept in `~/.cache/archilyzer/source-history` (or under `XDG_CACHE_HOME`), so later builds render only the new commits; `archilyzer doctor` shows where it is and its size.
## [0.10.0] - 2026-09-28
- **The homepage can be built and deployed from `/sites`.** Under a new **Homepage** section, after Hub, there is **Build homepage** (tick **Deploy after build** to ship it in the same job, only if the build succeeds) and **Deploy homepage**, which ships the build already in `homepage/out`. A **Preview branch** box beside them sends either deploy to a Cloudflare Pages preview of the `archilyzer` project instead of production, and shows the preview's address as you type; a name Cloudflare would refuse or rewrite, or `main`, greys the deploy buttons out and says why. A line under the buttons says what a deploy would ship: when `homepage/out` was built (or that it holds no build yet), and where it goes, with the live URL. Deploy homepage with nothing built is refused before any job starts. The homepage reads the search index as it stands, so run **Build index** first when its numbers should move. The jobs run the same code as `archilyzer build homepage` / `deploy homepage`, and show on `/jobs` as `build-homepage`, `deploy-homepage` and `build-deploy-homepage`. The Hub section no longer describes the homepage.
diff --git a/homepage/CHANGELOG.md b/homepage/CHANGELOG.md
@@ -1,6 +1,7 @@
# Homepage Changelog
## [Unreleased]
+- **`/source/` links the source's history.** A History block — how many commits (past 10,000, "the latest 10,000 of N"), the newest one (linking to its page), and links to the Log, the Refs and the Atom feed — shows when the build published the history pages (`/source/git/`, rendered by stagit); without them there is no History block. The history pages open on the homepage's ground (the reader's stored choice, else Dark; without JavaScript, the system's), start with one line back to `/source/`, and their Files page is an index into the raw tree. The e2e shows the page with and without a history from a fixture publish (`E2E_SOURCE_PUBLIC_DIR`, never read by a production build), and walks the real pages when the checkout has published them.
- **The growth chart draws its smallest instances together as Other.** Two or more instances that each hold under 5% of the chart's total are one band, **Other**, on top of the stack, in a near-neutral grey of its own (`--chart-other`: 7.36:1 on the Light ground, 3.22:1 on the Dark one, and apart from every instance colour for colour-blind readers); an instance at exactly 5% keeps its band, and a single one under 5% is not grouped. The other instances keep their bands and their colours. The legend lists them and Other; the caption says what Other is and the chart's description names the instances in it; every month's hover title and the Numbers by year table still name every instance. At this release's numbers Hasanalyzer, Rekietalyzer and Jasolyzer are Other. The instance cards and `/stats` are unchanged. The e2e fixture's fifth site transcribes 4 a day rather than 5, so two of its six sites are grouped.
- **An unlisted site is not on the homepage.** A site whose settings turn off **List on the Archilyzer homepage and hub** (`listed: false`) has no Official Instances card, chart series, `/stats` entry or recent item, is not in `channel-sites.json` or `stats/`, and the channels only it carries count in none of the numbers, the headline totals included. The summary's version is 6. The e2e fixture has a seventh, unlisted site that no page names.
- **`/#instances` goes straight to Official Instances.** The section carries `id="instances"`, clear of the sticky header, and every archive's header now links there (`INSTANCES_URL` in `common/lib/project.ts`). With no sites the link lands on the top of the page.
diff --git a/homepage/app/layout.tsx b/homepage/app/layout.tsx
@@ -2,6 +2,7 @@ import type { Metadata, Viewport } from "next";
import { fontVars } from "yt-dlp-transcript-common/styles/fonts";
import { ThemeScript } from "yt-dlp-transcript-common/components/ThemeScript";
import { ThemeProvider } from "yt-dlp-transcript-common/components/ThemeProvider";
+import { HOMEPAGE_DEFAULT_BASE } from "yt-dlp-transcript-common/lib/themeConfig";
import { BASE_GROUNDS, DEFAULT_ACCENT } from "yt-dlp-transcript-common/lib/brand";
import {
PROJECT_NAME,
@@ -73,8 +74,8 @@ export default function RootLayout({
{/* The project's own site opens on the dark base in Signal, the
family's accent. A reader cycles the base with the header's toggle;
the accent is always Signal, whatever this origin's storage holds. */}
- <ThemeScript defaultBase="dark" />
- <ThemeProvider defaultBase="dark" siteAccent={DEFAULT_ACCENT}>
+ <ThemeScript defaultBase={HOMEPAGE_DEFAULT_BASE} />
+ <ThemeProvider defaultBase={HOMEPAGE_DEFAULT_BASE} siteAccent={DEFAULT_ACCENT}>
<Header />
<main className="flex-1 w-full">{children}</main>
<Footer />
diff --git a/homepage/app/lib/headers.test.ts b/homepage/app/lib/headers.test.ts
@@ -101,3 +101,19 @@ test("the raw tree is text, its pages HTML, its binaries their own type — each
assert.match(served(RULES, "/source/tree/a.svg", "image/svg+xml").get("content-security-policy")!, /default-src 'none'/);
assert.equal(served(RULES, "/source/manifest.json", "application/json").get("content-type"), "application/json");
});
+
+test("the history pages (release 15 slice SG) are not indexed, like the raw tree, and keep their own types", () => {
+ const cases: Array<[string, string]> = [
+ ["/source/git/log.html", "text/html"],
+ ["/source/git/commit/0123456789abcdef0123456789abcdef01234567.html", "text/html"],
+ ["/source/git/atom.xml", "application/xml"],
+ ["/source/git/style.css", "text/css"],
+ ];
+ for (const [p, type] of cases) {
+ const h = served(RULES, p, type);
+ assert.equal(h.get("x-robots-tag"), "noindex", p);
+ assert.equal(h.get("content-type"), type, `${p} keeps its type: no /source/tree rule applies`);
+ }
+ assert.equal(served(RULES, "/source/tree/README.md", "text/markdown").get("x-robots-tag"), "noindex");
+ assert.equal(served(RULES, "/source/", "text/html").get("x-robots-tag"), null, "the /source/ page itself is indexed");
+});
diff --git a/homepage/app/lib/source.test.ts b/homepage/app/lib/source.test.ts
@@ -3,7 +3,7 @@ import assert from "node:assert/strict";
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import os from "node:os";
import path from "node:path";
-import { loadSourceManifest } from "./source";
+import { loadSourceManifest, sourcePublicDir } from "./source";
// Run with:
// pnpm --filter homepage test
@@ -34,7 +34,7 @@ const manifest = {
};
let n = 0;
-function pub(opts: { manifest?: unknown; refs?: boolean; tarball?: boolean }): string {
+function pub(opts: { manifest?: unknown; refs?: boolean; tarball?: boolean; log?: boolean }): string {
const p = path.join(TMP, `p${n++}`);
mkdirSync(path.join(p, "source", "archilyzer.git", "info"), { recursive: true });
mkdirSync(path.join(p, "downloads"), { recursive: true });
@@ -46,6 +46,10 @@ function pub(opts: { manifest?: unknown; refs?: boolean; tarball?: boolean }): s
}
if (opts.refs !== false) writeFileSync(path.join(p, "source", "archilyzer.git", "info", "refs"), "x\trefs/heads/main\n");
if (opts.tarball !== false) writeFileSync(path.join(p, "downloads", "archilyzer-source.tar.gz"), "t");
+ if (opts.log) {
+ mkdirSync(path.join(p, "source", "git"), { recursive: true });
+ writeFileSync(path.join(p, "source", "git", "log.html"), "<html></html>");
+ }
return p;
}
@@ -69,3 +73,36 @@ test("malformed, wrong-version or orphaned manifests are the empty state, never
assert.equal(loadSourceManifest(dir), null, what);
}
});
+
+// Release 15 slice SG: the history block is believed only beside its log page.
+const history = {
+ href: "/source/git/log.html",
+ commits: 3,
+ total: 3,
+ head: "2".repeat(40),
+ files: 9,
+ bytes: 1234,
+ sha256: "4".repeat(64),
+ tool: "stagit (sha256 0123456789ab)",
+};
+
+test("a history block beside its log page is believed; without the page the manifest comes back without it", () => {
+ assert.deepEqual(loadSourceManifest(pub({ manifest: { ...manifest, history }, log: true }))?.history, history);
+ const orphan = loadSourceManifest(pub({ manifest: { ...manifest, history } }));
+ assert.equal(orphan?.mirrorHead, "2".repeat(40), "the rest of the manifest stands");
+ assert.ok(orphan && !("history" in orphan), "no History links for pages that are not here");
+ assert.equal(loadSourceManifest(pub({ manifest, log: true }))?.history, undefined, "no block, no History");
+ // A malformed block is a manifest nobody believes (parseSourceManifest).
+ assert.equal(loadSourceManifest(pub({ manifest: { ...manifest, history: { ...history, commits: "3" } }, log: true })), null);
+});
+
+test("the e2e's fixture publish is read only outside a production build, and only when it holds a manifest", () => {
+ const fixture = pub({ manifest, log: true });
+ const empty = path.join(TMP, "empty-fixture");
+ mkdirSync(empty, { recursive: true });
+ const PUBLIC = "/checkout/homepage/public";
+ assert.equal(sourcePublicDir({ NODE_ENV: "development", E2E_SOURCE_PUBLIC_DIR: fixture }, PUBLIC), fixture);
+ assert.equal(sourcePublicDir({ NODE_ENV: "production", E2E_SOURCE_PUBLIC_DIR: fixture }, PUBLIC), PUBLIC);
+ assert.equal(sourcePublicDir({ NODE_ENV: "development", E2E_SOURCE_PUBLIC_DIR: empty }, PUBLIC), PUBLIC, "no manifest: public/");
+ assert.equal(sourcePublicDir({ NODE_ENV: "development" }, PUBLIC), PUBLIC);
+});
diff --git a/homepage/app/lib/source.ts b/homepage/app/lib/source.ts
@@ -1,6 +1,7 @@
import fs from "node:fs";
import path from "node:path";
import {
+ HISTORY_DIR,
MIRROR_DIR,
TARBALL_HREF,
parseSourceManifest,
@@ -15,27 +16,61 @@ import {
//
// Believed only when the files it describes are here too (the snapshot.ts
// rule): a manifest beside a missing mirror or tarball would advertise a
-// clone that fails and a download that 404s. A malformed or wrong-version
-// manifest is null too (parseSourceManifest checks every number the page
-// reads), so a bad file is the empty state, never a crash in `next build`.
-// `pub` is the test's seam.
+// clone that fails and a download that 404s. The same for the history block
+// (release 15 slice SG): without its log page beside it, the manifest is
+// returned WITHOUT it, and the page shows no History links. A malformed or
+// wrong-version manifest is null too (parseSourceManifest checks every number
+// the page reads), so a bad file is the empty state, never a crash in `next
+// build`. `pub` is the test's seam.
//
// The default is a DIRECTORY join on `process.cwd()`, which Turbopack would
// trace as every file under `public/` (the source mirror among them, and it
// grows with every publish): the opt-out keeps it a plain run-time path
// (plans/FACTS.md, "A path joined from `process.cwd()` …").
export function loadSourceManifest(
- pub: string = path.join(/* turbopackIgnore: true */ process.cwd(), "public"),
+ pub: string = sourcePublicDir(
+ process.env,
+ path.join(/* turbopackIgnore: true */ process.cwd(), "public"),
+ ),
): SourceManifest | null {
try {
const manifest = parseSourceManifest(
- JSON.parse(fs.readFileSync(path.join(pub, "source", "manifest.json"), "utf8")),
+ JSON.parse(fs.readFileSync(path.join(/* turbopackIgnore: true */ pub, "source", "manifest.json"), "utf8")),
);
if (!manifest) return null;
- if (!fs.statSync(path.join(pub, "source", MIRROR_DIR, "info", "refs")).isFile()) return null;
- if (!fs.statSync(path.join(pub, TARBALL_HREF.replace(/^\//, ""))).isFile()) return null;
+ if (!fs.statSync(path.join(/* turbopackIgnore: true */ pub, "source", MIRROR_DIR, "info", "refs")).isFile()) return null;
+ if (!fs.statSync(path.join(/* turbopackIgnore: true */ pub, TARBALL_HREF.replace(/^\//, ""))).isFile()) return null;
+ if (manifest.history && !isFile(path.join(/* turbopackIgnore: true */ pub, "source", HISTORY_DIR, "log.html"))) {
+ const without: SourceManifest = { ...manifest };
+ delete without.history;
+ return without;
+ }
return manifest;
} catch {
return null;
}
}
+
+function isFile(p: string): boolean {
+ try {
+ return fs.statSync(p).isFile();
+ } catch {
+ return false;
+ }
+}
+
+// The directory loadSourceManifest reads: `public/`, or the one
+// E2E_SOURCE_PUBLIC_DIR names when it holds a manifest — but ONLY outside a
+// production build. The homepage e2e's `next dev` points it at a fixture
+// publish its spec writes and removes (e2e/fixture-source.ts), so the page's
+// states with and without a history are exercised whatever this checkout has
+// published; with the fixture gone, the page reads `public/` again. `next
+// build` runs with NODE_ENV=production and ignores it (the summary.ts rule).
+export function sourcePublicDir(
+ env: Readonly<Record<string, string | undefined>>,
+ publicDir: string,
+): string {
+ const fixture = env.NODE_ENV !== "production" ? env.E2E_SOURCE_PUBLIC_DIR : undefined;
+ if (fixture && isFile(path.join(/* turbopackIgnore: true */ fixture, "source", "manifest.json"))) return fixture;
+ return publicDir;
+}
diff --git a/homepage/app/source/page.tsx b/homepage/app/source/page.tsx
@@ -1,6 +1,13 @@
import Link from "next/link";
import type { Metadata } from "next";
-import { CLONE_URL, TREE_HREF } from "yt-dlp-transcript-common/lib/sourceManifest";
+import {
+ CLONE_URL,
+ HISTORY_ATOM_HREF,
+ HISTORY_DIR,
+ HISTORY_LOG_HREF,
+ HISTORY_REFS_HREF,
+ TREE_HREF,
+} from "yt-dlp-transcript-common/lib/sourceManifest";
import { PageShell, PageHeading } from "../components/PageShell";
import { Fact } from "../components/Fact";
import { loadSourceManifest } from "../lib/source";
@@ -19,7 +26,8 @@ const linkClass =
// and the tarball are build artefacts, so this page has two states and the
// manifest decides which. A manifest is believed only when the files it
// describes are here too (lib/source.ts): no clone command or link for a file
-// that isn't there.
+// that isn't there. The History block likewise: only when the manifest has a
+// history (stagit rendered it) and its log page is here.
export default function SourcePage() {
const manifest = loadSourceManifest();
const date = manifest
@@ -79,6 +87,51 @@ export default function SourcePage() {
</Fact>
</div>
+ {manifest.history ? (
+ <div data-testid="source-history" className="doc-measure flex flex-col gap-2">
+ <p className="label-machine">History</p>
+ <p className="text-sm leading-[1.7] text-[var(--muted-foreground)]">
+ {manifest.history.total > manifest.history.commits
+ ? "The newest commits of main with their diffs, as static pages: the latest "
+ : "Every commit of main with its diff, as static pages: "}
+ <span data-testid="source-history-commits" className="tabular">
+ {manifest.history.commits.toLocaleString("en-US")}
+ </span>
+ {manifest.history.total > manifest.history.commits ? (
+ <>
+ {" "}of{" "}
+ <span data-testid="source-history-total" className="tabular">
+ {manifest.history.total.toLocaleString("en-US")}
+ </span>
+ </>
+ ) : null}{" "}
+ commit{manifest.history.total === 1 ? "" : "s"}, the newest{" "}
+ <a
+ data-testid="source-history-head"
+ href={`/source/${HISTORY_DIR}/commit/${manifest.history.head}.html`}
+ title={manifest.history.head}
+ className={`tabular ${linkClass}`}
+ >
+ {manifest.history.head.slice(0, 12)}
+ </a>
+ .
+ </p>
+ <p className="text-sm text-[var(--muted-foreground)]">
+ <a data-testid="source-history-log" href={HISTORY_LOG_HREF} className={linkClass}>
+ Log
+ </a>
+ {" · "}
+ <a data-testid="source-history-refs" href={HISTORY_REFS_HREF} className={linkClass}>
+ Refs
+ </a>
+ {" · "}
+ <a data-testid="source-history-atom" href={HISTORY_ATOM_HREF} className={linkClass}>
+ Atom feed
+ </a>
+ </p>
+ </div>
+ ) : null}
+
<div className="doc-measure flex flex-col gap-3">
<p>
<a data-testid="source-tree-link" href={TREE_HREF} className={`text-sm ${linkClass}`}>
diff --git a/homepage/content/docs/faq.md b/homepage/content/docs/faq.md
@@ -82,8 +82,8 @@ On this site, read-only: `git clone https://archilyzer.pages.dev/source/archilyz
It is the main branch with its whole history, regenerated from the private
repository with every deploy — machine paths scrubbed on the way out, which is
why its commit ids differ. There is no GitHub, and nothing takes a push or a
-pull request. [Source](/source/) also has every file to browse, and
-[Downloads](/downloads/) the same tree as a tarball.
+pull request. [Source](/source/) also has every file to browse and the history
+with every commit's diff, and [Downloads](/downloads/) the same tree as a tarball.
## Can I use it for one video?
diff --git a/homepage/content/docs/install.md b/homepage/content/docs/install.md
@@ -39,7 +39,8 @@ cd archilyzer
`git pull` brings you up to date; nothing takes a push. The commit ids differ
from the private repository's, because machine paths are scrubbed on the way
-out. [Source](/source/) has the details and a browsable copy of every file.
+out. [Source](/source/) has the details, a browsable copy of every file, and the
+history with every commit's diff.
No git? The same tree, without history, is a tarball:
diff --git a/homepage/e2e/fixture-source.ts b/homepage/e2e/fixture-source.ts
@@ -0,0 +1,70 @@
+import fs from "node:fs";
+import path from "node:path";
+
+// A FIXTURE PUBLISH for the /source/ page's History block (release 15 slice
+// SG). The page reads the directory E2E_SOURCE_PUBLIC_DIR names instead of
+// public/ while it holds a manifest (app/lib/source.ts sourcePublicDir,
+// honoured only outside a production build), so a spec can show the page with
+// a history and without one, whatever this checkout has published. The spec
+// writes it, and removes it after each test: with it gone, the page — and
+// every other spec — reads public/ again. playwright.config.ts clears it
+// before the server starts, in case a killed run left one.
+//
+// Only the files the loader requires: the manifest, the mirror's info/refs,
+// the tarball, and — with a history — the log page. Nothing here is served:
+// the dev server serves public/, so the History links are checked by their
+// hrefs, and the real pages by source.spec.ts.
+
+export const FIXTURE_SOURCE_NAME = ".e2e-source";
+
+export const FIXTURE_HISTORY = {
+ href: "/source/git/log.html",
+ commits: 1234,
+ total: 1234,
+ head: "c".repeat(40),
+ files: 1240,
+ bytes: 1_000_000,
+ sha256: "d".repeat(64),
+ tool: "stagit (sha256 0123456789ab)",
+};
+
+export const FIXTURE_MIRROR_HEAD = "c".repeat(40);
+
+// `total` over the history's `commits` is a history past the cap: the page
+// says "the latest N of M".
+export function writeFixtureSource(dir: string, o: { history: boolean; total?: number }): void {
+ clearFixtureSource(dir);
+ fs.mkdirSync(path.join(dir, "source", "archilyzer.git", "info"), { recursive: true });
+ fs.mkdirSync(path.join(dir, "downloads"), { recursive: true });
+ fs.writeFileSync(path.join(dir, "source", "archilyzer.git", "info", "refs"), `${FIXTURE_MIRROR_HEAD}\trefs/heads/main\n`);
+ fs.writeFileSync(path.join(dir, "downloads", "archilyzer-source.tar.gz"), "fixture");
+ if (o.history) {
+ fs.mkdirSync(path.join(dir, "source", "git"), { recursive: true });
+ fs.writeFileSync(path.join(dir, "source", "git", "log.html"), "<html></html>\n");
+ }
+ const manifest = {
+ version: 1,
+ generatedAt: "2026-09-30T12:00:00.000Z",
+ branch: "main",
+ sourceCommit: "b".repeat(40),
+ mirrorHead: FIXTURE_MIRROR_HEAD,
+ subject: "a fixture publish",
+ files: 10,
+ bytes: 100,
+ mirror: { files: 9, bytes: 80, packs: 1 },
+ tree: { files: 5, dirs: 2, bytes: 20 },
+ tarball: { href: "/downloads/archilyzer-source.tar.gz", bytes: 7, sha256: "e".repeat(64) },
+ cloneUrl: "https://archilyzer.pages.dev/source/archilyzer.git",
+ treeHref: "/source/tree/",
+ audit: { objects: 3, commits: 1, gitleaks: "skipped" },
+ tools: { git: "2.55.0", filterRepo: "fixture" },
+ ...(o.history ? { history: { ...FIXTURE_HISTORY, total: o.total ?? FIXTURE_HISTORY.total } } : {}),
+ };
+ // Written last, as the publish writes it: the page reads the fixture only
+ // once a manifest is there.
+ fs.writeFileSync(path.join(dir, "source", "manifest.json"), JSON.stringify(manifest, null, 2));
+}
+
+export function clearFixtureSource(dir: string): void {
+ fs.rmSync(dir, { recursive: true, force: true });
+}
diff --git a/homepage/e2e/source-history.spec.ts b/homepage/e2e/source-history.spec.ts
@@ -0,0 +1,58 @@
+import path from "node:path";
+import { test, expect } from "@playwright/test";
+import {
+ FIXTURE_HISTORY,
+ FIXTURE_SOURCE_NAME,
+ clearFixtureSource,
+ writeFixtureSource,
+} from "./fixture-source";
+
+// The /source/ page's History block (release 15 slice SG), in both states,
+// from a FIXTURE publish (e2e/fixture-source.ts): the dev server reads the
+// page's manifest from E2E_SOURCE_PUBLIC_DIR while it holds one
+// (playwright.config.ts, app/lib/source.ts). Each test writes it and removes
+// it, so every other spec reads public/. The history PAGES themselves — what
+// stagit rendered and the publish rewrote — are source.spec.ts's, over the
+// checkout's real publish.
+const FIXTURE = path.resolve(process.cwd(), "e2e", FIXTURE_SOURCE_NAME);
+
+test.describe.configure({ mode: "serial" });
+test.afterEach(() => clearFixtureSource(FIXTURE));
+
+test("with a history in the manifest: the History block names the commits and the head, and links the log, the refs and the feed", async ({ page }) => {
+ writeFixtureSource(FIXTURE, { history: true });
+ await page.goto("/source/");
+ await expect(page.getByTestId("source-mirror-head")).toHaveText(FIXTURE_HISTORY.head);
+ const block = page.getByTestId("source-history");
+ await expect(block).toBeVisible();
+ await expect(block).toContainText("Every commit of main with its diff, as static pages: 1,234 commits, the newest");
+ await expect(page.getByTestId("source-history-commits")).toHaveText("1,234");
+ await expect(page.getByTestId("source-history-total")).toHaveCount(0);
+ // The head, short, linking to its own commit page.
+ await expect(page.getByTestId("source-history-head")).toHaveText(FIXTURE_HISTORY.head.slice(0, 12));
+ await expect(page.getByTestId("source-history-head")).toHaveAttribute("href", `/source/git/commit/${FIXTURE_HISTORY.head}.html`);
+ await expect(page.getByTestId("source-history-log")).toHaveAttribute("href", "/source/git/log.html");
+ await expect(page.getByTestId("source-history-refs")).toHaveAttribute("href", "/source/git/refs.html");
+ await expect(page.getByTestId("source-history-atom")).toHaveAttribute("href", "/source/git/atom.xml");
+ await expect(page.getByTestId("source-history-log")).toHaveText("Log");
+ await expect(page.getByTestId("source-history-atom")).toHaveText("Atom feed");
+});
+
+test("past the cap: the History block says the latest N of M commits", async ({ page }) => {
+ writeFixtureSource(FIXTURE, { history: true, total: 12_345 });
+ await page.goto("/source/");
+ const block = page.getByTestId("source-history");
+ await expect(block).toContainText("The newest commits of main with their diffs, as static pages: the latest 1,234 of 12,345 commits, the newest");
+ await expect(page.getByTestId("source-history-commits")).toHaveText("1,234");
+ await expect(page.getByTestId("source-history-total")).toHaveText("12,345");
+ await expect(page.getByTestId("source-history-log")).toHaveAttribute("href", "/source/git/log.html");
+});
+
+test("without a history in the manifest: no History block, and the rest of the page as before", async ({ page }) => {
+ writeFixtureSource(FIXTURE, { history: false });
+ await page.goto("/source/");
+ await expect(page.getByTestId("source-clone")).toBeVisible();
+ await expect(page.getByTestId("source-tree-link")).toBeVisible();
+ await expect(page.getByTestId("source-history")).toHaveCount(0);
+ await expect(page.getByRole("link", { name: "Atom feed" })).toHaveCount(0);
+});
diff --git a/homepage/e2e/source.spec.ts b/homepage/e2e/source.spec.ts
@@ -106,6 +106,79 @@ test("the raw tree: an index per directory, encoded hrefs that resolve, brackets
expect(font.status()).toBe(200);
});
+// The history pages (release 15 slice SG): stagit's rendering, rewritten by
+// the publish. Run when the checkout's publish has one (a machine with
+// stagit); without, the page must show no History block.
+test("the history pages: the log, a commit, the Files index into the raw tree; the site's line, its theme by stored choice and by the OS; the feed untouched", async ({ page, browser, baseURL }) => {
+ if (!(await published(page))) {
+ await expect(page.getByTestId("source-history")).toHaveCount(0);
+ return;
+ }
+ const manifest = await (await page.request.get("/source/manifest.json")).json();
+ if (!manifest.history) {
+ await expect(page.getByTestId("source-history")).toHaveCount(0);
+ return;
+ }
+ await expect(page.getByTestId("source-history-head")).toHaveText(manifest.history.head.slice(0, 12));
+ const headHref = await page.getByTestId("source-history-head").getAttribute("href");
+ expect(headHref).toBe(`/source/git/commit/${manifest.history.head}.html`);
+ expect((await page.request.get(headHref!)).status()).toBe(200);
+ const shown = (await page.getByTestId("source-history-commits").textContent())!.replace(/\D/g, "");
+ expect(shown).toBe(String(manifest.history.commits));
+ // The log lists exactly the commits with a page (all of them under the cap).
+ const listed = await (await page.request.get("/source/git/log.html")).text();
+ expect(new Set([...listed.matchAll(/<a href="commit\/([0-9a-f]{40})\.html">/g)].map((m) => m[1])).size).toBe(manifest.history.commits);
+
+ // The log, with the one line the publish adds, linking back here, and the
+ // homepage's base (dark) for a reader who has stored none.
+ await page.getByTestId("source-history-log").click();
+ await expect(page).toHaveURL(/\/source\/git\/log\.html$/);
+ const back = page.getByRole("link", { name: "Archilyzer · Source" });
+ await expect(back).toHaveAttribute("href", "/source/");
+ await expect(page.locator("html")).toHaveAttribute("data-base", "dark");
+ await expect(page.locator("html")).toHaveAttribute("data-theme-ready", "1");
+ const bg = () => page.evaluate(() => getComputedStyle(document.body).backgroundColor);
+ expect(await bg()).toBe("rgb(12, 10, 8)");
+ // The stored choice is the homepage's (one origin, one key).
+ await page.evaluate(() => localStorage.setItem("ytdlp-tb:base", "light"));
+ await page.reload();
+ await expect(page.locator("html")).toHaveAttribute("data-base", "light");
+ expect(await bg()).toBe("rgb(243, 246, 247)");
+
+ // The newest commit's page resolves from the log, and its diff's file links
+ // go to the raw tree (no per-file pages are published).
+ const first = page.locator("#log a[href^='commit/']").first();
+ await expect(first).toHaveAttribute("href", `commit/${manifest.history.head}.html`);
+ const commit = await page.request.get(`/source/git/commit/${manifest.history.head}.html`);
+ expect(commit.status()).toBe(200);
+ const commitHtml = await commit.text();
+ expect(commitHtml).toContain('<a href="/source/">Archilyzer · Source</a>');
+ expect(commitHtml).not.toMatch(/href="(?:\.\.\/)*file\//);
+ // Files is an index into the tree: its first link is a raw file, served.
+ await page.goto("/source/git/files.html");
+ const href = await page.locator("#files a").first().getAttribute("href");
+ expect(href).toMatch(/^\.\.\/tree\//);
+ const target = new URL(href!, `${baseURL}/source/git/files.html`).pathname;
+ expect((await page.request.get(target)).status()).toBe(200);
+ expect((await page.request.get("/source/git/refs.html")).status()).toBe(200);
+ // The feeds are not pages: nothing is injected into them.
+ const atom = await (await page.request.get("/source/git/atom.xml")).text();
+ expect(atom).toContain("<feed");
+ expect(atom).toContain(`https://archilyzer.pages.dev/source/git/commit/${manifest.history.head}.html`);
+ expect(atom).not.toContain("archilyzer-source");
+ expect(atom).not.toContain("<script");
+
+ // Without the script (no JS), the OS decides: light, and dark.
+ for (const [scheme, rgb] of [["light", "rgb(243, 246, 247)"], ["dark", "rgb(12, 10, 8)"]] as const) {
+ const ctx = await browser.newContext({ javaScriptEnabled: false, colorScheme: scheme });
+ const p = await ctx.newPage();
+ await p.goto(`${baseURL}/source/git/log.html`);
+ expect(await p.locator("html").getAttribute("data-base"), "the script did not run").toBeNull();
+ expect(await p.evaluate(() => getComputedStyle(document.body).backgroundColor), `no JS, a ${scheme} OS`).toBe(rgb);
+ await ctx.close();
+ }
+});
+
test("the Downloads page points at the mirror for the history", async ({ page }) => {
await page.goto("/downloads/");
await expect(page.getByRole("link", { name: "read-only git mirror" })).toHaveAttribute("href", "/source/");
diff --git a/homepage/playwright.config.ts b/homepage/playwright.config.ts
@@ -11,6 +11,7 @@ import {
FIXTURE_SOCIAL_TRIO,
writeFixtureSettings,
} from "./e2e/fixture-social";
+import { FIXTURE_SOURCE_NAME, clearFixtureSource } from "./e2e/fixture-source";
// Homepage e2e. Runs against `next dev` (default mode) so it reflects uncommitted
// source. Kill any stale dev server on the port between runs.
@@ -38,6 +39,12 @@ import {
// `archilyzer source publish`): e2e/source.spec.ts
// FAILS on the /source/ page's empty state instead
// of accepting it
+// E2E_SOURCE_PUBLIC_DIR set below on the dev server: the directory the
+// /source/ page reads the publish from WHILE it
+// holds a manifest (app/lib/source.ts, ignored by
+// a production build). e2e/source-history.spec.ts
+// writes a fixture publish there and removes it;
+// empty, the page reads public/
const PORT = portFor("HOMEPAGE_E2E_PORT");
const baseURL = `http://localhost:${PORT}`;
@@ -56,6 +63,11 @@ writeFixtureSettings(FIXTURE_SETTINGS, FIXTURE_SOCIAL_TRIO);
const FIXTURE_SITES_DIR = path.resolve(process.cwd(), "e2e", ".e2e-sites");
fs.mkdirSync(FIXTURE_SITES_DIR, { recursive: true });
+// The fixture publish (e2e/fixture-source.ts): none until a spec writes one,
+// so a killed run's leftover never stands in for public/.
+const FIXTURE_SOURCE_DIR = path.resolve(process.cwd(), "e2e", FIXTURE_SOURCE_NAME);
+clearFixtureSource(FIXTURE_SOURCE_DIR);
+
export default defineConfig({
testDir: "./e2e",
timeout: 30_000,
@@ -73,6 +85,7 @@ export default defineConfig({
E2E_HOMEPAGE_SUMMARY_FILE: FIXTURE_SUMMARY,
SETTINGS_FILE: FIXTURE_SETTINGS,
SITES_DIR: FIXTURE_SITES_DIR,
+ E2E_SOURCE_PUBLIC_DIR: FIXTURE_SOURCE_DIR,
},
},
use: {
diff --git a/homepage/public/_headers b/homepage/public/_headers
@@ -47,3 +47,7 @@
! Content-Type
Content-Type: image/svg+xml
Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'
+# The history pages (stagit, sourceHistory.ts) are HTML by extension, and kept
+# out of search indexes like the raw tree (ruled, release 15 slice SG).
+/source/git/*
+ X-Robots-Tag: noindex
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -7773,3 +7773,106 @@ shipped". Anchors are at slice DT's tip (`r15/drive-timings`).
reconcile pass, are not capped); a hand-typed root is capped but never marked; a drive already
stalled at boot before the second counter sample, unless a page reaches it; per-click server
actions; the file route's stream; the corpus disk itself.
+
+## The source's history (stagit) (verified 2026-09-30, branch `r15/stagit`)
+
+Release 15 slice SG ([`release-15.md`](release-15.md)); the operator-facing doc is `PUBLISH.md`, "The
+source mirror (homepage)". Anchors are at the branch.
+
+- **stagit** (codemadness.org, C over libgit2; built from `519cae2f`, "bump version to 1.3",
+ 2026-09-14, against the system libgit2 1.9.7) renders the scrubbed mirror into `/source/git/`.
+ It is operator-installed like git-filter-repo: never vendored. `resolveStagit`
+ (`publish/sourceHistory.ts:122`): `paths.stagitBin` (`STAGIT_BIN`, else `"stagit"`, `lib/paths.ts:314`)
+ — a name with a slash is that file or nothing; a bare name is PATH, then `~/.local/bin/<name>`.
+ It has no version flag: `stagitIdentity` (`:136`) is the first 12 hex of its binary's sha256,
+ never the path (the manifest is published, and a home-dir path is a denied literal).
+- **What stagit does** (read in `stagit.c`): it writes into its CWD; the repository's NAME is its
+ directory's basename minus `.git` — so the step's scratch clone is `<scratch>/archilyzer.git`
+ (`source.ts:834`), no longer `bare`; `description` (read with `fgets`, the newline KEPT — write it
+ without one) and `url` (newline stripped) come from the repository directory; the header links
+ `<relpath>file/README.md.html` and `LICENSE` when HEAD has them, and the logo `<a href="../<relpath>">`
+ lands on `/source/` from every depth; `file/` pages are written for HEAD's whole tree on EVERY run;
+ every text from the repository is xmlencoded (`"` → `"`), so no page text can fake an
+ `href="…"`; `-l N` keeps the NEWEST N log lines (the revwalk is newest first) and ends "M more
+ commits remaining, fetch the repository", but still writes a page for EVERY commit (it only skips
+ the diffstat of a page that exists); `-c` with `-l` is a usage error; a diff's paths are percent-encoded (`/` and `,-.` kept); a commit over 1,000 files or
+ 100,000 added or deleted lines prints "Diff is too large, output suppressed"; `-c <file>` walks from
+ HEAD to the commit the cache file names — in COMMIT-DATE order, so a `--no-ff` merge of commits
+ older than that commit leaves them out (the step's count check then renders every page again,
+ about 8 s: common with parallel slices, and correct) — KEEPS every `commit/<sha>.html` already in
+ the CWD, and appends the cached log lines — so the cache is the whole output directory, and a cache
+ whose commit is not an ancestor duplicates the log. Its temp cache file (`cache.XXXXXXXXXXXX`) is made in the CWD.
+- **The cap** (`SOURCE_HISTORY_MAX_COMMITS`, 10,000; ruled 2026-09-30): up to it, `-c`; past it,
+ `-l 10000`. Either way the pages published are exactly the commits the log links
+ (`loggedCommits`: every `href="commit/<sha>.html"` in log.html), each of which must exist, and the
+ first must be the head. The manifest's `history.total` is every commit, `commits` those with a
+ page; `/source/` says "the latest N of M" when they differ. Past the cap each publish computes N
+ diffstats (no `-c`): 8.75 s at 1,834 commits with every page present, against 0.58 s with `-c`.
+ The oldest published page's parent link is then a 404.
+- **The step** (`source.ts`, step 12b) runs after the object audit and the mirror's staging,
+ before the manifest and the file audit: `stageHistoryPages` (`:1114`) → `renderHistory`
+ (`sourceHistory.ts:573`). **An allowlist is staged** (`TOP_FILES`, `COMMIT_PAGE`, `:94`): `log.html`,
+ `files.html`, `refs.html`, `atom.xml`, `tags.xml`, `commit/<40 hex>.html`, plus `style.css`. Never
+ `file/`, never a leftover. Pages are rewritten byte for byte (latin1 in and out; every injection is
+ ASCII: the back link says `·`).
+- **The post-pass** `rewriteHistoryPage` (`:327`), pages only: `<script>` + the homepage's pre-paint
+ script (`homepageThemeScript`, `:88` = `buildThemeScript({ defaultBase: HOMEPAGE_DEFAULT_BASE })`)
+ before `</head>`; `HISTORY_BACK_LINK` (`:85`) after `<body>`; `href="(../)*file/<p>.html"` →
+ `href="$1../tree/<p>"`; `logo.png`/`favicon.png` → `/icons/icon-32.png` (a route the homepage renders
+ at build). With the published set (`PublishedSet`: the staged raw tree's files, decoded; the log's
+ commits), a link to a file main no longer has (a diff of a deleted or renamed file: 3,342 anchors at
+ 1,882 commits) or to a commit with no page loses its ` href="…"` and keeps its text and its `id`
+ (a diff header is the diffstat's `#h<n>` target); `a:not([href])` is styled as text. At `4dfe21e5`
+ every one of the 32,609 remaining tree links and every commit link resolves. The theme config moved to `lib/themeConfig.ts` for this (publish may not import
+ components/; `components/themeConfig.ts` re-exports it); `HOMEPAGE_DEFAULT_BASE` (`:62`) is what the
+ homepage layout passes. The attribute the script sets is `data-base`, not `data-theme`.
+- **`style.css`** is `historyStylesheet` (`:215`) over `common/styles/tokens.css`: the light block
+ (`:root, html[data-base="light"]`), the dark block for `html[data-base="dark"]` AND for
+ `:root:not([data-base])` under `prefers-color-scheme: dark` (no JS); `HISTORY_TOKENS` and whatever
+ they reach through `var()`. Diff `+` lines are `--success`, `−` lines `--destructive`; there is no
+ chart-safe red in the tokens (the chart palette is blue, green, violet, amber, magenta, rust).
+- **Never a failed build over the history.** No stagit: one line with the install (`source.ts:973`).
+ A render that fails (`HistoryProblem`), a stagit timeout (600 s), unreadable tokens, more files
+ than the step's 15,000 in all, or a page over 24 MiB: a `[source] WARNING: …` and no history. A
+ cancel still cancels; any other error is a crash as before.
+- **The manifest's `history`** (`lib/sourceManifest.ts:88`, parsed at `:140`, all-or-nothing):
+ `href`, `commits` (`git rev-list --count`), `head`, `files`, `bytes`, `sha256` (`historyDigest`,
+ `source.ts:508`: `sourceDigest`'s walk over `source/git/**` alone), `tool`. `sourceDigest` (`:492`)
+ now walks `source/git` too. The state (`.source-publish.json`) gains `stagit` and `history:
+ {files, digest} | null`; the skip needs both to match, and a stagit present with no history never
+ skips. `SOURCE_STEP_VERSION` is 4 (`:109`).
+- **The deploy check** (`publishedSourceProblem`, `:1254`) names the history on its own (`:1336`):
+ `out/source/git`'s digest must be the state's (null when none) and the manifest's.
+- **The render cache** is `${XDG_CACHE_HOME:-~/.cache}/archilyzer/source-history/`
+ (`getPaths().sourceHistoryCacheDir`; an empty `XDG_CACHE_HOME` is unset; ruled 2026-09-30):
+ `key.json`, `stagit.cache`, `out/`, and `lock` while held. `historyCacheFor` (`source.ts`) gives
+ none for `--check` and none, with a line, inside the checkout or the public dir. `holdHistoryCache`
+ (`sourceHistory.ts:508`): the lock names the holder's pid; a live holder with a lock under an hour
+ old (`HISTORY_LOCK_STALE_MS`, `:498`) → render without the cache; a holder not running, or a lock
+ over an hour old → emptied and taken, with one line. An I/O error on the dir, the lock or the key
+ (`isIoError`, `:653`: an errno code — EACCES, EROFS, ENOSPC) → one line and a render without the
+ cache; a key that cannot be written is no key. Every line `renderHistory` logs goes through
+ `maskLiterals` (`source.ts`' `onLog` for it).
+ Its pages are kept when the key (`historyCacheKey`: rules hash, filter-repo, stagit, description,
+ clone URL, base URL) matches and `out/log.html` exists; its `-c` file only when the commit it names
+ is an ancestor of the head (`git merge-base --is-ancestor`), else that file alone goes. A page of a
+ commit no longer in history is never published (only the log's commits are). The key is removed
+ before a render and written after it. A cached render whose log names a page that is not there
+ is rendered again from nothing, once. `--force` renders
+ afresh; `--check` renders in its own scratch; a refusal removes the cache (`source.ts:716`).
+- **Measured (2026-09-30, `main` `6b8aa450` → mirror `8c924ca31701`):** 1,872 commits; 1,878 files,
+ 141,299,532 bytes; the largest `commit/7ddfc955….html`, 5,017,646 bytes; stagit 8 s of a 32 s
+ publish (every page); the whole publish 4,396 files, `homepage/out` 4,576. On the published mirror
+ at 1,834 commits: 8.0 s every page, 2.0 s from the cache with nothing new. The cache is 138 MB.
+ At `main` `4dfe21e5`: 1,882 commits, 1,888 files, 4,409 staged in all. One file per commit: the cap
+ keeps the history at 10,006 files at most; the 15,000-file drop is the last resort.
+- **`_headers`**: `/source/git/*` is `X-Robots-Tag: noindex` (ruled at review, M1), like the raw
+ tree; no other rule matches it, so the pages keep their own types (`headers.test.ts` pins both).
+- **Doctor** (`bin/doctor.ts`): the stagit line ends `; cache: <path>, <size>`. Since the merge of
+ slice DT it reads the settings before the corpus and applies `settings.storage.health`
+ (`applyHealthTimings`) before it inspects any drive, and prints the timings in force on a
+ `drive health` line.
+- **The homepage** believes `history` only beside `source/git/log.html` (`homepage/app/lib/source.ts:43`;
+ without it the manifest comes back without the block). Its e2e reads a fixture publish from
+ `E2E_SOURCE_PUBLIC_DIR` while it holds a manifest (`sourcePublicDir`, `:69`; never in a production
+ build).
diff --git a/plans/release-15.md b/plans/release-15.md
@@ -21,6 +21,7 @@ prompt carries its ruling, and this record carries what was built. Rules:
| UT | `r15/umtool-trace` | umtool's build stops tracing the whole `umtool/` folder | per its prompt |
| SS | `r15/site-scope` | The editor's site picker paints the stored site at once: the selection is a cookie | `editor/app/lib/activeSite{,Server,Actions}.ts` + `activeSite.test.ts`, `editor/app/components/SiteScope{Provider,Select}.tsx`, `editor/app/layout.tsx`, the scope lines of `editor/app/page.tsx` and `editor/app/channels/page.tsx`, a comment in `editor/next.config.ts`, `editor/e2e/site-scope.spec.ts`; records: `plans/FACTS.md` |
| DT | `r15/drive-timings` | The drive-health timings are settings (`settings.storage.health`), edited on `/storage` | new `common/lib/storageHealthTimings.ts` + test; `lib/{storageHealth,storageVolumes,channelMedia,storageLocations,settingsSchema,settingsDocs}.ts`, `controller/storageWatch.ts`, the `index` and `build stats` bins, `SETTINGS.md`; `/storage` (a form, its action and parse), the stall wording on `/channels` and the videos pages; `storage-locations.spec.ts`; records: `plans/FACTS.md` |
+| SG | `r15/stagit` | The source's history and diffs on the homepage, rendered by stagit at `/source/git/` | new `common/publish/sourceHistory.ts` + test, `common/publish/__fixtures__/fakeStagit.ts`; `common/publish/source.ts` (step 12b, the key, the digest, the deploy check) + test; `common/lib/{sourceManifest,paths,envVars}.ts`, `common/lib/themeConfig.ts` (moved from `components/`, which re-exports it); `common/bin/doctor.ts`; `homepage/app/{source/page.tsx,lib/source.ts,layout.tsx}`, `homepage/e2e/{source,source-history}.spec.ts`, `homepage/e2e/fixture-source.ts`, `homepage/playwright.config.ts`; `ENVIRONMENT.md`, `PUBLISH.md`; records: `plans/FACTS.md` |
**Order:** IG → DS. DS adds a health gate inside `inspectChannelMedia`, which IG's hold calls
through its public signature. UT is independent. The shared files are `editor/CHANGELOG.md`'s
@@ -1298,6 +1299,279 @@ collision in the three specs that open `/storage`.
| L3: a refused save clears the typed value (React's form reset) | As recorded in "Found and left" | as built |
| L4: the no-location timeout's words read the budget at throw time, not the one the call ran against | The detail carries `secondsText(budget)`; a test that changes the budget mid-call | `6a24ac4e` |
+### Slice SG, as shipped — the source's history and diffs, rendered by stagit (2026-09-30)
+
+Branch `r15/stagit` off `main` `6b8aa450`, worktree `~/Projects/r12-source-mirror` (block #13:
+editor 4301, test 4311, export 4310, homepage e2e 4340), one Opus implementer. Scratch files
+`sg-*` in the job's `tmp`. `main` did not move during the slice. The ruling (2026-09-30): visitors
+read the commit history and each commit's diff on the homepage; the tool is **stagit**; its
+per-file pages are dropped, and browsing stays the raw tree at `/source/tree/`. No vendor file:
+stagit is operator-installed, like git-filter-repo.
+
+**What shipped.**
+- **Where it runs** — `source.ts` step 12b, after the object audit and the mirror's staging, before
+ the manifest and the file audit. `stagit -c <cache> -u https://archilyzer.pages.dev/source/git/
+ <the scrubbed clone>`, in the render's directory. The scratch clone is now named
+ `archilyzer.git` (stagit names the repository after its directory; it was `bare`). Its
+ `description` is "Archilyzer" (no newline: stagit keeps it) and its `url` is the clone URL. Neither
+ file is published: the mirror is staged from its allowlist.
+- **What is published** — an allowlist of stagit's output: `log.html`, `files.html`, `refs.html`,
+ `atom.xml`, `tags.xml` and `commit/<40 hex>.html`, plus `style.css`. stagit writes `file/`
+ (HEAD's whole tree) on every run; it is deleted where it was rendered and never staged.
+- **The post-pass** (`rewriteHistoryPage`, pure, unit-tested; pages only, never the two feeds):
+ - the homepage's pre-paint theme script before `</head>`. It is the string `ThemeScript` emits:
+ `buildThemeScript({ defaultBase: HOMEPAGE_DEFAULT_BASE })`;
+ - one line after `<body>`: `Archilyzer · Source`, linking to `/source/`;
+ - `href="(../)*file/<path>.html"` → `href="$1../tree/<path>"`. This covers the Files index, the
+ header's README and LICENSE, and both sides of every diff header;
+ - stagit's `logo.png` and `favicon.png` → the site's `/icons/icon-32.png`. Without it, every page
+ has a broken image in its header.
+
+ The pages are rewritten byte for byte: latin1 in and out, and every injection is ASCII.
+- **The theme config moved to `lib/`.** The publish layer may not import `components/`
+ (`architecture.test.ts`), and the script lived in `components/themeConfig.ts`. That file now
+ re-exports `lib/themeConfig.ts`, so its importers are unchanged. `HOMEPAGE_DEFAULT_BASE` ("dark")
+ is new; the homepage layout passes it to `ThemeScript` and `ThemeProvider`.
+- **`style.css`** — `historyStylesheet(tokens.css)`, stagit's own rules recoloured:
+ - the light block, and the dark one for `html[data-base="dark"]`;
+ - the dark one also for `:root:not([data-base])` under `prefers-color-scheme: dark`, the no-JS
+ case;
+ - links in `--brand` (Signal), diff insertions in `--success`, deletions in `--destructive`;
+ - the clone line wraps at phone width.
+
+ Only the tokens the rules read are copied, with what they reach through `var()`.
+- **The gate covers it.** The pages are in the stage the file sweep reads. A denied literal planted in
+ a fake stagit's output refuses the publish, named as `file
+ source/git/commit/<sha>.html (contents, byte N)` and never by its bytes. The refusal withdraws the
+ publish and removes the render cache.
+- **The manifest and the key.**
+ - The manifest gains `history: {href, commits, total, head, files, bytes, sha256, tool}`
+ (`total` since ruling 1, below: `commits` is how many have a page).
+ `sha256` is `historyDigest`, the content digest's walk over `source/git/**`; `tool` is `stagit
+ (sha256 <12>)`, since stagit has no version flag.
+ - `sourceDigest` walks `source/git` too.
+ - The state gains `stagit` and `history: {files, digest} | null`. The skip needs both, and a stagit
+ present with no history never skips.
+ - `publishedSourceProblem` names history pages that are not the audited ones: "homepage/out's
+ history pages (/source/git/) are not the ones that were audited".
+ - `SOURCE_STEP_VERSION` is 4.
+- **The render cache** — ~~`<ARCHILYZER_SOURCE_SCRATCH>/archilyzer-source-history/`~~, since ruling 2
+ `${XDG_CACHE_HOME:-~/.cache}/archilyzer/source-history/` (below): the `-c` file, stagit's output, a
+ key.
+ - It is used only when the key matches (the rules hash, filter-repo, stagit, the header text),
+ the cached commit is an ancestor of the head, and the last run finished (the key is removed
+ before a render).
+ - One holder at a time, by a pid lock. A live holder means render without the cache; a dead one
+ means empty it and take it.
+ - A cached render whose page count is not the commit count renders every page again, once.
+ - `--force` renders afresh, `--check` renders in its own scratch, and a refusal removes the cache.
+- **Without stagit** there is one line: `[source] stagit not found (not on PATH, not in ~/.local/bin)
+ — publishing without the history pages (/source/git/); install it once: git clone
+ git://git.codemadness.org/stagit && make -C stagit && cp stagit/stagit ~/.local/bin/`. The manifest
+ then has no `history`, and `/source/` shows no History links.
+ - A render that fails, a timeout (600 s), unreadable tokens, or pages over the step's limits (15,000
+ files in all, 24 MiB a file) are a `[source] WARNING` and no history. The build never fails over
+ it.
+ - `STAGIT_BIN` (a `paths` variable, `getPaths().stagitBin`) overrides the lookup: stagit on PATH,
+ then `~/.local/bin/stagit`.
+ - `archilyzer doctor` prints `stagit <path>` right after filter-repo; a `STAGIT_BIN` that names
+ nothing is a warn.
+- **`/source/`'s History block**: the commit count, the newest commit (short, linking to its page),
+ and Log · Refs · Atom feed. It shows only when the manifest has a history and `source/git/log.html`
+ is beside it: the loader drops an orphaned block, and a malformed one untrusts the manifest.
+- **Docs.** PUBLISH.md's source-mirror section gains the history. There is no `operate.md` mention
+ of `/source/`; Install and the FAQ each gain half a sentence.
+
+**stagit, installed.** Cloned `git://git.codemadness.org/stagit` into `$T/sg-stagit/src` at
+**`519cae2fcab6f2427dc8f2805d6a5d1eee4bdb05`** ("bump version to 1.3", 2026-09-14). `make` built it
+against libgit2 1.9.7. The binary was copied to `~/.local/bin/stagit` (sha256 `898752011b07…`).
+
+| Commit | What |
+|---|---|
+| `b482f5fc` | `common:` the theme config moves to `lib/`; `components/themeConfig.ts` re-exports it; `HOMEPAGE_DEFAULT_BASE`, used by the homepage layout. |
+| `3cdf4773` | `common:` `sourceHistory.ts` + 11 tests; `source.ts` step 12b, the key, the digest, the deploy check, the withdrawal + 4 tests (and the old ones: `stagit: null`, the `archilyzer.git` scratch name, step ≥ 4); the manifest's `history` + 2 tests; `STAGIT_BIN`; `ENVIRONMENT.md`. |
+| `8394f4ca` | `common:` doctor's stagit line (its test extended). |
+| `17965d58` | `homepage:` the History block; the loader; `E2E_SOURCE_PUBLIC_DIR` (declared, `ENVIRONMENT.md`); `fixture-source.ts`, `source-history.spec.ts` (2), `source.spec.ts` (+1); +2 unit tests. |
+| `cbe0ac3a` | `homepage:` the head as a short link to its page; the clone line wraps. |
+| `3dd5238d` | `docs:` PUBLISH.md, Install, FAQ; the editor and homepage `[Unreleased]` bullets. |
+| this commit | `plans:` this record, the SG row, FACTS "The source's history (stagit)". |
+
+#### Gates (at `3dd5238d`; logs `$T/sg-*.log`)
+
+- **tsc** clean before every commit (`sg-tsc-{1..4}.log`, each `exit=0`).
+- **common 2,318/2,318** (2,301 at `main`, +17: `sourceHistory` 11, `source` +4, `sourceManifest`
+ +2; doctor's test extended), 0 skipped, 77 s. The filter-repo round trips and the real-stagit
+ test RAN.
+- **homepage unit 22/22** (+2). **editor unit 95/95.** **test:scripts 195 + 1 skipped.**
+- `docs env --check` and `docs files --check` both exit 0.
+- **Builds:**
+ - editor `next build` ok, 34 s. It bundles `buildHomepage`'s lazy import; no `.nft.json` names
+ `homepage/public`, `tokens.css` or the cache.
+ - export `next build` ok, 19 s (`export/public` linked from the primary, no dangling link).
+ - homepage `next build` under `systemd-run … MemoryMax=5G` ok, 12 s.
+- **homepage e2e**, `E2E_EXPECT_SOURCE=1`, over this worktree's publish (the queue was free):
+ - `source.spec` + `source-history.spec` + `downloads.spec`: **11 passed, 0 failed, 56 s**. The
+ history test RAN its published branch (10.1 s): the no-JS light and dark, the stored light
+ choice, a commit, the Files link into the tree, the feed.
+ - The full suite: **103 passed, 0 failed, 2.3 min**.
+- **The proof** — `pnpm -s archilyzer build homepage` in the worktree, with the operator's files
+ (read by the step; never printed), `~/.local/bin` on PATH:
+ - exit 0 in 50 s: `[source] history: stagit (sha256 898752011b07) — 1872 commits, 1872 pages
+ rendered; 1878 files, 134.8 MB, the largest git/commit/7ddfc955….html 4.8 MB (8 s)`;
+ - `[source] audit clean: 23,028 objects (1,872 commits), 4,396 staged files against 10 denied
+ literals; gitleaks clean`;
+ - `[source] published main 6b8aa450ad78 as 8c924ca31701: 4396 files, 211.5 MB (mirror 3 packs,
+ tree 414 dirs, history 1872 commits in 1878 files), tarball 7.3 MB sha256 a072630c6c3a (32 s)`.
+ - **`homepage/out/source/git/`:** 1,878 files, 141,299,532 bytes; the largest
+ `commit/7ddfc955….html` at 5,017,646 bytes. `homepage/out` holds 4,576 files, against Pages'
+ 20,000 and 25 MiB.
+ - `grep -rci "$(id -un)"` gives 0 files and 0 hits; the hostname gives 0. No `file/` directory, no
+ `.git` path segment in `out/`.
+ - `publishedSourceProblem` over the worktree's `out/` is null. With one byte added to
+ `out/source/git/log.html`, it gives the history sentence; restored, it is null again.
+ - A second build: `[source] up to date at 6b8aa450ad78; skipping` (17 s).
+ - `source publish --force` gave the same mirror head and tarball sha.
+ - `STAGIT_BIN=/nonexistent/stagit source publish --check`: the one line ("STAGIT_BIN names no
+ executable"), then `check passed … 2518 files, 76.8 MB (…, no history)`.
+ - doctor: `ok stagit ~/.local/bin/stagit`, right after filter-repo.
+- **The cache, measured** on a copy of the published mirror (1,834 commits): 8.0 s for every page,
+ 2.0 s with nothing new. The real cache is 138 MB in `/tmp`, which is tmpfs here.
+- **Numbers tool:** none.
+
+#### Found and left
+
+- ~~**The file limit is months away, not years.**~~ **Ruled and applied below: the history is capped
+ at the newest 10,000 commits**, so it is at most 10,006 files; the 15,000-file drop stays as the
+ last resort.
+- **`/source/git/` has no index page**, so the directory itself is a 404; `/source/` links
+ `log.html`. A copy of `log.html` as `index.html` would double 0.6 MB.
+- ~~**No `_headers` rule for `/source/git/*`.**~~ Since the review (M1, ruled): `X-Robots-Tag:
+ noindex`, like the raw tree. The pages are HTML by extension, and stagit encodes every repository
+ string. No CSP was added, because the inline theme script would need its hash in `_headers` on
+ every change.
+- **The skip does not see a code change to the step** (the post-pass, the stylesheet). The next
+ `main` does, and every merge moves `main`. `--force` does it at once.
+- **The mirror has 1,872 commits against 1,874 on the private main.** filter-repo prunes commits
+ that become empty. This predates the slice.
+
+#### Decisions the operator could overturn
+
+| What I assumed | The alternative |
+|---|---|
+| **Ruled (2026-09-30): stands.** Diff lines use `--success` and `--destructive`, with no new tokens: the chart palette has no red (it is categorical: blue, green, violet, amber, magenta, rust); both are text-grade on both grounds; every changed line keeps its `+`/`−`, so hue is never the only signal. | Two new tokens (`--diff-add`, `--diff-del`), validated as a pair. A green/red pair cannot be told apart by hue for every colour-blind reader, so the sign column carries it either way. |
+| A failed render, a timeout, unreadable tokens or pages over the limits are a WARNING, with no history. **Ruled: the history is capped by commit count instead (below); the file-limit drop stays only as the last resort.** | Refuse the publish: louder, but it takes the mirror down with the history. |
+| **Ruled (2026-09-30): the `components/themeConfig.ts` re-export stands**; its importers move to `lib/themeConfig` when a slice next touches them. | Move the 11 importers now. |
+| stagit's logo and favicon point at `/icons/icon-32.png`. | Leave them broken, or drop the `<img>`. The ruling said "nothing else is injected"; this is a link rewrite, like `file/`. |
+| The History block's head is 12 characters, linking to its commit page. The full sha is the Mirror head fact just above. | The full 40-character sha, as text. |
+| ~~The cache lives under the scratch root, a tmpfs `/tmp` by default here (138 MB of memory).~~ **Ruled (2026-09-30): `${XDG_CACHE_HOME:-~/.cache}/archilyzer/source-history/`** — applied, below. | — |
+| `--force` empties the cache. | Keep it, and force only the publish. |
+| `--check` renders without the cache (8 s rather than 2 s): it writes nothing outside its own scratch. **Ruled: stays cache-less.** | Use the cache. |
+| The e2e shows the page with and without a history from a fixture publish, through a non-production-only `E2E_SOURCE_PUBLIC_DIR`. The real pages are walked by `source.spec` when the checkout has a history. | Only the checkout's real state, as the other source specs do. |
+
+**For the rollout** (the parent's):
+- **Install stagit on the publishing machine**, the same way (`~/.local/bin/stagit`).
+- **Rebuild and restart the editor** before any `/sites` Homepage job; its built bundle runs the
+ old step.
+- The first build publishes the source again (step 4), with the history, and makes the render
+ cache at `~/.cache/archilyzer/source-history` (about 140 MB on disk).
+- A review should read one live page after the deploy; the preview's `_headers` do not touch
+ `/source/git/`.
+
+#### The parent's rulings, applied (2026-09-30), and the merge of `main` with slice DT
+
+| Ruling | What was done |
+|---|---|
+| 1. The history is capped by commit count: stagit `-l <SOURCE_HISTORY_MAX_COMMITS>`, 10,000, the newest first; `/source/` says "the latest N of M"; the 15,000-file drop only as the last resort | `d8c9e5fe`, `4409ab9d`. **Read in `stagit.c` and measured:** `-l N` keeps the NEWEST N log lines (the revwalk is newest first) and ends "M more commits remaining, fetch the repository". But it still writes a page for EVERY commit (only the diffstat of a page that exists is skipped), and stagit refuses `-c` with `-l` (usage error). So: up to the cap, `-c` as before; past it, `-l 10000`. The pages published are, in both, exactly the commits the log links (`loggedCommits`), and every one must be there. The manifest's `history` gains `total` (`commits` is how many have a page). Measured on the published mirror (1,834 commits, pages present): `-l 10000` 8.75 s, `-c` 0.58 s. Past the cap every publish computes 10,000 diffstats (~5 ms each, ~48 s). |
+| 2. The render cache is `~/.cache/archilyzer/source-history/` (`XDG_CACHE_HOME` honoured), made with a recursive mkdir, never inside the repo; `--check` stays cache-less; PUBLISH.md and doctor ("cache: <path>, <size>") | `d8c9e5fe`, `68670703`, `2acbd080`. `getPaths().sourceHistoryCacheDir`, from `XDG_CACHE_HOME` (empty = unset) else `~/.cache` (`XDG_CACHE_HOME` declared, `ENVIRONMENT.md`). `historyCacheFor` gives none for `--check` and none — with a line — inside the checkout or the public dir. Its pages are kept by key; its `-c` log lines only when they end at an ancestor (the pages stay otherwise). Doctor's stagit line ends `; cache: <path>, <size>` ("none yet"). The old `/tmp` cache was removed. |
+| 3. `--success` / `--destructive` stand | Recorded (the decisions table). |
+| 4. The `components/themeConfig.ts` re-export stands; importers move when a slice next touches them | Recorded (the decisions table). |
+| After DT merged: merge `main`; doctor's `applyHealthTimings(...)` line (DT's Found and left) | `2b2b5305` merges `main` `4dfe21e5`; the conflicts were `release-15.md` (DT's row, then SG's; DT's section, then SG's, before the Rollout) and the editor changelog (DT's bullet, then SG's). `53c1849d`: doctor reads the settings before the corpus and applies `settings.storage.health` before it inspects any drive, as the index and stats bins do; a "drive health" line in the corpus block names the timings in force and whether they are the defaults. |
+
+| Commit | What |
+|---|---|
+| `d8c9e5fe` | `common:` the cap (`SOURCE_HISTORY_MAX_COMMITS`, `-l`, `loggedCommits`, `history.total`); the cache in the XDG cache dir (`sourceHistoryCacheDir`, `historyCacheFor`, `XDG_CACHE_HOME`); one fake stagit (`__fixtures__/fakeStagit.ts`, `-c` and `-l` as `stagit.c` has them); tests +4 (the cap twice, `loggedCommits`, the cache's place) and the cache test rewritten. |
+| `68670703` | `common:` doctor's stagit line ends with the cache's path and size. |
+| `4409ab9d` | `homepage:` "the latest N of M commits"; the fixture can be capped (+1 e2e); `source.spec` checks the log lists exactly the commits with a page. |
+| `2acbd080` | `docs:` PUBLISH.md — the cap, the cache's place, doctor's line. |
+| `44ddb6b2` | `changelog:` the cap and the cache's place in both `[Unreleased]` bullets. |
+| `2b2b5305` | Merge `main` (DT). |
+| `53c1849d` | `common:` doctor applies the drive-health timings, and prints them (+1 test). |
+| this commit | `plans:` this subsection, the ruled rows, FACTS. |
+
+**Re-gates** (after the merge, at `53c1849d`; logs `$T/sg-*-2.log`, `sg-tsc-{5,6,7}.log`, `sg-pub2.log`):
+- **tsc** clean before each commit and on the merge.
+- **common 2,341/2,341**, 0 skipped: SG's 22 over `main` `4dfe21e5`'s (2,319 by difference; `sourceHistory` 13,
+ `source` +6, `sourceManifest` +2, doctor +1).
+- **homepage unit 22/22**; **test:scripts 195 + 1 skipped**; `docs env --check` 0,
+ `docs files --check` 0.
+- **The three source specs** (`source`, `source-history`, `downloads`; `E2E_EXPECT_SOURCE=1`, over
+ the worktree's fresh publish): **12 passed, 0 failed, 17.5 s** (+1: the capped block).
+- **`source publish`** (main `4dfe21e5`) in the worktree: `[source] history: stagit (sha256
+ 898752011b07) — 1882 commits, 1882 pages rendered; 1888 files, 135.3 MB, the largest
+ git/commit/7ddfc955….html 4.8 MB`; `audit clean: 23,128 objects (1,882 commits), 4,409 staged
+ files against 10 denied literals; gitleaks clean`; published as `f81edb46f7e3`, 4,409 files. The
+ cache was made at `~/.cache/archilyzer/source-history` (138 MB, mode 700).
+- **`source publish --check`**: `check passed — would publish main 4dfe21e5714a as f81edb46f7e3: 4409
+ files, 210.3 MB (…, history 1882 commits in 1888 files) …; nothing written (32 s)`. The cache's
+ key was untouched.
+- **doctor**: `ok stagit ~/.local/bin/stagit; cache: ~/.cache/archilyzer/source-history, 134.1 MB`
+ and `-- drive health a read may take 3 s, a check every 15 s (3 s each), a stall clears on a
+ clean check twice in a row, 4 reads in flight per drive — the defaults`.
+- Not re-run after the rulings: the full homepage suite (103 before), the builds. The rulings change
+ `source.ts`, `sourceHistory.ts`, doctor and one page component; the source specs cover the page.
+
+**Found and left, from the rulings:**
+- ~~**Past the cap, the oldest published commit page links its parent's page, which is not
+ published** (a 404). Every other link resolves.~~ Corrected at review (L4): 3,327 links to files
+ main no longer has were 404s too. Since the fix, a link to what is not published is text (below).
+- **Past the cap the render is not incremental**: stagit cannot combine `-l` with `-c`, so each
+ publish computes 10,000 diffstats (about 48 s by the per-commit rate measured here). The pages
+ themselves are still kept.
+
+#### Review: SHIP AFTER FIXES, and the fixes (2026-09-30)
+
+The review (`$T/sg-review.md`, at `10ae765b`) found no High, one Medium and eight Lows; the gate needs
+no change. The parent ruled on each; L7 (the live check greps the live history pages) is the
+parent's, for the runbook.
+
+| Finding | Ruling | Where |
+|---|---|---|
+| M1: the history pages are indexable; the raw tree is `noindex` | **Ruled: `noindex`** (the alternative, indexed, was not taken) | `dd5d03ee`: `_headers` `/source/git/*` `X-Robots-Tag: noindex`; `headers.test.ts` pins it and the pages' own types; PUBLISH.md `2f02cc7e` |
+| L1: the retry line quoted stagit's words unmasked (the home dir in its argv) | Mask | `8a960a0b`: `renderHistory`'s `onLog` masks every line; the cache-location line too; a test plants a literal in the cache's path |
+| L2: a cache I/O error failed the build | One line, render without the cache | `8a960a0b`: EACCES/EROFS/ENOSPC on the dir, the lock or the key → `the render cache <dir> is unusable (<code>); rendering without it`; a key that cannot be written is no key; tests with a read-only cache dir and an unmakeable one |
+| L3: stagit's `-c` misses a `--no-ff` merge of older commits | Say so | `8a960a0b`: the retry line says it ("stagit's -c stops at the last head it rendered, and a merge of older commits falls behind it"); a test merges two commits dated 2001; PUBLISH.md and FACTS |
+| L4: 3,327 tree links were 404s; the record said every link resolves | Unlink | `8a960a0b`: a link to a file not in the staged raw tree, or to a commit with no page, keeps its text and loses its `href` (a diff header keeps its `id`, the diffstat's target); the real render has 3,342 such anchors and 32,609 tree links, **every one resolving**, and every commit link; tests with a deleted file and past the cap (the parent) |
+| L5: stale records | Fix | this commit (the cache path, `total`, the link sentence); PUBLISH.md's timings say which is stagit's and which the step's (`2f02cc7e`); `source publish`'s usage names the history (`8a960a0b`) |
+| L6: the file-count drop was untested | Test | `8a960a0b`: a `historyFileLimit` seam; the test drops nine history files with the WARNING, and publishes the rest |
+| L8: a reused pid could hold the lock forever | Stale by pid or age | `8a960a0b`: a lock whose pid is not running, or over an hour old (`HISTORY_LOCK_STALE_MS`), is replaced with one line |
+| Re-review (SHIP): a withdrawal could throw over the render cache (`dropHistoryCache` on a read-only cache) and lose the refusal's exit | Catch it | `2aa0d7c3`: one masked line ("the history's render cache <dir> could not be removed (<code>); remove it by hand"); the exit 1 and the withdrawal stand; a test refuses with a read-only cache dir (`source` 22 tests, tsc clean) |
+
+| Commit | What |
+|---|---|
+| `dd5d03ee` | `homepage:` `/source/git/*` noindex; `headers.test.ts` +1. |
+| `8a960a0b` | `common:` L1, L2, L3, L4, L5 (usage), L6, L8; `sourceHistory` +4 tests, `source` +2. |
+| `2f02cc7e` | `docs:` PUBLISH.md. |
+| this commit | `plans:` this subsection, the corrections, FACTS. |
+
+**Re-gates** (at `2f02cc7e`; logs `$T/sg-*-3.log`, `sg-tsc-8.log`, `sg-pub3.log`, `sg-capped.log`):
+- **tsc** clean before each commit.
+- **common 2,347/2,347**, 0 skipped (+6: `sourceHistory` 17, `source` 21).
+- **homepage unit 23/23** (+1); **test:scripts 195 + 1 skipped**; `docs env --check` 0, `docs files
+ --check` 0.
+- **The three source specs** (`E2E_EXPECT_SOURCE=1`, over a fresh `--force` publish): **12 passed,
+ 0 failed, 17.2 s**.
+- **`source publish --force`**: `history: … 1882 commits, 1882 pages rendered; 1888 files, 135.1 MB`;
+ `audit clean: 23,128 objects (1,882 commits), 4,411 staged files …; gitleaks clean`; published as
+ `f81edb46f7e3`. Over the rendered pages: 32,609 tree links, 0 that do not resolve; 0 commit links to
+ an unpublished page; `grep -rli` of the user name and of the hostname: 0 files each.
+- **`source publish --check`**: `check passed — would publish main 4dfe21e5714a as f81edb46f7e3: 4411
+ files, 212.5 MB (…, history 1882 commits in 1888 files) …; nothing written (32 s)`.
+- **Capped builds with the corpus linked** (`transcripts/` moved aside, `ln -sT` to the primary's,
+ restored after; `systemd-run … MemoryMax=5G`): editor `next build` **ok, 34 s**; homepage `next
+ build` **ok, 13 s**. No editor `.nft.json` names `transcripts/channels`, `homepage/public/source` or
+ the cache.
+
+
## Rollout
Release 15 is slices IG (`r15/index-hold`, merged `ccf90892`), UT (`r15/umtool-trace`, `07c991be`),
