commit a527d3690a854b3292068c57786534f5c4488f71
parent a62cf6a859873f94dc5bc1d4c7e3998f4b8adf55
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date: Tue, 7 Jul 2026 12:59:12 -0400
Docker export build: parallel per-site container builds + serial deploy
Make buildPipeline.mode="docker" real. The export build has two natures the
single-process pipeline conflated: shared corpus-scale work (single-writer LMDB
index, staging, archive zips) and per-site work (compose + next build) that
collides on the fixed export/{public,out,.next}. Split them:
- Phase A (host, once): build:data, then a new build:archives step that warms
the shared archive cache for the union of all sites' channels — hoisting
archive generation out of per-site compose so parallel containers can't race
the shared cache. archive{Transcripts,LiveChat} gain a readOnly mode; compose
honors ARCHIVES_READONLY to materialize-only from the warmed cache.
- Phase B (parallel containers, capped by maxParallelBuilds): each site builds
in Dockerfile.build, read-only over the corpus/index/staging/archive cache,
writing an isolated out/ under export/.export-builds/<siteId>/ as the host uid.
- Phase C (host, serial): deploy each built site (R2 upload + wrangler),
tolerant of a single site failing.
UI: a "Build all sites" control on the Deploy page (build-only or build+deploy);
falls back to a serial host build+deploy when no container engine is available.
settings.json and the entrypoint are mounted fresh (not baked stale); outputs
are host-owned (-u). See DEPLOY_DOCKER.md.
Verified on the real corpus incl. a two-site concurrent fan-out: isolated
per-site outputs, shared cache unchanged (no write races), host-owned out/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Diffstat:
17 files changed, 951 insertions(+), 42 deletions(-)
diff --git a/.dockerignore b/.dockerignore
@@ -8,6 +8,9 @@
editor/test-transcripts/
editor/test-settings.json
+# The real settings.json is mounted fresh into build containers (SETTINGS_FILE) —
+# baking it would freeze build config (archive storage, size caps) at image time.
+settings.json
transcripts/
export/public/summaries/
@@ -17,6 +20,12 @@ export/public/code.tar.xz
export/public/transcripts.tar.gz
export/public/transcripts.tar.xz
+# Build staging / caches / per-site outputs — bind-mounted at run time, never baked.
+export/.export-index/
+export/.export-builds/
+export/.compose-cache/
+export/.r2-staging/
+
.git
.github
.claude
@@ -26,4 +35,5 @@ export/public/transcripts.tar.xz
.DS_Store
Dockerfile.test
+Dockerfile.build
.dockerignore
diff --git a/.gitignore b/.gitignore
@@ -72,6 +72,8 @@ yarn-error.log*
/export/.export-index/
# per-site incremental-compose signatures (skip-unchanged cache)
/export/.compose-cache/
+# per-site docker build outputs (out/, .next, public/, caches) — one dir per site
+/export/.export-builds/
/transcripts/index.mdb/
# homepage (hub) generated public data (regenerate with `pnpm build:homepage`)
diff --git a/DEPLOY_DOCKER.md b/DEPLOY_DOCKER.md
@@ -0,0 +1,76 @@
+# Docker export build pipeline
+
+Docker build mode builds **every site in parallel** in isolated containers, then
+deploys them serially — a large speedup when you host several sites, and stronger
+isolation than the basic single-process build. This is opt-in: set **Build
+pipeline → Docker** in Settings (or the toggle on the Deploy page). Basic mode is
+unchanged and remains the default.
+
+## Prerequisites
+
+- A container engine: **Docker**, or **podman** (set `DOCKER_BIN=podman`). Rootless
+ podman is a good fit — it maps container files to your host user automatically.
+- The editor host still needs Node + pnpm (Phase A and deploy run on the host) and
+ your Cloudflare/R2 credentials in the environment (see
+ [DEPLOY_CLOUDFLARE.md](./DEPLOY_CLOUDFLARE.md)). Credentials are **never** passed
+ into a container — deploy runs on the host.
+
+The build image is built (and cached) automatically from `Dockerfile.build` the
+first time you run; edit **Build image** / **Dockerfile** in Settings to override
+the tag/path.
+
+## How it works
+
+Trigger it with **Build all sites** on the Deploy page. One managed job runs three
+ordered phases:
+
+1. **Phase A — shared, on the host, once.** `build:data` (search index +
+ `.export-index` staging) then `build:archives` (warm the shared archive-zip
+ cache for the union of all sites' channels). Only the host writes this shared
+ state, so containers never race it. This phase is serial and is the long pole on
+ a cold build; on a warm rebuild it's near-instant (unchanged channels are
+ skipped).
+2. **Phase B — per-site, in parallel containers.** Each site's `compose:site +
+ next build` runs in its own container, capped by **Max parallel builds**. Each
+ writes an isolated `out/` under `export/.export-builds/<siteId>/`. Containers
+ mount the corpus/index/staging/archive cache **read-only**. (Network is left on:
+ `next build` fetches the site's fonts from Google via `next/font/google`;
+ isolation comes from the read-only mounts, per-site output dir, and non-root
+ user.)
+3. **Phase C — deploy, on the host, serially.** After every build finishes, each
+ built site is deployed in turn (oversize-archive R2 upload, then `wrangler pages
+ deploy`). A single site failing to build or deploy is reported and skipped; the
+ rest still ship.
+
+If no container engine is available, the action logs a notice and falls back to a
+serial host build+deploy (one site at a time).
+
+## Mounts (per Phase-B container)
+
+| Host | Container | Mode |
+|---|---|---|
+| `transcripts/` (corpus + `index.mdb` + archive cache) | `/data/transcripts` | ro |
+| `export/.export-index` (shared + per-site staging) | `/data/export/.export-index` | ro |
+| `export/.export-builds/<siteId>` (public/out/.next/caches) | `/site` | rw |
+| `settings.json` (build config, mounted fresh — not baked) | `/data/settings.json` | ro |
+
+The per-site `/site` mount is persistent, so incremental `next build` (`.next`) and
+incremental compose (`.compose-cache`) stay warm across builds.
+
+## Tuning & environment
+
+- **Max parallel builds** (setting) — how many site containers run at once. Each
+ `next build` can use up to ~8 GB; a safe starting point is `floor(RAM_GB / 9)`.
+- `DOCKER_BIN` — container binary (default `docker`; e.g. `podman`).
+- `DOCKER_BUILD_MEMORY`, `DOCKER_BUILD_CPUS` — optional per-container `--memory` /
+ `--cpus` caps so a fan-out can't OOM/peg the host.
+- `BUILD_ARCHIVES=0` (or the **Skip archive zips** checkbox) — skip the archive
+ warm + per-site archive materialize for a faster build with no download bundles.
+
+## Notes
+
+- Containers run as your host uid/gid (`-u`), so files under `.export-builds/` are
+ host-owned, not root-owned.
+- The image bakes the repo source + deps; a code change rebuilds it, but Docker
+ layer caching keeps that cheap (deps re-install only when the lockfile moves).
+- `.export-builds/` is gitignored and excluded from the image build context.
diff --git a/Dockerfile.build b/Dockerfile.build
@@ -0,0 +1,44 @@
+# Build image for the docker export pipeline (buildPipeline.mode = "docker").
+#
+# Bakes the repo source + installed deps so each per-site build container is
+# hermetic and reproducible. The corpus, the shared LMDB index, the .export-index
+# staging, and the archive cache are bind-mounted READ-ONLY at run time — never
+# baked (they're hundreds of GB and change constantly). Each container writes only
+# its per-site output mount (/site). Deploy never runs here; it stays on the host.
+#
+# Entry: docker/build-site.sh runs `compose:site + next build` for one SITE_ID.
+FROM node:20-bookworm-slim
+
+# Archive compressors the export build shells out to. `zip` is the default format;
+# `tar`/`xz`/`gzip` cover the other configurable archive formats.
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends zip tar xz-utils gzip \
+ && rm -rf /var/lib/apt/lists/*
+
+# Install pnpm as a plain global binary (NOT corepack): the fan-out containers run
+# fully offline (--network=none) as an arbitrary host uid with HOME=/tmp, where
+# corepack — lacking a packageManager pin — would try to fetch pnpm from the
+# registry and fail. A global install needs no network at run time.
+RUN npm install -g pnpm@9.15.4
+
+WORKDIR /repo
+
+# Install deps first for layer caching — rebuilds only when a manifest or the
+# lockfile moves. `onlyBuiltDependencies` in pnpm-workspace.yaml rebuilds the
+# native modules (lmdb, msgpackr-extract, esbuild).
+COPY pnpm-lock.yaml pnpm-workspace.yaml package.json ./
+COPY common/package.json common/package.json
+COPY export/package.json export/package.json
+RUN pnpm install --frozen-lockfile
+
+# Bake source last so a code change only re-runs from here.
+COPY . .
+
+# Containers run with `-u <host-uid>` (so /site outputs are host-owned, not root).
+# Next writes a couple of fixed-location files into the export package dir
+# (next-env.d.ts, tsconfig.tsbuildinfo) and the entrypoint symlinks .next/out from
+# there — so that one dir must be writable by an arbitrary runtime uid. The image
+# is ephemeral and isolated (--network=none), so widening it here is harmless.
+RUN chmod -R a+rwX /repo/export
+
+ENTRYPOINT ["bash", "docker/build-site.sh"]
diff --git a/common/bin/build-archives.ts b/common/bin/build-archives.ts
@@ -0,0 +1,91 @@
+#!/usr/bin/env tsx
+// Warm the shared, persistent archive cache (transcripts/export/archives) for the
+// UNION of every enabled site's member channels, in a single pass, on the host.
+//
+// This hoists archive-zip *generation* out of the per-site compose. Under the
+// docker build pipeline the per-site compose runs in parallel containers over a
+// read-only mount of this cache (ARCHIVES_READONLY=1) — so generation must happen
+// once, up front, or two sites sharing a cold channel would race to write the same
+// <slug>.zip. Running it here (Phase A, serial, before the fan-out) makes every
+// container's compose a pure materialize.
+//
+// It's signature-gated (see channelSignature.ts) exactly like the compose path,
+// so an unchanged channel is reused, not re-zipped — this is a no-op on a warm
+// cache and also speeds the basic (non-docker) build by collapsing N per-site
+// passes into one union pass.
+import { getPaths } from "../lib/paths";
+import { getSettings } from "../lib/settings";
+import { listSites } from "../lib/site";
+import { openChannelSigner } from "../lib/channelSignature";
+import {
+ archiveCacheDir,
+ archiveTranscripts,
+} from "../controller/archiveTranscripts";
+import { archiveLiveChat } from "../controller/archiveLiveChat";
+
+async function main(): Promise<void> {
+ // Honor the same global/per-build opt-outs the per-site compose respects
+ // (archivesEnabled in compose-site.ts). The per-site `archives` flag is applied
+ // below when building the union.
+ if (getSettings().buildArchives === false) {
+ console.log("[archives] disabled globally (buildArchives=false) — nothing to warm.");
+ return;
+ }
+ if (process.env.BUILD_ARCHIVES === "0") {
+ console.log("[archives] BUILD_ARCHIVES=0 — skipping archive cache warm.");
+ return;
+ }
+
+ const paths = getPaths();
+ const sites = listSites(paths);
+
+ // Union of member slugs across sites that ship archives. A channel shared by
+ // several sites is warmed once; the readonly per-site compose then materializes
+ // its subset.
+ const union = new Set<string>();
+ for (const site of sites) {
+ if (site.archives === false) continue;
+ for (const c of site.channels) union.add(c.slug);
+ }
+ const channelSlugs = [...union].sort();
+ if (channelSlugs.length === 0) {
+ console.log("[archives] no archive-enabled member channels — nothing to warm.");
+ return;
+ }
+
+ console.log(
+ `[archives] warming shared cache for ${channelSlugs.length} channel(s) across ${sites.length} site(s)…`,
+ );
+
+ // Match the per-site compose's build options EXACTLY so the sidecar signatures
+ // line up and the readonly compose sees cache hits (compose-site.ts uses
+ // `{ format: "zip" }`, i.e. the resolved defaults).
+ const build = { format: "zip" as const };
+ const channelConcurrency = Number(process.env.ARCHIVE_CHANNEL_CONCURRENCY) || 4;
+ const log = (m: string) => {
+ if (m) console.log(`[archives] ${m}`);
+ };
+
+ const signer = openChannelSigner(paths);
+ try {
+ const common = {
+ paths,
+ channelSlugs,
+ outDir: archiveCacheDir(paths),
+ build,
+ channelConcurrency,
+ onLog: log,
+ signer,
+ };
+ await archiveTranscripts(common);
+ await archiveLiveChat(common);
+ } finally {
+ await signer.close();
+ }
+ console.log("[archives] cache warm complete.");
+}
+
+main().catch((err) => {
+ console.error(err);
+ process.exit(1);
+});
diff --git a/common/bin/compose-site.ts b/common/bin/compose-site.ts
@@ -344,8 +344,16 @@ async function composeArchives(
// another site this run) is not re-zipped. One signer view is shared across the
// transcripts + live-chat passes. compose then copies this site's member subset
// out of the cache into public/archives below.
+ //
+ // ARCHIVES_READONLY (set by the docker per-site build container): the cache was
+ // already warmed on the host by `build:archives`, and here it's a read-only
+ // mount — so never generate or open the index, just materialize what's cached.
+ // This is what keeps parallel per-site containers from racing writes to the one
+ // shared cache.
const cacheDir = archiveCacheDir(paths);
- const signer = openChannelSigner(paths);
+ const readOnly = process.env.ARCHIVES_READONLY === "1";
+ if (readOnly) log("read-only cache mode — materializing pre-built archives");
+ const signer = readOnly ? undefined : openChannelSigner(paths);
let transcripts: Awaited<ReturnType<typeof archiveTranscripts>>;
let liveChat: Awaited<ReturnType<typeof archiveLiveChat>>;
try {
@@ -355,6 +363,7 @@ async function composeArchives(
outDir: cacheDir,
build,
onLog: log,
+ readOnly,
signer,
};
// Per-channel archives only — the combined "whole-site" bundles were dropped
@@ -363,7 +372,7 @@ async function composeArchives(
transcripts = await archiveTranscripts(common);
liveChat = await archiveLiveChat(common);
} finally {
- await signer.close();
+ if (signer) await signer.close();
}
// Materialize each cached archive into this site's served tree. Hard-link when
diff --git a/common/controller/archiveLiveChat.ts b/common/controller/archiveLiveChat.ts
@@ -70,6 +70,9 @@ export type ArchiveLiveChatOptions = {
// Reuse an already-open per-channel signer across passes (see
// ArchiveTranscriptsOptions.signer).
signer?: ChannelSigner;
+ // Materialize-only mode for the docker per-site compose (see
+ // ArchiveTranscriptsOptions.readOnly).
+ readOnly?: boolean;
};
export type ArchiveLiveChatResult = {
@@ -210,15 +213,19 @@ export async function archiveLiveChat(
);
const outDir = opts.outDir ?? archivesDir(opts.paths);
- await mkdir(outDir, { recursive: true });
- await mkdir(stagingDir(opts.paths), { recursive: true });
+ if (!opts.readOnly) {
+ await mkdir(outDir, { recursive: true });
+ await mkdir(stagingDir(opts.paths), { recursive: true });
+ }
const allChannels = await listChannels(opts.paths);
const target = opts.channelSlugs
? allChannels.filter((c) => opts.channelSlugs!.includes(c.slug))
: allChannels;
- const signer = opts.signer ?? openChannelSigner(opts.paths);
+ const signer = opts.readOnly
+ ? null
+ : (opts.signer ?? openChannelSigner(opts.paths));
const channelLimit = pLimit(opts.channelConcurrency ?? 4);
let reused = 0;
@@ -234,10 +241,25 @@ export async function archiveLiveChat(
outDir,
`${ch.slug}.${ARCHIVE_INFIX}.${archiveExtension(build.format)}`,
);
+ // Materialize-only: consume the pre-warmed cache, never generate.
+ if (opts.readOnly) {
+ if (await fileExists(archivePath)) {
+ const prev = await readArchiveSidecar(archivePath);
+ return {
+ archive: {
+ slug: ch.slug,
+ archivePath,
+ liveChatCount: prev?.count ?? 0,
+ },
+ reused: true,
+ };
+ }
+ return { skipped: { slug: ch.slug, reason: "no cached live-chat archive" } };
+ }
// `live-chat:` marks this signature as the live-chat variant so it never
// collides with the transcripts archive's sidecar (different files anyway,
// but keeps the two kinds' signatures independent).
- const sig = signer.signature(
+ const sig = signer!.signature(
ch.slug,
`live-chat:${archiveSignatureExtra(build, ch)}`,
);
@@ -306,7 +328,7 @@ export async function archiveLiveChat(
}
}
} finally {
- if (!opts.signer) await signer.close();
+ if (!opts.signer && signer) await signer.close();
}
if (reused > 0) log(`Reused ${reused} unchanged cached live-chat archive(s).`);
diff --git a/common/controller/archiveTranscripts.ts b/common/controller/archiveTranscripts.ts
@@ -75,6 +75,13 @@ export type ArchiveTranscriptsOptions = {
// caller can share one LMDB view across the transcripts + live-chat passes.
// When omitted, one is opened for the duration of this call.
signer?: ChannelSigner;
+ // Materialize-only mode for the docker per-site compose: never stage, compress,
+ // or write under the (read-only-mounted) cache. Report the already-cached
+ // archive for each member channel — the host's build:archives pass warmed the
+ // cache first. A member with no cached zip is a channel that legitimately had
+ // nothing to archive (build:archives ran over a superset of these slugs), so it
+ // is reported as skipped, exactly as a fresh generation would omit it.
+ readOnly?: boolean;
};
export type ArchiveTranscriptsResult = {
@@ -379,15 +386,21 @@ export async function archiveTranscripts(
);
const outDir = opts.outDir ?? archivesDir(opts.paths);
- await mkdir(outDir, { recursive: true });
- await mkdir(stagingDir(opts.paths), { recursive: true });
+ // In readOnly mode outDir is the shared cache on a read-only mount — never
+ // create dirs or touch staging there.
+ if (!opts.readOnly) {
+ await mkdir(outDir, { recursive: true });
+ await mkdir(stagingDir(opts.paths), { recursive: true });
+ }
const allChannels = await listChannels(opts.paths);
const target = opts.channelSlugs
? allChannels.filter((c) => opts.channelSlugs!.includes(c.slug))
: allChannels;
- const signer = opts.signer ?? openChannelSigner(opts.paths);
+ const signer = opts.readOnly
+ ? null
+ : (opts.signer ?? openChannelSigner(opts.paths));
const channelLimit = pLimit(opts.channelConcurrency ?? 4);
let reused = 0;
@@ -405,7 +418,22 @@ export async function archiveTranscripts(
outDir,
`${ch.slug}.${archiveExtension(build.format)}`,
);
- const sig = signer.signature(ch.slug, archiveSignatureExtra(build, ch));
+ // Materialize-only: consume the pre-warmed cache, never generate.
+ if (opts.readOnly) {
+ if (await fileExists(archivePath)) {
+ const prev = await readArchiveSidecar(archivePath);
+ return {
+ archive: {
+ slug: ch.slug,
+ archivePath,
+ transcriptCount: prev?.count ?? 0,
+ },
+ reused: true,
+ };
+ }
+ return { skipped: { slug: ch.slug, reason: "no cached archive" } };
+ }
+ const sig = signer!.signature(ch.slug, archiveSignatureExtra(build, ch));
if (sig) {
const prev = await readArchiveSidecar(archivePath);
if (prev && prev.signature === sig && (await fileExists(archivePath))) {
@@ -473,7 +501,7 @@ export async function archiveTranscripts(
}
}
} finally {
- if (!opts.signer) await signer.close();
+ if (!opts.signer && signer) await signer.close();
}
if (reused > 0) log(`Reused ${reused} unchanged cached archive(s).`);
diff --git a/docker/build-site.sh b/docker/build-site.sh
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+# Per-site build entrypoint for the docker export pipeline. Runs inside the
+# Dockerfile.build image, once per SITE_ID, in a parallel fan-out on the host.
+#
+# The host has already run Phase A (build:data + build:archives), so the shared
+# LMDB index, the .export-index staging, and the archive cache are all warm and
+# mounted READ-ONLY. This container only composes ITS site's public/ and runs
+# `next build`, writing everything into the per-site mount at /site.
+#
+# Expected run-time mounts (see runDockerBuildAllPhase):
+# <transcriptsDir> -> /data/transcripts (ro)
+# <export/.export-index> -> /data/export/.export-index (ro)
+# <exportBuildsDir>/<siteId> -> /site (rw)
+set -euo pipefail
+
+: "${SITE_ID:?SITE_ID is required}"
+
+export HOME="${HOME:-/tmp}"
+export NODE_ENV=production
+export TRANSCRIPTS_DIR=/data/transcripts
+export EXPORT_INDEX_DIR=/data/export/.export-index
+export EXPORT_PUBLIC_DIR=/site/public
+# Materialize-only: consume the host-warmed archive cache (read-only mount),
+# never re-generate — see compose-site.ts / archiveTranscripts.ts.
+export ARCHIVES_READONLY=1
+
+# Persist Next's build cache (.next) into the per-site mount so incremental
+# next builds stay warm across runs. A symlink is safe for .next because Next
+# reuses — never wholesale-replaces — its cache dir. The export dir was made
+# writable in the image so this symlink (and Next's next-env.d.ts) can be created
+# as an arbitrary runtime uid.
+mkdir -p /site/.next /site/out /site/public
+rm -rf export/.next
+ln -s /site/.next export/.next
+
+echo "[build-site] building site '${SITE_ID}'"
+# build:nodata = compose:site + next build, WITHOUT the prebuild data phase
+# (already done on the host). SITE_ID selects the site in compose-site.ts.
+# compose writes straight into the mounted /site/public (EXPORT_PUBLIC_DIR).
+pnpm --filter export run build:nodata
+
+# next build writes export/out as a FRESH real dir (it removes+recreates out, so
+# a symlink there wouldn't survive) — publish it into the per-site mount. Reached
+# only when the build succeeded (set -e aborts otherwise).
+echo "[build-site] publishing out/ -> /site/out"
+rm -rf /site/out
+mkdir -p /site/out
+cp -a export/out/. /site/out/
+echo "[build-site] done -> /site/out ($(find /site/out -type f | wc -l) files)"
diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md
@@ -1,5 +1,8 @@
# Changelog
+## [Unreleased]
+- **Docker build mode is now real: build every site in parallel, then deploy them serially.** The `Docker` build mode (Settings → Build pipeline) was previously a stub that fell back to the basic build. It now runs a proper pipeline, driven by a new **Build all sites** control on the Deploy page (one job, one log, one Cancel). The shared, corpus-scale work — the search index, the per-site staging, and the downloadable archive zips — runs **once on the host**; then each site's `compose + next build` runs in its **own container in parallel** (capped by the **Max parallel builds** setting), each writing an isolated per-site `out/` under `export/.export-builds/<siteId>/`; then the built sites **deploy serially** on the host (R2 upload + `wrangler pages deploy`), tolerant of a single site failing. Containers are read-only over the shared corpus/index/archive cache and run as your host user so outputs aren't root-owned. The image (`Dockerfile.build`, tag from **Build image**) is built/reused via Docker layer caching; when no container engine is available the action falls back to a serial host build+deploy. New env knobs: `DOCKER_BIN` (e.g. `podman`), `DOCKER_BUILD_MEMORY`/`DOCKER_BUILD_CPUS` (per-container caps). See `editor/app/deploy/buildDeployCore.ts` (`runDockerBuildAllPhase`/`runDockerDeployAllPhase`), `editor/app/build/buildAction.ts` (`buildAndDeployAllSitesAction`/`buildAllSitesAction`), `editor/app/deploy/components/BuildAllSitesButton.tsx`, `Dockerfile.build`, `docker/build-site.sh`, `common/bin/build-archives.ts`, and **[DEPLOY_DOCKER.md](../DEPLOY_DOCKER.md)**.
+
## [0.7.3] - 2026-07-07
- **Deploys no longer re-upload unchanged oversize archives to R2.** The deploy step used to stream every over-cap archive to R2 on every deploy, even ones byte-identical to what was already there. Because the export build now reuses an unchanged channel's cached zip verbatim, the upload step first does a cheap `HeadObject` and skips any archive whose R2 object already has the same size — so a redeploy after changing one channel only re-uploads that channel's oversize bundle. See `editor/app/deploy/buildDeployCore.ts`.
- **New "Search aliases" page for authoring known-term suggestions.** A concept is often spelled many ways — transcription in particular mangles them (AI expands `loli` to `lolly`/`loly`) — so searching one spelling silently misses the rest. This page authors a curated dictionary where one entry groups all the trigger spellings of a concept with a single robust replacement (e.g. `\blol(i|ly)`). A **Global** section applies to every site and ships a small seeded default set you can edit or remove; a **per-site** section (shown when a site is selected in the sidebar) adds to or overrides the global list by id — reuse a global id and mark it disabled to hide that alias for just that site. Nothing is forced: the entries only surface a *suggestion* in the viewer's search box, which the searcher can apply or ignore. Stored as `search-aliases.json` (global under the data dir, per-site under `sites/<id>/`) and merged into each site's bundle at build time. See `editor/app/aliases/{page.tsx,actions.ts,EditorAliasesClient.tsx}`, `common/lib/{searchAliases,aliasesStore}.ts`, `editor/app/lib/nav.ts`, and `editor/e2e/aliases.spec.ts`.
diff --git a/editor/app/build/buildAction.ts b/editor/app/build/buildAction.ts
@@ -15,16 +15,22 @@ import {
archiveCombinedLiveChat,
} from "yt-dlp-transcript-common/controller/archiveLiveChat";
import { getPaths } from "yt-dlp-transcript-common/lib/paths";
-import { getSite } from "yt-dlp-transcript-common/lib/site";
+import { getSite, listSites, type Site } from "yt-dlp-transcript-common/lib/site";
import {
runManagedFunction,
type StreamActionResult,
} from "yt-dlp-transcript-common/jobs/streamCommand";
import {
+ dockerAvailable,
+ dockerSiteOutDir,
resolveOutDir,
runArchiveUploadIntoLog,
runBuildPhase,
runDeployIntoLog,
+ runDockerBuildAllPhase,
+ runDockerDeployAllPhase,
+ type SiteBuildOutcome,
+ type SiteDeployOutcome,
} from "../deploy/buildDeployCore";
const DEFAULT_BUILD_QUEUE = "build";
@@ -237,3 +243,207 @@ export async function archiveCombinedLiveChatAction(
},
});
}
+
+// Build EVERY configured site, then deploy each — the docker-pipeline entry point.
+// With a container engine available: run the shared data phase + archive warm
+// once, fan the per-site builds out in parallel (capped by maxParallelBuilds),
+// then deploy the built sites SERIALLY on the host. Without one: fall back to a
+// serial host build+deploy per site. One managed job (single streamed log +
+// Cancel), serialized on the deploy queue so it never overlaps another deploy.
+export async function buildAndDeployAllSitesAction(
+ skipArchives?: boolean,
+): Promise<StreamActionResult> {
+ const paths = getPaths();
+ const sites = listSites(paths);
+ if (sites.length === 0) return { ok: false, error: "No sites configured." };
+ return runManagedFunction({
+ kind: "build-deploy-all",
+ queueKey: DEPLOY_QUEUE,
+ paths,
+ fn: async (onLog, signal) => {
+ const useDocker = await dockerAvailable(signal);
+ if (signal.aborted) return;
+
+ if (!useDocker) {
+ onLog(
+ "[notice] No container engine available (start the daemon, or set " +
+ "DOCKER_BIN) — falling back to serial host build+deploy.\n",
+ );
+ const deploys = await basicBuildAndDeployAll(
+ onLog,
+ signal,
+ sites,
+ paths,
+ skipArchives,
+ );
+ if (signal.aborted) return;
+ summarize(onLog, deploys);
+ return;
+ }
+
+ // Docker: build all (Phase A + B), barrier, then deploy all serially (C).
+ const outcomes = await runDockerBuildAllPhase(onLog, signal, sites, paths, {
+ skipArchives,
+ });
+ if (signal.aborted) return;
+ const builtOk = new Set(
+ outcomes.filter((o) => o.code === 0).map((o) => o.siteId),
+ );
+ onLog(`\n=== Phase C: deploy (${builtOk.size}/${sites.length} built) ===`);
+ const deploys = await runDockerDeployAllPhase(
+ onLog,
+ signal,
+ sites,
+ builtOk,
+ paths,
+ (id) => dockerSiteOutDir(paths, id),
+ );
+ if (signal.aborted) return;
+ summarize(onLog, deploys, outcomes);
+ },
+ });
+}
+
+// Build EVERY configured site WITHOUT deploying — e.g. to warm caches or validate
+// that each site composes and next-builds. Docker mode fans the builds out in
+// parallel; otherwise it's a serial host loop (whose shared export/out is
+// overwritten per site, so only the last survives — build-all is chiefly a docker
+// feature). Queued on the build queue.
+export async function buildAllSitesAction(
+ skipArchives?: boolean,
+): Promise<StreamActionResult> {
+ const paths = getPaths();
+ const sites = listSites(paths);
+ if (sites.length === 0) return { ok: false, error: "No sites configured." };
+ return runManagedFunction({
+ kind: "build-all",
+ queueKey: DEFAULT_BUILD_QUEUE,
+ paths,
+ fn: async (onLog, signal) => {
+ const useDocker = await dockerAvailable(signal);
+ if (signal.aborted) return;
+ let outcomes: SiteBuildOutcome[];
+ if (useDocker) {
+ outcomes = await runDockerBuildAllPhase(onLog, signal, sites, paths, {
+ skipArchives,
+ });
+ } else {
+ onLog(
+ "[notice] No container engine available — building sites serially on the host.\n",
+ );
+ outcomes = [];
+ for (let i = 0; i < sites.length; i++) {
+ if (signal.aborted) return;
+ const site = sites[i];
+ onLog(`\n=== Build ${site.siteId} (${i + 1}/${sites.length}) ===`);
+ const code = await runBuildPhase(onLog, signal, site.siteId, paths, {
+ skipData: i > 0,
+ skipArchives,
+ });
+ outcomes.push({ siteId: site.siteId, code });
+ }
+ }
+ if (signal.aborted) return;
+ const ok = outcomes.filter((o) => o.code === 0).length;
+ const failed = outcomes
+ .filter((o) => o.code !== 0)
+ .map((o) => `${o.siteId} (exit ${o.code})`);
+ onLog(`\n=== Summary: ${ok}/${sites.length} built ===`);
+ if (failed.length) throw new Error(`Build failures: ${failed.join(", ")}`);
+ },
+ });
+}
+
+// Serial host build+deploy fallback for buildAndDeployAllSitesAction when no
+// container engine is available. Interleaved (build A → deploy A → build B …)
+// because the basic build shares export/out — building all first would leave only
+// the last site's bundle. Rebuilds the pool-wide data once (first site).
+async function basicBuildAndDeployAll(
+ onLog: (line: string) => void,
+ signal: AbortSignal,
+ sites: Site[],
+ paths: ReturnType<typeof getPaths>,
+ skipArchives?: boolean,
+): Promise<SiteDeployOutcome[]> {
+ const basicOut = resolveOutDir("", paths);
+ const deploys: SiteDeployOutcome[] = [];
+ for (let i = 0; i < sites.length; i++) {
+ if (signal.aborted) break;
+ const site = sites[i];
+ onLog(`\n=== Build ${site.siteId} (${i + 1}/${sites.length}) ===`);
+ const code = await runBuildPhase(onLog, signal, site.siteId, paths, {
+ skipData: i > 0,
+ skipArchives,
+ });
+ if (signal.aborted) break;
+ if (code !== 0) {
+ onLog(`[${site.siteId}] build FAILED (exit ${code}) — skipping deploy`);
+ deploys.push({
+ siteId: site.siteId,
+ status: "skipped",
+ reason: `build exit ${code}`,
+ });
+ continue;
+ }
+ if (!site.cloudflareProject) {
+ onLog(`[${site.siteId}] deploy skipped — no Cloudflare project configured`);
+ deploys.push({
+ siteId: site.siteId,
+ status: "skipped",
+ reason: "no cloudflareProject",
+ });
+ continue;
+ }
+ onLog(`=== Deploy ${site.siteId} ===`);
+ const up = await runArchiveUploadIntoLog(onLog, signal, site, paths);
+ if (signal.aborted) break;
+ if (up !== 0) {
+ onLog(`[${site.siteId}] deploy FAILED — R2 upload exit ${up}`);
+ deploys.push({
+ siteId: site.siteId,
+ status: "failed",
+ reason: `R2 upload exit ${up}`,
+ });
+ continue;
+ }
+ const dep = await runDeployIntoLog(onLog, signal, site, basicOut, paths);
+ if (signal.aborted) break;
+ if (dep !== 0) {
+ onLog(`[${site.siteId}] deploy FAILED — exit ${dep}`);
+ deploys.push({
+ siteId: site.siteId,
+ status: "failed",
+ reason: `deploy exit ${dep}`,
+ });
+ continue;
+ }
+ onLog(`[${site.siteId}] deployed.`);
+ deploys.push({ siteId: site.siteId, status: "deployed" });
+ }
+ return deploys;
+}
+
+// Log the final tally and THROW if anything failed, so the managed job ends in a
+// failed state the UI surfaces. Skips are reported but not failures.
+function summarize(
+ onLog: (line: string) => void,
+ deploys: SiteDeployOutcome[],
+ builds?: SiteBuildOutcome[],
+): void {
+ const deployed = deploys
+ .filter((d) => d.status === "deployed")
+ .map((d) => d.siteId);
+ const skipped = deploys.filter((d) => d.status === "skipped");
+ const failed = [
+ ...(builds ?? [])
+ .filter((b) => b.code !== 0)
+ .map((b) => `${b.siteId} (build exit ${b.code})`),
+ ...deploys
+ .filter((d) => d.status === "failed")
+ .map((d) => `${d.siteId} (deploy: ${d.reason})`),
+ ];
+ onLog(`\n=== Summary: ${deployed.length} deployed ===`);
+ if (deployed.length) onLog(` deployed: ${deployed.join(", ")}`);
+ for (const s of skipped) onLog(` skipped ${s.siteId}: ${s.reason}`);
+ if (failed.length) throw new Error(`Failures: ${failed.join(", ")}`);
+}
diff --git a/editor/app/deploy/buildDeployCore.ts b/editor/app/deploy/buildDeployCore.ts
@@ -5,9 +5,8 @@
// here as plain helpers and import them into the thin action wrappers.
import path from "node:path";
-import { readdir } from "node:fs/promises";
-import { createReadStream } from "node:fs";
-import { stat } from "node:fs/promises";
+import { mkdir, readdir, stat } from "node:fs/promises";
+import { createReadStream, existsSync } from "node:fs";
import { S3Client, HeadObjectCommand } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import { runChildIntoLog } from "yt-dlp-transcript-common/jobs/runChild";
@@ -15,21 +14,32 @@ import type { Paths } from "yt-dlp-transcript-common/lib/paths";
import { getSettings } from "yt-dlp-transcript-common/lib/settings";
import type { Site } from "yt-dlp-transcript-common/lib/site";
-// Where the static output that should be deployed for a site lives. Today the
-// build always writes export/out (basic build), so deploy reads it in both
-// modes. The Docker follow-up will write each site's `out/` to
-// paths.exportBuildsDir/<siteId> and switch this to return that per-site dir in
-// docker mode; until then docker mode falls back to a basic build (see
-// runBuildPhase), so export/out is correct everywhere.
+// Where the basic (host) build writes the static bundle to deploy: the fixed
+// export/out, composed one site at a time. The docker fan-out writes per-site
+// out/ dirs instead — see dockerSiteOutDir.
export function resolveOutDir(_siteId: string, paths: Paths): string {
return path.join(paths.exportDir, "out");
}
-// Run the build phase for one site, streaming into `onLog`, returning the exit
-// code. Routes on the persisted build mode: "basic" runs `pnpm run build` in
-// export/ (serialized upstream on the build queue, since the export/ tree is
-// shared); "docker" is a follow-up — until the container pipeline lands it logs
-// a notice and falls back to the basic build so builds always succeed.
+// Where a docker per-site container writes its built bundle (mounted as /site/out
+// inside the container). Isolated per site so parallel builds never collide.
+export function dockerSiteOutDir(paths: Paths, siteId: string): string {
+ return path.join(paths.exportBuildsDir, siteId, "out");
+}
+
+// Where a docker per-site container stages oversize archives for R2 upload. The
+// container's EXPORT_PUBLIC_DIR is /site/public (host exportBuildsDir/<siteId>/
+// public), so compose writes .r2-staging as its sibling under exportBuildsDir.
+export function dockerSiteStagingDir(paths: Paths, siteId: string): string {
+ return path.join(paths.exportBuildsDir, siteId, ".r2-staging", siteId, "archives");
+}
+
+// Run the basic (host) build phase for one site, streaming into `onLog`,
+// returning the exit code. Runs `pnpm run build` in export/ (serialized upstream
+// on the build queue, since the export/ tree is shared). This is the single-site
+// build for basic mode, and the fallback the all-sites docker action drops to
+// when no container engine is available. The parallel per-site container path
+// lives in runDockerBuildAllPhase.
export async function runBuildPhase(
onLog: (line: string) => void,
signal: AbortSignal,
@@ -37,13 +47,6 @@ export async function runBuildPhase(
paths: Paths,
opts?: { skipData?: boolean; skipArchives?: boolean },
): Promise<number> {
- const { mode } = getSettings().buildPipeline;
- if (mode === "docker") {
- onLog(
- "[notice] Docker build mode isn't wired up yet — running the basic build " +
- "for now. Parallel container builds arrive in a follow-up.\n",
- );
- }
// When skipping the data rebuild, run the `build:nodata` script instead of
// `build`. `build:nodata` has the same body (compose:site + next build) but a
// different name, so npm's `prebuild` lifecycle hook (which runs build:data =
@@ -77,9 +80,10 @@ export async function runBuildPhase(
});
}
-// Where compose-site staged this site's oversize archives for R2 upload. Kept
-// outside export/public so they never ship as Pages assets. Mirrors the path
-// composeArchives writes to in common/bin/compose-site.ts.
+// Where the basic (host) compose staged this site's oversize archives for R2
+// upload. Kept outside export/public so they never ship as Pages assets. Mirrors
+// the path composeArchives writes to in common/bin/compose-site.ts. The docker
+// fan-out uses dockerSiteStagingDir instead, passed explicitly.
function archiveStagingDir(siteId: string, paths: Paths): string {
return path.join(
path.dirname(paths.exportPublicDir),
@@ -119,11 +123,15 @@ export async function runArchiveUploadIntoLog(
signal: AbortSignal,
site: Site,
paths: Paths,
+ // Where the oversize archives were staged. Defaults to the basic host location;
+ // the docker deploy phase passes dockerSiteStagingDir since its container wrote
+ // .r2-staging under exportBuildsDir/<siteId> instead.
+ stagingDirOverride?: string,
): Promise<number> {
const bucket = getSettings().archiveStorage?.bucket?.trim();
if (!bucket) return 0;
- const stagingDir = archiveStagingDir(site.siteId, paths);
+ const stagingDir = stagingDirOverride ?? archiveStagingDir(site.siteId, paths);
let files: string[];
try {
files = (await readdir(stagingDir)).filter((f) => !f.startsWith("."));
@@ -250,3 +258,270 @@ export async function runDeployIntoLog(
},
});
}
+
+// ---------------------------------------------------------------------------
+// Docker export pipeline (buildPipeline.mode = "docker")
+//
+// Three ordered phases (see DEPLOY_DOCKER.md):
+// A) HOST, serial: build:data (shared LMDB + .export-index) then build:archives
+// (warm the shared archive cache). One writer of the shared state.
+// B) CONTAINERS, parallel (cap maxParallelBuilds): each site's compose + next
+// build in its own container, read-only over the shared caches, writing only
+// its per-site out/ under exportBuildsDir/<siteId>.
+// C) HOST, serial: deploy each built site (handled by runDockerDeployAllPhase).
+// ---------------------------------------------------------------------------
+
+// The container engine binary. Defaults to `docker`; podman is a CLI drop-in
+// (and rootless podman yields host-owned outputs without needing `-u`).
+function dockerBin(): string {
+ return process.env.DOCKER_BIN?.trim() || "docker";
+}
+
+export type SiteBuildOutcome = { siteId: string; code: number };
+export type SiteDeployOutcome = {
+ siteId: string;
+ status: "deployed" | "skipped" | "failed";
+ reason?: string;
+};
+
+// Cheap probe: is the container engine installed and its daemon reachable? Used
+// to fall back to serial host builds when docker isn't available.
+export async function dockerAvailable(signal: AbortSignal): Promise<boolean> {
+ const code = await runChildIntoLog(() => {}, signal, {
+ command: dockerBin(),
+ args: ["version"],
+ cwd: process.cwd(),
+ });
+ return code === 0;
+}
+
+// Run a host-side export pnpm script (build:data / build:archives) for Phase A.
+// These are pool-wide: no SITE_ID, and no EXPORT_PUBLIC_DIR override so the
+// shared index/staging land at their canonical export/.export-index location
+// (exactly what the fan-out containers mount read-only).
+async function runHostScript(
+ onLog: (line: string) => void,
+ signal: AbortSignal,
+ paths: Paths,
+ script: string,
+ extraEnv?: Record<string, string>,
+): Promise<number> {
+ return runChildIntoLog(onLog, signal, {
+ command: "pnpm",
+ args: ["run", script],
+ cwd: paths.exportDir,
+ env: {
+ ...process.env,
+ NODE_ENV: "production",
+ TRANSCRIPTS_DIR: paths.transcriptsDir,
+ ...extraEnv,
+ },
+ });
+}
+
+// Build (or reuse cached layers of) the per-site build image.
+async function ensureBuildImage(
+ onLog: (line: string) => void,
+ signal: AbortSignal,
+ paths: Paths,
+): Promise<number> {
+ const { dockerImage, dockerfile } = getSettings().buildPipeline;
+ onLog(`[docker] building image "${dockerImage}" from ${dockerfile} (cached layers reused)`);
+ return runChildIntoLog(onLog, signal, {
+ command: dockerBin(),
+ args: ["build", "-f", dockerfile, "-t", dockerImage, "."],
+ cwd: paths.monorepoRoot,
+ });
+}
+
+// Run ONE site's build in a container. Mounts the corpus, the shared LMDB index,
+// and the .export-index staging read-only; mounts the per-site output dir rw.
+// Streams with a [siteId] prefix. Network is left ENABLED — `next build` uses
+// next/font/google, which fetches the site's fonts from Google at build time;
+// `--network=none` would fail the build. Isolation still comes from the per-site
+// output dir, the read-only shared mounts, and the non-root `-u` user.
+async function runDockerBuildOne(
+ onLog: (line: string) => void,
+ signal: AbortSignal,
+ siteId: string,
+ paths: Paths,
+ opts?: { skipArchives?: boolean },
+): Promise<number> {
+ const { dockerImage } = getSettings().buildPipeline;
+ const siteDir = path.join(paths.exportBuildsDir, siteId);
+ // Pre-create the mount target as the host user so container-written files are
+ // host-owned (paired with `-u` below), not created root-owned by the daemon.
+ await mkdir(siteDir, { recursive: true });
+
+ const args = ["run", "--rm", "--init"];
+ const uid = typeof process.getuid === "function" ? process.getuid() : null;
+ const gid = typeof process.getgid === "function" ? process.getgid() : null;
+ if (uid !== null && gid !== null) args.push("-u", `${uid}:${gid}`);
+ // Optional resource caps so N parallel builds (each next build can use ~8 GB)
+ // don't OOM the host. Sized by the operator; see DEPLOY_DOCKER.md.
+ const mem = process.env.DOCKER_BUILD_MEMORY?.trim();
+ const cpus = process.env.DOCKER_BUILD_CPUS?.trim();
+ if (mem) args.push("--memory", mem);
+ if (cpus) args.push("--cpus", cpus);
+ args.push(
+ "-v", `${paths.transcriptsDir}:/data/transcripts:ro`,
+ "-v", `${paths.exportIndexDir}:/data/export/.export-index:ro`,
+ "-v", `${siteDir}:/site`,
+ "-e", `SITE_ID=${siteId}`,
+ );
+ // Mount the host settings.json fresh (build config: archive storage, size caps)
+ // rather than relying on a possibly-stale copy — it is NOT baked into the image.
+ if (existsSync(paths.settingsFile)) {
+ args.push(
+ "-v", `${paths.settingsFile}:/data/settings.json:ro`,
+ "-e", "SETTINGS_FILE=/data/settings.json",
+ );
+ }
+ // Mount the entrypoint fresh over the baked copy so a script tweak takes effect
+ // without an image rebuild (the image still bakes it as a fallback).
+ const entrypoint = path.join(paths.monorepoRoot, "docker", "build-site.sh");
+ if (existsSync(entrypoint)) {
+ args.push("-v", `${entrypoint}:/repo/docker/build-site.sh:ro`);
+ }
+ if (opts?.skipArchives) args.push("-e", "BUILD_ARCHIVES=0");
+ args.push(dockerImage);
+
+ return runChildIntoLog(onLog, signal, {
+ command: dockerBin(),
+ args,
+ cwd: paths.monorepoRoot,
+ label: `[${siteId}] `,
+ });
+}
+
+// Phases A + B. Returns each site's build exit code (0 = ok). Throws only on an
+// infrastructure failure (data phase / archive warm / image build) that aborts
+// the whole run before any site could build; per-site build failures are
+// returned, not thrown, so one bad site never blocks the rest.
+export async function runDockerBuildAllPhase(
+ onLog: (line: string) => void,
+ signal: AbortSignal,
+ sites: Site[],
+ paths: Paths,
+ opts?: { skipArchives?: boolean },
+): Promise<SiteBuildOutcome[]> {
+ const { maxParallelBuilds } = getSettings().buildPipeline;
+
+ // --- Phase A: shared data + archive cache (host, serial) ---
+ onLog("=== Phase A: shared data + archive cache (host) ===");
+ const dataCode = await runHostScript(onLog, signal, paths, "build:data");
+ if (signal.aborted) return [];
+ if (dataCode !== 0) throw new Error(`Data phase failed (exit ${dataCode}).`);
+ if (!opts?.skipArchives) {
+ const archCode = await runHostScript(onLog, signal, paths, "build:archives");
+ if (signal.aborted) return [];
+ if (archCode !== 0) throw new Error(`Archive cache warm failed (exit ${archCode}).`);
+ }
+
+ const imgCode = await ensureBuildImage(onLog, signal, paths);
+ if (signal.aborted) return [];
+ if (imgCode !== 0) throw new Error(`Docker image build failed (exit ${imgCode}).`);
+
+ // --- Phase B: per-site fan-out (containers, parallel) ---
+ onLog(
+ `=== Phase B: building ${sites.length} site(s), up to ${maxParallelBuilds} in parallel ===`,
+ );
+ return runWithConcurrency(sites, maxParallelBuilds, async (site) => {
+ if (signal.aborted) return { siteId: site.siteId, code: 1 };
+ const code = await runDockerBuildOne(onLog, signal, site.siteId, paths, opts);
+ onLog(`[${site.siteId}] build ${code === 0 ? "ok" : `FAILED (exit ${code})`}`);
+ return { siteId: site.siteId, code };
+ });
+}
+
+// Phase C: deploy each built site SERIALLY on the host, after the build barrier.
+// Partial-failure tolerant — a site that fails to upload/deploy is recorded and
+// the loop continues. Sites that failed to build, or have no Cloudflare project,
+// are skipped. `outDirFor` resolves each site's built bundle (docker: per-site;
+// basic fallback: export/out).
+export async function runDockerDeployAllPhase(
+ onLog: (line: string) => void,
+ signal: AbortSignal,
+ sites: Site[],
+ builtOk: Set<string>,
+ paths: Paths,
+ outDirFor: (siteId: string) => string,
+): Promise<SiteDeployOutcome[]> {
+ const outcomes: SiteDeployOutcome[] = [];
+ for (const site of sites) {
+ if (signal.aborted) break;
+ if (!builtOk.has(site.siteId)) {
+ onLog(`[${site.siteId}] deploy skipped — build failed`);
+ outcomes.push({ siteId: site.siteId, status: "skipped", reason: "build failed" });
+ continue;
+ }
+ if (!site.cloudflareProject) {
+ onLog(`[${site.siteId}] deploy skipped — no Cloudflare project configured`);
+ outcomes.push({
+ siteId: site.siteId,
+ status: "skipped",
+ reason: "no cloudflareProject",
+ });
+ continue;
+ }
+ onLog(`=== Deploy ${site.siteId} ===`);
+ const uploadCode = await runArchiveUploadIntoLog(
+ onLog,
+ signal,
+ site,
+ paths,
+ dockerSiteStagingDir(paths, site.siteId),
+ );
+ if (signal.aborted) break;
+ if (uploadCode !== 0) {
+ onLog(`[${site.siteId}] deploy FAILED — R2 upload exit ${uploadCode}`);
+ outcomes.push({
+ siteId: site.siteId,
+ status: "failed",
+ reason: `R2 upload exit ${uploadCode}`,
+ });
+ continue;
+ }
+ const deployCode = await runDeployIntoLog(
+ onLog,
+ signal,
+ site,
+ outDirFor(site.siteId),
+ paths,
+ );
+ if (signal.aborted) break;
+ if (deployCode !== 0) {
+ onLog(`[${site.siteId}] deploy FAILED — exit ${deployCode}`);
+ outcomes.push({
+ siteId: site.siteId,
+ status: "failed",
+ reason: `deploy exit ${deployCode}`,
+ });
+ continue;
+ }
+ onLog(`[${site.siteId}] deployed.`);
+ outcomes.push({ siteId: site.siteId, status: "deployed" });
+ }
+ return outcomes;
+}
+
+// Bounded-concurrency map over a fixed work set, preserving input order in the
+// results. No external dep; a fresh worker pulls the next index until exhausted.
+async function runWithConcurrency<T, R>(
+ items: T[],
+ limit: number,
+ worker: (item: T) => Promise<R>,
+): Promise<R[]> {
+ const results: R[] = new Array(items.length);
+ let next = 0;
+ const width = Math.max(1, Math.min(limit, items.length));
+ const runners = Array.from({ length: width }, async () => {
+ while (true) {
+ const i = next++;
+ if (i >= items.length) break;
+ results[i] = await worker(items[i]);
+ }
+ });
+ await Promise.all(runners);
+ return results;
+}
diff --git a/editor/app/deploy/components/BuildAllSitesButton.tsx b/editor/app/deploy/components/BuildAllSitesButton.tsx
@@ -0,0 +1,78 @@
+"use client";
+
+import { useState } from "react";
+import {
+ buildAllSitesAction,
+ buildAndDeployAllSitesAction,
+} from "../../build/buildAction";
+import { JobLane } from "./JobLane";
+
+type Lane = { deploy: boolean; skipArchives: boolean; key: number };
+
+// One-click orchestrated pipeline over EVERY site as a SINGLE managed job (one
+// log, one Cancel): in Docker mode it runs the shared data phase + archive warm
+// once, fans the per-site builds out in parallel (capped by maxParallelBuilds),
+// then deploys the built sites serially. Without a container engine it falls back
+// to a serial host build+deploy. Distinct from the per-site panel below, which
+// launches one separate job per selected site.
+export function BuildAllSitesButton({ dockerMode }: { dockerMode: boolean }) {
+ const [deploy, setDeploy] = useState(true);
+ const [skipArchives, setSkipArchives] = useState(false);
+ const [lane, setLane] = useState<Lane | null>(null);
+ const [run, setRun] = useState(0);
+
+ function launch() {
+ const next = run + 1;
+ setRun(next);
+ setLane({ deploy, skipArchives, key: next });
+ }
+
+ return (
+ <div className="flex flex-col gap-3">
+ <div className="flex flex-wrap items-center gap-3">
+ <button
+ type="button"
+ onClick={launch}
+ className="px-3 py-2 rounded-md bg-primary text-primary-foreground text-sm font-medium hover:opacity-90"
+ >
+ {deploy ? "Build & deploy all sites" : "Build all sites"}
+ </button>
+ <label className="flex items-center gap-2 text-sm text-muted-foreground">
+ <input
+ type="checkbox"
+ checked={deploy}
+ onChange={(e) => setDeploy(e.target.checked)}
+ />
+ Deploy after build
+ </label>
+ <label className="flex items-center gap-2 text-sm text-muted-foreground">
+ <input
+ type="checkbox"
+ checked={skipArchives}
+ onChange={(e) => setSkipArchives(e.target.checked)}
+ />
+ Skip archive zips
+ </label>
+ </div>
+ <p className="text-xs text-muted-foreground">
+ {dockerMode
+ ? "Docker pipeline: shared data phase runs once, per-site builds run in parallel, then deploys run serially."
+ : "Docker mode is off — this runs a serial host build+deploy fallback (one site at a time)."}
+ </p>
+ {lane && (
+ <JobLane
+ key={lane.key}
+ title={lane.deploy ? "Build & deploy all sites" : "Build all sites"}
+ subtitle={
+ dockerMode ? "Docker pipeline · parallel builds" : "Serial host fallback"
+ }
+ trigger={() =>
+ lane.deploy
+ ? buildAndDeployAllSitesAction(lane.skipArchives)
+ : buildAllSitesAction(lane.skipArchives)
+ }
+ />
+ )}
+ </div>
+ );
+}
diff --git a/editor/app/deploy/components/BuildSitesPanel.tsx b/editor/app/deploy/components/BuildSitesPanel.tsx
@@ -120,7 +120,7 @@ export function BuildSitesPanel({
{serial && (
<p className="text-xs text-muted-foreground">
Basic mode runs these one at a time (the build output tree is shared).
- Switch to Docker mode for true parallel builds — a follow-up.
+ Use “Build all sites” above in Docker mode for true parallel builds.
</p>
)}
diff --git a/editor/app/deploy/page.tsx b/editor/app/deploy/page.tsx
@@ -14,6 +14,7 @@ import { getRegistry } from "yt-dlp-transcript-common/jobs/registry";
import { BuildExportButton } from "./components/BuildExportButton";
import { BuildDeployButton } from "./components/BuildDeployButton";
import { BuildModeToggle } from "./components/BuildModeToggle";
+import { BuildAllSitesButton } from "./components/BuildAllSitesButton";
import { BuildSitesPanel } from "./components/BuildSitesPanel";
import { CutReleaseForm } from "./components/CutReleaseForm";
import { DeployButton } from "./components/DeployButton";
@@ -169,10 +170,19 @@ export default async function DeployPage({
{/* 4 — Batch: build (and optionally deploy) several sites at once. */}
<section className="flex flex-col gap-3 border-t border-border pt-6">
<div>
- <h2 className="text-lg font-semibold">Build multiple sites</h2>
+ <h2 className="text-lg font-semibold">Build all sites</h2>
<p className="text-sm text-muted-foreground">
- Pick sites to build (and optionally deploy) together. Each gets its
- own live log lane.
+ One job over every site: the shared data phase runs once, per-site
+ builds run in parallel (Docker mode), then deploys run serially.
+ </p>
+ </div>
+ <BuildAllSitesButton dockerMode={buildMode === "docker"} />
+
+ <div className="mt-4">
+ <h3 className="font-semibold">Or pick specific sites</h3>
+ <p className="text-sm text-muted-foreground">
+ Build (and optionally deploy) selected sites, each in its own live log
+ lane.
</p>
</div>
<BuildSitesPanel
diff --git a/export/CHANGELOG.md b/export/CHANGELOG.md
@@ -1,6 +1,7 @@
# Changelog
## [Unreleased]
+- **The export build can now fan out across sites in parallel (Docker mode).** Archive-zip generation is hoisted into a new shared host step (`build:archives`) that warms the persistent cache once for the union of all sites' channels, so the per-site `compose:site` can run read-only over that cache (`ARCHIVES_READONLY=1`) without re-zipping or racing — which is what lets several sites build at once. No change to a site's output; this is a build-pipeline change. See `common/bin/build-archives.ts`, `common/controller/{archiveTranscripts,archiveLiveChat}.ts` (`readOnly`), and `common/bin/compose-site.ts`. (Orchestration + UI live in the editor changelog.)
- **The "Ask AI" chat now searches like an assistant — and follow-ups work.** Instead of one keyword search per message, the chat now runs an *agentic* loop: it decides what to look up, reads the results, and can refine and search again before answering — you see each search as it happens. Crucially, a follow-up that builds on the last answer (e.g. "put that on a timeline", "summarise it") no longer throws away the earlier context and re-searches your wording — earlier excerpts stay in play, so it just reformats what it already found. Answers render as **Markdown** (with a *Format answers* toggle to fall back to plain text if anything looks off), a live status shows the search→answer pipeline, and the model is told the archive's known transcription-misspelling aliases so it searches for the right variants. A **Search mode** control (Auto / Native tools / Scripted) chooses between your model's native function-calling and a provider-agnostic protocol, with automatic fallback. See `export/app/ask/*`, `export/app/lib/{searchAgent,askConversation,nativeTools/*}.ts`, `common/components/Markdown.tsx`, and `export/e2e/ask-chat.spec.ts`.
## [0.7.3] - 2026-07-07
diff --git a/export/package.json b/export/package.json
@@ -9,6 +9,7 @@
"build:stats": "NODE_OPTIONS=--max-old-space-size=8192 tsx ../common/bin/build-stats.ts",
"build:templates": "tsx ../common/bin/build-chart-templates.ts",
"build:data": "pnpm run build:index && pnpm run build:stats && pnpm run build:templates",
+ "build:archives": "tsx ../common/bin/build-archives.ts",
"detect:duplicates": "NODE_OPTIONS=--max-old-space-size=8192 tsx ../common/bin/duplicate-shorts.ts",
"compose:site": "tsx ../common/bin/compose-site.ts",
"compose:hub": "tsx ../common/bin/compose-hub.ts",
