commit 67be20c1eae3cb3186bec9e945fbba7b774277ed parent 62fad5ab343ebc02077f667bbe6af37696feb1ad Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st> Date: Mon, 5 Oct 2026 15:45:12 -0400 reports: the fixture site has two revisions; e2e checks the revision line, the history page and a git clone; Reports tab shows the revision; docs The report-site fixture commits report.v1.json then report.json through the exporter into a stage store and publishes their history; the specs check "Revision 2 · 2026-10-04 · history", both revisions' hashes, summaries and word diffs, a phone-width page, and a git clone over the e2e server. The history page builds its own static params. The editor's Reports tab shows each report's revision, commit and last change. plans/report-sites.md "History", PUBLISH.md, changelogs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Diffstat:
15 files changed, 449 insertions(+), 42 deletions(-)
diff --git a/PUBLISH.md b/PUBLISH.md @@ -85,8 +85,21 @@ Chromium; skipped with a note on a host without Playwright's browser), `report.m clips, stills and screenshots as files, so the clips play offline, plus the Markdown and the citations; packed by the system `zip`, which a host must have), with an `export.json` naming each file's size and checksum and the hash of the report.json it -was made from. Every export ends with a footer naming the report's date and the first -12 characters of that hash. +was made from. Every export ends with a footer naming the report's revision, its date +and the first 12 characters of that hash. + +Each export also keeps the report's **revision history** +(`common/publish/reportHistory.ts`): a bare git repository per report at +`sites/<id>/reports/<report>/history-git/` (add `history-git/` to the corpus +repository's `.gitignore`). When the report.json's sha256 differs from the newest +revision's, the export commits a new revision holding `report.json` (byte for byte), +`report.md` and `exports.json` (the sha256 of every export file and of the citations +JSON and CSV), with the message `Revision N` and a summary of what changed. A re-export +of an unchanged report commits nothing. Author and committer are the site (its title, +`noreply@<id>.invalid`), dated in UTC; git runs with no system or global config and none +of the operator's environment. A footer names the revision and the report's sha256, never +a commit (the commit holds the footer); the history page maps the sha256 to its commit, +and `export.json` records it. The build's compose then writes the reports from what prepare left (`common/publish/composeReports.ts`): each report's page, its citations as @@ -94,8 +107,15 @@ The build's compose then writes the reports from what prepare left `report.md`, `evidence-pack.zip` — only those made from the report.json as it is now, and only a file of at most 24 MiB: a larger pack stays on the host; the page's download line lists what was published), its cited stills (never a saved source copy), -one page per cited moment with the transcript lines around it, and the prepared clips -and captures — only those the published reports cite. Every quote is checked as it is +one page per cited moment with the transcript lines around it, the prepared clips +and captures — only those the published reports cite — and each report's revision +history: `reports/<id>/history/history.json` (every revision's number, date, commit, +report sha256, change summary and claim-level word diff, which `/reports/<id>/history/` +renders) and a dumb-HTTP clone at `reports/<id>/history/repo/`, staged as the source +mirror is (a fresh bare clone, `repack -a -d`, `update-server-info`, then only `HEAD`, +`packed-refs`, `info/refs`, `objects/info/packs`, the packs and `refs/heads/main`), so +`git clone <siteUrl>/reports/<id>/history/repo` works. The deploy audit refuses any other +file in a clone or one over 24 MiB, and counts the clone's files toward Pages' 20,000. Every quote is checked as it is composed: a span's against its cues within 5 s either side (a record whose `en` track has no cues is read from `en-orig`), a post's against its text, scored as the share of the quote's words found there; the score, the time and the method are written into the diff --git a/editor/CHANGELOG.md b/editor/CHANGELOG.md @@ -1,6 +1,7 @@ # Changelog ## [Unreleased] +- **Exporting a changed report records a new revision of it.** `reports export` (and **Export reports** on a site's Reports tab, and the end of a prepare) now commits a revision to the report's own git history, `sites/<site>/reports/<id>/history-git/`, whenever its `report.json` changed since the last one: the `report.json`, its Markdown export and the checksums of every export file, with a message of `Revision N` and a summary of the change. A re-export of an unchanged report records nothing. The commits carry the site's name and a `noreply@<site>.invalid` address with dates in UTC, never your git name, email or time zone. The Reports tab shows each report's revision, its commit and the last change under **Exports**, and the site's next build publishes the history. Add `history-git/` to the corpus repository's `.gitignore`. - **A forum thread can be archived as a posts source.** A XenForo thread URL (`…/threads/<title>.<id>/`; Kiwi Farms is recognised by host) makes a forum-thread channel — platform "xenforo", one channel per thread, each forum post a post — searchable and readable like X and Bluesky posts, in the editor, the export and the MCP (`get_thread` gives a forum post's conversation: the posts it quotes and the posts quoting it). **Fetch posts** reads the thread in a headless browser, newest page first, one page at a time with a 10–20 s pause (the channel key `postPagePauseSeconds` sets it), and stops at already-archived posts; a **Latest N pages** box (`archilyzer posts fetch --pages N`) caps a run, and the next run continues where it stopped. The browser keeps one profile per forum host, so a browser check it clears once (KiwiFlare's proof of work, say) stays cleared; a check that does not clear within a minute, a captcha, a login wall or a refusal stops the run with the reason and keeps its place — never retried at once. **Connect forum session** on the channel page opens that profile in a window on the editor's machine, at the thread, for the operator to clear it or log in. **Import saved pages** (`archilyzer posts import-html <slug> <file-or-dir>…`) reads thread pages saved from a browser ("Save page as", complete or HTML only) through the same parser: new posts are added and a post saved again after an edit is updated. **Capture posts** works on forum posts: a screenshot of the post and its attached files, through the same profile. A forum post keeps its thread title, page, position, author id, last-edit time, quoted posts and its media links; quoted text is marked with "> " lines. - **A site's reports can be exported as files a reader saves and hosts again.** `archilyzer reports export <site> [--report <id>] [--formats html,pdf,md,zip]`, the `reports-export` job (`POST /api/ops/reports-export`, `pnpm ops reports-export`, and **Export reports** on a site's Reports tab) write each published report, checked as the build checks it, into `.export-index/sites/<site>/report-exports/<report>/`: `report.html`, one self-contained page (its own style, no script, stills and post screenshots inlined and recompressed, clips linked on the site); `report.pdf`, that page printed by headless Chromium, skipped with a note where there is none; `report.md`, plain Markdown with numbered references; and `evidence-pack.zip`, the page with its clips, stills and screenshots as files plus the Markdown and the citations, packed by the system `zip` (a host without it fails that format, naming it). An `export.json` names each file's size and checksum and the checksum of the report.json it was made from; every export ends with the report's date and the start of that checksum. Preparing the evidence media now exports at its end when nothing is missing, on the same queue. The build publishes an export beside the report only when it was made from the report as it is now and is at most 24 MiB — a larger evidence pack stays local — and the Reports tab lists each report's exports, their sizes and which the next build publishes. The 24 MiB limit the source mirror and the evidence clips already kept is now one shared number. - **A report video's cue lookup names a site that publishes only its reports.** Pointing a report-to-video manifest at such a site (`corpus.json` `site.scope: "cited"`) used to fail with "channel … is not in corpus.json"; it now says the site publishes no transcripts and to use a full archive or a local corpus. The site form's **Publish** hint says a cited-only site is never listed on the homepage or the hub. diff --git a/editor/app/sites/[siteId]/reports/page.tsx b/editor/app/sites/[siteId]/reports/page.tsx @@ -36,7 +36,9 @@ export const metadata: Metadata = { title: "Reports" }; // what prepare and the build would refuse. Publish, unpublish and reorder write // that one key (lib/reportsActions.ts). The evidence media the published // reports cite is prepared here too, before a build, and the reports exported -// as files (HTML, PDF, Markdown, an evidence pack) for the build to publish. +// as files (HTML, PDF, Markdown, an evidence pack) for the build to publish; +// each export of a changed report.json commits a revision to the report's own +// history, and each report shows its revision and the last change. // // The documents themselves are written elsewhere (a converter, or by hand); // this tab never edits a report.json. @@ -163,7 +165,8 @@ export default async function SiteReportsPage({ evidence pack (the page with its clips, stills and screenshots) — for the site's next build to publish beside the report. Preparing the evidence media exports too. A file over 24 MiB stays - on this machine. + on this machine. Exporting a changed report also commits a new + revision to the report's history, which the site publishes. </p> </div> <ExportReportsButton siteId={siteId} /> @@ -368,6 +371,18 @@ function ExportSummary({ row }: { row: ReportExportsRow }) { <span className="text-muted-foreground">not exported yet</span> )} </p> + {manifest?.revision && ( + <p data-testid="report-revision" className="text-xs"> + Revision {manifest.revision.revision} · {manifest.revision.date.slice(0, 10)} ·{" "} + <code title={manifest.revision.commit}>{manifest.revision.commit.slice(0, 12)}</code> + {manifest.revision.summary.length > 0 && ( + <span className="text-muted-foreground"> + {" "} + — {manifest.revision.summary.join("; ")} + </span> + )} + </p> + )} {manifest && ( <ul className="flex flex-wrap gap-x-4 gap-y-1 text-xs"> {REPORT_EXPORT_FORMATS.filter((f) => manifest.files[f]).map((f) => ( diff --git a/export/CHANGELOG.md b/export/CHANGELOG.md @@ -1,6 +1,7 @@ # Changelog ## [Unreleased] +- **A report shows its revision, and every edit to it can be checked.** Under a report's dates its page now reads "Revision N · <date> · history". The history page, `/reports/<id>/history/`, lists every revision, newest first: its number, date (UTC), commit hash and the sha256 of its `report.json`, what changed (claims added or removed, verdicts changed, claims edited, citations added or removed, quotes edited, title, series or subtitle changed), and each changed claim's title, text, verdict and findings with the words removed struck through and the words added marked. The same data is in `history.json` beside the page. Each report's history is its own git repository, published for cloning: `git clone <site>/reports/<id>/history/repo`. Each commit names the site as its author, with its date in UTC. The footer of the report's HTML, PDF and Markdown downloads now begins with the revision number, and its sha256 can be looked up on the history page. Needs `reports export` and a rebuild and deploy of each site with reports. - **A report can be saved whole: as one HTML page, a PDF, Markdown, or an evidence pack.** A report page's download line now reads HTML · PDF · Markdown · Evidence pack · Citations JSON · CSV, each listed only when the site publishes it. The HTML is one file that opens with no network: the report with its verdicts, the document's sentences and the post screenshots inside it, numbered citations, and a reference list giving each quote's speaker, date, record, the original at its time and the moment page on the site. The PDF is that page printed. The Markdown is the same report as plain text with numbered references. The evidence pack is a zip of the page with its clips, stills and screenshots beside it, so the clips play offline. Each ends with a line naming the report's date and the start of its checksum. Needs `reports export` (or prepare) and a rebuild and deploy of each site with reports. - **A report's claim can carry a flag, its title a byline, and a site with one report names it in the browser tab.** `report.json` claim `flag` (one line, at most 60 characters) shows as a small pill in the accent colour beside the claim's verdict, e.g. "No source given". On a report-only site with one report, the home page's tab title is the report's, as on the report's own page, where it was the site's title alone. A report's page follows its title with a byline from the document under review, "by <author> · <publisher>", in the same line when it fits; the publisher (or, with none, the author) links to the document. Under it, a step apart, the page names its own: "Fact-check by <site title>" ("Report by …"), then the dates, then the subtitle; the kind's label no longer sits above the title on the report's page. A claim's sentence from the document under review no longer links up to the document's box: it sits on a rail in the document's colour, and the box's left edge wears the same colour (a source's `accent`, `"#rrggbb"`; without one, the border colour). A sentence of another document keeps its "from <title>" link. A report's citation can say where its evidence came from (`origin`: `"subject"`, the document under review gave it; `"added"`, the report's author found it). A claim lists what the report added first, each card marked with the Archilyzer mark and "Not in the article" ("Not in the source"), then evidence of unknown origin, then what the document gave itself folded under "In the article (n)"; the reference list marks an added citation with the mark alone, and a claim's flag pill wears the same mark. A fact-check's page reads in three tiers, each opened by a hairline with one, two or three dots and its reading time (at 230 words a minute): the quick take (the tally, the summary, and links to what the check found, every claim and the downloads); **What the check found**, every ruled claim grouped by verdict (contradicted, not found, partly, untestable, corroborated), one line each linking to the claim, with its `gist` (a new optional claim field, one line, at most 240 characters) and its flag; and **Every claim, with its evidence**, which opens with **How it was checked** (`method`, a new optional report field in markdown). A report of kind `sweep` has the first and last tiers only. Needs a rebuild and deploy of the site. - **Forum posts read like the other posts.** A post from a forum-thread channel shows its place in the thread (#N), an "edited" mark, the thread's title and its media as links, and opening its thread shows its conversation — the posts it quotes and the posts quoting it — rather than the whole forum thread. diff --git a/export/app/components/reports/ReportArticle.tsx b/export/app/components/reports/ReportArticle.tsx @@ -1,4 +1,3 @@ -import Link from "next/link"; import { ArrowDownToLine } from "lucide-react"; import { AddedMark, CitationCard } from "yt-dlp-transcript-common/components/citations/CitationCard"; import { CitationsProvider } from "yt-dlp-transcript-common/components/citations/CitationsContext"; @@ -303,9 +302,9 @@ export default function ReportArticle({ view, siteTitle }: { view: ReportPageVie ? `Revision ${view.history.revision} · ${dateLabel(view.history.date)}` : `Edited since revision ${view.history.revision}`} {" · "} - <Link href={view.history.href} className={textLink}> + <a href={view.history.href} className={textLink}> history - </Link> + </a> </p> )} </div> diff --git a/export/app/components/reports/ReportHistory.tsx b/export/app/components/reports/ReportHistory.tsx @@ -1,4 +1,3 @@ -import Link from "next/link"; import { reportHistoryViewPath, type ClaimChange, @@ -118,9 +117,9 @@ export default function ReportHistory({ view }: { view: ReportHistoryView }) { <article data-report-history={view.reportId} className="mx-auto flex w-full max-w-3xl flex-col gap-6"> <header className="flex flex-col gap-3"> <p className="text-sm"> - <Link href={reportPagePath(view.reportId)} className={textLink}> + <a href={reportPagePath(view.reportId)} className={textLink}> <ReportName series={view.series} title={view.title} /> - </Link> + </a> </p> <h1 className="font-display text-3xl font-semibold leading-tight tracking-tight text-foreground">Revision history</h1> <p className="text-sm text-muted-foreground"> diff --git a/export/app/lib/reports.test.ts b/export/app/lib/reports.test.ts @@ -90,7 +90,7 @@ test("the report page: header, tally, sections and claims, inline cites, referen // then ours, a step apart: the kind by the site, the dates under it, then the subtitle assert.match( html, - /<\/h1><div[^>]*><p data-report-attribution=""[^>]*>Fact-check by Demo Reports<\/p><p[^>]*>Published 2026-10-01 · Updated 2026-10-04<\/p><\/div><p[^>]*>Four claims about a demo channel/, + /<\/h1><div[^>]*><p data-report-attribution=""[^>]*>Fact-check by Demo Reports<\/p><p[^>]*>Published 2026-10-01 · Updated 2026-10-04<\/p><p data-report-revision="2"[^>]*>Revision 2 · 2026-10-04 · <a href="\/reports\/demo-factcheck\/history\/"[^>]*>history<\/a><\/p><\/div><p[^>]*>Four claims about a demo channel/, ); assert.doesNotMatch(html.slice(0, html.indexOf("<h1")), /Fact-check/, "no kind eyebrow above the title"); assert.match(html, /data-subject-source="s0"/); diff --git a/export/app/reports/[reportId]/history/page.tsx b/export/app/reports/[reportId]/history/page.tsx @@ -1,13 +1,19 @@ import type { Metadata } from "next"; -import { reportFullTitle } from "yt-dlp-transcript-common/lib/report/views"; +import { REPORT_PLACEHOLDER_ID, reportFullTitle } from "yt-dlp-transcript-common/lib/report/views"; import ReportHistory from "../../../components/reports/ReportHistory"; import { EmptyState } from "../../../components/reports/parts"; -import { readReportHistory } from "../../../lib/reports"; +import { readReportHistory, reportIds } from "../../../lib/reports"; // /reports/<reportId>/history/: the report's revisions, from -// /reports/<reportId>/history/history.json. Built for every report the parent -// segment lists (and its placeholder); a report with no published history -// says so. +// /reports/<reportId>/history/history.json. Static: built for every published +// report (and, with none, the placeholder, as /reports/<reportId>/ is); a +// report with no published history says so. +export const dynamicParams = false; + +export function generateStaticParams(): { reportId: string }[] { + const ids = reportIds(); + return (ids.length > 0 ? ids : [REPORT_PLACEHOLDER_ID]).map((reportId) => ({ reportId })); +} type Params = { params: Promise<{ reportId: string }> }; diff --git a/export/e2e-report/audit.spec.ts b/export/e2e-report/audit.spec.ts @@ -33,6 +33,12 @@ test("it holds the reports, the moments, the cited media and the contract", () = "reports/demo-factcheck/report.md", "reports/demo-factcheck/evidence-pack.zip", "reports/demo-factcheck/stills/a01.png", + "reports/demo-factcheck/history/index.html", + "reports/demo-factcheck/history/history.json", + "reports/demo-factcheck/history/repo/HEAD", + "reports/demo-factcheck/history/repo/info/refs", + "reports/demo-factcheck/history/repo/objects/info/packs", + "reports/demo-factcheck/history/repo/refs/heads/main", "m/index.json", "m/demo-channel/abc123/3126.00-3151.00/index.html", "m/demo-channel/abc123/3126.00-3151.00/moment.json", diff --git a/export/e2e-report/contract.ts b/export/e2e-report/contract.ts @@ -1,8 +1,10 @@ // The cited fixture site's compose step (stage.ts runs it, in a child process // with the stage's environment): what compose's reports stage writes beside -// the views — the citation downloads and the report's exports (HTML, PDF, -// Markdown, evidence pack, by publish/reportExports.ts's writer) — and the -// cited site's contract, by +// the views — the citation downloads, the report's exports (HTML, PDF, +// Markdown, evidence pack, by publish/reportExports.ts's writer), its two +// revisions (fixture.ts FIXTURE_REVISIONS, committed by that writer into a +// store in the stage) and their published history (history.json and the +// dumb-HTTP clone, publish/reportHistory.ts) — and the cited site's contract, by // compose's own functions (bin/compose-site.ts emitFederationFiles / // emitAiFiles), so the stage's public/ is what compose would leave for this // site. The view JSON, stills and media are the fixture's, already in place. @@ -21,7 +23,10 @@ const { citationSet, citationsCsv } = await import("yt-dlp-transcript-common/pub const { MOMENTS_INDEX_PATH, REPORTS_INDEX_PATH, reportCitationsDownloadPath, reportViewPath } = await import( "yt-dlp-transcript-common/lib/report/views" ); -const { FIXTURE_DIR, readFixtureReport } = await import("../fixtures/report-site/fixture"); +const { FIXTURE_REVISIONS, buildFixtureReportView, fixtureReportFile, readFixtureReport } = await import( + "../fixtures/report-site/fixture" +); +const { publishReportHistory, readReportHistoryView } = await import("yt-dlp-transcript-common/publish/reportHistory"); const { openPlaywrightPdfPrinter, writeReportExports } = await import("yt-dlp-transcript-common/publish/reportExports"); const { sha256Hex } = await import("yt-dlp-transcript-common/publish/reportExportFiles"); const { REPORT_EXPORT_FILENAMES, REPORT_EXPORT_FORMATS, momentViewPath, reportExportDownloadPath } = await import( @@ -39,6 +44,7 @@ const readJson = <T>(urlPath: string): T => JSON.parse(fs.readFileSync(pub(urlPa const index = readJson<ReportIndexView>(REPORTS_INDEX_PATH); const moments = readJson<MomentIndexView>(MOMENTS_INDEX_PATH).moments; const report = readFixtureReport(); +const histories: string[] = []; for (const entry of index.reports) { if (entry.id !== report.id) throw new Error(`contract.ts: the fixture has no report "${entry.id}"`); const view = readJson<ReportPageView>(reportViewPath(entry.id)); @@ -50,43 +56,79 @@ for (const entry of index.reports) { // The report's exports, by the host step's own writer, from the files // already in public/ (its stills, the post's shot, the clips), published - // where compose publishes them. + // where compose publishes them. Each earlier revision is exported first (its + // Markdown, which commits it); the last export commits the report as it is. const printer = openPlaywrightPdfPrinter(); - const dir = path.join(fs.mkdtempSync(path.join(os.tmpdir(), "report-site-exports-")), entry.id); + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "report-site-exports-")); + const dir = path.join(tmp, entry.id); + const gitDir = path.join(tmp, "history-git"); + const files = { + local: (p: string) => (fs.existsSync(pub(p)) ? pub(p) : undefined), + clip: (c: { moment: string }) => { + const m = readJson<MomentPageView>(momentViewPath(c.moment)); + return m.clip + ? { sitePath: m.clip.src, file: pub(m.clip.src), kind: m.kind === "audio" ? ("audio" as const) : ("video" as const) } + : undefined; + }, + }; try { + for (const rev of FIXTURE_REVISIONS.slice(0, -1)) { + const earlier = readFixtureReport(rev.file); + const reportJson = fs.readFileSync(fixtureReportFile(rev.file)); + const r = await writeReportExports({ + dir: path.join(tmp, `earlier-${rev.file}`), + site, + report: earlier, + view: buildFixtureReportView(earlier), + reportSha256: sha256Hex(reportJson), + files, + ffmpegBin: paths.ffmpegBin, + formats: ["md"], + printer: () => printer, + zipBin: "zip", + now: new Date(rev.date), + log: console.log, + history: { gitDir, reportJson }, + }); + if (r.problems.length > 0) throw new Error(`contract.ts: revision ${rev.file}: ${JSON.stringify(r.problems)}`); + } + const reportJson = fs.readFileSync(fixtureReportFile()); const { result, problems } = await writeReportExports({ dir, site, report, view, - reportSha256: sha256Hex(fs.readFileSync(path.join(FIXTURE_DIR, "source", entry.id, "report.json"))), - files: { - local: (p) => (fs.existsSync(pub(p)) ? pub(p) : undefined), - clip: (c) => { - const m = readJson<MomentPageView>(momentViewPath(c.moment)); - return m.clip ? { sitePath: m.clip.src, file: pub(m.clip.src), kind: m.kind === "audio" ? "audio" : "video" } : undefined; - }, - }, + reportSha256: sha256Hex(reportJson), + files, ffmpegBin: paths.ffmpegBin, formats: REPORT_EXPORT_FORMATS, printer: () => printer, zipBin: "zip", - now: new Date(), + now: new Date(FIXTURE_REVISIONS[FIXTURE_REVISIONS.length - 1].date), log: console.log, + history: { gitDir, reportJson }, }); + if (result.manifest.revision?.revision !== FIXTURE_REVISIONS.length) { + throw new Error(`contract.ts: the fixture's last export is not revision ${FIXTURE_REVISIONS.length}`); + } if (problems.length > 0 || result.manifest.notes.length > 0) { throw new Error(`contract.ts: the fixture's exports: ${JSON.stringify([...problems, ...result.manifest.notes])}`); } for (const f of REPORT_EXPORT_FORMATS) { fs.copyFileSync(path.join(dir, REPORT_EXPORT_FILENAMES[f]), pub(reportExportDownloadPath(entry.id, f))); } + // The history, published as compose publishes it. + const history = await readReportHistoryView(gitDir, { reportId: entry.id, siteUrl: site.siteUrl }); + if (!history) throw new Error("contract.ts: the fixture has no revision history"); + await publishReportHistory({ gitDir, publicDir: paths.exportPublicDir, view: history }); + histories.push(entry.id); } finally { const p = await printer; if (!("missing" in p)) await p.close(); - fs.rmSync(path.dirname(dir), { recursive: true, force: true }); + fs.rmSync(tmp, { recursive: true, force: true }); } } await emitFederationFiles(site, paths); -await emitAiFiles(site, paths, { reports: index.reports, moments, allowed: [] }); +await emitAiFiles(site, paths, { reports: index.reports, moments, allowed: [], histories }); console.log(`[report-site] composed the contract of cited site "${site.siteId}" into ${paths.exportPublicDir}`); diff --git a/export/e2e-report/report-site.spec.ts b/export/e2e-report/report-site.spec.ts @@ -1,5 +1,13 @@ +import { execFileSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; import { expect, test } from "./helpers"; import type { Locator, Page } from "@playwright/test"; +import { reportChangeSummary, type ReportHistoryView } from "yt-dlp-transcript-common/lib/report/revisions"; +import { historyGitEnv } from "yt-dlp-transcript-common/publish/reportHistory"; +import { readFixtureReport } from "../fixtures/report-site/fixture"; // A cited site, as a reader meets it: its one report as the home page, the // report (verdicts, tally, the source's sentence, numbered inline citations @@ -10,12 +18,14 @@ import type { Locator, Page } from "@playwright/test"; // The site is export/fixtures/report-site, staged and built by // e2e-report/stage.ts: one fact-check citing a video span (c01), an audio // span (au1), a post (p01), the article's own sentences (a01, a02) and a web -// page (w01). +// page (w01). It has two revisions (report.v1.json, then report.json), and +// publishes their history. const REPORT = "/reports/demo-factcheck/"; const VIDEO_MOMENT = "/m/demo-channel/abc123/3126.00-3151.00/"; const AUDIO_MOMENT = "/m/demo-podcast/ep-042/610.50-628.00/"; const POST_MOMENT = "/m/demo-social/1234567890/"; +const HISTORY = "/reports/demo-factcheck/history/"; const inlineCite = (page: Page, id: string): Locator => page.locator(`[data-inline-cite="${id}"]`).first(); const preview = (page: Page, id: string): Locator => page.locator(`[data-cite-preview="${id}"]`); @@ -271,6 +281,85 @@ test("report.html opens with no network: one file, no script, every image inline expect(asked).toEqual([]); }); +test("the report names its revision beside its dates, and links its history", async ({ page }) => { + await page.goto(REPORT); + const line = page.locator("[data-report-revision]"); + await expect(line).toHaveText("Revision 2 · 2026-10-04 · history"); + await line.getByRole("link", { name: "history" }).click(); + await expect(page).toHaveURL(HISTORY); + await expect(page.locator("h1")).toHaveText("Revision history"); +}); + +test("the history lists both revisions, newest first: hashes, the change summary, a claim-level word diff", async ({ + page, + request, +}) => { + const history = (await (await request.get(`${HISTORY}history.json`)).json()) as ReportHistoryView; + expect(history.revisions.map((r) => r.revision)).toEqual([1, 2]); + await page.goto(HISTORY); + const revisions = page.locator("li[data-revision]"); + await expect(revisions).toHaveCount(2); + await expect(revisions.nth(0)).toHaveAttribute("data-revision", "2"); + for (const r of history.revisions) { + const li = page.locator(`li[data-revision="${r.revision}"]`); + await expect(li.locator("h2")).toHaveText(`Revision ${r.revision}`); + await expect(li.locator("[data-commit]")).toHaveText(r.commit); + await expect(li.locator("[data-report-sha256]")).toHaveText(r.reportSha256); + } + // The summary is the commit's, from the two report.json files. + const v1 = readFixtureReport("report.v1.json"); + const v2 = readFixtureReport(); + await expect(page.locator('li[data-revision="1"] [data-revision-summary] li')).toHaveText(reportChangeSummary(null, v1)); + await expect(page.locator('li[data-revision="2"] [data-revision-summary] li')).toHaveText(reportChangeSummary(v1, v2)); + // The diff: a verdict changed, words of a claim replaced, a claim added. + const r2 = page.locator('li[data-revision="2"]'); + const verdict = r2.locator('[data-claim-change="claim-1"] [data-field="verdict"]'); + await expect(verdict.locator("del")).toHaveText("PARTLY"); + await expect(verdict.locator("ins")).toHaveText("CONTRADICTED"); + const text = r2.locator('[data-claim-change="claim-2"] [data-field="text"]'); + await expect(text.locator("del")).toHaveText("month."); + await expect(text.locator("ins")).toHaveText("year."); + await expect(r2.locator('[data-claim-change="claim-5"]')).toHaveAttribute("data-status", "added"); + await expect(page.locator('li[data-revision="1"] [data-claim-change]')).toHaveCount(0); + await expect(page.locator("[data-history-clone]")).toHaveText( + "git clone https://reports.example.org/reports/demo-factcheck/history/repo", + ); + // The footer of every export names the revision and its report sha256. + const md = await (await request.get(`${REPORT}report.md`)).text(); + expect(md).toContain(`Revision 2 · 2026-10-04 · report sha256 ${history.revisions[1].reportSha256.slice(0, 12)}`); +}); + +test("the history on a phone: no page-wide horizontal scroll", async ({ page }) => { + await page.setViewportSize({ width: 375, height: 800 }); + await page.goto(HISTORY); + await expect(page.locator("li[data-revision]")).toHaveCount(2); + const overflow = await page.evaluate(() => document.documentElement.scrollWidth - document.documentElement.clientWidth); + expect(overflow).toBeLessThanOrEqual(0); +}); + +test("the history clones over HTTP: its report.json is the one history.json names", async ({ baseURL, request }) => { + const history = (await (await request.get(`${HISTORY}history.json`)).json()) as ReportHistoryView; + const dest = fs.mkdtempSync(path.join(os.tmpdir(), "report-history-clone-")); + try { + execFileSync("git", ["clone", "--quiet", `${baseURL}${HISTORY}repo`, path.join(dest, "repo")], { + env: historyGitEnv(), + stdio: "pipe", + }); + const sha = (b: Buffer) => createHash("sha256").update(b).digest("hex"); + expect(sha(fs.readFileSync(path.join(dest, "repo", "report.json")))).toBe(history.revisions[1].reportSha256); + const log = execFileSync("git", ["-C", path.join(dest, "repo"), "log", "--format=%H %s"], { + env: historyGitEnv(), + encoding: "utf8", + }); + expect(log.trim().split("\n")).toEqual([ + `${history.revisions[1].commit} Revision 2`, + `${history.revisions[0].commit} Revision 1`, + ]); + } finally { + fs.rmSync(dest, { recursive: true, force: true }); + } +}); + test("a video citation opens its moment: the clip, the cue lines with the span marked, where it is cited", async ({ page, request, @@ -339,7 +428,7 @@ for (const route of ["/ask/", "/downloads/", "/duplicates/"]) { } test("reading the whole site never asks for the corpus", async ({ page, traffic }) => { - for (const route of ["/", REPORT, VIDEO_MOMENT, AUDIO_MOMENT, POST_MOMENT, "/ask/", "/downloads/"]) { + for (const route of ["/", REPORT, HISTORY, VIDEO_MOMENT, AUDIO_MOMENT, POST_MOMENT, "/ask/", "/downloads/"]) { await page.goto(route); await page.waitForLoadState("networkidle"); } diff --git a/export/fixtures/report-site/fixture.ts b/export/fixtures/report-site/fixture.ts @@ -8,6 +8,11 @@ // `write.ts` writes them under public/, and fixture.test.ts holds the written // files equal to a fresh build, so the JSON can never drift from the builder. // +// THE REVISIONS. The report has two (common/publish/reportHistory.ts): +// source/demo-factcheck/report.v1.json, then report.json — FIXTURE_REVISIONS, +// committed at fixed dates by e2e-report/contract.ts. The page view names the +// second as current. +// // Everything here is neutral demo data. import fs from "node:fs"; @@ -16,6 +21,7 @@ import { fileURLToPath } from "node:url"; import { buildCitedIn } from "yt-dlp-transcript-common/lib/report/citedIn"; import { parseReport } from "yt-dlp-transcript-common/lib/report/validate"; import type { Report } from "yt-dlp-transcript-common/lib/report/schema"; +import { reportHistoryPagePath } from "yt-dlp-transcript-common/lib/report/revisions"; import { momentKeyOf, parseMomentKey, type SpanMoment } from "yt-dlp-transcript-common/lib/citations/moments"; import { MOMENT_INDEX_FORMAT, @@ -43,6 +49,13 @@ export const FIXTURE_DIR = path.dirname(fileURLToPath(import.meta.url)); export const FIXTURE_PUBLIC_DIR = path.join(FIXTURE_DIR, "public"); export const FIXTURE_REPORT_ID = "demo-factcheck"; +// The report's revisions, oldest first: the file under source/<id>/ and the +// commit's date. +export const FIXTURE_REVISIONS: readonly { file: string; date: string }[] = [ + { file: "report.v1.json", date: "2026-10-01T09:00:00Z" }, + { file: "report.json", date: "2026-10-04T12:00:00Z" }, +]; + const RECORDS: Record<string, RecordView> = { "demo-channel/abc123": { channel: "demo-channel", @@ -76,10 +89,12 @@ function shotFor(c: { channel: string; id: string }): string { return `/media/posts/${c.channel}/${c.id}/shot.png`; } -export function readFixtureReport(): Report { - const raw = JSON.parse( - fs.readFileSync(path.join(FIXTURE_DIR, "source", FIXTURE_REPORT_ID, "report.json"), "utf8"), - ); +export function fixtureReportFile(file = "report.json"): string { + return path.join(FIXTURE_DIR, "source", FIXTURE_REPORT_ID, file); +} + +export function readFixtureReport(file = "report.json"): Report { + const raw = JSON.parse(fs.readFileSync(fixtureReportFile(file), "utf8")); const parsed = parseReport(raw); if (!parsed.ok || parsed.problems.length > 0) { throw new Error(`fixture report is not sound: ${JSON.stringify(parsed.problems)}`); @@ -110,6 +125,13 @@ export function buildFixtureReportView(report = readFixtureReport()): ReportPage json: reportCitationsDownloadPath(report.id, "json"), csv: reportCitationsDownloadPath(report.id, "csv"), }, + // The newest revision, as compose names it (contract.ts commits both). + history: { + revision: FIXTURE_REVISIONS.length, + date: FIXTURE_REVISIONS[FIXTURE_REVISIONS.length - 1].date, + href: reportHistoryPagePath(report.id), + current: true, + }, }); } diff --git a/export/fixtures/report-site/public/reports/demo-factcheck/page.json b/export/fixtures/report-site/public/reports/demo-factcheck/page.json @@ -230,5 +230,11 @@ "zip": "/reports/demo-factcheck/evidence-pack.zip", "json": "/reports/demo-factcheck/citations.json", "csv": "/reports/demo-factcheck/citations.csv" + }, + "history": { + "revision": 2, + "date": "2026-10-04T12:00:00Z", + "href": "/reports/demo-factcheck/history/", + "current": true } } diff --git a/export/fixtures/report-site/source/demo-factcheck/report.v1.json b/export/fixtures/report-site/source/demo-factcheck/report.v1.json @@ -0,0 +1,150 @@ +{ + "format": "archilyzer-report", + "version": 1, + "id": "demo-factcheck", + "kind": "factcheck", + "title": "Checking an example article", + "subtitle": "Three claims about a demo channel, tested against what was said on air", + "summary": "The article makes four claims. The recordings partly support two and contradict a third; the fourth cannot be tested. The host said the opening date out loud [on stream](cite:c01), and later [repeated it](cite:au1).", + "method": "Each claim was searched for in the channel's transcripts and the guest's podcast, and checked against the city's public records.", + "published": "2026-10-01", + "subject": { + "source": "s0" + }, + "sources": { + "s0": { + "kind": "article", + "title": "An example article about a demo channel", + "url": "https://example.org/articles/demo", + "publisher": "Example Gazette", + "author": "A. Writer", + "date": "2026-09-20", + "note": "Read in the edition published on the day; it has since been revised.", + "accent": "#8a6fb0", + "archives": [ + { + "label": "archive.org", + "url": "https://web.archive.org/web/2026/https://example.org/articles/demo", + "context": "as published on the day" + } + ] + } + }, + "citations": { + "c01": { + "kind": "video", + "channel": "demo-channel", + "id": "abc123", + "start": 3126, + "end": 3151, + "pad": { + "before": 5, + "after": 5 + }, + "quote": "The bridge opened in the spring of twenty nineteen, I was there for it.", + "origin": "added", + "verification": { + "quoteScore": 0.97, + "quoteCheckedAt": "2026-10-04T12:00:00Z", + "voiceChecked": true, + "method": "cue-window v1" + } + }, + "au1": { + "kind": "audio", + "channel": "demo-podcast", + "id": "ep-042", + "start": 610.5, + "end": 628, + "quote": "Like I said before, spring twenty nineteen.", + "speaker": "The guest", + "verification": { + "quoteScore": 0.88, + "quoteCheckedAt": "2026-10-04T12:00:00Z" + } + }, + "p01": { + "kind": "post", + "channel": "demo-social", + "id": "1234567890", + "quote": "Never said I would move. Not once.", + "date": "2025-03-14" + }, + "a01": { + "kind": "source", + "source": "s0", + "quote": "He opened the bridge himself in 2018.", + "image": "stills/a01.png" + }, + "a02": { + "kind": "source", + "source": "s0", + "quote": "He has said many times that he would move away." + }, + "unused": { + "kind": "page", + "url": "https://example.org/unused", + "quote": "Not cited anywhere." + } + }, + "sections": [ + { + "id": "bridge", + "title": "The bridge", + "body": "The article's first chapter is about the bridge.", + "claims": [ + { + "id": "claim-1", + "title": "“He opened it himself”", + "text": "He opened the bridge himself in 2018.", + "verdict": "PARTLY", + "gist": "He was there; the date is in doubt.", + "sourceQuote": { + "citation": "a01" + }, + "findings": "He was there, but the date is in doubt: he says so [on stream](cite:c01).", + "citations": [ + "c01", + "au1" + ] + }, + { + "id": "claim-2", + "text": "He went back to the bridge every month.", + "verdict": "PARTLY", + "gist": "Once is on record; every year is not.", + "findings": "He mentions going back [once](cite:au1); nothing in the recordings says every year.", + "citations": [ + "au1" + ] + } + ] + }, + { + "id": "moving", + "title": "Moving away", + "claims": [ + { + "id": "claim-3", + "text": "He has said many times that he would move away.", + "verdict": "CONTRADICTED", + "gist": "He denies ever saying it.", + "sourceQuote": { + "citation": "a02" + }, + "findings": "The article [puts it as a habit](cite:a02); he [denies it outright](cite:p01).", + "citations": [ + "p01" + ] + }, + { + "id": "claim-4", + "text": "He privately regrets it.", + "verdict": "UNTESTABLE", + "flag": "No source given", + "findings": "Nothing public speaks to a private feeling." + } + ] + } + ] +} diff --git a/plans/report-sites.md b/plans/report-sites.md @@ -1,6 +1,7 @@ # Report sites — a site built around cited reports -Status: BUILT — R0–R8 merged 2026-10-05 (plus R4b page size); first private report site built locally. 2026-10-05: `publish` replaced by a per-site `search` switch; report-only is derived (search off); export's `start` serves moment dirs (`scripts/serve-out.mjs`). Follow-ups: cited corpus.json still describes shards, voice checks recorded in citations, saveSiteAction onto patchSite. +Status: BUILT — R0–R8 merged 2026-10-05 (plus R4b page size); first private report site built locally. RH +(per-report revision history) built 2026-10-05, see "History". 2026-10-05: `publish` replaced by a per-site `search` switch; report-only is derived (search off); export's `start` serves moment dirs (`scripts/serve-out.mjs`). Follow-ups: cited corpus.json still describes shards, voice checks recorded in citations, saveSiteAction onto patchSite. ## What it is @@ -157,6 +158,56 @@ A report must survive a takedown as files anyone can save and host again (slice stays local. The view's `downloads` lists what was published and the report page's download line shows HTML · PDF · Markdown · Evidence pack · Citations JSON · CSV. `reports/` is allowed wholesale by the cited audit. +## History + +Readers can check that a report is the one published and see every edit, with hashes (slice RH). History is +PER REPORT, never site-wide. + +- **Store** — a bare git repo per report beside its report.json: `sites/<siteId>/reports/<id>/history-git/` + (`common/publish/reportHistory.ts`; never a path segment named `.git`). It is inside the corpus repo's tree; + nothing writes a `.gitignore` for it — the operator adds `history-git/` there. One branch, `main`. +- **Revisions** — `reports export` (`reportExports.ts` `writeReportExports`, given the store and the report.json's + bytes) commits one when the sha256 of report.json differs from the newest revision's; an unchanged report commits + nothing. A commit holds `report.json` (byte for byte), `report.md` (the Markdown export) and `exports.json` + (format `archilyzer-report-revision-exports`: the sha256 and size of every export file made and of the citations + JSON and CSV). Message: `Revision N`, a blank line, `- ` lines of `reportChangeSummary` + (`common/lib/report/revisions.ts`, pure): title/series/subtitle changes, claims added/removed, verdict changes, + claims edited (title, text, findings), citations added/removed, quotes edited, summary/method edited; the first + revision counts what it holds; anything else is one "Other edits" line. Written with git plumbing + (`hash-object`, `mktree`, `commit-tree --no-gpg-sign`, `update-ref` with the old value). +- **Identity** — author and committer = the site's title, `noreply@<siteId>.invalid`; `GIT_AUTHOR_DATE` / + `GIT_COMMITTER_DATE` as `@<seconds> +0000`. git runs with `extendEnv: false` and a minimal env through + `cleanGitEnv`: PATH, HOME/XDG_CONFIG_HOME at a path that does not exist, `GIT_CONFIG_NOSYSTEM=1`, + `GIT_CONFIG_GLOBAL=/dev/null`, `TZ=UTC`, plus `-c commit.gpgSign=false -c core.hooksPath=/dev/null`. +- **Footer and commit** — an export's footer is `Revision N · <date> · report sha256 <12>`: N is the revision the + export belongs to (the next one when report.json changed), computed before the files are written. The footer + never names the commit: the commit holds report.md, whose footer would have to name the commit holding it. The + history page and `history.json` map each revision's report sha256 to its commit; `export.json` records the + revision, commit, date and summary once committed. +- **Publish** — compose reads every revision into `reports/<id>/history/history.json` (`ReportHistoryView`, format + `archilyzer-report-history`: per revision N, date, commit, report sha256, summary, and `claimChanges` — each + changed claim's title/text/verdict/findings as a word diff of `eq`/`ins`/`del` runs) and stages a dumb-HTTP + clone at `reports/<id>/history/repo/` as the source mirror does: a fresh `clone --bare --no-local + --single-branch`, `repack -a -d --max-pack-size=20m`, `pack-refs --all`, `update-server-info`, then an + allowlisted copy (`HEAD`, `packed-refs`, `info/refs`, `objects/info/packs`, `objects/pack/pack-*.{pack,idx}`, + `refs/heads/main`). The page view carries `history: { revision, date, href, current }` (`current` false when + report.json changed after the newest revision and was not exported again — the line then reads "Edited since + revision N"). The sitemap lists the history page. +- **Export site** — `/reports/<id>/history/` (static params as the report page's): `git clone <siteUrl>/reports/<id>/history/repo` + (the site-root path when `siteUrl` is unset), then each revision newest first with its hashes, summary and inline + `<ins>`/`<del>` diff. The report header shows "Revision N · <date> · history" under the dates. +- **Audit** — `reportHistoryProblem` (`lib/builtExport.ts`, asked by `builtSiteProblem` and `builtBundleProblem` + for every build) refuses any file in a `reports/<id>/history/repo/` outside the allowlist or over + `PUBLISH_MAX_FILE_BYTES`; the cited audit counts the clone's files toward Pages' 20,000. +- **Editor** — the Reports tab's Exports list shows each report's revision, commit and last change (from + `export.json`). +- **Tests** — `lib/report/revisions.test.ts` (summary, word diff, claim changes), `publish/reportHistory.test.ts` + (commit only on change; site identity and UTC dates with an operator's GIT_*/EMAIL/TZ set; no operator string in + any object; the staged clone served by `export/scripts/serve-out.mjs` and cloned over HTTP), compose and audit + tests; the report-site e2e fixture has two revisions (`source/demo-factcheck/report.v1.json`, then + `report.json`) and checks the page line, the history page's hashes, summary and diff, and a `git clone` over the + e2e server. + ## Compose and the contract - `compose-site` gains a reports stage: resolve citations against the shared transcript trees, verify quotes,