archilyzer

commit 98aade8b9f30e6b161633e4b8e65fbb479beb2f2
parent cfc0823643f85125239cd6766302131ffed927bd
Author: I Mean I'm Just Saying <imeanimjustsaying@kiwifarms.st>
Date:   Thu,  1 Oct 2026 12:32:49 -0400

plans: slice XL's review — the fixes in the record and FACTS (gallery-dl's containers, the WAL-only line, x.com-only logins, the refresh that keeps a logged-in jar, the --test-type wording), L2 and I2–I5 found and left, the operator's step 0, the gates after it

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Diffstat:
M
plans/FACTS.md
 | 
60
++++++++++++++++++++++++++++++++++++++++++------------------
M
plans/release-16.md
 | 
104
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------

2 files changed, 131 insertions(+), 33 deletions(-)
diff --git a/plans/FACTS.md b/plans/FACTS.md
@@ -8103,9 +8103,16 @@ source mirror (homepage)". Anchors are at the branch.
     AutomationControlled` is what turns it off (false, headed and headless).
   - A system Chrome then draws "You are using an unsupported command-line flag" for
     `--disable-blink-features=AutomationControlled`, and for Playwright's default `--no-sandbox`.
-    `--test-type` (what ChromeDriver passes) suppresses the first; `chromiumSandbox: true` removes
-    the second (the sandbox starts on this box for both builds). The bundled Chrome for Testing
-    draws neither (it honours Playwright's `--disable-infobars`).
+    `chromiumSandbox: true` removes the second (the sandbox starts on this box for both builds).
+    `--test-type` removes the first: it is Chromium's internal test-harness switch, which makes the
+    browser skip its startup bars (`AddInfoBarsIfNecessary` returns before the bad-flags prompt), and
+    it sets no automation signal (measured in review: `navigator.webdriver`, `window.chrome`, the
+    user agent and the plugin count are the same with and without it; `runtime_features.cc` turns
+    `AutomationControlled` on only for `--enable-automation`, `--headless` and the debugging pipe or
+    port). ChromeDriver passes `--test-type=webdriver`; the bare switch is used here. It is NOT the
+    documented route: that is the `CommandLineFlagSecurityWarningsEnabled` policy, machine-wide,
+    root to set, and it would silence the operator's everyday browser too. The bundled Chrome for
+    Testing draws neither bar (it honours Playwright's `--disable-infobars`).
   - Not hidden by any of this: the debugging pipe itself, and the headless user agent
     (`HeadlessChrome/…`). Google's sign-in may still refuse; X's password login is the reliable path.
 - **The launch options are one pure builder**, `buildXBrowserLaunchOptions`
@@ -8124,11 +8131,14 @@ source mirror (homepage)". Anchors are at the branch.
   chromium-1234 (151) and two older revisions; this Playwright does not use them.
 - **The profile/executable pairing.** A connect writes `archilyzer-browser.json` into the profile
   dir (`{ executablePath | null, version, recordedAt }`). The headless refresh and the Playwright
-  fallback fetcher open the profile with the bundled build (`launchXProfile`) and fall back to the
-  recorded executable only when the bundled launch throws. Measured: a profile written by system
-  Chromium 153 (a cookie added, headed and headless) opens in the bundled headless shell 147 with the
-  cookie's value readable; `Last Version` stays `153.0.8010.47`; 153 re-opens it after. So the
-  fallback is a guard, not the common path.
+  fallback fetcher open the profile with the bundled build and fall back to the recorded executable
+  when the bundled launch throws. **The refresh never replaces a jar holding an `auth_token` with one
+  that has none:** a bundled read that comes back without the login is read again with the recorded
+  executable, and when no read finds it the jar is kept and the refresh refuses, saying so — a
+  profile written by a newer system browser could open in the bundled build without an error and
+  without its cookies. Measured: a profile written by system Chromium 153 (a cookie added, headed and
+  headless) opens in the bundled headless shell 147 with the cookie's value readable; `Last Version`
+  stays `153.0.8010.47`; 153 re-opens it after. So the fallback is a guard, not the common path.
 - **The X login source, `social.x.cookieSource`** (`common/social/xCookieSource.ts`, pure).
   `"browser"` | `"profile"`; absent = **resolved at read time, never stored**: `"browser"` when
   `cookiesFromBrowser` is set and no profile is connected (connected = the broker's exported jar
@@ -8142,16 +8152,30 @@ source mirror (homepage)". Anchors are at the branch.
   (yt-dlp's syntax plus `/DOMAIN`) and reads every browser it supports, Chromium's encrypted store
   included, on each run. The spec is passed verbatim (`galleryDlCookieChoice`).
 - **Reading Firefox's store ourselves** (`common/social/xBrowserLogin.ts`): for the Playwright
-  fallback and the Settings "Check". The store is found as gallery-dl and yt-dlp find it — a
-  profile path or name from the spec, else the most recently modified `cookies.sqlite` under
-  `~/.config/mozilla/firefox`, `~/.mozilla/firefox`, Snap, Flatpak, macOS (two levels deep).
-  `cookies.sqlite` **and its `-wal`** are copied to a private `mkdtemp` dir (a running Firefox
-  keeps fresh rows in the WAL until a checkpoint; a fixture held open in WAL mode proves it), read
-  there, and removed; only X's rows are selected; the profile is never opened in place. A
-  container (`::NAME`) is a `userContextId` from `containers.json` (`::none` = outside every
-  container), as in yt-dlp. `expiry` is seconds (read as ms when too large for seconds);
-  `lastAccessed` is PRTime (µs). The Chromium family is NOT read here (its values are encrypted
-  with a keyring key): the readers say so and gallery-dl reads it.
+  fallback and the Settings "Check". The store is found much as gallery-dl finds it — a profile path
+  or name from the spec, else the most recently modified `cookies.sqlite` under
+  `~/.config/mozilla/firefox`, `~/.mozilla/firefox`, Snap, Flatpak, macOS (two levels deep). Not
+  exactly: gallery-dl also honours `$XDG_CONFIG_HOME` and a second Flatpak root
+  (`~/.var/app/org.mozilla.firefox/config/mozilla/firefox`), and for a profile NAME takes the first
+  root holding `<name>/cookies.sqlite` where this takes the newest.
+  `cookies.sqlite` **and its `-wal`** are copied back to back to a private `mkdtemp` dir, and the db
+  alone copied again from there; both are read and removed; only X's rows are selected; the profile
+  is never opened in place. **Two views:** with the WAL (what Firefox holds now — the Playwright
+  fallback uses it) and the main file alone (what gallery-dl sees: it opens `cookies.sqlite`
+  `mode=ro&immutable=1`, which ignores the WAL, and its fallback copies the db file only). A fresh
+  login sits in the WAL until Firefox checkpoints (closing Firefox does it); the Check says so when
+  the `auth_token` is in the WAL only.
+  **Containers are read as gallery-dl reads them** (`cookies.py` `_firefox_cookies_database`): no
+  `::CONTAINER`, or `::none`, = only cookies outside every container (gallery-dl's default; yt-dlp
+  reads them all); `::all` = no filter; `::NAME` = the `userContextId` whose `name` is NAME, else whose
+  `l10nId` is `user-context-NAME` (or ends `-NAME`), else whose `l10nID` is `userContextNAME.label` —
+  case-sensitive. The Check says when an x.com login sits only in a container the spec does not read.
+  **A login counts on x.com only** (`.x.com`, `x.com`, subdomains): gallery-dl's twitter extractor
+  looks its `auth_token` up on `.x.com`; a twitter.com `auth_token` is reported as the old domain's
+  and not counted (`ct0` is not needed — gallery-dl makes one).
+  `expiry` is seconds (read as ms when too large for seconds); `lastAccessed` is PRTime (µs). The
+  Chromium family is NOT read here (its values are encrypted with a keyring key): the readers say so
+  and gallery-dl reads it.
 - **`node:sqlite`** (Node 22.23 here; unflagged since 22.13, an ExperimentalWarning once per process)
   is loaded at run time through an assembled specifier with `webpackIgnore`
   (`common/social/nodeSqlite.ts`), like `playwrightRuntime.ts`: the editor's Turbopack build has
diff --git a/plans/release-16.md b/plans/release-16.md
@@ -826,9 +826,13 @@ build and the recorded executable is a fallback, not the common path.
   cannot start one). Connect logs the browser (`[x-session] Opening Chromium 153.0.8010.47 Arch
   Linux at /usr/bin/chromium (found on PATH). …`) and writes it into the profile
   (`archilyzer-browser.json`); its success note names it.
-- **The profile opens headless through one function**, `launchXProfile` (the refresh and the
-  Playwright fallback fetcher): the bundled build with the same signals dropped, and the recorded
-  executable only when the bundled launch throws.
+- **The profile opens headless the same way for the refresh and the Playwright fallback fetcher**
+  (`launchXProfile`, and the refresh's own pass): the bundled build with the same signals dropped,
+  the recorded executable when the bundled launch throws. **The refresh never replaces a logged-in jar
+  with one without a login** (the review's L4): a bundled read with no `auth_token` while the jar holds
+  one is read again with the recorded executable, and when no read finds the login the jar is kept
+  and the refresh refuses, saying so ("…the logged-in cookie jar exported <when> was kept, not
+  replaced…").
 - **The login source, `social.x.cookieSource`** (`common/social/xCookieSource.ts`, pure; the one new
   settings key, `SETTINGS.md` and `settings.json.example` regenerated): `"browser"` | `"profile"`,
   absent by default and **resolved at read time** — `"browser"` when `cookiesFromBrowser` is set and
@@ -847,14 +851,22 @@ build and the recorded executable is a fallback, not the common path.
   - **Nitter** uses no X login and is unchanged.
 - **`xCookiesFromBrowser(spec)` and `readXLoginStatus(paths, settings)`**
   (`common/social/xBrowserLogin.ts`). The read takes the resolved SPEC, not the settings, so a
-  channel's own `cookiesFromBrowser` holds. **Firefox only, read-only**: the store is found as
-  gallery-dl and yt-dlp find it (a profile path or name, else the most recently modified
-  `cookies.sqlite` under the Firefox roots), `cookies.sqlite` and its `-wal` are copied into a private
-  `mkdtemp` dir, X's rows alone are selected, the copy is removed; a `::CONTAINER` is honoured.
-  **Chromium's encrypted store is not read here** — gallery-dl (and yt-dlp) read it; for such a spec
-  the Check and the fallback say so. The status: the source in use and whether it was chosen, whether
-  an `auth_token` for x.com is visible (`null` when the source cannot be read), and when it was last
-  seen — the browser's own `lastAccessed` for that cookie, or the jar's export time for the profile.
+  channel's own `cookiesFromBrowser` holds. **Firefox only, read-only**: the store is found much as
+  gallery-dl finds it (a profile path or name, else the most recently modified `cookies.sqlite` under
+  the Firefox roots), `cookies.sqlite` and its `-wal` are copied into a private `mkdtemp` dir, X's rows
+  alone are selected, the copies are removed. **It reads what gallery-dl reads** (the review's M1, L3):
+  the container as gallery-dl reads it — no `::CONTAINER` (or `::none`) is only the cookies outside
+  every container, `::all` is every container, `::NAME` one container matched as gallery-dl matches —
+  and a login counts only as an `auth_token` on x.com. Two views of one copy: with the WAL (what
+  Firefox holds now; the Playwright fallback uses it) and the main file alone (what gallery-dl's
+  immutable open sees). **Chromium's encrypted store is not read here** — gallery-dl (and yt-dlp) read
+  it; for such a spec the Check and the fallback say so. The status: the source in use and whether it
+  was chosen, whether an `auth_token` for x.com is visible (`null` when the source cannot be read), and
+  when it was last seen — the browser's own `lastAccessed` for that cookie, or the jar's export time
+  for the profile; and, for the browser, three plain lines when they apply: the login is in Firefox's
+  write-ahead log only ("Seen in Firefox's write-ahead log only; gallery-dl will see it after Firefox
+  checkpoints (closing Firefox does it)."), an old-domain cookie (a twitter.com `auth_token`) is
+  present and not used, and an x.com login sits only in a container the spec does not read.
   `node:sqlite` is loaded at run time (`common/social/nodeSqlite.ts`).
 - **The Settings section** (`XSessionSection.tsx`, `xSessionActions.ts`, `settings/page.tsx`): the
   intro names the two sources; a **Login source** select (Automatic — now: …, Browser login,
@@ -878,7 +890,9 @@ needs `{ exact: true }`, the next one contains it), `x cookie source in use`, `c
 | `cfb6dd10` | `common:` the Connect browser and its options, the login source, the browser-login reader and status, the fetchers, the `social` settings block; SETTINGS.md, settings.json.example, ENVIRONMENT.md |
 | `a266b18a` | `editor:` the X account session's source select, Check and Connect text; the actions; `x-session.spec.ts` |
 | `f295c263` | `plans:` this section; FACTS; the editor changelog |
-| this commit | `plans:` the e2e run |
+| `457f5961` | `plans:` the e2e run |
+| `b3ee652b` | `common:` the review's fixes — M1 (gallery-dl's containers), L1 (the WAL-only line), L3 (x.com-only logins), L4 (the refresh keeps a logged-in jar; `xSessionBroker.test.ts`), I1 (the `--test-type` wording) |
+| this commit | `plans:` the review, its record and FACTS lines, the gates after it |
 
 #### Gates (logs `$T/xl-*.log`)
 
@@ -924,12 +938,29 @@ The operator verifies by hand after the restart (below).
 - **Not run against the operator's own Firefox.** No run read a real browser profile, so whether
   `firefox` resolves to the profile the operator logs in to X with (the most recently used one) is
   the first thing the Check shows after the restart.
+- **Store discovery is close to gallery-dl's, not identical** (review L2). gallery-dl also honours
+  `$XDG_CONFIG_HOME` and a second Flatpak root (`~/.var/app/org.mozilla.firefox/config/mozilla/firefox`),
+  and for a profile NAME it takes the first root holding `<name>/cookies.sqlite` where this reader
+  takes the newest across roots. On a stock Linux Firefox (one root in use) both pick the same store.
+- **From the restart on, every X fetch runs as the operator's own X account** (review I2): the live
+  `settings.json` has `cookiesFromBrowser: "firefox"`, no `social` key and no exported jar, so the
+  default is the browser and every X fetch, scheduled syncs included, runs logged in as the everyday
+  account with no click. As ruled; the operator's steps below say it.
+- **gallery-dl is handed the whole spec** (review I3): `--cookies-from-browser firefox`, with no
+  `/DOMAIN`, loads every cookie outside a container from the operator's Firefox into gallery-dl's
+  process (it sends only X's to X). Giving gallery-dl's argv alone `firefox/.x.com…` would narrow it;
+  not done.
+- **A checkpoint between the two copies** (review I4): the db is copied before its WAL, so a
+  checkpoint and WAL reset in that instant can pair them wrongly — an "unreadable" result, reported as
+  such, or a stale view. Rare; no retry was added.
+- **The whole `cookies.sqlite` sits in a private temp dir during a read** (review I5): every site's
+  rows, mode 0700 under `os.tmpdir()`, removed in a `finally`; only a hard kill leaves it behind.
 
 #### Decisions the operator could overturn
 
 | What I did | The alternative |
 |---|---|
-| `--test-type` in the launch args and the sandbox on for the headed window, so a system Chrome draws no warning bar | The ruling's two flags alone: the window shows "unsupported command-line flag" for the blink flag (or for `--no-sandbox`) |
+| `--test-type` in the launch args and the sandbox on for the headed window, so a system Chrome draws no warning bar. `--test-type` is Chromium's internal test-harness switch that skips the startup bars, not the documented route (the machine-wide `CommandLineFlagSecurityWarningsEnabled` policy); it sets no automation signal (measured in review). ChromeDriver passes `--test-type=webdriver`; the bare switch is used | The ruling's two flags alone: the window shows "unsupported command-line flag" for the blink flag (or for `--no-sandbox`) |
 | The refresh and the fallback drop the same signals headless (`launchXProfile`) | Leave them on Playwright's defaults; the ruling named only the Connect window |
 | `ARCHILYZER_X_BROWSER` is a runtime variable read in `xBrowser.ts` | A `paths` entry in `getPaths()` beside `GALLERY_DL_BIN` (it would add a field every `Paths` literal must carry) |
 | "Automatic" is a third option in the select, storing nothing | Two options only: the read-time default would apply until the first choice and never again |
@@ -939,16 +970,59 @@ The operator verifies by hand after the restart (below).
 
 #### After the restart, by hand (the operator)
 
+0. **Every X fetch, scheduled syncs included, runs as your own account from now on** — your
+   everyday Firefox's X login, with no click (the default with `cookiesFromBrowser: "firefox"` and no
+   connected profile). If that is not wanted, choose **Connected profile** on `/settings` straight
+   after the restart, before the next scheduled X sync.
 1. `/settings` → X account session: **In use** should read `Browser login (firefox) (automatic)`
    (the live `settings.json` has `cookiesFromBrowser: "firefox"` and no exported jar). **Check**: an
-   X login visible in firefox, with when its `auth_token` was last used. If it says none, log in to
-   x.com in that Firefox profile and Check again.
+   X login visible in firefox, outside its containers, with when its `auth_token` was last used. If it
+   says none, log in to x.com in that Firefox profile (not in a container — or set
+   `cookiesFromBrowser` to `firefox::<container>`) and Check again. If it says "Seen in Firefox's
+   write-ahead log only", gallery-dl will not see the login until Firefox checkpoints: close Firefox
+   once (or wait), then Check again before step 2.
 2. One X channel's **Fetch posts**: the job log says `X login: the browser (firefox), read by
    gallery-dl.` before gallery-dl runs.
 3. Optionally **Connect X account**: the window should be Chromium 153, with no "controlled by
    automated test software" bar; log in with X's username and password (not Google), close the
    window. The source stays as chosen; on Automatic it moves to the connected profile.
 
+
+#### Review
+
+**Verdict: SHIP AFTER FIXES** (`xl-review.md` in the job's scratch): one Medium, four Lows, five
+infos. The coordinator ruled M1, L1, L3 and L4 in, the I1 wording, and L2 and I2–I5 into "Found and
+left" (I2 also into the operator's steps).
+
+| Finding | Where |
+|---|---|
+| M1: with no `::CONTAINER` the reader took cookies from every container (yt-dlp's default); gallery-dl, the fetcher this login feeds, takes only cookies outside every container and accepts `::all` — so the Check could say "visible" for a login gallery-dl never reads | `b3ee652b`: `firefoxContainerScope` / `inContainerScope` mirror gallery-dl 1.32.9's `_firefox_cookies_database` and its SQL (none = no container, `::all` = no filter, `::NAME` by `name`, `l10nId`, `l10nID`, case-sensitive); the container test expects `["default"]` with no container and has `::all`, `l10nId` and a 2-vs-12 case; the Check names a login that sits only in another container |
+| L1: the Check read the WAL; gallery-dl opens the db `immutable=1`, which ignores it, so just after a login the Check could say "visible" while gallery-dl runs as a guest | `b3ee652b`: one copy, two views (with the WAL; the main file alone); a login in the WAL only adds "Seen in Firefox's write-ahead log only; gallery-dl will see it after Firefox checkpoints (closing Firefox does it)."; the operator's step 1 says what to do |
+| L2: store discovery not exactly gallery-dl's | "Found and left"; FACTS and the module comment no longer say "as gallery-dl and yt-dlp find it" |
+| L3: "visible" also counted an `auth_token` on twitter.com; gallery-dl looks on `.x.com` | `b3ee652b`: `isXComAuth` counts x.com only (the Check and the fallback's warning); a twitter.com `auth_token` is reported as "An old-domain cookie is present too … gallery-dl does not use it." |
+| L4: the refresh fell back to the recorded browser only on a throw; a bundled read that opens a newer profile without its cookies would replace a logged-in jar with an empty one | `b3ee652b`: a bundled read with no `auth_token` while the jar holds one is read again with the recorded executable; when no read finds it the jar is kept and the refresh refuses, saying so. `refreshXCookies` takes an injectable `chromium`; `xSessionBroker.test.ts` (6 cases: login found, empty bundled read → recorded, nothing found → jar kept, no record → jar kept, a jar without a login replaced, bundled throws → recorded) |
+| I1: `--test-type` verified to set no automation signal; the wording "what ChromeDriver passes" was loose, and it is not the documented route | `b3ee652b` (`xBrowser.ts`'s comment), FACTS, the decisions table: Chromium's internal test-harness switch that skips the startup bars; the documented route is the machine-wide `CommandLineFlagSecurityWarningsEnabled` policy; ChromeDriver passes `--test-type=webdriver`, the bare switch is used |
+| I2: from the restart every X fetch runs as the operator's account | "Found and left"; the operator's step 0 |
+| I3: gallery-dl is handed the whole spec (every non-container cookie loads into its process) | "Found and left" |
+| I4: a checkpoint between the db and WAL copies can pair them wrongly | "Found and left" |
+| I5: the whole store sits in a private temp dir during a read | "Found and left" |
+
+**Gates after the review** (at `b3ee652b`; logs `$T/xl-tsc3.log`, `xl-common2.log`, `xl-build2.log`,
+`xl-e2e2.log`, `xl-e2e3.log`). `git merge main` was a no-op: `main` was still `a2229d68` (RM had not
+landed).
+- tsc (all workspaces) clean, 59 s. common **2,450/2,450**, 82 s (+11: the container case rewritten,
+  the two-views and x.com-domain cases, three status cases, `xSessionBroker.test.ts`'s 6). Editor unit
+  **109/109**. `archilyzer docs files --check`, `settings example --check`, `docs env --check`: exit 0.
+- The capped editor build with the corpus linked: exit 0, 69 s, 1.65 GB peak RSS, no warnings; the link
+  removed and the worktree's own `transcripts/` put back.
+
+  | Run | At | Specs | Result |
+  |---|---|---|---|
+  | 2 | `b3ee652b` | `x-session`, `settings` (also matches `operation-settings`), `social-channel` | 18 passed, **1 failed**, 2.5 min — `social-channel` "fetch-posts writes month-sharded JSONL…" hit the 30 s test timeout. The dev server was cold (started straight after the build), and the case is slow on this host anyway: 23.8 s in run 1, its offline neighbours about 12 s each. The gallery-dl path gained only a `stat` of the session dir |
+  | 3 | `b3ee652b` | `social-channel`, `x-session` | **7 passed**, 0 failed, 1.8 min (the fetch-posts case 25.7 s) |
+
+  Not measured: the same case on `main`'s code, to say how much of its time is older than this slice.
+
 ## Rollout
 
 Both slices are export- and homepage-side; the editor and umtool are not rebuilt for this release.