From ae3db89ab3f886b4abb872bf8416bd02c8d7fc0e Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 26 Oct 2026 22:12:32 +0100 Subject: [PATCH] night fixes 2026-10-26: live proofs (R-1220, R-1232, R-1187, R-1298), register states, design lines; R-1300 filed; R-528 design Evidence audits/night-fixes-2026-10-26/ (scanned: 0 secret hits). --- CONTEXT.md | 11 ++ STATUS.md | 8 +- documentation/architecture/03-host-agent.md | 2 +- .../architecture/05-hub-architecture.md | 2 +- .../architecture/07-backup-architecture.md | 6 + documentation/architecture/08-alarm-ladder.md | 2 + .../audits/night-fixes-2026-10-26/README.md | 67 +++++++ .../r1187-9202-run2.txt | 33 ++++ .../r1187-9202-run3.txt | 32 ++++ .../r1187-r1220-9202-run1.txt | 29 ++++ .../night-fixes-2026-10-26/r1187g-9202.txt | 27 +++ .../r1232/1-before-old-agent.txt | 41 +++++ .../r1232/2-after-fixed-agent.txt | 16 ++ .../r1232/3-first-run.txt | 35 ++++ .../r1232/install-console.txt | 163 ++++++++++++++++++ .../r1232/install-start.txt | 1 + .../night-fixes-2026-10-26/r1232/teardown.txt | 4 + ...uninstall-console-1-wrong-confirmation.txt | 26 +++ .../r1232/uninstall-console.txt | 84 +++++++++ .../r1298/clear-holds.txt | 18 ++ .../r1298/privatebin-remove.txt | 2 + .../tools/r1187_r1220_live.py | 125 ++++++++++++++ .../tools/r1187g_live.py | 100 +++++++++++ documentation/backlog/OPEN-ITEMS.md | 17 +- 24 files changed, 837 insertions(+), 14 deletions(-) create mode 100644 documentation/audits/night-fixes-2026-10-26/README.md create mode 100644 documentation/audits/night-fixes-2026-10-26/r1187-9202-run2.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/r1187-9202-run3.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/r1187-r1220-9202-run1.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/r1187g-9202.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/r1232/1-before-old-agent.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/r1232/2-after-fixed-agent.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/r1232/3-first-run.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/r1232/install-console.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/r1232/install-start.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/r1232/teardown.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/r1232/uninstall-console-1-wrong-confirmation.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/r1232/uninstall-console.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/r1298/clear-holds.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/r1298/privatebin-remove.txt create mode 100644 documentation/audits/night-fixes-2026-10-26/tools/r1187_r1220_live.py create mode 100644 documentation/audits/night-fixes-2026-10-26/tools/r1187g_live.py diff --git a/CONTEXT.md b/CONTEXT.md index 0afb63a4..1e7138d8 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,17 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-10-26 night (alone) — the data-safety fixes, on `main`, NOT released.** Controller: R-1220 `7e15a37` +> (`liveDriveBinds(drives)`: an unreadable app.yaml = installed), R-1218/R-1219 `1170b48` (stage when the password FILE +> exists; no ceremony during a recovery offer), R-1187 `dfd0f8e`…`90a2530` (.fab: links as links, FIFO refused, `os.Root` +> writes, downward-only links, owners/modes, busy and bad-path refusals; the import's compose gets `USERDATA_PATH`/ +> `IMPORT_PATH`; the DB loads through `appbackup.ImportDump` — the import's `mysqladmin`/`mysql` broke every MariaDB 11+ +> import), R-1298 `b156e45`, Go 1.27.3 `2560cf1`. Agent: R-1232 `3911cc6` (own-key archives only in the due-check), +> Go 1.27.3 `fbbb168`. Hub: R-1218 `09b5f02f` (409 on a key-less package over a keyed one), Go 1.27.3 `d625a143`. +> Live: `audits/night-fixes-2026-10-26/` (R-1220, R-1187, R-1298 on 9202 with local test images; R-1232 on a BYO Tester4 +> VM, uninstalled and destroyed). R-1218's live shape needs the operator (off-site off on Tester 1) — else reverted +> before the release. R-528: design in the row, not built. New: R-1299, R-1302. + > **2026-10-26 (night) — D55 = 260 (`09` §3 decision 268); the grant is SET.** CC used the operator's one-purpose token > `HUB_TOKEN_D55_WINDOW` („cc-d55-window", Off-site write only) once: `POST /offsite/window-grant/demo-hp` `max_remove=260` > → 200, 19:10:31Z; read back `offsite_window_grant:demo-hp = max:260` (hub DB with WAL) and the hub log line „granted ONE diff --git a/STATUS.md b/STATUS.md index 29d05145..2604a826 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-10-26 (night).** Your D55 token worked: demo-hp gets one clean-up window with a cap of 260 at its next nightly off-site run. Tonight CC fixes data-safety holes (saved only); Tuesday: the line test, then each release on your yes. +**Updated 2026-10-26 (night).** Saved and proven tonight, not released: kept data never offers a live app's folder; a reinstalled box backs up off-site at once; app-file import keeps links and owners and works again for MariaDB apps; Go 1.27.3. D55's window runs tonight. ## What works - **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241). @@ -23,8 +23,8 @@ - DooPlex: something sets the shared checkout to „bare" (R-935). ## What we work on -- Released: 0.345.1 (cloudflared 2026.10.3, security), every box; `main` and Sunday's golden carry it. Built, not released: a BYO box removes our SSH file in the same update (R-1285; the agent waits for a live BYO proof); held apps reach you (R-1291, controller + hub, proven on a test hub). -- Dated checks, run by CC. 2026-10-27: the qBittorrent line test on a seeding box, with DooPlex's qBittorrent as the peer (R-1250, your yes). Done 2026-10-26: the weekly clean-up windows removed only old copies and raised no false alarm. 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes). +- Released: 0.345.1 (cloudflared 2026.10.3, security), every box. Built, not released (Tuesday's block, each on your yes): R-1291, R-1220, R-1232, R-1187, R-1219, R-1298, R-972, R-1285. R-1218 needs you: off-site off on Tester 1 for minutes (hub customer page), or it leaves the release. +- Dated checks, run by CC. 2026-10-27: the line test on a seeding box, DooPlex's qBittorrent as the peer (R-1250, your yes). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes). ## Waiting on you — decide @@ -48,7 +48,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14. 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12. -**2026-10-26.** On the hub's Hosts page, delete `Tester4-b5ec39` (CC's test VM, destroyed). If not: the hub alarms that the box went silent. After demo-hp's window has run (CC says when, Tuesday morning), revoke the token `cc-d55-window` (R-1294). +**2026-10-26.** On the hub's Hosts page, delete `Tester4-b5dbbd` (CC's test VM of tonight, uninstalled and destroyed; you already deleted `Tester4-b5ec39`). If not: the hub alarms that the box went silent. After demo-hp's window has run (CC says when, Tuesday morning), revoke the token `cc-d55-window` (R-1294). **2026-10-23 do list.** 1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens. 2. On the hub's Hosts page: Discard appliance 49. If not: a dead entry alarms. diff --git a/documentation/architecture/03-host-agent.md b/documentation/architecture/03-host-agent.md index 6741a47b..8c0ff16f 100644 --- a/documentation/architecture/03-host-agent.md +++ b/documentation/architecture/03-host-agent.md @@ -510,7 +510,7 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba backup is due (`GET /backup/due`, §6) → **quiesces** (stops its app stacks) → `POST /backup` → polls `GET /backup/status` to `done` → **unquiesces** (restarts exactly the stacks it stopped). Implemented in `felhom-controller` v0.36.0 (`internal/quiesce`) + `felhom-agent` v0.11.0 (the - `/backup/due` cadence policy + `/backup/status` phases). **An agent-initiated vzdump is + `/backup/due` cadence policy + `/backup/status` phases). *[DESIGN] R-1232 (2026-10-26 night, agent main `3911cc6`, next agent release): on an encrypted tier the due-check counts only archives written with THIS storage's key (the R-727 discriminator — an archive carries no host id), so a reinstalled box does not read the old box's archive as its own last backup.* **An agent-initiated vzdump is crash-consistent only** (there is no inbound-to-guest channel to trigger a quiesce — §3/§5); the controller stopping its stacks first is what makes the captured state **clean-shutdown-consistent** (validated live: a quiesced postgres restore comes up clean — "database system was shut down" — vs diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index 9d90bba8..536a3e89 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -300,7 +300,7 @@ What remains: **`host_escrow.blob`** (and the identity blob beside it). **[DESIGN] 2026-10-24 (R-1254; committed felhom.eu `dcf4f84c`, hub next v0.170.0 — NOT deployed):** the escrow PUT stores the blob, its sealed-password hash and the identity blob in ONE transaction (`Store.SaveHostEscrowWithIdentity`; - the superseded copy is made inside it). A failure leaves all of the old package or all of the new — never the new hash + the superseded copy is made inside it). *[DESIGN] R-1218 (2026-10-26 night, hub main `09b5f02f`, next hub release): a package that seals the off-site repository password is never replaced by one that seals none — the PUT answers 409 and the good package stays current (it used to be demoted to operator-only custody, so the household's code no longer opened its off-site copy).* A failure leaves all of the old package or all of the new — never the new hash beside the old identity package, which a box would read as „escrowed" while the written code opened nothing. A malformed identity blob is refused before anything is stored. (The agent ceremony's „the previous code still works" after a lost answer, R-1254's other half, stays open.) diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 9b175da8..043dea5d 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -403,6 +403,8 @@ never moved by an update or an undo, and the unit restore accepts it. > copy protected the app's files. Since v0.245.0 every dashboard page carries the reminder (the R-241 > bar, second instance) and the Tier-1 sentence renders by state — „védené … szünetel" while paused. > Measured on a fresh box 2026-09-16: zero snapshots, and the page said the files were protected. +> +> *[DESIGN] R-1219 / R-1218 (2026-10-26 night, controller main `1170b48`, next release): during a recovery offer the bar does NOT urge the ceremony (a new code would replace the key that opens the old off-site copy) and the start answers 409; a ceremony stages the off-site password whenever its FILE is on the box, also with off-site switched off or cleared.* > **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does. @@ -923,6 +925,8 @@ run makes it longer. *Second-drive copies that are KEPT (D25) and the rule that a rejected backup block never deletes one: §8.2a (R-1252, 2026-10-24).* +*[DESIGN] R-1220 (2026-10-26 night, controller main `7e15a37`, ships with the next controller release): an app whose `app.yaml` EXISTS but does not parse counts as INSTALLED — every folder its compose binds, on every registered drive, is live and never listed as kept data (fail closed; one WARN names the app).* + **What it is.** "Remove the app, keep my data" leaves the app's drive folder (`/appdata/`, or the folder its definition binds through `${HDD_PATH}`) in place. A reinstall over it no longer runs silently into the old files (R-657): the install asks „A megőrzött adataimat használom" / "Use my kept data" (the database from the newest copy of @@ -1080,6 +1084,8 @@ second drive's whole restore (its `CheckOnly` pass carries the pull, so its file load from a unit and from off-site, and a set-aside unit (D47) — the off-site restore, the second drive's file restore and the no-unit fallback. Not covered: the `.fab` import — it removes the running app's volumes before its definition is pulled (R-1279). +*[DESIGN] The `.fab` route, R-1187 (2026-10-26 night, controller main, next release): the export carries a symlink AS A LINK and stops on a FIFO, socket or device naming its path; a drive folder that cannot be packed fails the export (never left out). The import writes only through `os.Root`, keeps owners and modes, restores a link only when it points DOWNWARD inside the folder (no absolute target, no „..”, no link on the way — others refused and listed), refuses while another operation holds the box and refuses an empty or relative `HDD_PATH`; its compose runs get the controller's own `USERDATA_PATH`/`IMPORT_PATH`; the database loads through the backup side (`appbackup.ImportDump`: mariadb / mariadb-admin, psql ON_ERROR_STOP) — the import's own `mysqladmin`/`mysql` calls failed on every MariaDB 11+ image. Live: `audits/night-fixes-2026-10-26/`.* + ### 6.7 Why an off-site run failed — the failure classifier (R-571) **[FACT, read from source 2026-10-05, felhom-controller `114ff27`]** When an off-site (restic) run fails, the diff --git a/documentation/architecture/08-alarm-ladder.md b/documentation/architecture/08-alarm-ladder.md index ef3e446c..7ff72eee 100644 --- a/documentation/architecture/08-alarm-ladder.md +++ b/documentation/architecture/08-alarm-ladder.md @@ -660,3 +660,5 @@ a row the box stopped repeating for 50 h (the list still shows it, marked). `ale every grace — a test configuration only. Live proof: `audits/d53-held-apps-2026-10-26/` (a crash-looped throwaway app on 9202 against a test hub with a 3-minute grace: listed on both pages, one operator mail, Start cleared it, the row went, every household leg `skipped operator_only`). + +*[DESIGN] R-1298 (2026-10-26 night, controller main `b156e45`): an install's hold (`unhealthy_stop`, `update_failed`, `putback_pending`) of an app that is NOT installed (no `app.yaml`) is not reported — it would remind weekly about an app that does not exist; a restore hold is always reported (R-379, a person clears it).* diff --git a/documentation/audits/night-fixes-2026-10-26/README.md b/documentation/audits/night-fixes-2026-10-26/README.md new file mode 100644 index 00000000..482d2a66 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/README.md @@ -0,0 +1,67 @@ +# Night fixes 2026-10-26 → 27: live proofs (R-1220, R-1232, R-1187, R-1298) + +Code on `main`, NOT released (the release block is Tuesday afternoon, attended). Test images were **local only** +(`0.346.0-night1/2/3`, `docker save`/`load` into 9202 — never pushed to the registry); the test agent +(`0.167.0-night1`) was a local build copied to the test VM only. Method: the endpoints the UI invokes, through the box +walk tools (`../catalog-sweep-2026-10-24/tools/`), and the product's own installer / uninstaller. + +## R-1220 — an unreadable `app.yaml` counts as installed (controller `7e15a37`) + +9202, throwaway Grimmory, its `app.yaml` made unreadable (byte copy saved, put back): after a stack rescan the +„Megőrzött adatok" page did **not** list its drive folder, and the controller logged the R-1220 WARN. +`r1187-r1220-9202-run1.txt` (step 2). PASS. + +## R-1232 — a reinstalled box's first whole-box copy is not a week late (agent `3911cc6`) + +VM 9375 on demo-hp's scratch NVMe: plain Proxmox (auto-install ISO with DooPlex's key), then the product's BYO installer +as **Tester4** (host `Tester4-b5dbbd`). Tester4's ep0 space held one archive of an EARLIER box (`ct/9201`, 2026-10-20, +key `23:58:…`); this box's key is `7c:3b:…`. + +| | Agent | `GET /backup/due?target=felhom-pbs` | +|---|---|---| +| before | 0.166.0 (released) | `due: false`, „within cadence window", age 546,807 s — the foreign archive counted | +| after | 0.167.0-night1 (`main`) | `due: true`, „no successful backup recorded yet", `age_state: absent` | + +The controller's first-backup valve then started the first whole-box copy at 20:13:55Z; it completed at 21:15:17 CET +(`ct/9201/2026-10-26T20:13:55Z`, this box's key) — 4 minutes after the swap. `r1232/1-…`, `2-…`, `3-first-run.txt`. +PASS. **Teardown:** the product's uninstall ran (`r1232/uninstall-console.txt`, `UNINSTALL_RC=0`; the first try stopped +at the confirmation prompt — it asks for the guest number, nothing destroyed), then VM 9375 destroyed, its ISO and work +files removed. **The operator deletes hub host `Tester4-b5dbbd`.** Tester4's ep0 space keeps this box's one archive +(a host delete keeps the tenancy, by design). + +## R-1187 — the `.fab` export and import (controller `dfd0f8e` … `90a2530`) + +9202, throwaway Grimmory (markers inside its bound drive folder `appdata/grimmory/data/cc-r1187`: a file owned +1000:1001 mode 0640, a link inside, a link to `/etc/hostname`): + +- (a) a FIFO in the folder → the export **fails**, naming the FIFO's path (`r1187-9202-run2.txt`, `-run3.txt`, step 3); + without it the export carries both links AS links (the tar listing, step 4). +- (b) after the import: the file is back as 1000:1001 mode 640, the inside link is back, the outside link is **refused** + and listed in the import's WARN (run 2 and run 3). +- **Two import defects found by this proof, older than tonight, fixed and re-proven:** run 1 and run 2 ended + „waiting for DB: timeout" and left Grimmory down: (1) the import's compose runs lacked the controller's own + `USERDATA_PATH`/`IMPORT_PATH`, and compose's warning was read as the DB container id (`a3ed63e`); (2) the import used + `mysqladmin`/`mysql`, which the MariaDB 11+ images do not ship, so EVERY MariaDB 11+ import timed out (`90a2530`: the + import now loads through `appbackup.ImportDump`). **Run 3 on `0.346.0-night3`: the import succeeds and the app + answers.** `r1187-9202-run3.txt`. +- (g) AdventureLog (postgis): a superuser made with the app's own `createsuperuser`, export, the user deleted in the live + app, import → the user is back (read through the app's own ORM, `manage.py shell`). My front-door check (the Django + admin login page) answered 302 with no form both BEFORE and after — an instrument that did not reach the form, not a + result. `r1187g-9202.txt`. PASS on the ORM readback. +- (d) and (f): their unit tests are the proof (a running restore cannot be staged safely; a crafted bundle would be a + hand-made input). + +## R-1298 — held-app reports only for installed apps (controller `b156e45`) + +9202 on `0.346.0-night1`: the reporter skipped its four orphan holds by name („not reported (R-1298)", 0 sent). Then, +the controller stopped, `-clear-restore-hold` for `chaoscrash`, `chaosoom`, `chaosoomb`, `gokapi` → „no restore holds in +force"; PrivateBin's leftover removed through the product (volume, `app.yaml`, backup gone). `r1298/`. + +## Teardown + +- **9202:** back on 0.343.0 (image file restored, the test images removed); its standing apps only; no holds (the four + orphans cleared on purpose, per the brief); the drive's `appdata/` holds only paperless; no bundles in `exports/`. +- **demo-hp:** VM 9375 destroyed (own id), its ISO removed; nothing else provisioned. +- **Hub:** host `Tester4-b5dbbd` — the operator's delete. **DooPlex:** the local test images removed. +- Secrets: the evidence was scanned (the dashboard passwords, Tester4's passphrase, both hub tokens, the Resend key): + 0 hits. diff --git a/documentation/audits/night-fixes-2026-10-26/r1187-9202-run2.txt b/documentation/audits/night-fixes-2026-10-26/r1187-9202-run2.txt new file mode 100644 index 00000000..eb6c7d25 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1187-9202-run2.txt @@ -0,0 +1,33 @@ + [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH'] + [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} + [1] deployed, controller state=running, pinned={'grimmory': 'ghcr.io/grimmory-tools/grimmory:v3.5.0', 'grimmory-db': 'mariadb:11.4'} + gate: grimd53 is gated — passed as the household (cookie set) +1. grimmory installed and answering: True + its appdata folder: /mnt/felhom-drives/scratch_hdd/appdata/grimmory + markers: lrwxrwxrwx 1 0 0 9 Oct 26 20:46 link-in -> owned.txt +lrwxrwxrwx 1 0 0 13 Oct 26 20:46 link-out -> /etc/hostname +-rw-r----- 1 1000 1001 2 Oct 26 20:46 owned.txt +2. R-1220 skipped (proven in the first run) +3. export with a FIFO -> 200 {'ok': True} + job: Felhasználói adatok mentése sikertelen: a(z) /mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/pipe különleges fájl (cső, foglalat vagy eszköz), ezt nem lehet menteni — az exportálás leállt, semmi nem maradt ki csendben +4. export -> 200 {'ok': True} + job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'export', 'ok': True, 'output_path': '/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-204714.fab', 'output_size': '3.2 MB', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Előkészítés', 'status': 'done'}, { + bundle: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-204714.fab + tar listing of the drive part: drwxr-xr-x root/root 0 2026-10-26 20:47 cc-r1187 +lrwxrwxrwx root/root 0 2026-10-26 20:46 cc-r1187/link-in -> owned.txt +lrwxrwxrwx root/root 0 2026-10-26 20:46 cc-r1187/link-out -> /etc/hostname +-rw-r----- 1000/1001 2 2026-10-26 20:46 cc-r1187/owned.txt + import -> 200 {'ok': True} + import job: {'display_name': 'Grimmory', 'done': True, 'error': 'Adatbázis visszaállítása sikertelen: waiting for DB: timeout waiting for mariadb to become ready', 'job_type': 'import', 'ok': True, 'output_path': '', 'output_size': '', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Csomag megn + after import: lrwxrwxrwx 1 0 0 9 Oct 26 20:47 link-in -> owned.txt +-rw-r----- 1 1000 1001 2 Oct 26 20:47 owned.txt +/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/owned.txt 1000:1001 640 +owned.txt +ls: cannot access '/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/link-out': No such file or directory + the import's WARN: 2026/10/26 20:47:31 restore.go:695: [WARN] Import: data — 1 link(s) pointing outside the app's folder were NOT restored (R-1187 b): cc-r1187/link-out + app never answered on grimd53/ (last rc=0 code=404) + the app answers: False + [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'grimmory', 'volumes_removed': ['grimmory_grimmory_db'], 'hdd_paths_removed': ['/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data (12K)'], 'hdd_paths_preserved': [] +5. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '/mnt/felhom-drives/scratch_hdd/appdata/grimmory', 'kept': '', 'subdomain': '404'} + bundles of this run removed: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-204714.fab +0 diff --git a/documentation/audits/night-fixes-2026-10-26/r1187-9202-run3.txt b/documentation/audits/night-fixes-2026-10-26/r1187-9202-run3.txt new file mode 100644 index 00000000..5360b8c3 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1187-9202-run3.txt @@ -0,0 +1,32 @@ + [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH'] + [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} + [1] deployed, controller state=running, pinned={'grimmory': 'ghcr.io/grimmory-tools/grimmory:v3.5.0', 'grimmory-db': 'mariadb:11.4'} + gate: grimd53 is gated — passed as the household (cookie set) +1. grimmory installed and answering: True + its appdata folder: /mnt/felhom-drives/scratch_hdd/appdata/grimmory + markers: lrwxrwxrwx 1 0 0 9 Oct 26 21:01 link-in -> owned.txt +lrwxrwxrwx 1 0 0 13 Oct 26 21:01 link-out -> /etc/hostname +-rw-r----- 1 1000 1001 2 Oct 26 21:01 owned.txt +2. R-1220 skipped (proven in the first run) +3. export with a FIFO -> 200 {'ok': True} + job: Felhasználói adatok mentése sikertelen: a(z) /mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/pipe különleges fájl (cső, foglalat vagy eszköz), ezt nem lehet menteni — az exportálás leállt, semmi nem maradt ki csendben +4. export -> 200 {'ok': True} + job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'export', 'ok': True, 'output_path': '/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-210142.fab', 'output_size': '3.2 MB', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Előkészítés', 'status': 'done'}, { + bundle: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-210142.fab + tar listing of the drive part: drwxr-xr-x root/root 0 2026-10-26 21:01 cc-r1187 +lrwxrwxrwx root/root 0 2026-10-26 21:01 cc-r1187/link-in -> owned.txt +lrwxrwxrwx root/root 0 2026-10-26 21:01 cc-r1187/link-out -> /etc/hostname +-rw-r----- 1000/1001 2 2026-10-26 21:01 cc-r1187/owned.txt + import -> 200 {'ok': True} + import job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'import', 'ok': True, 'output_path': '', 'output_size': '', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Csomag megnyitása', 'status': 'done'}, {'label': 'Alkalmazás előkészítése', 'status': 'done'}, {'label': 'K + after import: lrwxrwxrwx 1 0 0 9 Oct 26 21:01 link-in -> owned.txt +-rw-r----- 1 1000 1001 2 Oct 26 21:01 owned.txt +/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/owned.txt 1000:1001 640 +owned.txt +ls: cannot access '/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/link-out': No such file or directory + the import's WARN: 2026/10/26 21:01:57 restore.go:694: [WARN] Import: data — 1 link(s) pointing outside the app's folder were NOT restored (R-1187 b): cc-r1187/link-out + the app answers: True + [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'grimmory', 'volumes_removed': ['grimmory_grimmory_db'], 'hdd_paths_removed': ['/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data (12K)'], 'hdd_paths_preserved': [] +5. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '/mnt/felhom-drives/scratch_hdd/appdata/grimmory', 'kept': '', 'subdomain': '404'} + bundles of this run removed: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-210142.fab +0 diff --git a/documentation/audits/night-fixes-2026-10-26/r1187-r1220-9202-run1.txt b/documentation/audits/night-fixes-2026-10-26/r1187-r1220-9202-run1.txt new file mode 100644 index 00000000..29b74c7e --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1187-r1220-9202-run1.txt @@ -0,0 +1,29 @@ + [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH'] + [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} + [1] deployed, controller state=running, pinned={'grimmory': 'ghcr.io/grimmory-tools/grimmory:v3.5.0', 'grimmory-db': 'mariadb:11.4'} + gate: grimd53 is gated — passed as the household (cookie set) +1. grimmory installed and answering: True + its appdata folder: /mnt/felhom-drives/scratch_hdd/appdata/grimmory + markers: lrwxrwxrwx 1 0 0 9 Oct 26 20:15 link-in -> owned.txt +lrwxrwxrwx 1 0 0 13 Oct 26 20:15 link-out -> /etc/hostname +-rw-r----- 1 1000 1001 2 Oct 26 20:15 owned.txt +2. app.yaml made unreadable; rescan seen: True + R-1220: the kept-data page lists grimmory's folder: False (want False) | WARN lines: 2 + app.yaml put back, byte check: 2b7b111d6bd71008 +3. export with a FIFO -> 200 {'ok': True} + job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'export', 'ok': True, 'output_path': '/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201737.fab', 'output_size': '3.2 MB', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Előkészítés', 'status': 'done'}, { +4. export -> 200 {'ok': True} + job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'export', 'ok': True, 'output_path': '/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201745.fab', 'output_size': '3.2 MB', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Előkészítés', 'status': 'done'}, { + bundle: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201745.fab + tar listing of the drive part: + import -> 200 {'ok': True} + import job: {'display_name': 'Grimmory', 'done': True, 'error': 'Adatbázis visszaállítása sikertelen: waiting for DB: timeout waiting for mariadb to become ready', 'job_type': 'import', 'ok': True, 'output_path': '', 'output_size': '', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Csomag megn + after import: ls: cannot access '/mnt/felhom-drives/scratch_hdd/appdata/grimmory/cc-r1187/link-out': No such file or directory + the import's WARN: + app never answered on grimd53/ (last rc=0 code=404) + the app answers: False + [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'grimmory', 'volumes_removed': ['grimmory_grimmory_db'], 'hdd_paths_removed': ['/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data (4.0K)'], 'hdd_paths_preserved': [ +5. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '/mnt/felhom-drives/scratch_hdd/appdata/grimmory', 'kept': '', 'subdomain': '404'} + bundles of this run removed: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201737.fab +/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201745.fab +0 diff --git a/documentation/audits/night-fixes-2026-10-26/r1187g-9202.txt b/documentation/audits/night-fixes-2026-10-26/r1187g-9202.txt new file mode 100644 index 00000000..6496fa00 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1187g-9202.txt @@ -0,0 +1,27 @@ + [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} + [1] deployed, controller state=unhealthy, pinned={'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.13.0', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine'} + [1] NOTE: the controller's own state is 'unhealthy', not 'running' — recorded, not treated as a failure; the fixture's front-door wait is the real gate + gate: advd53 not routed right now (404) — not cached, asked again next call + gate: advd53 not routed right now (404) — not cached, asked again next call + gate: advd53 not routed right now (404) — not cached, asked again next call + gate: advd53 not routed right now (404) — not cached, asked again next call + gate: advd53 not routed right now (404) — not cached, asked again next call + gate: advd53 not routed right now (404) — not cached, asked again next call + gate: advd53 is gated — passed as the household (cookie set) +1. adventurelog installed and answering: True +2. createsuperuser: Superuser created successfully. | in the ORM: True + front door before: no login form (302) +3. export: ok 48.2 MB /mnt/felhom-drives/scratch_hdd/exports/adventurelog_20261026-210649.fab + the bundle's database: database/adventurelog-postgres.sql.gz +manifest.json +4. the user deleted in the live app (so only the import can bring it back) — in the ORM: False +5. import: ok | steps: [('Csomag megnyitása', 'done'), ('Alkalmazás előkészítése', 'done'), ('Konfiguráció visszaállítása', 'done'), ('Felhasználói adatok visszaállítása', 'done'), ('Adatbázis visszaállítása', 'done'), ('Alkalmazás indítása', 'done')] + app answers: True + R-1187 g — the user back in the ORM: True + R-1187 g — the front door after import: no login form (302) + the import's DB line: 2026/10/26 21:01:59 export.go:166: [DEBUG] [appexport] restoreDatabase: found DB service=grimmory-db type=mariadb container=grimmory-db +2026/10/26 21:07:18 export.go:166: [DEBUG] [appexport] restoring database: type=postgres +2026/10/26 21:07:18 export.go:166: [DEBUG] [appexport] restoreDatabase: found DB service=adventurelog-postgres type=postgres container=adventurelog-postgres + [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'adventurelog', 'volumes_removed': ['adventurelog_adventurelog_media', 'adventurelog_adventurelog_postgres_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], +6. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '', 'kept': '', 'subdomain': '404'} + bundles removed: 0 diff --git a/documentation/audits/night-fixes-2026-10-26/r1232/1-before-old-agent.txt b/documentation/audits/night-fixes-2026-10-26/r1232/1-before-old-agent.txt new file mode 100644 index 00000000..be78c843 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1232/1-before-old-agent.txt @@ -0,0 +1,41 @@ +== storage key fingerprint + encryption-key + namespace Tester4 +felhom-pbs:backup/ct/9201/2026-10-20T12:17:26Z 9201 1792498646 23:58:7a:71:46:72:0c:1e +own key: +== tiers (agent 0.166.0) +{ + "ok": true, + "data": { + "vmid": 9201, + "tiers": [ + { + "target": "local", + "cadence_seconds": 86400, + "primary": true, + "storage": "present" + }, + { + "target": "felhom-pbs", + "cadence_seconds": 604800, + "primary": false, + "storage": "present" + } + ] + } +} +20:10:33 +== due felhom-pbs (agent 0.166.0) +{ + "ok": true, + "data": { + "vmid": 9201, + "due": false, + "reason": "within cadence window", + "age_seconds": 546807, + "target": "felhom-pbs", + "age_state": "known" + } +} +own key: 7c:3b:36:51:bc:a6:b4:90 +20:10:53 diff --git a/documentation/audits/night-fixes-2026-10-26/r1232/2-after-fixed-agent.txt b/documentation/audits/night-fixes-2026-10-26/r1232/2-after-fixed-agent.txt new file mode 100644 index 00000000..fadcfda2 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1232/2-after-fixed-agent.txt @@ -0,0 +1,16 @@ +bin=/usr/local/bin/felhom-agent +active +felhom-agent 0.167.0-night1 +== due felhom-pbs (agent 0.167.0-night1, R-1232 fix) +{ + "ok": true, + "data": { + "vmid": 9201, + "due": true, + "reason": "no successful backup recorded yet", + "target": "felhom-pbs", + "age_state": "absent" + } +} +Oct 26 21:11:14 r1232-byo felhom-agent[17600]: time=2026-10-26T21:11:14.654+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=felhom-pbs volid=felhom-pbs:backup/ct/9201/202 +20:11:14 diff --git a/documentation/audits/night-fixes-2026-10-26/r1232/3-first-run.txt b/documentation/audits/night-fixes-2026-10-26/r1232/3-first-run.txt new file mode 100644 index 00000000..cf756dda --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1232/3-first-run.txt @@ -0,0 +1,35 @@ +== archives after +felhom-pbs:backup/ct/9201/2026-10-20T12:17:26Z 23:58:7a:71:46:72:0c:1e +felhom-pbs:backup/ct/9201/2026-10-26T20:13:55Z 7c:3b:36:51:bc:a6:b4:90 +== due felhom-pbs after the run +{ + "ok": true, + "data": { + "vmid": 9201, + "due": false, + "reason": "within cadence window", + "age_seconds": 99, + "target": "felhom-pbs", + "age_state": "known" + } +} +== agent journal (felhom-pbs, R-1232) +Oct 26 21:00:45 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:45.237+01:00 level=INFO msg="felhom-agent daemon starting" version=0.166.0 host_id=Tester4-b5dbbd hub_url=https://hub.felhom.eu interval_s=900 +Oct 26 21:00:45 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:45.255+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +Oct 26 21:00:46 r1232-byo sudo[3034]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-backup-target-apply grant felhom-pbs +Oct 26 21:00:47 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:47.454+01:00 level=ERROR msg="store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)" target=felhom-pbs privilege=Datastore.AllocateSpace action="felhom-bac +Oct 26 21:00:47 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:47.454+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:store-grant:felhom-pbs feature="backup tier felhom-pbs: the agent' +Oct 26 21:00:57 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:57.058+01:00 level=INFO msg="pbsdr: one-time token secret consumed (single-use; value withheld from logs)" storage_id=felhom-pbs secret_len=36 +Oct 26 21:00:57 r1232-byo sudo[3774]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply create felhom-pbs 10.77.0.1 felhom-offsite Tester4 felhom@pbs!Tester4 c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:6 +Oct 26 21:00:58 r1232-byo sudo[3788]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply grant felhom-pbs +Oct 26 21:00:59 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:59.566+01:00 level=INFO msg="pbsdr: seeded escrow.pbs_storage_id (the ceremony one-liner needs no flags)" storage_id=felhom-pbs +Oct 26 21:00:59 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:59.567+01:00 level=INFO msg="pbsdr: converged" state=applied storage_id=felhom-pbs +Oct 26 21:11:00 r1232-byo felhom-os-apply[17411]: os-apply: CONFFILE updated /etc/debian_version (it was not changed locally) +Oct 26 21:11:04 r1232-byo felhom-agent[17600]: time=2026-10-26T21:11:04.570+01:00 level=INFO msg="felhom-agent daemon starting" version=0.167.0-night1 host_id=Tester4-b5dbbd hub_url=https://hub.felhom.eu interval_s=900 +Oct 26 21:11:04 r1232-byo felhom-agent[17600]: time=2026-10-26T21:11:04.582+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +Oct 26 21:11:05 r1232-byo sudo[17768]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +Oct 26 21:11:06 r1232-byo sudo[17814]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +Oct 26 21:11:14 r1232-byo felhom-agent[17600]: time=2026-10-26T21:11:14.654+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=felhom-pbs volid=felhom-pbs:backup/ct/9201/2026-10-20T12:17:26Z size_bytes=2072117616 +Oct 26 21:13:56 r1232-byo felhom-agent[17600]: time=2026-10-26T21:13:56.633+01:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1793045635144752924 +Oct 26 21:15:17 r1232-byo felhom-agent[17600]: time=2026-10-26T21:15:17.431+01:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-10-26T20:13:55Z size_bytes=2072870896 uncovered_volumes=2 +Oct 26 21:15:17 r1232-byo felhom-agent[17600]: time=2026-10-26T21:15:17.431+01:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1793045635144752924 archive=felhom-pbs:backup/ct/9201/2026-10-26T20:13:55Z diff --git a/documentation/audits/night-fixes-2026-10-26/r1232/install-console.txt b/documentation/audits/night-fixes-2026-10-26/r1232/install-console.txt new file mode 100644 index 00000000..58e9a275 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1232/install-console.txt @@ -0,0 +1,163 @@ + +[INFO] felhom-host-install v1.36.1 — mode=byo customer=Tester4 vmid=9201 + +[STEP] 1/8 pre-flight +[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve) +[INFO] node: r1232-byo (explicit) +[INFO] agent config: /etc/felhom-agent/agent.json +[INFO] agent: not installed yet — will be fetched + installed in step 5/8 +[INFO] local-lvm free: ~29 GiB +[WARN] local-lvm free ~29 GiB is below the recommended 120 GiB — the install continues; proceed only if you sized the grows deliberately +[INFO] free RAM: ~6277 MiB +[INFO] existing guests on this host: 0 (pct+qm) +[INFO] archive storage 'local' present +[INFO] acl storages all present or pre-positioned: local local-lvm +[INFO] dnsmasq: not present before Felhom — recorded; uninstall will remove it again if we install it +[INFO] host DNS (:53): free +[INFO] hub reachable (https://hub.felhom.eu) +[OK] customer 'Tester4' exists + passphrase valid +[INFO] golden: none local — will fetch + verify from Gitea in step 7/8 +r1232-byo + +[STEP] BYO disclosure — installing Felhom on a host the operator does NOT own will create: + pveum: roles FelhomAgentBase/FelhomAgentGuest/FelhomAgentStore + user felhom-agent@pve + token felhom-agent@pve!agent + + scoped ACL grants (Base@/, Guest@/pool/felhom, Store@[local local-lvm]) + the 'felhom' pool + agent: service user felhom-agent + /usr/local/bin/felhom-agent (+ .bak-* backups; A/B slots via the guarded self-update) + + /etc/systemd/system/felhom-agent.service (+ felhom-agent-rollback.service + start-limit drop-in) + + /etc/sudoers.d/felhom-agent + state dir /var/lib/felhom-agent + config /etc/felhom-agent/agent.json (0600) + host: /usr/local/sbin/felhom-mkfs-guarded + /usr/local/sbin/felhom-selfupdate-guarded + + /usr/local/sbin/felhom-pbs-apply (PBS-DR apply wrapper — DR capability is baked + on every install; ACTIVATION stays a hub flag, off = zero effect on this host) + + felhom-mgmt-watchdog service+timer+script + /etc/tmpfiles.d/felhom-privsep.conf + + /usr/local/sbin/felhom-priv-apply + the guest-hook snippet under /var/lib/vz/snippets/ + (both from the agent's config bundle) + + the 'sudo' and 'age' packages if absent + install state dir /var/lib/felhom-install + wg: an OUTBOUND WireGuard tunnel to the Felhom hub (wg_tunnel.enabled=true — base + infrastructure like the cloudflared tunnel; hands-free peer registration; the + operator can disable it per-customer hub-side on request) + guest: the provisioned Felhom LXC (vmid 9201, capped 2 cores / 4096 MiB) + its volumes + + the golden vzdump imported onto storage 'local' + reuse: a golden already on 'local' is used only if its sha256 or its controller version + matches the hub's vouched golden; otherwise the vouched one is fetched (a --golden you name is refused) + update: operator-signed self-update authority: felhom-op-1 + NOT touched in byo mode: root@pam (no break-glass), host DNS (:53), OOB sshd. + +Type this host's short hostname (r1232-byo) to acknowledge the byo install: [OK] byo install acknowledged for host r1232-byo +[OK] pre-flight passed +[STEP] 2/8 Proxmox API token +[OK] token minted (secret captured, not logged) +[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm]) +[SKIP] old broad role FelhomAgent already absent +[STEP] 3/8 compute volume grows +[INFO] auto-computed from a 29 GiB thin pool (16 §7: 50/100/130 GiB) +[INFO] grows: rootfs +0G (->32G), data +26G (->50G, ONE volume) +[STEP] 4/8 host enrollment (POST /host-enroll) +[OK] host MINTED (first enroll) +[INFO] host_id: Tester4-b5dbbd (api_key captured, not logged) +[SKIP] break-glass (step 4b) SKIPPED (byo: the host owner manages break-glass; root@pam untouched, nothing vaulted) +[STEP] 5/8 agent install (fetch + verify + install) +[SKIP] apt repo alignment skipped (byo — the owner manages repos) +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] manifest: agent v0.166.0 (sha 3f22c65b78914632…), golden v0.344.0 +[INFO] fetching agent binary v0.166.0 from Gitea … +[OK] verified sha256 3f22c65b78914632… matches the hub manifest +[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.166.0) +[INFO] installing the 'sudo' package (required for the non-root agent model) … +[OK] sudo installed (Sudo version 1.9.16p2) +[INFO] installing the 'age' package (escrow ceremony identity-wrap dependency) … +[OK] age installed (1.2.1) +[OK] created service user felhom-agent +[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify) +[OK] verified sha256 093772c1bda8b974… matches the hub manifest + os-apply: BUNDLE /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf skipped: a BYO host keeps its own settings (D43) + os-apply: BUNDLE START agent=0.166.0 sha=093772c1bda8b974 authority=installer files=32 write=26 same=1 kept=0 skipped=5 + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mkfs-guarded (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-selfupdate-guarded (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-pbs-apply (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-backup-target-apply (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-vzdump-partial-clean (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-thin-check (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-internal-place (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-crash-guard (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-priv-apply (new) + os-apply: BUNDLE WROTE /var/lib/vz/snippets/felhom-guest-hook.sh (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-shared-parent.sh (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-shared-parent.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.timer (new) + os-apply: BUNDLE WROTE /etc/felhom/crash-guard.conf (new) + os-apply: BUNDLE WROTE /etc/grub.d/01_felhom_oneshot (new) + os-apply: BUNDLE WROTE /etc/grub.d/42_felhom_oneshot (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent-rollback.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mgmt-watchdog (new) + os-apply: BUNDLE WROTE /etc/tmpfiles.d/felhom-privsep.conf (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.timer (new) + os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-agent (new) + os-apply: BUNDLE DONE agent=0.166.0 written=26 same=1 self-check=ok signers-created=False +[OK] installed every root-owned file from config bundle v0.166.0 (sha 093772c1bda8b974…; checked, previous copies kept) +[SKIP] OOB (felhom-sshd) off (byo, or appliance --no-oob) — the bundle skipped its files +[STEP] 5b/8 internal data place (Belső tárhely) +[SKIP] byo: the host owner's thin pool is not ours to divide — no internal place +[STEP] 6/8 agent config + service +[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive. +[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard. +[INFO] node=r1232-byo local_api=192.168.0.127:8443 tls_fp=4B:20:94:3E:FB:AB… +[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent) +[OK] byo config asserts passed (lan_resolver.enable / oob.enabled off; deployment_mode=byo; wg_tunnel intentionally ON — base infrastructure) +[OK] agent --selftest (read-only) passed +[OK] felhom-agent service active (non-root felhom-agent reads the config OK) +[OK] wrote /etc/felhom/os-trust.json + operator-signers (felhom-op-1) — the root-owned slow-lane anchors +[STEP] 7/8 golden archive +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] fetching golden v0.344.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_10_26-21_00_49.tar.zst +[OK] verified sha256 767a90fe07ed47c8… matches the hub manifest +[OK] golden imported + verified: local:backup/vzdump-lxc-9100-2026_10_26-21_00_49.tar.zst +[STEP] 8/8 provision guest 9201 +[SKIP] pool felhom already exists +=== felhom-agent 0.166.0 selftest=provision (vmid=9201 customer=Tester4 hostname=tester4) === + --- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_10_26-21_00_49.tar.zst → vmid 9201 --- +time=2026-10-26T21:02:13.966+01:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=9201 pool=felhom + [OK] front half: vmid 9201 up (boot+running) in 1m14s; MAC=BC:24:11:06:5B:AC + --- back half: mint per-guest token + populate bootstrap config mount --- +time=2026-10-26T21:02:19.331+01:00 level=INFO msg="provision: back-half complete" vmid=9201 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=192.168.0.127:8443 + [OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 9201 (host dir /var/lib/felhom-agent/guests/9201/bootstrap) + local-api endpoint 192.168.0.127:8443 · leaf fp c0463612a04329e8b367f3aed7e925da5b5e44668df7b726fc5e7ac4dd6837ad · token: minted (not printed) +=== selftest=provision OK — guest 9201 provisioned + bootstrap-mounted (KEPT) === + next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller, + which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api. +[OK] provision completed +[INFO] rebooting guest 9201 so the baked controller-bootstrap unit picks up the mount +[STEP] verify +[OK] pct status: running +[OK] onboot: 1 + mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G + mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives + rootfs: local-lvm:vm-9201-disk-0,size=32G +[OK] pool: guest 9201 is a member of felhom +[OK] acl: FelhomAgentBase@/ present (user+token) +[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token) +[OK] acl: FelhomAgentStore@/storage/local present (user+token) +[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token) +[OK] authz signers: 2 (operator-signed self-update armed) +[OK] controller: Up 4 seconds (health: starting) (after ~0s) +[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.345.0 +[WARN] cloudflared not visible yet +[INFO] (confirm in the hub UI that host Tester4-b5dbbd reports guest 9201) + +[OK] Day-0 provision SUCCESS — vmid=9201 host_id=Tester4-b5dbbd customer=Tester4 golden=local:backup/vzdump-lxc-9100-2026_10_26-21_00_49.tar.zst +INSTALL_RC=0 +Connection to 192.168.0.127 closed. diff --git a/documentation/audits/night-fixes-2026-10-26/r1232/install-start.txt b/documentation/audits/night-fixes-2026-10-26/r1232/install-start.txt new file mode 100644 index 00000000..063cd9e0 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1232/install-start.txt @@ -0,0 +1 @@ +19:59:44 diff --git a/documentation/audits/night-fixes-2026-10-26/r1232/teardown.txt b/documentation/audits/night-fixes-2026-10-26/r1232/teardown.txt new file mode 100644 index 00000000..2375fb46 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1232/teardown.txt @@ -0,0 +1,4 @@ +purging VM 9375 from related configurations.. +Configuration file 'nodes/demo-hp/qemu-server/9375.conf' does not exist +ls: cannot access '/mnt/pve/nvme-scratch/images/9375': No such file or directory +felhom-installer-1.30.0-pve9.2-1.iso diff --git a/documentation/audits/night-fixes-2026-10-26/r1232/uninstall-console-1-wrong-confirmation.txt b/documentation/audits/night-fixes-2026-10-26/r1232/uninstall-console-1-wrong-confirmation.txt new file mode 100644 index 00000000..9d308cb2 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1232/uninstall-console-1-wrong-confirmation.txt @@ -0,0 +1,26 @@ + +[INFO] felhom-host-install v1.36.1 — mode=uninstall + +[STEP] UNINSTALL — local host teardown +[INFO] no --vmid given; using recorded provisioned_vmid=9201 from /var/lib/felhom-install/state.json +[INFO] vmid 9201 looks like a Felhom guest (has the /etc/felhom-bootstrap mount) +[INFO] pct config 9201: + arch: amd64 + cores: 2 + features: nesting=1,keyctl=1 + hookscript: local:snippets/felhom-guest-hook.sh + hostname: tester4 + memory: 4096 + mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G + mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives + mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1 + net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:06:5B:AC,ip=dhcp,type=veth + onboot: 1 + ostype: debian + rootfs: local-lvm:vm-9201-disk-0,size=32G + swap: 4096 + unprivileged: 1 +Type the vmid (9201) to confirm PERMANENT destruction: r1232-byo +[ERROR] confirmation mismatch (got 'r1232-byo', expected '9201') — aborting, nothing destroyed +UNINSTALL_RC=1 +Connection to 192.168.0.127 closed. diff --git a/documentation/audits/night-fixes-2026-10-26/r1232/uninstall-console.txt b/documentation/audits/night-fixes-2026-10-26/r1232/uninstall-console.txt new file mode 100644 index 00000000..85cec852 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1232/uninstall-console.txt @@ -0,0 +1,84 @@ + +[INFO] felhom-host-install v1.36.1 — mode=uninstall + +[STEP] UNINSTALL — local host teardown +[INFO] no --vmid given; using recorded provisioned_vmid=9201 from /var/lib/felhom-install/state.json +[INFO] vmid 9201 looks like a Felhom guest (has the /etc/felhom-bootstrap mount) +[INFO] pct config 9201: + arch: amd64 + cores: 2 + features: nesting=1,keyctl=1 + hookscript: local:snippets/felhom-guest-hook.sh + hostname: tester4 + memory: 4096 + mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G + mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives + mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1 + net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:06:5B:AC,ip=dhcp,type=veth + onboot: 1 + ostype: debian + rootfs: local-lvm:vm-9201-disk-0,size=32G + swap: 4096 + unprivileged: 1 +Type the vmid (9201) to confirm PERMANENT destruction: 9201 + Logical volume "vm-9201-disk-0" successfully removed. + Logical volume "vm-9201-disk-1" successfully removed. +[OK] guest 9201 destroyed +[STEP] host-level removal +Removed '/etc/systemd/system/multi-user.target.wants/felhom-agent.service'. +[OK] removed /etc/felhom-agent (the agent config and every copy of it) +Removed '/etc/systemd/system/multi-user.target.wants/wg-quick@wg-felhom.service'. +[OK] WireGuard tunnel to the Felhom off-site endpoint is down (wg-quick@wg-felhom disabled, conf removed) +Removed '/etc/systemd/system/timers.target.wants/felhom-mgmt-watchdog.timer'. +The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=, +Also=, or Alias= settings in the [Install] section, and DefaultInstance= for +template units). This means they are not meant to be enabled or disabled using systemctl. + +Possible reasons for having these kinds of units are: +• A unit may be statically enabled by being symlinked from another unit's + .wants/, .requires/, or .upholds/ directory. +• A unit's purpose may be to act as a helper for some other unit which has + a requirement dependency on it. +• A unit may be started when needed via activation (socket, path, timer, + D-Bus, udev, scripted systemctl call, ...). +• In case of template units, the unit is meant to be enabled with some + instance name specified. +Removed '/etc/systemd/system/pve-guests.service.wants/felhom-shared-parent.service'. +Removed '/etc/systemd/system/multi-user.target.wants/felhom-shared-parent.service'. +Generating grub configuration file ... +Found linux image: /boot/vmlinuz-7.0.2-6-pve +Found initrd image: /boot/initrd.img-7.0.2-6-pve +Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi +Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi +Found memtest86+ 64bit image: /boot/memtest86+x64.bin +Found memtest86+ 32bit image: /boot/memtest86+ia32.bin +Adding boot menu entry for UEFI Firmware Settings ... +done +[SKIP] old broad role FelhomAgent already absent +[SKIP] golden vzdump left in place (pass --remove-golden to remove) + +[STEP] kept vs wiped — read before pulling drives or closing the customer out + WIPED (this run): + - guest 9201 (container + its OS/Docker/user-data volumes) + - the felhom-agent: binary, unit, sudoers (+ its copies), state dir, service user, and its config + directory with every copy of the config in it + - the WireGuard tunnel to the Felhom off-site endpoint (wg-quick@wg-felhom disabled, /etc/wireguard/wg-felhom.conf removed) + - self-update artifacts: guarded wrapper, A/B slots (.prev/.new.*), rollback unit, start-limit drop-in + - break-glass watchdog + OOB artifacts (where present); guest-hook snippet; dnsmasq snippets; the mkfs, pbs-apply, + backup-target-apply, os-apply and priv-apply wrappers; the crash guard; the config-bundle record + - pveum: the Felhom roles/user/token/scoped ACL; the emptied felhom pool + - the install state file + KEPT (lives on deliberately — remove/rotate these out-of-band if the customer is leaving): + - the enrolled drives + ALL data under /mnt/felhom-drives — unmounted only, NEVER wiped; + the drives are physically removable now. + - the PBS backups + this customer's namespace on the PBS side — delete there if wanted + - the hub host/customer record + report history (operator UI / DB) + - this host's WireGuard PEER on the hub and the off-site endpoint (its /32 and public key) — the + tunnel is down from this side; delete the peer in the hub (operator) if the customer is leaving + - the escrow blob in the hub, if one exists (operator UI) + +[OK] UNINSTALL complete — removed: guest 9201, the felhom-agent (unit/sudoers/binary/state/config+baks/user + selfupdate-artifacts/shared-parent/mkfs-wrapper/pbs-apply-wrapper/hook-snippet/dnsmasq-snippets), the pveum role/user/token/ACL, the felhom pool, and /var/lib/felhom-install/state.json. +[INFO] golden vzdump: left in place (--remove-golden to remove). +[INFO] NOTE: the 'sudo' package was left INSTALLED (a system package, never ours to remove). dnsmasq is removed only when the install-time record says Felhom installed it; a resolver that pre-dated Felhom is never touched. The host record still exists in the hub — remove it there if desired. +UNINSTALL_RC=0 +Connection to 192.168.0.127 closed. diff --git a/documentation/audits/night-fixes-2026-10-26/r1298/clear-holds.txt b/documentation/audits/night-fixes-2026-10-26/r1298/clear-holds.txt new file mode 100644 index 00000000..05b74927 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1298/clear-holds.txt @@ -0,0 +1,18 @@ +== before (19:49:23) +chaoscrash held since 2026-09-24T12:16:24Z +chaosoom held since 2026-09-24T11:48:01Z +chaosoomb held since 2026-09-24T12:39:51Z +gokapi held since 2026-09-24T10:37:48Z +controller stopped 19:49:24 + Check the app's data first — the undo copies are in its unit's db-dumps dir. + chaoscrash rc=0 + Check the app's data first — the undo copies are in its unit's db-dumps dir. + chaosoom rc=0 + Check the app's data first — the undo copies are in its unit's db-dumps dir. + chaosoomb rc=0 +[INFO] [settings] Settings saved + gokapi rc=0 +== after +no restore holds in force +[INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json +controller started 19:49:26 diff --git a/documentation/audits/night-fixes-2026-10-26/r1298/privatebin-remove.txt b/documentation/audits/night-fixes-2026-10-26/r1298/privatebin-remove.txt new file mode 100644 index 00000000..aafffac1 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/r1298/privatebin-remove.txt @@ -0,0 +1,2 @@ + [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'privatebin', 'volumes_removed': ['privatebin_privatebin_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem tárolt saját adatot +leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '', 'kept': '', 'subdomain': '404'} diff --git a/documentation/audits/night-fixes-2026-10-26/tools/r1187_r1220_live.py b/documentation/audits/night-fixes-2026-10-26/tools/r1187_r1220_live.py new file mode 100644 index 00000000..27aaaa86 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/tools/r1187_r1220_live.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +"""r1187_r1220_live.py — R-1220 and R-1187 (a)(b) live on scratch guest 9202, through the endpoints the UI invokes. + +A THROWAWAY Grimmory (drive appdata binds). Steps: + 1. install; a marker folder in its drive appdata: an owned file (1000:1001, 0640), a link inside, a link outside; + 2. R-1220: its app.yaml made unreadable (a byte copy saved first) → after a stack rescan the „Megőrzött adatok" page does + NOT list its folder; the app.yaml put back byte-identical; + 3. R-1187 (a): a FIFO in the folder → the export FAILS naming the FIFO's path; the FIFO removed; + 4. R-1187 (a)(b): export → import over the running app → the owned file keeps owner and mode, the inside link comes + back, the outside link is refused (listed in the controller WARN); + 5. teardown: the app removed through the product (data and backups), the bundles this run made deleted. + +usage: SC=<0600 dir with .ctlpw> SWEEP_EV= TARGET=9202 DRIVE_ROOT=/mnt/felhom-drives/scratch_hdd python3 -u r1187_r1220_live.py +""" +import os +import signal +import sys +import time + +signal.signal(signal.SIGTERM, lambda *a: (_ for _ in ()).throw(SystemExit("SIGTERM — cleaning up"))) + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools")) +import sweep as sw # noqa: E402 + +w = sw.w +APP, SUB, DRIVE = "grimmory", "grimd53", os.environ.get("DRIVE_ROOT", "/mnt/felhom-drives/scratch_hdd") +say = print + + +def g(cmd): + return sw.g(cmd).strip() + + +def job(kind): + path = "/api/export/status" if kind == "export" else "/api/export/import/status" + for _ in range(180): + time.sleep(5) + code, d = w.ctl("GET", path) + j = d.get("data", d) if isinstance(d, dict) else {} + if j.get("done") or j.get("Done"): + return j + return {"timeout": True} + + +def scan_after(since): + """Wait for a stack rescan that started after `since` (the controller rescans on its own every ~2 min).""" + for _ in range(40): + time.sleep(10) + if g(f"docker logs --since {since} felhom-controller 2>&1 | grep -c 'ScanStacks complete'") not in ("", "0"): + return True + return False + + +def main(): + w.say = say + w.login() + try: + ok = w.deploy(APP, SUB, {"HDD_PATH": DRIVE}) and w.wait_app(SUB, "/", tries=90, delay=5) + say("1. grimmory installed and answering:", ok) + appdata = g(f"ls -d {DRIVE}/appdata/grimmory 2>/dev/null | head -1") + say(" its appdata folder:", appdata or "(not found)") + if not appdata: + raise SystemExit("INSTRUMENT: no appdata folder — no verdict") + m = f"{appdata}/data/cc-r1187" # INSIDE the bound folder (appdata/grimmory/data) — the export packs only binds + say(" markers:", g(f"mkdir -p {m} && printf hi > {m}/owned.txt && chown 1000:1001 {m}/owned.txt && chmod 0640 {m}/owned.txt" + f" && ln -sfn owned.txt {m}/link-in && ln -sfn /etc/hostname {m}/link-out && ls -ln {m} | tail -n +2")) + + # 2. R-1220 (SKIP_R1220=1: already proven in the first run, `r1187-r1220-9202-run1.txt`) + if os.environ.get("SKIP_R1220"): + say("2. R-1220 skipped (proven in the first run)") + else: + stack = f"/opt/docker/stacks/{APP}" + g(f"cp -p {stack}/app.yaml /root/r1220-app.yaml.save && printf 'deployed: true\\nenv: [broken: yaml: here\\n' > {stack}/app.yaml") + t = int(time.time()) + say("2. app.yaml made unreadable; rescan seen:", scan_after(t)) + page = w.page("/kept-data") + say(" R-1220: the kept-data page lists grimmory's folder:", appdata in page, "(want False) | WARN lines:", + g(f"docker logs --since {t} felhom-controller 2>&1 | grep -c 'R-1220'")) + g(f"cp -p /root/r1220-app.yaml.save {stack}/app.yaml && rm -f /root/r1220-app.yaml.save") + say(" app.yaml put back, byte check:", g(f"sha256sum {stack}/app.yaml | cut -c1-16")) + scan_after(int(time.time())) + + # 3. R-1187 (a) FIFO + g(f"mkfifo {m}/pipe") + code, d = w.ctl("POST", "/api/export/start", {"stack_name": APP, "dest_drive": DRIVE, "stop_app": False}) + say("3. export with a FIFO ->", code, str(d)[:160]) + if code != "200": + raise SystemExit("INSTRUMENT: the export start was refused — no verdict") + j = job("export") + say(" job:", str(j.get("error") or j.get("Error") or j)[:300]) + g(f"rm -f {m}/pipe") + if j.get("timeout"): + raise SystemExit("INSTRUMENT: the export job never ended — no verdict") + + # 4. R-1187 (a)(b) round trip + code, d = w.ctl("POST", "/api/export/start", {"stack_name": APP, "dest_drive": DRIVE, "stop_app": False}) + say("4. export ->", code, str(d)[:120]) + if code != "200": + raise SystemExit("INSTRUMENT: the export start was refused — no verdict") + j = job("export") + say(" job:", str(j)[:300]) + fab = g(f"ls -t {DRIVE}/exports/grimmory*.fab 2>/dev/null | head -1") + say(" bundle:", fab) + say(" tar listing of the drive part:", g(f"mkdir -p /root/r1187x && cd /root/r1187x && tar -xzf {fab} data/hdd 2>/dev/null; " + f"for f in data/hdd/*.tar; do tar -tvf $f | grep cc-r1187; done; rm -rf /root/r1187x")) + g(f"rm -rf {m}") # the import must bring it back + ti = int(time.time()) + code, d = w.ctl("POST", "/api/export/import", {"path": fab}) + say(" import ->", code, str(d)[:160]) + if code != "200": + raise SystemExit("INSTRUMENT: the import start was refused — no verdict") + j = job("import") + say(" import job:", str(j)[:300]) + say(" after import:", g(f"ls -ln {m} | tail -n +2; stat -c '%n %u:%g %a' {m}/owned.txt; readlink {m}/link-in; " + f"ls -l {m}/link-out 2>&1 | head -1")) + say(" the import's WARN:", g(f"docker logs --since {ti} felhom-controller 2>&1 | grep 'R-1187 b' | cut -c1-220")) + say(" the app answers:", w.wait_app(SUB, "/", tries=60, delay=5)) + finally: + sw.remove(APP, keep=False, say=say) + say("5. leftovers:", sw.leftovers(APP, SUB)) + say(" bundles of this run removed:", g(f"ls {DRIVE}/exports/grimmory*.fab 2>/dev/null; rm -f {DRIVE}/exports/grimmory*.fab; ls {DRIVE}/exports/ 2>/dev/null | grep -c grimmory")) + + +if __name__ == "__main__": + main() diff --git a/documentation/audits/night-fixes-2026-10-26/tools/r1187g_live.py b/documentation/audits/night-fixes-2026-10-26/tools/r1187g_live.py new file mode 100644 index 00000000..8f6a7a8d --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-26/tools/r1187g_live.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +"""r1187g_live.py — R-1187 (g) live on scratch guest 9202: a PostGIS app's database survives a `.fab` export → import. + +A THROWAWAY AdventureLog (postgis/postgis). A superuser is made with the app's own `createsuperuser`; the app is exported +and imported through the endpoints the UI invokes; then the user is read back through the app's FRONT DOOR — the +Django admin login page — and through its own ORM. Before the fix the import did not recognise postgis as a database, +so the dump was never loaded. Teardown through the product. The throwaway password lives only in this process. + +usage: SC=<0600 dir with .ctlpw> SWEEP_EV= TARGET=9202 DRIVE_ROOT=/mnt/felhom-drives/scratch_hdd python3 -u r1187g_live.py +""" +import os +import re +import secrets +import signal +import urllib.parse +import sys +import time + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools")) +import sweep as sw # noqa: E402 + +signal.signal(signal.SIGTERM, lambda *a: (_ for _ in ()).throw(SystemExit("SIGTERM — cleaning up"))) +w = sw.w +APP, SUB, DRIVE = "adventurelog", "advd53", os.environ.get("DRIVE_ROOT", "/mnt/felhom-drives/scratch_hdd") +say = print + + +def g(cmd): + return sw.g(cmd).strip() + + +def job(path): + for _ in range(180): + time.sleep(5) + code, d = w.ctl("GET", path) + if isinstance(d, dict) and d.get("done"): + return d + return {"timeout": True} + + +def orm_has(user): + out = g(f"docker exec adventurelog python manage.py shell -c " + f"\"from django.contrib.auth import get_user_model; print('ANS=' + str(get_user_model().objects.filter(username='{user}').exists()))\" 2>&1") + m = re.search(r"ANS=(True|False)", out) + return m.group(1) if m else "unreadable: " + out[-160:] + + +def admin_login(user, pw): + """The front door, the household's route (box_walk.app_curl carries the setup gate): GET /admin/login/ for the + form and its csrftoken cookie, POST the form (the password on stdin, never an argument). Success = 302 to /admin/.""" + rc, code, body = w.app_curl(SUB, "/admin/login/", "-D", "-") + tok = re.search(r'name="csrfmiddlewaretoken" value="([^"]+)"', body) + ck = re.search(r"(?im)^set-cookie:\s*csrftoken=([^;]+)", body) + if not tok or not ck: + return f"no login form ({code})" + form = urllib.parse.urlencode({"csrfmiddlewaretoken": tok.group(1), "username": user, "password": pw, "next": "/admin/"}) + rc, code, body = w.app_curl(SUB, "/admin/login/?next=/admin/", "-H", f"Cookie: csrftoken={ck.group(1)}", + "-H", "Content-Type: application/x-www-form-urlencoded", + "-H", f"Referer: https://{SUB}.{w.DOMAIN}/admin/login/", method="POST", data=form) + return code + (" (logged in)" if code == "302" else " (refused)") + + +def main(): + w.say = say + w.login() + user, pw = "cc" + secrets.token_hex(3), "Cc-" + secrets.token_hex(12) + try: + ok = w.deploy(APP, SUB) and w.wait_app(SUB, "/", tries=120, delay=5) + say("1. adventurelog installed and answering:", ok) + out = g(f"docker exec -e DJANGO_SUPERUSER_PASSWORD='{pw}' adventurelog python manage.py createsuperuser --noinput " + f"--username {user} --email {user}@cc.invalid 2>&1 | tail -1") + say("2. createsuperuser:", out[:120], "| in the ORM:", orm_has(user)) + say(" front door before:", admin_login(user, pw)) + code, d = w.ctl("POST", "/api/export/start", {"stack_name": APP, "dest_drive": DRIVE, "stop_app": False}) + if code != "200": + raise SystemExit(f"INSTRUMENT: export start refused {code} {d}") + j = job("/api/export/status") + say("3. export:", j.get("error") or "ok", j.get("output_size"), j.get("output_path")) + fab = j.get("output_path", "") + say(" the bundle's database:", g(f"tar -tzf {fab} | grep -E 'database/|manifest' | head -4")) + g("docker exec adventurelog python manage.py shell -c \"from django.contrib.auth import get_user_model; " + f"get_user_model().objects.filter(username='{user}').delete()\" >/dev/null 2>&1") + say("4. the user deleted in the live app (so only the import can bring it back) — in the ORM:", orm_has(user)) + code, d = w.ctl("POST", "/api/export/import", {"path": fab}) + if code != "200": + raise SystemExit(f"INSTRUMENT: import start refused {code} {d}") + j = job("/api/export/import/status") + say("5. import:", j.get("error") or "ok", "| steps:", [(s.get("label"), s.get("status")) for s in j.get("steps", [])]) + say(" app answers:", w.wait_app(SUB, "/", tries=120, delay=5)) + say(" R-1187 g — the user back in the ORM:", orm_has(user)) + say(" R-1187 g — the front door after import:", admin_login(user, pw)) + say(" the import's DB line:", g("docker logs --since 15m felhom-controller 2>&1 | grep -E 'restoring database|found DB service|DB dump imported|psql' | tail -3 | cut -c1-200")) + finally: + sw.remove(APP, keep=False, say=say) + say("6. leftovers:", sw.leftovers(APP, SUB)) + say(" bundles removed:", g(f"rm -f {DRIVE}/exports/adventurelog*.fab; ls {DRIVE}/exports 2>/dev/null | grep -c adventurelog")) + + +if __name__ == "__main__": + main() diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index a7167c9a..3787ee6c 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -193,7 +193,7 @@ stopping line that lies. | **R-1184** | Backup & restore | P3 | **A cut-off shared-folder restore can be „placed" and reported restored while files are missing; the download is then deleted.** Readiness is only „the scratch folder is not empty" (`internal/backup/shares_restore.go:131-140`) — the R-358 class fixed for apps, not for shares; 30-min download window (`web/offbox_handlers.go:716`); fixed 2 GiB free-space gate that fails open (`:107`). CONFIRMED in code. Data-loss hunt 2026-10-20 (C1 reviewer 1, R3). | OPEN | — | Fix: a completion marker after restic succeeds, as for apps | CC | | **R-1185** | Backup & restore | P3 | **A database dump that fails its own content check has already replaced last night's good dump, and the second drive copies it an hour later.** `appbackup/dbdump.go:388-422`: size check, rename over the good file (`:399`), then `ValidateDump` (`:419`). Volume tars are promoted with no content check either (`backup.go:902`). Tier 1 is one point per app by design (07 §6.1); the ORDER is the defect. PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/`, a docker stub prints a header-only dump): `ValidateDump` says invalid („dump file too small"), `DumpOneTo` returns no error, and last night's good dump (with its `CREATE TABLE`) is gone from disk. Data-loss hunt 2026-10-20 (D5). | OPEN | — | Fix: validate the temp file before the rename; keep the last good dump on a failed check | CC | | **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC | -| **R-1187** | Backup & restore | P2 | **A `.fab` export or import can lose or mix household data — code-confirmed paths from the data-loss hunt:** (a) an export silently drops a whole drive folder that holds a symlink and still ends „done” (`appexport/fabplan.go:292-310`, `export.go:712`; a FIFO also hangs the export on `os.Open`); (b) the import handles only folders and regular files: no owners set, symlinks dropped (`restore.go:1138-1183`); (c) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** an import over an installed app that is not running merged the old data with the bundle's (`docker cp` never deletes) — reproduced on 9202 before the fix (stopped: a marker written after the export SURVIVED the import; running control: replaced), now refused with a Hungarian sentence and the app untouched; red-proved (`r1187c_test.go`), proven on 9202 rc.2 (`audits/night-2026-10-22/P3-candidates/r1187c-9202.txt`); (d) PLAUSIBLE: no restore-busy check in `StartImport`, no undo, no record (`restore.go:181-186`); (f) the bundle's HDD_PATH is trusted, an empty one gives a relative path (`restore.go:589-631`); (g) the DB import misses postgis images (adventurelog, dawarich: `classifyDBImage`) and runs psql without ON_ERROR_STOP (`restore.go:1047-1056, 933-943`; the dump is `--clean --if-exists`, so ON_ERROR_STOP is safe). (e) was R-1191. Research `audits/night-2026-10-22/P3-candidates/analysis.md`; first list `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | OPEN — **2026-10-23: (c) and the C3/N9 refusal texts released in controller 0.340.0;** (a), (b), (f), (g) and the two import paths of C2 stay open. OPEN — **2026-10-22 night: re-ranked P3→P2** ((a) and (b) lose household files on a move to a new box with no error). (c) fixed in code; (a), (b), (f), (g) read in code, not live-proven. **2026-10-23 night — fresh review (`REPORT-night-2026-10-23.md`, C2, C3):** two more import paths still merge — a running app whose `RemoveStackVolumes` fails falls back to `StopStack` and goes on (`appexport/restore.go:375-380`), and `restoreHDDData` extracts drive-folder tars over the existing folder without clearing it (`restore.go:602-655`) for running and stopped apps alike. Both exist in 0.339.0 — not a hold for 0.340.0. The (c) refusal also catches an unhealthy or restarting app (`IsStackRunning`: running/degraded only); its text now says so and names removal (controller `943c517`). Recovery code: fix only with a red-proof + a 9202 proof. | — | Fix (a)+(b) with a red-proof and a 9202 proof (capture code — the night rule); (g) small fix with a 9202 postgis app; (d) after (a) | CC | +| **R-1187** | Backup & restore | P2 | **A `.fab` export or import can lose or mix household data — code-confirmed paths from the data-loss hunt:** (a) an export silently drops a whole drive folder that holds a symlink and still ends „done” (`appexport/fabplan.go:292-310`, `export.go:712`; a FIFO also hangs the export on `os.Open`); (b) the import handles only folders and regular files: no owners set, symlinks dropped (`restore.go:1138-1183`); (c) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** an import over an installed app that is not running merged the old data with the bundle's (`docker cp` never deletes) — reproduced on 9202 before the fix (stopped: a marker written after the export SURVIVED the import; running control: replaced), now refused with a Hungarian sentence and the app untouched; red-proved (`r1187c_test.go`), proven on 9202 rc.2 (`audits/night-2026-10-22/P3-candidates/r1187c-9202.txt`); (d) PLAUSIBLE: no restore-busy check in `StartImport`, no undo, no record (`restore.go:181-186`); (f) the bundle's HDD_PATH is trusted, an empty one gives a relative path (`restore.go:589-631`); (g) the DB import misses postgis images (adventurelog, dawarich: `classifyDBImage`) and runs psql without ON_ERROR_STOP (`restore.go:1047-1056, 933-943`; the dump is `--clean --if-exists`, so ON_ERROR_STOP is safe). (e) was R-1191. Research `audits/night-2026-10-22/P3-candidates/analysis.md`; first list `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | **VERIFY — fixed on `main`: (a), (b), (d), (f), (g) + two import defects found by the live proof; waits for the controller release** — 2026-10-26 night: controller `dfd0f8e`, `106325d`, `f24233c`, `cccb9ae` (three rounds of the automated security review on the link handling: writes only through `os.Root`, links only downward), `a3ed63e` (the import's compose runs get the controller's path variables; a warning is never the container id), `90a2530` (the database loads through `appbackup.ImportDump` — every MariaDB 11+ import had timed out on `mysqladmin`). Live on 9202 (Grimmory: FIFO refused naming the path, owner/mode/inside link back, outside link refused; the import now succeeds and the app answers; AdventureLog: a postgis user back after the import) — `audits/night-fixes-2026-10-26/`. (d), (f): unit tests. Still open from the row: C2's two merge paths of an import over an existing app. **Earlier:** OPEN — **2026-10-23: (c) and the C3/N9 refusal texts released in controller 0.340.0;** (a), (b), (f), (g) and the two import paths of C2 stay open. OPEN — **2026-10-22 night: re-ranked P3→P2** ((a) and (b) lose household files on a move to a new box with no error). (c) fixed in code; (a), (b), (f), (g) read in code, not live-proven. **2026-10-23 night — fresh review (`REPORT-night-2026-10-23.md`, C2, C3):** two more import paths still merge — a running app whose `RemoveStackVolumes` fails falls back to `StopStack` and goes on (`appexport/restore.go:375-380`), and `restoreHDDData` extracts drive-folder tars over the existing folder without clearing it (`restore.go:602-655`) for running and stopped apps alike. Both exist in 0.339.0 — not a hold for 0.340.0. The (c) refusal also catches an unhealthy or restarting app (`IsStackRunning`: running/degraded only); its text now says so and names removal (controller `943c517`). Recovery code: fix only with a red-proof + a 9202 proof. | — | Fix (a)+(b) with a red-proof and a 9202 proof (capture code — the night rule); (g) small fix with a 9202 postgis app; (d) after (a) | CC | | **R-1188** | Backup & restore | P4 | **Restore-path candidates from the data-loss hunt — what is left is unlikely or a sentence:** (b) a failed `volume rm` is ignored before the untar, which then lands in the old volume (`backup/restore.go:218`; `StopStack` is `compose down`, so the rm rarely fails); (d) the full off-site download stops at 30 min per press and the household is not told to press again (`web/offbox_handlers.go:455`; the scratch is kept, so a second press probably continues — not measured); (e) place and full restore re-read the LATEST snapshot, not the downloaded one (`offbox_restore.go:675-679`, `_ = id`). Closed parts: (a) with R-1195 (controller 0.338.0), (c) in controller 0.337.0 (`TestR1188c_ALeftoverTempDoesNotBlockTheRetry`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — **2026-10-22 night: re-ranked P3→P4**: (a) and (c) are shipped; the rest is low-chance or a sentence. | — | (d): measure a second press on a large download on 9202, then write the sentence; (b), (e): unit tests when the file is next touched | CC | | **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt — what is left:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it: the agent turns a future archive time into a negative age (`localapi/server.go:1179-1192`); half-live 2026-10-21 (B6 #8, `s8-wrong-clock/README.md`: a local archive dated 2028-10-21); agent code, so it rides an agent release and the operator's vouch; (c) the whole-guest quiesce waits for the catch-up and the update leg, not for a restore (`quiesce.go:213-275, 565`); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — a design question; (g) spring forward 2027-03-28 moves 02:30 onto the 03:30 second copy; (h) self-update busy check before the pull only (`updater.go:532-545, 764`; not re-read); (i) a double-clicked restore shows failed while it runs (check and begin not atomic, `web/handlers.go:1860, 1881`). **Moved out 2026-10-22 night:** (b) the skipped nights after an off-cycle run were FIXED by R-899 (controller 0.304.0; demo-hp's journal shows a night run after each of three off-cycle runs since), the creep → R-1221; (d) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** the catch-up now runs the off-site proof and the integrity check once after its legs, the false comment corrected; red-proved (`nightchain/r1189d_test.go`, `cmd/controller/r1189d_wiring_test.go`); (e) → R-1222 (hub). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — **2026-10-23: (d) released in controller 0.340.0;** (a), (c), (f), (g) stay as written. OPEN — 2026-10-22 night: (b) half closed, (d) fixed in code, (e) split out. | — | (a) with the next agent release (a unit test with a future archive); (c) a 9202 recipe; (g) a unit test with the 2027 clock; (f) to the operator at the next design review | CC | | **R-1190** | Backup & restore | P3 | **Key and retention candidates from the data-loss hunt — what is left:** (c) PLAUSIBLE: after a reuse-path reinstall the new PBS key K is never escrowed and nothing compares fingerprints (`felhom-pbs-apply:99`, `escrow_confirm.go:182-205`); (d) whole-guest off-site copies run before any escrow exists — a design question against `07` §6.1's Tier-3 gate (the operator's); (e) the escrow PUT is two statements; (f) the operator's off-site re-issue resets the provider password before saving it (hub `offsite.go:217-245`); (h) a unit restore can point an app back at an old drive and F5 then prunes the current drive's unit (`backup.go:1517-1574`) — (e), (f), (h) not re-read. **Moved out 2026-10-22 night:** (a) → R-1218, (b) → R-1219, (g) → R-1220. Evidence `audits/dataloss-hunt-2026-10-20/C1-keys-candidates.md`, `C1-delete-retention-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN | — | (c) a read-only fingerprint compare on a reinstalled test box; (e), (f), (h) re-read, then unit tests; (d) to the operator at the next design review | CC | @@ -201,14 +201,14 @@ stopping line that lies. | **R-1200** | Backup & restore | P3 | **A box installed in the afternoon gets no whole-box backup on its first night: the install's own backup starts the 24 h clock, so the family's whole first day reaches a whole-box copy only on the SECOND night (~38 h after install).** Rehearsal 2026-10-20 B5 (`audits/rehearsal-2026-10-20/B5-night/README.md`): Tester4's install backup at 12:09Z; at the 02:30–06:30Z window the agent answered „within cadence window" (`felhom-agent internal/localapi/server.go:1187`, `age >= tier.Cadence`, default 24 h). R-899 (closed) fixed the same shape for a daytime PRESS only. The other tiers did run that night (database copy, second drive, off-site — all PASS), so app data was covered; the whole-box (DR) tier was not. | **OPEN** — a design question: should the first scheduled night always run (like a press), or is one tier enough on night one? | — | decide; if „always run": treat the install backup like a press in the due rule, with a test on the first night | CC | | **R-1206** | Backup & restore | P3 | **Backups written while the box's clock ran ahead stay „the latest" after the clock is fixed.** The rehearsal's scenario #10 (clock set to 2026-10-28) left off-site snapshots dated 2026-10-28; on 2026-10-21 the rebuilt box's recovery page listed every app's „Legutóbbi mentés: 2026. 10. 28. 07:39", and the full restore said „mentés: 2026. 10. 28. 07:37" — a date in the future (`audits/fixes-2026-10-21/P1-live/07-unlock-right.txt`, `11-nextcloud-full-restore.txt`). restic orders snapshots by their own time, so the future ones are „newest" to every retention rule for a week. NOT measured: whether the box's keep-daily prune then drops a real snapshot. Related R-1202, R-1204 (clock). | NARROWED — the NAS tier only. **2026-10-22 night (read, `backup/offbox_window.go:22-24, 130-137`):** on the Felhom off-site tier no real snapshot is lost — the guard refuses a prune while any snapshot is future-dated (R-822, by design). The NAS tier prunes without that guard: open. **2026-10-25:** the kept-data choice no longer takes such a future-dated copy as the newest (R-1271, controller `bd9f4f7`); retention is unchanged — this row. | — | A pure unit test with the policy simulator on the NAS tier and one future-dated snapshot; if a real one is dropped, give the NAS tier the same guard | CC | | **R-1212** | Backup & restore | P4 | **Two household texts on the backup path do not follow the box's language:** the share-restore summary (`SharesRestoreResult.FlashMessage`, `controller/internal/backup/shares_restore.go`) is built from Hungarian Go literals, and the off-site `LastError` for an interrupted run is the Hungarian literal „megszakadt futás (a vezérlő újraindult futás közben)" (`controller/internal/backup/backup.go`). An English household reads Hungarian there. Found by the night helper 2026-10-21. | OPEN | — | Bundle keys for both (rendered in the box language when written, R-557), with parity fixtures | CC | -| **R-1218** | Backup & restore | P2 | **A recovery ceremony done while off-site is switched off (or cleared) seals no off-site repository password, and the hub then demotes the GOOD older package to operator-only custody: the household's own recovery code no longer opens its off-site copy.** REAL in code (night 2026-10-22, read; split from R-1190 (a)): `web/escrow_handlers.go:217-224` stages the password only if `OffboxConfigured()` (`backup/offbox.go:695-710`), false when the target is cleared or disabled — although `ClearOffboxTarget` keeps the password file on purpose (`offbox_clear.go` header); the agent then seals no password (`cmd/felhom-agent/main.go:2981-3003`) and uploads sha ""; hub `SaveHostEscrow` sees `curSHA != ""` and moves the good package to retained custody; `OffsiteRecoveryOffer` reads "" as „covers no key” (`offbox.go:1855-1873`) and says nothing. Evidence `audits/night-2026-10-22/P3-candidates/analysis.md` §R-1190(a). | OPEN — proposed fix, not built (key-handling code: a fix needs a red-proof AND a live proof, and 9202 has no hub to prove it with) | — | Fix (~6 lines): stage when the password FILE exists (`PushOffboxPasswordForEscrow`, `offbox.go:800`, reads only the file). Red-proof: a web test that the stage runs with off-site cleared; a hub store test pinning „abc”→"" ⇒ superseded (and refuse that demotion). Live proof on a test box with a hub (demo-hp 9201) | CC | -| **R-1219** | Backup & restore | P2 | **During a recovery offer the backups page hides the „create your recovery code” ceremony (it would replace the key that opens the old off-site copy), but the global top bar still links to it and the start handler does not refuse.** REAL in code (night 2026-10-22, read; split from R-1190 (b)): `web/escrow_banner.go:23-35` ignores `recoveryOffer()`; `templates/backups_remote.html:144-160` hides it and its own comment names the harm (R-199); `escrow_handlers.go:180-245` starts anyway; reachable in offer shape (c), `offbox.go:1866-1870`. Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — proposed fix, not built (key-handling code) | — | Fix: the bar hides its link during an offer, and the start handler answers 409 with a Hungarian sentence; red-proof with a handler test; live proof with an offer state on a test box with a hub | CC | -| **R-1220** | Backup & restore | P2 | **„Megőrzött adatok” (kept data) can list an INSTALLED app's live folder as kept — and offer to delete it — when that app's app.yaml cannot be read.** REAL in code (night 2026-10-22, read; split from R-1190 (g)): a nil AppConfig drops the app from `liveDriveBinds` (`stacks/kept.go:220-235, 336-360, 372-389`); a parse error gives nil (`deploy.go:1113-1125`). Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — proposed fix, not built (delete-path code) | — | Fix: an app whose app.yaml is unreadable counts as live (fail closed); its folders are never listed as kept. Red-proof: a unit test with a broken app.yaml; 9202 proof with a throwaway app | CC | +| **R-1218** | Backup & restore | P2 | **A recovery ceremony done while off-site is switched off (or cleared) seals no off-site repository password, and the hub then demotes the GOOD older package to operator-only custody: the household's own recovery code no longer opens its off-site copy.** REAL in code (night 2026-10-22, read; split from R-1190 (a)): `web/escrow_handlers.go:217-224` stages the password only if `OffboxConfigured()` (`backup/offbox.go:695-710`), false when the target is cleared or disabled — although `ClearOffboxTarget` keeps the password file on purpose (`offbox_clear.go` header); the agent then seals no password (`cmd/felhom-agent/main.go:2981-3003`) and uploads sha ""; hub `SaveHostEscrow` sees `curSHA != ""` and moves the good package to retained custody; `OffsiteRecoveryOffer` reads "" as „covers no key” (`offbox.go:1855-1873`) and says nothing. Evidence `audits/night-2026-10-22/P3-candidates/analysis.md` §R-1190(a). | **OPEN — fixed on `main` (both halves), red-proved; LIVE PROOF PENDING** — controller `1170b48` (stage when the password FILE exists) + hub `09b5f02f` (a keyed package is never replaced by a key-less one: 409). The live shape needs off-site switched OFF on a hub box: the household cannot switch a hub tier off (only per-app switches; a clear is refused by design), so it needs the operator's customer-page change on Tester 1 during Tuesday's attended block — otherwise both halves are REVERTED before the release (rule 2). **Earlier:** OPEN — proposed fix, not built (key-handling code: a fix needs a red-proof AND a live proof, and 9202 has no hub to prove it with) | — | Fix (~6 lines): stage when the password FILE exists (`PushOffboxPasswordForEscrow`, `offbox.go:800`, reads only the file). Red-proof: a web test that the stage runs with off-site cleared; a hub store test pinning „abc”→"" ⇒ superseded (and refuse that demotion). Live proof on a test box with a hub (demo-hp 9201) | CC | +| **R-1219** | Backup & restore | P2 | **During a recovery offer the backups page hides the „create your recovery code” ceremony (it would replace the key that opens the old off-site copy), but the global top bar still links to it and the start handler does not refuse.** REAL in code (night 2026-10-22, read; split from R-1190 (b)): `web/escrow_banner.go:23-35` ignores `recoveryOffer()`; `templates/backups_remote.html:144-160` hides it and its own comment names the harm (R-199); `escrow_handlers.go:180-245` starts anyway; reachable in offer shape (c), `offbox.go:1866-1870`. Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | **VERIFY — fixed on `main`, waits for the controller release** — controller `1170b48` (the top bar hides the ceremony during a recovery offer; the start handler answers 409 `api.escrow.recovery_offer_open`), red-proved with a real session and a no-offer control. The offer state cannot be staged without hand-set state, so the handler test is the proof (the brief's allowance). **Earlier:** OPEN — proposed fix, not built (key-handling code) | — | Fix: the bar hides its link during an offer, and the start handler answers 409 with a Hungarian sentence; red-proof with a handler test; live proof with an offer state on a test box with a hub | CC | +| **R-1220** | Backup & restore | P2 | **„Megőrzött adatok” (kept data) can list an INSTALLED app's live folder as kept — and offer to delete it — when that app's app.yaml cannot be read.** REAL in code (night 2026-10-22, read; split from R-1190 (g)): a nil AppConfig drops the app from `liveDriveBinds` (`stacks/kept.go:220-235, 336-360, 372-389`); a parse error gives nil (`deploy.go:1113-1125`). Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | **VERIFY — fixed on `main`, waits for the controller release** — 2026-10-26 night: controller `7e15a37` (an app whose `app.yaml` exists but does not parse counts as installed; its folders are never kept data), red-proved; live on 9202 (the kept-data page did not list the folder) — `audits/night-fixes-2026-10-26/`. **Earlier:** OPEN — proposed fix, not built (delete-path code) | — | Fix: an app whose app.yaml is unreadable counts as live (fail closed); its folders are never listed as kept. Red-proof: a unit test with a broken app.yaml; 9202 proof with a throwaway app | CC | | **R-1221** | Backup & restore | P4 | **The nightly whole-box backup starts ~1.5–5 min later each night, until one night is skipped (after ~70–80 nights with no press between).** REAL, low harm (night 2026-10-22; split from R-1189 (b)): agent `localapi/server.go:1184` is `age >= Cadence` with the age counted from the archive START; the controller polls every 5 min (`quiesce.go:150`) and the archive starts after the apps stop. Measured on demo-hp: 04:31:07 → 04:33:18 → 04:36:47 → 04:38:14, and 04:33:37 → 04:38:44 → 04:42:40. At W+6h one night is skipped (44 h is below the 48 h valve, `quiesce.go:807-818`); the hub's 26 h line (`deadline.go:29`) mails the operator. Any manual or off-cycle run resets the drift. Evidence `audits/night-2026-10-22/P3-candidates/demohp-whole-box-completions.txt`. | OPEN | — | Fix: inside the night window a tier is due at `age >= cadence - 4h`; the agent answers due-ness, so it is agent code (a unit test with an injected clock) and rides an agent release | CC | | **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN — **2026-10-23 — D35 ruled YES, after the release week (operator):** serve the website and the installer from Cloudflare like the ISO. Dated: start on or after 2026-10-30 (DUE-CHECKS). OPEN | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | CC (ruled yes 2026-10-23, D35) | | **R-1229** | Backup & restore | P4 | **ArgoCD's per-app auto-sync settings live only on the cluster, not in git.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G9): the Applications differ (`enabled` true/false, prune, selfHeal); a rebuilt cluster from `homelab-manifests` would not have them. | OPEN | — | Write each Application's sync policy into `homelab-manifests` (a homelab change, the operator's yes for the commit that changes live behaviour) | operator | | **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC | -| **R-1232** | Backup & restore | P2 | **A reinstalled box may take its first whole-box copy to ep0 up to a week late: the old box's copy in the same household space seems to count as its last one.** Test night 2026-10-23 (`audits/night-2026-10-23/P4-testnight/`): the new Tester4 box (guest 9201) had its PBS tier applied at 19:28 UTC, but no whole-box run to ep0 came all night; the only archive in the household's ep0 space is the rehearsal box's `ct/9201` of 2026-10-20 (same guest number), and the tier's cadence is 168 h. The agent's due-check reads the newest archive for that vmid on the tier's storage (`localapi/server.go` `newestArchiveOn`) — so the old box's copy reads as „2.6 days old, not due”. If so, the ep0 copy is the OLD box's for up to a week. PLAUSIBLE — read from the code and the night's silence, not proven by a log line (the controller logs the due-check only at debug). | OPEN | — | Prove: read `GET /backup/tiers` for the box (the agent's answer per tier) on the next reinstall; fix: a tier's „last backup” counts only archives this host wrote (or after the host's enrollment); an agent test with a foreign archive in the namespace | CC | +| **R-1232** | Backup & restore | P2 | **A reinstalled box may take its first whole-box copy to ep0 up to a week late: the old box's copy in the same household space seems to count as its last one.** Test night 2026-10-23 (`audits/night-2026-10-23/P4-testnight/`): the new Tester4 box (guest 9201) had its PBS tier applied at 19:28 UTC, but no whole-box run to ep0 came all night; the only archive in the household's ep0 space is the rehearsal box's `ct/9201` of 2026-10-20 (same guest number), and the tier's cadence is 168 h. The agent's due-check reads the newest archive for that vmid on the tier's storage (`localapi/server.go` `newestArchiveOn`) — so the old box's copy reads as „2.6 days old, not due”. If so, the ep0 copy is the OLD box's for up to a week. PLAUSIBLE — read from the code and the night's silence, not proven by a log line (the controller logs the due-check only at debug). | **VERIFY — fixed on `main`, waits for the agent release (the operator's vouch)** — 2026-10-26 night: agent `3911cc6` (the due-check counts only archives with this storage's key, the R-727 discriminator), red-proved; live on a fresh BYO Tester4 box (VM 9375): the released agent said `due: false` (the old box's archive, 6.3 days), the fixed one `due: true / absent`, and the first whole-box copy ran 4 minutes later — `audits/night-fixes-2026-10-26/`. Hub host `Tester4-b5dbbd` to delete (operator). **Earlier:** OPEN | — | Prove: read `GET /backup/tiers` for the box (the agent's answer per tier) on the next reinstall; fix: a tier's „last backup” counts only archives this host wrote (or after the host's enrollment); an agent test with a foreign archive in the namespace | CC | | **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC | | **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC | | **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC | @@ -269,7 +269,7 @@ stopping line that lies. | **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC | | **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC | | **R-974** | Box system & updates | P3 | **A compromised host agent can likely become root on the host through the root wrapper that applies PBS settings: the wrapper accepts a folder path inside the agent's own writable tree and then acts on it as root.** Static (the symlink behaviour was reasoned, not run). Same class as R-861. Fix: a fixed root-owned folder, and refuse symlinks before any root action. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix the wrapper with a test on a throwaway box | CC | -| **R-972** | Box system & updates | P2 | **The shipped controller and hub programs are built with Go 1.24.13 and golang.org/x/crypto v0.31.0, and the agent with Go 1.26.0, so dozens of fixed Go security bugs (including html/template escaping fixes) are still in what runs on boxes and the hub.** Go 1.24 has had no security fixes since Go 1.26 was released (Go supports the two newest major releases: https://go.dev/doc/devel/release#policy); the scanners name fixes in Go 1.26.9 and 1.27.2. Also: the controller image's Debian 12 base has 12 fixable critical packages, and the Samba image has 48 fixable high ones. Fix: build all three on a supported, patched Go; update x/crypto; rebuild the base images. Scanner output: `audits/security-review-2026-10-12/scanners/`. | **NARROWED** 2026-10-16 (burndown 2): **controller v0.322.0 released** on Go 1.27.2 + x/crypto v0.58.0, runtime debian:12.13-slim + `apt-get upgrade` (Debian 13 deliberately not: restic 0.18, R-1105) — scanner: critical 16 → 4, fixable criticals and highs in the OS 0; govulncheck 62 → 1 (module-level, `openpgp` not linked). **Hub v0.156.0 deployed** 2026-10-16 19:26 on the operator's yes (Go 1.27.2, alpine 3.24.2: 0 critical, 0 high); Synced/Healthy, reports from all three boxes after the start, the floor reads 0.322.0. **Agent: Go 1.27.2 committed (`37b99d7`), tested, NOT released** — the next vouched agent release carries it. Open: the Samba image (48 fixable high, not rebuilt); docker-compose plugin's own Go (4 high, upstream). Evidence `audits/burndown2-2026-10-16/P4/` **2026-10-16 (evening): the Samba part is DONE in controller v0.323.0** — felhom-samba 1.2.0 on Alpine 3.24.2 (0 critical / 0 high, was 48 fixable high), proven on demo-hp with the same household password (smbclient, nmbd, mDNS, WS-Discovery); on the three test boxes now, on every box with the global floor (R-1105's night check). The controller's Debian 13 OS part: 1 critical / 70 high, none fixable. Left: docker-compose plugin's own Go (4 high, upstream); the agent release; restic's binary is R-1107. **2026-10-17 (evening): agent v0.159.1 (Go 1.27.2) RELEASED and running on demo-hp, Tester 1 and demo-felhom** (signed jobs on the operator's yes); waiting for the operator's vouch — then only the upstream compose plugin is left (close then, recorded). | the next vouched agent release | Close when the global floor carries v0.323.0 and the next vouched agent release carries Go 1.27.2 (compose plugin: upstream, recorded) | CC | +| **R-972** | Box system & updates | P2 | **The shipped controller and hub programs are built with Go 1.24.13 and golang.org/x/crypto v0.31.0, and the agent with Go 1.26.0, so dozens of fixed Go security bugs (including html/template escaping fixes) are still in what runs on boxes and the hub.** Go 1.24 has had no security fixes since Go 1.26 was released (Go supports the two newest major releases: https://go.dev/doc/devel/release#policy); the scanners name fixes in Go 1.26.9 and 1.27.2. Also: the controller image's Debian 12 base has 12 fixable critical packages, and the Samba image has 48 fixable high ones. Fix: build all three on a supported, patched Go; update x/crypto; rebuild the base images. Scanner output: `audits/security-review-2026-10-12/scanners/`. | **NARROWED — Go 1.27.3 committed in all three, waits for the releases** — 2026-10-26 night: agent `fbbb168`, controller `2560cf1` (+ builder `golang:1.27.3-trixie`), hub `d625a143` (+ `golang:1.27.3-alpine3.24`); govulncheck 1 → 0 called vulnerabilities each (GO-2026-6809, `crypto/x509`); x/crypto v0.58.0 already newest; full tests green — `audits/r972-go1273-2026-10-26/`. **Earlier:** **NARROWED** 2026-10-16 (burndown 2): **controller v0.322.0 released** on Go 1.27.2 + x/crypto v0.58.0, runtime debian:12.13-slim + `apt-get upgrade` (Debian 13 deliberately not: restic 0.18, R-1105) — scanner: critical 16 → 4, fixable criticals and highs in the OS 0; govulncheck 62 → 1 (module-level, `openpgp` not linked). **Hub v0.156.0 deployed** 2026-10-16 19:26 on the operator's yes (Go 1.27.2, alpine 3.24.2: 0 critical, 0 high); Synced/Healthy, reports from all three boxes after the start, the floor reads 0.322.0. **Agent: Go 1.27.2 committed (`37b99d7`), tested, NOT released** — the next vouched agent release carries it. Open: the Samba image (48 fixable high, not rebuilt); docker-compose plugin's own Go (4 high, upstream). Evidence `audits/burndown2-2026-10-16/P4/` **2026-10-16 (evening): the Samba part is DONE in controller v0.323.0** — felhom-samba 1.2.0 on Alpine 3.24.2 (0 critical / 0 high, was 48 fixable high), proven on demo-hp with the same household password (smbclient, nmbd, mDNS, WS-Discovery); on the three test boxes now, on every box with the global floor (R-1105's night check). The controller's Debian 13 OS part: 1 critical / 70 high, none fixable. Left: docker-compose plugin's own Go (4 high, upstream); the agent release; restic's binary is R-1107. **2026-10-17 (evening): agent v0.159.1 (Go 1.27.2) RELEASED and running on demo-hp, Tester 1 and demo-felhom** (signed jobs on the operator's yes); waiting for the operator's vouch — then only the upstream compose plugin is left (close then, recorded). | the next vouched agent release | Close when the global floor carries v0.323.0 and the next vouched agent release carries Go 1.27.2 (compose plugin: upstream, recorded) | CC | | **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **2026-10-24 02:4xZ dated check read (weekend-night session, read only):** Tester 1 (8 GB host, guest 4 GB, controller 0.341.0) ran its 03:30 copy at 01:30:00–01:30:13Z (3 apps, 157 MB + 86 MB + 16 KB; no „capped copy container did not start" WARN, no leftover `felhom-tier2-copy-*`); the guest cgroup's memory „full" stall counter on the host (`/sys/fs/cgroup/lxc/9201/memory.pressure`) read **total 444 µs** over the whole 11.1 h since the host started (15:32Z), so „full" stayed far under 20 % (20 % of 10 s = 2 s); 0 OOM kills. **Not shown:** that the copy ran in the capped container (the INFO log does not name it; Docker keeps only ~200 events, from 02:36Z). **Weak load:** 0.24 GB, not the soak's large copy — so the row stays WATCHING; close after a night whose copy moves ≥ 1 GB, read from the same counter. WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC | | **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator | | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC | @@ -287,7 +287,7 @@ stopping line that lies. | **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). | **VERIFY** — RELEASED in controller v0.344.0 (global floor 2026-10-26 09:07:47Z, every live box on 0.344.0); the live test with a seeding box is Tuesday 10:00–16:00 (R-1250's dated check). **Earlier:** **VERIFY** — fixed on main 2026-10-25 night (controller `a79a292`), waits for the controller release (before any household gets agent 0.165.0 by signed update): every running seeder held, a removed app's record dropped, qBittorrent's slot named (`up_limit`/`alt_up_limit`). Red-proofs `audits/night-2026-10-25/redproof-r1261.txt`. **Live on the night's rig** (BYO VM, agent 0.165.0, rc.1, two seeders): both held, 94.8 Mbit/s, both limits back; a speed-mode switch mid-test left no 1 KiB/s (`rig/hold-watch*.txt`); the removed-app case by test only. **Earlier state:** OPEN — **now in controller 0.343.0 (released 2026-10-25); the three edges act only on a box whose agent is ≥ 0.165.0 (it alone reports `waiting_for_quiet`) — fix before any household gets agent 0.165.0 by signed update** | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC | | **R-1262** | Monitoring & notifications | P3 | **The upload watch treats ANY viewer as an explanation for any amount of upload, so a hacked Plex can send freely while one family member watches.** Found 2026-10-24 (SEC-1 X1): `felhom-controller/controller/internal/upwatch/watch.go:275,299`; `08` §9 states the rule, its limits list did not name this case. | OPEN | — | propose a cap (viewers × a bitrate ceiling) for the operator's R-1244 design; `08`'s limits list names the case meanwhile | CC | | **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC | -| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | +| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING — DESIGN WRITTEN 2026-10-26 night (the brief's Part 7: build only if time is left; it was not built — the line test and the attended release came first).** The host-side counter EXISTS and is readable (read on demo-hp 2026-10-26): `/sys/fs/cgroup/lxc//ns/system.slice/docker-<64-hex id>.scope/memory.events`, line `oom_kill N`. **Design:** (1) agent: a read-only collector (no root CLI — a sysfs read under the agent's own rights; if not readable, the existing sudoers fence gets one read-only `Cmnd_Alias`) that, for each guest in the `felhom` pool, globs those scopes and keeps `{container_id: oom_kill}`; served on the guest's own local API as `GET /guest/oom-kills` (`withGuest`, the guest's containers only). (2) controller: the 30-s dead-app scan asks once per pass; a RISE of a container's counter = an OOM kill whatever Docker's `OOMKilled` says → the existing `app_oom` line (R-636's per-run grain), `source: cgroup` in the details. An agent without the route = today's behaviour. (3) Tests: a fake cgroup tree (agent), a rising counter → one `app_oom` (controller, red-first). (4) Live: 9202, a throwaway app capped at 64M with a memory hog: the counter rises, `app_oom` names the container while `OOMKilled` reads false. ~1 day, two repos, one agent release. **Earlier:** WATCHING — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | | **R-1250** | Monitoring & notifications | P3 | **A qBittorrent that seeds all day is never line-tested, so it never gets the limit that exists for it.** The weekly test (R-1247) starts only after 10 minutes in which the host's uplink sent < 2 Mbit/s each minute; a seed at the line's speed keeps it busy through every window, so no test, no tested line, and no qBittorrent limit (D41 uses the tested line only). Found 2026-10-24 by reading the rule against the case it serves; not yet seen on a box (no test box seeds). | **VERIFY — released 2026-10-25, the live proof is a dated check:** controller 0.343.0 (global floor 17:53Z) + agent 0.165.0 (vouched 18:38 CET) carry it; the real proof cannot run before the next due window — the agent starts a test only on its due Tuesday 10:00–15:59 local, and every box measured on 2026-10-24. **DUE 2026-10-27:** a throwaway qBittorrent on demo-hp seeding to DooPlex's qBittorrent (operator's yes 2026-10-25, about 30 minutes), from before the host's slot; read the agent's line value and the limit back. **Earlier: NARROWED 2026-10-24 — FIXED on main, not released** (brief 2026-10-24 Part 4, commit only): agent `29bcf44`+`1e087bb` (`waiting_for_quiet`, `POST /host/line-test/run`), controller `046b808` (`internal/upwatch/linehold.go`: record first, hold at 1 KiB/s, test, the exact old limit back; the start-up check after a kill; the 50 % rule waits). Red-proofs in both CHANGELOGs. **Live on demo-hp with a real seed** (10 Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; held: uplink 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, back to 1 249 280 B/s exactly; a kill -9 mid-test left the record and the start-up check put the limit back (`audits/dooplex-alarms-2026-10-24/part4-r1250-live-proof.txt`). The live run used probes over plain HTTP (the pinned agent client is unit-tested, not live). | the next release train (agent ≥ 0.165.0 needs the operator's vouch) | Release agent + controller in the next normal train; on a seeding box read one Tuesday's `line test on request` line; then close | CC | | **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Design note (2026-10-21, after R-1192's diagnosis): the freeze's signature is cheap to see from the HOST side — the guest cgroup's memory PSI „full" above ~50 % for a minute, or its `memory.events max` climbing, while the host still reports; the agent could raise `guest_memory_thrash` (operator mail in its own words, not „no report for 45 m") and the household banner „A dobozod most nagyon lassú…". With the swap fix (R-1192) the measured thrash ends within a minute, so this alarm should be built only if a freeze is seen again after the agent release; until then it stays a design. Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC | | **R-1202** | Monitoring & notifications | P3 | **A box whose clock is behind blames the hub and stops an app without telling anyone.** Rehearsal 2026-10-20 B6 #8: RTC set 2 years back (the OS raised it to 2026-04-13): the dashboard shows only „Hub nem elérhető" while the log says „x509: certificate … not yet valid: current time 2026-04-13…"; Paperless crash-looped and the box stopped it with a restore hold; the `app_stopped_unhealthy` event failed (hub unreachable), so neither the household nor the operator heard. After the real time came back the app stayed stopped until a Start (clear sentence, 1 min). | **OPEN** | — | say „the box's clock is wrong (it says )" when TLS fails with not-yet-valid/expired against a hub that answers; keep the stop event queued until the hub is reachable | CC | @@ -399,5 +399,6 @@ stopping line that lies. | R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row | | **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC | -| **R-1298** | Monitoring & notifications | P3 | **Scratch guest 9202 carries leftovers made outside the product, and D53 would remind about them every week once released: four crash-loop holds since 2026-09-24 for apps that are not installed (`chaoscrash`, `chaosoom`, `chaosoomb`, `gokapi`), and a PrivateBin `app.yaml` (2026-10-25) with no container that the product reads as „installed".** Found 2026-10-26 evening in the D53 live proof: the new reporter sent the four holds at its first tick and the test hub mailed four reminders („held 32 days"). The product's removal clears an install's hold (R-704); these apps were removed by hand in a chaos drill. On a household box the same shape would mail the operator weekly about an app that does not exist. | OPEN | — | decide whether the reporter (or the hub) skips an `unhealthy_stop` / `update_failed` / `putback_pending` hold whose app is not installed (a restore hold stays reported — operator-cleared by design, R-379); then clear 9202's four holds with `-clear-restore-hold` and remove PrivateBin through the product | CC | +| **R-1298** | Monitoring & notifications | P3 | **Scratch guest 9202 carries leftovers made outside the product, and D53 would remind about them every week once released: four crash-loop holds since 2026-09-24 for apps that are not installed (`chaoscrash`, `chaosoom`, `chaosoomb`, `gokapi`), and a PrivateBin `app.yaml` (2026-10-25) with no container that the product reads as „installed".** Found 2026-10-26 evening in the D53 live proof: the new reporter sent the four holds at its first tick and the test hub mailed four reminders („held 32 days"). The product's removal clears an install's hold (R-704); these apps were removed by hand in a chaos drill. On a household box the same shape would mail the operator weekly about an app that does not exist. | **VERIFY — fixed on `main`, waits for the controller release** — controller `b156e45` (an install's hold of an app that is not installed is not reported; a restore hold always is), red-proved; live on 9202 (the four orphans skipped by name). 9202's four holds cleared with `-clear-restore-hold` and the PrivateBin leftover removed through the product — `audits/night-fixes-2026-10-26/`. Close at the release. **Earlier:** OPEN | — | decide whether the reporter (or the hub) skips an `unhealthy_stop` / `update_failed` / `putback_pending` hold whose app is not installed (a restore hold stays reported — operator-cleared by design, R-379); then clear 9202's four holds with `-clear-restore-hold` and remove PrivateBin through the product | CC | | **R-1299** | Process & tooling | P4 | **The operator's one-shot clean-up grant (`POST /offsite/window-grant/`, optional `max_remove`) has no button on any hub page** — the operator could not find it on 2026-10-26 (D55) and had to make a one-purpose access token so CC could call the route. The route, its scope (`offsite:write`) and its event exist (R-833); only the form is missing. | OPEN | — | a small form on the Off-site page (customer, optional cap ≤ 500), with a render test for both branches | CC | +| **R-1302** | Monitoring & notifications | P4 | **A fresh install's agent logs ERROR „store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)" and „capability DEGRADED" for `felhom-pbs` at its FIRST start, although the grant was simply never made yet** — the whole-guest tier is armed (21:00:45) before the PBS storage exists (created 21:00:57 on the hub's descriptor). Seen 2026-10-26 night on the R-1232 BYO box (VM 9375, `audits/night-fixes-2026-10-26/r1232/3-first-run.txt`) and on 2026-10-25's BYO box (`degraded=pve:store-grant:felhom-pbs`). An ERROR that says „investigate the loss" on every new box teaches the reader to skip it. | OPEN | — | on the first grant of a tier whose storage did not exist before, log INFO „granted" instead of the loss ERROR; keep the ERROR for a grant that existed and vanished; a test for both | CC |