night fixes 2026-10-26: live proofs (R-1220, R-1232, R-1187, R-1298), register states, design lines; R-1300 filed; R-528 design

Evidence audits/night-fixes-2026-10-26/ (scanned: 0 secret hits).

documentation/audits/night-fixes-2026-10-26/README.md

@@ -0,0 +1,67 @@
+# Night fixes 2026-10-26 → 27: live proofs (R-1220, R-1232, R-1187, R-1298)
+
+Code on `main`, NOT released (the release block is Tuesday afternoon, attended). Test images were **local only**
+(`0.346.0-night1/2/3`, `docker save`/`load` into 9202 — never pushed to the registry); the test agent
+(`0.167.0-night1`) was a local build copied to the test VM only. Method: the endpoints the UI invokes, through the box
+walk tools (`../catalog-sweep-2026-10-24/tools/`), and the product's own installer / uninstaller.
+
+## R-1220 — an unreadable `app.yaml` counts as installed (controller `7e15a37`)
+
+9202, throwaway Grimmory, its `app.yaml` made unreadable (byte copy saved, put back): after a stack rescan the
+„Megőrzött adatok" page did **not** list its drive folder, and the controller logged the R-1220 WARN.
+`r1187-r1220-9202-run1.txt` (step 2). PASS.
+
+## R-1232 — a reinstalled box's first whole-box copy is not a week late (agent `3911cc6`)
+
+VM 9375 on demo-hp's scratch NVMe: plain Proxmox (auto-install ISO with DooPlex's key), then the product's BYO installer
+as **Tester4** (host `Tester4-b5dbbd`). Tester4's ep0 space held one archive of an EARLIER box (`ct/9201`, 2026-10-20,
+key `23:58:…`); this box's key is `7c:3b:…`.
+
+| | Agent | `GET /backup/due?target=felhom-pbs` |
+|---|---|---|
+| before | 0.166.0 (released) | `due: false`, „within cadence window", age 546,807 s — the foreign archive counted |
+| after | 0.167.0-night1 (`main`) | `due: true`, „no successful backup recorded yet", `age_state: absent` |
+
+The controller's first-backup valve then started the first whole-box copy at 20:13:55Z; it completed at 21:15:17 CET
+(`ct/9201/2026-10-26T20:13:55Z`, this box's key) — 4 minutes after the swap. `r1232/1-…`, `2-…`, `3-first-run.txt`.
+PASS. **Teardown:** the product's uninstall ran (`r1232/uninstall-console.txt`, `UNINSTALL_RC=0`; the first try stopped
+at the confirmation prompt — it asks for the guest number, nothing destroyed), then VM 9375 destroyed, its ISO and work
+files removed. **The operator deletes hub host `Tester4-b5dbbd`.** Tester4's ep0 space keeps this box's one archive
+(a host delete keeps the tenancy, by design).
+
+## R-1187 — the `.fab` export and import (controller `dfd0f8e` … `90a2530`)
+
+9202, throwaway Grimmory (markers inside its bound drive folder `appdata/grimmory/data/cc-r1187`: a file owned
+1000:1001 mode 0640, a link inside, a link to `/etc/hostname`):
+
+- (a) a FIFO in the folder → the export **fails**, naming the FIFO's path (`r1187-9202-run2.txt`, `-run3.txt`, step 3);
+  without it the export carries both links AS links (the tar listing, step 4).
+- (b) after the import: the file is back as 1000:1001 mode 640, the inside link is back, the outside link is **refused**
+  and listed in the import's WARN (run 2 and run 3).
+- **Two import defects found by this proof, older than tonight, fixed and re-proven:** run 1 and run 2 ended
+  „waiting for DB: timeout" and left Grimmory down: (1) the import's compose runs lacked the controller's own
+  `USERDATA_PATH`/`IMPORT_PATH`, and compose's warning was read as the DB container id (`a3ed63e`); (2) the import used
+  `mysqladmin`/`mysql`, which the MariaDB 11+ images do not ship, so EVERY MariaDB 11+ import timed out (`90a2530`: the
+  import now loads through `appbackup.ImportDump`). **Run 3 on `0.346.0-night3`: the import succeeds and the app
+  answers.** `r1187-9202-run3.txt`.
+- (g) AdventureLog (postgis): a superuser made with the app's own `createsuperuser`, export, the user deleted in the live
+  app, import → the user is back (read through the app's own ORM, `manage.py shell`). My front-door check (the Django
+  admin login page) answered 302 with no form both BEFORE and after — an instrument that did not reach the form, not a
+  result. `r1187g-9202.txt`. PASS on the ORM readback.
+- (d) and (f): their unit tests are the proof (a running restore cannot be staged safely; a crafted bundle would be a
+  hand-made input).
+
+## R-1298 — held-app reports only for installed apps (controller `b156e45`)
+
+9202 on `0.346.0-night1`: the reporter skipped its four orphan holds by name („not reported (R-1298)", 0 sent). Then,
+the controller stopped, `-clear-restore-hold` for `chaoscrash`, `chaosoom`, `chaosoomb`, `gokapi` → „no restore holds in
+force"; PrivateBin's leftover removed through the product (volume, `app.yaml`, backup gone). `r1298/`.
+
+## Teardown
+
+- **9202:** back on 0.343.0 (image file restored, the test images removed); its standing apps only; no holds (the four
+  orphans cleared on purpose, per the brief); the drive's `appdata/` holds only paperless; no bundles in `exports/`.
+- **demo-hp:** VM 9375 destroyed (own id), its ISO removed; nothing else provisioned.
+- **Hub:** host `Tester4-b5dbbd` — the operator's delete. **DooPlex:** the local test images removed.
+- Secrets: the evidence was scanned (the dashboard passwords, Tester4's passphrase, both hub tokens, the Resend key):
+  0 hits.

documentation/audits/night-fixes-2026-10-26/r1187-9202-run2.txt

@@ -0,0 +1,33 @@
+  [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH']
+  [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
+  [1] deployed, controller state=running, pinned={'grimmory': 'ghcr.io/grimmory-tools/grimmory:v3.5.0', 'grimmory-db': 'mariadb:11.4'}
+    gate: grimd53 is gated — passed as the household (cookie set)
+1. grimmory installed and answering: True
+   its appdata folder: /mnt/felhom-drives/scratch_hdd/appdata/grimmory
+   markers: lrwxrwxrwx 1    0    0  9 Oct 26 20:46 link-in -> owned.txt
+lrwxrwxrwx 1    0    0 13 Oct 26 20:46 link-out -> /etc/hostname
+-rw-r----- 1 1000 1001  2 Oct 26 20:46 owned.txt
+2. R-1220 skipped (proven in the first run)
+3. export with a FIFO -> 200 {'ok': True}
+   job: Felhasználói adatok mentése sikertelen: a(z) /mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/pipe különleges fájl (cső, foglalat vagy eszköz), ezt nem lehet menteni — az exportálás leállt, semmi nem maradt ki csendben
+4. export -> 200 {'ok': True}
+   job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'export', 'ok': True, 'output_path': '/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-204714.fab', 'output_size': '3.2 MB', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Előkészítés', 'status': 'done'}, {
+   bundle: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-204714.fab
+   tar listing of the drive part: drwxr-xr-x root/root         0 2026-10-26 20:47 cc-r1187
+lrwxrwxrwx root/root         0 2026-10-26 20:46 cc-r1187/link-in -> owned.txt
+lrwxrwxrwx root/root         0 2026-10-26 20:46 cc-r1187/link-out -> /etc/hostname
+-rw-r----- 1000/1001         2 2026-10-26 20:46 cc-r1187/owned.txt
+   import -> 200 {'ok': True}
+   import job: {'display_name': 'Grimmory', 'done': True, 'error': 'Adatbázis visszaállítása sikertelen: waiting for DB: timeout waiting for mariadb to become ready', 'job_type': 'import', 'ok': True, 'output_path': '', 'output_size': '', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Csomag megn
+   after import: lrwxrwxrwx 1    0    0 9 Oct 26 20:47 link-in -> owned.txt
+-rw-r----- 1 1000 1001 2 Oct 26 20:47 owned.txt
+/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/owned.txt 1000:1001 640
+owned.txt
+ls: cannot access '/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/link-out': No such file or directory
+   the import's WARN: 2026/10/26 20:47:31 restore.go:695: [WARN] Import: data — 1 link(s) pointing outside the app's folder were NOT restored (R-1187 b): cc-r1187/link-out
+    app never answered on grimd53/ (last rc=0 code=404)
+   the app answers: False
+  [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'grimmory', 'volumes_removed': ['grimmory_grimmory_db'], 'hdd_paths_removed': ['/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data (12K)'], 'hdd_paths_preserved': []
+5. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '/mnt/felhom-drives/scratch_hdd/appdata/grimmory', 'kept': '', 'subdomain': '404'}
+   bundles of this run removed: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-204714.fab
+0

documentation/audits/night-fixes-2026-10-26/r1187-9202-run3.txt

@@ -0,0 +1,32 @@
+  [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH']
+  [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
+  [1] deployed, controller state=running, pinned={'grimmory': 'ghcr.io/grimmory-tools/grimmory:v3.5.0', 'grimmory-db': 'mariadb:11.4'}
+    gate: grimd53 is gated — passed as the household (cookie set)
+1. grimmory installed and answering: True
+   its appdata folder: /mnt/felhom-drives/scratch_hdd/appdata/grimmory
+   markers: lrwxrwxrwx 1    0    0  9 Oct 26 21:01 link-in -> owned.txt
+lrwxrwxrwx 1    0    0 13 Oct 26 21:01 link-out -> /etc/hostname
+-rw-r----- 1 1000 1001  2 Oct 26 21:01 owned.txt
+2. R-1220 skipped (proven in the first run)
+3. export with a FIFO -> 200 {'ok': True}
+   job: Felhasználói adatok mentése sikertelen: a(z) /mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/pipe különleges fájl (cső, foglalat vagy eszköz), ezt nem lehet menteni — az exportálás leállt, semmi nem maradt ki csendben
+4. export -> 200 {'ok': True}
+   job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'export', 'ok': True, 'output_path': '/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-210142.fab', 'output_size': '3.2 MB', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Előkészítés', 'status': 'done'}, {
+   bundle: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-210142.fab
+   tar listing of the drive part: drwxr-xr-x root/root         0 2026-10-26 21:01 cc-r1187
+lrwxrwxrwx root/root         0 2026-10-26 21:01 cc-r1187/link-in -> owned.txt
+lrwxrwxrwx root/root         0 2026-10-26 21:01 cc-r1187/link-out -> /etc/hostname
+-rw-r----- 1000/1001         2 2026-10-26 21:01 cc-r1187/owned.txt
+   import -> 200 {'ok': True}
+   import job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'import', 'ok': True, 'output_path': '', 'output_size': '', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Csomag megnyitása', 'status': 'done'}, {'label': 'Alkalmazás előkészítése', 'status': 'done'}, {'label': 'K
+   after import: lrwxrwxrwx 1    0    0 9 Oct 26 21:01 link-in -> owned.txt
+-rw-r----- 1 1000 1001 2 Oct 26 21:01 owned.txt
+/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/owned.txt 1000:1001 640
+owned.txt
+ls: cannot access '/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/link-out': No such file or directory
+   the import's WARN: 2026/10/26 21:01:57 restore.go:694: [WARN] Import: data — 1 link(s) pointing outside the app's folder were NOT restored (R-1187 b): cc-r1187/link-out
+   the app answers: True
+  [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'grimmory', 'volumes_removed': ['grimmory_grimmory_db'], 'hdd_paths_removed': ['/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data (12K)'], 'hdd_paths_preserved': []
+5. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '/mnt/felhom-drives/scratch_hdd/appdata/grimmory', 'kept': '', 'subdomain': '404'}
+   bundles of this run removed: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-210142.fab
+0

documentation/audits/night-fixes-2026-10-26/r1187-r1220-9202-run1.txt

@@ -0,0 +1,29 @@
+  [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH']
+  [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
+  [1] deployed, controller state=running, pinned={'grimmory': 'ghcr.io/grimmory-tools/grimmory:v3.5.0', 'grimmory-db': 'mariadb:11.4'}
+    gate: grimd53 is gated — passed as the household (cookie set)
+1. grimmory installed and answering: True
+   its appdata folder: /mnt/felhom-drives/scratch_hdd/appdata/grimmory
+   markers: lrwxrwxrwx 1    0    0  9 Oct 26 20:15 link-in -> owned.txt
+lrwxrwxrwx 1    0    0 13 Oct 26 20:15 link-out -> /etc/hostname
+-rw-r----- 1 1000 1001  2 Oct 26 20:15 owned.txt
+2. app.yaml made unreadable; rescan seen: True
+   R-1220: the kept-data page lists grimmory's folder: False (want False) | WARN lines: 2
+   app.yaml put back, byte check: 2b7b111d6bd71008
+3. export with a FIFO -> 200 {'ok': True}
+   job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'export', 'ok': True, 'output_path': '/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201737.fab', 'output_size': '3.2 MB', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Előkészítés', 'status': 'done'}, {
+4. export -> 200 {'ok': True}
+   job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'export', 'ok': True, 'output_path': '/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201745.fab', 'output_size': '3.2 MB', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Előkészítés', 'status': 'done'}, {
+   bundle: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201745.fab
+   tar listing of the drive part: 
+   import -> 200 {'ok': True}
+   import job: {'display_name': 'Grimmory', 'done': True, 'error': 'Adatbázis visszaállítása sikertelen: waiting for DB: timeout waiting for mariadb to become ready', 'job_type': 'import', 'ok': True, 'output_path': '', 'output_size': '', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Csomag megn
+   after import: ls: cannot access '/mnt/felhom-drives/scratch_hdd/appdata/grimmory/cc-r1187/link-out': No such file or directory
+   the import's WARN: 
+    app never answered on grimd53/ (last rc=0 code=404)
+   the app answers: False
+  [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'grimmory', 'volumes_removed': ['grimmory_grimmory_db'], 'hdd_paths_removed': ['/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data (4.0K)'], 'hdd_paths_preserved': [
+5. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '/mnt/felhom-drives/scratch_hdd/appdata/grimmory', 'kept': '', 'subdomain': '404'}
+   bundles of this run removed: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201737.fab
+/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201745.fab
+0

documentation/audits/night-fixes-2026-10-26/r1187g-9202.txt

@@ -0,0 +1,27 @@
+  [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
+  [1] deployed, controller state=unhealthy, pinned={'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.13.0', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine'}
+  [1] NOTE: the controller's own state is 'unhealthy', not 'running' — recorded, not treated as a failure; the fixture's front-door wait is the real gate
+    gate: advd53 not routed right now (404) — not cached, asked again next call
+    gate: advd53 not routed right now (404) — not cached, asked again next call
+    gate: advd53 not routed right now (404) — not cached, asked again next call
+    gate: advd53 not routed right now (404) — not cached, asked again next call
+    gate: advd53 not routed right now (404) — not cached, asked again next call
+    gate: advd53 not routed right now (404) — not cached, asked again next call
+    gate: advd53 is gated — passed as the household (cookie set)
+1. adventurelog installed and answering: True
+2. createsuperuser: Superuser created successfully. | in the ORM: True
+   front door before: no login form (302)
+3. export: ok 48.2 MB /mnt/felhom-drives/scratch_hdd/exports/adventurelog_20261026-210649.fab
+   the bundle's database: database/adventurelog-postgres.sql.gz
+manifest.json
+4. the user deleted in the live app (so only the import can bring it back) — in the ORM: False
+5. import: ok | steps: [('Csomag megnyitása', 'done'), ('Alkalmazás előkészítése', 'done'), ('Konfiguráció visszaállítása', 'done'), ('Felhasználói adatok visszaállítása', 'done'), ('Adatbázis visszaállítása', 'done'), ('Alkalmazás indítása', 'done')]
+   app answers: True
+   R-1187 g — the user back in the ORM: True
+   R-1187 g — the front door after import: no login form (302)
+   the import's DB line: 2026/10/26 21:01:59 export.go:166: [DEBUG] [appexport] restoreDatabase: found DB service=grimmory-db type=mariadb container=grimmory-db
+2026/10/26 21:07:18 export.go:166: [DEBUG] [appexport] restoring database: type=postgres
+2026/10/26 21:07:18 export.go:166: [DEBUG] [appexport] restoreDatabase: found DB service=adventurelog-postgres type=postgres container=adventurelog-postgres
+  [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'adventurelog', 'volumes_removed': ['adventurelog_adventurelog_media', 'adventurelog_adventurelog_postgres_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 
+6. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '', 'kept': '', 'subdomain': '404'}
+   bundles removed: 0

documentation/audits/night-fixes-2026-10-26/r1232/1-before-old-agent.txt

@@ -0,0 +1,41 @@
+== storage key fingerprint
+	encryption-key <present>
+	namespace Tester4
+felhom-pbs:backup/ct/9201/2026-10-20T12:17:26Z 9201 1792498646 23:58:7a:71:46:72:0c:1e
+own key: 
+== tiers (agent 0.166.0)
+{
+ "ok": true,
+ "data": {
+  "vmid": 9201,
+  "tiers": [
+   {
+    "target": "local",
+    "cadence_seconds": 86400,
+    "primary": true,
+    "storage": "present"
+   },
+   {
+    "target": "felhom-pbs",
+    "cadence_seconds": 604800,
+    "primary": false,
+    "storage": "present"
+   }
+  ]
+ }
+}
+20:10:33
+== due felhom-pbs (agent 0.166.0)
+{
+ "ok": true,
+ "data": {
+  "vmid": 9201,
+  "due": false,
+  "reason": "within cadence window",
+  "age_seconds": 546807,
+  "target": "felhom-pbs",
+  "age_state": "known"
+ }
+}
+own key: 7c:3b:36:51:bc:a6:b4:90
+20:10:53

documentation/audits/night-fixes-2026-10-26/r1232/2-after-fixed-agent.txt

@@ -0,0 +1,16 @@
+bin=/usr/local/bin/felhom-agent
+active
+felhom-agent 0.167.0-night1
+== due felhom-pbs (agent 0.167.0-night1, R-1232 fix)
+{
+ "ok": true,
+ "data": {
+  "vmid": 9201,
+  "due": true,
+  "reason": "no successful backup recorded yet",
+  "target": "felhom-pbs",
+  "age_state": "absent"
+ }
+}
+Oct 26 21:11:14 r1232-byo felhom-agent[17600]: time=2026-10-26T21:11:14.654+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=felhom-pbs volid=felhom-pbs:backup/ct/9201/202
+20:11:14

documentation/audits/night-fixes-2026-10-26/r1232/3-first-run.txt

@@ -0,0 +1,35 @@
+== archives after
+felhom-pbs:backup/ct/9201/2026-10-20T12:17:26Z 23:58:7a:71:46:72:0c:1e
+felhom-pbs:backup/ct/9201/2026-10-26T20:13:55Z 7c:3b:36:51:bc:a6:b4:90
+== due felhom-pbs after the run
+{
+ "ok": true,
+ "data": {
+  "vmid": 9201,
+  "due": false,
+  "reason": "within cadence window",
+  "age_seconds": 99,
+  "target": "felhom-pbs",
+  "age_state": "known"
+ }
+}
+== agent journal (felhom-pbs, R-1232)
+Oct 26 21:00:45 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:45.237+01:00 level=INFO msg="felhom-agent daemon starting" version=0.166.0 host_id=Tester4-b5dbbd hub_url=https://hub.felhom.eu interval_s=900
+Oct 26 21:00:45 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:45.255+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
+Oct 26 21:00:46 r1232-byo sudo[3034]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-backup-target-apply grant felhom-pbs
+Oct 26 21:00:47 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:47.454+01:00 level=ERROR msg="store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)" target=felhom-pbs privilege=Datastore.AllocateSpace action="felhom-bac
+Oct 26 21:00:47 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:47.454+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:store-grant:felhom-pbs feature="backup tier felhom-pbs: the agent'
+Oct 26 21:00:57 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:57.058+01:00 level=INFO msg="pbsdr: one-time token secret consumed (single-use; value withheld from logs)" storage_id=felhom-pbs secret_len=36
+Oct 26 21:00:57 r1232-byo sudo[3774]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply create felhom-pbs 10.77.0.1 felhom-offsite Tester4 felhom@pbs!Tester4 c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:6
+Oct 26 21:00:58 r1232-byo sudo[3788]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply grant felhom-pbs
+Oct 26 21:00:59 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:59.566+01:00 level=INFO msg="pbsdr: seeded escrow.pbs_storage_id (the ceremony one-liner needs no flags)" storage_id=felhom-pbs
+Oct 26 21:00:59 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:59.567+01:00 level=INFO msg="pbsdr: converged" state=applied storage_id=felhom-pbs
+Oct 26 21:11:00 r1232-byo felhom-os-apply[17411]: os-apply: CONFFILE updated /etc/debian_version (it was not changed locally)
+Oct 26 21:11:04 r1232-byo felhom-agent[17600]: time=2026-10-26T21:11:04.570+01:00 level=INFO msg="felhom-agent daemon starting" version=0.167.0-night1 host_id=Tester4-b5dbbd hub_url=https://hub.felhom.eu interval_s=900
+Oct 26 21:11:04 r1232-byo felhom-agent[17600]: time=2026-10-26T21:11:04.582+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
+Oct 26 21:11:05 r1232-byo sudo[17768]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
+Oct 26 21:11:06 r1232-byo sudo[17814]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
+Oct 26 21:11:14 r1232-byo felhom-agent[17600]: time=2026-10-26T21:11:14.654+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=felhom-pbs volid=felhom-pbs:backup/ct/9201/2026-10-20T12:17:26Z size_bytes=2072117616 
+Oct 26 21:13:56 r1232-byo felhom-agent[17600]: time=2026-10-26T21:13:56.633+01:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1793045635144752924
+Oct 26 21:15:17 r1232-byo felhom-agent[17600]: time=2026-10-26T21:15:17.431+01:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-10-26T20:13:55Z size_bytes=2072870896 uncovered_volumes=2
+Oct 26 21:15:17 r1232-byo felhom-agent[17600]: time=2026-10-26T21:15:17.431+01:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1793045635144752924 archive=felhom-pbs:backup/ct/9201/2026-10-26T20:13:55Z

documentation/audits/night-fixes-2026-10-26/r1232/install-console.txt

@@ -0,0 +1,163 @@
+
+[INFO] felhom-host-install v1.36.1 — mode=byo customer=Tester4 vmid=9201
+
+[STEP] 1/8 pre-flight
+[INFO]   pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
+[INFO]   node: r1232-byo (explicit)
+[INFO]   agent config: /etc/felhom-agent/agent.json
+[INFO]   agent: not installed yet — will be fetched + installed in step 5/8
+[INFO]   local-lvm free: ~29 GiB
+[WARN] local-lvm free ~29 GiB is below the recommended 120 GiB — the install continues; proceed only if you sized the grows deliberately
+[INFO]   free RAM: ~6277 MiB
+[INFO]   existing guests on this host: 0 (pct+qm)
+[INFO]   archive storage 'local' present
+[INFO]   acl storages all present or pre-positioned: local local-lvm
+[INFO]   dnsmasq: not present before Felhom — recorded; uninstall will remove it again if we install it
+[INFO]   host DNS (:53): free
+[INFO]   hub reachable (https://hub.felhom.eu)
+[OK]   customer 'Tester4' exists + passphrase valid
+[INFO]   golden: none local — will fetch + verify from Gitea in step 7/8
+r1232-byo
+
+[STEP] BYO disclosure — installing Felhom on a host the operator does NOT own will create:
+    pveum:  roles FelhomAgentBase/FelhomAgentGuest/FelhomAgentStore + user felhom-agent@pve + token felhom-agent@pve!agent
+            + scoped ACL grants (Base@/, Guest@/pool/felhom, Store@[local local-lvm]) + the 'felhom' pool
+    agent:  service user felhom-agent + /usr/local/bin/felhom-agent (+ .bak-* backups; A/B slots via the guarded self-update)
+            + /etc/systemd/system/felhom-agent.service (+ felhom-agent-rollback.service + start-limit drop-in)
+            + /etc/sudoers.d/felhom-agent + state dir /var/lib/felhom-agent + config /etc/felhom-agent/agent.json (0600)
+    host:   /usr/local/sbin/felhom-mkfs-guarded + /usr/local/sbin/felhom-selfupdate-guarded
+            + /usr/local/sbin/felhom-pbs-apply (PBS-DR apply wrapper — DR capability is baked
+              on every install; ACTIVATION stays a hub flag, off = zero effect on this host)
+            + felhom-mgmt-watchdog service+timer+script + /etc/tmpfiles.d/felhom-privsep.conf
+            + /usr/local/sbin/felhom-priv-apply + the guest-hook snippet under /var/lib/vz/snippets/
+              (both from the agent's config bundle)
+            + the 'sudo' and 'age' packages if absent + install state dir /var/lib/felhom-install
+    wg:     an OUTBOUND WireGuard tunnel to the Felhom hub (wg_tunnel.enabled=true — base
+            infrastructure like the cloudflared tunnel; hands-free peer registration; the
+            operator can disable it per-customer hub-side on request)
+    guest:  the provisioned Felhom LXC (vmid 9201, capped 2 cores / 4096 MiB) + its volumes
+            + the golden vzdump imported onto storage 'local'
+    reuse:  a golden already on 'local' is used only if its sha256 or its controller version
+            matches the hub's vouched golden; otherwise the vouched one is fetched (a --golden you name is refused)
+    update: operator-signed self-update authority: felhom-op-1
+    NOT touched in byo mode: root@pam (no break-glass), host DNS (:53), OOB sshd.
+
+Type this host's short hostname (r1232-byo) to acknowledge the byo install: [OK]   byo install acknowledged for host r1232-byo
+[OK] pre-flight passed
+[STEP] 2/8 Proxmox API token
+[OK]   token minted (secret captured, not logged)
+[OK]   scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm])
+[SKIP]   old broad role FelhomAgent already absent
+[STEP] 3/8 compute volume grows
+[INFO]   auto-computed from a 29 GiB thin pool (16 §7: 50/100/130 GiB)
+[INFO]   grows: rootfs +0G (->32G), data +26G (->50G, ONE volume)
+[STEP] 4/8 host enrollment (POST /host-enroll)
+[OK]   host MINTED (first enroll)
+[INFO]   host_id: Tester4-b5dbbd (api_key captured, not logged)
+[SKIP] break-glass (step 4b) SKIPPED (byo: the host owner manages break-glass; root@pam untouched, nothing vaulted)
+[STEP] 5/8 agent install (fetch + verify + install)
+[SKIP]   apt repo alignment skipped (byo — the owner manages repos)
+[WARN]   no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
+[INFO]   manifest: agent v0.166.0 (sha 3f22c65b78914632…), golden v0.344.0
+[INFO]   fetching agent binary v0.166.0 from Gitea …
+[OK]   verified sha256 3f22c65b78914632… matches the hub manifest
+[OK]   installed /usr/local/bin/felhom-agent (felhom-agent 0.166.0)
+[INFO]   installing the 'sudo' package (required for the non-root agent model) …
+[OK]   sudo installed (Sudo version 1.9.16p2)
+[INFO]   installing the 'age' package (escrow ceremony identity-wrap dependency) …
+[OK]   age installed (1.2.1)
+[OK]   created service user felhom-agent
+[OK]   added felhom-agent to systemd-journal (unprivileged journal read for NAS verify)
+[OK]   verified sha256 093772c1bda8b974… matches the hub manifest
+  os-apply: BUNDLE /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf skipped: a BYO host keeps its own settings (D43)
+  os-apply: BUNDLE START agent=0.166.0 sha=093772c1bda8b974 authority=installer files=32 write=26 same=1 kept=0 skipped=5
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mkfs-guarded (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-selfupdate-guarded (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-pbs-apply (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-backup-target-apply (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-vzdump-partial-clean (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-thin-check (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-internal-place (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-crash-guard (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-priv-apply (new)
+  os-apply: BUNDLE WROTE /var/lib/vz/snippets/felhom-guest-hook.sh (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-shared-parent.sh (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-shared-parent.service (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard.service (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.service (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.timer (new)
+  os-apply: BUNDLE WROTE /etc/felhom/crash-guard.conf (new)
+  os-apply: BUNDLE WROTE /etc/grub.d/01_felhom_oneshot (new)
+  os-apply: BUNDLE WROTE /etc/grub.d/42_felhom_oneshot (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent-rollback.service (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mgmt-watchdog (new)
+  os-apply: BUNDLE WROTE /etc/tmpfiles.d/felhom-privsep.conf (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.service (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.timer (new)
+  os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-agent (new)
+  os-apply: BUNDLE DONE agent=0.166.0 written=26 same=1 self-check=ok signers-created=False
+[OK]   installed every root-owned file from config bundle v0.166.0 (sha 093772c1bda8b974…; checked, previous copies kept)
+[SKIP]   OOB (felhom-sshd) off (byo, or appliance --no-oob) — the bundle skipped its files
+[STEP] 5b/8 internal data place (Belső tárhely)
+[SKIP]   byo: the host owner's thin pool is not ours to divide — no internal place
+[STEP] 6/8 agent config + service
+[WARN]   backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive.
+[WARN]   It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard.
+[INFO]   node=r1232-byo  local_api=192.168.0.127:8443  tls_fp=4B:20:94:3E:FB:AB…
+[OK]   wrote /etc/felhom-agent/agent.json (0600 felhom-agent)
+[OK]   byo config asserts passed (lan_resolver.enable / oob.enabled off; deployment_mode=byo; wg_tunnel intentionally ON — base infrastructure)
+[OK]   agent --selftest (read-only) passed
+[OK]   felhom-agent service active (non-root felhom-agent reads the config OK)
+[OK]   wrote /etc/felhom/os-trust.json + operator-signers (felhom-op-1) — the root-owned slow-lane anchors
+[STEP] 7/8 golden archive
+[WARN]   no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
+[INFO]   fetching golden v0.344.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_10_26-21_00_49.tar.zst
+[OK]   verified sha256 767a90fe07ed47c8… matches the hub manifest
+[OK]   golden imported + verified: local:backup/vzdump-lxc-9100-2026_10_26-21_00_49.tar.zst
+[STEP] 8/8 provision guest 9201
+[SKIP]   pool felhom already exists
+=== felhom-agent 0.166.0 selftest=provision (vmid=9201 customer=Tester4 hostname=tester4) ===
+  --- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_10_26-21_00_49.tar.zst → vmid 9201 ---
+time=2026-10-26T21:02:13.966+01:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=9201 pool=felhom
+  [OK] front half: vmid 9201 up (boot+running) in 1m14s; MAC=BC:24:11:06:5B:AC
+  --- back half: mint per-guest token + populate bootstrap config mount ---
+time=2026-10-26T21:02:19.331+01:00 level=INFO msg="provision: back-half complete" vmid=9201 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=192.168.0.127:8443
+  [OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 9201 (host dir /var/lib/felhom-agent/guests/9201/bootstrap)
+  local-api endpoint 192.168.0.127:8443 · leaf fp c0463612a04329e8b367f3aed7e925da5b5e44668df7b726fc5e7ac4dd6837ad · token: minted (not printed)
+=== selftest=provision OK — guest 9201 provisioned + bootstrap-mounted (KEPT) ===
+    next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller,
+    which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api.
+[OK]   provision completed
+[INFO]   rebooting guest 9201 so the baked controller-bootstrap unit picks up the mount
+[STEP] verify
+[OK]   pct status: running
+[OK]   onboot: 1
+    mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G
+    mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
+    rootfs: local-lvm:vm-9201-disk-0,size=32G
+[OK]   pool: guest 9201 is a member of felhom
+[OK]   acl: FelhomAgentBase@/ present (user+token)
+[OK]   acl: FelhomAgentGuest@/pool/felhom present (user+token)
+[OK]   acl: FelhomAgentStore@/storage/local present (user+token)
+[OK]   acl: FelhomAgentStore@/storage/local-lvm present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990000 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990001 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990002 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990003 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990004 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990005 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990006 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990007 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990008 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990009 present (user+token)
+[OK]   authz signers: 2 (operator-signed self-update armed)
+[OK]   controller: Up 4 seconds (health: starting) (after ~0s)
+[INFO]   controller image: gitea.dooplex.hu/admin/felhom-controller:0.345.0
+[WARN]   cloudflared not visible yet
+[INFO]   (confirm in the hub UI that host Tester4-b5dbbd reports guest 9201)
+
+[OK] Day-0 provision SUCCESS — vmid=9201 host_id=Tester4-b5dbbd customer=Tester4 golden=local:backup/vzdump-lxc-9100-2026_10_26-21_00_49.tar.zst
+INSTALL_RC=0
+Connection to 192.168.0.127 closed.

documentation/audits/night-fixes-2026-10-26/r1232/install-start.txt

@@ -0,0 +1 @@
+19:59:44

documentation/audits/night-fixes-2026-10-26/r1232/teardown.txt

@@ -0,0 +1,4 @@
+purging VM 9375 from related configurations..
+Configuration file 'nodes/demo-hp/qemu-server/9375.conf' does not exist
+ls: cannot access '/mnt/pve/nvme-scratch/images/9375': No such file or directory
+felhom-installer-1.30.0-pve9.2-1.iso

documentation/audits/night-fixes-2026-10-26/r1232/uninstall-console-1-wrong-confirmation.txt

@@ -0,0 +1,26 @@
+
+[INFO] felhom-host-install v1.36.1 — mode=uninstall
+
+[STEP] UNINSTALL — local host teardown
+[INFO]   no --vmid given; using recorded provisioned_vmid=9201 from /var/lib/felhom-install/state.json
+[INFO]   vmid 9201 looks like a Felhom guest (has the /etc/felhom-bootstrap mount)
+[INFO]   pct config 9201:
+    arch: amd64
+    cores: 2
+    features: nesting=1,keyctl=1
+    hookscript: local:snippets/felhom-guest-hook.sh
+    hostname: tester4
+    memory: 4096
+    mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G
+    mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
+    mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
+    net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:06:5B:AC,ip=dhcp,type=veth
+    onboot: 1
+    ostype: debian
+    rootfs: local-lvm:vm-9201-disk-0,size=32G
+    swap: 4096
+    unprivileged: 1
+Type the vmid (9201) to confirm PERMANENT destruction: r1232-byo
+[ERROR] confirmation mismatch (got 'r1232-byo', expected '9201') — aborting, nothing destroyed
+UNINSTALL_RC=1
+Connection to 192.168.0.127 closed.

documentation/audits/night-fixes-2026-10-26/r1232/uninstall-console.txt

@@ -0,0 +1,84 @@
+
+[INFO] felhom-host-install v1.36.1 — mode=uninstall
+
+[STEP] UNINSTALL — local host teardown
+[INFO]   no --vmid given; using recorded provisioned_vmid=9201 from /var/lib/felhom-install/state.json
+[INFO]   vmid 9201 looks like a Felhom guest (has the /etc/felhom-bootstrap mount)
+[INFO]   pct config 9201:
+    arch: amd64
+    cores: 2
+    features: nesting=1,keyctl=1
+    hookscript: local:snippets/felhom-guest-hook.sh
+    hostname: tester4
+    memory: 4096
+    mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G
+    mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
+    mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
+    net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:06:5B:AC,ip=dhcp,type=veth
+    onboot: 1
+    ostype: debian
+    rootfs: local-lvm:vm-9201-disk-0,size=32G
+    swap: 4096
+    unprivileged: 1
+Type the vmid (9201) to confirm PERMANENT destruction: 9201
+  Logical volume "vm-9201-disk-0" successfully removed.
+  Logical volume "vm-9201-disk-1" successfully removed.
+[OK]   guest 9201 destroyed
+[STEP] host-level removal
+Removed '/etc/systemd/system/multi-user.target.wants/felhom-agent.service'.
+[OK]   removed /etc/felhom-agent (the agent config and every copy of it)
+Removed '/etc/systemd/system/multi-user.target.wants/wg-quick@wg-felhom.service'.
+[OK]   WireGuard tunnel to the Felhom off-site endpoint is down (wg-quick@wg-felhom disabled, conf removed)
+Removed '/etc/systemd/system/timers.target.wants/felhom-mgmt-watchdog.timer'.
+The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=,
+Also=, or Alias= settings in the [Install] section, and DefaultInstance= for
+template units). This means they are not meant to be enabled or disabled using systemctl.
+ 
+Possible reasons for having these kinds of units are:
+• A unit may be statically enabled by being symlinked from another unit's
+  .wants/, .requires/, or .upholds/ directory.
+• A unit's purpose may be to act as a helper for some other unit which has
+  a requirement dependency on it.
+• A unit may be started when needed via activation (socket, path, timer,
+  D-Bus, udev, scripted systemctl call, ...).
+• In case of template units, the unit is meant to be enabled with some
+  instance name specified.
+Removed '/etc/systemd/system/pve-guests.service.wants/felhom-shared-parent.service'.
+Removed '/etc/systemd/system/multi-user.target.wants/felhom-shared-parent.service'.
+Generating grub configuration file ...
+Found linux image: /boot/vmlinuz-7.0.2-6-pve
+Found initrd image: /boot/initrd.img-7.0.2-6-pve
+Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
+Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
+Found memtest86+ 64bit image: /boot/memtest86+x64.bin
+Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
+Adding boot menu entry for UEFI Firmware Settings ...
+done
+[SKIP]   old broad role FelhomAgent already absent
+[SKIP]   golden vzdump left in place (pass --remove-golden to remove)
+
+[STEP] kept vs wiped — read before pulling drives or closing the customer out
+  WIPED (this run):
+    - guest 9201 (container + its OS/Docker/user-data volumes)
+    - the felhom-agent: binary, unit, sudoers (+ its copies), state dir, service user, and its config
+      directory with every copy of the config in it
+    - the WireGuard tunnel to the Felhom off-site endpoint (wg-quick@wg-felhom disabled, /etc/wireguard/wg-felhom.conf removed)
+    - self-update artifacts: guarded wrapper, A/B slots (.prev/.new.*), rollback unit, start-limit drop-in
+    - break-glass watchdog + OOB artifacts (where present); guest-hook snippet; dnsmasq snippets; the mkfs, pbs-apply,
+      backup-target-apply, os-apply and priv-apply wrappers; the crash guard; the config-bundle record
+    - pveum: the Felhom roles/user/token/scoped ACL; the emptied felhom pool
+    - the install state file
+  KEPT (lives on deliberately — remove/rotate these out-of-band if the customer is leaving):
+    - the enrolled drives + ALL data under /mnt/felhom-drives — unmounted only, NEVER wiped;
+      the drives are physically removable now.
+    - the PBS backups + this customer's namespace on the PBS side — delete there if wanted
+    - the hub host/customer record + report history (operator UI / DB)
+    - this host's WireGuard PEER on the hub and the off-site endpoint (its /32 and public key) — the
+      tunnel is down from this side; delete the peer in the hub (operator) if the customer is leaving
+    - the escrow blob in the hub, if one exists (operator UI)
+
+[OK] UNINSTALL complete — removed: guest 9201, the felhom-agent (unit/sudoers/binary/state/config+baks/user + selfupdate-artifacts/shared-parent/mkfs-wrapper/pbs-apply-wrapper/hook-snippet/dnsmasq-snippets), the pveum role/user/token/ACL, the felhom pool, and /var/lib/felhom-install/state.json.
+[INFO]   golden vzdump: left in place (--remove-golden to remove).
+[INFO]   NOTE: the 'sudo' package was left INSTALLED (a system package, never ours to remove). dnsmasq is removed only when the install-time record says Felhom installed it; a resolver that pre-dated Felhom is never touched. The host record still exists in the hub — remove it there if desired.
+UNINSTALL_RC=0
+Connection to 192.168.0.127 closed.

documentation/audits/night-fixes-2026-10-26/r1298/clear-holds.txt

@@ -0,0 +1,18 @@
+== before (19:49:23)
+chaoscrash	held since 2026-09-24T12:16:24Z
+chaosoom	held since 2026-09-24T11:48:01Z
+chaosoomb	held since 2026-09-24T12:39:51Z
+gokapi	held since 2026-09-24T10:37:48Z
+controller stopped 19:49:24
+  Check the app's data first — the undo copies are in its unit's db-dumps dir.
+  chaoscrash rc=0
+  Check the app's data first — the undo copies are in its unit's db-dumps dir.
+  chaosoom rc=0
+  Check the app's data first — the undo copies are in its unit's db-dumps dir.
+  chaosoomb rc=0
+[INFO] [settings] Settings saved
+  gokapi rc=0
+== after
+no restore holds in force
+[INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json
+controller started 19:49:26

documentation/audits/night-fixes-2026-10-26/r1298/privatebin-remove.txt

@@ -0,0 +1,2 @@
+  [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'privatebin', 'volumes_removed': ['privatebin_privatebin_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem tárolt saját adatot
+leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '', 'kept': '', 'subdomain': '404'}

documentation/audits/night-fixes-2026-10-26/tools/r1187_r1220_live.py

@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+"""r1187_r1220_live.py — R-1220 and R-1187 (a)(b) live on scratch guest 9202, through the endpoints the UI invokes.
+
+A THROWAWAY Grimmory (drive appdata binds). Steps:
+  1. install; a marker folder in its drive appdata: an owned file (1000:1001, 0640), a link inside, a link outside;
+  2. R-1220: its app.yaml made unreadable (a byte copy saved first) → after a stack rescan the „Megőrzött adatok" page does
+     NOT list its folder; the app.yaml put back byte-identical;
+  3. R-1187 (a): a FIFO in the folder → the export FAILS naming the FIFO's path; the FIFO removed;
+  4. R-1187 (a)(b): export → import over the running app → the owned file keeps owner and mode, the inside link comes
+     back, the outside link is refused (listed in the controller WARN);
+  5. teardown: the app removed through the product (data and backups), the bundles this run made deleted.
+
+usage: SC=<0600 dir with .ctlpw> SWEEP_EV=<evidence dir> TARGET=9202 DRIVE_ROOT=/mnt/felhom-drives/scratch_hdd python3 -u r1187_r1220_live.py
+"""
+import os
+import signal
+import sys
+import time
+
+signal.signal(signal.SIGTERM, lambda *a: (_ for _ in ()).throw(SystemExit("SIGTERM — cleaning up")))
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools"))
+import sweep as sw  # noqa: E402
+
+w = sw.w
+APP, SUB, DRIVE = "grimmory", "grimd53", os.environ.get("DRIVE_ROOT", "/mnt/felhom-drives/scratch_hdd")
+say = print
+
+
+def g(cmd):
+    return sw.g(cmd).strip()
+
+
+def job(kind):
+    path = "/api/export/status" if kind == "export" else "/api/export/import/status"
+    for _ in range(180):
+        time.sleep(5)
+        code, d = w.ctl("GET", path)
+        j = d.get("data", d) if isinstance(d, dict) else {}
+        if j.get("done") or j.get("Done"):
+            return j
+    return {"timeout": True}
+
+
+def scan_after(since):
+    """Wait for a stack rescan that started after `since` (the controller rescans on its own every ~2 min)."""
+    for _ in range(40):
+        time.sleep(10)
+        if g(f"docker logs --since {since} felhom-controller 2>&1 | grep -c 'ScanStacks complete'") not in ("", "0"):
+            return True
+    return False
+
+
+def main():
+    w.say = say
+    w.login()
+    try:
+        ok = w.deploy(APP, SUB, {"HDD_PATH": DRIVE}) and w.wait_app(SUB, "/", tries=90, delay=5)
+        say("1. grimmory installed and answering:", ok)
+        appdata = g(f"ls -d {DRIVE}/appdata/grimmory 2>/dev/null | head -1")
+        say("   its appdata folder:", appdata or "(not found)")
+        if not appdata:
+            raise SystemExit("INSTRUMENT: no appdata folder — no verdict")
+        m = f"{appdata}/data/cc-r1187"  # INSIDE the bound folder (appdata/grimmory/data) — the export packs only binds
+        say("   markers:", g(f"mkdir -p {m} && printf hi > {m}/owned.txt && chown 1000:1001 {m}/owned.txt && chmod 0640 {m}/owned.txt"
+                             f" && ln -sfn owned.txt {m}/link-in && ln -sfn /etc/hostname {m}/link-out && ls -ln {m} | tail -n +2"))
+
+        # 2. R-1220 (SKIP_R1220=1: already proven in the first run, `r1187-r1220-9202-run1.txt`)
+        if os.environ.get("SKIP_R1220"):
+            say("2. R-1220 skipped (proven in the first run)")
+        else:
+            stack = f"/opt/docker/stacks/{APP}"
+            g(f"cp -p {stack}/app.yaml /root/r1220-app.yaml.save && printf 'deployed: true\\nenv: [broken: yaml: here\\n' > {stack}/app.yaml")
+            t = int(time.time())
+            say("2. app.yaml made unreadable; rescan seen:", scan_after(t))
+            page = w.page("/kept-data")
+            say("   R-1220: the kept-data page lists grimmory's folder:", appdata in page, "(want False) | WARN lines:",
+                g(f"docker logs --since {t} felhom-controller 2>&1 | grep -c 'R-1220'"))
+            g(f"cp -p /root/r1220-app.yaml.save {stack}/app.yaml && rm -f /root/r1220-app.yaml.save")
+            say("   app.yaml put back, byte check:", g(f"sha256sum {stack}/app.yaml | cut -c1-16"))
+            scan_after(int(time.time()))
+
+        # 3. R-1187 (a) FIFO
+        g(f"mkfifo {m}/pipe")
+        code, d = w.ctl("POST", "/api/export/start", {"stack_name": APP, "dest_drive": DRIVE, "stop_app": False})
+        say("3. export with a FIFO ->", code, str(d)[:160])
+        if code != "200":
+            raise SystemExit("INSTRUMENT: the export start was refused — no verdict")
+        j = job("export")
+        say("   job:", str(j.get("error") or j.get("Error") or j)[:300])
+        g(f"rm -f {m}/pipe")
+        if j.get("timeout"):
+            raise SystemExit("INSTRUMENT: the export job never ended — no verdict")
+
+        # 4. R-1187 (a)(b) round trip
+        code, d = w.ctl("POST", "/api/export/start", {"stack_name": APP, "dest_drive": DRIVE, "stop_app": False})
+        say("4. export ->", code, str(d)[:120])
+        if code != "200":
+            raise SystemExit("INSTRUMENT: the export start was refused — no verdict")
+        j = job("export")
+        say("   job:", str(j)[:300])
+        fab = g(f"ls -t {DRIVE}/exports/grimmory*.fab 2>/dev/null | head -1")
+        say("   bundle:", fab)
+        say("   tar listing of the drive part:", g(f"mkdir -p /root/r1187x && cd /root/r1187x && tar -xzf {fab} data/hdd 2>/dev/null; "
+                                                   f"for f in data/hdd/*.tar; do tar -tvf $f | grep cc-r1187; done; rm -rf /root/r1187x"))
+        g(f"rm -rf {m}")  # the import must bring it back
+        ti = int(time.time())
+        code, d = w.ctl("POST", "/api/export/import", {"path": fab})
+        say("   import ->", code, str(d)[:160])
+        if code != "200":
+            raise SystemExit("INSTRUMENT: the import start was refused — no verdict")
+        j = job("import")
+        say("   import job:", str(j)[:300])
+        say("   after import:", g(f"ls -ln {m} | tail -n +2; stat -c '%n %u:%g %a' {m}/owned.txt; readlink {m}/link-in; "
+                                  f"ls -l {m}/link-out 2>&1 | head -1"))
+        say("   the import's WARN:", g(f"docker logs --since {ti} felhom-controller 2>&1 | grep 'R-1187 b' | cut -c1-220"))
+        say("   the app answers:", w.wait_app(SUB, "/", tries=60, delay=5))
+    finally:
+        sw.remove(APP, keep=False, say=say)
+        say("5. leftovers:", sw.leftovers(APP, SUB))
+        say("   bundles of this run removed:", g(f"ls {DRIVE}/exports/grimmory*.fab 2>/dev/null; rm -f {DRIVE}/exports/grimmory*.fab; ls {DRIVE}/exports/ 2>/dev/null | grep -c grimmory"))
+
+
+if __name__ == "__main__":
+    main()

documentation/audits/night-fixes-2026-10-26/tools/r1187g_live.py

@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+"""r1187g_live.py — R-1187 (g) live on scratch guest 9202: a PostGIS app's database survives a `.fab` export → import.
+
+A THROWAWAY AdventureLog (postgis/postgis). A superuser is made with the app's own `createsuperuser`; the app is exported
+and imported through the endpoints the UI invokes; then the user is read back through the app's FRONT DOOR — the
+Django admin login page — and through its own ORM. Before the fix the import did not recognise postgis as a database,
+so the dump was never loaded. Teardown through the product. The throwaway password lives only in this process.
+
+usage: SC=<0600 dir with .ctlpw> SWEEP_EV=<evidence dir> TARGET=9202 DRIVE_ROOT=/mnt/felhom-drives/scratch_hdd python3 -u r1187g_live.py
+"""
+import os
+import re
+import secrets
+import signal
+import urllib.parse
+import sys
+import time
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools"))
+import sweep as sw  # noqa: E402
+
+signal.signal(signal.SIGTERM, lambda *a: (_ for _ in ()).throw(SystemExit("SIGTERM — cleaning up")))
+w = sw.w
+APP, SUB, DRIVE = "adventurelog", "advd53", os.environ.get("DRIVE_ROOT", "/mnt/felhom-drives/scratch_hdd")
+say = print
+
+
+def g(cmd):
+    return sw.g(cmd).strip()
+
+
+def job(path):
+    for _ in range(180):
+        time.sleep(5)
+        code, d = w.ctl("GET", path)
+        if isinstance(d, dict) and d.get("done"):
+            return d
+    return {"timeout": True}
+
+
+def orm_has(user):
+    out = g(f"docker exec adventurelog python manage.py shell -c "
+            f"\"from django.contrib.auth import get_user_model; print('ANS=' + str(get_user_model().objects.filter(username='{user}').exists()))\" 2>&1")
+    m = re.search(r"ANS=(True|False)", out)
+    return m.group(1) if m else "unreadable: " + out[-160:]
+
+
+def admin_login(user, pw):
+    """The front door, the household's route (box_walk.app_curl carries the setup gate): GET /admin/login/ for the
+    form and its csrftoken cookie, POST the form (the password on stdin, never an argument). Success = 302 to /admin/."""
+    rc, code, body = w.app_curl(SUB, "/admin/login/", "-D", "-")
+    tok = re.search(r'name="csrfmiddlewaretoken" value="([^"]+)"', body)
+    ck = re.search(r"(?im)^set-cookie:\s*csrftoken=([^;]+)", body)
+    if not tok or not ck:
+        return f"no login form ({code})"
+    form = urllib.parse.urlencode({"csrfmiddlewaretoken": tok.group(1), "username": user, "password": pw, "next": "/admin/"})
+    rc, code, body = w.app_curl(SUB, "/admin/login/?next=/admin/", "-H", f"Cookie: csrftoken={ck.group(1)}",
+                                "-H", "Content-Type: application/x-www-form-urlencoded",
+                                "-H", f"Referer: https://{SUB}.{w.DOMAIN}/admin/login/", method="POST", data=form)
+    return code + (" (logged in)" if code == "302" else " (refused)")
+
+
+def main():
+    w.say = say
+    w.login()
+    user, pw = "cc" + secrets.token_hex(3), "Cc-" + secrets.token_hex(12)
+    try:
+        ok = w.deploy(APP, SUB) and w.wait_app(SUB, "/", tries=120, delay=5)
+        say("1. adventurelog installed and answering:", ok)
+        out = g(f"docker exec -e DJANGO_SUPERUSER_PASSWORD='{pw}' adventurelog python manage.py createsuperuser --noinput "
+                f"--username {user} --email {user}@cc.invalid 2>&1 | tail -1")
+        say("2. createsuperuser:", out[:120], "| in the ORM:", orm_has(user))
+        say("   front door before:", admin_login(user, pw))
+        code, d = w.ctl("POST", "/api/export/start", {"stack_name": APP, "dest_drive": DRIVE, "stop_app": False})
+        if code != "200":
+            raise SystemExit(f"INSTRUMENT: export start refused {code} {d}")
+        j = job("/api/export/status")
+        say("3. export:", j.get("error") or "ok", j.get("output_size"), j.get("output_path"))
+        fab = j.get("output_path", "")
+        say("   the bundle's database:", g(f"tar -tzf {fab} | grep -E 'database/|manifest' | head -4"))
+        g("docker exec adventurelog python manage.py shell -c \"from django.contrib.auth import get_user_model; "
+          f"get_user_model().objects.filter(username='{user}').delete()\" >/dev/null 2>&1")
+        say("4. the user deleted in the live app (so only the import can bring it back) — in the ORM:", orm_has(user))
+        code, d = w.ctl("POST", "/api/export/import", {"path": fab})
+        if code != "200":
+            raise SystemExit(f"INSTRUMENT: import start refused {code} {d}")
+        j = job("/api/export/import/status")
+        say("5. import:", j.get("error") or "ok", "| steps:", [(s.get("label"), s.get("status")) for s in j.get("steps", [])])
+        say("   app answers:", w.wait_app(SUB, "/", tries=120, delay=5))
+        say("   R-1187 g — the user back in the ORM:", orm_has(user))
+        say("   R-1187 g — the front door after import:", admin_login(user, pw))
+        say("   the import's DB line:", g("docker logs --since 15m felhom-controller 2>&1 | grep -E 'restoring database|found DB service|DB dump imported|psql' | tail -3 | cut -c1-200"))
+    finally:
+        sw.remove(APP, keep=False, say=say)
+        say("6. leftovers:", sw.leftovers(APP, SUB))
+        say("   bundles removed:", g(f"rm -f {DRIVE}/exports/adventurelog*.fab; ls {DRIVE}/exports 2>/dev/null | grep -c adventurelog"))
+
+
+if __name__ == "__main__":
+    main()