night fixes 2026-10-26: live proofs (R-1220, R-1232, R-1187, R-1298), register states, design lines; R-1300 filed; R-528 design
gates / gates (push) Successful in 1m56s
gates / gates (push) Successful in 1m56s
Evidence audits/night-fixes-2026-10-26/ (scanned: 0 secret hits).
This commit is contained in:
1 parent
cc0fb962f3
commit
ae3db89ab3
24 files changed
+837
-14
No files matched your search
+11
@@ -16,6 +16,17 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-26 night (alone) — the data-safety fixes, on `main`, NOT released.** Controller: R-1220 `7e15a37`
|
||||
> (`liveDriveBinds(drives)`: an unreadable app.yaml = installed), R-1218/R-1219 `1170b48` (stage when the password FILE
|
||||
> exists; no ceremony during a recovery offer), R-1187 `dfd0f8e`…`90a2530` (.fab: links as links, FIFO refused, `os.Root`
|
||||
> writes, downward-only links, owners/modes, busy and bad-path refusals; the import's compose gets `USERDATA_PATH`/
|
||||
> `IMPORT_PATH`; the DB loads through `appbackup.ImportDump` — the import's `mysqladmin`/`mysql` broke every MariaDB 11+
|
||||
> import), R-1298 `b156e45`, Go 1.27.3 `2560cf1`. Agent: R-1232 `3911cc6` (own-key archives only in the due-check),
|
||||
> Go 1.27.3 `fbbb168`. Hub: R-1218 `09b5f02f` (409 on a key-less package over a keyed one), Go 1.27.3 `d625a143`.
|
||||
> Live: `audits/night-fixes-2026-10-26/` (R-1220, R-1187, R-1298 on 9202 with local test images; R-1232 on a BYO Tester4
|
||||
> VM, uninstalled and destroyed). R-1218's live shape needs the operator (off-site off on Tester 1) — else reverted
|
||||
> before the release. R-528: design in the row, not built. New: R-1299, R-1302.
|
||||
|
||||
> **2026-10-26 (night) — D55 = 260 (`09` §3 decision 268); the grant is SET.** CC used the operator's one-purpose token
|
||||
> `HUB_TOKEN_D55_WINDOW` („cc-d55-window", Off-site write only) once: `POST /offsite/window-grant/demo-hp` `max_remove=260`
|
||||
> → 200, 19:10:31Z; read back `offsite_window_grant:demo-hp = max:260` (hub DB with WAL) and the hub log line „granted ONE
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
|
||||
|
||||
**Updated 2026-10-26 (night).** Your D55 token worked: demo-hp gets one clean-up window with a cap of 260 at its next nightly off-site run. Tonight CC fixes data-safety holes (saved only); Tuesday: the line test, then each release on your yes.
|
||||
**Updated 2026-10-26 (night).** Saved and proven tonight, not released: kept data never offers a live app's folder; a reinstalled box backs up off-site at once; app-file import keeps links and owners and works again for MariaDB apps; Go 1.27.3. D55's window runs tonight.
|
||||
|
||||
## What works
|
||||
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
|
||||
@@ -23,8 +23,8 @@
|
||||
- DooPlex: something sets the shared checkout to „bare" (R-935).
|
||||
|
||||
## What we work on
|
||||
- Released: 0.345.1 (cloudflared 2026.10.3, security), every box; `main` and Sunday's golden carry it. Built, not released: a BYO box removes our SSH file in the same update (R-1285; the agent waits for a live BYO proof); held apps reach you (R-1291, controller + hub, proven on a test hub).
|
||||
- Dated checks, run by CC. 2026-10-27: the qBittorrent line test on a seeding box, with DooPlex's qBittorrent as the peer (R-1250, your yes). Done 2026-10-26: the weekly clean-up windows removed only old copies and raised no false alarm. 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes).
|
||||
- Released: 0.345.1 (cloudflared 2026.10.3, security), every box. Built, not released (Tuesday's block, each on your yes): R-1291, R-1220, R-1232, R-1187, R-1219, R-1298, R-972, R-1285. R-1218 needs you: off-site off on Tester 1 for minutes (hub customer page), or it leaves the release.
|
||||
- Dated checks, run by CC. 2026-10-27: the line test on a seeding box, DooPlex's qBittorrent as the peer (R-1250, your yes). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes).
|
||||
|
||||
## Waiting on you — decide
|
||||
|
||||
@@ -48,7 +48,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14.
|
||||
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
|
||||
|
||||
**2026-10-26.** On the hub's Hosts page, delete `Tester4-b5ec39` (CC's test VM, destroyed). If not: the hub alarms that the box went silent. After demo-hp's window has run (CC says when, Tuesday morning), revoke the token `cc-d55-window` (R-1294).
|
||||
**2026-10-26.** On the hub's Hosts page, delete `Tester4-b5dbbd` (CC's test VM of tonight, uninstalled and destroyed; you already deleted `Tester4-b5ec39`). If not: the hub alarms that the box went silent. After demo-hp's window has run (CC says when, Tuesday morning), revoke the token `cc-d55-window` (R-1294).
|
||||
**2026-10-23 do list.**
|
||||
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
|
||||
2. On the hub's Hosts page: Discard appliance 49. If not: a dead entry alarms.
|
||||
|
||||
@@ -510,7 +510,7 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
|
||||
backup is due (`GET /backup/due`, §6) → **quiesces** (stops its app stacks) → `POST /backup` →
|
||||
polls `GET /backup/status` to `done` → **unquiesces** (restarts exactly the stacks it stopped).
|
||||
Implemented in `felhom-controller` v0.36.0 (`internal/quiesce`) + `felhom-agent` v0.11.0 (the
|
||||
`/backup/due` cadence policy + `/backup/status` phases). **An agent-initiated vzdump is
|
||||
`/backup/due` cadence policy + `/backup/status` phases). *[DESIGN] R-1232 (2026-10-26 night, agent main `3911cc6`, next agent release): on an encrypted tier the due-check counts only archives written with THIS storage's key (the R-727 discriminator — an archive carries no host id), so a reinstalled box does not read the old box's archive as its own last backup.* **An agent-initiated vzdump is
|
||||
crash-consistent only** (there is no inbound-to-guest channel to trigger a quiesce — §3/§5); the
|
||||
controller stopping its stacks first is what makes the captured state **clean-shutdown-consistent**
|
||||
(validated live: a quiesced postgres restore comes up clean — "database system was shut down" — vs
|
||||
|
||||
@@ -300,7 +300,7 @@ What remains:
|
||||
**`host_escrow.blob`** (and the identity blob beside it).
|
||||
**[DESIGN] 2026-10-24 (R-1254; committed felhom.eu `dcf4f84c`, hub next v0.170.0 — NOT deployed):** the escrow PUT
|
||||
stores the blob, its sealed-password hash and the identity blob in ONE transaction (`Store.SaveHostEscrowWithIdentity`;
|
||||
the superseded copy is made inside it). A failure leaves all of the old package or all of the new — never the new hash
|
||||
the superseded copy is made inside it). *[DESIGN] R-1218 (2026-10-26 night, hub main `09b5f02f`, next hub release): a package that seals the off-site repository password is never replaced by one that seals none — the PUT answers 409 and the good package stays current (it used to be demoted to operator-only custody, so the household's code no longer opened its off-site copy).* A failure leaves all of the old package or all of the new — never the new hash
|
||||
beside the old identity package, which a box would read as „escrowed" while the written code opened nothing. A
|
||||
malformed identity blob is refused before anything is stored. (The agent ceremony's „the previous code still works"
|
||||
after a lost answer, R-1254's other half, stays open.)
|
||||
|
||||
@@ -403,6 +403,8 @@ never moved by an update or an undo, and the unit restore accepts it.
|
||||
> copy protected the app's files. Since v0.245.0 every dashboard page carries the reminder (the R-241
|
||||
> bar, second instance) and the Tier-1 sentence renders by state — „védené … szünetel" while paused.
|
||||
> Measured on a fresh box 2026-09-16: zero snapshots, and the page said the files were protected.
|
||||
>
|
||||
> *[DESIGN] R-1219 / R-1218 (2026-10-26 night, controller main `1170b48`, next release): during a recovery offer the bar does NOT urge the ceremony (a new code would replace the key that opens the old off-site copy) and the start answers 409; a ceremony stages the off-site password whenever its FILE is on the box, also with off-site switched off or cleared.*
|
||||
|
||||
> **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does.
|
||||
|
||||
@@ -923,6 +925,8 @@ run makes it longer.
|
||||
|
||||
*Second-drive copies that are KEPT (D25) and the rule that a rejected backup block never deletes one: §8.2a (R-1252, 2026-10-24).*
|
||||
|
||||
*[DESIGN] R-1220 (2026-10-26 night, controller main `7e15a37`, ships with the next controller release): an app whose `app.yaml` EXISTS but does not parse counts as INSTALLED — every folder its compose binds, on every registered drive, is live and never listed as kept data (fail closed; one WARN names the app).*
|
||||
|
||||
**What it is.** "Remove the app, keep my data" leaves the app's drive folder (`<drive>/appdata/<app>`, or the folder
|
||||
its definition binds through `${HDD_PATH}`) in place. A reinstall over it no longer runs silently into the old files
|
||||
(R-657): the install asks „A megőrzött adataimat használom" / "Use my kept data" (the database from the newest copy of
|
||||
@@ -1080,6 +1084,8 @@ second drive's whole restore (its `CheckOnly` pass carries the pull, so its file
|
||||
load from a unit and from off-site, and a set-aside unit (D47) — the off-site restore, the second drive's file restore
|
||||
and the no-unit fallback. Not covered: the `.fab` import — it removes the running app's volumes before its definition is pulled (R-1279).
|
||||
|
||||
*[DESIGN] The `.fab` route, R-1187 (2026-10-26 night, controller main, next release): the export carries a symlink AS A LINK and stops on a FIFO, socket or device naming its path; a drive folder that cannot be packed fails the export (never left out). The import writes only through `os.Root`, keeps owners and modes, restores a link only when it points DOWNWARD inside the folder (no absolute target, no „..”, no link on the way — others refused and listed), refuses while another operation holds the box and refuses an empty or relative `HDD_PATH`; its compose runs get the controller's own `USERDATA_PATH`/`IMPORT_PATH`; the database loads through the backup side (`appbackup.ImportDump`: mariadb / mariadb-admin, psql ON_ERROR_STOP) — the import's own `mysqladmin`/`mysql` calls failed on every MariaDB 11+ image. Live: `audits/night-fixes-2026-10-26/`.*
|
||||
|
||||
### 6.7 Why an off-site run failed — the failure classifier (R-571)
|
||||
|
||||
**[FACT, read from source 2026-10-05, felhom-controller `114ff27`]** When an off-site (restic) run fails, the
|
||||
|
||||
@@ -660,3 +660,5 @@ a row the box stopped repeating for 50 h (the list still shows it, marked). `ale
|
||||
every grace — a test configuration only. Live proof: `audits/d53-held-apps-2026-10-26/` (a crash-looped throwaway app on
|
||||
9202 against a test hub with a 3-minute grace: listed on both pages, one operator mail, Start cleared it, the row went,
|
||||
every household leg `skipped operator_only`).
|
||||
|
||||
*[DESIGN] R-1298 (2026-10-26 night, controller main `b156e45`): an install's hold (`unhealthy_stop`, `update_failed`, `putback_pending`) of an app that is NOT installed (no `app.yaml`) is not reported — it would remind weekly about an app that does not exist; a restore hold is always reported (R-379, a person clears it).*
|
||||
@@ -0,0 +1,67 @@
|
||||
# Night fixes 2026-10-26 → 27: live proofs (R-1220, R-1232, R-1187, R-1298)
|
||||
|
||||
Code on `main`, NOT released (the release block is Tuesday afternoon, attended). Test images were **local only**
|
||||
(`0.346.0-night1/2/3`, `docker save`/`load` into 9202 — never pushed to the registry); the test agent
|
||||
(`0.167.0-night1`) was a local build copied to the test VM only. Method: the endpoints the UI invokes, through the box
|
||||
walk tools (`../catalog-sweep-2026-10-24/tools/`), and the product's own installer / uninstaller.
|
||||
|
||||
## R-1220 — an unreadable `app.yaml` counts as installed (controller `7e15a37`)
|
||||
|
||||
9202, throwaway Grimmory, its `app.yaml` made unreadable (byte copy saved, put back): after a stack rescan the
|
||||
„Megőrzött adatok" page did **not** list its drive folder, and the controller logged the R-1220 WARN.
|
||||
`r1187-r1220-9202-run1.txt` (step 2). PASS.
|
||||
|
||||
## R-1232 — a reinstalled box's first whole-box copy is not a week late (agent `3911cc6`)
|
||||
|
||||
VM 9375 on demo-hp's scratch NVMe: plain Proxmox (auto-install ISO with DooPlex's key), then the product's BYO installer
|
||||
as **Tester4** (host `Tester4-b5dbbd`). Tester4's ep0 space held one archive of an EARLIER box (`ct/9201`, 2026-10-20,
|
||||
key `23:58:…`); this box's key is `7c:3b:…`.
|
||||
|
||||
| | Agent | `GET /backup/due?target=felhom-pbs` |
|
||||
|---|---|---|
|
||||
| before | 0.166.0 (released) | `due: false`, „within cadence window", age 546,807 s — the foreign archive counted |
|
||||
| after | 0.167.0-night1 (`main`) | `due: true`, „no successful backup recorded yet", `age_state: absent` |
|
||||
|
||||
The controller's first-backup valve then started the first whole-box copy at 20:13:55Z; it completed at 21:15:17 CET
|
||||
(`ct/9201/2026-10-26T20:13:55Z`, this box's key) — 4 minutes after the swap. `r1232/1-…`, `2-…`, `3-first-run.txt`.
|
||||
PASS. **Teardown:** the product's uninstall ran (`r1232/uninstall-console.txt`, `UNINSTALL_RC=0`; the first try stopped
|
||||
at the confirmation prompt — it asks for the guest number, nothing destroyed), then VM 9375 destroyed, its ISO and work
|
||||
files removed. **The operator deletes hub host `Tester4-b5dbbd`.** Tester4's ep0 space keeps this box's one archive
|
||||
(a host delete keeps the tenancy, by design).
|
||||
|
||||
## R-1187 — the `.fab` export and import (controller `dfd0f8e` … `90a2530`)
|
||||
|
||||
9202, throwaway Grimmory (markers inside its bound drive folder `appdata/grimmory/data/cc-r1187`: a file owned
|
||||
1000:1001 mode 0640, a link inside, a link to `/etc/hostname`):
|
||||
|
||||
- (a) a FIFO in the folder → the export **fails**, naming the FIFO's path (`r1187-9202-run2.txt`, `-run3.txt`, step 3);
|
||||
without it the export carries both links AS links (the tar listing, step 4).
|
||||
- (b) after the import: the file is back as 1000:1001 mode 640, the inside link is back, the outside link is **refused**
|
||||
and listed in the import's WARN (run 2 and run 3).
|
||||
- **Two import defects found by this proof, older than tonight, fixed and re-proven:** run 1 and run 2 ended
|
||||
„waiting for DB: timeout" and left Grimmory down: (1) the import's compose runs lacked the controller's own
|
||||
`USERDATA_PATH`/`IMPORT_PATH`, and compose's warning was read as the DB container id (`a3ed63e`); (2) the import used
|
||||
`mysqladmin`/`mysql`, which the MariaDB 11+ images do not ship, so EVERY MariaDB 11+ import timed out (`90a2530`: the
|
||||
import now loads through `appbackup.ImportDump`). **Run 3 on `0.346.0-night3`: the import succeeds and the app
|
||||
answers.** `r1187-9202-run3.txt`.
|
||||
- (g) AdventureLog (postgis): a superuser made with the app's own `createsuperuser`, export, the user deleted in the live
|
||||
app, import → the user is back (read through the app's own ORM, `manage.py shell`). My front-door check (the Django
|
||||
admin login page) answered 302 with no form both BEFORE and after — an instrument that did not reach the form, not a
|
||||
result. `r1187g-9202.txt`. PASS on the ORM readback.
|
||||
- (d) and (f): their unit tests are the proof (a running restore cannot be staged safely; a crafted bundle would be a
|
||||
hand-made input).
|
||||
|
||||
## R-1298 — held-app reports only for installed apps (controller `b156e45`)
|
||||
|
||||
9202 on `0.346.0-night1`: the reporter skipped its four orphan holds by name („not reported (R-1298)", 0 sent). Then,
|
||||
the controller stopped, `-clear-restore-hold` for `chaoscrash`, `chaosoom`, `chaosoomb`, `gokapi` → „no restore holds in
|
||||
force"; PrivateBin's leftover removed through the product (volume, `app.yaml`, backup gone). `r1298/`.
|
||||
|
||||
## Teardown
|
||||
|
||||
- **9202:** back on 0.343.0 (image file restored, the test images removed); its standing apps only; no holds (the four
|
||||
orphans cleared on purpose, per the brief); the drive's `appdata/` holds only paperless; no bundles in `exports/`.
|
||||
- **demo-hp:** VM 9375 destroyed (own id), its ISO removed; nothing else provisioned.
|
||||
- **Hub:** host `Tester4-b5dbbd` — the operator's delete. **DooPlex:** the local test images removed.
|
||||
- Secrets: the evidence was scanned (the dashboard passwords, Tester4's passphrase, both hub tokens, the Resend key):
|
||||
0 hits.
|
||||
@@ -0,0 +1,33 @@
|
||||
[1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH']
|
||||
[1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
[1] deployed, controller state=running, pinned={'grimmory': 'ghcr.io/grimmory-tools/grimmory:v3.5.0', 'grimmory-db': 'mariadb:11.4'}
|
||||
gate: grimd53 is gated — passed as the household (cookie set)
|
||||
1. grimmory installed and answering: True
|
||||
its appdata folder: /mnt/felhom-drives/scratch_hdd/appdata/grimmory
|
||||
markers: lrwxrwxrwx 1 0 0 9 Oct 26 20:46 link-in -> owned.txt
|
||||
lrwxrwxrwx 1 0 0 13 Oct 26 20:46 link-out -> /etc/hostname
|
||||
-rw-r----- 1 1000 1001 2 Oct 26 20:46 owned.txt
|
||||
2. R-1220 skipped (proven in the first run)
|
||||
3. export with a FIFO -> 200 {'ok': True}
|
||||
job: Felhasználói adatok mentése sikertelen: a(z) /mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/pipe különleges fájl (cső, foglalat vagy eszköz), ezt nem lehet menteni — az exportálás leállt, semmi nem maradt ki csendben
|
||||
4. export -> 200 {'ok': True}
|
||||
job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'export', 'ok': True, 'output_path': '/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-204714.fab', 'output_size': '3.2 MB', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Előkészítés', 'status': 'done'}, {
|
||||
bundle: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-204714.fab
|
||||
tar listing of the drive part: drwxr-xr-x root/root 0 2026-10-26 20:47 cc-r1187
|
||||
lrwxrwxrwx root/root 0 2026-10-26 20:46 cc-r1187/link-in -> owned.txt
|
||||
lrwxrwxrwx root/root 0 2026-10-26 20:46 cc-r1187/link-out -> /etc/hostname
|
||||
-rw-r----- 1000/1001 2 2026-10-26 20:46 cc-r1187/owned.txt
|
||||
import -> 200 {'ok': True}
|
||||
import job: {'display_name': 'Grimmory', 'done': True, 'error': 'Adatbázis visszaállítása sikertelen: waiting for DB: timeout waiting for mariadb to become ready', 'job_type': 'import', 'ok': True, 'output_path': '', 'output_size': '', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Csomag megn
|
||||
after import: lrwxrwxrwx 1 0 0 9 Oct 26 20:47 link-in -> owned.txt
|
||||
-rw-r----- 1 1000 1001 2 Oct 26 20:47 owned.txt
|
||||
/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/owned.txt 1000:1001 640
|
||||
owned.txt
|
||||
ls: cannot access '/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/link-out': No such file or directory
|
||||
the import's WARN: 2026/10/26 20:47:31 restore.go:695: [WARN] Import: data — 1 link(s) pointing outside the app's folder were NOT restored (R-1187 b): cc-r1187/link-out
|
||||
app never answered on grimd53/ (last rc=0 code=404)
|
||||
the app answers: False
|
||||
[7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'grimmory', 'volumes_removed': ['grimmory_grimmory_db'], 'hdd_paths_removed': ['/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data (12K)'], 'hdd_paths_preserved': []
|
||||
5. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '/mnt/felhom-drives/scratch_hdd/appdata/grimmory', 'kept': '', 'subdomain': '404'}
|
||||
bundles of this run removed: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-204714.fab
|
||||
0
|
||||
@@ -0,0 +1,32 @@
|
||||
[1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH']
|
||||
[1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
[1] deployed, controller state=running, pinned={'grimmory': 'ghcr.io/grimmory-tools/grimmory:v3.5.0', 'grimmory-db': 'mariadb:11.4'}
|
||||
gate: grimd53 is gated — passed as the household (cookie set)
|
||||
1. grimmory installed and answering: True
|
||||
its appdata folder: /mnt/felhom-drives/scratch_hdd/appdata/grimmory
|
||||
markers: lrwxrwxrwx 1 0 0 9 Oct 26 21:01 link-in -> owned.txt
|
||||
lrwxrwxrwx 1 0 0 13 Oct 26 21:01 link-out -> /etc/hostname
|
||||
-rw-r----- 1 1000 1001 2 Oct 26 21:01 owned.txt
|
||||
2. R-1220 skipped (proven in the first run)
|
||||
3. export with a FIFO -> 200 {'ok': True}
|
||||
job: Felhasználói adatok mentése sikertelen: a(z) /mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/pipe különleges fájl (cső, foglalat vagy eszköz), ezt nem lehet menteni — az exportálás leállt, semmi nem maradt ki csendben
|
||||
4. export -> 200 {'ok': True}
|
||||
job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'export', 'ok': True, 'output_path': '/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-210142.fab', 'output_size': '3.2 MB', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Előkészítés', 'status': 'done'}, {
|
||||
bundle: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-210142.fab
|
||||
tar listing of the drive part: drwxr-xr-x root/root 0 2026-10-26 21:01 cc-r1187
|
||||
lrwxrwxrwx root/root 0 2026-10-26 21:01 cc-r1187/link-in -> owned.txt
|
||||
lrwxrwxrwx root/root 0 2026-10-26 21:01 cc-r1187/link-out -> /etc/hostname
|
||||
-rw-r----- 1000/1001 2 2026-10-26 21:01 cc-r1187/owned.txt
|
||||
import -> 200 {'ok': True}
|
||||
import job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'import', 'ok': True, 'output_path': '', 'output_size': '', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Csomag megnyitása', 'status': 'done'}, {'label': 'Alkalmazás előkészítése', 'status': 'done'}, {'label': 'K
|
||||
after import: lrwxrwxrwx 1 0 0 9 Oct 26 21:01 link-in -> owned.txt
|
||||
-rw-r----- 1 1000 1001 2 Oct 26 21:01 owned.txt
|
||||
/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/owned.txt 1000:1001 640
|
||||
owned.txt
|
||||
ls: cannot access '/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data/cc-r1187/link-out': No such file or directory
|
||||
the import's WARN: 2026/10/26 21:01:57 restore.go:694: [WARN] Import: data — 1 link(s) pointing outside the app's folder were NOT restored (R-1187 b): cc-r1187/link-out
|
||||
the app answers: True
|
||||
[7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'grimmory', 'volumes_removed': ['grimmory_grimmory_db'], 'hdd_paths_removed': ['/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data (12K)'], 'hdd_paths_preserved': []
|
||||
5. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '/mnt/felhom-drives/scratch_hdd/appdata/grimmory', 'kept': '', 'subdomain': '404'}
|
||||
bundles of this run removed: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-210142.fab
|
||||
0
|
||||
@@ -0,0 +1,29 @@
|
||||
[1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH']
|
||||
[1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
[1] deployed, controller state=running, pinned={'grimmory': 'ghcr.io/grimmory-tools/grimmory:v3.5.0', 'grimmory-db': 'mariadb:11.4'}
|
||||
gate: grimd53 is gated — passed as the household (cookie set)
|
||||
1. grimmory installed and answering: True
|
||||
its appdata folder: /mnt/felhom-drives/scratch_hdd/appdata/grimmory
|
||||
markers: lrwxrwxrwx 1 0 0 9 Oct 26 20:15 link-in -> owned.txt
|
||||
lrwxrwxrwx 1 0 0 13 Oct 26 20:15 link-out -> /etc/hostname
|
||||
-rw-r----- 1 1000 1001 2 Oct 26 20:15 owned.txt
|
||||
2. app.yaml made unreadable; rescan seen: True
|
||||
R-1220: the kept-data page lists grimmory's folder: False (want False) | WARN lines: 2
|
||||
app.yaml put back, byte check: 2b7b111d6bd71008
|
||||
3. export with a FIFO -> 200 {'ok': True}
|
||||
job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'export', 'ok': True, 'output_path': '/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201737.fab', 'output_size': '3.2 MB', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Előkészítés', 'status': 'done'}, {
|
||||
4. export -> 200 {'ok': True}
|
||||
job: {'display_name': 'Grimmory', 'done': True, 'error': '', 'job_type': 'export', 'ok': True, 'output_path': '/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201745.fab', 'output_size': '3.2 MB', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Előkészítés', 'status': 'done'}, {
|
||||
bundle: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201745.fab
|
||||
tar listing of the drive part:
|
||||
import -> 200 {'ok': True}
|
||||
import job: {'display_name': 'Grimmory', 'done': True, 'error': 'Adatbázis visszaállítása sikertelen: waiting for DB: timeout waiting for mariadb to become ready', 'job_type': 'import', 'ok': True, 'output_path': '', 'output_size': '', 'running': False, 'stack_name': 'grimmory', 'steps': [{'label': 'Csomag megn
|
||||
after import: ls: cannot access '/mnt/felhom-drives/scratch_hdd/appdata/grimmory/cc-r1187/link-out': No such file or directory
|
||||
the import's WARN:
|
||||
app never answered on grimd53/ (last rc=0 code=404)
|
||||
the app answers: False
|
||||
[7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'grimmory', 'volumes_removed': ['grimmory_grimmory_db'], 'hdd_paths_removed': ['/mnt/felhom-drives/scratch_hdd/appdata/grimmory/data (4.0K)'], 'hdd_paths_preserved': [
|
||||
5. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '/mnt/felhom-drives/scratch_hdd/appdata/grimmory', 'kept': '', 'subdomain': '404'}
|
||||
bundles of this run removed: /mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201737.fab
|
||||
/mnt/felhom-drives/scratch_hdd/exports/grimmory_20261026-201745.fab
|
||||
0
|
||||
@@ -0,0 +1,27 @@
|
||||
[1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
[1] deployed, controller state=unhealthy, pinned={'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.13.0', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine'}
|
||||
[1] NOTE: the controller's own state is 'unhealthy', not 'running' — recorded, not treated as a failure; the fixture's front-door wait is the real gate
|
||||
gate: advd53 not routed right now (404) — not cached, asked again next call
|
||||
gate: advd53 not routed right now (404) — not cached, asked again next call
|
||||
gate: advd53 not routed right now (404) — not cached, asked again next call
|
||||
gate: advd53 not routed right now (404) — not cached, asked again next call
|
||||
gate: advd53 not routed right now (404) — not cached, asked again next call
|
||||
gate: advd53 not routed right now (404) — not cached, asked again next call
|
||||
gate: advd53 is gated — passed as the household (cookie set)
|
||||
1. adventurelog installed and answering: True
|
||||
2. createsuperuser: Superuser created successfully. | in the ORM: True
|
||||
front door before: no login form (302)
|
||||
3. export: ok 48.2 MB /mnt/felhom-drives/scratch_hdd/exports/adventurelog_20261026-210649.fab
|
||||
the bundle's database: database/adventurelog-postgres.sql.gz
|
||||
manifest.json
|
||||
4. the user deleted in the live app (so only the import can bring it back) — in the ORM: False
|
||||
5. import: ok | steps: [('Csomag megnyitása', 'done'), ('Alkalmazás előkészítése', 'done'), ('Konfiguráció visszaállítása', 'done'), ('Felhasználói adatok visszaállítása', 'done'), ('Adatbázis visszaállítása', 'done'), ('Alkalmazás indítása', 'done')]
|
||||
app answers: True
|
||||
R-1187 g — the user back in the ORM: True
|
||||
R-1187 g — the front door after import: no login form (302)
|
||||
the import's DB line: 2026/10/26 21:01:59 export.go:166: [DEBUG] [appexport] restoreDatabase: found DB service=grimmory-db type=mariadb container=grimmory-db
|
||||
2026/10/26 21:07:18 export.go:166: [DEBUG] [appexport] restoring database: type=postgres
|
||||
2026/10/26 21:07:18 export.go:166: [DEBUG] [appexport] restoreDatabase: found DB service=adventurelog-postgres type=postgres container=adventurelog-postgres
|
||||
[7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'adventurelog', 'volumes_removed': ['adventurelog_adventurelog_media', 'adventurelog_adventurelog_postgres_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [],
|
||||
6. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '', 'kept': '', 'subdomain': '404'}
|
||||
bundles removed: 0
|
||||
@@ -0,0 +1,41 @@
|
||||
== storage key fingerprint
|
||||
encryption-key <present>
|
||||
namespace Tester4
|
||||
felhom-pbs:backup/ct/9201/2026-10-20T12:17:26Z 9201 1792498646 23:58:7a:71:46:72:0c:1e
|
||||
own key:
|
||||
== tiers (agent 0.166.0)
|
||||
{
|
||||
"ok": true,
|
||||
"data": {
|
||||
"vmid": 9201,
|
||||
"tiers": [
|
||||
{
|
||||
"target": "local",
|
||||
"cadence_seconds": 86400,
|
||||
"primary": true,
|
||||
"storage": "present"
|
||||
},
|
||||
{
|
||||
"target": "felhom-pbs",
|
||||
"cadence_seconds": 604800,
|
||||
"primary": false,
|
||||
"storage": "present"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
20:10:33
|
||||
== due felhom-pbs (agent 0.166.0)
|
||||
{
|
||||
"ok": true,
|
||||
"data": {
|
||||
"vmid": 9201,
|
||||
"due": false,
|
||||
"reason": "within cadence window",
|
||||
"age_seconds": 546807,
|
||||
"target": "felhom-pbs",
|
||||
"age_state": "known"
|
||||
}
|
||||
}
|
||||
own key: 7c:3b:36:51:bc:a6:b4:90
|
||||
20:10:53
|
||||
@@ -0,0 +1,16 @@
|
||||
bin=/usr/local/bin/felhom-agent
|
||||
active
|
||||
felhom-agent 0.167.0-night1
|
||||
== due felhom-pbs (agent 0.167.0-night1, R-1232 fix)
|
||||
{
|
||||
"ok": true,
|
||||
"data": {
|
||||
"vmid": 9201,
|
||||
"due": true,
|
||||
"reason": "no successful backup recorded yet",
|
||||
"target": "felhom-pbs",
|
||||
"age_state": "absent"
|
||||
}
|
||||
}
|
||||
Oct 26 21:11:14 r1232-byo felhom-agent[17600]: time=2026-10-26T21:11:14.654+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=felhom-pbs volid=felhom-pbs:backup/ct/9201/202
|
||||
20:11:14
|
||||
@@ -0,0 +1,35 @@
|
||||
== archives after
|
||||
felhom-pbs:backup/ct/9201/2026-10-20T12:17:26Z 23:58:7a:71:46:72:0c:1e
|
||||
felhom-pbs:backup/ct/9201/2026-10-26T20:13:55Z 7c:3b:36:51:bc:a6:b4:90
|
||||
== due felhom-pbs after the run
|
||||
{
|
||||
"ok": true,
|
||||
"data": {
|
||||
"vmid": 9201,
|
||||
"due": false,
|
||||
"reason": "within cadence window",
|
||||
"age_seconds": 99,
|
||||
"target": "felhom-pbs",
|
||||
"age_state": "known"
|
||||
}
|
||||
}
|
||||
== agent journal (felhom-pbs, R-1232)
|
||||
Oct 26 21:00:45 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:45.237+01:00 level=INFO msg="felhom-agent daemon starting" version=0.166.0 host_id=Tester4-b5dbbd hub_url=https://hub.felhom.eu interval_s=900
|
||||
Oct 26 21:00:45 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:45.255+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
|
||||
Oct 26 21:00:46 r1232-byo sudo[3034]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-backup-target-apply grant felhom-pbs
|
||||
Oct 26 21:00:47 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:47.454+01:00 level=ERROR msg="store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)" target=felhom-pbs privilege=Datastore.AllocateSpace action="felhom-bac
|
||||
Oct 26 21:00:47 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:47.454+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:store-grant:felhom-pbs feature="backup tier felhom-pbs: the agent'
|
||||
Oct 26 21:00:57 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:57.058+01:00 level=INFO msg="pbsdr: one-time token secret consumed (single-use; value withheld from logs)" storage_id=felhom-pbs secret_len=36
|
||||
Oct 26 21:00:57 r1232-byo sudo[3774]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply create felhom-pbs 10.77.0.1 felhom-offsite Tester4 felhom@pbs!Tester4 c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:6
|
||||
Oct 26 21:00:58 r1232-byo sudo[3788]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply grant felhom-pbs
|
||||
Oct 26 21:00:59 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:59.566+01:00 level=INFO msg="pbsdr: seeded escrow.pbs_storage_id (the ceremony one-liner needs no flags)" storage_id=felhom-pbs
|
||||
Oct 26 21:00:59 r1232-byo felhom-agent[2815]: time=2026-10-26T21:00:59.567+01:00 level=INFO msg="pbsdr: converged" state=applied storage_id=felhom-pbs
|
||||
Oct 26 21:11:00 r1232-byo felhom-os-apply[17411]: os-apply: CONFFILE updated /etc/debian_version (it was not changed locally)
|
||||
Oct 26 21:11:04 r1232-byo felhom-agent[17600]: time=2026-10-26T21:11:04.570+01:00 level=INFO msg="felhom-agent daemon starting" version=0.167.0-night1 host_id=Tester4-b5dbbd hub_url=https://hub.felhom.eu interval_s=900
|
||||
Oct 26 21:11:04 r1232-byo felhom-agent[17600]: time=2026-10-26T21:11:04.582+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
|
||||
Oct 26 21:11:05 r1232-byo sudo[17768]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
|
||||
Oct 26 21:11:06 r1232-byo sudo[17814]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
|
||||
Oct 26 21:11:14 r1232-byo felhom-agent[17600]: time=2026-10-26T21:11:14.654+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=felhom-pbs volid=felhom-pbs:backup/ct/9201/2026-10-20T12:17:26Z size_bytes=2072117616
|
||||
Oct 26 21:13:56 r1232-byo felhom-agent[17600]: time=2026-10-26T21:13:56.633+01:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1793045635144752924
|
||||
Oct 26 21:15:17 r1232-byo felhom-agent[17600]: time=2026-10-26T21:15:17.431+01:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-10-26T20:13:55Z size_bytes=2072870896 uncovered_volumes=2
|
||||
Oct 26 21:15:17 r1232-byo felhom-agent[17600]: time=2026-10-26T21:15:17.431+01:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1793045635144752924 archive=felhom-pbs:backup/ct/9201/2026-10-26T20:13:55Z
|
||||
@@ -0,0 +1,163 @@
|
||||
|
||||
[0;32m[INFO][0m felhom-host-install v1.36.1 — mode=byo customer=Tester4 vmid=9201
|
||||
|
||||
[0;34m[STEP][0m 1/8 pre-flight
|
||||
[0;32m[INFO][0m pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
|
||||
[0;32m[INFO][0m node: r1232-byo (explicit)
|
||||
[0;32m[INFO][0m agent config: /etc/felhom-agent/agent.json
|
||||
[0;32m[INFO][0m agent: not installed yet — will be fetched + installed in step 5/8
|
||||
[0;32m[INFO][0m local-lvm free: ~29 GiB
|
||||
[1;33m[WARN][0m local-lvm free ~29 GiB is below the recommended 120 GiB — the install continues; proceed only if you sized the grows deliberately
|
||||
[0;32m[INFO][0m free RAM: ~6277 MiB
|
||||
[0;32m[INFO][0m existing guests on this host: 0 (pct+qm)
|
||||
[0;32m[INFO][0m archive storage 'local' present
|
||||
[0;32m[INFO][0m acl storages all present or pre-positioned: local local-lvm
|
||||
[0;32m[INFO][0m dnsmasq: not present before Felhom — recorded; uninstall will remove it again if we install it
|
||||
[0;32m[INFO][0m host DNS (:53): free
|
||||
[0;32m[INFO][0m hub reachable (https://hub.felhom.eu)
|
||||
[0;32m[OK][0m customer 'Tester4' exists + passphrase valid
|
||||
[0;32m[INFO][0m golden: none local — will fetch + verify from Gitea in step 7/8
|
||||
r1232-byo
|
||||
|
||||
[0;34m[STEP][0m BYO disclosure — installing Felhom on a host the operator does NOT own will create:
|
||||
pveum: roles FelhomAgentBase/FelhomAgentGuest/FelhomAgentStore + user felhom-agent@pve + token felhom-agent@pve!agent
|
||||
+ scoped ACL grants (Base@/, Guest@/pool/felhom, Store@[local local-lvm]) + the 'felhom' pool
|
||||
agent: service user felhom-agent + /usr/local/bin/felhom-agent (+ .bak-* backups; A/B slots via the guarded self-update)
|
||||
+ /etc/systemd/system/felhom-agent.service (+ felhom-agent-rollback.service + start-limit drop-in)
|
||||
+ /etc/sudoers.d/felhom-agent + state dir /var/lib/felhom-agent + config /etc/felhom-agent/agent.json (0600)
|
||||
host: /usr/local/sbin/felhom-mkfs-guarded + /usr/local/sbin/felhom-selfupdate-guarded
|
||||
+ /usr/local/sbin/felhom-pbs-apply (PBS-DR apply wrapper — DR capability is baked
|
||||
on every install; ACTIVATION stays a hub flag, off = zero effect on this host)
|
||||
+ felhom-mgmt-watchdog service+timer+script + /etc/tmpfiles.d/felhom-privsep.conf
|
||||
+ /usr/local/sbin/felhom-priv-apply + the guest-hook snippet under /var/lib/vz/snippets/
|
||||
(both from the agent's config bundle)
|
||||
+ the 'sudo' and 'age' packages if absent + install state dir /var/lib/felhom-install
|
||||
wg: an OUTBOUND WireGuard tunnel to the Felhom hub (wg_tunnel.enabled=true — base
|
||||
infrastructure like the cloudflared tunnel; hands-free peer registration; the
|
||||
operator can disable it per-customer hub-side on request)
|
||||
guest: the provisioned Felhom LXC (vmid 9201, capped 2 cores / 4096 MiB) + its volumes
|
||||
+ the golden vzdump imported onto storage 'local'
|
||||
reuse: a golden already on 'local' is used only if its sha256 or its controller version
|
||||
matches the hub's vouched golden; otherwise the vouched one is fetched (a --golden you name is refused)
|
||||
update: operator-signed self-update authority: felhom-op-1
|
||||
NOT touched in byo mode: root@pam (no break-glass), host DNS (:53), OOB sshd.
|
||||
|
||||
Type this host's short hostname (r1232-byo) to acknowledge the byo install: [0;32m[OK][0m byo install acknowledged for host r1232-byo
|
||||
[0;32m[OK][0m pre-flight passed
|
||||
[0;34m[STEP][0m 2/8 Proxmox API token
|
||||
[0;32m[OK][0m token minted (secret captured, not logged)
|
||||
[0;32m[OK][0m scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm])
|
||||
[0;36m[SKIP][0m old broad role FelhomAgent already absent
|
||||
[0;34m[STEP][0m 3/8 compute volume grows
|
||||
[0;32m[INFO][0m auto-computed from a 29 GiB thin pool (16 §7: 50/100/130 GiB)
|
||||
[0;32m[INFO][0m grows: rootfs +0G (->32G), data +26G (->50G, ONE volume)
|
||||
[0;34m[STEP][0m 4/8 host enrollment (POST /host-enroll)
|
||||
[0;32m[OK][0m host MINTED (first enroll)
|
||||
[0;32m[INFO][0m host_id: Tester4-b5dbbd (api_key captured, not logged)
|
||||
[0;36m[SKIP][0m break-glass (step 4b) SKIPPED (byo: the host owner manages break-glass; root@pam untouched, nothing vaulted)
|
||||
[0;34m[STEP][0m 5/8 agent install (fetch + verify + install)
|
||||
[0;36m[SKIP][0m apt repo alignment skipped (byo — the owner manages repos)
|
||||
[1;33m[WARN][0m no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
|
||||
[0;32m[INFO][0m manifest: agent v0.166.0 (sha 3f22c65b78914632…), golden v0.344.0
|
||||
[0;32m[INFO][0m fetching agent binary v0.166.0 from Gitea …
|
||||
[0;32m[OK][0m verified sha256 3f22c65b78914632… matches the hub manifest
|
||||
[0;32m[OK][0m installed /usr/local/bin/felhom-agent (felhom-agent 0.166.0)
|
||||
[0;32m[INFO][0m installing the 'sudo' package (required for the non-root agent model) …
|
||||
[0;32m[OK][0m sudo installed (Sudo version 1.9.16p2)
|
||||
[0;32m[INFO][0m installing the 'age' package (escrow ceremony identity-wrap dependency) …
|
||||
[0;32m[OK][0m age installed (1.2.1)
|
||||
[0;32m[OK][0m created service user felhom-agent
|
||||
[0;32m[OK][0m added felhom-agent to systemd-journal (unprivileged journal read for NAS verify)
|
||||
[0;32m[OK][0m verified sha256 093772c1bda8b974… matches the hub manifest
|
||||
os-apply: BUNDLE /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf skipped: a BYO host keeps its own settings (D43)
|
||||
os-apply: BUNDLE START agent=0.166.0 sha=093772c1bda8b974 authority=installer files=32 write=26 same=1 kept=0 skipped=5
|
||||
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mkfs-guarded (new)
|
||||
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-selfupdate-guarded (new)
|
||||
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-pbs-apply (new)
|
||||
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-backup-target-apply (new)
|
||||
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-vzdump-partial-clean (new)
|
||||
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-thin-check (new)
|
||||
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-internal-place (new)
|
||||
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-crash-guard (new)
|
||||
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-priv-apply (new)
|
||||
os-apply: BUNDLE WROTE /var/lib/vz/snippets/felhom-guest-hook.sh (new)
|
||||
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-shared-parent.sh (new)
|
||||
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-shared-parent.service (new)
|
||||
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard.service (new)
|
||||
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.service (new)
|
||||
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.timer (new)
|
||||
os-apply: BUNDLE WROTE /etc/felhom/crash-guard.conf (new)
|
||||
os-apply: BUNDLE WROTE /etc/grub.d/01_felhom_oneshot (new)
|
||||
os-apply: BUNDLE WROTE /etc/grub.d/42_felhom_oneshot (new)
|
||||
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service (new)
|
||||
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent-rollback.service (new)
|
||||
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf (new)
|
||||
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mgmt-watchdog (new)
|
||||
os-apply: BUNDLE WROTE /etc/tmpfiles.d/felhom-privsep.conf (new)
|
||||
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.service (new)
|
||||
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.timer (new)
|
||||
os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-agent (new)
|
||||
os-apply: BUNDLE DONE agent=0.166.0 written=26 same=1 self-check=ok signers-created=False
|
||||
[0;32m[OK][0m installed every root-owned file from config bundle v0.166.0 (sha 093772c1bda8b974…; checked, previous copies kept)
|
||||
[0;36m[SKIP][0m OOB (felhom-sshd) off (byo, or appliance --no-oob) — the bundle skipped its files
|
||||
[0;34m[STEP][0m 5b/8 internal data place (Belső tárhely)
|
||||
[0;36m[SKIP][0m byo: the host owner's thin pool is not ours to divide — no internal place
|
||||
[0;34m[STEP][0m 6/8 agent config + service
|
||||
[1;33m[WARN][0m backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive.
|
||||
[1;33m[WARN][0m It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard.
|
||||
[0;32m[INFO][0m node=r1232-byo local_api=192.168.0.127:8443 tls_fp=4B:20:94:3E:FB:AB…
|
||||
[0;32m[OK][0m wrote /etc/felhom-agent/agent.json (0600 felhom-agent)
|
||||
[0;32m[OK][0m byo config asserts passed (lan_resolver.enable / oob.enabled off; deployment_mode=byo; wg_tunnel intentionally ON — base infrastructure)
|
||||
[0;32m[OK][0m agent --selftest (read-only) passed
|
||||
[0;32m[OK][0m felhom-agent service active (non-root felhom-agent reads the config OK)
|
||||
[0;32m[OK][0m wrote /etc/felhom/os-trust.json + operator-signers (felhom-op-1) — the root-owned slow-lane anchors
|
||||
[0;34m[STEP][0m 7/8 golden archive
|
||||
[1;33m[WARN][0m no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
|
||||
[0;32m[INFO][0m fetching golden v0.344.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_10_26-21_00_49.tar.zst
|
||||
[0;32m[OK][0m verified sha256 767a90fe07ed47c8… matches the hub manifest
|
||||
[0;32m[OK][0m golden imported + verified: local:backup/vzdump-lxc-9100-2026_10_26-21_00_49.tar.zst
|
||||
[0;34m[STEP][0m 8/8 provision guest 9201
|
||||
[0;36m[SKIP][0m pool felhom already exists
|
||||
=== felhom-agent 0.166.0 selftest=provision (vmid=9201 customer=Tester4 hostname=tester4) ===
|
||||
--- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_10_26-21_00_49.tar.zst → vmid 9201 ---
|
||||
time=2026-10-26T21:02:13.966+01:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=9201 pool=felhom
|
||||
[OK] front half: vmid 9201 up (boot+running) in 1m14s; MAC=BC:24:11:06:5B:AC
|
||||
--- back half: mint per-guest token + populate bootstrap config mount ---
|
||||
time=2026-10-26T21:02:19.331+01:00 level=INFO msg="provision: back-half complete" vmid=9201 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=192.168.0.127:8443
|
||||
[OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 9201 (host dir /var/lib/felhom-agent/guests/9201/bootstrap)
|
||||
local-api endpoint 192.168.0.127:8443 · leaf fp c0463612a04329e8b367f3aed7e925da5b5e44668df7b726fc5e7ac4dd6837ad · token: minted (not printed)
|
||||
=== selftest=provision OK — guest 9201 provisioned + bootstrap-mounted (KEPT) ===
|
||||
next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller,
|
||||
which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api.
|
||||
[0;32m[OK][0m provision completed
|
||||
[0;32m[INFO][0m rebooting guest 9201 so the baked controller-bootstrap unit picks up the mount
|
||||
[0;34m[STEP][0m verify
|
||||
[0;32m[OK][0m pct status: running
|
||||
[0;32m[OK][0m onboot: 1
|
||||
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G
|
||||
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
|
||||
rootfs: local-lvm:vm-9201-disk-0,size=32G
|
||||
[0;32m[OK][0m pool: guest 9201 is a member of felhom
|
||||
[0;32m[OK][0m acl: FelhomAgentBase@/ present (user+token)
|
||||
[0;32m[OK][0m acl: FelhomAgentGuest@/pool/felhom present (user+token)
|
||||
[0;32m[OK][0m acl: FelhomAgentStore@/storage/local present (user+token)
|
||||
[0;32m[OK][0m acl: FelhomAgentStore@/storage/local-lvm present (user+token)
|
||||
[0;32m[OK][0m acl: FelhomAgentGuest@/vms/990000 present (user+token)
|
||||
[0;32m[OK][0m acl: FelhomAgentGuest@/vms/990001 present (user+token)
|
||||
[0;32m[OK][0m acl: FelhomAgentGuest@/vms/990002 present (user+token)
|
||||
[0;32m[OK][0m acl: FelhomAgentGuest@/vms/990003 present (user+token)
|
||||
[0;32m[OK][0m acl: FelhomAgentGuest@/vms/990004 present (user+token)
|
||||
[0;32m[OK][0m acl: FelhomAgentGuest@/vms/990005 present (user+token)
|
||||
[0;32m[OK][0m acl: FelhomAgentGuest@/vms/990006 present (user+token)
|
||||
[0;32m[OK][0m acl: FelhomAgentGuest@/vms/990007 present (user+token)
|
||||
[0;32m[OK][0m acl: FelhomAgentGuest@/vms/990008 present (user+token)
|
||||
[0;32m[OK][0m acl: FelhomAgentGuest@/vms/990009 present (user+token)
|
||||
[0;32m[OK][0m authz signers: 2 (operator-signed self-update armed)
|
||||
[0;32m[OK][0m controller: Up 4 seconds (health: starting) (after ~0s)
|
||||
[0;32m[INFO][0m controller image: gitea.dooplex.hu/admin/felhom-controller:0.345.0
|
||||
[1;33m[WARN][0m cloudflared not visible yet
|
||||
[0;32m[INFO][0m (confirm in the hub UI that host Tester4-b5dbbd reports guest 9201)
|
||||
|
||||
[0;32m[OK][0m Day-0 provision SUCCESS — vmid=9201 host_id=Tester4-b5dbbd customer=Tester4 golden=local:backup/vzdump-lxc-9100-2026_10_26-21_00_49.tar.zst
|
||||
INSTALL_RC=0
|
||||
Connection to 192.168.0.127 closed.
|
||||
@@ -0,0 +1 @@
|
||||
19:59:44
|
||||
@@ -0,0 +1,4 @@
|
||||
purging VM 9375 from related configurations..
|
||||
Configuration file 'nodes/demo-hp/qemu-server/9375.conf' does not exist
|
||||
ls: cannot access '/mnt/pve/nvme-scratch/images/9375': No such file or directory
|
||||
felhom-installer-1.30.0-pve9.2-1.iso
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
|
||||
[0;32m[INFO][0m felhom-host-install v1.36.1 — mode=uninstall
|
||||
|
||||
[0;34m[STEP][0m UNINSTALL — local host teardown
|
||||
[0;32m[INFO][0m no --vmid given; using recorded provisioned_vmid=9201 from /var/lib/felhom-install/state.json
|
||||
[0;32m[INFO][0m vmid 9201 looks like a Felhom guest (has the /etc/felhom-bootstrap mount)
|
||||
[0;32m[INFO][0m pct config 9201:
|
||||
arch: amd64
|
||||
cores: 2
|
||||
features: nesting=1,keyctl=1
|
||||
hookscript: local:snippets/felhom-guest-hook.sh
|
||||
hostname: tester4
|
||||
memory: 4096
|
||||
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G
|
||||
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
|
||||
mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
|
||||
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:06:5B:AC,ip=dhcp,type=veth
|
||||
onboot: 1
|
||||
ostype: debian
|
||||
rootfs: local-lvm:vm-9201-disk-0,size=32G
|
||||
swap: 4096
|
||||
unprivileged: 1
|
||||
Type the vmid (9201) to confirm PERMANENT destruction: r1232-byo
|
||||
[0;31m[ERROR][0m confirmation mismatch (got 'r1232-byo', expected '9201') — aborting, nothing destroyed
|
||||
UNINSTALL_RC=1
|
||||
Connection to 192.168.0.127 closed.
|
||||
@@ -0,0 +1,84 @@
|
||||
|
||||
[0;32m[INFO][0m felhom-host-install v1.36.1 — mode=uninstall
|
||||
|
||||
[0;34m[STEP][0m UNINSTALL — local host teardown
|
||||
[0;32m[INFO][0m no --vmid given; using recorded provisioned_vmid=9201 from /var/lib/felhom-install/state.json
|
||||
[0;32m[INFO][0m vmid 9201 looks like a Felhom guest (has the /etc/felhom-bootstrap mount)
|
||||
[0;32m[INFO][0m pct config 9201:
|
||||
arch: amd64
|
||||
cores: 2
|
||||
features: nesting=1,keyctl=1
|
||||
hookscript: local:snippets/felhom-guest-hook.sh
|
||||
hostname: tester4
|
||||
memory: 4096
|
||||
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G
|
||||
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
|
||||
mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
|
||||
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:06:5B:AC,ip=dhcp,type=veth
|
||||
onboot: 1
|
||||
ostype: debian
|
||||
rootfs: local-lvm:vm-9201-disk-0,size=32G
|
||||
swap: 4096
|
||||
unprivileged: 1
|
||||
Type the vmid (9201) to confirm PERMANENT destruction: 9201
|
||||
Logical volume "vm-9201-disk-0" successfully removed.
|
||||
Logical volume "vm-9201-disk-1" successfully removed.
|
||||
[0;32m[OK][0m guest 9201 destroyed
|
||||
[0;34m[STEP][0m host-level removal
|
||||
Removed '/etc/systemd/system/multi-user.target.wants/felhom-agent.service'.
|
||||
[0;32m[OK][0m removed /etc/felhom-agent (the agent config and every copy of it)
|
||||
Removed '/etc/systemd/system/multi-user.target.wants/wg-quick@wg-felhom.service'.
|
||||
[0;32m[OK][0m WireGuard tunnel to the Felhom off-site endpoint is down (wg-quick@wg-felhom disabled, conf removed)
|
||||
Removed '/etc/systemd/system/timers.target.wants/felhom-mgmt-watchdog.timer'.
|
||||
The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=,
|
||||
Also=, or Alias= settings in the [Install] section, and DefaultInstance= for
|
||||
template units). This means they are not meant to be enabled or disabled using systemctl.
|
||||
|
||||
Possible reasons for having these kinds of units are:
|
||||
• A unit may be statically enabled by being symlinked from another unit's
|
||||
.wants/, .requires/, or .upholds/ directory.
|
||||
• A unit's purpose may be to act as a helper for some other unit which has
|
||||
a requirement dependency on it.
|
||||
• A unit may be started when needed via activation (socket, path, timer,
|
||||
D-Bus, udev, scripted systemctl call, ...).
|
||||
• In case of template units, the unit is meant to be enabled with some
|
||||
instance name specified.
|
||||
Removed '/etc/systemd/system/pve-guests.service.wants/felhom-shared-parent.service'.
|
||||
Removed '/etc/systemd/system/multi-user.target.wants/felhom-shared-parent.service'.
|
||||
Generating grub configuration file ...
|
||||
Found linux image: /boot/vmlinuz-7.0.2-6-pve
|
||||
Found initrd image: /boot/initrd.img-7.0.2-6-pve
|
||||
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
|
||||
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
|
||||
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
|
||||
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
|
||||
Adding boot menu entry for UEFI Firmware Settings ...
|
||||
done
|
||||
[0;36m[SKIP][0m old broad role FelhomAgent already absent
|
||||
[0;36m[SKIP][0m golden vzdump left in place (pass --remove-golden to remove)
|
||||
|
||||
[0;34m[STEP][0m kept vs wiped — read before pulling drives or closing the customer out
|
||||
WIPED (this run):
|
||||
- guest 9201 (container + its OS/Docker/user-data volumes)
|
||||
- the felhom-agent: binary, unit, sudoers (+ its copies), state dir, service user, and its config
|
||||
directory with every copy of the config in it
|
||||
- the WireGuard tunnel to the Felhom off-site endpoint (wg-quick@wg-felhom disabled, /etc/wireguard/wg-felhom.conf removed)
|
||||
- self-update artifacts: guarded wrapper, A/B slots (.prev/.new.*), rollback unit, start-limit drop-in
|
||||
- break-glass watchdog + OOB artifacts (where present); guest-hook snippet; dnsmasq snippets; the mkfs, pbs-apply,
|
||||
backup-target-apply, os-apply and priv-apply wrappers; the crash guard; the config-bundle record
|
||||
- pveum: the Felhom roles/user/token/scoped ACL; the emptied felhom pool
|
||||
- the install state file
|
||||
KEPT (lives on deliberately — remove/rotate these out-of-band if the customer is leaving):
|
||||
- the enrolled drives + ALL data under /mnt/felhom-drives — unmounted only, NEVER wiped;
|
||||
the drives are physically removable now.
|
||||
- the PBS backups + this customer's namespace on the PBS side — delete there if wanted
|
||||
- the hub host/customer record + report history (operator UI / DB)
|
||||
- this host's WireGuard PEER on the hub and the off-site endpoint (its /32 and public key) — the
|
||||
tunnel is down from this side; delete the peer in the hub (operator) if the customer is leaving
|
||||
- the escrow blob in the hub, if one exists (operator UI)
|
||||
|
||||
[0;32m[OK][0m UNINSTALL complete — removed: guest 9201, the felhom-agent (unit/sudoers/binary/state/config+baks/user + selfupdate-artifacts/shared-parent/mkfs-wrapper/pbs-apply-wrapper/hook-snippet/dnsmasq-snippets), the pveum role/user/token/ACL, the felhom pool, and /var/lib/felhom-install/state.json.
|
||||
[0;32m[INFO][0m golden vzdump: left in place (--remove-golden to remove).
|
||||
[0;32m[INFO][0m NOTE: the 'sudo' package was left INSTALLED (a system package, never ours to remove). dnsmasq is removed only when the install-time record says Felhom installed it; a resolver that pre-dated Felhom is never touched. The host record still exists in the hub — remove it there if desired.
|
||||
UNINSTALL_RC=0
|
||||
Connection to 192.168.0.127 closed.
|
||||
@@ -0,0 +1,18 @@
|
||||
== before (19:49:23)
|
||||
chaoscrash held since 2026-09-24T12:16:24Z
|
||||
chaosoom held since 2026-09-24T11:48:01Z
|
||||
chaosoomb held since 2026-09-24T12:39:51Z
|
||||
gokapi held since 2026-09-24T10:37:48Z
|
||||
controller stopped 19:49:24
|
||||
Check the app's data first — the undo copies are in its unit's db-dumps dir.
|
||||
chaoscrash rc=0
|
||||
Check the app's data first — the undo copies are in its unit's db-dumps dir.
|
||||
chaosoom rc=0
|
||||
Check the app's data first — the undo copies are in its unit's db-dumps dir.
|
||||
chaosoomb rc=0
|
||||
[INFO] [settings] Settings saved
|
||||
gokapi rc=0
|
||||
== after
|
||||
no restore holds in force
|
||||
[INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json
|
||||
controller started 19:49:26
|
||||
@@ -0,0 +1,2 @@
|
||||
[7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'privatebin', 'volumes_removed': ['privatebin_privatebin_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem tárolt saját adatot
|
||||
leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '', 'kept': '', 'subdomain': '404'}
|
||||
@@ -0,0 +1,125 @@
|
||||
#!/usr/bin/env python3
|
||||
"""r1187_r1220_live.py — R-1220 and R-1187 (a)(b) live on scratch guest 9202, through the endpoints the UI invokes.
|
||||
|
||||
A THROWAWAY Grimmory (drive appdata binds). Steps:
|
||||
1. install; a marker folder in its drive appdata: an owned file (1000:1001, 0640), a link inside, a link outside;
|
||||
2. R-1220: its app.yaml made unreadable (a byte copy saved first) → after a stack rescan the „Megőrzött adatok" page does
|
||||
NOT list its folder; the app.yaml put back byte-identical;
|
||||
3. R-1187 (a): a FIFO in the folder → the export FAILS naming the FIFO's path; the FIFO removed;
|
||||
4. R-1187 (a)(b): export → import over the running app → the owned file keeps owner and mode, the inside link comes
|
||||
back, the outside link is refused (listed in the controller WARN);
|
||||
5. teardown: the app removed through the product (data and backups), the bundles this run made deleted.
|
||||
|
||||
usage: SC=<0600 dir with .ctlpw> SWEEP_EV=<evidence dir> TARGET=9202 DRIVE_ROOT=/mnt/felhom-drives/scratch_hdd python3 -u r1187_r1220_live.py
|
||||
"""
|
||||
import os
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
|
||||
signal.signal(signal.SIGTERM, lambda *a: (_ for _ in ()).throw(SystemExit("SIGTERM — cleaning up")))
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools"))
|
||||
import sweep as sw # noqa: E402
|
||||
|
||||
w = sw.w
|
||||
APP, SUB, DRIVE = "grimmory", "grimd53", os.environ.get("DRIVE_ROOT", "/mnt/felhom-drives/scratch_hdd")
|
||||
say = print
|
||||
|
||||
|
||||
def g(cmd):
|
||||
return sw.g(cmd).strip()
|
||||
|
||||
|
||||
def job(kind):
|
||||
path = "/api/export/status" if kind == "export" else "/api/export/import/status"
|
||||
for _ in range(180):
|
||||
time.sleep(5)
|
||||
code, d = w.ctl("GET", path)
|
||||
j = d.get("data", d) if isinstance(d, dict) else {}
|
||||
if j.get("done") or j.get("Done"):
|
||||
return j
|
||||
return {"timeout": True}
|
||||
|
||||
|
||||
def scan_after(since):
|
||||
"""Wait for a stack rescan that started after `since` (the controller rescans on its own every ~2 min)."""
|
||||
for _ in range(40):
|
||||
time.sleep(10)
|
||||
if g(f"docker logs --since {since} felhom-controller 2>&1 | grep -c 'ScanStacks complete'") not in ("", "0"):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
w.say = say
|
||||
w.login()
|
||||
try:
|
||||
ok = w.deploy(APP, SUB, {"HDD_PATH": DRIVE}) and w.wait_app(SUB, "/", tries=90, delay=5)
|
||||
say("1. grimmory installed and answering:", ok)
|
||||
appdata = g(f"ls -d {DRIVE}/appdata/grimmory 2>/dev/null | head -1")
|
||||
say(" its appdata folder:", appdata or "(not found)")
|
||||
if not appdata:
|
||||
raise SystemExit("INSTRUMENT: no appdata folder — no verdict")
|
||||
m = f"{appdata}/data/cc-r1187" # INSIDE the bound folder (appdata/grimmory/data) — the export packs only binds
|
||||
say(" markers:", g(f"mkdir -p {m} && printf hi > {m}/owned.txt && chown 1000:1001 {m}/owned.txt && chmod 0640 {m}/owned.txt"
|
||||
f" && ln -sfn owned.txt {m}/link-in && ln -sfn /etc/hostname {m}/link-out && ls -ln {m} | tail -n +2"))
|
||||
|
||||
# 2. R-1220 (SKIP_R1220=1: already proven in the first run, `r1187-r1220-9202-run1.txt`)
|
||||
if os.environ.get("SKIP_R1220"):
|
||||
say("2. R-1220 skipped (proven in the first run)")
|
||||
else:
|
||||
stack = f"/opt/docker/stacks/{APP}"
|
||||
g(f"cp -p {stack}/app.yaml /root/r1220-app.yaml.save && printf 'deployed: true\\nenv: [broken: yaml: here\\n' > {stack}/app.yaml")
|
||||
t = int(time.time())
|
||||
say("2. app.yaml made unreadable; rescan seen:", scan_after(t))
|
||||
page = w.page("/kept-data")
|
||||
say(" R-1220: the kept-data page lists grimmory's folder:", appdata in page, "(want False) | WARN lines:",
|
||||
g(f"docker logs --since {t} felhom-controller 2>&1 | grep -c 'R-1220'"))
|
||||
g(f"cp -p /root/r1220-app.yaml.save {stack}/app.yaml && rm -f /root/r1220-app.yaml.save")
|
||||
say(" app.yaml put back, byte check:", g(f"sha256sum {stack}/app.yaml | cut -c1-16"))
|
||||
scan_after(int(time.time()))
|
||||
|
||||
# 3. R-1187 (a) FIFO
|
||||
g(f"mkfifo {m}/pipe")
|
||||
code, d = w.ctl("POST", "/api/export/start", {"stack_name": APP, "dest_drive": DRIVE, "stop_app": False})
|
||||
say("3. export with a FIFO ->", code, str(d)[:160])
|
||||
if code != "200":
|
||||
raise SystemExit("INSTRUMENT: the export start was refused — no verdict")
|
||||
j = job("export")
|
||||
say(" job:", str(j.get("error") or j.get("Error") or j)[:300])
|
||||
g(f"rm -f {m}/pipe")
|
||||
if j.get("timeout"):
|
||||
raise SystemExit("INSTRUMENT: the export job never ended — no verdict")
|
||||
|
||||
# 4. R-1187 (a)(b) round trip
|
||||
code, d = w.ctl("POST", "/api/export/start", {"stack_name": APP, "dest_drive": DRIVE, "stop_app": False})
|
||||
say("4. export ->", code, str(d)[:120])
|
||||
if code != "200":
|
||||
raise SystemExit("INSTRUMENT: the export start was refused — no verdict")
|
||||
j = job("export")
|
||||
say(" job:", str(j)[:300])
|
||||
fab = g(f"ls -t {DRIVE}/exports/grimmory*.fab 2>/dev/null | head -1")
|
||||
say(" bundle:", fab)
|
||||
say(" tar listing of the drive part:", g(f"mkdir -p /root/r1187x && cd /root/r1187x && tar -xzf {fab} data/hdd 2>/dev/null; "
|
||||
f"for f in data/hdd/*.tar; do tar -tvf $f | grep cc-r1187; done; rm -rf /root/r1187x"))
|
||||
g(f"rm -rf {m}") # the import must bring it back
|
||||
ti = int(time.time())
|
||||
code, d = w.ctl("POST", "/api/export/import", {"path": fab})
|
||||
say(" import ->", code, str(d)[:160])
|
||||
if code != "200":
|
||||
raise SystemExit("INSTRUMENT: the import start was refused — no verdict")
|
||||
j = job("import")
|
||||
say(" import job:", str(j)[:300])
|
||||
say(" after import:", g(f"ls -ln {m} | tail -n +2; stat -c '%n %u:%g %a' {m}/owned.txt; readlink {m}/link-in; "
|
||||
f"ls -l {m}/link-out 2>&1 | head -1"))
|
||||
say(" the import's WARN:", g(f"docker logs --since {ti} felhom-controller 2>&1 | grep 'R-1187 b' | cut -c1-220"))
|
||||
say(" the app answers:", w.wait_app(SUB, "/", tries=60, delay=5))
|
||||
finally:
|
||||
sw.remove(APP, keep=False, say=say)
|
||||
say("5. leftovers:", sw.leftovers(APP, SUB))
|
||||
say(" bundles of this run removed:", g(f"ls {DRIVE}/exports/grimmory*.fab 2>/dev/null; rm -f {DRIVE}/exports/grimmory*.fab; ls {DRIVE}/exports/ 2>/dev/null | grep -c grimmory"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env python3
|
||||
"""r1187g_live.py — R-1187 (g) live on scratch guest 9202: a PostGIS app's database survives a `.fab` export → import.
|
||||
|
||||
A THROWAWAY AdventureLog (postgis/postgis). A superuser is made with the app's own `createsuperuser`; the app is exported
|
||||
and imported through the endpoints the UI invokes; then the user is read back through the app's FRONT DOOR — the
|
||||
Django admin login page — and through its own ORM. Before the fix the import did not recognise postgis as a database,
|
||||
so the dump was never loaded. Teardown through the product. The throwaway password lives only in this process.
|
||||
|
||||
usage: SC=<0600 dir with .ctlpw> SWEEP_EV=<evidence dir> TARGET=9202 DRIVE_ROOT=/mnt/felhom-drives/scratch_hdd python3 -u r1187g_live.py
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import signal
|
||||
import urllib.parse
|
||||
import sys
|
||||
import time
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools"))
|
||||
import sweep as sw # noqa: E402
|
||||
|
||||
signal.signal(signal.SIGTERM, lambda *a: (_ for _ in ()).throw(SystemExit("SIGTERM — cleaning up")))
|
||||
w = sw.w
|
||||
APP, SUB, DRIVE = "adventurelog", "advd53", os.environ.get("DRIVE_ROOT", "/mnt/felhom-drives/scratch_hdd")
|
||||
say = print
|
||||
|
||||
|
||||
def g(cmd):
|
||||
return sw.g(cmd).strip()
|
||||
|
||||
|
||||
def job(path):
|
||||
for _ in range(180):
|
||||
time.sleep(5)
|
||||
code, d = w.ctl("GET", path)
|
||||
if isinstance(d, dict) and d.get("done"):
|
||||
return d
|
||||
return {"timeout": True}
|
||||
|
||||
|
||||
def orm_has(user):
|
||||
out = g(f"docker exec adventurelog python manage.py shell -c "
|
||||
f"\"from django.contrib.auth import get_user_model; print('ANS=' + str(get_user_model().objects.filter(username='{user}').exists()))\" 2>&1")
|
||||
m = re.search(r"ANS=(True|False)", out)
|
||||
return m.group(1) if m else "unreadable: " + out[-160:]
|
||||
|
||||
|
||||
def admin_login(user, pw):
|
||||
"""The front door, the household's route (box_walk.app_curl carries the setup gate): GET /admin/login/ for the
|
||||
form and its csrftoken cookie, POST the form (the password on stdin, never an argument). Success = 302 to /admin/."""
|
||||
rc, code, body = w.app_curl(SUB, "/admin/login/", "-D", "-")
|
||||
tok = re.search(r'name="csrfmiddlewaretoken" value="([^"]+)"', body)
|
||||
ck = re.search(r"(?im)^set-cookie:\s*csrftoken=([^;]+)", body)
|
||||
if not tok or not ck:
|
||||
return f"no login form ({code})"
|
||||
form = urllib.parse.urlencode({"csrfmiddlewaretoken": tok.group(1), "username": user, "password": pw, "next": "/admin/"})
|
||||
rc, code, body = w.app_curl(SUB, "/admin/login/?next=/admin/", "-H", f"Cookie: csrftoken={ck.group(1)}",
|
||||
"-H", "Content-Type: application/x-www-form-urlencoded",
|
||||
"-H", f"Referer: https://{SUB}.{w.DOMAIN}/admin/login/", method="POST", data=form)
|
||||
return code + (" (logged in)" if code == "302" else " (refused)")
|
||||
|
||||
|
||||
def main():
|
||||
w.say = say
|
||||
w.login()
|
||||
user, pw = "cc" + secrets.token_hex(3), "Cc-" + secrets.token_hex(12)
|
||||
try:
|
||||
ok = w.deploy(APP, SUB) and w.wait_app(SUB, "/", tries=120, delay=5)
|
||||
say("1. adventurelog installed and answering:", ok)
|
||||
out = g(f"docker exec -e DJANGO_SUPERUSER_PASSWORD='{pw}' adventurelog python manage.py createsuperuser --noinput "
|
||||
f"--username {user} --email {user}@cc.invalid 2>&1 | tail -1")
|
||||
say("2. createsuperuser:", out[:120], "| in the ORM:", orm_has(user))
|
||||
say(" front door before:", admin_login(user, pw))
|
||||
code, d = w.ctl("POST", "/api/export/start", {"stack_name": APP, "dest_drive": DRIVE, "stop_app": False})
|
||||
if code != "200":
|
||||
raise SystemExit(f"INSTRUMENT: export start refused {code} {d}")
|
||||
j = job("/api/export/status")
|
||||
say("3. export:", j.get("error") or "ok", j.get("output_size"), j.get("output_path"))
|
||||
fab = j.get("output_path", "")
|
||||
say(" the bundle's database:", g(f"tar -tzf {fab} | grep -E 'database/|manifest' | head -4"))
|
||||
g("docker exec adventurelog python manage.py shell -c \"from django.contrib.auth import get_user_model; "
|
||||
f"get_user_model().objects.filter(username='{user}').delete()\" >/dev/null 2>&1")
|
||||
say("4. the user deleted in the live app (so only the import can bring it back) — in the ORM:", orm_has(user))
|
||||
code, d = w.ctl("POST", "/api/export/import", {"path": fab})
|
||||
if code != "200":
|
||||
raise SystemExit(f"INSTRUMENT: import start refused {code} {d}")
|
||||
j = job("/api/export/import/status")
|
||||
say("5. import:", j.get("error") or "ok", "| steps:", [(s.get("label"), s.get("status")) for s in j.get("steps", [])])
|
||||
say(" app answers:", w.wait_app(SUB, "/", tries=120, delay=5))
|
||||
say(" R-1187 g — the user back in the ORM:", orm_has(user))
|
||||
say(" R-1187 g — the front door after import:", admin_login(user, pw))
|
||||
say(" the import's DB line:", g("docker logs --since 15m felhom-controller 2>&1 | grep -E 'restoring database|found DB service|DB dump imported|psql' | tail -3 | cut -c1-200"))
|
||||
finally:
|
||||
sw.remove(APP, keep=False, say=say)
|
||||
say("6. leftovers:", sw.leftovers(APP, SUB))
|
||||
say(" bundles removed:", g(f"rm -f {DRIVE}/exports/adventurelog*.fab; ls {DRIVE}/exports 2>/dev/null | grep -c adventurelog"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -193,7 +193,7 @@ stopping line that lies.
|
||||
| **R-1184** | Backup & restore | P3 | **A cut-off shared-folder restore can be „placed" and reported restored while files are missing; the download is then deleted.** Readiness is only „the scratch folder is not empty" (`internal/backup/shares_restore.go:131-140`) — the R-358 class fixed for apps, not for shares; 30-min download window (`web/offbox_handlers.go:716`); fixed 2 GiB free-space gate that fails open (`:107`). CONFIRMED in code. Data-loss hunt 2026-10-20 (C1 reviewer 1, R3). | OPEN | — | Fix: a completion marker after restic succeeds, as for apps | CC |
|
||||
| **R-1185** | Backup & restore | P3 | **A database dump that fails its own content check has already replaced last night's good dump, and the second drive copies it an hour later.** `appbackup/dbdump.go:388-422`: size check, rename over the good file (`:399`), then `ValidateDump` (`:419`). Volume tars are promoted with no content check either (`backup.go:902`). Tier 1 is one point per app by design (07 §6.1); the ORDER is the defect. PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/`, a docker stub prints a header-only dump): `ValidateDump` says invalid („dump file too small"), `DumpOneTo` returns no error, and last night's good dump (with its `CREATE TABLE`) is gone from disk. Data-loss hunt 2026-10-20 (D5). | OPEN | — | Fix: validate the temp file before the rename; keep the last good dump on a failed check | CC |
|
||||
| **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC |
|
||||
| **R-1187** | Backup & restore | P2 | **A `.fab` export or import can lose or mix household data — code-confirmed paths from the data-loss hunt:** (a) an export silently drops a whole drive folder that holds a symlink and still ends „done” (`appexport/fabplan.go:292-310`, `export.go:712`; a FIFO also hangs the export on `os.Open`); (b) the import handles only folders and regular files: no owners set, symlinks dropped (`restore.go:1138-1183`); (c) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** an import over an installed app that is not running merged the old data with the bundle's (`docker cp` never deletes) — reproduced on 9202 before the fix (stopped: a marker written after the export SURVIVED the import; running control: replaced), now refused with a Hungarian sentence and the app untouched; red-proved (`r1187c_test.go`), proven on 9202 rc.2 (`audits/night-2026-10-22/P3-candidates/r1187c-9202.txt`); (d) PLAUSIBLE: no restore-busy check in `StartImport`, no undo, no record (`restore.go:181-186`); (f) the bundle's HDD_PATH is trusted, an empty one gives a relative path (`restore.go:589-631`); (g) the DB import misses postgis images (adventurelog, dawarich: `classifyDBImage`) and runs psql without ON_ERROR_STOP (`restore.go:1047-1056, 933-943`; the dump is `--clean --if-exists`, so ON_ERROR_STOP is safe). (e) was R-1191. Research `audits/night-2026-10-22/P3-candidates/analysis.md`; first list `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | OPEN — **2026-10-23: (c) and the C3/N9 refusal texts released in controller 0.340.0;** (a), (b), (f), (g) and the two import paths of C2 stay open. OPEN — **2026-10-22 night: re-ranked P3→P2** ((a) and (b) lose household files on a move to a new box with no error). (c) fixed in code; (a), (b), (f), (g) read in code, not live-proven. **2026-10-23 night — fresh review (`REPORT-night-2026-10-23.md`, C2, C3):** two more import paths still merge — a running app whose `RemoveStackVolumes` fails falls back to `StopStack` and goes on (`appexport/restore.go:375-380`), and `restoreHDDData` extracts drive-folder tars over the existing folder without clearing it (`restore.go:602-655`) for running and stopped apps alike. Both exist in 0.339.0 — not a hold for 0.340.0. The (c) refusal also catches an unhealthy or restarting app (`IsStackRunning`: running/degraded only); its text now says so and names removal (controller `943c517`). Recovery code: fix only with a red-proof + a 9202 proof. | — | Fix (a)+(b) with a red-proof and a 9202 proof (capture code — the night rule); (g) small fix with a 9202 postgis app; (d) after (a) | CC |
|
||||
| **R-1187** | Backup & restore | P2 | **A `.fab` export or import can lose or mix household data — code-confirmed paths from the data-loss hunt:** (a) an export silently drops a whole drive folder that holds a symlink and still ends „done” (`appexport/fabplan.go:292-310`, `export.go:712`; a FIFO also hangs the export on `os.Open`); (b) the import handles only folders and regular files: no owners set, symlinks dropped (`restore.go:1138-1183`); (c) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** an import over an installed app that is not running merged the old data with the bundle's (`docker cp` never deletes) — reproduced on 9202 before the fix (stopped: a marker written after the export SURVIVED the import; running control: replaced), now refused with a Hungarian sentence and the app untouched; red-proved (`r1187c_test.go`), proven on 9202 rc.2 (`audits/night-2026-10-22/P3-candidates/r1187c-9202.txt`); (d) PLAUSIBLE: no restore-busy check in `StartImport`, no undo, no record (`restore.go:181-186`); (f) the bundle's HDD_PATH is trusted, an empty one gives a relative path (`restore.go:589-631`); (g) the DB import misses postgis images (adventurelog, dawarich: `classifyDBImage`) and runs psql without ON_ERROR_STOP (`restore.go:1047-1056, 933-943`; the dump is `--clean --if-exists`, so ON_ERROR_STOP is safe). (e) was R-1191. Research `audits/night-2026-10-22/P3-candidates/analysis.md`; first list `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | **VERIFY — fixed on `main`: (a), (b), (d), (f), (g) + two import defects found by the live proof; waits for the controller release** — 2026-10-26 night: controller `dfd0f8e`, `106325d`, `f24233c`, `cccb9ae` (three rounds of the automated security review on the link handling: writes only through `os.Root`, links only downward), `a3ed63e` (the import's compose runs get the controller's path variables; a warning is never the container id), `90a2530` (the database loads through `appbackup.ImportDump` — every MariaDB 11+ import had timed out on `mysqladmin`). Live on 9202 (Grimmory: FIFO refused naming the path, owner/mode/inside link back, outside link refused; the import now succeeds and the app answers; AdventureLog: a postgis user back after the import) — `audits/night-fixes-2026-10-26/`. (d), (f): unit tests. Still open from the row: C2's two merge paths of an import over an existing app. **Earlier:** OPEN — **2026-10-23: (c) and the C3/N9 refusal texts released in controller 0.340.0;** (a), (b), (f), (g) and the two import paths of C2 stay open. OPEN — **2026-10-22 night: re-ranked P3→P2** ((a) and (b) lose household files on a move to a new box with no error). (c) fixed in code; (a), (b), (f), (g) read in code, not live-proven. **2026-10-23 night — fresh review (`REPORT-night-2026-10-23.md`, C2, C3):** two more import paths still merge — a running app whose `RemoveStackVolumes` fails falls back to `StopStack` and goes on (`appexport/restore.go:375-380`), and `restoreHDDData` extracts drive-folder tars over the existing folder without clearing it (`restore.go:602-655`) for running and stopped apps alike. Both exist in 0.339.0 — not a hold for 0.340.0. The (c) refusal also catches an unhealthy or restarting app (`IsStackRunning`: running/degraded only); its text now says so and names removal (controller `943c517`). Recovery code: fix only with a red-proof + a 9202 proof. | — | Fix (a)+(b) with a red-proof and a 9202 proof (capture code — the night rule); (g) small fix with a 9202 postgis app; (d) after (a) | CC |
|
||||
| **R-1188** | Backup & restore | P4 | **Restore-path candidates from the data-loss hunt — what is left is unlikely or a sentence:** (b) a failed `volume rm` is ignored before the untar, which then lands in the old volume (`backup/restore.go:218`; `StopStack` is `compose down`, so the rm rarely fails); (d) the full off-site download stops at 30 min per press and the household is not told to press again (`web/offbox_handlers.go:455`; the scratch is kept, so a second press probably continues — not measured); (e) place and full restore re-read the LATEST snapshot, not the downloaded one (`offbox_restore.go:675-679`, `_ = id`). Closed parts: (a) with R-1195 (controller 0.338.0), (c) in controller 0.337.0 (`TestR1188c_ALeftoverTempDoesNotBlockTheRetry`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — **2026-10-22 night: re-ranked P3→P4**: (a) and (c) are shipped; the rest is low-chance or a sentence. | — | (d): measure a second press on a large download on 9202, then write the sentence; (b), (e): unit tests when the file is next touched | CC |
|
||||
| **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt — what is left:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it: the agent turns a future archive time into a negative age (`localapi/server.go:1179-1192`); half-live 2026-10-21 (B6 #8, `s8-wrong-clock/README.md`: a local archive dated 2028-10-21); agent code, so it rides an agent release and the operator's vouch; (c) the whole-guest quiesce waits for the catch-up and the update leg, not for a restore (`quiesce.go:213-275, 565`); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — a design question; (g) spring forward 2027-03-28 moves 02:30 onto the 03:30 second copy; (h) self-update busy check before the pull only (`updater.go:532-545, 764`; not re-read); (i) a double-clicked restore shows failed while it runs (check and begin not atomic, `web/handlers.go:1860, 1881`). **Moved out 2026-10-22 night:** (b) the skipped nights after an off-cycle run were FIXED by R-899 (controller 0.304.0; demo-hp's journal shows a night run after each of three off-cycle runs since), the creep → R-1221; (d) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** the catch-up now runs the off-site proof and the integrity check once after its legs, the false comment corrected; red-proved (`nightchain/r1189d_test.go`, `cmd/controller/r1189d_wiring_test.go`); (e) → R-1222 (hub). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — **2026-10-23: (d) released in controller 0.340.0;** (a), (c), (f), (g) stay as written. OPEN — 2026-10-22 night: (b) half closed, (d) fixed in code, (e) split out. | — | (a) with the next agent release (a unit test with a future archive); (c) a 9202 recipe; (g) a unit test with the 2027 clock; (f) to the operator at the next design review | CC |
|
||||
| **R-1190** | Backup & restore | P3 | **Key and retention candidates from the data-loss hunt — what is left:** (c) PLAUSIBLE: after a reuse-path reinstall the new PBS key K is never escrowed and nothing compares fingerprints (`felhom-pbs-apply:99`, `escrow_confirm.go:182-205`); (d) whole-guest off-site copies run before any escrow exists — a design question against `07` §6.1's Tier-3 gate (the operator's); (e) the escrow PUT is two statements; (f) the operator's off-site re-issue resets the provider password before saving it (hub `offsite.go:217-245`); (h) a unit restore can point an app back at an old drive and F5 then prunes the current drive's unit (`backup.go:1517-1574`) — (e), (f), (h) not re-read. **Moved out 2026-10-22 night:** (a) → R-1218, (b) → R-1219, (g) → R-1220. Evidence `audits/dataloss-hunt-2026-10-20/C1-keys-candidates.md`, `C1-delete-retention-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN | — | (c) a read-only fingerprint compare on a reinstalled test box; (e), (f), (h) re-read, then unit tests; (d) to the operator at the next design review | CC |
|
||||
@@ -201,14 +201,14 @@ stopping line that lies.
|
||||
| **R-1200** | Backup & restore | P3 | **A box installed in the afternoon gets no whole-box backup on its first night: the install's own backup starts the 24 h clock, so the family's whole first day reaches a whole-box copy only on the SECOND night (~38 h after install).** Rehearsal 2026-10-20 B5 (`audits/rehearsal-2026-10-20/B5-night/README.md`): Tester4's install backup at 12:09Z; at the 02:30–06:30Z window the agent answered „within cadence window" (`felhom-agent internal/localapi/server.go:1187`, `age >= tier.Cadence`, default 24 h). R-899 (closed) fixed the same shape for a daytime PRESS only. The other tiers did run that night (database copy, second drive, off-site — all PASS), so app data was covered; the whole-box (DR) tier was not. | **OPEN** — a design question: should the first scheduled night always run (like a press), or is one tier enough on night one? | — | decide; if „always run": treat the install backup like a press in the due rule, with a test on the first night | CC |
|
||||
| **R-1206** | Backup & restore | P3 | **Backups written while the box's clock ran ahead stay „the latest" after the clock is fixed.** The rehearsal's scenario #10 (clock set to 2026-10-28) left off-site snapshots dated 2026-10-28; on 2026-10-21 the rebuilt box's recovery page listed every app's „Legutóbbi mentés: 2026. 10. 28. 07:39", and the full restore said „mentés: 2026. 10. 28. 07:37" — a date in the future (`audits/fixes-2026-10-21/P1-live/07-unlock-right.txt`, `11-nextcloud-full-restore.txt`). restic orders snapshots by their own time, so the future ones are „newest" to every retention rule for a week. NOT measured: whether the box's keep-daily prune then drops a real snapshot. Related R-1202, R-1204 (clock). | NARROWED — the NAS tier only. **2026-10-22 night (read, `backup/offbox_window.go:22-24, 130-137`):** on the Felhom off-site tier no real snapshot is lost — the guard refuses a prune while any snapshot is future-dated (R-822, by design). The NAS tier prunes without that guard: open. **2026-10-25:** the kept-data choice no longer takes such a future-dated copy as the newest (R-1271, controller `bd9f4f7`); retention is unchanged — this row. | — | A pure unit test with the policy simulator on the NAS tier and one future-dated snapshot; if a real one is dropped, give the NAS tier the same guard | CC |
|
||||
| **R-1212** | Backup & restore | P4 | **Two household texts on the backup path do not follow the box's language:** the share-restore summary (`SharesRestoreResult.FlashMessage`, `controller/internal/backup/shares_restore.go`) is built from Hungarian Go literals, and the off-site `LastError` for an interrupted run is the Hungarian literal „megszakadt futás (a vezérlő újraindult futás közben)" (`controller/internal/backup/backup.go`). An English household reads Hungarian there. Found by the night helper 2026-10-21. | OPEN | — | Bundle keys for both (rendered in the box language when written, R-557), with parity fixtures | CC |
|
||||
| **R-1218** | Backup & restore | P2 | **A recovery ceremony done while off-site is switched off (or cleared) seals no off-site repository password, and the hub then demotes the GOOD older package to operator-only custody: the household's own recovery code no longer opens its off-site copy.** REAL in code (night 2026-10-22, read; split from R-1190 (a)): `web/escrow_handlers.go:217-224` stages the password only if `OffboxConfigured()` (`backup/offbox.go:695-710`), false when the target is cleared or disabled — although `ClearOffboxTarget` keeps the password file on purpose (`offbox_clear.go` header); the agent then seals no password (`cmd/felhom-agent/main.go:2981-3003`) and uploads sha ""; hub `SaveHostEscrow` sees `curSHA != ""` and moves the good package to retained custody; `OffsiteRecoveryOffer` reads "" as „covers no key” (`offbox.go:1855-1873`) and says nothing. Evidence `audits/night-2026-10-22/P3-candidates/analysis.md` §R-1190(a). | OPEN — proposed fix, not built (key-handling code: a fix needs a red-proof AND a live proof, and 9202 has no hub to prove it with) | — | Fix (~6 lines): stage when the password FILE exists (`PushOffboxPasswordForEscrow`, `offbox.go:800`, reads only the file). Red-proof: a web test that the stage runs with off-site cleared; a hub store test pinning „abc”→"" ⇒ superseded (and refuse that demotion). Live proof on a test box with a hub (demo-hp 9201) | CC |
|
||||
| **R-1219** | Backup & restore | P2 | **During a recovery offer the backups page hides the „create your recovery code” ceremony (it would replace the key that opens the old off-site copy), but the global top bar still links to it and the start handler does not refuse.** REAL in code (night 2026-10-22, read; split from R-1190 (b)): `web/escrow_banner.go:23-35` ignores `recoveryOffer()`; `templates/backups_remote.html:144-160` hides it and its own comment names the harm (R-199); `escrow_handlers.go:180-245` starts anyway; reachable in offer shape (c), `offbox.go:1866-1870`. Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — proposed fix, not built (key-handling code) | — | Fix: the bar hides its link during an offer, and the start handler answers 409 with a Hungarian sentence; red-proof with a handler test; live proof with an offer state on a test box with a hub | CC |
|
||||
| **R-1220** | Backup & restore | P2 | **„Megőrzött adatok” (kept data) can list an INSTALLED app's live folder as kept — and offer to delete it — when that app's app.yaml cannot be read.** REAL in code (night 2026-10-22, read; split from R-1190 (g)): a nil AppConfig drops the app from `liveDriveBinds` (`stacks/kept.go:220-235, 336-360, 372-389`); a parse error gives nil (`deploy.go:1113-1125`). Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — proposed fix, not built (delete-path code) | — | Fix: an app whose app.yaml is unreadable counts as live (fail closed); its folders are never listed as kept. Red-proof: a unit test with a broken app.yaml; 9202 proof with a throwaway app | CC |
|
||||
| **R-1218** | Backup & restore | P2 | **A recovery ceremony done while off-site is switched off (or cleared) seals no off-site repository password, and the hub then demotes the GOOD older package to operator-only custody: the household's own recovery code no longer opens its off-site copy.** REAL in code (night 2026-10-22, read; split from R-1190 (a)): `web/escrow_handlers.go:217-224` stages the password only if `OffboxConfigured()` (`backup/offbox.go:695-710`), false when the target is cleared or disabled — although `ClearOffboxTarget` keeps the password file on purpose (`offbox_clear.go` header); the agent then seals no password (`cmd/felhom-agent/main.go:2981-3003`) and uploads sha ""; hub `SaveHostEscrow` sees `curSHA != ""` and moves the good package to retained custody; `OffsiteRecoveryOffer` reads "" as „covers no key” (`offbox.go:1855-1873`) and says nothing. Evidence `audits/night-2026-10-22/P3-candidates/analysis.md` §R-1190(a). | **OPEN — fixed on `main` (both halves), red-proved; LIVE PROOF PENDING** — controller `1170b48` (stage when the password FILE exists) + hub `09b5f02f` (a keyed package is never replaced by a key-less one: 409). The live shape needs off-site switched OFF on a hub box: the household cannot switch a hub tier off (only per-app switches; a clear is refused by design), so it needs the operator's customer-page change on Tester 1 during Tuesday's attended block — otherwise both halves are REVERTED before the release (rule 2). **Earlier:** OPEN — proposed fix, not built (key-handling code: a fix needs a red-proof AND a live proof, and 9202 has no hub to prove it with) | — | Fix (~6 lines): stage when the password FILE exists (`PushOffboxPasswordForEscrow`, `offbox.go:800`, reads only the file). Red-proof: a web test that the stage runs with off-site cleared; a hub store test pinning „abc”→"" ⇒ superseded (and refuse that demotion). Live proof on a test box with a hub (demo-hp 9201) | CC |
|
||||
| **R-1219** | Backup & restore | P2 | **During a recovery offer the backups page hides the „create your recovery code” ceremony (it would replace the key that opens the old off-site copy), but the global top bar still links to it and the start handler does not refuse.** REAL in code (night 2026-10-22, read; split from R-1190 (b)): `web/escrow_banner.go:23-35` ignores `recoveryOffer()`; `templates/backups_remote.html:144-160` hides it and its own comment names the harm (R-199); `escrow_handlers.go:180-245` starts anyway; reachable in offer shape (c), `offbox.go:1866-1870`. Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | **VERIFY — fixed on `main`, waits for the controller release** — controller `1170b48` (the top bar hides the ceremony during a recovery offer; the start handler answers 409 `api.escrow.recovery_offer_open`), red-proved with a real session and a no-offer control. The offer state cannot be staged without hand-set state, so the handler test is the proof (the brief's allowance). **Earlier:** OPEN — proposed fix, not built (key-handling code) | — | Fix: the bar hides its link during an offer, and the start handler answers 409 with a Hungarian sentence; red-proof with a handler test; live proof with an offer state on a test box with a hub | CC |
|
||||
| **R-1220** | Backup & restore | P2 | **„Megőrzött adatok” (kept data) can list an INSTALLED app's live folder as kept — and offer to delete it — when that app's app.yaml cannot be read.** REAL in code (night 2026-10-22, read; split from R-1190 (g)): a nil AppConfig drops the app from `liveDriveBinds` (`stacks/kept.go:220-235, 336-360, 372-389`); a parse error gives nil (`deploy.go:1113-1125`). Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | **VERIFY — fixed on `main`, waits for the controller release** — 2026-10-26 night: controller `7e15a37` (an app whose `app.yaml` exists but does not parse counts as installed; its folders are never kept data), red-proved; live on 9202 (the kept-data page did not list the folder) — `audits/night-fixes-2026-10-26/`. **Earlier:** OPEN — proposed fix, not built (delete-path code) | — | Fix: an app whose app.yaml is unreadable counts as live (fail closed); its folders are never listed as kept. Red-proof: a unit test with a broken app.yaml; 9202 proof with a throwaway app | CC |
|
||||
| **R-1221** | Backup & restore | P4 | **The nightly whole-box backup starts ~1.5–5 min later each night, until one night is skipped (after ~70–80 nights with no press between).** REAL, low harm (night 2026-10-22; split from R-1189 (b)): agent `localapi/server.go:1184` is `age >= Cadence` with the age counted from the archive START; the controller polls every 5 min (`quiesce.go:150`) and the archive starts after the apps stop. Measured on demo-hp: 04:31:07 → 04:33:18 → 04:36:47 → 04:38:14, and 04:33:37 → 04:38:44 → 04:42:40. At W+6h one night is skipped (44 h is below the 48 h valve, `quiesce.go:807-818`); the hub's 26 h line (`deadline.go:29`) mails the operator. Any manual or off-cycle run resets the drift. Evidence `audits/night-2026-10-22/P3-candidates/demohp-whole-box-completions.txt`. | OPEN | — | Fix: inside the night window a tier is due at `age >= cadence - 4h`; the agent answers due-ness, so it is agent code (a unit test with an injected clock) and rides an agent release | CC |
|
||||
| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN — **2026-10-23 — D35 ruled YES, after the release week (operator):** serve the website and the installer from Cloudflare like the ISO. Dated: start on or after 2026-10-30 (DUE-CHECKS). OPEN | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | CC (ruled yes 2026-10-23, D35) |
|
||||
| **R-1229** | Backup & restore | P4 | **ArgoCD's per-app auto-sync settings live only on the cluster, not in git.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G9): the Applications differ (`enabled` true/false, prune, selfHeal); a rebuilt cluster from `homelab-manifests` would not have them. | OPEN | — | Write each Application's sync policy into `homelab-manifests` (a homelab change, the operator's yes for the commit that changes live behaviour) | operator |
|
||||
| **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC |
|
||||
| **R-1232** | Backup & restore | P2 | **A reinstalled box may take its first whole-box copy to ep0 up to a week late: the old box's copy in the same household space seems to count as its last one.** Test night 2026-10-23 (`audits/night-2026-10-23/P4-testnight/`): the new Tester4 box (guest 9201) had its PBS tier applied at 19:28 UTC, but no whole-box run to ep0 came all night; the only archive in the household's ep0 space is the rehearsal box's `ct/9201` of 2026-10-20 (same guest number), and the tier's cadence is 168 h. The agent's due-check reads the newest archive for that vmid on the tier's storage (`localapi/server.go` `newestArchiveOn`) — so the old box's copy reads as „2.6 days old, not due”. If so, the ep0 copy is the OLD box's for up to a week. PLAUSIBLE — read from the code and the night's silence, not proven by a log line (the controller logs the due-check only at debug). | OPEN | — | Prove: read `GET /backup/tiers` for the box (the agent's answer per tier) on the next reinstall; fix: a tier's „last backup” counts only archives this host wrote (or after the host's enrollment); an agent test with a foreign archive in the namespace | CC |
|
||||
| **R-1232** | Backup & restore | P2 | **A reinstalled box may take its first whole-box copy to ep0 up to a week late: the old box's copy in the same household space seems to count as its last one.** Test night 2026-10-23 (`audits/night-2026-10-23/P4-testnight/`): the new Tester4 box (guest 9201) had its PBS tier applied at 19:28 UTC, but no whole-box run to ep0 came all night; the only archive in the household's ep0 space is the rehearsal box's `ct/9201` of 2026-10-20 (same guest number), and the tier's cadence is 168 h. The agent's due-check reads the newest archive for that vmid on the tier's storage (`localapi/server.go` `newestArchiveOn`) — so the old box's copy reads as „2.6 days old, not due”. If so, the ep0 copy is the OLD box's for up to a week. PLAUSIBLE — read from the code and the night's silence, not proven by a log line (the controller logs the due-check only at debug). | **VERIFY — fixed on `main`, waits for the agent release (the operator's vouch)** — 2026-10-26 night: agent `3911cc6` (the due-check counts only archives with this storage's key, the R-727 discriminator), red-proved; live on a fresh BYO Tester4 box (VM 9375): the released agent said `due: false` (the old box's archive, 6.3 days), the fixed one `due: true / absent`, and the first whole-box copy ran 4 minutes later — `audits/night-fixes-2026-10-26/`. Hub host `Tester4-b5dbbd` to delete (operator). **Earlier:** OPEN | — | Prove: read `GET /backup/tiers` for the box (the agent's answer per tier) on the next reinstall; fix: a tier's „last backup” counts only archives this host wrote (or after the host's enrollment); an agent test with a foreign archive in the namespace | CC |
|
||||
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
|
||||
| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
|
||||
| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC |
|
||||
@@ -269,7 +269,7 @@ stopping line that lies.
|
||||
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
|
||||
| **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC |
|
||||
| **R-974** | Box system & updates | P3 | **A compromised host agent can likely become root on the host through the root wrapper that applies PBS settings: the wrapper accepts a folder path inside the agent's own writable tree and then acts on it as root.** Static (the symlink behaviour was reasoned, not run). Same class as R-861. Fix: a fixed root-owned folder, and refuse symlinks before any root action. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix the wrapper with a test on a throwaway box | CC |
|
||||
| **R-972** | Box system & updates | P2 | **The shipped controller and hub programs are built with Go 1.24.13 and golang.org/x/crypto v0.31.0, and the agent with Go 1.26.0, so dozens of fixed Go security bugs (including html/template escaping fixes) are still in what runs on boxes and the hub.** Go 1.24 has had no security fixes since Go 1.26 was released (Go supports the two newest major releases: https://go.dev/doc/devel/release#policy); the scanners name fixes in Go 1.26.9 and 1.27.2. Also: the controller image's Debian 12 base has 12 fixable critical packages, and the Samba image has 48 fixable high ones. Fix: build all three on a supported, patched Go; update x/crypto; rebuild the base images. Scanner output: `audits/security-review-2026-10-12/scanners/`. | **NARROWED** 2026-10-16 (burndown 2): **controller v0.322.0 released** on Go 1.27.2 + x/crypto v0.58.0, runtime debian:12.13-slim + `apt-get upgrade` (Debian 13 deliberately not: restic 0.18, R-1105) — scanner: critical 16 → 4, fixable criticals and highs in the OS 0; govulncheck 62 → 1 (module-level, `openpgp` not linked). **Hub v0.156.0 deployed** 2026-10-16 19:26 on the operator's yes (Go 1.27.2, alpine 3.24.2: 0 critical, 0 high); Synced/Healthy, reports from all three boxes after the start, the floor reads 0.322.0. **Agent: Go 1.27.2 committed (`37b99d7`), tested, NOT released** — the next vouched agent release carries it. Open: the Samba image (48 fixable high, not rebuilt); docker-compose plugin's own Go (4 high, upstream). Evidence `audits/burndown2-2026-10-16/P4/` **2026-10-16 (evening): the Samba part is DONE in controller v0.323.0** — felhom-samba 1.2.0 on Alpine 3.24.2 (0 critical / 0 high, was 48 fixable high), proven on demo-hp with the same household password (smbclient, nmbd, mDNS, WS-Discovery); on the three test boxes now, on every box with the global floor (R-1105's night check). The controller's Debian 13 OS part: 1 critical / 70 high, none fixable. Left: docker-compose plugin's own Go (4 high, upstream); the agent release; restic's binary is R-1107. **2026-10-17 (evening): agent v0.159.1 (Go 1.27.2) RELEASED and running on demo-hp, Tester 1 and demo-felhom** (signed jobs on the operator's yes); waiting for the operator's vouch — then only the upstream compose plugin is left (close then, recorded). | the next vouched agent release | Close when the global floor carries v0.323.0 and the next vouched agent release carries Go 1.27.2 (compose plugin: upstream, recorded) | CC |
|
||||
| **R-972** | Box system & updates | P2 | **The shipped controller and hub programs are built with Go 1.24.13 and golang.org/x/crypto v0.31.0, and the agent with Go 1.26.0, so dozens of fixed Go security bugs (including html/template escaping fixes) are still in what runs on boxes and the hub.** Go 1.24 has had no security fixes since Go 1.26 was released (Go supports the two newest major releases: https://go.dev/doc/devel/release#policy); the scanners name fixes in Go 1.26.9 and 1.27.2. Also: the controller image's Debian 12 base has 12 fixable critical packages, and the Samba image has 48 fixable high ones. Fix: build all three on a supported, patched Go; update x/crypto; rebuild the base images. Scanner output: `audits/security-review-2026-10-12/scanners/`. | **NARROWED — Go 1.27.3 committed in all three, waits for the releases** — 2026-10-26 night: agent `fbbb168`, controller `2560cf1` (+ builder `golang:1.27.3-trixie`), hub `d625a143` (+ `golang:1.27.3-alpine3.24`); govulncheck 1 → 0 called vulnerabilities each (GO-2026-6809, `crypto/x509`); x/crypto v0.58.0 already newest; full tests green — `audits/r972-go1273-2026-10-26/`. **Earlier:** **NARROWED** 2026-10-16 (burndown 2): **controller v0.322.0 released** on Go 1.27.2 + x/crypto v0.58.0, runtime debian:12.13-slim + `apt-get upgrade` (Debian 13 deliberately not: restic 0.18, R-1105) — scanner: critical 16 → 4, fixable criticals and highs in the OS 0; govulncheck 62 → 1 (module-level, `openpgp` not linked). **Hub v0.156.0 deployed** 2026-10-16 19:26 on the operator's yes (Go 1.27.2, alpine 3.24.2: 0 critical, 0 high); Synced/Healthy, reports from all three boxes after the start, the floor reads 0.322.0. **Agent: Go 1.27.2 committed (`37b99d7`), tested, NOT released** — the next vouched agent release carries it. Open: the Samba image (48 fixable high, not rebuilt); docker-compose plugin's own Go (4 high, upstream). Evidence `audits/burndown2-2026-10-16/P4/` **2026-10-16 (evening): the Samba part is DONE in controller v0.323.0** — felhom-samba 1.2.0 on Alpine 3.24.2 (0 critical / 0 high, was 48 fixable high), proven on demo-hp with the same household password (smbclient, nmbd, mDNS, WS-Discovery); on the three test boxes now, on every box with the global floor (R-1105's night check). The controller's Debian 13 OS part: 1 critical / 70 high, none fixable. Left: docker-compose plugin's own Go (4 high, upstream); the agent release; restic's binary is R-1107. **2026-10-17 (evening): agent v0.159.1 (Go 1.27.2) RELEASED and running on demo-hp, Tester 1 and demo-felhom** (signed jobs on the operator's yes); waiting for the operator's vouch — then only the upstream compose plugin is left (close then, recorded). | the next vouched agent release | Close when the global floor carries v0.323.0 and the next vouched agent release carries Go 1.27.2 (compose plugin: upstream, recorded) | CC |
|
||||
| **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **2026-10-24 02:4xZ dated check read (weekend-night session, read only):** Tester 1 (8 GB host, guest 4 GB, controller 0.341.0) ran its 03:30 copy at 01:30:00–01:30:13Z (3 apps, 157 MB + 86 MB + 16 KB; no „capped copy container did not start" WARN, no leftover `felhom-tier2-copy-*`); the guest cgroup's memory „full" stall counter on the host (`/sys/fs/cgroup/lxc/9201/memory.pressure`) read **total 444 µs** over the whole 11.1 h since the host started (15:32Z), so „full" stayed far under 20 % (20 % of 10 s = 2 s); 0 OOM kills. **Not shown:** that the copy ran in the capped container (the INFO log does not name it; Docker keeps only ~200 events, from 02:36Z). **Weak load:** 0.24 GB, not the soak's large copy — so the row stays WATCHING; close after a night whose copy moves ≥ 1 GB, read from the same counter. WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC |
|
||||
| **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator |
|
||||
| **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
|
||||
@@ -287,7 +287,7 @@ stopping line that lies.
|
||||
| **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). | **VERIFY** — RELEASED in controller v0.344.0 (global floor 2026-10-26 09:07:47Z, every live box on 0.344.0); the live test with a seeding box is Tuesday 10:00–16:00 (R-1250's dated check). **Earlier:** **VERIFY** — fixed on main 2026-10-25 night (controller `a79a292`), waits for the controller release (before any household gets agent 0.165.0 by signed update): every running seeder held, a removed app's record dropped, qBittorrent's slot named (`up_limit`/`alt_up_limit`). Red-proofs `audits/night-2026-10-25/redproof-r1261.txt`. **Live on the night's rig** (BYO VM, agent 0.165.0, rc.1, two seeders): both held, 94.8 Mbit/s, both limits back; a speed-mode switch mid-test left no 1 KiB/s (`rig/hold-watch*.txt`); the removed-app case by test only. **Earlier state:** OPEN — **now in controller 0.343.0 (released 2026-10-25); the three edges act only on a box whose agent is ≥ 0.165.0 (it alone reports `waiting_for_quiet`) — fix before any household gets agent 0.165.0 by signed update** | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC |
|
||||
| **R-1262** | Monitoring & notifications | P3 | **The upload watch treats ANY viewer as an explanation for any amount of upload, so a hacked Plex can send freely while one family member watches.** Found 2026-10-24 (SEC-1 X1): `felhom-controller/controller/internal/upwatch/watch.go:275,299`; `08` §9 states the rule, its limits list did not name this case. | OPEN | — | propose a cap (viewers × a bitrate ceiling) for the operator's R-1244 design; `08`'s limits list names the case meanwhile | CC |
|
||||
| **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC |
|
||||
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
|
||||
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING — DESIGN WRITTEN 2026-10-26 night (the brief's Part 7: build only if time is left; it was not built — the line test and the attended release came first).** The host-side counter EXISTS and is readable (read on demo-hp 2026-10-26): `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex id>.scope/memory.events`, line `oom_kill N`. **Design:** (1) agent: a read-only collector (no root CLI — a sysfs read under the agent's own rights; if not readable, the existing sudoers fence gets one read-only `Cmnd_Alias`) that, for each guest in the `felhom` pool, globs those scopes and keeps `{container_id: oom_kill}`; served on the guest's own local API as `GET /guest/oom-kills` (`withGuest`, the guest's containers only). (2) controller: the 30-s dead-app scan asks once per pass; a RISE of a container's counter = an OOM kill whatever Docker's `OOMKilled` says → the existing `app_oom` line (R-636's per-run grain), `source: cgroup` in the details. An agent without the route = today's behaviour. (3) Tests: a fake cgroup tree (agent), a rising counter → one `app_oom` (controller, red-first). (4) Live: 9202, a throwaway app capped at 64M with a memory hog: the counter rises, `app_oom` names the container while `OOMKilled` reads false. ~1 day, two repos, one agent release. **Earlier:** WATCHING — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a Line truncated
|
||||
| **R-1250** | Monitoring & notifications | P3 | **A qBittorrent that seeds all day is never line-tested, so it never gets the limit that exists for it.** The weekly test (R-1247) starts only after 10 minutes in which the host's uplink sent < 2 Mbit/s each minute; a seed at the line's speed keeps it busy through every window, so no test, no tested line, and no qBittorrent limit (D41 uses the tested line only). Found 2026-10-24 by reading the rule against the case it serves; not yet seen on a box (no test box seeds). | **VERIFY — released 2026-10-25, the live proof is a dated check:** controller 0.343.0 (global floor 17:53Z) + agent 0.165.0 (vouched 18:38 CET) carry it; the real proof cannot run before the next due window — the agent starts a test only on its due Tuesday 10:00–15:59 local, and every box measured on 2026-10-24. **DUE 2026-10-27:** a throwaway qBittorrent on demo-hp seeding to DooPlex's qBittorrent (operator's yes 2026-10-25, about 30 minutes), from before the host's slot; read the agent's line value and the limit back. **Earlier: NARROWED 2026-10-24 — FIXED on main, not released** (brief 2026-10-24 Part 4, commit only): agent `29bcf44`+`1e087bb` (`waiting_for_quiet`, `POST /host/line-test/run`), controller `046b808` (`internal/upwatch/linehold.go`: record first, hold at 1 KiB/s, test, the exact old limit back; the start-up check after a kill; the 50 % rule waits). Red-proofs in both CHANGELOGs. **Live on demo-hp with a real seed** (10 Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; held: uplink 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, back to 1 249 280 B/s exactly; a kill -9 mid-test left the record and the start-up check put the limit back (`audits/dooplex-alarms-2026-10-24/part4-r1250-live-proof.txt`). The live run used probes over plain HTTP (the pinned agent client is unit-tested, not live). | the next release train (agent ≥ 0.165.0 needs the operator's vouch) | Release agent + controller in the next normal train; on a seeding box read one Tuesday's `line test on request` line; then close | CC |
|
||||
| **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Design note (2026-10-21, after R-1192's diagnosis): the freeze's signature is cheap to see from the HOST side — the guest cgroup's memory PSI „full" above ~50 % for a minute, or its `memory.events max` climbing, while the host still reports; the agent could raise `guest_memory_thrash` (operator mail in its own words, not „no report for 45 m") and the household banner „A dobozod most nagyon lassú…". With the swap fix (R-1192) the measured thrash ends within a minute, so this alarm should be built only if a freeze is seen again after the agent release; until then it stays a design. Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC |
|
||||
| **R-1202** | Monitoring & notifications | P3 | **A box whose clock is behind blames the hub and stops an app without telling anyone.** Rehearsal 2026-10-20 B6 #8: RTC set 2 years back (the OS raised it to 2026-04-13): the dashboard shows only „Hub nem elérhető" while the log says „x509: certificate … not yet valid: current time 2026-04-13…"; Paperless crash-looped and the box stopped it with a restore hold; the `app_stopped_unhealthy` event failed (hub unreachable), so neither the household nor the operator heard. After the real time came back the app stayed stopped until a Start (clear sentence, 1 min). | **OPEN** | — | say „the box's clock is wrong (it says <date>)" when TLS fails with not-yet-valid/expired against a hub that answers; keep the stop event queued until the hub is reachable | CC |
|
||||
@@ -399,5 +399,6 @@ stopping line that lies.
|
||||
| R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row |
|
||||
<!-- DUE-CHECKS-END -->
|
||||
| **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC |
|
||||
| **R-1298** | Monitoring & notifications | P3 | **Scratch guest 9202 carries leftovers made outside the product, and D53 would remind about them every week once released: four crash-loop holds since 2026-09-24 for apps that are not installed (`chaoscrash`, `chaosoom`, `chaosoomb`, `gokapi`), and a PrivateBin `app.yaml` (2026-10-25) with no container that the product reads as „installed".** Found 2026-10-26 evening in the D53 live proof: the new reporter sent the four holds at its first tick and the test hub mailed four reminders („held 32 days"). The product's removal clears an install's hold (R-704); these apps were removed by hand in a chaos drill. On a household box the same shape would mail the operator weekly about an app that does not exist. | OPEN | — | decide whether the reporter (or the hub) skips an `unhealthy_stop` / `update_failed` / `putback_pending` hold whose app is not installed (a restore hold stays reported — operator-cleared by design, R-379); then clear 9202's four holds with `-clear-restore-hold` and remove PrivateBin through the product | CC |
|
||||
| **R-1298** | Monitoring & notifications | P3 | **Scratch guest 9202 carries leftovers made outside the product, and D53 would remind about them every week once released: four crash-loop holds since 2026-09-24 for apps that are not installed (`chaoscrash`, `chaosoom`, `chaosoomb`, `gokapi`), and a PrivateBin `app.yaml` (2026-10-25) with no container that the product reads as „installed".** Found 2026-10-26 evening in the D53 live proof: the new reporter sent the four holds at its first tick and the test hub mailed four reminders („held 32 days"). The product's removal clears an install's hold (R-704); these apps were removed by hand in a chaos drill. On a household box the same shape would mail the operator weekly about an app that does not exist. | **VERIFY — fixed on `main`, waits for the controller release** — controller `b156e45` (an install's hold of an app that is not installed is not reported; a restore hold always is), red-proved; live on 9202 (the four orphans skipped by name). 9202's four holds cleared with `-clear-restore-hold` and the PrivateBin leftover removed through the product — `audits/night-fixes-2026-10-26/`. Close at the release. **Earlier:** OPEN | — | decide whether the reporter (or the hub) skips an `unhealthy_stop` / `update_failed` / `putback_pending` hold whose app is not installed (a restore hold stays reported — operator-cleared by design, R-379); then clear 9202's four holds with `-clear-restore-hold` and remove PrivateBin through the product | CC |
|
||||
| **R-1299** | Process & tooling | P4 | **The operator's one-shot clean-up grant (`POST /offsite/window-grant/<customer>`, optional `max_remove`) has no button on any hub page** — the operator could not find it on 2026-10-26 (D55) and had to make a one-purpose access token so CC could call the route. The route, its scope (`offsite:write`) and its event exist (R-833); only the form is missing. | OPEN | — | a small form on the Off-site page (customer, optional cap ≤ 500), with a render test for both branches | CC |
|
||||
| **R-1302** | Monitoring & notifications | P4 | **A fresh install's agent logs ERROR „store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)" and „capability DEGRADED" for `felhom-pbs` at its FIRST start, although the grant was simply never made yet** — the whole-guest tier is armed (21:00:45) before the PBS storage exists (created 21:00:57 on the hub's descriptor). Seen 2026-10-26 night on the R-1232 BYO box (VM 9375, `audits/night-fixes-2026-10-26/r1232/3-first-run.txt`) and on 2026-10-25's BYO box (`degraded=pve:store-grant:felhom-pbs`). An ERROR that says „investigate the loss" on every new box teaches the reader to skip it. | OPEN | — | on the first grant of a tier whose storage did not exist before, log INFO „granted" instead of the loss ERROR; keep the ERROR for a grant that existed and vanished; a test for both | CC |
|
||||
Reference in new issue
Block a user