Review round 2, part 1 (batch 1): 01/03/04/05/06/08/12/13/_hub-review facts corrected against live code (each at file:line, EDITS.md); 20 capability rows marked re-check; poster timeline text; live-probes path; reviewer candidates and the H1/H2 proofs
gates / gates (push) Successful in 12m37s

This commit is contained in:
2026-10-24 15:46:33 +02:00
parent 44d1904286
commit 77015997f8
36 changed files with 886 additions and 91 deletions
+1 -1
View File
@@ -38,7 +38,7 @@ to probe a refusal at all.**
## If a probe does create something, remove it through the product
Not by hand, and not by `docker rm`: stop it, then `POST /api/stacks/<name>/remove` with
`remove_hdd_data` and `remove_backups`, and then **verify** — no container, no `/opt/felhom/stacks/<name>`,
`remove_hdd_data` and `remove_backups`, and then **verify** — no container, no `/opt/docker/stacks/<name>/app.yaml` (the folder itself stays — it holds the catalog's template files; path corrected 2026-10-24),
no volume. Say in the report that it happened. A tidy-up nobody is told about is how the next session
learns nothing.
+20 -20
View File
@@ -80,17 +80,17 @@
| <a id="a4"></a>[A4](capability-map-detail/A.md#a4) | Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; universal secret-free / operator-bind mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PROVEN-LIVE** | N100 (2026-07-18) and HP t740 (2026-07-21) | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/SPIKE-baremetal-iso-2026-07-16.md` (2026-07-16) |
| <a id="a5"></a>[A5](capability-map-detail/A.md#a5) | Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`<iso>.rootpw.txt`) | scripts v1.24.0 (`felhom-bootstrap.sh`, `build-felhom-iso.sh`) | **PROVEN-LIVE** | nested drill; metal proof pending | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` (2026-07-22) |
| <a id="a6"></a>[A6](capability-map-detail/A.md#a6) | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** | drill VM; no non-operator human yet (R-3) | `audits/DRILL-day0-vm-2026-07-12.md` (2026-07-12) |
| <a id="a7"></a>[A7](capability-map-detail/A.md#a7) | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** | customer-zero, on metal | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
| <a id="a8"></a>[A8](capability-map-detail/A.md#a8) | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** | drill VM, endpoint-exact; browser wizard first fired 2026-07-18 | `audits/SPIKE-controller-escrow-2026-07-13.md` (2026-07-13); `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
| <a id="a7"></a>[A7](capability-map-detail/A.md#a7) | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** | customer-zero, on metal; **re-check (2026-10-24):** mechanism changed after its evidence — f90e2229 (bind mail W3–W5) | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
| <a id="a8"></a>[A8](capability-map-detail/A.md#a8) | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** | drill VM, endpoint-exact; browser wizard first fired 2026-07-18; **re-check (2026-10-24):** mechanism changed after its evidence — agent c11e3e2 + controller ba4a6f0 (typed-back words, R-1151) | `audits/SPIKE-controller-escrow-2026-07-13.md` (2026-07-13); `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
| <a id="a9"></a>[A9](capability-map-detail/A.md#a9) | One name per secret on every surface — „Beállító kód" (the setup code) and „Tulajdonosi jelmondat" (the owner passphrase) — and the re-enrol mail points at a page a rebuilt box shows | controller v0.211.0, hub v0.104.0 + v0.105.0 (R-295, R-323) | **IMPLEMENTED** | written BUILT: source-read; no customer has typed the passphrase | R-295; R-323 |
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints; **re-check (2026-10-24):** mechanism changed after its evidence — hub 9688e76e (sealed package back to a rebuilt box, R-1176) | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only; **re-check (2026-10-24):** mechanism changed after its evidence — agent ae69ad4 (D30 WireGuard re-key) | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
| <a id="a12"></a>[A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES — the rebuilt Tester4 box (public ISO) got the household's sealed package from the hub, asked for the recovery code, opened the history; Nextcloud came back with 3 031 of 3 033 manifest files byte-identical (the 2 never reached the old box) and the next off-site runs appended (63 → 64). Earlier the same day it FAILED (R-1176, now closed); the failure paths are not claimed (R-379, R-380) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) |
| <a id="a13"></a>[A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES with ONE operator act by design (D19: delete the dead host) — install, bind, claim, the recovery offer, a wrong code refused, the right code, „Teljes visszaállítás", all through the household's own pages; the new-link page now tells a household whose old box is still linked to write to Felhom (R-1203). Earlier the same day it stopped at the rebuilt dashboard (R-1176) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) |
| <a id="a14"></a>[A14](capability-map-detail/A.md#a14) | A random night of household actions while random things go wrong — twelve rounds, unattended | controller v0.245.0, agent v0.131.0, hub v0.116.0, golden 0.245.0, ISO 1.28.0 | **PROVEN-LIVE** | nested box, twelve chaos rounds | `audits/DRILL-chaos-night-2026-09-17.md` (2026-09-17) |
| <a id="a15"></a>[A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.336.0, agent v0.161.0, hub v0.162.0, golden 0.331.0, installer 1.35.0, ISO 1.29.0 | **PARTIAL** | 2026-10-20: download → install → bind → claim → 8 apps → use → backup → remove → every restore path → power cut → mistyped codes all worked; the 8 GB box FROZE twice in the first afternoon and only a plug pull brought it back (R-1192), and the owner passphrase needed the operator | `audits/rehearsal-2026-10-20/journal.md` (2026-10-20); `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) |
| <a id="a16"></a>[A16](capability-map-detail/A.md#a16) | Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts | controller v0.197.0 | **PROVEN-LIVE** | both drive layouts | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04) |
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp; **re-check (2026-10-24):** mechanism changed after its evidence — agent 1b6e316 (local keep-daily, R-1149) | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
| <a id="a18"></a>[A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites per-repo CHANGELOGs and the spec `backlog/SPEC-r85-phase4-5-2026-07-26.md` — a spec, not a run record |
| <a id="a19"></a>[A19](capability-map-detail/A.md#a19) | A restore-test can never fill the box's disk: it sizes the restore first (uncompressed), keeps off the tested guest's pool when it can, refuses what does not fit, and retries a failed clean-up on a timer | agent v0.133.0 (RELEASED, NOT DELIVERED), hub v0.124.0 | **IMPLEMENTED** | written BUILT: the refusal proven on demo-hp 2026-09-24; the full-test path not live | `audits/r672-2026-09-24/` (2026-09-24); R-672 |
| <a id="a20"></a>[A20](capability-map-detail/A.md#a20) | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0; purge-first hub v0.142.0 | **PROVEN-LIVE** | two real firings (2026-07-18); third 2026-10-22 on a real customer (Tester-2) with the R-32 purge-first leg — all legs ok, others untouched | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/tester2-reset-2026-10-22/` |
@@ -102,17 +102,17 @@
| ID | Capability | Component | Status | Scope | Evidence |
|---|---|---|---|---|---|
| <a id="b1"></a>[B1](capability-map-detail/B.md#b1) | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (~52 apps, images pinned) | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/RERUN-p1p3-2026-07-06.md` (2026-07-06) |
| <a id="b2"></a>[B2](capability-map-detail/B.md#b2) | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | the actions; remove-with-data re-proven 2026-09-13 | `audits/R442-2026-09-13/` (2026-09-13); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
| <a id="b1"></a>[B1](capability-map-detail/B.md#b1) | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (~52 apps, images pinned) | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 6d90374 + 2fe6341 (allow-list R-971, appdata R-1142) | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/RERUN-p1p3-2026-07-06.md` (2026-07-06) |
| <a id="b2"></a>[B2](capability-map-detail/B.md#b2) | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | the actions; remove-with-data re-proven 2026-09-13; **re-check (2026-10-24):** mechanism changed after its evidence — 8ae9178 (removal deletes labelled volumes, R-1117) | `audits/R442-2026-09-13/` (2026-09-13); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
| <a id="b3"></a>[B3](capability-map-detail/B.md#b3) | Update is GUARDED: it refuses without a restorable backup, backs up first when the copy is stale, and HOLDS an app that does not come up — on ANY backup tier, and the release itself arrives by the managed floor | controller v0.237.0 + v0.238.0 + v0.238.1 + v0.239.0, hub v0.112.0 | **PROVEN-LIVE** | — | `audits/slice4-2026-09-13/` (2026-09-13); `audits/rulings-r472-r475-2026-09-13/` (2026-09-13) |
| <a id="b4"></a>[B4](capability-map-detail/B.md#b4) | A failed update is UNDONE by the box itself — the previous version back with its data from seconds before the update; the app is held only if that undo fails too | controller v0.263.2 (the undo), v0.264.0 + hub v0.120.0 (the mail) | **PROVEN-LIVE** | scratch 9202 | `audits/undo-live-2026-09-23/README.md` (2026-09-23) |
| <a id="b5"></a>[B5](capability-map-detail/B.md#b5) | The catalog holds only TESTED steps — an image move without a proven test record (bench + box, digests the registry still serves) is refused at push time | catalog `6db08a5` (test-record gate, `ladder.py`) | **PROVEN-LIVE** | written PROVEN: the catalog gate against the live registry | `audits/DRILL-night-2026-09-23.md` (2026-09-23) |
| <a id="b6"></a>[B6](capability-map-detail/B.md#b6) | A box behind climbs ONE tested step per press, each with its own definition (`09` §3 decision 14) | controller v0.268.0 + catalog `5ed599c` | **PROVEN-LIVE** | — | `audits/ladder-2026-09-24/partD/10-romm-two-steps.json` (2026-09-24) |
| <a id="b7"></a>[B7](capability-map-detail/B.md#b7) | A file app comes back WHOLE from the second drive — its files by four rules (never delete, never overwrite a newer file, bring back what is missing, keep an older replaced file beside it), then its settings and database (`09` §3 decision 26) | controller v0.269.0 (`backup/tier2_whole.go`) | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A1/21-readback.json` (2026-09-24) |
| <a id="b7"></a>[B7](capability-map-detail/B.md#b7) | A file app comes back WHOLE from the second drive — its files by four rules (never delete, never overwrite a newer file, bring back what is missing, keep an older replaced file beside it), then its settings and database (`09` §3 decision 26) | controller v0.269.0 (`backup/tier2_whole.go`) | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb + 5929b8e (D22, D25) | `audits/night-2026-09-24/A1/21-readback.json` (2026-09-24) |
| <a id="b8"></a>[B8](capability-map-detail/B.md#b8) | A crash-looping or out-of-memory app is STOPPED by the box, the household and operator are told, and Start gives one more try (decision 28) | controller v0.269.0, hub v0.123.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A3/21-gokapi-page-event.txt` (2026-09-24) |
| <a id="b9"></a>[B9](capability-map-detail/B.md#b9) | A held app whose page says support is informed can only be removed KEEPING its data (decision 27) | controller v0.269.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A2/10-held-no-copy-keep-data.json` (2026-09-24) |
| <a id="b10"></a>[B10](capability-map-detail/B.md#b10) | The box runs the exact TESTED image of a floating tag, and an installed app keeps its image until a guarded Update moves it (`09` §6.4 part 6, box half) | controller v0.269.1 (`stacks/digest.go`) | **PROVEN-LIVE** | — | `audits/night-2026-09-24/B/21-floating-tag-0.269.1.json` (2026-09-24) |
| <a id="b11"></a>[B11](capability-map-detail/B.md#b11) | Automatic updates at night — the update leg (`09` §6.4 part 7) | controller v0.271.0 | **PROVEN-LIVE** | scratch 9202, six simulated nights; demo boxes in Part D | `audits/DRILL-night-2026-09-25.md` (2026-09-25) |
| <a id="b11"></a>[B11](capability-map-detail/B.md#b11) | Automatic updates at night — the update leg (`09` §6.4 part 7) | controller v0.271.0 | **PROVEN-LIVE** | scratch 9202, six simulated nights; demo boxes in Part D; **re-check (2026-10-24):** mechanism changed after its evidence — 63dcccf (security_fix steps, 0.341.0) | `audits/DRILL-night-2026-09-25.md` (2026-09-25) |
| <a id="b12"></a>[B12](capability-map-detail/B.md#b12) | What `restart` and `update` do to a deployed app whose compose file the catalog already moved | controller v0.235.0 | **PROVEN-LIVE** | HISTORY: the behaviour up to v0.234.0; changed 2026-09-06 — current behaviour is B17 | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
| <a id="b13"></a>[B13](capability-map-detail/B.md#b13) | Whether the box UPGRADES an app by itself, with nobody pressing anything | controller | **PROVEN-LIVE** | answer: yes, only when an app fails to come back (measured 2026-09-01) | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
| <a id="b14"></a>[B14](capability-map-detail/B.md#b14) | Whether an app UPGRADE can be undone | controller + catalog | **PROVEN-LIVE** | answer: no, once a migration has run | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
@@ -147,10 +147,10 @@
|---|---|---|---|---|---|
| <a id="c1"></a>[C1](capability-map-detail/C.md#c1) | Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded DEGRADED rather than silently normal (installer Case A/B) | agent v0.113, host-install v1.22.0 | **PROVEN-LIVE** | — | `audits/E2D-fresh-vm-2026-07-29.md` (2026-07-29) |
| <a id="c2"></a>[C2](capability-map-detail/C.md#c2) | Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) |
| <a id="c3"></a>[C3](capability-map-detail/C.md#c3) | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
| <a id="c4"></a>[C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/VALIDATION-offbox-storagebox-2026-07-09.md` (2026-07-09) |
| <a id="c5"></a>[C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) |
| <a id="c6"></a>[C6](capability-map-detail/C.md#c6) | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6B-2026-07-14.md` (2026-07-14) |
| <a id="c3"></a>[C3](capability-map-detail/C.md#c3) | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 03c0a69 + 66a69d4 + 5929b8e (capped container R-1214, kept copies D25) | `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
| <a id="c4"></a>[C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 8f5bea0 + d5f88a2 (abandon R-1182, off-site proof R-1189) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/VALIDATION-offbox-storagebox-2026-07-09.md` (2026-07-09) |
| <a id="c5"></a>[C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 93fb251 + d582feb (D20, D22) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) |
| <a id="c6"></a>[C6](capability-map-detail/C.md#c6) | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d5f88a2 + 5929b8e (R-1187 c, R-1191) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6B-2026-07-14.md` (2026-07-14) |
| <a id="c7"></a>[C7](capability-map-detail/C.md#c7) | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `pilot/DRILL-GL6-2026-07-08.md` (2026-07-08) |
| <a id="c8"></a>[C8](capability-map-detail/C.md#c8) | PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | scoped to one host | `audits/SPIKE-pbsdr-selfheal-2026-07-15.md` (2026-07-15) |
| <a id="c9"></a>[C9](capability-map-detail/C.md#c9) | Box survives an unattended app or guest-network failure (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0→v0.190.0, agent v0.92.1 | **PROVEN-LIVE** | demo box; boot-orphan leg 6 of 6 hard resets | `audits/DIAG-bootrecon-drive-absent-2026-08-02.md` (2026-08-02) |
@@ -172,12 +172,12 @@
| <a id="d3"></a>[D3](capability-map-detail/D.md#d3) | Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14); `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
| <a id="d4"></a>[D4](capability-map-detail/D.md#d4) | NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | — | `audits/SPIKE-nas-verify-2026-07-11.md` (2026-07-11); `audits/SPIKE-nas-dsm-2026-07-11.md` (2026-07-11) |
| <a id="d5"></a>[D5](capability-map-detail/D.md#d5) | Network storage (NAS) is browse + bulk-media only — it may NOT host an app's data namespace | controller v0.187.0 | **PROVEN-LIVE** | — | `audits/R108-network-app-namespace-2026-07-30.md` (2026-07-30) |
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | — | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb (D22, R-1178) | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
| <a id="d7"></a>[D7](capability-map-detail/D.md#d7) | A restore SAYS what it returned, and refuses what it cannot do — the four restore-surface truth defects from the 2026-08-21 drill | controller v0.226.0 (R-353, R-357, R-358, R-360, R-396) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: three of four live; R-357 implemented only | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt` (2026-08-30) |
| <a id="d8"></a>[D8](capability-map-detail/D.md#d8) | The box PROVES its own off-site copy still HOLDS something — a backup that is intact and EMPTY is caught without a person | controller v0.231.0 (R-87) + hub v0.110.0 | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r87-offsite-proof-2026-08-31/` (2026-08-31) |
| <a id="d9"></a>[D9](capability-map-detail/D.md#d9) | The off-site store is VERIFIED on a cadence — something checks that the customer's backups are still readable | controller v0.228.0 (R-359, R-397, R-399) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r359-integrity-2026-08-30/` (2026-08-30) |
| <a id="d10"></a>[D10](capability-map-detail/D.md#d10) | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | — | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13) |
| <a id="d11"></a>[D11](capability-map-detail/D.md#d11) | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | — | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
| <a id="d10"></a>[D10](capability-map-detail/D.md#d10) | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 (R-1124) | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13) |
| <a id="d11"></a>[D11](capability-map-detail/D.md#d11) | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 + 5c3b218, controller 6c6bd87 (R-1124, R-1144) | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
| <a id="d12"></a>[D12](capability-map-detail/D.md#d12) | Boot ordering: automount + networking survive reboot; appliance self-heal watchdog | agent v0.85 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13); `audits/CAMPAIGN-6A-2026-07-14.md` (2026-07-14) |
| <a id="d13"></a>[D13](capability-map-detail/D.md#d13) | Media apps see EVERY drive; a drive added, pulled, back or safely removed is handled per app (notice, real restart, wiring, torrents paused/resumed) | controller v0.313.0, catalog `0188430`, hub v0.151.0 | **PROVEN-LIVE** | Tier-0 boxes (Tester 1, demo-hp 9201); the fleet with golden 0.313.0 + the floor | `audits/media-build-2026-10-15/G/` (2026-10-15); `audits/media-drive-events-2026-10-14/` (2026-10-14) |
| <a id="d14"></a>[D14](capability-map-detail/D.md#d14) | A second local copy COUNTS only on a proven different physical disk: the agent knows each place's disks; a VM box makes the copy but says it cannot check; a same-disk copy is not made; a NAS never hides the one-copy notice; a box with no drive refuses a data-folder app in words | agent v0.157.0, controller v0.317.0, hub v0.154.0 | **PARTIAL** | live: VM `unknown` (Tester 1) and two real-disk boxes `proven` (demo-hp, N100); unit only: the same-disk skip, the NAS notice, the no-drive sentence; `usb_bridge` never seen on a device | `audits/storage-identity-buildA-2026-10-15/README.md` (2026-10-15); `architecture/16-storage-physical-identity.md` §13 |
@@ -204,7 +204,7 @@
| <a id="e14"></a>[E14](capability-map-detail/E.md#e14) | Multiple household users / per-person accounts | — | **MISSING** | — | R-15 |
| <a id="e15"></a>[E15](capability-map-detail/E.md#e15) | A second login step for the dashboard (a TOTP code or a passkey) | — | **MISSING** | — | R-811 |
| <a id="e16"></a>[E16](capability-map-detail/E.md#e16) | The household can leave Felhom, or outlive it — the box runs without the hub, the household owns its domain, tunnel and off-site account, and can export everything | — | **MISSING** | as a written answer | R-810 |
| <a id="e17"></a>[E17](capability-map-detail/E.md#e17) | The host agent cannot reach root without the operator key: exact sudo patterns, no agent-written file installed where root reads it without a content check, the agent binary only by an operator-signed update checked as root | agent v0.146.1 (R-861; delivered by a step bundle, R-880) | **PROVEN-LIVE** | both demo boxes; three named residuals | `audits/hub-safety-2026-10-05/partF/` (2026-10-05) |
| <a id="e17"></a>[E17](capability-map-detail/E.md#e17) | The host agent cannot reach root without the operator key: exact sudo patterns, no agent-written file installed where root reads it without a content check, the agent binary only by an operator-signed update checked as root | agent v0.146.1 (R-861; delivered by a step bundle, R-880) | **PROVEN-LIVE** | both demo boxes; three named residuals; **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 + a0c88e5 (sudoers + priv-apply, D4 drop-in) | `audits/hub-safety-2026-10-05/partF/` (2026-10-05) |
| <a id="e18"></a>[E18](capability-map-detail/E.md#e18) | WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | — | `audits/SPIKE-oob-wg-operator-peer-2026-07-05.md` (2026-07-05); `audits/SPIKE-felhom-sshd-2026-07-05.md` (2026-07-05) |
| <a id="e19"></a>[E19](capability-map-detail/E.md#e19) | The operator can see WHERE a managed host is — its LAN address and its WireGuard address, on the host page | agent v0.119.0, hub v0.85.0 | **PROVEN-LIVE** | — | `audits/host-addresses-visible-2026-07-31.md` (2026-07-31) |
| <a id="e20"></a>[E20](capability-map-detail/E.md#e20) | The operator can see whether a managed host's guests still have working networking — and how often the watchdog had to repair them | agent v0.92.0 (emitter, 2026-07-21), hub v0.104.0 (reader, 2026-08-13) | **IMPLEMENTED** | not proven live | R-319 |
@@ -238,7 +238,7 @@
| <a id="g4"></a>[G4](capability-map-detail/G.md#g4) | Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules | hub v0.45/0.53, agent | **PARTIAL** | — | `runbooks/publish-train-rules.md` (2026-09-13); R-23 |
| <a id="g5"></a>[G5](capability-map-detail/G.md#g5) | Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed | agent v0.70+ | **PROVEN-LIVE** | demo | `audits/SPIKE-agent-selfupdate-2026-07-05.md` (2026-07-05) |
| <a id="g6"></a>[G6](capability-map-detail/G.md#g6) | Controller self-update: anonymous registry, no credentials in guest | controller v0.112 | **PROVEN-LIVE** | demo; the file found is the floor-driven self-update leg | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
| <a id="g7"></a>[G7](capability-map-detail/G.md#g7) | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | — | `audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md` (2026-07-09); `audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (2026-07-09) |
| <a id="g7"></a>[G7](capability-map-detail/G.md#g7) | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — hub 9688e76e (scoped abandonment, R-1182) | `audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md` (2026-07-09); `audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (2026-07-09) |
| <a id="g8"></a>[G8](capability-map-detail/G.md#g8) | Per-customer offsite fill + staleness + freeze lever | hub v0.41 | **IMPLEMENTED** | no live-fired leg | `audits/CAMPAIGN-offsite-overnight-2026-07-10.md` (2026-07-10) |
| <a id="g9"></a>[G9](capability-map-detail/G.md#g9) | Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert) | hub v0.64.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.64.0 code and unit tests (outside `documentation/`) |
| <a id="g10"></a>[G10](capability-map-detail/G.md#g10) | Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge) | hub v0.65.0 + tenantsync v1.2.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.65.0 code and unit tests (outside `documentation/`) |
@@ -246,10 +246,10 @@
| <a id="g12"></a>[G12](capability-map-detail/G.md#g12) | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | — | the 2026-07-13 hub closing bundle (no evidence document) |
| <a id="g13"></a>[G13](capability-map-detail/G.md#g13) | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | — | hub v0.54 (2026-07-13; no evidence document) |
| <a id="g14"></a>[G14](capability-map-detail/G.md#g14) | The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail | hub v0.135.0 (R-604, R-530) | **IMPLEMENTED** | the page live; the alarm and the mail unit-proven | `audits/hub-safety-2026-10-05/partD/` (2026-10-05) |
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | — | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g16"></a>[G16](capability-map-detail/G.md#g16) | Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail | `scripts/dooplex-offsite/` (R-232), homelab-manifests rules | **PROVEN-LIVE** | — | `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g17"></a>[G17](capability-map-detail/G.md#g17) | The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`) | hub v0.135.0 (R-135) | **PROVEN-LIVE** | — | `audits/hub-safety-2026-10-05/partA/live.txt` (2026-10-05) |
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
| <a id="g19"></a>[G19](capability-map-detail/G.md#g19) | An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps | controller v0.259.0 + hub v0.119.0 + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE** | walked on 0.258.0 with one blocker; fixed, the walk not repeated | `audits/DRILL-first-hour-en-0258-2026-09-20.md` (2026-09-20); `audits/i18n-closing-2026-09-21/live/` (2026-09-21) |
| <a id="g20"></a>[G20](capability-map-detail/G.md#g20) | A deletion of a customer's off-site history is NOTICED within a day | hub v0.111.0 (R-431) | **IMPLEMENTED** | not yet proven live | R-431 |
@@ -16,7 +16,7 @@
**Status:** draft (decisions from the topology/trust design sessions).
**Platform facts** referenced here live in `docs/proxmox-platform.md`; this document
**Platform facts** referenced here live in `documentation/proxmox-platform.md` (path corrected 2026-10-24); this document
records *Felhom's decisions*, not Proxmox behaviour.
---
@@ -105,7 +105,7 @@ credentials.
- **[FACT, 2026-10-16, R-970]** The last contradiction is gone: until controller v0.322.0 the box's own hub API key,
sent as a Bearer token, opened `/api/config/` (read `controller.yaml` with every secret in it, and REPLACE it,
unsigned) and the self-update and geo routes. Nothing called them (the hub never connects to a box). The config
routes are removed and no route accepts the hub key; every `/api/` route needs a dashboard session (pinned by
routes are removed and no route accepts the hub key; every `/api/` route but the read-only `/api/health` needs a dashboard session (pinned by
`TestR970_HubKeyOpensNoDoor` on the real route table). The box exposes no control endpoint, as stated above.
- All operator-initiated actions are recorded in a **customer-visible audit log**.
@@ -132,7 +132,7 @@ of all 53 apps, 2026-09-28).
**[DESIGN, 2026-10-17, R-1126, operator decision D11 = A] The tunnel runs before the claim, and FileBrowser never
answers on a default password.** The box starts cloudflared as soon as it has a tunnel token, before it is claimed,
because the claim itself is reached through it: the claim mail links `https://felhom.<domain>`
(`hub/internal/notify/templates.go:27`). Before the claim a stranger reaches only the claim page (a code: 72 h life, 5
(`hub/internal/notify/templates.go` `FormatClaimEmail`; line corrected 2026-10-24). Before the claim a stranger reaches only the claim page (a code: 72 h life, 5
tries, 15-minute lockout — controller `web/claim.go`) and `files.<domain>`. FileBrowser starts every new database on
admin/admin, and its own admin-password setting cannot replace that (it re-applies at every start and would overwrite
the household's own password — FileBrowser 1.5.6 `backend/cmd/user.go:50-57`, measured). So FileBrowser starts OFF
@@ -219,7 +219,10 @@ app. Controller down → the gated app answers an error, never the app. Measured
This removes any zero-touch remote-enrollment problem.
- A **one-time retrieval code** mints durable identity. Single-use (burned on the successful
config fetch) plus a short *pre-use* TTL; one-click regenerate for the only real failure
case (fetch fails before anything is persisted). After the fetch, the code is irrelevant —
case (fetch fails before anything is persisted). *(As built — corrected 2026-10-24: the passphrase is not burned
and has no TTL; it gives keys until the box has a host record AND a controller ≥ 0.334.0 has reported with its own
key, then `/config`, `/recovery` and `/host-enroll` answer 409 — D19, R-1159, `hub/internal/api/passphrase_scope.go:19-30`;
`05` §8 has the full rule.)* After the fetch, the code is irrelevant —
everything downstream runs on durable credentials, so retries don't need it.
- **Order:** the agent enrolls first (and, running as root at setup, mints its own scoped
operator-tier Proxmox token), then provisions the customer LXC from the golden template and
@@ -253,7 +256,7 @@ app. Controller down → the gated app answers an error, never the app. Measured
2026-09-14 first-hour drill used a `*.felhom.eu` customer domain with no tunnel, and the dashboard link
in the setup-code mail did not resolve.
- **A pasted Cloudflare API token must reach exactly the customer's own zone** (R-138 option C, `09` §3 decision 190;
hub main 2026-10-08, unreleased). On save the hub asks Cloudflare `GET /zones` with the token and keeps it only when
released in hub v0.144.0 — corrected 2026-10-24). On save the hub asks Cloudflare `GET /zones` with the token and keeps it only when
the token sees exactly ONE zone and that zone IS the customer's domain (a zone above it is refused: a sibling
customer added later would be reachable). Cloudflare unreachable → the save fails and the old token stays. The token
is never logged. **Limit:** `/zones` lists what the token can READ; a hand-built token with Zone:Read on one zone and
@@ -313,7 +316,8 @@ app. Controller down → the gated app answers an error, never the app. Measured
>
> **What being unmarked cost.** The consequence of this bullet — that 40 of 53 catalogue templates
> declare no configurable path because they are all-hot — is stated as **[FACT]** at
> `07-backup-architecture.md:296-299`. A reader met a marked observation beside an unmarked choice
> `07-backup-architecture.md:296-299`. *(Dated note 2026-10-24: that pointer is dead — those lines are about the local
> plaintext posture now, and no architecture file but this one states „40 of 53 … all-hot".)* A reader met a marked observation beside an unmarked choice
> and reasonably asked whether it *should* be so. **Between 19 and 22 August that reader called this
> decision a defect in four places** (R-370), and one session's work went into correcting the record
> rather than into the product.
@@ -343,7 +347,9 @@ app. Controller down → the gated app answers an error, never the app. Measured
restores from PBS; the controller returns on the same domain (tunnel reused from the hub
record). DR = provisioning + a restore mode, not a separate mechanism.
- **Snapshot-before-deploy:** controller asks the agent to snapshot, deploys, runs its
post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.)
post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.) **[DESIGN — not built,
noted 2026-10-24]:** the agent serves `/snapshot` and `/rollback`, but no controller code calls them
(`felhom-controller/controller/internal/agentapi`); the update's guard is the per-app backup + undo of `09` §3 decision 15.
---
+16 -12
View File
@@ -47,7 +47,7 @@ Owns:
1. **Proxmox lifecycle** — create/start/stop/destroy guests, snapshots, storage allocation. Via a scoped Proxmox API token (the **`FelhomAgent` operator role** — `proxmox-platform.md` §3.6, validated Phase 3 B3) for everything the API covers; raw host ops only where unavoidable.
2. **Storage management** — attach/classify targets, reconcile the storage manifest, mount USB-by-UUID, present mounts into guests.
3. **Backup/restore orchestration** — vzdump to the tiers, PBS, snapshot management, and the **self-restore-test**.
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Three states: `running` (healthy), `not_running` (stopped, absent, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Four states (corrected 2026-10-24): `running` (healthy), `not_running` (stopped, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms — and since agent v0.161.0 `absent` (the guest has no cloudflared container at all, R-1070; the hub never alarms on it) — `internal/hub/cloudflared.go:12-23`. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
Proxmox API, plus the OS wrapper's read-only `facts` (host Debian, next-boot kernel, held packages, taint, the crash
guard; guest Debian, Docker engine, containerd, live-restore), at most every 10 min. **Still no new sudoers line** — the
facts, `live-restore-on` and the Docker layer all ride `FELHOM_OSAPPLY`. New ROOT-owned files instead (installer 1.30.0;
@@ -129,13 +129,14 @@ the one trust root; the agent's API token is untouched; a box gets the new rule
`agent_config_update` (the order: signed `agent_update` first, then the bundle — after the bundle, an agent below
0.146.0 cannot update itself on that box).
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — built as a ready patch, NOT shipped] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — SHIPPED in agent v0.163.0 (`1109092`) and installer-v1.36.0 (`manifests/webpage.yaml:347`); corrected 2026-10-24] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
**[FACT, agent main `a0c88e5`, unreleased] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
**[FACT, agent `a0c88e5`, released in v0.163.0 — corrected 2026-10-24] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
**[DESIGN, operator ruling D30 2026-10-22; agent main `ae69ad4`, unreleased] The tunnel heals itself by a new key.** When
the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the old
private key kept as `wg/private.key.prev`, 0600 — the escrow may hold it) and registers the new public key on the
**[DESIGN, operator ruling D30 2026-10-22; agent `ae69ad4`, released in v0.163.0 — corrected 2026-10-24] The tunnel heals itself by a new key.** When
the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the FIRST old
private key kept as `wg/private.key.prev`, later ones as `wg/private.key.prev.<unix>`, the newest three kept, 0600 — the
escrow may hold one; the once-an-hour bound survives a restart in `wg/last-rekey` — `internal/wgtunnel/rekey.go`, corrected 2026-10-24) and registers the new public key on the
existing path, `POST /hosts/<id>/wg`: the hub re-keys the peer in place (same /32, generation bump) and its full-list
push to ep0 (`wgsync`, `wg syncconf`) drops the old key and adds the new one. A new key is a new peer for ep0, with no
stored timestamp. Once an hour at most; not while the unit is down; never during a re-register already in flight. The
@@ -250,8 +251,9 @@ The controller (in its LXC) reaches the agent (on the host) over the local bridg
- `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
- `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. **`host_busy`
(R-921, agent Unreleased `56aabc7`):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off), the
(R-921, agent `56aabc7`, released in v0.161.0 — corrected 2026-10-24):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off, and since v0.164.0 the
weekly upload line test — gate name `line-test`), the
state in which `POST /backup` answers 409; absent when the gate is not wired. The controller (≥ 0.335.0) reads it
before it stops any app. A read, never a reservation.
- **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record
@@ -314,7 +316,8 @@ token → guest and authorizes per guest, so a compromised controller's blast ra
quiesce-on-due consumer is 8B), the rest wrap the existing slice-5/6/7 machinery.
- **Transport / pin.** The agent serves a **persisted self-signed leaf** bound to the host bridge IP
on a fixed port (default `:8443`). The controller pins the **leaf-cert SHA-256** (decision:
on a fixed port — `local_api.listen_addr` is REQUIRED, there is no default (`internal/config/config.go:317-319`); an
island install (R-50) listens on `169.254.253.1:8443` on its private bridge (corrected 2026-10-24). The controller pins the **leaf-cert SHA-256** (decision:
consistency with the agent's Proxmox/PBS cert pinning), carried in its bootstrap. The leaf is
generated **once and persisted**, so its fingerprint is stable across agent restarts (a fresh cert
each boot would invalidate every already-issued bootstrap pin). Defense-in-depth: the listener
@@ -420,8 +423,8 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent Unreleased
> `56aabc7`):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent
> `56aabc7`, released in v0.161.0 — corrected 2026-10-24):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
> snapshot already gone), it retries once with Proxmox's own `force=1` on the same API call (no sudoers line, no
> root command); any other error is returned unchanged, and a failed forced delete is never forced again.
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
@@ -677,7 +680,8 @@ the *archive* and the *back half*, **and in identity policy** (below).
`swap`, and the agent never set it; on the 8 GB minimum machine (guest 4096 MB) the household's first uploads filled the
guest with its swap full, the kernel could evict only program code, and the box froze (memory „full" pressure 96 %, load
162, the dashboard dead 6½ minutes). `[DESIGN]` The size step sets `swap` = the guest's memory: at bring-up, at a RAM
resize (`POST /guest/memory`), and once a minute after agent start for every pool LXC whose swap is below its RAM (live,
resize (`POST /guest/memory`), and ONCE, one minute after agent start, for every pool LXC whose swap is below its RAM (`cmd/felhom-agent/main.go`
`EnsureGuestSwap`; „once a minute" corrected 2026-10-24) (live,
no restart, never lowers). Proven on a fresh box the night of 2026-10-21 (`audits/night-2026-10-21/P1-soak/`: 512 →
4096 one minute after the fixture agent started; the 2-hour household load then answered every 10-second probe). **The
host must have swap behind it:** our installer gives an 8 GB / 256 GB machine 7.75 GB (measured the same night); a BYO
@@ -15,9 +15,9 @@
> Status: design draft (decision content), grounded on `docs/tests/phase4-signing-findings.md`.
> Status: design draft (decision content), grounded on `documentation/tests/phase4-signing-findings.md`.
> To be reviewed by Claude Code against that spike + `03` §4, then placed at
> `docs/architecture/04-control-plane-authorization.md`.
> `documentation/architecture/04-control-plane-authorization.md` — where it is now (paths corrected 2026-10-24).
>
> Builds on Part 1 (enrollment / trust), Part 3 (the agent verifies + the §4 reversibility gate).
> This doc defines the **mechanism** behind `03` §4's "an operator signature the hub can't forge."
@@ -75,7 +75,12 @@ recorded last**, only after everything else passes, so an invalid signature can
nonce (DoS-safe). Each layer is mandatory and was proven to reject independently (Phase 4 §3–4):
- **target binding** — `target.host_id`/`guest_id` must equal *this* box/guest (a signature for box
A cannot be replayed at box B);
- **time window** — `now ∈ [issued_at, expires_at]`;
- **time window** — `now ∈ [issued_at, expires_at]`; as built, `issued_at` may be up to **2 minutes** in the future
(clock skew), `expires_at` is never extended — the agent (`felhom-agent/internal/authz/verifier.go:19-22,148-155`) and
the root wrapper (`configs/felhom-os-apply`) alike (corrected 2026-10-24). **[DESIGN — undocumented, found
2026-10-24]:** neither caps the window's LENGTH (`expires_at − issued_at`); `felhom-opsign` defaults to 30 min
(`cmd/felhom-opsign/main.go:57`), and the nonce store keeps a record until its expiry. A register row asks the
operator to confirm or cap it;
- **nonce** — unseen within the window (the nonce store **must be persistent across agent restarts**
and expiry-pruned; a non-persistent store reopens the replay window after a restart).
@@ -166,7 +171,7 @@ op the agent verifies** (same pipeline, §2.3) — never unauthenticated config.
queue — then the agent polls, verifies, executes, and audits. One command + passphrase, from the
desk. **Never** a site visit.
### 6.1 Operator actions in the report reply [DESIGN, R-314/R-279 — `09` §3 decision 185, controller + hub main 2026-10-08, unreleased]
### 6.1 Operator actions in the report reply [DESIGN, R-314/R-279 — `09` §3 decision 185; released in controller v0.304.0 + hub v0.144.0 — corrected 2026-10-24]
Routine, non-destructive operator requests need no signature (`03` §4 asks for one only to destroy or overwrite the
only copy). The hub stores a row, bumps the box's intent, and the report reply carries `operator_actions:[{id,action,arg}]`
@@ -22,8 +22,9 @@
> The hub is **not** greenfield — it's a mature service (felhom-hub **v0.11.0** as of 2026-06-13, Go +
> SQLite on k3s, `hub.felhom.eu`). This doc is the **deltas** to evolve it for the Proxmox model, plus
> the new data model. Builds on Part 1 (trust/enrollment), Part 3 (the agent + reconcile), Part 4 (signing).
> NOTE: this remains a design-draft for the deltas; the live hub is v0.11.0 (the prior "v0.6.3" reference
> was stale). Ground specifics against `felhom.eu/hub/` before relying on them.
> NOTE: this remains a design-draft for the deltas. It was written against hub v0.11.0 (2026-06-13); the deployed
> version is read from `manifests/hub.yaml`, never from this line (corrected 2026-10-24, review round 2: the old
> "the live hub is v0.11.0" was a current-state version literal). Ground specifics against `felhom.eu/hub/` before relying on them.
## 1. Source-of-truth model — two drivers, two directions
@@ -91,7 +92,7 @@ Evolves the existing staleness checker (60s **cadence**, a **configured** thresh
- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
liveness signal; a silent agent = the box is gone (the critical alert). **A hub restart does not re-mail a known
silence (R-1001, hub 0.162.0 — built 2026-10-19, not yet deployed):** a host whose newest `host_stale`/`host_down`
silence (R-1001, hub 0.162.0 — deployed since; corrected 2026-10-24):** a host whose newest `host_stale`/`host_down`
event is newer than its last report is seeded with that state at start, silently; `restore_test_stale` reads its edge
from the DB on the first sweep (a signal for the tier inside the read window and newer than its last proof). A failed
lookup lets the alarm through.
@@ -139,7 +140,7 @@ itself to the floor (R-1052: not yet proven end to end). Setting the global floo
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list
offers real agent packages only (R-1011, hub 0.162.0, not yet deployed):** the sha lookup asks for exactly the
offers real agent packages only (R-1011, hub 0.162.0, deployed since; corrected 2026-10-24):** the sha lookup asks for exactly the
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
(`artifact_file_missing`).
@@ -326,8 +327,9 @@ customer instead of a single controller.
- **DROP `reports` + CREATE `guest_reports`** (under the clean cutover this is drop+create with no data
migration, not an in-place rename); `guest_reports` adds `guest_id`, `host_id`; `cpu/memory` mean
guest-level; `backup_last_snapshot` goes quiet.
- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`; the
slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`.
- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`. ~~The
slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`~~ — retired, never built (§9
above, R-105) *(corrected 2026-10-24, review round 2)*.
- **DROP both** `infra_backup_versions` (current/live) **and** `infra_backups` (legacy mirror) — the DR
recipe, tenantsync and the escrow blob replace them (R-105).
- **KEEP** `customer_configs`, `events`, `customer_notifications`, `notification_log`,
@@ -350,7 +352,9 @@ customer instead of a single controller.
The box's console tells the volunteer to open „az e-mailben kapott link". That mail must exist whenever
a customer is **waiting for a box**, without an operator press. `autoMintSelfBindLink` runs on four
events: **customer creation**; **RESET completion**; **an e-mail set or changed on a customer with no
events (a fifth path mails a link too: the household's „new link" on a USED or EXPIRED link — one mail per customer
per hour, R-719; when the customer still has a linked host it shows the „oldbox" page in the customer's language and
tells the operator instead, R-1203 — `internal/web/selfbind.go` `handleBindResend`) *(corrected 2026-10-24, review round 2)*: **customer creation**; **RESET completion**; **an e-mail set or changed on a customer with no
bound host** (config save); **a host delete** that keeps the customer. The last two re-check that no
host is bound, so a customer who has a box is never sent a pairing link. **Appliance registration is
not a trigger** — it knows no customer (`api/appliance.go`). Every send is stored as a hub-internal
@@ -388,7 +392,7 @@ it has seen any box screen, and until v0.118.0 nothing here described that.
| Mail | Trigger | Rendered by | Language source |
|---|---|---|---|
| Event notification (39 event types) | a box event, or a hub checker | `FormatCustomerEmail` | reported → created-with → `hu` |
| Event notification (44 event types — the `mail.event.*` keys of `internal/i18n/locales/hu.json`, counted 2026-10-24) | a box event, or a hub checker | `FormatCustomerEmail` | reported → created-with → `hu` |
| Claim / reset / re-enroll / claimed | the claim arc | `FormatClaimEmail` | created-with (no box has reported yet) |
| Self-bind link | customer creation, or the operator's button | `FormatSelfBindEmail` | created-with |
| The public bind PAGE at `/bind/<token>` | the customer opens the link | one template per language | created-with, **except `expired`** — see 15.4 |
@@ -456,7 +460,7 @@ The operator mail renders the counts as one `Repair:` line. The household's side
date taken) are shown on its own app page, on its box, until it dismisses them — file names never leave the box.
### 15.4b A box whose tunnel to ep0 is dead — the clock-ahead alarm [hub 0.166.0, built, NOT deployed; R-1204]
### 15.4b A box whose tunnel to ep0 is dead — the clock-ahead alarm [hub 0.166.0, deployed with 0.167.0; R-1204 — corrected 2026-10-24]
**[FACT, hub `e267c5e2`]** The host report carries the agent's `wireguard` block (`hostWireguard`). `checkWGTunnel` (`api/host_tunnel.go`, called after `checkHostSwap`) calls the tunnel stale when the box reports `handshake_future_s` > 0 (the clock-ahead lock-out) or a last handshake older than 6 h (healthy boxes measured at most 113–124 s over 24 h). It is edge-triggered against the host's latest tunnel event (`store.LatestHostTunnelEvent`): `wg_tunnel_stale` once on the way down, `wg_tunnel_ok` once on the way back; both are allow-listed and operator-only (`notify.operatorOnlyEvents`, `wg_tunnel_operator_only_test.go`) — the household cannot repair a tunnel. A box with no `wireguard` block (an agent before the field, or no tunnel) raises nothing.
@@ -580,9 +584,13 @@ with the same `OFFSITE_SECRET_KEY`; today that key exists only on DooPlex (the k
on the same machine). The off-site plan for the database and its key: `runbooks/RUNBOOK-hub-db-offsite-backup.md`
(R-173 — option A decided 2026-10-05, `09` decision 125; §16.3).
### 16.3 The nightly database snapshot (R-173, hub v0.136.0)
### 16.3 The database snapshot — every 6 hours since D34 (R-173, hub v0.136.0; D34 in hub 0.167.0)
Every night at **02:00 Budapest** the hub writes `<data>/snapshots/hub-<UTC stamp>.db` with SQLite's `VACUUM INTO`:
**Corrected 2026-10-24 (review round 2):** since D34 (operator yes 2026-10-23) the hub writes a snapshot at **02:00,
08:00, 14:00 and 20:00 Budapest** (`internal/dbsnap/dbsnap.go` `Slots`), catches up at start when the newest is older
than **6 h** (`CatchUpAge`), and DooPlex pushes the newest at **:30 after each slot** (`felhom-hub-db-backup.timer`),
refusing one older than **7 h** (`felhom-hub-db-backup` `MAX_AGE_H`). The paragraph below is the original nightly
design; its mechanics hold per slot. Originally: every night at **02:00 Budapest** the hub writes `<data>/snapshots/hub-<UTC stamp>.db` with SQLite's `VACUUM INTO`:
one statement, one point in time, every committed write included — the rows still in `hub.db-wal` too, which a file copy
of `hub.db` loses (measured in `internal/dbsnap` tests: a plain copy held 0 of 150 fresh rows). Written as `.tmp`, mode
`0600`, then renamed, so a reader never sees half a file. The newest **2** stay (the volume grew from 1 GiB to 2 GiB
@@ -611,7 +619,7 @@ failure never fails or delays a request: lines go through a non-blocking buffer
### 16.5 Daily jobs and the clock change [DESIGN — hub v0.163.0, operator ruling D24 2026-10-21, R-1183]
Every daily hub job (`scheduleDaily`: db-snapshot 02:00, prune, deadline-check 05:00, offsite-key-audit 07:10) runs at
Every daily hub job (`scheduleDaily`: the four db-snapshot slots 02:00/08:00/14:00/20:00, prune at its configured time, deadline-check 05:00, offsite-key-audit 07:10 — slots corrected 2026-10-24) runs at
its Budapest wall-clock time **once per calendar day**: the next run is the calendar day after (`time.Date`, day+1),
never „now + 24 h", and the slot that ran last never runs again. On a 25-hour day (the last Sunday of October) the old
arithmetic ran every job after 02:00 twice; on a 23-hour day it ran them an hour late. Pinned by `TestR1183_*` for
@@ -19,7 +19,8 @@
> decisions; grounded against felhom.eu @ `bf099f6` and felhom-agent @ `4ba1b14` (v0.63.0).
> Evidence base: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (all
> measurements cited below as "spike Pn"). This document ends in an implementation slice roadmap
> (§8); **no slice is implemented yet** — each is a separate future task. Consistent with and
> (§8); **no slice is implemented yet** — each is a separate future task. *(Dated note 2026-10-24, review round 2: true when
> written; S1–S4 and S5's safe halves have since shipped — §8 and the agent's `internal/wgtunnel` say what is built.)* Consistent with and
> extending Part 01 (`01-topology-and-trust.md`); where this doc names a component's duty it uses
> Part 01 §3 vocabulary.
@@ -140,7 +141,9 @@ before this ships (the drilled `--rescope-acl` posture is unaffected).
4. **Hub → endpoint VM:** adds `{pubkey, /32}` to the WG server peer list (slice-1 surface).
5. **Box (agent):** next heartbeat sees the generation advance → fetches desired-state → writes
the WG config → `install` + `systemctl enable --now wg-felhom` (the sudoers `*.mount` pattern
of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3).
of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3). *(As built since R-861: no `install` sudoers
line; the conf is installed 0600 to its fixed path by the root wrapper `felhom-priv-apply wg` —
`felhom-agent/configs/felhom-agent.sudoers:223-229`; corrected 2026-10-24.)*
6. **Box:** tunnel handshakes (outbound, keepalive); PBS storage is (re)pointed at
`pbs_tunnel_ip:8007` (§3.4); first backup runs.
@@ -169,7 +172,8 @@ tenant ACL (`DatastoreBackup` on `/datastore/<ds>/<ns>` to BOTH the user and the
### 3.5 Key custody + DR re-establishment
The WG private key joins the **escrowed IdentityBundle**
(`internal/escrow/identity.go:24-27`, today `{TunnelToken, PBSToken}` → `+ WGPrivateKey`) — the
(`internal/escrow/identity.go`, then `{TunnelToken, PBSToken}` → `+ WGPrivateKey`; as built the bundle carries FOUR
fields — `TunnelToken`, `PBSToken`, `WGPrivateKey`, `ResticRepoPassword` — `identity.go:30-40`, corrected 2026-10-24) — the
same R-wrapped, zero-knowledge age blob the DR recipe already consumes. On host-loss
re-provisioning (01 §9 restore mode): the box recovers the bundle with the customer's recovery
code, restores the WG key, and re-establishes the tunnel; hub-side the peer entry still exists
@@ -199,7 +203,8 @@ disables the service, KEEPS its marker, and never re-registers on its own — re
revoked until the operator re-adds the peer (the heartbeat report carries the box's pubkey as
the recovery handle; **re-add via `POST /hosts/{id}/wg` with the global key** — the raw registry
add does not bump the host's generation). The only automatic re-registration is the
pubkey-mismatch (DR/re-key) path, backoff-bounded. Absent DATA (a failed fetch) is never a
pubkey-mismatch (DR/re-key) path, backoff-bounded — and, since D30 (agent v0.163.0), a key the agent makes itself after
a handshake stamped in the future for 15 min (`internal/wgtunnel/rekey.go`; corrected 2026-10-24). Absent DATA (a failed fetch) is never a
teardown signal. **Rollout gate:** `wg_tunnel.enabled` defaults FALSE agent-side until the
production endpoint exists.
@@ -285,7 +290,9 @@ production endpoint exists.
`wg-felhom` AllowedIPs so it survives the agent's self-heal ([OF-1]); PBS unaffected.
- **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one: no handshake
within ~3 keepalive periods → the offsite target reports unreachable → the existing alerting path
carries it. No new alarm channel.
carries it. No new alarm channel. *(Since hub 0.166.0 there is one: `wg_tunnel_stale` / `wg_tunnel_ok`,
operator-only — a handshake older than 6 h or stamped in the future, D30 / R-1204, `hub/internal/api/host_tunnel.go`;
corrected 2026-10-24.)*
**EXTENDED 2026-07-05 (TASK H1).** The agent's heartbeat now also carries an **`oob` stanza**
(`felhom_sshd_active`, `felhom_sshd_port`, `reachable`, `config_invalid`, `operator_peer_configured`,
`operator_key_configured`, `wg_handshake_age_s`) — the operator's "can I get into this box right
@@ -355,7 +362,8 @@ venue; it gates nothing. Runbook:
**Open sub-decisions (deferred by design):**
- Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — **slice-1 design point** (§5).
- WG key rotation cadence/mechanics — the reconcile channel can carry it (new pubkey → hub →
endpoint swap); design when fleet size warrants, possibly the moment the "separable fleet
endpoint swap); design when fleet size warrants *(one automatic rotation exists since D30: after a future-stamped
handshake for 15 min, at most once an hour — `rekey.go`; a scheduled rotation is still open; noted 2026-10-24)*, possibly the moment the "separable fleet
layer" (D1) conversation reopens.
- Endpoint scaling/sharding (N customers per VM, second region, HU-hosted option) — capacity
planning, not architecture; the peer registry already keys per-host.
@@ -378,6 +386,8 @@ venue; it gates nothing. Runbook:
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
**MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a
read-only prober that names itself "the seam for the tunnel-management slice") adopts the
S3-piloted agent-managed-service pattern in a later, separate slice.
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — then a
read-only prober that named itself "the seam for the tunnel-management slice") adopts the
S3-piloted agent-managed-service pattern in a later, separate slice. *(Dated note 2026-10-24: cloudflared is a GUEST
container the agent only reads — four states, `cloudflared.go:12-23`; `03` §2 says the agent will not manage it.
Whether this plan is withdrawn is the operator's — a row of review round 2.)*
+26 -9
View File
@@ -135,7 +135,9 @@ live 2026-08-23 — one event across 22 scans.
**The vocabulary is the HUB's and it is exact: `{info, warning, error, critical}`.** Anything else is
**coerced to `info` at ingest**, and `info` is dropped by `severityNotifies` before *both* delivery
legs. So a severity outside the set means the event is stored, answers `200`, shows on the dashboard —
legs — except the paired recoveries in `recoveredPairedDownTypes` (`node_recovered`, `host_recovered`,
`whole_guest_backup_recovered`, `pbsdr_box_recovered`, `offsite_box_recovered`, `storage_reconnected`,
`health_recovered`), which run before the gate and are mailed (`notify/dispatcher.go` — corrected 2026-10-24, review round 2). So a severity outside the set means the event is stored, answers `200`, shows on the dashboard —
and is e-mailed to **nobody**.
**This shipped twice.** `DiskAlertKind.Severity` emitted `"warn"` until controller v0.215.0;
@@ -155,8 +157,10 @@ Three things now hold it:
3. **The dispatcher's `unrecognized severity` branch is kept**, because the hub's own monitor checkers
call `ProcessEvent` directly and never pass the ingest handler. For them it is the only guard.
**Who gets it.** `processOperator` consults only `operatorOn`, the address and a 1-hour cooldown —
**never customer preferences** — so a valid severity always reaches the operator. `processCustomer`
**Who gets it.** `processOperator` consults only `operatorOn`, the address and a cooldown —
1 hour, except 5 min for node/host liveness and 6 h for a storage-fill warning (`operatorCooldownFor`); it records a
`backup_tier_skipped` in a box's FIRST hour as „suppressed" instead of mailing it (R-723), and the mail hold applies —
**never customer preferences** (cooldowns and R-723 corrected 2026-10-24) — so a valid severity always reaches the operator. `processCustomer`
consults `operatorOnlyEvents` and then the customer's `enabled_events`.
**`app_start_failed` is customer-switchable but OFF by default** [DESIGN, operator ruling 2026-08-23]:
@@ -180,7 +184,10 @@ rather than assumed (2026-08-30):
- **not** in `operatorOnlyEvents` — the operator leg ignores customer preferences anyway, so the
operator is always mailed; putting it here would only remove the customer's ability to opt in.
- **not** in `DefaultEnabledEvents` — customer-switchable, default OFF, the same ruling as
`app_start_failed`. The checkbox already exists at `settings_notifications.html:34`.
`app_start_failed`. **Superseded by operator Q1 (2026-10-12, R-388 slice 1):** since controller v0.309.0
`backup_integrity_failed` (with `crossdrive_failed`, `storage_reconnected`, `health_recovered`) IS in
`DefaultEnabledEvents` — the default for a household with no saved list (`internal/settings/settings.go`, pinned by
`TestR388_DefaultListAsGroups`; dated note 2026-10-24, review round 2). The checkbox already exists at `settings_notifications.html:34`.
**And a caveat that belongs in the alarm ladder rather than only in the backup document:** a
`backup_integrity_ok` at the shipped depth means *the index, the pack inventory and the snapshot graph
@@ -293,6 +300,7 @@ bands.
| app down (`app_start_failed`) | **per APP** | `…:<stack_name>` | no digest exists for it — see below |
| update outcome (`app_update_undone`, `app_update_held`) | **per APP**, both legs | `…:<stack_name>` | hub v0.120.0 — one mail per app per outcome; the household leg has its own register (`perAppCustomerCooldownEvents`) |
| OOM storm (`app_oom_storm`) | **per APP**, operator | `…:<stack_name>` | hub v0.121.0 — no digest; the controller already sends it at most once per container run |
| more per-APP types (added 2026-10-24 from `dispatcher.go`'s per-stack set) | **per APP** | `…:<stack_name>` | `app_hold_no_whole_copy`, `app_stopped_unhealthy` (both legs), `app_repaired_after_crash`, `app_unexplained_upload`, `app_unexplained_cpu` |
| backup run (`backup_run_failures`) | per RUN | `…:<run_id>` | a digest already lists every failing app; one per run |
| tiered backup (`whole_guest_backup_failed`, …) | per TIER | `…:<tier>` | the tiers fail independently and mean different things |
| everything else, incl. `crossdrive_failed` and `backup_integrity_failed` | per TYPE, per hour | — | coarse **on purpose** |
@@ -367,6 +375,14 @@ one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by
---
**More operator-only box alarms, listed 2026-10-24 (review round 2; each is in `notify/dispatcher.go`'s
`operatorOnlyEvents`, and each has its home elsewhere):** `wg_tunnel_stale` / `wg_tunnel_ok` — the box's tunnel to ep0:
a handshake older than 6 h or stamped in the future, edge-triggered (D30, R-1204; `api/host_tunnel.go`, `05` §15.4b);
`host_swap_short` / `host_swap_ok` — the host's swap below 90 % of min(the guests' swap allowance, 8192 MB) (R-1192;
`api/host_swap.go`); `bind_old_host_linked` — a household asked for a new bind link while its old host is still linked
(R-1203; `05` §13); `golden_not_vouched` — a published golden the hub does not vouch (decision 199, R-242);
`recovery_older_package` — a household's code opens, or may open, an older sealed package (decision 183, R-304 C).
## 6.4 A box that is not always on: the missed-backup deadline and the household's outage mail [DESIGN, hub v0.134.0, 2026-10-05]
Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *operator may reverse*).
@@ -376,7 +392,8 @@ Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *op
never judged (measured 2026-10-05: `1 skipped (down)`). Now a down box is judged on longer lines:
`expected_dbdump_missed` after **48 h** without a `db_dump_completed`, `expected_backup_missed` after **72 h**
without a whole-guest backup in any retained host report — never for a box first seen less than 48 h ago. A box that
died last night still raises only its staleness alarm. A `disabled` box is still skipped (R-321). Pinned by
died last night still raises only its staleness alarm. A `disabled` box is still skipped (R-321), and so is a box the household switched off on purpose — by its staleness
state or by the host row's hold (R-998, `monitor/deadline.go`; corrected 2026-10-24). Pinned by
`monitor/r872_down_box_test.go`.
- **R-873 — "your server cannot be reached" reaches the HOUSEHOLD at most once per 7 days** (`node_stale`,
`node_down`, `host_stale`, `host_down`, read from the persisted notification log, so a hub restart does not reset
@@ -387,7 +404,7 @@ Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *op
---
## 6.5 One lost off-site copy on a pinned tier [DESIGN, R-435 — `09` §3 decision 191, hub main 2026-10-08, unreleased]
## 6.5 One lost off-site copy on a pinned tier [DESIGN, R-435 — `09` §3 decision 191; released in hub v0.144.0 — corrected 2026-10-24]
On a **pinned** off-site tier (the hub holds a confirmed append-only key, decision 69) the only legitimate way the
snapshot count can fall is a clean-up window the hub opened (decision 68). So the hub raises
@@ -470,7 +487,7 @@ was ruled that way on its own merits and does not pre-judge the redesign.
**Filed as a PRODUCT DECISION, not a defect** — see the register. It is the operator's call to take
separately, and no part of it was implemented here.
### 8.1 The design — „only what you can act on" [DESIGN, R-388 — `09` §3 decision 200, 2026-10-12; NOT BUILT]
### 8.1 The design — „only what you can act on" [DESIGN, R-388 — `09` §3 decision 200, 2026-10-12; slice 1 BUILT in controller v0.309.0 — three group switches, details under Részletek (`internal/settings/notif_groups.go`); corrected 2026-10-24]
*Operator ruling 2026-10-12 (S5): design now, build in slices later. Written by CC from the direction above and the
live page (controller, `settings_notifications.html`: **18 event switches** plus the address, the cooldown and the
@@ -532,7 +549,7 @@ list is shown as custom and saved back unchanged; the default list maps to group
2. **The cooldown field („hány óránként legfeljebb"):** keep it on the page or move it under „Részletek"? Pick:
**under Részletek** — it is an engineering choice, which is what this redesign removes.
## 9. The upload and CPU watch — „does the app explain its own traffic?" [DESIGN — proposed; built in controller v0.342.0 + hub v0.169.0, NOT released]
## 9. The upload and CPU watch — „does the app explain its own traffic?" [DESIGN — built; released in controller v0.342.0 and hub v0.169.0 (deployed) on the operator's yes, 2026-10-24 — corrected 2026-10-24, review round 2]
**Why** (operator ruling 2026-10-23 19:22, `CONTEXT.md`): DooPlex's hacked Plex sent ~474 GB in a day with no viewer and
used ~10 cores with no film being converted (`audits/plex2shell-felhom-impact-2026-10-23/`). Nothing on a Felhom box
@@ -580,4 +597,4 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me
a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood
whatever the API says; not built. The box cannot tell LAN from internet bytes per app; a host-network container
(Samba) has no counter; the push floor is a lower bound, the weekly test is close to the line.
7. **Release:** not released; the operator's yes (R-1244).
7. **Release:** released 2026-10-24 on the operator's yes (controller v0.342.0, hub v0.169.0; R-1244) — corrected 2026-10-24.
@@ -20,8 +20,10 @@ workspace-root `CLAUDE.md` § „Artifact taxonomy". **[FACT]** The spec shape i
class). Validation is project Claude's unless CC is asked (`CLAUDE.md`, same section).
**A session with no spec** (a `/goal`, a night brief, „work the register") inherits the discipline a
spec used to carry from `.claude/rules/unprompted-work.md` — **[FACT]** five byte-identical copies:
the workspace root and each of the four repos (the file's own header says „change all five or none").
spec used to carry from `.claude/rules/unprompted-work.md` — **[FACT]** five copies:
the workspace root and each of the four repos (the file's own header says „change all five or none"). *(Measured
2026-10-24: they are NOT byte-identical — the felhom.eu and felhom-controller copies carry one more paragraph, „A
version number is pushed once" (added 2026-10-15); the review round 2 session aligned the other three, see its report.)*
## 2. Where an instruction lives
@@ -29,7 +31,7 @@ the workspace root and each of the four repos (the file's own header says „cha
|---|---|---|
| Cross-repo facts and standing rules (production host, gates, secrets, CHANGELOG/REPORT) | workspace-root `CLAUDE.md` | every session in the workspace |
| One repo's stable orientation | `<repo>/CLAUDE.md` | when a file in that repo is touched |
| Path-scoped rules (a subsystem's traps) | `<repo>/.claude/rules/*.md` | when a matching path is touched |
| Path-scoped rules (a subsystem's traps) | `<repo>/.claude/rules/*.md` | when a matching path is touched — a file with `unconditional: true` (e.g. `live-probes.md`) or without `paths:` (the `unprompted-work.md` copies) loads every session (corrected 2026-10-24) |
| A procedure with steps (build, test, diagnose, write for the operator) | a skill, `felhom.eu/skills/<name>/SKILL.md` | when its trigger matches, or by name |
| Current state, decisions, open work | `CONTEXT.md`, `STATUS.md`, `documentation/backlog/OPEN-ITEMS.md` | read on demand — never copied into an instruction file |
| Durable cross-session facts about the operator and the machines | the memory folder (`.claude-memory/`, index `MEMORY.md`) | the index each session; files on recall |
@@ -38,8 +40,9 @@ Which rung a piece of material belongs on is the doc-authoring skill's §3 („W
material sits") and §6 („One rule, one home"). This page does not repeat them.
**[FACT] Skills** are written in `felhom.eu/skills/`, installed by
`python3 felhom.eu/scripts/install_skills.py` as symlinks into `~/.claude/skills/` (so a repo edit is
live at once), and checked by `python3 felhom.eu/scripts/check_skills.py` (shape, the 150-line limit
`python3 felhom.eu/scripts/install_skills.py` as symlinks into `~/.claude/skills/` (so an edit to an INSTALLED skill is
live at once; a NEW skill folder is live only after `install_skills.py` runs again — measured 2026-10-24:
`skills/felhom-web-design` has no link yet), and checked by `python3 felhom.eu/scripts/check_skills.py` (shape, the 150-line limit
and its one grandfathered entry, R-394). The list is the folder; `skills/SOURCES.md` names where each
skill's material came from.
+10 -10
View File
@@ -47,8 +47,8 @@ scripts are held with the review's working notes.
| component | routes | no credential | listeners |
|---|---|---|---|
| controller | 231 (+8 in setup mode) | 21, of which 7 change state, each behind its own password, code or token | port 8080 on the guest's Docker networks (not published); mail relay 2525/2465/2526 |
| hub | 121 (1 health, 53 API, 67 web — v0.152.0 added the two token routes; both route tables are now pinned to the source by tests, R-952) | appliance register (rate-limited, by design), re-enroll (only while the operator has armed recovery mode), login, health | one ingress, path `/`, geo filter (HU, IE, LAN) |
| agent | 38 local-API routes (`POST /host/poweroff` added 2026-10-13, R-998) | 0 | local API on the island bridge; WireGuard; poke (tunnel only); felhom-sshd (tunnel only, measured) |
| hub | 121 (1 health, 53 API, 67 web — v0.152.0 added the two token routes; both route tables are now pinned to the source by tests, R-952) — *the API table has 54 entries since `/box-config/` (v0.161.0) and `/preferences` (v0.165.0); counted from `hub/internal/api/routes.go`, 2026-10-24* | appliance register (rate-limited, by design), re-enroll (only while the operator has armed recovery mode), login, health, and the public self-bind page `/bind/<token>` (a capability link, added to this list 2026-10-24) | one ingress, path `/`, geo filter (HU, IE, LAN) |
| agent | 41 local-API routes on `main` (counted 2026-10-24 in `felhom-agent/internal/localapi/server.go`; `POST /host/poweroff` added 2026-10-13, R-998; `POST /host/line-test/run` added 2026-10-24, R-1250, unreleased) | 0 | local API on the island bridge; WireGuard; poke (tunnel only); felhom-sshd (tunnel only, measured) |
**Outbound fetches that bring code or config onto a box:** the catalog (git, `main`), app images, the four infra
images, the controller image (by tag), `controller.yaml` from the hub, the agent binary and config bundle (signed),
@@ -58,11 +58,11 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
| boundary | verdict | evidence |
|---|---|---|
| end-user ↔ apps | holds, with gaps | traefik routing and header stripping hold; homepage has the Docker socket and no gate (R-981) |
| end-user ↔ apps | holds, with gaps | traefik routing and header stripping hold; homepage has the Docker socket and no gate (R-981). **Re-check (2026-10-24, review round 2):** the socket is gone from the homepage template (catalog `templates/homepage/docker-compose.yml:21`) and the box's allow-list is empty, so a socket bind is refused (`felhom-controller/controller/internal/privpolicy/allowlist.json`) |
| **added 2026-10-14 (R-1033, P1):** a visitor ↔ Cloudflare over plain http | holds on boxes with controller ≥ 0.312.0 | before: every public name answered `http://` (login pages too), and Sonarr/Radarr/Prowlarr redirected https visitors to an absolute `http://` login; a password could cross an untrusted network unencrypted (attacker: anyone on the visitor's network). Now traefik sends every `http://` request to https (301) before an app sees it, HSTS one day; boxes below 0.312.0 (Tester 2 at 0.292.0) keep the gap until the floor delivers it. `audits/https-only-2026-10-14/` |
| **added 2026-10-15 (R-1045):** a stranger ↔ a fresh Plex | holds on controller ≥ 0.313.0; on older ones only while the household does not press „Kész" before claiming | an unclaimed Plex answers everything to a private address with no token, and through the tunnel every request comes from one. The setup gate (catalog `9a23ee5`) closes the public path; 0.313.0 opens it only when Plex itself says claimed. A status probe meant to cover older controllers OPENED an unclaimed Plex while it started (`audits/media-build-2026-10-15/G7/`) and was removed |
| customer ↔ controller UI | partly | bcrypt and session hold; small gaps (R-983); family sessions survive a reset (R-976) |
| controller ↔ agent | partly | every route bound to the caller's guest; a compromised controller can still flush restore points (R-973), void the recovery code (R-978), and — since agent v0.155.0 — switch the whole machine off once per power-on, holding the hub's silence alarms for up to 14 days (R-998; mitigations and residual risk: `14-host-power-off.md` §5) |
| customer ↔ controller UI | partly | bcrypt and session hold; small gaps (R-983); family sessions survive a reset (R-976). **Re-check (2026-10-24, review round 2):** both fixed — a password change or a claim/reset ends every dashboard, family and gate session (controller v0.324.0, `internal/web/auth.go:352-366`) |
| controller ↔ agent | partly | every route bound to the caller's guest; a compromised controller can still flush restore points (R-973), void the recovery code (R-978) *(re-check 2026-10-24: both narrowed — a cadence cap on `POST /backup` since agent v0.160.0, `internal/localapi/backup_cap.go`; a new escrow package is uploaded only at the claim since agent v0.160.0, `escrow_ceremony.go`)*, and — since agent v0.155.0 — switch the whole machine off once per power-on, holding the hub's silence alarms for up to 14 days (R-998; mitigations and residual risk: `14-host-power-off.md` §5) |
| **added (R-1007):** controller → an app's own job queue | holds — nothing new | the `after_unclean_boot` commands run inside the app's container through the docker socket the controller already holds, and read the app's database with a read-only session: no key into the household's account, nothing a compromised controller could not already do (exec in any app, read its database). The household's lost-file names stay on the box (`app.yaml`, the app page); the operator event carries counts only, and a failing command's output is never logged (security review of controller `8d862e4`, fixed in `3b85a0c`) |
| **added:** agent ↔ root on the host | holds by the sudoers | every root verb is one regex-anchored line or a root wrapper that re-checks its input (`03` §3); agent v0.156.1 adds one delete: `felhom-vzdump-partial-clean` removes only a DEAD half-written vzdump of the named guest on a `dir` storage (no vzdump on the host, nothing holding it open) — a compromised agent gains the deletion of a leftover that is not a backup |
| agent ↔ hub | holds for destructive ops | signed jobs verified in the documented order; unsigned spec has no size bounds (R-984) |
@@ -70,13 +70,13 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
| controller ↔ hub | **did not hold until 2026-10-12** | the global key was public (R-950, rotated) and three routes needed no key (R-951, fixed); per-box keys are bound to their box (R-955) |
| box ↔ PBS | holds as written | ciphertext only; the zero-knowledge wording is stronger than a live box allows (R-985) |
| guest ↔ Proxmox host | holds | the guest has no Proxmox credentials (measured) |
| hub ↔ Cloudflare API | holds for the zone check | tokens are stored unsealed in the hub database (R-975); R-137 open |
| hub ↔ Cloudflare API | holds for the zone check | tokens are stored unsealed in the hub database (R-975); R-137 open. **Re-check (2026-10-24, review round 2):** the tokens are sealed (`enc:v1:`) since hub 0.168.0, deployed 2026-10-24 (`hub/internal/store/r975_config_token_seal.go`) |
| **added:** app ↔ app (shared Docker daemon) | partly | separate containers and networks; any docker-socket app is root in the guest (R-981) |
| **added:** app ↔ controller | partly | apps share a network with the controller; dashboard routes still need a session |
| **added:** LAN ↔ box | partly | root password SSH on the host from the LAN (R-982) |
| **added:** hub ↔ box (inbound) | **does not hold** | `01` §4 says the box exposes no control endpoint; a hub-key route can replace the box's settings (R-970) |
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971) |
| **added 2026-10-23 (R-1241):** a hacked app pod on DooPlex ↔ Felhom's secrets | **does not hold**; not used in the Plex2Shell window | two no-login doors: the Longhorn NFS export (the hub's unencrypted nightly copies with the unsealed tokens of R-975; by a published file-handle escape the whole sda1 disk, the operator signing keys included — not exercised) and the Longhorn API (backup a volume to any target). Plex2Shell, 2026-10-17 → 10-23: **NOT REACHED** — 0 new NFS connections in the window, no off-schedule backup or volume; the hub, the k8s API and the kubelet refuse a caller with no credential (measured). The export was narrowed to the pod network on 2026-10-23; the pod side stays open (`audits/plex2shell-felhom-impact-2026-10-23/`). **Felhom half, 2026-10-23 night:** the last plain tokens sealed in hub 0.168.0 (built, deploy pending — `audits/hub-token-seal-2026-10-23/`); the two plain Longhorn copies of the hub volume already in the folder are R-1243's decision. A household box's version of this attacker (a hacked app sending or mining) is what the upload watch (R-1244, not released) detects |
| **added:** LAN ↔ box | partly | root password SSH on the host from the LAN (R-982). **Re-check (2026-10-24, review round 2):** the config bundle's key-only drop-in shipped in agent v0.163.0 (`felhom-agent/configs/felhom-os-apply:230`) |
| **added:** hub ↔ box (inbound) | **does not hold** | `01` §4 says the box exposes no control endpoint; a hub-key route can replace the box's settings (R-970). **Re-check (2026-10-24, review round 2):** the config routes are removed and no route takes the hub key — pinned by `felhom-controller/controller/cmd/controller/r970_no_hub_key_door_test.go` (`01` §4 says the same) |
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971). **Re-check (2026-10-24, review round 2):** the box checks every rendered compose against its allow-list before any write since controller v0.328.0 (`internal/privpolicy/privpolicy.go`) |
| **added 2026-10-23 (R-1241):** a hacked app pod on DooPlex ↔ Felhom's secrets | **does not hold**; not used in the Plex2Shell window | two no-login doors: the Longhorn NFS export (the hub's unencrypted nightly copies with the unsealed tokens of R-975; by a published file-handle escape the whole sda1 disk, the operator signing keys included — not exercised) and the Longhorn API (backup a volume to any target). Plex2Shell, 2026-10-17 → 10-23: **NOT REACHED** — 0 new NFS connections in the window, no off-schedule backup or volume; the hub, the k8s API and the kubelet refuse a caller with no credential (measured). The export was narrowed to the pod network on 2026-10-23; the pod side stays open (`audits/plex2shell-felhom-impact-2026-10-23/`). **Felhom half, 2026-10-23 night:** the last plain tokens sealed in hub 0.168.0 (deployed 2026-10-24 — corrected 2026-10-24; `audits/hub-token-seal-2026-10-23/`); the two plain Longhorn copies of the hub volume already in the folder are R-1243's decision. A household box's version of this attacker (a hacked app sending or mining) is what the upload watch (R-1244, released 2026-10-24 in controller v0.342.0 + hub v0.169.0 — corrected 2026-10-24) detects |
## 6. What we accept (do not re-report)
@@ -1,5 +1,11 @@
# `05-hub-architecture.md` — critical review (grounded against felhom-hub v0.6.3 source + Parts 01–04)
> **HISTORICAL (banner added 2026-10-24, review round 2).** A dated review of hub v0.6.3. Many of its facts have moved
> since — for example the stale threshold is 45 min by config (code default 30 min), the operator cooldown is per event
> type (`notify/dispatcher.go` `operatorCooldownFor`), and the hub→controller push was retired in v0.26.0. Read
> `05-hub-architecture.md` and `08-alarm-ladder.md` for the current behaviour; this file is kept as the record of what
> the review found then.
Method: every claim about the existing hub was checked against `felhom.eu/hub/` source; every
cross-doc claim against Parts 01/03/04. Citations are `file:line`. Severity: **blocking** (wrong /
breaks an assumption) · **should-fix** (real gap or contradiction, low blast) · **minor**.
@@ -147,7 +147,7 @@ Title: **"Who tells a box what to do"**
## Bottom strip — "One night, hour by hour"
A horizontal timeline from **00:00 to 09:00**, two rows:
- **DooPlex row:** 00:00 database dumps · **00:20 DooPlex copy → ep0** · **02:30 hub database → ep0** · 03:30 ep0 prunes households · 03:45 ep0 prunes `operator` · **05:00 ep0-copy pull → DooPlex** · 07:30 ep0-copy prune · 08:00 deleted-customer clean-up. Sundays: 04:30 restore tests.
- **DooPlex row:** 00:00 database dumps · **00:20 DooPlex copy → ep0** · **02:30 hub database → ep0** (and again at 08:30; every 6 h since D34 — corrected 2026-10-24) · 03:30 ep0 prunes households · 03:45 ep0 prunes `operator` · **05:00 ep0-copy pull → DooPlex** · 07:30 ep0-copy prune · 08:00 deleted-customer clean-up. Sundays: 04:30 restore tests.
- **Customer box row:** **02:30 Tier 1** · **03:30 Tier 2** · **04:15 Tier 3 off-site** → app updates → **04:30–08:30 whole-guest backup** (local daily, off-site weekly) → Debian fixes → (when approved) Proxmox, Docker, kernel restart. 05:00 the hub checks every box made its backup.
- A small note: "Household mails about a kernel restart go out the day before, 09:00–20:00."
@@ -191,7 +191,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
<div style="position: absolute; left: 374px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 378px; top: 2040px; width: 220px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>00:20</b> DooPlex copy → ep0</div>
<div style="position: absolute; left: 894px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 898px; top: 2000px; width: 230px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>02:30</b> hub database → ep0</div>
<div style="position: absolute; left: 898px; top: 2000px; width: 230px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>02:30</b> hub DB → ep0, 6-hourly</div>
<div style="position: absolute; left: 1134px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 1138px; top: 2040px; width: 236px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>03:30</b> ep0 prunes households</div>
<div style="position: absolute; left: 1194px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
@@ -0,0 +1,17 @@
Sat Oct 24 12:46:20 UTC 2026
gitea.dooplex.hu/admin/felhom-controller:0.342.0
repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 4 hours (healthy)
filebrowser gtstef/filebrowser:1.5.6-stable Up 21 hours (healthy)
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 12 hours (healthy)
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 12 hours (healthy)
immich-redis redis:7-alpine Up 12 hours (healthy)
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 12 hours (healthy)
paperless-postgres postgres:18-alpine Up 12 hours (healthy)
paperless-redis redis:7-alpine Up 12 hours (healthy)
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 12 hours (healthy)
traefik traefik:v3.7.13 Up 21 hours
2628d11da1accbbb53a7fe9459b1a200 -
3
/dev/loop0 32G 2.7G 27G 10% /
/dev/nvme0n1 938G 227G 664G 26% /mnt/felhom-drives/scratch_hdd
@@ -0,0 +1,96 @@
# Document edits — review round 2 (2026-10-24)
Every edit below corrects a FACT the main session read at the cited `file:line` before editing (the reviewer's item id
in brackets; the candidate files are in `reviewers/`). Each edited place carries „corrected 2026-10-24" in the text.
Rules, fences, „never" lines and promises were NOT edited — those are rows for the operator.
## 05-hub-architecture.md (DOC-C1)
- [S5] „the live hub is v0.11.0" → written against v0.11.0; the deployed version is read from `manifests/hub.yaml` (`:135` = 0.169.0).
- [S4] „hub 0.162.0 — not yet deployed" ×2, §15.4b „0.166.0 NOT deployed" → deployed (hub CHANGELOG headers v0.164–v0.169).
- [C3] §11 „ADD … the slim DR record onto hosts" → struck through, retired (R-105, §9).
- [S3] `autoMintSelfBindLink` four events → plus the fifth path (R-719 / R-1203) — `hub/internal/web/selfbind.go:384-400`.
- [S12] „39 event types" → 44 (`mail.event.*` keys counted in `hub/internal/i18n/locales/hu.json`).
- [S1/C1] §16.3 nightly 02:00, catch-up 24 h, push 02:30 → four slots, catch-up 6 h, push :30 after each, refuse > 7 h — `hub/internal/dbsnap/dbsnap.go:31,36`; `hub/cmd/hub/main.go:964-968`; `scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10`.
- [S2] §16.5 `scheduleDaily` list → the four slots; prune at its configured time — `hub/cmd/hub/main.go:648,859,968,972`.
## 08-alarm-ladder.md (DOC-C1)
- [S11] info dropped before both legs → except the paired recoveries — `hub/internal/notify/dispatcher.go:137-167`.
- [S7] operator leg „only … a 1-hour cooldown" → per-type cooldown (5 min / 6 h / 1 h), R-723 first-hour suppression — `dispatcher.go:597-640`.
- [S9 — a dated note, the ruling line kept] backup_integrity_failed „default OFF" → superseded by operator Q1 2026-10-12: a default since controller v0.309.0 — `felhom-controller/controller/internal/settings/settings.go:713-721`.
- [S8] the per-APP grain table → five more per-app types — `dispatcher.go:470-480`.
- [S13] „a disabled box is still skipped" → also a box switched off on purpose (R-998) — `hub/internal/monitor/deadline.go:433-442`.
- [S10] §8.1 „NOT BUILT" → slice 1 built in v0.309.0 — `felhom-controller/controller/internal/settings/notif_groups.go:1-16`.
- [S6] §9 „NOT released" ×2 → released 2026-10-24 (controller CHANGELOG v0.342.0 header; `manifests/hub.yaml:135`).
- [U6/U7, listed as facts — each already has a ruling] the operator-only box alarms wg_tunnel_stale, host_swap_short, bind_old_host_linked, golden_not_vouched, recovery_older_package — `dispatcher.go:777,782,862,866,870`; `hub/internal/api/host_swap.go:25-37`.
- §6.5 header „hub main, unreleased" → released in hub v0.144.0 (hub CHANGELOG `:629` inside the v0.144.0 section).
## _hub-review.md (DOC-C1 S14)
- A HISTORICAL banner (stale threshold 45 m by config — `manifests/hub.yaml:88`; cooldown per type; push retired). The body is kept as the record.
## felhom-system-poster.facts.md + .html (DOC-C1 C2) — text only (the poster rule §6: facts file and matching html text)
- Timeline „02:30 hub database → ep0" → „… (and again at 08:30; every 6 h since D34)"; html „02:30 hub DB → ep0, 6-hourly".
## 03-host-agent.md (DOC-B)
- [S1] cloudflared three states → four (`absent`, v0.161.0) — `felhom-agent/internal/hub/cloudflared.go:12-23`.
- [S2] D4 „ready patch, NOT shipped" → shipped in v0.163.0 (`git tag --contains 1109092`) + installer-v1.36.0 (`manifests/webpage.yaml:347`).
- [S3/S4] `a0c88e5`, `ae69ad4` „unreleased" → in v0.163.0 (`git tag --contains`).
- [S5] old key `wg/private.key.prev` → first old key only; later `.prev.<unix>`, newest three; `wg/last-rekey` — `rekey.go:39,105-139`.
- [S6] host_busy „Unreleased 56aabc7" → v0.161.0; plus the line-test holder — `internal/linetest/linetest.go:62`.
- [S7] R-1010 „Unreleased 56aabc7" → v0.161.0.
- [S8] local API „default :8443" → `listen_addr` required, no default — `internal/config/config.go:317-319`.
- [S9] guest swap „once a minute after start" → once, one minute after start — `cmd/felhom-agent/main.go:940-955`.
- NOT edited (RULE → row): S10 break-glass „via the Cloudflare Tunnel behind Access".
## 06-offsite-connectivity.md (DOC-B) — dated notes; the design-of-record text kept
- [S11] „no slice is implemented yet" → note: S1–S4 shipped.
- [S12] sudoers `install … *.mount` → note: `felhom-priv-apply wg` — `felhom-agent/configs/felhom-agent.sudoers:223-229`.
- [S14] IdentityBundle `{TunnelToken, PBSToken}` → four fields — `felhom-agent/internal/escrow/identity.go:30-40`.
- [S15] the only automatic re-registration → plus D30 — `rekey.go`.
- [S16] „No new alarm channel" → `wg_tunnel_stale/ok` — `hub/internal/api/host_tunnel.go`.
- [S17] key rotation „design when fleet size warrants" → note: the D30 rotation exists.
- [S18] cloudflared „seam" → note: a guest container the agent only reads; withdrawing the plan is the operator's (row).
- NOT edited (RULE → row): S19 „no inbound listener … box→PBS only".
## 12-agent-tooling.md (DOC-B)
- [S20] „five byte-identical copies" → measured not identical; aligned (below).
- [S21] rules load „when a matching path is touched" → `unconditional: true` / no `paths:` load always — `felhom.eu/.claude/rules/live-probes.md:2`.
- [S22] skills „live at once … the list is the folder" → a NEW skill needs `install_skills.py` again (`felhom-web-design` has no link; not installed — no DooPlex change in this run).
## Instruction files (unprompted-work §5: a stale fact corrected / a proved fact added — named here)
- `felhom-agent/.claude/rules/unprompted-work.md`, `app-catalog-felhom.eu/.claude/rules/unprompted-work.md`, workspace root
`.claude/rules/unprompted-work.md` (unversioned): the missing item-6 paragraph „A version number is pushed once …"
added (before: 95 lines, md5 3b071ad2; after: byte-identical to the felhom.eu and controller copies, md5 b1891bd4).
Why: the file's own header says „change all five or none" and two copies had drifted; the paragraph states a fact
(`felhom-controller/controller/build.sh:115-125` refuses an existing tag and an unreachable registry).
- `felhom.eu/.claude/rules/live-probes.md:41`: before „no `/opt/felhom/stacks/<name>`"; after „no
`/opt/docker/stacks/<name>/app.yaml` (the folder itself stays — it holds the catalog's template files)". Why: the path
is `/opt/docker/stacks` (`felhom-controller/controller/internal/config/config.go:393`) and the folder stays after a
clean removal (the catalog's `box_walk.remove` checks `app.yaml` since 2026-10-24).
## 01-topology-and-trust.md (DOC-A)
- [S7] `docs/proxmox-platform.md` → `documentation/proxmox-platform.md` (the file exists; `docs/` does not).
- [S6] „every /api/ route needs a dashboard session" → but the read-only `/api/health` — `felhom-controller/controller/cmd/controller/main.go:4236`.
- [S1] claim mail link `templates.go:27` → `FormatClaimEmail` (`hub/internal/notify/templates.go:250-252`).
- [S2] the retrieval code „single-use … TTL" → a dated note: not burned, no TTL; D19/R-1159 rule — `hub/internal/api/passphrase_scope.go:19-30`.
- [S3] Cloudflare token reach check „hub main, unreleased" → released in hub v0.144.0.
- [S4] dead pointer `07:296-299` → a dated note.
- [S5] snapshot-before-deploy → [DESIGN — not built]: the agent serves `/snapshot`, `/rollback`; no controller caller (grep of `internal/agentapi`).
- NOT edited (RULE/promise → rows): C3 „the hub never connects into a box / no inbound control endpoint / no standing operator backdoor" vs poke + felhom-sshd; C4 „the operator cannot open the off-site backup" vs R-985.
## 04-control-plane-authorization.md (DOC-A)
- [S8] `docs/tests/…`, `docs/architecture/…` → `documentation/…`.
- [S10] time window → the 2-minute skew on `issued_at` — `felhom-agent/internal/authz/verifier.go:19-22,148-155`.
- [U2 → DESIGN — undocumented note + row] no cap on the window's length; opsign default 30 min — `felhom-agent/cmd/felhom-opsign/main.go:57`.
- [S9] §6.1 „controller + hub main, unreleased" → released in controller v0.304.0 + hub v0.144.0 (both CHANGELOGs).
## 13-threat-model.md (DOC-A) — facts corrected; every verdict cell LEFT AS IT WAS and marked „re-check" (a verdict is a judgement)
- [S11] „38 local-API routes" → 41 on main (`felhom-agent/internal/localapi/server.go`).
- [S12] hub routes → 54 API entries; `/bind/<token>` added to the no-credential list.
- [S13] homepage socket (R-981) → gone; the allow-list empty — catalog `templates/homepage/docker-compose.yml:21`; `privpolicy/allowlist.json:4`.
- [S14] R-983 / R-976 → fixed (controller v0.324.0, `internal/web/auth.go:352-366`).
- [S15] R-973 / R-978 → narrowed in agent v0.160.0 (`git tag --contains f17e197 / 8561dc8`).
- [S16] tokens unsealed (R-975) → sealed since hub 0.168.0 (deployed).
- [S17] root password SSH (R-982) → the drop-in shipped in agent v0.163.0.
- [S18] R-970 „does not hold" → the routes are removed, pinned by `r970_no_hub_key_door_test.go`.
- [S19] R-971 „design gap" → `privpolicy.Admit` before every write (controller v0.328.0).
- [S20] „hub 0.168.0 deploy pending", „R-1244 not released" → deployed / released 2026-10-24.
@@ -0,0 +1,80 @@
package backup
// REVIEW ROUND 2 (2026-10-24), HUNT-1 H1 — an EVIDENCE test kept in the audit, NOT committed to the repo.
// A restore of a DRIVE-backed app is cut after its undo copy (kept on the app's drive) is recorded. At the next
// start the drive is not bound yet: Recover finds the copy „gone", the gated start refuses (drive absent), the
// marker and its Undo are KEPT. The drive then binds and the app runs (the boot recreate checks only „held") and the
// household writes. At the NEXT controller start Recover finds the copy present and pours it back.
import (
"io"
"log"
"os"
"path/filepath"
"testing"
)
type recordingPutBacker struct{ calls []string }
func (p *recordingPutBacker) PutBackInterruptedRestore(stack, dir string) error {
p.calls = append(p.calls, stack+"@"+dir)
return nil
}
func TestReview2H1_DriveBackedCutRestore_PutBackRunsAfterTheAppAlreadyRan(t *testing.T) {
root := t.TempDir()
marker := filepath.Join(root, "appstop-state.json")
drive := filepath.Join(root, "drive")
undo := filepath.Join(drive, "ns", "backups", "restore-undo", "immich")
if err := os.MkdirAll(undo, 0o755); err != nil {
t.Fatal(err)
}
quiet := log.New(io.Discard, "", 0)
// the restore: Begin, the copy recorded, then the power cut (no End)
g0 := NewAppStopGuard(marker, quiet)
if err := g0.Begin("restore-1", ReasonUnitRestore, []string{"immich"}); err != nil {
t.Fatal(err)
}
if err := g0.SetUndo("immich", undo); err != nil {
t.Fatal(err)
}
// boot 1: the drive is not bound yet (its path is absent in the controller's view)
if err := os.Rename(drive, drive+".unbound"); err != nil {
t.Fatal(err)
}
pb1 := &recordingPutBacker{}
g1 := NewAppStopGuard(marker, quiet)
g1.SetStarter(&gatingStarter{refuse: map[string]string{"immich": "the data drive is not present"}})
g1.SetPutBacker(pb1)
r1 := g1.Recover()
t.Logf("boot 1: put-backs=%v refused=%v failed=%v restarted=%v marker kept=%v", pb1.calls, r1.Refused, r1.Failed, r1.Restarted, r2Exists(marker))
if len(pb1.calls) != 0 || len(r1.Refused) != 1 || !r2Exists(marker) {
t.Fatalf("boot 1 did not take the copy-gone, refused, marker-kept path")
}
// the drive binds; the boot recreate starts the app (it checks only appHeld) and the household writes
if err := os.Rename(drive+".unbound", drive); err != nil {
t.Fatal(err)
}
householdWrite := filepath.Join(drive, "ns", "new-photo-after-boot")
if err := os.WriteFile(householdWrite, []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
// boot 2 (any controller restart, e.g. the nightly self-update): the marker is still active
pb2 := &recordingPutBacker{}
g2 := NewAppStopGuard(marker, quiet)
g2.SetStarter(&fakeStarter{})
g2.SetPutBacker(pb2)
r2 := g2.Recover()
t.Logf("boot 2: put-backs=%v restarted=%v marker kept=%v", pb2.calls, r2.Restarted, r2Exists(marker))
if len(pb2.calls) == 1 {
t.Logf("DEFECT SHOWN: the copy taken BEFORE the restore is poured back AFTER the app already ran on the drive and the household wrote (%s)", householdWrite)
} else {
t.Fatalf("no put-back at boot 2 — the chain does not hold as described")
}
}
func r2Exists(p string) bool { _, err := os.Stat(p); return err == nil }
@@ -0,0 +1,8 @@
controller main 321df1f, lean clone, go test -run TestReview2H1 -v
=== RUN TestReview2H1_DriveBackedCutRestore_PutBackRunsAfterTheAppAlreadyRan
review2_h1_drive_crash_test.go:52: boot 1: put-backs=[] refused=[immich] failed=[] restarted=[] marker kept=true
review2_h1_drive_crash_test.go:72: boot 2: put-backs=[immich@<tmp>/drive/ns/backups/restore-undo/immich] restarted=[immich] marker kept=false
review2_h1_drive_crash_test.go:74: DEFECT SHOWN: the copy taken BEFORE the restore is poured back AFTER the app already ran on the drive and the household wrote (<tmp>/drive/ns/new-photo-after-boot)
--- PASS: TestReview2H1_DriveBackedCutRestore_PutBackRunsAfterTheAppAlreadyRan (0.00s)
PASS
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.006s
@@ -0,0 +1,6 @@
14:57:09 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH']
14:57:10 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
14:57:40 [1] deployed, controller state=running, pinned={'audiobookshelf': 'ghcr.io/advplyr/audiobookshelf:2.37.1'}
deploy: True
-rw-r--r-- 1 root 1000 46 Oct 24 12:57 /mnt/felhom-drives/scratch_hdd/userdata/media/audiobooks/review2-h1-marker.txt
@@ -0,0 +1,39 @@
night chain -> 202 {'data': {'legs': ['db-dump', 'tier2', 'update-leg']}, 'message': 'started', 'ok': True}
2026/10/24 12:57:55 night_chain.go:108: [INFO] [night-chain] manual run started from 172.18.0.4:40216: [db-dump tier2 update-leg]
2026/10/24 12:57:55 night_chain.go:84: [INFO] [night-chain] manual run of tonight's chain: dump → second drive → off-site → update leg
2026/10/24 12:57:55 night_chain.go:77: [INFO] [night-chain] db-dump: started
2026/10/24 12:58:50 night_chain.go:82: [INFO] [night-chain] db-dump: done in 55s
2026/10/24 12:58:50 night_chain.go:77: [INFO] [night-chain] tier2: started
2026/10/24 12:58:52 tier2.go:507: [INFO] [backup] Tier 2 copied audiobookshelf → /mnt/sizing-bench/backups/secondary/audiobookshelf (484.0 KB, 1 leg(s), 2s)
2026/10/24 12:59:11 tier2.go:507: [INFO] [backup] Tier 2 copied immich → /mnt/felhom-drives/scratch_hdd/backups/secondary/immich (1.1 GB, 2 leg(s), 19s)
2026/10/24 12:59:14 tier2.go:507: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sizing-bench/backups/secondary/paperless-ngx (128.2 MB, 1 leg(s), 3s)
2026/10/24 12:59:15 tier2.go:507: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/scratch_hdd/backups/secondary/privatebin (8.4 KB, 0 leg(s), 1s)
2026/10/24 12:59:15 tier2.go:569: [INFO] [backup] Tier 2 run complete: 4 app(s) processed (incl. volume-only — F6)
2026/10/24 12:59:15 night_chain.go:82: [INFO] [night-chain] tier2: done in 24s
2026/10/24 12:59:15 night_chain.go:90: [INFO] [night-chain] offsite: no off-site target on this box — skipped, as at night
2026/10/24 12:59:15 night_chain.go:77: [INFO] [night-chain] update-leg: started
2026/10/24 12:59:15 night_chain.go:82: [INFO] [night-chain] update-leg: done in 0s
2026/10/24 12:59:15 night_chain.go:93: [INFO] [night-chain] finished in 1m20s
/mnt/sizing-bench/backups/secondary/audiobookshelf
/mnt/sizing-bench/backups/secondary/audiobookshelf/.felhom-tier2-layout
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/.felhom.yml
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/app.yaml
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/docker-compose.yml
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/data-stamps.json
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/manifest.json
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps/audiobookshelf_audiobookshelf_config.tar
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps/audiobookshelf_audiobookshelf_metadata.tar
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media/audiobooks
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media/audiobooks/review2-h1-marker.txt
total 12
drwxrwsr-x 2 root 1000 4096 Oct 24 12:57 .
drwxr-xr-x 3 root root 4096 Oct 24 12:58 ..
-rw-r--r-- 1 root 1000 46 Oct 24 12:57 review2-h1-marker.txt
@@ -0,0 +1,39 @@
night chain -> 202 {'data': {'legs': ['db-dump', 'tier2', 'update-leg']}, 'message': 'started', 'ok': True}
2026/10/24 12:59:38 night_chain.go:108: [INFO] [night-chain] manual run started from 172.18.0.4:40216: [db-dump tier2 update-leg]
2026/10/24 12:59:38 night_chain.go:84: [INFO] [night-chain] manual run of tonight's chain: dump → second drive → off-site → update leg
2026/10/24 12:59:38 night_chain.go:77: [INFO] [night-chain] db-dump: started
2026/10/24 13:00:40 night_chain.go:82: [INFO] [night-chain] db-dump: done in 1m2s
2026/10/24 13:00:40 night_chain.go:77: [INFO] [night-chain] tier2: started
2026/10/24 13:00:41 tier2.go:507: [INFO] [backup] Tier 2 copied audiobookshelf → /mnt/sizing-bench/backups/secondary/audiobookshelf (487.0 KB, 1 leg(s), 1s)
2026/10/24 13:00:56 tier2.go:507: [INFO] [backup] Tier 2 copied immich → /mnt/felhom-drives/scratch_hdd/backups/secondary/immich (1.1 GB, 2 leg(s), 15s)
2026/10/24 13:00:58 tier2.go:507: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sizing-bench/backups/secondary/paperless-ngx (128.2 MB, 1 leg(s), 2s)
2026/10/24 13:00:59 tier2.go:507: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/scratch_hdd/backups/secondary/privatebin (8.4 KB, 0 leg(s), 1s)
2026/10/24 13:00:59 tier2.go:569: [INFO] [backup] Tier 2 run complete: 4 app(s) processed (incl. volume-only — F6)
2026/10/24 13:00:59 night_chain.go:82: [INFO] [night-chain] tier2: done in 19s
2026/10/24 13:00:59 night_chain.go:90: [INFO] [night-chain] offsite: no off-site target on this box — skipped, as at night
2026/10/24 13:00:59 night_chain.go:77: [INFO] [night-chain] update-leg: started
2026/10/24 13:00:59 night_chain.go:82: [INFO] [night-chain] update-leg: done in 0s
2026/10/24 13:00:59 night_chain.go:93: [INFO] [night-chain] finished in 1m21s
/mnt/sizing-bench/backups/secondary/audiobookshelf
/mnt/sizing-bench/backups/secondary/audiobookshelf/.felhom-tier2-layout
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/.felhom.yml
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/app.yaml
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/docker-compose.yml
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/data-stamps.json
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/manifest.json
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps/audiobookshelf_audiobookshelf_config.tar
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps/audiobookshelf_audiobookshelf_metadata.tar
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media/audiobooks
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media/audiobooks/review2-h1-marker.txt
total 12
drwxrwsr-x 2 root 1000 4096 Oct 24 12:57 .
drwxr-xr-x 3 root root 4096 Oct 24 12:58 ..
-rw-r--r-- 1 root 1000 46 Oct 24 12:57 review2-h1-marker.txt
@@ -0,0 +1,49 @@
28- # If Viktor rules excluded, flip THIS LINE ONLY before committing.
29: - path: media/review2-typo
30- class: optional # REVIEW-2 H1 PROOF: a path no compose bind has
night chain -> 202 {'data': {'legs': ['db-dump', 'tier2', 'update-leg']}, 'message': 'started', 'ok': True}
2026/10/24 13:01:28 night_chain.go:108: [INFO] [night-chain] manual run started from 172.18.0.4:40216: [db-dump tier2 update-leg]
2026/10/24 13:01:28 night_chain.go:84: [INFO] [night-chain] manual run of tonight's chain: dump → second drive → off-site → update leg
2026/10/24 13:01:28 night_chain.go:77: [INFO] [night-chain] db-dump: started
2026/10/24 13:01:30 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
2026/10/24 13:01:31 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
2026/10/24 13:01:31 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
2026/10/24 13:01:32 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
2026/10/24 13:01:32 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
2026/10/24 13:01:32 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
2026/10/24 13:01:32 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
2026/10/24 13:01:59 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
2026/10/24 13:02:24 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
2026/10/24 13:02:24 night_chain.go:82: [INFO] [night-chain] db-dump: done in 57s
2026/10/24 13:02:24 night_chain.go:77: [INFO] [night-chain] tier2: started
2026/10/24 13:02:24 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
2026/10/24 13:02:25 tier2.go:329: [INFO] [backup] Tier 2 reconcile: removed stale dest dir userdata/media
2026/10/24 13:02:25 tier2.go:507: [INFO] [backup] Tier 2 copied audiobookshelf → /mnt/sizing-bench/backups/secondary/audiobookshelf (489.9 KB, 0 leg(s), 1s)
2026/10/24 13:02:46 tier2.go:507: [INFO] [backup] Tier 2 copied immich → /mnt/felhom-drives/scratch_hdd/backups/secondary/immich (1.1 GB, 2 leg(s), 20s)
2026/10/24 13:02:49 tier2.go:507: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sizing-bench/backups/secondary/paperless-ngx (128.2 MB, 1 leg(s), 3s)
2026/10/24 13:02:50 tier2.go:507: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/scratch_hdd/backups/secondary/privatebin (8.4 KB, 0 leg(s), 0s)
2026/10/24 13:02:50 tier2.go:569: [INFO] [backup] Tier 2 run complete: 4 app(s) processed (incl. volume-only — F6)
2026/10/24 13:02:50 night_chain.go:82: [INFO] [night-chain] tier2: done in 25s
2026/10/24 13:02:50 night_chain.go:90: [INFO] [night-chain] offsite: no off-site target on this box — skipped, as at night
2026/10/24 13:02:50 night_chain.go:77: [INFO] [night-chain] update-leg: started
2026/10/24 13:02:50 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
2026/10/24 13:02:50 night_chain.go:82: [INFO] [night-chain] update-leg: done in 0s
2026/10/24 13:02:50 night_chain.go:93: [INFO] [night-chain] finished in 1m22s
/mnt/sizing-bench/backups/secondary/audiobookshelf
/mnt/sizing-bench/backups/secondary/audiobookshelf/.felhom-tier2-layout
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/.felhom.yml
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/app.yaml
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/docker-compose.yml
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/data-stamps.json
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/manifest.json
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps/audiobookshelf_audiobookshelf_config.tar
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps/audiobookshelf_audiobookshelf_metadata.tar
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata
ls: cannot access '/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media/audiobooks/': No such file or directory
@@ -0,0 +1,14 @@
0
restored
stop ('200', {'ok': True, 'message': 'Stack audiobookshelf stop completed'})
remove ('200', {'ok': True, 'data': {'removed': 'audiobookshelf', 'volumes_removed': ['audiobookshelf_audiobookshelf_config', 'audiobookshelf_audiobookshelf_metadata'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'userdata_kept': ['/mnt/felhom-drives/scratch_hdd/userdata/media/audiobooks (8.0K)', '/mnt/felhom-drives/scratch_hdd/userdata/media/podcasts (4.0K)'], 'backup_paths_removed': ['/mnt/felhom-drives/scratch_hdd/backups/primary/audiobookshelf (520K)', '/mnt/sizing-bench/backups/secondary/audiobookshelf (489.9 KB)'], 'verified': True}, 'message': 'Stack audiobookshelf removed'})
total 8
drwxrwsr-x 2 root 1000 4096 Oct 24 13:04 .
drwxrwsr-x 16 root 1000 4096 Oct 13 19:26 ..
paperless-ngx
docker-compose.yml
0
0
ls: cannot access '/mnt/felhom-drives/scratch_hdd/appdata/audiobookshelf': No such file or directory
@@ -0,0 +1,37 @@
# DOC-A — 01 topology & trust, 04 control-plane authorization, 13 threat model (reviewer candidates; verified by the main session)
Reviewer: read-only subagent, 2026-10-24. Counts: claims 62 · true 33 · stale 20 · untestable 6 · undocumented 4 · contradictions 4.
## STALE (all FACT)
- S1 01:135 claim mail link `templates.go:27` → :252
- S2 01:220-223 passphrase „single-use (burned on the successful config fetch) plus a short TTL" → not burned; keys until a host exists and a controller ≥ 0.334.0 reported, then 409 (R-1159, D19) — hub/internal/api/passphrase_scope.go:19-29,46-66
- S3 01:256 Cloudflare reach check „hub main, unreleased" → released v0.144.0
- S4 01:315-316 pointer `07:296-299` „40 of 53 … all-hot" → dead pointer
- S5 01:345-346 snapshot-before-deploy / rollback → agent serves /snapshot,/rollback; no controller caller — [DESIGN], not built
- S6 01:108 „every /api/ route needs a dashboard session" → /api/health is open (read-only) — cmd/controller/main.go:4236
- S7 01:19 `docs/proxmox-platform.md` → `documentation/proxmox-platform.md`
- S8 04:18-20 `docs/tests/…`, `docs/architecture/…` → `documentation/…`
- S9 04:169 operator actions „unreleased" → hub v0.144.0, controller v0.304.0
- S10 04:79 „now ∈ [issued_at, expires_at]" → 2-minute skew allowance — felhom-agent/internal/authz/verifier.go:21,148-155; configs/felhom-os-apply:658
- S11 13:51 „38 local-API routes" → 41 — felhom-agent/internal/localapi/server.go:562-634
- S12 13:50 hub route counts / no-credential list → 54 API routes; the public `/bind/<token>` page missing from the list
- S13 13:61 homepage socket (R-981) → gone; allow-list empty
- S14 13:64 R-983 gaps, R-976 family sessions survive a reset → both fixed (auth.go:352-366)
- S15 13:65 R-973 flush, R-978 void the code → cadence cap; package uploaded only at the claim
- S16 13:73 tokens unsealed (R-975) → sealed `enc:v1:` (hub 0.168.0, deployed)
- S17 13:76 root password SSH from the LAN (R-982) → the bundle's key-only drop-in
- S18 13:77 hub-key route can replace settings (R-970) → removed; pinned by cmd/controller/r970_no_hub_key_door_test.go
- S19 13:78 no box check on a catalog push (R-971) → privpolicy.Admit
- S20 13:79 hub 0.168.0 „deploy pending", R-1244 „not released" → deployed / shipped
## UNDOCUMENTED
- U1 FileBrowser SSO trusts a proxy header; the header NAME is the per-box secret; any app container reaches filebrowser:80 — controller internal/infra/filebrowser_sso.go:15-22; internal/web/files_sso.go:20-44 — 01 §5, 13 §5
- U2 a signed op's window has no maximum length (opsign default TTL 30 m) — felhom-agent/internal/authz/verifier.go:148-155 — 04 §2
- U3 „poke": the hub reaches into a box (SSH to ep0 → unauthenticated UDP to the box's WG /32:51822) — hub/internal/poke/client.go:1-16; agent poke.go:35 — 01 §4/§5
- U4 the guest can start a ~55 s line test on request (R-1250) — agent internal/localapi/linetest.go:67-73
## CONTRADICTIONS
- C1 01:105-109 R-970 closed vs 13:77 „does not hold" — code: closed
- C2 04:195 sessions end on a reset vs 13:64 „family sessions survive" — code: end
- C3 01:26-27,102,244-245,353 „the hub never connects into a box / no inbound control endpoint / no standing operator backdoor" vs 06:279-283 operator WG peer reaches felhom-sshd; 13:51 poke + felhom-sshd — RULE (a „never" line, a product value) → OPERATOR
- C4 01:328-329,352 the operator „can't open / cannot read" the off-site backup vs 13:71 + R-985 (a live box lets the operator reach the key) — RULE (a customer promise) → OPERATOR
@@ -0,0 +1,49 @@
# DOC-B — 03 host agent, 06 off-site connectivity, 12 agent tooling (reviewer candidates; verified by the main session before acting)
Reviewer: read-only subagent, 2026-10-24. Counts: claims 79 · true 45 · stale 22 · untestable 6 · undocumented 8 · contradictions 6.
`A/` = felhom-agent/.
## STALE
- S1 03:50,226 cloudflared three states → four (`absent`, R-1070, v0.161.0) — A/internal/hub/cloudflared.go:12-23 — FACT
- S2 03:132 D4 drop-in „ready patch, NOT shipped" → shipped (agent v0.163.0 `1109092`, installer-v1.36.0) — A/configs/felhom-os-apply:189,230; manifests/webpage.yaml:347 — FACT
- S3 03:134 `a0c88e5` „unreleased" → in v0.163.0 — FACT
- S4 03:136 `ae69ad4` „unreleased" → in v0.163.0 — FACT
- S5 03:137-138 old key `wg/private.key.prev` → only the first; later `private.key.prev.<unix>`, newest 3; bound in `wg/last-rekey` — A/internal/wgtunnel/rekey.go:107,137-139,161-173 — FACT
- S6 03:252-254 host_busy „Unreleased 56aabc7" → in v0.161.0; holder `line-test` too — A/internal/linetest/linetest.go:62 — FACT
- S7 03:423-424 R-1010 „Unreleased 56aabc7" → in v0.161.0 — FACT
- S8 03:316-317 local API „default :8443" → no default, `listen_addr` required; island 169.254.253.1:8443 — A/internal/config/config.go:251,260-269,319 — FACT
- S9 03:680 „once a minute after agent start" → ONCE, one minute after start — A/cmd/felhom-agent/main.go:940-955 — FACT
- S10 03:236-237 break-glass „via the Cloudflare Tunnel behind Access" → felhom-sshd over wg-felhom — A/internal/felhomsshd/claim.go:7 — RULE
- S11 06:19-21 „no slice is implemented yet" → S1–S4 shipped — FACT
- S12 06:69,142-143 sudoers `install … *.mount` → R-861 removed; `felhom-priv-apply wg` — A/configs/felhom-agent.sudoers:15-20,227-233 — FACT
- S13 06:71 report.go line citations moved (:628, :659-666) — FACT
- S14 06:172 IdentityBundle `{TunnelToken, PBSToken}` → 4 fields incl. ResticRepoPassword — A/internal/escrow/identity.go:30-40 — FACT
- S15 06:201-202 „only automatic re-registration is the pubkey-mismatch path" → D30 too — rekey.go:88-102 — FACT
- S16 06:286-288 „No new alarm channel" → hub wg_tunnel_stale/ok — hub/internal/api/host_tunnel.go:19-47 — FACT
- S17 06:357-359 „key rotation … design when fleet size warrants" → D30 rotation exists — FACT
- S18 06:381-383 cloudflared.go „seam" → reads the guest container; no seam — FACT
- S19 06:86,325-329 „No inbound listener … box→PBS only" → poke UDP 51822 on the WG /32, felhom-sshd, operator /32, iperf3 :5201 — A/internal/poke/poke.go:35,133 — RULE
- S20 12:23-24 „five byte-identical copies" → two copies carry an extra paragraph (b6638ac4 / 48d8674) — FACT (aligning them is rule text → operator)
- S21 12:32 rules load „when a matching path is touched" → some always load (`unconditional: true`, no `paths:`) — FACT
- S22 12:40-43 skills „live at once … the list is the folder" → a NEW skill needs install_skills.py again (felhom-web-design not linked) — FACT
## UNDOCUMENTED
- U1 03 §6 lists ~17 of 41 local-API routes — A/internal/localapi/server.go:562-634
- U2 the controller can start a HOST action: POST /host/line-test/run — server.go:579-580; linetest.go:346
- U3 local API over the R-50 island bridge — config.go:260-269
- U4 WG block carries oob_peer_ip, oob_operator_ssh_key — A/internal/hub/report.go:752,758
- U5 D30: re-key only inside Watchdog; `futureSince` in memory (a restart restarts the 15 min) — manager.go:621-630; rekey.go:45-80
- U6 poke binds UDP 51822 on the WG /32 only — poke.go:32-35,133
- U7 the GRANDFATHERED skill has no size cap (240 lines vs R-394's 179) — scripts/check_skills.py:45-47,98-101
- U8 line-test state `<oob state dir>/line-test.json`, switch `line_test.disable` — config.go:37; linetest.go:89
## CONTRADICTIONS
- C1 03:67,815 agent manages the tunnel service vs 03:73 cloudflared is a guest container — code: 03:73
- C2 06:381-383 cloudflared will adopt agent-managed pattern vs 03:73 — code: 03:73
- C3 06:325-333 no inbound listener vs 06:273-285 felhom-sshd over the tunnel — code: 06:273-285
- C4 03:234-238 break-glass over Cloudflare vs 06:280-285 over WG — code: 06
- C5 13:51 „38 local-API routes" vs 03 §6 ~17 — code: 41 on main
- C6 05:459 „0.166.0 NOT deployed" vs hub CHANGELOG v0.167.0 deployed
## UNTESTABLE
ep0:5201 never on the internet (nft lives on ep0); PBS sync/verify schedule (DooPlex job config); WG enabled default vs „production".
@@ -0,0 +1,38 @@
# DOC-C1 — 05 hub, 08 alarm ladder, _hub-review (reviewer candidates, read-only; the main session verified before acting)
Reviewer: read-only subagent, 2026-10-24. Counts: claims 52 · true 31 · stale 14 · untestable 5 · undocumented 7 · contradictions 4.
Prefixes: `H/` = felhom.eu/hub/, `C/` = felhom-controller/controller/, `F/` = felhom.eu/ (felhom.eu main 44d19042, controller main 321df1f).
## STALE
- S1 05:585,590-592 „Every night at 02:00 … older than 24 h … picks up at 02:30" → 02/08/14/20, catch-up 6 h, push :30 after each, refuse > 7 h (D34) — H/internal/dbsnap/dbsnap.go:32,36; H/cmd/hub/main.go:964-968; F/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10 — FACT
- S2 05:614 „scheduleDaily: db-snapshot 02:00, prune, …" → four db-snapshot slots; prune at the configured schedule (04:30) — H/cmd/hub/main.go:648,859,967-972 — FACT
- S3 05:353 „autoMintSelfBindLink runs on four events" → a fifth: the household's „new link" on a used/expired link (R-719, ≤1/h/customer); old host still linked → „oldbox" page + operator told — H/internal/web/selfbind.go:382,395-436 — FACT
- S4 05:94,142,459 version literals „hub 0.162.0 … not yet deployed", „0.166.0 built NOT deployed" → the manifest deploys 0.169.0 — F/manifests/hub.yaml:135 — FACT
- S5 05:22-26 „the live hub is v0.11.0" → version literal, false — F/manifests/hub.yaml:135 — FACT
- S6 08:535,583 „controller v0.342.0 + hub v0.169.0, NOT released" → hub half deployed (hub CHANGELOG:1) — FACT
- S7 08:158 „processOperator consults only operatorOn, the address and a 1-hour cooldown" → per-type cooldown (5 min liveness, 6 h storage fill, else 1 h), R-723 first-hour skip, mail hold — H/internal/notify/dispatcher.go:606-614,631-638,670 — FACT
- S8 08:291-298 per-app grain lists three types → also app_hold_no_whole_copy, app_stopped_unhealthy, app_repaired_after_crash, app_unexplained_upload/cpu — H/internal/notify/dispatcher.go:473-496,502-507 — FACT
- S9 08:182-183 backup_integrity_failed „not in DefaultEnabledEvents … default OFF" → a default since controller v0.309.0 („operator Q1 2026-10-12") — C/internal/settings/settings.go:713-721 — RULE
- S10 08:473 §8.1 „NOT BUILT" → slice 1 built (three group switches, v0.309.0) — C/internal/settings/notif_groups.go:4-16 — FACT
- S11 08:137-138 „info is dropped … before both legs" → paired info recoveries bypass the gate — H/internal/notify/dispatcher.go:137-167,225-228 — FACT
- S12 05:391 „39 event types" → 44 `mail.event.*` keys in hu.json — FACT
- S13 08:374-380 only `disabled` skipped → also a box the household switched off (R-998) — H/internal/monitor/deadline.go:433-442 — FACT
- S14 _hub-review.md (a dated v0.6.3 review) thresholds/cooldown/push → historical; add a banner — FACT
## UNDOCUMENTED
- U1 bind_old_host_linked (R-1203) — selfbind.go:395-421; dispatcher.go:870 — 05 §13, 08 §6.3
- U2 R-1230 the off-site delivery checker skips a box awaiting its recovery key — H/internal/monitor/offsite_delivery.go:105-110,149 — 07 / 05 §8
- U3 offsite_delivery_stuck + maybeHeal — offsite_delivery.go:113-125; dispatcher.go:847 — 07 §6
- U4 R-723 backup_tier_skipped suppressed in the first hour — dispatcher.go:627-638 — 08 §6.2
- U5 System page „Upload line" cell — H/internal/web/system.go:174-197 — 11 §5.7
- U6 wg_tunnel_stale, host_swap_short missing from 08 §6.3 — H/internal/api/host_tunnel.go:23,37-47; host_swap.go:31-43
- U7 golden_not_vouched, recovery_older_package in no document — dispatcher.go:776-782
## CONTRADICTIONS
- C1 05:569 „6-hourly VACUUM INTO" vs 05:585 „every night" — code: 6-hourly
- C2 poster.facts.md:125 (four push times) vs :150 („02:30" only) — code: four
- C3 05:291-297 slim DR record RETIRED vs 05:329-330 §11 „ADD … the slim DR record" — code: retired
- C4 08:182 „default OFF" vs 08:510 „the 17 defaults" (includes it) — code: default ON
## UNTESTABLE (live only)
05:571 token bytes freed; 05:569 Longhorn speed; 08:553/576 line test numbers; 05:268 R-1159 proof; 08:216 report gap.
@@ -0,0 +1,42 @@
# DOC-C2 — 09 update architecture (reviewer candidates; verified by the main session before acting)
Reviewer: read-only subagent, 2026-10-24. Counts: claims 63 · true 38 · stale 19 · untestable 6 · undocumented 6 · contradictions 7.
Paths under felhom-controller/controller/ unless named.
## STALE (all FACT)
- S1 09:68-75 `Manager.UpdateStack` „two compose commands … No safety copy. No rollback." → gone; StartGuardedUpdate — internal/api/router.go:794; internal/stacks/update.go:518-540,752-991
- S2 09:119-124 RestartStack comment → manager.go:1454; up -d --force-recreate, start-time check (R-1026) — manager.go:1470-1487
- S3 09:82-86,97-98,1958,2308 line citations moved (sync.go:319,362,436,700; config.go:399; appstop_marker.go:377; intermediary.go:228)
- S4 09:1979-1992 „four writers … UpdateStack" → advancePinTo; plus pinBack and the undo's restoreDefinition — update.go:921-941,1124; undo.go:531
- S5 09:2631-2632 recordInstalledImages callers incl. UpdateStack → StartStack, RestartStack, deploy, guarded update end, undo end — manager.go:1368,1495; deploy.go:683; update.go:1007; undo.go:555
- S6 09:1994 §5.4 render table → missing row: unreadable template images → compose left alone; R-971 admission before any write — internal/sync/sync.go:622-627; cmd/controller/main.go:421
- S7 09:2050,2273 slice 6 „OPEN — R-450" → built; R-450 closed 2026-10-05 — internal/stacks/unattended.go:16-37
- S8 09:2064-2075 §6.1 phase table → no `converting` row; bind-data order 2 → 3 → 1 — update.go:969-980; internal/backup/update_guard.go:130-155
- S9 09:2374 part 1 „DB up alone … empty-then-load … A volume tar" → what shipped is the folder copy (cp -a + marker) — undo.go:30-34,519-530
- S10 09:2378 „a step has no .felhom.yml of its own (R-664)" → steps/<key>.felhom.yml read when present — ladder.go:129-132,209-212
- S11 09:2379 „The box half … is not built" → built — digest.go:15-23,94-104,141-155; updateorder.go:76-112
- S12 09:2132-2133 R-646 residual → BackfillAppliedMeta at start — undo.go:646-676; cmd/controller/main.go:498
- S13 09:2119,2729-2733 „multi-major jump … slice 6" / §8.6 „Tier-2 copy … no automatic rollback" → any tier, undo, one step per press
- S14 09:2706-2708 digest comparison „R-446 §3b Q6" → built — updateorder.go:76-112
- S15 09:2736-2739 engine-major „until Slice 4" → MariaDB lifted; PostgreSQL with a two-venue proof + engine_conversion
- S16 09:269-273 `stacks.update_window` „no Go code reads it" → the field is removed (inside ruling 11: a dated note, not an edit)
- S17 09:427 `unattended.go:433–435` → :449-451
- S18 09:2344-2345 `config.go` L280-281, default L422 → L278-279, L419
- S19 09:2462,2477 refusal `current` → `already_current` — update.go:432
## UNDOCUMENTED
- U1 build.sh refuses --push from a dirty tree / HEAD ≠ origin/main — controller/build.sh:144-167 — 09 §0
- U2 the leg retries a transient refusal once in the same leg after RetryWait — unattended.go:65-66,294-306
- U3 the leg skips `stopped_by_household` (R-721) — unattended.go:54-55,416-418
- U4 a manual chain run's deadline = start + 195 min — unattended.go:234-239
- U5 preflight refusals `already_current`, `engine_no_test` — update.go:432,454
- U6 the syncer's R-971 allow-list before any compose write — cmd/controller/main.go:421
## CONTRADICTIONS
- C1 09:2050 slice 6 OPEN vs capability map B11 PROVEN-LIVE — code: built
- C2 09:68,1984,2632 + 02:838,861 UpdateStack vs controller REUSE.md:154 „DELETED in v0.237.0" — code: deleted
- C3 controller REUSE.md:164 lists UpdateStack as a caller vs REUSE.md:154
- C4 RULING decision 13 „CompareImageRefs moves to the catalog gate" vs 09:2377 „did NOT move" — code: controller only (updateorder.go:134) — OPERATOR
- C5 RULING decision 14 „never jumps" vs 09:2378 „an installed version matching no entry jumps" — a person's press jumps (ladder.go:34-36,175-176,193-194); the leg does not — OPERATOR
- C6 check-engine-major.py:24-26 „the Update button still takes no backup" vs 09 §8.6 CLOSED — code: update.go:819-841
- C7 RULING decision 250 item 5 „by its AGE (older than 7 days while the floor is above it)" vs golden_currency_gate.py:8-12 („release ahead" stands in for „floor ahead") — OPERATOR to know
@@ -0,0 +1,37 @@
# DOC-D1 — 07 backup architecture (reviewer candidates; verified by the main session before acting)
Reviewer: read-only subagent, 2026-10-24. Counts: claims 64 · true 44 · stale 11 · untestable 5 · undocumented 8 · contradictions 5.
Paths under felhom-controller/controller/ unless named.
## STALE
- S1 07:756 D22 „controller main, unreleased" → shipped v0.338.0 — CHANGELOG.md:237; internal/backup/restore_unit.go:442-512 — FACT
- S2 07:383 Tier-1 „secret-stripped app.yaml" → since D5 (schema 2) the unit's app.yaml (0600) carries the PORTABLE secrets — internal/backup/recovery_unit.go:25,261,272 — FACT
- S3 07:1537 (+1498, 1624) row 15 „DEFECT … no lock case → R-104" → R-104 closed v0.297.0; class `locked` — internal/backup/offbox.go:156-159,220-224,317 — FACT
- S4 07:1532 row 11 Longhorn target DooPlex „2 restore points" → D40 took the hub volume out of the recurring group — felhom.eu/manifests/hub.yaml:47-51 — FACT
- S5 07:256 „Hub v0.158.0 (built, deploy waits)" → deployed 2026-10-18 — hub/CHANGELOG.md:288 — FACT
- S6 07:1531 row 10 put-back „next slice", R-973 „agent main, unreleased" → both shipped (v0.336.0; agent 0.160.0) — internal/backup/offbox_restore_undo.go:17-33; felhom-agent/internal/localapi/backup_cap.go:41-52 — FACT
- S7 07:440 `cmd/controller/main.go:604-607` → main.go:1260-1264 via backupwindow.LegTimes — FACT
- S8 07:1344,1347,1349 line citations moved (GetAppDrivePath backup.go:441, F5 pruneStalePrimaryDirs backup.go:1514); mp1 gone (07:386) — FACT
- S9 07:450-451 „keep-last=1 … keep-last + 1" → keep-daily=N (R-1149) — felhom-agent/internal/config/config.go:558-559 — FACT
- S10 07:1601 „a restore is refused while any second-drive copy runs" → the install-time kept-data load checks only RestoreStatus/IsRunning — internal/web/restore_wizard.go:226; internal/api/kept_install.go:89-99 — FACT or a small code gap
- S11 07:975-976 §6.6 off-site row → the off-site restore also refuses a mixed/mismatched snapshot before anything is touched — internal/backup/offbox_reconstitute.go:829-832 — FACT
## UNDOCUMENTED
- U1 a copy is limited to 60 min; a stray that cannot be removed blocks every restore, retried each minute; copies into one destination serialized — internal/backup/tier2_copy_lifecycle.go:40,113-121,150-172 — 07 §8.2b
- U2 D25 „Eltávolítás" refused while any backup/restore/copy runs — internal/web/kept_copies.go:54-58
- U3 removing an app WITH its backups deletes all of backups/secondary/<app>, D25 kept legs included; conflicts with „only the household's Eltávolítás removes it" — internal/backup/r474_remove_mirrors.go:61,87; tier2.go:487-488; tier2_kept.go:18,76 — CANDIDATE DEFECT
- U4 the local unit restore skips the fit check when free space cannot be read (-1) — fails open — internal/backup/restore_unit.go:460; tier2_whole.go:374-375 — CANDIDATE DEFECT
- U5 a LOCAL restore's put-back mails „its off-site restore FAILED" (RunKind offsite-reconstitute) — cmd/controller/main.go:1514-1518; internal/backup/offbox_restore_undo.go:242; restore_unit.go:518 — CANDIDATE DEFECT (text)
- U6 a start that fails after a successful put-back: not held, copy kept, `restore_failed_put_back_not_started` — offbox_restore_undo.go:245-247
- U7 the release copy (D33) has no row in 07's matrix — felhom.eu/scripts/release-copy/
- U8 REUSE.md has no row for the stray-copy seams — tier2_copy_lifecycle.go:85,255,319,341
## CONTRADICTIONS
- C1 07:1021 `locked` class vs 07:1537/1498 open DEFECT; OPEN-ITEMS:94,100 R-104 open vs CLOSED-ITEMS:895 closed — code: closed
- C2 00-capability-map G15 „nightly" vs 07:1532 „6-hourly" — code: 6-hourly
- C3 07:383 „secret-stripped" vs 07:1523 „portable secrets" — code: portable
- C4 agent comment backup_cap.go:15 „keep-last" vs 07:386-387 keep-daily — code: keep-daily (the comment is stale)
- C5 controller REUSE.md:84-85 D22 „(unreleased)" vs CHANGELOG v0.338.0 released
## UNTESTABLE
ep0 keep-weekly / prune jobs (ep0 config); hub-DB retention 14 daily + 8 weekly (ep0); R-1214 stall numbers; „0.336.0 is the floor"; the carry end to end.
@@ -0,0 +1,33 @@
# DOC-D2 — 16 storage identity, 15 media library, _recovery-inventory-2026-07-28 (reviewer candidates; verified by the main session)
Reviewer: read-only subagent, 2026-10-24. Counts: claims 52 · true 30 · stale 13 · untestable 5 · undocumented 7 · contradictions 4.
## STALE (all FACT)
- S1 16:233-234 an empty partition asks for the typed confirmation (R-1093) → fixed agent 0.160.0 — felhom-agent/internal/storage/hostops.go:459-491
- S2 16:18 single-copy notice „a Go constant, not i18n" → i18n note.backup.single_copy(_unproven) — controller internal/backup/backup.go:1485-1494
- S3 16:19 `/disks/candidates` „whole disks only" → whole disks with nested partition candidates — agent internal/storage/candidates.go:87-88; partitions.go:84-96
- S4 16:17,19,21,129 line citations moved (install.sh:3294 = keep-daily=3 since R-1149; hu.json:330; offer.go:236; install.sh:2475)
- S5 15:197,213 a drive added/back → „up -d recreates … household: no" → only the binds are rewritten; the restart waits for „Újraindítás most" / „Később" (03:30); the file browser alone recreates itself — controller internal/stacks/media_library.go:286-288; internal/web/media_events.go:97-110,250-274
- S6 15:208 eject DEREGISTERS → a safe removal keeps the drive registered (SetEjected); its return re-binds with no step — media_events.go:304,319-352
- S7 15:166-168 Transmission watch/download folders off-site / `optional` → catalog c50adfc (R-1019) marks them `excluded`
- S8 inventory:194,392-396 rootfs + mp0 /var/lib/docker + mp1 /mnt/sys_drive → one mp0 at /var/lib/felhom; no mp1 — felhom-agent/configs/build-golden.sh:31-35,99-105,215-225
- S9 inventory:89,351,509-517 secrets never from the unit; unit app.yaml secret-stripped → schema-2 units carry the portable secrets; a restore takes the DB password from the unit (R-1106) — recovery_unit.go:30-41; restore_dbcreds.go:16-30
- S10 inventory:99,275,363 the copy's recovery-unit/ read by no code path (C9-F4) → RestoreTier2Unit (R-102) — tier2_restore.go:182-211; web/handlers.go:2324
- S11 inventory:138,276,373 named-volume tars never unpacked → reconstitute replays them (R-354) — offbox_reconstitute.go:72,1040-1047
- S12 inventory:383 ep0 keep-last 2; local keep-last 3 → ep0 keep-weekly 2 at 03:30 (R-1162/R-1165); local keep-daily=3 (R-1149)
- S13 inventory:374,656-673 restic 0.14.0 exits 0 on a missing source; namespace „root" → restic 0.19.1 exits 3; namespace from storage.cfg, "" for root
## UNDOCUMENTED
- U1 the „Később" 03:30 media restart is an in-memory timer (a restart forgets it) — media_events.go:250-264 — 15 §11
- U2 „Restart now"/„Később"/safe removal force-recreate EVERY deployed media app, including apps the household STOPPED (up -d --force-recreate starts them); held skipped; no backup-run check — media_library.go:270-280; manager.go:1477; media_events.go:196-205,306-315 — CANDIDATE DEFECT
- U3 the Sonarr/Radarr key is read from /config/config.xml (the „SET at install" option not built) — mediawire.go:336-345
- U4 systemd-detect-virt read once per agent; empty → unknown/vm (fails closed) — agent identity.go:195-205,370-372
- U5 an unmapped verdict falls back to st_dev; an unreadable path counts as the SAME disk (R-668) — backup.go:1408-1450
- U6 the internal place refuses eject/decommission/wipe, allows a data move — web/internal_place.go:22-25
- U7 the internal place subtracts every promised thin volume — agent configs/felhom-internal-place:11-14,71-73
## CONTRADICTIONS
- C1 08:535,583 upload watch „NOT released" vs capability map F13 PROVEN-LIVE on 0.342.0 — code/CHANGELOG: released
- C2 15:208 eject deregisters vs 15:251 safe removal keeps it — code: keeps
- C3 runbooks/RUNBOOK-manual-guest-restore.md:28,38,93 `mp1 … /mnt/sys_drive` vs 07:386 mp1 gone — code: gone (recovery-relevant)
- C4 .claude/rules/live-probes.md:41 `/opt/felhom/stacks/<name>` vs the real `/opt/docker/stacks/<app>/` (a path in a rule file)
@@ -0,0 +1,39 @@
# DOC-E — 02 controller module map, 10 localisation, 11 OS updates, 14 host power-off (reviewer candidates; verified by the main session)
Reviewer: read-only subagent, 2026-10-24. Counts: claims 121 · true 92 · stale 22 · untestable 6 · undocumented 5 · contradictions 3.
## STALE (all FACT)
- S1 14:36 heavy-op gate holders → also the weekly line test (agent v0.164.0) — felhom-agent/internal/linetest/linetest.go:62,320
- S2 14:25 „two exact sudoers vectors, FELHOM_POWEROFF" → FELHOM_POWEROFF holds only `systemctl poweroff`; `pct shutdown` is FELHOM_GUESTHOOK — felhom-agent.sudoers:81-90
- S3 14:89 website FAQ line „a later task (R-1000)" → present — website/gyik.html:128,603
- S4 14:39-40 a paragraph breaks the §3 table (rendering) — the doc itself
- S5 11:81 cloudflared `2026.6.0` → `2026.9.3` — controller internal/infra/infra.go:26
- S6 11:104-106 line citations moved (sudoers:68,228; lanresolver.go:133; manager.go:659)
- S7 11:253-256 „DRAFT … nothing installed … --repair-only" → built; sudoers has only `--plan` — sudoers:336-337
- S8 11:14 „§8 step 2 BUILT" → steps 2–6 built (the doc's own §8); „NOT RATIFIED" is the operator's
- S9 11:318-320 BUNDLE_FILES „22" → 32 — felhom-os-apply:191-233
- S10 11:482-483 HOST_SLOW_RE list → also linux-(image|headers|kbuild|modules|base), firmware-, proxmox-secure-boot-support — felhom-os-apply:100-102
- S11 10:202-204 the setup wizard still reachable, deletion R-554 → deleted v0.308.0; bootstrap.WaitForConfig
- S12 10:131 „Part B is the remaining piece" → built (v0.256.0) — controller internal/config/config.go:149-154
- S13 10:184-189 formal-form ceiling 16 → 0 (R-516 item 12) — controller scripts/i18n_missing_gate.py:41
- S14 10:237,310,313 „53 apps", „fifth row" → 65 apps; sixth row; fixtures still 53
- S15 10:229 metadata.go L316 → :475
- S16 02:108 „two in-flight markers" → also update-journal.json and .felhom-install-pending — update.go:49,1306; install_interrupted.go:16-28
- S17 02:171 „three holders" → four (1b IsUpdating) — cmd/controller/main.go:2459-2477
- S18 02:387-390 GetAlerts → GetBannerAlerts(page) filters before the cap (R-906) — internal/web/alerts.go:350-358
- S19 02:267,272,742 „no longer calls the Cloudflare API" → internal/cloudflare still wired when cf_api_token is set — main.go:1994-2000; config.go:160
- S20 02:328-329,724-725 state.go/updater.go „DELETE … removed" → both live; the swap goes to the agent (AgentSwapper) — internal/selfupdate/updater.go:38-47,87
- S21 02:821-825 line citations moved (sync.go:436; config.go:399)
- S22 02:888-890 locked_after_deploy „read by no code" → read into LockedFields; enforced only by an uncalled UpdateStackConfig — deploy.go:428,718-748,837
## UNDOCUMENTED
- U1 a re-enroll during the household power-off hold counts as „gone" (R-1154) — hub/internal/api/reenroll.go:67-68 — 14 §4
- U2 host_deliberate_off(_ended) recorded as info events, not mailed — hub/internal/api/deliberate_off.go:12-13,42-50
- U3 the unaccented confirmation word „KIKAPCSOLAS" is accepted — controller internal/web/poweroff_handlers.go:29
- U4 sockheal (R-860), channelhealth, ~20 post-v0.33 packages missing from 02 §1 — internal/sockheal/sockheal.go:1-6; internal/channelhealth/checker.go:1-6
- U5 hub System page „Upload line" cell (amber after 15 days) — hub/internal/web/system.go:174 — 11 §5.7
## CONTRADICTIONS
- C1 10:202-204 setup wizard reachable vs 02:56 deleted v0.308.0 — code: deleted
- C2 03:132 D4 „NOT shipped" / 13:76 root password SSH vs shipped in v0.163.0 — code: shipped
- C3 02:108 two markers vs 09:2102 update-journal.json — code: four
@@ -0,0 +1,15 @@
# HUNT-1 — restore and put-back (reviewer candidates; verified by the main session before any row)
Paths under felhom-controller/controller/.
- H1 P1 — a power cut during a restore of a DRIVE-backed app: Recover runs before the drive is bound → „restore copy … is gone — restarting as before" → the gated start refuses → app Refused, marker + Undo kept → intermediary.go:470-476 boot recreate checks only appHeld, not the marker → the app runs on the cut restore's data → the next Recover finds the copy and pours it back over the newer writes — internal/backup/offbox_restore_undo.go:47-48; appstop_marker.go:362-375,408-411; cmd/controller/main.go:2573-2576; internal/web/intermediary.go:400-403,470-476
- H2 P2 — a FAILED crash put-back leaves the app stopped, unheld; Start or the boot recreate brings it up; a later restart pours back an old copy — appstop_marker.go:368-371; api/router.go:690-709; intermediary.go:472
- H3 P2 — an export with „stop app first" during another app's restore overwrites the single marker file — appstop_marker.go:286-292,318; appexport/export.go:173-178,262-266
- H4 P3 — off-site restore: a definition-write failure restarts the app on a half-written definition — offbox_reconstitute.go:875-883,979-986; main.go:3318-3331
- H5 P3 — a put-back can merge (failed stop only logged, `volume rm -f` error ignored, tar over), then the copy is deleted — restore_crash_putback.go:155-171; offbox_restore_undo.go:223-225; restore.go:218,251
- H6 P3 — a second-drive mirror can start between the web refusal check and the restore's record — web/restore_wizard.go:226; opstatus.go:56-66; handlers.go:2209
- H7 P3 — Stop pressed during a restore: after a cut the put-back is skipped, marker cleared, copy left — appstop_marker.go:356-361
- H8 P4 — an unused restore copy is never cleaned up — offbox_restore_undo.go
- H9 P4 — a comment names a test file that does not exist (`r1195_crash_putback_test.go`) — restore_crash_putback.go:32
## CHECKED AND SOUND (reviewer)
put-back before any start; journal names the same stack; atomic writes; Undo recorded only after a whole copy; fit check before the stop; DBs are named volumes; hold path; capped copy --mount/no symlinks/--pull never; start-up stray clean-up; beginTier2Run under one lock; one single-flight flag.
@@ -0,0 +1,14 @@
# HUNT-2 — delete and retention (reviewer candidates; verified by the main session before any row)
- H1 P2 — one bad/rejected `.felhom.yml` backup block deletes the app's second-drive copies of every userdata/ and non-appdata hdd/ leg the same night — controller internal/stacks/metadata.go:550-557 (meta.Backup = nil on any defect, incl. classify.go:185 „matches no compose bind") → appbackup/classify.go:217-221 has=false → backup/tier2_capture.go:84-100 legacy legs hdd/appdata/<name> only → backup/tier2.go:492 tier2Reconcile removes every non-leg dir (tier2.go:325-329); also cs.Skipped legs (tier2_capture.go:56-61). Against D25's header tier2_kept.go:17-18.
- H2 P3 — a D25 kept copy is reconciled away when its leg later leaves the declared set; its record stays — tier2.go:485-492; settings/kept_copies.go; tier2_kept.go:93-95
- H3 P3 — R-1019 also moved MeTube (`media/metube`), NZBGet, SABnzbd to `excluded`; the reconcile deletes their copies — catalog templates/metube/.felhom.yml:33; commit c50adfc — a design call
- H4 P3 — Remove (even keep-data) deletes volumes labelled with the app's project that the definition no longer declares; no tier holds them — controller internal/stacks/delete.go:763-764 (8ae9178, R-1117)
- H5 P3 — the capped copy's rsync --delete sees the source in the daemon's mount view; the emptiness check runs in the controller's view — backup/tier2.go:834 vs :464
- H6 P3 — the release copy keeps the newest 5 agent versions, vouched or not — felhom.eu/scripts/release-copy/felhom-release-copy:199,218,237
- H7 P4 — release copy rsync --delete removes old versions during the transfer; a shrinking registry mirrors deletions — felhom-release-copy:190,237
- H8 P4 — hub-DB push has no lower age bound (a future-stamped snapshot stays „newest") — scripts/hub-db-backup/felhom-hub-db-backup:41,49-50
- H9 P4 — Remove with remove_backups deletes kept app copies under the mirror and leaves their records — api/router.go:1086-1087
## CHECKED AND SOUND (reviewer)
tier2SafeRemove bounded to /backups/secondary/ (tier2.go:260-269); RemoveKeptCopy same device+inode (tier2_kept.go:92-111); RemoveTier2Mirrors exact paths (r474_remove_mirrors.go:18-19,76-84); shares reconcile drive-scoped (tier2_shares.go:262,347-358); RemoveStack backup path bound (delete.go:839-846); release copy refuses before rsync, size cap first, rrsync; hub-db-backup STAGE default, integrity before push; ep0 prune jobs ns-scoped; agent keep-daily ≥ 1; R-1010 force-delete narrow.
@@ -0,0 +1,9 @@
# HUNT-3 — keys (reviewer candidates; verified by the main session before any row)
No P1.
- H1 P2 — the hub stores a household's new escrow package in TWO writes (SaveHostEscrow commits blob+hash, then SaveHostDRBundle sets identity_blob); a hub failure between them leaves the NEW hash beside the OLD identity package; the box's confirm passes and flips to escrowed; on a total loss the code opens nothing — hub/internal/api/handler.go:1415,1445; internal/store/store.go:3496,3511,3793; controller internal/report/escrow_confirm.go:195-217; agent sends once (cmd/felhom-agent/main.go:3366-3386)
- H2 P2/P3 — a lost/late hub answer tells the household „your previous code still works" when the hub may hold the new package — agent internal/localapi/escrow_ceremony.go:79,532-534,639-641; controller i18n en.json:2662,2664
- H3 P3 — the R-1176 carry can move a package with no repository password (checks identity_blob only), leaving the right package under the deleted host — hub/internal/store/escrow_carry.go:100-110; store.go:3533-3534,3631-3633; controller backup/offbox.go:608
- H4 P4 — same-day set-aside name reuse can purge a still-stored copy's package — hub/internal/offsitekeys/offsitekeys.go:378-385; store.go:4295-4300
## CHECKED AND SOUND (reviewer)
D30 key file atomic and heals; once an hour across restarts; hub swaps in place and refuses another host's key; PBS hook inert after a re-key; D31 re-issue changes the token only (comment wording); R-1176 customer-scoped, one transaction, PK stops a double carry; Reconcile before SetFloor; R-1182 purge never touches host_escrow; RESET needs acks; R-975 seal all-or-nothing; the mint guard.
@@ -0,0 +1,11 @@
# HUNT-4 — time and concurrency (reviewer candidates; verified by the main session before any row)
No P1 or P2. R-1250 (the hold + test on request) is committed, not released (agent „next v0.165.0", controller „next v0.343.0").
- H1 P4 — 2026-10-25 (fall back): DooPlex pushes the 20:00 hub snapshot again at 02:30 CEST and never the 02:00 one (Go puts the hub's 02:00 slot on the SECOND pass = 01:00Z, after the 00:30Z push); the copy is ~13.5 h old until 08:30 — scripts/hub-db-backup/felhom-hub-db-backup.timer; hub/cmd/hub/main.go:1092; hub/internal/dbsnap/dbsnap.go:32; felhom-hub-db-backup:19,50
- H2 P3 (2027-03-28) — spring forward: the leg's deadline is W + 5 h of REAL time, the gate counts wall-clock minutes; a step can start after the full-system backup stopped waiting — controller internal/stacks/unattended.go:225; internal/quiesce/quiesce.go:922-933
- H3 P4 (2027-03-28) — W in 02:00–02:59: the DB dump and the second-drive copy start at the same instant (the missing 02:30 maps to 03:30) — internal/scheduler/scheduler.go:509; backupwindow.go:17
- H4 P4 — two seeders behind one uplink: the line hold holds only the first, the test is refused „not quiet" every 15 min in the Tuesday window, never counted — controller internal/upwatch/linehold.go:155-160; agent internal/linetest/linetest.go:379-381 (unreleased)
- H5 P4 — a hold record of an app removed while held is never cleared; qBittorrent's 50 % rule never runs again — linehold.go:106-108; upwatch/service.go:149 (unreleased)
- H6 P4 — qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored — upwatch/explain.go:175; linehold.go:127-128 (unreleased)
## CHECKED AND SOUND (reviewer)
2026-10-25 controller legs fire once each; catch-up uses the same resolution; fall-back leg deadline earlier (safe); hub scheduleDaily day+1 + last guard; release copy 01:40 ends before the repeated hour; line test holds the gate without blocking; 2 min timeout > agent worst case; hold record written first, release deferred, recovery at start; second-drive copy one lock per destination; agent backup cap rolling 24 h.
@@ -0,0 +1,10 @@
# SEC-1 — hub + controller endpoints changed since 2026-10-17 (reviewer candidates; verified by the main session)
No P1 or P2 found.
- X1 P3 — the upload watch treats ANY viewer as an explanation for any upload size; a compromised Plex can upload freely while one member watches — controller internal/upwatch/watch.go:275,299; 08:559 states the rule, 08:579's limits list does not name this
- X2 P4 — after the operator deletes a household's host, the first enroll with only the passphrase becomes the live host and gets the sealed package (ciphertext; D19 accepts the window) — hub/internal/api/passphrase_scope.go:51-52; internal/store/escrow_carry.go:134-139
- X3 P4 — a passphrase holder can make the hub mail passphrase_keys_refused repeatedly — passphrase_scope.go:75-78; handler.go:1271
- X4 P4 — the qBittorrent admin password goes to whatever answers the DNS name `qbittorrent` — controller internal/upwatch/explain.go:94,108-113
- X5 P4 — kept-copy Remove: a gap between the identity check and RemoveAll — backup/tier2_kept.go:92-111; tier2.go:269
## CHECKED AND SOUND (reviewer)
kept remove behind session+CSRF; debug actions behind RequireAuth(CsrfProtect); .fab path checks; hub /preferences + /box-config self-scoped; /report body id = key; fhp_ refused on /api/v1, unclassified operator routes session-only; line-test run rate-bounded; Upload-line cell escaped; new events operator-only; bind page texts only for a real token; files forwardAuth strips X-Forwarded-*; network_mode refused.
@@ -0,0 +1,11 @@
# SEC-2 — agent local API, ep0 iperf3, installer sshd drop-in, DooPlex→ep0 jobs (reviewer candidates; verified by the main session)
- X1 P2 — a BYO host (the household owns it) gets root password SSH switched off by the config bundle; the installer's BYO screen says „NOT touched in byo mode: … OOB sshd" — felhom-agent/configs/felhom-os-apply:230 (the drop-in written for every mode; plan_writes 1985-2000 holds back only policy `oob`); felhom.eu/scripts/felhom-host-install.sh:2118, list 2092-2106 — proof: a byo config on a scratch host, sshd -T before/after the bundle
- X2 P3 — any tunnel peer (root on one box) can hold ep0's single iperf3 server (ErrBusy fleet-wide → MaxTriesPerDay) or ask `-R -t <long>` and fill ep0's uplink — felhom.eu/scripts/ep0-iperf3/felhom-iperf3.service:17 (no authorized users, no server-side caps)
- X3 P3 — the „in force" check probes only addr=192.0.2.1, so a `Match Address <LAN>` block with PermitRootLogin yes reads as in force — felhom-os-apply:2101-2110
- X4 P4 — `sshd -t` failing for any reason refuses and undoes the WHOLE bundle — felhom-os-apply:2102,2112-2115,2054-2061
- X5 P4 — the release-copy restore check trusts hashes stored on ep0 itself — runbooks/total-loss-of-dooplex.md
- X6 P4 — rrsync without -munge/-wo; a symlink push might escape `data/` depending on `.ssh` ownership — audits/day-2026-10-23/08-release-copy-D33.txt:6
## CHECKED AND SOUND (reviewer)
line-test routes bearer→VMID (server.go:579-580, 692-715); the POST is rate-bounded (due, window, 6/day, heavy-op gate); iperf3 on 10.77.0.1 only, wg0 + /24, forward drop; DynamicUser hardening; the drop-in value pinned to prohibit-password, sshd -t -f before write, undo reloads, uninstall removes; release-copy names validated, host key pinned, 0600 O_EXCL login file; read:package token; ep0 cannot write back to DooPlex; hub-db-backup name pattern, DatastoreBackup scope, shredded staging; ProtectSystem=strict units.
@@ -0,0 +1,43 @@
"""h1_tier2.py install|chain|copy|break|unbreak|remove — review round 2, HUNT-2 H1 on scratch 9202 (the product's own
endpoints; box_walk.py). A bad backup block in an app's .felhom.yml: does the night's second-drive copy then delete the
app's household-file copies?"""
import json, re, sys, time
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
import box_walk as w
APP, SUB = "audiobookshelf", "audiobookshelf"
MARK = "/mnt/felhom-drives/scratch_hdd/userdata/media/audiobooks/review2-h1-marker.txt"
COPY = "/mnt/sizing-bench/backups/secondary/audiobookshelf"
FY = "/opt/docker/stacks/audiobookshelf/.felhom.yml"
w.LOG = []
w.login()
cmd = sys.argv[1]
if cmd == "install":
print("deploy:", w.deploy(APP, SUB, {"HDD_PATH": "/mnt/felhom-drives/scratch_hdd"}))
print(w.guest(f"echo 'review round 2 H1 marker — a household file' > {MARK}; ls -la {MARK}"))
elif cmd == "chain":
since = w.guest("date -u +%Y-%m-%dT%H:%M:%SZ").strip()
code, d = w.ctl("POST", "/api/debug/backup/night-chain")
print("night chain ->", code, str(d)[:160])
out = ""
for _ in range(120):
time.sleep(10)
out = w.guest(f"docker logs --since {since} felhom-controller 2>&1 | grep -E 'night-chain|Tier 2|tier2|backup block|reconcile' | grep -v DEBUG | cut -c1-260")
if re.search(r"night-chain\] update-leg: done|night-chain\].*finished|update leg \(manual-chain\)", out):
break
print(out)
elif cmd == "copy":
print(w.guest(f"find {COPY} -maxdepth 5 | sort | head -40; echo; ls -la {COPY}/userdata/media/audiobooks/ 2>&1 | head"))
elif cmd == "break":
print(w.guest(f"""cp -p {FY} /root/review2-h1-felhom.yml.orig && python3 - <<'PY'
p='{FY}'; s=open(p).read()
old=' - path: media/podcasts\\n'
assert old in s, 'anchor'
s=s.replace(old, ' - path: media/review2-typo\\n class: optional # REVIEW-2 H1 PROOF: a path no compose bind has\\n'+old, 1)
open(p,'w').write(s)
PY
grep -n -B1 -A1 review2-typo {FY}"""))
elif cmd == "unbreak":
print(w.guest(f"cp -p /root/review2-h1-felhom.yml.orig {FY} && rm -f /root/review2-h1-felhom.yml.orig && grep -c review2-typo {FY}; echo restored"))
elif cmd == "remove":
print(w.remove(APP))
print(w.guest(f"rm -f {MARK}; ls {COPY} 2>&1 | head -3"))