Review round 2, part 1 (batch 1): 01/03/04/05/06/08/12/13/_hub-review facts corrected against live code (each at file:line, EDITS.md); 20 capability rows marked re-check; poster timeline text; live-probes path; reviewer candidates and the H1/H2 proofs
gates / gates (push) Successful in 12m37s
gates / gates (push) Successful in 12m37s
This commit is contained in:
@@ -38,7 +38,7 @@ to probe a refusal at all.**
|
||||
## If a probe does create something, remove it through the product
|
||||
|
||||
Not by hand, and not by `docker rm`: stop it, then `POST /api/stacks/<name>/remove` with
|
||||
`remove_hdd_data` and `remove_backups`, and then **verify** — no container, no `/opt/felhom/stacks/<name>`,
|
||||
`remove_hdd_data` and `remove_backups`, and then **verify** — no container, no `/opt/docker/stacks/<name>/app.yaml` (the folder itself stays — it holds the catalog's template files; path corrected 2026-10-24),
|
||||
no volume. Say in the report that it happened. A tidy-up nobody is told about is how the next session
|
||||
learns nothing.
|
||||
|
||||
|
||||
@@ -80,17 +80,17 @@
|
||||
| <a id="a4"></a>[A4](capability-map-detail/A.md#a4) | Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; universal secret-free / operator-bind mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PROVEN-LIVE** | N100 (2026-07-18) and HP t740 (2026-07-21) | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/SPIKE-baremetal-iso-2026-07-16.md` (2026-07-16) |
|
||||
| <a id="a5"></a>[A5](capability-map-detail/A.md#a5) | Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`<iso>.rootpw.txt`) | scripts v1.24.0 (`felhom-bootstrap.sh`, `build-felhom-iso.sh`) | **PROVEN-LIVE** | nested drill; metal proof pending | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` (2026-07-22) |
|
||||
| <a id="a6"></a>[A6](capability-map-detail/A.md#a6) | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** | drill VM; no non-operator human yet (R-3) | `audits/DRILL-day0-vm-2026-07-12.md` (2026-07-12) |
|
||||
| <a id="a7"></a>[A7](capability-map-detail/A.md#a7) | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** | customer-zero, on metal | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
|
||||
| <a id="a8"></a>[A8](capability-map-detail/A.md#a8) | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** | drill VM, endpoint-exact; browser wizard first fired 2026-07-18 | `audits/SPIKE-controller-escrow-2026-07-13.md` (2026-07-13); `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
|
||||
| <a id="a7"></a>[A7](capability-map-detail/A.md#a7) | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** | customer-zero, on metal; **re-check (2026-10-24):** mechanism changed after its evidence — f90e2229 (bind mail W3–W5) | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
|
||||
| <a id="a8"></a>[A8](capability-map-detail/A.md#a8) | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** | drill VM, endpoint-exact; browser wizard first fired 2026-07-18; **re-check (2026-10-24):** mechanism changed after its evidence — agent c11e3e2 + controller ba4a6f0 (typed-back words, R-1151) | `audits/SPIKE-controller-escrow-2026-07-13.md` (2026-07-13); `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
|
||||
| <a id="a9"></a>[A9](capability-map-detail/A.md#a9) | One name per secret on every surface — „Beállító kód" (the setup code) and „Tulajdonosi jelmondat" (the owner passphrase) — and the re-enrol mail points at a page a rebuilt box shows | controller v0.211.0, hub v0.104.0 + v0.105.0 (R-295, R-323) | **IMPLEMENTED** | written BUILT: source-read; no customer has typed the passphrase | R-295; R-323 |
|
||||
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
|
||||
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
||||
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints; **re-check (2026-10-24):** mechanism changed after its evidence — hub 9688e76e (sealed package back to a rebuilt box, R-1176) | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
|
||||
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only; **re-check (2026-10-24):** mechanism changed after its evidence — agent ae69ad4 (D30 WireGuard re-key) | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
||||
| <a id="a12"></a>[A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES — the rebuilt Tester4 box (public ISO) got the household's sealed package from the hub, asked for the recovery code, opened the history; Nextcloud came back with 3 031 of 3 033 manifest files byte-identical (the 2 never reached the old box) and the next off-site runs appended (63 → 64). Earlier the same day it FAILED (R-1176, now closed); the failure paths are not claimed (R-379, R-380) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) |
|
||||
| <a id="a13"></a>[A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES with ONE operator act by design (D19: delete the dead host) — install, bind, claim, the recovery offer, a wrong code refused, the right code, „Teljes visszaállítás", all through the household's own pages; the new-link page now tells a household whose old box is still linked to write to Felhom (R-1203). Earlier the same day it stopped at the rebuilt dashboard (R-1176) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) |
|
||||
| <a id="a14"></a>[A14](capability-map-detail/A.md#a14) | A random night of household actions while random things go wrong — twelve rounds, unattended | controller v0.245.0, agent v0.131.0, hub v0.116.0, golden 0.245.0, ISO 1.28.0 | **PROVEN-LIVE** | nested box, twelve chaos rounds | `audits/DRILL-chaos-night-2026-09-17.md` (2026-09-17) |
|
||||
| <a id="a15"></a>[A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.336.0, agent v0.161.0, hub v0.162.0, golden 0.331.0, installer 1.35.0, ISO 1.29.0 | **PARTIAL** | 2026-10-20: download → install → bind → claim → 8 apps → use → backup → remove → every restore path → power cut → mistyped codes all worked; the 8 GB box FROZE twice in the first afternoon and only a plug pull brought it back (R-1192), and the owner passphrase needed the operator | `audits/rehearsal-2026-10-20/journal.md` (2026-10-20); `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) |
|
||||
| <a id="a16"></a>[A16](capability-map-detail/A.md#a16) | Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts | controller v0.197.0 | **PROVEN-LIVE** | both drive layouts | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04) |
|
||||
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
|
||||
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp; **re-check (2026-10-24):** mechanism changed after its evidence — agent 1b6e316 (local keep-daily, R-1149) | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
|
||||
| <a id="a18"></a>[A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites per-repo CHANGELOGs and the spec `backlog/SPEC-r85-phase4-5-2026-07-26.md` — a spec, not a run record |
|
||||
| <a id="a19"></a>[A19](capability-map-detail/A.md#a19) | A restore-test can never fill the box's disk: it sizes the restore first (uncompressed), keeps off the tested guest's pool when it can, refuses what does not fit, and retries a failed clean-up on a timer | agent v0.133.0 (RELEASED, NOT DELIVERED), hub v0.124.0 | **IMPLEMENTED** | written BUILT: the refusal proven on demo-hp 2026-09-24; the full-test path not live | `audits/r672-2026-09-24/` (2026-09-24); R-672 |
|
||||
| <a id="a20"></a>[A20](capability-map-detail/A.md#a20) | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0; purge-first hub v0.142.0 | **PROVEN-LIVE** | two real firings (2026-07-18); third 2026-10-22 on a real customer (Tester-2) with the R-32 purge-first leg — all legs ok, others untouched | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/tester2-reset-2026-10-22/` |
|
||||
@@ -102,17 +102,17 @@
|
||||
|
||||
| ID | Capability | Component | Status | Scope | Evidence |
|
||||
|---|---|---|---|---|---|
|
||||
| <a id="b1"></a>[B1](capability-map-detail/B.md#b1) | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (~52 apps, images pinned) | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/RERUN-p1p3-2026-07-06.md` (2026-07-06) |
|
||||
| <a id="b2"></a>[B2](capability-map-detail/B.md#b2) | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | the actions; remove-with-data re-proven 2026-09-13 | `audits/R442-2026-09-13/` (2026-09-13); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
|
||||
| <a id="b1"></a>[B1](capability-map-detail/B.md#b1) | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (~52 apps, images pinned) | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 6d90374 + 2fe6341 (allow-list R-971, appdata R-1142) | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/RERUN-p1p3-2026-07-06.md` (2026-07-06) |
|
||||
| <a id="b2"></a>[B2](capability-map-detail/B.md#b2) | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | the actions; remove-with-data re-proven 2026-09-13; **re-check (2026-10-24):** mechanism changed after its evidence — 8ae9178 (removal deletes labelled volumes, R-1117) | `audits/R442-2026-09-13/` (2026-09-13); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
|
||||
| <a id="b3"></a>[B3](capability-map-detail/B.md#b3) | Update is GUARDED: it refuses without a restorable backup, backs up first when the copy is stale, and HOLDS an app that does not come up — on ANY backup tier, and the release itself arrives by the managed floor | controller v0.237.0 + v0.238.0 + v0.238.1 + v0.239.0, hub v0.112.0 | **PROVEN-LIVE** | — | `audits/slice4-2026-09-13/` (2026-09-13); `audits/rulings-r472-r475-2026-09-13/` (2026-09-13) |
|
||||
| <a id="b4"></a>[B4](capability-map-detail/B.md#b4) | A failed update is UNDONE by the box itself — the previous version back with its data from seconds before the update; the app is held only if that undo fails too | controller v0.263.2 (the undo), v0.264.0 + hub v0.120.0 (the mail) | **PROVEN-LIVE** | scratch 9202 | `audits/undo-live-2026-09-23/README.md` (2026-09-23) |
|
||||
| <a id="b5"></a>[B5](capability-map-detail/B.md#b5) | The catalog holds only TESTED steps — an image move without a proven test record (bench + box, digests the registry still serves) is refused at push time | catalog `6db08a5` (test-record gate, `ladder.py`) | **PROVEN-LIVE** | written PROVEN: the catalog gate against the live registry | `audits/DRILL-night-2026-09-23.md` (2026-09-23) |
|
||||
| <a id="b6"></a>[B6](capability-map-detail/B.md#b6) | A box behind climbs ONE tested step per press, each with its own definition (`09` §3 decision 14) | controller v0.268.0 + catalog `5ed599c` | **PROVEN-LIVE** | — | `audits/ladder-2026-09-24/partD/10-romm-two-steps.json` (2026-09-24) |
|
||||
| <a id="b7"></a>[B7](capability-map-detail/B.md#b7) | A file app comes back WHOLE from the second drive — its files by four rules (never delete, never overwrite a newer file, bring back what is missing, keep an older replaced file beside it), then its settings and database (`09` §3 decision 26) | controller v0.269.0 (`backup/tier2_whole.go`) | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A1/21-readback.json` (2026-09-24) |
|
||||
| <a id="b7"></a>[B7](capability-map-detail/B.md#b7) | A file app comes back WHOLE from the second drive — its files by four rules (never delete, never overwrite a newer file, bring back what is missing, keep an older replaced file beside it), then its settings and database (`09` §3 decision 26) | controller v0.269.0 (`backup/tier2_whole.go`) | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb + 5929b8e (D22, D25) | `audits/night-2026-09-24/A1/21-readback.json` (2026-09-24) |
|
||||
| <a id="b8"></a>[B8](capability-map-detail/B.md#b8) | A crash-looping or out-of-memory app is STOPPED by the box, the household and operator are told, and Start gives one more try (decision 28) | controller v0.269.0, hub v0.123.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A3/21-gokapi-page-event.txt` (2026-09-24) |
|
||||
| <a id="b9"></a>[B9](capability-map-detail/B.md#b9) | A held app whose page says support is informed can only be removed KEEPING its data (decision 27) | controller v0.269.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A2/10-held-no-copy-keep-data.json` (2026-09-24) |
|
||||
| <a id="b10"></a>[B10](capability-map-detail/B.md#b10) | The box runs the exact TESTED image of a floating tag, and an installed app keeps its image until a guarded Update moves it (`09` §6.4 part 6, box half) | controller v0.269.1 (`stacks/digest.go`) | **PROVEN-LIVE** | — | `audits/night-2026-09-24/B/21-floating-tag-0.269.1.json` (2026-09-24) |
|
||||
| <a id="b11"></a>[B11](capability-map-detail/B.md#b11) | Automatic updates at night — the update leg (`09` §6.4 part 7) | controller v0.271.0 | **PROVEN-LIVE** | scratch 9202, six simulated nights; demo boxes in Part D | `audits/DRILL-night-2026-09-25.md` (2026-09-25) |
|
||||
| <a id="b11"></a>[B11](capability-map-detail/B.md#b11) | Automatic updates at night — the update leg (`09` §6.4 part 7) | controller v0.271.0 | **PROVEN-LIVE** | scratch 9202, six simulated nights; demo boxes in Part D; **re-check (2026-10-24):** mechanism changed after its evidence — 63dcccf (security_fix steps, 0.341.0) | `audits/DRILL-night-2026-09-25.md` (2026-09-25) |
|
||||
| <a id="b12"></a>[B12](capability-map-detail/B.md#b12) | What `restart` and `update` do to a deployed app whose compose file the catalog already moved | controller v0.235.0 | **PROVEN-LIVE** | HISTORY: the behaviour up to v0.234.0; changed 2026-09-06 — current behaviour is B17 | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
|
||||
| <a id="b13"></a>[B13](capability-map-detail/B.md#b13) | Whether the box UPGRADES an app by itself, with nobody pressing anything | controller | **PROVEN-LIVE** | answer: yes, only when an app fails to come back (measured 2026-09-01) | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
|
||||
| <a id="b14"></a>[B14](capability-map-detail/B.md#b14) | Whether an app UPGRADE can be undone | controller + catalog | **PROVEN-LIVE** | answer: no, once a migration has run | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
|
||||
@@ -147,10 +147,10 @@
|
||||
|---|---|---|---|---|---|
|
||||
| <a id="c1"></a>[C1](capability-map-detail/C.md#c1) | Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded DEGRADED rather than silently normal (installer Case A/B) | agent v0.113, host-install v1.22.0 | **PROVEN-LIVE** | — | `audits/E2D-fresh-vm-2026-07-29.md` (2026-07-29) |
|
||||
| <a id="c2"></a>[C2](capability-map-detail/C.md#c2) | Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) |
|
||||
| <a id="c3"></a>[C3](capability-map-detail/C.md#c3) | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
|
||||
| <a id="c4"></a>[C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/VALIDATION-offbox-storagebox-2026-07-09.md` (2026-07-09) |
|
||||
| <a id="c5"></a>[C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) |
|
||||
| <a id="c6"></a>[C6](capability-map-detail/C.md#c6) | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6B-2026-07-14.md` (2026-07-14) |
|
||||
| <a id="c3"></a>[C3](capability-map-detail/C.md#c3) | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 03c0a69 + 66a69d4 + 5929b8e (capped container R-1214, kept copies D25) | `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
|
||||
| <a id="c4"></a>[C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 8f5bea0 + d5f88a2 (abandon R-1182, off-site proof R-1189) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/VALIDATION-offbox-storagebox-2026-07-09.md` (2026-07-09) |
|
||||
| <a id="c5"></a>[C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 93fb251 + d582feb (D20, D22) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) |
|
||||
| <a id="c6"></a>[C6](capability-map-detail/C.md#c6) | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d5f88a2 + 5929b8e (R-1187 c, R-1191) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6B-2026-07-14.md` (2026-07-14) |
|
||||
| <a id="c7"></a>[C7](capability-map-detail/C.md#c7) | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `pilot/DRILL-GL6-2026-07-08.md` (2026-07-08) |
|
||||
| <a id="c8"></a>[C8](capability-map-detail/C.md#c8) | PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | scoped to one host | `audits/SPIKE-pbsdr-selfheal-2026-07-15.md` (2026-07-15) |
|
||||
| <a id="c9"></a>[C9](capability-map-detail/C.md#c9) | Box survives an unattended app or guest-network failure (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0→v0.190.0, agent v0.92.1 | **PROVEN-LIVE** | demo box; boot-orphan leg 6 of 6 hard resets | `audits/DIAG-bootrecon-drive-absent-2026-08-02.md` (2026-08-02) |
|
||||
@@ -172,12 +172,12 @@
|
||||
| <a id="d3"></a>[D3](capability-map-detail/D.md#d3) | Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14); `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
|
||||
| <a id="d4"></a>[D4](capability-map-detail/D.md#d4) | NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | — | `audits/SPIKE-nas-verify-2026-07-11.md` (2026-07-11); `audits/SPIKE-nas-dsm-2026-07-11.md` (2026-07-11) |
|
||||
| <a id="d5"></a>[D5](capability-map-detail/D.md#d5) | Network storage (NAS) is browse + bulk-media only — it may NOT host an app's data namespace | controller v0.187.0 | **PROVEN-LIVE** | — | `audits/R108-network-app-namespace-2026-07-30.md` (2026-07-30) |
|
||||
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | — | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
|
||||
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb (D22, R-1178) | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
|
||||
| <a id="d7"></a>[D7](capability-map-detail/D.md#d7) | A restore SAYS what it returned, and refuses what it cannot do — the four restore-surface truth defects from the 2026-08-21 drill | controller v0.226.0 (R-353, R-357, R-358, R-360, R-396) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: three of four live; R-357 implemented only | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt` (2026-08-30) |
|
||||
| <a id="d8"></a>[D8](capability-map-detail/D.md#d8) | The box PROVES its own off-site copy still HOLDS something — a backup that is intact and EMPTY is caught without a person | controller v0.231.0 (R-87) + hub v0.110.0 | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r87-offsite-proof-2026-08-31/` (2026-08-31) |
|
||||
| <a id="d9"></a>[D9](capability-map-detail/D.md#d9) | The off-site store is VERIFIED on a cadence — something checks that the customer's backups are still readable | controller v0.228.0 (R-359, R-397, R-399) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r359-integrity-2026-08-30/` (2026-08-30) |
|
||||
| <a id="d10"></a>[D10](capability-map-detail/D.md#d10) | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | — | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13) |
|
||||
| <a id="d11"></a>[D11](capability-map-detail/D.md#d11) | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | — | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
|
||||
| <a id="d10"></a>[D10](capability-map-detail/D.md#d10) | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 (R-1124) | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13) |
|
||||
| <a id="d11"></a>[D11](capability-map-detail/D.md#d11) | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 + 5c3b218, controller 6c6bd87 (R-1124, R-1144) | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
|
||||
| <a id="d12"></a>[D12](capability-map-detail/D.md#d12) | Boot ordering: automount + networking survive reboot; appliance self-heal watchdog | agent v0.85 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13); `audits/CAMPAIGN-6A-2026-07-14.md` (2026-07-14) |
|
||||
| <a id="d13"></a>[D13](capability-map-detail/D.md#d13) | Media apps see EVERY drive; a drive added, pulled, back or safely removed is handled per app (notice, real restart, wiring, torrents paused/resumed) | controller v0.313.0, catalog `0188430`, hub v0.151.0 | **PROVEN-LIVE** | Tier-0 boxes (Tester 1, demo-hp 9201); the fleet with golden 0.313.0 + the floor | `audits/media-build-2026-10-15/G/` (2026-10-15); `audits/media-drive-events-2026-10-14/` (2026-10-14) |
|
||||
| <a id="d14"></a>[D14](capability-map-detail/D.md#d14) | A second local copy COUNTS only on a proven different physical disk: the agent knows each place's disks; a VM box makes the copy but says it cannot check; a same-disk copy is not made; a NAS never hides the one-copy notice; a box with no drive refuses a data-folder app in words | agent v0.157.0, controller v0.317.0, hub v0.154.0 | **PARTIAL** | live: VM `unknown` (Tester 1) and two real-disk boxes `proven` (demo-hp, N100); unit only: the same-disk skip, the NAS notice, the no-drive sentence; `usb_bridge` never seen on a device | `audits/storage-identity-buildA-2026-10-15/README.md` (2026-10-15); `architecture/16-storage-physical-identity.md` §13 |
|
||||
@@ -204,7 +204,7 @@
|
||||
| <a id="e14"></a>[E14](capability-map-detail/E.md#e14) | Multiple household users / per-person accounts | — | **MISSING** | — | R-15 |
|
||||
| <a id="e15"></a>[E15](capability-map-detail/E.md#e15) | A second login step for the dashboard (a TOTP code or a passkey) | — | **MISSING** | — | R-811 |
|
||||
| <a id="e16"></a>[E16](capability-map-detail/E.md#e16) | The household can leave Felhom, or outlive it — the box runs without the hub, the household owns its domain, tunnel and off-site account, and can export everything | — | **MISSING** | as a written answer | R-810 |
|
||||
| <a id="e17"></a>[E17](capability-map-detail/E.md#e17) | The host agent cannot reach root without the operator key: exact sudo patterns, no agent-written file installed where root reads it without a content check, the agent binary only by an operator-signed update checked as root | agent v0.146.1 (R-861; delivered by a step bundle, R-880) | **PROVEN-LIVE** | both demo boxes; three named residuals | `audits/hub-safety-2026-10-05/partF/` (2026-10-05) |
|
||||
| <a id="e17"></a>[E17](capability-map-detail/E.md#e17) | The host agent cannot reach root without the operator key: exact sudo patterns, no agent-written file installed where root reads it without a content check, the agent binary only by an operator-signed update checked as root | agent v0.146.1 (R-861; delivered by a step bundle, R-880) | **PROVEN-LIVE** | both demo boxes; three named residuals; **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 + a0c88e5 (sudoers + priv-apply, D4 drop-in) | `audits/hub-safety-2026-10-05/partF/` (2026-10-05) |
|
||||
| <a id="e18"></a>[E18](capability-map-detail/E.md#e18) | WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | — | `audits/SPIKE-oob-wg-operator-peer-2026-07-05.md` (2026-07-05); `audits/SPIKE-felhom-sshd-2026-07-05.md` (2026-07-05) |
|
||||
| <a id="e19"></a>[E19](capability-map-detail/E.md#e19) | The operator can see WHERE a managed host is — its LAN address and its WireGuard address, on the host page | agent v0.119.0, hub v0.85.0 | **PROVEN-LIVE** | — | `audits/host-addresses-visible-2026-07-31.md` (2026-07-31) |
|
||||
| <a id="e20"></a>[E20](capability-map-detail/E.md#e20) | The operator can see whether a managed host's guests still have working networking — and how often the watchdog had to repair them | agent v0.92.0 (emitter, 2026-07-21), hub v0.104.0 (reader, 2026-08-13) | **IMPLEMENTED** | not proven live | R-319 |
|
||||
@@ -238,7 +238,7 @@
|
||||
| <a id="g4"></a>[G4](capability-map-detail/G.md#g4) | Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules | hub v0.45/0.53, agent | **PARTIAL** | — | `runbooks/publish-train-rules.md` (2026-09-13); R-23 |
|
||||
| <a id="g5"></a>[G5](capability-map-detail/G.md#g5) | Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed | agent v0.70+ | **PROVEN-LIVE** | demo | `audits/SPIKE-agent-selfupdate-2026-07-05.md` (2026-07-05) |
|
||||
| <a id="g6"></a>[G6](capability-map-detail/G.md#g6) | Controller self-update: anonymous registry, no credentials in guest | controller v0.112 | **PROVEN-LIVE** | demo; the file found is the floor-driven self-update leg | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
||||
| <a id="g7"></a>[G7](capability-map-detail/G.md#g7) | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | — | `audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md` (2026-07-09); `audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (2026-07-09) |
|
||||
| <a id="g7"></a>[G7](capability-map-detail/G.md#g7) | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — hub 9688e76e (scoped abandonment, R-1182) | `audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md` (2026-07-09); `audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (2026-07-09) |
|
||||
| <a id="g8"></a>[G8](capability-map-detail/G.md#g8) | Per-customer offsite fill + staleness + freeze lever | hub v0.41 | **IMPLEMENTED** | no live-fired leg | `audits/CAMPAIGN-offsite-overnight-2026-07-10.md` (2026-07-10) |
|
||||
| <a id="g9"></a>[G9](capability-map-detail/G.md#g9) | Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert) | hub v0.64.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.64.0 code and unit tests (outside `documentation/`) |
|
||||
| <a id="g10"></a>[G10](capability-map-detail/G.md#g10) | Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge) | hub v0.65.0 + tenantsync v1.2.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.65.0 code and unit tests (outside `documentation/`) |
|
||||
@@ -246,10 +246,10 @@
|
||||
| <a id="g12"></a>[G12](capability-map-detail/G.md#g12) | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | — | the 2026-07-13 hub closing bundle (no evidence document) |
|
||||
| <a id="g13"></a>[G13](capability-map-detail/G.md#g13) | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | — | hub v0.54 (2026-07-13; no evidence document) |
|
||||
| <a id="g14"></a>[G14](capability-map-detail/G.md#g14) | The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail | hub v0.135.0 (R-604, R-530) | **IMPLEMENTED** | the page live; the alarm and the mail unit-proven | `audits/hub-safety-2026-10-05/partD/` (2026-10-05) |
|
||||
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | — | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
||||
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
||||
| <a id="g16"></a>[G16](capability-map-detail/G.md#g16) | Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail | `scripts/dooplex-offsite/` (R-232), homelab-manifests rules | **PROVEN-LIVE** | — | `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
||||
| <a id="g17"></a>[G17](capability-map-detail/G.md#g17) | The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`) | hub v0.135.0 (R-135) | **PROVEN-LIVE** | — | `audits/hub-safety-2026-10-05/partA/live.txt` (2026-10-05) |
|
||||
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
|
||||
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
|
||||
| <a id="g19"></a>[G19](capability-map-detail/G.md#g19) | An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps | controller v0.259.0 + hub v0.119.0 + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE** | walked on 0.258.0 with one blocker; fixed, the walk not repeated | `audits/DRILL-first-hour-en-0258-2026-09-20.md` (2026-09-20); `audits/i18n-closing-2026-09-21/live/` (2026-09-21) |
|
||||
| <a id="g20"></a>[G20](capability-map-detail/G.md#g20) | A deletion of a customer's off-site history is NOTICED within a day | hub v0.111.0 (R-431) | **IMPLEMENTED** | not yet proven live | R-431 |
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
|
||||
|
||||
**Status:** draft (decisions from the topology/trust design sessions).
|
||||
**Platform facts** referenced here live in `docs/proxmox-platform.md`; this document
|
||||
**Platform facts** referenced here live in `documentation/proxmox-platform.md` (path corrected 2026-10-24); this document
|
||||
records *Felhom's decisions*, not Proxmox behaviour.
|
||||
|
||||
---
|
||||
@@ -105,7 +105,7 @@ credentials.
|
||||
- **[FACT, 2026-10-16, R-970]** The last contradiction is gone: until controller v0.322.0 the box's own hub API key,
|
||||
sent as a Bearer token, opened `/api/config/` (read `controller.yaml` with every secret in it, and REPLACE it,
|
||||
unsigned) and the self-update and geo routes. Nothing called them (the hub never connects to a box). The config
|
||||
routes are removed and no route accepts the hub key; every `/api/` route needs a dashboard session (pinned by
|
||||
routes are removed and no route accepts the hub key; every `/api/` route but the read-only `/api/health` needs a dashboard session (pinned by
|
||||
`TestR970_HubKeyOpensNoDoor` on the real route table). The box exposes no control endpoint, as stated above.
|
||||
- All operator-initiated actions are recorded in a **customer-visible audit log**.
|
||||
|
||||
@@ -132,7 +132,7 @@ of all 53 apps, 2026-09-28).
|
||||
**[DESIGN, 2026-10-17, R-1126, operator decision D11 = A] The tunnel runs before the claim, and FileBrowser never
|
||||
answers on a default password.** The box starts cloudflared as soon as it has a tunnel token, before it is claimed,
|
||||
because the claim itself is reached through it: the claim mail links `https://felhom.<domain>`
|
||||
(`hub/internal/notify/templates.go:27`). Before the claim a stranger reaches only the claim page (a code: 72 h life, 5
|
||||
(`hub/internal/notify/templates.go` `FormatClaimEmail`; line corrected 2026-10-24). Before the claim a stranger reaches only the claim page (a code: 72 h life, 5
|
||||
tries, 15-minute lockout — controller `web/claim.go`) and `files.<domain>`. FileBrowser starts every new database on
|
||||
admin/admin, and its own admin-password setting cannot replace that (it re-applies at every start and would overwrite
|
||||
the household's own password — FileBrowser 1.5.6 `backend/cmd/user.go:50-57`, measured). So FileBrowser starts OFF
|
||||
@@ -219,7 +219,10 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
||||
This removes any zero-touch remote-enrollment problem.
|
||||
- A **one-time retrieval code** mints durable identity. Single-use (burned on the successful
|
||||
config fetch) plus a short *pre-use* TTL; one-click regenerate for the only real failure
|
||||
case (fetch fails before anything is persisted). After the fetch, the code is irrelevant —
|
||||
case (fetch fails before anything is persisted). *(As built — corrected 2026-10-24: the passphrase is not burned
|
||||
and has no TTL; it gives keys until the box has a host record AND a controller ≥ 0.334.0 has reported with its own
|
||||
key, then `/config`, `/recovery` and `/host-enroll` answer 409 — D19, R-1159, `hub/internal/api/passphrase_scope.go:19-30`;
|
||||
`05` §8 has the full rule.)* After the fetch, the code is irrelevant —
|
||||
everything downstream runs on durable credentials, so retries don't need it.
|
||||
- **Order:** the agent enrolls first (and, running as root at setup, mints its own scoped
|
||||
operator-tier Proxmox token), then provisions the customer LXC from the golden template and
|
||||
@@ -253,7 +256,7 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
||||
2026-09-14 first-hour drill used a `*.felhom.eu` customer domain with no tunnel, and the dashboard link
|
||||
in the setup-code mail did not resolve.
|
||||
- **A pasted Cloudflare API token must reach exactly the customer's own zone** (R-138 option C, `09` §3 decision 190;
|
||||
hub main 2026-10-08, unreleased). On save the hub asks Cloudflare `GET /zones` with the token and keeps it only when
|
||||
released in hub v0.144.0 — corrected 2026-10-24). On save the hub asks Cloudflare `GET /zones` with the token and keeps it only when
|
||||
the token sees exactly ONE zone and that zone IS the customer's domain (a zone above it is refused: a sibling
|
||||
customer added later would be reachable). Cloudflare unreachable → the save fails and the old token stays. The token
|
||||
is never logged. **Limit:** `/zones` lists what the token can READ; a hand-built token with Zone:Read on one zone and
|
||||
@@ -313,7 +316,8 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
||||
>
|
||||
> **What being unmarked cost.** The consequence of this bullet — that 40 of 53 catalogue templates
|
||||
> declare no configurable path because they are all-hot — is stated as **[FACT]** at
|
||||
> `07-backup-architecture.md:296-299`. A reader met a marked observation beside an unmarked choice
|
||||
> `07-backup-architecture.md:296-299`. *(Dated note 2026-10-24: that pointer is dead — those lines are about the local
|
||||
> plaintext posture now, and no architecture file but this one states „40 of 53 … all-hot".)* A reader met a marked observation beside an unmarked choice
|
||||
> and reasonably asked whether it *should* be so. **Between 19 and 22 August that reader called this
|
||||
> decision a defect in four places** (R-370), and one session's work went into correcting the record
|
||||
> rather than into the product.
|
||||
@@ -343,7 +347,9 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
||||
restores from PBS; the controller returns on the same domain (tunnel reused from the hub
|
||||
record). DR = provisioning + a restore mode, not a separate mechanism.
|
||||
- **Snapshot-before-deploy:** controller asks the agent to snapshot, deploys, runs its
|
||||
post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.)
|
||||
post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.) **[DESIGN — not built,
|
||||
noted 2026-10-24]:** the agent serves `/snapshot` and `/rollback`, but no controller code calls them
|
||||
(`felhom-controller/controller/internal/agentapi`); the update's guard is the per-app backup + undo of `09` §3 decision 15.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -47,7 +47,7 @@ Owns:
|
||||
1. **Proxmox lifecycle** — create/start/stop/destroy guests, snapshots, storage allocation. Via a scoped Proxmox API token (the **`FelhomAgent` operator role** — `proxmox-platform.md` §3.6, validated Phase 3 B3) for everything the API covers; raw host ops only where unavoidable.
|
||||
2. **Storage management** — attach/classify targets, reconcile the storage manifest, mount USB-by-UUID, present mounts into guests.
|
||||
3. **Backup/restore orchestration** — vzdump to the tiers, PBS, snapshot management, and the **self-restore-test**.
|
||||
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Three states: `running` (healthy), `not_running` (stopped, absent, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
|
||||
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Four states (corrected 2026-10-24): `running` (healthy), `not_running` (stopped, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms — and since agent v0.161.0 `absent` (the guest has no cloudflared container at all, R-1070; the hub never alarms on it) — `internal/hub/cloudflared.go:12-23`. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
|
||||
Proxmox API, plus the OS wrapper's read-only `facts` (host Debian, next-boot kernel, held packages, taint, the crash
|
||||
guard; guest Debian, Docker engine, containerd, live-restore), at most every 10 min. **Still no new sudoers line** — the
|
||||
facts, `live-restore-on` and the Docker layer all ride `FELHOM_OSAPPLY`. New ROOT-owned files instead (installer 1.30.0;
|
||||
@@ -129,13 +129,14 @@ the one trust root; the agent's API token is untouched; a box gets the new rule
|
||||
`agent_config_update` (the order: signed `agent_update` first, then the bundle — after the bundle, an agent below
|
||||
0.146.0 cannot update itself on that box).
|
||||
|
||||
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — built as a ready patch, NOT shipped] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
|
||||
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — SHIPPED in agent v0.163.0 (`1109092`) and installer-v1.36.0 (`manifests/webpage.yaml:347`); corrected 2026-10-24] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
|
||||
|
||||
**[FACT, agent main `a0c88e5`, unreleased] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
|
||||
**[FACT, agent `a0c88e5`, released in v0.163.0 — corrected 2026-10-24] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
|
||||
|
||||
**[DESIGN, operator ruling D30 2026-10-22; agent main `ae69ad4`, unreleased] The tunnel heals itself by a new key.** When
|
||||
the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the old
|
||||
private key kept as `wg/private.key.prev`, 0600 — the escrow may hold it) and registers the new public key on the
|
||||
**[DESIGN, operator ruling D30 2026-10-22; agent `ae69ad4`, released in v0.163.0 — corrected 2026-10-24] The tunnel heals itself by a new key.** When
|
||||
the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the FIRST old
|
||||
private key kept as `wg/private.key.prev`, later ones as `wg/private.key.prev.<unix>`, the newest three kept, 0600 — the
|
||||
escrow may hold one; the once-an-hour bound survives a restart in `wg/last-rekey` — `internal/wgtunnel/rekey.go`, corrected 2026-10-24) and registers the new public key on the
|
||||
existing path, `POST /hosts/<id>/wg`: the hub re-keys the peer in place (same /32, generation bump) and its full-list
|
||||
push to ep0 (`wgsync`, `wg syncconf`) drops the old key and adds the new one. A new key is a new peer for ep0, with no
|
||||
stored timestamp. Once an hour at most; not while the unit is down; never during a re-register already in flight. The
|
||||
@@ -250,8 +251,9 @@ The controller (in its LXC) reaches the agent (on the host) over the local bridg
|
||||
- `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
|
||||
- `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
|
||||
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. **`host_busy`
|
||||
(R-921, agent Unreleased `56aabc7`):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
|
||||
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off), the
|
||||
(R-921, agent `56aabc7`, released in v0.161.0 — corrected 2026-10-24):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
|
||||
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off, and since v0.164.0 the
|
||||
weekly upload line test — gate name `line-test`), the
|
||||
state in which `POST /backup` answers 409; absent when the gate is not wired. The controller (≥ 0.335.0) reads it
|
||||
before it stops any app. A read, never a reservation.
|
||||
- **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record
|
||||
@@ -314,7 +316,8 @@ token → guest and authorizes per guest, so a compromised controller's blast ra
|
||||
quiesce-on-due consumer is 8B), the rest wrap the existing slice-5/6/7 machinery.
|
||||
|
||||
- **Transport / pin.** The agent serves a **persisted self-signed leaf** bound to the host bridge IP
|
||||
on a fixed port (default `:8443`). The controller pins the **leaf-cert SHA-256** (decision:
|
||||
on a fixed port — `local_api.listen_addr` is REQUIRED, there is no default (`internal/config/config.go:317-319`); an
|
||||
island install (R-50) listens on `169.254.253.1:8443` on its private bridge (corrected 2026-10-24). The controller pins the **leaf-cert SHA-256** (decision:
|
||||
consistency with the agent's Proxmox/PBS cert pinning), carried in its bootstrap. The leaf is
|
||||
generated **once and persisted**, so its fingerprint is stable across agent restarts (a fresh cert
|
||||
each boot would invalidate every already-issued bootstrap pin). Defense-in-depth: the listener
|
||||
@@ -420,8 +423,8 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
|
||||
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
|
||||
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
|
||||
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
|
||||
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent Unreleased
|
||||
> `56aabc7`):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
|
||||
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent
|
||||
> `56aabc7`, released in v0.161.0 — corrected 2026-10-24):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
|
||||
> snapshot already gone), it retries once with Proxmox's own `force=1` on the same API call (no sudoers line, no
|
||||
> root command); any other error is returned unchanged, and a failed forced delete is never forced again.
|
||||
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
|
||||
@@ -677,7 +680,8 @@ the *archive* and the *back half*, **and in identity policy** (below).
|
||||
`swap`, and the agent never set it; on the 8 GB minimum machine (guest 4096 MB) the household's first uploads filled the
|
||||
guest with its swap full, the kernel could evict only program code, and the box froze (memory „full" pressure 96 %, load
|
||||
162, the dashboard dead 6½ minutes). `[DESIGN]` The size step sets `swap` = the guest's memory: at bring-up, at a RAM
|
||||
resize (`POST /guest/memory`), and once a minute after agent start for every pool LXC whose swap is below its RAM (live,
|
||||
resize (`POST /guest/memory`), and ONCE, one minute after agent start, for every pool LXC whose swap is below its RAM (`cmd/felhom-agent/main.go`
|
||||
`EnsureGuestSwap`; „once a minute" corrected 2026-10-24) (live,
|
||||
no restart, never lowers). Proven on a fresh box the night of 2026-10-21 (`audits/night-2026-10-21/P1-soak/`: 512 →
|
||||
4096 one minute after the fixture agent started; the 2-hour household load then answered every 10-second probe). **The
|
||||
host must have swap behind it:** our installer gives an 8 GB / 256 GB machine 7.75 GB (measured the same night); a BYO
|
||||
|
||||
@@ -15,9 +15,9 @@
|
||||
|
||||
|
||||
|
||||
> Status: design draft (decision content), grounded on `docs/tests/phase4-signing-findings.md`.
|
||||
> Status: design draft (decision content), grounded on `documentation/tests/phase4-signing-findings.md`.
|
||||
> To be reviewed by Claude Code against that spike + `03` §4, then placed at
|
||||
> `docs/architecture/04-control-plane-authorization.md`.
|
||||
> `documentation/architecture/04-control-plane-authorization.md` — where it is now (paths corrected 2026-10-24).
|
||||
>
|
||||
> Builds on Part 1 (enrollment / trust), Part 3 (the agent verifies + the §4 reversibility gate).
|
||||
> This doc defines the **mechanism** behind `03` §4's "an operator signature the hub can't forge."
|
||||
@@ -75,7 +75,12 @@ recorded last**, only after everything else passes, so an invalid signature can
|
||||
nonce (DoS-safe). Each layer is mandatory and was proven to reject independently (Phase 4 §3–4):
|
||||
- **target binding** — `target.host_id`/`guest_id` must equal *this* box/guest (a signature for box
|
||||
A cannot be replayed at box B);
|
||||
- **time window** — `now ∈ [issued_at, expires_at]`;
|
||||
- **time window** — `now ∈ [issued_at, expires_at]`; as built, `issued_at` may be up to **2 minutes** in the future
|
||||
(clock skew), `expires_at` is never extended — the agent (`felhom-agent/internal/authz/verifier.go:19-22,148-155`) and
|
||||
the root wrapper (`configs/felhom-os-apply`) alike (corrected 2026-10-24). **[DESIGN — undocumented, found
|
||||
2026-10-24]:** neither caps the window's LENGTH (`expires_at − issued_at`); `felhom-opsign` defaults to 30 min
|
||||
(`cmd/felhom-opsign/main.go:57`), and the nonce store keeps a record until its expiry. A register row asks the
|
||||
operator to confirm or cap it;
|
||||
- **nonce** — unseen within the window (the nonce store **must be persistent across agent restarts**
|
||||
and expiry-pruned; a non-persistent store reopens the replay window after a restart).
|
||||
|
||||
@@ -166,7 +171,7 @@ op the agent verifies** (same pipeline, §2.3) — never unauthenticated config.
|
||||
queue — then the agent polls, verifies, executes, and audits. One command + passphrase, from the
|
||||
desk. **Never** a site visit.
|
||||
|
||||
### 6.1 Operator actions in the report reply [DESIGN, R-314/R-279 — `09` §3 decision 185, controller + hub main 2026-10-08, unreleased]
|
||||
### 6.1 Operator actions in the report reply [DESIGN, R-314/R-279 — `09` §3 decision 185; released in controller v0.304.0 + hub v0.144.0 — corrected 2026-10-24]
|
||||
|
||||
Routine, non-destructive operator requests need no signature (`03` §4 asks for one only to destroy or overwrite the
|
||||
only copy). The hub stores a row, bumps the box's intent, and the report reply carries `operator_actions:[{id,action,arg}]`
|
||||
|
||||
@@ -22,8 +22,9 @@
|
||||
> The hub is **not** greenfield — it's a mature service (felhom-hub **v0.11.0** as of 2026-06-13, Go +
|
||||
> SQLite on k3s, `hub.felhom.eu`). This doc is the **deltas** to evolve it for the Proxmox model, plus
|
||||
> the new data model. Builds on Part 1 (trust/enrollment), Part 3 (the agent + reconcile), Part 4 (signing).
|
||||
> NOTE: this remains a design-draft for the deltas; the live hub is v0.11.0 (the prior "v0.6.3" reference
|
||||
> was stale). Ground specifics against `felhom.eu/hub/` before relying on them.
|
||||
> NOTE: this remains a design-draft for the deltas. It was written against hub v0.11.0 (2026-06-13); the deployed
|
||||
> version is read from `manifests/hub.yaml`, never from this line (corrected 2026-10-24, review round 2: the old
|
||||
> "the live hub is v0.11.0" was a current-state version literal). Ground specifics against `felhom.eu/hub/` before relying on them.
|
||||
|
||||
## 1. Source-of-truth model — two drivers, two directions
|
||||
|
||||
@@ -91,7 +92,7 @@ Evolves the existing staleness checker (60s **cadence**, a **configured** thresh
|
||||
|
||||
- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
|
||||
liveness signal; a silent agent = the box is gone (the critical alert). **A hub restart does not re-mail a known
|
||||
silence (R-1001, hub 0.162.0 — built 2026-10-19, not yet deployed):** a host whose newest `host_stale`/`host_down`
|
||||
silence (R-1001, hub 0.162.0 — deployed since; corrected 2026-10-24):** a host whose newest `host_stale`/`host_down`
|
||||
event is newer than its last report is seeded with that state at start, silently; `restore_test_stale` reads its edge
|
||||
from the DB on the first sweep (a signal for the tier inside the read window and newer than its last proof). A failed
|
||||
lookup lets the alarm through.
|
||||
@@ -139,7 +140,7 @@ itself to the floor (R-1052: not yet proven end to end). Setting the global floo
|
||||
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
|
||||
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
|
||||
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list
|
||||
offers real agent packages only (R-1011, hub 0.162.0, not yet deployed):** the sha lookup asks for exactly the
|
||||
offers real agent packages only (R-1011, hub 0.162.0, deployed since; corrected 2026-10-24):** the sha lookup asks for exactly the
|
||||
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
|
||||
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
|
||||
(`artifact_file_missing`).
|
||||
@@ -326,8 +327,9 @@ customer instead of a single controller.
|
||||
- **DROP `reports` + CREATE `guest_reports`** (under the clean cutover this is drop+create with no data
|
||||
migration, not an in-place rename); `guest_reports` adds `guest_id`, `host_id`; `cpu/memory` mean
|
||||
guest-level; `backup_last_snapshot` goes quiet.
|
||||
- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`; the
|
||||
slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`.
|
||||
- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`. ~~The
|
||||
slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`~~ — retired, never built (§9
|
||||
above, R-105) *(corrected 2026-10-24, review round 2)*.
|
||||
- **DROP both** `infra_backup_versions` (current/live) **and** `infra_backups` (legacy mirror) — the DR
|
||||
recipe, tenantsync and the escrow blob replace them (R-105).
|
||||
- **KEEP** `customer_configs`, `events`, `customer_notifications`, `notification_log`,
|
||||
@@ -350,7 +352,9 @@ customer instead of a single controller.
|
||||
|
||||
The box's console tells the volunteer to open „az e-mailben kapott link". That mail must exist whenever
|
||||
a customer is **waiting for a box**, without an operator press. `autoMintSelfBindLink` runs on four
|
||||
events: **customer creation**; **RESET completion**; **an e-mail set or changed on a customer with no
|
||||
events (a fifth path mails a link too: the household's „new link" on a USED or EXPIRED link — one mail per customer
|
||||
per hour, R-719; when the customer still has a linked host it shows the „oldbox" page in the customer's language and
|
||||
tells the operator instead, R-1203 — `internal/web/selfbind.go` `handleBindResend`) *(corrected 2026-10-24, review round 2)*: **customer creation**; **RESET completion**; **an e-mail set or changed on a customer with no
|
||||
bound host** (config save); **a host delete** that keeps the customer. The last two re-check that no
|
||||
host is bound, so a customer who has a box is never sent a pairing link. **Appliance registration is
|
||||
not a trigger** — it knows no customer (`api/appliance.go`). Every send is stored as a hub-internal
|
||||
@@ -388,7 +392,7 @@ it has seen any box screen, and until v0.118.0 nothing here described that.
|
||||
|
||||
| Mail | Trigger | Rendered by | Language source |
|
||||
|---|---|---|---|
|
||||
| Event notification (39 event types) | a box event, or a hub checker | `FormatCustomerEmail` | reported → created-with → `hu` |
|
||||
| Event notification (44 event types — the `mail.event.*` keys of `internal/i18n/locales/hu.json`, counted 2026-10-24) | a box event, or a hub checker | `FormatCustomerEmail` | reported → created-with → `hu` |
|
||||
| Claim / reset / re-enroll / claimed | the claim arc | `FormatClaimEmail` | created-with (no box has reported yet) |
|
||||
| Self-bind link | customer creation, or the operator's button | `FormatSelfBindEmail` | created-with |
|
||||
| The public bind PAGE at `/bind/<token>` | the customer opens the link | one template per language | created-with, **except `expired`** — see 15.4 |
|
||||
@@ -456,7 +460,7 @@ The operator mail renders the counts as one `Repair:` line. The household's side
|
||||
date taken) are shown on its own app page, on its box, until it dismisses them — file names never leave the box.
|
||||
|
||||
|
||||
### 15.4b A box whose tunnel to ep0 is dead — the clock-ahead alarm [hub 0.166.0, built, NOT deployed; R-1204]
|
||||
### 15.4b A box whose tunnel to ep0 is dead — the clock-ahead alarm [hub 0.166.0, deployed with 0.167.0; R-1204 — corrected 2026-10-24]
|
||||
|
||||
**[FACT, hub `e267c5e2`]** The host report carries the agent's `wireguard` block (`hostWireguard`). `checkWGTunnel` (`api/host_tunnel.go`, called after `checkHostSwap`) calls the tunnel stale when the box reports `handshake_future_s` > 0 (the clock-ahead lock-out) or a last handshake older than 6 h (healthy boxes measured at most 113–124 s over 24 h). It is edge-triggered against the host's latest tunnel event (`store.LatestHostTunnelEvent`): `wg_tunnel_stale` once on the way down, `wg_tunnel_ok` once on the way back; both are allow-listed and operator-only (`notify.operatorOnlyEvents`, `wg_tunnel_operator_only_test.go`) — the household cannot repair a tunnel. A box with no `wireguard` block (an agent before the field, or no tunnel) raises nothing.
|
||||
|
||||
@@ -580,9 +584,13 @@ with the same `OFFSITE_SECRET_KEY`; today that key exists only on DooPlex (the k
|
||||
on the same machine). The off-site plan for the database and its key: `runbooks/RUNBOOK-hub-db-offsite-backup.md`
|
||||
(R-173 — option A decided 2026-10-05, `09` decision 125; §16.3).
|
||||
|
||||
### 16.3 The nightly database snapshot (R-173, hub v0.136.0)
|
||||
### 16.3 The database snapshot — every 6 hours since D34 (R-173, hub v0.136.0; D34 in hub 0.167.0)
|
||||
|
||||
Every night at **02:00 Budapest** the hub writes `<data>/snapshots/hub-<UTC stamp>.db` with SQLite's `VACUUM INTO`:
|
||||
**Corrected 2026-10-24 (review round 2):** since D34 (operator yes 2026-10-23) the hub writes a snapshot at **02:00,
|
||||
08:00, 14:00 and 20:00 Budapest** (`internal/dbsnap/dbsnap.go` `Slots`), catches up at start when the newest is older
|
||||
than **6 h** (`CatchUpAge`), and DooPlex pushes the newest at **:30 after each slot** (`felhom-hub-db-backup.timer`),
|
||||
refusing one older than **7 h** (`felhom-hub-db-backup` `MAX_AGE_H`). The paragraph below is the original nightly
|
||||
design; its mechanics hold per slot. Originally: every night at **02:00 Budapest** the hub writes `<data>/snapshots/hub-<UTC stamp>.db` with SQLite's `VACUUM INTO`:
|
||||
one statement, one point in time, every committed write included — the rows still in `hub.db-wal` too, which a file copy
|
||||
of `hub.db` loses (measured in `internal/dbsnap` tests: a plain copy held 0 of 150 fresh rows). Written as `.tmp`, mode
|
||||
`0600`, then renamed, so a reader never sees half a file. The newest **2** stay (the volume grew from 1 GiB to 2 GiB
|
||||
@@ -611,7 +619,7 @@ failure never fails or delays a request: lines go through a non-blocking buffer
|
||||
|
||||
### 16.5 Daily jobs and the clock change [DESIGN — hub v0.163.0, operator ruling D24 2026-10-21, R-1183]
|
||||
|
||||
Every daily hub job (`scheduleDaily`: db-snapshot 02:00, prune, deadline-check 05:00, offsite-key-audit 07:10) runs at
|
||||
Every daily hub job (`scheduleDaily`: the four db-snapshot slots 02:00/08:00/14:00/20:00, prune at its configured time, deadline-check 05:00, offsite-key-audit 07:10 — slots corrected 2026-10-24) runs at
|
||||
its Budapest wall-clock time **once per calendar day**: the next run is the calendar day after (`time.Date`, day+1),
|
||||
never „now + 24 h", and the slot that ran last never runs again. On a 25-hour day (the last Sunday of October) the old
|
||||
arithmetic ran every job after 02:00 twice; on a 23-hour day it ran them an hour late. Pinned by `TestR1183_*` for
|
||||
|
||||
@@ -19,7 +19,8 @@
|
||||
> decisions; grounded against felhom.eu @ `bf099f6` and felhom-agent @ `4ba1b14` (v0.63.0).
|
||||
> Evidence base: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (all
|
||||
> measurements cited below as "spike Pn"). This document ends in an implementation slice roadmap
|
||||
> (§8); **no slice is implemented yet** — each is a separate future task. Consistent with and
|
||||
> (§8); **no slice is implemented yet** — each is a separate future task. *(Dated note 2026-10-24, review round 2: true when
|
||||
> written; S1–S4 and S5's safe halves have since shipped — §8 and the agent's `internal/wgtunnel` say what is built.)* Consistent with and
|
||||
> extending Part 01 (`01-topology-and-trust.md`); where this doc names a component's duty it uses
|
||||
> Part 01 §3 vocabulary.
|
||||
|
||||
@@ -140,7 +141,9 @@ before this ships (the drilled `--rescope-acl` posture is unaffected).
|
||||
4. **Hub → endpoint VM:** adds `{pubkey, /32}` to the WG server peer list (slice-1 surface).
|
||||
5. **Box (agent):** next heartbeat sees the generation advance → fetches desired-state → writes
|
||||
the WG config → `install` + `systemctl enable --now wg-felhom` (the sudoers `*.mount` pattern
|
||||
of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3).
|
||||
of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3). *(As built since R-861: no `install` sudoers
|
||||
line; the conf is installed 0600 to its fixed path by the root wrapper `felhom-priv-apply wg` —
|
||||
`felhom-agent/configs/felhom-agent.sudoers:223-229`; corrected 2026-10-24.)*
|
||||
6. **Box:** tunnel handshakes (outbound, keepalive); PBS storage is (re)pointed at
|
||||
`pbs_tunnel_ip:8007` (§3.4); first backup runs.
|
||||
|
||||
@@ -169,7 +172,8 @@ tenant ACL (`DatastoreBackup` on `/datastore/<ds>/<ns>` to BOTH the user and the
|
||||
### 3.5 Key custody + DR re-establishment
|
||||
|
||||
The WG private key joins the **escrowed IdentityBundle**
|
||||
(`internal/escrow/identity.go:24-27`, today `{TunnelToken, PBSToken}` → `+ WGPrivateKey`) — the
|
||||
(`internal/escrow/identity.go`, then `{TunnelToken, PBSToken}` → `+ WGPrivateKey`; as built the bundle carries FOUR
|
||||
fields — `TunnelToken`, `PBSToken`, `WGPrivateKey`, `ResticRepoPassword` — `identity.go:30-40`, corrected 2026-10-24) — the
|
||||
same R-wrapped, zero-knowledge age blob the DR recipe already consumes. On host-loss
|
||||
re-provisioning (01 §9 restore mode): the box recovers the bundle with the customer's recovery
|
||||
code, restores the WG key, and re-establishes the tunnel; hub-side the peer entry still exists
|
||||
@@ -199,7 +203,8 @@ disables the service, KEEPS its marker, and never re-registers on its own — re
|
||||
revoked until the operator re-adds the peer (the heartbeat report carries the box's pubkey as
|
||||
the recovery handle; **re-add via `POST /hosts/{id}/wg` with the global key** — the raw registry
|
||||
add does not bump the host's generation). The only automatic re-registration is the
|
||||
pubkey-mismatch (DR/re-key) path, backoff-bounded. Absent DATA (a failed fetch) is never a
|
||||
pubkey-mismatch (DR/re-key) path, backoff-bounded — and, since D30 (agent v0.163.0), a key the agent makes itself after
|
||||
a handshake stamped in the future for 15 min (`internal/wgtunnel/rekey.go`; corrected 2026-10-24). Absent DATA (a failed fetch) is never a
|
||||
teardown signal. **Rollout gate:** `wg_tunnel.enabled` defaults FALSE agent-side until the
|
||||
production endpoint exists.
|
||||
|
||||
@@ -285,7 +290,9 @@ production endpoint exists.
|
||||
`wg-felhom` AllowedIPs so it survives the agent's self-heal ([OF-1]); PBS unaffected.
|
||||
- **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one: no handshake
|
||||
within ~3 keepalive periods → the offsite target reports unreachable → the existing alerting path
|
||||
carries it. No new alarm channel.
|
||||
carries it. No new alarm channel. *(Since hub 0.166.0 there is one: `wg_tunnel_stale` / `wg_tunnel_ok`,
|
||||
operator-only — a handshake older than 6 h or stamped in the future, D30 / R-1204, `hub/internal/api/host_tunnel.go`;
|
||||
corrected 2026-10-24.)*
|
||||
**EXTENDED 2026-07-05 (TASK H1).** The agent's heartbeat now also carries an **`oob` stanza**
|
||||
(`felhom_sshd_active`, `felhom_sshd_port`, `reachable`, `config_invalid`, `operator_peer_configured`,
|
||||
`operator_key_configured`, `wg_handshake_age_s`) — the operator's "can I get into this box right
|
||||
@@ -355,7 +362,8 @@ venue; it gates nothing. Runbook:
|
||||
**Open sub-decisions (deferred by design):**
|
||||
- Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — **slice-1 design point** (§5).
|
||||
- WG key rotation cadence/mechanics — the reconcile channel can carry it (new pubkey → hub →
|
||||
endpoint swap); design when fleet size warrants, possibly the moment the "separable fleet
|
||||
endpoint swap); design when fleet size warrants *(one automatic rotation exists since D30: after a future-stamped
|
||||
handshake for 15 min, at most once an hour — `rekey.go`; a scheduled rotation is still open; noted 2026-10-24)*, possibly the moment the "separable fleet
|
||||
layer" (D1) conversation reopens.
|
||||
- Endpoint scaling/sharding (N customers per VM, second region, HU-hosted option) — capacity
|
||||
planning, not architecture; the peer registry already keys per-host.
|
||||
@@ -378,6 +386,8 @@ venue; it gates nothing. Runbook:
|
||||
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
|
||||
|
||||
**MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6
|
||||
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a
|
||||
read-only prober that names itself "the seam for the tunnel-management slice") adopts the
|
||||
S3-piloted agent-managed-service pattern in a later, separate slice.
|
||||
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — then a
|
||||
read-only prober that named itself "the seam for the tunnel-management slice") adopts the
|
||||
S3-piloted agent-managed-service pattern in a later, separate slice. *(Dated note 2026-10-24: cloudflared is a GUEST
|
||||
container the agent only reads — four states, `cloudflared.go:12-23`; `03` §2 says the agent will not manage it.
|
||||
Whether this plan is withdrawn is the operator's — a row of review round 2.)*
|
||||
|
||||
@@ -135,7 +135,9 @@ live 2026-08-23 — one event across 22 scans.
|
||||
|
||||
**The vocabulary is the HUB's and it is exact: `{info, warning, error, critical}`.** Anything else is
|
||||
**coerced to `info` at ingest**, and `info` is dropped by `severityNotifies` before *both* delivery
|
||||
legs. So a severity outside the set means the event is stored, answers `200`, shows on the dashboard —
|
||||
legs — except the paired recoveries in `recoveredPairedDownTypes` (`node_recovered`, `host_recovered`,
|
||||
`whole_guest_backup_recovered`, `pbsdr_box_recovered`, `offsite_box_recovered`, `storage_reconnected`,
|
||||
`health_recovered`), which run before the gate and are mailed (`notify/dispatcher.go` — corrected 2026-10-24, review round 2). So a severity outside the set means the event is stored, answers `200`, shows on the dashboard —
|
||||
and is e-mailed to **nobody**.
|
||||
|
||||
**This shipped twice.** `DiskAlertKind.Severity` emitted `"warn"` until controller v0.215.0;
|
||||
@@ -155,8 +157,10 @@ Three things now hold it:
|
||||
3. **The dispatcher's `unrecognized severity` branch is kept**, because the hub's own monitor checkers
|
||||
call `ProcessEvent` directly and never pass the ingest handler. For them it is the only guard.
|
||||
|
||||
**Who gets it.** `processOperator` consults only `operatorOn`, the address and a 1-hour cooldown —
|
||||
**never customer preferences** — so a valid severity always reaches the operator. `processCustomer`
|
||||
**Who gets it.** `processOperator` consults only `operatorOn`, the address and a cooldown —
|
||||
1 hour, except 5 min for node/host liveness and 6 h for a storage-fill warning (`operatorCooldownFor`); it records a
|
||||
`backup_tier_skipped` in a box's FIRST hour as „suppressed" instead of mailing it (R-723), and the mail hold applies —
|
||||
**never customer preferences** (cooldowns and R-723 corrected 2026-10-24) — so a valid severity always reaches the operator. `processCustomer`
|
||||
consults `operatorOnlyEvents` and then the customer's `enabled_events`.
|
||||
|
||||
**`app_start_failed` is customer-switchable but OFF by default** [DESIGN, operator ruling 2026-08-23]:
|
||||
@@ -180,7 +184,10 @@ rather than assumed (2026-08-30):
|
||||
- **not** in `operatorOnlyEvents` — the operator leg ignores customer preferences anyway, so the
|
||||
operator is always mailed; putting it here would only remove the customer's ability to opt in.
|
||||
- **not** in `DefaultEnabledEvents` — customer-switchable, default OFF, the same ruling as
|
||||
`app_start_failed`. The checkbox already exists at `settings_notifications.html:34`.
|
||||
`app_start_failed`. **Superseded by operator Q1 (2026-10-12, R-388 slice 1):** since controller v0.309.0
|
||||
`backup_integrity_failed` (with `crossdrive_failed`, `storage_reconnected`, `health_recovered`) IS in
|
||||
`DefaultEnabledEvents` — the default for a household with no saved list (`internal/settings/settings.go`, pinned by
|
||||
`TestR388_DefaultListAsGroups`; dated note 2026-10-24, review round 2). The checkbox already exists at `settings_notifications.html:34`.
|
||||
|
||||
**And a caveat that belongs in the alarm ladder rather than only in the backup document:** a
|
||||
`backup_integrity_ok` at the shipped depth means *the index, the pack inventory and the snapshot graph
|
||||
@@ -293,6 +300,7 @@ bands.
|
||||
| app down (`app_start_failed`) | **per APP** | `…:<stack_name>` | no digest exists for it — see below |
|
||||
| update outcome (`app_update_undone`, `app_update_held`) | **per APP**, both legs | `…:<stack_name>` | hub v0.120.0 — one mail per app per outcome; the household leg has its own register (`perAppCustomerCooldownEvents`) |
|
||||
| OOM storm (`app_oom_storm`) | **per APP**, operator | `…:<stack_name>` | hub v0.121.0 — no digest; the controller already sends it at most once per container run |
|
||||
| more per-APP types (added 2026-10-24 from `dispatcher.go`'s per-stack set) | **per APP** | `…:<stack_name>` | `app_hold_no_whole_copy`, `app_stopped_unhealthy` (both legs), `app_repaired_after_crash`, `app_unexplained_upload`, `app_unexplained_cpu` |
|
||||
| backup run (`backup_run_failures`) | per RUN | `…:<run_id>` | a digest already lists every failing app; one per run |
|
||||
| tiered backup (`whole_guest_backup_failed`, …) | per TIER | `…:<tier>` | the tiers fail independently and mean different things |
|
||||
| everything else, incl. `crossdrive_failed` and `backup_integrity_failed` | per TYPE, per hour | — | coarse **on purpose** |
|
||||
@@ -367,6 +375,14 @@ one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by
|
||||
|
||||
---
|
||||
|
||||
**More operator-only box alarms, listed 2026-10-24 (review round 2; each is in `notify/dispatcher.go`'s
|
||||
`operatorOnlyEvents`, and each has its home elsewhere):** `wg_tunnel_stale` / `wg_tunnel_ok` — the box's tunnel to ep0:
|
||||
a handshake older than 6 h or stamped in the future, edge-triggered (D30, R-1204; `api/host_tunnel.go`, `05` §15.4b);
|
||||
`host_swap_short` / `host_swap_ok` — the host's swap below 90 % of min(the guests' swap allowance, 8192 MB) (R-1192;
|
||||
`api/host_swap.go`); `bind_old_host_linked` — a household asked for a new bind link while its old host is still linked
|
||||
(R-1203; `05` §13); `golden_not_vouched` — a published golden the hub does not vouch (decision 199, R-242);
|
||||
`recovery_older_package` — a household's code opens, or may open, an older sealed package (decision 183, R-304 C).
|
||||
|
||||
## 6.4 A box that is not always on: the missed-backup deadline and the household's outage mail [DESIGN, hub v0.134.0, 2026-10-05]
|
||||
|
||||
Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *operator may reverse*).
|
||||
@@ -376,7 +392,8 @@ Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *op
|
||||
never judged (measured 2026-10-05: `1 skipped (down)`). Now a down box is judged on longer lines:
|
||||
`expected_dbdump_missed` after **48 h** without a `db_dump_completed`, `expected_backup_missed` after **72 h**
|
||||
without a whole-guest backup in any retained host report — never for a box first seen less than 48 h ago. A box that
|
||||
died last night still raises only its staleness alarm. A `disabled` box is still skipped (R-321). Pinned by
|
||||
died last night still raises only its staleness alarm. A `disabled` box is still skipped (R-321), and so is a box the household switched off on purpose — by its staleness
|
||||
state or by the host row's hold (R-998, `monitor/deadline.go`; corrected 2026-10-24). Pinned by
|
||||
`monitor/r872_down_box_test.go`.
|
||||
- **R-873 — "your server cannot be reached" reaches the HOUSEHOLD at most once per 7 days** (`node_stale`,
|
||||
`node_down`, `host_stale`, `host_down`, read from the persisted notification log, so a hub restart does not reset
|
||||
@@ -387,7 +404,7 @@ Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *op
|
||||
|
||||
---
|
||||
|
||||
## 6.5 One lost off-site copy on a pinned tier [DESIGN, R-435 — `09` §3 decision 191, hub main 2026-10-08, unreleased]
|
||||
## 6.5 One lost off-site copy on a pinned tier [DESIGN, R-435 — `09` §3 decision 191; released in hub v0.144.0 — corrected 2026-10-24]
|
||||
|
||||
On a **pinned** off-site tier (the hub holds a confirmed append-only key, decision 69) the only legitimate way the
|
||||
snapshot count can fall is a clean-up window the hub opened (decision 68). So the hub raises
|
||||
@@ -470,7 +487,7 @@ was ruled that way on its own merits and does not pre-judge the redesign.
|
||||
**Filed as a PRODUCT DECISION, not a defect** — see the register. It is the operator's call to take
|
||||
separately, and no part of it was implemented here.
|
||||
|
||||
### 8.1 The design — „only what you can act on" [DESIGN, R-388 — `09` §3 decision 200, 2026-10-12; NOT BUILT]
|
||||
### 8.1 The design — „only what you can act on" [DESIGN, R-388 — `09` §3 decision 200, 2026-10-12; slice 1 BUILT in controller v0.309.0 — three group switches, details under Részletek (`internal/settings/notif_groups.go`); corrected 2026-10-24]
|
||||
|
||||
*Operator ruling 2026-10-12 (S5): design now, build in slices later. Written by CC from the direction above and the
|
||||
live page (controller, `settings_notifications.html`: **18 event switches** plus the address, the cooldown and the
|
||||
@@ -532,7 +549,7 @@ list is shown as custom and saved back unchanged; the default list maps to group
|
||||
2. **The cooldown field („hány óránként legfeljebb"):** keep it on the page or move it under „Részletek"? Pick:
|
||||
**under Részletek** — it is an engineering choice, which is what this redesign removes.
|
||||
|
||||
## 9. The upload and CPU watch — „does the app explain its own traffic?" [DESIGN — proposed; built in controller v0.342.0 + hub v0.169.0, NOT released]
|
||||
## 9. The upload and CPU watch — „does the app explain its own traffic?" [DESIGN — built; released in controller v0.342.0 and hub v0.169.0 (deployed) on the operator's yes, 2026-10-24 — corrected 2026-10-24, review round 2]
|
||||
|
||||
**Why** (operator ruling 2026-10-23 19:22, `CONTEXT.md`): DooPlex's hacked Plex sent ~474 GB in a day with no viewer and
|
||||
used ~10 cores with no film being converted (`audits/plex2shell-felhom-impact-2026-10-23/`). Nothing on a Felhom box
|
||||
@@ -580,4 +597,4 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me
|
||||
a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood
|
||||
whatever the API says; not built. The box cannot tell LAN from internet bytes per app; a host-network container
|
||||
(Samba) has no counter; the push floor is a lower bound, the weekly test is close to the line.
|
||||
7. **Release:** not released; the operator's yes (R-1244).
|
||||
7. **Release:** released 2026-10-24 on the operator's yes (controller v0.342.0, hub v0.169.0; R-1244) — corrected 2026-10-24.
|
||||
|
||||
@@ -20,8 +20,10 @@ workspace-root `CLAUDE.md` § „Artifact taxonomy". **[FACT]** The spec shape i
|
||||
class). Validation is project Claude's unless CC is asked (`CLAUDE.md`, same section).
|
||||
|
||||
**A session with no spec** (a `/goal`, a night brief, „work the register") inherits the discipline a
|
||||
spec used to carry from `.claude/rules/unprompted-work.md` — **[FACT]** five byte-identical copies:
|
||||
the workspace root and each of the four repos (the file's own header says „change all five or none").
|
||||
spec used to carry from `.claude/rules/unprompted-work.md` — **[FACT]** five copies:
|
||||
the workspace root and each of the four repos (the file's own header says „change all five or none"). *(Measured
|
||||
2026-10-24: they are NOT byte-identical — the felhom.eu and felhom-controller copies carry one more paragraph, „A
|
||||
version number is pushed once" (added 2026-10-15); the review round 2 session aligned the other three, see its report.)*
|
||||
|
||||
## 2. Where an instruction lives
|
||||
|
||||
@@ -29,7 +31,7 @@ the workspace root and each of the four repos (the file's own header says „cha
|
||||
|---|---|---|
|
||||
| Cross-repo facts and standing rules (production host, gates, secrets, CHANGELOG/REPORT) | workspace-root `CLAUDE.md` | every session in the workspace |
|
||||
| One repo's stable orientation | `<repo>/CLAUDE.md` | when a file in that repo is touched |
|
||||
| Path-scoped rules (a subsystem's traps) | `<repo>/.claude/rules/*.md` | when a matching path is touched |
|
||||
| Path-scoped rules (a subsystem's traps) | `<repo>/.claude/rules/*.md` | when a matching path is touched — a file with `unconditional: true` (e.g. `live-probes.md`) or without `paths:` (the `unprompted-work.md` copies) loads every session (corrected 2026-10-24) |
|
||||
| A procedure with steps (build, test, diagnose, write for the operator) | a skill, `felhom.eu/skills/<name>/SKILL.md` | when its trigger matches, or by name |
|
||||
| Current state, decisions, open work | `CONTEXT.md`, `STATUS.md`, `documentation/backlog/OPEN-ITEMS.md` | read on demand — never copied into an instruction file |
|
||||
| Durable cross-session facts about the operator and the machines | the memory folder (`.claude-memory/`, index `MEMORY.md`) | the index each session; files on recall |
|
||||
@@ -38,8 +40,9 @@ Which rung a piece of material belongs on is the doc-authoring skill's §3 („W
|
||||
material sits") and §6 („One rule, one home"). This page does not repeat them.
|
||||
|
||||
**[FACT] Skills** are written in `felhom.eu/skills/`, installed by
|
||||
`python3 felhom.eu/scripts/install_skills.py` as symlinks into `~/.claude/skills/` (so a repo edit is
|
||||
live at once), and checked by `python3 felhom.eu/scripts/check_skills.py` (shape, the 150-line limit
|
||||
`python3 felhom.eu/scripts/install_skills.py` as symlinks into `~/.claude/skills/` (so an edit to an INSTALLED skill is
|
||||
live at once; a NEW skill folder is live only after `install_skills.py` runs again — measured 2026-10-24:
|
||||
`skills/felhom-web-design` has no link yet), and checked by `python3 felhom.eu/scripts/check_skills.py` (shape, the 150-line limit
|
||||
and its one grandfathered entry, R-394). The list is the folder; `skills/SOURCES.md` names where each
|
||||
skill's material came from.
|
||||
|
||||
|
||||
@@ -47,8 +47,8 @@ scripts are held with the review's working notes.
|
||||
| component | routes | no credential | listeners |
|
||||
|---|---|---|---|
|
||||
| controller | 231 (+8 in setup mode) | 21, of which 7 change state, each behind its own password, code or token | port 8080 on the guest's Docker networks (not published); mail relay 2525/2465/2526 |
|
||||
| hub | 121 (1 health, 53 API, 67 web — v0.152.0 added the two token routes; both route tables are now pinned to the source by tests, R-952) | appliance register (rate-limited, by design), re-enroll (only while the operator has armed recovery mode), login, health | one ingress, path `/`, geo filter (HU, IE, LAN) |
|
||||
| agent | 38 local-API routes (`POST /host/poweroff` added 2026-10-13, R-998) | 0 | local API on the island bridge; WireGuard; poke (tunnel only); felhom-sshd (tunnel only, measured) |
|
||||
| hub | 121 (1 health, 53 API, 67 web — v0.152.0 added the two token routes; both route tables are now pinned to the source by tests, R-952) — *the API table has 54 entries since `/box-config/` (v0.161.0) and `/preferences` (v0.165.0); counted from `hub/internal/api/routes.go`, 2026-10-24* | appliance register (rate-limited, by design), re-enroll (only while the operator has armed recovery mode), login, health, and the public self-bind page `/bind/<token>` (a capability link, added to this list 2026-10-24) | one ingress, path `/`, geo filter (HU, IE, LAN) |
|
||||
| agent | 41 local-API routes on `main` (counted 2026-10-24 in `felhom-agent/internal/localapi/server.go`; `POST /host/poweroff` added 2026-10-13, R-998; `POST /host/line-test/run` added 2026-10-24, R-1250, unreleased) | 0 | local API on the island bridge; WireGuard; poke (tunnel only); felhom-sshd (tunnel only, measured) |
|
||||
|
||||
**Outbound fetches that bring code or config onto a box:** the catalog (git, `main`), app images, the four infra
|
||||
images, the controller image (by tag), `controller.yaml` from the hub, the agent binary and config bundle (signed),
|
||||
@@ -58,11 +58,11 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
|
||||
|
||||
| boundary | verdict | evidence |
|
||||
|---|---|---|
|
||||
| end-user ↔ apps | holds, with gaps | traefik routing and header stripping hold; homepage has the Docker socket and no gate (R-981) |
|
||||
| end-user ↔ apps | holds, with gaps | traefik routing and header stripping hold; homepage has the Docker socket and no gate (R-981). **Re-check (2026-10-24, review round 2):** the socket is gone from the homepage template (catalog `templates/homepage/docker-compose.yml:21`) and the box's allow-list is empty, so a socket bind is refused (`felhom-controller/controller/internal/privpolicy/allowlist.json`) |
|
||||
| **added 2026-10-14 (R-1033, P1):** a visitor ↔ Cloudflare over plain http | holds on boxes with controller ≥ 0.312.0 | before: every public name answered `http://` (login pages too), and Sonarr/Radarr/Prowlarr redirected https visitors to an absolute `http://` login; a password could cross an untrusted network unencrypted (attacker: anyone on the visitor's network). Now traefik sends every `http://` request to https (301) before an app sees it, HSTS one day; boxes below 0.312.0 (Tester 2 at 0.292.0) keep the gap until the floor delivers it. `audits/https-only-2026-10-14/` |
|
||||
| **added 2026-10-15 (R-1045):** a stranger ↔ a fresh Plex | holds on controller ≥ 0.313.0; on older ones only while the household does not press „Kész" before claiming | an unclaimed Plex answers everything to a private address with no token, and through the tunnel every request comes from one. The setup gate (catalog `9a23ee5`) closes the public path; 0.313.0 opens it only when Plex itself says claimed. A status probe meant to cover older controllers OPENED an unclaimed Plex while it started (`audits/media-build-2026-10-15/G7/`) and was removed |
|
||||
| customer ↔ controller UI | partly | bcrypt and session hold; small gaps (R-983); family sessions survive a reset (R-976) |
|
||||
| controller ↔ agent | partly | every route bound to the caller's guest; a compromised controller can still flush restore points (R-973), void the recovery code (R-978), and — since agent v0.155.0 — switch the whole machine off once per power-on, holding the hub's silence alarms for up to 14 days (R-998; mitigations and residual risk: `14-host-power-off.md` §5) |
|
||||
| customer ↔ controller UI | partly | bcrypt and session hold; small gaps (R-983); family sessions survive a reset (R-976). **Re-check (2026-10-24, review round 2):** both fixed — a password change or a claim/reset ends every dashboard, family and gate session (controller v0.324.0, `internal/web/auth.go:352-366`) |
|
||||
| controller ↔ agent | partly | every route bound to the caller's guest; a compromised controller can still flush restore points (R-973), void the recovery code (R-978) *(re-check 2026-10-24: both narrowed — a cadence cap on `POST /backup` since agent v0.160.0, `internal/localapi/backup_cap.go`; a new escrow package is uploaded only at the claim since agent v0.160.0, `escrow_ceremony.go`)*, and — since agent v0.155.0 — switch the whole machine off once per power-on, holding the hub's silence alarms for up to 14 days (R-998; mitigations and residual risk: `14-host-power-off.md` §5) |
|
||||
| **added (R-1007):** controller → an app's own job queue | holds — nothing new | the `after_unclean_boot` commands run inside the app's container through the docker socket the controller already holds, and read the app's database with a read-only session: no key into the household's account, nothing a compromised controller could not already do (exec in any app, read its database). The household's lost-file names stay on the box (`app.yaml`, the app page); the operator event carries counts only, and a failing command's output is never logged (security review of controller `8d862e4`, fixed in `3b85a0c`) |
|
||||
| **added:** agent ↔ root on the host | holds by the sudoers | every root verb is one regex-anchored line or a root wrapper that re-checks its input (`03` §3); agent v0.156.1 adds one delete: `felhom-vzdump-partial-clean` removes only a DEAD half-written vzdump of the named guest on a `dir` storage (no vzdump on the host, nothing holding it open) — a compromised agent gains the deletion of a leftover that is not a backup |
|
||||
| agent ↔ hub | holds for destructive ops | signed jobs verified in the documented order; unsigned spec has no size bounds (R-984) |
|
||||
@@ -70,13 +70,13 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
|
||||
| controller ↔ hub | **did not hold until 2026-10-12** | the global key was public (R-950, rotated) and three routes needed no key (R-951, fixed); per-box keys are bound to their box (R-955) |
|
||||
| box ↔ PBS | holds as written | ciphertext only; the zero-knowledge wording is stronger than a live box allows (R-985) |
|
||||
| guest ↔ Proxmox host | holds | the guest has no Proxmox credentials (measured) |
|
||||
| hub ↔ Cloudflare API | holds for the zone check | tokens are stored unsealed in the hub database (R-975); R-137 open |
|
||||
| hub ↔ Cloudflare API | holds for the zone check | tokens are stored unsealed in the hub database (R-975); R-137 open. **Re-check (2026-10-24, review round 2):** the tokens are sealed (`enc:v1:`) since hub 0.168.0, deployed 2026-10-24 (`hub/internal/store/r975_config_token_seal.go`) |
|
||||
| **added:** app ↔ app (shared Docker daemon) | partly | separate containers and networks; any docker-socket app is root in the guest (R-981) |
|
||||
| **added:** app ↔ controller | partly | apps share a network with the controller; dashboard routes still need a session |
|
||||
| **added:** LAN ↔ box | partly | root password SSH on the host from the LAN (R-982) |
|
||||
| **added:** hub ↔ box (inbound) | **does not hold** | `01` §4 says the box exposes no control endpoint; a hub-key route can replace the box's settings (R-970) |
|
||||
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971) |
|
||||
| **added 2026-10-23 (R-1241):** a hacked app pod on DooPlex ↔ Felhom's secrets | **does not hold**; not used in the Plex2Shell window | two no-login doors: the Longhorn NFS export (the hub's unencrypted nightly copies with the unsealed tokens of R-975; by a published file-handle escape the whole sda1 disk, the operator signing keys included — not exercised) and the Longhorn API (backup a volume to any target). Plex2Shell, 2026-10-17 → 10-23: **NOT REACHED** — 0 new NFS connections in the window, no off-schedule backup or volume; the hub, the k8s API and the kubelet refuse a caller with no credential (measured). The export was narrowed to the pod network on 2026-10-23; the pod side stays open (`audits/plex2shell-felhom-impact-2026-10-23/`). **Felhom half, 2026-10-23 night:** the last plain tokens sealed in hub 0.168.0 (built, deploy pending — `audits/hub-token-seal-2026-10-23/`); the two plain Longhorn copies of the hub volume already in the folder are R-1243's decision. A household box's version of this attacker (a hacked app sending or mining) is what the upload watch (R-1244, not released) detects |
|
||||
| **added:** LAN ↔ box | partly | root password SSH on the host from the LAN (R-982). **Re-check (2026-10-24, review round 2):** the config bundle's key-only drop-in shipped in agent v0.163.0 (`felhom-agent/configs/felhom-os-apply:230`) |
|
||||
| **added:** hub ↔ box (inbound) | **does not hold** | `01` §4 says the box exposes no control endpoint; a hub-key route can replace the box's settings (R-970). **Re-check (2026-10-24, review round 2):** the config routes are removed and no route takes the hub key — pinned by `felhom-controller/controller/cmd/controller/r970_no_hub_key_door_test.go` (`01` §4 says the same) |
|
||||
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971). **Re-check (2026-10-24, review round 2):** the box checks every rendered compose against its allow-list before any write since controller v0.328.0 (`internal/privpolicy/privpolicy.go`) |
|
||||
| **added 2026-10-23 (R-1241):** a hacked app pod on DooPlex ↔ Felhom's secrets | **does not hold**; not used in the Plex2Shell window | two no-login doors: the Longhorn NFS export (the hub's unencrypted nightly copies with the unsealed tokens of R-975; by a published file-handle escape the whole sda1 disk, the operator signing keys included — not exercised) and the Longhorn API (backup a volume to any target). Plex2Shell, 2026-10-17 → 10-23: **NOT REACHED** — 0 new NFS connections in the window, no off-schedule backup or volume; the hub, the k8s API and the kubelet refuse a caller with no credential (measured). The export was narrowed to the pod network on 2026-10-23; the pod side stays open (`audits/plex2shell-felhom-impact-2026-10-23/`). **Felhom half, 2026-10-23 night:** the last plain tokens sealed in hub 0.168.0 (deployed 2026-10-24 — corrected 2026-10-24; `audits/hub-token-seal-2026-10-23/`); the two plain Longhorn copies of the hub volume already in the folder are R-1243's decision. A household box's version of this attacker (a hacked app sending or mining) is what the upload watch (R-1244, released 2026-10-24 in controller v0.342.0 + hub v0.169.0 — corrected 2026-10-24) detects |
|
||||
|
||||
## 6. What we accept (do not re-report)
|
||||
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
# `05-hub-architecture.md` — critical review (grounded against felhom-hub v0.6.3 source + Parts 01–04)
|
||||
|
||||
> **HISTORICAL (banner added 2026-10-24, review round 2).** A dated review of hub v0.6.3. Many of its facts have moved
|
||||
> since — for example the stale threshold is 45 min by config (code default 30 min), the operator cooldown is per event
|
||||
> type (`notify/dispatcher.go` `operatorCooldownFor`), and the hub→controller push was retired in v0.26.0. Read
|
||||
> `05-hub-architecture.md` and `08-alarm-ladder.md` for the current behaviour; this file is kept as the record of what
|
||||
> the review found then.
|
||||
|
||||
Method: every claim about the existing hub was checked against `felhom.eu/hub/` source; every
|
||||
cross-doc claim against Parts 01/03/04. Citations are `file:line`. Severity: **blocking** (wrong /
|
||||
breaks an assumption) · **should-fix** (real gap or contradiction, low blast) · **minor**.
|
||||
|
||||
@@ -147,7 +147,7 @@ Title: **"Who tells a box what to do"**
|
||||
## Bottom strip — "One night, hour by hour"
|
||||
|
||||
A horizontal timeline from **00:00 to 09:00**, two rows:
|
||||
- **DooPlex row:** 00:00 database dumps · **00:20 DooPlex copy → ep0** · **02:30 hub database → ep0** · 03:30 ep0 prunes households · 03:45 ep0 prunes `operator` · **05:00 ep0-copy pull → DooPlex** · 07:30 ep0-copy prune · 08:00 deleted-customer clean-up. Sundays: 04:30 restore tests.
|
||||
- **DooPlex row:** 00:00 database dumps · **00:20 DooPlex copy → ep0** · **02:30 hub database → ep0** (and again at 08:30; every 6 h since D34 — corrected 2026-10-24) · 03:30 ep0 prunes households · 03:45 ep0 prunes `operator` · **05:00 ep0-copy pull → DooPlex** · 07:30 ep0-copy prune · 08:00 deleted-customer clean-up. Sundays: 04:30 restore tests.
|
||||
- **Customer box row:** **02:30 Tier 1** · **03:30 Tier 2** · **04:15 Tier 3 off-site** → app updates → **04:30–08:30 whole-guest backup** (local daily, off-site weekly) → Debian fixes → (when approved) Proxmox, Docker, kernel restart. 05:00 the hub checks every box made its backup.
|
||||
- A small note: "Household mails about a kernel restart go out the day before, 09:00–20:00."
|
||||
|
||||
|
||||
@@ -191,7 +191,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
|
||||
<div style="position: absolute; left: 374px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
|
||||
<div style="position: absolute; left: 378px; top: 2040px; width: 220px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>00:20</b> DooPlex copy → ep0</div>
|
||||
<div style="position: absolute; left: 894px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
|
||||
<div style="position: absolute; left: 898px; top: 2000px; width: 230px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>02:30</b> hub database → ep0</div>
|
||||
<div style="position: absolute; left: 898px; top: 2000px; width: 230px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>02:30</b> hub DB → ep0, 6-hourly</div>
|
||||
<div style="position: absolute; left: 1134px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
|
||||
<div style="position: absolute; left: 1138px; top: 2040px; width: 236px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>03:30</b> ep0 prunes households</div>
|
||||
<div style="position: absolute; left: 1194px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
Sat Oct 24 12:46:20 UTC 2026
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.342.0
|
||||
repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 4 hours (healthy)
|
||||
filebrowser gtstef/filebrowser:1.5.6-stable Up 21 hours (healthy)
|
||||
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 12 hours (healthy)
|
||||
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 12 hours (healthy)
|
||||
immich-redis redis:7-alpine Up 12 hours (healthy)
|
||||
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 12 hours (healthy)
|
||||
paperless-postgres postgres:18-alpine Up 12 hours (healthy)
|
||||
paperless-redis redis:7-alpine Up 12 hours (healthy)
|
||||
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 12 hours (healthy)
|
||||
traefik traefik:v3.7.13 Up 21 hours
|
||||
2628d11da1accbbb53a7fe9459b1a200 -
|
||||
3
|
||||
/dev/loop0 32G 2.7G 27G 10% /
|
||||
/dev/nvme0n1 938G 227G 664G 26% /mnt/felhom-drives/scratch_hdd
|
||||
@@ -0,0 +1,96 @@
|
||||
# Document edits — review round 2 (2026-10-24)
|
||||
|
||||
Every edit below corrects a FACT the main session read at the cited `file:line` before editing (the reviewer's item id
|
||||
in brackets; the candidate files are in `reviewers/`). Each edited place carries „corrected 2026-10-24" in the text.
|
||||
Rules, fences, „never" lines and promises were NOT edited — those are rows for the operator.
|
||||
|
||||
## 05-hub-architecture.md (DOC-C1)
|
||||
- [S5] „the live hub is v0.11.0" → written against v0.11.0; the deployed version is read from `manifests/hub.yaml` (`:135` = 0.169.0).
|
||||
- [S4] „hub 0.162.0 — not yet deployed" ×2, §15.4b „0.166.0 NOT deployed" → deployed (hub CHANGELOG headers v0.164–v0.169).
|
||||
- [C3] §11 „ADD … the slim DR record onto hosts" → struck through, retired (R-105, §9).
|
||||
- [S3] `autoMintSelfBindLink` four events → plus the fifth path (R-719 / R-1203) — `hub/internal/web/selfbind.go:384-400`.
|
||||
- [S12] „39 event types" → 44 (`mail.event.*` keys counted in `hub/internal/i18n/locales/hu.json`).
|
||||
- [S1/C1] §16.3 nightly 02:00, catch-up 24 h, push 02:30 → four slots, catch-up 6 h, push :30 after each, refuse > 7 h — `hub/internal/dbsnap/dbsnap.go:31,36`; `hub/cmd/hub/main.go:964-968`; `scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10`.
|
||||
- [S2] §16.5 `scheduleDaily` list → the four slots; prune at its configured time — `hub/cmd/hub/main.go:648,859,968,972`.
|
||||
|
||||
## 08-alarm-ladder.md (DOC-C1)
|
||||
- [S11] info dropped before both legs → except the paired recoveries — `hub/internal/notify/dispatcher.go:137-167`.
|
||||
- [S7] operator leg „only … a 1-hour cooldown" → per-type cooldown (5 min / 6 h / 1 h), R-723 first-hour suppression — `dispatcher.go:597-640`.
|
||||
- [S9 — a dated note, the ruling line kept] backup_integrity_failed „default OFF" → superseded by operator Q1 2026-10-12: a default since controller v0.309.0 — `felhom-controller/controller/internal/settings/settings.go:713-721`.
|
||||
- [S8] the per-APP grain table → five more per-app types — `dispatcher.go:470-480`.
|
||||
- [S13] „a disabled box is still skipped" → also a box switched off on purpose (R-998) — `hub/internal/monitor/deadline.go:433-442`.
|
||||
- [S10] §8.1 „NOT BUILT" → slice 1 built in v0.309.0 — `felhom-controller/controller/internal/settings/notif_groups.go:1-16`.
|
||||
- [S6] §9 „NOT released" ×2 → released 2026-10-24 (controller CHANGELOG v0.342.0 header; `manifests/hub.yaml:135`).
|
||||
- [U6/U7, listed as facts — each already has a ruling] the operator-only box alarms wg_tunnel_stale, host_swap_short, bind_old_host_linked, golden_not_vouched, recovery_older_package — `dispatcher.go:777,782,862,866,870`; `hub/internal/api/host_swap.go:25-37`.
|
||||
- §6.5 header „hub main, unreleased" → released in hub v0.144.0 (hub CHANGELOG `:629` inside the v0.144.0 section).
|
||||
|
||||
## _hub-review.md (DOC-C1 S14)
|
||||
- A HISTORICAL banner (stale threshold 45 m by config — `manifests/hub.yaml:88`; cooldown per type; push retired). The body is kept as the record.
|
||||
|
||||
## felhom-system-poster.facts.md + .html (DOC-C1 C2) — text only (the poster rule §6: facts file and matching html text)
|
||||
- Timeline „02:30 hub database → ep0" → „… (and again at 08:30; every 6 h since D34)"; html „02:30 hub DB → ep0, 6-hourly".
|
||||
|
||||
## 03-host-agent.md (DOC-B)
|
||||
- [S1] cloudflared three states → four (`absent`, v0.161.0) — `felhom-agent/internal/hub/cloudflared.go:12-23`.
|
||||
- [S2] D4 „ready patch, NOT shipped" → shipped in v0.163.0 (`git tag --contains 1109092`) + installer-v1.36.0 (`manifests/webpage.yaml:347`).
|
||||
- [S3/S4] `a0c88e5`, `ae69ad4` „unreleased" → in v0.163.0 (`git tag --contains`).
|
||||
- [S5] old key `wg/private.key.prev` → first old key only; later `.prev.<unix>`, newest three; `wg/last-rekey` — `rekey.go:39,105-139`.
|
||||
- [S6] host_busy „Unreleased 56aabc7" → v0.161.0; plus the line-test holder — `internal/linetest/linetest.go:62`.
|
||||
- [S7] R-1010 „Unreleased 56aabc7" → v0.161.0.
|
||||
- [S8] local API „default :8443" → `listen_addr` required, no default — `internal/config/config.go:317-319`.
|
||||
- [S9] guest swap „once a minute after start" → once, one minute after start — `cmd/felhom-agent/main.go:940-955`.
|
||||
- NOT edited (RULE → row): S10 break-glass „via the Cloudflare Tunnel behind Access".
|
||||
|
||||
## 06-offsite-connectivity.md (DOC-B) — dated notes; the design-of-record text kept
|
||||
- [S11] „no slice is implemented yet" → note: S1–S4 shipped.
|
||||
- [S12] sudoers `install … *.mount` → note: `felhom-priv-apply wg` — `felhom-agent/configs/felhom-agent.sudoers:223-229`.
|
||||
- [S14] IdentityBundle `{TunnelToken, PBSToken}` → four fields — `felhom-agent/internal/escrow/identity.go:30-40`.
|
||||
- [S15] the only automatic re-registration → plus D30 — `rekey.go`.
|
||||
- [S16] „No new alarm channel" → `wg_tunnel_stale/ok` — `hub/internal/api/host_tunnel.go`.
|
||||
- [S17] key rotation „design when fleet size warrants" → note: the D30 rotation exists.
|
||||
- [S18] cloudflared „seam" → note: a guest container the agent only reads; withdrawing the plan is the operator's (row).
|
||||
- NOT edited (RULE → row): S19 „no inbound listener … box→PBS only".
|
||||
|
||||
## 12-agent-tooling.md (DOC-B)
|
||||
- [S20] „five byte-identical copies" → measured not identical; aligned (below).
|
||||
- [S21] rules load „when a matching path is touched" → `unconditional: true` / no `paths:` load always — `felhom.eu/.claude/rules/live-probes.md:2`.
|
||||
- [S22] skills „live at once … the list is the folder" → a NEW skill needs `install_skills.py` again (`felhom-web-design` has no link; not installed — no DooPlex change in this run).
|
||||
|
||||
## Instruction files (unprompted-work §5: a stale fact corrected / a proved fact added — named here)
|
||||
- `felhom-agent/.claude/rules/unprompted-work.md`, `app-catalog-felhom.eu/.claude/rules/unprompted-work.md`, workspace root
|
||||
`.claude/rules/unprompted-work.md` (unversioned): the missing item-6 paragraph „A version number is pushed once …"
|
||||
added (before: 95 lines, md5 3b071ad2; after: byte-identical to the felhom.eu and controller copies, md5 b1891bd4).
|
||||
Why: the file's own header says „change all five or none" and two copies had drifted; the paragraph states a fact
|
||||
(`felhom-controller/controller/build.sh:115-125` refuses an existing tag and an unreachable registry).
|
||||
- `felhom.eu/.claude/rules/live-probes.md:41`: before „no `/opt/felhom/stacks/<name>`"; after „no
|
||||
`/opt/docker/stacks/<name>/app.yaml` (the folder itself stays — it holds the catalog's template files)". Why: the path
|
||||
is `/opt/docker/stacks` (`felhom-controller/controller/internal/config/config.go:393`) and the folder stays after a
|
||||
clean removal (the catalog's `box_walk.remove` checks `app.yaml` since 2026-10-24).
|
||||
|
||||
## 01-topology-and-trust.md (DOC-A)
|
||||
- [S7] `docs/proxmox-platform.md` → `documentation/proxmox-platform.md` (the file exists; `docs/` does not).
|
||||
- [S6] „every /api/ route needs a dashboard session" → but the read-only `/api/health` — `felhom-controller/controller/cmd/controller/main.go:4236`.
|
||||
- [S1] claim mail link `templates.go:27` → `FormatClaimEmail` (`hub/internal/notify/templates.go:250-252`).
|
||||
- [S2] the retrieval code „single-use … TTL" → a dated note: not burned, no TTL; D19/R-1159 rule — `hub/internal/api/passphrase_scope.go:19-30`.
|
||||
- [S3] Cloudflare token reach check „hub main, unreleased" → released in hub v0.144.0.
|
||||
- [S4] dead pointer `07:296-299` → a dated note.
|
||||
- [S5] snapshot-before-deploy → [DESIGN — not built]: the agent serves `/snapshot`, `/rollback`; no controller caller (grep of `internal/agentapi`).
|
||||
- NOT edited (RULE/promise → rows): C3 „the hub never connects into a box / no inbound control endpoint / no standing operator backdoor" vs poke + felhom-sshd; C4 „the operator cannot open the off-site backup" vs R-985.
|
||||
|
||||
## 04-control-plane-authorization.md (DOC-A)
|
||||
- [S8] `docs/tests/…`, `docs/architecture/…` → `documentation/…`.
|
||||
- [S10] time window → the 2-minute skew on `issued_at` — `felhom-agent/internal/authz/verifier.go:19-22,148-155`.
|
||||
- [U2 → DESIGN — undocumented note + row] no cap on the window's length; opsign default 30 min — `felhom-agent/cmd/felhom-opsign/main.go:57`.
|
||||
- [S9] §6.1 „controller + hub main, unreleased" → released in controller v0.304.0 + hub v0.144.0 (both CHANGELOGs).
|
||||
|
||||
## 13-threat-model.md (DOC-A) — facts corrected; every verdict cell LEFT AS IT WAS and marked „re-check" (a verdict is a judgement)
|
||||
- [S11] „38 local-API routes" → 41 on main (`felhom-agent/internal/localapi/server.go`).
|
||||
- [S12] hub routes → 54 API entries; `/bind/<token>` added to the no-credential list.
|
||||
- [S13] homepage socket (R-981) → gone; the allow-list empty — catalog `templates/homepage/docker-compose.yml:21`; `privpolicy/allowlist.json:4`.
|
||||
- [S14] R-983 / R-976 → fixed (controller v0.324.0, `internal/web/auth.go:352-366`).
|
||||
- [S15] R-973 / R-978 → narrowed in agent v0.160.0 (`git tag --contains f17e197 / 8561dc8`).
|
||||
- [S16] tokens unsealed (R-975) → sealed since hub 0.168.0 (deployed).
|
||||
- [S17] root password SSH (R-982) → the drop-in shipped in agent v0.163.0.
|
||||
- [S18] R-970 „does not hold" → the routes are removed, pinned by `r970_no_hub_key_door_test.go`.
|
||||
- [S19] R-971 „design gap" → `privpolicy.Admit` before every write (controller v0.328.0).
|
||||
- [S20] „hub 0.168.0 deploy pending", „R-1244 not released" → deployed / released 2026-10-24.
|
||||
+80
@@ -0,0 +1,80 @@
|
||||
package backup
|
||||
|
||||
// REVIEW ROUND 2 (2026-10-24), HUNT-1 H1 — an EVIDENCE test kept in the audit, NOT committed to the repo.
|
||||
// A restore of a DRIVE-backed app is cut after its undo copy (kept on the app's drive) is recorded. At the next
|
||||
// start the drive is not bound yet: Recover finds the copy „gone", the gated start refuses (drive absent), the
|
||||
// marker and its Undo are KEPT. The drive then binds and the app runs (the boot recreate checks only „held") and the
|
||||
// household writes. At the NEXT controller start Recover finds the copy present and pours it back.
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type recordingPutBacker struct{ calls []string }
|
||||
|
||||
func (p *recordingPutBacker) PutBackInterruptedRestore(stack, dir string) error {
|
||||
p.calls = append(p.calls, stack+"@"+dir)
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestReview2H1_DriveBackedCutRestore_PutBackRunsAfterTheAppAlreadyRan(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
marker := filepath.Join(root, "appstop-state.json")
|
||||
drive := filepath.Join(root, "drive")
|
||||
undo := filepath.Join(drive, "ns", "backups", "restore-undo", "immich")
|
||||
if err := os.MkdirAll(undo, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
quiet := log.New(io.Discard, "", 0)
|
||||
|
||||
// the restore: Begin, the copy recorded, then the power cut (no End)
|
||||
g0 := NewAppStopGuard(marker, quiet)
|
||||
if err := g0.Begin("restore-1", ReasonUnitRestore, []string{"immich"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := g0.SetUndo("immich", undo); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// boot 1: the drive is not bound yet (its path is absent in the controller's view)
|
||||
if err := os.Rename(drive, drive+".unbound"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pb1 := &recordingPutBacker{}
|
||||
g1 := NewAppStopGuard(marker, quiet)
|
||||
g1.SetStarter(&gatingStarter{refuse: map[string]string{"immich": "the data drive is not present"}})
|
||||
g1.SetPutBacker(pb1)
|
||||
r1 := g1.Recover()
|
||||
t.Logf("boot 1: put-backs=%v refused=%v failed=%v restarted=%v marker kept=%v", pb1.calls, r1.Refused, r1.Failed, r1.Restarted, r2Exists(marker))
|
||||
if len(pb1.calls) != 0 || len(r1.Refused) != 1 || !r2Exists(marker) {
|
||||
t.Fatalf("boot 1 did not take the copy-gone, refused, marker-kept path")
|
||||
}
|
||||
|
||||
// the drive binds; the boot recreate starts the app (it checks only appHeld) and the household writes
|
||||
if err := os.Rename(drive+".unbound", drive); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
householdWrite := filepath.Join(drive, "ns", "new-photo-after-boot")
|
||||
if err := os.WriteFile(householdWrite, []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// boot 2 (any controller restart, e.g. the nightly self-update): the marker is still active
|
||||
pb2 := &recordingPutBacker{}
|
||||
g2 := NewAppStopGuard(marker, quiet)
|
||||
g2.SetStarter(&fakeStarter{})
|
||||
g2.SetPutBacker(pb2)
|
||||
r2 := g2.Recover()
|
||||
t.Logf("boot 2: put-backs=%v restarted=%v marker kept=%v", pb2.calls, r2.Restarted, r2Exists(marker))
|
||||
if len(pb2.calls) == 1 {
|
||||
t.Logf("DEFECT SHOWN: the copy taken BEFORE the restore is poured back AFTER the app already ran on the drive and the household wrote (%s)", householdWrite)
|
||||
} else {
|
||||
t.Fatalf("no put-back at boot 2 — the chain does not hold as described")
|
||||
}
|
||||
}
|
||||
|
||||
func r2Exists(p string) bool { _, err := os.Stat(p); return err == nil }
|
||||
@@ -0,0 +1,8 @@
|
||||
controller main 321df1f, lean clone, go test -run TestReview2H1 -v
|
||||
=== RUN TestReview2H1_DriveBackedCutRestore_PutBackRunsAfterTheAppAlreadyRan
|
||||
review2_h1_drive_crash_test.go:52: boot 1: put-backs=[] refused=[immich] failed=[] restarted=[] marker kept=true
|
||||
review2_h1_drive_crash_test.go:72: boot 2: put-backs=[immich@<tmp>/drive/ns/backups/restore-undo/immich] restarted=[immich] marker kept=false
|
||||
review2_h1_drive_crash_test.go:74: DEFECT SHOWN: the copy taken BEFORE the restore is poured back AFTER the app already ran on the drive and the household wrote (<tmp>/drive/ns/new-photo-after-boot)
|
||||
--- PASS: TestReview2H1_DriveBackedCutRestore_PutBackRunsAfterTheAppAlreadyRan (0.00s)
|
||||
PASS
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.006s
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
14:57:09 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH']
|
||||
14:57:10 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
14:57:40 [1] deployed, controller state=running, pinned={'audiobookshelf': 'ghcr.io/advplyr/audiobookshelf:2.37.1'}
|
||||
deploy: True
|
||||
-rw-r--r-- 1 root 1000 46 Oct 24 12:57 /mnt/felhom-drives/scratch_hdd/userdata/media/audiobooks/review2-h1-marker.txt
|
||||
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
night chain -> 202 {'data': {'legs': ['db-dump', 'tier2', 'update-leg']}, 'message': 'started', 'ok': True}
|
||||
2026/10/24 12:57:55 night_chain.go:108: [INFO] [night-chain] manual run started from 172.18.0.4:40216: [db-dump tier2 update-leg]
|
||||
2026/10/24 12:57:55 night_chain.go:84: [INFO] [night-chain] manual run of tonight's chain: dump → second drive → off-site → update leg
|
||||
2026/10/24 12:57:55 night_chain.go:77: [INFO] [night-chain] db-dump: started
|
||||
2026/10/24 12:58:50 night_chain.go:82: [INFO] [night-chain] db-dump: done in 55s
|
||||
2026/10/24 12:58:50 night_chain.go:77: [INFO] [night-chain] tier2: started
|
||||
2026/10/24 12:58:52 tier2.go:507: [INFO] [backup] Tier 2 copied audiobookshelf → /mnt/sizing-bench/backups/secondary/audiobookshelf (484.0 KB, 1 leg(s), 2s)
|
||||
2026/10/24 12:59:11 tier2.go:507: [INFO] [backup] Tier 2 copied immich → /mnt/felhom-drives/scratch_hdd/backups/secondary/immich (1.1 GB, 2 leg(s), 19s)
|
||||
2026/10/24 12:59:14 tier2.go:507: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sizing-bench/backups/secondary/paperless-ngx (128.2 MB, 1 leg(s), 3s)
|
||||
2026/10/24 12:59:15 tier2.go:507: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/scratch_hdd/backups/secondary/privatebin (8.4 KB, 0 leg(s), 1s)
|
||||
2026/10/24 12:59:15 tier2.go:569: [INFO] [backup] Tier 2 run complete: 4 app(s) processed (incl. volume-only — F6)
|
||||
2026/10/24 12:59:15 night_chain.go:82: [INFO] [night-chain] tier2: done in 24s
|
||||
2026/10/24 12:59:15 night_chain.go:90: [INFO] [night-chain] offsite: no off-site target on this box — skipped, as at night
|
||||
2026/10/24 12:59:15 night_chain.go:77: [INFO] [night-chain] update-leg: started
|
||||
2026/10/24 12:59:15 night_chain.go:82: [INFO] [night-chain] update-leg: done in 0s
|
||||
2026/10/24 12:59:15 night_chain.go:93: [INFO] [night-chain] finished in 1m20s
|
||||
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/.felhom-tier2-layout
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/.felhom.yml
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/app.yaml
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/docker-compose.yml
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/data-stamps.json
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/manifest.json
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps/audiobookshelf_audiobookshelf_config.tar
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps/audiobookshelf_audiobookshelf_metadata.tar
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media/audiobooks
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media/audiobooks/review2-h1-marker.txt
|
||||
|
||||
total 12
|
||||
drwxrwsr-x 2 root 1000 4096 Oct 24 12:57 .
|
||||
drwxr-xr-x 3 root root 4096 Oct 24 12:58 ..
|
||||
-rw-r--r-- 1 root 1000 46 Oct 24 12:57 review2-h1-marker.txt
|
||||
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
night chain -> 202 {'data': {'legs': ['db-dump', 'tier2', 'update-leg']}, 'message': 'started', 'ok': True}
|
||||
2026/10/24 12:59:38 night_chain.go:108: [INFO] [night-chain] manual run started from 172.18.0.4:40216: [db-dump tier2 update-leg]
|
||||
2026/10/24 12:59:38 night_chain.go:84: [INFO] [night-chain] manual run of tonight's chain: dump → second drive → off-site → update leg
|
||||
2026/10/24 12:59:38 night_chain.go:77: [INFO] [night-chain] db-dump: started
|
||||
2026/10/24 13:00:40 night_chain.go:82: [INFO] [night-chain] db-dump: done in 1m2s
|
||||
2026/10/24 13:00:40 night_chain.go:77: [INFO] [night-chain] tier2: started
|
||||
2026/10/24 13:00:41 tier2.go:507: [INFO] [backup] Tier 2 copied audiobookshelf → /mnt/sizing-bench/backups/secondary/audiobookshelf (487.0 KB, 1 leg(s), 1s)
|
||||
2026/10/24 13:00:56 tier2.go:507: [INFO] [backup] Tier 2 copied immich → /mnt/felhom-drives/scratch_hdd/backups/secondary/immich (1.1 GB, 2 leg(s), 15s)
|
||||
2026/10/24 13:00:58 tier2.go:507: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sizing-bench/backups/secondary/paperless-ngx (128.2 MB, 1 leg(s), 2s)
|
||||
2026/10/24 13:00:59 tier2.go:507: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/scratch_hdd/backups/secondary/privatebin (8.4 KB, 0 leg(s), 1s)
|
||||
2026/10/24 13:00:59 tier2.go:569: [INFO] [backup] Tier 2 run complete: 4 app(s) processed (incl. volume-only — F6)
|
||||
2026/10/24 13:00:59 night_chain.go:82: [INFO] [night-chain] tier2: done in 19s
|
||||
2026/10/24 13:00:59 night_chain.go:90: [INFO] [night-chain] offsite: no off-site target on this box — skipped, as at night
|
||||
2026/10/24 13:00:59 night_chain.go:77: [INFO] [night-chain] update-leg: started
|
||||
2026/10/24 13:00:59 night_chain.go:82: [INFO] [night-chain] update-leg: done in 0s
|
||||
2026/10/24 13:00:59 night_chain.go:93: [INFO] [night-chain] finished in 1m21s
|
||||
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/.felhom-tier2-layout
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/.felhom.yml
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/app.yaml
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/docker-compose.yml
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/data-stamps.json
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/manifest.json
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps/audiobookshelf_audiobookshelf_config.tar
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps/audiobookshelf_audiobookshelf_metadata.tar
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media/audiobooks
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media/audiobooks/review2-h1-marker.txt
|
||||
|
||||
total 12
|
||||
drwxrwsr-x 2 root 1000 4096 Oct 24 12:57 .
|
||||
drwxr-xr-x 3 root root 4096 Oct 24 12:58 ..
|
||||
-rw-r--r-- 1 root 1000 46 Oct 24 12:57 review2-h1-marker.txt
|
||||
|
||||
+49
@@ -0,0 +1,49 @@
|
||||
28- # If Viktor rules excluded, flip THIS LINE ONLY before committing.
|
||||
29: - path: media/review2-typo
|
||||
30- class: optional # REVIEW-2 H1 PROOF: a path no compose bind has
|
||||
|
||||
night chain -> 202 {'data': {'legs': ['db-dump', 'tier2', 'update-leg']}, 'message': 'started', 'ok': True}
|
||||
2026/10/24 13:01:28 night_chain.go:108: [INFO] [night-chain] manual run started from 172.18.0.4:40216: [db-dump tier2 update-leg]
|
||||
2026/10/24 13:01:28 night_chain.go:84: [INFO] [night-chain] manual run of tonight's chain: dump → second drive → off-site → update leg
|
||||
2026/10/24 13:01:28 night_chain.go:77: [INFO] [night-chain] db-dump: started
|
||||
2026/10/24 13:01:30 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
|
||||
2026/10/24 13:01:31 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
|
||||
2026/10/24 13:01:31 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
|
||||
2026/10/24 13:01:32 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
|
||||
2026/10/24 13:01:32 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
|
||||
2026/10/24 13:01:32 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
|
||||
2026/10/24 13:01:32 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
|
||||
2026/10/24 13:01:59 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
|
||||
2026/10/24 13:02:24 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
|
||||
2026/10/24 13:02:24 night_chain.go:82: [INFO] [night-chain] db-dump: done in 57s
|
||||
2026/10/24 13:02:24 night_chain.go:77: [INFO] [night-chain] tier2: started
|
||||
2026/10/24 13:02:24 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
|
||||
2026/10/24 13:02:25 tier2.go:329: [INFO] [backup] Tier 2 reconcile: removed stale dest dir userdata/media
|
||||
2026/10/24 13:02:25 tier2.go:507: [INFO] [backup] Tier 2 copied audiobookshelf → /mnt/sizing-bench/backups/secondary/audiobookshelf (489.9 KB, 0 leg(s), 1s)
|
||||
2026/10/24 13:02:46 tier2.go:507: [INFO] [backup] Tier 2 copied immich → /mnt/felhom-drives/scratch_hdd/backups/secondary/immich (1.1 GB, 2 leg(s), 20s)
|
||||
2026/10/24 13:02:49 tier2.go:507: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sizing-bench/backups/secondary/paperless-ngx (128.2 MB, 1 leg(s), 3s)
|
||||
2026/10/24 13:02:50 tier2.go:507: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/scratch_hdd/backups/secondary/privatebin (8.4 KB, 0 leg(s), 0s)
|
||||
2026/10/24 13:02:50 tier2.go:569: [INFO] [backup] Tier 2 run complete: 4 app(s) processed (incl. volume-only — F6)
|
||||
2026/10/24 13:02:50 night_chain.go:82: [INFO] [night-chain] tier2: done in 25s
|
||||
2026/10/24 13:02:50 night_chain.go:90: [INFO] [night-chain] offsite: no off-site target on this box — skipped, as at night
|
||||
2026/10/24 13:02:50 night_chain.go:77: [INFO] [night-chain] update-leg: started
|
||||
2026/10/24 13:02:50 [ERROR] [stacks] .felhom.yml backup block rejected in /opt/docker/stacks/audiobookshelf: userdata["media/review2-typo"]: matches no compose bind (stale or typoed path)
|
||||
2026/10/24 13:02:50 night_chain.go:82: [INFO] [night-chain] update-leg: done in 0s
|
||||
2026/10/24 13:02:50 night_chain.go:93: [INFO] [night-chain] finished in 1m22s
|
||||
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/.felhom-tier2-layout
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/.felhom.yml
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/app.yaml
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/compose/docker-compose.yml
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/data-stamps.json
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/manifest.json
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps/audiobookshelf_audiobookshelf_config.tar
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/recovery-unit/volume-dumps/audiobookshelf_audiobookshelf_metadata.tar
|
||||
/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata
|
||||
|
||||
ls: cannot access '/mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media/audiobooks/': No such file or directory
|
||||
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
0
|
||||
restored
|
||||
|
||||
stop ('200', {'ok': True, 'message': 'Stack audiobookshelf stop completed'})
|
||||
remove ('200', {'ok': True, 'data': {'removed': 'audiobookshelf', 'volumes_removed': ['audiobookshelf_audiobookshelf_config', 'audiobookshelf_audiobookshelf_metadata'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'userdata_kept': ['/mnt/felhom-drives/scratch_hdd/userdata/media/audiobooks (8.0K)', '/mnt/felhom-drives/scratch_hdd/userdata/media/podcasts (4.0K)'], 'backup_paths_removed': ['/mnt/felhom-drives/scratch_hdd/backups/primary/audiobookshelf (520K)', '/mnt/sizing-bench/backups/secondary/audiobookshelf (489.9 KB)'], 'verified': True}, 'message': 'Stack audiobookshelf removed'})
|
||||
total 8
|
||||
drwxrwsr-x 2 root 1000 4096 Oct 24 13:04 .
|
||||
drwxrwsr-x 16 root 1000 4096 Oct 13 19:26 ..
|
||||
paperless-ngx
|
||||
docker-compose.yml
|
||||
0
|
||||
0
|
||||
ls: cannot access '/mnt/felhom-drives/scratch_hdd/appdata/audiobookshelf': No such file or directory
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# DOC-A — 01 topology & trust, 04 control-plane authorization, 13 threat model (reviewer candidates; verified by the main session)
|
||||
|
||||
Reviewer: read-only subagent, 2026-10-24. Counts: claims 62 · true 33 · stale 20 · untestable 6 · undocumented 4 · contradictions 4.
|
||||
|
||||
## STALE (all FACT)
|
||||
- S1 01:135 claim mail link `templates.go:27` → :252
|
||||
- S2 01:220-223 passphrase „single-use (burned on the successful config fetch) plus a short TTL" → not burned; keys until a host exists and a controller ≥ 0.334.0 reported, then 409 (R-1159, D19) — hub/internal/api/passphrase_scope.go:19-29,46-66
|
||||
- S3 01:256 Cloudflare reach check „hub main, unreleased" → released v0.144.0
|
||||
- S4 01:315-316 pointer `07:296-299` „40 of 53 … all-hot" → dead pointer
|
||||
- S5 01:345-346 snapshot-before-deploy / rollback → agent serves /snapshot,/rollback; no controller caller — [DESIGN], not built
|
||||
- S6 01:108 „every /api/ route needs a dashboard session" → /api/health is open (read-only) — cmd/controller/main.go:4236
|
||||
- S7 01:19 `docs/proxmox-platform.md` → `documentation/proxmox-platform.md`
|
||||
- S8 04:18-20 `docs/tests/…`, `docs/architecture/…` → `documentation/…`
|
||||
- S9 04:169 operator actions „unreleased" → hub v0.144.0, controller v0.304.0
|
||||
- S10 04:79 „now ∈ [issued_at, expires_at]" → 2-minute skew allowance — felhom-agent/internal/authz/verifier.go:21,148-155; configs/felhom-os-apply:658
|
||||
- S11 13:51 „38 local-API routes" → 41 — felhom-agent/internal/localapi/server.go:562-634
|
||||
- S12 13:50 hub route counts / no-credential list → 54 API routes; the public `/bind/<token>` page missing from the list
|
||||
- S13 13:61 homepage socket (R-981) → gone; allow-list empty
|
||||
- S14 13:64 R-983 gaps, R-976 family sessions survive a reset → both fixed (auth.go:352-366)
|
||||
- S15 13:65 R-973 flush, R-978 void the code → cadence cap; package uploaded only at the claim
|
||||
- S16 13:73 tokens unsealed (R-975) → sealed `enc:v1:` (hub 0.168.0, deployed)
|
||||
- S17 13:76 root password SSH from the LAN (R-982) → the bundle's key-only drop-in
|
||||
- S18 13:77 hub-key route can replace settings (R-970) → removed; pinned by cmd/controller/r970_no_hub_key_door_test.go
|
||||
- S19 13:78 no box check on a catalog push (R-971) → privpolicy.Admit
|
||||
- S20 13:79 hub 0.168.0 „deploy pending", R-1244 „not released" → deployed / shipped
|
||||
|
||||
## UNDOCUMENTED
|
||||
- U1 FileBrowser SSO trusts a proxy header; the header NAME is the per-box secret; any app container reaches filebrowser:80 — controller internal/infra/filebrowser_sso.go:15-22; internal/web/files_sso.go:20-44 — 01 §5, 13 §5
|
||||
- U2 a signed op's window has no maximum length (opsign default TTL 30 m) — felhom-agent/internal/authz/verifier.go:148-155 — 04 §2
|
||||
- U3 „poke": the hub reaches into a box (SSH to ep0 → unauthenticated UDP to the box's WG /32:51822) — hub/internal/poke/client.go:1-16; agent poke.go:35 — 01 §4/§5
|
||||
- U4 the guest can start a ~55 s line test on request (R-1250) — agent internal/localapi/linetest.go:67-73
|
||||
|
||||
## CONTRADICTIONS
|
||||
- C1 01:105-109 R-970 closed vs 13:77 „does not hold" — code: closed
|
||||
- C2 04:195 sessions end on a reset vs 13:64 „family sessions survive" — code: end
|
||||
- C3 01:26-27,102,244-245,353 „the hub never connects into a box / no inbound control endpoint / no standing operator backdoor" vs 06:279-283 operator WG peer reaches felhom-sshd; 13:51 poke + felhom-sshd — RULE (a „never" line, a product value) → OPERATOR
|
||||
- C4 01:328-329,352 the operator „can't open / cannot read" the off-site backup vs 13:71 + R-985 (a live box lets the operator reach the key) — RULE (a customer promise) → OPERATOR
|
||||
+49
@@ -0,0 +1,49 @@
|
||||
# DOC-B — 03 host agent, 06 off-site connectivity, 12 agent tooling (reviewer candidates; verified by the main session before acting)
|
||||
|
||||
Reviewer: read-only subagent, 2026-10-24. Counts: claims 79 · true 45 · stale 22 · untestable 6 · undocumented 8 · contradictions 6.
|
||||
`A/` = felhom-agent/.
|
||||
|
||||
## STALE
|
||||
- S1 03:50,226 cloudflared three states → four (`absent`, R-1070, v0.161.0) — A/internal/hub/cloudflared.go:12-23 — FACT
|
||||
- S2 03:132 D4 drop-in „ready patch, NOT shipped" → shipped (agent v0.163.0 `1109092`, installer-v1.36.0) — A/configs/felhom-os-apply:189,230; manifests/webpage.yaml:347 — FACT
|
||||
- S3 03:134 `a0c88e5` „unreleased" → in v0.163.0 — FACT
|
||||
- S4 03:136 `ae69ad4` „unreleased" → in v0.163.0 — FACT
|
||||
- S5 03:137-138 old key `wg/private.key.prev` → only the first; later `private.key.prev.<unix>`, newest 3; bound in `wg/last-rekey` — A/internal/wgtunnel/rekey.go:107,137-139,161-173 — FACT
|
||||
- S6 03:252-254 host_busy „Unreleased 56aabc7" → in v0.161.0; holder `line-test` too — A/internal/linetest/linetest.go:62 — FACT
|
||||
- S7 03:423-424 R-1010 „Unreleased 56aabc7" → in v0.161.0 — FACT
|
||||
- S8 03:316-317 local API „default :8443" → no default, `listen_addr` required; island 169.254.253.1:8443 — A/internal/config/config.go:251,260-269,319 — FACT
|
||||
- S9 03:680 „once a minute after agent start" → ONCE, one minute after start — A/cmd/felhom-agent/main.go:940-955 — FACT
|
||||
- S10 03:236-237 break-glass „via the Cloudflare Tunnel behind Access" → felhom-sshd over wg-felhom — A/internal/felhomsshd/claim.go:7 — RULE
|
||||
- S11 06:19-21 „no slice is implemented yet" → S1–S4 shipped — FACT
|
||||
- S12 06:69,142-143 sudoers `install … *.mount` → R-861 removed; `felhom-priv-apply wg` — A/configs/felhom-agent.sudoers:15-20,227-233 — FACT
|
||||
- S13 06:71 report.go line citations moved (:628, :659-666) — FACT
|
||||
- S14 06:172 IdentityBundle `{TunnelToken, PBSToken}` → 4 fields incl. ResticRepoPassword — A/internal/escrow/identity.go:30-40 — FACT
|
||||
- S15 06:201-202 „only automatic re-registration is the pubkey-mismatch path" → D30 too — rekey.go:88-102 — FACT
|
||||
- S16 06:286-288 „No new alarm channel" → hub wg_tunnel_stale/ok — hub/internal/api/host_tunnel.go:19-47 — FACT
|
||||
- S17 06:357-359 „key rotation … design when fleet size warrants" → D30 rotation exists — FACT
|
||||
- S18 06:381-383 cloudflared.go „seam" → reads the guest container; no seam — FACT
|
||||
- S19 06:86,325-329 „No inbound listener … box→PBS only" → poke UDP 51822 on the WG /32, felhom-sshd, operator /32, iperf3 :5201 — A/internal/poke/poke.go:35,133 — RULE
|
||||
- S20 12:23-24 „five byte-identical copies" → two copies carry an extra paragraph (b6638ac4 / 48d8674) — FACT (aligning them is rule text → operator)
|
||||
- S21 12:32 rules load „when a matching path is touched" → some always load (`unconditional: true`, no `paths:`) — FACT
|
||||
- S22 12:40-43 skills „live at once … the list is the folder" → a NEW skill needs install_skills.py again (felhom-web-design not linked) — FACT
|
||||
|
||||
## UNDOCUMENTED
|
||||
- U1 03 §6 lists ~17 of 41 local-API routes — A/internal/localapi/server.go:562-634
|
||||
- U2 the controller can start a HOST action: POST /host/line-test/run — server.go:579-580; linetest.go:346
|
||||
- U3 local API over the R-50 island bridge — config.go:260-269
|
||||
- U4 WG block carries oob_peer_ip, oob_operator_ssh_key — A/internal/hub/report.go:752,758
|
||||
- U5 D30: re-key only inside Watchdog; `futureSince` in memory (a restart restarts the 15 min) — manager.go:621-630; rekey.go:45-80
|
||||
- U6 poke binds UDP 51822 on the WG /32 only — poke.go:32-35,133
|
||||
- U7 the GRANDFATHERED skill has no size cap (240 lines vs R-394's 179) — scripts/check_skills.py:45-47,98-101
|
||||
- U8 line-test state `<oob state dir>/line-test.json`, switch `line_test.disable` — config.go:37; linetest.go:89
|
||||
|
||||
## CONTRADICTIONS
|
||||
- C1 03:67,815 agent manages the tunnel service vs 03:73 cloudflared is a guest container — code: 03:73
|
||||
- C2 06:381-383 cloudflared will adopt agent-managed pattern vs 03:73 — code: 03:73
|
||||
- C3 06:325-333 no inbound listener vs 06:273-285 felhom-sshd over the tunnel — code: 06:273-285
|
||||
- C4 03:234-238 break-glass over Cloudflare vs 06:280-285 over WG — code: 06
|
||||
- C5 13:51 „38 local-API routes" vs 03 §6 ~17 — code: 41 on main
|
||||
- C6 05:459 „0.166.0 NOT deployed" vs hub CHANGELOG v0.167.0 deployed
|
||||
|
||||
## UNTESTABLE
|
||||
ep0:5201 never on the internet (nft lives on ep0); PBS sync/verify schedule (DooPlex job config); WG enabled default vs „production".
|
||||
@@ -0,0 +1,38 @@
|
||||
# DOC-C1 — 05 hub, 08 alarm ladder, _hub-review (reviewer candidates, read-only; the main session verified before acting)
|
||||
|
||||
Reviewer: read-only subagent, 2026-10-24. Counts: claims 52 · true 31 · stale 14 · untestable 5 · undocumented 7 · contradictions 4.
|
||||
Prefixes: `H/` = felhom.eu/hub/, `C/` = felhom-controller/controller/, `F/` = felhom.eu/ (felhom.eu main 44d19042, controller main 321df1f).
|
||||
|
||||
## STALE
|
||||
- S1 05:585,590-592 „Every night at 02:00 … older than 24 h … picks up at 02:30" → 02/08/14/20, catch-up 6 h, push :30 after each, refuse > 7 h (D34) — H/internal/dbsnap/dbsnap.go:32,36; H/cmd/hub/main.go:964-968; F/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10 — FACT
|
||||
- S2 05:614 „scheduleDaily: db-snapshot 02:00, prune, …" → four db-snapshot slots; prune at the configured schedule (04:30) — H/cmd/hub/main.go:648,859,967-972 — FACT
|
||||
- S3 05:353 „autoMintSelfBindLink runs on four events" → a fifth: the household's „new link" on a used/expired link (R-719, ≤1/h/customer); old host still linked → „oldbox" page + operator told — H/internal/web/selfbind.go:382,395-436 — FACT
|
||||
- S4 05:94,142,459 version literals „hub 0.162.0 … not yet deployed", „0.166.0 built NOT deployed" → the manifest deploys 0.169.0 — F/manifests/hub.yaml:135 — FACT
|
||||
- S5 05:22-26 „the live hub is v0.11.0" → version literal, false — F/manifests/hub.yaml:135 — FACT
|
||||
- S6 08:535,583 „controller v0.342.0 + hub v0.169.0, NOT released" → hub half deployed (hub CHANGELOG:1) — FACT
|
||||
- S7 08:158 „processOperator consults only operatorOn, the address and a 1-hour cooldown" → per-type cooldown (5 min liveness, 6 h storage fill, else 1 h), R-723 first-hour skip, mail hold — H/internal/notify/dispatcher.go:606-614,631-638,670 — FACT
|
||||
- S8 08:291-298 per-app grain lists three types → also app_hold_no_whole_copy, app_stopped_unhealthy, app_repaired_after_crash, app_unexplained_upload/cpu — H/internal/notify/dispatcher.go:473-496,502-507 — FACT
|
||||
- S9 08:182-183 backup_integrity_failed „not in DefaultEnabledEvents … default OFF" → a default since controller v0.309.0 („operator Q1 2026-10-12") — C/internal/settings/settings.go:713-721 — RULE
|
||||
- S10 08:473 §8.1 „NOT BUILT" → slice 1 built (three group switches, v0.309.0) — C/internal/settings/notif_groups.go:4-16 — FACT
|
||||
- S11 08:137-138 „info is dropped … before both legs" → paired info recoveries bypass the gate — H/internal/notify/dispatcher.go:137-167,225-228 — FACT
|
||||
- S12 05:391 „39 event types" → 44 `mail.event.*` keys in hu.json — FACT
|
||||
- S13 08:374-380 only `disabled` skipped → also a box the household switched off (R-998) — H/internal/monitor/deadline.go:433-442 — FACT
|
||||
- S14 _hub-review.md (a dated v0.6.3 review) thresholds/cooldown/push → historical; add a banner — FACT
|
||||
|
||||
## UNDOCUMENTED
|
||||
- U1 bind_old_host_linked (R-1203) — selfbind.go:395-421; dispatcher.go:870 — 05 §13, 08 §6.3
|
||||
- U2 R-1230 the off-site delivery checker skips a box awaiting its recovery key — H/internal/monitor/offsite_delivery.go:105-110,149 — 07 / 05 §8
|
||||
- U3 offsite_delivery_stuck + maybeHeal — offsite_delivery.go:113-125; dispatcher.go:847 — 07 §6
|
||||
- U4 R-723 backup_tier_skipped suppressed in the first hour — dispatcher.go:627-638 — 08 §6.2
|
||||
- U5 System page „Upload line" cell — H/internal/web/system.go:174-197 — 11 §5.7
|
||||
- U6 wg_tunnel_stale, host_swap_short missing from 08 §6.3 — H/internal/api/host_tunnel.go:23,37-47; host_swap.go:31-43
|
||||
- U7 golden_not_vouched, recovery_older_package in no document — dispatcher.go:776-782
|
||||
|
||||
## CONTRADICTIONS
|
||||
- C1 05:569 „6-hourly VACUUM INTO" vs 05:585 „every night" — code: 6-hourly
|
||||
- C2 poster.facts.md:125 (four push times) vs :150 („02:30" only) — code: four
|
||||
- C3 05:291-297 slim DR record RETIRED vs 05:329-330 §11 „ADD … the slim DR record" — code: retired
|
||||
- C4 08:182 „default OFF" vs 08:510 „the 17 defaults" (includes it) — code: default ON
|
||||
|
||||
## UNTESTABLE (live only)
|
||||
05:571 token bytes freed; 05:569 Longhorn speed; 08:553/576 line test numbers; 05:268 R-1159 proof; 08:216 report gap.
|
||||
@@ -0,0 +1,42 @@
|
||||
# DOC-C2 — 09 update architecture (reviewer candidates; verified by the main session before acting)
|
||||
|
||||
Reviewer: read-only subagent, 2026-10-24. Counts: claims 63 · true 38 · stale 19 · untestable 6 · undocumented 6 · contradictions 7.
|
||||
Paths under felhom-controller/controller/ unless named.
|
||||
|
||||
## STALE (all FACT)
|
||||
- S1 09:68-75 `Manager.UpdateStack` „two compose commands … No safety copy. No rollback." → gone; StartGuardedUpdate — internal/api/router.go:794; internal/stacks/update.go:518-540,752-991
|
||||
- S2 09:119-124 RestartStack comment → manager.go:1454; up -d --force-recreate, start-time check (R-1026) — manager.go:1470-1487
|
||||
- S3 09:82-86,97-98,1958,2308 line citations moved (sync.go:319,362,436,700; config.go:399; appstop_marker.go:377; intermediary.go:228)
|
||||
- S4 09:1979-1992 „four writers … UpdateStack" → advancePinTo; plus pinBack and the undo's restoreDefinition — update.go:921-941,1124; undo.go:531
|
||||
- S5 09:2631-2632 recordInstalledImages callers incl. UpdateStack → StartStack, RestartStack, deploy, guarded update end, undo end — manager.go:1368,1495; deploy.go:683; update.go:1007; undo.go:555
|
||||
- S6 09:1994 §5.4 render table → missing row: unreadable template images → compose left alone; R-971 admission before any write — internal/sync/sync.go:622-627; cmd/controller/main.go:421
|
||||
- S7 09:2050,2273 slice 6 „OPEN — R-450" → built; R-450 closed 2026-10-05 — internal/stacks/unattended.go:16-37
|
||||
- S8 09:2064-2075 §6.1 phase table → no `converting` row; bind-data order 2 → 3 → 1 — update.go:969-980; internal/backup/update_guard.go:130-155
|
||||
- S9 09:2374 part 1 „DB up alone … empty-then-load … A volume tar" → what shipped is the folder copy (cp -a + marker) — undo.go:30-34,519-530
|
||||
- S10 09:2378 „a step has no .felhom.yml of its own (R-664)" → steps/<key>.felhom.yml read when present — ladder.go:129-132,209-212
|
||||
- S11 09:2379 „The box half … is not built" → built — digest.go:15-23,94-104,141-155; updateorder.go:76-112
|
||||
- S12 09:2132-2133 R-646 residual → BackfillAppliedMeta at start — undo.go:646-676; cmd/controller/main.go:498
|
||||
- S13 09:2119,2729-2733 „multi-major jump … slice 6" / §8.6 „Tier-2 copy … no automatic rollback" → any tier, undo, one step per press
|
||||
- S14 09:2706-2708 digest comparison „R-446 §3b Q6" → built — updateorder.go:76-112
|
||||
- S15 09:2736-2739 engine-major „until Slice 4" → MariaDB lifted; PostgreSQL with a two-venue proof + engine_conversion
|
||||
- S16 09:269-273 `stacks.update_window` „no Go code reads it" → the field is removed (inside ruling 11: a dated note, not an edit)
|
||||
- S17 09:427 `unattended.go:433–435` → :449-451
|
||||
- S18 09:2344-2345 `config.go` L280-281, default L422 → L278-279, L419
|
||||
- S19 09:2462,2477 refusal `current` → `already_current` — update.go:432
|
||||
|
||||
## UNDOCUMENTED
|
||||
- U1 build.sh refuses --push from a dirty tree / HEAD ≠ origin/main — controller/build.sh:144-167 — 09 §0
|
||||
- U2 the leg retries a transient refusal once in the same leg after RetryWait — unattended.go:65-66,294-306
|
||||
- U3 the leg skips `stopped_by_household` (R-721) — unattended.go:54-55,416-418
|
||||
- U4 a manual chain run's deadline = start + 195 min — unattended.go:234-239
|
||||
- U5 preflight refusals `already_current`, `engine_no_test` — update.go:432,454
|
||||
- U6 the syncer's R-971 allow-list before any compose write — cmd/controller/main.go:421
|
||||
|
||||
## CONTRADICTIONS
|
||||
- C1 09:2050 slice 6 OPEN vs capability map B11 PROVEN-LIVE — code: built
|
||||
- C2 09:68,1984,2632 + 02:838,861 UpdateStack vs controller REUSE.md:154 „DELETED in v0.237.0" — code: deleted
|
||||
- C3 controller REUSE.md:164 lists UpdateStack as a caller vs REUSE.md:154
|
||||
- C4 RULING decision 13 „CompareImageRefs moves to the catalog gate" vs 09:2377 „did NOT move" — code: controller only (updateorder.go:134) — OPERATOR
|
||||
- C5 RULING decision 14 „never jumps" vs 09:2378 „an installed version matching no entry jumps" — a person's press jumps (ladder.go:34-36,175-176,193-194); the leg does not — OPERATOR
|
||||
- C6 check-engine-major.py:24-26 „the Update button still takes no backup" vs 09 §8.6 CLOSED — code: update.go:819-841
|
||||
- C7 RULING decision 250 item 5 „by its AGE (older than 7 days while the floor is above it)" vs golden_currency_gate.py:8-12 („release ahead" stands in for „floor ahead") — OPERATOR to know
|
||||
@@ -0,0 +1,37 @@
|
||||
# DOC-D1 — 07 backup architecture (reviewer candidates; verified by the main session before acting)
|
||||
|
||||
Reviewer: read-only subagent, 2026-10-24. Counts: claims 64 · true 44 · stale 11 · untestable 5 · undocumented 8 · contradictions 5.
|
||||
Paths under felhom-controller/controller/ unless named.
|
||||
|
||||
## STALE
|
||||
- S1 07:756 D22 „controller main, unreleased" → shipped v0.338.0 — CHANGELOG.md:237; internal/backup/restore_unit.go:442-512 — FACT
|
||||
- S2 07:383 Tier-1 „secret-stripped app.yaml" → since D5 (schema 2) the unit's app.yaml (0600) carries the PORTABLE secrets — internal/backup/recovery_unit.go:25,261,272 — FACT
|
||||
- S3 07:1537 (+1498, 1624) row 15 „DEFECT … no lock case → R-104" → R-104 closed v0.297.0; class `locked` — internal/backup/offbox.go:156-159,220-224,317 — FACT
|
||||
- S4 07:1532 row 11 Longhorn target DooPlex „2 restore points" → D40 took the hub volume out of the recurring group — felhom.eu/manifests/hub.yaml:47-51 — FACT
|
||||
- S5 07:256 „Hub v0.158.0 (built, deploy waits)" → deployed 2026-10-18 — hub/CHANGELOG.md:288 — FACT
|
||||
- S6 07:1531 row 10 put-back „next slice", R-973 „agent main, unreleased" → both shipped (v0.336.0; agent 0.160.0) — internal/backup/offbox_restore_undo.go:17-33; felhom-agent/internal/localapi/backup_cap.go:41-52 — FACT
|
||||
- S7 07:440 `cmd/controller/main.go:604-607` → main.go:1260-1264 via backupwindow.LegTimes — FACT
|
||||
- S8 07:1344,1347,1349 line citations moved (GetAppDrivePath backup.go:441, F5 pruneStalePrimaryDirs backup.go:1514); mp1 gone (07:386) — FACT
|
||||
- S9 07:450-451 „keep-last=1 … keep-last + 1" → keep-daily=N (R-1149) — felhom-agent/internal/config/config.go:558-559 — FACT
|
||||
- S10 07:1601 „a restore is refused while any second-drive copy runs" → the install-time kept-data load checks only RestoreStatus/IsRunning — internal/web/restore_wizard.go:226; internal/api/kept_install.go:89-99 — FACT or a small code gap
|
||||
- S11 07:975-976 §6.6 off-site row → the off-site restore also refuses a mixed/mismatched snapshot before anything is touched — internal/backup/offbox_reconstitute.go:829-832 — FACT
|
||||
|
||||
## UNDOCUMENTED
|
||||
- U1 a copy is limited to 60 min; a stray that cannot be removed blocks every restore, retried each minute; copies into one destination serialized — internal/backup/tier2_copy_lifecycle.go:40,113-121,150-172 — 07 §8.2b
|
||||
- U2 D25 „Eltávolítás" refused while any backup/restore/copy runs — internal/web/kept_copies.go:54-58
|
||||
- U3 removing an app WITH its backups deletes all of backups/secondary/<app>, D25 kept legs included; conflicts with „only the household's Eltávolítás removes it" — internal/backup/r474_remove_mirrors.go:61,87; tier2.go:487-488; tier2_kept.go:18,76 — CANDIDATE DEFECT
|
||||
- U4 the local unit restore skips the fit check when free space cannot be read (-1) — fails open — internal/backup/restore_unit.go:460; tier2_whole.go:374-375 — CANDIDATE DEFECT
|
||||
- U5 a LOCAL restore's put-back mails „its off-site restore FAILED" (RunKind offsite-reconstitute) — cmd/controller/main.go:1514-1518; internal/backup/offbox_restore_undo.go:242; restore_unit.go:518 — CANDIDATE DEFECT (text)
|
||||
- U6 a start that fails after a successful put-back: not held, copy kept, `restore_failed_put_back_not_started` — offbox_restore_undo.go:245-247
|
||||
- U7 the release copy (D33) has no row in 07's matrix — felhom.eu/scripts/release-copy/
|
||||
- U8 REUSE.md has no row for the stray-copy seams — tier2_copy_lifecycle.go:85,255,319,341
|
||||
|
||||
## CONTRADICTIONS
|
||||
- C1 07:1021 `locked` class vs 07:1537/1498 open DEFECT; OPEN-ITEMS:94,100 R-104 open vs CLOSED-ITEMS:895 closed — code: closed
|
||||
- C2 00-capability-map G15 „nightly" vs 07:1532 „6-hourly" — code: 6-hourly
|
||||
- C3 07:383 „secret-stripped" vs 07:1523 „portable secrets" — code: portable
|
||||
- C4 agent comment backup_cap.go:15 „keep-last" vs 07:386-387 keep-daily — code: keep-daily (the comment is stale)
|
||||
- C5 controller REUSE.md:84-85 D22 „(unreleased)" vs CHANGELOG v0.338.0 released
|
||||
|
||||
## UNTESTABLE
|
||||
ep0 keep-weekly / prune jobs (ep0 config); hub-DB retention 14 daily + 8 weekly (ep0); R-1214 stall numbers; „0.336.0 is the floor"; the carry end to end.
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
# DOC-D2 — 16 storage identity, 15 media library, _recovery-inventory-2026-07-28 (reviewer candidates; verified by the main session)
|
||||
|
||||
Reviewer: read-only subagent, 2026-10-24. Counts: claims 52 · true 30 · stale 13 · untestable 5 · undocumented 7 · contradictions 4.
|
||||
|
||||
## STALE (all FACT)
|
||||
- S1 16:233-234 an empty partition asks for the typed confirmation (R-1093) → fixed agent 0.160.0 — felhom-agent/internal/storage/hostops.go:459-491
|
||||
- S2 16:18 single-copy notice „a Go constant, not i18n" → i18n note.backup.single_copy(_unproven) — controller internal/backup/backup.go:1485-1494
|
||||
- S3 16:19 `/disks/candidates` „whole disks only" → whole disks with nested partition candidates — agent internal/storage/candidates.go:87-88; partitions.go:84-96
|
||||
- S4 16:17,19,21,129 line citations moved (install.sh:3294 = keep-daily=3 since R-1149; hu.json:330; offer.go:236; install.sh:2475)
|
||||
- S5 15:197,213 a drive added/back → „up -d recreates … household: no" → only the binds are rewritten; the restart waits for „Újraindítás most" / „Később" (03:30); the file browser alone recreates itself — controller internal/stacks/media_library.go:286-288; internal/web/media_events.go:97-110,250-274
|
||||
- S6 15:208 eject DEREGISTERS → a safe removal keeps the drive registered (SetEjected); its return re-binds with no step — media_events.go:304,319-352
|
||||
- S7 15:166-168 Transmission watch/download folders off-site / `optional` → catalog c50adfc (R-1019) marks them `excluded`
|
||||
- S8 inventory:194,392-396 rootfs + mp0 /var/lib/docker + mp1 /mnt/sys_drive → one mp0 at /var/lib/felhom; no mp1 — felhom-agent/configs/build-golden.sh:31-35,99-105,215-225
|
||||
- S9 inventory:89,351,509-517 secrets never from the unit; unit app.yaml secret-stripped → schema-2 units carry the portable secrets; a restore takes the DB password from the unit (R-1106) — recovery_unit.go:30-41; restore_dbcreds.go:16-30
|
||||
- S10 inventory:99,275,363 the copy's recovery-unit/ read by no code path (C9-F4) → RestoreTier2Unit (R-102) — tier2_restore.go:182-211; web/handlers.go:2324
|
||||
- S11 inventory:138,276,373 named-volume tars never unpacked → reconstitute replays them (R-354) — offbox_reconstitute.go:72,1040-1047
|
||||
- S12 inventory:383 ep0 keep-last 2; local keep-last 3 → ep0 keep-weekly 2 at 03:30 (R-1162/R-1165); local keep-daily=3 (R-1149)
|
||||
- S13 inventory:374,656-673 restic 0.14.0 exits 0 on a missing source; namespace „root" → restic 0.19.1 exits 3; namespace from storage.cfg, "" for root
|
||||
|
||||
## UNDOCUMENTED
|
||||
- U1 the „Később" 03:30 media restart is an in-memory timer (a restart forgets it) — media_events.go:250-264 — 15 §11
|
||||
- U2 „Restart now"/„Később"/safe removal force-recreate EVERY deployed media app, including apps the household STOPPED (up -d --force-recreate starts them); held skipped; no backup-run check — media_library.go:270-280; manager.go:1477; media_events.go:196-205,306-315 — CANDIDATE DEFECT
|
||||
- U3 the Sonarr/Radarr key is read from /config/config.xml (the „SET at install" option not built) — mediawire.go:336-345
|
||||
- U4 systemd-detect-virt read once per agent; empty → unknown/vm (fails closed) — agent identity.go:195-205,370-372
|
||||
- U5 an unmapped verdict falls back to st_dev; an unreadable path counts as the SAME disk (R-668) — backup.go:1408-1450
|
||||
- U6 the internal place refuses eject/decommission/wipe, allows a data move — web/internal_place.go:22-25
|
||||
- U7 the internal place subtracts every promised thin volume — agent configs/felhom-internal-place:11-14,71-73
|
||||
|
||||
## CONTRADICTIONS
|
||||
- C1 08:535,583 upload watch „NOT released" vs capability map F13 PROVEN-LIVE on 0.342.0 — code/CHANGELOG: released
|
||||
- C2 15:208 eject deregisters vs 15:251 safe removal keeps it — code: keeps
|
||||
- C3 runbooks/RUNBOOK-manual-guest-restore.md:28,38,93 `mp1 … /mnt/sys_drive` vs 07:386 mp1 gone — code: gone (recovery-relevant)
|
||||
- C4 .claude/rules/live-probes.md:41 `/opt/felhom/stacks/<name>` vs the real `/opt/docker/stacks/<app>/` (a path in a rule file)
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
# DOC-E — 02 controller module map, 10 localisation, 11 OS updates, 14 host power-off (reviewer candidates; verified by the main session)
|
||||
|
||||
Reviewer: read-only subagent, 2026-10-24. Counts: claims 121 · true 92 · stale 22 · untestable 6 · undocumented 5 · contradictions 3.
|
||||
|
||||
## STALE (all FACT)
|
||||
- S1 14:36 heavy-op gate holders → also the weekly line test (agent v0.164.0) — felhom-agent/internal/linetest/linetest.go:62,320
|
||||
- S2 14:25 „two exact sudoers vectors, FELHOM_POWEROFF" → FELHOM_POWEROFF holds only `systemctl poweroff`; `pct shutdown` is FELHOM_GUESTHOOK — felhom-agent.sudoers:81-90
|
||||
- S3 14:89 website FAQ line „a later task (R-1000)" → present — website/gyik.html:128,603
|
||||
- S4 14:39-40 a paragraph breaks the §3 table (rendering) — the doc itself
|
||||
- S5 11:81 cloudflared `2026.6.0` → `2026.9.3` — controller internal/infra/infra.go:26
|
||||
- S6 11:104-106 line citations moved (sudoers:68,228; lanresolver.go:133; manager.go:659)
|
||||
- S7 11:253-256 „DRAFT … nothing installed … --repair-only" → built; sudoers has only `--plan` — sudoers:336-337
|
||||
- S8 11:14 „§8 step 2 BUILT" → steps 2–6 built (the doc's own §8); „NOT RATIFIED" is the operator's
|
||||
- S9 11:318-320 BUNDLE_FILES „22" → 32 — felhom-os-apply:191-233
|
||||
- S10 11:482-483 HOST_SLOW_RE list → also linux-(image|headers|kbuild|modules|base), firmware-, proxmox-secure-boot-support — felhom-os-apply:100-102
|
||||
- S11 10:202-204 the setup wizard still reachable, deletion R-554 → deleted v0.308.0; bootstrap.WaitForConfig
|
||||
- S12 10:131 „Part B is the remaining piece" → built (v0.256.0) — controller internal/config/config.go:149-154
|
||||
- S13 10:184-189 formal-form ceiling 16 → 0 (R-516 item 12) — controller scripts/i18n_missing_gate.py:41
|
||||
- S14 10:237,310,313 „53 apps", „fifth row" → 65 apps; sixth row; fixtures still 53
|
||||
- S15 10:229 metadata.go L316 → :475
|
||||
- S16 02:108 „two in-flight markers" → also update-journal.json and .felhom-install-pending — update.go:49,1306; install_interrupted.go:16-28
|
||||
- S17 02:171 „three holders" → four (1b IsUpdating) — cmd/controller/main.go:2459-2477
|
||||
- S18 02:387-390 GetAlerts → GetBannerAlerts(page) filters before the cap (R-906) — internal/web/alerts.go:350-358
|
||||
- S19 02:267,272,742 „no longer calls the Cloudflare API" → internal/cloudflare still wired when cf_api_token is set — main.go:1994-2000; config.go:160
|
||||
- S20 02:328-329,724-725 state.go/updater.go „DELETE … removed" → both live; the swap goes to the agent (AgentSwapper) — internal/selfupdate/updater.go:38-47,87
|
||||
- S21 02:821-825 line citations moved (sync.go:436; config.go:399)
|
||||
- S22 02:888-890 locked_after_deploy „read by no code" → read into LockedFields; enforced only by an uncalled UpdateStackConfig — deploy.go:428,718-748,837
|
||||
|
||||
## UNDOCUMENTED
|
||||
- U1 a re-enroll during the household power-off hold counts as „gone" (R-1154) — hub/internal/api/reenroll.go:67-68 — 14 §4
|
||||
- U2 host_deliberate_off(_ended) recorded as info events, not mailed — hub/internal/api/deliberate_off.go:12-13,42-50
|
||||
- U3 the unaccented confirmation word „KIKAPCSOLAS" is accepted — controller internal/web/poweroff_handlers.go:29
|
||||
- U4 sockheal (R-860), channelhealth, ~20 post-v0.33 packages missing from 02 §1 — internal/sockheal/sockheal.go:1-6; internal/channelhealth/checker.go:1-6
|
||||
- U5 hub System page „Upload line" cell (amber after 15 days) — hub/internal/web/system.go:174 — 11 §5.7
|
||||
|
||||
## CONTRADICTIONS
|
||||
- C1 10:202-204 setup wizard reachable vs 02:56 deleted v0.308.0 — code: deleted
|
||||
- C2 03:132 D4 „NOT shipped" / 13:76 root password SSH vs shipped in v0.163.0 — code: shipped
|
||||
- C3 02:108 two markers vs 09:2102 update-journal.json — code: four
|
||||
@@ -0,0 +1,15 @@
|
||||
# HUNT-1 — restore and put-back (reviewer candidates; verified by the main session before any row)
|
||||
Paths under felhom-controller/controller/.
|
||||
|
||||
- H1 P1 — a power cut during a restore of a DRIVE-backed app: Recover runs before the drive is bound → „restore copy … is gone — restarting as before" → the gated start refuses → app Refused, marker + Undo kept → intermediary.go:470-476 boot recreate checks only appHeld, not the marker → the app runs on the cut restore's data → the next Recover finds the copy and pours it back over the newer writes — internal/backup/offbox_restore_undo.go:47-48; appstop_marker.go:362-375,408-411; cmd/controller/main.go:2573-2576; internal/web/intermediary.go:400-403,470-476
|
||||
- H2 P2 — a FAILED crash put-back leaves the app stopped, unheld; Start or the boot recreate brings it up; a later restart pours back an old copy — appstop_marker.go:368-371; api/router.go:690-709; intermediary.go:472
|
||||
- H3 P2 — an export with „stop app first" during another app's restore overwrites the single marker file — appstop_marker.go:286-292,318; appexport/export.go:173-178,262-266
|
||||
- H4 P3 — off-site restore: a definition-write failure restarts the app on a half-written definition — offbox_reconstitute.go:875-883,979-986; main.go:3318-3331
|
||||
- H5 P3 — a put-back can merge (failed stop only logged, `volume rm -f` error ignored, tar over), then the copy is deleted — restore_crash_putback.go:155-171; offbox_restore_undo.go:223-225; restore.go:218,251
|
||||
- H6 P3 — a second-drive mirror can start between the web refusal check and the restore's record — web/restore_wizard.go:226; opstatus.go:56-66; handlers.go:2209
|
||||
- H7 P3 — Stop pressed during a restore: after a cut the put-back is skipped, marker cleared, copy left — appstop_marker.go:356-361
|
||||
- H8 P4 — an unused restore copy is never cleaned up — offbox_restore_undo.go
|
||||
- H9 P4 — a comment names a test file that does not exist (`r1195_crash_putback_test.go`) — restore_crash_putback.go:32
|
||||
|
||||
## CHECKED AND SOUND (reviewer)
|
||||
put-back before any start; journal names the same stack; atomic writes; Undo recorded only after a whole copy; fit check before the stop; DBs are named volumes; hold path; capped copy --mount/no symlinks/--pull never; start-up stray clean-up; beginTier2Run under one lock; one single-flight flag.
|
||||
@@ -0,0 +1,14 @@
|
||||
# HUNT-2 — delete and retention (reviewer candidates; verified by the main session before any row)
|
||||
|
||||
- H1 P2 — one bad/rejected `.felhom.yml` backup block deletes the app's second-drive copies of every userdata/ and non-appdata hdd/ leg the same night — controller internal/stacks/metadata.go:550-557 (meta.Backup = nil on any defect, incl. classify.go:185 „matches no compose bind") → appbackup/classify.go:217-221 has=false → backup/tier2_capture.go:84-100 legacy legs hdd/appdata/<name> only → backup/tier2.go:492 tier2Reconcile removes every non-leg dir (tier2.go:325-329); also cs.Skipped legs (tier2_capture.go:56-61). Against D25's header tier2_kept.go:17-18.
|
||||
- H2 P3 — a D25 kept copy is reconciled away when its leg later leaves the declared set; its record stays — tier2.go:485-492; settings/kept_copies.go; tier2_kept.go:93-95
|
||||
- H3 P3 — R-1019 also moved MeTube (`media/metube`), NZBGet, SABnzbd to `excluded`; the reconcile deletes their copies — catalog templates/metube/.felhom.yml:33; commit c50adfc — a design call
|
||||
- H4 P3 — Remove (even keep-data) deletes volumes labelled with the app's project that the definition no longer declares; no tier holds them — controller internal/stacks/delete.go:763-764 (8ae9178, R-1117)
|
||||
- H5 P3 — the capped copy's rsync --delete sees the source in the daemon's mount view; the emptiness check runs in the controller's view — backup/tier2.go:834 vs :464
|
||||
- H6 P3 — the release copy keeps the newest 5 agent versions, vouched or not — felhom.eu/scripts/release-copy/felhom-release-copy:199,218,237
|
||||
- H7 P4 — release copy rsync --delete removes old versions during the transfer; a shrinking registry mirrors deletions — felhom-release-copy:190,237
|
||||
- H8 P4 — hub-DB push has no lower age bound (a future-stamped snapshot stays „newest") — scripts/hub-db-backup/felhom-hub-db-backup:41,49-50
|
||||
- H9 P4 — Remove with remove_backups deletes kept app copies under the mirror and leaves their records — api/router.go:1086-1087
|
||||
|
||||
## CHECKED AND SOUND (reviewer)
|
||||
tier2SafeRemove bounded to /backups/secondary/ (tier2.go:260-269); RemoveKeptCopy same device+inode (tier2_kept.go:92-111); RemoveTier2Mirrors exact paths (r474_remove_mirrors.go:18-19,76-84); shares reconcile drive-scoped (tier2_shares.go:262,347-358); RemoveStack backup path bound (delete.go:839-846); release copy refuses before rsync, size cap first, rrsync; hub-db-backup STAGE default, integrity before push; ep0 prune jobs ns-scoped; agent keep-daily ≥ 1; R-1010 force-delete narrow.
|
||||
@@ -0,0 +1,9 @@
|
||||
# HUNT-3 — keys (reviewer candidates; verified by the main session before any row)
|
||||
No P1.
|
||||
- H1 P2 — the hub stores a household's new escrow package in TWO writes (SaveHostEscrow commits blob+hash, then SaveHostDRBundle sets identity_blob); a hub failure between them leaves the NEW hash beside the OLD identity package; the box's confirm passes and flips to escrowed; on a total loss the code opens nothing — hub/internal/api/handler.go:1415,1445; internal/store/store.go:3496,3511,3793; controller internal/report/escrow_confirm.go:195-217; agent sends once (cmd/felhom-agent/main.go:3366-3386)
|
||||
- H2 P2/P3 — a lost/late hub answer tells the household „your previous code still works" when the hub may hold the new package — agent internal/localapi/escrow_ceremony.go:79,532-534,639-641; controller i18n en.json:2662,2664
|
||||
- H3 P3 — the R-1176 carry can move a package with no repository password (checks identity_blob only), leaving the right package under the deleted host — hub/internal/store/escrow_carry.go:100-110; store.go:3533-3534,3631-3633; controller backup/offbox.go:608
|
||||
- H4 P4 — same-day set-aside name reuse can purge a still-stored copy's package — hub/internal/offsitekeys/offsitekeys.go:378-385; store.go:4295-4300
|
||||
|
||||
## CHECKED AND SOUND (reviewer)
|
||||
D30 key file atomic and heals; once an hour across restarts; hub swaps in place and refuses another host's key; PBS hook inert after a re-key; D31 re-issue changes the token only (comment wording); R-1176 customer-scoped, one transaction, PK stops a double carry; Reconcile before SetFloor; R-1182 purge never touches host_escrow; RESET needs acks; R-975 seal all-or-nothing; the mint guard.
|
||||
@@ -0,0 +1,11 @@
|
||||
# HUNT-4 — time and concurrency (reviewer candidates; verified by the main session before any row)
|
||||
No P1 or P2. R-1250 (the hold + test on request) is committed, not released (agent „next v0.165.0", controller „next v0.343.0").
|
||||
- H1 P4 — 2026-10-25 (fall back): DooPlex pushes the 20:00 hub snapshot again at 02:30 CEST and never the 02:00 one (Go puts the hub's 02:00 slot on the SECOND pass = 01:00Z, after the 00:30Z push); the copy is ~13.5 h old until 08:30 — scripts/hub-db-backup/felhom-hub-db-backup.timer; hub/cmd/hub/main.go:1092; hub/internal/dbsnap/dbsnap.go:32; felhom-hub-db-backup:19,50
|
||||
- H2 P3 (2027-03-28) — spring forward: the leg's deadline is W + 5 h of REAL time, the gate counts wall-clock minutes; a step can start after the full-system backup stopped waiting — controller internal/stacks/unattended.go:225; internal/quiesce/quiesce.go:922-933
|
||||
- H3 P4 (2027-03-28) — W in 02:00–02:59: the DB dump and the second-drive copy start at the same instant (the missing 02:30 maps to 03:30) — internal/scheduler/scheduler.go:509; backupwindow.go:17
|
||||
- H4 P4 — two seeders behind one uplink: the line hold holds only the first, the test is refused „not quiet" every 15 min in the Tuesday window, never counted — controller internal/upwatch/linehold.go:155-160; agent internal/linetest/linetest.go:379-381 (unreleased)
|
||||
- H5 P4 — a hold record of an app removed while held is never cleared; qBittorrent's 50 % rule never runs again — linehold.go:106-108; upwatch/service.go:149 (unreleased)
|
||||
- H6 P4 — qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored — upwatch/explain.go:175; linehold.go:127-128 (unreleased)
|
||||
|
||||
## CHECKED AND SOUND (reviewer)
|
||||
2026-10-25 controller legs fire once each; catch-up uses the same resolution; fall-back leg deadline earlier (safe); hub scheduleDaily day+1 + last guard; release copy 01:40 ends before the repeated hour; line test holds the gate without blocking; 2 min timeout > agent worst case; hold record written first, release deferred, recovery at start; second-drive copy one lock per destination; agent backup cap rolling 24 h.
|
||||
@@ -0,0 +1,10 @@
|
||||
# SEC-1 — hub + controller endpoints changed since 2026-10-17 (reviewer candidates; verified by the main session)
|
||||
No P1 or P2 found.
|
||||
- X1 P3 — the upload watch treats ANY viewer as an explanation for any upload size; a compromised Plex can upload freely while one member watches — controller internal/upwatch/watch.go:275,299; 08:559 states the rule, 08:579's limits list does not name this
|
||||
- X2 P4 — after the operator deletes a household's host, the first enroll with only the passphrase becomes the live host and gets the sealed package (ciphertext; D19 accepts the window) — hub/internal/api/passphrase_scope.go:51-52; internal/store/escrow_carry.go:134-139
|
||||
- X3 P4 — a passphrase holder can make the hub mail passphrase_keys_refused repeatedly — passphrase_scope.go:75-78; handler.go:1271
|
||||
- X4 P4 — the qBittorrent admin password goes to whatever answers the DNS name `qbittorrent` — controller internal/upwatch/explain.go:94,108-113
|
||||
- X5 P4 — kept-copy Remove: a gap between the identity check and RemoveAll — backup/tier2_kept.go:92-111; tier2.go:269
|
||||
|
||||
## CHECKED AND SOUND (reviewer)
|
||||
kept remove behind session+CSRF; debug actions behind RequireAuth(CsrfProtect); .fab path checks; hub /preferences + /box-config self-scoped; /report body id = key; fhp_ refused on /api/v1, unclassified operator routes session-only; line-test run rate-bounded; Upload-line cell escaped; new events operator-only; bind page texts only for a real token; files forwardAuth strips X-Forwarded-*; network_mode refused.
|
||||
@@ -0,0 +1,11 @@
|
||||
# SEC-2 — agent local API, ep0 iperf3, installer sshd drop-in, DooPlex→ep0 jobs (reviewer candidates; verified by the main session)
|
||||
|
||||
- X1 P2 — a BYO host (the household owns it) gets root password SSH switched off by the config bundle; the installer's BYO screen says „NOT touched in byo mode: … OOB sshd" — felhom-agent/configs/felhom-os-apply:230 (the drop-in written for every mode; plan_writes 1985-2000 holds back only policy `oob`); felhom.eu/scripts/felhom-host-install.sh:2118, list 2092-2106 — proof: a byo config on a scratch host, sshd -T before/after the bundle
|
||||
- X2 P3 — any tunnel peer (root on one box) can hold ep0's single iperf3 server (ErrBusy fleet-wide → MaxTriesPerDay) or ask `-R -t <long>` and fill ep0's uplink — felhom.eu/scripts/ep0-iperf3/felhom-iperf3.service:17 (no authorized users, no server-side caps)
|
||||
- X3 P3 — the „in force" check probes only addr=192.0.2.1, so a `Match Address <LAN>` block with PermitRootLogin yes reads as in force — felhom-os-apply:2101-2110
|
||||
- X4 P4 — `sshd -t` failing for any reason refuses and undoes the WHOLE bundle — felhom-os-apply:2102,2112-2115,2054-2061
|
||||
- X5 P4 — the release-copy restore check trusts hashes stored on ep0 itself — runbooks/total-loss-of-dooplex.md
|
||||
- X6 P4 — rrsync without -munge/-wo; a symlink push might escape `data/` depending on `.ssh` ownership — audits/day-2026-10-23/08-release-copy-D33.txt:6
|
||||
|
||||
## CHECKED AND SOUND (reviewer)
|
||||
line-test routes bearer→VMID (server.go:579-580, 692-715); the POST is rate-bounded (due, window, 6/day, heavy-op gate); iperf3 on 10.77.0.1 only, wg0 + /24, forward drop; DynamicUser hardening; the drop-in value pinned to prohibit-password, sshd -t -f before write, undo reloads, uninstall removes; release-copy names validated, host key pinned, 0600 O_EXCL login file; read:package token; ep0 cannot write back to DooPlex; hub-db-backup name pattern, DatastoreBackup scope, shredded staging; ProtectSystem=strict units.
|
||||
@@ -0,0 +1,43 @@
|
||||
"""h1_tier2.py install|chain|copy|break|unbreak|remove — review round 2, HUNT-2 H1 on scratch 9202 (the product's own
|
||||
endpoints; box_walk.py). A bad backup block in an app's .felhom.yml: does the night's second-drive copy then delete the
|
||||
app's household-file copies?"""
|
||||
import json, re, sys, time
|
||||
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
|
||||
import box_walk as w
|
||||
APP, SUB = "audiobookshelf", "audiobookshelf"
|
||||
MARK = "/mnt/felhom-drives/scratch_hdd/userdata/media/audiobooks/review2-h1-marker.txt"
|
||||
COPY = "/mnt/sizing-bench/backups/secondary/audiobookshelf"
|
||||
FY = "/opt/docker/stacks/audiobookshelf/.felhom.yml"
|
||||
w.LOG = []
|
||||
w.login()
|
||||
cmd = sys.argv[1]
|
||||
if cmd == "install":
|
||||
print("deploy:", w.deploy(APP, SUB, {"HDD_PATH": "/mnt/felhom-drives/scratch_hdd"}))
|
||||
print(w.guest(f"echo 'review round 2 H1 marker — a household file' > {MARK}; ls -la {MARK}"))
|
||||
elif cmd == "chain":
|
||||
since = w.guest("date -u +%Y-%m-%dT%H:%M:%SZ").strip()
|
||||
code, d = w.ctl("POST", "/api/debug/backup/night-chain")
|
||||
print("night chain ->", code, str(d)[:160])
|
||||
out = ""
|
||||
for _ in range(120):
|
||||
time.sleep(10)
|
||||
out = w.guest(f"docker logs --since {since} felhom-controller 2>&1 | grep -E 'night-chain|Tier 2|tier2|backup block|reconcile' | grep -v DEBUG | cut -c1-260")
|
||||
if re.search(r"night-chain\] update-leg: done|night-chain\].*finished|update leg \(manual-chain\)", out):
|
||||
break
|
||||
print(out)
|
||||
elif cmd == "copy":
|
||||
print(w.guest(f"find {COPY} -maxdepth 5 | sort | head -40; echo; ls -la {COPY}/userdata/media/audiobooks/ 2>&1 | head"))
|
||||
elif cmd == "break":
|
||||
print(w.guest(f"""cp -p {FY} /root/review2-h1-felhom.yml.orig && python3 - <<'PY'
|
||||
p='{FY}'; s=open(p).read()
|
||||
old=' - path: media/podcasts\\n'
|
||||
assert old in s, 'anchor'
|
||||
s=s.replace(old, ' - path: media/review2-typo\\n class: optional # REVIEW-2 H1 PROOF: a path no compose bind has\\n'+old, 1)
|
||||
open(p,'w').write(s)
|
||||
PY
|
||||
grep -n -B1 -A1 review2-typo {FY}"""))
|
||||
elif cmd == "unbreak":
|
||||
print(w.guest(f"cp -p /root/review2-h1-felhom.yml.orig {FY} && rm -f /root/review2-h1-felhom.yml.orig && grep -c review2-typo {FY}; echo restored"))
|
||||
elif cmd == "remove":
|
||||
print(w.remove(APP))
|
||||
print(w.guest(f"rm -f {MARK}; ls {COPY} 2>&1 | head -3"))
|
||||
Reference in New Issue
Block a user