# Recovery baseline inventory — 2026-07-28 > **Class: INVENTORY, READ-ONLY.** Facts only. Nothing was changed, fixed, deployed or restarted. > No backup, restore, GC or prune was run to "check" something. `peti-felhom` was not touched. > No matrix is designed here, no target architecture is proposed, and no gap is ranked. > > **This is the input to a redesign, not the redesign.** ## 0. Method, baselines, and what "verified" means here Three claim kinds are distinguished throughout and labelled per claim: | label | meaning | |---|---| | **SOURCE** | read from code at a cited `file:line` in this workspace at the HEADs below | | **LIVE** | observed by running a read-only command against live infrastructure during this session, output pasted | | **DOC** | asserted by a document; unverified unless a SOURCE/LIVE line accompanies it | | **INFERENCE** | my reasoning over SOURCE/LIVE facts, marked as such | **Repo HEADs read (LIVE, `git rev-parse --short HEAD`, all clean except a pre-existing modification to `felhom.eu/documentation/PROMPT-TEMPLATE.md` which is unrelated to this task):** ``` felhom.eu ff050cf felhom-agent d5c7691 felhom-controller fd50a73 app-catalog-felhom.eu 4252121 ``` **Live fleet at inventory time (LIVE):** ``` hub gitea.dooplex.hu/admin/felhom-hub:0.80.0 (k3s felhom-system, 1/1, pod age 6h18m) agent felhom-agent 0.110.0 on demo-felhom AND demo-hp controller felhom-controller:0.183.0 in guest 9201 on BOTH boxes catalog 53 templates hosts demo-felhom (192.168.0.162) · demo-hp/`felhom-host` (192.168.0.87) · ep0 167.233.158.164 guests one per host: 9201, both `running`, both `onboot: 1` ``` Session clock: commands ran `2026-07-28 17:40–17:45 UTC` / `19:40–19:45 CEST`. Host clocks are CEST; hub, guests and PBS are UTC — every timestamp below carries its zone. **One derived artefact was made and is stated so it is not mistaken for a live read:** a hot copy of the hub SQLite DB was taken with `kubectl exec … cat /data/hub.db > hub.db.copy` at **17:40:00 UTC** (113,033,216 bytes) into the session scratchpad, and every hub-DB figure below comes from that copy. It is a hot copy of a live database; row counts and metadata are consistent enough for an inventory but are a snapshot of that instant, not a transactionally consistent dump. **Do not reuse this command as a recipe: since hub v0.88.0 the DB is in WAL mode (R-172), so `cat /data/hub.db` alone yields a copy that opens cleanly and silently omits the newest writes — the `-wal` must be copied beside it (`documentation/operations/nodes.md`).** The copy is read with `mode=ro`. No hub table was written. **Secret hygiene.** No secret value, key, token, password or key fingerprint is reproduced in this document. Where a secret is named, only its location, size, mode and owner are recorded. --- # Part A — every restore path that exists Twelve distinct restore paths were found. They are grouped by who can reach them, because that is the axis this task exists to establish. Route inventory method (so "nothing else found" is falsifiable): every `case path ==` in `controller/internal/web/server.go:360-551`; every route in `controller/internal/api/router.go:153-332`; every `mux.HandleFunc` in `controller/internal/setup/handlers.go:85-99`; every `--selftest` mode in `felhom-agent/cmd/felhom-agent/main.go:168`; every `RUNBOOK-*.md` under `felhom.eu/documentation/runbooks/` and `felhom.eu/documentation/pilot/`; and a repo-wide grep for `restore|Restore` over `felhom-controller/controller/internal` and `felhom-agent/internal`. ## A.1 — Customer-reachable, unassisted (dashboard password only) All five below sit behind the controller's `RequireAuth` mux gate — the single household dashboard password, which the **customer** owns (set at claim; `internal/web/auth.go:34-35` precedence `settings.json → password_hash` over `controller.yaml → web.password_hash`). No operator credential, no SSH, no hub action is needed. This is the whole customer-recovery surface. ### A.1.1 — „Visszaállítás indítása" — Tier-1 recovery-unit restore | field | detail | |---|---| | **Name (UI, HU)** | **„Visszaállítás indítása"** — page „Biztonsági mentés → Visszaállítás" | | **Entry point** | `POST /backup/restore` (`stack_name` + `snapshot_id`) — button `internal/web/templates/backups_restore.html:52`, form action `:346`; route `internal/web/server.go:466-467`; handler `internal/web/handlers.go:1241`; engine `internal/backup/restore_unit.go:95` `RestoreFromRecoveryUnit` | | **Source** | **Tier-1 only.** The app's own drive: `/backups/primary//` (`compose/`, `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json`). **Tier-2 copies are unreachable from here** — stated and pinned at `internal/backup/restore_points.go:16-18`: *"Tier-2 copies are NOT restorable through POST /backup/restore (it only reads the app's primary unit), so listing them would silently restore tier-1 data while claiming tier-2 — never emit them here."* | | **Scope** | compose + `.felhom.yml` + secret-stripped `app.yaml`; **Docker named volumes** (tars re-imported); **DB dumps replayed** into a DB-only bring-up (R-47); app redeployed on the pinned image. This is the **only** path that restores named-volume tars. | | **Semantics** | **DESTRUCTIVE/REPLACING** — volume contents are re-imported over the live volume and the stack is redeployed from the unit's compose. | | **Invocable by** | **CUSTOMER, UNASSISTED** | | **Prerequisites** | app deployed and its drive resolvable (`GetAppDrivePath` non-empty, `restore_unit.go:114-117`); a recovery unit present (absent ⇒ silent fallback to volume-only `RestoreApp`, `:120-127`); **the guest's live `app.yaml` must still hold the app's secrets** — see the fail-closed gate below; no concurrent backup/restore (`IsRunning()`, `handlers.go:1270`) | | **Fail-closed gate** | `reconcileRestoreSecrets` (`restore_unit.go:23`) **refuses** when a *data-encrypting* key is unrecoverable, because regenerating it would make the restored data unreadable; a missing *resettable* secret is regenerated and logged (`:143-163`). Secrets are recovered **from the guest, never from the unit** (`:132`). | | **Proven** | **PATH EXECUTES — content recovery NOT proven.** Campaign 9 A2 on `calibre-web`/demo-felhom: `15:28:34 Restoring calibre-web from recovery unit: images=1 … 15:28:52 Restore-from-unit completed: calibre-web (18.25s)`, app healthy 29 s later (`CAMPAIGN-9-restore-proof-2026-07-28.md:141-154`). The campaign states its own limit: *"It does not prove content recovery, because nothing was lost first"* (`:153-154`). Listed as **"the single most valuable unproven item"** (`:825-827`) and repeated in `felhom.eu/REPORT.md:96-97`. | ### A.1.2 — „Fájlok visszaállítása" — Tier-2 cross-drive file restore | field | detail | |---|---| | **Name (UI, HU)** | **„Fájlok visszaállítása"** — page „Biztonsági mentés → Alkalmazások" (`/backups/apps`) | | **Entry point** | `POST /backup/tier2/restore` (`stack_name`) — form `internal/web/templates/backups_apps.html:247,252`; route `server.go:469-470`; handler `handlers.go:1310`; engine `internal/backup/tier2_restore.go:118` `RestoreTier2Files` | | **Source** | the **RECORDED** Tier-2 destination (`settings.CrossDriveBackup.DestinationPath`), never a fresh target pick — `tier2_restore.go:114-116`. Only two subtrees are read: `destBase/hdd → liveNsRoot` and `destBase/userdata → liveNsRoot/userdata` (`:101-104`), each `os.Stat`-skipped when absent (`:114-116`). | | **Scope** | **file legs only.** The copy's `recovery-unit/` (DB dumps + named-volume tars) is written by every Tier-2 run (`tier2.go:369` "Unit leg (always)") and **read by no code path** — this is C9-F4. | | **Semantics** | **ADDITIVE / MISSING-ONLY.** `rsync -a --ignore-existing` (`tier2_restore.go:213-217`): existing live files are never overwritten, nothing is ever deleted. | | **Invocable by** | **CUSTOMER, UNASSISTED** | | **Prerequisites** | a recorded Tier-2 destination with the v2 layout marker; app stopped for the copy (stop-first is the locked consistency policy, `:116-117`); no concurrent op. Since v0.183.0 a **pre-flight coverage check refuses without stopping the app** when the copy holds no readable leg (`handlers.go:1341-1346`). | | **Proven** | **PROVEN, twice, on live hardware.** Campaign 9 **A1** (paperless-ngx / demo-hp, through traefik with a real session cookie and CSRF token): 6 deleted files returned byte-identical, 2 files created after the backup survived, 1 locally-modified file was not overwritten, 43+2 arithmetic closed, and the restored bytes were served back over paperless's own HTTP API with matching sha256 (`CAMPAIGN-9…:79-131`). Campaign 9 **A3**: the entire `appdata/paperless` directory destroyed while the app ran (verified by a positive observable — doc download 200 → **404**), then `43 fájl visszaállítva`, `documents_ok 16 of 16`, and 200s with matching hashes again (`:166-200`). Re-verified post-fix in v0.183.0 (`felhom.eu/REPORT.md:61`). | ### A.1.3 — „Visszaállítás a távoli tárolóból" — offsite restore to a verification copy | field | detail | |---|---| | **Name (UI, HU)** | **„Visszaállítás a távoli tárolóból"** / **„Teljes visszaállítás előkészítése"** — page „Biztonsági mentés → Visszaállítás" and the per-app wizard `/backups/restore/app` | | **Entry point** | `POST /backup/offbox/restore` (`app`, `mode=unit|full`, `confirm=1`) — forms `backups_restore.html:62,80` and `backups_restore_wizard.html:62`; route `server.go:482-483`; handler `internal/web/offbox_handlers.go:296`; engine `internal/backup/offbox_restore.go:182` `RestoreOffboxScratch` | | **Source** | Tier-3, `restic restore latest --tag ` from the Hetzner Storage Box over SFTP. Writes to an **on-data-drive** scratch `/backups/offsite-restore/` — never `cfg.Paths.DataDir` (the guest rootfs; the F-A1 filler), `offbox_restore.go:136-140`. | | **Scope** | `mode=unit` (default) restores the recovery unit only via `--include `; `mode=full` restores unit + mandatory userdata behind a size×1.1 headroom gate, **fail-closed on unknown size** (`:177-181`). | | **Semantics** | **NON-DESTRUCTIVE.** Nothing live is touched; this only materialises a verification copy. | | **Invocable by** | **CUSTOMER, UNASSISTED** | | **Prerequisites** | offsite configured and enabled; escrow ceremony completed (see Part C); restic repo reachable and **unlocked** (C9-F3); a data drive with headroom; no concurrent op; 30-minute background context (`offbox_handlers.go:335`) | | **Proven** | **PROVEN (bytes).** Campaign 8 Phase C, R-87: `bookstack` snapshot `b2d8936d`, 155.5 MB, 14 s, via the real two-step size gate — **6/7 files byte-identical by sha256** including a 160 MB MariaDB volume dump; the 7th differs only in `controller_version`/`created_at`, i.e. the restore correctly returned the snapshot's manifest (`CAMPAIGN-8…:246`). Campaign 8 also records what this does **not** prove: *"Restoring bytes is not the same as restoring service, and only the bytes were proven"* (`:255-258`). | ### A.1.4 — „Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" — offsite place-to-live | field | detail | |---|---| | **Name (UI, HU)** | **„Helyreállítás az élő adatok közé (csak a hiányzó fájlok)"** | | **Entry point** | `POST /backup/offbox/place` (`app`) — form `backups_restore_wizard.html:93`; route `server.go:484-485`; handler `offbox_handlers.go:459`; engine `offbox_restore.go:357` `PlaceOffsiteRestore` | | **Source** | a **completed full** offsite scratch (A.1.3 with `mode=full`) | | **Scope** | every captured path mapped back to its live location; the recovery unit is placed **only if the live unit is ABSENT** (`:352-356`). Named-volume tars are **not** unpacked; no DB is replayed. | | **Semantics** | **ADDITIVE / MISSING-ONLY** — `rsyncRestoreMissing`, explicitly never `rsyncMirror` (`--delete`), `offbox_restore.go:343-345` | | **Invocable by** | **CUSTOMER, UNASSISTED** | | **Prerequisites** | a completed *full* scratch (`OffboxFullScratchReady`, `:276`); whole-placement refusal on any path escaping the anchor (`mapOffsiteRestorePaths`, `:299-303`) — no partial writes | | **Proven** | **NOT PROVEN as a distinct action.** Campaign 8 lists *"the `place`/`reconstitute` legs that turn restored bytes back into a running app"* among the not-validated items (`CAMPAIGN-8…:520`). The C6 immich drill of 2026-07-20 that *is* cited as proven went through the reconstitute button (A.1.5), not this one. | ### A.1.5 — „Teljes visszaállítás (fájlok + adatbázis)" — offsite reconstitution | field | detail | |---|---| | **Name (UI, HU)** | **„Teljes visszaállítás (fájlok + adatbázis)"** / **„Teljes visszaállítás indítása"** | | **Entry point** | `POST /backup/offbox/reconstitute` (`app`, `confirm=1`) — form `backups_restore.html:121`; route `server.go:486-487`; handler `offbox_handlers.go:366`; engine `internal/backup/offbox_reconstitute.go:169` `ReconstituteFromOffsite` | | **Source** | the chosen offsite snapshot's coherent pair, out of the **scratch** (not the live unit) | | **Scope** | files **overwritten** to the snapshot's version + **that snapshot's DB dump replayed** into a DB-only bring-up + app restarted. **The unit itself is deliberately skipped** (`offbox_reconstitute.go:284-289`), so **named-volume tars are never unpacked by this path.** | | **Semantics** | **DESTRUCTIVE/REPLACING for files and the database**, but never deleting: *"NOTHING IS EVER DELETED. The file copy overwrites and adds; it never carries `--delete`"* (`:30-34`). | | **Undo** | a `pre-restore-*` safety dump of the live DB is taken **and verified present on disk** before anything stops; if it cannot be taken the operation refuses with zero changes (`:32-38`, `:257-268`). Fail-closed again if no DB service can be identified (`:272-276`). | | **Invocable by** | **CUSTOMER, UNASSISTED** | | **Prerequisites** | a completed **full** scratch; app deployed with a resolvable HDD path; explicit `confirm=1` (a stray click cannot fire it, `offbox_handlers.go:377-381`); 60-minute background context (`:388`) | | **Proven** | **PROVEN-LIVE (2026-07-20), operator-driven, through the customer's own buttons.** Photos deleted in immich's own UI **and trash emptied**, then restored: `40 file(s) placed`, 1 DB dump replayed rc-0, 11 assets active, `No schema drift detected`, timeline confirmed — `00-capability-map.md:66,75` citing `felhom-controller/REPORT.md` §4e. Residual recorded in the map itself: the run was performed by the **operator**, not a customer. | ### A.1.6 — „Megosztások visszaállítása" — shares restore (two-step) | field | detail | |---|---| | **Name (UI, HU)** | **„Megosztások visszaállítása"** then the place action | | **Entry point** | `POST /backup/shares/restore` (form `backups_restore.html:93`; route `server.go:493-494`; handler `offbox_handlers.go:498`; engine `internal/backup/shares_restore.go:87`) then `POST /backup/shares/place` (form `:97`; route `:495`; handler `:524`; engine `PlaceSharesRestore`) | | **Source** | the latest offsite `_shares` restic snapshot | | **Scope** | three things, in stated order of importance: **the files** (missing-only into each share's live folder), **the definitions** (merged back into the share registry), **the credential** (best-effort, into the samba named volume) — `shares_restore.go:20-23` | | **Semantics** | scratch step non-destructive; place step **ADDITIVE/MISSING-ONLY**, never overwriting | | **Guard** | a destination is written only if it resolves **strictly inside** a registered live storage root; a snapshot is treated as untrusted input; a `..` segment is refused outright rather than Clean'd away (`:25-28`, `:139-143`) | | **Invocable by** | **CUSTOMER, UNASSISTED** | | **Prerequisites** | offsite configured; a `_shares` snapshot; at least one live registered storage root | | **Proven** | **PROVEN-LIVE 2026-07-18** — `00-capability-map.md:96`: offsite snapshots `e0b9d723` and `4e2b15ec` carrying manifest + `passdb.tar`; the restore round-trip returned a deleted probe file byte-identical **and** a deleted share definition with its original flags, without overwriting live files. | ### A.1.7 — `.fab` bundle import | field | detail | |---|---| | **Name (UI, HU)** | the „Importálás" page (`/import`) | | **Entry point** | page `GET /import` (`server.go:526-527`, handler `internal/web/handler_export.go:115`); start `POST` → `apiImportStart` (`handler_export.go:327`) → `appexport.Exporter.StartImport`; status `apiImportStatus` (`:372`) | | **Source** | a `.fab` bundle discovered by `appexport.ScanForBundles` in the export dir of a **registered drive** (`internal/appexport/restore.go:41-50`); path validated by `isValidExportPath` before use (`handler_export.go:353`) | | **Scope** | config + DB dump + volume tars + the export's data mounts; optionally password-encrypted | | **Semantics** | **destructive re-import** of the bundled app (the P-FAB drill re-imported destructively) | | **Invocable by** | **CUSTOMER, UNASSISTED** — but the *upload* half is qualified below | | **Prerequisites** | the bundle must already be on a registered drive (or uploaded via the chunked upload path); the bundle password if encrypted | | **Proven** | **PROVEN-LIVE (partly).** `CAMPAIGN-6D` P-FAB / Accept #1: 1.7 GB full circle, byte-identical, app boots; chunking proven at the real Cloudflare edge (`00-capability-map.md:67`). **Still open in that same row:** *"the rendered browser file-picker upload leg is still Viktor's open full-circle test"*. | ## A.2 — Operator-only, via SSH / CLI (not reachable by any customer) ### A.2.1 — scheduled restore-test (unattended, agent) | field | detail | |---|---| | **Name** | restore-test (no customer-facing name; operator sees `restore_test_failed` / `restore_test_stale` on the hub) | | **Entry point** | scheduler inside the agent daemon; engine `felhom-agent/internal/reconcile/restoretest.go:126` `RunRestoreTest`; status readable at `GET /restore-test/status` on the per-guest local API (`internal/localapi/server.go:446`); manual form `felhom-agent --selftest=restore-test [-archive …]` (`cmd/felhom-agent/main.go:168`) | | **Source** | rotates **oldest-proven-first across every configured tier** (R-85): `local` (vzdump on `/var/lib/vz`) and `felhom-pbs` (the ep0 datastore over WireGuard) | | **Scope** | whole guest: restore into a scratch VMID in the `990000–990009` band → boot → verify running → **mount parity** against the archive's own config → teardown | | **Semantics** | **non-destructive to the live guest** — it restores into a *fresh* scratch guest and destroys it | | **Invocable by** | **automatic** (agent scheduler) or **operator via SSH** (`--selftest`). **Not customer-reachable at all.** | | **Prerequisites** | a candidate archive on the tier; the host-wide one-heavy-operation gate free (`internal/localapi/server.go:767`); scratch band free; teardown authority (see F-LEAK, Part G) | | **Proven** | **PROVEN-LIVE and currently running.** LIVE, `journalctl -u felhom-agent` on demo-hp: `18:08:33 backup: scheduled restore-test passed archive=local:backup/vzdump-lxc-9201-2026_07_28-17_43_05.tar.zst duration_s=89.204697113`. On demo-felhom, the **offsite** tier passed too: `Jul 28 00:04:27 … restore-test passed archive=felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z duration_s=1101.047742991`. LIVE `restore-test-state.json` shows both tiers tracked separately on both boxes. | ### A.2.2 — `pct restore` (manual whole-guest restore) | field | detail | |---|---| | **Name** | manual guest restore | | **Entry point** | `pct restore --storage --unprivileged 1`, per `documentation/runbooks/RUNBOOK-manual-guest-restore.md:80` | | **Source** | `local:backup/vzdump-lxc-9201-*.tar.zst` on the host, or `felhom-pbs:backup/ct/9201/` on ep0 | | **Scope** | whole guest: rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive`. **`mp8 /mnt/felhom-drives` and `mp9 /etc/felhom-bootstrap` are host binds and are NOT in the archive** (LIVE `pct config 9201`, both hosts; runbook `:34-39`). | | **Semantics** | **destructive/replacing** at guest granularity | | **Invocable by** | **OPERATOR VIA SSH ONLY** | | **Prerequisites** | root on the Proxmox host; the runbook's three pre-checks; the bind-strip-and-re-add step | | **Named hazard** | F-OPS: `mp9` embeds the **source** VMID, so a restore to a different VMID can bind **another guest's bootstrap credentials** (`RUNBOOK-manual-guest-restore.md:45-48`). The agent's own restore paths neutralise binds; the manual path does not (`:11-14`). | | **Proven** | **PROVEN-LIVE.** `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running + healthy), `00-capability-map.md:68`; and Campaign 8 Phase C restored a 5.82 GB vzdump into fresh CT 9299 and a 4.36 GB PBS snapshot into CT 9297, both with **exact mount parity** and `unprivileged: 1` preserved (`CAMPAIGN-8…:247-253`). Neither restored guest was booted, deliberately, because of F-OPS (`:589`). | ### A.2.3 — `proxmox-backup-client restore` (raw PBS restore) | field | detail | |---|---| | **Entry point** | invoked *by* `pct restore` from a `felhom-pbs:` volid — visible verbatim in a LIVE failure line on demo-felhom: `/usr/bin/proxmox-backup-client restore '--crypt-mode=encrypt' '--keyfd=13' ct/9201/2026-07-27T19:55:41Z root.pxar /var/lib/lxc/990000/rootfs --allow-existing-dirs --repository felhom@pbs!demo-felhom@10.77.0.1:felhom-offsite --ns demo-felhom`. Also used directly by the runbook to read `pct.conf` out of a snapshot without restoring (`RUNBOOK-manual-guest-restore.md:59`) | | **Source** | ep0 datastore `felhom-offsite`, per-customer namespace, over `wg-felhom` | | **Scope** | whatever archive is named (`root.pxar`, `pct.conf`, catalog) | | **Semantics** | destructive at the target path | | **Invocable by** | **OPERATOR VIA SSH ONLY** — needs the box's PBS token and the per-customer encryption key (`--crypt-mode=encrypt`) | | **Prerequisites** | WireGuard tunnel up to `10.77.0.1`; token valid; the encryption key present in `/etc/pve/priv/storage/felhom-pbs.enc` (or recovered from escrow) | | **Proven** | **PROVEN-LIVE** as part of A.2.1/A.2.2 above. | ### A.2.4 — `restic restore` (raw offsite restore) | field | detail | |---|---| | **Entry point** | the controller's own invocations only (`internal/backup/offbox.go:593-608`, env `RESTIC_PASSWORD_FILE=/offbox/repo_password`, mandatory `ConnectTimeout`). A human can run the same binary inside the guest with the same password file. | | **Source** | `sftp:u629488-sub1@u629488-sub1.your-storagebox.de:23/home/felhom-repo` (demo-felhom) / `u629488-sub3` (demo-hp) — LIVE from each guest's `settings.json` | | **Scope** | whatever the snapshot holds (see Part B) | | **Semantics** | non-destructive to a target dir | | **Invocable by** | **OPERATOR VIA SSH ONLY** for the raw form; the wrapped forms are A.1.3–A.1.6 | | **Prerequisites** | the repo password file, the SSH key, `known_hosts`, and — per C9-F3 — **no stale exclusive lock** | | **Proven** | proven through the wrapped path (A.1.3). Raw `restic restore` was also the manual remedy used in Campaign 9 Phase F for the lock (`restic unlock --remove-all`). | ### A.2.5 — agent DR bring-up (`--selftest=bring-up -mode dr`) | field | detail | |---|---| | **Name** | guest-loss DR bring-up | | **Entry point** | `felhom-agent --selftest=bring-up -mode dr -archive -vmid [-keep]` (`cmd/felhom-agent/main.go:168,171-172,174`); engine `internal/reconcile/bringup.go` (`ModeDRGuestLoss`, `:40`) | | **Source** | a customer backup archive (local or PBS) | | **Scope** | restore → **preserve continuity identity** (hostname, host keys, and by default MAC; reset only what collides, `:38-40`) → size → attach mounts → start link-up. Keeps the guest on success. | | **Semantics** | creates a new guest from an archive; destructive to the target VMID | | **Invocable by** | **OPERATOR VIA SSH ONLY** | | **Prerequisites** | archive; free VMID; restore storage; `-mode dr` requires the host-run seam (`:264`) | | **Proven** | **NEVER PROVEN.** Campaign 8's still-not-validated list: *"The agent's own DR bring-up path (`ModeDRGuestLoss`) — read in code, never executed"* (`CAMPAIGN-8…:522`). | ### A.2.6 — host-loss DR plan (derive-only — **executes nothing**) | field | detail | |---|---| | **Name** | host-loss restore plan | | **Entry point** | `felhom-agent/internal/dr/plan.go:45` `BuildRestorePlan`, driven by a `host_loss` `restore_directive` arriving on the desired-state channel (`:99-129`) | | **Source** | the hub's restore directive + the live DR recipe | | **Scope** | per guest: restore coordinates + sizing; per drive: `durable_id` → expected mount | | **Semantics** | **NONE — it is DERIVE-AND-SURFACE ONLY.** `plan.go:1-4`: *"the plan is logged (and exposed for the report), never executed … The Consumer has NO restore/destroy API by construction, so 'execute nothing' is a structural guarantee."* | | **Invocable by** | nobody executes it; the operator reads it | | **Proven** | unit-proven only (`06-offsite-connectivity.md:327`, S5 row). **See Part D2 — on the live fleet the inputs this plan consumes are empty or wrong.** | ### A.2.7 — escrow-driven secret recovery (the precondition for A.2.5/A.2.6) | field | detail | |---|---| | **Entry point** | `--selftest=escrow-consume -blob -fingerprint -keydest ` and `--selftest=identity-consume [-install-wg-key]` (`cmd/felhom-agent/main.go:186-189`); repo-password re-injection into a guest via `POST /backup/offbox/inject-password` (`server.go:503`, handler `offbox_handlers.go:189`, engine `internal/backup/offbox.go:541` `InjectOffboxPassword`) | | **Source** | the hub's `host_escrow` blob + the **customer's recovery code R** (never held by the operator or the hub) | | **Scope** | PBS client encryption key K; the identity bundle `{tunnel_token, pbs_token, wg_private_key, restic_repo_password}` (`felhom-agent/internal/escrow/identity.go:26-39`) | | **Invocable by** | **OPERATOR VIA SSH, with the customer present** to supply R | | **Proven** | ceremony + one-shot claim **PROVEN-LIVE** (`00-capability-map.md:38`). The **consume** side is proven at spike level (`documentation/tests/slice10d-identity-restore-spike-findings.md`); the destructive in-place drill is explicitly operator-gated and unrun (`06-offsite-connectivity.md:327`). | ## A.3 — Automatic self-healing (not restores, but recovery paths; recorded for completeness) | path | entry point | what it recovers | invocable by | proven | |---|---|---|---|---| | stale vzdump-lock recovery | `felhom-agent/internal/localapi/stalelock.go:62` `RecoverStaleLockedGuests` — **runs once at agent startup** | a guest left `backup`/`snapshot-delete` locked by a reboot mid-vzdump, plus a dangling `vzdump` snapshot | automatic | PROVEN-LIVE (TESTRUN F2-b lineage) | | guest-power watchdog | `felhom-agent/internal/localapi/guestpower.go` (60 s) | a guest that is `onboot:1`, stopped, unlocked, no vzdump in flight → started; bounded 3 attempts 1m/2m/4m | automatic | **PROVEN-LIVE 2026-07-28**: came back unattended in **120 s** against the **587 s** the original incident needed a human (`OPEN-ITEMS.md:38`) | | boot-orphan sweep + drive-backed gate | controller `bootrecon` | apps not started after a guest reboot; R-55 leaves deliberately-stopped drive-backed apps stopped | automatic | PROVEN-LIVE 2026-07-21 (`00-capability-map.md:70`) | | quiesce crash recovery | crash marker + `Recover()` | stacks left stopped by a SIGKILLed controller mid-quiesce | automatic | PROVEN-LIVE — recovered in **1 second** after a real SIGKILL; Campaign 8 corrects the spec's premise: *"`defer` played no part — it does not run on SIGKILL"* (`CAMPAIGN-8…:229-233`) | | restore-test scratch reaper | `reconcile.Engine.Recover` | a leaked scratch guest in the 990000 band | automatic | LIVE this session: `Jul 28 18:10:25 recover: destroyed leaked restore-test scratch guest op_id=scratch-restore-990000-9 vmid=990000` on demo-hp | ## A.4 — The "invocable by" summary (the point of Part A) | tier | customer-unassisted restore exists? | what it can return | |---|---|---| | **Tier-1** recovery unit | **YES** (A.1.1) | compose + DB dumps + **named volumes** — the only path that unpacks volume tars | | **Tier-2** cross-drive | **YES** (A.1.2) | **file legs only** — never the copy's own `recovery-unit/` (C9-F4) | | **Tier-3** restic offsite | **YES** (A.1.3–A.1.6) | files + DB replay; **never named volumes** (`offbox_reconstitute.go:284-289`) | | **Plane-2** whole-guest, local vzdump | **NO** | operator via SSH only | | **Plane-2** whole-guest, PBS offsite | **NO** | operator via SSH only | | **Host state** (Part D1) | **NO** | no restore path of any kind exists — see D1 | | **Hub state** (Part D2) | **NO** | Longhorn volume backup only, operator via `kubectl`/Longhorn UI | **INFERENCE, stated as such:** the customer can reach three tiers of app-level recovery and zero tiers of whole-guest or host recovery. Every path that returns a *whole system* is operator-only. --- # Part B — capture coverage per tier, per app ## B.1 — Catalog enumeration (SOURCE, computed this session) Method: parsed all 53 `templates/*/.felhom.yml` and `templates/*/docker-compose.yml` in `app-catalog-felhom.eu@4252121`, then applied the classifier's two-level default exactly as `internal/appbackup/classify.go` implements it (explicit `backup:` entry wins; else a `:ro` reader is **excluded**; else a writable bind is **mandatory**). ``` total templates 53 templates with a `backup:` block 13 templates with >=1 Docker NAMED volume 52 (only bentopdf has none) templates with a DB service (pg/mariadb/mysql/mongo) 14 templates binding ${HDD_PATH}/${USERDATA_PATH}/${IMPORT_PATH} 13 (exactly the block-carrying set) templates with such a bind and NO block 0 ``` Per-app classification of the 13 block-carrying apps (SOURCE, computed): | app | binds | mandatory | optional | excluded | |---|---|---|---|---| | audiobookshelf | 2 | 0 | 1 | 1 | | calibre-web | 1 | 1 | 0 | 0 | | emby | 1 | 0 | 0 | 1 | | immich | 2 | 1 | 1 | 0 | | jellyfin | 1 | 0 | 0 | 1 | | komga | 1 | 0 | 1 | 0 | | navidrome | 1 | 0 | 0 | 1 | | nextcloud | 1 | 1 | 0 | 0 | | paperless-ngx | 2 | 1 | 0 | 1 | | plex | 1 | 0 | 0 | 1 | | radarr | 2 | 0 | 0 | 2 | | romm | 2 | 0 | 1 | 1 | | sonarr | 2 | 0 | 0 | 2 | Derived counts (SOURCE): - **Tier-3 offsite** filters to `mandatory` only (`ComputeCaptureSet` with `TierOffsite`, `internal/appbackup/captureset.go:79`; offsite call site `internal/backup/offbox_capture.go:49`). **4 of 53** templates ever push a userdata path offsite: `calibre-web`, `immich`, `nextcloud`, `paperless-ngx`. The other 49 are **unit-only**, byte-identical to pre-v0.134.0 (`offbox_capture.go:30-31`). - **Tier-2** filters to `mandatory + optional` (`TierSecondary`, `internal/backup/tier2_capture.go:44`). **7 of 53** templates get a `hdd/` or `userdata/` leg: the 4 above plus `audiobookshelf`, `komga`, `romm`. The legacy resolver path (`tier2_capture.go:72-88`) adds legs for apps with an `appdata/` dir — **0 additional templates qualify**, because no non-block template binds a `${HDD_PATH}`/`${USERDATA_PATH}` path at all. - **6 of 13** block-carrying apps declare *only* excluded binds (`emby`, `jellyfin`, `navidrome`, `plex`, `radarr`, `sonarr`) — they carry a block and still get no automatic file leg on any tier. > **This does not reconcile with the shipped C9-F1 Phase-0 figure.** `felhom.eu/REPORT.md:17-24` and > `OPEN-ITEMS.md:31` record **A=9 / B=43 / C=1**. My independent enumeration gives **A=7 / B=45 / C=1** > by the same definition ("templates whose Tier-2 copy can hold a readable file leg"). The > difference is 2 apps and I could not identify them. Logged in **Contradictions** below; do not > treat either number as settled until the two methods are diffed. ## B.2 — What each tier captures, and where capture and restore disagree ### Tier-1 — recovery unit (per app, on the app's own drive) | | | |---|---| | **Location** | `/backups/primary//` | | **Captures** | `compose/docker-compose.yml`, `compose/.felhom.yml`, `compose/app.yaml` (**secret-stripped**), `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` — `recovery_unit.go:71-74,113-131` | | **Written by** | the nightly DB-dump leg and a periodic status refresh (`captureAllRecoveryUnits`, `recovery_unit.go:185`), **checksum-skipped** when nothing changed (`:76-77`) | | **Restore reads** | all of it (A.1.1) | | **Capture ⟷ restore disagreement** | **none in scope**, but one in *validation*: `CaptureRecoveryUnit` gates its write on a **config checksum change alone** — no run-state check, no health check, no test that the pinned images resolve (`recovery_unit.go:78-130`; Campaign 9 O-1, `CAMPAIGN-9…:538-584`). `ListRestorePoints` returns exactly **one** keep-side restore point per app (`restore_points.go:14-18`), so an unvalidated refresh replaces the only one there is. Campaign 9 measured the compensating control live: the catalog git-sync reverted the drift in **~3 minutes** and the unit followed. | ### Tier-2 — cross-drive secondary copy | | | |---|---| | **Location** | `/backups/secondary//` with the v2 layout: `.felhom-tier2-layout`, `recovery-unit/`, `hdd//`, `userdata//` (`07-backup-architecture.md:288-294`; writer `tier2.go:351-393`) | | **Captures** | **always** a full mirror of the recovery unit (`tier2.go:368-369`, comment "Unit leg (always)"; OPEN-ITEMS cites this as `tier2.go:369`) **plus** the `TierSecondary` capture legs | | **Restore reads** | `hdd/` and `userdata/` **only** (`tier2_restore.go:101-104`) | | **Capture ⟷ restore disagreement** | **This is the asymmetry.** Two distinct defects live here: **C9-F1** (shipped-fixed v0.183.0) — for the 45-of-53 apps with no leg, the button restored 0 files, took a real outage, and reported „Nincs hiányzó fájl — minden fájl megvan a helyén."; and **C9-F4** (OPEN) — *"Nothing reads the Tier-2 copy's `recovery-unit/` mirror"* (`OPEN-ITEMS.md:35`). `RecoveryUnitPath` resolves to `backups/**primary**/` (`appbackup/paths.go:46-48`) and the only reader of the secondary tree is `tier2_restore.go:79`. **Tier-2 exists for the case where the primary drive is lost — and in exactly that case the primary unit is gone while this mirror survives, unreachable by any customer action.** | | **LIVE confirmation of the asymmetry** | demo-felhom `backups/secondary/bookstack/` and `.../docmost/` contain **`recovery-unit` and nothing else** (no `hdd/`, no `userdata/`) at **156 MB** and **86 MB** respectively; `calibre-web` has `recovery-unit userdata`; `immich` has `hdd recovery-unit`. demo-hp `paperless-ngx` has `hdd recovery-unit`. | ### Tier-3 — restic offsite | | | |---|---| | **Location** | `sftp:u629488-sub@u629488-sub.your-storagebox.de:23/home/felhom-repo`, one multi-path snapshot per app per run, tagged `felhom-offbox` + `` | | **Captures** | the recovery unit (which itself contains DB dumps **and named-volume tars**) + `mandatory` binds only (`offbox_capture.go:28-31,49`); plus a separate `_shares` snapshot | | **Restore reads** | scratch restore reads whatever the snapshot holds (A.1.3); **place** and **reconstitute** both **skip the unit** when writing to live — `PlaceOffsiteRestore` places it only if the live unit is *absent* (`offbox_restore.go:352-356`), and `ReconstituteFromOffsite` skips it outright (`offbox_reconstitute.go:284-289`) | | **Capture ⟷ restore disagreement** | **The named-volume tars are captured on every offsite run and unpacked by no offsite action.** The DB *is* replayed (from the scratch unit's `db-dumps/`, `offbox_reconstitute.go:315`), but the volume tars are not. Since 52 of 53 templates keep data in named volumes (B.1), for most apps the offsite tier carries the volume data and cannot itself put it back. A **two-step** route exists — full restore → place (which lands the unit if absent) → then the Tier-1 restore, which does unpack tars — but **no single action does it and no UI routes it** (the C9-F1b shape, one tier over). This is my own enumeration; it is **not** currently filed as a finding. | | **Silent-gap guard that IS present** | a declared-but-absent mandatory path is stat-filtered *before* argv, because restic 0.14.0 skips a missing source with a stderr warning and **exits 0** (`07-backup-architecture.md:110-118`; implemented `offbox_capture.go:61-70`) | ### Plane-2 — whole-guest (local vzdump **and** PBS): do they differ? **They differ in three ways, all verified LIVE.** | | local (`local`, `/var/lib/vz/dump`) | offsite (`felhom-pbs` → ep0 `felhom-offsite`) | |---|---|---| | **cadence** | `backup_cadence_seconds: 0` ⇒ **24 h** | `604800` = **7 days** (LIVE, `/etc/felhom-agent/agent.json` `backup.backup_targets[]`, identical on both hosts) | | **retention** | `local_backup_retention: 3` | `keep_last: 0` box-side; retention is **server-side** on ep0 (LIVE `proxmox-backup-manager prune-job list`: `prune-demo-felhom` and `prune-demo-hp`, both `03:30`, `keep-last 2`) | | **encryption** | none — plain `.tar.zst` on the host | client-side, per-customer key (`encryption-key` in `/etc/pve/storage.cfg`, distinct per host) | | **transport** | local disk write | WireGuard `10.77.0.1` | | **verification** | none | `verify-new 1` on the datastore + the agent's own `VerifyLoop` (6 h, `internal/pbs/verify.go:13`) | | **contents** | **identical** — both are `vzdump` of the same guest | same | **Contents, identical for both and verified LIVE against `pct config 9201` on both hosts:** ``` rootfs local-lvm:vm-9201-disk-0 IN mp0 local-lvm:vm-9201-disk-1 mp=/var/lib/docker backup=1 IN mp1 local-lvm:vm-9201-disk-2 mp=/mnt/sys_drive backup=1 IN mp8 /mnt/felhom-drives mp=/mnt/felhom-drives OUT (host bind) mp9 /var/lib/felhom-agent/guests/9201/bootstrap ro=1 OUT (host bind) ``` **Consequence, stated as fact:** the whole-guest tiers carry `/var/lib/docker` — which is where the Docker named volumes and the controller's own data volume live — and **do not carry the customer's data drives at all**. On demo-felhom that is the 916 GB `hdd_1`; on demo-hp the 938 GB `nvme-1tb`. ## B.3 — Approximate data volume per app class, on the live fleet (LIVE) All figures from `du -sh` inside guest 9201 on each host, 2026-07-28 ~17:44 UTC. ### demo-felhom (namespace roots: `/mnt/felhom-drives/hdd_1` and `/mnt/sys_drive/felhom-data`) | item | size | |---|---| | `hdd_1/appdata/immich` | **252 M** | | `hdd_1/appdata/nextcloud` | 63 M | | `hdd_1/appdata/paperless` | 24 K | | `hdd_1/appdata/romm` | 12 K | | `hdd_1/userdata/media` | 472 K | | `hdd_1/userdata/{documents,downloads,roms}` | 4 K each | | `hdd_1/shares/Share` | 28 K | | **Tier-1 units** `hdd_1/backups/primary/immich` | **1.3 G** | | **Tier-1 units** `hdd_1/backups/primary/calibre-web` | 388 K | | **Tier-1 units** `sys_drive/…/primary/bookstack` | **157 M** | | **Tier-1 units** `sys_drive/…/primary/docmost` | **88 M** | | **Tier-1 units** — 19 further apps | 24 K each (config-only; never deployed) | | **Tier-2** `hdd_1/backups/secondary/bookstack` | **156 M** (recovery-unit ONLY) | | **Tier-2** `hdd_1/backups/secondary/docmost` | **86 M** (recovery-unit ONLY) | | **Tier-2** `sys_drive/…/secondary/immich` | **1.6 G** (`hdd` + `recovery-unit`) | | **Tier-2** `sys_drive/…/secondary/calibre-web` | 844 K (`recovery-unit` + `userdata`) | | **Tier-2** `sys_drive/…/secondary/_shares` | 884 K | | **Tier-3** restic repo | **1,103,918,002 B = 1.0 GB**, 32 snapshots (from `settings.json`) | | **Plane-2 local** | 3 archives; newest **5,929,975,285 B** (15 GiB written → 5.52 GB compressed) | | **Plane-2 PBS** | 3 snapshots in ns `demo-felhom`, oldest `2026-07-27T17:34:09Z` | | guest `/var/lib/docker` used | **12 G of 197 G** | | guest `/mnt/sys_drive` used | 1.8 G of 50 G | | data drive `hdd_1` used | **3.4 G of 916 G** | ### demo-hp (namespace roots: `/mnt/felhom-drives/nvme-1tb` and `/mnt/sys_drive/felhom-data`) | item | size | |---|---| | `nvme-1tb/appdata/paperless` | **8.6 M** | | `nvme-1tb/userdata/media` | 36 K | | **Tier-1** `nvme-1tb/backups/primary/paperless-ngx` | **78 M** | | **Tier-1** `sys_drive/…/primary/paperless` | 680 K | | **Tier-1** `sys_drive/…/primary/uptime-kuma` | 24 K | | **Tier-2** `sys_drive/…/secondary/paperless-ngx` | **86 M** (`hdd` + `recovery-unit`) | | **Tier-3** restic repo | **38,041,506 B = 36.3 MB**, 8 snapshots | | **Plane-2 local** | 3 archives; newest **1,679,958,230 B** (4.4 GiB written → 1.56 GB compressed) | | **Plane-2 PBS** | 6 snapshot dirs in ns `demo-hp`, one of which is the R-99 phantom (below) | | guest `/var/lib/docker` used | 5.3 G of 50 G | | data drive `nvme-1tb` used | **89 M of 938 G** | **ep0 (LIVE):** datastore `felhom-offsite` on a dedicated 98 G volume, **14 G used, 14 %**; per-namespace `du`: `demo-felhom` 568 K, `demo-hp` 444 K (index metadata only — the chunks are shared in `.chunks/`). The 13 G rollback copy `/srv/pbs-felhom` is present on the 38 G root disk (55 % full). **Fleet reality check, INFERENCE:** the largest single app-data object on the fleet today is immich's 252 MB appdata and its 1.3 GB Tier-1 unit. **Nothing on the live fleet exercises the WAN-restore regime the architecture must answer for** — there is no multi-hundred-GB media library anywhere, and both data drives are >99 % empty. Every measured timing in Part F is therefore a small-data timing. --- # Part C — keys, secrets, and what recovers them **A copy you cannot decrypt is not a copy.** Locations, counts, and recovery routes only; no values. ## C.1 — Inventory | # | secret | protects | where it lives (LIVE unless noted) | independent copies | recovers it if primary is lost | exercised? | |---|---|---|---|---|---|---| | 1 | **restic repo password** | **all Tier-3 offsite ciphertext** | guest `…/felhom-controller-data/_data/data/offbox/repo_password`, 64 B, `0600` root — LIVE on both boxes. Auto-generated once, 256-bit hex, never logged (`offbox.go:392-400,405-410`) | **3 conceptually**: the live file · inside every whole-guest archive (it sits under `/var/lib/docker` = mp0, `backup=1`) · **inside the hub's `host_escrow.identity_blob`** as `IdentityBundle.ResticRepoPassword` (`felhom-agent/internal/escrow/identity.go:34-38`). Plus a hash-only copy on the hub: `host_escrow.restic_pw_sha256`, **present for both hosts** (LIVE) | whole-guest restore, **or** escrow-consume with the customer's R, **or** `POST /backup/offbox/inject-password` (`offbox.go:541`) | escrow **creation** proven live; **consume never exercised destructively** | | 2 | **PBS datastore encryption key K** | all Plane-2 offsite ciphertext | host `/etc/pve/priv/storage/felhom-pbs.enc`, 255 B, `0600 root:www-data` — LIVE on both hosts; its fingerprint is also in `/etc/pve/storage.cfg` | 2: the live file · the hub's `host_escrow.blob` (383 B on both hosts, posture `zero_knowledge`, LIVE) | escrow-consume with R (`--selftest=escrow-consume -keydest …`) | ceremony proven; consume proven at spike level only | | 3 | **PBS access token secret** | write access to the box's own namespace | host `/etc/pve/priv/storage/felhom-pbs.pw`, 37 B, `0600 root:www-data` — LIVE both hosts | 3: live file · `IdentityBundle.PBSToken` in escrow · the hub can **re-mint** (`host_pbs_secrets`, LIVE: one consumed row per host, `generation 1`) | hub re-issue → descriptor `secret_generation` bump → agent re-applies | **PROVEN-LIVE 2026-07-21**, operator click to converged in **13 s** (`00-capability-map.md:39`) | | 4 | **agent's hub API key** | the box's identity to the hub | host `/etc/felhom-agent/agent.json`, `0600 felhom-agent` — LIVE both hosts; also `hosts.api_key` on the hub | 2 | hub-side re-enrolment | not exercised as a recovery | | 5 | **agent TLS key for the local API** | controller↔agent channel | host `/var/lib/felhom-agent/local-api.key`, 227 B, `0600` — LIVE | 1 (+ the pinned cert in the guest bootstrap) | regenerate + re-pin | n/a | | 6 | **guest bootstrap token** | the controller's hub identity + local-API token | host `/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json`, 380 B (demo-felhom) / 373 B (demo-hp), `0600` uid 100000 — LIVE; bind-mounted `ro` into the guest at `/etc/felhom-bootstrap` | 1 **on the host only** — this is `mp9`, which is **NOT in any guest archive** | re-provisioning back-half writes it | never rebuilt from nothing | | 7 | **WireGuard private key (box)** | the offsite transport | host `/etc/wireguard/wg-felhom.conf`, 327 B, `0600 root` + `/var/lib/felhom-agent/wg/` — LIVE both hosts | 2: live · `IdentityBundle.WGPrivateKey` in escrow (`identity.go:29-32`) | escrow-consume `-install-wg-key` (create-only), **or** fresh-key re-registration keeping the `/32` (`06-offsite-connectivity.md:327`) | unit-proven | | 8 | **offbox SSH key** | reaching the Storage Box | guest `…/data/offbox/ssh_key`, 400 B, `0600` — LIVE both boxes; `known_hosts` 290 B `0644` beside it | 2: live · inside whole-guest archives | hub re-provisions the sub-account credential | Hetzner sub-account provisioning PROVEN-LIVE (`00-capability-map.md:129`) | | 9 | **app.yaml encryption key** | **every per-app secret** | guest `…/data/encryption.key`, **32 B**, `0600` — LIVE both boxes; created at first boot (`internal/crypto/crypto.go:20-39`) | 2: live · inside whole-guest archives | **whole-guest restore only** | never exercised as a recovery | | 10 | **per-app secrets** (`DB_PASSWORD`, `SECRET_KEY`, `APP_SECRET`, …) | app data + app login | guest `/opt/docker/stacks//app.yaml`, AES-256-GCM `ENC:` under #9 | 1 logical copy, protected by #9 | recovered from the guest at restore time (`restore_unit.go:132`); **regenerated only for resettable secrets**, never for data keys | partially — data-key gate never fired in anger | | 11 | **dashboard password hash** | the whole customer UI | guest `…/data/settings.json` → `password_hash` (bcrypt) with `controller.yaml` → `web.password_hash` as fallback (`auth.go:34-35`) | 2 + archives | hub-issued **reset code** — `POST /claim/request-new-code` | **PROVEN-LIVE**: hash applied 1 s after request, code accepted first try (`00-capability-map.md:101`) | | 12 | **household SMB password** | the „Megosztás" shares | never persisted as plaintext (STDIN→`smbpasswd`); the passdb lives in the samba named volume; a copy is staged at guest `…/data/shares-payload/passdb.tar` — **LIVE, 855,040 B, `0600`, on both boxes** | 3: samba volume · `shares-payload/passdb.tar` · the offsite `_shares` snapshot | shares place-to-live restores the credential best-effort (`shares_restore.go:20-23`) | **PROVEN-LIVE 2026-07-18** | | 13 | **launcher share token + its bcrypt password hash** | the `/s/` guest launcher | `settings.json` → `launcher_share_token`, `launcher_share_password_hash` — LIVE, present on demo-felhom only | 1 + archives | rotate (the customer can) | n/a | | 14 | **`web.session_secret`** | session + CSRF + the share cookie HMAC | `controller.yaml` — LIVE both boxes | 1 + archives | regenerate (invalidates sessions and share cookies) | n/a | | 15 | **Cloudflare tunnel token + CF API token** | public reachability of every app | `controller.yaml` `infrastructure.cf_tunnel_token` / `cf_api_token` — LIVE both boxes; **also on the hub** in `customer_configs.config_json.infrastructure` (LIVE, both keys present for demo-felhom) | 2 + archives | the hub re-serves them on config pull | continuously exercised | | 16 | **`root@pam` console password (break-glass)** | PVE web console when SSH is dead | **hub only** — `host_recovery` table, LIVE: 3 rows, all `root@pam`, 32-byte secrets, `set_at` 2026-07-18/07-21/07-25 | 1 (hub) + whatever the operator saved out-of-band | `GET /api/v1/admin/hosts//recovery-credential` with the **global operator key** (`break-glass.md:42-48`) | PROVEN-LIVE (the sshd incident) | | 17 | **hub global bearer (`REPORT_API_KEY`)** | operator-tier hub API | k8s `Secret/report-api` + the operator's password manager | 2 | recreate from the out-of-band store | rotation runbook written, **not yet run** (`secrets.md:107-117`); **the git-history copy stays alive until rotated** (`:76-77`) | | 18 | **Resend API key** | all email — every notification and every claim/reset code | k8s `Secret/resend-api` + password manager + Gmail "Send mail as" | 3 | recreate from the store | rotation procedure written and ordered | | 19 | **Hetzner API token + pool box id** | Storage Box provisioning | k8s `Secret/storagebox` (`HETZNER_TOKEN`, `HETZNER_POOL_BOX_ID`) — LIVE from the hub Deployment env | 1 + password manager (assumed) | operator re-mints at Hetzner | n/a | | 20 | **three SSH private keys held by the hub** | ep0 peer-sync, tenantsync, and the desired-state poke | hub pod files (`WG_ENDPOINT_SSH_KEY_FILE`, `TENANTSYNC_SSH_KEY_FILE`, `POKE_SSH_KEY_FILE`) + `Secret/wg-endpoint-ssh` for the pinned host key — LIVE | 1 each | re-provision from the offsite-endpoint runbook | n/a | | 21 | **operator OOB SSH pubkey** | operator access to `felhom-sshd` on every box | `hub_settings.oob_operator_ssh_pubkey` (LIVE) → pushed to `/var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op` (93 B, LIVE both hosts) | 2 | hub re-push | continuously exercised | | 22 | **registry credentials** | pulling controller/agent images | `Secret/gitea-creds` | 1 | re-mint in Gitea | n/a | | 23 | **customer retrieval password** | pulling `controller.yaml` from the hub | `customer_configs.retrieval_password` on the hub; `settings.json.retrieval_password` in the guest | 2 | operator resets on the hub | exercised at every day-0 | | 24 | **the customer's recovery code R** | **unwraps #1, #2, #3, #7** | **nowhere in the system, by design** — the operator does not have it, the hub does not have it, the box does not retain it | **0 system copies** | nothing. If the customer loses R, the escrow blobs are unopenable. | the one-shot claim is PROVEN-LIVE; loss has never been exercised | ## C.2 — Secrets with exactly one copy | secret | why it is single-copy | consequence | |---|---|---| | **#6 guest bootstrap token** (`mp9`) | it is a **host bind mount** and therefore in **no guest archive** — verified LIVE against `pct config 9201` on both hosts | a whole-guest restore onto a fresh host produces a guest whose bootstrap dir must be rebuilt by the provisioning back-half; the manual runbook warns never to copy one from another guest because it carries that customer's tokens (`RUNBOOK-manual-guest-restore.md:97-100`) | | **#5 agent local-API TLS key** | regenerable, but the guest pins it | a regeneration requires re-pinning in the guest bootstrap | | **#16 break-glass console password** | hub-only by design (`break-glass.md:89`: *"The password is never logged or filed — it exists only in the hub vault"*) | **if the hub is gone, the break-glass password is gone** — see C.3 | | **#24 recovery code R** | zero system copies by design | irreducible; the premium operator-custody tier is the stated alternative (`05-hub-architecture.md:192`) | | **#19–#22 hub-held operator secrets** | one k8s Secret each, backed up only via the hub PVC (Part D2) | recreate from the out-of-band store; see Part E | ## C.3 — Circular dependencies (a recovery route that depends on something it also protects) Three found. All three are **SOURCE/LIVE-verified**, not inferred from docs. **C-1 — App-level restore cannot reconstitute an app without the guest that was lost.** Every tier's recovery unit is **secret-free by design** (`recovery_unit.go:73`: *"It NEVER writes a secret value"*), and `RestoreFromRecoveryUnit` recovers secrets **from the guest's live `app.yaml`** (`restore_unit.go:130-132`), which is encrypted under `encryption.key` — a 32-byte file that exists only inside the guest (LIVE) and only in whole-guest archives. `reconcileRestoreSecrets` states the consequence outright (`restore_unit.go:18-22`): a missing data-encrypting key is **FATAL** and *"we refuse and tell the operator to do a PBS whole-guest restore."* **So Tier-1, Tier-2 and Tier-3 are all conditioned on the whole-guest tier having survived.** They are app-recovery tiers, not system-recovery tiers, and the code says so. **C-2 — The offsite tier's own key rides only in tiers the same disaster removes.** The restic repo password (#1) has three copies: the guest file, the whole-guest archives, and the escrow blob. The first two are lost with the guest and the host. **The escrow blob is the only copy that survives a host loss — and opening it requires the customer's R (#24), which the system holds zero copies of.** So offsite-only recovery is possible exactly when the customer can produce R, and impossible otherwise, regardless of how healthy the ciphertext is. This is the intended zero-knowledge posture; it is recorded here because it is the recovery path's actual precondition and it has **never been exercised end to end** (A.2.7). **C-3 — The break-glass credential is protected by the plane it exists to rescue.** `host_recovery` lives **only** in the hub DB (LIVE: 3 rows). Retrieving it needs an authenticated call to `hub.felhom.eu` with the operator's global key (`break-glass.md:42-48`). The hub runs on DooPlex; `hub.felhom.eu` resolves to DooPlex's own home IP (Part E). **If DooPlex is down, the break-glass password for every customer host is unreachable** — including in the scenario where you need it because a host's SSH is dead. The runbook's own §5 note records the historical instance of this shape: *"A working break-glass path independent of sshd must always exist"* (`:73-77`) — the same argument applies one level up, to the hub. --- # Part D — host state and hub state ## D1 — What lives on the host and is in no guest backup Everything proven so far restores a **guest**. The following was inventoried LIVE on **both** hosts. None of it is inside any guest archive: `mp8` and `mp9` are host binds and out of `vzdump` scope, and nothing else on the host root filesystem is captured by anything at all. | # | host artefact | LIVE evidence | captured anywhere? | reconstructible from golden + provisioning? | |---|---|---|---|---| | 1 | agent binary + `felhom-agent.service`, `.service.d/`, `felhom-agent-rollback.service` | `/etc/systemd/system/` — 9 felhom units on demo-felhom, 11 on demo-hp | **NO** | yes — host-install + agent self-update A/B slots | | 2 | **`/etc/felhom-agent/agent.json`** (hub key, PBS/backup config, WG, escrow, oob, privileged) | 2465 B (demo-felhom) / 2459 B (demo-hp), `0600 felhom-agent`; **plus 6 ad-hoc `.bak`/`.campaign*` copies on each host** | **NO** | partially — the hub re-serves desired state, but `hub.api_key` is identity | | 3 | **`bootstrap.json`** (`mp9` source) | `/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json`, 380 B / 373 B, `0600` uid 100000 | **NO** — explicitly out of every archive | yes, by the provisioning back-half; **never** by a restore | | 4 | **island bridge `vmbr9`** + `/etc/network/interfaces` | both hosts carry `auto vmbr9 / 169.254.253.1/30 / bridge-ports none`; guest `net1` is `169.254.253.2/30` | **NO** | yes — `RUNBOOK-island-migration.md`, host-install v1.19.0+ | | 5 | **sudoers grants** | `/etc/sudoers.d/felhom-agent` **21,833 B** and `/etc/sudoers.d/felhom-op` 1,241 B, `0440`, identical byte-size on both hosts | **NO** | yes — host-install writes them; v1.21.0 added the `990000..990009` band | | 6 | **PBS storage config** `/etc/pve/storage.cfg` + `/etc/pve/priv/storage/felhom-pbs.{enc,pw}` | LIVE; the `.enc` is the per-customer client key | **NO** (`/etc/pve` is not in any guest archive) | key only via **escrow + R**; the descriptor via hub desired-state | | 7 | **WireGuard** `/etc/wireguard/wg-felhom.conf` | 327 B `0600` on both hosts | **NO** | key via escrow; peer entry survives on the hub (`wg_peers`, LIVE 4 rows) | | 8 | `onboot` flags | LIVE `onboot: 1` on both 9201s | inside the archived `pct.conf` | yes | | 9 | cron/systemd units for the mgmt-plane watchdog | `felhom-mgmt-watchdog.{service,timer}`, `felhom-privsep.tmpfiles`, `felhom-oob-nft.service`, `felhom-shared-parent.service`, `felhom-sshd.service` | **NO** | yes — host-install | | 10 | agent state dir `/var/lib/felhom-agent/` | LIVE: `journal.log` (9.7 K / 15.2 K), `restore-test-state.json`, `guest-binds.json`, `drive-intents.json`, `format-job.json`, `controller-swap-9201.json`, `local-api.{crt,key}`, `local-tokens.log`, `nonces.log`, `pbsdr/marker.json`, `units/*.mount`, `felhom-sshd/{authorized_keys.felhom-op,port,sshd_config}`, `wg/` | **NO** | mostly regenerable; `local-api.key` and the pinned cert are not (C.2) | | 11 | `/var/lib/felhom-install/state.json` | present on both hosts | **NO** | the `break_glass` marker matters — a from-scratch reinstall wipes it and re-randomises `root@pam` (`break-glass.md:83-88`) | | 12 | the local vzdump archives themselves | `/var/lib/vz/dump` | **they are the backup**; nothing backs them up | n/a | | 13 | `/mnt/felhom-drives/*` — **the customer's data drives** | 916 G `hdd_1` / 938 G `nvme-1tb`, bound via `mp8` | **NOT in any whole-guest archive**; only their felhom-managed subtrees ride Tier-2/Tier-3 | the drives are physical | **Has a host ever been rebuilt from nothing and verified?** **A host has been *installed* from nothing many times, on two different physical boards. A host has never been *rebuilt* from nothing — i.e. no host has ever been reconstructed as its former self, with its identity, keys and guests, and then verified.** The distinction is exactly the provision-vs-DR split in `bringup.go:36-40`, and only the `ModeProvision` half has ever run. Evidence for both halves: - *Install from nothing:* PROVEN-LIVE on two boards — N100 2026-07-18 and HP t740 2026-07-21, each a single unattended pass from a virgin ISO to a claimed, running box (`00-capability-map.md:34,37`). - *Rebuild as its former self:* `ModeDRGuestLoss` **"read in code, never executed"** (`CAMPAIGN-8…:522`); the destructive S5 host-loss drill is operator-gated and unrun (`06-offsite-connectivity.md:327`); and the host-loss plan builder executes nothing by construction (`dr/plan.go:1-4`). ## D2 — What the hub holds that exists nowhere else Hub DB: `/data/hub.db`, **113,033,216 B**, on PVC `hub-data` (1 Gi, Longhorn, RWO, bound 162 d). Row counts from the 17:40:00 UTC copy: ``` customer_configs 4 hosts 3 guests 5 host_escrow 2 host_escrow_superseded 0 host_recovery 3 host_pbs_secrets 2 dr_recipe 6 wg_endpoints 1 wg_peers 4 events 1617 host_reports 1760 reports 8010 notification_log 314 one_time_secrets 3 signed_jobs 0 hub_settings 9 appliance_registrations 8 selfbind_tokens 2 customer_claims 4 ``` ### D2.1 — What exists nowhere else | hub-only state | why it is hub-only | LIVE detail | |---|---|---| | **`host_escrow`** — the R-wrapped PBS key + identity bundle | the box produces it and then **cannot open it**; the hub is the custody store | 2 rows (`demo-felhom-8363b5` 2026-07-21, `demo-hp-bb76ea` 2026-07-23), both `posture=zero_knowledge`, blob 383 B, identity blob 572 B, `restic_pw_sha256` present, `stale_at` NULL | | **`host_recovery`** — break-glass `root@pam` per host | never logged or filed anywhere else by design | 3 rows | | **`host_pbs_secrets`** — the one-time PBS token secret | mint-and-consume | 2 rows, both consumed, `generation 1` | | **`customer_configs`** — identity, domain, email, retrieval password, **`config_json` incl. both Cloudflare tokens and the full offsite descriptor**, `min_controller_version`, `config_version`, `dr_tier` | the box only ever receives a rendered copy | 4 customers: `demo-felhom` (dr_tier 1, cfg v12), `peti-felhom` (dr_tier 0, v6), `demo-hp` (dr_tier 1, v3), `drill-r50` (blocked, v1) | | **`hosts.desired_json` + `desired_generation`** — operator intent | authored here | generations 8 / 5 / 1 | | **`wg_endpoints` + `wg_peers`** — the offsite peer registry | the endpoint is dumb and hub-driven (`06-…:34`) | endpoint `ep0` `ep0.felhom.eu:443`, subnet `10.77.0.0/24`, pbs tunnel ip `10.77.0.1`; peers `.2` demo-felhom, `.3` demo-hp, `.4` drill-r50, `.250` operator-oob | | **`hub_settings`** — fleet floor + signed artifact digests + operator password hash + operator OOB pubkey | the trust root the host bootstrap verifies against | `min_controller_version 0.156.0`, `artifact_golden_version 0.161.0`, `artifact_agent_version 0.96.0`, `artifact_min_agent 0.93.0` | | **`events` / `notification_log` / `host_reports` / `reports`** — the operator's only fleet-wide history | prunable by age, and stated to be *"convenience, not the DR source of record"* (`05-hub-architecture.md:170-171`) | 1617 / 314 / 1760 / 8010 rows | | **`customer_claims`, `selfbind_tokens`, `appliance_registrations`, `one_time_secrets`** | onboarding lifecycle | 4 / 2 / 8 / 3 | ### D2.2 — Is it backed up? (LIVE) **Yes, thinly, and to the same machine.** ``` PVC hub-data -> volume pvc-486c9809-4672-4b56-b70e-0bf01d0c3628 labels: recurring-job-group.longhorn.io/default = enabled <- in scope recurringjobs: backup-daily 0 4 * * * RETAIN 1 backup-weekly 0 5 * * 0 RETAIN 1 backuptarget "default": nfs://192.168.0.180:/mnt/5_hdd/backup/longhorn-pvc available=true backups of hub-data actually present: 2 2026-07-26T03:06:48Z Completed 425,721,856 2026-07-28T02:07:11Z Completed 425,721,856 ``` Facts that follow, stated plainly: 1. **The backup target is DooPlex itself.** `192.168.0.180` is the machine running k3s, the hub, and the Longhorn replicas. The replica data is on `/dev/sdb1` (`/var/lib/longhorn`, the 477 G root SSD, **81 % full**); the backup target is on `/dev/sda1` (`/mnt/5_hdd`, a 9.1 T HDD). Different physical disks, **same host, same building, same power, same operator uplink**. 2. **Exactly two restore points exist**, 2 days apart, because both recurring jobs use `RETAIN 1`. 3. There is **no off-machine copy of the hub DB** that I could find. Searched: Longhorn `backuptargets` (one, NFS to DooPlex), Longhorn `recurringjobs` (two, both `default` group), the hub Deployment's volumes, and `documentation/runbooks/` for any hub-DB export procedure. ### D2.3 — Two hub-held DR records that are empty or wrong on the live fleet Both are LIVE reads from the hub DB copy, cross-checked against ep0. **D2.3-a — `hosts.dr_record_json` is `{}` for every host.** ``` demo-felhom-8363b5 {} demo-hp-bb76ea {} drill-r50-0a4f9a {} ``` `05-hub-architecture.md:175-176` describes this column as *"a **slim DR record** on the `hosts` row (PBS namespace + repo fingerprint + the wrapped escrow key)"* and `:186` names it as one of the four durable sources host-loss recovery reads from. On the live fleet it holds nothing. The escrow itself *is* stored — in `host_escrow`, not here — so this may be a doc-vs-implementation drift rather than data loss; it is recorded as a fact, not diagnosed. **D2.3-b — `host_escrow.directive_json` is `{}` for both escrowed hosts.** The non-secret DR directive (PBS repo/ns, expected fingerprint, tunnel id) that `documentation/architecture/06-offsite-connectivity.md:148-150` says was uploaded with the escrow is 2 bytes — an empty object — on both rows. **D2.3-c — the DR recipe's PBS namespace is wrong on every box.** ``` dr_recipe.host_half.pbs (demo-felhom, updated 2026-07-28 17:31:13) { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" } dr_recipe.host_half.pbs (demo-hp, updated 2026-07-28 17:25:31) { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" } ``` `felhom-agent/internal/hub/dr_recipe.go:55` documents that field as *"PBS namespace the restore targets"*. The **real** namespaces, verified LIVE on ep0, are `demo-felhom` and `demo-hp` (`/mnt/pbs-datastore/ns/{demo-felhom,demo-hp}`), and the live `pct restore` command line quoted in A.2.3 carries `--ns demo-felhom`. Traced to source: `Snapshot.Namespace` is decoded from a `ns` JSON field (`internal/pbs/client.go:97`) that PBS does not echo per item when the request is already namespace-scoped via `?ns=` (`:117-120`), so the value is empty and `ToHub` normalises empty to `"root"` (`internal/pbs/report.go:23-27`), which `latestPBSCoord` then writes into the recipe (`dr_recipe.go:149`). `latest_snapshot_id: "9201"` is **not** a defect — it is `BackupID`, documented as *"a coordinate"* (`dr_recipe.go:56`). **D2.3-d — the DR recipe's drive list is empty on every box, including boxes with enrolled drives.** `"drives": []` for demo-felhom, demo-hp, peti-felhom and drill-r50. `BuildDRRecipeHostHalf` populates it from storage targets where `isUserDataDrive(t)` holds — `type` must be `usb` or `local-dir` **and** both `DurableID` and `MountPath` must be non-empty (`dr_recipe.go:129-136`). demo-felhom has an enrolled 916 G USB HDD at `/mnt/felhom-drives/hdd_1` and demo-hp an enrolled 938 G NVMe at `/mnt/felhom-drives/nvme-1tb` (both LIVE). `dr.PlannedDrive` is the *"re-attach BY durable_id (the wrong-disk guard)"* half of the host-loss plan (`dr/plan.go:34-35`); with an empty list that half of the plan is empty. **Not diagnosed here** — the storage-target type mapping was not traced end to end. ### D2.4 — What happens to a customer box if the hub is gone? Answered from SOURCE plus one LIVE observation. Split by duration. **For a day.** Nothing customer-visible breaks. The box's data plane is entirely local: apps, Traefik and cloudflared run in the guest; the cloudflared connector token is already in `controller.yaml`; DNS is at Cloudflare, not the hub. The backup tiers keep running — Tier-1/2/3 are scheduler jobs inside the controller (`cmd/controller/main.go:601-690`) and the whole-guest tiers are the agent's. Restores A.1.1–A.1.7 all work. Events queue rather than drop: `settings.PendingEvents` + `DrainPendingEvents` (`internal/settings/settings.go:94,1466-1485`). **For a week.** Three things degrade, all of them *knowledge* rather than *copies*: 1. **The operator alarm plane goes dark.** Every notification — customer Hungarian and operator English — is dispatched by the hub via Resend. A failing tier still records `last_status: error` in the guest's own `settings.json` and still shows in the customer UI, but nobody is told. This is R-100's exact scope note: *"the operator's fleet-wide alarm plane is silent, which is the plane that matters for an unattended appliance"* (`CAMPAIGN-8…:481-483`). 2. **No claim codes, no password resets, no self-bind.** `POST /claim/request-new-code` is a hub round-trip; a customer locked out of their dashboard stays locked out. 3. **No config or version convergence.** `min_controller_version`, the artifact digests and the desired-state channel all stop; a box stays on whatever it runs. 4. **`host_pbs_secrets` re-issue is impossible**, so the R-39 self-heal chain — proven to take 13 s with the hub present — has no mint to consume. **Permanently.** Two irreplaceable losses and one recoverable one: - **The escrow custody is gone** (`host_escrow`, 2 rows). The blobs are the only off-box copy of the PBS client key, the WG private key and the restic repo password. Losing them does not lose the ciphertext, but it removes the only route to it that survives a host loss (C-2). - **The break-glass credentials are gone** (`host_recovery`, 3 rows) — C-3. - The Cloudflare tokens, the offsite descriptor, and the customer identity are recoverable, because a live box already holds a rendered copy in `controller.yaml` (LIVE, both boxes). **The one LIVE observation that bounds all of this:** `hosts.last_report_at` is `2026-07-28 17:25:20` / `17:25:31` for the two live hosts — a ~15-minute-old mirror. The hub's copy of box reality is never more than one cycle stale, which is precisely why it is *convenience, not the DR source of record* (`05-hub-architecture.md:170-171`). --- # Part E — failure domains (facts; no ruling on acceptability) ## E.1 — Do restic and PBS offsite share a provider, account, login or payment method? | | Tier-3 restic | Plane-2 PBS offsite | |---|---|---| | **provider** | **Hetzner** — Storage Box `u629488`, sub-accounts `-sub1` (demo-felhom) and `-sub3` (demo-hp), host `u629488-subN.your-storagebox.de:23` (LIVE from each guest's `settings.json`) | **Hetzner** — Cloud server `ep0` at `167.233.158.164` (LIVE `ssh root@…` → `felhom-hetzner`), datastore on Cloud Volume `scsi-0HC_Volume_106469259` mounted at `/mnt/pbs-datastore` | | **provisioning credential** | hub `HETZNER_TOKEN` + `HETZNER_POOL_BOX_ID` (`Secret/storagebox`), API base `https://api.hetzner.com/v1` (`hub/internal/hetznerapi/hetznerapi.go:23`) | provisioned out of band; the hub reaches ep0 over SSH with a pinned host key, not the Hetzner API | | **shared?** | **PROVIDER: YES, both Hetzner.** **ACCOUNT / LOGIN / PAYMENT: UNKNOWN** — see Unknowns. I deliberately did not call the Hetzner API with the production token to find out. | **Additional shared-fate facts inside each tier:** - **All customers share one Storage Box.** `u629488-sub1`, `-sub2` (peti, untouched) and `-sub3` are sub-accounts of one box. A sub-account is an **access-control object, not a data object** — the 2026-07-18 RESET proved this live by deleting a sub-account and finding its `/home` intact (`00-capability-map.md:42`, → R-32). - **All customers share one PBS datastore.** `felhom-offsite` on ep0, isolated by namespace + a per-customer client-side key + a per-customer `DatastoreBackup` token (LIVE `storage.cfg` on both hosts, distinct `encryption-key` fingerprints, `username felhom@pbs!demo-felhom` / `!demo-hp`). Consequence recorded in `07-backup-architecture.md:388-391`: per-tenant encryption **precludes cross-customer dedup**, so cost scales linearly per customer. - **Both offsite tiers ride the same WireGuard endpoint host in one respect**: ep0 is *both* the WG server and the PBS server (`06-…:34`, D3). restic does **not** ride WG — it goes guest → SFTP directly to the Storage Box (Campaign 8 fault 4's failure to inject is the proof: *"restic runs inside the guest, so its traffic never enters the host's `output` hook"*, `CAMPAIGN-8…:567-573`). So the two offsite tiers share a provider but **not** a transport. ## E.2 — What depends on DooPlex, and what happens if DooPlex is lost DooPlex (192.168.0.180, public IP **37.191.56.193**, LIVE) is a single physical machine. **What runs there (LIVE):** k3s (single node — every pod on node `dooplex`), the hub, Longhorn (replicas on `/dev/sdb1`), the Longhorn backup NFS export (`/mnt/5_hdd/backup`), the container registry `gitea.dooplex.hu`, a PBS instance (`systemctl is-active proxmox-backup` → `active`; doc D6 says DooPlex PBS is dev/demo only), the felhom.eu website, contact-mailer, umami, filebrowser, and this Claude Code session. **If DooPlex is lost:** | lost | consequence | |---|---| | the hub | everything in D2.4 "permanently" | | the **only** hub-DB backup | it is on the same machine (D2.2) — both copies gone together | | `gitea.dooplex.hu` | **no controller or agent image can be pulled**, so no deploy, no self-update, and a guest that loses its image layers cannot be brought back on the pinned tag | | `hub.felhom.eu` and `felhom.eu` | they resolve **directly to 37.191.56.193** — see E.3 | | operator alerting (Healthchecks → monitoring@) | gone | **What does NOT depend on DooPlex:** every customer box's data plane; both offsite tiers (Storage Box and ep0 are elsewhere); the customers' public app URLs (Cloudflare edge → cloudflared in the guest). ## E.3 — What depends on Cloudflare, and what breaks without it **DNS (LIVE, `dig @1.1.1.1`):** ``` felhom.eu NS martha.ns.cloudflare.com. / tony.ns.cloudflare.com. demo-felhom.eu NS martha / tony enkisfelhom.hu NS martha / tony sajatfelhom.hu NS martha / tony ``` **One Cloudflare account authoritative for the operator's own domain and every customer domain.** **Records (LIVE):** ``` hub.felhom.eu CNAME dooplex.hopto.org. -> A 37.191.56.193 (NOT proxied - the origin IP is public) felhom.eu A 37.191.56.193 (NOT proxied) ep0.felhom.eu A 167.233.158.164 (NOT proxied) felhom.demo-felhom.eu A 104.21.11.129 / 172.67.149.59 (PROXIED - Cloudflare edge) felhom.enkisfelhom.hu A 104.21.3.175 / 172.67.130.252 (PROXIED) felhom.sajatfelhom.hu A 172.67.162.57 / 104.21.90.232 (PROXIED) ``` **What breaks without Cloudflare:** | Cloudflare component | what stops | |---|---| | **DNS** | *everything by name* — box→hub reporting, customer app access, ep0 endpoint resolution by the WG watchdog. The WG tunnel itself survives, because the agent resolves the A record and **writes the v4 literal** into the conf (`06-…:174-184`); PBS then dials `10.77.0.1`, an in-tunnel address that needs no DNS. | | **Tunnel / edge** (proxied records) | every customer app URL. The `lan_resolver` LAN path is the documented fallback and is **IMPLEMENTED, never drilled as a customer experience** (`00-capability-map.md:93` → R-19). | | **WAF / geo** | geo-restriction enforcement (hub-held CF API token) | **A third-party dependency in the same path, easy to miss:** `hub.felhom.eu` is a CNAME to **`dooplex.hopto.org`** — a **no-ip DynDNS** name. Box→hub reporting therefore depends on Cloudflare DNS **and** no-ip **and** the operator's home IP not changing faster than the DynDNS updater. ## E.4 — What depends on the k3s cluster k3s is **single-node** (LIVE: every pod on `dooplex`), so "the k3s cluster" and "DooPlex" are the same failure domain. Inside it: the hub Deployment, its Longhorn PVC, the nginx-internal ingress (`hub.felhom.eu` is `ingressClassName: nginx-internal`, `manifests/hub.yaml:323,329`), cert-manager for the hub certificate, ArgoCD (auto-sync **off**), and the Longhorn backup target. There is no second node and no failover. ## E.5 — Does any single credential or account failure take out more than one tier? | credential / account | tiers it can take out | |---|---| | **the Cloudflare account** | box→hub reporting **and** every customer's app access **and** the operator website — three planes, one login | | **the Hetzner account** (if one account covers both products — UNKNOWN) | Tier-3 restic **and** Plane-2 PBS offsite — i.e. **both offsite tiers at once**, leaving only the local vzdump tier, which lives on the same physical disk as the guest it backs up (E.6) | | **the hub global bearer** (`REPORT_API_KEY`) | operator-tier hub API incl. break-glass retrieval. **Its git-history copy is still live** — `secrets.md:76-77`: *"The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing."* Rotation runbook exists; no rotation date is recorded. | | **the Resend key** | every notification **and** every claim/reset code — the alarm plane and the account-recovery plane at once | | **the customer's recovery code R** | unwraps the PBS key, the WG key, the PBS token **and** the restic repo password — one loss, both offsite tiers unreadable after a host loss | | **the restic repo credential** | **can delete.** R-95, still open and ranked #1: *"restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only"* (`OPEN-ITEMS.md:13`). Contrast R-89, proven the other way for PBS: *"the box was correctly refused when it tried to delete its own offsite snapshot … A compromised box cannot destroy its own backups"* (`CAMPAIGN-8…:514-515`). **The two offsite tiers have opposite blast radii from a box compromise.** | ## E.6 — One more shared fate, found by inspection rather than by document **The local whole-guest tier shares its physical disk with the guest it backs up.** LIVE `lsblk` + `pvs` on both hosts: ``` demo-felhom: sda 476.9G -> sda3 -> VG pve -> { pve-root (/, holds /var/lib/vz/dump), pve-vm-9201-disk-0/1/2 (the guest) } sdb 931.5G -> /mnt/hdd_1 (the customer data drive, separate) demo-hp: sda 119.2G -> sda3 -> VG pve -> { pve-root (/, holds /var/lib/vz/dump), pve-vm-9201-disk-0/1/2, vm-300-disk-* } nvme0n1 953.9G -> /mnt/nvme-1tb (the customer data drive, separate) ``` `local` (the vzdump target, `/var/lib/vz`) and `local-lvm` (the guest's rootfs and both backup=1 mountpoints) are **the same physical device** on both hosts. A single-disk failure removes the guest and its entire local backup history together, leaving only the weekly PBS tier. Tier-2's `sys_drive` destination is on that same disk as well (`mp1` = `pve-vm-9201-disk-2`), so on demo-hp `paperless-ngx`'s Tier-2 copy is cross-*drive* from the source (`nvme-1tb` → `sda3`) but co-located with the local whole-guest archives. --- # Part F — measured timings Only measured numbers. **Nothing here is estimated**, and everything unmeasured is listed as such. ## F.1 — Backup timings (LIVE, this session) | operation | measurement | source | |---|---|---| | local vzdump, demo-felhom | **00:04:10** (15,062,241,280 B written → 5.42 GB archive, 59 MiB/s) | `/var/lib/vz/dump/vzdump-lxc-9201-2026_07_28-01_27_56.log` | | local vzdump, demo-felhom | **00:05:09** (15 GiB → 5.44 GB, 47 MiB/s) | `…-06_44_22.log` | | local vzdump, demo-felhom | **00:05:09** (15 GiB → 5.52 GB, 48 MiB/s) | `…-17_53_26.log` | | local vzdump, demo-hp | **00:01:05** (4,544,911,360 B → 1.50 GB, 71 MiB/s) | `…-09_42_36.log` | | local vzdump, demo-hp | **00:01:05** (4.3 GiB → 1.52 GB, 72 MiB/s) | `…-10_40_54.log` | | local vzdump, demo-hp | **00:01:06** (4.4 GiB → 1.56 GB, 71 MiB/s) | `…-17_43_05.log` | | restic offsite run, demo-felhom | **2m27s** (last successful, 2026-07-28T11:29:22Z; repo 1.0 GB / 32 snapshots) | guest `settings.json` `offbox.last_duration` | | restic offsite run, demo-hp | **1m35s** (2026-07-28T16:20:57Z; repo 36.3 MB / 8 snapshots) | guest `settings.json`; corroborated in `CAMPAIGN-9…:743-748` | | Tier-2 run, demo-felhom bookstack | **1s** for 155.7 MB | `settings.json` `app_backup.bookstack.cross_drive` | | Tier-2 run, demo-felhom immich | (duration not recorded in the row; size 1.5 GB) | as above | | Tier-2 run, demo-hp paperless-ngx | **0s** for 85.5 MB | `settings.json` | | PBS backup, demo-felhom | **5.7 GB / 00:03:12** | `CAMPAIGN-8…:75-76` | | PBS backup, demo-hp | **1.5 GB / 00:00:55** | `CAMPAIGN-8…:75-76` | ## F.2 — Restore timings | operation | measurement | source | |---|---|---| | **whole-guest restore-test, LOCAL tier, demo-hp** | **84.18 s / 84.21 s / 84.26 s / 89.20 s / 89.34 s** across five consecutive runs on the same 1.56 GB archive | LIVE `journalctl -u felhom-agent`, 2026-07-28 18:00–18:08 CEST | | **whole-guest restore-test, LOCAL tier, demo-felhom** | **93.84 s · 99.10 s · 111.82 s · 98.81 s** on 5.4–5.5 GB archives | LIVE `journalctl`, 2026-07-27 20:47 → 2026-07-28 00:32 CEST | | **whole-guest restore-test, PBS OFFSITE tier, demo-felhom** | **1101.05 s ≈ 18 m 21 s** — `archive=felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z`, passed 2026-07-28 00:04:27 CEST | LIVE `journalctl` | | whole-guest restore-test, PBS, demo-hp (manual `--selftest`) | **4 m 5 s** restore + boot + verify + teardown, `mount_parity: ok` | `00-capability-map.md:40` | | local whole-guest restore, Campaign 9 incidental | **89.19 s** — restored into scratch 990000, booted, torn down | `CAMPAIGN-9…:363-373` | | local vzdump → fresh CT 9299 | 5.82 GB archive, **15 GiB extracted @ 210 MiB/s**, exact mount parity | `CAMPAIGN-8…:247` | | PBS offsite → fresh CT 9297 | 4.36 GB encrypted snapshot, exact mount parity (no wall-clock recorded) | `CAMPAIGN-8…:248` | | **restic app-data restore (R-87)** | **14 s** for 155.5 MB, `bookstack` snapshot `b2d8936d`, 6/7 files byte-identical | `CAMPAIGN-8…:246` | | **Tier-2 file restore, gap-fill (A1)** | **39 s** — 6 files restored, incl. app stop→healthy | `CAMPAIGN-9…:105-109` | | **Tier-2 file restore, after total loss (A3)** | **46 s** — 43 files restored, 16/16 documents usable afterwards | `CAMPAIGN-9…:184` | | **Tier-1 recovery-unit restore (A2)** | **18.25 s**; app healthy 29 s later | `CAMPAIGN-9…:141-151` | | app restart-to-healthy after Tier-2 restore | `Up 5 seconds (health: starting)` → `Up 27 seconds (healthy)`, polled at 10 s | `CAMPAIGN-9…:112-114` | | **guest-power watchdog recovery** | **120 s** unattended, against the **587 s** the original incident needed with a human | `OPEN-ITEMS.md:38`; the incident's own figure is 9m47s = 587 s (`CAMPAIGN-8…:554-556`) | | quiesce crash recovery (unquiesce after SIGKILL) | **1 second** | `CAMPAIGN-8…:216,229-233` | | app downtime, one quiesce, two tiers | **86 s** (Campaign 8 A2) and **1m27s** (the R-82 UI-triggered proof) and **~97 s** (Campaign 9 B3) | `CAMPAIGN-8…:129`; `00-capability-map.md:40`; `CAMPAIGN-9…:337` | | PBS-DR credential self-heal, operator click → converged | **13 s** | `00-capability-map.md:39` | | controller floor-lift, operator save → healthy on the new version | **16 s** | `00-capability-map.md:124` | | customer bind → box running current | **2 min 44 s** | `00-capability-map.md:124` | ## F.3 — Unmeasured (listed, not estimated) - **Restore of a whole guest from PBS to a *different* host** — never run. - **Restore of a whole guest over the WAN at production scale** — the only offsite restore-test timing (1101 s) is a 4.4 GB guest to the same host over the existing tunnel. No large-guest number exists. - **A restic restore of anything larger than 155.5 MB.** - **Any restore of a customer data drive.** No tier holds one whole (Part B.2), so no such timing can exist. - **Weekly PBS incremental size and duration** — `07-backup-architecture.md:384` still records this as **UNMEASURED**; the datastore now holds 3 and 6 snapshots respectively, so the input exists but the measurement has not been made. - **Host rebuild from nothing** — never performed (D1). - **Escrow consume → tier re-established** — never performed destructively. - **Hub DB restore from a Longhorn backup** — never performed. - **`.fab` import wall-clock** — the 1.7 GB round trip is cited without a duration. - **Time to first byte for a customer restore over a home uplink** — no customer has ever driven a restore (`00-capability-map.md:75`). --- # Part G — what is actually proven Status vocabulary is the capability map's own strict enum (`00-capability-map.md:11-18`): a PROVEN-LIVE claim **must** cite a campaign/drill/validation doc. ## G.1 — Live, cited proof | path | proof | |---|---| | Tier-2 file restore, gap-fill | Campaign 9 A1 — byte-identical returns, both non-destruction promises kept, restored bytes served by paperless's own API | | Tier-2 file restore, after total directory loss | Campaign 9 A3 — 404 before, 200 with matching hashes after, 16/16 documents | | Tier-2 restore refuses without an outage for a no-coverage app | v0.183.0 live replay on demo-felhom — BookStack uptime unbroken (`felhom.eu/REPORT.md:60`) | | Tier-1 recovery-unit restore **executes** end to end | Campaign 9 A2 | | restic offsite restore of app data (bytes) | Campaign 8 R-87 | | offsite **reconstitution** of a DB-indexed app through the customer's own buttons | 2026-07-20 destructive immich drill (trash emptied), `00-capability-map.md:66,75` | | shares restore (files + definitions + credential) | 2026-07-18, `00-capability-map.md:96` | | `.fab` export/import full circle | `CAMPAIGN-6D` P-FAB, 1.7 GB byte-identical | | whole-guest `pct restore`, local and PBS, with exact mount parity | `CAMPAIGN-2` T-P9 + Campaign 8 Phase C | | a corrupted PBS snapshot **fails cleanly** | Campaign 8 fault 17 + Phase C — quarantined `.0.bad`, `verify_state: failed`, restore exit 255 naming the entry and the chunk, all three LVs rolled back, no debris | | a restore-test whose source vanishes fails loudly and does **not** advance the proven-restorable timestamp | Campaign 8 fault 18 | | the box cannot delete its own **PBS** snapshots | Campaign 8, R-89 | | unattended restore-test on **both** tiers | LIVE this session — `local` passes on both boxes; `felhom-pbs` passed on demo-felhom at 00:04:27 CEST | | guest-power watchdog | 120 s unattended recovery, 2026-07-28 | | F-CRIT-1 / F-CRIT-2 / F-A1 / F-LEAK / F-OBS / F-REBOOT fixes | each replayed live, `OPEN-ITEMS.md:25-40` | | C9-F2 crash-loop alarm | live replay: silent through ten 30 s samples, then `app_start_failed` at **5m25s**; heartbeat flipped `0 currently down` → `1 currently down` (`felhom.eu/REPORT.md:57-58`) | ## G.2 — Unit-tested / implemented only - **Restore-proof is UNATTENDED across every tier (R-85)** — `IMPLEMENTED`, explicitly *not* PROVEN-LIVE: *"rotation has not been observed selecting both tiers across consecutive UNATTENDED cadences"* (`00-capability-map.md:41`). My LIVE reads are consistent with rotation (both tiers have a proof timestamp on both boxes) but a single pass is not the multi-day observation the row asks for. - **PBS-DR secret self-heal** — reconciler still scoped to one host via `PBSDRHEAL_ONLY_HOST` (`00-capability-map.md:69`). - **Customer DELETE cascade** — unit-proven; the end-to-end live leg is not run (`00-capability-map.md:43`). - **F-DIAG's six offsite failure classes** — unit-proven; *"not yet exercised by a live offsite failure of each class"* (`OPEN-ITEMS.md:36`). Campaign 9 D10 did fire **one** of them (transport) live and confirmed the sanitiser. - **R-100's 48 h `offsite_stale` threshold** — the producer half is confirmed twice live; the alarm itself has **never fired** (`CAMPAIGN-9…:700-714`). - **Paired recovery notifications** — `IMPLEMENTED`, recovery leg PARTIAL (`00-capability-map.md:113`). ## G.3 — Doc claim only, or nothing | claim | status | |---|---| | agent DR bring-up `ModeDRGuestLoss` | code exists, **never executed** | | host-loss DR plan → an actual host restore | plan **executes nothing by construction**; the destructive drill is unrun | | escrow **consume** in a real recovery | spike-level only | | **Tier-1 content recovery after real loss** | **the single most valuable unproven item** (`CAMPAIGN-9…:825-827`, restated `felhom.eu/REPORT.md:96-97`) | | host reboot mid-backup | NOT REACHED in Campaign 8 (fault 12) and NOT STARTED in Campaign 9 (C6) | | three-way concurrency with GC | never | | a never-succeeded Tier-2 (Scenario C) live | never (`felhom-controller/REPORT.md:81-82`) | | F-HUB `SQLITE_BUSY` characterisation | never | | **a customer (not the operator) performing a restore via UI alone** | **MISSING as evidence** (`00-capability-map.md:75`) — no product gap blocks it since v0.153.0; the evidence run has simply never been done by a customer | | hub DB restore from its Longhorn backup | never | | FileBrowser browse/download | *"exercised in no doc"* (`00-capability-map.md:98`) | ## G.4 — Where a doc claims something the evidence does not support Each row cites **both** sides. | # | doc claim | contradicting evidence | |---|---|---| | G4-1 | `07-backup-architecture.md:85-91`: *"Both tiers are now restore-tested **unattended** … and each tier's last successful proof is reported."* | `00-capability-map.md:41` says the same capability is **IMPLEMENTED, not PROVEN-LIVE**, because *"rotation has not been observed selecting both tiers across consecutive UNATTENDED cadences."* The architecture doc states as settled what the capability map holds open. | | G4-2 | `07-backup-architecture.md:29,79`: the Tier-1 row describes `RestoreApp` as the restore path | `restore_unit.go:95` `RestoreFromRecoveryUnit` is the real path; `RestoreApp` is only the fallback when no unit exists (`:120-127`). The doc's own staleness banner (`:11`) covers this — it warns §0–§8 are *"historical intent, not current fact"* — so this is a disclosed, not a hidden, drift. | | G4-3 | `settings.go:148-150` (OffboxTarget doc comment): *"off-box of the secrets rides DR via the PBS whole-CT snapshot of **the rootfs**"* | LIVE: the secrets are in the Docker named volume `felhom-controller-data`, i.e. under `/var/lib/docker` = **mp0**, not the rootfs. The protection claim holds (mp0 is `backup=1`) but names the wrong mountpoint — a comment that would become false if mp0's `backup` flag ever changed, with no test pinning it. | | G4-4 | `06-offsite-connectivity.md:19-21`: *"The operator's public edge is Cloudflare-Tunnel → nginx-internal … DooPlex has **no public IP** and no public UDP."* | LIVE: `hub.felhom.eu` CNAMEs to `dooplex.hopto.org` → **A 37.191.56.193**, which is DooPlex's own public IP (`curl api.ipify.org` from DooPlex returns the same address). The record is **not** Cloudflare-proxied. The doc is from 2026-07-03 and the topology has since changed, or was never as described for the hub. | | G4-5 | `dr_recipe.go:55`: `Namespace` is *"PBS namespace the restore targets"* | LIVE: `"root"` on every box, while the real namespaces are `demo-felhom`/`demo-hp` (verified on ep0 and in a live `pct restore` command line). Root cause traced to `client.go:97` / `report.go:23-27` (D2.3-c). | | G4-6 | `05-hub-architecture.md:175-176,186`: the *"slim DR record"* on `hosts` holds PBS namespace + repo fingerprint + wrapped escrow key, and host-loss recovery reads from it | LIVE: `hosts.dr_record_json` is `{}` for all three hosts (D2.3-a). | | G4-7 | `06-offsite-connectivity.md:148-150`: the escrow upload included *"directive with the non-secret DR coords"* | LIVE: `host_escrow.directive_json` is `'{}'` — 2 bytes — for both escrowed hosts (D2.3-b). | | G4-8 | `dr/plan.go:34-35` describes `PlannedDrive` as the re-attach-by-`durable_id` wrong-disk guard, and `06-…:327` reports the matcher unit-proven | LIVE: `dr_recipe.host_half.drives` is `[]` on every customer, including two boxes with enrolled data drives (D2.3-d). The guard is correct; it currently has nothing to guard. | | G4-9 | `felhom.eu/REPORT.md:17-24` / `OPEN-ITEMS.md:31`: **A=9 / B=43 / C=1** over the 53 templates | My independent enumeration of the same catalog at `4252121`, applying `classify.go`'s documented default rules, gives **A=7 / B=45 / C=1** (Part B.1). Two apps unaccounted for. | | G4-10 | `felhom-controller/controller/README.md` (corrected 2026-07-28) claimed faults surface as *"exited/degraded/**restarting**/unhealthy"* | `StateRestarting` was in no down-set at all — the seventh shipped-invariant comment; already found and corrected in `felhom.eu/REPORT.md:84-87`. Recorded here because it is the same class as G4-3 and G4-5. | | G4-11 | `CAMPAIGN-8…:96` pre-registered the window-gate valve behaviour as an artifact of compression | Not a contradiction — recorded as the *good* case: the campaign declared its artifacts before running, and Campaign 9 repeated the practice (`CAMPAIGN-9…:302-313`). Included so the list is not read as one-sided. | --- # Unknowns Everything I could not establish, and what it would take. | # | unknown | what would settle it | |---|---|---| | U-1 | **Do the Storage Box and ep0 sit under one Hetzner account / login / payment method?** | An operator statement, or a read-only `GET /v1/servers` and `GET /v1/storage_boxes` with the hub's `HETZNER_TOKEN`. I deliberately did not make that call — it uses a production credential against an external service for an inventory question the operator can answer in one line. | | U-2 | **Which 2 apps account for the A=9 vs A=7 difference?** | Diff my enumeration script against the C9-F1 Phase-0 method. The Phase-0 method is described in prose (`felhom.eu/REPORT.md:15-24`) but the enumeration itself is not committed. | | U-3 | **Is `IdentityBundle.TunnelToken` populated on the current escrow blobs?** | Unopenable without R, by design. `06-…:151` records that the 2026-07-04 blob had it *"intentionally empty"*; the current blobs are from 2026-07-21/07-23 and are 572 B. Only a consume ceremony would tell. | | U-4 | **Whether `hosts.dr_record_json = {}` is a doc drift or an unwritten field.** | Read the hub's writer for that column; I did not trace it. | | U-5 | **Why `dr_recipe.drives` is empty** — whether the storage targets fail `isUserDataDrive`'s `type`/`DurableID`/`MountPath` test, and which of the three. | Read the live `host-report` JSON's `storage_targets[]` for one box. | | U-6 | **Whether the offsite restic repos contain what their snapshot counts imply.** | `restic snapshots` against each repo. Not run — it takes a repo lock, and C9-F3 makes lock handling a live hazard. Recorded counts used instead (32 / 8). | | U-7 | **The peti-felhom recovery posture.** LIVE: `dr_recipe` has a `peti-felhom` row referencing host `peti-felhom-86d37d`, but there is **no such row in `hosts`** and **no `host_escrow` row**. | Out of scope by instruction — peti was not touched. Flagged because it means one of four customers has no escrow custody at all. | | U-8 | **Whether the Longhorn hub-DB backups are restorable.** Two `Completed` backups exist; neither has been restored. | A restore drill into a scratch PVC. Not run (read-only task). | | U-9 | **Whether ep0 itself is backed up.** LIVE: the datastore is on a Cloud Volume and `/srv/pbs-felhom` holds a 13 G rollback copy on the root disk. `06-…:313` lists endpoint DR as a deferred slice-1 item. | Whether Hetzner Cloud snapshots/backups are enabled on `ep0` — a console or API question. | | U-10 | **Whether the `.fab` browser-upload leg works.** | A human click-through; `00-capability-map.md:67` still records it as Viktor's open test. | | U-11 | **What `app_backup..enabled` (Tier-1 toggle) actually gates today.** LIVE: it is `false` for **every** app on **both** boxes, while recovery units are demonstrably being captured and restored. | Trace the `Enabled` field's readers; my grep found only `offbox.go:451,1167` and `tier2.go:577-611`, none of which is a Tier-1 gate. Possibly vestigial (its comment still says *"includes app data in nightly restic (same drive)"*, a mechanism that moved to the agent). | | U-12 | **When the hub global bearer was last rotated** — the git-history copy is live until it is. | A password-manager entry date or a hub-side audit. | --- # Contradictions Doc / comment / README versus source or live state. G4-1 … G4-11 above are the full list; the four that are **new in this inventory** (not previously filed anywhere I could find) are restated here so they are not buried: 1. **G4-4 — the hub's public edge is not what doc 06 describes.** `hub.felhom.eu` resolves to DooPlex's own public IP through a no-ip DynDNS CNAME, unproxied. Doc 06 §1 says the edge is a Cloudflare Tunnel and that DooPlex has no public IP. 2. **G4-5 — the DR recipe records the PBS namespace as `root` on every box.** Traced to source; the real namespaces are per-customer. 3. **G4-6 / G4-7 / G4-8 — three DR inputs are empty on the live fleet**: `hosts.dr_record_json`, `host_escrow.directive_json`, and `dr_recipe.host_half.drives`. Each is named by a design doc as something host-loss recovery reads. 4. **G4-9 — the C9-F1 class counts do not reproduce.** 9/43/1 published, 7/45/1 by my enumeration. And one contradiction of **omission** rather than of statement: 5. **The offsite tier captures Docker named-volume tars on every run and no offsite action unpacks them.** `07-backup-architecture.md:81` describes Tier-3's restore path as *"staged scratch on a data drive → missing-only merge to live"* and §7.3 frames that as sufficient for the SQ3 acceptance. `offbox_reconstitute.go:284-289` skips the unit outright, and `offbox_restore.go:352-356` places it only when the live unit is absent. For the 52 of 53 templates with named volumes, the volume data is in the snapshot and no single offsite action returns it. Not currently filed. --- # Observations Noticed during the inventory and **deliberately not acted on**. **O-A — A plaintext restic repo password is sitting unwiped on demo-hp's host filesystem.** LIVE: `/var/lib/felhom-agent/escrow-stage/restic_repo_password`, 64 B, `0600 felhom-agent`, mtime **2026-07-28 13:26 CEST**. `felhom-agent/internal/escrow/identity.go:41-44` documents this path as a **transient** stage that the ceremony wipes after a successful escrow-create (`WipeStagedResticPassword`, `:47-53`). demo-felhom's `escrow-stage/` is empty, as expected. So on demo-hp the data key for the entire Tier-3 tier is currently a plaintext file on the host, outside the escrow, five days after that host's escrow was created (2026-07-23). Not touched. **O-B — Twelve stale backup copies of `agent.json` on the hosts.** LIVE: six per host (`.bak-pre-r82`, `.bak-pre-r85`, `.campaign8-before`, `.campaign9-before`, `.campaign9-prev`, `.pre-prunegate.bak`), each 2.3–2.5 KB and each containing the box's `hub.api_key`. Campaign 8 already noted that reading `agent.json` printed `hub.api_key` in cleartext because the redactor matched `*token*`/`*secret*`/`*password*` but not the bare `api_key` (`CAMPAIGN-8…:59-63`). Four similar copies of `controller.yaml` and three of `settings.json` exist inside each guest. Left alone. **O-C — The R-99 phantom snapshot is still on ep0 and still counted as an entry.** LIVE: `/mnt/pbs-datastore/ns/demo-hp/ct/9201/2026-07-28T05:31:14Z/` holds only `catalog.pcat1.tmp_didx` (0 B), `root.pxar.tmp_didx` (0 B) and `pct.conf.blob` (425 B) — no `index.json.blob`. It is exactly the artefact Campaign 9 saw the F-CRIT-2 size filter reject live (`CAMPAIGN-9…:341-361`). R-99 is filed and explicitly not automated (`OPEN-ITEMS.md:26`). Not deleted. **O-D — The setup wizard's restore path is dead code with live templates.** `internal/setup/templates/setup_hub_restore.html` and `setup_restore_exec.html` exist and are parsed by `ParseFS(templateFS, "templates/*.html")` (`handlers.go:78-81`), but no route serves them — `Handler()` registers only `/`, `/setup`, `/setup/fresh`, `/setup/manual`, `/setup/failed` and two static paths (`:85-99`), with the comment *"Disk-recovery setup paths (drive scan, infra-backup restore) have moved to the host agent (slice 8C)"*. An orphaned doc comment for `autoProcessHubRestore` now sits directly above `autoProcessFreshHub` (`:215-219`), describing a function that no longer exists. This is the "seam built but never wired" shape in reverse — a seam **unwired but never removed**. Not deleted. **O-E — `tier2DestRel` maps `RootImport` into the `hdd/` subtree.** `tier2_capture.go:26-33`: `base := "hdd"` unless the root is `RootUserdata`. So a `${IMPORT_PATH}` bind would be captured to `hdd/` and restored to `liveNsRoot/`, while its live location is `importRoot/` — the *system* namespace's `userdata/import`, deliberately a different root (`classify.go:37-41`). **Currently unreachable**: both catalog apps that bind `${IMPORT_PATH}` (`calibre-web`, `paperless-ngx`) class it `excluded`, so it is never captured. A latent mismatch that becomes real the day an import path is classed non-excluded. Not changed. **O-F — Nineteen Tier-1 recovery units on demo-felhom are 24 KB config-only shells for apps that are not deployed.** LIVE: `adventurelog`, `bentopdf`, `calcom`, `crafty-controller`, `ghost`, `glance`, `gokapi`, `gramps-web`, `home-assistant`, `homebox`, `kimai`, `n8n`, `outline`, `papra`, `rallly`, `recipe-importer`, `tandoor`, `wger`, `wishlist`, `zipline` — all under `backups/primary/` on `sys_drive`, none in `docker ps`. Harmless; noted because `ListRestorePoints` will offer a restore point for each, and the count of "apps with a restore point" is therefore not the count of "apps that exist". **O-G — The in-memory R-88 breaker still carries Campaign 9's failure counts.** Declared in the campaign's own report as a benign residue that clears on the next successful whole-guest backup or any controller restart (`CAMPAIGN-9…:762-768`). Both controllers have since restarted (v0.183.0 was deployed after the campaign), so this is very likely already cleared — I did not verify it, because verifying would mean reading a state the controller does not expose read-only. **O-H — DooPlex's root filesystem is at 81 % (`/dev/sdb1`, 341 G of 445 G used).** Below the 90 % abort threshold in the workspace `CLAUDE.md`, but it is the disk holding both the Longhorn replicas and the container images. Noted, not acted on. **O-I — `drill-r50` is both a blocked customer and a live WireGuard peer.** LIVE: `customer_configs` status `blocked`, `hosts.last_report_at 2026-07-27 05:44:53`, agent 0.96.0 (four minor versions behind the fleet), a `host_recovery` row, and peer `10.77.0.4` on ep0 with no handshake. Already filed as R-93 (`OPEN-ITEMS.md:43`). Recorded here because it means the fleet's peer list and host table both contain an entry whose recovery posture nobody maintains. **O-J — The window between "backup exists" and "backup is provable" is structurally different per tier, and only one tier's is short.** LIVE: local vzdump retention is 3 archives at a 24 h cadence (≈3 days of history); PBS offsite is `keep-last 2` at a 7-day cadence (≈2 weeks); restic is `--keep-daily 7 --keep-weekly 4 --keep-monthly 6`. The restore-test cadence is **84 h** on both boxes. So the local tier is re-proven roughly once per its own retention window, while the offsite PBS tier holds two snapshots and is proven at a cadence unrelated to when they were written — which is exactly what R-86 is filed to fix (*"Restore-tests are interval-scheduled, not backup-aligned"*, `OPEN-ITEMS.md:15`). Stated as a measured relationship, not a recommendation.