felhom.eu/documentation/architecture/_recovery-inventory-2026-07-28.md

Recovery baseline inventory — 2026-07-28

Class: INVENTORY, READ-ONLY. Facts only. Nothing was changed, fixed, deployed or restarted. No backup, restore, GC or prune was run to "check" something. peti-felhom was not touched. No matrix is designed here, no target architecture is proposed, and no gap is ranked.

This is the input to a redesign, not the redesign.

0. Method, baselines, and what "verified" means here

Three claim kinds are distinguished throughout and labelled per claim:

label	meaning
SOURCE	read from code at a cited file:line in this workspace at the HEADs below
LIVE	observed by running a read-only command against live infrastructure during this session, output pasted
DOC	asserted by a document; unverified unless a SOURCE/LIVE line accompanies it
INFERENCE	my reasoning over SOURCE/LIVE facts, marked as such

Repo HEADs read (LIVE, git rev-parse --short HEAD, all clean except a pre-existing modification to felhom.eu/documentation/PROMPT-TEMPLATE.md which is unrelated to this task):

felhom.eu               ff050cf
felhom-agent            d5c7691
felhom-controller       fd50a73
app-catalog-felhom.eu   4252121

Live fleet at inventory time (LIVE):

hub          gitea.dooplex.hu/admin/felhom-hub:0.80.0        (k3s felhom-system, 1/1, pod age 6h18m)
agent        felhom-agent 0.110.0                             on demo-felhom AND demo-hp
controller   felhom-controller:0.183.0                        in guest 9201 on BOTH boxes
catalog      53 templates
hosts        demo-felhom (192.168.0.162) · demo-hp/`felhom-host` (192.168.0.87) · ep0 167.233.158.164
guests       one per host: 9201, both `running`, both `onboot: 1`

Session clock: commands ran 2026-07-28 17:40–17:45 UTC / 19:40–19:45 CEST. Host clocks are CEST; hub, guests and PBS are UTC — every timestamp below carries its zone.

One derived artefact was made and is stated so it is not mistaken for a live read: a hot copy of the hub SQLite DB was taken with kubectl exec … cat /data/hub.db > hub.db.copy at 17:40:00 UTC (113,033,216 bytes) into the session scratchpad, and every hub-DB figure below comes from that copy. It is a hot copy of a live database; row counts and metadata are consistent enough for an inventory but are a snapshot of that instant, not a transactionally consistent dump. Do not reuse this command as a recipe: since hub v0.88.0 the DB is in WAL mode (R-172), so cat /data/hub.db alone yields a copy that opens cleanly and silently omits the newest writes — the -wal must be copied beside it (documentation/operations/nodes.md). The copy is read with mode=ro. No hub table was written.

Secret hygiene. No secret value, key, token, password or key fingerprint is reproduced in this document. Where a secret is named, only its location, size, mode and owner are recorded.

---

Part A — every restore path that exists

Twelve distinct restore paths were found. They are grouped by who can reach them, because that is the axis this task exists to establish.

Route inventory method (so "nothing else found" is falsifiable): every case path == in controller/internal/web/server.go:360-551; every route in controller/internal/api/router.go:153-332; every mux.HandleFunc in controller/internal/setup/handlers.go:85-99; every --selftest mode in felhom-agent/cmd/felhom-agent/main.go:168; every RUNBOOK-*.md under felhom.eu/documentation/runbooks/ and felhom.eu/documentation/pilot/; and a repo-wide grep for restore|Restore over felhom-controller/controller/internal and felhom-agent/internal.

A.1 — Customer-reachable, unassisted (dashboard password only)

All five below sit behind the controller's RequireAuth mux gate — the single household dashboard password, which the customer owns (set at claim; internal/web/auth.go:34-35 precedence settings.json → password_hash over controller.yaml → web.password_hash). No operator credential, no SSH, no hub action is needed. This is the whole customer-recovery surface.

A.1.1 — „Visszaállítás indítása" — Tier-1 recovery-unit restore

field	detail
Name (UI, HU)	„Visszaállítás indítása" — page „Biztonsági mentés → Visszaállítás"
Entry point	POST /backup/restore (stack_name + snapshot_id) — button internal/web/templates/backups_restore.html:52, form action :346; route internal/web/server.go:466-467; handler internal/web/handlers.go:1241; engine internal/backup/restore_unit.go:95 RestoreFromRecoveryUnit
Source	Tier-1 only. The app's own drive: <nsRoot>/backups/primary/<stack>/ (compose/, db-dumps/*.sql, volume-dumps/*.tar, manifest.json). Tier-2 copies are unreachable from here — stated and pinned at internal/backup/restore_points.go:16-18: "Tier-2 copies are NOT restorable through POST /backup/restore (it only reads the app's primary unit), so listing them would silently restore tier-1 data while claiming tier-2 — never emit them here."
Scope	compose + .felhom.yml + secret-stripped app.yaml; Docker named volumes (tars re-imported); DB dumps replayed into a DB-only bring-up (R-47); app redeployed on the pinned image. This is the only path that restores named-volume tars.
Semantics	DESTRUCTIVE/REPLACING — volume contents are re-imported over the live volume and the stack is redeployed from the unit's compose.
Invocable by	CUSTOMER, UNASSISTED
Prerequisites	app deployed and its drive resolvable (GetAppDrivePath non-empty, restore_unit.go:114-117); a recovery unit present (absent ⇒ silent fallback to volume-only RestoreApp, :120-127); the guest's live app.yaml must still hold the app's secrets — see the fail-closed gate below; no concurrent backup/restore (IsRunning(), handlers.go:1270)
Fail-closed gate	reconcileRestoreSecrets (restore_unit.go:23) refuses when a data-encrypting key is unrecoverable, because regenerating it would make the restored data unreadable; a missing resettable secret is regenerated and logged (:143-163). Secrets are recovered from the guest, never from the unit (:132).
Proven	PATH EXECUTES — content recovery NOT proven. Campaign 9 A2 on calibre-web/demo-felhom: 15:28:34 Restoring calibre-web from recovery unit: images=1 … 15:28:52 Restore-from-unit completed: calibre-web (18.25s), app healthy 29 s later (CAMPAIGN-9-restore-proof-2026-07-28.md:141-154). The campaign states its own limit: "It does not prove content recovery, because nothing was lost first" (:153-154). Listed as "the single most valuable unproven item" (:825-827) and repeated in felhom.eu/REPORT.md:96-97.

A.1.2 — „Fájlok visszaállítása" — Tier-2 cross-drive file restore

field	detail
Name (UI, HU)	„Fájlok visszaállítása" — page „Biztonsági mentés → Alkalmazások" (/backups/apps)
Entry point	POST /backup/tier2/restore (stack_name) — form internal/web/templates/backups_apps.html:247,252; route server.go:469-470; handler handlers.go:1310; engine internal/backup/tier2_restore.go:118 RestoreTier2Files
Source	the RECORDED Tier-2 destination (settings.CrossDriveBackup.DestinationPath), never a fresh target pick — tier2_restore.go:114-116. Only two subtrees are read: destBase/hdd → liveNsRoot and destBase/userdata → liveNsRoot/userdata (:101-104), each os.Stat-skipped when absent (:114-116).
Scope	file legs only. The copy's recovery-unit/ (DB dumps + named-volume tars) is written by every Tier-2 run (tier2.go:369 "Unit leg (always)") and read by no code path — this is C9-F4.
Semantics	ADDITIVE / MISSING-ONLY. rsync -a --ignore-existing (tier2_restore.go:213-217): existing live files are never overwritten, nothing is ever deleted.
Invocable by	CUSTOMER, UNASSISTED
Prerequisites	a recorded Tier-2 destination with the v2 layout marker; app stopped for the copy (stop-first is the locked consistency policy, :116-117); no concurrent op. Since v0.183.0 a pre-flight coverage check refuses without stopping the app when the copy holds no readable leg (handlers.go:1341-1346).
Proven	PROVEN, twice, on live hardware. Campaign 9 A1 (paperless-ngx / demo-hp, through traefik with a real session cookie and CSRF token): 6 deleted files returned byte-identical, 2 files created after the backup survived, 1 locally-modified file was not overwritten, 43+2 arithmetic closed, and the restored bytes were served back over paperless's own HTTP API with matching sha256 (CAMPAIGN-9…:79-131). Campaign 9 A3: the entire appdata/paperless directory destroyed while the app ran (verified by a positive observable — doc download 200 → 404), then 43 fájl visszaállítva, documents_ok 16 of 16, and 200s with matching hashes again (:166-200). Re-verified post-fix in v0.183.0 (felhom.eu/REPORT.md:61).

A.1.3 — „Visszaállítás a távoli tárolóból" — offsite restore to a verification copy

field	detail
Name (UI, HU)	„Visszaállítás a távoli tárolóból" / „Teljes visszaállítás előkészítése" — page „Biztonsági mentés → Visszaállítás" and the per-app wizard /backups/restore/app
Entry point	POST /backup/offbox/restore (app, `mode=unit
Source	Tier-3, restic restore latest --tag <stack> from the Hetzner Storage Box over SFTP. Writes to an on-data-drive scratch <nsRoot>/backups/offsite-restore/<app> — never cfg.Paths.DataDir (the guest rootfs; the F-A1 filler), offbox_restore.go:136-140.
Scope	mode=unit (default) restores the recovery unit only via --include <absolute-unit-path>; mode=full restores unit + mandatory userdata behind a size×1.1 headroom gate, fail-closed on unknown size (:177-181).
Semantics	NON-DESTRUCTIVE. Nothing live is touched; this only materialises a verification copy.
Invocable by	CUSTOMER, UNASSISTED
Prerequisites	offsite configured and enabled; escrow ceremony completed (see Part C); restic repo reachable and unlocked (C9-F3); a data drive with headroom; no concurrent op; 30-minute background context (offbox_handlers.go:335)
Proven	PROVEN (bytes). Campaign 8 Phase C, R-87: bookstack snapshot b2d8936d, 155.5 MB, 14 s, via the real two-step size gate — 6/7 files byte-identical by sha256 including a 160 MB MariaDB volume dump; the 7th differs only in controller_version/created_at, i.e. the restore correctly returned the snapshot's manifest (CAMPAIGN-8…:246). Campaign 8 also records what this does not prove: "Restoring bytes is not the same as restoring service, and only the bytes were proven" (:255-258).

A.1.4 — „Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" — offsite place-to-live

field	detail
Name (UI, HU)	„Helyreállítás az élő adatok közé (csak a hiányzó fájlok)"
Entry point	POST /backup/offbox/place (app) — form backups_restore_wizard.html:93; route server.go:484-485; handler offbox_handlers.go:459; engine offbox_restore.go:357 PlaceOffsiteRestore
Source	a completed full offsite scratch (A.1.3 with mode=full)
Scope	every captured path mapped back to its live location; the recovery unit is placed only if the live unit is ABSENT (:352-356). Named-volume tars are not unpacked; no DB is replayed.
Semantics	ADDITIVE / MISSING-ONLY — rsyncRestoreMissing, explicitly never rsyncMirror (--delete), offbox_restore.go:343-345
Invocable by	CUSTOMER, UNASSISTED
Prerequisites	a completed full scratch (OffboxFullScratchReady, :276); whole-placement refusal on any path escaping the anchor (mapOffsiteRestorePaths, :299-303) — no partial writes
Proven	NOT PROVEN as a distinct action. Campaign 8 lists "the place/reconstitute legs that turn restored bytes back into a running app" among the not-validated items (CAMPAIGN-8…:520). The C6 immich drill of 2026-07-20 that is cited as proven went through the reconstitute button (A.1.5), not this one.

A.1.5 — „Teljes visszaállítás (fájlok + adatbázis)" — offsite reconstitution

field	detail
Name (UI, HU)	„Teljes visszaállítás (fájlok + adatbázis)" / „Teljes visszaállítás indítása"
Entry point	POST /backup/offbox/reconstitute (app, confirm=1) — form backups_restore.html:121; route server.go:486-487; handler offbox_handlers.go:366; engine internal/backup/offbox_reconstitute.go:169 ReconstituteFromOffsite
Source	the chosen offsite snapshot's coherent pair, out of the scratch (not the live unit)
Scope	files overwritten to the snapshot's version + that snapshot's DB dump replayed into a DB-only bring-up + app restarted. The unit itself is deliberately skipped (offbox_reconstitute.go:284-289), so named-volume tars are never unpacked by this path.
Semantics	DESTRUCTIVE/REPLACING for files and the database, but never deleting: "NOTHING IS EVER DELETED. The file copy overwrites and adds; it never carries --delete" (:30-34).
Undo	a pre-restore-* safety dump of the live DB is taken and verified present on disk before anything stops; if it cannot be taken the operation refuses with zero changes (:32-38, :257-268). Fail-closed again if no DB service can be identified (:272-276).
Invocable by	CUSTOMER, UNASSISTED
Prerequisites	a completed full scratch; app deployed with a resolvable HDD path; explicit confirm=1 (a stray click cannot fire it, offbox_handlers.go:377-381); 60-minute background context (:388)
Proven	PROVEN-LIVE (2026-07-20), operator-driven, through the customer's own buttons. Photos deleted in immich's own UI and trash emptied, then restored: 40 file(s) placed, 1 DB dump replayed rc-0, 11 assets active, No schema drift detected, timeline confirmed — 00-capability-map.md:66,75 citing felhom-controller/REPORT.md §4e. Residual recorded in the map itself: the run was performed by the operator, not a customer.

A.1.6 — „Megosztások visszaállítása" — shares restore (two-step)

field	detail
Name (UI, HU)	„Megosztások visszaállítása" then the place action
Entry point	POST /backup/shares/restore (form backups_restore.html:93; route server.go:493-494; handler offbox_handlers.go:498; engine internal/backup/shares_restore.go:87) then POST /backup/shares/place (form :97; route :495; handler :524; engine PlaceSharesRestore)
Source	the latest offsite _shares restic snapshot
Scope	three things, in stated order of importance: the files (missing-only into each share's live folder), the definitions (merged back into the share registry), the credential (best-effort, into the samba named volume) — shares_restore.go:20-23
Semantics	scratch step non-destructive; place step ADDITIVE/MISSING-ONLY, never overwriting
Guard	a destination is written only if it resolves strictly inside a registered live storage root; a snapshot is treated as untrusted input; a .. segment is refused outright rather than Clean'd away (:25-28, :139-143)
Invocable by	CUSTOMER, UNASSISTED
Prerequisites	offsite configured; a _shares snapshot; at least one live registered storage root
Proven	PROVEN-LIVE 2026-07-18 — 00-capability-map.md:96: offsite snapshots e0b9d723 and 4e2b15ec carrying manifest + passdb.tar; the restore round-trip returned a deleted probe file byte-identical and a deleted share definition with its original flags, without overwriting live files.

A.1.7 — .fab bundle import

field	detail
Name (UI, HU)	the „Importálás" page (/import)
Entry point	page GET /import (server.go:526-527, handler internal/web/handler_export.go:115); start POST → apiImportStart (handler_export.go:327) → appexport.Exporter.StartImport; status apiImportStatus (:372)
Source	a .fab bundle discovered by appexport.ScanForBundles in the export dir of a registered drive (internal/appexport/restore.go:41-50); path validated by isValidExportPath before use (handler_export.go:353)
Scope	config + DB dump + volume tars + the export's data mounts; optionally password-encrypted
Semantics	destructive re-import of the bundled app (the P-FAB drill re-imported destructively)
Invocable by	CUSTOMER, UNASSISTED — but the upload half is qualified below
Prerequisites	the bundle must already be on a registered drive (or uploaded via the chunked upload path); the bundle password if encrypted
Proven	PROVEN-LIVE (partly). CAMPAIGN-6D P-FAB / Accept #1: 1.7 GB full circle, byte-identical, app boots; chunking proven at the real Cloudflare edge (00-capability-map.md:67). Still open in that same row: "the rendered browser file-picker upload leg is still Viktor's open full-circle test".

A.2 — Operator-only, via SSH / CLI (not reachable by any customer)

A.2.1 — scheduled restore-test (unattended, agent)

field	detail
Name	restore-test (no customer-facing name; operator sees restore_test_failed / restore_test_stale on the hub)
Entry point	scheduler inside the agent daemon; engine felhom-agent/internal/reconcile/restoretest.go:126 RunRestoreTest; status readable at GET /restore-test/status on the per-guest local API (internal/localapi/server.go:446); manual form felhom-agent --selftest=restore-test [-archive …] (cmd/felhom-agent/main.go:168)
Source	rotates oldest-proven-first across every configured tier (R-85): local (vzdump on /var/lib/vz) and felhom-pbs (the ep0 datastore over WireGuard)
Scope	whole guest: restore into a scratch VMID in the 990000–990009 band → boot → verify running → mount parity against the archive's own config → teardown
Semantics	non-destructive to the live guest — it restores into a fresh scratch guest and destroys it
Invocable by	automatic (agent scheduler) or operator via SSH (--selftest). Not customer-reachable at all.
Prerequisites	a candidate archive on the tier; the host-wide one-heavy-operation gate free (internal/localapi/server.go:767); scratch band free; teardown authority (see F-LEAK, Part G)
Proven	PROVEN-LIVE and currently running. LIVE, journalctl -u felhom-agent on demo-hp: 18:08:33 backup: scheduled restore-test passed archive=local:backup/vzdump-lxc-9201-2026_07_28-17_43_05.tar.zst duration_s=89.204697113. On demo-felhom, the offsite tier passed too: Jul 28 00:04:27 … restore-test passed archive=felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z duration_s=1101.047742991. LIVE restore-test-state.json shows both tiers tracked separately on both boxes.

A.2.2 — pct restore (manual whole-guest restore)

field	detail
Name	manual guest restore
Entry point	pct restore <TARGET-VMID> <volid> --storage <storage> --unprivileged 1, per documentation/runbooks/RUNBOOK-manual-guest-restore.md:80
Source	local:backup/vzdump-lxc-9201-*.tar.zst on the host, or felhom-pbs:backup/ct/9201/<ts> on ep0
Scope	whole guest: rootfs + mp0 /var/lib/docker + mp1 /mnt/sys_drive. mp8 /mnt/felhom-drives and mp9 /etc/felhom-bootstrap are host binds and are NOT in the archive (LIVE pct config 9201, both hosts; runbook :34-39).
Semantics	destructive/replacing at guest granularity
Invocable by	OPERATOR VIA SSH ONLY
Prerequisites	root on the Proxmox host; the runbook's three pre-checks; the bind-strip-and-re-add step
Named hazard	F-OPS: mp9 embeds the source VMID, so a restore to a different VMID can bind another guest's bootstrap credentials (RUNBOOK-manual-guest-restore.md:45-48). The agent's own restore paths neutralise binds; the manual path does not (:11-14).
Proven	PROVEN-LIVE. CAMPAIGN-2 T-P9-DESTROY-RESTORE (whole-guest pct restore of 9201 → running + healthy), 00-capability-map.md:68; and Campaign 8 Phase C restored a 5.82 GB vzdump into fresh CT 9299 and a 4.36 GB PBS snapshot into CT 9297, both with exact mount parity and unprivileged: 1 preserved (CAMPAIGN-8…:247-253). Neither restored guest was booted, deliberately, because of F-OPS (:589).

A.2.3 — proxmox-backup-client restore (raw PBS restore)

field	detail
Entry point	invoked by pct restore from a felhom-pbs: volid — visible verbatim in a LIVE failure line on demo-felhom: /usr/bin/proxmox-backup-client restore '--crypt-mode=encrypt' '--keyfd=13' ct/9201/2026-07-27T19:55:41Z root.pxar /var/lib/lxc/990000/rootfs --allow-existing-dirs --repository felhom@pbs!demo-felhom@10.77.0.1:felhom-offsite --ns demo-felhom. Also used directly by the runbook to read pct.conf out of a snapshot without restoring (RUNBOOK-manual-guest-restore.md:59)
Source	ep0 datastore felhom-offsite, per-customer namespace, over wg-felhom
Scope	whatever archive is named (root.pxar, pct.conf, catalog)
Semantics	destructive at the target path
Invocable by	OPERATOR VIA SSH ONLY — needs the box's PBS token and the per-customer encryption key (--crypt-mode=encrypt)
Prerequisites	WireGuard tunnel up to 10.77.0.1; token valid; the encryption key present in /etc/pve/priv/storage/felhom-pbs.enc (or recovered from escrow)
Proven	PROVEN-LIVE as part of A.2.1/A.2.2 above.

A.2.4 — restic restore (raw offsite restore)

field	detail
Entry point	the controller's own invocations only (internal/backup/offbox.go:593-608, env RESTIC_PASSWORD_FILE=<data>/offbox/repo_password, mandatory ConnectTimeout). A human can run the same binary inside the guest with the same password file.
Source	sftp:u629488-sub1@u629488-sub1.your-storagebox.de:23/home/felhom-repo (demo-felhom) / u629488-sub3 (demo-hp) — LIVE from each guest's settings.json
Scope	whatever the snapshot holds (see Part B)
Semantics	non-destructive to a target dir
Invocable by	OPERATOR VIA SSH ONLY for the raw form; the wrapped forms are A.1.3–A.1.6
Prerequisites	the repo password file, the SSH key, known_hosts, and — per C9-F3 — no stale exclusive lock
Proven	proven through the wrapped path (A.1.3). Raw restic restore was also the manual remedy used in Campaign 9 Phase F for the lock (restic unlock --remove-all).

A.2.5 — agent DR bring-up (--selftest=bring-up -mode dr)

field	detail
Name	guest-loss DR bring-up
Entry point	felhom-agent --selftest=bring-up -mode dr -archive <volid> -vmid <n> [-keep] (cmd/felhom-agent/main.go:168,171-172,174); engine internal/reconcile/bringup.go (ModeDRGuestLoss, :40)
Source	a customer backup archive (local or PBS)
Scope	restore → preserve continuity identity (hostname, host keys, and by default MAC; reset only what collides, :38-40) → size → attach mounts → start link-up. Keeps the guest on success.
Semantics	creates a new guest from an archive; destructive to the target VMID
Invocable by	OPERATOR VIA SSH ONLY
Prerequisites	archive; free VMID; restore storage; -mode dr requires the host-run seam (:264)
Proven	NEVER PROVEN. Campaign 8's still-not-validated list: "The agent's own DR bring-up path (ModeDRGuestLoss) — read in code, never executed" (CAMPAIGN-8…:522).

A.2.6 — host-loss DR plan (derive-only — executes nothing)

field	detail
Name	host-loss restore plan
Entry point	felhom-agent/internal/dr/plan.go:45 BuildRestorePlan, driven by a host_loss restore_directive arriving on the desired-state channel (:99-129)
Source	the hub's restore directive + the live DR recipe
Scope	per guest: restore coordinates + sizing; per drive: durable_id → expected mount
Semantics	NONE — it is DERIVE-AND-SURFACE ONLY. plan.go:1-4: "the plan is logged (and exposed for the report), never executed … The Consumer has NO restore/destroy API by construction, so 'execute nothing' is a structural guarantee."
Invocable by	nobody executes it; the operator reads it
Proven	unit-proven only (06-offsite-connectivity.md:327, S5 row). See Part D2 — on the live fleet the inputs this plan consumes are empty or wrong.

A.2.7 — escrow-driven secret recovery (the precondition for A.2.5/A.2.6)

field	detail
Entry point	--selftest=escrow-consume -blob <file> -fingerprint <fp> -keydest <path> and --selftest=identity-consume [-install-wg-key] (cmd/felhom-agent/main.go:186-189); repo-password re-injection into a guest via POST /backup/offbox/inject-password (server.go:503, handler offbox_handlers.go:189, engine internal/backup/offbox.go:541 InjectOffboxPassword)
Source	the hub's host_escrow blob + the customer's recovery code R (never held by the operator or the hub)
Scope	PBS client encryption key K; the identity bundle {tunnel_token, pbs_token, wg_private_key, restic_repo_password} (felhom-agent/internal/escrow/identity.go:26-39)
Invocable by	OPERATOR VIA SSH, with the customer present to supply R
Proven	ceremony + one-shot claim PROVEN-LIVE (00-capability-map.md:38). The consume side is proven at spike level (documentation/tests/slice10d-identity-restore-spike-findings.md); the destructive in-place drill is explicitly operator-gated and unrun (06-offsite-connectivity.md:327).

A.3 — Automatic self-healing (not restores, but recovery paths; recorded for completeness)

path	entry point	what it recovers	invocable by	proven
stale vzdump-lock recovery	felhom-agent/internal/localapi/stalelock.go:62 RecoverStaleLockedGuests — runs once at agent startup	a guest left backup/snapshot-delete locked by a reboot mid-vzdump, plus a dangling vzdump snapshot	automatic	PROVEN-LIVE (TESTRUN F2-b lineage)
guest-power watchdog	felhom-agent/internal/localapi/guestpower.go (60 s)	a guest that is onboot:1, stopped, unlocked, no vzdump in flight → started; bounded 3 attempts 1m/2m/4m	automatic	PROVEN-LIVE 2026-07-28: came back unattended in 120 s against the 587 s the original incident needed a human (OPEN-ITEMS.md:38)
boot-orphan sweep + drive-backed gate	controller bootrecon	apps not started after a guest reboot; R-55 leaves deliberately-stopped drive-backed apps stopped	automatic	PROVEN-LIVE 2026-07-21 (00-capability-map.md:70)
quiesce crash recovery	crash marker + Recover()	stacks left stopped by a SIGKILLed controller mid-quiesce	automatic	PROVEN-LIVE — recovered in 1 second after a real SIGKILL; Campaign 8 corrects the spec's premise: "defer played no part — it does not run on SIGKILL" (CAMPAIGN-8…:229-233)
restore-test scratch reaper	reconcile.Engine.Recover	a leaked scratch guest in the 990000 band	automatic	LIVE this session: Jul 28 18:10:25 recover: destroyed leaked restore-test scratch guest op_id=scratch-restore-990000-9 vmid=990000 on demo-hp

A.4 — The "invocable by" summary (the point of Part A)

tier	customer-unassisted restore exists?	what it can return
Tier-1 recovery unit	YES (A.1.1)	compose + DB dumps + named volumes — the only path that unpacks volume tars
Tier-2 cross-drive	YES (A.1.2)	file legs only — never the copy's own recovery-unit/ (C9-F4)
Tier-3 restic offsite	YES (A.1.3–A.1.6)	files + DB replay; never named volumes (offbox_reconstitute.go:284-289)
Plane-2 whole-guest, local vzdump	NO	operator via SSH only
Plane-2 whole-guest, PBS offsite	NO	operator via SSH only
Host state (Part D1)	NO	no restore path of any kind exists — see D1
Hub state (Part D2)	NO	Longhorn volume backup only, operator via kubectl/Longhorn UI

INFERENCE, stated as such: the customer can reach three tiers of app-level recovery and zero tiers of whole-guest or host recovery. Every path that returns a whole system is operator-only.

---

Part B — capture coverage per tier, per app

B.1 — Catalog enumeration (SOURCE, computed this session)

Method: parsed all 53 templates/*/.felhom.yml and templates/*/docker-compose.yml in app-catalog-felhom.eu@4252121, then applied the classifier's two-level default exactly as internal/appbackup/classify.go implements it (explicit backup: entry wins; else a :ro reader is excluded; else a writable bind is mandatory).

total templates                                   53
templates with a `backup:` block                  13
templates with >=1 Docker NAMED volume            52   (only bentopdf has none)
templates with a DB service (pg/mariadb/mysql/mongo) 14
templates binding ${HDD_PATH}/${USERDATA_PATH}/${IMPORT_PATH}  13  (exactly the block-carrying set)
templates with such a bind and NO block            0

Per-app classification of the 13 block-carrying apps (SOURCE, computed):

app	binds	mandatory	optional	excluded
audiobookshelf	2	0	1	1
calibre-web	1	1	0	0
emby	1	0	0	1
immich	2	1	1	0
jellyfin	1	0	0	1
komga	1	0	1	0
navidrome	1	0	0	1
nextcloud	1	1	0	0
paperless-ngx	2	1	0	1
plex	1	0	0	1
radarr	2	0	0	2
romm	2	0	1	1
sonarr	2	0	0	2

Derived counts (SOURCE):

• Tier-3 offsite filters to mandatory only (ComputeCaptureSet with TierOffsite, internal/appbackup/captureset.go:79; offsite call site internal/backup/offbox_capture.go:49). 4 of 53 templates ever push a userdata path offsite: calibre-web, immich, nextcloud, paperless-ngx. The other 49 are unit-only, byte-identical to pre-v0.134.0 (offbox_capture.go:30-31).
• Tier-2 filters to mandatory + optional (TierSecondary, internal/backup/tier2_capture.go:44). 7 of 53 templates get a hdd/ or userdata/ leg: the 4 above plus audiobookshelf, komga, romm. The legacy resolver path (tier2_capture.go:72-88) adds legs for apps with an appdata/<name> dir — 0 additional templates qualify, because no non-block template binds a ${HDD_PATH}/${USERDATA_PATH} path at all.
• 6 of 13 block-carrying apps declare only excluded binds (emby, jellyfin, navidrome, plex, radarr, sonarr) — they carry a block and still get no automatic file leg on any tier.

This does not reconcile with the shipped C9-F1 Phase-0 figure. felhom.eu/REPORT.md:17-24 and OPEN-ITEMS.md:31 record A=9 / B=43 / C=1. My independent enumeration gives A=7 / B=45 / C=1 by the same definition ("templates whose Tier-2 copy can hold a readable file leg"). The difference is 2 apps and I could not identify them. Logged in Contradictions below; do not treat either number as settled until the two methods are diffed.

B.2 — What each tier captures, and where capture and restore disagree

Tier-1 — recovery unit (per app, on the app's own drive)

Location	<nsRoot>/backups/primary/<stack>/
Captures	compose/docker-compose.yml, compose/.felhom.yml, compose/app.yaml (secret-stripped), db-dumps/*.sql, volume-dumps/*.tar, manifest.json — recovery_unit.go:71-74,113-131
Written by	the nightly DB-dump leg and a periodic status refresh (captureAllRecoveryUnits, recovery_unit.go:185), checksum-skipped when nothing changed (:76-77)
Restore reads	all of it (A.1.1)
Capture ⟷ restore disagreement	none in scope, but one in validation: CaptureRecoveryUnit gates its write on a config checksum change alone — no run-state check, no health check, no test that the pinned images resolve (recovery_unit.go:78-130; Campaign 9 O-1, CAMPAIGN-9…:538-584). ListRestorePoints returns exactly one keep-side restore point per app (restore_points.go:14-18), so an unvalidated refresh replaces the only one there is. Campaign 9 measured the compensating control live: the catalog git-sync reverted the drift in ~3 minutes and the unit followed.

Tier-2 — cross-drive secondary copy

Location	<target nsRoot>/backups/secondary/<stack>/ with the v2 layout: .felhom-tier2-layout, recovery-unit/, hdd/<relpath>/, userdata/<relpath>/ (07-backup-architecture.md:288-294; writer tier2.go:351-393)
Captures	always a full mirror of the recovery unit (tier2.go:368-369, comment "Unit leg (always)"; OPEN-ITEMS cites this as tier2.go:369) plus the TierSecondary capture legs
Restore reads	hdd/ and userdata/ only (tier2_restore.go:101-104)
Capture ⟷ restore disagreement	This is the asymmetry. Two distinct defects live here: C9-F1 (shipped-fixed v0.183.0) — for the 45-of-53 apps with no leg, the button restored 0 files, took a real outage, and reported „Nincs hiányzó fájl — minden fájl megvan a helyén."; and C9-F4 (OPEN) — "Nothing reads the Tier-2 copy's recovery-unit/ mirror" (OPEN-ITEMS.md:35). RecoveryUnitPath resolves to backups/**primary**/ (appbackup/paths.go:46-48) and the only reader of the secondary tree is tier2_restore.go:79. Tier-2 exists for the case where the primary drive is lost — and in exactly that case the primary unit is gone while this mirror survives, unreachable by any customer action.
LIVE confirmation of the asymmetry	demo-felhom backups/secondary/bookstack/ and .../docmost/ contain recovery-unit and nothing else (no hdd/, no userdata/) at 156 MB and 86 MB respectively; calibre-web has recovery-unit userdata; immich has hdd recovery-unit. demo-hp paperless-ngx has hdd recovery-unit.

Tier-3 — restic offsite

Location	sftp:u629488-sub<N>@u629488-sub<N>.your-storagebox.de:23/home/felhom-repo, one multi-path snapshot per app per run, tagged felhom-offbox + <stack>
Captures	the recovery unit (which itself contains DB dumps and named-volume tars) + mandatory binds only (offbox_capture.go:28-31,49); plus a separate _shares snapshot
Restore reads	scratch restore reads whatever the snapshot holds (A.1.3); place and reconstitute both skip the unit when writing to live — PlaceOffsiteRestore places it only if the live unit is absent (offbox_restore.go:352-356), and ReconstituteFromOffsite skips it outright (offbox_reconstitute.go:284-289)
Capture ⟷ restore disagreement	The named-volume tars are captured on every offsite run and unpacked by no offsite action. The DB is replayed (from the scratch unit's db-dumps/, offbox_reconstitute.go:315), but the volume tars are not. Since 52 of 53 templates keep data in named volumes (B.1), for most apps the offsite tier carries the volume data and cannot itself put it back. A two-step route exists — full restore → place (which lands the unit if absent) → then the Tier-1 restore, which does unpack tars — but no single action does it and no UI routes it (the C9-F1b shape, one tier over). This is my own enumeration; it is not currently filed as a finding.
Silent-gap guard that IS present	a declared-but-absent mandatory path is stat-filtered before argv, because restic 0.14.0 skips a missing source with a stderr warning and exits 0 (07-backup-architecture.md:110-118; implemented offbox_capture.go:61-70)

Plane-2 — whole-guest (local vzdump and PBS): do they differ?

They differ in three ways, all verified LIVE.

	local (local, /var/lib/vz/dump)	offsite (felhom-pbs → ep0 felhom-offsite)
cadence	backup_cadence_seconds: 0 ⇒ 24 h	604800 = 7 days (LIVE, /etc/felhom-agent/agent.json backup.backup_targets[], identical on both hosts)
retention	local_backup_retention: 3	keep_last: 0 box-side; retention is server-side on ep0 (LIVE proxmox-backup-manager prune-job list: prune-demo-felhom and prune-demo-hp, both 03:30, keep-last 2)
encryption	none — plain .tar.zst on the host	client-side, per-customer key (encryption-key in /etc/pve/storage.cfg, distinct per host)
transport	local disk write	WireGuard 10.77.0.1
verification	none	verify-new 1 on the datastore + the agent's own VerifyLoop (6 h, internal/pbs/verify.go:13)
contents	identical — both are vzdump of the same guest	same

Contents, identical for both and verified LIVE against pct config 9201 on both hosts:

rootfs  local-lvm:vm-9201-disk-0                                          IN
mp0     local-lvm:vm-9201-disk-1  mp=/var/lib/docker      backup=1        IN
mp1     local-lvm:vm-9201-disk-2  mp=/mnt/sys_drive       backup=1        IN
mp8     /mnt/felhom-drives        mp=/mnt/felhom-drives                   OUT (host bind)
mp9     /var/lib/felhom-agent/guests/9201/bootstrap  ro=1                 OUT (host bind)

Consequence, stated as fact: the whole-guest tiers carry /var/lib/docker — which is where the Docker named volumes and the controller's own data volume live — and do not carry the customer's data drives at all. On demo-felhom that is the 916 GB hdd_1; on demo-hp the 938 GB nvme-1tb.

B.3 — Approximate data volume per app class, on the live fleet (LIVE)

All figures from du -sh inside guest 9201 on each host, 2026-07-28 ~17:44 UTC.

demo-felhom (namespace roots: /mnt/felhom-drives/hdd_1 and /mnt/sys_drive/felhom-data)

item	size
hdd_1/appdata/immich	252 M
hdd_1/appdata/nextcloud	63 M
hdd_1/appdata/paperless	24 K
hdd_1/appdata/romm	12 K
hdd_1/userdata/media	472 K
hdd_1/userdata/{documents,downloads,roms}	4 K each
hdd_1/shares/Share	28 K
Tier-1 units hdd_1/backups/primary/immich	1.3 G
Tier-1 units hdd_1/backups/primary/calibre-web	388 K
Tier-1 units sys_drive/…/primary/bookstack	157 M
Tier-1 units sys_drive/…/primary/docmost	88 M
Tier-1 units — 19 further apps	24 K each (config-only; never deployed)
Tier-2 hdd_1/backups/secondary/bookstack	156 M (recovery-unit ONLY)
Tier-2 hdd_1/backups/secondary/docmost	86 M (recovery-unit ONLY)
Tier-2 sys_drive/…/secondary/immich	1.6 G (hdd + recovery-unit)
Tier-2 sys_drive/…/secondary/calibre-web	844 K (recovery-unit + userdata)
Tier-2 sys_drive/…/secondary/_shares	884 K
Tier-3 restic repo	1,103,918,002 B = 1.0 GB, 32 snapshots (from settings.json)
Plane-2 local	3 archives; newest 5,929,975,285 B (15 GiB written → 5.52 GB compressed)
Plane-2 PBS	3 snapshots in ns demo-felhom, oldest 2026-07-27T17:34:09Z
guest /var/lib/docker used	12 G of 197 G
guest /mnt/sys_drive used	1.8 G of 50 G
data drive hdd_1 used	3.4 G of 916 G

demo-hp (namespace roots: /mnt/felhom-drives/nvme-1tb and /mnt/sys_drive/felhom-data)

item	size
nvme-1tb/appdata/paperless	8.6 M
nvme-1tb/userdata/media	36 K
Tier-1 nvme-1tb/backups/primary/paperless-ngx	78 M
Tier-1 sys_drive/…/primary/paperless	680 K
Tier-1 sys_drive/…/primary/uptime-kuma	24 K
Tier-2 sys_drive/…/secondary/paperless-ngx	86 M (hdd + recovery-unit)
Tier-3 restic repo	38,041,506 B = 36.3 MB, 8 snapshots
Plane-2 local	3 archives; newest 1,679,958,230 B (4.4 GiB written → 1.56 GB compressed)
Plane-2 PBS	6 snapshot dirs in ns demo-hp, one of which is the R-99 phantom (below)
guest /var/lib/docker used	5.3 G of 50 G
data drive nvme-1tb used	89 M of 938 G

ep0 (LIVE): datastore felhom-offsite on a dedicated 98 G volume, 14 G used, 14 %; per-namespace du: demo-felhom 568 K, demo-hp 444 K (index metadata only — the chunks are shared in .chunks/). The 13 G rollback copy /srv/pbs-felhom is present on the 38 G root disk (55 % full).

Fleet reality check, INFERENCE: the largest single app-data object on the fleet today is immich's 252 MB appdata and its 1.3 GB Tier-1 unit. Nothing on the live fleet exercises the WAN-restore regime the architecture must answer for — there is no multi-hundred-GB media library anywhere, and both data drives are >99 % empty. Every measured timing in Part F is therefore a small-data timing.

---

Part C — keys, secrets, and what recovers them

A copy you cannot decrypt is not a copy. Locations, counts, and recovery routes only; no values.

C.1 — Inventory

#	secret	protects	where it lives (LIVE unless noted)	independent copies	recovers it if primary is lost	exercised?
1	restic repo password	all Tier-3 offsite ciphertext	guest …/felhom-controller-data/_data/data/offbox/repo_password, 64 B, 0600 root — LIVE on both boxes. Auto-generated once, 256-bit hex, never logged (offbox.go:392-400,405-410)	3 conceptually: the live file · inside every whole-guest archive (it sits under /var/lib/docker = mp0, backup=1) · inside the hub's host_escrow.identity_blob as IdentityBundle.ResticRepoPassword (felhom-agent/internal/escrow/identity.go:34-38). Plus a hash-only copy on the hub: host_escrow.restic_pw_sha256, present for both hosts (LIVE)	whole-guest restore, or escrow-consume with the customer's R, or POST /backup/offbox/inject-password (offbox.go:541)	escrow creation proven live; consume never exercised destructively
2	PBS datastore encryption key K	all Plane-2 offsite ciphertext	host /etc/pve/priv/storage/felhom-pbs.enc, 255 B, 0600 root:www-data — LIVE on both hosts; its fingerprint is also in /etc/pve/storage.cfg	2: the live file · the hub's host_escrow.blob (383 B on both hosts, posture zero_knowledge, LIVE)	escrow-consume with R (--selftest=escrow-consume -keydest …)	ceremony proven; consume proven at spike level only
3	PBS access token secret	write access to the box's own namespace	host /etc/pve/priv/storage/felhom-pbs.pw, 37 B, 0600 root:www-data — LIVE both hosts	3: live file · IdentityBundle.PBSToken in escrow · the hub can re-mint (host_pbs_secrets, LIVE: one consumed row per host, generation 1)	hub re-issue → descriptor secret_generation bump → agent re-applies	PROVEN-LIVE 2026-07-21, operator click to converged in 13 s (00-capability-map.md:39)
4	agent's hub API key	the box's identity to the hub	host /etc/felhom-agent/agent.json, 0600 felhom-agent — LIVE both hosts; also hosts.api_key on the hub	2	hub-side re-enrolment	not exercised as a recovery
5	agent TLS key for the local API	controller↔agent channel	host /var/lib/felhom-agent/local-api.key, 227 B, 0600 — LIVE	1 (+ the pinned cert in the guest bootstrap)	regenerate + re-pin	n/a
6	guest bootstrap token	the controller's hub identity + local-API token	host /var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json, 380 B (demo-felhom) / 373 B (demo-hp), 0600 uid 100000 — LIVE; bind-mounted ro into the guest at /etc/felhom-bootstrap	1 on the host only — this is mp9, which is NOT in any guest archive	re-provisioning back-half writes it	never rebuilt from nothing
7	WireGuard private key (box)	the offsite transport	host /etc/wireguard/wg-felhom.conf, 327 B, 0600 root + /var/lib/felhom-agent/wg/ — LIVE both hosts	2: live · IdentityBundle.WGPrivateKey in escrow (identity.go:29-32)	escrow-consume -install-wg-key (create-only), or fresh-key re-registration keeping the /32 (06-offsite-connectivity.md:327)	unit-proven
8	offbox SSH key	reaching the Storage Box	guest …/data/offbox/ssh_key, 400 B, 0600 — LIVE both boxes; known_hosts 290 B 0644 beside it	2: live · inside whole-guest archives	hub re-provisions the sub-account credential	Hetzner sub-account provisioning PROVEN-LIVE (00-capability-map.md:129)
9	app.yaml encryption key	every per-app secret	guest …/data/encryption.key, 32 B, 0600 — LIVE both boxes; created at first boot (internal/crypto/crypto.go:20-39)	2: live · inside whole-guest archives	whole-guest restore only	never exercised as a recovery
10	per-app secrets (DB_PASSWORD, SECRET_KEY, APP_SECRET, …)	app data + app login	guest /opt/docker/stacks/<app>/app.yaml, AES-256-GCM ENC: under #9	1 logical copy, protected by #9	recovered from the guest at restore time (restore_unit.go:132); regenerated only for resettable secrets, never for data keys	partially — data-key gate never fired in anger
11	dashboard password hash	the whole customer UI	guest …/data/settings.json → password_hash (bcrypt) with controller.yaml → web.password_hash as fallback (auth.go:34-35)	2 + archives	hub-issued reset code — POST /claim/request-new-code	PROVEN-LIVE: hash applied 1 s after request, code accepted first try (00-capability-map.md:101)
12	household SMB password	the „Megosztás" shares	never persisted as plaintext (STDIN→smbpasswd); the passdb lives in the samba named volume; a copy is staged at guest …/data/shares-payload/passdb.tar — LIVE, 855,040 B, 0600, on both boxes	3: samba volume · shares-payload/passdb.tar · the offsite _shares snapshot	shares place-to-live restores the credential best-effort (shares_restore.go:20-23)	PROVEN-LIVE 2026-07-18
13	launcher share token + its bcrypt password hash	the /s/<token> guest launcher	settings.json → launcher_share_token, launcher_share_password_hash — LIVE, present on demo-felhom only	1 + archives	rotate (the customer can)	n/a
14	web.session_secret	session + CSRF + the share cookie HMAC	controller.yaml — LIVE both boxes	1 + archives	regenerate (invalidates sessions and share cookies)	n/a
15	Cloudflare tunnel token + CF API token	public reachability of every app	controller.yaml infrastructure.cf_tunnel_token / cf_api_token — LIVE both boxes; also on the hub in customer_configs.config_json.infrastructure (LIVE, both keys present for demo-felhom)	2 + archives	the hub re-serves them on config pull	continuously exercised
16	root@pam console password (break-glass)	PVE web console when SSH is dead	hub only — host_recovery table, LIVE: 3 rows, all root@pam, 32-byte secrets, set_at 2026-07-18/07-21/07-25	1 (hub) + whatever the operator saved out-of-band	GET /api/v1/admin/hosts/<id>/recovery-credential with the global operator key (break-glass.md:42-48)	PROVEN-LIVE (the sshd incident)
17	hub global bearer (REPORT_API_KEY)	operator-tier hub API	k8s Secret/report-api + the operator's password manager	2	recreate from the out-of-band store	rotation runbook written, not yet run (secrets.md:107-117); the git-history copy stays alive until rotated (:76-77)
18	Resend API key	all email — every notification and every claim/reset code	k8s Secret/resend-api + password manager + Gmail "Send mail as"	3	recreate from the store	rotation procedure written and ordered
19	Hetzner API token + pool box id	Storage Box provisioning	k8s Secret/storagebox (HETZNER_TOKEN, HETZNER_POOL_BOX_ID) — LIVE from the hub Deployment env	1 + password manager (assumed)	operator re-mints at Hetzner	n/a
20	three SSH private keys held by the hub	ep0 peer-sync, tenantsync, and the desired-state poke	hub pod files (WG_ENDPOINT_SSH_KEY_FILE, TENANTSYNC_SSH_KEY_FILE, POKE_SSH_KEY_FILE) + Secret/wg-endpoint-ssh for the pinned host key — LIVE	1 each	re-provision from the offsite-endpoint runbook	n/a
21	operator OOB SSH pubkey	operator access to felhom-sshd on every box	hub_settings.oob_operator_ssh_pubkey (LIVE) → pushed to /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op (93 B, LIVE both hosts)	2	hub re-push	continuously exercised
22	registry credentials	pulling controller/agent images	Secret/gitea-creds	1	re-mint in Gitea	n/a
23	customer retrieval password	pulling controller.yaml from the hub	customer_configs.retrieval_password on the hub; settings.json.retrieval_password in the guest	2	operator resets on the hub	exercised at every day-0
24	the customer's recovery code R	unwraps #1, #2, #3, #7	nowhere in the system, by design — the operator does not have it, the hub does not have it, the box does not retain it	0 system copies	nothing. If the customer loses R, the escrow blobs are unopenable.	the one-shot claim is PROVEN-LIVE; loss has never been exercised

C.2 — Secrets with exactly one copy

secret	why it is single-copy	consequence
#6 guest bootstrap token (mp9)	it is a host bind mount and therefore in no guest archive — verified LIVE against pct config 9201 on both hosts	a whole-guest restore onto a fresh host produces a guest whose bootstrap dir must be rebuilt by the provisioning back-half; the manual runbook warns never to copy one from another guest because it carries that customer's tokens (RUNBOOK-manual-guest-restore.md:97-100)
#5 agent local-API TLS key	regenerable, but the guest pins it	a regeneration requires re-pinning in the guest bootstrap
#16 break-glass console password	hub-only by design (break-glass.md:89: "The password is never logged or filed — it exists only in the hub vault")	if the hub is gone, the break-glass password is gone — see C.3
#24 recovery code R	zero system copies by design	irreducible; the premium operator-custody tier is the stated alternative (05-hub-architecture.md:192)
#19–#22 hub-held operator secrets	one k8s Secret each, backed up only via the hub PVC (Part D2)	recreate from the out-of-band store; see Part E

C.3 — Circular dependencies (a recovery route that depends on something it also protects)

Three found. All three are SOURCE/LIVE-verified, not inferred from docs.

C-1 — App-level restore cannot reconstitute an app without the guest that was lost. Every tier's recovery unit is secret-free by design (recovery_unit.go:73: "It NEVER writes a secret value"), and RestoreFromRecoveryUnit recovers secrets from the guest's live app.yaml (restore_unit.go:130-132), which is encrypted under encryption.key — a 32-byte file that exists only inside the guest (LIVE) and only in whole-guest archives. reconcileRestoreSecrets states the consequence outright (restore_unit.go:18-22): a missing data-encrypting key is FATAL and "we refuse and tell the operator to do a PBS whole-guest restore." So Tier-1, Tier-2 and Tier-3 are all conditioned on the whole-guest tier having survived. They are app-recovery tiers, not system-recovery tiers, and the code says so.

C-2 — The offsite tier's own key rides only in tiers the same disaster removes. The restic repo password (#1) has three copies: the guest file, the whole-guest archives, and the escrow blob. The first two are lost with the guest and the host. The escrow blob is the only copy that survives a host loss — and opening it requires the customer's R (#24), which the system holds zero copies of. So offsite-only recovery is possible exactly when the customer can produce R, and impossible otherwise, regardless of how healthy the ciphertext is. This is the intended zero-knowledge posture; it is recorded here because it is the recovery path's actual precondition and it has never been exercised end to end (A.2.7).

C-3 — The break-glass credential is protected by the plane it exists to rescue. host_recovery lives only in the hub DB (LIVE: 3 rows). Retrieving it needs an authenticated call to hub.felhom.eu with the operator's global key (break-glass.md:42-48). The hub runs on DooPlex; hub.felhom.eu resolves to DooPlex's own home IP (Part E). If DooPlex is down, the break-glass password for every customer host is unreachable — including in the scenario where you need it because a host's SSH is dead. The runbook's own §5 note records the historical instance of this shape: "A working break-glass path independent of sshd must always exist" (:73-77) — the same argument applies one level up, to the hub.

---

Part D — host state and hub state

D1 — What lives on the host and is in no guest backup

Everything proven so far restores a guest. The following was inventoried LIVE on both hosts. None of it is inside any guest archive: mp8 and mp9 are host binds and out of vzdump scope, and nothing else on the host root filesystem is captured by anything at all.

#	host artefact	LIVE evidence	captured anywhere?	reconstructible from golden + provisioning?
1	agent binary + felhom-agent.service, .service.d/, felhom-agent-rollback.service	/etc/systemd/system/ — 9 felhom units on demo-felhom, 11 on demo-hp	NO	yes — host-install + agent self-update A/B slots
2	/etc/felhom-agent/agent.json (hub key, PBS/backup config, WG, escrow, oob, privileged)	2465 B (demo-felhom) / 2459 B (demo-hp), 0600 felhom-agent; plus 6 ad-hoc .bak/.campaign* copies on each host	NO	partially — the hub re-serves desired state, but hub.api_key is identity
3	bootstrap.json (mp9 source)	/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json, 380 B / 373 B, 0600 uid 100000	NO — explicitly out of every archive	yes, by the provisioning back-half; never by a restore
4	island bridge vmbr9 + /etc/network/interfaces	both hosts carry auto vmbr9 / 169.254.253.1/30 / bridge-ports none; guest net1 is 169.254.253.2/30	NO	yes — RUNBOOK-island-migration.md, host-install v1.19.0+
5	sudoers grants	/etc/sudoers.d/felhom-agent 21,833 B and /etc/sudoers.d/felhom-op 1,241 B, 0440, identical byte-size on both hosts	NO	yes — host-install writes them; v1.21.0 added the 990000..990009 band
6	PBS storage config /etc/pve/storage.cfg + /etc/pve/priv/storage/felhom-pbs.{enc,pw}	LIVE; the .enc is the per-customer client key	NO (/etc/pve is not in any guest archive)	key only via escrow + R; the descriptor via hub desired-state
7	WireGuard /etc/wireguard/wg-felhom.conf	327 B 0600 on both hosts	NO	key via escrow; peer entry survives on the hub (wg_peers, LIVE 4 rows)
8	onboot flags	LIVE onboot: 1 on both 9201s	inside the archived pct.conf	yes
9	cron/systemd units for the mgmt-plane watchdog	felhom-mgmt-watchdog.{service,timer}, felhom-privsep.tmpfiles, felhom-oob-nft.service, felhom-shared-parent.service, felhom-sshd.service	NO	yes — host-install
10	agent state dir /var/lib/felhom-agent/	LIVE: journal.log (9.7 K / 15.2 K), restore-test-state.json, guest-binds.json, drive-intents.json, format-job.json, controller-swap-9201.json, local-api.{crt,key}, local-tokens.log, nonces.log, pbsdr/marker.json, units/*.mount, felhom-sshd/{authorized_keys.felhom-op,port,sshd_config}, wg/	NO	mostly regenerable; local-api.key and the pinned cert are not (C.2)
11	/var/lib/felhom-install/state.json	present on both hosts	NO	the break_glass marker matters — a from-scratch reinstall wipes it and re-randomises root@pam (break-glass.md:83-88)
12	the local vzdump archives themselves	/var/lib/vz/dump	they are the backup; nothing backs them up	n/a
13	/mnt/felhom-drives/* — the customer's data drives	916 G hdd_1 / 938 G nvme-1tb, bound via mp8	NOT in any whole-guest archive; only their felhom-managed subtrees ride Tier-2/Tier-3	the drives are physical

Has a host ever been rebuilt from nothing and verified?

A host has been installed from nothing many times, on two different physical boards. A host has never been rebuilt from nothing — i.e. no host has ever been reconstructed as its former self, with its identity, keys and guests, and then verified. The distinction is exactly the provision-vs-DR split in bringup.go:36-40, and only the ModeProvision half has ever run.

Evidence for both halves:

• Install from nothing: PROVEN-LIVE on two boards — N100 2026-07-18 and HP t740 2026-07-21, each a single unattended pass from a virgin ISO to a claimed, running box (00-capability-map.md:34,37).
• Rebuild as its former self: ModeDRGuestLoss "read in code, never executed" (CAMPAIGN-8…:522); the destructive S5 host-loss drill is operator-gated and unrun (06-offsite-connectivity.md:327); and the host-loss plan builder executes nothing by construction (dr/plan.go:1-4).

D2 — What the hub holds that exists nowhere else

Hub DB: /data/hub.db, 113,033,216 B, on PVC hub-data (1 Gi, Longhorn, RWO, bound 162 d). Row counts from the 17:40:00 UTC copy:

customer_configs 4    hosts 3     guests 5     host_escrow 2   host_escrow_superseded 0
host_recovery 3       host_pbs_secrets 2       dr_recipe 6     wg_endpoints 1   wg_peers 4
events 1617           host_reports 1760        reports 8010    notification_log 314
one_time_secrets 3    signed_jobs 0            hub_settings 9  appliance_registrations 8
selfbind_tokens 2     customer_claims 4

D2.1 — What exists nowhere else

hub-only state	why it is hub-only	LIVE detail
host_escrow — the R-wrapped PBS key + identity bundle	the box produces it and then cannot open it; the hub is the custody store	2 rows (demo-felhom-8363b5 2026-07-21, demo-hp-bb76ea 2026-07-23), both posture=zero_knowledge, blob 383 B, identity blob 572 B, restic_pw_sha256 present, stale_at NULL
host_recovery — break-glass root@pam per host	never logged or filed anywhere else by design	3 rows
host_pbs_secrets — the one-time PBS token secret	mint-and-consume	2 rows, both consumed, generation 1
customer_configs — identity, domain, email, retrieval password, config_json incl. both Cloudflare tokens and the full offsite descriptor, min_controller_version, config_version, dr_tier	the box only ever receives a rendered copy	4 customers: demo-felhom (dr_tier 1, cfg v12), peti-felhom (dr_tier 0, v6), demo-hp (dr_tier 1, v3), drill-r50 (blocked, v1)
hosts.desired_json + desired_generation — operator intent	authored here	generations 8 / 5 / 1
wg_endpoints + wg_peers — the offsite peer registry	the endpoint is dumb and hub-driven (06-…:34)	endpoint ep0 ep0.felhom.eu:443, subnet 10.77.0.0/24, pbs tunnel ip 10.77.0.1; peers .2 demo-felhom, .3 demo-hp, .4 drill-r50, .250 operator-oob
hub_settings — fleet floor + signed artifact digests + operator password hash + operator OOB pubkey	the trust root the host bootstrap verifies against	min_controller_version 0.156.0, artifact_golden_version 0.161.0, artifact_agent_version 0.96.0, artifact_min_agent 0.93.0
events / notification_log / host_reports / reports — the operator's only fleet-wide history	prunable by age, and stated to be "convenience, not the DR source of record" (05-hub-architecture.md:170-171)	1617 / 314 / 1760 / 8010 rows
customer_claims, selfbind_tokens, appliance_registrations, one_time_secrets	onboarding lifecycle	4 / 2 / 8 / 3

D2.2 — Is it backed up? (LIVE)

Yes, thinly, and to the same machine.

PVC hub-data -> volume pvc-486c9809-4672-4b56-b70e-0bf01d0c3628
labels: recurring-job-group.longhorn.io/default = enabled          <- in scope
recurringjobs: backup-daily  0 4 * * *  RETAIN 1
               backup-weekly 0 5 * * 0  RETAIN 1
backuptarget "default": nfs://192.168.0.180:/mnt/5_hdd/backup/longhorn-pvc   available=true
backups of hub-data actually present:  2
  2026-07-26T03:06:48Z  Completed  425,721,856
  2026-07-28T02:07:11Z  Completed  425,721,856

Facts that follow, stated plainly:

1. The backup target is DooPlex itself. 192.168.0.180 is the machine running k3s, the hub, and the Longhorn replicas. The replica data is on /dev/sdb1 (/var/lib/longhorn, the 477 G root SSD, 81 % full); the backup target is on /dev/sda1 (/mnt/5_hdd, a 9.1 T HDD). Different physical disks, same host, same building, same power, same operator uplink.
2. Exactly two restore points exist, 2 days apart, because both recurring jobs use RETAIN 1.
3. There is no off-machine copy of the hub DB that I could find. Searched: Longhorn backuptargets (one, NFS to DooPlex), Longhorn recurringjobs (two, both default group), the hub Deployment's volumes, and documentation/runbooks/ for any hub-DB export procedure.

D2.3 — Two hub-held DR records that are empty or wrong on the live fleet

Both are LIVE reads from the hub DB copy, cross-checked against ep0.

D2.3-a — hosts.dr_record_json is {} for every host.

demo-felhom-8363b5  {}
demo-hp-bb76ea      {}
drill-r50-0a4f9a    {}

05-hub-architecture.md:175-176 describes this column as "a slim DR record on the hosts row (PBS namespace + repo fingerprint + the wrapped escrow key)" and :186 names it as one of the four durable sources host-loss recovery reads from. On the live fleet it holds nothing. The escrow itself is stored — in host_escrow, not here — so this may be a doc-vs-implementation drift rather than data loss; it is recorded as a fact, not diagnosed.

D2.3-b — host_escrow.directive_json is {} for both escrowed hosts.

The non-secret DR directive (PBS repo/ns, expected fingerprint, tunnel id) that documentation/architecture/06-offsite-connectivity.md:148-150 says was uploaded with the escrow is 2 bytes — an empty object — on both rows.

D2.3-c — the DR recipe's PBS namespace is wrong on every box.

dr_recipe.host_half.pbs  (demo-felhom, updated 2026-07-28 17:31:13)
  { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" }
dr_recipe.host_half.pbs  (demo-hp, updated 2026-07-28 17:25:31)
  { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" }

felhom-agent/internal/hub/dr_recipe.go:55 documents that field as "PBS namespace the restore targets". The real namespaces, verified LIVE on ep0, are demo-felhom and demo-hp (/mnt/pbs-datastore/ns/{demo-felhom,demo-hp}), and the live pct restore command line quoted in A.2.3 carries --ns demo-felhom. Traced to source: Snapshot.Namespace is decoded from a ns JSON field (internal/pbs/client.go:97) that PBS does not echo per item when the request is already namespace-scoped via ?ns= (:117-120), so the value is empty and ToHub normalises empty to "root" (internal/pbs/report.go:23-27), which latestPBSCoord then writes into the recipe (dr_recipe.go:149). latest_snapshot_id: "9201" is not a defect — it is BackupID, documented as "a coordinate" (dr_recipe.go:56).

D2.3-d — the DR recipe's drive list is empty on every box, including boxes with enrolled drives.

"drives": [] for demo-felhom, demo-hp, peti-felhom and drill-r50. BuildDRRecipeHostHalf populates it from storage targets where isUserDataDrive(t) holds — type must be usb or local-dir and both DurableID and MountPath must be non-empty (dr_recipe.go:129-136). demo-felhom has an enrolled 916 G USB HDD at /mnt/felhom-drives/hdd_1 and demo-hp an enrolled 938 G NVMe at /mnt/felhom-drives/nvme-1tb (both LIVE). dr.PlannedDrive is the "re-attach BY durable_id (the wrong-disk guard)" half of the host-loss plan (dr/plan.go:34-35); with an empty list that half of the plan is empty. Not diagnosed here — the storage-target type mapping was not traced end to end.

D2.4 — What happens to a customer box if the hub is gone?

Answered from SOURCE plus one LIVE observation. Split by duration.

For a day. Nothing customer-visible breaks. The box's data plane is entirely local: apps, Traefik and cloudflared run in the guest; the cloudflared connector token is already in controller.yaml; DNS is at Cloudflare, not the hub. The backup tiers keep running — Tier-1/2/3 are scheduler jobs inside the controller (cmd/controller/main.go:601-690) and the whole-guest tiers are the agent's. Restores A.1.1–A.1.7 all work. Events queue rather than drop: settings.PendingEvents

• DrainPendingEvents (internal/settings/settings.go:94,1466-1485).

For a week. Three things degrade, all of them knowledge rather than copies:

1. The operator alarm plane goes dark. Every notification — customer Hungarian and operator English — is dispatched by the hub via Resend. A failing tier still records last_status: error in the guest's own settings.json and still shows in the customer UI, but nobody is told. This is R-100's exact scope note: "the operator's fleet-wide alarm plane is silent, which is the plane that matters for an unattended appliance" (CAMPAIGN-8…:481-483).
2. No claim codes, no password resets, no self-bind. POST /claim/request-new-code is a hub round-trip; a customer locked out of their dashboard stays locked out.
3. No config or version convergence. min_controller_version, the artifact digests and the desired-state channel all stop; a box stays on whatever it runs.
4. host_pbs_secrets re-issue is impossible, so the R-39 self-heal chain — proven to take 13 s with the hub present — has no mint to consume.

Permanently. Two irreplaceable losses and one recoverable one:

• The escrow custody is gone (host_escrow, 2 rows). The blobs are the only off-box copy of the PBS client key, the WG private key and the restic repo password. Losing them does not lose the ciphertext, but it removes the only route to it that survives a host loss (C-2).
• The break-glass credentials are gone (host_recovery, 3 rows) — C-3.
• The Cloudflare tokens, the offsite descriptor, and the customer identity are recoverable, because a live box already holds a rendered copy in controller.yaml (LIVE, both boxes).

The one LIVE observation that bounds all of this: hosts.last_report_at is 2026-07-28 17:25:20 / 17:25:31 for the two live hosts — a ~15-minute-old mirror. The hub's copy of box reality is never more than one cycle stale, which is precisely why it is convenience, not the DR source of record (05-hub-architecture.md:170-171).

---

Part E — failure domains (facts; no ruling on acceptability)

E.1 — Do restic and PBS offsite share a provider, account, login or payment method?

	Tier-3 restic	Plane-2 PBS offsite
provider	Hetzner — Storage Box u629488, sub-accounts -sub1 (demo-felhom) and -sub3 (demo-hp), host u629488-subN.your-storagebox.de:23 (LIVE from each guest's settings.json)	Hetzner — Cloud server ep0 at 167.233.158.164 (LIVE ssh root@… → felhom-hetzner), datastore on Cloud Volume scsi-0HC_Volume_106469259 mounted at /mnt/pbs-datastore
provisioning credential	hub HETZNER_TOKEN + HETZNER_POOL_BOX_ID (Secret/storagebox), API base https://api.hetzner.com/v1 (hub/internal/hetznerapi/hetznerapi.go:23)	provisioned out of band; the hub reaches ep0 over SSH with a pinned host key, not the Hetzner API
shared?	PROVIDER: YES, both Hetzner. ACCOUNT / LOGIN / PAYMENT: UNKNOWN — see Unknowns. I deliberately did not call the Hetzner API with the production token to find out.

Additional shared-fate facts inside each tier:

• All customers share one Storage Box. u629488-sub1, -sub2 (peti, untouched) and -sub3 are sub-accounts of one box. A sub-account is an access-control object, not a data object — the 2026-07-18 RESET proved this live by deleting a sub-account and finding its /home intact (00-capability-map.md:42, → R-32).
• All customers share one PBS datastore. felhom-offsite on ep0, isolated by namespace + a per-customer client-side key + a per-customer DatastoreBackup token (LIVE storage.cfg on both hosts, distinct encryption-key fingerprints, username felhom@pbs!demo-felhom / !demo-hp). Consequence recorded in 07-backup-architecture.md:388-391: per-tenant encryption precludes cross-customer dedup, so cost scales linearly per customer.
• Both offsite tiers ride the same WireGuard endpoint host in one respect: ep0 is both the WG server and the PBS server (06-…:34, D3). restic does not ride WG — it goes guest → SFTP directly to the Storage Box (Campaign 8 fault 4's failure to inject is the proof: "restic runs inside the guest, so its traffic never enters the host's output hook", CAMPAIGN-8…:567-573). So the two offsite tiers share a provider but not a transport.

E.2 — What depends on DooPlex, and what happens if DooPlex is lost

DooPlex (192.168.0.180, public IP 37.191.56.193, LIVE) is a single physical machine.

What runs there (LIVE): k3s (single node — every pod on node dooplex), the hub, Longhorn (replicas on /dev/sdb1), the Longhorn backup NFS export (/mnt/5_hdd/backup), the container registry gitea.dooplex.hu, a PBS instance (systemctl is-active proxmox-backup → active; doc D6 says DooPlex PBS is dev/demo only), the felhom.eu website, contact-mailer, umami, filebrowser, and this Claude Code session.

If DooPlex is lost:

lost	consequence
the hub	everything in D2.4 "permanently"
the only hub-DB backup	it is on the same machine (D2.2) — both copies gone together
gitea.dooplex.hu	no controller or agent image can be pulled, so no deploy, no self-update, and a guest that loses its image layers cannot be brought back on the pinned tag
hub.felhom.eu and felhom.eu	they resolve directly to 37.191.56.193 — see E.3
operator alerting (Healthchecks → monitoring@)	gone

What does NOT depend on DooPlex: every customer box's data plane; both offsite tiers (Storage Box and ep0 are elsewhere); the customers' public app URLs (Cloudflare edge → cloudflared in the guest).

E.3 — What depends on Cloudflare, and what breaks without it

DNS (LIVE, dig @1.1.1.1):

felhom.eu           NS  martha.ns.cloudflare.com. / tony.ns.cloudflare.com.
demo-felhom.eu      NS  martha / tony
enkisfelhom.hu      NS  martha / tony
sajatfelhom.hu      NS  martha / tony

One Cloudflare account authoritative for the operator's own domain and every customer domain.

Records (LIVE):

hub.felhom.eu   CNAME dooplex.hopto.org.  ->  A 37.191.56.193     (NOT proxied - the origin IP is public)
felhom.eu       A     37.191.56.193                                (NOT proxied)
ep0.felhom.eu   A     167.233.158.164                              (NOT proxied)
felhom.demo-felhom.eu   A 104.21.11.129 / 172.67.149.59            (PROXIED - Cloudflare edge)
felhom.enkisfelhom.hu   A 104.21.3.175  / 172.67.130.252           (PROXIED)
felhom.sajatfelhom.hu   A 172.67.162.57 / 104.21.90.232            (PROXIED)

What breaks without Cloudflare:

Cloudflare component	what stops
DNS	everything by name — box→hub reporting, customer app access, ep0 endpoint resolution by the WG watchdog. The WG tunnel itself survives, because the agent resolves the A record and writes the v4 literal into the conf (06-…:174-184); PBS then dials 10.77.0.1, an in-tunnel address that needs no DNS.
Tunnel / edge (proxied records)	every customer app URL. The lan_resolver LAN path is the documented fallback and is IMPLEMENTED, never drilled as a customer experience (00-capability-map.md:93 → R-19).
WAF / geo	geo-restriction enforcement (hub-held CF API token)

A third-party dependency in the same path, easy to miss: hub.felhom.eu is a CNAME to dooplex.hopto.org — a no-ip DynDNS name. Box→hub reporting therefore depends on Cloudflare DNS and no-ip and the operator's home IP not changing faster than the DynDNS updater.

E.4 — What depends on the k3s cluster

k3s is single-node (LIVE: every pod on dooplex), so "the k3s cluster" and "DooPlex" are the same failure domain. Inside it: the hub Deployment, its Longhorn PVC, the nginx-internal ingress (hub.felhom.eu is ingressClassName: nginx-internal, manifests/hub.yaml:323,329), cert-manager for the hub certificate, ArgoCD (auto-sync off), and the Longhorn backup target. There is no second node and no failover.

E.5 — Does any single credential or account failure take out more than one tier?

credential / account	tiers it can take out
the Cloudflare account	box→hub reporting and every customer's app access and the operator website — three planes, one login
the Hetzner account (if one account covers both products — UNKNOWN)	Tier-3 restic and Plane-2 PBS offsite — i.e. both offsite tiers at once, leaving only the local vzdump tier, which lives on the same physical disk as the guest it backs up (E.6)
the hub global bearer (REPORT_API_KEY)	operator-tier hub API incl. break-glass retrieval. Its git-history copy is still live — secrets.md:76-77: "The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing." Rotation runbook exists; no rotation date is recorded.
the Resend key	every notification and every claim/reset code — the alarm plane and the account-recovery plane at once
the customer's recovery code R	unwraps the PBS key, the WG key, the PBS token and the restic repo password — one loss, both offsite tiers unreadable after a host loss
the restic repo credential	can delete. R-95, still open and ranked #1: "restic offsite credential can delete (readonly=False, forget --prune runs from the box); SFTP cannot express append-only" (OPEN-ITEMS.md:13). Contrast R-89, proven the other way for PBS: "the box was correctly refused when it tried to delete its own offsite snapshot … A compromised box cannot destroy its own backups" (CAMPAIGN-8…:514-515). The two offsite tiers have opposite blast radii from a box compromise.

E.6 — One more shared fate, found by inspection rather than by document

The local whole-guest tier shares its physical disk with the guest it backs up. LIVE lsblk + pvs on both hosts:

demo-felhom:  sda 476.9G -> sda3 -> VG pve -> { pve-root (/, holds /var/lib/vz/dump),
                                                pve-vm-9201-disk-0/1/2 (the guest) }
              sdb 931.5G -> /mnt/hdd_1                      (the customer data drive, separate)
demo-hp:      sda 119.2G -> sda3 -> VG pve -> { pve-root (/, holds /var/lib/vz/dump),
                                                pve-vm-9201-disk-0/1/2, vm-300-disk-* }
              nvme0n1 953.9G -> /mnt/nvme-1tb               (the customer data drive, separate)

local (the vzdump target, /var/lib/vz) and local-lvm (the guest's rootfs and both backup=1 mountpoints) are the same physical device on both hosts. A single-disk failure removes the guest and its entire local backup history together, leaving only the weekly PBS tier. Tier-2's sys_drive destination is on that same disk as well (mp1 = pve-vm-9201-disk-2), so on demo-hp paperless-ngx's Tier-2 copy is cross-drive from the source (nvme-1tb → sda3) but co-located with the local whole-guest archives.

---

Part F — measured timings

Only measured numbers. Nothing here is estimated, and everything unmeasured is listed as such.

F.1 — Backup timings (LIVE, this session)

operation	measurement	source
local vzdump, demo-felhom	00:04:10 (15,062,241,280 B written → 5.42 GB archive, 59 MiB/s)	/var/lib/vz/dump/vzdump-lxc-9201-2026_07_28-01_27_56.log
local vzdump, demo-felhom	00:05:09 (15 GiB → 5.44 GB, 47 MiB/s)	…-06_44_22.log
local vzdump, demo-felhom	00:05:09 (15 GiB → 5.52 GB, 48 MiB/s)	…-17_53_26.log
local vzdump, demo-hp	00:01:05 (4,544,911,360 B → 1.50 GB, 71 MiB/s)	…-09_42_36.log
local vzdump, demo-hp	00:01:05 (4.3 GiB → 1.52 GB, 72 MiB/s)	…-10_40_54.log
local vzdump, demo-hp	00:01:06 (4.4 GiB → 1.56 GB, 71 MiB/s)	…-17_43_05.log
restic offsite run, demo-felhom	2m27s (last successful, 2026-07-28T11:29:22Z; repo 1.0 GB / 32 snapshots)	guest settings.json offbox.last_duration
restic offsite run, demo-hp	1m35s (2026-07-28T16:20:57Z; repo 36.3 MB / 8 snapshots)	guest settings.json; corroborated in CAMPAIGN-9…:743-748
Tier-2 run, demo-felhom bookstack	1s for 155.7 MB	settings.json app_backup.bookstack.cross_drive
Tier-2 run, demo-felhom immich	(duration not recorded in the row; size 1.5 GB)	as above
Tier-2 run, demo-hp paperless-ngx	0s for 85.5 MB	settings.json
PBS backup, demo-felhom	5.7 GB / 00:03:12	CAMPAIGN-8…:75-76
PBS backup, demo-hp	1.5 GB / 00:00:55	CAMPAIGN-8…:75-76

F.2 — Restore timings

operation	measurement	source
whole-guest restore-test, LOCAL tier, demo-hp	84.18 s / 84.21 s / 84.26 s / 89.20 s / 89.34 s across five consecutive runs on the same 1.56 GB archive	LIVE journalctl -u felhom-agent, 2026-07-28 18:00–18:08 CEST
whole-guest restore-test, LOCAL tier, demo-felhom	93.84 s · 99.10 s · 111.82 s · 98.81 s on 5.4–5.5 GB archives	LIVE journalctl, 2026-07-27 20:47 → 2026-07-28 00:32 CEST
whole-guest restore-test, PBS OFFSITE tier, demo-felhom	1101.05 s ≈ 18 m 21 s — archive=felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z, passed 2026-07-28 00:04:27 CEST	LIVE journalctl
whole-guest restore-test, PBS, demo-hp (manual --selftest)	4 m 5 s restore + boot + verify + teardown, mount_parity: ok	00-capability-map.md:40
local whole-guest restore, Campaign 9 incidental	89.19 s — restored into scratch 990000, booted, torn down	CAMPAIGN-9…:363-373
local vzdump → fresh CT 9299	5.82 GB archive, 15 GiB extracted @ 210 MiB/s, exact mount parity	CAMPAIGN-8…:247
PBS offsite → fresh CT 9297	4.36 GB encrypted snapshot, exact mount parity (no wall-clock recorded)	CAMPAIGN-8…:248
restic app-data restore (R-87)	14 s for 155.5 MB, bookstack snapshot b2d8936d, 6/7 files byte-identical	CAMPAIGN-8…:246
Tier-2 file restore, gap-fill (A1)	39 s — 6 files restored, incl. app stop→healthy	CAMPAIGN-9…:105-109
Tier-2 file restore, after total loss (A3)	46 s — 43 files restored, 16/16 documents usable afterwards	CAMPAIGN-9…:184
Tier-1 recovery-unit restore (A2)	18.25 s; app healthy 29 s later	CAMPAIGN-9…:141-151
app restart-to-healthy after Tier-2 restore	Up 5 seconds (health: starting) → Up 27 seconds (healthy), polled at 10 s	CAMPAIGN-9…:112-114
guest-power watchdog recovery	120 s unattended, against the 587 s the original incident needed with a human	OPEN-ITEMS.md:38; the incident's own figure is 9m47s = 587 s (CAMPAIGN-8…:554-556)
quiesce crash recovery (unquiesce after SIGKILL)	1 second	CAMPAIGN-8…:216,229-233
app downtime, one quiesce, two tiers	86 s (Campaign 8 A2) and 1m27s (the R-82 UI-triggered proof) and ~97 s (Campaign 9 B3)	CAMPAIGN-8…:129; 00-capability-map.md:40; CAMPAIGN-9…:337
PBS-DR credential self-heal, operator click → converged	13 s	00-capability-map.md:39
controller floor-lift, operator save → healthy on the new version	16 s	00-capability-map.md:124
customer bind → box running current	2 min 44 s	00-capability-map.md:124

F.3 — Unmeasured (listed, not estimated)

• Restore of a whole guest from PBS to a different host — never run.
• Restore of a whole guest over the WAN at production scale — the only offsite restore-test timing (1101 s) is a 4.4 GB guest to the same host over the existing tunnel. No large-guest number exists.
• A restic restore of anything larger than 155.5 MB.
• Any restore of a customer data drive. No tier holds one whole (Part B.2), so no such timing can exist.
• Weekly PBS incremental size and duration — 07-backup-architecture.md:384 still records this as UNMEASURED; the datastore now holds 3 and 6 snapshots respectively, so the input exists but the measurement has not been made.
• Host rebuild from nothing — never performed (D1).
• Escrow consume → tier re-established — never performed destructively.
• Hub DB restore from a Longhorn backup — never performed.
• .fab import wall-clock — the 1.7 GB round trip is cited without a duration.
• Time to first byte for a customer restore over a home uplink — no customer has ever driven a restore (00-capability-map.md:75).

---

Part G — what is actually proven

Status vocabulary is the capability map's own strict enum (00-capability-map.md:11-18): a PROVEN-LIVE claim must cite a campaign/drill/validation doc.

G.1 — Live, cited proof

path	proof
Tier-2 file restore, gap-fill	Campaign 9 A1 — byte-identical returns, both non-destruction promises kept, restored bytes served by paperless's own API
Tier-2 file restore, after total directory loss	Campaign 9 A3 — 404 before, 200 with matching hashes after, 16/16 documents
Tier-2 restore refuses without an outage for a no-coverage app	v0.183.0 live replay on demo-felhom — BookStack uptime unbroken (felhom.eu/REPORT.md:60)
Tier-1 recovery-unit restore executes end to end	Campaign 9 A2
restic offsite restore of app data (bytes)	Campaign 8 R-87
offsite reconstitution of a DB-indexed app through the customer's own buttons	2026-07-20 destructive immich drill (trash emptied), 00-capability-map.md:66,75
shares restore (files + definitions + credential)	2026-07-18, 00-capability-map.md:96
.fab export/import full circle	CAMPAIGN-6D P-FAB, 1.7 GB byte-identical
whole-guest pct restore, local and PBS, with exact mount parity	CAMPAIGN-2 T-P9 + Campaign 8 Phase C
a corrupted PBS snapshot fails cleanly	Campaign 8 fault 17 + Phase C — quarantined .0.bad, verify_state: failed, restore exit 255 naming the entry and the chunk, all three LVs rolled back, no debris
a restore-test whose source vanishes fails loudly and does not advance the proven-restorable timestamp	Campaign 8 fault 18
the box cannot delete its own PBS snapshots	Campaign 8, R-89
unattended restore-test on both tiers	LIVE this session — local passes on both boxes; felhom-pbs passed on demo-felhom at 00:04:27 CEST
guest-power watchdog	120 s unattended recovery, 2026-07-28
F-CRIT-1 / F-CRIT-2 / F-A1 / F-LEAK / F-OBS / F-REBOOT fixes	each replayed live, OPEN-ITEMS.md:25-40
C9-F2 crash-loop alarm	live replay: silent through ten 30 s samples, then app_start_failed at 5m25s; heartbeat flipped 0 currently down → 1 currently down (felhom.eu/REPORT.md:57-58)

G.2 — Unit-tested / implemented only

• Restore-proof is UNATTENDED across every tier (R-85) — IMPLEMENTED, explicitly not PROVEN-LIVE: "rotation has not been observed selecting both tiers across consecutive UNATTENDED cadences" (00-capability-map.md:41). My LIVE reads are consistent with rotation (both tiers have a proof timestamp on both boxes) but a single pass is not the multi-day observation the row asks for.
• PBS-DR secret self-heal — reconciler still scoped to one host via PBSDRHEAL_ONLY_HOST (00-capability-map.md:69).
• Customer DELETE cascade — unit-proven; the end-to-end live leg is not run (00-capability-map.md:43).
• F-DIAG's six offsite failure classes — unit-proven; "not yet exercised by a live offsite failure of each class" (OPEN-ITEMS.md:36). Campaign 9 D10 did fire one of them (transport) live and confirmed the sanitiser.
• R-100's 48 h offsite_stale threshold — the producer half is confirmed twice live; the alarm itself has never fired (CAMPAIGN-9…:700-714).
• Paired recovery notifications — IMPLEMENTED, recovery leg PARTIAL (00-capability-map.md:113).

G.3 — Doc claim only, or nothing

claim	status
agent DR bring-up ModeDRGuestLoss	code exists, never executed
host-loss DR plan → an actual host restore	plan executes nothing by construction; the destructive drill is unrun
escrow consume in a real recovery	spike-level only
Tier-1 content recovery after real loss	the single most valuable unproven item (CAMPAIGN-9…:825-827, restated felhom.eu/REPORT.md:96-97)
host reboot mid-backup	NOT REACHED in Campaign 8 (fault 12) and NOT STARTED in Campaign 9 (C6)
three-way concurrency with GC	never
a never-succeeded Tier-2 (Scenario C) live	never (felhom-controller/REPORT.md:81-82)
F-HUB SQLITE_BUSY characterisation	never
a customer (not the operator) performing a restore via UI alone	MISSING as evidence (00-capability-map.md:75) — no product gap blocks it since v0.153.0; the evidence run has simply never been done by a customer
hub DB restore from its Longhorn backup	never
FileBrowser browse/download	"exercised in no doc" (00-capability-map.md:98)

G.4 — Where a doc claims something the evidence does not support

Each row cites both sides.

#	doc claim	contradicting evidence
G4-1	07-backup-architecture.md:85-91: "Both tiers are now restore-tested unattended … and each tier's last successful proof is reported."	00-capability-map.md:41 says the same capability is IMPLEMENTED, not PROVEN-LIVE, because "rotation has not been observed selecting both tiers across consecutive UNATTENDED cadences." The architecture doc states as settled what the capability map holds open.
G4-2	07-backup-architecture.md:29,79: the Tier-1 row describes RestoreApp as the restore path	restore_unit.go:95 RestoreFromRecoveryUnit is the real path; RestoreApp is only the fallback when no unit exists (:120-127). The doc's own staleness banner (:11) covers this — it warns §0–§8 are "historical intent, not current fact" — so this is a disclosed, not a hidden, drift.
G4-3	settings.go:148-150 (OffboxTarget doc comment): "off-box of the secrets rides DR via the PBS whole-CT snapshot of the rootfs"	LIVE: the secrets are in the Docker named volume felhom-controller-data, i.e. under /var/lib/docker = mp0, not the rootfs. The protection claim holds (mp0 is backup=1) but names the wrong mountpoint — a comment that would become false if mp0's backup flag ever changed, with no test pinning it.
G4-4	06-offsite-connectivity.md:19-21: "The operator's public edge is Cloudflare-Tunnel → nginx-internal … DooPlex has no public IP and no public UDP."	LIVE: hub.felhom.eu CNAMEs to dooplex.hopto.org → A 37.191.56.193, which is DooPlex's own public IP (curl api.ipify.org from DooPlex returns the same address). The record is not Cloudflare-proxied. The doc is from 2026-07-03 and the topology has since changed, or was never as described for the hub.
G4-5	dr_recipe.go:55: Namespace is "PBS namespace the restore targets"	LIVE: "root" on every box, while the real namespaces are demo-felhom/demo-hp (verified on ep0 and in a live pct restore command line). Root cause traced to client.go:97 / report.go:23-27 (D2.3-c).
G4-6	05-hub-architecture.md:175-176,186: the "slim DR record" on hosts holds PBS namespace + repo fingerprint + wrapped escrow key, and host-loss recovery reads from it	LIVE: hosts.dr_record_json is {} for all three hosts (D2.3-a).
G4-7	06-offsite-connectivity.md:148-150: the escrow upload included "directive with the non-secret DR coords"	LIVE: host_escrow.directive_json is '{}' — 2 bytes — for both escrowed hosts (D2.3-b).
G4-8	dr/plan.go:34-35 describes PlannedDrive as the re-attach-by-durable_id wrong-disk guard, and 06-…:327 reports the matcher unit-proven	LIVE: dr_recipe.host_half.drives is [] on every customer, including two boxes with enrolled data drives (D2.3-d). The guard is correct; it currently has nothing to guard.
G4-9	felhom.eu/REPORT.md:17-24 / OPEN-ITEMS.md:31: A=9 / B=43 / C=1 over the 53 templates	My independent enumeration of the same catalog at 4252121, applying classify.go's documented default rules, gives A=7 / B=45 / C=1 (Part B.1). Two apps unaccounted for.
G4-10	felhom-controller/controller/README.md (corrected 2026-07-28) claimed faults surface as "exited/degraded/restarting/unhealthy"	StateRestarting was in no down-set at all — the seventh shipped-invariant comment; already found and corrected in felhom.eu/REPORT.md:84-87. Recorded here because it is the same class as G4-3 and G4-5.
G4-11	CAMPAIGN-8…:96 pre-registered the window-gate valve behaviour as an artifact of compression	Not a contradiction — recorded as the good case: the campaign declared its artifacts before running, and Campaign 9 repeated the practice (CAMPAIGN-9…:302-313). Included so the list is not read as one-sided.

---

Unknowns

Everything I could not establish, and what it would take.

#	unknown	what would settle it
U-1	Do the Storage Box and ep0 sit under one Hetzner account / login / payment method?	An operator statement, or a read-only GET /v1/servers and GET /v1/storage_boxes with the hub's HETZNER_TOKEN. I deliberately did not make that call — it uses a production credential against an external service for an inventory question the operator can answer in one line.
U-2	Which 2 apps account for the A=9 vs A=7 difference?	Diff my enumeration script against the C9-F1 Phase-0 method. The Phase-0 method is described in prose (felhom.eu/REPORT.md:15-24) but the enumeration itself is not committed.
U-3	Is IdentityBundle.TunnelToken populated on the current escrow blobs?	Unopenable without R, by design. 06-…:151 records that the 2026-07-04 blob had it "intentionally empty"; the current blobs are from 2026-07-21/07-23 and are 572 B. Only a consume ceremony would tell.
U-4	Whether hosts.dr_record_json = {} is a doc drift or an unwritten field.	Read the hub's writer for that column; I did not trace it.
U-5	Why dr_recipe.drives is empty — whether the storage targets fail isUserDataDrive's type/DurableID/MountPath test, and which of the three.	Read the live host-report JSON's storage_targets[] for one box.
U-6	Whether the offsite restic repos contain what their snapshot counts imply.	restic snapshots against each repo. Not run — it takes a repo lock, and C9-F3 makes lock handling a live hazard. Recorded counts used instead (32 / 8).
U-7	The peti-felhom recovery posture. LIVE: dr_recipe has a peti-felhom row referencing host peti-felhom-86d37d, but there is no such row in hosts and no host_escrow row.	Out of scope by instruction — peti was not touched. Flagged because it means one of four customers has no escrow custody at all.
U-8	Whether the Longhorn hub-DB backups are restorable. Two Completed backups exist; neither has been restored.	A restore drill into a scratch PVC. Not run (read-only task).
U-9	Whether ep0 itself is backed up. LIVE: the datastore is on a Cloud Volume and /srv/pbs-felhom holds a 13 G rollback copy on the root disk. 06-…:313 lists endpoint DR as a deferred slice-1 item.	Whether Hetzner Cloud snapshots/backups are enabled on ep0 — a console or API question.
U-10	Whether the .fab browser-upload leg works.	A human click-through; 00-capability-map.md:67 still records it as Viktor's open test.
U-11	What app_backup.<app>.enabled (Tier-1 toggle) actually gates today. LIVE: it is false for every app on both boxes, while recovery units are demonstrably being captured and restored.	Trace the Enabled field's readers; my grep found only offbox.go:451,1167 and tier2.go:577-611, none of which is a Tier-1 gate. Possibly vestigial (its comment still says "includes app data in nightly restic (same drive)", a mechanism that moved to the agent).
U-12	When the hub global bearer was last rotated — the git-history copy is live until it is.	A password-manager entry date or a hub-side audit.

---

Contradictions

Doc / comment / README versus source or live state. G4-1 … G4-11 above are the full list; the four that are new in this inventory (not previously filed anywhere I could find) are restated here so they are not buried:

1. G4-4 — the hub's public edge is not what doc 06 describes. hub.felhom.eu resolves to DooPlex's own public IP through a no-ip DynDNS CNAME, unproxied. Doc 06 §1 says the edge is a Cloudflare Tunnel and that DooPlex has no public IP.
2. G4-5 — the DR recipe records the PBS namespace as root on every box. Traced to source; the real namespaces are per-customer.
3. G4-6 / G4-7 / G4-8 — three DR inputs are empty on the live fleet: hosts.dr_record_json, host_escrow.directive_json, and dr_recipe.host_half.drives. Each is named by a design doc as something host-loss recovery reads.
4. G4-9 — the C9-F1 class counts do not reproduce. 9/43/1 published, 7/45/1 by my enumeration.

And one contradiction of omission rather than of statement:

5. The offsite tier captures Docker named-volume tars on every run and no offsite action unpacks them. 07-backup-architecture.md:81 describes Tier-3's restore path as "staged scratch on a data drive → missing-only merge to live" and §7.3 frames that as sufficient for the SQ3 acceptance. offbox_reconstitute.go:284-289 skips the unit outright, and offbox_restore.go:352-356 places it only when the live unit is absent. For the 52 of 53 templates with named volumes, the volume data is in the snapshot and no single offsite action returns it. Not currently filed.

---

Observations

Noticed during the inventory and deliberately not acted on.

O-A — A plaintext restic repo password is sitting unwiped on demo-hp's host filesystem. LIVE: /var/lib/felhom-agent/escrow-stage/restic_repo_password, 64 B, 0600 felhom-agent, mtime 2026-07-28 13:26 CEST. felhom-agent/internal/escrow/identity.go:41-44 documents this path as a transient stage that the ceremony wipes after a successful escrow-create (WipeStagedResticPassword, :47-53). demo-felhom's escrow-stage/ is empty, as expected. So on demo-hp the data key for the entire Tier-3 tier is currently a plaintext file on the host, outside the escrow, five days after that host's escrow was created (2026-07-23). Not touched.

O-B — Twelve stale backup copies of agent.json on the hosts. LIVE: six per host (.bak-pre-r82, .bak-pre-r85, .campaign8-before, .campaign9-before, .campaign9-prev, .pre-prunegate.bak), each 2.3–2.5 KB and each containing the box's hub.api_key. Campaign 8 already noted that reading agent.json printed hub.api_key in cleartext because the redactor matched *token*/*secret*/*password* but not the bare api_key (CAMPAIGN-8…:59-63). Four similar copies of controller.yaml and three of settings.json exist inside each guest. Left alone.

O-C — The R-99 phantom snapshot is still on ep0 and still counted as an entry. LIVE: /mnt/pbs-datastore/ns/demo-hp/ct/9201/2026-07-28T05:31:14Z/ holds only catalog.pcat1.tmp_didx (0 B), root.pxar.tmp_didx (0 B) and pct.conf.blob (425 B) — no index.json.blob. It is exactly the artefact Campaign 9 saw the F-CRIT-2 size filter reject live (CAMPAIGN-9…:341-361). R-99 is filed and explicitly not automated (OPEN-ITEMS.md:26). Not deleted.

O-D — The setup wizard's restore path is dead code with live templates. internal/setup/templates/setup_hub_restore.html and setup_restore_exec.html exist and are parsed by ParseFS(templateFS, "templates/*.html") (handlers.go:78-81), but no route serves them — Handler() registers only /, /setup, /setup/fresh, /setup/manual, /setup/failed and two static paths (:85-99), with the comment "Disk-recovery setup paths (drive scan, infra-backup restore) have moved to the host agent (slice 8C)". An orphaned doc comment for autoProcessHubRestore now sits directly above autoProcessFreshHub (:215-219), describing a function that no longer exists. This is the "seam built but never wired" shape in reverse — a seam unwired but never removed. Not deleted.

O-E — tier2DestRel maps RootImport into the hdd/ subtree. tier2_capture.go:26-33: base := "hdd" unless the root is RootUserdata. So a ${IMPORT_PATH} bind would be captured to hdd/<rel> and restored to liveNsRoot/<rel>, while its live location is importRoot/<rel> — the system namespace's userdata/import, deliberately a different root (classify.go:37-41). Currently unreachable: both catalog apps that bind ${IMPORT_PATH} (calibre-web, paperless-ngx) class it excluded, so it is never captured. A latent mismatch that becomes real the day an import path is classed non-excluded. Not changed.

O-F — Nineteen Tier-1 recovery units on demo-felhom are 24 KB config-only shells for apps that are not deployed. LIVE: adventurelog, bentopdf, calcom, crafty-controller, ghost, glance, gokapi, gramps-web, home-assistant, homebox, kimai, n8n, outline, papra, rallly, recipe-importer, tandoor, wger, wishlist, zipline — all under backups/primary/ on sys_drive, none in docker ps. Harmless; noted because ListRestorePoints will offer a restore point for each, and the count of "apps with a restore point" is therefore not the count of "apps that exist".

O-G — The in-memory R-88 breaker still carries Campaign 9's failure counts. Declared in the campaign's own report as a benign residue that clears on the next successful whole-guest backup or any controller restart (CAMPAIGN-9…:762-768). Both controllers have since restarted (v0.183.0 was deployed after the campaign), so this is very likely already cleared — I did not verify it, because verifying would mean reading a state the controller does not expose read-only.

O-H — DooPlex's root filesystem is at 81 % (/dev/sdb1, 341 G of 445 G used). Below the 90 % abort threshold in the workspace CLAUDE.md, but it is the disk holding both the Longhorn replicas and the container images. Noted, not acted on.

O-I — drill-r50 is both a blocked customer and a live WireGuard peer. LIVE: customer_configs status blocked, hosts.last_report_at 2026-07-27 05:44:53, agent 0.96.0 (four minor versions behind the fleet), a host_recovery row, and peer 10.77.0.4 on ep0 with no handshake. Already filed as R-93 (OPEN-ITEMS.md:43). Recorded here because it means the fleet's peer list and host table both contain an entry whose recovery posture nobody maintains.

O-J — The window between "backup exists" and "backup is provable" is structurally different per tier, and only one tier's is short. LIVE: local vzdump retention is 3 archives at a 24 h cadence (≈3 days of history); PBS offsite is keep-last 2 at a 7-day cadence (≈2 weeks); restic is --keep-daily 7 --keep-weekly 4 --keep-monthly 6. The restore-test cadence is 84 h on both boxes. So the local tier is re-proven roughly once per its own retention window, while the offsite PBS tier holds two snapshots and is proven at a cadence unrelated to when they were written — which is exactly what R-86 is filed to fix ("Restore-tests are interval-scheduled, not backup-aligned", OPEN-ITEMS.md:15). Stated as a measured relationship, not a recommendation.