Live: hub 0.81.0, agent 0.113.0, controller 0.185.1 on both demo boxes;
host-install 1.22.0 (script; no reinstall performed).
E-2a wrapper proven live as root on demo-hp: F-1 subdirectory refused, F-2
unmounted path refused, root device refused, idempotent re-apply is a no-op,
repointing refused -- 0 stray storages. The agent PVE role was NOT widened.
Scenario E proven live on BOTH boxes: healthy renders nothing, no message key.
Records three defects I introduced and caught: unreachable routes (mounted
outside /api/storage/, caught by the first live call), a hollow test exposed by
its own red-proof, and another gofmt-realignment no-op.
Not live-proven: the degraded banner and offer acceptance (both boxes healthy),
backup_target_absent end-to-end, Case A/B on a real install, drive-loss recovery.
hub 0.81.0, agent 0.112.0, controller 0.184.1 live on BOTH demo boxes.
E-2c: eject/decommission of the backup-target drive refused 409 on both boxes,
drives unmoved. E-2b: the never-called disconnect seam is wired, with the target
case raising the specific backup_target_absent.
Records the keying bug caught before deploy (a.Path is the GUEST path, so the
target branch was unreachable -- 0.184.0 superseded, never deployed) and states
plainly that backup_target_absent is NOT proven end-to-end live: proving it needs
a live enrolled drive to go absent.
Parts 2/3/4 and E-2a remain open; Peti risk stays parked.
E-2 is partially shipped (hub v0.81.0 + controller Part 1). Filing the rest so a
foundation with no UI cannot quietly become a sixth seam-built-but-never-wired.
E-2 remaining: installer Case A/B, the offer + agent-side move, the degraded
banner, the controller half of the signal, red-proofs E/F, live validation.
Phase 0 INVERTED the emphasis: the installer has no drive-enrollment step,
so the common case at install is system-drive-only and Part 3 (drive added
later) is the PRIMARY path, not Case A.
E-2a the move needs a root-fenced wrapper -- the agent holds neither
Datastore.Allocate at /storage nor Permissions.Modify, and its sudoers has
no pvesm and no pveum. Use the guarded-wrapper pattern; do NOT widen the
agent's PVE role.
E-2b NotifyStorageDisconnected/Reconnected are defined and called NOWHERE, so a
drive going absent emits no event at all. Hub side is already plumbed, so
wiring needs no hub change.
E-2c E-1 put the whole-guest backups on a drive POST /disks/eject will eject
(RoleForStorage returns user-data for a local-dir on a non-system device).
Guard the eject specifically -- reclassifying the drive RoleBackup would
block legitimate ejects, since it is also the enrolled user-data drive.
PETI peti-felhom deliberately NOT migrated; drive failure there is offsite-only
recovery. Accepted until the operator's reinstall; re-evaluate if that
slips past ~2026-09-01.
Supervised operational run. No code, no version bump, nothing deleted.
Primary backup tier on both demo boxes moved from `local` (a dir storage on
/var/lib/vz -- the SAME physical device as the guest) to `felhom-backup`, a dir
storage on each box's secondary drive:
demo-hp /mnt/nvme-1tb uuid:91d2dc2d-... archive 2,256,044,492 B
demo-felhom /mnt/hdd_1 uuid:47a3361a-... archive 5,957,878,962 B
Both proven end to end via the real UI path: archive lands on the secondary
drive (df delta matches the archive byte-for-byte), restore-test auto-selects it
and passes with mount_parity: ok, and freshness survives an agent restart with
an empty in-memory store -- so the age can only have come from the new storage.
Phase 0: the target is CONFIGURATION, not converged (the sole writer of
agent.json touches only escrow.pbs_storage_id and preserves unknown keys), so
the runbook's STOP did not fire. No consumer hardcodes "local" on the backup path.
Findings:
- F-1 the storage path must BE the mountpoint; a subdirectory fails exactMount
and the target reports disconnected permanently (observe.go:321)
- F-2 --is_mountpoint 1 is load-bearing; proven live, an unguarded storage on a
non-mounted path reports active with the ROOT filesystem's free space and
had already created dump/ on pve-root -- a silent retarget onto the very
device this change escapes
- F-3 FelhomAgentStore is granted per storage path; without it every backup
403s. felhom-host-install.sh must issue it for new installs
- R-109 (new) the DR recipe records no backup target, and each box now carries
two content=backup dir storages, one live and one frozen
- R-105 narrowed and TRACED: dr_recipe drives was [] fleet-wide because the
enrolled drives were never PVE storages, so isUserDataDrive never saw
them. Both boxes now populate drives; SMART on the backup drives too
Absent-drive behaviour today is fail-loudly with no silent retarget (PVE half
live-proven; agent half source-traced). That is NOT the intended fall-back-and-
alarm design -- filed as E-2 with the honest single-drive label.
Reported in full in the record: the agent was restarted with a felhom-pbs backup
in flight, producing a spurious tier failure. The backup had in fact succeeded
(PVE task OK, 6,264,034,053 B snapshot) and the spurious failure reached no
channel -- R-84 ground truth superseded it.
Outstanding: full drive-loss recovery (needs physical access) and the agent half
of the absent-drive behaviour.
Replaces the 2026-07-14 DRAFT entirely - it was verified against controller v0.132.0,
51 versions stale, while being cited as authoritative. Now verified against controller
0.183.0 / agent 0.110.0 / hub 0.80.0 / catalog 4252121. NOT ratified.
Every statement is labelled [DESIGN] (a decision from the architecture discussion) or
[FACT] (file:line, live output, or a citation to _recovery-inventory-2026-07-28.md).
Phase 0 verified D5's premise and it FAILED. Ten read surfaces were checked; nine are
clean (SMB under both namespace shapes, FileBrowser for local drives, .fab import and
download, /api/debug/*, all three ServeFile sites, registering the backup dir as a
drive). One is not: FileBrowser binds a NETWORK share at its ROOT while local drives
are userdata-scoped, and no IsNetwork() filter guards the deploy dropdown, the per-app
migrate targets, or handleStorageMigrateApp. So an app namespace - and under D5 its
plaintext secrets - can land somewhere a browsing surface serves with download:true.
Not a leak today (the unit's app.yaml is secret-stripped); it becomes one under D5.
D5 is therefore written as a BLOCKED target, not as settled. -> R-108
Core artifact is the failure -> recovery matrix (15 rows). RTO cells carry only
measured durations; 12 are blank and each blank is listed explicitly as a finding.
RPO cells carry the configured cadence that bounds RPO, labelled as such - no RPO has
ever been measured from an incident and none is invented.
The 7/53 vs 9/43/1 Tier-2 coverage counts are both recorded, unresolved; neither adopted.
Coupling (S-1): capability map section C now cites the matrix instead of restating
routes; ROADMAP gains R-102..R-108; OPEN-ITEMS gains only the in-flight R-108 plus
C9-F1b/F3/F4 -> R-n cross-references; CONTEXT records D1-D6 as standing ruling S-3.
No code, no version bump, no deploy, no box touched.
Phase 0 sized C9-F1 properly before anything was designed: 43 of the 53 catalog apps have NO
subtree the Tier-2 restore can read (not 2), 9 are covered only for their file legs and never
their database or volumes, 1 is stateless. The asymmetry is Tier-2's alone — Tier-1 and offsite
both restore the unit and replay volume dumps, so BookStack always had a working restore and
only this button lied.
Shipped: the restore refuses BEFORE stopping the app and names the action that does work; a run
that proceeds claims only what it EXAMINED and discloses that the database and volumes are not
covered. C9-F2 alarms after a 5-minute sustained-restarting threshold, set above the 120s deploy
timeout, Mealie's 60s start_period and R-97b's 180s grace; StateRestarting is deliberately NOT
added to IsDownState.
Live: silent through ten 30s samples then app_start_failed at 5m25s, heartbeat now reads
"1 currently down" where Campaign 9 recorded 0; a real deploy stayed silent; bookstack refused
with its uptime unbroken; paperless re-restored 43/43 byte-identical, 16/16 docs clean.
Filed, not fixed: C9-F1b (route to the Tier-1 restore — its own task because it puts a
destructive operation behind a non-destructive button) and C9-F4 (nothing reads the Tier-2
copy's recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable
by any customer action — potentially larger than C9-F1).
Phase A is the headline and it passed on live hardware, through the real endpoints the UI
posts to: a customer who deletes files — or their entire app data directory — gets everything
back byte-identical, and the app works afterwards (paperless served the restored bytes over
its own API at the exact pre-deletion sha256). A1's two non-destruction promises both hold.
Three defects, recorded not fixed:
C9-F1 (HIGH) the Tier-2 restore button is offered for apps it can never restore, takes a
real outage, and reports "nothing was missing" — indistinguishable from a
genuine result, while 156 MB of that app's data sits unread in the same copy.
C9-F2 (HIGH) an app in a crash loop never alarms on any channel; StateRestarting is in no
down-set, so F-OBS's own heartbeat printed "0 currently down" for 9 minutes.
C9-F3 (MEDIUM) an interrupted offsite run leaves a lock the self-heal cannot reach; the tier
is dead until a human unlocks and the operator is told "unknown reason".
This answers Phase C item 8.
Two candidates were deliberately NOT filed: a recovery-unit poisoning the catalog sync healed
in ~3 min, and a snapshot_id that is documented as logging-only. Reporting either would have
been reporting an artifact.
Stopped at the end of Phase B (plus D10), then full recovery — both boxes healthy, real
cadences, offsite tier proven working again, no leaked scratch guests, peti untouched.
D11's approved staleAfter compression turned out not to be a knob; reported, not worked around.
F-LEAK's first fix (pool adoption) was refuted live and is recorded as such. R-100 is
F-CRIT-2's defect class on the hub for the restic tier: isStale reads only LastRun, which
the controller writes unconditionally on failure. Investigated, not fixed.
Both marked SHIPPED + PROVEN-LIVE in OPEN-ITEMS and the campaign doc. All three
of Campaign 8's alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1).
Adds the standing rule earned by this arc to the versioned workspace CLAUDE.md:
a comment asserting an invariant needs a test pinning it, or it is a wish — with
all six shipped-false-guarantee instances catalogued, and the corollary that a
test should assert the CONSEQUENCE (does the alarm fire?) not the MECHANISM
(does suppression expire?).
F-CRIT-2 marked SHIPPED + PROVEN-LIVE: NewestArchiveTime now counts only
plausibly-complete entries (measured 1 MiB floor), verified by replaying
campaign fault 2 on demo-hp in both directions.
Files two follow-ups:
- R-99: prune never REMOVES phantoms. It does NOT count them toward keep-last
(dry-run kept 2 real + the phantom), so there is no retention/data-loss bug —
but they accumulate one per aborted upload.
- F-CRIT-1: filed READY-HIGHEST. Campaign 8's other HIGH finding, untouched by
this fix.
Seam sweep: TieredBackend was the FIRST, not the only one. BackupArchiveLister
has the identical silent-degrade shape and a worse blast radius (it degrades to
the pre-R-84 in-memory-only behaviour), and no compile-time witness existed in
production code anywhere in either repo. No defect found, so no version bump and
no deploy — the witnesses are guards, proven by breaking a signature and watching
go build fail where it previously passed.
Live outage: age_state=unknown captured on real hardware for the first time, with
demo-felhom's local tier genuinely due throughout — the controller deferred and
zero app stacks were stopped. The R-88 breaker did NOT arm and no
whole_guest_backup_failed travelled, because felhom-pbs was not due; recorded as
conditions-did-not-arise rather than claimed as coverage.
Post-boot: the volume changed device name (sdb->sda) across the reboot and the
mount survived only because fstab uses by-id. That was never tested before.
R-88b closed (agent v0.105.0 + controller v0.178.0) — age_state gives 'unknown'
its own representation, with empty meaning legacy rather than unknown so the
first-backup valve keeps working on un-upgraded boxes.
R-97 note updated: hub v0.79.0 (R-97c) replaced a FALSE operator-only comment
with a real register — the comment claimed a guarantee the code did not provide.
Standing rule 4 (R-96): a recommendation that is not followed gets one line
saying why. Added to the live CLAUDE.md and this versioned copy — the live file
is not in a git repo, so committing to it alone would leave the rule as durable
as the chat it came from.
READY re-ranked: R-95 now leads.
R-97 collapsed to its shipped one-liner in ROADMAP and closed in OPEN-ITEMS.
PROMPT-TEMPLATE N.5 now names FOUR coupled artifacts instead of two: the
capability map, ROADMAP, the owning architecture doc (ruled as S-1 in CONTEXT.md
but never reflected in the template CC actually reads, so it bound nobody), and
OPEN-ITEMS.md. Tasks must now report which register rows they opened, closed or
re-ranked.
Ops: R-90 swap done (interim; CX33 still blocked), R-95 mitigation armed but zero
snapshots taken so it moves to WATCHING rather than closed, R-91 gate still not
satisfied. CONTEXT.md datastore path corrected to /mnt/pbs-datastore.
R-88 split: Part 1 (the failure breaker) SHIPPED in controller v0.176.0 and live
on both boxes; Part 2 (unknown != never) stays OPEN and is agent-side.
Phase 0 established the root cause at source: newestArchiveOn's (time.Time, bool)
signature cannot represent 'unknown', so a storage read ERROR collapses into a
positive 'no successful backup recorded yet'. The errored and genuine-never paths
are byte-identical on the wire, which is why Part 2 cannot be done controller-side.
R-97: the whole-guest backup tier has no failure signal to the hub at all —
internal/quiesce never imports internal/notify, so three failed backups and three
app-stack outages produced zero backup_failed events. Its only trace was a
customer-tier Hungarian app_start_failed for an app the backup itself had stopped.
Open state was spread across ROADMAP, CONTEXT.md, four audits/, three runbooks,
per-session REPORT.md files and a chat log. This is the one page to read first:
every row has a state (BLOCKED/READY/WAITING-ON-OPERATOR/WATCHING) and an owner,
and the READY rows are ranked with reasoning.
R-88 is the recommended next task — quiesce's nil-age fail-open stops every app
stack every 5 minutes with no backoff and bypasses the maintenance window, and
its trigger (a PBS read failure) is live given ep0's demonstrated OOM.
CONTEXT.md now records that OPEN-ITEMS.md is authoritative and that REPORT.md is
overwritten per session.
admin