BoundUnderParent is now a CONJUNCTION: bound under the parent AND the drive's
raw host mount still mounted. The raw mount is the device-bound systemd unit
that dies with the device; the agent's own bind is not, which is why the bind
outlived the device and the gate could never fire.
Conjunction deliberately, not replacement: the device half alone would regress
boot ordering (raw mounts early, bind lands ~18s later — that window must keep
reading absent), so existing behaviour is byte-identical and only the
unreachable case is closed. Unknown is never absent.
Controller UNCHANGED, no MinAgent bump — BoundUnderParent has exactly one
functional consumer (planDriveGates:226). A new DevicePresent bool was rejected:
absent-from-JSON decodes to false, so every drive on an older agent would have
read ABSENT and stopped its apps.
+6 tests (208->214), 4 red-proofs run and reverted. Deployed to demo-felhom and
the over-correction guard verified in production: raw mount present, drive still
reads present, 10/10 apps untouched, no gate action, no false alarm. demo-hp
deliberately left on 0.113.0 (the spec scoped deploy to felhom-pve).
SESSION C BLOCKER recorded on the row: the hub Day-0 manifest vouches agent
0.113.0, so a fresh drill box would install WITHOUT this fix and validate
nothing. Publish + vouch 0.114.0 first — R-111's trap in the same shape.
The delete was correctly refused at four successive gates while the host still
read ONLINE (acknowledgements -> typed confirm_id -> expect_hosts stale-preview
-> "host is ONLINE"). Rather than force it, the run waited for the destroyed
host to age to DOWN; delete-impact then reported deletable:true and the
documented cascade ran:
host deleted (escrow demoted to retained custody), tenantsync deprovisioned,
PBS tenancy deprovisioned, claim reset to unclaimed, residue purged
(reports=5 app_telemetry=5 notif_prefs=1 appliance_registrations=1)
Verified after: 0 occurrences of "e2d" anywhere on the hosts page; demo-felhom
and demo-hp ONLINE on agent 0.113.0; drill-r50 and peti-felhom unchanged;
demo-hp carries only guest 9201 and VM 300.
Scoping checked rather than assumed: the single purged appliance_registration
was this run's own appliance (810d10c5, bound to e2d-fresh). The unrelated stale
2026-07-25 appliance (206c8838 / QWA-WJE) was NOT touched by the cascade — the
operator removed it separately.
- OPEN-ITEMS.md: the drill-cleanup WATCHING row is removed (done, not open).
- audits/E2D-fresh-vm-2026-07-29.md §8 + REPORT-e2d.md: teardown recorded as
complete, with the cascade output and the appliance-scoping note.
Full ISO/PAIRING route on a nested PVE VM on demo-hp, after R-111 was fixed
earlier in the session. Bind -> running controller in 3m35s. The install fetched
the artifacts published an hour before and restored the golden baked 20 minutes
before, so the publish train is proven end to end on a real install.
C1 PROVEN: "felhom-host-install v1.22.0", "Day-0 provision SUCCESS", guest 9201
running, bootstrap unit wrote its done-flag and self-disabled. This retires E-2's
"installer-logic-tested, not install-tested".
C2 PROVEN: both DEGRADED warning lines verbatim, backup.local_backup_target=local,
no felhom-backup storage created, and the install did not abort.
C3/C4 PARTIAL and C5 FAILED — three findings, none fixed:
R-112 (P1): E-2's degraded banner and offer have NO UI CONSUMER. The endpoint
returns byte-exact copy; grep 'backup-target' across every html/js/css is 0 hits
and no page handler injects the state. Templates fetch 18 distinct /api/storage/*
endpoints; these two are the only ones with zero references. v0.185.1 fixed the
router mount and stopped one layer short of the render. Fifth instance of
seam-built-but-never-wired.
R-113 (P1): the drive-absent gate CANNOT FIRE on device loss. planDriveGates
reads presence from BoundUnderParent = "is this path in the guest's mountinfo".
The raw mount is a device-bound systemd unit and dies with the device; the
agent's own bind is not device-bound and outlives it, so the gate sees "present"
forever. Live: agent reported the drive absent every 20s for 4.5 minutes, the
controller logged 0 [gate] lines, the hub received zero events -- neither
backup_target_absent nor the generic storage_disconnected. Sixth instance of the
class: E-2b wired the seam to a condition that cannot occur.
R-114: on target-drive loss the message claims the backup is on the system disk
(false) and offers the drive that just vanished. Invisible only because of R-112,
so it must be fixed BEFORE R-112 is wired.
Also filed as a second instance under R-110 rather than a new ID: host-install
fetches nine files from raw/branch/main and the hub vouches a sha for one;
E-2a's wrapper is installed 0755 to /usr/local/sbin, root-fenced in sudoers,
validated only by bash -n.
C4 is fully proven at API level: decline path (registration confers no role),
restart_required:true, agent did NOT self-restart (in-flight check performed and
recorded first), E-2a wrapper created the storage at the drive's own mountpoint,
and healthy renders nothing.
Teardown: VM destroyed, scratch storage removed, pvesm status after == before
(local-lvm 38.77%), guest 9201 and drill-r50 untouched. Hub records for e2d-fresh
remain -- delete correctly refused at four gates, finally "host is ONLINE";
deletable once it ages to DOWN. Command recorded in OPEN-ITEMS.md.
capability-map NOT touched: the customer-facing legs are broken rather than
proven, and the map has no E-2 rows at all.
Found and fixed the same day. The channel was 17 agent releases stale — a box
installed today would have received agent 0.96.0 and controller 0.161.0.
- agent 0.113.0 built from the clean tree @ 58b598b and published via
scripts/publish-agent.sh; sha 5f3247f756cb658e…, round-trip GET verified.
- golden 0.185.1 baked on the nested drill VM embedding controller 0.185.1;
sha dba00f3e845c415e…. Bake clean: Result=success, overlay2, all 3 mounts
included (rootfs+mp0+mp1), 0 FATAL/exclusions, HTTP 201, token-leak grep 0.
GL-1 teardown: guest 9100 purged, secrets shredded, drill disk restored to
the virgin snapshot. Log saved to drill/bake-0.185.1.log.
- Hub Day-0 manifest: agent and golden moved TOGETHER in one POST so the
manifest never vouched a new agent against an old golden. min_agent
0.93.0 -> 0.113.0, which is what controller v0.185.0 declares. Zero fleet
impact, verified: all three enrolled hosts already run agent 0.113.0.
wrapper_sha256 preserved verbatim (re-checked, no drift).
- The global controller floor was deliberately NOT raised: the golden now
bakes 0.185.1, so a fresh box needs no self-update.
This unblocks E-2d C3/C4/C5, which the Phase 0 gate had blocked.
No VM created, no install run, no box touched. The run stopped at the Phase 0
gate per runbook §3, before provisioning.
felhom-host-install.sh does not install what is on main. resolve_artifacts()
(:423-436) reads the hub-vouched manifest and fetches Gitea GENERIC PACKAGES
(agent :1945, golden :2573). Gitea's newest are agent 0.96.0 and golden 0.161.0;
the hub manifest selects exactly those; the global floor v0.156.0 is below the
golden's 0.161.0 so nothing self-updates. A fresh box therefore lands on agent
0.96.0 + controller 0.161.0 against main's 0.113.0 / 0.185.1. Agent 0.113.0
reached both demo boxes by direct deploy and is not in the channel at all.
Claim impact, each pinned to its introducing commit:
- C1 (real rc=0 1.22.0 install) and C2 (Case B natural) — ACHIEVABLE, not run;
both are installer-side and host-install is served at 1.22.0.
- C3 — BLOCKED: banner + GET /api/storage/backup-target are controller v0.185.1
(cdaeb36), copy v0.185.0 (3f7cf2a). Unblocks cheaply by raising the hub floor
to >=0.185.0; measured fleet impact nil (both demo boxes already 0.185.1).
- C4 — BLOCKED: needs controller v0.185.1 + agent v0.113.0 (58b598b).
- C5 — BLOCKED: needs controller v0.184.0 (c1a63de) + agent v0.112.0.
Filed R-111 (P1): 17 unpublished agent releases (v0.97.0-v0.113.0) strand the
entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT, so a new customer's
box installs without them. Mirror of R-110, not a duplicate.
- audits/E2D-fresh-vm-2026-07-29.md — all four Phase 0 answers recorded so a
resumed run does not re-derive them (cadence 30s; hot-detach available; ISO
present; local-lvm fence re-measured at 38.77%, unchanged).
- OPEN-ITEMS.md — R-111 opened; E-2d re-stated, NOT closed.
- ROADMAP.md — R-111 under P1.
- capability map NOT touched: nothing was proven live.
The §5.1a operator STOP is retired — HUB_PW is in ~/.config/credentials and hub
auth was verified, so CC can bind on a resumed run.
Closes the record-hygiene rider. Part 3 of the spec (documenting a
ROADMAP/OPEN-ITEMS state convention) is deliberately NOT done — its stated
evidence is false; see REPORT-record-correction-2026-07-29.md.
- CONTEXT.md:540 — "scripts/hub_confirm_gate.py enforces" was present tense
about a gate invoked by nothing. Now says it asserts but is not enforced
(R-29). Third instance of the class after :564 and configs.go:27.
- REUSE.md:62 — same claim, "enforces zero". The RULE stays (never native
confirm()/prompt() is correct guidance and this is a reuse-reference row);
only the enforcement claim changes, and it now says the rule holds only as
long as you keep it.
- OPEN-ITEMS.md:4 — root REPORT.md is the overwritten per-session file;
REPORT-<topic>.md is the non-clobbering sibling form (CLAUDE.md:82-87), of
which 14 exist. The prohibition on durable content living only there stays.
- OPEN-ITEMS.md:55 — "Why the READY rows rank this way" promised a complete
ordering and listed 5 of ~15 open rows. Scoped to TOP, with a half-sentence
saying it is deliberately not a full ordering. No row added to the list.
hub/internal/web/configs.go:27 left alone (R-94 leg (b), needs a hub build).
No gate wired, run or fixed. Documentation only, no version bump, no CHANGELOG.
d4c07873 filed "hostinstall_gates.py is invoked by nothing" as a novel
observation. It is not novel — R-29 already names the class (green gates are
enforced nowhere; one sat RED for 16 releases while every REPORT said green),
and R-29 was missing from OPEN-ITEMS.md entirely, having never been carried
across the 2026-07-27 register rebuild. An open item about work not getting
done was absent from the page that decides what gets done.
Ruling on whether R-29 is the right home for a non-design-v2 gate: YES. Its
title says design-v2, but its own audit list already spans mount-safety,
secrets and dedup gates across four repos, and its part (b) — "the systemic
half is the real item" — is about the enforcement mechanism, which is
gate-agnostic. hub_confirm_gate.py is already on its list and sits in the same
scripts/ directory. No new ID minted; R-29's own text forbids it, and this is
the third re-raise it has absorbed.
- OPEN-ITEMS.md: open R-29 (READY, S(a)/M(b)), with the orphan evidence and
the two separable parts R-29 already defines.
- OPEN-ITEMS.md: R-94 leg (b) now points at R-29 as its class.
- ROADMAP.md:158: audit list extended with hostinstall_gates.py (RED today,
1.19.0 != 1.22.0) + hub_confirm_gate.py verified orphan. Entry not rewritten.
- ROADMAP.md:147: cited a non-existent R-164 — it means controller v0.164.0.
- CONTEXT.md:564: asserted in the present tense that the version cross-check is
"gated by scripts/hostinstall_gates.py". It exists, is red, and runs nowhere.
- OPEN-ITEMS.md: READY #1/#3/#4 markers dropped — they duplicated ranked-list
positions and the gap was left by the row merged in d4c07873.
- OPEN-ITEMS.md: E-2d citation :322-341 widened to :322-343; the invocation it
describes is at :343, two lines outside the old range.
- backlog/README.md: two-line lead naming OPEN-ITEMS.md and ROADMAP.md.
- REPORT-record-correction-2026-07-29.md: the report CLAUDE.md:82-87 requires
for both commits. Root REPORT.md (E-2 increment 1) untouched.
No gate wired, fixed, run or deleted — that is R-29 part (b), its own task.
Documentation only. No version bump, no CHANGELOG entry, no box touched.
The 2026-07-29 R-94/E-2d finding was written from an unverified claim and was
false. `felhom-bootstrap.sh:96` fetches the installer from the WEBSITE, not the
hub; the website git-syncs /scripts/ from main on a 30s period; every install
since 1.22.0 hit main this morning already runs 1.22.0. Confirmed by live fetch.
- OPEN-ITEMS.md: merge the two duplicate R-94 rows into one, retract the false
framing, re-scope to what it actually is (a drifting hand-synced constant plus
two pieces of dead safety equipment), unblock it from E-2d.
- OPEN-ITEMS.md: de-rank R-94 in the ranked list — the "high-consequence" reason
was the false claim in its most load-bearing form.
- OPEN-ITEMS.md: E-2d — the ISO is the STRONGER proof route, not an obstacle.
Phase 0 question answered at source: PAIRING falls through to run_direct in
the same invocation (:495-499), so it reaches the identical installer call.
- ROADMAP.md:149: same retraction; the original diagnosis (a hand-synced
constant in a second repo drifts every time the first ships) survives.
- ROADMAP.md + OPEN-ITEMS.md: new R-110 — main is the installer's publish
channel and there is no staging, tag, pinned path or rollback, for the one
artifact that runs as root on a virgin box. Operator ruling, not a defect.
- day0-install.md C.1: one sentence recording the same about the fetch URL.
Documentation only. No version bump, no CHANGELOG entry, no code, no box touched.
Space checked on the t740 -- NOT a blocker, with one constraint: the VM disk must
not go on local-lvm. That thin pool is over-subscribed (144G allocated against a
54G pool) on a box running a live customer guest, and a full thin pool corrupts
every guest on it. local has 23.7G on pve-root. Use /mnt/nvme-1tb (888G free).
Confirmed the ISO does NOT bake felhom-host-install.sh -- it ships
felhom-bootstrap.sh, which fetches the installer FROM THE HUB. Since the hub
serves 1.19.0, a fresh ISO install today would run the pre-E-2 installer and
exercise neither Case A nor Case B. So R-94 must be bumped only AFTER a real
1.22.0 run, not before -- which is the ordering already decided.
drill-r50 stays blocked and was restored to its r50pre state: the agent upgrade,
the added disk and the moved backup target from this session are all reverted.
Live: hub 0.81.0, agent 0.113.0, controller 0.185.1 on both demo boxes;
host-install 1.22.0 (script; no reinstall performed).
E-2a wrapper proven live as root on demo-hp: F-1 subdirectory refused, F-2
unmounted path refused, root device refused, idempotent re-apply is a no-op,
repointing refused -- 0 stray storages. The agent PVE role was NOT widened.
Scenario E proven live on BOTH boxes: healthy renders nothing, no message key.
Records three defects I introduced and caught: unreachable routes (mounted
outside /api/storage/, caught by the first live call), a hollow test exposed by
its own red-proof, and another gofmt-realignment no-op.
Not live-proven: the degraded banner and offer acceptance (both boxes healthy),
backup_target_absent end-to-end, Case A/B on a real install, drive-loss recovery.
hub 0.81.0, agent 0.112.0, controller 0.184.1 live on BOTH demo boxes.
E-2c: eject/decommission of the backup-target drive refused 409 on both boxes,
drives unmoved. E-2b: the never-called disconnect seam is wired, with the target
case raising the specific backup_target_absent.
Records the keying bug caught before deploy (a.Path is the GUEST path, so the
target branch was unreachable -- 0.184.0 superseded, never deployed) and states
plainly that backup_target_absent is NOT proven end-to-end live: proving it needs
a live enrolled drive to go absent.
Parts 2/3/4 and E-2a remain open; Peti risk stays parked.
E-2 is partially shipped (hub v0.81.0 + controller Part 1). Filing the rest so a
foundation with no UI cannot quietly become a sixth seam-built-but-never-wired.
E-2 remaining: installer Case A/B, the offer + agent-side move, the degraded
banner, the controller half of the signal, red-proofs E/F, live validation.
Phase 0 INVERTED the emphasis: the installer has no drive-enrollment step,
so the common case at install is system-drive-only and Part 3 (drive added
later) is the PRIMARY path, not Case A.
E-2a the move needs a root-fenced wrapper -- the agent holds neither
Datastore.Allocate at /storage nor Permissions.Modify, and its sudoers has
no pvesm and no pveum. Use the guarded-wrapper pattern; do NOT widen the
agent's PVE role.
E-2b NotifyStorageDisconnected/Reconnected are defined and called NOWHERE, so a
drive going absent emits no event at all. Hub side is already plumbed, so
wiring needs no hub change.
E-2c E-1 put the whole-guest backups on a drive POST /disks/eject will eject
(RoleForStorage returns user-data for a local-dir on a non-system device).
Guard the eject specifically -- reclassifying the drive RoleBackup would
block legitimate ejects, since it is also the enrolled user-data drive.
PETI peti-felhom deliberately NOT migrated; drive failure there is offsite-only
recovery. Accepted until the operator's reinstall; re-evaluate if that
slips past ~2026-09-01.
Supervised operational run. No code, no version bump, nothing deleted.
Primary backup tier on both demo boxes moved from `local` (a dir storage on
/var/lib/vz -- the SAME physical device as the guest) to `felhom-backup`, a dir
storage on each box's secondary drive:
demo-hp /mnt/nvme-1tb uuid:91d2dc2d-... archive 2,256,044,492 B
demo-felhom /mnt/hdd_1 uuid:47a3361a-... archive 5,957,878,962 B
Both proven end to end via the real UI path: archive lands on the secondary
drive (df delta matches the archive byte-for-byte), restore-test auto-selects it
and passes with mount_parity: ok, and freshness survives an agent restart with
an empty in-memory store -- so the age can only have come from the new storage.
Phase 0: the target is CONFIGURATION, not converged (the sole writer of
agent.json touches only escrow.pbs_storage_id and preserves unknown keys), so
the runbook's STOP did not fire. No consumer hardcodes "local" on the backup path.
Findings:
- F-1 the storage path must BE the mountpoint; a subdirectory fails exactMount
and the target reports disconnected permanently (observe.go:321)
- F-2 --is_mountpoint 1 is load-bearing; proven live, an unguarded storage on a
non-mounted path reports active with the ROOT filesystem's free space and
had already created dump/ on pve-root -- a silent retarget onto the very
device this change escapes
- F-3 FelhomAgentStore is granted per storage path; without it every backup
403s. felhom-host-install.sh must issue it for new installs
- R-109 (new) the DR recipe records no backup target, and each box now carries
two content=backup dir storages, one live and one frozen
- R-105 narrowed and TRACED: dr_recipe drives was [] fleet-wide because the
enrolled drives were never PVE storages, so isUserDataDrive never saw
them. Both boxes now populate drives; SMART on the backup drives too
Absent-drive behaviour today is fail-loudly with no silent retarget (PVE half
live-proven; agent half source-traced). That is NOT the intended fall-back-and-
alarm design -- filed as E-2 with the honest single-drive label.
Reported in full in the record: the agent was restarted with a felhom-pbs backup
in flight, producing a spurious tier failure. The backup had in fact succeeded
(PVE task OK, 6,264,034,053 B snapshot) and the spurious failure reached no
channel -- R-84 ground truth superseded it.
Outstanding: full drive-loss recovery (needs physical access) and the agent half
of the absent-drive behaviour.
Replaces the 2026-07-14 DRAFT entirely - it was verified against controller v0.132.0,
51 versions stale, while being cited as authoritative. Now verified against controller
0.183.0 / agent 0.110.0 / hub 0.80.0 / catalog 4252121. NOT ratified.
Every statement is labelled [DESIGN] (a decision from the architecture discussion) or
[FACT] (file:line, live output, or a citation to _recovery-inventory-2026-07-28.md).
Phase 0 verified D5's premise and it FAILED. Ten read surfaces were checked; nine are
clean (SMB under both namespace shapes, FileBrowser for local drives, .fab import and
download, /api/debug/*, all three ServeFile sites, registering the backup dir as a
drive). One is not: FileBrowser binds a NETWORK share at its ROOT while local drives
are userdata-scoped, and no IsNetwork() filter guards the deploy dropdown, the per-app
migrate targets, or handleStorageMigrateApp. So an app namespace - and under D5 its
plaintext secrets - can land somewhere a browsing surface serves with download:true.
Not a leak today (the unit's app.yaml is secret-stripped); it becomes one under D5.
D5 is therefore written as a BLOCKED target, not as settled. -> R-108
Core artifact is the failure -> recovery matrix (15 rows). RTO cells carry only
measured durations; 12 are blank and each blank is listed explicitly as a finding.
RPO cells carry the configured cadence that bounds RPO, labelled as such - no RPO has
ever been measured from an incident and none is invented.
The 7/53 vs 9/43/1 Tier-2 coverage counts are both recorded, unresolved; neither adopted.
Coupling (S-1): capability map section C now cites the matrix instead of restating
routes; ROADMAP gains R-102..R-108; OPEN-ITEMS gains only the in-flight R-108 plus
C9-F1b/F3/F4 -> R-n cross-references; CONTEXT records D1-D6 as standing ruling S-3.
No code, no version bump, no deploy, no box touched.
Phase 0 sized C9-F1 properly before anything was designed: 43 of the 53 catalog apps have NO
subtree the Tier-2 restore can read (not 2), 9 are covered only for their file legs and never
their database or volumes, 1 is stateless. The asymmetry is Tier-2's alone — Tier-1 and offsite
both restore the unit and replay volume dumps, so BookStack always had a working restore and
only this button lied.
Shipped: the restore refuses BEFORE stopping the app and names the action that does work; a run
that proceeds claims only what it EXAMINED and discloses that the database and volumes are not
covered. C9-F2 alarms after a 5-minute sustained-restarting threshold, set above the 120s deploy
timeout, Mealie's 60s start_period and R-97b's 180s grace; StateRestarting is deliberately NOT
added to IsDownState.
Live: silent through ten 30s samples then app_start_failed at 5m25s, heartbeat now reads
"1 currently down" where Campaign 9 recorded 0; a real deploy stayed silent; bookstack refused
with its uptime unbroken; paperless re-restored 43/43 byte-identical, 16/16 docs clean.
Filed, not fixed: C9-F1b (route to the Tier-1 restore — its own task because it puts a
destructive operation behind a non-destructive button) and C9-F4 (nothing reads the Tier-2
copy's recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable
by any customer action — potentially larger than C9-F1).
Phase A is the headline and it passed on live hardware, through the real endpoints the UI
posts to: a customer who deletes files — or their entire app data directory — gets everything
back byte-identical, and the app works afterwards (paperless served the restored bytes over
its own API at the exact pre-deletion sha256). A1's two non-destruction promises both hold.
Three defects, recorded not fixed:
C9-F1 (HIGH) the Tier-2 restore button is offered for apps it can never restore, takes a
real outage, and reports "nothing was missing" — indistinguishable from a
genuine result, while 156 MB of that app's data sits unread in the same copy.
C9-F2 (HIGH) an app in a crash loop never alarms on any channel; StateRestarting is in no
down-set, so F-OBS's own heartbeat printed "0 currently down" for 9 minutes.
C9-F3 (MEDIUM) an interrupted offsite run leaves a lock the self-heal cannot reach; the tier
is dead until a human unlocks and the operator is told "unknown reason".
This answers Phase C item 8.
Two candidates were deliberately NOT filed: a recovery-unit poisoning the catalog sync healed
in ~3 min, and a snapshot_id that is documented as logging-only. Reporting either would have
been reporting an artifact.
Stopped at the end of Phase B (plus D10), then full recovery — both boxes healthy, real
cadences, offsite tier proven working again, no leaked scratch guests, peti untouched.
D11's approved staleAfter compression turned out not to be a knob; reported, not worked around.
F-LEAK's first fix (pool adoption) was refuted live and is recorded as such. R-100 is
F-CRIT-2's defect class on the hub for the restic tier: isStale reads only LastRun, which
the controller writes unconditionally on failure. Investigated, not fixed.
Both marked SHIPPED + PROVEN-LIVE in OPEN-ITEMS and the campaign doc. All three
of Campaign 8's alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1).
Adds the standing rule earned by this arc to the versioned workspace CLAUDE.md:
a comment asserting an invariant needs a test pinning it, or it is a wish — with
all six shipped-false-guarantee instances catalogued, and the corollary that a
test should assert the CONSEQUENCE (does the alarm fire?) not the MECHANISM
(does suppression expire?).
F-CRIT-2 marked SHIPPED + PROVEN-LIVE: NewestArchiveTime now counts only
plausibly-complete entries (measured 1 MiB floor), verified by replaying
campaign fault 2 on demo-hp in both directions.
Files two follow-ups:
- R-99: prune never REMOVES phantoms. It does NOT count them toward keep-last
(dry-run kept 2 real + the phantom), so there is no retention/data-loss bug —
but they accumulate one per aborted upload.
- F-CRIT-1: filed READY-HIGHEST. Campaign 8's other HIGH finding, untouched by
this fix.
Seam sweep: TieredBackend was the FIRST, not the only one. BackupArchiveLister
has the identical silent-degrade shape and a worse blast radius (it degrades to
the pre-R-84 in-memory-only behaviour), and no compile-time witness existed in
production code anywhere in either repo. No defect found, so no version bump and
no deploy — the witnesses are guards, proven by breaking a signature and watching
go build fail where it previously passed.
Live outage: age_state=unknown captured on real hardware for the first time, with
demo-felhom's local tier genuinely due throughout — the controller deferred and
zero app stacks were stopped. The R-88 breaker did NOT arm and no
whole_guest_backup_failed travelled, because felhom-pbs was not due; recorded as
conditions-did-not-arise rather than claimed as coverage.
Post-boot: the volume changed device name (sdb->sda) across the reboot and the
mount survived only because fstab uses by-id. That was never tested before.
R-88b closed (agent v0.105.0 + controller v0.178.0) — age_state gives 'unknown'
its own representation, with empty meaning legacy rather than unknown so the
first-backup valve keeps working on un-upgraded boxes.
R-97 note updated: hub v0.79.0 (R-97c) replaced a FALSE operator-only comment
with a real register — the comment claimed a guarantee the code did not provide.
Standing rule 4 (R-96): a recommendation that is not followed gets one line
saying why. Added to the live CLAUDE.md and this versioned copy — the live file
is not in a git repo, so committing to it alone would leave the rule as durable
as the chat it came from.
READY re-ranked: R-95 now leads.
R-97 collapsed to its shipped one-liner in ROADMAP and closed in OPEN-ITEMS.
PROMPT-TEMPLATE N.5 now names FOUR coupled artifacts instead of two: the
capability map, ROADMAP, the owning architecture doc (ruled as S-1 in CONTEXT.md
but never reflected in the template CC actually reads, so it bound nobody), and
OPEN-ITEMS.md. Tasks must now report which register rows they opened, closed or
re-ranked.
Ops: R-90 swap done (interim; CX33 still blocked), R-95 mitigation armed but zero
snapshots taken so it moves to WATCHING rather than closed, R-91 gate still not
satisfied. CONTEXT.md datastore path corrected to /mnt/pbs-datastore.
R-88 split: Part 1 (the failure breaker) SHIPPED in controller v0.176.0 and live
on both boxes; Part 2 (unknown != never) stays OPEN and is agent-side.
Phase 0 established the root cause at source: newestArchiveOn's (time.Time, bool)
signature cannot represent 'unknown', so a storage read ERROR collapses into a
positive 'no successful backup recorded yet'. The errored and genuine-never paths
are byte-identical on the wire, which is why Part 2 cannot be done controller-side.
R-97: the whole-guest backup tier has no failure signal to the hub at all —
internal/quiesce never imports internal/notify, so three failed backups and three
app-stack outages produced zero backup_failed events. Its only trace was a
customer-tier Hungarian app_start_failed for an app the backup itself had stopped.
Read-only triage found work that was agreed or discovered but never given an id:
R-95 restic offsite credential CAN delete — answers the parallel question R-89
raised and left open. Per-customer subaccounts report readonly=False, the
controller runs forget --prune from the box, and the sftp: backend cannot
express append-only. Storage Box snapshots (snapshot_limit=10, plan=null,
0 used) are server-side and SFTP cannot delete them — an unused zero-code
mitigation.
R-94 hub pins hostInstallVersion 1.19.0 while host-install ships 1.20.0, so a
hub-driven install still gets the pre-R-82 backup default.
R-90 ep0 has no swap at all and OOM'd today; gates R-86.
R-91 the pre-migration 13 GB datastore copy still occupies ep0's root disk.
R-92 PBS-DR gauge granularity. R-93 drill-r50 fixture tension.
R-96 two standing rules agreed in chat and never committed (the third, N.5's
third leg, IS committed at CONTEXT.md:8).
Open state was spread across ROADMAP, CONTEXT.md, four audits/, three runbooks,
per-session REPORT.md files and a chat log. This is the one page to read first:
every row has a state (BLOCKED/READY/WAITING-ON-OPERATOR/WATCHING) and an owner,
and the READY rows are ranked with reasoning.
R-88 is the recommended next task — quiesce's nil-age fail-open stops every app
stack every 5 minutes with no backoff and bypasses the maintenance window, and
its trigger (a PBS read failure) is live given ep0's demonstrated OOM.
CONTEXT.md now records that OPEN-ITEMS.md is authoritative and that REPORT.md is
overwritten per session.
Supervised runbook execution. No code, no version bump.
The felhom-pbs tier had reported `job errors` on EVERY demo-hp backup
since the tier was created on 07-26, while the data landed correctly
every time: `DatastoreBackup` grants Datastore.Backup but not
Datastore.Prune, so the box's keep_last=2 prune was denied.
Operator ruling: retention is a COMMERCIAL attribute owned by the hub;
ep0 executes. Box tokens therefore stay write-only - a compromised box
must not be able to delete its own offsite backups. No grant was widened
and felhom-tenantsync.sh is unchanged (the ruling makes it correct).
Increment 1:
- boxes stop attempting prune. allowPBSPrune is DERIVED
(`!t.Primary && t.KeepLast > 0`), so keep_last: 0 on the PBS tier
disables both the --prune-backups value and the gate in one config
edit, and the tier stays armed. Verified prune_pbs_allowed=false on
both boxes with no tier REJECTED line.
- per-namespace prune jobs on ep0, keep-last 2, daily 03:30 UTC
(05:30 CEST), dry-run gated. demo-hp 3->2, demo-felhom untouched,
chunk count unchanged (prune removes indexes, not chunks).
Write proof CLOSED: 08:25:47 job errors -> 09:37:29 TASK OK, snapshot
2026-07-27T09:37:29Z, chunks 9787->9813, prune step absent entirely.
Driven through POST /api/guest-backup/trigger (the UI path), not
--selftest and not raw vzdump. Hub gauge evidence explicitly NOT
satisfied - the delta is below its 0.1 GB display granularity.
GC scheduled sun 04:30 UTC and deliberately NOT run: every chunk still
carries a fresh atime from the migration copy, so a run today would
reclaim nothing. verify-new enabled per operator ruling, turning an
inert hub alarm live.
Legacy demo-felhom-01 namespace deleted with its two ACL entries and its
token (operator ruling, confirmed twice) so nothing dangles.
R-89 records the target architecture and carries the unanswered parallel
question: does the restic key on storage-box-pool-1 have DELETE rights?
If so the daily app-data tier has the identical exposure and append-only
is the equivalent answer.
ep0 is Etc/UTC, not CEST - corrected in the record.
Corrects two wrong severity readings with evidence from the box and the code.
The PBS outage was ~15 min (07:00-07:18 UTC), caused by a global OOM at 06:58:12:
proxmox-backup-proxy peaked at 3.2G on a 3.8G box and a concurrent 1.9G rsync
tipped it over. Root SSH to that box works from DooPlex via the public IP, not
from felhom-pve via the tunnel IP — the documented path I failed to try first.
R-88: internal/quiesce has NO failure limiter, backoff or breaker; the loop
stopped after three cycles only because PBS recovered. Verified additionally that
scheduledRunAllowed (quiesce.go:476-478) returns true whenever lastAgeSecs is nil,
so the same missing value that makes every poll due also bypasses the time-of-day
gate — the cycles ran outside the [04:30,08:30) window. Fixing the due-verdict
without fixing the nil-age bypass would leave the hole open.
Measured on demo-felhom while the offsite PBS service was down: the controller
re-polls /backup/due every ~5 min, still gets 'due' (storage unreachable + cold
store), and runs the FULL quiesce cycle each time — all four customer app stacks
stopped and restarted for a backup that cannot succeed. ~19 s of app downtime per
cycle, unbounded. The first entry called this bounded and event-only; it is an
availability fault.
Observed live on demo-felhom 2026-07-27 07:02:57 UTC: an agent restart while the
offsite PBS service was down produced a doomed vzdump at that tier. R-84's
read-error fallback to the in-memory record is correct alone but empty after a
restart, so "cannot read the storage" resolved to "no backup has ever been
taken" = due. Same class R-81 fixed in the hub, one layer down in the agent:
unreachable must be UNKNOWN, not resolved.
The unattended offsite restore-test on demo-felhom passed: 14.46 GB archive,
duration_s=635.07 (10m35s), then it rotated to the local tier. Persisted state
confirms the credit: {"felhom-pbs": "2026-07-27T06:14:42Z"}.
CORRECTION: I estimated ~2 hours for this restore. It took 10m35s. I derived
the estimate from a download rate measured during the FAILED attempt, which was
running under contention; the real link does ~1.4 GB/min. I then used that wrong
figure to raise a design concern — that the heavy-op gate would block backups
for hours on this box — which at 10 minutes largely evaporates. An estimate
extrapolated from a degraded measurement is not a measurement.
The SPEC's closing risk note is corrected in place, with the original left
visible for the lesson.
R-86 (NEXT, operator ruling 2026-07-27): backup-ALIGNED restore-test scheduling
— test a tier ~1 day after ITS OWN backup. R-85 schedules on a free-running
interval, which cannot express 'the day after the PBS backup': any fixed offset
drifts, so alignment would be luck. Shape: trigger from the tier's own last
successful backup rather than a clock. Interim in force: 302400s (3.5d), which
lands each tier ~weekly — the cadence half of the ruling, not the alignment half.
R-87: the restic app-data offsite tier is NEVER restore-tested. R-85 covers
whole-guest vzdump tiers only; the agent has no restic surface. That is arguably
the tier that matters most — the only one that survives losing the box AND
carries the customer's app data, since the whole-guest snapshot excludes the
bind-mounted drives. Exactly the state PBS was in before R-85.
REPORT.md: the full R-80 -> R-85 arc, including a section on the seven mistakes
I made and the two recurring shapes behind them (inferring behaviour from an
artifact instead of the code that consumes it; reading a result without its exit
code). Records demo-felhom's restore-test as IN FLIGHT at close, with the manual
recovery step if the deferred restart watcher does not complete.
Hub gate green (17 packages, rc=0).
- ROADMAP: R-85 row. Code SHIPPED; rotation NOT YET OBSERVED LIVE, stated as
such rather than written as done.
- Capability map: a new row for UNATTENDED restore-proof, IMPLEMENTED not
PROVEN-LIVE, kept distinct from the R-82 row that a MANUAL selftest earned.
That distinction is the same one the activation-vs-arrival split made.
- 03-host-agent §8: the scheduler covers every tier, oldest-proven first; the
spec is per-run; a restore-test joins the one-heavy-op gate. The safety
properties that must not be re-derived are listed.
- 07: restore-proof recorded as a per-tier property. Doc still NOT ratified.
- 06: corrects S4.1's 'the offsite restore-test now runs unattended' — it
silently stopped being true when local_backup_target was retargeted to 'local',
the SECOND time in that doc that a correct mechanism was broken by its input
changing underneath it.
- CONTEXT + REUSE.
Hub gate green (17 packages, rc=0).
Written against verified state, not assumption. Records three gaps the original
Phase 5 ordering does not cover:
1. hub v0.77.0 is COMMITTED BUT NOT DEPLOYED — manifest pins 0.76.0 and the pod
runs 0.76.0, so the R-85 signal exists only in git. The original Phase 5 only
mentions the agent.
2. R-85 has no ROADMAP row.
3. The agent CHANGELOG says v0.104.0-dev; an ldflags version disagreeing with
the CHANGELOG is the reconciliation problem hub 0.73.2 already caused.
One ordering correction: THE HUB GOES FIRST. Agent v0.104.0 makes the offsite
tier testable; hub v0.77.0 makes a failure audible. Agent-first means rotation
begins with nothing listening — two tiers able to fail silently instead of one,
which is the fault R-85 exists to end. Also drops the retired drill box, so the
rollout is demo-hp -> demo-felhom.
Names the phase's most likely SILENT failure: the new rotation state lands at
/var/lib/felhom-agent/restore-test-state.json and the agent is non-root. If that
is not writable, RecordSuccess warns and continues — a quiet return to one tier
being starved, not a crash.
Flags for operator judgement: demo-felhom's 14.46 GB offsite archive makes its
unattended restore-test a ~2h operation every other day, holding the heavy-op
gate throughout. Ruled when the only measured restore was demo-hp's 4 minutes.
R-84 resolved by asking the STORAGE rather than persisting the store: ground
truth, so a pruned archive correctly stops counting where a persisted record
would keep claiming a backup that no longer exists. Proven live on both boxes
with the in-memory store cold.
demo-hp's FIRST EVER offsite backup landed (4.25 GB) — the R-82 finding closed
on the box where it was worst. Controller v0.175.0 deployed to both boxes.
Slice D.1 — host-install 1.20.0: a FRESH box defaults to local-daily +
offsite-weekly (felhom-pbs, 604800s, keep_last=2). setdefault semantics proven
both ways: fresh gets the tier, an UPGRADE preserves the existing backup block
verbatim — so an in-place upgrade can never silently start writing to an
offsite datastore. Existing boxes are migrated explicitly.
Slice E:
- 07-backup-architecture.md: honest status header per CONTEXT ruling S-2, with
an explicit STALE-outside-the-PBS-tier verdict (the controller tiers were last
verified 41 controller versions ago). The PBS row claimed 'PBS on DooPlex'
(the retired spike store) with no cadence; it now names felhom-pbs ->
felhom-offsite on ep0 over wg-felhom, weekly, keep_last=2. NOT marked
ratified — that is Viktor's review of the section 10 list. Discharges R-83.
- 06-offsite-connectivity.md: the target-split remaining-work note collapsed
(shipped), and records HOW S4.1's tier-aware timeout silently regressed — the
mechanism was never removed, its INPUT changed when local_backup_target was
retargeted to 'local'. Also notes S4.1 already diagnosed the teardown 403 as a
phantom (a timeout consequence, not an ACL gap).
- capability map: new row for recurring offsite backups actually LANDING, as
distinct from the existing row proving ACTIVATION. IMPLEMENTED, not
PROVEN-LIVE — the restore round-trip has not completed under the fixed code.
- ROADMAP: R-82 SHIPPED with its remaining gate named, R-83 DISCHARGED, R-84
left open.
- CONTEXT + REPORT: the arc, including the mid-arc correction I had to make.
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class.
On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp
and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel
claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump
archives were on disk. Cause: the agent backup store is in-memory, so the
R-50 fleet restart emptied `backups` until the next run, and the hub read
empty as "no backup exists".
- assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`;
absence is UNKNOWN until it outlives an anchored window. Still pure.
- store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own
retained history (bounded 7-day lookback, early-exit on fresh evidence) —
"when did I last SEE evidence of a backup?" The anchor was free: the hub
already retains 90 days. No agent change, no new persisted state.
- store.GetFirstHostReportAt anchors absence at first contact, reusing the
existing 26h threshold as the grace (no new knob, the v0.73.0 shape).
- Deferrals logged + counted; reason strings kept distinct.
- backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would
alarm six days in seven) and owned by R-82.
Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live
message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all
three now silent.
Source: documentation/audits/DIAG-backup-missed-2026-07-26.md
The allowlist entry is REQUIRED, not cosmetic: handleEvent 400s an unknown
event_type, so controller v0.173.0's new drift alert would be silently inert
without it. Shipped with the controller that emits it.
Docs:
- RUNBOOK-local-api-endpoint-drift.md — how to repair a drift, including the
step everyone will want to skip (establish which value is CORRECT from what
the agent is actually bound to, rather than assuming bootstrap.json wins) and
what success looks like (SILENCE, not a "recovered" line, because a fresh
controller's healthy first observation is not logged). Records both
2026-07-26 repairs.
- ROADMAP: R-77 shipped; R-78 the local_api authority ruling, with the
clobber-a-working-channel risk spelled out in BOTH directions so it is not
resolved opportunistically; R-79 the whole-surface English-strings sweep;
R-80 expected_backup_missed, flagged as likely outranking R-77 because 7.3
days of stale backup materially exceeds the ~1.5-day channel outage, so the
causal link the DIAG hedged on cannot be the whole story.
- Capability map: note against the drive-wizard row (every agent-backed
capability rides this channel) that a silent drift class is now detected.
NO row status flips — detection is not prevention.
New documentation/controller/import-and-data-paths.md: the canonical import root
(and why it is NOT a registered StoragePath), the three data_paths roles, the
Fork-3 validation asymmetry, the class-driven copy rule, and the seven
invariants a future change must not break.
Capability map "File access via browser" — status DELIBERATELY UNCHANGED. The
drop-zone now has its own FileBrowser source and the app page carries a deep
link, both verified live, but nothing drove the FileBrowser HTTP UI (no browser
on DooPlex), so the row's standing "browse is exercised in no doc" caveat still
holds and PROVEN-LIVE remains unearned.
R-75 collapsed to its shipped one-liner. R-76 left open — this task does not fix
it, and nothing built here assumes an import/* directory stays 2775.
R-75 (spiked, GO) names the capability-map row it would flip: "File access via
browser" (00-capability-map.md line 96), currently IMPLEMENTED with the caveat
that browse/download through FileBrowser is exercised in no doc. Carries the
mandatory determinism constraint from P6 (sort + red-proof, or FileBrowser
force-recreates on every sync pass), the zero-removals invariant for
`documents`, the url.PathEscape-not-QueryEscape trap, and the four design forks
with evidence + recommendation, all awaiting operator ruling.
R-76 is minted for the two PRE-EXISTING defects the spike surfaced and
deliberately did not fix: FileBrowser Quantum creating 0644/0755 without
propagating setgid (breaking the shared-group chain one level below any
customer-created folder -- latent only because every userdata-touching app runs
uid 1000), and import/calibre living at 755 on demo-felhom where its same-app
sibling media/books is 2775.
Source: audits/SPIKE-catalog-data-paths-2026-07-26.md
- B2 demo-hp + B3 demo-felhom migrated to the island (agent 0.96.0), apps
served throughout (0 container restarts), island /storage 200, LAN DNS pinned
to the LAN IP, hub reports 0.96.0. No rollback.
- capability-map 'site/network change' row PARTIAL -> PROVEN-LIVE
- ROADMAP R-50 -> SHIPPED (fleet-migrated); add R-74 (island on Peti's cluster)
- nodes.md: both boxes island-bound, agent 0.96.0
Provisioned nested-PVE drill 'drill-r50' (qm300 on demo-hp) via the v1.25.0
nested-vm ISO through the real day-0, then ran the R-50 empirical spike:
- vmbr9 portless island bridge + guest island NIC hot-add (LAN undisturbed)
- F1 replay money shot: LAN move survives on the island; LAN-literal bind
reproduces the 2026-07-20 daemon-exit bug verbatim
- dnsmasq trap confirmed live + lan_resolver.host_ip fix proven
- pin address-independent (leaf SHA-256 unchanged, HTTP 200 over island)
- survival matrix: agent/guest/host-cold-reboot all return on the island
Docs: SPIKE verdict BLOCKED->GO, ROADMAP R-50 SPIKED->GO, nodes.md drill VM,
REPORT overwrite.
New capability-map row (IMPLEMENTED; §13 endpoint-level live on 9201). Records the
ruling: member accounts are superseded by the capability-URL guest share for
launcher sharing; per-member tile visibility parked under the SSO/members arc (R-15).
Updated the launcher row's member-coupling note and R-15 accordingly.
admin
Claude Code