PART 1 — Session C unblocked.
Agent 0.114.0 (the R-113 fix) was built, pushed and deployed but never
published, so a fresh drill box would have installed 0.113.0 and proven the bug
rather than the fix. Published from the clean tree at b58d7bc via
scripts/publish-agent.sh; sha 5e4c15ebee2d7583d57301d1f7c9cc7d4276262966bf738b05e34653bfd18c31,
verified by an INDEPENDENT round-trip GET (http=200, sha match, binary
self-reports 0.114.0), and the hub manifest read back after the write.
Deliberately NOT done, each with a reason:
- No golden bake. The golden bakes the CONTROLLER, not the agent, and
host-install fetches them as separate generic packages (:1945 / :2573). Golden
0.185.1 is current, so there is no new-agent-against-old-golden risk.
- min_agent NOT raised, stays 0.113.0. It expresses what the CONTROLLER requires
of the agent, and controller v0.185.0 declares MinAgent 0.113.0 — which
0.114.0 already satisfies. Raising it to 0.114.0 would have been a false claim
AND would have held demo-hp and drill-r50. No box is held; no §3 STOP fired.
- Global controller floor NOT raised (v0.156.0), per R-111's reasoning.
- wrapper_sha256 preserved verbatim; re-checked against configs/felhom-pbs-apply
before and after — no drift both times.
demo-hp RULING: left on 0.113.0. The R-113 fix is not live-validated, so putting
it on a second box widens exposure for no proof, and Session C's nested box
takes its agent from the manifest, not from demo-hp's host agent. Move the fleet
once, after Session C.
PART 2 — R-115 opened (WAITING-ON-OPERATOR).
The finding is the RECURRENCE, not either instance: publishing is a remembered
step, and it was forgotten within eight hours of R-111 documenting it as
forgettable. Filed as a new ID with a back-pointer rather than reopening R-111,
because R-111's finding (the channel WAS stale) is closed and verified
end-to-end, while the process defect that caused it is a distinct problem with a
distinct fix and owner. Class cross-linked to R-29 (a control that exists and is
never walked) WITHOUT minting a second ID for it. Options are stated as the
operator's decision, with mechanisms (build-step, deploy gate) separated from
reminders (checklist, manual) — R-29's whole finding being that reminders do not
hold. No code written, by design.
R-111 gains a deferred-leg-recurred line; its shipped evidence is untouched and
it is NOT reopened. R-113 records that Session C is now unblocked.
BoundUnderParent is now a CONJUNCTION: bound under the parent AND the drive's
raw host mount still mounted. The raw mount is the device-bound systemd unit
that dies with the device; the agent's own bind is not, which is why the bind
outlived the device and the gate could never fire.
Conjunction deliberately, not replacement: the device half alone would regress
boot ordering (raw mounts early, bind lands ~18s later — that window must keep
reading absent), so existing behaviour is byte-identical and only the
unreachable case is closed. Unknown is never absent.
Controller UNCHANGED, no MinAgent bump — BoundUnderParent has exactly one
functional consumer (planDriveGates:226). A new DevicePresent bool was rejected:
absent-from-JSON decodes to false, so every drive on an older agent would have
read ABSENT and stopped its apps.
+6 tests (208->214), 4 red-proofs run and reverted. Deployed to demo-felhom and
the over-correction guard verified in production: raw mount present, drive still
reads present, 10/10 apps untouched, no gate action, no false alarm. demo-hp
deliberately left on 0.113.0 (the spec scoped deploy to felhom-pve).
SESSION C BLOCKER recorded on the row: the hub Day-0 manifest vouches agent
0.113.0, so a fresh drill box would install WITHOUT this fix and validate
nothing. Publish + vouch 0.114.0 first — R-111's trap in the same shape.
The delete was correctly refused at four successive gates while the host still
read ONLINE (acknowledgements -> typed confirm_id -> expect_hosts stale-preview
-> "host is ONLINE"). Rather than force it, the run waited for the destroyed
host to age to DOWN; delete-impact then reported deletable:true and the
documented cascade ran:
host deleted (escrow demoted to retained custody), tenantsync deprovisioned,
PBS tenancy deprovisioned, claim reset to unclaimed, residue purged
(reports=5 app_telemetry=5 notif_prefs=1 appliance_registrations=1)
Verified after: 0 occurrences of "e2d" anywhere on the hosts page; demo-felhom
and demo-hp ONLINE on agent 0.113.0; drill-r50 and peti-felhom unchanged;
demo-hp carries only guest 9201 and VM 300.
Scoping checked rather than assumed: the single purged appliance_registration
was this run's own appliance (810d10c5, bound to e2d-fresh). The unrelated stale
2026-07-25 appliance (206c8838 / QWA-WJE) was NOT touched by the cascade — the
operator removed it separately.
- OPEN-ITEMS.md: the drill-cleanup WATCHING row is removed (done, not open).
- audits/E2D-fresh-vm-2026-07-29.md §8 + REPORT-e2d.md: teardown recorded as
complete, with the cascade output and the appliance-scoping note.
Full ISO/PAIRING route on a nested PVE VM on demo-hp, after R-111 was fixed
earlier in the session. Bind -> running controller in 3m35s. The install fetched
the artifacts published an hour before and restored the golden baked 20 minutes
before, so the publish train is proven end to end on a real install.
C1 PROVEN: "felhom-host-install v1.22.0", "Day-0 provision SUCCESS", guest 9201
running, bootstrap unit wrote its done-flag and self-disabled. This retires E-2's
"installer-logic-tested, not install-tested".
C2 PROVEN: both DEGRADED warning lines verbatim, backup.local_backup_target=local,
no felhom-backup storage created, and the install did not abort.
C3/C4 PARTIAL and C5 FAILED — three findings, none fixed:
R-112 (P1): E-2's degraded banner and offer have NO UI CONSUMER. The endpoint
returns byte-exact copy; grep 'backup-target' across every html/js/css is 0 hits
and no page handler injects the state. Templates fetch 18 distinct /api/storage/*
endpoints; these two are the only ones with zero references. v0.185.1 fixed the
router mount and stopped one layer short of the render. Fifth instance of
seam-built-but-never-wired.
R-113 (P1): the drive-absent gate CANNOT FIRE on device loss. planDriveGates
reads presence from BoundUnderParent = "is this path in the guest's mountinfo".
The raw mount is a device-bound systemd unit and dies with the device; the
agent's own bind is not device-bound and outlives it, so the gate sees "present"
forever. Live: agent reported the drive absent every 20s for 4.5 minutes, the
controller logged 0 [gate] lines, the hub received zero events -- neither
backup_target_absent nor the generic storage_disconnected. Sixth instance of the
class: E-2b wired the seam to a condition that cannot occur.
R-114: on target-drive loss the message claims the backup is on the system disk
(false) and offers the drive that just vanished. Invisible only because of R-112,
so it must be fixed BEFORE R-112 is wired.
Also filed as a second instance under R-110 rather than a new ID: host-install
fetches nine files from raw/branch/main and the hub vouches a sha for one;
E-2a's wrapper is installed 0755 to /usr/local/sbin, root-fenced in sudoers,
validated only by bash -n.
C4 is fully proven at API level: decline path (registration confers no role),
restart_required:true, agent did NOT self-restart (in-flight check performed and
recorded first), E-2a wrapper created the storage at the drive's own mountpoint,
and healthy renders nothing.
Teardown: VM destroyed, scratch storage removed, pvesm status after == before
(local-lvm 38.77%), guest 9201 and drill-r50 untouched. Hub records for e2d-fresh
remain -- delete correctly refused at four gates, finally "host is ONLINE";
deletable once it ages to DOWN. Command recorded in OPEN-ITEMS.md.
capability-map NOT touched: the customer-facing legs are broken rather than
proven, and the map has no E-2 rows at all.
Found and fixed the same day. The channel was 17 agent releases stale — a box
installed today would have received agent 0.96.0 and controller 0.161.0.
- agent 0.113.0 built from the clean tree @ 58b598b and published via
scripts/publish-agent.sh; sha 5f3247f756cb658e…, round-trip GET verified.
- golden 0.185.1 baked on the nested drill VM embedding controller 0.185.1;
sha dba00f3e845c415e…. Bake clean: Result=success, overlay2, all 3 mounts
included (rootfs+mp0+mp1), 0 FATAL/exclusions, HTTP 201, token-leak grep 0.
GL-1 teardown: guest 9100 purged, secrets shredded, drill disk restored to
the virgin snapshot. Log saved to drill/bake-0.185.1.log.
- Hub Day-0 manifest: agent and golden moved TOGETHER in one POST so the
manifest never vouched a new agent against an old golden. min_agent
0.93.0 -> 0.113.0, which is what controller v0.185.0 declares. Zero fleet
impact, verified: all three enrolled hosts already run agent 0.113.0.
wrapper_sha256 preserved verbatim (re-checked, no drift).
- The global controller floor was deliberately NOT raised: the golden now
bakes 0.185.1, so a fresh box needs no self-update.
This unblocks E-2d C3/C4/C5, which the Phase 0 gate had blocked.
No VM created, no install run, no box touched. The run stopped at the Phase 0
gate per runbook §3, before provisioning.
felhom-host-install.sh does not install what is on main. resolve_artifacts()
(:423-436) reads the hub-vouched manifest and fetches Gitea GENERIC PACKAGES
(agent :1945, golden :2573). Gitea's newest are agent 0.96.0 and golden 0.161.0;
the hub manifest selects exactly those; the global floor v0.156.0 is below the
golden's 0.161.0 so nothing self-updates. A fresh box therefore lands on agent
0.96.0 + controller 0.161.0 against main's 0.113.0 / 0.185.1. Agent 0.113.0
reached both demo boxes by direct deploy and is not in the channel at all.
Claim impact, each pinned to its introducing commit:
- C1 (real rc=0 1.22.0 install) and C2 (Case B natural) — ACHIEVABLE, not run;
both are installer-side and host-install is served at 1.22.0.
- C3 — BLOCKED: banner + GET /api/storage/backup-target are controller v0.185.1
(cdaeb36), copy v0.185.0 (3f7cf2a). Unblocks cheaply by raising the hub floor
to >=0.185.0; measured fleet impact nil (both demo boxes already 0.185.1).
- C4 — BLOCKED: needs controller v0.185.1 + agent v0.113.0 (58b598b).
- C5 — BLOCKED: needs controller v0.184.0 (c1a63de) + agent v0.112.0.
Filed R-111 (P1): 17 unpublished agent releases (v0.97.0-v0.113.0) strand the
entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT, so a new customer's
box installs without them. Mirror of R-110, not a duplicate.
- audits/E2D-fresh-vm-2026-07-29.md — all four Phase 0 answers recorded so a
resumed run does not re-derive them (cadence 30s; hot-detach available; ISO
present; local-lvm fence re-measured at 38.77%, unchanged).
- OPEN-ITEMS.md — R-111 opened; E-2d re-stated, NOT closed.
- ROADMAP.md — R-111 under P1.
- capability map NOT touched: nothing was proven live.
The §5.1a operator STOP is retired — HUB_PW is in ~/.config/credentials and hub
auth was verified, so CC can bind on a resumed run.
Closes the record-hygiene rider. Part 3 of the spec (documenting a
ROADMAP/OPEN-ITEMS state convention) is deliberately NOT done — its stated
evidence is false; see REPORT-record-correction-2026-07-29.md.
- CONTEXT.md:540 — "scripts/hub_confirm_gate.py enforces" was present tense
about a gate invoked by nothing. Now says it asserts but is not enforced
(R-29). Third instance of the class after :564 and configs.go:27.
- REUSE.md:62 — same claim, "enforces zero". The RULE stays (never native
confirm()/prompt() is correct guidance and this is a reuse-reference row);
only the enforcement claim changes, and it now says the rule holds only as
long as you keep it.
- OPEN-ITEMS.md:4 — root REPORT.md is the overwritten per-session file;
REPORT-<topic>.md is the non-clobbering sibling form (CLAUDE.md:82-87), of
which 14 exist. The prohibition on durable content living only there stays.
- OPEN-ITEMS.md:55 — "Why the READY rows rank this way" promised a complete
ordering and listed 5 of ~15 open rows. Scoped to TOP, with a half-sentence
saying it is deliberately not a full ordering. No row added to the list.
hub/internal/web/configs.go:27 left alone (R-94 leg (b), needs a hub build).
No gate wired, run or fixed. Documentation only, no version bump, no CHANGELOG.
d4c07873 filed "hostinstall_gates.py is invoked by nothing" as a novel
observation. It is not novel — R-29 already names the class (green gates are
enforced nowhere; one sat RED for 16 releases while every REPORT said green),
and R-29 was missing from OPEN-ITEMS.md entirely, having never been carried
across the 2026-07-27 register rebuild. An open item about work not getting
done was absent from the page that decides what gets done.
Ruling on whether R-29 is the right home for a non-design-v2 gate: YES. Its
title says design-v2, but its own audit list already spans mount-safety,
secrets and dedup gates across four repos, and its part (b) — "the systemic
half is the real item" — is about the enforcement mechanism, which is
gate-agnostic. hub_confirm_gate.py is already on its list and sits in the same
scripts/ directory. No new ID minted; R-29's own text forbids it, and this is
the third re-raise it has absorbed.
- OPEN-ITEMS.md: open R-29 (READY, S(a)/M(b)), with the orphan evidence and
the two separable parts R-29 already defines.
- OPEN-ITEMS.md: R-94 leg (b) now points at R-29 as its class.
- ROADMAP.md:158: audit list extended with hostinstall_gates.py (RED today,
1.19.0 != 1.22.0) + hub_confirm_gate.py verified orphan. Entry not rewritten.
- ROADMAP.md:147: cited a non-existent R-164 — it means controller v0.164.0.
- CONTEXT.md:564: asserted in the present tense that the version cross-check is
"gated by scripts/hostinstall_gates.py". It exists, is red, and runs nowhere.
- OPEN-ITEMS.md: READY #1/#3/#4 markers dropped — they duplicated ranked-list
positions and the gap was left by the row merged in d4c07873.
- OPEN-ITEMS.md: E-2d citation :322-341 widened to :322-343; the invocation it
describes is at :343, two lines outside the old range.
- backlog/README.md: two-line lead naming OPEN-ITEMS.md and ROADMAP.md.
- REPORT-record-correction-2026-07-29.md: the report CLAUDE.md:82-87 requires
for both commits. Root REPORT.md (E-2 increment 1) untouched.
No gate wired, fixed, run or deleted — that is R-29 part (b), its own task.
Documentation only. No version bump, no CHANGELOG entry, no box touched.
The 2026-07-29 R-94/E-2d finding was written from an unverified claim and was
false. `felhom-bootstrap.sh:96` fetches the installer from the WEBSITE, not the
hub; the website git-syncs /scripts/ from main on a 30s period; every install
since 1.22.0 hit main this morning already runs 1.22.0. Confirmed by live fetch.
- OPEN-ITEMS.md: merge the two duplicate R-94 rows into one, retract the false
framing, re-scope to what it actually is (a drifting hand-synced constant plus
two pieces of dead safety equipment), unblock it from E-2d.
- OPEN-ITEMS.md: de-rank R-94 in the ranked list — the "high-consequence" reason
was the false claim in its most load-bearing form.
- OPEN-ITEMS.md: E-2d — the ISO is the STRONGER proof route, not an obstacle.
Phase 0 question answered at source: PAIRING falls through to run_direct in
the same invocation (:495-499), so it reaches the identical installer call.
- ROADMAP.md:149: same retraction; the original diagnosis (a hand-synced
constant in a second repo drifts every time the first ships) survives.
- ROADMAP.md + OPEN-ITEMS.md: new R-110 — main is the installer's publish
channel and there is no staging, tag, pinned path or rollback, for the one
artifact that runs as root on a virgin box. Operator ruling, not a defect.
- day0-install.md C.1: one sentence recording the same about the fetch URL.
Documentation only. No version bump, no CHANGELOG entry, no code, no box touched.
Space checked on the t740 -- NOT a blocker, with one constraint: the VM disk must
not go on local-lvm. That thin pool is over-subscribed (144G allocated against a
54G pool) on a box running a live customer guest, and a full thin pool corrupts
every guest on it. local has 23.7G on pve-root. Use /mnt/nvme-1tb (888G free).
Confirmed the ISO does NOT bake felhom-host-install.sh -- it ships
felhom-bootstrap.sh, which fetches the installer FROM THE HUB. Since the hub
serves 1.19.0, a fresh ISO install today would run the pre-E-2 installer and
exercise neither Case A nor Case B. So R-94 must be bumped only AFTER a real
1.22.0 run, not before -- which is the ordering already decided.
drill-r50 stays blocked and was restored to its r50pre state: the agent upgrade,
the added disk and the moved backup target from this session are all reverted.
Live: hub 0.81.0, agent 0.113.0, controller 0.185.1 on both demo boxes;
host-install 1.22.0 (script; no reinstall performed).
E-2a wrapper proven live as root on demo-hp: F-1 subdirectory refused, F-2
unmounted path refused, root device refused, idempotent re-apply is a no-op,
repointing refused -- 0 stray storages. The agent PVE role was NOT widened.
Scenario E proven live on BOTH boxes: healthy renders nothing, no message key.
Records three defects I introduced and caught: unreachable routes (mounted
outside /api/storage/, caught by the first live call), a hollow test exposed by
its own red-proof, and another gofmt-realignment no-op.
Not live-proven: the degraded banner and offer acceptance (both boxes healthy),
backup_target_absent end-to-end, Case A/B on a real install, drive-loss recovery.
hub 0.81.0, agent 0.112.0, controller 0.184.1 live on BOTH demo boxes.
E-2c: eject/decommission of the backup-target drive refused 409 on both boxes,
drives unmoved. E-2b: the never-called disconnect seam is wired, with the target
case raising the specific backup_target_absent.
Records the keying bug caught before deploy (a.Path is the GUEST path, so the
target branch was unreachable -- 0.184.0 superseded, never deployed) and states
plainly that backup_target_absent is NOT proven end-to-end live: proving it needs
a live enrolled drive to go absent.
Parts 2/3/4 and E-2a remain open; Peti risk stays parked.
E-2 is partially shipped (hub v0.81.0 + controller Part 1). Filing the rest so a
foundation with no UI cannot quietly become a sixth seam-built-but-never-wired.
E-2 remaining: installer Case A/B, the offer + agent-side move, the degraded
banner, the controller half of the signal, red-proofs E/F, live validation.
Phase 0 INVERTED the emphasis: the installer has no drive-enrollment step,
so the common case at install is system-drive-only and Part 3 (drive added
later) is the PRIMARY path, not Case A.
E-2a the move needs a root-fenced wrapper -- the agent holds neither
Datastore.Allocate at /storage nor Permissions.Modify, and its sudoers has
no pvesm and no pveum. Use the guarded-wrapper pattern; do NOT widen the
agent's PVE role.
E-2b NotifyStorageDisconnected/Reconnected are defined and called NOWHERE, so a
drive going absent emits no event at all. Hub side is already plumbed, so
wiring needs no hub change.
E-2c E-1 put the whole-guest backups on a drive POST /disks/eject will eject
(RoleForStorage returns user-data for a local-dir on a non-system device).
Guard the eject specifically -- reclassifying the drive RoleBackup would
block legitimate ejects, since it is also the enrolled user-data drive.
PETI peti-felhom deliberately NOT migrated; drive failure there is offsite-only
recovery. Accepted until the operator's reinstall; re-evaluate if that
slips past ~2026-09-01.
Supervised operational run. No code, no version bump, nothing deleted.
Primary backup tier on both demo boxes moved from `local` (a dir storage on
/var/lib/vz -- the SAME physical device as the guest) to `felhom-backup`, a dir
storage on each box's secondary drive:
demo-hp /mnt/nvme-1tb uuid:91d2dc2d-... archive 2,256,044,492 B
demo-felhom /mnt/hdd_1 uuid:47a3361a-... archive 5,957,878,962 B
Both proven end to end via the real UI path: archive lands on the secondary
drive (df delta matches the archive byte-for-byte), restore-test auto-selects it
and passes with mount_parity: ok, and freshness survives an agent restart with
an empty in-memory store -- so the age can only have come from the new storage.
Phase 0: the target is CONFIGURATION, not converged (the sole writer of
agent.json touches only escrow.pbs_storage_id and preserves unknown keys), so
the runbook's STOP did not fire. No consumer hardcodes "local" on the backup path.
Findings:
- F-1 the storage path must BE the mountpoint; a subdirectory fails exactMount
and the target reports disconnected permanently (observe.go:321)
- F-2 --is_mountpoint 1 is load-bearing; proven live, an unguarded storage on a
non-mounted path reports active with the ROOT filesystem's free space and
had already created dump/ on pve-root -- a silent retarget onto the very
device this change escapes
- F-3 FelhomAgentStore is granted per storage path; without it every backup
403s. felhom-host-install.sh must issue it for new installs
- R-109 (new) the DR recipe records no backup target, and each box now carries
two content=backup dir storages, one live and one frozen
- R-105 narrowed and TRACED: dr_recipe drives was [] fleet-wide because the
enrolled drives were never PVE storages, so isUserDataDrive never saw
them. Both boxes now populate drives; SMART on the backup drives too
Absent-drive behaviour today is fail-loudly with no silent retarget (PVE half
live-proven; agent half source-traced). That is NOT the intended fall-back-and-
alarm design -- filed as E-2 with the honest single-drive label.
Reported in full in the record: the agent was restarted with a felhom-pbs backup
in flight, producing a spurious tier failure. The backup had in fact succeeded
(PVE task OK, 6,264,034,053 B snapshot) and the spurious failure reached no
channel -- R-84 ground truth superseded it.
Outstanding: full drive-loss recovery (needs physical access) and the agent half
of the absent-drive behaviour.
Replaces the 2026-07-14 DRAFT entirely - it was verified against controller v0.132.0,
51 versions stale, while being cited as authoritative. Now verified against controller
0.183.0 / agent 0.110.0 / hub 0.80.0 / catalog 4252121. NOT ratified.
Every statement is labelled [DESIGN] (a decision from the architecture discussion) or
[FACT] (file:line, live output, or a citation to _recovery-inventory-2026-07-28.md).
Phase 0 verified D5's premise and it FAILED. Ten read surfaces were checked; nine are
clean (SMB under both namespace shapes, FileBrowser for local drives, .fab import and
download, /api/debug/*, all three ServeFile sites, registering the backup dir as a
drive). One is not: FileBrowser binds a NETWORK share at its ROOT while local drives
are userdata-scoped, and no IsNetwork() filter guards the deploy dropdown, the per-app
migrate targets, or handleStorageMigrateApp. So an app namespace - and under D5 its
plaintext secrets - can land somewhere a browsing surface serves with download:true.
Not a leak today (the unit's app.yaml is secret-stripped); it becomes one under D5.
D5 is therefore written as a BLOCKED target, not as settled. -> R-108
Core artifact is the failure -> recovery matrix (15 rows). RTO cells carry only
measured durations; 12 are blank and each blank is listed explicitly as a finding.
RPO cells carry the configured cadence that bounds RPO, labelled as such - no RPO has
ever been measured from an incident and none is invented.
The 7/53 vs 9/43/1 Tier-2 coverage counts are both recorded, unresolved; neither adopted.
Coupling (S-1): capability map section C now cites the matrix instead of restating
routes; ROADMAP gains R-102..R-108; OPEN-ITEMS gains only the in-flight R-108 plus
C9-F1b/F3/F4 -> R-n cross-references; CONTEXT records D1-D6 as standing ruling S-3.
No code, no version bump, no deploy, no box touched.
Phase 0 sized C9-F1 properly before anything was designed: 43 of the 53 catalog apps have NO
subtree the Tier-2 restore can read (not 2), 9 are covered only for their file legs and never
their database or volumes, 1 is stateless. The asymmetry is Tier-2's alone — Tier-1 and offsite
both restore the unit and replay volume dumps, so BookStack always had a working restore and
only this button lied.
Shipped: the restore refuses BEFORE stopping the app and names the action that does work; a run
that proceeds claims only what it EXAMINED and discloses that the database and volumes are not
covered. C9-F2 alarms after a 5-minute sustained-restarting threshold, set above the 120s deploy
timeout, Mealie's 60s start_period and R-97b's 180s grace; StateRestarting is deliberately NOT
added to IsDownState.
Live: silent through ten 30s samples then app_start_failed at 5m25s, heartbeat now reads
"1 currently down" where Campaign 9 recorded 0; a real deploy stayed silent; bookstack refused
with its uptime unbroken; paperless re-restored 43/43 byte-identical, 16/16 docs clean.
Filed, not fixed: C9-F1b (route to the Tier-1 restore — its own task because it puts a
destructive operation behind a non-destructive button) and C9-F4 (nothing reads the Tier-2
copy's recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable
by any customer action — potentially larger than C9-F1).
Phase A is the headline and it passed on live hardware, through the real endpoints the UI
posts to: a customer who deletes files — or their entire app data directory — gets everything
back byte-identical, and the app works afterwards (paperless served the restored bytes over
its own API at the exact pre-deletion sha256). A1's two non-destruction promises both hold.
Three defects, recorded not fixed:
C9-F1 (HIGH) the Tier-2 restore button is offered for apps it can never restore, takes a
real outage, and reports "nothing was missing" — indistinguishable from a
genuine result, while 156 MB of that app's data sits unread in the same copy.
C9-F2 (HIGH) an app in a crash loop never alarms on any channel; StateRestarting is in no
down-set, so F-OBS's own heartbeat printed "0 currently down" for 9 minutes.
C9-F3 (MEDIUM) an interrupted offsite run leaves a lock the self-heal cannot reach; the tier
is dead until a human unlocks and the operator is told "unknown reason".
This answers Phase C item 8.
Two candidates were deliberately NOT filed: a recovery-unit poisoning the catalog sync healed
in ~3 min, and a snapshot_id that is documented as logging-only. Reporting either would have
been reporting an artifact.
Stopped at the end of Phase B (plus D10), then full recovery — both boxes healthy, real
cadences, offsite tier proven working again, no leaked scratch guests, peti untouched.
D11's approved staleAfter compression turned out not to be a knob; reported, not worked around.
F-LEAK's first fix (pool adoption) was refuted live and is recorded as such. R-100 is
F-CRIT-2's defect class on the hub for the restic tier: isStale reads only LastRun, which
the controller writes unconditionally on failure. Investigated, not fixed.
Both marked SHIPPED + PROVEN-LIVE in OPEN-ITEMS and the campaign doc. All three
of Campaign 8's alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1).
Adds the standing rule earned by this arc to the versioned workspace CLAUDE.md:
a comment asserting an invariant needs a test pinning it, or it is a wish — with
all six shipped-false-guarantee instances catalogued, and the corollary that a
test should assert the CONSEQUENCE (does the alarm fire?) not the MECHANISM
(does suppression expire?).
F-CRIT-2 marked SHIPPED + PROVEN-LIVE: NewestArchiveTime now counts only
plausibly-complete entries (measured 1 MiB floor), verified by replaying
campaign fault 2 on demo-hp in both directions.
Files two follow-ups:
- R-99: prune never REMOVES phantoms. It does NOT count them toward keep-last
(dry-run kept 2 real + the phantom), so there is no retention/data-loss bug —
but they accumulate one per aborted upload.
- F-CRIT-1: filed READY-HIGHEST. Campaign 8's other HIGH finding, untouched by
this fix.
Seam sweep: TieredBackend was the FIRST, not the only one. BackupArchiveLister
has the identical silent-degrade shape and a worse blast radius (it degrades to
the pre-R-84 in-memory-only behaviour), and no compile-time witness existed in
production code anywhere in either repo. No defect found, so no version bump and
no deploy — the witnesses are guards, proven by breaking a signature and watching
go build fail where it previously passed.
Live outage: age_state=unknown captured on real hardware for the first time, with
demo-felhom's local tier genuinely due throughout — the controller deferred and
zero app stacks were stopped. The R-88 breaker did NOT arm and no
whole_guest_backup_failed travelled, because felhom-pbs was not due; recorded as
conditions-did-not-arise rather than claimed as coverage.
Post-boot: the volume changed device name (sdb->sda) across the reboot and the
mount survived only because fstab uses by-id. That was never tested before.
R-88b closed (agent v0.105.0 + controller v0.178.0) — age_state gives 'unknown'
its own representation, with empty meaning legacy rather than unknown so the
first-backup valve keeps working on un-upgraded boxes.
R-97 note updated: hub v0.79.0 (R-97c) replaced a FALSE operator-only comment
with a real register — the comment claimed a guarantee the code did not provide.
Standing rule 4 (R-96): a recommendation that is not followed gets one line
saying why. Added to the live CLAUDE.md and this versioned copy — the live file
is not in a git repo, so committing to it alone would leave the rule as durable
as the chat it came from.
READY re-ranked: R-95 now leads.
R-97 collapsed to its shipped one-liner in ROADMAP and closed in OPEN-ITEMS.
PROMPT-TEMPLATE N.5 now names FOUR coupled artifacts instead of two: the
capability map, ROADMAP, the owning architecture doc (ruled as S-1 in CONTEXT.md
but never reflected in the template CC actually reads, so it bound nobody), and
OPEN-ITEMS.md. Tasks must now report which register rows they opened, closed or
re-ranked.
Ops: R-90 swap done (interim; CX33 still blocked), R-95 mitigation armed but zero
snapshots taken so it moves to WATCHING rather than closed, R-91 gate still not
satisfied. CONTEXT.md datastore path corrected to /mnt/pbs-datastore.
R-88 split: Part 1 (the failure breaker) SHIPPED in controller v0.176.0 and live
on both boxes; Part 2 (unknown != never) stays OPEN and is agent-side.
Phase 0 established the root cause at source: newestArchiveOn's (time.Time, bool)
signature cannot represent 'unknown', so a storage read ERROR collapses into a
positive 'no successful backup recorded yet'. The errored and genuine-never paths
are byte-identical on the wire, which is why Part 2 cannot be done controller-side.
R-97: the whole-guest backup tier has no failure signal to the hub at all —
internal/quiesce never imports internal/notify, so three failed backups and three
app-stack outages produced zero backup_failed events. Its only trace was a
customer-tier Hungarian app_start_failed for an app the backup itself had stopped.
Open state was spread across ROADMAP, CONTEXT.md, four audits/, three runbooks,
per-session REPORT.md files and a chat log. This is the one page to read first:
every row has a state (BLOCKED/READY/WAITING-ON-OPERATOR/WATCHING) and an owner,
and the READY rows are ranked with reasoning.
R-88 is the recommended next task — quiesce's nil-age fail-open stops every app
stack every 5 minutes with no backoff and bypasses the maintenance window, and
its trigger (a PBS read failure) is live given ep0's demonstrated OOM.
CONTEXT.md now records that OPEN-ITEMS.md is authoritative and that REPORT.md is
overwritten per session.
admin