The credential existed and was not reachable when it was wanted. Every box has
had a strong random root@pam password since TASK G1, vaulted in the hub at day 0
and used for real during the sshd incident — but the only way to read it back was
a hand-written curl carrying the global operator key, a secret kept out-of-band.
In practice the PVE web console on a demo box felt locked.
The host page grows a Console access card: presence + username + set_at by
default, Reveal fetches the plaintext on demand for 60 s with a Copy button.
Masking clears the JS variable, and also fires on a second click and on
visibilitychange. A host with nothing vaulted says so, and says why.
The secret is NEVER rendered into the page, and that constraint shapes the
change. The render path uses a new store.GetHostRecoveryMeta whose struct and
SELECT both omit the secret column, so it is structurally incapable of carrying
one. The plaintext crosses the wire only in the response to POST
/hosts/{id}/reveal-recovery-credential (Cache-Control: no-store, CSRF-gated at
the ServeHTTP level; POST precisely so that gate applies and so no secret is
retrievable by URL alone). Deliberately NOT the customer page's data-secret
widget, which embeds the plaintext on every load.
A delivered reveal writes one recovery_credential_revealed event on the host's
customer timeline (info, source hub, Hungarian) via SaveEvent alone — no
dispatcher, nobody emailed, the log_tail_requested shape. Two reveals write two
events: the register records accesses, not states. A 404 is not an access. An
unbound host reveals fine and writes no event; the [INFO] hub line, carrying the
username and a length only, is then the record.
The global-key API path is untouched by design — it is the route for when the
hub UI itself is broken, and coupling it to the session layer would delete the
independence that makes it a fallback.
Recorded as a real trade: the hub session password alone now unlocks console root
fleet-wide, where retrieval previously also needed the global key. Accepted for a
single-operator, HU-geo-fenced hub that already stores these passwords in
plaintext at rest (CONTEXT.md ruling S-4). The plaintext-at-rest half is filed as
R-133 — every hub DB backup is a fleet-wide console-credential dump.
Tests 550 -> 559; four red-proofs (page leak, audit event, CSRF gate, route
order) each run, observed failing, and reverted. The route-order proof is a seam
test driving ServeHTTP: a handler-level test cannot see that defect, because the
handler is correct and simply never runs.
Live: hub 0.81.0, agent 0.113.0, controller 0.185.1 on both demo boxes;
host-install 1.22.0 (script; no reinstall performed).
E-2a wrapper proven live as root on demo-hp: F-1 subdirectory refused, F-2
unmounted path refused, root device refused, idempotent re-apply is a no-op,
repointing refused -- 0 stray storages. The agent PVE role was NOT widened.
Scenario E proven live on BOTH boxes: healthy renders nothing, no message key.
Records three defects I introduced and caught: unreachable routes (mounted
outside /api/storage/, caught by the first live call), a hollow test exposed by
its own red-proof, and another gofmt-realignment no-op.
Not live-proven: the degraded banner and offer acceptance (both boxes healthy),
backup_target_absent end-to-end, Case A/B on a real install, drive-loss recovery.
hub 0.81.0, agent 0.112.0, controller 0.184.1 live on BOTH demo boxes.
E-2c: eject/decommission of the backup-target drive refused 409 on both boxes,
drives unmoved. E-2b: the never-called disconnect seam is wired, with the target
case raising the specific backup_target_absent.
Records the keying bug caught before deploy (a.Path is the GUEST path, so the
target branch was unreachable -- 0.184.0 superseded, never deployed) and states
plainly that backup_target_absent is NOT proven end-to-end live: proving it needs
a live enrolled drive to go absent.
Parts 2/3/4 and E-2a remain open; Peti risk stays parked.
Supervised operational run. No code, no version bump, nothing deleted.
Primary backup tier on both demo boxes moved from `local` (a dir storage on
/var/lib/vz -- the SAME physical device as the guest) to `felhom-backup`, a dir
storage on each box's secondary drive:
demo-hp /mnt/nvme-1tb uuid:91d2dc2d-... archive 2,256,044,492 B
demo-felhom /mnt/hdd_1 uuid:47a3361a-... archive 5,957,878,962 B
Both proven end to end via the real UI path: archive lands on the secondary
drive (df delta matches the archive byte-for-byte), restore-test auto-selects it
and passes with mount_parity: ok, and freshness survives an agent restart with
an empty in-memory store -- so the age can only have come from the new storage.
Phase 0: the target is CONFIGURATION, not converged (the sole writer of
agent.json touches only escrow.pbs_storage_id and preserves unknown keys), so
the runbook's STOP did not fire. No consumer hardcodes "local" on the backup path.
Findings:
- F-1 the storage path must BE the mountpoint; a subdirectory fails exactMount
and the target reports disconnected permanently (observe.go:321)
- F-2 --is_mountpoint 1 is load-bearing; proven live, an unguarded storage on a
non-mounted path reports active with the ROOT filesystem's free space and
had already created dump/ on pve-root -- a silent retarget onto the very
device this change escapes
- F-3 FelhomAgentStore is granted per storage path; without it every backup
403s. felhom-host-install.sh must issue it for new installs
- R-109 (new) the DR recipe records no backup target, and each box now carries
two content=backup dir storages, one live and one frozen
- R-105 narrowed and TRACED: dr_recipe drives was [] fleet-wide because the
enrolled drives were never PVE storages, so isUserDataDrive never saw
them. Both boxes now populate drives; SMART on the backup drives too
Absent-drive behaviour today is fail-loudly with no silent retarget (PVE half
live-proven; agent half source-traced). That is NOT the intended fall-back-and-
alarm design -- filed as E-2 with the honest single-drive label.
Reported in full in the record: the agent was restarted with a felhom-pbs backup
in flight, producing a spurious tier failure. The backup had in fact succeeded
(PVE task OK, 6,264,034,053 B snapshot) and the spurious failure reached no
channel -- R-84 ground truth superseded it.
Outstanding: full drive-loss recovery (needs physical access) and the agent half
of the absent-drive behaviour.
Phase 0 sized C9-F1 properly before anything was designed: 43 of the 53 catalog apps have NO
subtree the Tier-2 restore can read (not 2), 9 are covered only for their file legs and never
their database or volumes, 1 is stateless. The asymmetry is Tier-2's alone — Tier-1 and offsite
both restore the unit and replay volume dumps, so BookStack always had a working restore and
only this button lied.
Shipped: the restore refuses BEFORE stopping the app and names the action that does work; a run
that proceeds claims only what it EXAMINED and discloses that the database and volumes are not
covered. C9-F2 alarms after a 5-minute sustained-restarting threshold, set above the 120s deploy
timeout, Mealie's 60s start_period and R-97b's 180s grace; StateRestarting is deliberately NOT
added to IsDownState.
Live: silent through ten 30s samples then app_start_failed at 5m25s, heartbeat now reads
"1 currently down" where Campaign 9 recorded 0; a real deploy stayed silent; bookstack refused
with its uptime unbroken; paperless re-restored 43/43 byte-identical, 16/16 docs clean.
Filed, not fixed: C9-F1b (route to the Tier-1 restore — its own task because it puts a
destructive operation behind a non-destructive button) and C9-F4 (nothing reads the Tier-2
copy's recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable
by any customer action — potentially larger than C9-F1).
Phase A is the headline and it passed on live hardware, through the real endpoints the UI
posts to: a customer who deletes files — or their entire app data directory — gets everything
back byte-identical, and the app works afterwards (paperless served the restored bytes over
its own API at the exact pre-deletion sha256). A1's two non-destruction promises both hold.
Three defects, recorded not fixed:
C9-F1 (HIGH) the Tier-2 restore button is offered for apps it can never restore, takes a
real outage, and reports "nothing was missing" — indistinguishable from a
genuine result, while 156 MB of that app's data sits unread in the same copy.
C9-F2 (HIGH) an app in a crash loop never alarms on any channel; StateRestarting is in no
down-set, so F-OBS's own heartbeat printed "0 currently down" for 9 minutes.
C9-F3 (MEDIUM) an interrupted offsite run leaves a lock the self-heal cannot reach; the tier
is dead until a human unlocks and the operator is told "unknown reason".
This answers Phase C item 8.
Two candidates were deliberately NOT filed: a recovery-unit poisoning the catalog sync healed
in ~3 min, and a snapshot_id that is documented as logging-only. Reporting either would have
been reporting an artifact.
Stopped at the end of Phase B (plus D10), then full recovery — both boxes healthy, real
cadences, offsite tier proven working again, no leaked scratch guests, peti untouched.
D11's approved staleAfter compression turned out not to be a knob; reported, not worked around.
Seam sweep: TieredBackend was the FIRST, not the only one. BackupArchiveLister
has the identical silent-degrade shape and a worse blast radius (it degrades to
the pre-R-84 in-memory-only behaviour), and no compile-time witness existed in
production code anywhere in either repo. No defect found, so no version bump and
no deploy — the witnesses are guards, proven by breaking a signature and watching
go build fail where it previously passed.
Live outage: age_state=unknown captured on real hardware for the first time, with
demo-felhom's local tier genuinely due throughout — the controller deferred and
zero app stacks were stopped. The R-88 breaker did NOT arm and no
whole_guest_backup_failed travelled, because felhom-pbs was not due; recorded as
conditions-did-not-arise rather than claimed as coverage.
Post-boot: the volume changed device name (sdb->sda) across the reboot and the
mount survived only because fstab uses by-id. That was never tested before.
Corrects two wrong severity readings with evidence from the box and the code.
The PBS outage was ~15 min (07:00-07:18 UTC), caused by a global OOM at 06:58:12:
proxmox-backup-proxy peaked at 3.2G on a 3.8G box and a concurrent 1.9G rsync
tipped it over. Root SSH to that box works from DooPlex via the public IP, not
from felhom-pve via the tunnel IP — the documented path I failed to try first.
R-88: internal/quiesce has NO failure limiter, backoff or breaker; the loop
stopped after three cycles only because PBS recovered. Verified additionally that
scheduledRunAllowed (quiesce.go:476-478) returns true whenever lastAgeSecs is nil,
so the same missing value that makes every poll due also bypasses the time-of-day
gate — the cycles ran outside the [04:30,08:30) window. Fixing the due-verdict
without fixing the nil-age bypass would leave the hole open.
Recorded after the session report was written. The offsite PBS service stopped
listening on 8007 five minutes after this session's 14.46 GB restore-test read
from it; the box is up and the tunnel is healthy, but no SSH key to it exists so
the cause is unestablished — the restore load is a plausible mechanism on a cx23
and is recorded as correlation, not cause.
The agent restart then exposed R-88: an unreachable target reads as 'no backup
exists', so the offsite tier is perpetually due and the controller runs a full
quiesce cycle every ~5 min. Operator ruling: leave it running, it self-heals when
PBS returns and masking it would hide the fault.
The unattended offsite restore-test on demo-felhom passed: 14.46 GB archive,
duration_s=635.07 (10m35s), then it rotated to the local tier. Persisted state
confirms the credit: {"felhom-pbs": "2026-07-27T06:14:42Z"}.
CORRECTION: I estimated ~2 hours for this restore. It took 10m35s. I derived
the estimate from a download rate measured during the FAILED attempt, which was
running under contention; the real link does ~1.4 GB/min. I then used that wrong
figure to raise a design concern — that the heavy-op gate would block backups
for hours on this box — which at 10 minutes largely evaporates. An estimate
extrapolated from a degraded measurement is not a measurement.
The SPEC's closing risk note is corrected in place, with the original left
visible for the lesson.
R-86 (NEXT, operator ruling 2026-07-27): backup-ALIGNED restore-test scheduling
— test a tier ~1 day after ITS OWN backup. R-85 schedules on a free-running
interval, which cannot express 'the day after the PBS backup': any fixed offset
drifts, so alignment would be luck. Shape: trigger from the tier's own last
successful backup rather than a clock. Interim in force: 302400s (3.5d), which
lands each tier ~weekly — the cadence half of the ruling, not the alignment half.
R-87: the restic app-data offsite tier is NEVER restore-tested. R-85 covers
whole-guest vzdump tiers only; the agent has no restic surface. That is arguably
the tier that matters most — the only one that survives losing the box AND
carries the customer's app data, since the whole-guest snapshot excludes the
bind-mounted drives. Exactly the state PBS was in before R-85.
REPORT.md: the full R-80 -> R-85 arc, including a section on the seven mistakes
I made and the two recurring shapes behind them (inferring behaviour from an
artifact instead of the code that consumes it; reading a result without its exit
code). Records demo-felhom's restore-test as IN FLIGHT at close, with the manual
recovery step if the deferred restart watcher does not complete.
Hub gate green (17 packages, rc=0).
Restore round-trip on demo-hp: pass=true, verified=boot+running,
mount_parity=ok, source_tier=pbs (the v0.100.0 fix — the earlier attempt said
'local' and died at 600s), 4m5s restore+boot+verify+teardown, clean teardown
with no 403 and no leak. That last point confirms 06's reading that the
teardown 403 was a phantom, and corrects my earlier framing of it as a standing
privilege gap.
Multi-tier quiesce driven through the REAL UI endpoint (authed+CSRF):
exactly ONE stop/start pair with BOTH backups inside it, local-first/PBS-last,
app quiesced through the non-last tier, early resume on the last tier's
snapshot. Total downtime 1m27s for both tiers; app healthy after.
Capability map row upgraded IMPLEMENTED -> PROVEN-LIVE, kept distinct from the
DR-tier row above which proves ACTIVATION not ARRIVAL. Remaining gaps recorded:
the SCHEDULED restore-test still only selects the primary tier (manual path
proven, unattended not), and the hub infers cadence from storage type.
Slice D.1 — host-install 1.20.0: a FRESH box defaults to local-daily +
offsite-weekly (felhom-pbs, 604800s, keep_last=2). setdefault semantics proven
both ways: fresh gets the tier, an UPGRADE preserves the existing backup block
verbatim — so an in-place upgrade can never silently start writing to an
offsite datastore. Existing boxes are migrated explicitly.
Slice E:
- 07-backup-architecture.md: honest status header per CONTEXT ruling S-2, with
an explicit STALE-outside-the-PBS-tier verdict (the controller tiers were last
verified 41 controller versions ago). The PBS row claimed 'PBS on DooPlex'
(the retired spike store) with no cadence; it now names felhom-pbs ->
felhom-offsite on ep0 over wg-felhom, weekly, keep_last=2. NOT marked
ratified — that is Viktor's review of the section 10 list. Discharges R-83.
- 06-offsite-connectivity.md: the target-split remaining-work note collapsed
(shipped), and records HOW S4.1's tier-aware timeout silently regressed — the
mechanism was never removed, its INPUT changed when local_backup_target was
retargeted to 'local'. Also notes S4.1 already diagnosed the teardown 403 as a
phantom (a timeout consequence, not an ACL gap).
- capability map: new row for recurring offsite backups actually LANDING, as
distinct from the existing row proving ACTIVATION. IMPLEMENTED, not
PROVEN-LIVE — the restore round-trip has not completed under the fixed code.
- ROADMAP: R-82 SHIPPED with its remaining gate named, R-83 DISCHARGED, R-84
left open.
- CONTEXT + REPORT: the arc, including the mid-arc correction I had to make.
- B2 demo-hp + B3 demo-felhom migrated to the island (agent 0.96.0), apps
served throughout (0 container restarts), island /storage 200, LAN DNS pinned
to the LAN IP, hub reports 0.96.0. No rollback.
- capability-map 'site/network change' row PARTIAL -> PROVEN-LIVE
- ROADMAP R-50 -> SHIPPED (fleet-migrated); add R-74 (island on Peti's cluster)
- nodes.md: both boxes island-bound, agent 0.96.0
- A4: scratch guest provisioned from golden on the island-configured drill came
up with net1 automatically (zero manual edits) — the v0.96.0 provisioning path
proven live. Spike method caveat CLOSED.
- REPORT: Phase A done + vouched 0.96.0; B0/B1/A4 done; B2/B3 await operator go.
Provisioned nested-PVE drill 'drill-r50' (qm300 on demo-hp) via the v1.25.0
nested-vm ISO through the real day-0, then ran the R-50 empirical spike:
- vmbr9 portless island bridge + guest island NIC hot-add (LAN undisturbed)
- F1 replay money shot: LAN move survives on the island; LAN-literal bind
reproduces the 2026-07-20 daemon-exit bug verbatim
- dnsmasq trap confirmed live + lan_resolver.host_ip fix proven
- pin address-independent (leaf SHA-256 unchanged, HTTP 200 over island)
- survival matrix: agent/guest/host-cold-reboot all return on the island
Docs: SPIKE verdict BLOCKED->GO, ROADMAP R-50 SPIKED->GO, nodes.md drill VM,
REPORT overwrite.
Virgin-ISO nested drill closed the train: dead-NIC install baked the
fallback (incl. the dead default gateway), the R-59 screen painted
(capture committed beside the spike doc), the cable move healed +
registered at the hub in 23s unaided, and the build's rootpw file
matched the installed box's shadow hash. R-59 SHIPPED with the recorded
deviation (first-boot gate; installer-initrd abort out of scope by
operator ack). R-60 SHIPPED (spike + drill cited; F-P9 route-flush fix
included). R-61 slice 1 SHIPPED. New R-62 row (hub delete-dialog
cosmetics, XS). Capability map: new PROVEN-LIVE row (nested != metal,
said so). Cleanup verified: felhom-pve interfaces byte-identical,
bridge/VMs/ISO removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Per the 2026-07-21 refresh brief: R-39 interim blocks (B4/E1) and the R-36
manual-Save block (C4) deleted — both shipped and proven live; freemail.hu
gate proven (R-4 COMPLETE); golden/floor-lift note now cites two shapes
(rehearsal + virgin HP t740 day-0 lift 0.153.0->0.156.0); A3 loader table
per operations/nodes.md (N100=mkimage/SB-off per record, HP t740=shim/SB
ENABLED); B2 multi-NIC cabled-port gotcha (R-59/R-60 pending); new A5 gate
(agent >=0.93.0 deployed box-side before the first escrow ceremony); D
offboarding pointer to §G (R-25b). DRAFT status and the C7 graduation gate
unchanged. ROADMAP R-25b pointer follows the rename.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Found validating v0.69.0 against the live hub. demo-vm-felhom was deleted
on 07-18 and was still on the Customers list AND still raising offsite_stale
(10 events, latest 07-21 17:34, operator email at 19:34) — because
GetCustomers() is report-derived and no lifecycle tier ever deleted a report.
New leg 3 (residue), before the record purge: reports, app_telemetry,
app_log_tails, log_tail_requests, customer_notifications, plus the
credential-bearing appliance_registrations and selfbind_tokens. Audit
(events, notification_log) and F-14 provenance still survive.
Ghost customers are now deletable: 404 means "nothing here", not "no config
row". With no config row the offsite descriptor is unknowable, so the Hetzner
and descriptor legs record skipped_no_config rather than a bare "skipped".
Two more red-proofs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
POST /configs/{id}/delete now runs hosts -> RESET -> purge behind three
acknowledgements, a typed customer-id, a stale-preview check and the
ONLINE-host refusal (every gate before any write, so a refusal has zero
side effects). The shallow handleConfigDelete is gone.
Two invariants are asserted, not just commented: ruling 3 is preserved by
construction (leg 2 never sees a host row) and retained escrow custody is
purged exactly once, in leg 3 (leg 2 runs with purgeEscrow=false).
handleCustomerReset's committed half was extracted as commitCustomerReset;
the standalone RESET path is byte-identical to v0.68.1 and its suite is
untouched. Five red-proofs run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
Dead primary: degraded in 13 s, exactly one app_start_failed, banner rendered and
self-cleared. Boot orphan: recovered in one attempt with zero alerts. Dead dhclient:
detected in 57 s on process liveness while the lease was still live, healed 120 s after the
kill — the tunnel never dropped, so the outage was prevented rather than observed.
P1 answered as a by-product: bookstack StartedAt == the moment bootrecon StartStack
returned, so unless-stopped did NOT resurrect it. F5 hypothesis confirmed.
New R-55, surfaced by the leg designed to prove the opposite: the boot bind gate recreates
and STARTS every deployed drive-backed app unconditionally, so a customer Stop does not
survive a reboot for those apps. Predates R-52 and does not implicate it, but it narrows
R-52's practical scope and needs a ruling.
R-51's roadmap diagnosis is corrected at the source: aggregation returned StateRunning
("partial") for a running/stopped mix, so the stack read RUNNING and IsDownState was never
consulted about at all — the constraint that row protects was never in tension
with the fix.
New R-54 row closes the INCIDENT-guest-dhclient-killed-2026-07-20 §5 OPEN RISK, and records
the design fact that makes it work: liveness of the DHCP client is itself a probe, because
the damage is timed and the address outlives its cause by 1-2 hours. The static-guest leg is
deliberately deferred to R-50.
New capability-map row is IMPLEMENTED, not PROVEN-LIVE: one leg is live (the watchdog's
healthy cycle on felhom-pve), the three that matter are destructive and operator-present and
have not run.
PROMPT-TEMPLATE §10 gains the seam-discipline row, including that a strings.Contains source
assertion is NOT sufficient — a commented-out call still contains the string.
admin
Claude Code