Link 7's only production caller was a --selftest reading R from an env var. Link 8 did not
exist: that selftest writes the whole bundle JSON and its success message named
"tunnel_token + pbs_token" -- accurate when written, a misstatement since v0.77.0 sealed the
offsite repository password into the same bundle. It now names what THIS bundle carried and
what it did not.
POST /escrow/recover-offsite-password: the controller supplies R, the agent fetches this
host's own blob from the hub (self-scoped by the per-host key), unseals it, and returns ONLY
the offsite restic repository password plus its sha256. Not the tunnel token, not the PBS
token, not the WG key -- the controller is a trust tier down and needs none of them.
R: in memory for one call, cleared on every path, never on disk, never in argv, never logged,
never echoed. A test redirects TMPDIR and asserts the tree is EMPTY afterwards -- emptiness
rather than a content scan, because a content scan is defeated by a later call overwriting the
leaked file, which is how the first version of that test passed its own red-proof while R sat
on disk.
Three distinct outcomes: no blob (404), a bundle that opens but predates the field (409), a
code that does not open it (400, fail-closed at the KDF, nothing written).
The wiring is asserted by an AST walk from func main() to the Options field, not by grep.
BoundUnderParent reported a namespace that returned EIO on every read and write
as healthy, and the gate restarted the customer's apps onto it. Both existing
terms parse a mountinfo line and then test only fields[4], the mount POINT.
Field 3 — major:minor — sat in the same parsed slice and was discarded.
Measured on hardware: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb with `shutdown`,
bound_under_parent true, EIO both directions, and the controller taking its
Return branch and emailing backup_target_restored with no alarm on any channel.
BoundUnderParent gains a third term at both /disks construction sites. The new
bindLiveness reads /proc only and asks two questions: the bind must name the
same device as the raw mount, and the filesystem must not have aborted (ext4
`shutdown` or `emergency_ro`).
The second check is not optional. A device that fails WITHOUT disappearing gives
the identical all-signals-healthy state with the devnos EQUAL and the drive never
Disconnected, so the gate produces neither a Stop nor a Return and nothing is
emitted on any channel, indefinitely (R-117a). A devno-only fix would have passed
every payload test.
Three states, never a bool: {Unknown, Live, StaleDevice, Aborted}, read through
Usable(), where Unknown counts as PRESENT — reporting absent stops a working
customer's apps.
No new recovery path; the existing one was unblocked. AttachDrive's normalize leg
already did the repair and three call sites already invoked it, including the
controller's Return branch before it restarts apps. All three died on
`if n == 1 && GuestSeesMount(...)` returning early. Now: StaleDevice ⇒ re-bind
(repairs live, guest never restarts); Aborted ⇒ quiet no-op, because a re-bind
lands on the same dead superblock and this runs every 20s — an infinite silent
retry that masks the state; it surfaces via BoundUnderParent=false instead.
Ordering trap caught by a test: reading the abort flag before comparing devices
classifies the real return state as aborted (its stale bind carries `shutdown`
too) and refuses the repair while still reporting correctly. The abort flag is
read off the RAW mount in the stale case.
Tests 849 → 863, 29/29 packages green. 6 red-proofs, each verified to have
landed. A hollow test was caught during them: the aborted fixture first used a
/dev/mapper device, for which RoleForStorage derives role=system — a system row
has no GuestPath, never runs the conjunction, and reports false by default, so
the assertion passed vacuously and no mutation could fail it. Found because RP1
failed to fail.
BoundUnderParent, the one field the controller's drive-absent gate keys on,
reported only "is this path a mount target in the guest's mountinfo". The
drive's raw mount at /mnt/<name> is a systemd mount unit bound to its device and
dies with it, but the agent's own bind of <raw>/felhom-data under the shared
parent is an ordinary bind: nothing ties it to the device, so its mountinfo
entry OUTLIVES the device as a stale shell. Presence read that survivor as true,
planDriveGates never produced a Stop action, and nothing fired on any channel --
not backup_target_absent, not the generic storage_disconnected. Measured live in
E-2d: detached at 10:58:37Z, silent for 4.5 minutes while the agent itself
logged "enrolled drive absent by UUID" every 20s (felhom.eu
audits/E2D-fresh-vm-2026-07-29.md §5.2).
The fix: BoundUnderParent becomes a CONJUNCTION -- bound under the parent AND
the drive's raw host mount still mounted (devicePresent, new deviceCheck seam
defaulting to isHostMountpoint). Applied at BOTH /disks construction sites. The
union path matters more, not less: it hardcodes State:"attached", so the
raw-mount check is the only device truth that row carries, and it is exactly the
shape E-2d detached.
Why a conjunction and not a replacement: half 2 alone would regress boot
ordering, where the raw drive mounts early and the bind lands ~18s later; the
gate depends on that window reading ABSENT. The conjunction leaves that
byte-identical and closes only the case the gate could never see.
Unknown is never absent: devicePresent("") returns TRUE. A false absent stops a
working customer's apps -- the failure mode of this fix, not of the bug.
Controller UNCHANGED, no MinAgent bump. BoundUnderParent has exactly one
functional consumer (planDriveGates, intermediary.go:226); every other mention
in both repos is a comment or a test, and boot convergence deliberately moved
off it to pollLiveBinds/driveBindLive. The alternative -- a new DevicePresent
bool the controller ANDs in -- was rejected as dangerous: a bool absent from an
older agent's JSON decodes to false, so every drive on a pre-0.114.0 agent would
have read ABSENT and stopped its apps.
Tests +6 in internal/localapi (208 -> 214): groups A-D plus a wire-contract test
asserting the ENCODED bound_under_parent, since that is what crosses to the
controller. Four red-proofs run and reverted (drop the conjunction on each path;
invert unknown; drop the bind half); disks.go verified byte-identical after.
NOT LIVE-VALIDATED. No drive was pulled. Leg awaiting Session C: device loss ->
gate Stop -> SetDisconnected -> backup_target_absent on the wire.
The agent cannot create a PVE storage (Datastore.Allocate at /storage) or grant
an ACL (Permissions.Modify) -- it holds neither by design, and widening the role
would trade the whole blast-radius containment model for one feature. The
privileged half therefore lives in a new fenced shim behind a literal
FELHOM_BACKUPTARGET sudoers alias, following the mkfs/pbs-apply pattern.
The wrapper enforces the two laws E-1 paid for on live hardware so no caller can
forget them: F-1 the path must BE the drive's own mountpoint, F-2 is_mountpoint 1
is hardcoded rather than a caller flag. It refuses a root-device target, has NO
storage-removal path of any kind (the pbs-apply no-delete law, grep-assertable),
is idempotent for the same path, and REFUSES to repoint an existing id.
POST /backup/target drives it in a fixed order: create -> grant -> config.
Reversed, a config pointing at an ungranted storage 403s every backup on first
run -- exactly E-1 finding F-3. A failed grant leaves the config untouched.
It deliberately does NOT restart the agent: restarting with a backup in flight
cancels the wait and records a spurious tier failure for a backup that actually
succeeded (E-1 did this to a real felhom-pbs run). It returns restart_required
and the caller restarts behind its own immediate in-flight check.
Config rewrite preserves unknown keys verbatim and writes in place, since
/etc/felhom-agent is root-owned while agent.json is agent-owned 0600.
Green gate: build + vet + test rc=0 (29 packages), run separately from this commit.
The v0.107.0 watchdog was silent on a healthy box, so its health could only be inferred
from absence — F-OBS's shape, shipped in the same session F-OBS was fixed. INFO summary
every 10th sweep with what it saw; aborted sweeps are not counted. Red-proofs 7 and 8.
F-REBOOT — a guest rebooted mid-backup never came back (fault 11: 9m47s of total
appliance outage, no lock, nothing retrying). The existing stale-lock recovery is
correct but missed it two ways: its predicate needs a stale vzdump lock and that
guest was unlocked, and it runs only at agent startup. New periodic guest-power
watchdog acts on 'should be running, is not, is not locked'.
onboot is the should-be-running signal, not invented here: stalelock.go already
uses it for this same decision, it is 0 on scratch/golden, and pve-guests uses it
at host boot. Guards: onboot:0 never touched (Scenario B), a locked guest is left
to the stale-lock path, a guest with a vzdump in flight is left stopped,
unprovable ownership acts on nothing, unconfirmable backup state fails safe.
Bounded retry 3x at 1/2/4m then ERROR (Scenario C) — a healthy start takes ~25s.
F-LEAK — a failed restore-test could not destroy its scratch (403 VM.Allocate).
It is pool membership, not privsep: VM.Allocate is granted at /pool/felhom only,
and a failed restore never completes the --pool association. Fix needs NO new
grant — Pool.Allocate is already held, so the teardown adopts the stranded
scratch into the pool and retries the destroy. Guarded by scratchAdoptAllowed:
scratch provenance AND the numeric band, both required (Scenario E).
Six red-proofs across both fixes, all observed failing.
newestArchiveOn's (time.Time, bool) signature could not express the 'unknown'
its own doc comment promised: a read error and a genuine not-found both returned
(zero,false), so /backup/due emitted a POSITIVE 'no successful backup recorded
yet' with a nil age, and the controller fired its window-gate valve on an
unreadable storage.
Three states now: known / absent / unknown, carried as a STRING enum so the zero
value unambiguously means 'legacy agent' rather than masquerading as an answer.
Fail-safe direction unchanged — unknown is still DUE; only the window-gate bypass
narrows to ABSENT.
A service with NO lister deliberately stays ABSENT: calling it unknown would stop
a genuinely new box on a pre-R-84 build from ever backing up outside its window.
An unparseable timestamp becomes unknown — a backup happened, we cannot date it.
The scheduler could only ever see cfg.Backup.BackupTarget(), so the offsite
tier's archives were never candidates — which is why demo-hp's DR tier reported
'applied' with zero snapshots for five days and nobody noticed.
Selection: oldest-first (operator ruling, Option 1). Never-proven sorts first,
which is where the offsite tier starts. Ties break on target id so ordering is
deterministic rather than following Go's randomised map order. Rotation credit
only on SUCCESS — a permanently failing tier must keep sorting first, not look
freshly proven and stop being retried.
- backup.RestoreTestState: persisted last-success per tier (atomic tmp+rename).
This genuinely needs persistence unlike R-84: R-84 had ground truth to consult
(the archive is still on the storage), whereas a restore-test destroys its
scratch and leaves no artifact. Corrupt/missing file -> 'nothing proven'.
- backup.InFlight: host-wide one-heavy-op gate shared with the local-API backup
path. A LINK concern, not a lock one — an offsite restore pulls multi-GB over
the same tunnel a backup pushes one, and at ~33 MB/min both drift toward
timeout, which is how a healthy tier gets recorded as failed. Callers DEFER,
never cancel.
- PickRestoreCandidateOn: newest archive on a named tier; '' is not an error, or
every fresh box looks broken for its first week.
- An empty tier is skipped and the next tried; it cannot starve, since it is
still least-recently-proven once it has an archive.
- POST /backup joins the gate (409 naming the holder).
Red-proofs A/E/F observed with the documented text. Full suite green (29
packages, rc=0).
Observed live: three redundant local backups on demo-felhom in one afternoon of
deploys. The backup Store is in-memory ('lost on restart; the cadence
re-populates'), so after every restart /backup/due said 'no successful backup
recorded yet' and the controller took another one. On the offsite tier that is a
wasted multi-hour WAN upload after every agent deploy.
- BackupRunner.NewestArchiveTime: when a backup last LANDED on this tier's
storage, read from the storage.
- localapi.BackupArchiveLister (optional BackupService extension): the due-check
takes whichever is newer, the in-memory record or the storage.
Asking the storage rather than persisting the store is deliberate: it is ground
truth (a pruned archive correctly stops counting, where a persisted record would
keep claiming a backup that no longer exists), needs no new on-disk state, and
answers only 'when did a backup last land' — the richer fields stay with real
records so the host-report never carries invented numbers.
Fail-safes: read error -> fall back to memory (never fake freshness, never
suppress); genuinely empty -> due; old archive -> still due; service without the
lister -> unchanged.
Red-proof observed; full suite green (29 packages).
Prerequisite for the installer default (host-install 1.20.0). A fresh box now
carries the offsite tier, but felhom-pbs only exists once the hub provisions the
DR tier. Without this the tier would report due in that window and the
controller would quiesce the apps and fire a vzdump at a missing storage every
cadence.
- GET /backup/due?target= defers when the target storage is absent
(targetStoragePresent): due:false with a reason that says why. The tier goes
live with NO restart once the storage appears.
Fail-safe: a storage-view ERROR returns present and the tier stays due. 'I could
not check' must never be read as 'not there' — that would silently suppress
backups, the absence-is-not-failure rule relearned three times now (R-80, R-81,
the R-82 wait timeout).
Full suite green.
Ruling 1 (2 weeks of weekly offsite backups): localPruneSpec's blanket PBS
refusal is now scoped — an ADDITIONAL tier with an explicit keep_last may
prune its PBS target. The refusal still applies in full to the PRIMARY tier,
because BackupTarget() defaults to felhom-pbs and KeepLast() defaults to 3, so
a box with neither key set would silently prune its offsite DR to 3 restore
points. An additional tier cannot have that accident (keep_last defaults to 0).
Ruling 3 (first backup runs as long as needed; nothing else starts until done):
- additional-tier wait bound 6h -> 12h (measured ~33 MB/min => ~5h for a first
full 10 GB snapshot; 12h gives margin but stays bounded so a hung task still
surfaces)
- ONE BACKUP AT A TIME PER GUEST across all tiers: POST /backup returns 409
when a DIFFERENT tier is in flight, naming the busy tier, with NO data object
so nothing is parseable as the caller's own job. Same tier still returns that
job (202, unchanged).
- snapshotted now counts as in-flight, not just running — after the snapshot the
vzdump is still uploading and holding the lock. The old check left a window
where a second POST started a real second vzdump. Latent bug, closed.
Full suite green (29 packages); red-proof observed and restored.
Found by live validation on demo-felhom, not by review.
The first real PBS-targeted backup ran past the runner's hard-coded 30-minute
WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE
VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences:
the tier stays permanently due, the next attempt collides with the guest lock
the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's
'applied and empty' fault re-created by a timeout.
Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot
projects to ~5h.
- BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED
(a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from
the measurement.
- backup.NewBackupRunnerWithWait: per-instance (per-tier) bound.
NewBackupRunner keeps its signature, so restore-test/selftest are untouched.
- localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the
tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h
outer context reproduces the same false failure four hours later.
Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait
LONGER. A slow backup is a slow backup; a false timeout is a corrupt status
plus lock contention.
Red-proof observed and restored; full suite green.
Mechanism only. No box changes behaviour until a backup_targets entry is
added to its config (Slice D); an untouched config resolves to exactly one
tier and behaves byte-identically to v0.96.0.
- config: BackupTargetConfig + ExtraTargets + BackupTiers(); each tier carries
its OWN cadence and retention (keep-last=3 is three days on a daily tier and
three weeks on a weekly one). A missing cadence is REJECTED, not defaulted —
a weekly DR tier silently running daily would fill the 37.2 GB datastore.
main.go logs every rejection at ERROR.
- /backup/due?target= judges a tier against its OWN newest successful backup.
Without that filter a fresh local backup satisfies the weekly PBS cadence and
the DR tier never runs — today's bug, re-created in code.
- GET /backup/tiers advertises the tiers; a 404 is the controller's pre-R-82
capability probe (Slice B).
- Jobs keyed by (vmid,target): single-flight is per tier, which is what lets
the weekly night run both backups in ONE quiesce window. Job ids are unique
per tier by construction, not by clock luck.
- One runner per tier: the runner holds target+retention as immutable state,
so parameterising one runner would risk pairing tier A's target with tier B's
retention.
COMPATIBILITY (frozen): untargeted /backup/due, POST /backup and
/backup/status keep the primary tier and the pre-R-82 response BYTES —
Target is omitempty and stays empty. The primary's job-id format is unchanged.
NOT changed: the local tier; PBS is still never pruned by the per-run flag
(keep_last defaults to 0 = never prune — enabling DR pruning is irreversible
and needs an operator ruling).
Tests 748->768. Red-proof #1 observed and restored.
Phase 0: felhom.eu/documentation/audits/SPIKE-r82-phase0-2026-07-26.md
Implements SPIKE-smart-coverage-2026-07-25 fixes B+A (additive; MinAgent unchanged).
Fix B: storage.SmartReader.SMARTForBacking wired into the /disks union path (localapi
Smart seam) so registry/USB drives get a real SMART read (watchdog Known stays
enrich-free). Fix A: smartDeviceFor resolves dm/LVM to the whole disk via
/sys/block/<dm>/slaves (recursive; skips >1-disk); the builtin local dir on the LVM
root gets a SMART-only device from its containing filesystem (never touches
backing/durable_id). SmartSummary.ModelName captured from smartctl. Fix C (-d sat)
stays rejected. Tests + red-proofs (dm multi-disk skip, enrich smartHint, union
routing); Known-path-never-SMARTs asserted.
MinAgent coupling: felhom-controller v0.143.0 gates its guest-memory-resize UI on
this agent (FeatureGuestMemoryResize, MinAgent 0.90.0).
R-24 guest RAM resize (internal/localapi/guestmemory.go): self-scoped GET/POST
/guest/memory. Agent enforces every bound FRESH per request (min 2048, max
host_total-2048, shrink floor max(2048, usage+512)); applies via PVE SetConfig —
live cgroup apply, no reboot (Phase-0 proven on the nested demo box). Verify-after-apply
re-reads maxmem before claiming success. New narrow MemoryOps seam (GuestAPI untouched);
Options.Memory nil -> 503. Memory only.
R-28 fast-tick (internal/fasttick): while any desired-state item is unapplied -
including the pre-tunnel window a hub poke can't reach - pulse the shared out-of-band
trigger every 30s, self-disarm on convergence. Four cached sources (desired-gen==0,
reconcile Planned-Pending>0, pbsdr waiting_secret only, wgtunnel desired-not-operational);
LOUD pbsdr states + pending_signature excluded. Seams: reconcile.Engine.LastResult() +
wgtunnel.Manager.TunnelConvergence() (cached, no per-tick exec).
Guests-0/0: hypothesis REFUTED live (9201 IS a pool member; 0/0 was the pre-provision
window; PoolAddVMID re-assert already covers restore-over-existing). No code change; the
fast-tick mitigates the window.
Tests + red-proofs (i floor guard, ii max guard, iii always-pulse) all restored green.
Agent half of the verify-before-commit task (SPIKE-nas-verify-2026-07-11, b57f6c1):
retry=0 in the production NFS options (Q4-vi); ClassifyNetVerifyFailure on the live
Q4 strings (nfs_export merges not-found/not-permitted); add = sync fast-fail (2s TCP
pre-probe, nothing installed) + detached in-memory verify job judging /proc/mounts
only, auto-rollback on failure; GET /netstorage/verify-status (phase none = the
controller's Scenario-F rollback signal); unprivileged journalctl (systemd-journal
group, NO new sudoers grants).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The staged offsite repo password was wiped only by the escrow-create
ceremony; a confirm without a fresh ceremony (password already escrowed)
left the 0600 staged file behind. The controller calls this on every
EscrowState flip to escrowed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
From AUDIT-blast-radius-hostroot-localapi-2026-07-02.md. Each fix ships with a
non-hollow test + a companion red-proof (shown failing on the pre-fix impl).
Sudoers install-source grants became globs — deploy the sudoers drop-in with
the binary. A1 (stale-lock pool-membership) deliberately excluded (spike).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
GET /disks/candidates enumerates host disks the Impl-1 unclaimed filter proves
free (init/attach split). RegistryKnownTargets sources the watchdog's known-drive
set from the intent registry + Felhom .mount units (not Observe/PVE storages) —
decouples drive health from PVE storage (closes the registry-only false-detach
class); Observe kept for real PVE storages + a deduped /disks union. Idempotent
existing-drive migration at start. Tests + red-proof (Observe misses a
registry-only drive; registry provider tracks it). go build/vet/test clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Host-side NFS/SMB automount of a bulk-media NAS share under /mnt/felhom-drives/<name>
(propagates into the guest via the existing shared bind), the +100000 uid recipe,
per-share liveness, and add/list/remove local-API endpoints. A NAS is a distinct class
that bypasses the drive enroll/eject/decommission/SMART/watchdog machinery.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
F2-b: at startup, recover a guest left with a stale vzdump lock by a
reboot-during-backup — pct unlock -> delete dangling vzdump snapshot ->
start iff onboot, guarded by a no-vzdump-running invariant (fail-safe).
New internal/localapi/stalelock.go; proxmox GuestConfig.Lock()/OnBoot(),
ListSnapshots, ListRunningTasks, Snapshot type. New narrow sudoers grant
FELHOM_STALELOCK (pct unlock) + Critical capability stalelock-unlock.
F2-a: EnsureSharedParent only redeployed the boot script when the UNIT
differed, so the v0.36.6 make-private fix never reached hosts whose unit
was current -> /mnt/felhom-drives stayed in root's shared:1 and doubled
every drive bind. New sharedParentInstallStale compares BOTH script and
unit. Boot-time-only; never churns the live mount.
Both root causes confirmed live on felhom-pve before fixing. Green gate
(build/vet/test) all pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
controllerHealthy reads RestartCount (running&&rc>0 -> not ok) + signals needsDwell for no-healthcheck;
verify requires verifyDwell(=3) consecutive ok polls for a no-healthcheck image (real healthcheck
trusted immediately). Closes the F1 hole (alpine crash-loop passed the point-in-time check). Red-proof
+ dwell + real-image tests. No sudoers/orchestration change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
New local-API POST /controller/swap (+ GET /controller/swap/status), withGuest-
scoped: the agent records the previous image, confirms the target is present,
rewrites /etc/felhom-controller-image, restarts felhom-controller-bootstrap.service,
verifies the new controller is healthy (docker inspect, <=90s), and ROLLS BACK to
the previous image if not. Single-flight per guest; strict image-ref validation;
crash-safety state file. GuestBinder.GuestExec is the pct-exec seam.
--selftest=controller-swap exercises it directly.
Tests: happy/rollback-on-unhealthy(+red-proof)/image-absent/no-healthcheck/
bad-image-400/single-flight-409.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7
Replaces the per-drive 'pct set -mpN' bind with ONE permanent parent bind
/mnt/felhom-drives plus host-side felhom-data swaps underneath it (propagates
into the running guest live, no pct, no reboot; C1-immune; confined; fail-closed
when absent). EnsureSharedParent installs a boot unit ordered Before=pve-guests.
ReassertGuestBinds is now a pure host-side reconcile. /disks reports GuestPath +
BoundUnderParent for the controller repoint+gate. Non-hollow tests + companions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
POST /disks/decommission mirrors eject (withGuest, user-data role gate) — no
operator signature, non-destructive (never formats): sets IntentDecommissioned,
prunes the GuestBindStore entry, unmounts. ReassertGuestBinds is now intent-aware
(skip non-enrolled) so a decommissioned-but-present drive never auto-rebinds on
agent restart — the load-bearing F9-reconnect fix. GuestBindStore.Remove added.
Operator-signed DecommissionExecutor + classify untouched. Non-hollow tests incl.
the intent-aware reassert companion (mutation-proven to fail on intent-blind code).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The format ran mkfs under the HTTP request context, so the controller's 15s client
timeout cancelled it → SIGKILL mid-write → corrupt disk. Now mkfs runs DETACHED off
s.baseCtx (a dropped request can't kill it) via a persisted formatJob record; the handler
still waits to return the synchronous result (backward-compatible with the v0.62.0
controller) but abandoning the wait on client-disconnect leaves the mkfs running to
completion. New GET /disks/format/status surfaces the job (additive). RecoverFormatJob
runs on agent startup: a record left 'running' (agent died mid-format) is re-resolved by
durable-id (anti-retarget — absent/swapped disk NOT re-formatted) and the mkfs re-run; a
blank/path-bound interrupted format is marked failed (retry), never auto-re-run.
Tests: detached run persists running→done + binds durable-id; status endpoint; recovery
re-runs an interrupted durable-id-bound format; skips blank; skips unresolvable durable-id.
Version 0.30.0 → 0.31.0.
The in-guest bind (pct set -mpN) is config state that a destroy+re-provision drops, and
nothing restored it — so a re-provisioned guest came up with its enrolled HDD unattached
(the live-drive F9 finding). New GuestBindStore persists, per guest, the durable-ids of
enrolled drives (recorded at guest-attach); ReassertGuestBinds runs on agent startup (the
host's bring-up/reconcile trigger) and re-adds any bind a guest is MISSING — but ONLY when
the durable-id still resolves to a present, mounted drive (a swapped/absent drive is never
auto-bound) and the guest lacks it (idempotent). The re-added bind activates on the guest's
next reboot, like the enroll flow. Wired in main.go (store opened beside drive-intents.json;
ReassertGuestBinds called before the local API serves).
Tests: restores a missing bind with no manual call (the operator's real-trigger proof);
skips absent/swapped durable-id; no-op when already bound; store survives reopen (restart).
F20-BUG2: the /disks list only carried DurableID in the uuid: scheme (for /disks/assign),
but the wipe gate resolves devices in the byid:/byuuid: scheme — so a customer confirming a
wipe with the advertised id was refused (binding_mismatch). Added a shared s.deviceDurableID
seam used by BOTH handleDisks (new DiskInfo.WipeDurableID) and the format gate, so the id the
customer copies from the list is exactly the id the gate accepts. DurableID (uuid:) is unchanged
(still feeds assign).
F9 (reporting half): added DiskInfo.GuestAttached — whether the drive's namespace is actually
bound into THIS guest's config (guestBoundPaths), distinct from mere host presence (State). This
is the signal whose absence made the HDD look available when it wasn't attached, and resolves the
F2 hdd_configured-vs-/disks disagreement.
Tests: wipe_durable_id is the gate scheme + distinct from uuid:; the list's wipe id matches the
gate's device-id binding (no mismatch); guest_attached true iff bound into the guest.
handleDiskFormat's customer-confirmed branch formatted the mutable req.Device
path; the durable id only bound the confirmation, never the mkfs target. A /dev
reassignment between inspect and mkfs could wipe the wrong physical disk.
Now mirrors signedjobs.WipeExecutor: resolve confirmed durable id -> current
device, re-derive + require exact match, re-inspect (still data-bearing), then
format THAT device. Any refusal -> 409, no mkfs. New antiRetargetResolve helper
(injected deps, unit-tested: mismatch/gone/blank/empty all refuse). Injectable
reresolveWipe seam on Server (defaults to real storage funcs).
BRANCH ONLY — pending supervised review/deploy (see AGENT-001-FIX-NOTES.md).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
POST /disks/guest-attach binds an enrolled drive's felhom-data namespace into
the guest (Model A: felhom-data is the bind source mounted at /mnt/<name>, so
only Felhom's namespace crosses in). GuestBinder does mkdir+chown(100000)+pct set
(RW bind) via the fenced runner. Idempotent, free-slot selection, path-validated.
Spike-proven on 9201. Pairs with controller P2C + golden /mnt:rslave (P2B).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
handleDiskEject now resolves the authoritative role of the storage at `where`
and refuses 403 (no Unmount) unless it is user-data. Fails safe to protected on
ambiguity. Adds roleForMountPath + an injectable HostReader seam for testability.
TestEject_RoleGated asserts protected mounts are refused with no Unmount.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A destructive op runs ONLY on a pinned-key-verified, nonce-fresh, in-window,
host-bound, durable-id-bound operator signature. New cmd/felhom-opsign signs
canonical OpBlobs offline via ssh-keygen -Y sign (hardware-ready); the signing
key is never in the hub or agent. New internal/signedjobs runner verifies each
queued blob through the gate and only on all-pass runs the WipeExecutor, which
re-resolves the DURABLE device id + re-inspects (8C) before mkfs — closing the
8C data-bearing-wipe pending_signature gap. New storage durable-device
resolution; authz.CanonicalBlob promoted to production. Real-crypto tests assert
valid executes and forged/replay/expired/retarget/non-pinned are rejected
(executor never called).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a host-wide, token-authed GET /host/metrics local-API endpoint that
re-serves the slice-4 collector's host + per-storage view to the customer
(the de-privileged controller can't read the host itself). Add the one new
collector — CPU/chassis temperature via sysfs hwmon/thermal-zones, graceful-
null — to the shared HostMetrics struct, so the hub report carries cpu_temp_c
too. Cross-repo host-report golden updated byte-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BackupRunner.BackupWithSnapshotHook tails the task log for the 'create storage
snapshot' marker (snapshot mode only) and fires onSnapshot once; localapi flips
/backup/status to 'snapshotted' before 'done' so the controller resumes early.
Phase 0 validated on PVE 9.2.2: marker confirmed, downtime ~24s->~1s (934MB).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
internal/localapi: real /backup/due (cadence; due when no successful backup or
newest older than backup.backup_cadence_seconds; false in-window after success;
failed doesn't count) + /backup/status phases (idle|running|done|failed + job
id) + POST /backup single-flight with job id. Drives the controller quiesce loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
admin
Claude Code