A hub the agent could not reach was reported to the customer as a bad recovery
code. Measured live 2026-08-05 (CAMPAIGN-11 F3): hub firewalled off, a CORRECT
current code, and the customer told it did not open their package — in 0.0556s
against ~1.0s for a real unseal. No unseal was attempted.
The discriminator existed here and this boundary threw it away: recover.go
fails at four distinguishable points and the local-api handler had cases for
two, with a default answering 'the recovery code did not open the sealed
bundle, OR the bundle could not be fetched'.
escrow.ErrBundleFetch now joins the fetch leg and the handler routes it to 502
with its own words — the code was NOT used. 502 not 4xx: the request was not
bad, an upstream dependency failed. Four situations, four statuses: 502 fetch /
400 fetched-and-refused / 404 no bundle / 409 predates the field. The
controller classifies on the STATUS and never parses the sentence.
A GREEN TEST NAMED THIS DEFECT AND DID NOT PREVENT IT.
TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct has said since v0.125.0
that the operator must not be sent to re-read their code because the hub was
unreachable — and passed throughout, because it asserted this package's error
STRING one layer below the merge, and a string is not something a caller can
branch on. Re-pointed at the sentinel, with a consequence-level twin asserting
the status.
Red-proofs: removing the %w join fails the sentinel test; deleting the handler
case makes fetch and wrong-code both answer 400 with the wrong-code sentence.
29 packages ok, vet clean, agent gates OK.
The workspace root is already documented (workspace-CLAUDE.md, the workspace-root
CLAUDE.md 'stay inside it') and work drifted into a home directory anyway. A rule
that has failed once as a reminder is not fixed by writing it down again, so it is
now asserted where it can bite.
A push is the right trigger: throwaway clones under /tmp for probes and red-proofs
never push, so nothing legitimate breaks. Symlinks are resolved on both sides; an
absent workspace root SKIPS the check rather than failing it, so this cannot brick
a legitimate clone on another machine. The only bypass is the documented
--no-verify, whose use is already reportable.
Identical in all four repos.
Link 7's only production caller was a --selftest reading R from an env var. Link 8 did not
exist: that selftest writes the whole bundle JSON and its success message named
"tunnel_token + pbs_token" -- accurate when written, a misstatement since v0.77.0 sealed the
offsite repository password into the same bundle. It now names what THIS bundle carried and
what it did not.
POST /escrow/recover-offsite-password: the controller supplies R, the agent fetches this
host's own blob from the hub (self-scoped by the per-host key), unseals it, and returns ONLY
the offsite restic repository password plus its sha256. Not the tunnel token, not the PBS
token, not the WG key -- the controller is a trust tier down and needs none of them.
R: in memory for one call, cleared on every path, never on disk, never in argv, never logged,
never echoed. A test redirects TMPDIR and asserts the tree is EMPTY afterwards -- emptiness
rather than a content scan, because a content scan is defeated by a later call overwriting the
leaked file, which is how the first version of that test passed its own red-proof while R sat
on disk.
Three distinct outcomes: no blob (404), a bundle that opens but predates the field (409), a
code that does not open it (400, fail-closed at the KDF, nothing written).
The wiring is asserted by an AST walk from func main() to the Options field, not by grep.
v0.124.0's transition record never reached the hub, and only the live run showed
it. The capability reported degraded for "one cycle" — the probe call that did the
repair. But probeAll is invoked independently by the self-check log and by the
collector building a host report. On the demo box the repairing call was the log's
(09:39:34, journal shows the self-repair and degraded=1) and the report three
seconds later found the grant present and sent ok. The agent's journal had the
record; the hub had nothing. That is the silence R-190 is about, re-created inside
its own mitigation, with every unit test green.
Fixed with a latch on TIME, not call count: a confirmed repair reports for 20
minutes, which exceeds the 900s report interval, so at least one report must carry
it. It clears on its own and is per tier.
Two hollow tests caught and fixed on the way — one asserting a value it built
itself, one asserting the latch helper rather than the path consuming it (its
red-proof duly passed). The decisions now live in storeGrantHealthyVerdict and
storeGrantRepairedVerdict and the tests call those.
R-190 is a grant that worked at 04:44 on 2026-08-03 and was gone by 09:24, with a
reinstall, logged pveum activity and cluster-log entries all ruled out. The cause
is open; the resilience need not wait for it.
Everything needed already existed and had only ever been called once: the root
wrapper's `grant` verb, its sudoers vector (`grant *`, any storage id — confirmed,
not assumed), and the exact command. The verb had only ever run at storage
creation — the "built but never wired" shape in a verb rather than a seam.
The probe now runs that wrapper on a missing grant and re-reads ONCE to confirm,
the pbsdr R-22 shape including its restraint.
The record is the half that matters. A repair leaving only "ok" behind destroys
the only evidence a permission vanished, so a recurring loss becomes undetectable
— worse than the fault. A confirmed repair therefore reports DEGRADED for exactly
one cycle with the explanation in Feature, because that is the field the hub puts
in the operator's email (Reason does not travel). Nothing new was built: the hub's
existing ok->degraded->ok edge is the channel, so one loss produces one alert pair.
No wire change, no hub change, no new event type.
Bounded at one attempt per tier per hour: a storage can be unreadable for reasons
an ACL cannot fix, and re-granting every cycle is a repair loop wearing a fix's
clothes. A failed repair never masks the fault.
The installer defect was NOT PVE_STORAGES as the row and the task assumed: the
create arm of configure_backup_target grants, the Scenario-F reuse arm did not.
Also records the measured trap (an ungranted path answers with INHERITED
privileges, not empty and not 403), the deviation from the spec's suggested
Prober generalisation in favour of the existing poolReadStatus precedent, the
hollow test caught before it shipped, and that demo-hp carried the same drift and
was fixed.
The missing grant is one command; the silence was the defect. On demo-felhom the
agent's token has FelhomAgentStore on local, local-lvm and felhom-pbs — and not
on felhom-backup, the storage the same installer configured as
local_backup_target. That storage answers {"data":[]} through the token while
root sees three archives.
An empty listing is what a FORBIDDEN tier and a NEWBORN tier both return, so
pickForThisRun skipped it as "no settled archive yet" and the tier was never
restore-testable on that box. The permission question, unlike the listing, has a
definite answer, so it is asked directly: Client.Permissions reads
/access/permissions as the agent's OWN token, and one capability.Status per
configured tier reports it — composed around the sudo prober, the way the
pool-read check already is.
Measured first, because the obvious reading is wrong: an ungranted path answers
neither empty nor 403, but with the privileges inherited from the box-wide grant
(Sys.Audit, SDN.Use, Datastore.Audit). Checking for Datastore.Audit would report
a blinded storage healthy — red-proved. The probe tests for
Datastore.AllocateSpace.
The probed set comes from the box's own config, never a fixed list: a hardcoded
probe list is the defect reproduced inside the fix. Critical, because the hub
alerts only on critical — except the "local" fallback target, which is reported
but does not page. It never looks at content, so it cannot alarm on a newborn
tier; it never reports ok when it could not ask. Status wire shape unchanged, so
no hub change.
Scenario A proven on demo-felhom against the exact observation that filed R-189:
a 675 s offsite restore-test passed, the agent was restarted 11 seconds later
(inside the reporting window), and the hub's very next report carried
'1 restore-tests' where the same sequence produced 0 this morning. The hub's own
database holds the archive, the tier, the pass and the ORIGINAL test time, with
the run mechanics deliberately zero.
Also records the property the validation surfaced: the state holds one proof per
tier, so proving an older archive re-arms a newer one — confirmed live after the
defaults were restored.
All three are the reporting and release path misreporting its own work. No
customer machine, no backup, no restore, no data. The restore-test itself and
when it runs are unchanged.
R-189 — a passing restore-test no longer vanishes on a restart. restore_tests[]
came only from the in-memory store, whose comment ("lost on restart; the cadence
re-populates") was true under a timer and stopped being true when R-86 made the
agent refuse to re-test a proven archive: the proof is then not repeated for a
whole archive generation. Observed live — a 14.5 GB offsite PASS reached no
host-report because the agent was restarted 2m43s later. RestoreTestState now
carries tier + verified beside the archive and renders reportable entries; the
collector merges them, one per tier, newest by TestedAt. It refuses to lie: a
record missing archive-or-tier produces no entry, and run mechanics are not
re-invented. Only successes are persisted, and the asymmetry is now written where
it will be read.
R-188 — a correct release stops emailing a failure. Only the tag PUSH moved
(build -> tag locally -> publish -> push tag): the push wakes CI, and a tag
visible before its package made the gate correctly fail a correct release about
half the time. The old order's invariant is asserted directly instead — the gate
now refuses a published version with no tag, as a bounded probe that prints its
own coverage, because the package listing api is still 401 without a token.
R-186 — a released binary can be verified by rebuilding it. -trimpath
-buildvcs=false: same source, same bytes, tag or no tag. Measured. publish-agent's
fallback also forced CGO_ENABLED=0 and produced a 74 KB different binary for the
same version; both paths now build identically. CLAUDE.md records the command.
Before R-86 every tick ran a heavy restore-test, so the scheduler was audible by
construction. After it, 'nothing is due' is the NORMAL outcome — and it was
logged at DEBUG, which journald drops. An empty journal would then be equally
consistent with a healthy loop and a dead goroutine: the shape the R-88 watcher
was retired for, re-created by making the quiet path the common one.
A not-due evaluation now logs one INFO line naming every tier's verdict (four
lines a day at the 6h default), and an unlistable tier reads UNKNOWN with its
error in that same line, so a lookup failure can never present as 'nothing due'.
Red-proved through the scheduler's own tick, not the helper.
R-115's whole point is that publishing cannot be forgotten because it rides the
release script. On the first real release through it (v0.121.0, R-86) it died at
exactly that leg:
scripts/release-agent.sh: line 101: .../scripts/publish-agent.sh: Permission denied
publish-agent.sh has been mode 0644 since it was created on 2026-06-28 — every
earlier caller ran it as `bash scripts/publish-agent.sh`, so nothing ever noticed,
and release-agent.sh (written the same day it was needed) called it directly.
Two fixes, both small and both wanted: restore the executable bit, and invoke it
through `bash` so the release no longer depends on a file mode — the kind of
thing a checkout, an archive or a copy loses again.
The v0.121.0 tag created by the failed run is withdrawn and recreated on this
commit; nothing was published under it (verified 404 on the package endpoint), so
one version name still means one binary.
The ticker survives as the EVALUATION interval only. A tier is DUE when its
newest archive that has settled for `settle` (default 24h) has not been proven:
daily tier -> proved daily on yesterday's archive, weekly tier -> weekly on its
own, newborn -> UNKNOWN.
The trap avoided: the literal reading ("newest archive is >= 24h old") is NEVER
true on a daily tier, so it silently switches restore-testing off where it
matters most. Red-proved at 0 runs over 5 simulated days.
- state records WHICH archive was proven; legacy files keep their time and yield
no proven archive (each tier due once after the upgrade, deliberately)
- two knobs replace one: restore_test_eval_interval_seconds (6h, measured) and
restore_test_settle_seconds (24h). The old cadence key keeps its DISABLE
meaning verbatim and now seeds the settle lag, with a start-up WARN.
- due-check runs BEFORE the heavy-op gate (a frequent poll must not make a
starting backup record a failure, F-A1)
- candidate picker skips implausible archives (a phantom would be due forever)
- new read-only --selftest=restore-test-due prints the verdict + its cost
NO VERSION BUMP and nothing built: no Go code changed. The agent stays v0.120.0.
scripts/release-agent.sh — THE way to release. build -> tag -> publish -> verify
by INDEPENDENT download. Publishing was a separate remembered step and was
forgotten three times in five days (R-111's 17 stranded releases, 0.114.0, and
0.120.0 — deployed to both demo hosts and undownloadable, so a documented-path
reinstall would have silently downgraded them WHILE REPORTING SUCCESS). R-111's
own closing line named this leg and closed SHIPPED without it; it recurred the
same afternoon, which is the evidence that a note is not a mechanism.
It tags because felhom-host-install.sh now fetches the sixteen agent config
files from raw/tag/v<version>/ (R-183): a released version with no tag 404s a
box mid-install, as root, on a virgin machine. It verifies by downloading what
it just published and comparing the sha to what it built — the publish step's
own success is a report on its own write; a fetch returning the right bytes is
a different claim. It refuses a dirty/unpushed tree and refuses to re-release an
existing version. It does NOT vouch: that points machines at a version and stays
the operator's act.
scripts/check-published-versions.py — the gate. Every v<semver> tag must have a
downloadable package AND a tag tree serving the agent's configs. Registered as
NOT --fast (needs network; a push must not fail because Gitea blinked), and the
CI workflow now runs the FULL gate set instead of --fast — otherwise the gate
would have been registered and never run, the built-but-never-wired failure this
project has shipped four times.
The invariant is not the one specified, and the reason was measured, not assumed:
the hub artifact manifest is 401 without a per-customer passphrase and Gitea's
package LISTING api is 401 without a token, while the package DOWNLOAD url and
the git TAGS api are anonymous. So CI cannot ask "what is vouched" without an
operator credential — whose addition is the operator's call. The tag-based
invariant needs none and catches all three recorded instances. What it does not
catch (the hub vouching a version never released at all) is filed as R-184.
v0.120.0 had been built, committed and deployed to both demo hosts but never
published: the Gitea generic package 404'd and the hub manifest vouched 0.119.0.
Installer step 5 skips only on an exact version match, so a documented-path
reinstall would have downgraded both boxes to the pre-merge agent -- and would
have succeeded, since step_grows passes -sysdata-grow 0 and 0.119.0's mp1 resize
never fires. Published from a clean tree (upload 201, round-trip GET verified,
sha a7763d31b55b5ce7...) and vouched; both reinstalls then fetched and verified
it over the real customer path.
Filed as the third instance of R-115. No version bump, nothing built.
Evidence: felhom.eu/REPORT.md
States the scope reduction first: Phases 6-7 (reinstall both demo boxes and
prove one end to end) were NOT done, nothing was wiped, and the golden is
deliberately left unvouched as a result. Filed as R-178.
Also records the two instrument errors this session: a census truncated by
head -10 that gave the wrong answer about --sysdata-grow (the installer does
pass it), and a wait gated on a marker the bake prints before publishing.
build-golden.sh 2.1.0 -> 3.0.0: a layout change is a major. The golden ships
ONE data volume at a NEUTRAL path (/var/lib/felhom); both /var/lib/docker
and /mnt/sys_drive are binds of subdirectories of it. mp1 is gone.
The variant was chosen on measurement. Three candidates were built and
rebooted (SPIKE-r165-phase0-2026-08-03.md); all three boot, reboot 3/3, give
ONE df figure and keep a container's statfs(/) on the merged volume — the
ordering worry that motivated the probe did not materialise. They differ
only in which guarantee they break: volume-at-docker puts customer backups
inside Docker's data-root; volume-at-sys_drive puts Docker's ENTIRE
data-root under /mnt, which the controller container mounts wholesale
(measured: it then sees /mnt/sys_drive/docker). V-c breaks neither.
The four assertions were RETARGETED, never deleted, and each was RUN against
a deliberately wrong shape — a real split guest and a real archive of it:
8 checks, 8 passed. A new 2b asserts both paths are ONE filesystem, which
catches the S2 shape the spike ranked worse than the split. Assertion 5
replaces the old "was mp1 excluded?" guard, whose pattern could no longer
match — a guard that cannot match has silently stopped guarding.
Provisioning: one volume, one grow. SysDataGrowGB is FOLDED IN rather than
dropped, because a census established that felhom-host-install.sh passes
-sysdata-grow and the two do not upgrade in the same instant; dropping it
would silently shrink every appliance by 42 of 250 GiB. The flags stay
accepted for the same reason. The existing test was retargeted to pin the
fold, and it caught the change before I did.
Same shape as the other repos. CI clones felhom.eu as a sibling because the shared
reuse_refs_check lives there and this repo's REUSE.md cites hub/internal/store/dr_recipe.go,
which lives in the hub. No uses: step, no version bump, nothing built or deployed.
A census of all thirteen gate scripts across the four felhom repos on 2026-08-02 found that
every check a CLAUDE.md names was passing and two of the four nobody is told to run were
failing. This repo was the extreme case: nothing ran against it at all, and its REUSE.md — 90
cited paths — was checked by no one.
agent_gates.py exists at ONE gate on purpose, so the agent is not the one repo with nowhere to
put a check and so the pre-push hook has the same entry point in all four repos. It grows when
the agent grows a second gate. The shared reuse checker stays in felhom.eu/scripts/ and is
invoked across the workspace — never copied here; an absent sibling clone FAILS the gate and
prints the path tried, which test_agent_gates.py pins by running the entry point from a lone
directory with no sibling.
.githooks/pre-push runs it with --fast and refuses the push. Per-clone and --no-verify-able,
both stated in the hook itself; a manual run WARNS when the clone is unarmed.
Tooling only: no Go change, no build, no deploy, no version bump.
A managed box's IP was invisible in every operator surface because nothing
reported one: HostMetrics carried node/cpu/mem/disk/load/uptime/temp/wrapper-sha
and no address of any kind. The hub could not show a host's LAN IP anywhere.
Two things that looked like the answer are traps, both checked before writing
code: lan_resolver.host_ip is an OPTIONAL config value absent unless that feature
is configured, and DeriveHostIP(local_api.listen_addr) returns 169.254.253.1 —
since R-50 the local API binds a link-local address identical on every box. Both
would have produced a confident wrong answer.
New wire field addresses[], one entry per (interface, address). Deliberately
iface+cidr rather than a single lan_ip: a Proxmox host legitimately holds several
(management bridge, tailnet, WG tunnel) and picking one to call "the" LAN IP is a
guess the agent is not entitled to make — silently wrong on a box whose bridge is
not vmbr0. The agent reports what exists; the hub does the labelling.
The filter is one predicate, chosen by MEASURING both demo hosts rather than by
reasoning about interface names. IsGlobalUnicast() alone drops loopback, IPv6
link-local (one per bridge, pure noise) and IPv4 link-local (169.254/16 — exactly
the island address above). It needs no veth/fwbr/tap denylist: that per-guest
plumbing carries no IP at all and self-excludes, verified on both boxes.
No new privilege and no block I/O — net.Interfaces() is a netlink/procfs read, so
the sudoers fence is untouched and the health-check rule is honoured.
The seam DEFAULTS to the real enumerator, inverting the nil-reporter-means-off
convention: this stanza has no config gate, so a forgotten wiring call would have
shipped it silently empty — the inert-seam failure recorded four times here.
Cross-repo: the golden is duplicated byte-identically in felhom.eu and the
contract test fails on top-level key drift, so both goldens moved together and
addresses[0]'s key set is asserted bidirectionally. The field marshals as [],
never null — the repo's own no-nulls invariant caught that on the first run.
Tests +9; three red-proofs (global-unicast filter, down-interface guard, inert
collectAddresses) each run, observed failing, and reverted.
Live validation caught what the tests could not. On demo-felhom the recipe read
namespace "root" with namespace_state "resolved" — confident and wrong, a worse
shape than the original defect.
mergeConfig overlays the cluster storage config onto the node entry through a
hand-listed set of fields and Namespace was not among them. NodeStorage does not
return the namespace at all, so PBSNamespace always read "" and latestPBSCoord
correctly treated that as the root namespace.
Every v0.118.0 test built StorageTarget values directly — including the two
through Collector.Collect(), which inject a fakeObserver — so nothing crossed the
merge. Two new tests drive the real Observe path with PVE's actual split returns
and table the merge itself. Red-proof: dropping the added line fails both.
Suite rc=0, 29 packages, 0 FAIL.
Both defects were live on both demo boxes: the recipe said namespace "root" while
storage.cfg said demo-felhom/demo-hp, and it never named which of two content=backup
dir storages holds the local archives.
R-106: the namespace came from the listed snapshot, but PBS omits `ns` per item once
the list is namespace-scoped, so it was always empty and normalised to "root". It now
resolves from the pbs STORAGE (storage.cfg's `namespace`) — the same field vzdump makes
PVE read, so the recipe cannot disagree with the backup.
R-109: backup_target resolves from the primary tier of cfg.Backup.BackupTiers(), the
function the scheduler consults, and carries the mountpoint that separates /mnt/hdd_1
from /var/lib/vz. The resolver reports the tier IN EFFECT (daemon-start config), not
agent.json on disk — a target move rewrites the file and deliberately does not restart.
Unresolvable is recorded as unresolvable: resolved|unknown plus a distinct reason,
never a default, an empty string, or a placeholder.
Needs hub v0.83.0 — AssembleDRRecipe allow-lists top-level keys, so backup_target
would otherwise be stored intact and dropped before any operator saw it.
9 tests, 4 red-proofs (each mutation asserted to have landed). Suite rc=0, 29 ok.
Predicate validated on hardware for both dead states (stale-device and
filesystem-aborted), 340-497us per call, no block I/O proven by strace (only
/proc/self/mountinfo, 0 statfs). No regression through the real pipeline: the
live backup-target drive reads bound_under_parent=True via GET /disks with the
controller's own credential.
Records what was NOT covered: the stale-bind repair on hardware (StablePathForRaw
hardcodes the live parent, so it would write into guest 9201's namespace -
R-117h), and sustained-load behaviour, still unmeasured.
BoundUnderParent reported a namespace that returned EIO on every read and write
as healthy, and the gate restarted the customer's apps onto it. Both existing
terms parse a mountinfo line and then test only fields[4], the mount POINT.
Field 3 — major:minor — sat in the same parsed slice and was discarded.
Measured on hardware: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb with `shutdown`,
bound_under_parent true, EIO both directions, and the controller taking its
Return branch and emailing backup_target_restored with no alarm on any channel.
BoundUnderParent gains a third term at both /disks construction sites. The new
bindLiveness reads /proc only and asks two questions: the bind must name the
same device as the raw mount, and the filesystem must not have aborted (ext4
`shutdown` or `emergency_ro`).
The second check is not optional. A device that fails WITHOUT disappearing gives
the identical all-signals-healthy state with the devnos EQUAL and the drive never
Disconnected, so the gate produces neither a Stop nor a Return and nothing is
emitted on any channel, indefinitely (R-117a). A devno-only fix would have passed
every payload test.
Three states, never a bool: {Unknown, Live, StaleDevice, Aborted}, read through
Usable(), where Unknown counts as PRESENT — reporting absent stops a working
customer's apps.
No new recovery path; the existing one was unblocked. AttachDrive's normalize leg
already did the repair and three call sites already invoked it, including the
controller's Return branch before it restarts apps. All three died on
`if n == 1 && GuestSeesMount(...)` returning early. Now: StaleDevice ⇒ re-bind
(repairs live, guest never restarts); Aborted ⇒ quiet no-op, because a re-bind
lands on the same dead superblock and this runs every 20s — an infinite silent
retry that masks the state; it surfaces via BoundUnderParent=false instead.
Ordering trap caught by a test: reading the abort flag before comparing devices
classifies the real return state as aborted (its stale bind carries `shutdown`
too) and refuses the repair while still reporting correctly. The abort flag is
read off the RAW mount in the stale case.
Tests 849 → 863, 29/29 packages green. 6 red-proofs, each verified to have
landed. A hollow test was caught during them: the aborted fixture first used a
/dev/mapper device, for which RoleForStorage derives role=system — a system row
has no GuestPath, never runs the conjunction, and reports false by default, so
the assertion passed vacuously and no mutation could fail it. Found because RP1
failed to fail.
felhom.eu/CLAUDE.md now carries the standing rule (with the measurement), but
that file does not load in an agent-only session — and health checks are written
in this repo. A standing rule that does not load where it binds is the inert-seam
shape applied to a rule, so the constraint is repeated here as a one-liner with a
pointer to the full text.
No code change, no version bump.
v0.116.0 validated on a fresh box: real day-0 from the v1.25.0 ISO on a nested PVE
on demo-hp (per runbooks/target-selection.md), agent installed unaided from the
vouched Day-0 manifest, drives enrolled through the real endpoints, device loss a
real hot-detach.
07:20:04 backup_target_absent (error) Cel meghajto <- TARGET, specific
07:22:34 backup_target_restored (info) Cel meghajto <- its matching pair
07:24:04 storage_disconnected (error) Adat meghajto <- NON-target, generic
07:25:34 storage_reconnected (info) Adat meghajto
All four reached the hub; gate fired in 3 s. Discrimination is proven NON-trivially
for the first time -- both prior runs had the target itself emit the generic event,
so their mirror proved nothing. Over-correction passes on a POSITIVE observable: 0
ABSENT lines and 0 drive events over 2m14s with both drives present, while two
RETURNED lines prove the gate was ticking rather than dead.
Caveat recorded, not a regression: the drill's controller was 0.185.1 from the
golden, which predates R-114, so its absent-state banner showed the old false
"backup is on the system disk" copy. The R-114 guard is pinned by unit test and by
the payload, but could not be confirmed on that box. Filed as R-120 -- the golden
is a release behind the deployed controller, which is R-115's class one layer up.
Teardown all three layers, including the hub: VM purged, storage removed with the
space measured back, hub records gate-blocked on ONLINE with the command recorded.
Suite rc=0 read separately from this commit.
The absent-drive alarm was generic while its recovery was specific -- a pair an
operator cannot match. Mechanism now measured, not reasoned (felhom.eu
audits/DIAG-r116-disks-payload-2026-07-30.md): with the device gone /disks returns
4 rows, not 3. The drive appears TWICE and the two facts the controller needs are
on different rows -- the Observe row has backup_target:true but mount_path:"" and
guest_path:"" (so driveTargetByPath registers NO key from it), while the registry
row owns /mnt/felhom-drives/<name>, the key the gate looks up, with BackupTarget
absent from its struct literal => false.
WHY v0.115.0 WAS INERT: its fallback computed StablePathForRaw(t.MountPath), and in
the absent state MountPath is ALSO "" -- emptied by the same exactMount failure
that empties BackingDevice. It assigned nothing. Its test passed because the fixture
supplied a MountPath production never supplies, and the harness left DriveTargets
nil so the union loop never ran. Both corrected here; red-proof 1 replays v0.115.0's
exact code against the real shape and it fails.
THE JOIN, which was the hard part: with the device gone the two records share no
runtime field -- no mount, no backing device, and the Observe row's DurableID has
degraded off the fs-UUID. They share CONFIGURATION: storage.cfg's path on one side,
the .mount unit's Where on the other, both yielding the same stable guest path. New
hub.StorageTarget.ConfigPath (json:"-" -- that struct is a cross-repo contract
pinned by the golden + contract_test key-set comparison, and nothing off-box needs
the value), set from s.Path in observe.go, consulted in disks.go only after MountPath
so the present-state path is byte-identical, plus a guest-path arm on the union dedup
so exactly one row carries the drive.
WHY NEITHER OBVIOUS OPTION WAS TAKEN -- both regress R-114, which shipped yesterday.
backup_target_offer.go:79 reads (BackupTarget && MountPath != "") as "a real
drive with its own mountpoint -- healthy" and returns before its TargetAbsent
branch. Back-filling MountPath onto the Observe row (the smallest change, and the
spec's lean) and teaching the registry row the flag (its MountPath is non-empty, read
from the stale unit file) BOTH manufacture that row while the drive is missing, which
would have told the customer the backup target is fine while its drive is gone.
R-114's correctness rests on the absent-state rows not combining the flag with a
mount path; that coupling was invisible until the payload existed. Pinned by
TestAbsentTargetKeepsR114DegradedSignal.
Role unchanged, BoundUnderParent conjunction not widened, no wire field changed.
Suppressing the registry row in the absent state also removes its false
state:"attached" and its root-filesystem-derived total_bytes -- R-118's symptom
goes incidentally; R-118 is NOT fixed and stays open.
Tests 845 -> 849, suite rc=0 read separately from this commit. Four red-proofs, each
mutation asserted to have landed first.
NOT live-validated at this commit: publish+vouch, C5, discrimination, over-correction.
Both found while writing felhom.eu runbooks/target-selection.md.
1. The demo host block said the t740 is the designated drill+build VM host "but no
drill VM is provisioned there yet". Stale since 2026-07-25 -- VM 300 (drill-r50)
has been there since. The sentence read as discouragement from the very box the
operator ruling designates, which is part of why a drill went to DooPlex instead.
Now says the ruling is realized and to start there, points at the new
target-selection runbook, and notes drill.qcow2 on DooPlex is a BAKE fixture, not
a drill target. Agent version dropped (it changes several times a day; the hub
host list and --version are the authorities) and the t740's PVE node name added.
2. RETRACTED the block's claim that the agent "does not run at all" at the remote
site because localapi binds the LAN literal 192.168.0.162 and the service has
"never started" -- with an outstanding config edit needing Viktor GO. That was
true before R-50 and is false now: since the island migration (2026-07-25)
localapi binds 169.254.253.1:8443 on vmbr9, which is location-independent by
design, and proxmox.endpoint is https://127.0.0.1:8006. Verified live 2026-07-30:
service active, version 0.115.0, and GET /disks answered over the island -- the
whole R-116 payload capture went through it. A session trusting the old text
would not have attempted the read that worked. The recorded remote-site address
was also wrong (.162, not .147), so it now says re-check instead of asserting one.
No code, no version bump, so no CHANGELOG entry (that file is version-keyed) and
REPORT.md is left holding the v0.115.0 record rather than being overwritten by a
docs fix.
Session C measured it live: a drive whose device vanished raised the GENERIC
storage_disconnected while its return raised the SPECIFIC
backup_target_restored -- an alarm and an all-clear an operator cannot pair.
backup_target_absent never fired at all.
The mechanism is not what the Session-C audit first said, and the difference
decides the fix. RoleForStorage returns RoleSystem whenever backingDevice == ""
(internal/storage/role.go:180-181). When the device goes, exactMountDevice
fails, BackingDevice becomes "", the target row's role flips to system and it
loses its guest path -- but keeps its MountPath. The union loop skips any drive
whose MountPath is already seen, so the registry row is DEDUPED AWAY ENTIRELY.
/disks carries no row with that guest path, so isTarget[guestPath] is a MISSING
KEY, not a false. Setting BackupTarget on the union row -- the obvious fix --
could not have worked, because that row is not emitted when the alarm is needed.
The audit is corrected in the same push.
Fix: on the Observe row only, carry the guest path when the row IS the backup
target and its role flipped because the device vanished.
Three gates, verified not assumed:
- t.BackingDevice == "" restricts it to the vanished-device flip; a genuinely
system-BACKED storage has a real device and is excluded, so a dir storage at
/mnt/<name> on the root disk cannot acquire a guest path.
- Case B, the common fresh-box shape, is safe twice over: its target is the
builtin local on /var/lib/vz and StablePathForRaw returns "" for anything not
exactly /mnt/<name>, so nothing is set even before the gates apply.
- It cannot make the gate read an absent drive as PRESENT. BoundUnderParent is
assigned at exactly two sites, both inside guest-path blocks a system-role row
never enters, so it stays false and planDriveGates computes false || false.
Pinned by TestAbsentTargetRowDoesNotRegisterPresence -- getting this backwards
would have silenced the alarm the fix exists to raise.
The :213-214 boundary stands: no system or backup mount gains a guest path.
Tests +5, asserting the emitted /disks JSON through a faithful copy of the
controller's driveTargetByPath, because the failure class is "the value is on
the wrong row". Red-proof: removing the block fails with "isTarget[...] is a
MISSING KEY"; reverted byte-identical.
Filed not closed: the two-row shape that produced this survives.
BoundUnderParent, the one field the controller's drive-absent gate keys on,
reported only "is this path a mount target in the guest's mountinfo". The
drive's raw mount at /mnt/<name> is a systemd mount unit bound to its device and
dies with it, but the agent's own bind of <raw>/felhom-data under the shared
parent is an ordinary bind: nothing ties it to the device, so its mountinfo
entry OUTLIVES the device as a stale shell. Presence read that survivor as true,
planDriveGates never produced a Stop action, and nothing fired on any channel --
not backup_target_absent, not the generic storage_disconnected. Measured live in
E-2d: detached at 10:58:37Z, silent for 4.5 minutes while the agent itself
logged "enrolled drive absent by UUID" every 20s (felhom.eu
audits/E2D-fresh-vm-2026-07-29.md §5.2).
The fix: BoundUnderParent becomes a CONJUNCTION -- bound under the parent AND
the drive's raw host mount still mounted (devicePresent, new deviceCheck seam
defaulting to isHostMountpoint). Applied at BOTH /disks construction sites. The
union path matters more, not less: it hardcodes State:"attached", so the
raw-mount check is the only device truth that row carries, and it is exactly the
shape E-2d detached.
Why a conjunction and not a replacement: half 2 alone would regress boot
ordering, where the raw drive mounts early and the bind lands ~18s later; the
gate depends on that window reading ABSENT. The conjunction leaves that
byte-identical and closes only the case the gate could never see.
Unknown is never absent: devicePresent("") returns TRUE. A false absent stops a
working customer's apps -- the failure mode of this fix, not of the bug.
Controller UNCHANGED, no MinAgent bump. BoundUnderParent has exactly one
functional consumer (planDriveGates, intermediary.go:226); every other mention
in both repos is a comment or a test, and boot convergence deliberately moved
off it to pollLiveBinds/driveBindLive. The alternative -- a new DevicePresent
bool the controller ANDs in -- was rejected as dangerous: a bool absent from an
older agent's JSON decodes to false, so every drive on a pre-0.114.0 agent would
have read ABSENT and stopped its apps.
Tests +6 in internal/localapi (208 -> 214): groups A-D plus a wire-contract test
asserting the ENCODED bound_under_parent, since that is what crosses to the
controller. Four red-proofs run and reverted (drop the conjunction on each path;
invert unknown; drop the bind half); disks.go verified byte-identical after.
NOT LIVE-VALIDATED. No drive was pulled. Leg awaiting Session C: device loss ->
gate Stop -> SetDisconnected -> backup_target_absent on the wire.
The agent cannot create a PVE storage (Datastore.Allocate at /storage) or grant
an ACL (Permissions.Modify) -- it holds neither by design, and widening the role
would trade the whole blast-radius containment model for one feature. The
privileged half therefore lives in a new fenced shim behind a literal
FELHOM_BACKUPTARGET sudoers alias, following the mkfs/pbs-apply pattern.
The wrapper enforces the two laws E-1 paid for on live hardware so no caller can
forget them: F-1 the path must BE the drive's own mountpoint, F-2 is_mountpoint 1
is hardcoded rather than a caller flag. It refuses a root-device target, has NO
storage-removal path of any kind (the pbs-apply no-delete law, grep-assertable),
is idempotent for the same path, and REFUSES to repoint an existing id.
POST /backup/target drives it in a fixed order: create -> grant -> config.
Reversed, a config pointing at an ungranted storage 403s every backup on first
run -- exactly E-1 finding F-3. A failed grant leaves the config untouched.
It deliberately does NOT restart the agent: restarting with a backup in flight
cancels the wait and records a spurious tier failure for a backup that actually
succeeded (E-1 did this to a real felhom-pbs run). It returns restart_required
and the caller restarts behind its own immediate in-flight check.
Config rewrite preserves unknown keys verbatim and writes in place, since
/etc/felhom-agent is root-owned while agent.json is agent-owned 0600.
Green gate: build + vet + test rc=0 (29 packages), run separately from this commit.
Additive backup_target field, true for the drive backing the PRIMARY tier.
The controller cannot work this out itself: settings.StoragePath.BackupTarget is
customer INTENT, and on the two boxes migrated by hand in E-1 that intent was
never recorded -- intent is empty while the drive really IS the target. Without
this flag the absent-target alarm could not name the drive on exactly the boxes
that currently have one.
omitempty + false on an older agent, so an old controller degrades to the generic
disconnect alarm rather than a wrong one.
Test asserts the target IS flagged AND the non-target is NOT, as a pair -- a
blanket true would satisfy a naive one-sided check.
A regression guard on a configuration that is live right now. E-1 moved each
demo box's whole-guest vzdump target onto its secondary drive at that drive's
own mountpoint -- but RoleForStorage types a local-dir on a non-system device as
user-data, so the existing eject role gate PASSED it. POST /disks/eject on
/mnt/nvme-1tb (demo-hp) or /mnt/hdd_1 (demo-felhom) would have SUCCEEDED
silently, taking the only local whole-guest backup with it, with no alarm and
the box still reporting a configured tier. Found by E-2 Phase 0, not by a
failure.
handleDiskEject and handleDiskDecommission now call refuseIfBackupTarget AFTER
the role gate and refuse with 409, naming the storage and the remedy -- the op
is ordered, not forbidden: reassign the target first.
NOT a role reclassification, which is the obvious fix and the wrong one: making
RoleForStorage return RoleBackup would refuse every legitimate eject of the
customer's own data drive, because on both demo boxes that drive IS the target.
That trades a silent failure for a permanent obstruction.
backupTargetAt resolves through the agent's own storage view, never the caller's
claim, and fails OPEN -- safe because it sits behind the role gate, which fails
SAFE on the same error.
Red-proofed both ways, mutations verified to land first:
- removing the eject guard -> "eject of the backup-target drive SUCCEEDED (200)"
- the over-correction (any backup-content dir storage is the target)
-> the gate blocks /mnt/spare, failing TestEjectStillAllowedOnANonTargetDrive
Harness note: normalizeBackupTiers DROPS tiers with a nil Service and falls back
to the legacy empty-TargetID tier -- an earlier version of this test exercised
nothing and reported the production bug as if the fix had failed.
Green gate: build + vet + test rc=0 (29 packages), run separately from this commit.
The per-VM ACL is consumed by the destroy it authorises (PVE remove_vm_access,
LXC.pm:906), so it works once per slot. Fourth root-fenced exception, band-enforced in
sudoers literally + in code + at the caller. API destroy still tried first.
The v0.107.0 watchdog was silent on a healthy box, so its health could only be inferred
from absence — F-OBS's shape, shipped in the same session F-OBS was fixed. INFO summary
every 10th sweep with what it saw; aborted sweeps are not counted. Red-proofs 7 and 8.
PUT /pools/{pool} ALSO requires VM.Allocate on the VM being added, so Pool.Allocate
cannot bootstrap its own membership. Proven live on demo-hp 2026-07-28. The real fix is
felhom-host-install v1.21.0 granting FelhomAgentGuest at /vms/990000..990009.
F-REBOOT — a guest rebooted mid-backup never came back (fault 11: 9m47s of total
appliance outage, no lock, nothing retrying). The existing stale-lock recovery is
correct but missed it two ways: its predicate needs a stale vzdump lock and that
guest was unlocked, and it runs only at agent startup. New periodic guest-power
watchdog acts on 'should be running, is not, is not locked'.
onboot is the should-be-running signal, not invented here: stalelock.go already
uses it for this same decision, it is 0 on scratch/golden, and pve-guests uses it
at host boot. Guards: onboot:0 never touched (Scenario B), a locked guest is left
to the stale-lock path, a guest with a vzdump in flight is left stopped,
unprovable ownership acts on nothing, unconfirmable backup state fails safe.
Bounded retry 3x at 1/2/4m then ERROR (Scenario C) — a healthy start takes ~25s.
F-LEAK — a failed restore-test could not destroy its scratch (403 VM.Allocate).
It is pool membership, not privsep: VM.Allocate is granted at /pool/felhom only,
and a failed restore never completes the --pool association. Fix needs NO new
grant — Pool.Allocate is already held, so the teardown adopts the stranded
scratch into the pool and retries the destroy. Guarded by scratchAdoptAllowed:
scratch provenance AND the numeric band, both required (Scenario E).
Six red-proofs across both fixes, all observed failing.
admin