Yesterday's drill proved a retained escrow package opens a set-aside store and
restores planted files byte-identical, while this agent answered the customer's
correct code with "the recovery code did not open the sealed bundle". Nothing had
ever tried the retained packages, so a correct-but-earlier code and a mistype were
genuinely indistinguishable.
OffsiteKeyRecoverer gains an optional FetchRetained, consulted ONLY after the
current package refuses, so the ordinary recovery pays nothing for it and cannot
fail because of it. A match returns ErrCodeOpensRetained wrapped in a
RetainedOpenedError carrying the supersession date - no material, no code, no
password. The local API answers 422: a FIFTH status added to the R-224 switch,
never a restructuring of it.
Fail-safe in every direction. Nil fetcher, a hub too old for the route (404 is a
clean "none"), a transport failure, a malformed package: each leaves the original
refusal standing. Attempts bounded at 6 because each unwrap is ~1s of scrypt.
Seven tests with REAL age crypto - the two situations are indistinguishable AT
THE UNWRAP, so a faked unwrap would prove nothing. Red-proof asserted applied:
remove the retained lookup and the fail-closed wrong-code error returns, which is
the lie in those exact words.
Gates only. No release, no version bump, no binary published; the agent stays
v0.128.0 at 28ba8593b8 and nothing on a customer's machine changes.
THE COUPLING DEFECT. The registry stopped serving 0.120.0 and older while
check-published-versions.py demanded every tag still be downloadable. Both rules
are sensible and together they are impossible, so CI went red at a commit whose
own run had been GREEN the day before -- and would have gone red again at the
next publish when 0.121.0 was evicted. scripts/retention-policy.json is now THE
number and both readers take it from there.
WHAT CI NO LONGER COVERS, and it prints this on every run rather than leaving it
to be discovered: a released version older than the retention window is no longer
asserted downloadable. Its git tag and its config tree ARE still asserted -- only
the binary's presence is dropped. A missing policy file is INCONCLUSIVE (exit 2),
never silently unbounded.
THE NUMBER IS NOT A LOCATED RULING and the file says so in its own header. Ten is
what the registry demonstrably holds; no register row records a prune, R-210 says
"Nothing was deleted; this is a list, not an action" and concerns local Docker
images, and container packages hold 19 each. The principled bound is the hub's
vouched min_agent floor -- nothing can install below it -- and that is the
recorded follow-up.
check-release-complete.py is the tag half as a machine. release-agent.sh already
warned that "a released version without a git tag 404s a box mid-install, as
root" and the step was still missed, so this is a gate and not a reminder. Legs
1-2 need no network and run in --fast, so the pre-push hook is the earliest
catch. Red-proved by repointing the CHANGELOG head at an unreleased v0.129.0:
both legs convicted and each named its fix command.
Three controls run: green at 10 naming what it dropped; widened to 11 the evicted
version re-enters and convicts; policy removed gives INCONCLUSIVE naming the path.
A rebuilt box could not run the escrow ceremony AT ALL, with no way forward from inside the product.
This was the only open item blocking a customer from something we promise them.
MECHANISM, established at file:line rather than assumed. The preflight refuses on
escrow.pbs_storage_id; the pbsdr bridge writes that key; and it wrote it from exactly one place —
finishConverged, reached only on the paths that actually converge.
The marker and the key live in different places and die at different times. The marker is host-side
(<agent-state>/pbsdr/marker.json). The key is in agent.json, which step_agent_config renders from
`base = {}` unless an explicit --preserve-from is given (felhom.eu/scripts/felhom-host-install.sh:
2396 the step, :2449 the render, :2579 the O_TRUNC write; the flag :1246, defaulting empty at :256)
— AND THE RENDER NEVER WRITES AN escrow SECTION AT ALL (grep over the whole heredoc: zero hits). So
a rebuild keeps the marker and takes the key: same descriptor, same hash, early return, and the seed
never runs again into a config that no longer has it.
A rebuild is only the case that was measured. The same hole opens for a hand-edited or restored
config, which is the honest reason this is a seam fix rather than an installer fix: the seed must be
a thing the loop ASSERTS, not a thing it did once.
Apply now re-asserts the seed BEFORE the idempotent early return. seedEscrowStorageID is unchanged
and still never clobbers a different existing value — an operator's own choice outranks the
descriptor's, with a warning naming both.
THE EARLY RETURN IS KEPT. It stops a converged box re-running Proxmox operations every 60s, and
TestSeedReasserted_OnConvergedTick_WithZeroProxmoxCalls asserts ZERO recorded runner calls on that
tick, so a "fix" that simply deleted the return would fail. Cost: one small file read plus a JSON
parse per tick, no exec, no network, early-returning once the value matches.
A seed failure can never un-converge the box: Warn plus a message on the published status, exactly
as finishConverged does it — no marker write, no state change.
Tests drive the REAL Apply with a real temp-dir agent.json and a call-recording runner; calling
seedEscrowStorageID directly cannot see the early return, which IS the defect. Production wiring
(pbsdr.NewManager(..., cfg.SourcePath, ...)) is asserted by walking main.go's AST, not by
strings.Contains, which a commented-out call also satisfies.
Red-proofs, each with the mutation asserted applied: removing the new call makes Scenario A fail on
today's tree (it did, with the intended message); removing the early return makes the
zero-Proxmox-calls assertion fail (it did).
go build / go vet / go test ./... green (29 packages), run separately from this commit.
Comment-only; no behaviour, no wire change, no version bump, nothing to rebuild.
HostReport.SelfUpdatePending / SelfUpdatePendingVersion carried "The hub reads an absent field as
pending=false, the correct default." The hub has NO FIELD for either, so it reads nothing — present
or absent — and encoding/json discards them on arrival. The sentence described an intent rather than
the code and read as settled long enough that a class sweep had to find it.
The emission is correct and stays: the agent reports the truth and the fault is entirely in the
receiving. The missing consumer is R-264 (OPEN). felhom.eu/scripts/wire_contract_gate.py now refuses
any NEW field of this shape and records the existing ones as reasoned allowlist entries.
After a rebuild the customer's own drives could not be re-attached: candidates
returned initialize:[] attach:[] while both drives sat there, and the deploy
refused with 'choose an attached drive from the list' — a list that was empty.
Measured live three times.
Mechanism: enrolment mounts a drive TWICE, at /mnt/felhom-drives/<name> and at
the raw /mnt/<name> it creates on the host. The host survives a guest rebuild;
the controller's registry does not. So classifyClaim saw a mount outside the
managed prefix and concluded 'claimed by something else' — about our own mount.
The fix is CORROBORATED, not a widened prefix: a non-managed mountpoint is
forgiven only when the SAME device is also mounted under the managed path, a
pairing only our enrolment produces. A disk another system uses — /srv/data,
/media/x, even /mnt/someone-elses-disk — has no counterpart and is STILL
refused, with its own test and a red-proof showing an over-wide fix offering it
for formatting.
Read from /proc/mounts deliberately: the lsblk invocation is pinned verbatim in
configs/felhom-agent.sudoers, so using the plural MOUNTPOINTS would have coupled
this to a sudoers rollout. /proc/mounts is world-readable — no sudo, no new
allowlisted command, no config change.
Fail-safe: an unreadable mount table corroborates NOTHING, so the device
classifies exactly as before. 'Could not corroborate' must never read as 'ours'.
29 packages ok, vet clean, agent gates OK.
175 -> 99 effective lines. New .claude/rules/{proxmox,localapi,backup,storage}.md alongside the
existing health-checks.md. The release section points at the felhom-build-deploy skill rather than
restating a table that drifts from the script; the layout section's per-package annotations moved
into the rule file for their area instead of being deleted.
Kept in the core because it is the only part re-injected after /compact: the root-CLI fence and its
three exceptions, the destructive-op gate, prove-ownership (audit A1), the gate entry point, the F9
live-validation fence, and the checklist.
health-checks.md overlaps localapi.md and storage.md on three globs -- deliberate, both load,
stated in each file. Go build/vet/test green and unchanged.
Corrected in all four instruction files across all four repos. Found while confirming this
session own push by run ID, which is precisely the check that catches it.
In felhom-agent/CLAUDE.md the sentence contradicted the same file release section, which
already said R-168 mails the failure -- a contradiction inside one instruction file, the exact
class the R-229 work exists to find.
REPORT.md deliberately NOT overwritten in the sibling repos: a one-line docs correction must not
destroy the record of their last real implementation.
Surgical corrections only; the file is deliberately NOT restructured (deferred).
Deleted the expired TEMPORARY block. It read "felhom-pve is at a remote site
(until ~2026-08-02) ... Delete this block on return" and was still being read as
current fact on 2026-08-06, four days past its own deadline, while
felhom-controller/CLAUDE.md asserted the opposite. The location-independence fact
worth keeping (localapi binds 169.254.253.1:8443 on vmbr9 since the R-50 island
migration) moved to an HTML comment.
Every component version literal is gone from effective text, including the
--version reading and the go.mod Go directive. Versions change several times a
day; ask the hub's /hosts + /configs or the box.
The drill-VM claim and the host addresses now point at
documentation/operations/nodes.md, which already stated both correctly. This
file's drill-VM claim was the correct one -- confirmed by qm list on demo-hp.
The R-115/R-188/R-186 release narratives moved to an HTML comment and to the
felhom-build-deploy skill; the directives stayed (never hand-roll the build; the
build -> tag -> publish -> push order; reproducible -trimpath -buildvcs=false).
The health-check block-I/O rule became .claude/rules/health-checks.md, scoped to
the five packages where health checks are written. It had been duplicated from
felhom.eu/CLAUDE.md with a note explaining that that file does not load in an
agent-only session -- correct reasoning, made obsolete by path-scoped rules.
agent_gates.py registers the shared instructions gate.
Docs only -- no Go, no version bump, nothing built or deployed.
Ledger: felhom.eu/documentation/audits/LEDGER-instruction-trim-2026-08-06.md
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JJc8sAGRWmavP3rMtdpkr2
A hub the agent could not reach was reported to the customer as a bad recovery
code. Measured live 2026-08-05 (CAMPAIGN-11 F3): hub firewalled off, a CORRECT
current code, and the customer told it did not open their package — in 0.0556s
against ~1.0s for a real unseal. No unseal was attempted.
The discriminator existed here and this boundary threw it away: recover.go
fails at four distinguishable points and the local-api handler had cases for
two, with a default answering 'the recovery code did not open the sealed
bundle, OR the bundle could not be fetched'.
escrow.ErrBundleFetch now joins the fetch leg and the handler routes it to 502
with its own words — the code was NOT used. 502 not 4xx: the request was not
bad, an upstream dependency failed. Four situations, four statuses: 502 fetch /
400 fetched-and-refused / 404 no bundle / 409 predates the field. The
controller classifies on the STATUS and never parses the sentence.
A GREEN TEST NAMED THIS DEFECT AND DID NOT PREVENT IT.
TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct has said since v0.125.0
that the operator must not be sent to re-read their code because the hub was
unreachable — and passed throughout, because it asserted this package's error
STRING one layer below the merge, and a string is not something a caller can
branch on. Re-pointed at the sentinel, with a consequence-level twin asserting
the status.
Red-proofs: removing the %w join fails the sentinel test; deleting the handler
case makes fetch and wrong-code both answer 400 with the wrong-code sentence.
29 packages ok, vet clean, agent gates OK.
Link 7's only production caller was a --selftest reading R from an env var. Link 8 did not
exist: that selftest writes the whole bundle JSON and its success message named
"tunnel_token + pbs_token" -- accurate when written, a misstatement since v0.77.0 sealed the
offsite repository password into the same bundle. It now names what THIS bundle carried and
what it did not.
POST /escrow/recover-offsite-password: the controller supplies R, the agent fetches this
host's own blob from the hub (self-scoped by the per-host key), unseals it, and returns ONLY
the offsite restic repository password plus its sha256. Not the tunnel token, not the PBS
token, not the WG key -- the controller is a trust tier down and needs none of them.
R: in memory for one call, cleared on every path, never on disk, never in argv, never logged,
never echoed. A test redirects TMPDIR and asserts the tree is EMPTY afterwards -- emptiness
rather than a content scan, because a content scan is defeated by a later call overwriting the
leaked file, which is how the first version of that test passed its own red-proof while R sat
on disk.
Three distinct outcomes: no blob (404), a bundle that opens but predates the field (409), a
code that does not open it (400, fail-closed at the KDF, nothing written).
The wiring is asserted by an AST walk from func main() to the Options field, not by grep.
v0.124.0's transition record never reached the hub, and only the live run showed
it. The capability reported degraded for "one cycle" — the probe call that did the
repair. But probeAll is invoked independently by the self-check log and by the
collector building a host report. On the demo box the repairing call was the log's
(09:39:34, journal shows the self-repair and degraded=1) and the report three
seconds later found the grant present and sent ok. The agent's journal had the
record; the hub had nothing. That is the silence R-190 is about, re-created inside
its own mitigation, with every unit test green.
Fixed with a latch on TIME, not call count: a confirmed repair reports for 20
minutes, which exceeds the 900s report interval, so at least one report must carry
it. It clears on its own and is per tier.
Two hollow tests caught and fixed on the way — one asserting a value it built
itself, one asserting the latch helper rather than the path consuming it (its
red-proof duly passed). The decisions now live in storeGrantHealthyVerdict and
storeGrantRepairedVerdict and the tests call those.
R-190 is a grant that worked at 04:44 on 2026-08-03 and was gone by 09:24, with a
reinstall, logged pveum activity and cluster-log entries all ruled out. The cause
is open; the resilience need not wait for it.
Everything needed already existed and had only ever been called once: the root
wrapper's `grant` verb, its sudoers vector (`grant *`, any storage id — confirmed,
not assumed), and the exact command. The verb had only ever run at storage
creation — the "built but never wired" shape in a verb rather than a seam.
The probe now runs that wrapper on a missing grant and re-reads ONCE to confirm,
the pbsdr R-22 shape including its restraint.
The record is the half that matters. A repair leaving only "ok" behind destroys
the only evidence a permission vanished, so a recurring loss becomes undetectable
— worse than the fault. A confirmed repair therefore reports DEGRADED for exactly
one cycle with the explanation in Feature, because that is the field the hub puts
in the operator's email (Reason does not travel). Nothing new was built: the hub's
existing ok->degraded->ok edge is the channel, so one loss produces one alert pair.
No wire change, no hub change, no new event type.
Bounded at one attempt per tier per hour: a storage can be unreadable for reasons
an ACL cannot fix, and re-granting every cycle is a repair loop wearing a fix's
clothes. A failed repair never masks the fault.
The missing grant is one command; the silence was the defect. On demo-felhom the
agent's token has FelhomAgentStore on local, local-lvm and felhom-pbs — and not
on felhom-backup, the storage the same installer configured as
local_backup_target. That storage answers {"data":[]} through the token while
root sees three archives.
An empty listing is what a FORBIDDEN tier and a NEWBORN tier both return, so
pickForThisRun skipped it as "no settled archive yet" and the tier was never
restore-testable on that box. The permission question, unlike the listing, has a
definite answer, so it is asked directly: Client.Permissions reads
/access/permissions as the agent's OWN token, and one capability.Status per
configured tier reports it — composed around the sudo prober, the way the
pool-read check already is.
Measured first, because the obvious reading is wrong: an ungranted path answers
neither empty nor 403, but with the privileges inherited from the box-wide grant
(Sys.Audit, SDN.Use, Datastore.Audit). Checking for Datastore.Audit would report
a blinded storage healthy — red-proved. The probe tests for
Datastore.AllocateSpace.
The probed set comes from the box's own config, never a fixed list: a hardcoded
probe list is the defect reproduced inside the fix. Critical, because the hub
alerts only on critical — except the "local" fallback target, which is reported
but does not page. It never looks at content, so it cannot alarm on a newborn
tier; it never reports ok when it could not ask. Status wire shape unchanged, so
no hub change.
All three are the reporting and release path misreporting its own work. No
customer machine, no backup, no restore, no data. The restore-test itself and
when it runs are unchanged.
R-189 — a passing restore-test no longer vanishes on a restart. restore_tests[]
came only from the in-memory store, whose comment ("lost on restart; the cadence
re-populates") was true under a timer and stopped being true when R-86 made the
agent refuse to re-test a proven archive: the proof is then not repeated for a
whole archive generation. Observed live — a 14.5 GB offsite PASS reached no
host-report because the agent was restarted 2m43s later. RestoreTestState now
carries tier + verified beside the archive and renders reportable entries; the
collector merges them, one per tier, newest by TestedAt. It refuses to lie: a
record missing archive-or-tier produces no entry, and run mechanics are not
re-invented. Only successes are persisted, and the asymmetry is now written where
it will be read.
R-188 — a correct release stops emailing a failure. Only the tag PUSH moved
(build -> tag locally -> publish -> push tag): the push wakes CI, and a tag
visible before its package made the gate correctly fail a correct release about
half the time. The old order's invariant is asserted directly instead — the gate
now refuses a published version with no tag, as a bounded probe that prints its
own coverage, because the package listing api is still 401 without a token.
R-186 — a released binary can be verified by rebuilding it. -trimpath
-buildvcs=false: same source, same bytes, tag or no tag. Measured. publish-agent's
fallback also forced CGO_ENABLED=0 and produced a 74 KB different binary for the
same version; both paths now build identically. CLAUDE.md records the command.
Before R-86 every tick ran a heavy restore-test, so the scheduler was audible by
construction. After it, 'nothing is due' is the NORMAL outcome — and it was
logged at DEBUG, which journald drops. An empty journal would then be equally
consistent with a healthy loop and a dead goroutine: the shape the R-88 watcher
was retired for, re-created by making the quiet path the common one.
A not-due evaluation now logs one INFO line naming every tier's verdict (four
lines a day at the 6h default), and an unlistable tier reads UNKNOWN with its
error in that same line, so a lookup failure can never present as 'nothing due'.
Red-proved through the scheduler's own tick, not the helper.
The ticker survives as the EVALUATION interval only. A tier is DUE when its
newest archive that has settled for `settle` (default 24h) has not been proven:
daily tier -> proved daily on yesterday's archive, weekly tier -> weekly on its
own, newborn -> UNKNOWN.
The trap avoided: the literal reading ("newest archive is >= 24h old") is NEVER
true on a daily tier, so it silently switches restore-testing off where it
matters most. Red-proved at 0 runs over 5 simulated days.
- state records WHICH archive was proven; legacy files keep their time and yield
no proven archive (each tier due once after the upgrade, deliberately)
- two knobs replace one: restore_test_eval_interval_seconds (6h, measured) and
restore_test_settle_seconds (24h). The old cadence key keeps its DISABLE
meaning verbatim and now seeds the settle lag, with a start-up WARN.
- due-check runs BEFORE the heavy-op gate (a frequent poll must not make a
starting backup record a failure, F-A1)
- candidate picker skips implausible archives (a phantom would be due forever)
- new read-only --selftest=restore-test-due prints the verdict + its cost
NO VERSION BUMP and nothing built: no Go code changed. The agent stays v0.120.0.
scripts/release-agent.sh — THE way to release. build -> tag -> publish -> verify
by INDEPENDENT download. Publishing was a separate remembered step and was
forgotten three times in five days (R-111's 17 stranded releases, 0.114.0, and
0.120.0 — deployed to both demo hosts and undownloadable, so a documented-path
reinstall would have silently downgraded them WHILE REPORTING SUCCESS). R-111's
own closing line named this leg and closed SHIPPED without it; it recurred the
same afternoon, which is the evidence that a note is not a mechanism.
It tags because felhom-host-install.sh now fetches the sixteen agent config
files from raw/tag/v<version>/ (R-183): a released version with no tag 404s a
box mid-install, as root, on a virgin machine. It verifies by downloading what
it just published and comparing the sha to what it built — the publish step's
own success is a report on its own write; a fetch returning the right bytes is
a different claim. It refuses a dirty/unpushed tree and refuses to re-release an
existing version. It does NOT vouch: that points machines at a version and stays
the operator's act.
scripts/check-published-versions.py — the gate. Every v<semver> tag must have a
downloadable package AND a tag tree serving the agent's configs. Registered as
NOT --fast (needs network; a push must not fail because Gitea blinked), and the
CI workflow now runs the FULL gate set instead of --fast — otherwise the gate
would have been registered and never run, the built-but-never-wired failure this
project has shipped four times.
The invariant is not the one specified, and the reason was measured, not assumed:
the hub artifact manifest is 401 without a per-customer passphrase and Gitea's
package LISTING api is 401 without a token, while the package DOWNLOAD url and
the git TAGS api are anonymous. So CI cannot ask "what is vouched" without an
operator credential — whose addition is the operator's call. The tag-based
invariant needs none and catches all three recorded instances. What it does not
catch (the hub vouching a version never released at all) is filed as R-184.
v0.120.0 had been built, committed and deployed to both demo hosts but never
published: the Gitea generic package 404'd and the hub manifest vouched 0.119.0.
Installer step 5 skips only on an exact version match, so a documented-path
reinstall would have downgraded both boxes to the pre-merge agent -- and would
have succeeded, since step_grows passes -sysdata-grow 0 and 0.119.0's mp1 resize
never fires. Published from a clean tree (upload 201, round-trip GET verified,
sha a7763d31b55b5ce7...) and vouched; both reinstalls then fetched and verified
it over the real customer path.
Filed as the third instance of R-115. No version bump, nothing built.
Evidence: felhom.eu/REPORT.md
build-golden.sh 2.1.0 -> 3.0.0: a layout change is a major. The golden ships
ONE data volume at a NEUTRAL path (/var/lib/felhom); both /var/lib/docker
and /mnt/sys_drive are binds of subdirectories of it. mp1 is gone.
The variant was chosen on measurement. Three candidates were built and
rebooted (SPIKE-r165-phase0-2026-08-03.md); all three boot, reboot 3/3, give
ONE df figure and keep a container's statfs(/) on the merged volume — the
ordering worry that motivated the probe did not materialise. They differ
only in which guarantee they break: volume-at-docker puts customer backups
inside Docker's data-root; volume-at-sys_drive puts Docker's ENTIRE
data-root under /mnt, which the controller container mounts wholesale
(measured: it then sees /mnt/sys_drive/docker). V-c breaks neither.
The four assertions were RETARGETED, never deleted, and each was RUN against
a deliberately wrong shape — a real split guest and a real archive of it:
8 checks, 8 passed. A new 2b asserts both paths are ONE filesystem, which
catches the S2 shape the spike ranked worse than the split. Assertion 5
replaces the old "was mp1 excluded?" guard, whose pattern could no longer
match — a guard that cannot match has silently stopped guarding.
Provisioning: one volume, one grow. SysDataGrowGB is FOLDED IN rather than
dropped, because a census established that felhom-host-install.sh passes
-sysdata-grow and the two do not upgrade in the same instant; dropping it
would silently shrink every appliance by 42 of 250 GiB. The flags stay
accepted for the same reason. The existing test was retargeted to pin the
fold, and it caught the change before I did.
A managed box's IP was invisible in every operator surface because nothing
reported one: HostMetrics carried node/cpu/mem/disk/load/uptime/temp/wrapper-sha
and no address of any kind. The hub could not show a host's LAN IP anywhere.
Two things that looked like the answer are traps, both checked before writing
code: lan_resolver.host_ip is an OPTIONAL config value absent unless that feature
is configured, and DeriveHostIP(local_api.listen_addr) returns 169.254.253.1 —
since R-50 the local API binds a link-local address identical on every box. Both
would have produced a confident wrong answer.
New wire field addresses[], one entry per (interface, address). Deliberately
iface+cidr rather than a single lan_ip: a Proxmox host legitimately holds several
(management bridge, tailnet, WG tunnel) and picking one to call "the" LAN IP is a
guess the agent is not entitled to make — silently wrong on a box whose bridge is
not vmbr0. The agent reports what exists; the hub does the labelling.
The filter is one predicate, chosen by MEASURING both demo hosts rather than by
reasoning about interface names. IsGlobalUnicast() alone drops loopback, IPv6
link-local (one per bridge, pure noise) and IPv4 link-local (169.254/16 — exactly
the island address above). It needs no veth/fwbr/tap denylist: that per-guest
plumbing carries no IP at all and self-excludes, verified on both boxes.
No new privilege and no block I/O — net.Interfaces() is a netlink/procfs read, so
the sudoers fence is untouched and the health-check rule is honoured.
The seam DEFAULTS to the real enumerator, inverting the nil-reporter-means-off
convention: this stanza has no config gate, so a forgotten wiring call would have
shipped it silently empty — the inert-seam failure recorded four times here.
Cross-repo: the golden is duplicated byte-identically in felhom.eu and the
contract test fails on top-level key drift, so both goldens moved together and
addresses[0]'s key set is asserted bidirectionally. The field marshals as [],
never null — the repo's own no-nulls invariant caught that on the first run.
Tests +9; three red-proofs (global-unicast filter, down-interface guard, inert
collectAddresses) each run, observed failing, and reverted.
Live validation caught what the tests could not. On demo-felhom the recipe read
namespace "root" with namespace_state "resolved" — confident and wrong, a worse
shape than the original defect.
mergeConfig overlays the cluster storage config onto the node entry through a
hand-listed set of fields and Namespace was not among them. NodeStorage does not
return the namespace at all, so PBSNamespace always read "" and latestPBSCoord
correctly treated that as the root namespace.
Every v0.118.0 test built StorageTarget values directly — including the two
through Collector.Collect(), which inject a fakeObserver — so nothing crossed the
merge. Two new tests drive the real Observe path with PVE's actual split returns
and table the merge itself. Red-proof: dropping the added line fails both.
Suite rc=0, 29 packages, 0 FAIL.
Both defects were live on both demo boxes: the recipe said namespace "root" while
storage.cfg said demo-felhom/demo-hp, and it never named which of two content=backup
dir storages holds the local archives.
R-106: the namespace came from the listed snapshot, but PBS omits `ns` per item once
the list is namespace-scoped, so it was always empty and normalised to "root". It now
resolves from the pbs STORAGE (storage.cfg's `namespace`) — the same field vzdump makes
PVE read, so the recipe cannot disagree with the backup.
R-109: backup_target resolves from the primary tier of cfg.Backup.BackupTiers(), the
function the scheduler consults, and carries the mountpoint that separates /mnt/hdd_1
from /var/lib/vz. The resolver reports the tier IN EFFECT (daemon-start config), not
agent.json on disk — a target move rewrites the file and deliberately does not restart.
Unresolvable is recorded as unresolvable: resolved|unknown plus a distinct reason,
never a default, an empty string, or a placeholder.
Needs hub v0.83.0 — AssembleDRRecipe allow-lists top-level keys, so backup_target
would otherwise be stored intact and dropped before any operator saw it.
9 tests, 4 red-proofs (each mutation asserted to have landed). Suite rc=0, 29 ok.
BoundUnderParent reported a namespace that returned EIO on every read and write
as healthy, and the gate restarted the customer's apps onto it. Both existing
terms parse a mountinfo line and then test only fields[4], the mount POINT.
Field 3 — major:minor — sat in the same parsed slice and was discarded.
Measured on hardware: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb with `shutdown`,
bound_under_parent true, EIO both directions, and the controller taking its
Return branch and emailing backup_target_restored with no alarm on any channel.
BoundUnderParent gains a third term at both /disks construction sites. The new
bindLiveness reads /proc only and asks two questions: the bind must name the
same device as the raw mount, and the filesystem must not have aborted (ext4
`shutdown` or `emergency_ro`).
The second check is not optional. A device that fails WITHOUT disappearing gives
the identical all-signals-healthy state with the devnos EQUAL and the drive never
Disconnected, so the gate produces neither a Stop nor a Return and nothing is
emitted on any channel, indefinitely (R-117a). A devno-only fix would have passed
every payload test.
Three states, never a bool: {Unknown, Live, StaleDevice, Aborted}, read through
Usable(), where Unknown counts as PRESENT — reporting absent stops a working
customer's apps.
No new recovery path; the existing one was unblocked. AttachDrive's normalize leg
already did the repair and three call sites already invoked it, including the
controller's Return branch before it restarts apps. All three died on
`if n == 1 && GuestSeesMount(...)` returning early. Now: StaleDevice ⇒ re-bind
(repairs live, guest never restarts); Aborted ⇒ quiet no-op, because a re-bind
lands on the same dead superblock and this runs every 20s — an infinite silent
retry that masks the state; it surfaces via BoundUnderParent=false instead.
Ordering trap caught by a test: reading the abort flag before comparing devices
classifies the real return state as aborted (its stale bind carries `shutdown`
too) and refuses the repair while still reporting correctly. The abort flag is
read off the RAW mount in the stale case.
Tests 849 → 863, 29/29 packages green. 6 red-proofs, each verified to have
landed. A hollow test was caught during them: the aborted fixture first used a
/dev/mapper device, for which RoleForStorage derives role=system — a system row
has no GuestPath, never runs the conjunction, and reports false by default, so
the assertion passed vacuously and no mutation could fail it. Found because RP1
failed to fail.
v0.116.0 validated on a fresh box: real day-0 from the v1.25.0 ISO on a nested PVE
on demo-hp (per runbooks/target-selection.md), agent installed unaided from the
vouched Day-0 manifest, drives enrolled through the real endpoints, device loss a
real hot-detach.
07:20:04 backup_target_absent (error) Cel meghajto <- TARGET, specific
07:22:34 backup_target_restored (info) Cel meghajto <- its matching pair
07:24:04 storage_disconnected (error) Adat meghajto <- NON-target, generic
07:25:34 storage_reconnected (info) Adat meghajto
All four reached the hub; gate fired in 3 s. Discrimination is proven NON-trivially
for the first time -- both prior runs had the target itself emit the generic event,
so their mirror proved nothing. Over-correction passes on a POSITIVE observable: 0
ABSENT lines and 0 drive events over 2m14s with both drives present, while two
RETURNED lines prove the gate was ticking rather than dead.
Caveat recorded, not a regression: the drill's controller was 0.185.1 from the
golden, which predates R-114, so its absent-state banner showed the old false
"backup is on the system disk" copy. The R-114 guard is pinned by unit test and by
the payload, but could not be confirmed on that box. Filed as R-120 -- the golden
is a release behind the deployed controller, which is R-115's class one layer up.
Teardown all three layers, including the hub: VM purged, storage removed with the
space measured back, hub records gate-blocked on ONLINE with the command recorded.
Suite rc=0 read separately from this commit.
The absent-drive alarm was generic while its recovery was specific -- a pair an
operator cannot match. Mechanism now measured, not reasoned (felhom.eu
audits/DIAG-r116-disks-payload-2026-07-30.md): with the device gone /disks returns
4 rows, not 3. The drive appears TWICE and the two facts the controller needs are
on different rows -- the Observe row has backup_target:true but mount_path:"" and
guest_path:"" (so driveTargetByPath registers NO key from it), while the registry
row owns /mnt/felhom-drives/<name>, the key the gate looks up, with BackupTarget
absent from its struct literal => false.
WHY v0.115.0 WAS INERT: its fallback computed StablePathForRaw(t.MountPath), and in
the absent state MountPath is ALSO "" -- emptied by the same exactMount failure
that empties BackingDevice. It assigned nothing. Its test passed because the fixture
supplied a MountPath production never supplies, and the harness left DriveTargets
nil so the union loop never ran. Both corrected here; red-proof 1 replays v0.115.0's
exact code against the real shape and it fails.
THE JOIN, which was the hard part: with the device gone the two records share no
runtime field -- no mount, no backing device, and the Observe row's DurableID has
degraded off the fs-UUID. They share CONFIGURATION: storage.cfg's path on one side,
the .mount unit's Where on the other, both yielding the same stable guest path. New
hub.StorageTarget.ConfigPath (json:"-" -- that struct is a cross-repo contract
pinned by the golden + contract_test key-set comparison, and nothing off-box needs
the value), set from s.Path in observe.go, consulted in disks.go only after MountPath
so the present-state path is byte-identical, plus a guest-path arm on the union dedup
so exactly one row carries the drive.
WHY NEITHER OBVIOUS OPTION WAS TAKEN -- both regress R-114, which shipped yesterday.
backup_target_offer.go:79 reads (BackupTarget && MountPath != "") as "a real
drive with its own mountpoint -- healthy" and returns before its TargetAbsent
branch. Back-filling MountPath onto the Observe row (the smallest change, and the
spec's lean) and teaching the registry row the flag (its MountPath is non-empty, read
from the stale unit file) BOTH manufacture that row while the drive is missing, which
would have told the customer the backup target is fine while its drive is gone.
R-114's correctness rests on the absent-state rows not combining the flag with a
mount path; that coupling was invisible until the payload existed. Pinned by
TestAbsentTargetKeepsR114DegradedSignal.
Role unchanged, BoundUnderParent conjunction not widened, no wire field changed.
Suppressing the registry row in the absent state also removes its false
state:"attached" and its root-filesystem-derived total_bytes -- R-118's symptom
goes incidentally; R-118 is NOT fixed and stays open.
Tests 845 -> 849, suite rc=0 read separately from this commit. Four red-proofs, each
mutation asserted to have landed first.
NOT live-validated at this commit: publish+vouch, C5, discrimination, over-correction.
Session C measured it live: a drive whose device vanished raised the GENERIC
storage_disconnected while its return raised the SPECIFIC
backup_target_restored -- an alarm and an all-clear an operator cannot pair.
backup_target_absent never fired at all.
The mechanism is not what the Session-C audit first said, and the difference
decides the fix. RoleForStorage returns RoleSystem whenever backingDevice == ""
(internal/storage/role.go:180-181). When the device goes, exactMountDevice
fails, BackingDevice becomes "", the target row's role flips to system and it
loses its guest path -- but keeps its MountPath. The union loop skips any drive
whose MountPath is already seen, so the registry row is DEDUPED AWAY ENTIRELY.
/disks carries no row with that guest path, so isTarget[guestPath] is a MISSING
KEY, not a false. Setting BackupTarget on the union row -- the obvious fix --
could not have worked, because that row is not emitted when the alarm is needed.
The audit is corrected in the same push.
Fix: on the Observe row only, carry the guest path when the row IS the backup
target and its role flipped because the device vanished.
Three gates, verified not assumed:
- t.BackingDevice == "" restricts it to the vanished-device flip; a genuinely
system-BACKED storage has a real device and is excluded, so a dir storage at
/mnt/<name> on the root disk cannot acquire a guest path.
- Case B, the common fresh-box shape, is safe twice over: its target is the
builtin local on /var/lib/vz and StablePathForRaw returns "" for anything not
exactly /mnt/<name>, so nothing is set even before the gates apply.
- It cannot make the gate read an absent drive as PRESENT. BoundUnderParent is
assigned at exactly two sites, both inside guest-path blocks a system-role row
never enters, so it stays false and planDriveGates computes false || false.
Pinned by TestAbsentTargetRowDoesNotRegisterPresence -- getting this backwards
would have silenced the alarm the fix exists to raise.
The :213-214 boundary stands: no system or backup mount gains a guest path.
Tests +5, asserting the emitted /disks JSON through a faithful copy of the
controller's driveTargetByPath, because the failure class is "the value is on
the wrong row". Red-proof: removing the block fails with "isTarget[...] is a
MISSING KEY"; reverted byte-identical.
Filed not closed: the two-row shape that produced this survives.
BoundUnderParent, the one field the controller's drive-absent gate keys on,
reported only "is this path a mount target in the guest's mountinfo". The
drive's raw mount at /mnt/<name> is a systemd mount unit bound to its device and
dies with it, but the agent's own bind of <raw>/felhom-data under the shared
parent is an ordinary bind: nothing ties it to the device, so its mountinfo
entry OUTLIVES the device as a stale shell. Presence read that survivor as true,
planDriveGates never produced a Stop action, and nothing fired on any channel --
not backup_target_absent, not the generic storage_disconnected. Measured live in
E-2d: detached at 10:58:37Z, silent for 4.5 minutes while the agent itself
logged "enrolled drive absent by UUID" every 20s (felhom.eu
audits/E2D-fresh-vm-2026-07-29.md §5.2).
The fix: BoundUnderParent becomes a CONJUNCTION -- bound under the parent AND
the drive's raw host mount still mounted (devicePresent, new deviceCheck seam
defaulting to isHostMountpoint). Applied at BOTH /disks construction sites. The
union path matters more, not less: it hardcodes State:"attached", so the
raw-mount check is the only device truth that row carries, and it is exactly the
shape E-2d detached.
Why a conjunction and not a replacement: half 2 alone would regress boot
ordering, where the raw drive mounts early and the bind lands ~18s later; the
gate depends on that window reading ABSENT. The conjunction leaves that
byte-identical and closes only the case the gate could never see.
Unknown is never absent: devicePresent("") returns TRUE. A false absent stops a
working customer's apps -- the failure mode of this fix, not of the bug.
Controller UNCHANGED, no MinAgent bump. BoundUnderParent has exactly one
functional consumer (planDriveGates, intermediary.go:226); every other mention
in both repos is a comment or a test, and boot convergence deliberately moved
off it to pollLiveBinds/driveBindLive. The alternative -- a new DevicePresent
bool the controller ANDs in -- was rejected as dangerous: a bool absent from an
older agent's JSON decodes to false, so every drive on a pre-0.114.0 agent would
have read ABSENT and stopped its apps.
Tests +6 in internal/localapi (208 -> 214): groups A-D plus a wire-contract test
asserting the ENCODED bound_under_parent, since that is what crosses to the
controller. Four red-proofs run and reverted (drop the conjunction on each path;
invert unknown; drop the bind half); disks.go verified byte-identical after.
NOT LIVE-VALIDATED. No drive was pulled. Leg awaiting Session C: device loss ->
gate Stop -> SetDisconnected -> backup_target_absent on the wire.
The agent cannot create a PVE storage (Datastore.Allocate at /storage) or grant
an ACL (Permissions.Modify) -- it holds neither by design, and widening the role
would trade the whole blast-radius containment model for one feature. The
privileged half therefore lives in a new fenced shim behind a literal
FELHOM_BACKUPTARGET sudoers alias, following the mkfs/pbs-apply pattern.
The wrapper enforces the two laws E-1 paid for on live hardware so no caller can
forget them: F-1 the path must BE the drive's own mountpoint, F-2 is_mountpoint 1
is hardcoded rather than a caller flag. It refuses a root-device target, has NO
storage-removal path of any kind (the pbs-apply no-delete law, grep-assertable),
is idempotent for the same path, and REFUSES to repoint an existing id.
POST /backup/target drives it in a fixed order: create -> grant -> config.
Reversed, a config pointing at an ungranted storage 403s every backup on first
run -- exactly E-1 finding F-3. A failed grant leaves the config untouched.
It deliberately does NOT restart the agent: restarting with a backup in flight
cancels the wait and records a spurious tier failure for a backup that actually
succeeded (E-1 did this to a real felhom-pbs run). It returns restart_required
and the caller restarts behind its own immediate in-flight check.
Config rewrite preserves unknown keys verbatim and writes in place, since
/etc/felhom-agent is root-owned while agent.json is agent-owned 0600.
Green gate: build + vet + test rc=0 (29 packages), run separately from this commit.
Additive backup_target field, true for the drive backing the PRIMARY tier.
The controller cannot work this out itself: settings.StoragePath.BackupTarget is
customer INTENT, and on the two boxes migrated by hand in E-1 that intent was
never recorded -- intent is empty while the drive really IS the target. Without
this flag the absent-target alarm could not name the drive on exactly the boxes
that currently have one.
omitempty + false on an older agent, so an old controller degrades to the generic
disconnect alarm rather than a wrong one.
Test asserts the target IS flagged AND the non-target is NOT, as a pair -- a
blanket true would satisfy a naive one-sided check.
A regression guard on a configuration that is live right now. E-1 moved each
demo box's whole-guest vzdump target onto its secondary drive at that drive's
own mountpoint -- but RoleForStorage types a local-dir on a non-system device as
user-data, so the existing eject role gate PASSED it. POST /disks/eject on
/mnt/nvme-1tb (demo-hp) or /mnt/hdd_1 (demo-felhom) would have SUCCEEDED
silently, taking the only local whole-guest backup with it, with no alarm and
the box still reporting a configured tier. Found by E-2 Phase 0, not by a
failure.
handleDiskEject and handleDiskDecommission now call refuseIfBackupTarget AFTER
the role gate and refuse with 409, naming the storage and the remedy -- the op
is ordered, not forbidden: reassign the target first.
NOT a role reclassification, which is the obvious fix and the wrong one: making
RoleForStorage return RoleBackup would refuse every legitimate eject of the
customer's own data drive, because on both demo boxes that drive IS the target.
That trades a silent failure for a permanent obstruction.
backupTargetAt resolves through the agent's own storage view, never the caller's
claim, and fails OPEN -- safe because it sits behind the role gate, which fails
SAFE on the same error.
Red-proofed both ways, mutations verified to land first:
- removing the eject guard -> "eject of the backup-target drive SUCCEEDED (200)"
- the over-correction (any backup-content dir storage is the target)
-> the gate blocks /mnt/spare, failing TestEjectStillAllowedOnANonTargetDrive
Harness note: normalizeBackupTiers DROPS tiers with a nil Service and falls back
to the legacy empty-TargetID tier -- an earlier version of this test exercised
nothing and reported the production bug as if the fix had failed.
Green gate: build + vet + test rc=0 (29 packages), run separately from this commit.
The per-VM ACL is consumed by the destroy it authorises (PVE remove_vm_access,
LXC.pm:906), so it works once per slot. Fourth root-fenced exception, band-enforced in
sudoers literally + in code + at the caller. API destroy still tried first.
The v0.107.0 watchdog was silent on a healthy box, so its health could only be inferred
from absence — F-OBS's shape, shipped in the same session F-OBS was fixed. INFO summary
every 10th sweep with what it saw; aborted sweeps are not counted. Red-proofs 7 and 8.
PUT /pools/{pool} ALSO requires VM.Allocate on the VM being added, so Pool.Allocate
cannot bootstrap its own membership. Proven live on demo-hp 2026-07-28. The real fix is
felhom-host-install v1.21.0 granting FelhomAgentGuest at /vms/990000..990009.
F-REBOOT — a guest rebooted mid-backup never came back (fault 11: 9m47s of total
appliance outage, no lock, nothing retrying). The existing stale-lock recovery is
correct but missed it two ways: its predicate needs a stale vzdump lock and that
guest was unlocked, and it runs only at agent startup. New periodic guest-power
watchdog acts on 'should be running, is not, is not locked'.
onboot is the should-be-running signal, not invented here: stalelock.go already
uses it for this same decision, it is 0 on scratch/golden, and pve-guests uses it
at host boot. Guards: onboot:0 never touched (Scenario B), a locked guest is left
to the stale-lock path, a guest with a vzdump in flight is left stopped,
unprovable ownership acts on nothing, unconfirmable backup state fails safe.
Bounded retry 3x at 1/2/4m then ERROR (Scenario C) — a healthy start takes ~25s.
F-LEAK — a failed restore-test could not destroy its scratch (403 VM.Allocate).
It is pool membership, not privsep: VM.Allocate is granted at /pool/felhom only,
and a failed restore never completes the --pool association. Fix needs NO new
grant — Pool.Allocate is already held, so the teardown adopts the stranded
scratch into the pool and retries the destroy. Guarded by scratchAdoptAllowed:
scratch provenance AND the numeric band, both required (Scenario E).
Six red-proofs across both fixes, all observed failing.
NewestArchiveTime counted an aborted PBS upload (1 byte, no manifest, NEWEST)
as a successful backup, so the tier reported fresh, went not-due, and was never
retried. On the real 168h offsite cadence that is 7 days of silence, and neither
the R-88 breaker (defers only DUE tiers) nor the hub deadline monitor (reads the
same freshness) can catch it.
R-84's storage-as-ground-truth was right; the bug is that presence was taken for
validity. Now only plausibly-complete entries count, via a measured size floor
(minPlausibleArchiveBytes = 1 MiB). Undecidable => not counted.
Size is the only tier-agnostic discriminator: verification and encrypted are
absent on EVERY local dir archive (and on a good PBS snapshot until verify-new
catches up), so gating on either would reject 100% of local backups and cause
fleet-wide backup thrash. Floor measured against the fleet: smallest real backup
is 612,397,450 B, so 1 MiB leaves 584x headroom — asserted by a test.
Rejections are announced at WARN once per distinct volid, naming snapshot and
reason; per-poll logging would emit ~288 lines/day and bury the signal.
Four red-proofs, all observed failing.
newestArchiveOn's (time.Time, bool) signature could not express the 'unknown'
its own doc comment promised: a read error and a genuine not-found both returned
(zero,false), so /backup/due emitted a POSITIVE 'no successful backup recorded
yet' with a nil age, and the controller fired its window-gate valve on an
unreadable storage.
Three states now: known / absent / unknown, carried as a STRING enum so the zero
value unambiguously means 'legacy agent' rather than masquerading as an answer.
Fail-safe direction unchanged — unknown is still DUE; only the window-gate bypass
narrows to ABSENT.
A service with NO lister deliberately stays ABSENT: calling it unknown would stop
a genuinely new box on a pre-R-84 build from ever backing up outside its window.
An unparseable timestamp becomes unknown — a backup happened, we cannot date it.
Version finalised from v0.104.0-dev — an ldflags version disagreeing with the
CHANGELOG is the reconciliation problem hub 0.73.2 already caused.
REUSE gains backup.InFlight, backup.RestoreTestState and the SpecBuilder/
TierPicker/PickRestoreCandidateOn trio, each with the decision someone could
unknowingly undo: rotation credit only on success; InFlight callers defer and
never cancel; a tier with no archive no-ops rather than failing; SourceTier
comes from the archive, never the configured target.
The scheduler could only ever see cfg.Backup.BackupTarget(), so the offsite
tier's archives were never candidates — which is why demo-hp's DR tier reported
'applied' with zero snapshots for five days and nobody noticed.
Selection: oldest-first (operator ruling, Option 1). Never-proven sorts first,
which is where the offsite tier starts. Ties break on target id so ordering is
deterministic rather than following Go's randomised map order. Rotation credit
only on SUCCESS — a permanently failing tier must keep sorting first, not look
freshly proven and stop being retried.
- backup.RestoreTestState: persisted last-success per tier (atomic tmp+rename).
This genuinely needs persistence unlike R-84: R-84 had ground truth to consult
(the archive is still on the storage), whereas a restore-test destroys its
scratch and leaves no artifact. Corrupt/missing file -> 'nothing proven'.
- backup.InFlight: host-wide one-heavy-op gate shared with the local-API backup
path. A LINK concern, not a lock one — an offsite restore pulls multi-GB over
the same tunnel a backup pushes one, and at ~33 MB/min both drift toward
timeout, which is how a healthy tier gets recorded as failed. Callers DEFER,
never cancel.
- PickRestoreCandidateOn: newest archive on a named tier; '' is not an error, or
every fresh box looks broken for its first week.
- An empty tier is skipped and the next tried; it cannot starve, since it is
still least-recently-proven once it has an archive.
- POST /backup joins the gate (409 naming the holder).
Red-proofs A/E/F observed with the documented text. Full suite green (29
packages, rc=0).
SchedulerOptions.Spec was a VALUE produced by an immediately-invoked function
at daemon start, so storageTier() and restoreTaskTimeout() were evaluated once
and reused for every run for the process lifetime. Nothing tier-varying was
expressible (the offsite tier could never be scheduled), and it was a latent
staleness bug besides: a storage-type or config change did not take effect
until restart.
- backup.SpecBuilder: func(ctx, archive) RestoreTestSpec, called once per run.
The archive is passed because the tier MUST come from it (v0.100.0 rule) —
config-derived is what classified a PBS archive as 'local' and killed a
14.46 GB WAN restore at the 10-minute local bound.
- A nil spec builder SKIPS loudly instead of panicking: a wiring bug must cost a
restore-test, never the daemon goroutine.
Red-proof observed. Full suite green (29 packages, rc=0).
Observed live: three redundant local backups on demo-felhom in one afternoon of
deploys. The backup Store is in-memory ('lost on restart; the cadence
re-populates'), so after every restart /backup/due said 'no successful backup
recorded yet' and the controller took another one. On the offsite tier that is a
wasted multi-hour WAN upload after every agent deploy.
- BackupRunner.NewestArchiveTime: when a backup last LANDED on this tier's
storage, read from the storage.
- localapi.BackupArchiveLister (optional BackupService extension): the due-check
takes whichever is newer, the in-memory record or the storage.
Asking the storage rather than persisting the store is deliberate: it is ground
truth (a pruned archive correctly stops counting, where a persisted record would
keep claiming a backup that no longer exists), needs no new on-disk state, and
answers only 'when did a backup last land' — the richer fields stay with real
records so the host-report never carries invented numbers.
Fail-safes: read error -> fall back to memory (never fake freshness, never
suppress); genuinely empty -> due; old archive -> still due; service without the
lister -> unchanged.
Red-proof observed; full suite green (29 packages).
Prerequisite for the installer default (host-install 1.20.0). A fresh box now
carries the offsite tier, but felhom-pbs only exists once the hub provisions the
DR tier. Without this the tier would report due in that window and the
controller would quiesce the apps and fire a vzdump at a missing storage every
cadence.
- GET /backup/due?target= defers when the target storage is absent
(targetStoragePresent): due:false with a reason that says why. The tier goes
live with NO restart once the storage appears.
Fail-safe: a storage-view ERROR returns present and the tier stays due. 'I could
not check' must never be read as 'not there' — that would silently suppress
backups, the absence-is-not-failure rule relearned three times now (R-80, R-81,
the R-82 wait timeout).
Full suite green.
CORRECTION: I earlier reported that the restore-test would boot a scratch guest
with the live guest's MAC/static island IP/hostname and break the control
plane. That was WRONG — RunRestoreTest step 2 link-downs EVERY interface
(withLinkDown, unit-tested) before the guest is ever started. The design
already handled it.
The real, narrower hazard: a restore that fails BEFORE step 2 (what the v0.100.0
wait bug caused) leaves a scratch holding the SOURCE guest's config verbatim,
including onboot:1. If teardown also fails (403 missing VM.Allocate — PVE
associates the pool only at restore completion), a host reboot would start that
leaked clone alongside the original with NICs up.
- proxmox.RestoreLXCOptions.ConfigOverrides: guest-config params applied AT
RESTORE TIME.
- The restore-test passes onboot=0 — at restore time, not after, because
'after' is exactly the path that leaks.
NOT changed: the link-down step (already correct, the primary defence); the
agent's Proxmox privileges (widening VM.Allocate to /vms would remove the
accidental guard that stopped a destructive mid-restore teardown).
restore_test_cadence_seconds was set to -1 on demo-felhom under the mistaken
reading; re-enabled.
Red-proof observed; full suite green (29 packages).
Found by the first real PBS restore round-trip, not by review.
Restoring a felhom-pbs: archive on a box whose primary target is 'local'
failed after exactly 600.76s — the 10-minute LOCAL wait — against a 14.46 GB
WAN restore needing ~2 hours. The selftest derived its tier from
cfg.Backup.BackupTarget() (the configured default), so restoreTaskTimeout
correctly returned the local bound for a PBS archive. The recorded result even
said source_tier=local for a PBS archive.
The tier-aware machinery was already right; it was fed the wrong input. What
broke is an assumption that stopped being true the moment a second tier
existed: 'the configured target' is no longer a proxy for 'the tier this
archive belongs to'.
RestoreTestSpec.RestoreTaskTimeout's doc comment predicts the consequence
exactly, and it happened: teardown fired at a still-restoring guest and was
refused with HTTP 403 missing privilege VM.Allocate (PVE associates the pool
only at restore COMPLETION, and the grant is on /pool/felhom not /vms). That
403 was load-bearing luck — the only reason a destructive teardown did not run
against a half-restored guest. The restore completed unharmed.
- restoreTierForArchive() derives the tier from the archive's own storage
(archiveStorageID parses the volid prefix), falling back to the configured
target only when there is no prefix.
Recorded, NOT fixed here: the daemon's scheduled restore-test still only covers
the PRIMARY tier (Pick uses a runner built on BackupTarget(); Spec is built once
at construction, not per tick) — so the offsite tier is never automatically
restore-tested. And the agent still cannot tear down a scratch guest until its
restore completes; widening the token's privileges is deliberately not the fix.
Full suite green (29 packages).
Ruling 1 (2 weeks of weekly offsite backups): localPruneSpec's blanket PBS
refusal is now scoped — an ADDITIONAL tier with an explicit keep_last may
prune its PBS target. The refusal still applies in full to the PRIMARY tier,
because BackupTarget() defaults to felhom-pbs and KeepLast() defaults to 3, so
a box with neither key set would silently prune its offsite DR to 3 restore
points. An additional tier cannot have that accident (keep_last defaults to 0).
Ruling 3 (first backup runs as long as needed; nothing else starts until done):
- additional-tier wait bound 6h -> 12h (measured ~33 MB/min => ~5h for a first
full 10 GB snapshot; 12h gives margin but stays bounded so a hung task still
surfaces)
- ONE BACKUP AT A TIME PER GUEST across all tiers: POST /backup returns 409
when a DIFFERENT tier is in flight, naming the busy tier, with NO data object
so nothing is parseable as the caller's own job. Same tier still returns that
job (202, unchanged).
- snapshotted now counts as in-flight, not just running — after the snapshot the
vzdump is still uploading and holding the lock. The old check left a window
where a second POST started a real second vzdump. Latent bug, closed.
Full suite green (29 packages); red-proof observed and restored.
Found by live validation on demo-felhom, not by review.
The first real PBS-targeted backup ran past the runner's hard-coded 30-minute
WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE
VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences:
the tier stays permanently due, the next attempt collides with the guest lock
the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's
'applied and empty' fault re-created by a timeout.
Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot
projects to ~5h.
- BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED
(a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from
the measurement.
- backup.NewBackupRunnerWithWait: per-instance (per-tier) bound.
NewBackupRunner keeps its signature, so restore-test/selftest are untouched.
- localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the
tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h
outer context reproduces the same false failure four hours later.
Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait
LONGER. A slow backup is a slow backup; a false timeout is a corrupt status
plus lock contention.
Red-proof observed and restored; full suite green.
Mechanism only. No box changes behaviour until a backup_targets entry is
added to its config (Slice D); an untouched config resolves to exactly one
tier and behaves byte-identically to v0.96.0.
- config: BackupTargetConfig + ExtraTargets + BackupTiers(); each tier carries
its OWN cadence and retention (keep-last=3 is three days on a daily tier and
three weeks on a weekly one). A missing cadence is REJECTED, not defaulted —
a weekly DR tier silently running daily would fill the 37.2 GB datastore.
main.go logs every rejection at ERROR.
- /backup/due?target= judges a tier against its OWN newest successful backup.
Without that filter a fresh local backup satisfies the weekly PBS cadence and
the DR tier never runs — today's bug, re-created in code.
- GET /backup/tiers advertises the tiers; a 404 is the controller's pre-R-82
capability probe (Slice B).
- Jobs keyed by (vmid,target): single-flight is per tier, which is what lets
the weekly night run both backups in ONE quiesce window. Job ids are unique
per tier by construction, not by clock luck.
- One runner per tier: the runner holds target+retention as immutable state,
so parameterising one runner would risk pairing tier A's target with tier B's
retention.
COMPATIBILITY (frozen): untargeted /backup/due, POST /backup and
/backup/status keep the primary tier and the pre-R-82 response BYTES —
Target is omitempty and stays empty. The primary's job-id format is unchanged.
NOT changed: the local tier; PBS is still never pruned by the per-run flag
(keep_last defaults to 0 = never prune — enabling DR pruning is irreversible
and needs an operator ruling).
Tests 748->768. Red-proof #1 observed and restored.
Phase 0: felhom.eu/documentation/audits/SPIKE-r82-phase0-2026-07-26.md
Implements SPIKE-smart-coverage-2026-07-25 fixes B+A (additive; MinAgent unchanged).
Fix B: storage.SmartReader.SMARTForBacking wired into the /disks union path (localapi
Smart seam) so registry/USB drives get a real SMART read (watchdog Known stays
enrich-free). Fix A: smartDeviceFor resolves dm/LVM to the whole disk via
/sys/block/<dm>/slaves (recursive; skips >1-disk); the builtin local dir on the LVM
root gets a SMART-only device from its containing filesystem (never touches
backing/durable_id). SmartSummary.ModelName captured from smartctl. Fix C (-d sat)
stays rejected. Tests + red-proofs (dm multi-disk skip, enrich smartHint, union
routing); Known-path-never-SMARTs asserted.
admin
Claude Code