Found by live validation on demo-felhom, not by review.
The first real PBS-targeted backup ran past the runner's hard-coded 30-minute
WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE
VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences:
the tier stays permanently due, the next attempt collides with the guest lock
the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's
'applied and empty' fault re-created by a timeout.
Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot
projects to ~5h.
- BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED
(a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from
the measurement.
- backup.NewBackupRunnerWithWait: per-instance (per-tier) bound.
NewBackupRunner keeps its signature, so restore-test/selftest are untouched.
- localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the
tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h
outer context reproduces the same false failure four hours later.
Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait
LONGER. A slow backup is a slow backup; a false timeout is a corrupt status
plus lock contention.
Red-proof observed and restored; full suite green.
Mechanism only. No box changes behaviour until a backup_targets entry is
added to its config (Slice D); an untouched config resolves to exactly one
tier and behaves byte-identically to v0.96.0.
- config: BackupTargetConfig + ExtraTargets + BackupTiers(); each tier carries
its OWN cadence and retention (keep-last=3 is three days on a daily tier and
three weeks on a weekly one). A missing cadence is REJECTED, not defaulted —
a weekly DR tier silently running daily would fill the 37.2 GB datastore.
main.go logs every rejection at ERROR.
- /backup/due?target= judges a tier against its OWN newest successful backup.
Without that filter a fresh local backup satisfies the weekly PBS cadence and
the DR tier never runs — today's bug, re-created in code.
- GET /backup/tiers advertises the tiers; a 404 is the controller's pre-R-82
capability probe (Slice B).
- Jobs keyed by (vmid,target): single-flight is per tier, which is what lets
the weekly night run both backups in ONE quiesce window. Job ids are unique
per tier by construction, not by clock luck.
- One runner per tier: the runner holds target+retention as immutable state,
so parameterising one runner would risk pairing tier A's target with tier B's
retention.
COMPATIBILITY (frozen): untargeted /backup/due, POST /backup and
/backup/status keep the primary tier and the pre-R-82 response BYTES —
Target is omitempty and stays empty. The primary's job-id format is unchanged.
NOT changed: the local tier; PBS is still never pruned by the per-run flag
(keep_last defaults to 0 = never prune — enabling DR pruning is irreversible
and needs an operator ruling).
Tests 748->768. Red-proof #1 observed and restored.
Phase 0: felhom.eu/documentation/audits/SPIKE-r82-phase0-2026-07-26.md
Implements SPIKE-smart-coverage-2026-07-25 fixes B+A (additive; MinAgent unchanged).
Fix B: storage.SmartReader.SMARTForBacking wired into the /disks union path (localapi
Smart seam) so registry/USB drives get a real SMART read (watchdog Known stays
enrich-free). Fix A: smartDeviceFor resolves dm/LVM to the whole disk via
/sys/block/<dm>/slaves (recursive; skips >1-disk); the builtin local dir on the LVM
root gets a SMART-only device from its containing filesystem (never touches
backing/durable_id). SmartSummary.ModelName captured from smartctl. Fix C (-d sat)
stays rejected. Tests + red-proofs (dm multi-disk skip, enrich smartHint, union
routing); Known-path-never-SMARTs asserted.
Additive, backward-compatible (MinAgent floor unchanged). The SMART is already
computed on the request path (storage.Observe -> enrich); this copies the target's
Smart into localapi.DiskInfo (pointer, omitempty) only when Health != "", so an
unread/absent summary stays omitted and the controller renders "Nincs adat".
No new smartctl load, endpoint, or sudoers change.
Test TestDisks_SmartSerialized + red-proof (drop the copy -> fails).
The EFF large list has exactly 4 entries containing the join separator
(drop-down, felt-tip, t-shirt, yo-yo). Drawing one made a code read as 11
words instead of 10 - ambiguous to transcribe in precisely the situation R
exists for. Filter them at init; the draw space goes 7776 -> 7772 and the
10-word code goes 129.248 -> 129.241 bits, still well over the 128 floor.
Generation-only: already-issued codes stay valid, R is verified as a whole
passphrase and never re-split.
Also fixes the ~1/5 flake this same defect caused: the test counted words by
splitting the joined string. It now counts what the generator drew and
asserts segmentation separately, plus a deterministic red-proof fixture.
v0.92.0 was published before live verification showed three of the four guestnet probes had
no sudoers grant, so that artifact lacks FELHOM_GUESTNET and the guestnet-* capability rows.
Superseded, not overwritten: a published version stays immutable (the v0.91.0 -> v0.91.1
precedent). Watchdog code is unchanged; the functional fix is the sudoers file, which must
ship with the binary.
The first sweep on felhom-pve logged 'dhclient liveness probe failed: sudo: a password is
required' and reported state=unknown — the watchdog fail-safed correctly, but it was blind.
TASK-D assumed no sudoers change; the existing grant only covered lanresolver's address read.
FELHOM_GUESTNET adds four FIXED vectors (route, interfaces, pgrep, and the incident's heal
invocation). Every argument after the numeric vmid is a literal, so nothing the guest or the
hub says can widen the grant. Plus four guestnet-* capability rows, non-critical, so a host
that has not taken the new sudoers is visibly degraded without paging an operator per box.
Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient
is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20
the guest ran another ~80 minutes on its unexpired lease, then lost its address and default
route and took the tunnel, hub reports, catalog sync and the controller->agent channel with
it (1h15m outage, healthy-looking for the first 80 minutes).
So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment
`pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address
to vanish is waiting out the silent window.
internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces
mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's
heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min
between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on
a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source
is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as
unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never
probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able
to kill the agent.
guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only
inward at guests we already own, and the failure exists on every box today.
Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN
direction in this repo, report stanzas are *Status.
Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy"
with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test.
Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not).
The probe logged only on failure, so a healthy one was silent: "no auth_failed" was
indistinguishable from "never probed", and the leg could not be demonstrated as running.
That is exactly how v0.91.0 shipped it inert unnoticed.
v0.91.0 built the AuthSink seam and the NoteAuthResult consumer, and main.go never
called SetAuthSink. The reporter skips probing when no sink is attached, so the entire
auth-honesty leg was silently inert — no probe, no auth_failed, no self-heal — and
nothing failed, because every unit test injected the sink directly.
Caught during STOP-1 live verification by checking the wiring instead of trusting it.
Same class as the controller v0.154.0 defect the day before: a table test over a seam
proves the seam, not the caller.
The published 0.91.0 artifact is superseded, not overwritten — a published version stays
immutable. TestLiveReporter_NoSinkMeansNoProbe pins the no-sink-no-probe contract so the
inert case is documented behaviour rather than an accident; only live evidence can prove
the wiring itself.
Closes the agent half of R-39's fleet fix. Requires hub >=0.68.0 for the re-arm signal;
that hub is safe for 0.90.0 agents (unknown key dropped), so it deploys first.
Three compounding defects let a box report `applied` while every PBS request 401'd:
1. The re-key was INVISIBLE. An ep0 re-issue rotates the secret of an existing token, so
token_id/fingerprint/datastore/namespace come back byte-identical and the descriptor
content hash never moved — the converged agent short-circuited and never consumed the
fresh secret. WirePBSDR.SecretGeneration (field-exact with the hub) is what moves the
hash now, because descriptorHash marshals this struct.
2. The agent could not READ its own credential. It writes /etc/pve/priv/storage/<id>.pw
through the root wrapper, but that dir is 0700 root:www-data and the wrapper had no
read verb — so the target resolver got "permission denied" every cycle, warned, and
skipped. The one loop that could have caught the 401 was blind BY CONSTRUCTION. Adds a
narrow `read` verb (+ exactly one sudoers line, + a pbsdr-read capability row): one
secret to stdout, no network, no mutation, never in argv (sudo logs argv), traversal
refused by the id grammar, the dir allowlist AND a resolved-path prefix assertion.
3. Nothing probed AUTHENTICATION. pbs.ProbeAuth (GET /version + an ErrUnauthorized
sentinel) runs on the 15-minute collect path and its verdict becomes a loud
`auth_failed` the hub escalates to a fresh mint. /version needs no datastore, namespace
or privilege, so a 401 means the CREDENTIAL is bad; 403 is deliberately NOT treated as
unauthorized, since re-keying a too-narrow token would mint forever without fixing
anything. A transport error is UNKNOWN, never a rejection — otherwise every network
blip burns a credential. Recovery self-clears.
R-50b(a): the report now carries the installed wrapper's sha256 so drift against the
vouched manifest value is answerable. Empty = unknown, never drift.
Three red-proofs, all at the assertion level. Removing SecretGeneration fails the re-arm
test with "consume calls=1, want 2". Swallowing the probe result leaves State:applied
AuthFailed:false — the July-18 shape exactly. Notably, deleting the wrapper's id charset
guard alone does NOT open a traversal hole (readlink + the prefix assertion still catch
it), so the isolating red-proof removes BOTH and shows the out-of-tree secret printed —
the layering is real, and a single-guard red-proof would have passed vacuously.
Enabling Megosztás on a fresh box pulled felhom-samba from the registry with
zero feedback — minutes of silent nothing. Observed live, twice.
Cause: this script carried its own hand-maintained array of three image tags,
with a comment telling the reader to keep it in sync with the controller's
internal/infra constants. It drifted the moment a fourth stack was added.
felhom-samba was never added here, so the golden baked 3 of 4.
The fix is structural rather than a fourth copy of the list: it now comes from
the controller image the bake just pulled (--print-infra-images, backed by
infra.Images(), which derives from the pins themselves). The golden bakes
exactly what THAT controller version will request, so the two cannot disagree.
Ordering fix this exposed: docker logout ran immediately after the controller
pull, but felhom-samba is on the same private registry, so the infra loop would
have 401'd. Logout moved after the loop, plus a hard assertion that no
credential remains in the guest before it is archived.
Pre-0.147.0 controllers have no such flag; the bake falls back to the historical
3-image list and says so loudly — the fallback IS the drift-prone thing being
removed, so it must never pass silently.
No agent version bump: build tooling only. Effective at the next golden build;
the current golden is not rebuilt for this.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Supersedes the "not yet healed" hand-off section. Viktor clicked "Re-issue PBS
credentials" and the chain closed in 9 seconds: hub re-issued 20:28:44 -> agent
consumed 20:28:51 -> `pbsdr: converged state=applied` 20:28:53, with the PATCHED
wrapper. consumed-failed.json cleared automatically, marker.json rewritten. The
agent picked it up unaided because marker.json was absent, so the L235 hash
short-circuit did not apply.
Proof the tier is alive rather than merely reporting alive:
pvesm status 401/inactive -> ACTIVE
token /version 401 -> 200
real backup none -> felhom-pbs:backup/ct/9201/
2026-07-18T18:31:06Z, 9744319312 B, 13m36s
Encrypted under fingerprint 7e:a6:af:f7:ea:6d:3e:d9 — the ESCROWED key, the one
customer zero holds the recovery code for. First real DR-tier backup on the
reborn box.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Config-only (wrapper + red-proof); the Go binary is unchanged, so this ships
with the next agent deploy as a config artifact.
The reconcile verb built `args=(--server "$server" --fingerprint "$fp")`. PVE
treats a PBS storage's `server` as a CREATE-ONLY parameter and rejects the
ENTIRE `pvesm set` call — "can't change value of fixed parameter 'server'" —
even when the value passed is byte-identical to the stored one. So reconcile
could never succeed against an existing entry; it exited 255 every time.
That is severe rather than cosmetic because the agent consumes the hub's
ONE-TIME PBS token secret BEFORE invoking the wrapper. Each hub "Re-issue PBS
credentials" therefore minted a secret, the agent burned it, the wrapper
rejected the apply, and the entry stayed pinned to the revoked credential —
a PBS DR tier authenticating 401 indefinitely while the agent reported
`pbsdr: converged state=applied`.
Live-diagnosed on the N100 during the rehearsal wrap (felhom.eu
tests/VALIDATION-n100-rehearsal-2026-07-18.md F2, ROADMAP R-39). Proven on the
live entry before writing code: `pvesm set <id> --server <same> --fingerprint
<same>` -> rejected; the same call without --server -> rc 0. K (<id>.enc) and
the .pw store verified byte-untouched after the rejected call — PVE rejects
atomically, so the set-only law held.
Fix: drop --server. The server address is immutable by construction (relocating
a PBS endpoint needs a fresh create), so there was never anything to reconcile
there. --fingerprint (+ --password when a secret is fed) remain.
Red-proof TestReconcileNeverPassesServerToPvesmSet: isolates the reconcile)
block from the shipped wrapper, asserts no --server reaches `pvesm set` and
that --fingerprint is still pushed. Verified RED on the unfixed wrapper, GREEN
after. Handles two vacuous-pass traps that both fired while authoring it: the
pattern is line-ending tolerant (\r?\n — this repo is cloned on Windows, and an
\n-only pattern matches nothing and passes silently), and comment lines are
stripped before matching (the WHY note quotes the very flag under test).
NOT fixed here, both still open and riding the spec'd R-39 agent train:
1. R-39's primary half — the agent re-applies on a change of the DESCRIPTOR
HASH (manager.go ~L235), but a credential re-issue leaves the descriptor
byte-identical (same token_id/fingerprint; only the side-table secret
rotates) and bumps only the generation, so a converged agent still ignores
a fresh secret. This makes the apply succeed once it re-applies; it does
not make it re-apply.
2. The verify loop reads /etc/pve/priv/storage/<id>.pw directly as non-root —
a path it can only ever WRITE through the root wrapper (0700 root:www-data;
sudoers exposes create|reconcile|grant, no read verb), so it is permanently
blind to the failure it exists to catch.
Demo box: wrapper hotfixed in place (.bak-20260718-preR39 kept). NOT yet healed
— diagnosis consumed the pending secret against the unfixed wrapper; the agent
parked correctly in consumed-failed (no burn loop). Healing needs Viktor to
click "Re-issue PBS credentials"; the agent will then pick it up unaided
(marker.json absent, so the L235 short-circuit does not apply).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
MinAgent coupling: felhom-controller v0.143.0 gates its guest-memory-resize UI on
this agent (FeatureGuestMemoryResize, MinAgent 0.90.0).
R-24 guest RAM resize (internal/localapi/guestmemory.go): self-scoped GET/POST
/guest/memory. Agent enforces every bound FRESH per request (min 2048, max
host_total-2048, shrink floor max(2048, usage+512)); applies via PVE SetConfig —
live cgroup apply, no reboot (Phase-0 proven on the nested demo box). Verify-after-apply
re-reads maxmem before claiming success. New narrow MemoryOps seam (GuestAPI untouched);
Options.Memory nil -> 503. Memory only.
R-28 fast-tick (internal/fasttick): while any desired-state item is unapplied -
including the pre-tunnel window a hub poke can't reach - pulse the shared out-of-band
trigger every 30s, self-disarm on convergence. Four cached sources (desired-gen==0,
reconcile Planned-Pending>0, pbsdr waiting_secret only, wgtunnel desired-not-operational);
LOUD pbsdr states + pending_signature excluded. Seams: reconcile.Engine.LastResult() +
wgtunnel.Manager.TunnelConvergence() (cached, no per-tick exec).
Guests-0/0: hypothesis REFUTED live (9201 IS a pool member; 0/0 was the pre-provision
window; PoolAddVMID re-assert already covers restore-over-existing). No code change; the
fast-tick mitigates the window.
Tests + red-proofs (i floor guard, ii max guard, iii always-pulse) all restored green.
- pbsdr: on a 403 pre-check (non-default storage id, no ACL yet) self-grant via the root wrapper then re-read, instead of aborting before the grant — closes F4/R-22. Red-proof TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant.
- escrow preflight: late-bound CurrentPBSStorageID re-reads agent.json so a pbsdr-seeded pbs_storage_id flips the row green in-process (no restart). Red-proof TestEscrowPreflight_PBSStorageIDLiveReload.
- internal/poke: contentless UDP poke listener bound exclusively to the box WG /32 (port 51822), leading-edge debounced, fires the hub-loop out-of-band trigger for an immediate desired-state cycle. First slice of R-13. Red-proofs TestBindConfinement + TestDebounceCoalescesBurst.
Operator ruling 2026-07-13: walk the root's backing device through /sys/block/<dev>/slaves
recursively down to physical disks (dm AND md; topology, never VG names); those + any mounted-ESP
holder are system; the all-system fail-safe returns to being the WALK-FAILURE error case only.
SAFETY DIRECTION: a root-backing disk classified candidate is made impossible — per-branch
conservatism (any unresolvable slave fails the WHOLE walk -> ok=false -> the unchanged all-system
path).
- physicalDisksOf/walkSlaves in role.go (symlink canon -> wholeDiskOf fast path -> recursive
slaves walk; cycle/depth guard; non-/dev sources unwalkable)
- HostReader.BlockSlaves(name) — the ONE new seam method; ProcHostReader reads
/sys/block/<name>/slaves; all four test fakes mirror it
- role_walk_test.go: signature table (root-backing disk ALWAYS system across legacy-LVM /
md-raid / EFI+raw / EFI+LVM / nested dm-on-md — NEVER weaken) + dead-wizard-lives +
dangling-slave fail-safe (real sysKnown=false path) + cycle + empty-slaves; red-proofs A/B/D
run->fail->revert (recorded in REPORT)
- §3 spike transcripts (drill legacy: dm-1->sda3->sda; felhom-pve: ESP+walk agree on sda ->
byte-identical regression); caller audit: none relied on all-system as a feature
- format/mkfs paths, data-bearing guards, wizard UI untouched
Claude Code
admin