Mechanism only. No box changes behaviour until a backup_targets entry is
added to its config (Slice D); an untouched config resolves to exactly one
tier and behaves byte-identically to v0.96.0.
- config: BackupTargetConfig + ExtraTargets + BackupTiers(); each tier carries
its OWN cadence and retention (keep-last=3 is three days on a daily tier and
three weeks on a weekly one). A missing cadence is REJECTED, not defaulted —
a weekly DR tier silently running daily would fill the 37.2 GB datastore.
main.go logs every rejection at ERROR.
- /backup/due?target= judges a tier against its OWN newest successful backup.
Without that filter a fresh local backup satisfies the weekly PBS cadence and
the DR tier never runs — today's bug, re-created in code.
- GET /backup/tiers advertises the tiers; a 404 is the controller's pre-R-82
capability probe (Slice B).
- Jobs keyed by (vmid,target): single-flight is per tier, which is what lets
the weekly night run both backups in ONE quiesce window. Job ids are unique
per tier by construction, not by clock luck.
- One runner per tier: the runner holds target+retention as immutable state,
so parameterising one runner would risk pairing tier A's target with tier B's
retention.
COMPATIBILITY (frozen): untargeted /backup/due, POST /backup and
/backup/status keep the primary tier and the pre-R-82 response BYTES —
Target is omitempty and stays empty. The primary's job-id format is unchanged.
NOT changed: the local tier; PBS is still never pruned by the per-run flag
(keep_last defaults to 0 = never prune — enabling DR pruning is irreversible
and needs an operator ruling).
Tests 748->768. Red-proof #1 observed and restored.
Phase 0: felhom.eu/documentation/audits/SPIKE-r82-phase0-2026-07-26.md
Implements SPIKE-smart-coverage-2026-07-25 fixes B+A (additive; MinAgent unchanged).
Fix B: storage.SmartReader.SMARTForBacking wired into the /disks union path (localapi
Smart seam) so registry/USB drives get a real SMART read (watchdog Known stays
enrich-free). Fix A: smartDeviceFor resolves dm/LVM to the whole disk via
/sys/block/<dm>/slaves (recursive; skips >1-disk); the builtin local dir on the LVM
root gets a SMART-only device from its containing filesystem (never touches
backing/durable_id). SmartSummary.ModelName captured from smartctl. Fix C (-d sat)
stays rejected. Tests + red-proofs (dm multi-disk skip, enrich smartHint, union
routing); Known-path-never-SMARTs asserted.
Additive, backward-compatible (MinAgent floor unchanged). The SMART is already
computed on the request path (storage.Observe -> enrich); this copies the target's
Smart into localapi.DiskInfo (pointer, omitempty) only when Health != "", so an
unread/absent summary stays omitted and the controller renders "Nincs adat".
No new smartctl load, endpoint, or sudoers change.
Test TestDisks_SmartSerialized + red-proof (drop the copy -> fails).
The EFF large list has exactly 4 entries containing the join separator
(drop-down, felt-tip, t-shirt, yo-yo). Drawing one made a code read as 11
words instead of 10 - ambiguous to transcribe in precisely the situation R
exists for. Filter them at init; the draw space goes 7776 -> 7772 and the
10-word code goes 129.248 -> 129.241 bits, still well over the 128 floor.
Generation-only: already-issued codes stay valid, R is verified as a whole
passphrase and never re-split.
Also fixes the ~1/5 flake this same defect caused: the test counted words by
splitting the joined string. It now counts what the generator drew and
asserts segmentation separately, plus a deterministic red-proof fixture.
The first sweep on felhom-pve logged 'dhclient liveness probe failed: sudo: a password is
required' and reported state=unknown — the watchdog fail-safed correctly, but it was blind.
TASK-D assumed no sudoers change; the existing grant only covered lanresolver's address read.
FELHOM_GUESTNET adds four FIXED vectors (route, interfaces, pgrep, and the incident's heal
invocation). Every argument after the numeric vmid is a literal, so nothing the guest or the
hub says can widen the grant. Plus four guestnet-* capability rows, non-critical, so a host
that has not taken the new sudoers is visibly degraded without paging an operator per box.
Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient
is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20
the guest ran another ~80 minutes on its unexpired lease, then lost its address and default
route and took the tunnel, hub reports, catalog sync and the controller->agent channel with
it (1h15m outage, healthy-looking for the first 80 minutes).
So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment
`pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address
to vanish is waiting out the silent window.
internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces
mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's
heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min
between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on
a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source
is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as
unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never
probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able
to kill the agent.
guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only
inward at guests we already own, and the failure exists on every box today.
Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN
direction in this repo, report stanzas are *Status.
Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy"
with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test.
Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not).
The probe logged only on failure, so a healthy one was silent: "no auth_failed" was
indistinguishable from "never probed", and the leg could not be demonstrated as running.
That is exactly how v0.91.0 shipped it inert unnoticed.
v0.91.0 built the AuthSink seam and the NoteAuthResult consumer, and main.go never
called SetAuthSink. The reporter skips probing when no sink is attached, so the entire
auth-honesty leg was silently inert — no probe, no auth_failed, no self-heal — and
nothing failed, because every unit test injected the sink directly.
Caught during STOP-1 live verification by checking the wiring instead of trusting it.
Same class as the controller v0.154.0 defect the day before: a table test over a seam
proves the seam, not the caller.
The published 0.91.0 artifact is superseded, not overwritten — a published version stays
immutable. TestLiveReporter_NoSinkMeansNoProbe pins the no-sink-no-probe contract so the
inert case is documented behaviour rather than an accident; only live evidence can prove
the wiring itself.
Closes the agent half of R-39's fleet fix. Requires hub >=0.68.0 for the re-arm signal;
that hub is safe for 0.90.0 agents (unknown key dropped), so it deploys first.
Three compounding defects let a box report `applied` while every PBS request 401'd:
1. The re-key was INVISIBLE. An ep0 re-issue rotates the secret of an existing token, so
token_id/fingerprint/datastore/namespace come back byte-identical and the descriptor
content hash never moved — the converged agent short-circuited and never consumed the
fresh secret. WirePBSDR.SecretGeneration (field-exact with the hub) is what moves the
hash now, because descriptorHash marshals this struct.
2. The agent could not READ its own credential. It writes /etc/pve/priv/storage/<id>.pw
through the root wrapper, but that dir is 0700 root:www-data and the wrapper had no
read verb — so the target resolver got "permission denied" every cycle, warned, and
skipped. The one loop that could have caught the 401 was blind BY CONSTRUCTION. Adds a
narrow `read` verb (+ exactly one sudoers line, + a pbsdr-read capability row): one
secret to stdout, no network, no mutation, never in argv (sudo logs argv), traversal
refused by the id grammar, the dir allowlist AND a resolved-path prefix assertion.
3. Nothing probed AUTHENTICATION. pbs.ProbeAuth (GET /version + an ErrUnauthorized
sentinel) runs on the 15-minute collect path and its verdict becomes a loud
`auth_failed` the hub escalates to a fresh mint. /version needs no datastore, namespace
or privilege, so a 401 means the CREDENTIAL is bad; 403 is deliberately NOT treated as
unauthorized, since re-keying a too-narrow token would mint forever without fixing
anything. A transport error is UNKNOWN, never a rejection — otherwise every network
blip burns a credential. Recovery self-clears.
R-50b(a): the report now carries the installed wrapper's sha256 so drift against the
vouched manifest value is answerable. Empty = unknown, never drift.
Three red-proofs, all at the assertion level. Removing SecretGeneration fails the re-arm
test with "consume calls=1, want 2". Swallowing the probe result leaves State:applied
AuthFailed:false — the July-18 shape exactly. Notably, deleting the wrapper's id charset
guard alone does NOT open a traversal hole (readlink + the prefix assertion still catch
it), so the isolating red-proof removes BOTH and shows the out-of-tree secret printed —
the layering is real, and a single-guard red-proof would have passed vacuously.
Config-only (wrapper + red-proof); the Go binary is unchanged, so this ships
with the next agent deploy as a config artifact.
The reconcile verb built `args=(--server "$server" --fingerprint "$fp")`. PVE
treats a PBS storage's `server` as a CREATE-ONLY parameter and rejects the
ENTIRE `pvesm set` call — "can't change value of fixed parameter 'server'" —
even when the value passed is byte-identical to the stored one. So reconcile
could never succeed against an existing entry; it exited 255 every time.
That is severe rather than cosmetic because the agent consumes the hub's
ONE-TIME PBS token secret BEFORE invoking the wrapper. Each hub "Re-issue PBS
credentials" therefore minted a secret, the agent burned it, the wrapper
rejected the apply, and the entry stayed pinned to the revoked credential —
a PBS DR tier authenticating 401 indefinitely while the agent reported
`pbsdr: converged state=applied`.
Live-diagnosed on the N100 during the rehearsal wrap (felhom.eu
tests/VALIDATION-n100-rehearsal-2026-07-18.md F2, ROADMAP R-39). Proven on the
live entry before writing code: `pvesm set <id> --server <same> --fingerprint
<same>` -> rejected; the same call without --server -> rc 0. K (<id>.enc) and
the .pw store verified byte-untouched after the rejected call — PVE rejects
atomically, so the set-only law held.
Fix: drop --server. The server address is immutable by construction (relocating
a PBS endpoint needs a fresh create), so there was never anything to reconcile
there. --fingerprint (+ --password when a secret is fed) remain.
Red-proof TestReconcileNeverPassesServerToPvesmSet: isolates the reconcile)
block from the shipped wrapper, asserts no --server reaches `pvesm set` and
that --fingerprint is still pushed. Verified RED on the unfixed wrapper, GREEN
after. Handles two vacuous-pass traps that both fired while authoring it: the
pattern is line-ending tolerant (\r?\n — this repo is cloned on Windows, and an
\n-only pattern matches nothing and passes silently), and comment lines are
stripped before matching (the WHY note quotes the very flag under test).
NOT fixed here, both still open and riding the spec'd R-39 agent train:
1. R-39's primary half — the agent re-applies on a change of the DESCRIPTOR
HASH (manager.go ~L235), but a credential re-issue leaves the descriptor
byte-identical (same token_id/fingerprint; only the side-table secret
rotates) and bumps only the generation, so a converged agent still ignores
a fresh secret. This makes the apply succeed once it re-applies; it does
not make it re-apply.
2. The verify loop reads /etc/pve/priv/storage/<id>.pw directly as non-root —
a path it can only ever WRITE through the root wrapper (0700 root:www-data;
sudoers exposes create|reconcile|grant, no read verb), so it is permanently
blind to the failure it exists to catch.
Demo box: wrapper hotfixed in place (.bak-20260718-preR39 kept). NOT yet healed
— diagnosis consumed the pending secret against the unfixed wrapper; the agent
parked correctly in consumed-failed (no burn loop). Healing needs Viktor to
click "Re-issue PBS credentials"; the agent will then pick it up unaided
(marker.json absent, so the L235 short-circuit does not apply).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
MinAgent coupling: felhom-controller v0.143.0 gates its guest-memory-resize UI on
this agent (FeatureGuestMemoryResize, MinAgent 0.90.0).
R-24 guest RAM resize (internal/localapi/guestmemory.go): self-scoped GET/POST
/guest/memory. Agent enforces every bound FRESH per request (min 2048, max
host_total-2048, shrink floor max(2048, usage+512)); applies via PVE SetConfig —
live cgroup apply, no reboot (Phase-0 proven on the nested demo box). Verify-after-apply
re-reads maxmem before claiming success. New narrow MemoryOps seam (GuestAPI untouched);
Options.Memory nil -> 503. Memory only.
R-28 fast-tick (internal/fasttick): while any desired-state item is unapplied -
including the pre-tunnel window a hub poke can't reach - pulse the shared out-of-band
trigger every 30s, self-disarm on convergence. Four cached sources (desired-gen==0,
reconcile Planned-Pending>0, pbsdr waiting_secret only, wgtunnel desired-not-operational);
LOUD pbsdr states + pending_signature excluded. Seams: reconcile.Engine.LastResult() +
wgtunnel.Manager.TunnelConvergence() (cached, no per-tick exec).
Guests-0/0: hypothesis REFUTED live (9201 IS a pool member; 0/0 was the pre-provision
window; PoolAddVMID re-assert already covers restore-over-existing). No code change; the
fast-tick mitigates the window.
Tests + red-proofs (i floor guard, ii max guard, iii always-pulse) all restored green.
- pbsdr: on a 403 pre-check (non-default storage id, no ACL yet) self-grant via the root wrapper then re-read, instead of aborting before the grant — closes F4/R-22. Red-proof TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant.
- escrow preflight: late-bound CurrentPBSStorageID re-reads agent.json so a pbsdr-seeded pbs_storage_id flips the row green in-process (no restart). Red-proof TestEscrowPreflight_PBSStorageIDLiveReload.
- internal/poke: contentless UDP poke listener bound exclusively to the box WG /32 (port 51822), leading-edge debounced, fires the hub-loop out-of-band trigger for an immediate desired-state cycle. First slice of R-13. Red-proofs TestBindConfinement + TestDebounceCoalescesBurst.
Operator ruling 2026-07-13: walk the root's backing device through /sys/block/<dev>/slaves
recursively down to physical disks (dm AND md; topology, never VG names); those + any mounted-ESP
holder are system; the all-system fail-safe returns to being the WALK-FAILURE error case only.
SAFETY DIRECTION: a root-backing disk classified candidate is made impossible — per-branch
conservatism (any unresolvable slave fails the WHOLE walk -> ok=false -> the unchanged all-system
path).
- physicalDisksOf/walkSlaves in role.go (symlink canon -> wholeDiskOf fast path -> recursive
slaves walk; cycle/depth guard; non-/dev sources unwalkable)
- HostReader.BlockSlaves(name) — the ONE new seam method; ProcHostReader reads
/sys/block/<name>/slaves; all four test fakes mirror it
- role_walk_test.go: signature table (root-backing disk ALWAYS system across legacy-LVM /
md-raid / EFI+raw / EFI+LVM / nested dm-on-md — NEVER weaken) + dead-wizard-lives +
dangling-slave fail-safe (real sysKnown=false path) + cycle + empty-slaves; red-proofs A/B/D
run->fail->revert (recorded in REPORT)
- §3 spike transcripts (drill legacy: dm-1->sda3->sda; felhom-pve: ESP+walk agree on sda ->
byte-identical regression); caller audit: none relied on all-system as a feature
- format/mkfs paths, data-bearing guards, wizard UI untouched
Agent half of the verify-before-commit task (SPIKE-nas-verify-2026-07-11, b57f6c1):
retry=0 in the production NFS options (Q4-vi); ClassifyNetVerifyFailure on the live
Q4 strings (nfs_export merges not-found/not-permitted); add = sync fast-fail (2s TCP
pre-probe, nothing installed) + detached in-memory verify job judging /proc/mounts
only, auto-rollback on failure; GET /netstorage/verify-status (phase none = the
controller's Scenario-F rollback signal); unprivileged journalctl (systemd-journal
group, NO new sudoers grants).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
HashResticPassword = sha256 hex over the trimmed password (pinned
cross-repo vector). escrowUploadRequest gains restic_pw_sha256,omitempty
— set only when a staged password was sealed into the blob. Contract test
updated; hub mirrors next.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The staged offsite repo password was wiped only by the escrow-create
ceremony; a confirm without a fresh ceremony (password already escrowed)
left the 0600 staged file behind. The controller calls this on every
EscrowState flip to escrowed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The restore-test had GL-5 finding #2's mirror image: its live-source-config
bind-override path tripped PVE's drop-unlisted-mountpoints rule, so scratch
guests boot-verified WITHOUT their storage mpN - weaker verification than
claimed. Params now derive from the ARCHIVE's own embedded config via
ExtractArchiveConfig + drRestoreOverrides (the object under test; full
layout, content genuinely extracted - the added runtime IS the
verification); unreadable/unknown-topology archives refuse up front. NEW
mount-parity assert (2b, pre-start): restored mpN set vs the archive's -
missing/mispathed/undersized/extra mpN fail the test naming the delta, so
constraint (b) can never regress into a green light. MountParity +
MountInventory ride the result + hub wire record (additive). Dead
bindMountOverrides/archiveVMID path deleted with its tests (no reachable
lookalike). DR bring-up untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The live scratch DR exposed the second half of PVE's all-or-nothing
explicit-params restore: mountpoints NOT named in the params are silently
DROPPED - the DR guest came up without its mp0/mp1 data volumes (boot
passed; the customer's world did not ride along). drRestoreOverrides now
derives the COMPLETE param set from the archive's extracted config:
explicit rootfs, every storage-backed mpN passed through (size + in-guest
path + backup flag preserved so vzrestore extracts its content), the two
structural binds replaced by 4d-swapped throwaways; unknown bind mpN or
unparseable size refuses loudly. Snapshot sections never shadow the
current config.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The live validation hit PVE's all-or-nothing restore rule: mpN params
without an explicit rootfs -> HTTP 500 "mount points configured, but
'rootfs' not set" (the same constraint restoretest.go:211 documents for the
live-config path; the spike never ran an override restore). The lost guest
has no live config, so the rootfs SIZE now comes from the archive's own
embedded config via NEW Client.ExtractArchiveConfig (GET vzdump/
extractconfig - verified live: answers 200 under the scoped agent token;
PBS keys stay server-side, the spike's candidate-1 rejection holds; used
for the SIZE ONLY - the bind layout stays the platform constants).
Unparseable/unreadable archive config -> clean refusal before any restore.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
ModeDRGuestLoss now passes restore-time MountOverrides for the two
platform-constant structural binds (mp8 parent, mp9 bootstrap) via the
shared throwaway-volume format helper - without them a customer-archive
restore under the privsep token fails outright ("restoring 'mp8' to bind
mount is only possible for root"). New post-restore step 4d swaps the real
binds in via the host runner (root pct set, one slot per call), deletes the
displaced unusedN volumes (API config PUT; a scoped-token refusal logs the
residue loudly instead of widening privileges), and respects the
committed/launched rollback envelope. Provision passes nil overrides -
byte-identical behavior (regression contract test).
Engine grows an optional HostRunner + StateDir seam (DR refuses up front
without a runner); selftest bring-up wires the ExecRunner + cleans the
scratch mp9 host dir on teardown; proxmox.GuestConfig.Unused() added.
6 new tests incl. C2 mid-swap rollback + C3 older-archive + 403-warn paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Pool membership is what lets the pool-scoped token reach a guest; pct restore
--pool sets it only at CREATE, so a restore over an existing VMID drops the guest
from the felhom pool and 403s the next restore-test/DR on VM.Audit. This empty-pool
state is the true root cause of the campaign's "R1" (bind-mount restore failing was
a symptom — restore-test's existing bind neutralization never ran without config-read).
Add Client.PoolAddVMID (PUT /pools, additive+idempotent, Pool.Allocate) and call it
in bring-up after liveness when spec.Pool!="" — warn-not-fail on a hiccup (liveness
wins). B3 scratch-teardown 403 diagnosed as a cascade (restoretest already passes
Pool). Role/ACL untouched. Tests + red-proof.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
roleForMountPath resolved role only from the PVE storage view; a bind-mounted
RAW enrolled user-data drive is not a PVE storage, so it fail-safe'd to system
and the eject/decommission gates 403'd EVERY user-data drive in the standard
topology (campaign F2, where=/mnt/teszt_enroll role=system). Add a mount-table
fallback mirroring durableIDForMount Impl-2b: device-keyed classification with a
whole-disk containment pass (new storage.SameWholeDisk) and the Observe-error
early return kept BEFORE the fallback (else a blind view -> permissive). Only
roleForMountPath touched. Tests A1/B1/B2/C1-C3 + 3 red-proofs; existing RoleGated
tests green unmodified.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
reachable now checks a bound listener via ss (the belt correctly drops even
localhost→felhom-sshd, so a dial always failed). operator_peer_configured derives
from the persistent wg-felhom.conf (2nd AllowedIPs /32) + operator_key from the
installed authorized_keys file — accurate immediately after an agent restart,
before the next desired-state fetch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Mirrors wgtunnel fetched=false-is-never-a-teardown: until the desired-state is
delivered, @operator_ips + felhom-op authorized_keys are left untouched (a nil
block on agent restart would otherwise empty @operator_ips and lock the operator
out until the next fetch). Belt.Sync split into SyncPort (always) + SyncOperator
(fetched only).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
On a continuous reconcile felhom-sshd itself listens on the claimed port, so
re-probing isFree(persisted) found it 'busy' by our own daemon and thrashed to
another candidate every tick. A persisted port is ours — keep it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
renderConf appends a validated, deterministically-SORTED oob_peer_ip /32 so the
operator peer survives self-heal/restart (a runtime wg set was wiped by self-heal).
WireWireguard gains oob_peer_ip (omitempty — absent = byte-identical pre-H1 conf).
Non-hollow tests: sorted+byte-stable, lower-IP-sorts-first, absent-identical,
invalid/v6 rejected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Prerequisite for felhom-sshd (H1). Closes the SPIKE-felhom-sshd §8 lockout: a
second sshd's RuntimeDirectory=sshd removed the SHARED /run/sshd privsep dir and
took stock sshd on :22 down (sessions reset after KEXINIT).
Host artifacts (configs/, installed by felhom-host-install):
- felhom-privsep.tmpfiles: layer 1, boot-persistent /run/sshd owned by no unit
- felhom-mgmt-watchdog.sh/.service/.timer: layer 2, AGENT-INDEPENDENT ~60s heal
(stat-first recreate + reset-failed sshd only if failed + heal-marker); never
RuntimeDirectory=, never restarts stock sshd, never touches a healthy dir.
Go (internal/mgmtplane): read-only Reporter → additive omitempty mgmt_plane
heartbeat stanza (privsep_dir_ok/sshd_reachable/healed_recently/privsep_healed_at),
wired via Collector.SetMgmtPlaneReporter. Non-hollow tests + red-proofs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
- reconcile: ClassAgentUpdate op class; always Destructive (no provenance
blesses replacing the root-adjacent binary). classify test + companion
(TestClassify_AgentUpdateAlwaysDestructive).
- opsign: `-op agent_update` with -agent-version + -sha256 (isHex64-validated);
params {version,sha256}. isHex64 test (Group D).
- config: SelfUpdateConfig{URLTemplate,Username,Token,StateDir,DwellSeconds}
+ WithDefaults + Token redaction.
- internal/selfupdate: Executor (download → verify vs the SIGNED sha → sudo -n
wrapper `apply`; sha is the only integrity root — mismatch refuses + removes,
agent untouched); Manager (startup dwell → `commit`; version-mismatch → no
commit + loud WARN + marker left for report visibility; shutdown-before-dwell
leaves pending). WrapperRunner seam → tests never shell out.
- hub report: additive selfupdate_pending(+version) via SetSelfUpdateReporter
seam; both omitempty (Wireguard precedent) so the cross-repo golden contract
stays byte-stable — no hub change.
- capability manifest: 3 non-critical FELHOM_SELFUPDATE probes.
- main.go: updateExec appended to the executor chain; commit-manager wired to
the report seam + MaybeCommit goroutine after core init.
Tests: Group A (executor happy/sha-mismatch+companion/bad-params/wrapper-fail),
B (agent_update rides the real gate: pinned-key executes, non-pinned +
retarget rejected), C (commit/version-mismatch/no-pending/shutdown), D (opsign).
C2 companion red-proof verified (neutered Go verify → bad binary reaches apply
→ test fails), reverted. Full go test ./... green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
wgtunnel.InstallRecoveredKey: write an escrow-recovered WG private key (create-
only, refuse-overwrite) so the tunnel re-establishes with the same identity/pubkey
(same /32), no keygen. Wired into identity-consume -install-wg-key (opt-in;
pre-S3 blob → logged fresh-keygen fallback). Value never logged.
internal/dr (new): consume the host_loss restore_directive (was logged-ignored)
into an inspectable RestorePlan via the AddConsumer raw seam — per guest
{vmid,archive,target,sizing} + per drive {durable_id→mount} + offsite PBS coord.
DERIVE-AND-SURFACE only; the Consumer has no restore/destroy dependency (execute-
nothing is structural). guest_loss/absent → no plan.
Tests + red-proofs (WG create-only overwrite; plan mode-gate). No secrets on
argv/stdout/logs. The destructive in-place restore is a separate operator-present
STOP-gated drill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
A WAN (pbs-tier) restore of a large guest exceeds the restore-task wait's 10m
default → the wait expired mid-restore, teardown fired against a still-restoring
(not-yet-pool-associated) scratch guest → leak + a phantom VM.Allocate 403.
- RestoreTestSpec.RestoreTaskTimeout (0→10m default); the restore WaitTask passes
it. Local tier unchanged (10m).
- config RestoreTestPBSRestoreTimeoutSeconds + accessor (default 120m).
- main restoreTaskTimeout(cfg,tier): configured PBS timeout only when tier==pbs,
else 0. Both scheduler + selftest spec builds.
- Tests + WaitOptions red-proof + accessor contract.
The "grant scratch-band VM.Allocate" follow-up is diagnosed not blind-applied:
the scratch is restored INTO /pool/felhom (ACL already grants VM.Allocate), so
the earlier 403 was a consequence of the timeout. No ACL/host-install change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Phase-1 live probe (felhom-hetzner) proved backup/restore/list/isolation over
the tunnel with a per-customer DatastoreBackup token, but the agent's PBS client
was namespace-unaware: Snapshots hit the datastore root (403 for a scoped token)
and Verify was whole-datastore (needs Datastore.Verify ~ admin). Operator-
approved fix.
- pbs.Config.Namespace + Client.namespace; Snapshots appends ?ns=; Verify sends
ns= (ns-scoped verify works with DatastoreBackup on the own ns — no admin
widening, Phase-1 confirmed). Root-ns clients unchanged (whole-datastore).
- proxmox.Storage.Namespace (parsed from /storage `namespace`).
- pbsTargetsFromPVE threads s.Namespace into the client.
Confirmed tenant ACL: DatastoreBackup on /datastore/felhom-offsite/<ns> (NOT
/ns/<ns>) to BOTH felhom@pbs (user) AND felhom@pbs!<ns> (token) — PBS privsep =
intersection; isolation holds (cross-ns 403 proven). TestClient_NamespaceScoping
red-proofed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Claude Code
admin