The ticker survives as the EVALUATION interval only. A tier is DUE when its
newest archive that has settled for `settle` (default 24h) has not been proven:
daily tier -> proved daily on yesterday's archive, weekly tier -> weekly on its
own, newborn -> UNKNOWN.
The trap avoided: the literal reading ("newest archive is >= 24h old") is NEVER
true on a daily tier, so it silently switches restore-testing off where it
matters most. Red-proved at 0 runs over 5 simulated days.
- state records WHICH archive was proven; legacy files keep their time and yield
no proven archive (each tier due once after the upgrade, deliberately)
- two knobs replace one: restore_test_eval_interval_seconds (6h, measured) and
restore_test_settle_seconds (24h). The old cadence key keeps its DISABLE
meaning verbatim and now seeds the settle lag, with a start-up WARN.
- due-check runs BEFORE the heavy-op gate (a frequent poll must not make a
starting backup record a failure, F-A1)
- candidate picker skips implausible archives (a phantom would be due forever)
- new read-only --selftest=restore-test-due prints the verdict + its cost
Ruling 1 (2 weeks of weekly offsite backups): localPruneSpec's blanket PBS
refusal is now scoped — an ADDITIONAL tier with an explicit keep_last may
prune its PBS target. The refusal still applies in full to the PRIMARY tier,
because BackupTarget() defaults to felhom-pbs and KeepLast() defaults to 3, so
a box with neither key set would silently prune its offsite DR to 3 restore
points. An additional tier cannot have that accident (keep_last defaults to 0).
Ruling 3 (first backup runs as long as needed; nothing else starts until done):
- additional-tier wait bound 6h -> 12h (measured ~33 MB/min => ~5h for a first
full 10 GB snapshot; 12h gives margin but stays bounded so a hung task still
surfaces)
- ONE BACKUP AT A TIME PER GUEST across all tiers: POST /backup returns 409
when a DIFFERENT tier is in flight, naming the busy tier, with NO data object
so nothing is parseable as the caller's own job. Same tier still returns that
job (202, unchanged).
- snapshotted now counts as in-flight, not just running — after the snapshot the
vzdump is still uploading and holding the lock. The old check left a window
where a second POST started a real second vzdump. Latent bug, closed.
Full suite green (29 packages); red-proof observed and restored.
Found by live validation on demo-felhom, not by review.
The first real PBS-targeted backup ran past the runner's hard-coded 30-minute
WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE
VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences:
the tier stays permanently due, the next attempt collides with the guest lock
the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's
'applied and empty' fault re-created by a timeout.
Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot
projects to ~5h.
- BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED
(a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from
the measurement.
- backup.NewBackupRunnerWithWait: per-instance (per-tier) bound.
NewBackupRunner keeps its signature, so restore-test/selftest are untouched.
- localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the
tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h
outer context reproduces the same false failure four hours later.
Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait
LONGER. A slow backup is a slow backup; a false timeout is a corrupt status
plus lock contention.
Red-proof observed and restored; full suite green.
Mechanism only. No box changes behaviour until a backup_targets entry is
added to its config (Slice D); an untouched config resolves to exactly one
tier and behaves byte-identically to v0.96.0.
- config: BackupTargetConfig + ExtraTargets + BackupTiers(); each tier carries
its OWN cadence and retention (keep-last=3 is three days on a daily tier and
three weeks on a weekly one). A missing cadence is REJECTED, not defaulted —
a weekly DR tier silently running daily would fill the 37.2 GB datastore.
main.go logs every rejection at ERROR.
- /backup/due?target= judges a tier against its OWN newest successful backup.
Without that filter a fresh local backup satisfies the weekly PBS cadence and
the DR tier never runs — today's bug, re-created in code.
- GET /backup/tiers advertises the tiers; a 404 is the controller's pre-R-82
capability probe (Slice B).
- Jobs keyed by (vmid,target): single-flight is per tier, which is what lets
the weekly night run both backups in ONE quiesce window. Job ids are unique
per tier by construction, not by clock luck.
- One runner per tier: the runner holds target+retention as immutable state,
so parameterising one runner would risk pairing tier A's target with tier B's
retention.
COMPATIBILITY (frozen): untargeted /backup/due, POST /backup and
/backup/status keep the primary tier and the pre-R-82 response BYTES —
Target is omitempty and stays empty. The primary's job-id format is unchanged.
NOT changed: the local tier; PBS is still never pruned by the per-run flag
(keep_last defaults to 0 = never prune — enabling DR pruning is irreversible
and needs an operator ruling).
Tests 748->768. Red-proof #1 observed and restored.
Phase 0: felhom.eu/documentation/audits/SPIKE-r82-phase0-2026-07-26.md
Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient
is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20
the guest ran another ~80 minutes on its unexpired lease, then lost its address and default
route and took the tunnel, hub reports, catalog sync and the controller->agent channel with
it (1h15m outage, healthy-looking for the first 80 minutes).
So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment
`pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address
to vanish is waiting out the silent window.
internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces
mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's
heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min
between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on
a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source
is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as
unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never
probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able
to kill the agent.
guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only
inward at guests we already own, and the failure exists on every box today.
Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN
direction in this repo, report stanzas are *Status.
Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy"
with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test.
Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not).
- reconcile: ClassAgentUpdate op class; always Destructive (no provenance
blesses replacing the root-adjacent binary). classify test + companion
(TestClassify_AgentUpdateAlwaysDestructive).
- opsign: `-op agent_update` with -agent-version + -sha256 (isHex64-validated);
params {version,sha256}. isHex64 test (Group D).
- config: SelfUpdateConfig{URLTemplate,Username,Token,StateDir,DwellSeconds}
+ WithDefaults + Token redaction.
- internal/selfupdate: Executor (download → verify vs the SIGNED sha → sudo -n
wrapper `apply`; sha is the only integrity root — mismatch refuses + removes,
agent untouched); Manager (startup dwell → `commit`; version-mismatch → no
commit + loud WARN + marker left for report visibility; shutdown-before-dwell
leaves pending). WrapperRunner seam → tests never shell out.
- hub report: additive selfupdate_pending(+version) via SetSelfUpdateReporter
seam; both omitempty (Wireguard precedent) so the cross-repo golden contract
stays byte-stable — no hub change.
- capability manifest: 3 non-critical FELHOM_SELFUPDATE probes.
- main.go: updateExec appended to the executor chain; commit-manager wired to
the report seam + MaybeCommit goroutine after core init.
Tests: Group A (executor happy/sha-mismatch+companion/bad-params/wrapper-fail),
B (agent_update rides the real gate: pinned-key executes, non-pinned +
retarget rejected), C (commit/version-mismatch/no-pending/shutdown), D (opsign).
C2 companion red-proof verified (neutered Go verify → bad binary reaches apply
→ test fails), reverted. Full go test ./... green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
A WAN (pbs-tier) restore of a large guest exceeds the restore-task wait's 10m
default → the wait expired mid-restore, teardown fired against a still-restoring
(not-yet-pool-associated) scratch guest → leak + a phantom VM.Allocate 403.
- RestoreTestSpec.RestoreTaskTimeout (0→10m default); the restore WaitTask passes
it. Local tier unchanged (10m).
- config RestoreTestPBSRestoreTimeoutSeconds + accessor (default 120m).
- main restoreTaskTimeout(cfg,tier): configured PBS timeout only when tier==pbs,
else 0. Both scheduler + selftest spec builds.
- Tests + WaitOptions red-proof + accessor contract.
The "grant scratch-band VM.Allocate" follow-up is diagnosed not blind-applied:
the scratch is restored INTO /pool/felhom (ACL already grants VM.Allocate), so
the earlier 403 was a consequence of the timeout. No ACL/host-install change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
v4-pin (doc 06 §4.2): renderConf takes a pre-resolved IPv4 literal and writes
Endpoint=<ip>:<port> — never the DNS name, never AAAA. Resolver seam (A records
only, LookupNetIP "ip4"); multiple A → lowest (deterministic fleet-wide);
renderConf stays pure. Resolved IP cached: steady-state Apply = zero DNS + zero
execs. DNS failure keeps the last conf (never a teardown).
Watchdog (loop-only, so Apply's zero-exec steady state is untouched): handshake
age > wg_tunnel.stale_after_seconds (default 180) → re-resolve; IP changed →
re-render + restart (endpoint re-IP recovery); IP same → no churn (throttled
warn). Staleness read reuses wg show latest-handshakes (never dump).
Capability: wg-conf-install/enable/restart/handshake-read flipped Critical=true
(backups ride the tunnel from S4); apt-install + disable stay non-critical.
TestWGCapabilityCriticality pins the set.
Tests + red-proofs a/b/d all fire. No new sudoers grant; no wire/JSON change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Sudoers: fixed-path conf install, enable/restart/disable, latest-handshakes-only
wg read (dump FORBIDDEN — the S1 incident). 6 capability-manifest entries
(Critical=false until S4 makes the tunnel load-bearing). WGTunnelConfig with
enabled=false DEFAULT (the safety gate: a v0.64.0 rollout without explicit
config is a no-op). Daemon wiring mirrors lanresolver + AddConsumer +
SetWireguardReporter; --selftest=wgtunnel single-shot. IdentityBundle
+wg_private_key (omitempty; pre-S3 blobs cannot be retrofitted — documented)
with escrow-create auto-inject (field name only in logs). Red-proof (e) run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The preventive counterpart to host_disk + storage_fill detectors: the periodic local
whole-guest vzdump now prunes its own old archives (keep-last=3, clamped >=1) so a box
can't refill its own root via its own backups. Local target only — PBS never pruned
(resolved via ListStorage; fail-safe skip on unknown). Seeded in host-install.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
Host-side NFS/SMB automount of a bulk-media NAS share under /mnt/felhom-drives/<name>
(propagates into the guest via the existing shared bind), the +100000 uid recipe,
per-share liveness, and add/list/remove local-API endpoints. A NAS is a distinct class
that bypasses the drive enroll/eject/decommission/SMART/watchdog machinery.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
- BackupConfig.BackupTarget() defaults whole-guest backup to felhom-pbs (separate
hardware = real DR), configurable via backup.local_backup_target; all NewBackupRunner
sites route through it. PBS round-trip proven live (snapshot marker + restore-test +
pct-restore) before the re-point.
- signedjobs DecommissionExecutor + ExecutorChain: makes IntentDecommissioned reachable
ONLY via a verified operator signature (keyed by the watchdog's storage durable-id);
felhom-opsign builds decommission params from -durable-id. Runner wiring moved below
the intent-store open.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Host-side dnsmasq the agent manages so LAN clients reach their guest directly
(same hostname + real wildcard cert, no Cloudflare hairpin). Renders local=/
+address=/ per customer (AAAA->NODATA via authoritative zone, wildcard A ->
live guest IP), forwards everything else. Manager ensures dnsmasq+base config,
discovers guest IP (pct exec ip) + domain (controller.yaml), write-if-changed +
reload. Loop (7th daemon goroutine) tracks DHCP IP changes per provisioned
guest. --selftest=lanresolver. FELHOM_DNSMASQ sudoers. Spiked live on felhom-pve.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
internal/localapi: real /backup/due (cadence; due when no successful backup or
newest older than backup.backup_cadence_seconds; false in-window after success;
failed doesn't count) + /backup/status phases (idle|running|done|failed + job
id) + POST /backup single-flight with job id. Drives the controller quiesce loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
internal/escrow: zero-knowledge escrow creation. R = 10 EFF-wordlist words
(crypto/rand, ~129 bits); wrap K under R via PBS-native key change-passphrase
driven over a stdlib pty (x/sys/unix; output discarded so R can't leak, F-A2);
self-verify the blob recovers K (fingerprint match) before shipping. Opt-in (b)
R-wrapped offline copy + (a) raw paperkey. Live K is byte-unchanged (operates on
a copy). --selftest=escrow-create (-storage/-paperkey/-offline/-upload). Posture
config field (zero_knowledge default). PBSEncKeyPath helper. Grounded by the
escrow spike findings.
Tests: R entropy>=128/format/uniqueness; integration round-trip (wrap->unwrap
fingerprint match, wrong-R fails, K byte-unchanged) guarded to linux+pbc.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spike-proven that backup/restore-to-PBS reuse Phase A unchanged; the only new code is
the verify capability, a small PBS-API client, and PBSSnapshot reporting.
- internal/pbs: fingerprint-pinned, token-authed PBS-API client (Verify/Snapshots/
TaskStatus, node-from-UPID; secret read from /etc/pve/priv/storage/<id>.pw at runtime,
never logged) + the verify maintenance loop (own cadence, default 6h, NOT gated/journaled,
like the watchdog) + SnapshotStore.
- hub: PBSSnapshot filled (namespace/type/id/time/size/owner/protected/encrypted/
verify_state/verify_upid); PBSReporter collector seam; cross-repo golden + bidirectional
key-set tests; hub handler parses pbs_snapshots + logs a failed-verify WARN.
- backup: report the ACTUAL vzdump mode (parsed from the task log; PVE may downgrade
snapshot->stop). proxmox.Storage.Username. config PBSVerifyCadence/secret-dir.
--selftest=pbs-verify. Backup/restore-to-PBS unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fill the slice-3 storage_targets stub and add the fast-poll storage watchdog.
Read-only this phase; the host-root surface (mounts/SMART/grow/destructive gate)
is Phase B. Hub-owned desired manifest is slice 10, so reconcile against it is
built-but-unfed.
- internal/storage: StorageTarget wire contract, durable_id derivation per type,
HostReader seam (procfs/sysfs, root-free), Observer (storage_targets from
ListStorage/NodeStorage + host reads, lvmthin thin-pool fill), and the watchdog
(third daemon goroutine; debounced out-of-band report on a known target's
attach/disconnect transition).
- proxmox.Storage: additive parse-only config fields (durable_id sources).
- collector StorageObserver seam; Loop.SetTrigger out-of-band report; daemon runs
the watchdog as a third goroutine; StorageConfig knobs.
- cross-repo golden kept byte-identical with felhom.eu/hub; bidirectional key-set test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
internal/hub: the agent's first daemon — a periodic read-only host-report POSTed to
the hub (the heartbeat; no separate ping).
- HostReport wire contract (shared field-for-field with the hub ingest): host
metrics, guests (vmid + spec), cloudflared status; storage/backups/restore-tests/
pbs/audit collections DEFINED but emitted empty (slices 5/6 fill).
- Collector over a read-only proxmoxReader (adapted to the real proxmox surface;
no proxmox changes) + a CloudflaredProber. Partial-failure: NodeStatus fail = hard
(skip POST); per-guest GuestConfig fail = status "unknown", still report.
- Client: Bearer-auth POST, standard TLS (system roots / optional ca_file), typed
TransportError/HTTPError, token never in errors.
- Loop: immediate first report, adopt hub poll_interval (clamp [60,3600]), resilient
to collect/report errors, clean ctx-cancel shutdown.
- ControlEnvelope: only poll_interval_seconds acted on; blocked/desired_generation/
has_signed_ops parsed-but-ignored (slice 4).
- config: HubConfig + FELHOM_AGENT_HUB_* overlay + mode-aware HubConfig.Validate +
WithDefaults + hub-key redaction; example config updated.
- main: no-selftest mode is now the daemon; added --selftest=hub. Version -> 0.3.0.
Tests: report serialization, client (incl. token-redaction), collector partial-
failure, loop continuation+interval adoption, config. internal/proxmox + internal/
authz untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stand up the felhom-agent project (module gitea.dooplex.hu/admin/felhom-agent,
binary felhom-agent) and the internal/proxmox package: the typed library every
other agent module calls to talk to Proxmox.
- API-first Client (hand-rolled REST over net/http, PVEAPIToken auth) with typed
read ops (version/nodes/status/lxc/config/storage) and async mutating ops
(restore/vzdump/snapshot/rollback/delete-snapshot/setconfig/start/stop), each
returning a UPID. WaitTask polls task status until stopped and asserts
exitstatus OK (authz can surface at task exec, not the POST — phase1-2 §1.3).
- Fenced Privileged (root-CLI) backend for the THREE proven exceptions only
(keyctl pct create, USB mount/fstab, SMART/sensors); each cites why it can't be
the API. Fence is structural (Client never shells out, Privileged never HTTPs)
and asserted in routing_test.go.
- TLS: SHA-256 leaf-cert pinning or CA file; insecure mode explicit + off by
default. No blanket verification disable.
- 403 -> privilege-named APIError; failed task -> privilege-named TaskError.
- JSON config + env overrides (token never logged); slog logging.
- cmd/felhom-agent --selftest (read-only health report) + gated --selftest=task
(reversible snapshot/rollback/delete exercise of WaitTask). No daemon loop yet.
- Types grounded in the spike findings and exact JSON shapes captured live from
demo-felhom (PVE 9.2.2). Unit tests use a mock transport + runner.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin
Claude Code