Commit Graph

31 Commits

Author SHA1 Message Date
admin 1c8a67eece R-106 + R-109: the DR recipe records the resolved namespace and names the backup target (v0.118.0)
Both defects were live on both demo boxes: the recipe said namespace "root" while
storage.cfg said demo-felhom/demo-hp, and it never named which of two content=backup
dir storages holds the local archives.

R-106: the namespace came from the listed snapshot, but PBS omits `ns` per item once
the list is namespace-scoped, so it was always empty and normalised to "root". It now
resolves from the pbs STORAGE (storage.cfg's `namespace`) — the same field vzdump makes
PVE read, so the recipe cannot disagree with the backup.

R-109: backup_target resolves from the primary tier of cfg.Backup.BackupTiers(), the
function the scheduler consults, and carries the mountpoint that separates /mnt/hdd_1
from /var/lib/vz. The resolver reports the tier IN EFFECT (daemon-start config), not
agent.json on disk — a target move rewrites the file and deliberately does not restart.

Unresolvable is recorded as unresolvable: resolved|unknown plus a distinct reason,
never a default, an empty string, or a placeholder.

Needs hub v0.83.0 — AssembleDRRecipe allow-lists top-level keys, so backup_target
would otherwise be stored intact and dropped before any operator saw it.

9 tests, 4 red-proofs (each mutation asserted to have landed). Suite rc=0, 29 ok.
2026-07-30 13:11:08 +02:00
admin 21b0164fad R-116 (v0.116.0): give the backup-target flag and the gate's key the same row
The absent-drive alarm was generic while its recovery was specific -- a pair an
operator cannot match. Mechanism now measured, not reasoned (felhom.eu
audits/DIAG-r116-disks-payload-2026-07-30.md): with the device gone /disks returns
4 rows, not 3. The drive appears TWICE and the two facts the controller needs are
on different rows -- the Observe row has backup_target:true but mount_path:"" and
guest_path:"" (so driveTargetByPath registers NO key from it), while the registry
row owns /mnt/felhom-drives/<name>, the key the gate looks up, with BackupTarget
absent from its struct literal => false.

WHY v0.115.0 WAS INERT: its fallback computed StablePathForRaw(t.MountPath), and in
the absent state MountPath is ALSO "" -- emptied by the same exactMount failure
that empties BackingDevice. It assigned nothing. Its test passed because the fixture
supplied a MountPath production never supplies, and the harness left DriveTargets
nil so the union loop never ran. Both corrected here; red-proof 1 replays v0.115.0's
exact code against the real shape and it fails.

THE JOIN, which was the hard part: with the device gone the two records share no
runtime field -- no mount, no backing device, and the Observe row's DurableID has
degraded off the fs-UUID. They share CONFIGURATION: storage.cfg's path on one side,
the .mount unit's Where on the other, both yielding the same stable guest path. New
hub.StorageTarget.ConfigPath (json:"-" -- that struct is a cross-repo contract
pinned by the golden + contract_test key-set comparison, and nothing off-box needs
the value), set from s.Path in observe.go, consulted in disks.go only after MountPath
so the present-state path is byte-identical, plus a guest-path arm on the union dedup
so exactly one row carries the drive.

WHY NEITHER OBVIOUS OPTION WAS TAKEN -- both regress R-114, which shipped yesterday.
backup_target_offer.go:79 reads (BackupTarget && MountPath != "") as "a real
drive with its own mountpoint -- healthy" and returns before its TargetAbsent
branch. Back-filling MountPath onto the Observe row (the smallest change, and the
spec's lean) and teaching the registry row the flag (its MountPath is non-empty, read
from the stale unit file) BOTH manufacture that row while the drive is missing, which
would have told the customer the backup target is fine while its drive is gone.
R-114's correctness rests on the absent-state rows not combining the flag with a
mount path; that coupling was invisible until the payload existed. Pinned by
TestAbsentTargetKeepsR114DegradedSignal.

Role unchanged, BoundUnderParent conjunction not widened, no wire field changed.
Suppressing the registry row in the absent state also removes its false
state:"attached" and its root-filesystem-derived total_bytes -- R-118's symptom
goes incidentally; R-118 is NOT fixed and stays open.

Tests 845 -> 849, suite rc=0 read separately from this commit. Four red-proofs, each
mutation asserted to have landed first.

NOT live-validated at this commit: publish+vouch, C5, discrimination, over-correction.
2026-07-30 08:48:31 +02:00
admin ed97232598 v0.95.0: SMART coverage — union-path drives + LVM/dm root + device model
Implements SPIKE-smart-coverage-2026-07-25 fixes B+A (additive; MinAgent unchanged).
Fix B: storage.SmartReader.SMARTForBacking wired into the /disks union path (localapi
Smart seam) so registry/USB drives get a real SMART read (watchdog Known stays
enrich-free). Fix A: smartDeviceFor resolves dm/LVM to the whole disk via
/sys/block/<dm>/slaves (recursive; skips >1-disk); the builtin local dir on the LVM
root gets a SMART-only device from its containing filesystem (never touches
backing/durable_id). SmartSummary.ModelName captured from smartctl. Fix C (-d sat)
stays rejected. Tests + red-proofs (dm multi-disk skip, enrich smartHint, union
routing); Known-path-never-SMARTs asserted.
2026-07-25 08:21:45 +02:00
admin c0966d753d feat(v0.92.0): guest-network watchdog (R-54) — supervise the guest's DHCP client
Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient
is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20
the guest ran another ~80 minutes on its unexpired lease, then lost its address and default
route and took the tunnel, hub reports, catalog sync and the controller->agent channel with
it (1h15m outage, healthy-looking for the first 80 minutes).

So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment
`pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address
to vanish is waiting out the silent window.

internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces
mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's
heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min
between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on
a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source
is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as
unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never
probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able
to kill the agent.

guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only
inward at guests we already own, and the failure exists on every box today.

Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN
direction in this repo, report stanzas are *Status.

Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy"
with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test.

Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not).
2026-07-21 12:29:37 +02:00
admin b2ca63ee9f v0.91.0 — the DR tier can no longer be applied and dead at the same time (R-39 + R-50b(a))
Closes the agent half of R-39's fleet fix. Requires hub >=0.68.0 for the re-arm signal;
that hub is safe for 0.90.0 agents (unknown key dropped), so it deploys first.

Three compounding defects let a box report `applied` while every PBS request 401'd:

1. The re-key was INVISIBLE. An ep0 re-issue rotates the secret of an existing token, so
   token_id/fingerprint/datastore/namespace come back byte-identical and the descriptor
   content hash never moved — the converged agent short-circuited and never consumed the
   fresh secret. WirePBSDR.SecretGeneration (field-exact with the hub) is what moves the
   hash now, because descriptorHash marshals this struct.

2. The agent could not READ its own credential. It writes /etc/pve/priv/storage/<id>.pw
   through the root wrapper, but that dir is 0700 root:www-data and the wrapper had no
   read verb — so the target resolver got "permission denied" every cycle, warned, and
   skipped. The one loop that could have caught the 401 was blind BY CONSTRUCTION. Adds a
   narrow `read` verb (+ exactly one sudoers line, + a pbsdr-read capability row): one
   secret to stdout, no network, no mutation, never in argv (sudo logs argv), traversal
   refused by the id grammar, the dir allowlist AND a resolved-path prefix assertion.

3. Nothing probed AUTHENTICATION. pbs.ProbeAuth (GET /version + an ErrUnauthorized
   sentinel) runs on the 15-minute collect path and its verdict becomes a loud
   `auth_failed` the hub escalates to a fresh mint. /version needs no datastore, namespace
   or privilege, so a 401 means the CREDENTIAL is bad; 403 is deliberately NOT treated as
   unauthorized, since re-keying a too-narrow token would mint forever without fixing
   anything. A transport error is UNKNOWN, never a rejection — otherwise every network
   blip burns a credential. Recovery self-clears.

R-50b(a): the report now carries the installed wrapper's sha256 so drift against the
vouched manifest value is answerable. Empty = unknown, never drift.

Three red-proofs, all at the assertion level. Removing SecretGeneration fails the re-arm
test with "consume calls=1, want 2". Swallowing the probe result leaves State:applied
AuthFailed:false — the July-18 shape exactly. Notably, deleting the wrapper's id charset
guard alone does NOT open a traversal hole (readlink + the prefix assertion still catch
it), so the isolating red-proof removes BOTH and shows the out-of-tree secret printed —
the layering is real, and a single-guard red-proof would have passed vacuously.
2026-07-21 10:12:31 +02:00
admin a659e5dc09 v0.89.0: pbsdr self-grant (R-22) + escrow config live-reload + agent-plane poke listener (Direction-2a)
- pbsdr: on a 403 pre-check (non-default storage id, no ACL yet) self-grant via the root wrapper then re-read, instead of aborting before the grant — closes F4/R-22. Red-proof TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant.
- escrow preflight: late-bound CurrentPBSStorageID re-reads agent.json so a pbsdr-seeded pbs_storage_id flips the row green in-process (no restart). Red-proof TestEscrowPreflight_PBSStorageIDLiveReload.
- internal/poke: contentless UDP poke listener bound exclusively to the box WG /32 (port 51822), leading-edge debounced, fires the hub-loop out-of-band trigger for an immediate desired-state cycle. First slice of R-13. Red-proofs TestBindConfinement + TestDebounceCoalescesBurst.
2026-07-16 22:47:22 +02:00
admin cb692f8788 v0.83.0: observability pass — always-DEBUG capture ring + GET /debug/logs + heartbeat log-pull + gap-fill sweep
Capture layer: applog.New returns (logger, Ring) — slog fan-out, stderr at the
configured level, ~1000-entry ring fixed at LevelDebug (remote diagnostics
without a config flip). GET /debug/logs (token-authed, ?raw=1) + request-level
DEBUG middleware. Heartbeat log-pull mirrors the report logtail pattern:
envelope log_tail_requested -> next heartbeat carries log_tail (128KB cap,
consume-once, failed-push retry proven). Gap-fill sweep over netverify/
netstorage/netmount/signedjobs/selfupdate/disks/controller-swap/desired/loop.
Red-proofs: ring-at-emit-level FAILs capture test; drain removed FAILs
consume-once; dropped phase line FAILs the S7 log-sequence smoke.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 16:24:07 +02:00
admin e5e8f3920a v0.80.0: PBS DR tier slice 2 — the apply-bridge (pbs_dr consumer, felhom-pbs-apply set-only wrapper, verify-pin-before-consume, adoption-first, loud consumed-failed, escrow seed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-10 21:50:36 +02:00
admin b1697874ec GL-5b: restore-test full-fidelity verification (v0.76.0)
The restore-test had GL-5 finding #2's mirror image: its live-source-config
bind-override path tripped PVE's drop-unlisted-mountpoints rule, so scratch
guests boot-verified WITHOUT their storage mpN - weaker verification than
claimed. Params now derive from the ARCHIVE's own embedded config via
ExtractArchiveConfig + drRestoreOverrides (the object under test; full
layout, content genuinely extracted - the added runtime IS the
verification); unreadable/unknown-topology archives refuse up front. NEW
mount-parity assert (2b, pre-start): restored mpN set vs the archive's -
missing/mispathed/undersized/extra mpN fail the test naming the delta, so
constraint (b) can never regress into a green light. MountParity +
MountInventory ride the result + hub wire record (additive). Dead
bindMountOverrides/archiveVMID path deleted with its tests (no reachable
lookalike). DR bring-up untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-08 09:46:30 +02:00
admin 9d9e4a5ea7 test(hub): collector OOB stanza wiring (H1)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 22:30:01 +02:00
admin c983a25609 feat(felhomsshd): dedicated OOB sshd instance + port-adaptive belt (H1 Parts 2-4 agent)
internal/felhomsshd: agent-managed felhom-sshd (claim port [8822,2222,8022,62222]
loud-fail-on-exhaustion; render config→sshd -t→reload never-restart-on-change
[SF-2]; operator authorized_keys from the hub block outside ~/.ssh [SF-3]); the
static-table nft belt mutating ONLY @operator_ips + @ssh_port [trap 4]; health/heal
(reset-failed-then-restart with 10min cooldown, NEVER restart onto an invalid
config) + the oob heartbeat stanza. configs/felhom-sshd.service (SAFE, no
RuntimeDirectory [SF-1]). FELHOM_SSHD + FELHOM_OOB sudoers (set-elements only).
oob.enabled config DEFAULT FALSE. Wired into main like wgtunnel.

Non-hollow tests: claim clean/contention/idempotent/exhaustion; config
safe+byte-stable+refuses-:22; belt mutate-then-idempotent + never-touches-rules;
heal no-restart-on-invalid-config + cooldown; status reflects block.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 22:27:02 +02:00
admin effff53f99 feat(wgtunnel): render operator OOB /32 into wg-felhom AllowedIPs (H1 Part 1, [OF-1])
renderConf appends a validated, deterministically-SORTED oob_peer_ip /32 so the
operator peer survives self-heal/restart (a runtime wg set was wiped by self-heal).
WireWireguard gains oob_peer_ip (omitempty — absent = byte-identical pre-H1 conf).
Non-hollow tests: sorted+byte-stable, lower-IP-sorts-first, absent-identical,
invalid/v6 rejected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 22:08:04 +02:00
admin fd4e177216 feat(mgmtplane): break-glass privsep-dir watchdog + mgmt_plane health (TASK G1) — v0.71.0
Prerequisite for felhom-sshd (H1). Closes the SPIKE-felhom-sshd §8 lockout: a
second sshd's RuntimeDirectory=sshd removed the SHARED /run/sshd privsep dir and
took stock sshd on :22 down (sessions reset after KEXINIT).

Host artifacts (configs/, installed by felhom-host-install):
- felhom-privsep.tmpfiles: layer 1, boot-persistent /run/sshd owned by no unit
- felhom-mgmt-watchdog.sh/.service/.timer: layer 2, AGENT-INDEPENDENT ~60s heal
  (stat-first recreate + reset-failed sshd only if failed + heal-marker); never
  RuntimeDirectory=, never restarts stock sshd, never touches a healthy dir.

Go (internal/mgmtplane): read-only Reporter → additive omitempty mgmt_plane
heartbeat stanza (privsep_dir_ok/sshd_reachable/healed_recently/privsep_healed_at),
wired via Collector.SetMgmtPlaneReporter. Non-hollow tests + red-proofs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 18:49:27 +02:00
admin 8033a522cd feat: D1 Part 2 — agent self-update Go plumbing (op class, opsign, executor, commit, report)
- reconcile: ClassAgentUpdate op class; always Destructive (no provenance
  blesses replacing the root-adjacent binary). classify test + companion
  (TestClassify_AgentUpdateAlwaysDestructive).
- opsign: `-op agent_update` with -agent-version + -sha256 (isHex64-validated);
  params {version,sha256}. isHex64 test (Group D).
- config: SelfUpdateConfig{URLTemplate,Username,Token,StateDir,DwellSeconds}
  + WithDefaults + Token redaction.
- internal/selfupdate: Executor (download → verify vs the SIGNED sha → sudo -n
  wrapper `apply`; sha is the only integrity root — mismatch refuses + removes,
  agent untouched); Manager (startup dwell → `commit`; version-mismatch → no
  commit + loud WARN + marker left for report visibility; shutdown-before-dwell
  leaves pending). WrapperRunner seam → tests never shell out.
- hub report: additive selfupdate_pending(+version) via SetSelfUpdateReporter
  seam; both omitempty (Wireguard precedent) so the cross-repo golden contract
  stays byte-stable — no hub change.
- capability manifest: 3 non-critical FELHOM_SELFUPDATE probes.
- main.go: updateExec appended to the executor chain; commit-manager wired to
  the report seam + MaybeCommit goroutine after core init.

Tests: Group A (executor happy/sha-mismatch+companion/bad-params/wrapper-fail),
B (agent_update rides the real gate: pinned-key executes, non-pinned +
retarget rejected), C (commit/version-mismatch/no-pending/shutdown), D (opsign).
C2 companion red-proof verified (neutered Go verify → bad binary reaches apply
→ test fails), reverted. Full go test ./... green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 15:32:15 +02:00
admin 312fd5ee29 wgtunnel: client MTU 1420 → 1280 (S3.1, resolve §4.3 open decision)
The 2026-07-04 CGNAT smoke test found MTU 1420 silently black-holes bulk TCP
on sub-~1480 paths (mobile ~1400, DS-Lite ~1452): handshake+ping stay healthy,
PBS TLS page (and at S4 the backup itself) drops. Set a fleet-wide, permanent,
family-agnostic client MTU of 1280 (RFC 8200 IPv6-minimum floor; outer 1340 v4
/ 1360 v6 fits every realistic path).

Client-only by construction — interface MTU caps box→PBS, advertised MSS caps
PBS→box; the endpoint's wg0 is untouched (zero live-endpoint risk). New const
clientMTU=1280 as the single home; golden pins exact "MTU = 1280" (red-proofed
against a 1420 flip). Stale report.go comment updated. No wire/JSON change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 13:57:20 +02:00
admin fb248961c6 wgtunnel: S3 Part 2 — manager state machine + loop + desired raw-consumer seam
Manager: one-shot registration (marker gate; backoff cap 15m), adopt-lost-marker,
re-key-on-mismatch, REVOKED-STAYS-REVOKED teardown (marker kept, zero execs on
later ticks), no-teardown-on-absent-data, hash-gated apply (zero execs steady
state), restart-not-reload on conf change, self-heal enable. Status stanza with
latest-handshakes-ONLY wg read. Collector WireguardReporter seam. desired.Syncer
AddConsumer fan-out with panic containment. Red-proofs a/b/d run + reverted;
no-key-material-in-logs asserted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 07:09:01 +02:00
admin 0daae92c4f wgtunnel: S3 Part 1 — pure-Go keygen + hub wire (WireWireguard, report stanza, RegisterWG)
key.go: create-once 0600/0700, corrupt-refusal (never overwrite — may be escrowed
identity), clamp for CANONICAL STORED form (x/crypto X25519 clamps derivation
internally — discovered during red-proof (c); the stored-clamped test is the
real anchor). Fixed vectors generated with real wg pubkey (provenance in test).
hub: WireDesiredState.Wireguard + WireguardStatus report stanza + RegisterWG
client (typed errors, token-free). S2 golden copied BYTE-IDENTICAL + field-exact
decode test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 07:00:19 +02:00
admin bf8e3be3f4 agent: report served local-API leaf fingerprint (hub re-key detection, Part A) v0.48.0
HostReport.LeafFingerprint rides the served fp (from EnsureLeaf) on every report; empty when local
API disabled. Collector.SetLeafFingerprint threads it like Capabilities. Hub watches it for a re-key.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
2026-06-29 23:14:52 +02:00
admin 25024d9dda capability: agent privileged-capability self-probe (manifest + build-test + runtime snapshot) v0.44.0
New internal/capability: Manifest of required sudo -n grants + Prober that LISTS each
via 'sudo -n -l' (never executes) + binary-exists check → ok/degraded snapshot on the hub
report. Build-time test asserts manifest⊆sudoers (red-proof: dropping lxc-info FAILs the
gate). Startup logs N/N ok + ERROR per degraded. Serve-degraded; no allowlist change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
2026-06-29 18:43:49 +02:00
admin fbe113011d v0.39.0 — DR-recipe completion: live PBS coord + drop role/restic_repo_coord from v1 drive shape
Live PBS coord: new internal/pbs/live_reporter.go (LiveSnapshotReporter implements
hub.PBSReporter via the cheap Client.Snapshots() list with last-known-good fallback,
bounded by an 8s timeout, list-only — never triggers a verify). Closes the gap where
the recipe's pbs block was omitted whenever the verify-loop SnapshotStore was empty
(one-shot collect + the first ~6h after a daemon restart). SnapshotStore.Get added
(per-datastore LKG). Wired into the collector in both runDaemon and runSelftestHub;
the verify loop keeps Recording into the SAME shared store via one hoisted pbsTargets.

v1 host-half drive shape: dropped drives[].role (hub/operator-owned manifest concept,
not host-derivable) and drives[].restic_repo_coord (named a backup tier that doesn't
exist). Drive shape is now {durable_id, mount_path, intent, fs_type?, total_bytes}.
Hub reads drives as json.RawMessage → no hub struct change; goldens re-pinned
byte-identical (agent + hub copies).

Tests: live_reporter_test.go (T1 load-bearing coord-without-verify + T2..T6),
TestDRRecipeHostHalf_V1DriveShape; each companion demonstrated to fail pre-fix then
reverted. go build/vet/test green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 20:25:09 +02:00
admin 8abc1b8852 agent v0.38.0: DR recipe — emit secret-free storage/guest/PBS half in host-report
DR recipe slice (agent half), grounded in SPIKE-dr-recipe-2026-06-16. Additive
`dr_recipe` host-report section = the non-secret reconstruction scaffolding the
operator must rebuild before PBS bytes can land. Built by pure
BuildDRRecipeHostHalf from facts the report already collects (no new reads):
guests[] sizing, drives[] (user-data by durable_id/role/mount/intent),
pve_storage[] (storage.cfg), pbs coordinates.

BOUNDARY (Phase-1 lesson): every field is an identifier/intent/size/coordinate —
never a key/password/token/hash/ENC:. PBS key stays in escrow; restic password
stays in escrow; the recipe names only the coordinates the restore targets.

Tests: BuildDRRecipeHostHalf selection, NoPBS, NoSecrets (boundary mirror),
dr_recipe key-set in the cross-repo golden contract test. recipe_version=1,
ignore-unknown on read.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 18:32:15 +02:00
admin 588fed2aa9 slice 10B: operator-signed destructive completion (offline key + signing CLI) (v0.16.0)
A destructive op runs ONLY on a pinned-key-verified, nonce-fresh, in-window,
host-bound, durable-id-bound operator signature. New cmd/felhom-opsign signs
canonical OpBlobs offline via ssh-keygen -Y sign (hardware-ready); the signing
key is never in the hub or agent. New internal/signedjobs runner verifies each
queued blob through the gate and only on all-pass runs the WipeExecutor, which
re-resolves the DURABLE device id + re-inspects (8C) before mkfs — closing the
8C data-bearing-wipe pending_signature gap. New storage durable-device
resolution; authz.CanonicalBlob promoted to production. Real-crypto tests assert
valid executes and forged/replay/expired/retarget/non-pinned are rejected
(executor never called).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 20:14:16 +02:00
admin 8ecf8929fb slice 10A: activate the control envelope (Down channel) + hub-backed desired provider (v0.15.0)
The control envelope becomes live: the agent caches the hub's desired-state +
generation and re-fetches GET /hosts/{id}/desired-state only when the
generation advances. A new internal/desired Syncer maps the wire shape into a
reconcile.CachingProvider feeding the engine; benign deltas reconcile, an
explicit guest decommission is gated pending_signature (exec is 10B). Adds the
DesiredStateResponse/WireDesiredState wire types + Client.FetchDesiredState +
the loop EnvelopeObserver seam. Cross-repo golden (envelope + desired-state)
byte-identical with the hub.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 19:02:59 +02:00
admin aa4dfb75ea slice 9: GET /host/metrics + CPU/chassis-temp collector (v0.14.0)
Add a host-wide, token-authed GET /host/metrics local-API endpoint that
re-serves the slice-4 collector's host + per-storage view to the customer
(the de-privileged controller can't read the host itself). Add the one new
collector — CPU/chassis temperature via sysfs hwmon/thermal-zones, graceful-
null — to the shared HostMetrics struct, so the hub report carries cpu_temp_c
too. Cross-repo host-report golden updated byte-identical.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 16:16:03 +02:00
admin 9f6753de0f restore-test wire contract: warnings + warnings_recognized in host-report golden (Phase B)
Adds the two new RestoreTest wire keys to the shared host-report golden and exercises
them in the bidirectional key-set contract test. Kept byte-identical with felhom-hub's
copy (sha256 e6999d77...). Testdata/tests only — the v0.7.0 binary is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 19:41:08 +02:00
admin 6e86483185 restore-test: verdict is liveness, not start-task exitstatus (v0.7.0)
Fixes the crying-wolf false-fail surfaced by the live hub-enrollment runbook:
PVE's guest-start task exits "WARNINGS: 1" for the benign systemd-nesting
advisory, and WaitTask treated any non-OK exitstatus as failure, so the verdict
was decided by an advisory exit code before the real boot check ran. Every
modern-distro restore-test reported pass:false.

- proxmox.WaitOptions.AllowWarnings (opt-in; default keeps all callers strict)
- restore-test start step accepts warnings, surfaces them, verdict stays waitRunning
- RestoreTestResult.StartWarnings/.WarningsRecognized + version-free "enable
  nesting" recognizer (can't rot back at systemd 258+); GuestAPI.TaskLogTail
- hub.RestoreTest.warnings/.warnings_recognized wire fields (consumed by hub v0.7.5)
- scheduler logs clean / passed-with-recognized / passed-with-unrecognized warnings
- tests: WaitTask warnings matrix; restore-test pass/fail-on-liveness; version-free
  regression guard (systemd 256-300)

Single agent bump 0.6.0 -> 0.7.0 covering the agent half of both task phases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 19:30:03 +02:00
admin 766500dfc3 v0.6.0: slice 6 Phase B — PBS offsite tier (verify + PBS-API client + reporting)
Spike-proven that backup/restore-to-PBS reuse Phase A unchanged; the only new code is
the verify capability, a small PBS-API client, and PBSSnapshot reporting.

- internal/pbs: fingerprint-pinned, token-authed PBS-API client (Verify/Snapshots/
  TaskStatus, node-from-UPID; secret read from /etc/pve/priv/storage/<id>.pw at runtime,
  never logged) + the verify maintenance loop (own cadence, default 6h, NOT gated/journaled,
  like the watchdog) + SnapshotStore.
- hub: PBSSnapshot filled (namespace/type/id/time/size/owner/protected/encrypted/
  verify_state/verify_upid); PBSReporter collector seam; cross-repo golden + bidirectional
  key-set tests; hub handler parses pbs_snapshots + logs a failed-verify WARN.
- backup: report the ACTUAL vzdump mode (parsed from the task log; PVE may downgrade
  snapshot->stop). proxmox.Storage.Username. config PBSVerifyCadence/secret-dir.
  --selftest=pbs-verify. Backup/restore-to-PBS unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 16:53:04 +02:00
admin b527430ec7 v0.6.0-rc1: slice 6 Phase A — backup + the self-restore-test (local target)
The guest-level backup layer + the journaled self-restore-test (restore→boot→verify→
teardown) that closes "a backup you haven't restored isn't a backup". All benign
(reuses the slice-4 classifier/gate/journal; no new destructive class/crypto). Local
target only; PBS = Phase B. Restore to a NEW guest only. Backups crash-consistent.

- proxmox: DestroyLXC, VzdumpOptions.Notes (notes-template), LatestBackupVolID.
- reconcile: Engine.RunRestoreTest (journal Scratch entry BEFORE mutation; net link-down
  pre-boot; defer teardown always; benign gated destroy) + Recover extended to reap a
  leaked scratch guest (Scratch flag, special-cased before the UPID path; idempotent).
- internal/backup: runner (vzdump + archive resolve + bulk-gap = backup!=1) + cadence
  scheduler (4th daemon goroutine, default 24h) + in-memory report store.
- hub: Backup/RestoreTest filled; collector seams; cross-repo golden byte-identical +
  bidirectional key-set tests; hub handler logs a FAILED restore-test prominently.
- config BackupConfig (band 990000-990009 default); --selftest=backup / restore-test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 13:49:39 +02:00
admin 27b68f043b v0.5.0-rc1: slice 5 Phase A — storage observe/report + watchdog (read-only, live)
Fill the slice-3 storage_targets stub and add the fast-poll storage watchdog.
Read-only this phase; the host-root surface (mounts/SMART/grow/destructive gate)
is Phase B. Hub-owned desired manifest is slice 10, so reconcile against it is
built-but-unfed.

- internal/storage: StorageTarget wire contract, durable_id derivation per type,
  HostReader seam (procfs/sysfs, root-free), Observer (storage_targets from
  ListStorage/NodeStorage + host reads, lvmthin thin-pool fill), and the watchdog
  (third daemon goroutine; debounced out-of-band report on a known target's
  attach/disconnect transition).
- proxmox.Storage: additive parse-only config fields (durable_id sources).
- collector StorageObserver seam; Loop.SetTrigger out-of-band report; daemon runs
  the watchdog as a third goroutine; StorageConfig knobs.
- cross-repo golden kept byte-identical with felhom.eu/hub; bidirectional key-set test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 09:59:05 +02:00
admin e68a7af4d3 fix(agent): slice-3 follow-ups — keep run-status on config fail, selftest usage, contract golden (v0.3.1)
- collect: a per-guest GuestConfig failure preserves the ListLXC run-status (only
  spec dropped); empty status normalized to "unknown". Test asserts preserved
  "running" + nil spec.
- main: --selftest usage error now reads (want read|task|hub).
- contract: testdata/host-report.golden.json + TestHostReport_ContractMatchesGolden
  (field-name key-set check vs golden; byte-identical with the hub copy).
- version 0.3.0 -> 0.3.1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 18:29:05 +02:00
admin ab77fa3544 feat(hub): host-report client + collector + first daemon loop (slice 3, v0.3.0)
internal/hub: the agent's first daemon — a periodic read-only host-report POSTed to
the hub (the heartbeat; no separate ping).

- HostReport wire contract (shared field-for-field with the hub ingest): host
  metrics, guests (vmid + spec), cloudflared status; storage/backups/restore-tests/
  pbs/audit collections DEFINED but emitted empty (slices 5/6 fill).
- Collector over a read-only proxmoxReader (adapted to the real proxmox surface;
  no proxmox changes) + a CloudflaredProber. Partial-failure: NodeStatus fail = hard
  (skip POST); per-guest GuestConfig fail = status "unknown", still report.
- Client: Bearer-auth POST, standard TLS (system roots / optional ca_file), typed
  TransportError/HTTPError, token never in errors.
- Loop: immediate first report, adopt hub poll_interval (clamp [60,3600]), resilient
  to collect/report errors, clean ctx-cancel shutdown.
- ControlEnvelope: only poll_interval_seconds acted on; blocked/desired_generation/
  has_signed_ops parsed-but-ignored (slice 4).
- config: HubConfig + FELHOM_AGENT_HUB_* overlay + mode-aware HubConfig.Validate +
  WithDefaults + hub-key redaction; example config updated.
- main: no-selftest mode is now the daemon; added --selftest=hub. Version -> 0.3.0.

Tests: report serialization, client (incl. token-redaction), collector partial-
failure, loop continuation+interval adoption, config. internal/proxmox + internal/
authz untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 16:20:09 +02:00