felhom-agent/REPORT.md

REPORT — RUNBOOK: hub enrollment round-trip + failed-PBS-verify validation (2026-06-09)

Overwrite-latest report (most recent significant work only). Cumulative history lives in CHANGELOG.md. This was an operational validation run (RUNBOOK, no code change) against the live, deployed v0.7.4 hub — it closes the "End-to-end hub reflection" gap that the prior slice-6-Phase-B report flagged as Not done.

Outcome

All runbook objectives met. The demo host (demo-felhom) was enrolled on the deployed hub for the first time, the daemon→hub round-trip was confirmed end-to-end on v0.7.4, and the failed-PBS-verify [WARN] — the loudest offsite-DR signal — was driven live by a real corrupted chunk in a dedicated throwaway datastore, with the encrypted production datastore (felhom-spike) proven untouched (blast radius contained). Teardown left the system clean; per explicit decision the demo host is kept enrolled as a persistent systemd service.

Components exercised live: agent felhom-agent v0.6.0 (daemon, PBS verify loop, host-report collector) ↔ hub felhom-hub v0.7.4 (host-report auth/ingest, storage/guests/pbs/restore reflection, failed-verify WARN).

Phase 0 — pre-flight (all green)

• Image gate: ArgoCD app felhom Synced + Healthy; hub Deployment image …/felhom-hub:v0.7.4; pod log [INFO] felhom-hub v0.7.4 starting. All three agree.
• Customer config: demo-felhom exists in customer_configs (confirmed via a non-mutating GET /api/v1/config/demo-felhom probe → 401 invalid password, i.e. row present).
• Existing enrollment: hosts table empty (count(*)=0) → minted fresh.
• Agent config: at /root/.config/felhom-agent/agent.json (no systemd unit; launched manually); no hub section; cadences restore_test=-1, pbs_verify=-1 (both disabled); local_backup_target=local; pbs_secret_dir=/etc/pve/priv/storage. Backed up to agent.json.pre-runbook. Scratch guest 9999 (felhom-selftest-scratch, LXC) present.

Phase A — happy-path round-trip (CHECKPOINT GREEN)

• A.1 mint: POST /api/v1/admin/hosts (global-key-gated, via the deployed hub) → 201, host_id=demo-felhom-01, 64-hex per-host key. Key transferred to the host out-of-band (stdin pipe → 0600 file → atomic config patch), never echoed.

• A.2 config: added the hub block (url, host_id, api_key) + test cadences (poll 60, pbs_verify 120, restore_test 300).

• A.3 seed: vzdump 9999 → local (mode downgraded snapshot→stop, expected for a stopped CT).

• A.4 daemon + reflection: daemon up (version=0.6.0 host_id=demo-felhom-01 hub_url=https://hub.felhom.eu); all loops started. The 18:17:07 host-report reflected, confirmed both in hub logs and the hub store (host_reports.report_json):

  • ✅ host-report auth + ingest (slice 3) — 2xx, no "Unknown host_id".
  • ✅ storage_targets (slice 5) — 4 targets incl. felhom-pbs.
  • ✅ guests reflected.
  • ✅ pbs_snapshots (slice-6 Phase B) — felhom-spike ct/9001 verify_state:"ok", encrypted:true → the encrypted-storage verify path observed ok live.
  • ✅ restore_tests (slice-6 Phase A) — recorded and reflected.
  • backups empty (expected — no RecordBackup caller until slice 10).

  Finding (benign false-negative): the scheduled restore-test reports pass:false because the restored scratch guest's vzstart returns exitstatus "WARNINGS: 1" — the warning is cosmetic: WARN: Systemd 257 detected. You may need to enable nesting. The guest boots fine; the agent's strict WaitTask (exitstatus == "OK") treats any WARNING as failure. This means every restore-test of a modern-distro (Debian 13 / systemd 257) LXC will false-fail, generating spurious DR alarms. It did, usefully, also exercise the hub's [WARN] restore-test FAILED path. (See "Follow-ups".)

Phase C — destructive failed-verify (fenced; the slice-6-Phase-B closure)

A dedicated throwaway datastore verifyfail (/mnt/5_hdd/pbs-verifyfail, unencrypted, one backup ⇒ every chunk exclusive) isolated the blast radius. felhom-spike and the homelab /mnt/5_hdd/backup/ tree were never touched.

• C.1–C.3: created verifyfail + DatastoreAdmin ACLs on both felhom@pbs and the token felhom@pbs!n100 (PBS privsep intersection); registered it as a PVE pbs storage on the demo host (token authed → status active); put exactly one backup (ct/9999) into it.

• C.4: baseline verify OK, then corrupted one chunk (.chunks/008f/008f536d…b121dd, first byte 0x31→0xCE, size unchanged → true digest mismatch with no size change, the realistic bit-rot case).

• C.5.1 — agent local read path: --selftest=pbs-verify → agent's Verify (ignore-verified=false re-read) detected it: datastore=verifyfail failed=1 total=1, the ct/9999 PBSSnapshot shows verify_state:"failed", while felhom-spike ct/9001 stays ok. Confirms report.go passes verification.state through and verify.go re-lists/counts.

• C.5.2 — live wire → hub WARN: the running daemon's verify loop caught it (18:24:09 … FAILED-verify snapshots), and the 18:32:07 host-report carried it. Hub logged:

  18:32:07 [WARN] host demo-felhom-01 PBS verify FAILED: ct/9999 ns=root owner=felhom@pbs!n100
  18:32:07 [INFO] host-report from demo-felhom-01 (3 guests, 5 storage targets, 0 backups,
                  1 restore-tests, 2 pbs-snapshots, 4928 bytes)
  

  Hub store confirmed containment: verifyfail ct/9999 encrypted:false verify_state:"failed", felhom-spike ct/9001 encrypted:true verify_state:"ok". Real corrupted chunk → agent verify loop → wire → hub failed-verify WARN, end to end, on v0.7.4.

Teardown (clean; demo host kept enrolled by decision)

• Daemon stopped; verifyfail PVE storage + verifyfail.pw removed; datastore destroyed (--destroy-data true), ACLs deleted, /mnt/5_hdd/pbs-verifyfail removed.
• Verified as-found: only felhom-spike datastore remains (verify → TASK OK); /mnt/5_hdd/backup/ tree byte-identical to the Phase-0 listing; no orphaned ACLs; seeded local backup of 9999 removed; stray temp/key files cleaned.
• Enrollment retained (explicit choice "keep enrolled as a service"): hub block kept; production cadences set (poll 900, pbs_verify 6h, restore_test -1/disabled to avoid the benign-warning spam); installed + enabled /etc/systemd/system/felhom-agent.service (active, enabled, survives reboot). First service report landed clean at 18:55:32 (1 pbs-snapshot, no WARN).

What this proved (and didn't)

• Proved live on v0.7.4: slice-3 host-report auth + ingest; slice-5 storage_targets reflection; slice-6-Phase-B pbs_snapshots reflection including the failed-verify [WARN] via a real corrupted chunk; slice-6-Phase-A restore_tests reflection; the encrypted-storage verify path observed ok on felhom-spike.
• Did not prove: backups reflection (no daemon RecordBackup caller — golden-only until slice 10); the corruption test datastore was unencrypted (the verify mechanism is identical server-side; the encrypted config path is exercised by the normal felhom-spike verify).

Follow-ups surfaced

1. Restore-test false-fail on benign LXC start warnings (new, notable): the agent treats vzstart WARNINGS: n as failure, so modern-distro LXC restore-tests always report pass:false. Options: treat WARNINGS distinct from non-OK, set scratch-guest features: nesting=1 on restore, or whitelist the systemd-257 warning. Left disabled on the demo host as a stopgap.
2. POST /api/v1/admin/hosts is global-key-gated only and reachable via the public hub URL — anyone with the global key can mint a host. Lock down at the slice 7–8 auth cutover (doc 05 §11).
3. Hub hardcodes defaultHostPollSeconds = 900 (handler.go) and the agent adopts it, silently overriding the configured poll_seconds — worth making configurable / documenting.
4. backups producer still absent (accept it lands with slice-10 desired-state).

No secrets committed

The hub global key, the per-host key, and the PBS token secret are referenced by location only (hub ConfigMap hub-config; /root/.config/felhom-agent/agent.json 0600; /etc/pve/priv/storage/*.pw 0600) — never echoed into logs or this report.