384 lines
34 KiB
Markdown
384 lines
34 KiB
Markdown
# CONTEXT — felhom-agent working state
|
||
|
||
> Snapshot of the current state + open threads. Authoritative history lives in `CHANGELOG.md` (top
|
||
> entry = current); the end-of-task detail lives in `REPORT.md`.
|
||
|
||
## Current
|
||
|
||
- **2026-07-28 — v0.107.0: F-REBOOT fixed — a guest rebooted mid-backup now comes back by itself.**
|
||
New `internal/localapi/guestpower.go`: a 60 s watchdog that starts a guest which is `onboot:1`,
|
||
stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let
|
||
`RecoverStaleLockedGuests` miss campaign fault 11 — that recovery acts only on a **stale vzdump
|
||
lock** (fault 11's guest was unlocked) and runs **once at agent startup** (fault 11's guest went
|
||
down while the agent was already up). `onboot` is the deliberate-stop discriminator and is *not*
|
||
invented here: it is already what `stalelock.go` uses for this decision, it is 0 on scratch/golden
|
||
guests, and it is what `pve-guests` consults at host boot — so the agent agrees with the platform
|
||
instead of keeping a second private definition of "should be running". Retry bounded at 3
|
||
(1m/2m/4m) then escalates **once**; an unbounded silent retry loop is the over-correction here.
|
||
Live on demo-hp: **120 s unattended** recovery vs the incident's **587 s** with a human; Scenario B
|
||
proven (an `onboot:0` guest left stopped throughout). Detail: `REPORT.md`.
|
||
|
||
- **2026-07-28 — F-LEAK took THREE attempts; v0.108.0 and v0.110.0 are the corrections.** The cause is
|
||
structural: `FelhomAgentGuest` is granted at `/pool/felhom` and a guest joins that pool only when its
|
||
restore **completes**, so a *failed* restore-test leaves a pool-less guest out of reach (403).
|
||
**(1) v0.107.0 pool adoption — REFUTED LIVE:** `PUT /pools/{pool}` also requires `VM.Allocate` on the
|
||
VM being added, so membership cannot bootstrap its own authority; removed in **v0.108.0**.
|
||
**(2) host-install v1.21.0 per-path `/vms/990000..990009` ACLs — works, but exactly ONCE per slot:**
|
||
PVE's destroy calls `AccessControl::remove_vm_access` (`API2/LXC.pm:906`) which deletes every ACL at
|
||
`/vms/<vmid>` (`AccessControl.pm:1898`) — **the grant is consumed by the op it authorises**. Caught by
|
||
counting ACL rows after the fix, not by reasoning. **(3) v0.110.0 SHIPPED —
|
||
`Privileged.DestroyScratchLXC`, the FOURTH root-fenced exception** (was exactly three: keyctl
|
||
`pct create`, USB mount/fstab, SMART/sensors). Band enforced in **sudoers literally**
|
||
(`pct destroy 99000[0-9] --purge`) + re-checked in code + journal provenance at the caller; none is
|
||
consumed by use. API destroy still tried FIRST; band ACLs stay provisioned so the common case needs no
|
||
privileged call. **Ships with a sudoers change — deploy `configs/felhom-agent.sudoers` WITH the
|
||
binary.** Live: token 403 on a stranded scratch → fenced path removed the guest and all 3 LVs; sudo
|
||
PERMITS the band and REFUSES `9201`/`9100`/`9999`/`990010`/`1`, and refuses `pct start 990000` too.
|
||
|
||
- **2026-07-28 — v0.109.0: the guest-power watchdog got the observable it shipped without.** A
|
||
self-correction: v0.107.0's watchdog logged only at startup and when it *acted*, so on a healthy box
|
||
its health could be read only from **absence** — F-OBS's exact shape, shipped in the same session
|
||
F-OBS was fixed in the controller. Now an INFO summary every 10th sweep carrying
|
||
`sweeps_since_boot`/`guests_evaluated`/`currently_stopped`. An **aborted** sweep (unproven
|
||
ownership) does not count, or the heartbeat would claim liveness for a watchdog examining nothing.
|
||
|
||
- **2026-07-28 — v0.106.0: F-CRIT-2 fixed — a failed backup no longer looks like a fresh one.**
|
||
`NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, NEWEST) as a successful
|
||
backup, so the tier reported fresh, went **not due**, and was never retried — 7 days of silence on
|
||
the real 168h cadence, invisible to both the R-88 breaker (defers only DUE tiers) and the hub
|
||
deadline monitor (reads the same freshness). Now only *plausibly complete* entries count, via a
|
||
measured floor `minPlausibleArchiveBytes` = 1 MiB; undecidable ⇒ not counted.
|
||
**Size is the only tier-agnostic discriminator** — `verification` and `encrypted` are absent on
|
||
every local (dir) archive and on a good PBS snapshot until verify-new catches up, so gating on
|
||
either would reject 100% of local backups and cause fleet-wide backup THRASH. Floor measured:
|
||
smallest real backup on the fleet is 612,397,450 B, so 1 MiB leaves 584x headroom (asserted by a
|
||
test). Rejections logged at WARN once per volid. Re-tested live by replaying campaign fault 2 on
|
||
demo-hp — both directions, incl. a no-thrash window with 91 scheduler ticks as the positive
|
||
observable. Deployed on both boxes. Detail: `REPORT.md`.
|
||
**Also established:** server-side prune does NOT count phantoms toward `keep-last` (dry-run kept
|
||
2 real + the phantom) ⇒ **no retention/data-loss bug** — but it never removes them either, so they
|
||
accumulate. Filed as R-99 (LOW).
|
||
|
||
- **2026-07-25 — v0.95.0 (additive): SMART coverage fixes (spike B+A) + device model.** Union-path
|
||
drives (USB/registry) now get SMART via `storage.SmartReader.SMARTForBacking` wired into the localapi
|
||
`/disks` union (localapi `Smart` seam); `smartDeviceFor` resolves dm/LVM to the whole disk via
|
||
`/sys/block/<dm>/slaves` (recursive, skips >1-disk); the builtin `local` dir on the LVM root gets a
|
||
**SMART-only** device from its containing filesystem (never touches backing/durable_id — the
|
||
removable-safety guard in build() stays intact); `SmartSummary.ModelName` captured from smartctl. The
|
||
watchdog `Known` path stays enrich-free. Consumed by controller v0.171.0. Source of WHERE:
|
||
`felhom.eu/documentation/audits/SPIKE-smart-coverage-2026-07-25.md`.
|
||
- **2026-07-24 — v0.94.0 (additive): SMART serialized into /disks.** `localapi.DiskInfo` gains
|
||
`Smart *hub.SmartSummary` (omitempty), copied from the target's already-computed Observe-time
|
||
enrichment when `Health != ""` — no new smartctl load, no endpoint, no sudoers/MinAgent change. The
|
||
controller v0.169.0 renders a "Lemezek állapota" card + 6h degradation alert from it; old controllers
|
||
ignore it. **NOTE: at the remote-site vacation window the agent is DOWN (localapi binds .162 → fails),
|
||
so live /disks-from-real-agent validation is deferred — the field is unit-proven; publish only.**
|
||
- **2026-07-22 — v0.93.0 is the FLEET AGENT.** Built, published (sha `a68b2ff73200622e…`),
|
||
Day-0-manifest-vouched (MinAgent also 0.93.0, operator-ruled) and deployed to BOTH boxes
|
||
(`demo-felhom-8363b5` + `demo-hp-bb76ea`, the latter over G1 break-glass — still no key baked);
|
||
clean-restart 5/5 on both, `.bak-0.92.1` retained. Discharges the onboarding runbook §A5
|
||
ceremony gate. Record: `felhom.eu/documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md`.
|
||
**The bullet below ("agent is DOWN … deployed 0.90.0") is SUPERSEDED history** — vmbr0 was made
|
||
static .162 on 2026-07-20 (F1 mitigation) and the agent has been up since; kept for the record.
|
||
|
||
- **2026-07-20 — REMOTE SITE until ~2026-08-02; the agent is DOWN there and cannot self-recover.**
|
||
felhom-pve moved off the home LAN; `ssh felhom-pve` = tailnet `100.70.170.35` (direct, ~37 ms). The
|
||
host is on DHCP and holds `192.168.0.147`, so `localapi`'s literal `192.168.0.162` bind fails with
|
||
`bind: cannot assign requested address` — the daemon exits ~1.1 s after start, systemd gave up after
|
||
4 retries, and a manual restart reproduces it exactly. Deployed binary is **0.90.0**. Fix needs
|
||
`listen_addr` in `/etc/felhom-agent/agent.json` **and** the guest bootstrap endpoint (plus the
|
||
pinned leaf's SAN) → **Viktor GO**; re-pinning to another literal just re-breaks on the next lease.
|
||
Also re-observed each start: `pbs: cannot read token secret … /etc/pve/priv/storage/felhom-pbs.pw:
|
||
permission denied` (R-39-adjacent). Evidence + ranked findings:
|
||
`felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
|
||
|
||
- **v0.90.0** (2026-07-17) — **agent train: guest RAM resize (R-24) + fast-tick (R-28); LIVE on BOTH
|
||
demo hosts (felhom-pve + nested demo-vm-felhom-4846bc).** MinAgent coupling: felhom-controller
|
||
v0.143.0 gates its resize UI on this agent. (1) **R-24 guest RAM resize (controller-direct)** —
|
||
self-scoped `GET`/`POST /guest/memory` (`internal/localapi/guestmemory.go`); the AGENT enforces
|
||
every bound fresh per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)) and
|
||
applies via PVE `SetConfig` — **live cgroup apply, no reboot** (Phase-0 PROVEN on the nested box; the
|
||
break-glass access path + the proof are in `~/.claude/.../nested-vm-access-breakglass.md`). Verify-
|
||
after-apply re-reads maxmem before claiming success. New narrow `MemoryOps` seam (GuestAPI untouched);
|
||
memory only. (2) **R-28 fast-tick** (`internal/fasttick/`) — while any desired-state item is unapplied
|
||
(esp. the pre-tunnel WG-registration window a hub poke can't reach) pulse the shared out-of-band
|
||
trigger every 30 s, self-disarm on convergence; four cached sources (desired-gen==0, reconcile
|
||
Planned−Pending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational). Seams:
|
||
`reconcile.Engine.LastResult()` + `wgtunnel.Manager.TunnelConvergence()` (cached — no per-tick exec).
|
||
(3) **Guests-0/0** REFUTED live: the 0/0 was the pre-provision window (guest not yet created), not a
|
||
pool-membership bug; the fast-tick shortens that window. **OPEN (operator GO):** publish 0.90.0 +
|
||
hub Day-0 manifest vouch + MinAgent-floor raise to 0.90.0 (password-gated UI; the safety gate — both
|
||
agents on 0.90.0 — is satisfied and the coupling is proven live via the version header). See REPORT.md.
|
||
|
||
- **v0.89.0** (2026-07-16) — **agent train: three bundled agent-plane items; built + published to
|
||
Gitea (sha256 `3969fd91…`); paired with hub 0.59.0 (LIVE).** (1) **pbsdr self-grant (R-22)** —
|
||
closes the F4 self-deadlock: a 403 on the token-auth `StorageEntry` pre-check now self-grants via
|
||
the root wrapper + re-reads instead of aborting before the grant (the demo's `felhom-offsite`
|
||
case). (2) **escrow config live-reload** — `/escrow/preflight`'s `pbs_storage_id` row now reads
|
||
the live agent.json (late-bound `CurrentPBSStorageID`) so a pbsdr-seeded id flips green with no
|
||
restart. (3) **agent-plane poke listener (Direction-2a)** — `internal/poke`: contentless UDP poke
|
||
bound to the box WG /32 (port **51822**), leading-edge debounced, fires the hub-loop out-of-band
|
||
trigger for an immediate desired-state cycle; enabled with `wg_tunnel.enabled`; first slice of
|
||
R-13. Red-proofs for all three (run-fail-revert). **ALL THREE LIVE LEGS PROVEN on the demo
|
||
(2026-07-17), demo now LIVE on 0.89.0:** Scenario 4 floor-driven A/B train 0.88→0.89 (operator
|
||
signed+enqueued the `agent_update` op — the vouch+floor alone does NOT trigger it; committed, no
|
||
rollback); Scenario 1 R-22 self-heal (marker aside + ACLs revoked → `pre-check 403 … self-granting
|
||
(R-22)` → `converged state=adopted` in ~3 s, ACLs restored, offsite active); Scenario 3 poke→tick
|
||
~31 ms ep0→box + immediate report cycle (save→tick ≈ ~0.45 s). Details: REPORT.md.
|
||
|
||
- **v0.88.0** (2026-07-13 eve) — **controller-driven escrow ceremony (agent half), LIVE on demo
|
||
host + drill VM (63/63 capabilities both).** `--output=json` machine mode (text mode
|
||
byte-identical; extraction into `escrowCeremony()`); the ONE fixed argv
|
||
(`escrow.CeremonyArgs()` — shared by the localapi exec + the `escrow-ceremony` capability
|
||
(Critical, pbs_dr-gated EXPLICIT) + the new `FELHOM_ESCROW` sudoers alias, three-way
|
||
pin-tested); localapi job endpoints (`POST /escrow/ceremony` single-flight 60 s,
|
||
status, ONE-SHOT claim → 410, 10-min TTL → `unclaimed_void`, `GET /escrow/preflight`).
|
||
R in-memory ONLY (never the job struct — snapshot-hygiene-tested; restart loses it safely).
|
||
Live-proven on drill endpoint-exact: stage → preflight all-green (live FELHOM_ESCROW
|
||
list-probe) → job ~4 s → hub blob `restic_pw_sha256` covering (repaired the spike's hash-less
|
||
blob) → claim 200 once → 410. Coupled: controller v0.127.0 (MinAgent 0.88.0 for the wizard).
|
||
**OPEN: publish 0.88.0 + Day-0 manifest vouch (operator) at the next train; deployed hosts got
|
||
direct deploys.** Details: REPORT.md + felhom.eu RUNBOOK-escrow-ceremony.md (F1 threat model).
|
||
|
||
- **v0.87.0** (2026-07-13) — **SystemDisks device-mapper walk (IA finding 2, MEDIUM):
|
||
legacy-boot hosts get a working drive wizard.** Operator ruling (approved 2026-07-13,
|
||
verbatim): *resolve device-mapper/raid parents — for the root filesystem's backing block
|
||
device, walk `/sys/block/<dev>/slaves` recursively down to physical disks; those, plus any
|
||
ESP holder when present, are system. Disks outside that set become wizard candidates (still
|
||
subject to the existing data-bearing guards). The all-system fail-safe remains ONLY for walk
|
||
failure — it returns to being the error case, not the legacy-boot common case.* Implemented as
|
||
`physicalDisksOf`/`walkSlaves` + `HostReader.BlockSlaves` (one seam method); per-branch
|
||
conservatism (any unresolvable slave → ok=false → unchanged all-system path); signature test
|
||
`TestSystemDisks_WalkTopologies` (root-backing disk ALWAYS system — never weaken). §3 spike
|
||
transcripts: drill (legacy) dm-1→sda3→sda; felhom-pve (EFI+LVM) ESP+walk agree on sda →
|
||
byte-identical regression. §13.2 wizard leg COMPLETE (offered → enrolled → formatted →
|
||
torn down, boxes as found) + Day-0 manifest vouched to 0.87.0 (operator). The leg also
|
||
surfaced two CONTROLLER bugs (fixed same-day: v0.126.3 claimed-box wizard CSRF, v0.126.4
|
||
502-through-CF + native-alert ban).
|
||
|
||
- **v0.83.0** (2026-07-11, LIVE on felhom-pve; NOT published — Peti stays 0.81.0) — **observability
|
||
pass** (pairs with controller v0.116.1 + hub v0.46.0). `applog.New` → `(logger, *Ring)`: slog
|
||
fan-out, journald at the configured level, ~1000-entry ring FIXED at DEBUG. `GET /debug/logs`
|
||
(local API, token-authed; the controller Debug page's Ügynök tab) + request-level DEBUG
|
||
middleware. Heartbeat log-pull: envelope `log_tail_requested` → next heartbeat ships `log_tail`
|
||
(128 KB, consume-once; failed push re-armed by the next envelope; `operator log pull served` INFO
|
||
on fulfillment). Gap-fill sweep: netverify phase/verdict lines (job start, trigger outcome,
|
||
/proc/mounts verdict, journal bytes, classification code, rollback outcome, durations), netmount
|
||
unit steps, signedjobs op-received (class/host/expiry — never signatures) + fetch duration,
|
||
selfupdate invariants + download sha/duration, disks outcome INFOs, controller-swap pre-pull +
|
||
health verdicts, desired/loop per-exchange DEBUG. Logging conventions:
|
||
`felhom.eu/documentation/runbooks/logging-conventions.md`. OPEN: the hub-side live pull awaits
|
||
the operator's button click (hub UI password-gated); pre-existing lanresolver
|
||
permission-denied WARN on /var/lib/felhom-agent/guests noted in REPORT.
|
||
|
||
- **v0.77.0** (2026-07-09) — **fork-4: escrow the offsite restic repo password under R.** `IdentityBundle`
|
||
gains `ResticRepoPassword` (rides the existing age-under-R `WrapIdentityBundle` path — validated by the
|
||
custody spike `febdc56`). New `POST /escrow/stage-secret` (`withGuest`) transiently stages the
|
||
controller-pushed password (0600, never logged), which the `--selftest=escrow-create` ceremony
|
||
auto-injects into the bundle and then wipes. `AttachResticPassword`/`StagedResticPasswordPath`/
|
||
`WipeStagedResticPassword` added. Pairs with controller v0.105.0 (push + atomicity gate + DR inject +
|
||
`DRResticCoord`). **NOT yet live-validated** — the supervised escrow ceremony is operator-run.
|
||
|
||
- **v0.76.0** (2026-07-08, LIVE on felhom-pve + **PUBLISHED sha `9828c5f7…f50b`** — THE Day-0
|
||
manifest bump target; **0.75.0 superseded unpublished**) — **GL-5b / G12: restore-test
|
||
full-fidelity**. Params derive from the ARCHIVE's embedded config (`drRestoreOverrides`, same as
|
||
DR — the old live-source-config path verified the wrong object AND dropped storage mpN per PVE's
|
||
all-or-nothing rule; deleted with `bindMountOverrides`/`archiveVMID`). NEW mount-parity assert
|
||
(restored mpN vs archive; miss/mispath/undersize/extra = FAIL naming the delta) + `MountParity`/
|
||
`MountInventory` on the wire record (additive). Live-proven: scratch 990000 ← 6.5GB 9201 archive,
|
||
parity ok, inventory mp0 200G+mp1 50G+2 throwaways, **3m4s local tier** (cheaper than feared);
|
||
rotated-out archive volid → clean up-front refusal (nice failure mode). bringup.go untouched.
|
||
|
||
- **v0.75.0** (2026-07-08, LIVE on felhom-pve) — **GL-5 / go-live G8: guest-loss DR bring-up
|
||
actually restores** (closes the v0.74.0 OPEN item + SPIKE-dr-bindmount-source §8). DR passes the
|
||
COMPLETE explicit restore param set derived from the archive's embedded config (NEW
|
||
`Client.ExtractArchiveConfig`, 200 under the scoped token) — **two live-discovered PVE rules: mpN
|
||
params need an explicit rootfs, AND unlisted mountpoints are silently DROPPED** (first run booted
|
||
without mp0/mp1!) — storage mpN passed through, structural mp8/mp9 → throwaways, then step 4d
|
||
swaps the REAL binds in via the host runner (root pct; new `EngineOptions.HostRunner`+`StateDir`
|
||
seam) and deletes the unusedN residue. Scratch-DR live-proven end-to-end (9310 from a real 9201
|
||
archive: mp0 200G + mp1 50G + real binds + no residue + clean teardown). Provision = nil
|
||
overrides (regression-tested). NOTE: published/vouch-pending agent is 0.74.0 — publish 0.75.0
|
||
before/with the manifest bump. OBSERVATION: the DR selftest hardcodes KeepMAC=true — a scratch DR
|
||
while the SOURCE guest is live briefly duplicates its MAC on the bridge (pre-existing; fine for
|
||
supervised runs, worth a -keep-mac flag someday). Full customer-data DR drill = GL-6/S5 family.
|
||
|
||
- **2026-07-07 — v0.74.0 Gitea-PUBLISHED (RUNBOOK GL-1)** — the LIVE felhom-pve binary's exact
|
||
bytes, sha256 `1ec3f58842edce1e…76af05`, anon-fetch-verified. This supersedes/closes every
|
||
standing "publish 0.6x + Day-0 vouch" OPEN item below (0.64→0.73 were never published; 0.74.0
|
||
is the vouch target). Golden 0.103.0 published in the same run (felhom.eu execution record
|
||
`documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`). **Day-0 manifest vouch = operator
|
||
step** (agent 0.74.0 / golden 0.103.0).
|
||
|
||
- **v0.74.0** (2026-07-07) — **campaign-2 R2 CLOSED; the mislabelled "R1" was a symptom** (LIVE on
|
||
felhom-pve). Pool membership is what lets the pool-scoped token reach a guest; `pct restore --pool`
|
||
sets it only at CREATE, so a restore-over-existing dropped 9201 from the `felhom` pool → no
|
||
`VM.Audit` → restore-test's *existing* `bindMountOverrides` never ran → "mp8 … only possible for
|
||
root". Fix: `Client.PoolAddVMID` + bring-up re-asserts membership post-restore (warn-not-fail).
|
||
Role/ACL + `bindMountOverrides` untouched (both correct). **Live restore-test PASSED for the first
|
||
time** once the pool was healed (Part A one-liner): read config → neutralize 2 binds → restore →
|
||
boot+running → clean teardown, 4m35s. B3 (scratch-teardown 403) confirmed a cascade — no code.
|
||
OPEN: DR `bring-up -mode dr` bind-override gap (spike `SPIKE-dr-bindmount-source-2026-07-07.md`:
|
||
small known-constant override reusing `bindMountOverrides`; mp8/mp9 are structural constants).
|
||
|
||
- **v0.73.0** (2026-07-06) — **F2 mount-role fallback CLOSED** (LIVE on felhom-pve). `roleForMountPath`
|
||
gained a mount-table fallback (Impl-2b style): a bind-mounted RAW enrolled user-data drive is not a PVE
|
||
storage, so it fail-safe'd to `system` and the eject/decommission gates 403'd EVERY user-data drive
|
||
(campaign F2, `where=/mnt/teszt_enroll role=system`). Device-keyed classification + whole-disk containment
|
||
(`storage.SameWholeDisk`); Observe-error keeps the fail-safe BEFORE the fallback. Only `roleForMountPath`
|
||
touched. Live-proven full lifecycle on teszt_enroll (eject/decommission 200, no-rebind across restart,
|
||
end==pre). OPEN follow-up: the `deviceRole`/`roleForMountPath` unification refactor (deferred).
|
||
|
||
- **v0.72.0** (2026-07-05) — **OOB operator access (merged E1+H1)** — TASK H1, provenance both
|
||
`SPIKE-{felhom-sshd,oob-wg-operator-peer}-2026-07-05`. Operator `/32` RENDERED into wg-felhom
|
||
AllowedIPs (survives self-heal, [OF-1]); dedicated `internal/felhomsshd` (port claim + config
|
||
render→sshd -t→reload + operator authorized_keys + heal + oob heartbeat stanza); static
|
||
`inet felhom_oob` belt (agent mutates SET ELEMENTS ONLY); `configs/felhom-sshd.service` (NO
|
||
RuntimeDirectory [SF-1]) + `felhom-oob.nft` + `felhom-op.sudoers`; `FELHOM_SSHD`+`FELHOM_OOB`
|
||
grants; `oob.enabled` DEFAULT FALSE. Live on felhom-pve (8822, belt filled, operator SSH as
|
||
felhom-op with scoped sudo); hub v0.35.0. Rollback `.bak-0.71.0`. 5 live-found bugs fixed (port
|
||
path, self-listen flip-flop, nil-block lockout, reachable-via-dial, operator-configured source).
|
||
- **v0.71.0** (2026-07-05) — **management-plane break-glass: privsep-dir watchdog + mgmt_plane
|
||
health** — TASK G1 (prereq for felhom-sshd/H1), provenance `SPIKE-felhom-sshd-2026-07-05` §8.
|
||
Host artifacts (`configs/felhom-privsep.tmpfiles` + `felhom-mgmt-watchdog.{sh,service,timer}`) make
|
||
`/run/sshd` boot-persistent AND auto-heal it every ~60s **agent-independently** (heals with the
|
||
agent stopped — proven live: `/run/sshd` removed → restored in 30.0s, `:22` back, no login).
|
||
`internal/mgmtplane` reports the additive `mgmt_plane` heartbeat stanza; hub v0.34.1 raises
|
||
`mgmt_plane_healed`. **NO unit declares `RuntimeDirectory=`** (the incident cause). H1 may now
|
||
assume `/run/sshd` is guaranteed present. Live on felhom-pve; rollback `.bak-0.70.0`.
|
||
- **v0.70.0** (2026-07-05) — **agent self-update (operator-signed A/B slots + crash-loop
|
||
auto-rollback)** — TASK D1, provenance `SPIKE-agent-selfupdate-2026-07-05`. An operator-signed
|
||
`agent_update` op (version+sha256, sha is the only integrity root) rides the signed-jobs gate;
|
||
`internal/selfupdate.Executor` downloads+verifies+hands to `felhom-selfupdate-guarded apply` (root
|
||
re-verify → A/B atomic flip → pending marker → detached restart); the new binary commits after a
|
||
60s dwell; a crash-looping binary is auto-reverted by `OnFailure=felhom-agent-rollback.service`
|
||
(first-crash trigger [SF-1]) with the tuned `[Unit]` start-limit (120s/4) as backstop. Host
|
||
artifacts + sudoers `FELHOM_SELFUPDATE` + `felhom-host-install.sh` day-0 install + report field
|
||
`selfupdate_pending`. Green tests + companions. **LIVE-VALIDATED on felhom-pve (2026-07-05): all 4
|
||
drills PASS** — happy path (0.70.0→0.70.1 signed op → download+verify+flip+commit), crash-rollback
|
||
(0.70.2-crash → OnFailure → **~2s crash-to-recovered**, byte-identical revert, no loop), no-pending
|
||
guard, gate refusal (non-pinned key). Full agent-side pipeline ran real (envelope injected into the
|
||
hub `signed_jobs` queue — CC lacks the hub global operator key; hub enqueue-auth is hub-unit-tested).
|
||
Box restored to canonical **v0.70.0** (host artifacts KEPT installed; scratch operator key REMOVED —
|
||
self-update dormant until an operator pins a real key, a Day-0-vouch-style follow-up). Rollback
|
||
`felhom-agent.bak-0.69.0`. OPEN (v1 scope-outs): no hub-floor auto-update, no failed-update
|
||
auto-retry, no pending-timeout auto-rollback; per-crash OnFailure can double-fire (idempotent —
|
||
future: serialize the rollback oneshot). Detail: REPORT.md.
|
||
|
||
- **v0.69.0** (2026-07-04, live on felhom-pve) — **S5: host-loss DR — safe halves shipped**.
|
||
**Part 1** `wgtunnel.InstallRecoveredKey` — writes an escrow-recovered WG privkey (create-only,
|
||
refuse-overwrite) so the tunnel re-establishes with the SAME identity/pubkey (same /32), no keygen;
|
||
wired into `--selftest=identity-consume -install-wg-key` (opt-in; pre-S3 blob → logged fresh-keygen
|
||
fallback). **Part 2** new `internal/dr` — consumes the host_loss `restore_directive` (was
|
||
logged-ignored) into an inspectable RestorePlan via AddConsumer: per-guest {vmid,archive,target,
|
||
sizing} + per-drive {durable_id→mount} + offsite PBS coord; DERIVE-AND-SURFACE only (Consumer has
|
||
no restore/destroy dep — execute-nothing is structural). Tests + red-proofs (WG create-only; plan
|
||
mode-gate). **Part 3** hub escrow-GET NOT needed (operator exports the blob via `sqlite3 writefile`
|
||
on a cp'd hub.db). **Part 4-A** re-attach wrong-disk safety already unit-proven
|
||
(`ResolveStorageDevice`: match resolves, absent/mismatch ERRORS, non-uuid scheme refused — never a
|
||
near disk). **Part 4-B (destructive in-place 9201 restore) PREPARED + OPERATOR-GATED, NOT executed**
|
||
— pre-flight green (offsite ct/9201 restorable per S4.1); the operator runs the R-consume steps +
|
||
confirms the destroy (§9-4a: CC never runs a consume/R command — see [[operator-present-one-time-secrets]]).
|
||
OPEN: the operator-run 4-B drill; guest_loss DR; hub-driven full-auto DR. Rollback
|
||
felhom-agent.bak-0.68.0. Detail: REPORT.md + doc-06 §3.5/S5.
|
||
- **v0.68.0** (2026-07-04, live on felhom-pve) — **S4.1: unattended offsite restore-test**.
|
||
**Tier-aware restore-task deadline:** `RestoreTestSpec.RestoreTaskTimeout` (0→10m default) from
|
||
`config.RestoreTestPBSRestoreTimeoutSeconds` (accessor default **120m**), set only when
|
||
`SourceTier=="pbs"` (`main.restoreTaskTimeout`); local tier UNCHANGED. Fixes the WAN restore being
|
||
killed at 10m → mid-restore teardown → leaked scratch. **Teardown "VM.Allocate" follow-up =
|
||
PHANTOM (diagnosed, not blind-fixed):** ran the restore-test on the AGENT-TOKEN path sourcing the
|
||
offsite (pbs) backup → `pass:true verified:boot+running`, teardown succeeded (`torn down
|
||
vmid=990000`, no 403), scratch band clean. The earlier 403 was the 10m-timeout consequence (guest
|
||
not yet pool-associated); the scratch is restored INTO `/pool/felhom` (ACL already grants
|
||
VM.Allocate) so teardown is authorized once the restore completes. **No ACL/host-install change.**
|
||
OPEN: publish 0.68.0 + Day-0 vouch; Tier-1/Tier-2 split for offsite-as-default; S5 DR consume.
|
||
Rollback `felhom-agent.bak-0.67.0`. Detail: REPORT.md.
|
||
- **v0.66.0 + v0.67.0** (2026-07-04, live on felhom-pve) — **S4: PBS over the tunnel**. **v0.66.0**:
|
||
wgtunnel **v4-pin** (renderConf writes the resolved A LITERAL, never DNS/AAAA; `Resolver` seam,
|
||
lowest addr; cached → steady-state zero-DNS/zero-exec) + **re-resolve watchdog** (`Manager.Watchdog`,
|
||
loop-only; handshake stale > `stale_after_seconds`=180 → re-resolve → IP-changed re-render+restart)
|
||
+ FELHOM_WG **Critical** flips (conf-install/enable/restart/handshake-read). **v0.67.0**:
|
||
**namespace-aware PBS client** (Config.Namespace → `Snapshots ?ns=`, `Verify ns=`; root-ns
|
||
unchanged) — the operator-approved fix after Phase-1 showed the ns-unaware datastore-root 403s a
|
||
per-tenant token. **Live Scenario-D (all green):** real vzdump of 9201 → **ciphertext** in ns
|
||
`demo-felhom-01` over the tunnel; ns-scoped verify=ok under the box's own `felhom@pbs!demo-felhom-01`
|
||
token; WARN gone; restore round-tripped (decrypt with box-born key → boot → teardown).
|
||
**Confirmed tenant ACL (felhom-hetzner):** `DatastoreBackup` on `/datastore/felhom-offsite/<ns>`
|
||
(NOT `/ns/<ns>`) to BOTH user `felhom@pbs` AND token (privsep=intersection; cross-ns 403);
|
||
DatastoreBackup can't prune (safety). **FINDINGS:** retarget field is `local_backup_target` (not
|
||
`backup_target`); retarget REVERTED to `local` (controller backs up ~every 30 min → single-target
|
||
offsite = near-continuous 20-min uploads; needs Tier-1/Tier-2 split); restore-test scheduler needs
|
||
a WAN restore deadline + scratch-band `VM.Allocate` before it runs offsite unattended. **OPEN:**
|
||
escrow-create (OPERATOR-PRESENT, new R); publish 0.66/0.67 + Day-0 vouch; S5 DR consume. Rollback:
|
||
`felhom-agent.bak-0.65.0`/`.bak-0.66.0`. Detail: REPORT.md + doc-06 §3.4/§4.2 + runbook §4a/§4b.
|
||
- **v0.65.0** (2026-07-04, live on felhom-pve) — **S3.1 offsite-tunnel client MTU 1420 → 1280**:
|
||
resolves `06 §4.3`'s OPEN DECISION left by the CGNAT smoke test. 1420 **silently black-holed bulk
|
||
TCP** on sub-~1480 paths (mobile ~1400, DS-Lite ~1452) — handshake+ping healthy, PBS TLS page
|
||
(and at S4 the backup itself) drops. New `const clientMTU = 1280` (RFC 8200 IPv6-minimum floor;
|
||
outer 1340 v4 / 1360 v6 fits every realistic path), **permanent + fleet-wide + family-agnostic**.
|
||
**Client-only by construction** — interface MTU caps box→PBS, advertised MSS caps PBS→box, so the
|
||
endpoint's `wg0` is untouched (zero live-endpoint risk). Golden pins exact `MTU = 1280`
|
||
(red-proofed vs a 1420 flip); no wire/JSON change. Live: agent re-rendered on restart (hash-gated
|
||
apply), conf + live iface both 1280, PBS page loads at 1280 (no regression on wired). OPEN:
|
||
true-CGNAT-SIM retest (low risk); publish 0.65.0 + Day-0 vouch (operator); S4 PBS-over-tunnel.
|
||
Rollback: `felhom-agent.bak-0.64.0` on the box. The v4-pin (§4.2 determinism) is a separate,
|
||
optional future note — NOT needed for MTU correctness.
|
||
- **v0.64.0** (2026-07-04, live on felhom-pve) — **S3 offsite WG tunnel**: new `internal/wgtunnel`
|
||
(keygen 0600/0700, marker-gated one-shot registration, agent-managed `wg-quick@wg-felhom` from
|
||
the hub's desired-state `wireguard` block via the new `desired.Syncer.AddConsumer` seam,
|
||
revoked-stays-revoked teardown, report stanza) + `FELHOM_WG` sudoers/capabilities +
|
||
`IdentityBundle.WGPrivateKey` escrow auto-inject. **`wg_tunnel.enabled` DEFAULTS FALSE** (safety
|
||
gate — rollout to Peti's box is a no-op until the production endpoint exists; enabled explicitly
|
||
on felhom-pve only). Live: tunnel to ep0.felhom.eu:443 up 3 s after enable (PBS page through
|
||
10.77.0.1:8007), reboot-persistent, revocation drill clean, 30-min keepalive soak. GOTCHAS:
|
||
hub envelope poll_interval_seconds (hub-side const 900 s) silently overrides agent poll_seconds
|
||
on cycle 1; `wg show <if> dump` leaks the PRIVATE key (forbidden everywhere — sudoers only
|
||
grants `latest-handshakes`). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix);
|
||
publish 0.64.0 to Gitea + Day-0 vouch (operator); S4 points PBS at the tunnel.
|
||
- **configs: build-golden.sh v2.0.0** (2026-07-03, @ `ceca355`; no agent version change) — **drill
|
||
findings B5 + B1 FIXED** (`DRILL-golden-098-2026-07-03.md`): the controller tag is a MANDATORY
|
||
argument (the default rotted twice — a fresh install booted a pre-floor controller, forcing the
|
||
guide's manual D.1b update) and the golden now bakes a `felhom-controller-bootstrap.path` unit
|
||
(controller deploys the moment the back-half hot-plugs the bootstrap mount — no reboot; installer
|
||
v1.9.1's reboot is a redundant belt, kept). **Golden 0.98.3** baked on the drill VM, clean-room
|
||
validated (bake integrity → isolated hot-plug proof → local-golden Day-0 → published-artifact
|
||
Day-0), published (sha256 b9a02ef1…fd01) + operator-vouched — Day-0 manifest now vouches
|
||
**agent 0.63.0 + golden 0.98.3** (the v0.63.0 vouch follow-up below is DONE). Fresh installs land
|
||
current and self-manage. NEW operator follow-up (SECURITY): the customer-config `git.token` has
|
||
Gitea package-WRITE rights — scope down + rotate (evidence-doc observation O1).
|
||
- **v0.63.0** (2026-07-03, live on felhom-pve + Gitea-published sha256 b4a89c81…) — **drill findings
|
||
B3 + B2 FIXED** (`DRILL-day0-cleanroom-2026-07-03.md`): `TokenStore.Lookup` reloads the append-only
|
||
store once on a miss (cross-process coherence with the one-shot provisioner — no more fresh-install
|
||
`/controller/swap` 401 / manual restart; size short-circuit bounds the cost; behind the
|
||
`TokenAuthority` seam) + `guesthook.InstallSnippet` issues a fenced `mkdir -p /var/lib/vz/snippets`
|
||
first (fresh boxes lacked the dir → the self-heal hook silently never installed). Sudoers gained
|
||
exactly that one grant — **ship sudoers WITH the binary** (done on felhom-pve). Red-proofed both;
|
||
Scenario-E method: compiled test suite run ON felhom-pve + live channel-health hit-path.
|
||
**OPERATOR FOLLOW-UP: bump the hub Day-0 manifest to agent 0.63.0** — until then fresh installs get
|
||
0.62.0 and the guide's D.1b restart-first step still applies (narrowed to "< v0.63.0" in the guide).
|
||
- **v0.62.0** (2026-07-03) — **audit A1 RESOLVED**: the stale-lock reaper's scan is now
|
||
pool-intersected (`staleLockController.Guests()` = `ListLXC` ∩ `Client.Pool("felhom")` members),
|
||
fail-safe skip on pool-read failure; `pve:pool-read` capability (non-critical) + `--selftest`
|
||
"pool read" line. Companion host-install **v1.9.0** adds `Pool.Audit` to `FelhomAgentGuest` —
|
||
**deploy order on any box: rescope ACL first, then this agent.** Per
|
||
`SPIKE-a1-pool-membership-read-2026-07-03.md`; red-proofed tests in stalelock_pool_test.go.
|
||
- **2026-07-03 — CLAUDE.md refreshed**: version narrative removed (state lives HERE + CHANGELOG top), layout completed (all 17 internal packages + cmd/felhom-opsign); deploy runbook now in the `felhom-build-deploy` skill (`felhom.eu/skills/`).
|
||
- **2026-07-03 — `REUSE.md` exists at the repo root** (canonical helpers / format-safety guards / traps / seams, code-verified); maintenance rule active: update it in the same commit that changes a shared helper.
|
||
- **v0.61.0** (2026-07-03) — blast-radius audit fixes **B1 + D1 + D2 + D3** from
|
||
`felhom.eu/documentation/audits/AUDIT-blast-radius-hostroot-localapi-2026-07-02.md`: random temp
|
||
staging for root-installed scripts (+ sudoers/manifest glob updates), mkfs-wrapper member/RO
|
||
re-checks (validated by `scripts/mkfs-guarded-harness.sh`), classifyClaim empty-lsblk fail-safe,
|
||
and the blank-format anti-retarget (durable-id-bound, AGENT-001's benign-branch twin).
|
||
- Deployed on demo host `felhom-pve` (node `demo-felhom`), non-root `felhom-agent` service user,
|
||
pool-scoped token (`felhom` pool).
|
||
|
||
## Open threads
|
||
|
||
- Deferred audit items (housekeeping/design, all INFO): C1 (controller-swap version floor), C2 (NAS
|
||
server allowlist), A2 (gate journal cross-check), B2–B5, E1/E2.
|
||
- Drive-enrollment leftovers: (a) `runStorageInit` slow-device detached-format polling; (b) Impl-3
|
||
shared-box operator format gate.
|
||
- BUNDLE leftover: non-root agent can't read the PBS key; migration must preserve cert/key/tokens.
|
||
- Not run (needs a supervised session): the destructive D1/D3 live proofs (real mkfs on a crafted
|
||
member; a live /dev re-enumeration race during a real format).
|