diff --git a/REPORT.md b/REPORT.md index b046ba6..e445d07 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,63 +1,58 @@ -# REPORT — agent v0.80.0: PBS DR tier SLICE 2, the apply-bridge — DEPLOYED + ADOPTION-PROVEN + PUBLISHED +# REPORT — agent v0.81.0: NAS verify-before-commit (retry=0 + classifier + detached verify job) — DEPLOYED -**Date:** 2026-07-10/11 · **Class:** implementation (agent) + demo live check + publish. Baseline -v0.79.0 → **v0.80.0**, live on felhom-pve (`felhom-agent 0.80.0`, capabilities **59/59, degraded=0**), -**PUBLISHED** to Gitea (sha `f2ba62ca6aca6e24a8d08706ea0dc3ae43940a63e9bdf57d4606ad1756cf06d2`; -anon round-trip verified; the deployed demo binary is byte-identical). -Consumes hub v0.44.0 (slice 1). **PENDING: the supervised Peti phase** — -`felhom.eu/documentation/pilot/RUNBOOK-peti-pbsdr-2026-07-11.md` (Viktor + Peti; his box has no -tunnel/OOB yet, so box prep is his hands). +**Date:** 2026-07-11 · **Class:** implementation (agent half of the cross-repo "NAS +verify-before-commit + page redesign" task). Baseline v0.80.0 (`300f06722b`) → **v0.81.0** +(`added9d`), live on felhom-pve. Pairs with controller v0.113.0 + host-install v1.13.0. +**Full task-wide record (red-proof outcomes, live A–E transcripts, teardown):** +`felhom-controller/REPORT.md`. Evidence base: SPIKE-nas-verify-2026-07-11 (b57f6c1). -## Part 0 — grant diagnosis (the carried spike §6 flag), evidence recorded +## What shipped (agent) -- Status reads (`/nodes/.../storage/.../status`) pass with NO storage grant — `FelhomAgentBase` - (`Datastore.Audit` at `/`, propagate) covers them. That + the demo's `local` backup target is - why the purged grants never showed. -- The WRITE path is the gap: token vzdump → **403 `(/storage/felhom-pbs, Datastore.AllocateSpace)`**. - After the §4b dual-grant re-assert (both entries, user+token): token vzdump of 9001 → `stopped OK` - (9001 also joined `/pool/felhom` — the VM.Backup half; pool-default conformance). -- The bridge's `grant` op therefore codifies EXACTLY the dual-grant, nothing more. +- **`retry=0` in the production NFS option string** (`netmount.go mountOptions`, Q4-vi): a dead-NAS + on-demand access fails clean in ~3.8 s (ENODEV) instead of a 91 s wedge; verify failures classify + as `No route to host` instead of a diagnostic-free systemd timeout. SMB unchanged (mount.cifs + rejects `retry=`). Installed units are NOT rewritten (none existed on the demo anyway). +- **`internal/storage/netverify.go`** — `ClassifyNetVerifyFailure(journalTail, tcpReachable)`: + pure, first-match table on the Q4 VERBATIM substrings → `unreachable | nfs_export | smb_auth | + smb_share | timeout | mount_failed`; `nfs_export` deliberately merges not-found/not-permitted + (NFSv4 identical strings). String-based BY DESIGN — every mount failure exits rc=32. +- **Detached verify job** (`localapi/netverifyjob.go`, formatjob shape but IN-MEMORY single slot — + a restart empties the slot; `GET /netstorage/verify-status` phase `none` is the CONTROLLER's + rollback signal, Scenario F): add = decode → role-gate → **sync fast-fail** (full spec validation + + 2 s TCP pre-probe; unreachable = HTTP 502 `{code:unreachable}`, NOTHING installed) → SMB creds + (0600) → EnsureNetworkMount → detached verify: trigger read through the automount, success judged + from **/proc/mounts ONLY** (§8: EACCES on a 0700 export is a GOOD mount), failure = journal + classification + **auto-rollback** (RemoveNetworkMount + creds). Single-flight: second add = 409. +- **Unprivileged journal read** (`journalctl -u -n 20 -o cat`, NO sudo, NO new sudoers + grant): needs the `systemd-journal` group (host-install v1.13.0; existing hosts: + `usermod -aG systemd-journal felhom-agent`). Unavailable ⇒ degrade to `mount_failed` + hint, + still rolled back. +- New exports: `storage.NetworkMountedAt`, `storage.NetworkEndpointReachable`. REUSE.md updated + (helpers + the in-memory-vs-persisted job pattern row + the four Server net-verify seams). -## What shipped (laws + red-proofs) +## Tests + red-proofs (agent side; outcomes verbatim in felhom-controller/REPORT.md) -- `internal/pbsdr` bridge (adoption-first / verify-pin-before-consume / set-only / loud - consumed-failed / escrow seed / descriptor-hash marker) + `configs/felhom-pbs-apply` + - `FELHOM_PBSDR` sudoers + 3 capability entries + `WirePBSDR`/`PBSDRStatus` wire/report + - `hub.ConsumePBSToken` (PLURAL `/hosts/` route). Details: CHANGELOG v0.80.0. -- **Red-proofs (all run, mutations reverted, gate green):** - - set-only: a remove+re-add path in the re-apply → `TestSetOnlyLaw` FAILS (recorded verbs - `[remove, create, grant]` ≠ set-only) — the K-destruction guard bites. - - stdin secret: secret passed as argv → `TestFreshPath_SecretOnStdinNeverArgv` FAILS with the - secret visible in Args. - - verify-before-consume: consume reordered before the probe → `TestVerifyPinBeforeConsume` - FAILS (`consume calls = 1, want 0`). +A1 retry=0 (revert → fail) · A2 classifier table incl. merged nfs_export + empty-journal +degradation (exit-code mutant → every non-generic row fails) · A3 rollback effects via fakes +(drop rollback → `removed=[]` + creds linger) · A4 §8 truth table (readability mutant → both rows +fail) · A5 pre-probe zero-install (skip probe → 200-with-install) · A6 single-flight + no-job shape +(drop check → 409 lost). Green gate `go build && go vet && go test ./...` PASS (known +TestGenerateRecoveryCode wordlist flake hit once, clean on re-run). -## Demo live check — the ADOPTION path (2026-07-10 22:00 CEST) +## Deployed + verified (felhom-pve) -Enabled "PBS DR tier" on demo-felhom with storage-id `felhom-offsite` (hub v0.44.0 UI, logged-in -browser). The full slice-1+2 chain fired live: -- hub: `tenantsync: provision ok (ns=demo-felhom)` on ep0 → secret staged consume-once → descriptor - → generation 10. -- agent: fetched gen 10 → **ADOPTED** the manually-built `felhom-offsite` entry — journal: - `adopted entry keeps its own tenancy (namespace "demo-felhom-01"; descriptor says "demo-felhom" - — entry wins, never repointed)` → `converged state=adopted`. -- **Asserts:** staged secret UNCONSUMED (`host_pbs_secrets.consumed_at = NULL` in the hub DB — the - no-consume law held); `pvesm status felhom-offsite` active (demo DR green); - `escrow.pbs_storage_id=felhom-offsite` seeded into agent.json. -- **Live finding, fixed same session (3d1323c):** the escrow seed's tmp+rename failed — - `/etc/felhom-agent` is root-owned while agent.json is agent-owned, so rename (a directory - mutation) is impossible for the non-root agent. Rewritten as park-a-recovery-copy-in-state-dir + - in-place O_TRUNC write; re-proven live (marker removed → re-adopt → seed OK). The warn-and- - converge fail-safe behaved as designed in between. -- Benign note: the report `pbs_dr` stanza appears from the SECOND heartbeat after a restart (the - startup collect races the first bridge Apply). +`.bak-0.80.0` kept → `install -m0755` → `usermod -aG systemd-journal felhom-agent` → +`systemctl restart felhom-agent`. Verified: `felhom-agent 0.81.0`; `id felhom-agent` → +`…,999(systemd-journal)`; clean start journal (enrolled drive re-bound, local-api listening, +desired-state gen 10). Live proof the journal path works: Scenario A's failure detail carried the +REAL `mount.nfs4 … reason given by server: No such file or directory` lines read unprivileged. -## Publish train + pending +**NOT published:** 0.81.0 is deployed on the demo only — no Gitea publish, Day-0 manifest untouched +(the publish train is the operator's separate decision); Peti's box untouched. -- 0.80.0 published (`publish-agent.sh` on 180): pre-delete 404 → PUT 201 → round-trip sha match. - Demo's live binary sha == published sha (the GL never-vouch-unpublished rule holds). -- **PENDING (the committed runbook, Viktor+Peti):** box prep (sudoers+wrapper+wg_tunnel enable — - his box has NO WG peer/OOB today), opsign signed update to 0.80.0, hub-UI tier enable (the FRESH - path: verify → consume → **K born** → degradation clears), then the ceremony one-liner (fresh R, - Peti keeps) → auto-confirm → first offsite run. Operator follow-up: Day-0 manifest vouch of - 0.80.0 when it becomes the floor target. +## Observations + +- Pre-existing (2026-07-08, unrelated): `lanresolver: cannot list provisioned guests: permission + denied` — `/var/lib/felhom-agent/guests` is root-0700 under the non-root agent. +- `RemoveNetworkMount` leaves the empty mountpoint dir + a `not-found failed` residual unit listing + (until `reset-failed`/reboot) — cosmetic, documented in the task report.