diff --git a/CONTEXT.md b/CONTEXT.md index e0878f7..f894c75 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,40 @@ > Snapshot of the current state + open threads. Authoritative history lives in `CHANGELOG.md` (top > entry = current); the end-of-task detail lives in `REPORT.md`. +## R-199 (v0.125.0) — links 6–8 of the recovery chain, assembled and walked + +`POST /escrow/recover-offsite-password` (pinned local API, `withGuest`): the controller supplies the +customer's recovery code, the agent fetches THIS host's own sealed blob from the hub +(`hub.Client.FetchIdentityEscrow` → `GET /hosts/{id}/escrow`, hub >= v0.94.0, self-scoped by the +per-host key), unseals it via `escrow.OffsiteKeyRecoverer`, and returns **only** the offsite restic +repository password plus its sha256. + +**Rules that must not erode:** +- **Only that field.** Not the tunnel token, not the PBS token, not the WG key — the controller is a + trust tier down and needs none of them. Narrowing cost nothing and is not recoverable later. +- **The unseal stays in the agent.** `age` is an agent runtime dependency (`/usr/bin/age` — hardcoded, + no config override; 1.2.1 on demo-felhom) and is deliberately absent from the controller image. +- **R:** in memory for one call, cleared on the success path AND every failure path, never on disk, + never in argv, never logged at any level including inside an error, never echoed. Verified live: 0 + log lines, 0 files, 0 leftover `felhom-idesc-*` dirs, with a positive control proving the search worked. +- **Three distinct outcomes**, not one generic failure: no blob (404), a bundle that opens but predates + the field (409 — pre-fork-4, cannot be retro-fitted), a code that does not open it (400 — fail-closed + at age's KDF, nothing written). +- **The wiring is pinned by an AST walk** (`cmd/felhom-agent/escrow_recover_wiring_test.go`): + `main` → `runDaemon` → `buildLocalAPIServer`, an `escrow.OffsiteKeyRecoverer` constructed there, the + `Options.EscrowRecovery` field present, and the fetcher calling the DAEMON's own `hubClient` (the + self-scoping that makes cross-host retrieval impossible is a property of WHICH key is used). + Links 6 and 7 were two of this project's six built-but-never-wired instances. + +**Proven live on demo-felhom 2026-08-04:** recovered sha256 == on-disk sha256 == the hub's stored hash. +A wrong code five minutes earlier failed closed. **The chain stops at link 8** — nothing installs a +recovered password, reopens a repository, or restores a file. + +**§8.6, fixed while here:** `runSelftestIdentityConsume`'s success line used to recite +"tunnel_token + pbs_token", which became a misstatement when v0.77.0 sealed the repository password +into the same bundle — anyone reading it would conclude the password was not there. It now names what +THIS bundle carried and what it did not. + ## Current - **2026-08-03 — v0.123.0 (R-185): a tier the box cannot READ now says so.** The agent's token had diff --git a/REPORT.md b/REPORT.md index c7df8ab..e8d6584 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,205 +1,57 @@ -# REPORT — R-185: a tier the box cannot READ must say so +# REPORT — agent v0.125.0: open the sealed bundle, return one field (2026-08-04, R-199 links 7–8) -**Date:** 2026-08-03 · **Repos:** `felhom-agent` **v0.122.0 → v0.123.0** (`fe14bc6`) · `felhom.eu` -installer **1.23.0 → 1.24.0** (`688470c`, tag `installer-v1.24.0`, manifest bump `311dc06`) · -**no hub change and no hub bump** — the hub already alerts on a degraded critical capability, which is -why that mechanism was chosen. +**Released and deployed.** Published sha256 `f7d8339b53d92a6c45be7eaf189469a041b6b00b758a64511c0479beae7016b3`, +**verified by independent download** before install; running on demo-felhom (`felhom-agent --version` +→ `0.125.0`, `systemctl is-active` → `active`). **NOT vouched** — vouching stays the operator's act. ---- +## What shipped -## 1. Baselines, re-read on arrival +- `internal/escrow/recover.go` — `OffsiteKeyRecoverer`: fetch → unseal → extract, with **distinct** + errors for "no blob", "the bundle predates the field", and "the code did not open it". +- `internal/hub/client.go` — `FetchIdentityEscrow` (link 6's first client, ever). +- `internal/localapi/escrow_recover.go` + route registration — link 7's first production caller. +- `cmd/felhom-agent/main.go` — the production wiring, and the §8.6 selftest message fix. -| Repo | `main` @ commit | Version | Matched §1? | -|---|---|---|---| -| `felhom-agent` | `0b28eae7bb14` | `v0.122.0` | **yes** | -| `felhom.eu` | `7a5694341d59` | installer `1.23.0`, both `--ref=installer-v1.23.0` (lines 327, 372) | **yes** | +## The result -Highest register ID in use **R-189**; R-190+ confirmed free by grep, and none was needed. +**Proven live on demo-felhom, 2026-08-04:** the recovered offsite repository password's sha256 equals +the one on the box's disk, and equals the hash the hub had independently recorded. A deliberately wrong +code five minutes earlier produced +`escrow: the recovery code did not unwrap the identity escrow … exit status 1` and wrote nothing — +which also proves the fetch and the unseal ran independently of the success. -## 2. Part 0 — the measurements, before anything was designed against them +**R persisted nowhere**, searched rather than claimed: 0 agent-journal lines, 0 controller-log lines, +0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover `felhom-idesc-*` staging +dirs — with a **positive control** (a planted copy found, then removed and not found) so the sweep is a +measurement. -**The row's three-way observation, reproduced unchanged:** +## Tests -| leg | result | -|---|---| -| root, `pvesh … /storage/felhom-backup/content` | **3 archives** — 6.1 / 6.2 / 6.3 GB, dated 08-01, 08-02, 08-03 | -| the **agent's token**, same endpoint | `{"data":[]}` | -| the agent's token, `/storage/local/content` | **8 entries** — the token works where it is granted | +`go build ./... && go vet ./... && go test ./...` — rc=0; `agent_gates.py --fast` — OK. +New: `internal/escrow/recover_test.go` (real `age` crypto) and +`cmd/felhom-agent/escrow_recover_wiring_test.go` (AST). -So the token is the variable, not the storage. Two further checks removed the obvious alternative -explanation: guest **9201 IS in the `felhom` pool** (so `VM.Backup` is not the discriminator), and -`pveum acl list` showed ACL rows for `/storage/{local,local-lvm,felhom-pbs}` and **none** for -`/storage/felhom-backup`. +**Red-proofs, both observed failing:** +- Returning `bundle.PBSToken` instead of `ResticRepoPassword` → *"the recovered key is not the sealed + repository password (len 24 vs 64)"*. That is the mutation that mattered — every field of that bundle + is a non-empty string that looks like a secret. +- `_ = escrowRecoverer` with the `Options.EscrowRecovery` field dropped → the AST test fails with + *"localapi.Options … carries no EscrowRecovery field"*. -**The permission query, asked by the token itself — and the obvious reading is wrong:** +**One red-proof did NOT work on the first attempt and is recorded, not quietly redone:** the R-leak +mutation wrote R into the walked directory and the test still passed, because a later failing call +overwrote the leak file with a different code while the test scanned only for the first. **The test was +wrong.** It now asserts the TMPDIR tree is EMPTY (nothing in the test creates anything there), with the +content scan kept as defence in depth. Re-run: FAIL, *"the unseal left 1 file(s) behind under TMPDIR"*. +A second mutation (commenting the wiring line) failed to compile, which is not a red-proof either; the +compiling version is above. -``` -/storage/felhom-pbs → {"Datastore.Allocate":1,"Datastore.AllocateSpace":1} -/storage/felhom-backup → {"Sys.Audit":1,"SDN.Use":1,"Datastore.Audit":1} -``` +## Scope -The ungranted path answers **neither empty nor 403**. It answers with the privileges **inherited** -from the box-wide `/` grant. A probe asking *"did the path come back?"* — or *"does it hold -`Datastore.Audit`?"* — would have reported the blinded storage **healthy**. This is exactly what §3 -required to be measured rather than assumed, and it changed the design: the probe tests -`Datastore.AllocateSpace` specifically, and a red-proof pins that choice. +Links 6–8 only. **Nothing installs a recovered password, reopens a repository, or restores a file** — +R-200's remaining half and R-201. -## 3. The probe +## Observation -`Client.Permissions` reads `/access/permissions?path=/storage/` **as the agent's own token** -(asking as root answers a different question and always says yes). `storeGrantStatuses` emits one -`capability.Status` per configured tier. - -**Deviation from §5/§8.1, stated because a recommendation not followed gets a line:** the spec asked -for the sudo `Prober` to be minimally generalised. This repo already has the better-established -pattern for exactly this — `poolReadStatus`, composed **around** the prober, with the comment *"an API -read does not belong inside the sudo-policy probe"* (v0.62.0, audit A1). The probe follows that -precedent instead. `capability.Status` is untouched either way, which is the constraint that mattered. - -**Decisions:** - -- **The probed set comes from the box's own `BackupTiers()`**, never a fixed list — a hardcoded probe - list is the defect reproduced inside the fix. -- **Critical** (§8.3): the hub alerts only on critical, so a non-critical entry would ride the report - and alert nobody — the same silence with extra steps. **Except** the `local` fallback target, which - host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it - does not page. Turning an ordinary documented setup into an alert is how a signal becomes something - an operator archives unread. -- **It never consults content**, so it cannot alarm on a newborn tier by construction — a stronger - guarantee than gating on emptiness would be. -- **It never reports ok when it could not ask.** Unreachable PVE is degraded: a self-check that fails - open converts *"I do not know"* into *"fine"*. - -## 4. The installer — the root cause was not where the row or the task expected - -Both assumed `PVE_STORAGES` (the fixed grant list) was the culprit. **It is not.** -`configure_backup_target` has two arms: - -- **Case A** creates the storage and calls `felhom-backup-target-apply grant` in the same breath — a - box that builds its own target has always been correct. -- **The Scenario-F arm** — *"the target already exists, leave it exactly as it is"* — **returned - without granting**. - -So a box whose `felhom-backup` pre-dated the install (created by the vzdump-target-move runbook, or -surviving a reinstall — which is both demo boxes) pointed `local_backup_target` at a storage its own -token could not read. The reuse arm now ensures the ACL through the same guarded wrapper. - -**Scenario F is unviolated:** the storage DEFINITION is still untouched. Granting the role the agent is -supposed to have on the target this same script is about to write into `agent.json` is finishing the -job, not retargeting the box; `pveum acl modify` is idempotent, so a box that already has it is -unchanged and a box whose token was rotated gets it back. - -**`$BACKUP_TARGET_ID` is deliberately still NOT in `PVE_STORAGES`,** and the comment now says why: that -list is granted in step 4/5, *before* `configure_backup_target` runs in step 6, and `--acl-storages` -entries are preflight-checked for existence. Adding it there would grant on a storage that may not yet -exist and would split ownership of the decision across two places. - -**A gate now asserts it:** every arm of `configure_backup_target` that resolves the target must also -grant on it — the check that would have caught this. - -## 5. Live validation, in order - -| # | evidence | -|---|---| -| 1 | Part 0's measurements above, taken **before** any change | -| 2 | **The signal that has never existed**, on the still-blind box: `capability DEGRADED … capability=pve:store-grant:felhom-backup … reason="the agent token lacks Datastore.AllocateSpace on /storage/felhom-backup (grant FelhomAgentStore there) — this tier's archives are INVISIBLE to the agent and it is never restore-tested" critical=true`, with `ok=69 total=70 degraded=1`. The hub: `Host capability: demo-felhom-8363b5 ok → degraded (agent_capability_degraded)` and **`Operator email sent`** | -| 3 | Grant applied (user **and** token — a privsep token's rights are the intersection); the token then lists **3 archives** where it listed none, and the permission answer becomes `{"Datastore.AllocateSpace":1,"Datastore.Allocate":1}` | -| 4 | `capabilities self-check ok=70 total=70 degraded=0`; the hub: `degraded → ok (agent_capability_recovered)` | -| 5 | **The host tier is a due-check candidate for the first time on that box**: `tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst" proven=""` — and the settle rule applies to it exactly as to the others, selecting the **08-02** archive because the 08-03 one has not settled 24 h | -| 6 | The served installer over HTTPS: `SCRIPT_VERSION="1.24.0"`, and the served bytes carry the fix itself, not merely the version | - -## 6. The other machines - -- **demo-hp CARRIES THE SAME DRIFT — and was fixed.** `local_backup_target=felhom-backup`, ACL rows for - `local`, `local-lvm`, `felhom-pbs` only. §8.6 assumed a single affected box; the same one-line, - additive, path-scoped, idempotent grant applies to the other, and leaving a known-blind backup tier - on a Tier-0 box after finding it would be this row happening twice. Granted (user + token); its - token now lists **4 archives**. It still runs agent `0.120.0`, so it has no probe yet — that arrives - when you vouch. -- **The tester's box was NOT touched** (Tier 2, protected). **What is known without connecting to it:** - it very likely carries the same drift — the mechanism is the Scenario-F reuse arm, which fires on - any box whose target pre-dated its install, and its target was moved by the very runbook that - creates that condition. It is due for reinstall, and installer 1.24.0 fixes it on the way in. - -## 7. Tests and red-proofs - -Green gate: `go build ./... && go vet ./... && go test ./...` — rc=0, plus `agent_gates.py` and -`repo_gates.py` all OK. Test runs and commits were always separate commands. - -| # | Test | Asserts | Mutation | Observed | -|---|---|---|---|---| -| A | `TestStoreGrant_ForbiddenStorageIsDegradedAndNamed` | degraded, critical, naming storage **and** role | probe removed from `probeAll` | **FAIL** — `main.go never calls storeGrantStatuses` (via the seam test); with the wrong-privilege mutation: `must be DEGRADED, not "ok"` | -| A′ | `TestStoreGrant_InheritedPrivilegesAreNotAGrant` | the measured trap: inherited ≠ granted | probe `Datastore.Audit` instead | **FAIL** — `checking for the wrong privilege reports a blinded storage healthy; got "ok"` | -| B | `TestStoreGrant_GrantedButEmptyIsHealthy` | a readable-but-empty tier is healthy | — (it never reads content, so emptiness cannot reach it) | pass | -| B′ | `TestStoreGrant_TheFallbackTargetIsNotCritical` | `local` is reported but does not page | gating removed (`return true`) | **FAIL** — `must not page the operator about an ordinary, documented configuration` | -| C | `TestStoreGrant_ForbiddenAndNewbornAreDistinguishable` | different status **and** different capability id | — | pass | -| — | `TestStoreGrant_UnreachablePVEIsDegradedNotOK` | unknown ≠ ok | — | pass | -| F | `hostinstall_gates.py` backup-target assertion | every resolving arm also grants | reuse arm reverted | **FAIL** — `resolves the backup target in 2 place(s) but grants in only 1` | -| H | `TestMainWiresTheStoreGrantProbe` | **AST** of `main.go` | call commented out | **FAIL** — a `strings.Contains` check would have passed | - -**A hollow test caught and fixed before it shipped:** the first draft of `storegrant_test.go` -re-implemented the verdict branch inside the test. It passed, and would have kept passing while -production diverged. The decision was extracted into `storeGrantVerdict` and the tests now call it. - -**Scenario B's red-proof, honestly:** the spec asked for "degrade on an empty content listing" as the -mutation. That is not a mutation of this code — the probe never looks at content, which is a stronger -guarantee than gating on emptiness. The gating red-proof above (`storeGrantCritical`) is the one that -exercises the guard that does exist, and it fails as required. - -## 8. Files, commits, tag - -`internal/proxmox/query.go` (`Permissions`), `cmd/felhom-agent/main.go` (`storeGrantStatuses`, -`storeGrantVerdict`, `storeGrantCritical`, `storeGrantRequiredPriv`, wiring), -`cmd/felhom-agent/storegrant_test.go`, `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`, `REPORT.md`. -`felhom.eu`: `scripts/felhom-host-install.sh`, `scripts/hostinstall_gates.py`, `scripts/CHANGELOG.md`, -`manifests/webpage.yaml`, `CONTEXT.md`, `STATUS.md`, `documentation/architecture/00-capability-map.md`, -`documentation/backlog/OPEN-ITEMS.md`, `documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md`. - -**Commits** — `felhom-agent`: `fe14bc6` (v0.123.0). `felhom.eu`: `688470c` (installer 1.24.0), `311dc06` -(manifest refs), `e3187c8` (docs). **Installer tag:** `installer-v1.24.0`. - -## 9. Deployment - -Agent released through `release-agent.sh` — tag `v0.123.0`, sha256 -`74910135ac4feb1b7f0ad4dbd1541d965cbc0fe70d4f47b62ebf7e4bfb962453`, round-trip verified. The -**published bytes** were downloaded and deployed: the running binary's sha matches the published one. -`felhom-agent --version` → **0.123.0**, `systemctl is-active` → active, prior kept as `.bak-0.122.0`. -**NOT VOUCHED** — that stays the operator's act. - -## 10. Registers - -- **R-185 → CLOSED** (shipped + proven live on both demo boxes), with the corrected root cause - recorded on the row. -- No new IDs minted; `ROADMAP.md` contains no R-185 row, so there was nothing to collapse. -- **The capability map's whole-guest row was OPTIMISTIC and now says so:** every live restore-test it - cited is on the OFFSITE tier, and the HOST tier was not merely unproven but *unprovable* on both - demo boxes. It now records that, the closure, and that it will carry a host-tier live proof when one - runs. -- The vzdump-target-move runbook's item 5 **predicted this** and is annotated, not rewritten: it - expected a 403 on backup, and the reason it did not surface that way is that `vzdump` writes through - a root path, so backups kept landing while the agent's *read* stayed blind. -- `CONTEXT.md`: agent-side entry, plus `felhom.eu` **S-21** (empty ≠ forbidden; the measured trap) and - **S-22** (the Scenario-F arm must finish the job). - -## 11. Teardown - -**Nothing was provisioned.** No scratch storage, no fixture grant, no probe tag, no scratch package -version. The two ACL grants are the intended durable change; the only other mutation was the -installer label, which is reversible by moving the tag. - -## 12. Observations — noticed, recorded, NOT acted on - -- **Both demo boxes are now due for a host-tier restore-test**, which has never run on either. The - scheduler will pick it up within 6 h unattended (a ~6 GB local restore — fast, and cheaper than the - offsite ones). Expected, not a defect, and the first host-tier proof this fleet will have. -- **`--acl-storages` semantics are unchanged and the automatic grant does not consult it.** If an - operator passes `--acl-storages` deliberately excluding the backup target, the target is still - granted by the resolution path. That is the correct precedence — a box cannot function with an - unreadable backup target — but it is a place where an override is not absolute, and it is written - here rather than left to be discovered. -- **`storeGrantRequiredPriv` is a single privilege**, chosen from measurement. If PVE ever changes - which privilege gates content listing, the probe would report healthy while the tier is blind. The - test asserts the constant's value so a change forces a re-measurement, but nothing detects a change - on PVE's side. -- **Ten pre-existing `gofmt`-unclean files** remain in the agent repo (unchanged from yesterday's - observation); every file touched here is clean. +`ageBinary` is the hardcoded `/usr/bin/age` with no config override. Present on demo-felhom (1.2.1); a +host that installs `age` elsewhere would fail the unseal at the last step with no configuration escape.