diff --git a/CONTEXT.md b/CONTEXT.md index 27ae036..80f1a56 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -17,6 +17,27 @@ ## Standing rulings +**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes +(2026-08-04). Scope: those two boxes, not the fleet.** + +Four SCHEDULED runs overnight, none triggered by hand: demo-felhom host **83.8 s** / offsite +**540.4 s**; demo-hp host **109.3 s** / offsite **300.1 s**. Every one restored into a scratch guest, +booted, verified and destroyed itself. + +*What this closes.* Until yesterday every live restore-proof this project held was on the OFFSITE +tier. The on-box tier — the one an ordinary recovery uses — had never been proven on either box, and +not because it failed: the agent could not read the storage it lives on (R-185), so it never saw an +archive there to test. + +*What was observed for the first time.* Both boxes had BOTH tiers due simultaneously. Never-proven +sorts first, so each took its host tier, deferred the offsite one, and picked that up on the next +evaluation six hours later — R-86's ordering and the one-heavy-operation gate, working together, +unsupervised. The host-tier proof then reached the hub through R-189's merge, which is that path +carrying a host-tier entry for the first time. + +*The asymmetry worth remembering:* a host-tier restore is **83–109 s**; an offsite one is +**300–540 s**. The tier that matters for an ordinary recovery is also the cheapest to prove. + **S-21 — an empty listing cannot distinguish FORBIDDEN from NEWBORN, so the box asks the permission question directly (2026-08-03, R-185; agent v0.123.0 + installer 1.24.0).** diff --git a/REPORT.md b/REPORT.md index 4e4dccc..87b5c57 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,137 +1,173 @@ -# REPORT — R-86 Part 2: the staleness window learns each tier's rhythm (2026-08-03) +# REPORT — RUNBOOK: the first host-tier restore-test, on both boxes -> **A LATER SESSION THE SAME DAY (R-189 · R-188 · R-186) touched this repo for REGISTERS AND DOCS -> ONLY** — `OPEN-ITEMS.md`, `STATUS.md`, `CONTEXT.md` (S-19, S-20) and the capability map. **No hub -> code changed and the hub was not bumped**: the defect was the agent no longer sending -> `restore_tests[]`, not the hub failing to read them. That session's report is -> `felhom-agent/REPORT.md`. What follows is the last hub IMPLEMENTATION, v0.91.1, which is what this -> file is for. +**Date:** 2026-08-03 → 2026-08-04 · **Repos:** `felhom.eu` docs + registers only. **Nothing was built +and no version was bumped.** The only binary that moved is the already-published `v0.123.0`, onto the +box that did not have it (P1). Baselines re-read and matched: `felhom-agent` `72161f6cf010` / +`v0.123.0`; `felhom.eu` `e3187c86d58d` / hub `v0.91.1`, installer `1.24.0`. Constants re-confirmed at +source: `defaultRestoreTestEvalInterval` **6 h**, `defaultRestoreTestSettle` **24 h**. -**Date:** 2026-08-03 · **Repo:** `felhom.eu` hub **v0.90.1 → v0.91.0 → v0.91.1** -(`323f45a`, `046df30`), manifests `687fedd`, `ff2655c`. Ships **with** `felhom-agent` v0.121.0, not -after it. Ruling recorded: **ep0 is Tier 2, protected**. +**Outcome: four scheduled runs, all passed, nothing triggered by hand.** -## 1. Baselines, re-read on arrival +--- -| Repo | `main` @ commit | Version | Matched §1? | -|---|---|---|---| -| `felhom.eu` | `e34b614e5b65` | hub CHANGELOG top entry `v0.90.0`; `manifests/hub.yaml` ran **`0.90.1`** | **Yes — and the discrepancy was real** | +## 1. Preconditions -The missing `v0.90.1` CHANGELOG entry is backfilled. It was built, deployed and recorded in -`REPORT.md` + the R-182 row on 2026-08-03 (`f21e7ca`) but never given a CHANGELOG entry — and -`REPORT.md` is overwritten every session, so the per-repo history under-reported what was running. +| # | demo-felhom | demo-hp | +|---|---|---| +| **P1** agent | `v0.123.0` ✓ | **`0.120.0` — below the 0.121.0 floor, so it could not become due at all.** Remediated with the published `v0.123.0` (sha `74910135…`, deployed sha identical) | +| **P2** tiers | host `felhom-backup` + offsite `felhom-pbs` (weekly) | **the same — the runbook expected demo-hp to have no offsite tier, and it has one**, active with 2 snapshots | +| **P3** grant | `ok=70 total=70 degraded=0` | `ok=70 total=70 degraded=0` (once the probe existed) | +| **P4** storage | `/dev/sdb → /mnt/hdd_1`, ext4, on the N100 | `/mnt/nvme-1tb` on the t740 — **different hosts, different disks ⇒ INDEPENDENT ⇒ parallel is safe** | +| **P5** space | target 889 GB free; restore pool `local-lvm` 358 GB | target 925 GB free; restore pool `data` **53.9 G at 30.79 %** (~37 GB free) against a 2.35 GB archive — adequate, and measured *because* that pool is the over-subscribed one | +| **P6** candidate | `…2026_08_02-04_42_14.tar.zst` (08-02) | `…2026_08_02-04_49_29.tar.zst` (08-02) — both correctly the settled archive, not the day's | +| **P7** other heavy work | daily backup ~04:44, outside the window | daily backup ~04:49, outside the window | -## 2. Why Part 2 could not ship later - -The agent now proves a tier **once per archive generation**, so a tier backed up weekly is proved -weekly — correctly, in perfect health. `restoreProvenStaleAfter` was a flat 7 days and its own comment -derived that number *from the cadence R-86 removes*. - -The brief said a weekly tier would "sit exactly on that line". That is literally true, and the -arithmetic is worth stating: proofs land at `archive + settle`, so just before the next proof the -current one is **exactly one interval old** — 168 h against a 168 h window, and `age > window` is false -by a hair. It did not sit near the line, it sat **on** it. Any ordinary delay — a backup landing late, -one evaluation deferred behind a running backup — tips a healthy tier into a nightly alarm. - -## 3. The window as derived +## 2. The due verdicts before the run, quoted ``` -interval = declaredArchiveInterval(tier) // host 26h, offsite 8d -if observedOK && observed > interval { interval = observed } // observation may only WIDEN -window = clamp(4 * interval, floor 7d, cap 12d) +demo-felhom tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst" + reason: newest settled archive … has not been proven; nothing proven on this tier yet + tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z" + reason: … has not been proven (last proven archive was a different one) + +demo-hp tier=felhom-backup due=true archive="…2026_08_02-04_49_29.tar.zst" + tier=felhom-pbs due=true archive="…2026-07-28T19:19:45Z" ``` -- **4 generations** — the settle generation plus ~3 missed opportunities: deliberately the same - tolerance the flat constant expressed. The change is to the *rhythm*, not to the patience. -- **The declared rhythm is `backupStaleAfter` (26 h) / `offsiteBackupStaleAfter` (8 d)** — the very - thresholds the backup-freshness checker already judges those tiers against. Not a second opinion: - if the two checkers disagreed about how often a tier receives an archive, one would be alarming on - the other's model. -- **The observation** comes from reports the hub already holds: `pbs_snapshots[]` plus successful - `backups[]` attributed by **target type** (slice-A.4 — a PBS-targeted vzdump appears in *both* - arrays, and classifying by array membership would attribute an offsite archive to the host tier). - Mean gap over distinct archive times; fewer than two generations ⇒ unobservable. -- **Floor 7 d** — the old constant kept as a floor, so no tier is judged more tightly than before. -- **Cap 12 d** — strictly inside the 2-week offsite retention with two days to spare, so a tier is - never called stale against an archive PBS has already pruned. +**Both boxes had BOTH tiers due**, which made §4's ordering question live rather than theoretical. -| tier | interval used | window | proof age in health | verdict | +## 3. The runs — all four SCHEDULED, none triggered + +| box | tier | due at | archive | result | |---|---|---|---|---| -| daily host | 26 h declared | 4×26 h → **floor 168 h** | ≤ ~54 h | OK — numerically unchanged from before | -| weekly offsite | 8 d declared (7 d observed loses to it) | 4×8 d → **cap 288 h** | ≤ ~198 h | OK, 3.75 d of margin | -| newborn offsite | 8 d declared | **288 h** | — | UNKNOWN until the anchor passes | +| demo-felhom | **host** | 00:55:21 | `…2026_08_02-04_42_14.tar.zst` | **passed, 83.8 s**, scratch torn down 00:56:45 | +| demo-felhom | offsite | 06:55:21 | `…2026-07-28T04:49:43Z` | **passed, 540.4 s**, torn down 07:04:21 | +| demo-hp | **host** | 02:05:39 | `…2026_08_02-04_49_29.tar.zst` | **passed, 109.3 s**, torn down 02:07:28 | +| demo-hp | offsite | 08:05:39 | `…2026-07-28T19:19:45Z` | **passed, 300.1 s**, torn down 08:10:39 | -## 4. v0.91.1 — a flaw v0.91.0 shipped with, found by checking the live box +**No box failed to fire, so Phase C was not entered and no `--selftest` was used as a proof.** The +only selftest invocations in this session were the read-only `restore-test-due` verdict prints in §2, +which start nothing. -demo-felhom's offsite tier holds two retained snapshots, `2026-07-27T19:55:41Z` and -`2026-07-28T04:49:43Z` — **8 h 54 m apart**, because one is a healing artefact and the other a real -weekly run. A mean-gap estimate therefore reads a **weekly** tier as nine-hourly: ×4 = 36 h, the floor -lifts it to 168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm -this whole task exists to prevent would have returned within a week, on the box it had just shipped -to.** +**§4's question, answered live:** each box took its **host** tier first — never-proven sorts ahead of +proven, and ahead on the id tie-break — deferred the offsite one, and picked it up on the **following +evaluation six hours later**. One heavy operation at a time, per box, with nobody sequencing it. That +is R-86's oldest-proven ordering and the heavy-operation gate observed together for the first time. -Fixed by `max(observed, declared)`. A gap *shorter* than the declared rhythm is routine and means -nothing (a retry, a manual run, a heal, a catch-up after an outage); a gap *longer* than it is real -information. **Cost, stated rather than hidden:** a tier that truly runs faster than its declared -rhythm gets a wider window than it needs, i.e. a slower `restore_test_stale`. Right direction for a -signal meaning *unverified* — *broken now* is `restore_test_failed`, immediate and untouched. +**The asymmetry worth keeping:** a host-tier restore is **83–109 s**; an offsite one **300–540 s**. The +tier an ordinary recovery uses is also the cheapest to prove. -## 5. Kept, because it was earned +## 4. What the runs left behind -- **Absence is UNKNOWN** until an anchored window has passed (R-81's structure, unchanged). -- **The stale signal stays edge-triggered** (`staleStates` untouched). -- **`restore_test_failed` and `restore_test_stale` stay DISTINCT** — one says your recovery is broken, - the other that it is unverified, and the second is the one that quietly becomes the first. -- **Every reason string now states the window it was judged against** — R-100's corollary: when a - verdict changes what it counts from, the alarm text must change with it, or an operator reads - "limit 168h" under a tier judged at 288 h and dismisses a true alarm. -- **The window READ is unchanged in cost** (14 days) — enough to find proof inside the widest window - and to see two generations of a weekly tier. +**Persisted state — v3, naming the archive, the tier and what was verified:** -## 6. Tests and red-proofs +```json +demo-felhom felhom-backup → {archive …2026_08_02-04_42_14.tar.zst, tier local, verified boot+running, + proven_at 2026-08-03T22:56:45Z} + felhom-pbs → {archive …2026-07-28T04:49:43Z, tier pbs, proven_at 2026-08-04T05:04:21Z} +demo-hp felhom-backup → {archive …2026_08_02-04_49_29.tar.zst, tier local, proven_at 2026-08-04T00:07:28Z} + felhom-pbs → {archive …2026-07-28T19:19:45Z, tier pbs, proven_at 2026-08-04T06:10:39Z} +``` -| Test | Asserts | Mutation | Observed | -|---|---|---|---| -| `TestRestoreTest_HealthyWeeklyTierIsNeverStale` | a healthy weekly tier **with jitter** is never stale across 6 weeks | window pinned flat at `restoreProvenWindowFloor` | **FAIL** — `week 0: … proof age 172h0m0s, window 168h0m0s … "limit 168h0m0s"` | -| `TestRestoreProvenWindow_Contract` | floor, cap, declared fallback, **observation may only widen** | `observed > interval` → `observed > 0` | **FAIL** — `window(pbs, observed=8h54m0s ok=true) = 168h0m0s, want 288h0m0s` | -| `TestObservedArchiveIntervals_FromReports` | rhythm observed per tier; a PBS vzdump is **not** counted into the host tier | — | pass | -| `TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms` | a window that never fires is a deletion, not a fix | — | pass | -| existing R-85/R-81 suite | anchored UNKNOWN, edge-trigger, distinct events, newborn silence | — | pass, unchanged | +**The hub received the host-tier proofs — R-189's path carrying one for the first time.** demo-felhom's +latest report holds **two** entries, one per tier; the `local` one can only have come from the +persisted state, because the in-memory store held only that morning's offsite run: -**A hollow test caught by its own red-proof.** The first Scenario-G fixture used a perfectly regular -weekly tier and **PASSED under the flat-window mutation**, because a regular tier's proof age lands on -exactly 168 h and `>` lets it through. It would have shipped Part 1 and its false alarm together while -looking like proof. The jitter is what makes it a test — and it is also the truth about the old -constant. +``` +demo-felhom tier=local pass=True archive=felhom-backup:…2026_08_02-04_42_14.tar.zst tested_at=2026-08-03T22:56:45Z + tier=pbs pass=True archive=felhom-pbs:…2026-07-28T04:49:43Z tested_at=2026-08-04T05:04:21Z +demo-hp tier=local pass=True archive=felhom-backup:…2026_08_02-04_49_29.tar.zst tested_at=2026-08-04T00:07:28Z +``` -## 7. Deployment +**A subsequent evaluation runs nothing on a proven tier**, quoted from demo-felhom now: -| Step | Evidence | -|---|---| -| Images built + pushed | `felhom-hub:0.91.0`, then `0.91.1` | -| Manifest bumped in git | `687fedd`, `ff2655c` — never `kubectl set image` | -| ArgoCD deliberate sync | `sync=Synced health=Healthy` | -| Running image | `gitea.dooplex.hu/admin/felhom-hub:0.91.1` | -| Startup log | checkers initialized; `Listening on :8080` | +``` +tier=felhom-pbs due=false proven="…2026-07-28T04:49:43Z" + reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven +``` -## 8. Part 3 — the ep0 ruling, recorded +…while its **host** tier reads `due=true` again — on the **08-03** archive, which has now settled 24 h. +That is not a defect: it is "proved daily, on its own archive", visible one day later. -`runbooks/target-selection.md` had carried *"D-d did not name ep0 either way. Confirm it explicitly"* -for two days. **Operator ruling, 2026-08-03: ep0 is protected.** Recorded three ways — the tier table -names it, the standing question is gone, and its per-machine section is retitled **Tier 2, PROTECTED**. -`CONTEXT.md` S-18 records that this **extends D-d's protected list to three machines** (DooPlex, -Peti's cluster, ep0) and that it is a *classification*, not new prohibitions: destroying datastores, -prune jobs, tunnel config or nftables rules was already forbidden by what it would destroy, and **the -ordinary off-site read a restore-test performs remains permitted** — which this session then exercised. +## 5. Teardown — three layers, per box -## 9. Registers and docs +| layer | demo-felhom | demo-hp | +|---|---|---| +| the machine | `pct list` → **0** entries for 990000 | **0** | +| the host | `lvs` → **0** volumes for 990000; `local-lvm` **1.95 % → 1.95 %** | **0**; `local-lvm` 30.79 % → 40.86 % during the offsite run → **30.83 %** after | +| the hub | the `restore_tests[]` entries are **RETAINED DELIBERATELY** — they *are* the proof the staleness check reads, so deleting them would delete the result | same | -- **R-86 → CLOSED**, shipped + proven live, with the trap in its own wording recorded on the row. -- **R-87 → re-ranked UP**: R-86 built most of what it waited for. What remains is restic-specific. -- **R-185 / R-186 / R-187 → filed** (grep established 185–187 free; R-184 was the highest in use). -- `ROADMAP.md` R-86 collapsed, keeping the reasoning and **correcting the shape the row itself - proposed** — which was the never-fires version. -- `STATUS.md` rewritten for the operator, trimmed back to one screen. -- `07-backup-architecture.md` §3 carries the new contract (S-1); `00-capability-map.md`'s restore-test - row upgraded with its live citation; `CONTEXT.md` gains S-17 (the rule + the trap + the config key) - and S-18 (ep0). +Nothing else was created: no scratch customer, no fixture storage, no probe tag, no package version. + +## 6. What the run surfaced — three findings, two of them corrections to my own record + +### R-190 (new) — a storage ACL that worked in the morning was gone by mid-morning + +A `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50** on +2026-08-03; the first `403 … missing privilege Datastore.Allocate` on that path is **09:24:56**, and by +~14:50 `pveum acl list` held **no row at all** for it. Ruled out by measurement: a host reinstall +(uptime 12 days), any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00, any cluster-log ACL entry. +Correlated but not established: guest 9201 was reprovisioned at 09:15–09:19, nine minutes before. +**A permission that can vanish silently makes every ACL-based guarantee on these hosts provisional**, +and v0.123.0's probe detects the *state* but says nothing about the *transition*. + +### R-191 (new) — every weekly offsite backup reports FAILED although it worked + +demo-felhom, 06:49–06:53 today: the upload **succeeded** (223 s, 629 MiB of 1.874 GiB, 67.2 % reused +incrementally) and the job then failed on the prune — +`missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `TASK ERROR: job +errors`, and the hub raised `whole_guest_backup_failed`. **The token behaves exactly as R-89 designed** +(box tokens are write-only; ep0 prunes). What did not follow is the config: **both** boxes still arm +the offsite tier with `keep_last=2 prune_pbs_allowed=true`, so every weekly run asks for a prune that +must fail. The data is safe; the verdict and the weekly e-mail are wrong, which is the R-100 corollary +— an alarm whose text is true and whose trigger is not the thing you would act on. **Not fixed here** +(§6 rule 2), and the fix needs one check first: whether ep0's prune jobs actually cover these two +namespaces. + +### Two corrections to yesterday's record + +1. **The R-185 drift was NOT silent on the write path.** demo-felhom's local-api backup jobs 403'd + **six times** (09:24 → 17:34) on that storage and privilege, and the hub raised + `whole_guest_backup_failed` at the first with edge-triggering suppressing the rest. My annotation + said backups kept landing because writes go through a root path — wrong, and now corrected in the + runbook and on the row. +2. **My "no `restore_test_*` events at all" was an instrument error.** The hub has no `/events` route; + I grepped a **404 page**. Read from the events table: five such events exist (2026-07-27/28), none + since the R-86 work — and one **more** since, below. + +**One further event, correctly raised and worth stating:** `restore_test_stale` for demo-felhom at +2026-08-03 22:33:42 UTC — **22 minutes before** the host-tier run. It was **true**: at that moment the +host tier had never been proven, and the hub said so on its own, without being asked. It has not +re-fired; the signal is edge-triggered, so the return to healthy is silent by design. + +## 7. The capability map + +The unattended restore-proof row now reads **PROVEN-LIVE for the host tier, unattended, on both demo +boxes**, with the four runs, their durations, the deferred-tier ordering, the hub-side proof and the +teardown cited — and it states its **scope explicitly**: `demo-felhom` and `demo-hp`. The tester's box +is untested and untouched, and one box proving something does not make it a fleet property. + +## 8. Registers + +- **R-185** — its consequence is now demonstrated; the row already closed yesterday, and carries the + correction in §6. +- **R-190**, **R-191** — filed. `grep` established R-190 and R-191 were free before minting (R-189 was + the highest in use). +- `ROADMAP.md` holds none of these rows, so nothing to collapse. +- `STATUS.md` rewritten for the operator and kept to one screen (85 lines); R-191 appears under + "What's broken" because it produces a weekly e-mail you would otherwise learn to ignore. + +## 9. Observations — noticed, NOT acted on + +- **demo-hp's `local-lvm` thin pool reached 40.86 %** during its offsite restore (from 30.79 %, + returning to 30.83 %). Comfortable, but that is the over-subscribed pool the target-selection notes + warn about, and the offsite archive is the larger of the two. A materially bigger guest would want + the restore pointed at `/mnt/nvme-1tb` instead. +- **demo-hp has an offsite tier**, contrary to the runbook's §2 premise and to the note that it "has + none". Nothing depends on that assumption now, but the operations notes still carry it. +- **The two boxes' daily archives are ~6.3 GB (demo-felhom) vs ~2.35 GB (demo-hp)** for the same guest + role — a 2.7× difference worth understanding before either is used to size anything. +- **Both waiters this session produced no output** despite the runs completing; the evidence was + gathered by direct query afterwards. A watcher that silently produces nothing is exactly the + instrument class this project distrusts — the conclusions here rest on the boxes' own journals and + the hub's database, not on the waiters. diff --git a/STATUS.md b/STATUS.md index c1e01e9..71e81ab 100644 --- a/STATUS.md +++ b/STATUS.md @@ -20,12 +20,23 @@ Proven end to end on real hardware. ## What's broken +- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then + trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and + you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one + setting per machine fixes it. *(R-191)* - **The off-site copy can be erased by the machine that made it.** The credential that writes it can - also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy. + also delete it. A daily snapshot is armed as a stopgap. *(R-95, R-87)* ## What shipped recently +- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both + demo machines restored their own on-machine backup into a throwaway machine overnight, booted it, + checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we + had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had + never been tested on either machine. Both also proved their off-site copy on the same night, one + after the other rather than at once, which is the machine deciding for itself what to do first. + *(closes the last open half of R-86/R-185)* - **A backup copy the machine was never allowed to read — and could not tell you about.** One demo machine kept its whole-machine backups on a dedicated storage area the agent had never been granted permission to read. Asked what was there it was told "nothing", while an administrator saw three @@ -45,9 +56,6 @@ Proven end to end on real hardware. published after the binary, and a new check catches the opposite mistake so nothing is traded away. **(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)* -- **Each backup is now proved, instead of the clock being obeyed** — tested once, about a day after - it is made, and not again until there is a newer one; the "not proved lately" alert learned each - copy's own rhythm in the same change. *(R-86)* ## What we're working on @@ -69,12 +77,9 @@ Proven end to end on real hardware. ## Changed since last update -- **2026-08-03** — Found and fixed a backup copy the machine was never permitted to read, on both demo - machines. The permission was one line; what mattered was that the machine now says so instead of - treating "I am not allowed" and "there is nothing here yet" as the same answer. *(R-185)* +- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own, + overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the + weekly off-site backup reports failure after a successful upload. *(R-185, R-191)* - **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup that could vanish on a restart (seen happening), a release that emailed a failure for a release that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)* -- **2026-08-03** — Backups are now proved one at a time, each about a day after it is made, instead of - on a timer; the "not proved lately" alert learned each copy's own rhythm. You settled that the - off-site endpoint is protected. *(R-86)* diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 7437711..eed52b3 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -39,7 +39,7 @@ | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test | | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). | | **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** | -| **Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard** | agent v0.104.0 → **v0.121.0**, hub v0.77.0 → **v0.91.0** | **PROVEN-LIVE (2026-08-03)** | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why this was NOT PROVEN-LIVE until now:** rotation had not been observed selecting both tiers across consecutive UNATTENDED cadences — at a 24h cadence a multi-day window — and a single passing run proves the code path, not the schedule. → **R-85**. **R-86 (agent v0.121.0 + hub v0.91.0, 2026-08-03) replaced the schedule and the observation became possible in one afternoon**, because what has to be observed is no longer a multi-day rotation but a RULE: a tier is due when its newest archive that has settled ~24 h has not been proven. **UPGRADE EVIDENCE — a real unattended run on demo-felhom (2026-08-03), triggered by DUE-NESS, not by a timer.** The scheduler's own log: `15:14:38 restore-test tier is DUE … target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z … reason="newest settled archive … has not been proven"` → `proxmox-backup-client restore --crypt-mode=encrypt` under the agent's own token → `15:25:08 gate decision class=guest_destroy guest=990000 allowed=true` → `15:25:14 scratch guest torn down` → **`15:25:14 backup: scheduled restore-test passed archive=felhom-pbs:… duration_s=635.1`**. A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored, booted, verified and destroyed in 635 s, unattended. **The three things a timer could not show**, all verified after it: the state names THAT archive (`{"felhom-pbs":{"archive":"…2026-07-28T04:49:43Z","proven_at":"2026-08-03T13:25:14Z"}}`); a second evaluation reports `due=false … is already proven` and runs nothing; and **an agent restart runs nothing**, which is the defect a person actually noticed — every deploy used to restart the timer. Teardown verified at all three layers: guest absent from `pct list`, **zero** `990000` volumes in `lvs`, and the hub-side `restore_tests[]` entry deliberately RETAINED (it IS the proof the staleness check reads). **R-189 (agent v0.122.0, 2026-08-03) closed the reporting half of this row, and it was a REAL gap in the evidence path:** the proof above reached the hub only because no restart intervened — `restore_tests[]` came solely from an in-memory store, so the 15:25:14 PASS was in fact LOST when the agent restarted 2 m 43 s later for a deploy (`0 restore-tests` on the next two host-reports). Under per-archive due-ness the box would not have repeated the work for a week. The persisted per-tier proof (with the archive, and now the tier) is merged into the report, so a proof survives a restart — one entry per tier, newest wins, and a record that cannot be described honestly is not emitted. **THE HOST TIER'S HALF OF THIS ROW WAS OPTIMISTIC UNTIL 2026-08-03, and it is worth saying plainly.** Every live restore-test cited here is on the OFFSITE tier. The HOST tier was not merely unproven on demo-felhom — it was **unprovable**, and on demo-hp equally: the agent's token had no ACL on `/storage/felhom-backup`, the storage both boxes configure as `local_backup_target`, so the content API answered `{"data":[]}` through the token while root listed three archives. The scheduler skipped it as *"no settled archive yet"* — which is exactly what a brand-new tier reports — so nothing ever said so (**R-185**). **CLOSED 2026-08-03 — agent v0.123.0 + installer 1.24.0.** The grant is applied on both demo boxes (the token now lists 3 and 4 archives), the installer's reuse arm grants on a pre-existing target, and the agent now **asks whether it may read each tier it depends on** instead of inferring it from an empty list: a critical degraded capability naming the storage and the missing role, which the hub alerted and emailed on **while the box was still blind**. **The host tier became a due-check candidate for the first time on that box** immediately after the grant, correctly selecting the 08-02 archive because the 08-03 one had not settled 24 h. An unattended host-tier restore-test is now possible; this row will carry its first live host-tier proof when one runs. Evidence: `felhom-agent/REPORT.md` | +| **Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard** | agent v0.104.0 → **v0.121.0**, hub v0.77.0 → **v0.91.0** | **PROVEN-LIVE (2026-08-03)** | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why this was NOT PROVEN-LIVE until now:** rotation had not been observed selecting both tiers across consecutive UNATTENDED cadences — at a 24h cadence a multi-day window — and a single passing run proves the code path, not the schedule. → **R-85**. **R-86 (agent v0.121.0 + hub v0.91.0, 2026-08-03) replaced the schedule and the observation became possible in one afternoon**, because what has to be observed is no longer a multi-day rotation but a RULE: a tier is due when its newest archive that has settled ~24 h has not been proven. **UPGRADE EVIDENCE — a real unattended run on demo-felhom (2026-08-03), triggered by DUE-NESS, not by a timer.** The scheduler's own log: `15:14:38 restore-test tier is DUE … target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z … reason="newest settled archive … has not been proven"` → `proxmox-backup-client restore --crypt-mode=encrypt` under the agent's own token → `15:25:08 gate decision class=guest_destroy guest=990000 allowed=true` → `15:25:14 scratch guest torn down` → **`15:25:14 backup: scheduled restore-test passed archive=felhom-pbs:… duration_s=635.1`**. A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored, booted, verified and destroyed in 635 s, unattended. **The three things a timer could not show**, all verified after it: the state names THAT archive (`{"felhom-pbs":{"archive":"…2026-07-28T04:49:43Z","proven_at":"2026-08-03T13:25:14Z"}}`); a second evaluation reports `due=false … is already proven` and runs nothing; and **an agent restart runs nothing**, which is the defect a person actually noticed — every deploy used to restart the timer. Teardown verified at all three layers: guest absent from `pct list`, **zero** `990000` volumes in `lvs`, and the hub-side `restore_tests[]` entry deliberately RETAINED (it IS the proof the staleness check reads). **R-189 (agent v0.122.0, 2026-08-03) closed the reporting half of this row, and it was a REAL gap in the evidence path:** the proof above reached the hub only because no restart intervened — `restore_tests[]` came solely from an in-memory store, so the 15:25:14 PASS was in fact LOST when the agent restarted 2 m 43 s later for a deploy (`0 restore-tests` on the next two host-reports). Under per-archive due-ness the box would not have repeated the work for a week. The persisted per-tier proof (with the archive, and now the tier) is merged into the report, so a proof survives a restart — one entry per tier, newest wins, and a record that cannot be described honestly is not emitted. **THE HOST TIER'S HALF OF THIS ROW WAS OPTIMISTIC UNTIL 2026-08-03, and it is worth saying plainly.** Every live restore-test cited here is on the OFFSITE tier. The HOST tier was not merely unproven on demo-felhom — it was **unprovable**, and on demo-hp equally: the agent's token had no ACL on `/storage/felhom-backup`, the storage both boxes configure as `local_backup_target`, so the content API answered `{"data":[]}` through the token while root listed three archives. The scheduler skipped it as *"no settled archive yet"* — which is exactly what a brand-new tier reports — so nothing ever said so (**R-185**). **CLOSED 2026-08-03 — agent v0.123.0 + installer 1.24.0.** The grant is applied on both demo boxes (the token now lists 3 and 4 archives), the installer's reuse arm grants on a pre-existing target, and the agent now **asks whether it may read each tier it depends on** instead of inferring it from an empty list: a critical degraded capability naming the storage and the missing role, which the hub alerted and emailed on **while the box was still blind**. **THE HOST TIER IS NOW PROVEN-LIVE, UNATTENDED, ON BOTH DEMO BOXES (2026-08-04) — this row's optimistic half is cashed.** Four SCHEDULED runs, nothing triggered by hand: **demo-felhom** host tier `…2026_08_02-04_42_14.tar.zst` **passed in 83.8 s** at 00:55, offsite `…2026-07-28T04:49:43Z` **passed in 540.4 s** at 06:55; **demo-hp** host tier `…2026_08_02-04_49_29.tar.zst` **passed in 109.3 s** at 02:05, offsite `…2026-07-28T19:19:45Z` **passed in 300.1 s** at 08:05. Each restored into a scratch guest, booted, verified and destroyed itself; `pct list` and `lvs` show zero `990000` afterwards on both, and both boxes' `local-lvm` returned to their pre-run figures (1.95 % and 30.83 %). **Both boxes had BOTH tiers due at once and the ordering was observed live for the first time under R-86's rule:** never-proven sorted first, so each box took its HOST tier, deferred the offsite one, and picked it up on the following evaluation six hours later — one heavy operation at a time, per box, without anyone sequencing it. **The proofs reached the hub**, which is R-189 carrying a host-tier entry for the first time: demo-felhom's report holds **two** entries, one per tier, and the `local` one can only have come from the persisted state because the in-memory store held only that morning's offsite run. **SCOPE, stated because one box proving something does not make it a fleet property:** this covers `demo-felhom` and `demo-hp`. The tester's box is untested and untouched. Evidence: `felhom-agent/REPORT.md`, `felhom.eu/REPORT.md` | | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below | | **Customer DELETE cascade** (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **UNIT-PROVEN; live leg PENDING** | `hub/internal/web/customer_delete_test.go` — leg ORDER observed from inside leg 2 (hosts already gone, customer row still present, custody still retained); 9 fail-closed gate cases each asserting zero mutations + zero external calls + no journal row; resume-after-external-failure converges; `purgeEscrow` custody semantics; preview leaks no secret. **5 red-proofs** (ack gate, stale-preview gate, ONLINE-host gate, leg order inverted, `purgeEscrow=true`) | Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **v0.70.0 (2026-07-21):** added the **residue** leg — `GetCustomers()` is REPORT-derived, so before it a fully deleted customer stayed on the Customers list and its report stream kept the staleness/offsite checkers alerting (live: `demo-vm-felhom` deleted 07-18, still emailing `offsite_stale` on 07-21). The leg also purges the credential-bearing `appliance_registrations` + `selfbind_tokens`. **Ghost customers (config row already gone) are now deletable** — 404 means "nothing here", not "no config row"; the Hetzner/descriptor legs record `skipped_no_config`. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run | | Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 43ed23a..81be0b9 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -34,6 +34,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-188** | **Every agent release has a ~50 % chance of emailing the operator a CI failure for a release that is correct.** `on: [push]` fires the gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag BEFORE publishing the package (deliberately — a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it). So CI can run the published-versions gate inside the seconds-long window where `v` exists and the package does not, and it correctly reports *"every released agent version must be INSTALLABLE"* — for a release that completes normally a moment later | **CLOSED — SHIPPED 2026-08-03** (agent **v0.122.0**) | — | **MEASURED, 2026-08-03, across two releases in one session:** v0.121.0 → runs #12 (id 79) **success** and #13 (id 81) **failure** on the same sha `4618169`; v0.121.1 → #17 (id 88) **failure** and #18 (id 89) **success** on the same sha `cd1b087`; and v0.121.0's second pair (#14/#15, `4d82591`) both **success** — i.e. it is a race, not a rule. Local gate afterwards: all three versions verify. **Why it matters more than it looks:** R-168 made CI email the operator on failure precisely so a red gate cannot be missed. A signal that cries wolf on every second correct release is how that mail becomes something you archive unread — the R-29 failure mode one layer up, and R-100's corollary in a new place (an alarm whose text is true but whose trigger is not the thing you would act on). **Fix shapes, cheapest first:** have the workflow skip tag refs (`if: !startsWith(github.ref, 'refs/tags/')` — the commit push already covers the content, and the tag adds no new code); or make `check-published-versions.py` tolerate a tag younger than ~2 minutes; or publish before tagging, which the release script's own comment argues against. **Do not "fix" it by quieting the alarm** **FIXED by moving ONLY the tag push** (build → tag locally → publish → push tag). The tag is still created before anything is published, so the build and the tag still describe the same commit; it becomes VISIBLE — to CI, and to any `raw/tag/…` fetch — only once the package is downloadable. **The invariant the old order protected was not traded away:** `check-published-versions.py` now asserts the converse directly — *no published version may be missing its tag* — as a BOUNDED probe of the frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set on every run because a check whose coverage is invisible reads as a guarantee it is not making. The package listing api still answers **401** without a token (re-measured 2026-08-03), so absence still cannot be enumerated and the script says so. **Both directions demonstrated:** a real published-but-untagged fixture (0.121.2, published then deleted — HTTP 201 then 204, absence re-verified by a 404) made the gate FAIL and print the one-line recovery; removing the converse block made it pass over that same orphan. A tag with no package still fails, shown against a local stand-in rather than by pushing a probe tag — pushing one would have emailed the operator a TRUE alarm about a fixture, which is the same attention cost this row removes. **A half-done release is now loud:** publish-succeeded-then-push-failed dies naming the exact recovery command, and a FAILED publish removes the local-only tag so the release can simply be retried instead of colliding with step 2's re-release guard. **Scenario E, live:** v0.122.0's own release produced CI runs **21 (task id 96)** and **22 (task id 97)** on the same sha — **both green**, where the same shape produced one red run on each of the two previous releases. | CC | | **R-189** | **A passing restore-test can be invisible to the hub forever — and R-86 made that window a week instead of a day.** The agent reports `restore_tests[]` from `internal/backup.Store`, which is **in-memory** (its own comment: *"lost on restart; the cadence re-populates"*), and the hub's report interval is 900 s. An agent restart inside that window drops the result. That used to self-heal within 24 h because the next cadence re-tested the same tier; **under per-archive due-ness the agent will NOT re-test a proven archive**, so the hub can stay ignorant of a proof until the NEXT archive generation — a week on the offsite tier. **Observed live, 2026-08-03:** the 15:25:14 PASS on `felhom-pbs` (a real 14.5 GB offsite restore) reached **no** host-report — the hub logged `0 restore-tests` at 15:28:00 and 15:32:14 — because the agent was restarted at 15:27:57 for the v0.121.1 deploy, 2 m 43 s after the run and before the next report | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.122.0**) | — | **The persisted proof already exists and is not reported.** `RestoreTestState` (on disk, per tier, R-86: with the archive id) survives exactly this, and `RestoreTestState.Snapshot()` carries the doc comment *"for the host-report gauge"* — **and has no production caller at all** (grep, 2026-08-03). That is this project's own *seam built but never wired* class, fifth instance, and a comment asserting an invariant with no test pinning it, both in one method. **Severity is bounded, and worth stating so it is not over-ranked:** the hub scans its retained window, archives keep arriving, and R-86's per-tier window for the offsite tier is 12 days against a 7-day archive rhythm — so ONE lost report is tolerated. What is not tolerable is that the loss is **silent** and that the box will not repeat the work. **Fix shape:** report the persisted per-tier proof (target, archive, proven_at) alongside the in-memory latest run, and let the hub's `lastProvenPerTier` read it — a report-contract change, so it touches `internal/hub/report.go` + `contract_test.go` + the hub's checker, and it is a spec, not a one-liner **FIXED.** `RestoreTestState` now stores the **tier** and what was **verified** beside the archive (recorded at proof time, from the run's own result — deriving them later would need a storage lookup that can fail on the one path where failing means mislabelling a proof), exposes `ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it with the in-memory result. **Merge rule: one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news and lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (two entries would read at the hub as two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **It refuses to lie:** a record missing the archive OR the tier produces NO entry, and run mechanics (scratch VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be. **The asymmetry is now written where it will be read:** only successes are persisted, because a success SUPPRESSES future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it believes) while a failure CAUSES future work and heals itself at the next evaluation. **`Store`'s comment was corrected in place** — *"lost on restart; the cadence re-populates"* was true under a timer and became false the moment R-86 shipped; leaving it is how the next reader concludes this is handled. **The seam that had no caller is now pinned by an AST test** (`TestMainWiresTheDurableRestoreTestProof`), because `Snapshot()` claimed a *"host-report gauge"* in its doc comment for weeks with nothing calling it. **MIGRATION, stated because it is visible on the live box:** a pre-R-189 record carries an archive but no tier and is therefore NOT reportable — upgrading does not retroactively make an old proof visible to the hub; the tier's next real proof fills it in. Observed on demo-felhom immediately after the upgrade: still `0 restore-tests` until a fresh run. | CC | | **R-190** | **A storage ACL that demonstrably WORKED in the morning was gone by mid-morning, and nothing recorded its removal.** On demo-felhom, a `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50 CEST 2026-08-03** (task log read in full). From **09:24:56** the same path returned `HTTP 403 … missing privilege Datastore.Allocate at /storage/felhom-backup`, six times through the day, until the grant was re-applied by hand at 18:54. By ~14:50 `pveum acl list` showed **no row at all** for that path | **OPEN** | — | **Why this is not just R-185 restated:** R-185's mechanism (the installer's Scenario-F arm resolves a pre-existing target without granting) explains a box that NEVER had the grant. This box HAD it and lost it, inside five hours, with the machine up throughout. **Ruled out, each by measurement:** a host reinstall (`uptime` = 12 days); any `pveum`/ACL/`user.cfg` activity in syslog between 04:00 and 10:00 (none); any ACL entry in `/cluster/log` (none). **Correlated, not established:** `host_leaf_changed` at 09:15 and `controller_started` at 09:19 — guest 9201 was reprovisioned nine minutes before the first 403. PVE removes ACLs at `/vms/` when a guest is destroyed (`AccessControl::remove_vm_access`, the F-LEAK mechanism); whether any path can take a `/storage/` row with it has NOT been established and is the first thing to check. **Why it matters more than the grant did:** a permission that can vanish silently makes every ACL-based guarantee on these hosts provisional, and the agent's new store-grant probe (v0.123.0) now detects the STATE but says nothing about the TRANSITION. **Worth pairing with:** whether the probe should report a grant it once had and no longer has as a distinct, louder signal than one it never had | CC | +| **R-191** | **Every weekly offsite backup UPLOADS successfully and then FAILS the job on a prune the box is deliberately not allowed to do — on both demo boxes.** Measured on demo-felhom 2026-08-04 06:49–06:53: the upload completed (223 s, 629 MiB of 1.874 GiB, **67.2 % reused incrementally**), then `ERROR: prune 'ct/9201': proxmox-backup-client failed: Error: permission check failed - missing Datastore.Modify\|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `ERROR: Backup of VM 9201 failed - error pruning backups` → `TASK ERROR: job errors`. The hub raised `whole_guest_backup_failed` | **OPEN** | — | **This is R-89's rule not reaching the config.** R-89 moved PBS pruning SERVER-SIDE — *"boxes set `keep_last: 0`, ep0 runs prune jobs; box tokens stay write-only, never widen the grant"*. The token behaves exactly as designed: it refuses. But **both** demo boxes still arm the offsite tier with `keep_last=2 prune_pbs_allowed=true` (`backup_targets: [{target_id: felhom-pbs, cadence_seconds: 604800, keep_last: 2}]`), so every run asks for a prune that must fail. **The data is SAFE and that is why this is not a P1:** the snapshot lands before the prune is attempted; what is wrong is the job's VERDICT and the weekly operator e-mail it produces. **But it is corrosive in the specific way this project keeps finding:** a backup that reports FAILED while succeeding trains the operator to discount `whole_guest_backup_failed`, which is the same alert that would carry a real one — and it is exactly the failure the R-100 corollary warns about, an alarm whose text is true and whose trigger is not the thing you would act on. **Fix is one config line per box** (`keep_last: 0` on the PBS tier) plus whatever writes it on a fresh install; **deliberately NOT applied in this session** — the session was a runbook with an explicit "change nothing, and if a change appears necessary, stop and report" rule, and a retention field on a live backup tier is not a change to slip into an observation run. **Check before fixing:** whether ep0's prune jobs actually cover these two namespaces, or the snapshots simply accumulate once the box stops asking | CC | | — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC | | — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator | | **R-90** | ~~ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged~~ | **CLOSED — the operator rescaled ep0 to a CX33 on 2026-08-03** | — | **MEASURED ON THE BOX, not read from an invoice:** `felhom-hetzner` reports `Mem: 7757` MB total (**8 GB**, was 3.8) and `nproc` **4**. **The interim lever survived and was checked rather than assumed** — a resize is a stop/start, so "the swapfile is still there" was an assumption until measured: `/swapfile`, 4 GiB, dated `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`), 0 B in use on an idle box. **THE 40 GB LOCAL DISK DID NOT CHANGE** and must not be "corrected" alongside the RAM: `/` is 38 G, 58% used. This was a CPU/RAM resize only, so every disk figure in the runbooks still stands — the separate 98 G volume at `/mnt/pbs-datastore` (R-82 P0.3) is unaffected. **Why this was BLOCKED and no longer is:** the row recorded CX33 as *"confirmed unavailable even powered OFF"* — the Cost-Optimized line's limited availability, not a power-state problem. It became available and the operator took it. **Documentation corrected** (`RUNBOOK-ep0-datastore-volume`, `RUNBOOK-pbs-prune-serverside` ×2, `runbooks/offsite-endpoint.md` ×2, `runbooks/target-selection.md`) and **audit/evidence documents ANNOTATED, not revised** (`SPIKE-connectivity-wireguard-2026-07-03`, campaign-10 `phaseA-journal`) — they record what was true when written and that is their value. **Still open and still the operator's, deliberately untouched:** `target-selection.md`'s *"D-d did not name ep0 either way. Confirm it explicitly."* | — |