diff --git a/REPORT.md b/REPORT.md index db42210..187cbe7 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,194 +1,245 @@ -# REPORT — R-82: the backup target split (2026-07-26) +# REPORT — R-80 → R-85: from a false alarm to an audible DR tier (2026-07-26 → 27) -Local **daily** + offsite **weekly**, made expressible at all. Spans four artifacts: -agent **v0.97.0 → v0.103.0**, controller **v0.174.0 → v0.175.0**, hub **v0.76.0**, -host-install **1.20.0**. +One long session. It began as a diagnostic into a nightly alarm and ended with the offsite DR tier +being **restore-proven unattended, with its failure audible**. Five roadmap items, four artifacts. -Phase-0 gates: `documentation/audits/SPIKE-r82-phase0-2026-07-26.md`. +| Repo | From → To | +|---|---| +| felhom-agent | v0.96.0 → **v0.104.0** | +| felhom-controller | v0.173.0 → **v0.175.0** | +| felhom.eu hub | v0.74.0 → **v0.77.0** | +| host-install | 1.19.0 → **1.20.0** | + +All live on demo-felhom and demo-hp. --- -## 1. What was wrong +## 1. The arc, in one line each -`BackupTarget()` returned ONE string and `BackupCadence()` ONE 24 h window, so "local daily **and** -PBS weekly" could not be said. The consequence was not theoretical: the DR tier reported `applied` -since 2026-07-21 while demo-felhom held **one** snapshot (2026-07-18, a healing artifact) and demo-hp -held **zero, ever**. R-39 was "applied and dead"; this was **applied and empty** — the same shape, -quieter, and it would have surfaced first at a real restore. - -## 2. Phase 0 — three gates - -| Gate | Verdict | +| Item | What it was | |---|---| -| P0.1 what is exposed for 7 days | **weekly CONFIRMED.** The only 7-day-exposed state is the non-SMB half of `settings.json`. `encryption.key` and the offbox credentials are **stable files unchanged since first boot**, so a week-old copy is byte-identical — that was the risk that could have overturned it | -| P0.2 the `pvesm status` 0/0/0 anomaly | **RESOLVED, benign.** PBS returns HTTP 200 with zeroed usage to a namespace-scoped token (`DatastoreBackup`, not `Datastore.Audit`). Ground truth via the hub's ep0 `df`: the datastore is real and writable | -| P0.3 capacity | **STOP raised; operator ruled to proceed and grow later.** 37.2 GB total. Per-tenant encryption means **no cross-customer dedup** | +| **R-80** | Diagnose `expected_backup_missed` firing on three boxes. **The premise was wrong on both counts** — it fired ONCE, not nightly, and no real external customer was notified | +| **R-81** | Fix the CLASS behind it: absence of a signal treated as evidence of failure. Third instance | +| **R-82** | The backup target split — "local daily + PBS weekly" was **not expressible at all**, which is why the DR tier was `applied` and empty | +| **R-84** | An agent restart no longer triggers a redundant backup | +| **R-85** | The DR tier is restore-tested **unattended**, and its failure is **heard** | -**Capacity, now measured rather than bracketed:** the second weekly snapshot cost **+2.7 GB on disk** -against 14.46 GB logical (~81 % dedup). Weekly top-ups are cheap; **first** snapshots are not — one -customer at two retained snapshots costs ~13.5 GB, so the 80 % warn arrives at roughly the **first** -additional customer, not the second as I first estimated. Recorded in `07-backup-architecture.md` -§9.1. +--- -## 3. What shipped +## 2. R-80 — the diagnosis, and what it actually found -- **Agent** — `backup_targets[]`: each tier carries its **own** cadence, retention and wait bound - (`keep-last=3` is three DAYS on a daily tier and three WEEKS on a weekly one; one shared knob - guarantees one of them is wrong). `/backup/due?target=` judges a tier against **its own** newest - successful backup. `GET /backup/tiers` is the controller's capability probe. One runner per tier. -- **Controller** — every due tier collected up front and run in **ONE quiesce window**. Two cycles on - the weekly night would mean two app outages for one night's work. The app stays quiesced until the - **last** tier snapshots, so every tier is app-consistent. -- **Hub** — per-tier thresholds (host 26 h, offsite 8 d), preserving R-81's three-valued verdicts, - anchored absence and distinct reason strings. Classification is by **target type** - (`target_id` → `storage_targets[].name` → `.type`), never by array membership. -- **Installer** — a fresh box defaults to local-daily + offsite-weekly; an unprovisioned tier - **defers** rather than firing at a storage that does not exist. +The alarm fired at 2026-07-26 03:00 UTC on demo-felhom, demo-hp and drill-r50. The brief described it +as nightly, on three customers, one reaching a customer channel with a 7.3-day staleness claim. -**The untargeted local-API contract is frozen.** No `?target=` ⇒ the primary tier, same response -**bytes** (`Target` is `omitempty` and stays empty). An old controller cannot tell the new agent from -the old one; a new controller against an old agent degrades on a 404 probe, logs once, and **still -takes the backup**. +**It fired exactly once.** The check ran and was correctly silent on every prior night. The one +customer-channel mail went to the operator's own mailbox; `peti-felhom` — the only real external +customer — did not fire at all. And the 7.3 days was the age of the **PBS** snapshot, reached only as +a fallback: the local tier had run on 07-24, 07-25 and 07-26. -## 4. Operator rulings (2026-07-26), all implemented +**Root cause:** the agent's backup store is in-memory (`"lost on restart; the cadence re-populates"`). +The R-50 fleet restart at 12:44 UTC emptied `backups[]`; the next backup landed at 07:03 the following +morning; the 03:00 check fell in that ~18 h blind window and read empty as *no backup exists*. -| Ruling | Implementation | +**One real finding stood:** the PBS/offsite tier had **no schedule at all**. demo-felhom held one +snapshot from a healing artifact; demo-hp held **zero, ever**, while reporting `applied` since 07-21. + +--- + +## 3. R-81 — fixing the class, not the instance + +`assessBackupFreshness` collapsed *absence of records* into *failure*. It now returns +**OK / UNKNOWN / MISSED**: absence is UNKNOWN until it outlives a window anchored at first contact. + +The anchor was **free** — a Phase-0 probe found the hub already retains 90 days of host-reports, so +`GetHostReportsSince` + `newestBackupEvidence` answer *"when did I last SEE evidence of a backup?"* +rather than *"what does the latest report say?"*. No agent change, no new state. + +**Not silence.** A genuinely dead box must still alarm — that is the half the naive fix breaks, and +`TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms` is what makes the suppression safe. + +The invariant is now written at the head of the function naming all three instances (v0.12.0, +v0.73.0, R-81), pinned by a boundary test whose name says what it protects. + +--- + +## 4. R-82 — the target split + +`BackupTarget()` returned ONE string and `BackupCadence()` ONE 24 h window. + +**Phase 0 gates:** weekly CONFIRMED (the only 7-day-exposed state is the non-SMB half of +`settings.json`; `encryption.key` and the offbox credentials are **stable files unchanged since first +boot**, so a week-old copy is byte-identical). The `pvesm status` 0/0/0 anomaly resolved as a +namespace-scoped-token reporting artifact. Capacity raised a **STOP**, which the operator ruled past. + +**Measured, not bracketed:** the second weekly snapshot cost **+2.7 GB** against 14.46 GB logical +(~81 % dedup). Weekly top-ups are cheap; **first** snapshots are not. Per-tenant encryption means +**no cross-customer dedup**, so the 80 % warn arrives at roughly the first additional customer. + +**Shipped:** per-tier cadence/retention/wait-bound (agent); **one quiesce window for both tiers** +(controller — two cycles would mean two app outages for one night's work); per-tier thresholds +(hub, host 26 h / offsite 8 d); fresh-install default (installer). + +### Four defects found by RUNNING it, not reviewing it + +1. **v0.98.0** — a 41-minute backup recorded `success:false` at 30 minutes **while still running**, + then completed `TASK OK`. Worse than "didn't happen": the tier stays due and the retry hits the + guest lock. +2. **v0.100.0** — the restore tier read from the configured target, not the archive. **A silent + regression of the S4.1 fix** — the mechanism was never removed; its *input* changed when + `local_backup_target` was retargeted to `local`. +3. **v0.101.0** — a leaked scratch kept `onboot: 1`; a host reboot would have started a clone of the + live guest. +4. **v0.102.0** — a tier fired at a storage that does not exist yet. + +--- + +## 5. R-84 — ground truth instead of memory + +Three redundant local backups ran in one afternoon of deploys, because a restart empties the store. +On the offsite tier that is a wasted multi-hour upload after **every** agent deploy. + +Fixed by **asking the storage**, not persisting the record: a pruned archive correctly stops counting, +where a persisted record would keep claiming a backup that no longer exists. Proven live on both +boxes with the store cold. + +--- + +## 6. R-85 — unattended, and audible + +Three defects, each verified at source: the scheduler only ever saw `BackupTarget()`; the spec was +**frozen at daemon start** (an immediately-invoked function — also a latent staleness bug); and a +failed restore-test was a `[WARN]` line with **no event, no notification, no gauge** — true for the +local tier that *was* being tested. + +**Ruling (operator): oldest-first.** Never-proven sorts first, which is where the offsite tier starts. +Rotation credit **only on success**, or a permanently failing tier looks freshly proven and stops +being retried. State persisted — unlike R-84 there is no ground truth, because a restore-test destroys +its scratch and leaves no artifact. + +**`backup.InFlight`** — a host-wide one-heavy-operation gate. Not a lock concern (the scratch VMID +never touches the live guest's vzdump lock) but a **LINK** one: an offsite restore PULLS multi-GB over +the tunnel a backup PUSHES one. Callers **defer**; they never cancel. + +**Two hub signals, never merged:** `restore_test_failed` (broken now) and `restore_test_stale` +(*unverified*, not known-broken). Anchored on R-81, operator-tier only. + +--- + +## 7. Live evidence + +**The restore round-trip PASSED** (demo-hp, manual): +``` +source_tier: pbs pass: true verified: boot+running mount_parity: ok 4m5s, clean teardown +``` +`mount_parity` is the non-hollow half — a boot-only verify cannot see a missing data volume. + +**The multi-tier quiesce ran** through the real UI endpoint (authed + CSRF): +exactly **ONE stop/start pair with BOTH backups inside it**, app quiesced through the non-last tier, +early resume on the last tier's snapshot. **Downtime 1m27s for both tiers.** + +**Rotation proven unattended** (demo-hp): +``` +tier selected (oldest-proven first) target=felhom-pbs ← never-proven sorted first +scheduled restore-test passed duration_s=194.8 +tier selected (oldest-proven first) target=local ← rotated +``` + +**The signal fired on its first sweep — on real faults:** +``` +[ERROR] drill-r50 Restore-test FAILED on the local tier … HTTP 403 missing privilege VM.Backup +[WARN] demo-felhom pbs tier: NEVER restore-proven in 205h (limit 168h) — unverified, not known-broken +[INFO] demo-hp pbs tier: watching 133h of a 168h grace — newborn, not a fault +``` +The first two are genuine, previously-invisible failures. All notifications went to the **operator** +channel; the DB confirms no customer routing. + +**demo-hp's first ever offsite backup landed** — 4.25 GB into a verifiably empty namespace. + +--- + +## 8. Mistakes I made, and what they cost + +Recorded because the pattern matters more than any one of them. + +1. **I claimed the restore-test would break the control plane** by booting a network-conflicting + clone. **Wrong** — it link-downs every NIC before boot, and that is unit-tested. I read a config + artifact and inferred behaviour without reading the code that consumes it, *then escalated before + finishing the check*. I also disabled a safety mechanism on that basis; it is re-enabled. +2. **I pushed one commit red** — read `packages ok: 28` and missed `rc=1` in the same command. Five + of my own tests were failing. +3. **I planted a time bomb in Slice C** — a test hard-coded the incident timestamp + `2026-07-18T18:31:06Z` while comparing against the real clock. It passed all day and began failing + at exactly 18:31 UTC, 8 days later. A test that passes at commit time and fails hours later is + worse than one that fails immediately. +4. **I left `REPORT.md` claiming "not deployed"** 26 minutes before deploying, and did not update it. +5. **Two bad observables** while verifying the controller: read a UTC timestamp as host time, then + checked for log lines the agent does not emit at INFO. Both produced confident wrong readings. +6. **I told the operator blocking would not silence the new alerts.** Wrong — blocked customers are + excluded by `GetActiveCustomerIDs` already. +7. **My own spec's §5.3 was wrong** — it said to force a failure with `--selftest`, but the selftest + runs in a separate process and never reaches a host-report. + +The recurring shape: **inferring behaviour from an artifact instead of reading the code path that +consumes it**, and **reading a result without reading its exit code**. The host=CEST / +component=UTC mismatch caught me twice in one session. + +--- + +## 9. State at close + +| | | |---|---| -| Two weeks of offsite backups | `keep_last=2`; the blanket PBS-prune refusal scoped to *additional* tiers with an explicit setting — the primary keeps the absolute refusal, because its target **and** retention both default and could prune the DR by accident | -| Grow the datastore before any real tester | recorded in `07` §9.1; no action taken | -| First backup runs as long as needed; nothing else starts until done | wait bound → 12 h (measured ~5 h for a first full snapshot); **one backup at a time per guest** — a second tier gets a 409 naming the busy tier, with no job id it could mistake for its own; a tier overrunning the quiesce bound defers the rest | -| Drill box is temporary | dropped from the rollout | -| Restore test, then next slice | done — see §6 | +| Backups | local daily + offsite weekly, both boxes | +| Offsite retention | 2 weeks (`keep_last=2`) | +| Restore-test | **rotating both tiers, unattended**; interim cadence 3.5 d → each tier ~weekly | +| Signals | failure + staleness, operator-only | +| drill-r50 | VM 300 **stopped**, `onboot=0`. Operator will block the customer | -## 5. Four defects found by RUNNING it, not reviewing it +### ⏳ IN FLIGHT AT CLOSE — check on this -1. **30-minute wait bound vs a 41-minute backup** (v0.98.0). The agent recorded `success:false` - **while the vzdump was still running**, and it later completed `TASK OK`. Not "the backup didn't - happen" but worse: the tier stays permanently due and the retry collides with the guest lock. -2. **The restore tier read from the configured target, not the archive** (v0.100.0). A `felhom-pbs:` - archive was classified `local` and got the 10-minute bound against a 14.46 GB WAN restore, failing - at 600 s. **This was a silent regression of the S4.1 fix** — the mechanism was never removed, its - *input* changed when `local_backup_target` was retargeted to `local`. The lesson is not "add a - timeout" (one was already there) but that a fix keyed on *"the configured target"* stops holding - the moment more than one target exists. Recorded in `06-offsite-connectivity.md`. -3. **A leaked scratch guest kept `onboot: 1`** (v0.101.0) — a host reboot would have started a clone - of the live guest. Now `onboot=0` is set **at restore time**, because "after" is the path that - leaks. -4. **A tier fires at a not-yet-provisioned storage** (v0.102.0) — would have quiesced the apps and - failed every cadence on a fresh box until DR provisioning. +**demo-felhom's offsite restore-test is still running.** Its archive is 14.46 GB, so expect roughly +two hours. Scratch guest `990000` is on the box while it runs; a clean finish tears it down. -## 6. Live validation +Its agent still has the **compressed 10-minute cadence in memory** — the reverted 3.5-day value is on +disk and takes effect at the next restart. A watcher is set to restart the agent once the test +finishes. **If that watcher did not complete, restart `felhom-agent` on demo-felhom manually** or the +box keeps restore-testing every 10 minutes. -**demo-felhom** — the first real PBS-targeted backup: **`TASK OK`, 41 minutes, 14.46 GB snapshot**, -and it **restored cleanly** (`vzrestore: stopped OK`, all volumes back). Both tiers armed and -verified over the real local API; the untargeted response confirmed byte-identical. +When it passes, demo-felhom's `restore_test_stale` clears on its own. -**demo-hp** — reached via the documented break-glass path; binary and config backed up first. -Its **first ever** PBS backup **landed**: 4.25 GB into a namespace that was verifiably empty. +--- -**THE RESTORE ROUND-TRIP PASSED** — the bar for calling a tier real: +## 10. NOT done — explicitly -``` -source_archive : felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z -source_tier : pbs <- the v0.100.0 fix; the earlier attempt said "local" and died at 600s -pass : true -verified : boot+running -mount_parity : ok <- mp0=/var/lib/docker 50G, mp1=/mnt/sys_drive 20G, mp8/mp9 stand-ins -duration : 4m5s (restore + boot + verify + teardown) -``` - -`mount_parity` is the non-hollow half: a boot-only verify cannot see a missing data volume. The -scratch guest tore down **cleanly** — no 403, no leak — which confirms `06-offsite-connectivity.md`'s -reading that the teardown 403 was a **phantom** (a consequence of the short timeout, not an ACL gap), -and corrects my earlier framing of it as a standing privilege gap. Afterwards: scratch band empty, -thin pool back to its exact pre-restore figure, live guest running, snapshot intact. - -**THE MULTI-TIER QUIESCE RAN LIVE**, through the real UI endpoint (`POST /api/guest-backup/trigger`, -session auth + CSRF — the exact call the "Mentés most" button makes): - -``` -17:01:39 manual backup requested — quiescing now -17:01:39 backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx] <- ONE stop -17:01:46 tier local: backup job ... started -17:02:56 tier local: ... done — next tier may start (app still quiesced) <- app stays DOWN -17:02:56 tier felhom-pbs: backup job backup-9201-felhom-pbs-... started -17:03:06 tier felhom-pbs: ... snapshotted — resuming app early (8B.2) -17:03:06 unquiescing (snapshotted (early resume, last tier)): restarting 1 stack(s) <- ONE start -``` - -**Exactly one stop/start pair with both backups inside it** — the assertion that matters, since -"both backups ran" would also pass against an implementation that quiesces twice. Tier order was -local-first/PBS-last as designed, the app stayed quiesced through the non-last tier (app-consistency -preserved on the DR tier), and it resumed at the **last** tier's snapshot rather than waiting for the -upload. **Total app downtime 1m27s for both tiers**; paperless came back healthy. - -**Hub Slice C replayed against the live DB before deploying:** - -``` -demo-felhom host=07-26T14:38Z offsite=07-26T12:21Z -> OK -demo-hp host=07-26T07:06Z offsite=none -> UNKNOWN (119h of a 192h grace) -drill-r50 host=none offsite=not expected -> MISSED (no evidence in 29h) -``` - -**No customer email results from the deploy.** demo-hp defers correctly and will alarm in ~3 days if -its offsite tier stays empty — the true finding arriving on schedule, not a false alarm. - -## 7. A correction I had to make mid-arc - -I reported that the restore-test would boot a scratch guest carrying the live guest's MAC, static -island IP and hostname, and so would break the controller→agent link. **That was wrong.** -`RunRestoreTest` step 2 link-downs **every** interface before the guest is started, and it is -unit-tested. I read a restored config artifact, inferred the boot behaviour from it, and escalated -before reading the code path that consumes it. I also disabled the scheduled restore-test on that -basis, which was an unnecessary reduction in safety coverage; it is re-enabled. - -The residual hazard was real but far narrower — it needed the restore to fail *before* the link-down -step, which is what defect 1 caused — and that is what v0.101.0 fixes. - -Separately, `06-offsite-connectivity.md` records that the teardown `403` I flagged as a standing -privilege gap was **already diagnosed in S4.1 as a phantom**: it is a consequence of the short -timeout, not an ACL problem. With the timeout fixed the guest is pool-associated by teardown time. - -## 8. Tests - -| Repo | Result | -|---|---| -| felhom-agent | `build/vet/test` rc=0, **29 packages** | -| felhom-controller | `build/vet/test` rc=0, **27 packages** | -| felhom.eu (hub) | `build/vet/test` rc=0, **17 packages** | - -Red-proofs observed and restored for every mandatory scenario: old-controller compat, new-controller -degrade (the hollow version asserts "no error" while silently skipping the backup), the both-due -night (**the COUNT is the assertion** — asserting only "both ran" passes against a double-quiesce), -the merged threshold, the per-tier wait bound, the `onboot` override, and the overrun defer. - -**A process failure worth recording:** I ran the agent suite and committed in the same command, read -`packages ok: 28`, and pushed **without reading `rc=1`**. Five of my own Slice A tests were failing — -a harness artifact, not a product bug, but the commit went out red. Fixed in `13ca2d9`. This is the -exact exit-code trap recorded twice earlier in this arc. - -## 9. NOT done — explicitly - -1. **The offsite tier is never AUTOMATICALLY restore-tested.** The scheduled restore-test picks - candidates from a runner built on the primary target, so it can never select a PBS archive. The - **manual/selftest** path is now proven end-to-end; the **unattended** one is not. Needs a per-tick - spec. -2. **The hub infers "PBS ⇒ weekly" from storage TYPE.** `defaultBackupTarget` is `felhom-pbs`, so a - box that never sets `local_backup_target` would run PBS as its **daily** tier and be judged - against 8 days — seven days of blindness. No box is in that shape today. The real fix is the agent - reporting each tier's actual cadence. -3. **The installer-default fleet flip** (Slice D step 4) waits on a full weekly cycle holding — a - genuine gate, not an oversight. -5. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header; +1. **The restic (app-data offsite) tier is NEVER restore-tested** → **R-87**. R-85 covers whole-guest + vzdump tiers only. This is arguably the tier that matters most: the only one that survives losing + the box **and** carries the customer's app data (the whole-guest snapshot excludes the + bind-mounted drives). It is exactly the state PBS was in yesterday. +2. **Restore-tests are interval-scheduled, not backup-aligned** → **R-86** (operator ruling + 2026-07-27: ~1 day after that tier's own backup). +3. **Rotation observed on demo-hp only**, and under a compressed cadence. It proves the rotation + logic, not the production interval. +4. **The hub infers "PBS ⇒ weekly" from storage TYPE**, not a reported cadence. No box is in the + wrong shape today. +5. **The installer-default fleet flip** awaits a full weekly cycle. +6. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header; ratification is Viktor's review of the §10 list. +7. **The customer-facing Hungarian copy still overstates scope** — unchanged since R-80 flagged it. +8. **Capacity:** the datastore still needs growing before the first real customer. -## 10. Observations +--- -- The `felhom-pbs` PVE storage will permanently show **0 %** in the PVE UI (namespace-scoped token). - Operators must read fill from the hub's PBS-DR gauge. Worth a runbook line. -- demo-felhom's guest grew **9.74 → 14.46 GB logical in eight days**. Probably one-off from app +## 11. Observations + +- **drill-r50's 403** (`missing privilege VM.Backup at /vms/9100`) is real and was invisible. The box + is being retired, so it needs a decision — fix or accept — not silence. +- **A blocked customer is hidden from the Dashboard and from every checker**, and it is reversible + (`configs.go` sets status back to `active`, with a UI affordance). Blocking is a blunt instrument + though: it silences *everything*, which is why the operator is building per-alert silencing. +- **The PVE UI will permanently show `felhom-pbs` at 0 %** (namespace-scoped token). Read fill from + the hub's PBS-DR gauge. +- **demo-felhom's guest grew 9.74 → 14.46 GB logical in eight days.** Probably one-off from app testing, but if it is a rate the capacity sizing changes quickly. -- demo-felhom was enabled **before** demo-hp, out of the specified rollout order, because Slice A - could not be validated otherwise. -- The customer-facing Hungarian copy still overstates scope ("A mai biztonsági mentés nem készült el - a határidőig!" covers only the host/PBS tier). Unchanged; flagged since the R-80 diagnostic. +- **Three of the five items in this arc were the same bug class** — absence of a signal read as + evidence, or a correct mechanism whose input changed underneath it. Both are now written into the + code as invariants rather than left as lore. diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 36e0e03..6353dfe 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -114,6 +114,8 @@ | R-83 | **Ratify or retire `07-backup-architecture.md`** | S | **DISCHARGED (2026-07-26) — brought current, NOT ratified** | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ **STALE outside the PBS tier** verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names `felhom-pbs` → `felhom-offsite` on ep0 over `wg-felhom`, weekly, keep_last=2 — and §9.1 records the measured capacity. **Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's.** R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. | | R-84 | **The agent's cold backup `Store` no longer causes a redundant backup** | S | **SHIPPED (agent v0.103.0, 2026-07-26)** | Observed live: three redundant local backups on demo-felhom in one afternoon of deploys. The `Store` is in-memory ("lost on restart; the cadence re-populates"), so after every restart `/backup/due` said *no successful backup recorded yet* and the controller took another — on the OFFSITE tier a wasted multi-hour WAN upload after every agent deploy. **Resolved by asking the STORAGE rather than persisting the store** (`BackupRunner.NewestArchiveTime` + the optional `localapi.BackupArchiveLister`): ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists; no new on-disk state, no migration. It answers only *when did a backup last land* — the richer fields stay with real records, so the host-report never carries invented numbers. Fail-safes: read error → fall back to memory (never fake freshness, never suppress); genuinely empty → due; old archive → still due. **Proven live on both boxes** immediately after a restart with the store cold: demo-felhom `local age=5298s / felhom-pbs age=14373s`, demo-hp `local age=498s / felhom-pbs age=2346s`, all `due:false`. | | R-85 | **The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD** | M | **Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE** | R-82 proved a PBS restore round-trip **by hand**. Three defects kept the unattended path from covering it, each verified at source: **(a)** `buildRestoreTestScheduler` took `cfg.Backup.BackupTarget()` and built ONE runner on it, so the offsite tier's archives were never candidates; **(b)** the `Spec` was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; **(c)** a failed restore-test was a `[WARN]` log line with **no event, no notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. **Selection ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported **so the achieved interval can be MEASURED before anyone decides whether a knob is needed** — measure, then configure, not the reverse. **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS, never fails); persisted `RestoreTestState` (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); `backup.InFlight`, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub `restore_test_failed` / `restore_test_stale`, kept **distinct** and anchored on R-81's verdict structure, operator-tier only. **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). | +| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — operator ruling 2026-07-27** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. | +| R-87 | **The restic (app-data offsite) tier is NEVER restore-tested** | M | idea — surfaced 2026-07-27 while closing R-85 | **R-85 covers whole-guest vzdump tiers only** (`local`, `felhom-pbs`); the agent has no restic surface at all. restic is the CONTROLLER's app-data offsite backup to the Hetzner Storage Box, a separate mechanism — so the tier that is arguably most important to a customer is the one nothing verifies. It is the only tier that survives losing the box **and** carries their actual app data: the whole-guest snapshot deliberately excludes the bind-mounted data drives (`/mnt/felhom-drives`). Restore code exists and has been exercised BY HAND (the immich destroy-and-recover drill, PROVEN-LIVE), but nothing tests it unattended — **exactly the state PBS was in before R-85: it works when someone tries it, and nobody would know if it stopped.** Needs its own design: a restic restore-test is controller-side, has no scratch-guest analogue, and would verify into a scratch dir rather than a booted guest, so R-85's machinery does not transfer. | | R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog | | R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | **SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result.** The universal secret-free ISO was booted on a completely different machine — an **HP t740 (Ryzen V1756B, AMI M42 firmware)** versus the N100 it was proven on — and the whole chain ran on **virgin hardware in one pass**: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host `demo-hp-bb76ea`. **Two things generalise from the second board specifically:** the **shim** loader booted with **Secure Boot ENABLED** (`mokutil --sb-state` → `SecureBoot enabled`), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's **1TB NVMe untouched and unenrolled** (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. **Not clean, and the failures are filed:** the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (**R-59**), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (**R-61**) | R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** |