Files
felhom.eu/documentation/architecture/capability-map-detail/G.md
T
admin 0d463a859e
gates / gates (push) Successful in 6m12s
answers 2026-10-12 Parts D, F, G: controller 0.308.0 delivered, the capability map restructured, the notification design; 128 -> 97
- closed: R-922 R-255 R-266 R-333 R-577 R-554 R-562 (controller 0.308.0 on demo-hp, demo-felhom, Tester 1), R-288 (map 212 KB -> 48 KB, detail per section)
- R-521 narrowed to F7; R-388 design in 08 §8.1 (two questions for the operator)
- privacy draft: why the registered address stays (S1); 02 module map: setup wizard deleted
- STATUS + REPORT-answers-2026-10-12.md; docs.md routing line names capability-map-detail/

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-12 15:54:44 +02:00

27 KiB
Raw Blame History

Capability map — detail, section G. Fleet & operator (hub)

What this file is. The long text of every row of section G of 00-capability-map.md, moved here verbatim on 2026-10-12 (R-288, R-290 folded in; 09 §3 decisions 214 and 229) from the map at commit 1b45bb4d. Each entry is one former table row, its cells laid out one per paragraph; the cell text itself was not edited. The map row carries the capability's ONE status word and its dated evidence pointer; this file keeps the argument, the corrections and the history. Where this text and the map row disagree on the status, the map row is current (see the map header for the words that were normalised).

A new correction goes under the entry it corrects, dated — never into the map row.

G1 — Customer/host management: 8-tab detail, scoped auto-refresh, safe stale-host deletion, capability chips

Map row: G1 · former map line 225.

Scenario (as written): Customer/host management: 8-tab detail, scoped auto-refresh, safe stale-host deletion, capability chips

Components (as written): hub v0.47–0.53

Status (as written): PROVEN-LIVE

Evidence (as written): hub v0.53.0 dead-host roll-up live on the Peti cluster (proxmox1 down 23h); CAMPAIGN-4-2026-07-13 (operator UI driven live); DRILL-day0-take2 F-16 (offsite/freeze buttons live)

Gap / roadmap (as written): 8-tab render + capability chips are render-test-validated (hub UI is password-gated; CC cannot log in). (Cited "daily operator use" was a no-doc citation; AUDIT-hub-gui-2026-06-30 predates these features at hub v0.25)


G2 — Config/state change round-trips in seconds (hub↔box immediacy; 15-min cycle stays the backbone): box→hub out-of-cycle report (Dir 1) + hub→box GET /api/v1/wait long-poll wake (Dir 2)

Map row: G2 · former map line 226.

Scenario (as written): Config/state change round-trips in seconds (hub↔box immediacy; 15-min cycle stays the backbone): box→hub out-of-cycle report (Dir 1) + hub→box GET /api/v1/wait long-poll wake (Dir 2)

Components (as written): controller v0.139/140, hub v0.58/0.63

Status (as written): PROVEN-LIVE (2026-07-21)

Evidence (as written): Transport proven live through the real DNS-only ingress: SPIKE-immediate-sync-transport-2026-07-16 + hub v0.58.0 / controller v0.140.0 REPORTs — 240 s no-annotation hold (25 s heartbeat defeats nginx's 60 s proxy_read_timeout, no ingress change), 0.047 s wake-on-change, hub rollout restart = 1 WARN + 0-storm reconnect; Dir-1 2 s box→hub round-trip live in controller v0.139.0

Gap / roadmap (as written): The operator-UI save→apply round-trip is not fired end-to-end live (hub UI password-gated; CC can't log in) → R-23; the wake transport and the ACK→config_version→ConfigRefresher delivery chain are each proven, only the UI-triggered bump leg is unexercised. Agent-plane (host-domain desired-state) poke first slice PROVEN-LIVE (Direction-2a, agent v0.89.0 + hub v0.59.0, 2026-07-17): contentless ep0-relayed UDP poke → agent immediate desired-state cycle, per SPIKE-immediate-sync-transport-2026-07-16 P4. Full path live-proven: a real operator manifest save fired poke: sync-poke delivered to 10.77.0.2; the box (0.89.0) received it and logged poke received → triggering an immediate desired-state cycle → out-of-band report triggered — ~31 ms ep0→box, sub-ms to the report cycle (WG-confined, from 10.77.0.1 to the 10.77.0.2-bound socket); save→tick ≈ ~0.45 s (SSH-dominated), well under ≤2–3 s. R-13 first slice (listener+sender only; the rest of the mutual-repair arc stays open). System-initiated immediacy wired (hub v0.63.0, this REPORT): the mutation sites that only OPERATOR actions used to notify now fire the correct plane's notifier when the hub itself mints state — agent-plane pokes at PBSDRAutoProvision (the observed slice-C lag), ReissuePBSDR (also the pbsdrheal escalation), handlePBSDRReissue, and the two admin desired-state api writers; controller-plane bump at reissueOnReenroll. Unit-tested + red-proofed, not yet fired on a real system event (folds into the rehearsal bind sequence). Still PARTIAL: the R-23 operator-UI save→apply leg and the agent fast-tick-until-first-convergence SECONDARY (the WG-registration leg a poke can't reach pre-tunnel) remain unfired live. Fast-tick SHIPPED (agent v0.90.0, R-28): while any desired-state item is unapplied — incl. the pre-tunnel window a poke can't reach — the agent pulses the out-of-band trigger every 30 s and self-disarms on convergence (state-based; four cached sources; LOUD states excluded). LIVE on both demo agents (the fast-tick armed: 30s … startup line verified). REAL-ONBOARDING PROOF DONE — tests/VALIDATION-n100-rehearsal-2026-07-18.md (ledger 8, S5): on a genuine first onboarding on metal, every post-bind leg landed seconds apart with no ~15-minute stall anywhere — bind 16:29:55 → credential delivered 16:30:21 (26 s) → agent 0.90.0 up 16:30:49 → WG registered + tunnel applied 16:30:51 (~2 s) → poke listener 16:30:54 → controller 16:32:28 → floor-lifted and running current 16:32:39. Bind → running-current = 2 min 44 s. The PBS-DR descriptor auto-provisioned on the same cadence (agent converged state=applied 16:45:53) — though see the DR-tier row: the descriptor converged while the credential behind it was already stale (R-39). The pre-tunnel fast-tick window is therefore proven in its real setting; the remaining PARTIAL is the R-23 operator-UI save→apply leg alone 2026-07-21 — THE LAST PARTIAL LEG IS CLOSED (R-23(a) restart leg). The operator saved the global floor to a version the box did NOT run (0.153.0 → v0.154.0) and the managed self-update fired exactly once: 06:57:13Z UpdateState pending (initiated_by=auto-floor) → 06:57:17Z agent controller swap requested → 06:57:21Z container restarted → 06:57:29Z new controller healthy. Save → healthy on the new version = 16 s. Over a 39-minute window: swap requests 1, agent-driven bootstrap restarts 1, rollbacks 0, container RestartCount 0; VerifyStartup confirmed on the next boot and the following periodic check logged Current version 0.154.0 is up to date (the at/above-floor branch doing nothing, as designed). The 2026-07-20 attempt could not prove this because it targeted an already-running version. Evidence: felhom-controller/REPORT.md §6.


G3 — Customer right-sizes guest RAM from the controller (agent-enforced bounds, live cgroup apply, no reboot)

Map row: G3 · former map line 227.

Scenario (as written): Customer right-sizes guest RAM from the controller (agent-enforced bounds, live cgroup apply, no reboot)

Components (as written): agent v0.90.0 + controller v0.143.0 (R-24)

Status (as written): PROVEN-LIVE (grow and shrink on metal, 2026-07-18)

Evidence (as written): tests/VALIDATION-n100-rehearsal-2026-07-18.md (ledger 9) — the apply is now proven in both directions on a normal-sized box: customer zero shrank 11675 → 8192 MB at 16:50:22 and grew 8192 → 12288 MB at 17:02:17, each a live cgroup apply with no reboot (local-api: guest-memory resized in the agent journal, [web] memory resized in the controller log), and the new total rippled into the deploy page's memory math at 17:05:15 (total=12288MB). F5 auto-sizing had landed the guest at 11675 MB. Controller-direct (R-24's hub-desired-state framing SUPERSEDED, Viktor 2026-07-17). Agent GET/POST /guest/memory enforces every bound FRESH (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)) + verify-after-apply; PVE SetConfig hot-applies (Phase-0 PROVEN on the nested box: maxmem moves with the guest running, /proc/meminfo ripples via lxcfs, no reboot). Controller "Szerver memória (RAM)" card + code→Hungarian map, gated on FeatureGuestMemoryResize (MinAgent 0.90.0). LIVE-validated end-to-end through the real endpoint on the demo (above_max + below_min refusals render the Hungarian, agent English never leaks; SupportYes via the version header)

Gap / roadmap (as written): Row complete as of the 2026-07-18 rehearsal — the refusals were proven on the nested demo, the applies on the N100. Cores stay observation


G4 — Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules

Map row: G4 · former map line 228.

Scenario (as written): Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules

Components (as written): hub v0.45/0.53, agent

Status (as written): PARTIAL

Evidence and gap (as written — this row's text carries a | outside code, so its last cells are kept together as they stood): runbooks/publish-train-rules.md; demo-fleet updates proven | Box-side floor lift PROVEN-LIVE on a fresh install (tests/VALIDATION-n100-rehearsal-2026-07-18.md): the day-0 golden deployed controller 0.143.0 at 16:32:28 and the managed floor lifted it to 0.145.0 by 16:32:34 — a 5-second, fully unattended update inside the first minute of controller life, update-state.json recording initiated_by: auto-floor with controller_updated pushed to the hub. So the mechanism is no longer nested-only. Still never proven on a real REMOTE customer — parked trains RUNBOOK-publish-0.79/0.81/0.85-* await Peti → R-1. Action before first invite: rebuild the golden to 0.145.x now that this evidence is banked, so fresh boxes don't sit two versions stale | Fresh-box floor lift proven 2026-07-21 (demo-hp): a brand-new box came up on golden 0.153.0 and self-updated to the fleet floor 0.156.0 during day-0, unattended — the floor mechanism works on first contact, not only on boxes that have been in the fleet a while. That is the R-23(a) single-swap behaviour observed on a machine with no history at all.


G5 — Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed

Map row: G5 · former map line 229.

Scenario (as written): Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed

Components (as written): agent v0.70+

Status (as written): PROVEN-LIVE (demo)

Evidence (as written): SPIKE-agent-selfupdate-2026-07-05

Gap / roadmap (as written): Remote-customer proof pending → R-1


G6 — Controller self-update: anonymous registry, no credentials in guest

Map row: G6 · former map line 230.

Scenario (as written): Controller self-update: anonymous registry, no credentials in guest

Components (as written): controller v0.112

Status (as written): PROVEN-LIVE (demo)

Evidence (as written): 07-10 arc

Gap / roadmap (as written):


G7 — Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue

Map row: G7 · former map line 231.

Scenario (as written): Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue

Components (as written): hub v0.37–0.39

Status (as written): PROVEN-LIVE

Evidence (as written): VALIDATION-offsite-provisioning-e2e-2026-07-09, SPIKE-hetzner-api-provisioning-2026-07-09

Gap / roadmap (as written):


G8 — Per-customer offsite fill + staleness + freeze lever

Map row: G8 · former map line 232.

Scenario (as written): Per-customer offsite fill + staleness + freeze lever

Components (as written): hub v0.41

Status (as written): IMPLEMENTED

Evidence (as written): OffsiteChecker (hub/internal/monitor/offsite.go): fill 90/95% vs soft quota, staleness >48h

Gap / roadmap (as written): No live-fired leg: CAMPAIGN-offsite-overnight-2026-07-10 recorded no quota/fill/staleness emails, and the freeze write-block was inconclusive (only the Hetzner readonly:true API op succeeded). Demoted


G9 — Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert)

Map row: G9 · former map line 233.

Scenario (as written): Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert)

Components (as written): hub v0.64.0 (R-5)

Status (as written): IMPLEMENTED (data pipeline PROVEN-LIVE)

Evidence (as written): monitor.OffsiteBoxChecker — fetch-throttled Hetzner GET (1/15 min), fill (used/storage_box_type.size, 80/90%) + oversubscription (Σ shared+enabled ConfigJSON quotas / capacity, 2.0×), escalation-only operator alert on the customer-less "pool-box" scope; Offsite-tab panel + dashboard tile. Phase-0-pinned live shape (box 611714) + live-computed in-cluster: 0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x. Tests + 4 red-proofs; hub v0.64.0 REPORT

Gap / roadmap (as written): Two open legs: the UI render is unit-verified only (hub UI password-gated → no screenshot); the alert emails are unit + red-proof verified but NOT fired live (real pool nominal — a live-fire emails Viktor). Thresholds pending Viktor's ruling (named config keys). READ-ONLY (GET)


G10 — Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge)

Map row: G10 · former map line 234.

Scenario (as written): Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge)

Components (as written): hub v0.65.0 + tenantsync v1.2.0 (R-5)

Status (as written): IMPLEMENTED (data pipeline PROVEN-LIVE)

Evidence (as written): The PBS DR datastore (felhom-offsite on ep0) fill — NOT a Hetzner box. Option A: a read-only usage op on the felhom-tenantsync ep0 forced command (twin of fingerprint; df on the datastore path — no customer_id, no admin token, NO mutation), polled by monitor.PBSDRBoxChecker (OffsiteBoxChecker clone; 15-min throttle; states ok/unavailable/degraded; fill 80/90% on the "pbsdr-box" operator scope). /offsite split into Restic + PBS DR tabs; two dashboard gauges. Graceful: hub deploy ⟂ ep0 update (ep0 ≤ v1.1.0 → gauge "n/a" until updated). Phase-0-pinned (df on ep0 PBS 4.2.3) + live-computed in-cluster (ep0 updated to v1.2.0 this session): 19.1% full (7.1 GB of 37.2 GB). 10 Go tests + a bash harness + 3 red-proofs; hub v0.65.0 REPORT

Gap / roadmap (as written): Open legs: UI render unit-verified only (hub UI password-gated); the fill alert email is unit + red-proof verified, NOT fired live (datastore nominal at 19%). Separate PBS threshold keys (default 80/90); no oversubscription (namespaces, not quotas). READ-ONLY


G11 — The hub reports LOSS OF VISIBILITY into either off-site store (not just how full it is)

Map row: G11 · former map line 235.

Scenario (as written): The hub reports LOSS OF VISIBILITY into either off-site store (not just how full it is)

Components (as written): hub v0.106.0 (R-339)

Status (as written): IMPLEMENTED — deliberately NOT proven-live

Evidence (as written): Both box checkers count consecutive failed fetch windows and emit pbsdr_box_unreachable / offsite_box_unreachable (severity warning) past a default 3 windows (≈30–45 min), each with a paired *_recovered all-clear routed via recoveredPairedDownTypes — required because the recoveries are severity info, which severityNotifies drops. Scopes stay customer-less (pbsdr-box / pool-box) → operator channel only. Fill logic untouched: a degraded read still drives no band transition. Evidence: internal/monitor/box_reachability_test.go + the cross-package wiring test in internal/notify/, which asserts an actual operator mail rather than a map entry. Filed BECAUSE of a measured gap, not a hypothesis: the 2026-08-18 ep0 outage ran 9 h 37 m with the hub silent

Gap / roadmap (as written): The gap that remains is R-340, and it is not small: the ep0 read is the usage op, which rides the LOCAL API daemon — the daemon that incident explicitly cleared — so this check would have shown GREEN for that entire outage. It closes "ep0 is unreachable as a host"; it does not close what actually happened. No live or constructed outage has exercised the emit path, and one cannot be manufactured against ep0 (Tier 2, protected)


G12 — Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store

Map row: G12 · former map line 236.

Scenario (as written): Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store

Components (as written): hub v0.53, conventions

Status (as written): IMPLEMENTED

Evidence (as written): 07-13 closing bundle

Gap / roadmap (as written):


G13 — Operator login password changeable from UI

Map row: G13 · former map line 237.

Scenario (as written): Operator login password changeable from UI

Components (as written): hub v0.54

Status (as written): IMPLEMENTED

Evidence (as written): 07-13

Gap / roadmap (as written):


G14 — The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail

Map row: G14 · former map line 238.

Scenario (as written): The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail

Components (as written): hub v0.135.0 (R-604, R-530)

Status (as written): IMPLEMENTED — the page PROVEN-LIVE, the alarm and the mail unit-proven

Evidence (as written): audits/hub-safety-2026-10-05/partD/ (live System page: three per-customer floors, Tester 2 0.142.0 → 0.145.0); osupdates/r530_agent_alarm_test.go, web/r604_floor_held_back_test.go

Gap / roadmap (as written): the mail fires only on a GLOBAL raise below an override — not exercised live


G15 — The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops

Map row: G15 · former map line 239.

Scenario (as written): The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops

Components (as written): hub v0.136.0 (R-173), scripts/hub-db-backup/, homelab-manifests rules

Status (as written): PROVEN-LIVE (2026-10-05) — first push, ep0 listing, restore test (4 hosts, 4 sealed, 0 readable), token limits, a key rebuilt from the paper copy decrypts, the saved seal key opens 4/4 console passwords in the restored copy; the alarm by promtool rule test + red-proofs ; 2026-10-09: the whole recovery (§3 steps 1–5) PROVEN on a throwaway k3s — the live hub image started on the restored copy, customers 4/4 and hosts 4/4 equal live, 4/4 console passwords revealed

Evidence (as written): audits/hub-db-offsite-2026-10-05/; audits/dooplex-survival-2026-10-09/partD-*.txt; runbooks/RUNBOOK-hub-db-offsite-backup.md

Gap / roadmap (as written): A restored hub mails households pending notices at once — a test restore has no network (runbook §3)


G16 — Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail

Map row: G16 · former map line 240.

Scenario (as written): Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail

Components (as written): scripts/dooplex-offsite/ (R-232), homelab-manifests rules

Status (as written): PROVEN-LIVE (2026-10-09) — first push 57 s, read back with the read-only token (27 805 files, 10 repos pass git fsck); Gitea restored into a throwaway and started: 10/10 repos, product main = live, a file byte for byte, a login; alarm by promtool + red-proofs; failure mail reached the inbox

Evidence (as written): audits/dooplex-survival-2026-10-09/; runbooks/gitea-restore.md

Gap / roadmap (as written): The container registry is NOT copied (images rebuild from the code); the on-box backup tree is still one writable path (R-232 c); the copy opens only with keys that must be on paper (runbooks/break-glass-sheet.md; the signing keys are in no copy, R-924)


G17 — The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + X-Felhom-Operator)

Map row: G17 · former map line 241.

Scenario (as written): The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + X-Felhom-Operator)

Components (as written): hub v0.135.0 (R-135)

Status (as written): PROVEN-LIVE (2026-10-05)

Evidence (as written): audits/hub-safety-2026-10-05/partA/live.txt (Basic, no header → 403 even on an unknown path; with the header → passes; header without credentials → 401); web/r135_csrf_test.go (39 paths)

Gap / roadmap (as written):


G18 — Box operating-system security updates (Proxmox host, guest Debian, Docker engine)

Map row: G18 · former map line 242.

Scenario (as written): Box operating-system security updates (Proxmox host, guest Debian, Docker engine)

Components (as written): agent v0.143.0, hub v0.133.0

Status (as written): PARTIAL — the GUEST and HOST Debian fast lanes and the DOCKER engine slow lane are PROVEN-LIVE (2026-10-04), with the System page, the fleet view and the alarms; the KERNEL lane is MISSING

Evidence (as written): Guest: audits/os-guest-lane-2026-10-04/. Host + fleet + alarms: audits/os-host-lane-2026-10-04/. Docker + System page + crash guard: audits/os-docker-crash-2026-10-04/ — live-restore on with the same container ids on every box; Docker 29.8.2 on both demo boxes; operator-approved Docker release; a signed undo and a signed ring-1 step; a replay refused; the crash guard restarted demo-hp twice and kept it off the third time. Design architecture/11-os-updates.md §5.8, §5.9, §8

Gap / roadmap (as written): No automatic undo (guest: last night's backup; host: by-hand runbook; Docker: a signed undo job); existing boxes get root-owned files by the signed config bundle since 2026-10-04 (R-840 CLOSED; a box from before agent 0.143.0 needs one by-hand bootstrap — Tester 2: R-862; audits/r840-config-bundle-2026-10-04/); test approvals now end with the test (R-859); the agent's sudoers is root-equivalent (R-861); the kernel lane (R-836); facts reach the hub late after a boot (R-853). 2026-10-05 (agent v0.144.1): R8 measures the real download (R-865); a killed pass still reports (R-868, live); the debug pass runs with the hub away (R-866, live); a power cut mid-update was proven by day on demo-hp — the box came back by itself in 37 s, but the next pass fails until dpkg --configure -a is run by hand (R-876, P2, open) — audits/night-fixes-2026-10-05/. 2026-10-05 afternoon (agent v0.145.0): R-876 FIXED and proven live — after a second crash mid-unpack the next pass repaired dpkg by itself (REPAIR … journal=1) and finished — audits/catchup-2026-10-05/partD/


G19 — An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps

Map row: G19 · former map line 243.

Scenario (as written): An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps

Components (as written): controller v0.259.0 + hub v0.119.0 + ISO 1.29.0 + the whole catalog

Status (as written): PROVEN-LIVE on 0.258.0 with one blocker; THE BLOCKER IS FIXED AND PROVEN, THE WALK IS NOT REPEATED

Evidence (as written): audits/DRILL-first-hour-en-0258-2026-09-20.md — a fresh install 2026-09-20, one intervention (R-494), stop rule not reached. Then audits/i18n-closing-2026-09-21/live/ — the three blockers fixed and each proven on a live box or in the operator's inbox: the claim page answers English through the real cookie path; the Backup page's tier names follow the language; and the setup mail carries four plain-ASCII English words where the drill's carried képző-szkítia-ásatás, one day apart in the same inbox.

Gap / roadmap (as written): R-596, R-597 and R-598 are CLOSED. What this row still does NOT claim: the fixed journey has not been walked end to end by a stranger on a fresh install. Three fixes proven at the endpoint are not an hour proven by a person, and this project's own rule is that fixes are not a journey (see the recovery-journey row). Also not walked: the recovery code (needs ep0), backup/restore/remove/power-cut (proven 2026-09-14), and the two Backup-page warnings themselves — guest 9201 is healthy and a healthy box renders none, so they are covered by handler render tests, not live. Verdict: nothing known now stands between an English-speaking tester and their box — and that is a different sentence from "the walk passed".


G20 — A deletion of a customer's off-site history is NOTICED within a day

Map row: G20 · former map line 244.

Scenario (as written): A deletion of a customer's off-site history is NOTICED within a day

Components (as written): hub v0.111.0 (R-431)

Status (as written): IMPLEMENTED — not yet PROVEN-LIVE

Evidence (as written): 09-01

Gap / roadmap (as written): hub/internal/monitor/offsite.go — third signal beside FILL and STALENESS. On the hub deliberately: a detector on the box is one the deletion can silence. Alarms when the reported count falls by more than HALF and by at least 5, guarded by StatsKnown (R-331), the declared State (R-204) and run success (R-100). Threshold reasoned, not invented: over 12 898 reports every decrease lands on ZERO and predates stats_known; in the 380-report stats_known window there are none. ACCEPTANCE: 9 009 real points replayed → ZERO alarms (offsite_r431_test.go, fixture committed). What PROVEN-LIVE would need and this does NOT have: a real drop observed on a live box producing a real mail — the live firing done at ship time was driven through the hub's own path with synthetic counts, which is an end-to-end delivery proof, not a proof that a genuine deletion is caught.